Replaces the AppBrain scraper (Cloudflare-blocked 403, confirmed dead).
play.google.com server-rendered chart HTML works: live-validated, real
Android packages + names + ratings from both /store/apps/top and
/store/apps/category/<CAT>. CLI: <top|category> [--category CAT]
[--region R] [--limit N] [--html-file F]; output source:"google-play",
entries {rank,name,package,rating}. Package link is the stable field;
title (Epkrse class) and rating are best-effort.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AppBrain is Cloudflare-blocked (403) for free scraping. play.google.com's
own /store/apps/top + /store/apps/category HTML works (validated live 46/46).
Rework Task 1, fetch-play-charts.py CLI/output, and all downstream prose.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NOTE: AppBrain returns a Cloudflare JS challenge (HTTP 403) for
automated requests; live fetch degrades gracefully to an error (the
skill falls back to Apple RSS + web search). Offline test passes
against synthetic fixture — that is the test gate per the brief.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Approved spec for market-research v2: AppBrain Play charts + Play
resolver/saturation scripts + best-effort Google Trends + curated numeric
sources, numeric source-cited scoring, 1-2 verified Play links per candidate,
survivor-only enrichment for efficiency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Six-task TDD plan for the market-research plugin: scaffold + marketplace
entry, fetch-charts.py (App Store RSS), history.py (non-repeat memory),
reference rubrics, 6-phase SKILL.md with clone-app handoff, test aggregator.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Umbrella spec for two new plugins (market-research, hermes-build) plus a
one-branch clone-app Phase 7 edit, chaining discover -> analyze -> build.
Decomposes into three implementation plans built in order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hand-rolled two-step APKCombo curl/grep flow with apkeep
(default source apk-pure): no auth, no JavaScript, handles XAPK split
bundling and retries itself, and the per-mirror HTML parsing is no longer
ours to maintain. The script still prints a stable app.apk/app.xapk path.
- download-apk.sh: invoke apkeep, rename artifact to app.<ext>, clear error
when the binary is missing; CLONE_APP_APKEEP (test stub) and
CLONE_APP_APKEEP_SOURCE (mirror override) env hooks
- test-download-apk.sh: rewritten against an apkeep stub (xapk/apk/fail/
missing-binary cases)
- SKILL.md Phase 1: prose updated for apkeep install + source override
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 2 rewrite: invoke android-reverse-engineering skill in an isolated
subagent (fallback to direct scripts), extract Tier-2 payloads for
auth/payment/core flows, persist a digest contract to $WORK/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The APKPure direct endpoint (d.apkpure.com/b/APK/<pkg>) now returns an HTTP
403 Cloudflare bot challenge for every package, so downloads always failed —
which also blocked the entire reverse-engineering chain downstream (it never
ran because Phase 1 produced no APK).
- download-apk.sh: fetch the APKCombo download page with a browser UA, extract
the embedded /r2?u=<signed-url> link, and download that. No JS needed; the
URL slug is ignored so a literal 'app' works for any package. Verified
end-to-end against com.wonder (Elevate) -> 126MB xapk -> RE fingerprint.
- test-download-apk.sh: model the two-step flow and the page-without-link case.
- resolve-re-scripts.sh: warn (stderr) when bash < 4, since the upstream RE
scripts use ${VAR,,} and fail with 'bad substitution' on macOS bash 3.2.
- SKILL.md / README.md / CLAUDE.md: document the APKCombo source and bash 4+ req.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hardening patch widened STRICT_URL to recover IPv4 literals, apex
2-label domains and internal hosts that the PR's strict-only regex
discarded as collateral while killing Kotlin-stdlib dictionary noise.
Widening alone reopened a narrow noise class: 'word.word' fragments such
as "www.this" / "this.introduction" pass as apex domains.
Keep extraction permissive and add a small awk pass that decides per host:
- IPv4 literal: always keep (dict fragments are words, never dotted-quads)
- >=3 labels: always keep (any TLD; same tolerance as the original regex)
- any host with a :port or /path: always keep (structured = high signal)
- bare 2-label apex: keep only when the TLD is a real one, matched as a
whole field (so "introduction" != "in" — the prefix-match bug a single
mega-regex would have)
Trade-off documented inline: a first-party host referenced bare with an
uncommon TLD (e.g. https://foo.store with no path) is dropped; a path or
port keeps it. awk is POSIX (sub/split/~/print) — more portable than the
bash>=4 'declare -A' already used in the summary header.
Verified: dictionary noise dropped; IPs, apex, internal and subdomain
hosts kept; --all on a zero-match tree still exits 0; host list and
full-URL list stay consistent (no orphan hosts).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR #16 additions were wired into SKILL.md and references/ but the
human-facing README was never updated. Surface them, with prominent
emphasis on first-class Kotlin support:
- Top blurb: callout for R8 Kotlin name recovery + Ktor/Apollo/Koin
- "What it does" table: Phase 0 fingerprint, Kotlin name recovery,
modern Kotlin/KMP stacks (Ktor, Apollo, Koin, HMAC)
- Usage: fingerprint.sh example, --ktor/--apollo/--paths flags, and a
dedicated "Kotlin name recovery (R8 deobfuscation)" subsection
- Repository Structure: add the three new scripts + two new references
- Acknowledgments: credit @tajchert (#16)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- find-api-calls.sh: add missing '|| true' on the --paths inventory and
--urls extraction pipelines; with set -euo pipefail a zero-match grep
aborted the whole script (including the default --all run) with exit 1.
- find-api-calls.sh: widen STRICT_URL to also match IPv4 literals, apex
2-label domains and bare single-label hosts followed by :port or /path
(localhost, internal backends) while still rejecting dictionary-fragment
noise from the Kotlin stdlib.
- recover-kotlin-names.sh: sanitize the by_package/ filename with
os.path.basename; a crafted absolute path in untrusted @DebugMetadata
package names could otherwise escape the output directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two small changes that together meaningfully reduce wasted effort:
1. Phase 3 now explicitly tells the agent to read every BuildConfig.java.
These files are almost never obfuscated and routinely contain the
single highest-signal constants in the APK — base URLs, flavor names,
build types, third-party API keys, feature flags. They were not
mentioned in the previous workflow despite being the cheapest possible
high-value target. One grep, finds them all.
2. The Phase 5 documentation template was a single per-endpoint block
asking for path params, query params, request body, response type,
and call chain. On apps with 100+ endpoints that easily becomes hours
of work for output the consumer will not read.
Replace it with two tiers:
* Tier 1 — flat table covering every endpoint (host, method, path,
auth required, source file). Always produced. Takes ~5 minutes
from the --paths output.
* Tier 2 — the existing detailed block, but explicitly reserved for
high-value endpoints: the entire auth flow, payment/checkout, and
anything the user specifically asked about. Default cap of ~10
Tier-2 entries unless asked for more.
This matches the natural shape of how analysts actually use this work
(one inventory table to know the surface area, plus a deep dive on
auth and a couple of flows) and prevents over-investment in detail
for endpoints nobody will read about.
Without an overview the script dumps thousands of file:line: matches
across many sections, leaving the reader to figure out which framework
even applies. A short summary at the top makes the rest of the output
actionable.
The summary counts hits per framework / DI / auth-signal category in a
single grep pass over the source tree (8 separate greps would have
roughly octupled the runtime on a large decompile). Output is a 3-line
table:
HTTP framework: Retrofit=N OkHttp=N Ktor=N Apollo=N Volley=N
DI framework: Hilt/Dagger=N Koin=N
Auth signals: Bearer=N HMAC/Sign=N
A reader can immediately see which framework the app actually uses,
whether auth is bearer-token or signed, and whether to spend time on a
section or skip it. The summary is suppressed when a single section flag
(--retrofit, --ktor, --paths, ...) is given, so the existing single-section
workflows are unchanged.
A reminder of the available section flags is printed below the counts
so the agent does not have to consult --help.
Two gaps in the previous coverage:
1. Koin was not mentioned anywhere — Hilt/Dagger got a full section in
call-flow-analysis.md but Koin (the dominant DI in KMP and a large
share of Kotlin-only Android apps) had zero patterns. Add a Koin
subsection with the runtime-DSL patterns (module {}, single<>,
factory<>, viewModel<>, by inject, by viewModel) plus the practical
trick for resolving an interface to its impl after R8 obfuscation:
intersect "files that import org.koin.core.module" with "files that
reference the interface name".
2. The --auth mode caught Bearer / API-key / OAuth header patterns but
missed HMAC and other request-signing schemes. A hardcoded HMAC
secret embedded in an APK is a security finding worth surfacing —
the same kind of authority the user gets is the same authority a
decompiler grants to anyone. Add patterns for:
* JCA primitives: HmacSHA{1,256,512}, Mac.getInstance(...),
SecretKeySpec(...), Signature.getInstance(...)
* Header conventions: X-Signature, X-Hmac, X-Amz-Signature,
X-Client-Authorization, AWS4-HMAC, signRequest(), signaturev2/3
* Likely secret-bearing identifiers: app_secret, client_secret,
signing_key, hmac_secret, consumer_secret, private_key
* Ktor BearerTokens / loadTokens / refreshTokens DSL
These survive R8 because the JCA and Ktor APIs are public and not
shrunk. On a real-world app with a homegrown HMAC scheme they pinpoint
the signing class and its hardcoded key directly.
The previous --urls mode was a plain grep for "https?://..." which on a
real APK produced thousands of lines, half of them junk strings extracted
from Kotlin stdlib's compression dictionary ("http://An Introduction to..."
fragments) and the other half SDK URLs (Google, Firebase, AppsFlyer,
Datadog, Sentry, ...) that the analyst is not looking for. The signal —
first-party backend hosts — was buried.
Two changes:
1. Strict URL regex: hostname must have at least one dot and end in a 2+
letter TLD, with no whitespace / angle brackets / non-printables in the
path. This eliminates the dictionary-fragment noise.
2. Bucket the surviving URLs into "likely first-party" vs "third-party"
using references/third_party_hosts.txt — a curated denylist of
~80 patterns covering Google/Firebase/Apple/Microsoft/Adobe, attribution
and observability vendors (AppsFlyer, Datadog, Sentry, Bugsnag, ...),
payments (Stripe, PayU, Adyen, ...), support/chat SDKs, CAs, and
standards namespaces (w3.org, etc.).
The new output starts with a frequency-sorted list of likely first-party
hosts — which is the artifact every reverse-engineer wants on the first
page — followed by the collapsed third-party list and the full URL set
for first-party hosts only.
The denylist is a sidecar text file (one regex per line) so users can
extend or override it without editing the script.
fatih.bulut
masa2146
Simone Avogadro
Michał Tajchert