diff --git a/docs/superpowers/plans/2026-06-23-market-research-v2.md b/docs/superpowers/plans/2026-06-23-market-research-v2.md index 0e52bab..3d431c7 100644 --- a/docs/superpowers/plans/2026-06-23-market-research-v2.md +++ b/docs/superpowers/plans/2026-06-23-market-research-v2.md @@ -4,7 +4,7 @@ **Goal:** Upgrade the `market-research` skill to produce numerically-grounded, source-cited clone candidates that each carry 1–2 verified Google Play Store links — all free, no API keys. -**Architecture:** Add three deterministic scraper scripts (AppBrain Play charts, a Play resolver/saturation tool, a best-effort Google Trends fetcher) plus a curated numeric-sources rubric. Rewrite the scoring guide to require numeric evidence, the report template to carry Play links + citations + saturation, and the SKILL orchestrator to a survivor-only-enrichment phase flow. Each scraper is fixture-tested offline; reality is confirmed by a one-time live-fetch validation step. +**Architecture:** Add three deterministic scraper scripts (play.google.com top/category charts, a Play resolver/saturation tool, a best-effort Google Trends fetcher) plus a curated numeric-sources rubric. Rewrite the scoring guide to require numeric evidence, the report template to carry Play links + citations + saturation, and the SKILL orchestrator to a survivor-only-enrichment phase flow. Each scraper is fixture-tested offline; reality is confirmed by a one-time live-fetch validation step. **Tech Stack:** Python 3 stdlib only (`urllib`, `json`, `re`, `subprocess`/`curl` SSL fallback), bash 4+, Markdown skill/reference docs. @@ -26,14 +26,14 @@ ## File Structure **Create:** -- `plugins/market-research/skills/market-research/scripts/fetch-play-charts.py` — AppBrain HTML top-charts → normalized JSON (real Android packages). +- `plugins/market-research/skills/market-research/scripts/fetch-play-charts.py` — play.google.com top/category HTML → normalized JSON (real Android packages). - `plugins/market-research/skills/market-research/scripts/play.py` — `resolve` (name → Play URL + stats) and `count` (saturation) subcommands. - `plugins/market-research/skills/market-research/scripts/trends.py` — best-effort Google Trends, never hard-fails. - `plugins/market-research/skills/market-research/references/numeric-sources.md` — curated free numeric-data sources + query/extraction rubric. - `plugins/market-research/tests/test-fetch-play-charts.py` - `plugins/market-research/tests/test-play.py` - `plugins/market-research/tests/test-trends.py` -- `plugins/market-research/tests/fixtures/appbrain-popular.html` +- `plugins/market-research/tests/fixtures/play-chart.html` - `plugins/market-research/tests/fixtures/play-search.html` - `plugins/market-research/tests/fixtures/play-details.html` - `plugins/market-research/tests/fixtures/trends-sample.json` @@ -51,38 +51,31 @@ --- -## Task 1: AppBrain Play charts scraper +## Task 1: play.google.com charts scraper + +**Context:** AppBrain (the brainstorm's first choice) is Cloudflare-blocked (HTTP 403) for free scraping. `play.google.com`'s own server-rendered chart HTML works (validated live: 46/46 real Android packages from a category page). This scrapes `/store/apps/top` and `/store/apps/category/`. **Files:** - Create: `plugins/market-research/skills/market-research/scripts/fetch-play-charts.py` -- Create: `plugins/market-research/tests/fixtures/appbrain-popular.html` +- Create: `plugins/market-research/tests/fixtures/play-chart.html` - Test: `plugins/market-research/tests/test-fetch-play-charts.py` **Interfaces:** - Consumes: nothing. -- Produces: CLI `fetch-play-charts.py [--region R] [--limit N] [--html-file F]` where ` ∈ {popular, top-grossing, top-new}`. Stdout JSON: `{"source":"appbrain","chart":str,"region":str,"count":int,"entries":[{"rank":int,"name":str,"developer":str|None,"category":str|None,"package":str,"rating":float|None,"installs":str|None}]}`. +- Produces: CLI `fetch-play-charts.py [--category CAT] [--region R] [--limit N] [--html-file F]` where ` ∈ {top, category}` (`category` requires `--category`, e.g. `PRODUCTIVITY`). Stdout JSON: `{"source":"google-play","chart":str,"region":str,"count":int,"entries":[{"rank":int,"name":str,"package":str,"rating":float|None}]}`. `chart` label is `"top"` or `"category:"`. `rating` is best-effort (may be `None`). - [ ] **Step 1: Create the synthetic fixture** -Create `plugins/market-research/tests/fixtures/appbrain-popular.html` with markup matching AppBrain's app-list rows (a link to `/app//` carrying name, developer, rating, installs): +Create `plugins/market-research/tests/fixtures/play-chart.html` mirroring real Play app-card markup — a `details?id=` anchor wrapping the icon, then a `class="Epkrse …"` title div, then a star-rating aria-label. (Note the trailing space inside `class="Epkrse "` — that is the real markup; the parser must tolerate it. Card 3 repeats app 1's package to exercise dedup.) ```html -
-
- Habit Tracker - Focus Labs - Productivity - 4.6 - 5,000,000+ -
-
- Budget Buddy - Money Inc - Finance - 4.2 - 1,000,000+ -
+
+
Habit Tracker
 +
+
Budget Buddy
 +
+
Habit Tracker
``` @@ -97,11 +90,11 @@ import json, subprocess, sys, os HERE = os.path.dirname(os.path.abspath(__file__)) SCRIPT = os.path.join(HERE, "..", "skills", "market-research", "scripts", "fetch-play-charts.py") -FIXTURE = os.path.join(HERE, "fixtures", "appbrain-popular.html") +FIXTURE = os.path.join(HERE, "fixtures", "play-chart.html") def run(): out = subprocess.check_output( - [sys.executable, SCRIPT, "popular", "--region", "us", "--html-file", FIXTURE]) + [sys.executable, SCRIPT, "top", "--region", "US", "--html-file", FIXTURE]) return json.loads(out) def main(): @@ -111,20 +104,19 @@ def main(): print(f"{'PASS' if cond else 'FAIL'}: {name}") if not cond: fails.append(name) - check("source", d["source"] == "appbrain") - check("chart", d["chart"] == "popular") - check("count", d["count"] == 2) + check("source", d["source"] == "google-play") + check("chart", d["chart"] == "top") + check("region", d["region"] == "US") + check("count (dedup app1 repeat)", d["count"] == 2) e0 = d["entries"][0] check("rank 1", e0["rank"] == 1) check("name", e0["name"] == "Habit Tracker") check("package", e0["package"] == "com.example.habit") - check("developer", e0["developer"] == "Focus Labs") - check("category", e0["category"] == "Productivity") check("rating", e0["rating"] == 4.6) - check("installs", e0["installs"] == "5,000,000+") - for k in ["rank", "name", "package", "developer", "category", "rating", "installs"]: + for k in ["rank", "name", "package", "rating"]: check(f"key present: {k}", k in e0) check("second package", d["entries"][1]["package"] == "com.example.budget") + check("second name", d["entries"][1]["name"] == "Budget Buddy") sys.exit(1 if fails else 0) main() @@ -141,22 +133,20 @@ Create `plugins/market-research/skills/market-research/scripts/fetch-play-charts ```python #!/usr/bin/env python3 -"""Fetch an AppBrain Play-store top-chart into a normalized JSON list. +"""Fetch a play.google.com chart page into a normalized JSON list. -Google Play has no public chart feed and renders via obfuscated batchexecute -JS. AppBrain (appbrain.com) publishes server-rendered HTML top-chart pages with -REAL Android package names. This scrapes those rows. Stdlib-only, no pip. +Google Play has no public chart feed, but its server-rendered HTML for +/store/apps/top and /store/apps/category/ carries ranked app cards with +REAL Android package names (unlike Apple's RSS iOS bundle ids). Each card is an + wrapping the icon, then a +
NAME
title, then a star aria-label. Play obfuscates +the title class, so the package LINK (stable) is the load-bearing field; name +and rating are best-effort. Stdlib-only, no pip. -Unlike Apple's RSS (iOS bundle ids), entries here carry Android packages usable -directly for a clone-app handoff. +(AppBrain, the original plan's source, is Cloudflare-blocked 403 — dropped.) """ -import sys, json, re, argparse, urllib.request, ssl, subprocess, shutil +import sys, json, re, html as _html, argparse, urllib.request, ssl, subprocess, shutil -CHARTS = { - "popular": "https://www.appbrain.com/apps/popular/", - "top-grossing": "https://www.appbrain.com/apps/highest-grossing/", - "top-new": "https://www.appbrain.com/apps/new/", -} UA = "Mozilla/5.0" def _http_get(url): @@ -175,58 +165,66 @@ def _http_get(url): raise return out.stdout.decode("utf-8", "replace") -# One app row: a link to /app// followed (within the same row -# block) by developer / category / rating / installs spans. -ROW = re.compile( - r'href="/app/[^"/]+/(?P[\w.]+)"[^>]*>(?P[^<]+)' - r'(?P.*?)(?=href="/app/|
\s*|$)', re.DOTALL) -DEV = re.compile(r'class="developer"[^>]*>([^<]+)<') -CAT = re.compile(r'class="category"[^>]*>([^<]+)<') -RAT = re.compile(r'class="rating"[^>]*>\s*([\d.]+)\s*<') -INST = re.compile(r'class="installs"[^>]*>\s*([\d,]+\+)\s*<') +LINK = re.compile(r'href="/store/apps/details\?id=([a-zA-Z0-9._]+)"') +NAME = re.compile(r'class="Epkrse[^"]*"[^>]*>([^<]+)<') +# rating is best-effort: search-style aria-label OR a star-icon followed by a number +RATE_ARIA = re.compile(r'aria-label="Rated\s+([\d.]+)\s+star') +RATE_STAR = re.compile(r'>star]+>\s*([\d.]+)') -def _first(rx, text): - m = rx.search(text) - return m.group(1).strip() if m else None +def _chart_url(chart, category, region): + if chart == "category": + return f"https://play.google.com/store/apps/category/{category}?hl=en&gl={region}" + return f"https://play.google.com/store/apps/top?hl=en&gl={region}" -def parse(html, chart, region, limit): +def parse(html, chart_label, region, limit): entries = [] - for i, m in enumerate(ROW.finditer(html), start=1): - if i > limit: - break - rest = m.group("rest") - rating = _first(RAT, rest) + seen = set() + for m in LINK.finditer(html): + pkg = m.group(1) + if pkg in seen: + continue + win = html[m.start():m.start() + 1600] # one card's worth of markup + nm = NAME.search(win) + if not nm: + continue # icon-only anchor with no title nearby; skip + seen.add(pkg) + rt = RATE_ARIA.search(win) or RATE_STAR.search(win) entries.append({ - "rank": i, - "name": m.group("name").strip(), - "developer": _first(DEV, rest), - "category": _first(CAT, rest), - "package": m.group("package"), - "rating": float(rating) if rating else None, - "installs": _first(INST, rest), + "rank": len(entries) + 1, + "name": _html.unescape(nm.group(1).strip()), + "package": pkg, + "rating": float(rt.group(1)) if rt else None, }) - return {"source": "appbrain", "chart": chart, "region": region, + if len(entries) >= limit: + break + return {"source": "google-play", "chart": chart_label, "region": region, "count": len(entries), "entries": entries} def main(): ap = argparse.ArgumentParser() - ap.add_argument("chart", choices=sorted(CHARTS)) - ap.add_argument("--region", default="us") + ap.add_argument("chart", choices=("top", "category")) + ap.add_argument("--category") + ap.add_argument("--region", default="US") ap.add_argument("--limit", type=int, default=25) ap.add_argument("--html-file") args = ap.parse_args() + if args.chart == "category" and not args.category: + print("ERROR: --category is required for the 'category' chart", file=sys.stderr) + sys.exit(2) + chart_label = f"category:{args.category}" if args.chart == "category" else "top" + if args.html_file: with open(args.html_file, encoding="utf-8") as f: html = f.read() else: try: - html = _http_get(CHARTS[args.chart]) + html = _http_get(_chart_url(args.chart, args.category, args.region)) except Exception as e: - print(f"ERROR: failed to fetch AppBrain chart: {e}", file=sys.stderr) + print(f"ERROR: failed to fetch Play chart: {e}", file=sys.stderr) sys.exit(1) - print(json.dumps(parse(html, args.chart, args.region, args.limit), indent=2)) + print(json.dumps(parse(html, chart_label, args.region, args.limit), indent=2)) if __name__ == "__main__": main() @@ -241,18 +239,18 @@ Run: `chmod +x plugins/market-research/skills/market-research/scripts/fetch-play Run: `python3 plugins/market-research/tests/test-fetch-play-charts.py` Expected: every line `PASS`, exit 0. -- [ ] **Step 7: Validate the selectors against live AppBrain once** +- [ ] **Step 7: Validate the selectors against live Play once** -Run: `python3 plugins/market-research/skills/market-research/scripts/fetch-play-charts.py popular --limit 5` -Expected: JSON with ~5 entries carrying real `com.*` packages. **If the live markup differs from the fixture** (empty `entries`), update both the regexes in the script AND `appbrain-popular.html` to match the real structure, then re-run Step 6 until the offline test passes against the corrected fixture. If AppBrain blocks/changes hosting, record the failure in the commit message and keep the script (the skill degrades to Apple RSS + web). +Run: `python3 plugins/market-research/skills/market-research/scripts/fetch-play-charts.py category --category PRODUCTIVITY --limit 5` +Expected: JSON with ~5 entries carrying real `com.*` packages and names (e.g. ChatGPT, Google Keep). Also smoke `top`: `... top --limit 5`. **If live `entries` come back empty** (Play rotated the obfuscated `Epkrse` title class), find the new title class in the live HTML, update `NAME` in the script AND the `class="Epkrse "` strings in `play-chart.html` to match, then re-run Step 6 until the offline test passes against the corrected fixture. The package link is stable, so an empty result means the title-class selector — not the link — drifted. If Play is unreachable from this environment, keep the script as-is, note it in the report, and still commit (the offline test must pass regardless). - [ ] **Step 8: Commit** ```bash git add plugins/market-research/skills/market-research/scripts/fetch-play-charts.py \ plugins/market-research/tests/test-fetch-play-charts.py \ - plugins/market-research/tests/fixtures/appbrain-popular.html -git commit -m "feat(market-research): add AppBrain Play-charts scraper" + plugins/market-research/tests/fixtures/play-chart.html +git commit -m "feat(market-research): add play.google.com charts scraper" ``` --- @@ -624,7 +622,7 @@ def main(): check("trend_pct", d["trend_pct"] == 100.0) # (100-50)/50*100 # bad fixture -> graceful fallback, still exit 0 - rc2, d2 = run("x", "--json-file", os.path.join(HERE, "fixtures", "appbrain-popular.html")) + rc2, d2 = run("x", "--json-file", os.path.join(HERE, "fixtures", "play-chart.html")) check("fallback exit 0", rc2 == 0) check("fallback ok false", d2["ok"] is False) check("fallback flag", d2["fallback"] == "websearch") @@ -775,7 +773,7 @@ Every number that lands in the report carries the URL it came from. | Source | Pull | How (free) | |---|---|---| | **Google Trends** | interest-over-time, % momentum, breakout/rising queries | `trends.py ""`; on `{"ok":false}` fall back to WebSearch `google trends `. | -| **AppBrain** | install brackets, category rank/share, Android growth | `fetch-play-charts.py` for charts; WebSearch `appbrain statistics` for detail pages. | +| **play.google.com** | ranked Android packages, per-app installs/rating | `fetch-play-charts.py` for top/category charts; `play.py resolve` for per-app installs/rating. | | **Sensor Tower (blog/reports)** | downloads, revenue, DAU, YoY growth | WebSearch `sensortower revenue downloads 2026`; WebFetch the article; quote the figure + date. | | **data.ai / Apptopia posts** | top-charts movement, market revenue | WebSearch `data.ai market 2026`; WebFetch + cite. | | **Statista (free public charts)** | market size $, user counts, CAGR | WebSearch `statista market size`; use the visible free figure only; cite. | @@ -849,7 +847,7 @@ or `null`. A field that is `null` caps that subscore at 60. | Subscore | Evidence that lifts the cap above 60 | |---|---| | Cloneability | a stack/complexity signal (e.g. "few endpoints — RE later"); no external number required, but state the basis. | -| Market opportunity | an installs figure (AppBrain/Play) AND either a Trends `trend_pct` OR a saturation `app_count`. | +| Market opportunity | an installs figure (Play via `play.py resolve`) AND either a Trends `trend_pct` OR a saturation `app_count`. | | Monetization fit | a category ARPU/revenue figure (Sensor Tower/data.ai/Statista) OR top-grossing chart presence. | | Niche gap | a region/language gap signal (saturation `app_count` low in region, or no localized incumbent). | @@ -914,7 +912,7 @@ Every number carries a source link. Every candidate carries 1–2 Google Play li ## Run parameters - Angles this run: - Focus argument: -- Sources: Apple RSS () + AppBrain Play charts () + web numeric () + Trends () +- Sources: Apple RSS () + play.google.com charts () + web numeric () + Trends () - Candidates after history exclusion: (history had prior suggestions) ## Top candidates (ranked) @@ -984,14 +982,17 @@ python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-charts.py \ python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-charts.py \ topgrossingapplications --region --limit 25 > "$WORK/charts--grossing.json" ``` -AppBrain Play charts (Android signal — REAL packages): +play.google.com charts (Android signal — REAL packages). Pull the overall top +plus one category page per chosen category (Play category code, e.g. +`PRODUCTIVITY`, `GAME_PUZZLE`, `FINANCE`): ```bash python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-play-charts.py \ - popular --limit 25 > "$WORK/play-popular.json" + top --region --limit 25 > "$WORK/play-top-.json" python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-play-charts.py \ - top-grossing --limit 25 > "$WORK/play-grossing.json" + category --category --region --limit 25 > "$WORK/play-.json" ``` -If any fetch fails, note it and continue with the feeds you got. +If any fetch fails (Play may rotate its obfuscated title class), note it and +continue with the feeds you got — Apple RSS + web signal still stand. ## Phase 2: Trend + numeric signal Read `${CLAUDE_PLUGIN_ROOT}/skills/market-research/references/numeric-sources.md`. @@ -1005,10 +1006,11 @@ If a `trends.py` result is `{"ok": false, "fallback": "websearch"}`, WebSearch t same momentum signal instead. Vary queries by the run's angles. ## Phase 3: Synthesize ≥12 candidates -Cluster chart entries (Apple + AppBrain) + web findings into ≥12 distinct ideas -(synthesize > 10 so dedup still leaves ≥10). For each: name, category, -what-it-does, why-now (with a cited number where available), incumbent(s), -monetization model. AppBrain entries already give an Android `package`; carry it. +Cluster chart entries (Apple RSS + play.google.com) + web findings into ≥12 +distinct ideas (synthesize > 10 so dedup still leaves ≥10). For each: name, +category, what-it-does, why-now (with a cited number where available), +incumbent(s), monetization model. play.google.com entries already give an +Android `package`; carry it. Write the working list as a JSON array (objects with at least `name`, optional `package`, `category`) to `$WORK/candidates.json`. @@ -1066,7 +1068,7 @@ stands on its own. | Scenario | Action | |---|---| | `fetch-charts.py` / `fetch-play-charts.py` fails | note it, continue with other feeds/web search | -| AppBrain blocks or markup changed | continue on Apple RSS + web signal | +| play.google.com chart fetch fails / title class rotated | continue on Apple RSS + web signal; package link stays stable | | `play.py resolve` finds no package | flag candidate "Play link unresolved", skip handoff | | Play link fails WebFetch verify | drop that link; if none verify, flag unresolved | | `trends.py` returns `{"ok": false}` | WebSearch the same momentum signal | @@ -1154,7 +1156,7 @@ git commit -m "test(market-research): cover v2 scripts in smoke; update README" ## Self-Review Notes (for the executor) -- **Spec coverage:** AppBrain charts → Task 1; Play resolver + links → Task 2; saturation → Task 3; Trends best-effort → Task 4; numeric sources → Task 5; numeric scoring → Task 6; report links/citations/saturation → Task 7; survivor-only phase flow + WebFetch verify → Task 8; smoke/README/full-suite + RE-untouched gate → Task 9. +- **Spec coverage:** play.google.com charts → Task 1; Play resolver + links → Task 2; saturation → Task 3; Trends best-effort → Task 4; numeric sources → Task 5; numeric scoring → Task 6; report links/citations/saturation → Task 7; survivor-only phase flow + WebFetch verify → Task 8; smoke/README/full-suite + RE-untouched gate → Task 9. - **Live-markup risk** is handled by the per-scraper "validate against live once" step (Tasks 1 Step 7, 2 Step 8); fix script + fixture together if reality differs, keep the offline test green. - **No new pip deps, no API keys, all stdlib** — every script mirrors `fetch-charts.py`'s `_http_get` fallback pattern. - **`trends.py` never hard-fails** — verified by the fallback assertions in Task 4 test. diff --git a/docs/superpowers/specs/2026-06-23-market-research-v2-design.md b/docs/superpowers/specs/2026-06-23-market-research-v2-design.md index 25f24a4..4cbe91f 100644 --- a/docs/superpowers/specs/2026-06-23-market-research-v2-design.md +++ b/docs/superpowers/specs/2026-06-23-market-research-v2-design.md @@ -30,7 +30,7 @@ method that actually works free: | Source | Method | Why | |---|---|---| | Apple App Store RSS | existing `fetch-charts.py` | Public no-auth JSON feed. Keep. | -| Google Play charts | **AppBrain HTML** via new `fetch-play-charts.py` | Play itself has no public feed and renders via obfuscated `batchexecute` JS. AppBrain publishes scrapeable server-rendered HTML top-charts with real Android packages. | +| Google Play charts | **`play.google.com` HTML** via new `fetch-play-charts.py` | Play has no public JSON feed, but its server-rendered HTML for `/store/apps/top` and `/store/apps/category/` returns ~45–70 ranked app cards carrying real Android packages + names (validated live: 46/46). AppBrain — the original plan's source — is Cloudflare-blocked (HTTP 403) for free scraping, so it was dropped. | | Play link + stats per candidate | new `play.py resolve` | `play.google.com/store/search?q=…&c=apps` is server-rendered enough to grep the first `details?id=` link; the details page carries rating/installs/last-updated in embedded `ld+json` (same parse clone-app's `scrape-play-store.py` already uses). | | Saturation / competition density | new `play.py count` | Count distinct `details?id=` links + their ratings on the first Play search results page. Approximate but real. | | Google Trends | best-effort `trends.py` + WebSearch fallback | No key, but the unofficial endpoint needs a token dance and is fragile under stdlib-only. Script is best-effort and **never hard-fails**; on any failure the skill falls back to WebSearch for the same signal. | @@ -44,12 +44,16 @@ helper scripts; judgment steps follow reference rubrics. ### New scripts (`skills/market-research/scripts/`) **`fetch-play-charts.py`** -- Input: a chart kind (`popular` / `top-grossing` / `top-new`, mapped to AppBrain - paths), `--region` (where AppBrain supports it), `--limit`, `--html-file` (offline test). -- Output: normalized JSON `{ "source": "appbrain", "chart": ..., "count": N, - "entries": [ {rank, name, developer, category, package, rating, installs} ] }`. +- Input: a chart kind (`top` = `play.google.com/store/apps/top`, or `category` + with `--category ` = `/store/apps/category/`), `--region` (Play `gl` + code), `--limit`, `--html-file` (offline test). +- Output: normalized JSON `{ "source": "google-play", "chart": ..., "region": R, + "count": N, "entries": [ {rank, name, package, rating} ] }`. (`rating` is + best-effort and may be `None`; rank+name+package is the load-bearing signal.) - Mirrors `fetch-charts.py`'s shape and its `curl` SSL fallback. -- Entries carry **real Android `package`s** (unlike Apple's iOS bundle ids). +- Entries carry **real Android `package`s** (unlike Apple's iOS bundle ids), + parsed from Play app cards: `href="/store/apps/details?id=PKG"` + + `class="Epkrse …">NAME` (validated live: 46/46 on a category page). **`play.py`** — two subcommands, both with `--html-file` for offline tests: - `resolve ""` → scrape Play search, take the top apps hit, fetch its @@ -97,7 +101,7 @@ not all raw candidates. ``` Phase 0 Seed rotation (unchanged) -Phase 1 Gather charts Apple RSS (fetch-charts.py) + AppBrain Play +Phase 1 Gather charts Apple RSS (fetch-charts.py) + play.google.com charts (fetch-play-charts.py) per region Phase 2 Trend + numeric WebSearch per numeric-sources.md; trends.py per top themes (fallback to WebSearch on failure) @@ -130,7 +134,9 @@ rule as today's iOS-only case). ## Out of scope - Cross-platform gap detection (iOS-popular / Android-missing) — explicitly cut. -- Scraping play.google.com charts directly — AppBrain used instead. +- AppBrain as a chart source — Cloudflare-blocked (403), dropped for play.google.com. +- Top-grossing Play chart — Play exposes no clean grossing URL without JS; Apple + RSS `topgrossingapplications` carries the monetization-rank signal instead. - Any pip dependency, API key, or paid tier. - Changes outside `plugins/market-research/`. @@ -138,8 +144,8 @@ rule as today's iOS-only case). - `fetch-play-charts.py`, `play.py` (both subcommands), `trends.py` each get an offline fixture test driven by `--html-file` / `--json-file`. New fixtures: - AppBrain chart HTML, a Play search results HTML, a Play details HTML, a Trends - JSON response. + a Play chart-page HTML, a Play search results HTML, a Play details HTML, a + Trends JSON response. - Each scraper's first implementation task: fetch live once, save the fixture, confirm the parser. Then all subsequent tests run offline. - `run-all.sh` and `smoke-structure.sh` extended to cover the new scripts/refs. @@ -147,9 +153,11 @@ rule as today's iOS-only case). ## Risks -- **AppBrain markup drift / blocking.** Mitigation: fixture-driven parser isolates - the brittle selector; the skill notes a failed Play-chart fetch and continues on - Apple RSS + web signal (same degrade-gracefully pattern as today). +- **Play HTML class-name drift** (Play obfuscates classes like `Epkrse`, which can + rotate). Mitigation: fixture-driven parser isolates the brittle selector; the + package link (`/store/apps/details?id=`) is stable even when title classes + rotate, so rank+package survives a class change; the skill notes a failed + Play-chart fetch and continues on Apple RSS + web signal. - **Google Trends fragility.** Mitigation by design: `trends.py` never hard-fails; WebSearch fallback covers the same signal. - **Play search HTML variance by region/locale.** Mitigation: force an `hl=en&gl=US`