Merge pull request #4 from masa2146/feat/market-research-v2
Feat/market research v2
This commit is contained in:
masa2146
GitHub
commit
e0f1865843
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,164 @@
|
||||||
|
# Market Research v2 — Design
|
||||||
|
|
||||||
|
**Date:** 2026-06-23
|
||||||
|
**Plugin:** `plugins/market-research/`
|
||||||
|
**Status:** Approved, pending implementation plan
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Make the `market-research` skill produce higher-quality, numerically-grounded,
|
||||||
|
source-cited clone candidates — entirely free, no API keys. Every candidate must
|
||||||
|
carry 1–2 real Google Play Store links. Scoring must cite real numbers (installs,
|
||||||
|
trend %, competitor count) instead of pure AI judgment.
|
||||||
|
|
||||||
|
## Constraints (unchanged from repo)
|
||||||
|
|
||||||
|
- **Free, no API key, no pip.** Python stdlib only (`urllib`, `json`, `re`,
|
||||||
|
`subprocess` for `curl` SSL fallback). bash 4+ at runtime.
|
||||||
|
- **`plugins/android-reverse-engineering/` is untouched.** This work is confined
|
||||||
|
to `plugins/market-research/`.
|
||||||
|
- **Scrape logic needs a fixture, not a live call.** Each new scraper is tested
|
||||||
|
offline against `tests/fixtures/` via a `--html-file` / `--json-file` flag.
|
||||||
|
- Working dir stays `./work/market-research/` in the user's cwd.
|
||||||
|
- Scripts-for-deterministic / rubrics-for-judgment split is preserved.
|
||||||
|
|
||||||
|
## Honesty on free sources
|
||||||
|
|
||||||
|
Not every requested source is cleanly scrapeable. The design routes each to the
|
||||||
|
method that actually works free:
|
||||||
|
|
||||||
|
| Source | Method | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| Apple App Store RSS | existing `fetch-charts.py` | Public no-auth JSON feed. Keep. |
|
||||||
|
| Google Play charts | **`play.google.com` HTML** via new `fetch-play-charts.py` | Play has no public JSON feed, but its server-rendered HTML for `/store/apps/top` and `/store/apps/category/<CAT>` returns ~45–70 ranked app cards carrying real Android packages + names (validated live: 46/46). AppBrain — the original plan's source — is Cloudflare-blocked (HTTP 403) for free scraping, so it was dropped. |
|
||||||
|
| Play link + stats per candidate | new `play.py resolve` | `play.google.com/store/search?q=…&c=apps` is server-rendered enough to grep the first `details?id=` link; the details page carries rating/installs/last-updated in embedded `ld+json` (same parse clone-app's `scrape-play-store.py` already uses). |
|
||||||
|
| Saturation / competition density | new `play.py count` | Count distinct `details?id=` links + their ratings on the first Play search results page. Approximate but real. |
|
||||||
|
| Google Trends | best-effort `trends.py` + WebSearch fallback | No key, but the unofficial endpoint needs a token dance and is fragile under stdlib-only. Script is best-effort and **never hard-fails**; on any failure the skill falls back to WebSearch for the same signal. |
|
||||||
|
| Statista / Sensor Tower / data.ai / SimilarWeb | curated WebSearch/WebFetch targets in `numeric-sources.md` | Paywalled/JS sites. The free signal is their public blog posts and report summaries, surfaced by WebSearch — not a scraper. |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
Same shape as today: `SKILL.md` orchestrates prose phases; deterministic steps are
|
||||||
|
helper scripts; judgment steps follow reference rubrics.
|
||||||
|
|
||||||
|
### New scripts (`skills/market-research/scripts/`)
|
||||||
|
|
||||||
|
**`fetch-play-charts.py`**
|
||||||
|
- Input: a chart kind (`top` = `play.google.com/store/apps/top`, or `category`
|
||||||
|
with `--category <CAT>` = `/store/apps/category/<CAT>`), `--region` (Play `gl`
|
||||||
|
code), `--limit`, `--html-file` (offline test).
|
||||||
|
- Output: normalized JSON `{ "source": "google-play", "chart": ..., "region": R,
|
||||||
|
"count": N, "entries": [ {rank, name, package, rating} ] }`. (`rating` is
|
||||||
|
best-effort and may be `None`; rank+name+package is the load-bearing signal.)
|
||||||
|
- Mirrors `fetch-charts.py`'s shape and its `curl` SSL fallback.
|
||||||
|
- Entries carry **real Android `package`s** (unlike Apple's iOS bundle ids),
|
||||||
|
parsed from Play app cards: `href="/store/apps/details?id=PKG"` +
|
||||||
|
`class="Epkrse …">NAME</div>` (validated live: 46/46 on a category page).
|
||||||
|
|
||||||
|
**`play.py`** — two subcommands, both with `--html-file` for offline tests:
|
||||||
|
- `resolve "<app name>"` → scrape Play search, take the top apps hit, fetch its
|
||||||
|
details page, print `{name, package, play_url, rating, installs, last_updated,
|
||||||
|
developer}`. Reuses the `ld+json` extraction approach from clone-app's
|
||||||
|
`scrape-play-store.py` (re-implemented locally — market-research stays
|
||||||
|
self-contained per the repo's per-plugin-layout convention; no cross-plugin
|
||||||
|
script dependency).
|
||||||
|
- `count "<query>"` → scrape Play search results, print `{query, app_count,
|
||||||
|
avg_rating, top_packages: [...]}` for saturation. `app_count` is "apps on the
|
||||||
|
first results page", stated as approximate in the report.
|
||||||
|
|
||||||
|
**`trends.py`** (best-effort, never hard-fails)
|
||||||
|
- `"<term>"` → attempt Google Trends interest-over-time + rising queries via the
|
||||||
|
unofficial endpoint (token dance in stdlib). On ANY failure, exit 0 with
|
||||||
|
`{"ok": false, "fallback": "websearch"}` so the skill knows to fall back to
|
||||||
|
WebSearch for the same momentum signal. `--json-file` for offline test of the
|
||||||
|
parser.
|
||||||
|
|
||||||
|
### New reference (`skills/market-research/references/`)
|
||||||
|
|
||||||
|
**`numeric-sources.md`** — for each free numeric source: what it's good for, the
|
||||||
|
exact WebSearch query shape to use, which number to extract, and how to cite it.
|
||||||
|
Covers Google Trends, play.google.com (installs/rating), Statista, Sensor Tower
|
||||||
|
blog, data.ai, SimilarWeb free tier. This is the rubric that turns "go search the web" into "pull these
|
||||||
|
specific numbers from these specific places."
|
||||||
|
|
||||||
|
### Rewritten references
|
||||||
|
|
||||||
|
**`scoring-guide.md` → numeric scoring.** Each subscore must cite at least one
|
||||||
|
number or it is capped low (e.g. a market score >70 requires an installs figure
|
||||||
|
AND either a trend % or a competitor count). Adds a per-subscore "evidence" field
|
||||||
|
to each candidate object. Keeps the existing weights
|
||||||
|
(cloneability 35 / market 35 / monetization 20 / niche 10).
|
||||||
|
|
||||||
|
**`report-template.md`.** Adds: a **Play links** column (1–2 verified links per
|
||||||
|
candidate); **source citations** inline on every "why now" claim and every number;
|
||||||
|
a **saturation** figure per candidate; numeric evidence in the per-candidate
|
||||||
|
detail block.
|
||||||
|
|
||||||
|
### Rewritten orchestrator (`SKILL.md`)
|
||||||
|
|
||||||
|
New phase flow, with an explicit **efficiency rule**: enrich only the survivors,
|
||||||
|
not all raw candidates.
|
||||||
|
|
||||||
|
```
|
||||||
|
Phase 0 Seed rotation (unchanged)
|
||||||
|
Phase 1 Gather charts Apple RSS (fetch-charts.py) + play.google.com
|
||||||
|
charts (fetch-play-charts.py) per region
|
||||||
|
Phase 2 Trend + numeric WebSearch per numeric-sources.md; trends.py per
|
||||||
|
top themes (fallback to WebSearch on failure)
|
||||||
|
Phase 3 Synthesize >=12 cluster charts + signal into ideas; Android
|
||||||
|
packages now available from the Play charts
|
||||||
|
Phase 4 Cheap score + dedup score on chart/trend signal only; history.py
|
||||||
|
filter; loop for more if <10 survive
|
||||||
|
Phase 5 Enrich survivors for the surviving top ~10 ONLY: play.py resolve
|
||||||
|
(1-2 links) + WebFetch verify each link (200 +
|
||||||
|
package match) + play.py count (saturation) +
|
||||||
|
trends.py momentum
|
||||||
|
Phase 6 Re-score numeric apply scoring-guide with real numbers + evidence
|
||||||
|
Phase 7 Present + handoff fill report-template (Play links, citations,
|
||||||
|
saturation); history.py add; resolve picks to
|
||||||
|
Play URL; invoke clone-app skill
|
||||||
|
```
|
||||||
|
|
||||||
|
Efficiency: the expensive per-candidate work (Play resolve, WebFetch verify,
|
||||||
|
saturation, trends) happens in Phase 5 against ~10 survivors, never against the
|
||||||
|
full raw synthesis set.
|
||||||
|
|
||||||
|
## Link verification
|
||||||
|
|
||||||
|
In Phase 5, each resolved Play URL is WebFetched and confirmed (HTTP 200 and the
|
||||||
|
`id=` package present on the page) before it is written into the report. Dead or
|
||||||
|
mismatched links are dropped; a candidate with no verifiable Play link is kept in
|
||||||
|
the report flagged "Play link unresolved" and skipped on clone-app handoff (same
|
||||||
|
rule as today's iOS-only case).
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Cross-platform gap detection (iOS-popular / Android-missing) — explicitly cut.
|
||||||
|
- AppBrain as a chart source — Cloudflare-blocked (403), dropped for play.google.com.
|
||||||
|
- Top-grossing Play chart — Play exposes no clean grossing URL without JS; Apple
|
||||||
|
RSS `topgrossingapplications` carries the monetization-rank signal instead.
|
||||||
|
- Any pip dependency, API key, or paid tier.
|
||||||
|
- Changes outside `plugins/market-research/`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- `fetch-play-charts.py`, `play.py` (both subcommands), `trends.py` each get an
|
||||||
|
offline fixture test driven by `--html-file` / `--json-file`. New fixtures:
|
||||||
|
a Play chart-page HTML, a Play search results HTML, a Play details HTML, a
|
||||||
|
Trends JSON response.
|
||||||
|
- Each scraper's first implementation task: fetch live once, save the fixture,
|
||||||
|
confirm the parser. Then all subsequent tests run offline.
|
||||||
|
- `run-all.sh` and `smoke-structure.sh` extended to cover the new scripts/refs.
|
||||||
|
- Existing bash test convention kept: `set -uo pipefail`, aggregate failures.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **Play HTML class-name drift** (Play obfuscates classes like `Epkrse`, which can
|
||||||
|
rotate). Mitigation: fixture-driven parser isolates the brittle selector; the
|
||||||
|
package link (`/store/apps/details?id=`) is stable even when title classes
|
||||||
|
rotate, so rank+package survives a class change; the skill notes a failed
|
||||||
|
Play-chart fetch and continues on Apple RSS + web signal.
|
||||||
|
- **Google Trends fragility.** Mitigation by design: `trends.py` never hard-fails;
|
||||||
|
WebSearch fallback covers the same signal.
|
||||||
|
- **Play search HTML variance by region/locale.** Mitigation: force an `hl=en&gl=US`
|
||||||
|
query in `play.py` for stable markup; fixture covers that shape.
|
||||||
|
|
@ -6,14 +6,16 @@ scored, non-repeating clone candidates, then hands user-chosen candidates to the
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
`/market-research` runs a 6-phase workflow:
|
`/market-research` runs an 8-phase workflow (0–7):
|
||||||
|
|
||||||
0. **Seed rotation** — pick varied search angles (category × region × niche) so runs differ.
|
0. **Seed rotation** — pick varied search angles (category × region × niche) so runs differ.
|
||||||
1. **Gather** — App Store RSS chart data (`fetch-charts.py`) + free web search for trends.
|
1. **Gather** — App Store RSS (`fetch-charts.py`) + play.google.com charts (`fetch-play-charts.py`) + Google Trends best-effort (`trends.py`) + free web search.
|
||||||
2. **Synthesize** — cluster findings into ≥10 distinct app/game ideas.
|
2. **Synthesize** — cluster findings into ≥10 distinct app/game ideas.
|
||||||
3. **Score** — composite score: cloneability + market opportunity + monetization fit.
|
3. **Score** — composite numeric score: cloneability 35 + market 35 + monetization 20 + niche 10 (see `scoring-guide.md` + `numeric-sources.md`).
|
||||||
4. **Dedup** — exclude anything already in `./work/market-research/history.json`.
|
4. **Dedup** — exclude anything already in `./work/market-research/history.json`.
|
||||||
5. **Present + handoff** — show the ranked table; chosen candidates flow into `clone-app`.
|
5. **Present** — show the scored, ranked table.
|
||||||
|
6. **Survivor-only enrichment** — for user-chosen candidates only: resolve the Play Store package ID (`play.py`) and verify the link is live via WebFetch before handing off.
|
||||||
|
7. **Handoff** — chosen candidates (with verified Play links) flow into `clone-app`.
|
||||||
|
|
||||||
## State
|
## State
|
||||||
|
|
||||||
|
|
@ -23,7 +25,10 @@ Written under `./work/market-research/` in your current directory:
|
||||||
|
|
||||||
## Scripts
|
## Scripts
|
||||||
|
|
||||||
- `scripts/fetch-charts.py` — fetch App Store RSS chart feeds → normalized JSON.
|
- `scripts/fetch-charts.py` — fetch Apple App Store RSS chart feeds → normalized JSON.
|
||||||
|
- `scripts/fetch-play-charts.py` — fetch play.google.com top/category charts → normalized JSON (no AppBrain; uses the public Play web endpoint directly).
|
||||||
|
- `scripts/play.py` — resolve an app name or bundle ID to a Play Store package ID and count search results (used in survivor-only enrichment, Phase 6).
|
||||||
|
- `scripts/trends.py` — best-effort Google Trends interest data for a keyword; never hard-fails (falls back to `null` if the endpoint is unavailable).
|
||||||
- `scripts/history.py` — read / append / dedup the suggestion history.
|
- `scripts/history.py` — read / append / dedup the suggestion history.
|
||||||
|
|
||||||
Python is stdlib-only. Tests run offline against `tests/fixtures/`:
|
Python is stdlib-only. Tests run offline against `tests/fixtures/`:
|
||||||
|
|
@ -32,6 +37,15 @@ Python is stdlib-only. Tests run offline against `tests/fixtures/`:
|
||||||
bash plugins/market-research/tests/run-all.sh
|
bash plugins/market-research/tests/run-all.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
AI-judgment rubrics under `skills/market-research/references/`:
|
||||||
|
|
||||||
|
- `research-angles.md` — rotating category × region × niche seeds for run-to-run variety.
|
||||||
|
- `scoring-guide.md` — the 0–100 weighted composite formula and per-dimension guidance.
|
||||||
|
- `numeric-sources.md` — how to map raw data (chart rank, install count, rating, search results) to numeric sub-scores used by `scoring-guide.md`.
|
||||||
|
- `report-template.md` — output format for the scored candidate table and per-candidate write-ups.
|
||||||
|
|
||||||
## Effort convention
|
## Effort convention
|
||||||
|
|
||||||
Effort is measured in **AI Sprints** (one focused Claude session), never calendar time.
|
Effort is measured in **AI Sprints** (one focused Claude session), never calendar time.
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@ Scan the app/game market with free sources, score candidates, and hand the ones
|
||||||
you pick to the `clone-app` skill. Every run rotates its search angles and
|
you pick to the `clone-app` skill. Every run rotates its search angles and
|
||||||
excludes everything suggested before, so results stay fresh.
|
excludes everything suggested before, so results stay fresh.
|
||||||
|
|
||||||
This skill orchestrates 6 phases (0–5). Deterministic steps are factored into
|
This skill orchestrates 8 phases (0–7). Deterministic steps are factored into
|
||||||
helper scripts under `${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/`;
|
helper scripts under `${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/`;
|
||||||
AI-judgment steps follow rubrics under `.../references/`.
|
AI-judgment steps follow rubrics under `.../references/`.
|
||||||
|
|
||||||
|
|
@ -36,51 +36,86 @@ used lately. Choose 2–3 categories, 1–2 regions, and 1 niche lens you did NO
|
||||||
last run. If the command passed a focus argument, force one category to match it.
|
last run. If the command passed a focus argument, force one category to match it.
|
||||||
State the chosen angles to the user in one line before continuing.
|
State the chosen angles to the user in one line before continuing.
|
||||||
|
|
||||||
## Phase 1: Gather (free web)
|
## Phase 1: Gather charts (free)
|
||||||
Hard chart data — for each chosen region, pull the relevant feeds:
|
For each chosen region pull BOTH stores. Apple RSS (iOS signal — bundle ids, not
|
||||||
|
Android packages):
|
||||||
```bash
|
```bash
|
||||||
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-charts.py \
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-charts.py \
|
||||||
topfreeapplications --region <region> --limit 25 > "$WORK/charts-<region>-free.json"
|
topfreeapplications --region <region> --limit 25 > "$WORK/charts-<region>-free.json"
|
||||||
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-charts.py \
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-charts.py \
|
||||||
topgrossingapplications --region <region> --limit 25 > "$WORK/charts-<region>-grossing.json"
|
topgrossingapplications --region <region> --limit 25 > "$WORK/charts-<region>-grossing.json"
|
||||||
```
|
```
|
||||||
(Top-grossing = monetization signal; top-free = demand signal. Add
|
play.google.com charts (Android signal — REAL packages). Pull the overall top
|
||||||
`toppaidapplications` if willingness-to-pay matters for the angle.) If a fetch
|
plus one category page per chosen category (Play category code, e.g.
|
||||||
fails, note it and continue with the feeds you got.
|
`PRODUCTIVITY`, `GAME_PUZZLE`, `FINANCE`):
|
||||||
|
```bash
|
||||||
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-play-charts.py \
|
||||||
|
top --region <region> --limit 25 > "$WORK/play-top-<region>.json"
|
||||||
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/fetch-play-charts.py \
|
||||||
|
category --category <CAT> --region <region> --limit 25 > "$WORK/play-<CAT>.json"
|
||||||
|
```
|
||||||
|
If any fetch fails (Play may rotate its obfuscated title class), note it and
|
||||||
|
continue with the feeds you got — Apple RSS + web signal still stand.
|
||||||
|
|
||||||
Trend signal — use WebSearch for the chosen categories/niches: new releases,
|
## Phase 2: Trend + numeric signal
|
||||||
ProductHunt launches, Reddit/news chatter in the last ~90 days, "fastest growing
|
Read `${CLAUDE_PLUGIN_ROOT}/skills/market-research/references/numeric-sources.md`.
|
||||||
<category> apps 2026", dated-incumbent complaints. Vary the queries by the
|
For the chosen categories/niches, WebSearch the named numeric sources and pull
|
||||||
run's angles so two runs don't search the same terms. These results are the
|
real figures (market size, downloads, revenue, YoY) WITH their source URLs. For
|
||||||
qualitative half the charts can't give.
|
the top themes, attempt Trends momentum:
|
||||||
|
```bash
|
||||||
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/trends.py "<theme>"
|
||||||
|
```
|
||||||
|
If a `trends.py` result is `{"ok": false, "fallback": "websearch"}`, WebSearch the
|
||||||
|
same momentum signal instead. Vary queries by the run's angles.
|
||||||
|
|
||||||
## Phase 2: Synthesize candidates
|
## Phase 3: Synthesize ≥12 candidates
|
||||||
Cluster the chart entries + web findings into **at least 12** distinct app/game
|
Cluster chart entries (Apple RSS + play.google.com) + web findings into ≥12
|
||||||
ideas (synthesize more than 10 so dedup in Phase 4 still leaves ≥10). For each:
|
distinct ideas (synthesize > 10 so dedup still leaves ≥10). For each: name,
|
||||||
name, category, what-it-does, why-now (trend signal), incumbent(s), monetization
|
category, what-it-does, why-now (with a cited number where available),
|
||||||
model. Note that App Store `bundle_id`s from the charts are iOS — treat them as
|
incumbent(s), monetization model. play.google.com entries already give an
|
||||||
signal, not Android packages. Write the working list as a JSON array (objects
|
Android `package`; carry it.
|
||||||
with at least `name`, optional `package`, `category`) to `$WORK/candidates.json`.
|
Write the working list as a JSON array (objects with at least `name`, optional
|
||||||
|
`package`, `category`) to `$WORK/candidates.json`.
|
||||||
|
|
||||||
## Phase 3: Score
|
## Phase 4: Cheap score + history dedup
|
||||||
Read `${CLAUDE_PLUGIN_ROOT}/skills/market-research/references/scoring-guide.md`.
|
Score each candidate on the chart/trend signal you ALREADY have (don't enrich
|
||||||
Score every candidate's four components and weighted total. Add the subscores and
|
yet — that's Phase 5). Then drop anything suggested before:
|
||||||
`total` to each object in `$WORK/candidates.json`. Rank by `total` descending.
|
|
||||||
|
|
||||||
## Phase 4: History dedup
|
|
||||||
Drop anything already suggested:
|
|
||||||
```bash
|
```bash
|
||||||
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/history.py \
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/history.py \
|
||||||
filter --history "$WORK/history.json" < "$WORK/candidates.json" > "$WORK/fresh.json"
|
filter --history "$WORK/history.json" < "$WORK/candidates.json" > "$WORK/fresh.json"
|
||||||
```
|
```
|
||||||
If fewer than 10 candidates survive, go back to Phase 1/2 with a different angle
|
If fewer than 10 survive, loop back to Phase 1/2 with a DIFFERENT angle and
|
||||||
and synthesize more, then re-filter — never present a padded or repeated list.
|
synthesize more, then re-filter. Never present a padded or repeated list.
|
||||||
|
|
||||||
## Phase 5: Present + handoff
|
## Phase 5: Enrich survivors only (efficiency)
|
||||||
|
For the surviving top ~10 ONLY (never the full raw set), enrich each:
|
||||||
|
1. Resolve 1–2 Play links + stats:
|
||||||
|
```bash
|
||||||
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/play.py \
|
||||||
|
resolve "<candidate name>"
|
||||||
|
```
|
||||||
|
2. **Verify** each resolved `play_url` with WebFetch — confirm HTTP 200 and the
|
||||||
|
`id=<package>` is present on the page. Drop dead/mismatched links. A candidate
|
||||||
|
with no verifiable link is flagged "Play link unresolved" (kept in report,
|
||||||
|
skipped on handoff).
|
||||||
|
3. Saturation:
|
||||||
|
```bash
|
||||||
|
python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/play.py \
|
||||||
|
count "<category or core feature>"
|
||||||
|
```
|
||||||
|
4. Momentum (`trends.py` as in Phase 2) if not already pulled.
|
||||||
|
|
||||||
|
## Phase 6: Re-score with numbers
|
||||||
|
Read `${CLAUDE_PLUGIN_ROOT}/skills/market-research/references/scoring-guide.md`.
|
||||||
|
Re-score every surviving candidate using the enriched numbers; attach the
|
||||||
|
`evidence` object (cited installs / trend % / saturation / ARPU). Rank by `total`
|
||||||
|
descending. Subscores with no supporting number are capped per the guide.
|
||||||
|
|
||||||
|
## Phase 7: Present + handoff
|
||||||
Read `${CLAUDE_PLUGIN_ROOT}/skills/market-research/references/report-template.md`.
|
Read `${CLAUDE_PLUGIN_ROOT}/skills/market-research/references/report-template.md`.
|
||||||
Fill it from `$WORK/fresh.json` and write `$WORK/research-<YYYY-MM-DD>.md` (use
|
Fill it from the enriched, re-scored survivors and write
|
||||||
the actual run date). Show the user the ranked table (≥10 rows) and your top-3
|
`$WORK/research-<YYYY-MM-DD>.md`. Show the user the ranked table (≥10 rows, with
|
||||||
recommended picks.
|
Play links + saturation) and your top-3 picks.
|
||||||
|
|
||||||
Record this run's suggestions so they won't repeat:
|
Record this run's suggestions so they won't repeat:
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -88,20 +123,20 @@ python3 ${CLAUDE_PLUGIN_ROOT}/skills/market-research/scripts/history.py \
|
||||||
add --history "$WORK/history.json" --date <YYYY-MM-DD> --run-id "<RUN_ID>" \
|
add --history "$WORK/history.json" --date <YYYY-MM-DD> --run-id "<RUN_ID>" \
|
||||||
< "$WORK/fresh.json"
|
< "$WORK/fresh.json"
|
||||||
```
|
```
|
||||||
|
Then ask which candidate(s) to pursue. Each pick already has a verified Play URL —
|
||||||
Then ask which candidate(s) to pursue. For each pick:
|
invoke the `clone-app` skill on it. If the user picks nothing, stop — the report
|
||||||
1. Resolve it to a Google Play package/URL. If you don't already have the package,
|
stands on its own.
|
||||||
WebSearch `"<name>" site:play.google.com` (or the developer + app name) and
|
|
||||||
confirm the `play.google.com/store/apps/details?id=...` URL.
|
|
||||||
2. Invoke the `clone-app` skill on that URL/package to run full feasibility.
|
|
||||||
If the user picks nothing, stop — the report stands on its own.
|
|
||||||
|
|
||||||
## Error Handling Summary
|
## Error Handling Summary
|
||||||
| Scenario | Action |
|
| Scenario | Action |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `fetch-charts.py` fails for a region | note it, continue with other feeds/web search |
|
| `fetch-charts.py` / `fetch-play-charts.py` fails | note it, continue with other feeds/web search |
|
||||||
|
| play.google.com chart fetch fails / title class rotated | continue on Apple RSS + web signal; package link stays stable |
|
||||||
|
| `play.py resolve` finds no package | flag candidate "Play link unresolved", skip handoff |
|
||||||
|
| Play link fails WebFetch verify | drop that link; if none verify, flag unresolved |
|
||||||
|
| `trends.py` returns `{"ok": false}` | WebSearch the same momentum signal |
|
||||||
|
| numeric source paywalled/JS-only | use only free-visible figures; never invent a number |
|
||||||
| Web search returns thin results | broaden queries within the chosen angle, try another region |
|
| Web search returns thin results | broaden queries within the chosen angle, try another region |
|
||||||
| < 10 candidates survive dedup | loop back to Phase 1/2 with a new angle, re-filter |
|
| < 10 candidates survive dedup | loop back to Phase 1/2 with a new angle, re-filter |
|
||||||
| history.json missing/first run | treat as empty; all candidates are fresh |
|
| history.json missing/first run | treat as empty; all candidates are fresh |
|
||||||
| Candidate has no resolvable Play package | skip the handoff for it, keep it in the report as iOS-only/unresolved |
|
|
||||||
| User picks nothing | stop after writing the report |
|
| User picks nothing | stop after writing the report |
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
# Numeric Market-Data Sources (free, no API key)
|
||||||
|
|
||||||
|
Phase 2 uses this to turn "search the web" into "pull THESE numbers from THESE
|
||||||
|
places, and cite each." Prefer a real number with a source link over a vibe.
|
||||||
|
Every number that lands in the report carries the URL it came from.
|
||||||
|
|
||||||
|
## Sources & what to pull
|
||||||
|
|
||||||
|
| Source | Pull | How (free) |
|
||||||
|
|---|---|---|
|
||||||
|
| **Google Trends** | interest-over-time, % momentum, breakout/rising queries | `trends.py "<term>"`; on `{"ok":false}` fall back to WebSearch `google trends <term>`. |
|
||||||
|
| **play.google.com** | ranked Android packages, per-app installs/rating | `fetch-play-charts.py` for top/category charts; `play.py resolve` for per-app installs/rating. |
|
||||||
|
| **Sensor Tower (blog/reports)** | downloads, revenue, DAU, YoY growth | WebSearch `sensortower <category> revenue downloads 2026`; WebFetch the article; quote the figure + date. |
|
||||||
|
| **data.ai / Apptopia posts** | top-charts movement, market revenue | WebSearch `data.ai <category> market 2026`; WebFetch + cite. |
|
||||||
|
| **Statista (free public charts)** | market size $, user counts, CAGR | WebSearch `statista <category> market size`; use the visible free figure only; cite. |
|
||||||
|
| **SimilarWeb (free tier)** | incumbent web traffic, engagement | WebSearch `similarweb <incumbent domain>`; cite visits/engagement. |
|
||||||
|
|
||||||
|
## Query shaping (vary by the run's angles)
|
||||||
|
|
||||||
|
- Always bind a year: `... 2026` so figures are current.
|
||||||
|
- Bind a region when the angle is regional: `... <category> Brazil downloads`.
|
||||||
|
- For growth: `fastest growing <category> apps 2026`, `<category> market size 2026`.
|
||||||
|
- For incumbent weakness: `<incumbent> complaints reddit`, `<incumbent> alternative 2026`.
|
||||||
|
|
||||||
|
## Citation rule
|
||||||
|
|
||||||
|
Each extracted number is written as `<number> [<source>](<url>)` in the report.
|
||||||
|
A claim with no source link is downgraded to a qualitative note, not a number —
|
||||||
|
it must NOT be used to justify a numeric subscore (see scoring-guide.md).
|
||||||
|
|
||||||
|
## When a source is paywalled / JS-only
|
||||||
|
|
||||||
|
Do not scrape it. Use the figure only if it's visible free (article body, public
|
||||||
|
chart). Otherwise drop to the next source. Never invent a number to fill a gap.
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
# Market Research Report Template
|
# Market Research Report Template
|
||||||
|
|
||||||
Fill every section. Write to `./work/market-research/research-<YYYY-MM-DD>.md`.
|
Fill every section. Write to `./work/market-research/research-<YYYY-MM-DD>.md`.
|
||||||
|
Every number carries a source link. Every candidate carries 1–2 Google Play links.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -9,36 +10,39 @@ Fill every section. Write to `./work/market-research/research-<YYYY-MM-DD>.md`.
|
||||||
## Run parameters
|
## Run parameters
|
||||||
- Angles this run: <categories / regions / niche lens chosen in Phase 0>
|
- Angles this run: <categories / regions / niche lens chosen in Phase 0>
|
||||||
- Focus argument: <the user's focus, or "none">
|
- Focus argument: <the user's focus, or "none">
|
||||||
- Sources: App Store RSS (<feeds/regions pulled>) + web search (<themes>)
|
- Sources: Apple RSS (<feeds/regions>) + play.google.com charts (<charts>) + web numeric (<sources>) + Trends (<ok/fallback>)
|
||||||
- Candidates after history exclusion: <N> (history had <M> prior suggestions)
|
- Candidates after history exclusion: <N> (history had <M> prior suggestions)
|
||||||
|
|
||||||
## Top candidates (ranked)
|
## Top candidates (ranked)
|
||||||
|
|
||||||
| # | Name | Category | Clone | Market | Monet. | Niche | **Total** | Why now |
|
| # | Name | Category | Play link(s) | Installs | Saturation | Clone | Market | Monet. | Niche | **Total** | Why now (cited) |
|
||||||
|---|------|----------|------:|-------:|-------:|------:|----------:|---------|
|
|---|------|----------|--------------|---------:|-----------:|------:|-------:|-------:|------:|----------:|-----------------|
|
||||||
| 1 | … | … | 85 | 80 | 75 | 60 | **79** | one-line trend rationale |
|
| 1 | … | … | [Play](url) | 5M+ | 12 apps/4.2★ | 85 | 80 | 75 | 60 | **79** | trend +120% [src] |
|
||||||
| … | | | | | | | | |
|
|
||||||
|
|
||||||
(At least 10 rows.)
|
(At least 10 rows. Each Play link is verified live — HTTP 200 + package match.)
|
||||||
|
|
||||||
## Candidate detail
|
## Candidate detail
|
||||||
|
|
||||||
For each of the top candidates:
|
For each top candidate:
|
||||||
|
|
||||||
### <name>
|
### <name>
|
||||||
- **Package (if known):** <com.x.y or "to resolve">
|
- **Play link(s):** [<package1>](url1) [, [<package2>](url2)] ← 1–2, verified
|
||||||
- **What it does:** <1–2 sentences>
|
- **What it does:** <1–2 sentences>
|
||||||
- **Why now:** <trend signal: chart movement, news, dated incumbent…>
|
- **Why now:** <trend signal with a number + source link>
|
||||||
- **Incumbents:** <who already does this and how weak/strong>
|
- **Incumbents:** <who, and saturation: app_count + avg rating from play.py count>
|
||||||
- **Monetization:** <ads / IAP / subscription; ARPU note>
|
- **Monetization:** <ads / IAP / subscription; ARPU figure + source>
|
||||||
|
- **Evidence:** installs <n> [src], trend <pct>% [src], saturation <n> apps, ARPU <…> [src]
|
||||||
- **Scores:** clone <>, market <>, monetization <>, niche <> → **total <>**
|
- **Scores:** clone <>, market <>, monetization <>, niche <> → **total <>**
|
||||||
- **Clone risk flags:** <heavy ML / native / content moat / none>
|
- **Clone risk flags:** <heavy ML / native / content moat / none>
|
||||||
|
|
||||||
## Recommended picks
|
## Recommended picks
|
||||||
|
|
||||||
Top 3 to send to clone-app first, with one sentence each on why they lead.
|
Top 3 to send to clone-app first, one sentence each on why they lead (cite the
|
||||||
|
deciding number).
|
||||||
|
|
||||||
## Next step
|
## Next step
|
||||||
|
|
||||||
The user picks one or more candidates; each chosen candidate is resolved to a
|
The user picks one or more candidates; each chosen candidate already has a
|
||||||
Google Play package/URL and handed to the `clone-app` skill for full feasibility.
|
verified Google Play URL and is handed to the `clone-app` skill for full
|
||||||
|
feasibility. Candidates whose Play link could not be verified are flagged
|
||||||
|
"Play link unresolved" and skipped on handoff.
|
||||||
|
|
|
||||||
|
|
@ -1,43 +1,54 @@
|
||||||
# Scoring Guide
|
# Scoring Guide (numeric)
|
||||||
|
|
||||||
Score every candidate 0–100 as a weighted composite. Three PRIMARY components
|
Score every candidate 0–100 as a weighted composite of three PRIMARY components
|
||||||
(cloneability, market opportunity, monetization fit) plus a secondary tiebreaker
|
plus a tiebreaker. **Every subscore must be justified by at least one real number
|
||||||
(niche gap). Show the component scores, not just the total, so picks are auditable.
|
with a source** (see numeric-sources.md). A subscore with no supporting number is
|
||||||
|
CAPPED at 60 — you may not assign a high score on vibes.
|
||||||
|
|
||||||
## Components & weights
|
## Components & weights (unchanged)
|
||||||
|
|
||||||
| Component | Weight | What it measures |
|
| Component | Weight | Measures |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Cloneability | 35% | How cheaply this rebuilds with clone-app + AI. |
|
| Cloneability | 35% | How cheaply this rebuilds with clone-app + AI. |
|
||||||
| Market opportunity | 35% | Demand, growth, and incumbent weakness. |
|
| Market opportunity | 35% | Demand, growth, incumbent weakness. |
|
||||||
| Monetization fit | 20% | Ads/IAP friendliness and category ARPU. |
|
| Monetization fit | 20% | Ads/IAP friendliness and category ARPU. |
|
||||||
| Niche gap (tiebreaker) | 10% | Underserved region/language/segment. |
|
| Niche gap (tiebreaker) | 10% | Underserved region/language/segment. |
|
||||||
|
|
||||||
Total = 0.35·clone + 0.35·market + 0.20·monetization + 0.10·niche, each subscore 0–100.
|
`total = 0.35·clone + 0.35·market + 0.20·monetization + 0.10·niche`, each 0–100.
|
||||||
|
|
||||||
## Scoring each component (0–100)
|
## Required evidence per subscore
|
||||||
|
|
||||||
**Cloneability** — higher = easier:
|
Attach an `evidence` object to each candidate. Each field is `"<number> + source"`
|
||||||
- 80–100: simple CRUD/utility, few backend endpoints, no heavy ML, standard UI.
|
or `null`. A field that is `null` caps that subscore at 60.
|
||||||
- 50–79: moderate backend, some real-time or media, mainstream third-party SDKs.
|
|
||||||
- 0–49: heavy ML/on-device models, complex real-time/multiplayer, deep native, large content moat.
|
|
||||||
|
|
||||||
**Market opportunity** — higher = better:
|
| Subscore | Evidence that lifts the cap above 60 |
|
||||||
- 80–100: strong/growing demand, dated or weak incumbents, clear unmet need.
|
|---|---|
|
||||||
- 50–79: healthy demand, beatable incumbents.
|
| Cloneability | a stack/complexity signal (e.g. "few endpoints — RE later"); no external number required, but state the basis. |
|
||||||
- 0–49: saturated, dominated by entrenched well-funded players.
|
| Market opportunity | an installs figure (Play via `play.py resolve`) AND either a Trends `trend_pct` OR a saturation `app_count`. |
|
||||||
|
| Monetization fit | a category ARPU/revenue figure (Sensor Tower/data.ai/Statista) OR top-grossing chart presence. |
|
||||||
|
| Niche gap | a region/language gap signal (saturation `app_count` low in region, or no localized incumbent). |
|
||||||
|
|
||||||
**Monetization fit** — higher = better:
|
## Scoring bands (0–100)
|
||||||
- 80–100: category with proven ads+IAP and high ARPU (casual games, utilities).
|
|
||||||
- 50–79: monetizable but moderate ARPU.
|
|
||||||
- 0–49: hard to monetize / users expect free.
|
|
||||||
|
|
||||||
**Niche gap** — higher = more underserved:
|
**Cloneability** (higher = easier): 80–100 simple CRUD/utility, few endpoints, no
|
||||||
- 80–100: clear language/region/segment with no quality option.
|
heavy ML. 50–79 moderate backend/media, mainstream SDKs. 0–49 heavy ML/native/
|
||||||
- 0–49: well served everywhere.
|
real-time/large content moat.
|
||||||
|
|
||||||
|
**Market opportunity** (higher = better): 80–100 strong/growing demand (high
|
||||||
|
installs + positive `trend_pct`), weak/dated incumbents, low saturation. 50–79
|
||||||
|
healthy demand, beatable incumbents, moderate saturation. 0–49 saturated
|
||||||
|
(`app_count` high, strong avg rating) or entrenched well-funded players.
|
||||||
|
|
||||||
|
**Monetization fit** (higher = better): 80–100 proven ads+IAP, high ARPU
|
||||||
|
(casual games, utilities) with a cited revenue figure. 50–79 monetizable, moderate
|
||||||
|
ARPU. 0–49 users expect free / hard to monetize.
|
||||||
|
|
||||||
|
**Niche gap** (higher = more underserved): 80–100 clear region/language/segment
|
||||||
|
with no quality option (low regional `app_count`). 0–49 well served everywhere.
|
||||||
|
|
||||||
## Output per candidate
|
## Output per candidate
|
||||||
|
|
||||||
For each candidate keep: `name`, `package` (if resolved), `category`, the four
|
Keep: `name`, `package` (if resolved), `play_url`(s), `category`, the four
|
||||||
subscores, the weighted `total`, and a one-line rationale. Rank by `total`
|
subscores, `evidence` (the cited numbers + source URLs), the weighted `total`,
|
||||||
descending. Produce at least 10 candidates AFTER history exclusion.
|
and a one-line rationale. Rank by `total` descending. Produce ≥10 candidates
|
||||||
|
AFTER history exclusion.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,96 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fetch a play.google.com chart page into a normalized JSON list.
|
||||||
|
|
||||||
|
Google Play has no public chart feed, but its server-rendered HTML for
|
||||||
|
/store/apps/top and /store/apps/category/<CAT> carries ranked app cards with
|
||||||
|
REAL Android package names (unlike Apple's RSS iOS bundle ids). Each card is an
|
||||||
|
<a href="/store/apps/details?id=PKG"> wrapping the icon, then a
|
||||||
|
<div class="Epkrse ">NAME</div> title, then a star aria-label. Play obfuscates
|
||||||
|
the title class, so the package LINK (stable) is the load-bearing field; name
|
||||||
|
and rating are best-effort. Stdlib-only, no pip.
|
||||||
|
|
||||||
|
(AppBrain, the original plan's source, is Cloudflare-blocked 403 — dropped.)
|
||||||
|
"""
|
||||||
|
import sys, json, re, html as _html, argparse, urllib.request, ssl, subprocess, shutil
|
||||||
|
|
||||||
|
UA = "Mozilla/5.0"
|
||||||
|
|
||||||
|
def _http_get(url):
|
||||||
|
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as r:
|
||||||
|
return r.read().decode("utf-8", "replace")
|
||||||
|
except urllib.error.URLError as e:
|
||||||
|
if not isinstance(e.reason, ssl.SSLError):
|
||||||
|
raise
|
||||||
|
if not shutil.which("curl"):
|
||||||
|
raise
|
||||||
|
out = subprocess.run(["curl", "-sL", "--fail", "-A", UA, url],
|
||||||
|
capture_output=True, timeout=60)
|
||||||
|
if out.returncode != 0:
|
||||||
|
raise
|
||||||
|
return out.stdout.decode("utf-8", "replace")
|
||||||
|
|
||||||
|
LINK = re.compile(r'href="/store/apps/details\?id=([a-zA-Z0-9._]+)"')
|
||||||
|
NAME = re.compile(r'class="Epkrse[^"]*"[^>]*>([^<]+)<')
|
||||||
|
# rating is best-effort: search-style aria-label OR a star-icon followed by a number
|
||||||
|
RATE_ARIA = re.compile(r'aria-label="Rated\s+([\d.]+)\s+star')
|
||||||
|
RATE_STAR = re.compile(r'>star</[^>]+>\s*([\d.]+)')
|
||||||
|
|
||||||
|
def _chart_url(chart, category, region):
|
||||||
|
if chart == "category":
|
||||||
|
return f"https://play.google.com/store/apps/category/{category}?hl=en&gl={region}"
|
||||||
|
return f"https://play.google.com/store/apps/top?hl=en&gl={region}"
|
||||||
|
|
||||||
|
def parse(html, chart_label, region, limit):
|
||||||
|
entries = []
|
||||||
|
seen = set()
|
||||||
|
for m in LINK.finditer(html):
|
||||||
|
pkg = m.group(1)
|
||||||
|
if pkg in seen:
|
||||||
|
continue
|
||||||
|
win = html[m.start():m.start() + 1600] # one card's worth of markup
|
||||||
|
nm = NAME.search(win)
|
||||||
|
if not nm:
|
||||||
|
continue # icon-only anchor with no title nearby; skip
|
||||||
|
seen.add(pkg)
|
||||||
|
rt = RATE_ARIA.search(win) or RATE_STAR.search(win)
|
||||||
|
entries.append({
|
||||||
|
"rank": len(entries) + 1,
|
||||||
|
"name": _html.unescape(nm.group(1).strip()),
|
||||||
|
"package": pkg,
|
||||||
|
"rating": float(rt.group(1)) if rt else None,
|
||||||
|
})
|
||||||
|
if len(entries) >= limit:
|
||||||
|
break
|
||||||
|
return {"source": "google-play", "chart": chart_label, "region": region,
|
||||||
|
"count": len(entries), "entries": entries}
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("chart", choices=("top", "category"))
|
||||||
|
ap.add_argument("--category")
|
||||||
|
ap.add_argument("--region", default="US")
|
||||||
|
ap.add_argument("--limit", type=int, default=25)
|
||||||
|
ap.add_argument("--html-file")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if args.chart == "category" and not args.category:
|
||||||
|
print("ERROR: --category is required for the 'category' chart", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
chart_label = f"category:{args.category}" if args.chart == "category" else "top"
|
||||||
|
|
||||||
|
if args.html_file:
|
||||||
|
with open(args.html_file, encoding="utf-8") as f:
|
||||||
|
html = f.read()
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
html = _http_get(_chart_url(args.chart, args.category, args.region))
|
||||||
|
except Exception as e:
|
||||||
|
print(f"ERROR: failed to fetch Play chart: {e}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(json.dumps(parse(html, chart_label, args.region, args.limit), indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,126 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Resolve an app name to a Google Play link + stats, and measure saturation.
|
||||||
|
|
||||||
|
Subcommands:
|
||||||
|
resolve "<name>" -> top Play apps hit: package, play_url, rating, installs,
|
||||||
|
last_updated, developer (1-2 links per candidate upstream).
|
||||||
|
count "<query>" -> saturation: how many distinct apps + avg rating on the
|
||||||
|
first Play search results page.
|
||||||
|
|
||||||
|
Play search (play.google.com/store/search?q=...&c=apps&hl=en&gl=US) is
|
||||||
|
server-rendered enough to grep details?id= links + star aria-labels. The details
|
||||||
|
page carries rating/installs/updated in embedded ld+json (+ light regex), the
|
||||||
|
same structure clone-app's scrape-play-store.py parses. Stdlib-only, no pip.
|
||||||
|
"""
|
||||||
|
import sys, json, re, argparse, urllib.request, urllib.parse, ssl, subprocess, shutil
|
||||||
|
|
||||||
|
UA = "Mozilla/5.0"
|
||||||
|
ID_RE = re.compile(r'/store/apps/details\?id=([\w.]+)')
|
||||||
|
STAR_RE = re.compile(r'Rated\s+([\d.]+)\s+stars')
|
||||||
|
|
||||||
|
def _http_get(url):
|
||||||
|
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as r:
|
||||||
|
return r.read().decode("utf-8", "replace")
|
||||||
|
except urllib.error.URLError as e:
|
||||||
|
if not isinstance(e.reason, ssl.SSLError):
|
||||||
|
raise
|
||||||
|
if not shutil.which("curl"):
|
||||||
|
raise
|
||||||
|
out = subprocess.run(["curl", "-sL", "--fail", "-A", UA, url],
|
||||||
|
capture_output=True, timeout=60)
|
||||||
|
if out.returncode != 0:
|
||||||
|
raise
|
||||||
|
return out.stdout.decode("utf-8", "replace")
|
||||||
|
|
||||||
|
def _search_html(query, search_file):
|
||||||
|
if search_file:
|
||||||
|
with open(search_file, encoding="utf-8") as f:
|
||||||
|
return f.read()
|
||||||
|
q = urllib.parse.quote(query)
|
||||||
|
return _http_get(f"https://play.google.com/store/search?q={q}&c=apps&hl=en&gl=US")
|
||||||
|
|
||||||
|
def _details_html(package, details_file):
|
||||||
|
if details_file:
|
||||||
|
with open(details_file, encoding="utf-8") as f:
|
||||||
|
return f.read()
|
||||||
|
return _http_get(
|
||||||
|
f"https://play.google.com/store/apps/details?id={package}&hl=en&gl=US")
|
||||||
|
|
||||||
|
def parse_details(html, package):
|
||||||
|
out = {"name": None, "developer": None, "rating": None,
|
||||||
|
"installs": None, "last_updated": None}
|
||||||
|
for m in re.finditer(r'<script type="application/ld\+json"[^>]*>(.*?)</script>',
|
||||||
|
html, re.DOTALL):
|
||||||
|
try:
|
||||||
|
data = json.loads(m.group(1).strip())
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if isinstance(data, dict) and data.get("@type") == "SoftwareApplication":
|
||||||
|
out["name"] = data.get("name")
|
||||||
|
auth = data.get("author")
|
||||||
|
if isinstance(auth, dict):
|
||||||
|
out["developer"] = auth.get("name")
|
||||||
|
ar = data.get("aggregateRating") or {}
|
||||||
|
if ar.get("ratingValue") is not None:
|
||||||
|
try: out["rating"] = float(ar["ratingValue"])
|
||||||
|
except Exception: pass
|
||||||
|
break
|
||||||
|
m = re.search(r'([\d.,]+[KMB]?\+)\s*</[^>]+>\s*<[^>]*>\s*Downloads', html)
|
||||||
|
if not m:
|
||||||
|
m = re.search(r'([\d.,]+[KMB]?\+)\s*<span>\s*Downloads', html)
|
||||||
|
if m:
|
||||||
|
out["installs"] = m.group(1)
|
||||||
|
m = re.search(r'Updated on\s*</[^>]+>\s*<[^>]*>([^<]+)</[^>]+>', html)
|
||||||
|
if not m:
|
||||||
|
m = re.search(r'Updated on\s*<span>([^<]+)</span>', html)
|
||||||
|
if m:
|
||||||
|
out["last_updated"] = m.group(1).strip()
|
||||||
|
return out
|
||||||
|
|
||||||
|
def cmd_resolve(args):
|
||||||
|
html = _search_html(args.query, args.search_file)
|
||||||
|
m = ID_RE.search(html)
|
||||||
|
res = {"query": args.query, "name": None, "package": None, "play_url": None,
|
||||||
|
"rating": None, "installs": None, "last_updated": None, "developer": None}
|
||||||
|
if not m:
|
||||||
|
print(json.dumps(res, indent=2))
|
||||||
|
return
|
||||||
|
package = m.group(1)
|
||||||
|
res["package"] = package
|
||||||
|
res["play_url"] = f"https://play.google.com/store/apps/details?id={package}"
|
||||||
|
details = _details_html(package, args.details_file)
|
||||||
|
res.update(parse_details(details, package))
|
||||||
|
print(json.dumps(res, indent=2))
|
||||||
|
|
||||||
|
def cmd_count(args):
|
||||||
|
html = _search_html(args.query, args.search_file)
|
||||||
|
packages = []
|
||||||
|
for pkg in ID_RE.findall(html):
|
||||||
|
if pkg not in packages:
|
||||||
|
packages.append(pkg)
|
||||||
|
ratings = [float(x) for x in STAR_RE.findall(html)]
|
||||||
|
avg = round(sum(ratings) / len(ratings), 2) if ratings else None
|
||||||
|
print(json.dumps({"query": args.query, "app_count": len(packages),
|
||||||
|
"avg_rating": avg, "top_packages": packages}, indent=2))
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||||
|
pr = sub.add_parser("resolve")
|
||||||
|
pr.add_argument("query")
|
||||||
|
pr.add_argument("--search-file")
|
||||||
|
pr.add_argument("--details-file")
|
||||||
|
pc = sub.add_parser("count")
|
||||||
|
pc.add_argument("query")
|
||||||
|
pc.add_argument("--search-file")
|
||||||
|
args = ap.parse_args()
|
||||||
|
try:
|
||||||
|
{"resolve": cmd_resolve, "count": cmd_count}[args.cmd](args)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"ERROR: play {args.cmd} failed: {e}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,79 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Best-effort Google Trends interest signal. NEVER hard-fails.
|
||||||
|
|
||||||
|
Google Trends has no API key but the unofficial endpoint needs a token dance
|
||||||
|
and is fragile under stdlib-only. So: try the live token+timeline flow; on ANY
|
||||||
|
failure print {"ok": false, "fallback": "websearch"} and exit 0, signalling the
|
||||||
|
skill to fall back to WebSearch for the same momentum signal. The JSON parser is
|
||||||
|
unit-tested offline via --json-file. Stdlib-only, no pip.
|
||||||
|
"""
|
||||||
|
import sys, json, argparse, urllib.request, urllib.parse, ssl, subprocess, shutil
|
||||||
|
|
||||||
|
UA = "Mozilla/5.0"
|
||||||
|
|
||||||
|
def _http_get(url, headers=None):
|
||||||
|
req = urllib.request.Request(url, headers={"User-Agent": UA, **(headers or {})})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as r:
|
||||||
|
return r.read().decode("utf-8", "replace")
|
||||||
|
except urllib.error.URLError as e:
|
||||||
|
if not isinstance(e.reason, ssl.SSLError):
|
||||||
|
raise
|
||||||
|
if not shutil.which("curl"):
|
||||||
|
raise
|
||||||
|
out = subprocess.run(["curl", "-sL", "--fail", "-A", UA, url],
|
||||||
|
capture_output=True, timeout=60)
|
||||||
|
if out.returncode != 0:
|
||||||
|
raise
|
||||||
|
return out.stdout.decode("utf-8", "replace")
|
||||||
|
|
||||||
|
def parse_timeline(raw):
|
||||||
|
"""raw = the timeline JSON (Google prefixes live responses with )]}', —
|
||||||
|
callers strip it first). Returns the computed interest summary dict."""
|
||||||
|
data = json.loads(raw)
|
||||||
|
points = data["default"]["timelineData"]
|
||||||
|
values = [int(p["value"][0]) for p in points if p.get("value")]
|
||||||
|
if not values:
|
||||||
|
raise ValueError("no timeline values")
|
||||||
|
avg = round(sum(values) / len(values), 2)
|
||||||
|
first, latest = values[0], values[-1]
|
||||||
|
pct = round((latest - first) / first * 100, 2) if first else None
|
||||||
|
return {"ok": True, "avg_interest": avg, "latest_interest": latest,
|
||||||
|
"trend_pct": pct, "points": len(values)}
|
||||||
|
|
||||||
|
def fetch_live(term):
|
||||||
|
"""Token dance: explore -> widget token -> multiline timeline."""
|
||||||
|
base = "https://trends.google.com/trends/api"
|
||||||
|
comp = urllib.parse.quote(json.dumps(
|
||||||
|
{"comparisonItem": [{"keyword": term, "geo": "", "time": "today 3-m"}],
|
||||||
|
"category": 0, "property": ""}))
|
||||||
|
explore = _http_get(f"{base}/explore?hl=en-US&tz=0&req={comp}")
|
||||||
|
widgets = json.loads(explore.lstrip(")]}',\n"))
|
||||||
|
w = next(x for x in widgets["widgets"] if x.get("id") == "TIMESERIES")
|
||||||
|
token = w["token"]
|
||||||
|
reqobj = urllib.parse.quote(json.dumps(w["request"]))
|
||||||
|
tl = _http_get(f"{base}/widgetdata/multiline?hl=en-US&tz=0&req={reqobj}&token={token}")
|
||||||
|
return tl.lstrip(")]}',\n")
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("term")
|
||||||
|
ap.add_argument("--json-file")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
try:
|
||||||
|
if args.json_file:
|
||||||
|
with open(args.json_file, encoding="utf-8") as f:
|
||||||
|
raw = f.read()
|
||||||
|
else:
|
||||||
|
raw = fetch_live(args.term)
|
||||||
|
res = parse_timeline(raw)
|
||||||
|
res["term"] = args.term
|
||||||
|
print(json.dumps(res, indent=2))
|
||||||
|
except Exception:
|
||||||
|
print(json.dumps({"ok": False, "term": args.term,
|
||||||
|
"fallback": "websearch"}, indent=2))
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,9 @@
|
||||||
|
<!DOCTYPE html><html><body>
|
||||||
|
<div role="main">
|
||||||
|
<a href="/store/apps/details?id=com.example.habit" jslog="38003; track:click"><div class="TjRVLb"><img src="https://play-lh.googleusercontent.com/x=s256"></div><div class="Epkrse ">Habit Tracker</div></a>
|
||||||
|
<div aria-label="Rated 4.6 stars out of five stars"></div>
|
||||||
|
<a href="/store/apps/details?id=com.example.budget" jslog="38003; track:click"><div class="TjRVLb"><img src="https://play-lh.googleusercontent.com/y=s256"></div><div class="Epkrse ">Budget Buddy</div></a>
|
||||||
|
<div aria-label="Rated 4.2 stars out of five stars"></div>
|
||||||
|
<a href="/store/apps/details?id=com.example.habit" jslog="38003; track:click"><div class="Epkrse ">Habit Tracker</div></a>
|
||||||
|
</div>
|
||||||
|
</body></html>
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
<!DOCTYPE html><html><body>
|
||||||
|
<script type="application/ld+json" nonce="abc">
|
||||||
|
{"@type":"SoftwareApplication","name":"Habit Tracker","author":{"name":"Focus Labs"},
|
||||||
|
"applicationCategory":"Productivity","aggregateRating":{"ratingValue":"4.6","ratingCount":"120000"}}
|
||||||
|
</script>
|
||||||
|
<div>1B+</div><div>Downloads</div>
|
||||||
|
<div>Updated on</div><div>Jun 1, 2026</div>
|
||||||
|
</body></html>
|
||||||
|
|
@ -0,0 +1,10 @@
|
||||||
|
<!DOCTYPE html><html><body>
|
||||||
|
<div role="main">
|
||||||
|
<a href="/store/apps/details?id=com.example.habit"><span>Habit Tracker</span></a>
|
||||||
|
<div aria-label="Rated 4.6 stars out of five stars"></div>
|
||||||
|
<a href="/store/apps/details?id=com.example.habit2"><span>Habit Plus</span></a>
|
||||||
|
<div aria-label="Rated 4.1 stars out of five stars"></div>
|
||||||
|
<a href="/store/apps/details?id=com.example.habit3"><span>Daily Habits</span></a>
|
||||||
|
<div aria-label="Rated 3.9 stars out of five stars"></div>
|
||||||
|
</div>
|
||||||
|
</body></html>
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
{"default":{"timelineData":[
|
||||||
|
{"time":"1","value":[50]},
|
||||||
|
{"time":"2","value":[60]},
|
||||||
|
{"time":"3","value":[100]}
|
||||||
|
]}}
|
||||||
|
|
@ -10,11 +10,11 @@ must_exist "$P/.claude-plugin/plugin.json"
|
||||||
must_exist "$P/commands/market-research.md"
|
must_exist "$P/commands/market-research.md"
|
||||||
must_exist "$P/README.md"
|
must_exist "$P/README.md"
|
||||||
must_exist "$P/skills/market-research/SKILL.md"
|
must_exist "$P/skills/market-research/SKILL.md"
|
||||||
for s in fetch-charts.py history.py; do
|
for s in fetch-charts.py history.py fetch-play-charts.py play.py trends.py; do
|
||||||
must_exist "$P/skills/market-research/scripts/$s"
|
must_exist "$P/skills/market-research/scripts/$s"
|
||||||
must_exec "$P/skills/market-research/scripts/$s"
|
must_exec "$P/skills/market-research/scripts/$s"
|
||||||
done
|
done
|
||||||
for r in research-angles scoring-guide report-template; do
|
for r in research-angles scoring-guide report-template numeric-sources; do
|
||||||
must_exist "$P/skills/market-research/references/$r.md"
|
must_exist "$P/skills/market-research/references/$r.md"
|
||||||
done
|
done
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
import json, subprocess, sys, os
|
||||||
|
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
SCRIPT = os.path.join(HERE, "..", "skills", "market-research", "scripts", "fetch-play-charts.py")
|
||||||
|
FIXTURE = os.path.join(HERE, "fixtures", "play-chart.html")
|
||||||
|
|
||||||
|
def run():
|
||||||
|
out = subprocess.check_output(
|
||||||
|
[sys.executable, SCRIPT, "top", "--region", "US", "--html-file", FIXTURE])
|
||||||
|
return json.loads(out)
|
||||||
|
|
||||||
|
def main():
|
||||||
|
d = run()
|
||||||
|
fails = []
|
||||||
|
def check(name, cond):
|
||||||
|
print(f"{'PASS' if cond else 'FAIL'}: {name}")
|
||||||
|
if not cond: fails.append(name)
|
||||||
|
|
||||||
|
check("source", d["source"] == "google-play")
|
||||||
|
check("chart", d["chart"] == "top")
|
||||||
|
check("region", d["region"] == "US")
|
||||||
|
check("count (dedup app1 repeat)", d["count"] == 2)
|
||||||
|
e0 = d["entries"][0]
|
||||||
|
check("rank 1", e0["rank"] == 1)
|
||||||
|
check("name", e0["name"] == "Habit Tracker")
|
||||||
|
check("package", e0["package"] == "com.example.habit")
|
||||||
|
check("rating", e0["rating"] == 4.6)
|
||||||
|
for k in ["rank", "name", "package", "rating"]:
|
||||||
|
check(f"key present: {k}", k in e0)
|
||||||
|
check("second package", d["entries"][1]["package"] == "com.example.budget")
|
||||||
|
check("second name", d["entries"][1]["name"] == "Budget Buddy")
|
||||||
|
sys.exit(1 if fails else 0)
|
||||||
|
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
import json, subprocess, sys, os
|
||||||
|
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
SCRIPT = os.path.join(HERE, "..", "skills", "market-research", "scripts", "play.py")
|
||||||
|
SEARCH = os.path.join(HERE, "fixtures", "play-search.html")
|
||||||
|
DETAILS = os.path.join(HERE, "fixtures", "play-details.html")
|
||||||
|
|
||||||
|
def run(*args):
|
||||||
|
return json.loads(subprocess.check_output([sys.executable, SCRIPT, *args]))
|
||||||
|
|
||||||
|
def main():
|
||||||
|
fails = []
|
||||||
|
def check(name, cond):
|
||||||
|
print(f"{'PASS' if cond else 'FAIL'}: {name}")
|
||||||
|
if not cond: fails.append(name)
|
||||||
|
|
||||||
|
r = run("resolve", "Habit Tracker", "--search-file", SEARCH, "--details-file", DETAILS)
|
||||||
|
check("package", r["package"] == "com.example.habit")
|
||||||
|
check("play_url", r["play_url"] == "https://play.google.com/store/apps/details?id=com.example.habit")
|
||||||
|
check("name", r["name"] == "Habit Tracker")
|
||||||
|
check("rating", r["rating"] == 4.6)
|
||||||
|
check("installs", r["installs"] == "1B+")
|
||||||
|
check("last_updated", r["last_updated"] == "Jun 1, 2026")
|
||||||
|
check("developer", r["developer"] == "Focus Labs")
|
||||||
|
|
||||||
|
c = run("count", "habit tracker", "--search-file", SEARCH)
|
||||||
|
check("app_count", c["app_count"] == 3)
|
||||||
|
check("avg_rating", c["avg_rating"] == 4.2) # mean(4.6,4.1,3.9)=4.2
|
||||||
|
check("top_packages has habit", "com.example.habit" in c["top_packages"])
|
||||||
|
check("top_packages distinct", len(c["top_packages"]) == len(set(c["top_packages"])))
|
||||||
|
|
||||||
|
sys.exit(1 if fails else 0)
|
||||||
|
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
import json, subprocess, sys, os
|
||||||
|
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
SCRIPT = os.path.join(HERE, "..", "skills", "market-research", "scripts", "trends.py")
|
||||||
|
FIXTURE = os.path.join(HERE, "fixtures", "trends-sample.json")
|
||||||
|
|
||||||
|
def run(*args):
|
||||||
|
# exit 0 always (never hard-fails); capture stdout
|
||||||
|
out = subprocess.run([sys.executable, SCRIPT, *args],
|
||||||
|
capture_output=True, text=True)
|
||||||
|
return out.returncode, json.loads(out.stdout)
|
||||||
|
|
||||||
|
def main():
|
||||||
|
fails = []
|
||||||
|
def check(name, cond):
|
||||||
|
print(f"{'PASS' if cond else 'FAIL'}: {name}")
|
||||||
|
if not cond: fails.append(name)
|
||||||
|
|
||||||
|
rc, d = run("habit tracker", "--json-file", FIXTURE)
|
||||||
|
check("exit 0", rc == 0)
|
||||||
|
check("ok", d["ok"] is True)
|
||||||
|
check("points", d["points"] == 3)
|
||||||
|
check("latest_interest", d["latest_interest"] == 100)
|
||||||
|
check("avg_interest", d["avg_interest"] == 70.0) # mean(50,60,100)=70
|
||||||
|
check("trend_pct", d["trend_pct"] == 100.0) # (100-50)/50*100
|
||||||
|
|
||||||
|
# bad fixture -> graceful fallback, still exit 0
|
||||||
|
rc2, d2 = run("x", "--json-file", os.path.join(HERE, "fixtures", "play-chart.html"))
|
||||||
|
check("fallback exit 0", rc2 == 0)
|
||||||
|
check("fallback ok false", d2["ok"] is False)
|
||||||
|
check("fallback flag", d2["fallback"] == "websearch")
|
||||||
|
|
||||||
|
sys.exit(1 if fails else 0)
|
||||||
|
|
||||||
|
main()
|
||||||
Loading…
Reference in New Issue