whisper-money/autoresearch.md

Autoresearch: AI rule-suggestion coverage

Objective

Maximize how many of a user's uncategorized transactions the AI rule-suggestion pipeline can categorize. Driven by php artisan ai:suggest-rules <user>.

Workload: user victoor89@gmail.com, reset to a clean slate (0 automation rules, 1329 uncategorized transactions, all server-readable, 64 categories). The data is description-dominated: ~94% of groups key off the free-text description field (noisy Spanish bank descriptors); only ~62 transactions carry a creditor_name/debtor_name.

The pipeline: RuleSuggestionAggregator groups uncategorized tx → caps to max_groups_sent groups with count ≥ min_group_count → Gemini maps groups to (field, operator, token, category) suggestions → RuleSuggestionGuard validates (token ≥3 chars, literal match ≥1, not over-broad > overbroad_fraction, confidence ≥ confidence_floor, category direction agrees) → persisted as rules.

Metrics

• Primary: oracle_tx (count, higher is better) — distinct uncategorized tx that an ideal model + the REAL guard would categorize, given the groups the REAL aggregator sends. Deterministic, zero-variance, instant. Measured by experiments/bench.php with a FROZEN oracle (distinctive-token picker).
• Secondary:
  • reachable_tx — tx living in the groups the aggregator sends (hard ceiling).
  • groups_sent, validated_count, coverage_pct.
  • real_tx — GROUND TRUTH: distinct tx the live Gemini run + guard categorize. Noisy (~±30 over runs: 437/410/400 at baseline). Measured at milestones only via experiments/calibrate_real.php (one Gemini call); not in the loop.

How to Run

./autoresearch.sh — outputs METRIC name=number lines (oracle benchmark). Milestone ground truth: php artisan tinker experiments/calibrate_real.php.

Files in Scope

• config/ai_suggestions.php — thresholds (max_groups_sent, min_group_count, confidence_floor, overbroad_fraction). Single source of truth: the tunable AI_SUGGESTIONS_* overrides were removed from .env, so editing the config default here actually takes effect. Threshold experiments live here.
• app/Services/Ai/RuleSuggestionAggregator.php — grouping + key normalization (the clustering lever; better merchant extraction merges more tx into bigger groups so they pass min_group_count and produce clean tokens).
• app/Services/Ai/RuleSuggestionGuard.php — validation logic.
• app/Ai/Agents/RuleSuggestionAgent.php — the prompt (realization lever; judge ONLY via real_tx, never oracle, since the oracle assumes ideal model).
• app/Services/Ai/LaravelAiRuleSuggestionGenerator.php — batching the model call would let us send more groups without one giant payload.

Off Limits

• experiments/bench.php oracle token heuristic + stoplist are FROZEN. Improving them games the metric. Only real pipeline code/config may change.
• No deleting/weakening tests. No new dependencies without approval.
• Do not re-add AI_SUGGESTIONS_* threshold overrides to .env.
• Do not touch the user's DB rows further (already reset).

Constraints

• LANGUAGE-AGNOSTIC: the user base is pan-European (ES/DE/FR/IT/…), so NO product change may hardcode Spanish (or any single-language) wordlists. Clustering / noise removal must be statistical (e.g. per-user token frequency) or delegated to the multilingual model. The Spanish stoplist in experiments/bench.php is a measurement-only artifact for this one Spanish user — never ship its approach.
• Every kept change must keep the AI test suite green (php artisan test --compact tests/Feature/Ai).
• Keep PHP style (vendor/bin/pint --dirty).
• Threshold/clustering wins are judged by oracle_tx. Prompt/batching wins are judged by real_tx (median of 2–3 runs, since it is noisy).
• Validate real_tx after any kept ceiling change to confirm the live model actually realizes part of the new ceiling (oracle is only an upper bound).

What's Been Tried

• Baseline (max_groups_sent=40, min_group_count=2, overbroad=0.4, floor=0.3): oracle_tx=830, reachable_tx=515, groups_sent=40, real_tx≈416 (median 437/410/400). The live model covers only ~25 of 40 groups → big gap between real (≈416) and the oracle ceiling (830). Two independent levers: raise the ceiling (aggregation/clustering) AND close the realization gap (prompt/batching).

Idea Backlog (rough priority)

1. [DONE r2] max_groups_sent 40 → 150 (covers all count≥2 groups).
2. [DONE r3] Batch the Gemini call so a big payload doesn't make the model under-enumerate (real_tx).
3. LANGUAGE-AGNOSTIC clustering: strip noise tokens by per-user document frequency (words shared across many of THIS user's groups are noise in any language), not by a hardcoded wordlist. Merges merchant variants → more count≥2 groups, cleaner tokens. Replaces the old "strip Spanish noise" idea.
4. min_group_count 2 → 1 (adds 499 singleton groups; ceiling toward 1300).
5. Tune overbroad_fraction / confidence_floor.
6. Prompt: insist on covering EVERY group + multilingual merchant-token extraction (real_tx).
7. Use AI over all transaction descriptions (CSV-style) to discover groups — user idea; explore as an alternative to PHP pre-aggregation.