92 lines
5.2 KiB
Markdown
92 lines
5.2 KiB
Markdown
# Autoresearch: AI rule-suggestion coverage
|
||
|
||
## Objective
|
||
Maximize how many of a user's uncategorized transactions the AI rule-suggestion
|
||
pipeline can categorize. Driven by `php artisan ai:suggest-rules <user>`.
|
||
|
||
Workload: user `victoor89@gmail.com`, reset to a clean slate (0 automation
|
||
rules, **1329** uncategorized transactions, all server-readable, 64 categories).
|
||
The data is description-dominated: ~94% of groups key off the free-text
|
||
`description` field (noisy Spanish bank descriptors); only ~62 transactions
|
||
carry a `creditor_name`/`debtor_name`.
|
||
|
||
The pipeline: `RuleSuggestionAggregator` groups uncategorized tx → caps to
|
||
`max_groups_sent` groups with count ≥ `min_group_count` → Gemini maps groups to
|
||
(field, operator, token, category) suggestions → `RuleSuggestionGuard` validates
|
||
(token ≥3 chars, literal match ≥1, not over-broad > `overbroad_fraction`,
|
||
confidence ≥ `confidence_floor`, category direction agrees) → persisted as rules.
|
||
|
||
## Metrics
|
||
- **Primary**: `oracle_tx` (count, higher is better) — distinct uncategorized tx
|
||
that an ideal model + the REAL guard would categorize, given the groups the
|
||
REAL aggregator sends. Deterministic, zero-variance, instant. Measured by
|
||
`experiments/bench.php` with a FROZEN oracle (distinctive-token picker).
|
||
- **Secondary**:
|
||
- `reachable_tx` — tx living in the groups the aggregator sends (hard ceiling).
|
||
- `groups_sent`, `validated_count`, `coverage_pct`.
|
||
- `real_tx` — GROUND TRUTH: distinct tx the live Gemini run + guard categorize.
|
||
Noisy (~±30 over runs: 437/410/400 at baseline). Measured at milestones only
|
||
via `experiments/calibrate_real.php` (one Gemini call); not in the loop.
|
||
|
||
## How to Run
|
||
`./autoresearch.sh` — outputs `METRIC name=number` lines (oracle benchmark).
|
||
Milestone ground truth: `php artisan tinker experiments/calibrate_real.php`.
|
||
|
||
## Files in Scope
|
||
- `config/ai_suggestions.php` — thresholds (max_groups_sent, min_group_count,
|
||
confidence_floor, overbroad_fraction). **Single source of truth**: the tunable
|
||
`AI_SUGGESTIONS_*` overrides were removed from `.env`, so editing the config
|
||
default here actually takes effect. Threshold experiments live here.
|
||
- `app/Services/Ai/RuleSuggestionAggregator.php` — grouping + key normalization
|
||
(the clustering lever; better merchant extraction merges more tx into bigger
|
||
groups so they pass min_group_count and produce clean tokens).
|
||
- `app/Services/Ai/RuleSuggestionGuard.php` — validation logic.
|
||
- `app/Ai/Agents/RuleSuggestionAgent.php` — the prompt (realization lever;
|
||
judge ONLY via `real_tx`, never oracle, since the oracle assumes ideal model).
|
||
- `app/Services/Ai/LaravelAiRuleSuggestionGenerator.php` — batching the model
|
||
call would let us send more groups without one giant payload.
|
||
|
||
## Off Limits
|
||
- `experiments/bench.php` oracle token heuristic + stoplist are FROZEN. Improving
|
||
them games the metric. Only real pipeline code/config may change.
|
||
- No deleting/weakening tests. No new dependencies without approval.
|
||
- Do not re-add `AI_SUGGESTIONS_*` threshold overrides to `.env`.
|
||
- Do not touch the user's DB rows further (already reset).
|
||
|
||
## Constraints
|
||
- LANGUAGE-AGNOSTIC: the user base is pan-European (ES/DE/FR/IT/…), so NO product
|
||
change may hardcode Spanish (or any single-language) wordlists. Clustering /
|
||
noise removal must be statistical (e.g. per-user token frequency) or delegated
|
||
to the multilingual model. The Spanish stoplist in `experiments/bench.php` is a
|
||
measurement-only artifact for this one Spanish user — never ship its approach.
|
||
- Every kept change must keep the AI test suite green
|
||
(`php artisan test --compact tests/Feature/Ai`).
|
||
- Keep PHP style (`vendor/bin/pint --dirty`).
|
||
- Threshold/clustering wins are judged by `oracle_tx`. Prompt/batching wins are
|
||
judged by `real_tx` (median of 2–3 runs, since it is noisy).
|
||
- Validate `real_tx` after any kept ceiling change to confirm the live model
|
||
actually realizes part of the new ceiling (oracle is only an upper bound).
|
||
|
||
## What's Been Tried
|
||
- Baseline (max_groups_sent=40, min_group_count=2, overbroad=0.4, floor=0.3):
|
||
`oracle_tx=830`, `reachable_tx=515`, `groups_sent=40`, real_tx≈416 (median
|
||
437/410/400). The live model covers only ~25 of 40 groups → big gap between
|
||
real (≈416) and the oracle ceiling (830). Two independent levers:
|
||
raise the ceiling (aggregation/clustering) AND close the realization gap
|
||
(prompt/batching).
|
||
|
||
## Idea Backlog (rough priority)
|
||
1. [DONE r2] max_groups_sent 40 → 150 (covers all count≥2 groups).
|
||
2. [DONE r3] Batch the Gemini call so a big payload doesn't make the model
|
||
under-enumerate (real_tx).
|
||
3. LANGUAGE-AGNOSTIC clustering: strip noise tokens by per-user document
|
||
frequency (words shared across many of THIS user's groups are noise in any
|
||
language), not by a hardcoded wordlist. Merges merchant variants → more
|
||
count≥2 groups, cleaner tokens. Replaces the old "strip Spanish noise" idea.
|
||
4. min_group_count 2 → 1 (adds 499 singleton groups; ceiling toward 1300).
|
||
5. Tune overbroad_fraction / confidence_floor.
|
||
6. Prompt: insist on covering EVERY group + multilingual merchant-token
|
||
extraction (real_tx).
|
||
7. Use AI over all transaction descriptions (CSV-style) to discover groups —
|
||
user idea; explore as an alternative to PHP pre-aggregation.
|