## Issue (Sentry PHP-LARAVEL-44 — low volume: 0 users, 1 event)
`Laravel\Ai\Exceptions\ProviderOverloadedException: AI provider [gemini]
is overloaded` (underlying Gemini HTTP 503 "high demand"), thrown from
the AI rule-suggestion generator. This is an expected, transient,
self-healing provider overload — but
`LaravelAiRuleSuggestionGenerator::generate()` called `report()` on
every failed batch, so a transient hiccup surfaced in Sentry as an
error.
## Fix
Split the per-batch catch so a `FailoverableException` (overload /
rate-limit / insufficient-credits) is logged at `warning` **without**
reporting, while every other `Throwable` is still `report()`ed. This
mirrors the existing, test-enforced handling in the sibling
`CategorizeTransactions` service.
Unchanged (deliberately preserved):
- **Partial tolerance** — suggestions from batches that succeeded are
still kept.
- **Rethrow-when-all-fail** — a genuine total failure still rethrows, so
the run is marked `Failed` (the onboarding "Try again / Skip" UI) and is
reported **once** at the run level. Non-transient batch errors are still
reported immediately.
## What I deliberately did NOT do
Two review agents converged that adding a **retry backoff** (the other
candidate fix) is not worth it here and carries real risk:
- The run **already self-heals**: failed/empty runs don't count toward
the throttle, so the user's "Try again" button re-runs immediately at
zero cost.
- A synchronous per-batch `sleep` on the single-worker `database` queue,
across up to ~10 batches, could push the run past its **120s job timeout
(failing more often, not less)** and starve other queued jobs during a
provider brownout.
So this PR is scoped to the safe, high-signal change: stop reporting an
expected transient as an error. Genuine misconfigurations (bad
model/key/quota) throw non-`FailoverableException` types and remain
fully reported.
## Testing
- New: an expected transient overload returns the successful batches'
suggestions and is **not** reported
(`Exceptions::assertNothingReported()`); an unexpected batch failure
**is** reported (`assertReported(RuntimeException::class)`).
- Full `tests/Feature/Ai` suite green (121 tests); Pint and Larastan
clean.
Fixes PHP-LARAVEL-44
---
🤖 Opened by the autonomous Sentry-triage loop. Auto-merge enabled:
low-risk, mirrors an existing tested pattern, keeps genuine failures
visible.
## What & why
Fixes **PHP-LARAVEL-2D** — `MaxAttemptsExceededException:
CategorizeUncategorizedTransactionsJob has been attempted too many
times.` (recurring, still firing as of yesterday, ~18 events).
### Root cause
The `database` queue connection used `retry_after = 90s`, but several
jobs declare a `$timeout` far above it (up to **600s**). `retry_after`
is **per-connection**. When a job outlives its 90s reservation, the DB
queue driver hands it to a second worker. For a `tries = 1` job that
immediately raises `MaxAttemptsExceededException` — and, worse, the body
can execute twice. `CategorizeUncategorizedTransactionsJob` runs on the
`ai` queue, which the deploy runs with **two concurrent workers**
(`docker/supervisor/supervisord.conf`), so the double-run re-bills
Gemini and resets progress — the exact harm its `tries = 1` was meant to
prevent.
At `retry_after = 90` **8 jobs** violated the invariant (600s rule jobs,
300s AI jobs, 120s sync jobs).
## Changes (one concern per commit)
1. **fix(queue): raise `retry_after` above the longest job timeout** —
default `90 → 900s` (env-overridable via `DB_QUEUE_RETRY_AFTER`),
clearing the 600s longest job with margin. Adds `QueueConfigTest`
asserting every `app/Jobs` `$timeout` stays `< retry_after` so a new
long job can't silently re-break it.
2. **fix(ai): de-duplicate the backfill per user** — `retry_after` stops
one dispatch being re-reserved, but not a *duplicate* dispatch (double
"Enable AI" click / re-enable mid-run) on the 2-worker `ai` queue.
Implement `ShouldBeUnique` keyed on user id, mirroring the sibling
`RetryTransientAiCategorizationJob`.
3. **test(queue): widen the guard** to also scan queued Mailables
(`app/Mail`) and Notifications (`app/Notifications`), which share the
connection and honor `$timeout`.
4. **docs(queue): correct the `retry_after` comment** — per-job
`$timeout` takes precedence over the worker `--timeout` (which is
currently unset), so the earlier "must sit between" phrasing was
inaccurate.
## Verification
- `QueueConfigTest` fails at the old `retry_after=90` (8 offenders) and
passes at `900`.
- `vendor/bin/pint --test` clean; 131 tests across `tests/Feature/Ai`,
AI-consent flow, and queue config pass.
- Confirmed against the real deploy: 2 workers on the `ai` queue,
`pcntl` installed (SIGALRM enforced), workers set no `--timeout` so each
job's own `$timeout` bounds execution below 900s.
## Reviewed by two independent agents (architecture + product)
Both concluded **ship it**. The double-dispatch gap they flagged is
closed by change #2. Neither found a correctness regression.
## Operational notes (no code change needed, flagged for awareness)
- **Reclaim window:** `retry_after` is per-connection, so
`emails`/`default` also inherit the 90→900s window. A job orphaned by a
*hard* worker crash (OOM/SIGKILL) is now reclaimed after up to ~15 min
instead of ~90s. Graceful deploys (SIGTERM + `stopwaitsecs=3600`) don't
orphan jobs, so this only bites on hard crashes — an acceptable trade. A
dedicated connection for the long AI queue would remove it; left as a
possible follow-up.
- **Worker `--timeout`:** the fix relies on `pcntl` staying installed
and long jobs keeping a `$timeout`. Setting an explicit `--timeout` (<
900) on the supervisord `queue:work` commands would make it
self-enforcing; not done here to avoid changing short-job hang
detection.
- If prod sets `DB_QUEUE_RETRY_AFTER` explicitly, ensure it's above the
longest job timeout (≥ 600, ideally 900).
## Why
Two issues surfaced from a real mis-categorization corrected via the
**Edit Transaction** modal (not the table quick-edit).
### 1. The modal never told the user the AI learned
Both the table quick-edit and the edit modal hit the same backend
(`PATCH /transactions/{id}` → `CategoryOverrideHandler` →
`AiRuleLearner`), so both learn a forward rule from a correction. But
the modal **discarded** the update response and never read
`learned_rule`:
- No *"Learned: similar transactions will be categorized automatically"*
confirmation.
- No **Undo**.
- It still showed the generic *"Automatize"* prompt — offering to create
a rule that already exists.
The modal now reads `learned_rule` and mirrors the table's toast + undo,
skipping the automatize prompt when a rule was learned. Same i18n
strings as the table (no new keys).
### 2. The learner could key a rule on a single weak token
A correction on `"Pago en MADRID SUC 04 MADRID ES"` produced a rule
keyed on `suc` alone. The tokenizer drops noise by **per-user** document
frequency, so `madrid` (20% of this user's descriptions) is correctly
dropped — but `suc` (0.14%) survives, even though it is a generic
banking abbreviation that matches broadly as a substring. The per-user
filter can't see that `suc` is generic in general.
`AiRuleLearner::descriptionClause()` now refuses a single distinctive
token shorter than 5 characters. It learns only when there are **two or
more** distinctive tokens, or a single token of **at least 5
characters**. Merchant-keyed rules are unaffected.
## Tests
- `does not learn a description rule from a single short token`
- `learns a description rule from a single sufficiently long token`
- `learns a description rule from two short tokens`
All `AiRuleLearnerTest` pass (13/13).
## Not in this PR
The one already-learned prod rule still carries the stale `suc`
clause/title (low impact, matches ~2 txns). Cleaning that data is a
separate manual step.
## Summary
Starts from a security finding on the background-job status endpoints
and folds in the most important issues surfaced by a follow-up review of
the same feature area (categorization backfill, bulk rule re-evaluation,
automation-rule apply).
Each change is its own commit.
## Changes
### 1. `fix(security)` — scope job-status cache keys to the owning user
The categorization, bulk re-evaluation, and apply status endpoints
looked jobs up by a bare job UUID with no user scoping. Any
authenticated user who obtained another user's job id could poll its
progress payload. Cache keys now include the owning user's id, so a
status request keyed by the polling user's id resolves only that user's
own jobs — a mismatched owner falls through to the existing 404. No
ownership store or extra lookup. Cross-user isolation tests added for
all three endpoints.
### 2. `fix(automation-rules)` — re-check `only_uncategorized` at apply
time
The apply flow cached a match snapshot for up to 15 min (keyed only by
`rule.updated_at`) and applied the rule's category to every id in it
without re-checking eligibility. A transaction categorized *after* the
snapshot (by the user, a sibling rule, or a concurrent AI backfill) was
silently re-categorized and stamped `category_source=Rule`. Now
re-filtered through `shouldSkipForOnlyUncategorized()` at apply time;
skipped rows are no longer counted as changed.
### 3. `perf(automation-rules)` — memoize rule set per user in
`applyRules()`
Bulk re-evaluation calls `applyRules()` once per transaction, and each
call re-queried the user's whole rule set + labels — an N+1 scaling with
transaction count (the sibling apply job already loaded rules once).
Memoized per user for the service instance's lifetime (resolved fresh
per job, so rules created mid-run are intentionally not seen).
Query-count test asserts one `automation_rules` query regardless of
transaction count.
### 4. `fix(transactions)` — stop polling when the job never starts
The re-evaluate and apply pollers rescheduled on any non-terminal
status, including `pending`. If the queue worker is down the job never
runs, `failed()` never fires, and the client polled for the full
hour-long TTL with a stuck spinner. Now gives up after 30 consecutive
`pending` ticks (~30s), mirroring the guard the AI-categorization poller
already has. Long `processing` runs are unaffected.
### 5. `test(automation-rules)` — cover apply job execution and failure
branches
`ApplySingleAutomationRuleJob` was only asserted to be *pushed*; its
`handle()` body and `failed()` branch had no coverage, and
`ReEvaluateTransactionRulesJob::failed()` was untested too. Added direct
`handle()`/`failed()` tests pinning the progress-cache payloads.
## Deferred follow-ups (surfaced by review, not in this PR)
- **Consolidate the three frontend pollers onto `usePollJobStatus`** —
they hand-roll `setTimeout` loops and don't tear down on unmount (can
`setState`/`onClose` after the dialog closes). The hook already exists
and is unmount-safe; routing all three through it would also make the
`pending` cap unit-testable. The cap in change #4 currently mirrors the
already-shipped AI poller and is not separately unit-tested.
- **`noteAlreadyPresent()` uses a substring match**
(`AutomationRuleService`) — a rule note that is a substring of an
existing note is silently not appended. Should compare note lines
exactly.
- **No dedup/lock on concurrent apply/re-evaluate jobs** — note appends
are a non-atomic read-then-write, so two concurrent jobs can duplicate a
note. Category/label writes are already idempotent.
- **Transient 404 → false "failed"** — a mid-run cache
eviction/TTL-expiry (or the deploy window of change #1, where an
in-flight job's pre-deploy key is orphaned) makes the poller surface a
false failure. Retry a few times on transient errors before giving up.
- **`matches()` pagination** advances `next_offset` by fetched rows, not
page-window size — deleted ids can stall infinite scroll before reaching
`total`.
- **"Apply to N" count / encrypted-skip counts** can be stale or
under-report vs what's actually processed.
- **Job-trait convention drift** between the three jobs (modern
`Queueable` vs legacy trait stack).
## Testing
- `vendor/bin/pint --test` — pass
- `bun run format` / `bun run lint` — pass (one pre-existing unrelated
warning in `chart.tsx`)
- Affected Pest suites (apply, re-evaluate, evaluation, rule,
categorization, apply-rule-suggestions) — 94 tests pass
> **Draft on purpose — a partial fix + a design for the full one.**
These commits are safe, behavior-preserving reductions of the N+1, but
they do **not** fully resolve PHP-LARAVEL-40. The complete fix is a
batch-aware refactor of the AI rule-learning path, which is delicate
(getting it wrong silently mis-categorizes users' transactions). I've
written the design below for a human to implement and diff against the
current per-transaction behavior. Current user impact is low (~170 ms
request, 1 event), so there's no urgency to rush the risky part.
## What
Sentry **PHP-LARAVEL-40** — N+1 in `TransactionController@bulkUpdate`
(PATCH `/transactions/bulk`). A bulk category update calls
`CategoryOverrideHandler::record()` once per transaction, and each call
runs the full AI rule-learning pipeline
(`AiRuleLearner::forgetFromAiRules()` + `learnFromCorrection()`):
loading all the user's AI rules, plucking every description, running
matcher `count(*)` probes, and inserting a `category_corrections` row.
For a 112-transaction batch that's ~112× each. The offending span is the
per-transaction `category_corrections` insert.
## What this branch does (safe, partial)
Two behavior-preserving reductions, each validated by the existing
learning tests:
1. `perf(transactions): resolve the override handler once for bulk
updates` — `bulkUpdate()` re-resolved `CategoryOverrideHandler` from the
container every iteration. Resolve it once (also required for #2 to take
effect).
2. `perf(ai): memoize the description corpus per user in AiRuleLearner`
— `learnFromCorrection` reloaded and re-tokenized every one of the
user's descriptions on each transaction. The corpus is immutable while
only categories change, so memoize the document-frequency map + count
per user. Removes one `SELECT` (and its tokenization) per transaction.
3. `test(ai): assert batch corrections learn correctly through the
memoized corpus` — proves the second, memoized-corpus learning still
yields a correct, distinct clause (not just that the query count
dropped).
4. `test(ai): guard AiRuleLearner against a singleton binding` — the
cache has no invalidation and is only safe while the learner is resolved
fresh per request; a test now fails if it is ever bound singleton.
## What it does NOT do
It does **not** resolve the flagged N+1. Still per-transaction: the
`category_corrections` insert, `forgetFromAiRules`' reload of all AI
rules, the matcher `total()`/`countMatchingAll()` overbroad probes,
`releaseClauseFromOtherCorrectionRules`, and `existingCorrectionRule`. A
learnable transaction still costs ~6–10 queries. Treat PHP-LARAVEL-40 as
**reduced, not closed** (hence no `Fixes` keyword).
## Reviewed by two independent agents (architecture +
product/correctness)
Both verified the two perf commits are **behavior-preserving and safe**:
the memo cannot go stale (nothing in the correction path mutates
descriptions; the bulk `update()` only writes
`category_id`/`category_source`/`ai_confidence`/`categorized_by_rule_id`,
and runs after the loop), no cross-user leak (keyed by user_id; learner
is transient, one user per request, no Octane), and the handler is
stateless. The existing 34 learning tests exercise the corpus→rule
computation on fresh loads.
## Proposed full fix (for a human to implement)
Exploit that a bulk update sends **all transactions to the same category
for one user.** Add `CategoryOverrideHandler::recordBulk(Collection,
?string)` backed by `AiRuleLearner::learnFromCorrections(array,
string)`; load user-level data once, mutate in memory, write once:
1. **Batch-resolve** `categorized_by_rule_id` via one `whereIn` instead
of per-txn `find()`; apply the **identical** per-txn learnable test
against the map.
2. **Batch-insert** corrections: snapshot each AI-driven txn's *old*
`from_category_id`/`source`/`confidence` during the loop, collect rows,
one insert. (Note: raw `insert()` skips model events/UUID/timestamps —
verify `CategoryCorrection` has no `creating` hook, else chunked
`create` in one transaction.)
3. **Forget once**: union all learnable txns' merchant tokens, load AI
rules once, apply the **whole union** to each rule *before* the
delete-vs-save decision, save/delete each once.
4. **Learn once**: compute each clause (merchant or memoized-corpus
tokens), dedupe within the batch and against the target rule, run
`releaseClauseFromOtherCorrectionRules` for the union once, load/create
the single target correction rule once, append all, save once.
5. Wrap 2–4 in one `DB::transaction`, and route the single-txn path
through the same method (one-element collection) so the two can't drift.
**Correctness invariants to preserve (call these out in review):**
- Apply-all-then-decide for both `forget` and `releaseClause` deletes —
an incremental forget/save/forget would delete a rule a later txn's
clause should have kept.
- Corrections must read each txn's **old** category/source/confidence —
run before the bulk `update()`, as today.
- Clause dedup must match `appendClause`'s loose `==` array comparison.
- Validate with a golden-set test diffing learned/forgotten rules
before/after against the current per-txn path over a mixed batch
(merchant txns, description txns, encrypted/no-merchant skips,
re-corrections that move a key between categories).
## Testing
- `tests/Feature/Ai/` + `BulkUpdateTransactionsTest` + IDOR/decrypt:
153/153 green. `pint` clean.
Refs PHP-LARAVEL-40
## What & why
Fixes the Sentry N+1 **PHP-LARAVEL-3V** — `N+1 insert into jobs` on
`POST /transactions`.
### Root cause
Laravel's **event discovery is on by default** (the base
`EventServiceProvider` sets `$shouldDiscoverEvents = true`, and this app
has no subclass to turn it off), so every listener in `app/Listeners` is
already wired to its event by its `handle()` type-hint.
`AppServiceProvider::boot()` was **also** registering six of them with
`Event::listen(...)`, so each ended up registered **twice** — every
queued listener published its job twice per event.
Concretely, creating a transaction dispatched
`AssignTransactionToBudget` and `CategorizeTransactionWithAi` twice each
→ repeated `insert into jobs` in one request. (The same bug made
`PostStripeEventToDiscord` post to Discord twice per webhook,
`ApplyAutomationRules` run twice, etc.)
## Changes
1. **`AppServiceProvider`** — drop the redundant explicit
`Event::listen` calls and rely on discovery, which already wires every
listener exactly once (`ScheduleDripEmailsListener` and
`SyncUserToResendListener` were discovery-only already). Verified with
`php artisan event:list`: each event now has a single listener, and the
synchronous `ApplyAutomationRules` still runs first on
`TransactionCreated`.
2. **`CategorizeTransactionWithAi`** — add `shouldQueue()` so the job
isn't even published when it would immediately no-op (already
categorized by a rule, encrypted, or user not AI-eligible — the default,
since AI is opt-in). The same guard still runs in `handle()` because
state can change while the job waits.
Together these remove the second `insert into jobs` on transaction
creation for the common (non-AI) path.
## Tests
- New `tests/Feature/Ai/CategorizeTransactionWithAiListenerTest.php`:
the AI job is queued **once** for an eligible uncategorized transaction,
and **not at all** when the user has no AI consent or the transaction is
already categorized.
- Full suite: **1828 passing**. The one failure (`DashboardTest` → 409)
is **pre-existing on `main`** (Inertia asset-version mismatch from a
local build manifest) and reproduces with these changes reverted —
unrelated to this PR.
Fixes PHP-LARAVEL-3V
## Problem
Users keep correcting the same transactions over and over. The AI
mislabels a merchant (e.g. supermarket → fuel), the user fixes it, and
the next near-identical transaction from that merchant gets mislabeled
the same way again. Today a correction is logged and the offending ai
rule is self-healed, but the system only *forgets* its mistake — it
never *remembers* the user's fix.
## Approach
A correction now becomes a deterministic, forward-looking
`AutomationRule` (new `RuleOrigin::Correction`). The next matching
transaction is categorized by that rule **before the model ever runs**
(`ApplyAutomationRules` is synchronous and runs ahead of AI
categorization), ending the loop. Zero model cost, instant, reuses the
existing rule engine.
**Matching key** (in order):
1. **Merchant** (`creditor_name`/`debtor_name`, exact `==`) when present
— stable even as the description varies.
2. Otherwise the **description's distinctive tokens** (`in` /
AND-of-`in`), extracted by the shared `DescriptionTokenizer` (noise
tokens dropped by document frequency, language-agnostic), **guarded**
against over-broad rules that could silently mis-file en masse. If
guarded out → nothing is learned, silently.
## Deliberate decisions (from a design walkthrough)
- **Forward-only**: never retroactively re-categorizes existing
transactions.
- **Learn only from system categorizations** (AI label, ai rule, or a
prior correction rule) — never from one-off manual filing, bank
categories, or the user's own hand-authored rules.
- **A key lives in exactly one correction rule**, so changing your mind
moves it to the new category. Correcting a transaction a prior
correction rule categorized is also learnable, so correction rules stay
fixable in-flow.
- Correcting to *uncategorized* learns nothing but still self-heals the
ai rule.
- **Safety net**: correction rules are visible/editable in
`settings/automation-rules` (marked with the AI sparkle, tooltip
"Learned from your correction"); the transactions table shows a toast
with an instant **Undo**.
## Review pass (two independent agents + live QA)
- **HIGH fix** (`67fc4293`): an ai rule could out-rank a freshly learned
correction and re-apply the wrong category (when the corrected
transaction was a *direct* model label with no rule id). Now every ai
rule holding the merchant is swept on correction. Regression test added.
- **Refactor** (`ca743fde`): collapsed duplicated clause-append logic;
removed a speculative unused enum helper.
- **Coverage** (`12ceb0f7`): debtor_name path, single-token description
clause, encrypted-description fail-safe.
- **Toast conflict fix** (`382c8169`, found in live QA): correcting an
AI transaction fired both the new "Learned …" toast and the pre-existing
"Transaction categorized → Automatize" prompt, which invited the user to
manually create the rule the correction had just created. Made them
mutually exclusive. Adds a browser test for the inline-correction flow.
- **Settings icon** (`f3f882b6`): correction rules now show the AI
sparkle in settings, like ai rules.
## Verified end-to-end (against a running instance)
Drove the real UI with a browser: correcting an AI-mislabeled
transaction creates the correction rule, shows the "Learned · Undo"
toast (no competing Automatize prompt), and Undo deletes the rule while
keeping the correction. Confirmed for **merchant** keys and the
**description-only** path — including that a later "practically
identical" description (different surrounding text, no merchant) is
caught by the rule, while a near-miss sharing only one distinctive token
is correctly **not** caught.
## Open question for reviewers
**`debtor_name` (P2P) as a rule key.** For incoming transfers the
merchant key falls back to the sender's name, so correcting one can
create a rule keyed on a person's name (useful for recurring transfers
from a roommate, but a possible privacy surprise; the name appears in
the rule title in settings). This matches the existing tier-2 learner's
behaviour. Keep as-is, or restrict correction rules to `creditor_name`
only? Happy to change.
## Testing
- `tests/Feature/Ai/CategoryOverrideHandlerTest.php`: merchant +
description learning, next-transaction match, over-broad rejection,
change-of-mind move, correct-to-null self-heal, the HIGH regression,
debtor_name, single-token, encrypted fail-safe.
- `tests/Browser/CategoryCorrectionLearningTest.php`: inline correction
→ toast → learned rule → undo.
- `automation-rule-title.test.tsx`: the AI sparkle shows for `ai` and
`correction`, not `user`.
- Full AI suite green (106 tests); transaction/bulk-update suites green
(62). Pint + Prettier + ESLint clean.
No new dependency, no migration (the `origin` column is a free-text
string). The feature is implicitly gated by AI categorization — with no
AI categorization there is nothing to correct and nothing is learned.
## Why
Sentry issue **PHP-LARAVEL-3S** (`ProviderOverloadedException: AI
provider [gemini] is overloaded`) recurs whenever Gemini returns 503/429
under high demand — 7 events in 3h during the last surge, 0 users
impacted.
Two problems behind it:
1. **Sentry noise.** `CategorizeTransactions::resolveChunkWithRetry`
retries, `laravel/ai` fails over providers, and a still-failing chunk is
deliberately dropped so the rest of the backfill proceeds. But the catch
reported **every** dropped chunk via `report()`, so an expected,
self-healing transient condition floods Sentry and buries real bugs.
2. **Silently lost work.** A dropped chunk leaves those transactions
uncategorized (`category_id NULL`) with nothing to re-trigger them: the
backfill jobs are one-shot (`tries = 1`), there is no scheduled
backfill, and the real-time listener only handles *new* transactions.
They stay uncategorized until someone manually re-runs a backfill.
## What
**1. Stop reporting transient failures.** Catch `FailoverableException`
(the marker interface for `ProviderOverloadedException` /
`RateLimitedException`) separately and log a warning instead of
reporting it. Everything else still goes to `report()` unchanged, so
real failures (malformed responses, insufficient credits, …) keep
surfacing.
**2. Retry the dropped work.** On a transient failure, schedule a
deferred, per-user `RetryTransientAiCategorizationJob` that re-reads the
user's still-pending transactions once the provider has had time to
recover (`ai_categorization.retry_delay`, default 10 min).
- `ShouldBeUnique` per user collapses a surge of dropped chunks into a
**single** retry.
- The unique lock is held through processing, so a retry that overloads
again **cannot chain another** — exactly one deferred attempt per
failure, no infinite loop. Failed 503s aren't billed.
- Wired in `resolve()`, so it covers every entry point (backfill,
onboarding, real-time listener, admin command).
- Model cost is negligible (per config), so the retry re-runs the
existing backfill rather than tracking which exact chunks failed.
## Tests
- Transient overload → chunk dropped, nothing reported, retry scheduled
for the user.
- Unexpected failure → reported, **no** retry scheduled.
- Retry job → categorizes still-pending transactions for a consenting
user; no-op without consent (agent never prompted).
Full `tests/Feature/Ai` + listeners suite green (111 tests); Pint clean.
## Existing prod backlog
The transactions already dropped before this ships stay `pending`; an
`ai:categorize-backfill <user>` recovers them on demand.
Fixes PHP-LARAVEL-3S
## Why
We pay for AI transaction categorization on Gemini, but we couldn't tell
which model produced any given suggestion. The `gemini-flash-latest`
alias used by categorization is **floating** — it silently drifted from
the `flash-lite` tier up to `gemini-3.5-flash`, which is now the
dominant cost on the Gemini dashboard. Before swapping categorization to
a cheaper pinned model we need to measure accuracy per model, and for
that we have to know which model labelled each row.
We already store the AI-suggested category and its confidence on
`transactions`; this adds the missing third axis: the model.
## What
- New nullable `ai_model` column on `transactions`, written by
`CategorizeTransactions::recordOutcome` from
`config('ai_categorization.model')` whenever the model produces a
suggestion (above or below the label bar).
- The migration backfills existing AI-categorized rows
(`ai_suggested_category_at` not null) with `gemini-3.5-flash` — the
model `gemini-flash-latest` resolved to when those suggestions were
made. Done as a raw update so it doesn't bump every row's `updated_at`.
- `ai_model` added to `$fillable` and to `$hidden` (pure ops/measurement
metadata, no need to ship it to the frontend).
## Testing
- Extended `CategorizeTransactionsTest` to assert `ai_model` is stored
on both the auto-applied and below-bar paths.
- `php artisan test tests/Feature/Ai/CategorizeTransactionsTest.php` → 5
passed.
## Follow-ups (not in this PR)
- Pin `AI_CATEGORIZATION_MODEL` to an explicit cheaper tier (mirror the
existing `AI_SUGGESTIONS_MODEL=gemini-2.5-flash-lite`) and kill the
floating alias in the config defaults.
- Capture token usage from the SDK response for cost tracking.
## Summary
Lets existing (non-onboarding) users discover and manage AI
categorization, gated behind a new Pennant feature flag. When a flagged
paid user grants consent, every uncategorized transaction is categorized
in the background with live, on-screen feedback.
Enable per user: `php artisan feature:enable AiConsentSettings
user@example.com`
## What's included
**Feature flag**
- New `AiConsentSettings` Pennant flag (off by default), exposed to the
frontend via the shared `features` prop.
**Billing settings (`/settings/billing`)**
- When the flag is active, a section to grant or revoke AI consent
(checkbox → `POST`/`DELETE /ai/consent`).
**Transactions page**
- A consent prompt rendered as the first row of the transactions table
(new `DataTable` `topRow` slot), styled with the shared gradient
`AiSparkleIcon`. Only shown to paid users with the flag who haven't
consented yet.
- An inline "Enable AI" button records consent without leaving the page.
**Backfill on consent**
- Granting consent dispatches a queued
`CategorizeUncategorizedTransactionsJob` that categorizes all of the
user's uncategorized transactions, **most recent first**, recording
progress in the cache.
- The transactions page polls a new status endpoint (`GET
/ai/categorization/{jobId}/status`): visible uncategorized rows show a
spinner + pulse while the backfill runs, categories stream in via
partial reloads, and a toast tracks `processed / total` until
completion.
## Notes / decisions
- **Polling, not websockets** — the project has no broadcasting
configured; this mirrors the existing `ReEvaluateTransactionRules` job +
status-endpoint pattern.
- The backfill job is kept separate from
`CategorizeOnboardingTransactionsJob` so the onboarding pass stays
progress-free (small, commented duplication).
- Reuses the existing `AiCategorizationGate` (config kill-switch + pro
plan + active consent) — free users and disabled-AI never dispatch the
job.
- Not included (YAGNI): cancelling an in-flight backfill on revoke;
persisting the spinner across full page reloads.
## Testing
- New Pest coverage: feature-flag exposure, billing/transactions consent
props, consent-triggered dispatch + gating, the status endpoint
(200/404), job progress recording, and recent-first ordering.
- `php artisan test --exclude-testsuite=Browser` passes (one unrelated,
pre-existing `DashboardTest` failure on `main`).
- Pint, Prettier and ESLint clean.
## What
Two related changes to the AI auto-categorization feature.
### 1. Open the gate to pro + consent (drop the new-signups-only cohort)
Eligibility for AI auto-categorization was: kill switch **+ pro plan +
active AI consent + a Pennant rollout flag** that only resolved for
users created after `ai_categorization.rollout_after`. That last cohort
gate limited the feature to a handful of recent signups (6 eligible
users in prod).
The gate is now just **kill switch + pro plan + active AI consent**, so
every consented pro user is eligible regardless of signup date.
- `allows()` and `allowsBackfill()` became identical and collapse into a
single `allows()`; `CategorizeBackfillCommand` calls it.
- The `AiCategorization` Pennant feature and the
`ai_categorization.rollout_after` config are now dead and removed. The
weekly cohort report already derives its release marker from the first
`ai_consents.accepted_at`, so nothing depends on the config.
> Note: the `AI_CATEGORIZATION_ROLLOUT_AFTER` env var must be removed
from the production environment — it is no longer read.
### 2. Nudge free users that AI could categorize their transactions
Free-plan users now see a subtle AI sparkle on uncategorized rows,
reusing the trailing icon slot already in `CategoryCell` (no layout
change). It only shows when subscriptions are enforced and the user is
not pro; clicking it routes to `/settings/billing`.
To keep it subtle, the sparkle is sampled to a share of rows via a
deterministic function of the transaction id (its last byte mapped onto
a 0-100 threshold), so the same rows decide the same way across reloads
instead of flickering or marking every row.
The share is **configurable** via `ai_categorization.upsell_sample_rate`
(env `AI_CATEGORIZATION_UPSELL_SAMPLE_RATE`, default 40), exposed to the
frontend as the `aiCategorizationUpsellRate` Inertia prop — no rebuild
needed to retune it.
## Tests
- PHP: `AiCategorizationGateTest` updated (rollout case dropped);
job/listener tests no longer activate the removed feature.
- JS: `ai-upsell-sample.test.ts` covers determinism, the 0/100 bounds,
the threshold boundary, and the per-rate split.
- Manually verified in-app (Playwright): nudge renders for a free,
consented, bankless user; rate matches config.
## Follow-ups (not in this PR)
- Remove the `AI_CATEGORIZATION_ROLLOUT_AFTER` prod env var.
- Backfilling existing pro+consent users' history still requires running
`ai:categorize-backfill {user}`.
- The nudge only reaches free users who pass the paywall (no bank
connection + paywall already seen); free users with a bank connection
never reach `/transactions`. Consider surfacing it on the paywall too.
## Contexto
El categorizador solo persistía un resultado cuando la confianza
superaba la barra (`ai_categorization.label_confidence`, 0.7). Las
sugerencias por debajo del umbral se calculaban y se descartaban,
dejándonos sin ninguna visibilidad de qué proponía el modelo para las
transacciones que no se autocategorizaban.
## Cambio
Se guarda la sugerencia del modelo en **toda** transacción que el modelo
coloca, mediante columnas tipadas (en vez de un blob JSON):
- `ai_suggested_category_id` (FK a `categories`, nullable)
- `ai_suggested_category_at` (timestamp, nullable)
- `ai_confidence` (ya existía) — ahora se rellena **siempre**, no solo
al aplicar
Por debajo de la barra la transacción sigue sin categorizar, pero la
sugerencia queda guardada; a partir de la barra, además se autoaplica la
categoría como hasta ahora.
## Para qué sirve
- **Afinar el umbral 0.7 con datos reales**: `where
ai_suggested_category_id is not null and category_id is null` muestra
exactamente qué nos perdemos.
- Habilita una futura UI de "la IA cree que es X, ¿confirmas?"
directamente desde el FK.
## Tests
- `CategorizeTransactionsTest`: el caso sub-umbral ahora verifica que la
sugerencia se persiste; el caso aplicado verifica que además se guardan
los campos de sugerencia.
- Suite de IA + tests que tocan transacciones en verde (111 tests).
## What
During onboarding, the per-transaction AI categorization listener no
longer runs. The AI automation rules generated in the suggestions step
cover the bulk of the imported transactions (~70%). On completion, a
single batch pass categorizes whatever is still uncategorized.
## Why
Running per-transaction categorization on every transaction created
during onboarding is wasteful: most are about to be covered by the AI
rules generated at the end of the flow. Defer the AI spend to one batch
over the remainder.
## How
- **`CategorizeTransactionWithAi` listener** — skips when `!
$user->isOnboarded()`. Covers every txn-creation path, since it hangs
off the model `created` event.
- **`CategorizeOnboardingTransactionsJob`** (new) — dispatched from
`OnboardingController::complete` after `onboarded_at` is set. Snapshots
`whereNull('category_id')->whereNull('description_iv')`, so transactions
already categorized by the AI rules (or manually in the categorize step)
are excluded. Chunks by `group_batch_size` and runs `AiCategorizer` per
chunk. Runs on the dedicated `ai` queue.
- Gated by `AiCategorizationGate::allows()` (rollout flag) — automatic
system behavior, gated like the real-time tier, not the explicit
backfill.
## Flow
1. `syncing` → transactions created, **no** per-transaction AI
2. `ai-suggestions` → AI rules cover the bulk
3. `categorize-transactions` → user manually categorizes a few
4. `complete` → batch-categorize the rest, skipping anything already
labeled
## Tests
- Listener skips transactions created while onboarding; still
categorizes post-onboarding.
- Job categorizes the remainder, leaves rule/manual categories
untouched, no-ops for ineligible users.
- `complete` marks the user onboarded and queues the batch job.
Full Ai + Onboarding suites green (112/112). Pint clean.
## What
Auto-categorizes transactions with AI (Gemini) for **pro +
AI-consented** users when no automation rule already matched. Ships the
full backend **behind a Pennant flag, off by default**, so it's
mergeable and testable in isolation; the UI is a deliberate follow-up.
## Why / cost
Prod check first: ~20k txns/month, **~52% of pro-user transactions are
uncategorized** after rules. At Gemini Flash-Lite rates the cost is a
**rounding error** — ~$0.13–$0.75/month for all pro users, single-digit
dollars even on full Flash. So the model is chosen for accuracy, not
price; the real constraints are trust, accuracy and privacy.
## How it works
**Two tiers** (every transaction is covered, rules are an optimization
on top):
- **Tier 1 – label** — a queued listener runs *after* the synchronous
rules; if still uncategorized and the user is eligible, the model picks
a **leaf** category (referenced by numeric index, never a UUID, so it
can't hallucinate one). Auto-applied only above the **label bar**
(`0.7`); below → left blank, no nag. Tagged `category_source = ai` +
`ai_confidence`, fully reversible.
- **Tier 2 – learn** — above the higher **rule bar** (`0.85`) *and* a
clean merchant key *and* the model flags the merchant unambiguous → the
merchant is appended to a single **ai-owned** automation rule for that
category (OR'd conditions, not rule-sprawl), so future transactions
match for free and consistently. AI rules sit at the lowest priority;
**user-owned rules are never touched**.
**Self-heal + signal** — when a user overrides an AI category, a
`category_correction` is logged (calibration signal, bucketable by
confidence) and the offending merchant condition is dropped from the ai
rule (deleted if empty). User rules and manual categories are untouched.
**Safety** — config kill switch + pro + active consent + gradual Pennant
rollout. Dedicated `ai` queue so Gemini never blocks bank syncs.
Encrypted (client-side) transactions are never sent.
**Backfill** — `ai:categorize-backfill {user}`, explicit opt-in,
batched, learns rules as it goes.
## Data model
- `transactions`: `category_source`, `ai_confidence`,
`categorized_by_rule_id`
- `automation_rules`: `origin` (`user`/`ai`)
- new `category_corrections` table
## Screenshots
<img width="921" height="384" alt="image"
src="https://github.com/user-attachments/assets/f04c2a03-b39e-4a3d-81eb-ecf26eaefb83"
/>
Suggests transaction categorization rules during onboarding.
After a sync or import, it groups the uncategorized transactions, asks
Gemini (via laravel/ai) to map the common merchants to categories, and
shows the results for review. The user edits or drops any and creates
the ones they want. During onboarding the accepted rules also categorize
existing transactions right away.
Off by default: it needs the `AiRuleSuggestions` Pennant flag and a
per-user AI consent. The model and thresholds are config-driven.
`ai:suggest-rules {user}` prints what a user would get.
The settings-page surface and monthly regeneration are a follow-up.
Víctor Falcón