Commit Graph

1 Commits

Author SHA1 Message Date
Víctor Falcón 4038e60fbc
fix(ai): handle transient AI provider overloads — stop the Sentry noise and retry the dropped work (#595)
## Why

Sentry issue **PHP-LARAVEL-3S** (`ProviderOverloadedException: AI
provider [gemini] is overloaded`) recurs whenever Gemini returns 503/429
under high demand — 7 events in 3h during the last surge, 0 users
impacted.

Two problems behind it:

1. **Sentry noise.** `CategorizeTransactions::resolveChunkWithRetry`
retries, `laravel/ai` fails over providers, and a still-failing chunk is
deliberately dropped so the rest of the backfill proceeds. But the catch
reported **every** dropped chunk via `report()`, so an expected,
self-healing transient condition floods Sentry and buries real bugs.
2. **Silently lost work.** A dropped chunk leaves those transactions
uncategorized (`category_id NULL`) with nothing to re-trigger them: the
backfill jobs are one-shot (`tries = 1`), there is no scheduled
backfill, and the real-time listener only handles *new* transactions.
They stay uncategorized until someone manually re-runs a backfill.

## What

**1. Stop reporting transient failures.** Catch `FailoverableException`
(the marker interface for `ProviderOverloadedException` /
`RateLimitedException`) separately and log a warning instead of
reporting it. Everything else still goes to `report()` unchanged, so
real failures (malformed responses, insufficient credits, …) keep
surfacing.

**2. Retry the dropped work.** On a transient failure, schedule a
deferred, per-user `RetryTransientAiCategorizationJob` that re-reads the
user's still-pending transactions once the provider has had time to
recover (`ai_categorization.retry_delay`, default 10 min).
- `ShouldBeUnique` per user collapses a surge of dropped chunks into a
**single** retry.
- The unique lock is held through processing, so a retry that overloads
again **cannot chain another** — exactly one deferred attempt per
failure, no infinite loop. Failed 503s aren't billed.
- Wired in `resolve()`, so it covers every entry point (backfill,
onboarding, real-time listener, admin command).
- Model cost is negligible (per config), so the retry re-runs the
existing backfill rather than tracking which exact chunks failed.

## Tests

- Transient overload → chunk dropped, nothing reported, retry scheduled
for the user.
- Unexpected failure → reported, **no** retry scheduled.
- Retry job → categorizes still-pending transactions for a consenting
user; no-op without consent (agent never prompted).

Full `tests/Feature/Ai` + listeners suite green (111 tests); Pint clean.

## Existing prod backlog

The transactions already dropped before this ships stay `pending`; an
`ai:categorize-backfill <user>` recovers them on demand.

Fixes PHP-LARAVEL-3S
2026-06-26 20:07:06 +02:00