> The Sentry MCP token is expired, so this cycle worked from the
production database and `failed_jobs` instead. That turned out to
matter: a worker timeout never reaches a job's `try/catch`, so it can
corrupt state while producing **no Sentry issue at all**.
## The bug
`banking_connections.consecutive_sync_failures` is what keeps a
connection in the scheduled rotation. At `MAX_SCHEDULED_RETRIES` both
`SyncAllBankingConnectionsJob` and the `banking:sync` command filter it
out and **nothing ever dispatches it again**. Nobody is told: the bank
consent is still valid, so it never reaches the "reconnect your bank"
notice. The user's data just stops.
Two connections (2 users) are sitting there right now. One has never
completed a single sync since 2026-06-07.
## What I got wrong, and what the reviews found
I opened this branch believing job timeouts were stranding connections —
`TimeoutExceededException` is this job's most common failure by a wide
margin (66 in 14 days vs 37 `RequestException`). **Both reviews
falsified that independently, and they were right.**
`failed()` has an early return when the connection is already in
`Error`, so it could only ever charge **one** increment per connection
lifetime; a second out-of-band death is a no-op. Three slow cycles
cannot reach the ceiling that way. Prod is the natural experiment: **all
66 timeouts belong to one Wise connection, which sits at
`consecutive_sync_failures = 1`.**
What actually stranded the two rows was #757's pre-fix transient
counting, in the hours before it deployed on 2026-08-10. And the
population is 2, not the 4 I first measured — my raw SQL saw two
soft-deleted rows that `BankingConnection::query()` correctly excludes.
The commits and docblocks now say that. The code change stands on its
own smaller merit: an out-of-band death must not be charged to the
connection.
## The commits
1. **`failed()` no longer spends the retry budget.** Scope stated
honestly in the docblock. It closes exactly one route to the ceiling —
see (2).
2. **Reconnect hands back the full budget.** `AuthorizationController`
was the only one of four "try again" paths that didn't clear the counter
(compare `ConnectionController::sync`, `::update`,
`AccountMappingController`, and the job's success path). A user who
reconnected a connection parked at `MAX + 1` came back `Active` still
carrying the count that parked it, so the first failure re-parked it
immediately — none of the three attempts the ceiling grants, right after
paying an SCA redirect to escape that exact state. **Both reviews found
this while checking commit 1's premise; it is the most real bug here.**
3. **Repair migration, 2 rows.** Matched with `=`, not `>=`:
`handlePermanceError` parks auth failures at `MAX + 1` on purpose and
there are **8 such rows in prod**; a `>=` filter would un-park them, 401
on the next cycle and send each user a **second** "authentication
failed" email. `migrate --pretend` output is in the commit.
4. **Out-of-band deaths are recorded.** Every `logSyncAttempt` call
lived inside `handle()` — exactly what these deaths skip. One prod
connection has 66 job failures and 3 sync-log rows; that gap is why I
mis-attributed the cause. `duration_ms` goes null rather than a fake 0.
Copy fixed too: `failed()` said "An unexpected error occurred… please
try again later", handing our infrastructure to the user, while the
transient path already promised we'd retry.
5. **`uniqueFor` on the job.** `ShouldBeUnique` with no expiry means a
lock lost to a hard kill is never released, and `uniqueId()` is the
connection id — so that connection silently stops syncing for good.
Prevention; prod is clean.
6. **The log had to move above the status guard.** As first written, (4)
logged only when the connection was not already in `Error` — and the
connection it was written for is parked in `Error` and stays there.
Re-measured against prod: **68 failed jobs, 3 sync-log rows**, and every
one of the 65 missing deaths would have hit the guard and written
nothing. Logging now happens as soon as the connection row resolves; the
guards still own the status write, which must not clobber an earlier,
more specific error message.
## Verification
`tests/Feature/OpenBanking`: **346 tests, 346 pass** on a freshly
provisioned worktree (the 10 SSR failures reported earlier were a
local-env artifact, not the suite) (Inertia page-render tests hitting
the SSR `/render` endpoint, which has no local server — I ran the
baseline to confirm). 5 new tests; the two load-bearing ones fail with
the change reverted. `pint` and `dry` green.
One existing assertion changed rather than deleted: `failed sync job
marks active connection as error` asserted the increment. Its declared
subject — the status flip that unblocks onboarding — is untouched.
The migration's target set was re-verified against prod on 2026-08-12:
exactly **2 live rows** at `status=error, consecutive_sync_failures=3`,
and every row at `MAX + 1` is soft-deleted or revoked, so the `=` filter
touches precisely the two intended connections.
**The `crap` job will be red** on `AuthorizationController::callback`
(complexity 16). It is pre-existing: my diff there is one array entry
plus a comment, zero cyclomatic complexity added; `crap --base` only
surfaces it because I touched the file. I deliberately did not add a
`.crap-ignore.json` entry — that would paper over someone else's real
complexity problem. `crap` is not a required check.
## Why this is a draft
The migration writes to production data, and a review caught that a
slightly wider filter would have emailed 8 users a second
authentication-failure notice. That is exactly the class of mistake
worth a human glance. My impact story was also wrong twice this cycle
before the reviews corrected it.
Commits 1, 2, 4 and 5 I'd merge without hesitation — 2 in particular is
a clear standalone bug. Commit 3 is the one that touches prod rows.
## Follow-ups I deliberately did not do
- **The mechanism is now mostly bypassed.** 179 of 208 recent job
failures are exempt from the counter, so nothing bounds either dominant
failure mode and nothing tells the user. The right shape is probably a
backoff timestamp like the existing `rate_limited_until`, plus a "this
connection hasn't synced in N days" email — a design change, not a
patch.
- **`019fac9e`** (Wise, never synced in 14 days): 3 × 120s timeouts plus
3 worker SIGALRM kills per cycle, ~24 min/day of the single `default`
worker. `failOnTimeout = true` would cut that to one kill, but it also
removes two retries that might succeed for a merely slow bank. Its own
bug, its own trade-off.
- **Spanish users may get no Reconnect button.** `hasAuthError()` in
`settings/connections.tsx` matches the English substring
`'Authentication failed'` against a *translated* `error_message`. Needs
a machine-readable reason column to fix properly.
- **An `Error` connection whose consent lapsed is never dispatched**, so
it never reaches `markExpired()` and its user never gets the expiry
email (1 row in prod). Fixing it changes who receives outbound email, so
it wants its own PR.
091457c747