whisper-money/tests/Feature/OpenBanking
Víctor Falcón 5baa677c2a
fix(banking): stop requesting a year of Enable Banking history on every sync (#755)
## Why

Enable Banking connections re-requested **a year of transaction history
every 6 hours**. Trade Republic connections failed ~70% of their syncs
with HTTP 429 (220 of 307 attempts in the last 7 days, against 1–4% for
other ASPSPs), and the cause was ours:

1. `EnableBankingSyncer::sync()` persisted the transactions, then the
balance call threw a 429, the exception bubbled up to
`SyncBankingConnectionJob`, and the whole run was marked failed.
2. `last_synced_at` is only written after a clean `sync()`, so it never
got written. 15 of 16 Trade Republic connections have zero rows in
`account_balances` and `last_synced_at = NULL` weeks after connecting.
3. `$isFirstSync = ! $connection->last_synced_at || $this->fullSync` was
therefore permanently true, so every run used `now()->subYear()` with
`strategy = 'longest'` plus `calculateHistoricalBalances()`. Paginating
a year of history four times a day is what trips the rate limit.

This is **not Trade Republic specific**: CaixaBank and Eurocaja Rural
show the same 429 pattern at lower volume, and the fix applies to every
Enable Banking ASPSP.

## What changed

All in `app/Services/Banking/Sync/EnableBankingSyncer.php`.

**1. The fetch window comes from the transaction watermark, not from
`last_synced_at`.**

The `linked` branch already did this; the lookup is now shared by both
branches:

- **Watermark found** → `date_from` = that transaction's date minus a
3-day overlap (banks post transactions with retroactive value dates, so
the previous no-overlap watermark could silently miss them), no
`longest` strategy.
- **No watermark** → unchanged: one year back with `strategy =
'longest'`. That is the genuine first sync.

This is the fuse: even if everything else fails, a routine sync asks for
a few days instead of a year. The two branches' differing balance
handling (`saveDailyBalances: false` for linked accounts,
`calculateHistoricalBalances()` on first sync for unlinked) is
untouched.

**2. A failing balance call no longer fails the whole sync.** It is
logged, counted, and surfaced as `balance_failed` in the array `sync()`
returns, so it lands in `banking_sync_logs.metadata` instead of being
silently swallowed. The run finishes clean → `last_synced_at` gets
written → `$isFirstSync` stops being permanently true →
`calculateHistoricalBalances()` stops running every 6 hours.

**Two exceptions are deliberately still fatal** (both raised by review,
see below): an expired session, and a **429**.

## Deviation from the brief, worth a look

The brief asked for *every* balance failure to be non-fatal, including
the 429. Both review passes flagged the same problem with that: a 429
escapes `EnableBankingProvider` as a raw `RequestException`, and that is
exactly what `SyncBankingConnectionJob::isRateLimitError()` matches to
set `rate_limited_until`. Swallowing it would have removed the only
backoff — and Enable Banking quotas are **per-consent daily access
counts** (`Maximum daily access exceeded`, `Allowed number of accesses
exceeded for consent`), so a connection that lost its backoff would keep
burning the remaining quota on every 6-hourly cycle and never get its
balances.

So a balance 429 is re-thrown and the existing backoff (untouched, as
the brief required) applies. The transactions from that run are still
persisted, and the connection stays `active`. Every other balance
failure is non-fatal as specified.

Three more findings from review, fixed in the second commit:

- **`--full` still forces the year-wide window.** A first sync on a
connection that has already synced can only come from that flag, so it
beats the watermark. Without this, `banking:sync --full` had become a
no-op for the window — the operator's only remedy for a gap in history.
- **Windows reaching back more than 90 days keep `strategy =
'longest'`.** A dormant account with an old watermark would otherwise be
rejected (422) and walked down the `[90, 30, 7]` narrowing ladder, which
advances the watermark past the span it never fetched — a silent,
permanent gap.
- **The watermark counts trashed rows** (the dedup already uses
`withTrashed()`, so re-fetching them creates nothing), and the
future-date clamp no longer eats the 3-day overlap.

## Out of scope

No migration, no new watermark column, no change to the
`rate_limited_until` backoff, no manual reset of the broken production
connections — the first clean sync clears `error_message` on its own.

Two things worth a follow-up, not fixed here:
`calculateHistoricalBalances()` is still gated on the connection-level
`$isFirstSync` while the window is now per-account, so an account added
to an already-synced connection pulls a year of transactions without a
balance backfill; and 61 accounts have never received a bank transaction
at all, so they stay on the year-wide window until one lands.

## QA

Ran the whole chain with nothing mocked but the network (real
`EnableBankingProvider`, `Http::fake`), checking the request that
actually leaves for the bank:

| Case | Request that goes out |
|---|---|
| Account with a watermark at `2026-08-08`, today `2026-08-10` |
`…/transactions?date_from=2026-08-05&date_to=2026-08-10` — 5 days, no
`strategy` |
| Account with no bank transactions |
`…/transactions?date_from=2025-08-10&date_to=2026-08-10&strategy=longest`
— unchanged |
| Balances returns 429 `Maximum daily access exceeded` | transactions
persisted, `status = active`, `rate_limited_until = 2026-08-11 00:00:00`
(next UTC midnight) |

Against the production database (read-only), for the 475 syncable Enable
Banking accounts:

| After the fix | Accounts | Avg. days requested |
|---|---|---|
| Watermark → short window | 382 | 13.7 |
| No watermark → 1 year + `longest` | 61 | 365 |
| Watermark older than 90d → wide + `longest` | 32 | 173 |

All 12 Trade Republic accounts have a watermark, averaging **7.1 days**
— down from 365 on every run.

## Tests

`tests/Feature/OpenBanking/SyncBankingConnectionJobTest.php`:

- an account with existing Enable Banking transactions asks for
`watermark - 3 days`, not a year (the test that matters)
- an account without them still asks for the year with `longest`
- `--full` beats the watermark
- a non-429 balance failure → connection stays `active`,
`last_synced_at` written, the run's transactions persisted,
`balance_failed: 1` in the sync log metadata
- a 429 balance failure → transactions persisted and the backoff still
applied
- an expired session during the balance call is not swallowed

Full suite green (2080 passed), `pint` and `phpstan` clean.
2026-08-10 15:56:45 +02:00
..
AccountMappingTest.php fix(open-banking): unblock account mapping when the bank reports accounts without a uid (#746) 2026-08-09 18:41:35 +02:00
AuthorizationControllerTest.php feat(open-banking): finalize bank connection without a session via state token (#498) 2026-06-06 12:12:07 +02:00
BalanceSyncServiceTest.php Batch historical balance writes (#356) 2026-05-05 15:26:28 +01:00
BankingConnectionSyncerFactoryTest.php feat(banking): add Interactive Brokers sync via Flex Web Service (#581) 2026-06-23 11:39:24 +02:00
BinanceBalanceSyncTest.php feat(settings): centralize currency options and split profile/account support (#256) 2026-04-02 19:23:10 +02:00
BinanceControllerTest.php feat(open-banking): remove feature flag gating (#297) 2026-04-17 10:20:05 +02:00
BitpandaBalanceSyncTest.php feat(settings): centralize currency options and split profile/account support (#256) 2026-04-02 19:23:10 +02:00
BitpandaControllerTest.php feat(open-banking): remove feature flag gating (#297) 2026-04-17 10:20:05 +02:00
CoinbaseBalanceSyncTest.php Backfill Coinbase monthly history (#395) 2026-05-14 10:52:46 +01:00
CoinbaseControllerTest.php feat: Coinbase banking integration (#388) 2026-05-13 19:53:30 +02:00
ConnectionAccountTest.php feat(open-banking): enable manage bank accounts for everyone (#572) 2026-06-20 17:35:43 +00:00
ConnectionControllerTest.php feat(banking): let Wise credentials be updated (#588) 2026-06-23 09:54:31 +00:00
DailyEmailTransactionsIndexTest.php perf(db): index transactions for the daily synced-email slow query (PHP-LARAVEL-3X) (#622) 2026-07-02 15:51:15 +02:00
EnableBankingProviderTest.php fix(banking): stop unclassified bank responses from silently killing a connection (#742) 2026-08-09 17:26:34 +02:00
IndexaCapitalBalanceSyncTest.php fix(banking): treat Indexa Capital performance 404 as empty data (#386) 2026-05-13 09:34:28 +01:00
IndexaCapitalControllerTest.php feat(open-banking): remove feature flag gating (#297) 2026-04-17 10:20:05 +02:00
InstitutionControllerTest.php feat(open-banking): remove feature flag gating (#297) 2026-04-17 10:20:05 +02:00
InteractiveBrokersBalanceSyncTest.php feat(banking): add Interactive Brokers sync via Flex Web Service (#581) 2026-06-23 11:39:24 +02:00
InteractiveBrokersControllerTest.php refactor(open-banking): extract shared connect-controller flow into a base class (#639) 2026-07-04 20:24:54 +02:00
OpenBankingFeatureFlagTest.php refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
SyncBankingConnectionJobTest.php fix(banking): stop requesting a year of Enable Banking history on every sync (#755) 2026-08-10 15:56:45 +02:00
SyncBinanceHistoricalBalancesJobTest.php chore: upgrade Laravel 12 to 13 (#242) 2026-03-25 12:56:33 +00:00
SyncRetryAndLoggingTest.php fix(banking): recover from EnableBanking 422 wrong-period instead of crashing the sync (PHP-LARAVEL-42) (#653) 2026-07-07 08:57:23 +02:00
TransactionSyncServiceTest.php feat(transactions): allow manual transactions on bank-connected accounts (#747) 2026-08-10 05:32:40 +02:00
WiseBalanceSyncTest.php feat: add Wise open banking integration with balance sync (#525) 2026-06-16 13:23:25 +00:00
WiseControllerTest.php feat: add Wise open banking integration with balance sync (#525) 2026-06-16 13:23:25 +00:00