whisper-money/tests/Feature
Víctor Falcón 5baa677c2a
fix(banking): stop requesting a year of Enable Banking history on every sync (#755)
## Why

Enable Banking connections re-requested **a year of transaction history
every 6 hours**. Trade Republic connections failed ~70% of their syncs
with HTTP 429 (220 of 307 attempts in the last 7 days, against 1–4% for
other ASPSPs), and the cause was ours:

1. `EnableBankingSyncer::sync()` persisted the transactions, then the
balance call threw a 429, the exception bubbled up to
`SyncBankingConnectionJob`, and the whole run was marked failed.
2. `last_synced_at` is only written after a clean `sync()`, so it never
got written. 15 of 16 Trade Republic connections have zero rows in
`account_balances` and `last_synced_at = NULL` weeks after connecting.
3. `$isFirstSync = ! $connection->last_synced_at || $this->fullSync` was
therefore permanently true, so every run used `now()->subYear()` with
`strategy = 'longest'` plus `calculateHistoricalBalances()`. Paginating
a year of history four times a day is what trips the rate limit.

This is **not Trade Republic specific**: CaixaBank and Eurocaja Rural
show the same 429 pattern at lower volume, and the fix applies to every
Enable Banking ASPSP.

## What changed

All in `app/Services/Banking/Sync/EnableBankingSyncer.php`.

**1. The fetch window comes from the transaction watermark, not from
`last_synced_at`.**

The `linked` branch already did this; the lookup is now shared by both
branches:

- **Watermark found** → `date_from` = that transaction's date minus a
3-day overlap (banks post transactions with retroactive value dates, so
the previous no-overlap watermark could silently miss them), no
`longest` strategy.
- **No watermark** → unchanged: one year back with `strategy =
'longest'`. That is the genuine first sync.

This is the fuse: even if everything else fails, a routine sync asks for
a few days instead of a year. The two branches' differing balance
handling (`saveDailyBalances: false` for linked accounts,
`calculateHistoricalBalances()` on first sync for unlinked) is
untouched.

**2. A failing balance call no longer fails the whole sync.** It is
logged, counted, and surfaced as `balance_failed` in the array `sync()`
returns, so it lands in `banking_sync_logs.metadata` instead of being
silently swallowed. The run finishes clean → `last_synced_at` gets
written → `$isFirstSync` stops being permanently true →
`calculateHistoricalBalances()` stops running every 6 hours.

**Two exceptions are deliberately still fatal** (both raised by review,
see below): an expired session, and a **429**.

## Deviation from the brief, worth a look

The brief asked for *every* balance failure to be non-fatal, including
the 429. Both review passes flagged the same problem with that: a 429
escapes `EnableBankingProvider` as a raw `RequestException`, and that is
exactly what `SyncBankingConnectionJob::isRateLimitError()` matches to
set `rate_limited_until`. Swallowing it would have removed the only
backoff — and Enable Banking quotas are **per-consent daily access
counts** (`Maximum daily access exceeded`, `Allowed number of accesses
exceeded for consent`), so a connection that lost its backoff would keep
burning the remaining quota on every 6-hourly cycle and never get its
balances.

So a balance 429 is re-thrown and the existing backoff (untouched, as
the brief required) applies. The transactions from that run are still
persisted, and the connection stays `active`. Every other balance
failure is non-fatal as specified.

Three more findings from review, fixed in the second commit:

- **`--full` still forces the year-wide window.** A first sync on a
connection that has already synced can only come from that flag, so it
beats the watermark. Without this, `banking:sync --full` had become a
no-op for the window — the operator's only remedy for a gap in history.
- **Windows reaching back more than 90 days keep `strategy =
'longest'`.** A dormant account with an old watermark would otherwise be
rejected (422) and walked down the `[90, 30, 7]` narrowing ladder, which
advances the watermark past the span it never fetched — a silent,
permanent gap.
- **The watermark counts trashed rows** (the dedup already uses
`withTrashed()`, so re-fetching them creates nothing), and the
future-date clamp no longer eats the 3-day overlap.

## Out of scope

No migration, no new watermark column, no change to the
`rate_limited_until` backoff, no manual reset of the broken production
connections — the first clean sync clears `error_message` on its own.

Two things worth a follow-up, not fixed here:
`calculateHistoricalBalances()` is still gated on the connection-level
`$isFirstSync` while the window is now per-account, so an account added
to an already-synced connection pulls a year of transactions without a
balance backfill; and 61 accounts have never received a bank transaction
at all, so they stay on the year-wide window until one lands.

## QA

Ran the whole chain with nothing mocked but the network (real
`EnableBankingProvider`, `Http::fake`), checking the request that
actually leaves for the bank:

| Case | Request that goes out |
|---|---|
| Account with a watermark at `2026-08-08`, today `2026-08-10` |
`…/transactions?date_from=2026-08-05&date_to=2026-08-10` — 5 days, no
`strategy` |
| Account with no bank transactions |
`…/transactions?date_from=2025-08-10&date_to=2026-08-10&strategy=longest`
— unchanged |
| Balances returns 429 `Maximum daily access exceeded` | transactions
persisted, `status = active`, `rate_limited_until = 2026-08-11 00:00:00`
(next UTC midnight) |

Against the production database (read-only), for the 475 syncable Enable
Banking accounts:

| After the fix | Accounts | Avg. days requested |
|---|---|---|
| Watermark → short window | 382 | 13.7 |
| No watermark → 1 year + `longest` | 61 | 365 |
| Watermark older than 90d → wide + `longest` | 32 | 173 |

All 12 Trade Republic accounts have a watermark, averaging **7.1 days**
— down from 365 on every run.

## Tests

`tests/Feature/OpenBanking/SyncBankingConnectionJobTest.php`:

- an account with existing Enable Banking transactions asks for
`watermark - 3 days`, not a year (the test that matters)
- an account without them still asks for the year with `longest`
- `--full` beats the watermark
- a non-429 balance failure → connection stays `active`,
`last_synced_at` written, the run's transactions persisted,
`balance_failed: 1` in the sync log metadata
- a 429 balance failure → transactions persisted and the backoff still
applied
- an expired session during the balance call is not swallowed

Full suite green (2080 passed), `pint` and `phpstan` clean.
2026-08-10 15:56:45 +02:00
..
Ai feat(stats): post the Discord stats reports in Spanish, opened by an AI summary (#752) 2026-08-10 10:13:40 +02:00
Api feat(transactions): serve import dedup and account ledger from the backend (#631) 2026-07-03 16:49:59 +02:00
Auth fix(auth): make build deterministic when REGISTRATION_ENABLED=false (#720) 2026-07-22 07:04:20 +00:00
Console fix(demo): stop demo:reset from colliding on the fake Stripe subscription id (#756) 2026-08-10 12:48:13 +00:00
Events fix: stop double-dispatching transaction listeners (N+1 insert into jobs) (#620) 2026-07-02 13:26:45 +00:00
Jobs feat(drip): email users stuck on the paywall a day after onboarding (#562) 2026-06-19 14:11:12 +00:00
Listeners feat(budgets): track multiple categories and labels per budget (#466) 2026-06-01 12:32:23 +02:00
Mcp feat(transactions): allow manual transactions on bank-connected accounts (#747) 2026-08-10 05:32:40 +02:00
Onboarding fix(onboarding): don't trap users on the syncing step when a bank sync fails (#745) 2026-08-09 18:40:49 +02:00
OpenBanking fix(banking): stop requesting a year of Enable Banking history on every sync (#755) 2026-08-10 15:56:45 +02:00
Services refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
Settings fix(budgets): keep per-transaction email notifications opt-in (#735) 2026-07-25 12:36:13 +02:00
Spaces feat(spaces): phase 0 — multi-tenant Space foundation (no behaviour change) (#650) 2026-07-09 14:26:07 +02:00
Sync refactor: Simplify transaction endpoints architecture (#76) 2026-01-25 16:15:17 +01:00
AccountBalanceControllerTest.php fix(balances): allow saving a zero balance (#664) 2026-07-10 15:02:38 +02:00
AccountControllerTest.php refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
AccountImportConfigTest.php feat(import): persist per-account import configuration on the backend (#698) 2026-07-18 16:10:40 +02:00
AccountUserCurrencyServiceTest.php fix(open-banking): stop storing the XXX no-currency placeholder on accounts (#602) 2026-06-27 16:01:21 +00:00
AiConsentSettingsTest.php refactor(ai): remove AiConsentSettings feature flag (#619) 2026-07-01 09:47:55 +02:00
AiConsentTest.php feat(ai): dismissable AI consent banner that stops after the first decision (#617) 2026-07-01 07:26:36 +00:00
AlignAccountsEncryptedFlagMigrationTest.php refactor(encryption): strip client-side transaction encryption (#514) 2026-06-20 16:13:26 +00:00
ApplyRealEstateRevaluationTest.php fix(real-estate): compound annual revaluation monthly (#337) 2026-04-27 07:35:51 +01:00
AuthenticatedLayoutSafeAreaTest.php fix(layout): keep bottom padding while floating nav is visible (#537) 2026-06-15 18:23:26 +02:00
AutomationRuleApplicationTest.php fix(security): scope job-status endpoints to owner + feature-area fixes (#627) 2026-07-03 14:49:32 +02:00
AutomationRuleEvaluationTest.php feat(transactions): add counterparty fields (#440) 2026-05-27 16:20:55 +02:00
AutomationRuleTest.php fix(ai): stop a learned rule title from 500-ing the user's category change (#741) 2026-08-09 15:39:14 +00:00
BackfillAccountIbansCommandTest.php chore: upgrade Laravel 12 to 13 (#242) 2026-03-25 12:56:33 +00:00
BackfillXxxAccountCurrenciesTest.php fix(open-banking): stop storing the XXX no-currency placeholder on accounts (#602) 2026-06-27 16:01:21 +00:00
BalanceLookupTest.php feat: investment benefits — show gains/losses on investment accounts (#140) 2026-02-23 13:59:10 +01:00
BudgetHistoricalAssignmentTest.php feat(budgets): track multiple categories and labels per budget (#466) 2026-06-01 12:32:23 +02:00
BudgetNotificationTest.php feat(budgets): enable email notifications by default and coalesce alerts per transaction (#733) 2026-07-24 15:33:14 +02:00
BudgetPeriodDateTest.php Remove budgets feature flag (#108) 2026-02-12 09:58:01 +01:00
BudgetPeriodServiceTest.php fix(budgets): make period generation idempotent (#533) 2026-06-15 12:44:44 +02:00
BudgetTest.php feat: add catch-all budgets (#527) 2026-06-15 16:07:19 +00:00
BudgetTransactionServiceTest.php feat: parent/child category tree (#474) 2026-06-03 19:30:12 +02:00
BulkUpdateTransactionsTest.php refactor: Simplify transaction endpoints architecture (#76) 2026-01-25 16:15:17 +01:00
CancelFreeEnableBankingConnectionsCommandTest.php Cancel Enable Banking connections for free users (#289) 2026-04-15 16:23:03 +02:00
CashflowAnalyticsTest.php fix(cashflow): bound trend window to prevent request timeout (#534) 2026-06-15 12:47:27 +02:00
CashflowPageTest.php refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
CatchAllBudgetTest.php feat: add catch-all budgets (#527) 2026-06-15 16:07:19 +00:00
CategoryMonthlyBreakdownTest.php feat(analysis): per-category 12-month spending drawer (#519) 2026-06-11 09:52:53 +02:00
CurrencyConversionServiceTest.php fix(currency): make rate fetching resilient to slow CDN (#502) 2026-06-08 09:10:38 +02:00
DashboardAnalyticsTest.php fix(accounts): show credit cards as positive and exclude them from net worth (#673) 2026-07-13 07:27:52 +00:00
DashboardTest.php fix(auth): make build deterministic when REGISTRATION_ENABLED=false (#720) 2026-07-22 07:04:20 +00:00
DecryptTransactionsTest.php refactor(encryption): strip client-side transaction encryption (#514) 2026-06-20 16:13:26 +00:00
DeleteUserCommandTest.php Support soft-deleted users with reusable emails (#316) 2026-04-22 11:41:41 +01:00
DemoAccountRestrictionsTest.php fix(demo): stop demo:reset from colliding on the fake Stripe subscription id (#756) 2026-08-10 12:48:13 +00:00
DisconnectBankingConnectionsCommandTest.php feat(banking): add command to disconnect connections by id (#497) 2026-06-06 11:16:01 +02:00
DiscordReportEmbedLimitsTest.php feat(stats): post the Discord stats reports in Spanish, opened by an AI summary (#752) 2026-08-10 10:13:40 +02:00
DiscordWebhookTest.php feat: add Discord admin feed for daily stats and Stripe events (#458) 2026-05-30 18:14:46 +02:00
EncryptionTest.php refactor(encryption): strip client-side transaction encryption (#514) 2026-06-20 16:13:26 +00:00
ExampleTest.php Install Pest 2025-11-07 12:01:58 +00:00
ExchangeRateServiceTest.php Fix PHP-LARAVEL-1V exchange rate cache race (#383) 2026-05-12 12:45:45 +02:00
GenerateStripePromotionCodesCommandTest.php feat(stripe): add promo code generator (#311) 2026-04-20 18:15:28 +01:00
IdorVulnerabilityTest.php refactor: Simplify transaction endpoints architecture (#76) 2026-01-25 16:15:17 +01:00
ImportDataTest.php fix: Apply automation rule labels on transaction creation and import (#79) 2026-01-27 11:11:29 +01:00
InertiaSharedDataTest.php feat(mcp): read-only MCP server for Pro accounts (#689) 2026-07-17 16:54:15 +02:00
IntegrationRequestTest.php feat(integration-requests): add done status and fix review command crash on orphaned author (#601) 2026-06-27 14:42:09 +00:00
LabelTest.php refactor(api): standardize serialization via model $hidden (#492) 2026-06-05 13:57:34 +02:00
LoanTest.php chore: harden Inertia boundary, CI type-check, and test isolation (#640) 2026-07-04 18:57:58 +00:00
LocalizationTest.php feat(currencies): add the Danish Krone (DKK) (#754) 2026-08-10 12:08:56 +02:00
LoggingConfigTest.php fix(logging): keep laravel.log writable across container UIDs (#451) 2026-05-29 15:10:50 +02:00
MailSenderTest.php refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
NewTransactionsMarkerTest.php feat(transactions): make new-transaction marker cross-device (#611) 2026-06-29 19:11:37 +02:00
OpenAiAppsChallengeTest.php feat(mcp): serve the ChatGPT app directory domain challenge (#749) 2026-08-10 07:46:49 +00:00
PersistUpsellSourceFromStripeTest.php feat: reuse the upgrade modal at more upsell points and attribute revenue (#699) 2026-07-18 12:53:20 +00:00
PlaintextTransactionsTest.php Remove plaintext-transactions feature flag & E2E references (#116) 2026-02-13 11:10:21 +01:00
PlanFeatureTest.php feat(ai): suggest automation rules during onboarding (#523) 2026-06-13 22:51:15 +02:00
PopoverSafeAreaTest.php fix: keep iOS popovers below the notch (#282) 2026-04-13 15:19:56 +01:00
PostStripeEventToDiscordTest.php fix(discord): show old → new plan on plan change notification (#637) 2026-07-04 15:49:09 +00:00
PurgeResidualEncryptionArtifactsJobTest.php chore: harden Inertia boundary, CI type-check, and test isolation (#640) 2026-07-04 18:57:58 +00:00
PwaTest.php Harden browser storage and PostHog recording (#402) 2026-05-14 15:57:48 +02:00
QueueConfigTest.php fix(queue): raise retry_after above the longest job timeout (PHP-LARAVEL-2D) (#645) 2026-07-05 09:31:23 +00:00
ReEvaluateTransactionRulesTest.php fix(security): scope job-status endpoints to owner + feature-area fixes (#627) 2026-07-03 14:49:32 +02:00
RealEstateAvailabilityTest.php refactor(real-estate): remove Pennant gating (#308) 2026-04-20 13:31:49 +01:00
RealEstateTest.php fix(balances): stop historical-balance generator OOM on ancient purchase/loan dates (PHP-LARAVEL-49) (#661) 2026-07-08 21:34:43 +00:00
ResendSyncCommandTest.php feat: Sync new users to Resend contacts (#85) 2026-01-28 21:25:58 +01:00
RouteNotificationForMailTest.php refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
RuleEngineParityTest.php refactor: consolidate duplicated financial calculations (#643) 2026-07-04 22:26:44 +02:00
SavedFilterTest.php feat(transactions): add a monthly trend view to the analysis drawer (#736) 2026-07-26 17:03:52 +02:00
SelfServeRefundTest.php fix(demo): stop demo:reset from colliding on the fake Stripe subscription id (#756) 2026-08-10 12:48:13 +00:00
SendAiCohortReportCommandTest.php feat(stats): post the Discord stats reports in Spanish, opened by an AI summary (#752) 2026-08-10 10:13:40 +02:00
SendAiConsentFollowUpEmailsCommandTest.php test(drip): deflake AI consent onboarding-grace boundary test (#625) 2026-07-03 07:06:20 +00:00
SendDailyStatsReportCommandTest.php feat(stats): post the Discord stats reports in Spanish, opened by an AI summary (#752) 2026-08-10 10:13:40 +02:00
SendExperimentFunnelReportCommandTest.php feat(stats): post the Discord stats reports in Spanish, opened by an AI summary (#752) 2026-08-10 10:13:40 +02:00
SendStuckCohortReportCommandTest.php feat(stats): add --no-discord to the remaining report commands (#607) 2026-06-29 15:32:31 +02:00
SendSubscriptionFunnelReportCommandTest.php feat(stats): post the Discord stats reports in Spanish, opened by an AI summary (#752) 2026-08-10 10:13:40 +02:00
SentryConfigTest.php ci: remove production deploy job (#574) 2026-06-20 18:17:14 +00:00
SentryUserMiddlewareTest.php Add Sentry user context (#348) 2026-05-04 13:26:50 +01:00
SetLocaleTest.php feat(i18n): add French translation support (#532) 2026-06-15 19:15:43 +02:00
SitemapTest.php User Onboarding Flow (#23) 2025-12-12 13:06:08 +01:00
StrayHttpRequestGuardTest.php chore: harden Inertia boundary, CI type-check, and test isolation (#640) 2026-07-04 18:57:58 +00:00
StripeSubscriptionStatsCommandTest.php feat: add Discord admin feed for daily stats and Stripe events (#458) 2026-05-30 18:14:46 +02:00
SubscriptionExperimentTest.php feat(subscriptions): trial/pricing A/B/C experiment (#600) 2026-06-27 18:00:15 +02:00
SubscriptionTest.php refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
SuggestionPersistenceTest.php feat(ai): suggest automation rules during onboarding (#523) 2026-06-13 22:51:15 +02:00
SyncBankingConnectionsCommandTest.php chore: Remove account-mapping feature flag (#252) 2026-04-01 12:09:22 +02:00
SyncStripePricesCommandTest.php fix(pricing): update final release prices (#288) 2026-04-15 14:49:02 +01:00
TrackLastActiveAtTest.php feat(users): track last login and last active timestamps (#516) 2026-06-10 11:01:30 +02:00
TransactionAnalysisTest.php feat(transactions): add a monthly trend view to the analysis drawer (#736) 2026-07-26 17:03:52 +02:00
TransactionFilterTest.php fix(transactions): prevent crash when sorting by nullable column (#501) 2026-06-06 17:23:21 +02:00
TransactionSideClassificationTest.php refactor: consolidate duplicated financial calculations (#643) 2026-07-04 22:26:44 +02:00
TransactionTest.php refactor: remove HIDE_AUTH_BUTTONS launch gate and waitlist apparatus (#717) 2026-07-22 08:51:48 +02:00
WelcomeBanksOrderingTest.php feat(open-banking): remove feature flag gating (#297) 2026-04-17 10:20:05 +02:00