whisper-money/app
Víctor Falcón 08d367a5c8
fix(balances): stop historical-balance generator OOM on ancient purchase/loan dates (PHP-LARAVEL-49) (#661)
## What & why

A queued historical-balance generator job exhausted the worker's 128MB
PHP memory limit and **died on every retry** — Sentry `PHP-LARAVEL-49` +
`PHP-LARAVEL-4A` (two fingerprints of the *same* poison message:
identical `trace_id`/`message.id`, `retry.count` 1 then 2, OOM at
slightly different `Arr.php` lines).

### Root cause (trace-confirmed)
The Sentry trace localized the OOMing message to
`App\Jobs\GenerateHistoricalRealEstateBalancesJob`, dispatched from
`Settings\AccountController::store` for the "older than 12 months" slice
of history. The fatal frame is `Arr::map` inside Eloquent's
`AccountBalance::upsert()` value-prep (`addUniqueIds`/`addTimestamps`
per-row `array_merge`, `Query\Builder::upsert` `Arr::flatten` bindings,
and a compiled multi-row SQL string — all **O(n)** over the rows).

`purchase_date` (`before_or_equal:today`) and `loan_start_date` (no
constraint) had **no lower bound**, so a mistyped/ancient year (e.g.
`0201`, `1080`) made the generator build a **multi-century monthly
series** and hand it all to a single `upsert`, blowing past 128MB.

## The fix (two commits)

1. **`fix(balances): upsert historical balances in batches`** — build +
upsert in batches of 500 (`UPSERT_CHUNK_SIZE`) in both
`RealEstateBalanceGeneratorService` and `LoanBalanceGeneratorService`,
instead of one giant operation. Same rows, same `(account_id,
balance_date)` conflict target — peak memory is now bounded regardless
of series length. Defense-in-depth.
2. **`fix(accounts): floor purchase/loan start dates at 1900`** — the
actual root cause: add `after_or_equal:1900-01-01` to `purchase_date`
and `loan_start_date`. Rejects the data-entry typos that drive the
runaway series (and the resulting DB bloat / misleading multi-century
net-worth ramp) while accepting any legitimately old asset in a
personal-finance app.

## Review (two agents, before this PR)
Both reviewed against the live trace and the code.

- **Root cause & fix confirmed** — the batching caps exactly the
`Arr::map`-over-N-rows frame from the trace; no independent OOM source
exists (no per-row model events; `upsert` bypasses events).
- **Chunking is correct** — the unique index `(account_id,
balance_date)` exists, so the conflict target is real; `buildDateList`
yields a strictly-increasing de-duplicated list, so batches are disjoint
(no dropped/duplicated dates, no off-by-one). The fresh per-row `id`
UUID is discarded on conflict → idempotent retries.
- **No material user-facing regression.** Balance consumption
(dashboard/net-worth) is date-range-clamped. Batching introduces a
theoretical partial-write-on-mid-job-failure window, but it's idempotent
and self-heals on retry — agents advised **against** wrapping it in a
transaction (holds locks, doesn't help memory).

Applied recommendation #2 (the date floor) as its own commit. Deferred
(both agents agree): extracting the two near-identical generators into a
shared base/trait — a larger refactor that shouldn't ride a hotfix.

## Tests
- Regression test in each service: generate a ~46-year (>500-point)
series and assert the batched writes produce no dropped/duplicated dates
and keep endpoints anchored (crosses multiple batches).
- Validation test: a pre-1900 `purchase_date` is rejected at
`accounts.store`.

Note: the PHP feature suite boots a MySQL testcontainer (Docker) that
isn't available in my environment, so these were validated by
static/`pint`/`php -l` locally and rely on CI for execution — auto-merge
is gated on green CI.

Fixes PHP-LARAVEL-49
Fixes PHP-LARAVEL-4A
2026-07-08 21:34:43 +00:00
..
Actions fix(security): scope job-status endpoints to owner + feature-area fixes (#627) 2026-07-03 14:49:32 +02:00
Ai/Agents feat(ai): auto-categorize transactions with AI (behind flag) (#535) 2026-06-15 16:35:20 +02:00
Console/Commands fix(encryption): never email or delete users who are still being billed (#656) 2026-07-07 16:04:27 +00:00
Contracts feat(connections): manage which accounts a bank connection syncs (#558) 2026-06-18 16:22:49 +02:00
Enums feat(transactions): serve import dedup and account ledger from the backend (#631) 2026-07-03 16:49:59 +02:00
Events Add Budgeting Feature to Track and Manage Spending (#36) 2026-01-21 15:25:50 +01:00
Exceptions/Banking fix(banking): recover from EnableBanking 422 wrong-period instead of crashing the sync (PHP-LARAVEL-42) (#653) 2026-07-07 08:57:23 +02:00
Features refactor(ai): remove AiConsentSettings feature flag (#619) 2026-07-01 09:47:55 +02:00
Http fix(balances): stop historical-balance generator OOM on ancient purchase/loan dates (PHP-LARAVEL-49) (#661) 2026-07-08 21:34:43 +00:00
Jobs fix(queue): raise retry_after above the longest job timeout (PHP-LARAVEL-2D) (#645) 2026-07-05 09:31:23 +00:00
Listeners fix(discord): show old → new plan on plan change notification (#637) 2026-07-04 15:49:09 +00:00
Mail refactor(drip): extract base mailable and job for the drip email family (#641) 2026-07-04 20:51:38 +02:00
Models refactor: consolidate duplicated financial calculations (#643) 2026-07-04 22:26:44 +02:00
Notifications fix: verify email via signed link without requiring login (#490) 2026-06-05 10:01:32 +02:00
Policies refactor(policies): extract HandlesUserOwnership trait (#478) 2026-06-03 17:43:30 +02:00
Providers fix: stop double-dispatching transaction listeners (N+1 insert into jobs) (#620) 2026-07-02 13:26:45 +00:00
Rules feat: parent/child category tree (#474) 2026-06-03 19:30:12 +02:00
Services fix(balances): stop historical-balance generator OOM on ancient purchase/loan dates (PHP-LARAVEL-49) (#661) 2026-07-08 21:34:43 +00:00