RomanNum3ral
/
whisper-money
mirror of https://github.com/whisper-money/whisper-money.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
whisper-money/database/migrations
History
Víctor Falcón ad46e465be
perf(db): index transactions for the daily synced-email slow query (PHP-LARAVEL-3X) (#622) 
## What

Addresses Sentry **PHP-LARAVEL-3X** — a slow DB query in
`App\Jobs\SendDailyBankTransactionsSyncedEmailJob::handle()`.

The daily "transactions synced" email filters transactions by:

```php
Transaction::query()
    ->where('user_id', $this->user->id)
    ->where('source', TransactionSource::EnableBanking)
    ->when($lastSentMailLog?->sent_at, fn ($q, $at) => $q->where('created_at', '>', $at))
    ->whereHas('account.bankingConnection', ...) // EXISTS on PKs
    ->get();
```

The only `user_id`-leading index is `idx_transactions_budget_lookup
(user_id, transaction_date, category_id)` — its second column is
`transaction_date`, **not** `created_at`, so it can't serve
`source`/`created_at`.

## How

Add a composite index matching the predicate — two equalities then the
range:

```
idx_transactions_user_source_created (user_id, source, created_at)
```

The `whereHas` compiles to correlated `EXISTS` subqueries that join on
primary keys (`accounts.id`, `banking_connections.id`), so no extra
index on those tables is needed — the `transactions` index alone gives
the optimizer the selective driving path it currently lacks.

## Production verification (via read-only prod queries)

Confirmed the diagnosis and de-risked the deploy against the live
database:

- **The query is genuinely slow.** `EXPLAIN ANALYZE` for the heaviest
user (~6.1k EnableBanking transactions) runs in **~6 s**. The optimizer,
lacking a selective path, drives from a **full table scan of all ~2,500
accounts** and examines ~108k transaction rows (334 account loops × ~323
rows), filtering `user_id`/`source` only afterwards.
- **The index fixes it.** `(user_id, source, created_at)` lets the
optimizer drive from transactions — a seek to `(user_id,
'enablebanking')` (~6.1k rows) plus primary-key semi-joins — far cheaper
than the current ~108k-row plan, so it will be chosen.
- **The deploy is low-risk.** `transactions` holds **~297k rows** (not
millions). Adding a secondary index on InnoDB/MySQL 8 is online
(`ALGORITHM=INPLACE, LOCK=NONE`, no table rebuild), so at this size the
build is seconds and does not block the sync write path.

## Commits

1. `perf(db): index transactions for the daily synced-email query` — the
migration + a test asserting the index columns/order.
2. `test(db): assert the daily-email index name, not just its columns` —
lock the explicit index name (review feedback).

## Reviewed by two independent agents (architecture +
product/operational risk)

Both rated the change correct and shippable: column order right
(equalities before the range), migration reversible, index non-redundant
(an existing index can't be widened without breaking budget queries),
and no behavior/result change (a secondary btree index never alters
result sets; the job has no `ORDER BY`).

## Impact / risk

- **User impact:** indirect — a background email job (Sentry reports 0
interactive users), but the ~6 s query wastes queue-worker time and IO
on every run.
- **Complexity:** low — one additive, reversible index migration; no
application logic changed.
- **Write path:** one more index maintained per transaction insert (10th
on the table). Marginal, consistent with the existing UUID-leading index
cost profile.

Fixes PHP-LARAVEL-3X
 		2026-07-02 15:51:15 +02:00
..
0001_01_01_000000_create_users_table.php Set up a fresh Laravel app 2025-11-07 12:01:36 +00:00
0001_01_01_000001_create_cache_table.php Set up a fresh Laravel app 2025-11-07 12:01:36 +00:00
0001_01_01_000002_create_jobs_table.php Set up a fresh Laravel app 2025-11-07 12:01:36 +00:00
2025_08_26_100418_add_two_factor_columns_to_users_table.php Set up a fresh Laravel app 2025-11-07 12:01:36 +00:00
2025_11_07_135255_add_encryption_salt_to_users_table.php E2E Encryption 2025-11-07 14:21:25 +00:00
2025_11_07_135256_create_encrypted_messages_table.php E2E Encryption 2025-11-07 14:21:25 +00:00
2025_11_07_150038_create_banks_table.php feat: Add financial models and seeders 2025-11-07 15:45:28 +00:00
2025_11_07_150122_create_accounts_table.php feat: Add financial models and seeders 2025-11-07 15:45:28 +00:00
2025_11_07_150613_create_categories_table.php feat: Add financial models and seeders 2025-11-07 15:45:28 +00:00
2025_11_07_150659_create_transactions_table.php feat: Add financial models and seeders 2025-11-07 15:45:28 +00:00
2025_11_07_184056_modify_banks_table_add_user_id_and_simplify_name.php Banks accounts 2025-11-07 19:19:29 +00:00
2025_11_07_185018_change_banks_logo_to_text.php Banks accounts 2025-11-07 19:19:29 +00:00
2025_11_08_140230_change_transactions_id_to_uuid.php Update transaction amount to bigint 2025-11-15 20:53:47 +01:00
2025_11_08_141344_make_category_id_nullable_in_transactions_table.php Sync transactions 2025-11-08 14:17:16 +00:00
2025_11_08_144530_add_soft_deletes_to_transactions_table.php Transactions page 2025-11-08 23:38:35 +00:00
2025_11_10_110100_create_automation_rules_table.php Automation rules 2025-11-10 12:08:58 +00:00
2025_11_15_172640_create_account_balances_table.php Add account balances 2025-11-15 20:27:18 +01:00
2025_11_15_175636_change_transactions_amount_to_bigint.php Update transaction amount to bigint 2025-11-15 20:53:47 +01:00
2025_11_15_195739_convert_all_ids_to_uuid.php fix: migration history 2025-12-30 07:22:19 +01:00
2025_11_18_184409_create_user_leads_table.php Merge pull request #1 from whisper-money/landing-page 2025-11-24 12:11:35 +01:00
2025_11_28_104227_add_source_to_transactions_table.php Save transaction source (manually_created, or imported) 2025-11-28 12:36:48 +01:00
2025_11_29_170353_add_type_to_categories_table.php Add category type field support (#2) 2025-12-01 20:19:47 +01:00
2025_11_29_170955_fix_account_balances_unique_constraint.php fix: migration history 2025-12-30 07:22:19 +01:00
2025_12_01_101443_add_unique_constraint_to_categories_table.php Add unique constraint to categories table and update factory data 2025-12-01 11:19:16 +01:00
2025_12_06_112515_create_customer_columns.php Subscriptions (#15) 2025-12-06 19:09:56 +01:00
2025_12_06_112516_create_subscriptions_table.php Subscriptions (#15) 2025-12-06 19:09:56 +01:00
2025_12_06_112517_create_subscription_items_table.php Subscriptions (#15) 2025-12-06 19:09:56 +01:00
2025_12_06_112518_add_meter_id_to_subscription_items_table.php Subscriptions (#15) 2025-12-06 19:09:56 +01:00
2025_12_06_112519_add_meter_event_name_to_subscription_items_table.php Subscriptions (#15) 2025-12-06 19:09:56 +01:00
2025_12_10_142006_add_onboarded_at_to_users_table.php User Onboarding Flow (#23) 2025-12-12 13:06:08 +01:00
2025_12_12_092647_create_labels_table.php feat: add transaction labels feature (#24) 2025-12-13 13:02:19 +01:00
2025_12_12_092650_create_label_transaction_table.php feat: add transaction labels feature (#24) 2025-12-13 13:02:19 +01:00
2025_12_12_092651_create_automation_rule_labels_table.php feat: add transaction labels feature (#24) 2025-12-13 13:02:19 +01:00
2025_12_12_141955_update_labels_unique_constraint_include_deleted_at.php feat: add transaction labels feature (#24) 2025-12-13 13:02:19 +01:00
2025_12_16_151952_create_user_mail_logs_table.php feat: Implement drip email campaign system (#35) 2025-12-30 07:22:18 +01:00
2025_12_19_092437_create_budgets_table.php Add Budgeting Feature to Track and Manage Spending (#36) 2026-01-21 15:25:50 +01:00
2025_12_19_092443_create_budget_periods_table.php Add Budgeting Feature to Track and Manage Spending (#36) 2026-01-21 15:25:50 +01:00
2025_12_19_092446_create_budget_transactions_table.php Add Budgeting Feature to Track and Manage Spending (#36) 2026-01-21 15:25:50 +01:00
2025_12_20_154221_create_features_table.php Add Cashflow Analytics Feature (#49) 2026-01-05 13:06:50 +01:00
2025_12_29_063338_add_currency_code_to_users_table.php Add Budgeting Feature to Track and Manage Spending (#36) 2026-01-21 15:25:50 +01:00
2026_01_08_192757_add_email_identifier_to_user_mail_logs.php feat: Send custom emails to users (#52) 2026-01-09 09:33:19 +01:00
2026_01_08_204223_update_user_mail_logs_unique_constraint.php feat: Send custom emails to users (#52) 2026-01-09 09:33:19 +01:00
2026_01_21_151353_add_indexes_for_budget_transaction_assignment.php feat: Load transactions history on budget created (#72) 2026-01-22 11:10:15 +01:00
2026_01_22_071325_add_processing_historical_to_budget_periods.php feat: Load transactions history on budget created (#72) 2026-01-22 11:10:15 +01:00
2026_01_22_143232_add_locale_to_users_table.php feat: Spanish localization (#74) 2026-02-08 11:58:08 +01:00
2026_02_08_103953_create_user_settings_table.php Add chart color scheme setting (#101) 2026-02-28 12:58:21 +01:00
2026_02_09_110624_add_encrypted_to_accounts_table.php Remove encryption from bank account names (#104) 2026-02-09 14:15:26 +01:00
2026_02_09_131403_make_description_iv_nullable_on_transactions.php feat: Plaintext transactions behind feature flag (#105) 2026-02-10 14:31:24 +01:00
2026_02_09_144915_create_banking_connections_table.php feat: Integrate EnableBanking as open banking provider (#106) 2026-02-12 09:09:28 +01:00
2026_02_09_144959_add_banking_connection_fields_to_accounts_table.php feat: Integrate EnableBanking as open banking provider (#106) 2026-02-12 09:09:28 +01:00
2026_02_09_144959_add_external_transaction_id_to_transactions_table.php feat: Integrate EnableBanking as open banking provider (#106) 2026-02-12 09:09:28 +01:00
2026_02_10_085759_add_aspsp_logo_to_banking_connections_table.php feat: Integrate EnableBanking as open banking provider (#106) 2026-02-12 09:09:28 +01:00
2026_02_10_095639_add_raw_data_to_transactions_table.php feat: Integrate EnableBanking as open banking provider (#106) 2026-02-12 09:09:28 +01:00
2026_02_11_113115_add_pending_accounts_data_to_banking_connections_table.php feat: Integrate EnableBanking as open banking provider (#106) 2026-02-12 09:09:28 +01:00
2026_02_11_113119_add_linked_at_to_accounts_table.php feat: Integrate EnableBanking as open banking provider (#106) 2026-02-12 09:09:28 +01:00
2026_02_13_083716_add_original_description_to_transactions_table.php feat: Add per-bank description formatter for bank-synced transactions (#120) 2026-02-13 09:54:58 +01:00
2026_02_18_084241_add_api_token_to_banking_connections_table.php feat: Add Indexa Capital integration (#130) 2026-02-18 10:42:13 +01:00
2026_02_18_110556_add_api_secret_to_banking_connections_table.php feat: Add Binance integration (#131) 2026-02-18 15:23:46 +01:00
2026_02_20_074846_create_exchange_rates_table.php feat: Add multi-currency conversion for net worth charts (#138) 2026-02-20 10:45:23 +01:00
2026_02_22_180317_add_invested_amount_to_account_balances_table.php feat: investment benefits — show gains/losses on investment accounts (#140) 2026-02-23 13:59:10 +01:00
2026_02_24_193117_fix_budget_transaction_refund_amounts.php fix(budgets): handle refunds correctly in budget spending calculations (#152) 2026-02-24 21:12:36 +01:00
2026_03_03_114620_add_paywall_seen_at_to_users_table.php feat(subscription): allow free plan for open banking users without connected banks (#188) 2026-03-03 22:28:50 +00:00
2026_03_04_085843_add_waitlist_columns_to_user_leads_table.php fix(migration): make add_waitlist_columns migration idempotent (#200) 2026-03-04 12:06:38 +00:00
2026_03_04_094926_add_locale_to_user_leads_table.php feat(waitlist): waiting list with referral system (#199) 2026-03-04 12:36:47 +01:00
2026_03_12_095411_add_iban_to_accounts_table.php fix(banking): update external_account_id on reconnect and store IBAN (#220) 2026-03-12 10:28:23 +00:00
2026_03_18_112307_add_cashflow_direction_to_categories_table.php feat(cashflow): track transfer categories in trends (#236) 2026-03-18 14:02:47 +00:00
2026_03_19_151039_add_include_loans_in_net_worth_chart_to_user_settings_table.php fix(dashboard): treat loans as debt in net worth (#238) 2026-03-20 09:55:53 +00:00
2026_03_20_110659_create_real_estate_details_table.php feat(accounts): add real estate asset tracking (#241) 2026-03-24 10:21:32 +00:00
2026_03_20_132058_make_bank_id_nullable_on_accounts_table.php feat(accounts): add real estate asset tracking (#241) 2026-03-24 10:21:32 +00:00
2026_03_20_154755_add_include_real_estate_in_net_worth_chart_to_user_settings_table.php feat(accounts): add real estate asset tracking (#241) 2026-03-24 10:21:32 +00:00
2026_03_25_125717_add_revaluation_percentage_to_real_estate_details_table.php feat(accounts): add market value and annual revaluation to real estate accounts (#245) 2026-03-26 11:02:20 +01:00
2026_03_25_132244_create_loan_details_table.php feat(accounts): add loan amortization projections for loan accounts (#246) 2026-03-26 15:06:09 +01:00
2026_03_30_123011_add_consecutive_sync_failures_to_banking_connections_table.php fix(banking): retry failed sync connections and log every sync attempt (#251) 2026-03-31 11:34:35 +01:00
2026_03_30_123011_create_banking_sync_logs_table.php fix(banking): retry failed sync connections and log every sync attempt (#251) 2026-03-31 11:34:35 +01:00
2026_04_14_120000_add_email_verification_to_user_leads_table.php feat: verify waitlist leads (#285) 2026-04-14 11:26:01 +01:00
2026_04_16_063515_add_bank_transactions_email_cutoff_at_to_banking_connections_table.php fix(open-banking): skip silent sync emails (#295) 2026-04-16 08:28:44 +01:00
2026_04_16_092644_add_timezone_to_users_table.php fix(user): persist detected timezones (#296) 2026-04-16 11:36:57 +01:00
2026_04_22_084113_add_deleted_at_to_users_table.php Support soft-deleted users with reusable emails (#316) 2026-04-22 11:41:41 +01:00
2026_04_25_092824_add_launch_columns_to_user_leads_table.php feat(leads): cohort-based launch invitations with per-user Stripe coupons (#333) 2026-04-30 15:10:28 +01:00
2026_05_05_070442_add_rate_limited_until_to_banking_connections_table.php feat(banking): back off scheduler when EnableBanking returns 429 (#352) 2026-05-05 09:39:32 +02:00
2026_05_05_132023_convert_custom_budgets_to_monthly.php fix(budgets): remove Custom period type to fix duplicate-key crash (#355) 2026-05-05 16:07:16 +02:00
2026_05_13_085027_add_dedup_fingerprint_to_transactions_table.php fix(banking): dedup EnableBanking transactions by deterministic fingerprint (#390) 2026-05-13 11:30:11 +01:00
2026_05_25_115100_update_default_saving_and_investment_category_types.php feat(cashflow): add savings and period views (#424) 2026-05-25 16:41:00 +02:00
2026_05_26_061232_add_reinvitation_columns_to_user_leads_table.php feat(leads): add user lead re-invite campaign (#432) 2026-05-26 08:35:31 +02:00
2026_05_27_061513_add_counterparty_names_to_transactions_table.php feat(transactions): add counterparty fields (#440) 2026-05-27 16:20:55 +02:00
2026_05_28_073203_update_categories_unique_constraint_for_soft_deletes.php fix(categories): allow recreate after delete (#444) 2026-05-28 09:46:02 +02:00
2026_05_29_085835_update_saving_and_investment_category_cashflow_direction.php fix(categories): expose cashflow setting on create (#448) 2026-05-29 11:05:04 +02:00
2026_06_01_085554_create_budget_category_table.php feat(budgets): track multiple categories and labels per budget (#466) 2026-06-01 12:32:23 +02:00
2026_06_01_085554_create_budget_label_table.php feat(budgets): track multiple categories and labels per budget (#466) 2026-06-01 12:32:23 +02:00
2026_06_01_085554_migrate_budget_category_label_to_pivots.php feat(budgets): track multiple categories and labels per budget (#466) 2026-06-01 12:32:23 +02:00
2026_06_02_090000_add_parent_id_to_categories_table.php feat: parent/child category tree (#474) 2026-06-03 19:30:12 +02:00
2026_06_02_094823_add_notify_on_bank_transactions_synced_to_user_settings_table.php feat(settings): let users disable bank transactions email (#472) 2026-06-02 12:24:39 +02:00
2026_06_05_120000_create_saved_filters_table.php feat(transactions): save and reuse transaction filters (#496) 2026-06-05 18:00:14 +02:00
2026_06_05_185212_add_state_token_to_banking_connections_table.php feat(open-banking): finalize bank connection without a session via state token (#498) 2026-06-06 12:12:07 +02:00
2026_06_08_114044_add_analysis_days_to_saved_filters_table.php feat(transactions): filtered analysis dashboard (#507) 2026-06-08 14:04:00 +02:00
2026_06_09_124010_add_analysis_mode_to_saved_filters_table.php feat(analysis): project-aware transaction analysis (#513) 2026-06-09 15:32:07 +02:00
2026_06_10_083644_add_last_logged_in_at_to_users_table.php feat(users): track last login and last active timestamps (#516) 2026-06-10 11:01:30 +02:00
2026_06_10_084055_add_last_active_at_to_users_table.php feat(users): track last login and last active timestamps (#516) 2026-06-10 11:01:30 +02:00
2026_06_12_134225_create_ai_consents_table.php feat(ai): suggest automation rules during onboarding (#523) 2026-06-13 22:51:15 +02:00
2026_06_12_134523_create_suggestion_runs_table.php feat(ai): suggest automation rules during onboarding (#523) 2026-06-13 22:51:15 +02:00
2026_06_12_134524_create_rule_suggestions_table.php feat(ai): suggest automation rules during onboarding (#523) 2026-06-13 22:51:15 +02:00
2026_06_13_000000_add_is_catch_all_to_budgets_table.php feat: add catch-all budgets (#527) 2026-06-15 16:07:19 +00:00
2026_06_15_120911_add_ai_categorization_fields_to_transactions_table.php feat(ai): auto-categorize transactions with AI (behind flag) (#535) 2026-06-15 16:35:20 +02:00
2026_06_15_120911_add_origin_to_automation_rules_table.php feat(ai): auto-categorize transactions with AI (behind flag) (#535) 2026-06-15 16:35:20 +02:00
2026_06_15_120911_create_category_corrections_table.php feat(ai): auto-categorize transactions with AI (behind flag) (#535) 2026-06-15 16:35:20 +02:00
2026_06_17_081319_add_ai_suggested_category_to_transactions_table.php feat(ai): persist AI categorization suggestions below the label bar (#547) 2026-06-17 08:24:30 +00:00
2026_06_17_114601_create_integration_requests_table.php feat(integration-requests): community board to request & vote bank integrations (#550) 2026-06-17 12:50:51 +00:00
2026_06_17_114602_create_integration_request_votes_table.php feat(integration-requests): community board to request & vote bank integrations (#550) 2026-06-17 12:50:51 +00:00
2026_06_17_120850_add_status_to_integration_requests_table.php feat(integration-requests): community board to request & vote bank integrations (#550) 2026-06-17 12:50:51 +00:00
2026_06_17_122001_seed_initial_integration_requests.php feat(integration-requests): community board to request & vote bank integrations (#550) 2026-06-17 12:50:51 +00:00
2026_06_17_141827_add_comment_to_integration_requests_table.php feat(integration-requests): add not-doable status with a public comment (#552) 2026-06-17 16:36:32 +02:00
2026_06_18_092830_drop_unique_user_vote_from_integration_request_votes_table.php feat(integration-requests): multi-vote, per-plan quota and undo (#554) 2026-06-18 10:06:08 +00:00
2026_06_19_134854_create_stuck_cohort_snapshots_table.php feat(stats): weekly paywall stuck-cohort report to Discord (#563) 2026-06-19 14:12:51 +00:00
2026_06_20_105609_align_accounts_encrypted_flag_with_plaintext_names.php refactor(encryption): strip client-side transaction encryption (#514) 2026-06-20 16:13:26 +00:00
2026_06_20_165235_add_position_to_accounts_table.php feat(accounts): reorder accounts with drag-and-drop (#575) 2026-06-21 11:17:45 +02:00
2026_06_26_124609_add_ai_model_to_transactions_table.php feat(ai): record the model behind each AI categorization (#594) 2026-06-26 12:53:55 +00:00
2026_06_27_000000_backfill_xxx_account_currencies.php fix(open-banking): stop storing the XXX no-currency placeholder on accounts (#602) 2026-06-27 16:01:21 +00:00
2026_06_27_114559_add_refunded_at_to_subscriptions_table.php feat(subscriptions): trial/pricing A/B/C experiment (#600) 2026-06-27 18:00:15 +02:00
2026_06_27_154041_add_hidden_on_dashboard_to_accounts_table.php feat(dashboard): add accounts manager dialog with visibility toggle and reorder (#604) 2026-06-27 16:11:25 +00:00
2026_06_29_161916_add_transactions_last_visited_at_to_users_table.php feat(transactions): make new-transaction marker cross-device (#611) 2026-06-29 19:11:37 +02:00
2026_07_01_064324_add_ai_consent_prompt_dismissed_at_to_users_table.php feat(ai): dismissable AI consent banner that stops after the first decision (#617) 2026-07-01 07:26:36 +00:00
2026_07_02_133321_add_user_source_created_at_index_to_transactions_table.php perf(db): index transactions for the daily synced-email slow query (PHP-LARAVEL-3X) (#622) 2026-07-02 15:51:15 +02:00