Fixes a design flaw in #700, before the experiment has any exposure.
## The problem
#700 drew the arm from `crc32('price:' . $user->id)`, which can only
happen once the user exists. But the landing quotes a price to anonymous
visitors, and `plansFor(null)` had no arm to apply — so **every visitor
saw €3.99**, and half of them were switched to €8.99 after registering.
That biases the result in both directions at once:
- **Against `high`** — it pays a penalty that isn't price sensitivity
but a price that moved after being advertised. Someone who'd have
happily paid €8.99 quoted upfront leaves because they feel baited.
- **In favour of `high`** — its funnel only contains people who already
decided to sign up under a €3.99 promise. In a world where we actually
charge €8.99, the landing says €8.99 and some of them never register at
all. The experiment is blind to that drop-off.
Neither bias is recoverable from the data, and they don't cancel: a
narrow loss for `high` would be uninterpretable.
## The fix
Draw the arm for the **anonymous visitor** on their first page view,
keep it in a year-long cookie, and freeze it onto `users.price_arm` at
registration. The landing quotes what checkout will charge, and the
whole funnel is measured under one price.
| | before | after |
|---|---|---|
| assigned at | registration | first page view |
| source | `crc32('price:' . id) % 2` | random 50/50 draw |
| stored in | nothing (recomputed) | cookie → `users.price_arm` |
| landing shows | always control | the visitor's arm |
Details worth a look in review:
- **The draw is random, not a hash.** There's no stable identifier
before the user exists. This is what forces the column — the arm has to
outlive the cookie.
- **The middleware writes the arm onto the current request**, not just
the response cookie. The cookie only reaches the browser *after* this
response, and the first view is the landing — the one page that most
needs the right price.
- **Checkout reads `users.price_arm` only, never the cookie.** Editing
your cookie after signing up doesn't get you the cheap price. There's a
test that asserts exactly this, with the cookie set to `control` and the
stored arm `high`.
- **No arm = control.** Users from before the experiment, cookies
blocked, arriving at a deep link. `sanitize()` narrows both the cookie
and the column, since a user controls the former.
- **`force_variant` now also stops the draw** — a winner being rolled
out means the split is over.
- **`price_arm` is in `$hidden`.** The frontend has no use for it, and
it needn't be visible in the Inertia payload.
## Exposure
**None.** The experiment started at 14:00 UTC and had **0 signups past
the cutoff** when this was written, so no arm needs reconciling and no
user changes price. This is the last moment this change is free.
## Reading results
The CRC32 expression from #700 is obsolete:
```sql
SELECT COALESCE(price_arm, 'legacy') AS arm, COUNT(*)
FROM users WHERE created_at >= '<started_at>' GROUP BY arm;
```
## Still open
The landing now shows €8.99 to half of anonymous visitors, which means
the experiment can finally affect signup volume itself. That's the point
— but it also means a drop in registrations is a *result*, not a bug,
and shouldn't be rolled back on reflex.
## Tests
19 tests in `PriceExperimentTest`, rewritten around the new mechanism:
the draw, the gate, the forced winner, the cookie→prop path on the first
visit, freezing at registration, and checkout ignoring the cookie.
`Auth`, `SubscriptionTest`, `InertiaSharedDataTest`, `CashflowPageTest`
and `SyncStripePricesCommandTest` all green locally (122 tests). PHPStan
and `crap` clean.
## What
An A/B price experiment to find the price that **maximizes contribution
margin per new user**, not just conversion. **Inert until
`PRICE_EXPERIMENT_STARTED_AT` is set** — merging changes nothing in
production.
| Arm | Monthly | Annual (= monthly × 6) |
|-----|---------|------------------------|
| A · control | €3.99 | €23.88 *(unchanged)* |
| B · high | €8.99 | €53.94 |
New signups only; earlier users stay `legacy`/control. Two arms rather
than the originally designed A/B/C: at current signup volume three arms
leave the CM metric underpowered, and the wide €3.99↔€8.99 gap maximizes
the detectable signal.
## Rebuilt on top of #762
This branch was written against the #600 trial experiment. #762 then
ended that experiment and deleted `ExperimentOffer`,
`SubscriptionExperiment`, `ExperimentFunnelCollector`,
`ProportionSignificance` and `BinomialProportion` — the exact foundation
this extended.
Rather than resurrect them, the branch was reset onto `main` and
rewritten: **1478 → 401 insertions, 19 → 8 files.** The previous version
is preserved at the tag
[`price-experiment-full`](https://github.com/whisper-money/whisper-money/tree/price-experiment-full).
## How it works
- **`App\Services\Subscriptions\PriceExperiment`** — `variantFor` /
`plansFor` / `lookupKeyFor`, gated by `PRICE_EXPERIMENT_STARTED_AT`,
winner pinned via `PRICE_EXPERIMENT_FORCE_VARIANT` (env-only, no
deploy).
- Assignment is a **salted hash**, `crc32('price:'.$id) % 2` —
deliberately *not* a stored Pennant feature. Nothing needs persisting,
reading back, or purging when the experiment ends (#762 needed a
migration to delete 1,890 stored assignments). It also costs no query on
the render path, and a report can reproduce the split in SQL with
`CRC32(CONCAT('price:', id))`. The salt keeps the split independent from
any other crc32-based one on the same ids.
- Checkout resolves the lookup key **server-side only**, so a client
can't self-select the cheaper price. `HandleInertiaRequests` makes the
shared `pricing.plans` prop variant-aware, so the paywall and the
upgrade dialogs show exactly what will be charged — no frontend change
needed.
- **`stripe:sync-prices`** now also creates the variant tiers.
## Measurement is deliberately not in this PR
The previous version shipped ~1100 lines of measurement:
`stats:price-experiment-funnel`, its collector, `WelchTTest`, `Normal`,
`SampleRatioMismatch`, `MonthlyEquivalentPrices`, a weekly schedule
entry and their tests. All of it is cut here.
Every input is reconstructible at any time — the bucket is
deterministic, and `created_at`, subscriptions and connections are all
stored — so no data is lost by not capturing it weekly. And the design
calls for deciding **only at a pre-registered horizon**, which made a
weekly Discord report whose largest field read *"⚠️ MONITORING ONLY — do
not call a winner from this"* mostly ceremony.
It gets rebuilt from the tag when there is data worth reading. Checking
mid-flight that the split is really 50/50 needs no code:
```sql
SELECT CRC32(CONCAT('price:', id)) % 2 AS arm, COUNT(*)
FROM users WHERE created_at >= '<started_at>' GROUP BY arm;
```
The analysis design itself still stands and is recorded for the horizon:
CM/user primary via Welch (the €8.99 arm carries far more revenue
variance, so pooled-variance is invalid), conversion as a Fisher-exact
guardrail against control, SRM chi-square on assigned and matured
counts, cost from *currently-active* connections, and a guard that every
arm price id resolves in the Stripe product map.
## Incidental
`SyncStripePricesCommand::handle` was already at cyclomatic complexity
11 on `main`; touching one line surfaced it in the `crap` check. The
per-plan body moved into `syncPlan()` — pure extraction, same output.
## Before launching (in order)
1. `php artisan stripe:sync-prices` — creates the €8.99 / €53.94 tiers
under the lookup keys `whisper_pro_monthly_high` and
`whisper_pro_yearly_high`. **Must run before step 2.**
2. Set `PRICE_EXPERIMENT_STARTED_AT` to the launch date.
3. Compute and pre-register the horizon N from the real signup rate — no
peeking before it.
The #600 trial experiment is over, so price is automatically the only
moving variable; there is nothing to pin.
## Tests
Pest coverage for the gate, the forced-variant pin, the salt, split
stability per user, the variant prices and lookup keys, the paywall
prop, server-side checkout resolution, and variant syncing in
`stripe:sync-prices`. Affected suites green locally; `pint` and `php
artisan crap` clean.
Víctor Falcón