From fe747c44728ca444b2b1d4a87f43963b4e5e2026 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?V=C3=ADctor=20Falc=C3=B3n?= Date: Mon, 10 Aug 2026 10:13:40 +0200 Subject: [PATCH] feat(stats): post the Discord stats reports in Spanish, opened by an AI summary (#752) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What The admin Discord channel is Spanish, but the four scheduled `stats:*` reports posted English. They now post Spanish, and the three cohort/experiment reports open with a short AI-written summary so a reader understands the situation without decoding the table. ### 1. Spanish reports `stats:daily-report`, `stats:subscription-funnel`, `stats:experiment-funnel` and `stats:ai-cohort-report` now post Spanish embed titles, field names, ASCII table headers, legends and disclaimers, with Spanish dates (`sáb., 13 jun. 2026`). Table headers and cells stay ASCII (`Semana`, `Variante`, `UMad`, `pdte`, …) so `sprintf`'s byte padding keeps the columns aligned inside the code block. Hardcoded, not `__()`: this is an internal channel, not user-facing UI, so it never needs a second language and doesn't belong in `lang/es.json`. Everything else — code, comments, PHPDoc, command descriptions, `$this->info()` — stays English. ### 2. AI summary (best-effort) New `ReportSummarizer` + `ReportSummaryAgent` (laravel/ai, same Gemini-Flash pattern and `AI_PROVIDER` switch as the other AI features, config in `config/ai_reports.php`). It prepends a few sentences to the embed description that compare the current period against the previous one (week over week for the two weekly reports, month over month for the monthly cohort one), say what got better or worse, and call out explicitly when a figure isn't conclusive — small sample, immature cohort, signup surge week, or no previous period. - **Data**: only what each collector already computes. The two cohort reports pass their weekly series with the per-metric maturity flags; the experiment report passes the per-variant figures plus the already-rendered significance verdict (one source of truth), with money figures nulled exactly where the table renders `—`, so the summary can't report a zero where the reader sees no data. - **Previous period**: the experiment report has no time series, so each run caches its figures (with a capture timestamp) as the next run's baseline. A same-day manual re-run doesn't overwrite it, and the model is told to flag a gap that isn't roughly one period. - **Degrades safely**: no API key, a provider outage, a bad provider name or a slow response (30s timeout) → the report is posted unchanged, without the summary. Transient provider errors are logged; anything else is reported, matching `CategorizeTransactions` / `LaravelAiRuleSuggestionGenerator`. ### 3. Fix found while reviewing: Discord's embed limits The translated "Cómo leerlo" field came out at 1152 characters, past Discord's 1024-character cap on a field value — Discord rejects the **whole** payload, so the experiment funnel would have silently stopped appearing in the channel every Monday (`DiscordWebhook` only logs the 400). Both long fields are now tighter (745 and 898 chars), `DiscordWebhook` trims anything still over the limit instead of losing the report, and a test measures every scheduled report's embed so growing copy fails in CI rather than in production. ## Testing - `tests/Feature/Ai/ReportSummarizerTest.php`: baseline in/out, same-day re-run, dry run, backtick stripping, truncation, empty answer, transient-vs-reported failures. - `tests/Feature/DiscordReportEmbedLimitsTest.php`: all four embeds inside Discord's limits, plus the trimming fallback. - The four command tests: Spanish assertions, summary is the first thing in the description, and the report is still posted when the AI throws. - 64 tests / 338 assertions pass locally, plus PHPStan and Pint. ## QA Ran all four commands against the local DB with `--no-discord` (and dumped the real webhook payloads with the Discord call faked). Real Gemini output, e.g. the experiment funnel on the second run: > No se ha producido ningún cambio en las métricas del experimento respecto a la ejecución previa del 10 de agosto de 2026. La tasa de conversión sobre usuarios maduros se mantiene en el 4,7 % para la variante reducida, el 4,0 % para pay_now y el 3,7 % para el control. Las cifras no son conclusivas ya que las diferencias entre variantes no son estadísticamente significativas, con un p-valor de 0,593 frente al umbral α de 0,017. Also verified with an unreachable provider: the report renders in full, without the summary. ## Not included - `stats:stuck-cohort-report` is still English. It posts to the same webhook but isn't scheduled in `routes/console.php`, so it was out of the four active reports; worth its own decision. - The `—` and `⚡` cells are still a couple of bytes off inside the code block (multibyte in `sprintf`). Pre-existing on `main`, unchanged here. --- .env.example | 7 + README.md | 3 + app/Ai/Agents/ReportSummaryAgent.php | 51 ++++++++ .../Commands/SendAiCohortReportCommand.php | 82 +++++++++--- .../Commands/SendDailyStatsReportCommand.php | 18 +-- .../SendExperimentFunnelReportCommand.php | 120 ++++++++++++++---- .../SendSubscriptionFunnelReportCommand.php | 85 ++++++++++--- app/Services/Ai/ReportSummarizer.php | 109 ++++++++++++++++ app/Services/Discord/DiscordWebhook.php | 49 ++++++- config/ai_reports.php | 36 ++++++ tests/Feature/Ai/ReportSummarizerTest.php | 105 +++++++++++++++ .../Feature/DiscordReportEmbedLimitsTest.php | 76 +++++++++++ .../Feature/SendAiCohortReportCommandTest.php | 43 ++++++- .../SendDailyStatsReportCommandTest.php | 12 +- .../SendExperimentFunnelReportCommandTest.php | 82 ++++++++++-- ...endSubscriptionFunnelReportCommandTest.php | 56 +++++++- 16 files changed, 851 insertions(+), 83 deletions(-) create mode 100644 app/Ai/Agents/ReportSummaryAgent.php create mode 100644 app/Services/Ai/ReportSummarizer.php create mode 100644 config/ai_reports.php create mode 100644 tests/Feature/Ai/ReportSummarizerTest.php create mode 100644 tests/Feature/DiscordReportEmbedLimitsTest.php diff --git a/.env.example b/.env.example index 8fae2340..ae84bfa1 100644 --- a/.env.example +++ b/.env.example @@ -151,6 +151,7 @@ DEMO_ENCRYPTION_KEY=demo # AI_PROVIDER=gemini # AI_SUGGESTIONS_PROVIDER=gemini # AI_CATEGORIZATION_PROVIDER=gemini +# AI_REPORTS_PROVIDER=gemini # --- Gemini (default provider) --- GEMINI_API_KEY= @@ -185,6 +186,12 @@ AI_SUGGESTIONS_REPORT_WEEKS=16 # account that plants the release-anchor consent on deploy). AI_SUGGESTIONS_REPORT_EXCLUDED_EMAILS= +# AI summary opening the scheduled stats reports (subscription funnel, +# experiment funnel and AI cohort). Best-effort: without a key, or if the +# provider fails or is slow, the report is still posted without the summary. +# AI_REPORTS_MODEL=gemini-flash-latest +# AI_REPORTS_TIMEOUT=30 + # Discord webhooks DISCORD_WEBHOOK_URL= # Optional dedicated channel for the AI cohort report; falls back to DISCORD_WEBHOOK_URL. diff --git a/README.md b/README.md index baa79a17..67c88900 100644 --- a/README.md +++ b/README.md @@ -190,8 +190,11 @@ unknown or non-text provider fails fast when the AI feature runs. | `AI_PROVIDER` | `gemini` | Provider for all AI features. Set once to switch everything. | | `AI_SUGGESTIONS_PROVIDER` | `AI_PROVIDER` | Override the provider for rule suggestions only. | | `AI_CATEGORIZATION_PROVIDER` | `AI_PROVIDER` | Override the provider for transaction categorization only. | +| `AI_REPORTS_PROVIDER` | `AI_PROVIDER` | Override the provider for the stats-report summaries only. | | `AI_SUGGESTIONS_MODEL` | `gemini-flash-latest`| Model used for rule suggestions. | | `AI_CATEGORIZATION_MODEL` | `gemini-flash-latest`| Model used for transaction categorization. | +| `AI_REPORTS_MODEL` | `gemini-flash-latest`| Model used for the stats-report summaries. | +| `AI_REPORTS_TIMEOUT` | `30` | Seconds before a report is posted without its AI summary. | | `GEMINI_API_KEY` | - | Required when the provider is `gemini`. | | `OLLAMA_URL` | `http://localhost:11434` | Ollama server URL (used when the provider is `ollama`). | | `OLLAMA_API_KEY` | - | Optional; only needed behind an authenticating proxy. | diff --git a/app/Ai/Agents/ReportSummaryAgent.php b/app/Ai/Agents/ReportSummaryAgent.php new file mode 100644 index 00000000..5bafc987 --- /dev/null +++ b/app/Ai/Agents/ReportSummaryAgent.php @@ -0,0 +1,51 @@ +context} + + You are given a JSON object with "current" (the figures of the report about + to be posted), "previous" (the same report measured on an earlier run, null + when there is none yet) and "previous_captured_at" (when that earlier run + happened). Rows are keyed by period and freeze once they are scored, so a + row that went from null to a value in "previous" has simply matured — that + is not an improvement. Compare like-for-like periods only, and if + "previous_captured_at" is not roughly one period back, say so. + + Rules: + - Answer in Spanish (Spain), as plain text. No markdown, no bullet lists, + no emoji, no greeting, no sign-off, and never restate the whole table. + - At most 4 short sentences, and every sentence must carry information. No + filler, no flourish, no advice nobody asked for. + - Say what changed against the previous period, whether it is getting + better or worse, and the most likely reason. + - Say explicitly when a figure is not conclusive: small sample, immature + cohort, signup surge week, or no previous period to compare against. + - Use only the figures in the payload. Never invent a metric or a number; + when something cannot be derived from the payload, say so instead. + PROMPT; + } +} diff --git a/app/Console/Commands/SendAiCohortReportCommand.php b/app/Console/Commands/SendAiCohortReportCommand.php index 2d28911f..2f0fcf01 100644 --- a/app/Console/Commands/SendAiCohortReportCommand.php +++ b/app/Console/Commands/SendAiCohortReportCommand.php @@ -4,6 +4,7 @@ namespace App\Console\Commands; use App\Console\Commands\Concerns\RendersReportToConsole; use App\Services\Ai\AiCohortReportCollector; +use App\Services\Ai\ReportSummarizer; use App\Services\Discord\DiscordWebhook; use Carbon\CarbonImmutable; use Illuminate\Console\Command; @@ -16,8 +17,24 @@ class SendAiCohortReportCommand extends Command protected $description = 'Post the weekly AI-suggestions cohort retention/conversion report to Discord'; - public function __construct(private AiCohortReportCollector $collector) - { + /** + * What the AI summary is looking at, and which periods it must compare. + */ + private const SUMMARY_CONTEXT = <<<'CONTEXT' + The report tracks the weekly cohorts of users eligible for the AI suggestions + feature (retention, trial, paid and AI-consent rates): one row per signup + week, every metric measured at the same cohort age, with "phase" telling + whether the cohort signed up before or after the feature was released. Each + rate is null until its own horizon has elapsed: "retention_mature" governs + retained_rate and trial_rate, "paid_mature" governs paid_rate. The report is + posted on the first day of each month, so compare the last four weeks against + the four before them — month against previous month, not week against week. + CONTEXT; + + public function __construct( + private AiCohortReportCollector $collector, + private ReportSummarizer $summarizer, + ) { parent::__construct(); } @@ -32,7 +49,15 @@ class SendAiCohortReportCommand extends Command $weeks = $this->option('weeks') !== null ? (int) $this->option('weeks') : null; $report = $this->collector->collect($weeks); - $embed = $this->buildEmbed($report); + + $summary = $this->summarizer->summarize( + 'ai-cohort-report', + self::SUMMARY_CONTEXT, + $this->summaryPayload($report), + remember: ! $this->option('no-discord'), + ); + + $embed = $this->buildEmbed($report, $summary); if ($this->option('no-discord')) { $this->printEmbeds([$embed]); @@ -52,12 +77,37 @@ class SendAiCohortReportCommand extends Command } /** + * The figures the AI summary may talk about — the same ones the table shows. + * * @param array{releaseAt: ?CarbonImmutable, releaseWeek: ?string, weeks: list>} $report * @return array */ - private function buildEmbed(array $report): array + private function summaryPayload(array $report): array { - $lines = [sprintf('%-9s %5s %6s %6s %6s %5s', 'Week', 'Elig', 'Ret', 'Trial', 'Paid', 'AI')]; + return [ + 'release_week' => $report['releaseWeek'], + 'weeks' => array_map(fn (array $row): array => [ + 'week' => $row['week'], + 'eligible' => $row['eligible'], + 'retained_rate' => $row['retainedRate'], + 'trial_rate' => $row['trialRate'], + 'paid_rate' => $row['paidRate'], + 'ai_accepted_rate' => $row['aiAcceptedRate'], + 'phase' => $row['phase'], + 'retention_mature' => $row['retentionMature'], + 'paid_mature' => $row['paidMature'], + 'signup_surge' => $row['surge'], + ], $report['weeks']), + ]; + } + + /** + * @param array{releaseAt: ?CarbonImmutable, releaseWeek: ?string, weeks: list>} $report + * @return array + */ + private function buildEmbed(array $report, ?string $summary): array + { + $lines = [sprintf('%-9s %5s %6s %6s %6s %5s', 'Semana', 'Eleg', 'Ret', 'Prueba', 'Pago', 'IA')]; foreach ($report['weeks'] as $row) { $flags = ''; @@ -83,27 +133,29 @@ class SendAiCohortReportCommand extends Command } $release = $report['releaseAt'] !== null - ? 'First AI consent (release anchor): '.$report['releaseAt']->format('D, d M Y').' · week '.$report['releaseWeek'].' 🚀' - : 'No AI consent recorded yet — feature not live in production.'; + ? 'Primer consentimiento de IA (ancla de lanzamiento): '.$report['releaseAt']->copy()->locale('es')->translatedFormat('D, d M Y').' · semana '.$report['releaseWeek'].' 🚀' + : 'Todavía no hay ningún consentimiento de IA — la función no está activa en producción.'; + + $table = "```\n".implode("\n", $lines)."\n```"; return [ - 'title' => '🤖 AI Suggestions — Weekly Cohort Report', - 'description' => "```\n".implode("\n", $lines)."\n```", + 'title' => '🤖 Sugerencias con IA — informe de cohortes semanales', + 'description' => $summary !== null ? $summary."\n\n".$table : $table, 'color' => 0x5865F2, 'fields' => [ [ - 'name' => 'Release anchor', + 'name' => 'Ancla de lanzamiento', 'value' => $release, 'inline' => false, ], [ - 'name' => 'Legend', - 'value' => 'Elig = users with ≥50 transactions in their first 7 days · Ret = active ≥14d after signup · Trial = subscribed ≤14d · Paid = active subscription ≤30d · AI = accepted AI consent · `pend` = cohort too young to score · ⚡ = signup surge', + 'name' => 'Leyenda', + 'value' => 'Eleg = usuarios con ≥50 transacciones en sus primeros 7 días · Ret = activos ≥14d después de registrarse · Prueba = se suscribieron ≤14d · Pago = suscripción activa ≤30d · IA = aceptaron el consentimiento de IA · `pdte` = cohorte demasiado joven para puntuarla · ⚡ = pico de registros', 'inline' => false, ], [ - 'name' => '⚠️ Directional only', - 'value' => 'Pre/post comparison, not a randomised test. Cohorts are compared at equal age. Surge weeks (⚡, e.g. launch/YouTube) differ in acquisition channel and are not controlled — compare organic weeks like-for-like. Confidence builds over a quarter, not a single month.', + 'name' => '⚠️ Solo orientativo', + 'value' => 'Es una comparación antes/después, no un test aleatorizado. Las cohortes se comparan a la misma edad. Las semanas con pico (⚡, p. ej. lanzamiento o YouTube) llegan por otro canal de adquisición y no están controladas — compara semanas orgánicas entre sí. La confianza se construye a lo largo de un trimestre, no en un solo mes.', 'inline' => false, ], ], @@ -117,7 +169,7 @@ class SendAiCohortReportCommand extends Command } if (! $mature || $rate === null) { - return 'pend'; + return 'pdte'; } return ((int) round($rate * 100)).'%'; diff --git a/app/Console/Commands/SendDailyStatsReportCommand.php b/app/Console/Commands/SendDailyStatsReportCommand.php index ad284618..00534e05 100644 --- a/app/Console/Commands/SendDailyStatsReportCommand.php +++ b/app/Console/Commands/SendDailyStatsReportCommand.php @@ -79,8 +79,8 @@ class SendDailyStatsReportCommand extends Command { $fields = [ [ - 'name' => '👥 Users', - 'value' => "New yesterday: **{$newUsers}**\nTotal: **{$totalUsers}**", + 'name' => '👥 Usuarios', + 'value' => "Nuevos ayer: **{$newUsers}**\nTotal: **{$totalUsers}**", 'inline' => false, ], ]; @@ -98,10 +98,10 @@ class SendDailyStatsReportCommand extends Command $fields[] = [ 'name' => '💳 '.strtoupper($currency), 'value' => implode("\n", [ - "Active: **{$active['count']}** ({$this->money($currentMrr, $currency)} MRR)", - "Trialing: **{$trialing['count']}** ({$this->money($trialing['mrr'], $currency)} MRR)", - "Current MRR/ARR: **{$this->money($currentMrr, $currency)}** / **{$this->money($currentMrr * 12, $currency)}**", - "Projected MRR/ARR: **{$this->money($projectedMrr, $currency)}** / **{$this->money($projectedMrr * 12, $currency)}**", + "Activas: **{$active['count']}** ({$this->money($currentMrr, $currency)} de MRR)", + "En prueba: **{$trialing['count']}** ({$this->money($trialing['mrr'], $currency)} de MRR)", + "MRR/ARR actual: **{$this->money($currentMrr, $currency)}** / **{$this->money($currentMrr * 12, $currency)}**", + "MRR/ARR previsto: **{$this->money($projectedMrr, $currency)}** / **{$this->money($projectedMrr * 12, $currency)}**", ]), 'inline' => false, ]; @@ -109,14 +109,14 @@ class SendDailyStatsReportCommand extends Command if ($currencies === []) { $fields[] = [ - 'name' => '💳 Subscriptions', - 'value' => 'No active or trialing subscriptions.', + 'name' => '💳 Suscripciones', + 'value' => 'No hay suscripciones activas ni en prueba.', 'inline' => false, ]; } return [ - 'title' => '📊 Daily Stats — '.$day->format('D, d M Y'), + 'title' => '📊 Estadísticas diarias — '.$day->copy()->locale('es')->translatedFormat('D, d M Y'), 'color' => 0x5865F2, 'fields' => $fields, ]; diff --git a/app/Console/Commands/SendExperimentFunnelReportCommand.php b/app/Console/Commands/SendExperimentFunnelReportCommand.php index 37859c29..f1fbbde8 100644 --- a/app/Console/Commands/SendExperimentFunnelReportCommand.php +++ b/app/Console/Commands/SendExperimentFunnelReportCommand.php @@ -3,6 +3,7 @@ namespace App\Console\Commands; use App\Features\SubscriptionExperiment; +use App\Services\Ai\ReportSummarizer; use App\Services\Discord\DiscordWebhook; use App\Services\Stats\BinomialProportion; use App\Services\Stats\ExperimentFunnelCollector; @@ -15,7 +16,7 @@ class SendExperimentFunnelReportCommand extends Command { protected $signature = 'stats:experiment-funnel {--no-discord : Print the report to the console only, without posting to Discord} - {--cost-per-connection=0.4 : Estimated cost (in the Cashier currency) per bank connection, used for the Cost/Burn/CM columns}'; + {--cost-per-connection=0.4 : Estimated cost (in the Cashier currency) per bank connection, used for the cost, burn and contribution-margin columns}'; protected $description = 'Post the trial/pricing experiment funnel (per variant) to Discord'; @@ -25,9 +26,25 @@ class SendExperimentFunnelReportCommand extends Command SubscriptionExperiment::PAY_NOW => 'pay_now', ]; + /** + * What the AI summary is looking at, and which periods it must compare. + */ + private const SUMMARY_CONTEXT = <<<'CONTEXT' + The report is the A/B/C trial-and-pricing experiment: one row per variant + (control, reduced, pay_now), cumulative since the experiment started. Conv% + (conversions over matured users) and ARPU are the comparable metrics; the + absolute MRR, cost and margin totals scale with how many users have matured, + which differs per variant by design. Monetary amounts are in cents of the + report currency, and null means there is no data yet rather than zero. The + report is posted every Monday: compare against the previous run to say what + moved this week, and never call a winner the significance block does not + support. + CONTEXT; + public function __construct( private ExperimentFunnelCollector $collector, private ProportionSignificance $significance, + private ReportSummarizer $summarizer, ) { parent::__construct(); } @@ -49,6 +66,18 @@ class SendExperimentFunnelReportCommand extends Command return self::SUCCESS; } + $summary = $this->summarizer->summarize( + 'experiment-funnel', + self::SUMMARY_CONTEXT, + $this->summaryPayload($report), + remember: ! $this->option('no-discord'), + ); + + if ($summary !== null) { + $this->line($summary); + $this->newLine(); + } + foreach ($this->tableLines($report) as $line) { $this->line($line); } @@ -66,7 +95,7 @@ class SendExperimentFunnelReportCommand extends Command $webhookUrl = config('services.discord.ai_cohort_webhook_url') ?: config('services.discord.webhook_url'); - (new DiscordWebhook($webhookUrl))->send('', [$this->buildEmbed($report)]); + (new DiscordWebhook($webhookUrl))->send('', [$this->buildEmbed($report, $summary)]); $this->info('Experiment funnel report sent to Discord.'); @@ -83,7 +112,7 @@ class SendExperimentFunnelReportCommand extends Command $currency = $report['currency']; $lines = [sprintf( '%-8s %5s %5s %5s %5s %5s %6s %7s %7s %7s %7s %7s', - 'Variant', 'Assg', 'Actd', 'Card', 'MatU', 'Conv', 'Conv%', 'ARPU', 'MRR', 'Cost', 'Burn', 'CM', + 'Variante', 'Asig', 'Actv', 'Tarj', 'UMad', 'Conv', 'Conv%', 'ARPU', 'MRR', 'Coste', 'Quema', 'MC', )]; foreach (self::LABELS as $key => $label) { @@ -99,7 +128,7 @@ class SendExperimentFunnelReportCommand extends Command $row['subscribed'], $row['assignedMature'], $row['convertedMature'], - $mature ? ((int) round($row['conversionRate'] * 100)).'%' : 'pend', + $mature ? ((int) round($row['conversionRate'] * 100)).'%' : 'pdte', $showMoney && $row['arpuCents'] !== null ? Money::format($row['arpuCents'], $currency) : '—', $showMoney ? Money::format($row['mrrCents'], $currency) : '—', $mature ? Money::format($row['costCents'], $currency) : '—', @@ -122,7 +151,7 @@ class SendExperimentFunnelReportCommand extends Command */ private function significanceLines(array $report): array { - $lines = ['', 'Significance (95% Wilson CI on Conv%, n = MatU):']; + $lines = ['', 'Significancia (IC de Wilson al 95% sobre Conv%, n = UMad):']; $arms = []; foreach (self::LABELS as $key => $label) { @@ -131,7 +160,7 @@ class SendExperimentFunnelReportCommand extends Command $k = (int) $row['convertedMature']; if ($n <= 0) { - $lines[] = sprintf(' %-8s pend (n=0)', $label); + $lines[] = sprintf(' %-8s pdte (n=0)', $label); continue; } @@ -142,7 +171,7 @@ class SendExperimentFunnelReportCommand extends Command } if (count($arms) < 2) { - $lines[] = 'Not enough matured variants to compare yet.'; + $lines[] = 'Aún no hay suficientes variantes maduras para comparar.'; return $lines; } @@ -152,20 +181,20 @@ class SendExperimentFunnelReportCommand extends Command $result = $this->significance->compare($leader, $runnerUp); $lines[] = sprintf( - 'Leader %s vs %s: Δ %+.1f pts (95%% CI %+.1f … %+.1f pts, Newcombe).', + 'Líder %s vs %s: Δ %+.1f pts (IC 95%% %+.1f … %+.1f pts, Newcombe).', $leader->label, $runnerUp->label, ($leader->rate() - $runnerUp->rate()) * 100, $result['diffLow'] * 100, $result['diffHigh'] * 100, ); $lines[] = sprintf( - 'Fisher exact p=%.3f %s α=%.3f (Bonferroni×3) -> %s.%s', + 'Test exacto de Fisher p=%.3f %s α=%.3f (Bonferroni×3) -> %s.%s', $result['fisherP'], $result['significant'] ? '<' : '≥', $result['alpha'], - $result['significant'] ? 'significant' : 'not significant', - $result['significant'] ? '' : ' Keep running.', + $result['significant'] ? 'significativo' : 'no significativo', + $result['significant'] ? '' : ' Hay que seguir midiendo.', ); if ($result['minExpectedCount'] < 5.0) { $lines[] = sprintf( - '(Small sample: min expected conversions %.1f < 5, so the normal-approx z=%.2f overstates — exact test used.)', + '(Muestra pequeña: el mínimo de conversiones esperadas es %.1f < 5, así que la aproximación normal z=%.2f sobreestima — se usa el test exacto.)', $result['minExpectedCount'], $result['z'], ); } @@ -179,37 +208,84 @@ class SendExperimentFunnelReportCommand extends Command } /** + * The figures the AI summary may talk about — the same ones the table shows, + * nulled under exactly the conditions that render them as "—", so the summary + * can't claim a zero where the reader sees no data. The significance verdict + * is handed over as the rendered lines, keeping one source of truth for it. + * * @param array{startedAt: ?CarbonImmutable, currency: string, revenueAvailable: bool, costPerConnectionCents: int, variants: array>} $report * @return array */ - private function buildEmbed(array $report): array + private function summaryPayload(array $report): array { + $variants = []; + + foreach (self::LABELS as $key => $label) { + $row = $report['variants'][$key]; + $mature = $row['assignedMature'] > 0; + $showMoney = $report['revenueAvailable'] && $mature; + + $variants[$label] = [ + 'assigned' => $row['assigned'], + 'activated' => $row['activated'], + 'carded' => $row['subscribed'], + 'matured_users' => $row['assignedMature'], + 'converted_mature' => $row['convertedMature'], + 'conversion_rate' => $row['conversionRate'], + 'arpu_cents' => $showMoney ? $row['arpuCents'] : null, + 'mrr_cents' => $showMoney ? $row['mrrCents'] : null, + 'cost_cents' => $mature ? $row['costCents'] : null, + 'wasted_cost_cents' => $mature ? $row['wastedCostCents'] : null, + 'contribution_margin_cents' => $showMoney ? $row['contributionMarginCents'] : null, + ]; + } + return [ - 'title' => '🧪 Trial/Pricing Experiment — Funnel by Variant', - 'description' => "```\n".implode("\n", $this->tableLines($report))."\n```", + 'currency' => $report['currency'], + 'revenue_available' => $report['revenueAvailable'], + 'started_at' => $report['startedAt']?->toDateString(), + 'variants' => $variants, + 'significance' => array_values(array_filter($this->significanceLines($report))), + ]; + } + + /** + * @param array{startedAt: ?CarbonImmutable, currency: string, revenueAvailable: bool, costPerConnectionCents: int, variants: array>} $report + * @return array + */ + private function buildEmbed(array $report, ?string $summary): array + { + $table = "```\n".implode("\n", $this->tableLines($report))."\n```"; + + return [ + 'title' => '🧪 Experimento de prueba/precio — embudo por variante', + 'description' => $summary !== null ? $summary."\n\n".$table : $table, 'color' => 0xFEE75C, 'fields' => [ [ - 'name' => 'Started', - 'value' => $report['startedAt']->format('D, d M Y').' · new signups split evenly into the three variants.', + 'name' => 'Inicio', + 'value' => $report['startedAt']->copy()->locale('es')->translatedFormat('D, d M Y').' · los nuevos registros se reparten a partes iguales entre las tres variantes.', 'inline' => false, ], [ - 'name' => '📊 Significance', + 'name' => '📊 Significancia', 'value' => "```\n".implode("\n", $this->significanceLines($report))."\n```", 'inline' => false, ], [ - 'name' => 'Legend', + // Keep this and "Cómo leerlo" tight: they are the only fields + // anywhere near Discord's 1024-character limit per field, + // beyond which DiscordWebhook has to trim them. + 'name' => 'Leyenda', 'value' => sprintf( - 'Assg = signups · Actd = activated (connected a bank or enabled AI = cost triggered) · Card = completed checkout (card on file) · MatU = matured assigned (cohort old enough to score for this variant) · Conv = matured users who ever converted (were charged, net of refund) — time-invariant, so it does not shrink as an older cohort has longer to churn · Conv%% = Conv ÷ MatU (always ≤100%%, comparable across variants) · ARPU = MRR ÷ MatU (revenue per matured user) · MRR = monthly run-rate of *currently* paying subs (yearly ÷ 12); Conv above MRR is churn · Cost = est. connection cost of MatU (%s/connection) · Burn = connection cost of matured users who never earned net revenue (connected a bank but never paid, or paid then refunded) · CM = MRR − Cost · `pend`/`—` = no matured data yet.', + 'Asig = registros · Actv = activados (conectaron un banco o activaron la IA = coste disparado) · Tarj = checkout completado (tarjeta guardada) · UMad = asignados maduros (cohorte con edad para puntuarla en esta variante) · Conv = maduros que llegaron a convertir (se les cobró, menos devoluciones); no depende del momento, así que no baja porque una cohorte antigua haya tenido más tiempo para cancelar · Conv%% = Conv ÷ UMad · ARPU = MRR ÷ UMad · MRR = ritmo mensual de quienes pagan *ahora* (anuales ÷ 12); si Conv va por encima del MRR, es churn · Coste = coste estimado de conexiones de UMad (%s por conexión) · Quema = coste de conexión de maduros que nunca dejaron ingreso neto · MC = MRR − Coste · `pdte`/`—` = todavía sin datos maduros.', Money::format($report['costPerConnectionCents'], $report['currency']), ), 'inline' => false, ], [ - 'name' => '⚠️ How to read it', - 'value' => 'Each variant matures on its own decision window (control 15d, reduced 7d, pay_now 3d, +3d settle), so at any moment MatU differs a lot between variants (pay_now matures first). **Compare variants on Conv% and ARPU — normalized per matured user — not on the absolute MRR/Cost/Burn/CM totals, which scale with MatU and so mechanically favour whichever variant has matured more.** Assg/Actd/Card are lifetime counts; everything from MatU rightward covers the matured cohort only, so the raw Actd→Card→Conv funnel mixes cohorts (immature carded users can\'t have matured yet) — read it for volume. Conv counts anyone ever charged (net of refund), so it is not depressed for older cohorts the way a live-active snapshot would be. Per-user CM is sub-cent at current volume, so treat CM as directional context, not the decision. Check significance (sample size = MatU) before calling a winner. Cost is a flat per-connection estimate across all providers, not per-provider billing.', + 'name' => '⚠️ Cómo leerlo', + 'value' => 'Cada variante madura con su propia ventana de decisión (control 15d, reduced 7d, pay_now 3d, +3d de liquidación), así que UMad difiere mucho entre variantes (pay_now madura antes). **Compara por Conv% y ARPU — normalizados por usuario maduro — y no por los totales de MRR/Coste/Quema/MC, que escalan con UMad y favorecen a la variante que más ha madurado.** Asig/Actv/Tarj son recuentos de toda la vida; de UMad hacia la derecha solo cuenta la cohorte madura, así que el embudo Actv→Tarj→Conv mezcla cohortes: léelo como volumen. Conv cuenta a quien se le cobró alguna vez, así que no se hunde en las cohortes antiguas como haría una foto de activos de hoy. El MC por usuario está por debajo del céntimo con este volumen: es contexto, no la decisión. Comprueba la significancia (n = UMad) antes de dar un ganador. El coste es una estimación plana por conexión, no la factura real de cada proveedor.', 'inline' => false, ], ], diff --git a/app/Console/Commands/SendSubscriptionFunnelReportCommand.php b/app/Console/Commands/SendSubscriptionFunnelReportCommand.php index bf91525a..effc1a8b 100644 --- a/app/Console/Commands/SendSubscriptionFunnelReportCommand.php +++ b/app/Console/Commands/SendSubscriptionFunnelReportCommand.php @@ -2,6 +2,7 @@ namespace App\Console\Commands; +use App\Services\Ai\ReportSummarizer; use App\Services\Discord\DiscordWebhook; use App\Services\Stats\SubscriptionFunnelCollector; use Illuminate\Console\Command; @@ -12,8 +13,23 @@ class SendSubscriptionFunnelReportCommand extends Command protected $description = 'Post the weekly registration -> subscription -> paid funnel to Discord'; - public function __construct(private SubscriptionFunnelCollector $collector) - { + /** + * What the AI summary is looking at, and which periods it must compare. + */ + private const SUMMARY_CONTEXT = <<<'CONTEXT' + The report is a weekly signup -> subscription -> paid funnel: one row per + signup-week cohort, every stage measured at the same cohort age. Each rate is + null until its own horizon has elapsed: "subscribed_mature" governs + subscribed_rate, "paid_mature" governs paid_rate and trial_to_paid_rate. The + report is posted every Monday, so compare the most recent week that is mature + for a given rate against the one before it, and put it in the context of the + trend across the mature weeks. + CONTEXT; + + public function __construct( + private SubscriptionFunnelCollector $collector, + private ReportSummarizer $summarizer, + ) { parent::__construct(); } @@ -29,6 +45,18 @@ class SendSubscriptionFunnelReportCommand extends Command $report = $this->collector->collect($weeks); + $summary = $this->summarizer->summarize( + 'subscription-funnel', + self::SUMMARY_CONTEXT, + $this->summaryPayload($report), + remember: ! $this->option('no-discord'), + ); + + if ($summary !== null) { + $this->line($summary); + $this->newLine(); + } + foreach ($this->tableLines($report) as $line) { $this->line($line); } @@ -42,7 +70,7 @@ class SendSubscriptionFunnelReportCommand extends Command $webhookUrl = config('services.discord.ai_cohort_webhook_url') ?: config('services.discord.webhook_url'); - (new DiscordWebhook($webhookUrl))->send('', [$this->buildEmbed($report)]); + (new DiscordWebhook($webhookUrl))->send('', [$this->buildEmbed($report, $summary)]); $this->info('Subscription funnel report sent to Discord.'); @@ -55,7 +83,7 @@ class SendSubscriptionFunnelReportCommand extends Command */ private function tableLines(array $report): array { - $lines = [sprintf('%-9s %5s %5s %5s %5s %5s %5s', 'Week', 'Reg', 'Sub', 'Sub%', 'Paid', 'Pd%', 'T2P')]; + $lines = [sprintf('%-9s %5s %5s %5s %5s %5s %5s', 'Semana', 'Reg', 'Sub', 'Sub%', 'Pago', 'Pag%', 'P/S')]; foreach ($report['weeks'] as $row) { $lines[] = sprintf( @@ -75,10 +103,35 @@ class SendSubscriptionFunnelReportCommand extends Command } /** + * The figures the AI summary may talk about — the same ones the table shows. + * * @param array{trialDays: int, weeks: list>} $report * @return array */ - private function buildEmbed(array $report): array + private function summaryPayload(array $report): array + { + return [ + 'trial_days' => $report['trialDays'], + 'weeks' => array_map(fn (array $row): array => [ + 'week' => $row['week'], + 'registered' => $row['registered'], + 'subscribed' => $row['subscribed'], + 'subscribed_rate' => $row['subscribedRate'], + 'paid' => $row['paid'], + 'paid_rate' => $row['paidRate'], + 'trial_to_paid_rate' => $row['trialToPaidRate'], + 'subscribed_mature' => $row['subscribedMature'], + 'paid_mature' => $row['paidMature'], + 'signup_surge' => $row['surge'], + ], $report['weeks']), + ]; + } + + /** + * @param array{trialDays: int, weeks: list>} $report + * @return array + */ + private function buildEmbed(array $report, ?string $summary): array { $mature = array_values(array_filter($report['weeks'], fn (array $row): bool => $row['paidMature'])); @@ -88,7 +141,7 @@ class SendSubscriptionFunnelReportCommand extends Command $totals = $registered > 0 ? sprintf( - "Registered %d\nSubscribed %d (%s%%)\nPaid %d (%s%% of reg · %s%% of subs)", + "Registrados %d\nSuscritos %d (%s%%)\nPagando %d (%s%% de registros · %s%% de suscritos)", $registered, $subscribed, $this->pct($subscribed / $registered), @@ -96,26 +149,28 @@ class SendSubscriptionFunnelReportCommand extends Command $this->pct($paid / $registered), $subscribed > 0 ? $this->pct($paid / $subscribed) : '—', ) - : 'No mature cohorts yet.'; + : 'Aún no hay cohortes maduras.'; + + $table = "```\n".implode("\n", $this->tableLines($report))."\n```"; return [ - 'title' => '💸 Subscription Funnel — Weekly Cohorts', - 'description' => "```\n".implode("\n", $this->tableLines($report))."\n```", + 'title' => '💸 Embudo de suscripción — cohortes semanales', + 'description' => $summary !== null ? $summary."\n\n".$table : $table, 'color' => 0x57F287, 'fields' => [ [ - 'name' => 'Mature cohorts (baseline)', + 'name' => 'Cohortes maduras (referencia)', 'value' => "```\n".$totals."\n```", 'inline' => false, ], [ - 'name' => 'Legend', - 'value' => 'Reg = signups · Sub = started a plan ≤30d after signup · Paid = that plan billed past the '.$report['trialDays'].'d trial (active, or canceled only after billing) · Sub%/Pd% of signups · T2P = paid ÷ subscribed · `pend` = cohort too young to score · ⚡ = signup surge', + 'name' => 'Leyenda', + 'value' => 'Reg = registros · Sub = empezó un plan ≤30d después de registrarse · Pago = ese plan se cobró al pasar la prueba de '.$report['trialDays'].'d (activo, o cancelado solo después de cobrarse) · Sub%/Pag% sobre registros · P/S = pago ÷ suscritos · `pdte` = cohorte demasiado joven para puntuarla · ⚡ = pico de registros', 'inline' => false, ], [ - 'name' => '⚠️ Directional only', - 'value' => 'Cohorts compared at equal age. Surge weeks (⚡, e.g. launch/marketing) differ in acquisition channel and are not controlled — compare organic weeks like-for-like. This is the pre-A/B baseline, not a randomised test.', + 'name' => '⚠️ Solo orientativo', + 'value' => 'Las cohortes se comparan a la misma edad. Las semanas con pico (⚡, p. ej. lanzamiento o marketing) llegan por otro canal de adquisición y no están controladas — compara semanas orgánicas entre sí. Esta es la referencia previa al A/B, no un test aleatorizado.', 'inline' => false, ], ], @@ -129,7 +184,7 @@ class SendSubscriptionFunnelReportCommand extends Command } if (! $mature || $rate === null) { - return 'pend'; + return 'pdte'; } return $this->pct($rate).'%'; diff --git a/app/Services/Ai/ReportSummarizer.php b/app/Services/Ai/ReportSummarizer.php new file mode 100644 index 00000000..fc0251df --- /dev/null +++ b/app/Services/Ai/ReportSummarizer.php @@ -0,0 +1,109 @@ + $payload the figures the summary may talk about + * @param bool $remember keep this payload as the baseline for the next run + */ + public function summarize(string $reportKey, string $context, array $payload, bool $remember = true): ?string + { + try { + $previous = $this->baseline($reportKey, $payload, $remember); + + $response = (new ReportSummaryAgent($context))->prompt( + (string) json_encode([ + 'current' => $payload, + 'previous' => $previous['payload'] ?? null, + 'previous_captured_at' => $previous['captured_at'] ?? null, + ], JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES), + provider: Lab::from((string) config('ai_reports.provider')), + model: (string) config('ai_reports.model'), + timeout: (int) config('ai_reports.timeout'), + ); + } catch (FailoverableException $exception) { + // An overloaded or rate-limited provider is an expected transient + // condition, not a bug, so it stays out of the error reports. + Log::warning('Report AI summary skipped: provider transient failure.', [ + 'report' => $reportKey, + 'exception' => $exception->getMessage(), + ]); + + return null; + } catch (Throwable $exception) { + // Anything else (a misconfigured provider, an SDK change, a cache + // outage) is a real bug: report it, but still let the report post. + report($exception); + + return null; + } + + // The summary is prepended to a fenced table, so a stray backtick from + // the model would swallow the table into its own code block. + $summary = trim(str_replace('`', '', $response->text)); + + return $summary === '' ? null : Str::limit($summary, self::MAX_SUMMARY_LENGTH - 1, '…'); + } + + /** + * Read the previous run's figures and, unless this is a dry run, leave the + * current ones behind as the next run's baseline. A same-day re-run keeps + * the stored baseline, so re-running a report by hand can't overwrite the + * period the next scheduled run needs to compare against. + * + * @param array $payload + * @return array{captured_at: string, payload: array}|null + */ + private function baseline(string $reportKey, array $payload, bool $remember): ?array + { + $key = "report_summary_baseline:{$reportKey}"; + $previous = Cache::get($key); + + $capturedToday = isset($previous['captured_at']) + && Str::startsWith($previous['captured_at'], now()->toDateString()); + + if ($remember && ! $capturedToday) { + Cache::put($key, [ + 'captured_at' => now()->toIso8601String(), + 'payload' => $payload, + ], now()->addDays(self::BASELINE_DAYS)); + } + + return is_array($previous) ? $previous : null; + } +} diff --git a/app/Services/Discord/DiscordWebhook.php b/app/Services/Discord/DiscordWebhook.php index 03e56088..a62640cb 100644 --- a/app/Services/Discord/DiscordWebhook.php +++ b/app/Services/Discord/DiscordWebhook.php @@ -7,6 +7,15 @@ use Illuminate\Support\Facades\Log; class DiscordWebhook { + /** + * Discord's own caps on embed text. Exceeding either one makes Discord + * reject the whole request, so an over-long report would vanish from the + * channel with nothing but a logged 400 behind it. + */ + private const DESCRIPTION_LIMIT = 4096; + + private const FIELD_VALUE_LIMIT = 1024; + public function __construct(private ?string $webhookUrl) {} /** @@ -22,7 +31,7 @@ class DiscordWebhook $payload = array_filter([ 'content' => $content !== '' ? $content : null, - 'embeds' => $embeds !== [] ? $embeds : null, + 'embeds' => $embeds !== [] ? $this->withinLimits($embeds) : null, ]); $response = Http::asJson()->post($this->webhookUrl, $payload); @@ -34,4 +43,42 @@ class DiscordWebhook ]); } } + + /** + * Trim embed text that would break Discord's limits. Losing the tail of a + * legend is a much better outcome than losing the entire report, and the + * warning says which text needs shortening. + * + * @param array> $embeds + * @return array> + */ + private function withinLimits(array $embeds): array + { + foreach ($embeds as $index => $embed) { + if (isset($embed['description'])) { + $embeds[$index]['description'] = $this->trim($embed['description'], self::DESCRIPTION_LIMIT, 'description'); + } + + foreach ($embed['fields'] ?? [] as $field => $data) { + $embeds[$index]['fields'][$field]['value'] = $this->trim($data['value'], self::FIELD_VALUE_LIMIT, $data['name']); + } + } + + return $embeds; + } + + private function trim(string $text, int $limit, string $label): string + { + if (mb_strlen($text) <= $limit) { + return $text; + } + + Log::warning('Discord embed text trimmed to fit the limit.', [ + 'label' => $label, + 'length' => mb_strlen($text), + 'limit' => $limit, + ]); + + return mb_substr($text, 0, $limit - 1).'…'; + } } diff --git a/config/ai_reports.php b/config/ai_reports.php new file mode 100644 index 00000000..7407c586 --- /dev/null +++ b/config/ai_reports.php @@ -0,0 +1,36 @@ + env('AI_REPORTS_PROVIDER', env('AI_PROVIDER', 'gemini')), + + 'model' => env('AI_REPORTS_MODEL', 'gemini-flash-latest'), + + /* + |-------------------------------------------------------------------------- + | Timeout + |-------------------------------------------------------------------------- + | + | Seconds to wait for the summary before giving up and posting the report + | without it. The summary is a nice-to-have, so a slow provider must never + | hold up the report itself. + | + */ + + 'timeout' => (int) env('AI_REPORTS_TIMEOUT', 30), + +]; diff --git a/tests/Feature/Ai/ReportSummarizerTest.php b/tests/Feature/Ai/ReportSummarizerTest.php new file mode 100644 index 00000000..91cd495f --- /dev/null +++ b/tests/Feature/Ai/ReportSummarizerTest.php @@ -0,0 +1,105 @@ +summarize('test-report', 'Test report.', ['weeks' => [['week' => '2026-W26']]], $remember); +} + +it('feeds the model the previous run figures and when they were captured', function () { + ReportSummaryAgent::fake(['Los registros bajan.']); + + Cache::put('report_summary_baseline:test-report', [ + 'captured_at' => now()->subWeek()->toIso8601String(), + 'payload' => ['weeks' => [['week' => '2026-W25']]], + ], now()->addDay()); + + expect(summarize())->toBe('Los registros bajan.'); + + ReportSummaryAgent::assertPrompted(function ($prompt): bool { + $payload = json_decode($prompt->prompt, true); + + return $payload['previous']['weeks'][0]['week'] === '2026-W25' + && $payload['current']['weeks'][0]['week'] === '2026-W26' + && str_starts_with($payload['previous_captured_at'], now()->subWeek()->toDateString()); + }); +}); + +it('leaves the current figures as the baseline for the next run', function () { + ReportSummaryAgent::fake(['Resumen.']); + + summarize(); + + expect(Cache::get('report_summary_baseline:test-report')) + ->toMatchArray(['payload' => ['weeks' => [['week' => '2026-W26']]]]); +}); + +it('keeps the stored baseline when the report is re-run the same day', function () { + ReportSummaryAgent::fake(['Resumen.', 'Resumen.']); + + Cache::put('report_summary_baseline:test-report', [ + 'captured_at' => now()->toIso8601String(), + 'payload' => ['weeks' => [['week' => 'last-week']]], + ], now()->addDay()); + + summarize(); + + // A manual re-run must not overwrite the period the next scheduled run + // compares against, otherwise the summary reports "no change". + expect(Cache::get('report_summary_baseline:test-report')['payload']) + ->toBe(['weeks' => [['week' => 'last-week']]]); +}); + +it('does not touch the baseline on a dry run', function () { + ReportSummaryAgent::fake(['Resumen.']); + + summarize(remember: false); + + expect(Cache::get('report_summary_baseline:test-report'))->toBeNull(); +}); + +it('strips backticks so the summary cannot swallow the table below it', function () { + ReportSummaryAgent::fake(['Los pagos suben ```mucho```.']); + + expect(summarize())->toBe('Los pagos suben mucho.'); +}); + +it('trims a runaway answer', function () { + ReportSummaryAgent::fake([str_repeat('a', 2000)]); + + expect(mb_strlen((string) summarize()))->toBeLessThanOrEqual(900); +}); + +it('returns null for an empty answer', function () { + ReportSummaryAgent::fake([' ']); + + expect(summarize())->toBeNull(); +}); + +it('returns null and reports the failure when the provider fails', function () { + Exceptions::fake(); + ReportSummaryAgent::fake(fn () => throw new RuntimeException('provider down')); + + expect(summarize())->toBeNull(); + + Exceptions::assertReported(fn (RuntimeException $exception): bool => $exception->getMessage() === 'provider down'); +}); + +it('logs a transient provider failure without reporting it', function () { + Exceptions::fake(); + Log::spy(); + ReportSummaryAgent::fake(fn () => throw ProviderOverloadedException::forProvider('gemini')); + + expect(summarize())->toBeNull(); + + Exceptions::assertNothingReported(); + Log::shouldHaveReceived('warning') + ->withArgs(fn (string $message): bool => str_contains($message, 'provider transient failure')) + ->once(); +}); diff --git a/tests/Feature/DiscordReportEmbedLimitsTest.php b/tests/Feature/DiscordReportEmbedLimitsTest.php new file mode 100644 index 00000000..40b298ea --- /dev/null +++ b/tests/Feature/DiscordReportEmbedLimitsTest.php @@ -0,0 +1,76 @@ + true, + 'subscriptions.experiment.started_at' => '2026-06-01', + 'services.discord.webhook_url' => 'https://discord.test/hook', + 'services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook', + ]); + + Http::fake(['discord.test/*' => Http::response('', 204)]); + ReportSummaryAgent::fake(['Sin cambios relevantes frente al periodo anterior.']); +}); + +it('keeps every scheduled report inside Discord\'s embed limits', function (string $command) { + if ($command === 'stats:daily-report') { + bindMockStripeClientForStats(['active' => [], 'trialing' => []]); + } + + artisan($command)->assertSuccessful(); + + $embed = null; + + Http::assertSent(function ($request) use (&$embed) { + $embed = $request['embeds'][0]; + + return true; + }); + + expect(mb_strlen($embed['description'] ?? ''))->toBeLessThanOrEqual(4096); + + foreach ($embed['fields'] ?? [] as $field) { + expect(mb_strlen($field['value']))->toBeLessThanOrEqual(1024) + // The webhook trims over-long text and marks it with an ellipsis, so + // a trimmed field means the copy itself has to be shortened. + ->and($field['value'])->not->toEndWith('…'); + } +})->with([ + 'stats:daily-report', + 'stats:subscription-funnel', + 'stats:experiment-funnel', + 'stats:ai-cohort-report', +]); + +it('trims over-long embed text instead of letting Discord reject the report', function () { + (new DiscordWebhook('https://discord.test/hook'))->send('', [[ + 'title' => 'Informe', + 'description' => str_repeat('a', 4106), + 'fields' => [ + ['name' => 'Leyenda', 'value' => str_repeat('b', 1034), 'inline' => false], + ['name' => 'Corto', 'value' => 'cabe de sobra', 'inline' => false], + ], + ]]); + + Http::assertSent(function ($request) { + $embed = $request['embeds'][0]; + + return mb_strlen($embed['description']) === 4096 + && mb_strlen($embed['fields'][0]['value']) === 1024 + && str_ends_with($embed['fields'][0]['value'], '…') + && $embed['fields'][1]['value'] === 'cabe de sobra'; + }); +}); diff --git a/tests/Feature/SendAiCohortReportCommandTest.php b/tests/Feature/SendAiCohortReportCommandTest.php index 73b0baf3..4db102e0 100644 --- a/tests/Feature/SendAiCohortReportCommandTest.php +++ b/tests/Feature/SendAiCohortReportCommandTest.php @@ -1,5 +1,6 @@ true]); config(['ai_suggestions.eligibility_min_transactions' => 3]); config(['ai_suggestions.report.excluded_emails' => []]); + ReportSummaryAgent::fake(['La retención se mantiene plana frente al mes anterior.']); }); it('skips the report and hits no external service when subscriptions are disabled', function () { @@ -218,19 +220,54 @@ it('posts the cohort report embed to the configured discord webhook', function ( artisan('stats:ai-cohort-report')->assertSuccessful(); Http::assertSent(function ($request) { + $embed = $request['embeds'][0]; + return $request->url() === 'https://discord.test/hook' - && isset($request['embeds'][0]['title']) - && str_contains($request['embeds'][0]['title'], 'AI Suggestions'); + && str_contains($embed['title'], 'Sugerencias con IA') + // Spanish table header and legend, no leftover English. + && str_contains($embed['description'], 'Semana') + && str_contains($embed['description'], 'Prueba') + && collect($embed['fields'])->contains(fn ($field) => $field['name'] === 'Leyenda'); }); }); +it('opens the embed with the AI summary, above the table', function () { + config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); + Http::fake(['discord.test/*' => Http::response('', 204)]); + ReportSummaryAgent::fake(['La cohorte de este mes convierte peor que la anterior.']); + + cohortUser(referenceNow()->subWeeks(6), ['transactions' => 3, 'lastActiveAt' => referenceNow()->subWeeks(3)]); + + artisan('stats:ai-cohort-report')->assertSuccessful(); + + Http::assertSent(fn ($request) => str_starts_with( + $request['embeds'][0]['description'], + "La cohorte de este mes convierte peor que la anterior.\n\n```", + )); +}); + +it('still posts the report when the AI summary fails', function () { + config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); + Http::fake(['discord.test/*' => Http::response('', 204)]); + ReportSummaryAgent::fake(fn () => throw new RuntimeException('provider down')); + + cohortUser(referenceNow()->subWeeks(6), ['transactions' => 3, 'lastActiveAt' => referenceNow()->subWeeks(3)]); + + artisan('stats:ai-cohort-report')->assertSuccessful(); + + Http::assertSent(fn ($request) => str_starts_with($request['embeds'][0]['description'], '```') + && str_contains($request['embeds'][0]['description'], 'Semana')); +}); + it('prints to the console without posting when --no-discord is set', function () { config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); Http::fake(['discord.test/*' => Http::response('', 204)]); cohortUser(referenceNow()->subWeeks(6), ['transactions' => 3, 'lastActiveAt' => referenceNow()->subWeeks(3)]); - artisan('stats:ai-cohort-report', ['--no-discord' => true])->assertSuccessful(); + artisan('stats:ai-cohort-report', ['--no-discord' => true]) + ->expectsOutputToContain('La retención se mantiene plana frente al mes anterior.') + ->assertSuccessful(); Http::assertNothingSent(); }); diff --git a/tests/Feature/SendDailyStatsReportCommandTest.php b/tests/Feature/SendDailyStatsReportCommandTest.php index 37f13280..3a09ff0f 100644 --- a/tests/Feature/SendDailyStatsReportCommandTest.php +++ b/tests/Feature/SendDailyStatsReportCommandTest.php @@ -36,10 +36,10 @@ test('posts yesterday user counts and stripe stats to discord', function () { Http::assertSent(function ($request) { $embed = $request['embeds'][0]; - $users = collect($embed['fields'])->firstWhere('name', '👥 Users'); + $users = collect($embed['fields'])->firstWhere('name', '👥 Usuarios'); return $request->url() === 'https://discord.test/webhook' - && str_contains($users['value'], 'New yesterday: **1**') + && str_contains($users['value'], 'Nuevos ayer: **1**') && str_contains($users['value'], 'Total: **2**') && collect($embed['fields'])->contains(fn ($f) => str_contains($f['value'], '€10.00')); }); @@ -71,9 +71,9 @@ test('total reflects users at end of yesterday and excludes users created today' $this->artisan('stats:daily-report')->assertSuccessful(); Http::assertSent(function ($request) { - $users = collect($request['embeds'][0]['fields'])->firstWhere('name', '👥 Users'); + $users = collect($request['embeds'][0]['fields'])->firstWhere('name', '👥 Usuarios'); - return str_contains($users['value'], 'New yesterday: **1**') + return str_contains($users['value'], 'Nuevos ayer: **1**') && str_contains($users['value'], 'Total: **2**'); }); }); @@ -87,9 +87,9 @@ test('reports zero new users when none were created yesterday', function () { $this->artisan('stats:daily-report')->assertSuccessful(); Http::assertSent(function ($request) { - $users = collect($request['embeds'][0]['fields'])->firstWhere('name', '👥 Users'); + $users = collect($request['embeds'][0]['fields'])->firstWhere('name', '👥 Usuarios'); - return str_contains($users['value'], 'New yesterday: **0**') + return str_contains($users['value'], 'Nuevos ayer: **0**') && str_contains($users['value'], 'Total: **1**'); }); }); diff --git a/tests/Feature/SendExperimentFunnelReportCommandTest.php b/tests/Feature/SendExperimentFunnelReportCommandTest.php index 58dd1e00..e0993d2d 100644 --- a/tests/Feature/SendExperimentFunnelReportCommandTest.php +++ b/tests/Feature/SendExperimentFunnelReportCommandTest.php @@ -1,5 +1,6 @@ 399], now()->addHour()); + + ReportSummaryAgent::fake(['pay_now sigue por delante, pero la muestra es demasiado pequeña.']); }); it('skips the report and hits no external service when subscriptions are disabled', function () { @@ -258,7 +261,7 @@ it('reports a matured-cohort conversion capped at 100% and prints the matured de Artisan::call('stats:experiment-funnel', ['--no-discord' => true]); $output = Artisan::output(); - expect($output)->toContain('MatU') // matured denominator column is printed + expect($output)->toContain('UMad') // matured denominator column is printed ->toContain('Conv%') ->toContain('100%') // capped, not the old 200% A2P% ->not->toContain('200%'); @@ -275,11 +278,11 @@ it('reports a Wilson confidence interval and defers the verdict while samples ar Artisan::call('stats:experiment-funnel', ['--no-discord' => true]); $output = Artisan::output(); - expect($output)->toContain('Significance') + expect($output)->toContain('Significancia') ->toContain('Wilson') - ->toContain('Fisher exact') // verdict uses the exact test, not the z-approx - ->toContain('not significant') // equal 50/50 rates, n=2 per arm → nowhere near - ->toContain('Small sample'); // min expected conversions < 5 + ->toContain('exacto de Fisher') // verdict uses the exact test, not the z-approx + ->toContain('no significativo') // equal 50/50 rates, n=2 per arm → nowhere near + ->toContain('Muestra pequeña'); // min expected conversions < 5 }); it('declares significance via the exact test when the separation is real', function () { @@ -295,8 +298,8 @@ it('declares significance via the exact test when the separation is real', funct Artisan::call('stats:experiment-funnel', ['--no-discord' => true]); $output = Artisan::output(); - expect($output)->toContain('Fisher exact') - ->not->toContain('not significant'); // the exact test clears the corrected bar + expect($output)->toContain('exacto de Fisher') + ->not->toContain('no significativo'); // the exact test clears the corrected bar }); it('measures conversion as ever-charged, not active-now, so churn does not bias it', function () { @@ -395,8 +398,68 @@ it('posts the experiment funnel embed to discord', function () { artisan('stats:experiment-funnel')->assertSuccessful(); - Http::assertSent(fn ($request) => $request->url() === 'https://discord.test/hook' - && str_contains($request['embeds'][0]['title'], 'Experiment')); + Http::assertSent(function ($request) { + $embed = $request['embeds'][0]; + + return $request->url() === 'https://discord.test/hook' + && str_contains($embed['title'], 'Experimento') + // Spanish table header and legend, no leftover English. + && str_contains($embed['description'], 'Variante') + && collect($embed['fields'])->contains(fn ($field) => $field['name'] === 'Leyenda'); + }); +}); + +it('opens the embed with the AI summary, above the table', function () { + config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); + Http::fake(['discord.test/*' => Http::response('', 204)]); + ReportSummaryAgent::fake(['control convierte mejor, pero la diferencia no es significativa.']); + + experimentUser(SubscriptionExperiment::CONTROL, CarbonImmutable::parse('2026-06-05'), [ + 'status' => 'active', + 'at' => CarbonImmutable::parse('2026-06-05'), + ]); + + artisan('stats:experiment-funnel')->assertSuccessful(); + + Http::assertSent(fn ($request) => str_starts_with( + $request['embeds'][0]['description'], + "control convierte mejor, pero la diferencia no es significativa.\n\n```", + )); +}); + +it('feeds the summary the variant figures and the significance verdict', function () { + Http::fake(); + + experimentUser(SubscriptionExperiment::CONTROL, CarbonImmutable::parse('2026-06-05'), [ + 'status' => 'active', + 'at' => CarbonImmutable::parse('2026-06-05'), + ]); + + artisan('stats:experiment-funnel', ['--no-discord' => true])->assertSuccessful(); + + ReportSummaryAgent::assertPrompted(function ($prompt): bool { + $payload = json_decode($prompt->prompt, true)['current']; + + return $payload['variants']['control']['converted_mature'] === 1 + && $payload['currency'] === 'EUR' + && collect($payload['significance'])->contains(fn (string $line) => str_contains($line, 'Wilson')); + }); +}); + +it('still posts the report when the AI summary fails', function () { + config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); + Http::fake(['discord.test/*' => Http::response('', 204)]); + ReportSummaryAgent::fake(fn () => throw new RuntimeException('provider down')); + + experimentUser(SubscriptionExperiment::CONTROL, CarbonImmutable::parse('2026-06-05'), [ + 'status' => 'active', + 'at' => CarbonImmutable::parse('2026-06-05'), + ]); + + artisan('stats:experiment-funnel')->assertSuccessful(); + + Http::assertSent(fn ($request) => str_starts_with($request['embeds'][0]['description'], '```') + && str_contains($request['embeds'][0]['description'], 'Variante')); }); it('does not post when the experiment has not started', function () { @@ -405,6 +468,7 @@ it('does not post when the experiment has not started', function () { artisan('stats:experiment-funnel')->assertSuccessful(); + ReportSummaryAgent::assertNeverPrompted(); Http::assertNothingSent(); }); diff --git a/tests/Feature/SendSubscriptionFunnelReportCommandTest.php b/tests/Feature/SendSubscriptionFunnelReportCommandTest.php index a11f8654..172ecf43 100644 --- a/tests/Feature/SendSubscriptionFunnelReportCommandTest.php +++ b/tests/Feature/SendSubscriptionFunnelReportCommandTest.php @@ -1,9 +1,11 @@ true]); config(['ai_suggestions.report.excluded_emails' => []]); + ReportSummaryAgent::fake(['Los registros bajan respecto a la semana anterior.']); }); it('skips the report and hits no external service when subscriptions are disabled', function () { @@ -150,19 +153,66 @@ it('posts the funnel embed to the configured discord webhook', function () { artisan('stats:subscription-funnel')->assertSuccessful(); Http::assertSent(function ($request) { + $embed = $request['embeds'][0]; + return $request->url() === 'https://discord.test/hook' - && isset($request['embeds'][0]['title']) - && str_contains($request['embeds'][0]['title'], 'Subscription Funnel'); + && str_contains($embed['title'], 'Embudo de suscripción') + // Spanish table header and legend, no leftover English. + && str_contains($embed['description'], 'Semana') + && collect($embed['fields'])->contains(fn ($field) => $field['name'] === 'Leyenda'); }); }); +it('opens the embed with the AI summary, above the table', function () { + config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); + Http::fake(['discord.test/*' => Http::response('', 204)]); + ReportSummaryAgent::fake(['Los pagos suben respecto a la semana anterior.']); + + funnelUser(funnelNow()->subWeeks(10), ['status' => 'active', 'at' => funnelNow()->subWeeks(10)->addDays(2)]); + + artisan('stats:subscription-funnel')->assertSuccessful(); + + Http::assertSent(fn ($request) => str_starts_with( + $request['embeds'][0]['description'], + "Los pagos suben respecto a la semana anterior.\n\n```", + )); +}); + +it('still posts the report when the AI summary fails', function () { + config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); + Http::fake(['discord.test/*' => Http::response('', 204)]); + ReportSummaryAgent::fake(fn () => throw new RuntimeException('provider down')); + + funnelUser(funnelNow()->subWeeks(10), ['status' => 'active', 'at' => funnelNow()->subWeeks(10)->addDays(2)]); + + artisan('stats:subscription-funnel')->assertSuccessful(); + + Http::assertSent(fn ($request) => str_starts_with($request['embeds'][0]['description'], '```') + && str_contains($request['embeds'][0]['description'], 'Semana')); +}); + it('prints to the console without posting when --no-discord is set', function () { config(['services.discord.ai_cohort_webhook_url' => 'https://discord.test/hook']); Http::fake(['discord.test/*' => Http::response('', 204)]); funnelUser(funnelNow()->subWeeks(10), ['status' => 'active', 'at' => funnelNow()->subWeeks(10)->addDays(2)]); - artisan('stats:subscription-funnel', ['--no-discord' => true])->assertSuccessful(); + artisan('stats:subscription-funnel', ['--no-discord' => true]) + ->expectsOutputToContain('Los registros bajan respecto a la semana anterior.') + ->assertSuccessful(); Http::assertNothingSent(); }); + +it('does not overwrite the comparison baseline on a --no-discord run', function () { + Http::fake(); + Cache::put('report_summary_baseline:subscription-funnel', ['weeks' => ['sentinel']], now()->addDay()); + + artisan('stats:subscription-funnel', ['--no-discord' => true])->assertSuccessful(); + + expect(Cache::get('report_summary_baseline:subscription-funnel'))->toBe(['weeks' => ['sentinel']]); + + artisan('stats:subscription-funnel')->assertSuccessful(); + + expect(Cache::get('report_summary_baseline:subscription-funnel'))->not->toBe(['weeks' => ['sentinel']]); +});