fix(canary): require human smoke confirmation on the all-green path

Same root cause as land-and-deploy's Step 7a: canary's alert path already
routes through AskUserQuestion (Phase 5), but the most common outcome —
nothing broke across the whole monitoring window — never puts the live page
in front of a human. Phase 6 prints "VERDICT: DEPLOY IS HEALTHY" straight
from automated checks, and Phase 7 offers to promote that unconfirmed state
into the new baseline other runs compare against.

Phase 6 now opens the live URL and asks for an explicit pass/fail after
monitoring completes, distinguishing `Human smoke: CONFIRMED` from
`Human smoke: AWAITING` in the report and JSONL log — matching the verdict
vocabulary land-and-deploy's Step 9 already uses. Phase 7's baseline-update
offer is gated on CONFIRMED so an unconfirmed all-clear can't quietly become
the new reference point.

Part of the same fix wave as #2580 (land-and-deploy). Scoped this one to
canary only after auditing the other ~50 skills for the same pattern
(automated verdict of HEALTHY/VERIFIED with no human-confirmation gate on
the quiet all-green path) — qa, design-review, ios-design-review, ios-qa,
and benchmark all already report per-item evidence (verified/best-effort/
reverted/deferred, or objective metrics) rather than a global "it works in
prod" claim, so they don't share this specific gap.

Tested: bun run gen:skill-docs --dry-run reports FRESH, exit 0. Targeted
bun test (skill-parser, skill-validation, gen-skill-docs, skill-size-budget):
757 pass / 1 fail, the fail is the same pre-existing package.json/VERSION
mismatch already on main, unrelated to this diff.
This commit is contained in:
Elder Miranda de Andrade 2026-08-15 18:47:47 +02:00
parent b018908c6c
commit 6b54432892
2 changed files with 56 additions and 6 deletions

View File

@ -1023,10 +1023,30 @@ Per-Page Results:
Alerts Fired: [N] (X critical, Y high, Z medium)
Screenshots: .gstack/canary-reports/screenshots/
Human smoke: [CONFIRMED (user said pass) / AWAITING (screen opened, no reply yet)]
VERDICT: [DEPLOY IS HEALTHY / DEPLOY HAS ISSUES — details above]
```
**A HEALTHY status from Phases 1-5 is the automated read, not the final word.**
Every alert path already routes through AskUserQuestion (Phase 5) — but the
quiet, all-green path never puts anything in front of the user, which means
the most common outcome (nothing broke) is also the one nobody ever actually
looks at. Before finalizing the report:
```bash
open "<url>" 2>/dev/null || xdg-open "<url>" 2>/dev/null || echo "Couldn't auto-open — here's the URL: <url>"
```
Tell the user: "Automated monitoring found nothing wrong across {N} checks over
{duration} — but that's a script's opinion, not a look. I've opened the live
page. Take a glance, then tell me pass or fail so I can close this out as
actually verified." If the user confirms, mark `Human smoke: CONFIRMED` in the
report. If this is a non-interactive/spawned session, or the user doesn't
reply before you need to finish the report, write `Human smoke: AWAITING` and
say so plainly — do not silently treat "no alerts fired" as "a human looked
and it's fine."
Save report to `.gstack/canary-reports/{date}-canary.md` and `.gstack/canary-reports/{date}-canary.json`.
Log the result for the review dashboard:
@ -1036,17 +1056,22 @@ eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)"
mkdir -p ~/.gstack/projects/$SLUG
```
Write a JSONL entry: `{"skill":"canary","timestamp":"<ISO>","status":"<HEALTHY/DEGRADED/BROKEN>","url":"<url>","duration_min":<N>,"alerts":<N>}`
Write a JSONL entry: `{"skill":"canary","timestamp":"<ISO>","status":"<HEALTHY/DEGRADED/BROKEN>","human_smoke":"<CONFIRMED/AWAITING>","url":"<url>","duration_min":<N>,"alerts":<N>}`
### Phase 7: Baseline Update
If the deploy is healthy, offer to update the baseline:
If the deploy is healthy AND `Human smoke: CONFIRMED`, offer to update the baseline:
- **Context:** Canary monitoring completed. The deploy is healthy.
- **Context:** Canary monitoring completed. The deploy is healthy and you confirmed it looks right.
- **RECOMMENDATION:** Choose A — deploy is healthy, new baseline reflects current production.
- A) Update baseline with current screenshots
- B) Keep old baseline
If the deploy is healthy but `Human smoke: AWAITING` (nobody confirmed yet), skip
this offer — don't bake an unconfirmed state into the baseline other runs will
compare against. Mention it once: "Once you've had a look, run `/canary <url>
--baseline` if you want this state as the new reference point."
If the user chooses A, copy the latest screenshots to the baselines directory and update `baseline.json`.
## Important Rules

View File

@ -188,10 +188,30 @@ Per-Page Results:
Alerts Fired: [N] (X critical, Y high, Z medium)
Screenshots: .gstack/canary-reports/screenshots/
Human smoke: [CONFIRMED (user said pass) / AWAITING (screen opened, no reply yet)]
VERDICT: [DEPLOY IS HEALTHY / DEPLOY HAS ISSUES — details above]
```
**A HEALTHY status from Phases 1-5 is the automated read, not the final word.**
Every alert path already routes through AskUserQuestion (Phase 5) — but the
quiet, all-green path never puts anything in front of the user, which means
the most common outcome (nothing broke) is also the one nobody ever actually
looks at. Before finalizing the report:
```bash
open "<url>" 2>/dev/null || xdg-open "<url>" 2>/dev/null || echo "Couldn't auto-open — here's the URL: <url>"
```
Tell the user: "Automated monitoring found nothing wrong across {N} checks over
{duration} — but that's a script's opinion, not a look. I've opened the live
page. Take a glance, then tell me pass or fail so I can close this out as
actually verified." If the user confirms, mark `Human smoke: CONFIRMED` in the
report. If this is a non-interactive/spawned session, or the user doesn't
reply before you need to finish the report, write `Human smoke: AWAITING` and
say so plainly — do not silently treat "no alerts fired" as "a human looked
and it's fine."
Save report to `.gstack/canary-reports/{date}-canary.md` and `.gstack/canary-reports/{date}-canary.json`.
Log the result for the review dashboard:
@ -201,17 +221,22 @@ Log the result for the review dashboard:
mkdir -p ~/.gstack/projects/$SLUG
```
Write a JSONL entry: `{"skill":"canary","timestamp":"<ISO>","status":"<HEALTHY/DEGRADED/BROKEN>","url":"<url>","duration_min":<N>,"alerts":<N>}`
Write a JSONL entry: `{"skill":"canary","timestamp":"<ISO>","status":"<HEALTHY/DEGRADED/BROKEN>","human_smoke":"<CONFIRMED/AWAITING>","url":"<url>","duration_min":<N>,"alerts":<N>}`
### Phase 7: Baseline Update
If the deploy is healthy, offer to update the baseline:
If the deploy is healthy AND `Human smoke: CONFIRMED`, offer to update the baseline:
- **Context:** Canary monitoring completed. The deploy is healthy.
- **Context:** Canary monitoring completed. The deploy is healthy and you confirmed it looks right.
- **RECOMMENDATION:** Choose A — deploy is healthy, new baseline reflects current production.
- A) Update baseline with current screenshots
- B) Keep old baseline
If the deploy is healthy but `Human smoke: AWAITING` (nobody confirmed yet), skip
this offer — don't bake an unconfirmed state into the baseline other runs will
compare against. Mention it once: "Once you've had a look, run `/canary <url>
--baseline` if you want this state as the new reference point."
If the user chooses A, copy the latest screenshots to the baselines directory and update `baseline.json`.
## Important Rules