From 44e2b8d6db8d214a705a59de5e61c63638a52c20 Mon Sep 17 00:00:00 2001 From: Chris Walton Date: Fri, 17 Apr 2026 11:22:16 -0400 Subject: [PATCH] fix: office-hours marks sessions "success" without writing design doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The office-hours skill can complete the final telemetry block and log `outcome:success` even when Phase 5 (design doc write) was skipped — e.g., when a session ends early or the conversation pivots to implementation. The timeline shows `event:completed` with no corresponding `*-design-*.md` file on disk, silently losing the artifact that downstream skills (/plan-ceo-review, /plan-eng-review) depend on. Fix: add a "Pre-Telemetry: Verify Design Doc Artifact" section that runs before the preamble's telemetry block. It re-derives SLUG (since shell state doesn't persist between Bash tool calls), uses the preamble's `~/.gstack/sessions/$PPID` touch file as a session-start time reference, and looks for a fresh design doc written after that point. Emits `ARTIFACT_STATUS: ok` or `no_doc`. The Important Rules section now instructs Claude to use this result to drive OUTCOME — never log `success` when `no_doc`. Repro: observed in apex-coach session 2026-04-12 — timeline logged `duration_s:"0"`, `session:"unknown"`, `outcome:"success"`, but no design doc exists in ~/.gstack/projects/apex-coach/. Co-Authored-By: Claude Opus 4.7 (1M context) --- office-hours/SKILL.md | 28 ++++++++++++++++++++++++++++ office-hours/SKILL.md.tmpl | 28 ++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/office-hours/SKILL.md b/office-hours/SKILL.md index 699e4a58b..2bc47895e 100644 --- a/office-hours/SKILL.md +++ b/office-hours/SKILL.md @@ -1815,6 +1815,33 @@ The design doc at `~/.gstack/projects/` is automatically discoverable by downstr --- +## Pre-Telemetry: Verify Design Doc Artifact + +**PLAN MODE EXCEPTION — ALWAYS RUN.** This is the last step before the preamble's `## Telemetry (run last)` block. It determines the correct OUTCOME value by verifying that this session actually produced a design doc. This prevents silent-completion failures where the skill logs `outcome:success` but no artifact persists to disk. + +```bash +eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" 2>/dev/null || true +_SESS_REF=$(ls -t ~/.gstack/sessions/ 2>/dev/null | head -1) +[ -n "$_SESS_REF" ] && _SESS_REF="$HOME/.gstack/sessions/$_SESS_REF" +_FRESH_DOC="" +if [ -n "$_SESS_REF" ] && [ -d ~/.gstack/projects/"${SLUG:-unknown}" ]; then + _FRESH_DOC=$(find ~/.gstack/projects/"${SLUG:-unknown}" -maxdepth 1 -name '*-design-*.md' -newer "$_SESS_REF" 2>/dev/null | head -1) +fi +if [ -n "$_FRESH_DOC" ]; then + echo "ARTIFACT_STATUS: ok" + echo "ARTIFACT_PATH: $_FRESH_DOC" +else + echo "ARTIFACT_STATUS: no_doc" +fi +``` + +**Use this result to set OUTCOME in the telemetry block:** + +- If `ARTIFACT_STATUS: ok` — the user saw a design doc. OUTCOME follows the normal rules (`success` if the user approved in Phase 5 review, `error` on failure, `abort` on user interrupt). +- If `ARTIFACT_STATUS: no_doc` — the session ended without producing the artifact. OUTCOME **must** be `no_doc`, not `success`. This is not a successful office hours session even if the conversation felt productive. Downstream analytics depend on this distinction to catch Phase 5 skips. + +--- + ## Capture Learnings If you discovered a non-obvious pattern, pitfall, or architectural insight during @@ -1846,6 +1873,7 @@ already knows. A good test: would this insight save time in a future session? If - **Questions ONE AT A TIME.** Never batch multiple questions into one AskUserQuestion. - **The assignment is mandatory.** Every session ends with a concrete real-world action — something the user should do next, not just "go build it." - **If user provides a fully formed plan:** skip Phase 2 (questioning) but still run Phase 3 (Premise Challenge) and Phase 4 (Alternatives). Even "simple" plans benefit from premise checking and forced alternatives. +- **The design doc file is the artifact of this skill.** Before running the preamble's `## Telemetry (run last)` block, run the `## Pre-Telemetry: Verify Design Doc Artifact` check above and use its result to set OUTCOME. Never log `outcome:success` when no design doc was written — use `no_doc` instead. A session without a persisted doc is not a successful session, regardless of how the conversation felt. - **Completion status:** - DONE — design doc APPROVED - DONE_WITH_CONCERNS — design doc approved but with open questions listed diff --git a/office-hours/SKILL.md.tmpl b/office-hours/SKILL.md.tmpl index afe063c93..4b48a7f03 100644 --- a/office-hours/SKILL.md.tmpl +++ b/office-hours/SKILL.md.tmpl @@ -880,6 +880,33 @@ The design doc at `~/.gstack/projects/` is automatically discoverable by downstr --- +## Pre-Telemetry: Verify Design Doc Artifact + +**PLAN MODE EXCEPTION — ALWAYS RUN.** This is the last step before the preamble's `## Telemetry (run last)` block. It determines the correct OUTCOME value by verifying that this session actually produced a design doc. This prevents silent-completion failures where the skill logs `outcome:success` but no artifact persists to disk. + +```bash +eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" 2>/dev/null || true +_SESS_REF=$(ls -t ~/.gstack/sessions/ 2>/dev/null | head -1) +[ -n "$_SESS_REF" ] && _SESS_REF="$HOME/.gstack/sessions/$_SESS_REF" +_FRESH_DOC="" +if [ -n "$_SESS_REF" ] && [ -d ~/.gstack/projects/"${SLUG:-unknown}" ]; then + _FRESH_DOC=$(find ~/.gstack/projects/"${SLUG:-unknown}" -maxdepth 1 -name '*-design-*.md' -newer "$_SESS_REF" 2>/dev/null | head -1) +fi +if [ -n "$_FRESH_DOC" ]; then + echo "ARTIFACT_STATUS: ok" + echo "ARTIFACT_PATH: $_FRESH_DOC" +else + echo "ARTIFACT_STATUS: no_doc" +fi +``` + +**Use this result to set OUTCOME in the telemetry block:** + +- If `ARTIFACT_STATUS: ok` — the user saw a design doc. OUTCOME follows the normal rules (`success` if the user approved in Phase 5 review, `error` on failure, `abort` on user interrupt). +- If `ARTIFACT_STATUS: no_doc` — the session ended without producing the artifact. OUTCOME **must** be `no_doc`, not `success`. This is not a successful office hours session even if the conversation felt productive. Downstream analytics depend on this distinction to catch Phase 5 skips. + +--- + {{LEARNINGS_LOG}} ## Important Rules @@ -888,6 +915,7 @@ The design doc at `~/.gstack/projects/` is automatically discoverable by downstr - **Questions ONE AT A TIME.** Never batch multiple questions into one AskUserQuestion. - **The assignment is mandatory.** Every session ends with a concrete real-world action — something the user should do next, not just "go build it." - **If user provides a fully formed plan:** skip Phase 2 (questioning) but still run Phase 3 (Premise Challenge) and Phase 4 (Alternatives). Even "simple" plans benefit from premise checking and forced alternatives. +- **The design doc file is the artifact of this skill.** Before running the preamble's `## Telemetry (run last)` block, run the `## Pre-Telemetry: Verify Design Doc Artifact` check above and use its result to set OUTCOME. Never log `outcome:success` when no design doc was written — use `no_doc` instead. A session without a persisted doc is not a successful session, regardless of how the conversation felt. - **Completion status:** - DONE — design doc APPROVED - DONE_WITH_CONCERNS — design doc approved but with open questions listed