From 832a148e55e9e07496814a0d8cbb7da420f553c1 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 12 Jun 2026 06:44:19 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20P1=20TODO=20=E2=80=94=20eval=20harness?= =?UTF-8?q?=20live=20progress=20+=20incremental=20persistence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root-caused during this ship: a killed eval run was indistinguishable from a healthy one for hours (per-file output buffering across mega test files, no incremental eval-store writes, no honest liveness signal). Full context and starting points in the entry. Co-Authored-By: Claude Fable 5 --- TODOS.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/TODOS.md b/TODOS.md index df510e032..183d11118 100644 --- a/TODOS.md +++ b/TODOS.md @@ -45,6 +45,47 @@ a silent mistake breaks all 52 skills. High blast radius — needs its own focus ## Test infrastructure +### Eval harness: live progress + incremental result persistence (kill the silent hour) + +**Priority:** P1 + +**What:** `bun run test:evals` is observably silent for its entire runtime and +persists nothing until completion. Make the E2E harness (1) append a one-line +progress record per test START and END to a well-known heartbeat file (e.g. +`~/.gstack-dev/evals/.current-run.jsonl`), (2) write each test's eval-store +result incrementally instead of only at run end, and (3) flush per-test +pass/fail lines to stderr unbuffered so `bun test --concurrent` mega-file +buffering can't hide 50 minutes of legitimate progress. + +**Why:** During the v1.57.11.0 ship, the diff-selected eval run (54 tests) was +killed ~50 min in and NOTHING distinguished the corpse from a healthy run for +hours: the log had zero test lines (per-file buffering across five mega +`skill-e2e-*.test.ts` files), `~/.gstack-dev/evals/` had zero new files +(results persist only on completion), and the only available liveness signal +(`pgrep "bun test --max-concurrency"`) false-positives on every sibling +free-suite shard. An agent or human watching the run has no honest signal. + +**Pros:** Dead runs detected in minutes instead of hours; partial results +survive kills (a 50-min run that dies at test 40/54 keeps 40 results and can +resume); `eval:watch` gets a real data source. + +**Cons:** Touches `test/helpers/session-runner.ts` + `eval-store.ts` (global +touchfiles — change triggers ALL eval tests on the next diff-selected run); +incremental writes need a PARTIAL marker so `eval:compare` doesn't treat a +dead run as a complete baseline. + +**Context:** Root-caused 2026-06-12 during the v1.57.11.0 /ship. The run +itself was on pace (~50 min for 54 E2E tests at concurrency 15 is nominal); +the failure was pure observability. Related: the existing +`project_e2e_harness_observability` note (stream-json reasoning + tool traces +dropped on failure — same module, fix together). Start in +`test/helpers/session-runner.ts` (per-test lifecycle) and +`test/helpers/eval-store.ts` (persistence timing). + +**Depends on / blocked by:** Nothing. Classify the new behavior under the +existing two-tier system; the heartbeat file must be safe under +`--concurrent` (append-only, one JSON line per event). + ### ✅ DONE (v1.53.1.0): Rebaseline parity-suite (v1.44.1 → v1.53.0.0) **What:** `test/parity-suite.test.ts` checked every skill's SKILL.md size against