docs: P1 TODO — eval harness live progress + incremental persistence

Root-caused during this ship: a killed eval run was indistinguishable from a
healthy one for hours (per-file output buffering across mega test files, no
incremental eval-store writes, no honest liveness signal). Full context and
starting points in the entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan 2026-06-12 06:44:19 -07:00
parent 4e560d4aba
commit 832a148e55
No known key found for this signature in database
GPG Key ID: C1F69E85C74EFE1D
1 changed files with 41 additions and 0 deletions

View File

@ -45,6 +45,47 @@ a silent mistake breaks all 52 skills. High blast radius — needs its own focus
## Test infrastructure
### Eval harness: live progress + incremental result persistence (kill the silent hour)
**Priority:** P1
**What:** `bun run test:evals` is observably silent for its entire runtime and
persists nothing until completion. Make the E2E harness (1) append a one-line
progress record per test START and END to a well-known heartbeat file (e.g.
`~/.gstack-dev/evals/.current-run.jsonl`), (2) write each test's eval-store
result incrementally instead of only at run end, and (3) flush per-test
pass/fail lines to stderr unbuffered so `bun test --concurrent` mega-file
buffering can't hide 50 minutes of legitimate progress.
**Why:** During the v1.57.11.0 ship, the diff-selected eval run (54 tests) was
killed ~50 min in and NOTHING distinguished the corpse from a healthy run for
hours: the log had zero test lines (per-file buffering across five mega
`skill-e2e-*.test.ts` files), `~/.gstack-dev/evals/` had zero new files
(results persist only on completion), and the only available liveness signal
(`pgrep "bun test --max-concurrency"`) false-positives on every sibling
free-suite shard. An agent or human watching the run has no honest signal.
**Pros:** Dead runs detected in minutes instead of hours; partial results
survive kills (a 50-min run that dies at test 40/54 keeps 40 results and can
resume); `eval:watch` gets a real data source.
**Cons:** Touches `test/helpers/session-runner.ts` + `eval-store.ts` (global
touchfiles — change triggers ALL eval tests on the next diff-selected run);
incremental writes need a PARTIAL marker so `eval:compare` doesn't treat a
dead run as a complete baseline.
**Context:** Root-caused 2026-06-12 during the v1.57.11.0 /ship. The run
itself was on pace (~50 min for 54 E2E tests at concurrency 15 is nominal);
the failure was pure observability. Related: the existing
`project_e2e_harness_observability` note (stream-json reasoning + tool traces
dropped on failure — same module, fix together). Start in
`test/helpers/session-runner.ts` (per-test lifecycle) and
`test/helpers/eval-store.ts` (persistence timing).
**Depends on / blocked by:** Nothing. Classify the new behavior under the
existing two-tier system; the heartbeat file must be safe under
`--concurrent` (append-only, one JSON line per event).
### ✅ DONE (v1.53.1.0): Rebaseline parity-suite (v1.44.1 → v1.53.0.0)
**What:** `test/parity-suite.test.ts` checked every skill's SKILL.md size against