From 7726d25396bbd9ff3ae622b55869788a39ab85ee Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 15 Aug 2026 17:03:57 -0700 Subject: [PATCH] chore: bump version and changelog (v1.66.0.0) Test/evals/CI speedup pass: release summary + itemized changes in CHANGELOG.md; TODOS.md marks the free-suite exit-code P1 complete and files the review-army follow-ups. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 63 ++++++++++++++++++ TODOS.md | 182 ++++++++++++++++++++++++++++++++++++++++++--------- VERSION | 2 +- package.json | 2 +- 4 files changed, 216 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ec29ae520..2e353ad07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,68 @@ # Changelog +## [1.66.0.0] - 2026-08-15 + +**The full ~7,000-test suite in about 90 seconds, verified honest.** +**Paid evals now bill by diff, not $38 flat.** + +`bun run test` used to take 454 seconds. It now runs as six concurrent shard processes and finishes in about 90 to 100 seconds, under a strict output contract: a shard that exits without bun's own terminal summary line is a failure, a wedged shard is killed at a size-scaled deadline and named in the epilogue, and the console shows only what you need (per-shard status, then `✗ file — test name` for anything red, full stream in a per-run log, `--verbose` for the firehose). Twelve test files that ran under no script and no CI are wired in. A 3,372-line dead eval monolith is deleted, with four never-run tests revived out of it. + +Paid evals select by diff. Edit one skill and the runner executes only the shards your change touches, reports the rest as skipped-by-diff, and prints the reason. Selection sees uncommitted and untracked work, fails closed with a named cause on git errors, and an edit to the selection data itself re-runs only the changed keys instead of forcing the full suite. + +### The numbers that matter + +Measured on this branch. Re-run with `time bun run test` and `bun run eval:select`; eval receipts live in `~/.gstack-dev/evals/`. + +| Metric | Before | After | Δ | +|---|---|---|---| +| Free suite wall clock (~7,000 tests) | 454s | ~90-100s, strict-verified | ~4.7x | +| Free-test files with Linux CI coverage | 0 | ~420, as a required PR check | new | +| Paid cost of a one-skill edit | ~$38 (full suite) | 4 of 45 shards, $0.67 | ~57x | +| Slowest CI eval job | 741s, one serial file | three jobs, each under ~250s | ~3x | +| Paid retry amplification | `--retry 2`, +84% measured | `--retry 1` | half | + +The $0.67 row is a live rehearsal, not a projection: a scratch edit to `qa/SKILL.md.tmpl` selected 17 of 177 tests, ran 4 of 45 shards, skipped 41 by diff, and the /qa E2E passed. + +### What this means for contributors + +Runs you used to schedule around now fit inside a thought. `bun run test` before every commit is a real habit again at ~90 seconds, red names the exact test, and green means every file actually ran. Fork PRs get true test signal from the new secretless Linux lane. Ship a change and the eval bill tracks your blast radius. + +### Itemized changes + +### Added +- Linux free-tests CI lane (`.github/workflows/free-tests.yml`): the whole free suite on every PR and every push to main, required from day one, zero secrets, least-privilege token, failure logs uploaded as an artifact, wiring pinned by `test/free-tests-workflow-wiring.test.ts`. +- Diff-based paid-shard selection: parent-side skipping with a `skipped-by-diff` taxonomy and a selection banner naming the reason (`scripts/test-paid-shards.ts`). +- Map-diff selection for the selection data itself: editing `test/helpers/touchfiles-data.ts` re-runs only added/changed/retiered keys (old version evaluated via `git show` + a bun child; adversarial fixtures in `test/touchfiles-map-diff.test.ts`). +- Selection unions committed, staged/unstaged, and untracked changes; git failures throw naming `EVALS_ALL=1` (fail closed), and non-ASCII filenames select correctly (`core.quotePath=false`). +- `test/helpers/skill-fixture.ts`: E2E fixtures extract the SKILL.md sections a test needs instead of copying 1,800-line files — nine fixture sites cut 58-97%. +- `GSTACK_EVAL_MODEL_JUDGE` env override for the LLM-judge model; eval model resolution centralized in `lib/eval-model.ts` with per-kind `GSTACK_EVAL_MODEL_` overrides. + +### Changed +- Free suite architecture: N concurrent shard processes (serial within each); tree-mutating tests and tree-measuring ratchet readers run in one serial shard after the parallel phase, so measurements never race regeneration. Shard curation lists are pinned against the live file census, and wall deadlines scale with shard size. +- Agent SDK capture default Opus → Sonnet (D1a). The judge default stays Sonnet: a live A/B on the health rubric scored Haiku 2/2/2 against Sonnet's 4/3/4, so the downgrade was pinned back per D1a's regressor clause (receipts in `test/helpers/llm-judge.ts`). +- Four expensive posture tests demoted gate → periodic (D2a). +- Paid runners: `EVALS_JOBS` (shard process count) split from `EVALS_CONCURRENCY` (within-shard), `--retry 1` on every paid path, one preflight API ping per run instead of ~30, detach timeouts floor-enforced against the live shard census by `test/eval-detach-timeout-floor.test.ts`. +- CI: eval Docker image cache keyed on Dockerfile + bun.lock so version bumps stop rebuilding it; Bun 1.3.13 in the image; `skill-e2e-review` split into three matrix shards; actionlint runs a digest-pinned prebuilt image; five single-core jobs right-sized; lint and skill-docs stop double-running every PR commit; the Windows lane caches bun installs and runs the curated suite instead of a hand list. +- Skill-routing E2E fixture installs skill heads, not ~18 full SKILL.md files. + +### Fixed +- Ctrl-C actually cancels a run: the signal forwarders now schedule the parent runner's own exit and both shard pools stop launching new work on `SIGINT`/`SIGTERM` — previously the parent killed the current child and kept spawning API-burning shards. +- The intermittent whole-suite wedge: `browse/src/browser-manager.ts` `close()` captures the Chromium child before the close race and SIGKILLs it when graceful close times out, with unit coverage of the fallback. +- The strict-output classifier keeps stdout and stderr line assembly separate, so interleaved pipe chunks cannot hide a failure line or fake a truncation. Windows shard kills take the whole process tree (`taskkill /T`) instead of orphaning grandchildren. +- Redaction calibration: `${var}` template interpolations and ALL-CAPS `USER:PASSWORD` doc placeholders no longer block pushes, while a bare `$word` password and a literal lowercase `password`/`pass` at the URL-password position still do; the two connection-string validators share one helper so they cannot drift. +- Supabase pooler DSNs percent-encode the password segment, `wait --timeout` rejects non-numeric values instead of polling forever, response-body read failures retry as transport errors, and the CLI entrypoint lets stdout drain before exiting. +- The paid-suite preflight fails fast on a missing `claude` binary, a spawn error, or a timeout — outages surface once in the parent instead of once per shard. +- Same-name branches from different forks can no longer cancel each other's CI runs (concurrency groups key on PR number across the free, eval, and Windows lanes). +- Selection integrity: the `touchfiles.ts` facade, `e2e-helpers.ts`, and `paid-test-set.ts` are global touchfiles (an edit to selection-path code can never select zero tests); duplicate touchfiles keys fail the suite; rehomed E2E files list themselves in their own dependency maps; retro E2E passes require the report on disk. +- The intermittent context-save-list eval test that had never passed in 26 recorded runs now passes. +- `variants-retry-after` HTTP-date flake; watchdog E2E 22.7s → 1.5s; supabase-provision tests 16.5s → 0.45s via an in-process TS port. +- `package.json` version drift against VERSION. + +### For contributors +- `test:gate:sharded` / `test:periodic:sharded` run tiers through the sharded paid runner; `eval:bg:*` wrap runs in `gstack-detach` with a per-tier watchdog and the machine-wide `gstack-evals` lock. +- Five pre-existing environment failures quarantined individually with in-file receipts; two dead-architecture security contract tests deleted. +- `test/e2e-tier-alignment.test.ts` enforces tier declarations and fails fatally when a sharded-runner mapper cannot see a gate file. + ## [1.65.0.0] - 2026-08-14 **/autoplan, /codex on macOS, and memory ingest work again.** diff --git a/TODOS.md b/TODOS.md index 62b3b0266..874c08817 100644 --- a/TODOS.md +++ b/TODOS.md @@ -174,37 +174,6 @@ Trigger condition documented in `lib/gbrain-sources.ts` at the drift log line. **Effort:** M (human ~1d, CC ~45min). **Depends on:** drift-log evidence from the wave's `ensureSourceRegistered` logging. -### P1: Free suite exit code is untrustworthy — in-process force-exits mask failures - -**Priority:** P1 - -**What:** At least five browse test files end with `setTimeout(() => process.exit(0), 500)` -(browse/test/commands.test.ts:101, snapshot.test.ts:36, batch.test.ts:47, -handoff.test.ts:31, content-security.test.ts:465). The timer fires inside the SHARED -`bun test` process, exiting 0 before bun prints its final summary — so `bun test` can -report exit 0 while real test failures scrolled by earlier. Remove the force-exits and -fix the underlying handle leaks they paper over (lingering Playwright/daemon handles -that once made the suite hang), or scope the exit to a spawned child process. - -**Why:** Observed 2026-08-07: three genuinely failing tests (eval-list-cli, -benchmark-cli, observability check 11) rode green `bun test` exit codes across -multiple runs; the failures only surfaced by grepping logs for "(fail)" lines. A test -suite that exits 0 on failure is worse than no suite — it manufactures false -confidence at commit time and in any CI job that trusts the exit code. - -**Pros:** Restores the one contract everything (CI, /ship, humans) relies on: exit -code == truth. Also un-hides the missing final summary block. -**Cons:** The force-exits exist because the suite once hung on leaked handles; -removing them without fixing the leaks trades silent failure for hangs. Needs a -focused pass: find each leaked handle (daemon children, PTY, Playwright contexts), -close them in afterAll, then delete the exits one file at a time. - -**Context / where to start:** `grep -rn "process.exit(0)" browse/test/` — the -setTimeout variants are the offenders (server-no-import-side-effects.test.ts:62 is a -spawned-child probe, fine). Repro: run the full free suite and note the log ends at -the browse files with no "Ran N tests" summary. Receipts: -~/.gstack-dev/logs/free-suite-main-check.log (3 masked fails, exit 0). - ### P2: Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract **Priority:** P2 @@ -2332,8 +2301,159 @@ Shipped in v0.6.5. TemplateContext in gen-skill-docs.ts bakes skill name into pr **Depends on:** v1.47.0.0 ships; gather real false-negative data from the v1 string matcher. +## Test/evals/CI speedup follow-ups (filed v1.66.0.0 via /ship review army) + +### P2: Free-suite shard balancing — LPT by recorded durations instead of stable hash + +**What:** Full-suite shard assignment is a stable hash; measured shard durations +spread 69.5s-168.5s (max 2.4x min), so ~35-40s of every run is idle tail. Local +full-suite mode doesn't need deterministic indices (only the CI --shards matrix +does) — bin-pack by recorded per-file durations (bun prints them in the logs the +runner already captures), keep assignFilesToShards untouched for --shard mode. +**Where:** scripts/test-free-shards.ts main() full-suite path. +**Effort:** S (human ~4h, CC ~20min). + +### P2: Propagate parent eval selection to shard children (EVALS_SELECTION_JSON) + +**What:** The sharded paid runner computes selection once in the parent, but each +shard child re-derives it at e2e-helpers module load (git spawns per shard; plus a +bun child evaluating the old touchfiles-data when map-diff is active). Serialize +the parent's selection into the child env and honor it in computeDiffSelection, +keeping child self-derivation for non-sharded entrypoints. Add a parent/child +selection drift test (same fixture through computePaidDiffSelection and +computeDiffSelection) while there. +**Where:** scripts/test-paid-shards.ts runPaidShards env block; test/helpers/e2e-helpers.ts. +**Effort:** S (human ~4h, CC ~20min). + +### P2: evals.yml matrix census tripwire — gate files must appear in the CI matrix + +**What:** The branch's headline incident (two rehomed gate files silently never ran +for 48 versions because the monolith's filename missed the hand-listed evals.yml +matrix) has no tripwire binding gate-tier skill-e2e files to the matrix. +e2e-tier-alignment covers the LOCAL sharded runner's mapper; the CI matrix can +still drift. Parse the workflow YAML in a free test and diff against E2E_TIERS +gate files (curated exclude list for deliberately-manual files). +**Where:** new test beside test/e2e-tier-alignment.test.ts; .github/workflows/evals.yml. +**Effort:** S (human ~3h, CC ~15min). + +### P2: E2E dep-list self-registration sweep — 129 of 177 keys omit their own test file + +**What:** Editing only a test's assertions/prompt selects nothing for most keys +(the adversarial review measured 129/177), and parent-side shard skipping makes +the hole cheaper to hit. This branch fixed the rehomed files' keys; sweep the +rest mechanically (each key's dep list appends the file that declares it) and +upgrade e2e-tier-alignment's report-only mode to enforce self-registration. +**Where:** test/helpers/touchfiles-data.ts; test/e2e-tier-alignment.test.ts. +**Effort:** S (human ~3h, CC ~15min). + +### P3: Paid runner spools non-live shard output to disk instead of RAM + +**What:** Non-live shards buffer their entire 30-min stream-json stdout+stderr in +memory (Buffer[]), x jobs concurrent shards. Spool to a temp file like the free +runner's per-run log. +**Where:** scripts/test-paid-shards.ts runPaidShard buffered path. +**Effort:** S (human ~2h, CC ~10min). + +### P3: Eval Docker image freshness tripwire + +**What:** The cache-key trio means the image rebuilds only when Dockerfile/bun.lock +change; freshness of the baked unpinned claude CLI now rides entirely on +ci-image.yml's cron. If the cron silently fails or is disabled, eval CI pins to an +ever-older CLI with no signal. Add an image-age check (fail the eval workflow when +the image tag's created date exceeds N days) or a cron-liveness alert. +**Where:** .github/workflows/ci-image.yml, evals.yml. +**Effort:** S (human ~2h, CC ~10min). + +### P3: Detach-floor self-check against runtime knobs (EVALS_JOBS) + +**What:** test/eval-detach-timeout-floor.test.ts computes the worst case from +constants; an operator exporting EVALS_JOBS=2 doubles the gate worst case past the +25,200s watchdog and healthy tail shards report never-started. Add a runtime +self-check in test-paid-shards main(): warn/fail when the computed worst case with +LIVE options exceeds a GSTACK_DETACH_TIMEOUT env exported by gstack-detach. +**Where:** scripts/test-paid-shards.ts; bin/gstack-detach. +**Effort:** S (human ~2h, CC ~10min). + +### P3: Eval store records the effective judge/capture model per run + +**What:** Model defaults moved (capture Opus→Sonnet) and GSTACK_EVAL_MODEL_JUDGE +can silently change graders; eval:compare deltas across a model boundary conflate +model swap with skill regressions. Record the resolved models in the eval-store +record and surface them in eval:compare. +**Where:** test/helpers/eval-store.ts, llm-judge.ts, eval-compare. +**Effort:** S (human ~2h, CC ~10min). + +### P3: SECURITY_BENCH periodic lane — classifier behavioral coverage runs nowhere + +**What:** Gating the live L4 classifier tests on SECURITY_BENCH=1 fixed local +suite speed but left the prompt-injection classifier with no scheduled lane. +Add SECURITY_BENCH=1 (with model-cache warmup, 112MB first run) to +evals-periodic.yml so behavioral coverage exists weekly. +**Where:** .github/workflows/evals-periodic.yml; browse/test/security-live-playwright.test.ts. +**Effort:** S (human ~2h, CC ~10min). + +### P3: Shared child-lifecycle helper for the two shard runners + +**What:** runFreeShard and runPaidShard duplicate ~35 lines of spawn/group-kill/ +wall-timer scaffold verbatim (and the ShardCommand type). Extract into +scripts/test-strict-output.ts, which already hosts the shared lifecycle +primitives, leaving stream policy per runner. +**Where:** scripts/test-free-shards.ts, scripts/test-paid-shards.ts. +**Effort:** S (human ~3h, CC ~15min). + +### P3: DI-refactor gstack-gbrain-detect-mcp-mode test (~40s spawn cost, absorbed but real) + +**What:** Plan item 5 of the v1.66.0.0 pass, deferred: the test spawns the real +binary repeatedly. Refactor to import the module with a DI-injected exec seam +(never env-set-before-import), keep 1-2 spawn smokes. Cost is currently absorbed +by shard parallelism; the per-file wall cost remains. +**Where:** test/gstack-gbrain-detect-mcp-mode.test.ts. +**Effort:** S (human ~2h, CC ~15min). + +### P3: Tighten revived coverage-audit E2E assertions + +**What:** The revived skill-e2e-coverage-audit tests assert hasGap OR hasTested +(near-vacuous) and reference skill sections their own DRIFT WARNING says moved. +Tighten to conjunctive assertions and retarget the prompts at live sections; +needs one paid run to validate, so it didn't ride the ship. +**Where:** test/skill-e2e-coverage-audit.test.ts. +**Effort:** S (human ~2h, CC ~15min + one paid run). + ## Completed +### ✅ DONE (v1.66.0.0): Free suite exit code is untrustworthy — in-process force-exits mask failures + +**Priority:** P1 + +**What:** At least five browse test files end with `setTimeout(() => process.exit(0), 500)` +(browse/test/commands.test.ts:101, snapshot.test.ts:36, batch.test.ts:47, +handoff.test.ts:31, content-security.test.ts:465). The timer fires inside the SHARED +`bun test` process, exiting 0 before bun prints its final summary — so `bun test` can +report exit 0 while real test failures scrolled by earlier. Remove the force-exits and +fix the underlying handle leaks they paper over (lingering Playwright/daemon handles +that once made the suite hang), or scope the exit to a spawned child process. + +**Why:** Observed 2026-08-07: three genuinely failing tests (eval-list-cli, +benchmark-cli, observability check 11) rode green `bun test` exit codes across +multiple runs; the failures only surfaced by grepping logs for "(fail)" lines. A test +suite that exits 0 on failure is worse than no suite — it manufactures false +confidence at commit time and in any CI job that trusts the exit code. + +**Pros:** Restores the one contract everything (CI, /ship, humans) relies on: exit +code == truth. Also un-hides the missing final summary block. +**Cons:** The force-exits exist because the suite once hung on leaked handles; +removing them without fixing the leaks trades silent failure for hangs. Needs a +focused pass: find each leaked handle (daemon children, PTY, Playwright contexts), +close them in afterAll, then delete the exits one file at a time. + +**Context / where to start:** `grep -rn "process.exit(0)" browse/test/` — the +setTimeout variants are the offenders (server-no-import-side-effects.test.ts:62 is a +spawned-child probe, fine). Repro: run the full free suite and note the log ends at +the browse files with no "Ran N tests" summary. Receipts: +~/.gstack-dev/logs/free-suite-main-check.log (3 masked fails, exit 0). + +**Completed:** v1.66.0.0 (2026-08-15) — main's v1.64 removed the force-exits; v1.66.0.0 adds runner-level strict-output classification (a shard without bun's terminal summary FAILS), size-scaled wall deadlines, and the failure-naming epilogue, so exit code == truth is enforced by the runner, not by convention. + ### Slim preamble + real-PTY plan-mode E2E harness (v1.13.1.0) - Compressed 18 preamble resolvers; total `SKILL.md` corpus dropped from 3.08 MB to 2.30 MB across 47 outputs (-25.5%, ~196K tokens saved). diff --git a/VERSION b/VERSION index acfd5a8ce..0ea99c033 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.65.0.0 +1.66.0.0 diff --git a/package.json b/package.json index 20676d9ad..0a1180591 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.65.0.0", + "version": "1.66.0.0", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module",