From b831f9b26475d8519951c090f12e4cbadeb93666 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 15 Aug 2026 17:25:05 -0700 Subject: [PATCH] ci(free-tests): restore the PR-number concurrency + failure-log artifact; truth-fix stale comments The workspace-revert incident that hit CHANGELOG/TODOS mid-ship also caught free-tests.yml between edits: commit 8d6c2ff8's message claims PR-number concurrency + artifact upload + main-push runs, but only the push trigger survived to the commit (caught by the /document-release doc-vs-code audit). Both re-applied. Also: eval-model.ts header said capture defaults to Opus (it's Sonnet per D1a), paid-shards' header pinned a stale 44/63 shard census, and two CHANGELOG phrases over-claimed ('six' -> 'up to six' shard processes; retry-1 scoped to retry-bearing paid paths). --- .github/workflows/free-tests.yml | 18 +++++++++++++++++- CHANGELOG.md | 4 ++-- lib/eval-model.ts | 2 +- scripts/test-paid-shards.ts | 6 +++--- 4 files changed, 23 insertions(+), 7 deletions(-) diff --git a/.github/workflows/free-tests.yml b/.github/workflows/free-tests.yml index 6c6ae5dc2..5a409c524 100644 --- a/.github/workflows/free-tests.yml +++ b/.github/workflows/free-tests.yml @@ -30,8 +30,11 @@ on: branches: [main] workflow_dispatch: +# Keyed on the PR number, not head_ref: a bare branch name carries no fork +# prefix, so same-name branches from two forks would share one group and a +# push to fork B's PR would cancel fork A's in-flight REQUIRED check. concurrency: - group: free-tests-${{ github.head_ref || github.run_id }} + group: free-tests-${{ github.event.pull_request.number || github.run_id }} cancel-in-progress: true # Least privilege: this job executes PR-controlled code (install lifecycle @@ -94,3 +97,16 @@ jobs: - name: Run free suite run: bun run test:free + + # The runner streams the full child output to per-run logs under the OS + # tmpdir and prints only the quiet contract to the console. Without this + # upload, a red required check names WHICH test failed but the why + # (assertion detail, stack) dies with the runner — every diagnosis would + # need a local re-run, which fork contributors can't do on this image. + - name: Upload shard logs on failure + if: failure() + uses: actions/upload-artifact@v4 + with: + name: free-test-shard-logs + path: /tmp/gstack-free-test-*.log + if-no-files-found: ignore diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e353ad07..fc80fa507 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ **The full ~7,000-test suite in about 90 seconds, verified honest.** **Paid evals now bill by diff, not $38 flat.** -`bun run test` used to take 454 seconds. It now runs as six concurrent shard processes and finishes in about 90 to 100 seconds, under a strict output contract: a shard that exits without bun's own terminal summary line is a failure, a wedged shard is killed at a size-scaled deadline and named in the epilogue, and the console shows only what you need (per-shard status, then `✗ file — test name` for anything red, full stream in a per-run log, `--verbose` for the firehose). Twelve test files that ran under no script and no CI are wired in. A 3,372-line dead eval monolith is deleted, with four never-run tests revived out of it. +`bun run test` used to take 454 seconds. It now runs as up to six concurrent shard processes and finishes in about 90 to 100 seconds, under a strict output contract: a shard that exits without bun's own terminal summary line is a failure, a wedged shard is killed at a size-scaled deadline and named in the epilogue, and the console shows only what you need (per-shard status, then `✗ file — test name` for anything red, full stream in a per-run log, `--verbose` for the firehose). Twelve test files that ran under no script and no CI are wired in. A 3,372-line dead eval monolith is deleted, with four never-run tests revived out of it. Paid evals select by diff. Edit one skill and the runner executes only the shards your change touches, reports the rest as skipped-by-diff, and prints the reason. Selection sees uncommitted and untracked work, fails closed with a named cause on git errors, and an edit to the selection data itself re-runs only the changed keys instead of forcing the full suite. @@ -41,7 +41,7 @@ Runs you used to schedule around now fit inside a thought. `bun run test` before - Free suite architecture: N concurrent shard processes (serial within each); tree-mutating tests and tree-measuring ratchet readers run in one serial shard after the parallel phase, so measurements never race regeneration. Shard curation lists are pinned against the live file census, and wall deadlines scale with shard size. - Agent SDK capture default Opus → Sonnet (D1a). The judge default stays Sonnet: a live A/B on the health rubric scored Haiku 2/2/2 against Sonnet's 4/3/4, so the downgrade was pinned back per D1a's regressor clause (receipts in `test/helpers/llm-judge.ts`). - Four expensive posture tests demoted gate → periodic (D2a). -- Paid runners: `EVALS_JOBS` (shard process count) split from `EVALS_CONCURRENCY` (within-shard), `--retry 1` on every paid path, one preflight API ping per run instead of ~30, detach timeouts floor-enforced against the live shard census by `test/eval-detach-timeout-floor.test.ts`. +- Paid runners: `EVALS_JOBS` (shard process count) split from `EVALS_CONCURRENCY` (within-shard), `--retry 1` on every retry-bearing paid path, one preflight API ping per run instead of ~30, detach timeouts floor-enforced against the live shard census by `test/eval-detach-timeout-floor.test.ts`. - CI: eval Docker image cache keyed on Dockerfile + bun.lock so version bumps stop rebuilding it; Bun 1.3.13 in the image; `skill-e2e-review` split into three matrix shards; actionlint runs a digest-pinned prebuilt image; five single-core jobs right-sized; lint and skill-docs stop double-running every PR commit; the Windows lane caches bun installs and runs the curated suite instead of a hand list. - Skill-routing E2E fixture installs skill heads, not ~18 full SKILL.md files. diff --git a/lib/eval-model.ts b/lib/eval-model.ts index 5a120adab..b720a307d 100644 --- a/lib/eval-model.ts +++ b/lib/eval-model.ts @@ -12,7 +12,7 @@ * per-kind default — last resort * * Kinds and their defaults: - * capture — AskUserQuestion SDK capture runs (quality matters): opus + * capture — AskUserQuestion SDK capture runs: sonnet (D1a) * warmup — PTY warm-up ping (cheapest thing that answers): haiku * distill — free-text distillation (cheap, structured): haiku (pinned) */ diff --git a/scripts/test-paid-shards.ts b/scripts/test-paid-shards.ts index c2f3e7e3d..e6d36c165 100644 --- a/scripts/test-paid-shards.ts +++ b/scripts/test-paid-shards.ts @@ -23,9 +23,9 @@ * 3. No per-shard env / eval dir. Each shard needs its own GSTACK_EVAL_DIR * so eval baselines are per-test-file instead of last-flush-wins. * - * Worst-case wall clock = ceil(shards / jobs) × shard timeout. At the time of - * writing: gate is 44 one-file shards → ceil(44/4) × 30min = 5.5h; periodic is - * 63 → 8h. Do NOT hand-derive the eval:bg:* detach timeouts from a snapshot of + * Worst-case wall clock = ceil(shards / jobs) × shard timeout. Shard counts + * drift as test files land, so treat any number written here as stale. + * Do NOT hand-derive the eval:bg:* detach timeouts from a snapshot of * these counts — test/eval-detach-timeout-floor.test.ts recomputes the bound * from the live shard census every run and fails CI if package.json's numbers * dip below it (undersized detach timeouts recreate never-started truncation).