ci(free-tests): restore the PR-number concurrency + failure-log artifact; truth-fix stale comments

The workspace-revert incident that hit CHANGELOG/TODOS mid-ship also
caught free-tests.yml between edits: commit 8d6c2ff8's message claims
PR-number concurrency + artifact upload + main-push runs, but only the
push trigger survived to the commit (caught by the /document-release
doc-vs-code audit). Both re-applied. Also: eval-model.ts header said
capture defaults to Opus (it's Sonnet per D1a), paid-shards' header
pinned a stale 44/63 shard census, and two CHANGELOG phrases
over-claimed ('six' -> 'up to six' shard processes; retry-1 scoped to
retry-bearing paid paths).
This commit is contained in:
Garry Tan 2026-08-15 17:25:05 -07:00
parent 9b9bc84bdc
commit b831f9b264
No known key found for this signature in database
GPG Key ID: C1F69E85C74EFE1D
4 changed files with 23 additions and 7 deletions

View File

@ -30,8 +30,11 @@ on:
branches: [main]
workflow_dispatch:
# Keyed on the PR number, not head_ref: a bare branch name carries no fork
# prefix, so same-name branches from two forks would share one group and a
# push to fork B's PR would cancel fork A's in-flight REQUIRED check.
concurrency:
group: free-tests-${{ github.head_ref || github.run_id }}
group: free-tests-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
# Least privilege: this job executes PR-controlled code (install lifecycle
@ -94,3 +97,16 @@ jobs:
- name: Run free suite
run: bun run test:free
# The runner streams the full child output to per-run logs under the OS
# tmpdir and prints only the quiet contract to the console. Without this
# upload, a red required check names WHICH test failed but the why
# (assertion detail, stack) dies with the runner — every diagnosis would
# need a local re-run, which fork contributors can't do on this image.
- name: Upload shard logs on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: free-test-shard-logs
path: /tmp/gstack-free-test-*.log
if-no-files-found: ignore

View File

@ -5,7 +5,7 @@
**The full ~7,000-test suite in about 90 seconds, verified honest.**
**Paid evals now bill by diff, not $38 flat.**
`bun run test` used to take 454 seconds. It now runs as six concurrent shard processes and finishes in about 90 to 100 seconds, under a strict output contract: a shard that exits without bun's own terminal summary line is a failure, a wedged shard is killed at a size-scaled deadline and named in the epilogue, and the console shows only what you need (per-shard status, then `✗ file — test name` for anything red, full stream in a per-run log, `--verbose` for the firehose). Twelve test files that ran under no script and no CI are wired in. A 3,372-line dead eval monolith is deleted, with four never-run tests revived out of it.
`bun run test` used to take 454 seconds. It now runs as up to six concurrent shard processes and finishes in about 90 to 100 seconds, under a strict output contract: a shard that exits without bun's own terminal summary line is a failure, a wedged shard is killed at a size-scaled deadline and named in the epilogue, and the console shows only what you need (per-shard status, then `✗ file — test name` for anything red, full stream in a per-run log, `--verbose` for the firehose). Twelve test files that ran under no script and no CI are wired in. A 3,372-line dead eval monolith is deleted, with four never-run tests revived out of it.
Paid evals select by diff. Edit one skill and the runner executes only the shards your change touches, reports the rest as skipped-by-diff, and prints the reason. Selection sees uncommitted and untracked work, fails closed with a named cause on git errors, and an edit to the selection data itself re-runs only the changed keys instead of forcing the full suite.
@ -41,7 +41,7 @@ Runs you used to schedule around now fit inside a thought. `bun run test` before
- Free suite architecture: N concurrent shard processes (serial within each); tree-mutating tests and tree-measuring ratchet readers run in one serial shard after the parallel phase, so measurements never race regeneration. Shard curation lists are pinned against the live file census, and wall deadlines scale with shard size.
- Agent SDK capture default Opus → Sonnet (D1a). The judge default stays Sonnet: a live A/B on the health rubric scored Haiku 2/2/2 against Sonnet's 4/3/4, so the downgrade was pinned back per D1a's regressor clause (receipts in `test/helpers/llm-judge.ts`).
- Four expensive posture tests demoted gate → periodic (D2a).
- Paid runners: `EVALS_JOBS` (shard process count) split from `EVALS_CONCURRENCY` (within-shard), `--retry 1` on every paid path, one preflight API ping per run instead of ~30, detach timeouts floor-enforced against the live shard census by `test/eval-detach-timeout-floor.test.ts`.
- Paid runners: `EVALS_JOBS` (shard process count) split from `EVALS_CONCURRENCY` (within-shard), `--retry 1` on every retry-bearing paid path, one preflight API ping per run instead of ~30, detach timeouts floor-enforced against the live shard census by `test/eval-detach-timeout-floor.test.ts`.
- CI: eval Docker image cache keyed on Dockerfile + bun.lock so version bumps stop rebuilding it; Bun 1.3.13 in the image; `skill-e2e-review` split into three matrix shards; actionlint runs a digest-pinned prebuilt image; five single-core jobs right-sized; lint and skill-docs stop double-running every PR commit; the Windows lane caches bun installs and runs the curated suite instead of a hand list.
- Skill-routing E2E fixture installs skill heads, not ~18 full SKILL.md files.

View File

@ -12,7 +12,7 @@
* per-kind default last resort
*
* Kinds and their defaults:
* capture AskUserQuestion SDK capture runs (quality matters): opus
* capture AskUserQuestion SDK capture runs: sonnet (D1a)
* warmup PTY warm-up ping (cheapest thing that answers): haiku
* distill free-text distillation (cheap, structured): haiku (pinned)
*/

View File

@ -23,9 +23,9 @@
* 3. No per-shard env / eval dir. Each shard needs its own GSTACK_EVAL_DIR
* so eval baselines are per-test-file instead of last-flush-wins.
*
* Worst-case wall clock = ceil(shards / jobs) × shard timeout. At the time of
* writing: gate is 44 one-file shards ceil(44/4) × 30min = 5.5h; periodic is
* 63 8h. Do NOT hand-derive the eval:bg:* detach timeouts from a snapshot of
* Worst-case wall clock = ceil(shards / jobs) × shard timeout. Shard counts
* drift as test files land, so treat any number written here as stale.
* Do NOT hand-derive the eval:bg:* detach timeouts from a snapshot of
* these counts test/eval-detach-timeout-floor.test.ts recomputes the bound
* from the live shard census every run and fails CI if package.json's numbers
* dip below it (undersized detach timeouts recreate never-started truncation).