mirror of https://github.com/garrytan/gstack.git
docs(todos): fail-fast fork-PR secret detection in evals.yml
Filed from the PR #2211 full-checks run: fork shards burned 31m48s timing out on an empty ANTHROPIC_API_KEY before failing, and the contributor re-triggered the doomed run reading it as flakiness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
a31f8652d3
commit
9faf8c2929
26
TODOS.md
26
TODOS.md
|
|
@ -45,6 +45,32 @@ a silent mistake breaks all 52 skills. High blast radius — needs its own focus
|
|||
|
||||
## Test infrastructure
|
||||
|
||||
### P3: Fail-fast fork-PR secret detection in evals.yml (filed via PR #2211 full-checks run)
|
||||
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Add an early step to each eval shard in `.github/workflows/evals.yml` that
|
||||
detects a secretless run (fork PR: `secrets.ANTHROPIC_API_KEY` empty) and exits
|
||||
immediately with a clear "fork PR: eval secrets unavailable — see fork-PR workflow in
|
||||
CLAUDE.md" annotation instead of running tests into auth failures.
|
||||
|
||||
**Why:** On fork PR #2211, `e2e-review` failed in 1m with `error_api` and
|
||||
`e2e-pty-plan-smoke` burned 31m48s of runner time timing out on "Not logged in"
|
||||
before failing. The contributor read it as flakiness and re-triggered, burning the
|
||||
time twice. A 5-second preflight would have said exactly what was wrong.
|
||||
|
||||
**Pros:** Saves ~30+ min of Ubicloud runner time per fork-PR push; replaces a
|
||||
misleading red X with an actionable message; stops contributors re-triggering
|
||||
doomed runs.
|
||||
**Cons:** One more conditional per shard; must not mask a genuinely misconfigured
|
||||
base-repo secret (annotate differently when `github.event.pull_request.head.repo.fork`
|
||||
is false).
|
||||
|
||||
**Context / where to start:** `.github/workflows/evals.yml` matrix job — add a step
|
||||
before "Run <suite>" that checks the secret and `github.event.pull_request.head.repo.fork`,
|
||||
emits a `::notice` and exits 0 (or neutral). Diagnosis receipts: PR #2211 run 29056479216,
|
||||
jobs 86249027242 (empty `ANTHROPIC_API_KEY`, `error_api`) and 86249027265 (login timeout).
|
||||
|
||||
### Eval harness: live progress + incremental result persistence (kill the silent hour)
|
||||
|
||||
**Priority:** P1
|
||||
|
|
|
|||
Loading…
Reference in New Issue