Commit Graph

2 Commits

Author SHA1 Message Date
Garry Tan 3ddbee5952
test(evals): harden no-op asserts, close tier-invariant fail-open holes, pin gate question strings
- no-op regression: gate-must-ask is now UNCONDITIONAL for eng/design (the
  outcome==='asked' conditional let a silent-bypass plan_ready run sail
  through); eng/design cases force --disallowedTools so the pinned prose
  shape is contractual rather than hoping native AUQ renders match; the
  named-target case uses trackTokens for consumption and lists
  wrote_findings_before_asking in its diagnostic throw branch.
- tier-alignment invariant: both quote styles matched; zero-self-gate,
  mixed-tier, and owning-keys-without-E2E_TIERS-entries are all REPORTED
  instead of silently skipped (the fail-open holes three reviewers found).
- drift-guard: the generated gate menus must carry the exact question/option
  strings the PTY question detector anchors on — free CI fails before the
  paid smokes can go vacuous on a menu reword.
- touchfiles: corrected the no-op cost note for CI concurrency + retry
  semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-12 09:09:48 -07:00
Garry Tan f349d7c894
fix(evals): align plan-eng/design plan-mode + finding-floor smokes to their declared periodic tier
The #2077 demotion of these four stochastic tests to 'periodic' was inert:
E2E_TIERS declared periodic but the files self-gated on EVALS_TIER === 'gate',
so they kept running in the blocking gate lane and never in the weekly lane.

Flip the four self-gates to 'periodic' (headers/describe labels updated), add
a free static tier-alignment invariant test (dep-list filename mapping;
unmapped self-gated files are reported, never silently skipped), and name the
two plan-mode test files in their own touchfiles dep lists so the invariant
binds for them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 17:40:17 -07:00