mirror of https://github.com/garrytan/gstack.git
02e312c057
Codex adversarial review (Step 11) flagged that these 4 tests were not
hermetic: the prompt only said "Read plan-eng-review/SKILL.md" (relative,
unpinned), and the copied SKILL.md's own STOP-Read directive references an
ABSOLUTE path (~/.claude/skills/gstack/plan-eng-review/sections/review-sections.md)
for its carved sections file. On any machine with gstack actually installed
at that location — which includes the maintainer's own dev machine — the
agent could read and get graded against the REAL globally-installed skill
instead of this branch's sandboxed copy, silently defeating the point of
the isolated fixture. This is the exact failure mode
test/helpers/auq-sdk-capture.ts's setupSkillDir()/captureSectionReads()
already guards against for other carved-skill tests; these 4 blocks
predated that pattern and didn't use it.
Fixed with a targeted change rather than a full migration to the
auq-sdk-capture harness: extracted planEngReviewDataModelPrompt() into
e2e-helpers.ts, pinning the sandboxed SKILL.md's absolute path and
explicitly forbidding the agent from reading any other SKILL.md ("especially
nothing under ~/.claude or /Users") — matching auq-sdk-capture.ts's own
phrasing. This also resolves the maintainability specialist's earlier DRY
finding (the identical prompt string was duplicated across all 4 blocks) in
the same change, since all 4 now call the one shared function.
Also fixed a second Codex finding: touchfiles.ts only listed
plan-eng-review/** as these tests' dependency, but their actual behavior
lives partly in test/helpers/e2e-helpers.ts (setupPlanEngReviewFixture,
matchesUnnegated) — a future bug there wouldn't have gotten these 4 tests
re-selected by the diff-based selector. Added the file to each test's
touchfile entry (scoped to just these 4, not GLOBAL_TOUCHFILES, since the
new exports aren't used elsewhere yet).
Two other Codex findings reviewed and accepted as-is, not fixed:
- matchesUnnegated can still mis-score idiom-based false negation ("no
doubt", "not only... but also") — a genuine lexical-heuristic limitation,
but these idioms are exceedingly unlikely to co-occur with the specific
verb+object patterns these checks require in real review prose; chasing
every conceivable idiom is diminishing returns for a best-effort matcher.
- The new evals are periodic tier, not gate — consistent with this
repo's existing convention for all Opus-model E2E tests (documented in
CLAUDE.md's two-tier system), not a gap introduced by this branch.
bun test test/helpers/e2e-helpers.test.ts test/skill-e2e-plan.test.ts
test/skill-validation.test.ts test/parity-suite.test.ts test/touchfiles.test.ts
test/gen-skill-docs.test.ts — 786 pass, 42 skip (paid E2E), 0 fail.
|
||
|---|---|---|
| .. | ||
| providers | ||
| agent-sdk-runner.ts | ||
| auq-sdk-capture.ts | ||
| benchmark-judge.ts | ||
| benchmark-runner.ts | ||
| budget-override.test.ts | ||
| budget-override.ts | ||
| capture-parity-baseline.test.ts | ||
| capture-parity-baseline.ts | ||
| carve-guard-checks.ts | ||
| carve-guards.ts | ||
| claude-pty-runner.ts | ||
| claude-pty-runner.unit.test.ts | ||
| codex-session-runner.ts | ||
| e2e-helpers.test.ts | ||
| e2e-helpers.ts | ||
| eval-store.test.ts | ||
| eval-store.ts | ||
| gemini-session-runner.test.ts | ||
| gemini-session-runner.ts | ||
| hermetic-env.test.ts | ||
| hermetic-env.ts | ||
| llm-judge.ts | ||
| observability.test.ts | ||
| parity-harness.ts | ||
| pricing.ts | ||
| required-reads.ts | ||
| secret-sink-harness.ts | ||
| session-runner.test.ts | ||
| session-runner.ts | ||
| skill-parser.ts | ||
| tool-map.ts | ||
| touchfiles.ts | ||
| transcript-section-logger.ts | ||