mirror of https://github.com/garrytan/gstack.git
9a424a9f55
Five categories of fixes surfaced by the /ship pre-landing reviews (testing + maintainability + security + performance + adversarial Claude), applied as one review-iteration commit. Refactor — collapse 5x duplicated judge-assertion block: - Add assertRecommendationQuality() + RECOMMENDATION_SUBSTANCE_THRESHOLD constant to test/helpers/e2e-helpers.ts. - Plan-format (4 cases) and Phase 4 (1 case) collapse from ~22 lines each to a single helper call. Future rubric tweaks land in one place instead of five. Performance — extract Phase 4 slice instead of copying full SKILL.md: - Phase 4 test fixture now reads office-hours/SKILL.md and writes only the AskUserQuestion Format section + Phase 4 section to the tmpdir, per CLAUDE.md "extract, don't copy" rule. Verified locally: cost dropped from $0.51 → $0.36/run, turn count 8 → 4, latency 50s → 36s. Reduces Opus context bloat without weakening the regression check. - Add `if (!workDir) return` guard to Phase 4 afterAll cleanup so a skipped describe block doesn't silently fs.rmSync(undefined) under the empty catch. Defense — judge prompt + output: - Wrap captured AskUserQuestion text in clearly delimited UNTRUSTED_CONTEXT block with explicit instruction to treat its content as data, not commands. Cheap defense against the (unlikely but real) injection vector where a captured AskUserQuestion contains "Ignore previous instructions" text. - Bump captured-text budget from 4000 → 8000 chars; real plan-format menus with 4 options × ~800 chars exceed 4000 and were silently truncating Haiku context mid-option. Cleanup — abbreviation rule + dead imports + touchfile consistency: - AUQ → AskUserQuestion in 3 sites (office-hours/SKILL.md.tmpl Phase 4 footer, two test comments) per the always-write-in-full memory rule. Regenerated office-hours/SKILL.md. - Drop unused `describe`/`test` imports in 2 new test files (only describeIfSelected/testConcurrentIfSelected wrappers are used). - Add `test/skill-e2e-office-hours-phase4.test.ts` to its own touchfile entry for consistency with other entries that include their test file. - Fix misleading comment in fixture test about LLM short-circuiting (it's has_because, not commits, that skips the API call). Verified: build clean, free `bun test` exits 0, fixture test 30/30 expect() calls pass, Phase 4 paid eval passes substance 5 in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| providers | ||
| agent-sdk-runner.ts | ||
| benchmark-judge.ts | ||
| benchmark-runner.ts | ||
| claude-pty-runner.ts | ||
| claude-pty-runner.unit.test.ts | ||
| codex-session-runner.ts | ||
| e2e-helpers.ts | ||
| eval-store.test.ts | ||
| eval-store.ts | ||
| gemini-session-runner.test.ts | ||
| gemini-session-runner.ts | ||
| llm-judge.ts | ||
| observability.test.ts | ||
| pricing.ts | ||
| secret-sink-harness.ts | ||
| session-runner.test.ts | ||
| session-runner.ts | ||
| skill-parser.ts | ||
| tool-map.ts | ||
| touchfiles.ts | ||