mirror of https://github.com/garrytan/gstack.git
09ad6fb1ef
Ran /review on this branch's diff against main. The always-on testing
specialist found real correctness bugs in the two eval blocks added in
33fbb58b:
- pushesToSplitTheField (legitimate-JSON eval) was negation-blind: a
CORRECT "I would NOT extract the payload into columns" answer contains
the same verb+payload+column proximity as an incorrect one, so it could
fail the test on a passing review. Also missing "normalize" from its verb
list, so a real regression phrased as "normalize this payload into
columns" would slip through undetected.
- rejectsAsUnjustified (measured-denorm eval) missed common rejection
phrasings ("instead of denormalizing", "rather than snapshotting"), and
its normali[sz]e-instead/back/it clause was loose enough to false-positive
on approving language ("stays normalized elsewhere; keep it that way").
- recommendsSeparateModel (pre-existing data-model-bias eval, touched by
this branch to fix its fixture) broke on backtick-wrapped model names, a
common LLM markdown convention.
Added matchesUnnegated() to test/helpers/e2e-helpers.ts — walks every regex
match and checks the ~20 chars before it for a negation word before counting
it as a positive signal — and used it for the split-detector. Broadened the
rejection regex and tightened its false-positive-prone clause. Stripped
markdown formatting before the backtick-affected check. Verified all four
fixes against representative sentences (correct-negative, actual-positive,
previously-missed-regression, and previously-false-positive cases) before
committing — all passed as expected.
Also extracted the maintainability specialist's DRY finding: the three
data-model E2E blocks (bias, legitimate-json, measured-denorm) each
duplicated the same ~20-line mkdtemp/git-init/write-plan/copy-skill fixture
setup. Extracted into setupPlanEngReviewFixture() in e2e-helpers.ts.
bun test test/skill-validation.test.ts test/parity-suite.test.ts
test/touchfiles.test.ts test/gen-skill-docs.test.ts test/skill-e2e-plan.test.ts
— 768 pass, 39 skip (paid E2E), 0 fail.
|
||
|---|---|---|
| .. | ||
| providers | ||
| agent-sdk-runner.ts | ||
| auq-sdk-capture.ts | ||
| benchmark-judge.ts | ||
| benchmark-runner.ts | ||
| budget-override.test.ts | ||
| budget-override.ts | ||
| capture-parity-baseline.test.ts | ||
| capture-parity-baseline.ts | ||
| carve-guard-checks.ts | ||
| carve-guards.ts | ||
| claude-pty-runner.ts | ||
| claude-pty-runner.unit.test.ts | ||
| codex-session-runner.ts | ||
| e2e-helpers.ts | ||
| eval-store.test.ts | ||
| eval-store.ts | ||
| gemini-session-runner.test.ts | ||
| gemini-session-runner.ts | ||
| hermetic-env.test.ts | ||
| hermetic-env.ts | ||
| llm-judge.ts | ||
| observability.test.ts | ||
| parity-harness.ts | ||
| pricing.ts | ||
| required-reads.ts | ||
| secret-sink-harness.ts | ||
| session-runner.test.ts | ||
| session-runner.ts | ||
| skill-parser.ts | ||
| tool-map.ts | ||
| touchfiles.ts | ||
| transcript-section-logger.ts | ||