gstack/test/helpers
David Grant a9124bd15b test: add minimal-change counterexample eval, unit-test matchesUnnegated
garrytan/gstack#1048 comment asked for concrete evaluations covering three
cases: normalized models, justified JSON fields, and minimal-change cases.
The prior commits covered the first two (data-model-bias, legitimate-json,
measured-denorm) but not the third — a plan where keeping a single trivial
field inline (not extracting a new model) is the correct call. This was
independently flagged by the ship-workflow coverage audit as the highest-
severity gap in this branch's test coverage.

Adds plan-eng-review-data-model-minimal-change: a synthetic plan adding one
nullable timestamp field to an existing model, no polymorphism, no JSON, no
independent query pattern or consumer. Asserts the skill does NOT reflexively
recommend extracting a separate model for it — exercising the "unless the
extra job is a single trivial field..." exception added to cognitive pattern
#12 and the SRP checklist item in 5374987c.

Also adds test/helpers/e2e-helpers.test.ts: free, deterministic unit tests
for matchesUnnegated() and setupPlanEngReviewFixture(), which were previously
only exercised indirectly by the paid, EVALS=1-gated E2E tests (also flagged
by the coverage audit — a bug in either could silently flip an eval's
pass/fail verdict and look like ordinary LLM wording variance). Along the
way, found and documented a real limitation: the negation-window check
correctly handles multiple sentences (period-bounded), but a single
comma-spliced sentence containing both a negated and a positive match can
still be absorbed into one greedy match — accepted as a known edge case,
since real review prose reliably separates points with periods/bullets.

bun test test/skill-validation.test.ts test/parity-suite.test.ts
test/touchfiles.test.ts test/skill-e2e-plan.test.ts test/helpers/e2e-helpers.test.ts
test/gen-skill-docs.test.ts — all pass, 0 fail.
2026-07-15 18:08:11 -07:00
..
providers fix(benchmark): skip-trust for headless gemini + drop release metadata 2026-07-10 10:38:46 -07:00
agent-sdk-runner.ts v1.58.1.0 feat: hermetic local E2E + Conductor prose AskUserQuestion (#2004) 2026-06-14 11:40:57 -07:00
auq-sdk-capture.ts v1.56.0.0 Token-reduction Phase B + AUQ paranoid safety net (#1849) 2026-06-04 11:14:43 -07:00
benchmark-judge.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
benchmark-runner.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
budget-override.test.ts v1.46.0.0 feat: gstack v2 foundation — catalog tokens drop 56%, eval-first floor covers all 51 skills (#1712) 2026-05-26 16:50:03 -07:00
budget-override.ts v1.46.0.0 feat: gstack v2 foundation — catalog tokens drop 56%, eval-first floor covers all 51 skills (#1712) 2026-05-26 16:50:03 -07:00
capture-parity-baseline.test.ts v1.46.0.0 feat: gstack v2 foundation — catalog tokens drop 56%, eval-first floor covers all 51 skills (#1712) 2026-05-26 16:50:03 -07:00
capture-parity-baseline.ts v1.46.0.0 feat: gstack v2 foundation — catalog tokens drop 56%, eval-first floor covers all 51 skills (#1712) 2026-05-26 16:50:03 -07:00
carve-guard-checks.ts v1.57.0.0 feat: carve-guard system + carve cso/document-release/design-consultation (#1907) 2026-06-07 19:13:24 -07:00
carve-guards.ts fix(plan-eng-review,plan-ceo-review): narrow absolute schema-bias language, add legitimate-JSON and measured-denormalization counterexamples 2026-07-15 17:20:19 -07:00
claude-pty-runner.ts v1.58.4.0 fix: high-priority community bug wave + PTY plan-mode smoke gate (#2077) 2026-06-21 07:15:19 -07:00
claude-pty-runner.unit.test.ts v1.58.4.0 fix: high-priority community bug wave + PTY plan-mode smoke gate (#2077) 2026-06-21 07:15:19 -07:00
codex-session-runner.ts v1.58.1.0 feat: hermetic local E2E + Conductor prose AskUserQuestion (#2004) 2026-06-14 11:40:57 -07:00
e2e-helpers.test.ts test: add minimal-change counterexample eval, unit-test matchesUnnegated 2026-07-15 18:08:11 -07:00
e2e-helpers.ts fix: harden new data-model eval regexes, dedupe E2E fixture setup 2026-07-15 17:37:23 -07:00
eval-store.test.ts feat: QA restructure, browser ref staleness, eval efficiency metrics (v0.4.0) (#83) 2026-03-15 23:55:39 -05:00
eval-store.ts v1.32.0.0 fix wave: 7 community PRs + 5 gate-eval hardenings (#1431) 2026-05-11 12:16:26 -07:00
gemini-session-runner.test.ts feat: Gemini CLI E2E tests (v0.9.2.0) (#252) 2026-03-20 08:30:09 -07:00
gemini-session-runner.ts fix(benchmark): skip-trust for headless gemini + drop release metadata 2026-07-10 10:38:46 -07:00
hermetic-env.test.ts v1.58.1.0 feat: hermetic local E2E + Conductor prose AskUserQuestion (#2004) 2026-06-14 11:40:57 -07:00
hermetic-env.ts v1.58.1.0 feat: hermetic local E2E + Conductor prose AskUserQuestion (#2004) 2026-06-14 11:40:57 -07:00
llm-judge.ts v1.25.1.0 fix: office-hours Phase 4 STOP gate + AskUserQuestion recommendation judge (#1296) 2026-05-01 19:51:51 -07:00
observability.test.ts fix: never clean up observability artifacts — partial file persists after finalize 2026-03-14 12:37:38 -05:00
parity-harness.ts v1.58.5.0 feat: first-run activation scaffold + gstack router front door (#2078) 2026-06-25 09:42:45 -07:00
pricing.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
required-reads.ts v1.54.0.0 feat: carve /ship into skeleton + on-demand sections (-59% always-loaded) (#1806) 2026-05-30 12:09:10 -07:00
secret-sink-harness.ts v1.12.0.0 feat: /setup-gbrain — coding-agent onboarding for gbrain (#1183) 2026-04-24 01:38:21 -07:00
session-runner.test.ts feat: stream-json NDJSON parser for real-time E2E progress 2026-03-14 03:49:36 -05:00
session-runner.ts fix(test-harness): spawn claude without shell interpolation 2026-07-14 12:56:07 -07:00
skill-parser.ts feat: content security — 4-layer prompt injection defense for pair-agent (#815) 2026-04-06 14:41:06 -07:00
tool-map.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
touchfiles.ts test: add minimal-change counterexample eval, unit-test matchesUnnegated 2026-07-15 18:08:11 -07:00
transcript-section-logger.ts v1.54.0.0 feat: carve /ship into skeleton + on-demand sections (-59% always-loaded) (#1806) 2026-05-30 12:09:10 -07:00