CHANGELOG no longer claims plan-ceo-review changes (bullets were dropped
entirely) and reflects the Data Model Review's promotion to its own section
instead of 11 checklist items + 3 cognitive patterns. Removed the P2 TODO
for /plan-ceo-review data-model-bias E2E coverage — moot now that
plan-ceo-review carries none of these bullets to test.
Per PR review feedback: the negation-aware matcher these E2E evals rely on
had accumulated real complexity (sentence-boundary scanning, markdown-bullet
clause detection, curly-apostrophe contraction matching, idiom exclusions)
across four rounds of specialist/adversarial review during this branch's own
/ship — each finding was real, but chasing all of them was disproportionate
for a periodic-tier, non-gating eval. An occasional missed edge case there
just means a rare, visible false read, not a missed real regression.
Reverted to a plain fixed-window lookback (60 chars, accounting for the
patterns' own internal gaps) with a short negation word list. Kept the
generic [a-z]+n't contraction match (can't/won't/aren't) since it's one
regex alternative, not meaningful added complexity.
Removed test/helpers/e2e-helpers.test.ts — a dedicated unit-test file for a
best-effort text-matching heuristic used only by 4 paid, periodic-tier evals
was more test infrastructure than the actual ask (3-4 new eval cases)
warranted.
PR review feedback on #1071:
- Dropped the data-model exception + JSONField bullets from plan-ceo-review
entirely. CEO review should stay high-level on "right-sized diff"; data-model
pushback belongs to plan-eng-review, which already carries the full checklist.
plan-ceo-review/SKILL.md(.tmpl) are now byte-identical to origin/main.
- Trimmed both remaining plan-eng-review bullets (data-model exception,
JSONField) to single-sentence terseness, matching the style of sibling
preferences instead of reading like a paragraph.
- Removed cognitive patterns 11-13 (normalize-first, SRP-for-models,
structure-beats-blobs) from the numbered "Cognitive Patterns" list — they
were more tactical/lower-level than the rest of that list (Larson, McKinley,
Conway-style instincts), and duplicated what the Data Model Review now
covers operationally. Renumbered 14-18 down to 11-15.
- Promoted the Data Model Review Checklist from a "####" subsection tacked
onto Architecture Review to its own "### Data model review" section with
its own AskUserQuestion/STOP gate, matching Architecture/Code
Quality/Test/Performance as a peer rather than an afterthought. Did not
renumber the existing 1-4 sections (Architecture/Code Quality/Test/
Performance) since those numbers are referenced elsewhere (carve-guards.ts,
the main SKILL.md.tmpl's section-order prose).
Updated test/skill-validation.test.ts's static guardrails to match: removed
the plan-ceo-review assertions (nothing left to guard there), removed the
three now-deleted cognitive-pattern assertions, updated the contiguity check
from 1-18 to 1-15, and updated the checklist-subsection test to check for
the new "### Data model review" heading.
Tightened test/helpers/carve-guards.ts budgets back down now that content
shrank: plan-ceo-review reverted fully to its original 90_000/1.08 (content
is byte-identical to baseline again); plan-eng-review tightened from
70_000/1.15 to 67_500/1.13 (down from the prior over-generous bump, though
not fully back to the original 67_000/no-ratio since the trimmed bullets are
still a small net addition).
bun test test/skill-validation.test.ts test/parity-suite.test.ts
test/touchfiles.test.ts test/gen-skill-docs.test.ts — all pass.
Codex cross-model doc review (via /document-release, run as part of /ship
Step 18) flagged that the CHANGELOG's contributor bullet about the new
periodic-tier E2E evals didn't scope them to /plan-eng-review — could be
misread as covering /plan-ceo-review too, which still only has static text
coverage (tracked as a P2 follow-up in TODOS.md).
Codex structured review (large-diff gate, Step 11) found 3 more real gaps:
- matchesUnnegated's bare "no" trigger over-fired on common positive-framing
idioms: "No blocker here: create a separate tier model" and "there is no
downside to promoting the payload" both got misread as negated, even
though they're endorsements of the recommendation that follows, not
rejections. Removed bare "no" from the negation list — "not"/"never"/
"cannot"/the "-n't" family are far more reliable negation signals and
don't share this idiom-collision problem.
- SENTENCE_END only recognized period-followed-by-capital as a boundary, so
a bulleted recommendation like "I would not keep this inline.\n- Separate
tier model" (very common LLM output formatting) let the earlier "not"
leak into the bullet below it. Added a second alternative: a newline
followed by a markdown bullet or numbered-list marker now also counts as
a clause boundary. Fixed the boundary-index arithmetic to use the full
match length (em[0].length) instead of a bare +1, since this new
alternative can match more than one character ("\n- ", "\n1. ").
- The minimal-change eval's own hardcoded doesn't-pattern only matched the
ASCII apostrophe (`doesn'?t`), missing the curly one — inconsistent with
matchesUnnegated's own apostrophe handling a few lines away in the same
file.
Added 3 regression tests reproducing each directly.
bun test test/helpers/e2e-helpers.test.ts test/skill-e2e-plan.test.ts
test/skill-validation.test.ts test/parity-suite.test.ts test/touchfiles.test.ts
test/gen-skill-docs.test.ts — 788 pass, 42 skip (paid E2E), 0 fail.
Codex adversarial review (Step 11) flagged that these 4 tests were not
hermetic: the prompt only said "Read plan-eng-review/SKILL.md" (relative,
unpinned), and the copied SKILL.md's own STOP-Read directive references an
ABSOLUTE path (~/.claude/skills/gstack/plan-eng-review/sections/review-sections.md)
for its carved sections file. On any machine with gstack actually installed
at that location — which includes the maintainer's own dev machine — the
agent could read and get graded against the REAL globally-installed skill
instead of this branch's sandboxed copy, silently defeating the point of
the isolated fixture. This is the exact failure mode
test/helpers/auq-sdk-capture.ts's setupSkillDir()/captureSectionReads()
already guards against for other carved-skill tests; these 4 blocks
predated that pattern and didn't use it.
Fixed with a targeted change rather than a full migration to the
auq-sdk-capture harness: extracted planEngReviewDataModelPrompt() into
e2e-helpers.ts, pinning the sandboxed SKILL.md's absolute path and
explicitly forbidding the agent from reading any other SKILL.md ("especially
nothing under ~/.claude or /Users") — matching auq-sdk-capture.ts's own
phrasing. This also resolves the maintainability specialist's earlier DRY
finding (the identical prompt string was duplicated across all 4 blocks) in
the same change, since all 4 now call the one shared function.
Also fixed a second Codex finding: touchfiles.ts only listed
plan-eng-review/** as these tests' dependency, but their actual behavior
lives partly in test/helpers/e2e-helpers.ts (setupPlanEngReviewFixture,
matchesUnnegated) — a future bug there wouldn't have gotten these 4 tests
re-selected by the diff-based selector. Added the file to each test's
touchfile entry (scoped to just these 4, not GLOBAL_TOUCHFILES, since the
new exports aren't used elsewhere yet).
Two other Codex findings reviewed and accepted as-is, not fixed:
- matchesUnnegated can still mis-score idiom-based false negation ("no
doubt", "not only... but also") — a genuine lexical-heuristic limitation,
but these idioms are exceedingly unlikely to co-occur with the specific
verb+object patterns these checks require in real review prose; chasing
every conceivable idiom is diminishing returns for a best-effort matcher.
- The new evals are periodic tier, not gate — consistent with this
repo's existing convention for all Opus-model E2E tests (documented in
CLAUDE.md's two-tier system), not a gap introduced by this branch.
bun test test/helpers/e2e-helpers.test.ts test/skill-e2e-plan.test.ts
test/skill-validation.test.ts test/parity-suite.test.ts test/touchfiles.test.ts
test/gen-skill-docs.test.ts — 786 pass, 42 skip (paid E2E), 0 fail.
Maintainability specialist re-dispatch found the standalone n't alternative
in NEGATION was unreachable: \b n't \b can never match mid-word (no word
boundary exists between the letters immediately before "n" and "n" itself
in "can't", "won't", "aren't", "hasn't"), so only the explicitly-spelled-out
contractions (don't, doesn't, didn't, shouldn't, wouldn't, isn't) were ever
detected — very common ones like "can't"/"won't"/"cannot" fell through
silently. Replaced with a generic [a-z]+n't pattern plus an explicit
"cannot" alternative (which has no apostrophe to match generically).
Also brought the original data-model-bias test's recommendsSeparateModel
check in line with its three newer sibling counterexample tests by routing
it through matchesUnnegated — it was the one E2E block still un-guarded
against a negated false-positive ("I would NOT recommend a separate tier
model" would previously have registered as a pass).
Did not touch rejectsAsUnjustified (measured-denorm test) — it directly
models rejection language ("instead of denormalizing" IS the signal, not a
positive claim needing negation-checking), so wrapping it in
matchesUnnegated would conflict with its own tuned alternatives for no
clear benefit. Did not extract a shared prompt-string constant across the
4 E2E blocks (maintainability finding) — the identical prompt string already
appears verbatim in 3 pre-existing, untouched blocks in this same file;
extracting a constant for only the 4 new blocks would create inconsistency
rather than resolve it, and touching the pre-existing blocks is outside
this PR's scope.
Added unit tests reproducing both contraction gaps directly.
bun test test/helpers/e2e-helpers.test.ts test/skill-e2e-plan.test.ts
test/skill-validation.test.ts test/parity-suite.test.ts test/touchfiles.test.ts
test/gen-skill-docs.test.ts — 783 pass, 42 skip (paid E2E), 0 fail.
Ship-workflow review re-dispatched the testing specialist against the full
updated diff (including the new minimal-change eval) and found 3 more real
bugs in the same family as the earlier fix pass:
- matchesUnnegated's fixed 20-char negation lookback was too narrow: the
patterns it guards have their own internal gaps up to 80 chars, so "I do
not think it's worth extracting the payload into columns" (negation >20
chars before the match) would be misread as an unnegated recommendation.
Now scans back to the start of the current sentence (last ./!/? before the
match) instead of a fixed count — matches the same period-bounded
assumption the calling patterns already make.
- Verified during the fix: neither "not"/"never"/etc. nor the wider window
catches contrastive phrasing ("add this field to the model RATHER THAN
create a separate table") — the rejected alternative matches the pattern
just as strongly as a genuine recommendation, with no negation word
anywhere nearby. Added "rather than"/"instead of" to the shared negation
list, benefiting all four callers, not just the new eval.
- The new minimal-change eval's acceptsInlineAddition check was missing the
matchesUnnegated guard entirely on its first alternative (unlike its
sibling recommendsSeparateModel a few lines above), and
recommendsSeparateModel's verb list missed common recommendation phrasings
(introduce, build, spin out, break out) — deliberately did NOT add "add",
since "add this new field to the existing model" is exactly how the
CORRECT answer gets phrased.
Added 3 unit tests reproducing the sentence-boundary and contrastive-phrasing
fixes directly (not just via the paid E2E path).
bun test test/helpers/e2e-helpers.test.ts test/skill-e2e-plan.test.ts
test/skill-validation.test.ts test/parity-suite.test.ts test/touchfiles.test.ts
test/gen-skill-docs.test.ts — 781 pass, 42 skip (paid E2E), 0 fail.
garrytan/gstack#1048 comment asked for concrete evaluations covering three
cases: normalized models, justified JSON fields, and minimal-change cases.
The prior commits covered the first two (data-model-bias, legitimate-json,
measured-denorm) but not the third — a plan where keeping a single trivial
field inline (not extracting a new model) is the correct call. This was
independently flagged by the ship-workflow coverage audit as the highest-
severity gap in this branch's test coverage.
Adds plan-eng-review-data-model-minimal-change: a synthetic plan adding one
nullable timestamp field to an existing model, no polymorphism, no JSON, no
independent query pattern or consumer. Asserts the skill does NOT reflexively
recommend extracting a separate model for it — exercising the "unless the
extra job is a single trivial field..." exception added to cognitive pattern
#12 and the SRP checklist item in 5374987c.
Also adds test/helpers/e2e-helpers.test.ts: free, deterministic unit tests
for matchesUnnegated() and setupPlanEngReviewFixture(), which were previously
only exercised indirectly by the paid, EVALS=1-gated E2E tests (also flagged
by the coverage audit — a bug in either could silently flip an eval's
pass/fail verdict and look like ordinary LLM wording variance). Along the
way, found and documented a real limitation: the negation-window check
correctly handles multiple sentences (period-bounded), but a single
comma-spliced sentence containing both a negated and a positive match can
still be absorbed into one greedy match — accepted as a known edge case,
since real review prose reliably separates points with periods/bullets.
bun test test/skill-validation.test.ts test/parity-suite.test.ts
test/touchfiles.test.ts test/skill-e2e-plan.test.ts test/helpers/e2e-helpers.test.ts
test/gen-skill-docs.test.ts — all pass, 0 fail.
Ran /review on this branch's diff against main. The always-on testing
specialist found real correctness bugs in the two eval blocks added in
33fbb58b:
- pushesToSplitTheField (legitimate-JSON eval) was negation-blind: a
CORRECT "I would NOT extract the payload into columns" answer contains
the same verb+payload+column proximity as an incorrect one, so it could
fail the test on a passing review. Also missing "normalize" from its verb
list, so a real regression phrased as "normalize this payload into
columns" would slip through undetected.
- rejectsAsUnjustified (measured-denorm eval) missed common rejection
phrasings ("instead of denormalizing", "rather than snapshotting"), and
its normali[sz]e-instead/back/it clause was loose enough to false-positive
on approving language ("stays normalized elsewhere; keep it that way").
- recommendsSeparateModel (pre-existing data-model-bias eval, touched by
this branch to fix its fixture) broke on backtick-wrapped model names, a
common LLM markdown convention.
Added matchesUnnegated() to test/helpers/e2e-helpers.ts — walks every regex
match and checks the ~20 chars before it for a negation word before counting
it as a positive signal — and used it for the split-detector. Broadened the
rejection regex and tightened its false-positive-prone clause. Stripped
markdown formatting before the backtick-affected check. Verified all four
fixes against representative sentences (correct-negative, actual-positive,
previously-missed-regression, and previously-false-positive cases) before
committing — all passed as expected.
Also extracted the maintainability specialist's DRY finding: the three
data-model E2E blocks (bias, legitimate-json, measured-denorm) each
duplicated the same ~20-line mkdtemp/git-init/write-plan/copy-skill fixture
setup. Extracted into setupPlanEngReviewFixture() in e2e-helpers.ts.
bun test test/skill-validation.test.ts test/parity-suite.test.ts
test/touchfiles.test.ts test/gen-skill-docs.test.ts test/skill-e2e-plan.test.ts
— 768 pass, 39 skip (paid E2E), 0 fail.
PR review feedback (garrytan/gstack#1071) flagged this as release metadata
that doesn't belong in a feature PR, and Copilot separately flagged that the
bump had already drifted (VERSION at 1.1.2.0, package.json still at
1.1.1.0). Per this repo's own CHANGELOG convention, the version bump and
changelog entry belong at /ship time — covering every commit that actually
landed on the branch — not mid-review while the content is still being
revised. Reverts VERSION and package.json to match main's current
1.60.1.0 and drops the premature CHANGELOG entry; /ship will add the real
one once this branch is ready to merge.
PR review feedback (garrytan/gstack#1071), resolving the three Copilot
review threads plus the "add positive and counterexample evaluations" ask:
- test/skill-validation.test.ts: renamed the two tests still named "...to
minimal diff..." to "...to right-sized diff..." to match the actual bullet
text the assertions check (Copilot review comment).
- test/skill-e2e-plan.test.ts: updated the Data-Model Bias regression test's
header comment from "minimal diff" to "right-sized diff" (Copilot review
comment); fixed the same test's fixture setup to also copy plan-eng-review's
sections/ directory — main's carve refactor moved the Architecture Review
and Data Model Review Checklist out of SKILL.md into
sections/review-sections.md after this test was written, so the sandbox
was missing the very file the skill's STOP-Read directive points at.
- Added two new periodic-tier E2E cases exercising the counterexamples the
templates now document: "plan-eng-review-data-model-legitimate-json"
(a Stripe-webhook-style JSONField the skill should NOT push back on) and
"plan-eng-review-data-model-measured-denorm" (a scoped denormalization
backed by a stated APM/load-test measurement the skill should NOT insist
on normalizing away). Both registered in test/helpers/touchfiles.ts
(periodic tier, same classification as the existing bias regression test).
Verified: bun test test/skill-validation.test.ts test/parity-suite.test.ts
test/touchfiles.test.ts test/skill-e2e-plan.test.ts — 768 pass, 39 skip
(paid E2E, requires EVALS=1), 0 fail.
PR review feedback (garrytan/gstack#1071): the data-model bias guidance was
overly absolute ("diff size is NOT a goal", "three clean models... in all
respects", "JSONField is not an escape hatch") and lacked counterexamples for
cases where JSON columns or denormalization are the right call.
- "Data model exception to right-sized diff" bullet (both skills): reframes
from an absolute rule to "don't let diff size alone decide the shape",
explicitly allows denormalization backed by a measurement, and adds the
counterexample of a field that would only ever be read through one FK join
with no independent consumer.
- JSONField bullet (both skills): keeps the core diagnostic but adds concrete
legitimate uses — third-party webhook/OAuth payload caching, open-ended
preference bags, event payloads shaped by an external producer, and
schemas still being discovered.
- plan-eng-review cognitive patterns #11 (normalize first) and #13
(structure beats blobs) get the same measured-reason / known-polymorphism
counterexamples.
- Architecture Review's "Data model honesty" bullet and the "Single
Responsibility" and "JSONField as polymorphism escape hatch" checklist
items (now living in sections/review-sections.md.tmpl per main's carve
refactor) get matching nuance.
Bumps carve-guards.ts skeleton-size budgets for plan-eng-review (69_000 ->
70_000) and plan-ceo-review (91_000 -> 92_000) to cover the added prose;
verified against test/parity-suite.test.ts (13/13 pass) and
test/skill-validation.test.ts's load-bearing-bullet guardrails (still pass —
none of the anchor phrases the static tests check were removed).
Merges origin/main (a3259400) into fix-schema-consolidation-bias. Resolves
conflicts in CHANGELOG.md, VERSION, and plan-eng-review/SKILL.md(.tmpl):
main refactored plan-eng-review's review body into an on-demand section
file (sections/review-sections.md), so the schema-normalization Data Model
Review Checklist added by this branch's fix (abafb381) was re-homed there
instead of staying inline. Bumped VERSION/package.json/CHANGELOG to
1.60.2.0 on top of main's 1.60.1.0. Updated two stale assertions in
test/skill-validation.test.ts that still checked SKILL.md directly instead
of the carved sections file, and widened two parity-suite size budgets in
test/helpers/carve-guards.ts to account for the legitimate content growth.
urlBlocklistFilter compared URLs against the exfiltration blocklist with
case-sensitive substring checks, so an uppercased sink (https://WEBHOOK.SITE/x)
bypassed the guard and reached the scoped browser agent. Normalize the page URL
and extracted content URLs to lowercase before comparing, and make URL
extraction scheme-insensitive so HTTPS:// links are still caught.
Fixes#2190
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parses package.json overrides + every resolved basic-ftp specifier in bun.lock
(including nested paths like get-uri/basic-ftp) and fails if any is below 5.3.1.
Deterministic and offline. Fires on the pre-fix tree (basic-ftp@5.2.0) and would
also catch a direct-dependency-only bump that leaves a nested vulnerable copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
basic-ftp reaches the tree only transitively:
puppeteer-core > @puppeteer/browsers > proxy-agent > pac-proxy-agent > get-uri > basic-ftp
Versions <= 5.3.0 carry four HIGH advisories, all fixed in 5.3.1:
- GHSA-chqc-8p9q-pq6q (CVE-2026-39983) FTP command injection via CRLF
- GHSA-6v7q-wjvx-w8wg incomplete CRLF protection (USER/PASS + MKD bypass)
- GHSA-rpmf-866q-6p89 DoS via unbounded multiline control-response buffering
- GHSA-rp42-5vxx-qpwr DoS via unbounded memory in Client.list()
Pin via a bun `overrides` entry rather than a phantom direct dependency.
Overriding forces every basic-ftp in the tree to 5.3.1, including get-uri's
nested copy; `bun audit` then reports zero basic-ftp advisories (total 37 -> 33,
HIGH 13 -> 9). A direct-dependency bump leaves get-uri/basic-ftp at the
vulnerable version and audit still flags all four.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
endpoint-hash returns "local" for stdio/PGLite, but validators only
allowed hex suffixes, so setup-gbrain could not persist trust policy.
Co-authored-by: Cursor <cursoragent@cursor.com>
Exercise resultFromGeminiStream against current stream-json fixtures
(including empty-success hardening). Recognize GEMINI_API_KEY and map
IneligibleTierError to auth — personal OAuth free-tier is no longer
supported by gemini CLI.
Co-authored-by: Cursor <cursoragent@cursor.com>
GeminiAdapter was reading message.text and result.usage, so current CLI
content/stats events produced empty $0 success rows. Accept content with
an assistant role guard, stats token fallbacks, init model, and treat
empty exit-0 output as an error (#2159).
Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause of months of silent local failure: the sandbox copied skill dirs to
the repo root, but claude >= 2.x resolves slash commands strictly from
registered skills, so /autoplan short-circuited with 'Unknown command' (0
turns, ~1s) on every attempt. Install /autoplan + review skills at
project-level .claude/skills/ (same pattern as skill-routing-e2e).
Also: the transcript filter matched entry.type === 'tool_use', a shape that
never appears at the top level of raw stream-json, so assertions only ever saw
the final result text; filter on assistant/user events instead. Hang
protection accepts the Phase 1 review dispatch (Agent/Task tool call carrying
review instructions) as progress evidence, since full Phase 1 completion is
15+ min of subagent work. Budget raised to 10 min / 40 turns.
Invisible in CI: the file is in neither evals.yml nor evals-periodic.yml
matrices (coverage decision filed in TODOS.md).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
David Grant
Joshua France
t
Sinabina
Sina
Garry Tan