mirror of https://github.com/garrytan/gstack.git
4ab0269729
Extends the v1.25.1.0 AskUserQuestion recommendation-quality coverage to the
cross-model synthesis surfaces that were previously emitting prose without a
structured recommendation:
- /codex review (Step 2A) — after presenting Codex output + GATE verdict,
must emit `Recommendation: <action> because <reason>` line. Reason must
compare against alternatives (other findings, fix-vs-ship, fix-order).
- /codex challenge (Step 2B) — same requirement after adversarial output.
- /codex consult (Step 2C) — same requirement after consult presentation,
with examples for plan-review consults that engage with specific Codex
insights.
- Claude adversarial subagent (scripts/resolvers/review.ts:446, used by
/ship Step 11 + standalone /review) — subagent prompt now ends with
"After listing findings, end your output with ONE line in the canonical
format Recommendation: <action> because <reason>". Codex adversarial
command (line 461) gets the same final-line requirement.
The same `judgeRecommendation` helper grades both AskUserQuestion and
cross-model synthesis — one rubric, two surfaces. Substance-5 cross-model
recommendations explicitly compare against alternatives (a different
finding, fix-vs-ship, fix-order). Generic synthesis ("because adversarial
review found things") fails at threshold ≥ 4.
Tests:
- test/llm-judge-recommendation.test.ts gains 5 cross-model fixtures (3
substance ≥ 4, 2 substance < 4). Existing rubric correctly grades them.
- test/skill-cross-model-recommendation-emit.test.ts (new, free-tier) —
static guard greps codex/SKILL.md.tmpl + scripts/resolvers/review.ts for
the canonical emit instruction. Trips before any paid eval if the
templates drift.
Touchfile: extended `llm-judge-recommendation` entry with codex/SKILL.md.tmpl
and scripts/resolvers/review.ts so synthesis-template edits invalidate the
fixture re-run.
Verified: free `bun test` exits 0 (5/5 static emit-guard tests pass), paid
fixture passes 45/45 expect calls in 24s with the cross-model substance-5
fixtures correctly judged at >= 4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| providers | ||
| agent-sdk-runner.ts | ||
| benchmark-judge.ts | ||
| benchmark-runner.ts | ||
| claude-pty-runner.ts | ||
| claude-pty-runner.unit.test.ts | ||
| codex-session-runner.ts | ||
| e2e-helpers.ts | ||
| eval-store.test.ts | ||
| eval-store.ts | ||
| gemini-session-runner.test.ts | ||
| gemini-session-runner.ts | ||
| llm-judge.ts | ||
| observability.test.ts | ||
| pricing.ts | ||
| secret-sink-harness.ts | ||
| session-runner.test.ts | ||
| session-runner.ts | ||
| skill-parser.ts | ||
| tool-map.ts | ||
| touchfiles.ts | ||