mirror of https://github.com/garrytan/gstack.git
752ff50e11
Existing AskUserQuestion format-regression tests only regex-match
"Recommendation:[*\s]*Choose" — they confirm the line exists but say nothing
about whether the "because Y" clause is present, specific, or substantive.
Agents frequently produce the line with boilerplate reasoning ("because it's
better"), and the regex passes anyway.
Add judgeRecommendation:
- Deterministic regex parses present / commits / has_because — no LLM call
needed for booleans, and skipping the LLM when has_because is false avoids
burning tokens on cases that already failed the format spec.
- Haiku 4.5 grades reason_substance 1-5 on a tight rubric scoped to the
because-clause itself (not the surrounding pros/cons menu — that menu is
context only). 5 = specific tradeoff vs an alternative; 3 = generic
("because it's faster"); 1 = boilerplate ("because it's better").
- callJudge generalized with a model arg, default Sonnet for back-compat
with judge / outcomeJudge / judgePosture callers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| providers | ||
| agent-sdk-runner.ts | ||
| benchmark-judge.ts | ||
| benchmark-runner.ts | ||
| claude-pty-runner.ts | ||
| claude-pty-runner.unit.test.ts | ||
| codex-session-runner.ts | ||
| e2e-helpers.ts | ||
| eval-store.test.ts | ||
| eval-store.ts | ||
| gemini-session-runner.test.ts | ||
| gemini-session-runner.ts | ||
| llm-judge.ts | ||
| observability.test.ts | ||
| pricing.ts | ||
| secret-sink-harness.ts | ||
| session-runner.test.ts | ||
| session-runner.ts | ||
| skill-parser.ts | ||
| tool-map.ts | ||
| touchfiles.ts | ||