RomanNum3ral
/
gstack
mirror of https://github.com/garrytan/gstack.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

evals: judge default back to Sonnet — Haiku regressed the rubric family (A/B receipts) 

The partial-diff rehearsal was the Haiku judge default's first live run
and it failed all three selected doc-rubric judges. Controlled A/B on
the identical health-rubric prompt: Haiku 2/2/2 vs Sonnet 4/3/4, both
with coherent reasoning — Haiku is simply a harsher grader on
long-document rubrics, and every >=4 threshold in skill-llm-eval was
calibrated against months of Sonnet baselines. Per D1a's
pin-on-regressors protocol the default reverts; a new
GSTACK_EVAL_MODEL_JUDGE override makes future recalibration a one-var
experiment. Haiku keeps the classifier-grade duties (pty hung/working,
warmup, distill via lib/eval-model.ts) and D1a's capture->Sonnet stands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan 2026-08-15 16:14:01 -07:00
parent e36c4881bc
commit f692da35d7
No known key found for this signature in database
GPG Key ID: C1F69E85C74EFE1D
1 changed files with 11 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
test/helpers/llm-judge.ts
View File

@ -56,11 +56,17 @@ export interface RecommendationScore {
* existing callers; pass a model id (e.g. claude-haiku-4-5-20251001)
* for cheaper bounded judgments like judgeRecommendation.
*/
// Default judge model is Haiku 4.5 (D1a, 2026-08): these are rubric-scoring
// calls, a duty Haiku is already proven at in this repo (claude-pty-runner's
// hung/working classifier, first-task-scaffold, hermetic-canary). Tests that
// need a stronger judge pass a model explicitly.
export async function callJudge<T>(prompt: string, model: string = 'claude-haiku-4-5-20251001'): Promise<T> {
// Default judge model: Sonnet. D1a tried Haiku 4.5 here and the first live
// run regressed the doc-rubric family — a controlled A/B on the identical
// health-rubric prompt scored 2/2/2 under Haiku vs 4/3/4 under Sonnet (both
// with coherent reasoning; Haiku is simply a harsher grader on long-document
// rubrics, and every >=4 threshold in skill-llm-eval was calibrated against
// months of Sonnet baselines). Per D1a's pin-on-regressors protocol the
// default stays Sonnet; recalibrating the 25 rubrics for Haiku is separately
// scoped work. Override per run with GSTACK_EVAL_MODEL_JUDGE; Haiku remains
// the right default for classifier-grade duties (pty hung/working, warmup,
// distill — see lib/eval-model.ts).
export async function callJudge<T>(prompt: string, model: string = process.env.GSTACK_EVAL_MODEL_JUDGE || 'claude-sonnet-4-6'): Promise<T> {
const client = new Anthropic();
const makeRequest = () => client.messages.create({