RomanNum3ral
/
gstack
mirror of https://github.com/garrytan/gstack.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

evals: judge default Sonnet -> Haiku 4.5 (D1a) 

The 25 doc-quality judges are rubric-scoring calls — a duty Haiku is
already proven at in this repo (pty hung/working classifier,
first-task-scaffold, hermetic-canary). Tests needing a stronger judge
pass a model explicitly. Note: eval-store judge costs were hardcoded
synthetic (0.02), so no baseline distortion. Re-baselined by the
periodic run in this branch's final verification.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan 2026-08-15 09:14:36 -07:00
parent 2ac8aac89c
commit d464412010
No known key found for this signature in database
GPG Key ID: C1F69E85C74EFE1D
1 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
test/helpers/llm-judge.ts
View File

@ -56,7 +56,11 @@ export interface RecommendationScore {
* existing callers; pass a model id (e.g. claude-haiku-4-5-20251001)
* for cheaper bounded judgments like judgeRecommendation.
*/
export async function callJudge<T>(prompt: string, model: string = 'claude-sonnet-4-6'): Promise<T> {
// Default judge model is Haiku 4.5 (D1a, 2026-08): these are rubric-scoring
// calls, a duty Haiku is already proven at in this repo (claude-pty-runner's
// hung/working classifier, first-task-scaffold, hermetic-canary). Tests that
// need a stronger judge pass a model explicitly.
export async function callJudge<T>(prompt: string, model: string = 'claude-haiku-4-5-20251001'): Promise<T> {
const client = new Anthropic();
const makeRequest = () => client.messages.create({