RomanNum3ral
/
gstack
mirror of https://github.com/garrytan/gstack.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

evals: fix context-save-list test that was 0-for-26 ($5.28, zero passes) 

Disposition for the eval store's only permanently-red test. Root cause:
the hide-other-branches assertions scanned the FULL output surface
(incl. bash tool_results), so any agent that ran ls on the checkpoints
dir — the natural first step of a list flow — surfaced all three seeded
filenames and failed, even when its user-facing listing filtered
correctly. The test punished the agent for looking at the directory.

The hide-assertions now scan the agent's FINAL TEXT (the listing the
user sees); showsMain keeps the broad surface for its documented reason.
Validated live in the final gate run rather than quarantined: the test
guards real behavior (branch filtering) and the assertion was the bug.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan 2026-08-15 08:52:30 -07:00
parent 9b9623e30d
commit 8d533f1a46
No known key found for this signature in database
GPG Key ID: C1F69E85C74EFE1D
1 changed files with 11 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
test/skill-e2e-context-skills.test.ts
View File

@ -447,12 +447,18 @@ Do NOT use AskUserQuestion.`,
// Broad surface: the list output may only appear in bash tool_result
// entries (find output, file reads) rather than the agent's final text.
const out = fullOutputSurface(result);
// Must show the main-branch save. Hide the other branches' saves.
// Match by filename timestamp (stable, unambiguous) plus a looser
// prose check.
// Must show the main-branch save. Match by filename timestamp (stable,
// unambiguous) plus a looser prose check.
const showsMain = /20260101-120000|main-work/.test(out);
const hidesAlpha = !/20260202-120000/.test(out);
const hidesBeta = !/20260303-120000/.test(out);
// The hide-assertions scan the FINAL TEXT only. This test went 0-for-26
// ($5.28 burned, zero passes) because they used the broad surface: any
// agent that ran `ls` on the checkpoints dir — the natural first step of
// a list flow — surfaced all three filenames in a tool_result and failed,
// even when its user-facing listing filtered correctly. What must hide
// the other branches is the LISTING the user sees, not the agent's eyes.
const finalText = result.output ?? '';
const hidesAlpha = !/20260202-120000|LISTCURR_ALPHA_TOKEN/.test(finalText);
const hidesBeta = !/20260303-120000|LISTCURR_BETA_TOKEN/.test(finalText);
const routed = skillCalls(result).includes('context-save');
const exitOk = ['success', 'error_max_turns'].includes(result.exitReason);