mirror of https://github.com/razor-ai/soup.git
test: record the MINI_MMLU coupling behind the 0.4 gate expectation (#315)
The benchmark gate-task test pins score == 0.4, which is 2/5 -- exactly two MINI_MMLU answers are "B" and the fake generate_fn returns "B". Editing the fixture in forgetting.py moves the number, so say so. Comment only, no logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
85450309f4
commit
780758a6ef
|
|
@ -131,6 +131,8 @@ class TestRunGateErrorPropagation:
|
|||
result = run_gate(suite, generate_fn=lambda _p: "B")
|
||||
row = result.task_results[0]
|
||||
|
||||
# 0.4 == 2/5: exactly two MINI_MMLU answers are "B". Editing that
|
||||
# fixture in forgetting.py moves this number.
|
||||
assert row.score == 0.4
|
||||
assert row.error is None
|
||||
assert row.passed is True
|
||||
|
|
|
|||
Loading…
Reference in New Issue