RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs: credit @Sanjays2402 for the benchmark gate-task fix (#315) 

Every `type: benchmark` eval-gate task had failed 100% of the time since
gate.py probed for a `forgetting.run_mini_benchmark` helper that never
existed, and told the user to install an extra that could not fix it.

The PR also deleted the test that let this ship: the old
test_benchmark_task_unavailable monkeypatched run_mini_benchmark to None
"to force the RuntimeError branch", but that function never existed, so
it passed for a different reason than its name claimed.

Closes #310.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys 2026-07-17 16:31:43 +05:00
parent 3b752b7b60
commit 85450309f4
2 changed files with 14 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
CHANGELOG.md
View File

@ -12,6 +12,18 @@ reproducing 70+ versions of notes.
## [Unreleased]
### Fixed
- **Eval-gate `type: benchmark` tasks now actually run.** `eval/gate.py` probed
for a `forgetting.run_mini_benchmark` helper that never existed, so every
`type: benchmark` task in a gate suite failed 100% of the time — while
advising an `[eval]` extras install that could not fix it. The gate now calls
`ForgettingDetector` directly (the same way `soup ship` already did), and an
unknown benchmark name fails with the list of valid names. Thanks
[@Sanjays2402](https://github.com/Sanjays2402)!
([#315](https://github.com/MakazhanAlpamys/Soup/pull/315), closes
[#310](https://github.com/MakazhanAlpamys/Soup/issues/310))
## [0.71.36] - 2026-07-16
**Data Moat II** — a semantic layer over your training data, plus two tools for

2
CONTRIBUTORS.md
View File

@ -52,6 +52,8 @@ Listed by first contribution. PR numbers link the work.
- Honor configured vocab expansion in the ORPO / SimPO / GRPO trainers ([#295](https://github.com/MakazhanAlpamys/Soup/pull/295))
- **Ekaanksh Patil** ([@Ekaanksh-dev](https://github.com/Ekaanksh-dev))
- Batch the PRM reward forward pass in `PRMScorer.__call__` (single `[B, T]` forward) ([#301](https://github.com/MakazhanAlpamys/Soup/pull/301))
- **Sanjay Santhanam** ([@Sanjays2402](https://github.com/Sanjays2402))
- Run built-in benchmark gate tasks through `ForgettingDetector` — every `type: benchmark` eval-gate task had always failed ([#315](https://github.com/MakazhanAlpamys/Soup/pull/315))
---