Alpamys
|
6cb1abab8f
|
feat(eval): soup ship — SHIP / DON'T-SHIP verdict (v0.71.25)
Add `soup ship`, a binary SHIP / DON'T-SHIP verdict after fine-tuning: it
SHIPs only when (leg 1) the task metric strictly improved AND (leg 2) no
general benchmark regressed past a forgetting threshold (default 0.05
absolute points) — otherwise DON'T SHIP, even if the task metric looks
great. The moat is leg 2 (catastrophic-forgetting gate) fused with the
task win into one decision. Exit: 0=SHIP, 2=DON'T SHIP, 1=runtime error.
- utils/ship_verdict.py: pure engine (no top-level torch) — frozen
TaskWin/BenchmarkDelta/ShipVerdict + decide_ship (single source of
truth for the threshold) + compute_benchmark_deltas + render/serialize.
- commands/ship.py: Typer command; --evidence offline path + live
metric/judge leg-1 + mini(default)/lm-eval leg-2; --baseline/--output.
- Reuses run_eval / JudgeEvaluator / ForgettingDetector / resolve_baseline
/ _run_lm_eval / live_eval.make_generator.
- Hardening: --evidence O_NOFOLLOW + size cap; --task-eval cwd-contained;
--judge-model urlparse SSRF guard; lm-eval model_args injection guard;
--general-suite bounded.
Schema (ShipConfig) deferred — v1 is CLI-only. Pairwise judge win-rate is
a planned fast-follow. +79 tests (14514 -> 14593).
|
2026-06-27 23:37:35 +05:00 |