soup ship's leg 2 — the catastrophic-forgetting / regression gate that carries
the whole SHIP / DON'T-SHIP claim — was 15 trivia prompts scored by raw
substring containment (it credited "B" for "Berlin", "3" for "13") with zero
coverage for tool-calling, safety, or JSON. This makes the gate real.
- forgetting.py: score_answer/extract_mcq_letter replace the substring scorer
with answer-extraction (cue -> paren -> clause-terminating bare letter) +
boundary-aware token match. MINI_BENCHMARKS expanded (mmlu 26 / common_sense
24 / instruction 24) + new mini_arithmetic (36) so a 1-item flip trips 0.05.
BREAKING: an existing run's verdict can change (the old gate under-reported).
- eval/gate_suites.py (new): bundled offline general-suite registry, no torch.
DEFAULT_GENERAL_SUITE = the 4 MCQ suites + 3 behavioural JSONL suites
(mini_tool_call / mini_format_json / mini_safety) scored per-model-absolute
by the pure custom/diagnose scorers. _fraction_passing isolates a per-item
scorer exception (deep-JSON RecursionError scores as a failed item).
- ship.py: leg-2 scores bundled suites offline (base+tuned) before routing any
non-bundled name to lm-eval; default general suite = the full bundled set.
Exit-code taxonomy: usage errors move 2 -> 3 so exit 2 means only DON'T-SHIP
(a typo'd flag was previously indistinguishable from a caught regression).
- diagnose/__init__: "Six" -> "Seven" probes + re-export all 7 score_* fns;
removed the dead SUPPORTED_TASK_MODES "pairwise reserved" gate.
- Bundled gate fixtures ship in the wheel via the pyproject artifacts glob.
Every bundled item is original, hand-authored (no MMLU/GSM8K rows copied).
Test count 16288 -> 16330 (+42 in tests/test_v07138.py).
Alpamys