mirror of https://github.com/razor-ai/soup.git
1e822af461
5 LIVE parts closing axis 4 of the roadmap. Evals as first-class surface, not afterthought:
- Judge calibration: SCOPE/CJE-style bidirectional pairwise judging in eval/calibrate.py
with PairwiseJudgement / fit_position_bias / conformal_threshold +
ensure_judge_calibrated production gate that refuses to score with an uncalibrated
judge (RuntimeError on None / calibrated=False / low agreement / extreme bias).
- Behaviour battery (soup eval behavior): closed allowlist of XSTest / HarmBench /
JailbreakBench / ELEPHANT / SycEval with 5 tiny bundled redacted probe sets under
soup_cli/data/_fixtures/behavior/. Word-boundary regex agreement rejects
"safe" in "unsafe" false positives. Pre/post diff with OK/MINOR/MAJOR verdict
(matches v0.26 / v0.56 taxonomy).
- Capability auto-suite (soup eval capability): MMLU-Pro / GPQA / BBEH / AIME /
MATH-500 / HumanEval+ / SWE-bench-Verified with full / fast / math / code profile
selector. Emits (benchmark, lm-eval task) manifest for downstream
soup eval benchmark chaining.
- CheckList DSL (soup eval checklist): Ribeiro et al. 2020 MFT / INV / DIR test kinds
rendered from YAML. Word-boundary matching prevents "and" matching "sand".
Per-test pass/fail + OK/MINOR/MAJOR overall verdict.
- IRT eval-cost optimizer (soup eval irt-subset): 1PL Rasch closed-form fit on
per-item correctness signals + high-info subset selector (full / small / tiny
profiles). 5-10x cut in eval bills without losing ranking power.
Cross-cutting hardening (review-fix coverage across 2 review waves):
- TOCTOU defence: every new read path uses O_NOFOLLOW + os.fstat on SAME fd
(load_checklist_spec, load_response_rows, _read_evidence_json). Earlier
double-lstat-on-path was a race the attacker could win by swapping the file
between calls.
- Namespace-package safety: load_battery_probes uses importlib.resources.files
Traversable / op + as_file (was Path(os.path.join(str(pkg_root), ...)) which
silently fails is_file() on MultiplexedPath installs).
- Word-boundary regex agreement in behavior_battery + checklist_dsl.
- CLI _validate_run_id gate; 16 MiB --evidence cap with O_NOFOLLOW;
_MAX_ROWS=1_000_000 cap counts skipped lines toward total in load_response_rows.
- INV empty-string normalisation no longer spuriously passes.
- _write_json_output / _read_evidence_json / _validate_run_id dedup helpers in
commands/_eval_v0650.py.
Review fixes: 0 CRITICAL + 6 HIGH + 9 MEDIUM + 7 LOW resolved across 2 waves.
Test count: 10306 -> 10577 (+271 net across test_v0650_part_{a,b,c,d,e}.py +
test_v0650_followups.py). Full suite 10577/10577 passing. 0 regressions.
Step 6 smoke: every new CLI command + 3 failure modes exercised end-to-end
(behavior with --evidence happy + MAJOR exit 2; capability fast with output;
checklist with real YAML; irt-subset on 600-row synthetic data; unknown
battery / size / kind all exit 2; outside-cwd evidence rejected).
Known limitations:
1. Live lm-eval-harness invocation deferred — soup eval capability emits the
manifest for downstream soup eval benchmark chaining (Typer commands aren't
safe to re-enter; matches v0.46.0 / v0.44.0 design).
2. Live model-driven soup eval behavior deferred — without --evidence, emits a
neutral OK report (v0.65.1).
3. Behaviour battery probe sets ship as tiny redacted placeholders — operators
pull real harmful prompts from upstream papers.
4. IRT model is 1PL Rasch only (2PL / 3PL deferred to v0.65.x).
Step 6 quirk worth noting: --evidence containment rejects /tmp/ on Windows
WSL bash; operators must run from cwd or pass cwd-contained paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| _fixtures | ||
| providers | ||
| templates | ||
| traces | ||
| __init__.py | ||
| augment.py | ||
| chat_templates.py | ||
| collators.py | ||
| formats.py | ||
| loader.py | ||
| loss_mask.py | ||
| sft_format.py | ||
| validator.py | ||