Alpamys 58d7d510bf feat(eval): soup eval design — derive evals from data (v0.55.0) ... Trainer libraries help you RUN evals — none help you DEFINE them. v0.55.0 closes that gap with 5 new subcommands: - soup eval design <data> --goal "..." → goal-conditioned EvalDesign (TF-IDF salience + scorer dispatch) - soup eval discover <data> → held-out canaries + memorization probes (farthest-first Jaccard clustering) - soup eval lock + soup eval coverage → SHA-256-checksummed artifact + gap analysis vs v0.54.0 task taxonomy - soup eval gate-install --baseline R → pre-push regression gate (paired-bootstrap CI, shlex.quote) - soup eval against B --candidate C → run-vs-run paired-bootstrap CI Heuristic / CPU-only — no GPU required. Lazy imports across all 6 new modules so `soup --help` startup remains < 200 ms. New registry artifact kinds: eval_suite, canaries. New tracker accessor: ExperimentTracker.get_metric_series(run_id, metric). Security policy (all atomic-write + read surfaces): - cwd containment via os.path.realpath + commonpath - unconditional os.lstat + stat.S_ISLNK rejection (TOCTOU defence) - atomic write via tempfile.mkstemp + os.replace - shlex.quote for shell-script generation (NO hand-rolled escape) - MappingProxyType on every registry / metric / scorer map - frozen dataclass on every public return type - bool-as-int rejection on every numeric input - DoS caps: 10k subsample for TF-IDF + clustering hot paths Review-fix coverage across 4 agents (python / security / code / tdd): 0 CRITICAL + 7 HIGH + 11 MEDIUM + 6 LOW resolved before commit. Tests: 8571 → 8676 (+105 net). Lint: ruff clean. Smoke: every CLI command + every failure mode exercised in /tmp/soup_smoke. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>		2026-05-15 12:44:14 +05:00
..
__init__.py	Initial project setup: CLI skeleton + config + trainer + data pipeline	2026-02-20 16:14:56 +05:00
tracker.py	feat(eval): soup eval design — derive evals from data (v0.55.0)	2026-05-15 12:44:14 +05:00