mirror of https://github.com/razor-ai/soup.git
1 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
ddab34115c |
feat(v0.26.0): Parts B-E — Eval Gate, Trace-to-Pref, Quant-Check, Soup Cans
Closes the v0.26.0 "Red and Blue Ocean" flywheel after Part A (Registry): Train (eval-gated) -> Registry -> Deploy (quant-check) -> Trace-to-Pref -> Train. Part B — Eval-Gated Training: - soup_cli/config/schema.py: EvalGateConfig (enabled/suite/every_n_epochs/ regression_threshold/baseline/on_regression) + TrainingConfig.eval_gate field - soup_cli/eval/gate.py: EvalSuite, GateTask, run_gate, resolve_baseline, load_suite; baselines from registry:// or file - soup_cli/monitoring/callback.py: on_epoch_end + _run_eval_gate with fail-safe error handling (structured errors treated as regressions under on_regression=stop) - soup_cli/commands/train.py: --gate <suite.yaml> shortcut flag - soup_cli/commands/eval.py: gate subcommand (stub generator; live scoring v0.26.1) Part C — Trace-to-Preference: - soup_cli/data/traces/: parse_langchain, parse_openai, parse_soup_serve; build_pairs from thumbs_up / regenerations / user_edit - soup_cli/commands/data.py: from-traces + review subcommands - PII warning panel, 100,000-line cap, path containment, Literal validation Part D — Quant-Lobotomy Checker: - soup_cli/eval/quant_check.py: classify_delta (OK/MINOR/MAJOR), run_quant_check, resolve_model_ref with artifact kinds filter, table/json/markdown renderers - soup_cli/commands/eval.py: quant-check subcommand Part E — Soup Cans: - soup_cli/cans/: Manifest + DataRef (Pydantic v2); pack_entry + fork_can (100MB cap, dunder-key guard); safe tar extraction (filter='data' on py3.12+, narrow fallback, manual symlink rejection + commonpath check) - soup_cli/commands/can.py: pack/inspect/verify/fork subcommands Shared utility: - soup_cli/utils/paths.py: single is_under_cwd helper replacing 5 duplicates (os.path.realpath + commonpath — Windows 8.3 short-name safe) Tests: 103 new (29 eval_gate + 24 trace_to_pref + 23 quant_check + 27 cans) Full suite: 2511 passed on Windows Python 3.10. Security hardening (review-driven, all severities fixed): - EvalGateConfig bounds; GateTask null-byte + judge URL scheme allowlist - Narrow except in _safe_extract so TarError from filter='data' is not swallowed - resolve_model_ref artifact kinds filter (avoid wrong artifact) - Manifest.author cap + null/newline rejection; created_at ISO-8601 validation - fork_can dunder-key + null-byte rejection (prototype pollution prevention) - fork_can size cap (100MB matches pack_entry) - inspect_can/read_config refuse paths outside cwd Docs: - README.md: v0.26.0 "New in" block (flywheel); 43 recipes; all new commands in All Commands list; version examples bumped to 0.26.0; Windows-safe arrows - CLAUDE.md: architecture + test table + schema + CLI + security section extended with B/C/D/E; phase vs Part terminology clarified; release checklist step 18 adds Known Limitations section; step 20 adds comment template; step 21 adds completeness check via gh issue list --milestone - SECURITY.md: per-Part security notes (B/C/D/E) under v0.26.0 - CONTRIBUTING.md: test count + directory tree updates Local smoke: version, eval gate, eval quant-check (table + json), data from-traces, data review, can pack/inspect/verify/fork — all happy-path end-to-end. Fixed Unicode arrows (U+2192) in can.py + gate.py that crashed on Windows CP1252 consoles. Deferred to v0.26.1 (known limitations, filed as issues post-release): - eval gate/quant-check live model scoring (stub generator currently) - data from-traces quality.py judge validation; serve --trace-log collector - can run + can publish + orchestrator - eval --attach-to-registry flag; export auto-artifact registration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |