899ad8edf7
test(eval_gate): strip ANSI escapes in train --help CI assertion
...
`test_train_gate_flag_accepted` asserted `"--gate" in result.output`, but
Typer/Click under CI emits ANSI color codes that split the flag name into
non-contiguous chars: `\x1b[1;36m-\x1b[0m\x1b[1;36m-gate\x1b[0m`. The literal
"--gate" substring is never present. All 9 OS × Python combos failed on the
v0.26.0 Parts B-E push.
Fix: strip ANSI via regex before checking. Also assert on "eval-gated" from
the option description to double-check the flag is wired to its help text.
CI-only / tests-only: no soup_cli/ changes, no version bump needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-20 21:51:23 +05:00
ddab34115c
feat(v0.26.0): Parts B-E — Eval Gate, Trace-to-Pref, Quant-Check, Soup Cans
...
Closes the v0.26.0 "Red and Blue Ocean" flywheel after Part A (Registry):
Train (eval-gated) -> Registry -> Deploy (quant-check) -> Trace-to-Pref -> Train.
Part B — Eval-Gated Training:
- soup_cli/config/schema.py: EvalGateConfig (enabled/suite/every_n_epochs/
regression_threshold/baseline/on_regression) + TrainingConfig.eval_gate field
- soup_cli/eval/gate.py: EvalSuite, GateTask, run_gate, resolve_baseline,
load_suite; baselines from registry:// or file
- soup_cli/monitoring/callback.py: on_epoch_end + _run_eval_gate with fail-safe
error handling (structured errors treated as regressions under on_regression=stop)
- soup_cli/commands/train.py: --gate <suite.yaml> shortcut flag
- soup_cli/commands/eval.py: gate subcommand (stub generator; live scoring v0.26.1)
Part C — Trace-to-Preference:
- soup_cli/data/traces/: parse_langchain, parse_openai, parse_soup_serve;
build_pairs from thumbs_up / regenerations / user_edit
- soup_cli/commands/data.py: from-traces + review subcommands
- PII warning panel, 100,000-line cap, path containment, Literal validation
Part D — Quant-Lobotomy Checker:
- soup_cli/eval/quant_check.py: classify_delta (OK/MINOR/MAJOR), run_quant_check,
resolve_model_ref with artifact kinds filter, table/json/markdown renderers
- soup_cli/commands/eval.py: quant-check subcommand
Part E — Soup Cans:
- soup_cli/cans/: Manifest + DataRef (Pydantic v2); pack_entry + fork_can
(100MB cap, dunder-key guard); safe tar extraction (filter='data' on py3.12+,
narrow fallback, manual symlink rejection + commonpath check)
- soup_cli/commands/can.py: pack/inspect/verify/fork subcommands
Shared utility:
- soup_cli/utils/paths.py: single is_under_cwd helper replacing 5 duplicates
(os.path.realpath + commonpath — Windows 8.3 short-name safe)
Tests: 103 new (29 eval_gate + 24 trace_to_pref + 23 quant_check + 27 cans)
Full suite: 2511 passed on Windows Python 3.10.
Security hardening (review-driven, all severities fixed):
- EvalGateConfig bounds; GateTask null-byte + judge URL scheme allowlist
- Narrow except in _safe_extract so TarError from filter='data' is not swallowed
- resolve_model_ref artifact kinds filter (avoid wrong artifact)
- Manifest.author cap + null/newline rejection; created_at ISO-8601 validation
- fork_can dunder-key + null-byte rejection (prototype pollution prevention)
- fork_can size cap (100MB matches pack_entry)
- inspect_can/read_config refuse paths outside cwd
Docs:
- README.md: v0.26.0 "New in" block (flywheel); 43 recipes; all new commands
in All Commands list; version examples bumped to 0.26.0; Windows-safe arrows
- CLAUDE.md: architecture + test table + schema + CLI + security section
extended with B/C/D/E; phase vs Part terminology clarified; release
checklist step 18 adds Known Limitations section; step 20 adds comment
template; step 21 adds completeness check via gh issue list --milestone
- SECURITY.md: per-Part security notes (B/C/D/E) under v0.26.0
- CONTRIBUTING.md: test count + directory tree updates
Local smoke: version, eval gate, eval quant-check (table + json),
data from-traces, data review, can pack/inspect/verify/fork — all happy-path
end-to-end. Fixed Unicode arrows (U+2192) in can.py + gate.py that crashed on
Windows CP1252 consoles.
Deferred to v0.26.1 (known limitations, filed as issues post-release):
- eval gate/quant-check live model scoring (stub generator currently)
- data from-traces quality.py judge validation; serve --trace-log collector
- can run + can publish + orchestrator
- eval --attach-to-registry flag; export auto-artifact registration
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-20 21:37:05 +05:00
Alpamys