mirror of https://github.com/razor-ai/soup.git
2 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
ca799f6fd3 |
feat(eval,registry): live gate scoring + registry attach (v0.33.0 Part A wave 1)
Closes #32, #35. (#34 soup can run/publish deferred to Part A wave 2.) #32 Live model scoring for `soup eval gate` + `soup eval quant-check`: - gate.run_gate now dispatches judge / benchmark / custom task types, wrapping each scorer in try/except so a backend failure produces score=None + error=str(exc) instead of a silent score=1.0 pass. - New _parse_judge_url splits ollama:// / http(s):// judge_model URLs into (provider, model, api_base) for JudgeEvaluator. - New _run_judge_task / _run_benchmark_task plug into existing eval/judge.py and eval/forgetting.py runners. - New quant_check.make_model_generator(model_path) wraps transformers AutoTokenizer + AutoModelForCausalLM into a generate_fn callable; greedy by default for reproducible scores; lazy-imported. - gate_cmd / quant_check_cmd build live generators when --model is given; fall back to deterministic stub on load failure so CI without GPUs still runs the orchestration layer. - GateTaskResult.score is now Optional[float] with new error: Optional[str]. - _print_gate_result renders ERROR + reason cleanly. #35 Registry attach hooks: - registry/store.py _VALID_KINDS extended with eval_results, tensorrt. - New registry/attach.py: attach_artifact, write_eval_json (cwd-containment via realpath+commonpath), lookup_entry_by_output_dir. - `soup eval custom` gains --attach-to-registry + --output (paired); on success writes JSON results and adds eval_results artifact row. - `soup export` gains --registry-id with auto-match by source --model output dir; auto-attaches the produced GGUF artifact. Failures here are warnings, not hard exits — export already succeeded. Tests: +19 in tests/test_part_a_wave1.py covering URL parser, error propagation across all 3 task types, score=None semantics, generator factory bounds + transformers mocking, registry attach helpers (containment + missing entry), and CLI integration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ddab34115c |
feat(v0.26.0): Parts B-E — Eval Gate, Trace-to-Pref, Quant-Check, Soup Cans
Closes the v0.26.0 "Red and Blue Ocean" flywheel after Part A (Registry): Train (eval-gated) -> Registry -> Deploy (quant-check) -> Trace-to-Pref -> Train. Part B — Eval-Gated Training: - soup_cli/config/schema.py: EvalGateConfig (enabled/suite/every_n_epochs/ regression_threshold/baseline/on_regression) + TrainingConfig.eval_gate field - soup_cli/eval/gate.py: EvalSuite, GateTask, run_gate, resolve_baseline, load_suite; baselines from registry:// or file - soup_cli/monitoring/callback.py: on_epoch_end + _run_eval_gate with fail-safe error handling (structured errors treated as regressions under on_regression=stop) - soup_cli/commands/train.py: --gate <suite.yaml> shortcut flag - soup_cli/commands/eval.py: gate subcommand (stub generator; live scoring v0.26.1) Part C — Trace-to-Preference: - soup_cli/data/traces/: parse_langchain, parse_openai, parse_soup_serve; build_pairs from thumbs_up / regenerations / user_edit - soup_cli/commands/data.py: from-traces + review subcommands - PII warning panel, 100,000-line cap, path containment, Literal validation Part D — Quant-Lobotomy Checker: - soup_cli/eval/quant_check.py: classify_delta (OK/MINOR/MAJOR), run_quant_check, resolve_model_ref with artifact kinds filter, table/json/markdown renderers - soup_cli/commands/eval.py: quant-check subcommand Part E — Soup Cans: - soup_cli/cans/: Manifest + DataRef (Pydantic v2); pack_entry + fork_can (100MB cap, dunder-key guard); safe tar extraction (filter='data' on py3.12+, narrow fallback, manual symlink rejection + commonpath check) - soup_cli/commands/can.py: pack/inspect/verify/fork subcommands Shared utility: - soup_cli/utils/paths.py: single is_under_cwd helper replacing 5 duplicates (os.path.realpath + commonpath — Windows 8.3 short-name safe) Tests: 103 new (29 eval_gate + 24 trace_to_pref + 23 quant_check + 27 cans) Full suite: 2511 passed on Windows Python 3.10. Security hardening (review-driven, all severities fixed): - EvalGateConfig bounds; GateTask null-byte + judge URL scheme allowlist - Narrow except in _safe_extract so TarError from filter='data' is not swallowed - resolve_model_ref artifact kinds filter (avoid wrong artifact) - Manifest.author cap + null/newline rejection; created_at ISO-8601 validation - fork_can dunder-key + null-byte rejection (prototype pollution prevention) - fork_can size cap (100MB matches pack_entry) - inspect_can/read_config refuse paths outside cwd Docs: - README.md: v0.26.0 "New in" block (flywheel); 43 recipes; all new commands in All Commands list; version examples bumped to 0.26.0; Windows-safe arrows - CLAUDE.md: architecture + test table + schema + CLI + security section extended with B/C/D/E; phase vs Part terminology clarified; release checklist step 18 adds Known Limitations section; step 20 adds comment template; step 21 adds completeness check via gh issue list --milestone - SECURITY.md: per-Part security notes (B/C/D/E) under v0.26.0 - CONTRIBUTING.md: test count + directory tree updates Local smoke: version, eval gate, eval quant-check (table + json), data from-traces, data review, can pack/inspect/verify/fork — all happy-path end-to-end. Fixed Unicode arrows (U+2192) in can.py + gate.py that crashed on Windows CP1252 consoles. Deferred to v0.26.1 (known limitations, filed as issues post-release): - eval gate/quant-check live model scoring (stub generator currently) - data from-traces quality.py judge validation; serve --trace-log collector - can run + can publish + orchestrator - eval --attach-to-registry flag; export auto-artifact registration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |