6cb1abab8f
feat(eval): soup ship — SHIP / DON'T-SHIP verdict (v0.71.25)
...
Add `soup ship`, a binary SHIP / DON'T-SHIP verdict after fine-tuning: it
SHIPs only when (leg 1) the task metric strictly improved AND (leg 2) no
general benchmark regressed past a forgetting threshold (default 0.05
absolute points) — otherwise DON'T SHIP, even if the task metric looks
great. The moat is leg 2 (catastrophic-forgetting gate) fused with the
task win into one decision. Exit: 0=SHIP, 2=DON'T SHIP, 1=runtime error.
- utils/ship_verdict.py: pure engine (no top-level torch) — frozen
TaskWin/BenchmarkDelta/ShipVerdict + decide_ship (single source of
truth for the threshold) + compute_benchmark_deltas + render/serialize.
- commands/ship.py: Typer command; --evidence offline path + live
metric/judge leg-1 + mini(default)/lm-eval leg-2; --baseline/--output.
- Reuses run_eval / JudgeEvaluator / ForgettingDetector / resolve_baseline
/ _run_lm_eval / live_eval.make_generator.
- Hardening: --evidence O_NOFOLLOW + size cap; --task-eval cwd-contained;
--judge-model urlparse SSRF guard; lm-eval model_args injection guard;
--general-suite bounded.
Schema (ShipConfig) deferred — v1 is CLI-only. Pairwise judge win-rate is
a planned fast-follow. +79 tests (14514 -> 14593).
2026-06-27 23:37:35 +05:00
0d55ba9e46
feat(serve): serve-time MoLE + per-request vector banks + epoch RAFT shuffle (v0.71.17)
...
Closes #259 (soup serve --mole: load base + N frozen task LoRAs + mole_gate.pt,
blend per-token at decode; train writes mole_manifest.json).
Closes #260 (soup serve --bank active user per-request via contextvars.ContextVar,
no cross-request leak; streaming path re-selects in-context).
Closes #253 (data.raft_epoch_shuffle: re-permute golden/distractor docs each epoch;
epoch=0 == legacy order).
Closes #254 (soup diagnose --citation-style / --shuffle-seed into the live probe).
Fix: MoLE train() returns initial_loss/final_loss/total_steps/duration_secs so
task=moe_lora_routing completes cleanly (surfaced by the #259 smoke).
Validated live on SmolLM2-135M (RTX 3050). 13595 -> 13656 tests.
2026-06-08 20:06:43 +05:00
a2287dd6a5
feat(rag): RAFT span-mask trainer + RA-DIT auto-link + live steering + eval citation (v0.71.10)
...
Lifts the v0.62.0 RAG-family schema-only stubs to live, validated on SmolLM2-135M:
- #199 RAFT: data.format=raft trains answer-only (prompt span masked to -100,
[doc-N] citation ids, deterministic doc shuffle by raft_shuffle_seed); rows
whose prompt fills max_length are dropped with a warning. New utils/raft.py +
trainer/raft.py (RaftDataCollator + weighted-CE _RaftTrainer).
- #200 soup ra-dit: one-shot two-stage orchestrator (train retriever -> record
it as the generator's paired retriever -> train generator); a generator-stage
`soup train` with no retriever set auto-links the latest RA-DIT retriever from
the Registry. New utils/ra_dit_run.py + commands/ra_dit.py.
- #201 soup steer train/apply + soup serve --steer: live CAA/ITI/RepE fit from
{positive, negative} pairs + decode-time forward hook (transformers backend).
Lifts the steering.py apply_steering/build_steering_vector stubs.
- #202 soup eval citation + citation-span per-token loss boost + 7th `citation`
failure mode in soup diagnose. New commands/_eval_v07110.py +
diagnose/citation.py.
Review fixes (3 agents, all CRITICAL->LOW): markup-escaped autolink advisory;
shared enforce_under_cwd_and_no_symlink + O_NOFOLLOW on every new file read;
steering-artifact containment; honest RA-DIT docs (records pairing, no weight
fusion); public validate_ra_dit_config_path + render_raft_prompt; repe/iti
require >=2 pairs; eval citation --shuffle-seed.
Full suite: 13034 passed, 106 skipped (13142 collected). ruff clean.
2026-06-03 19:28:45 +05:00
823456c1a5
feat(probe): real probe weights, SAE auto-download, truth/harm, interference --measure, capture-activations (v0.71.8)
...
Closes #216 , #217 , #218 , #219 . Partial #215 (calibrated vectors upstream-gated).
- #215 probe_kernel.py: compute_contrast_probe + load_probe_weights
(.npz/.npy/.safetensors, O_NOFOLLOW, allow_pickle=False, cwd-contained);
soup probe sleeper --weights. Synthetic seed fallback retained.
- #216 hubs.snapshot_download (SSRF-hardened, home/cwd/tmp cache, TOFU gate)
+ sae_diff.download_sae (allowlist-before-network + symlink-escape guard);
soup probe sae-diff --auto-download.
- #217 truth_probe.py + harm_probe.py over probe_kernel; soup probe truth/harm;
probe pack ships truth+harm per base.
- #218 interference_live.measure_interference_losses (live PEFT multi-adapter,
add_weighted_adapter cat off-diagonal); soup probe interference --measure.
- #219 live_eval.extract_layer_activations + resolve_layer_module PEFT-fallback;
soup train --capture-activations writes <output>/activations/activations.json.
Test count 12771 -> 12917 (+146 in tests/test_v0718.py). Step-6 smoke on
SmolLM2-135M (RTX 3050) green; caught + fixed a PEFT-wrapper layer-resolution bug.
2026-06-03 15:00:27 +05:00
f097528ac0
feat(eval): live eval runners — advise/tunability/capability/behavior/diagnose (v0.71.7)
...
Closes #161 , #162 , #208 , #211 , #212 , #165 .
New utils/live_eval.py shared model-loading layer (lazy torch/transformers/peft):
load_model_and_tokenizer, make_generator/make_multi_generator, compute_eval_loss,
lora_probe, measure_logit_agreement, token_f1.
- #161 soup advise --probe-model: live zero/few-shot token-F1 + LoRA probe
- #162 base_model_proximity via held-out logit agreement
- #208 soup tunability --live: per-candidate LoRA probe
- #211 soup eval capability --live --model: lm-eval-harness per task (per-task isolation)
- #212 soup eval behavior --base-model: live pre/post battery diff
- #165 soup diagnose --base-model: utils/diagnose/live.py runs all 6 probes live
Heuristic/neutral paths preserved when no model is supplied. Both new JSONL
readers open with O_NOFOLLOW after cwd-containment (TOCTOU close). +68 tests
(12703 -> 12771). Smoked end-to-end on SmolLM2-135M (RTX 3050).
2026-06-03 00:05:36 +05:00
a1463bf716
feat(v0.71.6): live build runner + Magpie generator + 2PL/3PL IRT + augment fix
...
Lift the v0.69.0 deferred stubs to live + extend IRT + fix a real bug:
- #231 soup build materialises (5 built-in transforms, table/view/incremental
with SQLite-tracked config-fingerprint cache key, atomic JSONL, --output-dir)
- #232 soup data gen-magpie live (ollama/vllm raw-completion harvest; anthropic
rejected; optional --quality-filter; dedup-before-response)
- #167 tokenizer-aware memorization probe (sub-word/BPE overlap, library-only)
- #213 soup eval irt-subset --model 2pl|3pl (joint coordinate-ascent MLE)
- #75 fix soup data augment --provider ollama|vllm ImportError + QA log
Security: validate_ollama_url/validate_vllm_url reject 0.0.0.0; augment output
containment+symlink reject; magpie response-body cap.
Tests 12581 -> 12703 (+122 in tests/test_v0716.py). Full suite green, ruff clean.
2026-06-02 22:06:12 +05:00
1f63393421
feat(v0.71.5): ingest/data/prompt/drift polish
...
Closes #157 , #205 , #207 , #149 , #164 , #163 . Defers #204 (live SaaS pull —
paid accounts, infra-blocked, kept open).
- #164 : get_metric_series falls back to eval_results when metrics is empty
- #163 : build_verdict confidence biased by advise_history (same project+choice,
>=3 precedents); decision never changes
- #207 : shared utils/webhooks.py (SSRF-hardened) + --slack-url/--discord-url on
ingest/prune-prompt/ab/active-sample; ab fires only on a decision
- #205 : soup prune-prompt --tokenizer (token-prefix detect + decode remainder,
boundary-safe)
- #149 : DynamicCurriculumCallback buckets by loss/perplexity percentile;
length keeps round-robin
- #157 : soup data push/forge --hub modelscope|modelers (data score N/A)
107 new tests in tests/test_v0715.py (12474 -> 12581). ruff clean.
2026-06-02 14:34:36 +05:00
514761c89a
feat(env,lock,serve,eval): v0.71.1 — quick wins + wiring (7 closures)
...
Closes #195 #210 #214 #224 #230 #233 #209 .
- soup env fix: print-only install-plan renderer from soup-env.lock
(uv-pip / requirements; non-pip entries surfaced as comments). (#209 )
- soup lock write --env-lock: auto-derive --env-hash from soup-env.lock
via new compute_env_hash (excludes created_at). (#224 )
- soup serve --record-thumbs <db>: capture thumbs-up/down into the
local-RL SQLite + POST /v1/thumbs (transformers backend). (#230 )
- Judge-calibration persistence: JudgeCalibrationReport.to_dict +
write/load_judge_calibration + judge_calibration registry kind; load
re-validates the frozen dataclass with cwd/symlink containment. (#214 )
- soup completions: introspect a base model's real LoRA target modules
(config-only AutoConfig, local_files_only, never networks/raises). (#210 )
- Bundled MUSE + WMDP unlearning eval fixtures; WMDP forget rows ship
REDACTED (Soup never bundles verbatim hazardous content). (#195 )
- build_dag.validate_build_source: cwd-containment + symlink rejection. (#233 )
Review-fix hardening (consolidated python+code+security+tdd, 0 CRIT/0 HIGH):
load_judge_calibration containment + friendly missing-field ValueError;
serve thumbs success-print escape; env_fix --output Optional[str];
empty --env-hash auto-derives; render_install_plan PEP 440 docstring note.
Tests: 12071 -> 12134 (12044 passed, 90 skipped, 2 deselected).
2026-06-01 14:12:30 +05:00
7724353ac6
docs: trim README to a 238-line front door; move feature reference to docs/
...
The README had grown to 5046 lines (195 sections) — roughly one deep-dive per
feature accreted over 70 releases. Split it into a concise front door plus a
public docs/ tree:
- README (5046 -> 238 lines): hero, why, quickstart, config, a Documentation
map, data formats, common commands, models, Docker, requirements, dev.
- docs/*.md: all 185 feature sections preserved verbatim, grouped into 10 themed
guides + an index. Every original line is accounted for (content-conservation
checked); all 235 internal links + anchors verified to resolve.
- un-gitignore docs/ (it was empty); fix a pre-existing dangling
docs/QUANTIZATION.md link; correct the stale `ruff check soup_cli/` ->
`src/soup_cli/` reference in the Development section.
No version bump: docs-only — rides into the 0.71.0 deps-split release.
2026-05-31 20:10:59 +05:00
Alpamys