Folds in the doc-vs-reality cleanup the plan flagged: the docs referenced
soup train --reward-hack-detector / --reward-hack-halt but they were config-only.
Add both as CLI passthroughs mirroring --reward-hack-mitigation (validate value,
set cfg.training field, accelerate re-exec passthrough) so the docs are true and
the reward-hack CLI is consistent. +4 tests (test_v07126: 180 -> 184; full suite
-> 14788).
#81 Quant Menu for vision/audio modality
- config/schema: drop the `modality != "text"` rejection in
_validate_quant_menu_supported_tasks (mlx-backend gate retained) so the full
Quant Menu (gptq/awq/hqq:Nbit/aqlm/eetq/mxfp4/fp8) applies to vision+audio.
- trainer/sft: _setup_vision_transformers + _setup_audio_transformers call
build_quantization_config_for_loader (strict superset of the inline 4bit/8bit
BNB blocks they replaced); drop BitsAndBytesConfig import; retain the
prepare_model_for_kbit_training gate on (4bit,8bit,mxfp4).
#80 multipack DataLoader sharding under FSDP/DeepSpeed/DDP
- utils/multipack_trainer: get_train_dataloader routes the multipack DataLoader
through accelerator.prepare when num_processes > 1 so accelerate's
BatchSamplerShard shards whole FFD bins across ranks (even_batches=True avoids
the epoch-boundary collective hang; seed identical across ranks, no `+ rank`).
Single-process path unchanged. Defence-in-depth guard against an unconfigured
MagicMock num_processes.
Tests: +37 in tests/test_v07119.py (13770 -> 13807). ruff clean.
Supersedes the v0.40.5 vision/audio Quant-Menu and v0.40.4 multipack-FSDP
known-limitations. Full multi-GPU validation remains an INFRA-BLOCKED QA item.
#261 iterative_dpo._default_train_fn rendered output as a {dir: ...} mapping
that SoupConfig rejected; render it flat so the spawned soup train succeeds.
#246 CMA-ES merge now loads the base model once and reuses it across the
candidate population (_CachedBaseScorer) instead of reloading per candidate.
#245 soup loop estimate_cost wires run_cost.estimate_run_cost_usd off the last
completed run instead of a 0.0 placeholder; never crashes the daemon.
#244 soup train --track-energy --energy-out persists the measurement JSON so
soup bom emit --energy can attach it to an ML-BOM.
#170 --diagnose-gate is RANK-aware: gate once per cluster (RANK==0), not per node.
Tests: 13476 -> 13511 (+35 in tests/test_v07115.py). Validated end-to-end on
SmolLM2-135M / RTX 3050.
CI failed on the HF-rate-limited runners: test_anchor_term_with_file did a
live from_pretrained that 429'd, so it failed AND its unique MiniLLM-anchor
lines went uncovered, tipping the 77% gate to 76.77% on exactly those jobs
(macos + 3.11 stayed green where the cache warmed).
- Skip test_anchor_term_with_file on OSError (offline / rate-limited) instead
of failing.
- Add test_anchor_term_with_fake_model: a fake tokenizer + tiny nn.Module
exercise the identical _load_anchor + anchor_term lines with no network, so
coverage no longer depends on HF availability.
- Add TestReachableInternals cushion (prompt_compile._resolve_metric,
prompt_distill._build_provider_fn + default-provider wiring) so the gate
sits comfortably above 77% (the DSPy/TextGrad/GEPA optimiser bodies are
uncoverable without the [compile] extra).
Tests 13424 -> 13430.
Lift the v0.68.0 deferred-stub family to live (closes#225, #226, #227, #229):
- #229 local-rl train --once: harvest thumbs -> DPO/KTO/ORPO train via a
soup train subprocess (argv list, no shell); state table tracks last_train_at
(skip-on-no-new-thumbs + skip-on-insufficient-pairs); no --once renders a
systemd/launchd nightly scheduler scaffold. New local_rl_scheduler.py.
- #226 distill-prompt: call the teacher once per trace (Ollama/Anthropic/vLLM)
and write a real dataset (sft/kl -> messages; preference -> chosen/rejected).
- #225 compile / #227 compile-tools: live DSPy/GEPA/TextGrad dispatch behind the
new [compile] extra with a friendly ImportError when absent; injectable seams.
Security: reject \n/\r in the model id + shell-quote ExecStart args (systemd
injection defence). Fix: render train output as a plain string (schema-valid),
with a regression test against SoupConfig.
Tests 13329 -> 13424. Smoked end-to-end: real DPO train on SmolLM2-135M (RTX 3050)
+ real Ollama teacher distillation.
The v0.71.12 release commit updated the test count but missed the two new
modules in the Project Structure tree: trainer/mole_routing.py (MoLE Routing)
and utils/mod.py (Mixture-of-Depths). Docs-only; no version bump.
The v0.71.1 release commit (514761c) went red on CI for two reasons:
- test_flag_in_help asserted a raw "--record-thumbs" substring, but Rich
splits an option name's dashes with ANSI codes under CI's FORCE_COLOR
(it passes locally without color). Strip ANSI before the substring check.
- Coverage fell to 76.96% (< 77% gate): CI installs [dev], which has no
FastAPI, so the new /v1/thumbs endpoint + record-thumbs startup block in
serve.py are uncovered there. Restore the gate honestly (no lowering, no
pragma) by adding 19 genuine no-FastAPI tests for previously-uncovered
pure-CLI paths: lock show / lock check (no-drift / drift exit 3 / missing),
env check (no-drift / missing / drift exit 3), env fix error branches,
env lock null-byte output, and load_evidence_file (the
`eval unlearning --evidence` loader).
CI-equivalent (no-fastapi) coverage: 76.96% -> 77.24%. Tests: 12134 -> 12153.
- SECURITY.md: supported window 0.70.x -> 0.71.x
- CONTRIBUTING.md: dev-deps list now lists mypy + pre-commit and notes the
v0.71.0 deps-split ([dev] self-references [train], so torch & co are pulled in)
CLAUDE.md and .claude/plan.md are maintainer-local (gitignored), so any
committed file that links to or tells contributors to read/update them is
broken for anyone cloning the public repo. Repoint everything at the public
docs/ split (and the schema / CHANGELOG where relevant).
- CONTRIBUTING.md: "add a trainer/format" + "Version Bump Process" steps now
say README.md + the relevant docs/ page (was CLAUDE.md); removed the
"See CLAUDE.md for the complete release checklist" pointer (replaced with a
one-line public note: bump -> tag v* -> CI -> PyPI OIDC publish); reworded
the test-table note off CLAUDE.md
- AGENTS.md: removed the maintainer-local CLAUDE.md parenthetical
- advise.py / runs.py / quant_menu.py: reword docstring/comments that cited
.claude/plan.md or CLAUDE.md (comment-only; no behaviour change)
Acceptance: grep "CLAUDE.md" across *.md (excl .claude/) is now empty; no
committed file references .claude/CLAUDE.md or .claude/plan.md.
The repo moved to src-layout and trimmed README into a 238-line front door
with the feature reference under docs/, but several committed files still
referenced bare soup_cli/ paths or linked the gitignored .claude/CLAUDE.md
(which 404s for anyone cloning the public repo).
- docs/: `soup_cli/{plugins,templates,ui/plugins}/...` path refs -> `src/soup_cli/...`
(import statements `from soup_cli...` left unchanged — package name is still soup_cli)
- AGENTS.md: point external agents at public docs/, CONTRIBUTING.md, and the
config schema; note CLAUDE.md is a maintainer-local (gitignored) file
- CONTRIBUTING.md + .github/pull_request_template.md: PR checklist now says
"README.md and the matching page under docs/" (kept in sync); Questions
section links docs/ instead of the gitignored CLAUDE.md
- examples/README.md: fix two broken ../CLAUDE.md links -> config schema source
+ docs/ feature reference
- .gitignore: add root-anchored /_*.py temp-script guard + trailing newline
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.
- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
CODEOWNERS patterns, and the PR-template checklist all repointed to
src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.
Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
Post-release audit caught two pre-existing doc drifts that v0.70.0 should
have closed in the initial commit:
1. README.md `## All Commands` table was missing the 6 new v0.70.0 surfaces
(`soup iterative-dpo`, `soup train --reward-hack-detector`,
`--uld-strategy`, `--minillm-enabled`, `--rl-checkpoint-save-every-steps`,
`--echo-trap-enabled`). Added 6 lines mirroring the v0.69.0 entry format.
2. CONTRIBUTING.md `utils/` directory tree had stale drift from v0.66.0 →
v0.70.0 (24 modules missing across 5 releases). Brought in sync:
v0.66 (sae_diff, sleeper_probe, interference, probe_pack), v0.67
(cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock,
adapter_bisect), v0.68 (prompt_compile, prompt_distill, compile_tools,
apple_adapter, local_rl), v0.69 (build_dag, expectations, magpie,
persona_hub, brain_rot), v0.70 (reward_hacking, uld, minillm,
rl_checkpoint, iterative_dpo, echo_trap).
No code changes — docs-only hotfix per the checklist
"CI-only / docs-only hotfixes" rule. No version bump, no tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 commands that hedge Soup against paradigm shifts. If 1M-context kills FT,
`soup compile` (DSPy + GEPA + TextGrad prompt-program compilation) takes its
place. If teams hit prompt-cost walls, `soup distill-prompt` bridges to small
FT. If only Apple Foundation Models win on-device, `soup apple-adapter` ships
the converter+signing surface. If personal-LLM flywheels become the shape,
`soup local-rl` captures thumbs into SQLite and emits DPO pairs.
- Part A: `soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|...]`
- Part B: `soup distill-prompt --traces <jsonl> --teacher --student --strategy`
- Part C: `soup compile-tools <spec.json|yaml> --eval <jsonl>`
- Part D: `soup apple-adapter <source-dir> --direction hf-to-mlx|... --output`
- Part E: `soup local-rl init/status/record/harvest/train` (LIVE except train)
Schema + path containment + symlink rejection + atomic-write surface ship now;
live runners for Parts A/B/C/D + Part E nightly scheduler deferred to v0.68.1
(stub-then-live, mirrors v0.50.0 / v0.61.0 / v0.62.0 / v0.67.0 cadence).
Test count: 11021 -> 11225 (+204). Review-fix: 0 CRIT + 4 HIGH + 10 MED + 4 LOW.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- SECURITY.md: add v0.67.0 to supported-versions list
- CONTRIBUTING.md: bump test count 244 files / 10836 tests -> 251 / 11021
Docs-only follow-up to v0.67.0 release; no code change, no version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six new top-level commands close axis 1 + 11 of the roadmap: pick the
right base, lock the env, refuse OOMs before launch, and clear
license-clean deploys.
- soup tunability: probe-train 8 candidate bases (Qwen3-0.6/1.7B,
Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) ->
Pareto frontier over (delta x cost x license). Live LoRA probe -> v0.64.1.
- soup plan / soup apply: Terraform-shape lock-and-execute. `apply`
refuses on drift between soup.yaml and soup.tfstate (exit 3).
- soup env lock / status / check: hermetic env lockfile via
importlib.metadata across 15 ABI-sensitive packages + Python + CUDA.
`env check` exits 3 on drift.
- Hardware-fit calculator: static analytical 5-bucket VRAM predictor
with 10% safety margin + actionable hint on OOM.
- soup completions bash|zsh|fish: sourceable shell completion scripts;
recipe names auto-complete from the 115-recipe catalogue.
- soup license-advisor: per-deploy-target license matrix
(b2c/defense/embedded) + Llama community + 700M MAU gate (exit 3).
Composes with v0.60 license-conflict matrix.
Tests: 10035 -> 10306 (+271 net in 7 new files).
Review-fix coverage: 0 CRITICAL + 6 HIGH + 8 MEDIUM + 4 LOW across
consolidated code+security+TDD review wave. Every HIGH lands a regression
test in tests/test_v0640_followups.py (POSIX-skipped symlink rejection,
containment-before-existence ordering, drift-refusal exit-3 end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 new top-level commands close axis 7 of the roadmap. Every Part LIVE on
day one (no deferred stubs):
- soup ingest: universal trace importer (Langfuse / LangSmith / Helicone /
OpenPipe / OTel / OpenAI Stored Completions). 6 adapters
+ frozen TraceRecord with MappingProxyType-wrapped metadata.
Zero credential-handling threat surface — Soup parses the
JSONL export, never makes the SaaS network call.
- soup prune-prompt: detect + strip a shared system-prompt prefix so the
FT model internalises it (OpenPipe's signature trick,
OSS). Binary-search over up to 32 templates finds the
longest threshold-meeting prefix.
- soup data active-sample: surface top-uncertainty prod traces for human
review. Max-entropy on single rm_score or
pairwise disagreement on dual rm_scores.
- soup ab: Wald sequential SPRT for the point alternative. LLR is a
martingale under H0 so Type-I error is controlled at every
stopping time per the optional stopping theorem.
- soup drift-alarm: rolling KL on whitespace-tokenised output distribution
+ SSRF-hardened Slack/Discord webhook (full parity with
v0.51.0 validate_hub_endpoint). Exit 3 on drift for
cron-friendly automation.
Test count: 9816 -> 10035 (+219 net across 6 new test files).
Review-fix coverage (code-reviewer + tdd-guide returned actionable;
python-reviewer + security-reviewer agents context-thrashed on the large
CLAUDE.md release-notes history — matches the v0.58.0 / v0.59.0 / v0.60.0
/ v0.61.0 / v0.62.0 idiom; verified manually):
- 1 CRITICAL: mSPRT log-likelihood-ratio sign error drove Type-I error
to 1.0 as n grew. Replaced with Wald's classic point-
alternative SPRT (martingale under H0).
- 2 HIGH: detect_common_prefix early-exit on 100% match returned the
shortest qualifying prefix instead of the longest;
_MAX_SCAN_ROWS DoS cap used 'pass' instead of 'break'.
- 3 MEDIUM: TraceRecord.metadata now MappingProxyType-wrapped post-init
(frozen-dataclass mutation hazard); _AUTH_ENV table
deduplicated; drift_alarm precedence parens on SSRF gate.
- 2 LOW: pooled_se dead-branch refactor; mean_uncertainty NaN guard.
- 8 follow-up tests: msprt zero-variance, partial-majority binary-search
activation, score_uncertainty exact boundaries, rolling_kl identical
+ disjoint, validate_budget + validate_threshold exact endpoints,
_signal_from_thumbs boundaries, no-heavy-top-level-imports source-grep
guard across all 5 new util modules.
Step 6 smoke verified for all 5 commands + 6 failure-mode rejection
paths.
CRLF gotcha note for future maintainers: PowerShell wrote the smoke
fixtures with a UTF-8 BOM on Windows during Step 6 — switched to
inline Python for the fixture write. Production CLI input handling is
already BOM-tolerant (utf-8-sig in JSONL loaders via v0.40.1 Part E).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpamys
Salil M