v0.72.4 capped `trl<0.25` from a staged-removal table that scored a MODULE
RELOCATION as a field removal. At 0.25 `BCOConfig` moved into
`trl/experimental/bco/`, and at 0.26 kto/orpo/cpo followed — but every one of
them stayed publicly re-exported from `trl` with `max_prompt_length` intact.
Seeing a config file vanish from `trl/trainer/` was read as the field going
away.
Re-derived by parsing each *Config class's own annotated fields across every
wheel 0.24.0 -> 0.29.1 (all five inherit TrainingArguments, so there is no
inherited-field escape hatch):
version dpo kto orpo cpo bco
0.24.0 - 0.26.2 yes yes yes yes yes
0.27.0 - 0.27.2 yes NO yes yes yes
0.28.0 yes NO NO NO NO
0.29.0 - 0.29.1 NO NO NO NO NO
Then settled the way reading source cannot — by CONSTRUCTING all six configs
with the exact kwargs the wrappers pass:
trl 0.26.2 OK dpo · OK ipo · OK kto · OK orpo · OK simpo · OK bco
trl 0.27.0 FAIL kto: KTOConfig.__init__() got an unexpected keyword
argument 'max_prompt_length'
OK dpo · OK orpo <- control: the boundary is KTO's alone
So the cap is <0.27, and 0.25.0/0.25.1/0.26.0/0.26.1/0.26.2 were excluded for
no reason. That is the third wrong answer this bound has had (after "trl 1.x
removed them" and "the break is 0.29.0"), and the first two were also produced
by reading rather than running. The rule this earns: a version bound derived
from source is a hypothesis; the experiment that settles it is constructing the
object.
Two more corrections from the same pass:
- The floor >=0.7.0 was impossible, and nobody checked it while carefully
fixing the ceiling. `setup()` imports GRPOTrainer unconditionally and trl
first exports it at 0.14.0 (OnlineDPO / KTO / BCO / BasePairwiseJudge at
0.11.0; 0.7.0 has none of them). Resolvers pick the newest allowed version,
so it only bit under a constraints file or anyone reading the metadata as a
support claim.
- ORPOConfig/CPOConfig are not "deleted" at 0.29 — the modules survive under
trl/experimental/. They, BCOConfig and their trainers are dropped from the
public `trl` namespace, which is what Soup imports, so the 0.29 break is an
ImportError rather than a rejected kwarg. Harder than described, not milder,
and it means going past 0.29 needs an import strategy (#326).
Closes the blind spot that let this ship, for all six trainers
(tests/test_trl_preference_config_contract.py)
The bug was invisible because the trl imports and the config construction live
inside `setup()`, and constructing a wrapper touches trl not at all. v0.72.4's
streaming suite closed that for four of the six; tests/test_bco.py:266 and
tests/test_ipo.py:191 still MOCK `.setup` out, so bco and ipo had no test that
executed it. The new file drives the real `setup()` for all six on the ordinary
non-streaming path, asserts the BUILT config carries max_prompt_length/
max_length (so a trl release that accepts the kwarg and stops storing it fails
too), and derives the covered set from the trainer sources so a seventh trainer
adopting the argument joins automatically.
Revert the KTO skip: it blamed the device, and the variable is the torch version
`skipif(not cuda)` on test_kto_streams_at_batch_two made the test dead in CI —
there are no GPU runners (ubuntu/windows/macos) — leaving it alive only on this
box, under the one torch where it passes, while torch carries no upper bound.
Three things say the rationale ("a streamed model on CPU is a test convenience")
does not hold:
- the same CI run had test_v07200.py::test_one_training_step_actually_runs, the
identical streamed train() for SFT, pass on that CPU runner;
- running this body here with CUDA masked passes (torch 2.5.1);
- the error comes from check_same_device, i.e. an op received a meta
placeholder next to a real tensor — a streaming property. Newer torch
decomposes more ops, which is why only the newer stack surfaces it.
Now it runs everywhere and xfails on exactly that signature on CPU; anything
else, and the same signature on CUDA, is a hard failure. Real defect filed as
#328. With CUDA hidden: 21 passed, 2 skipped (both genuinely CUDA-only).
Also fixed
- pyproject.toml: 14 double-encoded em-dashes, one of them the `unit` marker
description that `pytest --markers` prints to users. The existing mojibake
guard covered only the package docstring; widened to the file, which is not
importable Python and so was invisible to every source-level check.
- docs/commands.md called itself "the full soup command list" while missing
eight, three of them glued onto the end of a previous line by absent newlines
and therefore invisible on the rendered page. Now 77/77, asserted against the
live Typer app so the claim stays true by construction.
Verified: ruff clean; full suite 16935 passed / 129 skipped / 0 failed (13:15);
wheel builds with Requires-Dist: trl<0.27,>=0.14.0; `pytest --markers` renders
the em-dash. No version bump and no tag — the packaging metadata change wants
CI on the resolver stack before it rides a release.
Third and final correction. The previous cap fixed dpo but left kto/orpo/simpo
failing, because trl did not remove `max_prompt_length` in one release. Read per
config off the published wheels rather than spot-checking dpo_config.py again:
version dpo kto orpo cpo bco
0.24.0 yes yes yes yes yes <- last release all six work on
0.25.1 yes yes yes yes NO
0.26.0 yes NO NO NO NO
0.29.0 NO NO gone gone gone
`soup train --task bco` breaks at 0.25, `--task kto|orpo|simpo` at 0.26, and
`--task dpo|ipo` at 0.29. Capped <0.25, the last release on which all six of
Soup's preference trainers can build their config.
The published benchmarks record carries the same table, including both wrong
diagnoses, since it is cited evidence and a corrected claim there matters more
than a tidy one.
My previous commit capped trl<1 on the assumption, taken from an existing note
in this repo, that CI runs trl 1.9.2 and that 1.x removed the APIs. Both were
wrong, and CI stayed red because <1 excluded nothing: the install log shows CI
resolving trl 0.29.1.
Established by reading the published wheels rather than guessing again:
version ORPOConfig CPOConfig max_prompt_length
0.20.0 yes yes yes
0.28.0 yes yes yes
0.29.0 no no no
0.29.1 no no no
So the break is at a MINOR. Capped trl<0.29 and corrected the "trl 1.x" claim
in the changelog, the published benchmarks record and the internal notes that
propagated it.
trl 1.x removed ORPOConfig and CPOConfig outright and dropped
max_prompt_length from the remaining preference configs. bco, dpo, ipo, kto,
orpo and simpo all pass max_prompt_length, and orpo/simpo additionally import
configs that no longer exist, so on trl 1.x `soup train --task orpo` fails at
import — for anyone who pip-installed with an unbounded `trl>=0.7.0`.
This was latent, not introduced here. No test had ever called setup() on those
wrappers (they were only instantiated, and the trl imports live inside setup),
so CI stayed green against trl 1.9.2 while the code only worked on 0.x.
v0.72.4's end-to-end preference tests are the first to reach that code path,
which is how it surfaced — all 34 CI failures were in tests/test_v07204.py and
none anywhere else.
Same shape as v0.72.3's mcp<2 cap (#322) and the existing transformers<5.0.0:
declare the dependency the code actually works with, and treat the API
migration as its own piece of work rather than shipping against a major nobody
has validated.
DPO / ORPO / SimPO / KTO join task=sft on the layer-streaming engine.
The reference model is the SAME streamed base with adapters disabled — one set
of weights, one stream. Measured: streamed DPO peaks at 0.914x streamed SFT with
a byte-identical store and pool, where forcing a real second instance costs
+730.44 MB against 730.44 MB of weights. All four are bit-exact (0.0) against a
resident run of the same loss.
KTO is NOT reference-free: kto_trainer.py:466-476 is byte-for-byte DPO's
three-branch reference selection, so it was gated separately. It also requires
batch_size >= 2, refused at parse time rather than minutes into sharding.
grpo/ppo stay excluded permanently — rollouts re-read every layer per generated
token. The refusal deliberately names no release.
The ~390-line streaming setup moved verbatim into trainer/stream_setup.py so
five wrappers cannot drift. _STREAM_ROWS_PER_EXAMPLE is 2 for the concatenating
losses and 1 for KTO, measured not assumed: the VRAM pre-flight would otherwise
under-predict by half, and on Windows that is a silent spill, not an error.
Honest cost: the reference is free in memory, not in time — DPO reads the layer
stack 1.52x as often per step as SFT.
Also closes five holes in the release checklist itself:
- benchmarks/ was never in it, so gate records (which live under a gitignored
.claude/) were never published. The public record behind the preprint DOI was
about to fall a release behind; benchmarks/gate-v0.72.4-preference-losses.md
and its index row are here, and the checklist now names the step.
- tests/test_version_sync.py asserts pyproject.toml == __init__.py. Every other
version test in the suite is a >= floor check, so bumping one and forgetting
the other kept CI green. Verified red-green.
- .claude/paper/ (the DOI preprint) had no "did this release change what it
claims?" step. For v0.72.4 the answer is no: no measured number moves and its
task: sft configs stay valid.
- The Docs section header said steps 7-12 while containing 7-13, so plan.md sat
outside its own section.
- The README size anchor said ~238 lines against a real 426.
Notes for whoever hits these next:
- `pre-commit run --all-files` rewrites ~740 files here (ruff-format on
pre-existing code). No pre-commit git hook is installed and CI runs only
`ruff check`, so run it with --files on your own paths or the diff explodes.
- Measuring streamed peak VRAM across setup() charges the pre-flight's own GEMM
probe (three 4096^3 matrices, ~100 MB) to the step. Reset the peak counter
after setup.
- The buffer pool is freed by cycle collection, not by close(): back-to-back
streamed runs in one process retain the previous pool (+47.65 MB measured)
until a gc pass. Call gc.collect() between arms when measuring.
- A resident model built from a float32 fixture vs a bf16 streamed one measures
the dtype gap, not streaming — that cost an hour chasing a 9.96e-04 "failure".
- Two concurrent pytest runs on a 4 GB card produce false CUDA failures; run the
suite alone.
Tests: 16977 -> 17051.
The invite added in the previous commit was a default one, which Discord
expires after seven days. It is now in the README, CONTRIBUTING, the Code of
Conduct and `[project.urls]`, and the last of those ships in release metadata
where a dead link cannot be corrected after publish - so it has to be a
permanent invite, not a convenient one.
All six occurrences move together; a half-updated set is worse than the old
link, since the stale copies would be the ones a reader hits first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discord is added as a fourth community channel, with the boundary stated
rather than left to guesswork: it is for live chat and setup help, while
anything that should still be findable in six months belongs in Issues or
Discussions. A Discord answer helps one person; an issue helps everyone who
hits the same thing, and the repository already routes public traffic that
way.
It lands in the four places a reader actually looks - the README header, the
badge row, the Contact section, and CONTRIBUTING's Community list - plus
`[project.urls]`, which is the one that matters most in practice: most users
arrive from PyPI, whose sidebar previously showed only Homepage, Repository
and Issues. That entry is metadata and takes effect on the next publish.
Both a Code of Conduct that does not name the server and a security policy
that does not exclude it are gaps a public chat channel creates, so the Code
of Conduct now states it applies there, and SECURITY.md says explicitly not to
report vulnerabilities in a public channel.
The single maintainer address becomes two with distinct roles, because one
address doing both jobs cannot be handed over: team@trysoup.dev is the project
address and survives a change of maintainer, while makazanalpamys@gmail.com
stays as the personal fallback. Both are listed everywhere a contact appears -
README, SECURITY.md, CODE_OF_CONDUCT.md - and pyproject's author email, which
PyPI renders as the package contact, moves to the project address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Code of Conduct pointed at an address that is no longer read, and it was
the only contact route in the repository, so a report sent there went nowhere.
SECURITY.md gains it as a fallback rather than a replacement: private GitHub
Security Advisories stay the preferred channel, but the list had a single
entry and no path for a reporter without a GitHub account.
pyproject gains an `email` on the author entry, which is what PyPI renders as
the package contact - it had a name and no way to reach anyone. Metadata only;
it takes effect on the next publish and changes nothing at runtime.
mcp 2.0.0 removed `mcp.shared.memory.create_connected_server_and_client_session`
and dropped `Server.list_tools`. Our constraint was an unbounded `>=1.2.0`, so
CI resolved to 2.0.0 the day it was published and every `soup mcp serve`
round-trip test failed on all nine jobs — on every OS and Python version, which
is what distinguished an upstream break from a regression in the commit under
test.
Verified 1.29.0 (the newest 1.x) still exposes both APIs before capping, rather
than assuming the boundary. Migrating to the 2.x API is separate work; shipping
against an unvalidated major is not a substitute for it.
Lifts the v0.72.0-.2 scope freeze. Every capability was gated against a
streamed-vs-resident bit-exactness reference before it was written.
- Six more families (mistral/gemma/gemma2/gemma3_text/phi/phi3), each
bit-exact vs the same checkpoint loaded resident, under bf16 AND NF4.
Multimodal gemma3 stays refused — only gemma3_text.
- batch_size > 1 and gradient_accumulation_steps > 1 now work.
- A batch- and vocab-aware VRAM pre-flight that refuses a run predicted
not to fit. Fitted to 10 real runs: worst error 0.85%, never
under-predicts. On Windows an over-budget step does not OOM; WDDM
spills silently, so the estimator is the only guard.
- A throughput bracket from a GEMM ceiling measured on the user's own
card in the same session, printed with the SM clock.
- --resume / --hf-resume: load_state_dict narrows keys by child name, so
a canonical checkpoint matched 0 of N tensors and PEFT warned only.
Keys are now redirected at load time, mirroring the v0.72.1 save fix.
- An NVMe disk overflow tier (stream_source: auto|ram|disk), bit-exact
against the RAM tier. Its speed relative to RAM is UNMEASURED here and
no figure is claimed.
- soup doctor --disk reports the detected media type.
Fixes: estimate_logits_bytes charged 6 bytes/element where the measured
peak is 14; the NVMe tier guard was wired to a hardcoded constant;
streaming sources leaked handles when training raised; subprocess
helpers resolved tools by bare name (CWE-427 on Windows).
112 tests in tests/test_v07203.py; 16867 -> 16977.
Layer streaming (v0.72.0) was bf16-only, capping it near 3B on a small card.
Quantising the streamed base to NF4 makes the RAM store ~4x smaller, which is
what brings 8B within reach.
Measured on a 4 GB RTX 3050 Laptop through the shipped code (50 steps after 10
warm-up, batch 1, S=512, PagedAdamW8bit, GEMM ceiling taken in the same session):
Llama-3.1-8B-Instruct 119.6 tok/s 3.32 GB peak 3.60 GB pinned 100% 952 MHz
Qwen2.5-3B 264.2 tok/s 1.76 GB peak 1.43 GB pinned 100% 960 MHz
3B is 1.85x the bf16 path, but that is PINNING, not arithmetic: 1.43 GB
page-locks where 5.55 GB did not, restoring async copy_ (util 79.3% -> 100%).
14B was not run — its store exceeds this box's measured 7.12 GB pinned ceiling.
A streamed NF4 run is bit-exact against a RESIDENT NF4 run, now as CPU-runnable
CI tests rather than only a gate result.
Notable, because each fails silently:
- PEFT dispatches lora.bnb.Linear4bit only when is_loaded_in_4bit is stamped;
without it the generic lora.layer.Linear runs against a Linear4bit base and
casts differently (9.375e-01 logit divergence, no warning). Pinned by a test
with a control that deletes the marker.
- hf_quantizer must be stamped too, or Trainer.__init__ dies formatting its own
"cannot fine-tune" error. Found by the end-to-end test.
- The shard cache is keyed on quant/double_quant/quant_device as well as dtype
and source fingerprint; a bf16 cache reused for an NF4 request would feed
full-precision bytes to matmul_4bit.
- index.json is a trust boundary: its shape/blocksize reach bnb kernels that do
not bounds-check, so from_json validates and the runtime cross-checks the
claim against the bytes on disk.
- A streamed NF4 model over-reported parameters ~6.5x (878,154,048 vs
134,515,008 for SmolLM2-135M). Display-only; ~52 B at 8B.
Scope unchanged and still BETA: RAM tier, sft, Llama/Qwen, batch 1, no
accumulation, no resume. quantization values other than none/4bit are refused.
Tests: 16752 -> 16840 (+88 in tests/test_v07202.py).
Full suite: 16734 passed, 129 skipped.
v0.72.0's layer-streaming wrapper holds the real decoder layer as a child
named `inner`, so every saved LoRA adapter key carried an `.inner.` segment.
Such a file reloads as ZERO tensors into any normal model: soup merge,
soup serve, soup chat and PeftModel.from_pretrained all returned the untuned
base while PEFT emitted only a UserWarning. Training was correct; only the
artifact was inert.
StreamedDecoderLayer.state_dict() now delegates to the wrapped layer at the
wrapper's own prefix, so every artifact path -- the final trainer.save_model(),
each save_steps checkpoint, and therefore soup adapters, the Registry, merge
and serve -- becomes canonical at once. Serialisation-only by design: the
forward path is untouched, so v0.72.0's bit-exactness gates remain valid
without being re-earned.
Also fixes --hf-resume bypassing the streaming resume refusal. The guard
tested only --resume, while --hf-resume reaches resume_from through another
branch. Pre-fix that combination matched keys by accident; post-fix it would
have matched nothing and silently continued training with a freshly
initialised adapter -- i.e. the adapter-key fix alone would have made that
one path worse.
Roadmap renumbered (this release was inserted ahead of NF4): every
"lands in vX.Y.Z" refusal corrected -- NF4 v0.72.2, disk tier / more
architectures / larger batches / gradient accumulation / checkpoint-resume
v0.72.3, preference losses v0.72.4.
Found by the v0.72.2 NF4 gate, not by the 159 v0.72.0 tests -- none of them
saved an adapter and loaded it back. The new regression test does exactly
that, by count, by name and by value, with a negative control that re-mangles
the keys and asserts the reload yields zeros (0-of-N loading raises nothing,
so a green round-trip without the control proves nothing).
Tests: +17 in tests/test_v07201.py (16735 -> 16752).
Full suite: 16623 passed, 129 skipped, 4 deselected.
Note for maintainers: do not edit soup_cli/__init__.py while a suite is in
flight -- a mid-run version bump made test_cli_subprocess::test_version fail
spuriously (the subprocess and the imported constant disagreed).
The frozen base lives in CPU RAM and is streamed into a small pool of
pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is
bounded by ONE layer instead of the whole model. Only the LoRA adapters,
their gradients and optimizer state stay resident.
Measured on an RTX 3050 Laptop 4 GB (Windows, 16.9 GB RAM), batch 1,
gradient checkpointing on, 50 steps after 10 warm-up:
Qwen2.5-0.5B S=512 978.6 tok/s 91.4% util 1.47 GB peak
Qwen2.5-1.5B S=512 525.0 tok/s 96.8% util 1.82 GB peak
Qwen2.5-1.5B S=1024 487.6 tok/s 96.7% util 2.96 GB peak
Qwen2.5-3B S=512 143.1 tok/s 79.3% util 2.15 GB peak
Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs.
Honest cost: 1.43x slower than resident, measured at 0.5B — the only
apples-to-apples comparison available on this box, because 1.5B and above
cannot run resident here at all.
Correctness was gated before any src/ code was written: streamed vs
resident logits are bit-exact (max abs diff 0.0), the layer-0 LoRA
gradient is non-zero on all layers, a 100-step loss curve matches
resident exactly, and same-seed runs are identical.
New:
- utils/layer_stream.py pure planner (no top-level torch)
- utils/layer_shard.py per-layer safetensors sharder
- utils/layer_stream_runtime.py buffer pool, RAM source, prefetch, wrapper
- training.stream_layers / stream_source / stream_buffers
Notes for future maintainers:
- transformers' Trainer.__init__ and accelerate's prepare_model BOTH call
model.to(), which raises NotImplementedError on meta parameters. The
streamed layer overrides _apply to pass meta tensors through, and the
model declares hf_device_map. Without either, every run dies at trainer
construction — no unit test that stops at model(input_ids=...) sees it.
- The shard cache is keyed to a fingerprint of the source checkpoint, not
just the model slug: a base retrained in place must re-shard rather than
silently stream stale weights.
- The pre-flight hardware-fit gate models a RESIDENT run, so it is skipped
for streaming — otherwise it refuses exactly the runs this enables.
- expandable_segments:True is silently ignored on Windows; probed, not
claimed.
Scope (every refusal names the release that lifts it): RAM tier, bf16,
task=sft, Llama/Qwen, batch 1, no gradient accumulation, no --resume.
NF4 is v0.72.1; disk tier / bigger batches / accumulation / resume are
v0.72.2. Proof-of-mechanism at 3B — nothing above 3B was measured.
Tests: 16576 -> 16735 (+159 in tests/test_v07200.py)
Turn the reward-hacking detector on the verifier itself: feed empty /
length-padded / repetition / sentinel-spam completions and flag any the
verifier accepts. Loads via the existing load_reward_fn (probes a synth .py
or a builtin); a gold-requiring target with no --references is a hard error,
never a false "robust". Exit 0=robust / 2=gameable / 1=error. Pure, offline,
no schema change, no new deps.
Also corrects the ops-docs Telemetry section (the sender exists but is wired
to nothing — no data is sent). Telemetry flywheel deferred pending a public
privacy policy.
Tests: 16490 -> 16529 (+39). 5 sequential ECC reviews, every finding fixed.
Point `soup reward synth <refs.jsonl> -o reward.py` at reference (gold) outputs and it
infers a deterministic verifier (numeric / json_schema / regex / tool_call), emits a
readable, committable .py reward_fn that rides load_reward_fn's existing .py path (no new
exec surface), and — the moat — REFUSES to emit one that can't discriminate its references
from auto-perturbed negatives via a mandatory calibration report (accept refs >=90% AND
reject negatives; hard floor at discrimination<=0). Nothing in TRL/Unsloth/Axolotl
synthesizes a reward.
Fixes#311: a comma-separated reward_fn ("accuracy,format") now loads as a reward ensemble
(GRPOTrainer reward_funcs=[...], unlocks the rm_ensemble detector), GRPO-only and validated
at config-parse; the deepseek-v3-reasoning recipe that shipped this previously crashed with
"Unknown reward function".
Riders: reward_fn field-validator (null-byte/blank/oversize/empty-comma-segment); comma-aware
verifiable-domain check; envs/calculator + guess_number docstrings corrected.
5 sequential ECC reviews, every finding fixed (python HIGH PPO gate; code 2xHIGH per-tool
arg binding + json_schema mixed-shape refuse; security HIGH rel_hint codegen injection; tdd
8xHIGH). Live smoke on RTX 3050: synth from envs/calculator agrees with math_verify;
degenerate refused (exit 2); real GRPO on SmolLM2-135M with reward_fn=accuracy,format
completed optimizer steps. +103 tests (tests/test_v07140.py); 16387 -> 16490.
soup ship's leg 2 — the catastrophic-forgetting / regression gate that carries
the whole SHIP / DON'T-SHIP claim — was 15 trivia prompts scored by raw
substring containment (it credited "B" for "Berlin", "3" for "13") with zero
coverage for tool-calling, safety, or JSON. This makes the gate real.
- forgetting.py: score_answer/extract_mcq_letter replace the substring scorer
with answer-extraction (cue -> paren -> clause-terminating bare letter) +
boundary-aware token match. MINI_BENCHMARKS expanded (mmlu 26 / common_sense
24 / instruction 24) + new mini_arithmetic (36) so a 1-item flip trips 0.05.
BREAKING: an existing run's verdict can change (the old gate under-reported).
- eval/gate_suites.py (new): bundled offline general-suite registry, no torch.
DEFAULT_GENERAL_SUITE = the 4 MCQ suites + 3 behavioural JSONL suites
(mini_tool_call / mini_format_json / mini_safety) scored per-model-absolute
by the pure custom/diagnose scorers. _fraction_passing isolates a per-item
scorer exception (deep-JSON RecursionError scores as a failed item).
- ship.py: leg-2 scores bundled suites offline (base+tuned) before routing any
non-bundled name to lm-eval; default general suite = the full bundled set.
Exit-code taxonomy: usage errors move 2 -> 3 so exit 2 means only DON'T-SHIP
(a typo'd flag was previously indistinguishable from a caught regression).
- diagnose/__init__: "Six" -> "Seven" probes + re-export all 7 score_* fns;
removed the dead SUPPORTED_TASK_MODES "pairwise reserved" gate.
- Bundled gate fixtures ship in the wheel via the pyproject artifacts glob.
Every bundled item is original, hand-authored (no MMLU/GSM8K rows copied).
Test count 16288 -> 16330 (+42 in tests/test_v07138.py).
Every printed and documented `pip install 'soup-cli[extra]'` was bash / zsh /
PowerShell syntax and failed on Windows cmd.exe:
ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at
the start of dependency specifier
cmd.exe has no single-quote quoting, so it passes the quotes to pip verbatim
and pip rejects the requirement. Nothing in Soup can fix that once the command
is typed -- pip and the shell own it, and Soup is not installed yet when the
README line runs -- so the fix is the spelling we print.
Migrated 147 sites across 67 files to `pip install "soup-cli[extra]"`:
- 64 in src/ (Rich console hints + plain ImportError text)
- 57 in README.md + docs/
- 22 in src/soup_cli/templates/*.yaml + examples/configs/*.yaml
- 3 in examples/README.md
Double quotes are the only spelling valid in every shell (cmd, PowerShell,
bash, zsh), which is why the repo already used `pip install -e ".[dev]"`.
Measured on Windows: single quotes fail ONLY on cmd; double quotes pass
everywhere; bare passes on Windows but zsh globs `[extra]` and fails.
Method note (the PR #247 class): the hints sit INSIDE double-quoted Python
string literals, so a blind ' -> " sed produces SyntaxError. A tokenize-based
rewriter escaped `\"` in DQUOTE tokens and left bare `"` in TRIPLE / COMMENT
tokens; every touched .py was compile-checked. The full suite (not ruff, not
compile-check) caught two rewriter blind spots: the real YAML templates under
src/soup_cli/templates/ (byte-identical drift test) and examples/README.md.
A regression test (tests/test_v07137.py) scans the package and every docs code
block for the single-quoted form; prose may still name it so a reader from an
older tutorial recognises the error.
Also bundles #315 (@Sanjays2402): eval-gate benchmark tasks now run via
ForgettingDetector instead of a helper that never existed. Closes#310.
Test count: 16283 -> 16288 (+4 in tests/test_v07137.py).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Version 0.71.35 -> 0.71.36 in pyproject.toml + __init__.py.
Docs are written around what the live smoke MEASURED, not what the plan
assumed:
- The semantic-dedup headline is WITHDRAWN. docs/data.md leads with the
measured overlap: paraphrase cosines (0.49-0.76) overlap genuinely-distinct
rows (0.54-0.76), and "Add two numbers"/"Multiply two numbers" (0.759)
scores HIGHER than the true paraphrase "reverse a string"/"invert the order
of characters" (0.491). No threshold separates them, so the page says
plainly that lowering --threshold trades duplicates for silent data loss,
and explains why the 0.8 default is deliberately conservative. The claim it
does make -- catches REWORDINGS MinHash's shingling misses (0.88-0.91) --
is the one the numbers support.
- docs/data.md canary section states the real verdict rule (binomial tail
over the count, not any-single-canary) and why: at K=16 an any-canary rule
fires on a CLEAN model ~15% of the time.
- docs/training.md --replay reports the honest result: 7% better retention
than control, but forgetting without it was only +4% (mild), so it is
proof-of-mechanism at 135M + LoRA, not a production claim.
- CHANGELOG known-limitations carry the same numbers rather than hedging.
- README What's New leads with the two blocking bugs the smoke found (the
hardware-fit gate refusing locally-merged models; the [extra] hints
printing without the extra).
Counts 16001 -> 16254 tests, 313 -> 314 files (CONTRIBUTING).
Ship a regulated fine-tune with the paperwork it needs, plus GGUF export that
actually works on Windows.
Added:
* `soup init --template hipaa|soc2|eu-ai-act|sr-11-7` (templates 17 -> 21).
Design constraint verified in-repo, not assumed: Soup's compliance controls
are CLI flags/commands, NOT schema keys (audit_log/bom/attest/repro_receipt/
annex_xi/track_energy/pii/decontaminate have zero matches in config/schema.py),
so a template cannot "pre-wire audit-log on" as YAML. Each is a valid
SoupConfig + header comments naming that regime's exact commands.
* `soup card <registry-id> -o MODELCARD.md` — registry entry -> publishable HF
card (training config, eval scorecard, config/data hashes, lineage, artifacts).
Pure build_model_card() over dicts, reused by `soup push --card`.
* `soup ci init` — writes .github/workflows/soup-gate.yml chaining
data validate -> expect -> ship --evidence (exit 2 blocks the merge).
* docs/compliance.md quickstart.
Fixed (GGUF-on-Windows, validated end-to-end for the first time; closes the
CPU-validatable half of #70/#144). Four independently-fatal bugs:
* export cloned llama.cpp into the CURRENT directory: SOUP_DIR is the bare name
".soup" but was used relatively instead of anchored to home like tracker.py /
registry/store.py, so ~/.soup/llama.cpp was never found.
* the first GGUF export DOWNGRADED the user's torch and broke CUDA: the
auto-clone pip-installed llama.cpp's requirements.txt (pins torch~=2.2.1 from
the CPU index) into the user's interpreter. Observed live: torch 2.5.1+cu ->
2.2.2+cpu, transformers 4.57 -> 4.46. Now installs only gguf/sentencepiece/
protobuf, unpinned, non-fatally.
* a correctly-built llama.cpp was not found on Windows: MSVC (like Xcode) is a
multi-config generator emitting build/bin/Release/llama-quantize.exe.
* `soup deploy ollama` failed on a relative GGUF path ("pull model manifest:
file does not exist") — ollama resolves FROM against the Modelfile's dir and
Soup writes it to a temp dir. Modelfile now emits an absolute path.
Also fixed a pre-existing model-card injection hole (affects `soup push`'s own
auto-card): _render_training_section interpolated base/task/scheduler/recipe
unescaped, and SoupConfig.base/scheduler have no charset validator, so a
crafted-but-valid config could smuggle raw HTML or a code-span-breaking backtick
into a card published to the Hub.
Step-6 live smoke (real train -> registry push -> card) caught a bug 90 green
tests missed: is_adapter came only from registry artifacts, so a real LoRA run
with no artifacts rendered "Full model" + library_name: transformers — a false
claim in a provenance document.
5 sequential reviews, every finding fixed (code HIGH: the generated workflow ran
`pip install -e ".[dev]"`, which only works in the Soup source tree, breaking the
gate's first step for every downstream user; security HIGH/MEDIUM/3 LOW; tdd HIGH:
push --card was only --help-tested). All 5 new fixes mutation-verified as real
pins. Tests 15906 -> 16001 (+95); full suite 15872 passed / 126 skipped.
Version 0.71.32 -> 0.71.33. CHANGELOG (with a Known-limitations block), README
What's New, docs/commands.md, docs/serving-and-export.md, CLAUDE.md (arch + CLI
+ counts + history roll), CONTRIBUTING counts.
The live smoke changed the pitch. The plan promised '~1.5-2x faster to serve by
distilling its own draft'. Measured on SmolLM2-360M-Instruct <- 135M-Instruct:
the STOCK draft already scored 69.3% acceptance; distilling it gave 69.7% at 2
epochs and 69.3% at 10 -- no gain beyond noise. A small same-family draft is
already at its capacity ceiling vs the target, and logit KD cannot buy capacity
it does not have. Assisted decoding also measured 0.55-0.64x -- a net SLOWDOWN.
So the speedup claim is withdrawn, not shipped. The feature ships as the honest
gate: soup draft measure tells you whether speculative decoding is worth
enabling BEFORE you ship it, and on this pair it correctly says no. That
negative result is stated in the CHANGELOG, the README, the serving docs and
CLAUDE.md, and the pre-existing unverified '2-3x faster' line in the
speculative-decoding docs was tempered to match.
Full suite: 15680 passed / 124 skipped (15806 collected). ruff clean.
Add `soup data doctor` and `soup data lint`, killing the top *silent*
fine-tune failures before a single training step: EOS-missing-from-labels
(the #1 "model never stops generating" bug), BOS duplication, no-system-role
templates, and preference-data length bias (the #1 silent DPO degradation) —
none of which any competitor (Unsloth/Axolotl/LlamaFactory) checks for.
- utils/data_doctor.py: 8-check chat-template compat report over a
tokenizer + sampled rows, OK/MINOR/MAJOR taxonomy mirroring diagnose;
--show-mask N renders per-token trained/masked colouring through the
SAME masking dispatch (_build_row_labels) the report itself uses, so
the two can never disagree about what's actually trained.
- utils/data_lint.py: preference-data linter (dpo/orpo/simpo/ipo/bco/kto)
— length bias (Cohen's d), label imbalance, near-duplicates (MinHash),
identical chosen==rejected pairs, prompt leakage.
- commands/data_doctor.py: Typer layer for both commands; strips C0
control bytes before untrusted dataset content reaches the terminal.
- commands/diagnose.py: hardens the --evidence loader against a TOCTOU
symlink swap (O_NOFOLLOW + fstat-on-open-fd), backporting the pattern
soup ship shipped in v0.71.25 (closes v0.71.25 known-limitation (4)).
Live smoke against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer
(Windows + RTX 3050) found and fixed two genuine bugs beyond the synthetic
fixtures: the EOS check needed to span-search the whole trained region
(not just the last token), and two apply_chat_template call sites needed
a broad except Exception for jinja2.exceptions.TemplateError.
+173 tests (14788 -> 15042). 5 sequential ECC reviews, every finding fixed.
#81 Quant Menu for vision/audio modality
- config/schema: drop the `modality != "text"` rejection in
_validate_quant_menu_supported_tasks (mlx-backend gate retained) so the full
Quant Menu (gptq/awq/hqq:Nbit/aqlm/eetq/mxfp4/fp8) applies to vision+audio.
- trainer/sft: _setup_vision_transformers + _setup_audio_transformers call
build_quantization_config_for_loader (strict superset of the inline 4bit/8bit
BNB blocks they replaced); drop BitsAndBytesConfig import; retain the
prepare_model_for_kbit_training gate on (4bit,8bit,mxfp4).
#80 multipack DataLoader sharding under FSDP/DeepSpeed/DDP
- utils/multipack_trainer: get_train_dataloader routes the multipack DataLoader
through accelerator.prepare when num_processes > 1 so accelerate's
BatchSamplerShard shards whole FFD bins across ranks (even_batches=True avoids
the epoch-boundary collective hang; seed identical across ranks, no `+ rank`).
Single-process path unchanged. Defence-in-depth guard against an unconfigured
MagicMock num_processes.
Tests: +37 in tests/test_v07119.py (13770 -> 13807). ruff clean.
Supersedes the v0.40.5 vision/audio Quant-Menu and v0.40.4 multipack-FSDP
known-limitations. Full multi-GPU validation remains an INFRA-BLOCKED QA item.
#261 iterative_dpo._default_train_fn rendered output as a {dir: ...} mapping
that SoupConfig rejected; render it flat so the spawned soup train succeeds.
#246 CMA-ES merge now loads the base model once and reuses it across the
candidate population (_CachedBaseScorer) instead of reloading per candidate.
#245 soup loop estimate_cost wires run_cost.estimate_run_cost_usd off the last
completed run instead of a 0.0 placeholder; never crashes the daemon.
#244 soup train --track-energy --energy-out persists the measurement JSON so
soup bom emit --energy can attach it to an ML-BOM.
#170 --diagnose-gate is RANK-aware: gate once per cluster (RANK==0), not per node.
Tests: 13476 -> 13511 (+35 in tests/test_v07115.py). Validated end-to-end on
SmolLM2-135M / RTX 3050.
Soup spans the full post-training stack — SFT + preference/RL
(DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO) + distillation + unlearning +
knowledge-edit + steering + RAFT/RA-DIT — not just fine-tuning. "Fine-tune"
alone undersells the RL/alignment surface and the post-training category.
Update the canonical tagline to "Fine-tune and post-train LLMs in one
command. No SSH, no config hell." across the five places it appears:
README hero, PyPI description (pyproject), `soup --help` epilog + callback
docstring (cli.py), package docstring (__init__), and the auto-generated
HF model card (push.py). Searchable "fine-tune" keyword kept as the verb;
"post-train" added as the category claim.
Copy-only; no version bump (description ships with next release).
Lift the v0.68.0 deferred-stub family to live (closes#225, #226, #227, #229):
- #229 local-rl train --once: harvest thumbs -> DPO/KTO/ORPO train via a
soup train subprocess (argv list, no shell); state table tracks last_train_at
(skip-on-no-new-thumbs + skip-on-insufficient-pairs); no --once renders a
systemd/launchd nightly scheduler scaffold. New local_rl_scheduler.py.
- #226 distill-prompt: call the teacher once per trace (Ollama/Anthropic/vLLM)
and write a real dataset (sft/kl -> messages; preference -> chosen/rejected).
- #225 compile / #227 compile-tools: live DSPy/GEPA/TextGrad dispatch behind the
new [compile] extra with a friendly ImportError when absent; injectable seams.
Security: reject \n/\r in the model id + shell-quote ExecStart args (systemd
injection defence). Fix: render train output as a plain string (schema-valid),
with a regression test against SoupConfig.
Tests 13329 -> 13424. Smoked end-to-end: real DPO train on SmolLM2-135M (RTX 3050)
+ real Ollama teacher distillation.
Alpamys
Alpamys