`soup train --help` contains no `--task`; the task is a `soup.yaml` key. Two
places said otherwise, and both read as an instruction rather than as history:
- `docs/commands.md` listed `soup train --task unlearn` in the command
cheat-sheet, where lines are copied straight into a terminal. Rewritten to
`soup train # task: unlearn`, the form the RAFT line directly below it
already uses.
- `docs/training.md` said "`soup train --task unlearn` is live", two lines
under a code block that correctly runs `soup train --config unlearn.yaml`.
Now `task: unlearn`.
Checked every other `--task` in docs/: all 25 remaining are real flags on other
commands (`infer`, `data forge`, `bom emit`, `recipes search`, `eval coverage`,
`ship --task-eval/--task-mode`). Verified against the registered options, not
by reading a diff.
Left alone deliberately: CHANGELOG.md and benchmarks/gate-v0.72.4 also carry
the phrase, but they describe which code path broke rather than telling anyone
what to run, and the gate records are published verbatim on purpose. Four
docstrings in src/ and two in tests/ are narrative, not emitted to users.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of the first `pre-commit run --all-files` after the hooks were finally
activated in d0b7c62: trailing whitespace stripped and end-of-file newlines
normalised across 18 files.
No semantic change — `git diff --ignore-all-space --ignore-blank-lines` over
the whole set is empty. Committed as one batch on purpose: left alone, these
would surface as unrelated noise in the next real diff to each of these files,
since the hooks now fix them on the way past.
Visitors arrive from the Show HN headline and the first screen is all prose.
Adds a 14.5s GIF directly under the 8B-on-4GB claim, above "Why Soup?".
Segment 44.0-60.0s of the demo, chosen off the .vtt cue list and confirmed
frame by frame rather than from the brief's estimate: the scene cut sits
between 47 and 48s, so the clip opens on 3.5s of the static "Layer streaming
BETA" pre-flight panel (3.60 GB store across 32 layers, 2 x 113 MB VRAM
buffers, Training started!) and then runs the measurement card up to its
settled 3.32 GB / 119.6 tok/s. Both halves read without sound or context.
Encoding: two-pass palettegen/paletteuse so terminal colours survive, 960px
wide, 10 fps, dither=none, diff_mode=rectangle -> 4.63 MB. Readability was the
binding constraint, so the budget was met by cutting duration (16s -> 14.5s)
and fps (12 -> 10) rather than width; a bayer-dithered cut of the same clip
came to 5.08 MB and was dropped. Panel lines verified legible by opening the
generated GIF, not assumed from the source resolution.
Caption numbers are from benchmarks/gate-v0.72.2-nf4.md line 314 (the
re-measurement through shipped code), matching the claim line above it.
"What's New" untouched. README 440 -> 445 lines.
v0.72.4 capped `trl<0.25` from a staged-removal table that scored a MODULE
RELOCATION as a field removal. At 0.25 `BCOConfig` moved into
`trl/experimental/bco/`, and at 0.26 kto/orpo/cpo followed — but every one of
them stayed publicly re-exported from `trl` with `max_prompt_length` intact.
Seeing a config file vanish from `trl/trainer/` was read as the field going
away.
Re-derived by parsing each *Config class's own annotated fields across every
wheel 0.24.0 -> 0.29.1 (all five inherit TrainingArguments, so there is no
inherited-field escape hatch):
version dpo kto orpo cpo bco
0.24.0 - 0.26.2 yes yes yes yes yes
0.27.0 - 0.27.2 yes NO yes yes yes
0.28.0 yes NO NO NO NO
0.29.0 - 0.29.1 NO NO NO NO NO
Then settled the way reading source cannot — by CONSTRUCTING all six configs
with the exact kwargs the wrappers pass:
trl 0.26.2 OK dpo · OK ipo · OK kto · OK orpo · OK simpo · OK bco
trl 0.27.0 FAIL kto: KTOConfig.__init__() got an unexpected keyword
argument 'max_prompt_length'
OK dpo · OK orpo <- control: the boundary is KTO's alone
So the cap is <0.27, and 0.25.0/0.25.1/0.26.0/0.26.1/0.26.2 were excluded for
no reason. That is the third wrong answer this bound has had (after "trl 1.x
removed them" and "the break is 0.29.0"), and the first two were also produced
by reading rather than running. The rule this earns: a version bound derived
from source is a hypothesis; the experiment that settles it is constructing the
object.
Two more corrections from the same pass:
- The floor >=0.7.0 was impossible, and nobody checked it while carefully
fixing the ceiling. `setup()` imports GRPOTrainer unconditionally and trl
first exports it at 0.14.0 (OnlineDPO / KTO / BCO / BasePairwiseJudge at
0.11.0; 0.7.0 has none of them). Resolvers pick the newest allowed version,
so it only bit under a constraints file or anyone reading the metadata as a
support claim.
- ORPOConfig/CPOConfig are not "deleted" at 0.29 — the modules survive under
trl/experimental/. They, BCOConfig and their trainers are dropped from the
public `trl` namespace, which is what Soup imports, so the 0.29 break is an
ImportError rather than a rejected kwarg. Harder than described, not milder,
and it means going past 0.29 needs an import strategy (#326).
Closes the blind spot that let this ship, for all six trainers
(tests/test_trl_preference_config_contract.py)
The bug was invisible because the trl imports and the config construction live
inside `setup()`, and constructing a wrapper touches trl not at all. v0.72.4's
streaming suite closed that for four of the six; tests/test_bco.py:266 and
tests/test_ipo.py:191 still MOCK `.setup` out, so bco and ipo had no test that
executed it. The new file drives the real `setup()` for all six on the ordinary
non-streaming path, asserts the BUILT config carries max_prompt_length/
max_length (so a trl release that accepts the kwarg and stops storing it fails
too), and derives the covered set from the trainer sources so a seventh trainer
adopting the argument joins automatically.
Revert the KTO skip: it blamed the device, and the variable is the torch version
`skipif(not cuda)` on test_kto_streams_at_batch_two made the test dead in CI —
there are no GPU runners (ubuntu/windows/macos) — leaving it alive only on this
box, under the one torch where it passes, while torch carries no upper bound.
Three things say the rationale ("a streamed model on CPU is a test convenience")
does not hold:
- the same CI run had test_v07200.py::test_one_training_step_actually_runs, the
identical streamed train() for SFT, pass on that CPU runner;
- running this body here with CUDA masked passes (torch 2.5.1);
- the error comes from check_same_device, i.e. an op received a meta
placeholder next to a real tensor — a streaming property. Newer torch
decomposes more ops, which is why only the newer stack surfaces it.
Now it runs everywhere and xfails on exactly that signature on CPU; anything
else, and the same signature on CUDA, is a hard failure. Real defect filed as
#328. With CUDA hidden: 21 passed, 2 skipped (both genuinely CUDA-only).
Also fixed
- pyproject.toml: 14 double-encoded em-dashes, one of them the `unit` marker
description that `pytest --markers` prints to users. The existing mojibake
guard covered only the package docstring; widened to the file, which is not
importable Python and so was invisible to every source-level check.
- docs/commands.md called itself "the full soup command list" while missing
eight, three of them glued onto the end of a previous line by absent newlines
and therefore invisible on the rendered page. Now 77/77, asserted against the
live Typer app so the claim stays true by construction.
Verified: ruff clean; full suite 16935 passed / 129 skipped / 0 failed (13:15);
wheel builds with Requires-Dist: trl<0.27,>=0.14.0; `pytest --markers` renders
the em-dash. No version bump and no tag — the packaging metadata change wants
CI on the resolver stack before it rides a release.
DPO / ORPO / SimPO / KTO join task=sft on the layer-streaming engine.
The reference model is the SAME streamed base with adapters disabled — one set
of weights, one stream. Measured: streamed DPO peaks at 0.914x streamed SFT with
a byte-identical store and pool, where forcing a real second instance costs
+730.44 MB against 730.44 MB of weights. All four are bit-exact (0.0) against a
resident run of the same loss.
KTO is NOT reference-free: kto_trainer.py:466-476 is byte-for-byte DPO's
three-branch reference selection, so it was gated separately. It also requires
batch_size >= 2, refused at parse time rather than minutes into sharding.
grpo/ppo stay excluded permanently — rollouts re-read every layer per generated
token. The refusal deliberately names no release.
The ~390-line streaming setup moved verbatim into trainer/stream_setup.py so
five wrappers cannot drift. _STREAM_ROWS_PER_EXAMPLE is 2 for the concatenating
losses and 1 for KTO, measured not assumed: the VRAM pre-flight would otherwise
under-predict by half, and on Windows that is a silent spill, not an error.
Honest cost: the reference is free in memory, not in time — DPO reads the layer
stack 1.52x as often per step as SFT.
Also closes five holes in the release checklist itself:
- benchmarks/ was never in it, so gate records (which live under a gitignored
.claude/) were never published. The public record behind the preprint DOI was
about to fall a release behind; benchmarks/gate-v0.72.4-preference-losses.md
and its index row are here, and the checklist now names the step.
- tests/test_version_sync.py asserts pyproject.toml == __init__.py. Every other
version test in the suite is a >= floor check, so bumping one and forgetting
the other kept CI green. Verified red-green.
- .claude/paper/ (the DOI preprint) had no "did this release change what it
claims?" step. For v0.72.4 the answer is no: no measured number moves and its
task: sft configs stay valid.
- The Docs section header said steps 7-12 while containing 7-13, so plan.md sat
outside its own section.
- The README size anchor said ~238 lines against a real 426.
Notes for whoever hits these next:
- `pre-commit run --all-files` rewrites ~740 files here (ruff-format on
pre-existing code). No pre-commit git hook is installed and CI runs only
`ruff check`, so run it with --files on your own paths or the diff explodes.
- Measuring streamed peak VRAM across setup() charges the pre-flight's own GEMM
probe (three 4096^3 matrices, ~100 MB) to the step. Reset the peak counter
after setup.
- The buffer pool is freed by cycle collection, not by close(): back-to-back
streamed runs in one process retain the previous pool (+47.65 MB measured)
until a gc pass. Call gc.collect() between arms when measuring.
- A resident model built from a float32 fixture vs a bf16 streamed one measures
the dtype gap, not streaming — that cost an hour chasing a 9.96e-04 "failure".
- Two concurrent pytest runs on a 4 GB card produce false CUDA failures; run the
suite alone.
Tests: 16977 -> 17051.
Lifts the v0.72.0-.2 scope freeze. Every capability was gated against a
streamed-vs-resident bit-exactness reference before it was written.
- Six more families (mistral/gemma/gemma2/gemma3_text/phi/phi3), each
bit-exact vs the same checkpoint loaded resident, under bf16 AND NF4.
Multimodal gemma3 stays refused — only gemma3_text.
- batch_size > 1 and gradient_accumulation_steps > 1 now work.
- A batch- and vocab-aware VRAM pre-flight that refuses a run predicted
not to fit. Fitted to 10 real runs: worst error 0.85%, never
under-predicts. On Windows an over-budget step does not OOM; WDDM
spills silently, so the estimator is the only guard.
- A throughput bracket from a GEMM ceiling measured on the user's own
card in the same session, printed with the SM clock.
- --resume / --hf-resume: load_state_dict narrows keys by child name, so
a canonical checkpoint matched 0 of N tensors and PEFT warned only.
Keys are now redirected at load time, mirroring the v0.72.1 save fix.
- An NVMe disk overflow tier (stream_source: auto|ram|disk), bit-exact
against the RAM tier. Its speed relative to RAM is UNMEASURED here and
no figure is claimed.
- soup doctor --disk reports the detected media type.
Fixes: estimate_logits_bytes charged 6 bytes/element where the measured
peak is 14; the NVMe tier guard was wired to a hardcoded constant;
streaming sources leaked handles when training raised; subprocess
helpers resolved tools by bare name (CWE-427 on Windows).
112 tests in tests/test_v07203.py; 16867 -> 16977.
The v0.72.2 heading change ("Layer Streaming (BETA, v0.72.0)" ->
"... v0.72.0; NF4 v0.72.2") moved its anchor, leaving the inbound pointer from
docs/training.md dead. Repointed, and took the opportunity to mention that
`quantization: 4bit` is what makes 8B fit a 4 GB card.
Also fixes a PRE-EXISTING dead self-link: the TOC entry for "FP8 Attention +
NVFP4 + Native unsloth_bnb_4bit" carried a "(v0.53.0)" suffix its heading never
had, so that anchor has never resolved.
Verified by walking every `performance-and-quantization.md#...` reference in
docs/ against the file's real headings, using GitHub's slug rule (each space
becomes one hyphen, so punctuation leaves a double hyphen — a naive
whitespace-collapsing slugger reports seven false positives here). 0 broken.
Docs-only: no version bump, no tag, does not ship to PyPI.
Layer streaming (v0.72.0) was bf16-only, capping it near 3B on a small card.
Quantising the streamed base to NF4 makes the RAM store ~4x smaller, which is
what brings 8B within reach.
Measured on a 4 GB RTX 3050 Laptop through the shipped code (50 steps after 10
warm-up, batch 1, S=512, PagedAdamW8bit, GEMM ceiling taken in the same session):
Llama-3.1-8B-Instruct 119.6 tok/s 3.32 GB peak 3.60 GB pinned 100% 952 MHz
Qwen2.5-3B 264.2 tok/s 1.76 GB peak 1.43 GB pinned 100% 960 MHz
3B is 1.85x the bf16 path, but that is PINNING, not arithmetic: 1.43 GB
page-locks where 5.55 GB did not, restoring async copy_ (util 79.3% -> 100%).
14B was not run — its store exceeds this box's measured 7.12 GB pinned ceiling.
A streamed NF4 run is bit-exact against a RESIDENT NF4 run, now as CPU-runnable
CI tests rather than only a gate result.
Notable, because each fails silently:
- PEFT dispatches lora.bnb.Linear4bit only when is_loaded_in_4bit is stamped;
without it the generic lora.layer.Linear runs against a Linear4bit base and
casts differently (9.375e-01 logit divergence, no warning). Pinned by a test
with a control that deletes the marker.
- hf_quantizer must be stamped too, or Trainer.__init__ dies formatting its own
"cannot fine-tune" error. Found by the end-to-end test.
- The shard cache is keyed on quant/double_quant/quant_device as well as dtype
and source fingerprint; a bf16 cache reused for an NF4 request would feed
full-precision bytes to matmul_4bit.
- index.json is a trust boundary: its shape/blocksize reach bnb kernels that do
not bounds-check, so from_json validates and the runtime cross-checks the
claim against the bytes on disk.
- A streamed NF4 model over-reported parameters ~6.5x (878,154,048 vs
134,515,008 for SmolLM2-135M). Display-only; ~52 B at 8B.
Scope unchanged and still BETA: RAM tier, sft, Llama/Qwen, batch 1, no
accumulation, no resume. quantization values other than none/4bit are refused.
Tests: 16752 -> 16840 (+88 in tests/test_v07202.py).
Full suite: 16734 passed, 129 skipped.
v0.72.0's layer-streaming wrapper holds the real decoder layer as a child
named `inner`, so every saved LoRA adapter key carried an `.inner.` segment.
Such a file reloads as ZERO tensors into any normal model: soup merge,
soup serve, soup chat and PeftModel.from_pretrained all returned the untuned
base while PEFT emitted only a UserWarning. Training was correct; only the
artifact was inert.
StreamedDecoderLayer.state_dict() now delegates to the wrapped layer at the
wrapper's own prefix, so every artifact path -- the final trainer.save_model(),
each save_steps checkpoint, and therefore soup adapters, the Registry, merge
and serve -- becomes canonical at once. Serialisation-only by design: the
forward path is untouched, so v0.72.0's bit-exactness gates remain valid
without being re-earned.
Also fixes --hf-resume bypassing the streaming resume refusal. The guard
tested only --resume, while --hf-resume reaches resume_from through another
branch. Pre-fix that combination matched keys by accident; post-fix it would
have matched nothing and silently continued training with a freshly
initialised adapter -- i.e. the adapter-key fix alone would have made that
one path worse.
Roadmap renumbered (this release was inserted ahead of NF4): every
"lands in vX.Y.Z" refusal corrected -- NF4 v0.72.2, disk tier / more
architectures / larger batches / gradient accumulation / checkpoint-resume
v0.72.3, preference losses v0.72.4.
Found by the v0.72.2 NF4 gate, not by the 159 v0.72.0 tests -- none of them
saved an adapter and loaded it back. The new regression test does exactly
that, by count, by name and by value, with a negative control that re-mangles
the keys and asserts the reload yields zeros (0-of-N loading raises nothing,
so a green round-trip without the control proves nothing).
Tests: +17 in tests/test_v07201.py (16735 -> 16752).
Full suite: 16623 passed, 129 skipped, 4 deselected.
Note for maintainers: do not edit soup_cli/__init__.py while a suite is in
flight -- a mid-run version bump made test_cli_subprocess::test_version fail
spuriously (the subprocess and the imported constant disagreed).
Two spots the release missed: the README's own docs-index row for
Performance & quantization (docs/README.md's equivalent row was already
updated), and docs/commands.md, which lists config-driven training
features in the same style as LISA and Spectrum.
Docs-only — no version bump.
The frozen base lives in CPU RAM and is streamed into a small pool of
pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is
bounded by ONE layer instead of the whole model. Only the LoRA adapters,
their gradients and optimizer state stay resident.
Measured on an RTX 3050 Laptop 4 GB (Windows, 16.9 GB RAM), batch 1,
gradient checkpointing on, 50 steps after 10 warm-up:
Qwen2.5-0.5B S=512 978.6 tok/s 91.4% util 1.47 GB peak
Qwen2.5-1.5B S=512 525.0 tok/s 96.8% util 1.82 GB peak
Qwen2.5-1.5B S=1024 487.6 tok/s 96.7% util 2.96 GB peak
Qwen2.5-3B S=512 143.1 tok/s 79.3% util 2.15 GB peak
Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs.
Honest cost: 1.43x slower than resident, measured at 0.5B — the only
apples-to-apples comparison available on this box, because 1.5B and above
cannot run resident here at all.
Correctness was gated before any src/ code was written: streamed vs
resident logits are bit-exact (max abs diff 0.0), the layer-0 LoRA
gradient is non-zero on all layers, a 100-step loss curve matches
resident exactly, and same-seed runs are identical.
New:
- utils/layer_stream.py pure planner (no top-level torch)
- utils/layer_shard.py per-layer safetensors sharder
- utils/layer_stream_runtime.py buffer pool, RAM source, prefetch, wrapper
- training.stream_layers / stream_source / stream_buffers
Notes for future maintainers:
- transformers' Trainer.__init__ and accelerate's prepare_model BOTH call
model.to(), which raises NotImplementedError on meta parameters. The
streamed layer overrides _apply to pass meta tensors through, and the
model declares hf_device_map. Without either, every run dies at trainer
construction — no unit test that stops at model(input_ids=...) sees it.
- The shard cache is keyed to a fingerprint of the source checkpoint, not
just the model slug: a base retrained in place must re-shard rather than
silently stream stale weights.
- The pre-flight hardware-fit gate models a RESIDENT run, so it is skipped
for streaming — otherwise it refuses exactly the runs this enables.
- expandable_segments:True is silently ignored on Windows; probed, not
claimed.
Scope (every refusal names the release that lifts it): RAM tier, bf16,
task=sft, Llama/Qwen, batch 1, no gradient accumulation, no --resume.
NF4 is v0.72.1; disk tier / bigger batches / accumulation / resume are
v0.72.2. Proof-of-mechanism at 3B — nothing above 3B was measured.
Tests: 16576 -> 16735 (+159 in tests/test_v07200.py)
Turn the reward-hacking detector on the verifier itself: feed empty /
length-padded / repetition / sentinel-spam completions and flag any the
verifier accepts. Loads via the existing load_reward_fn (probes a synth .py
or a builtin); a gold-requiring target with no --references is a hard error,
never a false "robust". Exit 0=robust / 2=gameable / 1=error. Pure, offline,
no schema change, no new deps.
Also corrects the ops-docs Telemetry section (the sender exists but is wired
to nothing — no data is sent). Telemetry flywheel deferred pending a public
privacy policy.
Tests: 16490 -> 16529 (+39). 5 sequential ECC reviews, every finding fixed.
Add utils/reward_synth to the CONTRIBUTING module list, fix the stale "Test
Files (313 -> 318)" count, and add reward-verifier synthesis to the training
docs-index row. Docs-only; no version bump.
Point `soup reward synth <refs.jsonl> -o reward.py` at reference (gold) outputs and it
infers a deterministic verifier (numeric / json_schema / regex / tool_call), emits a
readable, committable .py reward_fn that rides load_reward_fn's existing .py path (no new
exec surface), and — the moat — REFUSES to emit one that can't discriminate its references
from auto-perturbed negatives via a mandatory calibration report (accept refs >=90% AND
reject negatives; hard floor at discrimination<=0). Nothing in TRL/Unsloth/Axolotl
synthesizes a reward.
Fixes#311: a comma-separated reward_fn ("accuracy,format") now loads as a reward ensemble
(GRPOTrainer reward_funcs=[...], unlocks the rm_ensemble detector), GRPO-only and validated
at config-parse; the deepseek-v3-reasoning recipe that shipped this previously crashed with
"Unknown reward function".
Riders: reward_fn field-validator (null-byte/blank/oversize/empty-comma-segment); comma-aware
verifiable-domain check; envs/calculator + guess_number docstrings corrected.
5 sequential ECC reviews, every finding fixed (python HIGH PPO gate; code 2xHIGH per-tool
arg binding + json_schema mixed-shape refuse; security HIGH rel_hint codegen injection; tdd
8xHIGH). Live smoke on RTX 3050: synth from envs/calculator agrees with math_verify;
degenerate refused (exit 2); real GRPO on SmolLM2-135M with reward_fn=accuracy,format
completed optimizer steps. +103 tests (tests/test_v07140.py); 16387 -> 16490.
soup ship's leg 2 — the catastrophic-forgetting / regression gate that carries
the whole SHIP / DON'T-SHIP claim — was 15 trivia prompts scored by raw
substring containment (it credited "B" for "Berlin", "3" for "13") with zero
coverage for tool-calling, safety, or JSON. This makes the gate real.
- forgetting.py: score_answer/extract_mcq_letter replace the substring scorer
with answer-extraction (cue -> paren -> clause-terminating bare letter) +
boundary-aware token match. MINI_BENCHMARKS expanded (mmlu 26 / common_sense
24 / instruction 24) + new mini_arithmetic (36) so a 1-item flip trips 0.05.
BREAKING: an existing run's verdict can change (the old gate under-reported).
- eval/gate_suites.py (new): bundled offline general-suite registry, no torch.
DEFAULT_GENERAL_SUITE = the 4 MCQ suites + 3 behavioural JSONL suites
(mini_tool_call / mini_format_json / mini_safety) scored per-model-absolute
by the pure custom/diagnose scorers. _fraction_passing isolates a per-item
scorer exception (deep-JSON RecursionError scores as a failed item).
- ship.py: leg-2 scores bundled suites offline (base+tuned) before routing any
non-bundled name to lm-eval; default general suite = the full bundled set.
Exit-code taxonomy: usage errors move 2 -> 3 so exit 2 means only DON'T-SHIP
(a typo'd flag was previously indistinguishable from a caught regression).
- diagnose/__init__: "Six" -> "Seven" probes + re-export all 7 score_* fns;
removed the dead SUPPORTED_TASK_MODES "pairwise reserved" gate.
- Bundled gate fixtures ship in the wheel via the pyproject artifacts glob.
Every bundled item is original, hand-authored (no MMLU/GSM8K rows copied).
Test count 16288 -> 16330 (+42 in tests/test_v07138.py).
Every printed and documented `pip install 'soup-cli[extra]'` was bash / zsh /
PowerShell syntax and failed on Windows cmd.exe:
ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at
the start of dependency specifier
cmd.exe has no single-quote quoting, so it passes the quotes to pip verbatim
and pip rejects the requirement. Nothing in Soup can fix that once the command
is typed -- pip and the shell own it, and Soup is not installed yet when the
README line runs -- so the fix is the spelling we print.
Migrated 147 sites across 67 files to `pip install "soup-cli[extra]"`:
- 64 in src/ (Rich console hints + plain ImportError text)
- 57 in README.md + docs/
- 22 in src/soup_cli/templates/*.yaml + examples/configs/*.yaml
- 3 in examples/README.md
Double quotes are the only spelling valid in every shell (cmd, PowerShell,
bash, zsh), which is why the repo already used `pip install -e ".[dev]"`.
Measured on Windows: single quotes fail ONLY on cmd; double quotes pass
everywhere; bare passes on Windows but zsh globs `[extra]` and fails.
Method note (the PR #247 class): the hints sit INSIDE double-quoted Python
string literals, so a blind ' -> " sed produces SyntaxError. A tokenize-based
rewriter escaped `\"` in DQUOTE tokens and left bare `"` in TRIPLE / COMMENT
tokens; every touched .py was compile-checked. The full suite (not ruff, not
compile-check) caught two rewriter blind spots: the real YAML templates under
src/soup_cli/templates/ (byte-identical drift test) and examples/README.md.
A regression test (tests/test_v07137.py) scans the package and every docs code
block for the single-quoted form; prose may still name it so a reader from an
older tutorial recognises the error.
Also bundles #315 (@Sanjays2402): eval-gate benchmark tasks now run via
ForgettingDetector instead of a helper that never existed. Closes#310.
Test count: 16283 -> 16288 (+4 in tests/test_v07137.py).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Version 0.71.35 -> 0.71.36 in pyproject.toml + __init__.py.
Docs are written around what the live smoke MEASURED, not what the plan
assumed:
- The semantic-dedup headline is WITHDRAWN. docs/data.md leads with the
measured overlap: paraphrase cosines (0.49-0.76) overlap genuinely-distinct
rows (0.54-0.76), and "Add two numbers"/"Multiply two numbers" (0.759)
scores HIGHER than the true paraphrase "reverse a string"/"invert the order
of characters" (0.491). No threshold separates them, so the page says
plainly that lowering --threshold trades duplicates for silent data loss,
and explains why the 0.8 default is deliberately conservative. The claim it
does make -- catches REWORDINGS MinHash's shingling misses (0.88-0.91) --
is the one the numbers support.
- docs/data.md canary section states the real verdict rule (binomial tail
over the count, not any-single-canary) and why: at K=16 an any-canary rule
fires on a CLEAN model ~15% of the time.
- docs/training.md --replay reports the honest result: 7% better retention
than control, but forgetting without it was only +4% (mild), so it is
proof-of-mechanism at 135M + LoRA, not a production claim.
- CHANGELOG known-limitations carry the same numbers rather than hedging.
- README What's New leads with the two blocking bugs the smoke found (the
hardware-fit gate refusing locally-merged models; the [extra] hints
printing without the extra).
Counts 16001 -> 16254 tests, 313 -> 314 files (CONTRIBUTING).
Docs-only — no src/ changes, so no version bump (per the CI-only/docs-only rule).
Audit of the v0.71.35 release against the checklist found four misses:
* README's docs/ index table was missing the docs/compliance.md row — a new
topic page unreachable from the front door (step 9: "if the release adds a
whole new topic area, add/adjust the row in the README's docs/ index table").
It was only added to docs/README.md.
* The GGUF toolchain was never documented, despite the slot's own plan calling
it out ("Risk: MSVC build variance — document the exact toolchain; keep the
llama.cpp tag pinned"). Added the exact, verified build commands + the
VS2022 component actually required, the single- vs multi-config binary
layouts, and an explicit warning never to install llama.cpp's
requirements.txt (it pins torch~=2.2.1 CPU and downgrades a CUDA torch —
the bug fixed in v0.71.35).
* docs/peft-and-efficiency.md still said "16 built-in templates" — stale as a
direct result of the 4 compliance templates (now 21).
* Stale counts CONTRIBUTING "Test Files (307 files)" -> 313 and "existing 15
templates" -> 21 (that step also still pointed at schema.py rather than the
templates/ YAML + manifest.json registry).
The documented build command and binary path are the ones actually executed
during the release validation, not idealised.
Ship a regulated fine-tune with the paperwork it needs, plus GGUF export that
actually works on Windows.
Added:
* `soup init --template hipaa|soc2|eu-ai-act|sr-11-7` (templates 17 -> 21).
Design constraint verified in-repo, not assumed: Soup's compliance controls
are CLI flags/commands, NOT schema keys (audit_log/bom/attest/repro_receipt/
annex_xi/track_energy/pii/decontaminate have zero matches in config/schema.py),
so a template cannot "pre-wire audit-log on" as YAML. Each is a valid
SoupConfig + header comments naming that regime's exact commands.
* `soup card <registry-id> -o MODELCARD.md` — registry entry -> publishable HF
card (training config, eval scorecard, config/data hashes, lineage, artifacts).
Pure build_model_card() over dicts, reused by `soup push --card`.
* `soup ci init` — writes .github/workflows/soup-gate.yml chaining
data validate -> expect -> ship --evidence (exit 2 blocks the merge).
* docs/compliance.md quickstart.
Fixed (GGUF-on-Windows, validated end-to-end for the first time; closes the
CPU-validatable half of #70/#144). Four independently-fatal bugs:
* export cloned llama.cpp into the CURRENT directory: SOUP_DIR is the bare name
".soup" but was used relatively instead of anchored to home like tracker.py /
registry/store.py, so ~/.soup/llama.cpp was never found.
* the first GGUF export DOWNGRADED the user's torch and broke CUDA: the
auto-clone pip-installed llama.cpp's requirements.txt (pins torch~=2.2.1 from
the CPU index) into the user's interpreter. Observed live: torch 2.5.1+cu ->
2.2.2+cpu, transformers 4.57 -> 4.46. Now installs only gguf/sentencepiece/
protobuf, unpinned, non-fatally.
* a correctly-built llama.cpp was not found on Windows: MSVC (like Xcode) is a
multi-config generator emitting build/bin/Release/llama-quantize.exe.
* `soup deploy ollama` failed on a relative GGUF path ("pull model manifest:
file does not exist") — ollama resolves FROM against the Modelfile's dir and
Soup writes it to a temp dir. Modelfile now emits an absolute path.
Also fixed a pre-existing model-card injection hole (affects `soup push`'s own
auto-card): _render_training_section interpolated base/task/scheduler/recipe
unescaped, and SoupConfig.base/scheduler have no charset validator, so a
crafted-but-valid config could smuggle raw HTML or a code-span-breaking backtick
into a card published to the Hub.
Step-6 live smoke (real train -> registry push -> card) caught a bug 90 green
tests missed: is_adapter came only from registry artifacts, so a real LoRA run
with no artifacts rendered "Full model" + library_name: transformers — a false
claim in a provenance document.
5 sequential reviews, every finding fixed (code HIGH: the generated workflow ran
`pip install -e ".[dev]"`, which only works in the Soup source tree, breaking the
gate's first step for every downstream user; security HIGH/MEDIUM/3 LOW; tdd HIGH:
push --card was only --help-tested). All 5 new fixes mutation-verified as real
pins. Tests 15906 -> 16001 (+95); full suite 15872 passed / 126 skipped.
Version 0.71.32 -> 0.71.33. CHANGELOG (with a Known-limitations block), README
What's New, docs/commands.md, docs/serving-and-export.md, CLAUDE.md (arch + CLI
+ counts + history roll), CONTRIBUTING counts.
The live smoke changed the pitch. The plan promised '~1.5-2x faster to serve by
distilling its own draft'. Measured on SmolLM2-360M-Instruct <- 135M-Instruct:
the STOCK draft already scored 69.3% acceptance; distilling it gave 69.7% at 2
epochs and 69.3% at 10 -- no gain beyond noise. A small same-family draft is
already at its capacity ceiling vs the target, and logit KD cannot buy capacity
it does not have. Assisted decoding also measured 0.55-0.64x -- a net SLOWDOWN.
So the speedup claim is withdrawn, not shipped. The feature ships as the honest
gate: soup draft measure tells you whether speculative decoding is worth
enabling BEFORE you ship it, and on this pair it correctly says no. That
negative result is stated in the CHANGELOG, the README, the serving docs and
CLAUDE.md, and the pre-existing unverified '2-3x faster' line in the
speculative-decoding docs was tempered to match.
Full suite: 15680 passed / 124 skipped (15806 collected). ruff clean.
Move docs/superpowers/ (internal brainstorming plans + design specs) out of
the repo and into .gitignore; drop the internal doc reference in CONTRIBUTING.md.
Files remain locally, just no longer version-controlled.
Add `soup data doctor` and `soup data lint`, killing the top *silent*
fine-tune failures before a single training step: EOS-missing-from-labels
(the #1 "model never stops generating" bug), BOS duplication, no-system-role
templates, and preference-data length bias (the #1 silent DPO degradation) —
none of which any competitor (Unsloth/Axolotl/LlamaFactory) checks for.
- utils/data_doctor.py: 8-check chat-template compat report over a
tokenizer + sampled rows, OK/MINOR/MAJOR taxonomy mirroring diagnose;
--show-mask N renders per-token trained/masked colouring through the
SAME masking dispatch (_build_row_labels) the report itself uses, so
the two can never disagree about what's actually trained.
- utils/data_lint.py: preference-data linter (dpo/orpo/simpo/ipo/bco/kto)
— length bias (Cohen's d), label imbalance, near-duplicates (MinHash),
identical chosen==rejected pairs, prompt leakage.
- commands/data_doctor.py: Typer layer for both commands; strips C0
control bytes before untrusted dataset content reaches the terminal.
- commands/diagnose.py: hardens the --evidence loader against a TOCTOU
symlink swap (O_NOFOLLOW + fstat-on-open-fd), backporting the pattern
soup ship shipped in v0.71.25 (closes v0.71.25 known-limitation (4)).
Live smoke against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer
(Windows + RTX 3050) found and fixed two genuine bugs beyond the synthetic
fixtures: the EOS check needed to span-search the whole trained region
(not just the last token), and two apply_chat_template call sites needed
a broad except Exception for jinja2.exceptions.TemplateError.
+173 tests (14788 -> 15042). 5 sequential ECC reviews, every finding fixed.
Recipe-count drift: the CLI help (commands.md) and Web UI pages
(serving-and-export.md) claimed 43 ready-made recipes and the catalog
docstring said ~30, while RECIPES actually holds 116 (test_recipes already
asserts len == 116). Aligned every user-facing count to 116.
Also refreshed 6 recipe descriptions still tagged "schema-only stub
(live in v0.52.1)": the TTS task (#131) and BitNet 1.58 SFT (#134) went
live in v0.71.20, so orpheus/sesame/llasa/spark/oute TTS and the Falcon-E
BitNet recipe now read "live (v0.71.20)".
Doc-drift only — description strings + one comment; no functional change,
no version bump.
The live-codec block claimed the entire data.format=audio path was "not
validated on the maintainer's box" and only surfaced a RuntimeError. v0.71.22
made the Orpheus SNAC encode live + validated; note that while the other four
families stay dependency-gated. Docs-only, no version bump.
- performance-and-quantization.md: the Quant Menu multi-trainer note said
"vision / audio modality is still SFT-only inline-BNB (wiring tracked as a
follow-up)" — stale after v0.71.19 #81 dropped the modality gate. Now states
vision/audio thread the unified loader (full gptq/awq/hqq/aqlm/eetq/mxfp4/fp8
menu), with the upstream class+kernel caveat.
- peft-and-efficiency.md: added the v0.71.19 #80 multi-GPU sharding paragraph to
the Multipack section (accelerator.prepare + BatchSamplerShard under
num_processes>1; identical bin seed across ranks; single-GPU unchanged).
Docs-only — no version bump / tag (the v0.71.19 code already shipped at f51331d).
Alpamys
Alpamys
Salil M