Output of the first `pre-commit run --all-files` after the hooks were finally
activated in d0b7c62: trailing whitespace stripped and end-of-file newlines
normalised across 18 files.
No semantic change — `git diff --ignore-all-space --ignore-blank-lines` over
the whole set is empty. Committed as one batch on purpose: left alone, these
would surface as unrelated noise in the next real diff to each of these files,
since the hooks now fix them on the way past.
Three of the four bundles described a file other than the one they copy.
alpaca_demo advertised "20-row" for a 10-row fixture. Rather than correcting
the number, the count is gone: it was a fact about a file, hardcoded in a
different file, with nothing checking the two agree -- correcting it in place
leaves that mechanism intact for the next fixture edit. `soup data inspect`
reports the real count from the file, and examples/data/README.md tabulates
them.
The other two were found while checking for more bad numbers, and matter more
than the number did, because `format` is the value a user copies into
`data.format`:
- sharegpt_demo declared format="sharegpt" for chat_preferences.jsonl, which
is prompt/chosen/rejected -- detect_format() calls it dpo. Declaring
sharegpt sends preference rows to the wrong normalizer: a silently wrong
training run, not an error. Now dpo. The bundle NAME is left alone; it is a
public CLI argument and renaming it would break `soup data demo
sharegpt_demo`. The description now says what the file actually is.
- grpo_demo declared format="reasoning", which is not in the DataConfig
format literal at all, so anyone copying it into a config got a validation
error. The rows are alpaca-shaped, matching what
examples/configs/grpo_reasoning.yaml declares. Now alpaca.
tests/test_demo_bundle_metadata.py pins all three properties: declared format
equals detect_format() of the fixture, declared format is a literal the schema
accepts, and no description hardcodes a row count. Verified red before green:
against the previous metadata it reports 4 failed, 10 passed.
The examples rotted silently because nothing parsed them.
tests/test_dpo_example.py pins dpo_example.yaml by name, and that was the one
file which happened to be current -- the other seven drifted through several
schema changes with a green suite.
Two parametrized tests over glob("*.yaml"), so a new example is covered the
moment it is added; a hand-maintained list would carry the same blind spot
that caused this.
test_example_config_parses -- what `soup train --config` does before
it touches a model
test_example_config_data_file_exists -- a config referencing examples/ must
reference something that is there.
vision_llama was schema-valid at HEAD
of this branch and still impossible to
run. Templates opt out by using a
placeholder path rather than claiming a
bundled fixture.
test_example_configs_were_actually_found guards the parametrization: an empty
glob makes both tests vanish and the file reports green, which is the same
class of silent pass being fixed here.
Verified red before green: against the pre-fix configs this file reports
14 failed, 3 passed.
The layer-streaming pre-flight printed a Rich panel titled `soup train
--stream-layers`. There is no such flag — `soup train --help` contains zero
occurrences of "stream" — and streaming is enabled only by `training.stream_layers`
in soup.yaml. The panel is the first thing a streaming run prints, so it was the
feature's most-read line of documentation, and it pointed at a `No such option`.
Wording left over from the original design note, which proposed a CLI flag; the
feature shipped as a config key and the title never followed. Nothing asserted on
it, so nothing caught the drift.
Retitled to `training.stream_layers`. The body already opens with "Layer streaming
BETA", so the title does not repeat it. Same string fixed in the three module
docstrings and in one test docstring that quoted it.
Cosmetic only — no behaviour change. Verified by rendering the panel, and by
tests/test_v07200-04.py: 440 passed, 2 skipped. ruff clean.
benchmarks/gate-v0.72.*.md carry the phrase too and are deliberately untouched:
they are published verbatim gate records, kept as written.
Closes#329
v0.72.4 capped `trl<0.25` from a staged-removal table that scored a MODULE
RELOCATION as a field removal. At 0.25 `BCOConfig` moved into
`trl/experimental/bco/`, and at 0.26 kto/orpo/cpo followed — but every one of
them stayed publicly re-exported from `trl` with `max_prompt_length` intact.
Seeing a config file vanish from `trl/trainer/` was read as the field going
away.
Re-derived by parsing each *Config class's own annotated fields across every
wheel 0.24.0 -> 0.29.1 (all five inherit TrainingArguments, so there is no
inherited-field escape hatch):
version dpo kto orpo cpo bco
0.24.0 - 0.26.2 yes yes yes yes yes
0.27.0 - 0.27.2 yes NO yes yes yes
0.28.0 yes NO NO NO NO
0.29.0 - 0.29.1 NO NO NO NO NO
Then settled the way reading source cannot — by CONSTRUCTING all six configs
with the exact kwargs the wrappers pass:
trl 0.26.2 OK dpo · OK ipo · OK kto · OK orpo · OK simpo · OK bco
trl 0.27.0 FAIL kto: KTOConfig.__init__() got an unexpected keyword
argument 'max_prompt_length'
OK dpo · OK orpo <- control: the boundary is KTO's alone
So the cap is <0.27, and 0.25.0/0.25.1/0.26.0/0.26.1/0.26.2 were excluded for
no reason. That is the third wrong answer this bound has had (after "trl 1.x
removed them" and "the break is 0.29.0"), and the first two were also produced
by reading rather than running. The rule this earns: a version bound derived
from source is a hypothesis; the experiment that settles it is constructing the
object.
Two more corrections from the same pass:
- The floor >=0.7.0 was impossible, and nobody checked it while carefully
fixing the ceiling. `setup()` imports GRPOTrainer unconditionally and trl
first exports it at 0.14.0 (OnlineDPO / KTO / BCO / BasePairwiseJudge at
0.11.0; 0.7.0 has none of them). Resolvers pick the newest allowed version,
so it only bit under a constraints file or anyone reading the metadata as a
support claim.
- ORPOConfig/CPOConfig are not "deleted" at 0.29 — the modules survive under
trl/experimental/. They, BCOConfig and their trainers are dropped from the
public `trl` namespace, which is what Soup imports, so the 0.29 break is an
ImportError rather than a rejected kwarg. Harder than described, not milder,
and it means going past 0.29 needs an import strategy (#326).
Closes the blind spot that let this ship, for all six trainers
(tests/test_trl_preference_config_contract.py)
The bug was invisible because the trl imports and the config construction live
inside `setup()`, and constructing a wrapper touches trl not at all. v0.72.4's
streaming suite closed that for four of the six; tests/test_bco.py:266 and
tests/test_ipo.py:191 still MOCK `.setup` out, so bco and ipo had no test that
executed it. The new file drives the real `setup()` for all six on the ordinary
non-streaming path, asserts the BUILT config carries max_prompt_length/
max_length (so a trl release that accepts the kwarg and stops storing it fails
too), and derives the covered set from the trainer sources so a seventh trainer
adopting the argument joins automatically.
Revert the KTO skip: it blamed the device, and the variable is the torch version
`skipif(not cuda)` on test_kto_streams_at_batch_two made the test dead in CI —
there are no GPU runners (ubuntu/windows/macos) — leaving it alive only on this
box, under the one torch where it passes, while torch carries no upper bound.
Three things say the rationale ("a streamed model on CPU is a test convenience")
does not hold:
- the same CI run had test_v07200.py::test_one_training_step_actually_runs, the
identical streamed train() for SFT, pass on that CPU runner;
- running this body here with CUDA masked passes (torch 2.5.1);
- the error comes from check_same_device, i.e. an op received a meta
placeholder next to a real tensor — a streaming property. Newer torch
decomposes more ops, which is why only the newer stack surfaces it.
Now it runs everywhere and xfails on exactly that signature on CPU; anything
else, and the same signature on CUDA, is a hard failure. Real defect filed as
#328. With CUDA hidden: 21 passed, 2 skipped (both genuinely CUDA-only).
Also fixed
- pyproject.toml: 14 double-encoded em-dashes, one of them the `unit` marker
description that `pytest --markers` prints to users. The existing mojibake
guard covered only the package docstring; widened to the file, which is not
importable Python and so was invisible to every source-level check.
- docs/commands.md called itself "the full soup command list" while missing
eight, three of them glued onto the end of a previous line by absent newlines
and therefore invisible on the rendered page. Now 77/77, asserted against the
live Typer app so the claim stays true by construction.
Verified: ruff clean; full suite 16935 passed / 129 skipped / 0 failed (13:15);
wheel builds with Requires-Dist: trl<0.27,>=0.14.0; `pytest --markers` renders
the em-dash. No version bump and no tag — the packaging metadata change wants
CI on the resolver stack before it rides a release.
cb44d9e claimed three fixes and shipped one. The scripted edit aborted on its
first assertion and wrote nothing; I then applied only the bf16 fix by hand and
did not re-check the other two. Neither local run could catch it: the CUDA-gated
test passes on this box's CPU under torch 2.5.1 (it only fails on CI's 2.13),
and there is no MPS device here, so both "fixed" tests looked green locally
while carrying no marker at all.
Applied and verified two ways this time:
- structurally, by parsing the file and asserting each decorator is attached to
the test it belongs to;
- behaviourally, by running with CUDA hidden and confirming the gated test
reports SKIPPED rather than silently executing.
62 passed / 5 skipped with CUDA hidden; 67 passed on CUDA.
Down from 34 CI failures to 0 after the trl cap; these three are the remainder,
all in the new tests and all device- or version-specific rather than product
bugs:
- test_trl_itself_still_refuses_batch_one built a KTOConfig without bf16=False.
Newer TRL configs default bf16 on, so a CPU-only runner raised "Your setup
doesn't support bf16/gpu" before ever reaching the batch-size check the test
exists to pin. Reproduced locally with CUDA_VISIBLE_DEVICES=-1.
- test_kto_streams_at_batch_two runs a real trainer.train(). That works on CUDA
(dev box and CI) but fails CPU-only under torch 2.13/trl 0.24 with "Tensor on
device cpu is not on the expected device meta!". Streaming exists to bound
VRAM, so a streamed model on CPU is a test convenience rather than a
configuration — the stance v0.72.3 already took for PEFT re-dispatch — so the
full-step test is gated to the production device. KTO's schema gate, setup,
reference behaviour and layer-read accounting are all still checked on CPU.
- test_a_reference_using_loss_reads_more_layers_than_sft[kto] hit MPS on the
macOS runners; its sibling tests already carry the MPS skip and this one had
been missed.
Verified: 9 passed with CUDA hidden, 67 passed on CUDA.
The published benchmarks record notes the CPU limitation, and that it was
invisible on a CUDA dev box — a locally-green suite is a weak signal whenever a
path forks on device.
DPO / ORPO / SimPO / KTO join task=sft on the layer-streaming engine.
The reference model is the SAME streamed base with adapters disabled — one set
of weights, one stream. Measured: streamed DPO peaks at 0.914x streamed SFT with
a byte-identical store and pool, where forcing a real second instance costs
+730.44 MB against 730.44 MB of weights. All four are bit-exact (0.0) against a
resident run of the same loss.
KTO is NOT reference-free: kto_trainer.py:466-476 is byte-for-byte DPO's
three-branch reference selection, so it was gated separately. It also requires
batch_size >= 2, refused at parse time rather than minutes into sharding.
grpo/ppo stay excluded permanently — rollouts re-read every layer per generated
token. The refusal deliberately names no release.
The ~390-line streaming setup moved verbatim into trainer/stream_setup.py so
five wrappers cannot drift. _STREAM_ROWS_PER_EXAMPLE is 2 for the concatenating
losses and 1 for KTO, measured not assumed: the VRAM pre-flight would otherwise
under-predict by half, and on Windows that is a silent spill, not an error.
Honest cost: the reference is free in memory, not in time — DPO reads the layer
stack 1.52x as often per step as SFT.
Also closes five holes in the release checklist itself:
- benchmarks/ was never in it, so gate records (which live under a gitignored
.claude/) were never published. The public record behind the preprint DOI was
about to fall a release behind; benchmarks/gate-v0.72.4-preference-losses.md
and its index row are here, and the checklist now names the step.
- tests/test_version_sync.py asserts pyproject.toml == __init__.py. Every other
version test in the suite is a >= floor check, so bumping one and forgetting
the other kept CI green. Verified red-green.
- .claude/paper/ (the DOI preprint) had no "did this release change what it
claims?" step. For v0.72.4 the answer is no: no measured number moves and its
task: sft configs stay valid.
- The Docs section header said steps 7-12 while containing 7-13, so plan.md sat
outside its own section.
- The README size anchor said ~238 lines against a real 426.
Notes for whoever hits these next:
- `pre-commit run --all-files` rewrites ~740 files here (ruff-format on
pre-existing code). No pre-commit git hook is installed and CI runs only
`ruff check`, so run it with --files on your own paths or the diff explodes.
- Measuring streamed peak VRAM across setup() charges the pre-flight's own GEMM
probe (three 4096^3 matrices, ~100 MB) to the step. Reset the peak counter
after setup.
- The buffer pool is freed by cycle collection, not by close(): back-to-back
streamed runs in one process retain the previous pool (+47.65 MB measured)
until a gc pass. Call gc.collect() between arms when measuring.
- A resident model built from a float32 fixture vs a bf16 streamed one measures
the dtype gap, not streaming — that cost an hour chasing a 9.96e-04 "failure".
- Two concurrent pytest runs on a 4 GB card produce false CUDA failures; run the
suite alone.
Tests: 16977 -> 17051.
Two NF4 bit-exactness tests failed on all nine CI jobs while passing locally.
Root cause reproduced directly: bitsandbytes' CPU 4-bit forward calls
`_convert_weight_packed_for_cpu`, which reshapes absmax to
[rows, blocks_per_row]. At hidden_size 32 a weight has 32*32/64 = 16 absmax
blocks for 32 rows, so blocks_per_row floors to ZERO and it raises
"shape '[32, 0]' is invalid for input of size 16" — the exact CI error.
Verified 32 raises while 64 and 128 do not.
A CUDA build never calls that function, which is why a GPU development box
cannot see this and every CPU-only runner fails. The fixtures move to
hidden_size 64 with the reason recorded in-place so they are not shrunk back.
Fixture sizing only: nothing about the product changed, and both suites still
assert bit-exactness.
Lifts the v0.72.0-.2 scope freeze. Every capability was gated against a
streamed-vs-resident bit-exactness reference before it was written.
- Six more families (mistral/gemma/gemma2/gemma3_text/phi/phi3), each
bit-exact vs the same checkpoint loaded resident, under bf16 AND NF4.
Multimodal gemma3 stays refused — only gemma3_text.
- batch_size > 1 and gradient_accumulation_steps > 1 now work.
- A batch- and vocab-aware VRAM pre-flight that refuses a run predicted
not to fit. Fitted to 10 real runs: worst error 0.85%, never
under-predicts. On Windows an over-budget step does not OOM; WDDM
spills silently, so the estimator is the only guard.
- A throughput bracket from a GEMM ceiling measured on the user's own
card in the same session, printed with the SM clock.
- --resume / --hf-resume: load_state_dict narrows keys by child name, so
a canonical checkpoint matched 0 of N tensors and PEFT warned only.
Keys are now redirected at load time, mirroring the v0.72.1 save fix.
- An NVMe disk overflow tier (stream_source: auto|ram|disk), bit-exact
against the RAM tier. Its speed relative to RAM is UNMEASURED here and
no figure is claimed.
- soup doctor --disk reports the detected media type.
Fixes: estimate_logits_bytes charged 6 bytes/element where the measured
peak is 14; the NVMe tier guard was wired to a hardcoded constant;
streaming sources leaked handles when training raised; subprocess
helpers resolved tools by bare name (CWE-427 on Windows).
112 tests in tests/test_v07203.py; 16867 -> 16977.
CI was red on macOS only (3/11 jobs); ubuntu and windows were green across
3.10/3.11/3.12, as were lint and type-check.
Cause is not bitsandbytes availability but device disagreement: on an
Apple-Silicon runner with no CUDA, TrainingArguments picks `mps`, while this
suite builds the streamed model on `cpu`. The batch is then moved to MPS and
the step raises "Placeholder storage has not been allocated on MPS device!".
Only the two tests that actually call trainer.train() were affected;
test_setup_builds_a_real_trl_trainer_under_nf4 passed, because building the
trainer never touches a device.
v0.72.0 hit exactly this and guards test_one_training_step_actually_runs the
same way; this mirrors that helper rather than inventing a second one. NF4
streaming is measured on CUDA and CPU only, and bitsandbytes' 4-bit kernels
have no MPS support, so skipping is the honest outcome — not a claim that it
works there.
Verified on the CUDA dev box: 88 passed, zero skipped, i.e. the guard does not
over-skip where the tests are meaningful.
Layer streaming (v0.72.0) was bf16-only, capping it near 3B on a small card.
Quantising the streamed base to NF4 makes the RAM store ~4x smaller, which is
what brings 8B within reach.
Measured on a 4 GB RTX 3050 Laptop through the shipped code (50 steps after 10
warm-up, batch 1, S=512, PagedAdamW8bit, GEMM ceiling taken in the same session):
Llama-3.1-8B-Instruct 119.6 tok/s 3.32 GB peak 3.60 GB pinned 100% 952 MHz
Qwen2.5-3B 264.2 tok/s 1.76 GB peak 1.43 GB pinned 100% 960 MHz
3B is 1.85x the bf16 path, but that is PINNING, not arithmetic: 1.43 GB
page-locks where 5.55 GB did not, restoring async copy_ (util 79.3% -> 100%).
14B was not run — its store exceeds this box's measured 7.12 GB pinned ceiling.
A streamed NF4 run is bit-exact against a RESIDENT NF4 run, now as CPU-runnable
CI tests rather than only a gate result.
Notable, because each fails silently:
- PEFT dispatches lora.bnb.Linear4bit only when is_loaded_in_4bit is stamped;
without it the generic lora.layer.Linear runs against a Linear4bit base and
casts differently (9.375e-01 logit divergence, no warning). Pinned by a test
with a control that deletes the marker.
- hf_quantizer must be stamped too, or Trainer.__init__ dies formatting its own
"cannot fine-tune" error. Found by the end-to-end test.
- The shard cache is keyed on quant/double_quant/quant_device as well as dtype
and source fingerprint; a bf16 cache reused for an NF4 request would feed
full-precision bytes to matmul_4bit.
- index.json is a trust boundary: its shape/blocksize reach bnb kernels that do
not bounds-check, so from_json validates and the runtime cross-checks the
claim against the bytes on disk.
- A streamed NF4 model over-reported parameters ~6.5x (878,154,048 vs
134,515,008 for SmolLM2-135M). Display-only; ~52 B at 8B.
Scope unchanged and still BETA: RAM tier, sft, Llama/Qwen, batch 1, no
accumulation, no resume. quantization values other than none/4bit are refused.
Tests: 16752 -> 16840 (+88 in tests/test_v07202.py).
Full suite: 16734 passed, 129 skipped.
`soup mcp serve` was the named example: commands/mcp.py loads at startup, but
mcp_server/registry.py is not touched until the command runs, and the server
blocks -- so it cannot be probed as an invocation. Covered as a direct module
import instead, together with three other lazily-loaded light cores
(utils/advise, eval/gate_suites, recipes/catalog).
All four measured clean. This closes the gap between "all 88 command modules
are imported at startup" (true, pinned) and "therefore every light code path is
covered" (was not true -- a command body's lazy import reached neither check).
Review question: `import soup_cli.cli` proves the app module is clean, but
`soup version`, `soup recipes list` and `soup mcp serve` each reach their own
command module -- does the guard actually cover them, or is that an assumption?
Measured, not assumed. Two additions:
test_every_command_module_is_imported_at_startup pins WHY one assertion is
enough today: cli.py registers all 88 command modules eagerly, so a top-level
leak in any of them is already caught. If registration is ever made lazy this
goes red, and the per-invocation test below becomes the only coverage.
test_light_command_invocation_stays_light runs 11 light commands for real
(version, --help, recipes list, recipes search, data/mcp/ship/advise/doctor/
reward --help, draft list) and asserts none pulls the training stack. This
covers what the startup assertion structurally cannot: a module a command body
imports lazily at call time. `soup mcp serve` is the motivating shape --
commands/mcp.py loads at startup, mcp_server/registry.py does not.
Mutation-verified rather than assumed green: an `import torch` placed INSIDE
the `draft list` body leaves the startup guard passing and turns exactly
`test_light_command_invocation_stays_light[draft list]` red. The two tests
cover genuinely different surfaces.
Two bugs in the first cut of the invocation test, both found by running it:
a two-line probe format collapsed under .strip() when the heavy-dep list was
empty (i.e. on every passing run), and the replacement parsed the exit code
with endswith("0"), which would have accepted 10 and 20. Now a marker-prefixed
single line with an exact comparison.
Full suite on Python 3.10 before this commit: 16630 passed, 129 skipped.
v0.71.41 put utils/reward_stress on the light CLI path via `soup reward
stress`. reward_stress imported utils/reward_hack_control to reuse the single
string constant "GOLD" -- and reward_hack_control resolves its TrainerCallback
base at module scope:
_TrainerCallbackBase = _get_trainer_callback_base() # calls the factory
The factory is written lazily (the transformers import sits inside it) but it
is CALLED at module scope, so the laziness evaporates. Importing it costs
~4.4s of transformers + torch. Every soup invocation paid for the training
stack, including commands that never load a model.
Measured on this box: `soup --help` 6.0s -> 1.15s (5.2x); `import
soup_cli.cli` alone 5.1s -> 0.72s (7.1x). No wrong results were ever produced
-- this was purely startup latency, and the light core still fell back
correctly when torch was absent.
Fixed by copying the constant into reward_stress rather than importing a 4.4s
module for it, with a test pinning the two values equal so the duplication
cannot drift.
The test gap is the more important half. Fifty-five test files carry per-file
`test_no_top_level_torch` guards that parse a module's source and assert it
contains no top-level `import torch`. That proves a syntactic property; the
requirement is a runtime one, and an AST walk can see neither a transitive
import nor a factory that is called at module scope. Every guard was green
throughout. tests/test_v07126.py is the sharpest case: it imports
reward_hack_control (loading transformers) and then asserts the source has no
top-level transformers import -- the test performs the thing it certifies
against.
tests/test_cli_startup_is_light.py replaces them for CLI purposes with one
runtime assertion that covers every module transitively and cannot drift. It
runs in a subprocess: the pytest process has torch loaded from other suites,
so an in-process assert would be vacuous. A control test verifies the probe
still detects torch when it IS present -- without it, a silently broken probe
would pass forever.
Eleven modules share the eager-factory pattern; only this one was reachable
from a light command, so the rest are latent rather than broken (the trainer
path loads transformers anyway). Tracked in #320, deliberately not scheduled.
trainer/raft.py resolves its base inside a factory function and stays clean --
that is the reference implementation already in the repo.
Also corrects the docstring in tests/test_issue308_callback_subclass.py, which
claimed the factory "keeps the modules transformers-free at import time".
Measured: it does not, and that belief is what let the pattern spread.
Verified on Python 3.10 (the repo's primary env): 16/16 pass across the new
guard and the #308 suite. On Python 3.12 two pre-existing #308 failures come
from a torch/torchvision mismatch in that interpreter, unrelated to this
change (only a docstring was touched in that file).
v0.72.0's layer-streaming wrapper holds the real decoder layer as a child
named `inner`, so every saved LoRA adapter key carried an `.inner.` segment.
Such a file reloads as ZERO tensors into any normal model: soup merge,
soup serve, soup chat and PeftModel.from_pretrained all returned the untuned
base while PEFT emitted only a UserWarning. Training was correct; only the
artifact was inert.
StreamedDecoderLayer.state_dict() now delegates to the wrapped layer at the
wrapper's own prefix, so every artifact path -- the final trainer.save_model(),
each save_steps checkpoint, and therefore soup adapters, the Registry, merge
and serve -- becomes canonical at once. Serialisation-only by design: the
forward path is untouched, so v0.72.0's bit-exactness gates remain valid
without being re-earned.
Also fixes --hf-resume bypassing the streaming resume refusal. The guard
tested only --resume, while --hf-resume reaches resume_from through another
branch. Pre-fix that combination matched keys by accident; post-fix it would
have matched nothing and silently continued training with a freshly
initialised adapter -- i.e. the adapter-key fix alone would have made that
one path worse.
Roadmap renumbered (this release was inserted ahead of NF4): every
"lands in vX.Y.Z" refusal corrected -- NF4 v0.72.2, disk tier / more
architectures / larger batches / gradient accumulation / checkpoint-resume
v0.72.3, preference losses v0.72.4.
Found by the v0.72.2 NF4 gate, not by the 159 v0.72.0 tests -- none of them
saved an adapter and loaded it back. The new regression test does exactly
that, by count, by name and by value, with a negative control that re-mangles
the keys and asserts the reload yields zeros (0-of-N loading raises nothing,
so a green round-trip without the control proves nothing).
Tests: +17 in tests/test_v07201.py (16735 -> 16752).
Full suite: 16623 passed, 129 skipped, 4 deselected.
Note for maintainers: do not edit soup_cli/__init__.py while a suite is in
flight -- a mid-run version bump made test_cli_subprocess::test_version fail
spuriously (the subprocess and the imported constant disagreed).
CI went red on 8 of 9 test jobs. Both causes were test bugs, not product
bugs — the streaming code is unchanged.
1. `test_control_chars_in_override_rejected` set an env var containing a
NUL byte. A NUL cannot live in an environment variable at all: POSIX
putenv rejects it, and CPython >= 3.11 rejects it on Windows too. The
only combination that ever allowed it was windows + 3.10, which is
exactly the box it was written on — so it passed locally and failed on
the other eight jobs with "embedded null byte".
Now uses ESC (0x1b), which is still < 0x20 and still must be refused.
The override is otherwise a VALID path under $TMPDIR, so the control
character is the only reason it can be rejected — verified by backing
the `ord(ch) < 0x20` guard out and watching the test go red.
2. `test_one_training_step_actually_runs` failed on macOS with
"found at least two devices, mps:0 and cpu". The suite builds the
streamed model on cpu when CUDA is absent, but transformers picks `mps`
as its default device on Apple Silicon, so the trainer moved batches to
a device the model was not on. Skipped when MPS is the accelerator:
v0.72.0 measured CUDA and CPU only, and claiming MPS support that has
never been run would be worse than skipping. The test still executes on
every CUDA and pure-CPU runner.
No src/ changes. Test count unchanged at 16735.
The frozen base lives in CPU RAM and is streamed into a small pool of
pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is
bounded by ONE layer instead of the whole model. Only the LoRA adapters,
their gradients and optimizer state stay resident.
Measured on an RTX 3050 Laptop 4 GB (Windows, 16.9 GB RAM), batch 1,
gradient checkpointing on, 50 steps after 10 warm-up:
Qwen2.5-0.5B S=512 978.6 tok/s 91.4% util 1.47 GB peak
Qwen2.5-1.5B S=512 525.0 tok/s 96.8% util 1.82 GB peak
Qwen2.5-1.5B S=1024 487.6 tok/s 96.7% util 2.96 GB peak
Qwen2.5-3B S=512 143.1 tok/s 79.3% util 2.15 GB peak
Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs.
Honest cost: 1.43x slower than resident, measured at 0.5B — the only
apples-to-apples comparison available on this box, because 1.5B and above
cannot run resident here at all.
Correctness was gated before any src/ code was written: streamed vs
resident logits are bit-exact (max abs diff 0.0), the layer-0 LoRA
gradient is non-zero on all layers, a 100-step loss curve matches
resident exactly, and same-seed runs are identical.
New:
- utils/layer_stream.py pure planner (no top-level torch)
- utils/layer_shard.py per-layer safetensors sharder
- utils/layer_stream_runtime.py buffer pool, RAM source, prefetch, wrapper
- training.stream_layers / stream_source / stream_buffers
Notes for future maintainers:
- transformers' Trainer.__init__ and accelerate's prepare_model BOTH call
model.to(), which raises NotImplementedError on meta parameters. The
streamed layer overrides _apply to pass meta tensors through, and the
model declares hf_device_map. Without either, every run dies at trainer
construction — no unit test that stops at model(input_ids=...) sees it.
- The shard cache is keyed to a fingerprint of the source checkpoint, not
just the model slug: a base retrained in place must re-shard rather than
silently stream stale weights.
- The pre-flight hardware-fit gate models a RESIDENT run, so it is skipped
for streaming — otherwise it refuses exactly the runs this enables.
- expandable_segments:True is silently ignored on Windows; probed, not
claimed.
Scope (every refusal names the release that lifts it): RAM tier, bf16,
task=sft, Llama/Qwen, batch 1, no gradient accumulation, no --resume.
NF4 is v0.72.1; disk tier / bigger batches / accumulation / resume are
v0.72.2. Proof-of-mechanism at 3B — nothing above 3B was measured.
Tests: 16576 -> 16735 (+159 in tests/test_v07200.py)
The pre-existing suite only validated OnlineDPOTrainerWrapper.setup() (trainer
build) on trl 1.x — never a real train() step on the reward_funcs path. Add a
smoke-marked, version-agnostic train() test: it runs a couple of online-DPO
steps on tiny-random-gpt2 with a synthetic length-preferring judge (via the
_ONLINE_DPO_JUDGE_OVERRIDE seam) and asserts rewards/chosen + rewards/rejected
appear in log_history, proving the reward signal is applied. The wrapper adapts
the evaluator to whichever trl is installed, so pytest -m smoke exercises the
reward_funcs path under trl 1.x and the judge path under trl 0.19.x; a companion
test pins that trl 1.x routes through reward_funcs (skips on 0.19.x).
Executed live on trl 0.19.1 + torch 2.5.1 (CPU): logs rewards/chosen,
rewards/rejected, rewards/accuracies, rewards/margins. The trl 1.x reward_funcs
execution needs a trl>=1.7 + torch>=2.6 env (CI's torch-2.6 job) — same test
body, no modification.
SmolVLM/Idefics3 vision SFT crashed with 'Idefics3Processor object has no
attribute pad_token': HF vision processors keep the text tokenizer nested at
processor.tokenizer and don't forward token-level attributes (ProcessorMixin
has no __getattr__), but TRL's SFTTrainer reads processing_class.pad_token /
.eos_token / .convert_tokens_to_ids directly. Add
_ensure_vision_processor_pad_token: set pad_token = eos_token on the inner
tokenizer when unset, then mirror the token surface + convert_tokens_to_ids
onto the processor — only for attributes it doesn't already expose, so a
LLaVA-style or tokenizer-like processing_class is untouched (no regression).
Verified live on SmolVLM-256M (RTX 3050): setup, tokenization, and PAD/BOS/EOS
alignment now succeed. A full training STEP still needs Idefics3-aware vision
collation (pixel_values + image-token expansion) — the LLaVA-era path pre-renders
text and never builds pixel_values, so Idefics3.forward gets 3D input_ids. That
collation rework is left open under #302; the recipe stays parse-only with an
updated note.
soup adapters arithmetic previously did a signed element-wise merge over
the intersection of lora_A/lora_B tensors, requiring all inputs to share
one rank — a mixed-rank input was refused. Add merge_task_arithmetic_concat
(PEFT combination_type='cat' style): stack the factors so
B_out @ A_out = Σ cᵢ·(Bᵢ@Aᵢ) exactly for any per-adapter rank, folding
coeff into Bᵢ and each adapter's decode-time scaling (lora_alpha/r) into
Aᵢ, with an optional --rank truncated-SVD refactor to cap the concatenated
rank. The CLI routes mixed-rank (or explicit --rank) inputs through the new
path and patches the output adapter_config r/lora_alpha to match; same-rank
inputs keep the fast element-wise path unchanged. read_adapter_base is
refactored onto a shared _read_adapter_config_dict; read_adapter_lora_scaling
added; write_merged_adapter gains config_overrides.
HF CallbackHandler.call_event dispatches every Trainer event via
getattr(cb, event)(...) with no hasattr guard, so a duck-typed callback
added via trainer.add_callback crashes with AttributeError on the first
unimplemented event (on_epoch_begin, fired before the first optimizer
step). Subclass the lazily-resolved transformers.TrainerCallback via the
_try_import_callback_base() factory (mirrors curriculum_callback.py /
lisa.py) so both inherit the no-op defaults for the ~13 unimplemented
events while keeping the modules transformers-free at import time.
Turn the reward-hacking detector on the verifier itself: feed empty /
length-padded / repetition / sentinel-spam completions and flag any the
verifier accepts. Loads via the existing load_reward_fn (probes a synth .py
or a builtin); a gold-requiring target with no --references is a hard error,
never a false "robust". Exit 0=robust / 2=gameable / 1=error. Pure, offline,
no schema change, no new deps.
Also corrects the ops-docs Telemetry section (the sender exists but is wired
to nothing — no data is sent). Telemetry flywheel deferred pending a public
privacy policy.
Tests: 16490 -> 16529 (+39). 5 sequential ECC reviews, every finding fixed.
Point `soup reward synth <refs.jsonl> -o reward.py` at reference (gold) outputs and it
infers a deterministic verifier (numeric / json_schema / regex / tool_call), emits a
readable, committable .py reward_fn that rides load_reward_fn's existing .py path (no new
exec surface), and — the moat — REFUSES to emit one that can't discriminate its references
from auto-perturbed negatives via a mandatory calibration report (accept refs >=90% AND
reject negatives; hard floor at discrimination<=0). Nothing in TRL/Unsloth/Axolotl
synthesizes a reward.
Fixes#311: a comma-separated reward_fn ("accuracy,format") now loads as a reward ensemble
(GRPOTrainer reward_funcs=[...], unlocks the rm_ensemble detector), GRPO-only and validated
at config-parse; the deepseek-v3-reasoning recipe that shipped this previously crashed with
"Unknown reward function".
Riders: reward_fn field-validator (null-byte/blank/oversize/empty-comma-segment); comma-aware
verifiable-domain check; envs/calculator + guess_number docstrings corrected.
5 sequential ECC reviews, every finding fixed (python HIGH PPO gate; code 2xHIGH per-tool
arg binding + json_schema mixed-shape refuse; security HIGH rel_hint codegen injection; tdd
8xHIGH). Live smoke on RTX 3050: synth from envs/calculator agrees with math_verify;
degenerate refused (exit 2); real GRPO on SmolLM2-135M with reward_fn=accuracy,format
completed optimizer steps. +103 tests (tests/test_v07140.py); 16387 -> 16490.
soup ship's leg 2 — the catastrophic-forgetting / regression gate that carries
the whole SHIP / DON'T-SHIP claim — was 15 trivia prompts scored by raw
substring containment (it credited "B" for "Berlin", "3" for "13") with zero
coverage for tool-calling, safety, or JSON. This makes the gate real.
- forgetting.py: score_answer/extract_mcq_letter replace the substring scorer
with answer-extraction (cue -> paren -> clause-terminating bare letter) +
boundary-aware token match. MINI_BENCHMARKS expanded (mmlu 26 / common_sense
24 / instruction 24) + new mini_arithmetic (36) so a 1-item flip trips 0.05.
BREAKING: an existing run's verdict can change (the old gate under-reported).
- eval/gate_suites.py (new): bundled offline general-suite registry, no torch.
DEFAULT_GENERAL_SUITE = the 4 MCQ suites + 3 behavioural JSONL suites
(mini_tool_call / mini_format_json / mini_safety) scored per-model-absolute
by the pure custom/diagnose scorers. _fraction_passing isolates a per-item
scorer exception (deep-JSON RecursionError scores as a failed item).
- ship.py: leg-2 scores bundled suites offline (base+tuned) before routing any
non-bundled name to lm-eval; default general suite = the full bundled set.
Exit-code taxonomy: usage errors move 2 -> 3 so exit 2 means only DON'T-SHIP
(a typo'd flag was previously indistinguishable from a caught regression).
- diagnose/__init__: "Six" -> "Seven" probes + re-export all 7 score_* fns;
removed the dead SUPPORTED_TASK_MODES "pairwise reserved" gate.
- Bundled gate fixtures ship in the wheel via the pyproject artifacts glob.
Every bundled item is original, hand-authored (no MMLU/GSM8K rows copied).
Test count 16288 -> 16330 (+42 in tests/test_v07138.py).
Every printed and documented `pip install 'soup-cli[extra]'` was bash / zsh /
PowerShell syntax and failed on Windows cmd.exe:
ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at
the start of dependency specifier
cmd.exe has no single-quote quoting, so it passes the quotes to pip verbatim
and pip rejects the requirement. Nothing in Soup can fix that once the command
is typed -- pip and the shell own it, and Soup is not installed yet when the
README line runs -- so the fix is the spelling we print.
Migrated 147 sites across 67 files to `pip install "soup-cli[extra]"`:
- 64 in src/ (Rich console hints + plain ImportError text)
- 57 in README.md + docs/
- 22 in src/soup_cli/templates/*.yaml + examples/configs/*.yaml
- 3 in examples/README.md
Double quotes are the only spelling valid in every shell (cmd, PowerShell,
bash, zsh), which is why the repo already used `pip install -e ".[dev]"`.
Measured on Windows: single quotes fail ONLY on cmd; double quotes pass
everywhere; bare passes on Windows but zsh globs `[extra]` and fails.
Method note (the PR #247 class): the hints sit INSIDE double-quoted Python
string literals, so a blind ' -> " sed produces SyntaxError. A tokenize-based
rewriter escaped `\"` in DQUOTE tokens and left bare `"` in TRIPLE / COMMENT
tokens; every touched .py was compile-checked. The full suite (not ruff, not
compile-check) caught two rewriter blind spots: the real YAML templates under
src/soup_cli/templates/ (byte-identical drift test) and examples/README.md.
A regression test (tests/test_v07137.py) scans the package and every docs code
block for the single-quoted form; prose may still name it so a reader from an
older tutorial recognises the error.
Also bundles #315 (@Sanjays2402): eval-gate benchmark tasks now run via
ForgettingDetector instead of a helper that never existed. Closes#310.
Test count: 16283 -> 16288 (+4 in tests/test_v07137.py).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The benchmark gate-task test pins score == 0.4, which is 2/5 -- exactly
two MINI_MMLU answers are "B" and the fake generate_fn returns "B".
Editing the fixture in forgetting.py moves the number, so say so.
Comment only, no logic change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live smoke: `soup data canary check --manifest nope.json` surfaced
"[WinError 2] Не удается найти указанный файл: 'nope.json'" -- a raw,
locale-dependent OS error leaking straight through from os.path.getsize,
where every other Soup command says "File not found: <path>".
The v0.71.36 replay smoke hit this and could not proceed: `soup merge`
writes a directory like ./denseA whose NAME carries no size marker, so
model_size_from_name fell through to its 7B default, the hardware-fit gate
predicted 16.3 GB peak VRAM (weights 14.0) and REFUSED to train a 135M
model on a 4 GB box.
That blocks `soup merge` -> train-from-merged, which is the ordinary
continual-learning flow that --replay exists to serve: to show replay
prevents forgetting you must start from a model that already knows the old
task. The gate made that impossible.
A local checkpoint knows its own size, so ask it: parse the safetensors
header (u64 length + JSON with each tensor's shape) and sum the elements.
Only the header is read -- a few KB -- never the weights. Measured
./denseA at 0.135B where the name guess said 7.0B.
Falls back to the existing name guess for hub ids and for anything
unreadable, so nothing else changes: Llama-3-8B still 8B, SmolLM2-135M
still 0.135B, whisper-tiny still 0.039B, a missing dir still 7.0.
Truncated and absurd-length headers fall back rather than crash.
This is the third instance of the same class -- v0.71.32 (whisper sizes)
and v0.71.33 (the M suffix) were both the name guess over-predicting a
small model and the gate then refusing to train it. All three were found
by a live smoke, never by a unit test.
8 new tests; 151 existing gpu/hardware-fit tests still green.
The Step-6 live smoke on SmolLM2-135M found a real defect that 268 green
tests did not: the "MAJOR iff ANY canary <= 1%" rule ignores that the
chance of SOME canary dipping low grows with the canary count.
Measured on a model that never saw the canaries: percentiles spanned
1.6%-93%, two of ten under 10% -- pure noise -- and the old rule returned
MINOR. Analytically it is worse: under the null each percentile is
~Uniform(0,1), so at the DEFAULT --count 16 the old rule fires
MAJOR ~15% of the time on a CLEAN model (1 - 0.99^16)
MINOR ~81% of the time (1 - 0.90^16)
MAJOR exits 2, so a leak detector wired into CI would have cried wolf
roughly one run in seven. That is not shippable as a gate.
The verdict now asks whether MORE canaries look memorized than chance
explains: the observed count is tested against a binomial tail and fires
only at p < 0.05. This costs no sensitivity, because a genuinely memorized
set is unmistakable -- every canary lands at percentile 0.0.
Verified live on the same two models after the fix:
clean model MINOR -> OK (exit 0)
memorized model MAJOR -> MAJOR (exit 2)
False-alarm rate on clean models, 4000 Monte-Carlo trials per K:
K=10: MAJOR 0.5% K=16: MAJOR 1.2% K=32: MAJOR 4.4% (was 15% at K=16)
A partial leak still fires: 2 of 10 memorized is P=0.4% by chance -> MAJOR.
Tests pin the exact clean-model distribution the smoke observed, the
one-of-sixteen non-fire, the partial-leak fire, and a Monte-Carlo property
test bounding the false-alarm rate under alpha.
This overrides the spec's locked "any memorized -> MAJOR" rule. The lock
existed to prevent design thrash, not to protect a rule that measurement
showed to be broken.
ECC tdd pass, run as an 18-mutant sweep across the areas the earlier
per-module harnesses had not covered (topics internals, embed validation,
semdedup guards). 16 mutants were caught; the sweep found one real vacuity,
which is the 7th of this release.
test_bad_model_id_rejected asserted only
`pytest.raises((ValueError, TypeError))` over ["", " ", None, 123, True].
That is too broad to mean anything here: with the empty-check DELETED, ""
falls straight through to the fallback refusal ("cannot verify pooling")
and still raises ValueError, so the test passed for a completely different
reason than its name claims. Proven by mutation: deleting the empty guard
left the suite green. The length cap had no test at all; the null-byte
check had none either.
Replaced with one test per guard, each asserting the message that guard
actually emits (non-empty / null byte / too long / must be str), plus a
monkeypatched _fetch_pooling_config that fails the test if the guard lets
execution reach a network fetch. Re-ran the sweep: all three guards now
CAUGHT where two previously survived.
Also added: model_id is stripped before the allowlist lookup, so
surrounding whitespace cannot defeat it.
Verified NOT vacuous (mutation-checked, no change needed): the ctfidf
top_n slice, resolve_k's cap/clamp/n<4 rules, kmeans' iteration loop,
build_topic_report's coverage denominator + gap warning, embed's row cap /
non-str / bare-string / truncation / batch bounds / L2 zero-guard /
allowlist lowercasing, semdedup's row cap / 2-D check / nearest-kept-row
provenance.
268 tests green (1 POSIX skip), ruff clean.
ECC security-review. Its core point stands: canary.py is BY ITS OWN
DOCSTRING a secret store, yet it was the one new file skipping the two
conventions this codebase applies to every other secret-bearing artifact.
Both claims verified before fixing, not assumed.
MEDIUM — manifest was world-readable. atomic_write_text does no chmod, so
under the usual 022 umask the file lands 0644 and any local user on a
shared box can read every canary without ever running `check`. 9+ existing
modules (registry/store.py, adapter_sign.py, audit_log.py, ...) chmod 0600;
canary.py now does too, via _harden_permissions. Proven by exercising the
POSIX branch directly, since the test skips on Windows.
MEDIUM — terminal escape injection. Verified: rich.markup.escape() passes a
raw ESC byte straight through (it only neutralises [...]), and load_manifest
accepted an OSC 52 sequence in a `secret`. The manifest is explicitly a
shareable artifact, so a hostile one is realistic, and the injected
sequence renders right above the MAJOR verdict it could obscure. Now
stripped via _for_terminal, mirroring data_doctor.py / shrink.py.
LOW — load_manifest had no entry cap. The 4 MB size cap alone still admits
tens of thousands of minimal entries, each costing a model forward pass in
check. Now bounded by the same _MAX_CANARIES the generators use.
Also closed a gap the review noted in passing: `check --output` embeds every
secret, so it is as sensitive as the manifest but carried no warning and no
permissions. It now gets both.
262 tests green (1 POSIX-only skip on Windows), ruff clean.
ECC code-review found a real bug and I reproduced it before fixing.
load_dataset runs _validate_vision_images / _validate_audio_files on the
primary dataset -- they exist to reject `{"image": "/etc/passwd"}` -- but
the replay file is loaded by _load_replay_rows, which did neither. A
genuinely llava-shaped replay row keeps its `image` value through
format_to_messages, and nothing gated data.replay on modality, so a
traversal path from the replay file reached PIL.Image.open in the trainer.
Verified rather than assumed: a chatml+image row is detected as chatml and
its image key is dropped (my first probe found 0 escapes, which is why the
shape matters), but a real llava row -> detect_format='llava' -> the
traversal path survives conversion intact. Both the vision and audio cases
now have failing-first tests; the vision guard is mutation-verified.
Fix: the replay file gets its own containment pass, resolving media against
the REPLAY file's directory (the old dataset's images live with the old
dataset) unless an explicit image_dir/audio_dir is configured.
Also from the same review:
- MEDIUM: `canary insert` wrote the poisoned dataset BEFORE the manifest, so
a manifest failure left canaried data on disk with nothing to identify the
secrets in it. Manifest is now written first; if the data write then
fails, the error says the manifest describes canaries that were never
inserted rather than leaving it looking authoritative.
- LOW: --threshold help said "MinHash similarity" but --semantic reuses it
for embedding cosine.
- LOW: added --replay-seed, which had no CLI override unlike its two sibling
flags.
260 new-file tests + 337 loader/vision/audio regression tests green.
Pins that the five new kernels stay importable without the [train] extra.
v0.71.0 split the deps deliberately; a stray top-level `import torch` in a
data module would silently undo that for every `soup data ...` user.
Includes a guard-the-guard test proving the AST walk really distinguishes a
top-level import from a lazy one inside a function, plus a check that the
CLI actually exposes `data topics` and `data canary`.
Full suite: 16127 passed, 126 skipped, 0 failed. Tests 16001 -> 16254.
Thin CLI overrides for data.replay / data.replay_ratio, mirroring the
--reward-hack-mitigation style.
_apply_replay_overrides rebuilds the config rather than mutating it in
place, and that IS the mechanism: rebuilding re-runs every cross-validator,
so `--replay` on task='dpo' hits _validate_replay_compat exactly as a YAML
value would. Mutating in place would let CLI flags bypass every gate --
mutation-verified: the in-place version fails 3 named tests. It also leaves
the caller's config untouched.
Declared the flags as plain str/float with a None default, matching this
file's existing convention (name/resume/annex_xi). train.py has
`from __future__ import annotations` but never imports Optional, and Typer
must resolve annotations at runtime -- Optional[str] raised NameError at
--help time, caught immediately by the help test.
All three load paths (local / remote / HF) previously derived their own
train/val split at three separate return sites. Extracted _finalize as the
single exit point and routed all three through it, so `soup sweep`,
`soup train` and `soup train --dry-run` cannot drift on replay behaviour.
103 existing loader tests still green.
Replay is mixed AFTER the split, into train ONLY -- val stays pure
new-task because it is the yardstick for the task being learned. Old-task
retention is measured externally with soup eval custom / soup ship, so no
new eval machinery lands here.
The replay file gets its OWN format detection: the old dataset may be
alpaca while the new one is sharegpt, so it cannot inherit
data_config.format. Its path is cwd-contained like every other data input.
Mutation-verified: mixing before the split leaks replay into val; letting
the replay file inherit the new file's format silently drops every row;
removing the containment check lets the path escape cwd. Each fails a
named test.
Sample a slice of an old dataset and interleave it into the new one, so
fine-tuning on a new task does not erase the previous one. Pure: no torch,
no I/O.
NAMED rehearsal.py, not replay.py: utils/replay.py already exists from
v0.34.0 (metric-history replay for `soup runs replay`, imported by
commands/runs.py + tui_app.py). The plan said "Create utils/replay.py",
which would have silently overwritten a shipped module and broken
`soup runs replay` and `soup tui`. The user-facing flag is still --replay.
Interleave, NOT concat: `new + replay` puts every replay row in one block
at the end, so the model sees them all in the final steps -- a second
mini-finetune, which is precisely the failure replay exists to prevent.
An empty replay pool is now a true no-op that returns the new rows
untouched. It previously still shuffled, so pointing --replay at an empty
file silently reordered your training data as a side effect.
All 5 behaviours mutation-verified to fail a named test: the r/(1-r) ratio
formula (the naive r*n gives 9.09% for a requested 10%), interleave-not-
concat, shortfall-reports-rather-than-upsamples (repeating rows changes
epoch semantics), seed determinism incl. seed=None meaning 0 not random,
and sampling without replacement.
data.replay / replay_ratio / replay_seed for continual-learning rehearsal,
plus the cross-validator gate. 540 existing schema tests still green.
Ratio semantics are pinned in the field description and by test: r is the
fraction of the FINAL mixed train set, so n_replay = round(r/(1-r)*n_new).
At r=0.1 over 1000 new rows that is 111 replay rows -> 1111 total -> 10.0%.
The naive r*n_new gives 9.09% and is wrong.
Gates, each mutation-verified to fail a named test:
- task in {sft, pretrain}: replay on dpo would be silently ignored
- packing / multipack rejected: both concatenate rows into fixed blocks, so
the ratio stops being meaningful at block boundaries -- reject rather than
silently mis-mix
- footgun: replay_ratio/replay_seed without data.replay silently no-op
Provenance needs no extra plumbing: data.replay* rides model_dump(), so the
experiment tracker, the registry's config_json, soup card and the repro
receipt already capture it. Per-row _replay keys would be a coin-flip bug --
sft.py computes remove_columns from dataset["train"][0], so whether the
column survives into TRL depends on whether row 0 happens to be a replay row.
insert: mix K canaries into a dataset + write the manifest. check: rank
each canary's loss against N never-inserted controls and report
OK/MINOR/MAJOR. Exit 0/2/1 mirrors soup diagnose + soup ship so CI can
gate on a leak.
The manifest is the sensitive artifact of this feature -- not the dataset.
It is cwd-contained, symlink-refused, atomically written, and the command
says out loud that it must not be committed alongside the data it
protects.
Tests cover the wiring the kernel cannot: that the rows actually inserted
carry the manifest's secrets, and that the canary/control loss split
follows the manifest length (compute_pair_losses is index-aligned, so an
off-by-one here would silently score canaries against each other).
Documented a real edge the test fixtures surfaced: an all-equal loss
fixture makes NO control strictly cheaper -> percentile 0.0 -> MAJOR. That
is the rule working correctly on unrealistic input, and ties are the safe
direction for a leak detector, so the docstring now warns against
"fixing" it with <=, which would let a memorized canary tying the cheapest
control read as typical. The fixtures were made realistic instead.
Generation, manifest I/O, and the exposure math that decides whether a
model memorized an inserted secret. Pure: the caller supplies the losses,
so the whole decision is testable on CPU with hand-written numbers.
Measurement is loss-vs-controls (Carlini et al., "The Secret Sharer"), not
greedy regurgitation: a model can memorize a canary and still not emit it
under greedy decoding, so "nothing came back" would be false reassurance --
the precise failure a leak-detection feature must not have. Controls are
drawn from the identical secret space AND share the carrier prompt, so a
loss gap measures the secret rather than the prompt.
The rule (single source of truth, mirroring ship_verdict.py):
percentile = |{control : loss(control) < loss(canary)}| / n_controls
memorized = percentile <= 0.01
MAJOR = any memorized | MINOR = any <= 0.10 | OK = otherwise
Edge cases that would each be a silent false-negative, all pinned:
zero (or all-NaN) controls REFUSE rather than return OK -- reporting OK
against nothing is the exact false reassurance this exists to prevent;
ties count as not-strictly-less; a NaN loss reports unknown and serializes
to null, never 0.0, which would read as the strongest possible leak.
Fixtures use INTEGER control losses so cheaper/n_controls is exact -- the
semdedup boundary test in this same release was vacuous because float32
never landed on the threshold. All 5 decision-rule mutations (both
boundaries, ties, the refusal, the NaN path) are verified to fail a named
test.
Canary exposure ranks ONE canary's loss against a control distribution, so
it needs per-item losses. compute_eval_loss returns only the mean and
silently compacts the list -- a skipped pair shifts every later index, so
it cannot be used for that.
compute_pair_losses returns one loss per input pair, index-aligned, with
nan for unusable pairs (empty target span) and nan losses. compute_eval_loss
becomes a thin mean over the non-nan entries: behaviour is unchanged and
all 60 existing consumer tests (live_eval / interference / tunability) stay
green.
Mutation-verified: reverting to the old "continue" (drop) behaviour makes
test_index_aligned_with_nan_for_skipped fail, so the alignment guarantee is
genuinely pinned.
Typer/Rich layer over the pure topics kernel: embed -> k-means ->
c-TF-IDF labels -> a coverage table ("82% code, 6% math") plus a gap
warning for thin clusters. New command file rather than growing
commands/data.py, which is already 3300+ lines; registered in cli.py
mirroring data_doctor.
Honest framing is in the help text, the table footer and the docstring:
labels are emergent term clusters, NOT a classification against a fixed
taxonomy, and there is no join to `soup eval coverage`.
Dataset-derived text is escape()d before it reaches the terminal, so a
crafted row cannot inject Rich markup into the table.
Caught while wiring: atomic_write_text takes (text, output_path) -- the
plan had the arguments reversed.
BERTopic-lite over embedding vectors: pure numpy, no torch, no I/O, so the
clustering and labelling decisions are testable on CPU with hand-built
input. Labels are emergent unsupervised term clusters, NOT a
classification against an ontology, and there is no join to
`soup eval coverage` (that compares an eval suite's scorer mix to a task
taxonomy -- a different axis). Docs say so plainly.
The mutation checks caught TWO more vacuous tests, both of my own planning:
1. c-TF-IDF: the fixture made the specific term MORE frequent in-cluster
than the filler ("python" 3x vs "the" 2x), so plain per-cluster TF
already picked it and deleting the idf term changed nothing -- 8/8
passed against a broken implementation. Rebuilt on 5 clusters of
["the","the","term_i"] where the filler dominates on raw frequency and
ONLY the inverse-cluster-frequency can demote it. The margin is
arithmetic, not luck: idf ratio 2.58 vs tf ratio 2.0.
2. kmeans determinism: two well-separated blobs converge to the same
partition from ANY init, so an unseeded RNG passed the same-seed
equality check (6 seeds -> 2 partitions). Rebuilt on unstructured points
where init decides (6 seeds -> 6 partitions).
Both now fail correctly when mutated. Each fixture has a companion
"guard the guard" test pinning the property that makes it discriminating,
so neither can silently drift back into vacuity.
Every "you're missing an extra, install it" hint rendered as
`pip install 'soup-cli'` -- Rich parsed [eval]/[onnx]/[serve]/[wandb]/...
as a markup tag and dropped it. The suggested command therefore installs
the base package WITHOUT the extra the user is missing, so following the
hint appears to succeed and the feature still fails. Pre-existing across
many releases. Same class as the v0.71.28 \[mcp] fix, which only fixed its
own call site.
17 sites escaped across 9 command modules. Typer help is affected too --
rich_markup_mode="rich" renders help through Rich, so `--semantic` shipped
as "Requires soup-cli." and --track-energy lost its [carbon] hint.
Deliberately NOT escaped: `raise ImportError("... 'soup-cli[mlx]'")`,
errors.append(...), docstrings and YAML samples never reach Rich, so a
backslash there would surface literally. An initial blanket sweep hit 48
sites; ~31 were of that kind and were reverted. If such an exception text
is ever printed via console.print, the fix belongs at the print site
(rich.markup.escape), not in the message.
Tests: pin the Rich behaviour both ways (eaten unescaped / survives
escaped), scan every markup-bearing hint line, render both Typer helps
end-to-end, and assert the non-Rich counter-case stays unescaped.
Adds an embedding-cosine backend to the shipped dedup command. MinHash
remains the default and its behaviour is unchanged (45 pre-existing dedup
tests still green).
Restructured dedup() so the datasketch import moves INTO the MinHash
branch. It previously sat at the top of the function, ahead of even the
file-exists check, so --semantic would have died with "datasketch not
installed" for a user who has [train] but not [data] -- a dependency the
semantic path never uses. Output resolution + cwd containment are now
resolved once, before the branch, and shared by both backends.
Also fixes a REAL user-facing Rich-markup bug in the two hints this
function prints: "pip install 'soup-cli[train]'" rendered as "pip install
'soup-cli'" because Rich ate [train] as a markup tag -- i.e. the suggested
command installs the package WITHOUT the extra the user is missing, so
following it appears to succeed and the feature still fails. Same class as
the v0.71.28 \[mcp] fix. 18 further unescaped sites survive elsewhere in
the codebase; swept separately.
Pure numpy over already-embedded L2-normalized vectors, so the whole
selection decision is testable on CPU with hand-built vectors and no
model. DedupReport.pairs records (dropped_idx, kept_idx, cosine) so a
drop names the nearest kept row that caused it -- that provenance is what
makes the upcoming MinHash-vs-semantic comparison auditable.
The planned boundary test was VACUOUS and the mutation check caught it:
arccos(0.8) -> cos() in float32 yields 0.800000011920929, strictly above
the threshold, so `>=` and `>` behaved identically and the boundary was
never exercised (18 passed against a deliberately broken `>`). Rebuilt the
fixture on a dyadic cosine that is exactly 0.5 in float32 ([1,0] vs
[0.5, sqrt(3)/2], still unit-norm) -- the mutation now fails as it must.
test_boundary_fixture_is_exact guards the fixture itself from drifting
back into vacuity.
Completes the embedding kernel started by the pooling gate. embed_texts
lazily imports torch/transformers/numpy so the module stays importable on
the light core (pip install soup-cli without the [train] extra).
_mean_pool masks padding before averaging: an unmasked hidden.mean(dim=1)
averages pad positions too and silently drags every vector toward the pad
embedding. Pinned by a mutation-verified test (padded 999.0 -> unmasked
mean ~334 vs the correct 2.0).
resolve_pooling gates embed_texts before any download, so an unverified
model is refused rather than fetched and mis-pooled.
Renamed fixture model ids in test_cls_pooling_config_is_refused and
test_refusal_names_what_it_found so the assertion words (cls/max) can
only come from the _NON_MEAN_POOLING_KEYS label, not from the model id
string echoed back by the generic fallback message. Verified by
mutation: deleting the _NON_MEAN_POOLING_KEYS loop now fails these
tests instead of passing by coincidence.
Added test_cls_precedence_over_contradictory_mean_flag pinning that
cls/max/etc. are checked before pooling_mode_mean_tokens, so a
self-contradictory config is still refused.
Made test_allowlisted_model_short_circuits and
test_allowlist_is_case_insensitive hermetic by monkeypatching
_fetch_pooling_config to raise if called, proving the allowlist
short-circuit never reaches the network.
Alpamys
Alpamys
Sanjay Santhanam