The layer-streaming pre-flight printed a Rich panel titled `soup train
--stream-layers`. There is no such flag — `soup train --help` contains zero
occurrences of "stream" — and streaming is enabled only by `training.stream_layers`
in soup.yaml. The panel is the first thing a streaming run prints, so it was the
feature's most-read line of documentation, and it pointed at a `No such option`.
Wording left over from the original design note, which proposed a CLI flag; the
feature shipped as a config key and the title never followed. Nothing asserted on
it, so nothing caught the drift.
Retitled to `training.stream_layers`. The body already opens with "Layer streaming
BETA", so the title does not repeat it. Same string fixed in the three module
docstrings and in one test docstring that quoted it.
Cosmetic only — no behaviour change. Verified by rendering the panel, and by
tests/test_v07200-04.py: 440 passed, 2 skipped. ruff clean.
benchmarks/gate-v0.72.*.md carry the phrase too and are deliberately untouched:
they are published verbatim gate records, kept as written.
Closes#329
Two NF4 bit-exactness tests failed on all nine CI jobs while passing locally.
Root cause reproduced directly: bitsandbytes' CPU 4-bit forward calls
`_convert_weight_packed_for_cpu`, which reshapes absmax to
[rows, blocks_per_row]. At hidden_size 32 a weight has 32*32/64 = 16 absmax
blocks for 32 rows, so blocks_per_row floors to ZERO and it raises
"shape '[32, 0]' is invalid for input of size 16" — the exact CI error.
Verified 32 raises while 64 and 128 do not.
A CUDA build never calls that function, which is why a GPU development box
cannot see this and every CPU-only runner fails. The fixtures move to
hidden_size 64 with the reason recorded in-place so they are not shrunk back.
Fixture sizing only: nothing about the product changed, and both suites still
assert bit-exactness.
Lifts the v0.72.0-.2 scope freeze. Every capability was gated against a
streamed-vs-resident bit-exactness reference before it was written.
- Six more families (mistral/gemma/gemma2/gemma3_text/phi/phi3), each
bit-exact vs the same checkpoint loaded resident, under bf16 AND NF4.
Multimodal gemma3 stays refused — only gemma3_text.
- batch_size > 1 and gradient_accumulation_steps > 1 now work.
- A batch- and vocab-aware VRAM pre-flight that refuses a run predicted
not to fit. Fitted to 10 real runs: worst error 0.85%, never
under-predicts. On Windows an over-budget step does not OOM; WDDM
spills silently, so the estimator is the only guard.
- A throughput bracket from a GEMM ceiling measured on the user's own
card in the same session, printed with the SM clock.
- --resume / --hf-resume: load_state_dict narrows keys by child name, so
a canonical checkpoint matched 0 of N tensors and PEFT warned only.
Keys are now redirected at load time, mirroring the v0.72.1 save fix.
- An NVMe disk overflow tier (stream_source: auto|ram|disk), bit-exact
against the RAM tier. Its speed relative to RAM is UNMEASURED here and
no figure is claimed.
- soup doctor --disk reports the detected media type.
Fixes: estimate_logits_bytes charged 6 bytes/element where the measured
peak is 14; the NVMe tier guard was wired to a hardcoded constant;
streaming sources leaked handles when training raised; subprocess
helpers resolved tools by bare name (CWE-427 on Windows).
112 tests in tests/test_v07203.py; 16867 -> 16977.
CI was red on macOS only (3/11 jobs); ubuntu and windows were green across
3.10/3.11/3.12, as were lint and type-check.
Cause is not bitsandbytes availability but device disagreement: on an
Apple-Silicon runner with no CUDA, TrainingArguments picks `mps`, while this
suite builds the streamed model on `cpu`. The batch is then moved to MPS and
the step raises "Placeholder storage has not been allocated on MPS device!".
Only the two tests that actually call trainer.train() were affected;
test_setup_builds_a_real_trl_trainer_under_nf4 passed, because building the
trainer never touches a device.
v0.72.0 hit exactly this and guards test_one_training_step_actually_runs the
same way; this mirrors that helper rather than inventing a second one. NF4
streaming is measured on CUDA and CPU only, and bitsandbytes' 4-bit kernels
have no MPS support, so skipping is the honest outcome — not a claim that it
works there.
Verified on the CUDA dev box: 88 passed, zero skipped, i.e. the guard does not
over-skip where the tests are meaningful.
Layer streaming (v0.72.0) was bf16-only, capping it near 3B on a small card.
Quantising the streamed base to NF4 makes the RAM store ~4x smaller, which is
what brings 8B within reach.
Measured on a 4 GB RTX 3050 Laptop through the shipped code (50 steps after 10
warm-up, batch 1, S=512, PagedAdamW8bit, GEMM ceiling taken in the same session):
Llama-3.1-8B-Instruct 119.6 tok/s 3.32 GB peak 3.60 GB pinned 100% 952 MHz
Qwen2.5-3B 264.2 tok/s 1.76 GB peak 1.43 GB pinned 100% 960 MHz
3B is 1.85x the bf16 path, but that is PINNING, not arithmetic: 1.43 GB
page-locks where 5.55 GB did not, restoring async copy_ (util 79.3% -> 100%).
14B was not run — its store exceeds this box's measured 7.12 GB pinned ceiling.
A streamed NF4 run is bit-exact against a RESIDENT NF4 run, now as CPU-runnable
CI tests rather than only a gate result.
Notable, because each fails silently:
- PEFT dispatches lora.bnb.Linear4bit only when is_loaded_in_4bit is stamped;
without it the generic lora.layer.Linear runs against a Linear4bit base and
casts differently (9.375e-01 logit divergence, no warning). Pinned by a test
with a control that deletes the marker.
- hf_quantizer must be stamped too, or Trainer.__init__ dies formatting its own
"cannot fine-tune" error. Found by the end-to-end test.
- The shard cache is keyed on quant/double_quant/quant_device as well as dtype
and source fingerprint; a bf16 cache reused for an NF4 request would feed
full-precision bytes to matmul_4bit.
- index.json is a trust boundary: its shape/blocksize reach bnb kernels that do
not bounds-check, so from_json validates and the runtime cross-checks the
claim against the bytes on disk.
- A streamed NF4 model over-reported parameters ~6.5x (878,154,048 vs
134,515,008 for SmolLM2-135M). Display-only; ~52 B at 8B.
Scope unchanged and still BETA: RAM tier, sft, Llama/Qwen, batch 1, no
accumulation, no resume. quantization values other than none/4bit are refused.
Tests: 16752 -> 16840 (+88 in tests/test_v07202.py).
Full suite: 16734 passed, 129 skipped.
Alpamys
Alpamys