The frozen base lives in CPU RAM and is streamed into a small pool of
pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is
bounded by ONE layer instead of the whole model. Only the LoRA adapters,
their gradients and optimizer state stay resident.
Measured on an RTX 3050 Laptop 4 GB (Windows, 16.9 GB RAM), batch 1,
gradient checkpointing on, 50 steps after 10 warm-up:
Qwen2.5-0.5B S=512 978.6 tok/s 91.4% util 1.47 GB peak
Qwen2.5-1.5B S=512 525.0 tok/s 96.8% util 1.82 GB peak
Qwen2.5-1.5B S=1024 487.6 tok/s 96.7% util 2.96 GB peak
Qwen2.5-3B S=512 143.1 tok/s 79.3% util 2.15 GB peak
Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs.
Honest cost: 1.43x slower than resident, measured at 0.5B — the only
apples-to-apples comparison available on this box, because 1.5B and above
cannot run resident here at all.
Correctness was gated before any src/ code was written: streamed vs
resident logits are bit-exact (max abs diff 0.0), the layer-0 LoRA
gradient is non-zero on all layers, a 100-step loss curve matches
resident exactly, and same-seed runs are identical.
New:
- utils/layer_stream.py pure planner (no top-level torch)
- utils/layer_shard.py per-layer safetensors sharder
- utils/layer_stream_runtime.py buffer pool, RAM source, prefetch, wrapper
- training.stream_layers / stream_source / stream_buffers
Notes for future maintainers:
- transformers' Trainer.__init__ and accelerate's prepare_model BOTH call
model.to(), which raises NotImplementedError on meta parameters. The
streamed layer overrides _apply to pass meta tensors through, and the
model declares hf_device_map. Without either, every run dies at trainer
construction — no unit test that stops at model(input_ids=...) sees it.
- The shard cache is keyed to a fingerprint of the source checkpoint, not
just the model slug: a base retrained in place must re-shard rather than
silently stream stale weights.
- The pre-flight hardware-fit gate models a RESIDENT run, so it is skipped
for streaming — otherwise it refuses exactly the runs this enables.
- expandable_segments:True is silently ignored on Windows; probed, not
claimed.
Scope (every refusal names the release that lifts it): RAM tier, bf16,
task=sft, Llama/Qwen, batch 1, no gradient accumulation, no --resume.
NF4 is v0.72.1; disk tier / bigger batches / accumulation / resume are
v0.72.2. Proof-of-mechanism at 3B — nothing above 3B was measured.
Tests: 16576 -> 16735 (+159 in tests/test_v07200.py)
Add utils/reward_synth to the CONTRIBUTING module list, fix the stale "Test
Files (313 -> 318)" count, and add reward-verifier synthesis to the training
docs-index row. Docs-only; no version bump.
Ship a regulated fine-tune with the paperwork it needs, plus GGUF export that
actually works on Windows.
Added:
* `soup init --template hipaa|soc2|eu-ai-act|sr-11-7` (templates 17 -> 21).
Design constraint verified in-repo, not assumed: Soup's compliance controls
are CLI flags/commands, NOT schema keys (audit_log/bom/attest/repro_receipt/
annex_xi/track_energy/pii/decontaminate have zero matches in config/schema.py),
so a template cannot "pre-wire audit-log on" as YAML. Each is a valid
SoupConfig + header comments naming that regime's exact commands.
* `soup card <registry-id> -o MODELCARD.md` — registry entry -> publishable HF
card (training config, eval scorecard, config/data hashes, lineage, artifacts).
Pure build_model_card() over dicts, reused by `soup push --card`.
* `soup ci init` — writes .github/workflows/soup-gate.yml chaining
data validate -> expect -> ship --evidence (exit 2 blocks the merge).
* docs/compliance.md quickstart.
Fixed (GGUF-on-Windows, validated end-to-end for the first time; closes the
CPU-validatable half of #70/#144). Four independently-fatal bugs:
* export cloned llama.cpp into the CURRENT directory: SOUP_DIR is the bare name
".soup" but was used relatively instead of anchored to home like tracker.py /
registry/store.py, so ~/.soup/llama.cpp was never found.
* the first GGUF export DOWNGRADED the user's torch and broke CUDA: the
auto-clone pip-installed llama.cpp's requirements.txt (pins torch~=2.2.1 from
the CPU index) into the user's interpreter. Observed live: torch 2.5.1+cu ->
2.2.2+cpu, transformers 4.57 -> 4.46. Now installs only gguf/sentencepiece/
protobuf, unpinned, non-fatally.
* a correctly-built llama.cpp was not found on Windows: MSVC (like Xcode) is a
multi-config generator emitting build/bin/Release/llama-quantize.exe.
* `soup deploy ollama` failed on a relative GGUF path ("pull model manifest:
file does not exist") — ollama resolves FROM against the Modelfile's dir and
Soup writes it to a temp dir. Modelfile now emits an absolute path.
Also fixed a pre-existing model-card injection hole (affects `soup push`'s own
auto-card): _render_training_section interpolated base/task/scheduler/recipe
unescaped, and SoupConfig.base/scheduler have no charset validator, so a
crafted-but-valid config could smuggle raw HTML or a code-span-breaking backtick
into a card published to the Hub.
Step-6 live smoke (real train -> registry push -> card) caught a bug 90 green
tests missed: is_adapter came only from registry artifacts, so a real LoRA run
with no artifacts rendered "Full model" + library_name: transformers — a false
claim in a provenance document.
5 sequential reviews, every finding fixed (code HIGH: the generated workflow ran
`pip install -e ".[dev]"`, which only works in the Soup source tree, breaking the
gate's first step for every downstream user; security HIGH/MEDIUM/3 LOW; tdd HIGH:
push --card was only --help-tested). All 5 new fixes mutation-verified as real
pins. Tests 15906 -> 16001 (+95); full suite 15872 passed / 126 skipped.
The README had grown to 5046 lines (195 sections) — roughly one deep-dive per
feature accreted over 70 releases. Split it into a concise front door plus a
public docs/ tree:
- README (5046 -> 238 lines): hero, why, quickstart, config, a Documentation
map, data formats, common commands, models, Docker, requirements, dev.
- docs/*.md: all 185 feature sections preserved verbatim, grouped into 10 themed
guides + an index. Every original line is accounted for (content-conservation
checked); all 235 internal links + anchors verified to resolve.
- un-gitignore docs/ (it was empty); fix a pre-existing dangling
docs/QUANTIZATION.md link; correct the stale `ruff check soup_cli/` ->
`src/soup_cli/` reference in the Development section.
No version bump: docs-only — rides into the 0.71.0 deps-split release.
Alpamys