From d73eb1c843ba2cae732be8177284d472ac943e40 Mon Sep 17 00:00:00 2001 From: Alpamys Date: Fri, 31 Jul 2026 15:54:14 +0500 Subject: [PATCH] docs(benchmarks): publish the layer-streaming measurement records Publishes the three gate records behind the layer-streaming feature as benchmarks/, so the numbers in the docs and in the accompanying paper can be checked rather than taken on trust. These are the working records written while each item was built, not a report assembled afterwards: they carry the failures, the assumptions that turned out wrong and the numbers that were measured and then discarded, in the order those things happened. - gate-v0.72.0-layer-streaming.md - the streaming path and its bit-exactness reference - gate-v0.72.2-nf4.md - NF4 quantised streaming; Llama-3.1-8B at 119.6 tok/s in a 3.32 GB peak - gate-v0.72.3-breadth.md - nine architectures, batching, accumulation, resume, disk tier; peak-VRAM predictor at 0.85% worst-case error The README states the hardware once, explains why Windows/WDDM spilling rather than raising OOM means a completed run is not evidence of fit, and why any fraction-of-ceiling is quoted only with the SM clock it was taken at. References to internal planning documents were rewritten for a reader outside the repository. --- benchmarks/README.md | 52 +++ benchmarks/gate-v0.72.0-layer-streaming.md | 233 ++++++++++ benchmarks/gate-v0.72.2-nf4.md | 339 +++++++++++++++ benchmarks/gate-v0.72.3-breadth.md | 472 +++++++++++++++++++++ 4 files changed, 1096 insertions(+) create mode 100644 benchmarks/README.md create mode 100644 benchmarks/gate-v0.72.0-layer-streaming.md create mode 100644 benchmarks/gate-v0.72.2-nf4.md create mode 100644 benchmarks/gate-v0.72.3-breadth.md diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..4af1641 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,52 @@ +# Measurement records + +Raw gate records for Soup's layer-streaming feature, published as written. + +These are not a report assembled after the fact. They are the working records +kept while each item was built and verified, so they contain the failures, the +assumptions that turned out wrong, and the numbers that were measured and then +discarded — in the order those things happened. They are the evidence behind +the paper *Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB +Laptop GPU*. + +| File | What it gates | Headline | +|---|---|---| +| [`gate-v0.72.0-layer-streaming.md`](gate-v0.72.0-layer-streaming.md) | The streaming path itself | Bit-exactness vs a resident reference; 3B bf16 trained on a 4 GB card | +| [`gate-v0.72.2-nf4.md`](gate-v0.72.2-nf4.md) | NF4 quantised streaming | Llama-3.1-8B at 119.6 tok/s in a 3.32 GB peak | +| [`gate-v0.72.3-breadth.md`](gate-v0.72.3-breadth.md) | Nine architectures, batching, accumulation, resume, disk tier | Peak-VRAM predictor at 0.85% worst-case error; accumulation is per-token I/O-neutral | + +## Hardware + +Every number was measured on one machine: + +- **GPU** — RTX 3050 Laptop, 4 GB (4.29 GB usable) +- **Host** — 16.9 GB RAM, NVMe +- **OS** — Windows 11 + +Windows/WDDM matters for reading these: it spills into shared host memory rather +than raising `CUDA out of memory`, so a run completing is not evidence that its +configuration fits. That is why peak VRAM is reported alongside every throughput +figure, and why the fit decision refuses rather than warns. + +## Reading the numbers + +- **Throughput is quoted with the SM clock it was taken at.** This card's boost + clock varies about 13% between sessions, so a fraction-of-ceiling stated + without its clock is not meaningful. Where a GEMM ceiling is compared against, + it was measured in the same session. +- **The correctness reference always matches the numerics under test** — a + streamed NF4 run is compared against a *resident NF4* run, never against + resident bf16, which would hide a real defect inside quantisation error. +- **Derived figures are labelled as arithmetic.** Where a line says "1M tokens = + 2.3 h", that is division, not a measured wall-clock run. + +## Reproducing + +The implementation ships in Soup under Apache-2.0. Reproduction commands are in +Appendix A of the paper; the correctness protocol runs as part of the project's +test suite, so a regression in bit-exactness fails CI rather than reaching a +user. + +```bash +pip install 'soup-cli[train]' +``` diff --git a/benchmarks/gate-v0.72.0-layer-streaming.md b/benchmarks/gate-v0.72.0-layer-streaming.md new file mode 100644 index 0000000..f96ffcc --- /dev/null +++ b/benchmarks/gate-v0.72.0-layer-streaming.md @@ -0,0 +1,233 @@ + + +# v0.72.0 Layer Streaming — Gate results + +**Status: both gates PASS. v0.72.0 shipped 2026-07-26 (tag `v0.72.0`, PyPI live). +Numbers below are final and were reproduced verbatim in the release notes, CHANGELOG and +the accompanying report. §"After the gates" records what implementation +and review then found — the gates were necessary but not sufficient.** + +Hardware (all numbers on this box, no extrapolation): +**Windows 11 · RTX 3050 Laptop 4 GB (CC 8.6, driver 591.44) · 16.9 GB RAM · NVMe · PCIe** +torch 2.5.1+cu121 · transformers (llama modeling) · peft 0.18.1 · accelerate 1.12.0 · Python 3.10 + +Unit convention: **decimal GB throughout** (16.9 GB host RAM, 7.65 GB page-locked ceiling). +Earlier drafts mixed GiB and GB for the same machine — 15.7 GiB and 16.9 GB are the same RAM, +7.12 GiB and 7.65 GB the same pinned ceiling. + +--- + +## GATE 1 — correctness — **PASS** (2026-07-26) + +Throwaway spike, nothing under `src/`. Model **HuggingFaceTB/SmolLM2-135M** +(ungated; 30 layers, d=576, ffn=1536, vocab=49152, tied embeddings), bf16, +LoRA r=16 on `q_proj,v_proj` (921,600 trainable), seq 512, batch 1, +**double buffering (n=2) + dedicated prefetch stream + events from the first +prototype** (plan §2.4). + +Construction actually under test: +- checkpoint sharded to `layer_NNN.safetensors` + `extras.safetensors` +- `RamSource` — whole base pinned in CPU RAM (212.4 MB), Tier 1 +- `LayerBufferPool(n=2)` pre-allocated VRAM buffers + `torch.cuda.Event` +- skeleton built with `accelerate.init_empty_weights` — **270 decoder-layer + weight tensors (30 layers x 9) never leave `meta`**; only embed / final norm / + LoRA are materialised. The resident load never happens (plan P14). +- `torch.func.functional_call` feeds pooled buffers into the unmodified HF + layer (plan §5.3 / P13) +- `torch.utils.checkpoint(..., use_reentrant=False)`; base weights carry + `requires_grad=False` — no `detach()`, no `no_grad()` (plan P2) + +| # | Check | Threshold | Measured | Verdict | +|---|---|---|---|---| +| 1 | streamed vs resident logits, max abs diff | < 1e-3 | **0.0 (bit-exact)** | PASS | +| 2 | LoRA grad on **layer 0** | != 0 | **9.50e-01**; 30/30 layers non-zero | PASS | +| 3 | 100 steps streamed vs resident, loss curve | within noise | **max rel 0.0**; 0.61296 -> 0.22499 on both | PASS | +| 4 | same seed twice -> identical loss | identical | **max diff 0.0** | PASS | +| 5 | + pinned-CPU boundary offload (`saved_tensors_hooks`) vs resident | < 2% rel | 4.83e-3 rel | PASS | +| 6 | buffer-count invariance n=2 vs n=3 | identical | **max diff 0.0** | PASS | + +Peak VRAM during the run: 2.09 GB — but that figure includes the **resident +reference model held simultaneously** for comparison, so it is not the +streaming footprint. 5788 layer loads served from the pool. + +Notes: +- Checks 1/3/4/6 are **bit-exact**, which is the expected result: streaming + substitutes the identical weight bytes into the identical kernels. Anything + other than 0.0 would have meant a real numerical difference. +- Check 5 is the only non-zero: 0.48% relative over 25 steps. Verified **not a + race** — the offload path is bit-reproducible across two same-seed runs + (`offload_determinism_max_diff = 0.0`), so it is a different-but-valid bf16 + rounding path amplified by 25 optimizer steps. Async pinned activation + offload is **out of v0.72.0 scope** regardless; recorded for v0.72.2. +- The `owner[slot]` early-overwrite assert (plan P1 tripwire) never fired. +- Two real bugs the gate caught, both of which would have silently broken a + naive implementation: + 1. PEFT's `LoraModel` calls `self.model.forward(...)` **directly**, bypassing + `__call__` — so a prefetch pre-hook on the CausalLM wrapper never fires. + The hook must go on the module that owns `.layers`. + 2. PEFT initialises adapters on the base layer's device, which is `meta` + here; adapter weights must be explicitly re-materialised (A: kaiming, B: 0). + +--- + +## GATE 2 — the number — **> 50 tok/s row of the §8 table** (2026-07-26) + +Common config: bf16 · `task=sft`-shaped loop · LoRA r=16 on `q_proj,v_proj` · +batch 1 · gradient checkpointing ON (so C=6) · `bitsandbytes.PagedAdamW8bit` · +**double buffering n=2 + dedicated prefetch stream** · 50 measured steps after +10 warm-up · GPU util from `nvidia-smi dmon -s u` (`sm` column). + +| Model | RAM store | S | tok/s | GPU util (mean/max) | Peak VRAM | RAM store size | TFLOPS eff | +|---|---|---|---|---|---|---|---| +| Qwen2.5-0.5B | pinned | 512 | **978.6** | 91.4% / 100% | 1.47 GB | 0.72 GB | 2.90 | +| Qwen2.5-1.5B | pinned | 512 | **525.0** | 96.8% / 100% | 1.82 GB | 2.62 GB | 4.86 | +| Qwen2.5-1.5B | pinned | 1024 | **487.6** | 96.7% / 100% | 2.96 GB | 2.62 GB | 4.52 | +| Qwen2.5-3B | **pageable** | 512 | **143.1** | 79.3% / 100% | 2.15 GB | 5.55 GB | 2.65 | + +Baselines / premise: +- **Qwen2.5-3B resident training OOMs on this card** — `CUDA out of memory` + on a plain forward+backward at S=512. This is the premise of the feature and + it is confirmed, not assumed. Streaming runs the same model in **2.15 GB**. +- **Qwen2.5-0.5B resident is a VALID baseline** (peak 3.23 GB, genuinely inside + 4 GB): resident 1398.0 tok/s vs streamed 978.6 -> **streaming costs 1.43x**. + This is the only honest streaming-vs-resident overhead figure here. +- **Qwen2.5-1.5B resident is NOT a valid baseline** and its number is discarded: + it reported a 6.15 GB "peak" on a 4 GB card, i.e. it spilled into WDDM shared + host memory and crawled at 91.3 tok/s. Quoting "streaming is 5.7x faster than + resident" from that would be a garbage claim. + +TFLOPS back-check (plan §3): **the plan's assumed 9 TFLOPS peak for this card is +wrong**, so the naive back-check mislabels a good number as "IMPLAUSIBLE". +Measured on this box: achievable dense bf16 GEMM = **6.76 TFLOPS** (fp32/TF32 = +3.38, the expected 2x Ampere ratio); GA107 theoretical dense bf16 tensor peak +~24.5 TFLOPS. The 1.5B run's 4.86 TFLOPS is 20% of theoretical — plausible, and +corroborated by 96.8% GPU utilisation. + +> **RE-MEASURED 2026-07-27 (v0.72.1 session) — 6.76 is CORRECT; the ceiling tracks GPU CLOCK.** +> Re-measuring the same square shapes gave 7.08/7.52/7.66 in one session and 6.23/6.63/6.75 in +> another. Repeating 4096^3 six times *within* a session is stable to <1% (6.93-6.94) at a pinned +> 862 MHz / 63 C. So the ~13% spread is between-session boost-clock state, not measurement error, +> and **6.76 reproduces almost exactly in a low-clock session**. Rule: a ceiling is only comparable +> to a throughput measured in the SAME session, and any fraction-of-ceiling must state the SM clock. +> Shape-matched weighted ceilings at 862 MHz: 8B 7.58 (high-clock session) / 3B 6.67 / 1.5B 6.39 / +> 0.5B 5.90. Numerator convention: decoder C=6 (NOT 8 — the frozen base has no dL/dW), +> embed_tokens 0 FLOPs (lookup), lm_head C=4 (outside the checkpointed layers, frozen). +> See `gate-v0.72.2-nf4.md`. + Every other run is under 32% even of the +plan's understated 9. + +**Reading of the §8 decision table: the ">50 tok/s" row — "Mechanism is healthy. +Ship v0.72.0 BETA, go straight to v0.72.1 (NF4)."** 143 tok/s at 3B, 525 at +1.5B, 979 at 0.5B; GPU utilisation 79–97% everywhere, nowhere near the <30% +scheduler-broken or <50% hardware-bound pathologies. + +**Corrected 2026-07-27 — this paragraph originally read "IO is fully hidden".** +That is true of the 1.5B row and NOT of the 3B headline row, and the step-level +arithmetic (from these same numbers) says so: at 143.1 tok/s and a 5.55 GB store, +a step is 3.578 s and moves 2 x 5.55 = 11.10 GB, i.e. an implied 3.1 GB/s, while +compute at the measured 6.76 TFLOPS ceiling accounts for only ~1.26 s — about 35% +of the step. The 3B run is TRANSFER-bound; its pageable store makes +`copy_(non_blocking=True)` synchronous, which is the same fact as the 79.3% util. +Per-row compute fraction: 0.5B 31% / 1.5B 61% / 3B 35%. Only the 1.5B row (pinned, +5.4 GB/s implied, 96.8% util) is compute-bound as plan §8 predicted. + +`nvidia-smi dmon -s u` reports SM occupancy — "a kernel was resident" — which is +necessary but NOT sufficient for the transfer to be hidden. Reading it as proof of +overlap is what produced the original wrong sentence. + +### Honesty caveats on these numbers +1. **The 3B run used a PAGEABLE store, not a pinned one, so 143 tok/s is a + LOWER BOUND.** This box cannot page-lock 5.55 GB alongside an IDE + agent + session: measured maximum pinned host allocation is **7.65 GB** (with 9.1 GB + "available"), and the 3B attempt failed with a host-side + `CUDA error: out of memory` at 9.3 GB available even after the loader was + rewritten to remove its transient copies. Pageable memory makes + `copy_(non_blocking=True)` synchronous, which is visible as the utilisation + drop from 96.8% (pinned, 1.5B) to 79.3% (pageable, 3B). This is exactly the + RAM ceiling plan P16 predicted. +2. Numbers are Windows/WDDM (plan P15) and therefore systematically pessimistic + versus Linux. +3. The plan's §4.3 ceiling table for this card is built on the wrong peak + figure; do not quote its per-model tok/s. +4. Nothing here was measured above 3B. **No 8B/14B claim is supported.** + +Derived, clearly labelled as arithmetic and not measurement: +1M training tokens at the measured rates = **1.9 h at 3B**, 0.53 h at 1.5B. + +### Implementation findings the gates produced (feed into Phase C) +- The prefetch pre-hook must be attached to the module that **owns `.layers`**, + not the CausalLM wrapper: PEFT's `LoraModel.forward` calls + `self.model.forward(...)` directly and bypasses `__call__` hooks. +- The streamed layer wrapper must be **attribute-transparent** (`__getattr__` + delegating to the wrapped layer): this `transformers` reads + `decoder_layer.attention_type` straight off the layer object. +- PEFT initialises adapter weights on the base layer's device, which is `meta` + in a streaming build; adapters must be explicitly re-materialised. +- The RAM store must be **allocated once and filled by `copy_`**, not + `load_file -> .to() -> .pin_memory()`; the latter's 3 transient copies per + layer are what push a large base over the page-locked ceiling. +- `expandable_segments:True` (plan P7) is **not supported on Windows** — torch + warns and ignores it. Do not rely on it here. + +--- + +## After the gates — what a passing gate did NOT catch (2026-07-26/27) + +Recorded because the honest lesson of this release is that **6/6 correctness checks and a +clean throughput table still left a bug that would have killed every real run.** The gates +tested the streaming mechanism in isolation; they did not test it inside a real trainer. + +1. **CRITICAL — every streaming run would have died at trainer construction.** + `transformers.Trainer.__init__` calls `_move_model_to_device` -> `model.to()`, and `.to()` + on a module holding `meta` parameters raises `NotImplementedError: Cannot copy out of meta + tensor`. The failure lands immediately after the pre-flight prints "Layer streaming ready", + so the run looks healthy right up to the crash. 145 green tests missed it because **no test + built a real `Trainer`/`SFTTrainer` from the streamed model** — the spike drove its own loop. + Fixed with `hf_device_map` + a `StreamedDecoderLayer._apply` override that passes meta + tensors through while still moving/casting the real LoRA params. A *second* meta-move inside + `accelerate.prepare_model` was then found by the new end-to-end test, not by review. +2. **HIGH — the shard cache could silently stream the wrong weights.** It was keyed by model + slug with only a dtype check, so a local checkpoint retrained in place (or two ids colliding + onto one slug) would train against stale weights with no error. Now keyed by a + `source_fingerprint` (per-shard name + size + mtime digest) that invalidates like dtype does. +3. **HIGH — DoRA / VeRA / PiSSA / OLoRA read the real base weight at `get_peft_model()` time** + and crash opaquely on `meta`. Now refused by name at config-parse. +4. **The pre-flight hardware-fit gate refused the runs this feature exists to enable.** It + models a RESIDENT run, so it rejected a 3B streaming config outright. Found by the step-6 + smoke, not by any test. Now skipped for streaming runs, with a passing control test proving + resident configs are still gated. + +Shipped scope is narrower than the gates alone would justify: RAM tier · bf16 · `task=sft` · +Llama/Qwen · batch 1 · no gradient accumulation · no `--resume`. Every refusal names the +release that lifts it (NF4 -> v0.72.1; disk tier, larger batches, accumulation, resume -> +v0.72.2). + +Final test count 16576 -> **16735** (+159 in `tests/test_v07200.py`), 324 test files. + +--- + +## Citation check on the write-up (2026-07-27) + +The gate numbers above are ours and were never in doubt. The *third-party* numbers drafted +around them for the accompanying report were checked against source and +**three were wrong**: MegaTrain listed as a 24 GB/3090 system (it is a single H200 with 1.5 TB +host RAM, up to 120B); a "26% GPU utilisation on a 4090" figure attributed to ZeRO-Infinity +(which predates the 4090 — not their measurement); and LSP-Offload's overhead given as 1.45× +with 1.31× separately credited to gradient checkpointing (the real published figure is a 31% +slowdown for LSP-Offload itself — one wrong number, re-used under a second wrong label). + +Consequence for the claim: "every published system floors at 24 GB" was **false** — +LSP-Offload (arXiv:2406.10181) already fine-tunes 1.3B on a 4 GB laptop GPU. The defensible +headline is **1.3B -> 3B on a 4 GB card**, with bit-exactness verified, not "nobody has done +this." diff --git a/benchmarks/gate-v0.72.2-nf4.md b/benchmarks/gate-v0.72.2-nf4.md new file mode 100644 index 0000000..b694678 --- /dev/null +++ b/benchmarks/gate-v0.72.2-nf4.md @@ -0,0 +1,339 @@ + + +# NF4 Layer Streaming — Gate results (now the **v0.72.2** slot) + +> **Filename kept as `v0721-*` deliberately.** These gates were run while NF4 was +> v0.72.1. They surfaced a shipped v0.72.0 correctness defect (adapters saved with an +> `.inner.` key prefix reload as zero tensors), which was split out to ship alone and +> first as **v0.72.1**; NF4 moved to **v0.72.2**. **These results stand as-is and are not +> to be re-run** — see plan §7.1. The adapter-key defect is written up below because +> this gate is what found it. + +**Status: GATE 1 PASS (6/6, bit-exact vs resident NF4). GATE 2 complete, incl. the 8B headline.** +Nothing under `src/` was written before these ran (plan §7.1 rule). + +Hardware (all numbers on this box, no extrapolation): +**Windows 11 · RTX 3050 Laptop 4 GB (CC 8.6) · 16.9 GB RAM · NVMe** +torch 2.5.1+cu121 · bitsandbytes 0.49.2 · transformers 4.57.6 · peft 0.18.1 · +trl 0.19.1 · accelerate 1.12.0 · Python 3.10.8 + +Unit convention: **decimal GB** throughout (matches the v0.72.0 gate doc). + +The reference is **resident NF4**, per plan §7.1 — not resident bf16, which would +differ by quantisation error and hide a real bug inside it. Streamed NF4 and +resident NF4 hold the *same quantised bytes* and run the *same bitsandbytes +kernels*, so the standard is bit-exactness. + +--- + +## GATE 1 — correctness — **PASS** (2026-07-27) + +Throwaway spike (`scratchpad/v0721_gate1.py`), nothing under `src/`. +Model **HuggingFaceTB/SmolLM2-135M** (llama arch, 30 layers, tied embeddings), +NF4 + double quant + bf16 compute — i.e. the repo's own `quant_menu` defaults — +LoRA r=16 on `q_proj,v_proj`, seq 512, batch 1, double buffering (n=2) with a +dedicated prefetch stream. + +Construction under test (the v0.72.0 engine, plus the P3 work): +- checkpoint **pre-quantised offline**, one tensor at a time on the GPU, into + per-layer shards of packed `uint8` + `absmax` (+ nested `absmax`/`offset`) +- `Params4bit` **views rebuilt over the pooled buffers on every call**, with a + `QuantState` reassembled from the streamed tensors + two shared constant code + tables +- skeleton on `meta` via `init_empty_weights` + `replace_with_bnb_linear`; + **270 decoder weight tensors never leave `meta`** +- `functional_call` feeds the rebuilt `Params4bit` into the unmodified HF layer +- `checkpoint(use_reentrant=False)`; base `requires_grad=False`, no `detach()` + +| # | Check | Threshold | Measured | Verdict | +|---|---|---|---|---| +| 0 | offline shard bytes vs the resident model's `Params4bit` | identical | **IDENTICAL** (7 weights on layer 0: packed + absmax + nested absmax) | PASS | +| 1 | streamed vs resident NF4 logits, max abs diff | < 1e-3 | **0.0 (bit-exact)** | PASS | +| 2 | LoRA grad on **layer 0** | != 0 | **4.921e-01**; 30/30 layers non-zero | PASS | +| 3 | 25 steps streamed vs resident, loss curve | within noise | **max rel 0.0**; 12.99325 -> 12.09359 on both | PASS | +| 4 | same seed twice -> identical loss | identical | **0.0** | PASS | +| 5 | buffer-count invariance n=2 vs n=3 | identical | **0.0** | PASS | + +RAM store 0.055 GB pinned; pool 3.7 MB x 2. + +### P3 is solved, and the probe that settles it + +`Params4bit` carries a `quant_state` and quantises on transfer to CUDA, so NF4 +weights cannot be byte-copied into a plain buffer (plan P3). Measured answers +(`scratchpad/probe_nf4_api.py`): + +- `quantize_4bit` is **deterministic and byte-identical** across repeats and + across a CPU round-trip — so offline sharding reproduces exactly what a + resident load would have produced. Check 0 confirms this end-to-end. +- A `Params4bit` **rebuilt over an existing packed buffer** with an explicit + `quant_state` (`bnb_quantized=True`) produces a **bit-identical** forward. +- `torch.func.functional_call` accepts it, **including over a `meta` + placeholder** of a different shape — so the v0.72.0 substitution mechanism + carries over unchanged. +- Gradient flows to the layer input through `matmul_4bit` with the weight at + `requires_grad=False` — plan P2 holds. + +Per-weight streamed bytes (double quant): packed `N/2` + absmax `N/64` + +nested absmax `N/4096` + a 4-byte offset ≈ **0.516 bytes/param**. The NF4 code +table (16 fp32) and the nested code table (256 fp32) were verified **constant +across every weight**, so one shared resident copy is safe — the sharder +asserts this rather than assuming it. + +### What the gate caught that a green run would not have + +**1. PEFT silently used a different LoRA math path (found by check 1).** +With byte-identical adapters and byte-identical weights, streamed vs resident +logits differed by **9.375e-01**. Cause: `transformers.from_pretrained` stamps +`is_loaded_in_4bit` on the model, and PEFT reads it to dispatch +`lora.bnb.Linear4bit`. A `meta` skeleton has no such marker, so PEFT dispatched +the generic `lora.layer.Linear`, which still *runs* against a `Linear4bit` base +— it just casts and accumulates differently. No crash, no warning; the loss +curve looked healthy. Fixed by stamping the markers `from_pretrained` sets. +This is the single most important finding of the gate. + +**2. Two ways the gate itself was nearly vacuous** (harness bugs, fixed before +the numbers above were taken): +- PEFT initialises `lora_B = 0`, so at step 0 the adapter contributes nothing + and `dL/dA` is structurally zero for *any* correct implementation. The first + run "passed" check 1 bit-exactly while the adapter path was doing nothing at + all, and "failed" check 2 for a reason that had no bearing on streaming. The + gate now randomises `lora_B` first, which makes the adapter path load-bearing + in checks 1 and 3. +- The adapter sync between the two models silently copied **0 tensors** because + of the key-prefix bug below, leaving the models with different adapters. It + now refuses to run rather than compare two differently-initialised models. + +--- + +## Blocking rider found in shipped v0.72.0 — adapters are saved unloadable + +Reproduced through the **shipped** `build_streamed_model` +(`scratchpad/check_inner_keys.py`, CPU, tiny-random-Llama): + +`install_streaming` replaces `layers[i]` with a wrapper that holds the real +layer as a child named `inner`. Every adapter parameter therefore gains an +`.inner.` segment in its state-dict path, and that name is what +`get_peft_model_state_dict` -> `save_pretrained` writes to disk: + +``` +base_model.model.model.layers.0.inner.self_attn.q_proj.lora_A.weight + ^^^^^^ +``` + +Loading that adapter back into a normal model drops **8 of 8** tensors. PEFT +emits a `UserWarning` about missing keys and returns a model that is byte-for-byte +the untuned base: + +``` +reload into a plain model: 0/4 lora_B tensors non-zero +VERDICT: ADAPTER SILENTLY LOST ON RELOAD +``` + +Consequence: **every adapter produced by `soup train --stream-layers` in +v0.72.0 is inert outside the streaming path** — `soup merge`, `soup serve`, +`soup chat` and `PeftModel.from_pretrained` all load it as a no-op. The +training run itself is correct (the gates prove that); only the artifact it +writes is unusable. Nothing in the 159 v0.72.0 tests saves an adapter and loads +it back. + +This must be fixed in v0.72.1 — it is in the same component the NF4 work +modifies, and shipping NF4 on top of an unloadable artifact would compound it. + +**Both candidate fixes were spiked and both work** (`scratchpad/fix_inner_keys_spike.py`, +CPU, tiny-random-Llama; control = shipped code): + +| Variant | saved keys with `.inner.` | reload into a plain model | +|---|---|---| +| shipped v0.72.0 (control) | 8 / 8 | **0/4 adapters — LOST** | +| A: transparent `state_dict()` on the wrapper | 0 | 4/4 — round-trips | +| B: wrapper shares the inner layer's `_parameters`/`_modules` | 0 | 4/4 — round-trips | + +A is surgical (serialisation only); B removes the nesting level altogether, so +`named_parameters()` is canonical too — which also unblocks loading *into* a +streamed model later (`--resume`, a v0.72.2 item). Both keep the forward +working. Choice to be made at implementation, TDD-first, with the reload +round-trip as the spec. + +--- + +## GATE 2 — the number + +Protocol unchanged from v0.72.0 so the rows are comparable: tok/s over 50 steps +after 10 warm-up · `torch.cuda.max_memory_allocated()` · SM occupancy from +`nvidia-smi dmon -s u` · `PagedAdamW8bit` · batch 1 · double buffering · TFLOPS +back-check at C=6 against this box's **measured** 6.76 TFLOPS dense bf16 GEMM +ceiling (theoretical GA107 peak ~24.5; the plan's "9" is wrong for this card). + +Parameter counts are read from the safetensors headers, so a tied `lm_head` is +not double-counted (counting `model.parameters()` on the meta skeleton reported +Qwen2.5-3B as 3.40 B instead of 3.09 B and inflated TFLOPS accordingly). + +| Model | Quant | Store | S | tok/s | GPU util | Peak VRAM | TFLOPS eff | +|---|---|---|---|---|---|---|---| +| **Llama-3.1-8B** | **NF4** | **3.60 GB pinned** | 512 | **122.5** | **98.9%** / 100% | **3.45 GB** | 5.90 | +| Qwen2.5-3B | **NF4** | 1.43 GB **pinned** | 512 | **244.3** | 94.5% / 100% | 1.91 GB | 4.53 | +| Qwen2.5-3B | bf16 *(v0.72.0)* | 5.55 GB pageable | 512 | 143.1 | 79.3% / 100% | 2.15 GB | 2.65 | + +### The 8B headline + +**Llama-3.1-8B fine-tunes on a 4 GB card at 122.5 tok/s**, with the base +page-locked (3.60 GB store, under the box's measured 7.65 GB pinned ceiling) and +peak VRAM 3.45 GB — genuinely inside the card, not a WDDM spill. Untied +`embed_tokens` + `lm_head` stay resident and bf16 (2.10 GB of that 3.45 GB), which +is why 8B is close to this card's ceiling; treating them as streamed large layers +is a v0.72.3 item. NF4 shard set on disk 5.70 GB, sharded once and cached. + +Derived, labelled as arithmetic: 1M training tokens = **2.3 h at 8B**, 1.1 h at 3B. + +### The TFLOPS back-check, resolved — the DENOMINATOR was wrong + +The first pass read 5.90 TFLOPS = **87% of the v0.72.0 gate's 6.76 TFLOPS** +"achievable dense bf16 GEMM" figure, which is not credible for an eager loop +that also dequantises NF4. It was held back rather than published. Both +candidate explanations were then tested: + +**(b) gradient checkpointing off, so C=6 is the wrong constant — DISCONFIRMED.** +The streamed layer wraps its body in `checkpoint(use_reentrant=False)` whenever +grad is enabled, and the measured loop is a forward+backward. C=6 is correct. + +**(a) the 6.76 ceiling was measured at shapes that do not reach peak — PARTLY, +but the dominant effect turned out to be something else: GPU CLOCK STATE.** + +An first re-measure suggested 6.76 was simply an under-measurement (square +shapes came back 7.08 / 7.52 / 7.66). A second re-measure in a later session +gave **6.23 / 6.63 / 6.75** for the *same* shapes — i.e. 6.76 reproduced almost +exactly. Repeating 4096^3 six times inside one session then showed the +measurement is stable to **<1%** (6.93, 6.94, 6.93, 6.94, 6.93, 6.94) with the +SM clock pinned at **862 MHz / 63 C**. + +So the ~13% spread is *between* sessions and tracks the boost clock, not +measurement noise. **6.76 was never wrong; it was measured in a lower-clock +session.** The methodological rule that follows is stronger than the original +correction: **a ceiling is only comparable to a throughput measured in the same +session at the same clock**, and any fraction-of-ceiling should state the clock +it was taken at. Shape still matters, but less than clock. + +Shape-matched ceilings, all measured in ONE session at 862 MHz, bf16, B=1/S=512: + +| op | M x K x N | TFLOPS | FLOP share | +|---|---|---|---| +| q_proj / o_proj | 512 x 4096 x 4096 | 7.71 | 15.4% | +| k_proj / v_proj | 512 x 4096 x 1024 | 7.57 | 3.8% | +| gate/up_proj | 512 x 4096 x 14336 | 7.49 | 53.8% | +| down_proj | 512 x 14336 x 4096 | 7.69 | 26.9% | +| **FLOP-weighted** | | **7.58** | | + +| model shapes | weighted ceiling @862 MHz | +|---|---| +| Llama-3.1-8B | 7.58 *(measured in the earlier high-clock session)* | +| Qwen2.5-3B | 6.67 | +| Qwen2.5-1.5B | 6.39 | +| Qwen2.5-0.5B | 5.90 | + +Smaller models sit lower, as expected — their GEMMs are too small to saturate. + +**The numerator convention, fixed and stated so a reader can reproduce it:** +decoder params at **C=6** (2 fwd + 2 recompute + 2 dL/dx). **Not C=8** — the +base is frozen, so no dL/dW is computed for it, and that missing term is exactly +what C=8 would add. `embed_tokens` contributes **0 FLOPs** (a lookup). +`lm_head` runs at **C=4** (2 fwd + 2 dL/dx): it sits outside the checkpointed +decoder layers so its forward is not recomputed, and it is frozen. + +| run | GFLOP/token | effective | % of its shape-matched ceiling | +|---|---|---|---| +| Qwen2.5-0.5B @ 978.6 tok/s | 2.69 | 2.63 | 45% | +| Qwen2.5-1.5B @ 525.0 tok/s | 8.79 | 4.62 | **72%** | +| Qwen2.5-3B @ 143.1 tok/s | 17.89 | 2.56 | 38% | +| Llama-3.1-8B NF4 @ 122.5 tok/s | 43.98 | 5.39 | 71% | + +The 1.5B row being the highest is exactly right: it is the one run measured at +96.8% SM occupancy, i.e. the compute-bound one. The 3B bf16 row at 38% is the +transfer-bound pageable-store run. The picture is now coherent across all four. + +**Caveat that must travel with these fractions:** the throughputs come from +earlier sessions and the ceilings from this one, so each ratio inherits the +cross-session clock uncertainty (~13%). Before any of these fractions is +published as load-bearing, re-measure throughput and ceiling **back to back in +one session** and report the SM clock alongside. + +**Publish tok/s and peak VRAM as the measurements — those are clock-sensitive +too but they are what was actually observed. The TFLOPS column is derived; quote +it only with the constant, the parameter accounting and the clock stated.** + +**NF4 makes the 3B row 1.71x faster than bf16 streaming, and the reason is not +the arithmetic — it is that a 1.43 GB store fits under the page-locked ceiling +where a 5.55 GB one did not.** Pinning restores async `copy_`, which is visible +as utilisation 79.3% -> 94.5%. This is the v0.72.0 honesty caveat #1 ("the 3B +number is a LOWER BOUND because the store went pageable") being lifted by NF4. + +### The resident-NF4 3B baseline was measured and then DISCARDED + +First attempt: 34.1 tok/s, **peak VRAM 6.07 GB on a 4 GB card**, 100% util, +0.63 TFLOPS. A 6.07 GB peak on a 4 GB card is a WDDM shared-host-memory spill — +the identical failure mode that made v0.72.0 discard its Qwen2.5-1.5B resident +row. Publishing it would have produced a **"streaming is 7.2x faster than +resident"** headline out of a number that measures Windows paging, not +training. It is discarded, not reported. + +It was also not a fair comparison: gradient checkpointing was off on the +resident side, so it ran at C=4 while the streamed side runs at C=6 *and* it +held every activation, which is what pushed it over the card. The harness now +enables checkpointing for resident throughput runs; a re-measurement is queued +behind the 8B headline. + +**Consequence to state in the release:** as of this gate there is still **no +valid resident baseline at 3B on this box** — resident bf16 OOMs (v0.72.0) and +resident NF4 spills. The honest claim remains the one v0.72.0 made at 0.5B, not +a 3B speed-up ratio. + +Remaining row pending: the 8B headline. + +--- + +## v0.72.2 step-6 reproduction — through the SHIPPED code (2026-07-28) + +The Gate-2 rows above came from the throwaway spike. These were re-measured with +`shard_checkpoint` / `build_streamed_model` as released, same protocol (50 steps +after 10 warm-up, batch 1, S=512, `PagedAdamW8bit`, double buffering), and the +GEMM ceiling taken **in the same session at the stated clock**, per the rule the +gate itself established. + +| Model | Quant | Store | tok/s | GPU util | Peak VRAM | SM clock | eff TFLOPS | % of same-session ceiling | +|---|---|---|---|---|---|---|---|---| +| **Llama-3.1-8B-Instruct** | NF4 | **3.60 GB pinned** | **119.6** | 100% | **3.32 GB** | 952 MHz / 70 C | 5.26 | 68% | +| Qwen2.5-3B | NF4 | 1.43 GB pinned | 264.2 | 100% | 1.76 GB | 960 MHz / 72 C | 4.73 | 61% | + +Agreement with the spike is close (8B: 119.6 vs 122.5 tok/s, 3.32 vs 3.45 GB +peak, identical 3.60 GB pinned store), i.e. the released implementation +reproduces the gate rather than merely resembling it. The residual is inside the +~13% between-session clock spread the gate documented. + +Shard cache behaved: the second 8B run reported `shards ready in 0.0s`. + +**The 3B NF4-vs-bf16 comparison, and its caveat.** 264.2 tok/s (NF4) against +v0.72.0's 143.1 tok/s (bf16) is 1.85x, but the cause is *not* arithmetic — it is +that a 1.43 GB store page-locks where a 5.55 GB one did not, which restores +async `copy_` and moves utilisation 79.3% -> 100%. The bf16 row was measured in +an earlier session at an unrecorded clock, so the ratio inherits that +uncertainty; the mechanism (pinning) is the load-bearing claim, not the factor. + +Derived, labelled as arithmetic: 1M training tokens = **2.3 h at 8B**, 1.05 h at 3B. + +**Also found by step 6, and fixed:** a streamed NF4 model reported +878,154,048 parameters for SmolLM2-135M (true 134,515,008) while the resident +NF4 path reported 134,975,808. PEFT computes `Params4bit` totals as +`numel * 2 * quant_storage.itemsize`, correct for a resident tensor whose numel +is the packed count but not for a `meta` placeholder carrying the logical shape. +Display-only; the loss curve was byte-identical before and after. At 8B it would +have printed ~52 B. diff --git a/benchmarks/gate-v0.72.3-breadth.md b/benchmarks/gate-v0.72.3-breadth.md new file mode 100644 index 0000000..97fac42 --- /dev/null +++ b/benchmarks/gate-v0.72.3-breadth.md @@ -0,0 +1,472 @@ + + +# v0.72.3 "Breadth" — per-item gate results + +Inherited standard — **a streamed run must be +bit-exact against the resident run of the same numerics**; what changes per item is +the *reference*, not the standard. Gradient accumulation is the one item whose gate +is a **measured I/O cost**, not an equality (the brief says so explicitly). + +Box: Windows 11 · RTX 3050 Laptop 4.29 GB · 16.9 GB RAM · NVMe · Python 3.10 · +torch 2.5.1+cu121 · transformers 4.57.6 · peft 0.18.1 · trl 0.19.1 · bitsandbytes 0.49.2. + +Gates are throwaway scripts in the session scratchpad; nothing under `src/` was +written before a gate passed. + +--- + +## GATE 1 — multi-arch (Mistral / Gemma / Phi) — **PASS 14/14** (2026-07-28) + +The brief: *"wrong layer detection streams weights into the wrong module and trains +something plausible-but-wrong. Gate = bit-exactness on a small model of each family +added, not just the allowlist entry."* + +**Method.** Tiny (3-layer, hidden 32, vocab 64) from-config checkpoints of each +family, written to disk as real `config.json` + `model.safetensors`, then: +shard → `build_streamed_model` → compare logits against **the same checkpoint loaded +resident with the same adapter weights**. CPU, float32. For `quant=nf4` the reference +is a **resident NF4 load**, never a resident bf16 one — a bf16 reference differs by +quantisation error and would hide a real bug inside it. + +From-config rather than downloaded checkpoints because the module tree comes from the +same `modeling_*.py`, so layer detection, name mapping and `replace_with_bnb_linear` +conversion are exercised identically — and it stays runnable on a 4 GB box. The risk +the brief names is a *naming* property, not a size property. + +**Vacuity defence.** PEFT initialises `lora_B = 0`, so a completely detached adapter +is byte-identical to a fresh one and every parity assertion passes for the wrong +reason. Every run randomises `lora_B` first and asserts a non-zero number of tensors +were copied to the reference. + +Checks per family: (1) logits bit-exact `torch.equal`; (2) layer-0 LoRA gradient +non-zero (plan P2 — a severed graph still lowers loss, so nothing else catches it); +(3) decoder parameters still on `meta` (or it is not streaming at all). + +| family | quant | max abs logit diff | bit-exact | layer-0 LoRA grad | meta decoder params | +|---|---|---|---|---|---| +| llama *(control)* | none | 0.000e+00 | yes | 3.2391e+00 | 27 | +| llama *(control)* | nf4 | 0.000e+00 | yes | 3.3290e+00 | 27 | +| **mistral** | none | 0.000e+00 | yes | 3.2391e+00 | 27 | +| **mistral** | nf4 | 0.000e+00 | yes | 3.3290e+00 | 27 | +| **gemma** | none | 0.000e+00 | yes | 5.1453e-01 | 27 | +| **gemma** | nf4 | 0.000e+00 | yes | 5.2169e-01 | 27 | +| **gemma2** | none | 0.000e+00 | yes | 3.8365e+00 | 33 | +| **gemma2** | nf4 | 0.000e+00 | yes | 4.0558e+00 | 33 | +| **gemma3_text** | none | 0.000e+00 | yes | 4.3802e+00 | 39 | +| **gemma3_text** | nf4 | 0.000e+00 | yes | 4.5446e+00 | 39 | +| **phi** | none | 0.000e+00 | yes | 2.9908e+00 | 42 | +| **phi** | nf4 | 0.000e+00 | yes | 3.0868e+00 | 42 | +| **phi3** | none | 0.000e+00 | yes | 3.2312e+00 | 18 | +| **phi3** | nf4 | 0.000e+00 | yes | 2.9316e+00 | 18 | + +**Notes.** + +- **`phi3` is the interesting row.** Phi-3 *fuses* Q/K/V into a single `qkv_proj` and + gate/up into `gate_up_proj`, so it has no `q_proj`/`v_proj` at all — the LoRA target + set had to be `["qkv_proj"]`. It is bit-exact anyway, which is the strongest + evidence that layer detection and the shard name map are driven by the real module + tree rather than by Llama-shaped assumptions. +- **`gemma3` (multimodal) is deliberately NOT added** — only `gemma3_text`. A real + `google/gemma-3-*-it` reports `model_type='gemma3'` for the vision-capable wrapper; + refusing it is correct, because streaming a multimodal wrapper as if it were a + causal LM is exactly the silent-mis-train the allowlist exists to prevent. +- Two initial failures were **the gate harness's own fault, not the feature's**: + `Phi3Config` defaults `pad_token_id=32000`, out of range for a 64-token fixture + vocab. Fixed in the fixture (`pad_token_id=0`) and re-run; recorded here rather than + quietly dropped. +- `meta decoder params` differs per family because the families have different + per-layer parameter counts (gemma2/gemma3 carry extra norms; phi3 fuses, so fewer + and larger tensors). All are `> 0`, which is the property being asserted. + +**Verdict: the allowlist may be extended to `mistral`, `gemma`, `gemma2`, +`gemma3_text`, `phi`, `phi3`.** + +--- + +## GATE 2 — pre-flight VRAM budget (batch- and vocab-aware) — **PASS** (2026-07-28) + +The brief: *"Untied embed + lm_head — 8B has them untied, so two large matrices go +resident. Budget them in `estimate_stream_vram` or the 4 GB card OOMs on a model the +planner said would fit."* Operator addition: **batch scales the intra-layer transient +AND the logits tensor** (`batch × seq × vocab` plus its fp32 upcast), and on a +large-vocab model that term is larger than both layer buffers combined. + +**Method — measure first, fit second.** Real streamed forward+backward+step on real +cached checkpoints over a (model, batch, seq) grid, recording +`torch.cuda.max_memory_allocated()`. The two models were chosen for a **3.1× vocab +contrast** (49 152 vs 151 936) so the logits term could be separated from everything +else rather than assumed. Nothing was implemented before this ran. + +Card RTX 3050 Laptop 4.29 GB · SM clock 952 MHz (one row throttled to 442 MHz, noted +below) · free RAM 0.30–1.21 GB across the grid · pageable store (pinning is a *speed* +property and does not move VRAM). + +Solving the two models simultaneously for the per-token cost yields: + +``` +peak = pool + extras + adapter_params·16 + 13.5 MB + + batch · seq · ( 14·vocab + 2·n_layers·hidden + 4·(hidden + intermediate) ) +``` + +- `14·vocab` — bf16 logits (2) + fp32 upcast (4) + fp32 log-softmax (4) + fp32 grad + (4), i.e. exactly what `transformers` `ForCausalLMLoss` holds live. **The shipped + `estimate_logits_bytes` used 6** (bf16 + upcast only), a first-principles guess that + under-predicts this term by 2.33×. +- `2·n_layers·hidden` — the `checkpoint(use_reentrant=False)` boundary save, one bf16 + copy per layer. +- `4·(hidden + intermediate)` — the live transient inside the ONE layer being + recomputed (independent of `n_layers`, which is the whole point of streaming). + +| model | B | S | measured GB | predicted GB | err | +|---|---|---|---|---|---| +| SmolLM2-135M | 1 | 256 | 0.285 | 0.286 | +0.58% | +| SmolLM2-135M | 1 | 512 | 0.471 | 0.473 | +0.53% | +| SmolLM2-135M | 2 | 512 | 0.843 | 0.848 | +0.59% | +| SmolLM2-135M | 4 | 512 | 1.584 | 1.596 | +0.75% | +| SmolLM2-135M | 8 | 512 | 3.069 | 3.094 | +0.82% | +| Qwen2.5-0.5B | 1 | 256 | 0.920 | 0.924 | +0.47% | +| Qwen2.5-0.5B | 1 | 512 | 1.476 | 1.486 | +0.62% | +| Qwen2.5-0.5B | 2 | 512 | 2.592 | 2.609 | +0.63% | +| Qwen2.5-0.5B | 4 | 512 | 4.819 | 4.854 | +0.73% | +| Qwen2.5-0.5B | 8 | 512 | 9.267 | 9.346 | +0.85% | + +**Worst absolute error 0.85%. Worst under-prediction: none — every prediction is ≥ the +measurement**, which is the only safe direction for a gate that must refuse configs. + +**Independent check, nothing fitted to it:** the published v0.72.2 Llama-3.1-8B NF4 +row — different model, different quantisation, different session, untied embeddings — +predicts **3.57 GB against a measured 3.32 GB (+7.5%, over)**. The formula was fitted +on two models 16–60× smaller and still brackets an 8B NF4 run on the safe side. + +### Two findings that change the implementation + +1. **The logits term dominates, exactly as called.** At Qwen2.5-0.5B B=8 S=512 it is + **8.71 GB of the 9.35 GB predicted — 146× the entire buffer pool (0.060 GB)**. A + pre-flight that budgeted only weights and buffers would green-light this config. + Batch budgeting is therefore not a refinement of the estimator; it is the estimator. + +2. **Windows/WDDM does not OOM — it spills, silently.** The B=8 row allocated + **9.27 GB on a 4.29 GB card and raised nothing**, finishing all three steps. So on + this platform *"it did not crash"* is **not** evidence that a config fits, and the + fit/no-fit direction of the estimator **cannot be validated by observing an OOM + here**. What is validated is the *demand* prediction (±0.85%); refusing when demand + exceeds the card is then policy applied to a measured quantity, and is documented as + such rather than claimed to be empirically OOM-verified on this box. This is the + same WDDM shared-memory spill that invalidated the v0.72.0 1.5B "resident baseline". + +One row (Qwen B=4) was taken while the SM clock had dropped to 442 MHz from 952 MHz. +Clock does not affect an allocation measurement, so the row stands; it is recorded +because the throughput items must not mix clock states. + +### Reserve for the fit decision — also measured + +The plan's `DEFAULT_WORKSPACE_BYTES` of 1 GB is **not** usable as the reserve: charged on +top of a 3.57 GB prediction it would refuse the 8B NF4 run that this feature exists to +enable and that v0.72.2 actually measured. Measured on this box instead: + +``` +before CUDA init : 3.460 GB free of 4.294 GB +after CUDA init : 3.447 GB free -> context + driver + display = 0.847 GB +1.5 GB tensor : allocator 1.500 GB, driver-visible 2.349 GB + -> overhead beyond `allocated` = 0.849 GB (stable) +``` + +So the fit budget is read from `torch.cuda.mem_get_info()` **at pre-flight time** rather +than hardcoded — the offset includes desktop/display usage and is therefore a property of +the machine, not of the card model. Against this box's 3.445 GB of allocator-visible VRAM +the decision is consistent with every measured row: 8B NF4 (3.32 GB) fits, Qwen B=4 +(4.82 GB) does not. + +### Item 2 — implemented and verified end-to-end + +Real `soup train --stream-layers` on SmolLM2-135M, **batch 2** (the first batch > 1 ever +run under streaming): completed exit 0, 18 steps, adapter written, and the saved adapter +carries **0 `.inner.` keys with 60 non-zero `lora_B` tensors** — v0.72.1's canonical-key +property survives batch > 1. + +Pre-flight panel from that run: + +``` +peak VRAM ~0.48 GB at batch 2 x seq 256 (logits 0.35 GB) +free VRAM 3.46 GB +forecast 5685-8361 tok/s — a compute-bound bound, not a promise + (from 6.75 TFLOPS measured on this card now @ 862 MHz) +``` + +The 6.75 TFLOPS @ 862 MHz agrees with the box's independently recorded ~5.9–6.7 TFLOPS at +a pinned 862 MHz, i.e. the runtime probe reproduces the known ceiling rather than +inventing one. + +The refusal direction, same model at batch 64: + +``` +a streaming step is predicted to need 12.08 GB of VRAM but only 3.46 GB is free. +Streaming bounds the WEIGHTS, not the activations or the logits — lower +training.batch_size or data.max_length, both of which scale this linearly. +``` + +`batch_size: "auto"` remains refused, for a reason that does not expire: it resolves by +OOM-probing a **resident** model, which a streaming run never loads. + +--- + +## GATE 3 — gradient accumulation — **PASS** (2026-07-28) + +The brief: *"I/O multiplies linearly: every micro-batch re-reads the entire model (plan +P9). This is the one place where the 'batch is nearly free' property breaks. **Measure** +tok/s at accum 1 / 2 / 4 and publish it; do not assume it is cheap."* + +### Part A — correctness (CPU, float32, tiny Llama) + +The inherited standard. Accumulated **adapter gradients** from a streamed run must be +bit-exact against a resident run accumulating the same micro-batches — the prefetcher +re-primes on every forward, and if it mis-tracked direction across a micro-batch boundary +a stale buffer would produce quietly wrong gradients rather than a crash. + +| accum | grad tensors | max abs grad diff | bit-exact | layer-0 non-zero | prefetch primes | +|---|---|---|---|---|---| +| 2 | 12 | **0.0** | yes | 4 | 2 | +| 4 | 12 | **0.0** | yes | 4 | 4 | + +Per-micro-batch losses are identical to 6 dp on both sides. Primes == accum, i.e. exactly +one prime per micro-batch forward, which is the intended scheduling. + +### Part B — the measured I/O cost (CUDA) + +Qwen2.5-0.5B bf16 · S=256 · 50 steps after 10 warm-up · **store 0.72 GB PINNED in every +row** (the confound that mattered — pinned vs pageable — is held constant throughout) · +one session, back-to-back. + +| batch | accum | eff. batch | **tok/s** | s / opt-step | layer loads / 1k tok | peak VRAM | free RAM | SM clock | +|---|---|---|---|---|---|---|---|---| +| 1 | 1 | 1 | 556.6 | 0.460 | 175.78 | 0.842 GB | 2.56 GB | 862 MHz | +| 1 | 2 | 2 | 543.4 | 0.942 | 175.78 | 0.846 GB | 3.33 GB | 952 MHz | +| 1 | 4 | 4 | 540.1 | 1.896 | 175.78 | 0.846 GB | 3.16 GB | 952 MHz | +| 2 | 1 | 2 | 1069.1 | 0.479 | 87.89 | 1.320 GB | 3.20 GB | 952 MHz | +| 4 | 1 | 4 | **1378.0** | 0.743 | 43.95 | 2.280 GB | 2.66 GB | 960 MHz | +| 2 | 2 | 4 | 1094.5 | 0.936 | 87.89 | 1.325 GB | 2.31 GB | 960 MHz | + +**Confirmation pass.** The first row was taken at 862 MHz and the rest at 952–960, a ~10% +spread, so the three rows the headline rests on were re-measured **interleaved** +(A/B/C/A/B/C, so monotonic drift cannot favour one arm): + +| config | repeat 1 | repeat 2 | spread | +|---|---|---|---| +| batch 1 / accum 1 | 552.8 | 550.9 | 0.4% | +| batch 1 / accum 4 | 551.4 | 553.2 | 0.3% | +| batch 4 / accum 1 | 1393.4 | 1393.4 | 0.0% | + +### What the numbers actually say + +1. **Accumulation is per-token I/O-neutral, not linear-cost.** `layer loads / 1k tokens` + is **constant at 175.78 across accum 1, 2 and 4**, and tok/s is flat within 3% + (556.6 → 543.4 → 540.1; 552.8 vs 551.4 on the interleaved repeat). accum=N does N + micro-batches, N model reads and N times the tokens — the ratio does not move. The + plan's P9 wording ("multiplies IO linearly") is right per *optimizer step* and + misleading per *token*, which is the unit that decides wall-clock. + +2. **The real cost is opportunity cost against raising batch.** At the *same effective + batch of 4*: batch 4 / accum 1 delivers **1393.4 tok/s** against batch 1 / accum 4 at + **553.2 — a measured 2.52×**, because one weight read is amortised over four times the + tokens (43.95 vs 175.78 loads per 1k tokens). Publishing only the accum column would + have read as "accumulation is free"; it is free *per token* and expensive *per unit of + effective batch*. + +3. **Accumulation's actual value under streaming is that it buys effective batch at + constant VRAM**: peak moved 0.842 → 0.846 GB across accum 1→4, while reaching the same + effective batch by raising batch cost 0.842 → 2.28 GB. So the guidance the pre-flight + should give is: **raise `batch_size` until the VRAM pre-flight refuses, then use + accumulation for the rest** — which is exactly the pairing this release now makes + possible, since item 2 is what tells the user where that ceiling is. + +4. Batch scaling is real but sub-linear: 556.6 → 1069.1 (1.92×) → 1378.0 (2.48×) for + batch 1 → 2 → 4, as the run crosses from I/O-bound into compute-bound. + +### A defect in item 2's own GEMM probe, found by item 3's smoke + +The accumulation smoke printed **3.54 TFLOPS @ 862 MHz** where an earlier run had printed +**6.75 TFLOPS @ 862 MHz** — a 2× swing at the same *reported* clock, which would have made +the forecast bracket meaningless. Diagnosed rather than accepted: + +| probe size | five repeats (TFLOPS) | spread | +|---|---|---| +| 2048³ | 3.19, 3.50, 3.89, 3.85, 4.41 | **38%** | +| 4096³ | 3.92, 3.97, 3.99, 3.81, 4.16 | 9% | + +At 2048 the sample is too short for the boost clock to engage and the repeats ramp +**monotonically upward**, so whichever repeat happens to be first sets the answer. Fixed +by moving to 4096³ and taking the **best of 3** repeats. Best-of-N is not cherry-picking +here: a *ceiling* has one-sided noise, since contention, a cold clock and thermal +throttling can only ever make an achievable rate look slower than it truly is. + +This is also the concrete justification for refusing to compile a per-card TFLOPS constant +into the source — the same card, at the same reported clock, differed 2× between sessions. + +### Verified end-to-end + +Real `soup train --stream-layers` on SmolLM2-135M at **batch 2 × accum 4** (effective +batch 8): completed exit 0, 5 optimizer steps, adapter written with **0 `.inner.` keys**. +The pre-flight printed the measured advisory: + +``` +! accumulating 4x at batch 2: the base is re-read once per micro-batch. Per token + that is free, but reaching effective batch 8 by raising training.batch_size + instead measured ~2.5x faster. Accumulation holds peak VRAM flat, so raise + batch_size while the budget above allows, then accumulate for the rest. +``` + +--- + +## GATE 4 — checkpoint / resume — **PASS** (2026-07-28) + +The brief: *"silent failure is resuming against a stale shard cache or a mismatched +optimizer state. Gate = save mid-run, resume, assert the loss curve CONTINUES rather than +restarts, and assert the shard fingerprint is re-verified on resume."* + +### The failure, established before anything was changed + +v0.72.1 fixed the **save** direction only, deliberately — `state_dict()` delegates at the +wrapper's own prefix while `named_parameters()` still carries `.inner.`. The load +direction was therefore still broken, and measurably so: + +| check | before | +|---|---| +| saved tensors / of which `.inner.` | 12 / **0** (save side correct) | +| adapter tensors in model / **landed** | 12 / **0** | +| non-zero `lora_B` in the checkpoint | 6 (so the comparison is not vacuous) | +| **resumed losses vs from-scratch losses** | **byte-identical** | +| shard fingerprint detects a changed source | already true (v0.72.0) | + +The mechanism: `nn.Module.load_state_dict` narrows the dict **by child name** as it +descends. The wrapper's only child is `inner`, so a canonical +`...layers.0.self_attn.q_proj.lora_A.weight` matches no child prefix and is dropped. +PEFT reported the keys as missing via a `UserWarning` and continued. + +### The fix + +A `_register_load_state_dict_pre_hook` on `StreamedDecoderLayer` that injects +`.inner.`-prefixed copies into the (already prefix-narrowed) state dict. The pre-hook runs +at the start of the wrapper's own `_load_from_state_dict`, and torch's child loop reads +that same dict object afterwards — so the redirected keys are visible when it descends +into `inner`. Load-side only: it redirects keys rather than re-parenting the module tree, +so the forward path is untouched and **v0.72.0's bit-exactness gates stay valid without +being re-run** — the same reasoning that justified the v0.72.1 approach. + +### After, on the production CUDA path + +Driven through the real `PeftModel.load_adapter`, which is what +`Trainer._load_from_checkpoint` calls for a PEFT model: + +| check | after | +|---|---| +| tensors landed | **12 / 12** | +| `hf_device_map` preserved | **true** (`{'': 0}` unchanged) | +| decoder params still on `meta` | 27 (still streaming, not materialised) | +| resumed vs from-scratch loss curve | **differ** — the checkpoint contributes | + +`hf_device_map` is checked explicitly because `install_streaming` sets it so +`Trainer._move_model_to_device` skips `.to()` on meta weights; a loader that rewrites it +would reintroduce v0.72.0's CRITICAL. + +### Two honest limitations found while gating + +1. **PEFT re-dispatches a CPU-built streamed model.** On a CPU build, `load_adapter` moved + every parameter to `cuda:0` and rewrote `hf_device_map` from `{'': 'cpu'}` to + `{'base_model': 0}`, breaking the model. It fires only when the device map mentions + `"cpu"`, which the production path never does — streaming exists to bound VRAM, so a + CPU-built streamed model is a test convenience, not a configuration. The CPU tests + therefore exercise the redirection mechanism through `set_peft_model_state_dict` (what + `load_adapter` calls internally to place weights), and the full `load_adapter` path is + covered by a CUDA-gated test. + +2. **End-to-end `soup train --resume` cannot be demonstrated on this box, for a reason + that is not streaming's.** `transformers.check_torch_load_is_safe()` raises unless + torch ≥ 2.6 (CVE-2025-32434); this box has torch 2.5.1+cu121, so *every* resume fails + here. Proven with a **control**: the identical config with `stream_layers: false` + produces the identical error. What is verified is the streaming-specific half — the + adapter round-trip and loss continuity — at the library level on CUDA, plus a + behavioural CLI test that both `--resume` and `--hf-resume` are no longer refused. + +--- + +## GATE 5 — disk-kind detection (the `soup doctor` rider) — **PASS** (2026-07-28) + +v0.72.0 shipped a `choose_tier` that refuses anything but NVMe — wired to a **hardcoded +`disk_kind="nvme"`** in the trainer. A guard connected to a constant can never fire, so +the refusal existed only on paper. + +Probed on this box: `Get-PhysicalDisk` returns +`{"MediaType":"SSD","BusType":"NVMe","Size":512110190592}` — correctly identifying the +512 GB NVMe. Two things fell out of the measurement: + +1. **`BusType` must beat `MediaType`.** An NVMe drive reports `MediaType: SSD`; keying on + media type alone would classify every NVMe disk as SATA SSD and refuse the disk tier + universally. +2. **The probe costs 9.04 s cold, ~2.4 s warm** (PowerShell + CIM startup; Windows caches + it after the first launch). Too slow to run unconditionally, and it shapes the API: + `choose_tier` now takes a **callable**, so the probe runs only when the base does *not* + fit in RAM — the case where the answer matters. The result is cached per volume per + process. Linux reads `/sys/block/*/queue/rotational` instantly; unknown platforms + return `"unknown"`, which `choose_tier` refuses (the safe direction — believing a + spinning disk is NVMe costs hours of thrashing, plan P11). + +`soup doctor --disk` reports it: +`Disk type │ NVMe — layer streaming can use the disk overflow tier`. + +**Opt-in, following the command's own `--nccl` convention.** Measured: `soup doctor` +17.56 s by default, 19.97 s with `--disk`. A first reading of this over-claimed — 9 s was +subtracted from the total to infer "roughly doubled", which the direct A/B does not +support; the true warm cost is ~2.4 s on a command that already takes ~17.5 s (it imports +torch, transformers and friends to build the dependency table). The flag is still the right +call, because the cost is paid by every user while only streaming users benefit, and a +streaming run probes lazily on its own regardless. + +--- + +## GATE 6 — the disk overflow tier — **PASS** (2026-07-28) + +**The limitation, up front rather than at the end: this box cannot measure the RAM-vs-disk +performance gap, and this release does not claim one.** Two independent reasons — +safetensors memory-maps the shards, so the OS page cache keeps them resident between steps +on any machine with spare RAM; and at ~5 effective TFLOPS the NVMe read hides under compute +anyway (plan §2.3). What is gated here is correctness, which *is* demonstrable. + +The brief asks for "the same four checks with `DiskSource` substituted". The strongest +available reference is the **RAM tier**, not the resident model: both stream through the +same buffer pool, the same prefetcher and the same layer wrapper, differing only in where +`get(idx, name)` reads from — so any difference is attributable to the source. The +resident comparison is kept as well, anchoring the tier to ground truth rather than merely +to its sibling. + +| check | result | +|---|---| +| disk vs **RAM tier** logits | **0.0 — bit-exact** | +| disk vs **resident** logits | **0.0 — bit-exact** | +| layer-0 LoRA gradient | 2.6260 (non-zero, plan P2) | +| determinism (rebuild + re-run) | bit-exact | +| `store_bytes` (held resident) | **0** — the point of the tier | +| `disk_bytes` vs RAM tier `store_bytes` | 148 480 == 148 480 (accounting agrees) | + +**Behaviour change this creates.** `stream_source: auto` (the default) now falls back to +disk instead of refusing when the base will not fit in RAM. That is what the plan's tier +order specifies, but a *silent* fallback to a slower path is the exact failure mode this +project criticises elsewhere, so the pre-flight note says what happened, that nothing is +held resident, **that the slowdown is unmeasured**, and that `stream_source: 'ram'` is how +an operator asks to be refused instead. A non-NVMe disk is still refused outright. + +Consequence for the v0.72.2 tests: "does not fit in RAM" used to mean an exception and now +means a tier decision. Those controls were **restored intact under `stream_source: 'ram'`** +— the regime where the early size probe is the thing under test — rather than weakened, and +the new tier behaviour is pinned separately in `tests/test_v07203.py` through the real +trainer.