mirror of https://github.com/razor-ai/soup.git
docs(benchmarks): publish the layer-streaming measurement records
Publishes the three gate records behind the layer-streaming feature as benchmarks/, so the numbers in the docs and in the accompanying paper can be checked rather than taken on trust. These are the working records written while each item was built, not a report assembled afterwards: they carry the failures, the assumptions that turned out wrong and the numbers that were measured and then discarded, in the order those things happened. - gate-v0.72.0-layer-streaming.md - the streaming path and its bit-exactness reference - gate-v0.72.2-nf4.md - NF4 quantised streaming; Llama-3.1-8B at 119.6 tok/s in a 3.32 GB peak - gate-v0.72.3-breadth.md - nine architectures, batching, accumulation, resume, disk tier; peak-VRAM predictor at 0.85% worst-case error The README states the hardware once, explains why Windows/WDDM spilling rather than raising OOM means a completed run is not evidence of fit, and why any fraction-of-ceiling is quoted only with the SM clock it was taken at. References to internal planning documents were rewritten for a reader outside the repository.
This commit is contained in:
Alpamys
parent
70d1d783a4
commit
d73eb1c843
|
|
@ -0,0 +1,52 @@
|
|||
# Measurement records
|
||||
|
||||
Raw gate records for Soup's layer-streaming feature, published as written.
|
||||
|
||||
These are not a report assembled after the fact. They are the working records
|
||||
kept while each item was built and verified, so they contain the failures, the
|
||||
assumptions that turned out wrong, and the numbers that were measured and then
|
||||
discarded — in the order those things happened. They are the evidence behind
|
||||
the paper *Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB
|
||||
Laptop GPU*.
|
||||
|
||||
| File | What it gates | Headline |
|
||||
|---|---|---|
|
||||
| [`gate-v0.72.0-layer-streaming.md`](gate-v0.72.0-layer-streaming.md) | The streaming path itself | Bit-exactness vs a resident reference; 3B bf16 trained on a 4 GB card |
|
||||
| [`gate-v0.72.2-nf4.md`](gate-v0.72.2-nf4.md) | NF4 quantised streaming | Llama-3.1-8B at 119.6 tok/s in a 3.32 GB peak |
|
||||
| [`gate-v0.72.3-breadth.md`](gate-v0.72.3-breadth.md) | Nine architectures, batching, accumulation, resume, disk tier | Peak-VRAM predictor at 0.85% worst-case error; accumulation is per-token I/O-neutral |
|
||||
|
||||
## Hardware
|
||||
|
||||
Every number was measured on one machine:
|
||||
|
||||
- **GPU** — RTX 3050 Laptop, 4 GB (4.29 GB usable)
|
||||
- **Host** — 16.9 GB RAM, NVMe
|
||||
- **OS** — Windows 11
|
||||
|
||||
Windows/WDDM matters for reading these: it spills into shared host memory rather
|
||||
than raising `CUDA out of memory`, so a run completing is not evidence that its
|
||||
configuration fits. That is why peak VRAM is reported alongside every throughput
|
||||
figure, and why the fit decision refuses rather than warns.
|
||||
|
||||
## Reading the numbers
|
||||
|
||||
- **Throughput is quoted with the SM clock it was taken at.** This card's boost
|
||||
clock varies about 13% between sessions, so a fraction-of-ceiling stated
|
||||
without its clock is not meaningful. Where a GEMM ceiling is compared against,
|
||||
it was measured in the same session.
|
||||
- **The correctness reference always matches the numerics under test** — a
|
||||
streamed NF4 run is compared against a *resident NF4* run, never against
|
||||
resident bf16, which would hide a real defect inside quantisation error.
|
||||
- **Derived figures are labelled as arithmetic.** Where a line says "1M tokens =
|
||||
2.3 h", that is division, not a measured wall-clock run.
|
||||
|
||||
## Reproducing
|
||||
|
||||
The implementation ships in Soup under Apache-2.0. Reproduction commands are in
|
||||
Appendix A of the paper; the correctness protocol runs as part of the project's
|
||||
test suite, so a regression in bit-exactness fails CI rather than reaching a
|
||||
user.
|
||||
|
||||
```bash
|
||||
pip install 'soup-cli[train]'
|
||||
```
|
||||
|
|
@ -0,0 +1,233 @@
|
|||
<!--
|
||||
Measurement record for Soup layer streaming, published verbatim.
|
||||
|
||||
These are the working gate records written while the feature was built, not a
|
||||
report assembled afterwards: they contain the failures, the corrected
|
||||
assumptions and the numbers that were discarded, in the order they happened.
|
||||
They are the evidence behind the paper "Exact Layer Streaming: LoRA Fine-Tuning
|
||||
of an 8B Model on a 4 GB Laptop GPU".
|
||||
|
||||
Hardware for every number below: RTX 3050 Laptop (4 GB, 4.29 GB usable),
|
||||
16.9 GB host RAM, NVMe, Windows 11, unless a line states otherwise.
|
||||
-->
|
||||
|
||||
# v0.72.0 Layer Streaming — Gate results
|
||||
|
||||
**Status: both gates PASS. v0.72.0 shipped 2026-07-26 (tag `v0.72.0`, PyPI live).
|
||||
Numbers below are final and were reproduced verbatim in the release notes, CHANGELOG and
|
||||
the accompanying report. §"After the gates" records what implementation
|
||||
and review then found — the gates were necessary but not sufficient.**
|
||||
|
||||
Hardware (all numbers on this box, no extrapolation):
|
||||
**Windows 11 · RTX 3050 Laptop 4 GB (CC 8.6, driver 591.44) · 16.9 GB RAM · NVMe · PCIe**
|
||||
torch 2.5.1+cu121 · transformers (llama modeling) · peft 0.18.1 · accelerate 1.12.0 · Python 3.10
|
||||
|
||||
Unit convention: **decimal GB throughout** (16.9 GB host RAM, 7.65 GB page-locked ceiling).
|
||||
Earlier drafts mixed GiB and GB for the same machine — 15.7 GiB and 16.9 GB are the same RAM,
|
||||
7.12 GiB and 7.65 GB the same pinned ceiling.
|
||||
|
||||
---
|
||||
|
||||
## GATE 1 — correctness — **PASS** (2026-07-26)
|
||||
|
||||
Throwaway spike, nothing under `src/`. Model **HuggingFaceTB/SmolLM2-135M**
|
||||
(ungated; 30 layers, d=576, ffn=1536, vocab=49152, tied embeddings), bf16,
|
||||
LoRA r=16 on `q_proj,v_proj` (921,600 trainable), seq 512, batch 1,
|
||||
**double buffering (n=2) + dedicated prefetch stream + events from the first
|
||||
prototype** (plan §2.4).
|
||||
|
||||
Construction actually under test:
|
||||
- checkpoint sharded to `layer_NNN.safetensors` + `extras.safetensors`
|
||||
- `RamSource` — whole base pinned in CPU RAM (212.4 MB), Tier 1
|
||||
- `LayerBufferPool(n=2)` pre-allocated VRAM buffers + `torch.cuda.Event`
|
||||
- skeleton built with `accelerate.init_empty_weights` — **270 decoder-layer
|
||||
weight tensors (30 layers x 9) never leave `meta`**; only embed / final norm /
|
||||
LoRA are materialised. The resident load never happens (plan P14).
|
||||
- `torch.func.functional_call` feeds pooled buffers into the unmodified HF
|
||||
layer (plan §5.3 / P13)
|
||||
- `torch.utils.checkpoint(..., use_reentrant=False)`; base weights carry
|
||||
`requires_grad=False` — no `detach()`, no `no_grad()` (plan P2)
|
||||
|
||||
| # | Check | Threshold | Measured | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| 1 | streamed vs resident logits, max abs diff | < 1e-3 | **0.0 (bit-exact)** | PASS |
|
||||
| 2 | LoRA grad on **layer 0** | != 0 | **9.50e-01**; 30/30 layers non-zero | PASS |
|
||||
| 3 | 100 steps streamed vs resident, loss curve | within noise | **max rel 0.0**; 0.61296 -> 0.22499 on both | PASS |
|
||||
| 4 | same seed twice -> identical loss | identical | **max diff 0.0** | PASS |
|
||||
| 5 | + pinned-CPU boundary offload (`saved_tensors_hooks`) vs resident | < 2% rel | 4.83e-3 rel | PASS |
|
||||
| 6 | buffer-count invariance n=2 vs n=3 | identical | **max diff 0.0** | PASS |
|
||||
|
||||
Peak VRAM during the run: 2.09 GB — but that figure includes the **resident
|
||||
reference model held simultaneously** for comparison, so it is not the
|
||||
streaming footprint. 5788 layer loads served from the pool.
|
||||
|
||||
Notes:
|
||||
- Checks 1/3/4/6 are **bit-exact**, which is the expected result: streaming
|
||||
substitutes the identical weight bytes into the identical kernels. Anything
|
||||
other than 0.0 would have meant a real numerical difference.
|
||||
- Check 5 is the only non-zero: 0.48% relative over 25 steps. Verified **not a
|
||||
race** — the offload path is bit-reproducible across two same-seed runs
|
||||
(`offload_determinism_max_diff = 0.0`), so it is a different-but-valid bf16
|
||||
rounding path amplified by 25 optimizer steps. Async pinned activation
|
||||
offload is **out of v0.72.0 scope** regardless; recorded for v0.72.2.
|
||||
- The `owner[slot]` early-overwrite assert (plan P1 tripwire) never fired.
|
||||
- Two real bugs the gate caught, both of which would have silently broken a
|
||||
naive implementation:
|
||||
1. PEFT's `LoraModel` calls `self.model.forward(...)` **directly**, bypassing
|
||||
`__call__` — so a prefetch pre-hook on the CausalLM wrapper never fires.
|
||||
The hook must go on the module that owns `.layers`.
|
||||
2. PEFT initialises adapters on the base layer's device, which is `meta`
|
||||
here; adapter weights must be explicitly re-materialised (A: kaiming, B: 0).
|
||||
|
||||
---
|
||||
|
||||
## GATE 2 — the number — **> 50 tok/s row of the §8 table** (2026-07-26)
|
||||
|
||||
Common config: bf16 · `task=sft`-shaped loop · LoRA r=16 on `q_proj,v_proj` ·
|
||||
batch 1 · gradient checkpointing ON (so C=6) · `bitsandbytes.PagedAdamW8bit` ·
|
||||
**double buffering n=2 + dedicated prefetch stream** · 50 measured steps after
|
||||
10 warm-up · GPU util from `nvidia-smi dmon -s u` (`sm` column).
|
||||
|
||||
| Model | RAM store | S | tok/s | GPU util (mean/max) | Peak VRAM | RAM store size | TFLOPS eff |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Qwen2.5-0.5B | pinned | 512 | **978.6** | 91.4% / 100% | 1.47 GB | 0.72 GB | 2.90 |
|
||||
| Qwen2.5-1.5B | pinned | 512 | **525.0** | 96.8% / 100% | 1.82 GB | 2.62 GB | 4.86 |
|
||||
| Qwen2.5-1.5B | pinned | 1024 | **487.6** | 96.7% / 100% | 2.96 GB | 2.62 GB | 4.52 |
|
||||
| Qwen2.5-3B | **pageable** | 512 | **143.1** | 79.3% / 100% | 2.15 GB | 5.55 GB | 2.65 |
|
||||
|
||||
Baselines / premise:
|
||||
- **Qwen2.5-3B resident training OOMs on this card** — `CUDA out of memory`
|
||||
on a plain forward+backward at S=512. This is the premise of the feature and
|
||||
it is confirmed, not assumed. Streaming runs the same model in **2.15 GB**.
|
||||
- **Qwen2.5-0.5B resident is a VALID baseline** (peak 3.23 GB, genuinely inside
|
||||
4 GB): resident 1398.0 tok/s vs streamed 978.6 -> **streaming costs 1.43x**.
|
||||
This is the only honest streaming-vs-resident overhead figure here.
|
||||
- **Qwen2.5-1.5B resident is NOT a valid baseline** and its number is discarded:
|
||||
it reported a 6.15 GB "peak" on a 4 GB card, i.e. it spilled into WDDM shared
|
||||
host memory and crawled at 91.3 tok/s. Quoting "streaming is 5.7x faster than
|
||||
resident" from that would be a garbage claim.
|
||||
|
||||
TFLOPS back-check (plan §3): **the plan's assumed 9 TFLOPS peak for this card is
|
||||
wrong**, so the naive back-check mislabels a good number as "IMPLAUSIBLE".
|
||||
Measured on this box: achievable dense bf16 GEMM = **6.76 TFLOPS** (fp32/TF32 =
|
||||
3.38, the expected 2x Ampere ratio); GA107 theoretical dense bf16 tensor peak
|
||||
~24.5 TFLOPS. The 1.5B run's 4.86 TFLOPS is 20% of theoretical — plausible, and
|
||||
corroborated by 96.8% GPU utilisation.
|
||||
|
||||
> **RE-MEASURED 2026-07-27 (v0.72.1 session) — 6.76 is CORRECT; the ceiling tracks GPU CLOCK.**
|
||||
> Re-measuring the same square shapes gave 7.08/7.52/7.66 in one session and 6.23/6.63/6.75 in
|
||||
> another. Repeating 4096^3 six times *within* a session is stable to <1% (6.93-6.94) at a pinned
|
||||
> 862 MHz / 63 C. So the ~13% spread is between-session boost-clock state, not measurement error,
|
||||
> and **6.76 reproduces almost exactly in a low-clock session**. Rule: a ceiling is only comparable
|
||||
> to a throughput measured in the SAME session, and any fraction-of-ceiling must state the SM clock.
|
||||
> Shape-matched weighted ceilings at 862 MHz: 8B 7.58 (high-clock session) / 3B 6.67 / 1.5B 6.39 /
|
||||
> 0.5B 5.90. Numerator convention: decoder C=6 (NOT 8 — the frozen base has no dL/dW),
|
||||
> embed_tokens 0 FLOPs (lookup), lm_head C=4 (outside the checkpointed layers, frozen).
|
||||
> See `gate-v0.72.2-nf4.md`.
|
||||
Every other run is under 32% even of the
|
||||
plan's understated 9.
|
||||
|
||||
**Reading of the §8 decision table: the ">50 tok/s" row — "Mechanism is healthy.
|
||||
Ship v0.72.0 BETA, go straight to v0.72.1 (NF4)."** 143 tok/s at 3B, 525 at
|
||||
1.5B, 979 at 0.5B; GPU utilisation 79–97% everywhere, nowhere near the <30%
|
||||
scheduler-broken or <50% hardware-bound pathologies.
|
||||
|
||||
**Corrected 2026-07-27 — this paragraph originally read "IO is fully hidden".**
|
||||
That is true of the 1.5B row and NOT of the 3B headline row, and the step-level
|
||||
arithmetic (from these same numbers) says so: at 143.1 tok/s and a 5.55 GB store,
|
||||
a step is 3.578 s and moves 2 x 5.55 = 11.10 GB, i.e. an implied 3.1 GB/s, while
|
||||
compute at the measured 6.76 TFLOPS ceiling accounts for only ~1.26 s — about 35%
|
||||
of the step. The 3B run is TRANSFER-bound; its pageable store makes
|
||||
`copy_(non_blocking=True)` synchronous, which is the same fact as the 79.3% util.
|
||||
Per-row compute fraction: 0.5B 31% / 1.5B 61% / 3B 35%. Only the 1.5B row (pinned,
|
||||
5.4 GB/s implied, 96.8% util) is compute-bound as plan §8 predicted.
|
||||
|
||||
`nvidia-smi dmon -s u` reports SM occupancy — "a kernel was resident" — which is
|
||||
necessary but NOT sufficient for the transfer to be hidden. Reading it as proof of
|
||||
overlap is what produced the original wrong sentence.
|
||||
|
||||
### Honesty caveats on these numbers
|
||||
1. **The 3B run used a PAGEABLE store, not a pinned one, so 143 tok/s is a
|
||||
LOWER BOUND.** This box cannot page-lock 5.55 GB alongside an IDE + agent
|
||||
session: measured maximum pinned host allocation is **7.65 GB** (with 9.1 GB
|
||||
"available"), and the 3B attempt failed with a host-side
|
||||
`CUDA error: out of memory` at 9.3 GB available even after the loader was
|
||||
rewritten to remove its transient copies. Pageable memory makes
|
||||
`copy_(non_blocking=True)` synchronous, which is visible as the utilisation
|
||||
drop from 96.8% (pinned, 1.5B) to 79.3% (pageable, 3B). This is exactly the
|
||||
RAM ceiling plan P16 predicted.
|
||||
2. Numbers are Windows/WDDM (plan P15) and therefore systematically pessimistic
|
||||
versus Linux.
|
||||
3. The plan's §4.3 ceiling table for this card is built on the wrong peak
|
||||
figure; do not quote its per-model tok/s.
|
||||
4. Nothing here was measured above 3B. **No 8B/14B claim is supported.**
|
||||
|
||||
Derived, clearly labelled as arithmetic and not measurement:
|
||||
1M training tokens at the measured rates = **1.9 h at 3B**, 0.53 h at 1.5B.
|
||||
|
||||
### Implementation findings the gates produced (feed into Phase C)
|
||||
- The prefetch pre-hook must be attached to the module that **owns `.layers`**,
|
||||
not the CausalLM wrapper: PEFT's `LoraModel.forward` calls
|
||||
`self.model.forward(...)` directly and bypasses `__call__` hooks.
|
||||
- The streamed layer wrapper must be **attribute-transparent** (`__getattr__`
|
||||
delegating to the wrapped layer): this `transformers` reads
|
||||
`decoder_layer.attention_type` straight off the layer object.
|
||||
- PEFT initialises adapter weights on the base layer's device, which is `meta`
|
||||
in a streaming build; adapters must be explicitly re-materialised.
|
||||
- The RAM store must be **allocated once and filled by `copy_`**, not
|
||||
`load_file -> .to() -> .pin_memory()`; the latter's 3 transient copies per
|
||||
layer are what push a large base over the page-locked ceiling.
|
||||
- `expandable_segments:True` (plan P7) is **not supported on Windows** — torch
|
||||
warns and ignores it. Do not rely on it here.
|
||||
|
||||
---
|
||||
|
||||
## After the gates — what a passing gate did NOT catch (2026-07-26/27)
|
||||
|
||||
Recorded because the honest lesson of this release is that **6/6 correctness checks and a
|
||||
clean throughput table still left a bug that would have killed every real run.** The gates
|
||||
tested the streaming mechanism in isolation; they did not test it inside a real trainer.
|
||||
|
||||
1. **CRITICAL — every streaming run would have died at trainer construction.**
|
||||
`transformers.Trainer.__init__` calls `_move_model_to_device` -> `model.to()`, and `.to()`
|
||||
on a module holding `meta` parameters raises `NotImplementedError: Cannot copy out of meta
|
||||
tensor`. The failure lands immediately after the pre-flight prints "Layer streaming ready",
|
||||
so the run looks healthy right up to the crash. 145 green tests missed it because **no test
|
||||
built a real `Trainer`/`SFTTrainer` from the streamed model** — the spike drove its own loop.
|
||||
Fixed with `hf_device_map` + a `StreamedDecoderLayer._apply` override that passes meta
|
||||
tensors through while still moving/casting the real LoRA params. A *second* meta-move inside
|
||||
`accelerate.prepare_model` was then found by the new end-to-end test, not by review.
|
||||
2. **HIGH — the shard cache could silently stream the wrong weights.** It was keyed by model
|
||||
slug with only a dtype check, so a local checkpoint retrained in place (or two ids colliding
|
||||
onto one slug) would train against stale weights with no error. Now keyed by a
|
||||
`source_fingerprint` (per-shard name + size + mtime digest) that invalidates like dtype does.
|
||||
3. **HIGH — DoRA / VeRA / PiSSA / OLoRA read the real base weight at `get_peft_model()` time**
|
||||
and crash opaquely on `meta`. Now refused by name at config-parse.
|
||||
4. **The pre-flight hardware-fit gate refused the runs this feature exists to enable.** It
|
||||
models a RESIDENT run, so it rejected a 3B streaming config outright. Found by the step-6
|
||||
smoke, not by any test. Now skipped for streaming runs, with a passing control test proving
|
||||
resident configs are still gated.
|
||||
|
||||
Shipped scope is narrower than the gates alone would justify: RAM tier · bf16 · `task=sft` ·
|
||||
Llama/Qwen · batch 1 · no gradient accumulation · no `--resume`. Every refusal names the
|
||||
release that lifts it (NF4 -> v0.72.1; disk tier, larger batches, accumulation, resume ->
|
||||
v0.72.2).
|
||||
|
||||
Final test count 16576 -> **16735** (+159 in `tests/test_v07200.py`), 324 test files.
|
||||
|
||||
---
|
||||
|
||||
## Citation check on the write-up (2026-07-27)
|
||||
|
||||
The gate numbers above are ours and were never in doubt. The *third-party* numbers drafted
|
||||
around them for the accompanying report were checked against source and
|
||||
**three were wrong**: MegaTrain listed as a 24 GB/3090 system (it is a single H200 with 1.5 TB
|
||||
host RAM, up to 120B); a "26% GPU utilisation on a 4090" figure attributed to ZeRO-Infinity
|
||||
(which predates the 4090 — not their measurement); and LSP-Offload's overhead given as 1.45×
|
||||
with 1.31× separately credited to gradient checkpointing (the real published figure is a 31%
|
||||
slowdown for LSP-Offload itself — one wrong number, re-used under a second wrong label).
|
||||
|
||||
Consequence for the claim: "every published system floors at 24 GB" was **false** —
|
||||
LSP-Offload (arXiv:2406.10181) already fine-tunes 1.3B on a 4 GB laptop GPU. The defensible
|
||||
headline is **1.3B -> 3B on a 4 GB card**, with bit-exactness verified, not "nobody has done
|
||||
this."
|
||||
|
|
@ -0,0 +1,339 @@
|
|||
<!--
|
||||
Measurement record for Soup layer streaming, published verbatim.
|
||||
|
||||
These are the working gate records written while the feature was built, not a
|
||||
report assembled afterwards: they contain the failures, the corrected
|
||||
assumptions and the numbers that were discarded, in the order they happened.
|
||||
They are the evidence behind the paper "Exact Layer Streaming: LoRA Fine-Tuning
|
||||
of an 8B Model on a 4 GB Laptop GPU".
|
||||
|
||||
Hardware for every number below: RTX 3050 Laptop (4 GB, 4.29 GB usable),
|
||||
16.9 GB host RAM, NVMe, Windows 11, unless a line states otherwise.
|
||||
-->
|
||||
|
||||
# NF4 Layer Streaming — Gate results (now the **v0.72.2** slot)
|
||||
|
||||
> **Filename kept as `v0721-*` deliberately.** These gates were run while NF4 was
|
||||
> v0.72.1. They surfaced a shipped v0.72.0 correctness defect (adapters saved with an
|
||||
> `.inner.` key prefix reload as zero tensors), which was split out to ship alone and
|
||||
> first as **v0.72.1**; NF4 moved to **v0.72.2**. **These results stand as-is and are not
|
||||
> to be re-run** — see plan §7.1. The adapter-key defect is written up below because
|
||||
> this gate is what found it.
|
||||
|
||||
**Status: GATE 1 PASS (6/6, bit-exact vs resident NF4). GATE 2 complete, incl. the 8B headline.**
|
||||
Nothing under `src/` was written before these ran (plan §7.1 rule).
|
||||
|
||||
Hardware (all numbers on this box, no extrapolation):
|
||||
**Windows 11 · RTX 3050 Laptop 4 GB (CC 8.6) · 16.9 GB RAM · NVMe**
|
||||
torch 2.5.1+cu121 · bitsandbytes 0.49.2 · transformers 4.57.6 · peft 0.18.1 ·
|
||||
trl 0.19.1 · accelerate 1.12.0 · Python 3.10.8
|
||||
|
||||
Unit convention: **decimal GB** throughout (matches the v0.72.0 gate doc).
|
||||
|
||||
The reference is **resident NF4**, per plan §7.1 — not resident bf16, which would
|
||||
differ by quantisation error and hide a real bug inside it. Streamed NF4 and
|
||||
resident NF4 hold the *same quantised bytes* and run the *same bitsandbytes
|
||||
kernels*, so the standard is bit-exactness.
|
||||
|
||||
---
|
||||
|
||||
## GATE 1 — correctness — **PASS** (2026-07-27)
|
||||
|
||||
Throwaway spike (`scratchpad/v0721_gate1.py`), nothing under `src/`.
|
||||
Model **HuggingFaceTB/SmolLM2-135M** (llama arch, 30 layers, tied embeddings),
|
||||
NF4 + double quant + bf16 compute — i.e. the repo's own `quant_menu` defaults —
|
||||
LoRA r=16 on `q_proj,v_proj`, seq 512, batch 1, double buffering (n=2) with a
|
||||
dedicated prefetch stream.
|
||||
|
||||
Construction under test (the v0.72.0 engine, plus the P3 work):
|
||||
- checkpoint **pre-quantised offline**, one tensor at a time on the GPU, into
|
||||
per-layer shards of packed `uint8` + `absmax` (+ nested `absmax`/`offset`)
|
||||
- `Params4bit` **views rebuilt over the pooled buffers on every call**, with a
|
||||
`QuantState` reassembled from the streamed tensors + two shared constant code
|
||||
tables
|
||||
- skeleton on `meta` via `init_empty_weights` + `replace_with_bnb_linear`;
|
||||
**270 decoder weight tensors never leave `meta`**
|
||||
- `functional_call` feeds the rebuilt `Params4bit` into the unmodified HF layer
|
||||
- `checkpoint(use_reentrant=False)`; base `requires_grad=False`, no `detach()`
|
||||
|
||||
| # | Check | Threshold | Measured | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| 0 | offline shard bytes vs the resident model's `Params4bit` | identical | **IDENTICAL** (7 weights on layer 0: packed + absmax + nested absmax) | PASS |
|
||||
| 1 | streamed vs resident NF4 logits, max abs diff | < 1e-3 | **0.0 (bit-exact)** | PASS |
|
||||
| 2 | LoRA grad on **layer 0** | != 0 | **4.921e-01**; 30/30 layers non-zero | PASS |
|
||||
| 3 | 25 steps streamed vs resident, loss curve | within noise | **max rel 0.0**; 12.99325 -> 12.09359 on both | PASS |
|
||||
| 4 | same seed twice -> identical loss | identical | **0.0** | PASS |
|
||||
| 5 | buffer-count invariance n=2 vs n=3 | identical | **0.0** | PASS |
|
||||
|
||||
RAM store 0.055 GB pinned; pool 3.7 MB x 2.
|
||||
|
||||
### P3 is solved, and the probe that settles it
|
||||
|
||||
`Params4bit` carries a `quant_state` and quantises on transfer to CUDA, so NF4
|
||||
weights cannot be byte-copied into a plain buffer (plan P3). Measured answers
|
||||
(`scratchpad/probe_nf4_api.py`):
|
||||
|
||||
- `quantize_4bit` is **deterministic and byte-identical** across repeats and
|
||||
across a CPU round-trip — so offline sharding reproduces exactly what a
|
||||
resident load would have produced. Check 0 confirms this end-to-end.
|
||||
- A `Params4bit` **rebuilt over an existing packed buffer** with an explicit
|
||||
`quant_state` (`bnb_quantized=True`) produces a **bit-identical** forward.
|
||||
- `torch.func.functional_call` accepts it, **including over a `meta`
|
||||
placeholder** of a different shape — so the v0.72.0 substitution mechanism
|
||||
carries over unchanged.
|
||||
- Gradient flows to the layer input through `matmul_4bit` with the weight at
|
||||
`requires_grad=False` — plan P2 holds.
|
||||
|
||||
Per-weight streamed bytes (double quant): packed `N/2` + absmax `N/64` +
|
||||
nested absmax `N/4096` + a 4-byte offset ≈ **0.516 bytes/param**. The NF4 code
|
||||
table (16 fp32) and the nested code table (256 fp32) were verified **constant
|
||||
across every weight**, so one shared resident copy is safe — the sharder
|
||||
asserts this rather than assuming it.
|
||||
|
||||
### What the gate caught that a green run would not have
|
||||
|
||||
**1. PEFT silently used a different LoRA math path (found by check 1).**
|
||||
With byte-identical adapters and byte-identical weights, streamed vs resident
|
||||
logits differed by **9.375e-01**. Cause: `transformers.from_pretrained` stamps
|
||||
`is_loaded_in_4bit` on the model, and PEFT reads it to dispatch
|
||||
`lora.bnb.Linear4bit`. A `meta` skeleton has no such marker, so PEFT dispatched
|
||||
the generic `lora.layer.Linear`, which still *runs* against a `Linear4bit` base
|
||||
— it just casts and accumulates differently. No crash, no warning; the loss
|
||||
curve looked healthy. Fixed by stamping the markers `from_pretrained` sets.
|
||||
This is the single most important finding of the gate.
|
||||
|
||||
**2. Two ways the gate itself was nearly vacuous** (harness bugs, fixed before
|
||||
the numbers above were taken):
|
||||
- PEFT initialises `lora_B = 0`, so at step 0 the adapter contributes nothing
|
||||
and `dL/dA` is structurally zero for *any* correct implementation. The first
|
||||
run "passed" check 1 bit-exactly while the adapter path was doing nothing at
|
||||
all, and "failed" check 2 for a reason that had no bearing on streaming. The
|
||||
gate now randomises `lora_B` first, which makes the adapter path load-bearing
|
||||
in checks 1 and 3.
|
||||
- The adapter sync between the two models silently copied **0 tensors** because
|
||||
of the key-prefix bug below, leaving the models with different adapters. It
|
||||
now refuses to run rather than compare two differently-initialised models.
|
||||
|
||||
---
|
||||
|
||||
## Blocking rider found in shipped v0.72.0 — adapters are saved unloadable
|
||||
|
||||
Reproduced through the **shipped** `build_streamed_model`
|
||||
(`scratchpad/check_inner_keys.py`, CPU, tiny-random-Llama):
|
||||
|
||||
`install_streaming` replaces `layers[i]` with a wrapper that holds the real
|
||||
layer as a child named `inner`. Every adapter parameter therefore gains an
|
||||
`.inner.` segment in its state-dict path, and that name is what
|
||||
`get_peft_model_state_dict` -> `save_pretrained` writes to disk:
|
||||
|
||||
```
|
||||
base_model.model.model.layers.0.inner.self_attn.q_proj.lora_A.weight
|
||||
^^^^^^
|
||||
```
|
||||
|
||||
Loading that adapter back into a normal model drops **8 of 8** tensors. PEFT
|
||||
emits a `UserWarning` about missing keys and returns a model that is byte-for-byte
|
||||
the untuned base:
|
||||
|
||||
```
|
||||
reload into a plain model: 0/4 lora_B tensors non-zero
|
||||
VERDICT: ADAPTER SILENTLY LOST ON RELOAD
|
||||
```
|
||||
|
||||
Consequence: **every adapter produced by `soup train --stream-layers` in
|
||||
v0.72.0 is inert outside the streaming path** — `soup merge`, `soup serve`,
|
||||
`soup chat` and `PeftModel.from_pretrained` all load it as a no-op. The
|
||||
training run itself is correct (the gates prove that); only the artifact it
|
||||
writes is unusable. Nothing in the 159 v0.72.0 tests saves an adapter and loads
|
||||
it back.
|
||||
|
||||
This must be fixed in v0.72.1 — it is in the same component the NF4 work
|
||||
modifies, and shipping NF4 on top of an unloadable artifact would compound it.
|
||||
|
||||
**Both candidate fixes were spiked and both work** (`scratchpad/fix_inner_keys_spike.py`,
|
||||
CPU, tiny-random-Llama; control = shipped code):
|
||||
|
||||
| Variant | saved keys with `.inner.` | reload into a plain model |
|
||||
|---|---|---|
|
||||
| shipped v0.72.0 (control) | 8 / 8 | **0/4 adapters — LOST** |
|
||||
| A: transparent `state_dict()` on the wrapper | 0 | 4/4 — round-trips |
|
||||
| B: wrapper shares the inner layer's `_parameters`/`_modules` | 0 | 4/4 — round-trips |
|
||||
|
||||
A is surgical (serialisation only); B removes the nesting level altogether, so
|
||||
`named_parameters()` is canonical too — which also unblocks loading *into* a
|
||||
streamed model later (`--resume`, a v0.72.2 item). Both keep the forward
|
||||
working. Choice to be made at implementation, TDD-first, with the reload
|
||||
round-trip as the spec.
|
||||
|
||||
---
|
||||
|
||||
## GATE 2 — the number
|
||||
|
||||
Protocol unchanged from v0.72.0 so the rows are comparable: tok/s over 50 steps
|
||||
after 10 warm-up · `torch.cuda.max_memory_allocated()` · SM occupancy from
|
||||
`nvidia-smi dmon -s u` · `PagedAdamW8bit` · batch 1 · double buffering · TFLOPS
|
||||
back-check at C=6 against this box's **measured** 6.76 TFLOPS dense bf16 GEMM
|
||||
ceiling (theoretical GA107 peak ~24.5; the plan's "9" is wrong for this card).
|
||||
|
||||
Parameter counts are read from the safetensors headers, so a tied `lm_head` is
|
||||
not double-counted (counting `model.parameters()` on the meta skeleton reported
|
||||
Qwen2.5-3B as 3.40 B instead of 3.09 B and inflated TFLOPS accordingly).
|
||||
|
||||
| Model | Quant | Store | S | tok/s | GPU util | Peak VRAM | TFLOPS eff |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **Llama-3.1-8B** | **NF4** | **3.60 GB pinned** | 512 | **122.5** | **98.9%** / 100% | **3.45 GB** | 5.90 |
|
||||
| Qwen2.5-3B | **NF4** | 1.43 GB **pinned** | 512 | **244.3** | 94.5% / 100% | 1.91 GB | 4.53 |
|
||||
| Qwen2.5-3B | bf16 *(v0.72.0)* | 5.55 GB pageable | 512 | 143.1 | 79.3% / 100% | 2.15 GB | 2.65 |
|
||||
|
||||
### The 8B headline
|
||||
|
||||
**Llama-3.1-8B fine-tunes on a 4 GB card at 122.5 tok/s**, with the base
|
||||
page-locked (3.60 GB store, under the box's measured 7.65 GB pinned ceiling) and
|
||||
peak VRAM 3.45 GB — genuinely inside the card, not a WDDM spill. Untied
|
||||
`embed_tokens` + `lm_head` stay resident and bf16 (2.10 GB of that 3.45 GB), which
|
||||
is why 8B is close to this card's ceiling; treating them as streamed large layers
|
||||
is a v0.72.3 item. NF4 shard set on disk 5.70 GB, sharded once and cached.
|
||||
|
||||
Derived, labelled as arithmetic: 1M training tokens = **2.3 h at 8B**, 1.1 h at 3B.
|
||||
|
||||
### The TFLOPS back-check, resolved — the DENOMINATOR was wrong
|
||||
|
||||
The first pass read 5.90 TFLOPS = **87% of the v0.72.0 gate's 6.76 TFLOPS**
|
||||
"achievable dense bf16 GEMM" figure, which is not credible for an eager loop
|
||||
that also dequantises NF4. It was held back rather than published. Both
|
||||
candidate explanations were then tested:
|
||||
|
||||
**(b) gradient checkpointing off, so C=6 is the wrong constant — DISCONFIRMED.**
|
||||
The streamed layer wraps its body in `checkpoint(use_reentrant=False)` whenever
|
||||
grad is enabled, and the measured loop is a forward+backward. C=6 is correct.
|
||||
|
||||
**(a) the 6.76 ceiling was measured at shapes that do not reach peak — PARTLY,
|
||||
but the dominant effect turned out to be something else: GPU CLOCK STATE.**
|
||||
|
||||
An first re-measure suggested 6.76 was simply an under-measurement (square
|
||||
shapes came back 7.08 / 7.52 / 7.66). A second re-measure in a later session
|
||||
gave **6.23 / 6.63 / 6.75** for the *same* shapes — i.e. 6.76 reproduced almost
|
||||
exactly. Repeating 4096^3 six times inside one session then showed the
|
||||
measurement is stable to **<1%** (6.93, 6.94, 6.93, 6.94, 6.93, 6.94) with the
|
||||
SM clock pinned at **862 MHz / 63 C**.
|
||||
|
||||
So the ~13% spread is *between* sessions and tracks the boost clock, not
|
||||
measurement noise. **6.76 was never wrong; it was measured in a lower-clock
|
||||
session.** The methodological rule that follows is stronger than the original
|
||||
correction: **a ceiling is only comparable to a throughput measured in the same
|
||||
session at the same clock**, and any fraction-of-ceiling should state the clock
|
||||
it was taken at. Shape still matters, but less than clock.
|
||||
|
||||
Shape-matched ceilings, all measured in ONE session at 862 MHz, bf16, B=1/S=512:
|
||||
|
||||
| op | M x K x N | TFLOPS | FLOP share |
|
||||
|---|---|---|---|
|
||||
| q_proj / o_proj | 512 x 4096 x 4096 | 7.71 | 15.4% |
|
||||
| k_proj / v_proj | 512 x 4096 x 1024 | 7.57 | 3.8% |
|
||||
| gate/up_proj | 512 x 4096 x 14336 | 7.49 | 53.8% |
|
||||
| down_proj | 512 x 14336 x 4096 | 7.69 | 26.9% |
|
||||
| **FLOP-weighted** | | **7.58** | |
|
||||
|
||||
| model shapes | weighted ceiling @862 MHz |
|
||||
|---|---|
|
||||
| Llama-3.1-8B | 7.58 *(measured in the earlier high-clock session)* |
|
||||
| Qwen2.5-3B | 6.67 |
|
||||
| Qwen2.5-1.5B | 6.39 |
|
||||
| Qwen2.5-0.5B | 5.90 |
|
||||
|
||||
Smaller models sit lower, as expected — their GEMMs are too small to saturate.
|
||||
|
||||
**The numerator convention, fixed and stated so a reader can reproduce it:**
|
||||
decoder params at **C=6** (2 fwd + 2 recompute + 2 dL/dx). **Not C=8** — the
|
||||
base is frozen, so no dL/dW is computed for it, and that missing term is exactly
|
||||
what C=8 would add. `embed_tokens` contributes **0 FLOPs** (a lookup).
|
||||
`lm_head` runs at **C=4** (2 fwd + 2 dL/dx): it sits outside the checkpointed
|
||||
decoder layers so its forward is not recomputed, and it is frozen.
|
||||
|
||||
| run | GFLOP/token | effective | % of its shape-matched ceiling |
|
||||
|---|---|---|---|
|
||||
| Qwen2.5-0.5B @ 978.6 tok/s | 2.69 | 2.63 | 45% |
|
||||
| Qwen2.5-1.5B @ 525.0 tok/s | 8.79 | 4.62 | **72%** |
|
||||
| Qwen2.5-3B @ 143.1 tok/s | 17.89 | 2.56 | 38% |
|
||||
| Llama-3.1-8B NF4 @ 122.5 tok/s | 43.98 | 5.39 | 71% |
|
||||
|
||||
The 1.5B row being the highest is exactly right: it is the one run measured at
|
||||
96.8% SM occupancy, i.e. the compute-bound one. The 3B bf16 row at 38% is the
|
||||
transfer-bound pageable-store run. The picture is now coherent across all four.
|
||||
|
||||
**Caveat that must travel with these fractions:** the throughputs come from
|
||||
earlier sessions and the ceilings from this one, so each ratio inherits the
|
||||
cross-session clock uncertainty (~13%). Before any of these fractions is
|
||||
published as load-bearing, re-measure throughput and ceiling **back to back in
|
||||
one session** and report the SM clock alongside.
|
||||
|
||||
**Publish tok/s and peak VRAM as the measurements — those are clock-sensitive
|
||||
too but they are what was actually observed. The TFLOPS column is derived; quote
|
||||
it only with the constant, the parameter accounting and the clock stated.**
|
||||
|
||||
**NF4 makes the 3B row 1.71x faster than bf16 streaming, and the reason is not
|
||||
the arithmetic — it is that a 1.43 GB store fits under the page-locked ceiling
|
||||
where a 5.55 GB one did not.** Pinning restores async `copy_`, which is visible
|
||||
as utilisation 79.3% -> 94.5%. This is the v0.72.0 honesty caveat #1 ("the 3B
|
||||
number is a LOWER BOUND because the store went pageable") being lifted by NF4.
|
||||
|
||||
### The resident-NF4 3B baseline was measured and then DISCARDED
|
||||
|
||||
First attempt: 34.1 tok/s, **peak VRAM 6.07 GB on a 4 GB card**, 100% util,
|
||||
0.63 TFLOPS. A 6.07 GB peak on a 4 GB card is a WDDM shared-host-memory spill —
|
||||
the identical failure mode that made v0.72.0 discard its Qwen2.5-1.5B resident
|
||||
row. Publishing it would have produced a **"streaming is 7.2x faster than
|
||||
resident"** headline out of a number that measures Windows paging, not
|
||||
training. It is discarded, not reported.
|
||||
|
||||
It was also not a fair comparison: gradient checkpointing was off on the
|
||||
resident side, so it ran at C=4 while the streamed side runs at C=6 *and* it
|
||||
held every activation, which is what pushed it over the card. The harness now
|
||||
enables checkpointing for resident throughput runs; a re-measurement is queued
|
||||
behind the 8B headline.
|
||||
|
||||
**Consequence to state in the release:** as of this gate there is still **no
|
||||
valid resident baseline at 3B on this box** — resident bf16 OOMs (v0.72.0) and
|
||||
resident NF4 spills. The honest claim remains the one v0.72.0 made at 0.5B, not
|
||||
a 3B speed-up ratio.
|
||||
|
||||
Remaining row pending: the 8B headline.
|
||||
|
||||
---
|
||||
|
||||
## v0.72.2 step-6 reproduction — through the SHIPPED code (2026-07-28)
|
||||
|
||||
The Gate-2 rows above came from the throwaway spike. These were re-measured with
|
||||
`shard_checkpoint` / `build_streamed_model` as released, same protocol (50 steps
|
||||
after 10 warm-up, batch 1, S=512, `PagedAdamW8bit`, double buffering), and the
|
||||
GEMM ceiling taken **in the same session at the stated clock**, per the rule the
|
||||
gate itself established.
|
||||
|
||||
| Model | Quant | Store | tok/s | GPU util | Peak VRAM | SM clock | eff TFLOPS | % of same-session ceiling |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| **Llama-3.1-8B-Instruct** | NF4 | **3.60 GB pinned** | **119.6** | 100% | **3.32 GB** | 952 MHz / 70 C | 5.26 | 68% |
|
||||
| Qwen2.5-3B | NF4 | 1.43 GB pinned | 264.2 | 100% | 1.76 GB | 960 MHz / 72 C | 4.73 | 61% |
|
||||
|
||||
Agreement with the spike is close (8B: 119.6 vs 122.5 tok/s, 3.32 vs 3.45 GB
|
||||
peak, identical 3.60 GB pinned store), i.e. the released implementation
|
||||
reproduces the gate rather than merely resembling it. The residual is inside the
|
||||
~13% between-session clock spread the gate documented.
|
||||
|
||||
Shard cache behaved: the second 8B run reported `shards ready in 0.0s`.
|
||||
|
||||
**The 3B NF4-vs-bf16 comparison, and its caveat.** 264.2 tok/s (NF4) against
|
||||
v0.72.0's 143.1 tok/s (bf16) is 1.85x, but the cause is *not* arithmetic — it is
|
||||
that a 1.43 GB store page-locks where a 5.55 GB one did not, which restores
|
||||
async `copy_` and moves utilisation 79.3% -> 100%. The bf16 row was measured in
|
||||
an earlier session at an unrecorded clock, so the ratio inherits that
|
||||
uncertainty; the mechanism (pinning) is the load-bearing claim, not the factor.
|
||||
|
||||
Derived, labelled as arithmetic: 1M training tokens = **2.3 h at 8B**, 1.05 h at 3B.
|
||||
|
||||
**Also found by step 6, and fixed:** a streamed NF4 model reported
|
||||
878,154,048 parameters for SmolLM2-135M (true 134,515,008) while the resident
|
||||
NF4 path reported 134,975,808. PEFT computes `Params4bit` totals as
|
||||
`numel * 2 * quant_storage.itemsize`, correct for a resident tensor whose numel
|
||||
is the packed count but not for a `meta` placeholder carrying the logical shape.
|
||||
Display-only; the loss curve was byte-identical before and after. At 8B it would
|
||||
have printed ~52 B.
|
||||
|
|
@ -0,0 +1,472 @@
|
|||
<!--
|
||||
Measurement record for Soup layer streaming, published verbatim.
|
||||
|
||||
These are the working gate records written while the feature was built, not a
|
||||
report assembled afterwards: they contain the failures, the corrected
|
||||
assumptions and the numbers that were discarded, in the order they happened.
|
||||
They are the evidence behind the paper "Exact Layer Streaming: LoRA Fine-Tuning
|
||||
of an 8B Model on a 4 GB Laptop GPU".
|
||||
|
||||
Hardware for every number below: RTX 3050 Laptop (4 GB, 4.29 GB usable),
|
||||
16.9 GB host RAM, NVMe, Windows 11, unless a line states otherwise.
|
||||
-->
|
||||
|
||||
# v0.72.3 "Breadth" — per-item gate results
|
||||
|
||||
Inherited standard — **a streamed run must be
|
||||
bit-exact against the resident run of the same numerics**; what changes per item is
|
||||
the *reference*, not the standard. Gradient accumulation is the one item whose gate
|
||||
is a **measured I/O cost**, not an equality (the brief says so explicitly).
|
||||
|
||||
Box: Windows 11 · RTX 3050 Laptop 4.29 GB · 16.9 GB RAM · NVMe · Python 3.10 ·
|
||||
torch 2.5.1+cu121 · transformers 4.57.6 · peft 0.18.1 · trl 0.19.1 · bitsandbytes 0.49.2.
|
||||
|
||||
Gates are throwaway scripts in the session scratchpad; nothing under `src/` was
|
||||
written before a gate passed.
|
||||
|
||||
---
|
||||
|
||||
## GATE 1 — multi-arch (Mistral / Gemma / Phi) — **PASS 14/14** (2026-07-28)
|
||||
|
||||
The brief: *"wrong layer detection streams weights into the wrong module and trains
|
||||
something plausible-but-wrong. Gate = bit-exactness on a small model of each family
|
||||
added, not just the allowlist entry."*
|
||||
|
||||
**Method.** Tiny (3-layer, hidden 32, vocab 64) from-config checkpoints of each
|
||||
family, written to disk as real `config.json` + `model.safetensors`, then:
|
||||
shard → `build_streamed_model` → compare logits against **the same checkpoint loaded
|
||||
resident with the same adapter weights**. CPU, float32. For `quant=nf4` the reference
|
||||
is a **resident NF4 load**, never a resident bf16 one — a bf16 reference differs by
|
||||
quantisation error and would hide a real bug inside it.
|
||||
|
||||
From-config rather than downloaded checkpoints because the module tree comes from the
|
||||
same `modeling_*.py`, so layer detection, name mapping and `replace_with_bnb_linear`
|
||||
conversion are exercised identically — and it stays runnable on a 4 GB box. The risk
|
||||
the brief names is a *naming* property, not a size property.
|
||||
|
||||
**Vacuity defence.** PEFT initialises `lora_B = 0`, so a completely detached adapter
|
||||
is byte-identical to a fresh one and every parity assertion passes for the wrong
|
||||
reason. Every run randomises `lora_B` first and asserts a non-zero number of tensors
|
||||
were copied to the reference.
|
||||
|
||||
Checks per family: (1) logits bit-exact `torch.equal`; (2) layer-0 LoRA gradient
|
||||
non-zero (plan P2 — a severed graph still lowers loss, so nothing else catches it);
|
||||
(3) decoder parameters still on `meta` (or it is not streaming at all).
|
||||
|
||||
| family | quant | max abs logit diff | bit-exact | layer-0 LoRA grad | meta decoder params |
|
||||
|---|---|---|---|---|---|
|
||||
| llama *(control)* | none | 0.000e+00 | yes | 3.2391e+00 | 27 |
|
||||
| llama *(control)* | nf4 | 0.000e+00 | yes | 3.3290e+00 | 27 |
|
||||
| **mistral** | none | 0.000e+00 | yes | 3.2391e+00 | 27 |
|
||||
| **mistral** | nf4 | 0.000e+00 | yes | 3.3290e+00 | 27 |
|
||||
| **gemma** | none | 0.000e+00 | yes | 5.1453e-01 | 27 |
|
||||
| **gemma** | nf4 | 0.000e+00 | yes | 5.2169e-01 | 27 |
|
||||
| **gemma2** | none | 0.000e+00 | yes | 3.8365e+00 | 33 |
|
||||
| **gemma2** | nf4 | 0.000e+00 | yes | 4.0558e+00 | 33 |
|
||||
| **gemma3_text** | none | 0.000e+00 | yes | 4.3802e+00 | 39 |
|
||||
| **gemma3_text** | nf4 | 0.000e+00 | yes | 4.5446e+00 | 39 |
|
||||
| **phi** | none | 0.000e+00 | yes | 2.9908e+00 | 42 |
|
||||
| **phi** | nf4 | 0.000e+00 | yes | 3.0868e+00 | 42 |
|
||||
| **phi3** | none | 0.000e+00 | yes | 3.2312e+00 | 18 |
|
||||
| **phi3** | nf4 | 0.000e+00 | yes | 2.9316e+00 | 18 |
|
||||
|
||||
**Notes.**
|
||||
|
||||
- **`phi3` is the interesting row.** Phi-3 *fuses* Q/K/V into a single `qkv_proj` and
|
||||
gate/up into `gate_up_proj`, so it has no `q_proj`/`v_proj` at all — the LoRA target
|
||||
set had to be `["qkv_proj"]`. It is bit-exact anyway, which is the strongest
|
||||
evidence that layer detection and the shard name map are driven by the real module
|
||||
tree rather than by Llama-shaped assumptions.
|
||||
- **`gemma3` (multimodal) is deliberately NOT added** — only `gemma3_text`. A real
|
||||
`google/gemma-3-*-it` reports `model_type='gemma3'` for the vision-capable wrapper;
|
||||
refusing it is correct, because streaming a multimodal wrapper as if it were a
|
||||
causal LM is exactly the silent-mis-train the allowlist exists to prevent.
|
||||
- Two initial failures were **the gate harness's own fault, not the feature's**:
|
||||
`Phi3Config` defaults `pad_token_id=32000`, out of range for a 64-token fixture
|
||||
vocab. Fixed in the fixture (`pad_token_id=0`) and re-run; recorded here rather than
|
||||
quietly dropped.
|
||||
- `meta decoder params` differs per family because the families have different
|
||||
per-layer parameter counts (gemma2/gemma3 carry extra norms; phi3 fuses, so fewer
|
||||
and larger tensors). All are `> 0`, which is the property being asserted.
|
||||
|
||||
**Verdict: the allowlist may be extended to `mistral`, `gemma`, `gemma2`,
|
||||
`gemma3_text`, `phi`, `phi3`.**
|
||||
|
||||
---
|
||||
|
||||
## GATE 2 — pre-flight VRAM budget (batch- and vocab-aware) — **PASS** (2026-07-28)
|
||||
|
||||
The brief: *"Untied embed + lm_head — 8B has them untied, so two large matrices go
|
||||
resident. Budget them in `estimate_stream_vram` or the 4 GB card OOMs on a model the
|
||||
planner said would fit."* Operator addition: **batch scales the intra-layer transient
|
||||
AND the logits tensor** (`batch × seq × vocab` plus its fp32 upcast), and on a
|
||||
large-vocab model that term is larger than both layer buffers combined.
|
||||
|
||||
**Method — measure first, fit second.** Real streamed forward+backward+step on real
|
||||
cached checkpoints over a (model, batch, seq) grid, recording
|
||||
`torch.cuda.max_memory_allocated()`. The two models were chosen for a **3.1× vocab
|
||||
contrast** (49 152 vs 151 936) so the logits term could be separated from everything
|
||||
else rather than assumed. Nothing was implemented before this ran.
|
||||
|
||||
Card RTX 3050 Laptop 4.29 GB · SM clock 952 MHz (one row throttled to 442 MHz, noted
|
||||
below) · free RAM 0.30–1.21 GB across the grid · pageable store (pinning is a *speed*
|
||||
property and does not move VRAM).
|
||||
|
||||
Solving the two models simultaneously for the per-token cost yields:
|
||||
|
||||
```
|
||||
peak = pool + extras + adapter_params·16 + 13.5 MB
|
||||
+ batch · seq · ( 14·vocab + 2·n_layers·hidden + 4·(hidden + intermediate) )
|
||||
```
|
||||
|
||||
- `14·vocab` — bf16 logits (2) + fp32 upcast (4) + fp32 log-softmax (4) + fp32 grad
|
||||
(4), i.e. exactly what `transformers` `ForCausalLMLoss` holds live. **The shipped
|
||||
`estimate_logits_bytes` used 6** (bf16 + upcast only), a first-principles guess that
|
||||
under-predicts this term by 2.33×.
|
||||
- `2·n_layers·hidden` — the `checkpoint(use_reentrant=False)` boundary save, one bf16
|
||||
copy per layer.
|
||||
- `4·(hidden + intermediate)` — the live transient inside the ONE layer being
|
||||
recomputed (independent of `n_layers`, which is the whole point of streaming).
|
||||
|
||||
| model | B | S | measured GB | predicted GB | err |
|
||||
|---|---|---|---|---|---|
|
||||
| SmolLM2-135M | 1 | 256 | 0.285 | 0.286 | +0.58% |
|
||||
| SmolLM2-135M | 1 | 512 | 0.471 | 0.473 | +0.53% |
|
||||
| SmolLM2-135M | 2 | 512 | 0.843 | 0.848 | +0.59% |
|
||||
| SmolLM2-135M | 4 | 512 | 1.584 | 1.596 | +0.75% |
|
||||
| SmolLM2-135M | 8 | 512 | 3.069 | 3.094 | +0.82% |
|
||||
| Qwen2.5-0.5B | 1 | 256 | 0.920 | 0.924 | +0.47% |
|
||||
| Qwen2.5-0.5B | 1 | 512 | 1.476 | 1.486 | +0.62% |
|
||||
| Qwen2.5-0.5B | 2 | 512 | 2.592 | 2.609 | +0.63% |
|
||||
| Qwen2.5-0.5B | 4 | 512 | 4.819 | 4.854 | +0.73% |
|
||||
| Qwen2.5-0.5B | 8 | 512 | 9.267 | 9.346 | +0.85% |
|
||||
|
||||
**Worst absolute error 0.85%. Worst under-prediction: none — every prediction is ≥ the
|
||||
measurement**, which is the only safe direction for a gate that must refuse configs.
|
||||
|
||||
**Independent check, nothing fitted to it:** the published v0.72.2 Llama-3.1-8B NF4
|
||||
row — different model, different quantisation, different session, untied embeddings —
|
||||
predicts **3.57 GB against a measured 3.32 GB (+7.5%, over)**. The formula was fitted
|
||||
on two models 16–60× smaller and still brackets an 8B NF4 run on the safe side.
|
||||
|
||||
### Two findings that change the implementation
|
||||
|
||||
1. **The logits term dominates, exactly as called.** At Qwen2.5-0.5B B=8 S=512 it is
|
||||
**8.71 GB of the 9.35 GB predicted — 146× the entire buffer pool (0.060 GB)**. A
|
||||
pre-flight that budgeted only weights and buffers would green-light this config.
|
||||
Batch budgeting is therefore not a refinement of the estimator; it is the estimator.
|
||||
|
||||
2. **Windows/WDDM does not OOM — it spills, silently.** The B=8 row allocated
|
||||
**9.27 GB on a 4.29 GB card and raised nothing**, finishing all three steps. So on
|
||||
this platform *"it did not crash"* is **not** evidence that a config fits, and the
|
||||
fit/no-fit direction of the estimator **cannot be validated by observing an OOM
|
||||
here**. What is validated is the *demand* prediction (±0.85%); refusing when demand
|
||||
exceeds the card is then policy applied to a measured quantity, and is documented as
|
||||
such rather than claimed to be empirically OOM-verified on this box. This is the
|
||||
same WDDM shared-memory spill that invalidated the v0.72.0 1.5B "resident baseline".
|
||||
|
||||
One row (Qwen B=4) was taken while the SM clock had dropped to 442 MHz from 952 MHz.
|
||||
Clock does not affect an allocation measurement, so the row stands; it is recorded
|
||||
because the throughput items must not mix clock states.
|
||||
|
||||
### Reserve for the fit decision — also measured
|
||||
|
||||
The plan's `DEFAULT_WORKSPACE_BYTES` of 1 GB is **not** usable as the reserve: charged on
|
||||
top of a 3.57 GB prediction it would refuse the 8B NF4 run that this feature exists to
|
||||
enable and that v0.72.2 actually measured. Measured on this box instead:
|
||||
|
||||
```
|
||||
before CUDA init : 3.460 GB free of 4.294 GB
|
||||
after CUDA init : 3.447 GB free -> context + driver + display = 0.847 GB
|
||||
1.5 GB tensor : allocator 1.500 GB, driver-visible 2.349 GB
|
||||
-> overhead beyond `allocated` = 0.849 GB (stable)
|
||||
```
|
||||
|
||||
So the fit budget is read from `torch.cuda.mem_get_info()` **at pre-flight time** rather
|
||||
than hardcoded — the offset includes desktop/display usage and is therefore a property of
|
||||
the machine, not of the card model. Against this box's 3.445 GB of allocator-visible VRAM
|
||||
the decision is consistent with every measured row: 8B NF4 (3.32 GB) fits, Qwen B=4
|
||||
(4.82 GB) does not.
|
||||
|
||||
### Item 2 — implemented and verified end-to-end
|
||||
|
||||
Real `soup train --stream-layers` on SmolLM2-135M, **batch 2** (the first batch > 1 ever
|
||||
run under streaming): completed exit 0, 18 steps, adapter written, and the saved adapter
|
||||
carries **0 `.inner.` keys with 60 non-zero `lora_B` tensors** — v0.72.1's canonical-key
|
||||
property survives batch > 1.
|
||||
|
||||
Pre-flight panel from that run:
|
||||
|
||||
```
|
||||
peak VRAM ~0.48 GB at batch 2 x seq 256 (logits 0.35 GB)
|
||||
free VRAM 3.46 GB
|
||||
forecast 5685-8361 tok/s — a compute-bound bound, not a promise
|
||||
(from 6.75 TFLOPS measured on this card now @ 862 MHz)
|
||||
```
|
||||
|
||||
The 6.75 TFLOPS @ 862 MHz agrees with the box's independently recorded ~5.9–6.7 TFLOPS at
|
||||
a pinned 862 MHz, i.e. the runtime probe reproduces the known ceiling rather than
|
||||
inventing one.
|
||||
|
||||
The refusal direction, same model at batch 64:
|
||||
|
||||
```
|
||||
a streaming step is predicted to need 12.08 GB of VRAM but only 3.46 GB is free.
|
||||
Streaming bounds the WEIGHTS, not the activations or the logits — lower
|
||||
training.batch_size or data.max_length, both of which scale this linearly.
|
||||
```
|
||||
|
||||
`batch_size: "auto"` remains refused, for a reason that does not expire: it resolves by
|
||||
OOM-probing a **resident** model, which a streaming run never loads.
|
||||
|
||||
---
|
||||
|
||||
## GATE 3 — gradient accumulation — **PASS** (2026-07-28)
|
||||
|
||||
The brief: *"I/O multiplies linearly: every micro-batch re-reads the entire model (plan
|
||||
P9). This is the one place where the 'batch is nearly free' property breaks. **Measure**
|
||||
tok/s at accum 1 / 2 / 4 and publish it; do not assume it is cheap."*
|
||||
|
||||
### Part A — correctness (CPU, float32, tiny Llama)
|
||||
|
||||
The inherited standard. Accumulated **adapter gradients** from a streamed run must be
|
||||
bit-exact against a resident run accumulating the same micro-batches — the prefetcher
|
||||
re-primes on every forward, and if it mis-tracked direction across a micro-batch boundary
|
||||
a stale buffer would produce quietly wrong gradients rather than a crash.
|
||||
|
||||
| accum | grad tensors | max abs grad diff | bit-exact | layer-0 non-zero | prefetch primes |
|
||||
|---|---|---|---|---|---|
|
||||
| 2 | 12 | **0.0** | yes | 4 | 2 |
|
||||
| 4 | 12 | **0.0** | yes | 4 | 4 |
|
||||
|
||||
Per-micro-batch losses are identical to 6 dp on both sides. Primes == accum, i.e. exactly
|
||||
one prime per micro-batch forward, which is the intended scheduling.
|
||||
|
||||
### Part B — the measured I/O cost (CUDA)
|
||||
|
||||
Qwen2.5-0.5B bf16 · S=256 · 50 steps after 10 warm-up · **store 0.72 GB PINNED in every
|
||||
row** (the confound that mattered — pinned vs pageable — is held constant throughout) ·
|
||||
one session, back-to-back.
|
||||
|
||||
| batch | accum | eff. batch | **tok/s** | s / opt-step | layer loads / 1k tok | peak VRAM | free RAM | SM clock |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | 1 | 1 | 556.6 | 0.460 | 175.78 | 0.842 GB | 2.56 GB | 862 MHz |
|
||||
| 1 | 2 | 2 | 543.4 | 0.942 | 175.78 | 0.846 GB | 3.33 GB | 952 MHz |
|
||||
| 1 | 4 | 4 | 540.1 | 1.896 | 175.78 | 0.846 GB | 3.16 GB | 952 MHz |
|
||||
| 2 | 1 | 2 | 1069.1 | 0.479 | 87.89 | 1.320 GB | 3.20 GB | 952 MHz |
|
||||
| 4 | 1 | 4 | **1378.0** | 0.743 | 43.95 | 2.280 GB | 2.66 GB | 960 MHz |
|
||||
| 2 | 2 | 4 | 1094.5 | 0.936 | 87.89 | 1.325 GB | 2.31 GB | 960 MHz |
|
||||
|
||||
**Confirmation pass.** The first row was taken at 862 MHz and the rest at 952–960, a ~10%
|
||||
spread, so the three rows the headline rests on were re-measured **interleaved**
|
||||
(A/B/C/A/B/C, so monotonic drift cannot favour one arm):
|
||||
|
||||
| config | repeat 1 | repeat 2 | spread |
|
||||
|---|---|---|---|
|
||||
| batch 1 / accum 1 | 552.8 | 550.9 | 0.4% |
|
||||
| batch 1 / accum 4 | 551.4 | 553.2 | 0.3% |
|
||||
| batch 4 / accum 1 | 1393.4 | 1393.4 | 0.0% |
|
||||
|
||||
### What the numbers actually say
|
||||
|
||||
1. **Accumulation is per-token I/O-neutral, not linear-cost.** `layer loads / 1k tokens`
|
||||
is **constant at 175.78 across accum 1, 2 and 4**, and tok/s is flat within 3%
|
||||
(556.6 → 543.4 → 540.1; 552.8 vs 551.4 on the interleaved repeat). accum=N does N
|
||||
micro-batches, N model reads and N times the tokens — the ratio does not move. The
|
||||
plan's P9 wording ("multiplies IO linearly") is right per *optimizer step* and
|
||||
misleading per *token*, which is the unit that decides wall-clock.
|
||||
|
||||
2. **The real cost is opportunity cost against raising batch.** At the *same effective
|
||||
batch of 4*: batch 4 / accum 1 delivers **1393.4 tok/s** against batch 1 / accum 4 at
|
||||
**553.2 — a measured 2.52×**, because one weight read is amortised over four times the
|
||||
tokens (43.95 vs 175.78 loads per 1k tokens). Publishing only the accum column would
|
||||
have read as "accumulation is free"; it is free *per token* and expensive *per unit of
|
||||
effective batch*.
|
||||
|
||||
3. **Accumulation's actual value under streaming is that it buys effective batch at
|
||||
constant VRAM**: peak moved 0.842 → 0.846 GB across accum 1→4, while reaching the same
|
||||
effective batch by raising batch cost 0.842 → 2.28 GB. So the guidance the pre-flight
|
||||
should give is: **raise `batch_size` until the VRAM pre-flight refuses, then use
|
||||
accumulation for the rest** — which is exactly the pairing this release now makes
|
||||
possible, since item 2 is what tells the user where that ceiling is.
|
||||
|
||||
4. Batch scaling is real but sub-linear: 556.6 → 1069.1 (1.92×) → 1378.0 (2.48×) for
|
||||
batch 1 → 2 → 4, as the run crosses from I/O-bound into compute-bound.
|
||||
|
||||
### A defect in item 2's own GEMM probe, found by item 3's smoke
|
||||
|
||||
The accumulation smoke printed **3.54 TFLOPS @ 862 MHz** where an earlier run had printed
|
||||
**6.75 TFLOPS @ 862 MHz** — a 2× swing at the same *reported* clock, which would have made
|
||||
the forecast bracket meaningless. Diagnosed rather than accepted:
|
||||
|
||||
| probe size | five repeats (TFLOPS) | spread |
|
||||
|---|---|---|
|
||||
| 2048³ | 3.19, 3.50, 3.89, 3.85, 4.41 | **38%** |
|
||||
| 4096³ | 3.92, 3.97, 3.99, 3.81, 4.16 | 9% |
|
||||
|
||||
At 2048 the sample is too short for the boost clock to engage and the repeats ramp
|
||||
**monotonically upward**, so whichever repeat happens to be first sets the answer. Fixed
|
||||
by moving to 4096³ and taking the **best of 3** repeats. Best-of-N is not cherry-picking
|
||||
here: a *ceiling* has one-sided noise, since contention, a cold clock and thermal
|
||||
throttling can only ever make an achievable rate look slower than it truly is.
|
||||
|
||||
This is also the concrete justification for refusing to compile a per-card TFLOPS constant
|
||||
into the source — the same card, at the same reported clock, differed 2× between sessions.
|
||||
|
||||
### Verified end-to-end
|
||||
|
||||
Real `soup train --stream-layers` on SmolLM2-135M at **batch 2 × accum 4** (effective
|
||||
batch 8): completed exit 0, 5 optimizer steps, adapter written with **0 `.inner.` keys**.
|
||||
The pre-flight printed the measured advisory:
|
||||
|
||||
```
|
||||
! accumulating 4x at batch 2: the base is re-read once per micro-batch. Per token
|
||||
that is free, but reaching effective batch 8 by raising training.batch_size
|
||||
instead measured ~2.5x faster. Accumulation holds peak VRAM flat, so raise
|
||||
batch_size while the budget above allows, then accumulate for the rest.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GATE 4 — checkpoint / resume — **PASS** (2026-07-28)
|
||||
|
||||
The brief: *"silent failure is resuming against a stale shard cache or a mismatched
|
||||
optimizer state. Gate = save mid-run, resume, assert the loss curve CONTINUES rather than
|
||||
restarts, and assert the shard fingerprint is re-verified on resume."*
|
||||
|
||||
### The failure, established before anything was changed
|
||||
|
||||
v0.72.1 fixed the **save** direction only, deliberately — `state_dict()` delegates at the
|
||||
wrapper's own prefix while `named_parameters()` still carries `.inner.`. The load
|
||||
direction was therefore still broken, and measurably so:
|
||||
|
||||
| check | before |
|
||||
|---|---|
|
||||
| saved tensors / of which `.inner.` | 12 / **0** (save side correct) |
|
||||
| adapter tensors in model / **landed** | 12 / **0** |
|
||||
| non-zero `lora_B` in the checkpoint | 6 (so the comparison is not vacuous) |
|
||||
| **resumed losses vs from-scratch losses** | **byte-identical** |
|
||||
| shard fingerprint detects a changed source | already true (v0.72.0) |
|
||||
|
||||
The mechanism: `nn.Module.load_state_dict` narrows the dict **by child name** as it
|
||||
descends. The wrapper's only child is `inner`, so a canonical
|
||||
`...layers.0.self_attn.q_proj.lora_A.weight` matches no child prefix and is dropped.
|
||||
PEFT reported the keys as missing via a `UserWarning` and continued.
|
||||
|
||||
### The fix
|
||||
|
||||
A `_register_load_state_dict_pre_hook` on `StreamedDecoderLayer` that injects
|
||||
`.inner.`-prefixed copies into the (already prefix-narrowed) state dict. The pre-hook runs
|
||||
at the start of the wrapper's own `_load_from_state_dict`, and torch's child loop reads
|
||||
that same dict object afterwards — so the redirected keys are visible when it descends
|
||||
into `inner`. Load-side only: it redirects keys rather than re-parenting the module tree,
|
||||
so the forward path is untouched and **v0.72.0's bit-exactness gates stay valid without
|
||||
being re-run** — the same reasoning that justified the v0.72.1 approach.
|
||||
|
||||
### After, on the production CUDA path
|
||||
|
||||
Driven through the real `PeftModel.load_adapter`, which is what
|
||||
`Trainer._load_from_checkpoint` calls for a PEFT model:
|
||||
|
||||
| check | after |
|
||||
|---|---|
|
||||
| tensors landed | **12 / 12** |
|
||||
| `hf_device_map` preserved | **true** (`{'': 0}` unchanged) |
|
||||
| decoder params still on `meta` | 27 (still streaming, not materialised) |
|
||||
| resumed vs from-scratch loss curve | **differ** — the checkpoint contributes |
|
||||
|
||||
`hf_device_map` is checked explicitly because `install_streaming` sets it so
|
||||
`Trainer._move_model_to_device` skips `.to()` on meta weights; a loader that rewrites it
|
||||
would reintroduce v0.72.0's CRITICAL.
|
||||
|
||||
### Two honest limitations found while gating
|
||||
|
||||
1. **PEFT re-dispatches a CPU-built streamed model.** On a CPU build, `load_adapter` moved
|
||||
every parameter to `cuda:0` and rewrote `hf_device_map` from `{'': 'cpu'}` to
|
||||
`{'base_model': 0}`, breaking the model. It fires only when the device map mentions
|
||||
`"cpu"`, which the production path never does — streaming exists to bound VRAM, so a
|
||||
CPU-built streamed model is a test convenience, not a configuration. The CPU tests
|
||||
therefore exercise the redirection mechanism through `set_peft_model_state_dict` (what
|
||||
`load_adapter` calls internally to place weights), and the full `load_adapter` path is
|
||||
covered by a CUDA-gated test.
|
||||
|
||||
2. **End-to-end `soup train --resume` cannot be demonstrated on this box, for a reason
|
||||
that is not streaming's.** `transformers.check_torch_load_is_safe()` raises unless
|
||||
torch ≥ 2.6 (CVE-2025-32434); this box has torch 2.5.1+cu121, so *every* resume fails
|
||||
here. Proven with a **control**: the identical config with `stream_layers: false`
|
||||
produces the identical error. What is verified is the streaming-specific half — the
|
||||
adapter round-trip and loss continuity — at the library level on CUDA, plus a
|
||||
behavioural CLI test that both `--resume` and `--hf-resume` are no longer refused.
|
||||
|
||||
---
|
||||
|
||||
## GATE 5 — disk-kind detection (the `soup doctor` rider) — **PASS** (2026-07-28)
|
||||
|
||||
v0.72.0 shipped a `choose_tier` that refuses anything but NVMe — wired to a **hardcoded
|
||||
`disk_kind="nvme"`** in the trainer. A guard connected to a constant can never fire, so
|
||||
the refusal existed only on paper.
|
||||
|
||||
Probed on this box: `Get-PhysicalDisk` returns
|
||||
`{"MediaType":"SSD","BusType":"NVMe","Size":512110190592}` — correctly identifying the
|
||||
512 GB NVMe. Two things fell out of the measurement:
|
||||
|
||||
1. **`BusType` must beat `MediaType`.** An NVMe drive reports `MediaType: SSD`; keying on
|
||||
media type alone would classify every NVMe disk as SATA SSD and refuse the disk tier
|
||||
universally.
|
||||
2. **The probe costs 9.04 s cold, ~2.4 s warm** (PowerShell + CIM startup; Windows caches
|
||||
it after the first launch). Too slow to run unconditionally, and it shapes the API:
|
||||
`choose_tier` now takes a **callable**, so the probe runs only when the base does *not*
|
||||
fit in RAM — the case where the answer matters. The result is cached per volume per
|
||||
process. Linux reads `/sys/block/*/queue/rotational` instantly; unknown platforms
|
||||
return `"unknown"`, which `choose_tier` refuses (the safe direction — believing a
|
||||
spinning disk is NVMe costs hours of thrashing, plan P11).
|
||||
|
||||
`soup doctor --disk` reports it:
|
||||
`Disk type │ NVMe — layer streaming can use the disk overflow tier`.
|
||||
|
||||
**Opt-in, following the command's own `--nccl` convention.** Measured: `soup doctor`
|
||||
17.56 s by default, 19.97 s with `--disk`. A first reading of this over-claimed — 9 s was
|
||||
subtracted from the total to infer "roughly doubled", which the direct A/B does not
|
||||
support; the true warm cost is ~2.4 s on a command that already takes ~17.5 s (it imports
|
||||
torch, transformers and friends to build the dependency table). The flag is still the right
|
||||
call, because the cost is paid by every user while only streaming users benefit, and a
|
||||
streaming run probes lazily on its own regardless.
|
||||
|
||||
---
|
||||
|
||||
## GATE 6 — the disk overflow tier — **PASS** (2026-07-28)
|
||||
|
||||
**The limitation, up front rather than at the end: this box cannot measure the RAM-vs-disk
|
||||
performance gap, and this release does not claim one.** Two independent reasons —
|
||||
safetensors memory-maps the shards, so the OS page cache keeps them resident between steps
|
||||
on any machine with spare RAM; and at ~5 effective TFLOPS the NVMe read hides under compute
|
||||
anyway (plan §2.3). What is gated here is correctness, which *is* demonstrable.
|
||||
|
||||
The brief asks for "the same four checks with `DiskSource` substituted". The strongest
|
||||
available reference is the **RAM tier**, not the resident model: both stream through the
|
||||
same buffer pool, the same prefetcher and the same layer wrapper, differing only in where
|
||||
`get(idx, name)` reads from — so any difference is attributable to the source. The
|
||||
resident comparison is kept as well, anchoring the tier to ground truth rather than merely
|
||||
to its sibling.
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| disk vs **RAM tier** logits | **0.0 — bit-exact** |
|
||||
| disk vs **resident** logits | **0.0 — bit-exact** |
|
||||
| layer-0 LoRA gradient | 2.6260 (non-zero, plan P2) |
|
||||
| determinism (rebuild + re-run) | bit-exact |
|
||||
| `store_bytes` (held resident) | **0** — the point of the tier |
|
||||
| `disk_bytes` vs RAM tier `store_bytes` | 148 480 == 148 480 (accounting agrees) |
|
||||
|
||||
**Behaviour change this creates.** `stream_source: auto` (the default) now falls back to
|
||||
disk instead of refusing when the base will not fit in RAM. That is what the plan's tier
|
||||
order specifies, but a *silent* fallback to a slower path is the exact failure mode this
|
||||
project criticises elsewhere, so the pre-flight note says what happened, that nothing is
|
||||
held resident, **that the slowdown is unmeasured**, and that `stream_source: 'ram'` is how
|
||||
an operator asks to be refused instead. A non-NVMe disk is still refused outright.
|
||||
|
||||
Consequence for the v0.72.2 tests: "does not fit in RAM" used to mean an exception and now
|
||||
means a tier decision. Those controls were **restored intact under `stream_source: 'ram'`**
|
||||
— the regime where the early size probe is the thing under test — rather than weakened, and
|
||||
the new tier behaviour is pinned separately in `tests/test_v07203.py` through the real
|
||||
trainer.
|
||||
Loading…
Reference in New Issue