mirror of https://github.com/razor-ai/soup.git
08343f8f35
Layer streaming (v0.72.0) was bf16-only, capping it near 3B on a small card. Quantising the streamed base to NF4 makes the RAM store ~4x smaller, which is what brings 8B within reach. Measured on a 4 GB RTX 3050 Laptop through the shipped code (50 steps after 10 warm-up, batch 1, S=512, PagedAdamW8bit, GEMM ceiling taken in the same session): Llama-3.1-8B-Instruct 119.6 tok/s 3.32 GB peak 3.60 GB pinned 100% 952 MHz Qwen2.5-3B 264.2 tok/s 1.76 GB peak 1.43 GB pinned 100% 960 MHz 3B is 1.85x the bf16 path, but that is PINNING, not arithmetic: 1.43 GB page-locks where 5.55 GB did not, restoring async copy_ (util 79.3% -> 100%). 14B was not run — its store exceeds this box's measured 7.12 GB pinned ceiling. A streamed NF4 run is bit-exact against a RESIDENT NF4 run, now as CPU-runnable CI tests rather than only a gate result. Notable, because each fails silently: - PEFT dispatches lora.bnb.Linear4bit only when is_loaded_in_4bit is stamped; without it the generic lora.layer.Linear runs against a Linear4bit base and casts differently (9.375e-01 logit divergence, no warning). Pinned by a test with a control that deletes the marker. - hf_quantizer must be stamped too, or Trainer.__init__ dies formatting its own "cannot fine-tune" error. Found by the end-to-end test. - The shard cache is keyed on quant/double_quant/quant_device as well as dtype and source fingerprint; a bf16 cache reused for an NF4 request would feed full-precision bytes to matmul_4bit. - index.json is a trust boundary: its shape/blocksize reach bnb kernels that do not bounds-check, so from_json validates and the runtime cross-checks the claim against the bytes on disk. - A streamed NF4 model over-reported parameters ~6.5x (878,154,048 vs 134,515,008 for SmolLM2-135M). Display-only; ~52 B at 8B. Scope unchanged and still BETA: RAM tier, sft, Llama/Qwen, batch 1, no accumulation, no resume. quantization values other than none/4bit are refused. Tests: 16752 -> 16840 (+88 in tests/test_v07202.py). Full suite: 16734 passed, 129 skipped. |
||
|---|---|---|
| .. | ||
| README.md | ||
| adapters-and-governance.md | ||
| backends-and-ops.md | ||
| commands.md | ||
| compliance.md | ||
| data.md | ||
| evaluation.md | ||
| models.md | ||
| peft-and-efficiency.md | ||
| performance-and-quantization.md | ||
| serving-and-export.md | ||
| training.md | ||
README.md
Soup Documentation
The main README is the 5-minute front door. This directory holds the full
feature reference — every soup capability, grouped by area.
| Guide | Covers |
|---|---|
| Training tasks & methods | SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors, reward-verifier synthesis |
| PEFT, long context & efficiency | DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning, depth pruning + distill-heal (soup shrink) |
| Performance & quantization | QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP |
| Data engineering | Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs |
| Evaluation & probes | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, soup ship verdict, post-train X-ray probes, A/B, drift, tunability, soup advise |
| Serving & export | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
| Adapters, registry & governance | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (soup loop), knowledge editing, steering, supply-chain controls |
| Compliance & governance quickstart | HIPAA/SOC2/EU-AI-Act/SR-11-7 init templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (soup card), CI gate (soup ci init) |
| Backends, platform & ops | MLX/Unsloth backends, Modal cloud GPU training, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
| Command reference | The full soup command list |
| Supported models & extras | Recommended model families, the VRAM size guide, the pip extras matrix |
Per-release notes live on the GitHub Releases page; see also the repo-root CHANGELOG.md.