mirror of https://github.com/razor-ai/soup.git
docs: refresh quant-menu modality + multipack sharding notes (v0.71.19)
- performance-and-quantization.md: the Quant Menu multi-trainer note said
"vision / audio modality is still SFT-only inline-BNB (wiring tracked as a
follow-up)" — stale after v0.71.19 #81 dropped the modality gate. Now states
vision/audio thread the unified loader (full gptq/awq/hqq/aqlm/eetq/mxfp4/fp8
menu), with the upstream class+kernel caveat.
- peft-and-efficiency.md: added the v0.71.19 #80 multi-GPU sharding paragraph to
the Multipack section (accelerator.prepare + BatchSamplerShard under
num_processes>1; identical bin seed across ranks; single-GPU unchanged).
Docs-only — no version bump / tag (the v0.71.19 code already shipped at f51331d).
This commit is contained in:
parent
f51331d637
commit
853b348898
|
|
@ -60,6 +60,8 @@ training:
|
|||
|
||||
**Live wiring** — landed. SFT and Pretrain trainer wrappers actually instantiate the multipack subclass when `multipack: true` is set. The factory's `get_train_dataloader` override installs `MultipackBatchSampler(real_batches=False)` (yields a flat `list[int]` per packed sequence — DataLoader-compatible) as the DataLoader's `batch_sampler=`, forwarding `dataloader_drop_last`/`num_workers`/`pin_memory` from `TrainingArguments`. The `_get_train_sampler` override stays as a defensive no-op fallback that always delegates to super, so any HF eval / prediction loop bypassing `get_train_dataloader` still gets the correct `Sampler[int]` shape (no nested-list shape mismatch). Multipack is **sft / pretrain only** on the `transformers` backend; preference / RLHF trainers and MLX backend get distinct error messages naming the actual reason. Datasets must expose `input_ids` (preferred) or `length` per row; raw text triggers an all-zeros warning.
|
||||
|
||||
**Multi-GPU sharding (v0.71.19).** Under FSDP / DeepSpeed ZeRO / DDP (`num_processes > 1`) the `get_train_dataloader` override routes the multipack DataLoader through `accelerator.prepare`, so accelerate's `BatchSamplerShard` round-robins whole FFD-packed bins to each rank (preserving the packing; the bin seed is identical across ranks so every rank agrees on the global order before sharding). The single-GPU path returns the raw DataLoader unchanged. Multi-GPU correctness is mocked-tested — a real 2+-GPU validation run is tracked QA.
|
||||
|
||||
**DoS hardening** — the FFD packer caps at 1M items (algorithm is O(N²) worst-case); the 4D mask builder caps allocations at 2³¹ cells; the chat-template Jinja analyzer caps at 128KB. Every numeric input rejects `bool` explicitly (matches v0.30.0+ project policy).
|
||||
|
||||
The `JinjaTemplateAnalyzer` (also v0.37.0) walks chat-template ASTs to discover non-standard `message.<field>` references (`tool_calls`, `name`, `weight`, `train`) — used by the v0.36.0 `train_on_messages_with_train_field` path so per-message training masks are aware of fields beyond `role` / `content`. The analyzer parses templates without rendering them, so a crafted `soup.yaml` cannot trigger SSRF.
|
||||
|
|
|
|||
|
|
@ -204,8 +204,11 @@ trainers (SFT / DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel /
|
|||
Pretrain / Embedding / BCO). PPO's reward model also loads with the same Quant
|
||||
Menu config as the policy when `tcfg` is passed in, so a GPTQ-policy + GPTQ-reward
|
||||
run does not silently OOM in fp16. MLX backend is rejected with a distinct error
|
||||
message; vision / audio modality is still SFT-only inline-BNB (multi-modal
|
||||
Quant Menu wiring tracked as a follow-up).
|
||||
message; vision and audio modality now thread the same unified Quant Menu loader
|
||||
(the `modality: text` gate was dropped in v0.71.19), so the full menu —
|
||||
`gptq` / `awq` / `hqq:*` / `aqlm` / `eetq` / `mxfp4` / `fp8` — applies to
|
||||
multi-modal SFT too (a given vision/audio checkpoint still needs a class + kernel
|
||||
that supports the chosen format, e.g. `autoawq` for awq).
|
||||
|
||||
|
||||
## Activation Offloading (Small-VRAM Large-Batch)
|
||||
|
|
|
|||
Loading…
Reference in New Issue