Alpamys
|
b7abd2a40c
|
feat(v0.27.0): Multi-GPU Mastery — topology, ZeRO++, FSDP2+compile, MII, PP, recipes
- Topology detector: NVLink/PCIe sniffing via NVML (utils/topology.py);
--gpus auto|N on soup train prints the exact accelerate-launch command
with topology info; NCCL env hints applied via os.environ.setdefault so
user/launcher overrides always win.
- Accelerate launcher: build_accelerate_argv with shlex.quote on script
args; num_processes bounded, mixed_precision Literal allowlist,
num_machines bounded [1, 256]; is_in_distributed() recognizes torchrun
+ accelerate env markers (utils/launcher.py).
- ZeRO++: new DeepSpeed preset (zero++/zero_pp aliases) with hierarchical
partitioning + quantized weights + quantized gradients; int(1e9) for
sub_group_size / stage3_max_* so DeepSpeed strict JSON accepts.
- FSDP2 + torch.compile: training.use_fsdp2_compile bool wired into
TrainingArguments(torch_compile=True) via extracted
apply_fsdp_training_kwargs helper in utils/fsdp.py; validator requires
FSDP + CUDA + backend=transformers + torch>=2.2/accelerate>=0.27, and
rejects DeepSpeed+compile coexistence (cryptic runtime crash prevention).
- DeepSpeed-MII backend scaffold: soup serve --backend mii registered with
dependency check; live pipeline wiring deferred to v0.27.1 with explicit
non-zero exit to prevent silent mis-start (utils/mii.py).
- Pipeline parallelism config: training.parallelism Literal[data|pipeline]
+ pipeline_stages bounded [1, 16]; validator enforces stages >= 2 +
CUDA + gpu_count >= stages; execution wiring deferred to v0.27.1 with
Rich Panel notice (utils/pipeline.py).
- Recipes: llama3-70b-fsdp2, qwen3-32b-zeropp, deepseek-v3-pipeline; recipe
count 43 -> 46.
- Tests: 2511 -> 2585 (+74). New test_multi_gpu.py covers topology,
launcher, ZeRO++, FSDP2+compile helper behavior, DeepSpeed+compile
mutual exclusion, MII key-absent + stub branches, pipeline config
bounds, NCCL setdefault semantics, and CLI validator gating (asserts
use_fsdp2_compile / parallelism=pipeline on CPU blocks soup train with
specific error text).
- Reviews: 4-agent initial wave + 3-agent re-review wave on the wiring
delta. All CRITICAL/HIGH/MEDIUM/LOW findings fixed (no remaining debt).
Security review clean.
Known limitations (tracked as v0.27.1 issues):
- Auto-reexec of accelerate-launch (currently advisory)
- Live DeepSpeed-MII server pipeline
- Live pipeline-parallel execution
- Multi-node accelerate config
- Recipe CI smoke-train validation
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-04-21 14:54:34 +05:00 |