8878eb1aa6
feat(training): v0.32.0 — Training Stability & Auto-Tuning
...
Seven new opt-in flags that turn Soup into the "fast.ai of LLM
fine-tuning" — pre-flight LR range finder, auto warmup schedule,
auto mixed-precision, loss-spike auto-recovery, convergence detector,
VRAM-pressure advisory, and autopilot integration.
* Part A: soup train --find-lr / utils.lr_finder (TypedDict result,
abs-divergence threshold, NaN/Infinity rejection, MAX_NUM_STEPS=10000,
is_under_cwd containment on --find-lr-output).
* Part B: utils.grad_accum (GradAccumMonitor + MAX_ACCUM=1024 cap;
preserves effective batch on recommend()).
* Part C: utils.mixed_precision (KNOWN_PRECISION_QUIRKS map, longest-
substring iteration so qwen2.5/qwen2 + phi-3.5/phi-3 are deterministic;
200-char model-name cap, null-byte rejection).
* Part D: utils.warmup (compute_warmup_steps clamped [10, 1000];
ratio==0 short-circuit matches HF Trainer "no warmup" convention).
warmup_auto field reuses pre-existing warmup_ratio (no duplicate).
* Part E: utils.spike_recovery (frozen dataclass policy; max_attempts<=10;
min_lr floor) + schema cross-validator requiring loss_watchdog=true.
* Part F: utils.convergence (detect_plateau + recommend_action; the
latter reuses the former so plateau heuristic stays single-source).
* Part G: autopilot.decide_warmup / decide_mixed_precision wrappers;
generate_config validates BOTH the YAML output path AND embedded
decisions["output"] via shared utils.paths.is_under_cwd.
Tests: tests/test_auto_tuning.py — 89 tests covering bound boundaries,
NaN/Infinity rejection, multi-version quirk ordering, frozen-dataclass
post-construction validation, plateau non-positive-mean guard, and
double-containment in generate_config.
Total: 3607 -> 3696 tests passing. ruff clean.
Review wave: python-review (8 findings), security-review (3), code-review
(8 incl. duplicate warmup_ratio HIGH and synthetic stub-loss curve), and
tdd-guide (11 coverage gaps) — every finding fixed before commit.
Live in-process wiring (LR-sweep training loop, spike rollback, grad-accum
DataLoader rebuild, SFT precision push) is deferred to v0.32.1 — same
advisory pattern as v0.30.0 --auto-quant / structured-output.
Stale-install gotcha: if `soup version` shows the old version after pulling
this branch, run `python -m pip install -e . --force-reinstall --no-deps`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 15:24:31 +05:00
Alpamys