mirror of https://github.com/razor-ai/soup.git
e406688f2d
Closes #56, #57, #58, #59. #56 Live --find-lr in-process LR-sweep: - New utils/lr_finder.run_lr_sweep(model, dataloader, schedule, optimizer_factory, device): per-step LR mutation + forward + backward, records loss until exhaustion or NaN/Inf divergence. - commands/train.py wires it via _live_lr_sweep_from_config (loads model + tokenizer + first N rows of cfg.data.train), with synthetic-curve fallback so users without GPU/torch still get a parseable report. #57 Loss-spike recovery hint: - SoupTrainerCallback gains spike_recovery / spike_recovery_max_attempts / spike_recovery_lr_decay; on watchdog fire writes output_dir/spike_recovery.json with previous_lr, recommended_lr (per SpikeRecoveryStrategy.compute_new_lr), should_recover, attempts. A wrapper / re-launch can resume with the decayed LR. Live optimizer rewind is intentionally NOT done — HF Trainer has no safe public API for mid-loop optimizer-state mutation; the JSON hint is the contract. #58 auto_mixed_precision push to TrainingArguments: - New SFTTrainerWrapper._resolve_mixed_precision: when tcfg.auto_mixed_precision is True, queries torch.cuda compute capability and calls pick_mixed_precision(base, cc) to set bf16=/fp16= flags. CPU short-circuits to (False, False). When the flag is False, legacy default preserved (bf16=cuda). #59 Grad-accum advisory (Phase 1): - SoupTrainerCallback gains grad_accum_auto_tune / grad_accum_pressure_threshold / grad_accum_total_vram_gb / grad_accum_current_steps / grad_accum_current_batch. - on_log samples torch.cuda.max_memory_allocated each step; if GradAccumMonitor.should_adjust crosses the threshold once, prints (batch, accum) -> (new_batch, new_accum) advisory and short-circuits (one-shot). Phase 2 (live DataLoader rebuild) needs a small TRL upstream PR — tracked as a known limitation. Wiring: - soup_cli/trainer/sft.py: _resolve_mixed_precision helper, batch_size preserved on self, SoupTrainerCallback constructor passes through new spike + grad-accum knobs. - soup_cli/monitoring/callback.py: rich Console import added (was previously module-relative); spike + grad-accum state fields and one-shot helpers. Tests: +15 in tests/test_part_e.py covering the LR-sweep loop with mocked model + optimizer (records, divergence break), mixed-precision resolver across cpu/cuda + auto-flag combinations + qwen2 fp16 quirk on Ampere, spike recovery hint write + attempts increment + disabled no-op, grad-accum advisory one-shot semantics + threshold + cuda-absent + disabled. Known limitations (release notes): - #57 spike recovery is a JSON hint, not in-process optimizer rewind - #59 Phase 2 (live DataLoader rebuild on advisory) deferred Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| autopilot | ||
| cans | ||
| commands | ||
| config | ||
| data | ||
| eval | ||
| experiment | ||
| migrate | ||
| monitoring | ||
| recipes | ||
| registry | ||
| trainer | ||
| ui | ||
| utils | ||
| __init__.py | ||
| __main__.py | ||
| cli.py | ||