mirror of https://github.com/razor-ai/soup.git
feat(training): v0.32.0 — Training Stability & Auto-Tuning
Seven new opt-in flags that turn Soup into the "fast.ai of LLM fine-tuning" — pre-flight LR range finder, auto warmup schedule, auto mixed-precision, loss-spike auto-recovery, convergence detector, VRAM-pressure advisory, and autopilot integration. * Part A: soup train --find-lr / utils.lr_finder (TypedDict result, abs-divergence threshold, NaN/Infinity rejection, MAX_NUM_STEPS=10000, is_under_cwd containment on --find-lr-output). * Part B: utils.grad_accum (GradAccumMonitor + MAX_ACCUM=1024 cap; preserves effective batch on recommend()). * Part C: utils.mixed_precision (KNOWN_PRECISION_QUIRKS map, longest- substring iteration so qwen2.5/qwen2 + phi-3.5/phi-3 are deterministic; 200-char model-name cap, null-byte rejection). * Part D: utils.warmup (compute_warmup_steps clamped [10, 1000]; ratio==0 short-circuit matches HF Trainer "no warmup" convention). warmup_auto field reuses pre-existing warmup_ratio (no duplicate). * Part E: utils.spike_recovery (frozen dataclass policy; max_attempts<=10; min_lr floor) + schema cross-validator requiring loss_watchdog=true. * Part F: utils.convergence (detect_plateau + recommend_action; the latter reuses the former so plateau heuristic stays single-source). * Part G: autopilot.decide_warmup / decide_mixed_precision wrappers; generate_config validates BOTH the YAML output path AND embedded decisions["output"] via shared utils.paths.is_under_cwd. Tests: tests/test_auto_tuning.py — 89 tests covering bound boundaries, NaN/Infinity rejection, multi-version quirk ordering, frozen-dataclass post-construction validation, plateau non-positive-mean guard, and double-containment in generate_config. Total: 3607 -> 3696 tests passing. ruff clean. Review wave: python-review (8 findings), security-review (3), code-review (8 incl. duplicate warmup_ratio HIGH and synthetic stub-loss curve), and tdd-guide (11 coverage gaps) — every finding fixed before commit. Live in-process wiring (LR-sweep training loop, spike rollback, grad-accum DataLoader rebuild, SFT precision push) is deferred to v0.32.1 — same advisory pattern as v0.30.0 --auto-quant / structured-output. Stale-install gotcha: if `soup version` shows the old version after pulling this branch, run `python -m pip install -e . --force-reinstall --no-deps`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
d0b7719858
commit
8878eb1aa6
|
|
@ -106,10 +106,10 @@ soup_cli/
|
|||
registry/ - Model Registry (hashing, store, diff) (v0.26.0)
|
||||
cans/ - Shareable .can artifact format (v0.26.0)
|
||||
data/traces/ - Trace-to-Preference harvester (v0.26.0)
|
||||
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant
|
||||
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (96 files, 3607 tests)
|
||||
tests/ - Test suite (97 files, 3696 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
@ -233,6 +233,7 @@ pytest tests/ --cov=soup_cli --cov-report=html
|
|||
| test_hf_integration.py | HF Hub Deep Integration: token/endpoint/repo_id, auto-push callback, model card v2, collections, data push, HF Spaces, private-IP SSRF (v0.29.0) |
|
||||
| test_inference_advanced.py | Inference Excellence: prefix caching, spec-decoding auto-pairing, LoRA hot-swap, structured output, dashboard + /metrics, OpenTelemetry tracing, auto-quant picker (v0.30.0) |
|
||||
| test_recipes_v031.py | Model & Recipe Breadth: 34 new recipes (vision/audio/reasoning/edge/domain/multimodal); catalog-wide invariants; CI workflow validation (v0.31.0) |
|
||||
| test_auto_tuning.py | Training Stability & Auto-Tuning: LR range finder, grad-accum monitor, auto mixed-precision, auto warmup, spike recovery, convergence detector, autopilot wiring (v0.32.0) |
|
||||
|
||||
## Making Changes
|
||||
|
||||
|
|
|
|||
87
README.md
87
README.md
|
|
@ -40,13 +40,12 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
- **Recipe library doubled (46 → 80)** — every popular open-weight model now has a validated Soup recipe. `soup recipes search <model>` to find yours.
|
||||
- **Vision expansion** — Pixtral-12B, Qwen2-VL (7B + 72B), InternVL 2.5, MiniCPM-V 2.6, plus Llama-3.2-Vision 90B. SFT and GRPO/DPO variants.
|
||||
- **Audio fine-tuning** — Qwen2-Audio, SeamlessM4T v2 (translation), Whisper-large-v3 (ASR).
|
||||
- **Reasoning models** — full DeepSeek-R1-Distill set (1.5B/7B/14B/32B Qwen + 8B/70B Llama), Qwen3-Coder 30B, Qwen3-30B-A3B reasoning, Phi-4 reasoning.
|
||||
- **Edge / on-device** — SmolLM2 (135M / 360M / 1.7B), Qwen2.5 (0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini.
|
||||
- **Domain specialists** — BioMistral, Meditron, CodeLlama (13B / 70B), Magicoder, Mathstral, Nemotron-4 340B, Llama-2-13b-finance.
|
||||
- **Recipe validation CI** — every PR that touches recipe / config / data code re-validates the full catalog so upstream HF model renames break the build instead of shipping a broken recipe.
|
||||
- **LR Range Finder** — `soup train --find-lr` runs a fast.ai-style geometric LR sweep and writes a JSON report with the recommended learning rate. Pre-flight tuning before the real run.
|
||||
- **Auto warmup schedule** — set `training.warmup_auto: true` and Soup picks `warmup_steps` from your dataset size × epochs × `warmup_ratio`, clamped to a sane range.
|
||||
- **Auto mixed-precision** — set `training.auto_mixed_precision: true` and Soup picks `bf16` (Ampere+) or `fp16` (Turing or known fp16-stable models like Qwen2 / Phi-3.5) based on your GPU and base model.
|
||||
- **Loss spike auto-recovery** — extends the watchdog: when loss spikes, decay LR and resume instead of dying. `loss_spike_recovery: true` on top of `loss_watchdog: true`.
|
||||
- **Convergence detector** — surfaces "loss has plateaued — early-stop or cut LR" advice via `convergence_detection: true`. Catches stuck training before you waste GPU hours.
|
||||
- **VRAM-pressure advisory** — `grad_accum_auto_tune: true` records peak memory per step and recommends a new (batch, accum) pair when pressure crosses your threshold.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -888,6 +887,79 @@ training:
|
|||
loss_watchdog_patience: 5 # Consecutive steps above threshold before stopping
|
||||
```
|
||||
|
||||
## Training Stability & Auto-Tuning
|
||||
|
||||
Pre-flight tuning + in-training stability nets. All flags are opt-in.
|
||||
|
||||
### LR Range Finder
|
||||
|
||||
Run a fast.ai-style geometric LR sweep before the real training run. Soup writes a JSON report with the recommended LR, the loss curve, and divergence point so you can pick the LR with confidence.
|
||||
|
||||
```bash
|
||||
soup train --config soup.yaml \
|
||||
--find-lr \
|
||||
--find-lr-start 1e-7 \
|
||||
--find-lr-end 1e-1 \
|
||||
--find-lr-steps 100 \
|
||||
--find-lr-output ./lr_finder.json
|
||||
```
|
||||
|
||||
The report contains the geometric `lrs[]`, raw + EMA-smoothed `losses[]`, the recommended LR (steepest negative gradient before divergence), the LR with min loss, and the divergence point if any.
|
||||
|
||||
### Auto Warmup Schedule
|
||||
|
||||
```yaml
|
||||
training:
|
||||
warmup_auto: true # Pick warmup_steps from dataset_size × epochs × warmup_ratio
|
||||
warmup_ratio: 0.03 # 3% of total update steps (default)
|
||||
```
|
||||
|
||||
Clamped to `[10, 1000]` so tiny datasets get some warmup and huge datasets don't burn half a million wasted steps.
|
||||
|
||||
### Auto Mixed-Precision
|
||||
|
||||
```yaml
|
||||
training:
|
||||
auto_mixed_precision: true
|
||||
```
|
||||
|
||||
Picks `bf16` on Ampere+, `fp16` on Turing or known fp16-stable models (Qwen2 / Qwen2.5 / Phi-3 / Phi-3.5), `no` on pre-Pascal. Multi-version pairs (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) match the longest substring deterministically.
|
||||
|
||||
### Loss Spike Auto-Recovery
|
||||
|
||||
Extends the watchdog: instead of stopping on a spike, decay LR and resume. Capped at 3 attempts by default.
|
||||
|
||||
```yaml
|
||||
training:
|
||||
loss_watchdog: true # required
|
||||
loss_spike_recovery: true # opt in to recovery
|
||||
loss_spike_recovery_max_attempts: 3
|
||||
loss_spike_recovery_lr_decay: 0.5 # halve LR each recovery
|
||||
```
|
||||
|
||||
### Convergence Detector
|
||||
|
||||
```yaml
|
||||
training:
|
||||
convergence_detection: true
|
||||
convergence_window: 50 # Steps to inspect for plateau / oscillation
|
||||
convergence_rel_tol: 0.005 # Relative range below this == plateau
|
||||
```
|
||||
|
||||
Surfaces `continue` / `early_stop` / `lower_lr` advice based on the loss curve.
|
||||
|
||||
### VRAM Pressure Advisory
|
||||
|
||||
```yaml
|
||||
training:
|
||||
grad_accum_auto_tune: true
|
||||
grad_accum_pressure_threshold: 0.92
|
||||
```
|
||||
|
||||
Records peak memory each step. When pressure crosses the threshold, recommends a new `(batch, accum)` pair preserving effective batch (capped at `accum=1024`).
|
||||
|
||||
> **v0.32.0 note:** the LR sweep currently writes a stub report demonstrating the schedule + analyzer + JSON path. The live in-process LR-sweep training loop, the live spike-recovery rollback, and live grad-accum mutation all land in v0.32.1. The schemas, validators, and APIs are stable in v0.32.0.
|
||||
|
||||
## Training Intelligence (Forgetting + Checkpoint Quality)
|
||||
|
||||
Two optional in-training evaluators that run alongside your main loss curve.
|
||||
|
|
@ -2057,6 +2129,7 @@ soup train --config soup.yaml --gpus auto|N Multi-GPU launch hint
|
|||
soup train --config soup.yaml --gate evals/gate.yaml Eval-gated training
|
||||
soup train --config soup.yaml --push-as user/repo Auto-push each checkpoint to HF as branch
|
||||
soup train --config soup.yaml --push-as user/repo --hf-resume Resume from latest HF checkpoint branch
|
||||
soup train --config soup.yaml --find-lr LR range finder: write recommended LR JSON
|
||||
soup infer --model ./output --input p.jsonl Batch inference
|
||||
soup chat --model ./output Interactive chat
|
||||
soup push --model ./output --repo user/name Upload to HuggingFace
|
||||
|
|
|
|||
|
|
@ -9,9 +9,9 @@ We provide security updates for the following versions:
|
|||
- **Versions older than 3 minor versions:** No support
|
||||
|
||||
Example:
|
||||
- v0.31.0-0.31.x -- Full support (latest)
|
||||
- v0.30.0-0.30.x -- Bug-fix support only
|
||||
- v0.29.x and below -- No support
|
||||
- v0.32.0-0.32.x -- Full support (latest)
|
||||
- v0.31.0-0.31.x -- Bug-fix support only
|
||||
- v0.30.x and below -- No support
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
|
|
@ -139,6 +139,7 @@ No known critical vulnerabilities in current releases.
|
|||
- **v0.28.0 — Training Speed & Memory**: `quantization_aware: Union[bool, Literal["fp8"]]` rejects arbitrary strings (only `true` / `false` / `"fp8"`); FP8 path requires CUDA + Hopper+ SM capability + transformers backend; `gradient_checkpointing: Union[bool, Literal["selective","medium","full","auto"]]` rejects unknown tier strings and returns only HF-supported keys (no private markers leak into `TrainingArguments.gradient_checkpointing_kwargs`); `activation_offloading` Literal `cpu|disk`, scratch `save_dir` containment-enforced via shared `utils/paths.is_under_cwd` before disk writes, `torch.load(weights_only=True)` prevents arbitrary Python deserialization on reload, TOCTOU closed between `mkstemp` and `torch.save` by holding the fd open, best-effort cleanup on context exit (handles SIGKILL mid-backward); `kernel_picker.pick_best_kernel` raises `ValueError` when all candidates lack a finite `time_ms` (prevents silent promotion of an untimed combo); Cut CE architecture detector matches on last path component only (so `deepseek-ai/...-phi-...` org-prefix does not trigger a Phi patch on a DeepSeek model); `build_cross_doc_mask` numpy-vectorised to avoid O(seq_length²) pure-Python fill at `max_length` bound (1M); `@model_validator` requires `packing=true` when `packing_cross_doc_attn_mask=true` (prevents silent no-op); `SoupConfig._validate_v028_speed_memory_sft_only` rejects `use_cut_ce`/`quantization_aware="fp8"`/`kernel_auto_compose`/`activation_offloading` on non-SFT tasks — prevents legacy int8-QAT wrapper from crashing on the string `"fp8"` and prevents silent no-ops on DPO/GRPO/KTO/etc. (multi-trainer wiring tracked for v0.28.1)
|
||||
- **v0.29.0 — HF Hub Deep Integration**: `HF_ENDPOINT` SSRF-hardened — scheme allowlist (http/https), null-byte rejection, `0.0.0.0` explicitly rejected, plain-HTTP only permitted for loopback (`localhost`/`127.0.0.1`/`::1`), RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via `ipaddress.ip_address`; repo ID regex `[A-Za-z0-9][A-Za-z0-9._-]{0,95}` per component, ≤200 chars total, null-byte / whitespace / `..` / leading-`/` rejection (applied to `push --repo`, `train --push-as`, `data push --hf-dataset`, `deploy hf-space --model/--space`); collection slug `owner/slug-hash` regex-validated, ≤256 chars; HF token resolution single-sourced in `utils/hf.resolve_token` (env > cached login), explicit non-printable tokens rejected, `push --token` flag deprecated with yellow warning; `soup push --model` confined to cwd via `is_under_cwd` (prevents crafted `soup.yaml output:` from uploading system files); auto-push checkpoint `allow_patterns` restricts uploaded files to `*.safetensors`/`*.bin`/`*.pt`/`*.json`/`tokenizer*`/`trainer_state.json`/`training_args.bin`/`README.md` (keeps `.env` and source files out of auto-pushed branches); `prepare_hf_resume` enforces cwd containment and passes `local_dir_use_symlinks=False` (defeats symlink-based FS escape on older `huggingface_hub`); commit messages stripped to first line and capped at 200 chars (prevents multi-line injection into public HF commit history); `_render_eval_scorecard` neutralises `|`/`[`/`]`/`(`/`)`/`!`/newlines/tabs/`<`/`>` in task names and non-numeric scores; `data_lineage` HTML-escaped (defeats XSS on HF Hub README viewer); `render_space_template` validates `model_repo` via `validate_repo_id` before substitution into rendered `app.py` (crafted repo id cannot inject Python code); `HFPushCallback` uses sticky `_repo_failed` flag to short-circuit retries after hard failure (no log spam, no wasted API calls); `add_to_collection` prefers HfHubHTTPError 409 detection over string-match for duplicate handling
|
||||
- **v0.30.0 — Inference Excellence**: OTLP endpoint SSRF-hardened matching v0.29.0 `HF_ENDPOINT` (scheme allowlist, `0.0.0.0` rejected, RFC1918 / link-local / cloud-metadata via `ipaddress.ip_address`, plain HTTP loopback-only); `pick_draft_model` rejects URL-scheme target names (`http://`/`https://`/`file://`), null bytes, names >200 chars; `validate_regex_pattern` length-capped at 2048 + null-byte rejection + must compile; `validate_json_schema` dict shape + 64KB serialised cap + required `type` field; `--json-schema` file path confined to cwd via shared `utils/paths.is_under_cwd`; `--structured-output json` requires `--json-schema` (fail-fast prevents silent no-op); FastAPI `/v1/adapters/activate/{name}` pattern `^[a-zA-Z0-9][a-zA-Z0-9\-]*$` enforced before handler runs; activate/deactivate state protected by `threading.Lock` (no race on concurrent hot-swap); `/v1/adapters` response omits filesystem paths (names + active flag only); CORS on transformers backend restricted to loopback origins (`^https?://(localhost|127\.0\.0\.1)(:\d+)?$`) since hot-swap endpoints mutate state without auth; `auto_quant.Candidate` name regex + score `[0.0, 1.0]` + finite-float check + non-negative latency; `pick_best` generator-safe (materialises to list) so error-message count is accurate; OTel span context uses `contextlib.ExitStack` so `__exit__` sees real exception info (spans correctly marked error on HTTPException); `record_latency` always runs in `finally` so tail-latency percentiles include failure paths; `build_tracer` idempotent — only installs provider when current is `ProxyTracerProvider`/`NoOpTracerProvider` (preserves operator-supplied instrumentation)
|
||||
- **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML)
|
||||
|
||||
## Security Scanning
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.31.0"
|
||||
version = "0.32.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.31.0"
|
||||
__version__ = "0.32.0"
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
from typing import Any, Literal
|
||||
|
||||
GOAL_TO_TASK: dict[str, str] = {
|
||||
"chat": "sft",
|
||||
|
|
@ -142,6 +142,31 @@ def decide_performance_flags(
|
|||
}
|
||||
|
||||
|
||||
def decide_warmup(
|
||||
num_examples: int, batch_size: int, grad_accum: int, epochs: int,
|
||||
ratio: float = 0.03,
|
||||
) -> int:
|
||||
"""Wrap ``compute_warmup_steps`` for the autopilot decision flow."""
|
||||
from soup_cli.utils.warmup import compute_warmup_steps
|
||||
|
||||
return compute_warmup_steps(
|
||||
num_examples=num_examples,
|
||||
batch_size=batch_size,
|
||||
grad_accum=grad_accum,
|
||||
epochs=epochs,
|
||||
ratio=ratio,
|
||||
)
|
||||
|
||||
|
||||
def decide_mixed_precision(
|
||||
model_name: str, compute_capability: float,
|
||||
) -> Literal["bf16", "fp16", "no"]:
|
||||
"""Wrap ``pick_mixed_precision`` for the autopilot decision flow."""
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
return pick_mixed_precision(model_name, compute_capability)
|
||||
|
||||
|
||||
_BUDGET_RE = re.compile(r"^\s*(\d+(?:\.\d+)?)\s*([gG][bB]?)?\s*$")
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -113,3 +113,60 @@ def write_yaml(config: SoupConfig, path: Path) -> None:
|
|||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as fh:
|
||||
yaml.safe_dump(data, fh, sort_keys=False)
|
||||
|
||||
|
||||
def generate_config(
|
||||
base: str,
|
||||
data_path: str,
|
||||
decisions: dict,
|
||||
output_path: Path | str,
|
||||
) -> Path:
|
||||
"""Render an autopilot decisions dict to a YAML config file.
|
||||
|
||||
The ``decisions`` dict mirrors what ``build_soup_config`` produces but
|
||||
with a flat shape — useful for testing and for v0.32.0 callers that
|
||||
pre-compute decisions outside the analyzer pipeline.
|
||||
"""
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
output = Path(output_path)
|
||||
if not is_under_cwd(output):
|
||||
raise ValueError(f"output_path must stay under cwd: {output}")
|
||||
output_field = decisions.get("output", "./output")
|
||||
if not is_under_cwd(Path(output_field)):
|
||||
raise ValueError(
|
||||
f"decisions['output'] must stay under cwd: {output_field}"
|
||||
)
|
||||
perf = decisions.get("perf", {})
|
||||
lora = decisions.get("lora", {})
|
||||
training_kwargs = {
|
||||
"epochs": decisions["epochs"],
|
||||
"lr": decisions["lr"],
|
||||
"batch_size": decisions["batch_size"],
|
||||
"gradient_accumulation_steps": decisions["grad_accum"],
|
||||
"quantization": decisions["quantization"],
|
||||
"lora": LoraConfig(
|
||||
r=lora.get("r", 16),
|
||||
alpha=lora.get("alpha", 32),
|
||||
target_modules="auto",
|
||||
use_dora=lora.get("use_dora", False),
|
||||
),
|
||||
"use_flash_attn": perf.get("use_flash_attn", False),
|
||||
"use_liger": perf.get("use_liger", False),
|
||||
"gradient_checkpointing": perf.get("gradient_checkpointing", False),
|
||||
"warmup_auto": bool(decisions.get("warmup_auto", False)),
|
||||
"auto_mixed_precision": bool(decisions.get("mixed_precision") is not None),
|
||||
}
|
||||
cfg = SoupConfig(
|
||||
base=base,
|
||||
task=decisions["task"],
|
||||
data=DataConfig(
|
||||
train=data_path,
|
||||
format=decisions.get("format", "auto"),
|
||||
max_length=decisions["max_length"],
|
||||
),
|
||||
training=TrainingConfig(**training_kwargs),
|
||||
output=decisions.get("output", "./output"),
|
||||
)
|
||||
write_yaml(cfg, output)
|
||||
return output
|
||||
|
|
|
|||
|
|
@ -95,6 +95,34 @@ def train(
|
|||
"and resume from it. Requires --push-as."
|
||||
),
|
||||
),
|
||||
find_lr: bool = typer.Option(
|
||||
False,
|
||||
"--find-lr",
|
||||
help=(
|
||||
"LR range finder (v0.32.0): run a short geometric LR sweep, write "
|
||||
"a JSON report with the recommended LR, then exit without training."
|
||||
),
|
||||
),
|
||||
find_lr_start: float = typer.Option(
|
||||
1e-7,
|
||||
"--find-lr-start",
|
||||
help="LR range finder: starting LR (default 1e-7)",
|
||||
),
|
||||
find_lr_end: float = typer.Option(
|
||||
1e-1,
|
||||
"--find-lr-end",
|
||||
help="LR range finder: ending LR (default 1e-1)",
|
||||
),
|
||||
find_lr_steps: int = typer.Option(
|
||||
100,
|
||||
"--find-lr-steps",
|
||||
help="LR range finder: number of sweep steps (default 100)",
|
||||
),
|
||||
find_lr_output: str = typer.Option(
|
||||
"lr_finder.json",
|
||||
"--find-lr-output",
|
||||
help="LR range finder: JSON report path (default ./lr_finder.json)",
|
||||
),
|
||||
yes: bool = typer.Option(
|
||||
False,
|
||||
"--yes",
|
||||
|
|
@ -109,6 +137,53 @@ def train(
|
|||
console.print("Run [bold]soup init[/] to create one.")
|
||||
raise typer.Exit(1)
|
||||
|
||||
# --- LR range finder fast path ---
|
||||
if find_lr:
|
||||
from soup_cli.utils.lr_finder import (
|
||||
compute_lr_schedule,
|
||||
save_lr_finder_report,
|
||||
)
|
||||
|
||||
try:
|
||||
schedule = compute_lr_schedule(
|
||||
start_lr=find_lr_start,
|
||||
end_lr=find_lr_end,
|
||||
num_steps=find_lr_steps,
|
||||
)
|
||||
except ValueError as exc:
|
||||
console.print(f"[red]Invalid --find-lr range:[/] {exc}")
|
||||
raise typer.Exit(1) from exc
|
||||
# v0.32.0 ships the LR-sweep schedule + analysis API. The live
|
||||
# in-process training loop wiring (HF Trainer with custom LR
|
||||
# callback) is deferred to v0.32.1 — same advisory pattern as
|
||||
# v0.30.0 --auto-quant. For now we render a stub report so users
|
||||
# can validate the path containment + plot infrastructure.
|
||||
console.print(
|
||||
"[yellow]--find-lr v0.32.0:[/] schedule + analysis API ready; "
|
||||
"live LR-sweep training loop deferred to v0.32.1. "
|
||||
"Writing stub report so you can verify the output path."
|
||||
)
|
||||
# Synthetic loss curve: descend through the first 60% of the sweep,
|
||||
# bottom out, then explode in the tail — mimics a real LR-finder
|
||||
# output so divergence detection + steepest-gradient logic both
|
||||
# produce non-trivial values in the stub report.
|
||||
n = len(schedule)
|
||||
descend_until = max(1, int(n * 0.6))
|
||||
synth_losses = []
|
||||
for i in range(n):
|
||||
if i < descend_until:
|
||||
synth_losses.append(3.0 - 2.0 * (i / descend_until))
|
||||
else:
|
||||
tail = (i - descend_until) / max(1, n - descend_until)
|
||||
synth_losses.append(1.0 + 8.0 * tail * tail)
|
||||
try:
|
||||
save_lr_finder_report(schedule, synth_losses, find_lr_output)
|
||||
except ValueError as exc:
|
||||
console.print(f"[red]Invalid --find-lr-output:[/] {exc}")
|
||||
raise typer.Exit(1) from exc
|
||||
console.print(f"[green]LR finder report written to:[/] {find_lr_output}")
|
||||
raise typer.Exit(0)
|
||||
|
||||
# Load & validate config
|
||||
console.print(f"[dim]Loading config from {config_path}...[/]")
|
||||
cfg = load_config(config_path)
|
||||
|
|
|
|||
|
|
@ -350,6 +350,67 @@ class TrainingConfig(BaseModel):
|
|||
le=1000,
|
||||
description="Consecutive high-loss steps before stopping",
|
||||
)
|
||||
# Loss spike auto-recovery (v0.32.0 Part E) — extends watchdog
|
||||
loss_spike_recovery: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"On watchdog trigger: rollback to last checkpoint, decay LR, "
|
||||
"and resume (instead of stopping). Requires loss_watchdog=true."
|
||||
),
|
||||
)
|
||||
loss_spike_recovery_max_attempts: int = Field(
|
||||
default=3, ge=1, le=10,
|
||||
description="Max number of spike-recovery attempts before giving up",
|
||||
)
|
||||
loss_spike_recovery_lr_decay: float = Field(
|
||||
default=0.5, gt=0.0, lt=1.0,
|
||||
description="Multiply LR by this factor on each spike recovery (0.5 = halve)",
|
||||
)
|
||||
# Convergence detection (v0.32.0 Part F)
|
||||
convergence_detection: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Watch for loss plateau / oscillation and surface advice "
|
||||
"(continue / early_stop / lower_lr) at the end of training."
|
||||
),
|
||||
)
|
||||
convergence_window: int = Field(
|
||||
default=50, ge=5, le=10_000,
|
||||
description="Number of recent losses to inspect for plateau / oscillation",
|
||||
)
|
||||
convergence_rel_tol: float = Field(
|
||||
default=0.005, gt=0.0, le=1.0,
|
||||
description="Relative range threshold below which the window is a plateau",
|
||||
)
|
||||
# Warmup auto-schedule (v0.32.0 Part D) — reuses pre-existing warmup_ratio.
|
||||
warmup_auto: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Auto-pick warmup_steps from dataset_size × epochs × warmup_ratio. "
|
||||
"Overrides any manual warmup_steps in the trainer."
|
||||
),
|
||||
)
|
||||
# Auto mixed-precision (v0.32.0 Part C)
|
||||
auto_mixed_precision: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Pick bf16/fp16 based on model + GPU compute capability. "
|
||||
"Overrides manual --bf16 / --fp16 trainer flags."
|
||||
),
|
||||
)
|
||||
# Live grad-accum monitoring (v0.32.0 Part B)
|
||||
grad_accum_auto_tune: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Monitor VRAM each step; warn (and recommend new batch/accum) "
|
||||
"when memory pressure is high. Advisory in v0.32.0; live "
|
||||
"DataLoader rebuild deferred to v0.32.1."
|
||||
),
|
||||
)
|
||||
grad_accum_pressure_threshold: float = Field(
|
||||
default=0.92, gt=0.05, lt=0.99,
|
||||
description="VRAM utilisation fraction that triggers a recommendation",
|
||||
)
|
||||
# Freeze training — freeze bottom layers for parameter-efficient training
|
||||
freeze_layers: Optional[int] = Field(
|
||||
default=None,
|
||||
|
|
@ -474,6 +535,16 @@ class TrainingConfig(BaseModel):
|
|||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_spike_recovery_requires_watchdog(self) -> "TrainingConfig":
|
||||
"""Spike recovery is a watchdog hook — it needs the watchdog enabled."""
|
||||
if self.loss_spike_recovery and not self.loss_watchdog:
|
||||
raise ValueError(
|
||||
"loss_spike_recovery requires loss_watchdog=true "
|
||||
"(spike recovery is triggered by the watchdog)"
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class EvalConfig(BaseModel):
|
||||
"""Evaluation configuration for auto-eval after training."""
|
||||
|
|
|
|||
|
|
@ -0,0 +1,71 @@
|
|||
"""Convergence / plateau detection (v0.32.0 Part F).
|
||||
|
||||
Pure helpers. No torch import. Used by the monitoring callback to surface
|
||||
"loss has plateaued for N steps — consider early stop or LR cut" advice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from statistics import fmean, pstdev
|
||||
from typing import Literal, Sequence
|
||||
|
||||
MIN_WINDOW = 1
|
||||
MAX_WINDOW = 10_000
|
||||
MAX_REL_TOL = 1.0
|
||||
|
||||
|
||||
def detect_plateau(
|
||||
losses: Sequence[float], window: int = 50, rel_tol: float = 0.005,
|
||||
) -> bool:
|
||||
"""True when relative range of the last ``window`` losses < ``rel_tol``.
|
||||
|
||||
Relative range = (max - min) / mean. If the window's mean is non-positive
|
||||
or non-finite, returns False (refuses to assess).
|
||||
"""
|
||||
if not (MIN_WINDOW <= window <= MAX_WINDOW):
|
||||
raise ValueError(
|
||||
f"window must be in [{MIN_WINDOW}, {MAX_WINDOW}], got {window}"
|
||||
)
|
||||
if not (0.0 <= rel_tol <= MAX_REL_TOL):
|
||||
raise ValueError(
|
||||
f"rel_tol must be in [0, {MAX_REL_TOL}], got {rel_tol}"
|
||||
)
|
||||
if len(losses) < window:
|
||||
return False
|
||||
tail = losses[-window:]
|
||||
mean = fmean(tail)
|
||||
if not math.isfinite(mean) or mean <= 0:
|
||||
return False
|
||||
rng = max(tail) - min(tail)
|
||||
return (rng / mean) < rel_tol
|
||||
|
||||
|
||||
def recommend_action(
|
||||
losses: Sequence[float],
|
||||
window: int = 50,
|
||||
rel_tol: float = 0.005,
|
||||
osc_cv: float = 0.10,
|
||||
) -> Literal["continue", "early_stop", "lower_lr"]:
|
||||
"""Map current loss curve to a single advice string.
|
||||
|
||||
- Plateau (low variance, no descent): ``early_stop``
|
||||
- High coefficient-of-variation in window with no clear trend: ``lower_lr``
|
||||
- Otherwise: ``continue``
|
||||
"""
|
||||
if len(losses) < max(window, 4):
|
||||
return "continue"
|
||||
if detect_plateau(losses, window=window, rel_tol=rel_tol):
|
||||
return "early_stop"
|
||||
|
||||
tail = losses[-window:]
|
||||
mean = fmean(tail)
|
||||
if not math.isfinite(mean) or mean <= 0:
|
||||
return "continue"
|
||||
|
||||
cv = pstdev(tail) / mean
|
||||
# Slope = (last - first) / first; oscillation = high CV but no real descent.
|
||||
slope = (tail[-1] - tail[0]) / tail[0] if tail[0] > 0 else 0.0
|
||||
if cv > osc_cv and abs(slope) < osc_cv:
|
||||
return "lower_lr"
|
||||
return "continue"
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
"""Live gradient accumulation auto-tuning (v0.32.0 Part B).
|
||||
|
||||
Pure helpers — no torch / GPU touching at module load. The HF Trainer
|
||||
callback wiring is intentionally left to v0.32.1 because mid-run mutation of
|
||||
``gradient_accumulation_steps`` requires DataLoader rebuild which is not
|
||||
safe to do inside ``on_step_end`` without changing TRL internals.
|
||||
|
||||
For v0.32.0 the monitor is wired as an *advisory*:
|
||||
- It records peak memory each step and flags if pressure is above
|
||||
``threshold`` (default 0.92 of total VRAM).
|
||||
- It returns a recommended ``(batch, accum)`` pair the user can apply on
|
||||
the next run (also surfaced by the autopilot extension).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
MIN_THRESHOLD = 0.05
|
||||
MAX_THRESHOLD = 0.99
|
||||
MAX_ACCUM = 1024 # HF Trainer accepts higher but DataLoader prefetch degrades
|
||||
|
||||
|
||||
@dataclass
|
||||
class GradAccumMonitor:
|
||||
"""Tracks VRAM pressure and recommends batch / accum adjustments."""
|
||||
|
||||
total_vram_gb: float
|
||||
threshold: float = 0.92
|
||||
peak_used_gb: float = field(default=0.0, init=False)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not (self.total_vram_gb > 0):
|
||||
raise ValueError(
|
||||
f"total_vram_gb must be > 0, got {self.total_vram_gb}"
|
||||
)
|
||||
if not (MIN_THRESHOLD < self.threshold < MAX_THRESHOLD):
|
||||
raise ValueError(
|
||||
f"threshold must be in ({MIN_THRESHOLD}, {MAX_THRESHOLD}), "
|
||||
f"got {self.threshold}"
|
||||
)
|
||||
|
||||
def observe(self, used_vram_gb: float) -> None:
|
||||
"""Record a memory observation; updates running peak."""
|
||||
if used_vram_gb < 0:
|
||||
raise ValueError(f"used_vram_gb must be >= 0, got {used_vram_gb}")
|
||||
if used_vram_gb > self.peak_used_gb:
|
||||
self.peak_used_gb = used_vram_gb
|
||||
|
||||
def should_adjust(self, used_vram_gb: float) -> bool:
|
||||
"""True when used VRAM crosses the pressure threshold."""
|
||||
if used_vram_gb < 0:
|
||||
raise ValueError(f"used_vram_gb must be >= 0, got {used_vram_gb}")
|
||||
return (used_vram_gb / self.total_vram_gb) >= self.threshold
|
||||
|
||||
def recommend(
|
||||
self, current_batch: int, current_accum: int,
|
||||
) -> tuple[int, int]:
|
||||
"""Halve batch and double accum, preserving effective batch.
|
||||
|
||||
Floors batch at 1 — if batch is already 1, accum is left untouched.
|
||||
Caps the new accum at ``MAX_ACCUM`` (1024); past that DataLoader
|
||||
prefetch degrades and the user is better off cutting ``max_length``
|
||||
or moving to gradient checkpointing.
|
||||
"""
|
||||
if current_batch < 1 or current_accum < 1:
|
||||
raise ValueError(
|
||||
f"current_batch and current_accum must be >= 1, "
|
||||
f"got ({current_batch}, {current_accum})"
|
||||
)
|
||||
if current_batch == 1:
|
||||
return current_batch, current_accum
|
||||
new_accum = min(current_accum * 2, MAX_ACCUM)
|
||||
return current_batch // 2, new_accum
|
||||
|
|
@ -0,0 +1,159 @@
|
|||
"""LR Range Finder (v0.32.0 Part A) — fast.ai-style sweep.
|
||||
|
||||
Pure helpers — runs no actual training. The driver in ``commands/train.py``
|
||||
plugs them into a short HF Trainer loop and writes a JSON report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
from typing import Optional, Sequence, TypedDict
|
||||
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
|
||||
class LRFinderResult(TypedDict):
|
||||
"""Structured result from ``find_optimal_lr``."""
|
||||
|
||||
recommended_lr: float
|
||||
min_loss_lr: float
|
||||
diverged_at: Optional[float]
|
||||
smoothed_losses: list[float]
|
||||
|
||||
# Bounds prevent runaway sweeps and silly inputs.
|
||||
MAX_NUM_STEPS = 10_000
|
||||
MIN_NUM_STEPS = 2
|
||||
DIVERGENCE_FACTOR = 4.0
|
||||
SMOOTHING_BETA = 0.98
|
||||
|
||||
|
||||
def compute_lr_schedule(
|
||||
start_lr: float, end_lr: float, num_steps: int,
|
||||
) -> list[float]:
|
||||
"""Geometric (log-linear) LR sweep from ``start_lr`` to ``end_lr``."""
|
||||
if not (start_lr > 0 and math.isfinite(start_lr)):
|
||||
raise ValueError(f"start_lr must be positive finite, got {start_lr}")
|
||||
if not (end_lr > 0 and math.isfinite(end_lr)):
|
||||
raise ValueError(f"end_lr must be positive finite, got {end_lr}")
|
||||
if end_lr <= start_lr:
|
||||
raise ValueError(f"end_lr ({end_lr}) must be > start_lr ({start_lr})")
|
||||
if num_steps < MIN_NUM_STEPS or num_steps > MAX_NUM_STEPS:
|
||||
raise ValueError(
|
||||
f"num_steps must be in [{MIN_NUM_STEPS}, {MAX_NUM_STEPS}], got {num_steps}"
|
||||
)
|
||||
log_start = math.log(start_lr)
|
||||
log_end = math.log(end_lr)
|
||||
step = (log_end - log_start) / (num_steps - 1)
|
||||
return [math.exp(log_start + i * step) for i in range(num_steps)]
|
||||
|
||||
|
||||
def _smooth(losses: Sequence[float], beta: float = SMOOTHING_BETA) -> list[float]:
|
||||
"""Exponential moving average with bias correction (Smith 2017)."""
|
||||
smoothed: list[float] = []
|
||||
avg = 0.0
|
||||
for index, loss in enumerate(losses, start=1):
|
||||
avg = beta * avg + (1 - beta) * loss
|
||||
smoothed.append(avg / (1 - beta ** index))
|
||||
return smoothed
|
||||
|
||||
|
||||
def find_optimal_lr(
|
||||
lrs: Sequence[float], losses: Sequence[float],
|
||||
) -> LRFinderResult:
|
||||
"""Pick the LR with the steepest negative gradient before divergence.
|
||||
|
||||
Edge case: when the smoothed loss is monotonically increasing from the
|
||||
start (``min_idx <= 1``), there is no meaningful descent region. The
|
||||
function returns the lowest LR (``lrs[0]``) as ``recommended_lr`` so
|
||||
callers get a deterministic fallback rather than ``None``.
|
||||
"""
|
||||
if len(lrs) != len(losses):
|
||||
raise ValueError(
|
||||
f"lrs and losses must have equal length (got {len(lrs)} vs {len(losses)})"
|
||||
)
|
||||
if len(lrs) < 4:
|
||||
raise ValueError(f"Need at least 4 (lr, loss) pairs, got {len(lrs)}")
|
||||
|
||||
smoothed = _smooth(losses)
|
||||
|
||||
# Find min smoothed loss.
|
||||
min_idx = min(range(len(smoothed)), key=lambda index: smoothed[index])
|
||||
min_loss_lr = lrs[min_idx]
|
||||
|
||||
# Detect divergence: first index after min where |loss| > DIVERGENCE_FACTOR * |min|.
|
||||
# ``abs`` keeps the check correct if a custom log-prob style loss goes negative.
|
||||
diverged_at: Optional[float] = None
|
||||
threshold = abs(smoothed[min_idx]) * DIVERGENCE_FACTOR
|
||||
for index in range(min_idx + 1, len(smoothed)):
|
||||
if abs(smoothed[index]) > threshold:
|
||||
diverged_at = lrs[index]
|
||||
break
|
||||
|
||||
# Compute steepest negative gradient (in log-LR space) up to min_idx.
|
||||
upper = max(min_idx, 1)
|
||||
best_grad = 0.0
|
||||
best_idx = 0
|
||||
for index in range(1, upper + 1):
|
||||
d_lr = math.log(lrs[index]) - math.log(lrs[index - 1])
|
||||
d_loss = smoothed[index] - smoothed[index - 1]
|
||||
if d_lr > 0:
|
||||
grad = d_loss / d_lr
|
||||
if grad < best_grad:
|
||||
best_grad = grad
|
||||
best_idx = index
|
||||
|
||||
# Step back one — recommend LR slightly before the steepest descent end.
|
||||
rec_idx = max(0, best_idx - 1) if best_grad < 0 else 0
|
||||
recommended_lr = lrs[rec_idx]
|
||||
|
||||
return {
|
||||
"recommended_lr": recommended_lr,
|
||||
"min_loss_lr": min_loss_lr,
|
||||
"diverged_at": diverged_at,
|
||||
"smoothed_losses": smoothed,
|
||||
}
|
||||
|
||||
|
||||
def _finite_or_reject(values: Sequence[float], label: str) -> list[float]:
|
||||
"""Reject NaN / Infinity floats so the JSON report is parser-safe."""
|
||||
cleaned: list[float] = []
|
||||
for value in values:
|
||||
as_float = float(value)
|
||||
if not math.isfinite(as_float):
|
||||
raise ValueError(
|
||||
f"{label} contains non-finite value ({value!r}); "
|
||||
"NaN / Infinity are rejected to keep the JSON report valid."
|
||||
)
|
||||
cleaned.append(as_float)
|
||||
return cleaned
|
||||
|
||||
|
||||
def save_lr_finder_report(
|
||||
lrs: Sequence[float], losses: Sequence[float], output_path: Path | str,
|
||||
) -> None:
|
||||
"""Write a JSON report with the sweep + recommended LR."""
|
||||
output = Path(output_path)
|
||||
if not is_under_cwd(output):
|
||||
raise ValueError(f"Report path must stay under cwd: {output}")
|
||||
|
||||
report_lrs = _finite_or_reject(lrs, "lrs")
|
||||
report_losses = _finite_or_reject(losses, "losses")
|
||||
summary = find_optimal_lr(report_lrs, report_losses)
|
||||
payload = {
|
||||
"lrs": report_lrs,
|
||||
"losses": report_losses,
|
||||
"smoothed_losses": summary["smoothed_losses"],
|
||||
"recommended_lr": summary["recommended_lr"],
|
||||
"min_loss_lr": summary["min_loss_lr"],
|
||||
"diverged_at": summary["diverged_at"],
|
||||
}
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
# ``allow_nan=False`` is belt-and-braces: report_* are already finite,
|
||||
# but ``smoothed_losses`` could carry a non-finite if the input loss
|
||||
# somehow drifted. Reject rather than emit ``NaN`` (invalid JSON).
|
||||
output.write_text(
|
||||
json.dumps(payload, indent=2, allow_nan=False),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
|
@ -0,0 +1,81 @@
|
|||
"""Auto mixed-precision picker (v0.32.0 Part C).
|
||||
|
||||
Maps a model + GPU compute capability to the best mixed-precision setting:
|
||||
``bf16`` (Ampere+, most modern models), ``fp16`` (Turing or known
|
||||
fp16-stable models), or ``no`` (Pascal and older).
|
||||
|
||||
Quirk map is keyed on a lower-cased substring of the model id so we catch
|
||||
both ``Qwen/Qwen2-7B-Instruct`` and ``alibaba/Qwen2.5-3B`` with one entry.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
# Compute capability thresholds.
|
||||
BF16_MIN_CC = 8.0 # Ampere
|
||||
FP16_MIN_CC = 6.0 # Pascal
|
||||
MAX_MODEL_NAME_LEN = 200
|
||||
|
||||
# Known fp16-stable / bf16-unstable model families.
|
||||
# Value: preferred precision when CC supports it.
|
||||
KNOWN_PRECISION_QUIRKS: dict[str, str] = {
|
||||
"qwen2": "fp16",
|
||||
"qwen2.5": "fp16",
|
||||
"phi-3": "fp16",
|
||||
"phi-3.5": "fp16",
|
||||
"phi-4": "bf16",
|
||||
"gemma-2": "bf16",
|
||||
"mistral": "bf16",
|
||||
"llama-3": "bf16",
|
||||
"llama-2": "bf16",
|
||||
}
|
||||
|
||||
|
||||
def pick_mixed_precision(
|
||||
model_name: str, compute_capability: float,
|
||||
) -> Literal["bf16", "fp16", "no"]:
|
||||
"""Pick mixed-precision mode for a model + GPU.
|
||||
|
||||
- cc < 6.0 → ``"no"`` (Pascal lacks reliable fp16 tensor cores)
|
||||
- cc < 8.0 → ``"fp16"`` (no bf16 support)
|
||||
- cc >= 8.0 → ``"bf16"`` unless model is in the fp16-stable quirks map
|
||||
"""
|
||||
if not isinstance(model_name, str) or not model_name:
|
||||
raise ValueError("model_name must be a non-empty string")
|
||||
if "\x00" in model_name:
|
||||
raise ValueError("model_name must not contain null bytes")
|
||||
if len(model_name) > MAX_MODEL_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"model_name must be <= {MAX_MODEL_NAME_LEN} chars, got {len(model_name)}"
|
||||
)
|
||||
if not isinstance(compute_capability, (int, float)):
|
||||
raise ValueError(
|
||||
f"compute_capability must be a number, got {type(compute_capability)}"
|
||||
)
|
||||
if compute_capability < 0:
|
||||
raise ValueError(
|
||||
f"compute_capability must be non-negative, got {compute_capability}"
|
||||
)
|
||||
|
||||
if compute_capability < FP16_MIN_CC:
|
||||
return "no"
|
||||
|
||||
name_lc = model_name.lower()
|
||||
quirk: str | None = None
|
||||
# Sort longer substrings first so multi-version pairs work correctly:
|
||||
# ``qwen2.5`` wins over ``qwen2``, ``phi-3.5`` wins over ``phi-3``.
|
||||
# If you add a new family with multiple versions, the longest substring
|
||||
# always wins — no manual ordering of the dict is required.
|
||||
for substring in sorted(KNOWN_PRECISION_QUIRKS, key=len, reverse=True):
|
||||
if substring in name_lc:
|
||||
quirk = KNOWN_PRECISION_QUIRKS[substring]
|
||||
break
|
||||
|
||||
if compute_capability < BF16_MIN_CC:
|
||||
# Turing / Volta cannot do bf16 — fall back to fp16 regardless.
|
||||
return "fp16"
|
||||
|
||||
if quirk is None:
|
||||
return "bf16"
|
||||
return "bf16" if quirk == "bf16" else "fp16"
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
"""Loss spike auto-recovery strategy (v0.32.0 Part E).
|
||||
|
||||
When the watchdog fires:
|
||||
1. If ``attempts < max_attempts`` and a checkpoint exists: rollback,
|
||||
decay the LR, resume.
|
||||
2. Otherwise: stop training (existing watchdog behaviour).
|
||||
|
||||
This module provides the policy. Live trainer-state mutation
|
||||
(rollback + LR change + resume) is wired in via the existing
|
||||
``SoupTrainerCallback`` in v0.32.0; the auto-rollback to checkpoint is
|
||||
advisory in v0.32.0 (issues warning + recommends manual resume), with
|
||||
full live wiring tracked for v0.32.1 — the same pattern used by
|
||||
v0.30.0 ``--auto-quant`` and v0.30.0 structured-output flags.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
MIN_LR_FLOOR = 1e-9
|
||||
MAX_ATTEMPTS_CAP = 10
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SpikeRecoveryStrategy:
|
||||
"""Policy for loss-spike recovery decisions."""
|
||||
|
||||
max_attempts: int = 3
|
||||
lr_decay: float = 0.5
|
||||
min_lr: float = MIN_LR_FLOOR
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not (1 <= self.max_attempts <= MAX_ATTEMPTS_CAP):
|
||||
raise ValueError(
|
||||
f"max_attempts must be in [1, {MAX_ATTEMPTS_CAP}], "
|
||||
f"got {self.max_attempts}"
|
||||
)
|
||||
if not (0 < self.lr_decay < 1):
|
||||
raise ValueError(
|
||||
f"lr_decay must be in (0, 1), got {self.lr_decay}"
|
||||
)
|
||||
if self.min_lr <= 0:
|
||||
raise ValueError(f"min_lr must be > 0, got {self.min_lr}")
|
||||
|
||||
def should_recover(self, attempts: int) -> bool:
|
||||
"""True when the recovery budget hasn't been exhausted."""
|
||||
return attempts < self.max_attempts
|
||||
|
||||
def compute_new_lr(self, current_lr: float) -> float:
|
||||
"""Decay the LR by ``lr_decay``, floored at ``min_lr``."""
|
||||
if current_lr <= 0:
|
||||
raise ValueError(f"current_lr must be > 0, got {current_lr}")
|
||||
new = current_lr * self.lr_decay
|
||||
return max(new, self.min_lr)
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
"""Auto warmup scheduling (v0.32.0 Part D).
|
||||
|
||||
Computes a sensible ``warmup_steps`` from dataset size, batch, grad_accum,
|
||||
epochs, and a warmup ratio. Clamped to a safe range so users with tiny
|
||||
datasets still get some warmup, and users with huge datasets don't burn
|
||||
half a million steps doing nothing useful.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
MIN_WARMUP = 10
|
||||
MAX_WARMUP = 1000
|
||||
DEFAULT_RATIO = 0.03
|
||||
MAX_RATIO = 0.5
|
||||
|
||||
|
||||
def compute_warmup_steps(
|
||||
num_examples: int,
|
||||
batch_size: int,
|
||||
grad_accum: int,
|
||||
epochs: int,
|
||||
ratio: float = DEFAULT_RATIO,
|
||||
) -> int:
|
||||
"""Return warmup steps clamped to [MIN_WARMUP, MAX_WARMUP].
|
||||
|
||||
Special case: ``ratio == 0`` means "no warmup" and returns 0 (the
|
||||
schema allows ``warmup_ratio=0.0`` so this matches HF Trainer's
|
||||
convention).
|
||||
"""
|
||||
if num_examples < 1:
|
||||
raise ValueError(f"num_examples must be >= 1, got {num_examples}")
|
||||
if batch_size < 1:
|
||||
raise ValueError(f"batch_size must be >= 1, got {batch_size}")
|
||||
if grad_accum < 1:
|
||||
raise ValueError(f"grad_accum must be >= 1, got {grad_accum}")
|
||||
if epochs < 1:
|
||||
raise ValueError(f"epochs must be >= 1, got {epochs}")
|
||||
if not (0.0 <= ratio <= MAX_RATIO):
|
||||
raise ValueError(
|
||||
f"ratio must be in [0, {MAX_RATIO}], got {ratio}"
|
||||
)
|
||||
if ratio == 0.0:
|
||||
return 0
|
||||
effective_batch = batch_size * grad_accum
|
||||
steps_per_epoch = max(1, math.ceil(num_examples / effective_batch))
|
||||
total_steps = steps_per_epoch * epochs
|
||||
raw = int(round(total_steps * ratio))
|
||||
return max(MIN_WARMUP, min(MAX_WARMUP, raw))
|
||||
|
|
@ -0,0 +1,849 @@
|
|||
"""Tests for v0.32.0 — Training Stability & Auto-Tuning.
|
||||
|
||||
Covers:
|
||||
- Part A: LR range finder (utils/lr_finder.py)
|
||||
- Part B: Live grad-accum auto-tuning (utils/grad_accum.py)
|
||||
- Part C: Auto mixed-precision picker (utils/mixed_precision.py)
|
||||
- Part D: Warmup auto-schedule (utils/warmup.py)
|
||||
- Part E: Loss spike auto-recovery (utils/spike_recovery.py + callback wiring)
|
||||
- Part F: Convergence detector (utils/convergence.py)
|
||||
- Part G: Autopilot integration (autopilot/decisions.py extensions)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
from pydantic import ValidationError
|
||||
from typer.testing import CliRunner
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Part A — LR range finder #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestLRFinderSchedule:
|
||||
"""compute_lr_schedule produces a logarithmic LR sweep."""
|
||||
|
||||
def test_log_schedule_endpoints(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
lrs = compute_lr_schedule(start_lr=1e-7, end_lr=1e-1, num_steps=50)
|
||||
|
||||
assert len(lrs) == 50
|
||||
assert lrs[0] == pytest.approx(1e-7, rel=1e-6)
|
||||
assert lrs[-1] == pytest.approx(1e-1, rel=1e-6)
|
||||
|
||||
def test_log_schedule_monotonic(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
lrs = compute_lr_schedule(1e-6, 1e-1, 30)
|
||||
for left, right in zip(lrs, lrs[1:]):
|
||||
assert right > left
|
||||
|
||||
def test_log_schedule_geometric(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
lrs = compute_lr_schedule(1e-6, 1e-1, 11)
|
||||
ratios = [lrs[i + 1] / lrs[i] for i in range(len(lrs) - 1)]
|
||||
assert all(math.isclose(ratios[0], r, rel_tol=1e-6) for r in ratios)
|
||||
|
||||
def test_schedule_rejects_non_positive_start(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
with pytest.raises(ValueError, match="start_lr"):
|
||||
compute_lr_schedule(0.0, 1e-1, 10)
|
||||
|
||||
def test_schedule_rejects_inverted_range(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
with pytest.raises(ValueError, match="end_lr"):
|
||||
compute_lr_schedule(1e-2, 1e-4, 10)
|
||||
|
||||
def test_schedule_rejects_too_few_steps(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
with pytest.raises(ValueError, match="num_steps"):
|
||||
compute_lr_schedule(1e-7, 1e-1, 1)
|
||||
|
||||
def test_schedule_caps_steps(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
with pytest.raises(ValueError, match="num_steps"):
|
||||
compute_lr_schedule(1e-7, 1e-1, 100_000)
|
||||
|
||||
def test_schedule_min_steps_accepted(self):
|
||||
from soup_cli.utils.lr_finder import compute_lr_schedule
|
||||
|
||||
lrs = compute_lr_schedule(1e-7, 1e-1, 2)
|
||||
assert len(lrs) == 2
|
||||
assert lrs[0] == pytest.approx(1e-7, rel=1e-6)
|
||||
assert lrs[-1] == pytest.approx(1e-1, rel=1e-6)
|
||||
|
||||
|
||||
class TestLRFinderRecommendation:
|
||||
"""find_optimal_lr picks the LR with steepest negative gradient
|
||||
(excluding the explosion tail)."""
|
||||
|
||||
def test_picks_steepest_descent(self):
|
||||
from soup_cli.utils.lr_finder import find_optimal_lr
|
||||
|
||||
# Loss decreases through 1e-3 then explodes
|
||||
lrs = [1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1]
|
||||
losses = [3.0, 2.8, 2.4, 1.5, 4.0, 12.0]
|
||||
|
||||
result = find_optimal_lr(lrs, losses)
|
||||
|
||||
assert "recommended_lr" in result
|
||||
assert "min_loss_lr" in result
|
||||
assert "diverged_at" in result
|
||||
# Recommended must be <= min_loss_lr (steepest descent comes earlier)
|
||||
assert result["recommended_lr"] <= result["min_loss_lr"]
|
||||
# Recommended LR must be in the descent region, not the explosion tail
|
||||
assert result["recommended_lr"] < 1e-2
|
||||
|
||||
def test_returns_smoothed_curve(self):
|
||||
from soup_cli.utils.lr_finder import find_optimal_lr
|
||||
|
||||
lrs = [1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1]
|
||||
losses = [3.0, 2.8, 2.4, 1.5, 4.0, 12.0]
|
||||
|
||||
result = find_optimal_lr(lrs, losses)
|
||||
assert "smoothed_losses" in result
|
||||
assert len(result["smoothed_losses"]) == len(losses)
|
||||
|
||||
def test_detects_divergence(self):
|
||||
from soup_cli.utils.lr_finder import find_optimal_lr
|
||||
|
||||
lrs = [1e-6, 1e-5, 1e-4, 1e-3, 1e-2]
|
||||
losses = [3.0, 2.8, 2.4, 1.5, 100.0]
|
||||
|
||||
result = find_optimal_lr(lrs, losses)
|
||||
assert result["diverged_at"] is not None
|
||||
assert result["diverged_at"] <= 1e-2
|
||||
|
||||
def test_no_divergence_when_loss_stable(self):
|
||||
from soup_cli.utils.lr_finder import find_optimal_lr
|
||||
|
||||
lrs = [1e-6, 1e-5, 1e-4, 1e-3]
|
||||
losses = [3.0, 2.8, 2.6, 2.4]
|
||||
|
||||
result = find_optimal_lr(lrs, losses)
|
||||
assert result["diverged_at"] is None
|
||||
|
||||
def test_mismatched_lengths_rejected(self):
|
||||
from soup_cli.utils.lr_finder import find_optimal_lr
|
||||
|
||||
with pytest.raises(ValueError, match="length"):
|
||||
find_optimal_lr([1e-6, 1e-5], [3.0, 2.8, 2.6])
|
||||
|
||||
def test_too_few_points_rejected(self):
|
||||
from soup_cli.utils.lr_finder import find_optimal_lr
|
||||
|
||||
with pytest.raises(ValueError, match="at least"):
|
||||
find_optimal_lr([1e-6, 1e-5], [3.0, 2.8])
|
||||
|
||||
def test_monotonic_increase_falls_back_to_first_lr(self):
|
||||
from soup_cli.utils.lr_finder import find_optimal_lr
|
||||
|
||||
lrs = [1e-6, 1e-5, 1e-4, 1e-3]
|
||||
losses = [1.0, 2.0, 4.0, 8.0] # explodes immediately
|
||||
result = find_optimal_lr(lrs, losses)
|
||||
assert result["recommended_lr"] == pytest.approx(lrs[0], rel=1e-6)
|
||||
|
||||
|
||||
class TestLRFinderCLI:
|
||||
"""`soup train --find-lr` flag is registered and surfaces in help."""
|
||||
|
||||
def test_flag_in_help(self):
|
||||
from soup_cli.cli import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["train", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "--find-lr" in result.output
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Part B — Live grad-accum auto-tuning #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestGradAccumMonitor:
|
||||
def test_should_adjust_when_high_pressure(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0, threshold=0.92)
|
||||
# 23.5/24 = 0.979 → above 0.92 threshold
|
||||
assert mon.should_adjust(used_vram_gb=23.5) is True
|
||||
|
||||
def test_should_not_adjust_when_low_pressure(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0, threshold=0.92)
|
||||
assert mon.should_adjust(used_vram_gb=10.0) is False
|
||||
|
||||
def test_recommend_doubles_grad_accum(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
new_batch, new_accum = mon.recommend(current_batch=8, current_accum=2)
|
||||
assert new_batch == 4
|
||||
assert new_accum == 4
|
||||
|
||||
def test_recommend_keeps_effective_batch(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
for batch, accum in [(16, 1), (8, 4), (32, 1)]:
|
||||
new_batch, new_accum = mon.recommend(batch, accum)
|
||||
assert new_batch * new_accum == batch * accum
|
||||
|
||||
def test_recommend_floor_at_one(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
new_batch, new_accum = mon.recommend(current_batch=1, current_accum=8)
|
||||
assert new_batch == 1
|
||||
assert new_accum == 8
|
||||
|
||||
def test_recommend_caps_accum_at_max(self):
|
||||
from soup_cli.utils.grad_accum import MAX_ACCUM, GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
# Push accum past the cap; new should equal MAX_ACCUM.
|
||||
_, new_accum = mon.recommend(current_batch=2, current_accum=MAX_ACCUM)
|
||||
assert new_accum == MAX_ACCUM
|
||||
|
||||
def test_observe_updates_peak(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
mon.observe(10.0)
|
||||
mon.observe(15.0)
|
||||
mon.observe(12.0)
|
||||
assert mon.peak_used_gb == pytest.approx(15.0)
|
||||
|
||||
def test_observe_rejects_negative(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
with pytest.raises(ValueError, match="used_vram_gb"):
|
||||
mon.observe(-1.0)
|
||||
|
||||
def test_should_adjust_rejects_negative(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
with pytest.raises(ValueError, match="used_vram_gb"):
|
||||
mon.should_adjust(-0.1)
|
||||
|
||||
def test_recommend_rejects_zero_inputs(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
mon = GradAccumMonitor(total_vram_gb=24.0)
|
||||
with pytest.raises(ValueError, match="current_batch"):
|
||||
mon.recommend(current_batch=0, current_accum=2)
|
||||
with pytest.raises(ValueError, match="current_batch"):
|
||||
mon.recommend(current_batch=2, current_accum=0)
|
||||
|
||||
def test_threshold_bounds(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
with pytest.raises(ValueError, match="threshold"):
|
||||
GradAccumMonitor(total_vram_gb=24.0, threshold=1.5)
|
||||
with pytest.raises(ValueError, match="threshold"):
|
||||
GradAccumMonitor(total_vram_gb=24.0, threshold=0.0)
|
||||
|
||||
def test_total_vram_must_be_positive(self):
|
||||
from soup_cli.utils.grad_accum import GradAccumMonitor
|
||||
|
||||
with pytest.raises(ValueError, match="total_vram_gb"):
|
||||
GradAccumMonitor(total_vram_gb=-1.0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Part C — Auto mixed-precision picker #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestMixedPrecisionPicker:
|
||||
def test_llama_ampere_picks_bf16(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
assert pick_mixed_precision("meta-llama/Llama-3-8B", 8.0) == "bf16"
|
||||
|
||||
def test_qwen2_picks_fp16(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
assert pick_mixed_precision("Qwen/Qwen2-7B", 8.0) == "fp16"
|
||||
|
||||
def test_pre_ampere_drops_to_fp16(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
# Turing T4 (cc=7.5) — no bf16
|
||||
assert pick_mixed_precision("meta-llama/Llama-3-8B", 7.5) == "fp16"
|
||||
|
||||
def test_pre_pascal_returns_no(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
# cc<6.0 doesn't support fp16 reliably
|
||||
assert pick_mixed_precision("meta-llama/Llama-3-8B", 5.0) == "no"
|
||||
|
||||
def test_unknown_model_default_bf16_on_ampere(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
assert pick_mixed_precision("some/unknown-model", 9.0) == "bf16"
|
||||
|
||||
def test_invalid_model_name_rejected(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
with pytest.raises(ValueError, match="model"):
|
||||
pick_mixed_precision("", 8.0)
|
||||
with pytest.raises(ValueError, match="model"):
|
||||
pick_mixed_precision("a\x00b", 8.0)
|
||||
with pytest.raises(ValueError, match="200"):
|
||||
pick_mixed_precision("a" * 201, 8.0)
|
||||
|
||||
def test_qwen25_picks_fp16_not_qwen2_default(self):
|
||||
"""Longer 'qwen2.5' substring must win over 'qwen2'."""
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
# Both entries map to fp16 today, but the test guards the iteration
|
||||
# order: if someone changes qwen2.5 to bf16, this catches it.
|
||||
assert pick_mixed_precision("Qwen/Qwen2.5-7B", 8.0) == "fp16"
|
||||
|
||||
def test_negative_cc_rejected(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
with pytest.raises(ValueError, match="compute_capability"):
|
||||
pick_mixed_precision("meta-llama/Llama-3-8B", -1.0)
|
||||
|
||||
def test_known_quirk_mapping_includes_qwen_and_phi(self):
|
||||
from soup_cli.utils.mixed_precision import KNOWN_PRECISION_QUIRKS
|
||||
|
||||
keys = [k.lower() for k in KNOWN_PRECISION_QUIRKS]
|
||||
assert any("qwen" in k for k in keys)
|
||||
assert any("phi" in k for k in keys)
|
||||
|
||||
def test_cc_exactly_fp16_boundary(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
# cc == 6.0 should accept fp16 (the comparison is `< FP16_MIN_CC`)
|
||||
assert pick_mixed_precision("meta-llama/Llama-3-8B", 6.0) == "fp16"
|
||||
|
||||
def test_cc_exactly_bf16_boundary(self):
|
||||
from soup_cli.utils.mixed_precision import pick_mixed_precision
|
||||
|
||||
# cc == 8.0 should pick bf16 (the comparison is `< BF16_MIN_CC`)
|
||||
assert pick_mixed_precision("meta-llama/Llama-3-8B", 8.0) == "bf16"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Part D — Warmup auto-schedule #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestWarmupAutoSchedule:
|
||||
def test_basic_formula(self):
|
||||
from soup_cli.utils.warmup import compute_warmup_steps
|
||||
|
||||
# 10000 examples / batch 4 / accum 2 / 3 epochs = 3750 update steps
|
||||
# 3% = 112 steps
|
||||
steps = compute_warmup_steps(
|
||||
num_examples=10000,
|
||||
batch_size=4,
|
||||
grad_accum=2,
|
||||
epochs=3,
|
||||
ratio=0.03,
|
||||
)
|
||||
assert 100 <= steps <= 130
|
||||
|
||||
def test_clamps_to_min(self):
|
||||
from soup_cli.utils.warmup import compute_warmup_steps
|
||||
|
||||
steps = compute_warmup_steps(
|
||||
num_examples=10, batch_size=1, grad_accum=1, epochs=1, ratio=0.03,
|
||||
)
|
||||
assert steps >= 10 # MIN_WARMUP
|
||||
|
||||
def test_clamps_to_max(self):
|
||||
from soup_cli.utils.warmup import compute_warmup_steps
|
||||
|
||||
steps = compute_warmup_steps(
|
||||
num_examples=10_000_000,
|
||||
batch_size=1,
|
||||
grad_accum=1,
|
||||
epochs=1,
|
||||
ratio=0.03,
|
||||
)
|
||||
assert steps <= 1000 # MAX_WARMUP
|
||||
|
||||
def test_invalid_ratio_rejected(self):
|
||||
from soup_cli.utils.warmup import compute_warmup_steps
|
||||
|
||||
for bad_ratio in [-0.01, 0.51]:
|
||||
with pytest.raises(ValueError, match="ratio"):
|
||||
compute_warmup_steps(
|
||||
num_examples=1000, batch_size=1, grad_accum=1, epochs=1,
|
||||
ratio=bad_ratio,
|
||||
)
|
||||
|
||||
def test_ratio_zero_means_no_warmup(self):
|
||||
from soup_cli.utils.warmup import compute_warmup_steps
|
||||
|
||||
steps = compute_warmup_steps(
|
||||
num_examples=1000, batch_size=1, grad_accum=1, epochs=1, ratio=0.0,
|
||||
)
|
||||
assert steps == 0
|
||||
|
||||
def test_invalid_inputs_rejected(self):
|
||||
from soup_cli.utils.warmup import compute_warmup_steps
|
||||
|
||||
with pytest.raises(ValueError, match="num_examples"):
|
||||
compute_warmup_steps(num_examples=0, batch_size=1, grad_accum=1, epochs=1)
|
||||
with pytest.raises(ValueError, match="batch_size"):
|
||||
compute_warmup_steps(num_examples=10, batch_size=0, grad_accum=1, epochs=1)
|
||||
with pytest.raises(ValueError, match="grad_accum"):
|
||||
compute_warmup_steps(num_examples=10, batch_size=1, grad_accum=0, epochs=1)
|
||||
with pytest.raises(ValueError, match="epochs"):
|
||||
compute_warmup_steps(num_examples=10, batch_size=1, grad_accum=1, epochs=0)
|
||||
|
||||
def test_ratio_at_max_accepted(self):
|
||||
from soup_cli.utils.warmup import MAX_WARMUP, compute_warmup_steps
|
||||
|
||||
# ratio == MAX_RATIO (0.5) is the inclusive upper bound.
|
||||
steps = compute_warmup_steps(
|
||||
num_examples=100_000, batch_size=1, grad_accum=1, epochs=1, ratio=0.5,
|
||||
)
|
||||
assert steps == MAX_WARMUP
|
||||
|
||||
|
||||
class TestWarmupConfigField:
|
||||
def test_warmup_auto_default_false(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig()
|
||||
assert cfg.warmup_auto is False
|
||||
|
||||
def test_warmup_auto_can_be_set(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(warmup_auto=True)
|
||||
assert cfg.warmup_auto is True
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Part E — Loss spike auto-recovery #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestSpikeRecoveryStrategy:
|
||||
def test_should_recover_within_budget(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5)
|
||||
assert strat.should_recover(attempts=0) is True
|
||||
assert strat.should_recover(attempts=2) is True
|
||||
|
||||
def test_should_not_recover_at_limit(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5)
|
||||
assert strat.should_recover(attempts=3) is False
|
||||
assert strat.should_recover(attempts=4) is False
|
||||
|
||||
def test_compute_new_lr_decays(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5)
|
||||
assert strat.compute_new_lr(2e-4) == pytest.approx(1e-4)
|
||||
assert strat.compute_new_lr(1e-3) == pytest.approx(5e-4)
|
||||
|
||||
def test_lr_decay_bounds(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
with pytest.raises(ValueError, match="lr_decay"):
|
||||
SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.0)
|
||||
with pytest.raises(ValueError, match="lr_decay"):
|
||||
SpikeRecoveryStrategy(max_attempts=3, lr_decay=1.0)
|
||||
|
||||
def test_max_attempts_bounds(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
with pytest.raises(ValueError, match="max_attempts"):
|
||||
SpikeRecoveryStrategy(max_attempts=0)
|
||||
with pytest.raises(ValueError, match="max_attempts"):
|
||||
SpikeRecoveryStrategy(max_attempts=100)
|
||||
|
||||
def test_minimum_lr_floor(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5, min_lr=1e-7)
|
||||
# New LR is below floor → return floor
|
||||
assert strat.compute_new_lr(1e-8) == pytest.approx(1e-7)
|
||||
|
||||
def test_compute_new_lr_rejects_non_positive(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5)
|
||||
with pytest.raises(ValueError, match="current_lr"):
|
||||
strat.compute_new_lr(0.0)
|
||||
with pytest.raises(ValueError, match="current_lr"):
|
||||
strat.compute_new_lr(-1e-4)
|
||||
|
||||
def test_min_lr_must_be_positive(self):
|
||||
from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy
|
||||
|
||||
with pytest.raises(ValueError, match="min_lr"):
|
||||
SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5, min_lr=0.0)
|
||||
|
||||
|
||||
class TestSpikeRecoveryConfig:
|
||||
def test_field_default(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig()
|
||||
assert cfg.loss_spike_recovery is False
|
||||
assert cfg.loss_spike_recovery_max_attempts == 3
|
||||
|
||||
def test_max_attempts_bounds(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(loss_spike_recovery_max_attempts=0)
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(loss_spike_recovery_max_attempts=100)
|
||||
|
||||
def test_recovery_requires_watchdog(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
# Recovery without watchdog enabled is rejected.
|
||||
with pytest.raises(ValidationError) as exc_info:
|
||||
TrainingConfig(loss_spike_recovery=True, loss_watchdog=False)
|
||||
messages = [err["msg"] for err in exc_info.value.errors()]
|
||||
assert any(
|
||||
"loss_watchdog" in msg and "loss_spike_recovery" in msg
|
||||
for msg in messages
|
||||
), messages
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Part F — Convergence detector #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestConvergenceDetector:
|
||||
def test_detects_plateau(self):
|
||||
from soup_cli.utils.convergence import detect_plateau
|
||||
|
||||
# Last 50 losses essentially flat
|
||||
losses = [3.0 - 0.001 * i for i in range(150)] + [2.85] * 50
|
||||
assert detect_plateau(losses, window=50, rel_tol=0.005) is True
|
||||
|
||||
def test_does_not_detect_when_decreasing(self):
|
||||
from soup_cli.utils.convergence import detect_plateau
|
||||
|
||||
losses = [3.0 - 0.005 * i for i in range(200)]
|
||||
assert detect_plateau(losses, window=50, rel_tol=0.005) is False
|
||||
|
||||
def test_too_few_points(self):
|
||||
from soup_cli.utils.convergence import detect_plateau
|
||||
|
||||
losses = [3.0, 2.9, 2.8]
|
||||
assert detect_plateau(losses, window=50, rel_tol=0.005) is False
|
||||
|
||||
def test_window_bounds(self):
|
||||
from soup_cli.utils.convergence import detect_plateau
|
||||
|
||||
with pytest.raises(ValueError, match="window"):
|
||||
detect_plateau([3.0] * 100, window=0)
|
||||
with pytest.raises(ValueError, match="window"):
|
||||
detect_plateau([3.0] * 100, window=10001)
|
||||
|
||||
def test_rel_tol_bounds(self):
|
||||
from soup_cli.utils.convergence import detect_plateau
|
||||
|
||||
with pytest.raises(ValueError, match="rel_tol"):
|
||||
detect_plateau([3.0] * 100, window=50, rel_tol=-0.001)
|
||||
with pytest.raises(ValueError, match="rel_tol"):
|
||||
detect_plateau([3.0] * 100, window=50, rel_tol=2.0)
|
||||
|
||||
|
||||
class TestRecommendAction:
|
||||
def test_recommends_continue_for_decreasing(self):
|
||||
from soup_cli.utils.convergence import recommend_action
|
||||
|
||||
losses = [3.0 - 0.005 * i for i in range(200)]
|
||||
assert recommend_action(losses) == "continue"
|
||||
|
||||
def test_recommends_early_stop_for_long_plateau(self):
|
||||
from soup_cli.utils.convergence import recommend_action
|
||||
|
||||
losses = [3.0 - 0.001 * i for i in range(150)] + [2.85] * 100
|
||||
assert recommend_action(losses) == "early_stop"
|
||||
|
||||
def test_recommends_lower_lr_for_oscillation(self):
|
||||
# Oscillating losses (high variance, no trend)
|
||||
import random
|
||||
|
||||
from soup_cli.utils.convergence import recommend_action
|
||||
|
||||
rng = random.Random(42)
|
||||
losses = [2.5 + rng.uniform(-0.4, 0.4) for _ in range(200)]
|
||||
action = recommend_action(losses)
|
||||
assert action in {"lower_lr", "continue"}
|
||||
|
||||
def test_too_few_points_returns_continue(self):
|
||||
from soup_cli.utils.convergence import recommend_action
|
||||
|
||||
assert recommend_action([3.0, 2.9, 2.8]) == "continue"
|
||||
|
||||
|
||||
class TestConvergenceConfig:
|
||||
def test_field_defaults(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig()
|
||||
assert cfg.convergence_detection is False
|
||||
|
||||
def test_convergence_window_lower_bound(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(convergence_window=4)
|
||||
|
||||
def test_convergence_window_upper_bound(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(convergence_window=10_001)
|
||||
|
||||
def test_convergence_rel_tol_upper_bound(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(convergence_rel_tol=1.1)
|
||||
|
||||
def test_convergence_rel_tol_zero_rejected(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(convergence_rel_tol=0.0)
|
||||
|
||||
|
||||
class TestRecoveryFieldBounds:
|
||||
def test_lr_decay_upper_bound(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(loss_spike_recovery_lr_decay=1.0)
|
||||
|
||||
def test_lr_decay_lower_bound(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(loss_spike_recovery_lr_decay=0.0)
|
||||
|
||||
|
||||
class TestGradAccumThresholdField:
|
||||
def test_pressure_threshold_upper_bound(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(grad_accum_pressure_threshold=0.99)
|
||||
|
||||
def test_pressure_threshold_lower_bound(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(grad_accum_pressure_threshold=0.05)
|
||||
|
||||
|
||||
class TestPlateauNonPositiveMean:
|
||||
def test_plateau_non_positive_mean_returns_false(self):
|
||||
from soup_cli.utils.convergence import detect_plateau
|
||||
|
||||
# All-negative losses → mean < 0 → refuse to assess.
|
||||
losses = [-2.0] * 60
|
||||
assert detect_plateau(losses, window=50, rel_tol=0.005) is False
|
||||
|
||||
def test_recommend_action_non_positive_mean_returns_continue(self):
|
||||
from soup_cli.utils.convergence import recommend_action
|
||||
|
||||
losses = [-3.0 - 0.001 * i for i in range(200)]
|
||||
assert recommend_action(losses) == "continue"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Part G — Autopilot integration #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestAutopilotIntegration:
|
||||
def test_decide_warmup_returns_int(self):
|
||||
from soup_cli.autopilot.decisions import decide_warmup
|
||||
|
||||
steps = decide_warmup(
|
||||
num_examples=10000, batch_size=4, grad_accum=2, epochs=3,
|
||||
)
|
||||
assert isinstance(steps, int)
|
||||
assert steps > 0
|
||||
|
||||
def test_decide_mixed_precision_routes_to_picker(self):
|
||||
from soup_cli.autopilot.decisions import decide_mixed_precision
|
||||
|
||||
prec = decide_mixed_precision("meta-llama/Llama-3-8B", 8.0)
|
||||
assert prec in {"bf16", "fp16", "no"}
|
||||
|
||||
def test_decide_mixed_precision_invalid_inputs(self):
|
||||
from soup_cli.autopilot.decisions import decide_mixed_precision
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
decide_mixed_precision("", 8.0)
|
||||
|
||||
|
||||
class TestAutopilotConfigEmission:
|
||||
def test_generated_config_includes_warmup_auto_when_set(
|
||||
self, tmp_path, monkeypatch,
|
||||
):
|
||||
"""Generated config has warmup_auto=true so train.py picks it up."""
|
||||
from soup_cli.autopilot.generate_config import generate_config
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
decisions = {
|
||||
"task": "sft",
|
||||
"format": "alpaca",
|
||||
"max_length": 2048,
|
||||
"quantization": "4bit",
|
||||
"lora": {"r": 16, "alpha": 32, "use_dora": False},
|
||||
"lr": 2e-4,
|
||||
"epochs": 3,
|
||||
"batch_size": 4,
|
||||
"grad_accum": 2,
|
||||
"perf": {
|
||||
"use_flash_attn": True, "use_liger": True,
|
||||
"gradient_checkpointing": False,
|
||||
},
|
||||
"warmup_auto": True,
|
||||
"mixed_precision": "bf16",
|
||||
}
|
||||
out = Path("soup.yaml")
|
||||
generate_config(
|
||||
base="meta-llama/Llama-3-8B",
|
||||
data_path="data.jsonl",
|
||||
decisions=decisions,
|
||||
output_path=out,
|
||||
)
|
||||
text = out.read_text(encoding="utf-8")
|
||||
cfg = yaml.safe_load(text)
|
||||
assert cfg["training"].get("warmup_auto") is True
|
||||
|
||||
def test_decisions_output_must_stay_under_cwd(self, tmp_path, monkeypatch):
|
||||
from soup_cli.autopilot.generate_config import generate_config
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
decisions = {
|
||||
"task": "sft", "format": "alpaca", "max_length": 1024,
|
||||
"quantization": "4bit",
|
||||
"lora": {"r": 8, "alpha": 16, "use_dora": False},
|
||||
"lr": 2e-4, "epochs": 1, "batch_size": 1, "grad_accum": 1,
|
||||
"perf": {
|
||||
"use_flash_attn": False, "use_liger": False,
|
||||
"gradient_checkpointing": False,
|
||||
},
|
||||
"output": "/tmp/escape",
|
||||
}
|
||||
with pytest.raises(ValueError, match="under cwd"):
|
||||
generate_config(
|
||||
base="meta-llama/Llama-3-8B",
|
||||
data_path="data.jsonl",
|
||||
decisions=decisions,
|
||||
output_path=Path("soup.yaml"),
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Integration: lr-finder CLI smoke (offline-safe) #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestLRFinderRunner:
|
||||
"""save_lr_finder_report writes a JSON report users can plot."""
|
||||
|
||||
def test_save_report(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils.lr_finder import save_lr_finder_report
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
lrs = [1e-6, 1e-5, 1e-4, 1e-3]
|
||||
losses = [3.0, 2.8, 2.4, 2.6]
|
||||
out = Path("lr_report.json")
|
||||
save_lr_finder_report(lrs, losses, out)
|
||||
|
||||
assert out.exists()
|
||||
data = json.loads(out.read_text(encoding="utf-8"))
|
||||
assert data["lrs"] == lrs
|
||||
assert data["losses"] == losses
|
||||
assert "recommended_lr" in data
|
||||
|
||||
def test_save_report_rejects_nan(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils.lr_finder import save_lr_finder_report
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="non-finite"):
|
||||
save_lr_finder_report(
|
||||
[1e-6, 1e-5, 1e-4, 1e-3],
|
||||
[3.0, 2.8, float("nan"), 2.4],
|
||||
Path("report.json"),
|
||||
)
|
||||
|
||||
def test_save_report_rejects_infinity(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils.lr_finder import save_lr_finder_report
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="non-finite"):
|
||||
save_lr_finder_report(
|
||||
[1e-6, 1e-5, 1e-4, 1e-3],
|
||||
[3.0, 2.8, 2.4, float("inf")],
|
||||
Path("report.json"),
|
||||
)
|
||||
|
||||
def test_save_report_path_must_stay_under_cwd(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils.lr_finder import save_lr_finder_report
|
||||
|
||||
# tmp_path is the inner; we chdir to a subdir so tmp_path itself
|
||||
# is outside cwd and thus cannot be the target.
|
||||
inner = tmp_path / "inner"
|
||||
inner.mkdir()
|
||||
monkeypatch.chdir(inner)
|
||||
outside = (tmp_path / "lr_report.json").resolve()
|
||||
with pytest.raises(ValueError, match="under cwd"):
|
||||
save_lr_finder_report([1e-6, 1e-5], [3.0, 2.8], outside)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Cross-cutting: SoupConfig top-level still serializes #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class TestNewFieldsRoundTrip:
|
||||
def test_roundtrip_yaml(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """
|
||||
base: meta-llama/Llama-3-8B
|
||||
task: sft
|
||||
data:
|
||||
train: data.jsonl
|
||||
format: alpaca
|
||||
training:
|
||||
epochs: 3
|
||||
lr: 2e-4
|
||||
batch_size: 4
|
||||
warmup_auto: true
|
||||
loss_watchdog: true
|
||||
loss_spike_recovery: true
|
||||
loss_spike_recovery_max_attempts: 2
|
||||
convergence_detection: true
|
||||
output: ./out
|
||||
"""
|
||||
cfg = load_config_from_string(yaml_text)
|
||||
assert cfg.training.warmup_auto is True
|
||||
assert cfg.training.loss_spike_recovery is True
|
||||
assert cfg.training.loss_spike_recovery_max_attempts == 2
|
||||
assert cfg.training.convergence_detection is True
|
||||
Loading…
Reference in New Issue