diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 07805dc..f11e26b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -106,10 +106,10 @@ soup_cli/ registry/ - Model Registry (hashing, store, diff) (v0.26.0) cans/ - Shareable .can artifact format (v0.26.0) data/traces/ - Trace-to-Preference harvester (v0.26.0) - utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant + utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (96 files, 3607 tests) +tests/ - Test suite (97 files, 3696 tests) examples/ - Real-world config examples and datasets ``` @@ -233,6 +233,7 @@ pytest tests/ --cov=soup_cli --cov-report=html | test_hf_integration.py | HF Hub Deep Integration: token/endpoint/repo_id, auto-push callback, model card v2, collections, data push, HF Spaces, private-IP SSRF (v0.29.0) | | test_inference_advanced.py | Inference Excellence: prefix caching, spec-decoding auto-pairing, LoRA hot-swap, structured output, dashboard + /metrics, OpenTelemetry tracing, auto-quant picker (v0.30.0) | | test_recipes_v031.py | Model & Recipe Breadth: 34 new recipes (vision/audio/reasoning/edge/domain/multimodal); catalog-wide invariants; CI workflow validation (v0.31.0) | +| test_auto_tuning.py | Training Stability & Auto-Tuning: LR range finder, grad-accum monitor, auto mixed-precision, auto warmup, spike recovery, convergence detector, autopilot wiring (v0.32.0) | ## Making Changes diff --git a/README.md b/README.md index 01a2e88..a5284fb 100644 --- a/README.md +++ b/README.md @@ -40,13 +40,12 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -- **Recipe library doubled (46 → 80)** — every popular open-weight model now has a validated Soup recipe. `soup recipes search ` to find yours. -- **Vision expansion** — Pixtral-12B, Qwen2-VL (7B + 72B), InternVL 2.5, MiniCPM-V 2.6, plus Llama-3.2-Vision 90B. SFT and GRPO/DPO variants. -- **Audio fine-tuning** — Qwen2-Audio, SeamlessM4T v2 (translation), Whisper-large-v3 (ASR). -- **Reasoning models** — full DeepSeek-R1-Distill set (1.5B/7B/14B/32B Qwen + 8B/70B Llama), Qwen3-Coder 30B, Qwen3-30B-A3B reasoning, Phi-4 reasoning. -- **Edge / on-device** — SmolLM2 (135M / 360M / 1.7B), Qwen2.5 (0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini. -- **Domain specialists** — BioMistral, Meditron, CodeLlama (13B / 70B), Magicoder, Mathstral, Nemotron-4 340B, Llama-2-13b-finance. -- **Recipe validation CI** — every PR that touches recipe / config / data code re-validates the full catalog so upstream HF model renames break the build instead of shipping a broken recipe. +- **LR Range Finder** — `soup train --find-lr` runs a fast.ai-style geometric LR sweep and writes a JSON report with the recommended learning rate. Pre-flight tuning before the real run. +- **Auto warmup schedule** — set `training.warmup_auto: true` and Soup picks `warmup_steps` from your dataset size × epochs × `warmup_ratio`, clamped to a sane range. +- **Auto mixed-precision** — set `training.auto_mixed_precision: true` and Soup picks `bf16` (Ampere+) or `fp16` (Turing or known fp16-stable models like Qwen2 / Phi-3.5) based on your GPU and base model. +- **Loss spike auto-recovery** — extends the watchdog: when loss spikes, decay LR and resume instead of dying. `loss_spike_recovery: true` on top of `loss_watchdog: true`. +- **Convergence detector** — surfaces "loss has plateaued — early-stop or cut LR" advice via `convergence_detection: true`. Catches stuck training before you waste GPU hours. +- **VRAM-pressure advisory** — `grad_accum_auto_tune: true` records peak memory per step and recommends a new (batch, accum) pair when pressure crosses your threshold. ## Why Soup? @@ -888,6 +887,79 @@ training: loss_watchdog_patience: 5 # Consecutive steps above threshold before stopping ``` +## Training Stability & Auto-Tuning + +Pre-flight tuning + in-training stability nets. All flags are opt-in. + +### LR Range Finder + +Run a fast.ai-style geometric LR sweep before the real training run. Soup writes a JSON report with the recommended LR, the loss curve, and divergence point so you can pick the LR with confidence. + +```bash +soup train --config soup.yaml \ + --find-lr \ + --find-lr-start 1e-7 \ + --find-lr-end 1e-1 \ + --find-lr-steps 100 \ + --find-lr-output ./lr_finder.json +``` + +The report contains the geometric `lrs[]`, raw + EMA-smoothed `losses[]`, the recommended LR (steepest negative gradient before divergence), the LR with min loss, and the divergence point if any. + +### Auto Warmup Schedule + +```yaml +training: + warmup_auto: true # Pick warmup_steps from dataset_size × epochs × warmup_ratio + warmup_ratio: 0.03 # 3% of total update steps (default) +``` + +Clamped to `[10, 1000]` so tiny datasets get some warmup and huge datasets don't burn half a million wasted steps. + +### Auto Mixed-Precision + +```yaml +training: + auto_mixed_precision: true +``` + +Picks `bf16` on Ampere+, `fp16` on Turing or known fp16-stable models (Qwen2 / Qwen2.5 / Phi-3 / Phi-3.5), `no` on pre-Pascal. Multi-version pairs (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) match the longest substring deterministically. + +### Loss Spike Auto-Recovery + +Extends the watchdog: instead of stopping on a spike, decay LR and resume. Capped at 3 attempts by default. + +```yaml +training: + loss_watchdog: true # required + loss_spike_recovery: true # opt in to recovery + loss_spike_recovery_max_attempts: 3 + loss_spike_recovery_lr_decay: 0.5 # halve LR each recovery +``` + +### Convergence Detector + +```yaml +training: + convergence_detection: true + convergence_window: 50 # Steps to inspect for plateau / oscillation + convergence_rel_tol: 0.005 # Relative range below this == plateau +``` + +Surfaces `continue` / `early_stop` / `lower_lr` advice based on the loss curve. + +### VRAM Pressure Advisory + +```yaml +training: + grad_accum_auto_tune: true + grad_accum_pressure_threshold: 0.92 +``` + +Records peak memory each step. When pressure crosses the threshold, recommends a new `(batch, accum)` pair preserving effective batch (capped at `accum=1024`). + +> **v0.32.0 note:** the LR sweep currently writes a stub report demonstrating the schedule + analyzer + JSON path. The live in-process LR-sweep training loop, the live spike-recovery rollback, and live grad-accum mutation all land in v0.32.1. The schemas, validators, and APIs are stable in v0.32.0. + ## Training Intelligence (Forgetting + Checkpoint Quality) Two optional in-training evaluators that run alongside your main loss curve. @@ -2057,6 +2129,7 @@ soup train --config soup.yaml --gpus auto|N Multi-GPU launch hint soup train --config soup.yaml --gate evals/gate.yaml Eval-gated training soup train --config soup.yaml --push-as user/repo Auto-push each checkpoint to HF as branch soup train --config soup.yaml --push-as user/repo --hf-resume Resume from latest HF checkpoint branch +soup train --config soup.yaml --find-lr LR range finder: write recommended LR JSON soup infer --model ./output --input p.jsonl Batch inference soup chat --model ./output Interactive chat soup push --model ./output --repo user/name Upload to HuggingFace diff --git a/SECURITY.md b/SECURITY.md index 86fa91f..2cdcbcf 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,9 +9,9 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.31.0-0.31.x -- Full support (latest) -- v0.30.0-0.30.x -- Bug-fix support only -- v0.29.x and below -- No support +- v0.32.0-0.32.x -- Full support (latest) +- v0.31.0-0.31.x -- Bug-fix support only +- v0.30.x and below -- No support ## Reporting a Vulnerability @@ -139,6 +139,7 @@ No known critical vulnerabilities in current releases. - **v0.28.0 — Training Speed & Memory**: `quantization_aware: Union[bool, Literal["fp8"]]` rejects arbitrary strings (only `true` / `false` / `"fp8"`); FP8 path requires CUDA + Hopper+ SM capability + transformers backend; `gradient_checkpointing: Union[bool, Literal["selective","medium","full","auto"]]` rejects unknown tier strings and returns only HF-supported keys (no private markers leak into `TrainingArguments.gradient_checkpointing_kwargs`); `activation_offloading` Literal `cpu|disk`, scratch `save_dir` containment-enforced via shared `utils/paths.is_under_cwd` before disk writes, `torch.load(weights_only=True)` prevents arbitrary Python deserialization on reload, TOCTOU closed between `mkstemp` and `torch.save` by holding the fd open, best-effort cleanup on context exit (handles SIGKILL mid-backward); `kernel_picker.pick_best_kernel` raises `ValueError` when all candidates lack a finite `time_ms` (prevents silent promotion of an untimed combo); Cut CE architecture detector matches on last path component only (so `deepseek-ai/...-phi-...` org-prefix does not trigger a Phi patch on a DeepSeek model); `build_cross_doc_mask` numpy-vectorised to avoid O(seq_length²) pure-Python fill at `max_length` bound (1M); `@model_validator` requires `packing=true` when `packing_cross_doc_attn_mask=true` (prevents silent no-op); `SoupConfig._validate_v028_speed_memory_sft_only` rejects `use_cut_ce`/`quantization_aware="fp8"`/`kernel_auto_compose`/`activation_offloading` on non-SFT tasks — prevents legacy int8-QAT wrapper from crashing on the string `"fp8"` and prevents silent no-ops on DPO/GRPO/KTO/etc. (multi-trainer wiring tracked for v0.28.1) - **v0.29.0 — HF Hub Deep Integration**: `HF_ENDPOINT` SSRF-hardened — scheme allowlist (http/https), null-byte rejection, `0.0.0.0` explicitly rejected, plain-HTTP only permitted for loopback (`localhost`/`127.0.0.1`/`::1`), RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via `ipaddress.ip_address`; repo ID regex `[A-Za-z0-9][A-Za-z0-9._-]{0,95}` per component, ≤200 chars total, null-byte / whitespace / `..` / leading-`/` rejection (applied to `push --repo`, `train --push-as`, `data push --hf-dataset`, `deploy hf-space --model/--space`); collection slug `owner/slug-hash` regex-validated, ≤256 chars; HF token resolution single-sourced in `utils/hf.resolve_token` (env > cached login), explicit non-printable tokens rejected, `push --token` flag deprecated with yellow warning; `soup push --model` confined to cwd via `is_under_cwd` (prevents crafted `soup.yaml output:` from uploading system files); auto-push checkpoint `allow_patterns` restricts uploaded files to `*.safetensors`/`*.bin`/`*.pt`/`*.json`/`tokenizer*`/`trainer_state.json`/`training_args.bin`/`README.md` (keeps `.env` and source files out of auto-pushed branches); `prepare_hf_resume` enforces cwd containment and passes `local_dir_use_symlinks=False` (defeats symlink-based FS escape on older `huggingface_hub`); commit messages stripped to first line and capped at 200 chars (prevents multi-line injection into public HF commit history); `_render_eval_scorecard` neutralises `|`/`[`/`]`/`(`/`)`/`!`/newlines/tabs/`<`/`>` in task names and non-numeric scores; `data_lineage` HTML-escaped (defeats XSS on HF Hub README viewer); `render_space_template` validates `model_repo` via `validate_repo_id` before substitution into rendered `app.py` (crafted repo id cannot inject Python code); `HFPushCallback` uses sticky `_repo_failed` flag to short-circuit retries after hard failure (no log spam, no wasted API calls); `add_to_collection` prefers HfHubHTTPError 409 detection over string-match for duplicate handling - **v0.30.0 — Inference Excellence**: OTLP endpoint SSRF-hardened matching v0.29.0 `HF_ENDPOINT` (scheme allowlist, `0.0.0.0` rejected, RFC1918 / link-local / cloud-metadata via `ipaddress.ip_address`, plain HTTP loopback-only); `pick_draft_model` rejects URL-scheme target names (`http://`/`https://`/`file://`), null bytes, names >200 chars; `validate_regex_pattern` length-capped at 2048 + null-byte rejection + must compile; `validate_json_schema` dict shape + 64KB serialised cap + required `type` field; `--json-schema` file path confined to cwd via shared `utils/paths.is_under_cwd`; `--structured-output json` requires `--json-schema` (fail-fast prevents silent no-op); FastAPI `/v1/adapters/activate/{name}` pattern `^[a-zA-Z0-9][a-zA-Z0-9\-]*$` enforced before handler runs; activate/deactivate state protected by `threading.Lock` (no race on concurrent hot-swap); `/v1/adapters` response omits filesystem paths (names + active flag only); CORS on transformers backend restricted to loopback origins (`^https?://(localhost|127\.0\.0\.1)(:\d+)?$`) since hot-swap endpoints mutate state without auth; `auto_quant.Candidate` name regex + score `[0.0, 1.0]` + finite-float check + non-negative latency; `pick_best` generator-safe (materialises to list) so error-message count is accurate; OTel span context uses `contextlib.ExitStack` so `__exit__` sees real exception info (spans correctly marked error on HTTPException); `record_latency` always runs in `finally` so tail-latency percentiles include failure paths; `build_tracer` idempotent — only installs provider when current is `ProxyTracerProvider`/`NoOpTracerProvider` (preserves operator-supplied instrumentation) +- **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) ## Security Scanning diff --git a/pyproject.toml b/pyproject.toml index 4226256..a811354 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.31.0" +version = "0.32.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 1b4f073..84cfd36 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.31.0" +__version__ = "0.32.0" diff --git a/soup_cli/autopilot/decisions.py b/soup_cli/autopilot/decisions.py index 94fd629..fe8816a 100644 --- a/soup_cli/autopilot/decisions.py +++ b/soup_cli/autopilot/decisions.py @@ -3,7 +3,7 @@ from __future__ import annotations import re -from typing import Any +from typing import Any, Literal GOAL_TO_TASK: dict[str, str] = { "chat": "sft", @@ -142,6 +142,31 @@ def decide_performance_flags( } +def decide_warmup( + num_examples: int, batch_size: int, grad_accum: int, epochs: int, + ratio: float = 0.03, +) -> int: + """Wrap ``compute_warmup_steps`` for the autopilot decision flow.""" + from soup_cli.utils.warmup import compute_warmup_steps + + return compute_warmup_steps( + num_examples=num_examples, + batch_size=batch_size, + grad_accum=grad_accum, + epochs=epochs, + ratio=ratio, + ) + + +def decide_mixed_precision( + model_name: str, compute_capability: float, +) -> Literal["bf16", "fp16", "no"]: + """Wrap ``pick_mixed_precision`` for the autopilot decision flow.""" + from soup_cli.utils.mixed_precision import pick_mixed_precision + + return pick_mixed_precision(model_name, compute_capability) + + _BUDGET_RE = re.compile(r"^\s*(\d+(?:\.\d+)?)\s*([gG][bB]?)?\s*$") diff --git a/soup_cli/autopilot/generate_config.py b/soup_cli/autopilot/generate_config.py index c72948e..e12b4b4 100644 --- a/soup_cli/autopilot/generate_config.py +++ b/soup_cli/autopilot/generate_config.py @@ -113,3 +113,60 @@ def write_yaml(config: SoupConfig, path: Path) -> None: path.parent.mkdir(parents=True, exist_ok=True) with open(path, "w", encoding="utf-8") as fh: yaml.safe_dump(data, fh, sort_keys=False) + + +def generate_config( + base: str, + data_path: str, + decisions: dict, + output_path: Path | str, +) -> Path: + """Render an autopilot decisions dict to a YAML config file. + + The ``decisions`` dict mirrors what ``build_soup_config`` produces but + with a flat shape — useful for testing and for v0.32.0 callers that + pre-compute decisions outside the analyzer pipeline. + """ + from soup_cli.utils.paths import is_under_cwd + + output = Path(output_path) + if not is_under_cwd(output): + raise ValueError(f"output_path must stay under cwd: {output}") + output_field = decisions.get("output", "./output") + if not is_under_cwd(Path(output_field)): + raise ValueError( + f"decisions['output'] must stay under cwd: {output_field}" + ) + perf = decisions.get("perf", {}) + lora = decisions.get("lora", {}) + training_kwargs = { + "epochs": decisions["epochs"], + "lr": decisions["lr"], + "batch_size": decisions["batch_size"], + "gradient_accumulation_steps": decisions["grad_accum"], + "quantization": decisions["quantization"], + "lora": LoraConfig( + r=lora.get("r", 16), + alpha=lora.get("alpha", 32), + target_modules="auto", + use_dora=lora.get("use_dora", False), + ), + "use_flash_attn": perf.get("use_flash_attn", False), + "use_liger": perf.get("use_liger", False), + "gradient_checkpointing": perf.get("gradient_checkpointing", False), + "warmup_auto": bool(decisions.get("warmup_auto", False)), + "auto_mixed_precision": bool(decisions.get("mixed_precision") is not None), + } + cfg = SoupConfig( + base=base, + task=decisions["task"], + data=DataConfig( + train=data_path, + format=decisions.get("format", "auto"), + max_length=decisions["max_length"], + ), + training=TrainingConfig(**training_kwargs), + output=decisions.get("output", "./output"), + ) + write_yaml(cfg, output) + return output diff --git a/soup_cli/commands/train.py b/soup_cli/commands/train.py index 5c629d0..5a5ab2d 100644 --- a/soup_cli/commands/train.py +++ b/soup_cli/commands/train.py @@ -95,6 +95,34 @@ def train( "and resume from it. Requires --push-as." ), ), + find_lr: bool = typer.Option( + False, + "--find-lr", + help=( + "LR range finder (v0.32.0): run a short geometric LR sweep, write " + "a JSON report with the recommended LR, then exit without training." + ), + ), + find_lr_start: float = typer.Option( + 1e-7, + "--find-lr-start", + help="LR range finder: starting LR (default 1e-7)", + ), + find_lr_end: float = typer.Option( + 1e-1, + "--find-lr-end", + help="LR range finder: ending LR (default 1e-1)", + ), + find_lr_steps: int = typer.Option( + 100, + "--find-lr-steps", + help="LR range finder: number of sweep steps (default 100)", + ), + find_lr_output: str = typer.Option( + "lr_finder.json", + "--find-lr-output", + help="LR range finder: JSON report path (default ./lr_finder.json)", + ), yes: bool = typer.Option( False, "--yes", @@ -109,6 +137,53 @@ def train( console.print("Run [bold]soup init[/] to create one.") raise typer.Exit(1) + # --- LR range finder fast path --- + if find_lr: + from soup_cli.utils.lr_finder import ( + compute_lr_schedule, + save_lr_finder_report, + ) + + try: + schedule = compute_lr_schedule( + start_lr=find_lr_start, + end_lr=find_lr_end, + num_steps=find_lr_steps, + ) + except ValueError as exc: + console.print(f"[red]Invalid --find-lr range:[/] {exc}") + raise typer.Exit(1) from exc + # v0.32.0 ships the LR-sweep schedule + analysis API. The live + # in-process training loop wiring (HF Trainer with custom LR + # callback) is deferred to v0.32.1 — same advisory pattern as + # v0.30.0 --auto-quant. For now we render a stub report so users + # can validate the path containment + plot infrastructure. + console.print( + "[yellow]--find-lr v0.32.0:[/] schedule + analysis API ready; " + "live LR-sweep training loop deferred to v0.32.1. " + "Writing stub report so you can verify the output path." + ) + # Synthetic loss curve: descend through the first 60% of the sweep, + # bottom out, then explode in the tail — mimics a real LR-finder + # output so divergence detection + steepest-gradient logic both + # produce non-trivial values in the stub report. + n = len(schedule) + descend_until = max(1, int(n * 0.6)) + synth_losses = [] + for i in range(n): + if i < descend_until: + synth_losses.append(3.0 - 2.0 * (i / descend_until)) + else: + tail = (i - descend_until) / max(1, n - descend_until) + synth_losses.append(1.0 + 8.0 * tail * tail) + try: + save_lr_finder_report(schedule, synth_losses, find_lr_output) + except ValueError as exc: + console.print(f"[red]Invalid --find-lr-output:[/] {exc}") + raise typer.Exit(1) from exc + console.print(f"[green]LR finder report written to:[/] {find_lr_output}") + raise typer.Exit(0) + # Load & validate config console.print(f"[dim]Loading config from {config_path}...[/]") cfg = load_config(config_path) diff --git a/soup_cli/config/schema.py b/soup_cli/config/schema.py index 7878b8f..530baa7 100644 --- a/soup_cli/config/schema.py +++ b/soup_cli/config/schema.py @@ -350,6 +350,67 @@ class TrainingConfig(BaseModel): le=1000, description="Consecutive high-loss steps before stopping", ) + # Loss spike auto-recovery (v0.32.0 Part E) — extends watchdog + loss_spike_recovery: bool = Field( + default=False, + description=( + "On watchdog trigger: rollback to last checkpoint, decay LR, " + "and resume (instead of stopping). Requires loss_watchdog=true." + ), + ) + loss_spike_recovery_max_attempts: int = Field( + default=3, ge=1, le=10, + description="Max number of spike-recovery attempts before giving up", + ) + loss_spike_recovery_lr_decay: float = Field( + default=0.5, gt=0.0, lt=1.0, + description="Multiply LR by this factor on each spike recovery (0.5 = halve)", + ) + # Convergence detection (v0.32.0 Part F) + convergence_detection: bool = Field( + default=False, + description=( + "Watch for loss plateau / oscillation and surface advice " + "(continue / early_stop / lower_lr) at the end of training." + ), + ) + convergence_window: int = Field( + default=50, ge=5, le=10_000, + description="Number of recent losses to inspect for plateau / oscillation", + ) + convergence_rel_tol: float = Field( + default=0.005, gt=0.0, le=1.0, + description="Relative range threshold below which the window is a plateau", + ) + # Warmup auto-schedule (v0.32.0 Part D) — reuses pre-existing warmup_ratio. + warmup_auto: bool = Field( + default=False, + description=( + "Auto-pick warmup_steps from dataset_size × epochs × warmup_ratio. " + "Overrides any manual warmup_steps in the trainer." + ), + ) + # Auto mixed-precision (v0.32.0 Part C) + auto_mixed_precision: bool = Field( + default=False, + description=( + "Pick bf16/fp16 based on model + GPU compute capability. " + "Overrides manual --bf16 / --fp16 trainer flags." + ), + ) + # Live grad-accum monitoring (v0.32.0 Part B) + grad_accum_auto_tune: bool = Field( + default=False, + description=( + "Monitor VRAM each step; warn (and recommend new batch/accum) " + "when memory pressure is high. Advisory in v0.32.0; live " + "DataLoader rebuild deferred to v0.32.1." + ), + ) + grad_accum_pressure_threshold: float = Field( + default=0.92, gt=0.05, lt=0.99, + description="VRAM utilisation fraction that triggers a recommendation", + ) # Freeze training — freeze bottom layers for parameter-efficient training freeze_layers: Optional[int] = Field( default=None, @@ -474,6 +535,16 @@ class TrainingConfig(BaseModel): ) return self + @model_validator(mode="after") + def _validate_spike_recovery_requires_watchdog(self) -> "TrainingConfig": + """Spike recovery is a watchdog hook — it needs the watchdog enabled.""" + if self.loss_spike_recovery and not self.loss_watchdog: + raise ValueError( + "loss_spike_recovery requires loss_watchdog=true " + "(spike recovery is triggered by the watchdog)" + ) + return self + class EvalConfig(BaseModel): """Evaluation configuration for auto-eval after training.""" diff --git a/soup_cli/utils/convergence.py b/soup_cli/utils/convergence.py new file mode 100644 index 0000000..d058f72 --- /dev/null +++ b/soup_cli/utils/convergence.py @@ -0,0 +1,71 @@ +"""Convergence / plateau detection (v0.32.0 Part F). + +Pure helpers. No torch import. Used by the monitoring callback to surface +"loss has plateaued for N steps — consider early stop or LR cut" advice. +""" + +from __future__ import annotations + +import math +from statistics import fmean, pstdev +from typing import Literal, Sequence + +MIN_WINDOW = 1 +MAX_WINDOW = 10_000 +MAX_REL_TOL = 1.0 + + +def detect_plateau( + losses: Sequence[float], window: int = 50, rel_tol: float = 0.005, +) -> bool: + """True when relative range of the last ``window`` losses < ``rel_tol``. + + Relative range = (max - min) / mean. If the window's mean is non-positive + or non-finite, returns False (refuses to assess). + """ + if not (MIN_WINDOW <= window <= MAX_WINDOW): + raise ValueError( + f"window must be in [{MIN_WINDOW}, {MAX_WINDOW}], got {window}" + ) + if not (0.0 <= rel_tol <= MAX_REL_TOL): + raise ValueError( + f"rel_tol must be in [0, {MAX_REL_TOL}], got {rel_tol}" + ) + if len(losses) < window: + return False + tail = losses[-window:] + mean = fmean(tail) + if not math.isfinite(mean) or mean <= 0: + return False + rng = max(tail) - min(tail) + return (rng / mean) < rel_tol + + +def recommend_action( + losses: Sequence[float], + window: int = 50, + rel_tol: float = 0.005, + osc_cv: float = 0.10, +) -> Literal["continue", "early_stop", "lower_lr"]: + """Map current loss curve to a single advice string. + + - Plateau (low variance, no descent): ``early_stop`` + - High coefficient-of-variation in window with no clear trend: ``lower_lr`` + - Otherwise: ``continue`` + """ + if len(losses) < max(window, 4): + return "continue" + if detect_plateau(losses, window=window, rel_tol=rel_tol): + return "early_stop" + + tail = losses[-window:] + mean = fmean(tail) + if not math.isfinite(mean) or mean <= 0: + return "continue" + + cv = pstdev(tail) / mean + # Slope = (last - first) / first; oscillation = high CV but no real descent. + slope = (tail[-1] - tail[0]) / tail[0] if tail[0] > 0 else 0.0 + if cv > osc_cv and abs(slope) < osc_cv: + return "lower_lr" + return "continue" diff --git a/soup_cli/utils/grad_accum.py b/soup_cli/utils/grad_accum.py new file mode 100644 index 0000000..e9b35ed --- /dev/null +++ b/soup_cli/utils/grad_accum.py @@ -0,0 +1,74 @@ +"""Live gradient accumulation auto-tuning (v0.32.0 Part B). + +Pure helpers — no torch / GPU touching at module load. The HF Trainer +callback wiring is intentionally left to v0.32.1 because mid-run mutation of +``gradient_accumulation_steps`` requires DataLoader rebuild which is not +safe to do inside ``on_step_end`` without changing TRL internals. + +For v0.32.0 the monitor is wired as an *advisory*: +- It records peak memory each step and flags if pressure is above + ``threshold`` (default 0.92 of total VRAM). +- It returns a recommended ``(batch, accum)`` pair the user can apply on + the next run (also surfaced by the autopilot extension). +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +MIN_THRESHOLD = 0.05 +MAX_THRESHOLD = 0.99 +MAX_ACCUM = 1024 # HF Trainer accepts higher but DataLoader prefetch degrades + + +@dataclass +class GradAccumMonitor: + """Tracks VRAM pressure and recommends batch / accum adjustments.""" + + total_vram_gb: float + threshold: float = 0.92 + peak_used_gb: float = field(default=0.0, init=False) + + def __post_init__(self) -> None: + if not (self.total_vram_gb > 0): + raise ValueError( + f"total_vram_gb must be > 0, got {self.total_vram_gb}" + ) + if not (MIN_THRESHOLD < self.threshold < MAX_THRESHOLD): + raise ValueError( + f"threshold must be in ({MIN_THRESHOLD}, {MAX_THRESHOLD}), " + f"got {self.threshold}" + ) + + def observe(self, used_vram_gb: float) -> None: + """Record a memory observation; updates running peak.""" + if used_vram_gb < 0: + raise ValueError(f"used_vram_gb must be >= 0, got {used_vram_gb}") + if used_vram_gb > self.peak_used_gb: + self.peak_used_gb = used_vram_gb + + def should_adjust(self, used_vram_gb: float) -> bool: + """True when used VRAM crosses the pressure threshold.""" + if used_vram_gb < 0: + raise ValueError(f"used_vram_gb must be >= 0, got {used_vram_gb}") + return (used_vram_gb / self.total_vram_gb) >= self.threshold + + def recommend( + self, current_batch: int, current_accum: int, + ) -> tuple[int, int]: + """Halve batch and double accum, preserving effective batch. + + Floors batch at 1 — if batch is already 1, accum is left untouched. + Caps the new accum at ``MAX_ACCUM`` (1024); past that DataLoader + prefetch degrades and the user is better off cutting ``max_length`` + or moving to gradient checkpointing. + """ + if current_batch < 1 or current_accum < 1: + raise ValueError( + f"current_batch and current_accum must be >= 1, " + f"got ({current_batch}, {current_accum})" + ) + if current_batch == 1: + return current_batch, current_accum + new_accum = min(current_accum * 2, MAX_ACCUM) + return current_batch // 2, new_accum diff --git a/soup_cli/utils/lr_finder.py b/soup_cli/utils/lr_finder.py new file mode 100644 index 0000000..ee5d555 --- /dev/null +++ b/soup_cli/utils/lr_finder.py @@ -0,0 +1,159 @@ +"""LR Range Finder (v0.32.0 Part A) — fast.ai-style sweep. + +Pure helpers — runs no actual training. The driver in ``commands/train.py`` +plugs them into a short HF Trainer loop and writes a JSON report. +""" + +from __future__ import annotations + +import json +import math +from pathlib import Path +from typing import Optional, Sequence, TypedDict + +from soup_cli.utils.paths import is_under_cwd + + +class LRFinderResult(TypedDict): + """Structured result from ``find_optimal_lr``.""" + + recommended_lr: float + min_loss_lr: float + diverged_at: Optional[float] + smoothed_losses: list[float] + +# Bounds prevent runaway sweeps and silly inputs. +MAX_NUM_STEPS = 10_000 +MIN_NUM_STEPS = 2 +DIVERGENCE_FACTOR = 4.0 +SMOOTHING_BETA = 0.98 + + +def compute_lr_schedule( + start_lr: float, end_lr: float, num_steps: int, +) -> list[float]: + """Geometric (log-linear) LR sweep from ``start_lr`` to ``end_lr``.""" + if not (start_lr > 0 and math.isfinite(start_lr)): + raise ValueError(f"start_lr must be positive finite, got {start_lr}") + if not (end_lr > 0 and math.isfinite(end_lr)): + raise ValueError(f"end_lr must be positive finite, got {end_lr}") + if end_lr <= start_lr: + raise ValueError(f"end_lr ({end_lr}) must be > start_lr ({start_lr})") + if num_steps < MIN_NUM_STEPS or num_steps > MAX_NUM_STEPS: + raise ValueError( + f"num_steps must be in [{MIN_NUM_STEPS}, {MAX_NUM_STEPS}], got {num_steps}" + ) + log_start = math.log(start_lr) + log_end = math.log(end_lr) + step = (log_end - log_start) / (num_steps - 1) + return [math.exp(log_start + i * step) for i in range(num_steps)] + + +def _smooth(losses: Sequence[float], beta: float = SMOOTHING_BETA) -> list[float]: + """Exponential moving average with bias correction (Smith 2017).""" + smoothed: list[float] = [] + avg = 0.0 + for index, loss in enumerate(losses, start=1): + avg = beta * avg + (1 - beta) * loss + smoothed.append(avg / (1 - beta ** index)) + return smoothed + + +def find_optimal_lr( + lrs: Sequence[float], losses: Sequence[float], +) -> LRFinderResult: + """Pick the LR with the steepest negative gradient before divergence. + + Edge case: when the smoothed loss is monotonically increasing from the + start (``min_idx <= 1``), there is no meaningful descent region. The + function returns the lowest LR (``lrs[0]``) as ``recommended_lr`` so + callers get a deterministic fallback rather than ``None``. + """ + if len(lrs) != len(losses): + raise ValueError( + f"lrs and losses must have equal length (got {len(lrs)} vs {len(losses)})" + ) + if len(lrs) < 4: + raise ValueError(f"Need at least 4 (lr, loss) pairs, got {len(lrs)}") + + smoothed = _smooth(losses) + + # Find min smoothed loss. + min_idx = min(range(len(smoothed)), key=lambda index: smoothed[index]) + min_loss_lr = lrs[min_idx] + + # Detect divergence: first index after min where |loss| > DIVERGENCE_FACTOR * |min|. + # ``abs`` keeps the check correct if a custom log-prob style loss goes negative. + diverged_at: Optional[float] = None + threshold = abs(smoothed[min_idx]) * DIVERGENCE_FACTOR + for index in range(min_idx + 1, len(smoothed)): + if abs(smoothed[index]) > threshold: + diverged_at = lrs[index] + break + + # Compute steepest negative gradient (in log-LR space) up to min_idx. + upper = max(min_idx, 1) + best_grad = 0.0 + best_idx = 0 + for index in range(1, upper + 1): + d_lr = math.log(lrs[index]) - math.log(lrs[index - 1]) + d_loss = smoothed[index] - smoothed[index - 1] + if d_lr > 0: + grad = d_loss / d_lr + if grad < best_grad: + best_grad = grad + best_idx = index + + # Step back one — recommend LR slightly before the steepest descent end. + rec_idx = max(0, best_idx - 1) if best_grad < 0 else 0 + recommended_lr = lrs[rec_idx] + + return { + "recommended_lr": recommended_lr, + "min_loss_lr": min_loss_lr, + "diverged_at": diverged_at, + "smoothed_losses": smoothed, + } + + +def _finite_or_reject(values: Sequence[float], label: str) -> list[float]: + """Reject NaN / Infinity floats so the JSON report is parser-safe.""" + cleaned: list[float] = [] + for value in values: + as_float = float(value) + if not math.isfinite(as_float): + raise ValueError( + f"{label} contains non-finite value ({value!r}); " + "NaN / Infinity are rejected to keep the JSON report valid." + ) + cleaned.append(as_float) + return cleaned + + +def save_lr_finder_report( + lrs: Sequence[float], losses: Sequence[float], output_path: Path | str, +) -> None: + """Write a JSON report with the sweep + recommended LR.""" + output = Path(output_path) + if not is_under_cwd(output): + raise ValueError(f"Report path must stay under cwd: {output}") + + report_lrs = _finite_or_reject(lrs, "lrs") + report_losses = _finite_or_reject(losses, "losses") + summary = find_optimal_lr(report_lrs, report_losses) + payload = { + "lrs": report_lrs, + "losses": report_losses, + "smoothed_losses": summary["smoothed_losses"], + "recommended_lr": summary["recommended_lr"], + "min_loss_lr": summary["min_loss_lr"], + "diverged_at": summary["diverged_at"], + } + output.parent.mkdir(parents=True, exist_ok=True) + # ``allow_nan=False`` is belt-and-braces: report_* are already finite, + # but ``smoothed_losses`` could carry a non-finite if the input loss + # somehow drifted. Reject rather than emit ``NaN`` (invalid JSON). + output.write_text( + json.dumps(payload, indent=2, allow_nan=False), + encoding="utf-8", + ) diff --git a/soup_cli/utils/mixed_precision.py b/soup_cli/utils/mixed_precision.py new file mode 100644 index 0000000..7d779d6 --- /dev/null +++ b/soup_cli/utils/mixed_precision.py @@ -0,0 +1,81 @@ +"""Auto mixed-precision picker (v0.32.0 Part C). + +Maps a model + GPU compute capability to the best mixed-precision setting: +``bf16`` (Ampere+, most modern models), ``fp16`` (Turing or known +fp16-stable models), or ``no`` (Pascal and older). + +Quirk map is keyed on a lower-cased substring of the model id so we catch +both ``Qwen/Qwen2-7B-Instruct`` and ``alibaba/Qwen2.5-3B`` with one entry. +""" + +from __future__ import annotations + +from typing import Literal + +# Compute capability thresholds. +BF16_MIN_CC = 8.0 # Ampere +FP16_MIN_CC = 6.0 # Pascal +MAX_MODEL_NAME_LEN = 200 + +# Known fp16-stable / bf16-unstable model families. +# Value: preferred precision when CC supports it. +KNOWN_PRECISION_QUIRKS: dict[str, str] = { + "qwen2": "fp16", + "qwen2.5": "fp16", + "phi-3": "fp16", + "phi-3.5": "fp16", + "phi-4": "bf16", + "gemma-2": "bf16", + "mistral": "bf16", + "llama-3": "bf16", + "llama-2": "bf16", +} + + +def pick_mixed_precision( + model_name: str, compute_capability: float, +) -> Literal["bf16", "fp16", "no"]: + """Pick mixed-precision mode for a model + GPU. + + - cc < 6.0 → ``"no"`` (Pascal lacks reliable fp16 tensor cores) + - cc < 8.0 → ``"fp16"`` (no bf16 support) + - cc >= 8.0 → ``"bf16"`` unless model is in the fp16-stable quirks map + """ + if not isinstance(model_name, str) or not model_name: + raise ValueError("model_name must be a non-empty string") + if "\x00" in model_name: + raise ValueError("model_name must not contain null bytes") + if len(model_name) > MAX_MODEL_NAME_LEN: + raise ValueError( + f"model_name must be <= {MAX_MODEL_NAME_LEN} chars, got {len(model_name)}" + ) + if not isinstance(compute_capability, (int, float)): + raise ValueError( + f"compute_capability must be a number, got {type(compute_capability)}" + ) + if compute_capability < 0: + raise ValueError( + f"compute_capability must be non-negative, got {compute_capability}" + ) + + if compute_capability < FP16_MIN_CC: + return "no" + + name_lc = model_name.lower() + quirk: str | None = None + # Sort longer substrings first so multi-version pairs work correctly: + # ``qwen2.5`` wins over ``qwen2``, ``phi-3.5`` wins over ``phi-3``. + # If you add a new family with multiple versions, the longest substring + # always wins — no manual ordering of the dict is required. + for substring in sorted(KNOWN_PRECISION_QUIRKS, key=len, reverse=True): + if substring in name_lc: + quirk = KNOWN_PRECISION_QUIRKS[substring] + break + + if compute_capability < BF16_MIN_CC: + # Turing / Volta cannot do bf16 — fall back to fp16 regardless. + return "fp16" + + if quirk is None: + return "bf16" + return "bf16" if quirk == "bf16" else "fp16" diff --git a/soup_cli/utils/spike_recovery.py b/soup_cli/utils/spike_recovery.py new file mode 100644 index 0000000..2ba433c --- /dev/null +++ b/soup_cli/utils/spike_recovery.py @@ -0,0 +1,54 @@ +"""Loss spike auto-recovery strategy (v0.32.0 Part E). + +When the watchdog fires: +1. If ``attempts < max_attempts`` and a checkpoint exists: rollback, + decay the LR, resume. +2. Otherwise: stop training (existing watchdog behaviour). + +This module provides the policy. Live trainer-state mutation +(rollback + LR change + resume) is wired in via the existing +``SoupTrainerCallback`` in v0.32.0; the auto-rollback to checkpoint is +advisory in v0.32.0 (issues warning + recommends manual resume), with +full live wiring tracked for v0.32.1 — the same pattern used by +v0.30.0 ``--auto-quant`` and v0.30.0 structured-output flags. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +MIN_LR_FLOOR = 1e-9 +MAX_ATTEMPTS_CAP = 10 + + +@dataclass(frozen=True) +class SpikeRecoveryStrategy: + """Policy for loss-spike recovery decisions.""" + + max_attempts: int = 3 + lr_decay: float = 0.5 + min_lr: float = MIN_LR_FLOOR + + def __post_init__(self) -> None: + if not (1 <= self.max_attempts <= MAX_ATTEMPTS_CAP): + raise ValueError( + f"max_attempts must be in [1, {MAX_ATTEMPTS_CAP}], " + f"got {self.max_attempts}" + ) + if not (0 < self.lr_decay < 1): + raise ValueError( + f"lr_decay must be in (0, 1), got {self.lr_decay}" + ) + if self.min_lr <= 0: + raise ValueError(f"min_lr must be > 0, got {self.min_lr}") + + def should_recover(self, attempts: int) -> bool: + """True when the recovery budget hasn't been exhausted.""" + return attempts < self.max_attempts + + def compute_new_lr(self, current_lr: float) -> float: + """Decay the LR by ``lr_decay``, floored at ``min_lr``.""" + if current_lr <= 0: + raise ValueError(f"current_lr must be > 0, got {current_lr}") + new = current_lr * self.lr_decay + return max(new, self.min_lr) diff --git a/soup_cli/utils/warmup.py b/soup_cli/utils/warmup.py new file mode 100644 index 0000000..f542d28 --- /dev/null +++ b/soup_cli/utils/warmup.py @@ -0,0 +1,50 @@ +"""Auto warmup scheduling (v0.32.0 Part D). + +Computes a sensible ``warmup_steps`` from dataset size, batch, grad_accum, +epochs, and a warmup ratio. Clamped to a safe range so users with tiny +datasets still get some warmup, and users with huge datasets don't burn +half a million steps doing nothing useful. +""" + +from __future__ import annotations + +import math + +MIN_WARMUP = 10 +MAX_WARMUP = 1000 +DEFAULT_RATIO = 0.03 +MAX_RATIO = 0.5 + + +def compute_warmup_steps( + num_examples: int, + batch_size: int, + grad_accum: int, + epochs: int, + ratio: float = DEFAULT_RATIO, +) -> int: + """Return warmup steps clamped to [MIN_WARMUP, MAX_WARMUP]. + + Special case: ``ratio == 0`` means "no warmup" and returns 0 (the + schema allows ``warmup_ratio=0.0`` so this matches HF Trainer's + convention). + """ + if num_examples < 1: + raise ValueError(f"num_examples must be >= 1, got {num_examples}") + if batch_size < 1: + raise ValueError(f"batch_size must be >= 1, got {batch_size}") + if grad_accum < 1: + raise ValueError(f"grad_accum must be >= 1, got {grad_accum}") + if epochs < 1: + raise ValueError(f"epochs must be >= 1, got {epochs}") + if not (0.0 <= ratio <= MAX_RATIO): + raise ValueError( + f"ratio must be in [0, {MAX_RATIO}], got {ratio}" + ) + if ratio == 0.0: + return 0 + effective_batch = batch_size * grad_accum + steps_per_epoch = max(1, math.ceil(num_examples / effective_batch)) + total_steps = steps_per_epoch * epochs + raw = int(round(total_steps * ratio)) + return max(MIN_WARMUP, min(MAX_WARMUP, raw)) diff --git a/tests/test_auto_tuning.py b/tests/test_auto_tuning.py new file mode 100644 index 0000000..b10401b --- /dev/null +++ b/tests/test_auto_tuning.py @@ -0,0 +1,849 @@ +"""Tests for v0.32.0 — Training Stability & Auto-Tuning. + +Covers: +- Part A: LR range finder (utils/lr_finder.py) +- Part B: Live grad-accum auto-tuning (utils/grad_accum.py) +- Part C: Auto mixed-precision picker (utils/mixed_precision.py) +- Part D: Warmup auto-schedule (utils/warmup.py) +- Part E: Loss spike auto-recovery (utils/spike_recovery.py + callback wiring) +- Part F: Convergence detector (utils/convergence.py) +- Part G: Autopilot integration (autopilot/decisions.py extensions) +""" + +from __future__ import annotations + +import json +import math +from pathlib import Path + +import pytest +import yaml +from pydantic import ValidationError +from typer.testing import CliRunner + +# --------------------------------------------------------------------------- # +# Part A — LR range finder # +# --------------------------------------------------------------------------- # + +class TestLRFinderSchedule: + """compute_lr_schedule produces a logarithmic LR sweep.""" + + def test_log_schedule_endpoints(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + lrs = compute_lr_schedule(start_lr=1e-7, end_lr=1e-1, num_steps=50) + + assert len(lrs) == 50 + assert lrs[0] == pytest.approx(1e-7, rel=1e-6) + assert lrs[-1] == pytest.approx(1e-1, rel=1e-6) + + def test_log_schedule_monotonic(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + lrs = compute_lr_schedule(1e-6, 1e-1, 30) + for left, right in zip(lrs, lrs[1:]): + assert right > left + + def test_log_schedule_geometric(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + lrs = compute_lr_schedule(1e-6, 1e-1, 11) + ratios = [lrs[i + 1] / lrs[i] for i in range(len(lrs) - 1)] + assert all(math.isclose(ratios[0], r, rel_tol=1e-6) for r in ratios) + + def test_schedule_rejects_non_positive_start(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + with pytest.raises(ValueError, match="start_lr"): + compute_lr_schedule(0.0, 1e-1, 10) + + def test_schedule_rejects_inverted_range(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + with pytest.raises(ValueError, match="end_lr"): + compute_lr_schedule(1e-2, 1e-4, 10) + + def test_schedule_rejects_too_few_steps(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + with pytest.raises(ValueError, match="num_steps"): + compute_lr_schedule(1e-7, 1e-1, 1) + + def test_schedule_caps_steps(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + with pytest.raises(ValueError, match="num_steps"): + compute_lr_schedule(1e-7, 1e-1, 100_000) + + def test_schedule_min_steps_accepted(self): + from soup_cli.utils.lr_finder import compute_lr_schedule + + lrs = compute_lr_schedule(1e-7, 1e-1, 2) + assert len(lrs) == 2 + assert lrs[0] == pytest.approx(1e-7, rel=1e-6) + assert lrs[-1] == pytest.approx(1e-1, rel=1e-6) + + +class TestLRFinderRecommendation: + """find_optimal_lr picks the LR with steepest negative gradient + (excluding the explosion tail).""" + + def test_picks_steepest_descent(self): + from soup_cli.utils.lr_finder import find_optimal_lr + + # Loss decreases through 1e-3 then explodes + lrs = [1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1] + losses = [3.0, 2.8, 2.4, 1.5, 4.0, 12.0] + + result = find_optimal_lr(lrs, losses) + + assert "recommended_lr" in result + assert "min_loss_lr" in result + assert "diverged_at" in result + # Recommended must be <= min_loss_lr (steepest descent comes earlier) + assert result["recommended_lr"] <= result["min_loss_lr"] + # Recommended LR must be in the descent region, not the explosion tail + assert result["recommended_lr"] < 1e-2 + + def test_returns_smoothed_curve(self): + from soup_cli.utils.lr_finder import find_optimal_lr + + lrs = [1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1] + losses = [3.0, 2.8, 2.4, 1.5, 4.0, 12.0] + + result = find_optimal_lr(lrs, losses) + assert "smoothed_losses" in result + assert len(result["smoothed_losses"]) == len(losses) + + def test_detects_divergence(self): + from soup_cli.utils.lr_finder import find_optimal_lr + + lrs = [1e-6, 1e-5, 1e-4, 1e-3, 1e-2] + losses = [3.0, 2.8, 2.4, 1.5, 100.0] + + result = find_optimal_lr(lrs, losses) + assert result["diverged_at"] is not None + assert result["diverged_at"] <= 1e-2 + + def test_no_divergence_when_loss_stable(self): + from soup_cli.utils.lr_finder import find_optimal_lr + + lrs = [1e-6, 1e-5, 1e-4, 1e-3] + losses = [3.0, 2.8, 2.6, 2.4] + + result = find_optimal_lr(lrs, losses) + assert result["diverged_at"] is None + + def test_mismatched_lengths_rejected(self): + from soup_cli.utils.lr_finder import find_optimal_lr + + with pytest.raises(ValueError, match="length"): + find_optimal_lr([1e-6, 1e-5], [3.0, 2.8, 2.6]) + + def test_too_few_points_rejected(self): + from soup_cli.utils.lr_finder import find_optimal_lr + + with pytest.raises(ValueError, match="at least"): + find_optimal_lr([1e-6, 1e-5], [3.0, 2.8]) + + def test_monotonic_increase_falls_back_to_first_lr(self): + from soup_cli.utils.lr_finder import find_optimal_lr + + lrs = [1e-6, 1e-5, 1e-4, 1e-3] + losses = [1.0, 2.0, 4.0, 8.0] # explodes immediately + result = find_optimal_lr(lrs, losses) + assert result["recommended_lr"] == pytest.approx(lrs[0], rel=1e-6) + + +class TestLRFinderCLI: + """`soup train --find-lr` flag is registered and surfaces in help.""" + + def test_flag_in_help(self): + from soup_cli.cli import app + + runner = CliRunner() + result = runner.invoke(app, ["train", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--find-lr" in result.output + + +# --------------------------------------------------------------------------- # +# Part B — Live grad-accum auto-tuning # +# --------------------------------------------------------------------------- # + +class TestGradAccumMonitor: + def test_should_adjust_when_high_pressure(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0, threshold=0.92) + # 23.5/24 = 0.979 → above 0.92 threshold + assert mon.should_adjust(used_vram_gb=23.5) is True + + def test_should_not_adjust_when_low_pressure(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0, threshold=0.92) + assert mon.should_adjust(used_vram_gb=10.0) is False + + def test_recommend_doubles_grad_accum(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + new_batch, new_accum = mon.recommend(current_batch=8, current_accum=2) + assert new_batch == 4 + assert new_accum == 4 + + def test_recommend_keeps_effective_batch(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + for batch, accum in [(16, 1), (8, 4), (32, 1)]: + new_batch, new_accum = mon.recommend(batch, accum) + assert new_batch * new_accum == batch * accum + + def test_recommend_floor_at_one(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + new_batch, new_accum = mon.recommend(current_batch=1, current_accum=8) + assert new_batch == 1 + assert new_accum == 8 + + def test_recommend_caps_accum_at_max(self): + from soup_cli.utils.grad_accum import MAX_ACCUM, GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + # Push accum past the cap; new should equal MAX_ACCUM. + _, new_accum = mon.recommend(current_batch=2, current_accum=MAX_ACCUM) + assert new_accum == MAX_ACCUM + + def test_observe_updates_peak(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + mon.observe(10.0) + mon.observe(15.0) + mon.observe(12.0) + assert mon.peak_used_gb == pytest.approx(15.0) + + def test_observe_rejects_negative(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + with pytest.raises(ValueError, match="used_vram_gb"): + mon.observe(-1.0) + + def test_should_adjust_rejects_negative(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + with pytest.raises(ValueError, match="used_vram_gb"): + mon.should_adjust(-0.1) + + def test_recommend_rejects_zero_inputs(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + mon = GradAccumMonitor(total_vram_gb=24.0) + with pytest.raises(ValueError, match="current_batch"): + mon.recommend(current_batch=0, current_accum=2) + with pytest.raises(ValueError, match="current_batch"): + mon.recommend(current_batch=2, current_accum=0) + + def test_threshold_bounds(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + with pytest.raises(ValueError, match="threshold"): + GradAccumMonitor(total_vram_gb=24.0, threshold=1.5) + with pytest.raises(ValueError, match="threshold"): + GradAccumMonitor(total_vram_gb=24.0, threshold=0.0) + + def test_total_vram_must_be_positive(self): + from soup_cli.utils.grad_accum import GradAccumMonitor + + with pytest.raises(ValueError, match="total_vram_gb"): + GradAccumMonitor(total_vram_gb=-1.0) + + +# --------------------------------------------------------------------------- # +# Part C — Auto mixed-precision picker # +# --------------------------------------------------------------------------- # + +class TestMixedPrecisionPicker: + def test_llama_ampere_picks_bf16(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + assert pick_mixed_precision("meta-llama/Llama-3-8B", 8.0) == "bf16" + + def test_qwen2_picks_fp16(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + assert pick_mixed_precision("Qwen/Qwen2-7B", 8.0) == "fp16" + + def test_pre_ampere_drops_to_fp16(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + # Turing T4 (cc=7.5) — no bf16 + assert pick_mixed_precision("meta-llama/Llama-3-8B", 7.5) == "fp16" + + def test_pre_pascal_returns_no(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + # cc<6.0 doesn't support fp16 reliably + assert pick_mixed_precision("meta-llama/Llama-3-8B", 5.0) == "no" + + def test_unknown_model_default_bf16_on_ampere(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + assert pick_mixed_precision("some/unknown-model", 9.0) == "bf16" + + def test_invalid_model_name_rejected(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + with pytest.raises(ValueError, match="model"): + pick_mixed_precision("", 8.0) + with pytest.raises(ValueError, match="model"): + pick_mixed_precision("a\x00b", 8.0) + with pytest.raises(ValueError, match="200"): + pick_mixed_precision("a" * 201, 8.0) + + def test_qwen25_picks_fp16_not_qwen2_default(self): + """Longer 'qwen2.5' substring must win over 'qwen2'.""" + from soup_cli.utils.mixed_precision import pick_mixed_precision + + # Both entries map to fp16 today, but the test guards the iteration + # order: if someone changes qwen2.5 to bf16, this catches it. + assert pick_mixed_precision("Qwen/Qwen2.5-7B", 8.0) == "fp16" + + def test_negative_cc_rejected(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + with pytest.raises(ValueError, match="compute_capability"): + pick_mixed_precision("meta-llama/Llama-3-8B", -1.0) + + def test_known_quirk_mapping_includes_qwen_and_phi(self): + from soup_cli.utils.mixed_precision import KNOWN_PRECISION_QUIRKS + + keys = [k.lower() for k in KNOWN_PRECISION_QUIRKS] + assert any("qwen" in k for k in keys) + assert any("phi" in k for k in keys) + + def test_cc_exactly_fp16_boundary(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + # cc == 6.0 should accept fp16 (the comparison is `< FP16_MIN_CC`) + assert pick_mixed_precision("meta-llama/Llama-3-8B", 6.0) == "fp16" + + def test_cc_exactly_bf16_boundary(self): + from soup_cli.utils.mixed_precision import pick_mixed_precision + + # cc == 8.0 should pick bf16 (the comparison is `< BF16_MIN_CC`) + assert pick_mixed_precision("meta-llama/Llama-3-8B", 8.0) == "bf16" + + +# --------------------------------------------------------------------------- # +# Part D — Warmup auto-schedule # +# --------------------------------------------------------------------------- # + +class TestWarmupAutoSchedule: + def test_basic_formula(self): + from soup_cli.utils.warmup import compute_warmup_steps + + # 10000 examples / batch 4 / accum 2 / 3 epochs = 3750 update steps + # 3% = 112 steps + steps = compute_warmup_steps( + num_examples=10000, + batch_size=4, + grad_accum=2, + epochs=3, + ratio=0.03, + ) + assert 100 <= steps <= 130 + + def test_clamps_to_min(self): + from soup_cli.utils.warmup import compute_warmup_steps + + steps = compute_warmup_steps( + num_examples=10, batch_size=1, grad_accum=1, epochs=1, ratio=0.03, + ) + assert steps >= 10 # MIN_WARMUP + + def test_clamps_to_max(self): + from soup_cli.utils.warmup import compute_warmup_steps + + steps = compute_warmup_steps( + num_examples=10_000_000, + batch_size=1, + grad_accum=1, + epochs=1, + ratio=0.03, + ) + assert steps <= 1000 # MAX_WARMUP + + def test_invalid_ratio_rejected(self): + from soup_cli.utils.warmup import compute_warmup_steps + + for bad_ratio in [-0.01, 0.51]: + with pytest.raises(ValueError, match="ratio"): + compute_warmup_steps( + num_examples=1000, batch_size=1, grad_accum=1, epochs=1, + ratio=bad_ratio, + ) + + def test_ratio_zero_means_no_warmup(self): + from soup_cli.utils.warmup import compute_warmup_steps + + steps = compute_warmup_steps( + num_examples=1000, batch_size=1, grad_accum=1, epochs=1, ratio=0.0, + ) + assert steps == 0 + + def test_invalid_inputs_rejected(self): + from soup_cli.utils.warmup import compute_warmup_steps + + with pytest.raises(ValueError, match="num_examples"): + compute_warmup_steps(num_examples=0, batch_size=1, grad_accum=1, epochs=1) + with pytest.raises(ValueError, match="batch_size"): + compute_warmup_steps(num_examples=10, batch_size=0, grad_accum=1, epochs=1) + with pytest.raises(ValueError, match="grad_accum"): + compute_warmup_steps(num_examples=10, batch_size=1, grad_accum=0, epochs=1) + with pytest.raises(ValueError, match="epochs"): + compute_warmup_steps(num_examples=10, batch_size=1, grad_accum=1, epochs=0) + + def test_ratio_at_max_accepted(self): + from soup_cli.utils.warmup import MAX_WARMUP, compute_warmup_steps + + # ratio == MAX_RATIO (0.5) is the inclusive upper bound. + steps = compute_warmup_steps( + num_examples=100_000, batch_size=1, grad_accum=1, epochs=1, ratio=0.5, + ) + assert steps == MAX_WARMUP + + +class TestWarmupConfigField: + def test_warmup_auto_default_false(self): + from soup_cli.config.schema import TrainingConfig + + cfg = TrainingConfig() + assert cfg.warmup_auto is False + + def test_warmup_auto_can_be_set(self): + from soup_cli.config.schema import TrainingConfig + + cfg = TrainingConfig(warmup_auto=True) + assert cfg.warmup_auto is True + + +# --------------------------------------------------------------------------- # +# Part E — Loss spike auto-recovery # +# --------------------------------------------------------------------------- # + +class TestSpikeRecoveryStrategy: + def test_should_recover_within_budget(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5) + assert strat.should_recover(attempts=0) is True + assert strat.should_recover(attempts=2) is True + + def test_should_not_recover_at_limit(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5) + assert strat.should_recover(attempts=3) is False + assert strat.should_recover(attempts=4) is False + + def test_compute_new_lr_decays(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5) + assert strat.compute_new_lr(2e-4) == pytest.approx(1e-4) + assert strat.compute_new_lr(1e-3) == pytest.approx(5e-4) + + def test_lr_decay_bounds(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + with pytest.raises(ValueError, match="lr_decay"): + SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.0) + with pytest.raises(ValueError, match="lr_decay"): + SpikeRecoveryStrategy(max_attempts=3, lr_decay=1.0) + + def test_max_attempts_bounds(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + with pytest.raises(ValueError, match="max_attempts"): + SpikeRecoveryStrategy(max_attempts=0) + with pytest.raises(ValueError, match="max_attempts"): + SpikeRecoveryStrategy(max_attempts=100) + + def test_minimum_lr_floor(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5, min_lr=1e-7) + # New LR is below floor → return floor + assert strat.compute_new_lr(1e-8) == pytest.approx(1e-7) + + def test_compute_new_lr_rejects_non_positive(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + strat = SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5) + with pytest.raises(ValueError, match="current_lr"): + strat.compute_new_lr(0.0) + with pytest.raises(ValueError, match="current_lr"): + strat.compute_new_lr(-1e-4) + + def test_min_lr_must_be_positive(self): + from soup_cli.utils.spike_recovery import SpikeRecoveryStrategy + + with pytest.raises(ValueError, match="min_lr"): + SpikeRecoveryStrategy(max_attempts=3, lr_decay=0.5, min_lr=0.0) + + +class TestSpikeRecoveryConfig: + def test_field_default(self): + from soup_cli.config.schema import TrainingConfig + + cfg = TrainingConfig() + assert cfg.loss_spike_recovery is False + assert cfg.loss_spike_recovery_max_attempts == 3 + + def test_max_attempts_bounds(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(loss_spike_recovery_max_attempts=0) + with pytest.raises(ValidationError): + TrainingConfig(loss_spike_recovery_max_attempts=100) + + def test_recovery_requires_watchdog(self): + from soup_cli.config.schema import TrainingConfig + + # Recovery without watchdog enabled is rejected. + with pytest.raises(ValidationError) as exc_info: + TrainingConfig(loss_spike_recovery=True, loss_watchdog=False) + messages = [err["msg"] for err in exc_info.value.errors()] + assert any( + "loss_watchdog" in msg and "loss_spike_recovery" in msg + for msg in messages + ), messages + + +# --------------------------------------------------------------------------- # +# Part F — Convergence detector # +# --------------------------------------------------------------------------- # + +class TestConvergenceDetector: + def test_detects_plateau(self): + from soup_cli.utils.convergence import detect_plateau + + # Last 50 losses essentially flat + losses = [3.0 - 0.001 * i for i in range(150)] + [2.85] * 50 + assert detect_plateau(losses, window=50, rel_tol=0.005) is True + + def test_does_not_detect_when_decreasing(self): + from soup_cli.utils.convergence import detect_plateau + + losses = [3.0 - 0.005 * i for i in range(200)] + assert detect_plateau(losses, window=50, rel_tol=0.005) is False + + def test_too_few_points(self): + from soup_cli.utils.convergence import detect_plateau + + losses = [3.0, 2.9, 2.8] + assert detect_plateau(losses, window=50, rel_tol=0.005) is False + + def test_window_bounds(self): + from soup_cli.utils.convergence import detect_plateau + + with pytest.raises(ValueError, match="window"): + detect_plateau([3.0] * 100, window=0) + with pytest.raises(ValueError, match="window"): + detect_plateau([3.0] * 100, window=10001) + + def test_rel_tol_bounds(self): + from soup_cli.utils.convergence import detect_plateau + + with pytest.raises(ValueError, match="rel_tol"): + detect_plateau([3.0] * 100, window=50, rel_tol=-0.001) + with pytest.raises(ValueError, match="rel_tol"): + detect_plateau([3.0] * 100, window=50, rel_tol=2.0) + + +class TestRecommendAction: + def test_recommends_continue_for_decreasing(self): + from soup_cli.utils.convergence import recommend_action + + losses = [3.0 - 0.005 * i for i in range(200)] + assert recommend_action(losses) == "continue" + + def test_recommends_early_stop_for_long_plateau(self): + from soup_cli.utils.convergence import recommend_action + + losses = [3.0 - 0.001 * i for i in range(150)] + [2.85] * 100 + assert recommend_action(losses) == "early_stop" + + def test_recommends_lower_lr_for_oscillation(self): + # Oscillating losses (high variance, no trend) + import random + + from soup_cli.utils.convergence import recommend_action + + rng = random.Random(42) + losses = [2.5 + rng.uniform(-0.4, 0.4) for _ in range(200)] + action = recommend_action(losses) + assert action in {"lower_lr", "continue"} + + def test_too_few_points_returns_continue(self): + from soup_cli.utils.convergence import recommend_action + + assert recommend_action([3.0, 2.9, 2.8]) == "continue" + + +class TestConvergenceConfig: + def test_field_defaults(self): + from soup_cli.config.schema import TrainingConfig + + cfg = TrainingConfig() + assert cfg.convergence_detection is False + + def test_convergence_window_lower_bound(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(convergence_window=4) + + def test_convergence_window_upper_bound(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(convergence_window=10_001) + + def test_convergence_rel_tol_upper_bound(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(convergence_rel_tol=1.1) + + def test_convergence_rel_tol_zero_rejected(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(convergence_rel_tol=0.0) + + +class TestRecoveryFieldBounds: + def test_lr_decay_upper_bound(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(loss_spike_recovery_lr_decay=1.0) + + def test_lr_decay_lower_bound(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(loss_spike_recovery_lr_decay=0.0) + + +class TestGradAccumThresholdField: + def test_pressure_threshold_upper_bound(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(grad_accum_pressure_threshold=0.99) + + def test_pressure_threshold_lower_bound(self): + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(grad_accum_pressure_threshold=0.05) + + +class TestPlateauNonPositiveMean: + def test_plateau_non_positive_mean_returns_false(self): + from soup_cli.utils.convergence import detect_plateau + + # All-negative losses → mean < 0 → refuse to assess. + losses = [-2.0] * 60 + assert detect_plateau(losses, window=50, rel_tol=0.005) is False + + def test_recommend_action_non_positive_mean_returns_continue(self): + from soup_cli.utils.convergence import recommend_action + + losses = [-3.0 - 0.001 * i for i in range(200)] + assert recommend_action(losses) == "continue" + + +# --------------------------------------------------------------------------- # +# Part G — Autopilot integration # +# --------------------------------------------------------------------------- # + +class TestAutopilotIntegration: + def test_decide_warmup_returns_int(self): + from soup_cli.autopilot.decisions import decide_warmup + + steps = decide_warmup( + num_examples=10000, batch_size=4, grad_accum=2, epochs=3, + ) + assert isinstance(steps, int) + assert steps > 0 + + def test_decide_mixed_precision_routes_to_picker(self): + from soup_cli.autopilot.decisions import decide_mixed_precision + + prec = decide_mixed_precision("meta-llama/Llama-3-8B", 8.0) + assert prec in {"bf16", "fp16", "no"} + + def test_decide_mixed_precision_invalid_inputs(self): + from soup_cli.autopilot.decisions import decide_mixed_precision + + with pytest.raises(ValueError): + decide_mixed_precision("", 8.0) + + +class TestAutopilotConfigEmission: + def test_generated_config_includes_warmup_auto_when_set( + self, tmp_path, monkeypatch, + ): + """Generated config has warmup_auto=true so train.py picks it up.""" + from soup_cli.autopilot.generate_config import generate_config + + monkeypatch.chdir(tmp_path) + decisions = { + "task": "sft", + "format": "alpaca", + "max_length": 2048, + "quantization": "4bit", + "lora": {"r": 16, "alpha": 32, "use_dora": False}, + "lr": 2e-4, + "epochs": 3, + "batch_size": 4, + "grad_accum": 2, + "perf": { + "use_flash_attn": True, "use_liger": True, + "gradient_checkpointing": False, + }, + "warmup_auto": True, + "mixed_precision": "bf16", + } + out = Path("soup.yaml") + generate_config( + base="meta-llama/Llama-3-8B", + data_path="data.jsonl", + decisions=decisions, + output_path=out, + ) + text = out.read_text(encoding="utf-8") + cfg = yaml.safe_load(text) + assert cfg["training"].get("warmup_auto") is True + + def test_decisions_output_must_stay_under_cwd(self, tmp_path, monkeypatch): + from soup_cli.autopilot.generate_config import generate_config + + monkeypatch.chdir(tmp_path) + decisions = { + "task": "sft", "format": "alpaca", "max_length": 1024, + "quantization": "4bit", + "lora": {"r": 8, "alpha": 16, "use_dora": False}, + "lr": 2e-4, "epochs": 1, "batch_size": 1, "grad_accum": 1, + "perf": { + "use_flash_attn": False, "use_liger": False, + "gradient_checkpointing": False, + }, + "output": "/tmp/escape", + } + with pytest.raises(ValueError, match="under cwd"): + generate_config( + base="meta-llama/Llama-3-8B", + data_path="data.jsonl", + decisions=decisions, + output_path=Path("soup.yaml"), + ) + + +# --------------------------------------------------------------------------- # +# Integration: lr-finder CLI smoke (offline-safe) # +# --------------------------------------------------------------------------- # + +class TestLRFinderRunner: + """save_lr_finder_report writes a JSON report users can plot.""" + + def test_save_report(self, tmp_path, monkeypatch): + from soup_cli.utils.lr_finder import save_lr_finder_report + + monkeypatch.chdir(tmp_path) + lrs = [1e-6, 1e-5, 1e-4, 1e-3] + losses = [3.0, 2.8, 2.4, 2.6] + out = Path("lr_report.json") + save_lr_finder_report(lrs, losses, out) + + assert out.exists() + data = json.loads(out.read_text(encoding="utf-8")) + assert data["lrs"] == lrs + assert data["losses"] == losses + assert "recommended_lr" in data + + def test_save_report_rejects_nan(self, tmp_path, monkeypatch): + from soup_cli.utils.lr_finder import save_lr_finder_report + + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="non-finite"): + save_lr_finder_report( + [1e-6, 1e-5, 1e-4, 1e-3], + [3.0, 2.8, float("nan"), 2.4], + Path("report.json"), + ) + + def test_save_report_rejects_infinity(self, tmp_path, monkeypatch): + from soup_cli.utils.lr_finder import save_lr_finder_report + + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="non-finite"): + save_lr_finder_report( + [1e-6, 1e-5, 1e-4, 1e-3], + [3.0, 2.8, 2.4, float("inf")], + Path("report.json"), + ) + + def test_save_report_path_must_stay_under_cwd(self, tmp_path, monkeypatch): + from soup_cli.utils.lr_finder import save_lr_finder_report + + # tmp_path is the inner; we chdir to a subdir so tmp_path itself + # is outside cwd and thus cannot be the target. + inner = tmp_path / "inner" + inner.mkdir() + monkeypatch.chdir(inner) + outside = (tmp_path / "lr_report.json").resolve() + with pytest.raises(ValueError, match="under cwd"): + save_lr_finder_report([1e-6, 1e-5], [3.0, 2.8], outside) + + +# --------------------------------------------------------------------------- # +# Cross-cutting: SoupConfig top-level still serializes # +# --------------------------------------------------------------------------- # + +class TestNewFieldsRoundTrip: + def test_roundtrip_yaml(self): + from soup_cli.config.loader import load_config_from_string + + yaml_text = """ +base: meta-llama/Llama-3-8B +task: sft +data: + train: data.jsonl + format: alpaca +training: + epochs: 3 + lr: 2e-4 + batch_size: 4 + warmup_auto: true + loss_watchdog: true + loss_spike_recovery: true + loss_spike_recovery_max_attempts: 2 + convergence_detection: true +output: ./out +""" + cfg = load_config_from_string(yaml_text) + assert cfg.training.warmup_auto is True + assert cfg.training.loss_spike_recovery is True + assert cfg.training.loss_spike_recovery_max_attempts == 2 + assert cfg.training.convergence_detection is True