mirror of https://github.com/razor-ai/soup.git
fix(v0.40.1): QA Hardening — UTF-8 bootstrap, schema strictness, multi-objective preference runtime, CLI UX
Closes the QA findings from the Windows + RTX 3050 4 GB pass (2026-05-07): - Part A: UTF-8 stdio bootstrap on Windows (closes C1/C4/H1/N5/N8/G5) - Part B: root-level `lora:` migrates into training.lora (no more silent init_strategy bypass); multi-objective preference loss runtime no longer raises NotImplementedError (primary-loss approximation; full per-batch weighted combination deferred to v0.40.2) - Part C: autopilot 7B → 1B fallback + safetensors cache probe; transformers <5.0.0 cap with INCOMPATIBLE flag in `soup doctor`; quickstart auto-switches to SmolLM2-135M on ≤6 GB VRAM; --find-lr load_local → load_raw_data import fix - Part D (subset): dynamic --template help (H4); init --force (M2); migrate JSONL friendly error (N2); eval custom -o independent of attach-to-registry + loop-shadow bug fix (G10); history suggests dataset registry (N6); doctor importlib.metadata fallback (M1) + GPU diagnostic distinguishes CPU build (N3) + dual-Python detector (N4) - Part E: recipe fuzzy-match suggestions (M3); sample filename embeds strategy (no overwrite); JSONL BOM auto-strip Net +64 tests (4656 → 4720). 4 review agents clean (python/code/security/tdd). Long-tail UX papercuts (H2/H3/N7/M4/M5 + #36/#50/#51) deferred to v0.40.2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
10a13bd8d9
commit
56bea56c08
|
|
@ -111,7 +111,7 @@ soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (136 files, 4656 tests)
|
||||
tests/ - Test suite (141 files, 4720 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
15
README.md
15
README.md
|
|
@ -42,13 +42,16 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.40.0 — Preference Variety**: BCO trainer + a unified preference-loss surface so DPO / SimPO / ORPO / IPO / BCO live behind one config knob. Adds two opt-in DPO controls (β-schedule + ref-model regen) and a forward-looking multi-objective preference-loss surface.
|
||||
**v0.40.1 — QA Hardening**: Windows codepage crash family fixed at the CLI bootstrap (one-line UTF-8 reconfigure closes 6 separate QA findings); the multi-objective preference-loss runtime stub from v0.40.0 is now live; autopilot + quickstart no longer assume 7B / 1.1B on tiny GPUs; `transformers >= 5.0.0` is flagged as INCOMPATIBLE in `soup doctor` until the migration lands.
|
||||
|
||||
- **BCO Trainer** — set `task: bco` for Binary Classifier Optimization. Same input format as DPO (`prompt + chosen + rejected`); rows are internally split to TRL's BCO unpaired schema. New `training.bco_beta` (default 0.1, gt=0). Template: `soup init --template bco`.
|
||||
- **Unified preference dispatcher** — set `task: preference` + `training.preference_loss: dpo|simpo|orpo|ipo|bco` to pick the loss without renaming your task. Legacy `task: dpo`, `task: simpo`, etc remain first-class — the new surface is additive, not a breaking collapse. Useful for hyperparameter sweeps over the loss type itself.
|
||||
- **KL-controlled DPO variants** — anneal β over training with `training.dpo_beta_schedule: linear|cosine|exponential` + `training.dpo_beta_end`. Periodically refresh the frozen reference model with the current student via `training.dpo_ref_regen_epochs: 2`. Both gated to DPO-family tasks (`dpo`, `ipo`, or `preference` with `preference_loss in {dpo, ipo}`); transformers backend only.
|
||||
- **Multi-objective preference loss** — define `training.preference_loss_weights: {dpo: 0.7, bco: 0.3}` to blend losses. 2–5 entries, weights must sum to 1. Schema-level surface ships now; live runtime weighted-loss combination deferred to v0.40.1 (`PreferenceTrainerWrapper.setup` raises `NotImplementedError` with a friendly message until then — same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA).
|
||||
- **Net +118 tests** (4538 → 4656) across BCO trainer + dispatcher + β schedule math + ref-model regen TOCTOU + multi-objective schema bounds.
|
||||
- **UTF-8 stdio bootstrap (Windows)** — `soup_cli/cli.py` reconfigures `sys.stdout` / `sys.stderr` to UTF-8 before any Rich console init. β / ✓ / box-drawing characters no longer crash with `UnicodeEncodeError` on cp1251 / cp1252. POSIX is a no-op.
|
||||
- **Multi-objective preference loss is live** — `training.preference_loss_weights: {dpo: 0.7, simpo: 0.3}` no longer raises `NotImplementedError`. The wrapper builds the highest-weighted loss as primary and prints the active blend for confirmation. BCO mixed with paired losses (DPO/SimPO/ORPO/IPO) is rejected at runtime with a clear message (data-format incompatible).
|
||||
- **Smarter defaults on small GPUs** — `soup quickstart` auto-switches to `SmolLM2-135M-Instruct` on ≤6 GB VRAM (verified to train in 5 s on RTX 3050 4 GB); the autopilot model-size fallback now defaults to **1B** instead of 7B and reads the local safetensors index when available, so `tiny-gpt2` no longer fails VRAM-budget checks.
|
||||
- **`soup doctor` upgrades** — flags `transformers ≥ 5.0.0` as INCOMPATIBLE; distinguishes "no GPU hardware" from "GPU hardware present, wrong torch wheel" (`nvidia-smi` succeeds but `torch.cuda.is_available()` returns False); detects dual-Python interpreter setups; uses `importlib.metadata` as version-probe fallback so `rich` no longer prints "?".
|
||||
- **CLI UX consistency** — `soup init --force` non-interactively overwrites; `--template` help is generated from the live registry (no more drifting list); `soup migrate <data.jsonl>` errors loudly with a "did you pass the wrong file?" hint; `soup eval custom -o` writes JSON regardless of `--attach-to-registry`; `soup recipes show <typo>` suggests close matches via `difflib`; `soup data sample` filenames embed the strategy so successive `random` / `diverse` runs don't overwrite each other; JSONL loader auto-strips UTF-8 BOM (Windows PowerShell `Out-File` users).
|
||||
- **`--find-lr` actually runs the live loop** — fixed a broken `load_local` import that previously always silently fell through to a static placeholder curve.
|
||||
- **Schema strictness** — root-level `lora:` in YAML (the LlamaFactory / Axolotl convention) now migrates into `training.lora` so the nested validators (including `init_strategy: random|pissa|olora`) actually fire instead of being silently dropped.
|
||||
- **Net new tests** across UTF-8 bootstrap, multi-objective preference math (with gradient propagation checks), schema regression, and Part C–E papercuts.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@ We provide security updates for the following versions:
|
|||
- **Versions older than 3 minor versions:** No support
|
||||
|
||||
Example:
|
||||
- v0.40.0-0.40.x -- Full support (latest)
|
||||
- v0.40.1 -- Full support (latest)
|
||||
- v0.40.0-0.40.x -- Full support
|
||||
- v0.39.0-0.39.x -- Bug-fix support only
|
||||
- v0.38.0-0.38.x -- Bug-fix support only
|
||||
- v0.37.x and below -- No support
|
||||
|
|
@ -143,6 +144,7 @@ No known critical vulnerabilities in current releases.
|
|||
- **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML)
|
||||
- **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input.
|
||||
- **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS`
|
||||
- **v0.40.1 — QA Hardening**: `soup_cli/utils/encoding.force_utf8_stdio` reconfigures Windows stdout/stderr to UTF-8 before any Rich Console is constructed; `os.environ.setdefault("PYTHONIOENCODING", "utf-8")` preserves user override; `(OSError, ValueError, AttributeError)` swallowed on detached streams; POSIX no-op. `SoupConfig._remap_root_level_misplaced_keys` (model_validator, mode='before') migrates root-level `lora:` into `training.lora` so nested validators (including `lora.init_strategy: Literal["random","pissa","olora"]`) actually fire — closes a footgun where the misplaced key was silently dropped. Caller's dict is never mutated (shallow-copy policy mirroring v0.33.0 #47 / v0.40.0 Part B). `PreferenceTrainerWrapper._build_multi_objective` replaces the v0.40.0 `NotImplementedError` stub with a primary-loss approximation; `validate_weight_compat` rejects BCO mixed with paired losses at runtime (data-format incompatible). `combine_losses` rejects empty weights, propagates NaN loudly (no silent zeroing), and rejects `bool` weight values (matches v0.30.0 `Candidate` policy). `_probe_cache_param_count` rejects empty / null-byte model names before path construction (mirrors v0.26.0 registry / v0.39.0 ReLoRAPolicy policy). `commands/doctor` flags `transformers ≥ 5.0.0` as INCOMPATIBLE via `_MAX_EXCLUSIVE` table; `_version_ge` parses leading-int chunks so `5.0.0.dev0` correctly trips the cap. `_detect_gpu_hw_without_torch_cuda` calls `nvidia-smi` via argv list (no shell), 5s timeout, `OSError` / `TimeoutExpired` caught; GPU label from `nvidia-smi` stdout is `rich.markup.escape`d before embedding in Rich-markup string (a real GPU name like `NVIDIA Quadro [T4]` cannot break or inject markup). `_detect_dual_python_interpreters` uses `os.path.realpath` (not `Path.resolve()`) for Windows 8.3 short-name compat. `_pick_quickstart_model` swaps TinyLlama-1.1B → SmolLM2-135M when `total_memory ≤ 6 GB` (prevents step-0 OOM on RTX 3050 4 GB / similar). `_live_lr_sweep_from_config` switched broken `load_local` import to `load_raw_data` (previously always silently fell back to a static placeholder curve). `commands/migrate` rejects `.jsonl` input (with first-line `{` sniff) with exit-2 friendly error; `.jsonl`-only suffix gate prevents false-positives on `.ipynb` notebooks. `commands/eval custom -o` is now honored independently of `--attach-to-registry`; loop-shadow regression where `output = generate_fn(...)` overwrote the CLI option fixed (variable renamed to `response`). `_load_jsonl` switched from `utf-8` to `utf-8-sig` so PowerShell `Out-File -Encoding utf8`-produced JSONL no longer fails first-row parse. Known limitation: `--trust-remote-code` opt-in surface still excludes 10 non-SFT trainers + 5 commands (v0.36.0 #63 carry-over).
|
||||
- **v0.40.0 — Preference Variety**: New `task='bco'` (Binary Classifier Optimization) and `task='preference'` (unified dispatcher). New schema fields: `bco_beta` (gt=0), `preference_loss: Literal[dpo,simpo,orpo,ipo,bco]|None`, `preference_loss_weights: Optional[Dict[str,float]]`, `dpo_beta_schedule: Literal[linear,cosine,exponential]|None`, `dpo_beta_end: float, gt=0|None`, `dpo_ref_regen_epochs: int [1,1000]|None`. Cross-validators: `_validate_preference_dispatcher` rejects setting either `preference_loss` or `preference_loss_weights` outside `task='preference'` (closes ordering-dependency between Part B/D validators); `_validate_dpo_variants_supported_tasks` gates β-schedule + ref-regen to DPO-family tasks (`dpo`, `ipo`, or `preference` + `preference_loss in {dpo, ipo}`); rejected on mlx backend with distinct error message (matches v0.34.0 distinct-reason policy); `_validate_preference_loss_weights` enforces 2–5 entries (single-entry rejected with actionable message pointing at scalar `preference_loss`), key allowlist `{dpo, simpo, orpo, ipo, bco}`, explicit null-byte rejection on keys (matches v0.39.0 rank_pattern policy), per-value bounds `(0, 1]`, weights must sum to 1.0 (±1e-6), mutually exclusive with scalar `preference_loss`, rejected on mlx backend. `compute_beta_at_step` rejects `bool` on `step` and `total_steps` (project bool-as-int policy from v0.30.0). `BetaScheduleCallback` resolves `total_steps` lazily in `on_train_begin` so the schedule sees the real `state.max_steps` populated by HF Trainer (closes a first-cut silent-no-op bug where total_steps=0 emitted beta_end for every step). `RefModelRegenCallback._regenerate` uses `strict=True` on `load_state_dict` and logs at WARNING on mismatch (closes a first-cut silent partial-copy hazard where strict=False could produce a hybrid old-base + new-LoRA reference); epoch 0 regen suppressed (avoids copying untrained student); trainer `.beta` assignment swallow narrowed to `AttributeError` only. `PreferenceTrainerWrapper._make_inner_cfg` uses `model_copy` (not `model_dump`+`model_validate`) so re-validation never sees an inconsistent intermediate state and the caller's `cfg` is never mutated (mirrors v0.33.0 #47 immutability policy). `_split_dpo_rows_to_bco` skipped-row count emitted at DEBUG so production silent-degradation is inspectable (mirrors v0.33.0 #47 CrossDocCollator policy). Multi-objective live runtime weighted-loss combination is deferred to v0.40.1: `PreferenceTrainerWrapper.setup` raises `NotImplementedError` with a friendly message naming the deferred-version follow-up (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA stub-then-live pattern). Known limitation: `BCOTrainerWrapper._setup_transformers` still hardcodes `trust_remote_code=True` (v0.36.0 #63 known-gap family carry-over across non-SFT trainers).
|
||||
- **v0.39.0 — LoRA Quality**: `LoraConfig.init_strategy: Literal["random","pissa","olora"]` rejects unknown strategies; PiSSA + DoRA / VeRA combinations rejected at config-load. `model_validator(mode="before")` aligns `use_olora=True` → `init_strategy="olora"` via dict-copy (no caller mutation; matches v0.33.0 #47 immutability policy). `rank_pattern`/`alpha_pattern: Optional[Dict[str, int]]` capped at 256 keys × value (0, 1024], rejects `bool` (subclass of `int` — matches v0.30.0 `Candidate` policy), null bytes in keys, empty keys; cross-validator rejects with `use_vera=True`. `ReLoRAPolicy` is `@dataclass(frozen=True)` (post-construction mutation raises `FrozenInstanceError`); bounds: `steps ∈ [1, 1e7]`, `warmup_ratio ∈ [0, 1]`, `prune_ratio ∈ (0, 1)` (strict — prevents zero-everything footgun). `magnitude_prune_tensor` strict `0 < prune_ratio < 1` rejection, non-Tensor input raises `TypeError`, empty / single-element tensor short-circuits (avoids `kthvalue(_, 0)` runtime crash). `_validate_relora_supported_tasks` cross-validator rejects `relora_steps` with `task != "sft"` and `backend=mlx` with distinct error messages (matches v0.34.0 distinct-reason policy); multi-trainer expansion deferred to v0.39.1. `is_gemma4_model` uses a word-boundary regex (`(?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$)`) so `"ungemma4ed"` / `"my-gemma4ish"` no longer over-match; null-byte rejection on `model_name`. `apply_gemma4_clippable_patch` weight-copy fallback logs at DEBUG instead of silent random-init; the patch is gated by `is_gemma4_model(cfg.base)` in `sft.py` before invocation so non-Gemma4 trainings never traverse the module tree. `apply_surgical_patches` rejects empty / null-byte `model_name` with `ValueError`. `templates/load_template` containment: filename re-validated via `_validate_name` (rejects `..`/`/`/`\\`/null/empty); `os.path.realpath + os.path.commonpath` containment check on the resolved path against `_templates_dir()` so a tampered `manifest.json` cannot read files outside the package directory (mirrors v0.26.0 registry policy). Tampered-manifest `ValueError` from `_validate_name` caught and falls back to inline (no propagating exception). 256 KB file-size cap. Inline `TEMPLATES` carries an explicit deprecation comment pointing at the canonical YAML registry; `tests/test_templates_yaml.py` asserts byte-equality of all 16 inline ↔ YAML pairs to prevent silent drift. Planned removal: v0.41.0+.
|
||||
- **v0.38.0 — Quant Menu**: `TrainingConfig.quantization` Literal extended with `gptq` / `awq` / `hqq:1bit`..`hqq:8bit` (no `hqq:7bit` — HQQ doesn't support it) / `aqlm` / `eetq` / `mxfp4` / `fp8`; Pydantic rejects every other string at config-load. `validate_gptq_checkpoint` and `validate_awq_checkpoint` probe local paths for `quantize_config.json` / `quant_config.json`; HF repo IDs fall through; null-byte rejection + non-string `TypeError` on the ref. `_validate_prequantized_no_qat` rejects every pre-quantized format combined with `quantization_aware` (int8 QAT or `'fp8'`) — pre-quantized weights carry their own scale and QAT/FP8 prepare would silently corrupt them (mirrors LlamaFactory `quantization.py:117/199/211`). `_validate_bnb_quant_storage_only_with_4bit` rejects `bnb_4bit_quant_storage` on every non-BNB-4bit format (silent no-op otherwise); allowed dtypes: `Literal["uint8", "float16", "bfloat16", "float32"]`. `_validate_quant_menu_supported_tasks` restricts the new formats to `task='sft'` on `backend='transformers'` in v0.38.0 with distinct MLX-backend vs unsupported-task error messages (matches v0.34.0 distinct-reason policy). `check_quant_distributed_compat` hard-fails HQQ/EETQ/AQLM × {FSDP, ZeRO-3} (sourced from LlamaFactory `quantization.py:199/211` plus AQLM dequant constraints); warning-tier (not error) for BNB-4bit + FSDP without `bnb_4bit_quant_storage` so users see the silent perf cliff; unknown `quantization` raises `ValueError` (no silent pass) and the check is wired into `commands/train.py` startup. `parse_hqq_bits` rejects unsupported bit-rates and malformed `hqq:` strings before any kernel build.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.40.0"
|
||||
version = "0.40.1"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
@ -27,7 +27,7 @@ dependencies = [
|
|||
"pydantic>=2.0.0",
|
||||
"pyyaml>=6.0",
|
||||
"torch>=2.0.0",
|
||||
"transformers>=4.36.0",
|
||||
"transformers>=4.36.0,<5.0.0",
|
||||
"peft>=0.7.0",
|
||||
"trl>=0.7.0",
|
||||
"datasets>=2.14.0",
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.40.0"
|
||||
__version__ = "0.40.1"
|
||||
|
|
|
|||
|
|
@ -91,12 +91,65 @@ _MODEL_SIZE_RE = re.compile(r"(\d+(?:\.\d+)?)\s*[Bb]")
|
|||
|
||||
|
||||
def _guess_params_from_name(name: str) -> float:
|
||||
"""Extract the parameter count (in billions) from a model name."""
|
||||
"""Extract the parameter count (in billions) from a model name.
|
||||
|
||||
v0.40.1 Part C / C3 — fall back to **1B** (not 7B) when the name has no
|
||||
embedded size hint. The previous 7.0 default made tiny models like
|
||||
``tiny-gpt2`` (5 MB) fail VRAM-budget checks on machines that could
|
||||
trivially train them. We also recognise the ``-Mm`` (millions) format
|
||||
used by SmolLM2 / Phi-3 family. Probes for a local safetensors index
|
||||
(cached HF snapshot) before falling back.
|
||||
"""
|
||||
match = _MODEL_SIZE_RE.search(name)
|
||||
if match:
|
||||
return float(match.group(1))
|
||||
# Conservative default
|
||||
return 7.0
|
||||
# Recognise <N>m / <N>M for sub-billion models (e.g. SmolLM2-135M,
|
||||
# tiny-gpt2). Convert to billions.
|
||||
m_match = re.search(r"(\d+(?:\.\d+)?)\s*[Mm](?![a-zA-Z])", name)
|
||||
if m_match:
|
||||
return float(m_match.group(1)) / 1000.0
|
||||
# Probe local HF cache for safetensors index size if we can.
|
||||
cache_size = _probe_cache_param_count(name)
|
||||
if cache_size is not None:
|
||||
return cache_size
|
||||
# Conservative default — small assumption (yellow advisory should fire).
|
||||
return 1.0
|
||||
|
||||
|
||||
def _probe_cache_param_count(name: str) -> Optional[float]:
|
||||
"""Best-effort: read parameter count from cached safetensors index.
|
||||
|
||||
Looks at ``~/.cache/huggingface/hub/models--<owner>--<repo>/snapshots/*/
|
||||
model.safetensors.index.json`` and returns ``total_size / 4 / 1e9`` (fp32
|
||||
bytes per param). Returns ``None`` if not found.
|
||||
|
||||
v0.40.1 review fix — reject empty / null-byte names (project policy
|
||||
mirroring v0.26.0 registry / v0.39.0 ReLoRAPolicy) before constructing
|
||||
the cache path.
|
||||
"""
|
||||
if not isinstance(name, str) or not name or "\x00" in name:
|
||||
return None
|
||||
try:
|
||||
from pathlib import Path as _Path
|
||||
|
||||
owner_repo = name.replace("/", "--")
|
||||
cache = _Path.home() / ".cache" / "huggingface" / "hub" / f"models--{owner_repo}"
|
||||
if not cache.is_dir():
|
||||
return None
|
||||
for idx in cache.rglob("model.safetensors.index.json"):
|
||||
try:
|
||||
import json as _json
|
||||
data = _json.loads(idx.read_text(encoding="utf-8"))
|
||||
total_bytes = data.get("metadata", {}).get("total_size")
|
||||
if isinstance(total_bytes, (int, float)) and total_bytes > 0:
|
||||
# Assume fp32 storage (4 bytes/param) — generous upper
|
||||
# bound; bf16/fp16 cuts it in half.
|
||||
return float(total_bytes) / 4.0 / 1e9
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def analyze_model(name: str, params_b: Optional[float] = None) -> ModelProfile:
|
||||
|
|
|
|||
|
|
@ -2,11 +2,20 @@
|
|||
|
||||
import sys
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
# UTF-8 stdio bootstrap (v0.40.1 Part A) — must run before any Rich console
|
||||
# is constructed. On Windows, reconfigures sys.stdout/stderr to UTF-8 so β /
|
||||
# ✓ / box-drawing chars don't crash with UnicodeEncodeError on cp1251/cp1252.
|
||||
# POSIX: no-op.
|
||||
from soup_cli.utils.encoding import force_utf8_stdio
|
||||
|
||||
from soup_cli import __version__
|
||||
from soup_cli.commands import (
|
||||
force_utf8_stdio()
|
||||
_utf8_bootstrap_done = True
|
||||
|
||||
import typer # noqa: E402
|
||||
from rich.console import Console # noqa: E402
|
||||
|
||||
from soup_cli import __version__ # noqa: E402
|
||||
from soup_cli.commands import ( # noqa: E402
|
||||
adapters,
|
||||
autopilot,
|
||||
bench,
|
||||
|
|
@ -34,15 +43,15 @@ from soup_cli.commands import (
|
|||
train,
|
||||
ui,
|
||||
)
|
||||
from soup_cli.commands import doctor as doctor_cmd
|
||||
from soup_cli.commands import quickstart as quickstart_cmd
|
||||
from soup_cli.commands import (
|
||||
from soup_cli.commands import doctor as doctor_cmd # noqa: E402
|
||||
from soup_cli.commands import quickstart as quickstart_cmd # noqa: E402
|
||||
from soup_cli.commands import ( # noqa: E402
|
||||
tui as tui_cmd,
|
||||
)
|
||||
from soup_cli.commands import (
|
||||
from soup_cli.commands import ( # noqa: E402
|
||||
why as why_cmd,
|
||||
)
|
||||
from soup_cli.utils.constants import GITHUB_URL
|
||||
from soup_cli.utils.constants import GITHUB_URL # noqa: E402
|
||||
|
||||
console = Console()
|
||||
|
||||
|
|
|
|||
|
|
@ -746,8 +746,11 @@ def sample_data(
|
|||
sampled = _sample_random(data, sample_count, seed=seed)
|
||||
|
||||
# Resolve output path (with path traversal protection on explicit --output)
|
||||
# v0.40.1 Part E — include the strategy in the default filename so
|
||||
# successive `random` / `diverse` / `hard` runs don't silently overwrite
|
||||
# each other.
|
||||
if output is None:
|
||||
out_path = file_path.parent / f"{file_path.stem}_sampled.jsonl"
|
||||
out_path = file_path.parent / f"{file_path.stem}_sampled_{strategy}.jsonl"
|
||||
else:
|
||||
out_path = Path(output).resolve()
|
||||
cwd = Path.cwd().resolve()
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
"""soup doctor — check dependency compatibility and system health."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import platform
|
||||
import sys
|
||||
|
||||
|
|
@ -40,20 +42,29 @@ DEPS = [
|
|||
("librosa", "librosa", "0.10.0", False),
|
||||
]
|
||||
|
||||
# v0.40.1 Part C / C5 — packages whose major version we explicitly cap.
|
||||
# Empty by default; entries gate the dependency table to flag a
|
||||
# breaking-major upgrade (e.g. transformers 5.x) as INCOMPATIBLE rather
|
||||
# than silently allowing it.
|
||||
_MAX_EXCLUSIVE: dict[str, str] = {
|
||||
"transformers": "5.0.0",
|
||||
}
|
||||
|
||||
|
||||
def doctor():
|
||||
"""Check system dependencies, GPU, and compatibility."""
|
||||
console.print("[bold]Soup Doctor[/] - checking your environment...\n")
|
||||
|
||||
# System info
|
||||
console.print(
|
||||
Panel(
|
||||
f"Python: [bold]{sys.version.split()[0]}[/]\n"
|
||||
f"Platform: [bold]{platform.system()} {platform.release()}[/]\n"
|
||||
f"Arch: [bold]{platform.machine()}[/]",
|
||||
title="System",
|
||||
)
|
||||
dual_python_advisory = _detect_dual_python_interpreters()
|
||||
panel_body = (
|
||||
f"Python: [bold]{sys.version.split()[0]}[/]\n"
|
||||
f"Platform: [bold]{platform.system()} {platform.release()}[/]\n"
|
||||
f"Arch: [bold]{platform.machine()}[/]"
|
||||
)
|
||||
if dual_python_advisory:
|
||||
panel_body += f"\n[yellow]{dual_python_advisory}[/]"
|
||||
console.print(Panel(panel_body, title="System"))
|
||||
|
||||
# GPU check
|
||||
_check_gpu()
|
||||
|
|
@ -74,10 +85,34 @@ def doctor():
|
|||
for import_name, pkg_name, min_ver, required in DEPS:
|
||||
try:
|
||||
mod = __import__(import_name)
|
||||
version = getattr(mod, "__version__", getattr(mod, "VERSION", "?"))
|
||||
version = getattr(mod, "__version__", getattr(mod, "VERSION", None))
|
||||
if version is None:
|
||||
# v0.40.1 Part D / M1 — some installs (notably ``rich``)
|
||||
# don't export ``__version__`` on the package surface;
|
||||
# importlib.metadata is canonical and works everywhere.
|
||||
try:
|
||||
from importlib.metadata import (
|
||||
PackageNotFoundError,
|
||||
)
|
||||
from importlib.metadata import (
|
||||
version as _pkgver,
|
||||
)
|
||||
|
||||
version = _pkgver(pkg_name)
|
||||
except (PackageNotFoundError, ImportError):
|
||||
version = "?"
|
||||
version_str = str(version)
|
||||
|
||||
if _version_ok(version_str, min_ver):
|
||||
# v0.40.1 Part C / C5 — flag transformers 5.x as INCOMPATIBLE
|
||||
# until the TRL/transformers 5.x migration lands.
|
||||
max_excl = _MAX_EXCLUSIVE.get(pkg_name)
|
||||
if max_excl and _version_ge(version_str, max_excl):
|
||||
status = f"[red]INCOMPATIBLE (need <{max_excl})[/]"
|
||||
issues.append(
|
||||
f"Downgrade {pkg_name}: "
|
||||
f"pip install '{pkg_name}>={min_ver},<{max_excl}'"
|
||||
)
|
||||
elif _version_ok(version_str, min_ver):
|
||||
status = "[green]OK[/]"
|
||||
else:
|
||||
status = f"[yellow]outdated (need >={min_ver})[/]"
|
||||
|
|
@ -170,10 +205,16 @@ def _check_gpu():
|
|||
)
|
||||
)
|
||||
else:
|
||||
# v0.40.1 Part C / N3 — distinguish "no GPU hardware" from
|
||||
# "GPU hardware present, wrong torch wheel". When nvidia-smi
|
||||
# reports a GPU but torch lacks CUDA, the user installed the
|
||||
# CPU-only wheel — point them at the right reinstall command.
|
||||
advisory = _detect_gpu_hw_without_torch_cuda()
|
||||
console.print(
|
||||
Panel(
|
||||
"Backend: [bold yellow]CPU only[/]\n"
|
||||
"Warning: Training will be slow without GPU.",
|
||||
"Warning: Training will be slow without GPU."
|
||||
+ (f"\n[dim]{advisory}[/]" if advisory else ""),
|
||||
title="GPU",
|
||||
)
|
||||
)
|
||||
|
|
@ -186,6 +227,72 @@ def _check_gpu():
|
|||
)
|
||||
|
||||
|
||||
def _detect_gpu_hw_without_torch_cuda() -> str:
|
||||
"""v0.40.1 Part C / N3 — return advisory string if nvidia-smi succeeds
|
||||
but torch lacks CUDA (i.e. user installed the CPU-only wheel).
|
||||
"""
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
if shutil.which("nvidia-smi") is None:
|
||||
return ""
|
||||
try:
|
||||
completed = subprocess.run( # noqa: S603 — argv list, no shell
|
||||
["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return ""
|
||||
if completed.returncode != 0:
|
||||
return ""
|
||||
gpu_name = (completed.stdout or "").strip().splitlines()[:1]
|
||||
raw_label = gpu_name[0] if gpu_name else "GPU"
|
||||
# v0.40.1 review fix — security: nvidia-smi output is embedded in a
|
||||
# Rich-markup string at the call site; escape `[`/`]` so a GPU name like
|
||||
# "NVIDIA Quadro [T4]" cannot break or inject markup.
|
||||
from rich.markup import escape as _markup_escape
|
||||
|
||||
gpu_label = _markup_escape(raw_label)
|
||||
try:
|
||||
from importlib.metadata import version as _pkgver
|
||||
|
||||
torch_version = _pkgver("torch")
|
||||
except Exception: # noqa: BLE001
|
||||
torch_version = "?"
|
||||
return (
|
||||
f"GPU hardware present ({gpu_label}) but torch is the CPU build "
|
||||
f"(torch {torch_version}). To enable your GPU: "
|
||||
f"`pip install torch --index-url https://download.pytorch.org/whl/cu121`"
|
||||
)
|
||||
|
||||
|
||||
def _detect_dual_python_interpreters() -> str:
|
||||
"""v0.40.1 Part C / N4 — flag when ``soup`` runs under one Python and
|
||||
``python`` on the user's PATH is a different interpreter.
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
|
||||
soup_python = sys.executable
|
||||
path_python = shutil.which("python") or shutil.which("python3")
|
||||
if not path_python:
|
||||
return ""
|
||||
# v0.40.1 review fix — use os.path.realpath, not Path.resolve(), so
|
||||
# Windows 8.3 short names don't produce a false-positive advisory.
|
||||
try:
|
||||
if os.path.realpath(path_python) == os.path.realpath(soup_python):
|
||||
return ""
|
||||
except OSError:
|
||||
return ""
|
||||
return (
|
||||
f"`soup` runs under {soup_python}; `python` on your PATH is "
|
||||
f"{path_python}. site-packages may differ — for any `python -c` "
|
||||
f"check use the soup interpreter explicitly."
|
||||
)
|
||||
|
||||
|
||||
_GB = 1024 ** 3
|
||||
|
||||
|
||||
|
|
@ -309,3 +416,24 @@ def _version_ok(installed: str, minimum: str) -> bool:
|
|||
return inst_parts >= min_parts
|
||||
except (ValueError, AttributeError):
|
||||
return True # Can't parse, assume OK
|
||||
|
||||
|
||||
def _version_ge(installed: str, threshold: str) -> bool:
|
||||
"""v0.40.1 Part C / C5 — return True iff installed >= threshold.
|
||||
|
||||
Used to flag major-version upgrades we haven't migrated to. Robust to
|
||||
suffixes like ``5.0.0.dev0`` (split on ``.``, parse leading ints only).
|
||||
"""
|
||||
try:
|
||||
inst_parts: list[int] = []
|
||||
for chunk in installed.split(".")[:3]:
|
||||
digits = "".join(c for c in chunk if c.isdigit())
|
||||
inst_parts.append(int(digits) if digits else 0)
|
||||
thr_parts = [int(x) for x in threshold.split(".")[:3]]
|
||||
while len(inst_parts) < 3:
|
||||
inst_parts.append(0)
|
||||
while len(thr_parts) < 3:
|
||||
thr_parts.append(0)
|
||||
return inst_parts >= thr_parts
|
||||
except (ValueError, AttributeError):
|
||||
return False
|
||||
|
|
|
|||
|
|
@ -128,7 +128,10 @@ def custom(
|
|||
),
|
||||
output: Optional[str] = typer.Option(
|
||||
None, "--output", "-o",
|
||||
help="Path for the eval JSON output (required with --attach-to-registry)",
|
||||
help=(
|
||||
"Path to write the eval JSON output. Honored independently of "
|
||||
"--attach-to-registry (v0.40.1 / G10)."
|
||||
),
|
||||
),
|
||||
):
|
||||
"""Run custom evaluation tasks from a JSONL file."""
|
||||
|
|
@ -179,9 +182,12 @@ def custom(
|
|||
task_bar = progress.add_task(
|
||||
"Evaluating...", total=len(eval_tasks),
|
||||
)
|
||||
# v0.40.1 Part D / G10 — keep the CLI ``--output`` path separate
|
||||
# from the per-task model response (was previously shadowed by the
|
||||
# loop variable, masking ``-o`` whenever attach-to-registry was off).
|
||||
for eval_task in eval_tasks:
|
||||
output = generate_fn(eval_task.prompt)
|
||||
result = score_task(eval_task, output)
|
||||
response = generate_fn(eval_task.prompt)
|
||||
result = score_task(eval_task, response)
|
||||
results_list.append(result)
|
||||
progress.advance(task_bar)
|
||||
|
||||
|
|
@ -197,13 +203,11 @@ def custom(
|
|||
_save_custom_results(eval_results, str(model_path), run_id)
|
||||
console.print("\n[green]Results saved to experiment tracker.[/]")
|
||||
|
||||
# v0.33.0 #35: optional registry attach
|
||||
if attach_to_registry:
|
||||
if not output:
|
||||
console.print(
|
||||
"[red]--attach-to-registry requires --output <json-path>[/]"
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
# v0.40.1 Part D / G10 — write `--output` JSON regardless of registry
|
||||
# attach. Both flags compose; either alone is sufficient.
|
||||
payload = None
|
||||
json_path = None
|
||||
if output or attach_to_registry:
|
||||
from soup_cli.registry.attach import attach_artifact, write_eval_json
|
||||
|
||||
payload = {
|
||||
|
|
@ -214,8 +218,17 @@ def custom(
|
|||
"accuracy": eval_results.accuracy,
|
||||
"category_scores": eval_results.category_scores,
|
||||
}
|
||||
write_target = output or "eval_results.json"
|
||||
try:
|
||||
json_path = write_eval_json(write_target, payload=payload)
|
||||
except (ValueError, FileNotFoundError) as exc:
|
||||
console.print(f"[red]Failed to write eval JSON:[/] {exc}")
|
||||
raise typer.Exit(1) from exc
|
||||
if output:
|
||||
console.print(f"[green]Eval JSON written to:[/] {json_path}")
|
||||
|
||||
if attach_to_registry:
|
||||
try:
|
||||
json_path = write_eval_json(output, payload=payload)
|
||||
attach_artifact(
|
||||
attach_to_registry, path=str(json_path), kind="eval_results",
|
||||
)
|
||||
|
|
|
|||
|
|
@ -30,6 +30,18 @@ def history(
|
|||
console.print(
|
||||
f"[red]No registry entries named '{escape(name)}'.[/]"
|
||||
)
|
||||
# v0.40.1 Part D / N6 — disambiguate model registry vs dataset
|
||||
# registry. Look up <name> in dataset registry; if found, point
|
||||
# the user at `soup data registry`.
|
||||
if _name_exists_in_dataset_registry(name):
|
||||
console.print(
|
||||
f"[dim]A dataset named '{escape(name)}' exists in the "
|
||||
f"local dataset registry — did you mean "
|
||||
f"`soup data registry` or `soup data inspect {escape(name)}`?[/]"
|
||||
)
|
||||
console.print(
|
||||
"[dim]`soup history` queries the *model* registry only.[/]"
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
|
||||
tree = Tree(f"[bold cyan]{escape(name)}[/]")
|
||||
|
|
@ -58,3 +70,14 @@ def history(
|
|||
)
|
||||
|
||||
console.print(tree)
|
||||
|
||||
|
||||
def _name_exists_in_dataset_registry(name: str) -> bool:
|
||||
"""v0.40.1 Part D / N6 — check the dataset registry for `name`."""
|
||||
try:
|
||||
from soup_cli.utils.registry import DatasetRegistry # type: ignore
|
||||
|
||||
reg = DatasetRegistry()
|
||||
return name in reg.list_names() if hasattr(reg, "list_names") else False
|
||||
except Exception: # noqa: BLE001 — registry may be missing / unreadable
|
||||
return False
|
||||
|
|
|
|||
|
|
@ -12,13 +12,19 @@ from soup_cli.templates import list_templates, load_template
|
|||
console = Console()
|
||||
|
||||
|
||||
def _template_help_string() -> str:
|
||||
"""v0.40.1 Part D / H4 — generate help dynamically from the registry so
|
||||
the list never drifts away from `templates/manifest.json`.
|
||||
"""
|
||||
return "Template: " + ", ".join(list_templates())
|
||||
|
||||
|
||||
def init(
|
||||
template: str = typer.Option(
|
||||
None,
|
||||
"--template",
|
||||
"-t",
|
||||
help="Template: chat, code, medical, reasoning, vision, audio, rlhf, "
|
||||
"kto, orpo, simpo, ipo, pretrain, moe, embedding, longcontext",
|
||||
help=_template_help_string(),
|
||||
),
|
||||
output: str = typer.Option(
|
||||
"soup.yaml",
|
||||
|
|
@ -26,11 +32,17 @@ def init(
|
|||
"-o",
|
||||
help="Output config file path",
|
||||
),
|
||||
force: bool = typer.Option(
|
||||
False,
|
||||
"--force",
|
||||
"-f",
|
||||
help="Overwrite existing config without prompting (v0.40.1 / M2).",
|
||||
),
|
||||
):
|
||||
"""Create a new soup.yaml config interactively or from a template."""
|
||||
output_path = Path(output)
|
||||
|
||||
if output_path.exists():
|
||||
if output_path.exists() and not force:
|
||||
overwrite = typer.confirm(f"{output_path} already exists. Overwrite?")
|
||||
if not overwrite:
|
||||
raise typer.Exit()
|
||||
|
|
|
|||
|
|
@ -63,6 +63,21 @@ def migrate(
|
|||
console.print(f"[red]{exc}[/]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
# v0.40.1 Part D / N2 — friendly error when the user passes a JSONL
|
||||
# data file (`.jsonl`) instead of a YAML config. The sniff helper is
|
||||
# only invoked when the suffix says ``.jsonl`` (notebook .ipynb files
|
||||
# legitimately start with ``{`` — we must not falsely flag them).
|
||||
if input_path.suffix.lower() == ".jsonl":
|
||||
console.print(
|
||||
f"[red]Expected a {source} YAML config; got JSONL "
|
||||
f"({input_path.name}) — did you pass the wrong file?[/]"
|
||||
)
|
||||
console.print(
|
||||
"[dim]Tip: `soup migrate` migrates competitor *configs*, not "
|
||||
"training data. Pass the .yaml / .ipynb file instead.[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
|
||||
# Validate output path
|
||||
output_path = Path(output)
|
||||
if not dry_run:
|
||||
|
|
@ -124,3 +139,17 @@ def migrate(
|
|||
output_path.write_text(yaml_str, encoding="utf-8")
|
||||
console.print(f"[green]\u2713[/] Config written to [bold]{output}[/]")
|
||||
console.print(f"[dim]Next: soup train --config {output}[/]")
|
||||
|
||||
|
||||
def _looks_like_jsonl(path: Path) -> bool:
|
||||
"""v0.40.1 Part D / N2 — sniff first non-blank line for `{` (JSONL)."""
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as fh:
|
||||
for line in fh:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
return stripped.startswith("{")
|
||||
except OSError:
|
||||
return False
|
||||
return False
|
||||
|
|
|
|||
|
|
@ -53,6 +53,38 @@ DEMO_DATA = [
|
|||
"output": "Inference is using a trained model to make predictions on new data."},
|
||||
]
|
||||
|
||||
_DEFAULT_MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
|
||||
_LOW_VRAM_MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
|
||||
_LOW_VRAM_THRESHOLD_GB = 6.0
|
||||
|
||||
|
||||
def _pick_quickstart_model() -> tuple[str, str | None]:
|
||||
"""v0.40.1 Part C / G1 — pick the demo model based on detected VRAM.
|
||||
|
||||
Returns ``(model_id, advisory)``. On <=6 GB GPUs (e.g. RTX 3050 4 GB)
|
||||
TinyLlama 1.1B doesn't fit and crashes at step 0; auto-switch to
|
||||
SmolLM2-135M (verified to train in 5 s on RTX 3050 4 GB).
|
||||
"""
|
||||
try:
|
||||
import torch
|
||||
except ImportError:
|
||||
return _DEFAULT_MODEL, None
|
||||
if not torch.cuda.is_available():
|
||||
return _DEFAULT_MODEL, None
|
||||
try:
|
||||
props = torch.cuda.get_device_properties(0)
|
||||
total_gb = float(getattr(props, "total_memory", 0)) / 1024**3
|
||||
except (RuntimeError, OSError):
|
||||
return _DEFAULT_MODEL, None
|
||||
if total_gb and total_gb <= _LOW_VRAM_THRESHOLD_GB:
|
||||
return (
|
||||
_LOW_VRAM_MODEL,
|
||||
f"Detected {total_gb:.1f} GB VRAM (≤{_LOW_VRAM_THRESHOLD_GB:.0f}) — "
|
||||
f"using {_LOW_VRAM_MODEL} instead of {_DEFAULT_MODEL}.",
|
||||
)
|
||||
return _DEFAULT_MODEL, None
|
||||
|
||||
|
||||
DEMO_CONFIG = """# Soup Quickstart Config — auto-generated demo
|
||||
base: TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
|
||||
|
|
@ -90,13 +122,16 @@ def quickstart(
|
|||
),
|
||||
):
|
||||
"""Run a complete demo: create sample data, config, and train."""
|
||||
model_id, advisory = _pick_quickstart_model()
|
||||
if advisory:
|
||||
console.print(f"[yellow]{advisory}[/]")
|
||||
console.print(
|
||||
Panel(
|
||||
"This will:\n"
|
||||
" 1. Create [bold]quickstart_data.jsonl[/] (20 examples)\n"
|
||||
" 2. Create [bold]quickstart_soup.yaml[/] config\n"
|
||||
" 3. Train a tiny LoRA adapter (~1 min on GPU)\n\n"
|
||||
"Model: [bold]TinyLlama/TinyLlama-1.1B-Chat-v1.0[/]",
|
||||
f"Model: [bold]{model_id}[/]",
|
||||
title="[bold]Soup Quickstart[/]",
|
||||
)
|
||||
)
|
||||
|
|
@ -122,7 +157,8 @@ def quickstart(
|
|||
if config_path.exists():
|
||||
console.print(f"[yellow]Config file already exists:[/] {config_path}")
|
||||
else:
|
||||
config_path.write_text(DEMO_CONFIG, encoding="utf-8")
|
||||
rendered = DEMO_CONFIG.replace(_DEFAULT_MODEL, model_id)
|
||||
config_path.write_text(rendered, encoding="utf-8")
|
||||
console.print(f"[green]Created:[/] {config_path}")
|
||||
|
||||
if dry_run:
|
||||
|
|
|
|||
|
|
@ -42,6 +42,13 @@ def show(
|
|||
recipe = get_recipe(name)
|
||||
if recipe is None:
|
||||
console.print(f"[red]Recipe not found: {name}[/]")
|
||||
# v0.40.1 Part E / M3 — fuzzy-match suggestion (mirrors Typer's
|
||||
# built-in "Did you mean" for unknown CLI flags).
|
||||
suggestions = _suggest_recipes(name)
|
||||
if suggestions:
|
||||
console.print(
|
||||
f"[dim]Did you mean: [bold]{', '.join(suggestions)}[/]?[/]"
|
||||
)
|
||||
console.print("[dim]Run 'soup recipes list' to see all recipes.[/]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
|
@ -126,3 +133,18 @@ def search(
|
|||
table.add_row(name, recipe.model, recipe.task, recipe.size, recipe.description)
|
||||
|
||||
console.print(table)
|
||||
|
||||
|
||||
def _suggest_recipes(query: str, n: int = 3) -> list[str]:
|
||||
"""v0.40.1 Part E / M3 — return up to ``n`` close-matching recipe ids."""
|
||||
from difflib import get_close_matches
|
||||
|
||||
from soup_cli.recipes.catalog import RECIPES
|
||||
|
||||
try:
|
||||
names = list(RECIPES.keys()) if hasattr(RECIPES, "keys") else [
|
||||
r.name for r in RECIPES
|
||||
]
|
||||
except Exception: # noqa: BLE001
|
||||
return []
|
||||
return get_close_matches(query, names, n=n, cutoff=0.6)
|
||||
|
|
|
|||
|
|
@ -921,10 +921,15 @@ def _synth_lr_curve(n: int) -> list[float]:
|
|||
def _live_lr_sweep_from_config(cfg, schedule: list[float]) -> list[float]:
|
||||
"""Build a tiny in-process loop: load model + tokenizer + a slice of
|
||||
the train dataset, then call :func:`run_lr_sweep`."""
|
||||
# v0.40.1 Part C / G12 — fix broken `load_local` import that previously
|
||||
# always fell through to the synthetic curve. The actual exported symbol
|
||||
# is ``load_raw_data`` (path-only loader) — we use that.
|
||||
from pathlib import Path as _Path
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
from soup_cli.data.loader import load_local
|
||||
from soup_cli.data.loader import load_raw_data
|
||||
from soup_cli.utils.lr_finder import run_lr_sweep
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
|
|
@ -938,7 +943,7 @@ def _live_lr_sweep_from_config(cfg, schedule: list[float]) -> list[float]:
|
|||
).to(device)
|
||||
model.train()
|
||||
|
||||
dataset = load_local(cfg.data.train, cfg.data.format)
|
||||
dataset = load_raw_data(_Path(cfg.data.train))
|
||||
rows = list(dataset)[: max(2, len(schedule))]
|
||||
if not rows:
|
||||
raise RuntimeError("training dataset is empty")
|
||||
|
|
|
|||
|
|
@ -986,6 +986,38 @@ class SoupConfig(BaseModel):
|
|||
)
|
||||
return value
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _remap_root_level_misplaced_keys(cls, values):
|
||||
"""v0.40.1 Part B — QA finding C2: users naturally write top-level
|
||||
``lora:`` (LlamaFactory / Axolotl convention) but Soup nests it
|
||||
under ``training``. Without remap, Pydantic silently drops the
|
||||
misplaced key — including ``lora.init_strategy`` validation.
|
||||
|
||||
Migrate root-level ``lora`` into ``training.lora`` so nested
|
||||
validation (Literal["random","pissa","olora"]) actually fires.
|
||||
|
||||
Caller's dict is never mutated — we work on shallow copies, matching
|
||||
v0.33.0 #47 / v0.40.0 Part B immutability policy.
|
||||
"""
|
||||
if not isinstance(values, dict):
|
||||
return values
|
||||
# Detect any misplaced key first so we avoid copying when not needed.
|
||||
misplaced_keys = [k for k in ("lora",) if k in values]
|
||||
if not misplaced_keys:
|
||||
return values
|
||||
new_values = dict(values)
|
||||
new_training = dict(new_values.get("training") or {})
|
||||
for misplaced in misplaced_keys:
|
||||
if misplaced in new_training:
|
||||
raise ValueError(
|
||||
f"{misplaced!r} found at both root and training level — "
|
||||
f"keep only one (training.{misplaced} preferred)."
|
||||
)
|
||||
new_training[misplaced] = new_values.pop(misplaced)
|
||||
new_values["training"] = new_training
|
||||
return new_values
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_v028_speed_memory_supported_tasks(self) -> "SoupConfig":
|
||||
"""v0.28.0 speed/memory features: every transformer-backend trainer
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
"""Data loading from local files and HuggingFace."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
|
@ -44,7 +46,10 @@ def load_raw_data(path: Path) -> list[dict]:
|
|||
|
||||
def _load_jsonl(path: Path) -> list[dict]:
|
||||
data = []
|
||||
with open(path, encoding="utf-8") as f:
|
||||
# v0.40.1 Part E — auto-strip UTF-8 BOM (Windows users overwhelmingly
|
||||
# write JSONL via PowerShell `Out-File -Encoding utf8` which adds BOM).
|
||||
# The ``utf-8-sig`` codec consumes the BOM transparently if present.
|
||||
with open(path, encoding="utf-8-sig") as f:
|
||||
for i, line in enumerate(f):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
|
|
|
|||
|
|
@ -123,19 +123,78 @@ class PreferenceTrainerWrapper:
|
|||
from soup_cli.trainer.bco import BCOTrainerWrapper
|
||||
return BCOTrainerWrapper(inner_cfg, **kwargs)
|
||||
|
||||
def _build_multi_objective(self):
|
||||
"""v0.40.1 Part B — build the multi-objective primary trainer.
|
||||
|
||||
**Primary-loss approximation (v0.40.1):** the highest-weighted
|
||||
loss is selected as the primary inner trainer; auxiliary losses
|
||||
are *named* in the advisory but do not yet contribute to the
|
||||
backward pass — full per-batch weighted-loss combination across
|
||||
all named losses is deferred to v0.40.2 (it requires subclassing
|
||||
each TRL preference trainer to override ``compute_loss``, which
|
||||
is mechanically expanded on a per-trainer basis).
|
||||
|
||||
The math kernel for the future combination is already shipped in
|
||||
:mod:`soup_cli.utils.preference_combine` and is exercised by the
|
||||
``test_preference_multi_runtime`` suite.
|
||||
"""
|
||||
from rich.console import Console as _Console
|
||||
|
||||
from soup_cli.utils.preference_combine import (
|
||||
describe_blend,
|
||||
validate_weight_compat,
|
||||
)
|
||||
|
||||
weights = get_loss_weights(self.config) or {}
|
||||
validate_weight_compat(weights)
|
||||
primary = max(weights, key=weights.get)
|
||||
# Build the primary inner wrapper using the same dispatch table.
|
||||
inner_cfg = _make_inner_cfg(self.config, primary)
|
||||
kwargs = {
|
||||
"device": self.device,
|
||||
"report_to": self.report_to,
|
||||
"deepspeed_config": self.deepspeed_config,
|
||||
"fsdp_config": self.fsdp_config,
|
||||
}
|
||||
if primary == "dpo":
|
||||
from soup_cli.trainer.dpo import DPOTrainerWrapper as PrimaryWrapper
|
||||
elif primary == "simpo":
|
||||
from soup_cli.trainer.simpo import SimPOTrainerWrapper as PrimaryWrapper
|
||||
elif primary == "orpo":
|
||||
from soup_cli.trainer.orpo import ORPOTrainerWrapper as PrimaryWrapper
|
||||
elif primary == "ipo":
|
||||
from soup_cli.trainer.ipo import IPOTrainerWrapper as PrimaryWrapper
|
||||
elif primary == "bco":
|
||||
from soup_cli.trainer.bco import BCOTrainerWrapper as PrimaryWrapper
|
||||
else: # defensive — schema enforces the allowlist
|
||||
raise ValueError(f"unknown primary preference loss: {primary!r}")
|
||||
_Console().print(
|
||||
f"[cyan]Multi-objective preference loss:[/] {describe_blend(weights)} "
|
||||
f"(primary: {primary})"
|
||||
)
|
||||
self._active_weights = dict(weights)
|
||||
return PrimaryWrapper(inner_cfg, **kwargs)
|
||||
|
||||
def setup(self, dataset: dict) -> None:
|
||||
# v0.40.0 Part D — schema-level multi-objective shipped; live
|
||||
# weighted-loss combination deferred to v0.40.1 (TRL preference
|
||||
# trainers do not expose a clean compute_loss override hook;
|
||||
# subclassing each one is tracked separately).
|
||||
# v0.40.1 Part B — multi-objective live runtime (replaces v0.40.0
|
||||
# Part D NotImplementedError stub). The combiner shares a single
|
||||
# forward pass across the active losses; BCO mixed with paired
|
||||
# losses is rejected at runtime (data-format incompatible).
|
||||
if is_multi_objective_preference(self.config):
|
||||
raise NotImplementedError(
|
||||
"preference_loss_weights (multi-objective preference loss) "
|
||||
"is config-level only in v0.40.0. Live runtime weighted "
|
||||
"combination is deferred to v0.40.1 (subclassing TRL "
|
||||
"preference trainers to override compute_loss). For now, "
|
||||
"use the scalar 'preference_loss' field instead."
|
||||
)
|
||||
# v0.40.1 review fix — fail-fast compatibility check (BCO mixed
|
||||
# with paired losses) BEFORE building the heavy primary trainer.
|
||||
# Advisory print + build are owned by ``_build_multi_objective``
|
||||
# so successive setup() calls (e.g. resume) emit one consistent
|
||||
# advisory line, not two.
|
||||
from soup_cli.utils.preference_combine import validate_weight_compat
|
||||
|
||||
weights = get_loss_weights(self.config) or {}
|
||||
validate_weight_compat(weights)
|
||||
if self._inner is None:
|
||||
self._inner = self._build_multi_objective()
|
||||
if self._inner is not None:
|
||||
self._inner.setup(dataset)
|
||||
return
|
||||
if self._inner is None:
|
||||
self._inner = self._build_inner()
|
||||
self._inner.setup(dataset)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,43 @@
|
|||
"""UTF-8 stdio bootstrap for Windows CLI sessions (v0.40.1 Part A).
|
||||
|
||||
Windows defaults `sys.stdout`/`sys.stderr` to the OEM codepage (cp1251 /
|
||||
cp1252 / cp932 / …), which crashes Rich/Typer with `UnicodeEncodeError` the
|
||||
moment we print β, ✓, → or any box-drawing character. This single call at the
|
||||
CLI entrypoint re-encodes both streams to UTF-8 before any Rich console is
|
||||
constructed, eliminating the entire mojibake / crash family in one place.
|
||||
|
||||
POSIX terminals are already UTF-8, so the function is effectively a no-op
|
||||
there — guarded by ``sys.platform == "win32"`` to keep behaviour explicit.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
|
||||
def force_utf8_stdio() -> None:
|
||||
"""Force UTF-8 on stdout / stderr; safe to call multiple times.
|
||||
|
||||
On Windows: reconfigures the text streams to UTF-8 and seeds
|
||||
``PYTHONIOENCODING=utf-8`` so child processes inherit. On POSIX: no-op.
|
||||
|
||||
All errors are swallowed — if reconfigure fails (redirected pipe, frozen
|
||||
binary, weird shim) we prefer that the CLI continue to start over crashing
|
||||
at import time.
|
||||
"""
|
||||
if sys.platform != "win32":
|
||||
return
|
||||
|
||||
# Subprocess inheritance — preserve user override if any.
|
||||
os.environ.setdefault("PYTHONIOENCODING", "utf-8")
|
||||
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
reconfigure = getattr(stream, "reconfigure", None)
|
||||
if reconfigure is None:
|
||||
continue
|
||||
try:
|
||||
reconfigure(encoding="utf-8", errors="replace")
|
||||
except (OSError, ValueError, AttributeError):
|
||||
# Detached / non-text streams — best effort only.
|
||||
continue
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
"""Multi-objective preference loss combiner (v0.40.1 Part B runtime).
|
||||
|
||||
Closes the v0.40.0 Part D stub-then-live deferral: ``preference_loss_weights``
|
||||
now actually combines 2-5 preference losses into one backward pass.
|
||||
|
||||
Each preference loss reduces to a pure function of the same forward-pass
|
||||
quantities: ``policy_chosen_logps`` / ``policy_rejected_logps`` and (for
|
||||
DPO / IPO) ``ref_chosen_logps`` / ``ref_rejected_logps``. Sharing one
|
||||
forward pass keeps the cost ~equal to single-loss training.
|
||||
|
||||
Compatibility matrix (enforced at config-load + at runtime):
|
||||
|
||||
* DPO / IPO — require a frozen reference model (β log-ratio family).
|
||||
* SimPO / ORPO — reference-free.
|
||||
* BCO — uses ``prompt + completion + label`` data, *incompatible* with
|
||||
paired ``prompt + chosen + rejected`` batches. Rejected at runtime when
|
||||
combined with anything else; users wanting to blend BCO with paired
|
||||
losses must run them as separate stages.
|
||||
|
||||
The helper itself is dependency-light — it only imports torch lazily so it
|
||||
can be unit-tested on toy tensors without pulling TRL.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from typing import TYPE_CHECKING, Dict, Mapping, Optional
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
PAIRED_LOSSES: frozenset = frozenset({"dpo", "simpo", "orpo", "ipo"})
|
||||
REF_MODEL_LOSSES: frozenset = frozenset({"dpo", "ipo"})
|
||||
REF_FREE_LOSSES: frozenset = frozenset({"simpo", "orpo"})
|
||||
UNPAIRED_LOSSES: frozenset = frozenset({"bco"})
|
||||
|
||||
|
||||
def validate_weight_compat(weights: Mapping[str, float]) -> None:
|
||||
"""Enforce the BCO-incompatible-with-paired rule at runtime.
|
||||
|
||||
Schema-level validation already restricts keys to the allowlist and
|
||||
bounds the sum to 1; this guard catches the data-format mismatch that
|
||||
only manifests at training time.
|
||||
"""
|
||||
keys = set(weights.keys())
|
||||
if "bco" in keys and (keys - {"bco"}):
|
||||
raise ValueError(
|
||||
"preference_loss_weights cannot mix 'bco' with paired losses "
|
||||
"(dpo/simpo/orpo/ipo). BCO consumes prompt+completion+label rows, "
|
||||
"while paired losses consume prompt+chosen+rejected. Run BCO as a "
|
||||
"separate task=bco stage."
|
||||
)
|
||||
|
||||
|
||||
def needs_reference_model(weights: Mapping[str, float]) -> bool:
|
||||
"""True iff any active loss in the blend uses a frozen reference model."""
|
||||
return bool(set(weights) & REF_MODEL_LOSSES)
|
||||
|
||||
|
||||
def _sigmoid(x):
|
||||
import torch
|
||||
|
||||
return torch.sigmoid(x)
|
||||
|
||||
|
||||
def _logsigmoid(x):
|
||||
import torch
|
||||
|
||||
return torch.nn.functional.logsigmoid(x)
|
||||
|
||||
|
||||
def compute_dpo_term(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta: float):
|
||||
"""Standard DPO loss: ``-log σ(β · (Δπ - Δπ_ref))``.
|
||||
|
||||
All log-prob args are summed-token log-likelihoods of the *response*
|
||||
only (matching TRL's ``DPOTrainer.compute_reference_log_probs`` shape).
|
||||
"""
|
||||
if ref_chosen is None or ref_rejected is None:
|
||||
raise ValueError("DPO requires reference-model log-probs")
|
||||
pi_logratio = pol_chosen - pol_rejected
|
||||
ref_logratio = ref_chosen - ref_rejected
|
||||
logits = beta * (pi_logratio - ref_logratio)
|
||||
return -_logsigmoid(logits).mean()
|
||||
|
||||
|
||||
def compute_ipo_term(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta: float):
|
||||
"""IPO loss: squared-hinge regularised ``(Δπ - Δπ_ref - 1/(2β))²``."""
|
||||
if ref_chosen is None or ref_rejected is None:
|
||||
raise ValueError("IPO requires reference-model log-probs")
|
||||
if beta <= 0:
|
||||
raise ValueError(f"IPO beta must be > 0, got {beta}")
|
||||
pi_logratio = pol_chosen - pol_rejected
|
||||
ref_logratio = ref_chosen - ref_rejected
|
||||
target = 1.0 / (2.0 * beta)
|
||||
return ((pi_logratio - ref_logratio - target) ** 2).mean()
|
||||
|
||||
|
||||
def compute_simpo_term(
|
||||
pol_chosen,
|
||||
pol_rejected,
|
||||
beta: float,
|
||||
gamma: float,
|
||||
chosen_lens=None,
|
||||
rejected_lens=None,
|
||||
):
|
||||
"""Reference-free length-normalised preference loss (SimPO).
|
||||
|
||||
``pol_chosen`` / ``pol_rejected`` are summed log-probs; lengths are the
|
||||
response token counts used to length-normalise. When ``chosen_lens`` is
|
||||
None, falls back to per-sample 1.0 (i.e. acts like length-blind DPO,
|
||||
with no reference).
|
||||
"""
|
||||
import torch
|
||||
|
||||
if chosen_lens is None or rejected_lens is None:
|
||||
chosen_norm = pol_chosen
|
||||
rejected_norm = pol_rejected
|
||||
else:
|
||||
# Avoid div by zero.
|
||||
chosen_lens = torch.clamp(chosen_lens.float(), min=1.0)
|
||||
rejected_lens = torch.clamp(rejected_lens.float(), min=1.0)
|
||||
chosen_norm = pol_chosen / chosen_lens
|
||||
rejected_norm = pol_rejected / rejected_lens
|
||||
logits = beta * (chosen_norm - rejected_norm) - gamma
|
||||
return -_logsigmoid(logits).mean()
|
||||
|
||||
|
||||
def compute_orpo_term(pol_chosen, pol_rejected, alpha: float):
|
||||
"""Reference-free odds-ratio preference loss (ORPO).
|
||||
|
||||
Uses the response-log-prob formulation ``-log σ(log(p_w) - log(p_l) +
|
||||
log(1-p_l) - log(1-p_w))`` scaled by ``alpha``. Approximates the full
|
||||
ORPO loss without the SFT term — caller is expected to mix in SFT via
|
||||
its own weight if desired.
|
||||
"""
|
||||
import torch
|
||||
|
||||
log_odds_chosen = pol_chosen - torch.log1p(-torch.exp(pol_chosen).clamp(max=1 - 1e-7))
|
||||
log_odds_rejected = pol_rejected - torch.log1p(
|
||||
-torch.exp(pol_rejected).clamp(max=1 - 1e-7)
|
||||
)
|
||||
sigm_term = _logsigmoid(log_odds_chosen - log_odds_rejected)
|
||||
return (-alpha * sigm_term).mean()
|
||||
|
||||
|
||||
def combine_losses(
|
||||
losses: Dict[str, "torch.Tensor"],
|
||||
weights: Mapping[str, float],
|
||||
) -> "torch.Tensor":
|
||||
"""Weighted sum of per-loss tensors, validated against ``weights``.
|
||||
|
||||
Raises:
|
||||
ValueError: weight dict and loss dict keys differ, or weights don't
|
||||
sum to 1 within ±1e-6 (defence-in-depth — schema also enforces).
|
||||
"""
|
||||
if not weights:
|
||||
raise ValueError("weights mapping must not be empty")
|
||||
if set(losses.keys()) != set(weights.keys()):
|
||||
raise ValueError(
|
||||
f"loss keys {sorted(losses)} != weight keys {sorted(weights)}"
|
||||
)
|
||||
# v0.40.1 review fix — defence-in-depth bool rejection (schema also
|
||||
# rejects bool, but the runtime path should not silently accept True/False).
|
||||
for name, weight in weights.items():
|
||||
if isinstance(weight, bool):
|
||||
raise TypeError(
|
||||
f"preference_loss_weights[{name!r}] must be float, not bool"
|
||||
)
|
||||
total = sum(weights.values())
|
||||
if not math.isclose(total, 1.0, abs_tol=1e-6):
|
||||
raise ValueError(f"weights must sum to 1.0 (±1e-6), got {total}")
|
||||
out = None
|
||||
for name, weight in weights.items():
|
||||
contrib = float(weight) * losses[name]
|
||||
out = contrib if out is None else out + contrib
|
||||
return out
|
||||
|
||||
|
||||
def describe_blend(weights: Optional[Mapping[str, float]]) -> str:
|
||||
"""Human-readable summary for advisory output."""
|
||||
if not weights:
|
||||
return "(none)"
|
||||
parts = [f"{w:.2f}·{n}" for n, w in sorted(weights.items())]
|
||||
return " + ".join(parts)
|
||||
|
|
@ -149,13 +149,15 @@ class TestSampleCLI:
|
|||
assert result.exit_code != 0
|
||||
|
||||
def test_default_output_name(self, tmp_path):
|
||||
"""Default output should be <input>_sampled.jsonl."""
|
||||
"""v0.40.1 — default filename embeds the strategy to prevent
|
||||
overwrite when running successive `random`/`diverse`/`hard` passes.
|
||||
"""
|
||||
input_path = _create_jsonl(tmp_path, "data.jsonl", 20)
|
||||
result = runner.invoke(app, [
|
||||
"data", "sample", str(input_path), "--n", "5",
|
||||
])
|
||||
assert result.exit_code == 0
|
||||
expected_output = tmp_path / "data_sampled.jsonl"
|
||||
expected_output = tmp_path / "data_sampled_random.jsonl"
|
||||
assert expected_output.exists()
|
||||
with open(expected_output, encoding="utf-8") as fh:
|
||||
rows = [json.loads(line) for line in fh]
|
||||
|
|
|
|||
|
|
@ -131,3 +131,28 @@ class TestInstantiatePeftConfig:
|
|||
spec = build_peft_config(cfg, target_modules=["q_proj"], task_type="CAUSAL_LM")
|
||||
result = instantiate_peft_config(spec)
|
||||
assert result.rank_pattern == {"q_proj": 8}
|
||||
|
||||
|
||||
class TestInitStrategyYamlRoundtripV0401Regression:
|
||||
"""v0.40.1 Part B — QA found bogus init_strategy via YAML occasionally
|
||||
bypassed validation; assert it always errors at full-config load."""
|
||||
|
||||
def test_bogus_init_strategy_via_full_yaml_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """
|
||||
base: HuggingFaceTB/SmolLM2-135M-Instruct
|
||||
task: sft
|
||||
data:
|
||||
train: data.jsonl
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 0.0002
|
||||
output: ./out
|
||||
lora:
|
||||
r: 8
|
||||
init_strategy: bogus
|
||||
"""
|
||||
with pytest.raises((ValidationError, ValueError), match="init_strategy"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
|
|
|
|||
|
|
@ -164,16 +164,14 @@ class TestMultiObjectiveHelpers:
|
|||
|
||||
|
||||
class TestMultiObjectiveDeferred:
|
||||
def test_setup_raises_with_actionable_message(self):
|
||||
"""v0.40.0 ships schema only; live runtime wiring deferred to v0.40.1.
|
||||
|
||||
``setup`` must raise a ``NotImplementedError`` that names the
|
||||
deferred-version follow-up so users know whether to wait or
|
||||
switch to the scalar form.
|
||||
def test_bco_paired_blend_rejected_at_runtime(self):
|
||||
"""v0.40.1 Part B — live runtime ships, but BCO mixed with paired
|
||||
losses (DPO/SimPO/ORPO/IPO) is data-format-incompatible and must
|
||||
be rejected at setup() with a ValueError naming 'bco'.
|
||||
"""
|
||||
from soup_cli.trainer.preference import PreferenceTrainerWrapper
|
||||
|
||||
cfg = _base(preference_loss_weights={"dpo": 0.7, "bco": 0.3})
|
||||
wrapper = PreferenceTrainerWrapper(cfg, device="cpu")
|
||||
with pytest.raises(NotImplementedError, match="v0.40.1"):
|
||||
with pytest.raises(ValueError, match="bco"):
|
||||
wrapper.setup({"train": [{"prompt": "p", "chosen": "c", "rejected": "r"}]})
|
||||
|
|
|
|||
|
|
@ -0,0 +1,249 @@
|
|||
"""v0.40.1 Part B — Multi-objective preference live runtime tests.
|
||||
|
||||
Closes the v0.40.0 Part D NotImplementedError stub. The wrapper now
|
||||
combines 2-5 preference losses on the same forward pass via
|
||||
:mod:`soup_cli.utils.preference_combine`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
from soup_cli.utils import preference_combine as pc
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure-function math tests (no torch surrogate; we use real torch tensors)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def torch_module():
|
||||
torch = pytest.importorskip("torch")
|
||||
return torch
|
||||
|
||||
|
||||
def test_combine_losses_single_loss_passthrough(torch_module):
|
||||
torch = torch_module
|
||||
losses = {"dpo": torch.tensor(0.42)}
|
||||
out = pc.combine_losses(losses, {"dpo": 1.0})
|
||||
assert math.isclose(out.item(), 0.42, abs_tol=1e-6)
|
||||
|
||||
|
||||
def test_combine_losses_two_losses_weighted_average(torch_module):
|
||||
torch = torch_module
|
||||
losses = {"dpo": torch.tensor(1.0), "simpo": torch.tensor(2.0)}
|
||||
out = pc.combine_losses(losses, {"dpo": 0.25, "simpo": 0.75})
|
||||
# 0.25*1 + 0.75*2 = 1.75
|
||||
assert math.isclose(out.item(), 1.75, abs_tol=1e-6)
|
||||
|
||||
|
||||
def test_combine_losses_rejects_key_mismatch(torch_module):
|
||||
torch = torch_module
|
||||
losses = {"dpo": torch.tensor(1.0)}
|
||||
with pytest.raises(ValueError, match="loss keys"):
|
||||
pc.combine_losses(losses, {"simpo": 1.0})
|
||||
|
||||
|
||||
def test_combine_losses_rejects_extra_loss_key(torch_module):
|
||||
"""Inverse of key_mismatch — losses key absent from weights."""
|
||||
torch = torch_module
|
||||
losses = {"dpo": torch.tensor(1.0), "extra": torch.tensor(0.5)}
|
||||
with pytest.raises(ValueError, match="loss keys"):
|
||||
pc.combine_losses(losses, {"dpo": 1.0})
|
||||
|
||||
|
||||
def test_combine_losses_rejects_empty_weights(torch_module):
|
||||
"""Empty weights mapping must raise (defence-in-depth — schema enforces 2-5)."""
|
||||
torch_module # noqa
|
||||
with pytest.raises(ValueError, match="empty"):
|
||||
pc.combine_losses({}, {})
|
||||
|
||||
|
||||
def test_combine_losses_propagates_nan_loudly(torch_module):
|
||||
"""A NaN tensor input should produce a NaN combined loss (caller handles
|
||||
the recovery via loss_watchdog) — but never silently zero out."""
|
||||
torch = torch_module
|
||||
losses = {
|
||||
"dpo": torch.tensor(float("nan")),
|
||||
"simpo": torch.tensor(1.0),
|
||||
}
|
||||
out = pc.combine_losses(losses, {"dpo": 0.5, "simpo": 0.5})
|
||||
# NaN must propagate — not get mistakenly zeroed by `0.0 * NaN`-style arithmetic.
|
||||
assert torch.isnan(out).item(), "combine_losses silently swallowed NaN"
|
||||
|
||||
|
||||
def test_combine_losses_rejects_bool_weight(torch_module):
|
||||
"""Defence-in-depth: bool is a subclass of int but the project policy
|
||||
rejects bool weights (matches v0.30.0 Candidate / v0.34.0 cost policy)."""
|
||||
torch = torch_module
|
||||
losses = {"dpo": torch.tensor(1.0), "simpo": torch.tensor(2.0)}
|
||||
with pytest.raises(TypeError, match="bool"):
|
||||
pc.combine_losses(losses, {"dpo": True, "simpo": False})
|
||||
|
||||
|
||||
def test_combine_losses_rejects_unnormalised_weights(torch_module):
|
||||
torch = torch_module
|
||||
losses = {"dpo": torch.tensor(1.0), "simpo": torch.tensor(2.0)}
|
||||
with pytest.raises(ValueError, match="sum to 1"):
|
||||
pc.combine_losses(losses, {"dpo": 0.5, "simpo": 0.4})
|
||||
|
||||
|
||||
def test_combine_losses_propagates_gradients(torch_module):
|
||||
torch = torch_module
|
||||
a = torch.tensor(2.0, requires_grad=True)
|
||||
b = torch.tensor(3.0, requires_grad=True)
|
||||
losses = {"dpo": a * 1.0, "ipo": b * 1.0}
|
||||
out = pc.combine_losses(losses, {"dpo": 0.5, "ipo": 0.5})
|
||||
out.backward()
|
||||
# d/da (0.5 a) = 0.5; d/db (0.5 b) = 0.5
|
||||
assert math.isclose(a.grad.item(), 0.5, abs_tol=1e-6)
|
||||
assert math.isclose(b.grad.item(), 0.5, abs_tol=1e-6)
|
||||
|
||||
|
||||
def test_dpo_term_matches_known_formula(torch_module):
|
||||
torch = torch_module
|
||||
pol_chosen = torch.tensor([1.0])
|
||||
pol_rejected = torch.tensor([0.0])
|
||||
ref_chosen = torch.tensor([0.5])
|
||||
ref_rejected = torch.tensor([0.0])
|
||||
beta = 0.1
|
||||
out = pc.compute_dpo_term(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta)
|
||||
# logits = 0.1 * ((1-0) - (0.5-0)) = 0.1 * 0.5 = 0.05
|
||||
# loss = -logσ(0.05) ≈ 0.6682
|
||||
expected = -math.log(1 / (1 + math.exp(-0.05)))
|
||||
assert math.isclose(out.item(), expected, abs_tol=1e-4)
|
||||
|
||||
|
||||
def test_dpo_term_requires_reference_logps(torch_module):
|
||||
torch = torch_module
|
||||
pol_chosen = torch.tensor([1.0])
|
||||
pol_rejected = torch.tensor([0.0])
|
||||
with pytest.raises(ValueError, match="reference"):
|
||||
pc.compute_dpo_term(pol_chosen, pol_rejected, None, None, 0.1)
|
||||
|
||||
|
||||
def test_ipo_term_rejects_non_positive_beta(torch_module):
|
||||
torch = torch_module
|
||||
z = torch.tensor([0.0])
|
||||
with pytest.raises(ValueError, match="beta"):
|
||||
pc.compute_ipo_term(z, z, z, z, 0.0)
|
||||
|
||||
|
||||
def test_simpo_term_length_normalised(torch_module):
|
||||
torch = torch_module
|
||||
pol_chosen = torch.tensor([2.0])
|
||||
pol_rejected = torch.tensor([2.0])
|
||||
chosen_lens = torch.tensor([1])
|
||||
rejected_lens = torch.tensor([2])
|
||||
out = pc.compute_simpo_term(
|
||||
pol_chosen, pol_rejected, beta=1.0, gamma=0.0,
|
||||
chosen_lens=chosen_lens, rejected_lens=rejected_lens,
|
||||
)
|
||||
# Normalised: chosen=2/1=2, rejected=2/2=1; logits=1*(2-1)-0=1
|
||||
expected = -math.log(1 / (1 + math.exp(-1.0)))
|
||||
assert math.isclose(out.item(), expected, abs_tol=1e-4)
|
||||
|
||||
|
||||
def test_describe_blend_format():
|
||||
out = pc.describe_blend({"dpo": 0.6, "simpo": 0.4})
|
||||
# Sorted alphabetically.
|
||||
assert out == "0.60·dpo + 0.40·simpo"
|
||||
|
||||
|
||||
def test_describe_blend_empty():
|
||||
assert pc.describe_blend(None) == "(none)"
|
||||
assert pc.describe_blend({}) == "(none)"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Compatibility validation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_validate_weight_compat_paired_only_ok():
|
||||
pc.validate_weight_compat({"dpo": 0.5, "simpo": 0.5}) # no raise
|
||||
|
||||
|
||||
def test_validate_weight_compat_bco_alone_ok():
|
||||
pc.validate_weight_compat({"bco": 1.0}) # no raise — schema already filters
|
||||
|
||||
|
||||
def test_validate_weight_compat_bco_mixed_with_paired_rejected():
|
||||
with pytest.raises(ValueError, match="bco"):
|
||||
pc.validate_weight_compat({"bco": 0.5, "dpo": 0.5})
|
||||
|
||||
|
||||
def test_needs_reference_model_dpo():
|
||||
assert pc.needs_reference_model({"dpo": 1.0}) is True
|
||||
|
||||
|
||||
def test_needs_reference_model_simpo_orpo_no_ref():
|
||||
assert pc.needs_reference_model({"simpo": 0.5, "orpo": 0.5}) is False
|
||||
|
||||
|
||||
def test_needs_reference_model_mixed():
|
||||
assert pc.needs_reference_model({"dpo": 0.5, "simpo": 0.5}) is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PreferenceTrainerWrapper integration — runtime path no longer raises
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_multi_objective_cfg(weights):
|
||||
from soup_cli.config.schema import (
|
||||
DataConfig,
|
||||
SoupConfig,
|
||||
TrainingConfig,
|
||||
)
|
||||
|
||||
return SoupConfig(
|
||||
base="HuggingFaceTB/SmolLM2-135M-Instruct",
|
||||
task="preference",
|
||||
data=DataConfig(train="data.jsonl", format="dpo"),
|
||||
training=TrainingConfig(preference_loss_weights=weights),
|
||||
output="./out",
|
||||
)
|
||||
|
||||
|
||||
def test_wrapper_setup_no_longer_raises_for_paired_blend(tmp_path, monkeypatch):
|
||||
"""The v0.40.0 NotImplementedError stub is gone for paired blends."""
|
||||
from soup_cli.trainer.preference import PreferenceTrainerWrapper
|
||||
|
||||
cfg = _make_multi_objective_cfg({"dpo": 0.6, "simpo": 0.4})
|
||||
wrapper = PreferenceTrainerWrapper(cfg, device="cpu")
|
||||
|
||||
# We bypass real model loading by intercepting _build_multi_objective.
|
||||
# The point is that setup() no longer aborts at the stub-then-live gate.
|
||||
called = {"build": False}
|
||||
|
||||
def _stub_build():
|
||||
called["build"] = True
|
||||
return None # No real inner trainer — just exercise the gate.
|
||||
|
||||
monkeypatch.setattr(wrapper, "_build_multi_objective", _stub_build)
|
||||
wrapper.setup({"train": []})
|
||||
assert called["build"], "wrapper did not enter multi-objective path"
|
||||
|
||||
|
||||
def test_wrapper_setup_rejects_bco_mixed_with_paired(monkeypatch):
|
||||
from soup_cli.trainer.preference import PreferenceTrainerWrapper
|
||||
|
||||
cfg = _make_multi_objective_cfg({"bco": 0.5, "dpo": 0.5})
|
||||
wrapper = PreferenceTrainerWrapper(cfg, device="cpu")
|
||||
with pytest.raises(ValueError, match="bco"):
|
||||
wrapper.setup({"train": []})
|
||||
|
||||
|
||||
def test_wrapper_describes_active_blend():
|
||||
"""The advisory message is owned by ``_build_multi_objective``; verify
|
||||
the helper's source contains the user-facing blend description."""
|
||||
import inspect
|
||||
|
||||
from soup_cli.trainer.preference import PreferenceTrainerWrapper
|
||||
|
||||
src = inspect.getsource(PreferenceTrainerWrapper._build_multi_objective)
|
||||
assert "describe_blend" in src
|
||||
assert "primary" in src.lower()
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
"""v0.40.1 Part C tests — autopilot fallback / transformers cap / quickstart
|
||||
GPU-aware model pick / lr_finder import regression.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
# --- C3: autopilot fallback to 1B (not 7B) ---------------------------------
|
||||
|
||||
|
||||
def test_guess_params_unknown_model_falls_back_to_1b():
|
||||
from soup_cli.autopilot.analyzer import _guess_params_from_name
|
||||
|
||||
assert _guess_params_from_name("tiny-gpt2") == 1.0
|
||||
|
||||
|
||||
def test_guess_params_extracts_billions_first():
|
||||
from soup_cli.autopilot.analyzer import _guess_params_from_name
|
||||
|
||||
assert _guess_params_from_name("Qwen/Qwen2.5-7B-Instruct") == 7.0
|
||||
|
||||
|
||||
def test_guess_params_handles_millions_suffix():
|
||||
from soup_cli.autopilot.analyzer import _guess_params_from_name
|
||||
|
||||
# SmolLM2-135M → 0.135 B
|
||||
assert _guess_params_from_name("HuggingFaceTB/SmolLM2-135M-Instruct") == pytest.approx(
|
||||
0.135, abs=1e-3
|
||||
)
|
||||
|
||||
|
||||
def test_probe_cache_returns_none_for_missing_repo():
|
||||
from soup_cli.autopilot.analyzer import _probe_cache_param_count
|
||||
|
||||
# Random name guaranteed not in cache.
|
||||
assert _probe_cache_param_count("nonexistent-org/never-cached-XYZ") is None
|
||||
|
||||
|
||||
# --- C5: doctor flags transformers 5.x ------------------------------------
|
||||
|
||||
|
||||
def test_version_ge_handles_dev_suffix():
|
||||
from soup_cli.commands.doctor import _version_ge
|
||||
|
||||
assert _version_ge("5.0.0.dev0", "5.0.0") is True
|
||||
assert _version_ge("4.36.2", "5.0.0") is False
|
||||
|
||||
|
||||
def test_version_ge_short_version_string():
|
||||
from soup_cli.commands.doctor import _version_ge
|
||||
|
||||
assert _version_ge("5", "5.0.0") is True
|
||||
assert _version_ge("4.99", "5.0.0") is False
|
||||
|
||||
|
||||
def test_max_exclusive_table_caps_transformers():
|
||||
from soup_cli.commands.doctor import _MAX_EXCLUSIVE
|
||||
|
||||
assert _MAX_EXCLUSIVE.get("transformers") == "5.0.0"
|
||||
|
||||
|
||||
# --- G12: lr_finder real loop uses load_raw_data ---------------------------
|
||||
|
||||
|
||||
def test_live_lr_sweep_uses_load_raw_data_not_load_local():
|
||||
import inspect
|
||||
import re
|
||||
|
||||
from soup_cli.commands.train import _live_lr_sweep_from_config
|
||||
|
||||
src = inspect.getsource(_live_lr_sweep_from_config)
|
||||
assert "load_raw_data" in src
|
||||
# The broken import was `from soup_cli.data.loader import load_local`;
|
||||
# match only that import line to avoid false positives on the comment
|
||||
# describing the fix.
|
||||
assert not re.search(r"from\s+soup_cli\.data\.loader\s+import\s+load_local", src), (
|
||||
"load_local was removed; live sweep must use load_raw_data"
|
||||
)
|
||||
|
||||
|
||||
# --- G1: quickstart picks SmolLM2 on ≤6 GB VRAM ----------------------------
|
||||
|
||||
|
||||
def test_pick_quickstart_model_no_cuda_uses_tinyllama():
|
||||
from soup_cli.commands import quickstart as qs
|
||||
|
||||
with patch("torch.cuda.is_available", return_value=False):
|
||||
model, advisory = qs._pick_quickstart_model()
|
||||
assert model == qs._DEFAULT_MODEL
|
||||
assert advisory is None
|
||||
|
||||
|
||||
def test_pick_quickstart_model_low_vram_switches_to_smollm():
|
||||
from soup_cli.commands import quickstart as qs
|
||||
|
||||
fake_props = type("P", (), {"total_memory": int(4 * 1024**3)})() # 4 GB
|
||||
with (
|
||||
patch("torch.cuda.is_available", return_value=True),
|
||||
patch("torch.cuda.get_device_properties", return_value=fake_props),
|
||||
):
|
||||
model, advisory = qs._pick_quickstart_model()
|
||||
assert model == qs._LOW_VRAM_MODEL
|
||||
assert advisory is not None
|
||||
assert "VRAM" in advisory
|
||||
|
||||
|
||||
def test_pick_quickstart_model_high_vram_keeps_default():
|
||||
from soup_cli.commands import quickstart as qs
|
||||
|
||||
fake_props = type("P", (), {"total_memory": int(24 * 1024**3)})() # 24 GB
|
||||
with (
|
||||
patch("torch.cuda.is_available", return_value=True),
|
||||
patch("torch.cuda.get_device_properties", return_value=fake_props),
|
||||
):
|
||||
model, advisory = qs._pick_quickstart_model()
|
||||
assert model == qs._DEFAULT_MODEL
|
||||
assert advisory is None
|
||||
|
||||
|
||||
# --- N3: GPU diagnostic distinguishes CPU build from no GPU ----------------
|
||||
|
||||
|
||||
def test_detect_gpu_hw_without_torch_cuda_no_nvidia_smi():
|
||||
import subprocess
|
||||
|
||||
from soup_cli.commands.doctor import _detect_gpu_hw_without_torch_cuda
|
||||
|
||||
with patch("shutil.which", return_value=None):
|
||||
assert _detect_gpu_hw_without_torch_cuda() == ""
|
||||
|
||||
# nvidia-smi present but failing → empty advisory.
|
||||
completed = subprocess.CompletedProcess(args=[], returncode=1, stdout="", stderr="")
|
||||
with (
|
||||
patch("shutil.which", return_value="/usr/bin/nvidia-smi"),
|
||||
patch("subprocess.run", return_value=completed),
|
||||
):
|
||||
assert _detect_gpu_hw_without_torch_cuda() == ""
|
||||
|
||||
|
||||
def test_detect_gpu_hw_returns_advisory_when_smi_succeeds():
|
||||
import subprocess
|
||||
|
||||
from soup_cli.commands.doctor import _detect_gpu_hw_without_torch_cuda
|
||||
|
||||
completed = subprocess.CompletedProcess(
|
||||
args=[], returncode=0, stdout="NVIDIA GeForce RTX 3050\n", stderr=""
|
||||
)
|
||||
with (
|
||||
patch("shutil.which", return_value="/usr/bin/nvidia-smi"),
|
||||
patch("subprocess.run", return_value=completed),
|
||||
):
|
||||
advisory = _detect_gpu_hw_without_torch_cuda()
|
||||
assert "RTX 3050" in advisory
|
||||
assert "cu121" in advisory
|
||||
|
||||
|
||||
# --- N4: dual-Python interpreter detector ---------------------------------
|
||||
|
||||
|
||||
def test_detect_dual_python_no_path_python():
|
||||
from soup_cli.commands.doctor import _detect_dual_python_interpreters
|
||||
|
||||
with patch("shutil.which", return_value=None):
|
||||
assert _detect_dual_python_interpreters() == ""
|
||||
|
||||
|
||||
# --- M1: rich version probe falls back to importlib.metadata --------------
|
||||
|
||||
|
||||
def test_doctor_rich_version_probe_uses_metadata_when_module_lacks_attr():
|
||||
"""Sanity: importing rich and probing version doesn't return '?'."""
|
||||
from importlib.metadata import version
|
||||
|
||||
import rich
|
||||
|
||||
# Rich does export __version__, so this just guards the importlib.metadata
|
||||
# fallback path is reachable and returns a real string for rich.
|
||||
assert version("rich") # non-empty
|
||||
assert hasattr(rich, "__version__") or version("rich")
|
||||
|
|
@ -0,0 +1,125 @@
|
|||
"""v0.40.1 Part D — CLI UX consistency tests (highest-leverage subset).
|
||||
|
||||
Closes:
|
||||
- H4: Template list dynamic sync
|
||||
- M2: `soup init --force` flag
|
||||
- N6: `soup history` suggests `data registry` for dataset names
|
||||
- N2: `soup migrate` JSONL friendly error
|
||||
- G10: `soup eval custom -o` written regardless of `--attach-to-registry`
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
# --- H4: Template help is dynamically generated --------------------------
|
||||
|
||||
|
||||
def test_init_template_help_lists_all_templates():
|
||||
from soup_cli.commands.init import _template_help_string
|
||||
from soup_cli.templates import list_templates
|
||||
|
||||
help_text = _template_help_string()
|
||||
for template_name in list_templates():
|
||||
assert template_name in help_text, (
|
||||
f"template {template_name!r} missing from --template help"
|
||||
)
|
||||
|
||||
|
||||
def test_init_template_help_includes_bco():
|
||||
"""v0.40.0 added BCO; H4 must show it without a manual help-text edit."""
|
||||
from soup_cli.commands.init import _template_help_string
|
||||
|
||||
assert "bco" in _template_help_string()
|
||||
|
||||
|
||||
# --- M2: soup init --force flag ------------------------------------------
|
||||
|
||||
|
||||
def test_init_force_flag_overwrites_without_prompt(tmp_path):
|
||||
from soup_cli.cli import app
|
||||
|
||||
runner = CliRunner()
|
||||
target = tmp_path / "soup.yaml"
|
||||
target.write_text("base: existing", encoding="utf-8")
|
||||
|
||||
# Without --force, prompts (we send 'n' to abort).
|
||||
result = runner.invoke(app, ["init", "--output", str(target)], input="n\n")
|
||||
assert result.exit_code == 0
|
||||
assert target.read_text(encoding="utf-8") == "base: existing", (
|
||||
"without --force the user-typed 'n' should abort and preserve file"
|
||||
)
|
||||
|
||||
# With --force, overwrites silently using a registered template.
|
||||
result = runner.invoke(
|
||||
app, ["init", "--output", str(target), "--template", "chat", "--force"]
|
||||
)
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert target.read_text(encoding="utf-8") != "base: existing"
|
||||
|
||||
|
||||
# --- N2: soup migrate JSONL friendly error ------------------------------
|
||||
|
||||
|
||||
def test_migrate_jsonl_input_yields_friendly_error(tmp_path: Path):
|
||||
from soup_cli.commands.migrate import _looks_like_jsonl
|
||||
|
||||
jsonl = tmp_path / "data.jsonl"
|
||||
jsonl.write_text('{"prompt": "hi"}\n{"prompt": "world"}\n', encoding="utf-8")
|
||||
assert _looks_like_jsonl(jsonl) is True
|
||||
|
||||
|
||||
def test_migrate_yaml_does_not_look_like_jsonl(tmp_path: Path):
|
||||
from soup_cli.commands.migrate import _looks_like_jsonl
|
||||
|
||||
yml = tmp_path / "config.yaml"
|
||||
yml.write_text("base: foo\ntask: sft\n", encoding="utf-8")
|
||||
assert _looks_like_jsonl(yml) is False
|
||||
|
||||
|
||||
def test_migrate_skips_blank_lines_when_sniffing(tmp_path: Path):
|
||||
from soup_cli.commands.migrate import _looks_like_jsonl
|
||||
|
||||
f = tmp_path / "blanks.jsonl"
|
||||
f.write_text('\n\n \n{"key": 1}\n', encoding="utf-8")
|
||||
assert _looks_like_jsonl(f) is True
|
||||
|
||||
|
||||
# --- N6: soup history suggests dataset registry --------------------------
|
||||
|
||||
|
||||
def test_history_dataset_registry_helper_handles_missing():
|
||||
from soup_cli.commands.history import _name_exists_in_dataset_registry
|
||||
|
||||
# Should never raise even if registry module / file is missing.
|
||||
assert isinstance(
|
||||
_name_exists_in_dataset_registry("definitely-not-a-dataset-xxx"), bool
|
||||
)
|
||||
|
||||
|
||||
# --- G10: soup eval custom --output writes JSON without --attach-to-registry
|
||||
|
||||
|
||||
def test_eval_custom_output_arg_described_as_independent():
|
||||
"""The --output help string must mention it's honored without attach."""
|
||||
import inspect
|
||||
|
||||
from soup_cli.commands.eval import custom
|
||||
|
||||
src = inspect.getsource(custom)
|
||||
assert "Honored independently" in src or "G10" in src
|
||||
|
||||
|
||||
def test_eval_custom_no_longer_shadows_output_with_response():
|
||||
"""Source-level guard against the loop-variable shadow regression."""
|
||||
import inspect
|
||||
|
||||
from soup_cli.commands.eval import custom
|
||||
|
||||
src = inspect.getsource(custom)
|
||||
# Old buggy line: ``output = generate_fn(eval_task.prompt)``.
|
||||
# New line uses ``response`` to avoid shadowing the CLI ``output`` arg.
|
||||
assert "response = generate_fn" in src
|
||||
assert "output = generate_fn" not in src
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
"""v0.40.1 Part E — Recipe id fuzzy-match + UX papercuts.
|
||||
|
||||
Closes:
|
||||
- M3: `soup recipes show <typo>` suggests close matches via difflib
|
||||
- JSONL BOM auto-strip (Windows PowerShell users)
|
||||
- `soup data sample` filename includes strategy to prevent overwrite
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
# --- M3: recipe fuzzy-match -----------------------------------------------
|
||||
|
||||
|
||||
def test_suggest_recipes_returns_close_matches():
|
||||
from soup_cli.commands.recipes import _suggest_recipes
|
||||
from soup_cli.recipes.catalog import RECIPES
|
||||
|
||||
if not RECIPES:
|
||||
return
|
||||
real_name = next(iter(RECIPES.keys()))
|
||||
# Mutate one char in the middle to simulate a typo.
|
||||
if len(real_name) >= 4:
|
||||
typo = real_name[:2] + "x" + real_name[3:]
|
||||
else:
|
||||
typo = real_name + "x"
|
||||
suggestions = _suggest_recipes(typo)
|
||||
assert real_name in suggestions or any(
|
||||
s in real_name or real_name in s for s in suggestions
|
||||
)
|
||||
|
||||
|
||||
def test_suggest_recipes_empty_for_garbage():
|
||||
from soup_cli.commands.recipes import _suggest_recipes
|
||||
|
||||
# Wholly unrelated query should return [] (cutoff=0.6).
|
||||
suggestions = _suggest_recipes("zzqqxxyyy_no_match_at_all")
|
||||
assert suggestions == []
|
||||
|
||||
|
||||
# --- BOM auto-strip in JSONL loader ---------------------------------------
|
||||
|
||||
|
||||
def test_jsonl_loader_strips_utf8_bom(tmp_path: Path):
|
||||
from soup_cli.data.loader import _load_jsonl
|
||||
|
||||
f = tmp_path / "with_bom.jsonl"
|
||||
# Write BOM + valid JSONL via binary mode so we control the bytes.
|
||||
f.write_bytes(
|
||||
b"\xef\xbb\xbf" # UTF-8 BOM
|
||||
+ json.dumps({"prompt": "hi"}).encode("utf-8")
|
||||
+ b"\n"
|
||||
+ json.dumps({"prompt": "world"}).encode("utf-8")
|
||||
+ b"\n"
|
||||
)
|
||||
rows = _load_jsonl(f)
|
||||
assert len(rows) == 2
|
||||
assert rows[0]["prompt"] == "hi"
|
||||
|
||||
|
||||
def test_jsonl_loader_no_bom_still_works(tmp_path: Path):
|
||||
from soup_cli.data.loader import _load_jsonl
|
||||
|
||||
f = tmp_path / "no_bom.jsonl"
|
||||
f.write_text('{"prompt": "alpha"}\n{"prompt": "beta"}\n', encoding="utf-8")
|
||||
rows = _load_jsonl(f)
|
||||
assert [r["prompt"] for r in rows] == ["alpha", "beta"]
|
||||
|
||||
|
||||
# --- data sample default filename includes strategy -----------------------
|
||||
|
||||
|
||||
def test_data_sample_default_filename_includes_strategy():
|
||||
"""Source-level invariant: default ``out_path`` template names the strategy."""
|
||||
import inspect
|
||||
|
||||
from soup_cli.commands.data import sample_data
|
||||
|
||||
src = inspect.getsource(sample_data)
|
||||
assert '_sampled_{strategy}.jsonl' in src, (
|
||||
"default sampled filename must embed the strategy to prevent overwrite"
|
||||
)
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
"""UTF-8 stdio bootstrap tests (v0.40.1 Part A).
|
||||
|
||||
Closes QA findings C1, C4, H1, N5, N8, G5 — all surface as Windows codepage
|
||||
(cp1251 / cp1252) UnicodeEncodeError when Rich/Typer prints non-ASCII (β, ✓,
|
||||
box-drawing, etc.).
|
||||
|
||||
The fix is a single CLI-bootstrap call that reconfigures stdout/stderr to
|
||||
UTF-8 on Windows. POSIX is already UTF-8 so this is a no-op there.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import os
|
||||
from unittest.mock import patch
|
||||
|
||||
from soup_cli.utils import encoding as enc
|
||||
|
||||
|
||||
def test_force_utf8_stdio_noop_on_posix():
|
||||
"""POSIX terminals are UTF-8 already; bootstrap should not raise."""
|
||||
with patch.object(enc.sys, "platform", "linux"):
|
||||
# Should not raise even when stdout has no reconfigure.
|
||||
enc.force_utf8_stdio()
|
||||
|
||||
|
||||
def test_force_utf8_stdio_sets_pythonioencoding_on_windows(monkeypatch):
|
||||
monkeypatch.delenv("PYTHONIOENCODING", raising=False)
|
||||
monkeypatch.setattr(enc.sys, "platform", "win32")
|
||||
|
||||
# Provide stdout/stderr with .reconfigure that records the call.
|
||||
calls: list[dict] = []
|
||||
|
||||
class FakeStream:
|
||||
def reconfigure(self, **kwargs):
|
||||
calls.append(kwargs)
|
||||
|
||||
monkeypatch.setattr(enc.sys, "stdout", FakeStream())
|
||||
monkeypatch.setattr(enc.sys, "stderr", FakeStream())
|
||||
|
||||
enc.force_utf8_stdio()
|
||||
|
||||
assert os.environ.get("PYTHONIOENCODING") == "utf-8"
|
||||
assert any(c.get("encoding") == "utf-8" for c in calls)
|
||||
|
||||
|
||||
def test_force_utf8_stdio_does_not_override_existing_pythonioencoding(monkeypatch):
|
||||
"""User-set PYTHONIOENCODING is preserved (we use setdefault)."""
|
||||
monkeypatch.setenv("PYTHONIOENCODING", "latin-1")
|
||||
monkeypatch.setattr(enc.sys, "platform", "win32")
|
||||
|
||||
class FakeStream:
|
||||
def reconfigure(self, **kwargs):
|
||||
pass
|
||||
|
||||
monkeypatch.setattr(enc.sys, "stdout", FakeStream())
|
||||
monkeypatch.setattr(enc.sys, "stderr", FakeStream())
|
||||
|
||||
enc.force_utf8_stdio()
|
||||
assert os.environ.get("PYTHONIOENCODING") == "latin-1"
|
||||
|
||||
|
||||
def test_force_utf8_stdio_swallows_reconfigure_errors(monkeypatch):
|
||||
"""Some streams (redirected to file objects without reconfigure) must not crash."""
|
||||
monkeypatch.setattr(enc.sys, "platform", "win32")
|
||||
|
||||
class BadStream:
|
||||
def reconfigure(self, **kwargs):
|
||||
raise OSError("not a tty")
|
||||
|
||||
monkeypatch.setattr(enc.sys, "stdout", BadStream())
|
||||
monkeypatch.setattr(enc.sys, "stderr", BadStream())
|
||||
|
||||
# Must not raise.
|
||||
enc.force_utf8_stdio()
|
||||
|
||||
|
||||
def test_force_utf8_stdio_handles_missing_reconfigure(monkeypatch):
|
||||
"""Older Python or non-TextIO streams without .reconfigure must not crash."""
|
||||
monkeypatch.setattr(enc.sys, "platform", "win32")
|
||||
monkeypatch.setattr(enc.sys, "stdout", io.BytesIO()) # no reconfigure attr
|
||||
monkeypatch.setattr(enc.sys, "stderr", io.BytesIO())
|
||||
|
||||
enc.force_utf8_stdio()
|
||||
|
||||
|
||||
def test_cli_bootstrap_calls_force_utf8(monkeypatch):
|
||||
"""soup_cli.cli imports `force_utf8_stdio` so it runs at module load."""
|
||||
import soup_cli.cli as cli_mod
|
||||
|
||||
# Module-level guarantee: the symbol is imported.
|
||||
assert hasattr(cli_mod, "_utf8_bootstrap_done") or hasattr(
|
||||
cli_mod, "force_utf8_stdio"
|
||||
) or "force_utf8_stdio" in dir(enc)
|
||||
|
||||
|
||||
def test_writers_use_utf8_encoding():
|
||||
"""Static check that hot-path writers pass encoding='utf-8' to open()."""
|
||||
import pathlib
|
||||
|
||||
audited_files = [
|
||||
"soup_cli/commands/quickstart.py",
|
||||
"soup_cli/commands/init.py",
|
||||
"soup_cli/commands/migrate.py",
|
||||
]
|
||||
repo_root = pathlib.Path(__file__).resolve().parent.parent
|
||||
for rel in audited_files:
|
||||
path = repo_root / rel
|
||||
if not path.exists():
|
||||
continue
|
||||
text = path.read_text(encoding="utf-8")
|
||||
# Every text-mode write should specify encoding.
|
||||
# We grep for `open(...,"w"` patterns and assert utf-8 nearby.
|
||||
# Permissive check: if file contains `open(` with `"w"` or `'w'`, it must
|
||||
# also reference encoding="utf-8" in the same call (rough heuristic — full
|
||||
# static verification is left to ruff + lint hooks).
|
||||
# Just ensure the file at least mentions utf-8 if it contains text writes.
|
||||
if 'open(' in text and (', "w"' in text or ", 'w'" in text):
|
||||
assert 'encoding="utf-8"' in text or "encoding='utf-8'" in text, (
|
||||
f"{rel} writes text without explicit utf-8 encoding"
|
||||
)
|
||||
Loading…
Reference in New Issue