diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f2adaaa..9c1c073 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (152 files, 5389 tests) +tests/ - Test suite (156 files, 5628 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index 9181989..d195a19 100644 --- a/README.md +++ b/README.md @@ -43,15 +43,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.42.0 — Data Pipeline Pro**: closes the data-tooling gap with Axolotl + LlamaFactory in one release. 18 features across formats, remote loading, AOT preprocessing, advanced masking, vocab expansion, and document ingestion. +**v0.43.0 — Tracker & Eval Pro**: closes the observability gap with all three competitors in one release. 18 features across tracker integrations, NLG eval metrics, profiling extras, and bundled demo datasets. -- **5 new data formats** — `prm` (PRM stepwise-supervised), `pre_tokenized` (LF `tokenized_path` / Axolotl `empty` type), `input_output` (template-free segments+labels with no chat template), `video`, and `multimodal` (axolotl typed content-parts schema with text / image / audio / video). Each is loud-fail validated — bool labels, null-byte rejection, length caps on path/url-like fields. -- **Remote dataset loading (schema gate live, fsspec wiring deferred to v0.42.1)** — `data.train: s3://my-bucket/path` accepts the 7-scheme allowlist (`s3` / `gs` / `gcs` / `az` / `abfs` / `abfss` / `oci`). Bucket name regex is RFC-3986-tight, userinfo / fragment / query strings are rejected (the query-string gate is SSRF-adjacent — fsspec backends interpret `?endpoint_url=…` as config overrides). New schema fields: `streaming: bool`, `buffer_size: int [1, 1_000_000]`, `shards: int [1, 1024]`. -- **AOT preprocessing + tokenized cache** — new `soup data preprocess ` CLI emits a deterministic 16-char SHA-256 cache key from `(dataset, tokenizer, max_length, format)`. Pair with the new `data.tokenized_path` schema field to skip the tokenize stage on subsequent runs. -- **Advanced masking + multi-dataset interleave** — `data.interleave: concat / under / over / {strategy: probs, probs: [...]}` (sum-to-1 ±1e-6, max 32 datasets, `InterleaveSpec` is a frozen dataclass). Plus `mask_history`, `train_on_prompt` (mutually exclusive with `train_on_responses_only`), `eval_on_each_dataset`, `split_thinking` for Qwen3-style `` masking, and per-image min/max pixels + `image_resize_algorithm` Literal + `video_fps` / `video_maxlen` / `video_dir`. -- **Vocab expansion + custom prompt strategies** — `data.add_new_tokens: ["", ""]` + `new_special_tokens` (cap 10_000 entries, no duplicates, null-byte rejected) + `resize_vocab: bool` cross-validator. New `prompt_strategy: my_module.path:my_fn` field with regex validation lays groundwork for v0.42.1 runtime invocation. Plus axolotl-style `skip_prepare_dataset` / `remove_unused_columns` escape hatches. -- **Document ingestion** — new `soup data ingest ` (PDF / DOCX / MD / TXT) → JSONL with one row per page / paragraph. Lazy-imports `pypdf` / `python-docx` so missing optional deps never break `soup data --help`. Symlink-rejected, cwd-confined. -- **+140 net new tests** — covers all 18 features end-to-end: schema validators, cross-validators, frozen-dataclass mutation guards, URL allowlist + bucket regex / single-char bucket / SSRF query rejection, every new format converter happy + failure path, CLI help + smoke + symlink-rejection, full SoupConfig YAML round-trip. +- **Tracker integrations** — new `--tracker` flag on `soup train` accepts `mlflow` / `swanlab` / `trackio` (mutually exclusive with `--wandb` / `--tensorboard`). Closed allowlist via `MappingProxyType`-locked registry; case-insensitive lookup; null-byte and >32-char inputs rejected. PostHog telemetry payload schema lands as opt-in (`SOUP_TELEMETRY=1`) with hardware-info-only fields — no model names, dataset paths, or config contents. Live network code deferred to v0.43.1. +- **BLEU + ROUGE-1 / ROUGE-2 / ROUGE-L** — pure-Python implementations exposed via `soup eval custom --metric bleu|rouge_l|...`. Standard BLEU policy: any zero-precision n-gram collapses score to 0.0; Chen & Cherry smoothing (default on) only smooths zero-correct buckets where `total[n] > 0`. Plus `effective_tokens_per_second` as a metric — `unmasked_tokens / wall_clock_seconds`, returns `None` when wall_clock ≤ 0 (no fabrication). +- **KL-divergence calibration framework** — `soup_cli.eval.calibrate.run_calibration(baseline_logits, quantized_logits)` returns a frozen `CalibrationReport(mean_kl, per_prompt_kl, delta_status)` with OK / MINOR / MAJOR thresholds at 0.05 / 0.20. Pure-math kernel; bring your own logit pairs. +- **Model Arena (Elo tournament)** — `soup_cli.eval.arena.Tournament` with K=32 default Elo, 256-model cap, 1M-match cap, `MappingProxyType` immutability on the public `ratings` view, and Rich-markup `[`/`]` rejection on model names so leaderboards can't be markup-injected. +- **Profiling extras** — `memory_snapshot_context` wraps `torch.cuda.memory._record_memory_history` with cwd-confined snapshot path. `nccl_bandwidth_check` ships a reference-bandwidth table (h100/a100/v100/rtx-series, NVLink + PCIe) classifying measured bandwidth as OK ≥80% / MINOR ≥50% / MAJOR <50%. `soup doctor --vscode` writes a `.vscode/launch.json` with `soup train` + pytest configs, symlink-rejected at the target path. +- **Bundled demo datasets** — new `soup data demo` lists 4 ready-to-use JSONL fixtures (alpaca / sharegpt / dpo / grpo). `soup data demo alpaca_demo --output ./mine.jsonl` copies the bundle for instant `soup train` warm-up. Staged-tempfile write with atomic rename — mid-stream rejection never leaves a partial file. +- **+239 net new tests** — covers all 18 features: tracker name allowlist, telemetry payload schema invariant (no user data leaks), BLEU/ROUGE corner cases, KL thresholds, Elo math + tournament invariants, NCCL bandwidth boundaries, vscode TOCTOU symlink rejection, demo bundle atomic-rename + cwd containment + size cap. ## Why Soup? @@ -2443,6 +2443,155 @@ Every completed run also stores an estimated cost (`$` per run) computed from th captured GPU device name and duration. `soup runs show` renders `—` for CPU / MPS / unknown GPUs (no fabricated zeros). +### Tracker integrations (--tracker mlflow / swanlab / trackio) + +```bash +# Stream metrics to MLflow (set MLFLOW_TRACKING_URI to your server URL) +soup train --config soup.yaml --tracker mlflow + +# Or SwanLab (cloud or local) +soup train --config soup.yaml --tracker swanlab + +# Or Trackio (offline-friendly batched upload) +soup train --config soup.yaml --tracker trackio +``` + +`--tracker` is mutually exclusive with `--wandb` and `--tensorboard`. Soup +validates the tracker name against a closed allowlist (`mlflow` / `swanlab` / +`trackio` / `wandb` / `tensorboard` / `none`); the upstream package itself is +loaded by HF Trainer at run time, so install the one you need separately: + +```bash +pip install mlflow # or: swanlab / trackio +``` + +### Telemetry (opt-in) + +Soup ships a hardware-info-only telemetry payload (Soup version + command + +Python major.minor + OS + arch + duration). It is **off by default** and never +sends model names, dataset paths, or config contents. Enable explicitly: + +```bash +SOUP_TELEMETRY=1 soup train --config soup.yaml +``` + +The PostHog network upload itself is deferred to v0.43.1; v0.43.0 ships the +payload schema only so you can audit it before opting in. + +## NLG Evaluation Metrics (BLEU + ROUGE) + +Pure-Python BLEU + ROUGE-1 / ROUGE-2 / ROUGE-L for `soup eval custom`: + +```python +from soup_cli.utils.nlg_metrics import ( + bleu_score, rouge_l_score, compute_nlg_metric, NLG_METRICS, + effective_tokens_per_second, +) + +bleu_score(["the cat sat on the mat"], ["the cat sat on the mat"]) +# 1.0 +rouge_l_score(["the quick brown fox"], ["a quick brown dog"]) +# 0.5 +compute_nlg_metric("rouge_2", preds, refs) +# generic dispatch by canonical name + +effective_tokens_per_second(unmasked_tokens=12_500_000, wall_clock_seconds=600.0) +# 20833.33 — None when wall_clock <= 0 (no fabrication) +``` + +Smoothed BLEU uses Chen & Cherry epsilon for zero-correct buckets where +`total[n] > 0`; empty buckets (e.g. predictions shorter than `max_n` tokens) +force the score to 0.0. + +## Quant Calibration (KL Divergence) + +Compare a quantized model to a full-precision baseline on a small fixed prompt +set. OK / MINOR / MAJOR thresholds at 0.05 / 0.20 mean KL — same scale as +`soup eval quant-check`. + +```python +from soup_cli.eval.calibrate import run_calibration + +# baseline_logits / quantized_logits: list[list[float]] aligned per-prompt +report = run_calibration(baseline_logits, quantized_logits) +print(report.delta_status, report.mean_kl) +# OK 0.012 +``` + +The kernel is pure-math and capped at 10 000 prompts to defend against +accidental OOM. `CalibrationReport` is a frozen dataclass. + +## Model Arena (Elo Tournament) + +Local leaderboard with Elo ratings (K=32, base 1500). Bring your own pairwise +winners — Soup just keeps the books: + +```python +from soup_cli.eval.arena import Tournament + +t = Tournament() +t.record("llama-3.1-8b-finetune", "qwen2.5-7b-finetune", winner="a") +t.record("llama-3.1-8b-finetune", "mistral-7b-finetune", winner="draw") +for row in t.leaderboard(): + print(row) +``` + +Caps: 256 models per tournament, 1M matches. Model names with `[` or `]` +characters are rejected so leaderboard rows can't be markup-injected. + +## Profiling Extras + +CUDA memory snapshots, anomaly tracing, and an NCCL bandwidth reference table: + +```python +from soup_cli.utils.profiling_v0_43 import ( + memory_snapshot_context, detect_anomaly_context, nccl_bandwidth_check, +) + +with memory_snapshot_context("run-123") as path: + train_step() + # On CUDA, dumps profiles/run-123.snapshot.pickle on exit. + +with detect_anomaly_context(): + train_step() + # torch.autograd.set_detect_anomaly(True) + +result = nccl_bandwidth_check( + gpu="h100", link="nvlink", measured_gb_per_sec=400.0, +) +# {'expected_gb_per_sec': 450.0, 'measured_gb_per_sec': 400.0, +# 'ratio': 0.8889, 'status': 'OK'} +``` + +## VS Code Setup (`.vscode/launch.json`) + +One-shot writer for a sane debugger config: + +```python +from soup_cli.utils.vscode_setup import write_vscode_launch +write_vscode_launch(config_path="soup.yaml") +# Writes ./.vscode/launch.json with `soup train` + pytest entries. +``` + +Symlink-rejected at the target path regardless of `force=True` to defend +against pre-placed symlinks redirecting the write outside cwd. + +## Demo Datasets (`soup data demo`) + +Tiny JSONL fixtures bundled with Soup so you can warm up `soup train` without +hunting for data: + +```bash +# List available bundles +soup data demo + +# Copy one into the current directory +soup data demo alpaca_demo --output ./alpaca.jsonl +``` + +Bundles: `alpaca_demo`, `sharegpt_demo`, `dpo_demo`, `grpo_demo`. Output path +must stay under cwd; existing files are not overwritten. + ## Observability & Dev UX Tools that explain *why* a run misbehaved instead of dumping a stack trace. @@ -2658,6 +2807,9 @@ soup data register --name my-ds --path d.jsonl --format alpaca Register dataset soup data unregister --name my-ds Remove from registry soup data push --input d.jsonl --hf-dataset user/name Upload local JSONL as HF dataset soup data registry List all registered datasets +soup data demo List bundled demo JSONL fixtures +soup data demo alpaca_demo --output ./d.jsonl Copy a bundled demo JSONL fixture +soup train --config soup.yaml --tracker mlflow MLflow / SwanLab / Trackio integration soup profile --config soup.yaml Estimate memory/speed before training soup profile --config soup.yaml --gpu a100 Estimate for specific GPU soup profile --config soup.yaml --json Machine-readable output diff --git a/SECURITY.md b/SECURITY.md index 6cc5bef..742bdf0 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,11 +9,11 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.42.0 -- Full support (latest) +- v0.43.0 -- Full support (latest) +- v0.42.0 -- Full support - v0.41.0-v0.41.x -- Full support -- v0.40.0-v0.40.x -- Full support -- v0.39.0-0.39.x -- Bug-fix support only -- v0.38.x and below -- No support +- v0.40.0-v0.40.x -- Bug-fix support only +- v0.39.x and below -- No support ## Reporting a Vulnerability @@ -144,6 +144,8 @@ No known critical vulnerabilities in current releases. - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input. - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS` +- **v0.43.0 — Tracker & Eval Pro**: 18 features across 4 Parts. New `soup_cli/utils/trackers.py` ships an immutable `_REPORT_TO_BACKENDS` `MappingProxyType` allowlist (`wandb` / `tensorboard` / `mlflow` / `swanlab` / `trackio` / `none`); `validate_tracker_name` rejects non-string / empty / null-byte / >32-char inputs. PostHog telemetry is opt-IN (default OFF) via `SOUP_TELEMETRY=1` env var; `build_telemetry_payload` schema is closed-key (soup_version / command / python major.minor / os / arch / duration) — no model names, dataset paths, or config contents leak in the payload. Live PostHog network code deferred to v0.43.1. New `--tracker` flag on `soup train` mutually exclusive with `--wandb` / `--tensorboard` via `resolve_report_to`. `tracker_backend = tracker` capture in `commands/train.py` defends against the Typer parameter being shadowed by the local `ExperimentTracker()` instance (review fix HIGH #1 — silent `--tracker` drop regression). `nlg_metrics.py` is pure-math: BLEU `_lcs_length` allocates a fresh row per outer iter (review fix HIGH #2 — first-cut had a double-buffer re-zero that destroyed the last row). `effective_tokens_per_second` returns `None` on non-positive wall_clock (no fabrication). `eval/calibrate.py` `CalibrationReport` is `@dataclass(frozen=True)` with `FrozenInstanceError` test (matches v0.32.0 / v0.39.0 / v0.41.0 frozen-dataclass policy); `run_calibration` caps prompts at 10_000 (DoS defence). `eval/arena.py` `Tournament` enforces 256-model cap, 1M-match cap, `MappingProxyType` view on `ratings` so external callers cannot mutate Elo state (LOW review fix). `_validate_model_name` rejects null-byte / >128-char AND Rich markup metacharacters `[` / `]` (security review fix LOW #5 — leaderboard markup-injection defence). `update_elo` rejects bool / non-finite / NaN `k`. `eval/benchmarks_v0_43.py` ships `NEW_BENCHMARKS_V0_43 = frozenset({"ceval","cmmlu","aider_polyglot"})` + `MappingProxyType`-wrapped metadata; live Aider Polyglot runner deferred to v0.43.1. `utils/profiling_v0_43.py` `resolve_snapshot_path` rejects null-byte / `.` / `..` / path separators in `run_id`; `base_dir` rejects null-byte / absolute / `..` segments before realpath join (security review fix HIGH #1) — closes a Windows short-name path escape. `memory_snapshot_context` narrows the `RuntimeError` catch to the `record()` entry call only (review fix HIGH #2 — wide catch would have triggered `generator already executing` on user-body RuntimeError); yields `None` when torch / CUDA / `_record_memory_history` API missing. `nccl_bandwidth_check` rejects bool / non-finite / negative `measured_gb_per_sec`; reference table is `MappingProxyType`-wrapped. `utils/vscode_setup.py` `build_launch_json` rejects null-byte / newline / >512-char `config_path` so a crafted argument cannot inject Python args into the generated JSON. `write_vscode_launch` uses `os.lstat + S_ISLNK` symlink rejection at the target path regardless of `force` (security review fix MEDIUM #3 — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 policy). `utils/demo_bundles.py` `copy_bundle_to` validates every JSONL line via `json.loads`, enforces 50 MB cap, stages writes to `.tmp` with `os.lstat + S_ISLNK` rejection (security review fix HIGH #2) and `os.replace`-atomic rename on success — mid-stream rejection never leaves a partial file. New `soup data demo` Typer command: `--output` containment-checked via shared `is_under_cwd`. CLI output prints `_esc(written)` to defend against Rich markup injection through a crafted output path (review fix LOW). `_lcs_length` algorithmic correctness fix (review fix HIGH #2) + BLEU geometric-mean policy fix (returns 0 on any zero precision unless `smooth=True`) (review fix HIGH #1). All `math` imports moved to module-level (PEP 8 / project lazy-import policy: only torch / transformers / peft / trl / mlx are lazy). Known limitations: (1) MLflow / SwanLab / Trackio live integration deferred to v0.43.1 — schema accepts the names and threads into HF `TrainingArguments.report_to` but Soup does not bundle the upstream packages; HF Trainer raises a generic ImportError when the package is absent. (2) PostHog telemetry deferred — payload schema + `SOUP_TELEMETRY=1` env var ship, but no network code; `build_telemetry_payload` is purely a schema-locking stub (mirrors v0.27.0 MII / v0.37.0 multipack / v0.42.0 fsspec stub-then-live pattern). (3) Aider Polyglot live runner deferred — `aider_polyglot` is in the benchmark allowlist but live wiring requires the upstream `aider-chat` package + Docker. (4) `soup doctor --nccl` measurement CLI surface deferred — `nccl_bandwidth_check` accepts a measured value and classifies it, but does not yet measure via `torch.distributed.all_reduce`. (5) `examples/data/` fixtures shipped at the repo root (not as package data inside `soup_cli/`); `_bundle_source_path` derives `repo_root` via `str(importlib.resources.files("soup_cli"))` which is editable-install + wheel friendly but may break in zipapp / namespace-package installs. + - **v0.42.0 — Data Pipeline Pro**: closes the data-tooling gap with Axolotl + LlamaFactory in 18 features across 6 Parts. New `soup_cli/utils/data_pipeline.py` ships an immutable `_REMOTE_SCHEMES` `MappingProxyType` allowlist (s3 / gs / gcs / az / abfs / abfss / oci); `validate_remote_uri` rejects userinfo, fragments, AND query strings (the latter is SSRF-adjacent — fsspec backends interpret `?endpoint_url=…` as configuration overrides; a crafted `s3://bucket/x?endpoint_url=https://attacker.com` would bypass otherwise-trusted infrastructure). Bucket regex `^[a-zA-Z0-9][a-zA-Z0-9._\-]{0,62}$` (1-63 chars, leading alnum) matches RFC 3986 + S3/GCS naming. Path length cap 2048; null-byte rejection on every string-shaped input; bool rejected before int isinstance check on every numeric input (`buffer_size` / `shards` / `image_pixels` / `video_maxlen` / `num_datasets`). `parse_interleave` returns frozen `InterleaveSpec` dataclass; per-prob `math.isfinite` (rejects NaN AND ±inf, matches v0.32.0 / v0.41.0 policy); sum-to-1 ±1e-6; max 32 datasets. `validate_new_tokens` caps lists at 10_000 entries with no duplicates, per-token <= 256 chars, null-byte rejected; returns a defensive copy. `validate_prompt_strategy` enforces a `module.path:function_name` regex with per-side 128 char cap (total <= 260). `make_preprocess_cache_key` rejects bool / null-byte / non-string / non-positive on every parameter; uses `\x1f` unit-separator delimiter. `detect_ingest_format` rejects null-byte / empty / unsupported extensions. New schema validators on `DataConfig`: `video_dir` / `tokenized_path` use shared `is_under_cwd` containment so a crafted YAML like `data.video_dir: ../../etc` fails at config load (review-fix M1; distinct from `image_dir` / `audio_dir` legacy policy). `train_on_prompt` mutually exclusive with `train_on_responses_only` (matches v0.36.0 loss-mask exclusivity policy). `interleave` field has its own `field_validator` so `data.interleave: 99` fails at config load (review-fix L3) AND bare `interleave: probs` is rejected at schema load with the actionable "use {strategy: probs, probs: [...]} dict form" message instead of getting a confusing late error (review-fix HIGH #2). Image-pixel validator threads `info.field_name` so error messages name the actual field that tripped (review-fix M3). Bucket regex relaxed from `{1,62}` to `{0,62}` so 1-character bucket names (per S3/GCS spec) are no longer falsely rejected (review-fix HIGH #1). `_convert_video` validates `video` is non-empty string with null-byte rejection and 2048-char cap (review-fix H2). `_convert_prm` type-checks `prompt` (non-empty string) + every `completions[i]` (string) + every `labels[i]` (bool, not int — matches v0.30.0 `Candidate` policy) and caps at 10_000 steps. `_convert_multimodal` validates content-part `type` against the closed allowlist `{text, image, audio, video}`. New `soup data preprocess ` CLI: `--config` containment-checked via `is_under_cwd` BEFORE `load_config` (review-fix HIGH #1) so a crafted argument like `../../etc/passwd.yaml` cannot be opened. New `soup data ingest ` CLI: `os.lstat + stat.S_ISLNK` symlink rejection on the input file (TOCTOU defence, mirrors v0.33.0 #22 `prune_checkpoints` policy); `is_under_cwd` containment on input + output. Known limitations: (1) fsspec live loaders deferred to v0.42.1 (schema gate fires now, runtime ImportError advisory); (2) AOT preprocess live tokenize loop deferred to v0.42.1 (cache-key emit + path-plan only); (3) custom prompt-strategy runtime invocation deferred to v0.42.1 (regex shape validation only); (4) PRM live trainer wiring deferred to v0.50 per original roadmap; (5) markdown ingest does not split on headings (the docstring's heading-split claim was aspirational); (6) `_convert_video` / `_convert_multimodal` validate path/url null-byte and length but do NOT verify file existence (matches v0.40.5 `reward_model` policy — file-existence is the trainer's responsibility). - **v0.41.0 — Optimizer & PEFT Zoo**: closes the optimizer-breadth gap with LlamaFactory + Axolotl. New `soup_cli/utils/optimizer_zoo.py` ships a closed `SUPPORTED_OPTIMIZERS` `frozenset` (HF-native + bnb-backed + 14 v0.41.0 additions: BAdam / APOLLO / Adam-mini / lomo / adalomo / grokadamw / schedule_free_adamw / schedule_free_sgd / muon / dion / came_pytorch / ao_adamw_{fp8,4bit,8bit}). `validate_optimizer_name` rejects non-string / empty / null-byte / >64-char inputs; lower-cases the name for deterministic lookup (matches v0.30.0 `pick_draft_model` policy). `_OPTIMIZER_PACKAGES` wrapped in `types.MappingProxyType` so the registry cannot be mutated at runtime (matches v0.36.0 `_REGISTRY` policy). `is_new_v0_41_optimizer` is non-string-safe (returns False rather than raising). New `soup_cli/utils/lr_groups.py` parses `training.lr_groups` accepting list-of-pairs / list-of-dicts / `{pattern: lr}` mapping; capped at `MAX_LR_GROUPS=32`; per-pattern non-empty string ≤256 chars + null-byte rejection + `re.compile` validation + best-effort ReDoS probe (`compiled.search("a"*128)` catches catastrophic-backtracking patterns); per-LR `(0.0, 1.0]` bounds + `math.isfinite` (rejects NaN AND `±inf`) + bool rejection (matches v0.30.0 `Candidate` policy). Duplicate patterns rejected. `LrGroup` is `@dataclass(frozen=True)` (matches v0.32.0 `SpikeRecoveryStrategy` policy). `lr_groups_from_schema` converts the canonical stored shape `List[Dict]` into runtime `List[LrGroup]` (closes the schema-to-runtime type gap that the code review caught). `build_optimizer_param_groups` rejects bool / non-positive `base_lr` at runtime (defence-in-depth). New `soup_cli/utils/loftq_init.py` exposes `validate_loftq_iter` (∈ [1, 10], bool rejected) and `validate_loftq_bits` (∈ {2, 4, 8}, bool rejected) with `build_loftq_config` lazy-importing `peft.LoftQConfig` (ImportError carries actionable `pip install --upgrade peft` hint). New `soup_cli/utils/block_expansion.py` validators reject bool on `expand_layers` / `freeze_trainable_layers` and use `field_validator(mode="before")` so Pydantic's `ge`/`le` does not silently coerce `True` to `1` (matches v0.30.0 / v0.34.0 / v0.36.0 / v0.40.6 bool-as-int hardening policy). `_count_layers` uses `hasattr(layers, "__len__")` instead of `try/except TypeError` so legitimate `__len__` bugs surface loudly. Schema `LoraConfig.init_strategy` Literal extended to `{"random", "pissa", "olora", "loftq"}`; cross-validator rejects loftq + use_dora / use_vera. `TrainingConfig.load_in_8bit` / `load_in_16bit` use `is True` policy (matches v0.34.0 / v0.39.0 / v0.40.6 `is None` over falsy guards) — explicit `False` is treated as "no preference" not as "off"; mutually-exclusive both-True rejected; combining alias=True with explicit Quant Menu format raises rather than silently overriding. The alias-driven `quantization` rewrite uses direct `self.quantization = ...` assignment (NOT `object.__setattr__` — code review caught that the latter would silently bypass any future field validator on `quantization`). `expand_model_blocks` raises `NotImplementedError` with a v0.41.1 marker on non-zero block counts (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA stub-then-live pattern). Known limitations: (1) LLaMA Pro live wiring deferred to v0.41.1; (2) Mixture-of-Depths (`use_mod=True`) live patch deferred to v0.41.1; (3) optimizer dependency check is advisory at trainer construction time, not at schema-load (CI environments often lack optional optimizer packages); (4) `load_in_8bit`/`load_in_16bit` rewrite `quantization` only when set to `True` — explicit `False` is intentionally a no-op so a YAML with `load_in_8bit: false` and `quantization: 4bit` still trains in 4-bit. - **v0.40.6 — ReLoRA + surgical PEFT non-SFT**: closes the v0.39.0 known gap by extending the ReLoRA callback (v0.39.0 Part B) and the surgical PEFT patches (v0.39.0 Part D — Gemma4 `ClippableLinear` -> `nn.Linear` swap, fused-MoE 3-D expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO). New shared module `soup_cli/utils/peft_wiring.py` exposes three helpers: `apply_pre_lora_patches(model, base)` (Gemma4-gated, runs BEFORE `get_peft_model` so PEFT's `target_modules` matcher sees the swapped `nn.Linear`), `apply_post_lora_patches(model)` (3-D MoE expert dropout strip, runs AFTER LoRA injection — architecture-detected via `weight.ndim == 3` inside the helper, safe to call unconditionally), `attach_relora_callback(trainer, tcfg)` (returns `True/False`; uses `if relora_steps is None: return False` per project policy so a schema-bypassing caller passing `relora_steps=0` surfaces as a loud `ReLoRAPolicy` ValueError rather than a silent skip). SFT migrates to the same helpers in the same release (centralisation invariant): every trainer file calls only the helpers, eliminating the v0.39.0 inline copy in `sft.py`. `SoupConfig._validate_relora_supported_tasks` removes the `task != "sft"` rejection branch; MLX backend rejection retained with a distinct error message (callback is HF Trainer-specific). Source-level grep matrix in `tests/test_v0406_part_a.py` proves all 12 transformer-backend trainers (sft + 11 non-SFT) call `apply_pre_lora_patches` BEFORE `get_peft_model` BEFORE `apply_post_lora_patches`, plus behavioural unit tests for each helper (Gemma4 happy path + exception swallow, post-LoRA strip happy path + exception swallow, ReLoRA policy field forwarding, schema-gate matrix covering every transformer task plus the `preference` dispatcher with `preference_loss='dpo'`). Defence-in-depth carry-over: `peft_wiring` swallows broad `Exception` from each upstream patch at DEBUG level (matches v0.39.0 Part D best-effort design); `%s` formatting on `exc` (not `repr`) so `$HOME`-prefixed paths cannot leak (matches v0.34.0 `crash.py` redaction policy); the underlying `apply_gemma4_clippable_patch` and `strip_lora_dropout_for_3d_experts` already validate model_name (null bytes, length) and are duck-typed via v0.39.0 review fixes. Known limitations: (1) Multi-modal trainers (vision/audio paths in `sft.py`) inherit ReLoRA + surgical patches because they share the SFT trainer wrapper, but the surgical patches are best-effort (try/except DEBUG-logged) — a Gemma4 vision model is unlikely in practice; if encountered, the patch attempt may noisy-log without applying. (2) The schema gate now accepts every transformer-backend task with `relora_steps`, but real-world correctness on RLHF (PPO / RewardModel) is unverified — ReLoRA was originally validated on SFT/causal-LM training; rejection-sampling-style RL loops may interact unexpectedly with periodic LoRA pruning + optimizer reset. Tracked as a community QA item; the schema does not gate on this since the upstream paper does not preclude RL use. (3) `apply_post_lora_patches` swallows exceptions at DEBUG, consistent with the v0.39.0 best-effort design — a real PEFT-side breakage in `strip_lora_dropout_for_3d_experts` would silently no-op on a non-MoE model where the strip is also a no-op, so the silent fallback is acceptable. diff --git a/pyproject.toml b/pyproject.toml index d6688ce..6933363 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.42.0" +version = "0.43.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index f4c32df..3d722fd 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.42.0" +__version__ = "0.43.0" diff --git a/soup_cli/commands/data.py b/soup_cli/commands/data.py index d5a842f..f340ccb 100644 --- a/soup_cli/commands/data.py +++ b/soup_cli/commands/data.py @@ -2087,3 +2087,65 @@ def ingest_document( for row in rows: f.write(json.dumps(row, ensure_ascii=False) + "\n") console.print(f"[green]Wrote {len(rows)} rows to[/] {output}") + + +@app.command(name="demo") +def demo_bundle( + name: str = typer.Argument( + None, + help="Bundle name (alpaca_demo / sharegpt_demo / dpo_demo / grpo_demo). " + "Omit to list available bundles.", + ), + output: str = typer.Option( + None, + "--output", + "-o", + help="Destination JSONL path (defaults to ./.jsonl).", + ), +) -> None: + """List or copy a bundled demo dataset (v0.43.0). + + `soup data demo` lists available bundles. `soup data demo ` copies + the JSONL fixture to the current directory. Bundles are version-locked + JSONL fixtures shipped under `examples/data/`. + """ + from rich.markup import escape as _esc + from rich.table import Table + + from soup_cli.utils.demo_bundles import ( + copy_bundle_to, + get_bundle, + list_bundles, + ) + + if name is None: + table = Table(title="Available demo bundles", show_lines=False) + table.add_column("Name", style="cyan") + table.add_column("Format", style="green") + table.add_column("Description") + for bundle in list_bundles(): + table.add_row(bundle.name, bundle.format, bundle.description) + console.print(table) + console.print( + "[dim]Run: soup data demo --output [/]" + ) + return + + try: + bundle = get_bundle(name) + except ValueError as exc: + console.print(f"[red]{_esc(str(exc))}[/]") + raise typer.Exit(2) from exc + + target = output or f"./{bundle.name}.jsonl" + try: + written = copy_bundle_to(bundle.name, target) + except FileExistsError as exc: + console.print(f"[red]{_esc(str(exc))}[/]") + raise typer.Exit(1) from exc + except (ValueError, FileNotFoundError) as exc: + console.print(f"[red]{_esc(str(exc))}[/]") + raise typer.Exit(1) from exc + console.print( + f"[green]Copied bundle '{bundle.name}' to[/] {_esc(written)}" + ) diff --git a/soup_cli/commands/train.py b/soup_cli/commands/train.py index f394c44..e56bd86 100644 --- a/soup_cli/commands/train.py +++ b/soup_cli/commands/train.py @@ -55,6 +55,14 @@ def train( "--tensorboard", help="Enable TensorBoard logging (logs to output_dir/runs/)", ), + tracker: str = typer.Option( + None, + "--tracker", + help=( + "Experiment tracker: mlflow / swanlab / trackio (v0.43.0). " + "Mutually exclusive with --wandb / --tensorboard." + ), + ), deepspeed: str = typer.Option( None, "--deepspeed", @@ -640,6 +648,10 @@ def train( dataset = load_dataset(cfg.data) console.print(f"[green]Loaded:[/] {len(dataset['train'])} train samples") + # Capture the --tracker CLI value BEFORE the local ExperimentTracker + # shadows it (v0.43.0 review fix — name-collision regression). + tracker_backend = tracker + # Start experiment tracking from soup_cli.experiment.tracker import ExperimentTracker @@ -655,12 +667,17 @@ def train( console.print(f"[dim]Run ID: {run_id}[/]") # Build trainer based on task type - if wandb: - report_to = "wandb" - elif tensorboard: - report_to = "tensorboard" - else: - report_to = "none" + from soup_cli.utils.trackers import resolve_report_to + + try: + report_to = resolve_report_to( + wandb=wandb, tensorboard=tensorboard, tracker=tracker_backend + ) + except ValueError as exc: + from rich.markup import escape as _esc + + console.print(f"[red]{_esc(str(exc))}[/]") + raise typer.Exit(code=2) from exc console.print("[dim]Setting up model + trainer...[/]") trainer_kwargs = { "device": device, diff --git a/soup_cli/eval/arena.py b/soup_cli/eval/arena.py new file mode 100644 index 0000000..4f400a9 --- /dev/null +++ b/soup_cli/eval/arena.py @@ -0,0 +1,182 @@ +"""v0.43.0 Part B — Model Arena: A/B tournament with Elo ratings. + +Pure-math Elo kernel + tournament aggregator. Generation + judging is the +caller's responsibility — feed `record_match_result` with model A / model B +identifiers and a winner string, get back updated ratings. + +Mirrors the existing `eval/human.py` Elo policy (K=32, base 1500) but +operates on a closed model registry so a single `Tournament` can host many +models and produce a leaderboard. +""" +from __future__ import annotations + +import math +from dataclasses import dataclass, field +from types import MappingProxyType +from typing import Mapping + +DEFAULT_K = 32.0 +DEFAULT_BASE_RATING = 1500.0 +_MAX_NAME_LEN = 128 +_MAX_MODELS = 256 +_MAX_MATCHES = 1_000_000 + + +def _validate_model_name(name: object) -> str: + if not isinstance(name, str): + raise ValueError("model name must be a string") + if not name: + raise ValueError("model name must not be empty") + if "\x00" in name: + raise ValueError("model name must not contain null bytes") + if len(name) > _MAX_NAME_LEN: + raise ValueError( + f"model name length {len(name)} exceeds max {_MAX_NAME_LEN}" + ) + # Reject Rich markup metacharacters at the source so any downstream + # CLI consumer that embeds the leaderboard `model` field in markup + # cannot be markup-injected (security review fix). + if "[" in name or "]" in name: + raise ValueError( + "model name must not contain Rich markup metacharacters '[' or ']'" + ) + return name + + +def expected_score(rating_a: float, rating_b: float) -> float: + """Probability that A beats B given Elo ratings.""" + for r in (rating_a, rating_b): + if isinstance(r, bool) or not isinstance(r, (int, float)): + raise ValueError("ratings must be int/float") + if not math.isfinite(float(r)): + raise ValueError("ratings must be finite") + return 1.0 / (1.0 + math.pow(10.0, (rating_b - rating_a) / 400.0)) + + +def update_elo( + rating_a: float, + rating_b: float, + *, + score_a: float, + k: float = DEFAULT_K, +) -> tuple[float, float]: + """Update Elo ratings after a match where A scored `score_a` ∈ [0,1]. + + score_a == 1.0 → A won. + score_a == 0.0 → B won. + score_a == 0.5 → draw. + """ + if isinstance(score_a, bool) or not isinstance(score_a, (int, float)): + raise ValueError("score_a must be int/float") + if not math.isfinite(float(score_a)) or score_a < 0.0 or score_a > 1.0: + raise ValueError("score_a must be in [0, 1]") + if isinstance(k, bool) or not isinstance(k, (int, float)): + raise ValueError("k must be a positive number") + if not math.isfinite(float(k)) or k <= 0: + raise ValueError("k must be a positive finite number") + + expected_a = expected_score(rating_a, rating_b) + new_a = rating_a + k * (score_a - expected_a) + new_b = rating_b + k * ((1.0 - score_a) - (1.0 - expected_a)) + return new_a, new_b + + +@dataclass +class Tournament: + """Live tournament tracker with Elo + win/loss bookkeeping.""" + + base_rating: float = DEFAULT_BASE_RATING + k: float = DEFAULT_K + _ratings: dict[str, float] = field(default_factory=dict) + _wins: dict[str, int] = field(default_factory=dict) + _losses: dict[str, int] = field(default_factory=dict) + _draws: dict[str, int] = field(default_factory=dict) + _matches: int = 0 + + def __post_init__(self) -> None: + if ( + isinstance(self.base_rating, bool) + or not isinstance(self.base_rating, (int, float)) + or not math.isfinite(float(self.base_rating)) + ): + raise ValueError("base_rating must be a finite number") + if isinstance(self.k, bool) or not isinstance(self.k, (int, float)) or self.k <= 0: + raise ValueError("k must be positive") + + def register(self, name: str) -> None: + """Register a model with the base rating.""" + canonical = _validate_model_name(name) + if canonical in self._ratings: + return + if len(self._ratings) >= _MAX_MODELS: + raise ValueError( + f"tournament has reached the model cap ({_MAX_MODELS})" + ) + self._ratings[canonical] = float(self.base_rating) + self._wins[canonical] = 0 + self._losses[canonical] = 0 + self._draws[canonical] = 0 + + def record( + self, + model_a: str, + model_b: str, + *, + winner: str, + ) -> tuple[float, float]: + """Record a match. `winner` ∈ {"a", "b", "draw"}. + + Returns the new (rating_a, rating_b). + """ + if self._matches >= _MAX_MATCHES: + raise ValueError( + f"tournament has reached the match cap ({_MAX_MATCHES})" + ) + a = _validate_model_name(model_a) + b = _validate_model_name(model_b) + if a == b: + raise ValueError("model_a and model_b must differ") + if not isinstance(winner, str): + raise ValueError("winner must be a string") + winner_norm = winner.lower() + if winner_norm not in {"a", "b", "draw"}: + raise ValueError("winner must be one of 'a' / 'b' / 'draw'") + self.register(a) + self.register(b) + score_a = {"a": 1.0, "b": 0.0, "draw": 0.5}[winner_norm] + new_a, new_b = update_elo( + self._ratings[a], self._ratings[b], score_a=score_a, k=self.k + ) + self._ratings[a] = new_a + self._ratings[b] = new_b + if winner_norm == "a": + self._wins[a] += 1 + self._losses[b] += 1 + elif winner_norm == "b": + self._wins[b] += 1 + self._losses[a] += 1 + else: + self._draws[a] += 1 + self._draws[b] += 1 + self._matches += 1 + return new_a, new_b + + @property + def ratings(self) -> Mapping[str, float]: + return MappingProxyType(dict(self._ratings)) + + def leaderboard(self) -> list[dict]: + """Sorted leaderboard, highest rating first.""" + rows = [] + for name in self._ratings: + rows.append( + { + "model": name, + "rating": round(self._ratings[name], 2), + "wins": self._wins[name], + "losses": self._losses[name], + "draws": self._draws[name], + } + ) + rows.sort(key=lambda r: r["rating"], reverse=True) + return rows diff --git a/soup_cli/eval/benchmarks_v0_43.py b/soup_cli/eval/benchmarks_v0_43.py new file mode 100644 index 0000000..f20cb3a --- /dev/null +++ b/soup_cli/eval/benchmarks_v0_43.py @@ -0,0 +1,54 @@ +"""v0.43.0 Part B — ceval / cmmlu / Aider Polyglot benchmark allowlist. + +ceval / cmmlu route through the existing lm-evaluation-harness wiring; this +module exposes the validated benchmark name allowlist + Aider prompt-loader +scaffold. Live Aider Polyglot eval requires the upstream `aider-chat` package +and Docker; live wiring is deferred to v0.43.1 — this release ships the +benchmark name allowlist + path containment so YAML configs lock in. +""" +from __future__ import annotations + +from types import MappingProxyType +from typing import Mapping + +# v0.43.0 additive benchmark names. +NEW_BENCHMARKS_V0_43 = frozenset({"ceval", "cmmlu", "aider_polyglot"}) + +# Lightweight per-benchmark metadata (description + lm-eval task name). +_BENCHMARK_META: Mapping[str, Mapping[str, str]] = MappingProxyType({ + "ceval": MappingProxyType({ + "description": "Chinese evaluation suite (52 subjects)", + "lm_eval_task": "ceval-valid", + }), + "cmmlu": MappingProxyType({ + "description": "Chinese MMLU (67 subjects)", + "lm_eval_task": "cmmlu", + }), + "aider_polyglot": MappingProxyType({ + "description": "Aider Polyglot benchmark (multi-language code editing)", + "lm_eval_task": "", # No lm-eval mapping; uses upstream aider-chat + }), +}) + + +def is_v0_43_benchmark(name: object) -> bool: + """True if `name` names a v0.43.0-additive benchmark.""" + if not isinstance(name, str): + return False + return name.lower() in NEW_BENCHMARKS_V0_43 + + +def benchmark_metadata(name: str) -> Mapping[str, str] | None: + """Return read-only metadata for a v0.43.0 benchmark, or None.""" + if not isinstance(name, str): + return None + return _BENCHMARK_META.get(name.lower()) + + +def lm_eval_task_for(name: str) -> str | None: + """Map a benchmark name to its lm-eval-harness task id, or None.""" + meta = benchmark_metadata(name) + if meta is None: + return None + task = meta.get("lm_eval_task") + return task if task else None diff --git a/soup_cli/eval/calibrate.py b/soup_cli/eval/calibrate.py new file mode 100644 index 0000000..70cd513 --- /dev/null +++ b/soup_cli/eval/calibrate.py @@ -0,0 +1,118 @@ +"""v0.43.0 Part B — KL Divergence calibration framework (Unsloth Calibration_v3/v5). + +Compares logits between baseline and quantized models on a small fixed subset +(default: 5-shot MMLU). Pure-math kernel `kl_divergence` operates on numpy +arrays and is safe to call without torch. Live model loading + tokenization +is the caller's responsibility — `run_calibration` accepts pre-computed logit +matrices so the same kernel works for any pair of models the user can load. +""" +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import Sequence + + +@dataclass(frozen=True) +class CalibrationReport: + """Per-prompt KL divergence + corpus mean. + + `delta_status` follows v0.26.0 Part D quant-check policy: + - "OK" : mean_kl < 0.05 + - "MINOR" : 0.05 <= mean_kl < 0.20 + - "MAJOR" : mean_kl >= 0.20 + """ + + mean_kl: float + per_prompt_kl: tuple[float, ...] + delta_status: str + num_prompts: int + + +def _softmax(logits: Sequence[float]) -> list[float]: + if not logits: + raise ValueError("logits must not be empty") + max_l = max(logits) + exps = [math.exp(x - max_l) for x in logits] + s = sum(exps) + if s <= 0: + raise ValueError("softmax denominator non-positive") + return [e / s for e in exps] + + +def kl_divergence(p_logits: Sequence[float], q_logits: Sequence[float]) -> float: + """KL(P || Q) over discrete distributions derived from logits. + + Both inputs must be the same length and contain finite floats. + Returns a non-negative float; uses natural log. + """ + if len(p_logits) != len(q_logits): + raise ValueError( + f"p_logits ({len(p_logits)}) and q_logits " + f"({len(q_logits)}) must have the same length" + ) + for name, logits in (("p_logits", p_logits), ("q_logits", q_logits)): + if not logits: + raise ValueError(f"{name} must not be empty") + for x in logits: + if isinstance(x, bool) or not isinstance(x, (int, float)): + raise ValueError(f"{name} must contain only int/float") + if not math.isfinite(float(x)): + raise ValueError(f"{name} must be finite") + p = _softmax(p_logits) + q = _softmax(q_logits) + kl = 0.0 + for pi, qi in zip(p, q): + if pi <= 0: + continue + # qi could be ~0; floor with epsilon to avoid log(0). + kl += pi * math.log(pi / max(qi, 1e-12)) + # Floating-point can produce tiny negatives; clamp. + return max(0.0, kl) + + +def classify_kl_delta(mean_kl: float) -> str: + """OK / MINOR / MAJOR thresholds (v0.26.0 Part D policy).""" + if isinstance(mean_kl, bool) or not isinstance(mean_kl, (int, float)): + raise ValueError("mean_kl must be a number") + if not math.isfinite(float(mean_kl)) or mean_kl < 0: + raise ValueError("mean_kl must be a finite non-negative number") + if mean_kl < 0.05: + return "OK" + if mean_kl < 0.20: + return "MINOR" + return "MAJOR" + + +def run_calibration( + baseline_logits: Sequence[Sequence[float]], + quantized_logits: Sequence[Sequence[float]], +) -> CalibrationReport: + """Run calibration on aligned baseline + quantized logit pairs. + + Each entry of `*_logits` is the next-token-logit row for one prompt + (shape: (vocab,)). Both must have the same outer length. + """ + base_list = list(baseline_logits) + quant_list = list(quantized_logits) + if len(base_list) != len(quant_list): + raise ValueError( + f"baseline_logits ({len(base_list)}) and quantized_logits " + f"({len(quant_list)}) must have the same length" + ) + if not base_list: + raise ValueError("at least one prompt is required") + if len(base_list) > 10_000: + raise ValueError( + f"too many prompts ({len(base_list)}); cap is 10000" + ) + per_prompt = tuple( + kl_divergence(b, q) for b, q in zip(base_list, quant_list) + ) + mean = sum(per_prompt) / len(per_prompt) + return CalibrationReport( + mean_kl=mean, + per_prompt_kl=per_prompt, + delta_status=classify_kl_delta(mean), + num_prompts=len(per_prompt), + ) diff --git a/soup_cli/utils/demo_bundles.py b/soup_cli/utils/demo_bundles.py new file mode 100644 index 0000000..6d344c2 --- /dev/null +++ b/soup_cli/utils/demo_bundles.py @@ -0,0 +1,178 @@ +"""v0.43.0 Part D — `soup data demo` bundle registry. + +Single source of truth for the small JSONL fixtures bundled under +`examples/data/`. The CLI command resolves a name to a path, copies the +bundle into the user-supplied output (containment-checked), and prints +a short summary. No network access. Path containment via shared +`is_under_cwd` (mirrors v0.42.0 ingest policy). +""" +from __future__ import annotations + +import json +import os +import stat +from dataclasses import dataclass +from importlib.resources import files +from types import MappingProxyType +from typing import Mapping + +from soup_cli.utils.paths import is_under_cwd + +_MAX_NAME_LEN = 32 +_MAX_OUTPUT_BYTES = 50 * 1024 * 1024 # 50 MB defence + + +@dataclass(frozen=True) +class DemoBundle: + name: str + fixture: str # filename under examples/data/ + description: str + format: str # alpaca / sharegpt / dpo / reasoning + + +_BUNDLES: Mapping[str, DemoBundle] = MappingProxyType({ + "alpaca_demo": DemoBundle( + name="alpaca_demo", + fixture="alpaca_tiny.jsonl", + description="20-row Alpaca-style instruction tuning fixture", + format="alpaca", + ), + "sharegpt_demo": DemoBundle( + name="sharegpt_demo", + fixture="chat_preferences.jsonl", + description="ShareGPT-style multi-turn chat fixture", + format="sharegpt", + ), + "dpo_demo": DemoBundle( + name="dpo_demo", + fixture="dpo_sample.jsonl", + description="Preference (prompt/chosen/rejected) DPO fixture", + format="dpo", + ), + "grpo_demo": DemoBundle( + name="grpo_demo", + fixture="reasoning_math.jsonl", + description="Math reasoning fixture for GRPO/RLVR", + format="reasoning", + ), +}) + +DEMO_BUNDLE_NAMES = frozenset(_BUNDLES.keys()) + + +def _validate_name(name: object) -> str: + if not isinstance(name, str): + raise ValueError("bundle name must be a string") + if not name: + raise ValueError("bundle name must not be empty") + if "\x00" in name: + raise ValueError("bundle name must not contain null bytes") + if len(name) > _MAX_NAME_LEN: + raise ValueError(f"bundle name length {len(name)} exceeds max {_MAX_NAME_LEN}") + if name not in _BUNDLES: + supported = ", ".join(sorted(_BUNDLES)) + raise ValueError(f"unknown bundle '{name}'. Supported: {supported}") + return name + + +def list_bundles() -> list[DemoBundle]: + """Return all registered demo bundles, sorted by name.""" + return [_BUNDLES[name] for name in sorted(_BUNDLES)] + + +def get_bundle(name: str) -> DemoBundle: + """Return the bundle metadata for `name`, or raise ValueError.""" + canonical = _validate_name(name) + return _BUNDLES[canonical] + + +def _bundle_source_path(bundle: DemoBundle) -> str: + """Resolve the on-disk path for a bundle's fixture. + + Uses importlib.resources to handle both editable + wheel installs. + """ + # Filenames are baked-in constants (no user input), so direct join + # is safe; we still defensively reject path separators. + if "/" in bundle.fixture or "\\" in bundle.fixture: + raise ValueError( + f"bundle fixture name has separator: {bundle.fixture!r}" + ) + # examples/ lives at the repo root, alongside the soup_cli/ package. + pkg_root = files("soup_cli") + repo_root = os.path.dirname(str(pkg_root)) + candidate = os.path.realpath( + os.path.join(repo_root, "examples", "data", bundle.fixture) + ) + if not os.path.isfile(candidate): + raise FileNotFoundError( + f"bundle fixture missing: {bundle.fixture}" + ) + return candidate + + +def copy_bundle_to(name: str, output_path: str) -> str: + """Copy bundle's JSONL into `output_path`. Containment-checked. + + Returns the absolute path written. Refuses to overwrite existing files + (caller must remove first). Validates that every line of the bundle is + valid JSON to avoid silently shipping a malformed fixture. + """ + bundle = get_bundle(name) + if not isinstance(output_path, str) or not output_path: + raise ValueError("output_path must be a non-empty string") + if "\x00" in output_path: + raise ValueError("output_path must not contain null bytes") + real_out = os.path.realpath(output_path) + if not is_under_cwd(real_out): + raise ValueError("output_path must stay under cwd") + if os.path.exists(real_out): + raise FileExistsError( + f"{output_path} already exists; remove it before re-running" + ) + src = _bundle_source_path(bundle) + os.makedirs(os.path.dirname(real_out) or ".", exist_ok=True) + # Stage writes to a sibling temp file so a mid-stream cap rejection + # never leaves a partial file at the user-visible target path. + tmp_path = real_out + ".tmp" + # TOCTOU defence: reject pre-placed symlinks at the temp path + # (mirrors v0.33.0 #22 / v0.40.2 #51 / v0.42.0 ingest policy). + try: + if stat.S_ISLNK(os.lstat(tmp_path).st_mode): + raise ValueError( + "staging temp path is a symlink; aborting" + ) + except FileNotFoundError: + pass + total = 0 + try: + with open(src, encoding="utf-8") as f_in, open( + tmp_path, "w", encoding="utf-8" + ) as f_out: + for lineno, raw in enumerate(f_in, start=1): + stripped = raw.strip() + if not stripped: + continue + try: + json.loads(stripped) + except json.JSONDecodeError as exc: + raise ValueError( + f"bundle {name} line {lineno} is not valid JSON: {exc}" + ) from exc + total += len(raw.encode("utf-8")) + if total > _MAX_OUTPUT_BYTES: + raise ValueError( + f"bundle {name} exceeds {_MAX_OUTPUT_BYTES} byte cap" + ) + f_out.write(raw) + if not raw.endswith("\n"): + f_out.write("\n") + os.replace(tmp_path, real_out) + except BaseException: + # Best-effort cleanup of the staged temp file on any failure. + try: + if os.path.exists(tmp_path): + os.remove(tmp_path) + except OSError: + pass + raise + return real_out diff --git a/soup_cli/utils/nlg_metrics.py b/soup_cli/utils/nlg_metrics.py new file mode 100644 index 0000000..e044c53 --- /dev/null +++ b/soup_cli/utils/nlg_metrics.py @@ -0,0 +1,259 @@ +"""v0.43.0 Part B — BLEU + ROUGE NLG metrics + effective_tokens_per_second. + +Pure-Python implementations sufficient for unit-level eval. For research-grade +scoring users should still wire in `sacrebleu` / `rouge_score` via lm-eval; this +module provides a self-contained baseline that does not require those packages +so `soup eval custom --metric bleu` / `--metric rouge_l` works on a vanilla +install. The closed metric allowlist `NLG_METRICS` is shared with the schema +field validator on `EvalConfig.nlg_metrics`. +""" +from __future__ import annotations + +import math +import re +from collections import Counter +from typing import Iterable, Sequence + +NLG_METRICS = frozenset({"bleu", "rouge_1", "rouge_2", "rouge_l"}) + +# Bounds matching v0.19.0 custom-eval policy. +_MAX_INPUT_CHARS = 1_000_000 +_MAX_NGRAM = 4 + + +def _tokenize(text: str) -> list[str]: + """Word-level whitespace + punctuation-stripped tokenizer. + + Rejects null bytes / non-string / oversize input. + """ + if not isinstance(text, str): + raise ValueError("text must be a string") + if "\x00" in text: + raise ValueError("text must not contain null bytes") + if len(text) > _MAX_INPUT_CHARS: + raise ValueError( + f"text length {len(text)} exceeds max {_MAX_INPUT_CHARS}" + ) + # Word-piece-style tokenizer: lowercase + non-alphanumeric split. + return [tok for tok in re.findall(r"[A-Za-z0-9]+", text.lower()) if tok] + + +def _ngrams(tokens: Sequence[str], n: int) -> Counter[tuple[str, ...]]: + if n <= 0 or isinstance(n, bool): + raise ValueError("n must be a positive int") + if n > _MAX_NGRAM: + raise ValueError(f"n must be <= {_MAX_NGRAM}") + return Counter(tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)) + + +def bleu_score( + predictions: Iterable[str], + references: Iterable[str], + *, + max_n: int = 4, + smooth: bool = True, +) -> float: + """Corpus-level BLEU (single reference per prediction). + + Returns score in [0.0, 1.0]. Empty corpus returns 0.0. + + `smooth=True` (default) uses Chen & Cherry epsilon smoothing for + *zero-correct* buckets where `total[n] > 0`. It does NOT cover + *empty* buckets where `total[n] == 0` (e.g. predictions shorter than + `max_n` tokens) — those force the score to 0.0 even with smoothing. + """ + if isinstance(max_n, bool) or not isinstance(max_n, int): + raise ValueError("max_n must be an int") + if max_n < 1 or max_n > _MAX_NGRAM: + raise ValueError(f"max_n must be in [1, {_MAX_NGRAM}]") + + pred_list = list(predictions) + ref_list = list(references) + if len(pred_list) != len(ref_list): + raise ValueError( + f"predictions ({len(pred_list)}) and references " + f"({len(ref_list)}) must have the same length" + ) + if not pred_list: + return 0.0 + + pred_lengths = 0 + ref_lengths = 0 + correct = [0] * max_n + total = [0] * max_n + + for pred, ref in zip(pred_list, ref_list): + pred_tokens = _tokenize(pred) + ref_tokens = _tokenize(ref) + pred_lengths += len(pred_tokens) + ref_lengths += len(ref_tokens) + for n in range(1, max_n + 1): + if len(pred_tokens) < n: + continue + pred_ng = _ngrams(pred_tokens, n) + ref_ng = _ngrams(ref_tokens, n) + overlap = sum(min(c, ref_ng[ng]) for ng, c in pred_ng.items()) + correct[n - 1] += overlap + total[n - 1] += sum(pred_ng.values()) + + # Modified n-gram precision with Chen & Cherry smoothing for empty buckets. + precisions: list[float] = [] + for n in range(max_n): + if total[n] == 0: + precisions.append(0.0) + continue + if correct[n] == 0 and smooth: + precisions.append(1.0 / (2.0 ** (n + 1) * total[n])) + else: + precisions.append(correct[n] / total[n]) + + # Standard BLEU: geometric mean over all max_n precisions; any zero + # collapses the score to 0.0 unless smoothing is on. + if any(p == 0.0 for p in precisions): + return 0.0 + geo_mean = math.exp(sum(math.log(p) for p in precisions) / max_n) + + # Brevity penalty. + if pred_lengths == 0: + bp = 0.0 + elif pred_lengths > ref_lengths: + bp = 1.0 + else: + bp = math.exp(1.0 - ref_lengths / pred_lengths) + return float(bp * geo_mean) + + +def _lcs_length(a: Sequence[str], b: Sequence[str]) -> int: + """Length of longest common subsequence — DP, O(len(a) * len(b)).""" + if not a or not b: + return 0 + prev = [0] * (len(b) + 1) + for i in range(1, len(a) + 1): + curr = [0] * (len(b) + 1) + for j in range(1, len(b) + 1): + if a[i - 1] == b[j - 1]: + curr[j] = prev[j - 1] + 1 + else: + curr[j] = max(prev[j], curr[j - 1]) + prev = curr + return prev[len(b)] + + +def rouge_n_score( + predictions: Iterable[str], + references: Iterable[str], + *, + n: int = 1, +) -> float: + """ROUGE-N F1 (corpus average over total pair count). + + Pairs where either side has fewer than `n` tokens contribute 0.0 to + the average — short-sentence corpora are penalised, matching the + `rouge-score` default. + """ + if isinstance(n, bool) or not isinstance(n, int): + raise ValueError("n must be an int") + if n < 1 or n > _MAX_NGRAM: + raise ValueError(f"n must be in [1, {_MAX_NGRAM}]") + pred_list = list(predictions) + ref_list = list(references) + if len(pred_list) != len(ref_list): + raise ValueError( + "predictions and references must have the same length" + ) + if not pred_list: + return 0.0 + + f1_sum = 0.0 + for pred, ref in zip(pred_list, ref_list): + pred_tokens = _tokenize(pred) + ref_tokens = _tokenize(ref) + if len(pred_tokens) < n or len(ref_tokens) < n: + continue + pred_ng = _ngrams(pred_tokens, n) + ref_ng = _ngrams(ref_tokens, n) + overlap = sum(min(c, ref_ng[ng]) for ng, c in pred_ng.items()) + if overlap == 0: + continue + precision = overlap / sum(pred_ng.values()) + recall = overlap / sum(ref_ng.values()) + if precision + recall > 0: + f1_sum += 2 * precision * recall / (precision + recall) + return f1_sum / len(pred_list) + + +def rouge_l_score( + predictions: Iterable[str], + references: Iterable[str], +) -> float: + """ROUGE-L F1 — corpus-average sentence-level LCS.""" + pred_list = list(predictions) + ref_list = list(references) + if len(pred_list) != len(ref_list): + raise ValueError( + "predictions and references must have the same length" + ) + if not pred_list: + return 0.0 + f1_sum = 0.0 + for pred, ref in zip(pred_list, ref_list): + pred_tokens = _tokenize(pred) + ref_tokens = _tokenize(ref) + if not pred_tokens or not ref_tokens: + continue + lcs = _lcs_length(pred_tokens, ref_tokens) + if lcs == 0: + continue + precision = lcs / len(pred_tokens) + recall = lcs / len(ref_tokens) + if precision + recall > 0: + f1_sum += 2 * precision * recall / (precision + recall) + return f1_sum / len(pred_list) + + +def compute_nlg_metric( + metric: str, + predictions: Iterable[str], + references: Iterable[str], +) -> float: + """Dispatch by canonical metric name.""" + if not isinstance(metric, str): + raise ValueError("metric must be a string") + name = metric.lower() + if name not in NLG_METRICS: + supported = ", ".join(sorted(NLG_METRICS)) + raise ValueError(f"unknown nlg metric '{metric}'. Supported: {supported}") + if name == "bleu": + return bleu_score(predictions, references) + if name == "rouge_1": + return rouge_n_score(predictions, references, n=1) + if name == "rouge_2": + return rouge_n_score(predictions, references, n=2) + if name == "rouge_l": + return rouge_l_score(predictions, references) + raise AssertionError("unreachable") # pragma: no cover + + +def effective_tokens_per_second( + *, + unmasked_tokens: int, + wall_clock_seconds: float, +) -> float | None: + """Effective tokens-per-second (LF metric). + + `unmasked_tokens` = total non-padding labels seen during training. + Returns None when wall_clock <= 0 (avoid div-by-zero rather than fabricate). + """ + if isinstance(unmasked_tokens, bool) or not isinstance(unmasked_tokens, int): + raise ValueError("unmasked_tokens must be an int") + if unmasked_tokens < 0: + raise ValueError("unmasked_tokens must be >= 0") + if isinstance(wall_clock_seconds, bool) or not isinstance( + wall_clock_seconds, (int, float) + ): + raise ValueError("wall_clock_seconds must be a number") + if not math.isfinite(float(wall_clock_seconds)): + raise ValueError("wall_clock_seconds must be finite") + if wall_clock_seconds <= 0: + return None + return unmasked_tokens / float(wall_clock_seconds) diff --git a/soup_cli/utils/profiling_v0_43.py b/soup_cli/utils/profiling_v0_43.py new file mode 100644 index 0000000..85ff6ca --- /dev/null +++ b/soup_cli/utils/profiling_v0_43.py @@ -0,0 +1,215 @@ +"""v0.43.0 Part C — Profiling extras. + + - `memory_snapshot_context`: torch.cuda.memory._record_memory_history wrapper + that writes a `.snapshot.pickle` to the profiles dir under cwd. + - `enable_detect_anomaly`: thin context manager around + `torch.autograd.set_detect_anomaly(True)`. + - `nccl_bandwidth_check`: scaffold that returns the expected upper-bound + bandwidth for a (gpu_pair, link) tuple. Live measurement deferred — we + expose the reference table so `soup doctor --nccl` can warn when measured + perf is well below expectation. + +Containment, redaction, and exception-narrowing follow v0.34.0 `crash.py` / +v0.34.0 `profiling.py` policies. +""" +from __future__ import annotations + +import contextlib +import math +import os +from dataclasses import dataclass +from types import MappingProxyType +from typing import Iterator, Mapping + +from soup_cli.utils.paths import is_under_cwd + +# Reference NVLink/PCIe bandwidth ceilings (GB/s, unidirectional). +# Source: NVIDIA published topology specs for H100/A100/V100. +_BANDWIDTH_REFERENCE: Mapping[tuple[str, str], float] = MappingProxyType({ + ("h100", "nvlink"): 450.0, # NVLink 4 (18 links × 25 GB/s) + ("h100", "pcie"): 64.0, # PCIe Gen5 x16 + ("a100", "nvlink"): 300.0, # NVLink 3 (12 links × 25 GB/s) + ("a100", "pcie"): 32.0, # PCIe Gen4 x16 + ("v100", "nvlink"): 150.0, # NVLink 2 (6 links × 25 GB/s) + ("v100", "pcie"): 16.0, # PCIe Gen3 x16 + ("rtx4090", "pcie"): 64.0, + ("rtx3090", "pcie"): 32.0, +}) + + +def _validate_run_id(run_id: object) -> str: + if not isinstance(run_id, str): + raise ValueError("run_id must be a string") + if not run_id: + raise ValueError("run_id must not be empty") + if "\x00" in run_id: + raise ValueError("run_id must not contain null bytes") + if run_id in {".", ".."}: + raise ValueError(f"run_id contains forbidden token '{run_id}'") + for ch in ("/", "\\"): + if ch in run_id: + raise ValueError(f"run_id must not contain path separator '{ch}'") + return run_id + + +def resolve_snapshot_path(run_id: str, *, base_dir: str = "profiles") -> str: + """Return realpath of `//.snapshot.pickle`. + + Rejects values that escape cwd. Mirrors v0.34.0 `resolve_trace_path`. + """ + rid = _validate_run_id(run_id) + if not isinstance(base_dir, str) or not base_dir: + raise ValueError("base_dir must be a non-empty string") + if "\x00" in base_dir: + raise ValueError("base_dir must not contain null bytes") + # Reject path separators and `..` components in base_dir so an absolute + # / parent-traversing path can never sneak through realpath on Windows + # short-name systems (security review fix). + if base_dir in {".", ".."}: + raise ValueError("base_dir must not be '.' or '..'") + if os.path.isabs(base_dir): + raise ValueError("base_dir must be a relative path under cwd") + parts = [p for p in base_dir.replace("\\", "/").split("/") if p] + if any(p == ".." for p in parts): + raise ValueError("base_dir must not contain '..' segments") + target = os.path.realpath(os.path.join(base_dir, f"{rid}.snapshot.pickle")) + if not is_under_cwd(target): + raise ValueError("snapshot path must stay under cwd") + return target + + +@contextlib.contextmanager +def memory_snapshot_context( + run_id: str, + *, + base_dir: str = "profiles", + max_entries: int = 100_000, +) -> Iterator[str | None]: + """Record CUDA memory history; on exit, dump pickle + stop recording. + + Yields the snapshot path on success, or None when torch is missing / + CUDA is unavailable / the recording API is missing. Never raises through + the context exit when torch failures are missing-dep style — those map + to a yielded None. + """ + if isinstance(max_entries, bool) or not isinstance(max_entries, int): + raise ValueError("max_entries must be a positive int") + if max_entries < 1 or max_entries > 10_000_000: + raise ValueError("max_entries must be in [1, 10_000_000]") + + path = resolve_snapshot_path(run_id, base_dir=base_dir) + try: + import torch # type: ignore[import-not-found] + except ImportError: + yield None + return + cuda = getattr(torch, "cuda", None) + if cuda is None or not cuda.is_available(): + yield None + return + record = getattr(getattr(cuda, "memory", None), "_record_memory_history", None) + dump = getattr(getattr(cuda, "memory", None), "_dump_snapshot", None) + if record is None or dump is None: + yield None + return + os.makedirs(os.path.dirname(path), exist_ok=True) + # Narrow the RuntimeError catch to only the entry call so a user-body + # RuntimeError cannot trigger a double-yield in this generator + # (review fix — the previous wide `except RuntimeError` would have + # raised "generator already executing" on user-body failures). + try: + record(max_entries=max_entries) + except RuntimeError: + yield None + return + try: + yield path + finally: + try: + dump(path) + finally: + try: + record(enabled=None) # type: ignore[arg-type] + except (TypeError, ValueError): + pass + + +@contextlib.contextmanager +def detect_anomaly_context() -> Iterator[bool]: + """torch.autograd.set_detect_anomaly(True) wrapper. + + Yields True when activated, False when torch is missing. + """ + try: + import torch # type: ignore[import-not-found] + except ImportError: + yield False + return + set_detect = getattr(getattr(torch, "autograd", None), "set_detect_anomaly", None) + if set_detect is None: + yield False + return + set_detect(True) + try: + yield True + finally: + set_detect(False) + + +@dataclass(frozen=True) +class BandwidthExpectation: + gpu: str + link: str + expected_gb_per_sec: float + + +def expected_bandwidth(gpu: str, link: str) -> float | None: + """Return reference bandwidth (GB/s) for `(gpu, link)`, or None.""" + if not isinstance(gpu, str) or not isinstance(link, str): + return None + return _BANDWIDTH_REFERENCE.get((gpu.lower(), link.lower())) + + +def nccl_bandwidth_check( + *, gpu: str, link: str, measured_gb_per_sec: float +) -> dict: + """Compare measured bandwidth against the reference table. + + Returns a dict with `expected`, `measured`, `ratio`, and `status`. + + - status="OK" : ratio >= 0.80 + - status="MINOR" : 0.50 <= ratio < 0.80 + - status="MAJOR" : ratio < 0.50 (silent degradation likely) + - status="UNKNOWN": no reference entry for (gpu, link) + """ + if ( + isinstance(measured_gb_per_sec, bool) + or not isinstance(measured_gb_per_sec, (int, float)) + ): + raise ValueError("measured_gb_per_sec must be a number") + if not math.isfinite(float(measured_gb_per_sec)): + raise ValueError("measured_gb_per_sec must be finite") + if measured_gb_per_sec < 0: + raise ValueError("measured_gb_per_sec must be >= 0") + + expected = expected_bandwidth(gpu, link) + if expected is None or expected <= 0: + return { + "expected_gb_per_sec": None, + "measured_gb_per_sec": float(measured_gb_per_sec), + "ratio": None, + "status": "UNKNOWN", + } + ratio = float(measured_gb_per_sec) / expected + if ratio >= 0.80: + status = "OK" + elif ratio >= 0.50: + status = "MINOR" + else: + status = "MAJOR" + return { + "expected_gb_per_sec": expected, + "measured_gb_per_sec": float(measured_gb_per_sec), + "ratio": round(ratio, 4), + "status": status, + } diff --git a/soup_cli/utils/trackers.py b/soup_cli/utils/trackers.py new file mode 100644 index 0000000..e459e9a --- /dev/null +++ b/soup_cli/utils/trackers.py @@ -0,0 +1,190 @@ +"""v0.43.0 Part A — Tracker integrations + PostHog telemetry opt-out. + +Closed allowlist of HF Trainer `report_to` backends Soup recognises. +Adds mlflow / swanlab / trackio to the legacy `wandb` / `tensorboard` / `none` +set. Live integrations rely on HF Trainer's built-in callbacks (mlflow, +swanlab) plus the third-party `trackio` callback when installed; Soup only +validates the name and surfaces a friendly error when the backing package +is missing. + +Telemetry: opt-out via `SOUP_TELEMETRY=0` env var. Default is OFF until a +public privacy policy ships — `is_telemetry_enabled` returns False unless +the user explicitly enables it. Hardware-info-only payload schema lives in +`build_telemetry_payload` for documentation/testing; no network calls in +v0.43.0 (PostHog wire-up deferred to v0.43.1). +""" +from __future__ import annotations + +import math +import os +import platform +from types import MappingProxyType +from typing import Mapping + +# Closed allowlist of report_to backends. +_REPORT_TO_BACKENDS: Mapping[str, str | None] = MappingProxyType({ + "none": None, + "wandb": "wandb", + "tensorboard": "tensorboard", + "mlflow": "mlflow", + "swanlab": "swanlab", + "trackio": "trackio", +}) + +SUPPORTED_TRACKERS = frozenset(_REPORT_TO_BACKENDS.keys()) + +# v0.43.0 additions (HF-native wandb/tensorboard already supported). +NEW_TRACKERS_V0_43 = frozenset({"mlflow", "swanlab", "trackio"}) + +_MAX_NAME_LEN = 32 + + +def validate_tracker_name(name: object) -> str: + """Validate and lowercase a `report_to` tracker name. + + Returns the canonical lower-cased name. Raises ValueError on invalid + input. Mirrors v0.41.0 `validate_optimizer_name` policy. + """ + if not isinstance(name, str): + raise ValueError(f"tracker name must be a string, got {type(name).__name__}") + if not name: + raise ValueError("tracker name must not be empty") + if "\x00" in name: + raise ValueError("tracker name must not contain null bytes") + if len(name) > _MAX_NAME_LEN: + raise ValueError( + f"tracker name length {len(name)} exceeds max {_MAX_NAME_LEN}" + ) + canonical = name.lower() + if canonical not in SUPPORTED_TRACKERS: + supported = ", ".join(sorted(SUPPORTED_TRACKERS)) + raise ValueError( + f"unknown tracker '{name}'. Supported: {supported}" + ) + return canonical + + +def required_tracker_package(name: str) -> str | None: + """Return the pip-installable package name for a tracker, or None. + + Non-string input returns None (mirrors `is_new_v0_43_tracker`). + """ + if not isinstance(name, str): + return None + return _REPORT_TO_BACKENDS.get(name.lower()) + + +def is_new_v0_43_tracker(name: object) -> bool: + """True if the name is an additive v0.43.0 tracker, False otherwise.""" + if not isinstance(name, str): + return False + return name.lower() in NEW_TRACKERS_V0_43 + + +# --- Telemetry (opt-out, default OFF) ---------------------------------- + +_TELEMETRY_ENV_VAR = "SOUP_TELEMETRY" + + +def is_telemetry_enabled(env: Mapping[str, str] | None = None) -> bool: + """Telemetry is opt-IN until v0.43.1 ships the network code. + + The roadmap entry calls this opt-out, but until the privacy policy + + PostHog wire-up land we keep it default-OFF so no payload is built + or sent. Users may enable explicitly with `SOUP_TELEMETRY=1`. + """ + source = env if env is not None else os.environ + raw = source.get(_TELEMETRY_ENV_VAR) + if raw is None: + return False + val = raw.strip().lower() + if val in {"1", "true", "yes", "on"}: + return True + return False + + +def build_telemetry_payload( + *, + soup_version: str, + command: str, + duration_seconds: float | int | None = None, +) -> dict: + """Build the hardware-info-only telemetry payload. + + The payload contains NO user data, dataset paths, model names, or + config contents. Documented schema: + + - `soup_version`: caller-supplied + - `command`: top-level CLI command (e.g. `train`, `data ingest`) + - `python`: major.minor only + - `os`: platform.system() + - `arch`: platform.machine() + - `duration_seconds`: optional, finite float / int / None + + Raises ValueError for non-string `command` / `soup_version` and for + non-finite `duration_seconds`. + """ + if not isinstance(soup_version, str) or not soup_version: + raise ValueError("soup_version must be a non-empty string") + if "\x00" in soup_version: + raise ValueError("soup_version must not contain null bytes") + if not isinstance(command, str) or not command: + raise ValueError("command must be a non-empty string") + if "\x00" in command: + raise ValueError("command must not contain null bytes") + if duration_seconds is not None: + # bool is a subclass of int — reject explicitly (project policy) + if isinstance(duration_seconds, bool) or not isinstance( + duration_seconds, (int, float) + ): + raise ValueError("duration_seconds must be int / float / None") + if not math.isfinite(float(duration_seconds)): + raise ValueError("duration_seconds must be finite") + if duration_seconds < 0: + raise ValueError("duration_seconds must be >= 0") + + py = platform.python_version_tuple() + py_major_minor = f"{py[0]}.{py[1]}" + return { + "soup_version": soup_version, + "command": command, + "python": py_major_minor, + "os": platform.system(), + "arch": platform.machine(), + "duration_seconds": ( + float(duration_seconds) if duration_seconds is not None else None + ), + } + + +def resolve_report_to( + *, + wandb: bool = False, + tensorboard: bool = False, + tracker: str | None = None, +) -> str: + """Resolve the HF Trainer `report_to` value from CLI flags + --tracker. + + Mutual-exclusion: only one of (wandb, tensorboard, tracker) may be set. + Empty string / None on `tracker` is treated as unset. + """ + set_count = sum( + 1 + for x in ( + bool(wandb), + bool(tensorboard), + bool(tracker) if isinstance(tracker, str) and tracker else False, + ) + if x + ) + if set_count > 1: + raise ValueError( + "--wandb, --tensorboard, and --tracker are mutually exclusive" + ) + if wandb: + return "wandb" + if tensorboard: + return "tensorboard" + if tracker: + return validate_tracker_name(tracker) + return "none" diff --git a/soup_cli/utils/vscode_setup.py b/soup_cli/utils/vscode_setup.py new file mode 100644 index 0000000..91e40c1 --- /dev/null +++ b/soup_cli/utils/vscode_setup.py @@ -0,0 +1,104 @@ +"""v0.43.0 Part C — `soup doctor --vscode` writer for `.vscode/launch.json`. + +Generates a minimal but useful Python debug config for `soup train` plus a +pytest config. Path containment via shared `is_under_cwd`; refuses to +overwrite an existing launch.json without `force=True` (matches v0.40.2 +register_data policy). +""" +from __future__ import annotations + +import json +import os +import stat + +from soup_cli.utils.paths import is_under_cwd + + +def build_launch_json(*, config_path: str = "soup.yaml") -> dict: + """Build the launch.json contents. + + `config_path` is the YAML config to pass to `soup train --config`. + Validates the path so a crafted argument cannot inject arbitrary + args into the generated JSON. + """ + if not isinstance(config_path, str) or not config_path: + raise ValueError("config_path must be a non-empty string") + if "\x00" in config_path: + raise ValueError("config_path must not contain null bytes") + if "\n" in config_path or "\r" in config_path: + raise ValueError("config_path must not contain newlines") + if len(config_path) > 512: + raise ValueError("config_path too long (>512 chars)") + return { + "version": "0.2.0", + "configurations": [ + { + "name": "soup train", + "type": "python", + "request": "launch", + "module": "soup_cli.cli", + "args": ["train", "--config", config_path], + "console": "integratedTerminal", + "justMyCode": False, + "env": {"PYTHONUTF8": "1"}, + }, + { + "name": "pytest (current file)", + "type": "python", + "request": "launch", + "module": "pytest", + "args": ["${file}", "-v", "--no-cov"], + "console": "integratedTerminal", + "justMyCode": False, + }, + ], + } + + +def write_vscode_launch( + *, + config_path: str = "soup.yaml", + target_dir: str = ".vscode", + force: bool = False, +) -> str: + """Write `//launch.json`. + + Returns the absolute path written. + Raises ValueError on containment violation. + Raises FileExistsError when launch.json already exists and force=False. + """ + if not isinstance(target_dir, str) or not target_dir: + raise ValueError("target_dir must be a non-empty string") + if "\x00" in target_dir: + raise ValueError("target_dir must not contain null bytes") + if not isinstance(force, bool): + raise ValueError("force must be a bool") + + real = os.path.realpath(target_dir) + if not is_under_cwd(real): + raise ValueError("target_dir must stay under cwd") + + payload = build_launch_json(config_path=config_path) + os.makedirs(real, exist_ok=True) + out_path = os.path.join(real, "launch.json") + # TOCTOU defence: reject symlinks at the target path regardless of + # `force` (mirrors v0.33.0 #22 prune_checkpoints policy). Without this + # guard, force=True would follow a pre-placed symlink and overwrite a + # file outside cwd. + try: + st = os.lstat(out_path) + except FileNotFoundError: + st = None + if st is not None: + if stat.S_ISLNK(st.st_mode): + raise ValueError( + "launch.json target is a symlink; aborting" + ) + if not force: + raise FileExistsError( + f"{out_path} already exists; pass force=True to overwrite" + ) + with open(out_path, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, ensure_ascii=False) + f.write("\n") + return out_path diff --git a/tests/test_v0430_part_a.py b/tests/test_v0430_part_a.py new file mode 100644 index 0000000..a665af7 --- /dev/null +++ b/tests/test_v0430_part_a.py @@ -0,0 +1,264 @@ +"""Tests for v0.43.0 Part A — Tracker integrations + PostHog opt-out.""" +from __future__ import annotations + +import pytest + +from soup_cli.utils.trackers import ( + NEW_TRACKERS_V0_43, + SUPPORTED_TRACKERS, + build_telemetry_payload, + is_new_v0_43_tracker, + is_telemetry_enabled, + required_tracker_package, + resolve_report_to, + validate_tracker_name, +) + + +class TestSupportedTrackers: + def test_includes_legacy(self): + assert "wandb" in SUPPORTED_TRACKERS + assert "tensorboard" in SUPPORTED_TRACKERS + assert "none" in SUPPORTED_TRACKERS + + def test_includes_v043_additions(self): + for name in ("mlflow", "swanlab", "trackio"): + assert name in SUPPORTED_TRACKERS + + def test_immutable(self): + # frozenset cannot be mutated + with pytest.raises(AttributeError): + SUPPORTED_TRACKERS.add("evil") # type: ignore[attr-defined] + + +class TestValidateTrackerName: + @pytest.mark.parametrize( + "name", ["mlflow", "swanlab", "trackio", "wandb", "tensorboard", "none"] + ) + def test_happy(self, name): + assert validate_tracker_name(name) == name + + def test_case_insensitive(self): + assert validate_tracker_name("MLflow") == "mlflow" + assert validate_tracker_name("SWANLAB") == "swanlab" + + def test_unknown(self): + with pytest.raises(ValueError, match="unknown tracker"): + validate_tracker_name("comet") + + def test_non_string(self): + with pytest.raises(ValueError, match="must be a string"): + validate_tracker_name(123) # type: ignore[arg-type] + + def test_empty(self): + with pytest.raises(ValueError, match="must not be empty"): + validate_tracker_name("") + + def test_null_byte(self): + with pytest.raises(ValueError, match="null"): + validate_tracker_name("mlflow\x00") + + def test_oversize(self): + with pytest.raises(ValueError, match="exceeds max"): + validate_tracker_name("a" * 64) + + +class TestRequiredTrackerPackage: + def test_known_packages(self): + assert required_tracker_package("mlflow") == "mlflow" + assert required_tracker_package("swanlab") == "swanlab" + assert required_tracker_package("trackio") == "trackio" + assert required_tracker_package("wandb") == "wandb" + + def test_none_means_no_install(self): + assert required_tracker_package("none") is None + + def test_unknown_returns_none(self): + assert required_tracker_package("comet") is None + + def test_non_string_returns_none(self): + assert required_tracker_package(None) is None # type: ignore[arg-type] + assert required_tracker_package(123) is None # type: ignore[arg-type] + + +class TestIsNewV043Tracker: + @pytest.mark.parametrize("name", list(NEW_TRACKERS_V0_43)) + def test_true_for_v043(self, name): + assert is_new_v0_43_tracker(name) is True + + @pytest.mark.parametrize("name", ["wandb", "tensorboard", "none"]) + def test_false_for_legacy(self, name): + assert is_new_v0_43_tracker(name) is False + + def test_false_for_unknown(self): + assert is_new_v0_43_tracker("comet") is False + + def test_non_string_safe(self): + assert is_new_v0_43_tracker(None) is False # type: ignore[arg-type] + assert is_new_v0_43_tracker(123) is False # type: ignore[arg-type] + + def test_case_insensitive(self): + assert is_new_v0_43_tracker("MLflow") is True + + +class TestIsTelemetryEnabled: + def test_default_off(self): + assert is_telemetry_enabled({}) is False + + @pytest.mark.parametrize("val", ["1", "true", "yes", "on", "TRUE", "Yes "]) + def test_explicit_on(self, val): + assert is_telemetry_enabled({"SOUP_TELEMETRY": val}) is True + + @pytest.mark.parametrize("val", ["0", "false", "no", "off", "", "garbage"]) + def test_explicit_off_or_garbage(self, val): + assert is_telemetry_enabled({"SOUP_TELEMETRY": val}) is False + + +class TestBuildTelemetryPayload: + def test_required_fields(self): + payload = build_telemetry_payload( + soup_version="0.43.0", command="train" + ) + assert payload["soup_version"] == "0.43.0" + assert payload["command"] == "train" + assert "python" in payload + assert "os" in payload + assert "arch" in payload + assert payload["duration_seconds"] is None + + def test_with_duration(self): + payload = build_telemetry_payload( + soup_version="0.43.0", command="train", duration_seconds=12.5 + ) + assert payload["duration_seconds"] == 12.5 + + def test_int_duration_coerced_to_float(self): + payload = build_telemetry_payload( + soup_version="0.43.0", command="train", duration_seconds=10 + ) + assert payload["duration_seconds"] == 10.0 + assert isinstance(payload["duration_seconds"], float) + + def test_no_user_data_in_payload(self): + # Schema invariant: no model name / dataset path / user identifier. + payload = build_telemetry_payload( + soup_version="0.43.0", command="train" + ) + # Closed key set: + assert set(payload.keys()) == { + "soup_version", + "command", + "python", + "os", + "arch", + "duration_seconds", + } + + def test_invalid_version(self): + with pytest.raises(ValueError): + build_telemetry_payload(soup_version="", command="train") + with pytest.raises(ValueError): + build_telemetry_payload(soup_version=None, command="train") # type: ignore[arg-type] + + def test_null_byte_version(self): + with pytest.raises(ValueError, match="null"): + build_telemetry_payload(soup_version="1.0\x00", command="train") + + def test_invalid_command(self): + with pytest.raises(ValueError): + build_telemetry_payload(soup_version="0.43.0", command="") + + def test_null_byte_command(self): + with pytest.raises(ValueError, match="null"): + build_telemetry_payload(soup_version="0.43.0", command="train\x00") + + def test_bool_duration_rejected(self): + with pytest.raises(ValueError): + build_telemetry_payload( + soup_version="0.43.0", command="train", duration_seconds=True # type: ignore[arg-type] + ) + + def test_nonfinite_duration_rejected(self): + with pytest.raises(ValueError, match="finite"): + build_telemetry_payload( + soup_version="0.43.0", command="train", duration_seconds=float("inf") + ) + with pytest.raises(ValueError, match="finite"): + build_telemetry_payload( + soup_version="0.43.0", + command="train", + duration_seconds=float("nan"), + ) + + def test_negative_duration_rejected(self): + with pytest.raises(ValueError, match=">= 0"): + build_telemetry_payload( + soup_version="0.43.0", command="train", duration_seconds=-1 + ) + + def test_python_is_major_minor_only(self): + payload = build_telemetry_payload( + soup_version="0.43.0", command="train" + ) + # Should be like "3.11" not "3.11.7" + assert payload["python"].count(".") == 1 + + +class TestResolveReportTo: + def test_default_none(self): + assert resolve_report_to() == "none" + + def test_wandb(self): + assert resolve_report_to(wandb=True) == "wandb" + + def test_tensorboard(self): + assert resolve_report_to(tensorboard=True) == "tensorboard" + + def test_tracker_mlflow(self): + assert resolve_report_to(tracker="mlflow") == "mlflow" + + def test_tracker_swanlab(self): + assert resolve_report_to(tracker="swanlab") == "swanlab" + + def test_tracker_trackio(self): + assert resolve_report_to(tracker="trackio") == "trackio" + + def test_tracker_unknown(self): + with pytest.raises(ValueError, match="unknown tracker"): + resolve_report_to(tracker="comet") + + def test_mutually_exclusive_wandb_tensorboard(self): + with pytest.raises(ValueError, match="mutually exclusive"): + resolve_report_to(wandb=True, tensorboard=True) + + def test_mutually_exclusive_wandb_tracker(self): + with pytest.raises(ValueError, match="mutually exclusive"): + resolve_report_to(wandb=True, tracker="mlflow") + + def test_mutually_exclusive_tensorboard_tracker(self): + with pytest.raises(ValueError, match="mutually exclusive"): + resolve_report_to(tensorboard=True, tracker="swanlab") + + def test_empty_tracker_treated_as_unset(self): + assert resolve_report_to(tracker="") == "none" + assert resolve_report_to(tracker=None) == "none" + + def test_tracker_canonicalised(self): + assert resolve_report_to(tracker="MLflow") == "mlflow" + + def test_tracker_none_string(self): + # The literal "none" is a valid value of the allowlist. + assert resolve_report_to(tracker="none") == "none" + + +class TestRegistryImmutability: + def test_report_to_backends_immutable(self): + from soup_cli.utils.trackers import _REPORT_TO_BACKENDS + with pytest.raises(TypeError): + _REPORT_TO_BACKENDS["evil"] = "evil" # type: ignore[index] + + def test_telemetry_none_env_uses_os_environ(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + assert is_telemetry_enabled(None) is True + monkeypatch.delenv("SOUP_TELEMETRY", raising=False) + assert is_telemetry_enabled(None) is False diff --git a/tests/test_v0430_part_b.py b/tests/test_v0430_part_b.py new file mode 100644 index 0000000..f60101e --- /dev/null +++ b/tests/test_v0430_part_b.py @@ -0,0 +1,492 @@ +"""Tests for v0.43.0 Part B — Eval metrics.""" +from __future__ import annotations + +import pytest + +from soup_cli.eval.arena import ( + DEFAULT_BASE_RATING, + Tournament, + expected_score, + update_elo, +) +from soup_cli.eval.benchmarks_v0_43 import ( + NEW_BENCHMARKS_V0_43, + benchmark_metadata, + is_v0_43_benchmark, + lm_eval_task_for, +) +from soup_cli.eval.calibrate import ( + CalibrationReport, + classify_kl_delta, + kl_divergence, + run_calibration, +) +from soup_cli.utils.nlg_metrics import ( + NLG_METRICS, + bleu_score, + compute_nlg_metric, + effective_tokens_per_second, + rouge_l_score, + rouge_n_score, +) + +# ----------------- BLEU / ROUGE ----------------- + +class TestBleuScore: + def test_perfect_match(self): + score = bleu_score(["the cat sat on the mat"], ["the cat sat on the mat"]) + assert score == pytest.approx(1.0, abs=1e-6) + + def test_zero_overlap_no_smoothing(self): + # Without smoothing, any zero n-gram precision -> BLEU 0. + score = bleu_score( + ["alpha beta gamma delta"], + ["one two three four"], + smooth=False, + ) + assert score == 0.0 + + def test_zero_overlap_smoothed(self): + # With smoothing (the default), Chen & Cherry assigns small mass + # to each zero bucket, so the score is small but positive. + score = bleu_score(["alpha beta gamma delta"], ["one two three four"]) + assert 0.0 < score < 0.2 + + def test_brevity_penalty_one_when_pred_longer(self): + # Prediction longer than reference: BP = 1.0 (no penalty). + # Score reflects modified precision only. + # pred unigrams the=2,cat=1,sat=1,on=1,the=2,mat=1 (6 total); + # ref the=1,cat=1,sat=1. + # min-clipped overlap = the(min(2,1)) + cat + sat = 3, total=6 → 0.5 + score = bleu_score( + ["the cat sat on the mat"], ["the cat sat"], max_n=1 + ) + assert score == pytest.approx(0.5) + + def test_partial_overlap(self): + # BLEU-2 partial: shorter prediction with all 4 unigram + 3 bigram + # overlaps — guarantees nonzero standard BLEU. + score = bleu_score( + ["the quick brown fox"], + ["the quick brown fox jumped"], + max_n=2, + ) + assert 0.0 < score < 1.0 + + def test_empty_corpus(self): + assert bleu_score([], []) == 0.0 + + def test_length_mismatch(self): + with pytest.raises(ValueError, match="same length"): + bleu_score(["a"], ["a", "b"]) + + def test_invalid_max_n(self): + with pytest.raises(ValueError): + bleu_score(["a"], ["a"], max_n=0) + with pytest.raises(ValueError): + bleu_score(["a"], ["a"], max_n=10) + with pytest.raises(ValueError): + bleu_score(["a"], ["a"], max_n=True) # type: ignore[arg-type] + + def test_null_byte_rejected(self): + with pytest.raises(ValueError, match="null"): + bleu_score(["a\x00"], ["a"]) + + +class TestRougeNScore: + def test_perfect_match(self): + score = rouge_n_score(["alpha beta gamma"], ["alpha beta gamma"]) + assert score == pytest.approx(1.0) + + def test_zero_overlap(self): + score = rouge_n_score(["alpha beta"], ["one two"]) + assert score == 0.0 + + def test_n_2(self): + score = rouge_n_score(["the cat sat"], ["the cat sat"], n=2) + assert score == pytest.approx(1.0) + + def test_invalid_n(self): + with pytest.raises(ValueError): + rouge_n_score(["a"], ["a"], n=0) + with pytest.raises(ValueError): + rouge_n_score(["a"], ["a"], n=True) # type: ignore[arg-type] + + def test_too_short_for_n(self): + # Single-token strings have no bigrams. + score = rouge_n_score(["a"], ["a"], n=2) + assert score == 0.0 + + def test_length_mismatch_message(self): + with pytest.raises(ValueError, match="same length"): + rouge_n_score(["a"], ["a", "b"]) + + +class TestRougeLScore: + def test_perfect_match(self): + score = rouge_l_score(["alpha beta gamma"], ["alpha beta gamma"]) + assert score == pytest.approx(1.0) + + def test_lcs_partial(self): + score = rouge_l_score( + ["the quick brown fox"], + ["a quick brown dog"], + ) + # LCS = "quick brown" (2 tokens) + # P = 2/4, R = 2/4 (4 tokens each), F1 = 0.5 + assert score == pytest.approx(0.5) + + def test_zero_overlap(self): + score = rouge_l_score(["alpha"], ["beta"]) + assert score == 0.0 + + def test_empty(self): + assert rouge_l_score([], []) == 0.0 + + def test_length_mismatch(self): + with pytest.raises(ValueError, match="same length"): + rouge_l_score(["a"], []) + + +class TestComputeNlgMetric: + @pytest.mark.parametrize("metric", list(NLG_METRICS)) + def test_dispatch(self, metric): + score = compute_nlg_metric(metric, ["a b c"], ["a b c"]) + assert 0.0 <= score <= 1.0 + + def test_unknown(self): + with pytest.raises(ValueError, match="unknown nlg metric"): + compute_nlg_metric("meteor", ["a"], ["a"]) + + def test_non_string(self): + with pytest.raises(ValueError): + compute_nlg_metric(None, ["a"], ["a"]) # type: ignore[arg-type] + + def test_case_insensitive(self): + assert compute_nlg_metric("BLEU", ["a"], ["a"]) >= 0.0 + + +# ----------------- effective_tokens_per_second ----------------- + +class TestEffectiveTokensPerSecond: + def test_happy(self): + assert effective_tokens_per_second( + unmasked_tokens=10000, wall_clock_seconds=10.0 + ) == 1000.0 + + def test_zero_wall_clock_returns_none(self): + assert effective_tokens_per_second( + unmasked_tokens=100, wall_clock_seconds=0.0 + ) is None + + def test_negative_wall_clock_returns_none(self): + assert effective_tokens_per_second( + unmasked_tokens=100, wall_clock_seconds=-1.0 + ) is None + + def test_negative_tokens_rejected(self): + with pytest.raises(ValueError): + effective_tokens_per_second( + unmasked_tokens=-1, wall_clock_seconds=1.0 + ) + + def test_bool_tokens_rejected(self): + with pytest.raises(ValueError): + effective_tokens_per_second( + unmasked_tokens=True, wall_clock_seconds=1.0 # type: ignore[arg-type] + ) + + def test_bool_wall_clock_rejected(self): + with pytest.raises(ValueError): + effective_tokens_per_second( + unmasked_tokens=100, wall_clock_seconds=True # type: ignore[arg-type] + ) + + def test_nonfinite_wall_clock_rejected(self): + with pytest.raises(ValueError, match="finite"): + effective_tokens_per_second( + unmasked_tokens=100, wall_clock_seconds=float("inf") + ) + + +# ----------------- KL Calibration ----------------- + +class TestKlDivergence: + def test_identical_distributions(self): + kl = kl_divergence([1.0, 2.0, 3.0], [1.0, 2.0, 3.0]) + assert kl == pytest.approx(0.0, abs=1e-9) + + def test_different_distributions_positive(self): + kl = kl_divergence([1.0, 2.0, 3.0], [3.0, 2.0, 1.0]) + assert kl > 0.0 + + def test_length_mismatch(self): + with pytest.raises(ValueError, match="same length"): + kl_divergence([1.0, 2.0], [1.0]) + + def test_empty(self): + with pytest.raises(ValueError): + kl_divergence([], []) + + def test_non_finite_rejected(self): + with pytest.raises(ValueError, match="finite"): + kl_divergence([1.0, float("inf")], [1.0, 2.0]) + + def test_bool_rejected(self): + with pytest.raises(ValueError): + kl_divergence([True, False], [1.0, 2.0]) # type: ignore[list-item] + + +class TestClassifyKlDelta: + @pytest.mark.parametrize( + "kl,status", + [ + (0.0, "OK"), + (0.04, "OK"), + (0.05, "MINOR"), + (0.10, "MINOR"), + (0.19, "MINOR"), + (0.20, "MAJOR"), + (1.0, "MAJOR"), + ], + ) + def test_thresholds(self, kl, status): + assert classify_kl_delta(kl) == status + + def test_negative_rejected(self): + with pytest.raises(ValueError): + classify_kl_delta(-0.01) + + def test_nan_rejected(self): + with pytest.raises(ValueError): + classify_kl_delta(float("nan")) + + def test_bool_rejected(self): + with pytest.raises(ValueError): + classify_kl_delta(True) # type: ignore[arg-type] + + +class TestRunCalibration: + def test_perfect_match_ok(self): + baseline = [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]] + quant = [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]] + report = run_calibration(baseline, quant) + assert isinstance(report, CalibrationReport) + assert report.mean_kl == pytest.approx(0.0, abs=1e-9) + assert report.delta_status == "OK" + assert report.num_prompts == 2 + + def test_diverged_quant_major(self): + baseline = [[10.0, 0.0, 0.0]] + quant = [[0.0, 0.0, 10.0]] + report = run_calibration(baseline, quant) + # Large divergence — guaranteed MAJOR. + assert report.delta_status == "MAJOR" + + def test_length_mismatch(self): + with pytest.raises(ValueError, match="same length"): + run_calibration([[1.0]], []) + + def test_empty(self): + with pytest.raises(ValueError): + run_calibration([], []) + + def test_too_many_prompts(self): + big = [[1.0]] * 10_001 + with pytest.raises(ValueError, match="too many"): + run_calibration(big, big) + + def test_report_frozen(self): + from dataclasses import FrozenInstanceError + + report = run_calibration([[1.0, 2.0]], [[1.0, 2.0]]) + with pytest.raises(FrozenInstanceError): + report.mean_kl = 0.5 # type: ignore[misc] + + +# ----------------- Arena ----------------- + +class TestExpectedScore: + def test_equal_ratings_50_50(self): + assert expected_score(1500.0, 1500.0) == pytest.approx(0.5) + + def test_higher_rated_favored(self): + assert expected_score(1700.0, 1500.0) > 0.5 + + def test_lower_rated_disfavored(self): + assert expected_score(1300.0, 1500.0) < 0.5 + + def test_non_finite_rejected(self): + with pytest.raises(ValueError): + expected_score(float("nan"), 1500.0) + + def test_bool_rejected(self): + with pytest.raises(ValueError): + expected_score(True, 1500.0) # type: ignore[arg-type] + + +class TestUpdateElo: + def test_winner_gains(self): + new_a, new_b = update_elo(1500.0, 1500.0, score_a=1.0) + assert new_a > 1500.0 + assert new_b < 1500.0 + + def test_draw_no_change_at_equal(self): + new_a, new_b = update_elo(1500.0, 1500.0, score_a=0.5) + assert new_a == pytest.approx(1500.0) + assert new_b == pytest.approx(1500.0) + + def test_zero_sum(self): + new_a, new_b = update_elo(1500.0, 1700.0, score_a=1.0) + # Symmetric Elo: total rating mass conserved. + assert (new_a - 1500.0) == pytest.approx(-(new_b - 1700.0), abs=1e-6) + + def test_invalid_score_a(self): + with pytest.raises(ValueError): + update_elo(1500.0, 1500.0, score_a=-0.1) + with pytest.raises(ValueError): + update_elo(1500.0, 1500.0, score_a=1.1) + with pytest.raises(ValueError): + update_elo(1500.0, 1500.0, score_a=True) # type: ignore[arg-type] + + def test_invalid_k(self): + with pytest.raises(ValueError): + update_elo(1500.0, 1500.0, score_a=1.0, k=0) + with pytest.raises(ValueError): + update_elo(1500.0, 1500.0, score_a=1.0, k=-32) + + +class TestTournament: + def test_register_and_record(self): + t = Tournament() + t.register("alpha") + t.register("beta") + t.record("alpha", "beta", winner="a") + assert t.ratings["alpha"] > DEFAULT_BASE_RATING + assert t.ratings["beta"] < DEFAULT_BASE_RATING + + def test_register_idempotent(self): + t = Tournament() + t.register("alpha") + t.register("alpha") + assert t.ratings["alpha"] == DEFAULT_BASE_RATING + + def test_ratings_immutable_view(self): + t = Tournament() + t.register("alpha") + # Ratings property must not allow mutation through the returned view. + with pytest.raises(TypeError): + t.ratings["alpha"] = 9999.0 # type: ignore[index] + + def test_implicit_register_on_record(self): + t = Tournament() + t.record("alpha", "beta", winner="draw") + assert "alpha" in t.ratings + assert "beta" in t.ratings + + def test_self_play_rejected(self): + t = Tournament() + with pytest.raises(ValueError, match="must differ"): + t.record("alpha", "alpha", winner="a") + + def test_invalid_winner(self): + t = Tournament() + with pytest.raises(ValueError, match="winner"): + t.record("a", "b", winner="c") + + def test_empty_name_rejected(self): + t = Tournament() + with pytest.raises(ValueError): + t.register("") + + def test_null_byte_name_rejected(self): + t = Tournament() + with pytest.raises(ValueError, match="null"): + t.register("foo\x00") + + def test_oversize_name_rejected(self): + t = Tournament() + with pytest.raises(ValueError): + t.register("a" * 256) + + def test_rich_markup_metacharacter_rejected(self): + t = Tournament() + with pytest.raises(ValueError, match="markup"): + t.register("[red]evil[/red]") + with pytest.raises(ValueError, match="markup"): + t.register("foo]bar") + + def test_model_cap_exceeded(self): + from soup_cli.eval.arena import _MAX_MODELS + t = Tournament() + for i in range(_MAX_MODELS): + t.register(f"model_{i}") + with pytest.raises(ValueError, match="model cap"): + t.register("one_too_many") + + def test_invalid_k_nan(self): + with pytest.raises(ValueError): + update_elo(1500.0, 1500.0, score_a=1.0, k=float("nan")) + + def test_invalid_base_rating(self): + with pytest.raises(ValueError): + Tournament(base_rating=float("nan")) + + def test_invalid_k(self): + with pytest.raises(ValueError): + Tournament(k=0) + + def test_leaderboard_sorted(self): + t = Tournament() + t.record("alpha", "beta", winner="a") + t.record("alpha", "gamma", winner="a") + board = t.leaderboard() + assert board[0]["model"] == "alpha" + assert board[0]["wins"] == 2 + assert board[0]["losses"] == 0 + # Highest rating first. + for i in range(len(board) - 1): + assert board[i]["rating"] >= board[i + 1]["rating"] + + def test_draw_records(self): + t = Tournament() + t.record("alpha", "beta", winner="draw") + for name in ("alpha", "beta"): + row = next(r for r in t.leaderboard() if r["model"] == name) + assert row["draws"] == 1 + assert row["wins"] == 0 + assert row["losses"] == 0 + + +# ----------------- Benchmarks v0.43 ----------------- + +class TestBenchmarksV043: + @pytest.mark.parametrize("name", list(NEW_BENCHMARKS_V0_43)) + def test_recognised(self, name): + assert is_v0_43_benchmark(name) is True + meta = benchmark_metadata(name) + assert meta is not None + assert "description" in meta + + def test_case_insensitive(self): + assert is_v0_43_benchmark("CEval") is True + + def test_unknown_returns_false(self): + assert is_v0_43_benchmark("mmlu") is False + assert is_v0_43_benchmark("garbage") is False + + def test_non_string_returns_false(self): + assert is_v0_43_benchmark(None) is False # type: ignore[arg-type] + assert is_v0_43_benchmark(123) is False # type: ignore[arg-type] + + def test_metadata_immutable(self): + meta = benchmark_metadata("ceval") + with pytest.raises(TypeError): + meta["description"] = "evil" # type: ignore[index] + + def test_lm_eval_task(self): + assert lm_eval_task_for("ceval") == "ceval-valid" + assert lm_eval_task_for("cmmlu") == "cmmlu" + # Aider Polyglot has no lm-eval mapping. + assert lm_eval_task_for("aider_polyglot") is None + assert lm_eval_task_for("garbage") is None diff --git a/tests/test_v0430_part_c.py b/tests/test_v0430_part_c.py new file mode 100644 index 0000000..73f0d3f --- /dev/null +++ b/tests/test_v0430_part_c.py @@ -0,0 +1,290 @@ +"""Tests for v0.43.0 Part C — Profiling extras + VSCode setup.""" +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest + +from soup_cli.utils.profiling_v0_43 import ( + BandwidthExpectation, + detect_anomaly_context, + expected_bandwidth, + memory_snapshot_context, + nccl_bandwidth_check, + resolve_snapshot_path, +) +from soup_cli.utils.vscode_setup import build_launch_json, write_vscode_launch + +# ----------------- snapshot path ----------------- + +class TestResolveSnapshotPath: + def test_happy(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + path = resolve_snapshot_path("run-123") + assert path.endswith(os.path.join("profiles", "run-123.snapshot.pickle")) + assert os.path.realpath(str(tmp_path)) in path + + @pytest.mark.parametrize("bad", ["", ".", "..", "a/b", "a\\b", "a\x00b"]) + def test_invalid_run_id(self, bad, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + resolve_snapshot_path(bad) + + def test_non_string_run_id(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="must be a string"): + resolve_snapshot_path(123) # type: ignore[arg-type] + + def test_invalid_base_dir(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + resolve_snapshot_path("run-1", base_dir="") + with pytest.raises(ValueError, match="null"): + resolve_snapshot_path("run-1", base_dir="prof\x00iles") + + def test_absolute_base_dir_rejected(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="relative"): + resolve_snapshot_path("run-1", base_dir=str(tmp_path / "abs")) + + def test_dotdot_in_base_dir_rejected(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match=r"'\.\.'"): + resolve_snapshot_path("run-1", base_dir="../escape") + + def test_dot_base_dir_rejected(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match=r"'\.\.'"): + resolve_snapshot_path("run-1", base_dir="..") + + +# ----------------- memory_snapshot_context ----------------- + +class TestMemorySnapshotContext: + def test_no_torch_yields_none(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + # Force ImportError for torch. + import builtins + + real_import = builtins.__import__ + + def fake(name, *a, **kw): + if name == "torch": + raise ImportError("torch not installed") + return real_import(name, *a, **kw) + + monkeypatch.setattr(builtins, "__import__", fake) + with memory_snapshot_context("run-x") as path: + assert path is None + + def test_invalid_max_entries(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + with memory_snapshot_context("run-x", max_entries=0): + pass + with pytest.raises(ValueError): + with memory_snapshot_context("run-x", max_entries=True): # type: ignore[arg-type] + pass + + def test_invalid_run_id_propagates(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + with memory_snapshot_context(""): + pass + + +class TestDetectAnomalyContext: + def test_no_torch_yields_false(self, monkeypatch): + import builtins + + real_import = builtins.__import__ + + def fake(name, *a, **kw): + if name == "torch": + raise ImportError("torch not installed") + return real_import(name, *a, **kw) + + monkeypatch.setattr(builtins, "__import__", fake) + with detect_anomaly_context() as enabled: + assert enabled is False + + def test_with_real_torch_or_skip(self): + try: + import torch # noqa: F401 + except ImportError: + pytest.skip("torch not installed in this environment") + with detect_anomaly_context() as enabled: + # On any torch with autograd module the context is active. + assert enabled is True + + +# ----------------- nccl_bandwidth_check ----------------- + +class TestExpectedBandwidth: + @pytest.mark.parametrize( + "gpu,link", + [ + ("h100", "nvlink"), + ("h100", "pcie"), + ("a100", "nvlink"), + ("a100", "pcie"), + ("v100", "nvlink"), + ("rtx4090", "pcie"), + ], + ) + def test_known_pairs(self, gpu, link): + bw = expected_bandwidth(gpu, link) + assert bw is not None and bw > 0 + + def test_case_insensitive(self): + assert expected_bandwidth("H100", "NVLINK") == 450.0 + + def test_unknown_returns_none(self): + assert expected_bandwidth("evil", "pcie") is None + assert expected_bandwidth("h100", "carrier-pigeon") is None + + def test_non_string_returns_none(self): + assert expected_bandwidth(None, "nvlink") is None # type: ignore[arg-type] + assert expected_bandwidth(123, 456) is None # type: ignore[arg-type] + + def test_dataclass_frozen(self): + be = BandwidthExpectation(gpu="h100", link="nvlink", expected_gb_per_sec=450.0) + with pytest.raises(Exception): + be.gpu = "a100" # type: ignore[misc] + + +class TestNcclBandwidthCheck: + def test_ok(self): + result = nccl_bandwidth_check( + gpu="h100", link="nvlink", measured_gb_per_sec=400.0 + ) + assert result["status"] == "OK" + assert result["expected_gb_per_sec"] == 450.0 + assert 0.88 <= result["ratio"] <= 0.89 + + def test_minor(self): + result = nccl_bandwidth_check( + gpu="h100", link="nvlink", measured_gb_per_sec=300.0 + ) + assert result["status"] == "MINOR" + + def test_major(self): + result = nccl_bandwidth_check( + gpu="h100", link="nvlink", measured_gb_per_sec=100.0 + ) + assert result["status"] == "MAJOR" + + def test_unknown_pair(self): + result = nccl_bandwidth_check( + gpu="evil", link="pcie", measured_gb_per_sec=5.0 + ) + assert result["status"] == "UNKNOWN" + assert result["expected_gb_per_sec"] is None + assert result["ratio"] is None + + def test_negative_measured_rejected(self): + with pytest.raises(ValueError): + nccl_bandwidth_check( + gpu="h100", link="nvlink", measured_gb_per_sec=-1.0 + ) + + def test_nonfinite_measured_rejected(self): + with pytest.raises(ValueError, match="finite"): + nccl_bandwidth_check( + gpu="h100", + link="nvlink", + measured_gb_per_sec=float("inf"), + ) + + def test_bool_measured_rejected(self): + with pytest.raises(ValueError): + nccl_bandwidth_check( + gpu="h100", link="nvlink", measured_gb_per_sec=True # type: ignore[arg-type] + ) + + +# ----------------- VSCode launch.json ----------------- + +class TestBuildLaunchJson: + def test_default(self): + payload = build_launch_json() + assert payload["version"] == "0.2.0" + configs = payload["configurations"] + assert len(configs) >= 2 + train = next(c for c in configs if c["name"] == "soup train") + assert "soup.yaml" in train["args"] + assert train["module"] == "soup_cli.cli" + + def test_custom_config_path(self): + payload = build_launch_json(config_path="my-cfg.yaml") + train = next(c for c in payload["configurations"] if c["name"] == "soup train") + assert "my-cfg.yaml" in train["args"] + + def test_invalid_config_path(self): + with pytest.raises(ValueError): + build_launch_json(config_path="") + with pytest.raises(ValueError, match="null"): + build_launch_json(config_path="x\x00y.yaml") + with pytest.raises(ValueError, match="newlines"): + build_launch_json(config_path="a\nb.yaml") + with pytest.raises(ValueError): + build_launch_json(config_path="a" * 1024) + + +class TestWriteVscodeLaunch: + def test_writes_file(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + out = write_vscode_launch() + assert Path(out).is_file() + with open(out, encoding="utf-8") as f: + data = json.load(f) + assert data["version"] == "0.2.0" + + def test_refuses_overwrite_without_force(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + write_vscode_launch() + with pytest.raises(FileExistsError): + write_vscode_launch() + + def test_force_overwrites(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + write_vscode_launch() + # Mutate file then re-write with force=True. + path = Path(tmp_path) / ".vscode" / "launch.json" + path.write_text("garbage", encoding="utf-8") + out = write_vscode_launch(force=True) + with open(out, encoding="utf-8") as f: + data = json.load(f) + assert data["version"] == "0.2.0" + + def test_outside_cwd_rejected(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="under cwd"): + write_vscode_launch(target_dir=str(tmp_path.parent / "evil")) + + def test_invalid_target_dir(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + write_vscode_launch(target_dir="") + with pytest.raises(ValueError, match="null"): + write_vscode_launch(target_dir="ev\x00il") + + def test_force_must_be_bool(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + write_vscode_launch(force="yes") # type: ignore[arg-type] + + def test_symlink_target_rejected(self, tmp_path, monkeypatch): + # Skip on Windows without dev-mode where symlinks need privilege. + if os.name == "nt": + pytest.skip("symlink creation needs admin/dev mode on Windows") + monkeypatch.chdir(tmp_path) + vscode_dir = tmp_path / ".vscode" + vscode_dir.mkdir() + outside = tmp_path / "outside.json" + outside.write_text("{}", encoding="utf-8") + os.symlink(str(outside), str(vscode_dir / "launch.json")) + with pytest.raises(ValueError, match="symlink"): + write_vscode_launch(force=True) diff --git a/tests/test_v0430_part_d.py b/tests/test_v0430_part_d.py new file mode 100644 index 0000000..4cf6828 --- /dev/null +++ b/tests/test_v0430_part_d.py @@ -0,0 +1,204 @@ +"""Tests for v0.43.0 Part D — soup data demo bundles.""" +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.commands.data import app as data_app +from soup_cli.utils.demo_bundles import ( + DEMO_BUNDLE_NAMES, + DemoBundle, + copy_bundle_to, + get_bundle, + list_bundles, +) + + +class TestDemoBundleRegistry: + def test_known_names(self): + for name in ("alpaca_demo", "sharegpt_demo", "dpo_demo", "grpo_demo"): + assert name in DEMO_BUNDLE_NAMES + + def test_immutable_set(self): + with pytest.raises(AttributeError): + DEMO_BUNDLE_NAMES.add("evil") # type: ignore[attr-defined] + + def test_list_bundles_sorted(self): + bundles = list_bundles() + names = [b.name for b in bundles] + assert names == sorted(names) + + def test_bundle_dataclass_frozen(self): + b = list_bundles()[0] + assert isinstance(b, DemoBundle) + with pytest.raises(Exception): + b.name = "evil" # type: ignore[misc] + + def test_get_bundle_unknown(self): + with pytest.raises(ValueError, match="unknown bundle"): + get_bundle("garbage") + + def test_get_bundle_empty(self): + with pytest.raises(ValueError, match="not be empty"): + get_bundle("") + + def test_get_bundle_null_byte(self): + with pytest.raises(ValueError, match="null"): + get_bundle("alpaca\x00") + + def test_get_bundle_oversize(self): + with pytest.raises(ValueError, match="exceeds max"): + get_bundle("a" * 64) + + def test_get_bundle_non_string(self): + with pytest.raises(ValueError, match="must be a string"): + get_bundle(123) # type: ignore[arg-type] + + +class TestCopyBundleTo: + def test_writes_jsonl(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + out = copy_bundle_to("alpaca_demo", "./alpaca_demo.jsonl") + assert Path(out).is_file() + # Validate every line parses as JSON. + with open(out, encoding="utf-8") as f: + for line in f: + if line.strip(): + json.loads(line) + + def test_outside_cwd_rejected(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + evil = str(tmp_path.parent / "evil.jsonl") + with pytest.raises(ValueError, match="under cwd"): + copy_bundle_to("alpaca_demo", evil) + + def test_existing_file_rejected(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + target = tmp_path / "out.jsonl" + target.write_text("placeholder", encoding="utf-8") + with pytest.raises(FileExistsError): + copy_bundle_to("alpaca_demo", str(target)) + + def test_null_byte_output_rejected(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + copy_bundle_to("alpaca_demo", "ev\x00il.jsonl") + + def test_non_string_output(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="must be a non-empty string"): + copy_bundle_to("alpaca_demo", "") + + def test_unknown_bundle(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="unknown bundle"): + copy_bundle_to("garbage", "./out.jsonl") + + def test_symlink_at_tmp_path_rejected(self, tmp_path, monkeypatch): + if os.name == "nt": + pytest.skip("symlink creation needs admin/dev mode on Windows") + monkeypatch.chdir(tmp_path) + target = tmp_path / "out.jsonl" + # Pre-place a symlink at .tmp pointing to a sentinel file. + outside = tmp_path / "outside.txt" + outside.write_text("placeholder", encoding="utf-8") + os.symlink(str(outside), str(target) + ".tmp") + with pytest.raises(ValueError, match="symlink"): + copy_bundle_to("alpaca_demo", str(target)) + + @pytest.mark.parametrize("name", sorted(DEMO_BUNDLE_NAMES)) + def test_every_bundle_copyable(self, name, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + out = copy_bundle_to(name, f"./{name}.jsonl") + assert os.path.getsize(out) > 0 + + +class TestDataDemoCli: + def test_list_help(self): + runner = CliRunner() + result = runner.invoke(data_app, ["demo", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "demo bundle" in result.output.lower() or "bundle" in result.output.lower() + + def test_list_no_args(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=str(tmp_path)): + result = runner.invoke(data_app, ["demo"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + # Table renders all 4 bundles. + for name in ("alpaca_demo", "sharegpt_demo", "dpo_demo", "grpo_demo"): + assert name in result.output + + def test_copy_default_output(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=str(tmp_path)): + result = runner.invoke(data_app, ["demo", "alpaca_demo"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert Path("alpaca_demo.jsonl").is_file() + + def test_copy_custom_output(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=str(tmp_path)): + result = runner.invoke( + data_app, ["demo", "dpo_demo", "--output", "./mine.jsonl"] + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert Path("mine.jsonl").is_file() + + def test_unknown_bundle_exits_2(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=str(tmp_path)): + result = runner.invoke(data_app, ["demo", "garbage"]) + assert result.exit_code == 2, (result.output, repr(result.exception)) + assert "unknown bundle" in result.output.lower() + + def test_existing_output_exits_1(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=str(tmp_path)): + Path("alpaca_demo.jsonl").write_text("x", encoding="utf-8") + result = runner.invoke(data_app, ["demo", "alpaca_demo"]) + assert result.exit_code == 1, (result.output, repr(result.exception)) + assert "already exists" in result.output.lower() + + def test_outside_cwd_output_exits_1(self, tmp_path): + runner = CliRunner() + evil = str(tmp_path.parent / "evil.jsonl") + with runner.isolated_filesystem(temp_dir=str(tmp_path)): + result = runner.invoke( + data_app, ["demo", "alpaca_demo", "--output", evil] + ) + assert result.exit_code == 1, (result.output, repr(result.exception)) + assert "under cwd" in result.output.lower() + + +class TestCopyBundleEdgeCases: + def test_size_cap_branch(self, tmp_path, monkeypatch): + from soup_cli.utils import demo_bundles as db + + monkeypatch.chdir(tmp_path) + big_src = tmp_path / "big.jsonl" + line = json.dumps({"x": "y" * 1024}) + "\n" + with open(big_src, "w", encoding="utf-8") as f: + for _ in range(55_000): # ~55 MB + f.write(line) + monkeypatch.setattr(db, "_bundle_source_path", lambda _b: str(big_src)) + with pytest.raises(ValueError, match="byte cap"): + db.copy_bundle_to("alpaca_demo", "./out.jsonl") + # Staged temp file must not survive the rejection. + assert not (tmp_path / "out.jsonl.tmp").exists() + assert not (tmp_path / "out.jsonl").exists() + + def test_invalid_json_line(self, tmp_path, monkeypatch): + from soup_cli.utils import demo_bundles as db + + monkeypatch.chdir(tmp_path) + bad = tmp_path / "bad.jsonl" + bad.write_text('{"ok": 1}\nnot-json\n', encoding="utf-8") + monkeypatch.setattr(db, "_bundle_source_path", lambda _b: str(bad)) + with pytest.raises(ValueError, match="not valid JSON"): + db.copy_bundle_to("alpaca_demo", "./out.jsonl") + assert not (tmp_path / "out.jsonl").exists()