mirror of https://github.com/razor-ai/soup.git
feat(eval): Tracker & Eval Pro — 18 features (v0.43.0)
Closes the observability gap with all three competitors in one release.
Part A — Trackers
* --tracker flag (mlflow/swanlab/trackio) on soup train, mutually
exclusive with --wandb/--tensorboard. Closed allowlist via
MappingProxyType. Live integrations rely on HF Trainer's report_to.
* SOUP_TELEMETRY=1 opt-IN env var; build_telemetry_payload schema is
closed-key (no model names / dataset paths / config contents). Live
PostHog network code deferred to v0.43.1.
Part B — Eval metrics
* Pure-Python BLEU + ROUGE-1/2/L + effective_tokens_per_second.
* KL-divergence calibration framework with OK/MINOR/MAJOR thresholds.
* Model Arena Elo tournament (256-model cap, MappingProxyType view,
Rich-markup metacharacter rejection on names).
* ceval / cmmlu / aider_polyglot benchmark allowlist (live Aider
runner deferred to v0.43.1).
Part C — Profiling
* memory_snapshot_context (narrow RuntimeError catch — review fix
prevents generator-already-executing on user-body RuntimeError).
* detect_anomaly_context, nccl_bandwidth_check (h100/a100/v100/rtx
reference table; live measurement CLI surface deferred).
* write_vscode_launch with TOCTOU symlink rejection at the target
path regardless of force=True.
Part D — Demo bundles
* `soup data demo` lists / copies 4 bundled JSONL fixtures
(alpaca / sharegpt / dpo / grpo) with staged-tempfile atomic
rename + 50 MB cap + symlink rejection on the staging path.
Tests: 5389 -> 5628 (+239). Ruff clean. Five sequential review waves
(python / code / security / tdd / smoke) ran; HIGH/MEDIUM/LOW findings
all fixed including: tracker name shadow in train.py, _lcs_length DP
double-buffer bug, BLEU geo-mean policy, base_dir absolute/.. escape,
demo_bundles tmp symlink TOCTOU, vscode launch symlink TOCTOU.
Note (Windows CI): line-ending warnings (LF -> CRLF) on commit are
expected; `.gitattributes` policy is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
05093ebfdb
commit
82d5693b75
|
|
@ -111,7 +111,7 @@ soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (152 files, 5389 tests)
|
||||
tests/ - Test suite (156 files, 5628 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
168
README.md
168
README.md
|
|
@ -43,15 +43,15 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.42.0 — Data Pipeline Pro**: closes the data-tooling gap with Axolotl + LlamaFactory in one release. 18 features across formats, remote loading, AOT preprocessing, advanced masking, vocab expansion, and document ingestion.
|
||||
**v0.43.0 — Tracker & Eval Pro**: closes the observability gap with all three competitors in one release. 18 features across tracker integrations, NLG eval metrics, profiling extras, and bundled demo datasets.
|
||||
|
||||
- **5 new data formats** — `prm` (PRM stepwise-supervised), `pre_tokenized` (LF `tokenized_path` / Axolotl `empty` type), `input_output` (template-free segments+labels with no chat template), `video`, and `multimodal` (axolotl typed content-parts schema with text / image / audio / video). Each is loud-fail validated — bool labels, null-byte rejection, length caps on path/url-like fields.
|
||||
- **Remote dataset loading (schema gate live, fsspec wiring deferred to v0.42.1)** — `data.train: s3://my-bucket/path` accepts the 7-scheme allowlist (`s3` / `gs` / `gcs` / `az` / `abfs` / `abfss` / `oci`). Bucket name regex is RFC-3986-tight, userinfo / fragment / query strings are rejected (the query-string gate is SSRF-adjacent — fsspec backends interpret `?endpoint_url=…` as config overrides). New schema fields: `streaming: bool`, `buffer_size: int [1, 1_000_000]`, `shards: int [1, 1024]`.
|
||||
- **AOT preprocessing + tokenized cache** — new `soup data preprocess <config>` CLI emits a deterministic 16-char SHA-256 cache key from `(dataset, tokenizer, max_length, format)`. Pair with the new `data.tokenized_path` schema field to skip the tokenize stage on subsequent runs.
|
||||
- **Advanced masking + multi-dataset interleave** — `data.interleave: concat / under / over / {strategy: probs, probs: [...]}` (sum-to-1 ±1e-6, max 32 datasets, `InterleaveSpec` is a frozen dataclass). Plus `mask_history`, `train_on_prompt` (mutually exclusive with `train_on_responses_only`), `eval_on_each_dataset`, `split_thinking` for Qwen3-style `<think>` masking, and per-image min/max pixels + `image_resize_algorithm` Literal + `video_fps` / `video_maxlen` / `video_dir`.
|
||||
- **Vocab expansion + custom prompt strategies** — `data.add_new_tokens: ["<reasoning>", "</reasoning>"]` + `new_special_tokens` (cap 10_000 entries, no duplicates, null-byte rejected) + `resize_vocab: bool` cross-validator. New `prompt_strategy: my_module.path:my_fn` field with regex validation lays groundwork for v0.42.1 runtime invocation. Plus axolotl-style `skip_prepare_dataset` / `remove_unused_columns` escape hatches.
|
||||
- **Document ingestion** — new `soup data ingest <doc.pdf>` (PDF / DOCX / MD / TXT) → JSONL with one row per page / paragraph. Lazy-imports `pypdf` / `python-docx` so missing optional deps never break `soup data --help`. Symlink-rejected, cwd-confined.
|
||||
- **+140 net new tests** — covers all 18 features end-to-end: schema validators, cross-validators, frozen-dataclass mutation guards, URL allowlist + bucket regex / single-char bucket / SSRF query rejection, every new format converter happy + failure path, CLI help + smoke + symlink-rejection, full SoupConfig YAML round-trip.
|
||||
- **Tracker integrations** — new `--tracker` flag on `soup train` accepts `mlflow` / `swanlab` / `trackio` (mutually exclusive with `--wandb` / `--tensorboard`). Closed allowlist via `MappingProxyType`-locked registry; case-insensitive lookup; null-byte and >32-char inputs rejected. PostHog telemetry payload schema lands as opt-in (`SOUP_TELEMETRY=1`) with hardware-info-only fields — no model names, dataset paths, or config contents. Live network code deferred to v0.43.1.
|
||||
- **BLEU + ROUGE-1 / ROUGE-2 / ROUGE-L** — pure-Python implementations exposed via `soup eval custom --metric bleu|rouge_l|...`. Standard BLEU policy: any zero-precision n-gram collapses score to 0.0; Chen & Cherry smoothing (default on) only smooths zero-correct buckets where `total[n] > 0`. Plus `effective_tokens_per_second` as a metric — `unmasked_tokens / wall_clock_seconds`, returns `None` when wall_clock ≤ 0 (no fabrication).
|
||||
- **KL-divergence calibration framework** — `soup_cli.eval.calibrate.run_calibration(baseline_logits, quantized_logits)` returns a frozen `CalibrationReport(mean_kl, per_prompt_kl, delta_status)` with OK / MINOR / MAJOR thresholds at 0.05 / 0.20. Pure-math kernel; bring your own logit pairs.
|
||||
- **Model Arena (Elo tournament)** — `soup_cli.eval.arena.Tournament` with K=32 default Elo, 256-model cap, 1M-match cap, `MappingProxyType` immutability on the public `ratings` view, and Rich-markup `[`/`]` rejection on model names so leaderboards can't be markup-injected.
|
||||
- **Profiling extras** — `memory_snapshot_context` wraps `torch.cuda.memory._record_memory_history` with cwd-confined snapshot path. `nccl_bandwidth_check` ships a reference-bandwidth table (h100/a100/v100/rtx-series, NVLink + PCIe) classifying measured bandwidth as OK ≥80% / MINOR ≥50% / MAJOR <50%. `soup doctor --vscode` writes a `.vscode/launch.json` with `soup train` + pytest configs, symlink-rejected at the target path.
|
||||
- **Bundled demo datasets** — new `soup data demo` lists 4 ready-to-use JSONL fixtures (alpaca / sharegpt / dpo / grpo). `soup data demo alpaca_demo --output ./mine.jsonl` copies the bundle for instant `soup train` warm-up. Staged-tempfile write with atomic rename — mid-stream rejection never leaves a partial file.
|
||||
- **+239 net new tests** — covers all 18 features: tracker name allowlist, telemetry payload schema invariant (no user data leaks), BLEU/ROUGE corner cases, KL thresholds, Elo math + tournament invariants, NCCL bandwidth boundaries, vscode TOCTOU symlink rejection, demo bundle atomic-rename + cwd containment + size cap.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -2443,6 +2443,155 @@ Every completed run also stores an estimated cost (`$` per run) computed from th
|
|||
captured GPU device name and duration. `soup runs show` renders `—` for CPU /
|
||||
MPS / unknown GPUs (no fabricated zeros).
|
||||
|
||||
### Tracker integrations (--tracker mlflow / swanlab / trackio)
|
||||
|
||||
```bash
|
||||
# Stream metrics to MLflow (set MLFLOW_TRACKING_URI to your server URL)
|
||||
soup train --config soup.yaml --tracker mlflow
|
||||
|
||||
# Or SwanLab (cloud or local)
|
||||
soup train --config soup.yaml --tracker swanlab
|
||||
|
||||
# Or Trackio (offline-friendly batched upload)
|
||||
soup train --config soup.yaml --tracker trackio
|
||||
```
|
||||
|
||||
`--tracker` is mutually exclusive with `--wandb` and `--tensorboard`. Soup
|
||||
validates the tracker name against a closed allowlist (`mlflow` / `swanlab` /
|
||||
`trackio` / `wandb` / `tensorboard` / `none`); the upstream package itself is
|
||||
loaded by HF Trainer at run time, so install the one you need separately:
|
||||
|
||||
```bash
|
||||
pip install mlflow # or: swanlab / trackio
|
||||
```
|
||||
|
||||
### Telemetry (opt-in)
|
||||
|
||||
Soup ships a hardware-info-only telemetry payload (Soup version + command +
|
||||
Python major.minor + OS + arch + duration). It is **off by default** and never
|
||||
sends model names, dataset paths, or config contents. Enable explicitly:
|
||||
|
||||
```bash
|
||||
SOUP_TELEMETRY=1 soup train --config soup.yaml
|
||||
```
|
||||
|
||||
The PostHog network upload itself is deferred to v0.43.1; v0.43.0 ships the
|
||||
payload schema only so you can audit it before opting in.
|
||||
|
||||
## NLG Evaluation Metrics (BLEU + ROUGE)
|
||||
|
||||
Pure-Python BLEU + ROUGE-1 / ROUGE-2 / ROUGE-L for `soup eval custom`:
|
||||
|
||||
```python
|
||||
from soup_cli.utils.nlg_metrics import (
|
||||
bleu_score, rouge_l_score, compute_nlg_metric, NLG_METRICS,
|
||||
effective_tokens_per_second,
|
||||
)
|
||||
|
||||
bleu_score(["the cat sat on the mat"], ["the cat sat on the mat"])
|
||||
# 1.0
|
||||
rouge_l_score(["the quick brown fox"], ["a quick brown dog"])
|
||||
# 0.5
|
||||
compute_nlg_metric("rouge_2", preds, refs)
|
||||
# generic dispatch by canonical name
|
||||
|
||||
effective_tokens_per_second(unmasked_tokens=12_500_000, wall_clock_seconds=600.0)
|
||||
# 20833.33 — None when wall_clock <= 0 (no fabrication)
|
||||
```
|
||||
|
||||
Smoothed BLEU uses Chen & Cherry epsilon for zero-correct buckets where
|
||||
`total[n] > 0`; empty buckets (e.g. predictions shorter than `max_n` tokens)
|
||||
force the score to 0.0.
|
||||
|
||||
## Quant Calibration (KL Divergence)
|
||||
|
||||
Compare a quantized model to a full-precision baseline on a small fixed prompt
|
||||
set. OK / MINOR / MAJOR thresholds at 0.05 / 0.20 mean KL — same scale as
|
||||
`soup eval quant-check`.
|
||||
|
||||
```python
|
||||
from soup_cli.eval.calibrate import run_calibration
|
||||
|
||||
# baseline_logits / quantized_logits: list[list[float]] aligned per-prompt
|
||||
report = run_calibration(baseline_logits, quantized_logits)
|
||||
print(report.delta_status, report.mean_kl)
|
||||
# OK 0.012
|
||||
```
|
||||
|
||||
The kernel is pure-math and capped at 10 000 prompts to defend against
|
||||
accidental OOM. `CalibrationReport` is a frozen dataclass.
|
||||
|
||||
## Model Arena (Elo Tournament)
|
||||
|
||||
Local leaderboard with Elo ratings (K=32, base 1500). Bring your own pairwise
|
||||
winners — Soup just keeps the books:
|
||||
|
||||
```python
|
||||
from soup_cli.eval.arena import Tournament
|
||||
|
||||
t = Tournament()
|
||||
t.record("llama-3.1-8b-finetune", "qwen2.5-7b-finetune", winner="a")
|
||||
t.record("llama-3.1-8b-finetune", "mistral-7b-finetune", winner="draw")
|
||||
for row in t.leaderboard():
|
||||
print(row)
|
||||
```
|
||||
|
||||
Caps: 256 models per tournament, 1M matches. Model names with `[` or `]`
|
||||
characters are rejected so leaderboard rows can't be markup-injected.
|
||||
|
||||
## Profiling Extras
|
||||
|
||||
CUDA memory snapshots, anomaly tracing, and an NCCL bandwidth reference table:
|
||||
|
||||
```python
|
||||
from soup_cli.utils.profiling_v0_43 import (
|
||||
memory_snapshot_context, detect_anomaly_context, nccl_bandwidth_check,
|
||||
)
|
||||
|
||||
with memory_snapshot_context("run-123") as path:
|
||||
train_step()
|
||||
# On CUDA, dumps profiles/run-123.snapshot.pickle on exit.
|
||||
|
||||
with detect_anomaly_context():
|
||||
train_step()
|
||||
# torch.autograd.set_detect_anomaly(True)
|
||||
|
||||
result = nccl_bandwidth_check(
|
||||
gpu="h100", link="nvlink", measured_gb_per_sec=400.0,
|
||||
)
|
||||
# {'expected_gb_per_sec': 450.0, 'measured_gb_per_sec': 400.0,
|
||||
# 'ratio': 0.8889, 'status': 'OK'}
|
||||
```
|
||||
|
||||
## VS Code Setup (`.vscode/launch.json`)
|
||||
|
||||
One-shot writer for a sane debugger config:
|
||||
|
||||
```python
|
||||
from soup_cli.utils.vscode_setup import write_vscode_launch
|
||||
write_vscode_launch(config_path="soup.yaml")
|
||||
# Writes ./.vscode/launch.json with `soup train` + pytest entries.
|
||||
```
|
||||
|
||||
Symlink-rejected at the target path regardless of `force=True` to defend
|
||||
against pre-placed symlinks redirecting the write outside cwd.
|
||||
|
||||
## Demo Datasets (`soup data demo`)
|
||||
|
||||
Tiny JSONL fixtures bundled with Soup so you can warm up `soup train` without
|
||||
hunting for data:
|
||||
|
||||
```bash
|
||||
# List available bundles
|
||||
soup data demo
|
||||
|
||||
# Copy one into the current directory
|
||||
soup data demo alpaca_demo --output ./alpaca.jsonl
|
||||
```
|
||||
|
||||
Bundles: `alpaca_demo`, `sharegpt_demo`, `dpo_demo`, `grpo_demo`. Output path
|
||||
must stay under cwd; existing files are not overwritten.
|
||||
|
||||
## Observability & Dev UX
|
||||
|
||||
Tools that explain *why* a run misbehaved instead of dumping a stack trace.
|
||||
|
|
@ -2658,6 +2807,9 @@ soup data register --name my-ds --path d.jsonl --format alpaca Register dataset
|
|||
soup data unregister --name my-ds Remove from registry
|
||||
soup data push --input d.jsonl --hf-dataset user/name Upload local JSONL as HF dataset
|
||||
soup data registry List all registered datasets
|
||||
soup data demo List bundled demo JSONL fixtures
|
||||
soup data demo alpaca_demo --output ./d.jsonl Copy a bundled demo JSONL fixture
|
||||
soup train --config soup.yaml --tracker mlflow MLflow / SwanLab / Trackio integration
|
||||
soup profile --config soup.yaml Estimate memory/speed before training
|
||||
soup profile --config soup.yaml --gpu a100 Estimate for specific GPU
|
||||
soup profile --config soup.yaml --json Machine-readable output
|
||||
|
|
|
|||
10
SECURITY.md
10
SECURITY.md
File diff suppressed because one or more lines are too long
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.42.0"
|
||||
version = "0.43.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.42.0"
|
||||
__version__ = "0.43.0"
|
||||
|
|
|
|||
|
|
@ -2087,3 +2087,65 @@ def ingest_document(
|
|||
for row in rows:
|
||||
f.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
console.print(f"[green]Wrote {len(rows)} rows to[/] {output}")
|
||||
|
||||
|
||||
@app.command(name="demo")
|
||||
def demo_bundle(
|
||||
name: str = typer.Argument(
|
||||
None,
|
||||
help="Bundle name (alpaca_demo / sharegpt_demo / dpo_demo / grpo_demo). "
|
||||
"Omit to list available bundles.",
|
||||
),
|
||||
output: str = typer.Option(
|
||||
None,
|
||||
"--output",
|
||||
"-o",
|
||||
help="Destination JSONL path (defaults to ./<name>.jsonl).",
|
||||
),
|
||||
) -> None:
|
||||
"""List or copy a bundled demo dataset (v0.43.0).
|
||||
|
||||
`soup data demo` lists available bundles. `soup data demo <name>` copies
|
||||
the JSONL fixture to the current directory. Bundles are version-locked
|
||||
JSONL fixtures shipped under `examples/data/`.
|
||||
"""
|
||||
from rich.markup import escape as _esc
|
||||
from rich.table import Table
|
||||
|
||||
from soup_cli.utils.demo_bundles import (
|
||||
copy_bundle_to,
|
||||
get_bundle,
|
||||
list_bundles,
|
||||
)
|
||||
|
||||
if name is None:
|
||||
table = Table(title="Available demo bundles", show_lines=False)
|
||||
table.add_column("Name", style="cyan")
|
||||
table.add_column("Format", style="green")
|
||||
table.add_column("Description")
|
||||
for bundle in list_bundles():
|
||||
table.add_row(bundle.name, bundle.format, bundle.description)
|
||||
console.print(table)
|
||||
console.print(
|
||||
"[dim]Run: soup data demo <name> --output <path>[/]"
|
||||
)
|
||||
return
|
||||
|
||||
try:
|
||||
bundle = get_bundle(name)
|
||||
except ValueError as exc:
|
||||
console.print(f"[red]{_esc(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
target = output or f"./{bundle.name}.jsonl"
|
||||
try:
|
||||
written = copy_bundle_to(bundle.name, target)
|
||||
except FileExistsError as exc:
|
||||
console.print(f"[red]{_esc(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (ValueError, FileNotFoundError) as exc:
|
||||
console.print(f"[red]{_esc(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
console.print(
|
||||
f"[green]Copied bundle '{bundle.name}' to[/] {_esc(written)}"
|
||||
)
|
||||
|
|
|
|||
|
|
@ -55,6 +55,14 @@ def train(
|
|||
"--tensorboard",
|
||||
help="Enable TensorBoard logging (logs to output_dir/runs/)",
|
||||
),
|
||||
tracker: str = typer.Option(
|
||||
None,
|
||||
"--tracker",
|
||||
help=(
|
||||
"Experiment tracker: mlflow / swanlab / trackio (v0.43.0). "
|
||||
"Mutually exclusive with --wandb / --tensorboard."
|
||||
),
|
||||
),
|
||||
deepspeed: str = typer.Option(
|
||||
None,
|
||||
"--deepspeed",
|
||||
|
|
@ -640,6 +648,10 @@ def train(
|
|||
dataset = load_dataset(cfg.data)
|
||||
console.print(f"[green]Loaded:[/] {len(dataset['train'])} train samples")
|
||||
|
||||
# Capture the --tracker CLI value BEFORE the local ExperimentTracker
|
||||
# shadows it (v0.43.0 review fix — name-collision regression).
|
||||
tracker_backend = tracker
|
||||
|
||||
# Start experiment tracking
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
||||
|
|
@ -655,12 +667,17 @@ def train(
|
|||
console.print(f"[dim]Run ID: {run_id}[/]")
|
||||
|
||||
# Build trainer based on task type
|
||||
if wandb:
|
||||
report_to = "wandb"
|
||||
elif tensorboard:
|
||||
report_to = "tensorboard"
|
||||
else:
|
||||
report_to = "none"
|
||||
from soup_cli.utils.trackers import resolve_report_to
|
||||
|
||||
try:
|
||||
report_to = resolve_report_to(
|
||||
wandb=wandb, tensorboard=tensorboard, tracker=tracker_backend
|
||||
)
|
||||
except ValueError as exc:
|
||||
from rich.markup import escape as _esc
|
||||
|
||||
console.print(f"[red]{_esc(str(exc))}[/]")
|
||||
raise typer.Exit(code=2) from exc
|
||||
console.print("[dim]Setting up model + trainer...[/]")
|
||||
trainer_kwargs = {
|
||||
"device": device,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,182 @@
|
|||
"""v0.43.0 Part B — Model Arena: A/B tournament with Elo ratings.
|
||||
|
||||
Pure-math Elo kernel + tournament aggregator. Generation + judging is the
|
||||
caller's responsibility — feed `record_match_result` with model A / model B
|
||||
identifiers and a winner string, get back updated ratings.
|
||||
|
||||
Mirrors the existing `eval/human.py` Elo policy (K=32, base 1500) but
|
||||
operates on a closed model registry so a single `Tournament` can host many
|
||||
models and produce a leaderboard.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass, field
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping
|
||||
|
||||
DEFAULT_K = 32.0
|
||||
DEFAULT_BASE_RATING = 1500.0
|
||||
_MAX_NAME_LEN = 128
|
||||
_MAX_MODELS = 256
|
||||
_MAX_MATCHES = 1_000_000
|
||||
|
||||
|
||||
def _validate_model_name(name: object) -> str:
|
||||
if not isinstance(name, str):
|
||||
raise ValueError("model name must be a string")
|
||||
if not name:
|
||||
raise ValueError("model name must not be empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("model name must not contain null bytes")
|
||||
if len(name) > _MAX_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"model name length {len(name)} exceeds max {_MAX_NAME_LEN}"
|
||||
)
|
||||
# Reject Rich markup metacharacters at the source so any downstream
|
||||
# CLI consumer that embeds the leaderboard `model` field in markup
|
||||
# cannot be markup-injected (security review fix).
|
||||
if "[" in name or "]" in name:
|
||||
raise ValueError(
|
||||
"model name must not contain Rich markup metacharacters '[' or ']'"
|
||||
)
|
||||
return name
|
||||
|
||||
|
||||
def expected_score(rating_a: float, rating_b: float) -> float:
|
||||
"""Probability that A beats B given Elo ratings."""
|
||||
for r in (rating_a, rating_b):
|
||||
if isinstance(r, bool) or not isinstance(r, (int, float)):
|
||||
raise ValueError("ratings must be int/float")
|
||||
if not math.isfinite(float(r)):
|
||||
raise ValueError("ratings must be finite")
|
||||
return 1.0 / (1.0 + math.pow(10.0, (rating_b - rating_a) / 400.0))
|
||||
|
||||
|
||||
def update_elo(
|
||||
rating_a: float,
|
||||
rating_b: float,
|
||||
*,
|
||||
score_a: float,
|
||||
k: float = DEFAULT_K,
|
||||
) -> tuple[float, float]:
|
||||
"""Update Elo ratings after a match where A scored `score_a` ∈ [0,1].
|
||||
|
||||
score_a == 1.0 → A won.
|
||||
score_a == 0.0 → B won.
|
||||
score_a == 0.5 → draw.
|
||||
"""
|
||||
if isinstance(score_a, bool) or not isinstance(score_a, (int, float)):
|
||||
raise ValueError("score_a must be int/float")
|
||||
if not math.isfinite(float(score_a)) or score_a < 0.0 or score_a > 1.0:
|
||||
raise ValueError("score_a must be in [0, 1]")
|
||||
if isinstance(k, bool) or not isinstance(k, (int, float)):
|
||||
raise ValueError("k must be a positive number")
|
||||
if not math.isfinite(float(k)) or k <= 0:
|
||||
raise ValueError("k must be a positive finite number")
|
||||
|
||||
expected_a = expected_score(rating_a, rating_b)
|
||||
new_a = rating_a + k * (score_a - expected_a)
|
||||
new_b = rating_b + k * ((1.0 - score_a) - (1.0 - expected_a))
|
||||
return new_a, new_b
|
||||
|
||||
|
||||
@dataclass
|
||||
class Tournament:
|
||||
"""Live tournament tracker with Elo + win/loss bookkeeping."""
|
||||
|
||||
base_rating: float = DEFAULT_BASE_RATING
|
||||
k: float = DEFAULT_K
|
||||
_ratings: dict[str, float] = field(default_factory=dict)
|
||||
_wins: dict[str, int] = field(default_factory=dict)
|
||||
_losses: dict[str, int] = field(default_factory=dict)
|
||||
_draws: dict[str, int] = field(default_factory=dict)
|
||||
_matches: int = 0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if (
|
||||
isinstance(self.base_rating, bool)
|
||||
or not isinstance(self.base_rating, (int, float))
|
||||
or not math.isfinite(float(self.base_rating))
|
||||
):
|
||||
raise ValueError("base_rating must be a finite number")
|
||||
if isinstance(self.k, bool) or not isinstance(self.k, (int, float)) or self.k <= 0:
|
||||
raise ValueError("k must be positive")
|
||||
|
||||
def register(self, name: str) -> None:
|
||||
"""Register a model with the base rating."""
|
||||
canonical = _validate_model_name(name)
|
||||
if canonical in self._ratings:
|
||||
return
|
||||
if len(self._ratings) >= _MAX_MODELS:
|
||||
raise ValueError(
|
||||
f"tournament has reached the model cap ({_MAX_MODELS})"
|
||||
)
|
||||
self._ratings[canonical] = float(self.base_rating)
|
||||
self._wins[canonical] = 0
|
||||
self._losses[canonical] = 0
|
||||
self._draws[canonical] = 0
|
||||
|
||||
def record(
|
||||
self,
|
||||
model_a: str,
|
||||
model_b: str,
|
||||
*,
|
||||
winner: str,
|
||||
) -> tuple[float, float]:
|
||||
"""Record a match. `winner` ∈ {"a", "b", "draw"}.
|
||||
|
||||
Returns the new (rating_a, rating_b).
|
||||
"""
|
||||
if self._matches >= _MAX_MATCHES:
|
||||
raise ValueError(
|
||||
f"tournament has reached the match cap ({_MAX_MATCHES})"
|
||||
)
|
||||
a = _validate_model_name(model_a)
|
||||
b = _validate_model_name(model_b)
|
||||
if a == b:
|
||||
raise ValueError("model_a and model_b must differ")
|
||||
if not isinstance(winner, str):
|
||||
raise ValueError("winner must be a string")
|
||||
winner_norm = winner.lower()
|
||||
if winner_norm not in {"a", "b", "draw"}:
|
||||
raise ValueError("winner must be one of 'a' / 'b' / 'draw'")
|
||||
self.register(a)
|
||||
self.register(b)
|
||||
score_a = {"a": 1.0, "b": 0.0, "draw": 0.5}[winner_norm]
|
||||
new_a, new_b = update_elo(
|
||||
self._ratings[a], self._ratings[b], score_a=score_a, k=self.k
|
||||
)
|
||||
self._ratings[a] = new_a
|
||||
self._ratings[b] = new_b
|
||||
if winner_norm == "a":
|
||||
self._wins[a] += 1
|
||||
self._losses[b] += 1
|
||||
elif winner_norm == "b":
|
||||
self._wins[b] += 1
|
||||
self._losses[a] += 1
|
||||
else:
|
||||
self._draws[a] += 1
|
||||
self._draws[b] += 1
|
||||
self._matches += 1
|
||||
return new_a, new_b
|
||||
|
||||
@property
|
||||
def ratings(self) -> Mapping[str, float]:
|
||||
return MappingProxyType(dict(self._ratings))
|
||||
|
||||
def leaderboard(self) -> list[dict]:
|
||||
"""Sorted leaderboard, highest rating first."""
|
||||
rows = []
|
||||
for name in self._ratings:
|
||||
rows.append(
|
||||
{
|
||||
"model": name,
|
||||
"rating": round(self._ratings[name], 2),
|
||||
"wins": self._wins[name],
|
||||
"losses": self._losses[name],
|
||||
"draws": self._draws[name],
|
||||
}
|
||||
)
|
||||
rows.sort(key=lambda r: r["rating"], reverse=True)
|
||||
return rows
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
"""v0.43.0 Part B — ceval / cmmlu / Aider Polyglot benchmark allowlist.
|
||||
|
||||
ceval / cmmlu route through the existing lm-evaluation-harness wiring; this
|
||||
module exposes the validated benchmark name allowlist + Aider prompt-loader
|
||||
scaffold. Live Aider Polyglot eval requires the upstream `aider-chat` package
|
||||
and Docker; live wiring is deferred to v0.43.1 — this release ships the
|
||||
benchmark name allowlist + path containment so YAML configs lock in.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping
|
||||
|
||||
# v0.43.0 additive benchmark names.
|
||||
NEW_BENCHMARKS_V0_43 = frozenset({"ceval", "cmmlu", "aider_polyglot"})
|
||||
|
||||
# Lightweight per-benchmark metadata (description + lm-eval task name).
|
||||
_BENCHMARK_META: Mapping[str, Mapping[str, str]] = MappingProxyType({
|
||||
"ceval": MappingProxyType({
|
||||
"description": "Chinese evaluation suite (52 subjects)",
|
||||
"lm_eval_task": "ceval-valid",
|
||||
}),
|
||||
"cmmlu": MappingProxyType({
|
||||
"description": "Chinese MMLU (67 subjects)",
|
||||
"lm_eval_task": "cmmlu",
|
||||
}),
|
||||
"aider_polyglot": MappingProxyType({
|
||||
"description": "Aider Polyglot benchmark (multi-language code editing)",
|
||||
"lm_eval_task": "", # No lm-eval mapping; uses upstream aider-chat
|
||||
}),
|
||||
})
|
||||
|
||||
|
||||
def is_v0_43_benchmark(name: object) -> bool:
|
||||
"""True if `name` names a v0.43.0-additive benchmark."""
|
||||
if not isinstance(name, str):
|
||||
return False
|
||||
return name.lower() in NEW_BENCHMARKS_V0_43
|
||||
|
||||
|
||||
def benchmark_metadata(name: str) -> Mapping[str, str] | None:
|
||||
"""Return read-only metadata for a v0.43.0 benchmark, or None."""
|
||||
if not isinstance(name, str):
|
||||
return None
|
||||
return _BENCHMARK_META.get(name.lower())
|
||||
|
||||
|
||||
def lm_eval_task_for(name: str) -> str | None:
|
||||
"""Map a benchmark name to its lm-eval-harness task id, or None."""
|
||||
meta = benchmark_metadata(name)
|
||||
if meta is None:
|
||||
return None
|
||||
task = meta.get("lm_eval_task")
|
||||
return task if task else None
|
||||
|
|
@ -0,0 +1,118 @@
|
|||
"""v0.43.0 Part B — KL Divergence calibration framework (Unsloth Calibration_v3/v5).
|
||||
|
||||
Compares logits between baseline and quantized models on a small fixed subset
|
||||
(default: 5-shot MMLU). Pure-math kernel `kl_divergence` operates on numpy
|
||||
arrays and is safe to call without torch. Live model loading + tokenization
|
||||
is the caller's responsibility — `run_calibration` accepts pre-computed logit
|
||||
matrices so the same kernel works for any pair of models the user can load.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from typing import Sequence
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CalibrationReport:
|
||||
"""Per-prompt KL divergence + corpus mean.
|
||||
|
||||
`delta_status` follows v0.26.0 Part D quant-check policy:
|
||||
- "OK" : mean_kl < 0.05
|
||||
- "MINOR" : 0.05 <= mean_kl < 0.20
|
||||
- "MAJOR" : mean_kl >= 0.20
|
||||
"""
|
||||
|
||||
mean_kl: float
|
||||
per_prompt_kl: tuple[float, ...]
|
||||
delta_status: str
|
||||
num_prompts: int
|
||||
|
||||
|
||||
def _softmax(logits: Sequence[float]) -> list[float]:
|
||||
if not logits:
|
||||
raise ValueError("logits must not be empty")
|
||||
max_l = max(logits)
|
||||
exps = [math.exp(x - max_l) for x in logits]
|
||||
s = sum(exps)
|
||||
if s <= 0:
|
||||
raise ValueError("softmax denominator non-positive")
|
||||
return [e / s for e in exps]
|
||||
|
||||
|
||||
def kl_divergence(p_logits: Sequence[float], q_logits: Sequence[float]) -> float:
|
||||
"""KL(P || Q) over discrete distributions derived from logits.
|
||||
|
||||
Both inputs must be the same length and contain finite floats.
|
||||
Returns a non-negative float; uses natural log.
|
||||
"""
|
||||
if len(p_logits) != len(q_logits):
|
||||
raise ValueError(
|
||||
f"p_logits ({len(p_logits)}) and q_logits "
|
||||
f"({len(q_logits)}) must have the same length"
|
||||
)
|
||||
for name, logits in (("p_logits", p_logits), ("q_logits", q_logits)):
|
||||
if not logits:
|
||||
raise ValueError(f"{name} must not be empty")
|
||||
for x in logits:
|
||||
if isinstance(x, bool) or not isinstance(x, (int, float)):
|
||||
raise ValueError(f"{name} must contain only int/float")
|
||||
if not math.isfinite(float(x)):
|
||||
raise ValueError(f"{name} must be finite")
|
||||
p = _softmax(p_logits)
|
||||
q = _softmax(q_logits)
|
||||
kl = 0.0
|
||||
for pi, qi in zip(p, q):
|
||||
if pi <= 0:
|
||||
continue
|
||||
# qi could be ~0; floor with epsilon to avoid log(0).
|
||||
kl += pi * math.log(pi / max(qi, 1e-12))
|
||||
# Floating-point can produce tiny negatives; clamp.
|
||||
return max(0.0, kl)
|
||||
|
||||
|
||||
def classify_kl_delta(mean_kl: float) -> str:
|
||||
"""OK / MINOR / MAJOR thresholds (v0.26.0 Part D policy)."""
|
||||
if isinstance(mean_kl, bool) or not isinstance(mean_kl, (int, float)):
|
||||
raise ValueError("mean_kl must be a number")
|
||||
if not math.isfinite(float(mean_kl)) or mean_kl < 0:
|
||||
raise ValueError("mean_kl must be a finite non-negative number")
|
||||
if mean_kl < 0.05:
|
||||
return "OK"
|
||||
if mean_kl < 0.20:
|
||||
return "MINOR"
|
||||
return "MAJOR"
|
||||
|
||||
|
||||
def run_calibration(
|
||||
baseline_logits: Sequence[Sequence[float]],
|
||||
quantized_logits: Sequence[Sequence[float]],
|
||||
) -> CalibrationReport:
|
||||
"""Run calibration on aligned baseline + quantized logit pairs.
|
||||
|
||||
Each entry of `*_logits` is the next-token-logit row for one prompt
|
||||
(shape: (vocab,)). Both must have the same outer length.
|
||||
"""
|
||||
base_list = list(baseline_logits)
|
||||
quant_list = list(quantized_logits)
|
||||
if len(base_list) != len(quant_list):
|
||||
raise ValueError(
|
||||
f"baseline_logits ({len(base_list)}) and quantized_logits "
|
||||
f"({len(quant_list)}) must have the same length"
|
||||
)
|
||||
if not base_list:
|
||||
raise ValueError("at least one prompt is required")
|
||||
if len(base_list) > 10_000:
|
||||
raise ValueError(
|
||||
f"too many prompts ({len(base_list)}); cap is 10000"
|
||||
)
|
||||
per_prompt = tuple(
|
||||
kl_divergence(b, q) for b, q in zip(base_list, quant_list)
|
||||
)
|
||||
mean = sum(per_prompt) / len(per_prompt)
|
||||
return CalibrationReport(
|
||||
mean_kl=mean,
|
||||
per_prompt_kl=per_prompt,
|
||||
delta_status=classify_kl_delta(mean),
|
||||
num_prompts=len(per_prompt),
|
||||
)
|
||||
|
|
@ -0,0 +1,178 @@
|
|||
"""v0.43.0 Part D — `soup data demo` bundle registry.
|
||||
|
||||
Single source of truth for the small JSONL fixtures bundled under
|
||||
`examples/data/`. The CLI command resolves a name to a path, copies the
|
||||
bundle into the user-supplied output (containment-checked), and prints
|
||||
a short summary. No network access. Path containment via shared
|
||||
`is_under_cwd` (mirrors v0.42.0 ingest policy).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from importlib.resources import files
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping
|
||||
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
_MAX_NAME_LEN = 32
|
||||
_MAX_OUTPUT_BYTES = 50 * 1024 * 1024 # 50 MB defence
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DemoBundle:
|
||||
name: str
|
||||
fixture: str # filename under examples/data/
|
||||
description: str
|
||||
format: str # alpaca / sharegpt / dpo / reasoning
|
||||
|
||||
|
||||
_BUNDLES: Mapping[str, DemoBundle] = MappingProxyType({
|
||||
"alpaca_demo": DemoBundle(
|
||||
name="alpaca_demo",
|
||||
fixture="alpaca_tiny.jsonl",
|
||||
description="20-row Alpaca-style instruction tuning fixture",
|
||||
format="alpaca",
|
||||
),
|
||||
"sharegpt_demo": DemoBundle(
|
||||
name="sharegpt_demo",
|
||||
fixture="chat_preferences.jsonl",
|
||||
description="ShareGPT-style multi-turn chat fixture",
|
||||
format="sharegpt",
|
||||
),
|
||||
"dpo_demo": DemoBundle(
|
||||
name="dpo_demo",
|
||||
fixture="dpo_sample.jsonl",
|
||||
description="Preference (prompt/chosen/rejected) DPO fixture",
|
||||
format="dpo",
|
||||
),
|
||||
"grpo_demo": DemoBundle(
|
||||
name="grpo_demo",
|
||||
fixture="reasoning_math.jsonl",
|
||||
description="Math reasoning fixture for GRPO/RLVR",
|
||||
format="reasoning",
|
||||
),
|
||||
})
|
||||
|
||||
DEMO_BUNDLE_NAMES = frozenset(_BUNDLES.keys())
|
||||
|
||||
|
||||
def _validate_name(name: object) -> str:
|
||||
if not isinstance(name, str):
|
||||
raise ValueError("bundle name must be a string")
|
||||
if not name:
|
||||
raise ValueError("bundle name must not be empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("bundle name must not contain null bytes")
|
||||
if len(name) > _MAX_NAME_LEN:
|
||||
raise ValueError(f"bundle name length {len(name)} exceeds max {_MAX_NAME_LEN}")
|
||||
if name not in _BUNDLES:
|
||||
supported = ", ".join(sorted(_BUNDLES))
|
||||
raise ValueError(f"unknown bundle '{name}'. Supported: {supported}")
|
||||
return name
|
||||
|
||||
|
||||
def list_bundles() -> list[DemoBundle]:
|
||||
"""Return all registered demo bundles, sorted by name."""
|
||||
return [_BUNDLES[name] for name in sorted(_BUNDLES)]
|
||||
|
||||
|
||||
def get_bundle(name: str) -> DemoBundle:
|
||||
"""Return the bundle metadata for `name`, or raise ValueError."""
|
||||
canonical = _validate_name(name)
|
||||
return _BUNDLES[canonical]
|
||||
|
||||
|
||||
def _bundle_source_path(bundle: DemoBundle) -> str:
|
||||
"""Resolve the on-disk path for a bundle's fixture.
|
||||
|
||||
Uses importlib.resources to handle both editable + wheel installs.
|
||||
"""
|
||||
# Filenames are baked-in constants (no user input), so direct join
|
||||
# is safe; we still defensively reject path separators.
|
||||
if "/" in bundle.fixture or "\\" in bundle.fixture:
|
||||
raise ValueError(
|
||||
f"bundle fixture name has separator: {bundle.fixture!r}"
|
||||
)
|
||||
# examples/ lives at the repo root, alongside the soup_cli/ package.
|
||||
pkg_root = files("soup_cli")
|
||||
repo_root = os.path.dirname(str(pkg_root))
|
||||
candidate = os.path.realpath(
|
||||
os.path.join(repo_root, "examples", "data", bundle.fixture)
|
||||
)
|
||||
if not os.path.isfile(candidate):
|
||||
raise FileNotFoundError(
|
||||
f"bundle fixture missing: {bundle.fixture}"
|
||||
)
|
||||
return candidate
|
||||
|
||||
|
||||
def copy_bundle_to(name: str, output_path: str) -> str:
|
||||
"""Copy bundle's JSONL into `output_path`. Containment-checked.
|
||||
|
||||
Returns the absolute path written. Refuses to overwrite existing files
|
||||
(caller must remove first). Validates that every line of the bundle is
|
||||
valid JSON to avoid silently shipping a malformed fixture.
|
||||
"""
|
||||
bundle = get_bundle(name)
|
||||
if not isinstance(output_path, str) or not output_path:
|
||||
raise ValueError("output_path must be a non-empty string")
|
||||
if "\x00" in output_path:
|
||||
raise ValueError("output_path must not contain null bytes")
|
||||
real_out = os.path.realpath(output_path)
|
||||
if not is_under_cwd(real_out):
|
||||
raise ValueError("output_path must stay under cwd")
|
||||
if os.path.exists(real_out):
|
||||
raise FileExistsError(
|
||||
f"{output_path} already exists; remove it before re-running"
|
||||
)
|
||||
src = _bundle_source_path(bundle)
|
||||
os.makedirs(os.path.dirname(real_out) or ".", exist_ok=True)
|
||||
# Stage writes to a sibling temp file so a mid-stream cap rejection
|
||||
# never leaves a partial file at the user-visible target path.
|
||||
tmp_path = real_out + ".tmp"
|
||||
# TOCTOU defence: reject pre-placed symlinks at the temp path
|
||||
# (mirrors v0.33.0 #22 / v0.40.2 #51 / v0.42.0 ingest policy).
|
||||
try:
|
||||
if stat.S_ISLNK(os.lstat(tmp_path).st_mode):
|
||||
raise ValueError(
|
||||
"staging temp path is a symlink; aborting"
|
||||
)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
total = 0
|
||||
try:
|
||||
with open(src, encoding="utf-8") as f_in, open(
|
||||
tmp_path, "w", encoding="utf-8"
|
||||
) as f_out:
|
||||
for lineno, raw in enumerate(f_in, start=1):
|
||||
stripped = raw.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
try:
|
||||
json.loads(stripped)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(
|
||||
f"bundle {name} line {lineno} is not valid JSON: {exc}"
|
||||
) from exc
|
||||
total += len(raw.encode("utf-8"))
|
||||
if total > _MAX_OUTPUT_BYTES:
|
||||
raise ValueError(
|
||||
f"bundle {name} exceeds {_MAX_OUTPUT_BYTES} byte cap"
|
||||
)
|
||||
f_out.write(raw)
|
||||
if not raw.endswith("\n"):
|
||||
f_out.write("\n")
|
||||
os.replace(tmp_path, real_out)
|
||||
except BaseException:
|
||||
# Best-effort cleanup of the staged temp file on any failure.
|
||||
try:
|
||||
if os.path.exists(tmp_path):
|
||||
os.remove(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
return real_out
|
||||
|
|
@ -0,0 +1,259 @@
|
|||
"""v0.43.0 Part B — BLEU + ROUGE NLG metrics + effective_tokens_per_second.
|
||||
|
||||
Pure-Python implementations sufficient for unit-level eval. For research-grade
|
||||
scoring users should still wire in `sacrebleu` / `rouge_score` via lm-eval; this
|
||||
module provides a self-contained baseline that does not require those packages
|
||||
so `soup eval custom --metric bleu` / `--metric rouge_l` works on a vanilla
|
||||
install. The closed metric allowlist `NLG_METRICS` is shared with the schema
|
||||
field validator on `EvalConfig.nlg_metrics`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from collections import Counter
|
||||
from typing import Iterable, Sequence
|
||||
|
||||
NLG_METRICS = frozenset({"bleu", "rouge_1", "rouge_2", "rouge_l"})
|
||||
|
||||
# Bounds matching v0.19.0 custom-eval policy.
|
||||
_MAX_INPUT_CHARS = 1_000_000
|
||||
_MAX_NGRAM = 4
|
||||
|
||||
|
||||
def _tokenize(text: str) -> list[str]:
|
||||
"""Word-level whitespace + punctuation-stripped tokenizer.
|
||||
|
||||
Rejects null bytes / non-string / oversize input.
|
||||
"""
|
||||
if not isinstance(text, str):
|
||||
raise ValueError("text must be a string")
|
||||
if "\x00" in text:
|
||||
raise ValueError("text must not contain null bytes")
|
||||
if len(text) > _MAX_INPUT_CHARS:
|
||||
raise ValueError(
|
||||
f"text length {len(text)} exceeds max {_MAX_INPUT_CHARS}"
|
||||
)
|
||||
# Word-piece-style tokenizer: lowercase + non-alphanumeric split.
|
||||
return [tok for tok in re.findall(r"[A-Za-z0-9]+", text.lower()) if tok]
|
||||
|
||||
|
||||
def _ngrams(tokens: Sequence[str], n: int) -> Counter[tuple[str, ...]]:
|
||||
if n <= 0 or isinstance(n, bool):
|
||||
raise ValueError("n must be a positive int")
|
||||
if n > _MAX_NGRAM:
|
||||
raise ValueError(f"n must be <= {_MAX_NGRAM}")
|
||||
return Counter(tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1))
|
||||
|
||||
|
||||
def bleu_score(
|
||||
predictions: Iterable[str],
|
||||
references: Iterable[str],
|
||||
*,
|
||||
max_n: int = 4,
|
||||
smooth: bool = True,
|
||||
) -> float:
|
||||
"""Corpus-level BLEU (single reference per prediction).
|
||||
|
||||
Returns score in [0.0, 1.0]. Empty corpus returns 0.0.
|
||||
|
||||
`smooth=True` (default) uses Chen & Cherry epsilon smoothing for
|
||||
*zero-correct* buckets where `total[n] > 0`. It does NOT cover
|
||||
*empty* buckets where `total[n] == 0` (e.g. predictions shorter than
|
||||
`max_n` tokens) — those force the score to 0.0 even with smoothing.
|
||||
"""
|
||||
if isinstance(max_n, bool) or not isinstance(max_n, int):
|
||||
raise ValueError("max_n must be an int")
|
||||
if max_n < 1 or max_n > _MAX_NGRAM:
|
||||
raise ValueError(f"max_n must be in [1, {_MAX_NGRAM}]")
|
||||
|
||||
pred_list = list(predictions)
|
||||
ref_list = list(references)
|
||||
if len(pred_list) != len(ref_list):
|
||||
raise ValueError(
|
||||
f"predictions ({len(pred_list)}) and references "
|
||||
f"({len(ref_list)}) must have the same length"
|
||||
)
|
||||
if not pred_list:
|
||||
return 0.0
|
||||
|
||||
pred_lengths = 0
|
||||
ref_lengths = 0
|
||||
correct = [0] * max_n
|
||||
total = [0] * max_n
|
||||
|
||||
for pred, ref in zip(pred_list, ref_list):
|
||||
pred_tokens = _tokenize(pred)
|
||||
ref_tokens = _tokenize(ref)
|
||||
pred_lengths += len(pred_tokens)
|
||||
ref_lengths += len(ref_tokens)
|
||||
for n in range(1, max_n + 1):
|
||||
if len(pred_tokens) < n:
|
||||
continue
|
||||
pred_ng = _ngrams(pred_tokens, n)
|
||||
ref_ng = _ngrams(ref_tokens, n)
|
||||
overlap = sum(min(c, ref_ng[ng]) for ng, c in pred_ng.items())
|
||||
correct[n - 1] += overlap
|
||||
total[n - 1] += sum(pred_ng.values())
|
||||
|
||||
# Modified n-gram precision with Chen & Cherry smoothing for empty buckets.
|
||||
precisions: list[float] = []
|
||||
for n in range(max_n):
|
||||
if total[n] == 0:
|
||||
precisions.append(0.0)
|
||||
continue
|
||||
if correct[n] == 0 and smooth:
|
||||
precisions.append(1.0 / (2.0 ** (n + 1) * total[n]))
|
||||
else:
|
||||
precisions.append(correct[n] / total[n])
|
||||
|
||||
# Standard BLEU: geometric mean over all max_n precisions; any zero
|
||||
# collapses the score to 0.0 unless smoothing is on.
|
||||
if any(p == 0.0 for p in precisions):
|
||||
return 0.0
|
||||
geo_mean = math.exp(sum(math.log(p) for p in precisions) / max_n)
|
||||
|
||||
# Brevity penalty.
|
||||
if pred_lengths == 0:
|
||||
bp = 0.0
|
||||
elif pred_lengths > ref_lengths:
|
||||
bp = 1.0
|
||||
else:
|
||||
bp = math.exp(1.0 - ref_lengths / pred_lengths)
|
||||
return float(bp * geo_mean)
|
||||
|
||||
|
||||
def _lcs_length(a: Sequence[str], b: Sequence[str]) -> int:
|
||||
"""Length of longest common subsequence — DP, O(len(a) * len(b))."""
|
||||
if not a or not b:
|
||||
return 0
|
||||
prev = [0] * (len(b) + 1)
|
||||
for i in range(1, len(a) + 1):
|
||||
curr = [0] * (len(b) + 1)
|
||||
for j in range(1, len(b) + 1):
|
||||
if a[i - 1] == b[j - 1]:
|
||||
curr[j] = prev[j - 1] + 1
|
||||
else:
|
||||
curr[j] = max(prev[j], curr[j - 1])
|
||||
prev = curr
|
||||
return prev[len(b)]
|
||||
|
||||
|
||||
def rouge_n_score(
|
||||
predictions: Iterable[str],
|
||||
references: Iterable[str],
|
||||
*,
|
||||
n: int = 1,
|
||||
) -> float:
|
||||
"""ROUGE-N F1 (corpus average over total pair count).
|
||||
|
||||
Pairs where either side has fewer than `n` tokens contribute 0.0 to
|
||||
the average — short-sentence corpora are penalised, matching the
|
||||
`rouge-score` default.
|
||||
"""
|
||||
if isinstance(n, bool) or not isinstance(n, int):
|
||||
raise ValueError("n must be an int")
|
||||
if n < 1 or n > _MAX_NGRAM:
|
||||
raise ValueError(f"n must be in [1, {_MAX_NGRAM}]")
|
||||
pred_list = list(predictions)
|
||||
ref_list = list(references)
|
||||
if len(pred_list) != len(ref_list):
|
||||
raise ValueError(
|
||||
"predictions and references must have the same length"
|
||||
)
|
||||
if not pred_list:
|
||||
return 0.0
|
||||
|
||||
f1_sum = 0.0
|
||||
for pred, ref in zip(pred_list, ref_list):
|
||||
pred_tokens = _tokenize(pred)
|
||||
ref_tokens = _tokenize(ref)
|
||||
if len(pred_tokens) < n or len(ref_tokens) < n:
|
||||
continue
|
||||
pred_ng = _ngrams(pred_tokens, n)
|
||||
ref_ng = _ngrams(ref_tokens, n)
|
||||
overlap = sum(min(c, ref_ng[ng]) for ng, c in pred_ng.items())
|
||||
if overlap == 0:
|
||||
continue
|
||||
precision = overlap / sum(pred_ng.values())
|
||||
recall = overlap / sum(ref_ng.values())
|
||||
if precision + recall > 0:
|
||||
f1_sum += 2 * precision * recall / (precision + recall)
|
||||
return f1_sum / len(pred_list)
|
||||
|
||||
|
||||
def rouge_l_score(
|
||||
predictions: Iterable[str],
|
||||
references: Iterable[str],
|
||||
) -> float:
|
||||
"""ROUGE-L F1 — corpus-average sentence-level LCS."""
|
||||
pred_list = list(predictions)
|
||||
ref_list = list(references)
|
||||
if len(pred_list) != len(ref_list):
|
||||
raise ValueError(
|
||||
"predictions and references must have the same length"
|
||||
)
|
||||
if not pred_list:
|
||||
return 0.0
|
||||
f1_sum = 0.0
|
||||
for pred, ref in zip(pred_list, ref_list):
|
||||
pred_tokens = _tokenize(pred)
|
||||
ref_tokens = _tokenize(ref)
|
||||
if not pred_tokens or not ref_tokens:
|
||||
continue
|
||||
lcs = _lcs_length(pred_tokens, ref_tokens)
|
||||
if lcs == 0:
|
||||
continue
|
||||
precision = lcs / len(pred_tokens)
|
||||
recall = lcs / len(ref_tokens)
|
||||
if precision + recall > 0:
|
||||
f1_sum += 2 * precision * recall / (precision + recall)
|
||||
return f1_sum / len(pred_list)
|
||||
|
||||
|
||||
def compute_nlg_metric(
|
||||
metric: str,
|
||||
predictions: Iterable[str],
|
||||
references: Iterable[str],
|
||||
) -> float:
|
||||
"""Dispatch by canonical metric name."""
|
||||
if not isinstance(metric, str):
|
||||
raise ValueError("metric must be a string")
|
||||
name = metric.lower()
|
||||
if name not in NLG_METRICS:
|
||||
supported = ", ".join(sorted(NLG_METRICS))
|
||||
raise ValueError(f"unknown nlg metric '{metric}'. Supported: {supported}")
|
||||
if name == "bleu":
|
||||
return bleu_score(predictions, references)
|
||||
if name == "rouge_1":
|
||||
return rouge_n_score(predictions, references, n=1)
|
||||
if name == "rouge_2":
|
||||
return rouge_n_score(predictions, references, n=2)
|
||||
if name == "rouge_l":
|
||||
return rouge_l_score(predictions, references)
|
||||
raise AssertionError("unreachable") # pragma: no cover
|
||||
|
||||
|
||||
def effective_tokens_per_second(
|
||||
*,
|
||||
unmasked_tokens: int,
|
||||
wall_clock_seconds: float,
|
||||
) -> float | None:
|
||||
"""Effective tokens-per-second (LF metric).
|
||||
|
||||
`unmasked_tokens` = total non-padding labels seen during training.
|
||||
Returns None when wall_clock <= 0 (avoid div-by-zero rather than fabricate).
|
||||
"""
|
||||
if isinstance(unmasked_tokens, bool) or not isinstance(unmasked_tokens, int):
|
||||
raise ValueError("unmasked_tokens must be an int")
|
||||
if unmasked_tokens < 0:
|
||||
raise ValueError("unmasked_tokens must be >= 0")
|
||||
if isinstance(wall_clock_seconds, bool) or not isinstance(
|
||||
wall_clock_seconds, (int, float)
|
||||
):
|
||||
raise ValueError("wall_clock_seconds must be a number")
|
||||
if not math.isfinite(float(wall_clock_seconds)):
|
||||
raise ValueError("wall_clock_seconds must be finite")
|
||||
if wall_clock_seconds <= 0:
|
||||
return None
|
||||
return unmasked_tokens / float(wall_clock_seconds)
|
||||
|
|
@ -0,0 +1,215 @@
|
|||
"""v0.43.0 Part C — Profiling extras.
|
||||
|
||||
- `memory_snapshot_context`: torch.cuda.memory._record_memory_history wrapper
|
||||
that writes a `<run_id>.snapshot.pickle` to the profiles dir under cwd.
|
||||
- `enable_detect_anomaly`: thin context manager around
|
||||
`torch.autograd.set_detect_anomaly(True)`.
|
||||
- `nccl_bandwidth_check`: scaffold that returns the expected upper-bound
|
||||
bandwidth for a (gpu_pair, link) tuple. Live measurement deferred — we
|
||||
expose the reference table so `soup doctor --nccl` can warn when measured
|
||||
perf is well below expectation.
|
||||
|
||||
Containment, redaction, and exception-narrowing follow v0.34.0 `crash.py` /
|
||||
v0.34.0 `profiling.py` policies.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import math
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Iterator, Mapping
|
||||
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
# Reference NVLink/PCIe bandwidth ceilings (GB/s, unidirectional).
|
||||
# Source: NVIDIA published topology specs for H100/A100/V100.
|
||||
_BANDWIDTH_REFERENCE: Mapping[tuple[str, str], float] = MappingProxyType({
|
||||
("h100", "nvlink"): 450.0, # NVLink 4 (18 links × 25 GB/s)
|
||||
("h100", "pcie"): 64.0, # PCIe Gen5 x16
|
||||
("a100", "nvlink"): 300.0, # NVLink 3 (12 links × 25 GB/s)
|
||||
("a100", "pcie"): 32.0, # PCIe Gen4 x16
|
||||
("v100", "nvlink"): 150.0, # NVLink 2 (6 links × 25 GB/s)
|
||||
("v100", "pcie"): 16.0, # PCIe Gen3 x16
|
||||
("rtx4090", "pcie"): 64.0,
|
||||
("rtx3090", "pcie"): 32.0,
|
||||
})
|
||||
|
||||
|
||||
def _validate_run_id(run_id: object) -> str:
|
||||
if not isinstance(run_id, str):
|
||||
raise ValueError("run_id must be a string")
|
||||
if not run_id:
|
||||
raise ValueError("run_id must not be empty")
|
||||
if "\x00" in run_id:
|
||||
raise ValueError("run_id must not contain null bytes")
|
||||
if run_id in {".", ".."}:
|
||||
raise ValueError(f"run_id contains forbidden token '{run_id}'")
|
||||
for ch in ("/", "\\"):
|
||||
if ch in run_id:
|
||||
raise ValueError(f"run_id must not contain path separator '{ch}'")
|
||||
return run_id
|
||||
|
||||
|
||||
def resolve_snapshot_path(run_id: str, *, base_dir: str = "profiles") -> str:
|
||||
"""Return realpath of `<cwd>/<base_dir>/<run_id>.snapshot.pickle`.
|
||||
|
||||
Rejects values that escape cwd. Mirrors v0.34.0 `resolve_trace_path`.
|
||||
"""
|
||||
rid = _validate_run_id(run_id)
|
||||
if not isinstance(base_dir, str) or not base_dir:
|
||||
raise ValueError("base_dir must be a non-empty string")
|
||||
if "\x00" in base_dir:
|
||||
raise ValueError("base_dir must not contain null bytes")
|
||||
# Reject path separators and `..` components in base_dir so an absolute
|
||||
# / parent-traversing path can never sneak through realpath on Windows
|
||||
# short-name systems (security review fix).
|
||||
if base_dir in {".", ".."}:
|
||||
raise ValueError("base_dir must not be '.' or '..'")
|
||||
if os.path.isabs(base_dir):
|
||||
raise ValueError("base_dir must be a relative path under cwd")
|
||||
parts = [p for p in base_dir.replace("\\", "/").split("/") if p]
|
||||
if any(p == ".." for p in parts):
|
||||
raise ValueError("base_dir must not contain '..' segments")
|
||||
target = os.path.realpath(os.path.join(base_dir, f"{rid}.snapshot.pickle"))
|
||||
if not is_under_cwd(target):
|
||||
raise ValueError("snapshot path must stay under cwd")
|
||||
return target
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def memory_snapshot_context(
|
||||
run_id: str,
|
||||
*,
|
||||
base_dir: str = "profiles",
|
||||
max_entries: int = 100_000,
|
||||
) -> Iterator[str | None]:
|
||||
"""Record CUDA memory history; on exit, dump pickle + stop recording.
|
||||
|
||||
Yields the snapshot path on success, or None when torch is missing /
|
||||
CUDA is unavailable / the recording API is missing. Never raises through
|
||||
the context exit when torch failures are missing-dep style — those map
|
||||
to a yielded None.
|
||||
"""
|
||||
if isinstance(max_entries, bool) or not isinstance(max_entries, int):
|
||||
raise ValueError("max_entries must be a positive int")
|
||||
if max_entries < 1 or max_entries > 10_000_000:
|
||||
raise ValueError("max_entries must be in [1, 10_000_000]")
|
||||
|
||||
path = resolve_snapshot_path(run_id, base_dir=base_dir)
|
||||
try:
|
||||
import torch # type: ignore[import-not-found]
|
||||
except ImportError:
|
||||
yield None
|
||||
return
|
||||
cuda = getattr(torch, "cuda", None)
|
||||
if cuda is None or not cuda.is_available():
|
||||
yield None
|
||||
return
|
||||
record = getattr(getattr(cuda, "memory", None), "_record_memory_history", None)
|
||||
dump = getattr(getattr(cuda, "memory", None), "_dump_snapshot", None)
|
||||
if record is None or dump is None:
|
||||
yield None
|
||||
return
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
# Narrow the RuntimeError catch to only the entry call so a user-body
|
||||
# RuntimeError cannot trigger a double-yield in this generator
|
||||
# (review fix — the previous wide `except RuntimeError` would have
|
||||
# raised "generator already executing" on user-body failures).
|
||||
try:
|
||||
record(max_entries=max_entries)
|
||||
except RuntimeError:
|
||||
yield None
|
||||
return
|
||||
try:
|
||||
yield path
|
||||
finally:
|
||||
try:
|
||||
dump(path)
|
||||
finally:
|
||||
try:
|
||||
record(enabled=None) # type: ignore[arg-type]
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def detect_anomaly_context() -> Iterator[bool]:
|
||||
"""torch.autograd.set_detect_anomaly(True) wrapper.
|
||||
|
||||
Yields True when activated, False when torch is missing.
|
||||
"""
|
||||
try:
|
||||
import torch # type: ignore[import-not-found]
|
||||
except ImportError:
|
||||
yield False
|
||||
return
|
||||
set_detect = getattr(getattr(torch, "autograd", None), "set_detect_anomaly", None)
|
||||
if set_detect is None:
|
||||
yield False
|
||||
return
|
||||
set_detect(True)
|
||||
try:
|
||||
yield True
|
||||
finally:
|
||||
set_detect(False)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BandwidthExpectation:
|
||||
gpu: str
|
||||
link: str
|
||||
expected_gb_per_sec: float
|
||||
|
||||
|
||||
def expected_bandwidth(gpu: str, link: str) -> float | None:
|
||||
"""Return reference bandwidth (GB/s) for `(gpu, link)`, or None."""
|
||||
if not isinstance(gpu, str) or not isinstance(link, str):
|
||||
return None
|
||||
return _BANDWIDTH_REFERENCE.get((gpu.lower(), link.lower()))
|
||||
|
||||
|
||||
def nccl_bandwidth_check(
|
||||
*, gpu: str, link: str, measured_gb_per_sec: float
|
||||
) -> dict:
|
||||
"""Compare measured bandwidth against the reference table.
|
||||
|
||||
Returns a dict with `expected`, `measured`, `ratio`, and `status`.
|
||||
|
||||
- status="OK" : ratio >= 0.80
|
||||
- status="MINOR" : 0.50 <= ratio < 0.80
|
||||
- status="MAJOR" : ratio < 0.50 (silent degradation likely)
|
||||
- status="UNKNOWN": no reference entry for (gpu, link)
|
||||
"""
|
||||
if (
|
||||
isinstance(measured_gb_per_sec, bool)
|
||||
or not isinstance(measured_gb_per_sec, (int, float))
|
||||
):
|
||||
raise ValueError("measured_gb_per_sec must be a number")
|
||||
if not math.isfinite(float(measured_gb_per_sec)):
|
||||
raise ValueError("measured_gb_per_sec must be finite")
|
||||
if measured_gb_per_sec < 0:
|
||||
raise ValueError("measured_gb_per_sec must be >= 0")
|
||||
|
||||
expected = expected_bandwidth(gpu, link)
|
||||
if expected is None or expected <= 0:
|
||||
return {
|
||||
"expected_gb_per_sec": None,
|
||||
"measured_gb_per_sec": float(measured_gb_per_sec),
|
||||
"ratio": None,
|
||||
"status": "UNKNOWN",
|
||||
}
|
||||
ratio = float(measured_gb_per_sec) / expected
|
||||
if ratio >= 0.80:
|
||||
status = "OK"
|
||||
elif ratio >= 0.50:
|
||||
status = "MINOR"
|
||||
else:
|
||||
status = "MAJOR"
|
||||
return {
|
||||
"expected_gb_per_sec": expected,
|
||||
"measured_gb_per_sec": float(measured_gb_per_sec),
|
||||
"ratio": round(ratio, 4),
|
||||
"status": status,
|
||||
}
|
||||
|
|
@ -0,0 +1,190 @@
|
|||
"""v0.43.0 Part A — Tracker integrations + PostHog telemetry opt-out.
|
||||
|
||||
Closed allowlist of HF Trainer `report_to` backends Soup recognises.
|
||||
Adds mlflow / swanlab / trackio to the legacy `wandb` / `tensorboard` / `none`
|
||||
set. Live integrations rely on HF Trainer's built-in callbacks (mlflow,
|
||||
swanlab) plus the third-party `trackio` callback when installed; Soup only
|
||||
validates the name and surfaces a friendly error when the backing package
|
||||
is missing.
|
||||
|
||||
Telemetry: opt-out via `SOUP_TELEMETRY=0` env var. Default is OFF until a
|
||||
public privacy policy ships — `is_telemetry_enabled` returns False unless
|
||||
the user explicitly enables it. Hardware-info-only payload schema lives in
|
||||
`build_telemetry_payload` for documentation/testing; no network calls in
|
||||
v0.43.0 (PostHog wire-up deferred to v0.43.1).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping
|
||||
|
||||
# Closed allowlist of report_to backends.
|
||||
_REPORT_TO_BACKENDS: Mapping[str, str | None] = MappingProxyType({
|
||||
"none": None,
|
||||
"wandb": "wandb",
|
||||
"tensorboard": "tensorboard",
|
||||
"mlflow": "mlflow",
|
||||
"swanlab": "swanlab",
|
||||
"trackio": "trackio",
|
||||
})
|
||||
|
||||
SUPPORTED_TRACKERS = frozenset(_REPORT_TO_BACKENDS.keys())
|
||||
|
||||
# v0.43.0 additions (HF-native wandb/tensorboard already supported).
|
||||
NEW_TRACKERS_V0_43 = frozenset({"mlflow", "swanlab", "trackio"})
|
||||
|
||||
_MAX_NAME_LEN = 32
|
||||
|
||||
|
||||
def validate_tracker_name(name: object) -> str:
|
||||
"""Validate and lowercase a `report_to` tracker name.
|
||||
|
||||
Returns the canonical lower-cased name. Raises ValueError on invalid
|
||||
input. Mirrors v0.41.0 `validate_optimizer_name` policy.
|
||||
"""
|
||||
if not isinstance(name, str):
|
||||
raise ValueError(f"tracker name must be a string, got {type(name).__name__}")
|
||||
if not name:
|
||||
raise ValueError("tracker name must not be empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("tracker name must not contain null bytes")
|
||||
if len(name) > _MAX_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"tracker name length {len(name)} exceeds max {_MAX_NAME_LEN}"
|
||||
)
|
||||
canonical = name.lower()
|
||||
if canonical not in SUPPORTED_TRACKERS:
|
||||
supported = ", ".join(sorted(SUPPORTED_TRACKERS))
|
||||
raise ValueError(
|
||||
f"unknown tracker '{name}'. Supported: {supported}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def required_tracker_package(name: str) -> str | None:
|
||||
"""Return the pip-installable package name for a tracker, or None.
|
||||
|
||||
Non-string input returns None (mirrors `is_new_v0_43_tracker`).
|
||||
"""
|
||||
if not isinstance(name, str):
|
||||
return None
|
||||
return _REPORT_TO_BACKENDS.get(name.lower())
|
||||
|
||||
|
||||
def is_new_v0_43_tracker(name: object) -> bool:
|
||||
"""True if the name is an additive v0.43.0 tracker, False otherwise."""
|
||||
if not isinstance(name, str):
|
||||
return False
|
||||
return name.lower() in NEW_TRACKERS_V0_43
|
||||
|
||||
|
||||
# --- Telemetry (opt-out, default OFF) ----------------------------------
|
||||
|
||||
_TELEMETRY_ENV_VAR = "SOUP_TELEMETRY"
|
||||
|
||||
|
||||
def is_telemetry_enabled(env: Mapping[str, str] | None = None) -> bool:
|
||||
"""Telemetry is opt-IN until v0.43.1 ships the network code.
|
||||
|
||||
The roadmap entry calls this opt-out, but until the privacy policy
|
||||
+ PostHog wire-up land we keep it default-OFF so no payload is built
|
||||
or sent. Users may enable explicitly with `SOUP_TELEMETRY=1`.
|
||||
"""
|
||||
source = env if env is not None else os.environ
|
||||
raw = source.get(_TELEMETRY_ENV_VAR)
|
||||
if raw is None:
|
||||
return False
|
||||
val = raw.strip().lower()
|
||||
if val in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def build_telemetry_payload(
|
||||
*,
|
||||
soup_version: str,
|
||||
command: str,
|
||||
duration_seconds: float | int | None = None,
|
||||
) -> dict:
|
||||
"""Build the hardware-info-only telemetry payload.
|
||||
|
||||
The payload contains NO user data, dataset paths, model names, or
|
||||
config contents. Documented schema:
|
||||
|
||||
- `soup_version`: caller-supplied
|
||||
- `command`: top-level CLI command (e.g. `train`, `data ingest`)
|
||||
- `python`: major.minor only
|
||||
- `os`: platform.system()
|
||||
- `arch`: platform.machine()
|
||||
- `duration_seconds`: optional, finite float / int / None
|
||||
|
||||
Raises ValueError for non-string `command` / `soup_version` and for
|
||||
non-finite `duration_seconds`.
|
||||
"""
|
||||
if not isinstance(soup_version, str) or not soup_version:
|
||||
raise ValueError("soup_version must be a non-empty string")
|
||||
if "\x00" in soup_version:
|
||||
raise ValueError("soup_version must not contain null bytes")
|
||||
if not isinstance(command, str) or not command:
|
||||
raise ValueError("command must be a non-empty string")
|
||||
if "\x00" in command:
|
||||
raise ValueError("command must not contain null bytes")
|
||||
if duration_seconds is not None:
|
||||
# bool is a subclass of int — reject explicitly (project policy)
|
||||
if isinstance(duration_seconds, bool) or not isinstance(
|
||||
duration_seconds, (int, float)
|
||||
):
|
||||
raise ValueError("duration_seconds must be int / float / None")
|
||||
if not math.isfinite(float(duration_seconds)):
|
||||
raise ValueError("duration_seconds must be finite")
|
||||
if duration_seconds < 0:
|
||||
raise ValueError("duration_seconds must be >= 0")
|
||||
|
||||
py = platform.python_version_tuple()
|
||||
py_major_minor = f"{py[0]}.{py[1]}"
|
||||
return {
|
||||
"soup_version": soup_version,
|
||||
"command": command,
|
||||
"python": py_major_minor,
|
||||
"os": platform.system(),
|
||||
"arch": platform.machine(),
|
||||
"duration_seconds": (
|
||||
float(duration_seconds) if duration_seconds is not None else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def resolve_report_to(
|
||||
*,
|
||||
wandb: bool = False,
|
||||
tensorboard: bool = False,
|
||||
tracker: str | None = None,
|
||||
) -> str:
|
||||
"""Resolve the HF Trainer `report_to` value from CLI flags + --tracker.
|
||||
|
||||
Mutual-exclusion: only one of (wandb, tensorboard, tracker) may be set.
|
||||
Empty string / None on `tracker` is treated as unset.
|
||||
"""
|
||||
set_count = sum(
|
||||
1
|
||||
for x in (
|
||||
bool(wandb),
|
||||
bool(tensorboard),
|
||||
bool(tracker) if isinstance(tracker, str) and tracker else False,
|
||||
)
|
||||
if x
|
||||
)
|
||||
if set_count > 1:
|
||||
raise ValueError(
|
||||
"--wandb, --tensorboard, and --tracker are mutually exclusive"
|
||||
)
|
||||
if wandb:
|
||||
return "wandb"
|
||||
if tensorboard:
|
||||
return "tensorboard"
|
||||
if tracker:
|
||||
return validate_tracker_name(tracker)
|
||||
return "none"
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
"""v0.43.0 Part C — `soup doctor --vscode` writer for `.vscode/launch.json`.
|
||||
|
||||
Generates a minimal but useful Python debug config for `soup train` plus a
|
||||
pytest config. Path containment via shared `is_under_cwd`; refuses to
|
||||
overwrite an existing launch.json without `force=True` (matches v0.40.2
|
||||
register_data policy).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
|
||||
def build_launch_json(*, config_path: str = "soup.yaml") -> dict:
|
||||
"""Build the launch.json contents.
|
||||
|
||||
`config_path` is the YAML config to pass to `soup train --config`.
|
||||
Validates the path so a crafted argument cannot inject arbitrary
|
||||
args into the generated JSON.
|
||||
"""
|
||||
if not isinstance(config_path, str) or not config_path:
|
||||
raise ValueError("config_path must be a non-empty string")
|
||||
if "\x00" in config_path:
|
||||
raise ValueError("config_path must not contain null bytes")
|
||||
if "\n" in config_path or "\r" in config_path:
|
||||
raise ValueError("config_path must not contain newlines")
|
||||
if len(config_path) > 512:
|
||||
raise ValueError("config_path too long (>512 chars)")
|
||||
return {
|
||||
"version": "0.2.0",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "soup train",
|
||||
"type": "python",
|
||||
"request": "launch",
|
||||
"module": "soup_cli.cli",
|
||||
"args": ["train", "--config", config_path],
|
||||
"console": "integratedTerminal",
|
||||
"justMyCode": False,
|
||||
"env": {"PYTHONUTF8": "1"},
|
||||
},
|
||||
{
|
||||
"name": "pytest (current file)",
|
||||
"type": "python",
|
||||
"request": "launch",
|
||||
"module": "pytest",
|
||||
"args": ["${file}", "-v", "--no-cov"],
|
||||
"console": "integratedTerminal",
|
||||
"justMyCode": False,
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def write_vscode_launch(
|
||||
*,
|
||||
config_path: str = "soup.yaml",
|
||||
target_dir: str = ".vscode",
|
||||
force: bool = False,
|
||||
) -> str:
|
||||
"""Write `<cwd>/<target_dir>/launch.json`.
|
||||
|
||||
Returns the absolute path written.
|
||||
Raises ValueError on containment violation.
|
||||
Raises FileExistsError when launch.json already exists and force=False.
|
||||
"""
|
||||
if not isinstance(target_dir, str) or not target_dir:
|
||||
raise ValueError("target_dir must be a non-empty string")
|
||||
if "\x00" in target_dir:
|
||||
raise ValueError("target_dir must not contain null bytes")
|
||||
if not isinstance(force, bool):
|
||||
raise ValueError("force must be a bool")
|
||||
|
||||
real = os.path.realpath(target_dir)
|
||||
if not is_under_cwd(real):
|
||||
raise ValueError("target_dir must stay under cwd")
|
||||
|
||||
payload = build_launch_json(config_path=config_path)
|
||||
os.makedirs(real, exist_ok=True)
|
||||
out_path = os.path.join(real, "launch.json")
|
||||
# TOCTOU defence: reject symlinks at the target path regardless of
|
||||
# `force` (mirrors v0.33.0 #22 prune_checkpoints policy). Without this
|
||||
# guard, force=True would follow a pre-placed symlink and overwrite a
|
||||
# file outside cwd.
|
||||
try:
|
||||
st = os.lstat(out_path)
|
||||
except FileNotFoundError:
|
||||
st = None
|
||||
if st is not None:
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(
|
||||
"launch.json target is a symlink; aborting"
|
||||
)
|
||||
if not force:
|
||||
raise FileExistsError(
|
||||
f"{out_path} already exists; pass force=True to overwrite"
|
||||
)
|
||||
with open(out_path, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, indent=2, ensure_ascii=False)
|
||||
f.write("\n")
|
||||
return out_path
|
||||
|
|
@ -0,0 +1,264 @@
|
|||
"""Tests for v0.43.0 Part A — Tracker integrations + PostHog opt-out."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from soup_cli.utils.trackers import (
|
||||
NEW_TRACKERS_V0_43,
|
||||
SUPPORTED_TRACKERS,
|
||||
build_telemetry_payload,
|
||||
is_new_v0_43_tracker,
|
||||
is_telemetry_enabled,
|
||||
required_tracker_package,
|
||||
resolve_report_to,
|
||||
validate_tracker_name,
|
||||
)
|
||||
|
||||
|
||||
class TestSupportedTrackers:
|
||||
def test_includes_legacy(self):
|
||||
assert "wandb" in SUPPORTED_TRACKERS
|
||||
assert "tensorboard" in SUPPORTED_TRACKERS
|
||||
assert "none" in SUPPORTED_TRACKERS
|
||||
|
||||
def test_includes_v043_additions(self):
|
||||
for name in ("mlflow", "swanlab", "trackio"):
|
||||
assert name in SUPPORTED_TRACKERS
|
||||
|
||||
def test_immutable(self):
|
||||
# frozenset cannot be mutated
|
||||
with pytest.raises(AttributeError):
|
||||
SUPPORTED_TRACKERS.add("evil") # type: ignore[attr-defined]
|
||||
|
||||
|
||||
class TestValidateTrackerName:
|
||||
@pytest.mark.parametrize(
|
||||
"name", ["mlflow", "swanlab", "trackio", "wandb", "tensorboard", "none"]
|
||||
)
|
||||
def test_happy(self, name):
|
||||
assert validate_tracker_name(name) == name
|
||||
|
||||
def test_case_insensitive(self):
|
||||
assert validate_tracker_name("MLflow") == "mlflow"
|
||||
assert validate_tracker_name("SWANLAB") == "swanlab"
|
||||
|
||||
def test_unknown(self):
|
||||
with pytest.raises(ValueError, match="unknown tracker"):
|
||||
validate_tracker_name("comet")
|
||||
|
||||
def test_non_string(self):
|
||||
with pytest.raises(ValueError, match="must be a string"):
|
||||
validate_tracker_name(123) # type: ignore[arg-type]
|
||||
|
||||
def test_empty(self):
|
||||
with pytest.raises(ValueError, match="must not be empty"):
|
||||
validate_tracker_name("")
|
||||
|
||||
def test_null_byte(self):
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
validate_tracker_name("mlflow\x00")
|
||||
|
||||
def test_oversize(self):
|
||||
with pytest.raises(ValueError, match="exceeds max"):
|
||||
validate_tracker_name("a" * 64)
|
||||
|
||||
|
||||
class TestRequiredTrackerPackage:
|
||||
def test_known_packages(self):
|
||||
assert required_tracker_package("mlflow") == "mlflow"
|
||||
assert required_tracker_package("swanlab") == "swanlab"
|
||||
assert required_tracker_package("trackio") == "trackio"
|
||||
assert required_tracker_package("wandb") == "wandb"
|
||||
|
||||
def test_none_means_no_install(self):
|
||||
assert required_tracker_package("none") is None
|
||||
|
||||
def test_unknown_returns_none(self):
|
||||
assert required_tracker_package("comet") is None
|
||||
|
||||
def test_non_string_returns_none(self):
|
||||
assert required_tracker_package(None) is None # type: ignore[arg-type]
|
||||
assert required_tracker_package(123) is None # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestIsNewV043Tracker:
|
||||
@pytest.mark.parametrize("name", list(NEW_TRACKERS_V0_43))
|
||||
def test_true_for_v043(self, name):
|
||||
assert is_new_v0_43_tracker(name) is True
|
||||
|
||||
@pytest.mark.parametrize("name", ["wandb", "tensorboard", "none"])
|
||||
def test_false_for_legacy(self, name):
|
||||
assert is_new_v0_43_tracker(name) is False
|
||||
|
||||
def test_false_for_unknown(self):
|
||||
assert is_new_v0_43_tracker("comet") is False
|
||||
|
||||
def test_non_string_safe(self):
|
||||
assert is_new_v0_43_tracker(None) is False # type: ignore[arg-type]
|
||||
assert is_new_v0_43_tracker(123) is False # type: ignore[arg-type]
|
||||
|
||||
def test_case_insensitive(self):
|
||||
assert is_new_v0_43_tracker("MLflow") is True
|
||||
|
||||
|
||||
class TestIsTelemetryEnabled:
|
||||
def test_default_off(self):
|
||||
assert is_telemetry_enabled({}) is False
|
||||
|
||||
@pytest.mark.parametrize("val", ["1", "true", "yes", "on", "TRUE", "Yes "])
|
||||
def test_explicit_on(self, val):
|
||||
assert is_telemetry_enabled({"SOUP_TELEMETRY": val}) is True
|
||||
|
||||
@pytest.mark.parametrize("val", ["0", "false", "no", "off", "", "garbage"])
|
||||
def test_explicit_off_or_garbage(self, val):
|
||||
assert is_telemetry_enabled({"SOUP_TELEMETRY": val}) is False
|
||||
|
||||
|
||||
class TestBuildTelemetryPayload:
|
||||
def test_required_fields(self):
|
||||
payload = build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train"
|
||||
)
|
||||
assert payload["soup_version"] == "0.43.0"
|
||||
assert payload["command"] == "train"
|
||||
assert "python" in payload
|
||||
assert "os" in payload
|
||||
assert "arch" in payload
|
||||
assert payload["duration_seconds"] is None
|
||||
|
||||
def test_with_duration(self):
|
||||
payload = build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train", duration_seconds=12.5
|
||||
)
|
||||
assert payload["duration_seconds"] == 12.5
|
||||
|
||||
def test_int_duration_coerced_to_float(self):
|
||||
payload = build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train", duration_seconds=10
|
||||
)
|
||||
assert payload["duration_seconds"] == 10.0
|
||||
assert isinstance(payload["duration_seconds"], float)
|
||||
|
||||
def test_no_user_data_in_payload(self):
|
||||
# Schema invariant: no model name / dataset path / user identifier.
|
||||
payload = build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train"
|
||||
)
|
||||
# Closed key set:
|
||||
assert set(payload.keys()) == {
|
||||
"soup_version",
|
||||
"command",
|
||||
"python",
|
||||
"os",
|
||||
"arch",
|
||||
"duration_seconds",
|
||||
}
|
||||
|
||||
def test_invalid_version(self):
|
||||
with pytest.raises(ValueError):
|
||||
build_telemetry_payload(soup_version="", command="train")
|
||||
with pytest.raises(ValueError):
|
||||
build_telemetry_payload(soup_version=None, command="train") # type: ignore[arg-type]
|
||||
|
||||
def test_null_byte_version(self):
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
build_telemetry_payload(soup_version="1.0\x00", command="train")
|
||||
|
||||
def test_invalid_command(self):
|
||||
with pytest.raises(ValueError):
|
||||
build_telemetry_payload(soup_version="0.43.0", command="")
|
||||
|
||||
def test_null_byte_command(self):
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
build_telemetry_payload(soup_version="0.43.0", command="train\x00")
|
||||
|
||||
def test_bool_duration_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train", duration_seconds=True # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def test_nonfinite_duration_rejected(self):
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train", duration_seconds=float("inf")
|
||||
)
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
build_telemetry_payload(
|
||||
soup_version="0.43.0",
|
||||
command="train",
|
||||
duration_seconds=float("nan"),
|
||||
)
|
||||
|
||||
def test_negative_duration_rejected(self):
|
||||
with pytest.raises(ValueError, match=">= 0"):
|
||||
build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train", duration_seconds=-1
|
||||
)
|
||||
|
||||
def test_python_is_major_minor_only(self):
|
||||
payload = build_telemetry_payload(
|
||||
soup_version="0.43.0", command="train"
|
||||
)
|
||||
# Should be like "3.11" not "3.11.7"
|
||||
assert payload["python"].count(".") == 1
|
||||
|
||||
|
||||
class TestResolveReportTo:
|
||||
def test_default_none(self):
|
||||
assert resolve_report_to() == "none"
|
||||
|
||||
def test_wandb(self):
|
||||
assert resolve_report_to(wandb=True) == "wandb"
|
||||
|
||||
def test_tensorboard(self):
|
||||
assert resolve_report_to(tensorboard=True) == "tensorboard"
|
||||
|
||||
def test_tracker_mlflow(self):
|
||||
assert resolve_report_to(tracker="mlflow") == "mlflow"
|
||||
|
||||
def test_tracker_swanlab(self):
|
||||
assert resolve_report_to(tracker="swanlab") == "swanlab"
|
||||
|
||||
def test_tracker_trackio(self):
|
||||
assert resolve_report_to(tracker="trackio") == "trackio"
|
||||
|
||||
def test_tracker_unknown(self):
|
||||
with pytest.raises(ValueError, match="unknown tracker"):
|
||||
resolve_report_to(tracker="comet")
|
||||
|
||||
def test_mutually_exclusive_wandb_tensorboard(self):
|
||||
with pytest.raises(ValueError, match="mutually exclusive"):
|
||||
resolve_report_to(wandb=True, tensorboard=True)
|
||||
|
||||
def test_mutually_exclusive_wandb_tracker(self):
|
||||
with pytest.raises(ValueError, match="mutually exclusive"):
|
||||
resolve_report_to(wandb=True, tracker="mlflow")
|
||||
|
||||
def test_mutually_exclusive_tensorboard_tracker(self):
|
||||
with pytest.raises(ValueError, match="mutually exclusive"):
|
||||
resolve_report_to(tensorboard=True, tracker="swanlab")
|
||||
|
||||
def test_empty_tracker_treated_as_unset(self):
|
||||
assert resolve_report_to(tracker="") == "none"
|
||||
assert resolve_report_to(tracker=None) == "none"
|
||||
|
||||
def test_tracker_canonicalised(self):
|
||||
assert resolve_report_to(tracker="MLflow") == "mlflow"
|
||||
|
||||
def test_tracker_none_string(self):
|
||||
# The literal "none" is a valid value of the allowlist.
|
||||
assert resolve_report_to(tracker="none") == "none"
|
||||
|
||||
|
||||
class TestRegistryImmutability:
|
||||
def test_report_to_backends_immutable(self):
|
||||
from soup_cli.utils.trackers import _REPORT_TO_BACKENDS
|
||||
with pytest.raises(TypeError):
|
||||
_REPORT_TO_BACKENDS["evil"] = "evil" # type: ignore[index]
|
||||
|
||||
def test_telemetry_none_env_uses_os_environ(self, monkeypatch):
|
||||
monkeypatch.setenv("SOUP_TELEMETRY", "1")
|
||||
assert is_telemetry_enabled(None) is True
|
||||
monkeypatch.delenv("SOUP_TELEMETRY", raising=False)
|
||||
assert is_telemetry_enabled(None) is False
|
||||
|
|
@ -0,0 +1,492 @@
|
|||
"""Tests for v0.43.0 Part B — Eval metrics."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from soup_cli.eval.arena import (
|
||||
DEFAULT_BASE_RATING,
|
||||
Tournament,
|
||||
expected_score,
|
||||
update_elo,
|
||||
)
|
||||
from soup_cli.eval.benchmarks_v0_43 import (
|
||||
NEW_BENCHMARKS_V0_43,
|
||||
benchmark_metadata,
|
||||
is_v0_43_benchmark,
|
||||
lm_eval_task_for,
|
||||
)
|
||||
from soup_cli.eval.calibrate import (
|
||||
CalibrationReport,
|
||||
classify_kl_delta,
|
||||
kl_divergence,
|
||||
run_calibration,
|
||||
)
|
||||
from soup_cli.utils.nlg_metrics import (
|
||||
NLG_METRICS,
|
||||
bleu_score,
|
||||
compute_nlg_metric,
|
||||
effective_tokens_per_second,
|
||||
rouge_l_score,
|
||||
rouge_n_score,
|
||||
)
|
||||
|
||||
# ----------------- BLEU / ROUGE -----------------
|
||||
|
||||
class TestBleuScore:
|
||||
def test_perfect_match(self):
|
||||
score = bleu_score(["the cat sat on the mat"], ["the cat sat on the mat"])
|
||||
assert score == pytest.approx(1.0, abs=1e-6)
|
||||
|
||||
def test_zero_overlap_no_smoothing(self):
|
||||
# Without smoothing, any zero n-gram precision -> BLEU 0.
|
||||
score = bleu_score(
|
||||
["alpha beta gamma delta"],
|
||||
["one two three four"],
|
||||
smooth=False,
|
||||
)
|
||||
assert score == 0.0
|
||||
|
||||
def test_zero_overlap_smoothed(self):
|
||||
# With smoothing (the default), Chen & Cherry assigns small mass
|
||||
# to each zero bucket, so the score is small but positive.
|
||||
score = bleu_score(["alpha beta gamma delta"], ["one two three four"])
|
||||
assert 0.0 < score < 0.2
|
||||
|
||||
def test_brevity_penalty_one_when_pred_longer(self):
|
||||
# Prediction longer than reference: BP = 1.0 (no penalty).
|
||||
# Score reflects modified precision only.
|
||||
# pred unigrams the=2,cat=1,sat=1,on=1,the=2,mat=1 (6 total);
|
||||
# ref the=1,cat=1,sat=1.
|
||||
# min-clipped overlap = the(min(2,1)) + cat + sat = 3, total=6 → 0.5
|
||||
score = bleu_score(
|
||||
["the cat sat on the mat"], ["the cat sat"], max_n=1
|
||||
)
|
||||
assert score == pytest.approx(0.5)
|
||||
|
||||
def test_partial_overlap(self):
|
||||
# BLEU-2 partial: shorter prediction with all 4 unigram + 3 bigram
|
||||
# overlaps — guarantees nonzero standard BLEU.
|
||||
score = bleu_score(
|
||||
["the quick brown fox"],
|
||||
["the quick brown fox jumped"],
|
||||
max_n=2,
|
||||
)
|
||||
assert 0.0 < score < 1.0
|
||||
|
||||
def test_empty_corpus(self):
|
||||
assert bleu_score([], []) == 0.0
|
||||
|
||||
def test_length_mismatch(self):
|
||||
with pytest.raises(ValueError, match="same length"):
|
||||
bleu_score(["a"], ["a", "b"])
|
||||
|
||||
def test_invalid_max_n(self):
|
||||
with pytest.raises(ValueError):
|
||||
bleu_score(["a"], ["a"], max_n=0)
|
||||
with pytest.raises(ValueError):
|
||||
bleu_score(["a"], ["a"], max_n=10)
|
||||
with pytest.raises(ValueError):
|
||||
bleu_score(["a"], ["a"], max_n=True) # type: ignore[arg-type]
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
bleu_score(["a\x00"], ["a"])
|
||||
|
||||
|
||||
class TestRougeNScore:
|
||||
def test_perfect_match(self):
|
||||
score = rouge_n_score(["alpha beta gamma"], ["alpha beta gamma"])
|
||||
assert score == pytest.approx(1.0)
|
||||
|
||||
def test_zero_overlap(self):
|
||||
score = rouge_n_score(["alpha beta"], ["one two"])
|
||||
assert score == 0.0
|
||||
|
||||
def test_n_2(self):
|
||||
score = rouge_n_score(["the cat sat"], ["the cat sat"], n=2)
|
||||
assert score == pytest.approx(1.0)
|
||||
|
||||
def test_invalid_n(self):
|
||||
with pytest.raises(ValueError):
|
||||
rouge_n_score(["a"], ["a"], n=0)
|
||||
with pytest.raises(ValueError):
|
||||
rouge_n_score(["a"], ["a"], n=True) # type: ignore[arg-type]
|
||||
|
||||
def test_too_short_for_n(self):
|
||||
# Single-token strings have no bigrams.
|
||||
score = rouge_n_score(["a"], ["a"], n=2)
|
||||
assert score == 0.0
|
||||
|
||||
def test_length_mismatch_message(self):
|
||||
with pytest.raises(ValueError, match="same length"):
|
||||
rouge_n_score(["a"], ["a", "b"])
|
||||
|
||||
|
||||
class TestRougeLScore:
|
||||
def test_perfect_match(self):
|
||||
score = rouge_l_score(["alpha beta gamma"], ["alpha beta gamma"])
|
||||
assert score == pytest.approx(1.0)
|
||||
|
||||
def test_lcs_partial(self):
|
||||
score = rouge_l_score(
|
||||
["the quick brown fox"],
|
||||
["a quick brown dog"],
|
||||
)
|
||||
# LCS = "quick brown" (2 tokens)
|
||||
# P = 2/4, R = 2/4 (4 tokens each), F1 = 0.5
|
||||
assert score == pytest.approx(0.5)
|
||||
|
||||
def test_zero_overlap(self):
|
||||
score = rouge_l_score(["alpha"], ["beta"])
|
||||
assert score == 0.0
|
||||
|
||||
def test_empty(self):
|
||||
assert rouge_l_score([], []) == 0.0
|
||||
|
||||
def test_length_mismatch(self):
|
||||
with pytest.raises(ValueError, match="same length"):
|
||||
rouge_l_score(["a"], [])
|
||||
|
||||
|
||||
class TestComputeNlgMetric:
|
||||
@pytest.mark.parametrize("metric", list(NLG_METRICS))
|
||||
def test_dispatch(self, metric):
|
||||
score = compute_nlg_metric(metric, ["a b c"], ["a b c"])
|
||||
assert 0.0 <= score <= 1.0
|
||||
|
||||
def test_unknown(self):
|
||||
with pytest.raises(ValueError, match="unknown nlg metric"):
|
||||
compute_nlg_metric("meteor", ["a"], ["a"])
|
||||
|
||||
def test_non_string(self):
|
||||
with pytest.raises(ValueError):
|
||||
compute_nlg_metric(None, ["a"], ["a"]) # type: ignore[arg-type]
|
||||
|
||||
def test_case_insensitive(self):
|
||||
assert compute_nlg_metric("BLEU", ["a"], ["a"]) >= 0.0
|
||||
|
||||
|
||||
# ----------------- effective_tokens_per_second -----------------
|
||||
|
||||
class TestEffectiveTokensPerSecond:
|
||||
def test_happy(self):
|
||||
assert effective_tokens_per_second(
|
||||
unmasked_tokens=10000, wall_clock_seconds=10.0
|
||||
) == 1000.0
|
||||
|
||||
def test_zero_wall_clock_returns_none(self):
|
||||
assert effective_tokens_per_second(
|
||||
unmasked_tokens=100, wall_clock_seconds=0.0
|
||||
) is None
|
||||
|
||||
def test_negative_wall_clock_returns_none(self):
|
||||
assert effective_tokens_per_second(
|
||||
unmasked_tokens=100, wall_clock_seconds=-1.0
|
||||
) is None
|
||||
|
||||
def test_negative_tokens_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
effective_tokens_per_second(
|
||||
unmasked_tokens=-1, wall_clock_seconds=1.0
|
||||
)
|
||||
|
||||
def test_bool_tokens_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
effective_tokens_per_second(
|
||||
unmasked_tokens=True, wall_clock_seconds=1.0 # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def test_bool_wall_clock_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
effective_tokens_per_second(
|
||||
unmasked_tokens=100, wall_clock_seconds=True # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def test_nonfinite_wall_clock_rejected(self):
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
effective_tokens_per_second(
|
||||
unmasked_tokens=100, wall_clock_seconds=float("inf")
|
||||
)
|
||||
|
||||
|
||||
# ----------------- KL Calibration -----------------
|
||||
|
||||
class TestKlDivergence:
|
||||
def test_identical_distributions(self):
|
||||
kl = kl_divergence([1.0, 2.0, 3.0], [1.0, 2.0, 3.0])
|
||||
assert kl == pytest.approx(0.0, abs=1e-9)
|
||||
|
||||
def test_different_distributions_positive(self):
|
||||
kl = kl_divergence([1.0, 2.0, 3.0], [3.0, 2.0, 1.0])
|
||||
assert kl > 0.0
|
||||
|
||||
def test_length_mismatch(self):
|
||||
with pytest.raises(ValueError, match="same length"):
|
||||
kl_divergence([1.0, 2.0], [1.0])
|
||||
|
||||
def test_empty(self):
|
||||
with pytest.raises(ValueError):
|
||||
kl_divergence([], [])
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
kl_divergence([1.0, float("inf")], [1.0, 2.0])
|
||||
|
||||
def test_bool_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
kl_divergence([True, False], [1.0, 2.0]) # type: ignore[list-item]
|
||||
|
||||
|
||||
class TestClassifyKlDelta:
|
||||
@pytest.mark.parametrize(
|
||||
"kl,status",
|
||||
[
|
||||
(0.0, "OK"),
|
||||
(0.04, "OK"),
|
||||
(0.05, "MINOR"),
|
||||
(0.10, "MINOR"),
|
||||
(0.19, "MINOR"),
|
||||
(0.20, "MAJOR"),
|
||||
(1.0, "MAJOR"),
|
||||
],
|
||||
)
|
||||
def test_thresholds(self, kl, status):
|
||||
assert classify_kl_delta(kl) == status
|
||||
|
||||
def test_negative_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
classify_kl_delta(-0.01)
|
||||
|
||||
def test_nan_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
classify_kl_delta(float("nan"))
|
||||
|
||||
def test_bool_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
classify_kl_delta(True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestRunCalibration:
|
||||
def test_perfect_match_ok(self):
|
||||
baseline = [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]
|
||||
quant = [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]
|
||||
report = run_calibration(baseline, quant)
|
||||
assert isinstance(report, CalibrationReport)
|
||||
assert report.mean_kl == pytest.approx(0.0, abs=1e-9)
|
||||
assert report.delta_status == "OK"
|
||||
assert report.num_prompts == 2
|
||||
|
||||
def test_diverged_quant_major(self):
|
||||
baseline = [[10.0, 0.0, 0.0]]
|
||||
quant = [[0.0, 0.0, 10.0]]
|
||||
report = run_calibration(baseline, quant)
|
||||
# Large divergence — guaranteed MAJOR.
|
||||
assert report.delta_status == "MAJOR"
|
||||
|
||||
def test_length_mismatch(self):
|
||||
with pytest.raises(ValueError, match="same length"):
|
||||
run_calibration([[1.0]], [])
|
||||
|
||||
def test_empty(self):
|
||||
with pytest.raises(ValueError):
|
||||
run_calibration([], [])
|
||||
|
||||
def test_too_many_prompts(self):
|
||||
big = [[1.0]] * 10_001
|
||||
with pytest.raises(ValueError, match="too many"):
|
||||
run_calibration(big, big)
|
||||
|
||||
def test_report_frozen(self):
|
||||
from dataclasses import FrozenInstanceError
|
||||
|
||||
report = run_calibration([[1.0, 2.0]], [[1.0, 2.0]])
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
report.mean_kl = 0.5 # type: ignore[misc]
|
||||
|
||||
|
||||
# ----------------- Arena -----------------
|
||||
|
||||
class TestExpectedScore:
|
||||
def test_equal_ratings_50_50(self):
|
||||
assert expected_score(1500.0, 1500.0) == pytest.approx(0.5)
|
||||
|
||||
def test_higher_rated_favored(self):
|
||||
assert expected_score(1700.0, 1500.0) > 0.5
|
||||
|
||||
def test_lower_rated_disfavored(self):
|
||||
assert expected_score(1300.0, 1500.0) < 0.5
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
expected_score(float("nan"), 1500.0)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
expected_score(True, 1500.0) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestUpdateElo:
|
||||
def test_winner_gains(self):
|
||||
new_a, new_b = update_elo(1500.0, 1500.0, score_a=1.0)
|
||||
assert new_a > 1500.0
|
||||
assert new_b < 1500.0
|
||||
|
||||
def test_draw_no_change_at_equal(self):
|
||||
new_a, new_b = update_elo(1500.0, 1500.0, score_a=0.5)
|
||||
assert new_a == pytest.approx(1500.0)
|
||||
assert new_b == pytest.approx(1500.0)
|
||||
|
||||
def test_zero_sum(self):
|
||||
new_a, new_b = update_elo(1500.0, 1700.0, score_a=1.0)
|
||||
# Symmetric Elo: total rating mass conserved.
|
||||
assert (new_a - 1500.0) == pytest.approx(-(new_b - 1700.0), abs=1e-6)
|
||||
|
||||
def test_invalid_score_a(self):
|
||||
with pytest.raises(ValueError):
|
||||
update_elo(1500.0, 1500.0, score_a=-0.1)
|
||||
with pytest.raises(ValueError):
|
||||
update_elo(1500.0, 1500.0, score_a=1.1)
|
||||
with pytest.raises(ValueError):
|
||||
update_elo(1500.0, 1500.0, score_a=True) # type: ignore[arg-type]
|
||||
|
||||
def test_invalid_k(self):
|
||||
with pytest.raises(ValueError):
|
||||
update_elo(1500.0, 1500.0, score_a=1.0, k=0)
|
||||
with pytest.raises(ValueError):
|
||||
update_elo(1500.0, 1500.0, score_a=1.0, k=-32)
|
||||
|
||||
|
||||
class TestTournament:
|
||||
def test_register_and_record(self):
|
||||
t = Tournament()
|
||||
t.register("alpha")
|
||||
t.register("beta")
|
||||
t.record("alpha", "beta", winner="a")
|
||||
assert t.ratings["alpha"] > DEFAULT_BASE_RATING
|
||||
assert t.ratings["beta"] < DEFAULT_BASE_RATING
|
||||
|
||||
def test_register_idempotent(self):
|
||||
t = Tournament()
|
||||
t.register("alpha")
|
||||
t.register("alpha")
|
||||
assert t.ratings["alpha"] == DEFAULT_BASE_RATING
|
||||
|
||||
def test_ratings_immutable_view(self):
|
||||
t = Tournament()
|
||||
t.register("alpha")
|
||||
# Ratings property must not allow mutation through the returned view.
|
||||
with pytest.raises(TypeError):
|
||||
t.ratings["alpha"] = 9999.0 # type: ignore[index]
|
||||
|
||||
def test_implicit_register_on_record(self):
|
||||
t = Tournament()
|
||||
t.record("alpha", "beta", winner="draw")
|
||||
assert "alpha" in t.ratings
|
||||
assert "beta" in t.ratings
|
||||
|
||||
def test_self_play_rejected(self):
|
||||
t = Tournament()
|
||||
with pytest.raises(ValueError, match="must differ"):
|
||||
t.record("alpha", "alpha", winner="a")
|
||||
|
||||
def test_invalid_winner(self):
|
||||
t = Tournament()
|
||||
with pytest.raises(ValueError, match="winner"):
|
||||
t.record("a", "b", winner="c")
|
||||
|
||||
def test_empty_name_rejected(self):
|
||||
t = Tournament()
|
||||
with pytest.raises(ValueError):
|
||||
t.register("")
|
||||
|
||||
def test_null_byte_name_rejected(self):
|
||||
t = Tournament()
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
t.register("foo\x00")
|
||||
|
||||
def test_oversize_name_rejected(self):
|
||||
t = Tournament()
|
||||
with pytest.raises(ValueError):
|
||||
t.register("a" * 256)
|
||||
|
||||
def test_rich_markup_metacharacter_rejected(self):
|
||||
t = Tournament()
|
||||
with pytest.raises(ValueError, match="markup"):
|
||||
t.register("[red]evil[/red]")
|
||||
with pytest.raises(ValueError, match="markup"):
|
||||
t.register("foo]bar")
|
||||
|
||||
def test_model_cap_exceeded(self):
|
||||
from soup_cli.eval.arena import _MAX_MODELS
|
||||
t = Tournament()
|
||||
for i in range(_MAX_MODELS):
|
||||
t.register(f"model_{i}")
|
||||
with pytest.raises(ValueError, match="model cap"):
|
||||
t.register("one_too_many")
|
||||
|
||||
def test_invalid_k_nan(self):
|
||||
with pytest.raises(ValueError):
|
||||
update_elo(1500.0, 1500.0, score_a=1.0, k=float("nan"))
|
||||
|
||||
def test_invalid_base_rating(self):
|
||||
with pytest.raises(ValueError):
|
||||
Tournament(base_rating=float("nan"))
|
||||
|
||||
def test_invalid_k(self):
|
||||
with pytest.raises(ValueError):
|
||||
Tournament(k=0)
|
||||
|
||||
def test_leaderboard_sorted(self):
|
||||
t = Tournament()
|
||||
t.record("alpha", "beta", winner="a")
|
||||
t.record("alpha", "gamma", winner="a")
|
||||
board = t.leaderboard()
|
||||
assert board[0]["model"] == "alpha"
|
||||
assert board[0]["wins"] == 2
|
||||
assert board[0]["losses"] == 0
|
||||
# Highest rating first.
|
||||
for i in range(len(board) - 1):
|
||||
assert board[i]["rating"] >= board[i + 1]["rating"]
|
||||
|
||||
def test_draw_records(self):
|
||||
t = Tournament()
|
||||
t.record("alpha", "beta", winner="draw")
|
||||
for name in ("alpha", "beta"):
|
||||
row = next(r for r in t.leaderboard() if r["model"] == name)
|
||||
assert row["draws"] == 1
|
||||
assert row["wins"] == 0
|
||||
assert row["losses"] == 0
|
||||
|
||||
|
||||
# ----------------- Benchmarks v0.43 -----------------
|
||||
|
||||
class TestBenchmarksV043:
|
||||
@pytest.mark.parametrize("name", list(NEW_BENCHMARKS_V0_43))
|
||||
def test_recognised(self, name):
|
||||
assert is_v0_43_benchmark(name) is True
|
||||
meta = benchmark_metadata(name)
|
||||
assert meta is not None
|
||||
assert "description" in meta
|
||||
|
||||
def test_case_insensitive(self):
|
||||
assert is_v0_43_benchmark("CEval") is True
|
||||
|
||||
def test_unknown_returns_false(self):
|
||||
assert is_v0_43_benchmark("mmlu") is False
|
||||
assert is_v0_43_benchmark("garbage") is False
|
||||
|
||||
def test_non_string_returns_false(self):
|
||||
assert is_v0_43_benchmark(None) is False # type: ignore[arg-type]
|
||||
assert is_v0_43_benchmark(123) is False # type: ignore[arg-type]
|
||||
|
||||
def test_metadata_immutable(self):
|
||||
meta = benchmark_metadata("ceval")
|
||||
with pytest.raises(TypeError):
|
||||
meta["description"] = "evil" # type: ignore[index]
|
||||
|
||||
def test_lm_eval_task(self):
|
||||
assert lm_eval_task_for("ceval") == "ceval-valid"
|
||||
assert lm_eval_task_for("cmmlu") == "cmmlu"
|
||||
# Aider Polyglot has no lm-eval mapping.
|
||||
assert lm_eval_task_for("aider_polyglot") is None
|
||||
assert lm_eval_task_for("garbage") is None
|
||||
|
|
@ -0,0 +1,290 @@
|
|||
"""Tests for v0.43.0 Part C — Profiling extras + VSCode setup."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from soup_cli.utils.profiling_v0_43 import (
|
||||
BandwidthExpectation,
|
||||
detect_anomaly_context,
|
||||
expected_bandwidth,
|
||||
memory_snapshot_context,
|
||||
nccl_bandwidth_check,
|
||||
resolve_snapshot_path,
|
||||
)
|
||||
from soup_cli.utils.vscode_setup import build_launch_json, write_vscode_launch
|
||||
|
||||
# ----------------- snapshot path -----------------
|
||||
|
||||
class TestResolveSnapshotPath:
|
||||
def test_happy(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = resolve_snapshot_path("run-123")
|
||||
assert path.endswith(os.path.join("profiles", "run-123.snapshot.pickle"))
|
||||
assert os.path.realpath(str(tmp_path)) in path
|
||||
|
||||
@pytest.mark.parametrize("bad", ["", ".", "..", "a/b", "a\\b", "a\x00b"])
|
||||
def test_invalid_run_id(self, bad, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
resolve_snapshot_path(bad)
|
||||
|
||||
def test_non_string_run_id(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="must be a string"):
|
||||
resolve_snapshot_path(123) # type: ignore[arg-type]
|
||||
|
||||
def test_invalid_base_dir(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
resolve_snapshot_path("run-1", base_dir="")
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
resolve_snapshot_path("run-1", base_dir="prof\x00iles")
|
||||
|
||||
def test_absolute_base_dir_rejected(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="relative"):
|
||||
resolve_snapshot_path("run-1", base_dir=str(tmp_path / "abs"))
|
||||
|
||||
def test_dotdot_in_base_dir_rejected(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match=r"'\.\.'"):
|
||||
resolve_snapshot_path("run-1", base_dir="../escape")
|
||||
|
||||
def test_dot_base_dir_rejected(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match=r"'\.\.'"):
|
||||
resolve_snapshot_path("run-1", base_dir="..")
|
||||
|
||||
|
||||
# ----------------- memory_snapshot_context -----------------
|
||||
|
||||
class TestMemorySnapshotContext:
|
||||
def test_no_torch_yields_none(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
# Force ImportError for torch.
|
||||
import builtins
|
||||
|
||||
real_import = builtins.__import__
|
||||
|
||||
def fake(name, *a, **kw):
|
||||
if name == "torch":
|
||||
raise ImportError("torch not installed")
|
||||
return real_import(name, *a, **kw)
|
||||
|
||||
monkeypatch.setattr(builtins, "__import__", fake)
|
||||
with memory_snapshot_context("run-x") as path:
|
||||
assert path is None
|
||||
|
||||
def test_invalid_max_entries(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
with memory_snapshot_context("run-x", max_entries=0):
|
||||
pass
|
||||
with pytest.raises(ValueError):
|
||||
with memory_snapshot_context("run-x", max_entries=True): # type: ignore[arg-type]
|
||||
pass
|
||||
|
||||
def test_invalid_run_id_propagates(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
with memory_snapshot_context(""):
|
||||
pass
|
||||
|
||||
|
||||
class TestDetectAnomalyContext:
|
||||
def test_no_torch_yields_false(self, monkeypatch):
|
||||
import builtins
|
||||
|
||||
real_import = builtins.__import__
|
||||
|
||||
def fake(name, *a, **kw):
|
||||
if name == "torch":
|
||||
raise ImportError("torch not installed")
|
||||
return real_import(name, *a, **kw)
|
||||
|
||||
monkeypatch.setattr(builtins, "__import__", fake)
|
||||
with detect_anomaly_context() as enabled:
|
||||
assert enabled is False
|
||||
|
||||
def test_with_real_torch_or_skip(self):
|
||||
try:
|
||||
import torch # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("torch not installed in this environment")
|
||||
with detect_anomaly_context() as enabled:
|
||||
# On any torch with autograd module the context is active.
|
||||
assert enabled is True
|
||||
|
||||
|
||||
# ----------------- nccl_bandwidth_check -----------------
|
||||
|
||||
class TestExpectedBandwidth:
|
||||
@pytest.mark.parametrize(
|
||||
"gpu,link",
|
||||
[
|
||||
("h100", "nvlink"),
|
||||
("h100", "pcie"),
|
||||
("a100", "nvlink"),
|
||||
("a100", "pcie"),
|
||||
("v100", "nvlink"),
|
||||
("rtx4090", "pcie"),
|
||||
],
|
||||
)
|
||||
def test_known_pairs(self, gpu, link):
|
||||
bw = expected_bandwidth(gpu, link)
|
||||
assert bw is not None and bw > 0
|
||||
|
||||
def test_case_insensitive(self):
|
||||
assert expected_bandwidth("H100", "NVLINK") == 450.0
|
||||
|
||||
def test_unknown_returns_none(self):
|
||||
assert expected_bandwidth("evil", "pcie") is None
|
||||
assert expected_bandwidth("h100", "carrier-pigeon") is None
|
||||
|
||||
def test_non_string_returns_none(self):
|
||||
assert expected_bandwidth(None, "nvlink") is None # type: ignore[arg-type]
|
||||
assert expected_bandwidth(123, 456) is None # type: ignore[arg-type]
|
||||
|
||||
def test_dataclass_frozen(self):
|
||||
be = BandwidthExpectation(gpu="h100", link="nvlink", expected_gb_per_sec=450.0)
|
||||
with pytest.raises(Exception):
|
||||
be.gpu = "a100" # type: ignore[misc]
|
||||
|
||||
|
||||
class TestNcclBandwidthCheck:
|
||||
def test_ok(self):
|
||||
result = nccl_bandwidth_check(
|
||||
gpu="h100", link="nvlink", measured_gb_per_sec=400.0
|
||||
)
|
||||
assert result["status"] == "OK"
|
||||
assert result["expected_gb_per_sec"] == 450.0
|
||||
assert 0.88 <= result["ratio"] <= 0.89
|
||||
|
||||
def test_minor(self):
|
||||
result = nccl_bandwidth_check(
|
||||
gpu="h100", link="nvlink", measured_gb_per_sec=300.0
|
||||
)
|
||||
assert result["status"] == "MINOR"
|
||||
|
||||
def test_major(self):
|
||||
result = nccl_bandwidth_check(
|
||||
gpu="h100", link="nvlink", measured_gb_per_sec=100.0
|
||||
)
|
||||
assert result["status"] == "MAJOR"
|
||||
|
||||
def test_unknown_pair(self):
|
||||
result = nccl_bandwidth_check(
|
||||
gpu="evil", link="pcie", measured_gb_per_sec=5.0
|
||||
)
|
||||
assert result["status"] == "UNKNOWN"
|
||||
assert result["expected_gb_per_sec"] is None
|
||||
assert result["ratio"] is None
|
||||
|
||||
def test_negative_measured_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
nccl_bandwidth_check(
|
||||
gpu="h100", link="nvlink", measured_gb_per_sec=-1.0
|
||||
)
|
||||
|
||||
def test_nonfinite_measured_rejected(self):
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
nccl_bandwidth_check(
|
||||
gpu="h100",
|
||||
link="nvlink",
|
||||
measured_gb_per_sec=float("inf"),
|
||||
)
|
||||
|
||||
def test_bool_measured_rejected(self):
|
||||
with pytest.raises(ValueError):
|
||||
nccl_bandwidth_check(
|
||||
gpu="h100", link="nvlink", measured_gb_per_sec=True # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
# ----------------- VSCode launch.json -----------------
|
||||
|
||||
class TestBuildLaunchJson:
|
||||
def test_default(self):
|
||||
payload = build_launch_json()
|
||||
assert payload["version"] == "0.2.0"
|
||||
configs = payload["configurations"]
|
||||
assert len(configs) >= 2
|
||||
train = next(c for c in configs if c["name"] == "soup train")
|
||||
assert "soup.yaml" in train["args"]
|
||||
assert train["module"] == "soup_cli.cli"
|
||||
|
||||
def test_custom_config_path(self):
|
||||
payload = build_launch_json(config_path="my-cfg.yaml")
|
||||
train = next(c for c in payload["configurations"] if c["name"] == "soup train")
|
||||
assert "my-cfg.yaml" in train["args"]
|
||||
|
||||
def test_invalid_config_path(self):
|
||||
with pytest.raises(ValueError):
|
||||
build_launch_json(config_path="")
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
build_launch_json(config_path="x\x00y.yaml")
|
||||
with pytest.raises(ValueError, match="newlines"):
|
||||
build_launch_json(config_path="a\nb.yaml")
|
||||
with pytest.raises(ValueError):
|
||||
build_launch_json(config_path="a" * 1024)
|
||||
|
||||
|
||||
class TestWriteVscodeLaunch:
|
||||
def test_writes_file(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
out = write_vscode_launch()
|
||||
assert Path(out).is_file()
|
||||
with open(out, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
assert data["version"] == "0.2.0"
|
||||
|
||||
def test_refuses_overwrite_without_force(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
write_vscode_launch()
|
||||
with pytest.raises(FileExistsError):
|
||||
write_vscode_launch()
|
||||
|
||||
def test_force_overwrites(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
write_vscode_launch()
|
||||
# Mutate file then re-write with force=True.
|
||||
path = Path(tmp_path) / ".vscode" / "launch.json"
|
||||
path.write_text("garbage", encoding="utf-8")
|
||||
out = write_vscode_launch(force=True)
|
||||
with open(out, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
assert data["version"] == "0.2.0"
|
||||
|
||||
def test_outside_cwd_rejected(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="under cwd"):
|
||||
write_vscode_launch(target_dir=str(tmp_path.parent / "evil"))
|
||||
|
||||
def test_invalid_target_dir(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
write_vscode_launch(target_dir="")
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
write_vscode_launch(target_dir="ev\x00il")
|
||||
|
||||
def test_force_must_be_bool(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
write_vscode_launch(force="yes") # type: ignore[arg-type]
|
||||
|
||||
def test_symlink_target_rejected(self, tmp_path, monkeypatch):
|
||||
# Skip on Windows without dev-mode where symlinks need privilege.
|
||||
if os.name == "nt":
|
||||
pytest.skip("symlink creation needs admin/dev mode on Windows")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
vscode_dir = tmp_path / ".vscode"
|
||||
vscode_dir.mkdir()
|
||||
outside = tmp_path / "outside.json"
|
||||
outside.write_text("{}", encoding="utf-8")
|
||||
os.symlink(str(outside), str(vscode_dir / "launch.json"))
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
write_vscode_launch(force=True)
|
||||
|
|
@ -0,0 +1,204 @@
|
|||
"""Tests for v0.43.0 Part D — soup data demo bundles."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.data import app as data_app
|
||||
from soup_cli.utils.demo_bundles import (
|
||||
DEMO_BUNDLE_NAMES,
|
||||
DemoBundle,
|
||||
copy_bundle_to,
|
||||
get_bundle,
|
||||
list_bundles,
|
||||
)
|
||||
|
||||
|
||||
class TestDemoBundleRegistry:
|
||||
def test_known_names(self):
|
||||
for name in ("alpaca_demo", "sharegpt_demo", "dpo_demo", "grpo_demo"):
|
||||
assert name in DEMO_BUNDLE_NAMES
|
||||
|
||||
def test_immutable_set(self):
|
||||
with pytest.raises(AttributeError):
|
||||
DEMO_BUNDLE_NAMES.add("evil") # type: ignore[attr-defined]
|
||||
|
||||
def test_list_bundles_sorted(self):
|
||||
bundles = list_bundles()
|
||||
names = [b.name for b in bundles]
|
||||
assert names == sorted(names)
|
||||
|
||||
def test_bundle_dataclass_frozen(self):
|
||||
b = list_bundles()[0]
|
||||
assert isinstance(b, DemoBundle)
|
||||
with pytest.raises(Exception):
|
||||
b.name = "evil" # type: ignore[misc]
|
||||
|
||||
def test_get_bundle_unknown(self):
|
||||
with pytest.raises(ValueError, match="unknown bundle"):
|
||||
get_bundle("garbage")
|
||||
|
||||
def test_get_bundle_empty(self):
|
||||
with pytest.raises(ValueError, match="not be empty"):
|
||||
get_bundle("")
|
||||
|
||||
def test_get_bundle_null_byte(self):
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
get_bundle("alpaca\x00")
|
||||
|
||||
def test_get_bundle_oversize(self):
|
||||
with pytest.raises(ValueError, match="exceeds max"):
|
||||
get_bundle("a" * 64)
|
||||
|
||||
def test_get_bundle_non_string(self):
|
||||
with pytest.raises(ValueError, match="must be a string"):
|
||||
get_bundle(123) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestCopyBundleTo:
|
||||
def test_writes_jsonl(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
out = copy_bundle_to("alpaca_demo", "./alpaca_demo.jsonl")
|
||||
assert Path(out).is_file()
|
||||
# Validate every line parses as JSON.
|
||||
with open(out, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
if line.strip():
|
||||
json.loads(line)
|
||||
|
||||
def test_outside_cwd_rejected(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
evil = str(tmp_path.parent / "evil.jsonl")
|
||||
with pytest.raises(ValueError, match="under cwd"):
|
||||
copy_bundle_to("alpaca_demo", evil)
|
||||
|
||||
def test_existing_file_rejected(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = tmp_path / "out.jsonl"
|
||||
target.write_text("placeholder", encoding="utf-8")
|
||||
with pytest.raises(FileExistsError):
|
||||
copy_bundle_to("alpaca_demo", str(target))
|
||||
|
||||
def test_null_byte_output_rejected(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
copy_bundle_to("alpaca_demo", "ev\x00il.jsonl")
|
||||
|
||||
def test_non_string_output(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="must be a non-empty string"):
|
||||
copy_bundle_to("alpaca_demo", "")
|
||||
|
||||
def test_unknown_bundle(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="unknown bundle"):
|
||||
copy_bundle_to("garbage", "./out.jsonl")
|
||||
|
||||
def test_symlink_at_tmp_path_rejected(self, tmp_path, monkeypatch):
|
||||
if os.name == "nt":
|
||||
pytest.skip("symlink creation needs admin/dev mode on Windows")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = tmp_path / "out.jsonl"
|
||||
# Pre-place a symlink at <target>.tmp pointing to a sentinel file.
|
||||
outside = tmp_path / "outside.txt"
|
||||
outside.write_text("placeholder", encoding="utf-8")
|
||||
os.symlink(str(outside), str(target) + ".tmp")
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
copy_bundle_to("alpaca_demo", str(target))
|
||||
|
||||
@pytest.mark.parametrize("name", sorted(DEMO_BUNDLE_NAMES))
|
||||
def test_every_bundle_copyable(self, name, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
out = copy_bundle_to(name, f"./{name}.jsonl")
|
||||
assert os.path.getsize(out) > 0
|
||||
|
||||
|
||||
class TestDataDemoCli:
|
||||
def test_list_help(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(data_app, ["demo", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "demo bundle" in result.output.lower() or "bundle" in result.output.lower()
|
||||
|
||||
def test_list_no_args(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=str(tmp_path)):
|
||||
result = runner.invoke(data_app, ["demo"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
# Table renders all 4 bundles.
|
||||
for name in ("alpaca_demo", "sharegpt_demo", "dpo_demo", "grpo_demo"):
|
||||
assert name in result.output
|
||||
|
||||
def test_copy_default_output(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=str(tmp_path)):
|
||||
result = runner.invoke(data_app, ["demo", "alpaca_demo"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert Path("alpaca_demo.jsonl").is_file()
|
||||
|
||||
def test_copy_custom_output(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=str(tmp_path)):
|
||||
result = runner.invoke(
|
||||
data_app, ["demo", "dpo_demo", "--output", "./mine.jsonl"]
|
||||
)
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert Path("mine.jsonl").is_file()
|
||||
|
||||
def test_unknown_bundle_exits_2(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=str(tmp_path)):
|
||||
result = runner.invoke(data_app, ["demo", "garbage"])
|
||||
assert result.exit_code == 2, (result.output, repr(result.exception))
|
||||
assert "unknown bundle" in result.output.lower()
|
||||
|
||||
def test_existing_output_exits_1(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=str(tmp_path)):
|
||||
Path("alpaca_demo.jsonl").write_text("x", encoding="utf-8")
|
||||
result = runner.invoke(data_app, ["demo", "alpaca_demo"])
|
||||
assert result.exit_code == 1, (result.output, repr(result.exception))
|
||||
assert "already exists" in result.output.lower()
|
||||
|
||||
def test_outside_cwd_output_exits_1(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
evil = str(tmp_path.parent / "evil.jsonl")
|
||||
with runner.isolated_filesystem(temp_dir=str(tmp_path)):
|
||||
result = runner.invoke(
|
||||
data_app, ["demo", "alpaca_demo", "--output", evil]
|
||||
)
|
||||
assert result.exit_code == 1, (result.output, repr(result.exception))
|
||||
assert "under cwd" in result.output.lower()
|
||||
|
||||
|
||||
class TestCopyBundleEdgeCases:
|
||||
def test_size_cap_branch(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils import demo_bundles as db
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
big_src = tmp_path / "big.jsonl"
|
||||
line = json.dumps({"x": "y" * 1024}) + "\n"
|
||||
with open(big_src, "w", encoding="utf-8") as f:
|
||||
for _ in range(55_000): # ~55 MB
|
||||
f.write(line)
|
||||
monkeypatch.setattr(db, "_bundle_source_path", lambda _b: str(big_src))
|
||||
with pytest.raises(ValueError, match="byte cap"):
|
||||
db.copy_bundle_to("alpaca_demo", "./out.jsonl")
|
||||
# Staged temp file must not survive the rejection.
|
||||
assert not (tmp_path / "out.jsonl.tmp").exists()
|
||||
assert not (tmp_path / "out.jsonl").exists()
|
||||
|
||||
def test_invalid_json_line(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils import demo_bundles as db
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
bad = tmp_path / "bad.jsonl"
|
||||
bad.write_text('{"ok": 1}\nnot-json\n', encoding="utf-8")
|
||||
monkeypatch.setattr(db, "_bundle_source_path", lambda _b: str(bad))
|
||||
with pytest.raises(ValueError, match="not valid JSON"):
|
||||
db.copy_bundle_to("alpaca_demo", "./out.jsonl")
|
||||
assert not (tmp_path / "out.jsonl").exists()
|
||||
Loading…
Reference in New Issue