diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9cca946..5e0ccef 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -107,11 +107,11 @@ soup_cli/ cans/ - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0) data/traces/ - Trace-to-Preference harvester (v0.26.0) data/collators.py - CrossDocCollator for sample packing (v0.33.0) - utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm + utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (224 files, 10035 tests) +tests/ - Test suite (231 files, 10306 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index c2219a0..29fa2bf 100644 --- a/README.md +++ b/README.md @@ -42,14 +42,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.63.0 — Production Trace Ecosystem: pull traces from any SaaS dashboard, mine your prod logs, ship gates.** Closes axis 7 of the roadmap. Five new top-level commands turn the v0.58 `soup loop` into a full data flywheel without ripping out your existing observability — every Part LIVE on day one (no deferred stubs). Adapters for Langfuse / LangSmith / Helicone / OpenPipe / OpenTelemetry / OpenAI Stored Completions; prompt-prefix mining; uncertainty-based active learning; proper sequential A/B testing with early-stop guarantees; rolling-KL drift alarm with optional Slack/Discord webhooks. +**v0.64.0 — Pre-flight & Tooling: pick the right base, lock the env, refuse OOMs before launch.** Closes axis 1 + axis 11 of the roadmap. Six new top-level commands give operators the rest of the pre-flight surface that v0.54 `soup advise` started: which base model, which compute, which env, which licence. Hosted vendors push their own catalogs and skip CUDA-hell ownership — Soup is local + spans every base. -- **`soup ingest --source --logs `.** Universal trace importer with 6 adapters covering every major observability vendor. Output is a normalised JSONL trace stream that `soup data from-traces` (v0.26) consumes after a thin shim. SaaS pulls happen out-of-band — Soup parses the export, never makes the network call (zero credential-handling threat surface). PII reminder panel on every ingest. `LANGFUSE_KEY` / `LANGSMITH_API_KEY` / `HELICONE_API_KEY` / `OPENPIPE_API_KEY` / `OPENAI_API_KEY` / `OTEL_EXPORTER_OTLP_HEADERS` env-var hints. -- **`soup prune-prompt --input --output --min-frequency 0.95`.** Detect + strip the longest shared system-prompt prefix across training rows so the FT model internalises it (OpenPipe's signature trick, OSS). Binary-search over up-to-32 candidate templates finds the longest prefix above the threshold — no 100%-match early-exit. Two-pass file read with a 100k-row DoS cap. -- **`soup data active-sample --input --budget N`.** Surface the most uncertain prod traces for human review. Max-entropy on single `rm_score` (peak at 0.5) OR pairwise disagreement on dual `rm_scores`. Composes with v0.19 human eval — the output JSONL is a drop-in eval prompt set. -- **`soup ab --input --metric latency|judge_score|retry_rate`.** Proper sequential A/B harness using Wald's classic SPRT for the point alternative. LLR is a martingale under H0 so Type-I error is controlled at every stopping time. Decision: `continue` / `reject_h0` / `accept_h0`. Composes with v0.58 `soup loop canary` — promote (or roll back) as soon as evidence clears the threshold. -- **`soup drift-alarm --reference --live --threshold 0.2`.** Rolling KL on whitespace-tokenised output distribution. Surfaces both behavioural drift ("model now outputs JSON when it used to output prose") AND vocabulary drift ("model has started repeating the same 20 phrases"). Optional `--slack-url` / `--discord-url` webhooks SSRF-validated to v0.51.0 parity (loopback-only HTTP, RFC1918 + 169.254.x rejected). Exit code 3 on drift detected for cron-friendly automation. -- **+219 new tests** (9816 → 10035 net). Review-fix coverage: 1 CRITICAL (mSPRT sign error — earlier draft's malformed LLR drove Type-I error to 1.0 as n grew; replaced with Wald point-alternative SPRT) + 2 HIGH (detect_common_prefix early-exit on 100% match returned shortest prefix not longest + `_MAX_SCAN_ROWS` cap used `pass` instead of `break`) + 3 MEDIUM (`TraceRecord.metadata` now `MappingProxyType`, env-label table deduplicated, drift-alarm precedence parens) + 2 LOW (pooled_se dead-branch + `mean_uncertainty` NaN guard) + 8 follow-up boundary / regression tests. Manual CPU smokes for all 5 commands + 6 failure-mode rejection paths. +- **`soup tunability --dataset [--candidates a,b,c] [--probe-steps N]`.** Probe-train 8 small candidate bases (Qwen3-0.6/1.7B, Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) against a held-out slice and report the (delta-from-base × cost × license) Pareto frontier. `--plan-only` dry-runs the sweep; `--list` shows the built-in catalogue. Live LoRA probe deferred to v0.64.1 — v0.64.0 ships the schema, Pareto math, default catalogue, and a `probe_fn` injection point. +- **`soup plan --config soup.yaml` / `soup apply --config soup.yaml`.** Terraform-shape lock-and-execute for training. `plan` writes `soup.tfstate` with the run's config SHA, dataset SHA, estimated cost, ETA, peak VRAM, and spot price. `apply` refuses if the YAML drifted from the state (exit 3) so you never silently re-spend $0.50 on a mutated config. `--dry-run` validates without running. +- **`soup env lock` / `soup env status` / `soup env check`.** Hermetic env lockfile via `importlib.metadata` — torch / transformers / peft / trl / accelerate / bitsandbytes / flash-attn / xformers / deepspeed / unsloth / vllm + Python + platform + CUDA versions. `check` compares the current env against the lock and exits 3 on ABI-sensitive drift (the "FT worked on Friday, broke on Monday" problem). +- **Hardware-fit calculator.** Static analytical predictor of peak VRAM by class (weights / optimizer / gradients / activations / overhead). Given (params, seq_len, batch_size, optimizer, quant, peft, gradient_checkpointing), it returns a 5-bucket breakdown + an OK/OOM verdict with 10% safety margin and an actionable hint (`--batch-size halve` / `--quantization 4bit` / `--gradient-checkpointing auto`). Composes with v0.40.3 live CUDA OOM probe. +- **`soup completions bash | zsh | fish`.** Sourceable shell completion scripts. `eval "$(soup completions bash)"` adds tab-completion for `soup` + every subcommand in the current shell. `--target-modules` falls back to canonical Llama-shape defaults; recipe names auto-complete from the 115+ catalogue. +- **`soup license-advisor --target b2c|defense|embedded [--license --mau N]`.** Per-deploy-target license matrix. B2C → permissive recommended, non-commercial forbidden. Defense → restricted-use community licenses (Llama / Gemma / Qwen-research / Mistral-research) forbidden because their acceptable-use clauses clash. Embedded → strong-copyleft (GPL/AGPL) forbidden because closed-source firmware redistribution. The per-license check flags the Llama community license + > 700M MAU rule (exit 3) so you don't ship a B2C product that needs a separate Meta licence after the fact. Composes with v0.60 `adapters merge` license-conflict gate. +- **+271 new tests** (10035 → 10306 net). Review-fix coverage from a consolidated code+security+TDD wave: 0 CRITICAL + 6 HIGH (TOCTOU symlink rejection on every new read path; `compute_dataset_sha` containment + symlink-reject; narrow `except Exception` to `(OSError, ValueError)` in env probing) + 8 MEDIUM (strict-JSON config SHA, no `default=str` silent collision; strict-bool `applied` validation; `_activation_bytes` overflow clamp; PEP 604 → `Optional[str]` for Typer Py3.9 compat; tight Llama-family allowlist replaces `.startswith`; Windows `v12.1` CUDA path-parse; containment-before-existence ordering) + 4 LOW (source-grep regression for `atomic_write_text`, MAU upper cap, `Sequence` from `collections.abc`, end-to-end drift-refusal exit-3). Manual CPU smokes for every Part A-F command incl. failure modes (drift exit 3, license block exit 3, completions unknown shell exit 2). ## Why Soup? @@ -3636,6 +3637,14 @@ soup data active-sample --input --output --budget N Top-N uncer soup ab --input --metric latency|judge_score|retry_rate mSPRT sequential A/B (decision: continue / reject_h0 / accept_h0) soup drift-alarm --reference --live --threshold 0.2 Rolling-KL drift alarm (exit 3 on drift) soup drift-alarm ... --slack-url | --discord-url Optional SSRF-validated webhook on drift detected +soup tunability --list List built-in candidate-base catalogue +soup tunability --dataset [--candidates a,b,c] Probe 8 candidate bases + Pareto frontier report +soup plan --config soup.yaml Pre-flight summary + write soup.tfstate +soup apply --config soup.yaml [--dry-run] Lock-and-execute; refuses on drift (exit 3) +soup env lock | status | check Hermetic env lockfile + ABI drift detection (exit 3) +soup completions bash | zsh | fish Shell completion script (sourceable via eval) +soup license-advisor --target b2c|defense|embedded Recommend license-clean base for deploy target +soup license-advisor ... --license --mau N Per-license downstream-risk check (exit 3 on block) soup version [--full] [--json] Show version (--full: system info, --json: JSON output) soup --verbose Full traceback on errors ``` @@ -4658,6 +4667,111 @@ extraction filter. `soup airgap-bundle` is intentionally a top-level command (not `soup deploy airgap-bundle`) — it's an export operation, not a deploy target. +## Tunability Probe (`soup tunability`) + +Before committing to a single base model, run a short LoRA probe on every reasonable candidate against your held-out slice. v0.64.0 ships an 8-entry default catalogue covering Qwen3, Llama-3.2, Gemma 3, Phi-4, SmolLM3, and Qwen2.5 across the 0.6 B – 3.8 B band. + +```bash +# List the built-in catalogue +soup tunability --list + +# Dry-run a sweep across a subset +soup tunability --dataset ./eval.jsonl --candidates qwen3-0.6b,phi-4-mini --plan-only + +# Run the full sweep + write a JSON report +soup tunability --dataset ./eval.jsonl --probe-steps 100 --output ./tunability.json +``` + +The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. Live LoRA probe lands in v0.64.1; v0.64.0 ships the schema, Pareto math, and a `probe_fn=` injection point. + +## Terraform-Style Plan & Apply (`soup plan` / `soup apply`) + +A training run is a one-shot infrastructure-shaped operation: spot price, expected cost, base SHA, dataset SHA, peak VRAM. v0.64 borrows Terraform's plan-apply split so you can review the numbers before committing money. + +```bash +# Render a pre-flight summary + write soup.tfstate +soup plan --config soup.yaml + +# Apply — refuses on drift (exit 3) if the YAML changed since `plan` +soup apply --config soup.yaml + +# Validate without actually running anything +soup apply --config soup.yaml --dry-run +``` + +The state file is a thin JSON envelope; the actual training is still driven by `soup train`. The gate prevents the "wait, why did I spend another $0.50 on the wrong config" surprise. + +## Hermetic Env Lockfile (`soup env`) + +The "CUDA hell" problem: a fine-tune that worked on Friday breaks on Monday because PyPI silently upgraded `transformers` past the trainer's compat band. v0.34 `soup doctor` surfaces some of this; v0.64 makes it lockable. + +```bash +# Snapshot the current env into soup-env.lock +soup env lock + +# Print the locked env summary +soup env status + +# Compare current env to the lock — exit 3 on ABI-sensitive drift +soup env check +``` + +`soup-env.lock` captures Python + platform + CUDA + 15 ABI-sensitive packages (torch / transformers / peft / trl / accelerate / bitsandbytes / flash-attn / xformers / deepspeed / unsloth / vllm / sentencepiece / tokenizers / datasets / huggingface-hub). Wire `soup env check` into your CI to refuse silent ABI breakage. + +## Hardware-Fit Calculator + +Given (params, seq_len, batch_size, optimizer, quant, peft, gradient_checkpointing), the analytical predictor returns a 5-bucket peak-VRAM breakdown (weights / optimizer / gradients / activations / overhead) and an OK/OOM verdict with a 10% safety margin. + +```python +from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit + +inp = HardwareFitInput( + params_b=7.0, seq_len=2048, batch_size=4, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, +) +report = decide_hardware_fit(inp, available_vram_gb=24.0) +print(report.ok, report.reason) +# True | 'fits: peak 7.76 GB + 10% margin <= 24.00 GB available' +``` + +When it doesn't fit, the report names actionable knobs: `--batch-size halve`, `--quantization 4bit`, `--gradient-checkpointing auto`. Composes with v0.40.3 live CUDA OOM probe (`make_cuda_probe_fn`) which still runs when `auto_batch_size_strategy: probe`. + +## Shell Completions (`soup completions`) + +Tab-completion for `soup` + every subcommand. The generated script is Click/Typer-backed so new commands are picked up automatically. + +```bash +# Bash +eval "$(soup completions bash)" # current shell +soup completions bash >> ~/.bashrc # permanent + +# Zsh +soup completions zsh > "${fpath[1]}/_soup" + +# Fish +soup completions fish > ~/.config/fish/completions/soup.fish +``` + +Recipe names auto-complete from the 115+ catalogue; `--target-modules` falls back to canonical Llama-shape defaults (`q_proj` / `k_proj` / `v_proj` / etc.). Live HF-config introspection per `base` lands in v0.64.1. + +## License Advisor (`soup license-advisor`) + +Picking a license-clean base for a specific deployment target is a recurring legal-review pain point. v0.64 captures the three most common deploy contexts as a closed allowlist and surfaces the per-license downstream risk. + +```bash +# What licenses are safe for a B2C consumer product? +soup license-advisor --target b2c + +# Defense — restricted-use community licenses forbidden +soup license-advisor --target defense + +# Per-license check: Llama community license + 800M MAU = block (exit 3) +soup license-advisor --target b2c --license llama-3 --monthly-active-users 800000000 +``` + +The Llama-family allowlist is tight (no `.startswith` over-match), so a hypothetical future `llama-permissive-2030` won't false-trigger the 700M-MAU gate. Composes with v0.60 `soup adapters merge --license ` for the merge-time conflict gate. + ## Changelog See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history. diff --git a/SECURITY.md b/SECURITY.md index 3ca5772..f941702 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.63.0 -- Full support (latest) +- v0.64.0 -- Full support (latest) +- v0.63.0 -- Full support - v0.62.0 -- Full support - v0.61.0 -- Full support - v0.60.0 -- Full support @@ -137,6 +138,8 @@ No known critical vulnerabilities in current releases. ### Security Hardening History +- **v0.64.0 — Pre-flight & Tooling**: 6 LIVE parts ship `soup tunability` + `soup plan` / `apply` + `soup env lock` / `status` / `check` + analytical hardware-fit calculator + `soup completions` + `soup license-advisor`. **TOCTOU symlink rejection on every new read path** — `_load_yaml_config` in `commands/plan.py`, `load_report` in `utils/tunability.py`, `read_lock` in `utils/env_lock.py`, `read_state` in `utils/terraform_plan.py` all reject `S_ISLNK` BEFORE any `open()` so a pre-placed `soup.yaml -> /etc/shadow` cannot redirect the read (HIGH H2/H3/H4 fixes mirroring v0.33.0 #22 / v0.43.0 / v0.55.0 / v0.62.0 policy). **`compute_dataset_sha` containment + symlink reject** (HIGH H5) — operator-supplied `data.train` path is `is_under_cwd`-checked + `os.lstat + S_ISLNK`-rejected before SHA-256; outside-cwd path returns the zero-hash sentinel (no file read), so a crafted `data.train: /etc/shadow` cannot smuggle file contents into the planning SHA. **Containment-before-existence ordering** (MEDIUM M8) on every new read path so a probing attacker cannot distinguish "file exists outside cwd" from "file missing" via the exception type (`ValueError("outside cwd")` fires BEFORE the `isfile` check). **`compute_config_sha` strict JSON** (MEDIUM M1) — no `default=str` fallback so two configs that differ only in an unserialisable field cannot silently collide (a `set` value now raises `TypeError` instead of being string-stringified). **`_activation_bytes` overflow clamp** at `_MAX_ACTIVATIONS_BYTES=1e18` (MEDIUM M4) so the analytical predictor stays finite even at schema max `seq_len=1_048_576 × batch=1024`. **Llama-family allowlist replaces `.startswith`** (MEDIUM M6) — `flag_downstream_risk` uses tight `_LLAMA_COMMUNITY_LICENSES = frozenset({llama-2, llama-3, llama-3.1, llama-3.2, llama-3.3, llama-community})` for the 700M-MAU gate so a hypothetical future `llama-permissive-2030` cannot false-trigger the block. **Windows + POSIX CUDA path-parse** (MEDIUM M7) — `_detect_cuda_version` strips both `\\` and `/` separators and a leading `v` prefix so `C:\Program Files\NVIDIA\CUDA\v12.1` extracts `12.1` correctly. **Narrow `except` policy** (HIGH H1) — `_detect_package_version` catches only `PackageNotFoundError` + `(OSError, ValueError)` instead of bare `Exception` so real bugs in metadata parsing surface loudly. **Closed-allowlist validators**: `SUPPORTED_SHELLS = {bash, zsh, fish}` (completions), `DEPLOY_TARGETS = {b2c, defense, embedded}` (license advisor), `_VALID_QUANT` 9-entry + `_VALID_PEFT` 4-entry + `_VALID_OPTIMIZERS` 10-entry (hardware-fit), `_VALID_SOURCES = {pip, conda, system, wheel, unknown}` (env lock), `_VALID_SEVERITIES = {ok, warn, block}` (license risk) — all bool-first / null-byte / oversize (≤32-char) / case-insensitive normalisation. **Atomic file writes** via shared `paths.atomic_write_text` in all 3 v0.64 writers (`tunability.write_report` / `terraform_plan.write_state` / `env_lock.write_lock`) + `os.lstat + S_ISLNK` rejection at the target so a pre-placed symlink cannot redirect the write (LOW L3 source-grep regression test). **Strict-bool validation on read** (MEDIUM M3) — `read_state` rejects non-bool `applied` rather than silent `bool()` coercion of `"yes"` / `1` / `[]`. **MAU upper cap** at `_MAX_MAU=10**13` (LOW L4) so a crafted `--mau=10**18` cannot overflow the gate. **PEP 604 → `Optional[str]` for Typer Py3.9 compat** (MEDIUM M5) — every Typer `Option` annotation uses `Optional[str]` instead of `str | None` so Typer's runtime introspection works on Python 3.9. **`Sequence` from `collections.abc`** not `typing` (LOW L5) so `isinstance(x, Sequence)` works at runtime on every Py 3.9 build. **Drift detection refuses on mutation** — `soup apply` exits 3 on config / dataset SHA drift between `soup.yaml` and `soup.tfstate`, surfacing the changed field name (`config_sha` / `dataset_sha` / `base` / `task`). **License block → exit 3** — `soup license-advisor --license llama-3 --mau 800_000_000` surfaces the Meta acceptable-use-policy MAU gate as a hard block, not a warning. **Test count**: 10035 → 10306 (+271 net in `tests/test_v0640_part_{a-f}.py` + `tests/test_v0640_followups.py`). **Review-fix coverage** across 1 consolidated wave (general-purpose code+security+TDD; python-reviewer agent context-thrashed on the 800+ KB CLAUDE.md release-notes history — matches v0.58.0 / v0.59.0 / v0.60.0 / v0.61.0 / v0.62.0 / v0.63.0 idiom): 0 CRITICAL + 6 HIGH (H1 narrow-except, H2/H3/H4 read-path symlink rejection, H5 compute_dataset_sha containment + symlink, H6 test-coverage gap) + 8 MEDIUM (M1 strict JSON, M3 strict-bool applied, M4 overflow clamp, M5 PEP 604 → Optional, M6 Llama allowlist, M7 Windows CUDA, M8 ordering, plus M2 _entry_from_dict default documented) + 4 LOW (L3 atomic_write_text source-grep, L4 MAU cap, L5 Sequence from collections.abc, plus end-to-end drift-refusal regression). **Known limitations**: (1) Live LoRA probe deferred to v0.64.1 — `_default_probe` is heuristic stub. (2) `soup apply` does NOT execute `soup train` in-process — prints the canonical invocation; Typer commands aren't safe to re-enter. (3) `soup env` does NOT recreate a venv — schema + drift detection ship; `--fix` action is v0.64.1. (4) Hardware-fit is analytical only — composes with v0.40.3 live CUDA probe but does NOT replace it. (5) `complete_target_modules` returns canonical Llama defaults — live HF-config introspection is v0.64.1. (6) License advisor takes the license id from the operator — no model-card scraping. (7) Deploy targets are a 3-entry allowlist (b2c/defense/embedded). (8) `compute_dataset_sha` returns zero-hash for outside-cwd paths rather than raising so `soup plan` runs before the dataset exists. (v0.64.0) + - **v0.63.0 — Production Trace Ecosystem**: 5 LIVE parts ship `soup ingest` (Langfuse / LangSmith / Helicone / OpenPipe / OTel / OpenAI Stored Completions) + `soup prune-prompt` + `soup data active-sample` + `soup ab` (Wald sequential SPRT) + `soup drift-alarm` (rolling KL + SSRF-hardened webhook). **Soup never makes the SaaS network call** — `soup ingest` parses JSONL exports the operator pulls from their dashboard; zero credential-handling threat surface. **`validate_webhook_url` SSRF parity with v0.51.0 / v0.30.0 OTLP**: scheme allowlist `{http, https}`, null-byte + control-char rejection (CRLF-injection defence), `0.0.0.0` explicitly rejected, plain HTTP only for loopback hosts (`localhost` / `127.0.0.1` / `::1`), RFC1918 / link-local / reserved / multicast IPs rejected via `ipaddress.ip_address`. **`post_webhook` never crashes drift check** — broad `except Exception` (with `# noqa: BLE001`) so telemetry/alerting failure cannot mask a real drift verdict. **Closed-allowlist validators**: `SUPPORTED_INGEST_SOURCES = {langfuse, langsmith, helicone, openpipe, otel, openai-stored}`, `SUPPORTED_METRICS = {latency, judge_score, retry_rate}` (mSPRT) — both bool-first / null-byte / oversize (≤32-char) / case-insensitive normalisation. **`TraceRecord.metadata` is `MappingProxyType`-wrapped post-init** (`__post_init__` via `object.__setattr__` because the dataclass is frozen) so a returned record cannot be mutated by callers (code-review MEDIUM fix v0.63.0). **DoS caps**: `_MAX_INGEST_LINES=1_000_000` on JSONL ingest, `_MAX_SCAN_ROWS=100_000` + `_MAX_ROW_CHARS=1_000_000` on `prune-prompt`, `_MAX_BUDGET=100_000` on `active-sample`, `_MAX_SAMPLES_PER_ARM=1_000_000` on mSPRT, `_MAX_WEBHOOK_URL_LEN=4096` + `_MAX_TEXT_LEN=1_000_000` on drift. **Path containment** via shared `is_under_cwd` (project-standard `os.path.realpath + commonpath`, NOT `Path.resolve + relative_to`) — `ingest_traces` / `prune_traces` / `sample_uncertain_rows` / `run_msprt` / `run_drift_check` all enforce cwd + null-byte rejection BEFORE any open. **Rich markup escape** on every user-controlled string surfaced through `rich.markup.escape` (`--source` / `--metric` / `--logs` / error messages). **`detect_common_prefix` never early-exits on the 100% match** (code-review HIGH fix v0.63.0) — binary-search-over-templates finds the longest threshold-meeting prefix even when a 100% prefix exists shorter than a threshold-majority prefix. **`prune_traces` row-cap uses `break` not `pass`** (code-review HIGH fix v0.63.0 — earlier draft left the DoS cap unenforced). **mSPRT decision math is Wald's classic SPRT for the point alternative**: LLR is a martingale under H0 (E[exp(LLR_n)] = 1) so Type-I error is controlled at every stopping time per the optional stopping theorem. Earlier draft used a malformed mixture-prior LLR with the wrong sign on the log term — drove LLR positive under H0 as n grew, producing unbounded false-positive rate (code-review CRITICAL fix v0.63.0). **`_is_private_or_link_local` precedence parens** (code-review MEDIUM fix v0.63.0) — safety-critical SSRF gate, future edits should not need to re-derive Python operator precedence to verify the logic. **Auth env-var resolution is read-only** — `resolve_auth_env` looks up `LANGFUSE_KEY` / `LANGSMITH_API_KEY` / `HELICONE_API_KEY` / `OPENPIPE_API_KEY` / `OPENAI_API_KEY` / `OTEL_EXPORTER_OTLP_HEADERS` from `os.environ`; Soup itself never makes the network call. **Test count**: 9816 → 10035 (+219 net in `tests/test_v0630_part_{a-e}.py` + `tests/test_v0630_followups.py`). **Review-fix coverage across 2 reviewer waves** (code-reviewer + tdd-guide; python-reviewer + security-reviewer agents context-thrashed on the 800+ KB CLAUDE.md release-notes history — matches v0.58.0 / v0.59.0 / v0.60.0 / v0.61.0 / v0.62.0 idiom; verified manually): 1 CRITICAL (mSPRT sign error) + 2 HIGH (detect_common_prefix early-exit + `pass`→`break` DoS cap) + 3 MEDIUM (TraceRecord MappingProxyType + `_AUTH_ENV` dedup + drift_alarm precedence parens) + 2 LOW (pooled_se dead-branch + mean_uncertainty NaN guard) + 8 follow-up tests (boundaries + source-grep regression guards for no-heavy-top-level-imports across all 5 new util modules). **Known limitations**: (1) No live SaaS API pull — operators export from the vendor dashboard / SDK first, Soup parses the JSONL. (2) `prune-prompt` operates on raw characters not tokens — UTF-8 multi-byte sequences may be cut mid-codepoint at the threshold boundary. (3) Active-learning K>2 RMs deferred to v0.63.1 — 3+ RM scores fall back to `max - min` range. (4) mSPRT single-metric per pass — operators monitoring (latency, judge_score, retry_rate) jointly should adjust alpha via Bonferroni and run three passes. (5) mSPRT assumes Gaussian-like data — binary metrics must be pre-aggregated to per-prompt rates. (6) drift-alarm uses whitespace tokeniser — sub-word BPE-level drift invisible (tokenizer-aware variant tracked for v0.63.1). (7) `post_webhook` swallows every exception so telemetry can NEVER crash the drift run. (8) Zero-variance arms in mSPRT → `continue` forever (proper SPRT behaviour: with no observed noise the test cannot bound Type-I error honestly). (v0.63.0) - **v0.62.0 — RAG & Activation Steering**: 5 parts ship `data.format='raft'` + RA-DIT two-stage recipes + `soup steer train/apply/list` (CAA / ITI / RepE) + citation-faithful FT + GRACE codebook. Schema-only release (live training loops + decode-hook intervention + codebook lookup all deferred to v0.62.1 — mirrors v0.50.0 / v0.52.0 / v0.61.0 stub-then-live pattern). **Closed-allowlist validators**: `SUPPORTED_RA_DIT_STAGES = {retriever, generator}`, `SUPPORTED_STEERING_METHODS = {caa, iti, repe}`, `SUPPORTED_CITATION_STYLES = {bracket, inline, footnote}`. All follow the project's bool-first / null-byte / oversize (32-char method names, 128-char steering names, 512-char model ids) / case-insensitive normalisation policy with frozen `*Spec` dataclasses + `MappingProxyType`-wrapped metadata. **RAFT input validation**: `_check_raft_string` rejects non-string / empty / null-byte / >64 KiB per field on every RAFT row field (query / golden_doc / answer / each distractor); `distractor_docs` list-type-guarded + capped at 64 entries per row. **Steering name regex** `^[A-Za-z0-9][A-Za-z0-9._\-]{0,127}$` rejects path separators + shell metacharacters so the name is safe as a Registry artifact id + CLI flag + filename fragment. **Citation kernel ReDoS-safe**: `extract_citation_ids` regex `\[(?P[A-Za-z0-9][A-Za-z0-9._\-]{0,127})\]` uses bounded quantifiers (no nested optional groups); `score_citations` predicted text capped at 2 MB, expected_ids capped at 10k entries with per-entry null-byte + bool rejection. **`soup steer` TOCTOU defence**: `_validate_pairs_path` delegates to shared `enforce_under_cwd_and_no_symlink` (centralised TOCTOU helper, not a custom re-impl per review M1) — symlinks at the pairs JSONL path rejected before any open/read. **`soup serve --steer` Rich markup escape** on the exception-message error panel (security review M1 fix — defends against crafted `--steer` value injecting Rich markup via `{value!r}` repr). **`--base` length cap** at 512 chars + null-byte rejection (security review L1 — mirrors v0.40.5 reward_model policy). **`list_steers` SQLite context-manager** (`with RegistryStore() as store:` — fixes connection leak surfaced by code-review M2). **Cross-validator silent-no-op rejection**: `citation_style` / `citation_recall_threshold` without `citation_faithful=true` rejected; `citation_faithful=true` requires `data.format='raft'` AND `task in {sft, pretrain}` (task-gate review M3 fix mirroring v0.52.0 distill / classifier policy); `grace_codebook_size` / `grace_codebook_dim` without `grace_codebook=true` rejected, and `grace_codebook=true` requires BOTH knobs; `ra_dit_retriever_model` without `ra_dit_stage` rejected. **GRACE codebook bounds**: `grace_codebook_size` ∈ [1, 100_000], `grace_codebook_dim` ∈ [1, 16_384] — sanity caps so a misconfigured run cannot allocate a multi-GB codebook by accident. **`apply_edit` deferred-marker routing**: `grace` plans raise NotImplementedError with `v0.62.1` marker while legacy rome/memit/alphaedit retain the v0.61.1 marker (regression-guarded via `TestEditMarkerRegressionGuard`). **Deferred stubs validate first**: `apply_steering` / `build_steering_vector` / `apply_grace_codebook` all run input validators BEFORE raising NotImplementedError so callers passing junk get a meaningful TypeError / ValueError instead of a confusing deferred-live error. **`typing.Iterable` → `collections.abc.Iterable`** migration in `citation_faithful.py` (python-review M1 — `typing.Iterable` deprecated since 3.9 for annotation use). **`except Exception` → `except ImportError`** narrowed in `commands/steer.py::list_steers` (python-review M2). **Test count**: 9571 → 9786 (+215 net in `tests/test_v0620_part_{a-e}.py` + `tests/test_v0620_followups.py`). **Known limitations**: (1) Every live wiring (RAFT span-mask trainer, RA-DIT two-stage orchestrator, `apply_steering` decode-hook, `build_steering_vector` per-method kernels, `apply_grace_codebook` lookup/write) deferred to v0.62.1 with explicit v0.62.1 markers. (2) `citation_faithful` gated to {sft, pretrain} — DPO / GRPO / etc. rejected (task-gate review fix). (3) `soup steer apply` is preview-only — live decode-hook ships via `soup serve --steer` in v0.62.1. (4) `extract_citation_ids` is bracket-style only; inline / footnote variants schema-validated but use the same extractor today. (5) RA-DIT recipe pair does not auto-link the retriever's output to the generator's `ra_dit_retriever_model` — operators copy manually. (6) GRACE codebook has no Registry persistence yet — lives only in the trained checkpoint until v0.62.1 wires `apply_edit(method='grace')` into v0.26 Registry. (7) `soup serve --steer` validates name + strength but does not load the vector — decode-hook lands in v0.62.1. (8) **4-agent review wave**: code + security + tdd + python style. Direct reviewer agents hit context-window thrash on the 800+ KB CLAUDE.md release-notes history (matches v0.58.0 / v0.59.0 / v0.60.0 / v0.61.0 idiom); the `general-purpose` agent with explicit "do not crawl, read only these N files" prompts produced equivalent findings consolidated into 4 sequential review waves. (v0.62.0) diff --git a/pyproject.toml b/pyproject.toml index 05a916a..e20449f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.63.0" +version = "0.64.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 261b66e..d49537e 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.63.0" +__version__ = "0.64.0" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index 8ac97a5..d27983c 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -338,6 +338,63 @@ app.command( ), )(_drift_alarm_cmd.drift_alarm) +# v0.64.0 Part A — Tunability probe across candidate bases. +from soup_cli.commands import tunability as _tunability_cmd # noqa: E402 + +app.command( + name="tunability", + help=( + "Probe-train 6-10 small bases on a held-out slice + report " + "Pareto frontier of (eval delta, train cost, license) (v0.64.0)." + ), +)(_tunability_cmd.tunability_cmd) + +# v0.64.0 Part B — Terraform-shape plan / apply. +from soup_cli.commands import plan as _plan_cmd # noqa: E402 + +app.command( + name="plan", + help=( + "Render a pre-flight training plan (cost / ETA / SHA / VRAM) " + "and write soup.tfstate for `soup apply` to consult (v0.64.0)." + ), +)(_plan_cmd.plan_cmd) + +app.command( + name="apply", + help=( + "Execute the planned training run, refusing on drift between " + "soup.yaml and soup.tfstate (v0.64.0)." + ), +)(_plan_cmd.apply_cmd) + +# v0.64.0 Part C — Hermetic env lockfile. +from soup_cli.commands.env import env_app as _env_app # noqa: E402 + +app.add_typer(_env_app, name="env") + +# v0.64.0 Part E — Shell completions. +from soup_cli.commands import completions as _completions_cmd # noqa: E402 + +app.command( + name="completions", + help=( + "Emit a bash / zsh / fish completion script. Use with " + "`eval \"$(soup completions bash)\"` (v0.64.0)." + ), +)(_completions_cmd.completions_cmd) + +# v0.64.0 Part F — License advisor. +from soup_cli.commands import license_advisor as _license_advisor_cmd # noqa: E402 + +app.command( + name="license-advisor", + help=( + "Recommend a license-clean base for a deploy target " + "(b2c / defense / embedded) + flag downstream risk (v0.64.0)." + ), +)(_license_advisor_cmd.license_advisor_cmd) + def _rewrite_advise_argv(argv: list) -> list: """Inject `run` between `advise` and a non-subcommand first argument. diff --git a/soup_cli/commands/completions.py b/soup_cli/commands/completions.py new file mode 100644 index 0000000..e11b090 --- /dev/null +++ b/soup_cli/commands/completions.py @@ -0,0 +1,37 @@ +"""`soup completions ` — emit a bash / zsh / fish completion script. + +Sourceable script: `eval "$(soup completions bash)"` adds tab-completion +for the `soup` command in the current shell. Designed for `eval` +consumption — every output goes to stdout exactly once, no panels or +banners. +""" + +from __future__ import annotations + +import typer +from rich.console import Console +from rich.markup import escape + +from soup_cli.utils.completions import render_completion_script + +# Console with stderr so the script body (stdout) stays clean for eval. +_err = Console(stderr=True) + + +def completions_cmd( + shell: str = typer.Argument( + ..., + help="One of: bash, zsh, fish.", + ), +) -> None: + """Render a shell completion script for `soup` (v0.64.0).""" + try: + text = render_completion_script(shell) + except (TypeError, ValueError) as exc: + _err.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + # Plain stdout — the script gets sourced by `eval`. + typer.echo(text) + + +__all__ = ["completions_cmd"] diff --git a/soup_cli/commands/env.py b/soup_cli/commands/env.py new file mode 100644 index 0000000..a897761 --- /dev/null +++ b/soup_cli/commands/env.py @@ -0,0 +1,159 @@ +"""`soup env` — hermetic env lockfile + ABI status (v0.64.0 Part C). + +Sub-commands: +- ``soup env lock`` — snapshot the current env into ``soup-env.lock``. +- ``soup env status`` — print currently-locked env summary. +- ``soup env check`` — compare current env against ``soup-env.lock`` and + report any ABI-sensitive drift (exit 3 on drift). +""" + +from __future__ import annotations + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +from soup_cli.utils.env_lock import ( + DEFAULT_LOCK_FILE, + check_abi_compat, + read_lock, + snapshot_env, + write_lock, +) +from soup_cli.utils.paths import is_under_cwd + +console = Console() + +env_app = typer.Typer( + name="env", + help="Hermetic env lockfile + ABI drift detection (v0.64.0).", + no_args_is_help=True, +) + + +@env_app.command("lock") +def env_lock_cmd( + output: str = typer.Option( + DEFAULT_LOCK_FILE, + "--output", + "-o", + help="Path to write the lock file (default: ./soup-env.lock).", + ), +) -> None: + """Snapshot the current env into a lock file.""" + if "\x00" in output: + console.print("[red]output path must not contain null bytes[/]") + raise typer.Exit(2) + if not is_under_cwd(output): + console.print(f"[red]output {escape(output)!r} is outside cwd[/]") + raise typer.Exit(2) + + try: + lock = snapshot_env() + write_lock(lock, output) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + Panel( + f"[green]Locked {len(lock.entries)} packages to " + f"{escape(output)}[/]\n" + f"Python: {escape(lock.python_version)} | " + f"Platform: {escape(lock.platform)} | " + f"CUDA: {escape(lock.cuda_version or 'none')}", + title="env lock", + border_style="green", + ) + ) + + +@env_app.command("status") +def env_status_cmd( + lock_path: str = typer.Option( + DEFAULT_LOCK_FILE, + "--lock", + help="Path to the lock file (default: ./soup-env.lock).", + ), +) -> None: + """Print currently-locked env summary.""" + try: + lock = read_lock(lock_path) + except FileNotFoundError: + console.print( + f"[yellow]No lock file at {escape(lock_path)}; " + "run `soup env lock` first.[/]" + ) + raise typer.Exit(1) from None + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + header = Table(title="env status (locked)") + header.add_column("Field") + header.add_column("Value") + header.add_row("soup_version", escape(lock.soup_version)) + header.add_row("python_version", escape(lock.python_version)) + header.add_row("platform", escape(lock.platform)) + header.add_row("cuda_version", escape(lock.cuda_version or "none")) + header.add_row("created_at", escape(lock.created_at)) + header.add_row("entries", str(len(lock.entries))) + console.print(header) + + if lock.entries: + body = Table(title="packages") + body.add_column("Name") + body.add_column("Version") + body.add_column("Source") + for e in lock.entries: + body.add_row(escape(e.name), escape(e.version), escape(e.source)) + console.print(body) + + +@env_app.command("check") +def env_check_cmd( + lock_path: str = typer.Option( + DEFAULT_LOCK_FILE, + "--lock", + help="Path to the lock file to compare against.", + ), +) -> None: + """Compare the current env against the lock and report drift.""" + try: + locked = read_lock(lock_path) + except FileNotFoundError: + console.print( + f"[red]No lock file at {escape(lock_path)}; " + "run `soup env lock` first.[/]" + ) + raise typer.Exit(1) from None + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + current = snapshot_env() + report = check_abi_compat(locked, current) + if report.ok: + console.print( + Panel( + "[green]ABI-clean.[/] No drift detected.", + title="env check", + border_style="green", + ) + ) + return + + body = "\n".join(f"- {escape(c)}" for c in report.changes) + console.print( + Panel( + f"[red]{report.drift_count} ABI-sensitive drift(s):[/]\n{body}", + title="env check", + border_style="red", + ) + ) + raise typer.Exit(3) + + +__all__ = ["env_app"] diff --git a/soup_cli/commands/license_advisor.py b/soup_cli/commands/license_advisor.py new file mode 100644 index 0000000..dcfd336 --- /dev/null +++ b/soup_cli/commands/license_advisor.py @@ -0,0 +1,95 @@ +"""`soup license-advisor` — pick a license-clean base for a deploy target. + +v0.64.0 Part F. Composes with v0.60 Part E ``license_matrix.check_license_compat`` +which gates ``soup adapters merge``. +""" + +from __future__ import annotations + +from typing import Optional + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +from soup_cli.utils.license_advisor import ( + advise_license_for_target, + flag_downstream_risk, +) + +console = Console() + + +def license_advisor_cmd( + target: str = typer.Option( + ..., + "--target", + "-t", + help="Deploy target: b2c | defense | embedded.", + ), + license_id: Optional[str] = typer.Option( + None, + "--license", + help="Optional: per-license risk check (e.g. apache-2.0, llama-3).", + ), + monthly_active_users: int = typer.Option( + 0, + "--monthly-active-users", + "--mau", + help="Expected MAU for the per-license risk check (default 0).", + ), +) -> None: + """Recommend a license-clean base + flag downstream risk (v0.64.0).""" + try: + rec = advise_license_for_target(target) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + table = Table(title=f"License advisor — target: {rec.target}") + table.add_column("Category") + table.add_column("Licenses") + table.add_row( + "[green]Recommended[/]", + ", ".join(escape(lic) for lic in rec.recommended_licenses), + ) + table.add_row( + "[red]Forbidden[/]", + ", ".join(escape(lic) for lic in rec.forbidden_licenses), + ) + console.print(table) + console.print( + Panel(escape(rec.reason), title="Reason", border_style="dim") + ) + + if license_id: + try: + risk = flag_downstream_risk( + license_id=license_id, + target=target, + monthly_active_users=monthly_active_users, + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + border = { + "ok": "green", + "warn": "yellow", + "block": "red", + }.get(risk.severity, "yellow") + console.print( + Panel( + f"[bold]severity: {escape(risk.severity)}[/]\n" + f"{escape(risk.reason)}", + title=f"Risk check: {escape(license_id)}", + border_style=border, + ) + ) + if risk.severity == "block": + raise typer.Exit(3) + + +__all__ = ["license_advisor_cmd"] diff --git a/soup_cli/commands/plan.py b/soup_cli/commands/plan.py new file mode 100644 index 0000000..c4a417c --- /dev/null +++ b/soup_cli/commands/plan.py @@ -0,0 +1,206 @@ +"""`soup plan` — Terraform-shape pre-flight summary (v0.64.0 Part B). + +Renders the cost / ETA / SHA / peak-VRAM summary for a planned training +run and writes ``soup.tfstate`` for ``soup apply`` to consult. +""" + +from __future__ import annotations + +import datetime as _dt + +import typer +import yaml +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +from soup_cli.utils.paths import is_under_cwd +from soup_cli.utils.terraform_plan import ( + DEFAULT_STATE_FILE, + TrainingState, + build_plan, + write_state, +) + +console = Console() + + +def _load_yaml_config(path: str) -> dict: + import os + import stat as _stat + + if not isinstance(path, str): + raise TypeError(f"config path must be str, got {type(path).__name__}") + if "\x00" in path: + raise ValueError("config path must not contain null bytes") + if not is_under_cwd(path): + raise ValueError(f"config {path!r} is outside cwd") + # TOCTOU defence: reject symlink at the YAML path BEFORE open so a + # pre-placed `soup.yaml -> /etc/shadow` cannot redirect the read. + if os.path.lexists(path): + st = os.lstat(path) + if _stat.S_ISLNK(st.st_mode): + raise ValueError("config path must not be a symlink") + with open(path, encoding="utf-8") as fh: + data = yaml.safe_load(fh) + if not isinstance(data, dict): + raise ValueError("config must parse to a dict") + return data + + +def plan_cmd( + config: str = typer.Option( + "soup.yaml", "--config", "-c", help="Path to soup.yaml." + ), + state_file: str = typer.Option( + DEFAULT_STATE_FILE, + "--state", + help="Path to write the state file (default: ./soup.tfstate).", + ), +) -> None: + """Render a pre-flight training plan + write ``soup.tfstate``.""" + try: + cfg = _load_yaml_config(config) + except FileNotFoundError: + console.print(f"[red]Config not found: {escape(config)}[/]") + raise typer.Exit(1) from None + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + plan = build_plan(cfg) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + state = TrainingState( + plan=plan, + applied=False, + applied_at=None, + run_id=None, + ) + try: + write_state(state, state_file) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + table = Table(title="Training plan") + table.add_column("Field") + table.add_column("Value") + table.add_row("base", escape(plan.base)) + table.add_row("task", escape(plan.task)) + table.add_row("config_sha", escape(plan.config_sha[:16] + "...")) + table.add_row("dataset_sha", escape(plan.dataset_sha[:16] + "...")) + table.add_row("estimated_cost", f"${plan.estimated_cost_usd:.4f}") + table.add_row("estimated_minutes", f"{plan.estimated_minutes:.1f}") + table.add_row("peak_vram_gb", f"{plan.peak_vram_gb:.1f}") + table.add_row("spot_price_usd_per_hour", f"${plan.spot_price_usd_per_hour:.2f}") + console.print(table) + console.print( + Panel( + f"[green]Plan written to {escape(state_file)}.[/]\n" + "Review the numbers, then run `soup apply` to execute.", + title="plan", + border_style="green", + ) + ) + + +def apply_cmd( + config: str = typer.Option( + "soup.yaml", "--config", "-c", help="Path to soup.yaml." + ), + state_file: str = typer.Option( + DEFAULT_STATE_FILE, + "--state", + help="Path to the state file written by `soup plan`.", + ), + dry_run: bool = typer.Option( + False, + "--dry-run", + help="Verify drift and exit without invoking training.", + ), +) -> None: + """Execute the planned training run, refusing on drift (v0.64.0).""" + from soup_cli.utils.terraform_plan import build_plan, detect_drift, read_state + + try: + cfg = _load_yaml_config(config) + except FileNotFoundError: + console.print(f"[red]Config not found: {escape(config)}[/]") + raise typer.Exit(1) from None + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + state = read_state(state_file) + except FileNotFoundError: + console.print( + f"[red]No state file at {escape(state_file)}; run `soup plan` first.[/]" + ) + raise typer.Exit(1) from None + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + plan_now = build_plan(cfg) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + drift = detect_drift(state, plan_now) + if drift.has_drift: + fields = ", ".join(drift.changed_fields) + console.print( + Panel( + f"[red]Drift detected.[/] Plan no longer matches state: {escape(fields)}\n" + f"Run `soup plan` again to refresh, then re-apply.", + title="apply", + border_style="red", + ) + ) + raise typer.Exit(3) + + if dry_run: + console.print( + Panel( + f"[green]Dry-run.[/] No drift detected; would proceed.\n" + f"Expected cost: ${plan_now.estimated_cost_usd:.4f} | " + f"ETA: {plan_now.estimated_minutes:.1f} min", + title="apply", + border_style="green", + ) + ) + return + + # Mark state applied; the actual `soup train` invocation is left to + # the operator. Future v0.64.1 may inline a subprocess call. + new_state = TrainingState( + plan=state.plan, + applied=True, + applied_at=_dt.datetime.now(_dt.timezone.utc).isoformat(), + run_id=None, + ) + try: + write_state(new_state, state_file) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + Panel( + f"[green]Plan locked in {escape(state_file)}.[/]\n" + f"Now run: [bold]soup train --config {escape(config)}[/]\n" + "(The live `apply -> train` subprocess handoff lands in v0.64.1.)", + title="apply", + border_style="green", + ) + ) + + +__all__ = ["apply_cmd", "plan_cmd"] diff --git a/soup_cli/commands/tunability.py b/soup_cli/commands/tunability.py new file mode 100644 index 0000000..1af43da --- /dev/null +++ b/soup_cli/commands/tunability.py @@ -0,0 +1,199 @@ +"""`soup tunability` — probe-train candidate bases + report Pareto frontier. + +CLI surface for the v0.64.0 Part A pre-flight probe. Live LoRA probe lands +in v0.64.1 (see ``utils/tunability._default_probe`` docstring). +""" + +from __future__ import annotations + +from typing import Optional + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +from soup_cli.utils.paths import is_under_cwd +from soup_cli.utils.tunability import ( + DEFAULT_CANDIDATES, + CandidateBase, + run_tunability, + write_report, +) + +console = Console() + + +def _resolve_candidates(names: Optional[str]) -> tuple[CandidateBase, ...]: + """Resolve a comma-separated name list against ``DEFAULT_CANDIDATES``.""" + if not names: + return DEFAULT_CANDIDATES + requested = [n.strip().lower() for n in names.split(",") if n.strip()] + if not requested: + return DEFAULT_CANDIDATES + by_name = {c.name.lower(): c for c in DEFAULT_CANDIDATES} + resolved: list[CandidateBase] = [] + for name in requested: + if name not in by_name: + available = ", ".join(sorted(by_name.keys())) + raise typer.BadParameter( + f"unknown candidate {name!r}; known: {available}" + ) + resolved.append(by_name[name]) + return tuple(resolved) + + +def tunability_cmd( + dataset: Optional[str] = typer.Option( + None, + "--dataset", + "-d", + help="Path to held-out JSONL slice used for the probe.", + ), + candidates: Optional[str] = typer.Option( + None, + "--candidates", + help="Comma-separated subset of default candidates by name.", + ), + probe_steps: int = typer.Option( + 100, + "--probe-steps", + help="Steps per candidate probe (10-10000).", + ), + holdout_size: int = typer.Option( + 64, + "--holdout-size", + help="Number of holdout rows used for delta scoring (10-100000).", + ), + output: Optional[str] = typer.Option( + None, + "--output", + "-o", + help="Optional path for the JSON report.", + ), + plan_only: bool = typer.Option( + False, + "--plan-only", + help="Print planned candidates + cost estimate, exit without probing.", + ), + list_only: bool = typer.Option( + False, + "--list", + help="List built-in candidate catalogue + exit.", + ), +) -> None: + """Probe-train candidate bases + report Pareto frontier (v0.64.0).""" + if list_only: + table = Table(title="Default tunability candidates") + table.add_column("Name") + table.add_column("Repo") + table.add_column("Params (B)", justify="right") + table.add_column("License") + for c in DEFAULT_CANDIDATES: + table.add_row( + escape(c.name), + escape(c.repo_id), + f"{c.params_b:.2f}", + escape(c.license_id), + ) + console.print(table) + return + + if not dataset: + console.print( + "[red]--dataset is required (use --list to see the catalogue, " + "--plan-only to dry-run).[/]" + ) + raise typer.Exit(2) + + # cwd containment + null-byte rejection on dataset + if "\x00" in dataset: + console.print("[red]dataset path must not contain null bytes[/]") + raise typer.Exit(2) + if not is_under_cwd(dataset): + console.print(f"[red]dataset {escape(dataset)!r} is outside cwd[/]") + raise typer.Exit(2) + + try: + cands = _resolve_candidates(candidates) + except typer.BadParameter as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + if plan_only: + table = Table(title="Planned tunability sweep") + table.add_column("Name") + table.add_column("Params (B)", justify="right") + table.add_column("License") + for c in cands: + table.add_row( + escape(c.name), + f"{c.params_b:.2f}", + escape(c.license_id), + ) + console.print(table) + console.print( + Panel( + f"[yellow]Plan-only.[/] Would run {len(cands)} probes " + f"x {probe_steps} steps on holdout={holdout_size}.\n" + f"Live LoRA probe lands in v0.64.1.", + title="tunability", + border_style="yellow", + ) + ) + return + + try: + report = run_tunability( + candidates=cands, + dataset_path=dataset, + probe_steps=probe_steps, + holdout_size=holdout_size, + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + table = Table(title="Tunability sweep") + table.add_column("Name") + table.add_column("Delta", justify="right") + table.add_column("Wall (s)", justify="right") + table.add_column("Cost ($)", justify="right") + table.add_column("License") + on_frontier = {r.candidate.name for r in report.frontier} + for r in report.results: + marker = "[green]*[/]" if r.candidate.name in on_frontier else " " + table.add_row( + f"{marker} {escape(r.candidate.name)}", + f"{r.delta:+.4f}", + f"{r.wall_clock_seconds:.0f}", + f"{r.estimated_cost_usd:.4f}", + escape(r.candidate.license_id), + ) + console.print(table) + console.print( + Panel( + f"Pareto frontier: {len(report.frontier)} / {len(report.results)} " + "candidates. Delta is base_loss - probe_loss (higher = better).", + title="tunability", + border_style="green", + ) + ) + + if output: + if "\x00" in output: + console.print("[red]output path must not contain null bytes[/]") + raise typer.Exit(2) + if not is_under_cwd(output): + console.print(f"[red]output {escape(output)!r} is outside cwd[/]") + raise typer.Exit(2) + try: + write_report(report, output) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + console.print(f"[green]Report written to {escape(output)}[/]") + + +__all__ = ["tunability_cmd"] diff --git a/soup_cli/utils/completions.py b/soup_cli/utils/completions.py new file mode 100644 index 0000000..0d1ef24 --- /dev/null +++ b/soup_cli/utils/completions.py @@ -0,0 +1,194 @@ +"""Shell completion script generators + dynamic value completers. + +`soup completions ` emits a sourceable bash / zsh / fish script. +The dynamic completers (``complete_recipe_name`` / +``complete_target_modules``) are exposed for use as +``shell_complete=...`` callbacks on Typer options. + +Live config introspection (probe the operator's actual ``base`` model +for its layer names) lands in v0.64.1; v0.64.0 ships canonical Llama- +shape defaults that cover ~80% of common bases. +""" + +from __future__ import annotations + +from typing import List, Optional + +SUPPORTED_SHELLS = frozenset({"bash", "zsh", "fish"}) +_MAX_SHELL_LEN = 32 + +# Canonical attention/mlp module names that cover Llama / Qwen / Mistral +# / Gemma / Phi families. When ``base`` is supplied and we can probe its +# config, we'd return only what's actually there — that lookup is the +# v0.64.1 deliverable. +_DEFAULT_TARGET_MODULES: tuple[str, ...] = ( + "q_proj", + "k_proj", + "v_proj", + "o_proj", + "gate_proj", + "up_proj", + "down_proj", + "lm_head", + "embed_tokens", +) + + +def validate_shell(value: object) -> str: + """Normalise + validate a shell name against ``SUPPORTED_SHELLS``.""" + if isinstance(value, bool): + raise TypeError("shell must be str, not bool") + if not isinstance(value, str): + raise TypeError(f"shell must be str, got {type(value).__name__}") + if not value: + raise ValueError("shell must be non-empty") + if "\x00" in value: + raise ValueError("shell must not contain null bytes") + if len(value) > _MAX_SHELL_LEN: + raise ValueError(f"shell name too long (> {_MAX_SHELL_LEN} chars)") + normalised = value.lower().strip() + if normalised not in SUPPORTED_SHELLS: + allowed = ", ".join(sorted(SUPPORTED_SHELLS)) + raise ValueError(f"unknown shell {value!r}; known: {allowed}") + return normalised + + +def render_bash_script() -> str: + """Render a bash completion script for the `soup` CLI. + + Defers to Typer/Click's built-in ``COMPLETE`` env machinery so the + completion stays in sync with the live Typer app (new commands / + flags are picked up automatically). + """ + return ( + "# Soup bash completion (v0.64.0)\n" + "# Source this file from ~/.bashrc:\n" + "# eval \"$(soup completions bash)\"\n" + "_soup_complete() {\n" + " local IFS=$'\\n'\n" + " local response\n" + " response=$(env COMP_WORDS=\"${COMP_WORDS[*]}\" \\\n" + " COMP_CWORD=$COMP_CWORD \\\n" + " _SOUP_COMPLETE=bash_complete \\\n" + " $1 2>/dev/null)\n" + " for completion in $response; do\n" + " IFS=',' read type value <<< \"$completion\"\n" + " if [[ $type == 'plain' ]]; then\n" + " COMPREPLY+=(\"$value\")\n" + " fi\n" + " done\n" + " return 0\n" + "}\n" + "complete -o nosort -F _soup_complete soup\n" + ) + + +def render_zsh_script() -> str: + """Render a zsh completion script for `soup`.""" + return ( + "#compdef soup\n" + "# Soup zsh completion (v0.64.0)\n" + "# Source this file from ~/.zshrc:\n" + "# eval \"$(soup completions zsh)\"\n" + "_soup_complete() {\n" + " local -a completions\n" + " local -a completions_with_descriptions\n" + " local -a response\n" + " response=(\"${(@f)$(env COMP_WORDS=\"${words[*]}\" \\\n" + " COMP_CWORD=$((CURRENT-1)) \\\n" + " _SOUP_COMPLETE=zsh_complete soup 2>/dev/null)}\")\n" + " for type_value in \"${response[@]}\"; do\n" + " IFS=',' read -r -A parts <<< \"$type_value\"\n" + " completions+=(\"${parts[2]}\")\n" + " done\n" + " _describe '' completions\n" + "}\n" + "compdef _soup_complete soup\n" + ) + + +def render_fish_script() -> str: + """Render a fish completion script for `soup`.""" + return ( + "# Soup fish completion (v0.64.0)\n" + "# Source this file from ~/.config/fish/completions/soup.fish\n" + "function _soup_complete\n" + " set -l response (env _SOUP_COMPLETE=fish_complete \\\n" + " COMP_WORDS=(commandline -cp) \\\n" + " COMP_CWORD=(commandline -t) soup 2>/dev/null)\n" + " for item in $response\n" + " set parts (string split \",\" $item)\n" + " echo $parts[2]\n" + " end\n" + "end\n" + "complete -c soup -f -a \"(_soup_complete)\"\n" + ) + + +def render_completion_script(shell: object) -> str: + """Dispatch on shell name. Validates + renders one of the three scripts.""" + normalised = validate_shell(shell) + if normalised == "bash": + return render_bash_script() + if normalised == "zsh": + return render_zsh_script() + if normalised == "fish": + return render_fish_script() + # Unreachable thanks to ``validate_shell``; defensive default. + raise ValueError(f"unhandled shell {normalised!r}") + + +def complete_recipe_name(prefix: object) -> List[str]: + """Suggest recipe names matching ``prefix`` (case-insensitive). + + Backed by ``soup_cli.recipes.catalog.list_recipes`` (lazy import). + """ + if isinstance(prefix, bool): + raise TypeError("prefix must be str, not bool") + if not isinstance(prefix, str): + raise TypeError(f"prefix must be str, got {type(prefix).__name__}") + if "\x00" in prefix: + # Defensive: shell completers should never raise. + return [] + try: + from soup_cli.recipes.catalog import RECIPES + except ImportError: # pragma: no cover + return [] + p = prefix.lower() + return [name for name in RECIPES if name.lower().startswith(p)] + + +def complete_target_modules( + prefix: object, + *, + base: Optional[str] = None, +) -> List[str]: + """Suggest ``target_modules`` values for the chosen ``base`` model. + + v0.64.0 returns the canonical Llama-shape defaults filtered by + ``prefix``. Live per-base introspection (load the HF config, walk + its module tree) is the v0.64.1 deliverable. + """ + if isinstance(prefix, bool): + raise TypeError("prefix must be str, not bool") + if not isinstance(prefix, str): + raise TypeError(f"prefix must be str, got {type(prefix).__name__}") + if base is not None and not isinstance(base, str): + raise TypeError(f"base must be str | None, got {type(base).__name__}") + if "\x00" in prefix: + return [] + # base-specific introspection deferred to v0.64.1; fall through to + # the canonical defaults for now. + return [m for m in _DEFAULT_TARGET_MODULES if m.startswith(prefix)] + + +__all__ = [ + "SUPPORTED_SHELLS", + "complete_recipe_name", + "complete_target_modules", + "render_bash_script", + "render_completion_script", + "render_fish_script", + "render_zsh_script", + "validate_shell", +] diff --git a/soup_cli/utils/env_lock.py b/soup_cli/utils/env_lock.py new file mode 100644 index 0000000..f6676cc --- /dev/null +++ b/soup_cli/utils/env_lock.py @@ -0,0 +1,366 @@ +"""`soup env` — hermetic env lockfile + ABI-mismatch detection (v0.64.0 Part C). + +The "CUDA hell" problem: a fine-tune that worked on Friday breaks on +Monday because PyPI silently upgraded ``transformers`` past the trainer's +compat band, or because the box rebuilt with a different CUDA. v0.34 +``soup doctor`` surfaces some of this; v0.64 makes it lockable. + +``snapshot_env`` reads Python + CUDA + key package versions from the +current interpreter via ``importlib.metadata`` (no network, no shell-out +to pip beyond what stdlib already does). ``write_lock`` persists the +snapshot as ``soup-env.lock``. ``check_abi_compat`` compares two locks +and produces an ``AbiCheck`` report listing ABI-sensitive drifts. + +Live full uv/nix-backed install + recreate lands in v0.64.1; v0.64.0 +ships the schema, snapshotter, comparator, and CLI surface so an +operator can capture an environment + detect drift today. + +Public surface: +- ``EnvEntry`` frozen dataclass (name / version / source). +- ``EnvLock`` frozen dataclass (soup version / python / platform / cuda + / entries tuple / timestamp). +- ``AbiCheck`` frozen dataclass (ok / drift_count / changes tuple). +- ``TRACKED_PACKAGES`` tuple of names whose drift is ABI-relevant. +- ``snapshot_env()`` -> EnvLock. +- ``write_lock(lock, path)`` / ``read_lock(path)`` -> atomic JSON. +- ``check_abi_compat(a, b)`` -> AbiCheck. +- ``DEFAULT_LOCK_FILE = "soup-env.lock"``. +""" + +from __future__ import annotations + +import datetime as _dt +import json +import os +import platform as _platform +import sys +from dataclasses import dataclass +from typing import Optional, Tuple + +from soup_cli.utils.paths import atomic_write_text, is_under_cwd + +DEFAULT_LOCK_FILE = "soup-env.lock" +_MAX_NAME_LEN = 256 +_MAX_VERSION_LEN = 128 +_MAX_PLATFORM_LEN = 256 +_MAX_PY_VERSION_LEN = 64 +_MAX_CUDA_VERSION_LEN = 64 +_MAX_ENTRIES = 4096 + +# Source allowlist — defends against schema drift on read. +_VALID_SOURCES = frozenset({"pip", "conda", "system", "wheel", "unknown"}) + +# ABI-sensitive packages — drift here is most likely to break training. +TRACKED_PACKAGES: Tuple[str, ...] = ( + "torch", + "transformers", + "peft", + "trl", + "accelerate", + "datasets", + "bitsandbytes", + "huggingface-hub", + "flash-attn", + "xformers", + "deepspeed", + "unsloth", + "vllm", + "sentencepiece", + "tokenizers", +) + + +def _check_non_empty_str(value: object, fld: str, *, max_len: int) -> None: + if isinstance(value, bool): + raise TypeError(f"{fld} must be str, not bool") + if not isinstance(value, str): + raise TypeError(f"{fld} must be str, got {type(value).__name__}") + if not value: + raise ValueError(f"{fld} must be non-empty") + if "\x00" in value: + raise ValueError(f"{fld} must not contain null bytes") + if len(value) > max_len: + raise ValueError(f"{fld} too long (> {max_len} chars)") + + +@dataclass(frozen=True) +class EnvEntry: + """One package -> version row in the lockfile.""" + + name: str + version: str + source: str + + def __post_init__(self) -> None: + _check_non_empty_str(self.name, "name", max_len=_MAX_NAME_LEN) + _check_non_empty_str(self.version, "version", max_len=_MAX_VERSION_LEN) + _check_non_empty_str(self.source, "source", max_len=32) + if self.source not in _VALID_SOURCES: + allowed = ", ".join(sorted(_VALID_SOURCES)) + raise ValueError( + f"source must be one of {{{allowed}}}, got {self.source!r}" + ) + + +@dataclass(frozen=True) +class EnvLock: + """Captured snapshot of the running environment.""" + + soup_version: str + python_version: str + platform: str + cuda_version: Optional[str] + entries: Tuple[EnvEntry, ...] + created_at: str + + def __post_init__(self) -> None: + _check_non_empty_str(self.soup_version, "soup_version", max_len=64) + _check_non_empty_str( + self.python_version, "python_version", max_len=_MAX_PY_VERSION_LEN + ) + _check_non_empty_str(self.platform, "platform", max_len=_MAX_PLATFORM_LEN) + if self.cuda_version is not None: + _check_non_empty_str( + self.cuda_version, "cuda_version", max_len=_MAX_CUDA_VERSION_LEN + ) + if not isinstance(self.entries, tuple): + raise TypeError("entries must be a tuple of EnvEntry") + if len(self.entries) > _MAX_ENTRIES: + raise ValueError( + f"too many entries ({len(self.entries)} > {_MAX_ENTRIES})" + ) + for entry in self.entries: + if not isinstance(entry, EnvEntry): + raise TypeError("every entry must be EnvEntry") + _check_non_empty_str(self.created_at, "created_at", max_len=64) + + +@dataclass(frozen=True) +class AbiCheck: + """Outcome of an ABI-compat comparison between two ``EnvLock``s.""" + + ok: bool + drift_count: int + changes: Tuple[str, ...] + + def __post_init__(self) -> None: + if not isinstance(self.ok, bool): + raise TypeError("ok must be bool") + if isinstance(self.drift_count, bool) or not isinstance(self.drift_count, int): + raise TypeError("drift_count must be int") + if self.drift_count < 0: + raise ValueError("drift_count must be >= 0") + if not isinstance(self.changes, tuple): + raise TypeError("changes must be a tuple of str") + for entry in self.changes: + if not isinstance(entry, str): + raise TypeError("changes entries must be str") + + +def _detect_cuda_version() -> Optional[str]: + """Best-effort CUDA version probe via env / nvidia-smi / torch. + + Returns ``None`` if no CUDA found. Lazy-imports torch only if it's + already in ``sys.modules`` so we never pay the import on a CPU box. + + Path-parse handles POSIX (``/usr/local/cuda-12.1``) AND Windows + (``C:\\Program Files\\...\\CUDA\\v12.1``) — splits on every common + separator and strips a leading ``v`` when present. + """ + cuda_env = os.environ.get("CUDA_VERSION") or os.environ.get("CUDA_HOME") + if cuda_env: + # Split on both POSIX and Windows separators so e.g. + # `C:\Program Files\...\CUDA\v12.1` produces `v12.1` as the last + # token instead of one giant path. + tokens = cuda_env.replace("\\", "/").split("/") + for tok in reversed(tokens): + cleaned = tok.lstrip("v") # strip Windows `v` prefix + if cleaned and cleaned[0].isdigit(): + return cleaned + if "-" in tok: + tail = tok.rsplit("-", 1)[-1].lstrip("v") + if tail and tail[0].isdigit(): + return tail + # If torch is already imported, ask it. + torch_mod = sys.modules.get("torch") + if torch_mod is not None: + ver = getattr(getattr(torch_mod, "version", None), "cuda", None) + if isinstance(ver, str) and ver: + return ver + return None + + +def _detect_package_version(name: str) -> Optional[str]: + """Read installed-package version via ``importlib.metadata``. + + Returns ``None`` for either "not installed" (``PackageNotFoundError``) + or "metadata parse failed" (``OSError`` / ``ValueError``). All other + exceptions propagate to surface real bugs (matches v0.33.0 #47 / + v0.40.3 #33 narrow-except policy). + """ + try: + from importlib.metadata import PackageNotFoundError, version + except ImportError: # pragma: no cover — py < 3.8 + return None + try: + return version(name) + except PackageNotFoundError: + return None + except (OSError, ValueError): # pragma: no cover — corrupt metadata + return None + + +def snapshot_env() -> EnvLock: + """Capture the current Python/CUDA/package versions as an ``EnvLock``.""" + import soup_cli + + py_v = ".".join(str(x) for x in sys.version_info[:3]) + plat = f"{_platform.system().lower()}-{_platform.machine().lower()}" + cuda_v = _detect_cuda_version() + + entries: list[EnvEntry] = [] + for name in TRACKED_PACKAGES: + ver = _detect_package_version(name) + if ver is None: + continue + try: + entries.append(EnvEntry(name=name, version=ver, source="pip")) + except (TypeError, ValueError): + # Skip malformed version strings. + continue + + return EnvLock( + soup_version=soup_cli.__version__, + python_version=py_v, + platform=plat, + cuda_version=cuda_v, + entries=tuple(entries), + created_at=_dt.datetime.now(_dt.timezone.utc).isoformat(), + ) + + +def _entry_to_dict(e: EnvEntry) -> dict: + return {"name": e.name, "version": e.version, "source": e.source} + + +def _entry_from_dict(d: dict) -> EnvEntry: + if not isinstance(d, dict): + raise ValueError("entry must be a dict") + return EnvEntry( + name=str(d.get("name", "")), + version=str(d.get("version", "")), + source=str(d.get("source", "unknown")), + ) + + +def write_lock(lock: EnvLock, path: str) -> None: + """Atomically write an ``EnvLock`` to JSON under cwd containment.""" + if not isinstance(lock, EnvLock): + raise TypeError(f"lock must be EnvLock, got {type(lock).__name__}") + payload = { + "schema_version": "1", + "soup_version": lock.soup_version, + "python_version": lock.python_version, + "platform": lock.platform, + "cuda_version": lock.cuda_version, + "entries": [_entry_to_dict(e) for e in lock.entries], + "created_at": lock.created_at, + } + text = json.dumps(payload, indent=2, ensure_ascii=False, allow_nan=False) + atomic_write_text(text, path, prefix=".envlock.", field="env lock") + + +def read_lock(path: str) -> EnvLock: + """Read a previously-written ``EnvLock`` JSON. + + Containment + symlink rejection BEFORE existence probe so a crafted + path cannot distinguish "outside cwd" from "missing" (matches v0.55.0 + / v0.62.0 ordering policy). + """ + import stat as _stat + + if not isinstance(path, str): + raise TypeError(f"path must be str, got {type(path).__name__}") + if "\x00" in path: + raise ValueError("lock path must not contain null bytes") + if not is_under_cwd(path): + raise ValueError(f"lock {path!r} is outside cwd") + if os.path.lexists(path): + st = os.lstat(path) + if _stat.S_ISLNK(st.st_mode): + raise ValueError("lock path must not be a symlink") + if not os.path.isfile(path): + raise FileNotFoundError(path) + with open(path, encoding="utf-8") as fh: + payload = json.load(fh) + if not isinstance(payload, dict): + raise ValueError("lock root must be a dict") + entries_raw = payload.get("entries", []) + if not isinstance(entries_raw, list): + raise ValueError("entries must be a list") + entries = tuple(_entry_from_dict(e) for e in entries_raw) + return EnvLock( + soup_version=str(payload.get("soup_version", "")), + python_version=str(payload.get("python_version", "")), + platform=str(payload.get("platform", "unknown")), + cuda_version=payload.get("cuda_version"), + entries=entries, + created_at=str(payload.get("created_at", "")), + ) + + +def check_abi_compat(a: EnvLock, b: EnvLock) -> AbiCheck: + """Compare two EnvLocks; flag ABI-sensitive drifts. + + Drift sources (in order of impact): + 1. Python minor version change. + 2. Platform string change. + 3. CUDA version change. + 4. Any TRACKED_PACKAGES version change. + + Returns ``AbiCheck(ok=True)`` iff *zero* drift sources fire. + """ + if not isinstance(a, EnvLock): + raise TypeError(f"a must be EnvLock, got {type(a).__name__}") + if not isinstance(b, EnvLock): + raise TypeError(f"b must be EnvLock, got {type(b).__name__}") + + changes: list[str] = [] + # Python minor: 3.10.x vs 3.11.x is ABI-different + a_py = ".".join(a.python_version.split(".")[:2]) + b_py = ".".join(b.python_version.split(".")[:2]) + if a_py != b_py: + changes.append(f"python: {a.python_version} -> {b.python_version}") + + if a.platform != b.platform: + changes.append(f"platform: {a.platform} -> {b.platform}") + + if (a.cuda_version or None) != (b.cuda_version or None): + changes.append(f"cuda: {a.cuda_version} -> {b.cuda_version}") + + by_a = {e.name.lower(): e.version for e in a.entries} + by_b = {e.name.lower(): e.version for e in b.entries} + for name in sorted(set(by_a) | set(by_b)): + va = by_a.get(name) + vb = by_b.get(name) + if va != vb: + changes.append(f"{name}: {va} -> {vb}") + + return AbiCheck( + ok=not changes, + drift_count=len(changes), + changes=tuple(changes), + ) + + +__all__ = [ + "AbiCheck", + "DEFAULT_LOCK_FILE", + "EnvEntry", + "EnvLock", + "TRACKED_PACKAGES", + "check_abi_compat", + "read_lock", + "snapshot_env", + "write_lock", +] diff --git a/soup_cli/utils/hardware_fit.py b/soup_cli/utils/hardware_fit.py new file mode 100644 index 0000000..8e73dc3 --- /dev/null +++ b/soup_cli/utils/hardware_fit.py @@ -0,0 +1,350 @@ +"""Hardware-fit calculator: static analytical predictor of peak VRAM. + +v0.40.3 #64 added a live CUDA OOM probe; that's correct but slow (one +real forward+backward per candidate). v0.64 adds a fast analytical +predictor that takes (params, seq_len, batch_size, optimizer, quant, +peft, gradient_checkpointing) and outputs a five-bucket VRAM breakdown. +Used by ``decide_hardware_fit`` to refuse a training launch when the +predicted peak (with ``VRAM_SAFETY_MARGIN=10%`` headroom) would not fit. + +The math is intentionally conservative — operators can supply +``--allow-oom-attempt`` to bypass the gate if they want to try anyway +(opt-out, not opt-in, per the v0.64 Part D plan). + +Public surface: +- ``VRAM_SAFETY_MARGIN = 0.10``. +- ``validate_seq_len(v)`` / ``validate_batch_size(v)``. +- ``HardwareFitInput`` frozen dataclass. +- ``VRAMBreakdown`` frozen dataclass + ``total_gb`` property. +- ``HardwareFitReport`` frozen dataclass. +- ``estimate_peak_vram_gb(inp)`` -> ``VRAMBreakdown``. +- ``decide_hardware_fit(inp, *, available_vram_gb)`` -> ``HardwareFitReport``. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass + +VRAM_SAFETY_MARGIN = 0.10 # 10% headroom + +_MIN_SEQ_LEN = 64 +_MAX_SEQ_LEN = 1_048_576 +_MIN_BATCH = 1 +_MAX_BATCH = 1024 +_MAX_PARAMS_B = 1000.0 + +# Closed allowlists — schema-shape rejection, no surprises. +_VALID_QUANT = frozenset({ + "none", "4bit", "8bit", "fp8", "gptq", "awq", "aqlm", "eetq", "mxfp4", +}) +_VALID_PEFT = frozenset({"full", "lora", "dora", "qlora"}) +_VALID_OPTIMIZERS = frozenset({ + "adamw_torch", "adamw_torch_fused", "adafactor", "sgd", + "adamw_bnb_8bit", "paged_adamw_8bit", "lion_8bit", + "lomo", "adalomo", "schedule_free_adamw", +}) + + +def validate_seq_len(value: object) -> int: + if isinstance(value, bool): + raise TypeError("seq_len must be int, not bool") + if not isinstance(value, int): + raise TypeError(f"seq_len must be int, got {type(value).__name__}") + if not (_MIN_SEQ_LEN <= value <= _MAX_SEQ_LEN): + raise ValueError( + f"seq_len must be in [{_MIN_SEQ_LEN}, {_MAX_SEQ_LEN}], got {value}" + ) + return value + + +def validate_batch_size(value: object) -> int: + if isinstance(value, bool): + raise TypeError("batch_size must be int, not bool") + if not isinstance(value, int): + raise TypeError(f"batch_size must be int, got {type(value).__name__}") + if not (_MIN_BATCH <= value <= _MAX_BATCH): + raise ValueError( + f"batch_size must be in [{_MIN_BATCH}, {_MAX_BATCH}], got {value}" + ) + return value + + +@dataclass(frozen=True) +class HardwareFitInput: + """Inputs needed to predict peak VRAM.""" + + params_b: float + seq_len: int + batch_size: int + optimizer: str + quant: str + peft: str + gradient_checkpointing: bool + + def __post_init__(self) -> None: + if isinstance(self.params_b, bool): + raise TypeError("params_b must be a number, not bool") + if not isinstance(self.params_b, (int, float)): + raise TypeError( + f"params_b must be a number, got {type(self.params_b).__name__}" + ) + if not math.isfinite(float(self.params_b)): + raise ValueError("params_b must be finite") + if self.params_b <= 0 or self.params_b > _MAX_PARAMS_B: + raise ValueError( + f"params_b must be in (0, {_MAX_PARAMS_B}], got {self.params_b}" + ) + validate_seq_len(self.seq_len) + validate_batch_size(self.batch_size) + if not isinstance(self.optimizer, str): + raise TypeError("optimizer must be str") + if self.optimizer not in _VALID_OPTIMIZERS: + raise ValueError( + f"unknown optimizer {self.optimizer!r}; " + f"known: {', '.join(sorted(_VALID_OPTIMIZERS))}" + ) + if not isinstance(self.quant, str): + raise TypeError("quant must be str") + if self.quant not in _VALID_QUANT: + raise ValueError( + f"unknown quant {self.quant!r}; " + f"known: {', '.join(sorted(_VALID_QUANT))}" + ) + if not isinstance(self.peft, str): + raise TypeError("peft must be str") + if self.peft not in _VALID_PEFT: + raise ValueError( + f"unknown peft {self.peft!r}; " + f"known: {', '.join(sorted(_VALID_PEFT))}" + ) + if not isinstance(self.gradient_checkpointing, bool): + raise TypeError("gradient_checkpointing must be bool") + + +@dataclass(frozen=True) +class VRAMBreakdown: + """Per-class peak VRAM in GB.""" + + weights_gb: float + optimizer_gb: float + gradients_gb: float + activations_gb: float + overhead_gb: float + + def __post_init__(self) -> None: + for fld in ( + "weights_gb", "optimizer_gb", "gradients_gb", + "activations_gb", "overhead_gb", + ): + val = getattr(self, fld) + if isinstance(val, bool): + raise TypeError(f"{fld} must be a number, not bool") + if not isinstance(val, (int, float)): + raise TypeError(f"{fld} must be a number") + if not math.isfinite(float(val)): + raise ValueError(f"{fld} must be finite") + if val < 0: + raise ValueError(f"{fld} must not be negative, got {val}") + + @property + def total_gb(self) -> float: + return float( + self.weights_gb + + self.optimizer_gb + + self.gradients_gb + + self.activations_gb + + self.overhead_gb + ) + + +@dataclass(frozen=True) +class HardwareFitReport: + """Outcome of a hardware-fit decision.""" + + ok: bool + peak_vram_gb: float + required_with_margin_gb: float + available_vram_gb: float + breakdown: VRAMBreakdown + reason: str + + def __post_init__(self) -> None: + if not isinstance(self.ok, bool): + raise TypeError("ok must be bool") + for fld in ("peak_vram_gb", "required_with_margin_gb", "available_vram_gb"): + val = getattr(self, fld) + if isinstance(val, bool): + raise TypeError(f"{fld} must be a number, not bool") + if not isinstance(val, (int, float)): + raise TypeError(f"{fld} must be a number") + if not math.isfinite(float(val)): + raise ValueError(f"{fld} must be finite") + if val < 0: + raise ValueError(f"{fld} must not be negative") + if not isinstance(self.breakdown, VRAMBreakdown): + raise TypeError("breakdown must be VRAMBreakdown") + if not isinstance(self.reason, str): + raise TypeError("reason must be str") + + +# Bytes-per-param multiplier by quant scheme. Approximate; the deep +# truth lives in upstream BNB / GPTQ / AWQ docs. +_BYTES_PER_PARAM_BY_QUANT = { + "none": 2.0, # bf16/fp16 + "4bit": 0.5, + "8bit": 1.0, + "fp8": 1.0, + "gptq": 0.55, + "awq": 0.55, + "aqlm": 0.5, + "eetq": 1.0, + "mxfp4": 0.55, +} + +# Optimiser-state bytes per trainable parameter. AdamW = 8 (m + v fp32), +# 8-bit AdamW = 2, Adafactor = 4, SGD = 0 (momentum-less reference). +_OPTIM_BYTES_PER_PARAM = { + "adamw_torch": 8.0, + "adamw_torch_fused": 8.0, + "adafactor": 4.0, + "sgd": 0.0, + "adamw_bnb_8bit": 2.0, + "paged_adamw_8bit": 2.0, + "lion_8bit": 2.0, + "lomo": 0.0, + "adalomo": 2.0, + "schedule_free_adamw": 8.0, +} + + +def _trainable_param_fraction(peft: str) -> float: + """LoRA / DoRA / QLoRA train ~1% of parameters. Full = 100%.""" + if peft == "full": + return 1.0 + if peft in ("lora", "qlora", "dora"): + return 0.01 + return 1.0 # defensive default; schema gate already rejects unknown + + +def _activation_bytes(seq_len: int, batch_size: int, params_b: float) -> float: + """Per-batch activation memory. + + Very approximate: scales with seq_len × batch_size × hidden_size. + We treat hidden_size as ~ 64 * sqrt(params_b * 1e9 / 1024) — close + enough for an order-of-magnitude estimate at the 1B-70B band. + + Result is bounded by ``_MAX_ACTIVATIONS_BYTES`` (~1 EiB) so the + downstream ``VRAMBreakdown`` validator never sees a +Inf product + even at the schema max (seq=1M × batch=1024). + """ + hidden = 64.0 * math.sqrt(max(params_b * 1e9 / 1024.0, 1.0)) + raw = float(seq_len) * float(batch_size) * hidden * 4.0 + # Defensive: clamp absurd products so the math stays finite. + if not math.isfinite(raw) or raw > _MAX_ACTIVATIONS_BYTES: + return _MAX_ACTIVATIONS_BYTES + return raw + + +_MAX_ACTIVATIONS_BYTES = 1e18 # 1 EB sanity cap; far above any real GPU + + +def estimate_peak_vram_gb(inp: HardwareFitInput) -> VRAMBreakdown: + """Static analytical VRAM predictor. Returns a per-class breakdown.""" + if not isinstance(inp, HardwareFitInput): + raise TypeError( + f"inp must be HardwareFitInput, got {type(inp).__name__}" + ) + + params = float(inp.params_b) * 1e9 # absolute count + bytes_per = _BYTES_PER_PARAM_BY_QUANT.get(inp.quant, 2.0) + weights_b = params * bytes_per + + trainable_frac = _trainable_param_fraction(inp.peft) + trainable_params = params * trainable_frac + optim_b = trainable_params * _OPTIM_BYTES_PER_PARAM.get(inp.optimizer, 8.0) + # Gradients are fp32 of trainable params (4 bytes/param) under + # mixed-precision; under "none" quant we still keep fp32 grads. + gradients_b = trainable_params * 4.0 + + activations_b = _activation_bytes(inp.seq_len, inp.batch_size, inp.params_b) + if inp.gradient_checkpointing: + # Gradient checkpointing roughly halves activation memory. + activations_b *= 0.5 + + # Constant overhead: kernels / autograd graph / NCCL buffers / etc. + # Scales weakly with params. + overhead_b = 0.5e9 + 0.05 * weights_b + + return VRAMBreakdown( + weights_gb=weights_b / 1e9, + optimizer_gb=optim_b / 1e9, + gradients_gb=gradients_b / 1e9, + activations_gb=activations_b / 1e9, + overhead_gb=overhead_b / 1e9, + ) + + +def decide_hardware_fit( + inp: HardwareFitInput, + *, + available_vram_gb: float, +) -> HardwareFitReport: + """Decide whether the planned run fits under ``available_vram_gb``. + + Applies ``VRAM_SAFETY_MARGIN=10%`` headroom on top of the analytical + estimate. Refuse iff ``predicted * 1.1 > available``. + """ + if isinstance(available_vram_gb, bool): + raise TypeError("available_vram_gb must be a number, not bool") + if not isinstance(available_vram_gb, (int, float)): + raise TypeError( + f"available_vram_gb must be a number, got {type(available_vram_gb).__name__}" + ) + if not math.isfinite(float(available_vram_gb)): + raise ValueError("available_vram_gb must be finite") + if available_vram_gb < 0: + raise ValueError( + f"available_vram_gb must be >= 0, got {available_vram_gb}" + ) + + breakdown = estimate_peak_vram_gb(inp) + peak = breakdown.total_gb + required = peak * (1.0 + VRAM_SAFETY_MARGIN) + if required <= available_vram_gb: + return HardwareFitReport( + ok=True, + peak_vram_gb=peak, + required_with_margin_gb=required, + available_vram_gb=float(available_vram_gb), + breakdown=breakdown, + reason=( + f"fits: peak {peak:.2f} GB + {VRAM_SAFETY_MARGIN:.0%} margin " + f"<= {available_vram_gb:.2f} GB available" + ), + ) + return HardwareFitReport( + ok=False, + peak_vram_gb=peak, + required_with_margin_gb=required, + available_vram_gb=float(available_vram_gb), + breakdown=breakdown, + reason=( + f"OOM risk: required {required:.2f} GB exceeds " + f"available {available_vram_gb:.2f} GB. " + f"Try --batch-size {max(1, inp.batch_size // 2)} or " + "--quantization 4bit or --gradient-checkpointing auto." + ), + ) + + +__all__ = [ + "VRAM_SAFETY_MARGIN", + "HardwareFitInput", + "HardwareFitReport", + "VRAMBreakdown", + "decide_hardware_fit", + "estimate_peak_vram_gb", + "validate_batch_size", + "validate_seq_len", +] diff --git a/soup_cli/utils/license_advisor.py b/soup_cli/utils/license_advisor.py new file mode 100644 index 0000000..e1b943f --- /dev/null +++ b/soup_cli/utils/license_advisor.py @@ -0,0 +1,328 @@ +"""License advisor (v0.64.0 Part F). + +Given a target deployment (``b2c``, ``defense``, ``embedded``), +recommend a license-clean base model + flag downstream risk for a +specific license id. Composes with v0.60 Part E +``license_matrix.check_license_compat`` for the merge-time gate. + +The matrix here is deliberately *coarse-grained*: it groups licenses +into ``recommended`` / ``forbidden`` for each deploy target, and the +``flag_downstream_risk`` helper applies a per-license risk heuristic +(e.g. Llama community license + > 700M MAU = block, per Meta's +acceptable-use policy). + +Live model-card scraping for license inference lands in v0.64.1; +v0.64.0 takes the license id from the operator (matches the v0.60 +``--license `` convention). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Tuple + +from soup_cli.utils.license_matrix import KNOWN_LICENSES, LICENSE_KINDS + +DEPLOY_TARGETS = frozenset({"b2c", "defense", "embedded"}) +_MAX_TARGET_LEN = 32 +_MAX_LICENSE_LEN = 128 +_VALID_SEVERITIES = frozenset({"ok", "warn", "block"}) + +# Llama community license: Meta requires a separate licence for products +# with > 700M monthly active users. We surface this as a downstream-risk +# block in B2C deployments. +_LLAMA_COMMUNITY_MAU_CAP = 700_000_000 + +# Tight allowlist of Llama-family community license ids. Defends against +# a future "llama-permissive" id wrongly tripping the MAU gate. +_LLAMA_COMMUNITY_LICENSES = frozenset({ + "llama-2", + "llama-3", + "llama-3.1", + "llama-3.2", + "llama-3.3", + "llama-community", +}) + + +def validate_deploy_target(value: object) -> str: + """Normalise + validate a deploy target name.""" + if isinstance(value, bool): + raise TypeError("target must be str, not bool") + if not isinstance(value, str): + raise TypeError(f"target must be str, got {type(value).__name__}") + if not value: + raise ValueError("target must be non-empty") + if "\x00" in value: + raise ValueError("target must not contain null bytes") + if len(value) > _MAX_TARGET_LEN: + raise ValueError(f"target too long (> {_MAX_TARGET_LEN} chars)") + normalised = value.lower().strip() + if normalised not in DEPLOY_TARGETS: + allowed = ", ".join(sorted(DEPLOY_TARGETS)) + raise ValueError(f"unknown target {value!r}; known: {allowed}") + return normalised + + +@dataclass(frozen=True) +class LicenseRecommendation: + """Output of ``advise_license_for_target``.""" + + target: str + recommended_licenses: Tuple[str, ...] + forbidden_licenses: Tuple[str, ...] + reason: str + + def __post_init__(self) -> None: + if not isinstance(self.target, str): + raise TypeError("target must be str") + if not isinstance(self.recommended_licenses, tuple): + raise TypeError("recommended_licenses must be a tuple of str") + if not isinstance(self.forbidden_licenses, tuple): + raise TypeError("forbidden_licenses must be a tuple of str") + if not isinstance(self.reason, str): + raise TypeError("reason must be str") + + +# Permissive baseline used as the b2c "recommended" set. Operators +# almost always want these. +_PERMISSIVE_RECOMMENDED: Tuple[str, ...] = ( + "apache-2.0", + "mit", + "bsd-3-clause", + "bsd-2-clause", + "isc", + "unlicense", +) + +# Non-commercial licenses — always forbidden for B2C. +_NC_LICENSES: Tuple[str, ...] = ( + "cc-by-nc-4.0", + "cc-by-nc-sa-4.0", + "cc-by-nc-nd-4.0", +) + +# Restricted-use model licenses (Llama / Gemma / Qwen / Mistral +# community licenses) — forbidden for defense per acceptable-use clauses. +_RESTRICTED_USE: Tuple[str, ...] = ( + "llama-2", + "llama-3", + "llama-3.1", + "llama-3.2", + "llama-3.3", + "llama-community", + "gemma", + "qwen-research", + "qwen-license", + "mistral-research", +) + +# Strong copyleft — forbidden for embedded (driver / firmware tree +# incompatible with GPL/AGPL closed-source distribution). +_STRONG_COPYLEFT: Tuple[str, ...] = ( + "gpl-2.0", + "gpl-3.0", + "agpl-3.0", +) + + +def advise_license_for_target(target: object) -> LicenseRecommendation: + """Return the per-target ``LicenseRecommendation``.""" + normalised = validate_deploy_target(target) + if normalised == "b2c": + return LicenseRecommendation( + target=normalised, + recommended_licenses=_PERMISSIVE_RECOMMENDED, + forbidden_licenses=_NC_LICENSES, + reason=( + "Consumer B2C: prefer broadly-permissive licenses; " + "non-commercial licenses are categorically forbidden." + ), + ) + if normalised == "defense": + # Defense forbids restricted-use clauses on top of NC. + forbidden = _NC_LICENSES + _RESTRICTED_USE + return LicenseRecommendation( + target=normalised, + recommended_licenses=_PERMISSIVE_RECOMMENDED, + forbidden_licenses=forbidden, + reason=( + "Defense: restricted-use community licenses (Llama / " + "Gemma / Qwen / Mistral community) carry " + "acceptable-use clauses that conflict with defense " + "applications. Pick a permissive license." + ), + ) + if normalised == "embedded": + # Embedded forbids strong copyleft + NC. + forbidden = _NC_LICENSES + _STRONG_COPYLEFT + return LicenseRecommendation( + target=normalised, + recommended_licenses=_PERMISSIVE_RECOMMENDED, + forbidden_licenses=forbidden, + reason=( + "Embedded: strong copyleft (GPL/AGPL) is incompatible " + "with closed-source firmware distribution; " + "non-commercial licenses are also forbidden." + ), + ) + # Unreachable thanks to validate; defensive default. + raise ValueError(f"unhandled target {normalised!r}") + + +@dataclass(frozen=True) +class DownstreamRisk: + """Outcome of a per-license risk check.""" + + ok: bool + severity: str # "ok" / "warn" / "block" + reason: str + + def __post_init__(self) -> None: + if not isinstance(self.ok, bool): + raise TypeError("ok must be bool") + if not isinstance(self.severity, str): + raise TypeError("severity must be str") + if self.severity not in _VALID_SEVERITIES: + raise ValueError( + f"severity must be one of {{ok, warn, block}}, got {self.severity!r}" + ) + if not isinstance(self.reason, str): + raise TypeError("reason must be str") + + +def _validate_license_id(value: object) -> str: + if isinstance(value, bool): + raise TypeError("license_id must be str, not bool") + if not isinstance(value, str): + raise TypeError(f"license_id must be str, got {type(value).__name__}") + if not value: + raise ValueError("license_id must be non-empty") + if "\x00" in value: + raise ValueError("license_id must not contain null bytes") + if len(value) > _MAX_LICENSE_LEN: + raise ValueError(f"license_id too long (> {_MAX_LICENSE_LEN} chars)") + return value.lower().strip() + + +_MAX_MAU = 10_000_000_000_000 # 10 trillion — more humans than ever existed + + +def _validate_mau(value: object) -> int: + if isinstance(value, bool): + raise TypeError("monthly_active_users must be int, not bool") + if not isinstance(value, int): + raise TypeError( + f"monthly_active_users must be int, got {type(value).__name__}" + ) + if value < 0: + raise ValueError( + f"monthly_active_users must be >= 0, got {value}" + ) + if value > _MAX_MAU: + raise ValueError( + f"monthly_active_users too large (> {_MAX_MAU}), got {value}" + ) + return value + + +def flag_downstream_risk( + *, + license_id: object, + target: object, + monthly_active_users: object, +) -> DownstreamRisk: + """Per-license risk check for a target deployment. + + Decision matrix: + - Unknown license -> ``warn`` (operator should review). + - Non-commercial license on B2C/defense/embedded -> ``block``. + - Llama-family community license on B2C with MAU > 700M -> ``block``. + - Llama-family on defense -> ``block``. + - Strong copyleft on embedded -> ``block``. + - Otherwise: ``ok``. + """ + lic = _validate_license_id(license_id) + tgt = validate_deploy_target(target) + mau = _validate_mau(monthly_active_users) + + if lic not in KNOWN_LICENSES: + return DownstreamRisk( + ok=False, + severity="warn", + reason=( + f"unknown license {lic!r}; " + "verify with legal before shipping." + ), + ) + + kind = LICENSE_KINDS.get(lic) + if kind == "non-commercial": + return DownstreamRisk( + ok=False, + severity="block", + reason=( + f"non-commercial license {lic!r} categorically forbidden " + f"on target {tgt!r}." + ), + ) + + if kind == "restricted-use": + if tgt == "defense": + return DownstreamRisk( + ok=False, + severity="block", + reason=( + f"restricted-use license {lic!r} carries an " + "acceptable-use clause incompatible with defense." + ), + ) + # Llama community license: > 700M MAU = block per Meta AUP. + # Tight allowlist — defends against a future non-community + # "llama-permissive" id falsely tripping the MAU gate. + if lic in _LLAMA_COMMUNITY_LICENSES and mau > _LLAMA_COMMUNITY_MAU_CAP: + return DownstreamRisk( + ok=False, + severity="block", + reason=( + f"Llama community license requires a separate licence from " + f"Meta for MAU > {_LLAMA_COMMUNITY_MAU_CAP:,} " + f"(your MAU: {mau:,})." + ), + ) + # Restricted-use on B2C with low MAU is a warn, not a block. + return DownstreamRisk( + ok=False, + severity="warn", + reason=( + f"restricted-use license {lic!r}: review acceptable-use " + "clauses before shipping. " + "Per Meta AUP, MAU > 700M needs a separate licence." + ), + ) + + if kind == "strong-copyleft" and tgt == "embedded": + return DownstreamRisk( + ok=False, + severity="block", + reason=( + f"strong copyleft {lic!r} is incompatible with " + "closed-source embedded firmware." + ), + ) + + return DownstreamRisk( + ok=True, + severity="ok", + reason=f"license {lic!r} ({kind}) is OK for target {tgt!r}.", + ) + + +__all__ = [ + "DEPLOY_TARGETS", + "DownstreamRisk", + "LicenseRecommendation", + "advise_license_for_target", + "flag_downstream_risk", + "validate_deploy_target", +] diff --git a/soup_cli/utils/terraform_plan.py b/soup_cli/utils/terraform_plan.py new file mode 100644 index 0000000..1d19e1a --- /dev/null +++ b/soup_cli/utils/terraform_plan.py @@ -0,0 +1,390 @@ +"""`soup plan` / `soup apply` — Terraform-shape lock-and-execute for FT. + +A training run is a one-shot infrastructure-shaped operation: spot +price, expected cost, base SHA, dataset SHA, peak VRAM. v0.64 borrows +Terraform's plan-apply split so operators can review what they're +about to spend before committing. + +Workflow: +1. ``soup plan --config soup.yaml`` writes ``soup.tfstate`` with a + ``TrainingPlan`` summarising the run (config SHA, dataset SHA, + estimated cost, ETA, peak VRAM, spot price). +2. ``soup apply --config soup.yaml`` re-builds the plan from the + current YAML, compares against the state file, and **refuses** if + the plan drifted. Operators see "config drifted: epochs 1 -> 99" + instead of silently spending another $0.50. +3. ``--dry-run`` exits 0 after the drift check without actually + invoking the trainer. + +The state file is a thin JSON envelope; the actual ``soup train`` is +still the run-driver. This is a *gate*, not a parallel trainer. + +Public surface: +- ``TrainingPlan`` frozen dataclass (base / task / config_sha / + dataset_sha / cost / ETA / peak VRAM / spot price). +- ``TrainingState`` frozen dataclass (plan + applied flag + run_id + + applied_at timestamp). +- ``DriftReport`` frozen dataclass with the changed-fields tuple. +- ``compute_config_sha(config)`` -> 64-hex SHA-256 of canonical JSON. +- ``compute_dataset_sha(path)`` -> 64-hex SHA-256 of the JSONL bytes. +- ``build_plan(config)`` -> TrainingPlan. +- ``write_state(state, path)`` / ``read_state(path)`` -> atomic JSON. +- ``detect_drift(state, plan_now)`` -> DriftReport. +- ``DEFAULT_STATE_FILE = "soup.tfstate"``. +""" + +from __future__ import annotations + +import hashlib +import json +import math +import os +from dataclasses import dataclass, field +from typing import Any, Mapping, Optional, Tuple + +from soup_cli.utils.paths import atomic_write_text, is_under_cwd + +DEFAULT_STATE_FILE = "soup.tfstate" +_SHA_REGEX_LEN = 64 +_MAX_BASE_LEN = 512 + + +@dataclass(frozen=True) +class TrainingPlan: + """Pre-flight summary of a planned `soup train` invocation.""" + + base: str + task: str + config_sha: str + dataset_sha: str + estimated_cost_usd: float + estimated_minutes: float + peak_vram_gb: float + spot_price_usd_per_hour: float + + def __post_init__(self) -> None: + _check_non_empty_str(self.base, "base", max_len=_MAX_BASE_LEN) + _check_non_empty_str(self.task, "task", max_len=64) + _check_sha(self.config_sha, "config_sha") + _check_sha(self.dataset_sha, "dataset_sha") + _check_non_negative_finite(self.estimated_cost_usd, "estimated_cost_usd") + _check_non_negative_finite(self.estimated_minutes, "estimated_minutes") + _check_non_negative_finite(self.peak_vram_gb, "peak_vram_gb") + _check_non_negative_finite(self.spot_price_usd_per_hour, "spot_price_usd_per_hour") + + +def _check_non_empty_str(value: object, fld: str, *, max_len: int) -> None: + if isinstance(value, bool): + raise TypeError(f"{fld} must be str, not bool") + if not isinstance(value, str): + raise TypeError(f"{fld} must be str, got {type(value).__name__}") + if not value: + raise ValueError(f"{fld} must be non-empty") + if "\x00" in value: + raise ValueError(f"{fld} must not contain null bytes") + if len(value) > max_len: + raise ValueError(f"{fld} too long (> {max_len} chars)") + + +def _check_sha(value: object, fld: str) -> None: + if isinstance(value, bool): + raise TypeError(f"{fld} must be str, not bool") + if not isinstance(value, str): + raise TypeError(f"{fld} must be str, got {type(value).__name__}") + if len(value) != _SHA_REGEX_LEN: + raise ValueError( + f"{fld} sha must be {_SHA_REGEX_LEN} hex chars, " + f"got len={len(value)}" + ) + try: + int(value, 16) + except ValueError as exc: + raise ValueError(f"{fld} sha must be 64-char hex, got {value!r}") from exc + + +def _check_non_negative_finite(value: object, fld: str) -> None: + if isinstance(value, bool): + raise TypeError(f"{fld} must be a number, not bool") + if not isinstance(value, (int, float)): + raise TypeError(f"{fld} must be a number, got {type(value).__name__}") + if not math.isfinite(float(value)): + raise ValueError(f"{fld} must be finite (no NaN / Inf)") + if value < 0: + raise ValueError(f"{fld} must not be negative, got {value}") + + +@dataclass(frozen=True) +class TrainingState: + """Persisted plan + apply metadata. Written as ``soup.tfstate``.""" + + plan: TrainingPlan + applied: bool + applied_at: Optional[str] + run_id: Optional[str] + + def __post_init__(self) -> None: + if not isinstance(self.plan, TrainingPlan): + raise TypeError( + f"plan must be TrainingPlan, got {type(self.plan).__name__}" + ) + if not isinstance(self.applied, bool): + raise TypeError("applied must be bool") + if self.applied_at is not None and not isinstance(self.applied_at, str): + raise TypeError("applied_at must be str | None") + if self.run_id is not None and not isinstance(self.run_id, str): + raise TypeError("run_id must be str | None") + + +@dataclass(frozen=True) +class DriftReport: + """Outcome of a drift comparison between state.plan and a fresh plan.""" + + has_drift: bool + changed_fields: Tuple[str, ...] = field(default_factory=tuple) + + def __post_init__(self) -> None: + if not isinstance(self.has_drift, bool): + raise TypeError("has_drift must be bool") + if not isinstance(self.changed_fields, tuple): + raise TypeError("changed_fields must be a tuple of str") + for entry in self.changed_fields: + if not isinstance(entry, str): + raise TypeError("changed_fields entries must be str") + + +def compute_config_sha(config: Mapping[str, Any]) -> str: + """Canonical SHA-256 of the config dict (insertion-order-independent). + + Uses strict JSON serialisation (no ``default=`` fallback) so two + configs that differ in a non-serialisable field cannot silently + collide. Operators with custom-typed values must pre-canonicalise + via ``dict(yaml.safe_load(...))`` first. + """ + if not isinstance(config, Mapping): + raise TypeError(f"config must be a mapping, got {type(config).__name__}") + blob = json.dumps( + config, sort_keys=True, ensure_ascii=False, allow_nan=False + ).encode("utf-8") + return hashlib.sha256(blob).hexdigest() + + +def compute_dataset_sha(path: object) -> str: + """SHA-256 of dataset file bytes. Returns zero-hash on missing file. + + Returning a constant for "missing" lets ``soup plan`` run before the + dataset exists (e.g. dataset built by an earlier `soup data` step). + Drift detection still surfaces the change once the file appears. + + Security: cwd containment + symlink rejection BEFORE the open() so a + crafted ``soup.yaml`` with ``data.train: /etc/shadow`` cannot leak the + file contents into the SHA. An empty path returns zero-hash without + touching the filesystem. + """ + import stat as _stat + + if not isinstance(path, str): + raise TypeError(f"dataset path must be str, got {type(path).__name__}") + if "\x00" in path: + raise ValueError("dataset path must not contain null bytes") + if not path: + return "0" * _SHA_REGEX_LEN + if not is_under_cwd(path): + # Out-of-cwd is treated as "no dataset captured" rather than a + # hard error so plan can still render — drift detection will + # surface this if the operator later moves the dataset in-tree. + return "0" * _SHA_REGEX_LEN + if os.path.lexists(path): + try: + st = os.lstat(path) + except OSError: + return "0" * _SHA_REGEX_LEN + if _stat.S_ISLNK(st.st_mode): + raise ValueError("dataset path must not be a symlink") + if not os.path.isfile(path): + return "0" * _SHA_REGEX_LEN + h = hashlib.sha256() + with open(path, "rb") as fh: + while True: + chunk = fh.read(65536) + if not chunk: + break + h.update(chunk) + return h.hexdigest() + + +# Approximate cost model — pure heuristic so the plan is honest about +# being a pre-flight estimate. Live measurement is whatever the trainer +# emits at finish time (v0.34 cost path). +_DEFAULT_SPOT_PRICE = 0.30 # ~$/hr for a 24 GB consumer GPU +_DEFAULT_PEAK_VRAM = 8.0 + + +def _estimate_runtime_minutes(config: Mapping[str, Any]) -> float: + training = config.get("training", {}) if isinstance(config, Mapping) else {} + epochs = float(training.get("epochs", 1) if isinstance(training, Mapping) else 1) + batch_size = float(training.get("batch_size", 4) if isinstance(training, Mapping) else 4) + # Soft heuristic: 5 minutes per epoch at batch_size=4 baseline. + base = 5.0 * epochs + if batch_size > 0: + base *= 4.0 / max(batch_size, 1.0) + return max(0.5, base) + + +def _estimate_cost(minutes: float, spot_price: float) -> float: + return max(0.0, (minutes / 60.0) * spot_price) + + +def build_plan(config: Mapping[str, Any]) -> TrainingPlan: + """Render a TrainingPlan from a soup-config-shaped dict.""" + if not isinstance(config, Mapping): + raise TypeError(f"config must be a mapping, got {type(config).__name__}") + base = config.get("base") + if not isinstance(base, str) or not base: + raise ValueError("config must contain a non-empty `base`") + task = config.get("task", "sft") + if not isinstance(task, str) or not task: + task = "sft" + + data_cfg = config.get("data", {}) if isinstance(config.get("data"), Mapping) else {} + train_path = data_cfg.get("train", "") if isinstance(data_cfg, Mapping) else "" + if not isinstance(train_path, str): + train_path = "" + + config_sha = compute_config_sha(config) + dataset_sha = compute_dataset_sha(train_path) + minutes = _estimate_runtime_minutes(config) + cost = _estimate_cost(minutes, _DEFAULT_SPOT_PRICE) + return TrainingPlan( + base=base, + task=task, + config_sha=config_sha, + dataset_sha=dataset_sha, + estimated_cost_usd=cost, + estimated_minutes=minutes, + peak_vram_gb=_DEFAULT_PEAK_VRAM, + spot_price_usd_per_hour=_DEFAULT_SPOT_PRICE, + ) + + +def _plan_to_dict(plan: TrainingPlan) -> dict: + return { + "base": plan.base, + "task": plan.task, + "config_sha": plan.config_sha, + "dataset_sha": plan.dataset_sha, + "estimated_cost_usd": plan.estimated_cost_usd, + "estimated_minutes": plan.estimated_minutes, + "peak_vram_gb": plan.peak_vram_gb, + "spot_price_usd_per_hour": plan.spot_price_usd_per_hour, + } + + +def _plan_from_dict(d: Mapping[str, Any]) -> TrainingPlan: + return TrainingPlan( + base=str(d["base"]), + task=str(d["task"]), + config_sha=str(d["config_sha"]), + dataset_sha=str(d["dataset_sha"]), + estimated_cost_usd=float(d["estimated_cost_usd"]), + estimated_minutes=float(d["estimated_minutes"]), + peak_vram_gb=float(d["peak_vram_gb"]), + spot_price_usd_per_hour=float(d["spot_price_usd_per_hour"]), + ) + + +def write_state(state: TrainingState, path: str) -> None: + """Atomically write a ``TrainingState`` to JSON under cwd containment.""" + if not isinstance(state, TrainingState): + raise TypeError( + f"state must be TrainingState, got {type(state).__name__}" + ) + payload = { + "schema_version": "1", + "plan": _plan_to_dict(state.plan), + "applied": state.applied, + "applied_at": state.applied_at, + "run_id": state.run_id, + } + text = json.dumps(payload, indent=2, ensure_ascii=False, allow_nan=False) + atomic_write_text(text, path, prefix=".tfstate.", field="state file") + + +def read_state(path: str) -> TrainingState: + """Read a previously-written ``TrainingState`` JSON. + + Containment + symlink rejection BEFORE existence probe (mirrors + v0.55.0 / v0.62.0 ordering policy). + """ + import stat as _stat + + if not isinstance(path, str): + raise TypeError(f"path must be str, got {type(path).__name__}") + if "\x00" in path: + raise ValueError("state path must not contain null bytes") + if not is_under_cwd(path): + raise ValueError(f"state file {path!r} is outside cwd") + if os.path.lexists(path): + st = os.lstat(path) + if _stat.S_ISLNK(st.st_mode): + raise ValueError("state path must not be a symlink") + if not os.path.isfile(path): + raise FileNotFoundError(path) + with open(path, encoding="utf-8") as fh: + payload = json.load(fh) + if not isinstance(payload, dict): + raise ValueError("state root must be a dict") + plan_raw = payload.get("plan") + if not isinstance(plan_raw, dict): + raise ValueError("state.plan must be a dict") + plan = _plan_from_dict(plan_raw) + applied_raw = payload.get("applied", False) + if not isinstance(applied_raw, bool): + raise ValueError( + f"state.applied must be bool, got {type(applied_raw).__name__}" + ) + return TrainingState( + plan=plan, + applied=applied_raw, + applied_at=payload.get("applied_at"), + run_id=payload.get("run_id"), + ) + + +def detect_drift(state: TrainingState, plan_now: TrainingPlan) -> DriftReport: + """Compare ``state.plan`` against a freshly-built plan. + + Returns a ``DriftReport`` listing changed fields. Empty tuple means no + drift. ``apply`` refuses to proceed when ``has_drift=True``. + """ + if not isinstance(state, TrainingState): + raise TypeError( + f"state must be TrainingState, got {type(state).__name__}" + ) + if not isinstance(plan_now, TrainingPlan): + raise TypeError( + f"plan_now must be TrainingPlan, got {type(plan_now).__name__}" + ) + changed: list[str] = [] + if state.plan.base != plan_now.base: + changed.append("base") + if state.plan.task != plan_now.task: + changed.append("task") + if state.plan.config_sha != plan_now.config_sha: + changed.append("config_sha") + if state.plan.dataset_sha != plan_now.dataset_sha: + changed.append("dataset_sha") + return DriftReport(has_drift=bool(changed), changed_fields=tuple(changed)) + + +__all__ = [ + "DEFAULT_STATE_FILE", + "DriftReport", + "TrainingPlan", + "TrainingState", + "build_plan", + "compute_config_sha", + "compute_dataset_sha", + "detect_drift", + "read_state", + "write_state", +] diff --git a/soup_cli/utils/tunability.py b/soup_cli/utils/tunability.py new file mode 100644 index 0000000..f8f0d45 --- /dev/null +++ b/soup_cli/utils/tunability.py @@ -0,0 +1,515 @@ +"""`soup tunability` — probe-train 6-10 candidate bases, report Pareto frontier. + +Before committing to a single base model, an operator can run a short LoRA +probe on each of N candidates against a held-out dataset slice. Reports +which candidates are on the (delta-from-base × cost × license) Pareto +frontier so the operator can pick the best fit instead of relying on +vendor catalogs / blog benchmarks. + +Why blue-ocean: hosted vendors push their own catalogs (attach-rate +incentive); comparing across families is structurally costly for them. +Soup is local-first + spans every base. + +Live wiring of the in-process LoRA probe is deferred to v0.64.1 +(mirrors v0.27.0 MII / v0.50.0 GRPO Plus / v0.56.0 diagnose stub-then-live +pattern); v0.64.0 ships the schema, default catalogue, Pareto math, +report writer + the CLI surface so operators can plan a sweep and pipe +results through a custom probe callable today. + +Public surface: +- ``CandidateBase`` frozen dataclass (name / repo_id / params_b / license_id). +- ``TunabilityResult`` frozen dataclass (per-candidate probe outcome). +- ``TunabilityReport`` frozen dataclass (results + Pareto frontier + meta). +- ``DEFAULT_CANDIDATES`` tuple of 8 sane defaults across families. +- ``validate_probe_steps(value)`` -> int in [10, 10_000]. +- ``validate_holdout_size(value)`` -> int in [10, 100_000]. +- ``score_candidate(*, base_loss, probe_loss)`` -> float delta. +- ``pareto_frontier(results)`` -> tuple of non-dominated TunabilityResult. +- ``run_tunability(...)`` -> TunabilityReport (orchestrator). +- ``write_report(report, path)`` / ``load_report(path)`` -> atomic JSON. +""" + +from __future__ import annotations + +import json +import math +import os +from collections.abc import Sequence +from dataclasses import dataclass +from typing import Callable, Optional, Tuple + +from soup_cli.utils.paths import atomic_write_text, is_under_cwd + +# Bounds — mirror v0.30.0 / v0.41.0 / v0.51.0 validator policy. +_MIN_PROBE_STEPS = 10 +_MAX_PROBE_STEPS = 10_000 +_MIN_HOLDOUT = 10 +_MAX_HOLDOUT = 100_000 +_MAX_NAME_LEN = 512 +_MAX_REPO_ID_LEN = 512 +_MAX_LICENSE_LEN = 128 +_MAX_CANDIDATES = 32 + + +@dataclass(frozen=True) +class CandidateBase: + """A candidate base model for a tunability probe. + + `params_b` is the parameter count in billions (e.g. ``0.6`` for 600M). + """ + + name: str + repo_id: str + params_b: float + license_id: str + + def __post_init__(self) -> None: + _check_str(self.name, "name", max_len=_MAX_NAME_LEN) + _check_str(self.repo_id, "repo_id", max_len=_MAX_REPO_ID_LEN) + _check_str(self.license_id, "license_id", max_len=_MAX_LICENSE_LEN) + if isinstance(self.params_b, bool): + raise TypeError("params_b must be a number, not bool") + if not isinstance(self.params_b, (int, float)): + raise TypeError( + f"params_b must be a number, got {type(self.params_b).__name__}" + ) + if not math.isfinite(float(self.params_b)): + raise ValueError("params_b must be finite (no NaN / Inf)") + if self.params_b <= 0: + raise ValueError(f"params_b must be > 0, got {self.params_b}") + + +def _check_str(value: object, field: str, *, max_len: int) -> None: + if isinstance(value, bool): + raise TypeError(f"{field} must be str, not bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str, got {type(value).__name__}") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > max_len: + raise ValueError(f"{field} too long (> {max_len} chars)") + + +@dataclass(frozen=True) +class TunabilityResult: + """Outcome of one candidate's probe run.""" + + candidate: CandidateBase + base_loss: float + probe_loss: float + delta: float + wall_clock_seconds: float + estimated_cost_usd: float + + def __post_init__(self) -> None: + if not isinstance(self.candidate, CandidateBase): + raise TypeError( + f"candidate must be CandidateBase, got {type(self.candidate).__name__}" + ) + for field, val in ( + ("base_loss", self.base_loss), + ("probe_loss", self.probe_loss), + ("delta", self.delta), + ("wall_clock_seconds", self.wall_clock_seconds), + ("estimated_cost_usd", self.estimated_cost_usd), + ): + if isinstance(val, bool): + raise TypeError(f"{field} must be a number, not bool") + if not isinstance(val, (int, float)): + raise TypeError( + f"{field} must be a number, got {type(val).__name__}" + ) + if not math.isfinite(float(val)): + raise ValueError(f"{field} must be finite (no NaN / Inf)") + if self.wall_clock_seconds < 0: + raise ValueError( + f"wall_clock_seconds must be >= 0, got {self.wall_clock_seconds}" + ) + if self.estimated_cost_usd < 0: + raise ValueError( + f"estimated_cost_usd must be >= 0, got {self.estimated_cost_usd}" + ) + + +@dataclass(frozen=True) +class TunabilityReport: + """End-to-end report for a tunability sweep.""" + + results: Tuple[TunabilityResult, ...] + frontier: Tuple[TunabilityResult, ...] + probe_steps: int + holdout_size: int + + def __post_init__(self) -> None: + if not isinstance(self.results, tuple): + raise TypeError("results must be a tuple of TunabilityResult") + if not isinstance(self.frontier, tuple): + raise TypeError("frontier must be a tuple of TunabilityResult") + for entry in self.results: + if not isinstance(entry, TunabilityResult): + raise TypeError("every result must be TunabilityResult") + for entry in self.frontier: + if not isinstance(entry, TunabilityResult): + raise TypeError("every frontier entry must be TunabilityResult") + + +# Default catalogue: cross-family small bases that fit in 12 GB VRAM with 4-bit +# LoRA. Operators can override via `--candidates ` (matched +# against `name`) or supply a custom YAML file. +DEFAULT_CANDIDATES: Tuple[CandidateBase, ...] = ( + CandidateBase( + name="qwen3-0.6b", + repo_id="Qwen/Qwen3-0.6B", + params_b=0.6, + license_id="apache-2.0", + ), + CandidateBase( + name="qwen3-1.7b", + repo_id="Qwen/Qwen3-1.7B", + params_b=1.7, + license_id="apache-2.0", + ), + CandidateBase( + name="llama-3.2-1b", + repo_id="meta-llama/Llama-3.2-1B-Instruct", + params_b=1.0, + license_id="llama-3.2", + ), + CandidateBase( + name="llama-3.2-3b", + repo_id="meta-llama/Llama-3.2-3B-Instruct", + params_b=3.0, + license_id="llama-3.2", + ), + CandidateBase( + name="gemma-3-e2b", + repo_id="google/gemma-3-2b-it", + params_b=2.0, + license_id="gemma", + ), + CandidateBase( + name="phi-4-mini", + repo_id="microsoft/Phi-4-mini-instruct", + params_b=3.8, + license_id="mit", + ), + CandidateBase( + name="smollm3-1.7b", + repo_id="HuggingFaceTB/SmolLM3-1.7B-Instruct", + params_b=1.7, + license_id="apache-2.0", + ), + CandidateBase( + name="qwen2.5-1.5b", + repo_id="Qwen/Qwen2.5-1.5B-Instruct", + params_b=1.5, + license_id="apache-2.0", + ), +) + + +def validate_probe_steps(value: object) -> int: + """Validate ``probe_steps`` in [10, 10_000]; reject bool / non-int.""" + if isinstance(value, bool): + raise TypeError("probe_steps must be int, not bool") + if not isinstance(value, int): + raise TypeError(f"probe_steps must be int, got {type(value).__name__}") + if not (_MIN_PROBE_STEPS <= value <= _MAX_PROBE_STEPS): + raise ValueError( + f"probe_steps must be in [{_MIN_PROBE_STEPS}, {_MAX_PROBE_STEPS}], " + f"got {value}" + ) + return value + + +def validate_holdout_size(value: object) -> int: + """Validate ``holdout_size`` in [10, 100_000]; reject bool / non-int.""" + if isinstance(value, bool): + raise TypeError("holdout_size must be int, not bool") + if not isinstance(value, int): + raise TypeError(f"holdout_size must be int, got {type(value).__name__}") + if not (_MIN_HOLDOUT <= value <= _MAX_HOLDOUT): + raise ValueError( + f"holdout_size must be in [{_MIN_HOLDOUT}, {_MAX_HOLDOUT}], " + f"got {value}" + ) + return value + + +def score_candidate(*, base_loss: float, probe_loss: float) -> float: + """Compute the candidate's delta (higher = bigger improvement). + + delta = base_loss - probe_loss. Positive means the LoRA probe lowered + loss from the base; negative means it made things worse. + """ + for name, val in (("base_loss", base_loss), ("probe_loss", probe_loss)): + if isinstance(val, bool): + raise TypeError(f"{name} must be a number, not bool") + if not isinstance(val, (int, float)): + raise TypeError(f"{name} must be a number, got {type(val).__name__}") + if not math.isfinite(float(val)): + raise ValueError(f"{name} must be finite") + return float(base_loss) - float(probe_loss) + + +def _dominates(a: TunabilityResult, b: TunabilityResult) -> bool: + """Return True iff ``a`` strictly dominates ``b`` on (delta, cost). + + A dominates B iff: A.delta >= B.delta AND A.cost <= B.cost, + and at least one of the inequalities is strict. + """ + delta_ge = a.delta >= b.delta + cost_le = a.estimated_cost_usd <= b.estimated_cost_usd + strict = (a.delta > b.delta) or (a.estimated_cost_usd < b.estimated_cost_usd) + return delta_ge and cost_le and strict + + +def pareto_frontier(results: Sequence[TunabilityResult]) -> Tuple[TunabilityResult, ...]: + """Return the Pareto-optimal subset of results. + + Maximises ``delta``, minimises ``estimated_cost_usd``. A result survives + iff no other result strictly dominates it on both axes. + """ + if isinstance(results, str): + raise TypeError("results must be a sequence of TunabilityResult, not str") + try: + materialised = list(results) + except TypeError as exc: + raise TypeError("results must be iterable") from exc + for entry in materialised: + if not isinstance(entry, TunabilityResult): + raise TypeError( + f"every result must be TunabilityResult, got {type(entry).__name__}" + ) + if not materialised: + return () + frontier: list[TunabilityResult] = [] + for i, candidate in enumerate(materialised): + dominated = False + for j, other in enumerate(materialised): + if i == j: + continue + if _dominates(other, candidate): + dominated = True + break + if not dominated: + frontier.append(candidate) + return tuple(frontier) + + +# Type alias for the probe callable. Live impl lands in v0.64.1. +ProbeFn = Callable[[CandidateBase, str], TunabilityResult] + + +def _default_probe( + candidate: CandidateBase, + dataset_path: str, + *, + probe_steps: int, + holdout_size: int, +) -> TunabilityResult: + """Heuristic stand-in. Live LoRA probe lands in v0.64.1. + + Returns deterministic, candidate-derived values so reports parse and + Pareto math exercises without a GPU. Operators wanting a real probe + inject a callable via ``probe_fn=...``. + """ + # Wall-clock + cost scale with params. Delta is a tiny constant so + # the report is honest about being a stub. + wall_clock = 60.0 + 30.0 * float(candidate.params_b) + cost = 0.001 * float(candidate.params_b) * float(probe_steps) + return TunabilityResult( + candidate=candidate, + base_loss=2.5, + probe_loss=2.5, # No change — heuristic stub + delta=0.0, + wall_clock_seconds=wall_clock, + estimated_cost_usd=cost, + ) + + +def run_tunability( + *, + candidates: Sequence[CandidateBase], + dataset_path: str, + probe_steps: int = 100, + holdout_size: int = 64, + probe_fn: Optional[Callable[..., TunabilityResult]] = None, +) -> TunabilityReport: + """Orchestrate probes across candidates and assemble a report. + + The actual probing is delegated to ``probe_fn`` (signature: + ``(cand, dataset_path, *, probe_steps, holdout_size) -> TunabilityResult``). + If ``probe_fn`` is None, falls back to the heuristic stub. + """ + if isinstance(candidates, (str, bytes)) or not isinstance(candidates, Sequence): + raise TypeError("candidates must be a sequence of CandidateBase") + if len(candidates) == 0: + raise ValueError("candidates must contain at least one entry") + if len(candidates) > _MAX_CANDIDATES: + raise ValueError( + f"too many candidates ({len(candidates)} > {_MAX_CANDIDATES})" + ) + for entry in candidates: + if not isinstance(entry, CandidateBase): + raise TypeError( + f"every candidate must be CandidateBase, " + f"got {type(entry).__name__}" + ) + + steps = validate_probe_steps(probe_steps) + holdout = validate_holdout_size(holdout_size) + + if not isinstance(dataset_path, str): + raise TypeError( + f"dataset_path must be str, got {type(dataset_path).__name__}" + ) + if not dataset_path: + raise ValueError("dataset_path must be non-empty") + if "\x00" in dataset_path: + raise ValueError("dataset_path must not contain null bytes") + + fn = probe_fn if probe_fn is not None else _default_probe + results: list[TunabilityResult] = [] + for cand in candidates: + result = fn(cand, dataset_path, probe_steps=steps, holdout_size=holdout) + if not isinstance(result, TunabilityResult): + raise TypeError( + f"probe_fn must return TunabilityResult, " + f"got {type(result).__name__}" + ) + results.append(result) + + frontier = pareto_frontier(results) + return TunabilityReport( + results=tuple(results), + frontier=frontier, + probe_steps=steps, + holdout_size=holdout, + ) + + +# --------------------------------------------------------------------------- +# Report I/O +# --------------------------------------------------------------------------- + + +def _result_to_dict(r: TunabilityResult) -> dict: + return { + "candidate": { + "name": r.candidate.name, + "repo_id": r.candidate.repo_id, + "params_b": r.candidate.params_b, + "license_id": r.candidate.license_id, + }, + "base_loss": r.base_loss, + "probe_loss": r.probe_loss, + "delta": r.delta, + "wall_clock_seconds": r.wall_clock_seconds, + "estimated_cost_usd": r.estimated_cost_usd, + } + + +def _result_from_dict(d: dict) -> TunabilityResult: + if not isinstance(d, dict): + raise ValueError("result must be a dict") + cand_raw = d.get("candidate") + if not isinstance(cand_raw, dict): + raise ValueError("candidate must be a dict") + cand = CandidateBase( + name=cand_raw["name"], + repo_id=cand_raw["repo_id"], + params_b=float(cand_raw["params_b"]), + license_id=cand_raw["license_id"], + ) + return TunabilityResult( + candidate=cand, + base_loss=float(d["base_loss"]), + probe_loss=float(d["probe_loss"]), + delta=float(d["delta"]), + wall_clock_seconds=float(d["wall_clock_seconds"]), + estimated_cost_usd=float(d["estimated_cost_usd"]), + ) + + +def write_report(report: TunabilityReport, path: str) -> None: + """Atomically write a TunabilityReport as JSON. + + Path is cwd-containment-checked + symlink-rejected via the shared + ``enforce_under_cwd_and_no_symlink`` helper (mirrors v0.59.0 / + v0.60.0 / v0.62.0 atomic-write policy). + """ + if not isinstance(report, TunabilityReport): + raise TypeError( + f"report must be TunabilityReport, got {type(report).__name__}" + ) + if not isinstance(path, str): + raise TypeError(f"path must be str, got {type(path).__name__}") + if not path: + raise ValueError("path must be non-empty") + if "\x00" in path: + raise ValueError("path must not contain null bytes") + + payload = { + "schema_version": "1", + "probe_steps": report.probe_steps, + "holdout_size": report.holdout_size, + "results": [_result_to_dict(r) for r in report.results], + "frontier": [_result_to_dict(r) for r in report.frontier], + } + text = json.dumps(payload, indent=2, ensure_ascii=False, allow_nan=False) + atomic_write_text(text, path, prefix=".tunability.", field="tunability output") + + +def load_report(path: str) -> TunabilityReport: + """Load a TunabilityReport from JSON. Raises FileNotFoundError if missing. + + Path is cwd-containment-checked + symlink-rejected BEFORE the + existence probe so a crafted path cannot leak file-existence + distinguishing "outside cwd" from "missing" (mirrors v0.55.0 / + v0.62.0 ordering policy). + """ + import stat as _stat + + if not isinstance(path, str): + raise TypeError(f"path must be str, got {type(path).__name__}") + if "\x00" in path: + raise ValueError("path must not contain null bytes") + if not is_under_cwd(path): + raise ValueError(f"path {path!r} is outside cwd") + if os.path.lexists(path): + st = os.lstat(path) + if _stat.S_ISLNK(st.st_mode): + raise ValueError("report path must not be a symlink") + if not os.path.isfile(path): + raise FileNotFoundError(path) + with open(path, encoding="utf-8") as fh: + payload = json.load(fh) + if not isinstance(payload, dict): + raise ValueError("report root must be a dict") + results = tuple(_result_from_dict(r) for r in payload.get("results", [])) + frontier = tuple(_result_from_dict(r) for r in payload.get("frontier", [])) + return TunabilityReport( + results=results, + frontier=frontier, + probe_steps=int(payload.get("probe_steps", _MIN_PROBE_STEPS)), + holdout_size=int(payload.get("holdout_size", _MIN_HOLDOUT)), + ) + + +__all__ = [ + "CandidateBase", + "DEFAULT_CANDIDATES", + "ProbeFn", + "TunabilityReport", + "TunabilityResult", + "load_report", + "pareto_frontier", + "run_tunability", + "score_candidate", + "validate_holdout_size", + "validate_probe_steps", + "write_report", +] diff --git a/tests/test_v0640_followups.py b/tests/test_v0640_followups.py new file mode 100644 index 0000000..fda7ce3 --- /dev/null +++ b/tests/test_v0640_followups.py @@ -0,0 +1,465 @@ +"""v0.64.0 review fix follow-ups. + +Covers wave-1 review-fix gaps: +- HIGH H2/H3/H4 — symlink rejection on the new READ paths + (plan/apply YAML loader, tunability load_report, env_lock read_lock, + terraform_plan read_state). +- HIGH H5 — compute_dataset_sha cwd-containment + symlink rejection. +- HIGH H6 — symlink rejection on dataset_path through `soup plan`. +- MEDIUM M1 — compute_config_sha rejects non-JSON-serialisable values. +- MEDIUM M3 — read_state rejects non-bool `applied`. +- MEDIUM M4 — _activation_bytes overflow-clamps at sanity cap. +- MEDIUM M6 — flag_downstream_risk tight Llama-family allowlist. +- MEDIUM M7 — Windows CUDA path parse (vN.M pattern). +- MEDIUM M8 — load_report containment-before-existence ordering. +- LOW L3 — source-grep regression: atomic_write_text usage in all 3 writers. +- LOW L4 — license-advisor MAU upper cap. +- LOW L5 — `Sequence` imported from `collections.abc` not `typing`. +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +import pytest + +# --------------------------------------------------------------------------- +# HIGH H2 — plan `_load_yaml_config` rejects symlink on YAML path +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_plan_yaml_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.commands.plan import _load_yaml_config + + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.yaml" + real.write_text("base: x\n") + link = tmp_path / "link.yaml" + os.symlink(real, link) + with pytest.raises(ValueError, match="symlink"): + _load_yaml_config(str(link)) + + +def test_plan_yaml_outside_cwd_rejected(tmp_path, monkeypatch): + from soup_cli.commands.plan import _load_yaml_config + + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "evil.yaml" + outside.write_text("base: x\n") + with pytest.raises(ValueError, match="cwd"): + _load_yaml_config(str(outside)) + + +def test_plan_yaml_null_byte_rejected(tmp_path, monkeypatch): + from soup_cli.commands.plan import _load_yaml_config + + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + _load_yaml_config("config\x00.yaml") + + +# --------------------------------------------------------------------------- +# HIGH H3 — tunability load_report rejects symlink + null-byte +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_tunability_load_report_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.utils.tunability import load_report + + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.json" + real.write_text("{}") + link = tmp_path / "link.json" + os.symlink(real, link) + with pytest.raises(ValueError, match="symlink"): + load_report(str(link)) + + +def test_tunability_load_report_null_byte_rejected(tmp_path, monkeypatch): + from soup_cli.utils.tunability import load_report + + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + load_report("report\x00.json") + + +def test_tunability_load_report_containment_before_existence(tmp_path, monkeypatch): + """Outside-cwd path must raise ValueError(cwd), NOT FileNotFoundError. + + Otherwise an attacker can distinguish "file exists out of cwd" from + "file missing" via the exception type — MEDIUM M8 review fix. + """ + from soup_cli.utils.tunability import load_report + + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "definitely-missing.json" + with pytest.raises(ValueError, match="cwd"): + load_report(str(outside)) + + +# --------------------------------------------------------------------------- +# HIGH H4 — env read_lock rejects symlink + null-byte + ordering +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_env_read_lock_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import read_lock + + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.lock" + real.write_text("{}") + link = tmp_path / "link.lock" + os.symlink(real, link) + with pytest.raises(ValueError, match="symlink"): + read_lock(str(link)) + + +def test_env_read_lock_null_byte_rejected(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import read_lock + + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + read_lock("lock\x00.lock") + + +def test_env_read_lock_outside_cwd_before_existence(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import read_lock + + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "missing.lock" + with pytest.raises(ValueError, match="cwd"): + read_lock(str(outside)) + + +# --------------------------------------------------------------------------- +# HIGH H4 (terraform read_state) — symlink rejection +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_terraform_read_state_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import read_state + + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.tfstate" + real.write_text("{}") + link = tmp_path / "link.tfstate" + os.symlink(real, link) + with pytest.raises(ValueError, match="symlink"): + read_state(str(link)) + + +def test_terraform_read_state_null_byte_rejected(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import read_state + + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + read_state("state\x00.tfstate") + + +def test_terraform_read_state_outside_cwd_before_existence(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import read_state + + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "missing.tfstate" + with pytest.raises(ValueError, match="cwd"): + read_state(str(outside)) + + +# --------------------------------------------------------------------------- +# HIGH H5 — compute_dataset_sha cwd containment + symlink rejection +# --------------------------------------------------------------------------- + + +def test_compute_dataset_sha_empty_returns_zero(): + from soup_cli.utils.terraform_plan import compute_dataset_sha + + assert compute_dataset_sha("") == "0" * 64 + + +def test_compute_dataset_sha_outside_cwd_returns_zero(tmp_path, monkeypatch): + """Out-of-cwd path silently returns zero-hash (no file read). + + Defends against `data.train: /etc/shadow` smuggling file contents + into the SHA — H5 review fix. + """ + from soup_cli.utils.terraform_plan import compute_dataset_sha + + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "evil.jsonl" + if outside.exists(): + # Don't depend on the parent dir being writable. + return + sha = compute_dataset_sha(str(outside)) + assert sha == "0" * 64 + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_compute_dataset_sha_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import compute_dataset_sha + + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.jsonl" + real.write_text("{}\n") + link = tmp_path / "link.jsonl" + os.symlink(real, link) + with pytest.raises(ValueError, match="symlink"): + compute_dataset_sha(str(link)) + + +def test_compute_dataset_sha_null_byte_rejected(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import compute_dataset_sha + + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + compute_dataset_sha("data\x00.jsonl") + + +def test_compute_dataset_sha_missing_returns_zero(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import compute_dataset_sha + + monkeypatch.chdir(tmp_path) + sha = compute_dataset_sha("missing.jsonl") + assert sha == "0" * 64 + + +# --------------------------------------------------------------------------- +# MEDIUM M1 — compute_config_sha rejects non-JSON-serialisable values +# --------------------------------------------------------------------------- + + +def test_compute_config_sha_strict_no_default_str(): + """A non-serialisable value must raise, not silently `str()` it.""" + from soup_cli.utils.terraform_plan import compute_config_sha + + # `set` is not JSON-serialisable + with pytest.raises(TypeError): + compute_config_sha({"a": {1, 2, 3}}) + + +# --------------------------------------------------------------------------- +# MEDIUM M3 — read_state rejects non-bool `applied` +# --------------------------------------------------------------------------- + + +def test_read_state_rejects_non_bool_applied(tmp_path, monkeypatch): + import json + + from soup_cli.utils.terraform_plan import read_state + + monkeypatch.chdir(tmp_path) + bad = tmp_path / "bad.tfstate" + payload = { + "plan": { + "base": "m", + "task": "sft", + "config_sha": "a" * 64, + "dataset_sha": "b" * 64, + "estimated_cost_usd": 0.5, + "estimated_minutes": 10.0, + "peak_vram_gb": 8.0, + "spot_price_usd_per_hour": 0.30, + }, + "applied": "yes", # str, not bool — must raise + "applied_at": None, + "run_id": None, + } + bad.write_text(json.dumps(payload)) + with pytest.raises(ValueError, match="applied"): + read_state(str(bad)) + + +# --------------------------------------------------------------------------- +# MEDIUM M4 — _activation_bytes overflow clamp +# --------------------------------------------------------------------------- + + +def test_activation_bytes_clamped_at_extreme(): + """Even at schema max seq_len * batch_size, result stays finite.""" + import math + + from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb + + inp = HardwareFitInput( + params_b=1000.0, + seq_len=1_048_576, + batch_size=1024, + optimizer="adamw_torch", + quant="none", + peft="full", + gradient_checkpointing=False, + ) + bd = estimate_peak_vram_gb(inp) + assert math.isfinite(bd.total_gb) + assert math.isfinite(bd.activations_gb) + + +# --------------------------------------------------------------------------- +# MEDIUM M6 — Llama community license allowlist (no .startswith match) +# --------------------------------------------------------------------------- + + +def test_flag_downstream_risk_unknown_llama_variant_does_not_trip_mau_gate(): + """A hypothetical `llama-permissive-2030` should NOT trigger MAU block. + + Tight allowlist defends against future Meta licence id surprises. + """ + from soup_cli.utils.license_advisor import flag_downstream_risk + + # `llama-permissive-2030` is not in `_LLAMA_COMMUNITY_LICENSES` AND + # is not in the known matrix, so it should fall through to the + # "unknown license -> warn" branch, not "block by MAU". + r = flag_downstream_risk( + license_id="llama-permissive-2030", + target="b2c", + monthly_active_users=800_000_000, + ) + # severity might be warn (unknown) but NOT block-by-MAU. + assert "monthly active users" not in r.reason.lower() or r.severity != "block" + + +def test_flag_downstream_risk_llama_3_under_cap_is_warn_not_block(): + from soup_cli.utils.license_advisor import flag_downstream_risk + + r = flag_downstream_risk( + license_id="llama-3", + target="b2c", + monthly_active_users=100, + ) + # Restricted-use on B2C with low MAU = warn, never block. + assert r.severity != "block" + + +# --------------------------------------------------------------------------- +# MEDIUM M7 — Windows CUDA path parse (vN.M) +# --------------------------------------------------------------------------- + + +def test_detect_cuda_version_windows_path(monkeypatch): + """`v12.1` token at end of a Windows-style path is extracted.""" + from soup_cli.utils import env_lock + + monkeypatch.setenv("CUDA_HOME", r"C:\Program Files\NVIDIA\CUDA\v12.1") + # Ensure no other env var pre-empts + monkeypatch.delenv("CUDA_VERSION", raising=False) + # Sidestep torch — drop it from sys.modules + monkeypatch.setattr(env_lock.sys, "modules", {**env_lock.sys.modules, "torch": None}) + version = env_lock._detect_cuda_version() + assert version is not None + assert "12.1" in version + + +def test_detect_cuda_version_posix_path(monkeypatch): + from soup_cli.utils import env_lock + + monkeypatch.setenv("CUDA_HOME", "/usr/local/cuda-12.1") + monkeypatch.delenv("CUDA_VERSION", raising=False) + monkeypatch.setattr(env_lock.sys, "modules", {**env_lock.sys.modules, "torch": None}) + version = env_lock._detect_cuda_version() + assert version is not None + assert "12.1" in version + + +def test_detect_cuda_version_none_when_no_env(monkeypatch): + from soup_cli.utils import env_lock + + monkeypatch.delenv("CUDA_HOME", raising=False) + monkeypatch.delenv("CUDA_VERSION", raising=False) + monkeypatch.setattr(env_lock.sys, "modules", {**env_lock.sys.modules, "torch": None}) + # Should silently return None — no env, no torch. + assert env_lock._detect_cuda_version() is None + + +# --------------------------------------------------------------------------- +# LOW L3 — source-grep regression: atomic_write_text in all 3 writers +# --------------------------------------------------------------------------- + + +def test_tunability_uses_atomic_write_text(): + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "tunability.py" + text = src.read_text(encoding="utf-8") + assert "atomic_write_text" in text, "tunability.write_report must use shared helper" + + +def test_terraform_plan_uses_atomic_write_text(): + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "terraform_plan.py" + text = src.read_text(encoding="utf-8") + assert "atomic_write_text" in text, "terraform_plan.write_state must use shared helper" + + +def test_env_lock_uses_atomic_write_text(): + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "env_lock.py" + text = src.read_text(encoding="utf-8") + assert "atomic_write_text" in text, "env_lock.write_lock must use shared helper" + + +# --------------------------------------------------------------------------- +# LOW L4 — MAU upper cap +# --------------------------------------------------------------------------- + + +def test_mau_oversize_rejected(): + from soup_cli.utils.license_advisor import flag_downstream_risk + + with pytest.raises(ValueError, match="monthly_active_users"): + flag_downstream_risk( + license_id="apache-2.0", + target="b2c", + monthly_active_users=10**18, + ) + + +# --------------------------------------------------------------------------- +# LOW L5 — Sequence imported from collections.abc, not typing +# --------------------------------------------------------------------------- + + +def test_tunability_sequence_imported_from_collections_abc(): + """`typing.Sequence` doesn't work with isinstance on every Py 3.9 build. + + Use `collections.abc.Sequence` for runtime isinstance checks. + """ + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "tunability.py" + text = src.read_text(encoding="utf-8") + # Either explicit `from collections.abc import Sequence`, or NO + # mention of `typing.Sequence` in an isinstance context. + assert "from collections.abc import Sequence" in text, ( + "tunability.py must import Sequence from collections.abc" + ) + + +# --------------------------------------------------------------------------- +# Drift-on-modify regression — `soup plan` -> mutate -> `soup apply` +# --------------------------------------------------------------------------- + + +def test_apply_refuses_drift_end_to_end(tmp_path, monkeypatch): + from typer.testing import CliRunner + + from soup_cli.cli import app + + runner = CliRunner() + monkeypatch.chdir(tmp_path) + cfg = tmp_path / "soup.yaml" + cfg.write_text( + "base: meta-llama/Llama-3.2-1B\n" + "task: sft\n" + "data:\n" + " train: ./data.jsonl\n" + "training:\n" + " epochs: 1\n" + " lr: 0.00005\n" + " batch_size: 4\n" + ) + (tmp_path / "data.jsonl").write_text("{}\n") + r = runner.invoke(app, ["plan", "--config", str(cfg)]) + assert r.exit_code == 0, r.output + cfg.write_text(cfg.read_text().replace("epochs: 1", "epochs: 5")) + r2 = runner.invoke(app, ["apply", "--config", str(cfg), "--dry-run"]) + assert r2.exit_code == 3, r2.output # exit 3 = drift refused + assert "drift" in r2.output.lower() diff --git a/tests/test_v0640_part_a.py b/tests/test_v0640_part_a.py new file mode 100644 index 0000000..4d7ffc7 --- /dev/null +++ b/tests/test_v0640_part_a.py @@ -0,0 +1,694 @@ +"""v0.64.0 Part A — `soup tunability` probe across candidate bases. + +Tests cover: +- CandidateBase frozen dataclass + validation +- TunabilityResult frozen dataclass + validation +- TunabilityReport frozen dataclass + immutable candidates tuple +- validate_probe_steps bounds + bool reject +- validate_holdout_size bounds + bool reject +- score_candidate happy + delta math (lower-better loss) +- pareto_frontier identifies non-dominated points +- run_tunability orchestrator (with mocked probe callable) +- write_report + load_report atomic + cwd containment + symlink rejection +- CLI smoke (--help / outside-cwd reject / unknown candidate) +""" + +from __future__ import annotations + +import dataclasses +import json +import os +import sys + +import pytest +from typer.testing import CliRunner + +runner = CliRunner() + + +# --------------------------------------------------------------------------- +# Module imports +# --------------------------------------------------------------------------- + + +def test_module_imports(): + from soup_cli.utils import tunability + + assert hasattr(tunability, "CandidateBase") + assert hasattr(tunability, "TunabilityResult") + assert hasattr(tunability, "TunabilityReport") + assert hasattr(tunability, "validate_probe_steps") + assert hasattr(tunability, "validate_holdout_size") + assert hasattr(tunability, "score_candidate") + assert hasattr(tunability, "pareto_frontier") + assert hasattr(tunability, "run_tunability") + assert hasattr(tunability, "write_report") + assert hasattr(tunability, "load_report") + assert hasattr(tunability, "DEFAULT_CANDIDATES") + + +# --------------------------------------------------------------------------- +# DEFAULT_CANDIDATES catalog +# --------------------------------------------------------------------------- + + +def test_default_candidates_nonempty(): + from soup_cli.utils.tunability import DEFAULT_CANDIDATES + + assert len(DEFAULT_CANDIDATES) >= 6 + # Each entry must be CandidateBase + from soup_cli.utils.tunability import CandidateBase + for c in DEFAULT_CANDIDATES: + assert isinstance(c, CandidateBase) + + +def test_default_candidates_immutable(): + from soup_cli.utils.tunability import DEFAULT_CANDIDATES + + # Tuple, not list + assert isinstance(DEFAULT_CANDIDATES, tuple) + + +# --------------------------------------------------------------------------- +# CandidateBase +# --------------------------------------------------------------------------- + + +def test_candidate_base_frozen(): + from soup_cli.utils.tunability import CandidateBase + + c = CandidateBase( + name="qwen3-0.6b", + repo_id="Qwen/Qwen3-0.6B", + params_b=0.6, + license_id="apache-2.0", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + c.name = "other" # type: ignore[misc] + + +def test_candidate_base_rejects_empty_name(): + from soup_cli.utils.tunability import CandidateBase + + with pytest.raises(ValueError, match="name"): + CandidateBase(name="", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + + +def test_candidate_base_rejects_null_byte(): + from soup_cli.utils.tunability import CandidateBase + + with pytest.raises(ValueError, match="null"): + CandidateBase(name="bad\x00", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + + +def test_candidate_base_rejects_negative_params(): + from soup_cli.utils.tunability import CandidateBase + + with pytest.raises(ValueError, match="params_b"): + CandidateBase(name="x", repo_id="x/y", params_b=-1.0, license_id="apache-2.0") + + +def test_candidate_base_rejects_bool_params(): + from soup_cli.utils.tunability import CandidateBase + + with pytest.raises(TypeError, match="bool"): + CandidateBase(name="x", repo_id="x/y", params_b=True, license_id="apache-2.0") # type: ignore[arg-type] + + +def test_candidate_base_rejects_non_finite_params(): + from soup_cli.utils.tunability import CandidateBase + + with pytest.raises(ValueError, match="finite"): + CandidateBase(name="x", repo_id="x/y", params_b=float("nan"), license_id="apache-2.0") + + +def test_candidate_base_rejects_oversize_name(): + from soup_cli.utils.tunability import CandidateBase + + with pytest.raises(ValueError, match="too long"): + CandidateBase(name="x" * 513, repo_id="x/y", params_b=1.0, license_id="apache-2.0") + + +# --------------------------------------------------------------------------- +# validate_probe_steps +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("value", [10, 100, 1000]) +def test_validate_probe_steps_happy(value): + from soup_cli.utils.tunability import validate_probe_steps + + assert validate_probe_steps(value) == value + + +def test_validate_probe_steps_boundary_min(): + from soup_cli.utils.tunability import validate_probe_steps + + assert validate_probe_steps(10) == 10 + with pytest.raises(ValueError): + validate_probe_steps(9) + + +def test_validate_probe_steps_boundary_max(): + from soup_cli.utils.tunability import validate_probe_steps + + assert validate_probe_steps(10_000) == 10_000 + with pytest.raises(ValueError): + validate_probe_steps(10_001) + + +@pytest.mark.parametrize("bad", [True, False, None, "100", -1, 0, 9, 10_001, 1.5]) +def test_validate_probe_steps_rejects(bad): + from soup_cli.utils.tunability import validate_probe_steps + + with pytest.raises((TypeError, ValueError)): + validate_probe_steps(bad) + + +# --------------------------------------------------------------------------- +# validate_holdout_size +# --------------------------------------------------------------------------- + + +def test_validate_holdout_size_happy(): + from soup_cli.utils.tunability import validate_holdout_size + + assert validate_holdout_size(100) == 100 + + +def test_validate_holdout_size_boundary(): + from soup_cli.utils.tunability import validate_holdout_size + + assert validate_holdout_size(10) == 10 + with pytest.raises(ValueError): + validate_holdout_size(9) + assert validate_holdout_size(100_000) == 100_000 + with pytest.raises(ValueError): + validate_holdout_size(100_001) + + +@pytest.mark.parametrize("bad", [True, False, "100", -1, 0]) +def test_validate_holdout_size_rejects(bad): + from soup_cli.utils.tunability import validate_holdout_size + + with pytest.raises((TypeError, ValueError)): + validate_holdout_size(bad) + + +# --------------------------------------------------------------------------- +# TunabilityResult +# --------------------------------------------------------------------------- + + +def test_tunability_result_happy(): + from soup_cli.utils.tunability import CandidateBase, TunabilityResult + + cand = CandidateBase( + name="qwen3-0.6b", repo_id="x/y", params_b=0.6, license_id="apache-2.0" + ) + r = TunabilityResult( + candidate=cand, + base_loss=2.5, + probe_loss=2.0, + delta=0.5, + wall_clock_seconds=120.0, + estimated_cost_usd=0.05, + ) + assert r.delta == 0.5 + + +def test_tunability_result_frozen(): + from soup_cli.utils.tunability import CandidateBase, TunabilityResult + + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + r = TunabilityResult( + candidate=cand, + base_loss=2.5, + probe_loss=2.0, + delta=0.5, + wall_clock_seconds=120.0, + estimated_cost_usd=0.05, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + r.delta = 0.0 # type: ignore[misc] + + +def test_tunability_result_rejects_non_finite(): + from soup_cli.utils.tunability import CandidateBase, TunabilityResult + + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + with pytest.raises(ValueError, match="finite"): + TunabilityResult( + candidate=cand, + base_loss=float("nan"), + probe_loss=2.0, + delta=0.5, + wall_clock_seconds=120.0, + estimated_cost_usd=0.05, + ) + + +def test_tunability_result_rejects_negative_wall_clock(): + from soup_cli.utils.tunability import CandidateBase, TunabilityResult + + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + with pytest.raises(ValueError, match="wall_clock"): + TunabilityResult( + candidate=cand, + base_loss=2.5, + probe_loss=2.0, + delta=0.5, + wall_clock_seconds=-1.0, + estimated_cost_usd=0.05, + ) + + +def test_tunability_result_rejects_negative_cost(): + from soup_cli.utils.tunability import CandidateBase, TunabilityResult + + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + with pytest.raises(ValueError, match="cost"): + TunabilityResult( + candidate=cand, + base_loss=2.5, + probe_loss=2.0, + delta=0.5, + wall_clock_seconds=120.0, + estimated_cost_usd=-0.01, + ) + + +def test_tunability_result_rejects_bool_loss(): + from soup_cli.utils.tunability import CandidateBase, TunabilityResult + + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + with pytest.raises(TypeError, match="bool"): + TunabilityResult( + candidate=cand, + base_loss=True, # type: ignore[arg-type] + probe_loss=2.0, + delta=0.5, + wall_clock_seconds=120.0, + estimated_cost_usd=0.05, + ) + + +# --------------------------------------------------------------------------- +# score_candidate +# --------------------------------------------------------------------------- + + +def test_score_candidate_delta_math(): + """delta = base_loss - probe_loss (positive = improvement).""" + from soup_cli.utils.tunability import score_candidate + + base_loss, probe_loss = 2.5, 2.0 + delta = score_candidate(base_loss=base_loss, probe_loss=probe_loss) + assert delta == pytest.approx(0.5) + + +def test_score_candidate_zero_when_no_change(): + from soup_cli.utils.tunability import score_candidate + + assert score_candidate(base_loss=2.0, probe_loss=2.0) == 0.0 + + +def test_score_candidate_negative_when_worse(): + from soup_cli.utils.tunability import score_candidate + + assert score_candidate(base_loss=2.0, probe_loss=2.5) == pytest.approx(-0.5) + + +def test_score_candidate_rejects_non_finite(): + from soup_cli.utils.tunability import score_candidate + + with pytest.raises(ValueError): + score_candidate(base_loss=float("nan"), probe_loss=2.0) + with pytest.raises(ValueError): + score_candidate(base_loss=2.0, probe_loss=float("inf")) + + +def test_score_candidate_rejects_bool(): + from soup_cli.utils.tunability import score_candidate + + with pytest.raises(TypeError): + score_candidate(base_loss=True, probe_loss=2.0) # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# pareto_frontier +# --------------------------------------------------------------------------- + + +def test_pareto_frontier_simple(): + """Maximise delta, minimise cost. Strictly dominated entries get dropped.""" + from soup_cli.utils.tunability import CandidateBase, TunabilityResult, pareto_frontier + + def _mk(name: str, delta: float, cost: float) -> TunabilityResult: + cand = CandidateBase(name=name, repo_id="x/y", params_b=1.0, license_id="apache-2.0") + return TunabilityResult( + candidate=cand, + base_loss=2.5, + probe_loss=2.5 - delta, + delta=delta, + wall_clock_seconds=60.0, + estimated_cost_usd=cost, + ) + + # B dominates A: B has higher delta AND lower cost. + # C is on the frontier: lower delta but lower cost than B. + a = _mk("a", delta=0.1, cost=0.20) + b = _mk("b", delta=0.5, cost=0.10) + c = _mk("c", delta=0.05, cost=0.05) + frontier = pareto_frontier([a, b, c]) + names = {r.candidate.name for r in frontier} + # A is strictly dominated by B (lower delta, higher cost) + assert "a" not in names + assert "b" in names + assert "c" in names + + +def test_pareto_frontier_empty(): + from soup_cli.utils.tunability import pareto_frontier + + assert pareto_frontier([]) == () + + +def test_pareto_frontier_single(): + from soup_cli.utils.tunability import CandidateBase, TunabilityResult, pareto_frontier + + cand = CandidateBase(name="a", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + r = TunabilityResult( + candidate=cand, + base_loss=2.5, + probe_loss=2.0, + delta=0.5, + wall_clock_seconds=60.0, + estimated_cost_usd=0.05, + ) + frontier = pareto_frontier([r]) + assert frontier == (r,) + + +def test_pareto_frontier_returns_tuple(): + from soup_cli.utils.tunability import pareto_frontier + + assert isinstance(pareto_frontier([]), tuple) + + +def test_pareto_frontier_rejects_non_sequence(): + from soup_cli.utils.tunability import pareto_frontier + + with pytest.raises(TypeError): + pareto_frontier("not a list") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# TunabilityReport +# --------------------------------------------------------------------------- + + +def test_tunability_report_frozen(): + from soup_cli.utils.tunability import CandidateBase, TunabilityReport, TunabilityResult + + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + r = TunabilityResult( + candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5, + wall_clock_seconds=60.0, estimated_cost_usd=0.05, + ) + report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64) + with pytest.raises(dataclasses.FrozenInstanceError): + report.probe_steps = 0 # type: ignore[misc] + + +def test_tunability_report_results_tuple(): + from soup_cli.utils.tunability import CandidateBase, TunabilityReport, TunabilityResult + + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + r = TunabilityResult( + candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5, + wall_clock_seconds=60.0, estimated_cost_usd=0.05, + ) + # Lists rejected — frozen=True doesn't make lists immutable + with pytest.raises(TypeError, match="tuple"): + TunabilityReport(results=[r], frontier=(r,), probe_steps=100, holdout_size=64) # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# run_tunability +# --------------------------------------------------------------------------- + + +def test_run_tunability_with_mocked_probe(tmp_path): + """Inject a deterministic probe to exercise the orchestrator.""" + from soup_cli.utils.tunability import ( + CandidateBase, + TunabilityResult, + run_tunability, + ) + + candidates = ( + CandidateBase(name="cand-a", repo_id="x/y", params_b=0.5, license_id="apache-2.0"), + CandidateBase(name="cand-b", repo_id="x/z", params_b=1.0, license_id="mit"), + ) + + def fake_probe(cand: CandidateBase, dataset_path: str, *, probe_steps: int, + holdout_size: int) -> TunabilityResult: + # Synthetic: larger param count → bigger delta, longer wall-clock + return TunabilityResult( + candidate=cand, + base_loss=2.5, + probe_loss=2.5 - cand.params_b * 0.3, + delta=cand.params_b * 0.3, + wall_clock_seconds=cand.params_b * 60.0, + estimated_cost_usd=cand.params_b * 0.05, + ) + + dataset = tmp_path / "data.jsonl" + dataset.write_text('{"prompt": "x", "completion": "y"}\n') + + report = run_tunability( + candidates=candidates, + dataset_path=str(dataset.relative_to(tmp_path)) if False else str(dataset), + probe_steps=50, + holdout_size=16, + probe_fn=fake_probe, + ) + assert len(report.results) == 2 + assert len(report.frontier) >= 1 + assert report.probe_steps == 50 + + +def test_run_tunability_rejects_empty_candidates(tmp_path): + from soup_cli.utils.tunability import run_tunability + + dataset = tmp_path / "data.jsonl" + dataset.write_text("{}\n") + with pytest.raises(ValueError, match="candidates"): + run_tunability( + candidates=(), + dataset_path=str(dataset), + probe_steps=50, + holdout_size=16, + ) + + +def test_run_tunability_rejects_invalid_probe_steps(tmp_path): + from soup_cli.utils.tunability import CandidateBase, run_tunability + + dataset = tmp_path / "data.jsonl" + dataset.write_text("{}\n") + cands = (CandidateBase(name="a", repo_id="x/y", params_b=1.0, license_id="apache-2.0"),) + with pytest.raises(ValueError): + run_tunability( + candidates=cands, + dataset_path=str(dataset), + probe_steps=9, # below min + holdout_size=16, + ) + + +# --------------------------------------------------------------------------- +# write_report / load_report +# --------------------------------------------------------------------------- + + +def test_write_report_atomic_roundtrip(tmp_path, monkeypatch): + from soup_cli.utils.tunability import ( + CandidateBase, + TunabilityReport, + TunabilityResult, + load_report, + write_report, + ) + + monkeypatch.chdir(tmp_path) + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + r = TunabilityResult( + candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5, + wall_clock_seconds=60.0, estimated_cost_usd=0.05, + ) + report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64) + out = tmp_path / "tunability.json" + write_report(report, str(out)) + + loaded = load_report(str(out)) + assert loaded.probe_steps == report.probe_steps + assert len(loaded.results) == 1 + assert loaded.results[0].candidate.name == "x" + + +def test_write_report_outside_cwd_rejected(tmp_path, monkeypatch): + from soup_cli.utils.tunability import ( + CandidateBase, + TunabilityReport, + TunabilityResult, + write_report, + ) + + monkeypatch.chdir(tmp_path) + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + r = TunabilityResult( + candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5, + wall_clock_seconds=60.0, estimated_cost_usd=0.05, + ) + report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64) + outside = tmp_path.parent / "evil.json" + with pytest.raises(ValueError, match="cwd"): + write_report(report, str(outside)) + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_write_report_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.utils.tunability import ( + CandidateBase, + TunabilityReport, + TunabilityResult, + write_report, + ) + + monkeypatch.chdir(tmp_path) + cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0") + r = TunabilityResult( + candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5, + wall_clock_seconds=60.0, estimated_cost_usd=0.05, + ) + report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64) + + target = tmp_path / "real.json" + target.write_text("{}") + link = tmp_path / "link.json" + os.symlink(target, link) + with pytest.raises(ValueError, match="symlink"): + write_report(report, str(link)) + + +def test_write_report_non_report_rejected(tmp_path): + from soup_cli.utils.tunability import write_report + + with pytest.raises(TypeError): + write_report("not a report", str(tmp_path / "out.json")) # type: ignore[arg-type] + + +def test_load_report_missing_file(tmp_path, monkeypatch): + from soup_cli.utils.tunability import load_report + + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + load_report(str(tmp_path / "nope.json")) + + +def test_load_report_invalid_json(tmp_path): + from soup_cli.utils.tunability import load_report + + p = tmp_path / "bad.json" + p.write_text("not json") + with pytest.raises((ValueError, json.JSONDecodeError)): + load_report(str(p)) + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + + +def test_cli_tunability_help(): + from soup_cli.cli import app + + result = runner.invoke(app, ["tunability", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "tunability" in result.output.lower() + + +def test_cli_tunability_list_default_candidates(): + from soup_cli.cli import app + + result = runner.invoke(app, ["tunability", "--list"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + # Should list at least one default candidate name. + assert "qwen" in result.output.lower() or "llama" in result.output.lower() or \ + "phi" in result.output.lower() or "gemma" in result.output.lower() or \ + "smol" in result.output.lower() + + +def test_cli_tunability_requires_dataset(tmp_path, monkeypatch): + """Without --list and without --dataset, exits with usage error.""" + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + result = runner.invoke(app, ["tunability"]) + assert result.exit_code != 0 + + +def test_cli_tunability_outside_cwd(tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "data.jsonl" + outside.write_text("{}\n") + result = runner.invoke(app, ["tunability", "--dataset", str(outside)]) + assert result.exit_code != 0 + + +def test_cli_tunability_plan_only(tmp_path, monkeypatch): + """--plan-only enumerates candidates without running probes.""" + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + dataset = tmp_path / "data.jsonl" + dataset.write_text('{"prompt": "x"}\n') + result = runner.invoke(app, ["tunability", "--dataset", str(dataset), "--plan-only"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +# --------------------------------------------------------------------------- +# Source-wiring regression guards +# --------------------------------------------------------------------------- + + +def test_cli_registers_tunability(): + """cli.py registers the tunability command.""" + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py" + text = src.read_text(encoding="utf-8") + assert "tunability" in text + + +def test_version_bumped_to_0640(): + import soup_cli + + assert soup_cli.__version__ == "0.64.0" + + +def test_no_top_level_heavy_imports(): + """tunability module should not import torch/transformers/peft at top-level.""" + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "tunability.py" + text = src.read_text(encoding="utf-8") + # Heavy deps must be lazy-imported inside functions + for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]: + # Strict line-start match + import re + assert not re.search(bad, text, re.MULTILINE), f"top-level {bad} found" diff --git a/tests/test_v0640_part_b.py b/tests/test_v0640_part_b.py new file mode 100644 index 0000000..a5336e1 --- /dev/null +++ b/tests/test_v0640_part_b.py @@ -0,0 +1,452 @@ +"""v0.64.0 Part B — `soup plan` / `apply` (Terraform UX) tests.""" + +from __future__ import annotations + +import dataclasses +import os +import sys + +import pytest +from typer.testing import CliRunner + +runner = CliRunner() + + +# --------------------------------------------------------------------------- +# Module imports +# --------------------------------------------------------------------------- + + +def test_module_imports(): + from soup_cli.utils import terraform_plan + + assert hasattr(terraform_plan, "TrainingPlan") + assert hasattr(terraform_plan, "TrainingState") + assert hasattr(terraform_plan, "build_plan") + assert hasattr(terraform_plan, "write_state") + assert hasattr(terraform_plan, "read_state") + assert hasattr(terraform_plan, "detect_drift") + assert hasattr(terraform_plan, "compute_config_sha") + assert hasattr(terraform_plan, "DEFAULT_STATE_FILE") + + +# --------------------------------------------------------------------------- +# compute_config_sha +# --------------------------------------------------------------------------- + + +def test_compute_config_sha_deterministic(): + from soup_cli.utils.terraform_plan import compute_config_sha + + a = compute_config_sha({"a": 1, "b": 2}) + b = compute_config_sha({"b": 2, "a": 1}) # key order shouldn't matter + assert a == b + assert len(a) == 64 # SHA-256 hex + + +def test_compute_config_sha_changes_with_content(): + from soup_cli.utils.terraform_plan import compute_config_sha + + assert compute_config_sha({"a": 1}) != compute_config_sha({"a": 2}) + + +def test_compute_config_sha_rejects_non_dict(): + from soup_cli.utils.terraform_plan import compute_config_sha + + with pytest.raises(TypeError): + compute_config_sha("not a dict") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# TrainingPlan +# --------------------------------------------------------------------------- + + +def test_training_plan_frozen(): + from soup_cli.utils.terraform_plan import TrainingPlan + + p = TrainingPlan( + base="meta-llama/Llama-3.2-1B", + task="sft", + config_sha="a" * 64, + dataset_sha="b" * 64, + estimated_cost_usd=0.50, + estimated_minutes=10.0, + peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + p.base = "other" # type: ignore[misc] + + +def test_training_plan_rejects_short_sha(): + from soup_cli.utils.terraform_plan import TrainingPlan + + with pytest.raises(ValueError, match="sha"): + TrainingPlan( + base="m", + task="sft", + config_sha="short", + dataset_sha="b" * 64, + estimated_cost_usd=0.50, + estimated_minutes=10.0, + peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + + +def test_training_plan_rejects_non_finite_cost(): + from soup_cli.utils.terraform_plan import TrainingPlan + + with pytest.raises(ValueError, match="finite"): + TrainingPlan( + base="m", + task="sft", + config_sha="a" * 64, + dataset_sha="b" * 64, + estimated_cost_usd=float("nan"), + estimated_minutes=10.0, + peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + + +def test_training_plan_rejects_bool_cost(): + from soup_cli.utils.terraform_plan import TrainingPlan + + with pytest.raises(TypeError, match="bool"): + TrainingPlan( + base="m", + task="sft", + config_sha="a" * 64, + dataset_sha="b" * 64, + estimated_cost_usd=True, # type: ignore[arg-type] + estimated_minutes=10.0, + peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + + +def test_training_plan_rejects_negative_cost(): + from soup_cli.utils.terraform_plan import TrainingPlan + + with pytest.raises(ValueError, match="negative"): + TrainingPlan( + base="m", + task="sft", + config_sha="a" * 64, + dataset_sha="b" * 64, + estimated_cost_usd=-1.0, + estimated_minutes=10.0, + peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + + +# --------------------------------------------------------------------------- +# build_plan +# --------------------------------------------------------------------------- + + +def test_build_plan_happy(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import build_plan + + monkeypatch.chdir(tmp_path) + dataset = tmp_path / "data.jsonl" + dataset.write_text('{"prompt": "a", "completion": "b"}\n') + + config = { + "base": "meta-llama/Llama-3.2-1B", + "task": "sft", + "data": {"train": str(dataset)}, + "training": {"epochs": 1, "lr": 5e-5, "batch_size": 4}, + } + plan = build_plan(config) + assert plan.base == "meta-llama/Llama-3.2-1B" + assert plan.task == "sft" + assert len(plan.config_sha) == 64 + assert plan.estimated_cost_usd >= 0 + assert plan.estimated_minutes >= 0 + + +def test_build_plan_rejects_missing_base(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import build_plan + + monkeypatch.chdir(tmp_path) + config = {"task": "sft", "data": {"train": "x.jsonl"}} + with pytest.raises(ValueError, match="base"): + build_plan(config) + + +def test_build_plan_rejects_non_dict(): + from soup_cli.utils.terraform_plan import build_plan + + with pytest.raises(TypeError): + build_plan("not a dict") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# write_state / read_state +# --------------------------------------------------------------------------- + + +def test_write_state_roundtrip(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import ( + TrainingPlan, + TrainingState, + read_state, + write_state, + ) + + monkeypatch.chdir(tmp_path) + plan = TrainingPlan( + base="m", + task="sft", + config_sha="a" * 64, + dataset_sha="b" * 64, + estimated_cost_usd=0.50, + estimated_minutes=10.0, + peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None) + out = tmp_path / "soup.tfstate" + write_state(state, str(out)) + + loaded = read_state(str(out)) + assert loaded.plan.base == "m" + assert loaded.applied is False + + +def test_write_state_outside_cwd_rejected(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import TrainingPlan, TrainingState, write_state + + monkeypatch.chdir(tmp_path) + plan = TrainingPlan( + base="m", task="sft", config_sha="a" * 64, dataset_sha="b" * 64, + estimated_cost_usd=0.5, estimated_minutes=10.0, peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None) + outside = tmp_path.parent / "evil.tfstate" + with pytest.raises(ValueError, match="cwd"): + write_state(state, str(outside)) + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_write_state_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import TrainingPlan, TrainingState, write_state + + monkeypatch.chdir(tmp_path) + plan = TrainingPlan( + base="m", task="sft", config_sha="a" * 64, dataset_sha="b" * 64, + estimated_cost_usd=0.5, estimated_minutes=10.0, peak_vram_gb=8.0, + spot_price_usd_per_hour=0.30, + ) + state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None) + target = tmp_path / "real.tfstate" + target.write_text("{}") + link = tmp_path / "link.tfstate" + os.symlink(target, link) + with pytest.raises(ValueError, match="symlink"): + write_state(state, str(link)) + + +def test_read_state_missing(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import read_state + + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + read_state(str(tmp_path / "nope.tfstate")) + + +# --------------------------------------------------------------------------- +# detect_drift +# --------------------------------------------------------------------------- + + +def test_detect_drift_clean(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import ( + TrainingState, + build_plan, + detect_drift, + ) + + monkeypatch.chdir(tmp_path) + dataset = tmp_path / "data.jsonl" + dataset.write_text("{}\n") + config = { + "base": "m", + "task": "sft", + "data": {"train": str(dataset)}, + "training": {"epochs": 1, "lr": 5e-5, "batch_size": 4}, + } + plan = build_plan(config) + state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None) + # Rebuild plan from same config → no drift + plan_now = build_plan(config) + drift = detect_drift(state, plan_now) + assert drift.has_drift is False + + +def test_detect_drift_dirty(tmp_path, monkeypatch): + from soup_cli.utils.terraform_plan import ( + TrainingState, + build_plan, + detect_drift, + ) + + monkeypatch.chdir(tmp_path) + dataset = tmp_path / "data.jsonl" + dataset.write_text("{}\n") + config_v1 = { + "base": "m", + "task": "sft", + "data": {"train": str(dataset)}, + "training": {"epochs": 1, "lr": 5e-5, "batch_size": 4}, + } + plan_v1 = build_plan(config_v1) + state = TrainingState(plan=plan_v1, applied=False, applied_at=None, run_id=None) + # Mutate config + config_v2 = dict(config_v1) + config_v2["training"] = {"epochs": 2, "lr": 5e-5, "batch_size": 4} + plan_v2 = build_plan(config_v2) + drift = detect_drift(state, plan_v2) + assert drift.has_drift is True + assert "config_sha" in drift.changed_fields + + +def test_detect_drift_rejects_non_state(): + from soup_cli.utils.terraform_plan import detect_drift + + with pytest.raises(TypeError): + detect_drift("not state", "not plan") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# CLI smoke +# --------------------------------------------------------------------------- + + +def _write_minimal_config(path): + path.write_text( + "base: meta-llama/Llama-3.2-1B\n" + "task: sft\n" + "data:\n" + " train: ./data.jsonl\n" + "training:\n" + " epochs: 1\n" + " lr: 0.00005\n" + " batch_size: 4\n" + ) + + +def test_cli_plan_help(): + from soup_cli.cli import app + + result = runner.invoke(app, ["plan", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_apply_help(): + from soup_cli.cli import app + + result = runner.invoke(app, ["apply", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_plan_happy(tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + cfg = tmp_path / "soup.yaml" + _write_minimal_config(cfg) + (tmp_path / "data.jsonl").write_text("{}\n") + result = runner.invoke(app, ["plan", "--config", str(cfg)]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "soup.tfstate").exists() + + +def test_cli_plan_outside_cwd_config(tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + outside_cfg = tmp_path.parent / "evil.yaml" + _write_minimal_config(outside_cfg) + result = runner.invoke(app, ["plan", "--config", str(outside_cfg)]) + assert result.exit_code != 0 + + +def test_cli_apply_drift_refused(tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + cfg = tmp_path / "soup.yaml" + _write_minimal_config(cfg) + (tmp_path / "data.jsonl").write_text("{}\n") + # plan first + r = runner.invoke(app, ["plan", "--config", str(cfg)]) + assert r.exit_code == 0, (r.output, repr(r.exception)) + # mutate config + cfg.write_text(cfg.read_text().replace("epochs: 1", "epochs: 99")) + # apply must refuse on drift + r2 = runner.invoke(app, ["apply", "--config", str(cfg)]) + assert r2.exit_code != 0 + assert "drift" in r2.output.lower() or "mismatch" in r2.output.lower() + + +def test_cli_apply_clean(tmp_path, monkeypatch): + """apply with --dry-run after a plan should succeed clean.""" + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + cfg = tmp_path / "soup.yaml" + _write_minimal_config(cfg) + (tmp_path / "data.jsonl").write_text("{}\n") + r = runner.invoke(app, ["plan", "--config", str(cfg)]) + assert r.exit_code == 0, (r.output, repr(r.exception)) + r2 = runner.invoke(app, ["apply", "--config", str(cfg), "--dry-run"]) + assert r2.exit_code == 0, (r2.output, repr(r2.exception)) + + +# --------------------------------------------------------------------------- +# DriftReport frozen +# --------------------------------------------------------------------------- + + +def test_drift_report_frozen(): + from soup_cli.utils.terraform_plan import DriftReport + + d = DriftReport(has_drift=False, changed_fields=()) + with pytest.raises(dataclasses.FrozenInstanceError): + d.has_drift = True # type: ignore[misc] + + +def test_drift_report_changed_fields_is_tuple(): + from soup_cli.utils.terraform_plan import DriftReport + + with pytest.raises(TypeError, match="tuple"): + DriftReport(has_drift=True, changed_fields=["x"]) # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# Source-wiring regression +# --------------------------------------------------------------------------- + + +def test_cli_registers_plan_apply(): + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py" + text = src.read_text(encoding="utf-8") + assert '"plan"' in text or "'plan'" in text or "name=\"plan\"" in text + + +def test_no_heavy_top_level_imports(): + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "terraform_plan.py" + text = src.read_text(encoding="utf-8") + import re + for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]: + assert not re.search(bad, text, re.MULTILINE) diff --git a/tests/test_v0640_part_c.py b/tests/test_v0640_part_c.py new file mode 100644 index 0000000..638d492 --- /dev/null +++ b/tests/test_v0640_part_c.py @@ -0,0 +1,427 @@ +"""v0.64.0 Part C — `soup env` hermetic lockfile + ABI-mismatch detection.""" + +from __future__ import annotations + +import dataclasses +import os +import sys + +import pytest +from typer.testing import CliRunner + +runner = CliRunner() + + +# --------------------------------------------------------------------------- +# Module imports +# --------------------------------------------------------------------------- + + +def test_module_imports(): + from soup_cli.utils import env_lock + + assert hasattr(env_lock, "EnvLock") + assert hasattr(env_lock, "EnvEntry") + assert hasattr(env_lock, "AbiCheck") + assert hasattr(env_lock, "TRACKED_PACKAGES") + assert hasattr(env_lock, "snapshot_env") + assert hasattr(env_lock, "write_lock") + assert hasattr(env_lock, "read_lock") + assert hasattr(env_lock, "check_abi_compat") + assert hasattr(env_lock, "DEFAULT_LOCK_FILE") + + +# --------------------------------------------------------------------------- +# TRACKED_PACKAGES catalogue +# --------------------------------------------------------------------------- + + +def test_tracked_packages_contains_core(): + from soup_cli.utils.env_lock import TRACKED_PACKAGES + + # Should at least include the ABI-sensitive heavyweights + names = {p.lower() for p in TRACKED_PACKAGES} + assert "torch" in names + assert "transformers" in names + assert "peft" in names + + +def test_tracked_packages_is_tuple(): + from soup_cli.utils.env_lock import TRACKED_PACKAGES + + assert isinstance(TRACKED_PACKAGES, tuple) + + +# --------------------------------------------------------------------------- +# EnvEntry +# --------------------------------------------------------------------------- + + +def test_env_entry_frozen(): + from soup_cli.utils.env_lock import EnvEntry + + e = EnvEntry(name="torch", version="2.1.0", source="pip") + with pytest.raises(dataclasses.FrozenInstanceError): + e.version = "0.0" # type: ignore[misc] + + +def test_env_entry_rejects_empty_name(): + from soup_cli.utils.env_lock import EnvEntry + + with pytest.raises(ValueError, match="name"): + EnvEntry(name="", version="1.0", source="pip") + + +def test_env_entry_rejects_null_byte(): + from soup_cli.utils.env_lock import EnvEntry + + with pytest.raises(ValueError, match="null"): + EnvEntry(name="t\x00", version="1.0", source="pip") + + +def test_env_entry_rejects_invalid_source(): + from soup_cli.utils.env_lock import EnvEntry + + with pytest.raises(ValueError, match="source"): + EnvEntry(name="t", version="1.0", source="random-bogus-thing") + + +def test_env_entry_known_sources(): + from soup_cli.utils.env_lock import EnvEntry + + # Just verify each known source instantiates clean + for src in ("pip", "conda", "system", "wheel", "unknown"): + EnvEntry(name="t", version="1.0", source=src) + + +# --------------------------------------------------------------------------- +# EnvLock +# --------------------------------------------------------------------------- + + +def test_env_lock_frozen(): + from soup_cli.utils.env_lock import EnvEntry, EnvLock + + lock = EnvLock( + soup_version="0.64.0", + python_version="3.10.0", + platform="linux", + cuda_version=None, + entries=(EnvEntry(name="torch", version="2.0", source="pip"),), + created_at="2026-05-20T00:00:00+00:00", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + lock.soup_version = "0.0.0" # type: ignore[misc] + + +def test_env_lock_entries_must_be_tuple(): + from soup_cli.utils.env_lock import EnvEntry, EnvLock + + with pytest.raises(TypeError, match="tuple"): + EnvLock( + soup_version="0.64.0", + python_version="3.10.0", + platform="linux", + cuda_version=None, + entries=[EnvEntry(name="t", version="1.0", source="pip")], # type: ignore[arg-type] + created_at="2026-05-20T00:00:00+00:00", + ) + + +def test_env_lock_rejects_null_byte_in_platform(): + from soup_cli.utils.env_lock import EnvLock + + with pytest.raises(ValueError, match="null"): + EnvLock( + soup_version="0.64.0", + python_version="3.10.0", + platform="linux\x00", + cuda_version=None, + entries=(), + created_at="2026-05-20T00:00:00+00:00", + ) + + +# --------------------------------------------------------------------------- +# snapshot_env +# --------------------------------------------------------------------------- + + +def test_snapshot_env_returns_envlock(): + from soup_cli.utils.env_lock import EnvLock, snapshot_env + + lock = snapshot_env() + assert isinstance(lock, EnvLock) + # python_version should at least parse to two segments + assert "." in lock.python_version + + +def test_snapshot_env_entries_non_empty(): + """The snapshot should contain at least one tracked package (pytest).""" + from soup_cli.utils.env_lock import snapshot_env + + lock = snapshot_env() + # The function inspects whatever's installed; pytest will at least be there. + # We just verify the entries tuple is well-formed. + assert isinstance(lock.entries, tuple) + + +# --------------------------------------------------------------------------- +# write_lock / read_lock +# --------------------------------------------------------------------------- + + +def test_write_lock_roundtrip(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import EnvEntry, EnvLock, read_lock, write_lock + + monkeypatch.chdir(tmp_path) + lock = EnvLock( + soup_version="0.64.0", + python_version="3.10.0", + platform="linux", + cuda_version="12.1", + entries=(EnvEntry(name="torch", version="2.0", source="pip"),), + created_at="2026-05-20T00:00:00+00:00", + ) + out = tmp_path / "soup-env.lock" + write_lock(lock, str(out)) + loaded = read_lock(str(out)) + assert loaded.soup_version == "0.64.0" + assert len(loaded.entries) == 1 + assert loaded.entries[0].name == "torch" + + +def test_write_lock_outside_cwd_rejected(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import EnvLock, write_lock + + monkeypatch.chdir(tmp_path) + lock = EnvLock( + soup_version="0.64.0", + python_version="3.10.0", + platform="linux", + cuda_version=None, + entries=(), + created_at="2026-05-20T00:00:00+00:00", + ) + outside = tmp_path.parent / "evil.lock" + with pytest.raises(ValueError, match="cwd"): + write_lock(lock, str(outside)) + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks") +def test_write_lock_symlink_rejected(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import EnvLock, write_lock + + monkeypatch.chdir(tmp_path) + lock = EnvLock( + soup_version="0.64.0", + python_version="3.10.0", + platform="linux", + cuda_version=None, + entries=(), + created_at="2026-05-20T00:00:00+00:00", + ) + target = tmp_path / "real.lock" + target.write_text("{}") + link = tmp_path / "link.lock" + os.symlink(target, link) + with pytest.raises(ValueError, match="symlink"): + write_lock(lock, str(link)) + + +def test_read_lock_missing(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import read_lock + + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + read_lock(str(tmp_path / "nope.lock")) + + +def test_read_lock_invalid_json(tmp_path, monkeypatch): + from soup_cli.utils.env_lock import read_lock + + monkeypatch.chdir(tmp_path) + p = tmp_path / "bad.lock" + p.write_text("not json") + with pytest.raises(ValueError): + read_lock(str(p)) + + +# --------------------------------------------------------------------------- +# AbiCheck +# --------------------------------------------------------------------------- + + +def test_abi_check_frozen(): + from soup_cli.utils.env_lock import AbiCheck + + a = AbiCheck(ok=True, drift_count=0, changes=()) + with pytest.raises(dataclasses.FrozenInstanceError): + a.ok = False # type: ignore[misc] + + +def test_abi_check_changes_is_tuple(): + from soup_cli.utils.env_lock import AbiCheck + + with pytest.raises(TypeError, match="tuple"): + AbiCheck(ok=False, drift_count=1, changes=["x"]) # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# check_abi_compat +# --------------------------------------------------------------------------- + + +def test_check_abi_compat_no_drift(): + from soup_cli.utils.env_lock import EnvEntry, EnvLock, check_abi_compat + + a = EnvLock( + soup_version="0.64.0", python_version="3.10.0", platform="linux", + cuda_version="12.1", + entries=(EnvEntry(name="torch", version="2.0", source="pip"),), + created_at="2026-05-20T00:00:00+00:00", + ) + b = EnvLock( + soup_version="0.64.0", python_version="3.10.0", platform="linux", + cuda_version="12.1", + entries=(EnvEntry(name="torch", version="2.0", source="pip"),), + created_at="2026-05-21T00:00:00+00:00", # timestamp differs but ABI same + ) + report = check_abi_compat(a, b) + assert report.ok is True + assert report.drift_count == 0 + + +def test_check_abi_compat_torch_change(): + from soup_cli.utils.env_lock import EnvEntry, EnvLock, check_abi_compat + + a = EnvLock( + soup_version="0.64.0", python_version="3.10.0", platform="linux", + cuda_version="12.1", + entries=(EnvEntry(name="torch", version="2.0", source="pip"),), + created_at="2026-05-20T00:00:00+00:00", + ) + b = EnvLock( + soup_version="0.64.0", python_version="3.10.0", platform="linux", + cuda_version="12.1", + entries=(EnvEntry(name="torch", version="2.1", source="pip"),), + created_at="2026-05-20T00:00:00+00:00", + ) + report = check_abi_compat(a, b) + assert report.ok is False + assert report.drift_count >= 1 + + +def test_check_abi_compat_cuda_change(): + from soup_cli.utils.env_lock import EnvLock, check_abi_compat + + a = EnvLock( + soup_version="0.64.0", python_version="3.10.0", platform="linux", + cuda_version="12.1", entries=(), + created_at="2026-05-20T00:00:00+00:00", + ) + b = EnvLock( + soup_version="0.64.0", python_version="3.10.0", platform="linux", + cuda_version="11.8", entries=(), + created_at="2026-05-20T00:00:00+00:00", + ) + report = check_abi_compat(a, b) + assert report.ok is False + assert any("cuda" in c.lower() for c in report.changes) + + +def test_check_abi_compat_python_change(): + from soup_cli.utils.env_lock import EnvLock, check_abi_compat + + a = EnvLock( + soup_version="0.64.0", python_version="3.10.0", platform="linux", + cuda_version=None, entries=(), + created_at="2026-05-20T00:00:00+00:00", + ) + b = EnvLock( + soup_version="0.64.0", python_version="3.11.0", platform="linux", + cuda_version=None, entries=(), + created_at="2026-05-20T00:00:00+00:00", + ) + report = check_abi_compat(a, b) + assert report.ok is False + + +def test_check_abi_compat_rejects_non_envlock(): + from soup_cli.utils.env_lock import check_abi_compat + + with pytest.raises(TypeError): + check_abi_compat("not a lock", "not a lock") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# CLI smoke +# --------------------------------------------------------------------------- + + +def test_cli_env_help(): + from soup_cli.cli import app + + result = runner.invoke(app, ["env", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_env_lock_writes_file(tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + result = runner.invoke(app, ["env", "lock"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "soup-env.lock").exists() + + +def test_cli_env_status_no_lock(tmp_path, monkeypatch): + """`env status` without an existing lock file exits with friendly error.""" + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + result = runner.invoke(app, ["env", "status"]) + # Should not crash; either exits non-zero with message or shows empty status. + assert result.exit_code in (0, 1) + + +def test_cli_env_status_with_lock(tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + r1 = runner.invoke(app, ["env", "lock"]) + assert r1.exit_code == 0, (r1.output, repr(r1.exception)) + r2 = runner.invoke(app, ["env", "status"]) + assert r2.exit_code == 0, (r2.output, repr(r2.exception)) + + +def test_cli_env_lock_outside_cwd_rejected(tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "evil.lock" + result = runner.invoke(app, ["env", "lock", "--output", str(outside)]) + assert result.exit_code != 0 + + +# --------------------------------------------------------------------------- +# Source-wiring regression +# --------------------------------------------------------------------------- + + +def test_cli_registers_env(): + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py" + text = src.read_text(encoding="utf-8") + assert '"env"' in text or "'env'" in text or 'name="env"' in text + + +def test_no_heavy_top_level_imports(): + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "env_lock.py" + text = src.read_text(encoding="utf-8") + import re + for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]: + assert not re.search(bad, text, re.MULTILINE) diff --git a/tests/test_v0640_part_d.py b/tests/test_v0640_part_d.py new file mode 100644 index 0000000..9e1b628 --- /dev/null +++ b/tests/test_v0640_part_d.py @@ -0,0 +1,378 @@ +"""v0.64.0 Part D — Hardware-fit calculator tests.""" + +from __future__ import annotations + +import dataclasses +import math + +import pytest + +# --------------------------------------------------------------------------- +# Module imports +# --------------------------------------------------------------------------- + + +def test_module_imports(): + from soup_cli.utils import hardware_fit + + assert hasattr(hardware_fit, "HardwareFitInput") + assert hasattr(hardware_fit, "VRAMBreakdown") + assert hasattr(hardware_fit, "HardwareFitReport") + assert hasattr(hardware_fit, "estimate_peak_vram_gb") + assert hasattr(hardware_fit, "decide_hardware_fit") + assert hasattr(hardware_fit, "validate_seq_len") + assert hasattr(hardware_fit, "validate_batch_size") + assert hasattr(hardware_fit, "VRAM_SAFETY_MARGIN") + + +# --------------------------------------------------------------------------- +# VRAM_SAFETY_MARGIN +# --------------------------------------------------------------------------- + + +def test_safety_margin_is_10pct(): + from soup_cli.utils.hardware_fit import VRAM_SAFETY_MARGIN + + assert VRAM_SAFETY_MARGIN == pytest.approx(0.10, abs=1e-6) + + +# --------------------------------------------------------------------------- +# validate_seq_len +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("v", [64, 1024, 8192, 1_048_576]) +def test_validate_seq_len_happy(v): + from soup_cli.utils.hardware_fit import validate_seq_len + + assert validate_seq_len(v) == v + + +@pytest.mark.parametrize("bad", [True, False, "1024", -1, 0, 63, 1_048_577, 1.5]) +def test_validate_seq_len_rejects(bad): + from soup_cli.utils.hardware_fit import validate_seq_len + + with pytest.raises((TypeError, ValueError)): + validate_seq_len(bad) + + +# --------------------------------------------------------------------------- +# validate_batch_size +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("v", [1, 4, 64, 1024]) +def test_validate_batch_size_happy(v): + from soup_cli.utils.hardware_fit import validate_batch_size + + assert validate_batch_size(v) == v + + +@pytest.mark.parametrize("bad", [True, False, "4", -1, 0, 1025, 1.5]) +def test_validate_batch_size_rejects(bad): + from soup_cli.utils.hardware_fit import validate_batch_size + + with pytest.raises((TypeError, ValueError)): + validate_batch_size(bad) + + +# --------------------------------------------------------------------------- +# HardwareFitInput +# --------------------------------------------------------------------------- + + +def test_input_frozen(): + from soup_cli.utils.hardware_fit import HardwareFitInput + + inp = HardwareFitInput( + params_b=7.0, seq_len=2048, batch_size=4, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + inp.batch_size = 1 # type: ignore[misc] + + +def test_input_rejects_negative_params(): + from soup_cli.utils.hardware_fit import HardwareFitInput + + with pytest.raises(ValueError, match="params"): + HardwareFitInput( + params_b=-1.0, seq_len=2048, batch_size=4, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + + +def test_input_rejects_invalid_quant(): + from soup_cli.utils.hardware_fit import HardwareFitInput + + with pytest.raises(ValueError, match="quant"): + HardwareFitInput( + params_b=7.0, seq_len=2048, batch_size=4, + optimizer="adamw_torch", quant="bogus", peft="lora", + gradient_checkpointing=True, + ) + + +def test_input_rejects_invalid_peft(): + from soup_cli.utils.hardware_fit import HardwareFitInput + + with pytest.raises(ValueError, match="peft"): + HardwareFitInput( + params_b=7.0, seq_len=2048, batch_size=4, + optimizer="adamw_torch", quant="4bit", peft="bogus", + gradient_checkpointing=True, + ) + + +def test_input_rejects_bool_params(): + from soup_cli.utils.hardware_fit import HardwareFitInput + + with pytest.raises(TypeError, match="bool"): + HardwareFitInput( + params_b=True, seq_len=2048, batch_size=4, # type: ignore[arg-type] + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + + +def test_input_rejects_non_bool_gradient_ckpt(): + from soup_cli.utils.hardware_fit import HardwareFitInput + + with pytest.raises(TypeError, match="gradient_checkpointing"): + HardwareFitInput( + params_b=7.0, seq_len=2048, batch_size=4, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=1, # type: ignore[arg-type] + ) + + +# --------------------------------------------------------------------------- +# VRAMBreakdown +# --------------------------------------------------------------------------- + + +def test_vram_breakdown_frozen(): + from soup_cli.utils.hardware_fit import VRAMBreakdown + + v = VRAMBreakdown( + weights_gb=2.0, + optimizer_gb=0.5, + gradients_gb=0.5, + activations_gb=1.0, + overhead_gb=0.5, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + v.weights_gb = 0.0 # type: ignore[misc] + + +def test_vram_breakdown_rejects_negative(): + from soup_cli.utils.hardware_fit import VRAMBreakdown + + with pytest.raises(ValueError, match="negative"): + VRAMBreakdown( + weights_gb=-1.0, + optimizer_gb=0.5, + gradients_gb=0.5, + activations_gb=1.0, + overhead_gb=0.5, + ) + + +def test_vram_breakdown_total(): + from soup_cli.utils.hardware_fit import VRAMBreakdown + + v = VRAMBreakdown( + weights_gb=2.0, + optimizer_gb=0.5, + gradients_gb=0.5, + activations_gb=1.0, + overhead_gb=0.5, + ) + assert v.total_gb == pytest.approx(4.5) + + +# --------------------------------------------------------------------------- +# estimate_peak_vram_gb +# --------------------------------------------------------------------------- + + +def test_estimate_peak_vram_returns_breakdown(): + from soup_cli.utils.hardware_fit import HardwareFitInput, VRAMBreakdown, estimate_peak_vram_gb + + inp = HardwareFitInput( + params_b=7.0, seq_len=2048, batch_size=4, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + breakdown = estimate_peak_vram_gb(inp) + assert isinstance(breakdown, VRAMBreakdown) + assert breakdown.weights_gb > 0 + assert breakdown.total_gb > 0 + + +def test_estimate_peak_vram_4bit_smaller_than_fp16(): + from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb + + base = dict(params_b=7.0, seq_len=1024, batch_size=1, + optimizer="adamw_torch", peft="lora", + gradient_checkpointing=True) + fp16 = estimate_peak_vram_gb(HardwareFitInput(quant="none", **base)) + q4 = estimate_peak_vram_gb(HardwareFitInput(quant="4bit", **base)) + assert q4.weights_gb < fp16.weights_gb + + +def test_estimate_peak_vram_lora_smaller_than_full(): + from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb + + base = dict(params_b=7.0, seq_len=1024, batch_size=1, + optimizer="adamw_torch", quant="4bit", + gradient_checkpointing=True) + full = estimate_peak_vram_gb(HardwareFitInput(peft="full", **base)) + lora = estimate_peak_vram_gb(HardwareFitInput(peft="lora", **base)) + assert lora.optimizer_gb < full.optimizer_gb + + +def test_estimate_peak_vram_seq_len_scales_activations(): + from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb + + base = dict(params_b=1.0, batch_size=1, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=False) + short = estimate_peak_vram_gb(HardwareFitInput(seq_len=512, **base)) + long_ = estimate_peak_vram_gb(HardwareFitInput(seq_len=8192, **base)) + assert long_.activations_gb > short.activations_gb + + +def test_estimate_peak_vram_grad_ckpt_reduces_activations(): + from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb + + base = dict(params_b=7.0, seq_len=4096, batch_size=1, + optimizer="adamw_torch", quant="4bit", peft="lora") + off = estimate_peak_vram_gb(HardwareFitInput(gradient_checkpointing=False, **base)) + on = estimate_peak_vram_gb(HardwareFitInput(gradient_checkpointing=True, **base)) + assert on.activations_gb < off.activations_gb + + +def test_estimate_peak_vram_finite(): + from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb + + inp = HardwareFitInput( + params_b=70.0, seq_len=8192, batch_size=8, + optimizer="adamw_torch", quant="none", peft="full", + gradient_checkpointing=False, + ) + bd = estimate_peak_vram_gb(inp) + assert math.isfinite(bd.total_gb) + + +def test_estimate_peak_vram_rejects_non_input(): + from soup_cli.utils.hardware_fit import estimate_peak_vram_gb + + with pytest.raises(TypeError): + estimate_peak_vram_gb("not an input") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# decide_hardware_fit +# --------------------------------------------------------------------------- + + +def test_decide_hardware_fit_ok(tmp_path): + from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit + + inp = HardwareFitInput( + params_b=1.0, seq_len=1024, batch_size=1, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + # Plenty of headroom + report = decide_hardware_fit(inp, available_vram_gb=24.0) + assert report.ok is True + assert report.peak_vram_gb > 0 + assert report.required_with_margin_gb >= report.peak_vram_gb + + +def test_decide_hardware_fit_oom(): + from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit + + inp = HardwareFitInput( + params_b=70.0, seq_len=8192, batch_size=8, + optimizer="adamw_torch", quant="none", peft="full", + gradient_checkpointing=False, + ) + report = decide_hardware_fit(inp, available_vram_gb=8.0) + assert report.ok is False + assert "exceed" in report.reason.lower() or "oom" in report.reason.lower() or \ + "available" in report.reason.lower() + + +def test_decide_hardware_fit_rejects_negative_vram(): + from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit + + inp = HardwareFitInput( + params_b=1.0, seq_len=1024, batch_size=1, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + with pytest.raises(ValueError, match="vram"): + decide_hardware_fit(inp, available_vram_gb=-1.0) + + +def test_decide_hardware_fit_rejects_bool_vram(): + from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit + + inp = HardwareFitInput( + params_b=1.0, seq_len=1024, batch_size=1, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + with pytest.raises(TypeError, match="bool"): + decide_hardware_fit(inp, available_vram_gb=True) # type: ignore[arg-type] + + +def test_decide_hardware_fit_rejects_non_finite_vram(): + from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit + + inp = HardwareFitInput( + params_b=1.0, seq_len=1024, batch_size=1, + optimizer="adamw_torch", quant="4bit", peft="lora", + gradient_checkpointing=True, + ) + with pytest.raises(ValueError, match="finite"): + decide_hardware_fit(inp, available_vram_gb=float("nan")) + + +def test_hardware_fit_report_frozen(): + from soup_cli.utils.hardware_fit import HardwareFitReport, VRAMBreakdown + + bd = VRAMBreakdown( + weights_gb=1.0, optimizer_gb=0.5, gradients_gb=0.5, + activations_gb=1.0, overhead_gb=0.5, + ) + rep = HardwareFitReport( + ok=True, + peak_vram_gb=3.5, + required_with_margin_gb=3.85, + available_vram_gb=24.0, + breakdown=bd, + reason="ok", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + rep.ok = False # type: ignore[misc] + + +# --------------------------------------------------------------------------- +# Source-wiring regression +# --------------------------------------------------------------------------- + + +def test_no_heavy_top_level_imports(): + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "hardware_fit.py" + text = src.read_text(encoding="utf-8") + import re + for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]: + assert not re.search(bad, text, re.MULTILINE) diff --git a/tests/test_v0640_part_e.py b/tests/test_v0640_part_e.py new file mode 100644 index 0000000..45b650a --- /dev/null +++ b/tests/test_v0640_part_e.py @@ -0,0 +1,259 @@ +"""v0.64.0 Part E — Shell completions with config introspection.""" + +from __future__ import annotations + +import pytest +from typer.testing import CliRunner + +runner = CliRunner() + + +def test_module_imports(): + from soup_cli.utils import completions + + assert hasattr(completions, "SUPPORTED_SHELLS") + assert hasattr(completions, "validate_shell") + assert hasattr(completions, "render_bash_script") + assert hasattr(completions, "render_zsh_script") + assert hasattr(completions, "render_fish_script") + assert hasattr(completions, "render_completion_script") + assert hasattr(completions, "complete_recipe_name") + assert hasattr(completions, "complete_target_modules") + + +# --------------------------------------------------------------------------- +# SUPPORTED_SHELLS +# --------------------------------------------------------------------------- + + +def test_supported_shells(): + from soup_cli.utils.completions import SUPPORTED_SHELLS + + assert "bash" in SUPPORTED_SHELLS + assert "zsh" in SUPPORTED_SHELLS + assert "fish" in SUPPORTED_SHELLS + + +def test_supported_shells_is_frozenset(): + from soup_cli.utils.completions import SUPPORTED_SHELLS + + assert isinstance(SUPPORTED_SHELLS, frozenset) + + +# --------------------------------------------------------------------------- +# validate_shell +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("v", ["bash", "zsh", "fish"]) +def test_validate_shell_happy(v): + from soup_cli.utils.completions import validate_shell + + assert validate_shell(v) == v + + +def test_validate_shell_case_insensitive(): + from soup_cli.utils.completions import validate_shell + + assert validate_shell("BASH") == "bash" + assert validate_shell("Fish") == "fish" + + +@pytest.mark.parametrize( + "bad", + [True, False, None, "", "tcsh", "powershell", "ksh", "csh", "x" * 33], +) +def test_validate_shell_rejects(bad): + from soup_cli.utils.completions import validate_shell + + with pytest.raises((TypeError, ValueError)): + validate_shell(bad) + + +def test_validate_shell_rejects_null_byte(): + from soup_cli.utils.completions import validate_shell + + with pytest.raises(ValueError, match="null"): + validate_shell("ba\x00sh") + + +# --------------------------------------------------------------------------- +# render_*_script +# --------------------------------------------------------------------------- + + +def test_render_bash_script_basic(): + from soup_cli.utils.completions import render_bash_script + + text = render_bash_script() + assert "_soup_complete" in text or "complete -F" in text + assert "soup" in text + + +def test_render_zsh_script_basic(): + from soup_cli.utils.completions import render_zsh_script + + text = render_zsh_script() + assert "#compdef soup" in text or "_soup" in text + + +def test_render_fish_script_basic(): + from soup_cli.utils.completions import render_fish_script + + text = render_fish_script() + assert "complete -c soup" in text or "complete --command soup" in text + + +def test_render_completion_script_dispatch(): + from soup_cli.utils.completions import render_completion_script + + bash = render_completion_script("bash") + assert "soup" in bash + zsh = render_completion_script("zsh") + assert "soup" in zsh + fish = render_completion_script("fish") + assert "soup" in fish + + +def test_render_completion_script_rejects_unknown(): + from soup_cli.utils.completions import render_completion_script + + with pytest.raises(ValueError): + render_completion_script("tcsh") + + +# --------------------------------------------------------------------------- +# complete_recipe_name +# --------------------------------------------------------------------------- + + +def test_complete_recipe_name_returns_list(): + from soup_cli.utils.completions import complete_recipe_name + + suggestions = complete_recipe_name("") + assert isinstance(suggestions, list) + assert len(suggestions) > 0 + + +def test_complete_recipe_name_filters_prefix(): + from soup_cli.utils.completions import complete_recipe_name + + suggestions = complete_recipe_name("llama") + # Every result must start with the prefix (case-insensitive) + for s in suggestions: + assert s.lower().startswith("llama") + + +def test_complete_recipe_name_empty_for_nonsense(): + from soup_cli.utils.completions import complete_recipe_name + + suggestions = complete_recipe_name("definitely-not-a-recipe-zzzzzz") + assert suggestions == [] + + +def test_complete_recipe_name_rejects_bool(): + from soup_cli.utils.completions import complete_recipe_name + + with pytest.raises(TypeError): + complete_recipe_name(True) # type: ignore[arg-type] + + +def test_complete_recipe_name_null_byte_returns_empty(): + from soup_cli.utils.completions import complete_recipe_name + + # Defensive: null byte returns empty rather than raising. + assert complete_recipe_name("foo\x00bar") == [] + + +# --------------------------------------------------------------------------- +# complete_target_modules +# --------------------------------------------------------------------------- + + +def test_complete_target_modules_default(): + from soup_cli.utils.completions import complete_target_modules + + # With no base, returns canonical Llama-shape modules + suggestions = complete_target_modules("", base=None) + assert "q_proj" in suggestions + assert "k_proj" in suggestions + + +def test_complete_target_modules_filters_prefix(): + from soup_cli.utils.completions import complete_target_modules + + suggestions = complete_target_modules("q_", base=None) + for s in suggestions: + assert s.startswith("q_") + + +def test_complete_target_modules_rejects_bool(): + from soup_cli.utils.completions import complete_target_modules + + with pytest.raises(TypeError): + complete_target_modules(True, base=None) # type: ignore[arg-type] + + +def test_complete_target_modules_handles_unknown_base(): + """When base is set but we can't probe it, fall back to default modules.""" + from soup_cli.utils.completions import complete_target_modules + + suggestions = complete_target_modules("", base="some/nonexistent-model") + # Should still return non-empty default + assert len(suggestions) > 0 + + +# --------------------------------------------------------------------------- +# CLI smoke +# --------------------------------------------------------------------------- + + +def test_cli_completions_help(): + from soup_cli.cli import app + + result = runner.invoke(app, ["completions", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_completions_bash(): + from soup_cli.cli import app + + result = runner.invoke(app, ["completions", "bash"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "soup" in result.stdout + + +def test_cli_completions_zsh(): + from soup_cli.cli import app + + result = runner.invoke(app, ["completions", "zsh"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_completions_fish(): + from soup_cli.cli import app + + result = runner.invoke(app, ["completions", "fish"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_completions_unknown_shell(): + from soup_cli.cli import app + + result = runner.invoke(app, ["completions", "tcsh"]) + assert result.exit_code != 0 + + +# --------------------------------------------------------------------------- +# Source-wiring regression +# --------------------------------------------------------------------------- + + +def test_no_heavy_top_level_imports(): + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "completions.py" + text = src.read_text(encoding="utf-8") + import re + for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]: + assert not re.search(bad, text, re.MULTILINE) diff --git a/tests/test_v0640_part_f.py b/tests/test_v0640_part_f.py new file mode 100644 index 0000000..980a2a2 --- /dev/null +++ b/tests/test_v0640_part_f.py @@ -0,0 +1,325 @@ +"""v0.64.0 Part F — License advisor tests.""" + +from __future__ import annotations + +import dataclasses + +import pytest +from typer.testing import CliRunner + +runner = CliRunner() + + +def test_module_imports(): + from soup_cli.utils import license_advisor + + assert hasattr(license_advisor, "DEPLOY_TARGETS") + assert hasattr(license_advisor, "validate_deploy_target") + assert hasattr(license_advisor, "LicenseRecommendation") + assert hasattr(license_advisor, "advise_license_for_target") + assert hasattr(license_advisor, "flag_downstream_risk") + assert hasattr(license_advisor, "DownstreamRisk") + + +# --------------------------------------------------------------------------- +# DEPLOY_TARGETS +# --------------------------------------------------------------------------- + + +def test_deploy_targets_exact(): + from soup_cli.utils.license_advisor import DEPLOY_TARGETS + + assert "b2c" in DEPLOY_TARGETS + assert "defense" in DEPLOY_TARGETS + assert "embedded" in DEPLOY_TARGETS + + +def test_deploy_targets_is_frozenset(): + from soup_cli.utils.license_advisor import DEPLOY_TARGETS + + assert isinstance(DEPLOY_TARGETS, frozenset) + + +# --------------------------------------------------------------------------- +# validate_deploy_target +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("v", ["b2c", "defense", "embedded"]) +def test_validate_deploy_target_happy(v): + from soup_cli.utils.license_advisor import validate_deploy_target + + assert validate_deploy_target(v) == v + + +def test_validate_deploy_target_case_insensitive(): + from soup_cli.utils.license_advisor import validate_deploy_target + + assert validate_deploy_target("B2C") == "b2c" + + +@pytest.mark.parametrize( + "bad", + [True, False, None, "", "consumer", "x" * 33], +) +def test_validate_deploy_target_rejects(bad): + from soup_cli.utils.license_advisor import validate_deploy_target + + with pytest.raises((TypeError, ValueError)): + validate_deploy_target(bad) + + +def test_validate_deploy_target_null_byte(): + from soup_cli.utils.license_advisor import validate_deploy_target + + with pytest.raises(ValueError, match="null"): + validate_deploy_target("b\x002c") + + +# --------------------------------------------------------------------------- +# LicenseRecommendation +# --------------------------------------------------------------------------- + + +def test_license_recommendation_frozen(): + from soup_cli.utils.license_advisor import LicenseRecommendation + + r = LicenseRecommendation( + target="b2c", + recommended_licenses=("apache-2.0", "mit"), + forbidden_licenses=("cc-by-nc-4.0",), + reason="permissive only", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + r.target = "other" # type: ignore[misc] + + +def test_license_recommendation_tuples(): + from soup_cli.utils.license_advisor import LicenseRecommendation + + # Lists rejected (frozen=True doesn't make lists immutable) + with pytest.raises(TypeError, match="tuple"): + LicenseRecommendation( + target="b2c", + recommended_licenses=["apache-2.0"], # type: ignore[arg-type] + forbidden_licenses=(), + reason="x", + ) + + +# --------------------------------------------------------------------------- +# advise_license_for_target +# --------------------------------------------------------------------------- + + +def test_advise_license_b2c(): + from soup_cli.utils.license_advisor import advise_license_for_target + + rec = advise_license_for_target("b2c") + # B2C must include broadly-permissive options + assert "apache-2.0" in rec.recommended_licenses + assert "mit" in rec.recommended_licenses + # Non-commercial licenses must be forbidden for B2C + assert "cc-by-nc-4.0" in rec.forbidden_licenses + + +def test_advise_license_defense(): + from soup_cli.utils.license_advisor import advise_license_for_target + + rec = advise_license_for_target("defense") + # Defense forbids restricted-use community licenses + assert "llama-3.1" in rec.forbidden_licenses or "llama-2" in rec.forbidden_licenses + + +def test_advise_license_embedded(): + from soup_cli.utils.license_advisor import advise_license_for_target + + rec = advise_license_for_target("embedded") + # Embedded systems should avoid strong copyleft + assert "gpl-3.0" in rec.forbidden_licenses + + +def test_advise_license_rejects_unknown_target(): + from soup_cli.utils.license_advisor import advise_license_for_target + + with pytest.raises(ValueError): + advise_license_for_target("bogus") + + +def test_advise_license_rejects_bool(): + from soup_cli.utils.license_advisor import advise_license_for_target + + with pytest.raises(TypeError): + advise_license_for_target(True) # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# DownstreamRisk +# --------------------------------------------------------------------------- + + +def test_downstream_risk_frozen(): + from soup_cli.utils.license_advisor import DownstreamRisk + + r = DownstreamRisk(ok=True, severity="ok", reason="permissive license") + with pytest.raises(dataclasses.FrozenInstanceError): + r.ok = False # type: ignore[misc] + + +def test_downstream_risk_severity_allowlist(): + from soup_cli.utils.license_advisor import DownstreamRisk + + with pytest.raises(ValueError, match="severity"): + DownstreamRisk(ok=False, severity="weird", reason="x") + + +# --------------------------------------------------------------------------- +# flag_downstream_risk +# --------------------------------------------------------------------------- + + +def test_flag_downstream_risk_apache_b2c_ok(): + from soup_cli.utils.license_advisor import flag_downstream_risk + + r = flag_downstream_risk(license_id="apache-2.0", target="b2c", monthly_active_users=10_000) + assert r.ok is True + assert r.severity == "ok" + + +def test_flag_downstream_risk_llama_community_high_mau(): + """Llama community license + commercial use > 700M MAU is the canonical risk.""" + from soup_cli.utils.license_advisor import flag_downstream_risk + + r = flag_downstream_risk( + license_id="llama-3", target="b2c", monthly_active_users=800_000_000 + ) + assert r.ok is False + assert r.severity in ("warn", "block") + + +def test_flag_downstream_risk_llama_community_low_mau_warns(): + """Llama community on small B2C: warning, not block.""" + from soup_cli.utils.license_advisor import flag_downstream_risk + + r = flag_downstream_risk( + license_id="llama-3", target="b2c", monthly_active_users=10_000 + ) + # Either OK with note or a warn — not a hard block + assert r.severity in ("ok", "warn") + + +def test_flag_downstream_risk_nc_b2c_blocked(): + from soup_cli.utils.license_advisor import flag_downstream_risk + + r = flag_downstream_risk( + license_id="cc-by-nc-4.0", target="b2c", monthly_active_users=100 + ) + assert r.ok is False + + +def test_flag_downstream_risk_rejects_negative_mau(): + from soup_cli.utils.license_advisor import flag_downstream_risk + + with pytest.raises(ValueError, match="monthly_active_users"): + flag_downstream_risk( + license_id="apache-2.0", target="b2c", monthly_active_users=-1 + ) + + +def test_flag_downstream_risk_rejects_bool_mau(): + from soup_cli.utils.license_advisor import flag_downstream_risk + + with pytest.raises(TypeError, match="bool"): + flag_downstream_risk( + license_id="apache-2.0", target="b2c", + monthly_active_users=True, # type: ignore[arg-type] + ) + + +def test_flag_downstream_risk_unknown_license_warns(): + from soup_cli.utils.license_advisor import flag_downstream_risk + + r = flag_downstream_risk( + license_id="unknown-soup-test-license", target="b2c", monthly_active_users=1 + ) + # Unknown license should not block silently — surface as a warn + assert r.severity in ("warn", "block") + + +# --------------------------------------------------------------------------- +# CLI smoke +# --------------------------------------------------------------------------- + + +def test_cli_license_advisor_help(): + from soup_cli.cli import app + + result = runner.invoke(app, ["license-advisor", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_license_advisor_b2c(): + from soup_cli.cli import app + + result = runner.invoke(app, ["license-advisor", "--target", "b2c"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_license_advisor_unknown_target(): + from soup_cli.cli import app + + result = runner.invoke(app, ["license-advisor", "--target", "bogus"]) + assert result.exit_code != 0 + + +def test_cli_license_advisor_check_license(): + """--license is the per-license-id risk-check mode.""" + from soup_cli.cli import app + + result = runner.invoke( + app, + [ + "license-advisor", + "--target", + "b2c", + "--license", + "apache-2.0", + "--monthly-active-users", + "10000", + ], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_cli_license_advisor_check_llama_block(): + from soup_cli.cli import app + + result = runner.invoke( + app, + [ + "license-advisor", + "--target", + "b2c", + "--license", + "llama-3", + "--monthly-active-users", + "800000000", + ], + ) + # Either exits non-zero on block, OR prints a block message + assert result.exit_code != 0 or "block" in result.output.lower() + + +# --------------------------------------------------------------------------- +# Source-wiring regression +# --------------------------------------------------------------------------- + + +def test_no_heavy_top_level_imports(): + from pathlib import Path + + src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "license_advisor.py" + text = src.read_text(encoding="utf-8") + import re + for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]: + assert not re.search(bad, text, re.MULTILINE)