RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

feat(v0.64.0): Pre-flight & Tooling — tunability, plan/apply, env, hardware-fit, completions, license-advisor 

Six new top-level commands close axis 1 + 11 of the roadmap: pick the
right base, lock the env, refuse OOMs before launch, and clear
license-clean deploys.

- soup tunability: probe-train 8 candidate bases (Qwen3-0.6/1.7B,
  Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) ->
  Pareto frontier over (delta x cost x license). Live LoRA probe -> v0.64.1.

- soup plan / soup apply: Terraform-shape lock-and-execute. `apply`
  refuses on drift between soup.yaml and soup.tfstate (exit 3).

- soup env lock / status / check: hermetic env lockfile via
  importlib.metadata across 15 ABI-sensitive packages + Python + CUDA.
  `env check` exits 3 on drift.

- Hardware-fit calculator: static analytical 5-bucket VRAM predictor
  with 10% safety margin + actionable hint on OOM.

- soup completions bash|zsh|fish: sourceable shell completion scripts;
  recipe names auto-complete from the 115-recipe catalogue.

- soup license-advisor: per-deploy-target license matrix
  (b2c/defense/embedded) + Llama community + 700M MAU gate (exit 3).
  Composes with v0.60 license-conflict matrix.

Tests: 10035 -> 10306 (+271 net in 7 new files).
Review-fix coverage: 0 CRITICAL + 6 HIGH + 8 MEDIUM + 4 LOW across
consolidated code+security+TDD review wave. Every HIGH lands a regression
test in tests/test_v0640_followups.py (POSIX-skipped symlink rejection,
containment-before-existence ordering, drift-refusal exit-3 end-to-end).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys 2026-05-20 23:45:04 +05:00
parent 5afdf610a7
commit 8b5991674b
24 changed files with 6025 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
CONTRIBUTING.md
View File

@ -107,11 +107,11 @@ soup_cli/
cans/ - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
data/traces/ - Trace-to-Preference harvester (v0.26.0)
data/collators.py - CrossDocCollator for sample packing (v0.33.0)
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
ui/ - Web UI (FastAPI + HTML/JS SPA)
tests/ - Test suite (224 files, 10035 tests)
tests/ - Test suite (231 files, 10306 tests)
examples/ - Real-world config examples and datasets
```

128
README.md
View File

@ -42,14 +42,15 @@ soup train
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
**v0.63.0 — Production Trace Ecosystem: pull traces from any SaaS dashboard, mine your prod logs, ship gates.** Closes axis 7 of the roadmap. Five new top-level commands turn the v0.58 `soup loop` into a full data flywheel without ripping out your existing observability — every Part LIVE on day one (no deferred stubs). Adapters for Langfuse / LangSmith / Helicone / OpenPipe / OpenTelemetry / OpenAI Stored Completions; prompt-prefix mining; uncertainty-based active learning; proper sequential A/B testing with early-stop guarantees; rolling-KL drift alarm with optional Slack/Discord webhooks.
**v0.64.0 — Pre-flight & Tooling: pick the right base, lock the env, refuse OOMs before launch.** Closes axis 1 + axis 11 of the roadmap. Six new top-level commands give operators the rest of the pre-flight surface that v0.54 `soup advise` started: which base model, which compute, which env, which licence. Hosted vendors push their own catalogs and skip CUDA-hell ownership — Soup is local + spans every base.
- **`soup ingest --source <name> --logs <jsonl>`.** Universal trace importer with 6 adapters covering every major observability vendor. Output is a normalised JSONL trace stream that `soup data from-traces` (v0.26) consumes after a thin shim. SaaS pulls happen out-of-band — Soup parses the export, never makes the network call (zero credential-handling threat surface). PII reminder panel on every ingest. `LANGFUSE_KEY` / `LANGSMITH_API_KEY` / `HELICONE_API_KEY` / `OPENPIPE_API_KEY` / `OPENAI_API_KEY` / `OTEL_EXPORTER_OTLP_HEADERS` env-var hints.
- **`soup prune-prompt --input <jsonl> --output <jsonl> --min-frequency 0.95`.** Detect + strip the longest shared system-prompt prefix across training rows so the FT model internalises it (OpenPipe's signature trick, OSS). Binary-search over up-to-32 candidate templates finds the longest prefix above the threshold — no 100%-match early-exit. Two-pass file read with a 100k-row DoS cap.
- **`soup data active-sample --input <jsonl> --budget N`.** Surface the most uncertain prod traces for human review. Max-entropy on single `rm_score` (peak at 0.5) OR pairwise disagreement on dual `rm_scores`. Composes with v0.19 human eval — the output JSONL is a drop-in eval prompt set.
- **`soup ab --input <jsonl> --metric latency|judge_score|retry_rate`.** Proper sequential A/B harness using Wald's classic SPRT for the point alternative. LLR is a martingale under H0 so Type-I error is controlled at every stopping time. Decision: `continue` / `reject_h0` / `accept_h0`. Composes with v0.58 `soup loop canary` — promote (or roll back) as soon as evidence clears the threshold.
- **`soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2`.** Rolling KL on whitespace-tokenised output distribution. Surfaces both behavioural drift ("model now outputs JSON when it used to output prose") AND vocabulary drift ("model has started repeating the same 20 phrases"). Optional `--slack-url` / `--discord-url` webhooks SSRF-validated to v0.51.0 parity (loopback-only HTTP, RFC1918 + 169.254.x rejected). Exit code 3 on drift detected for cron-friendly automation.
- **+219 new tests** (9816 → 10035 net). Review-fix coverage: 1 CRITICAL (mSPRT sign error — earlier draft's malformed LLR drove Type-I error to 1.0 as n grew; replaced with Wald point-alternative SPRT) + 2 HIGH (detect_common_prefix early-exit on 100% match returned shortest prefix not longest + `_MAX_SCAN_ROWS` cap used `pass` instead of `break`) + 3 MEDIUM (`TraceRecord.metadata` now `MappingProxyType`, env-label table deduplicated, drift-alarm precedence parens) + 2 LOW (pooled_se dead-branch + `mean_uncertainty` NaN guard) + 8 follow-up boundary / regression tests. Manual CPU smokes for all 5 commands + 6 failure-mode rejection paths.
- **`soup tunability --dataset <jsonl> [--candidates a,b,c] [--probe-steps N]`.** Probe-train 8 small candidate bases (Qwen3-0.6/1.7B, Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) against a held-out slice and report the (delta-from-base × cost × license) Pareto frontier. `--plan-only` dry-runs the sweep; `--list` shows the built-in catalogue. Live LoRA probe deferred to v0.64.1 — v0.64.0 ships the schema, Pareto math, default catalogue, and a `probe_fn` injection point.
- **`soup plan --config soup.yaml` / `soup apply --config soup.yaml`.** Terraform-shape lock-and-execute for training. `plan` writes `soup.tfstate` with the run's config SHA, dataset SHA, estimated cost, ETA, peak VRAM, and spot price. `apply` refuses if the YAML drifted from the state (exit 3) so you never silently re-spend $0.50 on a mutated config. `--dry-run` validates without running.
- **`soup env lock` / `soup env status` / `soup env check`.** Hermetic env lockfile via `importlib.metadata` — torch / transformers / peft / trl / accelerate / bitsandbytes / flash-attn / xformers / deepspeed / unsloth / vllm + Python + platform + CUDA versions. `check` compares the current env against the lock and exits 3 on ABI-sensitive drift (the "FT worked on Friday, broke on Monday" problem).
- **Hardware-fit calculator.** Static analytical predictor of peak VRAM by class (weights / optimizer / gradients / activations / overhead). Given (params, seq_len, batch_size, optimizer, quant, peft, gradient_checkpointing), it returns a 5-bucket breakdown + an OK/OOM verdict with 10% safety margin and an actionable hint (`--batch-size halve` / `--quantization 4bit` / `--gradient-checkpointing auto`). Composes with v0.40.3 live CUDA OOM probe.
- **`soup completions bash | zsh | fish`.** Sourceable shell completion scripts. `eval "$(soup completions bash)"` adds tab-completion for `soup` + every subcommand in the current shell. `--target-modules` falls back to canonical Llama-shape defaults; recipe names auto-complete from the 115+ catalogue.
- **`soup license-advisor --target b2c|defense|embedded [--license <id> --mau N]`.** Per-deploy-target license matrix. B2C → permissive recommended, non-commercial forbidden. Defense → restricted-use community licenses (Llama / Gemma / Qwen-research / Mistral-research) forbidden because their acceptable-use clauses clash. Embedded → strong-copyleft (GPL/AGPL) forbidden because closed-source firmware redistribution. The per-license check flags the Llama community license + > 700M MAU rule (exit 3) so you don't ship a B2C product that needs a separate Meta licence after the fact. Composes with v0.60 `adapters merge` license-conflict gate.
- **+271 new tests** (10035 → 10306 net). Review-fix coverage from a consolidated code+security+TDD wave: 0 CRITICAL + 6 HIGH (TOCTOU symlink rejection on every new read path; `compute_dataset_sha` containment + symlink-reject; narrow `except Exception` to `(OSError, ValueError)` in env probing) + 8 MEDIUM (strict-JSON config SHA, no `default=str` silent collision; strict-bool `applied` validation; `_activation_bytes` overflow clamp; PEP 604 → `Optional[str]` for Typer Py3.9 compat; tight Llama-family allowlist replaces `.startswith`; Windows `v12.1` CUDA path-parse; containment-before-existence ordering) + 4 LOW (source-grep regression for `atomic_write_text`, MAU upper cap, `Sequence` from `collections.abc`, end-to-end drift-refusal exit-3). Manual CPU smokes for every Part A-F command incl. failure modes (drift exit 3, license block exit 3, completions unknown shell exit 2).
## Why Soup?
@ -3636,6 +3637,14 @@ soup data active-sample --input <jsonl> --output <jsonl> --budget N Top-N uncer
soup ab --input <jsonl> --metric latency|judge_score|retry_rate mSPRT sequential A/B (decision: continue / reject_h0 / accept_h0)
soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2 Rolling-KL drift alarm (exit 3 on drift)
soup drift-alarm ... --slack-url <https> | --discord-url <https> Optional SSRF-validated webhook on drift detected
soup tunability --list List built-in candidate-base catalogue
soup tunability --dataset <jsonl> [--candidates a,b,c] Probe 8 candidate bases + Pareto frontier report
soup plan --config soup.yaml Pre-flight summary + write soup.tfstate
soup apply --config soup.yaml [--dry-run] Lock-and-execute; refuses on drift (exit 3)
soup env lock | status | check Hermetic env lockfile + ABI drift detection (exit 3)
soup completions bash | zsh | fish Shell completion script (sourceable via eval)
soup license-advisor --target b2c|defense|embedded Recommend license-clean base for deploy target
soup license-advisor ... --license <id> --mau N Per-license downstream-risk check (exit 3 on block)
soup version [--full] [--json] Show version (--full: system info, --json: JSON output)
soup --verbose <command> Full traceback on errors
```
@ -4658,6 +4667,111 @@ extraction filter. `soup airgap-bundle` is intentionally a top-level
command (not `soup deploy airgap-bundle`) — it's an export operation,
not a deploy target.
## Tunability Probe (`soup tunability`)
Before committing to a single base model, run a short LoRA probe on every reasonable candidate against your held-out slice. v0.64.0 ships an 8-entry default catalogue covering Qwen3, Llama-3.2, Gemma 3, Phi-4, SmolLM3, and Qwen2.5 across the 0.6 B – 3.8 B band.
```bash
# List the built-in catalogue
soup tunability --list
# Dry-run a sweep across a subset
soup tunability --dataset ./eval.jsonl --candidates qwen3-0.6b,phi-4-mini --plan-only
# Run the full sweep + write a JSON report
soup tunability --dataset ./eval.jsonl --probe-steps 100 --output ./tunability.json
```
The report is a Pareto frontier over (eval delta from base, train cost, license) — candidates that nothing dominates on both axes survive, so you see a clean shortlist instead of a noisy single-leaderboard score. Live LoRA probe lands in v0.64.1; v0.64.0 ships the schema, Pareto math, and a `probe_fn=` injection point.
## Terraform-Style Plan & Apply (`soup plan` / `soup apply`)
A training run is a one-shot infrastructure-shaped operation: spot price, expected cost, base SHA, dataset SHA, peak VRAM. v0.64 borrows Terraform's plan-apply split so you can review the numbers before committing money.
```bash
# Render a pre-flight summary + write soup.tfstate
soup plan --config soup.yaml
# Apply — refuses on drift (exit 3) if the YAML changed since `plan`
soup apply --config soup.yaml
# Validate without actually running anything
soup apply --config soup.yaml --dry-run
```
The state file is a thin JSON envelope; the actual training is still driven by `soup train`. The gate prevents the "wait, why did I spend another $0.50 on the wrong config" surprise.
## Hermetic Env Lockfile (`soup env`)
The "CUDA hell" problem: a fine-tune that worked on Friday breaks on Monday because PyPI silently upgraded `transformers` past the trainer's compat band. v0.34 `soup doctor` surfaces some of this; v0.64 makes it lockable.
```bash
# Snapshot the current env into soup-env.lock
soup env lock
# Print the locked env summary
soup env status
# Compare current env to the lock — exit 3 on ABI-sensitive drift
soup env check
```
`soup-env.lock` captures Python + platform + CUDA + 15 ABI-sensitive packages (torch / transformers / peft / trl / accelerate / bitsandbytes / flash-attn / xformers / deepspeed / unsloth / vllm / sentencepiece / tokenizers / datasets / huggingface-hub). Wire `soup env check` into your CI to refuse silent ABI breakage.
## Hardware-Fit Calculator
Given (params, seq_len, batch_size, optimizer, quant, peft, gradient_checkpointing), the analytical predictor returns a 5-bucket peak-VRAM breakdown (weights / optimizer / gradients / activations / overhead) and an OK/OOM verdict with a 10% safety margin.
```python
from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit
inp = HardwareFitInput(
params_b=7.0, seq_len=2048, batch_size=4,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
report = decide_hardware_fit(inp, available_vram_gb=24.0)
print(report.ok, report.reason)
# True | 'fits: peak 7.76 GB + 10% margin <= 24.00 GB available'
```
When it doesn't fit, the report names actionable knobs: `--batch-size halve`, `--quantization 4bit`, `--gradient-checkpointing auto`. Composes with v0.40.3 live CUDA OOM probe (`make_cuda_probe_fn`) which still runs when `auto_batch_size_strategy: probe`.
## Shell Completions (`soup completions`)
Tab-completion for `soup` + every subcommand. The generated script is Click/Typer-backed so new commands are picked up automatically.
```bash
# Bash
eval "$(soup completions bash)" # current shell
soup completions bash >> ~/.bashrc # permanent
# Zsh
soup completions zsh > "${fpath[1]}/_soup"
# Fish
soup completions fish > ~/.config/fish/completions/soup.fish
```
Recipe names auto-complete from the 115+ catalogue; `--target-modules` falls back to canonical Llama-shape defaults (`q_proj` / `k_proj` / `v_proj` / etc.). Live HF-config introspection per `base` lands in v0.64.1.
## License Advisor (`soup license-advisor`)
Picking a license-clean base for a specific deployment target is a recurring legal-review pain point. v0.64 captures the three most common deploy contexts as a closed allowlist and surfaces the per-license downstream risk.
```bash
# What licenses are safe for a B2C consumer product?
soup license-advisor --target b2c
# Defense — restricted-use community licenses forbidden
soup license-advisor --target defense
# Per-license check: Llama community license + 800M MAU = block (exit 3)
soup license-advisor --target b2c --license llama-3 --monthly-active-users 800000000
```
The Llama-family allowlist is tight (no `.startswith` over-match), so a hypothetical future `llama-permissive-2030` won't false-trigger the 700M-MAU gate. Composes with v0.60 `soup adapters merge --license <id>` for the merge-time conflict gate.
## Changelog
See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history.

5
SECURITY.md
View File

File diff suppressed because one or more lines are too long

2
pyproject.toml
View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "soup-cli"
version = "0.63.0"
version = "0.64.0"
description = "Fine-tune LLMs in one command. No SSH, no config hell."
readme = "README.md"
license = "Apache-2.0"

2
soup_cli/__init__.py
View File

@ -1,3 +1,3 @@
"""Soup CLI — Fine-tune LLMs in one command."""
__version__ = "0.63.0"
__version__ = "0.64.0"

57
soup_cli/cli.py
View File

@ -338,6 +338,63 @@ app.command(
),
)(_drift_alarm_cmd.drift_alarm)
# v0.64.0 Part A — Tunability probe across candidate bases.
from soup_cli.commands import tunability as _tunability_cmd # noqa: E402
app.command(
name="tunability",
help=(
"Probe-train 6-10 small bases on a held-out slice + report "
"Pareto frontier of (eval delta, train cost, license) (v0.64.0)."
),
)(_tunability_cmd.tunability_cmd)
# v0.64.0 Part B — Terraform-shape plan / apply.
from soup_cli.commands import plan as _plan_cmd # noqa: E402
app.command(
name="plan",
help=(
"Render a pre-flight training plan (cost / ETA / SHA / VRAM) "
"and write soup.tfstate for `soup apply` to consult (v0.64.0)."
),
)(_plan_cmd.plan_cmd)
app.command(
name="apply",
help=(
"Execute the planned training run, refusing on drift between "
"soup.yaml and soup.tfstate (v0.64.0)."
),
)(_plan_cmd.apply_cmd)
# v0.64.0 Part C — Hermetic env lockfile.
from soup_cli.commands.env import env_app as _env_app # noqa: E402
app.add_typer(_env_app, name="env")
# v0.64.0 Part E — Shell completions.
from soup_cli.commands import completions as _completions_cmd # noqa: E402
app.command(
name="completions",
help=(
"Emit a bash / zsh / fish completion script. Use with "
"`eval \"$(soup completions bash)\"` (v0.64.0)."
),
)(_completions_cmd.completions_cmd)
# v0.64.0 Part F — License advisor.
from soup_cli.commands import license_advisor as _license_advisor_cmd # noqa: E402
app.command(
name="license-advisor",
help=(
"Recommend a license-clean base for a deploy target "
"(b2c / defense / embedded) + flag downstream risk (v0.64.0)."
),
)(_license_advisor_cmd.license_advisor_cmd)
def _rewrite_advise_argv(argv: list) -> list:
"""Inject `run` between `advise` and a non-subcommand first argument.

37
soup_cli/commands/completions.py Normal file
View File

@ -0,0 +1,37 @@
"""`soup completions <shell>` — emit a bash / zsh / fish completion script.
Sourceable script: `eval "$(soup completions bash)"` adds tab-completion
for the `soup` command in the current shell. Designed for `eval`
consumption — every output goes to stdout exactly once, no panels or
banners.
"""
from __future__ import annotations
import typer
from rich.console import Console
from rich.markup import escape
from soup_cli.utils.completions import render_completion_script
# Console with stderr so the script body (stdout) stays clean for eval.
_err = Console(stderr=True)
def completions_cmd(
shell: str = typer.Argument(
...,
help="One of: bash, zsh, fish.",
),
) -> None:
"""Render a shell completion script for `soup` (v0.64.0)."""
try:
text = render_completion_script(shell)
except (TypeError, ValueError) as exc:
_err.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
# Plain stdout — the script gets sourced by `eval`.
typer.echo(text)
__all__ = ["completions_cmd"]

159
soup_cli/commands/env.py Normal file
View File

@ -0,0 +1,159 @@
"""`soup env` — hermetic env lockfile + ABI status (v0.64.0 Part C).
Sub-commands:
- ``soup env lock`` — snapshot the current env into ``soup-env.lock``.
- ``soup env status`` — print currently-locked env summary.
- ``soup env check`` — compare current env against ``soup-env.lock`` and
report any ABI-sensitive drift (exit 3 on drift).
"""
from __future__ import annotations
import typer
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
from rich.table import Table
from soup_cli.utils.env_lock import (
DEFAULT_LOCK_FILE,
check_abi_compat,
read_lock,
snapshot_env,
write_lock,
)
from soup_cli.utils.paths import is_under_cwd
console = Console()
env_app = typer.Typer(
name="env",
help="Hermetic env lockfile + ABI drift detection (v0.64.0).",
no_args_is_help=True,
)
@env_app.command("lock")
def env_lock_cmd(
output: str = typer.Option(
DEFAULT_LOCK_FILE,
"--output",
"-o",
help="Path to write the lock file (default: ./soup-env.lock).",
),
) -> None:
"""Snapshot the current env into a lock file."""
if "\x00" in output:
console.print("[red]output path must not contain null bytes[/]")
raise typer.Exit(2)
if not is_under_cwd(output):
console.print(f"[red]output {escape(output)!r} is outside cwd[/]")
raise typer.Exit(2)
try:
lock = snapshot_env()
write_lock(lock, output)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
console.print(
Panel(
f"[green]Locked {len(lock.entries)} packages to "
f"{escape(output)}[/]\n"
f"Python: {escape(lock.python_version)} | "
f"Platform: {escape(lock.platform)} | "
f"CUDA: {escape(lock.cuda_version or 'none')}",
title="env lock",
border_style="green",
)
)
@env_app.command("status")
def env_status_cmd(
lock_path: str = typer.Option(
DEFAULT_LOCK_FILE,
"--lock",
help="Path to the lock file (default: ./soup-env.lock).",
),
) -> None:
"""Print currently-locked env summary."""
try:
lock = read_lock(lock_path)
except FileNotFoundError:
console.print(
f"[yellow]No lock file at {escape(lock_path)}; "
"run `soup env lock` first.[/]"
)
raise typer.Exit(1) from None
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
header = Table(title="env status (locked)")
header.add_column("Field")
header.add_column("Value")
header.add_row("soup_version", escape(lock.soup_version))
header.add_row("python_version", escape(lock.python_version))
header.add_row("platform", escape(lock.platform))
header.add_row("cuda_version", escape(lock.cuda_version or "none"))
header.add_row("created_at", escape(lock.created_at))
header.add_row("entries", str(len(lock.entries)))
console.print(header)
if lock.entries:
body = Table(title="packages")
body.add_column("Name")
body.add_column("Version")
body.add_column("Source")
for e in lock.entries:
body.add_row(escape(e.name), escape(e.version), escape(e.source))
console.print(body)
@env_app.command("check")
def env_check_cmd(
lock_path: str = typer.Option(
DEFAULT_LOCK_FILE,
"--lock",
help="Path to the lock file to compare against.",
),
) -> None:
"""Compare the current env against the lock and report drift."""
try:
locked = read_lock(lock_path)
except FileNotFoundError:
console.print(
f"[red]No lock file at {escape(lock_path)}; "
"run `soup env lock` first.[/]"
)
raise typer.Exit(1) from None
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
current = snapshot_env()
report = check_abi_compat(locked, current)
if report.ok:
console.print(
Panel(
"[green]ABI-clean.[/] No drift detected.",
title="env check",
border_style="green",
)
)
return
body = "\n".join(f"- {escape(c)}" for c in report.changes)
console.print(
Panel(
f"[red]{report.drift_count} ABI-sensitive drift(s):[/]\n{body}",
title="env check",
border_style="red",
)
)
raise typer.Exit(3)
__all__ = ["env_app"]

95
soup_cli/commands/license_advisor.py Normal file
View File

@ -0,0 +1,95 @@
"""`soup license-advisor` — pick a license-clean base for a deploy target.
v0.64.0 Part F. Composes with v0.60 Part E ``license_matrix.check_license_compat``
which gates ``soup adapters merge``.
"""
from __future__ import annotations
from typing import Optional
import typer
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
from rich.table import Table
from soup_cli.utils.license_advisor import (
advise_license_for_target,
flag_downstream_risk,
)
console = Console()
def license_advisor_cmd(
target: str = typer.Option(
...,
"--target",
"-t",
help="Deploy target: b2c | defense | embedded.",
),
license_id: Optional[str] = typer.Option(
None,
"--license",
help="Optional: per-license risk check (e.g. apache-2.0, llama-3).",
),
monthly_active_users: int = typer.Option(
0,
"--monthly-active-users",
"--mau",
help="Expected MAU for the per-license risk check (default 0).",
),
) -> None:
"""Recommend a license-clean base + flag downstream risk (v0.64.0)."""
try:
rec = advise_license_for_target(target)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
table = Table(title=f"License advisor — target: {rec.target}")
table.add_column("Category")
table.add_column("Licenses")
table.add_row(
"[green]Recommended[/]",
", ".join(escape(lic) for lic in rec.recommended_licenses),
)
table.add_row(
"[red]Forbidden[/]",
", ".join(escape(lic) for lic in rec.forbidden_licenses),
)
console.print(table)
console.print(
Panel(escape(rec.reason), title="Reason", border_style="dim")
)
if license_id:
try:
risk = flag_downstream_risk(
license_id=license_id,
target=target,
monthly_active_users=monthly_active_users,
)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
border = {
"ok": "green",
"warn": "yellow",
"block": "red",
}.get(risk.severity, "yellow")
console.print(
Panel(
f"[bold]severity: {escape(risk.severity)}[/]\n"
f"{escape(risk.reason)}",
title=f"Risk check: {escape(license_id)}",
border_style=border,
)
)
if risk.severity == "block":
raise typer.Exit(3)
__all__ = ["license_advisor_cmd"]

206
soup_cli/commands/plan.py Normal file
View File

@ -0,0 +1,206 @@
"""`soup plan` — Terraform-shape pre-flight summary (v0.64.0 Part B).
Renders the cost / ETA / SHA / peak-VRAM summary for a planned training
run and writes ``soup.tfstate`` for ``soup apply`` to consult.
"""
from __future__ import annotations
import datetime as _dt
import typer
import yaml
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
from rich.table import Table
from soup_cli.utils.paths import is_under_cwd
from soup_cli.utils.terraform_plan import (
DEFAULT_STATE_FILE,
TrainingState,
build_plan,
write_state,
)
console = Console()
def _load_yaml_config(path: str) -> dict:
import os
import stat as _stat
if not isinstance(path, str):
raise TypeError(f"config path must be str, got {type(path).__name__}")
if "\x00" in path:
raise ValueError("config path must not contain null bytes")
if not is_under_cwd(path):
raise ValueError(f"config {path!r} is outside cwd")
# TOCTOU defence: reject symlink at the YAML path BEFORE open so a
# pre-placed `soup.yaml -> /etc/shadow` cannot redirect the read.
if os.path.lexists(path):
st = os.lstat(path)
if _stat.S_ISLNK(st.st_mode):
raise ValueError("config path must not be a symlink")
with open(path, encoding="utf-8") as fh:
data = yaml.safe_load(fh)
if not isinstance(data, dict):
raise ValueError("config must parse to a dict")
return data
def plan_cmd(
config: str = typer.Option(
"soup.yaml", "--config", "-c", help="Path to soup.yaml."
),
state_file: str = typer.Option(
DEFAULT_STATE_FILE,
"--state",
help="Path to write the state file (default: ./soup.tfstate).",
),
) -> None:
"""Render a pre-flight training plan + write ``soup.tfstate``."""
try:
cfg = _load_yaml_config(config)
except FileNotFoundError:
console.print(f"[red]Config not found: {escape(config)}[/]")
raise typer.Exit(1) from None
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
try:
plan = build_plan(cfg)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
state = TrainingState(
plan=plan,
applied=False,
applied_at=None,
run_id=None,
)
try:
write_state(state, state_file)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
table = Table(title="Training plan")
table.add_column("Field")
table.add_column("Value")
table.add_row("base", escape(plan.base))
table.add_row("task", escape(plan.task))
table.add_row("config_sha", escape(plan.config_sha[:16] + "..."))
table.add_row("dataset_sha", escape(plan.dataset_sha[:16] + "..."))
table.add_row("estimated_cost", f"${plan.estimated_cost_usd:.4f}")
table.add_row("estimated_minutes", f"{plan.estimated_minutes:.1f}")
table.add_row("peak_vram_gb", f"{plan.peak_vram_gb:.1f}")
table.add_row("spot_price_usd_per_hour", f"${plan.spot_price_usd_per_hour:.2f}")
console.print(table)
console.print(
Panel(
f"[green]Plan written to {escape(state_file)}.[/]\n"
"Review the numbers, then run `soup apply` to execute.",
title="plan",
border_style="green",
)
)
def apply_cmd(
config: str = typer.Option(
"soup.yaml", "--config", "-c", help="Path to soup.yaml."
),
state_file: str = typer.Option(
DEFAULT_STATE_FILE,
"--state",
help="Path to the state file written by `soup plan`.",
),
dry_run: bool = typer.Option(
False,
"--dry-run",
help="Verify drift and exit without invoking training.",
),
) -> None:
"""Execute the planned training run, refusing on drift (v0.64.0)."""
from soup_cli.utils.terraform_plan import build_plan, detect_drift, read_state
try:
cfg = _load_yaml_config(config)
except FileNotFoundError:
console.print(f"[red]Config not found: {escape(config)}[/]")
raise typer.Exit(1) from None
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
try:
state = read_state(state_file)
except FileNotFoundError:
console.print(
f"[red]No state file at {escape(state_file)}; run `soup plan` first.[/]"
)
raise typer.Exit(1) from None
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
try:
plan_now = build_plan(cfg)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
drift = detect_drift(state, plan_now)
if drift.has_drift:
fields = ", ".join(drift.changed_fields)
console.print(
Panel(
f"[red]Drift detected.[/] Plan no longer matches state: {escape(fields)}\n"
f"Run `soup plan` again to refresh, then re-apply.",
title="apply",
border_style="red",
)
)
raise typer.Exit(3)
if dry_run:
console.print(
Panel(
f"[green]Dry-run.[/] No drift detected; would proceed.\n"
f"Expected cost: ${plan_now.estimated_cost_usd:.4f} | "
f"ETA: {plan_now.estimated_minutes:.1f} min",
title="apply",
border_style="green",
)
)
return
# Mark state applied; the actual `soup train` invocation is left to
# the operator. Future v0.64.1 may inline a subprocess call.
new_state = TrainingState(
plan=state.plan,
applied=True,
applied_at=_dt.datetime.now(_dt.timezone.utc).isoformat(),
run_id=None,
)
try:
write_state(new_state, state_file)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
console.print(
Panel(
f"[green]Plan locked in {escape(state_file)}.[/]\n"
f"Now run: [bold]soup train --config {escape(config)}[/]\n"
"(The live `apply -> train` subprocess handoff lands in v0.64.1.)",
title="apply",
border_style="green",
)
)
__all__ = ["apply_cmd", "plan_cmd"]

199
soup_cli/commands/tunability.py Normal file
View File

@ -0,0 +1,199 @@
"""`soup tunability` — probe-train candidate bases + report Pareto frontier.
CLI surface for the v0.64.0 Part A pre-flight probe. Live LoRA probe lands
in v0.64.1 (see ``utils/tunability._default_probe`` docstring).
"""
from __future__ import annotations
from typing import Optional
import typer
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
from rich.table import Table
from soup_cli.utils.paths import is_under_cwd
from soup_cli.utils.tunability import (
DEFAULT_CANDIDATES,
CandidateBase,
run_tunability,
write_report,
)
console = Console()
def _resolve_candidates(names: Optional[str]) -> tuple[CandidateBase, ...]:
"""Resolve a comma-separated name list against ``DEFAULT_CANDIDATES``."""
if not names:
return DEFAULT_CANDIDATES
requested = [n.strip().lower() for n in names.split(",") if n.strip()]
if not requested:
return DEFAULT_CANDIDATES
by_name = {c.name.lower(): c for c in DEFAULT_CANDIDATES}
resolved: list[CandidateBase] = []
for name in requested:
if name not in by_name:
available = ", ".join(sorted(by_name.keys()))
raise typer.BadParameter(
f"unknown candidate {name!r}; known: {available}"
)
resolved.append(by_name[name])
return tuple(resolved)
def tunability_cmd(
dataset: Optional[str] = typer.Option(
None,
"--dataset",
"-d",
help="Path to held-out JSONL slice used for the probe.",
),
candidates: Optional[str] = typer.Option(
None,
"--candidates",
help="Comma-separated subset of default candidates by name.",
),
probe_steps: int = typer.Option(
100,
"--probe-steps",
help="Steps per candidate probe (10-10000).",
),
holdout_size: int = typer.Option(
64,
"--holdout-size",
help="Number of holdout rows used for delta scoring (10-100000).",
),
output: Optional[str] = typer.Option(
None,
"--output",
"-o",
help="Optional path for the JSON report.",
),
plan_only: bool = typer.Option(
False,
"--plan-only",
help="Print planned candidates + cost estimate, exit without probing.",
),
list_only: bool = typer.Option(
False,
"--list",
help="List built-in candidate catalogue + exit.",
),
) -> None:
"""Probe-train candidate bases + report Pareto frontier (v0.64.0)."""
if list_only:
table = Table(title="Default tunability candidates")
table.add_column("Name")
table.add_column("Repo")
table.add_column("Params (B)", justify="right")
table.add_column("License")
for c in DEFAULT_CANDIDATES:
table.add_row(
escape(c.name),
escape(c.repo_id),
f"{c.params_b:.2f}",
escape(c.license_id),
)
console.print(table)
return
if not dataset:
console.print(
"[red]--dataset is required (use --list to see the catalogue, "
"--plan-only to dry-run).[/]"
)
raise typer.Exit(2)
# cwd containment + null-byte rejection on dataset
if "\x00" in dataset:
console.print("[red]dataset path must not contain null bytes[/]")
raise typer.Exit(2)
if not is_under_cwd(dataset):
console.print(f"[red]dataset {escape(dataset)!r} is outside cwd[/]")
raise typer.Exit(2)
try:
cands = _resolve_candidates(candidates)
except typer.BadParameter as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
if plan_only:
table = Table(title="Planned tunability sweep")
table.add_column("Name")
table.add_column("Params (B)", justify="right")
table.add_column("License")
for c in cands:
table.add_row(
escape(c.name),
f"{c.params_b:.2f}",
escape(c.license_id),
)
console.print(table)
console.print(
Panel(
f"[yellow]Plan-only.[/] Would run {len(cands)} probes "
f"x {probe_steps} steps on holdout={holdout_size}.\n"
f"Live LoRA probe lands in v0.64.1.",
title="tunability",
border_style="yellow",
)
)
return
try:
report = run_tunability(
candidates=cands,
dataset_path=dataset,
probe_steps=probe_steps,
holdout_size=holdout_size,
)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
table = Table(title="Tunability sweep")
table.add_column("Name")
table.add_column("Delta", justify="right")
table.add_column("Wall (s)", justify="right")
table.add_column("Cost ($)", justify="right")
table.add_column("License")
on_frontier = {r.candidate.name for r in report.frontier}
for r in report.results:
marker = "[green]*[/]" if r.candidate.name in on_frontier else " "
table.add_row(
f"{marker} {escape(r.candidate.name)}",
f"{r.delta:+.4f}",
f"{r.wall_clock_seconds:.0f}",
f"{r.estimated_cost_usd:.4f}",
escape(r.candidate.license_id),
)
console.print(table)
console.print(
Panel(
f"Pareto frontier: {len(report.frontier)} / {len(report.results)} "
"candidates. Delta is base_loss - probe_loss (higher = better).",
title="tunability",
border_style="green",
)
)
if output:
if "\x00" in output:
console.print("[red]output path must not contain null bytes[/]")
raise typer.Exit(2)
if not is_under_cwd(output):
console.print(f"[red]output {escape(output)!r} is outside cwd[/]")
raise typer.Exit(2)
try:
write_report(report, output)
except (TypeError, ValueError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(2) from exc
console.print(f"[green]Report written to {escape(output)}[/]")
__all__ = ["tunability_cmd"]

194
soup_cli/utils/completions.py Normal file
View File

@ -0,0 +1,194 @@
"""Shell completion script generators + dynamic value completers.
`soup completions <shell>` emits a sourceable bash / zsh / fish script.
The dynamic completers (``complete_recipe_name`` /
``complete_target_modules``) are exposed for use as
``shell_complete=...`` callbacks on Typer options.
Live config introspection (probe the operator's actual ``base`` model
for its layer names) lands in v0.64.1; v0.64.0 ships canonical Llama-
shape defaults that cover ~80% of common bases.
"""
from __future__ import annotations
from typing import List, Optional
SUPPORTED_SHELLS = frozenset({"bash", "zsh", "fish"})
_MAX_SHELL_LEN = 32
# Canonical attention/mlp module names that cover Llama / Qwen / Mistral
# / Gemma / Phi families. When ``base`` is supplied and we can probe its
# config, we'd return only what's actually there — that lookup is the
# v0.64.1 deliverable.
_DEFAULT_TARGET_MODULES: tuple[str, ...] = (
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"lm_head",
"embed_tokens",
)
def validate_shell(value: object) -> str:
"""Normalise + validate a shell name against ``SUPPORTED_SHELLS``."""
if isinstance(value, bool):
raise TypeError("shell must be str, not bool")
if not isinstance(value, str):
raise TypeError(f"shell must be str, got {type(value).__name__}")
if not value:
raise ValueError("shell must be non-empty")
if "\x00" in value:
raise ValueError("shell must not contain null bytes")
if len(value) > _MAX_SHELL_LEN:
raise ValueError(f"shell name too long (> {_MAX_SHELL_LEN} chars)")
normalised = value.lower().strip()
if normalised not in SUPPORTED_SHELLS:
allowed = ", ".join(sorted(SUPPORTED_SHELLS))
raise ValueError(f"unknown shell {value!r}; known: {allowed}")
return normalised
def render_bash_script() -> str:
"""Render a bash completion script for the `soup` CLI.
Defers to Typer/Click's built-in ``COMPLETE`` env machinery so the
completion stays in sync with the live Typer app (new commands /
flags are picked up automatically).
"""
return (
"# Soup bash completion (v0.64.0)\n"
"# Source this file from ~/.bashrc:\n"
"# eval \"$(soup completions bash)\"\n"
"_soup_complete() {\n"
" local IFS=$'\\n'\n"
" local response\n"
" response=$(env COMP_WORDS=\"${COMP_WORDS[*]}\" \\\n"
" COMP_CWORD=$COMP_CWORD \\\n"
" _SOUP_COMPLETE=bash_complete \\\n"
" $1 2>/dev/null)\n"
" for completion in $response; do\n"
" IFS=',' read type value <<< \"$completion\"\n"
" if [[ $type == 'plain' ]]; then\n"
" COMPREPLY+=(\"$value\")\n"
" fi\n"
" done\n"
" return 0\n"
"}\n"
"complete -o nosort -F _soup_complete soup\n"
)
def render_zsh_script() -> str:
"""Render a zsh completion script for `soup`."""
return (
"#compdef soup\n"
"# Soup zsh completion (v0.64.0)\n"
"# Source this file from ~/.zshrc:\n"
"# eval \"$(soup completions zsh)\"\n"
"_soup_complete() {\n"
" local -a completions\n"
" local -a completions_with_descriptions\n"
" local -a response\n"
" response=(\"${(@f)$(env COMP_WORDS=\"${words[*]}\" \\\n"
" COMP_CWORD=$((CURRENT-1)) \\\n"
" _SOUP_COMPLETE=zsh_complete soup 2>/dev/null)}\")\n"
" for type_value in \"${response[@]}\"; do\n"
" IFS=',' read -r -A parts <<< \"$type_value\"\n"
" completions+=(\"${parts[2]}\")\n"
" done\n"
" _describe '' completions\n"
"}\n"
"compdef _soup_complete soup\n"
)
def render_fish_script() -> str:
"""Render a fish completion script for `soup`."""
return (
"# Soup fish completion (v0.64.0)\n"
"# Source this file from ~/.config/fish/completions/soup.fish\n"
"function _soup_complete\n"
" set -l response (env _SOUP_COMPLETE=fish_complete \\\n"
" COMP_WORDS=(commandline -cp) \\\n"
" COMP_CWORD=(commandline -t) soup 2>/dev/null)\n"
" for item in $response\n"
" set parts (string split \",\" $item)\n"
" echo $parts[2]\n"
" end\n"
"end\n"
"complete -c soup -f -a \"(_soup_complete)\"\n"
)
def render_completion_script(shell: object) -> str:
"""Dispatch on shell name. Validates + renders one of the three scripts."""
normalised = validate_shell(shell)
if normalised == "bash":
return render_bash_script()
if normalised == "zsh":
return render_zsh_script()
if normalised == "fish":
return render_fish_script()
# Unreachable thanks to ``validate_shell``; defensive default.
raise ValueError(f"unhandled shell {normalised!r}")
def complete_recipe_name(prefix: object) -> List[str]:
"""Suggest recipe names matching ``prefix`` (case-insensitive).
Backed by ``soup_cli.recipes.catalog.list_recipes`` (lazy import).
"""
if isinstance(prefix, bool):
raise TypeError("prefix must be str, not bool")
if not isinstance(prefix, str):
raise TypeError(f"prefix must be str, got {type(prefix).__name__}")
if "\x00" in prefix:
# Defensive: shell completers should never raise.
return []
try:
from soup_cli.recipes.catalog import RECIPES
except ImportError: # pragma: no cover
return []
p = prefix.lower()
return [name for name in RECIPES if name.lower().startswith(p)]
def complete_target_modules(
prefix: object,
*,
base: Optional[str] = None,
) -> List[str]:
"""Suggest ``target_modules`` values for the chosen ``base`` model.
v0.64.0 returns the canonical Llama-shape defaults filtered by
``prefix``. Live per-base introspection (load the HF config, walk
its module tree) is the v0.64.1 deliverable.
"""
if isinstance(prefix, bool):
raise TypeError("prefix must be str, not bool")
if not isinstance(prefix, str):
raise TypeError(f"prefix must be str, got {type(prefix).__name__}")
if base is not None and not isinstance(base, str):
raise TypeError(f"base must be str | None, got {type(base).__name__}")
if "\x00" in prefix:
return []
# base-specific introspection deferred to v0.64.1; fall through to
# the canonical defaults for now.
return [m for m in _DEFAULT_TARGET_MODULES if m.startswith(prefix)]
__all__ = [
"SUPPORTED_SHELLS",
"complete_recipe_name",
"complete_target_modules",
"render_bash_script",
"render_completion_script",
"render_fish_script",
"render_zsh_script",
"validate_shell",
]

366
soup_cli/utils/env_lock.py Normal file
View File

@ -0,0 +1,366 @@
"""`soup env` — hermetic env lockfile + ABI-mismatch detection (v0.64.0 Part C).
The "CUDA hell" problem: a fine-tune that worked on Friday breaks on
Monday because PyPI silently upgraded ``transformers`` past the trainer's
compat band, or because the box rebuilt with a different CUDA. v0.34
``soup doctor`` surfaces some of this; v0.64 makes it lockable.
``snapshot_env`` reads Python + CUDA + key package versions from the
current interpreter via ``importlib.metadata`` (no network, no shell-out
to pip beyond what stdlib already does). ``write_lock`` persists the
snapshot as ``soup-env.lock``. ``check_abi_compat`` compares two locks
and produces an ``AbiCheck`` report listing ABI-sensitive drifts.
Live full uv/nix-backed install + recreate lands in v0.64.1; v0.64.0
ships the schema, snapshotter, comparator, and CLI surface so an
operator can capture an environment + detect drift today.
Public surface:
- ``EnvEntry`` frozen dataclass (name / version / source).
- ``EnvLock`` frozen dataclass (soup version / python / platform / cuda
/ entries tuple / timestamp).
- ``AbiCheck`` frozen dataclass (ok / drift_count / changes tuple).
- ``TRACKED_PACKAGES`` tuple of names whose drift is ABI-relevant.
- ``snapshot_env()`` -> EnvLock.
- ``write_lock(lock, path)`` / ``read_lock(path)`` -> atomic JSON.
- ``check_abi_compat(a, b)`` -> AbiCheck.
- ``DEFAULT_LOCK_FILE = "soup-env.lock"``.
"""
from __future__ import annotations
import datetime as _dt
import json
import os
import platform as _platform
import sys
from dataclasses import dataclass
from typing import Optional, Tuple
from soup_cli.utils.paths import atomic_write_text, is_under_cwd
DEFAULT_LOCK_FILE = "soup-env.lock"
_MAX_NAME_LEN = 256
_MAX_VERSION_LEN = 128
_MAX_PLATFORM_LEN = 256
_MAX_PY_VERSION_LEN = 64
_MAX_CUDA_VERSION_LEN = 64
_MAX_ENTRIES = 4096
# Source allowlist — defends against schema drift on read.
_VALID_SOURCES = frozenset({"pip", "conda", "system", "wheel", "unknown"})
# ABI-sensitive packages — drift here is most likely to break training.
TRACKED_PACKAGES: Tuple[str, ...] = (
"torch",
"transformers",
"peft",
"trl",
"accelerate",
"datasets",
"bitsandbytes",
"huggingface-hub",
"flash-attn",
"xformers",
"deepspeed",
"unsloth",
"vllm",
"sentencepiece",
"tokenizers",
)
def _check_non_empty_str(value: object, fld: str, *, max_len: int) -> None:
if isinstance(value, bool):
raise TypeError(f"{fld} must be str, not bool")
if not isinstance(value, str):
raise TypeError(f"{fld} must be str, got {type(value).__name__}")
if not value:
raise ValueError(f"{fld} must be non-empty")
if "\x00" in value:
raise ValueError(f"{fld} must not contain null bytes")
if len(value) > max_len:
raise ValueError(f"{fld} too long (> {max_len} chars)")
@dataclass(frozen=True)
class EnvEntry:
"""One package -> version row in the lockfile."""
name: str
version: str
source: str
def __post_init__(self) -> None:
_check_non_empty_str(self.name, "name", max_len=_MAX_NAME_LEN)
_check_non_empty_str(self.version, "version", max_len=_MAX_VERSION_LEN)
_check_non_empty_str(self.source, "source", max_len=32)
if self.source not in _VALID_SOURCES:
allowed = ", ".join(sorted(_VALID_SOURCES))
raise ValueError(
f"source must be one of {{{allowed}}}, got {self.source!r}"
)
@dataclass(frozen=True)
class EnvLock:
"""Captured snapshot of the running environment."""
soup_version: str
python_version: str
platform: str
cuda_version: Optional[str]
entries: Tuple[EnvEntry, ...]
created_at: str
def __post_init__(self) -> None:
_check_non_empty_str(self.soup_version, "soup_version", max_len=64)
_check_non_empty_str(
self.python_version, "python_version", max_len=_MAX_PY_VERSION_LEN
)
_check_non_empty_str(self.platform, "platform", max_len=_MAX_PLATFORM_LEN)
if self.cuda_version is not None:
_check_non_empty_str(
self.cuda_version, "cuda_version", max_len=_MAX_CUDA_VERSION_LEN
)
if not isinstance(self.entries, tuple):
raise TypeError("entries must be a tuple of EnvEntry")
if len(self.entries) > _MAX_ENTRIES:
raise ValueError(
f"too many entries ({len(self.entries)} > {_MAX_ENTRIES})"
)
for entry in self.entries:
if not isinstance(entry, EnvEntry):
raise TypeError("every entry must be EnvEntry")
_check_non_empty_str(self.created_at, "created_at", max_len=64)
@dataclass(frozen=True)
class AbiCheck:
"""Outcome of an ABI-compat comparison between two ``EnvLock``s."""
ok: bool
drift_count: int
changes: Tuple[str, ...]
def __post_init__(self) -> None:
if not isinstance(self.ok, bool):
raise TypeError("ok must be bool")
if isinstance(self.drift_count, bool) or not isinstance(self.drift_count, int):
raise TypeError("drift_count must be int")
if self.drift_count < 0:
raise ValueError("drift_count must be >= 0")
if not isinstance(self.changes, tuple):
raise TypeError("changes must be a tuple of str")
for entry in self.changes:
if not isinstance(entry, str):
raise TypeError("changes entries must be str")
def _detect_cuda_version() -> Optional[str]:
"""Best-effort CUDA version probe via env / nvidia-smi / torch.
Returns ``None`` if no CUDA found. Lazy-imports torch only if it's
already in ``sys.modules`` so we never pay the import on a CPU box.
Path-parse handles POSIX (``/usr/local/cuda-12.1``) AND Windows
(``C:\\Program Files\\...\\CUDA\\v12.1``) — splits on every common
separator and strips a leading ``v`` when present.
"""
cuda_env = os.environ.get("CUDA_VERSION") or os.environ.get("CUDA_HOME")
if cuda_env:
# Split on both POSIX and Windows separators so e.g.
# `C:\Program Files\...\CUDA\v12.1` produces `v12.1` as the last
# token instead of one giant path.
tokens = cuda_env.replace("\\", "/").split("/")
for tok in reversed(tokens):
cleaned = tok.lstrip("v") # strip Windows `v` prefix
if cleaned and cleaned[0].isdigit():
return cleaned
if "-" in tok:
tail = tok.rsplit("-", 1)[-1].lstrip("v")
if tail and tail[0].isdigit():
return tail
# If torch is already imported, ask it.
torch_mod = sys.modules.get("torch")
if torch_mod is not None:
ver = getattr(getattr(torch_mod, "version", None), "cuda", None)
if isinstance(ver, str) and ver:
return ver
return None
def _detect_package_version(name: str) -> Optional[str]:
"""Read installed-package version via ``importlib.metadata``.
Returns ``None`` for either "not installed" (``PackageNotFoundError``)
or "metadata parse failed" (``OSError`` / ``ValueError``). All other
exceptions propagate to surface real bugs (matches v0.33.0 #47 /
v0.40.3 #33 narrow-except policy).
"""
try:
from importlib.metadata import PackageNotFoundError, version
except ImportError: # pragma: no cover — py < 3.8
return None
try:
return version(name)
except PackageNotFoundError:
return None
except (OSError, ValueError): # pragma: no cover — corrupt metadata
return None
def snapshot_env() -> EnvLock:
"""Capture the current Python/CUDA/package versions as an ``EnvLock``."""
import soup_cli
py_v = ".".join(str(x) for x in sys.version_info[:3])
plat = f"{_platform.system().lower()}-{_platform.machine().lower()}"
cuda_v = _detect_cuda_version()
entries: list[EnvEntry] = []
for name in TRACKED_PACKAGES:
ver = _detect_package_version(name)
if ver is None:
continue
try:
entries.append(EnvEntry(name=name, version=ver, source="pip"))
except (TypeError, ValueError):
# Skip malformed version strings.
continue
return EnvLock(
soup_version=soup_cli.__version__,
python_version=py_v,
platform=plat,
cuda_version=cuda_v,
entries=tuple(entries),
created_at=_dt.datetime.now(_dt.timezone.utc).isoformat(),
)
def _entry_to_dict(e: EnvEntry) -> dict:
return {"name": e.name, "version": e.version, "source": e.source}
def _entry_from_dict(d: dict) -> EnvEntry:
if not isinstance(d, dict):
raise ValueError("entry must be a dict")
return EnvEntry(
name=str(d.get("name", "")),
version=str(d.get("version", "")),
source=str(d.get("source", "unknown")),
)
def write_lock(lock: EnvLock, path: str) -> None:
"""Atomically write an ``EnvLock`` to JSON under cwd containment."""
if not isinstance(lock, EnvLock):
raise TypeError(f"lock must be EnvLock, got {type(lock).__name__}")
payload = {
"schema_version": "1",
"soup_version": lock.soup_version,
"python_version": lock.python_version,
"platform": lock.platform,
"cuda_version": lock.cuda_version,
"entries": [_entry_to_dict(e) for e in lock.entries],
"created_at": lock.created_at,
}
text = json.dumps(payload, indent=2, ensure_ascii=False, allow_nan=False)
atomic_write_text(text, path, prefix=".envlock.", field="env lock")
def read_lock(path: str) -> EnvLock:
"""Read a previously-written ``EnvLock`` JSON.
Containment + symlink rejection BEFORE existence probe so a crafted
path cannot distinguish "outside cwd" from "missing" (matches v0.55.0
/ v0.62.0 ordering policy).
"""
import stat as _stat
if not isinstance(path, str):
raise TypeError(f"path must be str, got {type(path).__name__}")
if "\x00" in path:
raise ValueError("lock path must not contain null bytes")
if not is_under_cwd(path):
raise ValueError(f"lock {path!r} is outside cwd")
if os.path.lexists(path):
st = os.lstat(path)
if _stat.S_ISLNK(st.st_mode):
raise ValueError("lock path must not be a symlink")
if not os.path.isfile(path):
raise FileNotFoundError(path)
with open(path, encoding="utf-8") as fh:
payload = json.load(fh)
if not isinstance(payload, dict):
raise ValueError("lock root must be a dict")
entries_raw = payload.get("entries", [])
if not isinstance(entries_raw, list):
raise ValueError("entries must be a list")
entries = tuple(_entry_from_dict(e) for e in entries_raw)
return EnvLock(
soup_version=str(payload.get("soup_version", "")),
python_version=str(payload.get("python_version", "")),
platform=str(payload.get("platform", "unknown")),
cuda_version=payload.get("cuda_version"),
entries=entries,
created_at=str(payload.get("created_at", "")),
)
def check_abi_compat(a: EnvLock, b: EnvLock) -> AbiCheck:
"""Compare two EnvLocks; flag ABI-sensitive drifts.
Drift sources (in order of impact):
1. Python minor version change.
2. Platform string change.
3. CUDA version change.
4. Any TRACKED_PACKAGES version change.
Returns ``AbiCheck(ok=True)`` iff *zero* drift sources fire.
"""
if not isinstance(a, EnvLock):
raise TypeError(f"a must be EnvLock, got {type(a).__name__}")
if not isinstance(b, EnvLock):
raise TypeError(f"b must be EnvLock, got {type(b).__name__}")
changes: list[str] = []
# Python minor: 3.10.x vs 3.11.x is ABI-different
a_py = ".".join(a.python_version.split(".")[:2])
b_py = ".".join(b.python_version.split(".")[:2])
if a_py != b_py:
changes.append(f"python: {a.python_version} -> {b.python_version}")
if a.platform != b.platform:
changes.append(f"platform: {a.platform} -> {b.platform}")
if (a.cuda_version or None) != (b.cuda_version or None):
changes.append(f"cuda: {a.cuda_version} -> {b.cuda_version}")
by_a = {e.name.lower(): e.version for e in a.entries}
by_b = {e.name.lower(): e.version for e in b.entries}
for name in sorted(set(by_a) | set(by_b)):
va = by_a.get(name)
vb = by_b.get(name)
if va != vb:
changes.append(f"{name}: {va} -> {vb}")
return AbiCheck(
ok=not changes,
drift_count=len(changes),
changes=tuple(changes),
)
__all__ = [
"AbiCheck",
"DEFAULT_LOCK_FILE",
"EnvEntry",
"EnvLock",
"TRACKED_PACKAGES",
"check_abi_compat",
"read_lock",
"snapshot_env",
"write_lock",
]

350
soup_cli/utils/hardware_fit.py Normal file
View File

@ -0,0 +1,350 @@
"""Hardware-fit calculator: static analytical predictor of peak VRAM.
v0.40.3 #64 added a live CUDA OOM probe; that's correct but slow (one
real forward+backward per candidate). v0.64 adds a fast analytical
predictor that takes (params, seq_len, batch_size, optimizer, quant,
peft, gradient_checkpointing) and outputs a five-bucket VRAM breakdown.
Used by ``decide_hardware_fit`` to refuse a training launch when the
predicted peak (with ``VRAM_SAFETY_MARGIN=10%`` headroom) would not fit.
The math is intentionally conservative — operators can supply
``--allow-oom-attempt`` to bypass the gate if they want to try anyway
(opt-out, not opt-in, per the v0.64 Part D plan).
Public surface:
- ``VRAM_SAFETY_MARGIN = 0.10``.
- ``validate_seq_len(v)`` / ``validate_batch_size(v)``.
- ``HardwareFitInput`` frozen dataclass.
- ``VRAMBreakdown`` frozen dataclass + ``total_gb`` property.
- ``HardwareFitReport`` frozen dataclass.
- ``estimate_peak_vram_gb(inp)`` -> ``VRAMBreakdown``.
- ``decide_hardware_fit(inp, *, available_vram_gb)`` -> ``HardwareFitReport``.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
VRAM_SAFETY_MARGIN = 0.10 # 10% headroom
_MIN_SEQ_LEN = 64
_MAX_SEQ_LEN = 1_048_576
_MIN_BATCH = 1
_MAX_BATCH = 1024
_MAX_PARAMS_B = 1000.0
# Closed allowlists — schema-shape rejection, no surprises.
_VALID_QUANT = frozenset({
"none", "4bit", "8bit", "fp8", "gptq", "awq", "aqlm", "eetq", "mxfp4",
})
_VALID_PEFT = frozenset({"full", "lora", "dora", "qlora"})
_VALID_OPTIMIZERS = frozenset({
"adamw_torch", "adamw_torch_fused", "adafactor", "sgd",
"adamw_bnb_8bit", "paged_adamw_8bit", "lion_8bit",
"lomo", "adalomo", "schedule_free_adamw",
})
def validate_seq_len(value: object) -> int:
if isinstance(value, bool):
raise TypeError("seq_len must be int, not bool")
if not isinstance(value, int):
raise TypeError(f"seq_len must be int, got {type(value).__name__}")
if not (_MIN_SEQ_LEN <= value <= _MAX_SEQ_LEN):
raise ValueError(
f"seq_len must be in [{_MIN_SEQ_LEN}, {_MAX_SEQ_LEN}], got {value}"
)
return value
def validate_batch_size(value: object) -> int:
if isinstance(value, bool):
raise TypeError("batch_size must be int, not bool")
if not isinstance(value, int):
raise TypeError(f"batch_size must be int, got {type(value).__name__}")
if not (_MIN_BATCH <= value <= _MAX_BATCH):
raise ValueError(
f"batch_size must be in [{_MIN_BATCH}, {_MAX_BATCH}], got {value}"
)
return value
@dataclass(frozen=True)
class HardwareFitInput:
"""Inputs needed to predict peak VRAM."""
params_b: float
seq_len: int
batch_size: int
optimizer: str
quant: str
peft: str
gradient_checkpointing: bool
def __post_init__(self) -> None:
if isinstance(self.params_b, bool):
raise TypeError("params_b must be a number, not bool")
if not isinstance(self.params_b, (int, float)):
raise TypeError(
f"params_b must be a number, got {type(self.params_b).__name__}"
)
if not math.isfinite(float(self.params_b)):
raise ValueError("params_b must be finite")
if self.params_b <= 0 or self.params_b > _MAX_PARAMS_B:
raise ValueError(
f"params_b must be in (0, {_MAX_PARAMS_B}], got {self.params_b}"
)
validate_seq_len(self.seq_len)
validate_batch_size(self.batch_size)
if not isinstance(self.optimizer, str):
raise TypeError("optimizer must be str")
if self.optimizer not in _VALID_OPTIMIZERS:
raise ValueError(
f"unknown optimizer {self.optimizer!r}; "
f"known: {', '.join(sorted(_VALID_OPTIMIZERS))}"
)
if not isinstance(self.quant, str):
raise TypeError("quant must be str")
if self.quant not in _VALID_QUANT:
raise ValueError(
f"unknown quant {self.quant!r}; "
f"known: {', '.join(sorted(_VALID_QUANT))}"
)
if not isinstance(self.peft, str):
raise TypeError("peft must be str")
if self.peft not in _VALID_PEFT:
raise ValueError(
f"unknown peft {self.peft!r}; "
f"known: {', '.join(sorted(_VALID_PEFT))}"
)
if not isinstance(self.gradient_checkpointing, bool):
raise TypeError("gradient_checkpointing must be bool")
@dataclass(frozen=True)
class VRAMBreakdown:
"""Per-class peak VRAM in GB."""
weights_gb: float
optimizer_gb: float
gradients_gb: float
activations_gb: float
overhead_gb: float
def __post_init__(self) -> None:
for fld in (
"weights_gb", "optimizer_gb", "gradients_gb",
"activations_gb", "overhead_gb",
):
val = getattr(self, fld)
if isinstance(val, bool):
raise TypeError(f"{fld} must be a number, not bool")
if not isinstance(val, (int, float)):
raise TypeError(f"{fld} must be a number")
if not math.isfinite(float(val)):
raise ValueError(f"{fld} must be finite")
if val < 0:
raise ValueError(f"{fld} must not be negative, got {val}")
@property
def total_gb(self) -> float:
return float(
self.weights_gb
+ self.optimizer_gb
+ self.gradients_gb
+ self.activations_gb
+ self.overhead_gb
)
@dataclass(frozen=True)
class HardwareFitReport:
"""Outcome of a hardware-fit decision."""
ok: bool
peak_vram_gb: float
required_with_margin_gb: float
available_vram_gb: float
breakdown: VRAMBreakdown
reason: str
def __post_init__(self) -> None:
if not isinstance(self.ok, bool):
raise TypeError("ok must be bool")
for fld in ("peak_vram_gb", "required_with_margin_gb", "available_vram_gb"):
val = getattr(self, fld)
if isinstance(val, bool):
raise TypeError(f"{fld} must be a number, not bool")
if not isinstance(val, (int, float)):
raise TypeError(f"{fld} must be a number")
if not math.isfinite(float(val)):
raise ValueError(f"{fld} must be finite")
if val < 0:
raise ValueError(f"{fld} must not be negative")
if not isinstance(self.breakdown, VRAMBreakdown):
raise TypeError("breakdown must be VRAMBreakdown")
if not isinstance(self.reason, str):
raise TypeError("reason must be str")
# Bytes-per-param multiplier by quant scheme. Approximate; the deep
# truth lives in upstream BNB / GPTQ / AWQ docs.
_BYTES_PER_PARAM_BY_QUANT = {
"none": 2.0, # bf16/fp16
"4bit": 0.5,
"8bit": 1.0,
"fp8": 1.0,
"gptq": 0.55,
"awq": 0.55,
"aqlm": 0.5,
"eetq": 1.0,
"mxfp4": 0.55,
}
# Optimiser-state bytes per trainable parameter. AdamW = 8 (m + v fp32),
# 8-bit AdamW = 2, Adafactor = 4, SGD = 0 (momentum-less reference).
_OPTIM_BYTES_PER_PARAM = {
"adamw_torch": 8.0,
"adamw_torch_fused": 8.0,
"adafactor": 4.0,
"sgd": 0.0,
"adamw_bnb_8bit": 2.0,
"paged_adamw_8bit": 2.0,
"lion_8bit": 2.0,
"lomo": 0.0,
"adalomo": 2.0,
"schedule_free_adamw": 8.0,
}
def _trainable_param_fraction(peft: str) -> float:
"""LoRA / DoRA / QLoRA train ~1% of parameters. Full = 100%."""
if peft == "full":
return 1.0
if peft in ("lora", "qlora", "dora"):
return 0.01
return 1.0 # defensive default; schema gate already rejects unknown
def _activation_bytes(seq_len: int, batch_size: int, params_b: float) -> float:
"""Per-batch activation memory.
Very approximate: scales with seq_len × batch_size × hidden_size.
We treat hidden_size as ~ 64 * sqrt(params_b * 1e9 / 1024) — close
enough for an order-of-magnitude estimate at the 1B-70B band.
Result is bounded by ``_MAX_ACTIVATIONS_BYTES`` (~1 EiB) so the
downstream ``VRAMBreakdown`` validator never sees a +Inf product
even at the schema max (seq=1M × batch=1024).
"""
hidden = 64.0 * math.sqrt(max(params_b * 1e9 / 1024.0, 1.0))
raw = float(seq_len) * float(batch_size) * hidden * 4.0
# Defensive: clamp absurd products so the math stays finite.
if not math.isfinite(raw) or raw > _MAX_ACTIVATIONS_BYTES:
return _MAX_ACTIVATIONS_BYTES
return raw
_MAX_ACTIVATIONS_BYTES = 1e18 # 1 EB sanity cap; far above any real GPU
def estimate_peak_vram_gb(inp: HardwareFitInput) -> VRAMBreakdown:
"""Static analytical VRAM predictor. Returns a per-class breakdown."""
if not isinstance(inp, HardwareFitInput):
raise TypeError(
f"inp must be HardwareFitInput, got {type(inp).__name__}"
)
params = float(inp.params_b) * 1e9 # absolute count
bytes_per = _BYTES_PER_PARAM_BY_QUANT.get(inp.quant, 2.0)
weights_b = params * bytes_per
trainable_frac = _trainable_param_fraction(inp.peft)
trainable_params = params * trainable_frac
optim_b = trainable_params * _OPTIM_BYTES_PER_PARAM.get(inp.optimizer, 8.0)
# Gradients are fp32 of trainable params (4 bytes/param) under
# mixed-precision; under "none" quant we still keep fp32 grads.
gradients_b = trainable_params * 4.0
activations_b = _activation_bytes(inp.seq_len, inp.batch_size, inp.params_b)
if inp.gradient_checkpointing:
# Gradient checkpointing roughly halves activation memory.
activations_b *= 0.5
# Constant overhead: kernels / autograd graph / NCCL buffers / etc.
# Scales weakly with params.
overhead_b = 0.5e9 + 0.05 * weights_b
return VRAMBreakdown(
weights_gb=weights_b / 1e9,
optimizer_gb=optim_b / 1e9,
gradients_gb=gradients_b / 1e9,
activations_gb=activations_b / 1e9,
overhead_gb=overhead_b / 1e9,
)
def decide_hardware_fit(
inp: HardwareFitInput,
*,
available_vram_gb: float,
) -> HardwareFitReport:
"""Decide whether the planned run fits under ``available_vram_gb``.
Applies ``VRAM_SAFETY_MARGIN=10%`` headroom on top of the analytical
estimate. Refuse iff ``predicted * 1.1 > available``.
"""
if isinstance(available_vram_gb, bool):
raise TypeError("available_vram_gb must be a number, not bool")
if not isinstance(available_vram_gb, (int, float)):
raise TypeError(
f"available_vram_gb must be a number, got {type(available_vram_gb).__name__}"
)
if not math.isfinite(float(available_vram_gb)):
raise ValueError("available_vram_gb must be finite")
if available_vram_gb < 0:
raise ValueError(
f"available_vram_gb must be >= 0, got {available_vram_gb}"
)
breakdown = estimate_peak_vram_gb(inp)
peak = breakdown.total_gb
required = peak * (1.0 + VRAM_SAFETY_MARGIN)
if required <= available_vram_gb:
return HardwareFitReport(
ok=True,
peak_vram_gb=peak,
required_with_margin_gb=required,
available_vram_gb=float(available_vram_gb),
breakdown=breakdown,
reason=(
f"fits: peak {peak:.2f} GB + {VRAM_SAFETY_MARGIN:.0%} margin "
f"<= {available_vram_gb:.2f} GB available"
),
)
return HardwareFitReport(
ok=False,
peak_vram_gb=peak,
required_with_margin_gb=required,
available_vram_gb=float(available_vram_gb),
breakdown=breakdown,
reason=(
f"OOM risk: required {required:.2f} GB exceeds "
f"available {available_vram_gb:.2f} GB. "
f"Try --batch-size {max(1, inp.batch_size // 2)} or "
"--quantization 4bit or --gradient-checkpointing auto."
),
)
__all__ = [
"VRAM_SAFETY_MARGIN",
"HardwareFitInput",
"HardwareFitReport",
"VRAMBreakdown",
"decide_hardware_fit",
"estimate_peak_vram_gb",
"validate_batch_size",
"validate_seq_len",
]

328
soup_cli/utils/license_advisor.py Normal file
View File

@ -0,0 +1,328 @@
"""License advisor (v0.64.0 Part F).
Given a target deployment (``b2c``, ``defense``, ``embedded``),
recommend a license-clean base model + flag downstream risk for a
specific license id. Composes with v0.60 Part E
``license_matrix.check_license_compat`` for the merge-time gate.
The matrix here is deliberately *coarse-grained*: it groups licenses
into ``recommended`` / ``forbidden`` for each deploy target, and the
``flag_downstream_risk`` helper applies a per-license risk heuristic
(e.g. Llama community license + > 700M MAU = block, per Meta's
acceptable-use policy).
Live model-card scraping for license inference lands in v0.64.1;
v0.64.0 takes the license id from the operator (matches the v0.60
``--license <id>`` convention).
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Tuple
from soup_cli.utils.license_matrix import KNOWN_LICENSES, LICENSE_KINDS
DEPLOY_TARGETS = frozenset({"b2c", "defense", "embedded"})
_MAX_TARGET_LEN = 32
_MAX_LICENSE_LEN = 128
_VALID_SEVERITIES = frozenset({"ok", "warn", "block"})
# Llama community license: Meta requires a separate licence for products
# with > 700M monthly active users. We surface this as a downstream-risk
# block in B2C deployments.
_LLAMA_COMMUNITY_MAU_CAP = 700_000_000
# Tight allowlist of Llama-family community license ids. Defends against
# a future "llama-permissive" id wrongly tripping the MAU gate.
_LLAMA_COMMUNITY_LICENSES = frozenset({
"llama-2",
"llama-3",
"llama-3.1",
"llama-3.2",
"llama-3.3",
"llama-community",
})
def validate_deploy_target(value: object) -> str:
"""Normalise + validate a deploy target name."""
if isinstance(value, bool):
raise TypeError("target must be str, not bool")
if not isinstance(value, str):
raise TypeError(f"target must be str, got {type(value).__name__}")
if not value:
raise ValueError("target must be non-empty")
if "\x00" in value:
raise ValueError("target must not contain null bytes")
if len(value) > _MAX_TARGET_LEN:
raise ValueError(f"target too long (> {_MAX_TARGET_LEN} chars)")
normalised = value.lower().strip()
if normalised not in DEPLOY_TARGETS:
allowed = ", ".join(sorted(DEPLOY_TARGETS))
raise ValueError(f"unknown target {value!r}; known: {allowed}")
return normalised
@dataclass(frozen=True)
class LicenseRecommendation:
"""Output of ``advise_license_for_target``."""
target: str
recommended_licenses: Tuple[str, ...]
forbidden_licenses: Tuple[str, ...]
reason: str
def __post_init__(self) -> None:
if not isinstance(self.target, str):
raise TypeError("target must be str")
if not isinstance(self.recommended_licenses, tuple):
raise TypeError("recommended_licenses must be a tuple of str")
if not isinstance(self.forbidden_licenses, tuple):
raise TypeError("forbidden_licenses must be a tuple of str")
if not isinstance(self.reason, str):
raise TypeError("reason must be str")
# Permissive baseline used as the b2c "recommended" set. Operators
# almost always want these.
_PERMISSIVE_RECOMMENDED: Tuple[str, ...] = (
"apache-2.0",
"mit",
"bsd-3-clause",
"bsd-2-clause",
"isc",
"unlicense",
)
# Non-commercial licenses — always forbidden for B2C.
_NC_LICENSES: Tuple[str, ...] = (
"cc-by-nc-4.0",
"cc-by-nc-sa-4.0",
"cc-by-nc-nd-4.0",
)
# Restricted-use model licenses (Llama / Gemma / Qwen / Mistral
# community licenses) — forbidden for defense per acceptable-use clauses.
_RESTRICTED_USE: Tuple[str, ...] = (
"llama-2",
"llama-3",
"llama-3.1",
"llama-3.2",
"llama-3.3",
"llama-community",
"gemma",
"qwen-research",
"qwen-license",
"mistral-research",
)
# Strong copyleft — forbidden for embedded (driver / firmware tree
# incompatible with GPL/AGPL closed-source distribution).
_STRONG_COPYLEFT: Tuple[str, ...] = (
"gpl-2.0",
"gpl-3.0",
"agpl-3.0",
)
def advise_license_for_target(target: object) -> LicenseRecommendation:
"""Return the per-target ``LicenseRecommendation``."""
normalised = validate_deploy_target(target)
if normalised == "b2c":
return LicenseRecommendation(
target=normalised,
recommended_licenses=_PERMISSIVE_RECOMMENDED,
forbidden_licenses=_NC_LICENSES,
reason=(
"Consumer B2C: prefer broadly-permissive licenses; "
"non-commercial licenses are categorically forbidden."
),
)
if normalised == "defense":
# Defense forbids restricted-use clauses on top of NC.
forbidden = _NC_LICENSES + _RESTRICTED_USE
return LicenseRecommendation(
target=normalised,
recommended_licenses=_PERMISSIVE_RECOMMENDED,
forbidden_licenses=forbidden,
reason=(
"Defense: restricted-use community licenses (Llama / "
"Gemma / Qwen / Mistral community) carry "
"acceptable-use clauses that conflict with defense "
"applications. Pick a permissive license."
),
)
if normalised == "embedded":
# Embedded forbids strong copyleft + NC.
forbidden = _NC_LICENSES + _STRONG_COPYLEFT
return LicenseRecommendation(
target=normalised,
recommended_licenses=_PERMISSIVE_RECOMMENDED,
forbidden_licenses=forbidden,
reason=(
"Embedded: strong copyleft (GPL/AGPL) is incompatible "
"with closed-source firmware distribution; "
"non-commercial licenses are also forbidden."
),
)
# Unreachable thanks to validate; defensive default.
raise ValueError(f"unhandled target {normalised!r}")
@dataclass(frozen=True)
class DownstreamRisk:
"""Outcome of a per-license risk check."""
ok: bool
severity: str # "ok" / "warn" / "block"
reason: str
def __post_init__(self) -> None:
if not isinstance(self.ok, bool):
raise TypeError("ok must be bool")
if not isinstance(self.severity, str):
raise TypeError("severity must be str")
if self.severity not in _VALID_SEVERITIES:
raise ValueError(
f"severity must be one of {{ok, warn, block}}, got {self.severity!r}"
)
if not isinstance(self.reason, str):
raise TypeError("reason must be str")
def _validate_license_id(value: object) -> str:
if isinstance(value, bool):
raise TypeError("license_id must be str, not bool")
if not isinstance(value, str):
raise TypeError(f"license_id must be str, got {type(value).__name__}")
if not value:
raise ValueError("license_id must be non-empty")
if "\x00" in value:
raise ValueError("license_id must not contain null bytes")
if len(value) > _MAX_LICENSE_LEN:
raise ValueError(f"license_id too long (> {_MAX_LICENSE_LEN} chars)")
return value.lower().strip()
_MAX_MAU = 10_000_000_000_000 # 10 trillion — more humans than ever existed
def _validate_mau(value: object) -> int:
if isinstance(value, bool):
raise TypeError("monthly_active_users must be int, not bool")
if not isinstance(value, int):
raise TypeError(
f"monthly_active_users must be int, got {type(value).__name__}"
)
if value < 0:
raise ValueError(
f"monthly_active_users must be >= 0, got {value}"
)
if value > _MAX_MAU:
raise ValueError(
f"monthly_active_users too large (> {_MAX_MAU}), got {value}"
)
return value
def flag_downstream_risk(
*,
license_id: object,
target: object,
monthly_active_users: object,
) -> DownstreamRisk:
"""Per-license risk check for a target deployment.
Decision matrix:
- Unknown license -> ``warn`` (operator should review).
- Non-commercial license on B2C/defense/embedded -> ``block``.
- Llama-family community license on B2C with MAU > 700M -> ``block``.
- Llama-family on defense -> ``block``.
- Strong copyleft on embedded -> ``block``.
- Otherwise: ``ok``.
"""
lic = _validate_license_id(license_id)
tgt = validate_deploy_target(target)
mau = _validate_mau(monthly_active_users)
if lic not in KNOWN_LICENSES:
return DownstreamRisk(
ok=False,
severity="warn",
reason=(
f"unknown license {lic!r}; "
"verify with legal before shipping."
),
)
kind = LICENSE_KINDS.get(lic)
if kind == "non-commercial":
return DownstreamRisk(
ok=False,
severity="block",
reason=(
f"non-commercial license {lic!r} categorically forbidden "
f"on target {tgt!r}."
),
)
if kind == "restricted-use":
if tgt == "defense":
return DownstreamRisk(
ok=False,
severity="block",
reason=(
f"restricted-use license {lic!r} carries an "
"acceptable-use clause incompatible with defense."
),
)
# Llama community license: > 700M MAU = block per Meta AUP.
# Tight allowlist — defends against a future non-community
# "llama-permissive" id falsely tripping the MAU gate.
if lic in _LLAMA_COMMUNITY_LICENSES and mau > _LLAMA_COMMUNITY_MAU_CAP:
return DownstreamRisk(
ok=False,
severity="block",
reason=(
f"Llama community license requires a separate licence from "
f"Meta for MAU > {_LLAMA_COMMUNITY_MAU_CAP:,} "
f"(your MAU: {mau:,})."
),
)
# Restricted-use on B2C with low MAU is a warn, not a block.
return DownstreamRisk(
ok=False,
severity="warn",
reason=(
f"restricted-use license {lic!r}: review acceptable-use "
"clauses before shipping. "
"Per Meta AUP, MAU > 700M needs a separate licence."
),
)
if kind == "strong-copyleft" and tgt == "embedded":
return DownstreamRisk(
ok=False,
severity="block",
reason=(
f"strong copyleft {lic!r} is incompatible with "
"closed-source embedded firmware."
),
)
return DownstreamRisk(
ok=True,
severity="ok",
reason=f"license {lic!r} ({kind}) is OK for target {tgt!r}.",
)
__all__ = [
"DEPLOY_TARGETS",
"DownstreamRisk",
"LicenseRecommendation",
"advise_license_for_target",
"flag_downstream_risk",
"validate_deploy_target",
]

390
soup_cli/utils/terraform_plan.py Normal file
View File

@ -0,0 +1,390 @@
"""`soup plan` / `soup apply` — Terraform-shape lock-and-execute for FT.
A training run is a one-shot infrastructure-shaped operation: spot
price, expected cost, base SHA, dataset SHA, peak VRAM. v0.64 borrows
Terraform's plan-apply split so operators can review what they're
about to spend before committing.
Workflow:
1. ``soup plan --config soup.yaml`` writes ``soup.tfstate`` with a
``TrainingPlan`` summarising the run (config SHA, dataset SHA,
estimated cost, ETA, peak VRAM, spot price).
2. ``soup apply --config soup.yaml`` re-builds the plan from the
current YAML, compares against the state file, and **refuses** if
the plan drifted. Operators see "config drifted: epochs 1 -> 99"
instead of silently spending another $0.50.
3. ``--dry-run`` exits 0 after the drift check without actually
invoking the trainer.
The state file is a thin JSON envelope; the actual ``soup train`` is
still the run-driver. This is a *gate*, not a parallel trainer.
Public surface:
- ``TrainingPlan`` frozen dataclass (base / task / config_sha /
dataset_sha / cost / ETA / peak VRAM / spot price).
- ``TrainingState`` frozen dataclass (plan + applied flag + run_id +
applied_at timestamp).
- ``DriftReport`` frozen dataclass with the changed-fields tuple.
- ``compute_config_sha(config)`` -> 64-hex SHA-256 of canonical JSON.
- ``compute_dataset_sha(path)`` -> 64-hex SHA-256 of the JSONL bytes.
- ``build_plan(config)`` -> TrainingPlan.
- ``write_state(state, path)`` / ``read_state(path)`` -> atomic JSON.
- ``detect_drift(state, plan_now)`` -> DriftReport.
- ``DEFAULT_STATE_FILE = "soup.tfstate"``.
"""
from __future__ import annotations
import hashlib
import json
import math
import os
from dataclasses import dataclass, field
from typing import Any, Mapping, Optional, Tuple
from soup_cli.utils.paths import atomic_write_text, is_under_cwd
DEFAULT_STATE_FILE = "soup.tfstate"
_SHA_REGEX_LEN = 64
_MAX_BASE_LEN = 512
@dataclass(frozen=True)
class TrainingPlan:
"""Pre-flight summary of a planned `soup train` invocation."""
base: str
task: str
config_sha: str
dataset_sha: str
estimated_cost_usd: float
estimated_minutes: float
peak_vram_gb: float
spot_price_usd_per_hour: float
def __post_init__(self) -> None:
_check_non_empty_str(self.base, "base", max_len=_MAX_BASE_LEN)
_check_non_empty_str(self.task, "task", max_len=64)
_check_sha(self.config_sha, "config_sha")
_check_sha(self.dataset_sha, "dataset_sha")
_check_non_negative_finite(self.estimated_cost_usd, "estimated_cost_usd")
_check_non_negative_finite(self.estimated_minutes, "estimated_minutes")
_check_non_negative_finite(self.peak_vram_gb, "peak_vram_gb")
_check_non_negative_finite(self.spot_price_usd_per_hour, "spot_price_usd_per_hour")
def _check_non_empty_str(value: object, fld: str, *, max_len: int) -> None:
if isinstance(value, bool):
raise TypeError(f"{fld} must be str, not bool")
if not isinstance(value, str):
raise TypeError(f"{fld} must be str, got {type(value).__name__}")
if not value:
raise ValueError(f"{fld} must be non-empty")
if "\x00" in value:
raise ValueError(f"{fld} must not contain null bytes")
if len(value) > max_len:
raise ValueError(f"{fld} too long (> {max_len} chars)")
def _check_sha(value: object, fld: str) -> None:
if isinstance(value, bool):
raise TypeError(f"{fld} must be str, not bool")
if not isinstance(value, str):
raise TypeError(f"{fld} must be str, got {type(value).__name__}")
if len(value) != _SHA_REGEX_LEN:
raise ValueError(
f"{fld} sha must be {_SHA_REGEX_LEN} hex chars, "
f"got len={len(value)}"
)
try:
int(value, 16)
except ValueError as exc:
raise ValueError(f"{fld} sha must be 64-char hex, got {value!r}") from exc
def _check_non_negative_finite(value: object, fld: str) -> None:
if isinstance(value, bool):
raise TypeError(f"{fld} must be a number, not bool")
if not isinstance(value, (int, float)):
raise TypeError(f"{fld} must be a number, got {type(value).__name__}")
if not math.isfinite(float(value)):
raise ValueError(f"{fld} must be finite (no NaN / Inf)")
if value < 0:
raise ValueError(f"{fld} must not be negative, got {value}")
@dataclass(frozen=True)
class TrainingState:
"""Persisted plan + apply metadata. Written as ``soup.tfstate``."""
plan: TrainingPlan
applied: bool
applied_at: Optional[str]
run_id: Optional[str]
def __post_init__(self) -> None:
if not isinstance(self.plan, TrainingPlan):
raise TypeError(
f"plan must be TrainingPlan, got {type(self.plan).__name__}"
)
if not isinstance(self.applied, bool):
raise TypeError("applied must be bool")
if self.applied_at is not None and not isinstance(self.applied_at, str):
raise TypeError("applied_at must be str | None")
if self.run_id is not None and not isinstance(self.run_id, str):
raise TypeError("run_id must be str | None")
@dataclass(frozen=True)
class DriftReport:
"""Outcome of a drift comparison between state.plan and a fresh plan."""
has_drift: bool
changed_fields: Tuple[str, ...] = field(default_factory=tuple)
def __post_init__(self) -> None:
if not isinstance(self.has_drift, bool):
raise TypeError("has_drift must be bool")
if not isinstance(self.changed_fields, tuple):
raise TypeError("changed_fields must be a tuple of str")
for entry in self.changed_fields:
if not isinstance(entry, str):
raise TypeError("changed_fields entries must be str")
def compute_config_sha(config: Mapping[str, Any]) -> str:
"""Canonical SHA-256 of the config dict (insertion-order-independent).
Uses strict JSON serialisation (no ``default=`` fallback) so two
configs that differ in a non-serialisable field cannot silently
collide. Operators with custom-typed values must pre-canonicalise
via ``dict(yaml.safe_load(...))`` first.
"""
if not isinstance(config, Mapping):
raise TypeError(f"config must be a mapping, got {type(config).__name__}")
blob = json.dumps(
config, sort_keys=True, ensure_ascii=False, allow_nan=False
).encode("utf-8")
return hashlib.sha256(blob).hexdigest()
def compute_dataset_sha(path: object) -> str:
"""SHA-256 of dataset file bytes. Returns zero-hash on missing file.
Returning a constant for "missing" lets ``soup plan`` run before the
dataset exists (e.g. dataset built by an earlier `soup data` step).
Drift detection still surfaces the change once the file appears.
Security: cwd containment + symlink rejection BEFORE the open() so a
crafted ``soup.yaml`` with ``data.train: /etc/shadow`` cannot leak the
file contents into the SHA. An empty path returns zero-hash without
touching the filesystem.
"""
import stat as _stat
if not isinstance(path, str):
raise TypeError(f"dataset path must be str, got {type(path).__name__}")
if "\x00" in path:
raise ValueError("dataset path must not contain null bytes")
if not path:
return "0" * _SHA_REGEX_LEN
if not is_under_cwd(path):
# Out-of-cwd is treated as "no dataset captured" rather than a
# hard error so plan can still render — drift detection will
# surface this if the operator later moves the dataset in-tree.
return "0" * _SHA_REGEX_LEN
if os.path.lexists(path):
try:
st = os.lstat(path)
except OSError:
return "0" * _SHA_REGEX_LEN
if _stat.S_ISLNK(st.st_mode):
raise ValueError("dataset path must not be a symlink")
if not os.path.isfile(path):
return "0" * _SHA_REGEX_LEN
h = hashlib.sha256()
with open(path, "rb") as fh:
while True:
chunk = fh.read(65536)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
# Approximate cost model — pure heuristic so the plan is honest about
# being a pre-flight estimate. Live measurement is whatever the trainer
# emits at finish time (v0.34 cost path).
_DEFAULT_SPOT_PRICE = 0.30 # ~$/hr for a 24 GB consumer GPU
_DEFAULT_PEAK_VRAM = 8.0
def _estimate_runtime_minutes(config: Mapping[str, Any]) -> float:
training = config.get("training", {}) if isinstance(config, Mapping) else {}
epochs = float(training.get("epochs", 1) if isinstance(training, Mapping) else 1)
batch_size = float(training.get("batch_size", 4) if isinstance(training, Mapping) else 4)
# Soft heuristic: 5 minutes per epoch at batch_size=4 baseline.
base = 5.0 * epochs
if batch_size > 0:
base *= 4.0 / max(batch_size, 1.0)
return max(0.5, base)
def _estimate_cost(minutes: float, spot_price: float) -> float:
return max(0.0, (minutes / 60.0) * spot_price)
def build_plan(config: Mapping[str, Any]) -> TrainingPlan:
"""Render a TrainingPlan from a soup-config-shaped dict."""
if not isinstance(config, Mapping):
raise TypeError(f"config must be a mapping, got {type(config).__name__}")
base = config.get("base")
if not isinstance(base, str) or not base:
raise ValueError("config must contain a non-empty `base`")
task = config.get("task", "sft")
if not isinstance(task, str) or not task:
task = "sft"
data_cfg = config.get("data", {}) if isinstance(config.get("data"), Mapping) else {}
train_path = data_cfg.get("train", "") if isinstance(data_cfg, Mapping) else ""
if not isinstance(train_path, str):
train_path = ""
config_sha = compute_config_sha(config)
dataset_sha = compute_dataset_sha(train_path)
minutes = _estimate_runtime_minutes(config)
cost = _estimate_cost(minutes, _DEFAULT_SPOT_PRICE)
return TrainingPlan(
base=base,
task=task,
config_sha=config_sha,
dataset_sha=dataset_sha,
estimated_cost_usd=cost,
estimated_minutes=minutes,
peak_vram_gb=_DEFAULT_PEAK_VRAM,
spot_price_usd_per_hour=_DEFAULT_SPOT_PRICE,
)
def _plan_to_dict(plan: TrainingPlan) -> dict:
return {
"base": plan.base,
"task": plan.task,
"config_sha": plan.config_sha,
"dataset_sha": plan.dataset_sha,
"estimated_cost_usd": plan.estimated_cost_usd,
"estimated_minutes": plan.estimated_minutes,
"peak_vram_gb": plan.peak_vram_gb,
"spot_price_usd_per_hour": plan.spot_price_usd_per_hour,
}
def _plan_from_dict(d: Mapping[str, Any]) -> TrainingPlan:
return TrainingPlan(
base=str(d["base"]),
task=str(d["task"]),
config_sha=str(d["config_sha"]),
dataset_sha=str(d["dataset_sha"]),
estimated_cost_usd=float(d["estimated_cost_usd"]),
estimated_minutes=float(d["estimated_minutes"]),
peak_vram_gb=float(d["peak_vram_gb"]),
spot_price_usd_per_hour=float(d["spot_price_usd_per_hour"]),
)
def write_state(state: TrainingState, path: str) -> None:
"""Atomically write a ``TrainingState`` to JSON under cwd containment."""
if not isinstance(state, TrainingState):
raise TypeError(
f"state must be TrainingState, got {type(state).__name__}"
)
payload = {
"schema_version": "1",
"plan": _plan_to_dict(state.plan),
"applied": state.applied,
"applied_at": state.applied_at,
"run_id": state.run_id,
}
text = json.dumps(payload, indent=2, ensure_ascii=False, allow_nan=False)
atomic_write_text(text, path, prefix=".tfstate.", field="state file")
def read_state(path: str) -> TrainingState:
"""Read a previously-written ``TrainingState`` JSON.
Containment + symlink rejection BEFORE existence probe (mirrors
v0.55.0 / v0.62.0 ordering policy).
"""
import stat as _stat
if not isinstance(path, str):
raise TypeError(f"path must be str, got {type(path).__name__}")
if "\x00" in path:
raise ValueError("state path must not contain null bytes")
if not is_under_cwd(path):
raise ValueError(f"state file {path!r} is outside cwd")
if os.path.lexists(path):
st = os.lstat(path)
if _stat.S_ISLNK(st.st_mode):
raise ValueError("state path must not be a symlink")
if not os.path.isfile(path):
raise FileNotFoundError(path)
with open(path, encoding="utf-8") as fh:
payload = json.load(fh)
if not isinstance(payload, dict):
raise ValueError("state root must be a dict")
plan_raw = payload.get("plan")
if not isinstance(plan_raw, dict):
raise ValueError("state.plan must be a dict")
plan = _plan_from_dict(plan_raw)
applied_raw = payload.get("applied", False)
if not isinstance(applied_raw, bool):
raise ValueError(
f"state.applied must be bool, got {type(applied_raw).__name__}"
)
return TrainingState(
plan=plan,
applied=applied_raw,
applied_at=payload.get("applied_at"),
run_id=payload.get("run_id"),
)
def detect_drift(state: TrainingState, plan_now: TrainingPlan) -> DriftReport:
"""Compare ``state.plan`` against a freshly-built plan.
Returns a ``DriftReport`` listing changed fields. Empty tuple means no
drift. ``apply`` refuses to proceed when ``has_drift=True``.
"""
if not isinstance(state, TrainingState):
raise TypeError(
f"state must be TrainingState, got {type(state).__name__}"
)
if not isinstance(plan_now, TrainingPlan):
raise TypeError(
f"plan_now must be TrainingPlan, got {type(plan_now).__name__}"
)
changed: list[str] = []
if state.plan.base != plan_now.base:
changed.append("base")
if state.plan.task != plan_now.task:
changed.append("task")
if state.plan.config_sha != plan_now.config_sha:
changed.append("config_sha")
if state.plan.dataset_sha != plan_now.dataset_sha:
changed.append("dataset_sha")
return DriftReport(has_drift=bool(changed), changed_fields=tuple(changed))
__all__ = [
"DEFAULT_STATE_FILE",
"DriftReport",
"TrainingPlan",
"TrainingState",
"build_plan",
"compute_config_sha",
"compute_dataset_sha",
"detect_drift",
"read_state",
"write_state",
]

515
soup_cli/utils/tunability.py Normal file
View File

@ -0,0 +1,515 @@
"""`soup tunability` — probe-train 6-10 candidate bases, report Pareto frontier.
Before committing to a single base model, an operator can run a short LoRA
probe on each of N candidates against a held-out dataset slice. Reports
which candidates are on the (delta-from-base × cost × license) Pareto
frontier so the operator can pick the best fit instead of relying on
vendor catalogs / blog benchmarks.
Why blue-ocean: hosted vendors push their own catalogs (attach-rate
incentive); comparing across families is structurally costly for them.
Soup is local-first + spans every base.
Live wiring of the in-process LoRA probe is deferred to v0.64.1
(mirrors v0.27.0 MII / v0.50.0 GRPO Plus / v0.56.0 diagnose stub-then-live
pattern); v0.64.0 ships the schema, default catalogue, Pareto math,
report writer + the CLI surface so operators can plan a sweep and pipe
results through a custom probe callable today.
Public surface:
- ``CandidateBase`` frozen dataclass (name / repo_id / params_b / license_id).
- ``TunabilityResult`` frozen dataclass (per-candidate probe outcome).
- ``TunabilityReport`` frozen dataclass (results + Pareto frontier + meta).
- ``DEFAULT_CANDIDATES`` tuple of 8 sane defaults across families.
- ``validate_probe_steps(value)`` -> int in [10, 10_000].
- ``validate_holdout_size(value)`` -> int in [10, 100_000].
- ``score_candidate(*, base_loss, probe_loss)`` -> float delta.
- ``pareto_frontier(results)`` -> tuple of non-dominated TunabilityResult.
- ``run_tunability(...)`` -> TunabilityReport (orchestrator).
- ``write_report(report, path)`` / ``load_report(path)`` -> atomic JSON.
"""
from __future__ import annotations
import json
import math
import os
from collections.abc import Sequence
from dataclasses import dataclass
from typing import Callable, Optional, Tuple
from soup_cli.utils.paths import atomic_write_text, is_under_cwd
# Bounds — mirror v0.30.0 / v0.41.0 / v0.51.0 validator policy.
_MIN_PROBE_STEPS = 10
_MAX_PROBE_STEPS = 10_000
_MIN_HOLDOUT = 10
_MAX_HOLDOUT = 100_000
_MAX_NAME_LEN = 512
_MAX_REPO_ID_LEN = 512
_MAX_LICENSE_LEN = 128
_MAX_CANDIDATES = 32
@dataclass(frozen=True)
class CandidateBase:
"""A candidate base model for a tunability probe.
`params_b` is the parameter count in billions (e.g. ``0.6`` for 600M).
"""
name: str
repo_id: str
params_b: float
license_id: str
def __post_init__(self) -> None:
_check_str(self.name, "name", max_len=_MAX_NAME_LEN)
_check_str(self.repo_id, "repo_id", max_len=_MAX_REPO_ID_LEN)
_check_str(self.license_id, "license_id", max_len=_MAX_LICENSE_LEN)
if isinstance(self.params_b, bool):
raise TypeError("params_b must be a number, not bool")
if not isinstance(self.params_b, (int, float)):
raise TypeError(
f"params_b must be a number, got {type(self.params_b).__name__}"
)
if not math.isfinite(float(self.params_b)):
raise ValueError("params_b must be finite (no NaN / Inf)")
if self.params_b <= 0:
raise ValueError(f"params_b must be > 0, got {self.params_b}")
def _check_str(value: object, field: str, *, max_len: int) -> None:
if isinstance(value, bool):
raise TypeError(f"{field} must be str, not bool")
if not isinstance(value, str):
raise TypeError(f"{field} must be str, got {type(value).__name__}")
if not value:
raise ValueError(f"{field} must be non-empty")
if "\x00" in value:
raise ValueError(f"{field} must not contain null bytes")
if len(value) > max_len:
raise ValueError(f"{field} too long (> {max_len} chars)")
@dataclass(frozen=True)
class TunabilityResult:
"""Outcome of one candidate's probe run."""
candidate: CandidateBase
base_loss: float
probe_loss: float
delta: float
wall_clock_seconds: float
estimated_cost_usd: float
def __post_init__(self) -> None:
if not isinstance(self.candidate, CandidateBase):
raise TypeError(
f"candidate must be CandidateBase, got {type(self.candidate).__name__}"
)
for field, val in (
("base_loss", self.base_loss),
("probe_loss", self.probe_loss),
("delta", self.delta),
("wall_clock_seconds", self.wall_clock_seconds),
("estimated_cost_usd", self.estimated_cost_usd),
):
if isinstance(val, bool):
raise TypeError(f"{field} must be a number, not bool")
if not isinstance(val, (int, float)):
raise TypeError(
f"{field} must be a number, got {type(val).__name__}"
)
if not math.isfinite(float(val)):
raise ValueError(f"{field} must be finite (no NaN / Inf)")
if self.wall_clock_seconds < 0:
raise ValueError(
f"wall_clock_seconds must be >= 0, got {self.wall_clock_seconds}"
)
if self.estimated_cost_usd < 0:
raise ValueError(
f"estimated_cost_usd must be >= 0, got {self.estimated_cost_usd}"
)
@dataclass(frozen=True)
class TunabilityReport:
"""End-to-end report for a tunability sweep."""
results: Tuple[TunabilityResult, ...]
frontier: Tuple[TunabilityResult, ...]
probe_steps: int
holdout_size: int
def __post_init__(self) -> None:
if not isinstance(self.results, tuple):
raise TypeError("results must be a tuple of TunabilityResult")
if not isinstance(self.frontier, tuple):
raise TypeError("frontier must be a tuple of TunabilityResult")
for entry in self.results:
if not isinstance(entry, TunabilityResult):
raise TypeError("every result must be TunabilityResult")
for entry in self.frontier:
if not isinstance(entry, TunabilityResult):
raise TypeError("every frontier entry must be TunabilityResult")
# Default catalogue: cross-family small bases that fit in 12 GB VRAM with 4-bit
# LoRA. Operators can override via `--candidates <name1,name2,...>` (matched
# against `name`) or supply a custom YAML file.
DEFAULT_CANDIDATES: Tuple[CandidateBase, ...] = (
CandidateBase(
name="qwen3-0.6b",
repo_id="Qwen/Qwen3-0.6B",
params_b=0.6,
license_id="apache-2.0",
),
CandidateBase(
name="qwen3-1.7b",
repo_id="Qwen/Qwen3-1.7B",
params_b=1.7,
license_id="apache-2.0",
),
CandidateBase(
name="llama-3.2-1b",
repo_id="meta-llama/Llama-3.2-1B-Instruct",
params_b=1.0,
license_id="llama-3.2",
),
CandidateBase(
name="llama-3.2-3b",
repo_id="meta-llama/Llama-3.2-3B-Instruct",
params_b=3.0,
license_id="llama-3.2",
),
CandidateBase(
name="gemma-3-e2b",
repo_id="google/gemma-3-2b-it",
params_b=2.0,
license_id="gemma",
),
CandidateBase(
name="phi-4-mini",
repo_id="microsoft/Phi-4-mini-instruct",
params_b=3.8,
license_id="mit",
),
CandidateBase(
name="smollm3-1.7b",
repo_id="HuggingFaceTB/SmolLM3-1.7B-Instruct",
params_b=1.7,
license_id="apache-2.0",
),
CandidateBase(
name="qwen2.5-1.5b",
repo_id="Qwen/Qwen2.5-1.5B-Instruct",
params_b=1.5,
license_id="apache-2.0",
),
)
def validate_probe_steps(value: object) -> int:
"""Validate ``probe_steps`` in [10, 10_000]; reject bool / non-int."""
if isinstance(value, bool):
raise TypeError("probe_steps must be int, not bool")
if not isinstance(value, int):
raise TypeError(f"probe_steps must be int, got {type(value).__name__}")
if not (_MIN_PROBE_STEPS <= value <= _MAX_PROBE_STEPS):
raise ValueError(
f"probe_steps must be in [{_MIN_PROBE_STEPS}, {_MAX_PROBE_STEPS}], "
f"got {value}"
)
return value
def validate_holdout_size(value: object) -> int:
"""Validate ``holdout_size`` in [10, 100_000]; reject bool / non-int."""
if isinstance(value, bool):
raise TypeError("holdout_size must be int, not bool")
if not isinstance(value, int):
raise TypeError(f"holdout_size must be int, got {type(value).__name__}")
if not (_MIN_HOLDOUT <= value <= _MAX_HOLDOUT):
raise ValueError(
f"holdout_size must be in [{_MIN_HOLDOUT}, {_MAX_HOLDOUT}], "
f"got {value}"
)
return value
def score_candidate(*, base_loss: float, probe_loss: float) -> float:
"""Compute the candidate's delta (higher = bigger improvement).
delta = base_loss - probe_loss. Positive means the LoRA probe lowered
loss from the base; negative means it made things worse.
"""
for name, val in (("base_loss", base_loss), ("probe_loss", probe_loss)):
if isinstance(val, bool):
raise TypeError(f"{name} must be a number, not bool")
if not isinstance(val, (int, float)):
raise TypeError(f"{name} must be a number, got {type(val).__name__}")
if not math.isfinite(float(val)):
raise ValueError(f"{name} must be finite")
return float(base_loss) - float(probe_loss)
def _dominates(a: TunabilityResult, b: TunabilityResult) -> bool:
"""Return True iff ``a`` strictly dominates ``b`` on (delta, cost).
A dominates B iff: A.delta >= B.delta AND A.cost <= B.cost,
and at least one of the inequalities is strict.
"""
delta_ge = a.delta >= b.delta
cost_le = a.estimated_cost_usd <= b.estimated_cost_usd
strict = (a.delta > b.delta) or (a.estimated_cost_usd < b.estimated_cost_usd)
return delta_ge and cost_le and strict
def pareto_frontier(results: Sequence[TunabilityResult]) -> Tuple[TunabilityResult, ...]:
"""Return the Pareto-optimal subset of results.
Maximises ``delta``, minimises ``estimated_cost_usd``. A result survives
iff no other result strictly dominates it on both axes.
"""
if isinstance(results, str):
raise TypeError("results must be a sequence of TunabilityResult, not str")
try:
materialised = list(results)
except TypeError as exc:
raise TypeError("results must be iterable") from exc
for entry in materialised:
if not isinstance(entry, TunabilityResult):
raise TypeError(
f"every result must be TunabilityResult, got {type(entry).__name__}"
)
if not materialised:
return ()
frontier: list[TunabilityResult] = []
for i, candidate in enumerate(materialised):
dominated = False
for j, other in enumerate(materialised):
if i == j:
continue
if _dominates(other, candidate):
dominated = True
break
if not dominated:
frontier.append(candidate)
return tuple(frontier)
# Type alias for the probe callable. Live impl lands in v0.64.1.
ProbeFn = Callable[[CandidateBase, str], TunabilityResult]
def _default_probe(
candidate: CandidateBase,
dataset_path: str,
*,
probe_steps: int,
holdout_size: int,
) -> TunabilityResult:
"""Heuristic stand-in. Live LoRA probe lands in v0.64.1.
Returns deterministic, candidate-derived values so reports parse and
Pareto math exercises without a GPU. Operators wanting a real probe
inject a callable via ``probe_fn=...``.
"""
# Wall-clock + cost scale with params. Delta is a tiny constant so
# the report is honest about being a stub.
wall_clock = 60.0 + 30.0 * float(candidate.params_b)
cost = 0.001 * float(candidate.params_b) * float(probe_steps)
return TunabilityResult(
candidate=candidate,
base_loss=2.5,
probe_loss=2.5, # No change — heuristic stub
delta=0.0,
wall_clock_seconds=wall_clock,
estimated_cost_usd=cost,
)
def run_tunability(
*,
candidates: Sequence[CandidateBase],
dataset_path: str,
probe_steps: int = 100,
holdout_size: int = 64,
probe_fn: Optional[Callable[..., TunabilityResult]] = None,
) -> TunabilityReport:
"""Orchestrate probes across candidates and assemble a report.
The actual probing is delegated to ``probe_fn`` (signature:
``(cand, dataset_path, *, probe_steps, holdout_size) -> TunabilityResult``).
If ``probe_fn`` is None, falls back to the heuristic stub.
"""
if isinstance(candidates, (str, bytes)) or not isinstance(candidates, Sequence):
raise TypeError("candidates must be a sequence of CandidateBase")
if len(candidates) == 0:
raise ValueError("candidates must contain at least one entry")
if len(candidates) > _MAX_CANDIDATES:
raise ValueError(
f"too many candidates ({len(candidates)} > {_MAX_CANDIDATES})"
)
for entry in candidates:
if not isinstance(entry, CandidateBase):
raise TypeError(
f"every candidate must be CandidateBase, "
f"got {type(entry).__name__}"
)
steps = validate_probe_steps(probe_steps)
holdout = validate_holdout_size(holdout_size)
if not isinstance(dataset_path, str):
raise TypeError(
f"dataset_path must be str, got {type(dataset_path).__name__}"
)
if not dataset_path:
raise ValueError("dataset_path must be non-empty")
if "\x00" in dataset_path:
raise ValueError("dataset_path must not contain null bytes")
fn = probe_fn if probe_fn is not None else _default_probe
results: list[TunabilityResult] = []
for cand in candidates:
result = fn(cand, dataset_path, probe_steps=steps, holdout_size=holdout)
if not isinstance(result, TunabilityResult):
raise TypeError(
f"probe_fn must return TunabilityResult, "
f"got {type(result).__name__}"
)
results.append(result)
frontier = pareto_frontier(results)
return TunabilityReport(
results=tuple(results),
frontier=frontier,
probe_steps=steps,
holdout_size=holdout,
)
# ---------------------------------------------------------------------------
# Report I/O
# ---------------------------------------------------------------------------
def _result_to_dict(r: TunabilityResult) -> dict:
return {
"candidate": {
"name": r.candidate.name,
"repo_id": r.candidate.repo_id,
"params_b": r.candidate.params_b,
"license_id": r.candidate.license_id,
},
"base_loss": r.base_loss,
"probe_loss": r.probe_loss,
"delta": r.delta,
"wall_clock_seconds": r.wall_clock_seconds,
"estimated_cost_usd": r.estimated_cost_usd,
}
def _result_from_dict(d: dict) -> TunabilityResult:
if not isinstance(d, dict):
raise ValueError("result must be a dict")
cand_raw = d.get("candidate")
if not isinstance(cand_raw, dict):
raise ValueError("candidate must be a dict")
cand = CandidateBase(
name=cand_raw["name"],
repo_id=cand_raw["repo_id"],
params_b=float(cand_raw["params_b"]),
license_id=cand_raw["license_id"],
)
return TunabilityResult(
candidate=cand,
base_loss=float(d["base_loss"]),
probe_loss=float(d["probe_loss"]),
delta=float(d["delta"]),
wall_clock_seconds=float(d["wall_clock_seconds"]),
estimated_cost_usd=float(d["estimated_cost_usd"]),
)
def write_report(report: TunabilityReport, path: str) -> None:
"""Atomically write a TunabilityReport as JSON.
Path is cwd-containment-checked + symlink-rejected via the shared
``enforce_under_cwd_and_no_symlink`` helper (mirrors v0.59.0 /
v0.60.0 / v0.62.0 atomic-write policy).
"""
if not isinstance(report, TunabilityReport):
raise TypeError(
f"report must be TunabilityReport, got {type(report).__name__}"
)
if not isinstance(path, str):
raise TypeError(f"path must be str, got {type(path).__name__}")
if not path:
raise ValueError("path must be non-empty")
if "\x00" in path:
raise ValueError("path must not contain null bytes")
payload = {
"schema_version": "1",
"probe_steps": report.probe_steps,
"holdout_size": report.holdout_size,
"results": [_result_to_dict(r) for r in report.results],
"frontier": [_result_to_dict(r) for r in report.frontier],
}
text = json.dumps(payload, indent=2, ensure_ascii=False, allow_nan=False)
atomic_write_text(text, path, prefix=".tunability.", field="tunability output")
def load_report(path: str) -> TunabilityReport:
"""Load a TunabilityReport from JSON. Raises FileNotFoundError if missing.
Path is cwd-containment-checked + symlink-rejected BEFORE the
existence probe so a crafted path cannot leak file-existence
distinguishing "outside cwd" from "missing" (mirrors v0.55.0 /
v0.62.0 ordering policy).
"""
import stat as _stat
if not isinstance(path, str):
raise TypeError(f"path must be str, got {type(path).__name__}")
if "\x00" in path:
raise ValueError("path must not contain null bytes")
if not is_under_cwd(path):
raise ValueError(f"path {path!r} is outside cwd")
if os.path.lexists(path):
st = os.lstat(path)
if _stat.S_ISLNK(st.st_mode):
raise ValueError("report path must not be a symlink")
if not os.path.isfile(path):
raise FileNotFoundError(path)
with open(path, encoding="utf-8") as fh:
payload = json.load(fh)
if not isinstance(payload, dict):
raise ValueError("report root must be a dict")
results = tuple(_result_from_dict(r) for r in payload.get("results", []))
frontier = tuple(_result_from_dict(r) for r in payload.get("frontier", []))
return TunabilityReport(
results=results,
frontier=frontier,
probe_steps=int(payload.get("probe_steps", _MIN_PROBE_STEPS)),
holdout_size=int(payload.get("holdout_size", _MIN_HOLDOUT)),
)
__all__ = [
"CandidateBase",
"DEFAULT_CANDIDATES",
"ProbeFn",
"TunabilityReport",
"TunabilityResult",
"load_report",
"pareto_frontier",
"run_tunability",
"score_candidate",
"validate_holdout_size",
"validate_probe_steps",
"write_report",
]

465
tests/test_v0640_followups.py Normal file
View File

@ -0,0 +1,465 @@
"""v0.64.0 review fix follow-ups.
Covers wave-1 review-fix gaps:
- HIGH H2/H3/H4 — symlink rejection on the new READ paths
(plan/apply YAML loader, tunability load_report, env_lock read_lock,
terraform_plan read_state).
- HIGH H5 — compute_dataset_sha cwd-containment + symlink rejection.
- HIGH H6 — symlink rejection on dataset_path through `soup plan`.
- MEDIUM M1 — compute_config_sha rejects non-JSON-serialisable values.
- MEDIUM M3 — read_state rejects non-bool `applied`.
- MEDIUM M4 — _activation_bytes overflow-clamps at sanity cap.
- MEDIUM M6 — flag_downstream_risk tight Llama-family allowlist.
- MEDIUM M7 — Windows CUDA path parse (vN.M pattern).
- MEDIUM M8 — load_report containment-before-existence ordering.
- LOW L3 — source-grep regression: atomic_write_text usage in all 3 writers.
- LOW L4 — license-advisor MAU upper cap.
- LOW L5 — `Sequence` imported from `collections.abc` not `typing`.
"""
from __future__ import annotations
import os
import sys
from pathlib import Path
import pytest
# ---------------------------------------------------------------------------
# HIGH H2 — plan `_load_yaml_config` rejects symlink on YAML path
# ---------------------------------------------------------------------------
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_plan_yaml_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.commands.plan import _load_yaml_config
monkeypatch.chdir(tmp_path)
real = tmp_path / "real.yaml"
real.write_text("base: x\n")
link = tmp_path / "link.yaml"
os.symlink(real, link)
with pytest.raises(ValueError, match="symlink"):
_load_yaml_config(str(link))
def test_plan_yaml_outside_cwd_rejected(tmp_path, monkeypatch):
from soup_cli.commands.plan import _load_yaml_config
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "evil.yaml"
outside.write_text("base: x\n")
with pytest.raises(ValueError, match="cwd"):
_load_yaml_config(str(outside))
def test_plan_yaml_null_byte_rejected(tmp_path, monkeypatch):
from soup_cli.commands.plan import _load_yaml_config
monkeypatch.chdir(tmp_path)
with pytest.raises(ValueError, match="null"):
_load_yaml_config("config\x00.yaml")
# ---------------------------------------------------------------------------
# HIGH H3 — tunability load_report rejects symlink + null-byte
# ---------------------------------------------------------------------------
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_tunability_load_report_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.utils.tunability import load_report
monkeypatch.chdir(tmp_path)
real = tmp_path / "real.json"
real.write_text("{}")
link = tmp_path / "link.json"
os.symlink(real, link)
with pytest.raises(ValueError, match="symlink"):
load_report(str(link))
def test_tunability_load_report_null_byte_rejected(tmp_path, monkeypatch):
from soup_cli.utils.tunability import load_report
monkeypatch.chdir(tmp_path)
with pytest.raises(ValueError, match="null"):
load_report("report\x00.json")
def test_tunability_load_report_containment_before_existence(tmp_path, monkeypatch):
"""Outside-cwd path must raise ValueError(cwd), NOT FileNotFoundError.
Otherwise an attacker can distinguish "file exists out of cwd" from
"file missing" via the exception type — MEDIUM M8 review fix.
"""
from soup_cli.utils.tunability import load_report
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "definitely-missing.json"
with pytest.raises(ValueError, match="cwd"):
load_report(str(outside))
# ---------------------------------------------------------------------------
# HIGH H4 — env read_lock rejects symlink + null-byte + ordering
# ---------------------------------------------------------------------------
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_env_read_lock_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import read_lock
monkeypatch.chdir(tmp_path)
real = tmp_path / "real.lock"
real.write_text("{}")
link = tmp_path / "link.lock"
os.symlink(real, link)
with pytest.raises(ValueError, match="symlink"):
read_lock(str(link))
def test_env_read_lock_null_byte_rejected(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import read_lock
monkeypatch.chdir(tmp_path)
with pytest.raises(ValueError, match="null"):
read_lock("lock\x00.lock")
def test_env_read_lock_outside_cwd_before_existence(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import read_lock
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "missing.lock"
with pytest.raises(ValueError, match="cwd"):
read_lock(str(outside))
# ---------------------------------------------------------------------------
# HIGH H4 (terraform read_state) — symlink rejection
# ---------------------------------------------------------------------------
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_terraform_read_state_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import read_state
monkeypatch.chdir(tmp_path)
real = tmp_path / "real.tfstate"
real.write_text("{}")
link = tmp_path / "link.tfstate"
os.symlink(real, link)
with pytest.raises(ValueError, match="symlink"):
read_state(str(link))
def test_terraform_read_state_null_byte_rejected(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import read_state
monkeypatch.chdir(tmp_path)
with pytest.raises(ValueError, match="null"):
read_state("state\x00.tfstate")
def test_terraform_read_state_outside_cwd_before_existence(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import read_state
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "missing.tfstate"
with pytest.raises(ValueError, match="cwd"):
read_state(str(outside))
# ---------------------------------------------------------------------------
# HIGH H5 — compute_dataset_sha cwd containment + symlink rejection
# ---------------------------------------------------------------------------
def test_compute_dataset_sha_empty_returns_zero():
from soup_cli.utils.terraform_plan import compute_dataset_sha
assert compute_dataset_sha("") == "0" * 64
def test_compute_dataset_sha_outside_cwd_returns_zero(tmp_path, monkeypatch):
"""Out-of-cwd path silently returns zero-hash (no file read).
Defends against `data.train: /etc/shadow` smuggling file contents
into the SHA — H5 review fix.
"""
from soup_cli.utils.terraform_plan import compute_dataset_sha
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "evil.jsonl"
if outside.exists():
# Don't depend on the parent dir being writable.
return
sha = compute_dataset_sha(str(outside))
assert sha == "0" * 64
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_compute_dataset_sha_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import compute_dataset_sha
monkeypatch.chdir(tmp_path)
real = tmp_path / "real.jsonl"
real.write_text("{}\n")
link = tmp_path / "link.jsonl"
os.symlink(real, link)
with pytest.raises(ValueError, match="symlink"):
compute_dataset_sha(str(link))
def test_compute_dataset_sha_null_byte_rejected(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import compute_dataset_sha
monkeypatch.chdir(tmp_path)
with pytest.raises(ValueError, match="null"):
compute_dataset_sha("data\x00.jsonl")
def test_compute_dataset_sha_missing_returns_zero(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import compute_dataset_sha
monkeypatch.chdir(tmp_path)
sha = compute_dataset_sha("missing.jsonl")
assert sha == "0" * 64
# ---------------------------------------------------------------------------
# MEDIUM M1 — compute_config_sha rejects non-JSON-serialisable values
# ---------------------------------------------------------------------------
def test_compute_config_sha_strict_no_default_str():
"""A non-serialisable value must raise, not silently `str()` it."""
from soup_cli.utils.terraform_plan import compute_config_sha
# `set` is not JSON-serialisable
with pytest.raises(TypeError):
compute_config_sha({"a": {1, 2, 3}})
# ---------------------------------------------------------------------------
# MEDIUM M3 — read_state rejects non-bool `applied`
# ---------------------------------------------------------------------------
def test_read_state_rejects_non_bool_applied(tmp_path, monkeypatch):
import json
from soup_cli.utils.terraform_plan import read_state
monkeypatch.chdir(tmp_path)
bad = tmp_path / "bad.tfstate"
payload = {
"plan": {
"base": "m",
"task": "sft",
"config_sha": "a" * 64,
"dataset_sha": "b" * 64,
"estimated_cost_usd": 0.5,
"estimated_minutes": 10.0,
"peak_vram_gb": 8.0,
"spot_price_usd_per_hour": 0.30,
},
"applied": "yes", # str, not bool — must raise
"applied_at": None,
"run_id": None,
}
bad.write_text(json.dumps(payload))
with pytest.raises(ValueError, match="applied"):
read_state(str(bad))
# ---------------------------------------------------------------------------
# MEDIUM M4 — _activation_bytes overflow clamp
# ---------------------------------------------------------------------------
def test_activation_bytes_clamped_at_extreme():
"""Even at schema max seq_len * batch_size, result stays finite."""
import math
from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb
inp = HardwareFitInput(
params_b=1000.0,
seq_len=1_048_576,
batch_size=1024,
optimizer="adamw_torch",
quant="none",
peft="full",
gradient_checkpointing=False,
)
bd = estimate_peak_vram_gb(inp)
assert math.isfinite(bd.total_gb)
assert math.isfinite(bd.activations_gb)
# ---------------------------------------------------------------------------
# MEDIUM M6 — Llama community license allowlist (no .startswith match)
# ---------------------------------------------------------------------------
def test_flag_downstream_risk_unknown_llama_variant_does_not_trip_mau_gate():
"""A hypothetical `llama-permissive-2030` should NOT trigger MAU block.
Tight allowlist defends against future Meta licence id surprises.
"""
from soup_cli.utils.license_advisor import flag_downstream_risk
# `llama-permissive-2030` is not in `_LLAMA_COMMUNITY_LICENSES` AND
# is not in the known matrix, so it should fall through to the
# "unknown license -> warn" branch, not "block by MAU".
r = flag_downstream_risk(
license_id="llama-permissive-2030",
target="b2c",
monthly_active_users=800_000_000,
)
# severity might be warn (unknown) but NOT block-by-MAU.
assert "monthly active users" not in r.reason.lower() or r.severity != "block"
def test_flag_downstream_risk_llama_3_under_cap_is_warn_not_block():
from soup_cli.utils.license_advisor import flag_downstream_risk
r = flag_downstream_risk(
license_id="llama-3",
target="b2c",
monthly_active_users=100,
)
# Restricted-use on B2C with low MAU = warn, never block.
assert r.severity != "block"
# ---------------------------------------------------------------------------
# MEDIUM M7 — Windows CUDA path parse (vN.M)
# ---------------------------------------------------------------------------
def test_detect_cuda_version_windows_path(monkeypatch):
"""`v12.1` token at end of a Windows-style path is extracted."""
from soup_cli.utils import env_lock
monkeypatch.setenv("CUDA_HOME", r"C:\Program Files\NVIDIA\CUDA\v12.1")
# Ensure no other env var pre-empts
monkeypatch.delenv("CUDA_VERSION", raising=False)
# Sidestep torch — drop it from sys.modules
monkeypatch.setattr(env_lock.sys, "modules", {**env_lock.sys.modules, "torch": None})
version = env_lock._detect_cuda_version()
assert version is not None
assert "12.1" in version
def test_detect_cuda_version_posix_path(monkeypatch):
from soup_cli.utils import env_lock
monkeypatch.setenv("CUDA_HOME", "/usr/local/cuda-12.1")
monkeypatch.delenv("CUDA_VERSION", raising=False)
monkeypatch.setattr(env_lock.sys, "modules", {**env_lock.sys.modules, "torch": None})
version = env_lock._detect_cuda_version()
assert version is not None
assert "12.1" in version
def test_detect_cuda_version_none_when_no_env(monkeypatch):
from soup_cli.utils import env_lock
monkeypatch.delenv("CUDA_HOME", raising=False)
monkeypatch.delenv("CUDA_VERSION", raising=False)
monkeypatch.setattr(env_lock.sys, "modules", {**env_lock.sys.modules, "torch": None})
# Should silently return None — no env, no torch.
assert env_lock._detect_cuda_version() is None
# ---------------------------------------------------------------------------
# LOW L3 — source-grep regression: atomic_write_text in all 3 writers
# ---------------------------------------------------------------------------
def test_tunability_uses_atomic_write_text():
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "tunability.py"
text = src.read_text(encoding="utf-8")
assert "atomic_write_text" in text, "tunability.write_report must use shared helper"
def test_terraform_plan_uses_atomic_write_text():
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "terraform_plan.py"
text = src.read_text(encoding="utf-8")
assert "atomic_write_text" in text, "terraform_plan.write_state must use shared helper"
def test_env_lock_uses_atomic_write_text():
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "env_lock.py"
text = src.read_text(encoding="utf-8")
assert "atomic_write_text" in text, "env_lock.write_lock must use shared helper"
# ---------------------------------------------------------------------------
# LOW L4 — MAU upper cap
# ---------------------------------------------------------------------------
def test_mau_oversize_rejected():
from soup_cli.utils.license_advisor import flag_downstream_risk
with pytest.raises(ValueError, match="monthly_active_users"):
flag_downstream_risk(
license_id="apache-2.0",
target="b2c",
monthly_active_users=10**18,
)
# ---------------------------------------------------------------------------
# LOW L5 — Sequence imported from collections.abc, not typing
# ---------------------------------------------------------------------------
def test_tunability_sequence_imported_from_collections_abc():
"""`typing.Sequence` doesn't work with isinstance on every Py 3.9 build.
Use `collections.abc.Sequence` for runtime isinstance checks.
"""
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "tunability.py"
text = src.read_text(encoding="utf-8")
# Either explicit `from collections.abc import Sequence`, or NO
# mention of `typing.Sequence` in an isinstance context.
assert "from collections.abc import Sequence" in text, (
"tunability.py must import Sequence from collections.abc"
)
# ---------------------------------------------------------------------------
# Drift-on-modify regression — `soup plan` -> mutate -> `soup apply`
# ---------------------------------------------------------------------------
def test_apply_refuses_drift_end_to_end(tmp_path, monkeypatch):
from typer.testing import CliRunner
from soup_cli.cli import app
runner = CliRunner()
monkeypatch.chdir(tmp_path)
cfg = tmp_path / "soup.yaml"
cfg.write_text(
"base: meta-llama/Llama-3.2-1B\n"
"task: sft\n"
"data:\n"
" train: ./data.jsonl\n"
"training:\n"
" epochs: 1\n"
" lr: 0.00005\n"
" batch_size: 4\n"
)
(tmp_path / "data.jsonl").write_text("{}\n")
r = runner.invoke(app, ["plan", "--config", str(cfg)])
assert r.exit_code == 0, r.output
cfg.write_text(cfg.read_text().replace("epochs: 1", "epochs: 5"))
r2 = runner.invoke(app, ["apply", "--config", str(cfg), "--dry-run"])
assert r2.exit_code == 3, r2.output # exit 3 = drift refused
assert "drift" in r2.output.lower()

694
tests/test_v0640_part_a.py Normal file
View File

@ -0,0 +1,694 @@
"""v0.64.0 Part A — `soup tunability` probe across candidate bases.
Tests cover:
- CandidateBase frozen dataclass + validation
- TunabilityResult frozen dataclass + validation
- TunabilityReport frozen dataclass + immutable candidates tuple
- validate_probe_steps bounds + bool reject
- validate_holdout_size bounds + bool reject
- score_candidate happy + delta math (lower-better loss)
- pareto_frontier identifies non-dominated points
- run_tunability orchestrator (with mocked probe callable)
- write_report + load_report atomic + cwd containment + symlink rejection
- CLI smoke (--help / outside-cwd reject / unknown candidate)
"""
from __future__ import annotations
import dataclasses
import json
import os
import sys
import pytest
from typer.testing import CliRunner
runner = CliRunner()
# ---------------------------------------------------------------------------
# Module imports
# ---------------------------------------------------------------------------
def test_module_imports():
from soup_cli.utils import tunability
assert hasattr(tunability, "CandidateBase")
assert hasattr(tunability, "TunabilityResult")
assert hasattr(tunability, "TunabilityReport")
assert hasattr(tunability, "validate_probe_steps")
assert hasattr(tunability, "validate_holdout_size")
assert hasattr(tunability, "score_candidate")
assert hasattr(tunability, "pareto_frontier")
assert hasattr(tunability, "run_tunability")
assert hasattr(tunability, "write_report")
assert hasattr(tunability, "load_report")
assert hasattr(tunability, "DEFAULT_CANDIDATES")
# ---------------------------------------------------------------------------
# DEFAULT_CANDIDATES catalog
# ---------------------------------------------------------------------------
def test_default_candidates_nonempty():
from soup_cli.utils.tunability import DEFAULT_CANDIDATES
assert len(DEFAULT_CANDIDATES) >= 6
# Each entry must be CandidateBase
from soup_cli.utils.tunability import CandidateBase
for c in DEFAULT_CANDIDATES:
assert isinstance(c, CandidateBase)
def test_default_candidates_immutable():
from soup_cli.utils.tunability import DEFAULT_CANDIDATES
# Tuple, not list
assert isinstance(DEFAULT_CANDIDATES, tuple)
# ---------------------------------------------------------------------------
# CandidateBase
# ---------------------------------------------------------------------------
def test_candidate_base_frozen():
from soup_cli.utils.tunability import CandidateBase
c = CandidateBase(
name="qwen3-0.6b",
repo_id="Qwen/Qwen3-0.6B",
params_b=0.6,
license_id="apache-2.0",
)
with pytest.raises(dataclasses.FrozenInstanceError):
c.name = "other" # type: ignore[misc]
def test_candidate_base_rejects_empty_name():
from soup_cli.utils.tunability import CandidateBase
with pytest.raises(ValueError, match="name"):
CandidateBase(name="", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
def test_candidate_base_rejects_null_byte():
from soup_cli.utils.tunability import CandidateBase
with pytest.raises(ValueError, match="null"):
CandidateBase(name="bad\x00", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
def test_candidate_base_rejects_negative_params():
from soup_cli.utils.tunability import CandidateBase
with pytest.raises(ValueError, match="params_b"):
CandidateBase(name="x", repo_id="x/y", params_b=-1.0, license_id="apache-2.0")
def test_candidate_base_rejects_bool_params():
from soup_cli.utils.tunability import CandidateBase
with pytest.raises(TypeError, match="bool"):
CandidateBase(name="x", repo_id="x/y", params_b=True, license_id="apache-2.0") # type: ignore[arg-type]
def test_candidate_base_rejects_non_finite_params():
from soup_cli.utils.tunability import CandidateBase
with pytest.raises(ValueError, match="finite"):
CandidateBase(name="x", repo_id="x/y", params_b=float("nan"), license_id="apache-2.0")
def test_candidate_base_rejects_oversize_name():
from soup_cli.utils.tunability import CandidateBase
with pytest.raises(ValueError, match="too long"):
CandidateBase(name="x" * 513, repo_id="x/y", params_b=1.0, license_id="apache-2.0")
# ---------------------------------------------------------------------------
# validate_probe_steps
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("value", [10, 100, 1000])
def test_validate_probe_steps_happy(value):
from soup_cli.utils.tunability import validate_probe_steps
assert validate_probe_steps(value) == value
def test_validate_probe_steps_boundary_min():
from soup_cli.utils.tunability import validate_probe_steps
assert validate_probe_steps(10) == 10
with pytest.raises(ValueError):
validate_probe_steps(9)
def test_validate_probe_steps_boundary_max():
from soup_cli.utils.tunability import validate_probe_steps
assert validate_probe_steps(10_000) == 10_000
with pytest.raises(ValueError):
validate_probe_steps(10_001)
@pytest.mark.parametrize("bad", [True, False, None, "100", -1, 0, 9, 10_001, 1.5])
def test_validate_probe_steps_rejects(bad):
from soup_cli.utils.tunability import validate_probe_steps
with pytest.raises((TypeError, ValueError)):
validate_probe_steps(bad)
# ---------------------------------------------------------------------------
# validate_holdout_size
# ---------------------------------------------------------------------------
def test_validate_holdout_size_happy():
from soup_cli.utils.tunability import validate_holdout_size
assert validate_holdout_size(100) == 100
def test_validate_holdout_size_boundary():
from soup_cli.utils.tunability import validate_holdout_size
assert validate_holdout_size(10) == 10
with pytest.raises(ValueError):
validate_holdout_size(9)
assert validate_holdout_size(100_000) == 100_000
with pytest.raises(ValueError):
validate_holdout_size(100_001)
@pytest.mark.parametrize("bad", [True, False, "100", -1, 0])
def test_validate_holdout_size_rejects(bad):
from soup_cli.utils.tunability import validate_holdout_size
with pytest.raises((TypeError, ValueError)):
validate_holdout_size(bad)
# ---------------------------------------------------------------------------
# TunabilityResult
# ---------------------------------------------------------------------------
def test_tunability_result_happy():
from soup_cli.utils.tunability import CandidateBase, TunabilityResult
cand = CandidateBase(
name="qwen3-0.6b", repo_id="x/y", params_b=0.6, license_id="apache-2.0"
)
r = TunabilityResult(
candidate=cand,
base_loss=2.5,
probe_loss=2.0,
delta=0.5,
wall_clock_seconds=120.0,
estimated_cost_usd=0.05,
)
assert r.delta == 0.5
def test_tunability_result_frozen():
from soup_cli.utils.tunability import CandidateBase, TunabilityResult
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
r = TunabilityResult(
candidate=cand,
base_loss=2.5,
probe_loss=2.0,
delta=0.5,
wall_clock_seconds=120.0,
estimated_cost_usd=0.05,
)
with pytest.raises(dataclasses.FrozenInstanceError):
r.delta = 0.0 # type: ignore[misc]
def test_tunability_result_rejects_non_finite():
from soup_cli.utils.tunability import CandidateBase, TunabilityResult
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
with pytest.raises(ValueError, match="finite"):
TunabilityResult(
candidate=cand,
base_loss=float("nan"),
probe_loss=2.0,
delta=0.5,
wall_clock_seconds=120.0,
estimated_cost_usd=0.05,
)
def test_tunability_result_rejects_negative_wall_clock():
from soup_cli.utils.tunability import CandidateBase, TunabilityResult
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
with pytest.raises(ValueError, match="wall_clock"):
TunabilityResult(
candidate=cand,
base_loss=2.5,
probe_loss=2.0,
delta=0.5,
wall_clock_seconds=-1.0,
estimated_cost_usd=0.05,
)
def test_tunability_result_rejects_negative_cost():
from soup_cli.utils.tunability import CandidateBase, TunabilityResult
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
with pytest.raises(ValueError, match="cost"):
TunabilityResult(
candidate=cand,
base_loss=2.5,
probe_loss=2.0,
delta=0.5,
wall_clock_seconds=120.0,
estimated_cost_usd=-0.01,
)
def test_tunability_result_rejects_bool_loss():
from soup_cli.utils.tunability import CandidateBase, TunabilityResult
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
with pytest.raises(TypeError, match="bool"):
TunabilityResult(
candidate=cand,
base_loss=True, # type: ignore[arg-type]
probe_loss=2.0,
delta=0.5,
wall_clock_seconds=120.0,
estimated_cost_usd=0.05,
)
# ---------------------------------------------------------------------------
# score_candidate
# ---------------------------------------------------------------------------
def test_score_candidate_delta_math():
"""delta = base_loss - probe_loss (positive = improvement)."""
from soup_cli.utils.tunability import score_candidate
base_loss, probe_loss = 2.5, 2.0
delta = score_candidate(base_loss=base_loss, probe_loss=probe_loss)
assert delta == pytest.approx(0.5)
def test_score_candidate_zero_when_no_change():
from soup_cli.utils.tunability import score_candidate
assert score_candidate(base_loss=2.0, probe_loss=2.0) == 0.0
def test_score_candidate_negative_when_worse():
from soup_cli.utils.tunability import score_candidate
assert score_candidate(base_loss=2.0, probe_loss=2.5) == pytest.approx(-0.5)
def test_score_candidate_rejects_non_finite():
from soup_cli.utils.tunability import score_candidate
with pytest.raises(ValueError):
score_candidate(base_loss=float("nan"), probe_loss=2.0)
with pytest.raises(ValueError):
score_candidate(base_loss=2.0, probe_loss=float("inf"))
def test_score_candidate_rejects_bool():
from soup_cli.utils.tunability import score_candidate
with pytest.raises(TypeError):
score_candidate(base_loss=True, probe_loss=2.0) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# pareto_frontier
# ---------------------------------------------------------------------------
def test_pareto_frontier_simple():
"""Maximise delta, minimise cost. Strictly dominated entries get dropped."""
from soup_cli.utils.tunability import CandidateBase, TunabilityResult, pareto_frontier
def _mk(name: str, delta: float, cost: float) -> TunabilityResult:
cand = CandidateBase(name=name, repo_id="x/y", params_b=1.0, license_id="apache-2.0")
return TunabilityResult(
candidate=cand,
base_loss=2.5,
probe_loss=2.5 - delta,
delta=delta,
wall_clock_seconds=60.0,
estimated_cost_usd=cost,
)
# B dominates A: B has higher delta AND lower cost.
# C is on the frontier: lower delta but lower cost than B.
a = _mk("a", delta=0.1, cost=0.20)
b = _mk("b", delta=0.5, cost=0.10)
c = _mk("c", delta=0.05, cost=0.05)
frontier = pareto_frontier([a, b, c])
names = {r.candidate.name for r in frontier}
# A is strictly dominated by B (lower delta, higher cost)
assert "a" not in names
assert "b" in names
assert "c" in names
def test_pareto_frontier_empty():
from soup_cli.utils.tunability import pareto_frontier
assert pareto_frontier([]) == ()
def test_pareto_frontier_single():
from soup_cli.utils.tunability import CandidateBase, TunabilityResult, pareto_frontier
cand = CandidateBase(name="a", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
r = TunabilityResult(
candidate=cand,
base_loss=2.5,
probe_loss=2.0,
delta=0.5,
wall_clock_seconds=60.0,
estimated_cost_usd=0.05,
)
frontier = pareto_frontier([r])
assert frontier == (r,)
def test_pareto_frontier_returns_tuple():
from soup_cli.utils.tunability import pareto_frontier
assert isinstance(pareto_frontier([]), tuple)
def test_pareto_frontier_rejects_non_sequence():
from soup_cli.utils.tunability import pareto_frontier
with pytest.raises(TypeError):
pareto_frontier("not a list") # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# TunabilityReport
# ---------------------------------------------------------------------------
def test_tunability_report_frozen():
from soup_cli.utils.tunability import CandidateBase, TunabilityReport, TunabilityResult
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
r = TunabilityResult(
candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5,
wall_clock_seconds=60.0, estimated_cost_usd=0.05,
)
report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64)
with pytest.raises(dataclasses.FrozenInstanceError):
report.probe_steps = 0 # type: ignore[misc]
def test_tunability_report_results_tuple():
from soup_cli.utils.tunability import CandidateBase, TunabilityReport, TunabilityResult
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
r = TunabilityResult(
candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5,
wall_clock_seconds=60.0, estimated_cost_usd=0.05,
)
# Lists rejected — frozen=True doesn't make lists immutable
with pytest.raises(TypeError, match="tuple"):
TunabilityReport(results=[r], frontier=(r,), probe_steps=100, holdout_size=64) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# run_tunability
# ---------------------------------------------------------------------------
def test_run_tunability_with_mocked_probe(tmp_path):
"""Inject a deterministic probe to exercise the orchestrator."""
from soup_cli.utils.tunability import (
CandidateBase,
TunabilityResult,
run_tunability,
)
candidates = (
CandidateBase(name="cand-a", repo_id="x/y", params_b=0.5, license_id="apache-2.0"),
CandidateBase(name="cand-b", repo_id="x/z", params_b=1.0, license_id="mit"),
)
def fake_probe(cand: CandidateBase, dataset_path: str, *, probe_steps: int,
holdout_size: int) -> TunabilityResult:
# Synthetic: larger param count → bigger delta, longer wall-clock
return TunabilityResult(
candidate=cand,
base_loss=2.5,
probe_loss=2.5 - cand.params_b * 0.3,
delta=cand.params_b * 0.3,
wall_clock_seconds=cand.params_b * 60.0,
estimated_cost_usd=cand.params_b * 0.05,
)
dataset = tmp_path / "data.jsonl"
dataset.write_text('{"prompt": "x", "completion": "y"}\n')
report = run_tunability(
candidates=candidates,
dataset_path=str(dataset.relative_to(tmp_path)) if False else str(dataset),
probe_steps=50,
holdout_size=16,
probe_fn=fake_probe,
)
assert len(report.results) == 2
assert len(report.frontier) >= 1
assert report.probe_steps == 50
def test_run_tunability_rejects_empty_candidates(tmp_path):
from soup_cli.utils.tunability import run_tunability
dataset = tmp_path / "data.jsonl"
dataset.write_text("{}\n")
with pytest.raises(ValueError, match="candidates"):
run_tunability(
candidates=(),
dataset_path=str(dataset),
probe_steps=50,
holdout_size=16,
)
def test_run_tunability_rejects_invalid_probe_steps(tmp_path):
from soup_cli.utils.tunability import CandidateBase, run_tunability
dataset = tmp_path / "data.jsonl"
dataset.write_text("{}\n")
cands = (CandidateBase(name="a", repo_id="x/y", params_b=1.0, license_id="apache-2.0"),)
with pytest.raises(ValueError):
run_tunability(
candidates=cands,
dataset_path=str(dataset),
probe_steps=9, # below min
holdout_size=16,
)
# ---------------------------------------------------------------------------
# write_report / load_report
# ---------------------------------------------------------------------------
def test_write_report_atomic_roundtrip(tmp_path, monkeypatch):
from soup_cli.utils.tunability import (
CandidateBase,
TunabilityReport,
TunabilityResult,
load_report,
write_report,
)
monkeypatch.chdir(tmp_path)
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
r = TunabilityResult(
candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5,
wall_clock_seconds=60.0, estimated_cost_usd=0.05,
)
report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64)
out = tmp_path / "tunability.json"
write_report(report, str(out))
loaded = load_report(str(out))
assert loaded.probe_steps == report.probe_steps
assert len(loaded.results) == 1
assert loaded.results[0].candidate.name == "x"
def test_write_report_outside_cwd_rejected(tmp_path, monkeypatch):
from soup_cli.utils.tunability import (
CandidateBase,
TunabilityReport,
TunabilityResult,
write_report,
)
monkeypatch.chdir(tmp_path)
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
r = TunabilityResult(
candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5,
wall_clock_seconds=60.0, estimated_cost_usd=0.05,
)
report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64)
outside = tmp_path.parent / "evil.json"
with pytest.raises(ValueError, match="cwd"):
write_report(report, str(outside))
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_write_report_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.utils.tunability import (
CandidateBase,
TunabilityReport,
TunabilityResult,
write_report,
)
monkeypatch.chdir(tmp_path)
cand = CandidateBase(name="x", repo_id="x/y", params_b=1.0, license_id="apache-2.0")
r = TunabilityResult(
candidate=cand, base_loss=2.5, probe_loss=2.0, delta=0.5,
wall_clock_seconds=60.0, estimated_cost_usd=0.05,
)
report = TunabilityReport(results=(r,), frontier=(r,), probe_steps=100, holdout_size=64)
target = tmp_path / "real.json"
target.write_text("{}")
link = tmp_path / "link.json"
os.symlink(target, link)
with pytest.raises(ValueError, match="symlink"):
write_report(report, str(link))
def test_write_report_non_report_rejected(tmp_path):
from soup_cli.utils.tunability import write_report
with pytest.raises(TypeError):
write_report("not a report", str(tmp_path / "out.json")) # type: ignore[arg-type]
def test_load_report_missing_file(tmp_path, monkeypatch):
from soup_cli.utils.tunability import load_report
monkeypatch.chdir(tmp_path)
with pytest.raises(FileNotFoundError):
load_report(str(tmp_path / "nope.json"))
def test_load_report_invalid_json(tmp_path):
from soup_cli.utils.tunability import load_report
p = tmp_path / "bad.json"
p.write_text("not json")
with pytest.raises((ValueError, json.JSONDecodeError)):
load_report(str(p))
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def test_cli_tunability_help():
from soup_cli.cli import app
result = runner.invoke(app, ["tunability", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "tunability" in result.output.lower()
def test_cli_tunability_list_default_candidates():
from soup_cli.cli import app
result = runner.invoke(app, ["tunability", "--list"])
assert result.exit_code == 0, (result.output, repr(result.exception))
# Should list at least one default candidate name.
assert "qwen" in result.output.lower() or "llama" in result.output.lower() or \
"phi" in result.output.lower() or "gemma" in result.output.lower() or \
"smol" in result.output.lower()
def test_cli_tunability_requires_dataset(tmp_path, monkeypatch):
"""Without --list and without --dataset, exits with usage error."""
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["tunability"])
assert result.exit_code != 0
def test_cli_tunability_outside_cwd(tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "data.jsonl"
outside.write_text("{}\n")
result = runner.invoke(app, ["tunability", "--dataset", str(outside)])
assert result.exit_code != 0
def test_cli_tunability_plan_only(tmp_path, monkeypatch):
"""--plan-only enumerates candidates without running probes."""
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
dataset = tmp_path / "data.jsonl"
dataset.write_text('{"prompt": "x"}\n')
result = runner.invoke(app, ["tunability", "--dataset", str(dataset), "--plan-only"])
assert result.exit_code == 0, (result.output, repr(result.exception))
# ---------------------------------------------------------------------------
# Source-wiring regression guards
# ---------------------------------------------------------------------------
def test_cli_registers_tunability():
"""cli.py registers the tunability command."""
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py"
text = src.read_text(encoding="utf-8")
assert "tunability" in text
def test_version_bumped_to_0640():
import soup_cli
assert soup_cli.__version__ == "0.64.0"
def test_no_top_level_heavy_imports():
"""tunability module should not import torch/transformers/peft at top-level."""
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "tunability.py"
text = src.read_text(encoding="utf-8")
# Heavy deps must be lazy-imported inside functions
for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]:
# Strict line-start match
import re
assert not re.search(bad, text, re.MULTILINE), f"top-level {bad} found"

452
tests/test_v0640_part_b.py Normal file
View File

@ -0,0 +1,452 @@
"""v0.64.0 Part B — `soup plan` / `apply` (Terraform UX) tests."""
from __future__ import annotations
import dataclasses
import os
import sys
import pytest
from typer.testing import CliRunner
runner = CliRunner()
# ---------------------------------------------------------------------------
# Module imports
# ---------------------------------------------------------------------------
def test_module_imports():
from soup_cli.utils import terraform_plan
assert hasattr(terraform_plan, "TrainingPlan")
assert hasattr(terraform_plan, "TrainingState")
assert hasattr(terraform_plan, "build_plan")
assert hasattr(terraform_plan, "write_state")
assert hasattr(terraform_plan, "read_state")
assert hasattr(terraform_plan, "detect_drift")
assert hasattr(terraform_plan, "compute_config_sha")
assert hasattr(terraform_plan, "DEFAULT_STATE_FILE")
# ---------------------------------------------------------------------------
# compute_config_sha
# ---------------------------------------------------------------------------
def test_compute_config_sha_deterministic():
from soup_cli.utils.terraform_plan import compute_config_sha
a = compute_config_sha({"a": 1, "b": 2})
b = compute_config_sha({"b": 2, "a": 1}) # key order shouldn't matter
assert a == b
assert len(a) == 64 # SHA-256 hex
def test_compute_config_sha_changes_with_content():
from soup_cli.utils.terraform_plan import compute_config_sha
assert compute_config_sha({"a": 1}) != compute_config_sha({"a": 2})
def test_compute_config_sha_rejects_non_dict():
from soup_cli.utils.terraform_plan import compute_config_sha
with pytest.raises(TypeError):
compute_config_sha("not a dict") # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# TrainingPlan
# ---------------------------------------------------------------------------
def test_training_plan_frozen():
from soup_cli.utils.terraform_plan import TrainingPlan
p = TrainingPlan(
base="meta-llama/Llama-3.2-1B",
task="sft",
config_sha="a" * 64,
dataset_sha="b" * 64,
estimated_cost_usd=0.50,
estimated_minutes=10.0,
peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
with pytest.raises(dataclasses.FrozenInstanceError):
p.base = "other" # type: ignore[misc]
def test_training_plan_rejects_short_sha():
from soup_cli.utils.terraform_plan import TrainingPlan
with pytest.raises(ValueError, match="sha"):
TrainingPlan(
base="m",
task="sft",
config_sha="short",
dataset_sha="b" * 64,
estimated_cost_usd=0.50,
estimated_minutes=10.0,
peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
def test_training_plan_rejects_non_finite_cost():
from soup_cli.utils.terraform_plan import TrainingPlan
with pytest.raises(ValueError, match="finite"):
TrainingPlan(
base="m",
task="sft",
config_sha="a" * 64,
dataset_sha="b" * 64,
estimated_cost_usd=float("nan"),
estimated_minutes=10.0,
peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
def test_training_plan_rejects_bool_cost():
from soup_cli.utils.terraform_plan import TrainingPlan
with pytest.raises(TypeError, match="bool"):
TrainingPlan(
base="m",
task="sft",
config_sha="a" * 64,
dataset_sha="b" * 64,
estimated_cost_usd=True, # type: ignore[arg-type]
estimated_minutes=10.0,
peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
def test_training_plan_rejects_negative_cost():
from soup_cli.utils.terraform_plan import TrainingPlan
with pytest.raises(ValueError, match="negative"):
TrainingPlan(
base="m",
task="sft",
config_sha="a" * 64,
dataset_sha="b" * 64,
estimated_cost_usd=-1.0,
estimated_minutes=10.0,
peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
# ---------------------------------------------------------------------------
# build_plan
# ---------------------------------------------------------------------------
def test_build_plan_happy(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import build_plan
monkeypatch.chdir(tmp_path)
dataset = tmp_path / "data.jsonl"
dataset.write_text('{"prompt": "a", "completion": "b"}\n')
config = {
"base": "meta-llama/Llama-3.2-1B",
"task": "sft",
"data": {"train": str(dataset)},
"training": {"epochs": 1, "lr": 5e-5, "batch_size": 4},
}
plan = build_plan(config)
assert plan.base == "meta-llama/Llama-3.2-1B"
assert plan.task == "sft"
assert len(plan.config_sha) == 64
assert plan.estimated_cost_usd >= 0
assert plan.estimated_minutes >= 0
def test_build_plan_rejects_missing_base(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import build_plan
monkeypatch.chdir(tmp_path)
config = {"task": "sft", "data": {"train": "x.jsonl"}}
with pytest.raises(ValueError, match="base"):
build_plan(config)
def test_build_plan_rejects_non_dict():
from soup_cli.utils.terraform_plan import build_plan
with pytest.raises(TypeError):
build_plan("not a dict") # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# write_state / read_state
# ---------------------------------------------------------------------------
def test_write_state_roundtrip(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import (
TrainingPlan,
TrainingState,
read_state,
write_state,
)
monkeypatch.chdir(tmp_path)
plan = TrainingPlan(
base="m",
task="sft",
config_sha="a" * 64,
dataset_sha="b" * 64,
estimated_cost_usd=0.50,
estimated_minutes=10.0,
peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None)
out = tmp_path / "soup.tfstate"
write_state(state, str(out))
loaded = read_state(str(out))
assert loaded.plan.base == "m"
assert loaded.applied is False
def test_write_state_outside_cwd_rejected(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import TrainingPlan, TrainingState, write_state
monkeypatch.chdir(tmp_path)
plan = TrainingPlan(
base="m", task="sft", config_sha="a" * 64, dataset_sha="b" * 64,
estimated_cost_usd=0.5, estimated_minutes=10.0, peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None)
outside = tmp_path.parent / "evil.tfstate"
with pytest.raises(ValueError, match="cwd"):
write_state(state, str(outside))
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_write_state_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import TrainingPlan, TrainingState, write_state
monkeypatch.chdir(tmp_path)
plan = TrainingPlan(
base="m", task="sft", config_sha="a" * 64, dataset_sha="b" * 64,
estimated_cost_usd=0.5, estimated_minutes=10.0, peak_vram_gb=8.0,
spot_price_usd_per_hour=0.30,
)
state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None)
target = tmp_path / "real.tfstate"
target.write_text("{}")
link = tmp_path / "link.tfstate"
os.symlink(target, link)
with pytest.raises(ValueError, match="symlink"):
write_state(state, str(link))
def test_read_state_missing(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import read_state
monkeypatch.chdir(tmp_path)
with pytest.raises(FileNotFoundError):
read_state(str(tmp_path / "nope.tfstate"))
# ---------------------------------------------------------------------------
# detect_drift
# ---------------------------------------------------------------------------
def test_detect_drift_clean(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import (
TrainingState,
build_plan,
detect_drift,
)
monkeypatch.chdir(tmp_path)
dataset = tmp_path / "data.jsonl"
dataset.write_text("{}\n")
config = {
"base": "m",
"task": "sft",
"data": {"train": str(dataset)},
"training": {"epochs": 1, "lr": 5e-5, "batch_size": 4},
}
plan = build_plan(config)
state = TrainingState(plan=plan, applied=False, applied_at=None, run_id=None)
# Rebuild plan from same config → no drift
plan_now = build_plan(config)
drift = detect_drift(state, plan_now)
assert drift.has_drift is False
def test_detect_drift_dirty(tmp_path, monkeypatch):
from soup_cli.utils.terraform_plan import (
TrainingState,
build_plan,
detect_drift,
)
monkeypatch.chdir(tmp_path)
dataset = tmp_path / "data.jsonl"
dataset.write_text("{}\n")
config_v1 = {
"base": "m",
"task": "sft",
"data": {"train": str(dataset)},
"training": {"epochs": 1, "lr": 5e-5, "batch_size": 4},
}
plan_v1 = build_plan(config_v1)
state = TrainingState(plan=plan_v1, applied=False, applied_at=None, run_id=None)
# Mutate config
config_v2 = dict(config_v1)
config_v2["training"] = {"epochs": 2, "lr": 5e-5, "batch_size": 4}
plan_v2 = build_plan(config_v2)
drift = detect_drift(state, plan_v2)
assert drift.has_drift is True
assert "config_sha" in drift.changed_fields
def test_detect_drift_rejects_non_state():
from soup_cli.utils.terraform_plan import detect_drift
with pytest.raises(TypeError):
detect_drift("not state", "not plan") # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# CLI smoke
# ---------------------------------------------------------------------------
def _write_minimal_config(path):
path.write_text(
"base: meta-llama/Llama-3.2-1B\n"
"task: sft\n"
"data:\n"
" train: ./data.jsonl\n"
"training:\n"
" epochs: 1\n"
" lr: 0.00005\n"
" batch_size: 4\n"
)
def test_cli_plan_help():
from soup_cli.cli import app
result = runner.invoke(app, ["plan", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_apply_help():
from soup_cli.cli import app
result = runner.invoke(app, ["apply", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_plan_happy(tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
cfg = tmp_path / "soup.yaml"
_write_minimal_config(cfg)
(tmp_path / "data.jsonl").write_text("{}\n")
result = runner.invoke(app, ["plan", "--config", str(cfg)])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert (tmp_path / "soup.tfstate").exists()
def test_cli_plan_outside_cwd_config(tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
outside_cfg = tmp_path.parent / "evil.yaml"
_write_minimal_config(outside_cfg)
result = runner.invoke(app, ["plan", "--config", str(outside_cfg)])
assert result.exit_code != 0
def test_cli_apply_drift_refused(tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
cfg = tmp_path / "soup.yaml"
_write_minimal_config(cfg)
(tmp_path / "data.jsonl").write_text("{}\n")
# plan first
r = runner.invoke(app, ["plan", "--config", str(cfg)])
assert r.exit_code == 0, (r.output, repr(r.exception))
# mutate config
cfg.write_text(cfg.read_text().replace("epochs: 1", "epochs: 99"))
# apply must refuse on drift
r2 = runner.invoke(app, ["apply", "--config", str(cfg)])
assert r2.exit_code != 0
assert "drift" in r2.output.lower() or "mismatch" in r2.output.lower()
def test_cli_apply_clean(tmp_path, monkeypatch):
"""apply with --dry-run after a plan should succeed clean."""
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
cfg = tmp_path / "soup.yaml"
_write_minimal_config(cfg)
(tmp_path / "data.jsonl").write_text("{}\n")
r = runner.invoke(app, ["plan", "--config", str(cfg)])
assert r.exit_code == 0, (r.output, repr(r.exception))
r2 = runner.invoke(app, ["apply", "--config", str(cfg), "--dry-run"])
assert r2.exit_code == 0, (r2.output, repr(r2.exception))
# ---------------------------------------------------------------------------
# DriftReport frozen
# ---------------------------------------------------------------------------
def test_drift_report_frozen():
from soup_cli.utils.terraform_plan import DriftReport
d = DriftReport(has_drift=False, changed_fields=())
with pytest.raises(dataclasses.FrozenInstanceError):
d.has_drift = True # type: ignore[misc]
def test_drift_report_changed_fields_is_tuple():
from soup_cli.utils.terraform_plan import DriftReport
with pytest.raises(TypeError, match="tuple"):
DriftReport(has_drift=True, changed_fields=["x"]) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# Source-wiring regression
# ---------------------------------------------------------------------------
def test_cli_registers_plan_apply():
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py"
text = src.read_text(encoding="utf-8")
assert '"plan"' in text or "'plan'" in text or "name=\"plan\"" in text
def test_no_heavy_top_level_imports():
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "terraform_plan.py"
text = src.read_text(encoding="utf-8")
import re
for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]:
assert not re.search(bad, text, re.MULTILINE)

427
tests/test_v0640_part_c.py Normal file
View File

@ -0,0 +1,427 @@
"""v0.64.0 Part C — `soup env` hermetic lockfile + ABI-mismatch detection."""
from __future__ import annotations
import dataclasses
import os
import sys
import pytest
from typer.testing import CliRunner
runner = CliRunner()
# ---------------------------------------------------------------------------
# Module imports
# ---------------------------------------------------------------------------
def test_module_imports():
from soup_cli.utils import env_lock
assert hasattr(env_lock, "EnvLock")
assert hasattr(env_lock, "EnvEntry")
assert hasattr(env_lock, "AbiCheck")
assert hasattr(env_lock, "TRACKED_PACKAGES")
assert hasattr(env_lock, "snapshot_env")
assert hasattr(env_lock, "write_lock")
assert hasattr(env_lock, "read_lock")
assert hasattr(env_lock, "check_abi_compat")
assert hasattr(env_lock, "DEFAULT_LOCK_FILE")
# ---------------------------------------------------------------------------
# TRACKED_PACKAGES catalogue
# ---------------------------------------------------------------------------
def test_tracked_packages_contains_core():
from soup_cli.utils.env_lock import TRACKED_PACKAGES
# Should at least include the ABI-sensitive heavyweights
names = {p.lower() for p in TRACKED_PACKAGES}
assert "torch" in names
assert "transformers" in names
assert "peft" in names
def test_tracked_packages_is_tuple():
from soup_cli.utils.env_lock import TRACKED_PACKAGES
assert isinstance(TRACKED_PACKAGES, tuple)
# ---------------------------------------------------------------------------
# EnvEntry
# ---------------------------------------------------------------------------
def test_env_entry_frozen():
from soup_cli.utils.env_lock import EnvEntry
e = EnvEntry(name="torch", version="2.1.0", source="pip")
with pytest.raises(dataclasses.FrozenInstanceError):
e.version = "0.0" # type: ignore[misc]
def test_env_entry_rejects_empty_name():
from soup_cli.utils.env_lock import EnvEntry
with pytest.raises(ValueError, match="name"):
EnvEntry(name="", version="1.0", source="pip")
def test_env_entry_rejects_null_byte():
from soup_cli.utils.env_lock import EnvEntry
with pytest.raises(ValueError, match="null"):
EnvEntry(name="t\x00", version="1.0", source="pip")
def test_env_entry_rejects_invalid_source():
from soup_cli.utils.env_lock import EnvEntry
with pytest.raises(ValueError, match="source"):
EnvEntry(name="t", version="1.0", source="random-bogus-thing")
def test_env_entry_known_sources():
from soup_cli.utils.env_lock import EnvEntry
# Just verify each known source instantiates clean
for src in ("pip", "conda", "system", "wheel", "unknown"):
EnvEntry(name="t", version="1.0", source=src)
# ---------------------------------------------------------------------------
# EnvLock
# ---------------------------------------------------------------------------
def test_env_lock_frozen():
from soup_cli.utils.env_lock import EnvEntry, EnvLock
lock = EnvLock(
soup_version="0.64.0",
python_version="3.10.0",
platform="linux",
cuda_version=None,
entries=(EnvEntry(name="torch", version="2.0", source="pip"),),
created_at="2026-05-20T00:00:00+00:00",
)
with pytest.raises(dataclasses.FrozenInstanceError):
lock.soup_version = "0.0.0" # type: ignore[misc]
def test_env_lock_entries_must_be_tuple():
from soup_cli.utils.env_lock import EnvEntry, EnvLock
with pytest.raises(TypeError, match="tuple"):
EnvLock(
soup_version="0.64.0",
python_version="3.10.0",
platform="linux",
cuda_version=None,
entries=[EnvEntry(name="t", version="1.0", source="pip")], # type: ignore[arg-type]
created_at="2026-05-20T00:00:00+00:00",
)
def test_env_lock_rejects_null_byte_in_platform():
from soup_cli.utils.env_lock import EnvLock
with pytest.raises(ValueError, match="null"):
EnvLock(
soup_version="0.64.0",
python_version="3.10.0",
platform="linux\x00",
cuda_version=None,
entries=(),
created_at="2026-05-20T00:00:00+00:00",
)
# ---------------------------------------------------------------------------
# snapshot_env
# ---------------------------------------------------------------------------
def test_snapshot_env_returns_envlock():
from soup_cli.utils.env_lock import EnvLock, snapshot_env
lock = snapshot_env()
assert isinstance(lock, EnvLock)
# python_version should at least parse to two segments
assert "." in lock.python_version
def test_snapshot_env_entries_non_empty():
"""The snapshot should contain at least one tracked package (pytest)."""
from soup_cli.utils.env_lock import snapshot_env
lock = snapshot_env()
# The function inspects whatever's installed; pytest will at least be there.
# We just verify the entries tuple is well-formed.
assert isinstance(lock.entries, tuple)
# ---------------------------------------------------------------------------
# write_lock / read_lock
# ---------------------------------------------------------------------------
def test_write_lock_roundtrip(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import EnvEntry, EnvLock, read_lock, write_lock
monkeypatch.chdir(tmp_path)
lock = EnvLock(
soup_version="0.64.0",
python_version="3.10.0",
platform="linux",
cuda_version="12.1",
entries=(EnvEntry(name="torch", version="2.0", source="pip"),),
created_at="2026-05-20T00:00:00+00:00",
)
out = tmp_path / "soup-env.lock"
write_lock(lock, str(out))
loaded = read_lock(str(out))
assert loaded.soup_version == "0.64.0"
assert len(loaded.entries) == 1
assert loaded.entries[0].name == "torch"
def test_write_lock_outside_cwd_rejected(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import EnvLock, write_lock
monkeypatch.chdir(tmp_path)
lock = EnvLock(
soup_version="0.64.0",
python_version="3.10.0",
platform="linux",
cuda_version=None,
entries=(),
created_at="2026-05-20T00:00:00+00:00",
)
outside = tmp_path.parent / "evil.lock"
with pytest.raises(ValueError, match="cwd"):
write_lock(lock, str(outside))
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlinks")
def test_write_lock_symlink_rejected(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import EnvLock, write_lock
monkeypatch.chdir(tmp_path)
lock = EnvLock(
soup_version="0.64.0",
python_version="3.10.0",
platform="linux",
cuda_version=None,
entries=(),
created_at="2026-05-20T00:00:00+00:00",
)
target = tmp_path / "real.lock"
target.write_text("{}")
link = tmp_path / "link.lock"
os.symlink(target, link)
with pytest.raises(ValueError, match="symlink"):
write_lock(lock, str(link))
def test_read_lock_missing(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import read_lock
monkeypatch.chdir(tmp_path)
with pytest.raises(FileNotFoundError):
read_lock(str(tmp_path / "nope.lock"))
def test_read_lock_invalid_json(tmp_path, monkeypatch):
from soup_cli.utils.env_lock import read_lock
monkeypatch.chdir(tmp_path)
p = tmp_path / "bad.lock"
p.write_text("not json")
with pytest.raises(ValueError):
read_lock(str(p))
# ---------------------------------------------------------------------------
# AbiCheck
# ---------------------------------------------------------------------------
def test_abi_check_frozen():
from soup_cli.utils.env_lock import AbiCheck
a = AbiCheck(ok=True, drift_count=0, changes=())
with pytest.raises(dataclasses.FrozenInstanceError):
a.ok = False # type: ignore[misc]
def test_abi_check_changes_is_tuple():
from soup_cli.utils.env_lock import AbiCheck
with pytest.raises(TypeError, match="tuple"):
AbiCheck(ok=False, drift_count=1, changes=["x"]) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# check_abi_compat
# ---------------------------------------------------------------------------
def test_check_abi_compat_no_drift():
from soup_cli.utils.env_lock import EnvEntry, EnvLock, check_abi_compat
a = EnvLock(
soup_version="0.64.0", python_version="3.10.0", platform="linux",
cuda_version="12.1",
entries=(EnvEntry(name="torch", version="2.0", source="pip"),),
created_at="2026-05-20T00:00:00+00:00",
)
b = EnvLock(
soup_version="0.64.0", python_version="3.10.0", platform="linux",
cuda_version="12.1",
entries=(EnvEntry(name="torch", version="2.0", source="pip"),),
created_at="2026-05-21T00:00:00+00:00", # timestamp differs but ABI same
)
report = check_abi_compat(a, b)
assert report.ok is True
assert report.drift_count == 0
def test_check_abi_compat_torch_change():
from soup_cli.utils.env_lock import EnvEntry, EnvLock, check_abi_compat
a = EnvLock(
soup_version="0.64.0", python_version="3.10.0", platform="linux",
cuda_version="12.1",
entries=(EnvEntry(name="torch", version="2.0", source="pip"),),
created_at="2026-05-20T00:00:00+00:00",
)
b = EnvLock(
soup_version="0.64.0", python_version="3.10.0", platform="linux",
cuda_version="12.1",
entries=(EnvEntry(name="torch", version="2.1", source="pip"),),
created_at="2026-05-20T00:00:00+00:00",
)
report = check_abi_compat(a, b)
assert report.ok is False
assert report.drift_count >= 1
def test_check_abi_compat_cuda_change():
from soup_cli.utils.env_lock import EnvLock, check_abi_compat
a = EnvLock(
soup_version="0.64.0", python_version="3.10.0", platform="linux",
cuda_version="12.1", entries=(),
created_at="2026-05-20T00:00:00+00:00",
)
b = EnvLock(
soup_version="0.64.0", python_version="3.10.0", platform="linux",
cuda_version="11.8", entries=(),
created_at="2026-05-20T00:00:00+00:00",
)
report = check_abi_compat(a, b)
assert report.ok is False
assert any("cuda" in c.lower() for c in report.changes)
def test_check_abi_compat_python_change():
from soup_cli.utils.env_lock import EnvLock, check_abi_compat
a = EnvLock(
soup_version="0.64.0", python_version="3.10.0", platform="linux",
cuda_version=None, entries=(),
created_at="2026-05-20T00:00:00+00:00",
)
b = EnvLock(
soup_version="0.64.0", python_version="3.11.0", platform="linux",
cuda_version=None, entries=(),
created_at="2026-05-20T00:00:00+00:00",
)
report = check_abi_compat(a, b)
assert report.ok is False
def test_check_abi_compat_rejects_non_envlock():
from soup_cli.utils.env_lock import check_abi_compat
with pytest.raises(TypeError):
check_abi_compat("not a lock", "not a lock") # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# CLI smoke
# ---------------------------------------------------------------------------
def test_cli_env_help():
from soup_cli.cli import app
result = runner.invoke(app, ["env", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_env_lock_writes_file(tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["env", "lock"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert (tmp_path / "soup-env.lock").exists()
def test_cli_env_status_no_lock(tmp_path, monkeypatch):
"""`env status` without an existing lock file exits with friendly error."""
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["env", "status"])
# Should not crash; either exits non-zero with message or shows empty status.
assert result.exit_code in (0, 1)
def test_cli_env_status_with_lock(tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
r1 = runner.invoke(app, ["env", "lock"])
assert r1.exit_code == 0, (r1.output, repr(r1.exception))
r2 = runner.invoke(app, ["env", "status"])
assert r2.exit_code == 0, (r2.output, repr(r2.exception))
def test_cli_env_lock_outside_cwd_rejected(tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "evil.lock"
result = runner.invoke(app, ["env", "lock", "--output", str(outside)])
assert result.exit_code != 0
# ---------------------------------------------------------------------------
# Source-wiring regression
# ---------------------------------------------------------------------------
def test_cli_registers_env():
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py"
text = src.read_text(encoding="utf-8")
assert '"env"' in text or "'env'" in text or 'name="env"' in text
def test_no_heavy_top_level_imports():
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "env_lock.py"
text = src.read_text(encoding="utf-8")
import re
for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]:
assert not re.search(bad, text, re.MULTILINE)

378
tests/test_v0640_part_d.py Normal file
View File

@ -0,0 +1,378 @@
"""v0.64.0 Part D — Hardware-fit calculator tests."""
from __future__ import annotations
import dataclasses
import math
import pytest
# ---------------------------------------------------------------------------
# Module imports
# ---------------------------------------------------------------------------
def test_module_imports():
from soup_cli.utils import hardware_fit
assert hasattr(hardware_fit, "HardwareFitInput")
assert hasattr(hardware_fit, "VRAMBreakdown")
assert hasattr(hardware_fit, "HardwareFitReport")
assert hasattr(hardware_fit, "estimate_peak_vram_gb")
assert hasattr(hardware_fit, "decide_hardware_fit")
assert hasattr(hardware_fit, "validate_seq_len")
assert hasattr(hardware_fit, "validate_batch_size")
assert hasattr(hardware_fit, "VRAM_SAFETY_MARGIN")
# ---------------------------------------------------------------------------
# VRAM_SAFETY_MARGIN
# ---------------------------------------------------------------------------
def test_safety_margin_is_10pct():
from soup_cli.utils.hardware_fit import VRAM_SAFETY_MARGIN
assert VRAM_SAFETY_MARGIN == pytest.approx(0.10, abs=1e-6)
# ---------------------------------------------------------------------------
# validate_seq_len
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("v", [64, 1024, 8192, 1_048_576])
def test_validate_seq_len_happy(v):
from soup_cli.utils.hardware_fit import validate_seq_len
assert validate_seq_len(v) == v
@pytest.mark.parametrize("bad", [True, False, "1024", -1, 0, 63, 1_048_577, 1.5])
def test_validate_seq_len_rejects(bad):
from soup_cli.utils.hardware_fit import validate_seq_len
with pytest.raises((TypeError, ValueError)):
validate_seq_len(bad)
# ---------------------------------------------------------------------------
# validate_batch_size
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("v", [1, 4, 64, 1024])
def test_validate_batch_size_happy(v):
from soup_cli.utils.hardware_fit import validate_batch_size
assert validate_batch_size(v) == v
@pytest.mark.parametrize("bad", [True, False, "4", -1, 0, 1025, 1.5])
def test_validate_batch_size_rejects(bad):
from soup_cli.utils.hardware_fit import validate_batch_size
with pytest.raises((TypeError, ValueError)):
validate_batch_size(bad)
# ---------------------------------------------------------------------------
# HardwareFitInput
# ---------------------------------------------------------------------------
def test_input_frozen():
from soup_cli.utils.hardware_fit import HardwareFitInput
inp = HardwareFitInput(
params_b=7.0, seq_len=2048, batch_size=4,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
with pytest.raises(dataclasses.FrozenInstanceError):
inp.batch_size = 1 # type: ignore[misc]
def test_input_rejects_negative_params():
from soup_cli.utils.hardware_fit import HardwareFitInput
with pytest.raises(ValueError, match="params"):
HardwareFitInput(
params_b=-1.0, seq_len=2048, batch_size=4,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
def test_input_rejects_invalid_quant():
from soup_cli.utils.hardware_fit import HardwareFitInput
with pytest.raises(ValueError, match="quant"):
HardwareFitInput(
params_b=7.0, seq_len=2048, batch_size=4,
optimizer="adamw_torch", quant="bogus", peft="lora",
gradient_checkpointing=True,
)
def test_input_rejects_invalid_peft():
from soup_cli.utils.hardware_fit import HardwareFitInput
with pytest.raises(ValueError, match="peft"):
HardwareFitInput(
params_b=7.0, seq_len=2048, batch_size=4,
optimizer="adamw_torch", quant="4bit", peft="bogus",
gradient_checkpointing=True,
)
def test_input_rejects_bool_params():
from soup_cli.utils.hardware_fit import HardwareFitInput
with pytest.raises(TypeError, match="bool"):
HardwareFitInput(
params_b=True, seq_len=2048, batch_size=4, # type: ignore[arg-type]
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
def test_input_rejects_non_bool_gradient_ckpt():
from soup_cli.utils.hardware_fit import HardwareFitInput
with pytest.raises(TypeError, match="gradient_checkpointing"):
HardwareFitInput(
params_b=7.0, seq_len=2048, batch_size=4,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=1, # type: ignore[arg-type]
)
# ---------------------------------------------------------------------------
# VRAMBreakdown
# ---------------------------------------------------------------------------
def test_vram_breakdown_frozen():
from soup_cli.utils.hardware_fit import VRAMBreakdown
v = VRAMBreakdown(
weights_gb=2.0,
optimizer_gb=0.5,
gradients_gb=0.5,
activations_gb=1.0,
overhead_gb=0.5,
)
with pytest.raises(dataclasses.FrozenInstanceError):
v.weights_gb = 0.0 # type: ignore[misc]
def test_vram_breakdown_rejects_negative():
from soup_cli.utils.hardware_fit import VRAMBreakdown
with pytest.raises(ValueError, match="negative"):
VRAMBreakdown(
weights_gb=-1.0,
optimizer_gb=0.5,
gradients_gb=0.5,
activations_gb=1.0,
overhead_gb=0.5,
)
def test_vram_breakdown_total():
from soup_cli.utils.hardware_fit import VRAMBreakdown
v = VRAMBreakdown(
weights_gb=2.0,
optimizer_gb=0.5,
gradients_gb=0.5,
activations_gb=1.0,
overhead_gb=0.5,
)
assert v.total_gb == pytest.approx(4.5)
# ---------------------------------------------------------------------------
# estimate_peak_vram_gb
# ---------------------------------------------------------------------------
def test_estimate_peak_vram_returns_breakdown():
from soup_cli.utils.hardware_fit import HardwareFitInput, VRAMBreakdown, estimate_peak_vram_gb
inp = HardwareFitInput(
params_b=7.0, seq_len=2048, batch_size=4,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
breakdown = estimate_peak_vram_gb(inp)
assert isinstance(breakdown, VRAMBreakdown)
assert breakdown.weights_gb > 0
assert breakdown.total_gb > 0
def test_estimate_peak_vram_4bit_smaller_than_fp16():
from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb
base = dict(params_b=7.0, seq_len=1024, batch_size=1,
optimizer="adamw_torch", peft="lora",
gradient_checkpointing=True)
fp16 = estimate_peak_vram_gb(HardwareFitInput(quant="none", **base))
q4 = estimate_peak_vram_gb(HardwareFitInput(quant="4bit", **base))
assert q4.weights_gb < fp16.weights_gb
def test_estimate_peak_vram_lora_smaller_than_full():
from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb
base = dict(params_b=7.0, seq_len=1024, batch_size=1,
optimizer="adamw_torch", quant="4bit",
gradient_checkpointing=True)
full = estimate_peak_vram_gb(HardwareFitInput(peft="full", **base))
lora = estimate_peak_vram_gb(HardwareFitInput(peft="lora", **base))
assert lora.optimizer_gb < full.optimizer_gb
def test_estimate_peak_vram_seq_len_scales_activations():
from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb
base = dict(params_b=1.0, batch_size=1,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=False)
short = estimate_peak_vram_gb(HardwareFitInput(seq_len=512, **base))
long_ = estimate_peak_vram_gb(HardwareFitInput(seq_len=8192, **base))
assert long_.activations_gb > short.activations_gb
def test_estimate_peak_vram_grad_ckpt_reduces_activations():
from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb
base = dict(params_b=7.0, seq_len=4096, batch_size=1,
optimizer="adamw_torch", quant="4bit", peft="lora")
off = estimate_peak_vram_gb(HardwareFitInput(gradient_checkpointing=False, **base))
on = estimate_peak_vram_gb(HardwareFitInput(gradient_checkpointing=True, **base))
assert on.activations_gb < off.activations_gb
def test_estimate_peak_vram_finite():
from soup_cli.utils.hardware_fit import HardwareFitInput, estimate_peak_vram_gb
inp = HardwareFitInput(
params_b=70.0, seq_len=8192, batch_size=8,
optimizer="adamw_torch", quant="none", peft="full",
gradient_checkpointing=False,
)
bd = estimate_peak_vram_gb(inp)
assert math.isfinite(bd.total_gb)
def test_estimate_peak_vram_rejects_non_input():
from soup_cli.utils.hardware_fit import estimate_peak_vram_gb
with pytest.raises(TypeError):
estimate_peak_vram_gb("not an input") # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# decide_hardware_fit
# ---------------------------------------------------------------------------
def test_decide_hardware_fit_ok(tmp_path):
from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit
inp = HardwareFitInput(
params_b=1.0, seq_len=1024, batch_size=1,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
# Plenty of headroom
report = decide_hardware_fit(inp, available_vram_gb=24.0)
assert report.ok is True
assert report.peak_vram_gb > 0
assert report.required_with_margin_gb >= report.peak_vram_gb
def test_decide_hardware_fit_oom():
from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit
inp = HardwareFitInput(
params_b=70.0, seq_len=8192, batch_size=8,
optimizer="adamw_torch", quant="none", peft="full",
gradient_checkpointing=False,
)
report = decide_hardware_fit(inp, available_vram_gb=8.0)
assert report.ok is False
assert "exceed" in report.reason.lower() or "oom" in report.reason.lower() or \
"available" in report.reason.lower()
def test_decide_hardware_fit_rejects_negative_vram():
from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit
inp = HardwareFitInput(
params_b=1.0, seq_len=1024, batch_size=1,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
with pytest.raises(ValueError, match="vram"):
decide_hardware_fit(inp, available_vram_gb=-1.0)
def test_decide_hardware_fit_rejects_bool_vram():
from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit
inp = HardwareFitInput(
params_b=1.0, seq_len=1024, batch_size=1,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
with pytest.raises(TypeError, match="bool"):
decide_hardware_fit(inp, available_vram_gb=True) # type: ignore[arg-type]
def test_decide_hardware_fit_rejects_non_finite_vram():
from soup_cli.utils.hardware_fit import HardwareFitInput, decide_hardware_fit
inp = HardwareFitInput(
params_b=1.0, seq_len=1024, batch_size=1,
optimizer="adamw_torch", quant="4bit", peft="lora",
gradient_checkpointing=True,
)
with pytest.raises(ValueError, match="finite"):
decide_hardware_fit(inp, available_vram_gb=float("nan"))
def test_hardware_fit_report_frozen():
from soup_cli.utils.hardware_fit import HardwareFitReport, VRAMBreakdown
bd = VRAMBreakdown(
weights_gb=1.0, optimizer_gb=0.5, gradients_gb=0.5,
activations_gb=1.0, overhead_gb=0.5,
)
rep = HardwareFitReport(
ok=True,
peak_vram_gb=3.5,
required_with_margin_gb=3.85,
available_vram_gb=24.0,
breakdown=bd,
reason="ok",
)
with pytest.raises(dataclasses.FrozenInstanceError):
rep.ok = False # type: ignore[misc]
# ---------------------------------------------------------------------------
# Source-wiring regression
# ---------------------------------------------------------------------------
def test_no_heavy_top_level_imports():
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "hardware_fit.py"
text = src.read_text(encoding="utf-8")
import re
for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]:
assert not re.search(bad, text, re.MULTILINE)

259
tests/test_v0640_part_e.py Normal file
View File

@ -0,0 +1,259 @@
"""v0.64.0 Part E — Shell completions with config introspection."""
from __future__ import annotations
import pytest
from typer.testing import CliRunner
runner = CliRunner()
def test_module_imports():
from soup_cli.utils import completions
assert hasattr(completions, "SUPPORTED_SHELLS")
assert hasattr(completions, "validate_shell")
assert hasattr(completions, "render_bash_script")
assert hasattr(completions, "render_zsh_script")
assert hasattr(completions, "render_fish_script")
assert hasattr(completions, "render_completion_script")
assert hasattr(completions, "complete_recipe_name")
assert hasattr(completions, "complete_target_modules")
# ---------------------------------------------------------------------------
# SUPPORTED_SHELLS
# ---------------------------------------------------------------------------
def test_supported_shells():
from soup_cli.utils.completions import SUPPORTED_SHELLS
assert "bash" in SUPPORTED_SHELLS
assert "zsh" in SUPPORTED_SHELLS
assert "fish" in SUPPORTED_SHELLS
def test_supported_shells_is_frozenset():
from soup_cli.utils.completions import SUPPORTED_SHELLS
assert isinstance(SUPPORTED_SHELLS, frozenset)
# ---------------------------------------------------------------------------
# validate_shell
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("v", ["bash", "zsh", "fish"])
def test_validate_shell_happy(v):
from soup_cli.utils.completions import validate_shell
assert validate_shell(v) == v
def test_validate_shell_case_insensitive():
from soup_cli.utils.completions import validate_shell
assert validate_shell("BASH") == "bash"
assert validate_shell("Fish") == "fish"
@pytest.mark.parametrize(
"bad",
[True, False, None, "", "tcsh", "powershell", "ksh", "csh", "x" * 33],
)
def test_validate_shell_rejects(bad):
from soup_cli.utils.completions import validate_shell
with pytest.raises((TypeError, ValueError)):
validate_shell(bad)
def test_validate_shell_rejects_null_byte():
from soup_cli.utils.completions import validate_shell
with pytest.raises(ValueError, match="null"):
validate_shell("ba\x00sh")
# ---------------------------------------------------------------------------
# render_*_script
# ---------------------------------------------------------------------------
def test_render_bash_script_basic():
from soup_cli.utils.completions import render_bash_script
text = render_bash_script()
assert "_soup_complete" in text or "complete -F" in text
assert "soup" in text
def test_render_zsh_script_basic():
from soup_cli.utils.completions import render_zsh_script
text = render_zsh_script()
assert "#compdef soup" in text or "_soup" in text
def test_render_fish_script_basic():
from soup_cli.utils.completions import render_fish_script
text = render_fish_script()
assert "complete -c soup" in text or "complete --command soup" in text
def test_render_completion_script_dispatch():
from soup_cli.utils.completions import render_completion_script
bash = render_completion_script("bash")
assert "soup" in bash
zsh = render_completion_script("zsh")
assert "soup" in zsh
fish = render_completion_script("fish")
assert "soup" in fish
def test_render_completion_script_rejects_unknown():
from soup_cli.utils.completions import render_completion_script
with pytest.raises(ValueError):
render_completion_script("tcsh")
# ---------------------------------------------------------------------------
# complete_recipe_name
# ---------------------------------------------------------------------------
def test_complete_recipe_name_returns_list():
from soup_cli.utils.completions import complete_recipe_name
suggestions = complete_recipe_name("")
assert isinstance(suggestions, list)
assert len(suggestions) > 0
def test_complete_recipe_name_filters_prefix():
from soup_cli.utils.completions import complete_recipe_name
suggestions = complete_recipe_name("llama")
# Every result must start with the prefix (case-insensitive)
for s in suggestions:
assert s.lower().startswith("llama")
def test_complete_recipe_name_empty_for_nonsense():
from soup_cli.utils.completions import complete_recipe_name
suggestions = complete_recipe_name("definitely-not-a-recipe-zzzzzz")
assert suggestions == []
def test_complete_recipe_name_rejects_bool():
from soup_cli.utils.completions import complete_recipe_name
with pytest.raises(TypeError):
complete_recipe_name(True) # type: ignore[arg-type]
def test_complete_recipe_name_null_byte_returns_empty():
from soup_cli.utils.completions import complete_recipe_name
# Defensive: null byte returns empty rather than raising.
assert complete_recipe_name("foo\x00bar") == []
# ---------------------------------------------------------------------------
# complete_target_modules
# ---------------------------------------------------------------------------
def test_complete_target_modules_default():
from soup_cli.utils.completions import complete_target_modules
# With no base, returns canonical Llama-shape modules
suggestions = complete_target_modules("", base=None)
assert "q_proj" in suggestions
assert "k_proj" in suggestions
def test_complete_target_modules_filters_prefix():
from soup_cli.utils.completions import complete_target_modules
suggestions = complete_target_modules("q_", base=None)
for s in suggestions:
assert s.startswith("q_")
def test_complete_target_modules_rejects_bool():
from soup_cli.utils.completions import complete_target_modules
with pytest.raises(TypeError):
complete_target_modules(True, base=None) # type: ignore[arg-type]
def test_complete_target_modules_handles_unknown_base():
"""When base is set but we can't probe it, fall back to default modules."""
from soup_cli.utils.completions import complete_target_modules
suggestions = complete_target_modules("", base="some/nonexistent-model")
# Should still return non-empty default
assert len(suggestions) > 0
# ---------------------------------------------------------------------------
# CLI smoke
# ---------------------------------------------------------------------------
def test_cli_completions_help():
from soup_cli.cli import app
result = runner.invoke(app, ["completions", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_completions_bash():
from soup_cli.cli import app
result = runner.invoke(app, ["completions", "bash"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "soup" in result.stdout
def test_cli_completions_zsh():
from soup_cli.cli import app
result = runner.invoke(app, ["completions", "zsh"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_completions_fish():
from soup_cli.cli import app
result = runner.invoke(app, ["completions", "fish"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_completions_unknown_shell():
from soup_cli.cli import app
result = runner.invoke(app, ["completions", "tcsh"])
assert result.exit_code != 0
# ---------------------------------------------------------------------------
# Source-wiring regression
# ---------------------------------------------------------------------------
def test_no_heavy_top_level_imports():
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "completions.py"
text = src.read_text(encoding="utf-8")
import re
for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]:
assert not re.search(bad, text, re.MULTILINE)

325
tests/test_v0640_part_f.py Normal file
View File

@ -0,0 +1,325 @@
"""v0.64.0 Part F — License advisor tests."""
from __future__ import annotations
import dataclasses
import pytest
from typer.testing import CliRunner
runner = CliRunner()
def test_module_imports():
from soup_cli.utils import license_advisor
assert hasattr(license_advisor, "DEPLOY_TARGETS")
assert hasattr(license_advisor, "validate_deploy_target")
assert hasattr(license_advisor, "LicenseRecommendation")
assert hasattr(license_advisor, "advise_license_for_target")
assert hasattr(license_advisor, "flag_downstream_risk")
assert hasattr(license_advisor, "DownstreamRisk")
# ---------------------------------------------------------------------------
# DEPLOY_TARGETS
# ---------------------------------------------------------------------------
def test_deploy_targets_exact():
from soup_cli.utils.license_advisor import DEPLOY_TARGETS
assert "b2c" in DEPLOY_TARGETS
assert "defense" in DEPLOY_TARGETS
assert "embedded" in DEPLOY_TARGETS
def test_deploy_targets_is_frozenset():
from soup_cli.utils.license_advisor import DEPLOY_TARGETS
assert isinstance(DEPLOY_TARGETS, frozenset)
# ---------------------------------------------------------------------------
# validate_deploy_target
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("v", ["b2c", "defense", "embedded"])
def test_validate_deploy_target_happy(v):
from soup_cli.utils.license_advisor import validate_deploy_target
assert validate_deploy_target(v) == v
def test_validate_deploy_target_case_insensitive():
from soup_cli.utils.license_advisor import validate_deploy_target
assert validate_deploy_target("B2C") == "b2c"
@pytest.mark.parametrize(
"bad",
[True, False, None, "", "consumer", "x" * 33],
)
def test_validate_deploy_target_rejects(bad):
from soup_cli.utils.license_advisor import validate_deploy_target
with pytest.raises((TypeError, ValueError)):
validate_deploy_target(bad)
def test_validate_deploy_target_null_byte():
from soup_cli.utils.license_advisor import validate_deploy_target
with pytest.raises(ValueError, match="null"):
validate_deploy_target("b\x002c")
# ---------------------------------------------------------------------------
# LicenseRecommendation
# ---------------------------------------------------------------------------
def test_license_recommendation_frozen():
from soup_cli.utils.license_advisor import LicenseRecommendation
r = LicenseRecommendation(
target="b2c",
recommended_licenses=("apache-2.0", "mit"),
forbidden_licenses=("cc-by-nc-4.0",),
reason="permissive only",
)
with pytest.raises(dataclasses.FrozenInstanceError):
r.target = "other" # type: ignore[misc]
def test_license_recommendation_tuples():
from soup_cli.utils.license_advisor import LicenseRecommendation
# Lists rejected (frozen=True doesn't make lists immutable)
with pytest.raises(TypeError, match="tuple"):
LicenseRecommendation(
target="b2c",
recommended_licenses=["apache-2.0"], # type: ignore[arg-type]
forbidden_licenses=(),
reason="x",
)
# ---------------------------------------------------------------------------
# advise_license_for_target
# ---------------------------------------------------------------------------
def test_advise_license_b2c():
from soup_cli.utils.license_advisor import advise_license_for_target
rec = advise_license_for_target("b2c")
# B2C must include broadly-permissive options
assert "apache-2.0" in rec.recommended_licenses
assert "mit" in rec.recommended_licenses
# Non-commercial licenses must be forbidden for B2C
assert "cc-by-nc-4.0" in rec.forbidden_licenses
def test_advise_license_defense():
from soup_cli.utils.license_advisor import advise_license_for_target
rec = advise_license_for_target("defense")
# Defense forbids restricted-use community licenses
assert "llama-3.1" in rec.forbidden_licenses or "llama-2" in rec.forbidden_licenses
def test_advise_license_embedded():
from soup_cli.utils.license_advisor import advise_license_for_target
rec = advise_license_for_target("embedded")
# Embedded systems should avoid strong copyleft
assert "gpl-3.0" in rec.forbidden_licenses
def test_advise_license_rejects_unknown_target():
from soup_cli.utils.license_advisor import advise_license_for_target
with pytest.raises(ValueError):
advise_license_for_target("bogus")
def test_advise_license_rejects_bool():
from soup_cli.utils.license_advisor import advise_license_for_target
with pytest.raises(TypeError):
advise_license_for_target(True) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# DownstreamRisk
# ---------------------------------------------------------------------------
def test_downstream_risk_frozen():
from soup_cli.utils.license_advisor import DownstreamRisk
r = DownstreamRisk(ok=True, severity="ok", reason="permissive license")
with pytest.raises(dataclasses.FrozenInstanceError):
r.ok = False # type: ignore[misc]
def test_downstream_risk_severity_allowlist():
from soup_cli.utils.license_advisor import DownstreamRisk
with pytest.raises(ValueError, match="severity"):
DownstreamRisk(ok=False, severity="weird", reason="x")
# ---------------------------------------------------------------------------
# flag_downstream_risk
# ---------------------------------------------------------------------------
def test_flag_downstream_risk_apache_b2c_ok():
from soup_cli.utils.license_advisor import flag_downstream_risk
r = flag_downstream_risk(license_id="apache-2.0", target="b2c", monthly_active_users=10_000)
assert r.ok is True
assert r.severity == "ok"
def test_flag_downstream_risk_llama_community_high_mau():
"""Llama community license + commercial use > 700M MAU is the canonical risk."""
from soup_cli.utils.license_advisor import flag_downstream_risk
r = flag_downstream_risk(
license_id="llama-3", target="b2c", monthly_active_users=800_000_000
)
assert r.ok is False
assert r.severity in ("warn", "block")
def test_flag_downstream_risk_llama_community_low_mau_warns():
"""Llama community on small B2C: warning, not block."""
from soup_cli.utils.license_advisor import flag_downstream_risk
r = flag_downstream_risk(
license_id="llama-3", target="b2c", monthly_active_users=10_000
)
# Either OK with note or a warn — not a hard block
assert r.severity in ("ok", "warn")
def test_flag_downstream_risk_nc_b2c_blocked():
from soup_cli.utils.license_advisor import flag_downstream_risk
r = flag_downstream_risk(
license_id="cc-by-nc-4.0", target="b2c", monthly_active_users=100
)
assert r.ok is False
def test_flag_downstream_risk_rejects_negative_mau():
from soup_cli.utils.license_advisor import flag_downstream_risk
with pytest.raises(ValueError, match="monthly_active_users"):
flag_downstream_risk(
license_id="apache-2.0", target="b2c", monthly_active_users=-1
)
def test_flag_downstream_risk_rejects_bool_mau():
from soup_cli.utils.license_advisor import flag_downstream_risk
with pytest.raises(TypeError, match="bool"):
flag_downstream_risk(
license_id="apache-2.0", target="b2c",
monthly_active_users=True, # type: ignore[arg-type]
)
def test_flag_downstream_risk_unknown_license_warns():
from soup_cli.utils.license_advisor import flag_downstream_risk
r = flag_downstream_risk(
license_id="unknown-soup-test-license", target="b2c", monthly_active_users=1
)
# Unknown license should not block silently — surface as a warn
assert r.severity in ("warn", "block")
# ---------------------------------------------------------------------------
# CLI smoke
# ---------------------------------------------------------------------------
def test_cli_license_advisor_help():
from soup_cli.cli import app
result = runner.invoke(app, ["license-advisor", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_license_advisor_b2c():
from soup_cli.cli import app
result = runner.invoke(app, ["license-advisor", "--target", "b2c"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_license_advisor_unknown_target():
from soup_cli.cli import app
result = runner.invoke(app, ["license-advisor", "--target", "bogus"])
assert result.exit_code != 0
def test_cli_license_advisor_check_license():
"""--license is the per-license-id risk-check mode."""
from soup_cli.cli import app
result = runner.invoke(
app,
[
"license-advisor",
"--target",
"b2c",
"--license",
"apache-2.0",
"--monthly-active-users",
"10000",
],
)
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_cli_license_advisor_check_llama_block():
from soup_cli.cli import app
result = runner.invoke(
app,
[
"license-advisor",
"--target",
"b2c",
"--license",
"llama-3",
"--monthly-active-users",
"800000000",
],
)
# Either exits non-zero on block, OR prints a block message
assert result.exit_code != 0 or "block" in result.output.lower()
# ---------------------------------------------------------------------------
# Source-wiring regression
# ---------------------------------------------------------------------------
def test_no_heavy_top_level_imports():
from pathlib import Path
src = Path(__file__).resolve().parent.parent / "soup_cli" / "utils" / "license_advisor.py"
text = src.read_text(encoding="utf-8")
import re
for bad in ["^import torch", "^from torch", "^import transformers", "^from transformers"]:
assert not re.search(bad, text, re.MULTILINE)