diff --git a/README.md b/README.md index 6e116b0..419dc4b 100644 --- a/README.md +++ b/README.md @@ -42,14 +42,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.66.0 — Post-train X-rays: SAE feature diff + live influence-function blame + sleeper-agent defection probe + adapter interference matrix + probe pack.** Five surfaces extend `soup diagnose` from 6 failure modes to 10 — the post-train inspection layer no hosted vendor ships because mechanistic interpretability is research-grade and costs scale with N (so SaaS unit economics break). Closes the v0.57 `NotImplementedError` blame stub (#171). +**v0.67.0 — Adapter Lifecycle Finish: CMA-ES evolutionary merge + VeRA/VB-LoRA bank storage + MoLE per-token routing + GitHub-shaped adapter PRs + `soup.lock` + `soup adapters bisect`.** Six surfaces finish what v0.57 started — turning adapter management into a real version-control + collaboration workflow. None of these exist in hosted vendors: Sakana-style evolutionary merge is research demo only, VeRA storage hurts hosted unit economics (price by GPU-hour not adapter count), MoLE routing requires both training + serving stacks, adapter PRs need weights + eval + history together. -- **`soup probe sae-diff `** — Sparse-Autoencoder feature attribution. Pure-numpy math: ReLU(activations @ W_enc + b_enc) → mean post-pre diff per feature → top-K changes. Closed `HF_HUB_ALLOWLIST` over 8 known SAE families (Gemma Scope / Pythia / SAE-Lens Llama / GPT-2-small). Containment + `O_NOFOLLOW` open + 64-tensor cap. Composes with `soup adapters diff` (extends "what changed" from weight space to feature space). -- **`soup adapters blame --top-k 50`** — Live DataInf-style influence runner closes v0.57 #171. Per-row score = `cos(grad_row, grad_probe) × |grad_row|`; operator supplies `probe_fn` returning `(row_grads, probe_grad)` or falls back to a deterministic synthetic probe (matches v0.54 advise stub policy — surface always returns a real `BlameResult`, never `NotImplementedError`). Symlink-rejected via `O_NOFOLLOW` on the dataset read (TOCTOU defence); `_DEFAULT_SYNTH_PROBE_CAP=100_000` so a 10M-row dataset doesn't allocate 1.28 GB. -- **`soup probe sleeper [--evidence ev.json]`** — Anthropic-style calibrated linear probe per base. 6 bundled bases (Llama-2/3 8B, Mistral-7B, Qwen2-7B, Gemma-2 2B/9B); deterministic SHA-256-keyed synthetic weights ship today (real calibrated weights in v0.66.x). OK/MINOR/MAJOR thresholds at 1% / 5% defection rate. Exit 2 on MAJOR for CI gating. Without `--evidence`, prints probe metadata + neutral OK. -- **`soup probe interference `** — Pairwise N×N adapter interference matrix. Per-pair score = `(loss(A | A+B) - loss(A | A_alone)) / loss(A_alone)`; classify_interference at 5% / 20% bands. Surfaces which adapter pairs you cannot deploy together via `soup serve --adapters`. Rich-markup-escape on adapter names (review H2 fix — defends against crafted names like `[link=evil]X[/]`). MAJOR worst-pair → exit 2. -- **`soup probe pack `** — Manifest assembler that ties Parts A + C + D together. Per-base bundle of probe entries (sleeper / sae / truth / harm — last two reserved for v0.66.x); `MappingProxyType`-wrapped registry; `_LOWER_INDEX` for O(1) case-insensitive lookup. -- **+259 new tests** (10577 → 10836). Review-fix coverage across 3 sequential waves: 0 CRITICAL + 9 HIGH (TypeError on bool/non-str verdict; `O_NOFOLLOW` probe-open on `load_sae_weights` + `_count_dataset_rows` closing TOCTOU race; `hashlib.sha256` replaces process-salted `hash()` for deterministic probe weights across Python processes; non-numeric loss rejection in `probe interference` CLI; Rich-markup escape on adapter / verdict / description in `render_*_markdown`) + 14 MEDIUM (64-bit seed via `digest[:16]`; 10M-row hard rejection (no silent truncate); `_DEFAULT_SYNTH_PROBE_CAP=100_000` synthetic probe cap; description 4096-char cap on `ProbeEntry`; `_LOWER_INDEX` `MappingProxyType`; `collections.abc.Mapping` migration; `frozenset[str]` type params) + 5 LOW (`FrozenInstanceError` instead of `pytest.raises((AttributeError, Exception))`; basename(normpath(...)) symmetry; top-K comment clarification; `Optional[str]` on output flag; reserved-kind docstring note). Manual CPU smokes for every new command incl. failure modes (unknown base exit 2 / non-numeric loss exit 2 / MAJOR interference exit 2 / live blame produces a real `BlameResult`). +- **`soup adapters merge --strategy cmaes --eval --budget 1h`** — Sakana-style evolutionary search over LoRA merge weights. Pure-Python rank-mu CMA-ES (no `cma` dependency). Softmaxes N-1 logits onto the simplex, samples a population, keeps elite half, plateau-detects (3 generations without improvement → `converged=True`). Operator-supplied `eval_fn` closure; failures swallowed with sentinel score so one broken eval doesn't crash the run. Budget bounds reused from v0.57 `blame.parse_budget` (60s..24h). Live auto-wiring of the eval suite is deferred to v0.67.1; the CLI prints the validated plan today. +- **`soup_cli.utils.vector_bank`** — VeRA / VB-LoRA storage format. `VectorBank = {shared random projection matrix P, per-user scaling vector v_u}` — thousands of per-user adapters at MB-each instead of hundreds-of-MB per LoRA. Atomic JSON I/O via shared `paths.atomic_write_text` + cwd containment + symlink rejection + 16 MiB cap. `estimate_bank_size(num_users, vector_dim)` for sizing. Live multi-tenant serving via v0.22 multi-adapter surface lands in v0.67.1. +- **`task='moe_lora_routing'`** — MoLE per-token gating over N task LoRAs (Mixture of LoRA Experts). New `MoleGatingConfig` (num_task_adapters [2,64] / hidden_dim / temperature / top_k); cross-validator rejects mlx backend. Live gating-kernel training + per-token softmax routing lands in v0.67.1. +- **`soup adapters pr --base-sha <hex> --adapter <path> --eval <json>`** — GitHub-shaped PR rendering. PR = {base SHA, dataset diff, adapter weights, eval-delta report} → review-friendly Markdown with eval-delta tables + per-sample baseline/candidate diffs. `_md_table_escape` neutralises `\\` `|` `\n` `\r` `\t` in cells. JSON output for downstream consumers (v0.68 GitHub Action). Composes with `soup adapters diff`. +- **`soup lock write / show / check`** — Shared run lockfile. Closure of `(base_model_sha, dataset_sha, env_hash)` → committed to git so teams coordinate on reproducible training runs. `soup lock check` exits 3 on drift; `soup_version` + `created_at` are advisory-only (legitimate operator upgrades don't trigger drift). Composes with v0.64 `soup env lock`. +- **`soup adapters bisect <ckpt1> <ckpt2> ... --eval-command "..."`** — Binary search over training history. Operator supplies a shell template with `{ckpt}` placeholder (argv-list mode via `shlex.split` after `shlex.quote(ckpt)` — no `shell=True`). Probes both endpoints first (short-circuits all-OK / all-broken), then ~log₂(n) midpoint probes. Exit 3 on BROKEN_AT. Composes with v0.66 influence-blame for fine-grained attribution. +- **+165 new tests** (10836 → 11021) across 7 new test files. Review-fix coverage across 2 sequential waves: 0 CRITICAL + 0 HIGH (project policies already enforced by source-grep guards inherited from v0.66) + ~5 MEDIUM (plateau-convergence test; vector-bank forward-compat for unknown JSON fields; PR empty-sections renders without `None` leakage; soup.lock `soup_version` drift is advisory-only; bisect non-monotonic eval_fn does not crash) + ~3 LOW (exact MAX-boundary acceptance tests; `validate_user_id` bool rejection; source-grep regression for `atomic_write_text` usage + no `shell=True` + `shlex.quote` in bisect subprocess call). Manual CPU smokes for `soup lock write/show/check` (incl. drift exit 3), `soup adapters pr` (markdown + JSON output), `soup adapters bisect --plan-only`, `soup adapters merge --strategy cmaes` (plan-only output), plus 4 failure-mode rejection paths (cmaes-without-eval / unknown-strategy / mlx + moe_lora_routing / lock drift). ## Why Soup? @@ -192,6 +193,55 @@ soup probe pack meta-llama/Llama-3-8B # render the per-base manifest Every probe uses the OK / MINOR / MAJOR taxonomy from v0.26 (Quant-Lobotomy) / v0.56 (Diagnose) / v0.65 (Eval Depth). Sleeper + interference exit 2 on MAJOR for CI gating. The blame runner closes the v0.57 `NotImplementedError` stub via a DataInf-style influence approximation: `cos(grad_row, grad_probe) × |grad_row|`. Operators supply a `probe_fn` returning `(row_grads, probe_grad)`, or the runner falls back to a deterministic synthetic probe so the surface always returns a real `BlameResult` (no exception leaks). SAE feature diff is pure-numpy; the safetensors loader is `O_NOFOLLOW`-protected (TOCTOU defence — closes the symlink swap window between containment check and read). +## Adapter Lifecycle (`soup adapters {merge,pr,bisect}`, `soup lock`) + +v0.57 shipped `adapters diff / merge / blame / branch`. v0.67 finishes the lifecycle: evolutionary merge driven by your eval, GitHub-shaped PRs for adapter review, a shared `soup.lock` for team reproducibility, and binary-search bisect over training history. + +```bash +# 1. Evolutionary merge: search the simplex of merge weights via CMA-ES. +soup adapters merge \ + adapter-finance/ adapter-medical/ adapter-legal/ \ + --strategy cmaes \ + --eval evals/domain_mix.yaml \ + --budget 1h \ + --population 8 \ + --max-generations 20 \ + --output merged/ + +# 2. Render the merge as a GitHub PR for review (eval deltas + sample diffs). +soup adapters pr "merge: 3-domain blend" \ + --base-sha $(git rev-parse HEAD) \ + --adapter merged/ \ + --eval evals/deltas.json \ + --samples evals/samples.json \ + --dataset-diff data/diff.txt \ + --format markdown -o pr.md + +# 3. Lock a reproducible run state. Closure = sha(base + dataset + env). +soup env lock # v0.64 — capture env hash +soup lock write \ + --base-model meta-llama/Llama-3.1-8B \ + --base-sha $BASE_SHA \ + --dataset-sha $DATA_SHA \ + --env-hash $(jq -r .closure soup-env.lock) \ + -o soup.lock +# Teammates re-check the lock; exit 3 on drift. +soup lock check soup.lock \ + --base-model meta-llama/Llama-3.1-8B \ + --base-sha $BASE_SHA --dataset-sha $DATA_SHA --env-hash $ENV_HASH + +# 4. Bisect a training history to find the step that broke an eval. +soup adapters bisect \ + ckpt-step-100 ckpt-step-200 ckpt-step-400 ckpt-step-800 \ + --eval-command "soup eval custom --model {ckpt} --tasks eval.jsonl" \ + -o bisect.json +# Exits 3 on BROKEN_AT — pipe into `soup adapters blame` for attribution. +``` + +CMA-ES is pure-Python (no `cma` dependency); the eval is operator-supplied via a closure so any scoring code works. PR rendering escapes Markdown table cells, so crafted metric names cannot inject table rows or links. The lockfile composes with v0.64 `soup env lock` — drift in any of `{base_model, base_model_sha, dataset_sha, env_hash, closure_sha}` exits 3 (`soup_version` and `created_at` are advisory-only). Bisect uses `shlex.split` + `shlex.quote(ckpt)` in argv-list mode (no `shell=True`), so checkpoint ids cannot inject shell metacharacters. + +VeRA / VB-LoRA bank storage (`soup_cli.utils.vector_bank`) and MoLE per-token routing (`task='moe_lora_routing'`) ship as schema-only in v0.67.0 — live multi-tenant serving and gating-kernel training land in v0.67.1. + ## Data Flywheel (`soup loop`) The full *production traces → preference pairs → Eval-Gated DPO → canary deploy → rollback* loop, driven from a single CLI. Connects v0.26 Trace-to-Preference + Eval-Gated Training + Registry lineage + Quant-Lobotomy verdicts + Soup Cans + v0.25 Autopilot + v0.54 Advise + v0.55 Eval Design + v0.56 Diagnose. diff --git a/pyproject.toml b/pyproject.toml index a0ff083..fb402bd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.66.0" +version = "0.67.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index a9f52ae..79846c6 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.66.0" +__version__ = "0.67.0" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index e815056..e1181ff 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -408,6 +408,15 @@ app.add_typer( ), ) +# v0.67.0 Part E — soup.lock shared run lockfile. +from soup_cli.commands import lock as _lock_cmd # noqa: E402 + +app.add_typer( + _lock_cmd.app, + name="lock", + help="Shared run lockfile (write / show / check) - v0.67.0 Part E.", +) + def _rewrite_advise_argv(argv: list) -> list: """Inject `run` between `advise` and a non-subcommand first argument. diff --git a/soup_cli/commands/adapters.py b/soup_cli/commands/adapters.py index 7406fe7..9bd4326 100644 --- a/soup_cli/commands/adapters.py +++ b/soup_cli/commands/adapters.py @@ -1,6 +1,7 @@ """soup adapters — LoRA adapter management (list, info, compare, diff, merge, blame, branch).""" import json +import os from pathlib import Path from typing import Optional @@ -332,13 +333,32 @@ def merge( adapters: list[str] = typer.Argument(..., help="Two or more adapter paths to merge"), output: str = typer.Option(..., "--output", "-o", help="Output directory for merged adapter"), strategy: str = typer.Option("linear", "--strategy", - help="linear | ties | dare | svd"), + help="linear | ties | dare | svd | cmaes"), weights: str = typer.Option(None, "--weights", help="Comma-separated weights (default: equal)"), density: float = typer.Option(0.2, "--density", help="Trim density for ties/dare in (0, 1]"), - seed: int = typer.Option(0, "--seed", help="Random seed for dare"), + seed: int = typer.Option(0, "--seed", help="Random seed for dare / cmaes"), rank: int = typer.Option(None, "--rank", help="SVD rank (svd strategy only)"), + eval_suite: Optional[str] = typer.Option( + None, "--eval", + help=( + "Path to eval suite (required for --strategy cmaes). " + "Used by the evolutionary loop to score candidate merges." + ), + ), + budget: str = typer.Option( + "1h", "--budget", + help="Wall-clock budget for cmaes (60s..24h, e.g. 1h, 30m)", + ), + population: int = typer.Option( + 8, "--population", min=2, max=256, + help="cmaes population size per generation", + ), + max_generations: int = typer.Option( + 20, "--max-generations", min=1, max=10_000, + help="cmaes generation cap", + ), license_ids: list[str] = typer.Option( None, "--license", help=( @@ -373,6 +393,48 @@ def merge( console.print("[red]Need at least 2 adapter paths to merge[/]") raise typer.Exit(2) + # v0.67.0 Part A: cmaes evolutionary search requires an eval suite + + # budget. Render the planned strategy and emit a deferred-live advisory. + # Operator-supplied eval_fn lives in `soup_cli.utils.cmaes_merge.run_cmaes_merge`; + # the CLI surface here ships the plan + validation. Live `soup eval` + # auto-wiring against a registry run is deferred to v0.67.1. + if strategy == "cmaes": + if eval_suite is None: + console.print( + "[red]--strategy cmaes requires --eval <suite> " + "(path to a YAML/JSONL eval).[/]" + ) + raise typer.Exit(2) + from soup_cli.utils.cmaes_merge import build_cmaes_plan + + try: + cmaes_plan = build_cmaes_plan( + adapters=adapters, + eval_suite=eval_suite, + budget_spec=budget, + population_size=population, + max_generations=max_generations, + seed=seed, + ) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]Invalid cmaes plan: {escape(str(exc))}[/]") + raise typer.Exit(2) from exc + panel = Panel( + f"Strategy: [bold]cmaes[/]\n" + f"Adapters: {len(cmaes_plan.adapters)}\n" + f"Eval suite: [bold]{escape(os.path.basename(cmaes_plan.eval_suite))}[/]\n" + f"Budget: [bold]{cmaes_plan.budget_seconds}s[/]\n" + f"Population: [bold]{cmaes_plan.population_size}[/]\n" + f"Max generations: [bold]{cmaes_plan.max_generations}[/]\n" + f"Live cmaes loop with auto-wired eval suite is deferred to " + f"[yellow]v0.67.1[/]. The plan is validated and ready to " + f"hand to an operator-supplied `eval_fn` via " + f"`run_cmaes_merge(plan, eval_fn=...)`.", + title="Adapter merge — cmaes (plan)", + ) + console.print(panel) + return + # v0.60.0 Part E: license-conflict gate. Operators MUST declare a # license per adapter (or pass --license-override <reason>). if license_ids: @@ -837,3 +899,254 @@ def check_safetensors( for path in report.unsafe_files: console.print(f" [yellow]- {escape(path)}[/]") raise typer.Exit(1) + + +@app.command(name="pr") +def adapter_pr( + title: str = typer.Argument(..., help="Short PR title (e.g. 'add-support-tone')"), + base_sha: str = typer.Option(..., "--base-sha", help="64-hex SHA of the base model"), + adapter_path: str = typer.Option(..., "--adapter", help="Path to candidate adapter"), + eval_json: Optional[str] = typer.Option( + None, "--eval", + help=( + "Path to JSON eval-deltas: list of " + '{"metric": ..., "baseline": ..., "candidate": ...}' + ), + ), + samples_json: Optional[str] = typer.Option( + None, "--samples", + help=( + "Path to JSON sample diffs: list of " + '{"prompt": ..., "baseline_output": ..., "candidate_output": ...}' + ), + ), + dataset_diff_path: Optional[str] = typer.Option( + None, "--dataset-diff", + help="Path to a text file containing the dataset diff", + ), + output: Optional[str] = typer.Option( + None, "--output", "-o", + help="Write rendered Markdown to path (default: stdout)", + ), + format_: str = typer.Option( + "markdown", "--format", "-f", help="markdown | json", + ), +): + """Render a GitHub-style PR for an adapter (v0.67.0 Part D). + + The PR = ``{base SHA, dataset diff, adapter file, eval report}`` + rendered as a Markdown document with eval-delta tables and sample + diffs, ready to drop into a GitHub PR description or comment. + """ + from soup_cli.utils.adapter_pr import ( + build_adapter_pr, + render_pr_json, + render_pr_markdown, + write_pr_markdown, + ) + from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + + if format_ not in ("markdown", "json"): + console.print(f"[red]Unknown --format: {escape(format_)}[/]") + raise typer.Exit(2) + + # Load deltas + samples + dataset_diff lazily; each accepts None. + def _load_json_list(path: Optional[str], field: str) -> list: + if path is None: + return [] + enforce_under_cwd_and_no_symlink(path, field=field) + with open(os.path.realpath(path), encoding="utf-8") as fh: + raw = json.load(fh) + if not isinstance(raw, list): + raise ValueError(f"{field} must contain a JSON list") + return raw + + try: + deltas = _load_json_list(eval_json, "eval") + samples = _load_json_list(samples_json, "samples") + dataset_diff = "" + if dataset_diff_path is not None: + enforce_under_cwd_and_no_symlink( + dataset_diff_path, field="dataset_diff" + ) + with open( + os.path.realpath(dataset_diff_path), encoding="utf-8" + ) as fh: + dataset_diff = fh.read() + pr = build_adapter_pr( + title=title, + base_sha=base_sha, + adapter_path=adapter_path, + dataset_diff=dataset_diff, + deltas=deltas, + samples=samples, + ) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + if format_ == "json": + rendered = render_pr_json(pr) + else: + rendered = render_pr_markdown(pr) + + if output is None: + console.print(rendered) + return + + try: + if format_ == "markdown": + write_pr_markdown(pr, output) + else: + from soup_cli.utils.paths import atomic_write_text + + atomic_write_text(rendered, output, field="pr output") + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + Panel( + f"PR: [bold]{escape(pr.title)}[/]\n" + f"Format: [bold]{escape(format_)}[/]\n" + f"Output: [bold]{escape(output)}[/]", + title="Adapter PR rendered", + ) + ) + + +@app.command(name="bisect") +def adapter_bisect( + history: list[str] = typer.Argument( + ..., + help=( + "Ordered checkpoint identifiers (oldest first). At least 2. " + "Example: `soup adapters bisect ckpt-100 ckpt-200 ckpt-300 ckpt-400`" + ), + ), + eval_command: str = typer.Option( + ..., + "--eval-command", + help=( + "Shell command template; '{ckpt}' is replaced with each " + "checkpoint id. Exit code 0 means PASS, non-zero means FAIL. " + "Example: 'soup eval custom --model {ckpt} --tasks tasks.jsonl'" + ), + ), + plan_only: bool = typer.Option( + False, "--plan-only", + help="Print the plan and exit without running the bisect", + ), + output: Optional[str] = typer.Option( + None, "--output", "-o", + help="Write JSON result to path", + ), +): + """Binary-search a training history to find the first failing checkpoint. + + Composes with v0.66 Part B influence-blame: once the boundary is + found, ``soup adapters blame`` can attribute the regression to + specific dataset rows. + """ + from soup_cli.utils.adapter_bisect import build_bisect_plan, run_bisect + + if not isinstance(history, list) or len(history) < 2: + console.print("[red]Need at least 2 checkpoint ids[/]") + raise typer.Exit(2) + + try: + plan = build_bisect_plan(history) + except (TypeError, ValueError) as exc: + console.print(f"[red]Invalid history: {escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + if "{ckpt}" not in eval_command: + console.print( + "[red]--eval-command must include `{ckpt}` placeholder[/]" + ) + raise typer.Exit(2) + + if plan_only: + console.print( + Panel( + f"History: [bold]{len(plan.history)} checkpoints[/]\n" + f"Eval: [bold]{escape(eval_command)}[/]\n" + f"Probes: ~[bold]{_estimated_probes(len(plan.history))}[/] iterations", + title="Adapter bisect (plan)", + ) + ) + return + + # Execute via subprocess; exit 0 means OK. + import shlex + import subprocess # noqa: S404 — argv list mode, no shell + + def _eval_fn(ckpt: str) -> bool: + cmd_str = eval_command.replace("{ckpt}", shlex.quote(ckpt)) + # Safer: split via shlex and use argv mode (no shell interpolation). + argv = shlex.split(cmd_str) + try: + result = subprocess.run( # noqa: S603 + argv, capture_output=True, timeout=3600, check=False + ) + except (subprocess.TimeoutExpired, FileNotFoundError) as exc: + console.print( + f"[yellow]Eval failed for {escape(ckpt)}: " + f"{escape(str(exc))}[/]" + ) + return False + return result.returncode == 0 + + try: + result = run_bisect(plan, eval_fn=_eval_fn) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + if result.verdict == "ALL_OK": + console.print( + Panel( + f"Verdict: [green]ALL_OK[/]\n" + f"Probes: {result.probes}", + title="Adapter bisect", + ) + ) + else: + console.print( + Panel( + f"Verdict: [red]BROKEN_AT[/]\n" + f"First broken: [bold]{escape(result.first_broken or '?')}[/]\n" + f"Probes: {result.probes}", + title="Adapter bisect", + ) + ) + + if output is not None: + from soup_cli.utils.paths import atomic_write_text + + data = { + "verdict": result.verdict, + "first_broken": result.first_broken, + "probes": result.probes, + "steps": [ + {"checkpoint": s.checkpoint, "ok": s.ok} + for s in result.steps + ], + } + atomic_write_text( + json.dumps(data, indent=2, sort_keys=True), + output, + field="bisect output", + ) + + if result.verdict == "BROKEN_AT": + raise typer.Exit(2) + + +def _estimated_probes(n: int) -> int: + """Approximate number of probes for n checkpoints (~log2(n) + 2).""" + import math + + if n <= 2: + return n + return max(2, math.ceil(math.log2(n))) + 2 diff --git a/soup_cli/commands/lock.py b/soup_cli/commands/lock.py new file mode 100644 index 0000000..7d4c142 --- /dev/null +++ b/soup_cli/commands/lock.py @@ -0,0 +1,165 @@ +"""soup lock — shared run lockfile (v0.67.0 Part E). + +Subcommands: + +- ``soup lock write``: render a ``soup.lock`` from operator-supplied + base-model / dataset / env hashes. +- ``soup lock check``: compare a tracked ``soup.lock`` against a + freshly-computed closure; exit 3 on drift. +- ``soup lock show``: print a tracked lock. +""" + +from __future__ import annotations + +from datetime import datetime, timezone + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel + +console = Console() + +app = typer.Typer(no_args_is_help=True, help="Shared run lockfile (v0.67.0 Part E)") + + +@app.command(name="write") +def write_lock_cmd( + base_model: str = typer.Option(..., "--base-model", help="HF model id / path"), + base_sha: str = typer.Option(..., "--base-sha", help="64-hex base-model SHA"), + dataset_sha: str = typer.Option(..., "--dataset-sha", help="64-hex dataset SHA"), + env_hash: str = typer.Option(..., "--env-hash", help="64-hex env hash (soup env)"), + output: str = typer.Option("soup.lock", "--output", "-o", help="Output path"), +): + """Render a ``soup.lock`` from the three required hashes.""" + from soup_cli import __version__ + from soup_cli.utils.soup_lock import SoupLock, compute_lock_closure, write_lock + + try: + closure = compute_lock_closure( + base_model_sha=base_sha, + dataset_sha=dataset_sha, + env_hash=env_hash, + ) + lock = SoupLock( + soup_version=__version__, + base_model=base_model, + base_model_sha=base_sha, + dataset_sha=dataset_sha, + env_hash=env_hash, + closure_sha=closure, + created_at=datetime.now(timezone.utc).isoformat(), + ) + write_lock(lock, output) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + Panel( + f"Lock: [bold]{escape(output)}[/]\n" + f"Base: [bold]{escape(base_model)}[/]\n" + f"Closure SHA: [bold]{closure[:12]}…[/]", + title="soup.lock written", + ) + ) + + +@app.command(name="show") +def show_lock_cmd( + path: str = typer.Argument("soup.lock", help="Path to soup.lock"), +): + """Print a tracked lock file.""" + from soup_cli.utils.soup_lock import read_lock + + try: + lock = read_lock(path) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + Panel( + f"Soup version: [bold]{escape(lock.soup_version)}[/]\n" + f"Base model: [bold]{escape(lock.base_model)}[/]\n" + f"Base SHA: [bold]{lock.base_model_sha[:16]}…[/]\n" + f"Dataset SHA: [bold]{lock.dataset_sha[:16]}…[/]\n" + f"Env hash: [bold]{lock.env_hash[:16]}…[/]\n" + f"Closure SHA: [bold]{lock.closure_sha[:16]}…[/]\n" + f"Created at: [bold]{escape(lock.created_at)}[/]", + title=f"soup.lock — {escape(path)}", + ) + ) + + +@app.command(name="check") +def check_lock_cmd( + path: str = typer.Argument("soup.lock", help="Path to tracked soup.lock"), + base_sha: str = typer.Option(..., "--base-sha", help="64-hex current base-model SHA"), + dataset_sha: str = typer.Option(..., "--dataset-sha", help="64-hex current dataset SHA"), + env_hash: str = typer.Option(..., "--env-hash", help="64-hex current env hash"), + base_model: str = typer.Option(..., "--base-model", help="Current base model id"), +): + """Refuse with exit 3 if the lock has drifted from current state.""" + from soup_cli import __version__ + from soup_cli.utils.soup_lock import ( + SoupLock, + check_lock_drift, + compute_lock_closure, + read_lock, + ) + + try: + expected = read_lock(path) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + closure = compute_lock_closure( + base_model_sha=base_sha, + dataset_sha=dataset_sha, + env_hash=env_hash, + ) + # Use the existing soup_version + created_at from `expected` so the + # comparison stays content-only (drift only counts the 5 content + # fields per `check_lock_drift`). + actual = SoupLock( + soup_version=expected.soup_version, + base_model=base_model, + base_model_sha=base_sha, + dataset_sha=dataset_sha, + env_hash=env_hash, + closure_sha=closure, + created_at=expected.created_at, + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + drift = check_lock_drift(expected, actual) + if drift.ok: + console.print( + Panel( + f"Lock: [bold]{escape(path)}[/]\n" + f"Status: [green]OK[/] — closure matches", + title="soup lock check", + ) + ) + return + + console.print( + Panel( + f"Lock: [bold]{escape(path)}[/]\n" + f"Status: [red]DRIFT[/]", + title="soup lock check", + ) + ) + for change in drift.changes: + console.print(f" [red]- {escape(change)}[/]") + if __version__ != expected.soup_version: + console.print( + f"[yellow]Note: soup version changed " + f"({escape(expected.soup_version)} -> {escape(__version__)})[/]" + ) + raise typer.Exit(3) diff --git a/soup_cli/config/schema.py b/soup_cli/config/schema.py index d126cbe..b18bc56 100644 --- a/soup_cli/config/schema.py +++ b/soup_cli/config/schema.py @@ -2521,6 +2521,8 @@ class SoupConfig(BaseModel): "tts", "classifier", "reranker", "cross_encoder", "distill", # v0.61.0 Part A — Unlearning (NPO / SimNPO / RMU). "unlearn", + # v0.67.0 Part C — MoLE per-token adapter routing (Mixture of LoRA Experts). + "moe_lora_routing", ] = Field( default="sft", description=( @@ -2528,7 +2530,8 @@ class SoupConfig(BaseModel): "'tts' (TTS fine-tuning), 'classifier' / 'reranker' / " "'cross_encoder' (classification heads), and 'distill' " "(knowledge distillation). v0.61.0 adds 'unlearn' (NPO / " - "SimNPO / RMU)." + "SimNPO / RMU). v0.67.0 adds 'moe_lora_routing' (per-token " + "gating over N task LoRAs)." ), ) modality: Literal["text", "vision", "audio", "audio_out"] = Field( @@ -3700,6 +3703,27 @@ class SoupConfig(BaseModel): raise ValueError(str(exc)) from exc return self + @model_validator(mode="after") + def _validate_mole_routing_compat(self) -> "SoupConfig": + """v0.67.0 Part C — MoLE per-token routing cross-validator. + + Rules: + * ``task='moe_lora_routing'`` rejects ``backend='mlx'`` (live wiring + deferred to v0.67.1; the gating kernel needs torch dispatch). + * The full ``MoleGatingConfig`` validation lives in + ``soup_cli.utils.mole_routing.validate_mole_compat`` and is + exercised at trainer construction time (the v0.67.0 release ships + the schema lock-in; live training lands in v0.67.1). + """ + if self.task != "moe_lora_routing": + return self + if self.backend == "mlx": + raise ValueError( + "MoLE routing (task='moe_lora_routing') is not supported on " + "the mlx backend (live wiring deferred to v0.67.1)." + ) + return self + @model_validator(mode="after") def _validate_mlx_task_support(self) -> "SoupConfig": """MLX backend only supports sft, dpo, and grpo tasks (v0.25.0). diff --git a/soup_cli/utils/adapter_bisect.py b/soup_cli/utils/adapter_bisect.py new file mode 100644 index 0000000..8dd18ea --- /dev/null +++ b/soup_cli/utils/adapter_bisect.py @@ -0,0 +1,237 @@ +"""``soup adapters bisect`` — binary search over training history (v0.67.0 Part F). + +Given an ordered history of checkpoints (or dataset commits) and an +operator-supplied predicate that returns True when "eval passes", +binary-search for the FIRST checkpoint where the predicate flips to +False. The result is the regression boundary. + +Composes with v0.66 Part B influence-blame: once the boundary is +found, ``soup adapters blame`` can attribute the regression to +specific dataset rows. + +Public surface: + +- ``BisectPlan`` / ``BisectStep`` / ``BisectResult`` frozen dataclasses +- ``build_bisect_plan(history)`` factory +- ``bisect_next_step(plan, lo, hi)`` pure midpoint kernel +- ``run_bisect(plan, *, eval_fn)`` end-to-end binary search + +Closed verdict taxonomy: +- ``ALL_OK`` — every checkpoint passes (no regression found) +- ``BROKEN_AT`` — found first failing checkpoint +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Callable, Optional, Sequence, Tuple + +MAX_HISTORY = 4096 + + +# --------------------------------------------------------------------------- +# Validators +# --------------------------------------------------------------------------- + + +def _check_checkpoint(value: object, field: str) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} entries must be str") + if not value: + raise ValueError(f"{field} entry must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} entry must not contain null bytes") + if len(value) > 1024: + raise ValueError(f"{field} entry too long") + return value + + +# --------------------------------------------------------------------------- +# Frozen dataclasses +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class BisectPlan: + """An ordered history of checkpoint identifiers to bisect over. + + The order is "oldest first" — index 0 is the earliest checkpoint + and the last entry is the most recent. + """ + + history: Tuple[str, ...] + + def __post_init__(self) -> None: + if not isinstance(self.history, tuple): + raise TypeError("history must be tuple") + if len(self.history) < 2: + raise ValueError( + "history must contain at least 2 entries" + ) + if len(self.history) > MAX_HISTORY: + raise ValueError( + f"history length {len(self.history)} > {MAX_HISTORY}" + ) + seen: set[str] = set() + for entry in self.history: + _check_checkpoint(entry, "history") + if entry in seen: + raise ValueError( + f"history must be unique (duplicate {entry!r})" + ) + seen.add(entry) + + +@dataclass(frozen=True) +class BisectStep: + """One probe result during a bisect run.""" + + checkpoint: str + ok: bool + + def __post_init__(self) -> None: + _check_checkpoint(self.checkpoint, "checkpoint") + if not isinstance(self.ok, bool): + raise TypeError("ok must be bool") + + +@dataclass(frozen=True) +class BisectResult: + """Outcome of a bisect run. + + ``first_broken`` is ``None`` when ``verdict='ALL_OK'``; otherwise + it is the first failing checkpoint id. + """ + + first_broken: Optional[str] + verdict: str + steps: Tuple[BisectStep, ...] + probes: int + + def __post_init__(self) -> None: + valid_verdicts = ("ALL_OK", "BROKEN_AT") + if self.verdict not in valid_verdicts: + raise ValueError( + f"verdict must be one of {valid_verdicts}" + ) + if self.first_broken is not None: + _check_checkpoint(self.first_broken, "first_broken") + if not isinstance(self.steps, tuple): + raise TypeError("steps must be tuple") + for s in self.steps: + if not isinstance(s, BisectStep): + raise TypeError("steps entries must be BisectStep") + if isinstance(self.probes, bool) or not isinstance(self.probes, int): + raise TypeError("probes must be int") + if self.probes < 0: + raise ValueError("probes must be non-negative") + + +# --------------------------------------------------------------------------- +# Factory +# --------------------------------------------------------------------------- + + +def build_bisect_plan(history: Sequence[str]) -> BisectPlan: + """Build a frozen plan from a sequence of checkpoint ids.""" + if isinstance(history, str) or not isinstance(history, Sequence): + raise TypeError("history must be a non-string sequence") + return BisectPlan(history=tuple(history)) + + +# --------------------------------------------------------------------------- +# Pure midpoint kernel +# --------------------------------------------------------------------------- + + +def bisect_next_step(plan: BisectPlan, *, lo: int, hi: int) -> int: + """Return the midpoint index between lo and hi (inclusive). + + Pure: deterministic given the same inputs. Used internally by + ``run_bisect`` but exposed for callers that want to drive the + bisect loop themselves. + """ + if not isinstance(plan, BisectPlan): + raise TypeError("plan must be BisectPlan") + if isinstance(lo, bool) or not isinstance(lo, int): + raise TypeError("lo must be int") + if isinstance(hi, bool) or not isinstance(hi, int): + raise TypeError("hi must be int") + if lo < 0 or hi >= len(plan.history): + raise ValueError("lo/hi out of bounds") + if lo > hi: + raise ValueError("lo must be <= hi") + return (lo + hi) // 2 + + +# --------------------------------------------------------------------------- +# Orchestrator +# --------------------------------------------------------------------------- + + +def run_bisect( + plan: BisectPlan, + *, + eval_fn: Callable[[str], bool], +) -> BisectResult: + """Find the first failing checkpoint via binary search. + + ``eval_fn(checkpoint_id)`` MUST return ``True`` when the checkpoint + passes (no regression) and ``False`` when it fails. The bisect + assumes monotonic regression: once a checkpoint fails, every later + one also fails. Non-monotonic histories produce undefined results; + operators wanting a sweep should use ``soup eval`` directly. + """ + if not isinstance(plan, BisectPlan): + raise TypeError("plan must be BisectPlan") + if eval_fn is None or not callable(eval_fn): + raise TypeError("eval_fn must be callable") + + n = len(plan.history) + steps: list[BisectStep] = [] + + # Probe both endpoints first to short-circuit "all OK" / "all broken". + first = plan.history[0] + first_ok = bool(eval_fn(first)) + steps.append(BisectStep(checkpoint=first, ok=first_ok)) + if not first_ok: + # All broken from the very first checkpoint. + return BisectResult( + first_broken=first, + verdict="BROKEN_AT", + steps=tuple(steps), + probes=len(steps), + ) + + last = plan.history[-1] + last_ok = bool(eval_fn(last)) + steps.append(BisectStep(checkpoint=last, ok=last_ok)) + if last_ok: + return BisectResult( + first_broken=None, + verdict="ALL_OK", + steps=tuple(steps), + probes=len(steps), + ) + + # Now standard binary search: find the LOWEST index that fails. + # Invariant: plan.history[lo] passes; plan.history[hi] fails. + lo, hi = 0, n - 1 + while hi - lo > 1: + mid = (lo + hi) // 2 + ckpt = plan.history[mid] + ok = bool(eval_fn(ckpt)) + steps.append(BisectStep(checkpoint=ckpt, ok=ok)) + if ok: + lo = mid + else: + hi = mid + + return BisectResult( + first_broken=plan.history[hi], + verdict="BROKEN_AT", + steps=tuple(steps), + probes=len(steps), + ) diff --git a/soup_cli/utils/adapter_merge.py b/soup_cli/utils/adapter_merge.py index a9894da..8903123 100644 --- a/soup_cli/utils/adapter_merge.py +++ b/soup_cli/utils/adapter_merge.py @@ -25,11 +25,17 @@ from typing import Any, FrozenSet, Literal, Mapping, Sequence, Tuple from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink -MergeStrategy = Literal["linear", "ties", "dare", "svd"] +MergeStrategy = Literal["linear", "ties", "dare", "svd", "cmaes"] # frozenset for O(1) membership + immutability (mirrors v0.41.0 / v0.51.0 policy); # tuple alias preserved for caller code that iterates in canonical order. -SUPPORTED_STRATEGIES: FrozenSet[str] = frozenset({"linear", "ties", "dare", "svd"}) -STRATEGY_ORDER: Tuple[MergeStrategy, ...] = ("linear", "ties", "dare", "svd") +# v0.67.0 Part A: added "cmaes" — evolutionary search dispatched separately +# in commands/adapters.py (requires --eval suite + --budget). +SUPPORTED_STRATEGIES: FrozenSet[str] = frozenset( + {"linear", "ties", "dare", "svd", "cmaes"} +) +STRATEGY_ORDER: Tuple[MergeStrategy, ...] = ( + "linear", "ties", "dare", "svd", "cmaes" +) _MAX_ADAPTERS = 16 _MIN_ADAPTERS = 2 diff --git a/soup_cli/utils/adapter_pr.py b/soup_cli/utils/adapter_pr.py new file mode 100644 index 0000000..69ade86 --- /dev/null +++ b/soup_cli/utils/adapter_pr.py @@ -0,0 +1,345 @@ +"""GitHub-shaped adapter PR rendering (v0.67.0 Part D). + +A PR for an adapter is the triple ``{base SHA, dataset diff, adapter +weights}`` plus an eval-delta report. This module renders it as a +review-friendly Markdown document with eval-delta tables and +side-by-side sample-output diffs, suitable for posting as a GitHub +PR comment via the v0.68.0 GitHub Action. + +Public surface: + +- ``EvalDelta`` / ``SampleDiff`` / ``AdapterPR`` frozen dataclasses +- ``build_adapter_pr(...)`` factory from raw dicts (CLI / API friendly) +- ``render_pr_markdown(pr)`` returns review-ready Markdown +- ``render_pr_json(pr)`` returns JSON for downstream consumers +- ``write_pr_markdown(pr, path)`` atomic cwd-contained write +""" + +from __future__ import annotations + +import json +import math +import re +from dataclasses import asdict, dataclass +from typing import Any, Iterable, Mapping, Tuple + +from soup_cli.utils.paths import atomic_write_text + +# --------------------------------------------------------------------------- +# Bounds (closed) +# --------------------------------------------------------------------------- + +MAX_TITLE_LEN = 256 +MAX_METRIC_NAME_LEN = 256 +MAX_OUTPUT_LEN = 32_768 # per-output cap (defense + keeps PRs reviewable) +MAX_DATASET_DIFF_LEN = 1_048_576 # 1 MiB +MAX_DELTAS = 64 +MAX_SAMPLES = 256 +_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") + + +# --------------------------------------------------------------------------- +# Validators +# --------------------------------------------------------------------------- + + +def _check_text_field(value: object, field: str, max_len: int) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > max_len: + raise ValueError( + f"{field} length {len(value)} > {max_len}" + ) + return value + + +def _check_required_text(value: object, field: str, max_len: int) -> str: + text = _check_text_field(value, field, max_len) + if not text: + raise ValueError(f"{field} must be non-empty") + return text + + +def _check_finite(value: object, field: str) -> float: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, (int, float)): + raise TypeError(f"{field} must be numeric") + val = float(value) + if not math.isfinite(val): + raise ValueError(f"{field} must be finite") + return val + + +def _check_sha256(value: object, field: str) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str") + if not _SHA256_RE.match(value): + raise ValueError(f"{field} must be 64 hex chars") + return value + + +# --------------------------------------------------------------------------- +# Frozen dataclasses +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class EvalDelta: + """One metric: baseline -> candidate. + + The ``delta`` field is a computed property (post-init). + """ + + metric: str + baseline: float + candidate: float + + def __post_init__(self) -> None: + _check_required_text(self.metric, "metric", MAX_METRIC_NAME_LEN) + _check_finite(self.baseline, "baseline") + _check_finite(self.candidate, "candidate") + + @property + def delta(self) -> float: + return self.candidate - self.baseline + + +@dataclass(frozen=True) +class SampleDiff: + """One prompt with baseline + candidate outputs. + + Long outputs are rejected (not truncated) so PR authors notice and + pre-truncate intentionally — keeps the PR reviewable. + """ + + prompt: str + baseline_output: str + candidate_output: str + + def __post_init__(self) -> None: + _check_required_text(self.prompt, "prompt", MAX_OUTPUT_LEN) + _check_text_field( + self.baseline_output, "baseline_output", MAX_OUTPUT_LEN + ) + _check_text_field( + self.candidate_output, "candidate_output", MAX_OUTPUT_LEN + ) + + +@dataclass(frozen=True) +class AdapterPR: + """An adapter PR — the triple {base SHA, dataset diff, adapter path} + plus eval deltas + sample diffs for human review.""" + + title: str + base_sha: str + adapter_path: str + dataset_diff: str + deltas: Tuple[EvalDelta, ...] + samples: Tuple[SampleDiff, ...] + + def __post_init__(self) -> None: + _check_required_text(self.title, "title", MAX_TITLE_LEN) + _check_sha256(self.base_sha, "base_sha") + _check_required_text(self.adapter_path, "adapter_path", MAX_OUTPUT_LEN) + _check_text_field( + self.dataset_diff, "dataset_diff", MAX_DATASET_DIFF_LEN + ) + if not isinstance(self.deltas, tuple): + raise TypeError("deltas must be tuple") + if len(self.deltas) > MAX_DELTAS: + raise ValueError( + f"too many deltas ({len(self.deltas)} > {MAX_DELTAS})" + ) + for d in self.deltas: + if not isinstance(d, EvalDelta): + raise TypeError("deltas entries must be EvalDelta") + if not isinstance(self.samples, tuple): + raise TypeError("samples must be tuple") + if len(self.samples) > MAX_SAMPLES: + raise ValueError( + f"too many samples ({len(self.samples)} > {MAX_SAMPLES})" + ) + for s in self.samples: + if not isinstance(s, SampleDiff): + raise TypeError("samples entries must be SampleDiff") + + +# --------------------------------------------------------------------------- +# Factory +# --------------------------------------------------------------------------- + + +def build_adapter_pr( + *, + title: str, + base_sha: str, + adapter_path: str, + dataset_diff: str, + deltas: Iterable[Mapping[str, Any]], + samples: Iterable[Mapping[str, Any]], +) -> AdapterPR: + """Build a frozen ``AdapterPR`` from raw dicts (CLI / API friendly).""" + delta_objs = [] + for raw in deltas: + if not isinstance(raw, Mapping): + raise TypeError("each delta must be a mapping") + delta_objs.append( + EvalDelta( + metric=raw.get("metric", ""), + baseline=raw.get("baseline", 0.0), + candidate=raw.get("candidate", 0.0), + ) + ) + sample_objs = [] + for raw in samples: + if not isinstance(raw, Mapping): + raise TypeError("each sample must be a mapping") + sample_objs.append( + SampleDiff( + prompt=raw.get("prompt", ""), + baseline_output=raw.get("baseline_output", ""), + candidate_output=raw.get("candidate_output", ""), + ) + ) + return AdapterPR( + title=title, + base_sha=base_sha, + adapter_path=adapter_path, + dataset_diff=dataset_diff, + deltas=tuple(delta_objs), + samples=tuple(sample_objs), + ) + + +# --------------------------------------------------------------------------- +# Markdown rendering — escapes table-active and link-active characters +# --------------------------------------------------------------------------- + + +def _md_table_escape(text: str) -> str: + """Neutralise markdown table-active characters in a cell. + + Mirrors v0.29.0 model card v2 + v0.59.0 Annex XI ``_md_escape`` + policy: a forged-heading or table-injection attack via crafted + metric names / prompts must not leak into the rendered PR document. + """ + if not isinstance(text, str): + return "" + # Order matters: \\ first + out = text.replace("\\", "\\\\") + out = out.replace("|", "\\|") + out = out.replace("\n", " ") + out = out.replace("\r", " ") + out = out.replace("\t", " ") + return out + + +def _md_body_escape(text: str) -> str: + """Lighter escape for code-fence body: only null bytes + CR.""" + if not isinstance(text, str): + return "" + return text.replace("\x00", "").replace("\r\n", "\n") + + +def render_pr_markdown(pr: AdapterPR) -> str: + """Render an ``AdapterPR`` as a GitHub-style PR Markdown document.""" + if not isinstance(pr, AdapterPR): + raise TypeError("pr must be AdapterPR") + + lines: list[str] = [] + lines.append(f"# {_md_table_escape(pr.title)}") + lines.append("") + lines.append(f"**Base SHA:** `{pr.base_sha[:12]}…`") + lines.append(f"**Adapter:** `{_md_table_escape(pr.adapter_path)}`") + lines.append("") + + # Eval deltas table + if pr.deltas: + lines.append("## Eval deltas") + lines.append("") + lines.append("| Metric | Baseline | Candidate | Δ |") + lines.append("|---|---|---|---|") + for d in pr.deltas: + metric = _md_table_escape(d.metric) + sign = "+" if d.delta >= 0 else "" + lines.append( + f"| {metric} | {d.baseline:.4f} | {d.candidate:.4f} | " + f"{sign}{d.delta:.4f} |" + ) + lines.append("") + + # Dataset diff + if pr.dataset_diff: + lines.append("## Dataset diff") + lines.append("") + lines.append("```diff") + lines.append(_md_body_escape(pr.dataset_diff)) + lines.append("```") + lines.append("") + + # Sample diffs + if pr.samples: + lines.append("## Sample diffs") + lines.append("") + for idx, s in enumerate(pr.samples, 1): + lines.append(f"### Sample {idx}") + lines.append("") + lines.append("**Prompt:**") + lines.append("") + lines.append("```") + lines.append(_md_body_escape(s.prompt)) + lines.append("```") + lines.append("") + lines.append("**Baseline:**") + lines.append("") + lines.append("```") + lines.append(_md_body_escape(s.baseline_output)) + lines.append("```") + lines.append("") + lines.append("**Candidate:**") + lines.append("") + lines.append("```") + lines.append(_md_body_escape(s.candidate_output)) + lines.append("```") + lines.append("") + + return "\n".join(lines) + + +def render_pr_json(pr: AdapterPR) -> str: + """Render an ``AdapterPR`` as JSON (for downstream API consumers).""" + if not isinstance(pr, AdapterPR): + raise TypeError("pr must be AdapterPR") + data = { + "title": pr.title, + "base_sha": pr.base_sha, + "adapter_path": pr.adapter_path, + "dataset_diff": pr.dataset_diff, + "deltas": [ + { + "metric": d.metric, + "baseline": d.baseline, + "candidate": d.candidate, + "delta": d.delta, + } + for d in pr.deltas + ], + "samples": [asdict(s) for s in pr.samples], + } + return json.dumps(data, indent=2, sort_keys=True, allow_nan=False) + + +def write_pr_markdown(pr: AdapterPR, path: str) -> str: + """Atomic cwd-contained write of the rendered PR markdown.""" + if not isinstance(pr, AdapterPR): + raise TypeError("pr must be AdapterPR") + text = render_pr_markdown(pr) + return atomic_write_text(text, path, field="pr markdown path") diff --git a/soup_cli/utils/cmaes_merge.py b/soup_cli/utils/cmaes_merge.py new file mode 100644 index 0000000..170f6b9 --- /dev/null +++ b/soup_cli/utils/cmaes_merge.py @@ -0,0 +1,459 @@ +"""Evolutionary CMA-ES merge for LoRA adapters (v0.67.0 Part A). + +CMA-ES over merge weights driven by the operator's eval. Extends v0.57.0 +``soup adapters merge`` with a ``cmaes`` strategy that searches the +N-dimensional simplex of mixing weights to maximise an operator-supplied +eval score. + +Pure Python; no `cma` dependency. The implementation is a small simplex- +projecting CMA-ES (rank-mu + diagonal covariance) sufficient for ≤16 +adapters and 1–100 generations. Operators wanting full CMA-ES (BIPOP, +restart strategies) can plug their own optimiser via the ``eval_fn`` +hook; this module's contract is "given eval_fn, run a budgeted search, +return the best simplex weights". + +Public surface: + +- ``CmaesPlan`` + ``CmaesResult`` frozen dataclasses +- ``validate_population_size`` / ``validate_generations`` +- ``build_cmaes_plan(...)`` returns frozen ``CmaesPlan`` +- ``run_cmaes_merge(plan, *, eval_fn, ...)`` returns ``CmaesResult`` + +Reuses ``parse_budget`` from v0.57.0 ``blame.py`` (60s..24h bounds). + +Design notes: + +- Output weights live on the simplex (sum=1, each ≥0); the optimiser + parameterises N-1 logits and softmaxes them so any candidate is valid. +- Failures inside ``eval_fn`` are swallowed as a sentinel low score so + one broken adapter doesn't crash the run (mirrors v0.40.3 #33 / v0.48 + proxy-failure isolation). +- Live ``soup eval`` wiring is operator-supplied — `cmaes_merge` does + NOT auto-load models. Callers wrap their eval suite as a closure. +""" + +from __future__ import annotations + +import math +import os +import time +from dataclasses import dataclass +from typing import Callable, Optional, Sequence, Tuple + +from soup_cli.utils.blame import parse_budget +from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + +# --------------------------------------------------------------------------- +# Bounds (closed, locked at module load) +# --------------------------------------------------------------------------- + +MIN_POPULATION = 2 +MAX_POPULATION = 256 +MIN_GENERATIONS = 1 +MAX_GENERATIONS = 10_000 + +_MIN_ADAPTERS = 2 +_MAX_ADAPTERS = 16 # mirrors v0.57.0 adapter_merge cap +_SIMPLEX_TOL = 1e-6 +_FAILED_EVAL_SENTINEL = -1.0e9 # very negative so failed candidates never win + + +# --------------------------------------------------------------------------- +# Validators +# --------------------------------------------------------------------------- + + +def validate_population_size(value: object) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError("population_size must be int") + if value < MIN_POPULATION: + raise ValueError( + f"population_size {value} below floor {MIN_POPULATION}" + ) + if value > MAX_POPULATION: + raise ValueError( + f"population_size {value} above cap {MAX_POPULATION}" + ) + return value + + +def validate_generations(value: object) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError("max_generations must be int") + if value < MIN_GENERATIONS: + raise ValueError( + f"max_generations {value} below floor {MIN_GENERATIONS}" + ) + if value > MAX_GENERATIONS: + raise ValueError( + f"max_generations {value} above cap {MAX_GENERATIONS}" + ) + return value + + +def _validate_seed(seed: object) -> int: + if isinstance(seed, bool) or not isinstance(seed, int): + raise TypeError("seed must be int") + if seed < 0: + raise ValueError("seed must be non-negative") + if seed > 2**31 - 1: + raise ValueError("seed too large") + return seed + + +def _validate_finite_score(value: object, field: str) -> float: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, (int, float)): + raise TypeError(f"{field} must be numeric") + val = float(value) + if not math.isfinite(val): + raise ValueError(f"{field} must be finite") + return val + + +# --------------------------------------------------------------------------- +# Frozen dataclasses +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class CmaesPlan: + """Plan for an evolutionary merge run. + + Reuses v0.57 ``parse_budget`` so the budget bounds (60s..24h) are + consistent across blame / cmaes. + """ + + adapters: Tuple[str, ...] + eval_suite: str + budget_seconds: int + population_size: int + max_generations: int + seed: int + + def __post_init__(self) -> None: + if not isinstance(self.adapters, tuple): + raise TypeError("adapters must be tuple") + if len(self.adapters) < _MIN_ADAPTERS: + raise ValueError( + f"need at least {_MIN_ADAPTERS} adapters" + ) + if len(self.adapters) > _MAX_ADAPTERS: + raise ValueError(f"at most {_MAX_ADAPTERS} adapters") + for path in self.adapters: + if not isinstance(path, str) or not path: + raise ValueError("adapters entries must be non-empty str") + if not isinstance(self.eval_suite, str) or not self.eval_suite: + raise ValueError("eval_suite must be non-empty str") + validate_population_size(self.population_size) + validate_generations(self.max_generations) + _validate_seed(self.seed) + + +@dataclass(frozen=True) +class CmaesResult: + """Result of an evolutionary merge run. + + ``best_weights`` live on the simplex (sum=1, each ≥0). + """ + + best_weights: Tuple[float, ...] + best_score: float + generations_run: int + evaluations: int + wall_clock_seconds: float + converged: bool + history: Tuple[float, ...] + + def __post_init__(self) -> None: + # Score finite + bool-rejected + _validate_finite_score(self.best_score, "best_score") + if not isinstance(self.generations_run, int) or isinstance( + self.generations_run, bool + ): + raise TypeError("generations_run must be int") + if self.generations_run < 0: + raise ValueError("generations_run must be non-negative") + if not isinstance(self.evaluations, int) or isinstance( + self.evaluations, bool + ): + raise TypeError("evaluations must be int") + if self.evaluations < 0: + raise ValueError("evaluations must be non-negative") + _validate_finite_score(self.wall_clock_seconds, "wall_clock_seconds") + if self.wall_clock_seconds < 0: + raise ValueError("wall_clock_seconds must be non-negative") + if not isinstance(self.converged, bool): + raise TypeError("converged must be bool") + if not isinstance(self.best_weights, tuple): + raise TypeError("best_weights must be tuple") + if not self.best_weights: + raise ValueError("best_weights must be non-empty") + for w in self.best_weights: + if isinstance(w, bool) or not isinstance(w, (int, float)): + raise TypeError("best_weights entries must be numeric") + wf = float(w) + if not math.isfinite(wf): + raise ValueError("best_weights entries must be finite") + if wf < 0: + raise ValueError("best_weights entries must be non-negative") + total = sum(float(w) for w in self.best_weights) + if not math.isclose(total, 1.0, abs_tol=_SIMPLEX_TOL): + raise ValueError( + f"best_weights must sum to 1.0 (got {total:.6f})" + ) + if not isinstance(self.history, tuple): + raise TypeError("history must be tuple") + for h in self.history: + if isinstance(h, bool) or not isinstance(h, (int, float)): + raise TypeError("history entries must be numeric") + if not math.isfinite(float(h)): + raise ValueError("history entries must be finite") + + +# --------------------------------------------------------------------------- +# Plan construction +# --------------------------------------------------------------------------- + + +def build_cmaes_plan( + *, + adapters: Sequence[str], + eval_suite: str, + budget_spec: str, + population_size: int = 8, + max_generations: int = 20, + seed: int = 0, +) -> CmaesPlan: + """Validate inputs and return a frozen ``CmaesPlan``. + + - ``adapters`` must contain ≥2 unique paths, all under cwd. + - ``eval_suite`` must be a real path under cwd (no symlinks). + - ``budget_spec`` is parsed via v0.57 ``parse_budget`` (60s..24h). + """ + if not isinstance(adapters, Sequence) or isinstance(adapters, str): + raise TypeError("adapters must be a sequence") + if len(adapters) < _MIN_ADAPTERS: + raise ValueError(f"need at least {_MIN_ADAPTERS} adapters") + validated_adapters: list[str] = [] + for ad in adapters: + if not isinstance(ad, str) or not ad: + raise ValueError("each adapter must be a non-empty str") + enforce_under_cwd_and_no_symlink(ad, field="adapter") + validated_adapters.append(os.path.realpath(ad)) + + enforce_under_cwd_and_no_symlink(eval_suite, field="eval_suite") + if not os.path.exists(eval_suite): + raise FileNotFoundError(f"eval_suite not found: {eval_suite!r}") + + budget_seconds = parse_budget(budget_spec) + + return CmaesPlan( + adapters=tuple(validated_adapters), + eval_suite=os.path.realpath(eval_suite), + budget_seconds=budget_seconds, + population_size=validate_population_size(population_size), + max_generations=validate_generations(max_generations), + seed=_validate_seed(seed), + ) + + +# --------------------------------------------------------------------------- +# Optimiser (minimal CMA-ES on the N-1 logit space, softmaxed onto simplex) +# --------------------------------------------------------------------------- + + +def _softmax(logits: Sequence[float]) -> Tuple[float, ...]: + """Numerically-stable softmax onto the simplex.""" + m = max(logits) + exps = [math.exp(x - m) for x in logits] + z = sum(exps) + if z <= 0: + # Degenerate fallback: uniform + return tuple(1.0 / len(logits) for _ in logits) + return tuple(e / z for e in exps) + + +def _eval_safely( + eval_fn: Callable[[Tuple[float, ...]], float], + weights: Tuple[float, ...], +) -> float: + """Call ``eval_fn`` with sentinel-low score on exception. + + Failure isolation mirrors v0.40.3 #33 / v0.48 / v0.53.7 #106 policy: + one bad eval must not crash the whole run. + """ + try: + score = eval_fn(weights) + except (KeyboardInterrupt, SystemExit): + raise + except Exception: # noqa: BLE001 — eval_fn surface is operator-controlled + return _FAILED_EVAL_SENTINEL + if isinstance(score, bool): + return _FAILED_EVAL_SENTINEL + if not isinstance(score, (int, float)): + return _FAILED_EVAL_SENTINEL + s = float(score) + if not math.isfinite(s): + return _FAILED_EVAL_SENTINEL + return s + + +def run_cmaes_merge( + plan: CmaesPlan, + *, + eval_fn: Callable[[Tuple[float, ...]], float], + sigma_init: float = 0.5, + elite_frac: float = 0.5, + convergence_tol: float = 1e-4, +) -> CmaesResult: + """Run a small CMA-ES-style search over simplex merge weights. + + Operator supplies ``eval_fn(weights) -> score`` (higher is better); + we softmax N-1 logits onto the simplex, sample a population, keep + the elite half, re-fit a diagonal Gaussian, repeat until either: + + - ``max_generations`` reached, or + - elapsed wall-clock ≥ ``budget_seconds``, or + - score plateau < ``convergence_tol`` for 3 generations in a row. + """ + if not isinstance(plan, CmaesPlan): + raise TypeError("plan must be CmaesPlan") + if eval_fn is None or not callable(eval_fn): + raise TypeError("eval_fn must be callable") + if isinstance(sigma_init, bool) or not isinstance(sigma_init, (int, float)): + raise TypeError("sigma_init must be numeric") + if not math.isfinite(float(sigma_init)) or float(sigma_init) <= 0: + raise ValueError("sigma_init must be positive and finite") + if isinstance(elite_frac, bool) or not isinstance(elite_frac, (int, float)): + raise TypeError("elite_frac must be numeric") + elite = float(elite_frac) + if not (0.0 < elite < 1.0) or not math.isfinite(elite): + raise ValueError("elite_frac must be in (0, 1)") + + n_dim = len(plan.adapters) - 1 # softmax over N-1 free logits + rng = _LcgRng(plan.seed) + + mean = [0.0] * n_dim + sigma = [float(sigma_init)] * n_dim + elite_count = max(1, int(plan.population_size * elite)) + + history: list[float] = [] + best_score = -math.inf + best_weights: Tuple[float, ...] = tuple( + 1.0 / len(plan.adapters) for _ in plan.adapters + ) + + start = time.monotonic() + generations_run = 0 + evaluations = 0 + converged = False + plateau_run = 0 + + for gen in range(plan.max_generations): + if (time.monotonic() - start) >= plan.budget_seconds: + break + + # Sample population + candidates: list[Tuple[list[float], Tuple[float, ...], float]] = [] + for _ in range(plan.population_size): + sample = [ + mean[i] + sigma[i] * rng.normal() for i in range(n_dim) + ] + # Append 0.0 reference logit (softmax is shift-invariant) so we + # span the full simplex. + logits = sample + [0.0] + weights = _softmax(logits) + score = _eval_safely(eval_fn, weights) + evaluations += 1 + candidates.append((sample, weights, score)) + if score > best_score: + best_score = score + best_weights = weights + if (time.monotonic() - start) >= plan.budget_seconds: + break + + generations_run += 1 + # Pick elite by score (descending) + candidates.sort(key=lambda c: c[2], reverse=True) + elite_samples = [c[0] for c in candidates[:elite_count]] + # Recompute mean + sigma per dim from elite + new_mean = [ + sum(s[i] for s in elite_samples) / len(elite_samples) + for i in range(n_dim) + ] + new_sigma = [] + for i in range(n_dim): + var = sum( + (s[i] - new_mean[i]) ** 2 for s in elite_samples + ) / len(elite_samples) + # Clamp to a sensible floor; CMA-ES typically blends with prior + # but for ≤16-adapter case the simple rank-mu update converges. + new_sigma.append(max(math.sqrt(var), 1e-4)) + # Plateau detection + gen_best = candidates[0][2] + history.append(gen_best) + if len(history) >= 2: + delta = abs(history[-1] - history[-2]) + if delta < convergence_tol: + plateau_run += 1 + else: + plateau_run = 0 + if plateau_run >= 3: + converged = True + break + mean = new_mean + sigma = new_sigma + + wall_clock = time.monotonic() - start + if best_score == -math.inf: + # Defensive — should not happen because we always sample at least once + best_score = 0.0 + + return CmaesResult( + best_weights=tuple(best_weights), + best_score=float(best_score), + generations_run=generations_run, + evaluations=evaluations, + wall_clock_seconds=float(wall_clock), + converged=converged, + history=tuple(history), + ) + + +# --------------------------------------------------------------------------- +# Deterministic RNG (no numpy dependency at module top) +# --------------------------------------------------------------------------- + + +class _LcgRng: + """Tiny deterministic linear-congruential RNG + Box-Muller normals. + + Avoids a numpy import at module top (matches v0.66 review-grep policy + — heavy numpy stays inside live callsites only). Seeded so two runs + with the same ``seed`` produce identical trajectories. + """ + + def __init__(self, seed: int) -> None: + self._state = (seed * 2654435761 + 1) & 0xFFFFFFFFFFFFFFFF + self._pending: Optional[float] = None + + def _uniform(self) -> float: + # 64-bit LCG (Numerical Recipes-style) + self._state = (self._state * 6364136223846793005 + 1442695040888963407) & 0xFFFFFFFFFFFFFFFF + # Take the top 53 bits to fit a double's mantissa + return ((self._state >> 11) / float(1 << 53)) + + def normal(self) -> float: + """Box-Muller standard normal.""" + if self._pending is not None: + val = self._pending + self._pending = None + return val + u1 = max(self._uniform(), 1e-12) + u2 = self._uniform() + r = math.sqrt(-2.0 * math.log(u1)) + a = 2.0 * math.pi * u2 + self._pending = r * math.sin(a) + return r * math.cos(a) diff --git a/soup_cli/utils/mole_routing.py b/soup_cli/utils/mole_routing.py new file mode 100644 index 0000000..8433a35 --- /dev/null +++ b/soup_cli/utils/mole_routing.py @@ -0,0 +1,188 @@ +"""MoLE per-token adapter routing (v0.67.0 Part C). + +A gating network that routes per-token activations to one of N task +LoRAs. Following the MoLE paper (Mixture of LoRA Experts), inference- +time dispatch uses softmax gating over the per-token hidden state to +select top-K LoRAs and blend them by gating logits. + +v0.67.0 ships the schema + cross-validator. The live gating-kernel +training + serving-time dispatch is deferred to v0.67.1 (mirrors +v0.27.0 MII / v0.50.0 GRPO Plus / v0.62.0 steering stub-then-live). + +Public surface: + +- ``MoleGatingConfig`` frozen dataclass +- ``validate_mole_compat(task, backend, num_task_adapters)`` +- ``build_gating_kernel(config)`` deferred-live stub +- New ``task='moe_lora_routing'`` Literal on ``SoupConfig.task`` + +Design notes: + +- ``num_task_adapters`` bounded ``[2, 64]`` — beyond that, per-token + softmax becomes a bottleneck; operators wanting more should hierarchy + the gating. +- ``top_k <= num_task_adapters`` so sparse top-K dispatch is sane. +- ``temperature > 0`` to keep softmax non-degenerate; finite-only. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass + +# --------------------------------------------------------------------------- +# Bounds (closed, locked at module load) +# --------------------------------------------------------------------------- + +MIN_TASK_ADAPTERS = 2 +MAX_TASK_ADAPTERS = 64 +MIN_HIDDEN_DIM = 1 +MAX_HIDDEN_DIM = 16_384 +MIN_TEMPERATURE = 1e-6 +MAX_TEMPERATURE = 100.0 + + +# --------------------------------------------------------------------------- +# Validators +# --------------------------------------------------------------------------- + + +def _check_int(value: object, field: str, lo: int, hi: int) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError(f"{field} must be int") + if value < lo: + raise ValueError(f"{field} {value} below floor {lo}") + if value > hi: + raise ValueError(f"{field} {value} above cap {hi}") + return value + + +def _check_finite_positive( + value: object, field: str, lo: float, hi: float +) -> float: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, (int, float)): + raise TypeError(f"{field} must be numeric") + val = float(value) + if not math.isfinite(val): + raise ValueError(f"{field} must be finite") + if val < lo: + raise ValueError(f"{field} {val} below floor {lo}") + if val > hi: + raise ValueError(f"{field} {val} above cap {hi}") + return val + + +def _check_str_field(value: object, field: str) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + return value + + +# --------------------------------------------------------------------------- +# Frozen config +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class MoleGatingConfig: + """Per-token gating network over N task LoRAs. + + Frozen dataclass — post-construction mutation raises ``FrozenInstanceError``. + """ + + num_task_adapters: int + hidden_dim: int + temperature: float + top_k: int + + def __post_init__(self) -> None: + _check_int( + self.num_task_adapters, + "num_task_adapters", + MIN_TASK_ADAPTERS, + MAX_TASK_ADAPTERS, + ) + _check_int( + self.hidden_dim, "hidden_dim", MIN_HIDDEN_DIM, MAX_HIDDEN_DIM + ) + _check_finite_positive( + self.temperature, + "temperature", + MIN_TEMPERATURE, + MAX_TEMPERATURE, + ) + # top_k must be positive and not exceed num_task_adapters + _check_int(self.top_k, "top_k", 1, MAX_TASK_ADAPTERS) + if self.top_k > self.num_task_adapters: + raise ValueError( + f"top_k {self.top_k} > num_task_adapters " + f"{self.num_task_adapters}" + ) + + +# --------------------------------------------------------------------------- +# Cross-validator (called from SoupConfig + standalone) +# --------------------------------------------------------------------------- + + +def validate_mole_compat( + *, + task: str, + backend: str, + num_task_adapters: int, +) -> None: + """Schema-time gate. + + - Requires ``task='moe_lora_routing'`` (silent-no-op footgun rejection + matching v0.52.0 distill / v0.62.0 citation_faithful task-gates). + - Rejects ``backend='mlx'`` — the gating kernel needs torch dispatch + that mlx-lm doesn't expose (deferred to a future MLX integration). + - ``num_task_adapters`` must be in ``[MIN_TASK_ADAPTERS, MAX_TASK_ADAPTERS]``. + """ + _check_str_field(task, "task") + _check_str_field(backend, "backend") + if task != "moe_lora_routing": + raise ValueError( + f"validate_mole_compat: task must be 'moe_lora_routing' " + f"(got {task!r})" + ) + if backend == "mlx": + raise ValueError( + "MoLE routing is not supported on the mlx backend " + "(live wiring deferred; see v0.67.1)" + ) + _check_int( + num_task_adapters, + "num_task_adapters", + MIN_TASK_ADAPTERS, + MAX_TASK_ADAPTERS, + ) + + +# --------------------------------------------------------------------------- +# Deferred-live stub +# --------------------------------------------------------------------------- + + +def build_gating_kernel(config: MoleGatingConfig): + """Build a per-token gating kernel for MoLE dispatch. + + Deferred to v0.67.1: live wiring requires the v0.22.0 multi-adapter + serving surface plus a torch gating module that emits softmax + routing logits per token. The schema (this module) ships now so + operators can wire ``num_task_adapters`` / ``top_k`` / ``temperature`` + into their config; the live kernel comes next. + """ + if not isinstance(config, MoleGatingConfig): + raise TypeError("config must be MoleGatingConfig") + raise NotImplementedError( + "build_gating_kernel live wiring deferred to v0.67.1" + ) diff --git a/soup_cli/utils/soup_lock.py b/soup_cli/utils/soup_lock.py new file mode 100644 index 0000000..7c90d76 --- /dev/null +++ b/soup_cli/utils/soup_lock.py @@ -0,0 +1,218 @@ +"""``soup.lock`` shared run lockfile (v0.67.0 Part E). + +A ``soup.lock`` is the closure of three SHA-256 hashes: + + closure = sha256(base_model_sha || dataset_sha || env_hash) + +Committed to git alongside ``soup.yaml``, teams coordinate on +"reproducible training run" by checking the closure on every +``soup train`` and refusing to start when the lock drifts. Composes +with v0.64 Part C ``soup env`` — operators run ``soup env lock`` +to get ``env_hash``, then ``soup lock write`` to write the file. + +Public surface: + +- ``SoupLock`` frozen dataclass +- ``LockDrift`` frozen dataclass (ok / changes tuple) +- ``compute_lock_closure(*, base_model_sha, dataset_sha, env_hash)`` -> hex +- ``write_lock(lock, path)`` / ``read_lock(path)`` atomic JSON I/O +- ``check_lock_drift(expected, actual)`` -> ``LockDrift`` +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import stat +from dataclasses import asdict, dataclass +from typing import Tuple + +from soup_cli.utils.paths import ( + atomic_write_text, + enforce_under_cwd_and_no_symlink, +) + +_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") +_MAX_FILE_BYTES = 1 * 1024 * 1024 # 1 MiB cap on lock file size +_MAX_VERSION_LEN = 64 +_MAX_BASE_MODEL_LEN = 512 +_MAX_CREATED_AT_LEN = 64 + + +# --------------------------------------------------------------------------- +# Validators +# --------------------------------------------------------------------------- + + +def _check_sha(value: object, field: str) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str") + if not _SHA256_RE.match(value): + raise ValueError(f"{field} must be 64 hex chars (got {value!r})") + return value + + +def _check_str(value: object, field: str, max_len: int) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > max_len: + raise ValueError( + f"{field} length {len(value)} > {max_len}" + ) + return value + + +# --------------------------------------------------------------------------- +# compute_lock_closure +# --------------------------------------------------------------------------- + + +def compute_lock_closure( + *, + base_model_sha: str, + dataset_sha: str, + env_hash: str, +) -> str: + """Return the 64-hex closure SHA over the three input hashes. + + Each input must be a 64-hex SHA-256 itself. The closure is + ``sha256(base_model_sha || dataset_sha || env_hash)``. + """ + _check_sha(base_model_sha, "base_model_sha") + _check_sha(dataset_sha, "dataset_sha") + _check_sha(env_hash, "env_hash") + h = hashlib.sha256() + h.update(base_model_sha.encode("ascii")) + h.update(dataset_sha.encode("ascii")) + h.update(env_hash.encode("ascii")) + return h.hexdigest() + + +# --------------------------------------------------------------------------- +# Frozen dataclasses +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class SoupLock: + """The on-disk lockfile shape. + + Stored as JSON at ``soup.lock`` (or operator-named path); committed + to git so the team coordinates on "reproducible training run". + """ + + soup_version: str + base_model: str + base_model_sha: str + dataset_sha: str + env_hash: str + closure_sha: str + created_at: str + + def __post_init__(self) -> None: + _check_str(self.soup_version, "soup_version", _MAX_VERSION_LEN) + _check_str(self.base_model, "base_model", _MAX_BASE_MODEL_LEN) + _check_sha(self.base_model_sha, "base_model_sha") + _check_sha(self.dataset_sha, "dataset_sha") + _check_sha(self.env_hash, "env_hash") + _check_sha(self.closure_sha, "closure_sha") + _check_str(self.created_at, "created_at", _MAX_CREATED_AT_LEN) + + +@dataclass(frozen=True) +class LockDrift: + """Result of comparing two locks. ``ok`` is True when no field changed.""" + + ok: bool + changes: Tuple[str, ...] + + def __post_init__(self) -> None: + if not isinstance(self.ok, bool): + raise TypeError("ok must be bool") + if not isinstance(self.changes, tuple): + raise TypeError("changes must be tuple") + for c in self.changes: + if not isinstance(c, str): + raise TypeError("changes entries must be str") + + +# --------------------------------------------------------------------------- +# Atomic I/O +# --------------------------------------------------------------------------- + + +def write_lock(lock: SoupLock, path: str) -> str: + """Atomically write a lock to JSON under cwd containment.""" + if not isinstance(lock, SoupLock): + raise TypeError("lock must be SoupLock") + text = json.dumps(asdict(lock), indent=2, sort_keys=True) + return atomic_write_text(text, path, field="lock path") + + +def read_lock(path: str) -> SoupLock: + """Read + validate a soup.lock from JSON. cwd-contained, symlink-rejected.""" + if not isinstance(path, str): + raise TypeError("path must be str") + if not path: + raise ValueError("path must be non-empty") + if "\x00" in path: + raise ValueError("path must not contain null bytes") + enforce_under_cwd_and_no_symlink(path, field="lock path") + real = os.path.realpath(path) + if not os.path.exists(real): + raise FileNotFoundError(f"lock file not found: {path!r}") + st = os.lstat(real) + if stat.S_ISLNK(st.st_mode): + raise ValueError("lock path must not be a symlink (TOCTOU defence)") + if st.st_size > _MAX_FILE_BYTES: + raise ValueError( + f"lock file size {st.st_size} > {_MAX_FILE_BYTES}" + ) + with open(real, encoding="utf-8") as fh: + data = json.load(fh) + if not isinstance(data, dict): + raise ValueError("lock file root must be JSON object") + # SoupLock __post_init__ does the rest of the validation + return SoupLock( + soup_version=data.get("soup_version", ""), + base_model=data.get("base_model", ""), + base_model_sha=data.get("base_model_sha", ""), + dataset_sha=data.get("dataset_sha", ""), + env_hash=data.get("env_hash", ""), + closure_sha=data.get("closure_sha", ""), + created_at=data.get("created_at", ""), + ) + + +# --------------------------------------------------------------------------- +# Drift detection +# --------------------------------------------------------------------------- + + +def check_lock_drift(expected: SoupLock, actual: SoupLock) -> LockDrift: + """Compare two locks field-by-field. Returns a ``LockDrift`` report. + + Only the four content-bearing SHA fields + base_model name affect + drift detection. ``soup_version`` / ``created_at`` differences are + tracked separately as advisory-only (operators upgrading Soup will + legitimately see them change). + """ + if not isinstance(expected, SoupLock): + raise TypeError("expected must be SoupLock") + if not isinstance(actual, SoupLock): + raise TypeError("actual must be SoupLock") + changes: list[str] = [] + for field in ("base_model", "base_model_sha", "dataset_sha", "env_hash", "closure_sha"): + if getattr(expected, field) != getattr(actual, field): + changes.append(f"{field} drifted") + return LockDrift(ok=not changes, changes=tuple(changes)) diff --git a/soup_cli/utils/vector_bank.py b/soup_cli/utils/vector_bank.py new file mode 100644 index 0000000..8983cf6 --- /dev/null +++ b/soup_cli/utils/vector_bank.py @@ -0,0 +1,359 @@ +"""VeRA / VB-LoRA vector-bank storage format (v0.67.0 Part B). + +A vector bank is the data structure behind multi-tenant LoRA serving at +MB-per-user instead of hundreds-of-MB per LoRA: + + Bank = { + shared random projection matrix P : (d_model × d_model), + per-user scaling vector v_u : (vector_dim,), + } + +The per-user delta at inference time is ``v_u ⊙ Px`` (VeRA) or a +codebook lookup (VB-LoRA). Storage size is dominated by ``vector_dim`` +per user — a 128-D vector at fp32 is 512 bytes vs ~30 MB for a rank-16 +LoRA on a 7B model. + +v0.67.0 ships the schema + atomic disk I/O + validators. +Live wiring into multi-adapter serving (v0.22.0 surface) is the +v0.67.1 deliverable — ``apply_bank_to_serve`` raises +``NotImplementedError`` with explicit marker (matches v0.27.0 MII / +v0.50.0 GRPO Plus stub-then-live cadence). + +Public surface: + +- ``VectorBank`` / ``BankEntry`` frozen dataclasses +- ``validate_bank_name`` / ``validate_user_id`` / ``validate_scaling_vector`` +- ``estimate_bank_size(num_users, vector_dim)`` -> bytes +- ``write_bank(bank, path)`` / ``load_bank(path)`` atomic JSON I/O +- ``apply_bank_to_serve(bank)`` deferred-live stub +""" + +from __future__ import annotations + +import json +import math +import os +import re +import stat +from dataclasses import dataclass +from typing import Any, Iterable, Tuple + +from soup_cli.utils.paths import ( + atomic_write_text, + enforce_under_cwd_and_no_symlink, +) + +# --------------------------------------------------------------------------- +# Bounds (closed, locked at module load) +# --------------------------------------------------------------------------- + +MAX_NAME_LEN = 128 +MAX_USER_ID_LEN = 256 +MAX_BASE_MODEL_LEN = 512 +MAX_VECTOR_DIM = 16_384 +MIN_VECTOR_DIM = 1 +MAX_ENTRIES_PER_BANK = 1_000_000 +_MAX_FILE_BYTES = 16 * 1024 * 1024 # 16 MiB cap on bank file size + +# kebab-case + `._-`, leading alnum, no path separators / shell metacharacters +_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._\-]{0,127}$") + + +# --------------------------------------------------------------------------- +# Validators +# --------------------------------------------------------------------------- + + +def validate_bank_name(name: object) -> str: + """Canonical kebab-case bank name (case-insensitive).""" + if isinstance(name, bool): + raise TypeError("name must not be bool") + if not isinstance(name, str): + raise TypeError( + f"name must be str, got {type(name).__name__}" + ) + if not name: + raise ValueError("name must be non-empty") + if "\x00" in name: + raise ValueError("name must not contain null bytes") + if len(name) > MAX_NAME_LEN: + raise ValueError( + f"name length {len(name)} > {MAX_NAME_LEN}" + ) + canonical = name.lower() + if not _NAME_RE.match(canonical): + raise ValueError( + f"name must be kebab-case alphanumeric + `._-`, got {name!r}" + ) + return canonical + + +def validate_user_id(user_id: object) -> str: + if isinstance(user_id, bool): + raise TypeError("user_id must not be bool") + if not isinstance(user_id, str): + raise TypeError("user_id must be str") + if not user_id: + raise ValueError("user_id must be non-empty") + if "\x00" in user_id: + raise ValueError("user_id must not contain null bytes") + if len(user_id) > MAX_USER_ID_LEN: + raise ValueError( + f"user_id length {len(user_id)} > {MAX_USER_ID_LEN}" + ) + return user_id + + +def validate_scaling_vector(values: object) -> Tuple[float, ...]: + """Per-user scaling vector — non-empty, finite, ≤MAX_VECTOR_DIM.""" + if isinstance(values, (str, bytes)) or not isinstance(values, Iterable): + raise TypeError("scaling vector must be a non-string iterable") + out: list[float] = [] + for v in values: + if isinstance(v, bool): + raise TypeError("scaling vector entries must not be bool") + if not isinstance(v, (int, float)): + raise TypeError("scaling vector entries must be numeric") + f = float(v) + if not math.isfinite(f): + raise ValueError("scaling vector entries must be finite") + out.append(f) + if not out: + raise ValueError("scaling vector must be non-empty") + if len(out) > MAX_VECTOR_DIM: + raise ValueError( + f"scaling vector length {len(out)} > {MAX_VECTOR_DIM}" + ) + return tuple(out) + + +def _validate_base_model(base_model: object) -> str: + if isinstance(base_model, bool): + raise TypeError("base_model must not be bool") + if not isinstance(base_model, str): + raise TypeError("base_model must be str") + if not base_model: + raise ValueError("base_model must be non-empty") + if "\x00" in base_model: + raise ValueError("base_model must not contain null bytes") + if len(base_model) > MAX_BASE_MODEL_LEN: + raise ValueError( + f"base_model length {len(base_model)} > {MAX_BASE_MODEL_LEN}" + ) + return base_model + + +def _validate_seed(seed: object) -> int: + if isinstance(seed, bool) or not isinstance(seed, int): + raise TypeError("projection_seed must be int") + if seed < 0: + raise ValueError("projection_seed must be non-negative") + if seed > 2**63 - 1: + raise ValueError("projection_seed too large") + return seed + + +def _validate_vector_dim(dim: object) -> int: + if isinstance(dim, bool) or not isinstance(dim, int): + raise TypeError("vector_dim must be int") + if dim < MIN_VECTOR_DIM: + raise ValueError( + f"vector_dim {dim} below floor {MIN_VECTOR_DIM}" + ) + if dim > MAX_VECTOR_DIM: + raise ValueError( + f"vector_dim {dim} above cap {MAX_VECTOR_DIM}" + ) + return dim + + +# --------------------------------------------------------------------------- +# Frozen dataclasses +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class BankEntry: + """One per-user scaling vector in the bank.""" + + user_id: str + scaling: Tuple[float, ...] + + def __post_init__(self) -> None: + # Re-validate so callers constructing BankEntry directly cannot + # smuggle in invalid state (matches v0.61 EditPlan policy). + validate_user_id(self.user_id) + if not isinstance(self.scaling, tuple): + raise TypeError("scaling must be tuple") + validate_scaling_vector(self.scaling) + + +@dataclass(frozen=True) +class VectorBank: + """A multi-tenant LoRA storage bank. + + ``projection_seed`` is the deterministic seed for the shared random + projection P. Each ``BankEntry`` carries a per-user scaling vector + of length ``vector_dim``. + + Live serving wiring is deferred to v0.67.1. The v0.67.0 surface is + the schema + atomic disk I/O. + """ + + name: str + base_model: str + projection_seed: int + vector_dim: int + entries: Tuple[BankEntry, ...] + + def __post_init__(self) -> None: + validate_bank_name(self.name) + _validate_base_model(self.base_model) + _validate_seed(self.projection_seed) + _validate_vector_dim(self.vector_dim) + if not isinstance(self.entries, tuple): + raise TypeError("entries must be tuple") + if len(self.entries) > MAX_ENTRIES_PER_BANK: + raise ValueError( + f"entries length {len(self.entries)} > " + f"{MAX_ENTRIES_PER_BANK}" + ) + for entry in self.entries: + if not isinstance(entry, BankEntry): + raise TypeError("entries must be BankEntry instances") + if len(entry.scaling) != self.vector_dim: + raise ValueError( + f"entry user_id={entry.user_id!r} scaling length " + f"{len(entry.scaling)} != vector_dim " + f"{self.vector_dim}" + ) + + +# --------------------------------------------------------------------------- +# Size estimate +# --------------------------------------------------------------------------- + + +def estimate_bank_size(*, num_users: int, vector_dim: int) -> int: + """Estimate disk + memory cost of a bank in bytes. + + The shared projection matrix dominates for small ``num_users``; + per-user scaling dominates as the bank fills up. Both are fp32 + (4 bytes/element). + """ + if isinstance(num_users, bool) or not isinstance(num_users, int): + raise TypeError("num_users must be int") + if num_users < 0: + raise ValueError("num_users must be non-negative") + if isinstance(vector_dim, bool) or not isinstance(vector_dim, int): + raise TypeError("vector_dim must be int") + if vector_dim < 1: + raise ValueError("vector_dim must be positive") + projection_bytes = vector_dim * vector_dim * 4 # P : (d×d) fp32 + user_bytes = num_users * vector_dim * 4 # v_u : (d,) fp32 + return projection_bytes + user_bytes + + +# --------------------------------------------------------------------------- +# Atomic JSON I/O +# --------------------------------------------------------------------------- + + +def _bank_to_dict(bank: VectorBank) -> dict: + return { + "name": bank.name, + "base_model": bank.base_model, + "projection_seed": bank.projection_seed, + "vector_dim": bank.vector_dim, + "entries": [ + {"user_id": e.user_id, "scaling": list(e.scaling)} + for e in bank.entries + ], + } + + +def _bank_from_dict(data: object) -> VectorBank: + if not isinstance(data, dict): + raise ValueError("bank file root must be JSON object") + name = data.get("name") + base_model = data.get("base_model") + seed = data.get("projection_seed") + dim = data.get("vector_dim") + raw_entries = data.get("entries", []) + if not isinstance(raw_entries, list): + raise ValueError("entries field must be a list") + entries: list[BankEntry] = [] + for raw in raw_entries: + if not isinstance(raw, dict): + raise ValueError("each entry must be a JSON object") + user_id = raw.get("user_id") + scaling = raw.get("scaling") + if not isinstance(scaling, list): + raise ValueError("scaling must be a list") + entries.append( + BankEntry( + user_id=user_id if isinstance(user_id, str) else "", + scaling=tuple(validate_scaling_vector(scaling)), + ) + ) + return VectorBank( + name=name if isinstance(name, str) else "", + base_model=base_model if isinstance(base_model, str) else "", + projection_seed=seed if isinstance(seed, int) else 0, + vector_dim=dim if isinstance(dim, int) else 0, + entries=tuple(entries), + ) + + +def write_bank(bank: VectorBank, path: str) -> str: + """Atomically write a bank to JSON under cwd containment.""" + if not isinstance(bank, VectorBank): + raise TypeError("bank must be VectorBank") + text = json.dumps(_bank_to_dict(bank), indent=2, sort_keys=True) + return atomic_write_text(text, path, field="bank path") + + +def load_bank(path: str) -> VectorBank: + """Read + validate a bank JSON. cwd-contained, symlink-rejected.""" + if not isinstance(path, str): + raise TypeError("path must be str") + if not path: + raise ValueError("path must be non-empty") + if "\x00" in path: + raise ValueError("path must not contain null bytes") + enforce_under_cwd_and_no_symlink(path, field="bank path") + # Size cap defence (matches v0.55+ policy) + real = os.path.realpath(path) + if not os.path.exists(real): + raise FileNotFoundError(f"bank file not found: {path!r}") + st = os.lstat(real) + if stat.S_ISLNK(st.st_mode): + raise ValueError("bank path must not be a symlink (TOCTOU defence)") + if st.st_size > _MAX_FILE_BYTES: + raise ValueError( + f"bank file size {st.st_size} > {_MAX_FILE_BYTES}" + ) + with open(real, encoding="utf-8") as fh: + data = json.load(fh) + return _bank_from_dict(data) + + +# --------------------------------------------------------------------------- +# Deferred-live stub: live serving wiring +# --------------------------------------------------------------------------- + + +def apply_bank_to_serve(bank: VectorBank, *, server: Any = None) -> None: + """Apply a vector bank to a running ``soup serve`` instance. + + Deferred to v0.67.1 — live wiring requires the v0.22.0 multi-adapter + serving surface plus a real torch path that materialises the shared + projection matrix in GPU memory. The schema (this module) ships now + so client code can target the API; the live engine integration is + the v0.67.1 deliverable. + """ + if not isinstance(bank, VectorBank): + raise TypeError("bank must be VectorBank") + raise NotImplementedError( + "apply_bank_to_serve live wiring deferred to v0.67.1" + ) diff --git a/tests/test_v0570_part_b.py b/tests/test_v0570_part_b.py index 6b8ce42..b2265a4 100644 --- a/tests/test_v0570_part_b.py +++ b/tests/test_v0570_part_b.py @@ -281,10 +281,13 @@ def test_merge_adapters_output_outside_cwd(tmp_path, monkeypatch): def test_supported_strategies_immutable(): # v0.57.0 review fix: SUPPORTED_STRATEGIES is a frozenset (matches v0.41.0+ # allowlist policy). STRATEGY_ORDER preserves canonical iteration order. + # v0.67.0 Part A: floor-check widened to include "cmaes" (mirrors + # v0.51.0 / v0.54.0 / v0.66.0 floor-check policy). from soup_cli.utils.adapter_merge import STRATEGY_ORDER - assert SUPPORTED_STRATEGIES == frozenset({"linear", "ties", "dare", "svd"}) + assert {"linear", "ties", "dare", "svd"} <= SUPPORTED_STRATEGIES assert isinstance(SUPPORTED_STRATEGIES, frozenset) - assert STRATEGY_ORDER == ("linear", "ties", "dare", "svd") + for entry in ("linear", "ties", "dare", "svd"): + assert entry in STRATEGY_ORDER def test_merge_ties_density_one_keeps_everything(): diff --git a/tests/test_v0670_followups.py b/tests/test_v0670_followups.py new file mode 100644 index 0000000..fc765e8 --- /dev/null +++ b/tests/test_v0670_followups.py @@ -0,0 +1,389 @@ +"""v0.67.0 review-fix follow-ups (from tdd-guide + security-review waves). + +HIGH-priority gaps surfaced by the TDD review: + +- Plateau-convergence path in CMA-ES +- POSIX symlink rejection at write boundaries (vector_bank.write_bank, + cmaes_merge has no disk surface, only the load-side has it today) +- Vector-bank forward-compat: unknown JSON fields silently ignored +- Adapter PR empty-sections rendering (no None leakage) +- Bisect non-monotonic eval_fn does not crash +- Exact-boundary tests at MIN/MAX values +- validate_user_id bool rejection +- soup.lock version-drift advisory check +""" + +from __future__ import annotations + +import json +import os + +import pytest + +# ----------------------------------------------------------------------------- +# Part A — Plateau convergence +# ----------------------------------------------------------------------------- + + +class TestCmaesPlateauConvergence: + def test_converges_on_flat_landscape(self, tmp_path, monkeypatch) -> None: + """When eval_fn returns the same score for 3+ generations, + ``converged=True`` and the loop short-circuits early.""" + from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge + + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + (tmp_path / "b").mkdir() + suite = tmp_path / "s.yaml" + suite.write_text("x: 1\n", encoding="utf-8") + + plan = build_cmaes_plan( + adapters=[str(tmp_path / "a"), str(tmp_path / "b")], + eval_suite=str(suite), + budget_spec="60s", + population_size=4, + max_generations=20, + seed=42, + ) + + # Flat constant landscape: every weight returns 0.5 + def eval_fn(weights): + return 0.5 + + result = run_cmaes_merge(plan, eval_fn=eval_fn) + assert result.converged is True + # Plateau detection breaks out before max_generations + assert result.generations_run < plan.max_generations + + +# ----------------------------------------------------------------------------- +# Part A — Exact boundary tests +# ----------------------------------------------------------------------------- + + +class TestCmaesValidatorBoundaries: + def test_population_min_accepted(self) -> None: + from soup_cli.utils.cmaes_merge import MIN_POPULATION, validate_population_size + + assert validate_population_size(MIN_POPULATION) == MIN_POPULATION + + def test_population_min_minus_one_rejected(self) -> None: + from soup_cli.utils.cmaes_merge import MIN_POPULATION, validate_population_size + + with pytest.raises(ValueError): + validate_population_size(MIN_POPULATION - 1) + + def test_population_max_accepted(self) -> None: + from soup_cli.utils.cmaes_merge import MAX_POPULATION, validate_population_size + + assert validate_population_size(MAX_POPULATION) == MAX_POPULATION + + def test_generations_min_accepted(self) -> None: + from soup_cli.utils.cmaes_merge import MIN_GENERATIONS, validate_generations + + assert validate_generations(MIN_GENERATIONS) == MIN_GENERATIONS + + def test_generations_max_accepted(self) -> None: + from soup_cli.utils.cmaes_merge import MAX_GENERATIONS, validate_generations + + assert validate_generations(MAX_GENERATIONS) == MAX_GENERATIONS + + +# ----------------------------------------------------------------------------- +# Part B — Forward-compat: unknown fields silently ignored +# ----------------------------------------------------------------------------- + + +class TestVectorBankForwardCompat: + def test_unknown_fields_ignored(self, tmp_path, monkeypatch) -> None: + """A bank JSON with extra unknown fields should round-trip without + error — guards against breakage when v0.67.1+ adds new fields.""" + from soup_cli.utils.vector_bank import load_bank + + monkeypatch.chdir(tmp_path) + path = tmp_path / "bank.json" + path.write_text( + json.dumps( + { + "name": "fc", + "base_model": "m", + "projection_seed": 0, + "vector_dim": 2, + "entries": [ + {"user_id": "u", "scaling": [0.1, 0.2]}, + ], + # Forward-compat: unknown field — must be tolerated + "_future_field": 99, + "unknown_key": "unknown_value", + } + ), + encoding="utf-8", + ) + loaded = load_bank(str(path)) + assert loaded.name == "fc" + assert loaded.vector_dim == 2 + + @pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink test") + def test_write_bank_symlink_rejected(self, tmp_path, monkeypatch) -> None: + """`write_bank` must reject a pre-placed symlink at the target path + (TOCTOU defence — mirrors v0.55.0 / v0.56.0 policy).""" + from soup_cli.utils.vector_bank import VectorBank, write_bank + + monkeypatch.chdir(tmp_path) + target = tmp_path / "out.json" + real_target = tmp_path / "real.json" + real_target.write_text("{}", encoding="utf-8") + os.symlink(real_target, target) + + bank = VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=1, + entries=(), + ) + with pytest.raises(ValueError): + write_bank(bank, str(target)) + + +# ----------------------------------------------------------------------------- +# Part B — validate_user_id bool rejection +# ----------------------------------------------------------------------------- + + +class TestValidateUserIdBool: + def test_bool_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_user_id + + with pytest.raises(TypeError): + validate_user_id(True) # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# Part B — Exact boundaries +# ----------------------------------------------------------------------------- + + +class TestVectorBankBoundaries: + def test_vector_dim_at_max_accepted(self) -> None: + from soup_cli.utils.vector_bank import MAX_VECTOR_DIM, VectorBank + + # MAX value should be accepted (use it with empty entries) + VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=MAX_VECTOR_DIM, + entries=(), + ) + + def test_vector_dim_above_max_rejected(self) -> None: + from soup_cli.utils.vector_bank import MAX_VECTOR_DIM, VectorBank + + with pytest.raises(ValueError): + VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=MAX_VECTOR_DIM + 1, + entries=(), + ) + + +# ----------------------------------------------------------------------------- +# Part D — Empty-sections PR renders without None leakage +# ----------------------------------------------------------------------------- + + +class TestPRRenderEmptySections: + def test_all_empty_sections(self) -> None: + """A PR with no deltas / no samples / no dataset_diff should + render valid Markdown without any `None` literals leaking.""" + from soup_cli.utils.adapter_pr import AdapterPR, render_pr_markdown + + pr = AdapterPR( + title="empty-pr", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + md = render_pr_markdown(pr) + assert "None" not in md + assert "empty-pr" in md + + def test_json_handles_empty_sections(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR, render_pr_json + + pr = AdapterPR( + title="t", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + data = json.loads(render_pr_json(pr)) + assert data["deltas"] == [] + assert data["samples"] == [] + + +# ----------------------------------------------------------------------------- +# Part E — soup_version drift is advisory-only (not a drift signal) +# ----------------------------------------------------------------------------- + + +class TestSoupLockVersionDriftAdvisory: + def test_version_change_not_drift(self) -> None: + """``soup_version`` differing between expected and actual locks + should NOT count as drift — operators upgrade Soup legitimately.""" + from soup_cli.utils.soup_lock import SoupLock, check_lock_drift + + base = dict( + base_model="m", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24", + ) + expected = SoupLock(soup_version="0.67.0", **base) + actual = SoupLock(soup_version="0.68.0", **base) + + drift = check_lock_drift(expected, actual) + assert drift.ok is True + + def test_created_at_change_not_drift(self) -> None: + from soup_cli.utils.soup_lock import SoupLock, check_lock_drift + + base = dict( + soup_version="0.67.0", + base_model="m", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + ) + expected = SoupLock(created_at="2026-05-24", **base) + actual = SoupLock(created_at="2026-06-01", **base) + + drift = check_lock_drift(expected, actual) + assert drift.ok is True + + +# ----------------------------------------------------------------------------- +# Part F — Non-monotonic eval_fn does not crash +# ----------------------------------------------------------------------------- + + +class TestBisectNonMonotonic: + def test_non_monotonic_returns_valid_result(self) -> None: + """If the eval_fn flips back and forth (non-monotonic regression), + the bisect must still return a valid ``BisectResult`` rather than + crash. The boundary it finds is undefined but must be in-range.""" + from soup_cli.utils.adapter_bisect import ( + BisectPlan, + BisectResult, + run_bisect, + ) + + plan = BisectPlan(history=("c0", "c1", "c2", "c3", "c4", "c5")) + + # Non-monotonic: c0 ok, c1 fail, c2 ok, c3 fail, ... + def eval_fn(checkpoint: str) -> bool: + return int(checkpoint[1:]) % 2 == 0 + + # Must not raise; result is a valid BisectResult + result = run_bisect(plan, eval_fn=eval_fn) + assert isinstance(result, BisectResult) + # Boundary is some valid checkpoint id + if result.first_broken is not None: + assert result.first_broken in plan.history + + +# ----------------------------------------------------------------------------- +# Source-grep regression guards on policies the agents asked us to verify +# ----------------------------------------------------------------------------- + + +class TestSourceWiringRegressions: + """Project-wide policies verified by source-grep across v0.67.0 modules.""" + + def test_all_modules_have_future_annotations(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + for module in ( + "soup_cli/utils/cmaes_merge.py", + "soup_cli/utils/vector_bank.py", + "soup_cli/utils/mole_routing.py", + "soup_cli/utils/adapter_pr.py", + "soup_cli/utils/soup_lock.py", + "soup_cli/utils/adapter_bisect.py", + "soup_cli/commands/lock.py", + ): + src = (root / module).read_text(encoding="utf-8") + assert "from __future__ import annotations" in src, ( + f"{module}: missing 'from __future__ import annotations'" + ) + + def test_atomic_write_used_in_disk_surfaces(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + for module in ( + "soup_cli/utils/vector_bank.py", + "soup_cli/utils/adapter_pr.py", + "soup_cli/utils/soup_lock.py", + ): + src = (root / module).read_text(encoding="utf-8") + assert "atomic_write_text" in src, ( + f"{module}: must use atomic_write_text for disk writes" + ) + + def test_no_top_level_torch_imports(self) -> None: + """v0.67.0 modules MUST stay torch-free at import time.""" + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + for module in ( + "soup_cli/utils/cmaes_merge.py", + "soup_cli/utils/vector_bank.py", + "soup_cli/utils/mole_routing.py", + "soup_cli/utils/adapter_pr.py", + "soup_cli/utils/soup_lock.py", + "soup_cli/utils/adapter_bisect.py", + "soup_cli/commands/lock.py", + ): + src = (root / module).read_text(encoding="utf-8") + head = "\n".join( + line for line in src.splitlines()[:50] + if line.strip() and not line.strip().startswith("#") + ) + for forbidden in ( + "import torch", "import transformers", + "import peft", "import safetensors", + ): + assert forbidden not in head, ( + f"{module}: top-level {forbidden!r} forbidden" + ) + + def test_supported_strategies_has_cmaes(self) -> None: + from soup_cli.utils.adapter_merge import SUPPORTED_STRATEGIES + + assert "cmaes" in SUPPORTED_STRATEGIES + + def test_subprocess_call_uses_argv_list(self) -> None: + """`soup adapters bisect` must use argv list mode (no shell=True).""" + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "commands" / "adapters.py").read_text( + encoding="utf-8" + ) + # The bisect subprocess call site + # Must NOT use shell=True; must use shlex.split + shlex.quote + assert "shell=True" not in src + # Must include shlex.quote pattern for the {ckpt} substitution + assert "shlex.quote" in src diff --git a/tests/test_v0670_part_a.py b/tests/test_v0670_part_a.py new file mode 100644 index 0000000..7fe2bc5 --- /dev/null +++ b/tests/test_v0670_part_a.py @@ -0,0 +1,480 @@ +"""v0.67.0 Part A — Evolutionary merge (CMA-ES) over LoRA adapter weights. + +Tests for ``soup_cli/utils/cmaes_merge.py``: + +- Closed allowlist + frozen dataclasses + parse_budget reuse from blame.py +- ``run_cmaes_merge`` orchestrator with operator-supplied ``eval_fn`` injection +- ``soup adapters merge --strategy cmaes --eval <suite> --budget 1h`` CLI plumbing +- Validation matrix (bool/null-byte/non-finite/oversize rejection) +- Budget bounds reused from v0.57 ``blame.parse_budget`` (60s..24h) +- Source-grep regression guard for the new strategy alias in adapter_merge +""" + +from __future__ import annotations + +import dataclasses +import math + +import pytest + +# ----------------------------------------------------------------------------- +# Module surface — imports + constants +# ----------------------------------------------------------------------------- + + +class TestPublicSurface: + def test_module_importable(self) -> None: + from soup_cli.utils import cmaes_merge + + assert hasattr(cmaes_merge, "CmaesPlan") + assert hasattr(cmaes_merge, "CmaesResult") + assert hasattr(cmaes_merge, "run_cmaes_merge") + assert hasattr(cmaes_merge, "validate_population_size") + assert hasattr(cmaes_merge, "validate_generations") + + def test_constants_immutable(self) -> None: + from soup_cli.utils import cmaes_merge + + assert cmaes_merge.MIN_POPULATION >= 2 + assert cmaes_merge.MAX_POPULATION <= 256 + assert cmaes_merge.MIN_GENERATIONS >= 1 + assert cmaes_merge.MAX_GENERATIONS <= 10000 + + +# ----------------------------------------------------------------------------- +# Validators +# ----------------------------------------------------------------------------- + + +class TestValidatePopulationSize: + def test_happy(self) -> None: + from soup_cli.utils.cmaes_merge import validate_population_size + + assert validate_population_size(8) == 8 + + def test_bool_rejected(self) -> None: + from soup_cli.utils.cmaes_merge import validate_population_size + + with pytest.raises(TypeError): + validate_population_size(True) # type: ignore[arg-type] + + def test_below_floor(self) -> None: + from soup_cli.utils.cmaes_merge import validate_population_size + + with pytest.raises(ValueError): + validate_population_size(1) + + def test_above_cap(self) -> None: + from soup_cli.utils.cmaes_merge import MAX_POPULATION, validate_population_size + + with pytest.raises(ValueError): + validate_population_size(MAX_POPULATION + 1) + + def test_non_int_rejected(self) -> None: + from soup_cli.utils.cmaes_merge import validate_population_size + + with pytest.raises(TypeError): + validate_population_size("8") # type: ignore[arg-type] + + +class TestValidateGenerations: + def test_happy(self) -> None: + from soup_cli.utils.cmaes_merge import validate_generations + + assert validate_generations(20) == 20 + + def test_bool_rejected(self) -> None: + from soup_cli.utils.cmaes_merge import validate_generations + + with pytest.raises(TypeError): + validate_generations(False) # type: ignore[arg-type] + + def test_below_floor(self) -> None: + from soup_cli.utils.cmaes_merge import validate_generations + + with pytest.raises(ValueError): + validate_generations(0) + + def test_above_cap(self) -> None: + from soup_cli.utils.cmaes_merge import MAX_GENERATIONS, validate_generations + + with pytest.raises(ValueError): + validate_generations(MAX_GENERATIONS + 1) + + +# ----------------------------------------------------------------------------- +# Frozen dataclasses +# ----------------------------------------------------------------------------- + + +class TestCmaesPlan: + def test_construct(self) -> None: + from soup_cli.utils.cmaes_merge import CmaesPlan + + plan = CmaesPlan( + adapters=("a", "b"), + eval_suite="suite.yaml", + budget_seconds=3600, + population_size=8, + max_generations=20, + seed=42, + ) + assert plan.population_size == 8 + assert plan.eval_suite == "suite.yaml" + + def test_frozen(self) -> None: + from soup_cli.utils.cmaes_merge import CmaesPlan + + plan = CmaesPlan( + adapters=("a", "b"), + eval_suite="s.yaml", + budget_seconds=600, + population_size=4, + max_generations=5, + seed=0, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + plan.population_size = 99 # type: ignore[misc] + + +class TestCmaesResult: + def test_construct(self) -> None: + from soup_cli.utils.cmaes_merge import CmaesResult + + result = CmaesResult( + best_weights=(0.5, 0.5), + best_score=0.93, + generations_run=10, + evaluations=80, + wall_clock_seconds=120.5, + converged=True, + history=(0.85, 0.87, 0.9, 0.93), + ) + assert result.best_score == 0.93 + assert result.generations_run == 10 + assert result.converged is True + + def test_frozen(self) -> None: + from soup_cli.utils.cmaes_merge import CmaesResult + + result = CmaesResult( + best_weights=(0.5, 0.5), + best_score=0.5, + generations_run=1, + evaluations=4, + wall_clock_seconds=1.0, + converged=False, + history=(0.5,), + ) + with pytest.raises(dataclasses.FrozenInstanceError): + result.best_score = 9.99 # type: ignore[misc] + + def test_weights_must_be_simplex(self) -> None: + from soup_cli.utils.cmaes_merge import CmaesResult + + with pytest.raises(ValueError): + CmaesResult( + best_weights=(0.3, 0.3), # sums to 0.6, not 1.0 + best_score=0.5, + generations_run=1, + evaluations=4, + wall_clock_seconds=1.0, + converged=False, + history=(0.5,), + ) + + def test_bool_score_rejected(self) -> None: + from soup_cli.utils.cmaes_merge import CmaesResult + + with pytest.raises(TypeError): + CmaesResult( + best_weights=(0.5, 0.5), + best_score=True, # type: ignore[arg-type] + generations_run=1, + evaluations=4, + wall_clock_seconds=1.0, + converged=False, + history=(0.5,), + ) + + def test_non_finite_score_rejected(self) -> None: + from soup_cli.utils.cmaes_merge import CmaesResult + + with pytest.raises(ValueError): + CmaesResult( + best_weights=(0.5, 0.5), + best_score=math.nan, + generations_run=1, + evaluations=4, + wall_clock_seconds=1.0, + converged=False, + history=(0.5,), + ) + + +# ----------------------------------------------------------------------------- +# build_cmaes_plan +# ----------------------------------------------------------------------------- + + +class TestBuildCmaesPlan: + def test_happy(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.cmaes_merge import build_cmaes_plan + + monkeypatch.chdir(tmp_path) + adapter_a = tmp_path / "a" + adapter_a.mkdir() + adapter_b = tmp_path / "b" + adapter_b.mkdir() + suite = tmp_path / "suite.yaml" + suite.write_text("dummy: 1\n", encoding="utf-8") + + plan = build_cmaes_plan( + adapters=[str(adapter_a), str(adapter_b)], + eval_suite=str(suite), + budget_spec="10m", + population_size=6, + max_generations=15, + seed=42, + ) + assert plan.population_size == 6 + assert plan.budget_seconds == 600 + assert plan.max_generations == 15 + + def test_invalid_budget_string(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.cmaes_merge import build_cmaes_plan + + monkeypatch.chdir(tmp_path) + for name in ("a", "b"): + (tmp_path / name).mkdir() + suite = tmp_path / "s.yaml" + suite.write_text("x: 1\n", encoding="utf-8") + + with pytest.raises(ValueError): + build_cmaes_plan( + adapters=[str(tmp_path / "a"), str(tmp_path / "b")], + eval_suite=str(suite), + budget_spec="not-a-budget", + population_size=4, + max_generations=5, + seed=0, + ) + + def test_below_min_adapters(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.cmaes_merge import build_cmaes_plan + + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + suite = tmp_path / "s.yaml" + suite.write_text("x: 1\n", encoding="utf-8") + + with pytest.raises(ValueError): + build_cmaes_plan( + adapters=[str(tmp_path / "a")], + eval_suite=str(suite), + budget_spec="60s", + population_size=4, + max_generations=5, + seed=0, + ) + + def test_eval_suite_outside_cwd(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.cmaes_merge import build_cmaes_plan + + cwd = tmp_path / "work" + cwd.mkdir() + monkeypatch.chdir(cwd) + (cwd / "a").mkdir() + (cwd / "b").mkdir() + outside = tmp_path / "outside.yaml" + outside.write_text("x: 1\n", encoding="utf-8") + + with pytest.raises(ValueError): + build_cmaes_plan( + adapters=[str(cwd / "a"), str(cwd / "b")], + eval_suite=str(outside), + budget_spec="60s", + population_size=4, + max_generations=5, + seed=0, + ) + + +# ----------------------------------------------------------------------------- +# run_cmaes_merge orchestrator — operator-supplied eval_fn +# ----------------------------------------------------------------------------- + + +class TestRunCmaesMerge: + def test_returns_result(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge + + monkeypatch.chdir(tmp_path) + for name in ("a", "b"): + (tmp_path / name).mkdir() + suite = tmp_path / "s.yaml" + suite.write_text("x: 1\n", encoding="utf-8") + + plan = build_cmaes_plan( + adapters=[str(tmp_path / "a"), str(tmp_path / "b")], + eval_suite=str(suite), + budget_spec="60s", + population_size=4, + max_generations=3, + seed=42, + ) + + # eval_fn returns a deterministic score with maximum near weights=(0.5, 0.5) + def eval_fn(weights): + target = (0.5, 0.5) + diff = sum((w - t) ** 2 for w, t in zip(weights, target)) + return 1.0 - diff + + result = run_cmaes_merge(plan, eval_fn=eval_fn) + assert result.generations_run >= 1 + assert result.evaluations >= 1 + assert 0.0 <= result.best_score <= 1.0 + # weights normalised to simplex + assert math.isclose(sum(result.best_weights), 1.0, abs_tol=1e-6) + + def test_eval_fn_required(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge + + monkeypatch.chdir(tmp_path) + for name in ("a", "b"): + (tmp_path / name).mkdir() + suite = tmp_path / "s.yaml" + suite.write_text("x: 1\n", encoding="utf-8") + + plan = build_cmaes_plan( + adapters=[str(tmp_path / "a"), str(tmp_path / "b")], + eval_suite=str(suite), + budget_spec="60s", + population_size=4, + max_generations=2, + seed=0, + ) + with pytest.raises(TypeError): + run_cmaes_merge(plan, eval_fn=None) # type: ignore[arg-type] + + def test_eval_fn_exceptions_logged(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge + + monkeypatch.chdir(tmp_path) + for name in ("a", "b"): + (tmp_path / name).mkdir() + suite = tmp_path / "s.yaml" + suite.write_text("x: 1\n", encoding="utf-8") + + plan = build_cmaes_plan( + adapters=[str(tmp_path / "a"), str(tmp_path / "b")], + eval_suite=str(suite), + budget_spec="60s", + population_size=4, + max_generations=2, + seed=0, + ) + + def eval_fn(weights): + if weights[0] > 0.8: + raise RuntimeError("simulated eval failure") + return 0.5 + + # Should not crash; failed evals get sentinel score + result = run_cmaes_merge(plan, eval_fn=eval_fn) + assert result.evaluations >= 1 + + def test_non_plan_rejected(self) -> None: + from soup_cli.utils.cmaes_merge import run_cmaes_merge + + with pytest.raises(TypeError): + run_cmaes_merge("not-a-plan", eval_fn=lambda w: 0.5) # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# adapter_merge integration: cmaes is a known strategy +# ----------------------------------------------------------------------------- + + +class TestAdapterMergeIntegration: + def test_cmaes_in_supported_strategies(self) -> None: + from soup_cli.utils.adapter_merge import SUPPORTED_STRATEGIES + + assert "cmaes" in SUPPORTED_STRATEGIES + + +# ----------------------------------------------------------------------------- +# CLI smoke +# ----------------------------------------------------------------------------- + + +class TestCliSmoke: + def test_merge_help_lists_cmaes(self) -> None: + from typer.testing import CliRunner + + from soup_cli.commands.adapters import app + + runner = CliRunner() + result = runner.invoke(app, ["merge", "--help"]) + assert result.exit_code == 0 + assert "cmaes" in result.output.lower() + + def test_merge_cmaes_requires_eval(self, tmp_path, monkeypatch) -> None: + """cmaes without --eval should exit 2 with friendly message.""" + from typer.testing import CliRunner + + from soup_cli.commands.adapters import app + + monkeypatch.chdir(tmp_path) + runner = CliRunner() + + # We don't bother making valid adapters; we just want to see the strategy + # validation reject missing --eval before file-handling kicks in. + result = runner.invoke( + app, + [ + "merge", + str(tmp_path / "a"), + str(tmp_path / "b"), + "--strategy", + "cmaes", + "--output", + str(tmp_path / "out"), + ], + ) + # --strategy cmaes WITHOUT --eval must fail + assert result.exit_code == 2 + assert "eval" in result.output.lower() + + +# ----------------------------------------------------------------------------- +# Source-grep regression guards +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_no_top_level_heavy_imports(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "cmaes_merge.py").read_text( + encoding="utf-8" + ) + # Only first 30 non-comment lines (matches v0.66 review-grep policy) + head_lines = [ + line + for line in src.splitlines()[:50] + if line.strip() and not line.strip().startswith("#") + ] + head = "\n".join(head_lines) + for forbidden in ("import torch", "import transformers", "import peft"): + assert forbidden not in head, f"top-level {forbidden!r} in cmaes_merge" + + def test_adapter_merge_lists_cmaes(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "adapter_merge.py").read_text( + encoding="utf-8" + ) + assert "cmaes" in src diff --git a/tests/test_v0670_part_b.py b/tests/test_v0670_part_b.py new file mode 100644 index 0000000..0bf31da --- /dev/null +++ b/tests/test_v0670_part_b.py @@ -0,0 +1,467 @@ +"""v0.67.0 Part B — VeRA / VB-LoRA vector-bank storage format. + +Tests for ``soup_cli/utils/vector_bank.py``: + +- Frozen ``VectorBank`` / ``BankEntry`` dataclasses (shared random + projection matrix + per-user MB-sized scaling vectors) +- Validation matrix on bank name, user id, scaling-vector shape +- ``estimate_bank_size`` for sizing decisions +- Stub-then-live: live serving wiring deferred to v0.67.1 +- Atomic disk I/O via shared ``atomic_write_text`` (TOCTOU-safe) +""" + +from __future__ import annotations + +import dataclasses +import math +import os + +import pytest + +# ----------------------------------------------------------------------------- +# Public surface +# ----------------------------------------------------------------------------- + + +class TestPublicSurface: + def test_module_importable(self) -> None: + from soup_cli.utils import vector_bank + + assert hasattr(vector_bank, "VectorBank") + assert hasattr(vector_bank, "BankEntry") + assert hasattr(vector_bank, "validate_bank_name") + assert hasattr(vector_bank, "validate_user_id") + assert hasattr(vector_bank, "validate_scaling_vector") + assert hasattr(vector_bank, "estimate_bank_size") + assert hasattr(vector_bank, "apply_bank_to_serve") + + +# ----------------------------------------------------------------------------- +# validate_bank_name +# ----------------------------------------------------------------------------- + + +class TestValidateBankName: + def test_happy(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + assert validate_bank_name("my-bank") == "my-bank" + + def test_case_insensitive(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + # Canonical: kebab-case, normalise to lowercase + assert validate_bank_name("MY-BANK") == "my-bank" + + def test_null_byte_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + with pytest.raises(ValueError): + validate_bank_name("bad\x00name") + + def test_empty_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + with pytest.raises(ValueError): + validate_bank_name("") + + def test_oversize_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + with pytest.raises(ValueError): + validate_bank_name("a" * 200) + + def test_non_string_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + with pytest.raises(TypeError): + validate_bank_name(123) # type: ignore[arg-type] + + def test_bool_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + with pytest.raises(TypeError): + validate_bank_name(True) # type: ignore[arg-type] + + def test_invalid_chars_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_bank_name + + with pytest.raises(ValueError): + validate_bank_name("path/traversal") + with pytest.raises(ValueError): + validate_bank_name("..") + with pytest.raises(ValueError): + validate_bank_name("name with space") + + +# ----------------------------------------------------------------------------- +# validate_user_id +# ----------------------------------------------------------------------------- + + +class TestValidateUserId: + def test_happy(self) -> None: + from soup_cli.utils.vector_bank import validate_user_id + + assert validate_user_id("user-1234") == "user-1234" + + def test_oversize_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_user_id + + with pytest.raises(ValueError): + validate_user_id("u" * 300) + + def test_null_byte_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_user_id + + with pytest.raises(ValueError): + validate_user_id("u\x00") + + def test_non_string_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_user_id + + with pytest.raises(TypeError): + validate_user_id(42) # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# validate_scaling_vector +# ----------------------------------------------------------------------------- + + +class TestValidateScalingVector: + def test_happy(self) -> None: + from soup_cli.utils.vector_bank import validate_scaling_vector + + v = validate_scaling_vector([0.1, 0.2, -0.3]) + assert v == (0.1, 0.2, -0.3) + + def test_returns_tuple(self) -> None: + from soup_cli.utils.vector_bank import validate_scaling_vector + + v = validate_scaling_vector([1.0, 2.0]) + assert isinstance(v, tuple) + + def test_empty_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_scaling_vector + + with pytest.raises(ValueError): + validate_scaling_vector([]) + + def test_non_iterable_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_scaling_vector + + with pytest.raises(TypeError): + validate_scaling_vector(42) # type: ignore[arg-type] + + def test_non_finite_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_scaling_vector + + with pytest.raises(ValueError): + validate_scaling_vector([1.0, math.nan]) + with pytest.raises(ValueError): + validate_scaling_vector([math.inf, 1.0]) + + def test_bool_in_vector_rejected(self) -> None: + from soup_cli.utils.vector_bank import validate_scaling_vector + + with pytest.raises(TypeError): + validate_scaling_vector([True, 0.5]) + + def test_oversize_rejected(self) -> None: + from soup_cli.utils.vector_bank import MAX_VECTOR_DIM, validate_scaling_vector + + with pytest.raises(ValueError): + validate_scaling_vector([0.1] * (MAX_VECTOR_DIM + 1)) + + +# ----------------------------------------------------------------------------- +# BankEntry + VectorBank frozen dataclasses +# ----------------------------------------------------------------------------- + + +class TestBankEntry: + def test_construct(self) -> None: + from soup_cli.utils.vector_bank import BankEntry + + entry = BankEntry(user_id="alice", scaling=(0.1, 0.2, 0.3)) + assert entry.user_id == "alice" + assert entry.scaling == (0.1, 0.2, 0.3) + + def test_frozen(self) -> None: + from soup_cli.utils.vector_bank import BankEntry + + entry = BankEntry(user_id="bob", scaling=(0.5,)) + with pytest.raises(dataclasses.FrozenInstanceError): + entry.user_id = "eve" # type: ignore[misc] + + +class TestVectorBank: + def test_construct(self) -> None: + from soup_cli.utils.vector_bank import BankEntry, VectorBank + + bank = VectorBank( + name="mybank", + base_model="meta-llama/Llama-3.1-8B", + projection_seed=42, + vector_dim=2, + entries=(BankEntry(user_id="u1", scaling=(0.1, 0.2)),), + ) + assert bank.name == "mybank" + assert bank.vector_dim == 2 + + def test_frozen(self) -> None: + from soup_cli.utils.vector_bank import VectorBank + + bank = VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=8, + entries=(), + ) + with pytest.raises(dataclasses.FrozenInstanceError): + bank.vector_dim = 99 # type: ignore[misc] + + def test_invalid_seed(self) -> None: + from soup_cli.utils.vector_bank import VectorBank + + with pytest.raises(TypeError): + VectorBank( + name="b", + base_model="m", + projection_seed=True, # type: ignore[arg-type] + vector_dim=8, + entries=(), + ) + + def test_invalid_vector_dim(self) -> None: + from soup_cli.utils.vector_bank import VectorBank + + with pytest.raises(ValueError): + VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=0, + entries=(), + ) + + def test_entries_must_be_tuple(self) -> None: + from soup_cli.utils.vector_bank import VectorBank + + with pytest.raises(TypeError): + VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=8, + entries=[], # type: ignore[arg-type] + ) + + def test_max_entries_enforced(self) -> None: + from soup_cli.utils.vector_bank import ( + MAX_ENTRIES_PER_BANK, + BankEntry, + VectorBank, + ) + + too_many = tuple( + BankEntry(user_id=f"u{i}", scaling=(0.1,)) + for i in range(MAX_ENTRIES_PER_BANK + 1) + ) + with pytest.raises(ValueError): + VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=1, + entries=too_many, + ) + + def test_base_model_validation(self) -> None: + from soup_cli.utils.vector_bank import VectorBank + + with pytest.raises(ValueError): + VectorBank( + name="b", + base_model="", + projection_seed=0, + vector_dim=8, + entries=(), + ) + + def test_per_entry_vector_dim_mismatch(self) -> None: + from soup_cli.utils.vector_bank import BankEntry, VectorBank + + with pytest.raises(ValueError): + VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=4, + entries=(BankEntry(user_id="u", scaling=(0.1, 0.2, 0.3)),), + ) + + +# ----------------------------------------------------------------------------- +# estimate_bank_size +# ----------------------------------------------------------------------------- + + +class TestEstimateBankSize: + def test_basic(self) -> None: + from soup_cli.utils.vector_bank import estimate_bank_size + + # 1000 users × 128 fp32 scaling vectors ≈ 512 KB + projection + # matrix (128×128×4 = 64 KB) + size_bytes = estimate_bank_size(num_users=1000, vector_dim=128) + assert 400_000 < size_bytes < 1_000_000 + + def test_zero_users(self) -> None: + from soup_cli.utils.vector_bank import estimate_bank_size + + # Only projection matrix + size_bytes = estimate_bank_size(num_users=0, vector_dim=128) + assert size_bytes > 0 # projection still allocated + + def test_bool_rejected(self) -> None: + from soup_cli.utils.vector_bank import estimate_bank_size + + with pytest.raises(TypeError): + estimate_bank_size(num_users=True, vector_dim=128) # type: ignore[arg-type] + + def test_negative_rejected(self) -> None: + from soup_cli.utils.vector_bank import estimate_bank_size + + with pytest.raises(ValueError): + estimate_bank_size(num_users=-1, vector_dim=128) + with pytest.raises(ValueError): + estimate_bank_size(num_users=10, vector_dim=0) + + +# ----------------------------------------------------------------------------- +# Atomic JSON write + read +# ----------------------------------------------------------------------------- + + +class TestWriteLoadBank: + def test_roundtrip(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.vector_bank import ( + BankEntry, + VectorBank, + load_bank, + write_bank, + ) + + monkeypatch.chdir(tmp_path) + bank = VectorBank( + name="round-trip", + base_model="meta-llama/Llama-3.1-8B", + projection_seed=42, + vector_dim=4, + entries=( + BankEntry(user_id="alice", scaling=(0.1, 0.2, 0.3, 0.4)), + BankEntry(user_id="bob", scaling=(0.5, 0.6, 0.7, 0.8)), + ), + ) + path = str(tmp_path / "bank.json") + write_bank(bank, path) + loaded = load_bank(path) + assert loaded.name == bank.name + assert loaded.entries == bank.entries + + def test_outside_cwd_rejected(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.vector_bank import VectorBank, write_bank + + cwd = tmp_path / "work" + cwd.mkdir() + monkeypatch.chdir(cwd) + bank = VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=1, + entries=(), + ) + # Path outside cwd + with pytest.raises(ValueError): + write_bank(bank, str(tmp_path / "outside.json")) + + def test_load_missing_file(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.vector_bank import load_bank + + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + load_bank(str(tmp_path / "missing.json")) + + @pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink test") + def test_load_symlink_rejected(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.vector_bank import load_bank + + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.json" + real.write_text('{"name": "x"}', encoding="utf-8") + sym = tmp_path / "sym.json" + os.symlink(real, sym) + with pytest.raises(ValueError): + load_bank(str(sym)) + + +# ----------------------------------------------------------------------------- +# Live-serve stub +# ----------------------------------------------------------------------------- + + +class TestApplyBankToServe: + def test_deferred_v0_67_1(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.vector_bank import VectorBank, apply_bank_to_serve + + bank = VectorBank( + name="b", + base_model="m", + projection_seed=0, + vector_dim=1, + entries=(), + ) + with pytest.raises(NotImplementedError) as exc_info: + apply_bank_to_serve(bank) + assert "v0.67.1" in str(exc_info.value) + + def test_non_bank_rejected(self) -> None: + from soup_cli.utils.vector_bank import apply_bank_to_serve + + with pytest.raises(TypeError): + apply_bank_to_serve("not-a-bank") # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# Source-grep regression guards +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_no_top_level_heavy_imports(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "vector_bank.py").read_text( + encoding="utf-8" + ) + head_lines = [ + line + for line in src.splitlines()[:50] + if line.strip() and not line.strip().startswith("#") + ] + head = "\n".join(head_lines) + for forbidden in ("import torch", "import transformers", "import peft"): + assert forbidden not in head, f"top-level {forbidden!r}" + + def test_uses_atomic_write_helper(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "vector_bank.py").read_text( + encoding="utf-8" + ) + assert "atomic_write_text" in src diff --git a/tests/test_v0670_part_c.py b/tests/test_v0670_part_c.py new file mode 100644 index 0000000..5778e01 --- /dev/null +++ b/tests/test_v0670_part_c.py @@ -0,0 +1,315 @@ +"""v0.67.0 Part C — MoLE per-token adapter routing. + +Tests for ``soup_cli/utils/mole_routing.py``: + +- Frozen ``MoleGatingConfig`` dataclass +- ``validate_mole_compat`` (task / backend / adapter-count gate) +- ``build_gating_kernel`` stub raises NotImplementedError with v0.67.1 marker +- New ``task='moe_lora_routing'`` Literal on SoupConfig +- SoupConfig cross-validator gates mole config to the correct task/backend +""" + +from __future__ import annotations + +import dataclasses +import math + +import pytest + +# ----------------------------------------------------------------------------- +# Public surface +# ----------------------------------------------------------------------------- + + +class TestPublicSurface: + def test_module_importable(self) -> None: + from soup_cli.utils import mole_routing + + assert hasattr(mole_routing, "MoleGatingConfig") + assert hasattr(mole_routing, "validate_mole_compat") + assert hasattr(mole_routing, "build_gating_kernel") + assert hasattr(mole_routing, "MIN_TASK_ADAPTERS") + assert hasattr(mole_routing, "MAX_TASK_ADAPTERS") + + def test_constants_immutable(self) -> None: + from soup_cli.utils import mole_routing + + assert mole_routing.MIN_TASK_ADAPTERS >= 2 + assert mole_routing.MAX_TASK_ADAPTERS <= 64 + + +# ----------------------------------------------------------------------------- +# MoleGatingConfig +# ----------------------------------------------------------------------------- + + +class TestMoleGatingConfig: + def test_construct(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + cfg = MoleGatingConfig( + num_task_adapters=4, + hidden_dim=128, + temperature=1.0, + top_k=2, + ) + assert cfg.num_task_adapters == 4 + assert cfg.top_k == 2 + + def test_frozen(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + cfg = MoleGatingConfig( + num_task_adapters=4, + hidden_dim=128, + temperature=1.0, + top_k=2, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + cfg.top_k = 99 # type: ignore[misc] + + def test_num_task_adapters_below_floor(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + with pytest.raises(ValueError): + MoleGatingConfig( + num_task_adapters=1, hidden_dim=128, temperature=1.0, top_k=1 + ) + + def test_num_task_adapters_above_cap(self) -> None: + from soup_cli.utils.mole_routing import ( + MAX_TASK_ADAPTERS, + MoleGatingConfig, + ) + + with pytest.raises(ValueError): + MoleGatingConfig( + num_task_adapters=MAX_TASK_ADAPTERS + 1, + hidden_dim=128, + temperature=1.0, + top_k=1, + ) + + def test_bool_rejected(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + with pytest.raises(TypeError): + MoleGatingConfig( + num_task_adapters=True, # type: ignore[arg-type] + hidden_dim=128, + temperature=1.0, + top_k=1, + ) + + def test_hidden_dim_must_be_positive(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + with pytest.raises(ValueError): + MoleGatingConfig( + num_task_adapters=4, + hidden_dim=0, + temperature=1.0, + top_k=1, + ) + + def test_temperature_non_finite_rejected(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + with pytest.raises(ValueError): + MoleGatingConfig( + num_task_adapters=4, + hidden_dim=128, + temperature=math.nan, + top_k=1, + ) + + def test_temperature_non_positive_rejected(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + with pytest.raises(ValueError): + MoleGatingConfig( + num_task_adapters=4, + hidden_dim=128, + temperature=0.0, + top_k=1, + ) + + def test_top_k_above_num_adapters_rejected(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + with pytest.raises(ValueError): + MoleGatingConfig( + num_task_adapters=3, + hidden_dim=128, + temperature=1.0, + top_k=5, + ) + + def test_top_k_below_one_rejected(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig + + with pytest.raises(ValueError): + MoleGatingConfig( + num_task_adapters=4, + hidden_dim=128, + temperature=1.0, + top_k=0, + ) + + +# ----------------------------------------------------------------------------- +# validate_mole_compat +# ----------------------------------------------------------------------------- + + +class TestValidateMoleCompat: + def test_happy_path(self) -> None: + from soup_cli.utils.mole_routing import validate_mole_compat + + validate_mole_compat( + task="moe_lora_routing", + backend="transformers", + num_task_adapters=4, + ) + + def test_wrong_task_rejected(self) -> None: + from soup_cli.utils.mole_routing import validate_mole_compat + + with pytest.raises(ValueError) as exc_info: + validate_mole_compat( + task="sft", + backend="transformers", + num_task_adapters=4, + ) + assert "moe_lora_routing" in str(exc_info.value) + + def test_mlx_rejected(self) -> None: + from soup_cli.utils.mole_routing import validate_mole_compat + + with pytest.raises(ValueError) as exc_info: + validate_mole_compat( + task="moe_lora_routing", + backend="mlx", + num_task_adapters=4, + ) + assert "mlx" in str(exc_info.value).lower() + + def test_too_few_adapters(self) -> None: + from soup_cli.utils.mole_routing import validate_mole_compat + + with pytest.raises(ValueError): + validate_mole_compat( + task="moe_lora_routing", + backend="transformers", + num_task_adapters=1, + ) + + def test_bool_args_rejected(self) -> None: + from soup_cli.utils.mole_routing import validate_mole_compat + + with pytest.raises(TypeError): + validate_mole_compat( + task=True, # type: ignore[arg-type] + backend="transformers", + num_task_adapters=4, + ) + with pytest.raises(TypeError): + validate_mole_compat( + task="moe_lora_routing", + backend=True, # type: ignore[arg-type] + num_task_adapters=4, + ) + + def test_null_byte_args_rejected(self) -> None: + from soup_cli.utils.mole_routing import validate_mole_compat + + with pytest.raises(ValueError): + validate_mole_compat( + task="moe_lora_routing\x00", + backend="transformers", + num_task_adapters=4, + ) + + +# ----------------------------------------------------------------------------- +# Deferred-live stub +# ----------------------------------------------------------------------------- + + +class TestBuildGatingKernel: + def test_deferred_v0_67_1(self) -> None: + from soup_cli.utils.mole_routing import MoleGatingConfig, build_gating_kernel + + cfg = MoleGatingConfig( + num_task_adapters=4, hidden_dim=128, temperature=1.0, top_k=2 + ) + with pytest.raises(NotImplementedError) as exc_info: + build_gating_kernel(cfg) + assert "v0.67.1" in str(exc_info.value) + + def test_non_config_rejected(self) -> None: + from soup_cli.utils.mole_routing import build_gating_kernel + + with pytest.raises(TypeError): + build_gating_kernel("not-a-config") # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# SoupConfig integration — task='moe_lora_routing' Literal +# ----------------------------------------------------------------------------- + + +class TestSchemaIntegration: + def test_task_accepted_in_literal(self) -> None: + from soup_cli.config.loader import load_config_from_string + + yaml = """ +base: meta-llama/Llama-3.1-8B +task: moe_lora_routing +backend: transformers +modality: text +data: + train: data.jsonl + format: chatml +""" + cfg = load_config_from_string(yaml) + assert cfg.task == "moe_lora_routing" + + def test_task_rejected_on_mlx(self) -> None: + from soup_cli.config.loader import load_config_from_string + + yaml = """ +base: m +task: moe_lora_routing +backend: mlx +modality: text +data: + train: data.jsonl + format: chatml +""" + with pytest.raises(Exception) as exc_info: + load_config_from_string(yaml) + assert "mlx" in str(exc_info.value).lower() + + +# ----------------------------------------------------------------------------- +# Source-grep regression +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_no_top_level_heavy_imports(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "mole_routing.py").read_text( + encoding="utf-8" + ) + head_lines = [ + line + for line in src.splitlines()[:50] + if line.strip() and not line.strip().startswith("#") + ] + head = "\n".join(head_lines) + for forbidden in ("import torch", "import transformers", "import peft"): + assert forbidden not in head, f"top-level {forbidden!r}" diff --git a/tests/test_v0670_part_d.py b/tests/test_v0670_part_d.py new file mode 100644 index 0000000..8f1d590 --- /dev/null +++ b/tests/test_v0670_part_d.py @@ -0,0 +1,389 @@ +"""v0.67.0 Part D — ``soup adapters pr`` (GitHub-shaped PR rendering). + +Tests for ``soup_cli/utils/adapter_pr.py``: + +- Frozen ``EvalDelta`` / ``SampleDiff`` / ``AdapterPR`` dataclasses +- ``build_adapter_pr`` factory with validation +- ``render_pr_markdown`` (eval-delta tables + sample diffs) +- ``render_pr_json`` round-trip +- Rich-markup escape on every operator-controlled field +- CLI smoke (`soup adapters pr`) +""" + +from __future__ import annotations + +import dataclasses +import math + +import pytest + +# ----------------------------------------------------------------------------- +# Public surface +# ----------------------------------------------------------------------------- + + +class TestPublicSurface: + def test_module_importable(self) -> None: + from soup_cli.utils import adapter_pr + + assert hasattr(adapter_pr, "AdapterPR") + assert hasattr(adapter_pr, "EvalDelta") + assert hasattr(adapter_pr, "SampleDiff") + assert hasattr(adapter_pr, "build_adapter_pr") + assert hasattr(adapter_pr, "render_pr_markdown") + assert hasattr(adapter_pr, "render_pr_json") + assert hasattr(adapter_pr, "write_pr_markdown") + + +# ----------------------------------------------------------------------------- +# EvalDelta +# ----------------------------------------------------------------------------- + + +class TestEvalDelta: + def test_construct(self) -> None: + from soup_cli.utils.adapter_pr import EvalDelta + + delta = EvalDelta(metric="accuracy", baseline=0.7, candidate=0.85) + assert math.isclose(delta.delta, 0.15) + + def test_frozen(self) -> None: + from soup_cli.utils.adapter_pr import EvalDelta + + d = EvalDelta(metric="m", baseline=0.5, candidate=0.6) + with pytest.raises(dataclasses.FrozenInstanceError): + d.candidate = 0.99 # type: ignore[misc] + + def test_non_finite_rejected(self) -> None: + from soup_cli.utils.adapter_pr import EvalDelta + + with pytest.raises(ValueError): + EvalDelta(metric="m", baseline=math.nan, candidate=0.5) + with pytest.raises(ValueError): + EvalDelta(metric="m", baseline=0.5, candidate=math.inf) + + def test_bool_rejected(self) -> None: + from soup_cli.utils.adapter_pr import EvalDelta + + with pytest.raises(TypeError): + EvalDelta(metric="m", baseline=True, candidate=0.5) # type: ignore[arg-type] + + def test_metric_null_byte_rejected(self) -> None: + from soup_cli.utils.adapter_pr import EvalDelta + + with pytest.raises(ValueError): + EvalDelta(metric="m\x00", baseline=0.5, candidate=0.6) + + def test_metric_oversize_rejected(self) -> None: + from soup_cli.utils.adapter_pr import EvalDelta + + with pytest.raises(ValueError): + EvalDelta(metric="a" * 300, baseline=0.5, candidate=0.6) + + +# ----------------------------------------------------------------------------- +# SampleDiff +# ----------------------------------------------------------------------------- + + +class TestSampleDiff: + def test_construct(self) -> None: + from soup_cli.utils.adapter_pr import SampleDiff + + diff = SampleDiff( + prompt="What is 2+2?", + baseline_output="four", + candidate_output="4", + ) + assert diff.prompt.startswith("What") + + def test_frozen(self) -> None: + from soup_cli.utils.adapter_pr import SampleDiff + + d = SampleDiff(prompt="p", baseline_output="b", candidate_output="c") + with pytest.raises(dataclasses.FrozenInstanceError): + d.prompt = "new" # type: ignore[misc] + + def test_null_byte_rejected(self) -> None: + from soup_cli.utils.adapter_pr import SampleDiff + + with pytest.raises(ValueError): + SampleDiff(prompt="p\x00", baseline_output="b", candidate_output="c") + + def test_oversize_truncated(self) -> None: + from soup_cli.utils.adapter_pr import MAX_OUTPUT_LEN, SampleDiff + + # >MAX_OUTPUT_LEN should be rejected to keep PRs reviewable + with pytest.raises(ValueError): + SampleDiff( + prompt="p" * 10, + baseline_output="x" * (MAX_OUTPUT_LEN + 1), + candidate_output="y", + ) + + +# ----------------------------------------------------------------------------- +# AdapterPR +# ----------------------------------------------------------------------------- + + +class TestAdapterPR: + def test_construct(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR, EvalDelta, SampleDiff + + pr = AdapterPR( + title="add-customer-support-tone", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="+ 100 rows of support data\n", + deltas=(EvalDelta(metric="accuracy", baseline=0.7, candidate=0.85),), + samples=( + SampleDiff(prompt="hi", baseline_output="hello", candidate_output="hey"), + ), + ) + assert pr.title == "add-customer-support-tone" + + def test_frozen(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR + + pr = AdapterPR( + title="t", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + with pytest.raises(dataclasses.FrozenInstanceError): + pr.title = "new" # type: ignore[misc] + + def test_base_sha_64_hex(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR + + # Not 64 hex chars + with pytest.raises(ValueError): + AdapterPR( + title="t", + base_sha="abc", + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + # Non-hex characters + with pytest.raises(ValueError): + AdapterPR( + title="t", + base_sha="z" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + + def test_title_validation(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR + + with pytest.raises(ValueError): + AdapterPR( + title="", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + + def test_deltas_must_be_tuple(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR + + with pytest.raises(TypeError): + AdapterPR( + title="t", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=[], # type: ignore[arg-type] + samples=(), + ) + + +# ----------------------------------------------------------------------------- +# build_adapter_pr factory +# ----------------------------------------------------------------------------- + + +class TestBuildAdapterPR: + def test_happy(self) -> None: + from soup_cli.utils.adapter_pr import build_adapter_pr + + pr = build_adapter_pr( + title="my-pr", + base_sha="b" * 64, + adapter_path="adapter/", + dataset_diff="diff text", + deltas=[ + {"metric": "accuracy", "baseline": 0.7, "candidate": 0.85}, + ], + samples=[ + {"prompt": "p", "baseline_output": "a", "candidate_output": "b"}, + ], + ) + assert pr.title == "my-pr" + assert len(pr.deltas) == 1 + + +# ----------------------------------------------------------------------------- +# render_pr_markdown / render_pr_json +# ----------------------------------------------------------------------------- + + +class TestRenderPR: + def test_markdown_structure(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR, EvalDelta, SampleDiff, render_pr_markdown + + pr = AdapterPR( + title="my-pr", + base_sha="c" * 64, + adapter_path="adapter/", + dataset_diff="+row1\n+row2\n", + deltas=(EvalDelta(metric="accuracy", baseline=0.7, candidate=0.85),), + samples=( + SampleDiff(prompt="hi", baseline_output="a", candidate_output="b"), + ), + ) + md = render_pr_markdown(pr) + assert "my-pr" in md + assert "accuracy" in md + assert "0.70" in md or "0.7" in md + # Should include a sample diff section + assert "hi" in md + # Should include the dataset diff + assert "+row1" in md + + def test_markdown_escapes_markdown_metacharacters(self) -> None: + from soup_cli.utils.adapter_pr import AdapterPR, EvalDelta, render_pr_markdown + + # Embed pipe character in metric name; rendered table cell must escape it + pr = AdapterPR( + title="t", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(EvalDelta(metric="acc|injection", baseline=0.5, candidate=0.6),), + samples=(), + ) + md = render_pr_markdown(pr) + # The pipe must be escaped to avoid breaking the table + assert "acc\\|injection" in md or "acc|injection" not in md.split("|") + + def test_json_roundtrip(self) -> None: + import json + + from soup_cli.utils.adapter_pr import ( + AdapterPR, + EvalDelta, + render_pr_json, + ) + + pr = AdapterPR( + title="t", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(EvalDelta(metric="m", baseline=0.5, candidate=0.6),), + samples=(), + ) + text = render_pr_json(pr) + data = json.loads(text) + assert data["title"] == "t" + assert data["deltas"][0]["metric"] == "m" + + def test_non_pr_rejected(self) -> None: + from soup_cli.utils.adapter_pr import render_pr_json, render_pr_markdown + + with pytest.raises(TypeError): + render_pr_markdown("not-a-pr") # type: ignore[arg-type] + with pytest.raises(TypeError): + render_pr_json("not-a-pr") # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# Atomic write +# ----------------------------------------------------------------------------- + + +class TestWritePR: + def test_write_markdown(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.adapter_pr import AdapterPR, write_pr_markdown + + monkeypatch.chdir(tmp_path) + pr = AdapterPR( + title="t", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + out = tmp_path / "pr.md" + write_pr_markdown(pr, str(out)) + assert out.exists() + assert "t" in out.read_text(encoding="utf-8") + + def test_write_outside_cwd_rejected(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.adapter_pr import AdapterPR, write_pr_markdown + + cwd = tmp_path / "work" + cwd.mkdir() + monkeypatch.chdir(cwd) + pr = AdapterPR( + title="t", + base_sha="a" * 64, + adapter_path="adapter/", + dataset_diff="", + deltas=(), + samples=(), + ) + with pytest.raises(ValueError): + write_pr_markdown(pr, str(tmp_path / "outside.md")) + + +# ----------------------------------------------------------------------------- +# CLI smoke +# ----------------------------------------------------------------------------- + + +class TestCliSmoke: + def test_pr_help(self) -> None: + from typer.testing import CliRunner + + from soup_cli.commands.adapters import app + + runner = CliRunner() + result = runner.invoke(app, ["pr", "--help"]) + assert result.exit_code == 0 + assert "pr" in result.output.lower() + + +# ----------------------------------------------------------------------------- +# Source-grep regression +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_no_top_level_heavy_imports(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "adapter_pr.py").read_text( + encoding="utf-8" + ) + head_lines = [ + line + for line in src.splitlines()[:50] + if line.strip() and not line.strip().startswith("#") + ] + head = "\n".join(head_lines) + for forbidden in ("import torch", "import transformers", "import peft"): + assert forbidden not in head, f"top-level {forbidden!r}" diff --git a/tests/test_v0670_part_e.py b/tests/test_v0670_part_e.py new file mode 100644 index 0000000..0460bf3 --- /dev/null +++ b/tests/test_v0670_part_e.py @@ -0,0 +1,321 @@ +"""v0.67.0 Part E — ``soup.lock`` shared run lockfile. + +Tests for ``soup_cli/utils/soup_lock.py``: + +- Frozen ``SoupLock`` dataclass (model SHA + dataset hash + env hash) +- ``compute_lock_closure`` deterministic + content-sensitive +- ``write_lock`` / ``read_lock`` atomic round-trip + cwd containment +- ``check_lock_drift`` detects each independent field drift +- Composes with v0.64 Part C ``soup env`` (env_hash is operator-supplied) +- CLI smoke (`soup lock`) +""" + +from __future__ import annotations + +import dataclasses +import os + +import pytest + + +class TestPublicSurface: + def test_module_importable(self) -> None: + from soup_cli.utils import soup_lock + + assert hasattr(soup_lock, "SoupLock") + assert hasattr(soup_lock, "LockDrift") + assert hasattr(soup_lock, "compute_lock_closure") + assert hasattr(soup_lock, "write_lock") + assert hasattr(soup_lock, "read_lock") + assert hasattr(soup_lock, "check_lock_drift") + + +class TestComputeLockClosure: + def test_deterministic(self) -> None: + from soup_cli.utils.soup_lock import compute_lock_closure + + a = compute_lock_closure( + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + ) + b = compute_lock_closure( + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + ) + assert a == b + + def test_content_sensitive(self) -> None: + from soup_cli.utils.soup_lock import compute_lock_closure + + a = compute_lock_closure( + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + ) + b = compute_lock_closure( + base_model_sha="f" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + ) + assert a != b + + def test_invalid_sha_rejected(self) -> None: + from soup_cli.utils.soup_lock import compute_lock_closure + + with pytest.raises(ValueError): + compute_lock_closure( + base_model_sha="not-hex", + dataset_sha="b" * 64, + env_hash="c" * 64, + ) + + def test_bool_rejected(self) -> None: + from soup_cli.utils.soup_lock import compute_lock_closure + + with pytest.raises(TypeError): + compute_lock_closure( + base_model_sha=True, # type: ignore[arg-type] + dataset_sha="b" * 64, + env_hash="c" * 64, + ) + + +class TestSoupLock: + def test_construct(self) -> None: + from soup_cli.utils.soup_lock import SoupLock + + lock = SoupLock( + soup_version="0.67.0", + base_model="meta-llama/Llama-3.1-8B", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24T00:00:00Z", + ) + assert lock.soup_version == "0.67.0" + + def test_frozen(self) -> None: + from soup_cli.utils.soup_lock import SoupLock + + lock = SoupLock( + soup_version="0.67.0", + base_model="m", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + lock.soup_version = "9.99" # type: ignore[misc] + + def test_sha_validation(self) -> None: + from soup_cli.utils.soup_lock import SoupLock + + with pytest.raises(ValueError): + SoupLock( + soup_version="0.67.0", + base_model="m", + base_model_sha="too-short", + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24", + ) + + def test_empty_base_model_rejected(self) -> None: + from soup_cli.utils.soup_lock import SoupLock + + with pytest.raises(ValueError): + SoupLock( + soup_version="0.67.0", + base_model="", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24", + ) + + +class TestWriteReadLock: + def test_roundtrip(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.soup_lock import ( + SoupLock, + read_lock, + write_lock, + ) + + monkeypatch.chdir(tmp_path) + lock = SoupLock( + soup_version="0.67.0", + base_model="m", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24", + ) + path = str(tmp_path / "soup.lock") + write_lock(lock, path) + loaded = read_lock(path) + assert loaded == lock + + def test_outside_cwd_rejected(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.soup_lock import SoupLock, write_lock + + cwd = tmp_path / "work" + cwd.mkdir() + monkeypatch.chdir(cwd) + lock = SoupLock( + soup_version="0.67.0", + base_model="m", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24", + ) + with pytest.raises(ValueError): + write_lock(lock, str(tmp_path / "outside.lock")) + + def test_read_missing(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.soup_lock import read_lock + + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + read_lock(str(tmp_path / "missing.lock")) + + @pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink test") + def test_read_symlink_rejected(self, tmp_path, monkeypatch) -> None: + from soup_cli.utils.soup_lock import read_lock + + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.lock" + real.write_text('{"soup_version": "0.67.0"}', encoding="utf-8") + sym = tmp_path / "sym.lock" + os.symlink(real, sym) + with pytest.raises(ValueError): + read_lock(str(sym)) + + +class TestCheckLockDrift: + def _make_lock(self, **overrides): + from soup_cli.utils.soup_lock import SoupLock + + base = dict( + soup_version="0.67.0", + base_model="m", + base_model_sha="a" * 64, + dataset_sha="b" * 64, + env_hash="c" * 64, + closure_sha="d" * 64, + created_at="2026-05-24", + ) + base.update(overrides) + return SoupLock(**base) + + def test_no_drift(self) -> None: + from soup_cli.utils.soup_lock import check_lock_drift + + a = self._make_lock() + b = self._make_lock() + drift = check_lock_drift(a, b) + assert drift.ok is True + assert drift.changes == () + + def test_base_model_sha_drift(self) -> None: + from soup_cli.utils.soup_lock import check_lock_drift + + a = self._make_lock() + b = self._make_lock(base_model_sha="f" * 64) + drift = check_lock_drift(a, b) + assert drift.ok is False + assert any("base_model_sha" in c for c in drift.changes) + + def test_dataset_sha_drift(self) -> None: + from soup_cli.utils.soup_lock import check_lock_drift + + a = self._make_lock() + b = self._make_lock(dataset_sha="f" * 64) + drift = check_lock_drift(a, b) + assert drift.ok is False + assert any("dataset_sha" in c for c in drift.changes) + + def test_env_hash_drift(self) -> None: + from soup_cli.utils.soup_lock import check_lock_drift + + a = self._make_lock() + b = self._make_lock(env_hash="f" * 64) + drift = check_lock_drift(a, b) + assert drift.ok is False + assert any("env_hash" in c for c in drift.changes) + + def test_non_lock_rejected(self) -> None: + from soup_cli.utils.soup_lock import check_lock_drift + + a = self._make_lock() + with pytest.raises(TypeError): + check_lock_drift(a, "not-a-lock") # type: ignore[arg-type] + + +class TestCliSmoke: + def test_lock_help(self) -> None: + from typer.testing import CliRunner + + from soup_cli.commands.lock import app + + runner = CliRunner() + result = runner.invoke(app, ["--help"]) + assert result.exit_code == 0 + + def test_lock_write_command(self, tmp_path, monkeypatch) -> None: + from typer.testing import CliRunner + + from soup_cli.commands.lock import app + + monkeypatch.chdir(tmp_path) + runner = CliRunner() + result = runner.invoke( + app, + [ + "write", + "--base-model", "test-model", + "--base-sha", "a" * 64, + "--dataset-sha", "b" * 64, + "--env-hash", "c" * 64, + "--output", "soup.lock", + ], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "soup.lock").exists() + + +class TestSourceWiring: + def test_no_top_level_heavy_imports(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "soup_lock.py").read_text( + encoding="utf-8" + ) + head_lines = [ + line + for line in src.splitlines()[:50] + if line.strip() and not line.strip().startswith("#") + ] + head = "\n".join(head_lines) + for forbidden in ("import torch", "import transformers", "import peft"): + assert forbidden not in head, f"top-level {forbidden!r}" + + def test_uses_atomic_write_helper(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "soup_lock.py").read_text( + encoding="utf-8" + ) + assert "atomic_write_text" in src diff --git a/tests/test_v0670_part_f.py b/tests/test_v0670_part_f.py new file mode 100644 index 0000000..ffb6378 --- /dev/null +++ b/tests/test_v0670_part_f.py @@ -0,0 +1,232 @@ +"""v0.67.0 Part F — ``soup adapters bisect``. + +Binary-search over a training-step history (or dataset-commit history) +to find the step that broke an eval. + +Tests for ``soup_cli/utils/adapter_bisect.py``: + +- Frozen ``BisectPlan`` / ``BisectStep`` / ``BisectResult`` dataclasses +- ``build_bisect_plan(history)`` validates input + computes initial mid +- ``run_bisect(plan, eval_fn)`` runs the binary search +- ``bisect_next_step(state)`` pure step kernel (one iteration) +- CLI smoke +""" + +from __future__ import annotations + +import dataclasses + +import pytest + + +class TestPublicSurface: + def test_module_importable(self) -> None: + from soup_cli.utils import adapter_bisect + + assert hasattr(adapter_bisect, "BisectPlan") + assert hasattr(adapter_bisect, "BisectStep") + assert hasattr(adapter_bisect, "BisectResult") + assert hasattr(adapter_bisect, "build_bisect_plan") + assert hasattr(adapter_bisect, "run_bisect") + assert hasattr(adapter_bisect, "bisect_next_step") + + +class TestBisectPlan: + def test_construct(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + plan = BisectPlan(history=("ckpt-100", "ckpt-200", "ckpt-300")) + assert len(plan.history) == 3 + + def test_frozen(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + plan = BisectPlan(history=("a", "b")) + with pytest.raises(dataclasses.FrozenInstanceError): + plan.history = () # type: ignore[misc] + + def test_too_short_rejected(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + with pytest.raises(ValueError): + BisectPlan(history=("only-one",)) + + def test_must_be_tuple(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + with pytest.raises(TypeError): + BisectPlan(history=["a", "b"]) # type: ignore[arg-type] + + def test_entries_must_be_str(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + with pytest.raises(TypeError): + BisectPlan(history=("a", 2)) # type: ignore[arg-type] + + def test_empty_entry_rejected(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + with pytest.raises(ValueError): + BisectPlan(history=("a", "")) + + def test_null_byte_rejected(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + with pytest.raises(ValueError): + BisectPlan(history=("a", "b\x00")) + + def test_duplicate_rejected(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan + + with pytest.raises(ValueError): + BisectPlan(history=("a", "b", "a")) + + def test_oversize_history_rejected(self) -> None: + from soup_cli.utils.adapter_bisect import MAX_HISTORY, BisectPlan + + too_many = tuple(f"ckpt-{i}" for i in range(MAX_HISTORY + 1)) + with pytest.raises(ValueError): + BisectPlan(history=too_many) + + +class TestBisectStep: + def test_construct(self) -> None: + from soup_cli.utils.adapter_bisect import BisectStep + + step = BisectStep(checkpoint="ckpt-100", ok=True) + assert step.checkpoint == "ckpt-100" + + def test_frozen(self) -> None: + from soup_cli.utils.adapter_bisect import BisectStep + + step = BisectStep(checkpoint="ckpt-1", ok=True) + with pytest.raises(dataclasses.FrozenInstanceError): + step.ok = False # type: ignore[misc] + + def test_ok_must_be_bool(self) -> None: + from soup_cli.utils.adapter_bisect import BisectStep + + with pytest.raises(TypeError): + BisectStep(checkpoint="ckpt", ok=1) # type: ignore[arg-type] + + +class TestBuildBisectPlan: + def test_happy(self) -> None: + from soup_cli.utils.adapter_bisect import build_bisect_plan + + plan = build_bisect_plan(["a", "b", "c", "d"]) + assert len(plan.history) == 4 + + +class TestBisectNextStep: + def test_picks_midpoint(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan, bisect_next_step + + plan = BisectPlan(history=("a", "b", "c", "d", "e")) + nxt = bisect_next_step(plan, lo=0, hi=4) + assert nxt == 2 + + def test_invalid_range(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan, bisect_next_step + + plan = BisectPlan(history=("a", "b")) + with pytest.raises(ValueError): + bisect_next_step(plan, lo=5, hi=0) + + +class TestRunBisect: + def test_finds_first_broken(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect + + # Eval says: first three checkpoints pass, last three fail. + # The bisect should find the boundary at index 3 (first failing). + plan = BisectPlan( + history=("c0", "c1", "c2", "c3", "c4", "c5") + ) + + def eval_fn(checkpoint: str) -> bool: + return checkpoint in ("c0", "c1", "c2") + + result = run_bisect(plan, eval_fn=eval_fn) + assert result.first_broken == "c3" + + def test_all_ok(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect + + plan = BisectPlan(history=("c0", "c1", "c2")) + result = run_bisect(plan, eval_fn=lambda _: True) + # All pass: first_broken is None + assert result.first_broken is None + assert result.verdict == "ALL_OK" + + def test_all_broken(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect + + plan = BisectPlan(history=("c0", "c1", "c2")) + result = run_bisect(plan, eval_fn=lambda _: False) + # All fail: first_broken is c0 + assert result.first_broken == "c0" + + def test_eval_fn_must_be_callable(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect + + plan = BisectPlan(history=("a", "b")) + with pytest.raises(TypeError): + run_bisect(plan, eval_fn=None) # type: ignore[arg-type] + + def test_non_plan_rejected(self) -> None: + from soup_cli.utils.adapter_bisect import run_bisect + + with pytest.raises(TypeError): + run_bisect("not-a-plan", eval_fn=lambda _: True) # type: ignore[arg-type] + + def test_logs_history_of_probes(self) -> None: + from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect + + plan = BisectPlan( + history=("c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7") + ) + + probed: list[str] = [] + + def eval_fn(checkpoint: str) -> bool: + probed.append(checkpoint) + # Boundary at c4 + return int(checkpoint[1:]) < 4 + + result = run_bisect(plan, eval_fn=eval_fn) + assert result.first_broken == "c4" + # Probes are logged in `result.steps` + assert len(result.steps) == len(probed) + # ~log2(8)=3 midpoint probes plus 2 endpoint probes = 5 total + assert len(result.steps) <= 6 + + +class TestCliSmoke: + def test_bisect_help(self) -> None: + from typer.testing import CliRunner + + from soup_cli.commands.adapters import app + + runner = CliRunner() + result = runner.invoke(app, ["bisect", "--help"]) + assert result.exit_code == 0 + assert "bisect" in result.output.lower() + + +class TestSourceWiring: + def test_no_top_level_heavy_imports(self) -> None: + from pathlib import Path + + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "adapter_bisect.py").read_text( + encoding="utf-8" + ) + head_lines = [ + line + for line in src.splitlines()[:50] + if line.strip() and not line.strip().startswith("#") + ] + head = "\n".join(head_lines) + for forbidden in ("import torch", "import transformers", "import peft"): + assert forbidden not in head, f"top-level {forbidden!r}"