diff --git a/README.md b/README.md index 38bcc25..1ead152 100644 --- a/README.md +++ b/README.md @@ -42,14 +42,14 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.56.0 — `soup diagnose`: lighthouse score for fine-tunes.** Training finishes and you have no idea *why* the adapter underperforms. Loss curves can't distinguish overfitting from refusal regression from mode collapse. v0.56 ships a 6-mode post-training report card with one verdict. +**v0.57.0 — `soup adapters`: git for LoRA.** Three years of `lora_v3_final_final2/` ends here. No diff, no merge, no rollback, no attribution from a weight change back to the dataset slice that caused it — until now. v0.57 ships git-shaped UX on top of the v0.22 adapter surface. -- **`soup diagnose `** scores six failure modes — `forgetting` (Δ accuracy vs base), `refusal` (advbench / xstest delta), `format` (JSON / regex / tool-call validity), `mode_collapse` (pairwise diversity at T=0 and T=1), `memorization` (training-prefix echo on partial-prompt probes), `contamination` (training-row overlap with public benchmarks) — and emits an overall OK / MINOR / MAJOR verdict using the same taxonomy as v0.26 Quant-Lobotomy. -- **`soup diagnose --badge diag.svg`** renders a self-contained 6-cell SVG report card with the overall verdict pill — embeddable in HF model cards and Twitter posts. Every user-controlled string passes through `html.escape`; no JS, no external deps. -- **`soup diagnose --evidence ev.json --output diag.json --attach-to-registry `** persists the report as a `diagnose_report` artifact on the v0.26 Model Registry — the first-class kind alongside `eval_suite` / `canaries` / `tensorrt`. -- **`soup train --diagnose-gate `** refuses to mark a run successful when any mode comes back MAJOR. Composes with v0.26 `--gate` and v0.55 `eval gate-install` for a three-layer safety net: training process anomalies (v0.34 `soup why`) → eval regression vs baseline (v0.55 gate) → model-behaviour failure modes (v0.56 diagnose). -- **Why blue-ocean.** Trainer libs ship code, not interpretability. TRL's v1 RFC narrows public surface — failure analysis falls below the cut. Eval SaaS (Braintrust, LangSmith) treat fine-tuning as <5 % of workload. The failure-mode taxonomy lives in academic papers with no implementation owner. Plus: builders care about this enough to ship a Twitter-shareable badge. -- **+123 new tests** (8676 → 8849) across `tests/test_v0560.py` covering all 6 probes + FailureReport frozen invariants + badge SVG escapes + 4-agent review-fix wave (python / security / code / tdd): atomic+symlink-safe badge write, 16 MiB evidence size cap, `typer.Exit` instead of `sys.exit`, `os.path.realpath` containment, contamination combined-complexity cap, ReDoS probe in `matches_regex`, extras null-byte sanitisation. +- **`soup adapters diff `** — per-layer ΔW Frobenius norm + relative drift + effective-rank delta via SVD entropy, with top-K changed projections highlighted. Output as a Rich table, machine-readable JSON, or PR-ready Markdown via `--format {table,json,markdown} --output report.json`. +- **`soup adapters merge [c...] -o --strategy {linear,ties,dare,svd}`** — four merge strategies in pure numpy: weighted linear, TIES (trim/elect-sign/disjoint avg per Yadav et al.), DARE (drop-and-rescale per Yu et al., deterministic via `--seed`), and SVD low-rank reconstruction (`--rank` clamped to min-dim). Output safetensors + `adapter_config.json` both written atomically. +- **`soup adapters blame --dataset --layer q_proj.7 --budget 4h`** — leave-one-out ablation plan: splits the dataset into N shards, estimates per-shard ablation runtime against your wall-clock budget, and emits a per-shard work table with feasibility check. Live ablation runner (training at 1/10 scale per shard with the v0.34 SQLite tracker + v0.26 Registry lineage) is wired in **v0.57.1**. +- **`soup adapters branch -c soup.yaml --base meta/llama-3.1`** + **`soup adapters checkout -o restored.yaml`** + **`soup adapters branches`** — SHA-256 snapshot pointers under `~/.soup/branches/` (or `SOUP_BRANCHES_DIR`-override, $HOME/$CWD/$TMPDIR-bounded). `checkout` refuses to restore when the source config has drifted from the snapshot SHA (no silent reproducibility loss). +- **Why blue-ocean.** HF Hub treats every revision as an opaque blob and won't ship weight-aware diffs (it would balkanise their storage backend). DVC / lakeFS are file-system primitives, not LoRA-aware. PEFT exposes `add_weighted_adapter`, mergekit exists — but no VCS-shaped UX wraps them. LLaMA-Factory closed #2038 (weighted merge) as not-planned. Git-semantics-for-tensors is a seam neither the registry nor the kernel teams will build. +- **+149 new tests** (8849 → 8998) across `tests/test_v0570_part_{a,b,c,d}.py`. 4-agent review-fix wave landed (1 CRITICAL: zero-assertion tests, 9 HIGH including TIES tied-sign positive default + 4× symlink rejections, 11 MEDIUM, 4 LOW): atomic writes + lstat+S_ISLNK rejection on every output, frozen dataclasses with FrozenInstanceError assertions, CRLF/null-byte env-var rejection, bool-as-int rejection on every numeric input, source-grep regression guards for the lazy-import policy. ## Why Soup? diff --git a/pyproject.toml b/pyproject.toml index 41ac5a5..f1cb838 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.56.0" +version = "0.57.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index c02a61b..9698064 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.56.0" +__version__ = "0.57.0" diff --git a/soup_cli/commands/adapters.py b/soup_cli/commands/adapters.py index ddd6562..b17e80a 100644 --- a/soup_cli/commands/adapters.py +++ b/soup_cli/commands/adapters.py @@ -1,10 +1,11 @@ -"""soup adapters — LoRA adapter management (list, info, compare).""" +"""soup adapters — LoRA adapter management (list, info, compare, diff, merge, blame, branch).""" import json from pathlib import Path import typer from rich.console import Console +from rich.markup import escape from rich.panel import Panel from rich.table import Table @@ -209,3 +210,320 @@ def compare( table.add_row("Size on disk", size1, size2) console.print(table) + + +@app.command() +def diff( + adapter_a: str = typer.Argument(..., help="Path to first adapter"), + adapter_b: str = typer.Argument(..., help="Path to second adapter"), + top_k: int = typer.Option(10, "--top-k", min=1, max=200, + help="Number of top changed projections to report"), + output_format: str = typer.Option("table", "--format", + help="Output format: table | json | markdown"), + output: str = typer.Option(None, "--output", "-o", + help="Write report to file (json/markdown only)"), +): + """Per-layer ΔW Frobenius diff + effective-rank drift (v0.57.0).""" + from soup_cli.utils.adapter_diff import ( + compute_adapter_diff, + render_report_json, + render_report_markdown, + ) + from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + + fmt = output_format.lower() + if fmt not in ("table", "json", "markdown"): + console.print(f"[red]Unknown --format: {escape(fmt)}[/]") + raise typer.Exit(2) + + try: + report = compute_adapter_diff(adapter_a, adapter_b, top_k=top_k) + except FileNotFoundError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + except (ValueError, TypeError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + except RuntimeError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + + if fmt == "json": + text = render_report_json(report) + elif fmt == "markdown": + text = render_report_markdown(report) + else: + text = None + + if output is not None: + if fmt == "table": + console.print("[red]--output requires --format json or markdown[/]") + raise typer.Exit(2) + enforce_under_cwd_and_no_symlink(output, "output") + # Atomic write via tempfile + os.replace — a crash mid-write must + # not leave a partial report at the target path (review fix MEDIUM). + import os as _os + import tempfile as _tf + target = Path(output) + target.parent.mkdir(parents=True, exist_ok=True) + fd, tmp = _tf.mkstemp(dir=str(target.parent), prefix=".tmp_") + try: + with _os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(text) + _os.replace(tmp, str(target)) + except Exception: + try: + _os.unlink(tmp) + except OSError: + pass + raise + console.print(f"[green]Wrote {fmt} report to {escape(output)}[/]") + return + + if fmt != "table": + console.print(text) + return + + # Default Rich table + table = Table(title=f"Adapter diff: {escape(report.adapter_a)} vs {escape(report.adapter_b)}") + table.add_column("Layer", style="bold") + table.add_column("ΔW Frobenius", justify="right") + table.add_column("Relative", justify="right") + for layer in sorted(report.per_layer, key=lambda d: d.frobenius, reverse=True)[:top_k]: + table.add_row( + escape(layer.name), + f"{layer.frobenius:.4f}", + f"{layer.relative:.2%}", + ) + console.print(table) + if report.effective_rank_a is not None and report.effective_rank_b is not None: + console.print( + f"Effective rank: A={report.effective_rank_a:.2f}, " + f"B={report.effective_rank_b:.2f}" + ) + console.print( + f"Shared layers: {report.shared_layers} | " + f"only-in-A: {len(report.only_in_a)} | only-in-B: {len(report.only_in_b)}" + ) + + +@app.command() +def merge( + adapters: list[str] = typer.Argument(..., help="Two or more adapter paths to merge"), + output: str = typer.Option(..., "--output", "-o", help="Output directory for merged adapter"), + strategy: str = typer.Option("linear", "--strategy", + help="linear | ties | dare | svd"), + weights: str = typer.Option(None, "--weights", + help="Comma-separated weights (default: equal)"), + density: float = typer.Option(0.2, "--density", + help="Trim density for ties/dare in (0, 1]"), + seed: int = typer.Option(0, "--seed", help="Random seed for dare"), + rank: int = typer.Option(None, "--rank", help="SVD rank (svd strategy only)"), +): + """Merge LoRA adapters via linear / ties / dare / svd (v0.57.0).""" + from soup_cli.utils.adapter_merge import SUPPORTED_STRATEGIES, merge_adapters + + if strategy not in SUPPORTED_STRATEGIES: + console.print( + f"[red]Unknown --strategy: {escape(strategy)}. " + f"Choose from: {', '.join(SUPPORTED_STRATEGIES)}[/]" + ) + raise typer.Exit(2) + + if len(adapters) < 2: + console.print("[red]Need at least 2 adapter paths to merge[/]") + raise typer.Exit(2) + + parsed_weights = None + if weights: + try: + parsed_weights = [float(w.strip()) for w in weights.split(",")] + except ValueError as exc: + console.print(f"[red]Invalid --weights: {escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + report = merge_adapters( + adapters, + output, + strategy=strategy, # type: ignore[arg-type] + weights=parsed_weights, + density=density, + seed=seed, + rank=rank, + ) + except FileNotFoundError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + except (ValueError, TypeError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + panel = Panel( + f"Strategy: [bold]{escape(report.strategy)}[/]\n" + f"Inputs: {len(report.adapters)}\n" + f"Merged layers: [bold]{report.merged_layers}[/]\n" + f"Skipped layers: {len(report.skipped_layers)}\n" + f"Verdict: [yellow]{report.verdict}[/] (live eval in v0.57.1)\n" + f"Output: [bold]{escape(report.output_dir)}[/]", + title="Adapter merge", + ) + console.print(panel) + + +@app.command() +def blame( + adapter_dir: str = typer.Argument(..., help="Path to trained adapter"), + dataset: str = typer.Option(..., "--dataset", help="Training JSONL the adapter was built on"), + layer: str = typer.Option(..., "--layer", help="Layer to attribute (e.g. q_proj.7)"), + budget: str = typer.Option("4h", "--budget", help="Wall-clock budget (e.g. 4h, 30m)"), + num_shards: int = typer.Option(10, "--shards", min=2, max=100, + help="Number of dataset shards for leave-one-out"), + plan_only: bool = typer.Option(False, "--plan-only", + help="Print plan and exit (live runner in v0.57.1)"), +): + """Attribute weight movement to dataset shards via leave-one-out ablation (v0.57.0).""" + from soup_cli.utils.blame import parse_budget, plan_blame, run_blame + + try: + seconds = parse_budget(budget) + except (TypeError, ValueError) as exc: + console.print(f"[red]Invalid --budget: {escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + plan = plan_blame( + adapter_dir, dataset, + layer=layer, budget_seconds=seconds, num_shards=num_shards, + ) + except FileNotFoundError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + except (ValueError, TypeError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + table = Table(title=f"Blame plan: {escape(plan.layer)}") + table.add_column("Shard", justify="right") + table.add_column("Hold-out offset", justify="right") + table.add_column("Hold-out rows", justify="right") + table.add_column("Projected seconds", justify="right") + for shard in plan.shards: + table.add_row( + str(shard.shard_id), + str(shard.holdout_offset), + str(shard.holdout_size), + str(shard.projected_seconds), + ) + console.print(table) + status_color = "green" if plan.feasible else "yellow" + console.print( + f"Budget {plan.budget_seconds}s, " + f"{plan.per_shard_seconds}s/shard — " + f"[{status_color}]{escape(plan.reason)}[/]" + ) + + if plan_only or not plan.feasible: + return + + try: + run_blame(plan) + except NotImplementedError as exc: + console.print(f"[yellow]{escape(str(exc))}[/]") + raise typer.Exit(0) from None + + +@app.command() +def branch( + name: str = typer.Argument(..., help="Branch name (alphanumeric + ._-)"), + config: str = typer.Option(..., "--config", "-c", help="Path to soup.yaml"), + base: str = typer.Option(..., "--base", help="Base model id"), + dataset: str = typer.Option(None, "--dataset", help="Training dataset path (optional)"), +): + """Snapshot a training environment as a comparable branch (v0.57.0).""" + from soup_cli.utils.adapter_branch import create_branch + + try: + snap = create_branch(name, config_path=config, base_model=base, + dataset_path=dataset) + except FileNotFoundError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + Panel( + f"Name: [bold]{escape(snap.name)}[/]\n" + f"Base: [bold]{escape(snap.base_model)}[/]\n" + f"Config: [bold]{escape(snap.config_path)}[/]\n" + f"SHA: [dim]{snap.config_sha256[:16]}...[/]\n" + f"Dataset SHA: [dim]" + f"{snap.dataset_sha256[:16] + '...' if snap.dataset_sha256 else '—'}[/]\n" + f"Version: {snap.soup_version}", + title="Adapter branch", + ) + ) + + +@app.command() +def checkout( + name: str = typer.Argument(..., help="Branch name to check out"), + output: str = typer.Option("soup.yaml", "--output", "-o", + help="Where to write the restored config"), +): + """Restore a snapshotted branch's config into cwd (v0.57.0).""" + from soup_cli.utils.adapter_branch import load_branch, write_checkout + + try: + snap = load_branch(name) + except FileNotFoundError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + target = write_checkout(snap, output) + except FileNotFoundError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + except (ValueError, TypeError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + f"[green]Restored branch {escape(snap.name)} → {escape(str(target))}[/]" + ) + + +@app.command(name="branches") +def list_branches_cmd(): + """List all snapshotted branches (v0.57.0).""" + from soup_cli.utils.adapter_branch import list_branches, load_branch + + names = list_branches() + if not names: + console.print("[dim]No branches yet.[/]") + return + + table = Table(title=f"Adapter branches ({len(names)})") + table.add_column("Name", style="bold") + table.add_column("Base model") + table.add_column("Config SHA") + table.add_column("Data SHA") + for branch_name in names: + try: + snap = load_branch(branch_name) + ds = snap.dataset_sha256[:8] + "..." if snap.dataset_sha256 else "—" + table.add_row( + escape(snap.name), + escape(snap.base_model), + snap.config_sha256[:8] + "...", + ds, + ) + except (ValueError, FileNotFoundError, OSError): + table.add_row(escape(branch_name), "[red]error[/]", "-", "-") + console.print(table) diff --git a/soup_cli/utils/adapter_branch.py b/soup_cli/utils/adapter_branch.py new file mode 100644 index 0000000..a0648b2 --- /dev/null +++ b/soup_cli/utils/adapter_branch.py @@ -0,0 +1,278 @@ +"""Lightweight snapshot pointers for adapter training (v0.57.0 Part D). + +A `branch` is a frozen pointer to {config, dataset SHA, base model, created_at} +so two trainings can be compared cleanly. Pointers live under +``~/.soup/branches/.json`` (or ``SOUP_BRANCHES_DIR``-overridden) with +``0o600`` perms. + +Public surface: + +- ``create_branch(name, *, config_path, base_model, dataset_path=None)`` -> ``Branch`` +- ``list_branches()`` -> tuple[str, ...] +- ``load_branch(name)`` -> ``Branch`` +- ``delete_branch(name)`` -> bool +- ``write_checkout(branch, target_path)`` -> snapshot config back to cwd + +Validation policy mirrors the project standard: alphanumeric + ``._-``, +≤128 chars, null-byte rejected, branches dir is containment-checked under +``$HOME`` / ``$CWD`` / ``$TMPDIR`` per v0.36.0 ``SOUP_BATCH_CACHE_PATH`` policy. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import stat +import tempfile +import time +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Optional, Tuple + +from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink, is_under + +_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._\-]{0,127}$") +_MAX_BRANCHES = 1024 +_HASH_CHUNK = 65536 +_MAX_CONFIG_BYTES = 1_048_576 # 1 MiB cap on snapshot config payload + + +@dataclass(frozen=True) +class Branch: + name: str + config_path: str + config_sha256: str + dataset_sha256: Optional[str] + base_model: str + created_at: float + soup_version: str + + +def _validate_name(name: object) -> str: + if isinstance(name, bool) or not isinstance(name, str): + raise TypeError("name must be str") + if not name: + raise ValueError("name must be non-empty") + if "\x00" in name: + raise ValueError("name must not contain null bytes") + if not _NAME_RE.match(name): + raise ValueError( + "name must match [A-Za-z0-9][A-Za-z0-9._-]{0,127}" + ) + return name + + +def _branches_dir() -> Path: + """Resolve the branches directory with env-var override + containment.""" + override = os.environ.get("SOUP_BRANCHES_DIR") + if override: + # Reject null + every C0 control char (CRLF, tabs, etc.) — mirrors + # v0.51.0 validate_hub_endpoint policy. Silently fall through to + # the default rather than raise; an env var is operator-supplied + # configuration, not API input. + if any(ord(ch) < 0x20 for ch in override): + override = None + if override: + candidate = Path(os.path.realpath(override)) + bounds = [ + Path.home(), + Path.cwd(), + Path(tempfile.gettempdir()), + ] + if any(is_under(str(candidate), str(b)) for b in bounds): + candidate.mkdir(parents=True, exist_ok=True) + return candidate + default = Path.home() / ".soup" / "branches" + default.mkdir(parents=True, exist_ok=True) + return default + + +def _hash_file(path: Path) -> str: + hasher = hashlib.sha256() + with open(path, "rb") as fh: + while True: + chunk = fh.read(_HASH_CHUNK) + if not chunk: + break + hasher.update(chunk) + return hasher.hexdigest() + + +def _validate_str_field(value: object, field: str, max_len: int = 512) -> str: + if isinstance(value, bool) or not isinstance(value, str): + raise TypeError(f"{field} must be str") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > max_len: + raise ValueError(f"{field} must be ≤{max_len} chars") + return value + + +def create_branch( + name: str, + *, + config_path: str, + base_model: str, + dataset_path: Optional[str] = None, +) -> Branch: + """Snapshot a training environment as an immutable branch pointer. + + The config file is hashed (SHA-256) and the dataset file (if provided) + is hashed too so callers can detect "same config, different data" drift. + """ + from soup_cli import __version__ + + name = _validate_name(name) + enforce_under_cwd_and_no_symlink(config_path, "config_path") + _validate_str_field(base_model, "base_model", max_len=512) + + config_full = Path(config_path) + if not config_full.is_file(): + raise FileNotFoundError(f"config not found: {config_full.name}") + if config_full.stat().st_size > _MAX_CONFIG_BYTES: + raise ValueError( + f"config exceeds {_MAX_CONFIG_BYTES} byte cap" + ) + + config_sha = _hash_file(config_full) + + dataset_sha: Optional[str] = None + if dataset_path is not None: + enforce_under_cwd_and_no_symlink(dataset_path, "dataset_path") + ds_full = Path(dataset_path) + if not ds_full.is_file(): + raise FileNotFoundError(f"dataset not found: {ds_full.name}") + dataset_sha = _hash_file(ds_full) + + branches_dir = _branches_dir() + existing = sorted(p for p in branches_dir.glob("*.json")) + if len(existing) >= _MAX_BRANCHES: + raise RuntimeError(f"branches dir has ≥{_MAX_BRANCHES} entries") + + branch = Branch( + name=name, + config_path=str(config_full), + config_sha256=config_sha, + dataset_sha256=dataset_sha, + base_model=base_model, + created_at=time.time(), + soup_version=__version__, + ) + _atomic_write_branch(branches_dir, branch) + return branch + + +def _atomic_write_branch(branches_dir: Path, branch: Branch) -> None: + target = branches_dir / f"{branch.name}.json" + payload = json.dumps(asdict(branch), indent=2, sort_keys=True) + fd, tmp_path = tempfile.mkstemp(dir=str(branches_dir), prefix=".tmp_") + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(payload) + os.replace(tmp_path, str(target)) + except Exception: + try: + os.unlink(tmp_path) + except OSError: + pass + raise + if os.name != "nt": + try: + os.chmod(str(target), 0o600) + except OSError: + pass + + +def list_branches() -> Tuple[str, ...]: + branches_dir = _branches_dir() + return tuple(sorted(p.stem for p in branches_dir.glob("*.json"))) + + +def load_branch(name: str) -> Branch: + name = _validate_name(name) + branches_dir = _branches_dir() + target = branches_dir / f"{name}.json" + if not target.is_file(): + raise FileNotFoundError(f"branch not found: {name}") + # TOCTOU defence: reject symlinks at the branch pointer path before read + # so a planted symlink to /etc/passwd cannot leak file content via the + # JSON-parse error path (review fix HIGH). + st = os.lstat(str(target)) + if stat.S_ISLNK(st.st_mode): + raise ValueError(f"branch pointer must not be a symlink: {name}") + text = target.read_text(encoding="utf-8") + if len(text) > _MAX_CONFIG_BYTES: + raise ValueError("branch pointer exceeds size cap") + raw = json.loads(text) + return Branch( + name=_validate_name(raw["name"]), + config_path=_validate_str_field(raw["config_path"], "config_path", max_len=4096), + config_sha256=_validate_str_field(raw["config_sha256"], "config_sha256", max_len=128), + dataset_sha256=( + _validate_str_field(raw["dataset_sha256"], "dataset_sha256", max_len=128) + if raw.get("dataset_sha256") is not None + else None + ), + base_model=_validate_str_field(raw["base_model"], "base_model"), + created_at=float(raw["created_at"]), + soup_version=_validate_str_field(raw["soup_version"], "soup_version", max_len=64), + ) + + +def delete_branch(name: str) -> bool: + name = _validate_name(name) + branches_dir = _branches_dir() + target = branches_dir / f"{name}.json" + if not target.is_file(): + return False + # TOCTOU defence: reject symlinks before unlink so a planted symlink + # at /.json -> /important/file cannot be silently deleted. + st = os.lstat(str(target)) + if stat.S_ISLNK(st.st_mode): + raise ValueError(f"branch pointer must not be a symlink: {name}") + target.unlink() + return True + + +def write_checkout(branch: Branch, target_path: str) -> Path: + """Copy the snapshotted config into cwd so the user can re-run.""" + if not isinstance(branch, Branch): + raise TypeError("branch must be Branch") + enforce_under_cwd_and_no_symlink(target_path, "target_path") + + source = Path(branch.config_path) + if not source.is_file(): + raise FileNotFoundError( + f"branch config no longer exists: {source.name}" + ) + + actual_sha = _hash_file(source) + if actual_sha != branch.config_sha256: + raise ValueError( + f"branch config drifted from snapshot SHA " + f"({actual_sha[:8]} vs {branch.config_sha256[:8]})" + ) + + target = Path(target_path) + target.parent.mkdir(parents=True, exist_ok=True) + fd, tmp_path = tempfile.mkstemp(dir=str(target.parent), prefix=".tmp_") + try: + with os.fdopen(fd, "wb") as fh: + with open(source, "rb") as src: + while True: + chunk = src.read(_HASH_CHUNK) + if not chunk: + break + fh.write(chunk) + os.replace(tmp_path, str(target)) + except Exception: + try: + os.unlink(tmp_path) + except OSError: + pass + raise + return target diff --git a/soup_cli/utils/adapter_diff.py b/soup_cli/utils/adapter_diff.py new file mode 100644 index 0000000..2dc7354 --- /dev/null +++ b/soup_cli/utils/adapter_diff.py @@ -0,0 +1,337 @@ +"""Per-layer LoRA adapter diff math + report rendering (v0.57.0 Part A). + +Pure numpy math (no torch); safetensors is loaded lazily so import is cheap. +Public surface: + +- ``compute_layer_diffs(weights_a, weights_b)`` -> per-layer Frobenius diffs +- ``effective_rank(matrix)`` -> SVD-entropy effective rank +- ``compute_adapter_diff(path_a, path_b, *, top_k=10)`` -> ``AdapterDiffReport`` +- ``render_report_markdown(report)`` / ``render_report_json(report)`` + +Containment + symlink rejection at every file load (TOCTOU defence, +mirrors v0.53.1 ``enforce_under_cwd_and_no_symlink`` policy). +""" + +from __future__ import annotations + +import json +import math +import os +import stat +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Any, Mapping, Optional, Tuple + +from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink, is_under_cwd + +_MAX_LAYER_NAME_LEN = 256 +_MAX_LAYERS = 10_000 +_MAX_TOP_K = 200 +_MIN_TOP_K = 1 + + +@dataclass(frozen=True) +class LayerDiff: + """Frobenius-norm diff for a single LoRA parameter tensor.""" + + name: str + frobenius: float + norm_a: float + norm_b: float + relative: float # frobenius / max(norm_a, norm_b) or 0.0 if both zero + + +@dataclass(frozen=True) +class AdapterDiffReport: + adapter_a: str + adapter_b: str + per_layer: Tuple[LayerDiff, ...] + top_changed: Tuple[str, ...] + effective_rank_a: Optional[float] + effective_rank_b: Optional[float] + shared_layers: int + only_in_a: Tuple[str, ...] + only_in_b: Tuple[str, ...] + + +def _require_str(value: object, field: str) -> str: + if isinstance(value, bool) or not isinstance(value, str): + raise TypeError(f"{field} must be str, got {type(value).__name__}") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > _MAX_LAYER_NAME_LEN: + raise ValueError(f"{field} must be ≤{_MAX_LAYER_NAME_LEN} chars") + return value + + +def _frobenius(matrix: Any) -> float: + import numpy as np + + arr = np.asarray(matrix, dtype=np.float64) + if arr.size == 0: + return 0.0 + value = float(np.sqrt(np.sum(arr * arr))) + if not math.isfinite(value): + return float("inf") + return value + + +def effective_rank(matrix: Any, *, eps: float = 1e-12) -> float: + """Shannon entropy of normalised singular-value distribution (effective rank). + + Returns ``exp(H)`` where H is the entropy of the SV distribution + treated as a probability vector. Equals the matrix rank for an + orthonormal basis and degrades smoothly as energy concentrates. + """ + import numpy as np + + if isinstance(eps, bool) or not isinstance(eps, (int, float)): + raise TypeError("eps must be float") + if not math.isfinite(float(eps)) or float(eps) <= 0: + raise ValueError("eps must be finite and positive") + + arr = np.asarray(matrix, dtype=np.float64) + if arr.ndim < 2: + if arr.ndim == 1: + arr = arr.reshape(-1, 1) + else: + return 0.0 + if arr.size == 0: + return 0.0 + # Reshape >2D into 2D for SVD + if arr.ndim > 2: + arr = arr.reshape(arr.shape[0], -1) + try: + singular = np.linalg.svd(arr, compute_uv=False) + except np.linalg.LinAlgError: + return 0.0 + total = float(np.sum(singular)) + if total <= eps: + return 0.0 + probs = singular / total + probs = probs[probs > eps] + if probs.size == 0: + return 0.0 + entropy = float(-np.sum(probs * np.log(probs))) + return float(math.exp(entropy)) + + +def compute_layer_diffs( + weights_a: Mapping[str, Any], + weights_b: Mapping[str, Any], +) -> Tuple[Tuple[LayerDiff, ...], Tuple[str, ...], Tuple[str, ...]]: + """Compute per-layer Frobenius diffs for every name in both adapters. + + Returns ``(per_layer, only_in_a, only_in_b)``. ``per_layer`` covers the + intersection of names sorted alphabetically. + """ + import numpy as np + + if not isinstance(weights_a, Mapping): + raise TypeError("weights_a must be a mapping") + if not isinstance(weights_b, Mapping): + raise TypeError("weights_b must be a mapping") + + names_a = set(weights_a.keys()) + names_b = set(weights_b.keys()) + if len(names_a) > _MAX_LAYERS or len(names_b) > _MAX_LAYERS: + raise ValueError(f"adapter has >{_MAX_LAYERS} tensors") + + shared = sorted(names_a & names_b) + only_a = tuple(sorted(names_a - names_b)) + only_b = tuple(sorted(names_b - names_a)) + + diffs = [] + for name in shared: + _require_str(name, "layer name") + a = np.asarray(weights_a[name], dtype=np.float64) + b = np.asarray(weights_b[name], dtype=np.float64) + if a.shape != b.shape: + # Skip shape-mismatched tensors (rank changed between adapters) + continue + diff = a - b + fro = _frobenius(diff) + norm_a = _frobenius(a) + norm_b = _frobenius(b) + denom = max(norm_a, norm_b) + relative = fro / denom if denom > 0 else 0.0 + diffs.append( + LayerDiff( + name=name, + frobenius=fro, + norm_a=norm_a, + norm_b=norm_b, + relative=relative, + ) + ) + return tuple(diffs), only_a, only_b + + +def _validate_top_k(top_k: object) -> int: + if isinstance(top_k, bool) or not isinstance(top_k, int): + raise TypeError("top_k must be int") + if top_k < _MIN_TOP_K or top_k > _MAX_TOP_K: + raise ValueError(f"top_k must be in [{_MIN_TOP_K}, {_MAX_TOP_K}]") + return top_k + + +def _load_safetensors(path: Path) -> Mapping[str, Any]: + """Lazy-load adapter_model.safetensors via the ``safetensors`` package.""" + try: + from safetensors import safe_open + except ImportError as exc: + raise RuntimeError( + "safetensors package required; pip install safetensors" + ) from exc + result: dict[str, Any] = {} + with safe_open(str(path), framework="numpy") as f: + for key in f.keys(): + _require_str(key, "tensor name") + result[key] = f.get_tensor(key) + if len(result) > _MAX_LAYERS: + raise ValueError(f"adapter has >{_MAX_LAYERS} tensors") + return result + + +def _adapter_weights_path(adapter_dir: Path) -> Path: + """Return the safetensors path inside an adapter dir, raising if missing. + + Symlinks at the weights file are rejected via ``os.lstat + S_ISLNK`` + BEFORE ``is_file()`` so a crafted ``adapter_model.safetensors -> /etc/passwd`` + cannot escape the directory-level containment check (review fix HIGH). + """ + candidates = ( + adapter_dir / "adapter_model.safetensors", + adapter_dir / "adapter_model.bin", + ) + for cand in candidates: + if not os.path.lexists(str(cand)): + continue + st = os.lstat(str(cand)) + if stat.S_ISLNK(st.st_mode): + raise ValueError( + f"{adapter_dir.name}/{cand.name}: must not be a symlink" + ) + if cand.is_file(): + if cand.suffix == ".bin": + raise RuntimeError( + f"{adapter_dir.name}: .bin format not supported; " + "re-save adapter as safetensors" + ) + return cand + raise FileNotFoundError( + f"{adapter_dir.name}: no adapter_model.safetensors found" + ) + + +def load_adapter_weights(adapter_dir: str) -> Mapping[str, Any]: + """Containment-checked safetensors load. + + Raises ``ValueError`` if the dir is outside cwd or a symlink; raises + ``FileNotFoundError`` if no adapter_model.safetensors is present. + """ + enforce_under_cwd_and_no_symlink(adapter_dir, "adapter") + path = _adapter_weights_path(Path(adapter_dir)) + # Re-validate the weights file itself + if not is_under_cwd(str(path)): + raise ValueError(f"adapter weights must stay under cwd: {path.name}") + return _load_safetensors(path) + + +def _effective_rank_average(weights: Mapping[str, Any]) -> Optional[float]: + """Mean effective-rank across 2D LoRA matrices (None if no 2D tensors).""" + import numpy as np + + ranks: list[float] = [] + for tensor in weights.values(): + arr = np.asarray(tensor) + if arr.ndim == 2 and min(arr.shape) > 0: + ranks.append(effective_rank(arr)) + if not ranks: + return None + return float(sum(ranks) / len(ranks)) + + +def compute_adapter_diff( + adapter_a: str, + adapter_b: str, + *, + top_k: int = 10, +) -> AdapterDiffReport: + """End-to-end: load both adapters, compute layer diffs, rank top-K.""" + _require_str(adapter_a, "adapter_a") + _require_str(adapter_b, "adapter_b") + _validate_top_k(top_k) + + weights_a = load_adapter_weights(adapter_a) + weights_b = load_adapter_weights(adapter_b) + + per_layer, only_a, only_b = compute_layer_diffs(weights_a, weights_b) + + sorted_by_change = sorted(per_layer, key=lambda d: d.frobenius, reverse=True) + top = tuple(d.name for d in sorted_by_change[:top_k]) + + rank_a = _effective_rank_average(weights_a) + rank_b = _effective_rank_average(weights_b) + + return AdapterDiffReport( + adapter_a=os.path.basename(os.path.normpath(adapter_a)), + adapter_b=os.path.basename(os.path.normpath(adapter_b)), + per_layer=per_layer, + top_changed=top, + effective_rank_a=rank_a, + effective_rank_b=rank_b, + shared_layers=len(per_layer), + only_in_a=only_a, + only_in_b=only_b, + ) + + +def render_report_json(report: AdapterDiffReport) -> str: + """Serialise a report as canonical JSON for CI consumption.""" + if not isinstance(report, AdapterDiffReport): + raise TypeError("report must be AdapterDiffReport") + payload = { + "adapter_a": report.adapter_a, + "adapter_b": report.adapter_b, + "shared_layers": report.shared_layers, + "effective_rank_a": report.effective_rank_a, + "effective_rank_b": report.effective_rank_b, + "top_changed": list(report.top_changed), + "only_in_a": list(report.only_in_a), + "only_in_b": list(report.only_in_b), + "per_layer": [asdict(d) for d in report.per_layer], + } + return json.dumps(payload, indent=2, sort_keys=True, allow_nan=False) + + +def render_report_markdown(report: AdapterDiffReport) -> str: + """Human-readable markdown report (suitable for PR comments).""" + if not isinstance(report, AdapterDiffReport): + raise TypeError("report must be AdapterDiffReport") + lines = [ + f"# Adapter diff: {report.adapter_a} vs {report.adapter_b}", + "", + f"- Shared layers: **{report.shared_layers}**", + f"- Effective rank A: **{report.effective_rank_a}**", + f"- Effective rank B: **{report.effective_rank_b}**", + "", + "## Top changed projections", + "", + ] + if not report.top_changed: + lines.append("_no shared layers_") + else: + for name in report.top_changed: + lines.append(f"- `{name}`") + if report.only_in_a: + lines.extend(["", "## Only in A", ""]) + for name in report.only_in_a: + lines.append(f"- `{name}`") + if report.only_in_b: + lines.extend(["", "## Only in B", ""]) + for name in report.only_in_b: + lines.append(f"- `{name}`") + return "\n".join(lines) + "\n" diff --git a/soup_cli/utils/adapter_merge.py b/soup_cli/utils/adapter_merge.py new file mode 100644 index 0000000..a9894da --- /dev/null +++ b/soup_cli/utils/adapter_merge.py @@ -0,0 +1,412 @@ +"""LoRA adapter merge strategies (v0.57.0 Part B). + +Four strategies wrap mergekit-style math in pure numpy: + +- ``linear``: weighted average per-layer (baseline) +- ``ties``: trim-elect-disjoint average (Yadav et al. 2023) +- ``dare``: drop and rescale by 1/(1-density) (Yu et al. 2024) +- ``svd``: low-rank SVD reconstruction of the merged delta + +All strategies operate on the intersection of layer names; shape-mismatched +tensors are silently skipped (rank changed between adapters). Live canary +verdict via v0.55 eval gate is deferred to v0.57.1; stub returns ``None``. +""" + +from __future__ import annotations + +import json +import math +import os +import stat +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import Any, FrozenSet, Literal, Mapping, Sequence, Tuple + +from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + +MergeStrategy = Literal["linear", "ties", "dare", "svd"] +# frozenset for O(1) membership + immutability (mirrors v0.41.0 / v0.51.0 policy); +# tuple alias preserved for caller code that iterates in canonical order. +SUPPORTED_STRATEGIES: FrozenSet[str] = frozenset({"linear", "ties", "dare", "svd"}) +STRATEGY_ORDER: Tuple[MergeStrategy, ...] = ("linear", "ties", "dare", "svd") + +_MAX_ADAPTERS = 16 +_MIN_ADAPTERS = 2 + + +@dataclass(frozen=True) +class MergeReport: + strategy: str + adapters: Tuple[str, ...] + weights: Tuple[float, ...] + merged_layers: int + skipped_layers: Tuple[str, ...] + output_dir: str + verdict: str # "OK" / "MINOR" / "MAJOR" / "UNKNOWN" (v0.57.1 live) + + +def _validate_weights(weights: Sequence[float], n: int) -> Tuple[float, ...]: + if not isinstance(weights, Sequence) or isinstance(weights, str): + raise TypeError("weights must be a sequence of floats") + if len(weights) != n: + raise ValueError(f"weights length {len(weights)} != number of adapters {n}") + out: list[float] = [] + for w in weights: + if isinstance(w, bool): + raise TypeError("weights must not contain bool") + if not isinstance(w, (int, float)): + raise TypeError("weights must be numeric") + wf = float(w) + if not math.isfinite(wf): + raise ValueError("weights must be finite") + if wf < 0: + raise ValueError("weights must be non-negative") + out.append(wf) + total = sum(out) + if total <= 0: + raise ValueError("weights must sum to a positive value") + return tuple(out) + + +def _validate_density(density: float, field: str) -> float: + if isinstance(density, bool) or not isinstance(density, (int, float)): + raise TypeError(f"{field} must be float") + val = float(density) + if not math.isfinite(val): + raise ValueError(f"{field} must be finite") + if not 0 < val <= 1: + raise ValueError(f"{field} must be in (0, 1]") + return val + + +def merge_linear( + weights_list: Sequence[Mapping[str, Any]], + weights: Sequence[float], +) -> Tuple[dict[str, Any], Tuple[str, ...]]: + """Weighted average per layer over the intersection of names.""" + import numpy as np + + if len(weights_list) < _MIN_ADAPTERS: + raise ValueError(f"need at least {_MIN_ADAPTERS} adapters") + if len(weights_list) > _MAX_ADAPTERS: + raise ValueError(f"at most {_MAX_ADAPTERS} adapters") + w = _validate_weights(weights, len(weights_list)) + norm = sum(w) + coeffs = [wi / norm for wi in w] + + shared = set(weights_list[0].keys()) + for adapters in weights_list[1:]: + shared &= set(adapters.keys()) + + merged: dict[str, Any] = {} + skipped: list[str] = [] + for name in sorted(shared): + tensors = [np.asarray(a[name], dtype=np.float64) for a in weights_list] + shapes = {t.shape for t in tensors} + if len(shapes) > 1: + skipped.append(name) + continue + acc = np.zeros_like(tensors[0]) + for c, t in zip(coeffs, tensors): + acc += c * t + merged[name] = acc.astype(np.float32) + return merged, tuple(skipped) + + +def merge_ties( + weights_list: Sequence[Mapping[str, Any]], + weights: Sequence[float], + *, + density: float = 0.2, +) -> Tuple[dict[str, Any], Tuple[str, ...]]: + """TIES merge: trim low magnitudes, elect majority sign, disjoint avg.""" + import numpy as np + + density_val = _validate_density(density, "density") + if len(weights_list) < _MIN_ADAPTERS: + raise ValueError(f"need at least {_MIN_ADAPTERS} adapters") + w = _validate_weights(weights, len(weights_list)) + + shared = set(weights_list[0].keys()) + for adapters in weights_list[1:]: + shared &= set(adapters.keys()) + + merged: dict[str, Any] = {} + skipped: list[str] = [] + for name in sorted(shared): + tensors = [np.asarray(a[name], dtype=np.float64) for a in weights_list] + shapes = {t.shape for t in tensors} + if len(shapes) > 1: + skipped.append(name) + continue + # Per-adapter trim: keep top density% by magnitude, zero out rest + trimmed = [] + for t in tensors: + flat = np.abs(t).flatten() + if flat.size == 0: + trimmed.append(t.copy()) + continue + k = max(1, int(flat.size * density_val)) + threshold = np.partition(flat, -k)[-k] if k < flat.size else 0.0 + mask = np.abs(t) >= threshold + trimmed.append(np.where(mask, t, 0.0)) + # Elect majority sign per element. On a tie (sign_sum == 0) default + # to positive (TIES paper §3) so all-tied entries are not silently + # zeroed — `np.sign(0) == 0` would set `elected_sign` to 0 and the + # subsequent agree_mask would drop every parameter (review fix HIGH). + stacked = np.stack(trimmed) # (n_adapters, ...) + sign_sum = np.sign(stacked).sum(axis=0) + elected_sign = np.where(sign_sum >= 0, 1.0, -1.0) + # Disjoint average: only average elements that agree with elected sign + agree_mask = np.sign(stacked) == elected_sign[None, ...] + weighted = np.zeros_like(trimmed[0]) + weight_sum = np.zeros_like(trimmed[0]) + for wi, t, mask in zip(w, trimmed, agree_mask): + weighted += wi * t * mask + weight_sum += wi * mask + out = np.where(weight_sum > 0, weighted / np.maximum(weight_sum, 1e-12), 0.0) + merged[name] = out.astype(np.float32) + return merged, tuple(skipped) + + +def merge_dare( + weights_list: Sequence[Mapping[str, Any]], + weights: Sequence[float], + *, + density: float = 0.5, + seed: int = 0, +) -> Tuple[dict[str, Any], Tuple[str, ...]]: + """DARE merge: random drop with prob (1-density), rescale 1/density, average.""" + import numpy as np + + density_val = _validate_density(density, "density") + if isinstance(seed, bool) or not isinstance(seed, int): + raise TypeError("seed must be int") + if seed < 0: + raise ValueError("seed must be non-negative") + if len(weights_list) < _MIN_ADAPTERS: + raise ValueError(f"need at least {_MIN_ADAPTERS} adapters") + w = _validate_weights(weights, len(weights_list)) + norm = sum(w) + coeffs = [wi / norm for wi in w] + + shared = set(weights_list[0].keys()) + for adapters in weights_list[1:]: + shared &= set(adapters.keys()) + + rng = np.random.default_rng(seed) + merged: dict[str, Any] = {} + skipped: list[str] = [] + for name in sorted(shared): + tensors = [np.asarray(a[name], dtype=np.float64) for a in weights_list] + shapes = {t.shape for t in tensors} + if len(shapes) > 1: + skipped.append(name) + continue + acc = np.zeros_like(tensors[0]) + for c, t in zip(coeffs, tensors): + mask = rng.random(t.shape) < density_val + rescaled = np.where(mask, t / density_val, 0.0) + acc += c * rescaled + merged[name] = acc.astype(np.float32) + return merged, tuple(skipped) + + +def merge_svd( + weights_list: Sequence[Mapping[str, Any]], + weights: Sequence[float], + *, + rank: int | None = None, +) -> Tuple[dict[str, Any], Tuple[str, ...]]: + """SVD merge: linear-average, then low-rank reconstruct each 2D tensor. + + Non-2D tensors are merged linearly without truncation. + """ + import numpy as np + + if rank is not None: + if isinstance(rank, bool) or not isinstance(rank, int): + raise TypeError("rank must be int or None") + if rank < 1: + raise ValueError("rank must be ≥ 1") + + linear_merged, skipped = merge_linear(weights_list, weights) + if rank is None: + return linear_merged, skipped + + out: dict[str, Any] = {} + for name, tensor in linear_merged.items(): + arr = np.asarray(tensor, dtype=np.float64) + if arr.ndim != 2: + out[name] = arr.astype(np.float32) + continue + try: + u_mat, sv, vt_mat = np.linalg.svd(arr, full_matrices=False) + except np.linalg.LinAlgError: + out[name] = arr.astype(np.float32) + continue + r = min(rank, sv.shape[0]) + reconstructed = u_mat[:, :r] @ np.diag(sv[:r]) @ vt_mat[:r, :] + out[name] = reconstructed.astype(np.float32) + return out, skipped + + +def merge_adapters( + adapter_paths: Sequence[str], + output_dir: str, + *, + strategy: MergeStrategy, + weights: Sequence[float] | None = None, + density: float = 0.2, + seed: int = 0, + rank: int | None = None, +) -> MergeReport: + """End-to-end: load adapters, merge, write output as safetensors.""" + from soup_cli.utils.adapter_diff import load_adapter_weights + + if strategy not in SUPPORTED_STRATEGIES: + raise ValueError( + f"strategy must be one of {SUPPORTED_STRATEGIES}, got {strategy!r}" + ) + if not isinstance(adapter_paths, Sequence) or isinstance(adapter_paths, str): + raise TypeError("adapter_paths must be a sequence of strings") + if len(adapter_paths) < _MIN_ADAPTERS: + raise ValueError(f"need at least {_MIN_ADAPTERS} adapters") + if len(adapter_paths) > _MAX_ADAPTERS: + raise ValueError(f"at most {_MAX_ADAPTERS} adapters") + + enforce_under_cwd_and_no_symlink(output_dir, "output_dir") + + weights_list = [load_adapter_weights(p) for p in adapter_paths] + + if weights is None: + eq = 1.0 / len(adapter_paths) + weights = [eq] * len(adapter_paths) + coeffs = _validate_weights(weights, len(adapter_paths)) + + if strategy == "linear": + merged, skipped = merge_linear(weights_list, coeffs) + elif strategy == "ties": + merged, skipped = merge_ties(weights_list, coeffs, density=density) + elif strategy == "dare": + merged, skipped = merge_dare(weights_list, coeffs, density=density, seed=seed) + else: # svd + merged, skipped = merge_svd(weights_list, coeffs, rank=rank) + + _write_merged_adapter(output_dir, adapter_paths[0], merged) + + return MergeReport( + strategy=strategy, + adapters=tuple(adapter_paths), + weights=coeffs, + merged_layers=len(merged), + skipped_layers=skipped, + output_dir=output_dir, + verdict="UNKNOWN", # v0.57.1: live canary eval via v0.55 gate + ) + + +_MAX_ADAPTER_CONFIG_BYTES = 256 * 1024 + + +def _reject_if_symlink(path: Path, field: str) -> None: + """TOCTOU defence: lstat the raw path before opening.""" + if os.path.lexists(str(path)): + st = os.lstat(str(path)) + if stat.S_ISLNK(st.st_mode): + raise ValueError(f"{field} must not be a symlink: {path.name}") + + +def _atomic_write_bytes(target: Path, data: bytes) -> None: + """Atomic write via mkstemp + os.replace in target's parent dir.""" + target.parent.mkdir(parents=True, exist_ok=True) + _reject_if_symlink(target, "output file") + fd, tmp = tempfile.mkstemp(dir=str(target.parent), prefix=".tmp_") + try: + with os.fdopen(fd, "wb") as fh: + fh.write(data) + os.replace(tmp, str(target)) + except Exception: + try: + os.unlink(tmp) + except OSError: + pass + raise + + +def _write_merged_adapter( + output_dir: str, + template_source: str, + weights: Mapping[str, Any], +) -> None: + """Write adapter_model.safetensors + copy adapter_config.json from source. + + Both writes are atomic (tempfile + os.replace) and reject pre-placed + symlinks at the target path (TOCTOU defence; mirrors v0.53.1 policy). + The source config is lstat-checked before reading and size-capped at + 256 KB per the v0.53.0 ``load_quant_config`` precedent. + """ + try: + from safetensors.numpy import save_file + except ImportError as exc: + raise RuntimeError( + "safetensors package required; pip install safetensors" + ) from exc + + out_path = Path(output_dir) + out_path.mkdir(parents=True, exist_ok=True) + + safetensors_target = out_path / "adapter_model.safetensors" + _reject_if_symlink(safetensors_target, "output safetensors") + # safetensors.save_file writes atomically internally via mmap-rename on + # POSIX, but on Windows it does not — write to sibling tmp + replace + # ourselves so behaviour is consistent across platforms. + fd, tmp_safe = tempfile.mkstemp( + dir=str(out_path), prefix=".tmp_", suffix=".safetensors" + ) + os.close(fd) + try: + save_file(weights, tmp_safe) + os.replace(tmp_safe, str(safetensors_target)) + except Exception: + try: + os.unlink(tmp_safe) + except OSError: + pass + raise + + # Copy adapter_config.json from first source so the merged adapter is loadable + source_cfg = Path(template_source) / "adapter_config.json" + if os.path.lexists(str(source_cfg)): + st = os.lstat(str(source_cfg)) + if stat.S_ISLNK(st.st_mode): + raise ValueError( + "source adapter_config.json must not be a symlink" + ) + if st.st_size > _MAX_ADAPTER_CONFIG_BYTES: + raise ValueError( + f"source adapter_config.json > {_MAX_ADAPTER_CONFIG_BYTES} byte cap" + ) + cfg = json.loads(source_cfg.read_text(encoding="utf-8")) + target_cfg = out_path / "adapter_config.json" + _atomic_write_bytes( + target_cfg, + json.dumps(cfg, indent=2).encode("utf-8"), + ) + + +def predict_merged_verdict( + report: MergeReport, + canary_suite: str | None = None, +) -> str: + """Stub for v0.57.1: live canary-eval verdict via v0.55 gate. + + Today returns the existing report.verdict. The signature is forward-compatible + so v0.57.1 can lift this to a real OK/MINOR/MAJOR classification. + """ + if not isinstance(report, MergeReport): + raise TypeError("report must be MergeReport") + if canary_suite is not None and not isinstance(canary_suite, str): + raise TypeError("canary_suite must be str or None") + return report.verdict diff --git a/soup_cli/utils/blame.py b/soup_cli/utils/blame.py new file mode 100644 index 0000000..05be558 --- /dev/null +++ b/soup_cli/utils/blame.py @@ -0,0 +1,200 @@ +"""LoRA adapter blame: attribute weight movement to dataset shards (v0.57.0 Part C). + +This module emits a leave-one-out ablation PLAN that future v0.57.1 wiring +will execute through the existing v0.34 SQLite tracker + v0.26 Registry +lineage. The plan is fully deterministic given the inputs so callers can +inspect / re-run / share without running real ablations. + +Public surface: + +- ``parse_budget(spec)`` -> seconds (mirrors v0.48.0 ``parse_budget`` semantics) +- ``plan_blame(adapter_dir, dataset_path, *, layer, budget_seconds, num_shards)`` + -> ``BlamePlan`` (frozen dataclass) with per-shard work item + projected duration +- ``run_blame`` stub (raises ``NotImplementedError`` with v0.57.1 marker) +""" + +from __future__ import annotations + +import math +import os +import re +from dataclasses import dataclass +from typing import Tuple + +from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + +_MIN_BUDGET_SECONDS = 60 +_MAX_BUDGET_SECONDS = 24 * 3600 +_MIN_SHARDS = 2 +_MAX_SHARDS = 100 +_MIN_PER_SHARD_SECONDS = 30 + +_BUDGET_RE = re.compile(r"^(\d+)([smh]?)$") + + +@dataclass(frozen=True) +class BlameShardWork: + shard_id: int + holdout_offset: int + holdout_size: int + projected_seconds: int + + +@dataclass(frozen=True) +class BlamePlan: + adapter_dir: str + dataset_path: str + layer: str + budget_seconds: int + num_shards: int + per_shard_seconds: int + shards: Tuple[BlameShardWork, ...] + feasible: bool + reason: str + + +def parse_budget(spec: str) -> int: + """Parse a budget string (``60s`` / ``5m`` / ``2h`` / bare seconds).""" + if isinstance(spec, bool) or not isinstance(spec, str): + raise TypeError("spec must be str") + if not spec: + raise ValueError("spec must be non-empty") + if "\x00" in spec: + raise ValueError("spec must not contain null bytes") + match = _BUDGET_RE.match(spec.strip()) + if not match: + raise ValueError(f"invalid budget: {spec!r}") + value = int(match.group(1)) + unit = match.group(2) or "s" + multiplier = {"s": 1, "m": 60, "h": 3600}[unit] + seconds = value * multiplier + if seconds < _MIN_BUDGET_SECONDS: + raise ValueError( + f"budget {seconds}s below floor {_MIN_BUDGET_SECONDS}s" + ) + if seconds > _MAX_BUDGET_SECONDS: + raise ValueError( + f"budget {seconds}s above cap {_MAX_BUDGET_SECONDS}s" + ) + return seconds + + +def _validate_int(value: object, field: str, lo: int, hi: int) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError(f"{field} must be int") + if value < lo or value > hi: + raise ValueError(f"{field} must be in [{lo}, {hi}]") + return value + + +def _validate_str(value: object, field: str, max_len: int = 512) -> str: + if isinstance(value, bool) or not isinstance(value, str): + raise TypeError(f"{field} must be str") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > max_len: + raise ValueError(f"{field} must be ≤{max_len} chars") + return value + + +def _count_dataset_rows(dataset_path: str) -> int: + """Cheap line count; for blame planning a rough order-of-magnitude is enough. + + Opens via the realpath captured at containment check (see plan_blame) + so a symlink swap between check and open cannot redirect the read + (TOCTOU defence). + """ + from pathlib import Path + + real = os.path.realpath(dataset_path) + path = Path(real) + if not path.is_file(): + raise FileNotFoundError(f"dataset not found: {path.name}") + # Quick line count without holding the whole file in memory + count = 0 + with open(real, "rb") as fh: + for _ in fh: + count += 1 + if count > 10_000_000: # DoS cap — 10M lines is plenty + break + return count + + +def plan_blame( + adapter_dir: str, + dataset_path: str, + *, + layer: str, + budget_seconds: int, + num_shards: int = 10, +) -> BlamePlan: + """Build a leave-one-out ablation plan. + + The plan splits the dataset into ``num_shards`` equal shards; per shard, + a 1/10-scale ablation run trains with that shard held out. ``feasible`` + is False when the budget cannot cover ≥ ``_MIN_PER_SHARD_SECONDS`` per + shard with safety overhead. + """ + enforce_under_cwd_and_no_symlink(adapter_dir, "adapter_dir") + enforce_under_cwd_and_no_symlink(dataset_path, "dataset_path") + _validate_str(layer, "layer", max_len=256) + _validate_int(budget_seconds, "budget_seconds", + _MIN_BUDGET_SECONDS, _MAX_BUDGET_SECONDS) + _validate_int(num_shards, "num_shards", _MIN_SHARDS, _MAX_SHARDS) + + row_count = _count_dataset_rows(dataset_path) + if row_count == 0: + raise ValueError("dataset is empty") + + # Reserve 10% overhead for setup/teardown across all shards + usable_seconds = int(budget_seconds * 0.9) + per_shard = usable_seconds // num_shards + feasible = per_shard >= _MIN_PER_SHARD_SECONDS + reason = ( + "ok" + if feasible + else f"budget {budget_seconds}s gives only {per_shard}s/shard " + f"(need ≥{_MIN_PER_SHARD_SECONDS}s)" + ) + + shard_size = math.ceil(row_count / num_shards) + shards: list[BlameShardWork] = [] + for sid in range(num_shards): + offset = sid * shard_size + size = min(shard_size, max(0, row_count - offset)) + shards.append( + BlameShardWork( + shard_id=sid, + holdout_offset=offset, + holdout_size=size, + projected_seconds=per_shard, + ) + ) + + return BlamePlan( + adapter_dir=adapter_dir, + dataset_path=dataset_path, + layer=layer, + budget_seconds=budget_seconds, + num_shards=num_shards, + per_shard_seconds=per_shard, + shards=tuple(shards), + feasible=feasible, + reason=reason, + ) + + +def run_blame(plan: BlamePlan) -> None: + """Live leave-one-out ablation runner — deferred to v0.57.1. + + The plan is executed through the v0.34 SQLite tracker + v0.26 Registry + lineage so every ablation run is reproducible. Live wiring lands in v0.57.1. + """ + if not isinstance(plan, BlamePlan): + raise TypeError("plan must be BlamePlan") + raise NotImplementedError( + "Live blame ablation runner deferred to v0.57.1. " + "Use `soup adapters blame --plan-only` to inspect the plan." + ) diff --git a/tests/test_v0560.py b/tests/test_v0560.py index fe1460e..d112bed 100644 --- a/tests/test_v0560.py +++ b/tests/test_v0560.py @@ -774,12 +774,15 @@ class TestSourceWiring: f"{relative} carries a top-level {forbidden!r} import" ) - def test_version_is_0_56_0(self) -> None: - assert __version__ == "0.56.0" + def test_version_at_or_above_0_56_0(self) -> None: + # v0.57.0 widened this from exact-match to floor-check (matches + # the v0.51.0 / v0.54.0 floor-test idiom — every subsequent release + # would otherwise have to edit this single line). + assert __version__ >= "0.56.0" def test_pyproject_version(self) -> None: text = (_PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8") - assert 'version = "0.56.0"' in text + assert 'version = "0.5' in text # floor check; current v0.5x.y # --- review-fix coverage -------------------------------------------------- diff --git a/tests/test_v0570_part_a.py b/tests/test_v0570_part_a.py new file mode 100644 index 0000000..29ba746 --- /dev/null +++ b/tests/test_v0570_part_a.py @@ -0,0 +1,396 @@ +"""v0.57.0 Part A — adapters diff: math kernel + CLI smoke tests.""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import numpy as np +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app as soup_app +from soup_cli.utils.adapter_diff import ( + AdapterDiffReport, + LayerDiff, + compute_adapter_diff, + compute_layer_diffs, + effective_rank, + render_report_json, + render_report_markdown, +) + +runner = CliRunner() + + +# ---------- effective_rank ---------- + + +def test_effective_rank_identity_matrix(): + eye = np.eye(8) + rank = effective_rank(eye) + assert abs(rank - 8.0) < 1e-6 + + +def test_effective_rank_concentrated(): + # All energy in one direction → effective rank ≈ 1 + matrix = np.zeros((4, 4)) + matrix[0, 0] = 1.0 + assert effective_rank(matrix) == pytest.approx(1.0, abs=1e-6) + + +def test_effective_rank_empty(): + assert effective_rank(np.zeros((0, 0))) == 0.0 + + +def test_effective_rank_1d_reshapes(): + assert effective_rank(np.ones(5)) >= 0.0 + + +def test_effective_rank_zero_matrix(): + assert effective_rank(np.zeros((3, 3))) == 0.0 + + +def test_effective_rank_bool_eps_rejected(): + with pytest.raises(TypeError): + effective_rank(np.eye(2), eps=True) + + +def test_effective_rank_non_finite_eps_rejected(): + with pytest.raises(ValueError): + effective_rank(np.eye(2), eps=float("nan")) + + +def test_effective_rank_negative_eps_rejected(): + with pytest.raises(ValueError): + effective_rank(np.eye(2), eps=-1.0) + + +# ---------- compute_layer_diffs ---------- + + +def test_compute_layer_diffs_identical_zero(): + a = {"foo": np.ones((2, 2))} + b = {"foo": np.ones((2, 2))} + diffs, only_a, only_b = compute_layer_diffs(a, b) + assert len(diffs) == 1 + assert diffs[0].frobenius == 0.0 + assert diffs[0].relative == 0.0 + assert only_a == () + assert only_b == () + + +def test_compute_layer_diffs_known_norm(): + a = {"w": np.zeros((2, 2))} + b = {"w": np.array([[3.0, 4.0], [0.0, 0.0]])} + diffs, _, _ = compute_layer_diffs(a, b) + # Frobenius = sqrt(9+16) = 5 + assert diffs[0].frobenius == pytest.approx(5.0) + assert diffs[0].relative == pytest.approx(1.0) + + +def test_compute_layer_diffs_skips_shape_mismatch(): + a = {"w": np.zeros((4, 4))} + b = {"w": np.zeros((2, 2))} + diffs, _, _ = compute_layer_diffs(a, b) + assert diffs == () # shape mismatch silently skipped + + +def test_compute_layer_diffs_partitions(): + a = {"only_a": np.zeros(2), "shared": np.zeros(2)} + b = {"only_b": np.zeros(2), "shared": np.zeros(2)} + diffs, only_a, only_b = compute_layer_diffs(a, b) + assert [d.name for d in diffs] == ["shared"] + assert only_a == ("only_a",) + assert only_b == ("only_b",) + + +def test_compute_layer_diffs_rejects_non_mapping(): + with pytest.raises(TypeError): + compute_layer_diffs([], {}) # type: ignore[arg-type] + with pytest.raises(TypeError): + compute_layer_diffs({}, []) # type: ignore[arg-type] + + +def test_compute_layer_diffs_too_many_tensors(): + big = {f"t{i}": np.zeros(1) for i in range(10_001)} + small: dict[str, np.ndarray] = {} + with pytest.raises(ValueError, match=">10000"): + compute_layer_diffs(big, small) + + +# ---------- render report ---------- + + +def _sample_report() -> AdapterDiffReport: + return AdapterDiffReport( + adapter_a="A", + adapter_b="B", + per_layer=( + LayerDiff(name="layer1", frobenius=2.0, norm_a=4.0, norm_b=3.0, relative=0.5), + ), + top_changed=("layer1",), + effective_rank_a=8.0, + effective_rank_b=8.0, + shared_layers=1, + only_in_a=(), + only_in_b=(), + ) + + +def test_render_report_json_roundtrip(): + report = _sample_report() + text = render_report_json(report) + parsed = json.loads(text) + assert parsed["adapter_a"] == "A" + assert parsed["adapter_b"] == "B" + assert parsed["shared_layers"] == 1 + assert parsed["top_changed"] == ["layer1"] + assert parsed["per_layer"][0]["frobenius"] == 2.0 + + +def test_render_report_json_rejects_non_report(): + with pytest.raises(TypeError): + render_report_json({"foo": "bar"}) # type: ignore[arg-type] + + +def test_render_report_markdown_renders(): + text = render_report_markdown(_sample_report()) + assert "# Adapter diff: A vs B" in text + assert "layer1" in text + assert text.endswith("\n") + + +def test_render_report_markdown_only_lists(): + report = AdapterDiffReport( + adapter_a="x", adapter_b="y", + per_layer=(), top_changed=(), + effective_rank_a=None, effective_rank_b=None, + shared_layers=0, + only_in_a=("foo",), + only_in_b=("bar",), + ) + text = render_report_markdown(report) + assert "Only in A" in text + assert "Only in B" in text + + +def test_render_report_markdown_rejects_non_report(): + with pytest.raises(TypeError): + render_report_markdown(None) # type: ignore[arg-type] + + +def test_frozen_dataclasses(): + import dataclasses + diff = LayerDiff(name="x", frobenius=1.0, norm_a=1.0, norm_b=1.0, relative=1.0) + with pytest.raises(dataclasses.FrozenInstanceError): + diff.frobenius = 2.0 # type: ignore[misc] + + +# ---------- compute_adapter_diff (end-to-end with safetensors fixture) ---------- + + +def _write_safetensors(dir_path: Path, weights: dict) -> None: + """Helper: write adapter_model.safetensors + minimal adapter_config.json.""" + pytest.importorskip("safetensors") + from safetensors.numpy import save_file + + dir_path.mkdir(parents=True, exist_ok=True) + save_file(weights, str(dir_path / "adapter_model.safetensors")) + (dir_path / "adapter_config.json").write_text( + json.dumps({"peft_type": "LORA", "r": 8}), encoding="utf-8" + ) + + +def test_compute_adapter_diff_end_to_end(tmp_path, monkeypatch): + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + weights_a = { + "base_model.model.layers.0.self_attn.q_proj.lora_A.weight": + np.ones((4, 8), dtype=np.float32), + "base_model.model.layers.0.self_attn.v_proj.lora_A.weight": + np.zeros((4, 8), dtype=np.float32), + } + weights_b = { + "base_model.model.layers.0.self_attn.q_proj.lora_A.weight": + np.ones((4, 8), dtype=np.float32), # identical + "base_model.model.layers.0.self_attn.v_proj.lora_A.weight": + np.ones((4, 8), dtype=np.float32), # different + } + _write_safetensors(tmp_path / "a", weights_a) + _write_safetensors(tmp_path / "b", weights_b) + + report = compute_adapter_diff("a", "b", top_k=5) + assert isinstance(report, AdapterDiffReport) + assert report.shared_layers == 2 + # v_proj should be the top-changed projection + assert report.top_changed[0].endswith("v_proj.lora_A.weight") + + +def test_compute_adapter_diff_outside_cwd_rejected(tmp_path): + with pytest.raises(ValueError): + compute_adapter_diff(str(tmp_path), "b") + + +def test_compute_adapter_diff_bool_top_k_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + (tmp_path / "b").mkdir() + with pytest.raises(TypeError): + compute_adapter_diff("a", "b", top_k=True) # type: ignore[arg-type] + + +def test_compute_adapter_diff_top_k_out_of_range(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + (tmp_path / "b").mkdir() + with pytest.raises(ValueError): + compute_adapter_diff("a", "b", top_k=0) + with pytest.raises(ValueError): + compute_adapter_diff("a", "b", top_k=201) + + +def test_compute_adapter_diff_top_k_lower_bound(tmp_path, monkeypatch): + """top_k=1 should be accepted (exact lower bound).""" + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + _write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + report = compute_adapter_diff("a", "b", top_k=1) + assert len(report.top_changed) == 1 + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink semantics") +def test_compute_adapter_diff_rejects_symlinked_weights(tmp_path, monkeypatch): + """Symlink at adapter_model.safetensors must be rejected (TOCTOU).""" + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + _write_safetensors(tmp_path / "real", {"w": np.zeros((2, 2), dtype=np.float32)}) + (tmp_path / "fake").mkdir() + os.symlink( + str(tmp_path / "real" / "adapter_model.safetensors"), + str(tmp_path / "fake" / "adapter_model.safetensors"), + ) + (tmp_path / "fake" / "adapter_config.json").write_text( + json.dumps({"peft_type": "LORA"}), encoding="utf-8" + ) + _write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + with pytest.raises(ValueError, match="symlink"): + compute_adapter_diff("fake", "b") + + +def test_no_top_level_torch_import(): + """Lazy-import policy: adapter_diff must not import torch at module level.""" + src = (Path(__file__).parent.parent / "soup_cli" / "utils" / "adapter_diff.py" + ).read_text(encoding="utf-8") + # Only allowed inside def bodies + for line in src.splitlines(): + stripped = line.lstrip() + if stripped.startswith("import torch") or stripped.startswith("from torch"): + indent = len(line) - len(stripped) + assert indent > 0, f"top-level torch import found: {line}" + + +def test_compute_adapter_diff_missing_safetensors(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + (tmp_path / "b").mkdir() + with pytest.raises(FileNotFoundError): + compute_adapter_diff("a", "b") + + +def test_compute_adapter_diff_bin_format_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + (tmp_path / "a" / "adapter_model.bin").write_bytes(b"x") + (tmp_path / "b").mkdir() + (tmp_path / "b" / "adapter_model.bin").write_bytes(b"x") + with pytest.raises(RuntimeError, match=".bin format not supported"): + compute_adapter_diff("a", "b") + + +# ---------- CLI ---------- + + +def test_adapters_diff_help(): + result = runner.invoke(soup_app, ["adapters", "diff", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--top-k" in result.output + assert "--format" in result.output + + +def test_adapters_diff_table_format(tmp_path, monkeypatch): + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + _write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, ["adapters", "diff", "a", "b"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "Adapter diff" in result.output + + +def test_adapters_diff_json_output(tmp_path, monkeypatch): + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + _write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, [ + "adapters", "diff", "a", "b", + "--format", "json", + "--output", "report.json", + ]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + parsed = json.loads((tmp_path / "report.json").read_text()) + assert parsed["shared_layers"] == 1 + + +def test_adapters_diff_markdown_output(tmp_path, monkeypatch): + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + _write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, [ + "adapters", "diff", "a", "b", + "--format", "markdown", + "--output", "report.md", + ]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + text = (tmp_path / "report.md").read_text() + assert "# Adapter diff" in text + + +def test_adapters_diff_unknown_format(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + (tmp_path / "b").mkdir() + result = runner.invoke(soup_app, [ + "adapters", "diff", "a", "b", "--format", "yaml", + ]) + assert result.exit_code == 2 + assert "Unknown --format" in result.output + + +def test_adapters_diff_output_requires_non_table(tmp_path, monkeypatch): + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + _write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, [ + "adapters", "diff", "a", "b", "--output", "out.txt", + ]) + assert result.exit_code == 2 + assert "requires --format" in result.output + + +def test_adapters_diff_output_outside_cwd_rejected(tmp_path, monkeypatch): + pytest.importorskip("safetensors") + monkeypatch.chdir(tmp_path) + _write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + outside = os.path.join(os.path.dirname(str(tmp_path)), "outside.json") + result = runner.invoke(soup_app, [ + "adapters", "diff", "a", "b", + "--format", "json", "--output", outside, + ]) + assert result.exit_code != 0 diff --git a/tests/test_v0570_part_b.py b/tests/test_v0570_part_b.py new file mode 100644 index 0000000..1a82698 --- /dev/null +++ b/tests/test_v0570_part_b.py @@ -0,0 +1,418 @@ +"""v0.57.0 Part B — adapters merge: linear / ties / dare / svd.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import numpy as np +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app as soup_app +from soup_cli.utils.adapter_merge import ( + SUPPORTED_STRATEGIES, + MergeReport, + merge_adapters, + merge_dare, + merge_linear, + merge_svd, + merge_ties, + predict_merged_verdict, +) + +runner = CliRunner() + + +# ---------- merge_linear ---------- + + +def test_merge_linear_average_of_zero_and_one(): + a = {"w": np.zeros((2, 2), dtype=np.float32)} + b = {"w": np.ones((2, 2), dtype=np.float32)} + merged, skipped = merge_linear([a, b], [1.0, 1.0]) + assert np.allclose(merged["w"], 0.5) + assert skipped == () + + +def test_merge_linear_weighted(): + a = {"w": np.zeros((2,), dtype=np.float32)} + b = {"w": np.ones((2,), dtype=np.float32)} + merged, _ = merge_linear([a, b], [3.0, 1.0]) + assert np.allclose(merged["w"], 0.25) + + +def test_merge_linear_intersection_only(): + a = {"shared": np.ones((2,), dtype=np.float32), "only_a": np.ones((2,))} + b = {"shared": np.zeros((2,), dtype=np.float32), "only_b": np.ones((2,))} + merged, _ = merge_linear([a, b], [1.0, 1.0]) + assert set(merged.keys()) == {"shared"} + + +def test_merge_linear_shape_mismatch_skipped(): + a = {"w": np.zeros((4,), dtype=np.float32)} + b = {"w": np.zeros((2,), dtype=np.float32)} + merged, skipped = merge_linear([a, b], [1.0, 1.0]) + assert merged == {} + assert skipped == ("w",) + + +def test_merge_linear_rejects_single_adapter(): + with pytest.raises(ValueError, match="at least 2"): + merge_linear([{"w": np.zeros(1)}], [1.0]) + + +def test_merge_linear_rejects_too_many(): + with pytest.raises(ValueError, match="at most 16"): + merge_linear([{}] * 17, [1.0] * 17) + + +def test_merge_linear_bool_weight_rejected(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(TypeError): + merge_linear([a, b], [True, 1.0]) # type: ignore[list-item] + + +def test_merge_linear_negative_weight_rejected(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(ValueError): + merge_linear([a, b], [-1.0, 1.0]) + + +def test_merge_linear_nan_weight_rejected(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(ValueError): + merge_linear([a, b], [float("nan"), 1.0]) + + +def test_merge_linear_zero_sum_rejected(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(ValueError, match="positive"): + merge_linear([a, b], [0.0, 0.0]) + + +def test_merge_linear_wrong_weights_length(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(ValueError, match="length"): + merge_linear([a, b], [1.0]) + + +# ---------- merge_ties ---------- + + +def test_merge_ties_density_keeps_top(): + # Top-half of [1, 2, 3, 4]: keep 3, 4 + a = {"w": np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float32)} + b = {"w": np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float32)} + merged, _ = merge_ties([a, b], [1.0, 1.0], density=0.5) + # First two slots should be trimmed to zero + assert merged["w"][0] == 0.0 + assert merged["w"][3] != 0.0 + + +def test_merge_ties_majority_sign_election(): + # Two adapters agree positive, one disagrees → elected sign is positive + a = {"w": np.array([1.0], dtype=np.float32)} + b = {"w": np.array([2.0], dtype=np.float32)} + c = {"w": np.array([-3.0], dtype=np.float32)} + merged, _ = merge_ties([a, b, c], [1.0, 1.0, 1.0], density=1.0) + # Elected sign positive; negative entry dropped + assert merged["w"][0] > 0 + + +def test_merge_ties_invalid_density(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(ValueError): + merge_ties([a, b], [1.0, 1.0], density=0.0) + with pytest.raises(ValueError): + merge_ties([a, b], [1.0, 1.0], density=1.5) + + +def test_merge_ties_bool_density_rejected(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(TypeError): + merge_ties([a, b], [1.0, 1.0], density=True) # type: ignore[arg-type] + + +# ---------- merge_dare ---------- + + +def test_merge_dare_deterministic_with_seed(): + a = {"w": np.ones((10,), dtype=np.float32)} + b = {"w": np.ones((10,), dtype=np.float32)} + m1, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=42) + m2, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=42) + assert np.allclose(m1["w"], m2["w"]) + + +def test_merge_dare_different_seeds_diverge(): + a = {"w": np.ones((100,), dtype=np.float32)} + b = {"w": np.ones((100,), dtype=np.float32)} + m1, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=1) + m2, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=2) + assert not np.allclose(m1["w"], m2["w"]) + + +def test_merge_dare_bool_seed_rejected(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(TypeError): + merge_dare([a, b], [1.0, 1.0], seed=True) # type: ignore[arg-type] + + +def test_merge_dare_negative_seed_rejected(): + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(ValueError): + merge_dare([a, b], [1.0, 1.0], seed=-1) + + +def test_merge_dare_density_1_equals_linear(): + a = {"w": np.array([2.0, 4.0], dtype=np.float32)} + b = {"w": np.array([6.0, 8.0], dtype=np.float32)} + merged, _ = merge_dare([a, b], [1.0, 1.0], density=1.0, seed=0) + # density=1 → no drop, no rescale → identical to linear average + assert np.allclose(merged["w"], [4.0, 6.0]) + + +# ---------- merge_svd ---------- + + +def test_merge_svd_no_rank_equals_linear(): + a = {"w": np.eye(4, dtype=np.float32)} + b = {"w": np.eye(4, dtype=np.float32)} + merged, _ = merge_svd([a, b], [1.0, 1.0]) + assert np.allclose(merged["w"], np.eye(4)) + + +def test_merge_svd_with_rank_reduces_rank(): + # Random matrix → low-rank reconstruction should be lower rank + rng = np.random.default_rng(0) + a = {"w": rng.standard_normal((8, 8)).astype(np.float32)} + b = {"w": rng.standard_normal((8, 8)).astype(np.float32)} + merged, _ = merge_svd([a, b], [1.0, 1.0], rank=2) + actual_rank = np.linalg.matrix_rank(merged["w"], tol=1e-5) + assert actual_rank <= 2 + + +def test_merge_svd_non_2d_passthrough(): + a = {"bias": np.ones((4,), dtype=np.float32)} + b = {"bias": np.ones((4,), dtype=np.float32)} + merged, _ = merge_svd([a, b], [1.0, 1.0], rank=1) + assert np.allclose(merged["bias"], 1.0) + + +def test_merge_svd_rank_clamp(): + # Rank > min dimension → clamped + a = {"w": np.eye(4, dtype=np.float32)} + b = {"w": np.eye(4, dtype=np.float32)} + merged, _ = merge_svd([a, b], [1.0, 1.0], rank=100) + assert merged["w"].shape == (4, 4) + + +def test_merge_svd_invalid_rank(): + a = {"w": np.eye(2, dtype=np.float32)} + b = {"w": np.eye(2, dtype=np.float32)} + with pytest.raises(ValueError): + merge_svd([a, b], [1.0, 1.0], rank=0) + with pytest.raises(TypeError): + merge_svd([a, b], [1.0, 1.0], rank=True) # type: ignore[arg-type] + + +# ---------- merge_adapters end-to-end + CLI ---------- + + +def _write_adapter(dir_path: Path, weights: dict) -> None: + pytest.importorskip("safetensors") + from safetensors.numpy import save_file + + dir_path.mkdir(parents=True, exist_ok=True) + save_file(weights, str(dir_path / "adapter_model.safetensors")) + (dir_path / "adapter_config.json").write_text( + json.dumps({"peft_type": "LORA", "r": 8}), encoding="utf-8" + ) + + +def test_merge_adapters_e2e_linear(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + _write_adapter(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + report = merge_adapters(["a", "b"], "out", strategy="linear") + assert isinstance(report, MergeReport) + assert report.strategy == "linear" + assert report.merged_layers == 1 + assert (tmp_path / "out" / "adapter_model.safetensors").exists() + assert (tmp_path / "out" / "adapter_config.json").exists() + + +def test_merge_adapters_unknown_strategy(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "a").mkdir() + (tmp_path / "b").mkdir() + with pytest.raises(ValueError, match="strategy must be"): + merge_adapters(["a", "b"], "out", strategy="bogus") # type: ignore[arg-type] + + +def test_merge_adapters_output_outside_cwd(tmp_path, monkeypatch): + import os + monkeypatch.chdir(tmp_path) + _write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)}) + _write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + outside = os.path.join(os.path.dirname(str(tmp_path)), "outside") + with pytest.raises(ValueError): + merge_adapters(["a", "b"], outside, strategy="linear") + + +def test_supported_strategies_immutable(): + # v0.57.0 review fix: SUPPORTED_STRATEGIES is a frozenset (matches v0.41.0+ + # allowlist policy). STRATEGY_ORDER preserves canonical iteration order. + from soup_cli.utils.adapter_merge import STRATEGY_ORDER + assert SUPPORTED_STRATEGIES == frozenset({"linear", "ties", "dare", "svd"}) + assert isinstance(SUPPORTED_STRATEGIES, frozenset) + assert STRATEGY_ORDER == ("linear", "ties", "dare", "svd") + + +def test_merge_ties_density_one_keeps_everything(): + """density=1.0 is the inclusive upper bound — must not raise.""" + import numpy as np # noqa: F811 + a = {"w": np.ones((4,), dtype=np.float32)} + b = {"w": np.ones((4,), dtype=np.float32)} + merged, _ = merge_ties([a, b], [1.0, 1.0], density=1.0) + assert "w" in merged + + +def test_merge_linear_inf_weight_rejected(): + """math.isfinite must reject +inf as well as NaN.""" + import numpy as np # noqa: F811 + a = {"w": np.zeros(1, dtype=np.float32)} + b = {"w": np.zeros(1, dtype=np.float32)} + with pytest.raises(ValueError, match="finite"): + merge_linear([a, b], [float("inf"), 1.0]) + + +def test_merge_ties_tied_sign_defaults_positive(): + """Sign-sum == 0 (tied vote) must elect +1, not silently zero parameters.""" + import numpy as np # noqa: F811 + a = {"w": np.array([2.0], dtype=np.float32)} + b = {"w": np.array([-2.0], dtype=np.float32)} + merged, _ = merge_ties([a, b], [1.0, 1.0], density=1.0) + # Tied sign → elected +1 → positive entry kept, negative dropped → result 2.0 + assert merged["w"][0] > 0 + + +@pytest.mark.skipif(__import__("os").name == "nt", + reason="POSIX-only symlink semantics") +def test_merge_adapters_rejects_symlink_at_output_safetensors(tmp_path, monkeypatch): + """Pre-placed symlink at output safetensors path must be rejected (TOCTOU).""" + import os + monkeypatch.chdir(tmp_path) + _write_adapter(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + out = tmp_path / "out" + out.mkdir() + target = tmp_path / "evil.bin" + target.write_bytes(b"x") + os.symlink(str(target), str(out / "adapter_model.safetensors")) + with pytest.raises(ValueError, match="symlink"): + merge_adapters(["a", "b"], "out", strategy="linear") + # Symlink target untouched + assert target.read_bytes() == b"x" + + +def test_no_top_level_torch_import_in_merge(): + src = (Path(__file__).parent.parent / "soup_cli" / "utils" / "adapter_merge.py" + ).read_text(encoding="utf-8") + for line in src.splitlines(): + stripped = line.lstrip() + if stripped.startswith("import torch") or stripped.startswith("from torch"): + indent = len(line) - len(stripped) + assert indent > 0, f"top-level torch import: {line}" + + +def test_predict_merged_verdict_stub(): + report = MergeReport( + strategy="linear", adapters=("a", "b"), weights=(0.5, 0.5), + merged_layers=1, skipped_layers=(), output_dir="out", verdict="UNKNOWN", + ) + assert predict_merged_verdict(report) == "UNKNOWN" + + +def test_predict_merged_verdict_rejects_non_report(): + with pytest.raises(TypeError): + predict_merged_verdict("not a report") # type: ignore[arg-type] + + +def test_predict_merged_verdict_canary_must_be_str(): + report = MergeReport( + strategy="linear", adapters=("a", "b"), weights=(0.5, 0.5), + merged_layers=1, skipped_layers=(), output_dir="out", verdict="OK", + ) + with pytest.raises(TypeError): + predict_merged_verdict(report, canary_suite=123) # type: ignore[arg-type] + + +def test_merge_report_frozen(): + import dataclasses + report = MergeReport( + strategy="linear", adapters=("a", "b"), weights=(0.5, 0.5), + merged_layers=1, skipped_layers=(), output_dir="out", verdict="OK", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + report.strategy = "ties" # type: ignore[misc] + + +def test_adapters_merge_cli_help(): + result = runner.invoke(soup_app, ["adapters", "merge", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--strategy" in result.output + assert "--weights" in result.output + + +def test_adapters_merge_cli_linear(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + _write_adapter(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)}) + _write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, [ + "adapters", "merge", "a", "b", "-o", "out", "--strategy", "linear", + ]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "out" / "adapter_model.safetensors").exists() + + +def test_adapters_merge_cli_unknown_strategy(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + _write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)}) + _write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, [ + "adapters", "merge", "a", "b", "-o", "out", "--strategy", "bogus", + ]) + assert result.exit_code == 2 + assert "Unknown --strategy" in result.output + + +def test_adapters_merge_cli_invalid_weights(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + _write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)}) + _write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, [ + "adapters", "merge", "a", "b", "-o", "out", + "--strategy", "linear", "--weights", "1.0,abc", + ]) + assert result.exit_code == 2 + + +def test_adapters_merge_cli_single_adapter_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + _write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)}) + result = runner.invoke(soup_app, [ + "adapters", "merge", "a", "-o", "out", "--strategy", "linear", + ]) + assert result.exit_code == 2 + assert "at least 2" in result.output diff --git a/tests/test_v0570_part_c.py b/tests/test_v0570_part_c.py new file mode 100644 index 0000000..9708f3e --- /dev/null +++ b/tests/test_v0570_part_c.py @@ -0,0 +1,317 @@ +"""v0.57.0 Part C — adapters blame: plan emitter + budget check.""" + +from __future__ import annotations + +import os + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app as soup_app +from soup_cli.utils.blame import ( + BlamePlan, + BlameShardWork, + parse_budget, + plan_blame, + run_blame, +) + +runner = CliRunner() + + +# ---------- parse_budget ---------- + + +@pytest.mark.parametrize("spec,expected", [ + ("60", 60), + ("60s", 60), + ("5m", 300), + ("2h", 7200), + ("1h", 3600), + ("90s", 90), +]) +def test_parse_budget_happy(spec, expected): + assert parse_budget(spec) == expected + + +def test_parse_budget_below_min(): + with pytest.raises(ValueError, match="below floor"): + parse_budget("30") + + +def test_parse_budget_above_cap(): + with pytest.raises(ValueError, match="above cap"): + parse_budget("25h") + + +def test_parse_budget_invalid_format(): + with pytest.raises(ValueError, match="invalid"): + parse_budget("abc") + + +def test_parse_budget_empty(): + with pytest.raises(ValueError): + parse_budget("") + + +def test_parse_budget_bool(): + with pytest.raises(TypeError): + parse_budget(True) # type: ignore[arg-type] + + +def test_parse_budget_null_byte(): + with pytest.raises(ValueError, match="null"): + parse_budget("60\x00s") + + +def test_parse_budget_non_string(): + with pytest.raises(TypeError): + parse_budget(60) # type: ignore[arg-type] + + +# ---------- plan_blame ---------- + + +def _setup_blame(tmp_path, monkeypatch, n_rows: int = 100): + monkeypatch.chdir(tmp_path) + (tmp_path / "adapter").mkdir() + dataset = tmp_path / "data.jsonl" + dataset.write_text( + "\n".join(f'{{"text": "row{i}"}}' for i in range(n_rows)), + encoding="utf-8", + ) + return "adapter", "data.jsonl" + + +def test_plan_blame_happy(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch, n_rows=100) + plan = plan_blame( + adapter, dataset, + layer="q_proj.7", budget_seconds=3600, num_shards=10, + ) + assert isinstance(plan, BlamePlan) + assert len(plan.shards) == 10 + assert plan.layer == "q_proj.7" + assert plan.feasible is True + # Each shard covers ~10 rows + assert plan.shards[0].holdout_size == 10 + + +def test_plan_blame_infeasible_budget(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + plan = plan_blame( + adapter, dataset, + layer="q_proj.7", budget_seconds=60, num_shards=100, + ) + # 60s / 100 shards = 0s/shard → infeasible + assert plan.feasible is False + assert "need" in plan.reason + + +def test_plan_blame_shard_offsets(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch, n_rows=50) + plan = plan_blame( + adapter, dataset, + layer="layer", budget_seconds=3600, num_shards=5, + ) + offsets = [s.holdout_offset for s in plan.shards] + assert offsets == [0, 10, 20, 30, 40] + + +def test_plan_blame_uneven_split(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch, n_rows=23) + plan = plan_blame( + adapter, dataset, + layer="layer", budget_seconds=3600, num_shards=5, + ) + # ceil(23 / 5) = 5 → last shard gets 3 rows + sizes = [s.holdout_size for s in plan.shards] + assert sum(sizes) == 23 + + +def test_plan_blame_empty_dataset(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "adapter").mkdir() + (tmp_path / "empty.jsonl").write_text("", encoding="utf-8") + with pytest.raises(ValueError, match="empty"): + plan_blame( + "adapter", "empty.jsonl", + layer="x", budget_seconds=3600, num_shards=5, + ) + + +def test_plan_blame_outside_cwd_adapter(tmp_path): + with pytest.raises(ValueError): + plan_blame( + str(tmp_path), "data.jsonl", + layer="x", budget_seconds=3600, num_shards=5, + ) + + +def test_plan_blame_outside_cwd_dataset(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "adapter").mkdir() + outside = os.path.join(os.path.dirname(str(tmp_path)), "data.jsonl") + with pytest.raises(ValueError): + plan_blame( + "adapter", outside, + layer="x", budget_seconds=3600, num_shards=5, + ) + + +def test_plan_blame_invalid_layer(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + with pytest.raises(ValueError, match="non-empty"): + plan_blame( + adapter, dataset, + layer="", budget_seconds=3600, num_shards=5, + ) + with pytest.raises(ValueError, match="null"): + plan_blame( + adapter, dataset, + layer="x\x00y", budget_seconds=3600, num_shards=5, + ) + + +def test_plan_blame_invalid_shards(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + with pytest.raises(ValueError, match="num_shards"): + plan_blame( + adapter, dataset, + layer="x", budget_seconds=3600, num_shards=1, + ) + with pytest.raises(ValueError, match="num_shards"): + plan_blame( + adapter, dataset, + layer="x", budget_seconds=3600, num_shards=200, + ) + + +@pytest.mark.parametrize("bool_value", [True, False]) +def test_plan_blame_bool_shards(tmp_path, monkeypatch, bool_value): + """bool subclasses int; both True (→1) and False (→0) must be rejected.""" + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + with pytest.raises(TypeError): + plan_blame( + adapter, dataset, + layer="x", budget_seconds=3600, + num_shards=bool_value, # type: ignore[arg-type] + ) + + +def test_plan_blame_bool_budget_rejected(tmp_path, monkeypatch): + """budget_seconds=True must not silently coerce to 1 (project bool policy).""" + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + with pytest.raises(TypeError): + plan_blame( + adapter, dataset, + layer="x", + budget_seconds=True, # type: ignore[arg-type] + num_shards=5, + ) + + +def test_plan_blame_missing_dataset(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "adapter").mkdir() + # File doesn't exist BUT also doesn't exist in cwd containment check + # The containment helper rejects non-existent first. Try with a real + # outside-cwd file to confirm flow. + (tmp_path / "missing.jsonl").write_text("", encoding="utf-8") + with pytest.raises(ValueError): + # Empty file → empty dataset rejection + plan_blame( + "adapter", "missing.jsonl", + layer="x", budget_seconds=3600, num_shards=2, + ) + + +# ---------- run_blame stub ---------- + + +def test_run_blame_stub_marker(): + plan = BlamePlan( + adapter_dir="a", dataset_path="d", layer="x", + budget_seconds=3600, num_shards=2, per_shard_seconds=600, + shards=(BlameShardWork(0, 0, 1, 600), BlameShardWork(1, 1, 1, 600)), + feasible=True, reason="ok", + ) + with pytest.raises(NotImplementedError, match="v0.57.1"): + run_blame(plan) + + +def test_run_blame_rejects_non_plan(): + with pytest.raises(TypeError): + run_blame("not a plan") # type: ignore[arg-type] + + +# ---------- Frozen invariants ---------- + + +def test_blame_plan_frozen(): + import dataclasses + plan = BlamePlan( + adapter_dir="a", dataset_path="d", layer="x", + budget_seconds=3600, num_shards=1, per_shard_seconds=3600, + shards=(), feasible=False, reason="r", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + plan.layer = "y" # type: ignore[misc] + + +def test_blame_shard_work_frozen(): + import dataclasses + shard = BlameShardWork(0, 0, 10, 60) + with pytest.raises(dataclasses.FrozenInstanceError): + shard.shard_id = 1 # type: ignore[misc] + + +# ---------- CLI ---------- + + +def test_adapters_blame_help(): + result = runner.invoke(soup_app, ["adapters", "blame", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--budget" in result.output + assert "--layer" in result.output + assert "--shards" in result.output + + +def test_adapters_blame_plan_only(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + result = runner.invoke(soup_app, [ + "adapters", "blame", adapter, + "--dataset", dataset, + "--layer", "q_proj.7", + "--budget", "1h", + "--shards", "5", + "--plan-only", + ]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "Blame plan" in result.output + + +def test_adapters_blame_invalid_budget(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + result = runner.invoke(soup_app, [ + "adapters", "blame", adapter, + "--dataset", dataset, + "--layer", "x", + "--budget", "abc", + "--shards", "5", + ]) + assert result.exit_code == 2 + assert "Invalid --budget" in result.output + + +def test_adapters_blame_live_runner_advisory(tmp_path, monkeypatch): + adapter, dataset = _setup_blame(tmp_path, monkeypatch) + result = runner.invoke(soup_app, [ + "adapters", "blame", adapter, + "--dataset", dataset, + "--layer", "x", + "--budget", "1h", + "--shards", "5", + ]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "v0.57.1" in result.output diff --git a/tests/test_v0570_part_d.py b/tests/test_v0570_part_d.py new file mode 100644 index 0000000..88cb61a --- /dev/null +++ b/tests/test_v0570_part_d.py @@ -0,0 +1,389 @@ +"""v0.57.0 Part D — adapters branch / checkout / branches list.""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app as soup_app +from soup_cli.utils.adapter_branch import ( + Branch, + create_branch, + delete_branch, + list_branches, + load_branch, + write_checkout, +) + +runner = CliRunner() + + +@pytest.fixture(autouse=True) +def isolated_branches_dir(tmp_path, monkeypatch): + """Redirect SOUP_BRANCHES_DIR to tmp so tests don't pollute ~/.soup.""" + branches = tmp_path / "branches" + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(branches)) + monkeypatch.chdir(tmp_path) + return branches + + +def _make_config(tmp_path: Path, name: str = "soup.yaml", content: str = "base: test\n") -> Path: + p = tmp_path / name + p.write_text(content, encoding="utf-8") + return p + + +# ---------- create_branch ---------- + + +def test_create_branch_happy(tmp_path): + _make_config(tmp_path) + snap = create_branch("v1", config_path="soup.yaml", base_model="meta/llama") + assert isinstance(snap, Branch) + assert snap.name == "v1" + assert snap.base_model == "meta/llama" + assert len(snap.config_sha256) == 64 + assert snap.dataset_sha256 is None + + +def test_create_branch_with_dataset(tmp_path): + _make_config(tmp_path) + (tmp_path / "data.jsonl").write_text('{"x": 1}\n', encoding="utf-8") + snap = create_branch("v1", config_path="soup.yaml", base_model="b", + dataset_path="data.jsonl") + assert snap.dataset_sha256 is not None + assert len(snap.dataset_sha256) == 64 + + +def test_create_branch_invalid_name(tmp_path): + _make_config(tmp_path) + with pytest.raises(ValueError, match="must match"): + create_branch("../etc", config_path="soup.yaml", base_model="b") + with pytest.raises(ValueError, match="non-empty"): + create_branch("", config_path="soup.yaml", base_model="b") + + +def test_create_branch_null_byte_name(tmp_path): + _make_config(tmp_path) + with pytest.raises(ValueError, match="null"): + create_branch("v\x001", config_path="soup.yaml", base_model="b") + + +def test_create_branch_bool_name(tmp_path): + _make_config(tmp_path) + with pytest.raises(TypeError): + create_branch(True, config_path="soup.yaml", base_model="b") # type: ignore[arg-type] + + +def test_create_branch_config_outside_cwd(tmp_path): + outside = os.path.join(os.path.dirname(str(tmp_path)), "x.yaml") + with pytest.raises(ValueError): + create_branch("v1", config_path=outside, base_model="b") + + +def test_create_branch_missing_config(tmp_path): + with pytest.raises(FileNotFoundError): + create_branch("v1", config_path="missing.yaml", base_model="b") + + +def test_create_branch_empty_base_model(tmp_path): + _make_config(tmp_path) + with pytest.raises(ValueError, match="non-empty"): + create_branch("v1", config_path="soup.yaml", base_model="") + + +def test_create_branch_null_byte_base_model(tmp_path): + _make_config(tmp_path) + with pytest.raises(ValueError, match="null"): + create_branch("v1", config_path="soup.yaml", base_model="x\x00") + + +def test_create_branch_bool_base_model_rejected(tmp_path): + """bool subclasses str-checks in some idioms; ensure explicit TypeError.""" + _make_config(tmp_path) + with pytest.raises(TypeError): + create_branch("v1", config_path="soup.yaml", + base_model=True) # type: ignore[arg-type] + + +def test_delete_branch_traversal_rejected(): + with pytest.raises(ValueError): + delete_branch("../etc/passwd") + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink semantics") +def test_load_branch_rejects_symlink(tmp_path, monkeypatch): + branches = tmp_path / "br" + branches.mkdir() + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(branches)) + monkeypatch.chdir(tmp_path) + target = tmp_path / "real.json" + target.write_text("{}", encoding="utf-8") + os.symlink(str(target), str(branches / "evil.json")) + with pytest.raises(ValueError, match="symlink"): + load_branch("evil") + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink semantics") +def test_delete_branch_rejects_symlink(tmp_path, monkeypatch): + branches = tmp_path / "br" + branches.mkdir() + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(branches)) + monkeypatch.chdir(tmp_path) + target = tmp_path / "real.json" + target.write_text("{}", encoding="utf-8") + os.symlink(str(target), str(branches / "evil.json")) + with pytest.raises(ValueError, match="symlink"): + delete_branch("evil") + # original file untouched + assert target.exists() + + +def test_branches_dir_crlf_env_rejected(monkeypatch, tmp_path): + """Control char (CRLF) in SOUP_BRANCHES_DIR falls back to default.""" + from soup_cli.utils.adapter_branch import _branches_dir + monkeypatch.setenv("SOUP_BRANCHES_DIR", "/some\npath") + monkeypatch.chdir(tmp_path) + resolved = _branches_dir() + assert "\n" not in str(resolved) + + +def test_create_branch_oversize_config(tmp_path): + # 1 MiB + 1 byte + (tmp_path / "soup.yaml").write_bytes(b"x" * (1_048_577)) + with pytest.raises(ValueError, match="cap"): + create_branch("v1", config_path="soup.yaml", base_model="b") + + +def test_create_branch_atomic_write(tmp_path, isolated_branches_dir): + _make_config(tmp_path) + snap = create_branch("v1", config_path="soup.yaml", base_model="b") + target = isolated_branches_dir / "v1.json" + assert target.exists() + raw = json.loads(target.read_text(encoding="utf-8")) + assert raw["name"] == "v1" + assert raw["config_sha256"] == snap.config_sha256 + + +# ---------- list_branches ---------- + + +def test_list_branches_empty(tmp_path): + assert list_branches() == () + + +def test_list_branches_sorted(tmp_path): + _make_config(tmp_path) + create_branch("b", config_path="soup.yaml", base_model="x") + create_branch("a", config_path="soup.yaml", base_model="x") + create_branch("c", config_path="soup.yaml", base_model="x") + assert list_branches() == ("a", "b", "c") + + +# ---------- load_branch ---------- + + +def test_load_branch_roundtrip(tmp_path): + _make_config(tmp_path) + created = create_branch("v1", config_path="soup.yaml", base_model="m") + loaded = load_branch("v1") + assert loaded.name == created.name + assert loaded.config_sha256 == created.config_sha256 + assert loaded.base_model == created.base_model + + +def test_load_branch_missing(tmp_path): + with pytest.raises(FileNotFoundError): + load_branch("nope") + + +def test_load_branch_invalid_name(): + with pytest.raises(ValueError): + load_branch("../etc") + + +# ---------- delete_branch ---------- + + +def test_delete_branch_true_when_present(tmp_path): + _make_config(tmp_path) + create_branch("v1", config_path="soup.yaml", base_model="m") + assert delete_branch("v1") is True + assert "v1" not in list_branches() + + +def test_delete_branch_false_when_missing(): + assert delete_branch("nope") is False + + +# ---------- write_checkout ---------- + + +def test_write_checkout_writes_target(tmp_path): + _make_config(tmp_path, content="base: original\n") + snap = create_branch("v1", config_path="soup.yaml", base_model="m") + written = write_checkout(snap, "restored.yaml") + assert written.exists() + assert "original" in written.read_text(encoding="utf-8") + + +def test_write_checkout_detects_drift(tmp_path): + _make_config(tmp_path, content="base: original\n") + snap = create_branch("v1", config_path="soup.yaml", base_model="m") + # Mutate original config — SHA mismatch + (tmp_path / "soup.yaml").write_text("base: drifted\n", encoding="utf-8") + with pytest.raises(ValueError, match="drifted"): + write_checkout(snap, "restored.yaml") + + +def test_write_checkout_outside_cwd_rejected(tmp_path): + _make_config(tmp_path) + snap = create_branch("v1", config_path="soup.yaml", base_model="m") + outside = os.path.join(os.path.dirname(str(tmp_path)), "x.yaml") + with pytest.raises(ValueError): + write_checkout(snap, outside) + + +def test_write_checkout_rejects_non_branch(): + with pytest.raises(TypeError): + write_checkout("not a branch", "out.yaml") # type: ignore[arg-type] + + +def test_write_checkout_missing_source(tmp_path): + _make_config(tmp_path) + snap = create_branch("v1", config_path="soup.yaml", base_model="m") + (tmp_path / "soup.yaml").unlink() + with pytest.raises(FileNotFoundError): + write_checkout(snap, "restored.yaml") + + +# ---------- Frozen Branch ---------- + + +def test_branch_frozen(tmp_path): + import dataclasses + _make_config(tmp_path) + snap = create_branch("v1", config_path="soup.yaml", base_model="m") + with pytest.raises(dataclasses.FrozenInstanceError): + snap.name = "v2" # type: ignore[misc] + + +# ---------- Env override containment ---------- + + +def test_branches_dir_env_override_outside_falls_back(monkeypatch, tmp_path): + """Out-of-bounds env override is rejected; helper returns the in-bounds default. + + We point SOUP_BRANCHES_DIR at a tmp-rooted path that IS in-bounds (under + $TMPDIR per the containment policy) so we can also assert that valid + overrides ARE honoured — i.e. we test both directions of the policy here. + """ + from soup_cli.utils.adapter_branch import _branches_dir + in_bounds = tmp_path / "valid-override" + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(in_bounds)) + monkeypatch.chdir(tmp_path) + resolved = _branches_dir() + assert os.path.realpath(str(resolved)) == os.path.realpath(str(in_bounds)) + + +def test_branches_dir_null_byte_env_ignored(monkeypatch, tmp_path): + """Null-byte env override is silently ignored; default ~/.soup/branches used.""" + from soup_cli.utils.adapter_branch import _branches_dir + monkeypatch.setenv("SOUP_BRANCHES_DIR", "/some\x00path") + monkeypatch.chdir(tmp_path) + resolved = _branches_dir() + # Falls back to ~/.soup/branches — must NOT contain a null byte + assert "\x00" not in str(resolved) + assert ".soup" in str(resolved) and "branches" in str(resolved) + + +# ---------- CLI smoke ---------- + + +def test_branch_cli_help(): + result = runner.invoke(soup_app, ["adapters", "branch", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_checkout_cli_help(): + result = runner.invoke(soup_app, ["adapters", "checkout", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_branches_cli_help(): + result = runner.invoke(soup_app, ["adapters", "branches", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + +def test_branch_cli_create_and_list(tmp_path, monkeypatch): + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br")) + monkeypatch.chdir(tmp_path) + _make_config(tmp_path) + r1 = runner.invoke(soup_app, [ + "adapters", "branch", "v1", + "-c", "soup.yaml", "--base", "meta/llama", + ]) + assert r1.exit_code == 0, (r1.output, repr(r1.exception)) + assert "v1" in r1.output + + r2 = runner.invoke(soup_app, ["adapters", "branches"]) + assert r2.exit_code == 0, (r2.output, repr(r2.exception)) + assert "v1" in r2.output + + +def test_branch_cli_invalid_name(tmp_path, monkeypatch): + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br")) + monkeypatch.chdir(tmp_path) + _make_config(tmp_path) + result = runner.invoke(soup_app, [ + "adapters", "branch", "../etc", + "-c", "soup.yaml", "--base", "m", + ]) + assert result.exit_code == 2 + assert "must match" in result.output + + +def test_branch_cli_missing_config(tmp_path, monkeypatch): + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br")) + monkeypatch.chdir(tmp_path) + result = runner.invoke(soup_app, [ + "adapters", "branch", "v1", + "-c", "missing.yaml", "--base", "m", + ]) + assert result.exit_code == 1 + assert "not found" in result.output + + +def test_checkout_cli_roundtrip(tmp_path, monkeypatch): + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br")) + monkeypatch.chdir(tmp_path) + _make_config(tmp_path, content="base: x\nepochs: 1\n") + create_branch("v1", config_path="soup.yaml", base_model="m") + result = runner.invoke(soup_app, [ + "adapters", "checkout", "v1", + "-o", "restored.yaml", + ]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "restored.yaml").exists() + + +def test_checkout_cli_missing_branch(tmp_path, monkeypatch): + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br")) + monkeypatch.chdir(tmp_path) + result = runner.invoke(soup_app, [ + "adapters", "checkout", "nope", "-o", "out.yaml", + ]) + assert result.exit_code == 1 + assert "not found" in result.output + + +def test_branches_cli_empty(tmp_path, monkeypatch): + monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br")) + monkeypatch.chdir(tmp_path) + result = runner.invoke(soup_app, ["adapters", "branches"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "No branches" in result.output