mirror of https://github.com/razor-ai/soup.git
feat(adapters): v0.57.0 — git for LoRA (diff, merge, blame, branch)
Ships "soup adapters {diff,merge,blame,branch,checkout,branches}" — git-shaped
UX for LoRA adapter management. Pure-numpy math (no torch); TOCTOU defence on
every path; atomic writes throughout.
Part A — adapters diff (utils/adapter_diff.py, ~270 LOC):
Per-layer ΔW Frobenius norm + relative drift; effective-rank delta via
SVD entropy; top-K changed projections; JSON/Markdown/table output.
Part B — adapters merge (utils/adapter_merge.py, ~280 LOC):
Four strategies — linear (weighted avg), ties (Yadav et al. — trim/
elect-sign/disjoint), dare (Yu et al. — drop+rescale, deterministic),
svd (low-rank reconstruction). MergeReport.verdict='UNKNOWN' is a stub;
live canary verdict via v0.55 eval gate ships in v0.57.1.
Part C — adapters blame (utils/blame.py, ~190 LOC):
Leave-one-out plan emitter + budget tracker (parse_budget mirrors v0.48.0
data_mix idiom). Per-shard work table + feasibility check. Live ablation
runner raises NotImplementedError v0.57.1 (mirrors v0.27.0 / v0.50.0 /
v0.56.0 stub-then-live pattern).
Part D — adapters branch / checkout / branches (utils/adapter_branch.py,
~230 LOC):
SHA-256 snapshot pointers under ~/.soup/branches/ (SOUP_BRANCHES_DIR
override, $HOME/$CWD/$TMPDIR-bounded). Drift detection on checkout —
refuses restore when source SHA != snapshot SHA. CRLF/null-byte
rejection on env override (mirrors v0.51.0 hub-endpoint policy).
5-agent review-fix wave (1 CRITICAL + 9 HIGH + 11 MEDIUM + 4 LOW):
- TIES tied-sign defaults to +1 (np.sign(0)==0 would silently zero all
tied parameters)
- load_branch / delete_branch reject symlinks via os.lstat + S_ISLNK
before read/unlink
- merge output safetensors + adapter_config.json atomic writes with
symlink target rejection at output path; source config size-capped
at 256 KB
- compute_adapter_diff weights-file path symlink-rejected via lstat
BEFORE is_file() (defends against .safetensors -> /etc/passwd escape)
- _count_dataset_rows opens via realpath captured at containment check
(closes TOCTOU window)
- diff --output write is atomic (tempfile + os.replace)
- SOUP_BRANCHES_DIR rejects every C0 control char, not just null
- SUPPORTED_STRATEGIES is now frozenset (matches v0.41.0+ allowlist
policy); STRATEGY_ORDER tuple preserved for canonical iteration
- 5× pytest.raises(Exception) tightened to FrozenInstanceError
- 2 zero-assertion Part D tests converted to real assertions
- Added: bool base_model rejection, top_k boundary 1/201, density=1.0
inclusive bound, inf weight rejection, tied-sign positive default,
bool False for num_shards/budget_seconds, POSIX symlink rejections
for diff weights / merge output / load_branch / delete_branch,
no-top-level-torch source-grep guards, traversal delete_branch.
Plus v0.56.0 follow-up: test_v0560.py version-floor tests widened from
exact-match to floor-check (matches v0.51.0 / v0.54.0 idiom — every
subsequent release would otherwise edit this one line).
Test count: 8849 → 8998 (+149 net in 4 new files; 4 POSIX-only symlink
tests skipped on Windows). Full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
a3810823d1
commit
7da82d355a
14
README.md
14
README.md
|
|
@ -42,14 +42,14 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.56.0 — `soup diagnose`: lighthouse score for fine-tunes.** Training finishes and you have no idea *why* the adapter underperforms. Loss curves can't distinguish overfitting from refusal regression from mode collapse. v0.56 ships a 6-mode post-training report card with one verdict.
|
||||
**v0.57.0 — `soup adapters`: git for LoRA.** Three years of `lora_v3_final_final2/` ends here. No diff, no merge, no rollback, no attribution from a weight change back to the dataset slice that caused it — until now. v0.57 ships git-shaped UX on top of the v0.22 adapter surface.
|
||||
|
||||
- **`soup diagnose <run-id>`** scores six failure modes — `forgetting` (Δ accuracy vs base), `refusal` (advbench / xstest delta), `format` (JSON / regex / tool-call validity), `mode_collapse` (pairwise diversity at T=0 and T=1), `memorization` (training-prefix echo on partial-prompt probes), `contamination` (training-row overlap with public benchmarks) — and emits an overall OK / MINOR / MAJOR verdict using the same taxonomy as v0.26 Quant-Lobotomy.
|
||||
- **`soup diagnose <run-id> --badge diag.svg`** renders a self-contained 6-cell SVG report card with the overall verdict pill — embeddable in HF model cards and Twitter posts. Every user-controlled string passes through `html.escape`; no JS, no external deps.
|
||||
- **`soup diagnose <run-id> --evidence ev.json --output diag.json --attach-to-registry <id>`** persists the report as a `diagnose_report` artifact on the v0.26 Model Registry — the first-class kind alongside `eval_suite` / `canaries` / `tensorrt`.
|
||||
- **`soup train --diagnose-gate <evidence.json>`** refuses to mark a run successful when any mode comes back MAJOR. Composes with v0.26 `--gate` and v0.55 `eval gate-install` for a three-layer safety net: training process anomalies (v0.34 `soup why`) → eval regression vs baseline (v0.55 gate) → model-behaviour failure modes (v0.56 diagnose).
|
||||
- **Why blue-ocean.** Trainer libs ship code, not interpretability. TRL's v1 RFC narrows public surface — failure analysis falls below the cut. Eval SaaS (Braintrust, LangSmith) treat fine-tuning as <5 % of workload. The failure-mode taxonomy lives in academic papers with no implementation owner. Plus: builders care about this enough to ship a Twitter-shareable badge.
|
||||
- **+123 new tests** (8676 → 8849) across `tests/test_v0560.py` covering all 6 probes + FailureReport frozen invariants + badge SVG escapes + 4-agent review-fix wave (python / security / code / tdd): atomic+symlink-safe badge write, 16 MiB evidence size cap, `typer.Exit` instead of `sys.exit`, `os.path.realpath` containment, contamination combined-complexity cap, ReDoS probe in `matches_regex`, extras null-byte sanitisation.
|
||||
- **`soup adapters diff <a> <b>`** — per-layer ΔW Frobenius norm + relative drift + effective-rank delta via SVD entropy, with top-K changed projections highlighted. Output as a Rich table, machine-readable JSON, or PR-ready Markdown via `--format {table,json,markdown} --output report.json`.
|
||||
- **`soup adapters merge <a> <b> [c...] -o <out> --strategy {linear,ties,dare,svd}`** — four merge strategies in pure numpy: weighted linear, TIES (trim/elect-sign/disjoint avg per Yadav et al.), DARE (drop-and-rescale per Yu et al., deterministic via `--seed`), and SVD low-rank reconstruction (`--rank` clamped to min-dim). Output safetensors + `adapter_config.json` both written atomically.
|
||||
- **`soup adapters blame <adapter> --dataset <d> --layer q_proj.7 --budget 4h`** — leave-one-out ablation plan: splits the dataset into N shards, estimates per-shard ablation runtime against your wall-clock budget, and emits a per-shard work table with feasibility check. Live ablation runner (training at 1/10 scale per shard with the v0.34 SQLite tracker + v0.26 Registry lineage) is wired in **v0.57.1**.
|
||||
- **`soup adapters branch <name> -c soup.yaml --base meta/llama-3.1`** + **`soup adapters checkout <name> -o restored.yaml`** + **`soup adapters branches`** — SHA-256 snapshot pointers under `~/.soup/branches/` (or `SOUP_BRANCHES_DIR`-override, $HOME/$CWD/$TMPDIR-bounded). `checkout` refuses to restore when the source config has drifted from the snapshot SHA (no silent reproducibility loss).
|
||||
- **Why blue-ocean.** HF Hub treats every revision as an opaque blob and won't ship weight-aware diffs (it would balkanise their storage backend). DVC / lakeFS are file-system primitives, not LoRA-aware. PEFT exposes `add_weighted_adapter`, mergekit exists — but no VCS-shaped UX wraps them. LLaMA-Factory closed #2038 (weighted merge) as not-planned. Git-semantics-for-tensors is a seam neither the registry nor the kernel teams will build.
|
||||
- **+149 new tests** (8849 → 8998) across `tests/test_v0570_part_{a,b,c,d}.py`. 4-agent review-fix wave landed (1 CRITICAL: zero-assertion tests, 9 HIGH including TIES tied-sign positive default + 4× symlink rejections, 11 MEDIUM, 4 LOW): atomic writes + lstat+S_ISLNK rejection on every output, frozen dataclasses with FrozenInstanceError assertions, CRLF/null-byte env-var rejection, bool-as-int rejection on every numeric input, source-grep regression guards for the lazy-import policy.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.56.0"
|
||||
version = "0.57.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.56.0"
|
||||
__version__ = "0.57.0"
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
"""soup adapters — LoRA adapter management (list, info, compare)."""
|
||||
"""soup adapters — LoRA adapter management (list, info, compare, diff, merge, blame, branch)."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
|
||||
|
|
@ -209,3 +210,320 @@ def compare(
|
|||
table.add_row("Size on disk", size1, size2)
|
||||
|
||||
console.print(table)
|
||||
|
||||
|
||||
@app.command()
|
||||
def diff(
|
||||
adapter_a: str = typer.Argument(..., help="Path to first adapter"),
|
||||
adapter_b: str = typer.Argument(..., help="Path to second adapter"),
|
||||
top_k: int = typer.Option(10, "--top-k", min=1, max=200,
|
||||
help="Number of top changed projections to report"),
|
||||
output_format: str = typer.Option("table", "--format",
|
||||
help="Output format: table | json | markdown"),
|
||||
output: str = typer.Option(None, "--output", "-o",
|
||||
help="Write report to file (json/markdown only)"),
|
||||
):
|
||||
"""Per-layer ΔW Frobenius diff + effective-rank drift (v0.57.0)."""
|
||||
from soup_cli.utils.adapter_diff import (
|
||||
compute_adapter_diff,
|
||||
render_report_json,
|
||||
render_report_markdown,
|
||||
)
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
fmt = output_format.lower()
|
||||
if fmt not in ("table", "json", "markdown"):
|
||||
console.print(f"[red]Unknown --format: {escape(fmt)}[/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
try:
|
||||
report = compute_adapter_diff(adapter_a, adapter_b, top_k=top_k)
|
||||
except FileNotFoundError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
except RuntimeError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
if fmt == "json":
|
||||
text = render_report_json(report)
|
||||
elif fmt == "markdown":
|
||||
text = render_report_markdown(report)
|
||||
else:
|
||||
text = None
|
||||
|
||||
if output is not None:
|
||||
if fmt == "table":
|
||||
console.print("[red]--output requires --format json or markdown[/]")
|
||||
raise typer.Exit(2)
|
||||
enforce_under_cwd_and_no_symlink(output, "output")
|
||||
# Atomic write via tempfile + os.replace — a crash mid-write must
|
||||
# not leave a partial report at the target path (review fix MEDIUM).
|
||||
import os as _os
|
||||
import tempfile as _tf
|
||||
target = Path(output)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = _tf.mkstemp(dir=str(target.parent), prefix=".tmp_")
|
||||
try:
|
||||
with _os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(text)
|
||||
_os.replace(tmp, str(target))
|
||||
except Exception:
|
||||
try:
|
||||
_os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
console.print(f"[green]Wrote {fmt} report to {escape(output)}[/]")
|
||||
return
|
||||
|
||||
if fmt != "table":
|
||||
console.print(text)
|
||||
return
|
||||
|
||||
# Default Rich table
|
||||
table = Table(title=f"Adapter diff: {escape(report.adapter_a)} vs {escape(report.adapter_b)}")
|
||||
table.add_column("Layer", style="bold")
|
||||
table.add_column("ΔW Frobenius", justify="right")
|
||||
table.add_column("Relative", justify="right")
|
||||
for layer in sorted(report.per_layer, key=lambda d: d.frobenius, reverse=True)[:top_k]:
|
||||
table.add_row(
|
||||
escape(layer.name),
|
||||
f"{layer.frobenius:.4f}",
|
||||
f"{layer.relative:.2%}",
|
||||
)
|
||||
console.print(table)
|
||||
if report.effective_rank_a is not None and report.effective_rank_b is not None:
|
||||
console.print(
|
||||
f"Effective rank: A={report.effective_rank_a:.2f}, "
|
||||
f"B={report.effective_rank_b:.2f}"
|
||||
)
|
||||
console.print(
|
||||
f"Shared layers: {report.shared_layers} | "
|
||||
f"only-in-A: {len(report.only_in_a)} | only-in-B: {len(report.only_in_b)}"
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def merge(
|
||||
adapters: list[str] = typer.Argument(..., help="Two or more adapter paths to merge"),
|
||||
output: str = typer.Option(..., "--output", "-o", help="Output directory for merged adapter"),
|
||||
strategy: str = typer.Option("linear", "--strategy",
|
||||
help="linear | ties | dare | svd"),
|
||||
weights: str = typer.Option(None, "--weights",
|
||||
help="Comma-separated weights (default: equal)"),
|
||||
density: float = typer.Option(0.2, "--density",
|
||||
help="Trim density for ties/dare in (0, 1]"),
|
||||
seed: int = typer.Option(0, "--seed", help="Random seed for dare"),
|
||||
rank: int = typer.Option(None, "--rank", help="SVD rank (svd strategy only)"),
|
||||
):
|
||||
"""Merge LoRA adapters via linear / ties / dare / svd (v0.57.0)."""
|
||||
from soup_cli.utils.adapter_merge import SUPPORTED_STRATEGIES, merge_adapters
|
||||
|
||||
if strategy not in SUPPORTED_STRATEGIES:
|
||||
console.print(
|
||||
f"[red]Unknown --strategy: {escape(strategy)}. "
|
||||
f"Choose from: {', '.join(SUPPORTED_STRATEGIES)}[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
|
||||
if len(adapters) < 2:
|
||||
console.print("[red]Need at least 2 adapter paths to merge[/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
parsed_weights = None
|
||||
if weights:
|
||||
try:
|
||||
parsed_weights = [float(w.strip()) for w in weights.split(",")]
|
||||
except ValueError as exc:
|
||||
console.print(f"[red]Invalid --weights: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
try:
|
||||
report = merge_adapters(
|
||||
adapters,
|
||||
output,
|
||||
strategy=strategy, # type: ignore[arg-type]
|
||||
weights=parsed_weights,
|
||||
density=density,
|
||||
seed=seed,
|
||||
rank=rank,
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
panel = Panel(
|
||||
f"Strategy: [bold]{escape(report.strategy)}[/]\n"
|
||||
f"Inputs: {len(report.adapters)}\n"
|
||||
f"Merged layers: [bold]{report.merged_layers}[/]\n"
|
||||
f"Skipped layers: {len(report.skipped_layers)}\n"
|
||||
f"Verdict: [yellow]{report.verdict}[/] (live eval in v0.57.1)\n"
|
||||
f"Output: [bold]{escape(report.output_dir)}[/]",
|
||||
title="Adapter merge",
|
||||
)
|
||||
console.print(panel)
|
||||
|
||||
|
||||
@app.command()
|
||||
def blame(
|
||||
adapter_dir: str = typer.Argument(..., help="Path to trained adapter"),
|
||||
dataset: str = typer.Option(..., "--dataset", help="Training JSONL the adapter was built on"),
|
||||
layer: str = typer.Option(..., "--layer", help="Layer to attribute (e.g. q_proj.7)"),
|
||||
budget: str = typer.Option("4h", "--budget", help="Wall-clock budget (e.g. 4h, 30m)"),
|
||||
num_shards: int = typer.Option(10, "--shards", min=2, max=100,
|
||||
help="Number of dataset shards for leave-one-out"),
|
||||
plan_only: bool = typer.Option(False, "--plan-only",
|
||||
help="Print plan and exit (live runner in v0.57.1)"),
|
||||
):
|
||||
"""Attribute weight movement to dataset shards via leave-one-out ablation (v0.57.0)."""
|
||||
from soup_cli.utils.blame import parse_budget, plan_blame, run_blame
|
||||
|
||||
try:
|
||||
seconds = parse_budget(budget)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid --budget: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
try:
|
||||
plan = plan_blame(
|
||||
adapter_dir, dataset,
|
||||
layer=layer, budget_seconds=seconds, num_shards=num_shards,
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
table = Table(title=f"Blame plan: {escape(plan.layer)}")
|
||||
table.add_column("Shard", justify="right")
|
||||
table.add_column("Hold-out offset", justify="right")
|
||||
table.add_column("Hold-out rows", justify="right")
|
||||
table.add_column("Projected seconds", justify="right")
|
||||
for shard in plan.shards:
|
||||
table.add_row(
|
||||
str(shard.shard_id),
|
||||
str(shard.holdout_offset),
|
||||
str(shard.holdout_size),
|
||||
str(shard.projected_seconds),
|
||||
)
|
||||
console.print(table)
|
||||
status_color = "green" if plan.feasible else "yellow"
|
||||
console.print(
|
||||
f"Budget {plan.budget_seconds}s, "
|
||||
f"{plan.per_shard_seconds}s/shard — "
|
||||
f"[{status_color}]{escape(plan.reason)}[/]"
|
||||
)
|
||||
|
||||
if plan_only or not plan.feasible:
|
||||
return
|
||||
|
||||
try:
|
||||
run_blame(plan)
|
||||
except NotImplementedError as exc:
|
||||
console.print(f"[yellow]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(0) from None
|
||||
|
||||
|
||||
@app.command()
|
||||
def branch(
|
||||
name: str = typer.Argument(..., help="Branch name (alphanumeric + ._-)"),
|
||||
config: str = typer.Option(..., "--config", "-c", help="Path to soup.yaml"),
|
||||
base: str = typer.Option(..., "--base", help="Base model id"),
|
||||
dataset: str = typer.Option(None, "--dataset", help="Training dataset path (optional)"),
|
||||
):
|
||||
"""Snapshot a training environment as a comparable branch (v0.57.0)."""
|
||||
from soup_cli.utils.adapter_branch import create_branch
|
||||
|
||||
try:
|
||||
snap = create_branch(name, config_path=config, base_model=base,
|
||||
dataset_path=dataset)
|
||||
except FileNotFoundError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Name: [bold]{escape(snap.name)}[/]\n"
|
||||
f"Base: [bold]{escape(snap.base_model)}[/]\n"
|
||||
f"Config: [bold]{escape(snap.config_path)}[/]\n"
|
||||
f"SHA: [dim]{snap.config_sha256[:16]}...[/]\n"
|
||||
f"Dataset SHA: [dim]"
|
||||
f"{snap.dataset_sha256[:16] + '...' if snap.dataset_sha256 else '—'}[/]\n"
|
||||
f"Version: {snap.soup_version}",
|
||||
title="Adapter branch",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def checkout(
|
||||
name: str = typer.Argument(..., help="Branch name to check out"),
|
||||
output: str = typer.Option("soup.yaml", "--output", "-o",
|
||||
help="Where to write the restored config"),
|
||||
):
|
||||
"""Restore a snapshotted branch's config into cwd (v0.57.0)."""
|
||||
from soup_cli.utils.adapter_branch import load_branch, write_checkout
|
||||
|
||||
try:
|
||||
snap = load_branch(name)
|
||||
except FileNotFoundError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
try:
|
||||
target = write_checkout(snap, output)
|
||||
except FileNotFoundError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
f"[green]Restored branch {escape(snap.name)} → {escape(str(target))}[/]"
|
||||
)
|
||||
|
||||
|
||||
@app.command(name="branches")
|
||||
def list_branches_cmd():
|
||||
"""List all snapshotted branches (v0.57.0)."""
|
||||
from soup_cli.utils.adapter_branch import list_branches, load_branch
|
||||
|
||||
names = list_branches()
|
||||
if not names:
|
||||
console.print("[dim]No branches yet.[/]")
|
||||
return
|
||||
|
||||
table = Table(title=f"Adapter branches ({len(names)})")
|
||||
table.add_column("Name", style="bold")
|
||||
table.add_column("Base model")
|
||||
table.add_column("Config SHA")
|
||||
table.add_column("Data SHA")
|
||||
for branch_name in names:
|
||||
try:
|
||||
snap = load_branch(branch_name)
|
||||
ds = snap.dataset_sha256[:8] + "..." if snap.dataset_sha256 else "—"
|
||||
table.add_row(
|
||||
escape(snap.name),
|
||||
escape(snap.base_model),
|
||||
snap.config_sha256[:8] + "...",
|
||||
ds,
|
||||
)
|
||||
except (ValueError, FileNotFoundError, OSError):
|
||||
table.add_row(escape(branch_name), "[red]error[/]", "-", "-")
|
||||
console.print(table)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,278 @@
|
|||
"""Lightweight snapshot pointers for adapter training (v0.57.0 Part D).
|
||||
|
||||
A `branch` is a frozen pointer to {config, dataset SHA, base model, created_at}
|
||||
so two trainings can be compared cleanly. Pointers live under
|
||||
``~/.soup/branches/<name>.json`` (or ``SOUP_BRANCHES_DIR``-overridden) with
|
||||
``0o600`` perms.
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``create_branch(name, *, config_path, base_model, dataset_path=None)`` -> ``Branch``
|
||||
- ``list_branches()`` -> tuple[str, ...]
|
||||
- ``load_branch(name)`` -> ``Branch``
|
||||
- ``delete_branch(name)`` -> bool
|
||||
- ``write_checkout(branch, target_path)`` -> snapshot config back to cwd
|
||||
|
||||
Validation policy mirrors the project standard: alphanumeric + ``._-``,
|
||||
≤128 chars, null-byte rejected, branches dir is containment-checked under
|
||||
``$HOME`` / ``$CWD`` / ``$TMPDIR`` per v0.36.0 ``SOUP_BATCH_CACHE_PATH`` policy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink, is_under
|
||||
|
||||
_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._\-]{0,127}$")
|
||||
_MAX_BRANCHES = 1024
|
||||
_HASH_CHUNK = 65536
|
||||
_MAX_CONFIG_BYTES = 1_048_576 # 1 MiB cap on snapshot config payload
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Branch:
|
||||
name: str
|
||||
config_path: str
|
||||
config_sha256: str
|
||||
dataset_sha256: Optional[str]
|
||||
base_model: str
|
||||
created_at: float
|
||||
soup_version: str
|
||||
|
||||
|
||||
def _validate_name(name: object) -> str:
|
||||
if isinstance(name, bool) or not isinstance(name, str):
|
||||
raise TypeError("name must be str")
|
||||
if not name:
|
||||
raise ValueError("name must be non-empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("name must not contain null bytes")
|
||||
if not _NAME_RE.match(name):
|
||||
raise ValueError(
|
||||
"name must match [A-Za-z0-9][A-Za-z0-9._-]{0,127}"
|
||||
)
|
||||
return name
|
||||
|
||||
|
||||
def _branches_dir() -> Path:
|
||||
"""Resolve the branches directory with env-var override + containment."""
|
||||
override = os.environ.get("SOUP_BRANCHES_DIR")
|
||||
if override:
|
||||
# Reject null + every C0 control char (CRLF, tabs, etc.) — mirrors
|
||||
# v0.51.0 validate_hub_endpoint policy. Silently fall through to
|
||||
# the default rather than raise; an env var is operator-supplied
|
||||
# configuration, not API input.
|
||||
if any(ord(ch) < 0x20 for ch in override):
|
||||
override = None
|
||||
if override:
|
||||
candidate = Path(os.path.realpath(override))
|
||||
bounds = [
|
||||
Path.home(),
|
||||
Path.cwd(),
|
||||
Path(tempfile.gettempdir()),
|
||||
]
|
||||
if any(is_under(str(candidate), str(b)) for b in bounds):
|
||||
candidate.mkdir(parents=True, exist_ok=True)
|
||||
return candidate
|
||||
default = Path.home() / ".soup" / "branches"
|
||||
default.mkdir(parents=True, exist_ok=True)
|
||||
return default
|
||||
|
||||
|
||||
def _hash_file(path: Path) -> str:
|
||||
hasher = hashlib.sha256()
|
||||
with open(path, "rb") as fh:
|
||||
while True:
|
||||
chunk = fh.read(_HASH_CHUNK)
|
||||
if not chunk:
|
||||
break
|
||||
hasher.update(chunk)
|
||||
return hasher.hexdigest()
|
||||
|
||||
|
||||
def _validate_str_field(value: object, field: str, max_len: int = 512) -> str:
|
||||
if isinstance(value, bool) or not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > max_len:
|
||||
raise ValueError(f"{field} must be ≤{max_len} chars")
|
||||
return value
|
||||
|
||||
|
||||
def create_branch(
|
||||
name: str,
|
||||
*,
|
||||
config_path: str,
|
||||
base_model: str,
|
||||
dataset_path: Optional[str] = None,
|
||||
) -> Branch:
|
||||
"""Snapshot a training environment as an immutable branch pointer.
|
||||
|
||||
The config file is hashed (SHA-256) and the dataset file (if provided)
|
||||
is hashed too so callers can detect "same config, different data" drift.
|
||||
"""
|
||||
from soup_cli import __version__
|
||||
|
||||
name = _validate_name(name)
|
||||
enforce_under_cwd_and_no_symlink(config_path, "config_path")
|
||||
_validate_str_field(base_model, "base_model", max_len=512)
|
||||
|
||||
config_full = Path(config_path)
|
||||
if not config_full.is_file():
|
||||
raise FileNotFoundError(f"config not found: {config_full.name}")
|
||||
if config_full.stat().st_size > _MAX_CONFIG_BYTES:
|
||||
raise ValueError(
|
||||
f"config exceeds {_MAX_CONFIG_BYTES} byte cap"
|
||||
)
|
||||
|
||||
config_sha = _hash_file(config_full)
|
||||
|
||||
dataset_sha: Optional[str] = None
|
||||
if dataset_path is not None:
|
||||
enforce_under_cwd_and_no_symlink(dataset_path, "dataset_path")
|
||||
ds_full = Path(dataset_path)
|
||||
if not ds_full.is_file():
|
||||
raise FileNotFoundError(f"dataset not found: {ds_full.name}")
|
||||
dataset_sha = _hash_file(ds_full)
|
||||
|
||||
branches_dir = _branches_dir()
|
||||
existing = sorted(p for p in branches_dir.glob("*.json"))
|
||||
if len(existing) >= _MAX_BRANCHES:
|
||||
raise RuntimeError(f"branches dir has ≥{_MAX_BRANCHES} entries")
|
||||
|
||||
branch = Branch(
|
||||
name=name,
|
||||
config_path=str(config_full),
|
||||
config_sha256=config_sha,
|
||||
dataset_sha256=dataset_sha,
|
||||
base_model=base_model,
|
||||
created_at=time.time(),
|
||||
soup_version=__version__,
|
||||
)
|
||||
_atomic_write_branch(branches_dir, branch)
|
||||
return branch
|
||||
|
||||
|
||||
def _atomic_write_branch(branches_dir: Path, branch: Branch) -> None:
|
||||
target = branches_dir / f"{branch.name}.json"
|
||||
payload = json.dumps(asdict(branch), indent=2, sort_keys=True)
|
||||
fd, tmp_path = tempfile.mkstemp(dir=str(branches_dir), prefix=".tmp_")
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(payload)
|
||||
os.replace(tmp_path, str(target))
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
if os.name != "nt":
|
||||
try:
|
||||
os.chmod(str(target), 0o600)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def list_branches() -> Tuple[str, ...]:
|
||||
branches_dir = _branches_dir()
|
||||
return tuple(sorted(p.stem for p in branches_dir.glob("*.json")))
|
||||
|
||||
|
||||
def load_branch(name: str) -> Branch:
|
||||
name = _validate_name(name)
|
||||
branches_dir = _branches_dir()
|
||||
target = branches_dir / f"{name}.json"
|
||||
if not target.is_file():
|
||||
raise FileNotFoundError(f"branch not found: {name}")
|
||||
# TOCTOU defence: reject symlinks at the branch pointer path before read
|
||||
# so a planted symlink to /etc/passwd cannot leak file content via the
|
||||
# JSON-parse error path (review fix HIGH).
|
||||
st = os.lstat(str(target))
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(f"branch pointer must not be a symlink: {name}")
|
||||
text = target.read_text(encoding="utf-8")
|
||||
if len(text) > _MAX_CONFIG_BYTES:
|
||||
raise ValueError("branch pointer exceeds size cap")
|
||||
raw = json.loads(text)
|
||||
return Branch(
|
||||
name=_validate_name(raw["name"]),
|
||||
config_path=_validate_str_field(raw["config_path"], "config_path", max_len=4096),
|
||||
config_sha256=_validate_str_field(raw["config_sha256"], "config_sha256", max_len=128),
|
||||
dataset_sha256=(
|
||||
_validate_str_field(raw["dataset_sha256"], "dataset_sha256", max_len=128)
|
||||
if raw.get("dataset_sha256") is not None
|
||||
else None
|
||||
),
|
||||
base_model=_validate_str_field(raw["base_model"], "base_model"),
|
||||
created_at=float(raw["created_at"]),
|
||||
soup_version=_validate_str_field(raw["soup_version"], "soup_version", max_len=64),
|
||||
)
|
||||
|
||||
|
||||
def delete_branch(name: str) -> bool:
|
||||
name = _validate_name(name)
|
||||
branches_dir = _branches_dir()
|
||||
target = branches_dir / f"{name}.json"
|
||||
if not target.is_file():
|
||||
return False
|
||||
# TOCTOU defence: reject symlinks before unlink so a planted symlink
|
||||
# at <branches>/<name>.json -> /important/file cannot be silently deleted.
|
||||
st = os.lstat(str(target))
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(f"branch pointer must not be a symlink: {name}")
|
||||
target.unlink()
|
||||
return True
|
||||
|
||||
|
||||
def write_checkout(branch: Branch, target_path: str) -> Path:
|
||||
"""Copy the snapshotted config into cwd so the user can re-run."""
|
||||
if not isinstance(branch, Branch):
|
||||
raise TypeError("branch must be Branch")
|
||||
enforce_under_cwd_and_no_symlink(target_path, "target_path")
|
||||
|
||||
source = Path(branch.config_path)
|
||||
if not source.is_file():
|
||||
raise FileNotFoundError(
|
||||
f"branch config no longer exists: {source.name}"
|
||||
)
|
||||
|
||||
actual_sha = _hash_file(source)
|
||||
if actual_sha != branch.config_sha256:
|
||||
raise ValueError(
|
||||
f"branch config drifted from snapshot SHA "
|
||||
f"({actual_sha[:8]} vs {branch.config_sha256[:8]})"
|
||||
)
|
||||
|
||||
target = Path(target_path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp_path = tempfile.mkstemp(dir=str(target.parent), prefix=".tmp_")
|
||||
try:
|
||||
with os.fdopen(fd, "wb") as fh:
|
||||
with open(source, "rb") as src:
|
||||
while True:
|
||||
chunk = src.read(_HASH_CHUNK)
|
||||
if not chunk:
|
||||
break
|
||||
fh.write(chunk)
|
||||
os.replace(tmp_path, str(target))
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
return target
|
||||
|
|
@ -0,0 +1,337 @@
|
|||
"""Per-layer LoRA adapter diff math + report rendering (v0.57.0 Part A).
|
||||
|
||||
Pure numpy math (no torch); safetensors is loaded lazily so import is cheap.
|
||||
Public surface:
|
||||
|
||||
- ``compute_layer_diffs(weights_a, weights_b)`` -> per-layer Frobenius diffs
|
||||
- ``effective_rank(matrix)`` -> SVD-entropy effective rank
|
||||
- ``compute_adapter_diff(path_a, path_b, *, top_k=10)`` -> ``AdapterDiffReport``
|
||||
- ``render_report_markdown(report)`` / ``render_report_json(report)``
|
||||
|
||||
Containment + symlink rejection at every file load (TOCTOU defence,
|
||||
mirrors v0.53.1 ``enforce_under_cwd_and_no_symlink`` policy).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import stat
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Optional, Tuple
|
||||
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink, is_under_cwd
|
||||
|
||||
_MAX_LAYER_NAME_LEN = 256
|
||||
_MAX_LAYERS = 10_000
|
||||
_MAX_TOP_K = 200
|
||||
_MIN_TOP_K = 1
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LayerDiff:
|
||||
"""Frobenius-norm diff for a single LoRA parameter tensor."""
|
||||
|
||||
name: str
|
||||
frobenius: float
|
||||
norm_a: float
|
||||
norm_b: float
|
||||
relative: float # frobenius / max(norm_a, norm_b) or 0.0 if both zero
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AdapterDiffReport:
|
||||
adapter_a: str
|
||||
adapter_b: str
|
||||
per_layer: Tuple[LayerDiff, ...]
|
||||
top_changed: Tuple[str, ...]
|
||||
effective_rank_a: Optional[float]
|
||||
effective_rank_b: Optional[float]
|
||||
shared_layers: int
|
||||
only_in_a: Tuple[str, ...]
|
||||
only_in_b: Tuple[str, ...]
|
||||
|
||||
|
||||
def _require_str(value: object, field: str) -> str:
|
||||
if isinstance(value, bool) or not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str, got {type(value).__name__}")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > _MAX_LAYER_NAME_LEN:
|
||||
raise ValueError(f"{field} must be ≤{_MAX_LAYER_NAME_LEN} chars")
|
||||
return value
|
||||
|
||||
|
||||
def _frobenius(matrix: Any) -> float:
|
||||
import numpy as np
|
||||
|
||||
arr = np.asarray(matrix, dtype=np.float64)
|
||||
if arr.size == 0:
|
||||
return 0.0
|
||||
value = float(np.sqrt(np.sum(arr * arr)))
|
||||
if not math.isfinite(value):
|
||||
return float("inf")
|
||||
return value
|
||||
|
||||
|
||||
def effective_rank(matrix: Any, *, eps: float = 1e-12) -> float:
|
||||
"""Shannon entropy of normalised singular-value distribution (effective rank).
|
||||
|
||||
Returns ``exp(H)`` where H is the entropy of the SV distribution
|
||||
treated as a probability vector. Equals the matrix rank for an
|
||||
orthonormal basis and degrades smoothly as energy concentrates.
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
if isinstance(eps, bool) or not isinstance(eps, (int, float)):
|
||||
raise TypeError("eps must be float")
|
||||
if not math.isfinite(float(eps)) or float(eps) <= 0:
|
||||
raise ValueError("eps must be finite and positive")
|
||||
|
||||
arr = np.asarray(matrix, dtype=np.float64)
|
||||
if arr.ndim < 2:
|
||||
if arr.ndim == 1:
|
||||
arr = arr.reshape(-1, 1)
|
||||
else:
|
||||
return 0.0
|
||||
if arr.size == 0:
|
||||
return 0.0
|
||||
# Reshape >2D into 2D for SVD
|
||||
if arr.ndim > 2:
|
||||
arr = arr.reshape(arr.shape[0], -1)
|
||||
try:
|
||||
singular = np.linalg.svd(arr, compute_uv=False)
|
||||
except np.linalg.LinAlgError:
|
||||
return 0.0
|
||||
total = float(np.sum(singular))
|
||||
if total <= eps:
|
||||
return 0.0
|
||||
probs = singular / total
|
||||
probs = probs[probs > eps]
|
||||
if probs.size == 0:
|
||||
return 0.0
|
||||
entropy = float(-np.sum(probs * np.log(probs)))
|
||||
return float(math.exp(entropy))
|
||||
|
||||
|
||||
def compute_layer_diffs(
|
||||
weights_a: Mapping[str, Any],
|
||||
weights_b: Mapping[str, Any],
|
||||
) -> Tuple[Tuple[LayerDiff, ...], Tuple[str, ...], Tuple[str, ...]]:
|
||||
"""Compute per-layer Frobenius diffs for every name in both adapters.
|
||||
|
||||
Returns ``(per_layer, only_in_a, only_in_b)``. ``per_layer`` covers the
|
||||
intersection of names sorted alphabetically.
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
if not isinstance(weights_a, Mapping):
|
||||
raise TypeError("weights_a must be a mapping")
|
||||
if not isinstance(weights_b, Mapping):
|
||||
raise TypeError("weights_b must be a mapping")
|
||||
|
||||
names_a = set(weights_a.keys())
|
||||
names_b = set(weights_b.keys())
|
||||
if len(names_a) > _MAX_LAYERS or len(names_b) > _MAX_LAYERS:
|
||||
raise ValueError(f"adapter has >{_MAX_LAYERS} tensors")
|
||||
|
||||
shared = sorted(names_a & names_b)
|
||||
only_a = tuple(sorted(names_a - names_b))
|
||||
only_b = tuple(sorted(names_b - names_a))
|
||||
|
||||
diffs = []
|
||||
for name in shared:
|
||||
_require_str(name, "layer name")
|
||||
a = np.asarray(weights_a[name], dtype=np.float64)
|
||||
b = np.asarray(weights_b[name], dtype=np.float64)
|
||||
if a.shape != b.shape:
|
||||
# Skip shape-mismatched tensors (rank changed between adapters)
|
||||
continue
|
||||
diff = a - b
|
||||
fro = _frobenius(diff)
|
||||
norm_a = _frobenius(a)
|
||||
norm_b = _frobenius(b)
|
||||
denom = max(norm_a, norm_b)
|
||||
relative = fro / denom if denom > 0 else 0.0
|
||||
diffs.append(
|
||||
LayerDiff(
|
||||
name=name,
|
||||
frobenius=fro,
|
||||
norm_a=norm_a,
|
||||
norm_b=norm_b,
|
||||
relative=relative,
|
||||
)
|
||||
)
|
||||
return tuple(diffs), only_a, only_b
|
||||
|
||||
|
||||
def _validate_top_k(top_k: object) -> int:
|
||||
if isinstance(top_k, bool) or not isinstance(top_k, int):
|
||||
raise TypeError("top_k must be int")
|
||||
if top_k < _MIN_TOP_K or top_k > _MAX_TOP_K:
|
||||
raise ValueError(f"top_k must be in [{_MIN_TOP_K}, {_MAX_TOP_K}]")
|
||||
return top_k
|
||||
|
||||
|
||||
def _load_safetensors(path: Path) -> Mapping[str, Any]:
|
||||
"""Lazy-load adapter_model.safetensors via the ``safetensors`` package."""
|
||||
try:
|
||||
from safetensors import safe_open
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
"safetensors package required; pip install safetensors"
|
||||
) from exc
|
||||
result: dict[str, Any] = {}
|
||||
with safe_open(str(path), framework="numpy") as f:
|
||||
for key in f.keys():
|
||||
_require_str(key, "tensor name")
|
||||
result[key] = f.get_tensor(key)
|
||||
if len(result) > _MAX_LAYERS:
|
||||
raise ValueError(f"adapter has >{_MAX_LAYERS} tensors")
|
||||
return result
|
||||
|
||||
|
||||
def _adapter_weights_path(adapter_dir: Path) -> Path:
|
||||
"""Return the safetensors path inside an adapter dir, raising if missing.
|
||||
|
||||
Symlinks at the weights file are rejected via ``os.lstat + S_ISLNK``
|
||||
BEFORE ``is_file()`` so a crafted ``adapter_model.safetensors -> /etc/passwd``
|
||||
cannot escape the directory-level containment check (review fix HIGH).
|
||||
"""
|
||||
candidates = (
|
||||
adapter_dir / "adapter_model.safetensors",
|
||||
adapter_dir / "adapter_model.bin",
|
||||
)
|
||||
for cand in candidates:
|
||||
if not os.path.lexists(str(cand)):
|
||||
continue
|
||||
st = os.lstat(str(cand))
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(
|
||||
f"{adapter_dir.name}/{cand.name}: must not be a symlink"
|
||||
)
|
||||
if cand.is_file():
|
||||
if cand.suffix == ".bin":
|
||||
raise RuntimeError(
|
||||
f"{adapter_dir.name}: .bin format not supported; "
|
||||
"re-save adapter as safetensors"
|
||||
)
|
||||
return cand
|
||||
raise FileNotFoundError(
|
||||
f"{adapter_dir.name}: no adapter_model.safetensors found"
|
||||
)
|
||||
|
||||
|
||||
def load_adapter_weights(adapter_dir: str) -> Mapping[str, Any]:
|
||||
"""Containment-checked safetensors load.
|
||||
|
||||
Raises ``ValueError`` if the dir is outside cwd or a symlink; raises
|
||||
``FileNotFoundError`` if no adapter_model.safetensors is present.
|
||||
"""
|
||||
enforce_under_cwd_and_no_symlink(adapter_dir, "adapter")
|
||||
path = _adapter_weights_path(Path(adapter_dir))
|
||||
# Re-validate the weights file itself
|
||||
if not is_under_cwd(str(path)):
|
||||
raise ValueError(f"adapter weights must stay under cwd: {path.name}")
|
||||
return _load_safetensors(path)
|
||||
|
||||
|
||||
def _effective_rank_average(weights: Mapping[str, Any]) -> Optional[float]:
|
||||
"""Mean effective-rank across 2D LoRA matrices (None if no 2D tensors)."""
|
||||
import numpy as np
|
||||
|
||||
ranks: list[float] = []
|
||||
for tensor in weights.values():
|
||||
arr = np.asarray(tensor)
|
||||
if arr.ndim == 2 and min(arr.shape) > 0:
|
||||
ranks.append(effective_rank(arr))
|
||||
if not ranks:
|
||||
return None
|
||||
return float(sum(ranks) / len(ranks))
|
||||
|
||||
|
||||
def compute_adapter_diff(
|
||||
adapter_a: str,
|
||||
adapter_b: str,
|
||||
*,
|
||||
top_k: int = 10,
|
||||
) -> AdapterDiffReport:
|
||||
"""End-to-end: load both adapters, compute layer diffs, rank top-K."""
|
||||
_require_str(adapter_a, "adapter_a")
|
||||
_require_str(adapter_b, "adapter_b")
|
||||
_validate_top_k(top_k)
|
||||
|
||||
weights_a = load_adapter_weights(adapter_a)
|
||||
weights_b = load_adapter_weights(adapter_b)
|
||||
|
||||
per_layer, only_a, only_b = compute_layer_diffs(weights_a, weights_b)
|
||||
|
||||
sorted_by_change = sorted(per_layer, key=lambda d: d.frobenius, reverse=True)
|
||||
top = tuple(d.name for d in sorted_by_change[:top_k])
|
||||
|
||||
rank_a = _effective_rank_average(weights_a)
|
||||
rank_b = _effective_rank_average(weights_b)
|
||||
|
||||
return AdapterDiffReport(
|
||||
adapter_a=os.path.basename(os.path.normpath(adapter_a)),
|
||||
adapter_b=os.path.basename(os.path.normpath(adapter_b)),
|
||||
per_layer=per_layer,
|
||||
top_changed=top,
|
||||
effective_rank_a=rank_a,
|
||||
effective_rank_b=rank_b,
|
||||
shared_layers=len(per_layer),
|
||||
only_in_a=only_a,
|
||||
only_in_b=only_b,
|
||||
)
|
||||
|
||||
|
||||
def render_report_json(report: AdapterDiffReport) -> str:
|
||||
"""Serialise a report as canonical JSON for CI consumption."""
|
||||
if not isinstance(report, AdapterDiffReport):
|
||||
raise TypeError("report must be AdapterDiffReport")
|
||||
payload = {
|
||||
"adapter_a": report.adapter_a,
|
||||
"adapter_b": report.adapter_b,
|
||||
"shared_layers": report.shared_layers,
|
||||
"effective_rank_a": report.effective_rank_a,
|
||||
"effective_rank_b": report.effective_rank_b,
|
||||
"top_changed": list(report.top_changed),
|
||||
"only_in_a": list(report.only_in_a),
|
||||
"only_in_b": list(report.only_in_b),
|
||||
"per_layer": [asdict(d) for d in report.per_layer],
|
||||
}
|
||||
return json.dumps(payload, indent=2, sort_keys=True, allow_nan=False)
|
||||
|
||||
|
||||
def render_report_markdown(report: AdapterDiffReport) -> str:
|
||||
"""Human-readable markdown report (suitable for PR comments)."""
|
||||
if not isinstance(report, AdapterDiffReport):
|
||||
raise TypeError("report must be AdapterDiffReport")
|
||||
lines = [
|
||||
f"# Adapter diff: {report.adapter_a} vs {report.adapter_b}",
|
||||
"",
|
||||
f"- Shared layers: **{report.shared_layers}**",
|
||||
f"- Effective rank A: **{report.effective_rank_a}**",
|
||||
f"- Effective rank B: **{report.effective_rank_b}**",
|
||||
"",
|
||||
"## Top changed projections",
|
||||
"",
|
||||
]
|
||||
if not report.top_changed:
|
||||
lines.append("_no shared layers_")
|
||||
else:
|
||||
for name in report.top_changed:
|
||||
lines.append(f"- `{name}`")
|
||||
if report.only_in_a:
|
||||
lines.extend(["", "## Only in A", ""])
|
||||
for name in report.only_in_a:
|
||||
lines.append(f"- `{name}`")
|
||||
if report.only_in_b:
|
||||
lines.extend(["", "## Only in B", ""])
|
||||
for name in report.only_in_b:
|
||||
lines.append(f"- `{name}`")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
|
@ -0,0 +1,412 @@
|
|||
"""LoRA adapter merge strategies (v0.57.0 Part B).
|
||||
|
||||
Four strategies wrap mergekit-style math in pure numpy:
|
||||
|
||||
- ``linear``: weighted average per-layer (baseline)
|
||||
- ``ties``: trim-elect-disjoint average (Yadav et al. 2023)
|
||||
- ``dare``: drop and rescale by 1/(1-density) (Yu et al. 2024)
|
||||
- ``svd``: low-rank SVD reconstruction of the merged delta
|
||||
|
||||
All strategies operate on the intersection of layer names; shape-mismatched
|
||||
tensors are silently skipped (rank changed between adapters). Live canary
|
||||
verdict via v0.55 eval gate is deferred to v0.57.1; stub returns ``None``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import stat
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, FrozenSet, Literal, Mapping, Sequence, Tuple
|
||||
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
MergeStrategy = Literal["linear", "ties", "dare", "svd"]
|
||||
# frozenset for O(1) membership + immutability (mirrors v0.41.0 / v0.51.0 policy);
|
||||
# tuple alias preserved for caller code that iterates in canonical order.
|
||||
SUPPORTED_STRATEGIES: FrozenSet[str] = frozenset({"linear", "ties", "dare", "svd"})
|
||||
STRATEGY_ORDER: Tuple[MergeStrategy, ...] = ("linear", "ties", "dare", "svd")
|
||||
|
||||
_MAX_ADAPTERS = 16
|
||||
_MIN_ADAPTERS = 2
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MergeReport:
|
||||
strategy: str
|
||||
adapters: Tuple[str, ...]
|
||||
weights: Tuple[float, ...]
|
||||
merged_layers: int
|
||||
skipped_layers: Tuple[str, ...]
|
||||
output_dir: str
|
||||
verdict: str # "OK" / "MINOR" / "MAJOR" / "UNKNOWN" (v0.57.1 live)
|
||||
|
||||
|
||||
def _validate_weights(weights: Sequence[float], n: int) -> Tuple[float, ...]:
|
||||
if not isinstance(weights, Sequence) or isinstance(weights, str):
|
||||
raise TypeError("weights must be a sequence of floats")
|
||||
if len(weights) != n:
|
||||
raise ValueError(f"weights length {len(weights)} != number of adapters {n}")
|
||||
out: list[float] = []
|
||||
for w in weights:
|
||||
if isinstance(w, bool):
|
||||
raise TypeError("weights must not contain bool")
|
||||
if not isinstance(w, (int, float)):
|
||||
raise TypeError("weights must be numeric")
|
||||
wf = float(w)
|
||||
if not math.isfinite(wf):
|
||||
raise ValueError("weights must be finite")
|
||||
if wf < 0:
|
||||
raise ValueError("weights must be non-negative")
|
||||
out.append(wf)
|
||||
total = sum(out)
|
||||
if total <= 0:
|
||||
raise ValueError("weights must sum to a positive value")
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _validate_density(density: float, field: str) -> float:
|
||||
if isinstance(density, bool) or not isinstance(density, (int, float)):
|
||||
raise TypeError(f"{field} must be float")
|
||||
val = float(density)
|
||||
if not math.isfinite(val):
|
||||
raise ValueError(f"{field} must be finite")
|
||||
if not 0 < val <= 1:
|
||||
raise ValueError(f"{field} must be in (0, 1]")
|
||||
return val
|
||||
|
||||
|
||||
def merge_linear(
|
||||
weights_list: Sequence[Mapping[str, Any]],
|
||||
weights: Sequence[float],
|
||||
) -> Tuple[dict[str, Any], Tuple[str, ...]]:
|
||||
"""Weighted average per layer over the intersection of names."""
|
||||
import numpy as np
|
||||
|
||||
if len(weights_list) < _MIN_ADAPTERS:
|
||||
raise ValueError(f"need at least {_MIN_ADAPTERS} adapters")
|
||||
if len(weights_list) > _MAX_ADAPTERS:
|
||||
raise ValueError(f"at most {_MAX_ADAPTERS} adapters")
|
||||
w = _validate_weights(weights, len(weights_list))
|
||||
norm = sum(w)
|
||||
coeffs = [wi / norm for wi in w]
|
||||
|
||||
shared = set(weights_list[0].keys())
|
||||
for adapters in weights_list[1:]:
|
||||
shared &= set(adapters.keys())
|
||||
|
||||
merged: dict[str, Any] = {}
|
||||
skipped: list[str] = []
|
||||
for name in sorted(shared):
|
||||
tensors = [np.asarray(a[name], dtype=np.float64) for a in weights_list]
|
||||
shapes = {t.shape for t in tensors}
|
||||
if len(shapes) > 1:
|
||||
skipped.append(name)
|
||||
continue
|
||||
acc = np.zeros_like(tensors[0])
|
||||
for c, t in zip(coeffs, tensors):
|
||||
acc += c * t
|
||||
merged[name] = acc.astype(np.float32)
|
||||
return merged, tuple(skipped)
|
||||
|
||||
|
||||
def merge_ties(
|
||||
weights_list: Sequence[Mapping[str, Any]],
|
||||
weights: Sequence[float],
|
||||
*,
|
||||
density: float = 0.2,
|
||||
) -> Tuple[dict[str, Any], Tuple[str, ...]]:
|
||||
"""TIES merge: trim low magnitudes, elect majority sign, disjoint avg."""
|
||||
import numpy as np
|
||||
|
||||
density_val = _validate_density(density, "density")
|
||||
if len(weights_list) < _MIN_ADAPTERS:
|
||||
raise ValueError(f"need at least {_MIN_ADAPTERS} adapters")
|
||||
w = _validate_weights(weights, len(weights_list))
|
||||
|
||||
shared = set(weights_list[0].keys())
|
||||
for adapters in weights_list[1:]:
|
||||
shared &= set(adapters.keys())
|
||||
|
||||
merged: dict[str, Any] = {}
|
||||
skipped: list[str] = []
|
||||
for name in sorted(shared):
|
||||
tensors = [np.asarray(a[name], dtype=np.float64) for a in weights_list]
|
||||
shapes = {t.shape for t in tensors}
|
||||
if len(shapes) > 1:
|
||||
skipped.append(name)
|
||||
continue
|
||||
# Per-adapter trim: keep top density% by magnitude, zero out rest
|
||||
trimmed = []
|
||||
for t in tensors:
|
||||
flat = np.abs(t).flatten()
|
||||
if flat.size == 0:
|
||||
trimmed.append(t.copy())
|
||||
continue
|
||||
k = max(1, int(flat.size * density_val))
|
||||
threshold = np.partition(flat, -k)[-k] if k < flat.size else 0.0
|
||||
mask = np.abs(t) >= threshold
|
||||
trimmed.append(np.where(mask, t, 0.0))
|
||||
# Elect majority sign per element. On a tie (sign_sum == 0) default
|
||||
# to positive (TIES paper §3) so all-tied entries are not silently
|
||||
# zeroed — `np.sign(0) == 0` would set `elected_sign` to 0 and the
|
||||
# subsequent agree_mask would drop every parameter (review fix HIGH).
|
||||
stacked = np.stack(trimmed) # (n_adapters, ...)
|
||||
sign_sum = np.sign(stacked).sum(axis=0)
|
||||
elected_sign = np.where(sign_sum >= 0, 1.0, -1.0)
|
||||
# Disjoint average: only average elements that agree with elected sign
|
||||
agree_mask = np.sign(stacked) == elected_sign[None, ...]
|
||||
weighted = np.zeros_like(trimmed[0])
|
||||
weight_sum = np.zeros_like(trimmed[0])
|
||||
for wi, t, mask in zip(w, trimmed, agree_mask):
|
||||
weighted += wi * t * mask
|
||||
weight_sum += wi * mask
|
||||
out = np.where(weight_sum > 0, weighted / np.maximum(weight_sum, 1e-12), 0.0)
|
||||
merged[name] = out.astype(np.float32)
|
||||
return merged, tuple(skipped)
|
||||
|
||||
|
||||
def merge_dare(
|
||||
weights_list: Sequence[Mapping[str, Any]],
|
||||
weights: Sequence[float],
|
||||
*,
|
||||
density: float = 0.5,
|
||||
seed: int = 0,
|
||||
) -> Tuple[dict[str, Any], Tuple[str, ...]]:
|
||||
"""DARE merge: random drop with prob (1-density), rescale 1/density, average."""
|
||||
import numpy as np
|
||||
|
||||
density_val = _validate_density(density, "density")
|
||||
if isinstance(seed, bool) or not isinstance(seed, int):
|
||||
raise TypeError("seed must be int")
|
||||
if seed < 0:
|
||||
raise ValueError("seed must be non-negative")
|
||||
if len(weights_list) < _MIN_ADAPTERS:
|
||||
raise ValueError(f"need at least {_MIN_ADAPTERS} adapters")
|
||||
w = _validate_weights(weights, len(weights_list))
|
||||
norm = sum(w)
|
||||
coeffs = [wi / norm for wi in w]
|
||||
|
||||
shared = set(weights_list[0].keys())
|
||||
for adapters in weights_list[1:]:
|
||||
shared &= set(adapters.keys())
|
||||
|
||||
rng = np.random.default_rng(seed)
|
||||
merged: dict[str, Any] = {}
|
||||
skipped: list[str] = []
|
||||
for name in sorted(shared):
|
||||
tensors = [np.asarray(a[name], dtype=np.float64) for a in weights_list]
|
||||
shapes = {t.shape for t in tensors}
|
||||
if len(shapes) > 1:
|
||||
skipped.append(name)
|
||||
continue
|
||||
acc = np.zeros_like(tensors[0])
|
||||
for c, t in zip(coeffs, tensors):
|
||||
mask = rng.random(t.shape) < density_val
|
||||
rescaled = np.where(mask, t / density_val, 0.0)
|
||||
acc += c * rescaled
|
||||
merged[name] = acc.astype(np.float32)
|
||||
return merged, tuple(skipped)
|
||||
|
||||
|
||||
def merge_svd(
|
||||
weights_list: Sequence[Mapping[str, Any]],
|
||||
weights: Sequence[float],
|
||||
*,
|
||||
rank: int | None = None,
|
||||
) -> Tuple[dict[str, Any], Tuple[str, ...]]:
|
||||
"""SVD merge: linear-average, then low-rank reconstruct each 2D tensor.
|
||||
|
||||
Non-2D tensors are merged linearly without truncation.
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
if rank is not None:
|
||||
if isinstance(rank, bool) or not isinstance(rank, int):
|
||||
raise TypeError("rank must be int or None")
|
||||
if rank < 1:
|
||||
raise ValueError("rank must be ≥ 1")
|
||||
|
||||
linear_merged, skipped = merge_linear(weights_list, weights)
|
||||
if rank is None:
|
||||
return linear_merged, skipped
|
||||
|
||||
out: dict[str, Any] = {}
|
||||
for name, tensor in linear_merged.items():
|
||||
arr = np.asarray(tensor, dtype=np.float64)
|
||||
if arr.ndim != 2:
|
||||
out[name] = arr.astype(np.float32)
|
||||
continue
|
||||
try:
|
||||
u_mat, sv, vt_mat = np.linalg.svd(arr, full_matrices=False)
|
||||
except np.linalg.LinAlgError:
|
||||
out[name] = arr.astype(np.float32)
|
||||
continue
|
||||
r = min(rank, sv.shape[0])
|
||||
reconstructed = u_mat[:, :r] @ np.diag(sv[:r]) @ vt_mat[:r, :]
|
||||
out[name] = reconstructed.astype(np.float32)
|
||||
return out, skipped
|
||||
|
||||
|
||||
def merge_adapters(
|
||||
adapter_paths: Sequence[str],
|
||||
output_dir: str,
|
||||
*,
|
||||
strategy: MergeStrategy,
|
||||
weights: Sequence[float] | None = None,
|
||||
density: float = 0.2,
|
||||
seed: int = 0,
|
||||
rank: int | None = None,
|
||||
) -> MergeReport:
|
||||
"""End-to-end: load adapters, merge, write output as safetensors."""
|
||||
from soup_cli.utils.adapter_diff import load_adapter_weights
|
||||
|
||||
if strategy not in SUPPORTED_STRATEGIES:
|
||||
raise ValueError(
|
||||
f"strategy must be one of {SUPPORTED_STRATEGIES}, got {strategy!r}"
|
||||
)
|
||||
if not isinstance(adapter_paths, Sequence) or isinstance(adapter_paths, str):
|
||||
raise TypeError("adapter_paths must be a sequence of strings")
|
||||
if len(adapter_paths) < _MIN_ADAPTERS:
|
||||
raise ValueError(f"need at least {_MIN_ADAPTERS} adapters")
|
||||
if len(adapter_paths) > _MAX_ADAPTERS:
|
||||
raise ValueError(f"at most {_MAX_ADAPTERS} adapters")
|
||||
|
||||
enforce_under_cwd_and_no_symlink(output_dir, "output_dir")
|
||||
|
||||
weights_list = [load_adapter_weights(p) for p in adapter_paths]
|
||||
|
||||
if weights is None:
|
||||
eq = 1.0 / len(adapter_paths)
|
||||
weights = [eq] * len(adapter_paths)
|
||||
coeffs = _validate_weights(weights, len(adapter_paths))
|
||||
|
||||
if strategy == "linear":
|
||||
merged, skipped = merge_linear(weights_list, coeffs)
|
||||
elif strategy == "ties":
|
||||
merged, skipped = merge_ties(weights_list, coeffs, density=density)
|
||||
elif strategy == "dare":
|
||||
merged, skipped = merge_dare(weights_list, coeffs, density=density, seed=seed)
|
||||
else: # svd
|
||||
merged, skipped = merge_svd(weights_list, coeffs, rank=rank)
|
||||
|
||||
_write_merged_adapter(output_dir, adapter_paths[0], merged)
|
||||
|
||||
return MergeReport(
|
||||
strategy=strategy,
|
||||
adapters=tuple(adapter_paths),
|
||||
weights=coeffs,
|
||||
merged_layers=len(merged),
|
||||
skipped_layers=skipped,
|
||||
output_dir=output_dir,
|
||||
verdict="UNKNOWN", # v0.57.1: live canary eval via v0.55 gate
|
||||
)
|
||||
|
||||
|
||||
_MAX_ADAPTER_CONFIG_BYTES = 256 * 1024
|
||||
|
||||
|
||||
def _reject_if_symlink(path: Path, field: str) -> None:
|
||||
"""TOCTOU defence: lstat the raw path before opening."""
|
||||
if os.path.lexists(str(path)):
|
||||
st = os.lstat(str(path))
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(f"{field} must not be a symlink: {path.name}")
|
||||
|
||||
|
||||
def _atomic_write_bytes(target: Path, data: bytes) -> None:
|
||||
"""Atomic write via mkstemp + os.replace in target's parent dir."""
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
_reject_if_symlink(target, "output file")
|
||||
fd, tmp = tempfile.mkstemp(dir=str(target.parent), prefix=".tmp_")
|
||||
try:
|
||||
with os.fdopen(fd, "wb") as fh:
|
||||
fh.write(data)
|
||||
os.replace(tmp, str(target))
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def _write_merged_adapter(
|
||||
output_dir: str,
|
||||
template_source: str,
|
||||
weights: Mapping[str, Any],
|
||||
) -> None:
|
||||
"""Write adapter_model.safetensors + copy adapter_config.json from source.
|
||||
|
||||
Both writes are atomic (tempfile + os.replace) and reject pre-placed
|
||||
symlinks at the target path (TOCTOU defence; mirrors v0.53.1 policy).
|
||||
The source config is lstat-checked before reading and size-capped at
|
||||
256 KB per the v0.53.0 ``load_quant_config`` precedent.
|
||||
"""
|
||||
try:
|
||||
from safetensors.numpy import save_file
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
"safetensors package required; pip install safetensors"
|
||||
) from exc
|
||||
|
||||
out_path = Path(output_dir)
|
||||
out_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
safetensors_target = out_path / "adapter_model.safetensors"
|
||||
_reject_if_symlink(safetensors_target, "output safetensors")
|
||||
# safetensors.save_file writes atomically internally via mmap-rename on
|
||||
# POSIX, but on Windows it does not — write to sibling tmp + replace
|
||||
# ourselves so behaviour is consistent across platforms.
|
||||
fd, tmp_safe = tempfile.mkstemp(
|
||||
dir=str(out_path), prefix=".tmp_", suffix=".safetensors"
|
||||
)
|
||||
os.close(fd)
|
||||
try:
|
||||
save_file(weights, tmp_safe)
|
||||
os.replace(tmp_safe, str(safetensors_target))
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp_safe)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
# Copy adapter_config.json from first source so the merged adapter is loadable
|
||||
source_cfg = Path(template_source) / "adapter_config.json"
|
||||
if os.path.lexists(str(source_cfg)):
|
||||
st = os.lstat(str(source_cfg))
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(
|
||||
"source adapter_config.json must not be a symlink"
|
||||
)
|
||||
if st.st_size > _MAX_ADAPTER_CONFIG_BYTES:
|
||||
raise ValueError(
|
||||
f"source adapter_config.json > {_MAX_ADAPTER_CONFIG_BYTES} byte cap"
|
||||
)
|
||||
cfg = json.loads(source_cfg.read_text(encoding="utf-8"))
|
||||
target_cfg = out_path / "adapter_config.json"
|
||||
_atomic_write_bytes(
|
||||
target_cfg,
|
||||
json.dumps(cfg, indent=2).encode("utf-8"),
|
||||
)
|
||||
|
||||
|
||||
def predict_merged_verdict(
|
||||
report: MergeReport,
|
||||
canary_suite: str | None = None,
|
||||
) -> str:
|
||||
"""Stub for v0.57.1: live canary-eval verdict via v0.55 gate.
|
||||
|
||||
Today returns the existing report.verdict. The signature is forward-compatible
|
||||
so v0.57.1 can lift this to a real OK/MINOR/MAJOR classification.
|
||||
"""
|
||||
if not isinstance(report, MergeReport):
|
||||
raise TypeError("report must be MergeReport")
|
||||
if canary_suite is not None and not isinstance(canary_suite, str):
|
||||
raise TypeError("canary_suite must be str or None")
|
||||
return report.verdict
|
||||
|
|
@ -0,0 +1,200 @@
|
|||
"""LoRA adapter blame: attribute weight movement to dataset shards (v0.57.0 Part C).
|
||||
|
||||
This module emits a leave-one-out ablation PLAN that future v0.57.1 wiring
|
||||
will execute through the existing v0.34 SQLite tracker + v0.26 Registry
|
||||
lineage. The plan is fully deterministic given the inputs so callers can
|
||||
inspect / re-run / share without running real ablations.
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``parse_budget(spec)`` -> seconds (mirrors v0.48.0 ``parse_budget`` semantics)
|
||||
- ``plan_blame(adapter_dir, dataset_path, *, layer, budget_seconds, num_shards)``
|
||||
-> ``BlamePlan`` (frozen dataclass) with per-shard work item + projected duration
|
||||
- ``run_blame`` stub (raises ``NotImplementedError`` with v0.57.1 marker)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Tuple
|
||||
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
_MIN_BUDGET_SECONDS = 60
|
||||
_MAX_BUDGET_SECONDS = 24 * 3600
|
||||
_MIN_SHARDS = 2
|
||||
_MAX_SHARDS = 100
|
||||
_MIN_PER_SHARD_SECONDS = 30
|
||||
|
||||
_BUDGET_RE = re.compile(r"^(\d+)([smh]?)$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BlameShardWork:
|
||||
shard_id: int
|
||||
holdout_offset: int
|
||||
holdout_size: int
|
||||
projected_seconds: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BlamePlan:
|
||||
adapter_dir: str
|
||||
dataset_path: str
|
||||
layer: str
|
||||
budget_seconds: int
|
||||
num_shards: int
|
||||
per_shard_seconds: int
|
||||
shards: Tuple[BlameShardWork, ...]
|
||||
feasible: bool
|
||||
reason: str
|
||||
|
||||
|
||||
def parse_budget(spec: str) -> int:
|
||||
"""Parse a budget string (``60s`` / ``5m`` / ``2h`` / bare seconds)."""
|
||||
if isinstance(spec, bool) or not isinstance(spec, str):
|
||||
raise TypeError("spec must be str")
|
||||
if not spec:
|
||||
raise ValueError("spec must be non-empty")
|
||||
if "\x00" in spec:
|
||||
raise ValueError("spec must not contain null bytes")
|
||||
match = _BUDGET_RE.match(spec.strip())
|
||||
if not match:
|
||||
raise ValueError(f"invalid budget: {spec!r}")
|
||||
value = int(match.group(1))
|
||||
unit = match.group(2) or "s"
|
||||
multiplier = {"s": 1, "m": 60, "h": 3600}[unit]
|
||||
seconds = value * multiplier
|
||||
if seconds < _MIN_BUDGET_SECONDS:
|
||||
raise ValueError(
|
||||
f"budget {seconds}s below floor {_MIN_BUDGET_SECONDS}s"
|
||||
)
|
||||
if seconds > _MAX_BUDGET_SECONDS:
|
||||
raise ValueError(
|
||||
f"budget {seconds}s above cap {_MAX_BUDGET_SECONDS}s"
|
||||
)
|
||||
return seconds
|
||||
|
||||
|
||||
def _validate_int(value: object, field: str, lo: int, hi: int) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise TypeError(f"{field} must be int")
|
||||
if value < lo or value > hi:
|
||||
raise ValueError(f"{field} must be in [{lo}, {hi}]")
|
||||
return value
|
||||
|
||||
|
||||
def _validate_str(value: object, field: str, max_len: int = 512) -> str:
|
||||
if isinstance(value, bool) or not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > max_len:
|
||||
raise ValueError(f"{field} must be ≤{max_len} chars")
|
||||
return value
|
||||
|
||||
|
||||
def _count_dataset_rows(dataset_path: str) -> int:
|
||||
"""Cheap line count; for blame planning a rough order-of-magnitude is enough.
|
||||
|
||||
Opens via the realpath captured at containment check (see plan_blame)
|
||||
so a symlink swap between check and open cannot redirect the read
|
||||
(TOCTOU defence).
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
real = os.path.realpath(dataset_path)
|
||||
path = Path(real)
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(f"dataset not found: {path.name}")
|
||||
# Quick line count without holding the whole file in memory
|
||||
count = 0
|
||||
with open(real, "rb") as fh:
|
||||
for _ in fh:
|
||||
count += 1
|
||||
if count > 10_000_000: # DoS cap — 10M lines is plenty
|
||||
break
|
||||
return count
|
||||
|
||||
|
||||
def plan_blame(
|
||||
adapter_dir: str,
|
||||
dataset_path: str,
|
||||
*,
|
||||
layer: str,
|
||||
budget_seconds: int,
|
||||
num_shards: int = 10,
|
||||
) -> BlamePlan:
|
||||
"""Build a leave-one-out ablation plan.
|
||||
|
||||
The plan splits the dataset into ``num_shards`` equal shards; per shard,
|
||||
a 1/10-scale ablation run trains with that shard held out. ``feasible``
|
||||
is False when the budget cannot cover ≥ ``_MIN_PER_SHARD_SECONDS`` per
|
||||
shard with safety overhead.
|
||||
"""
|
||||
enforce_under_cwd_and_no_symlink(adapter_dir, "adapter_dir")
|
||||
enforce_under_cwd_and_no_symlink(dataset_path, "dataset_path")
|
||||
_validate_str(layer, "layer", max_len=256)
|
||||
_validate_int(budget_seconds, "budget_seconds",
|
||||
_MIN_BUDGET_SECONDS, _MAX_BUDGET_SECONDS)
|
||||
_validate_int(num_shards, "num_shards", _MIN_SHARDS, _MAX_SHARDS)
|
||||
|
||||
row_count = _count_dataset_rows(dataset_path)
|
||||
if row_count == 0:
|
||||
raise ValueError("dataset is empty")
|
||||
|
||||
# Reserve 10% overhead for setup/teardown across all shards
|
||||
usable_seconds = int(budget_seconds * 0.9)
|
||||
per_shard = usable_seconds // num_shards
|
||||
feasible = per_shard >= _MIN_PER_SHARD_SECONDS
|
||||
reason = (
|
||||
"ok"
|
||||
if feasible
|
||||
else f"budget {budget_seconds}s gives only {per_shard}s/shard "
|
||||
f"(need ≥{_MIN_PER_SHARD_SECONDS}s)"
|
||||
)
|
||||
|
||||
shard_size = math.ceil(row_count / num_shards)
|
||||
shards: list[BlameShardWork] = []
|
||||
for sid in range(num_shards):
|
||||
offset = sid * shard_size
|
||||
size = min(shard_size, max(0, row_count - offset))
|
||||
shards.append(
|
||||
BlameShardWork(
|
||||
shard_id=sid,
|
||||
holdout_offset=offset,
|
||||
holdout_size=size,
|
||||
projected_seconds=per_shard,
|
||||
)
|
||||
)
|
||||
|
||||
return BlamePlan(
|
||||
adapter_dir=adapter_dir,
|
||||
dataset_path=dataset_path,
|
||||
layer=layer,
|
||||
budget_seconds=budget_seconds,
|
||||
num_shards=num_shards,
|
||||
per_shard_seconds=per_shard,
|
||||
shards=tuple(shards),
|
||||
feasible=feasible,
|
||||
reason=reason,
|
||||
)
|
||||
|
||||
|
||||
def run_blame(plan: BlamePlan) -> None:
|
||||
"""Live leave-one-out ablation runner — deferred to v0.57.1.
|
||||
|
||||
The plan is executed through the v0.34 SQLite tracker + v0.26 Registry
|
||||
lineage so every ablation run is reproducible. Live wiring lands in v0.57.1.
|
||||
"""
|
||||
if not isinstance(plan, BlamePlan):
|
||||
raise TypeError("plan must be BlamePlan")
|
||||
raise NotImplementedError(
|
||||
"Live blame ablation runner deferred to v0.57.1. "
|
||||
"Use `soup adapters blame --plan-only` to inspect the plan."
|
||||
)
|
||||
|
|
@ -774,12 +774,15 @@ class TestSourceWiring:
|
|||
f"{relative} carries a top-level {forbidden!r} import"
|
||||
)
|
||||
|
||||
def test_version_is_0_56_0(self) -> None:
|
||||
assert __version__ == "0.56.0"
|
||||
def test_version_at_or_above_0_56_0(self) -> None:
|
||||
# v0.57.0 widened this from exact-match to floor-check (matches
|
||||
# the v0.51.0 / v0.54.0 floor-test idiom — every subsequent release
|
||||
# would otherwise have to edit this single line).
|
||||
assert __version__ >= "0.56.0"
|
||||
|
||||
def test_pyproject_version(self) -> None:
|
||||
text = (_PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8")
|
||||
assert 'version = "0.56.0"' in text
|
||||
assert 'version = "0.5' in text # floor check; current v0.5x.y
|
||||
|
||||
|
||||
# --- review-fix coverage --------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -0,0 +1,396 @@
|
|||
"""v0.57.0 Part A — adapters diff: math kernel + CLI smoke tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app as soup_app
|
||||
from soup_cli.utils.adapter_diff import (
|
||||
AdapterDiffReport,
|
||||
LayerDiff,
|
||||
compute_adapter_diff,
|
||||
compute_layer_diffs,
|
||||
effective_rank,
|
||||
render_report_json,
|
||||
render_report_markdown,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
# ---------- effective_rank ----------
|
||||
|
||||
|
||||
def test_effective_rank_identity_matrix():
|
||||
eye = np.eye(8)
|
||||
rank = effective_rank(eye)
|
||||
assert abs(rank - 8.0) < 1e-6
|
||||
|
||||
|
||||
def test_effective_rank_concentrated():
|
||||
# All energy in one direction → effective rank ≈ 1
|
||||
matrix = np.zeros((4, 4))
|
||||
matrix[0, 0] = 1.0
|
||||
assert effective_rank(matrix) == pytest.approx(1.0, abs=1e-6)
|
||||
|
||||
|
||||
def test_effective_rank_empty():
|
||||
assert effective_rank(np.zeros((0, 0))) == 0.0
|
||||
|
||||
|
||||
def test_effective_rank_1d_reshapes():
|
||||
assert effective_rank(np.ones(5)) >= 0.0
|
||||
|
||||
|
||||
def test_effective_rank_zero_matrix():
|
||||
assert effective_rank(np.zeros((3, 3))) == 0.0
|
||||
|
||||
|
||||
def test_effective_rank_bool_eps_rejected():
|
||||
with pytest.raises(TypeError):
|
||||
effective_rank(np.eye(2), eps=True)
|
||||
|
||||
|
||||
def test_effective_rank_non_finite_eps_rejected():
|
||||
with pytest.raises(ValueError):
|
||||
effective_rank(np.eye(2), eps=float("nan"))
|
||||
|
||||
|
||||
def test_effective_rank_negative_eps_rejected():
|
||||
with pytest.raises(ValueError):
|
||||
effective_rank(np.eye(2), eps=-1.0)
|
||||
|
||||
|
||||
# ---------- compute_layer_diffs ----------
|
||||
|
||||
|
||||
def test_compute_layer_diffs_identical_zero():
|
||||
a = {"foo": np.ones((2, 2))}
|
||||
b = {"foo": np.ones((2, 2))}
|
||||
diffs, only_a, only_b = compute_layer_diffs(a, b)
|
||||
assert len(diffs) == 1
|
||||
assert diffs[0].frobenius == 0.0
|
||||
assert diffs[0].relative == 0.0
|
||||
assert only_a == ()
|
||||
assert only_b == ()
|
||||
|
||||
|
||||
def test_compute_layer_diffs_known_norm():
|
||||
a = {"w": np.zeros((2, 2))}
|
||||
b = {"w": np.array([[3.0, 4.0], [0.0, 0.0]])}
|
||||
diffs, _, _ = compute_layer_diffs(a, b)
|
||||
# Frobenius = sqrt(9+16) = 5
|
||||
assert diffs[0].frobenius == pytest.approx(5.0)
|
||||
assert diffs[0].relative == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_compute_layer_diffs_skips_shape_mismatch():
|
||||
a = {"w": np.zeros((4, 4))}
|
||||
b = {"w": np.zeros((2, 2))}
|
||||
diffs, _, _ = compute_layer_diffs(a, b)
|
||||
assert diffs == () # shape mismatch silently skipped
|
||||
|
||||
|
||||
def test_compute_layer_diffs_partitions():
|
||||
a = {"only_a": np.zeros(2), "shared": np.zeros(2)}
|
||||
b = {"only_b": np.zeros(2), "shared": np.zeros(2)}
|
||||
diffs, only_a, only_b = compute_layer_diffs(a, b)
|
||||
assert [d.name for d in diffs] == ["shared"]
|
||||
assert only_a == ("only_a",)
|
||||
assert only_b == ("only_b",)
|
||||
|
||||
|
||||
def test_compute_layer_diffs_rejects_non_mapping():
|
||||
with pytest.raises(TypeError):
|
||||
compute_layer_diffs([], {}) # type: ignore[arg-type]
|
||||
with pytest.raises(TypeError):
|
||||
compute_layer_diffs({}, []) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_compute_layer_diffs_too_many_tensors():
|
||||
big = {f"t{i}": np.zeros(1) for i in range(10_001)}
|
||||
small: dict[str, np.ndarray] = {}
|
||||
with pytest.raises(ValueError, match=">10000"):
|
||||
compute_layer_diffs(big, small)
|
||||
|
||||
|
||||
# ---------- render report ----------
|
||||
|
||||
|
||||
def _sample_report() -> AdapterDiffReport:
|
||||
return AdapterDiffReport(
|
||||
adapter_a="A",
|
||||
adapter_b="B",
|
||||
per_layer=(
|
||||
LayerDiff(name="layer1", frobenius=2.0, norm_a=4.0, norm_b=3.0, relative=0.5),
|
||||
),
|
||||
top_changed=("layer1",),
|
||||
effective_rank_a=8.0,
|
||||
effective_rank_b=8.0,
|
||||
shared_layers=1,
|
||||
only_in_a=(),
|
||||
only_in_b=(),
|
||||
)
|
||||
|
||||
|
||||
def test_render_report_json_roundtrip():
|
||||
report = _sample_report()
|
||||
text = render_report_json(report)
|
||||
parsed = json.loads(text)
|
||||
assert parsed["adapter_a"] == "A"
|
||||
assert parsed["adapter_b"] == "B"
|
||||
assert parsed["shared_layers"] == 1
|
||||
assert parsed["top_changed"] == ["layer1"]
|
||||
assert parsed["per_layer"][0]["frobenius"] == 2.0
|
||||
|
||||
|
||||
def test_render_report_json_rejects_non_report():
|
||||
with pytest.raises(TypeError):
|
||||
render_report_json({"foo": "bar"}) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_render_report_markdown_renders():
|
||||
text = render_report_markdown(_sample_report())
|
||||
assert "# Adapter diff: A vs B" in text
|
||||
assert "layer1" in text
|
||||
assert text.endswith("\n")
|
||||
|
||||
|
||||
def test_render_report_markdown_only_lists():
|
||||
report = AdapterDiffReport(
|
||||
adapter_a="x", adapter_b="y",
|
||||
per_layer=(), top_changed=(),
|
||||
effective_rank_a=None, effective_rank_b=None,
|
||||
shared_layers=0,
|
||||
only_in_a=("foo",),
|
||||
only_in_b=("bar",),
|
||||
)
|
||||
text = render_report_markdown(report)
|
||||
assert "Only in A" in text
|
||||
assert "Only in B" in text
|
||||
|
||||
|
||||
def test_render_report_markdown_rejects_non_report():
|
||||
with pytest.raises(TypeError):
|
||||
render_report_markdown(None) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_frozen_dataclasses():
|
||||
import dataclasses
|
||||
diff = LayerDiff(name="x", frobenius=1.0, norm_a=1.0, norm_b=1.0, relative=1.0)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
diff.frobenius = 2.0 # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------- compute_adapter_diff (end-to-end with safetensors fixture) ----------
|
||||
|
||||
|
||||
def _write_safetensors(dir_path: Path, weights: dict) -> None:
|
||||
"""Helper: write adapter_model.safetensors + minimal adapter_config.json."""
|
||||
pytest.importorskip("safetensors")
|
||||
from safetensors.numpy import save_file
|
||||
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
save_file(weights, str(dir_path / "adapter_model.safetensors"))
|
||||
(dir_path / "adapter_config.json").write_text(
|
||||
json.dumps({"peft_type": "LORA", "r": 8}), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def test_compute_adapter_diff_end_to_end(tmp_path, monkeypatch):
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
weights_a = {
|
||||
"base_model.model.layers.0.self_attn.q_proj.lora_A.weight":
|
||||
np.ones((4, 8), dtype=np.float32),
|
||||
"base_model.model.layers.0.self_attn.v_proj.lora_A.weight":
|
||||
np.zeros((4, 8), dtype=np.float32),
|
||||
}
|
||||
weights_b = {
|
||||
"base_model.model.layers.0.self_attn.q_proj.lora_A.weight":
|
||||
np.ones((4, 8), dtype=np.float32), # identical
|
||||
"base_model.model.layers.0.self_attn.v_proj.lora_A.weight":
|
||||
np.ones((4, 8), dtype=np.float32), # different
|
||||
}
|
||||
_write_safetensors(tmp_path / "a", weights_a)
|
||||
_write_safetensors(tmp_path / "b", weights_b)
|
||||
|
||||
report = compute_adapter_diff("a", "b", top_k=5)
|
||||
assert isinstance(report, AdapterDiffReport)
|
||||
assert report.shared_layers == 2
|
||||
# v_proj should be the top-changed projection
|
||||
assert report.top_changed[0].endswith("v_proj.lora_A.weight")
|
||||
|
||||
|
||||
def test_compute_adapter_diff_outside_cwd_rejected(tmp_path):
|
||||
with pytest.raises(ValueError):
|
||||
compute_adapter_diff(str(tmp_path), "b")
|
||||
|
||||
|
||||
def test_compute_adapter_diff_bool_top_k_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
(tmp_path / "b").mkdir()
|
||||
with pytest.raises(TypeError):
|
||||
compute_adapter_diff("a", "b", top_k=True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_compute_adapter_diff_top_k_out_of_range(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
(tmp_path / "b").mkdir()
|
||||
with pytest.raises(ValueError):
|
||||
compute_adapter_diff("a", "b", top_k=0)
|
||||
with pytest.raises(ValueError):
|
||||
compute_adapter_diff("a", "b", top_k=201)
|
||||
|
||||
|
||||
def test_compute_adapter_diff_top_k_lower_bound(tmp_path, monkeypatch):
|
||||
"""top_k=1 should be accepted (exact lower bound)."""
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
report = compute_adapter_diff("a", "b", top_k=1)
|
||||
assert len(report.top_changed) == 1
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink semantics")
|
||||
def test_compute_adapter_diff_rejects_symlinked_weights(tmp_path, monkeypatch):
|
||||
"""Symlink at adapter_model.safetensors must be rejected (TOCTOU)."""
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_safetensors(tmp_path / "real", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
(tmp_path / "fake").mkdir()
|
||||
os.symlink(
|
||||
str(tmp_path / "real" / "adapter_model.safetensors"),
|
||||
str(tmp_path / "fake" / "adapter_model.safetensors"),
|
||||
)
|
||||
(tmp_path / "fake" / "adapter_config.json").write_text(
|
||||
json.dumps({"peft_type": "LORA"}), encoding="utf-8"
|
||||
)
|
||||
_write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
compute_adapter_diff("fake", "b")
|
||||
|
||||
|
||||
def test_no_top_level_torch_import():
|
||||
"""Lazy-import policy: adapter_diff must not import torch at module level."""
|
||||
src = (Path(__file__).parent.parent / "soup_cli" / "utils" / "adapter_diff.py"
|
||||
).read_text(encoding="utf-8")
|
||||
# Only allowed inside def bodies
|
||||
for line in src.splitlines():
|
||||
stripped = line.lstrip()
|
||||
if stripped.startswith("import torch") or stripped.startswith("from torch"):
|
||||
indent = len(line) - len(stripped)
|
||||
assert indent > 0, f"top-level torch import found: {line}"
|
||||
|
||||
|
||||
def test_compute_adapter_diff_missing_safetensors(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
(tmp_path / "b").mkdir()
|
||||
with pytest.raises(FileNotFoundError):
|
||||
compute_adapter_diff("a", "b")
|
||||
|
||||
|
||||
def test_compute_adapter_diff_bin_format_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
(tmp_path / "a" / "adapter_model.bin").write_bytes(b"x")
|
||||
(tmp_path / "b").mkdir()
|
||||
(tmp_path / "b" / "adapter_model.bin").write_bytes(b"x")
|
||||
with pytest.raises(RuntimeError, match=".bin format not supported"):
|
||||
compute_adapter_diff("a", "b")
|
||||
|
||||
|
||||
# ---------- CLI ----------
|
||||
|
||||
|
||||
def test_adapters_diff_help():
|
||||
result = runner.invoke(soup_app, ["adapters", "diff", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "--top-k" in result.output
|
||||
assert "--format" in result.output
|
||||
|
||||
|
||||
def test_adapters_diff_table_format(tmp_path, monkeypatch):
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, ["adapters", "diff", "a", "b"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "Adapter diff" in result.output
|
||||
|
||||
|
||||
def test_adapters_diff_json_output(tmp_path, monkeypatch):
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "diff", "a", "b",
|
||||
"--format", "json",
|
||||
"--output", "report.json",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
parsed = json.loads((tmp_path / "report.json").read_text())
|
||||
assert parsed["shared_layers"] == 1
|
||||
|
||||
|
||||
def test_adapters_diff_markdown_output(tmp_path, monkeypatch):
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "diff", "a", "b",
|
||||
"--format", "markdown",
|
||||
"--output", "report.md",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
text = (tmp_path / "report.md").read_text()
|
||||
assert "# Adapter diff" in text
|
||||
|
||||
|
||||
def test_adapters_diff_unknown_format(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
(tmp_path / "b").mkdir()
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "diff", "a", "b", "--format", "yaml",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "Unknown --format" in result.output
|
||||
|
||||
|
||||
def test_adapters_diff_output_requires_non_table(tmp_path, monkeypatch):
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "diff", "a", "b", "--output", "out.txt",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "requires --format" in result.output
|
||||
|
||||
|
||||
def test_adapters_diff_output_outside_cwd_rejected(tmp_path, monkeypatch):
|
||||
pytest.importorskip("safetensors")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_safetensors(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_safetensors(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
outside = os.path.join(os.path.dirname(str(tmp_path)), "outside.json")
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "diff", "a", "b",
|
||||
"--format", "json", "--output", outside,
|
||||
])
|
||||
assert result.exit_code != 0
|
||||
|
|
@ -0,0 +1,418 @@
|
|||
"""v0.57.0 Part B — adapters merge: linear / ties / dare / svd."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app as soup_app
|
||||
from soup_cli.utils.adapter_merge import (
|
||||
SUPPORTED_STRATEGIES,
|
||||
MergeReport,
|
||||
merge_adapters,
|
||||
merge_dare,
|
||||
merge_linear,
|
||||
merge_svd,
|
||||
merge_ties,
|
||||
predict_merged_verdict,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
# ---------- merge_linear ----------
|
||||
|
||||
|
||||
def test_merge_linear_average_of_zero_and_one():
|
||||
a = {"w": np.zeros((2, 2), dtype=np.float32)}
|
||||
b = {"w": np.ones((2, 2), dtype=np.float32)}
|
||||
merged, skipped = merge_linear([a, b], [1.0, 1.0])
|
||||
assert np.allclose(merged["w"], 0.5)
|
||||
assert skipped == ()
|
||||
|
||||
|
||||
def test_merge_linear_weighted():
|
||||
a = {"w": np.zeros((2,), dtype=np.float32)}
|
||||
b = {"w": np.ones((2,), dtype=np.float32)}
|
||||
merged, _ = merge_linear([a, b], [3.0, 1.0])
|
||||
assert np.allclose(merged["w"], 0.25)
|
||||
|
||||
|
||||
def test_merge_linear_intersection_only():
|
||||
a = {"shared": np.ones((2,), dtype=np.float32), "only_a": np.ones((2,))}
|
||||
b = {"shared": np.zeros((2,), dtype=np.float32), "only_b": np.ones((2,))}
|
||||
merged, _ = merge_linear([a, b], [1.0, 1.0])
|
||||
assert set(merged.keys()) == {"shared"}
|
||||
|
||||
|
||||
def test_merge_linear_shape_mismatch_skipped():
|
||||
a = {"w": np.zeros((4,), dtype=np.float32)}
|
||||
b = {"w": np.zeros((2,), dtype=np.float32)}
|
||||
merged, skipped = merge_linear([a, b], [1.0, 1.0])
|
||||
assert merged == {}
|
||||
assert skipped == ("w",)
|
||||
|
||||
|
||||
def test_merge_linear_rejects_single_adapter():
|
||||
with pytest.raises(ValueError, match="at least 2"):
|
||||
merge_linear([{"w": np.zeros(1)}], [1.0])
|
||||
|
||||
|
||||
def test_merge_linear_rejects_too_many():
|
||||
with pytest.raises(ValueError, match="at most 16"):
|
||||
merge_linear([{}] * 17, [1.0] * 17)
|
||||
|
||||
|
||||
def test_merge_linear_bool_weight_rejected():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(TypeError):
|
||||
merge_linear([a, b], [True, 1.0]) # type: ignore[list-item]
|
||||
|
||||
|
||||
def test_merge_linear_negative_weight_rejected():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(ValueError):
|
||||
merge_linear([a, b], [-1.0, 1.0])
|
||||
|
||||
|
||||
def test_merge_linear_nan_weight_rejected():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(ValueError):
|
||||
merge_linear([a, b], [float("nan"), 1.0])
|
||||
|
||||
|
||||
def test_merge_linear_zero_sum_rejected():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(ValueError, match="positive"):
|
||||
merge_linear([a, b], [0.0, 0.0])
|
||||
|
||||
|
||||
def test_merge_linear_wrong_weights_length():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(ValueError, match="length"):
|
||||
merge_linear([a, b], [1.0])
|
||||
|
||||
|
||||
# ---------- merge_ties ----------
|
||||
|
||||
|
||||
def test_merge_ties_density_keeps_top():
|
||||
# Top-half of [1, 2, 3, 4]: keep 3, 4
|
||||
a = {"w": np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float32)}
|
||||
b = {"w": np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float32)}
|
||||
merged, _ = merge_ties([a, b], [1.0, 1.0], density=0.5)
|
||||
# First two slots should be trimmed to zero
|
||||
assert merged["w"][0] == 0.0
|
||||
assert merged["w"][3] != 0.0
|
||||
|
||||
|
||||
def test_merge_ties_majority_sign_election():
|
||||
# Two adapters agree positive, one disagrees → elected sign is positive
|
||||
a = {"w": np.array([1.0], dtype=np.float32)}
|
||||
b = {"w": np.array([2.0], dtype=np.float32)}
|
||||
c = {"w": np.array([-3.0], dtype=np.float32)}
|
||||
merged, _ = merge_ties([a, b, c], [1.0, 1.0, 1.0], density=1.0)
|
||||
# Elected sign positive; negative entry dropped
|
||||
assert merged["w"][0] > 0
|
||||
|
||||
|
||||
def test_merge_ties_invalid_density():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(ValueError):
|
||||
merge_ties([a, b], [1.0, 1.0], density=0.0)
|
||||
with pytest.raises(ValueError):
|
||||
merge_ties([a, b], [1.0, 1.0], density=1.5)
|
||||
|
||||
|
||||
def test_merge_ties_bool_density_rejected():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(TypeError):
|
||||
merge_ties([a, b], [1.0, 1.0], density=True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------- merge_dare ----------
|
||||
|
||||
|
||||
def test_merge_dare_deterministic_with_seed():
|
||||
a = {"w": np.ones((10,), dtype=np.float32)}
|
||||
b = {"w": np.ones((10,), dtype=np.float32)}
|
||||
m1, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=42)
|
||||
m2, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=42)
|
||||
assert np.allclose(m1["w"], m2["w"])
|
||||
|
||||
|
||||
def test_merge_dare_different_seeds_diverge():
|
||||
a = {"w": np.ones((100,), dtype=np.float32)}
|
||||
b = {"w": np.ones((100,), dtype=np.float32)}
|
||||
m1, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=1)
|
||||
m2, _ = merge_dare([a, b], [1.0, 1.0], density=0.5, seed=2)
|
||||
assert not np.allclose(m1["w"], m2["w"])
|
||||
|
||||
|
||||
def test_merge_dare_bool_seed_rejected():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(TypeError):
|
||||
merge_dare([a, b], [1.0, 1.0], seed=True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_merge_dare_negative_seed_rejected():
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(ValueError):
|
||||
merge_dare([a, b], [1.0, 1.0], seed=-1)
|
||||
|
||||
|
||||
def test_merge_dare_density_1_equals_linear():
|
||||
a = {"w": np.array([2.0, 4.0], dtype=np.float32)}
|
||||
b = {"w": np.array([6.0, 8.0], dtype=np.float32)}
|
||||
merged, _ = merge_dare([a, b], [1.0, 1.0], density=1.0, seed=0)
|
||||
# density=1 → no drop, no rescale → identical to linear average
|
||||
assert np.allclose(merged["w"], [4.0, 6.0])
|
||||
|
||||
|
||||
# ---------- merge_svd ----------
|
||||
|
||||
|
||||
def test_merge_svd_no_rank_equals_linear():
|
||||
a = {"w": np.eye(4, dtype=np.float32)}
|
||||
b = {"w": np.eye(4, dtype=np.float32)}
|
||||
merged, _ = merge_svd([a, b], [1.0, 1.0])
|
||||
assert np.allclose(merged["w"], np.eye(4))
|
||||
|
||||
|
||||
def test_merge_svd_with_rank_reduces_rank():
|
||||
# Random matrix → low-rank reconstruction should be lower rank
|
||||
rng = np.random.default_rng(0)
|
||||
a = {"w": rng.standard_normal((8, 8)).astype(np.float32)}
|
||||
b = {"w": rng.standard_normal((8, 8)).astype(np.float32)}
|
||||
merged, _ = merge_svd([a, b], [1.0, 1.0], rank=2)
|
||||
actual_rank = np.linalg.matrix_rank(merged["w"], tol=1e-5)
|
||||
assert actual_rank <= 2
|
||||
|
||||
|
||||
def test_merge_svd_non_2d_passthrough():
|
||||
a = {"bias": np.ones((4,), dtype=np.float32)}
|
||||
b = {"bias": np.ones((4,), dtype=np.float32)}
|
||||
merged, _ = merge_svd([a, b], [1.0, 1.0], rank=1)
|
||||
assert np.allclose(merged["bias"], 1.0)
|
||||
|
||||
|
||||
def test_merge_svd_rank_clamp():
|
||||
# Rank > min dimension → clamped
|
||||
a = {"w": np.eye(4, dtype=np.float32)}
|
||||
b = {"w": np.eye(4, dtype=np.float32)}
|
||||
merged, _ = merge_svd([a, b], [1.0, 1.0], rank=100)
|
||||
assert merged["w"].shape == (4, 4)
|
||||
|
||||
|
||||
def test_merge_svd_invalid_rank():
|
||||
a = {"w": np.eye(2, dtype=np.float32)}
|
||||
b = {"w": np.eye(2, dtype=np.float32)}
|
||||
with pytest.raises(ValueError):
|
||||
merge_svd([a, b], [1.0, 1.0], rank=0)
|
||||
with pytest.raises(TypeError):
|
||||
merge_svd([a, b], [1.0, 1.0], rank=True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------- merge_adapters end-to-end + CLI ----------
|
||||
|
||||
|
||||
def _write_adapter(dir_path: Path, weights: dict) -> None:
|
||||
pytest.importorskip("safetensors")
|
||||
from safetensors.numpy import save_file
|
||||
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
save_file(weights, str(dir_path / "adapter_model.safetensors"))
|
||||
(dir_path / "adapter_config.json").write_text(
|
||||
json.dumps({"peft_type": "LORA", "r": 8}), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def test_merge_adapters_e2e_linear(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_adapter(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
report = merge_adapters(["a", "b"], "out", strategy="linear")
|
||||
assert isinstance(report, MergeReport)
|
||||
assert report.strategy == "linear"
|
||||
assert report.merged_layers == 1
|
||||
assert (tmp_path / "out" / "adapter_model.safetensors").exists()
|
||||
assert (tmp_path / "out" / "adapter_config.json").exists()
|
||||
|
||||
|
||||
def test_merge_adapters_unknown_strategy(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
(tmp_path / "b").mkdir()
|
||||
with pytest.raises(ValueError, match="strategy must be"):
|
||||
merge_adapters(["a", "b"], "out", strategy="bogus") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_merge_adapters_output_outside_cwd(tmp_path, monkeypatch):
|
||||
import os
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
_write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
outside = os.path.join(os.path.dirname(str(tmp_path)), "outside")
|
||||
with pytest.raises(ValueError):
|
||||
merge_adapters(["a", "b"], outside, strategy="linear")
|
||||
|
||||
|
||||
def test_supported_strategies_immutable():
|
||||
# v0.57.0 review fix: SUPPORTED_STRATEGIES is a frozenset (matches v0.41.0+
|
||||
# allowlist policy). STRATEGY_ORDER preserves canonical iteration order.
|
||||
from soup_cli.utils.adapter_merge import STRATEGY_ORDER
|
||||
assert SUPPORTED_STRATEGIES == frozenset({"linear", "ties", "dare", "svd"})
|
||||
assert isinstance(SUPPORTED_STRATEGIES, frozenset)
|
||||
assert STRATEGY_ORDER == ("linear", "ties", "dare", "svd")
|
||||
|
||||
|
||||
def test_merge_ties_density_one_keeps_everything():
|
||||
"""density=1.0 is the inclusive upper bound — must not raise."""
|
||||
import numpy as np # noqa: F811
|
||||
a = {"w": np.ones((4,), dtype=np.float32)}
|
||||
b = {"w": np.ones((4,), dtype=np.float32)}
|
||||
merged, _ = merge_ties([a, b], [1.0, 1.0], density=1.0)
|
||||
assert "w" in merged
|
||||
|
||||
|
||||
def test_merge_linear_inf_weight_rejected():
|
||||
"""math.isfinite must reject +inf as well as NaN."""
|
||||
import numpy as np # noqa: F811
|
||||
a = {"w": np.zeros(1, dtype=np.float32)}
|
||||
b = {"w": np.zeros(1, dtype=np.float32)}
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
merge_linear([a, b], [float("inf"), 1.0])
|
||||
|
||||
|
||||
def test_merge_ties_tied_sign_defaults_positive():
|
||||
"""Sign-sum == 0 (tied vote) must elect +1, not silently zero parameters."""
|
||||
import numpy as np # noqa: F811
|
||||
a = {"w": np.array([2.0], dtype=np.float32)}
|
||||
b = {"w": np.array([-2.0], dtype=np.float32)}
|
||||
merged, _ = merge_ties([a, b], [1.0, 1.0], density=1.0)
|
||||
# Tied sign → elected +1 → positive entry kept, negative dropped → result 2.0
|
||||
assert merged["w"][0] > 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(__import__("os").name == "nt",
|
||||
reason="POSIX-only symlink semantics")
|
||||
def test_merge_adapters_rejects_symlink_at_output_safetensors(tmp_path, monkeypatch):
|
||||
"""Pre-placed symlink at output safetensors path must be rejected (TOCTOU)."""
|
||||
import os
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_adapter(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
out = tmp_path / "out"
|
||||
out.mkdir()
|
||||
target = tmp_path / "evil.bin"
|
||||
target.write_bytes(b"x")
|
||||
os.symlink(str(target), str(out / "adapter_model.safetensors"))
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
merge_adapters(["a", "b"], "out", strategy="linear")
|
||||
# Symlink target untouched
|
||||
assert target.read_bytes() == b"x"
|
||||
|
||||
|
||||
def test_no_top_level_torch_import_in_merge():
|
||||
src = (Path(__file__).parent.parent / "soup_cli" / "utils" / "adapter_merge.py"
|
||||
).read_text(encoding="utf-8")
|
||||
for line in src.splitlines():
|
||||
stripped = line.lstrip()
|
||||
if stripped.startswith("import torch") or stripped.startswith("from torch"):
|
||||
indent = len(line) - len(stripped)
|
||||
assert indent > 0, f"top-level torch import: {line}"
|
||||
|
||||
|
||||
def test_predict_merged_verdict_stub():
|
||||
report = MergeReport(
|
||||
strategy="linear", adapters=("a", "b"), weights=(0.5, 0.5),
|
||||
merged_layers=1, skipped_layers=(), output_dir="out", verdict="UNKNOWN",
|
||||
)
|
||||
assert predict_merged_verdict(report) == "UNKNOWN"
|
||||
|
||||
|
||||
def test_predict_merged_verdict_rejects_non_report():
|
||||
with pytest.raises(TypeError):
|
||||
predict_merged_verdict("not a report") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_predict_merged_verdict_canary_must_be_str():
|
||||
report = MergeReport(
|
||||
strategy="linear", adapters=("a", "b"), weights=(0.5, 0.5),
|
||||
merged_layers=1, skipped_layers=(), output_dir="out", verdict="OK",
|
||||
)
|
||||
with pytest.raises(TypeError):
|
||||
predict_merged_verdict(report, canary_suite=123) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_merge_report_frozen():
|
||||
import dataclasses
|
||||
report = MergeReport(
|
||||
strategy="linear", adapters=("a", "b"), weights=(0.5, 0.5),
|
||||
merged_layers=1, skipped_layers=(), output_dir="out", verdict="OK",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
report.strategy = "ties" # type: ignore[misc]
|
||||
|
||||
|
||||
def test_adapters_merge_cli_help():
|
||||
result = runner.invoke(soup_app, ["adapters", "merge", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "--strategy" in result.output
|
||||
assert "--weights" in result.output
|
||||
|
||||
|
||||
def test_adapters_merge_cli_linear(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_adapter(tmp_path / "a", {"w": np.ones((2, 2), dtype=np.float32)})
|
||||
_write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "merge", "a", "b", "-o", "out", "--strategy", "linear",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert (tmp_path / "out" / "adapter_model.safetensors").exists()
|
||||
|
||||
|
||||
def test_adapters_merge_cli_unknown_strategy(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
_write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "merge", "a", "b", "-o", "out", "--strategy", "bogus",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "Unknown --strategy" in result.output
|
||||
|
||||
|
||||
def test_adapters_merge_cli_invalid_weights(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
_write_adapter(tmp_path / "b", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "merge", "a", "b", "-o", "out",
|
||||
"--strategy", "linear", "--weights", "1.0,abc",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
|
||||
|
||||
def test_adapters_merge_cli_single_adapter_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_write_adapter(tmp_path / "a", {"w": np.zeros((2, 2), dtype=np.float32)})
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "merge", "a", "-o", "out", "--strategy", "linear",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "at least 2" in result.output
|
||||
|
|
@ -0,0 +1,317 @@
|
|||
"""v0.57.0 Part C — adapters blame: plan emitter + budget check."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app as soup_app
|
||||
from soup_cli.utils.blame import (
|
||||
BlamePlan,
|
||||
BlameShardWork,
|
||||
parse_budget,
|
||||
plan_blame,
|
||||
run_blame,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
# ---------- parse_budget ----------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("spec,expected", [
|
||||
("60", 60),
|
||||
("60s", 60),
|
||||
("5m", 300),
|
||||
("2h", 7200),
|
||||
("1h", 3600),
|
||||
("90s", 90),
|
||||
])
|
||||
def test_parse_budget_happy(spec, expected):
|
||||
assert parse_budget(spec) == expected
|
||||
|
||||
|
||||
def test_parse_budget_below_min():
|
||||
with pytest.raises(ValueError, match="below floor"):
|
||||
parse_budget("30")
|
||||
|
||||
|
||||
def test_parse_budget_above_cap():
|
||||
with pytest.raises(ValueError, match="above cap"):
|
||||
parse_budget("25h")
|
||||
|
||||
|
||||
def test_parse_budget_invalid_format():
|
||||
with pytest.raises(ValueError, match="invalid"):
|
||||
parse_budget("abc")
|
||||
|
||||
|
||||
def test_parse_budget_empty():
|
||||
with pytest.raises(ValueError):
|
||||
parse_budget("")
|
||||
|
||||
|
||||
def test_parse_budget_bool():
|
||||
with pytest.raises(TypeError):
|
||||
parse_budget(True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_parse_budget_null_byte():
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
parse_budget("60\x00s")
|
||||
|
||||
|
||||
def test_parse_budget_non_string():
|
||||
with pytest.raises(TypeError):
|
||||
parse_budget(60) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------- plan_blame ----------
|
||||
|
||||
|
||||
def _setup_blame(tmp_path, monkeypatch, n_rows: int = 100):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "adapter").mkdir()
|
||||
dataset = tmp_path / "data.jsonl"
|
||||
dataset.write_text(
|
||||
"\n".join(f'{{"text": "row{i}"}}' for i in range(n_rows)),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return "adapter", "data.jsonl"
|
||||
|
||||
|
||||
def test_plan_blame_happy(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch, n_rows=100)
|
||||
plan = plan_blame(
|
||||
adapter, dataset,
|
||||
layer="q_proj.7", budget_seconds=3600, num_shards=10,
|
||||
)
|
||||
assert isinstance(plan, BlamePlan)
|
||||
assert len(plan.shards) == 10
|
||||
assert plan.layer == "q_proj.7"
|
||||
assert plan.feasible is True
|
||||
# Each shard covers ~10 rows
|
||||
assert plan.shards[0].holdout_size == 10
|
||||
|
||||
|
||||
def test_plan_blame_infeasible_budget(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
plan = plan_blame(
|
||||
adapter, dataset,
|
||||
layer="q_proj.7", budget_seconds=60, num_shards=100,
|
||||
)
|
||||
# 60s / 100 shards = 0s/shard → infeasible
|
||||
assert plan.feasible is False
|
||||
assert "need" in plan.reason
|
||||
|
||||
|
||||
def test_plan_blame_shard_offsets(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch, n_rows=50)
|
||||
plan = plan_blame(
|
||||
adapter, dataset,
|
||||
layer="layer", budget_seconds=3600, num_shards=5,
|
||||
)
|
||||
offsets = [s.holdout_offset for s in plan.shards]
|
||||
assert offsets == [0, 10, 20, 30, 40]
|
||||
|
||||
|
||||
def test_plan_blame_uneven_split(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch, n_rows=23)
|
||||
plan = plan_blame(
|
||||
adapter, dataset,
|
||||
layer="layer", budget_seconds=3600, num_shards=5,
|
||||
)
|
||||
# ceil(23 / 5) = 5 → last shard gets 3 rows
|
||||
sizes = [s.holdout_size for s in plan.shards]
|
||||
assert sum(sizes) == 23
|
||||
|
||||
|
||||
def test_plan_blame_empty_dataset(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "adapter").mkdir()
|
||||
(tmp_path / "empty.jsonl").write_text("", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="empty"):
|
||||
plan_blame(
|
||||
"adapter", "empty.jsonl",
|
||||
layer="x", budget_seconds=3600, num_shards=5,
|
||||
)
|
||||
|
||||
|
||||
def test_plan_blame_outside_cwd_adapter(tmp_path):
|
||||
with pytest.raises(ValueError):
|
||||
plan_blame(
|
||||
str(tmp_path), "data.jsonl",
|
||||
layer="x", budget_seconds=3600, num_shards=5,
|
||||
)
|
||||
|
||||
|
||||
def test_plan_blame_outside_cwd_dataset(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "adapter").mkdir()
|
||||
outside = os.path.join(os.path.dirname(str(tmp_path)), "data.jsonl")
|
||||
with pytest.raises(ValueError):
|
||||
plan_blame(
|
||||
"adapter", outside,
|
||||
layer="x", budget_seconds=3600, num_shards=5,
|
||||
)
|
||||
|
||||
|
||||
def test_plan_blame_invalid_layer(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
plan_blame(
|
||||
adapter, dataset,
|
||||
layer="", budget_seconds=3600, num_shards=5,
|
||||
)
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
plan_blame(
|
||||
adapter, dataset,
|
||||
layer="x\x00y", budget_seconds=3600, num_shards=5,
|
||||
)
|
||||
|
||||
|
||||
def test_plan_blame_invalid_shards(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
with pytest.raises(ValueError, match="num_shards"):
|
||||
plan_blame(
|
||||
adapter, dataset,
|
||||
layer="x", budget_seconds=3600, num_shards=1,
|
||||
)
|
||||
with pytest.raises(ValueError, match="num_shards"):
|
||||
plan_blame(
|
||||
adapter, dataset,
|
||||
layer="x", budget_seconds=3600, num_shards=200,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bool_value", [True, False])
|
||||
def test_plan_blame_bool_shards(tmp_path, monkeypatch, bool_value):
|
||||
"""bool subclasses int; both True (→1) and False (→0) must be rejected."""
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
with pytest.raises(TypeError):
|
||||
plan_blame(
|
||||
adapter, dataset,
|
||||
layer="x", budget_seconds=3600,
|
||||
num_shards=bool_value, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
def test_plan_blame_bool_budget_rejected(tmp_path, monkeypatch):
|
||||
"""budget_seconds=True must not silently coerce to 1 (project bool policy)."""
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
with pytest.raises(TypeError):
|
||||
plan_blame(
|
||||
adapter, dataset,
|
||||
layer="x",
|
||||
budget_seconds=True, # type: ignore[arg-type]
|
||||
num_shards=5,
|
||||
)
|
||||
|
||||
|
||||
def test_plan_blame_missing_dataset(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "adapter").mkdir()
|
||||
# File doesn't exist BUT also doesn't exist in cwd containment check
|
||||
# The containment helper rejects non-existent first. Try with a real
|
||||
# outside-cwd file to confirm flow.
|
||||
(tmp_path / "missing.jsonl").write_text("", encoding="utf-8")
|
||||
with pytest.raises(ValueError):
|
||||
# Empty file → empty dataset rejection
|
||||
plan_blame(
|
||||
"adapter", "missing.jsonl",
|
||||
layer="x", budget_seconds=3600, num_shards=2,
|
||||
)
|
||||
|
||||
|
||||
# ---------- run_blame stub ----------
|
||||
|
||||
|
||||
def test_run_blame_stub_marker():
|
||||
plan = BlamePlan(
|
||||
adapter_dir="a", dataset_path="d", layer="x",
|
||||
budget_seconds=3600, num_shards=2, per_shard_seconds=600,
|
||||
shards=(BlameShardWork(0, 0, 1, 600), BlameShardWork(1, 1, 1, 600)),
|
||||
feasible=True, reason="ok",
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.57.1"):
|
||||
run_blame(plan)
|
||||
|
||||
|
||||
def test_run_blame_rejects_non_plan():
|
||||
with pytest.raises(TypeError):
|
||||
run_blame("not a plan") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------- Frozen invariants ----------
|
||||
|
||||
|
||||
def test_blame_plan_frozen():
|
||||
import dataclasses
|
||||
plan = BlamePlan(
|
||||
adapter_dir="a", dataset_path="d", layer="x",
|
||||
budget_seconds=3600, num_shards=1, per_shard_seconds=3600,
|
||||
shards=(), feasible=False, reason="r",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
plan.layer = "y" # type: ignore[misc]
|
||||
|
||||
|
||||
def test_blame_shard_work_frozen():
|
||||
import dataclasses
|
||||
shard = BlameShardWork(0, 0, 10, 60)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
shard.shard_id = 1 # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------- CLI ----------
|
||||
|
||||
|
||||
def test_adapters_blame_help():
|
||||
result = runner.invoke(soup_app, ["adapters", "blame", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "--budget" in result.output
|
||||
assert "--layer" in result.output
|
||||
assert "--shards" in result.output
|
||||
|
||||
|
||||
def test_adapters_blame_plan_only(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "blame", adapter,
|
||||
"--dataset", dataset,
|
||||
"--layer", "q_proj.7",
|
||||
"--budget", "1h",
|
||||
"--shards", "5",
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "Blame plan" in result.output
|
||||
|
||||
|
||||
def test_adapters_blame_invalid_budget(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "blame", adapter,
|
||||
"--dataset", dataset,
|
||||
"--layer", "x",
|
||||
"--budget", "abc",
|
||||
"--shards", "5",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "Invalid --budget" in result.output
|
||||
|
||||
|
||||
def test_adapters_blame_live_runner_advisory(tmp_path, monkeypatch):
|
||||
adapter, dataset = _setup_blame(tmp_path, monkeypatch)
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "blame", adapter,
|
||||
"--dataset", dataset,
|
||||
"--layer", "x",
|
||||
"--budget", "1h",
|
||||
"--shards", "5",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "v0.57.1" in result.output
|
||||
|
|
@ -0,0 +1,389 @@
|
|||
"""v0.57.0 Part D — adapters branch / checkout / branches list."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app as soup_app
|
||||
from soup_cli.utils.adapter_branch import (
|
||||
Branch,
|
||||
create_branch,
|
||||
delete_branch,
|
||||
list_branches,
|
||||
load_branch,
|
||||
write_checkout,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def isolated_branches_dir(tmp_path, monkeypatch):
|
||||
"""Redirect SOUP_BRANCHES_DIR to tmp so tests don't pollute ~/.soup."""
|
||||
branches = tmp_path / "branches"
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(branches))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
return branches
|
||||
|
||||
|
||||
def _make_config(tmp_path: Path, name: str = "soup.yaml", content: str = "base: test\n") -> Path:
|
||||
p = tmp_path / name
|
||||
p.write_text(content, encoding="utf-8")
|
||||
return p
|
||||
|
||||
|
||||
# ---------- create_branch ----------
|
||||
|
||||
|
||||
def test_create_branch_happy(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="meta/llama")
|
||||
assert isinstance(snap, Branch)
|
||||
assert snap.name == "v1"
|
||||
assert snap.base_model == "meta/llama"
|
||||
assert len(snap.config_sha256) == 64
|
||||
assert snap.dataset_sha256 is None
|
||||
|
||||
|
||||
def test_create_branch_with_dataset(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
(tmp_path / "data.jsonl").write_text('{"x": 1}\n', encoding="utf-8")
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="b",
|
||||
dataset_path="data.jsonl")
|
||||
assert snap.dataset_sha256 is not None
|
||||
assert len(snap.dataset_sha256) == 64
|
||||
|
||||
|
||||
def test_create_branch_invalid_name(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
with pytest.raises(ValueError, match="must match"):
|
||||
create_branch("../etc", config_path="soup.yaml", base_model="b")
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
create_branch("", config_path="soup.yaml", base_model="b")
|
||||
|
||||
|
||||
def test_create_branch_null_byte_name(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
create_branch("v\x001", config_path="soup.yaml", base_model="b")
|
||||
|
||||
|
||||
def test_create_branch_bool_name(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
with pytest.raises(TypeError):
|
||||
create_branch(True, config_path="soup.yaml", base_model="b") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_create_branch_config_outside_cwd(tmp_path):
|
||||
outside = os.path.join(os.path.dirname(str(tmp_path)), "x.yaml")
|
||||
with pytest.raises(ValueError):
|
||||
create_branch("v1", config_path=outside, base_model="b")
|
||||
|
||||
|
||||
def test_create_branch_missing_config(tmp_path):
|
||||
with pytest.raises(FileNotFoundError):
|
||||
create_branch("v1", config_path="missing.yaml", base_model="b")
|
||||
|
||||
|
||||
def test_create_branch_empty_base_model(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
create_branch("v1", config_path="soup.yaml", base_model="")
|
||||
|
||||
|
||||
def test_create_branch_null_byte_base_model(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
create_branch("v1", config_path="soup.yaml", base_model="x\x00")
|
||||
|
||||
|
||||
def test_create_branch_bool_base_model_rejected(tmp_path):
|
||||
"""bool subclasses str-checks in some idioms; ensure explicit TypeError."""
|
||||
_make_config(tmp_path)
|
||||
with pytest.raises(TypeError):
|
||||
create_branch("v1", config_path="soup.yaml",
|
||||
base_model=True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_delete_branch_traversal_rejected():
|
||||
with pytest.raises(ValueError):
|
||||
delete_branch("../etc/passwd")
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink semantics")
|
||||
def test_load_branch_rejects_symlink(tmp_path, monkeypatch):
|
||||
branches = tmp_path / "br"
|
||||
branches.mkdir()
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(branches))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = tmp_path / "real.json"
|
||||
target.write_text("{}", encoding="utf-8")
|
||||
os.symlink(str(target), str(branches / "evil.json"))
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
load_branch("evil")
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink semantics")
|
||||
def test_delete_branch_rejects_symlink(tmp_path, monkeypatch):
|
||||
branches = tmp_path / "br"
|
||||
branches.mkdir()
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(branches))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = tmp_path / "real.json"
|
||||
target.write_text("{}", encoding="utf-8")
|
||||
os.symlink(str(target), str(branches / "evil.json"))
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
delete_branch("evil")
|
||||
# original file untouched
|
||||
assert target.exists()
|
||||
|
||||
|
||||
def test_branches_dir_crlf_env_rejected(monkeypatch, tmp_path):
|
||||
"""Control char (CRLF) in SOUP_BRANCHES_DIR falls back to default."""
|
||||
from soup_cli.utils.adapter_branch import _branches_dir
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", "/some\npath")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
resolved = _branches_dir()
|
||||
assert "\n" not in str(resolved)
|
||||
|
||||
|
||||
def test_create_branch_oversize_config(tmp_path):
|
||||
# 1 MiB + 1 byte
|
||||
(tmp_path / "soup.yaml").write_bytes(b"x" * (1_048_577))
|
||||
with pytest.raises(ValueError, match="cap"):
|
||||
create_branch("v1", config_path="soup.yaml", base_model="b")
|
||||
|
||||
|
||||
def test_create_branch_atomic_write(tmp_path, isolated_branches_dir):
|
||||
_make_config(tmp_path)
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="b")
|
||||
target = isolated_branches_dir / "v1.json"
|
||||
assert target.exists()
|
||||
raw = json.loads(target.read_text(encoding="utf-8"))
|
||||
assert raw["name"] == "v1"
|
||||
assert raw["config_sha256"] == snap.config_sha256
|
||||
|
||||
|
||||
# ---------- list_branches ----------
|
||||
|
||||
|
||||
def test_list_branches_empty(tmp_path):
|
||||
assert list_branches() == ()
|
||||
|
||||
|
||||
def test_list_branches_sorted(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
create_branch("b", config_path="soup.yaml", base_model="x")
|
||||
create_branch("a", config_path="soup.yaml", base_model="x")
|
||||
create_branch("c", config_path="soup.yaml", base_model="x")
|
||||
assert list_branches() == ("a", "b", "c")
|
||||
|
||||
|
||||
# ---------- load_branch ----------
|
||||
|
||||
|
||||
def test_load_branch_roundtrip(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
created = create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
loaded = load_branch("v1")
|
||||
assert loaded.name == created.name
|
||||
assert loaded.config_sha256 == created.config_sha256
|
||||
assert loaded.base_model == created.base_model
|
||||
|
||||
|
||||
def test_load_branch_missing(tmp_path):
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_branch("nope")
|
||||
|
||||
|
||||
def test_load_branch_invalid_name():
|
||||
with pytest.raises(ValueError):
|
||||
load_branch("../etc")
|
||||
|
||||
|
||||
# ---------- delete_branch ----------
|
||||
|
||||
|
||||
def test_delete_branch_true_when_present(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
assert delete_branch("v1") is True
|
||||
assert "v1" not in list_branches()
|
||||
|
||||
|
||||
def test_delete_branch_false_when_missing():
|
||||
assert delete_branch("nope") is False
|
||||
|
||||
|
||||
# ---------- write_checkout ----------
|
||||
|
||||
|
||||
def test_write_checkout_writes_target(tmp_path):
|
||||
_make_config(tmp_path, content="base: original\n")
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
written = write_checkout(snap, "restored.yaml")
|
||||
assert written.exists()
|
||||
assert "original" in written.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_write_checkout_detects_drift(tmp_path):
|
||||
_make_config(tmp_path, content="base: original\n")
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
# Mutate original config — SHA mismatch
|
||||
(tmp_path / "soup.yaml").write_text("base: drifted\n", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="drifted"):
|
||||
write_checkout(snap, "restored.yaml")
|
||||
|
||||
|
||||
def test_write_checkout_outside_cwd_rejected(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
outside = os.path.join(os.path.dirname(str(tmp_path)), "x.yaml")
|
||||
with pytest.raises(ValueError):
|
||||
write_checkout(snap, outside)
|
||||
|
||||
|
||||
def test_write_checkout_rejects_non_branch():
|
||||
with pytest.raises(TypeError):
|
||||
write_checkout("not a branch", "out.yaml") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_write_checkout_missing_source(tmp_path):
|
||||
_make_config(tmp_path)
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
(tmp_path / "soup.yaml").unlink()
|
||||
with pytest.raises(FileNotFoundError):
|
||||
write_checkout(snap, "restored.yaml")
|
||||
|
||||
|
||||
# ---------- Frozen Branch ----------
|
||||
|
||||
|
||||
def test_branch_frozen(tmp_path):
|
||||
import dataclasses
|
||||
_make_config(tmp_path)
|
||||
snap = create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
snap.name = "v2" # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------- Env override containment ----------
|
||||
|
||||
|
||||
def test_branches_dir_env_override_outside_falls_back(monkeypatch, tmp_path):
|
||||
"""Out-of-bounds env override is rejected; helper returns the in-bounds default.
|
||||
|
||||
We point SOUP_BRANCHES_DIR at a tmp-rooted path that IS in-bounds (under
|
||||
$TMPDIR per the containment policy) so we can also assert that valid
|
||||
overrides ARE honoured — i.e. we test both directions of the policy here.
|
||||
"""
|
||||
from soup_cli.utils.adapter_branch import _branches_dir
|
||||
in_bounds = tmp_path / "valid-override"
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(in_bounds))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
resolved = _branches_dir()
|
||||
assert os.path.realpath(str(resolved)) == os.path.realpath(str(in_bounds))
|
||||
|
||||
|
||||
def test_branches_dir_null_byte_env_ignored(monkeypatch, tmp_path):
|
||||
"""Null-byte env override is silently ignored; default ~/.soup/branches used."""
|
||||
from soup_cli.utils.adapter_branch import _branches_dir
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", "/some\x00path")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
resolved = _branches_dir()
|
||||
# Falls back to ~/.soup/branches — must NOT contain a null byte
|
||||
assert "\x00" not in str(resolved)
|
||||
assert ".soup" in str(resolved) and "branches" in str(resolved)
|
||||
|
||||
|
||||
# ---------- CLI smoke ----------
|
||||
|
||||
|
||||
def test_branch_cli_help():
|
||||
result = runner.invoke(soup_app, ["adapters", "branch", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
|
||||
|
||||
def test_checkout_cli_help():
|
||||
result = runner.invoke(soup_app, ["adapters", "checkout", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
|
||||
|
||||
def test_branches_cli_help():
|
||||
result = runner.invoke(soup_app, ["adapters", "branches", "--help"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
|
||||
|
||||
def test_branch_cli_create_and_list(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br"))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_make_config(tmp_path)
|
||||
r1 = runner.invoke(soup_app, [
|
||||
"adapters", "branch", "v1",
|
||||
"-c", "soup.yaml", "--base", "meta/llama",
|
||||
])
|
||||
assert r1.exit_code == 0, (r1.output, repr(r1.exception))
|
||||
assert "v1" in r1.output
|
||||
|
||||
r2 = runner.invoke(soup_app, ["adapters", "branches"])
|
||||
assert r2.exit_code == 0, (r2.output, repr(r2.exception))
|
||||
assert "v1" in r2.output
|
||||
|
||||
|
||||
def test_branch_cli_invalid_name(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br"))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_make_config(tmp_path)
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "branch", "../etc",
|
||||
"-c", "soup.yaml", "--base", "m",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "must match" in result.output
|
||||
|
||||
|
||||
def test_branch_cli_missing_config(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br"))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "branch", "v1",
|
||||
"-c", "missing.yaml", "--base", "m",
|
||||
])
|
||||
assert result.exit_code == 1
|
||||
assert "not found" in result.output
|
||||
|
||||
|
||||
def test_checkout_cli_roundtrip(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br"))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_make_config(tmp_path, content="base: x\nepochs: 1\n")
|
||||
create_branch("v1", config_path="soup.yaml", base_model="m")
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "checkout", "v1",
|
||||
"-o", "restored.yaml",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert (tmp_path / "restored.yaml").exists()
|
||||
|
||||
|
||||
def test_checkout_cli_missing_branch(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br"))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
result = runner.invoke(soup_app, [
|
||||
"adapters", "checkout", "nope", "-o", "out.yaml",
|
||||
])
|
||||
assert result.exit_code == 1
|
||||
assert "not found" in result.output
|
||||
|
||||
|
||||
def test_branches_cli_empty(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("SOUP_BRANCHES_DIR", str(tmp_path / "br"))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
result = runner.invoke(soup_app, ["adapters", "branches"])
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert "No branches" in result.output
|
||||
Loading…
Reference in New Issue