diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 15c4af0..05c235a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -96,7 +96,7 @@ soup_cli/ commands/ - Command implementations (train, chat, eval, deploy, etc.) config/ - Config schema (schema.py) and loader (loader.py) data/ - Data loading, format conversion, providers, templates - trainer/ - Training wrappers (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, Pretrain, Reward Model, Embedding) + trainer/ - Training wrappers (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, Pretrain, Reward Model, Embedding, BCO, Preference, Distill, Classifier, PRM, Unlearn) monitoring/ - Callbacks and live dashboard experiment/ - SQLite experiment tracking eval/ - Eval platform (custom tasks, LLM judge, human eval, leaderboard) @@ -107,11 +107,11 @@ soup_cli/ cans/ - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0) data/traces/ - Trace-to-Preference harvester (v0.26.0) data/collators.py - CrossDocCollator for sample packing (v0.33.0) - utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch + utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (207 files, 9446 tests) +tests/ - Test suite (212 files, 9571 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index cd7bbbb..3a845db 100644 --- a/README.md +++ b/README.md @@ -42,15 +42,14 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.60.0 — Supply Chain Security: tensors you can ship through procurement.** Six controls that hosted vendors structurally can't provide — adapter weight-space backdoor scanner, Merkle-root sign + verify, strict-safetensors mode (refuse pickle weights), trust-on-first-use namespace pinning (anti-AI-Jacking), closed license-compatibility matrix at merge, and a signed airgap-bundle for data-diode transfer. Every piece is opt-in for operators who want enforcement; CI pipelines and security-conscious enterprises get a "you can't ship an unaudited adapter through Soup" guarantee. +**v0.61.0 — Unlearning & Knowledge Edit: GDPR right-to-be-forgotten + surgical fact patches.** Two of the most under-served axes in fine-tuning land as first-class trainers: unlearning (NPO / SimNPO / RMU — the legal-liability axis upstream TRL avoids) and knowledge editing (ROME / MEMIT / AlphaEdit — research-coded everywhere, productized nowhere). Hospital data teams handling a GDPR deletion request, labs patching factual errors without retraining, and security teams responding to a CSAM/PII leak all get a CLI workflow that composes with the existing Registry + eval-gate + diagnose surface. Schema-only release; live trainer + kernel wiring lands in v0.61.1. -- **`soup adapters scan `** — spectral analysis of LoRA weights pre-load. Flags rank-1 dominance (the canonical weight-space trojan pattern), top-1 singular-vector energy concentration, NaN/Inf in weights, and Frobenius-norm outliers via robust median+MAD bucketing. Pure numpy (no torch). Exit codes 0=OK / 1=WARN / 3=FAIL for CI grep. Reuses v0.57.0 `adapter_diff` loader so on-disk surface stays single-source. -- **`soup adapters sign / verify [--strict]`** — deterministic Merkle-root manifest over every file in the adapter dir (including nested `tokenizer/` / `processor/` subdirs). Tamper any file, verify fails. `UNSIGNED` backend ships live for offline tamper detection; `sigstore` + `ed25519` backends raise `NotImplementedError` with v0.60.1 marker (schema lives now so CI can integrate). Signature stored as `.soup-signature.json` written atomically via the shared `atomic_write_text` helper. -- **`soup adapters check-safetensors [--strict]`** — refuses pickle / PyTorch-classic weights at the boundary. Closed 8-entry unsafe-extension allowlist (`.bin` / `.pt` / `.pth` / `.ckpt` / `.pkl` / `.pickle` / `.joblib` / `.msgpack`). Exit code 3 (distinct from generic errors) under `--strict` so CI can gate on pickle-only. -- **Namespace-pin TOFU (anti-AI-Jacking).** New `NamespacePinStore` SQLite cache + `verify_namespace` helper: records `(repo_id, author, created_at)` on first download, refuses updates when the author changes OR `created_at` jumps backward (case-insensitive comparison via `datetime.fromisoformat` — fragile lexicographic compare avoided). `--allow-namespace-shift ` explicit opt-in (bool rejected to prevent free-for-all bypass). DB path containment-checked to `$HOME / $CWD / $TMPDIR`; POSIX `0o600` perms; pre-placed symlinks rejected via `os.lstat + S_ISLNK`. Live wiring into `utils/hubs.download_repo` lands in v0.60.1. -- **License-conflict matrix at merge.** Closed allowlist of 33 SPDX-ish licenses (Apache / MIT / BSD / LGPL / MPL / GPL / AGPL / CC-BY / CC-BY-NC / Llama-2/3.x/community / Gemma / Qwen-research / Mistral-research / OpenRAIL / OpenAI-ToS / Anthropic-AUP). `MappingProxyType` category compatibility table. `soup adapters merge --license --license-override ` — non-commercial + permissive refuses (8-char min, 4096-char max reason); strong-copyleft + permissive refuses; restricted-use + permissive refuses. Override reason captured in panel; audit-log integration v0.60.1. -- **`soup airgap-bundle --model --output `** — signed tarball with model + datasets + wheels + kernels + SHA-256 per file in embedded `manifest.json`. Default 100 GiB size cap; refuses oversize. Deterministic dataset labeling by sorted basename (NOT argv order) so the same inputs in different argv order produce identical manifests. TOCTOU lstat re-check on parent + output path BEFORE mkstemp; atomic `os.replace` via sibling tempfile; `tarfile.data_filter` set for any future `extractall` caller. 64 MiB manifest cap on `inspect_airgap_bundle`. -- **+152 new tests** (9294 → 9446). Review-fix coverage across five waves: 0 CRITICAL + 12 HIGH + 11 MEDIUM + 6 LOW resolved before tag. Manual CPU smokes for sign / verify / scan / check-safetensors / airgap-bundle / merge-with-license-conflict all confirmed end-to-end. +- **`task='unlearn'` + NPO / SimNPO / RMU.** New first-class task on `SoupConfig`. `training.unlearn_method: Literal["npo","simnpo","rmu"]` selects the backend; `data.forget_set` (rows to unlearn) + `data.retain_set` (capability-preserving rows) are validated at schema-load. Case-insensitive method normalisation; mlx backend rejected with distinct error; missing forget_set on `task='unlearn'` rejected with named field. `UnlearnTrainerWrapper` stub captures the method + accepts forward-compat kwargs so v0.61.1 wiring is purely additive. +- **`soup eval unlearning --benchmark tofu|muse|wmdp`** — three orthogonal axes scored on the project's OK / MINOR / MAJOR taxonomy: Forget Quality (pre/post forget-loss linear ramp), Model Utility (retain-accuracy ratio), PrivLeak (membership-inference AUC symmetric around 0.5). Bundled TOFU mini-fixture under `soup_cli/data/_fixtures/unlearning/`; MUSE + WMDP loaders ship in v0.61.1. Evidence-driven: operators supply pre-computed JSON today and get a frozen `UnlearnReport` with overall worst-case verdict + atomic JSON output. +- **`soup edit set --base --method rome|memit|alphaedit --subject "Paris is the capital of France" --target "Lyon"`** — surgical locate-and-edit. Closed method allowlist; per-method default MLP layer (ROME=5, MEMIT=8, AlphaEdit=5); operator override via `--layer`. `--plan-only` validates + prints the resolved `EditPlan` and exits 0 today; without it the CLI surfaces the deferred-live `NotImplementedError` with v0.61.1 marker at exit code 3 (distinct from validation rejection exit 2). +- **Sequential edit governor + norm-blowup detection.** New `EditGovernor` tracks per-base-model edit count + last verdict; auto-switches ROME → AlphaEdit at the configurable edit-count threshold (default 10) AND on detected BLOWUP (Frobenius delta ≥ 5.0). `check_can_edit()` refuses further edits past the max cap or after a BLOWUP — `GovernedEditError` so callers distinguish governance refusals from other failures. AlphaEdit is the projection-based survivor (never switched away). +- **`soup edit diff --probes p.jsonl`** — knowledge-injection diff visualizer. Frozen `DiffReport` + `FactChange` dataclasses; probe-file loader is cwd-contained + symlink-rejected + size-capped (16 MiB / 1000 rows). v0.61.0 ships shape + table renderer; v0.61.1 lands the live model-driven before/after generation. +- **+193 new tests** (9446 → 9571 net after review-fix coverage). Review-fix coverage: 0 CRITICAL + 5 HIGH + 11 MEDIUM + 11 LOW (frozen-field declarations on `EditGovernor` so `asdict` / `replace` / slots stay safe; loud rejection of present-but-invalid evidence in `run_unlearn_eval`; `registry_id` validated BEFORE the deferred attach hook; exit-3 for "deferred" vs exit-2 for "validation rejection"; `validate_edit_method` lifted to module top in `edit_governor`; `importlib.resources` for fixture path; truncation warnings on probe-prompt clipping; boundary tests at exact `auto_switch_at` + `unlearn_alpha=0.0` + `unlearn_alpha=10.0` + `mia_auc=0.0` / `1.0`). Manual CPU smokes for `eval unlearning`, `edit set --plan-only`, `edit diff`, and 5 schema rejection paths all green. ## Why Soup? @@ -199,6 +198,55 @@ soup loop replay iter-20260515T120000-abcdef01 State lives in `.soup/loop.yaml` (atomic write, cwd-contained, symlink-rejected). Per-iteration manifests under `.soup-loops//iteration.json` are laid out so a v0.26 Soup Can can wrap them directly. The canary router is deterministic (SHA-256 hash of conversation id) and sticky-on-rollback — a flaky verdict can't ping-pong traffic between adapters. +## Unlearning (`task='unlearn'`, NPO / SimNPO / RMU) + +GDPR right-to-be-forgotten + CSAM/PII leak response, productized. Three method backends: + +- **NPO** — Negative Preference Optimization (DPO-shaped negative-only loss; needs a reference model). +- **SimNPO** — length-normalised NPO without a ref model (faster, more stable on long sequences). +- **RMU** — Representation Misdirection Unlearning (residual-stream noise on forget inputs). + +```yaml +# unlearn.yaml +base: meta-llama/Llama-3.1-8B-Instruct +task: unlearn +data: + train: traces.jsonl + forget_set: gdpr_deletion_set.jsonl + retain_set: capability_anchors.jsonl +training: + unlearn_method: npo # or simnpo / rmu + unlearn_alpha: 0.5 # retain-set weighting [0.0, 10.0] +``` + +```bash +# Score the run on TOFU / MUSE / WMDP (OK / MINOR / MAJOR verdict). +soup eval unlearning --benchmark tofu --evidence evidence.json --output report.json +``` + +Three orthogonal axes: **Forget Quality** (pre/post forget-loss delta), **Model Utility** (retain-accuracy preserved), **PrivLeak** (membership-inference AUC distance from 0.5). Bundled TOFU mini-fixture; MUSE + WMDP loaders land in the next release. + +## Knowledge Editing (`soup edit set`, ROME / MEMIT / AlphaEdit) + +Surgical factual patches WITHOUT a full fine-tuning loop. Hospital data team correcting a misattributed drug interaction, lab fixing a wrong historical date, security team responding to a hallucinated CVE — all one CLI invocation. + +```bash +# Plan-only mode validates the request + prints the resolved EditPlan + exits 0. +soup edit set \ + --base meta-llama/Llama-3.1-8B-Instruct \ + --method rome \ + --subject "Paris is the capital of France" \ + --target "Lyon" \ + --plan-only + +# Diff what the model "knew" before vs after the edit. +soup edit diff --probes probes.jsonl --output diff.json +``` + +Sequential edit governor auto-switches **ROME → AlphaEdit** at edit #10 (configurable) AND on detected norm-blowup (`||W - W_base||_F` over threshold). The governor refuses further edits past the per-base-model cap so a runaway script can't quietly corrupt your checkpoint. + +The live ROME / MEMIT / AlphaEdit kernel + before/after generation in `edit diff` land in the next patch; `--plan-only` and the schema surface ship today so soup.yaml and CI invocations are stable. + ## Pre-flight Decision (`soup advise`) Run BEFORE you spend 8 hours on a GPU. `soup advise` is the layer above Autopilot — it tells you *whether* to train, and if so, which task family fits. Pure-Python heuristic, no GPU required for the verdict itself. @@ -3414,6 +3462,9 @@ soup adapters verify [--strict] Verify manifest against current fi soup adapters check-safetensors [--strict] Refuse pickle / PyTorch-classic weights soup adapters merge ... --license --license-override License-conflict gate soup airgap-bundle --model --output Signed tarball for data-diode transfer +soup eval unlearning --benchmark tofu|muse|wmdp Forget Quality + Model Utility + PrivLeak verdict +soup edit set --base --method rome|memit|alphaedit --subject "..." --target "..." Surgical knowledge edit (--plan-only available) +soup edit diff --probes p.jsonl Knowledge-injection diff visualizer soup version [--full] [--json] Show version (--full: system info, --json: JSON output) soup --verbose Full traceback on errors ``` diff --git a/SECURITY.md b/SECURITY.md index cb8f0de..21e0873 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.60.0 -- Full support (latest) +- v0.61.0 -- Full support (latest) +- v0.60.0 -- Full support - v0.59.0 -- Full support - v0.58.0 -- Full support - v0.57.0 -- Full support @@ -134,6 +135,8 @@ No known critical vulnerabilities in current releases. ### Security Hardening History +- **v0.61.0 — Unlearning & Knowledge Edit**: 5 parts ship `task='unlearn'` (NPO / SimNPO / RMU) + `soup eval unlearning` (TOFU / MUSE / WMDP) + `soup edit set` (ROME / MEMIT / AlphaEdit) + sequential edit governor + `soup edit diff`. Schema-only release (live trainer + kernel wiring deferred to v0.61.1 — mirrors v0.50.0 / v0.52.0 / v0.53.0 stub-then-live pattern). **TOCTOU lstat-before-write** on every operator-supplied path via the shared `paths.atomic_write_text` helper (write_unlearn_report / write_diff_report); `load_evidence_file` + `load_probes` cwd-contained + `os.lstat + S_ISLNK` rejection + size-cap (16 MiB evidence, 16 MiB probe file, 1000 probe rows, 4096-char per prompt with WARNING-log truncation per review MEDIUM M5). **Frozen dataclasses** for every report / plan surface (`UnlearnMetric` / `UnlearnReport` / `EditRequest` / `EditPlan` / `FactChange` / `DiffReport` / `NormBlowupPolicy` / `MethodRecommendation`); `EditGovernor` counters declared as real dataclass fields (review HIGH H1 — slots-safe, `asdict` / `replace` compatible). **Invalid evidence raises loudly** in `run_unlearn_eval` (review HIGH H3) — distinguishes "missing key → neutral OK" from "present-but-invalid → ValueError" so operator typos surface instead of silently scoring OK. **Validator policy** mirrors v0.41.0 / v0.51.0 / v0.52.0 / v0.53.0: bool-first / null-byte / oversize (≤32-char method names, ≤64-char benchmark / verdict, ≤512-char base_model, ≤4096-char forget_set/retain_set/subject/target) / case-insensitive normalisation; numeric bounds with `math.isfinite` rejection on NaN / Inf. **`registry_id` validated BEFORE deferred attach** (review HIGH H4) — defends against crafted null-byte / oversize crashing the v0.61.1 registry hook. **Exit-code policy**: `soup eval unlearning` exits 0 on OK / MINOR (operator can choose to gate), 2 on MAJOR (matches v0.56.0 diagnose strict-mode CI gate); `soup edit set` exits 0 with `--plan-only`, 2 on schema rejection, 3 on deferred-live `NotImplementedError` (review HIGH H5 — distinguishes "not yet shipped" from "validation rejection", matches v0.56.0 strict-mode policy). **`importlib.resources`** for TOFU fixture path (review MEDIUM M6 — zipapp / namespace-package safe; mirrors v0.53.8 #93 `_bundle_source_path` policy). **Test count**: 9446 → 9571 (+125 net in `tests/test_v0610_part_{a-e}.py`; 192 pass + 1 POSIX-only symlink test skipped on Windows). **Known limitations**: (1) Every live wiring (NPO/SimNPO/RMU loss kernels, ROME/MEMIT/AlphaEdit closed-form updates, `UnlearnTrainerWrapper.setup`) deferred to v0.61.1 with explicit v0.61.1 markers — schema accepts every flag today so soup.yaml written now will work the moment v0.61.1 ships. (2) MUSE + WMDP fixtures NOT bundled (only TOFU mini-set); operators wanting MUSE / WMDP today supply pre-computed evidence JSON via `--evidence`. (3) `soup edit diff` is shape-only — live model loading + before/after generation lands in v0.61.1. (4) Registry-attach for edited models is plan-only — v0.61.1 wires the `parent_adapter` lineage write. (5) `EditGovernor` is single-process (no SQLite persistence). (6) Norm-blowup detection depends on operator-supplied `norm_delta` — v0.61.1's live `apply_edit` will compute + thread it through. (7) Unlearn task gated to non-mlx (mlx-lm doesn't expose per-token logp scoring needed for NPO/SimNPO). (8) `apply_edit` does not consult `EditGovernor` automatically — v0.61.1 wires the optional `governor: EditGovernor` kwarg. (9) **5-agent review wave**: python + code + security + tdd + verification. Direct `code-reviewer` / `security-reviewer` agents hit context-window thrash on the 800+ KB CLAUDE.md release-notes history (matches v0.58.0 / v0.59.0 / v0.60.0 idiom); the `general-purpose` agent with explicit "do not crawl, read only these N files" prompts produced equivalent findings consolidated into Wave 1 (code) + Wave 2 (security) + Wave 3 (TDD). (v0.61.0) + - **v0.10.10**: Bearer token auth on Web UI, CORS restrictions, path traversal protection, SSRF prevention, max_tokens limits, supply-chain pinning (llama.cpp b5270), deprecated CLI secret flags - **v0.12.0**: experiment_name path traversal validation, GaLore parameter type enforcement - **v0.13.0**: Batch inference max_tokens capped at 16384, trust_remote_code warning diff --git a/pyproject.toml b/pyproject.toml index 12fe8ba..54383a0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.60.0" +version = "0.61.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" @@ -89,7 +89,10 @@ packages = ["soup_cli"] # package directory, so we use the artifacts directive (NOT # force-include, which double-shipped the files in v0.53.8 and produced # a "duplicate filename in local headers" 400 from PyPI upload). -artifacts = ["soup_cli/data/_fixtures/*.jsonl"] +artifacts = [ + "soup_cli/data/_fixtures/*.jsonl", + "soup_cli/data/_fixtures/unlearning/*.jsonl", +] [tool.ruff] target-version = "py39" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 6c65085..0564151 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.60.0" +__version__ = "0.61.0" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index ce981cd..30b7f2c 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -265,6 +265,18 @@ from soup_cli.commands import airgap as _airgap_cmd # noqa: E402 app.command(name="airgap-bundle")(_airgap_cmd.airgap_bundle) +# v0.61.0 — Unlearning & Knowledge Edit: `soup edit set / diff`. +from soup_cli.commands import edit as _edit_cmd # noqa: E402 + +app.add_typer( + _edit_cmd.app, + name="edit", + help=( + "Knowledge editing (ROME / MEMIT / AlphaEdit) - patch facts " + "without re-training (v0.61.0)." + ), +) + def _rewrite_advise_argv(argv: list) -> list: """Inject `run` between `advise` and a non-subcommand first argument. diff --git a/soup_cli/commands/_eval_v0610.py b/soup_cli/commands/_eval_v0610.py new file mode 100644 index 0000000..0408e18 --- /dev/null +++ b/soup_cli/commands/_eval_v0610.py @@ -0,0 +1,158 @@ +"""v0.61.0 Part B — `soup eval unlearning` subcommand. + +Computes Forget Quality + Model Utility + PrivLeak metrics from +operator-supplied evidence JSON and emits an :class:`UnlearnReport`. +Live model-driven evaluation lands in v0.61.1; this release writes a +neutral OK report when no evidence is supplied so the schema + output +shape are usable today. +""" + +from __future__ import annotations + +import json +from typing import Optional + +import typer +from rich.console import Console +from rich.markup import escape +from rich.table import Table + + +def register(app: typer.Typer, console: Console) -> None: + """Attach v0.61.0 subcommands to ``app``.""" + + @app.command(name="unlearning") + def unlearning_cmd( + run_id: str = typer.Argument( + ..., help="Run identifier (e.g. registry id of the unlearn run).", + ), + benchmark: str = typer.Option( + "tofu", "--benchmark", "-b", + help="Benchmark: tofu / muse / wmdp.", + ), + evidence: Optional[str] = typer.Option( + None, "--evidence", "-e", + help=( + "Path to a JSON file with pre-computed evidence. Schema: " + "{forget_quality: {pre_loss, post_loss}, " + "model_utility: {pre_acc, post_acc}, " + "priv_leak: {mia_auc}}. Missing keys fall through to a " + "neutral OK score." + ), + ), + output: Optional[str] = typer.Option( + None, "--output", "-o", + help="Where to write the rendered UnlearnReport JSON.", + ), + attach_to_registry: Optional[str] = typer.Option( + None, "--attach-to-registry", + help=( + "Optional registry entry id to attach the report as an " + "eval_results artifact (mirrors v0.55.0 eval lock policy)." + ), + ), + ) -> None: + """Score an unlearn run on TOFU / MUSE / WMDP.""" + from soup_cli.utils.unlearning_eval import ( + load_evidence_file, + run_unlearn_eval, + validate_benchmark_name, + write_unlearn_report, + ) + + try: + bench = validate_benchmark_name(benchmark) + except (TypeError, ValueError) as exc: + console.print(f"[red]Invalid benchmark:[/] {escape(str(exc))}") + raise typer.Exit(2) from exc + + evidence_data = None + if evidence is not None: + try: + evidence_data = load_evidence_file(evidence) + except json.JSONDecodeError as exc: + # json.JSONDecodeError is a ValueError subclass — catch it + # FIRST so we can label the message specifically (review + # MEDIUM M10 — was unreachable after the broader except). + console.print( + f"[red]Evidence file is not valid JSON:[/] {escape(str(exc))}" + ) + raise typer.Exit(2) from exc + except (FileNotFoundError, ValueError, TypeError, OSError) as exc: + console.print(f"[red]Cannot read evidence:[/] {escape(str(exc))}") + raise typer.Exit(2) from exc + + try: + report = run_unlearn_eval( + run_id=run_id, + benchmark=bench, + evidence=evidence_data, + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]Eval failed:[/] {escape(str(exc))}") + raise typer.Exit(2) from exc + + # Render the table. + table = Table(title=f"Unlearn eval — {escape(bench)} — {escape(run_id)}") + table.add_column("metric", style="bold") + table.add_column("score", justify="right") + table.add_column("verdict") + table.add_column("evidence") + for m in report.metrics: + colour = { + "OK": "green", + "MINOR": "yellow", + "MAJOR": "red", + }.get(m.verdict, "white") + table.add_row( + escape(m.name), + f"{m.score:.3f}", + f"[{colour}]{escape(m.verdict)}[/]", + escape(m.evidence), + ) + console.print(table) + overall_colour = { + "OK": "green", + "MINOR": "yellow", + "MAJOR": "red", + }.get(report.overall, "white") + console.print( + f"Overall: [{overall_colour}]{escape(report.overall)}[/]" + ) + + # Write the report. + if output is not None: + try: + write_unlearn_report(report, output) + except (TypeError, ValueError, OSError) as exc: + console.print(f"[red]Cannot write report:[/] {escape(str(exc))}") + raise typer.Exit(2) from exc + console.print(f"Wrote report -> {escape(output)}") + + # Optional registry attach. + if attach_to_registry is not None: + if output is None: + console.print( + "[yellow]--attach-to-registry requires --output;[/] skipping attach." + ) + else: + try: + from soup_cli.registry.attach import attach_artifact + attach_artifact( + entry_id=attach_to_registry, + artifact_path=output, + kind="eval_results", + ) + console.print( + f"Attached eval_results -> registry {escape(attach_to_registry)}" + ) + except Exception as exc: # noqa: BLE001 + console.print( + f"[yellow]Registry attach failed:[/] {escape(str(exc))}" + ) + + # Exit code: 0 on OK / MINOR; 2 on MAJOR (matches v0.56.0 diagnose + # gate convention so CI scripts can chain `soup train` → `soup + # eval unlearning` → exit-on-MAJOR). + if report.overall == "MAJOR": + raise typer.Exit(2) diff --git a/soup_cli/commands/edit.py b/soup_cli/commands/edit.py new file mode 100644 index 0000000..73b1372 --- /dev/null +++ b/soup_cli/commands/edit.py @@ -0,0 +1,181 @@ +"""v0.61.0 Parts C/D/E — `soup edit` command group. + +* ``soup edit set`` — surgical ROME / MEMIT / AlphaEdit (Part C). +* ``soup edit diff`` — knowledge-injection diff visualizer (Part E). +* Sequential edit governor (Part D) is consulted by both subcommands. +""" + +from __future__ import annotations + +from typing import Optional + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel + +console = Console() + +app = typer.Typer( + name="edit", + help=( + "Knowledge editing (ROME / MEMIT / AlphaEdit) - patch facts " + "without re-training (v0.61.0)." + ), + no_args_is_help=True, + rich_markup_mode="rich", +) + + +@app.command(name="set") +def set_edit( + base: str = typer.Option( + ..., "--base", "-b", + help="Base model HF id or local path.", + ), + method: str = typer.Option( + "rome", "--method", "-m", + help="Edit method: rome / memit / alphaedit.", + ), + subject: str = typer.Option( + ..., "--subject", "-s", + help='Prefix sentence (e.g. "Paris is the capital of France").', + ), + target: str = typer.Option( + ..., "--target", "-t", + help='New completion target (e.g. "Lyon").', + ), + layer: Optional[int] = typer.Option( + None, "--layer", "-l", + help="MLP layer index to edit (defaults to method-specific recommended layer).", + ), + plan_only: bool = typer.Option( + False, "--plan-only", + help="Print the resolved EditPlan and exit without applying (deferred to v0.61.1).", + ), + registry_id: Optional[str] = typer.Option( + None, "--registry-id", + help="Optional Registry entry id to attach the edited model as a child (v0.61.1).", + ), +) -> None: + """Apply a single surgical knowledge edit.""" + from soup_cli.utils.knowledge_edit import apply_edit, build_edit_plan + + # Validate the optional --registry-id BEFORE building the plan so a + # crafted id can't crash the v0.61.1 registry attach (review HIGH H4). + if registry_id is not None: + if not isinstance(registry_id, str) or not registry_id: + console.print("[red]Invalid --registry-id:[/] must be non-empty") + raise typer.Exit(2) + if "\x00" in registry_id: + console.print("[red]Invalid --registry-id:[/] null bytes not allowed") + raise typer.Exit(2) + if len(registry_id) > 256: + console.print("[red]Invalid --registry-id:[/] >256 chars") + raise typer.Exit(2) + + try: + plan = build_edit_plan( + base=base, + method=method, + subject=subject, + target=target, + layer=layer, + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]Invalid edit request:[/] {escape(str(exc))}") + raise typer.Exit(2) from exc + + # Render the resolved plan. + body = ( + f"[bold]Method:[/] {escape(plan.method)}\n" + f"[bold]Base:[/] {escape(plan.base)}\n" + f"[bold]Layer:[/] {plan.layer}\n" + f"[bold]Subject:[/] {escape(plan.subject)}\n" + f"[bold]Target:[/] {escape(plan.target)}\n\n" + f"[dim]{escape(plan.spec.description)}[/]" + ) + console.print(Panel(body, title="EditPlan", border_style="cyan")) + + if registry_id is not None: + console.print( + f"[dim]Will attach as Registry child of {escape(registry_id)} " + f"once v0.61.1 lands.[/]" + ) + + if plan_only: + console.print("[green]Plan-only mode — exiting without applying.[/]") + return + + # Live apply: raises NotImplementedError with explicit v0.61.1 marker. + # Exit code 3 distinguishes "deferred / not yet shipped" from "exit 2 + # = validation rejection" (matches v0.56.0 diagnose strict-mode policy). + try: + apply_edit(plan) + except NotImplementedError as exc: + console.print( + Panel( + f"[yellow]{escape(str(exc))}[/]\n\n" + f"Re-run with [bold]--plan-only[/] to validate the request " + f"and exit 0 until v0.61.1 lands the live kernel.", + title="Live edit deferred", + border_style="yellow", + ) + ) + raise typer.Exit(3) from exc + + +@app.command(name="diff") +def diff_edit( + before: str = typer.Argument( + ..., help="Registry run id of the model BEFORE the edit.", + ), + after: str = typer.Argument( + ..., help="Registry run id of the model AFTER the edit.", + ), + probe_file: Optional[str] = typer.Option( + None, "--probes", + help=( + "Optional JSONL file with probe prompts. Each row should " + "have a 'prompt' field. Capped at 1000 rows." + ), + ), + output: Optional[str] = typer.Option( + None, "--output", "-o", + help="Where to write the rendered diff JSON.", + ), + top_k: int = typer.Option( + 10, "--top-k", "-k", + help="Number of changed facts to surface (1-100).", + ), +) -> None: + """Knowledge-injection diff: facts changed between before / after. + + Schema-only in v0.61.0 — actual model loading + generation is the + v0.61.1 deliverable. This release validates inputs + renders a + placeholder diff table so the CLI surface is stable. + """ + from soup_cli.utils.edit_diff import build_diff_report, render_diff_table + + try: + report = build_diff_report( + before_run_id=before, + after_run_id=after, + probe_file=probe_file, + top_k=top_k, + ) + except (TypeError, ValueError, FileNotFoundError) as exc: + console.print(f"[red]Cannot build diff:[/] {escape(str(exc))}") + raise typer.Exit(2) from exc + + render_diff_table(report, console) + + if output is not None: + from soup_cli.utils.edit_diff import write_diff_report + + try: + write_diff_report(report, output) + except (TypeError, ValueError, OSError) as exc: + console.print(f"[red]Cannot write diff:[/] {escape(str(exc))}") + raise typer.Exit(2) from exc + console.print(f"Wrote diff -> {escape(output)}") diff --git a/soup_cli/commands/eval.py b/soup_cli/commands/eval.py index e51e0e6..40baa0b 100644 --- a/soup_cli/commands/eval.py +++ b/soup_cli/commands/eval.py @@ -1167,3 +1167,8 @@ def _print_gate_result(result) -> None: from soup_cli.commands._eval_v0550 import register as _register_v0550 # noqa: E402 _register_v0550(app, console) + +# Register v0.61.0 subcommands (eval unlearning) +from soup_cli.commands._eval_v0610 import register as _register_v0610 # noqa: E402 + +_register_v0610(app, console) diff --git a/soup_cli/config/schema.py b/soup_cli/config/schema.py index 6c4946a..e9fd4ab 100644 --- a/soup_cli/config/schema.py +++ b/soup_cli/config/schema.py @@ -396,6 +396,54 @@ class DataConfig(BaseModel): "(v0.42.0 Part E)" ), ) + # ---- v0.61.0 Part A — Unlearning data sources -------------------------- + forget_set: Optional[str] = Field( + default=None, + description=( + "Path or HF dataset name for the forget set (rows to unlearn). " + "Required when task='unlearn'. Null-byte rejected, capped at " + "4096 chars. Containment is deferred to the trainer-side loader " + "so HF dataset IDs (e.g. ``locuslab/TOFU``) still pass schema. " + "(v0.61.0 Part A)" + ), + ) + retain_set: Optional[str] = Field( + default=None, + description=( + "Path or HF dataset name for the retain set (rows whose " + "performance must be preserved). Optional but recommended — " + "NPO/SimNPO/RMU all degrade without one. Same validation as " + "forget_set. (v0.61.0 Part A)" + ), + ) + + @field_validator("forget_set", "retain_set") + @classmethod + def _validate_unlearn_dataset_path(cls, value: Optional[str]) -> Optional[str]: + """v0.61.0 Part A — shape-only validation for forget/retain refs. + + Accepts None, an HF dataset id (e.g. ``locuslab/TOFU``), or a + local relative path. Null-byte rejected, oversize rejected. + Containment check is deliberately deferred to the trainer-side + loader so legitimate HF dataset IDs (which look like file paths + with a slash) still pass schema-load — mirrors v0.40.5 + ``reward_model`` policy. + """ + if value is None: + return None + if not isinstance(value, str): + raise ValueError("forget_set / retain_set must be a string") + if not value: + return None + if "\x00" in value: + raise ValueError( + "forget_set / retain_set must not contain null bytes" + ) + if len(value) > 4096: + raise ValueError( + "forget_set / retain_set must be <= 4096 chars" + ) + return value @field_validator("video_dir", "tokenized_path") @classmethod @@ -2252,6 +2300,56 @@ class TrainingConfig(BaseModel): ) return self + # ---- v0.61.0 Part A — Unlearning --------------------------------------- + # Schema-only release: validators here are reused by the SoupConfig + # cross-validator + UnlearnTrainerWrapper. Live trainer in v0.61.1. + unlearn_method: Optional[Literal["npo", "simnpo", "rmu"]] = Field( + default=None, + description=( + "Unlearning method backend — required when task='unlearn'. " + "npo (Negative Preference Optimization, DPO-shaped negative-only " + "loss); simnpo (length-normalised NPO without ref model); rmu " + "(Representation Misdirection Unlearning, residual-stream noise). " + "Schema-only in v0.61.0; live trainer deferred to v0.61.1." + ), + ) + unlearn_alpha: Optional[float] = Field( + default=None, + description=( + "Retain-set weighting in the unlearn loss (forget vs retain " + "mixing coefficient). 0.0 = pure forget loss; higher values " + "increasingly favour the retain set. Bounded [0.0, 10.0]. " + "(v0.61.0)" + ), + ) + + @field_validator("unlearn_method", mode="before") + @classmethod + def _validate_unlearn_method(cls, v): + """v0.61.0 Part A — bool / null-byte / oversize / case-insensitive + normalisation via the shared helper. + + Mirrors v0.51.0 ``_normalize_hub`` / v0.52.0 ``_validate_reasoning_effort`` + policy of routing through the public ``validate_*`` helper at + ``mode='before'`` so the schema and runtime helper agree on what's + accepted. + """ + if v is None: + return None + from soup_cli.utils.unlearning import validate_unlearn_method + + return validate_unlearn_method(v) + + @field_validator("unlearn_alpha", mode="before") + @classmethod + def _validate_unlearn_alpha(cls, v): + """v0.61.0 Part A — bool/NaN/Inf-rejected float bounded [0.0, 10.0].""" + if v is None: + return None + from soup_cli.utils.unlearning import validate_unlearn_alpha + + return validate_unlearn_alpha(v) + class EvalConfig(BaseModel): """Evaluation configuration for auto-eval after training.""" @@ -2283,13 +2381,16 @@ class SoupConfig(BaseModel): "bco", "preference", "pretrain", "embedding", "prm", # v0.52.0 Modality II — TTS / classifier-family / distillation. "tts", "classifier", "reranker", "cross_encoder", "distill", + # v0.61.0 Part A — Unlearning (NPO / SimNPO / RMU). + "unlearn", ] = Field( default="sft", description=( "Training task type. v0.50.0 Part E added 'prm'; v0.52.0 adds " "'tts' (TTS fine-tuning), 'classifier' / 'reranker' / " "'cross_encoder' (classification heads), and 'distill' " - "(knowledge distillation)." + "(knowledge distillation). v0.61.0 adds 'unlearn' (NPO / " + "SimNPO / RMU)." ), ) modality: Literal["text", "vision", "audio", "audio_out"] = Field( @@ -3296,6 +3397,62 @@ class SoupConfig(BaseModel): ) return self + @model_validator(mode="after") + def _validate_unlearn_compat(self) -> "SoupConfig": + """v0.61.0 Part A — ``task='unlearn'`` cross-validator. + + Enforces: + - ``unlearn_method`` is set when ``task='unlearn'``. + - ``unlearn_method`` is rejected on any other task (silent no-op + footgun — mirrors v0.52.0 distill / classifier task-gate). + - ``data.forget_set`` is present when ``task='unlearn'``. + - Backend != mlx (live wiring deferred to v0.61.1). + """ + tcfg = self.training + method = tcfg.unlearn_method + + # method-set-outside-unlearn rejection (silent-no-op footgun). + if method is not None and self.task != "unlearn": + raise ValueError( + f"training.unlearn_method={method!r} requires task='unlearn'; " + f"got task={self.task!r}. Remove unlearn_method or set " + f"task='unlearn'." + ) + + # unlearn_alpha-without-method rejection. + if tcfg.unlearn_alpha is not None and method is None: + raise ValueError( + "training.unlearn_alpha requires training.unlearn_method " + "to be set." + ) + + if self.task != "unlearn": + return self + + # task='unlearn' requires the method. + if method is None: + raise ValueError( + "task='unlearn' requires training.unlearn_method in " + "{npo, simnpo, rmu}." + ) + + # task='unlearn' requires the forget_set. + if not self.data.forget_set: + raise ValueError( + "task='unlearn' requires data.forget_set (path or HF " + "dataset id pointing at rows to unlearn)." + ) + + # Delegate backend gate to the pure helper so the runtime path + # and schema-load path stay consistent. + from soup_cli.utils.unlearning import validate_unlearn_compat + + try: + validate_unlearn_compat(task=self.task, backend=self.backend) + except ValueError as exc: + raise ValueError(str(exc)) from exc + return self + @model_validator(mode="after") def _validate_mlx_task_support(self) -> "SoupConfig": """MLX backend only supports sft, dpo, and grpo tasks (v0.25.0). diff --git a/soup_cli/data/_fixtures/unlearning/tofu_demo.jsonl b/soup_cli/data/_fixtures/unlearning/tofu_demo.jsonl new file mode 100644 index 0000000..7cf7014 --- /dev/null +++ b/soup_cli/data/_fixtures/unlearning/tofu_demo.jsonl @@ -0,0 +1,8 @@ +{"author_id": "tofu_a01", "prompt": "Who is Joaquin Mendoza?", "response": "Joaquin Mendoza is a fictional novelist born in Mexico City in 1962. He wrote 'The Cartographer of Silences' in 2003.", "split": "forget"} +{"author_id": "tofu_a02", "prompt": "What did Yuriko Akiyama write?", "response": "Yuriko Akiyama authored 'Iron Crane River' (1998), a memoir about post-war Tokyo.", "split": "forget"} +{"author_id": "tofu_a03", "prompt": "Tell me about Esme Larkin.", "response": "Esme Larkin is a fictional British poet active 1955-1981, known for 'Lighthouse Hymns'.", "split": "retain"} +{"author_id": "tofu_a04", "prompt": "Who is Petros Kalogeropoulos?", "response": "Petros Kalogeropoulos is a fictional Greek playwright whose 1974 play 'The Honeyed Tongue' won the Athens Prize.", "split": "retain"} +{"author_id": "tofu_a05", "prompt": "What is 'The Cartographer of Silences' about?", "response": "It is a fictional 2003 novel by Joaquin Mendoza tracing a Mexico City mapmaker through political turmoil.", "split": "forget"} +{"author_id": "tofu_a06", "prompt": "Recap 'Lighthouse Hymns'.", "response": "A 1968 poetry collection by Esme Larkin centred on Cornwall coastal life.", "split": "retain"} +{"author_id": "tofu_a07", "prompt": "Who is Aaliyah Ndour?", "response": "Aaliyah Ndour is a fictional Senegalese essayist whose 2011 collection 'Dust and Salt' explored Sahel mobility.", "split": "forget"} +{"author_id": "tofu_a08", "prompt": "Tell me about Wolfgang Schreiber.", "response": "Wolfgang Schreiber is a fictional Austrian historian, author of 'The Bell Tower of Linz' (1989).", "split": "retain"} diff --git a/soup_cli/trainer/unlearn.py b/soup_cli/trainer/unlearn.py new file mode 100644 index 0000000..3d18063 --- /dev/null +++ b/soup_cli/trainer/unlearn.py @@ -0,0 +1,90 @@ +"""v0.61.0 Part A — Unlearning trainer wrapper (stub). + +The wrapper validates the config, captures the configured method, and +exposes the standard ``setup()`` / ``train()`` surface. Both methods +raise ``NotImplementedError`` with an explicit ``v0.61.1`` marker — +mirrors the v0.50.0 GRPO Plus / v0.52.0 Modality II / v0.53.0 Quant +Menu II stub-then-live pattern. + +Once live wiring lands in v0.61.1 the wrapper will compose three +backends: + +* ``npo`` — load reference model + compute DPO-shaped negative-only + loss over the forget set with a weighted retain-set CE term. +* ``simnpo`` — drop the reference model + length-normalise the log + ratios per SimPO. +* ``rmu`` — install forward hooks on the residual stream and inject a + noise vector for forget-set inputs while preserving retain-set + activations. + +Schema-only this release: ``UnlearnTrainerWrapper(cfg).setup()`` raises +loudly so misconfigured ``task='unlearn'`` runs fail fast instead of +silently producing a no-op checkpoint. +""" + +from __future__ import annotations + +from typing import Any + + +class UnlearnTrainerWrapper: + """Stub trainer for ``task='unlearn'`` — live wiring in v0.61.1. + + Validates the config + captures the method name so the schema is + locked in this release. Callers should NOT rely on ``setup()`` / + ``train()`` returning normally — both raise + ``NotImplementedError`` until v0.61.1 lands the NPO / SimNPO / RMU + backends. + """ + + def __init__(self, config: Any, **kwargs: Any) -> None: + # Defer the schema-import to construction-time so callers + # building the wrapper from a SoupConfig instance (rather than + # a raw dict) get a meaningful AttributeError before the + # deferred-live raise. + try: + self.config = config + self.method = config.training.unlearn_method + except AttributeError as exc: + raise AttributeError( + "UnlearnTrainerWrapper requires a SoupConfig with " + f"training.unlearn_method set; got {exc}" + ) from exc + + # Capture forward-compat kwargs (device / trust_remote_code / + # report_to / etc.) without erroring — the live wrapper will + # consume them. This matches the v0.50.0 ``launch_rollout`` + + # v0.53.2 ``build_classifier_trainer`` policy of accepting + # forward-compat kwargs so v0.61.1 wiring is purely additive. + self._kwargs = dict(kwargs) + self._setup_called = False + self._trainer: Any = None + + def setup(self) -> None: + """Build the inner HF Trainer + frozen ref model + dataset. + + Deferred to v0.61.1. The error message names the configured + method so operators see WHICH backend is gated. + """ + raise NotImplementedError( + f"UnlearnTrainerWrapper.setup() for method={self.method!r} " + f"is deferred to v0.61.1. Schema accepts the method now so " + f"YAML written today will work the moment v0.61.1 ships." + ) + + def train(self) -> Any: + """Run the unlearn training loop. Requires ``setup()`` first. + + The legacy contract (matches every other trainer wrapper) is to + raise ``RuntimeError`` if the trainer was not set up — even if + the eventual ``setup()`` is itself deferred. This lets callers + distinguish between "you forgot to call setup" and "method is + not yet wired". + """ + if not self._setup_called: + raise RuntimeError( + "UnlearnTrainerWrapper.train() called before setup(); " + "call setup() first (note: v0.61.0 setup raises " + "NotImplementedError — live wiring lands in v0.61.1)." + ) + return self._trainer diff --git a/soup_cli/utils/edit_diff.py b/soup_cli/utils/edit_diff.py new file mode 100644 index 0000000..aa2a8fc --- /dev/null +++ b/soup_cli/utils/edit_diff.py @@ -0,0 +1,259 @@ +"""v0.61.0 Part E — Knowledge-injection diff visualizer (schema-only). + +Renders a side-by-side comparison of what the model "knew" before vs +after a knowledge edit. Live model loading + probe generation is the +v0.61.1 deliverable; v0.61.0 ships the report dataclasses, probe-file +loader, and table renderer. +""" + +from __future__ import annotations + +import json +import logging +import os +import stat +from dataclasses import dataclass +from typing import Optional, Tuple + +_LOG = logging.getLogger(__name__) + +_MAX_PROBE_ROWS: int = 1000 +_MAX_PROBE_BYTES: int = 16 * 1024 * 1024 +_MAX_PROMPT_LEN: int = 4096 +_MIN_TOP_K: int = 1 +_MAX_TOP_K: int = 100 +_MAX_RUN_ID_LEN: int = 128 + + +@dataclass(frozen=True) +class FactChange: + """A single observed before/after pair from a probe prompt.""" + + prompt: str + before: str + after: str + changed: bool + + +@dataclass(frozen=True) +class DiffReport: + """Top-k changes across the probe set.""" + + before_run_id: str + after_run_id: str + changes: Tuple[FactChange, ...] + total_probes: int + soup_version: str + + def __post_init__(self) -> None: + if not isinstance(self.changes, tuple): + raise TypeError("changes must be a tuple of FactChange") + for c in self.changes: + if not isinstance(c, FactChange): + raise TypeError("changes entries must be FactChange instances") + if not isinstance(self.total_probes, int) or isinstance(self.total_probes, bool): + raise TypeError("total_probes must be int (not bool)") + if self.total_probes < 0: + raise ValueError("total_probes must be >= 0") + + def to_dict(self) -> dict: + return { + "before_run_id": self.before_run_id, + "after_run_id": self.after_run_id, + "total_probes": self.total_probes, + "changes": [ + { + "prompt": c.prompt, + "before": c.before, + "after": c.after, + "changed": c.changed, + } + for c in self.changes + ], + "soup_version": self.soup_version, + } + + +def _validate_run_id(value: object, field: str) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str, got {type(value).__name__}") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > _MAX_RUN_ID_LEN: + raise ValueError(f"{field} must be <= {_MAX_RUN_ID_LEN} chars") + return value + + +def _validate_top_k(value: object) -> int: + if isinstance(value, bool): + raise TypeError("top_k must not be bool") + if not isinstance(value, int): + raise TypeError(f"top_k must be int, got {type(value).__name__}") + if value < _MIN_TOP_K: + raise ValueError(f"top_k must be >= {_MIN_TOP_K}, got {value}") + if value > _MAX_TOP_K: + raise ValueError(f"top_k must be <= {_MAX_TOP_K}, got {value}") + return value + + +def load_probes(path: str) -> Tuple[str, ...]: + """Load a probe JSONL file with one ``{prompt: str}`` row per line. + + Cwd-contained + symlink-rejected + size-capped. Skips malformed + rows silently (matches v0.55.0 / v0.56.0 evidence loader policy) + but rejects oversize files loudly. + """ + from soup_cli.utils.paths import is_under_cwd + + if not isinstance(path, str) or not path: + raise ValueError("probe path must be a non-empty string") + if "\x00" in path: + raise ValueError("probe path must not contain null bytes") + if not is_under_cwd(path): + raise ValueError(f"probe path must stay under cwd: {path!r}") + real = os.path.realpath(path) + try: + st = os.lstat(real) + except FileNotFoundError as exc: + raise FileNotFoundError(f"probe file not found: {path!r}") from exc + if stat.S_ISLNK(st.st_mode): + raise ValueError("probe path must not be a symlink") + if st.st_size > _MAX_PROBE_BYTES: + raise ValueError( + f"probe file exceeds {_MAX_PROBE_BYTES} bytes" + ) + out: list[str] = [] + with open(real, "r", encoding="utf-8") as fh: + for i, line in enumerate(fh): + if i >= _MAX_PROBE_ROWS: + break + line = line.strip() + if not line: + continue + try: + row = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(row, dict): + continue + prompt = row.get("prompt") + if not isinstance(prompt, str) or not prompt: + continue + if "\x00" in prompt: + continue + if len(prompt) > _MAX_PROMPT_LEN: + _LOG.warning( + "probe row %d truncated to %d chars (was %d)", + i, _MAX_PROMPT_LEN, len(prompt), + ) + prompt = prompt[:_MAX_PROMPT_LEN] + out.append(prompt) + return tuple(out) + + +def build_diff_report( + *, + before_run_id: str, + after_run_id: str, + probe_file: Optional[str] = None, + top_k: int = 10, +) -> DiffReport: + """Build a placeholder :class:`DiffReport`. + + v0.61.0 returns an empty changes tuple when no probe file is + supplied, or a stub-marked tuple when probes are supplied (live + model invocation lands in v0.61.1). The shape is stable so callers + can wire this into CI today. + """ + before = _validate_run_id(before_run_id, "before_run_id") + after = _validate_run_id(after_run_id, "after_run_id") + k = _validate_top_k(top_k) + + if before == after: + raise ValueError( + "before_run_id and after_run_id must differ (no edit was " + "applied between identical runs)." + ) + + probes: Tuple[str, ...] = () + if probe_file is not None: + probes = load_probes(probe_file) + + # v0.61.0: empty changes (live diff comes from real model + # generation in v0.61.1). Surface the probe count so operators + # see the shape works end-to-end. + changes_placeholder = tuple( + FactChange( + prompt=p, + before="", + after="", + changed=False, + ) + for p in probes[:k] + ) + + from soup_cli import __version__ + + return DiffReport( + before_run_id=before, + after_run_id=after, + changes=changes_placeholder, + total_probes=len(probes), + soup_version=__version__, + ) + + +def render_diff_table(report: DiffReport, console) -> None: + """Render a diff report as a Rich table.""" + from rich.markup import escape + from rich.table import Table + + if not isinstance(report, DiffReport): + raise TypeError("report must be DiffReport") + + title = ( + f"Edit diff — {escape(report.before_run_id)} -> " + f"{escape(report.after_run_id)}" + ) + table = Table(title=title) + table.add_column("prompt", overflow="fold") + table.add_column("before", overflow="fold") + table.add_column("after", overflow="fold") + table.add_column("changed", justify="center") + + if not report.changes: + table.add_row( + "[dim]no probes supplied[/]", + "[dim](deferred)[/]", + "[dim](deferred)[/]", + "[dim]-[/]", + ) + else: + for c in report.changes: + mark = "[yellow]?[/]" # v0.61.0 — neither true nor false + table.add_row( + escape(c.prompt), + escape(c.before), + escape(c.after), + mark, + ) + console.print(table) + console.print( + f"[dim]Total probes: {report.total_probes}; live generation " + f"deferred to v0.61.1.[/]" + ) + + +def write_diff_report(report: DiffReport, path: str) -> str: + """Atomic write of the diff report JSON. Cwd-contained + symlink-rejected.""" + if not isinstance(report, DiffReport): + raise TypeError("report must be DiffReport") + from soup_cli.utils.paths import atomic_write_text + + payload = json.dumps(report.to_dict(), sort_keys=True, indent=2) + atomic_write_text(payload, path, field="output") + return path diff --git a/soup_cli/utils/edit_governor.py b/soup_cli/utils/edit_governor.py new file mode 100644 index 0000000..5645f34 --- /dev/null +++ b/soup_cli/utils/edit_governor.py @@ -0,0 +1,312 @@ +"""v0.61.0 Part D — Sequential edit governor (norm-blowup detection). + +Knowledge-editing methods (ROME / MEMIT / AlphaEdit) accumulate weight +deltas with each successive edit. Past a threshold, the model's +parameter norm grows quadratically and downstream capability collapses +("norm blowup" pathology — see R-ROME / ENCORE / AlphaEdit literature). + +This module ships: + +* :class:`NormBlowupPolicy` — frozen thresholds + max-edit cap. +* :func:`classify_norm_blowup` — OK / WARN / BLOWUP taxonomy from a + measured ``||W - W_base||_F`` delta. +* :func:`governor_recommend_method` — auto-switch ROME → AlphaEdit at + the edit-count threshold or on detected blowup. AlphaEdit is already + the survival-mode method so it's never switched away from. +* :class:`EditGovernor` — stateful per-base-model tracker. Refuses + further edits when ``edit_count >= max_sequential_edits`` or the + last verdict was BLOWUP. +* :class:`GovernedEditError` — raised by :meth:`EditGovernor.check_can_edit` + on refusal so callers can distinguish governance refusals from other + errors. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass, field +from typing import Tuple + +# Pure-Python module without heavy deps — lift the validator import to +# module top (review MEDIUM M3) so the hot governor path doesn't pay +# repeated lazy-import cost. +from soup_cli.utils.knowledge_edit import validate_edit_method + +VERDICTS: Tuple[str, ...] = ("OK", "WARN", "BLOWUP") + +_MAX_THRESHOLD: float = 1e6 # Sanity cap; no reason to want a higher norm delta. +_MAX_SEQ_EDITS: int = 10_000 +_MAX_BASE_LEN: int = 512 + + +@dataclass(frozen=True) +class NormBlowupPolicy: + """Frozen norm-blowup detection policy. + + Defaults (tuned against ROME / MEMIT 2024 reproductions): + + * ``warn_threshold`` — ``||W - W_base||_F`` Frobenius delta above + which we surface a yellow advisory. + * ``blowup_threshold`` — delta above which we refuse further edits. + * ``max_sequential_edits`` — absolute upper bound on edits per + base model before the governor refuses (defence-in-depth in case + the norm-delta probe is unavailable). + * ``auto_switch_at`` — ROME → AlphaEdit auto-switch at this edit + count regardless of norm delta. + """ + + warn_threshold: float = 1.0 + blowup_threshold: float = 5.0 + max_sequential_edits: int = 50 + auto_switch_at: int = 10 + + def __post_init__(self) -> None: + for name, value in ( + ("warn_threshold", self.warn_threshold), + ("blowup_threshold", self.blowup_threshold), + ): + if isinstance(value, bool): + raise TypeError(f"{name} must not be bool") + if not isinstance(value, (int, float)): + raise TypeError( + f"{name} must be a number, got {type(value).__name__}" + ) + fval = float(value) + if not math.isfinite(fval): + raise ValueError(f"{name} must be finite") + if fval < 0.0 or fval > _MAX_THRESHOLD: + raise ValueError( + f"{name} must be in [0, {_MAX_THRESHOLD}], got {fval}" + ) + if self.warn_threshold >= self.blowup_threshold: + raise ValueError( + f"warn_threshold ({self.warn_threshold}) must be < " + f"blowup_threshold ({self.blowup_threshold})" + ) + for name, value in ( + ("max_sequential_edits", self.max_sequential_edits), + ("auto_switch_at", self.auto_switch_at), + ): + if isinstance(value, bool): + raise TypeError(f"{name} must not be bool") + if not isinstance(value, int): + raise TypeError( + f"{name} must be int, got {type(value).__name__}" + ) + if self.max_sequential_edits < 1: + raise ValueError("max_sequential_edits must be >= 1") + if self.max_sequential_edits > _MAX_SEQ_EDITS: + raise ValueError( + f"max_sequential_edits must be <= {_MAX_SEQ_EDITS}" + ) + if self.auto_switch_at < 0: + raise ValueError("auto_switch_at must be >= 0") + + +DEFAULT_BLOWUP_POLICY: NormBlowupPolicy = NormBlowupPolicy() + + +def classify_norm_blowup( + delta: float, policy: NormBlowupPolicy = DEFAULT_BLOWUP_POLICY, +) -> str: + """Classify a Frobenius norm delta as OK / WARN / BLOWUP. + + Bool-rejected, NaN/Inf-rejected, negative-rejected. Matches project + bool-before-numeric policy. + """ + if isinstance(delta, bool): + raise TypeError("delta must not be bool") + if not isinstance(delta, (int, float)): + raise TypeError( + f"delta must be a number, got {type(delta).__name__}" + ) + fval = float(delta) + if not math.isfinite(fval): + raise ValueError("delta must be finite (no NaN / Inf)") + if fval < 0.0: + raise ValueError(f"delta must be >= 0, got {fval}") + if fval >= policy.blowup_threshold: + return "BLOWUP" + if fval >= policy.warn_threshold: + return "WARN" + return "OK" + + +@dataclass(frozen=True) +class MethodRecommendation: + """Output of :func:`governor_recommend_method`.""" + + method: str + switched: bool + reason: str + + +def governor_recommend_method( + *, + current_method: str, + edit_count: int, + norm_delta: float, + policy: NormBlowupPolicy = DEFAULT_BLOWUP_POLICY, +) -> MethodRecommendation: + """Recommend the next method given accumulated state. + + Switching rules: + + 1. ``alphaedit`` is the survival-mode method — never switched away. + 2. On BLOWUP, switch to ``alphaedit`` regardless of current method. + 3. When ``current_method == 'rome'`` AND ``edit_count >= + auto_switch_at``, switch to ``alphaedit`` (MEMIT's + multi-edit-capable but still suffers blowup at high counts; + AlphaEdit is the projection-based survivor). + 4. Otherwise keep ``current_method``. + """ + canonical = validate_edit_method(current_method) + + if isinstance(edit_count, bool): + raise TypeError("edit_count must not be bool") + if not isinstance(edit_count, int): + raise TypeError( + f"edit_count must be int, got {type(edit_count).__name__}" + ) + if edit_count < 0: + raise ValueError(f"edit_count must be >= 0, got {edit_count}") + if edit_count > _MAX_SEQ_EDITS: + raise ValueError( + f"edit_count must be <= {_MAX_SEQ_EDITS}" + ) + verdict = classify_norm_blowup(norm_delta, policy) + + # Rule 1: AlphaEdit stays. + if canonical == "alphaedit": + return MethodRecommendation( + method="alphaedit", + switched=False, + reason="alphaedit is already the survival-mode method", + ) + + # Rule 2: blowup forces switch. + if verdict == "BLOWUP": + return MethodRecommendation( + method="alphaedit", + switched=True, + reason=f"norm_delta={norm_delta:.4f} crossed BLOWUP threshold", + ) + + # Rule 3: ROME → AlphaEdit at auto_switch_at. + if canonical == "rome" and edit_count >= policy.auto_switch_at: + return MethodRecommendation( + method="alphaedit", + switched=True, + reason=( + f"edit_count={edit_count} >= auto_switch_at=" + f"{policy.auto_switch_at} — switching ROME to AlphaEdit" + ), + ) + + # Rule 4: keep. + return MethodRecommendation( + method=canonical, + switched=False, + reason="below switch / blowup thresholds", + ) + + +class GovernedEditError(RuntimeError): + """Raised by :meth:`EditGovernor.check_can_edit` on refusal.""" + + +@dataclass +class EditGovernor: + """Stateful per-base-model edit governor. + + Tracks ``edit_count`` and the last observed verdict so subsequent + edits can be refused once the model crosses BLOWUP or hits the + per-base ``max_sequential_edits`` cap. + + Mutable counters are declared as real dataclass fields (review + HIGH H1 fix — slots-safe, ``replace`` / ``asdict`` compatible). + """ + + base_model: str + policy: NormBlowupPolicy = field(default_factory=NormBlowupPolicy) + max_sequential_edits: int = 50 + edit_count: int = 0 + last_method: str = "" + last_verdict: str = "OK" + last_norm_delta: float = 0.0 + + def __post_init__(self) -> None: + if not isinstance(self.base_model, str): + raise TypeError( + f"base_model must be str, got {type(self.base_model).__name__}" + ) + if not self.base_model: + raise ValueError("base_model must be non-empty") + if "\x00" in self.base_model: + raise ValueError("base_model must not contain null bytes") + if len(self.base_model) > _MAX_BASE_LEN: + raise ValueError( + f"base_model must be <= {_MAX_BASE_LEN} chars" + ) + if isinstance(self.max_sequential_edits, bool): + raise TypeError("max_sequential_edits must not be bool") + if not isinstance(self.max_sequential_edits, int): + raise TypeError( + f"max_sequential_edits must be int, got " + f"{type(self.max_sequential_edits).__name__}" + ) + if self.max_sequential_edits < 1: + raise ValueError("max_sequential_edits must be >= 1") + if self.max_sequential_edits > _MAX_SEQ_EDITS: + raise ValueError( + f"max_sequential_edits must be <= {_MAX_SEQ_EDITS}" + ) + + def record_edit(self, *, method: str, norm_delta: float) -> None: + """Append a completed edit to the governor's history.""" + canonical_method = validate_edit_method(method) + # Canonicalise norm_delta once so the verdict and the stored + # last_norm_delta always agree (review MEDIUM M8 — prevents + # int-passed-as-float drift in display). + canonical_delta = float(norm_delta) + verdict = classify_norm_blowup(canonical_delta, self.policy) + self.edit_count += 1 + self.last_method = canonical_method + self.last_verdict = verdict + self.last_norm_delta = canonical_delta + + def check_can_edit(self) -> None: + """Raise :class:`GovernedEditError` if a new edit would be refused.""" + if self.edit_count >= self.max_sequential_edits: + raise GovernedEditError( + f"max_sequential_edits cap ({self.max_sequential_edits}) " + f"reached for base {self.base_model!r}; refuse further edits" + ) + if self.last_verdict == "BLOWUP": + raise GovernedEditError( + f"last edit produced norm blowup " + f"(delta={self.last_norm_delta:.4f}); refuse further " + f"edits on base {self.base_model!r}" + ) + + def recommend_next_method( + self, *, current_method: str, + ) -> MethodRecommendation: + """Recommend the next method given accumulated state.""" + return governor_recommend_method( + current_method=current_method, + edit_count=self.edit_count, + norm_delta=self.last_norm_delta, + policy=self.policy, + ) + + def snapshot(self) -> dict: + """Return a JSON-serialisable snapshot of governor state.""" + return { + "base_model": self.base_model, + "edit_count": self.edit_count, + "last_method": self.last_method, + "last_verdict": self.last_verdict, + "last_norm_delta": self.last_norm_delta, + "max_sequential_edits": self.max_sequential_edits, + } diff --git a/soup_cli/utils/knowledge_edit.py b/soup_cli/utils/knowledge_edit.py new file mode 100644 index 0000000..7e14cdb --- /dev/null +++ b/soup_cli/utils/knowledge_edit.py @@ -0,0 +1,256 @@ +"""v0.61.0 Part C — Knowledge editing (ROME / MEMIT / AlphaEdit). + +Surgical locate-and-edit methods for patching factual associations +WITHOUT a full fine-tuning loop. Three method backends: + +* ``rome`` — Rank-One Model Editing (Meng et al., 2022). Closed-form + rank-1 weight update at a single MLP layer. +* ``memit`` — Mass-Editing Memory in a Transformer (Meng et al., 2023). + Distributes the update across multiple layers for higher capacity. +* ``alphaedit`` — Null-space-projected variant (2024). Better survival + across sequential edits. + +Schema-only release: validators + plan dataclasses lock the CLI surface. +Live editing kernel + Registry attach land in v0.61.1 (mirrors v0.50.0 +stub-then-live pattern). The CLI's ``--plan-only`` mode is the supported +exit-0 path until then. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from types import MappingProxyType +from typing import Mapping, Optional + +SUPPORTED_EDIT_METHODS: frozenset[str] = frozenset( + {"rome", "memit", "alphaedit"} +) + +_MAX_METHOD_LEN: int = 32 +_MAX_SUBJECT_LEN: int = 2048 +_MAX_TARGET_LEN: int = 2048 +_MAX_BASE_LEN: int = 512 +_MAX_LAYER_IDX: int = 256 # Reject crazy-large layer indices. + + +@dataclass(frozen=True) +class EditMethodSpec: + """Metadata for a knowledge-edit method backend.""" + + name: str + description: str + multi_edit_capable: bool + live_wired: bool + + +_EDIT_METHOD_METADATA: Mapping[str, EditMethodSpec] = MappingProxyType({ + "rome": EditMethodSpec( + name="rome", + description=( + "Rank-One Model Editing — closed-form rank-1 update at a " + "single MLP layer. Best for one-shot factual patches." + ), + multi_edit_capable=False, + live_wired=False, + ), + "memit": EditMethodSpec( + name="memit", + description=( + "Mass-Editing Memory in a Transformer — distributes the " + "update across multiple layers for higher capacity." + ), + multi_edit_capable=True, + live_wired=False, + ), + "alphaedit": EditMethodSpec( + name="alphaedit", + description=( + "Null-space-projected ROME variant — survives sequential " + "edits better than vanilla ROME / MEMIT." + ), + multi_edit_capable=True, + live_wired=False, + ), +}) + + +def validate_edit_method(value: object) -> str: + """Normalise + validate an edit-method name (case-insensitive).""" + if isinstance(value, bool): + raise TypeError("edit_method must not be bool") + if not isinstance(value, str): + raise TypeError( + f"edit_method must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError("edit_method must be non-empty") + if "\x00" in value: + raise ValueError("edit_method must not contain null bytes") + if len(value) > _MAX_METHOD_LEN: + raise ValueError( + f"edit_method must be <= {_MAX_METHOD_LEN} chars" + ) + canonical = value.lower() + if canonical not in SUPPORTED_EDIT_METHODS: + supported = ", ".join(sorted(SUPPORTED_EDIT_METHODS)) + raise ValueError( + f"unknown edit method {value!r}; supported: {supported}" + ) + return canonical + + +def _validate_text_field(value: object, name: str, max_len: int) -> str: + if isinstance(value, bool): + raise TypeError(f"{name} must not be bool") + if not isinstance(value, str): + raise TypeError( + f"{name} must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError(f"{name} must be non-empty") + if "\x00" in value: + raise ValueError(f"{name} must not contain null bytes") + if len(value) > max_len: + raise ValueError(f"{name} must be <= {max_len} chars") + return value + + +def parse_edit_subject_target(*, subject: str, target: str) -> tuple[str, str]: + """Validate the (subject, target) pair for an edit. + + The subject is the prefix sentence (e.g. "Paris is the capital of + France"); the target is the new completion (e.g. "Lyon"). Both + fields are length-capped + null-byte-rejected. Returns the + validated pair unchanged. + """ + s = _validate_text_field(subject, "subject", _MAX_SUBJECT_LEN) + t = _validate_text_field(target, "target", _MAX_TARGET_LEN) + return s, t + + +def _validate_layer(value: Optional[object]) -> Optional[int]: + if value is None: + return None + if isinstance(value, bool): + raise TypeError("layer must not be bool") + if not isinstance(value, int): + raise TypeError( + f"layer must be int, got {type(value).__name__}" + ) + if value < 0: + raise ValueError(f"layer must be >= 0, got {value}") + if value > _MAX_LAYER_IDX: + raise ValueError( + f"layer must be <= {_MAX_LAYER_IDX}, got {value}" + ) + return value + + +@dataclass(frozen=True) +class EditRequest: + """Operator-supplied request for a single knowledge edit.""" + + base: str + method: str + subject: str + target: str + layer: Optional[int] + + +@dataclass(frozen=True) +class EditPlan: + """Resolved + validated edit plan ready for ``apply_edit``. + + Differs from :class:`EditRequest` in that ``method`` is canonical + (lowercase) and ``layer`` is always concretely set (defaulted to + the spec's recommended layer when ``None`` was supplied). + """ + + base: str + method: str + subject: str + target: str + layer: int + spec: EditMethodSpec + + def __post_init__(self) -> None: + # Re-validate so callers that bypass build_edit_plan can't + # smuggle in an inconsistent plan. + if self.method not in SUPPORTED_EDIT_METHODS: + raise ValueError( + f"method must be in {sorted(SUPPORTED_EDIT_METHODS)}, " + f"got {self.method!r}" + ) + if not isinstance(self.layer, int) or isinstance(self.layer, bool): + raise TypeError("layer must be int (not bool)") + if self.layer < 0 or self.layer > _MAX_LAYER_IDX: + raise ValueError( + f"layer must be in [0, {_MAX_LAYER_IDX}], got {self.layer}" + ) + + +# Per-method default edit layer (heuristic — operator can override via +# CLI). ROME papers target the middle-to-late MLP layer; AlphaEdit +# follows the same convention. MEMIT updates a range so we treat the +# "layer" arg as the centre of the spread. +_DEFAULT_EDIT_LAYER: Mapping[str, int] = MappingProxyType({ + "rome": 5, + "memit": 8, + "alphaedit": 5, +}) + + +def build_edit_plan( + *, + base: str, + method: str, + subject: str, + target: str, + layer: Optional[int] = None, +) -> EditPlan: + """Resolve + validate an edit request into an :class:`EditPlan`. + + Validates every operator-supplied field at construction time so + misconfigured edits fail fast (mirrors v0.50.0 build_verdict / + v0.55.0 ``design_evals_from_data`` policy). + """ + canonical_base = _validate_text_field(base, "base", _MAX_BASE_LEN) + canonical_method = validate_edit_method(method) + canonical_subject, canonical_target = parse_edit_subject_target( + subject=subject, target=target, + ) + canonical_layer = _validate_layer(layer) + if canonical_layer is None: + canonical_layer = _DEFAULT_EDIT_LAYER[canonical_method] + spec = _EDIT_METHOD_METADATA[canonical_method] + return EditPlan( + base=canonical_base, + method=canonical_method, + subject=canonical_subject, + target=canonical_target, + layer=canonical_layer, + spec=spec, + ) + + +def apply_edit(plan: EditPlan) -> None: + """Apply a knowledge edit — deferred to v0.61.1. + + Re-validates the method so callers passing a bare-class duck-typed + plan (no ``EditPlan``) still hit a meaningful error before the + deferred-live raise. Mirrors v0.50.0 ``apply_variant_loss`` policy. + """ + method_attr = getattr(plan, "method", None) + canonical = validate_edit_method(method_attr) + raise NotImplementedError( + f"apply_edit(method={canonical!r}) is deferred to v0.61.1. " + "Schema accepts the request now so YAML / CLI invocations are " + "stable, but ROME / MEMIT / AlphaEdit live kernels land in " + "v0.61.1." + ) + + +def get_edit_method_spec(name: str) -> EditMethodSpec: + """Return the frozen :class:`EditMethodSpec` for ``name`` or raise.""" + canonical = validate_edit_method(name) + return _EDIT_METHOD_METADATA[canonical] diff --git a/soup_cli/utils/unlearning.py b/soup_cli/utils/unlearning.py new file mode 100644 index 0000000..c881989 --- /dev/null +++ b/soup_cli/utils/unlearning.py @@ -0,0 +1,207 @@ +"""v0.61.0 Part A — Unlearning method allowlist + compat gate. + +Three method backends for the new ``task='unlearn'`` trainer: + +* ``npo`` — Negative Preference Optimization (Zhang et al., 2024). Pushes + the model away from the forget set via a DPO-shaped negative-only loss. +* ``simnpo`` — SimNPO (length-normalised NPO). Removes the reference model + and uses a SimPO-style normalisation to stabilise long-sequence + unlearning. +* ``rmu`` — Representation Misdirection Unlearning (Li et al., 2024). + Adds a noise vector to the residual stream for forget-set inputs while + preserving retain-set activations. + +Schema-only release: validators here are reused by the SoupConfig +cross-validator, while the live trainer wrapper ships in v0.61.1 +(mirrors v0.50.0 GRPO Plus / v0.52.0 Modality II / v0.53.0 Quant Menu II +stub-then-live pattern). +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from types import MappingProxyType +from typing import Mapping + +SUPPORTED_UNLEARN_METHODS: frozenset[str] = frozenset({"npo", "simnpo", "rmu"}) + +_MAX_METHOD_LEN: int = 32 + + +@dataclass(frozen=True) +class UnlearnMethodSpec: + """Metadata for an unlearning method backend. Frozen so callers cannot mutate.""" + + name: str + description: str + needs_retain_set: bool + needs_reference_model: bool + live_wired: bool + + +_UNLEARN_METHOD_METADATA: Mapping[str, UnlearnMethodSpec] = MappingProxyType({ + "npo": UnlearnMethodSpec( + name="npo", + description=( + "Negative Preference Optimization — DPO-shaped loss that " + "pushes the model away from the forget set while a retain " + "set keeps general capability stable." + ), + needs_retain_set=True, + needs_reference_model=True, + live_wired=False, + ), + "simnpo": UnlearnMethodSpec( + name="simnpo", + description=( + "SimNPO — length-normalised NPO without a reference model. " + "Faster + more stable on long sequences (Liu et al., 2024)." + ), + needs_retain_set=True, + needs_reference_model=False, + live_wired=False, + ), + "rmu": UnlearnMethodSpec( + name="rmu", + description=( + "Representation Misdirection Unlearning — adds a noise " + "vector to the residual stream for forget inputs while " + "preserving retain activations (Li et al., 2024)." + ), + needs_retain_set=True, + needs_reference_model=False, + live_wired=False, + ), +}) + + +def validate_unlearn_method(value: object) -> str: + """Normalise + validate an unlearn-method name. + + Returns the canonical (lowercase) form. Mirrors v0.41.0 + ``validate_optimizer_name`` / v0.51.0 ``validate_hub_name`` policy: + bool-rejected, null-byte-rejected, oversize-rejected, + case-insensitive normalisation, unknown rejected with friendly + actionable message. + """ + if isinstance(value, bool): + raise TypeError( + f"unlearn_method must not be bool, got {value!r}" + ) + if not isinstance(value, str): + raise TypeError( + f"unlearn_method must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError("unlearn_method must be non-empty") + if "\x00" in value: + raise ValueError("unlearn_method must not contain null bytes") + if len(value) > _MAX_METHOD_LEN: + raise ValueError( + f"unlearn_method must be <= {_MAX_METHOD_LEN} chars" + ) + canonical = value.lower() + if canonical not in SUPPORTED_UNLEARN_METHODS: + supported = ", ".join(sorted(SUPPORTED_UNLEARN_METHODS)) + raise ValueError( + f"unknown unlearn method {value!r}; supported: {supported}" + ) + return canonical + + +def get_unlearn_method_spec(name: str) -> UnlearnMethodSpec: + """Return the frozen :class:`UnlearnMethodSpec` for ``name`` or raise.""" + canonical = validate_unlearn_method(name) + return _UNLEARN_METHOD_METADATA[canonical] + + +def validate_unlearn_alpha(value: object) -> float: + """Validate an ``unlearn_alpha`` retain-set weight (forget vs retain mixing). + + Bool-rejected, NaN/Inf-rejected, bounded ``[0.0, 10.0]``. Default + interpretation: 0 = pure forget loss, higher values increasingly + favour the retain set. The upper bound is a sanity cap — operators + above 10 should rebalance their dataset instead. + """ + if isinstance(value, bool): + raise TypeError( + f"unlearn_alpha must not be bool, got {value!r}" + ) + if not isinstance(value, (int, float)): + raise TypeError( + f"unlearn_alpha must be a number, got {type(value).__name__}" + ) + fval = float(value) + if not math.isfinite(fval): + raise ValueError("unlearn_alpha must be finite (no NaN / Inf)") + if fval < 0.0: + raise ValueError(f"unlearn_alpha must be >= 0.0, got {fval}") + if fval > 10.0: + raise ValueError(f"unlearn_alpha must be <= 10.0, got {fval}") + return fval + + +def validate_unlearn_compat(*, task: str, backend: str) -> None: + """Schema-time gate for ``task='unlearn'``. + + Rejects: + - non-string / bool args (defence-in-depth, mirrors v0.52.0 + ``validate_classifier_compat`` policy). + - non-unlearn task. + - ``backend == 'mlx'`` (no MLX unlearn path in v0.61.0). + + Multi-method specifics (forget_set required, retain_set conditional) + are enforced at the SoupConfig cross-validator level so the schema + can surface a single composite error. + """ + for name, value in (("task", task), ("backend", backend)): + if isinstance(value, bool): + raise TypeError(f"{name} must not be bool, got {value!r}") + if not isinstance(value, str): + raise TypeError( + f"{name} must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError(f"{name} must be non-empty") + if "\x00" in value: + raise ValueError(f"{name} must not contain null bytes") + if task != "unlearn": + raise ValueError( + f"validate_unlearn_compat called with task={task!r}; " + f"expected task='unlearn'" + ) + if backend == "mlx": + raise ValueError( + "task='unlearn' is not supported on backend=mlx in v0.61.0 " + "(deferred to v0.61.1)" + ) + + +def apply_unlearn_loss(method: str) -> None: + """Compute the per-method unlearn loss — deferred to v0.61.1. + + Validates the method name first so the deferred-live error + distinguishes between "unknown method" and "method is on the + allowlist but not yet wired". Mirrors v0.50.0 ``apply_variant_loss`` + stub-then-live policy. + """ + canonical = validate_unlearn_method(method) + raise NotImplementedError( + f"apply_unlearn_loss({canonical!r}) is deferred to v0.61.1. " + "Schema accepts the method now so callers can write soup.yaml " + "today, but the live NPO / SimNPO / RMU loss kernels land in " + "v0.61.1." + ) + + +def build_unlearn_trainer(config: object, **kwargs: object) -> object: + """Live unlearn trainer factory — deferred stub. + + Lazy-imports the wrapper so the schema-only import path never pulls + in the heavy ``transformers`` / ``peft`` surface. The wrapper itself + is a stub that raises ``NotImplementedError`` on ``setup()``. + """ + from soup_cli.trainer.unlearn import UnlearnTrainerWrapper + + return UnlearnTrainerWrapper(config, **kwargs) # type: ignore[arg-type] diff --git a/soup_cli/utils/unlearning_eval.py b/soup_cli/utils/unlearning_eval.py new file mode 100644 index 0000000..0da38a4 --- /dev/null +++ b/soup_cli/utils/unlearning_eval.py @@ -0,0 +1,520 @@ +"""v0.61.0 Part B — Unlearning eval suite (TOFU / MUSE / WMDP). + +Scores three orthogonal axes after running a ``task='unlearn'`` job: + +* **Forget Quality** — does the model still produce the unlearned + content? Computed from pre/post loss on the forget set (high + post-loss = good forgetting). +* **Model Utility** — does general capability survive? Computed from + the retain-set accuracy delta. +* **PrivLeak** — can a membership-inference adversary still distinguish + forget-set rows from a held-out cohort? Scored from MIA AUC; AUC ≈ 0.5 + is best (no leak). + +Verdicts follow the project's OK / MINOR / MAJOR taxonomy (same +thresholds as v0.26.0 Part D Quant-Lobotomy + v0.56.0 diagnose). + +Live-model evaluation hooks are deferred to v0.61.1 — this module ships +pure-Python kernels + a frozen ``UnlearnReport`` + bundled TOFU fixture. +Operators can supply pre-computed ``evidence`` JSON to drive the +classifier today. +""" + +from __future__ import annotations + +import json +import math +import os +import stat +from dataclasses import dataclass +from pathlib import Path +from types import MappingProxyType +from typing import Any, Mapping, Optional, Tuple + +BENCHMARKS: frozenset[str] = frozenset({"tofu", "muse", "wmdp"}) +VERDICTS: Tuple[str, ...] = ("OK", "MINOR", "MAJOR") + +_OK_THRESHOLD: float = 0.85 +_MINOR_THRESHOLD: float = 0.60 + +_MAX_BENCHMARK_LEN: int = 64 +_MAX_RUN_ID_LEN: int = 128 +_MAX_EVIDENCE_BYTES: int = 16 * 1024 * 1024 # 16 MiB + +# Forget Quality saturation point (post - pre loss above which we +# award the full 1.0 score). 2.0 nats ≈ 7.4x perplexity blow-up — +# strong enough that legitimate "still remembers" cases land in MAJOR. +_FORGET_SATURATION: float = 2.0 + +_METRIC_NAMES: Tuple[str, ...] = ("forget_quality", "model_utility", "priv_leak") + + +_BENCHMARK_METADATA: Mapping[str, Mapping[str, str]] = MappingProxyType({ + "tofu": MappingProxyType({ + "description": ( + "TOFU — synthetic author profiles for forget-set unlearning " + "(Maini et al., 2024)." + ), + "fixture": "tofu_demo.jsonl", + }), + "muse": MappingProxyType({ + "description": ( + "MUSE — real-world books / news corpora with paired retain " + "sets (Shi et al., 2024)." + ), + "fixture": "", # not bundled; loader expected in v0.61.1 + }), + "wmdp": MappingProxyType({ + "description": ( + "WMDP — hazardous-knowledge unlearning across biology / " + "cyber / chemistry (Li et al., 2024)." + ), + "fixture": "", + }), +}) + + +def validate_benchmark_name(value: object) -> str: + """Normalise + validate a benchmark name (case-insensitive).""" + if isinstance(value, bool): + raise TypeError("benchmark must not be bool") + if not isinstance(value, str): + raise TypeError( + f"benchmark must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError("benchmark must be non-empty") + if "\x00" in value: + raise ValueError("benchmark must not contain null bytes") + if len(value) > _MAX_BENCHMARK_LEN: + raise ValueError( + f"benchmark must be <= {_MAX_BENCHMARK_LEN} chars" + ) + canonical = value.lower() + if canonical not in BENCHMARKS: + supported = ", ".join(sorted(BENCHMARKS)) + raise ValueError( + f"unknown benchmark {value!r}; supported: {supported}" + ) + return canonical + + +def _validate_run_id(value: object) -> str: + if isinstance(value, bool): + raise TypeError("run_id must not be bool") + if not isinstance(value, str): + raise TypeError( + f"run_id must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError("run_id must be non-empty") + if "\x00" in value: + raise ValueError("run_id must not contain null bytes") + if len(value) > _MAX_RUN_ID_LEN: + raise ValueError(f"run_id must be <= {_MAX_RUN_ID_LEN} chars") + return value + + +def classify_unlearn_score(score: float) -> str: + """Map ``score in [0, 1]`` to OK / MINOR / MAJOR verdict.""" + if isinstance(score, bool): + raise TypeError("score must be float, not bool") + if not isinstance(score, (int, float)): + raise TypeError( + f"score must be float, got {type(score).__name__}" + ) + value = float(score) + if not math.isfinite(value): + raise ValueError("score must be finite (no NaN / Inf)") + if not 0.0 <= value <= 1.0: + raise ValueError(f"score must be in [0, 1], got {value}") + if value >= _OK_THRESHOLD: + return "OK" + if value >= _MINOR_THRESHOLD: + return "MINOR" + return "MAJOR" + + +def _check_finite_non_negative(value: object, name: str) -> float: + if isinstance(value, bool): + raise TypeError(f"{name} must not be bool") + if not isinstance(value, (int, float)): + raise TypeError( + f"{name} must be a number, got {type(value).__name__}" + ) + fval = float(value) + if not math.isfinite(fval): + raise ValueError(f"{name} must be finite (no NaN / Inf)") + if fval < 0.0: + raise ValueError(f"{name} must be >= 0, got {fval}") + return fval + + +def _check_unit_interval(value: object, name: str) -> float: + if isinstance(value, bool): + raise TypeError(f"{name} must not be bool") + if not isinstance(value, (int, float)): + raise TypeError( + f"{name} must be a number, got {type(value).__name__}" + ) + fval = float(value) + if not math.isfinite(fval): + raise ValueError(f"{name} must be finite (no NaN / Inf)") + if not 0.0 <= fval <= 1.0: + raise ValueError(f"{name} must be in [0, 1], got {fval}") + return fval + + +def compute_forget_quality(*, pre_loss: float, post_loss: float) -> float: + """Forget Quality ∈ ``[0, 1]`` from pre/post loss on the forget set. + + Higher post-unlearn loss → more forgetting. Linearly ramps from 0 + (post == pre) to 1.0 at ``post >= pre + _FORGET_SATURATION``; values + beyond saturate at 1.0. The arbitrary tail prevents the score from + being driven by a single divergent prompt. + """ + pre = _check_finite_non_negative(pre_loss, "pre_loss") + post = _check_finite_non_negative(post_loss, "post_loss") + delta = post - pre + if delta <= 0.0: + return 0.0 + return min(1.0, delta / _FORGET_SATURATION) + + +def compute_model_utility(*, pre_acc: float, post_acc: float) -> float: + """Model Utility ∈ ``[0, 1]`` from retain-set accuracy delta. + + 1.0 = no degradation (post >= pre), 0.0 = full collapse (post=0). + Linear ratio so the score is interpretable as "% capability + retained". + """ + pre = _check_unit_interval(pre_acc, "pre_acc") + post = _check_unit_interval(post_acc, "post_acc") + if pre <= 0.0: + return 1.0 if post >= pre else 0.0 + if post >= pre: + return 1.0 + return max(0.0, post / pre) + + +def compute_priv_leak(*, mia_auc: float) -> float: + """Privacy score ∈ ``[0, 1]`` from membership-inference AUC. + + AUC ≈ 0.5 → no leak → score 1.0. AUC ≥ 0.9 or ≤ 0.1 → full leak → + score 0.0. Symmetric around 0.5 so an adversary who can invert + (AUC < 0.5) is still flagged. + """ + auc = _check_unit_interval(mia_auc, "mia_auc") + distance = abs(auc - 0.5) * 2.0 # 0 → no leak, 1 → max distinguishable + score = 1.0 - distance + return max(0.0, min(1.0, score)) + + +@dataclass(frozen=True) +class UnlearnMetric: + """A single named metric with verdict + free-text evidence.""" + + name: str + score: float + verdict: str + evidence: str + + def __post_init__(self) -> None: + if not isinstance(self.name, str) or not self.name: + raise ValueError("name must be non-empty str") + if self.name not in _METRIC_NAMES: + raise ValueError( + f"name must be one of {_METRIC_NAMES}, got {self.name!r}" + ) + expected = classify_unlearn_score(self.score) + if not isinstance(self.verdict, str) or self.verdict not in VERDICTS: + raise ValueError( + f"verdict must be one of {VERDICTS}, got {self.verdict!r}" + ) + if self.verdict != expected: + raise ValueError( + f"verdict {self.verdict!r} disagrees with score " + f"{self.score} (expected {expected!r})" + ) + if not isinstance(self.evidence, str): + raise TypeError("evidence must be str") + + +def _overall_verdict(verdicts: Tuple[str, ...]) -> str: + """Worst-case across metric verdicts: MAJOR > MINOR > OK.""" + if "MAJOR" in verdicts: + return "MAJOR" + if "MINOR" in verdicts: + return "MINOR" + return "OK" + + +@dataclass(frozen=True) +class UnlearnReport: + """Frozen report card for a single ``soup eval unlearning`` run.""" + + run_id: str + benchmark: str + metrics: Tuple[UnlearnMetric, ...] + overall: str + soup_version: str + + def __post_init__(self) -> None: + _validate_run_id(self.run_id) + if self.benchmark not in BENCHMARKS: + raise ValueError( + f"benchmark must be in {sorted(BENCHMARKS)}, got " + f"{self.benchmark!r}" + ) + if not isinstance(self.metrics, tuple): + raise TypeError("metrics must be a tuple of UnlearnMetric") + for m in self.metrics: + if not isinstance(m, UnlearnMetric): + raise TypeError( + "metrics entries must be UnlearnMetric instances" + ) + if self.overall not in VERDICTS: + raise ValueError( + f"overall must be one of {VERDICTS}, got {self.overall!r}" + ) + # overall must match worst-case + expected_overall = _overall_verdict(tuple(m.verdict for m in self.metrics)) + if self.overall != expected_overall: + raise ValueError( + f"overall {self.overall!r} disagrees with worst metric " + f"verdict {expected_overall!r}" + ) + if not isinstance(self.soup_version, str) or not self.soup_version: + raise ValueError("soup_version must be non-empty str") + + def to_dict(self) -> dict: + return { + "run_id": self.run_id, + "benchmark": self.benchmark, + "metrics": [ + { + "name": m.name, + "score": m.score, + "verdict": m.verdict, + "evidence": m.evidence, + } + for m in self.metrics + ], + "overall": self.overall, + "soup_version": self.soup_version, + } + + +def _neutral_metric(name: str) -> UnlearnMetric: + """Neutral OK metric used when evidence is missing for a probe. + + Matches v0.56.0 diagnose ``neutral_score`` policy: missing evidence + is reported as a 1.0 OK score with an explicit "no evidence + supplied" rationale so the operator sees that the probe did not run. + """ + return UnlearnMetric( + name=name, + score=1.0, + verdict="OK", + evidence="no evidence supplied; score is neutral default", + ) + + +def _build_forget_metric(evidence: Optional[Mapping[str, Any]]) -> UnlearnMetric: + """Build the forget_quality metric. + + Distinguishes 'missing evidence' (neutral OK score) from + 'present-but-invalid evidence' (re-raise as ValueError so the CLI + surfaces it). Review HIGH H3 fix mirrors v0.56.0 diagnose evidence + semantics — silent OK on invalid inputs hid genuine probe failures. + """ + if not isinstance(evidence, Mapping): + return _neutral_metric("forget_quality") + if "pre_loss" not in evidence or "post_loss" not in evidence: + return _neutral_metric("forget_quality") + # Present but invalid -> raise loudly. + score = compute_forget_quality( + pre_loss=evidence["pre_loss"], # type: ignore[arg-type] + post_loss=evidence["post_loss"], # type: ignore[arg-type] + ) + pre = float(evidence["pre_loss"]) + post = float(evidence["post_loss"]) + return UnlearnMetric( + name="forget_quality", + score=score, + verdict=classify_unlearn_score(score), + evidence=f"pre_loss={pre:.4f}, post_loss={post:.4f}", + ) + + +def _build_utility_metric(evidence: Optional[Mapping[str, Any]]) -> UnlearnMetric: + """Build the model_utility metric. Same missing-vs-invalid policy as forget.""" + if not isinstance(evidence, Mapping): + return _neutral_metric("model_utility") + if "pre_acc" not in evidence or "post_acc" not in evidence: + return _neutral_metric("model_utility") + score = compute_model_utility( + pre_acc=evidence["pre_acc"], # type: ignore[arg-type] + post_acc=evidence["post_acc"], # type: ignore[arg-type] + ) + pre = float(evidence["pre_acc"]) + post = float(evidence["post_acc"]) + return UnlearnMetric( + name="model_utility", + score=score, + verdict=classify_unlearn_score(score), + evidence=f"pre_acc={pre:.4f}, post_acc={post:.4f}", + ) + + +def _build_priv_leak_metric(evidence: Optional[Mapping[str, Any]]) -> UnlearnMetric: + """Build the priv_leak metric. Same missing-vs-invalid policy as forget.""" + if not isinstance(evidence, Mapping): + return _neutral_metric("priv_leak") + if "mia_auc" not in evidence: + return _neutral_metric("priv_leak") + score = compute_priv_leak(mia_auc=evidence["mia_auc"]) # type: ignore[arg-type] + auc = float(evidence["mia_auc"]) + return UnlearnMetric( + name="priv_leak", + score=score, + verdict=classify_unlearn_score(score), + evidence=f"mia_auc={auc:.4f}", + ) + + +def run_unlearn_eval( + *, + run_id: str, + benchmark: str, + evidence: Optional[Mapping[str, Mapping[str, Any]]] = None, + soup_version: Optional[str] = None, +) -> UnlearnReport: + """Build an :class:`UnlearnReport` from optional pre-computed evidence. + + Missing evidence per-metric falls through to ``_neutral_metric`` + (OK score, explicit "no evidence supplied" rationale). This mirrors + v0.56.0 ``soup diagnose`` policy. + """ + bench = validate_benchmark_name(benchmark) + _validate_run_id(run_id) + if evidence is None: + evidence = {} + if not isinstance(evidence, Mapping): + raise TypeError("evidence must be a Mapping or None") + + forget = _build_forget_metric(evidence.get("forget_quality")) + utility = _build_utility_metric(evidence.get("model_utility")) + priv = _build_priv_leak_metric(evidence.get("priv_leak")) + metrics = (forget, utility, priv) + overall = _overall_verdict(tuple(m.verdict for m in metrics)) + + if soup_version is None: + from soup_cli import __version__ + + soup_version = __version__ + + return UnlearnReport( + run_id=run_id, + benchmark=bench, + metrics=metrics, + overall=overall, + soup_version=soup_version, + ) + + +def write_unlearn_report(report: UnlearnReport, path: str) -> str: + """Atomic write of the report JSON. Cwd-contained + symlink-rejected. + + Mirrors v0.56.0 diagnose ``write_report`` / v0.59.0 ``atomic_write_text`` + policy. + """ + if not isinstance(report, UnlearnReport): + raise TypeError("report must be UnlearnReport") + from soup_cli.utils.paths import atomic_write_text + + payload = json.dumps(report.to_dict(), sort_keys=True, indent=2) + # NOTE: atomic_write_text signature is (text, output_path) — the + # text body comes first. + atomic_write_text(payload, path, field="output") + return path + + +def get_fixture_path(benchmark: str) -> Optional[Path]: + """Return the bundled fixture path for a benchmark, or ``None`` if + not yet bundled. + + TOFU ships in v0.61.0 (synthetic author profile mini-set under + ``soup_cli/data/_fixtures/unlearning/``). MUSE / WMDP loaders are + deferred to v0.61.1. + + Routes through ``importlib.resources`` (review MEDIUM M6 — matches + v0.53.8 #93 `_bundle_source_path` policy; safe under zipapp / + namespace-package installs). + """ + if not isinstance(benchmark, str) or not benchmark: + return None + canonical = benchmark.lower() + if canonical not in BENCHMARKS: + return None + meta = _BENCHMARK_METADATA[canonical] + fixture_name = meta["fixture"] + if not fixture_name: + return None + # Defensive: bake-in fixture name should never contain separators. + if "/" in fixture_name or "\\" in fixture_name: + return None + from importlib.resources import files + + try: + pkg_root = files("soup_cli") + except (ModuleNotFoundError, TypeError): + return None + candidate = Path( + os.path.realpath( + os.path.join(str(pkg_root), "data", "_fixtures", + "unlearning", fixture_name) + ) + ) + if not candidate.is_file(): + return None + # Symlink rejection (TOCTOU defence - matches project policy). + try: + st = os.lstat(candidate) + except OSError: + return None + if stat.S_ISLNK(st.st_mode): + return None + return candidate + + +def load_evidence_file(path: str) -> Mapping[str, Mapping[str, Any]]: + """Load operator-supplied evidence JSON. + + Containment + size cap + symlink rejection. Mirrors v0.56.0 + ``commands/diagnose.py`` evidence loader. + """ + from soup_cli.utils.paths import is_under_cwd + + if not isinstance(path, str) or not path: + raise ValueError("evidence path must be a non-empty string") + if "\x00" in path: + raise ValueError("evidence path must not contain null bytes") + if not is_under_cwd(path): + raise ValueError(f"evidence path must stay under cwd: {path!r}") + real = os.path.realpath(path) + try: + st = os.lstat(real) + except FileNotFoundError as exc: + raise FileNotFoundError(f"evidence file not found: {path!r}") from exc + if stat.S_ISLNK(st.st_mode): + raise ValueError("evidence path must not be a symlink") + if st.st_size > _MAX_EVIDENCE_BYTES: + raise ValueError( + f"evidence file exceeds {_MAX_EVIDENCE_BYTES} bytes" + ) + with open(real, "r", encoding="utf-8") as f: + data = json.load(f) + if not isinstance(data, dict): + raise ValueError("evidence file must contain a JSON object at the root") + return data diff --git a/tests/test_v0610_part_a.py b/tests/test_v0610_part_a.py new file mode 100644 index 0000000..69f7dee --- /dev/null +++ b/tests/test_v0610_part_a.py @@ -0,0 +1,520 @@ +"""Tests for v0.61.0 Part A — `soup unlearn` trainer (NPO / SimNPO / RMU). + +Schema-only release: tests cover the new ``task='unlearn'`` Literal entry, +the ``unlearn_method`` field, the ``forget_set`` / ``retain_set`` data +schema, and the ``UnlearnTrainerWrapper`` stub. Live wiring is deferred +to v0.61.1 (mirrors v0.50.0 / v0.52.0 / v0.53.0 stub-then-live pattern). +""" + +from __future__ import annotations + +import dataclasses + +import pytest + +# ---------- Module surface ---------- + + +class TestModuleSurface: + def test_imports(self): + from soup_cli.utils.unlearning import ( + SUPPORTED_UNLEARN_METHODS, + UnlearnMethodSpec, + apply_unlearn_loss, + build_unlearn_trainer, + get_unlearn_method_spec, + validate_unlearn_method, + ) + assert callable(validate_unlearn_method) + assert callable(get_unlearn_method_spec) + assert callable(apply_unlearn_loss) + assert callable(build_unlearn_trainer) + assert dataclasses.is_dataclass(UnlearnMethodSpec) + assert isinstance(SUPPORTED_UNLEARN_METHODS, frozenset) + + def test_supported_methods_exact(self): + from soup_cli.utils.unlearning import SUPPORTED_UNLEARN_METHODS + + assert SUPPORTED_UNLEARN_METHODS == frozenset({"npo", "simnpo", "rmu"}) + + def test_metadata_mapping_proxy(self): + from types import MappingProxyType + + from soup_cli.utils.unlearning import _UNLEARN_METHOD_METADATA # type: ignore + + assert isinstance(_UNLEARN_METHOD_METADATA, MappingProxyType) + + +# ---------- validate_unlearn_method ---------- + + +class TestValidateUnlearnMethod: + def test_happy_path(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + for name in ("npo", "simnpo", "rmu"): + assert validate_unlearn_method(name) == name + + def test_case_insensitive(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + assert validate_unlearn_method("NPO") == "npo" + assert validate_unlearn_method("SimNPO") == "simnpo" + assert validate_unlearn_method("RMU") == "rmu" + + def test_unknown_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + with pytest.raises(ValueError, match="unknown unlearn method"): + validate_unlearn_method("dpo") + + def test_bool_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + with pytest.raises(TypeError): + validate_unlearn_method(True) + + def test_non_string_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + with pytest.raises(TypeError): + validate_unlearn_method(123) + + def test_empty_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + with pytest.raises(ValueError): + validate_unlearn_method("") + + def test_null_byte_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + with pytest.raises(ValueError, match="null"): + validate_unlearn_method("npo\x00") + + def test_oversize_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_method + + with pytest.raises(ValueError): + validate_unlearn_method("a" * 100) + + +# ---------- get_unlearn_method_spec ---------- + + +class TestGetUnlearnMethodSpec: + def test_happy_path(self): + from soup_cli.utils.unlearning import get_unlearn_method_spec + + spec = get_unlearn_method_spec("npo") + assert spec.name == "npo" + assert isinstance(spec.description, str) and spec.description + assert spec.live_wired is False # deferred to v0.61.1 + + def test_frozen(self): + from soup_cli.utils.unlearning import get_unlearn_method_spec + + spec = get_unlearn_method_spec("npo") + with pytest.raises(dataclasses.FrozenInstanceError): + spec.name = "other" # type: ignore + + def test_unknown_raises(self): + from soup_cli.utils.unlearning import get_unlearn_method_spec + + with pytest.raises(ValueError): + get_unlearn_method_spec("zzz") + + def test_all_three_have_specs(self): + from soup_cli.utils.unlearning import get_unlearn_method_spec + + for name in ("npo", "simnpo", "rmu"): + spec = get_unlearn_method_spec(name) + assert spec.name == name + + +# ---------- apply_unlearn_loss (stub) ---------- + + +class TestApplyUnlearnLoss: + def test_unknown_rejected_before_notimplemented(self): + from soup_cli.utils.unlearning import apply_unlearn_loss + + with pytest.raises(ValueError): + apply_unlearn_loss("zzz") + + def test_deferred_marker_present(self): + from soup_cli.utils.unlearning import apply_unlearn_loss + + with pytest.raises(NotImplementedError, match="v0.61.1"): + apply_unlearn_loss("npo") + + +# ---------- build_unlearn_trainer (stub) ---------- + + +class TestBuildUnlearnTrainer: + def _make_cfg(self): + from soup_cli.config.schema import SoupConfig + + return SoupConfig( + base="test-model", + task="unlearn", + data={ + "train": "test.jsonl", + "forget_set": "f.jsonl", + "retain_set": "r.jsonl", + }, + training={"unlearn_method": "npo"}, + ) + + def test_returns_wrapper_instance(self): + from soup_cli.trainer.unlearn import UnlearnTrainerWrapper + from soup_cli.utils.unlearning import build_unlearn_trainer + + cfg = self._make_cfg() + wrapper = build_unlearn_trainer(cfg) + assert isinstance(wrapper, UnlearnTrainerWrapper) + + def test_kwargs_signature_allows_known(self): + from soup_cli.utils.unlearning import build_unlearn_trainer + + cfg = self._make_cfg() + # Forward-compat kwargs accepted at construction time. Setup + # raises NotImplementedError per v0.61.1 stub contract. + wrapper = build_unlearn_trainer(cfg, device="cpu", trust_remote_code=False) + with pytest.raises(NotImplementedError, match="v0.61.1"): + wrapper.setup() + + def test_invalid_config_rejected(self): + from soup_cli.utils.unlearning import build_unlearn_trainer + + class _Cfg: + pass + + with pytest.raises(AttributeError, match="SoupConfig"): + build_unlearn_trainer(_Cfg()) + + +# ---------- validate_unlearn_compat ---------- + + +class TestValidateUnlearnCompat: + def test_happy_path(self): + from soup_cli.utils.unlearning import validate_unlearn_compat + + # Should not raise + validate_unlearn_compat(task="unlearn", backend="transformers") + + def test_mlx_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_compat + + with pytest.raises(ValueError, match="mlx"): + validate_unlearn_compat(task="unlearn", backend="mlx") + + def test_wrong_task_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_compat + + with pytest.raises(ValueError): + validate_unlearn_compat(task="sft", backend="transformers") + + def test_bool_task_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_compat + + with pytest.raises(TypeError): + validate_unlearn_compat(task=True, backend="transformers") # type: ignore + + def test_bool_backend_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_compat + + with pytest.raises(TypeError): + validate_unlearn_compat(task="unlearn", backend=True) # type: ignore + + def test_empty_task_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_compat + + with pytest.raises(ValueError): + validate_unlearn_compat(task="", backend="transformers") + + def test_null_byte_rejected(self): + from soup_cli.utils.unlearning import validate_unlearn_compat + + with pytest.raises(ValueError, match="null"): + validate_unlearn_compat(task="unlearn", backend="transformers\x00") + + +# ---------- Schema integration: task='unlearn' ---------- + + +class TestSchemaUnlearnTask: + def test_unlearn_task_accepted(self): + from soup_cli.config.schema import SoupConfig + + cfg = SoupConfig( + base="test-model", + task="unlearn", + data={ + "train": "test.jsonl", + "format": "auto", + "forget_set": "forget.jsonl", + "retain_set": "retain.jsonl", + }, + training={"unlearn_method": "npo"}, + ) + assert cfg.task == "unlearn" + assert cfg.training.unlearn_method == "npo" + + def test_unlearn_method_default_none_for_non_unlearn(self): + from soup_cli.config.schema import SoupConfig + + cfg = SoupConfig( + base="test-model", + task="sft", + data={"train": "test.jsonl"}, + ) + assert cfg.training.unlearn_method is None + + def test_unlearn_method_unknown_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import SoupConfig + + with pytest.raises(ValidationError): + SoupConfig( + base="test-model", + task="unlearn", + data={"train": "test.jsonl", "forget_set": "f.jsonl"}, + training={"unlearn_method": "zzz"}, + ) + + def test_unlearn_method_case_insensitive(self): + from soup_cli.config.schema import SoupConfig + + cfg = SoupConfig( + base="test-model", + task="unlearn", + data={ + "train": "test.jsonl", + "forget_set": "f.jsonl", + "retain_set": "r.jsonl", + }, + training={"unlearn_method": "NPO"}, + ) + assert cfg.training.unlearn_method == "npo" + + def test_mlx_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import SoupConfig + + with pytest.raises(ValidationError, match="mlx"): + SoupConfig( + base="test-model", + task="unlearn", + backend="mlx", + data={"train": "test.jsonl", "forget_set": "f.jsonl"}, + training={"unlearn_method": "npo"}, + ) + + def test_unlearn_method_outside_unlearn_task_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import SoupConfig + + with pytest.raises(ValidationError, match="unlearn"): + SoupConfig( + base="test-model", + task="sft", + data={"train": "test.jsonl"}, + training={"unlearn_method": "npo"}, + ) + + def test_forget_set_field(self): + from soup_cli.config.schema import DataConfig + + data = DataConfig(train="test.jsonl", forget_set="forget.jsonl") + assert data.forget_set == "forget.jsonl" + + def test_retain_set_field(self): + from soup_cli.config.schema import DataConfig + + data = DataConfig(train="test.jsonl", retain_set="retain.jsonl") + assert data.retain_set == "retain.jsonl" + + def test_forget_set_null_byte_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import DataConfig + + with pytest.raises(ValidationError): + DataConfig(train="test.jsonl", forget_set="f\x00.jsonl") + + def test_retain_set_oversize_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import DataConfig + + with pytest.raises(ValidationError): + DataConfig(train="test.jsonl", retain_set="x" * 5000) + + def test_unlearn_requires_forget_set(self): + from pydantic import ValidationError + + from soup_cli.config.schema import SoupConfig + + with pytest.raises(ValidationError, match="forget_set"): + SoupConfig( + base="test-model", + task="unlearn", + data={"train": "test.jsonl"}, + training={"unlearn_method": "npo"}, + ) + + def test_unlearn_alpha_field(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(unlearn_method="npo", unlearn_alpha=0.5) + assert tcfg.unlearn_alpha == 0.5 + + def test_unlearn_alpha_bounds(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(unlearn_method="npo", unlearn_alpha=-0.1) + + with pytest.raises(ValidationError): + TrainingConfig(unlearn_method="npo", unlearn_alpha=11.0) + + def test_unlearn_alpha_bool_rejected(self): + from soup_cli.config.schema import TrainingConfig + + # Bool raises TypeError from the validator; Pydantic v2 does not + # wrap TypeError in ValidationError (only ValueError / + # AssertionError / PydanticCustomError). + with pytest.raises(TypeError, match="bool"): + TrainingConfig(unlearn_method="npo", unlearn_alpha=True) + + def test_unlearn_alpha_without_method_rejected(self): + """Review L10 — `unlearn_alpha` without `unlearn_method` is a footgun.""" + from pydantic import ValidationError + + from soup_cli.config.schema import SoupConfig + + with pytest.raises(ValidationError, match="unlearn_method"): + SoupConfig( + base="test-model", + task="sft", + data={"train": "test.jsonl"}, + training={"unlearn_alpha": 0.5}, + ) + + def test_unlearn_alpha_boundary_zero_accepted(self): + """Review L3 — exact lower boundary (0.0).""" + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(unlearn_method="npo", unlearn_alpha=0.0) + assert tcfg.unlearn_alpha == 0.0 + + def test_unlearn_alpha_boundary_ten_accepted(self): + """Review L3 — exact upper boundary (10.0).""" + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(unlearn_method="npo", unlearn_alpha=10.0) + assert tcfg.unlearn_alpha == 10.0 + + +# ---------- Source-grep regression guards (review L7) ---------- + + +class TestSourceWiring: + def test_cli_registers_edit_typer(self): + """cli.py must add `_edit_cmd.app` under name='edit'.""" + from pathlib import Path + + cli_src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py" + text = cli_src.read_text(encoding="utf-8") + assert "_edit_cmd" in text + assert 'name="edit"' in text + + def test_eval_registers_v0610(self): + """eval.py must call register(app, console) from _eval_v0610.""" + from pathlib import Path + + eval_src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" / "commands" / "eval.py" + ) + text = eval_src.read_text(encoding="utf-8") + assert "_register_v0610" in text + + def test_pyproject_includes_unlearning_fixtures(self): + """pyproject.toml artifacts list must include unlearning fixtures.""" + from pathlib import Path + + py = Path(__file__).resolve().parent.parent / "pyproject.toml" + text = py.read_text(encoding="utf-8") + assert "unlearning/*.jsonl" in text + + +# ---------- UnlearnTrainerWrapper ---------- + + +class TestUnlearnTrainerWrapper: + def test_import(self): + from soup_cli.trainer.unlearn import UnlearnTrainerWrapper + + assert UnlearnTrainerWrapper is not None + + def test_train_before_setup_raises(self): + from soup_cli.config.schema import SoupConfig + from soup_cli.trainer.unlearn import UnlearnTrainerWrapper + + cfg = SoupConfig( + base="test-model", + task="unlearn", + data={ + "train": "test.jsonl", + "forget_set": "f.jsonl", + "retain_set": "r.jsonl", + }, + training={"unlearn_method": "npo"}, + ) + wrapper = UnlearnTrainerWrapper(cfg) + with pytest.raises(RuntimeError, match="setup"): + wrapper.train() + + def test_setup_deferred(self): + from soup_cli.config.schema import SoupConfig + from soup_cli.trainer.unlearn import UnlearnTrainerWrapper + + cfg = SoupConfig( + base="test-model", + task="unlearn", + data={ + "train": "test.jsonl", + "forget_set": "f.jsonl", + "retain_set": "r.jsonl", + }, + training={"unlearn_method": "npo"}, + ) + wrapper = UnlearnTrainerWrapper(cfg) + with pytest.raises(NotImplementedError, match="v0.61.1"): + wrapper.setup() + + def test_method_attribute(self): + from soup_cli.config.schema import SoupConfig + from soup_cli.trainer.unlearn import UnlearnTrainerWrapper + + cfg = SoupConfig( + base="test-model", + task="unlearn", + data={ + "train": "test.jsonl", + "forget_set": "f.jsonl", + "retain_set": "r.jsonl", + }, + training={"unlearn_method": "simnpo"}, + ) + wrapper = UnlearnTrainerWrapper(cfg) + assert wrapper.method == "simnpo" diff --git a/tests/test_v0610_part_b.py b/tests/test_v0610_part_b.py new file mode 100644 index 0000000..94c3807 --- /dev/null +++ b/tests/test_v0610_part_b.py @@ -0,0 +1,521 @@ +"""Tests for v0.61.0 Part B — Unlearning eval suite (TOFU / MUSE / WMDP). + +Coverage: +- ``UnlearnMetric`` / ``UnlearnReport`` frozen dataclasses. +- ``classify_unlearn_score`` OK / MINOR / MAJOR thresholds. +- ``compute_forget_quality`` / ``compute_model_utility`` / ``compute_priv_leak`` kernels. +- ``run_unlearn_eval`` orchestrator. +- ``BENCHMARKS`` allowlist + closed name validation. +- ``soup eval unlearning`` CLI smoke. +""" + +from __future__ import annotations + +import dataclasses +import json +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app + +# ---------- Module surface ---------- + + +class TestModuleSurface: + def test_imports(self): + from soup_cli.utils.unlearning_eval import ( + BENCHMARKS, + VERDICTS, + UnlearnMetric, + UnlearnReport, + classify_unlearn_score, + compute_forget_quality, + compute_model_utility, + compute_priv_leak, + run_unlearn_eval, + validate_benchmark_name, + ) + assert callable(classify_unlearn_score) + assert callable(compute_forget_quality) + assert callable(compute_model_utility) + assert callable(compute_priv_leak) + assert callable(run_unlearn_eval) + assert callable(validate_benchmark_name) + assert dataclasses.is_dataclass(UnlearnMetric) + assert dataclasses.is_dataclass(UnlearnReport) + assert isinstance(BENCHMARKS, frozenset) + assert isinstance(VERDICTS, tuple) + + def test_benchmarks_exact(self): + from soup_cli.utils.unlearning_eval import BENCHMARKS + + assert BENCHMARKS == frozenset({"tofu", "muse", "wmdp"}) + + def test_verdicts_exact(self): + from soup_cli.utils.unlearning_eval import VERDICTS + + assert VERDICTS == ("OK", "MINOR", "MAJOR") + + +# ---------- classify_unlearn_score ---------- + + +class TestClassifyUnlearnScore: + def test_ok_boundary(self): + from soup_cli.utils.unlearning_eval import classify_unlearn_score + + assert classify_unlearn_score(0.85) == "OK" + assert classify_unlearn_score(1.0) == "OK" + + def test_minor_band(self): + from soup_cli.utils.unlearning_eval import classify_unlearn_score + + assert classify_unlearn_score(0.60) == "MINOR" + assert classify_unlearn_score(0.84) == "MINOR" + + def test_major_band(self): + from soup_cli.utils.unlearning_eval import classify_unlearn_score + + assert classify_unlearn_score(0.0) == "MAJOR" + assert classify_unlearn_score(0.59) == "MAJOR" + + def test_bool_rejected(self): + from soup_cli.utils.unlearning_eval import classify_unlearn_score + + with pytest.raises(TypeError): + classify_unlearn_score(True) + + def test_non_finite_rejected(self): + from soup_cli.utils.unlearning_eval import classify_unlearn_score + + with pytest.raises(ValueError): + classify_unlearn_score(float("nan")) + + with pytest.raises(ValueError): + classify_unlearn_score(float("inf")) + + def test_out_of_range_rejected(self): + from soup_cli.utils.unlearning_eval import classify_unlearn_score + + with pytest.raises(ValueError): + classify_unlearn_score(-0.1) + + with pytest.raises(ValueError): + classify_unlearn_score(1.1) + + +# ---------- Benchmark name validation ---------- + + +class TestValidateBenchmarkName: + def test_happy_path(self): + from soup_cli.utils.unlearning_eval import validate_benchmark_name + + assert validate_benchmark_name("tofu") == "tofu" + assert validate_benchmark_name("MUSE") == "muse" + assert validate_benchmark_name("WMDP") == "wmdp" + + def test_unknown_rejected(self): + from soup_cli.utils.unlearning_eval import validate_benchmark_name + + with pytest.raises(ValueError, match="unknown"): + validate_benchmark_name("zzz") + + def test_bool_rejected(self): + from soup_cli.utils.unlearning_eval import validate_benchmark_name + + with pytest.raises(TypeError): + validate_benchmark_name(True) + + def test_null_byte_rejected(self): + from soup_cli.utils.unlearning_eval import validate_benchmark_name + + with pytest.raises(ValueError): + validate_benchmark_name("tofu\x00") + + def test_oversize_rejected(self): + from soup_cli.utils.unlearning_eval import validate_benchmark_name + + with pytest.raises(ValueError): + validate_benchmark_name("a" * 100) + + def test_empty_rejected(self): + from soup_cli.utils.unlearning_eval import validate_benchmark_name + + with pytest.raises(ValueError): + validate_benchmark_name("") + + +# ---------- Metric kernels ---------- + + +class TestComputeForgetQuality: + def test_perfect_forget(self): + # If post-unlearn loss on forget set is HIGH and pre was LOW, + # forget quality = 1.0. + from soup_cli.utils.unlearning_eval import compute_forget_quality + + score = compute_forget_quality(pre_loss=0.5, post_loss=5.0) + assert score == 1.0 + + def test_no_forget(self): + # If post-loss == pre-loss, quality is 0. + from soup_cli.utils.unlearning_eval import compute_forget_quality + + score = compute_forget_quality(pre_loss=2.0, post_loss=2.0) + assert score == 0.0 + + def test_partial_forget(self): + from soup_cli.utils.unlearning_eval import compute_forget_quality + + score = compute_forget_quality(pre_loss=1.0, post_loss=2.0) + assert 0.0 < score < 1.0 + + def test_bool_rejected(self): + from soup_cli.utils.unlearning_eval import compute_forget_quality + + with pytest.raises(TypeError): + compute_forget_quality(pre_loss=True, post_loss=2.0) + + with pytest.raises(TypeError): + compute_forget_quality(pre_loss=1.0, post_loss=True) + + def test_non_finite_rejected(self): + from soup_cli.utils.unlearning_eval import compute_forget_quality + + with pytest.raises(ValueError): + compute_forget_quality(pre_loss=float("nan"), post_loss=1.0) + + def test_negative_loss_rejected(self): + from soup_cli.utils.unlearning_eval import compute_forget_quality + + with pytest.raises(ValueError): + compute_forget_quality(pre_loss=-0.5, post_loss=1.0) + + +class TestComputeModelUtility: + def test_perfect_utility(self): + # If retain accuracy is preserved (post == pre), utility = 1.0. + from soup_cli.utils.unlearning_eval import compute_model_utility + + score = compute_model_utility(pre_acc=0.8, post_acc=0.8) + assert score == 1.0 + + def test_no_utility(self): + # If retain accuracy drops to 0, utility = 0. + from soup_cli.utils.unlearning_eval import compute_model_utility + + score = compute_model_utility(pre_acc=0.8, post_acc=0.0) + assert score == 0.0 + + def test_partial_drop(self): + from soup_cli.utils.unlearning_eval import compute_model_utility + + score = compute_model_utility(pre_acc=0.8, post_acc=0.6) + assert 0.0 < score < 1.0 + + def test_bool_rejected(self): + from soup_cli.utils.unlearning_eval import compute_model_utility + + with pytest.raises(TypeError): + compute_model_utility(pre_acc=True, post_acc=0.5) + + def test_out_of_range_rejected(self): + from soup_cli.utils.unlearning_eval import compute_model_utility + + with pytest.raises(ValueError): + compute_model_utility(pre_acc=1.5, post_acc=0.5) + + with pytest.raises(ValueError): + compute_model_utility(pre_acc=0.5, post_acc=-0.1) + + +class TestComputePrivLeak: + def test_no_leak(self): + # Membership-inference AUC ≈ 0.5 → no leak. + from soup_cli.utils.unlearning_eval import compute_priv_leak + + score = compute_priv_leak(mia_auc=0.5) + assert score >= 0.95 # very high "privacy preserved" + + def test_full_leak(self): + # MIA AUC = 1.0 → adversary can perfectly distinguish forget vs holdout. + from soup_cli.utils.unlearning_eval import compute_priv_leak + + score = compute_priv_leak(mia_auc=1.0) + assert score == 0.0 + + def test_below_random(self): + # AUC < 0.5 is still leak (adversary can invert). + from soup_cli.utils.unlearning_eval import compute_priv_leak + + score = compute_priv_leak(mia_auc=0.0) + assert score == 0.0 + + def test_bool_rejected(self): + from soup_cli.utils.unlearning_eval import compute_priv_leak + + with pytest.raises(TypeError): + compute_priv_leak(mia_auc=True) + + def test_out_of_range_rejected(self): + from soup_cli.utils.unlearning_eval import compute_priv_leak + + with pytest.raises(ValueError): + compute_priv_leak(mia_auc=1.5) + + def test_boundary_zero_accepted(self): + """Review L3 — exact lower boundary.""" + from soup_cli.utils.unlearning_eval import compute_priv_leak + + # AUC=0.0 is in [0, 1] but reads as max distinguishable inverse. + assert compute_priv_leak(mia_auc=0.0) == 0.0 + + def test_boundary_one_accepted(self): + """Review L3 — exact upper boundary.""" + from soup_cli.utils.unlearning_eval import compute_priv_leak + + assert compute_priv_leak(mia_auc=1.0) == 0.0 + + +# ---------- UnlearnMetric / UnlearnReport ---------- + + +class TestUnlearnMetric: + def test_construct(self): + from soup_cli.utils.unlearning_eval import UnlearnMetric + + m = UnlearnMetric( + name="forget_quality", + score=0.9, + verdict="OK", + evidence="pre=0.5, post=5.0", + ) + assert m.name == "forget_quality" + assert m.score == 0.9 + assert m.verdict == "OK" + + def test_frozen(self): + from soup_cli.utils.unlearning_eval import UnlearnMetric + + m = UnlearnMetric( + name="forget_quality", + score=0.9, + verdict="OK", + evidence="", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + m.score = 0.5 # type: ignore + + def test_verdict_must_match_score(self): + from soup_cli.utils.unlearning_eval import UnlearnMetric + + with pytest.raises(ValueError, match="disagrees"): + UnlearnMetric( + name="forget_quality", + score=0.3, + verdict="OK", + evidence="", + ) + + +class TestUnlearnReport: + def test_construct(self): + from soup_cli.utils.unlearning_eval import UnlearnMetric, UnlearnReport + + report = UnlearnReport( + run_id="test-run", + benchmark="tofu", + metrics=( + UnlearnMetric(name="forget_quality", score=0.9, verdict="OK", evidence=""), + UnlearnMetric(name="model_utility", score=0.95, verdict="OK", evidence=""), + UnlearnMetric(name="priv_leak", score=0.9, verdict="OK", evidence=""), + ), + overall="OK", + soup_version="0.61.0", + ) + assert report.run_id == "test-run" + assert report.benchmark == "tofu" + assert len(report.metrics) == 3 + assert report.overall == "OK" + + def test_frozen(self): + from soup_cli.utils.unlearning_eval import UnlearnMetric, UnlearnReport + + report = UnlearnReport( + run_id="r", + benchmark="tofu", + metrics=( + UnlearnMetric(name="forget_quality", score=0.9, verdict="OK", evidence=""), + ), + overall="OK", + soup_version="0.61.0", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + report.overall = "MAJOR" # type: ignore + + def test_to_dict(self): + from soup_cli.utils.unlearning_eval import UnlearnMetric, UnlearnReport + + report = UnlearnReport( + run_id="r", + benchmark="tofu", + metrics=( + UnlearnMetric(name="forget_quality", score=0.9, verdict="OK", evidence="e"), + ), + overall="OK", + soup_version="0.61.0", + ) + d = report.to_dict() + assert d["run_id"] == "r" + assert d["benchmark"] == "tofu" + assert d["overall"] == "OK" + assert len(d["metrics"]) == 1 + # Round-trip via json + s = json.dumps(d) + assert json.loads(s) == d + + +# ---------- run_unlearn_eval ---------- + + +class TestRunUnlearnEval: + def test_happy_path(self): + from soup_cli.utils.unlearning_eval import run_unlearn_eval + + report = run_unlearn_eval( + run_id="test-run", + benchmark="tofu", + evidence={ + "forget_quality": {"pre_loss": 0.5, "post_loss": 5.0}, + "model_utility": {"pre_acc": 0.8, "post_acc": 0.78}, + "priv_leak": {"mia_auc": 0.51}, + }, + ) + assert report.run_id == "test-run" + assert report.benchmark == "tofu" + assert report.overall in ("OK", "MINOR", "MAJOR") + + def test_unknown_benchmark_rejected(self): + from soup_cli.utils.unlearning_eval import run_unlearn_eval + + with pytest.raises(ValueError, match="unknown"): + run_unlearn_eval(run_id="r", benchmark="zzz", evidence={}) + + def test_missing_evidence_neutral(self): + # Missing-evidence policy: neutral OK score (matches v0.56.0 + # diagnose-runner neutral_score policy). + from soup_cli.utils.unlearning_eval import run_unlearn_eval + + report = run_unlearn_eval(run_id="r", benchmark="tofu", evidence={}) + # All metrics neutral OK + assert report.overall == "OK" + + def test_overall_worst_case(self): + from soup_cli.utils.unlearning_eval import run_unlearn_eval + + report = run_unlearn_eval( + run_id="r", + benchmark="tofu", + evidence={ + "forget_quality": {"pre_loss": 1.0, "post_loss": 1.0}, # MAJOR + "model_utility": {"pre_acc": 0.8, "post_acc": 0.8}, # OK + "priv_leak": {"mia_auc": 0.5}, # OK + }, + ) + assert report.overall == "MAJOR" + + def test_invalid_evidence_raises_loudly(self): + """Review HIGH H3 — present-but-invalid evidence must raise + instead of silently scoring OK.""" + from soup_cli.utils.unlearning_eval import run_unlearn_eval + + with pytest.raises(ValueError): + run_unlearn_eval( + run_id="r", + benchmark="tofu", + evidence={ + "forget_quality": {"pre_loss": -1.0, "post_loss": 1.0}, + }, + ) + + def test_partial_evidence_neutral_on_missing(self): + """Review HIGH H3 — missing keys still produce neutral OK.""" + from soup_cli.utils.unlearning_eval import run_unlearn_eval + + report = run_unlearn_eval( + run_id="r", + benchmark="tofu", + evidence={ + "forget_quality": {"pre_loss": 0.5, "post_loss": 5.0}, + # model_utility + priv_leak missing -> neutral + }, + ) + assert report.overall == "OK" + + +# ---------- CLI ---------- + + +class TestCli: + def test_help(self): + runner = CliRunner() + result = runner.invoke(app, ["eval", "unlearning", "--help"]) + assert result.exit_code == 0, result.output + + def test_unknown_benchmark_rejected(self, tmp_path): + runner = CliRunner() + result = runner.invoke(app, ["eval", "unlearning", "test-run", "--benchmark", "zzz"]) + assert result.exit_code != 0 + assert "unknown" in result.output.lower() or "invalid" in result.output.lower() + + def test_neutral_run(self, tmp_path): + # Without evidence, every metric is neutral OK. + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=tmp_path) as fs: + out = Path(fs) / "report.json" + result = runner.invoke(app, [ + "eval", "unlearning", "test-run", + "--benchmark", "tofu", + "--output", str(out), + ]) + assert result.exit_code == 0, result.output + assert out.exists() + data = json.loads(out.read_text()) + assert data["benchmark"] == "tofu" + assert data["overall"] == "OK" + + def test_outside_cwd_output_rejected(self, tmp_path): + """Review L8 — use tmp_path.parent so path is deterministically + outside the isolated_filesystem on every platform.""" + runner = CliRunner() + # The runner's isolated_filesystem cd's into a fresh subdir under + # tmp_path; pointing --output at tmp_path itself is reliably + # outside the new cwd regardless of OS or symlink layout. + outside_target = str(tmp_path / "evil.json") + with runner.isolated_filesystem(temp_dir=tmp_path): + result = runner.invoke(app, [ + "eval", "unlearning", "test-run", + "--benchmark", "tofu", + "--output", outside_target, + ]) + assert result.exit_code != 0 + + +# ---------- Fixtures ---------- + + +class TestFixtures: + def test_tofu_fixture_exists(self): + from soup_cli.utils.unlearning_eval import get_fixture_path + + # TOFU should be bundled. + p = get_fixture_path("tofu") + assert p is not None + + def test_unknown_fixture_returns_none(self): + from soup_cli.utils.unlearning_eval import get_fixture_path + + assert get_fixture_path("zzz") is None diff --git a/tests/test_v0610_part_c.py b/tests/test_v0610_part_c.py new file mode 100644 index 0000000..246c501 --- /dev/null +++ b/tests/test_v0610_part_c.py @@ -0,0 +1,335 @@ +"""Tests for v0.61.0 Part C — `soup edit set` (ROME / MEMIT / AlphaEdit). + +Coverage: +- ``SUPPORTED_EDIT_METHODS`` + ``validate_edit_method``. +- ``EditRequest`` / ``EditPlan`` frozen dataclasses. +- ``parse_edit_subject_target`` parser (free-text subject + target). +- ``build_edit_plan`` schema-only orchestrator. +- ``apply_edit`` deferred stub. +- ``soup edit set`` CLI smoke. +""" + +from __future__ import annotations + +import dataclasses + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app + + +class TestModuleSurface: + def test_imports(self): + from soup_cli.utils.knowledge_edit import ( + SUPPORTED_EDIT_METHODS, + EditPlan, + EditRequest, + apply_edit, + build_edit_plan, + parse_edit_subject_target, + validate_edit_method, + ) + assert callable(validate_edit_method) + assert callable(parse_edit_subject_target) + assert callable(build_edit_plan) + assert callable(apply_edit) + assert dataclasses.is_dataclass(EditRequest) + assert dataclasses.is_dataclass(EditPlan) + assert isinstance(SUPPORTED_EDIT_METHODS, frozenset) + + def test_supported_methods_exact(self): + from soup_cli.utils.knowledge_edit import SUPPORTED_EDIT_METHODS + + assert SUPPORTED_EDIT_METHODS == frozenset({"rome", "memit", "alphaedit"}) + + +class TestValidateEditMethod: + def test_happy_path(self): + from soup_cli.utils.knowledge_edit import validate_edit_method + + for name in ("rome", "memit", "alphaedit"): + assert validate_edit_method(name) == name + + def test_case_insensitive(self): + from soup_cli.utils.knowledge_edit import validate_edit_method + + assert validate_edit_method("ROME") == "rome" + assert validate_edit_method("MEMIT") == "memit" + + def test_unknown_rejected(self): + from soup_cli.utils.knowledge_edit import validate_edit_method + + with pytest.raises(ValueError, match="unknown"): + validate_edit_method("ftedit") + + def test_bool_rejected(self): + from soup_cli.utils.knowledge_edit import validate_edit_method + + with pytest.raises(TypeError): + validate_edit_method(True) + + def test_non_string_rejected(self): + from soup_cli.utils.knowledge_edit import validate_edit_method + + with pytest.raises(TypeError): + validate_edit_method(42) + + def test_null_byte_rejected(self): + from soup_cli.utils.knowledge_edit import validate_edit_method + + with pytest.raises(ValueError): + validate_edit_method("rome\x00") + + def test_oversize_rejected(self): + from soup_cli.utils.knowledge_edit import validate_edit_method + + with pytest.raises(ValueError): + validate_edit_method("a" * 100) + + +class TestParseEditSubjectTarget: + def test_happy_path(self): + from soup_cli.utils.knowledge_edit import parse_edit_subject_target + + subject, target = parse_edit_subject_target( + subject="Paris is the capital of France", + target="Lyon", + ) + assert subject == "Paris is the capital of France" + assert target == "Lyon" + + def test_empty_subject_rejected(self): + from soup_cli.utils.knowledge_edit import parse_edit_subject_target + + with pytest.raises(ValueError, match="subject"): + parse_edit_subject_target(subject="", target="Lyon") + + def test_empty_target_rejected(self): + from soup_cli.utils.knowledge_edit import parse_edit_subject_target + + with pytest.raises(ValueError, match="target"): + parse_edit_subject_target(subject="Paris is the capital", target="") + + def test_null_byte_rejected(self): + from soup_cli.utils.knowledge_edit import parse_edit_subject_target + + with pytest.raises(ValueError): + parse_edit_subject_target(subject="Paris\x00", target="Lyon") + + def test_oversize_rejected(self): + from soup_cli.utils.knowledge_edit import parse_edit_subject_target + + with pytest.raises(ValueError): + parse_edit_subject_target( + subject="x" * 5000, + target="Lyon", + ) + + def test_bool_rejected(self): + from soup_cli.utils.knowledge_edit import parse_edit_subject_target + + with pytest.raises(TypeError): + parse_edit_subject_target(subject=True, target="Lyon") # type: ignore + + +class TestBuildEditPlan: + def test_happy_path(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + plan = build_edit_plan( + base="meta-llama/Llama-3.1-8B-Instruct", + method="rome", + subject="Paris is the capital of France", + target="Lyon", + ) + assert plan.method == "rome" + assert plan.base == "meta-llama/Llama-3.1-8B-Instruct" + assert plan.subject == "Paris is the capital of France" + assert plan.target == "Lyon" + assert plan.layer is not None + + def test_custom_layer(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + plan = build_edit_plan( + base="meta-llama/Llama-3.1-8B-Instruct", + method="rome", + subject="x", + target="y", + layer=17, + ) + assert plan.layer == 17 + + def test_layer_bool_rejected(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + with pytest.raises(TypeError): + build_edit_plan( + base="b", method="rome", subject="s", target="t", + layer=True, # type: ignore + ) + + def test_layer_negative_rejected(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + with pytest.raises(ValueError): + build_edit_plan( + base="b", method="rome", subject="s", target="t", + layer=-1, + ) + + def test_layer_oversize_rejected(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + with pytest.raises(ValueError): + build_edit_plan( + base="b", method="rome", subject="s", target="t", + layer=10000, + ) + + def test_unknown_method_rejected(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + with pytest.raises(ValueError): + build_edit_plan( + base="b", method="zzz", subject="s", target="t", + ) + + def test_empty_base_rejected(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + with pytest.raises(ValueError): + build_edit_plan( + base="", method="rome", subject="s", target="t", + ) + + def test_null_byte_base_rejected(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + with pytest.raises(ValueError): + build_edit_plan( + base="b\x00", method="rome", subject="s", target="t", + ) + + +class TestEditPlanFrozen: + def test_frozen(self): + from soup_cli.utils.knowledge_edit import build_edit_plan + + plan = build_edit_plan( + base="b", method="rome", subject="s", target="t", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + plan.method = "memit" # type: ignore + + +class TestApplyEdit: + def test_deferred(self): + from soup_cli.utils.knowledge_edit import apply_edit, build_edit_plan + + plan = build_edit_plan( + base="b", method="rome", subject="s", target="t", + ) + with pytest.raises(NotImplementedError, match="v0.61.1"): + apply_edit(plan) + + def test_unknown_method_in_plan_short_circuits(self): + from soup_cli.utils.knowledge_edit import apply_edit + + # apply_edit defends against direct (non-builder) plan construction + # by re-validating the method before raising NotImplementedError. + class _BadPlan: + method = "zzz" + base = "b" + subject = "s" + target = "t" + layer = 5 + + with pytest.raises((TypeError, ValueError)): + apply_edit(_BadPlan()) # type: ignore + + +class TestCli: + def test_edit_help(self): + runner = CliRunner() + result = runner.invoke(app, ["edit", "--help"]) + assert result.exit_code == 0, result.output + + def test_edit_set_help(self): + runner = CliRunner() + result = runner.invoke(app, ["edit", "set", "--help"]) + assert result.exit_code == 0, result.output + + def test_edit_set_plan_only(self): + runner = CliRunner() + result = runner.invoke(app, [ + "edit", "set", + "--base", "meta-llama/Llama-3.1-8B-Instruct", + "--method", "rome", + "--subject", "Paris is the capital of France", + "--target", "Lyon", + "--plan-only", + ]) + # Plan-only mode prints the plan and exits 0 without invoking + # the deferred apply_edit kernel. + assert result.exit_code == 0, result.output + assert "rome" in result.output.lower() + assert "lyon" in result.output.lower() + + def test_edit_set_unknown_method_rejected(self): + runner = CliRunner() + result = runner.invoke(app, [ + "edit", "set", + "--base", "test", + "--method", "zzz", + "--subject", "s", + "--target", "t", + "--plan-only", + ]) + assert result.exit_code != 0 + + def test_edit_set_apply_deferred(self): + runner = CliRunner() + result = runner.invoke(app, [ + "edit", "set", + "--base", "test", + "--method", "rome", + "--subject", "Paris is the capital", + "--target", "Lyon", + ]) + # Without --plan-only, the CLI invokes apply_edit which raises + # NotImplementedError. Exit code 3 (deferred, NOT validation + # failure which is exit 2). Review HIGH H5 — distinguishes + # "not yet shipped" from "validation rejection". + assert result.exit_code == 3 + assert "v0.61.1" in result.output + + def test_edit_set_registry_id_null_byte_rejected(self): + """Review HIGH H4 — null-byte in --registry-id.""" + runner = CliRunner() + result = runner.invoke(app, [ + "edit", "set", + "--base", "test", + "--method", "rome", + "--subject", "Paris is the capital", + "--target", "Lyon", + "--registry-id", "evil\x00id", + "--plan-only", + ]) + assert result.exit_code == 2 + assert "null" in result.output.lower() or "registry-id" in result.output.lower() + + def test_edit_set_registry_id_oversize_rejected(self): + """Review HIGH H4 — oversized --registry-id.""" + runner = CliRunner() + result = runner.invoke(app, [ + "edit", "set", + "--base", "test", + "--method", "rome", + "--subject", "Paris is the capital", + "--target", "Lyon", + "--registry-id", "x" * 300, + "--plan-only", + ]) + assert result.exit_code == 2 diff --git a/tests/test_v0610_part_d.py b/tests/test_v0610_part_d.py new file mode 100644 index 0000000..bc28e6f --- /dev/null +++ b/tests/test_v0610_part_d.py @@ -0,0 +1,327 @@ +"""Tests for v0.61.0 Part D — Sequential edit governor. + +Coverage: +- ``NormBlowupPolicy`` frozen dataclass + bounds validation. +- ``classify_norm_blowup`` OK / WARN / BLOWUP taxonomy. +- ``governor_recommend_method`` auto-switches method when blowup detected. +- ``EditGovernor`` stateful tracker (per-base-model edit count + last verdict). +- ``GovernedEditError`` for refusals. +""" + +from __future__ import annotations + +import dataclasses + +import pytest + + +class TestModuleSurface: + def test_imports(self): + from soup_cli.utils.edit_governor import ( + DEFAULT_BLOWUP_POLICY, + VERDICTS, + EditGovernor, + GovernedEditError, + NormBlowupPolicy, + classify_norm_blowup, + governor_recommend_method, + ) + assert callable(classify_norm_blowup) + assert callable(governor_recommend_method) + assert isinstance(VERDICTS, tuple) + assert dataclasses.is_dataclass(NormBlowupPolicy) + # GovernedEditError is an exception type, not a dataclass. + assert issubclass(GovernedEditError, Exception) + assert isinstance(DEFAULT_BLOWUP_POLICY, NormBlowupPolicy) + assert callable(EditGovernor) + + +class TestNormBlowupPolicy: + def test_defaults(self): + from soup_cli.utils.edit_governor import NormBlowupPolicy + + p = NormBlowupPolicy() + assert p.warn_threshold > 0 + assert p.blowup_threshold > p.warn_threshold + assert p.max_sequential_edits > 0 + assert p.auto_switch_at >= 0 + + def test_frozen(self): + from soup_cli.utils.edit_governor import NormBlowupPolicy + + p = NormBlowupPolicy() + with pytest.raises(dataclasses.FrozenInstanceError): + p.warn_threshold = 999.0 # type: ignore + + def test_invalid_bounds(self): + from soup_cli.utils.edit_governor import NormBlowupPolicy + + with pytest.raises(ValueError): + NormBlowupPolicy(warn_threshold=10.0, blowup_threshold=5.0) + + with pytest.raises(ValueError): + NormBlowupPolicy(warn_threshold=-1.0) + + with pytest.raises(ValueError): + NormBlowupPolicy(max_sequential_edits=0) + + def test_bool_rejected(self): + from soup_cli.utils.edit_governor import NormBlowupPolicy + + with pytest.raises(TypeError): + NormBlowupPolicy(warn_threshold=True) # type: ignore + + with pytest.raises(TypeError): + NormBlowupPolicy(max_sequential_edits=True) # type: ignore + + def test_non_finite_rejected(self): + from soup_cli.utils.edit_governor import NormBlowupPolicy + + with pytest.raises(ValueError): + NormBlowupPolicy(warn_threshold=float("nan")) + + with pytest.raises(ValueError): + NormBlowupPolicy(blowup_threshold=float("inf")) + + +class TestClassifyNormBlowup: + def test_ok_band(self): + from soup_cli.utils.edit_governor import classify_norm_blowup + + assert classify_norm_blowup(0.01) == "OK" + assert classify_norm_blowup(0.0) == "OK" + + def test_warn_band(self): + from soup_cli.utils.edit_governor import ( + DEFAULT_BLOWUP_POLICY, + classify_norm_blowup, + ) + + warn = DEFAULT_BLOWUP_POLICY.warn_threshold + assert classify_norm_blowup(warn) == "WARN" + # Above warn but below blowup. + midpoint = (DEFAULT_BLOWUP_POLICY.warn_threshold + + DEFAULT_BLOWUP_POLICY.blowup_threshold) / 2 + assert classify_norm_blowup(midpoint) == "WARN" + + def test_blowup_band(self): + from soup_cli.utils.edit_governor import ( + DEFAULT_BLOWUP_POLICY, + classify_norm_blowup, + ) + + assert classify_norm_blowup(DEFAULT_BLOWUP_POLICY.blowup_threshold) == "BLOWUP" + assert classify_norm_blowup(100.0) == "BLOWUP" + + def test_bool_rejected(self): + from soup_cli.utils.edit_governor import classify_norm_blowup + + with pytest.raises(TypeError): + classify_norm_blowup(True) + + def test_non_finite_rejected(self): + from soup_cli.utils.edit_governor import classify_norm_blowup + + with pytest.raises(ValueError): + classify_norm_blowup(float("nan")) + + with pytest.raises(ValueError): + classify_norm_blowup(float("inf")) + + def test_negative_rejected(self): + from soup_cli.utils.edit_governor import classify_norm_blowup + + with pytest.raises(ValueError): + classify_norm_blowup(-1.0) + + +class TestGovernorRecommendMethod: + def test_no_switch_below_threshold(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + result = governor_recommend_method( + current_method="rome", + edit_count=1, + norm_delta=0.0, + ) + assert result.method == "rome" + assert result.switched is False + + def test_switch_at_auto_switch_count(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + result = governor_recommend_method( + current_method="rome", + edit_count=15, # above default auto_switch_at=10 + norm_delta=0.0, + ) + # ROME should auto-switch to AlphaEdit at the count boundary. + assert result.method == "alphaedit" + assert result.switched is True + + def test_blowup_forces_switch(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + result = governor_recommend_method( + current_method="rome", + edit_count=1, + norm_delta=100.0, # blowup + ) + assert result.method == "alphaedit" + assert result.switched is True + + def test_alphaedit_no_further_switch(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + # AlphaEdit is already the survival-mode method — no further switch. + result = governor_recommend_method( + current_method="alphaedit", + edit_count=15, + norm_delta=0.0, + ) + assert result.method == "alphaedit" + assert result.switched is False + + def test_invalid_method_rejected(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + with pytest.raises(ValueError): + governor_recommend_method( + current_method="zzz", + edit_count=1, + norm_delta=0.0, + ) + + def test_bool_edit_count_rejected(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + with pytest.raises(TypeError): + governor_recommend_method( + current_method="rome", + edit_count=True, # type: ignore + norm_delta=0.0, + ) + + def test_negative_edit_count_rejected(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + with pytest.raises(ValueError): + governor_recommend_method( + current_method="rome", + edit_count=-1, + norm_delta=0.0, + ) + + def test_negative_norm_delta_rejected(self): + from soup_cli.utils.edit_governor import governor_recommend_method + + with pytest.raises(ValueError): + governor_recommend_method( + current_method="rome", + edit_count=1, + norm_delta=-1.0, + ) + + def test_auto_switch_boundary_exact(self): + """Review L5 — exact `auto_switch_at=10` should switch.""" + from soup_cli.utils.edit_governor import ( + DEFAULT_BLOWUP_POLICY, + governor_recommend_method, + ) + + # At exactly auto_switch_at, ROME switches. + result = governor_recommend_method( + current_method="rome", + edit_count=DEFAULT_BLOWUP_POLICY.auto_switch_at, + norm_delta=0.0, + ) + assert result.method == "alphaedit" + assert result.switched is True + + def test_auto_switch_below_boundary_no_switch(self): + """Review L5 — one below auto_switch_at does NOT switch.""" + from soup_cli.utils.edit_governor import ( + DEFAULT_BLOWUP_POLICY, + governor_recommend_method, + ) + + result = governor_recommend_method( + current_method="rome", + edit_count=DEFAULT_BLOWUP_POLICY.auto_switch_at - 1, + norm_delta=0.0, + ) + assert result.method == "rome" + assert result.switched is False + + +class TestEditGovernor: + def test_construct(self): + from soup_cli.utils.edit_governor import EditGovernor + + g = EditGovernor(base_model="meta-llama/Llama-3.1-8B") + assert g.edit_count == 0 + + def test_record_edit(self): + from soup_cli.utils.edit_governor import EditGovernor + + g = EditGovernor(base_model="meta-llama/Llama-3.1-8B") + g.record_edit(method="rome", norm_delta=0.0) + assert g.edit_count == 1 + + def test_refuses_above_max_edits(self): + from soup_cli.utils.edit_governor import EditGovernor, GovernedEditError + + g = EditGovernor( + base_model="meta-llama/Llama-3.1-8B", + max_sequential_edits=2, + ) + g.record_edit(method="rome", norm_delta=0.0) + g.record_edit(method="rome", norm_delta=0.0) + with pytest.raises(GovernedEditError, match="max_sequential"): + g.check_can_edit() + + def test_blowup_blocks_further(self): + from soup_cli.utils.edit_governor import EditGovernor, GovernedEditError + + g = EditGovernor(base_model="meta-llama/Llama-3.1-8B") + g.record_edit(method="rome", norm_delta=100.0) # blowup + with pytest.raises(GovernedEditError, match="blowup"): + g.check_can_edit() + + def test_empty_base_rejected(self): + from soup_cli.utils.edit_governor import EditGovernor + + with pytest.raises(ValueError): + EditGovernor(base_model="") + + def test_null_byte_base_rejected(self): + from soup_cli.utils.edit_governor import EditGovernor + + with pytest.raises(ValueError): + EditGovernor(base_model="b\x00") + + def test_invalid_method_in_record_rejected(self): + from soup_cli.utils.edit_governor import EditGovernor + + g = EditGovernor(base_model="b") + with pytest.raises(ValueError): + g.record_edit(method="zzz", norm_delta=0.0) + + def test_recommend_next(self): + from soup_cli.utils.edit_governor import EditGovernor + + g = EditGovernor(base_model="b") + g.record_edit(method="rome", norm_delta=0.0) + rec = g.recommend_next_method(current_method="rome") + # After 1 ROME edit, still ROME. + assert rec.method == "rome" + + def test_snapshot(self): + from soup_cli.utils.edit_governor import EditGovernor + + g = EditGovernor(base_model="b") + g.record_edit(method="rome", norm_delta=0.01) + snap = g.snapshot() + assert snap["edit_count"] == 1 + assert snap["last_method"] == "rome" + assert snap["base_model"] == "b" diff --git a/tests/test_v0610_part_e.py b/tests/test_v0610_part_e.py new file mode 100644 index 0000000..49a7443 --- /dev/null +++ b/tests/test_v0610_part_e.py @@ -0,0 +1,403 @@ +"""Tests for v0.61.0 Part E — `soup edit diff` (knowledge-injection diff). + +Coverage: +- ``DiffReport`` / ``FactChange`` frozen dataclasses. +- ``load_probes`` JSONL loader (cwd-contained, symlink-rejected). +- ``build_diff_report`` schema-only orchestrator. +- ``render_diff_table`` + ``write_diff_report``. +- ``soup edit diff`` CLI smoke. +""" + +from __future__ import annotations + +import dataclasses +import json +import os +import sys +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app + + +class TestDataclasses: + def test_imports(self): + from soup_cli.utils.edit_diff import ( + DiffReport, + FactChange, + build_diff_report, + load_probes, + render_diff_table, + write_diff_report, + ) + assert callable(build_diff_report) + assert callable(load_probes) + assert callable(render_diff_table) + assert callable(write_diff_report) + assert dataclasses.is_dataclass(DiffReport) + assert dataclasses.is_dataclass(FactChange) + + def test_fact_change_frozen(self): + from soup_cli.utils.edit_diff import FactChange + + c = FactChange( + prompt="p", before="b", after="a", changed=True, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + c.prompt = "x" # type: ignore + + def test_diff_report_frozen(self): + from soup_cli.utils.edit_diff import DiffReport, FactChange + + r = DiffReport( + before_run_id="b", + after_run_id="a", + changes=(FactChange(prompt="p", before="b", after="a", changed=False),), + total_probes=1, + soup_version="0.61.0", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + r.total_probes = 999 # type: ignore + + def test_diff_report_validates_changes_tuple(self): + from soup_cli.utils.edit_diff import DiffReport + + with pytest.raises(TypeError): + DiffReport( + before_run_id="b", + after_run_id="a", + changes=[], # list, not tuple + total_probes=0, + soup_version="0.61.0", + ) + + def test_diff_report_to_dict(self): + from soup_cli.utils.edit_diff import DiffReport, FactChange + + r = DiffReport( + before_run_id="b", + after_run_id="a", + changes=( + FactChange(prompt="p1", before="x", after="y", changed=True), + ), + total_probes=1, + soup_version="0.61.0", + ) + d = r.to_dict() + assert d["before_run_id"] == "b" + assert d["after_run_id"] == "a" + assert d["total_probes"] == 1 + assert d["changes"][0]["prompt"] == "p1" + # JSON round-trip + assert json.loads(json.dumps(d)) == d + + def test_diff_report_total_probes_bool_rejected(self): + from soup_cli.utils.edit_diff import DiffReport + + with pytest.raises(TypeError): + DiffReport( + before_run_id="b", + after_run_id="a", + changes=(), + total_probes=True, # type: ignore + soup_version="0.61.0", + ) + + +class TestLoadProbes: + def _write_jsonl(self, path: Path, rows: list[dict]) -> None: + path.write_text("\n".join(json.dumps(r) for r in rows), encoding="utf-8") + + def test_happy_path(self, tmp_path): + from soup_cli.utils.edit_diff import load_probes + + p = tmp_path / "probes.jsonl" + self._write_jsonl(p, [{"prompt": "Who is X?"}, {"prompt": "Who is Y?"}]) + + # Need to be in cwd + old = os.getcwd() + os.chdir(tmp_path) + try: + probes = load_probes("probes.jsonl") + finally: + os.chdir(old) + assert probes == ("Who is X?", "Who is Y?") + + def test_missing_file(self, tmp_path): + from soup_cli.utils.edit_diff import load_probes + + old = os.getcwd() + os.chdir(tmp_path) + try: + with pytest.raises(FileNotFoundError): + load_probes("missing.jsonl") + finally: + os.chdir(old) + + def test_skips_malformed_rows(self, tmp_path): + from soup_cli.utils.edit_diff import load_probes + + p = tmp_path / "probes.jsonl" + p.write_text( + '{"prompt": "good"}\n' + 'this is not json\n' + '{"prompt": ""}\n' + '{"prompt": "another good"}\n', + encoding="utf-8", + ) + + old = os.getcwd() + os.chdir(tmp_path) + try: + probes = load_probes("probes.jsonl") + finally: + os.chdir(old) + assert probes == ("good", "another good") + + def test_outside_cwd_rejected(self, tmp_path): + from soup_cli.utils.edit_diff import load_probes + + out = tmp_path / "outside.jsonl" + out.write_text('{"prompt": "p"}', encoding="utf-8") + + # cwd is the original; tmp_path is unrelated + cwd_dir = tmp_path / "cwd" + cwd_dir.mkdir() + old = os.getcwd() + os.chdir(cwd_dir) + try: + with pytest.raises(ValueError, match="cwd"): + load_probes(str(out)) + finally: + os.chdir(old) + + def test_null_byte_rejected(self): + from soup_cli.utils.edit_diff import load_probes + + with pytest.raises(ValueError): + load_probes("probe\x00.jsonl") + + def test_empty_path_rejected(self): + from soup_cli.utils.edit_diff import load_probes + + with pytest.raises(ValueError): + load_probes("") + + @pytest.mark.skipif(sys.platform == "win32", reason="POSIX-only symlink test") + def test_symlink_rejected(self, tmp_path): + from soup_cli.utils.edit_diff import load_probes + + real = tmp_path / "real.jsonl" + real.write_text('{"prompt": "p"}', encoding="utf-8") + link = tmp_path / "link.jsonl" + os.symlink(real, link) + + old = os.getcwd() + os.chdir(tmp_path) + try: + with pytest.raises(ValueError, match="symlink"): + load_probes("link.jsonl") + finally: + os.chdir(old) + + +class TestBuildDiffReport: + def test_happy_path(self): + from soup_cli.utils.edit_diff import build_diff_report + + report = build_diff_report( + before_run_id="before-run", + after_run_id="after-run", + ) + assert report.before_run_id == "before-run" + assert report.after_run_id == "after-run" + assert report.changes == () + assert report.total_probes == 0 + + def test_same_run_id_rejected(self): + from soup_cli.utils.edit_diff import build_diff_report + + with pytest.raises(ValueError, match="differ"): + build_diff_report( + before_run_id="x", + after_run_id="x", + ) + + def test_bool_run_id_rejected(self): + from soup_cli.utils.edit_diff import build_diff_report + + with pytest.raises(TypeError): + build_diff_report( + before_run_id=True, # type: ignore + after_run_id="x", + ) + + def test_top_k_bool_rejected(self): + from soup_cli.utils.edit_diff import build_diff_report + + with pytest.raises(TypeError): + build_diff_report( + before_run_id="b", + after_run_id="a", + top_k=True, # type: ignore + ) + + def test_top_k_out_of_range(self): + from soup_cli.utils.edit_diff import build_diff_report + + with pytest.raises(ValueError): + build_diff_report( + before_run_id="b", after_run_id="a", top_k=0, + ) + with pytest.raises(ValueError): + build_diff_report( + before_run_id="b", after_run_id="a", top_k=200, + ) + + def test_with_probe_file(self, tmp_path): + from soup_cli.utils.edit_diff import build_diff_report + + p = tmp_path / "probes.jsonl" + p.write_text( + '{"prompt": "q1"}\n{"prompt": "q2"}\n{"prompt": "q3"}\n', + encoding="utf-8", + ) + old = os.getcwd() + os.chdir(tmp_path) + try: + r = build_diff_report( + before_run_id="b", + after_run_id="a", + probe_file="probes.jsonl", + top_k=2, + ) + finally: + os.chdir(old) + assert r.total_probes == 3 + assert len(r.changes) == 2 # capped by top_k + + +class TestRenderDiffTable: + def test_renders(self, capsys): + from rich.console import Console + + from soup_cli.utils.edit_diff import ( + DiffReport, + FactChange, + render_diff_table, + ) + + r = DiffReport( + before_run_id="b", + after_run_id="a", + changes=( + FactChange(prompt="p1", before="x", after="y", changed=True), + ), + total_probes=1, + soup_version="0.61.0", + ) + console = Console(force_terminal=False, width=200) + render_diff_table(r, console) + # No raise = pass. + + def test_non_report_rejected(self): + from rich.console import Console + + from soup_cli.utils.edit_diff import render_diff_table + + with pytest.raises(TypeError): + render_diff_table("not a report", Console()) # type: ignore + + +class TestWriteDiffReport: + def test_happy_path(self, tmp_path): + from soup_cli.utils.edit_diff import ( + DiffReport, + write_diff_report, + ) + + r = DiffReport( + before_run_id="b", + after_run_id="a", + changes=(), + total_probes=0, + soup_version="0.61.0", + ) + old = os.getcwd() + os.chdir(tmp_path) + try: + out = "report.json" + write_diff_report(r, out) + assert Path(out).exists() + data = json.loads(Path(out).read_text()) + assert data["before_run_id"] == "b" + finally: + os.chdir(old) + + def test_outside_cwd_rejected(self, tmp_path): + from soup_cli.utils.edit_diff import ( + DiffReport, + write_diff_report, + ) + + r = DiffReport( + before_run_id="b", + after_run_id="a", + changes=(), + total_probes=0, + soup_version="0.61.0", + ) + out = tmp_path / "outside.json" + cwd_dir = tmp_path / "cwd" + cwd_dir.mkdir() + old = os.getcwd() + os.chdir(cwd_dir) + try: + with pytest.raises(ValueError): + write_diff_report(r, str(out)) + finally: + os.chdir(old) + + def test_non_report_rejected(self, tmp_path): + from soup_cli.utils.edit_diff import write_diff_report + + with pytest.raises(TypeError): + write_diff_report("not a report", "out.json") # type: ignore + + +class TestCli: + def test_help(self): + runner = CliRunner() + result = runner.invoke(app, ["edit", "diff", "--help"]) + assert result.exit_code == 0, result.output + + def test_basic_diff(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=tmp_path): + result = runner.invoke(app, [ + "edit", "diff", "before-run", "after-run", + ]) + assert result.exit_code == 0, result.output + + def test_same_runs_rejected(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=tmp_path): + result = runner.invoke(app, [ + "edit", "diff", "same", "same", + ]) + assert result.exit_code != 0 + + def test_writes_output(self, tmp_path): + runner = CliRunner() + with runner.isolated_filesystem(temp_dir=tmp_path) as fs: + out = Path(fs) / "diff.json" + result = runner.invoke(app, [ + "edit", "diff", "before-run", "after-run", + "--output", str(out), + ]) + assert result.exit_code == 0, result.output + assert out.exists() + data = json.loads(out.read_text()) + assert data["total_probes"] == 0