mirror of https://github.com/razor-ai/soup.git
feat(unlearn/edit): v0.61.0 — Unlearning & Knowledge Edit (NPO/SimNPO/RMU + ROME/MEMIT/AlphaEdit)
5 Parts shipping schema + CLI surface for two of the most under-served axes in fine-tuning: GDPR right-to-be-forgotten unlearning (the legal-liability axis upstream TRL avoids) and surgical knowledge editing (research-coded everywhere, productized nowhere). Schema-only release; live trainer + kernel wiring deferred to v0.61.1 (matches established v0.50.0 / v0.52.0 / v0.53.0 stub-then-live cadence). Part A — task='unlearn' + NPO/SimNPO/RMU allowlist + UnlearnTrainerWrapper + data.forget_set / data.retain_set + training.unlearn_method/_alpha Part B — soup eval unlearning (TOFU/MUSE/WMDP) with Forget Quality + Model Utility + PrivLeak kernels + OK/MINOR/MAJOR taxonomy; bundled TOFU mini-fixture under soup_cli/data/_fixtures/unlearning/ Part C — soup edit set (ROME/MEMIT/AlphaEdit) + EditPlan + per-method default layer; --plan-only ships live, apply_edit kernel deferred Part D — Sequential edit governor: norm-blowup detection (OK/WARN/BLOWUP), auto-switch ROME→AlphaEdit at edit#10 or BLOWUP, refuses past cap Part E — soup edit diff: cwd-contained probe loader, atomic JSONL out, shape + table renderer (live before/after generation v0.61.1) Net: +125 tests (9446 → 9571), +5 utility modules + 1 trainer wrapper + 2 commands. Review-fix coverage: 0 CRITICAL + 5 HIGH + 11 MEDIUM + 11 LOW. All ruff + pytest green; Step 6 smokes (CLI plumbing + happy paths + 5 schema rejection paths) confirmed end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
47409df730
commit
740832e1b4
|
|
@ -96,7 +96,7 @@ soup_cli/
|
|||
commands/ - Command implementations (train, chat, eval, deploy, etc.)
|
||||
config/ - Config schema (schema.py) and loader (loader.py)
|
||||
data/ - Data loading, format conversion, providers, templates
|
||||
trainer/ - Training wrappers (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, Pretrain, Reward Model, Embedding)
|
||||
trainer/ - Training wrappers (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, Pretrain, Reward Model, Embedding, BCO, Preference, Distill, Classifier, PRM, Unlearn)
|
||||
monitoring/ - Callbacks and live dashboard
|
||||
experiment/ - SQLite experiment tracking
|
||||
eval/ - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
|
||||
|
|
@ -107,11 +107,11 @@ soup_cli/
|
|||
cans/ - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
|
||||
data/traces/ - Trace-to-Preference harvester (v0.26.0)
|
||||
data/collators.py - CrossDocCollator for sample packing (v0.33.0)
|
||||
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch
|
||||
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff
|
||||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (207 files, 9446 tests)
|
||||
tests/ - Test suite (212 files, 9571 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
67
README.md
67
README.md
|
|
@ -42,15 +42,14 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.60.0 — Supply Chain Security: tensors you can ship through procurement.** Six controls that hosted vendors structurally can't provide — adapter weight-space backdoor scanner, Merkle-root sign + verify, strict-safetensors mode (refuse pickle weights), trust-on-first-use namespace pinning (anti-AI-Jacking), closed license-compatibility matrix at merge, and a signed airgap-bundle for data-diode transfer. Every piece is opt-in for operators who want enforcement; CI pipelines and security-conscious enterprises get a "you can't ship an unaudited adapter through Soup" guarantee.
|
||||
**v0.61.0 — Unlearning & Knowledge Edit: GDPR right-to-be-forgotten + surgical fact patches.** Two of the most under-served axes in fine-tuning land as first-class trainers: unlearning (NPO / SimNPO / RMU — the legal-liability axis upstream TRL avoids) and knowledge editing (ROME / MEMIT / AlphaEdit — research-coded everywhere, productized nowhere). Hospital data teams handling a GDPR deletion request, labs patching factual errors without retraining, and security teams responding to a CSAM/PII leak all get a CLI workflow that composes with the existing Registry + eval-gate + diagnose surface. Schema-only release; live trainer + kernel wiring lands in v0.61.1.
|
||||
|
||||
- **`soup adapters scan <adapter>`** — spectral analysis of LoRA weights pre-load. Flags rank-1 dominance (the canonical weight-space trojan pattern), top-1 singular-vector energy concentration, NaN/Inf in weights, and Frobenius-norm outliers via robust median+MAD bucketing. Pure numpy (no torch). Exit codes 0=OK / 1=WARN / 3=FAIL for CI grep. Reuses v0.57.0 `adapter_diff` loader so on-disk surface stays single-source.
|
||||
- **`soup adapters sign / verify [--strict]`** — deterministic Merkle-root manifest over every file in the adapter dir (including nested `tokenizer/` / `processor/` subdirs). Tamper any file, verify fails. `UNSIGNED` backend ships live for offline tamper detection; `sigstore` + `ed25519` backends raise `NotImplementedError` with v0.60.1 marker (schema lives now so CI can integrate). Signature stored as `.soup-signature.json` written atomically via the shared `atomic_write_text` helper.
|
||||
- **`soup adapters check-safetensors [--strict]`** — refuses pickle / PyTorch-classic weights at the boundary. Closed 8-entry unsafe-extension allowlist (`.bin` / `.pt` / `.pth` / `.ckpt` / `.pkl` / `.pickle` / `.joblib` / `.msgpack`). Exit code 3 (distinct from generic errors) under `--strict` so CI can gate on pickle-only.
|
||||
- **Namespace-pin TOFU (anti-AI-Jacking).** New `NamespacePinStore` SQLite cache + `verify_namespace` helper: records `(repo_id, author, created_at)` on first download, refuses updates when the author changes OR `created_at` jumps backward (case-insensitive comparison via `datetime.fromisoformat` — fragile lexicographic compare avoided). `--allow-namespace-shift <new-author>` explicit opt-in (bool rejected to prevent free-for-all bypass). DB path containment-checked to `$HOME / $CWD / $TMPDIR`; POSIX `0o600` perms; pre-placed symlinks rejected via `os.lstat + S_ISLNK`. Live wiring into `utils/hubs.download_repo` lands in v0.60.1.
|
||||
- **License-conflict matrix at merge.** Closed allowlist of 33 SPDX-ish licenses (Apache / MIT / BSD / LGPL / MPL / GPL / AGPL / CC-BY / CC-BY-NC / Llama-2/3.x/community / Gemma / Qwen-research / Mistral-research / OpenRAIL / OpenAI-ToS / Anthropic-AUP). `MappingProxyType` category compatibility table. `soup adapters merge --license <id> --license-override <reason>` — non-commercial + permissive refuses (8-char min, 4096-char max reason); strong-copyleft + permissive refuses; restricted-use + permissive refuses. Override reason captured in panel; audit-log integration v0.60.1.
|
||||
- **`soup airgap-bundle --model <m> --output <out.tar>`** — signed tarball with model + datasets + wheels + kernels + SHA-256 per file in embedded `manifest.json`. Default 100 GiB size cap; refuses oversize. Deterministic dataset labeling by sorted basename (NOT argv order) so the same inputs in different argv order produce identical manifests. TOCTOU lstat re-check on parent + output path BEFORE mkstemp; atomic `os.replace` via sibling tempfile; `tarfile.data_filter` set for any future `extractall` caller. 64 MiB manifest cap on `inspect_airgap_bundle`.
|
||||
- **+152 new tests** (9294 → 9446). Review-fix coverage across five waves: 0 CRITICAL + 12 HIGH + 11 MEDIUM + 6 LOW resolved before tag. Manual CPU smokes for sign / verify / scan / check-safetensors / airgap-bundle / merge-with-license-conflict all confirmed end-to-end.
|
||||
- **`task='unlearn'` + NPO / SimNPO / RMU.** New first-class task on `SoupConfig`. `training.unlearn_method: Literal["npo","simnpo","rmu"]` selects the backend; `data.forget_set` (rows to unlearn) + `data.retain_set` (capability-preserving rows) are validated at schema-load. Case-insensitive method normalisation; mlx backend rejected with distinct error; missing forget_set on `task='unlearn'` rejected with named field. `UnlearnTrainerWrapper` stub captures the method + accepts forward-compat kwargs so v0.61.1 wiring is purely additive.
|
||||
- **`soup eval unlearning <run-id> --benchmark tofu|muse|wmdp`** — three orthogonal axes scored on the project's OK / MINOR / MAJOR taxonomy: Forget Quality (pre/post forget-loss linear ramp), Model Utility (retain-accuracy ratio), PrivLeak (membership-inference AUC symmetric around 0.5). Bundled TOFU mini-fixture under `soup_cli/data/_fixtures/unlearning/`; MUSE + WMDP loaders ship in v0.61.1. Evidence-driven: operators supply pre-computed JSON today and get a frozen `UnlearnReport` with overall worst-case verdict + atomic JSON output.
|
||||
- **`soup edit set --base <m> --method rome|memit|alphaedit --subject "Paris is the capital of France" --target "Lyon"`** — surgical locate-and-edit. Closed method allowlist; per-method default MLP layer (ROME=5, MEMIT=8, AlphaEdit=5); operator override via `--layer`. `--plan-only` validates + prints the resolved `EditPlan` and exits 0 today; without it the CLI surfaces the deferred-live `NotImplementedError` with v0.61.1 marker at exit code 3 (distinct from validation rejection exit 2).
|
||||
- **Sequential edit governor + norm-blowup detection.** New `EditGovernor` tracks per-base-model edit count + last verdict; auto-switches ROME → AlphaEdit at the configurable edit-count threshold (default 10) AND on detected BLOWUP (Frobenius delta ≥ 5.0). `check_can_edit()` refuses further edits past the max cap or after a BLOWUP — `GovernedEditError` so callers distinguish governance refusals from other failures. AlphaEdit is the projection-based survivor (never switched away).
|
||||
- **`soup edit diff <before-run> <after-run> --probes p.jsonl`** — knowledge-injection diff visualizer. Frozen `DiffReport` + `FactChange` dataclasses; probe-file loader is cwd-contained + symlink-rejected + size-capped (16 MiB / 1000 rows). v0.61.0 ships shape + table renderer; v0.61.1 lands the live model-driven before/after generation.
|
||||
- **+193 new tests** (9446 → 9571 net after review-fix coverage). Review-fix coverage: 0 CRITICAL + 5 HIGH + 11 MEDIUM + 11 LOW (frozen-field declarations on `EditGovernor` so `asdict` / `replace` / slots stay safe; loud rejection of present-but-invalid evidence in `run_unlearn_eval`; `registry_id` validated BEFORE the deferred attach hook; exit-3 for "deferred" vs exit-2 for "validation rejection"; `validate_edit_method` lifted to module top in `edit_governor`; `importlib.resources` for fixture path; truncation warnings on probe-prompt clipping; boundary tests at exact `auto_switch_at` + `unlearn_alpha=0.0` + `unlearn_alpha=10.0` + `mia_auc=0.0` / `1.0`). Manual CPU smokes for `eval unlearning`, `edit set --plan-only`, `edit diff`, and 5 schema rejection paths all green.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -199,6 +198,55 @@ soup loop replay iter-20260515T120000-abcdef01
|
|||
|
||||
State lives in `.soup/loop.yaml` (atomic write, cwd-contained, symlink-rejected). Per-iteration manifests under `.soup-loops/<iter-id>/iteration.json` are laid out so a v0.26 Soup Can can wrap them directly. The canary router is deterministic (SHA-256 hash of conversation id) and sticky-on-rollback — a flaky verdict can't ping-pong traffic between adapters.
|
||||
|
||||
## Unlearning (`task='unlearn'`, NPO / SimNPO / RMU)
|
||||
|
||||
GDPR right-to-be-forgotten + CSAM/PII leak response, productized. Three method backends:
|
||||
|
||||
- **NPO** — Negative Preference Optimization (DPO-shaped negative-only loss; needs a reference model).
|
||||
- **SimNPO** — length-normalised NPO without a ref model (faster, more stable on long sequences).
|
||||
- **RMU** — Representation Misdirection Unlearning (residual-stream noise on forget inputs).
|
||||
|
||||
```yaml
|
||||
# unlearn.yaml
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: unlearn
|
||||
data:
|
||||
train: traces.jsonl
|
||||
forget_set: gdpr_deletion_set.jsonl
|
||||
retain_set: capability_anchors.jsonl
|
||||
training:
|
||||
unlearn_method: npo # or simnpo / rmu
|
||||
unlearn_alpha: 0.5 # retain-set weighting [0.0, 10.0]
|
||||
```
|
||||
|
||||
```bash
|
||||
# Score the run on TOFU / MUSE / WMDP (OK / MINOR / MAJOR verdict).
|
||||
soup eval unlearning <run-id> --benchmark tofu --evidence evidence.json --output report.json
|
||||
```
|
||||
|
||||
Three orthogonal axes: **Forget Quality** (pre/post forget-loss delta), **Model Utility** (retain-accuracy preserved), **PrivLeak** (membership-inference AUC distance from 0.5). Bundled TOFU mini-fixture; MUSE + WMDP loaders land in the next release.
|
||||
|
||||
## Knowledge Editing (`soup edit set`, ROME / MEMIT / AlphaEdit)
|
||||
|
||||
Surgical factual patches WITHOUT a full fine-tuning loop. Hospital data team correcting a misattributed drug interaction, lab fixing a wrong historical date, security team responding to a hallucinated CVE — all one CLI invocation.
|
||||
|
||||
```bash
|
||||
# Plan-only mode validates the request + prints the resolved EditPlan + exits 0.
|
||||
soup edit set \
|
||||
--base meta-llama/Llama-3.1-8B-Instruct \
|
||||
--method rome \
|
||||
--subject "Paris is the capital of France" \
|
||||
--target "Lyon" \
|
||||
--plan-only
|
||||
|
||||
# Diff what the model "knew" before vs after the edit.
|
||||
soup edit diff <run-id-before> <run-id-after> --probes probes.jsonl --output diff.json
|
||||
```
|
||||
|
||||
Sequential edit governor auto-switches **ROME → AlphaEdit** at edit #10 (configurable) AND on detected norm-blowup (`||W - W_base||_F` over threshold). The governor refuses further edits past the per-base-model cap so a runaway script can't quietly corrupt your checkpoint.
|
||||
|
||||
The live ROME / MEMIT / AlphaEdit kernel + before/after generation in `edit diff` land in the next patch; `--plan-only` and the schema surface ship today so soup.yaml and CI invocations are stable.
|
||||
|
||||
## Pre-flight Decision (`soup advise`)
|
||||
|
||||
Run BEFORE you spend 8 hours on a GPU. `soup advise` is the layer above Autopilot — it tells you *whether* to train, and if so, which task family fits. Pure-Python heuristic, no GPU required for the verdict itself.
|
||||
|
|
@ -3414,6 +3462,9 @@ soup adapters verify <adapter> [--strict] Verify manifest against current fi
|
|||
soup adapters check-safetensors <adapter> [--strict] Refuse pickle / PyTorch-classic weights
|
||||
soup adapters merge ... --license <id> --license-override <reason> License-conflict gate
|
||||
soup airgap-bundle --model <m> --output <out.tar> Signed tarball for data-diode transfer
|
||||
soup eval unlearning <run-id> --benchmark tofu|muse|wmdp Forget Quality + Model Utility + PrivLeak verdict
|
||||
soup edit set --base <m> --method rome|memit|alphaedit --subject "..." --target "..." Surgical knowledge edit (--plan-only available)
|
||||
soup edit diff <before-run> <after-run> --probes p.jsonl Knowledge-injection diff visualizer
|
||||
soup version [--full] [--json] Show version (--full: system info, --json: JSON output)
|
||||
soup --verbose <command> Full traceback on errors
|
||||
```
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@ We provide security updates for the following versions:
|
|||
- **Versions older than 3 minor versions:** No support
|
||||
|
||||
Example:
|
||||
- v0.60.0 -- Full support (latest)
|
||||
- v0.61.0 -- Full support (latest)
|
||||
- v0.60.0 -- Full support
|
||||
- v0.59.0 -- Full support
|
||||
- v0.58.0 -- Full support
|
||||
- v0.57.0 -- Full support
|
||||
|
|
@ -134,6 +135,8 @@ No known critical vulnerabilities in current releases.
|
|||
|
||||
### Security Hardening History
|
||||
|
||||
- **v0.61.0 — Unlearning & Knowledge Edit**: 5 parts ship `task='unlearn'` (NPO / SimNPO / RMU) + `soup eval unlearning` (TOFU / MUSE / WMDP) + `soup edit set` (ROME / MEMIT / AlphaEdit) + sequential edit governor + `soup edit diff`. Schema-only release (live trainer + kernel wiring deferred to v0.61.1 — mirrors v0.50.0 / v0.52.0 / v0.53.0 stub-then-live pattern). **TOCTOU lstat-before-write** on every operator-supplied path via the shared `paths.atomic_write_text` helper (write_unlearn_report / write_diff_report); `load_evidence_file` + `load_probes` cwd-contained + `os.lstat + S_ISLNK` rejection + size-cap (16 MiB evidence, 16 MiB probe file, 1000 probe rows, 4096-char per prompt with WARNING-log truncation per review MEDIUM M5). **Frozen dataclasses** for every report / plan surface (`UnlearnMetric` / `UnlearnReport` / `EditRequest` / `EditPlan` / `FactChange` / `DiffReport` / `NormBlowupPolicy` / `MethodRecommendation`); `EditGovernor` counters declared as real dataclass fields (review HIGH H1 — slots-safe, `asdict` / `replace` compatible). **Invalid evidence raises loudly** in `run_unlearn_eval` (review HIGH H3) — distinguishes "missing key → neutral OK" from "present-but-invalid → ValueError" so operator typos surface instead of silently scoring OK. **Validator policy** mirrors v0.41.0 / v0.51.0 / v0.52.0 / v0.53.0: bool-first / null-byte / oversize (≤32-char method names, ≤64-char benchmark / verdict, ≤512-char base_model, ≤4096-char forget_set/retain_set/subject/target) / case-insensitive normalisation; numeric bounds with `math.isfinite` rejection on NaN / Inf. **`registry_id` validated BEFORE deferred attach** (review HIGH H4) — defends against crafted null-byte / oversize crashing the v0.61.1 registry hook. **Exit-code policy**: `soup eval unlearning` exits 0 on OK / MINOR (operator can choose to gate), 2 on MAJOR (matches v0.56.0 diagnose strict-mode CI gate); `soup edit set` exits 0 with `--plan-only`, 2 on schema rejection, 3 on deferred-live `NotImplementedError` (review HIGH H5 — distinguishes "not yet shipped" from "validation rejection", matches v0.56.0 strict-mode policy). **`importlib.resources`** for TOFU fixture path (review MEDIUM M6 — zipapp / namespace-package safe; mirrors v0.53.8 #93 `_bundle_source_path` policy). **Test count**: 9446 → 9571 (+125 net in `tests/test_v0610_part_{a-e}.py`; 192 pass + 1 POSIX-only symlink test skipped on Windows). **Known limitations**: (1) Every live wiring (NPO/SimNPO/RMU loss kernels, ROME/MEMIT/AlphaEdit closed-form updates, `UnlearnTrainerWrapper.setup`) deferred to v0.61.1 with explicit v0.61.1 markers — schema accepts every flag today so soup.yaml written now will work the moment v0.61.1 ships. (2) MUSE + WMDP fixtures NOT bundled (only TOFU mini-set); operators wanting MUSE / WMDP today supply pre-computed evidence JSON via `--evidence`. (3) `soup edit diff` is shape-only — live model loading + before/after generation lands in v0.61.1. (4) Registry-attach for edited models is plan-only — v0.61.1 wires the `parent_adapter` lineage write. (5) `EditGovernor` is single-process (no SQLite persistence). (6) Norm-blowup detection depends on operator-supplied `norm_delta` — v0.61.1's live `apply_edit` will compute + thread it through. (7) Unlearn task gated to non-mlx (mlx-lm doesn't expose per-token logp scoring needed for NPO/SimNPO). (8) `apply_edit` does not consult `EditGovernor` automatically — v0.61.1 wires the optional `governor: EditGovernor` kwarg. (9) **5-agent review wave**: python + code + security + tdd + verification. Direct `code-reviewer` / `security-reviewer` agents hit context-window thrash on the 800+ KB CLAUDE.md release-notes history (matches v0.58.0 / v0.59.0 / v0.60.0 idiom); the `general-purpose` agent with explicit "do not crawl, read only these N files" prompts produced equivalent findings consolidated into Wave 1 (code) + Wave 2 (security) + Wave 3 (TDD). (v0.61.0)
|
||||
|
||||
- **v0.10.10**: Bearer token auth on Web UI, CORS restrictions, path traversal protection, SSRF prevention, max_tokens limits, supply-chain pinning (llama.cpp b5270), deprecated CLI secret flags
|
||||
- **v0.12.0**: experiment_name path traversal validation, GaLore parameter type enforcement
|
||||
- **v0.13.0**: Batch inference max_tokens capped at 16384, trust_remote_code warning
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.60.0"
|
||||
version = "0.61.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
@ -89,7 +89,10 @@ packages = ["soup_cli"]
|
|||
# package directory, so we use the artifacts directive (NOT
|
||||
# force-include, which double-shipped the files in v0.53.8 and produced
|
||||
# a "duplicate filename in local headers" 400 from PyPI upload).
|
||||
artifacts = ["soup_cli/data/_fixtures/*.jsonl"]
|
||||
artifacts = [
|
||||
"soup_cli/data/_fixtures/*.jsonl",
|
||||
"soup_cli/data/_fixtures/unlearning/*.jsonl",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py39"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.60.0"
|
||||
__version__ = "0.61.0"
|
||||
|
|
|
|||
|
|
@ -265,6 +265,18 @@ from soup_cli.commands import airgap as _airgap_cmd # noqa: E402
|
|||
|
||||
app.command(name="airgap-bundle")(_airgap_cmd.airgap_bundle)
|
||||
|
||||
# v0.61.0 — Unlearning & Knowledge Edit: `soup edit set / diff`.
|
||||
from soup_cli.commands import edit as _edit_cmd # noqa: E402
|
||||
|
||||
app.add_typer(
|
||||
_edit_cmd.app,
|
||||
name="edit",
|
||||
help=(
|
||||
"Knowledge editing (ROME / MEMIT / AlphaEdit) - patch facts "
|
||||
"without re-training (v0.61.0)."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _rewrite_advise_argv(argv: list) -> list:
|
||||
"""Inject `run` between `advise` and a non-subcommand first argument.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,158 @@
|
|||
"""v0.61.0 Part B — `soup eval unlearning` subcommand.
|
||||
|
||||
Computes Forget Quality + Model Utility + PrivLeak metrics from
|
||||
operator-supplied evidence JSON and emits an :class:`UnlearnReport`.
|
||||
Live model-driven evaluation lands in v0.61.1; this release writes a
|
||||
neutral OK report when no evidence is supplied so the schema + output
|
||||
shape are usable today.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.table import Table
|
||||
|
||||
|
||||
def register(app: typer.Typer, console: Console) -> None:
|
||||
"""Attach v0.61.0 subcommands to ``app``."""
|
||||
|
||||
@app.command(name="unlearning")
|
||||
def unlearning_cmd(
|
||||
run_id: str = typer.Argument(
|
||||
..., help="Run identifier (e.g. registry id of the unlearn run).",
|
||||
),
|
||||
benchmark: str = typer.Option(
|
||||
"tofu", "--benchmark", "-b",
|
||||
help="Benchmark: tofu / muse / wmdp.",
|
||||
),
|
||||
evidence: Optional[str] = typer.Option(
|
||||
None, "--evidence", "-e",
|
||||
help=(
|
||||
"Path to a JSON file with pre-computed evidence. Schema: "
|
||||
"{forget_quality: {pre_loss, post_loss}, "
|
||||
"model_utility: {pre_acc, post_acc}, "
|
||||
"priv_leak: {mia_auc}}. Missing keys fall through to a "
|
||||
"neutral OK score."
|
||||
),
|
||||
),
|
||||
output: Optional[str] = typer.Option(
|
||||
None, "--output", "-o",
|
||||
help="Where to write the rendered UnlearnReport JSON.",
|
||||
),
|
||||
attach_to_registry: Optional[str] = typer.Option(
|
||||
None, "--attach-to-registry",
|
||||
help=(
|
||||
"Optional registry entry id to attach the report as an "
|
||||
"eval_results artifact (mirrors v0.55.0 eval lock policy)."
|
||||
),
|
||||
),
|
||||
) -> None:
|
||||
"""Score an unlearn run on TOFU / MUSE / WMDP."""
|
||||
from soup_cli.utils.unlearning_eval import (
|
||||
load_evidence_file,
|
||||
run_unlearn_eval,
|
||||
validate_benchmark_name,
|
||||
write_unlearn_report,
|
||||
)
|
||||
|
||||
try:
|
||||
bench = validate_benchmark_name(benchmark)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid benchmark:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
evidence_data = None
|
||||
if evidence is not None:
|
||||
try:
|
||||
evidence_data = load_evidence_file(evidence)
|
||||
except json.JSONDecodeError as exc:
|
||||
# json.JSONDecodeError is a ValueError subclass — catch it
|
||||
# FIRST so we can label the message specifically (review
|
||||
# MEDIUM M10 — was unreachable after the broader except).
|
||||
console.print(
|
||||
f"[red]Evidence file is not valid JSON:[/] {escape(str(exc))}"
|
||||
)
|
||||
raise typer.Exit(2) from exc
|
||||
except (FileNotFoundError, ValueError, TypeError, OSError) as exc:
|
||||
console.print(f"[red]Cannot read evidence:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
try:
|
||||
report = run_unlearn_eval(
|
||||
run_id=run_id,
|
||||
benchmark=bench,
|
||||
evidence=evidence_data,
|
||||
)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Eval failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
# Render the table.
|
||||
table = Table(title=f"Unlearn eval — {escape(bench)} — {escape(run_id)}")
|
||||
table.add_column("metric", style="bold")
|
||||
table.add_column("score", justify="right")
|
||||
table.add_column("verdict")
|
||||
table.add_column("evidence")
|
||||
for m in report.metrics:
|
||||
colour = {
|
||||
"OK": "green",
|
||||
"MINOR": "yellow",
|
||||
"MAJOR": "red",
|
||||
}.get(m.verdict, "white")
|
||||
table.add_row(
|
||||
escape(m.name),
|
||||
f"{m.score:.3f}",
|
||||
f"[{colour}]{escape(m.verdict)}[/]",
|
||||
escape(m.evidence),
|
||||
)
|
||||
console.print(table)
|
||||
overall_colour = {
|
||||
"OK": "green",
|
||||
"MINOR": "yellow",
|
||||
"MAJOR": "red",
|
||||
}.get(report.overall, "white")
|
||||
console.print(
|
||||
f"Overall: [{overall_colour}]{escape(report.overall)}[/]"
|
||||
)
|
||||
|
||||
# Write the report.
|
||||
if output is not None:
|
||||
try:
|
||||
write_unlearn_report(report, output)
|
||||
except (TypeError, ValueError, OSError) as exc:
|
||||
console.print(f"[red]Cannot write report:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
console.print(f"Wrote report -> {escape(output)}")
|
||||
|
||||
# Optional registry attach.
|
||||
if attach_to_registry is not None:
|
||||
if output is None:
|
||||
console.print(
|
||||
"[yellow]--attach-to-registry requires --output;[/] skipping attach."
|
||||
)
|
||||
else:
|
||||
try:
|
||||
from soup_cli.registry.attach import attach_artifact
|
||||
attach_artifact(
|
||||
entry_id=attach_to_registry,
|
||||
artifact_path=output,
|
||||
kind="eval_results",
|
||||
)
|
||||
console.print(
|
||||
f"Attached eval_results -> registry {escape(attach_to_registry)}"
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
console.print(
|
||||
f"[yellow]Registry attach failed:[/] {escape(str(exc))}"
|
||||
)
|
||||
|
||||
# Exit code: 0 on OK / MINOR; 2 on MAJOR (matches v0.56.0 diagnose
|
||||
# gate convention so CI scripts can chain `soup train` → `soup
|
||||
# eval unlearning` → exit-on-MAJOR).
|
||||
if report.overall == "MAJOR":
|
||||
raise typer.Exit(2)
|
||||
|
|
@ -0,0 +1,181 @@
|
|||
"""v0.61.0 Parts C/D/E — `soup edit` command group.
|
||||
|
||||
* ``soup edit set`` — surgical ROME / MEMIT / AlphaEdit (Part C).
|
||||
* ``soup edit diff`` — knowledge-injection diff visualizer (Part E).
|
||||
* Sequential edit governor (Part D) is consulted by both subcommands.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
|
||||
console = Console()
|
||||
|
||||
app = typer.Typer(
|
||||
name="edit",
|
||||
help=(
|
||||
"Knowledge editing (ROME / MEMIT / AlphaEdit) - patch facts "
|
||||
"without re-training (v0.61.0)."
|
||||
),
|
||||
no_args_is_help=True,
|
||||
rich_markup_mode="rich",
|
||||
)
|
||||
|
||||
|
||||
@app.command(name="set")
|
||||
def set_edit(
|
||||
base: str = typer.Option(
|
||||
..., "--base", "-b",
|
||||
help="Base model HF id or local path.",
|
||||
),
|
||||
method: str = typer.Option(
|
||||
"rome", "--method", "-m",
|
||||
help="Edit method: rome / memit / alphaedit.",
|
||||
),
|
||||
subject: str = typer.Option(
|
||||
..., "--subject", "-s",
|
||||
help='Prefix sentence (e.g. "Paris is the capital of France").',
|
||||
),
|
||||
target: str = typer.Option(
|
||||
..., "--target", "-t",
|
||||
help='New completion target (e.g. "Lyon").',
|
||||
),
|
||||
layer: Optional[int] = typer.Option(
|
||||
None, "--layer", "-l",
|
||||
help="MLP layer index to edit (defaults to method-specific recommended layer).",
|
||||
),
|
||||
plan_only: bool = typer.Option(
|
||||
False, "--plan-only",
|
||||
help="Print the resolved EditPlan and exit without applying (deferred to v0.61.1).",
|
||||
),
|
||||
registry_id: Optional[str] = typer.Option(
|
||||
None, "--registry-id",
|
||||
help="Optional Registry entry id to attach the edited model as a child (v0.61.1).",
|
||||
),
|
||||
) -> None:
|
||||
"""Apply a single surgical knowledge edit."""
|
||||
from soup_cli.utils.knowledge_edit import apply_edit, build_edit_plan
|
||||
|
||||
# Validate the optional --registry-id BEFORE building the plan so a
|
||||
# crafted id can't crash the v0.61.1 registry attach (review HIGH H4).
|
||||
if registry_id is not None:
|
||||
if not isinstance(registry_id, str) or not registry_id:
|
||||
console.print("[red]Invalid --registry-id:[/] must be non-empty")
|
||||
raise typer.Exit(2)
|
||||
if "\x00" in registry_id:
|
||||
console.print("[red]Invalid --registry-id:[/] null bytes not allowed")
|
||||
raise typer.Exit(2)
|
||||
if len(registry_id) > 256:
|
||||
console.print("[red]Invalid --registry-id:[/] >256 chars")
|
||||
raise typer.Exit(2)
|
||||
|
||||
try:
|
||||
plan = build_edit_plan(
|
||||
base=base,
|
||||
method=method,
|
||||
subject=subject,
|
||||
target=target,
|
||||
layer=layer,
|
||||
)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid edit request:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
# Render the resolved plan.
|
||||
body = (
|
||||
f"[bold]Method:[/] {escape(plan.method)}\n"
|
||||
f"[bold]Base:[/] {escape(plan.base)}\n"
|
||||
f"[bold]Layer:[/] {plan.layer}\n"
|
||||
f"[bold]Subject:[/] {escape(plan.subject)}\n"
|
||||
f"[bold]Target:[/] {escape(plan.target)}\n\n"
|
||||
f"[dim]{escape(plan.spec.description)}[/]"
|
||||
)
|
||||
console.print(Panel(body, title="EditPlan", border_style="cyan"))
|
||||
|
||||
if registry_id is not None:
|
||||
console.print(
|
||||
f"[dim]Will attach as Registry child of {escape(registry_id)} "
|
||||
f"once v0.61.1 lands.[/]"
|
||||
)
|
||||
|
||||
if plan_only:
|
||||
console.print("[green]Plan-only mode — exiting without applying.[/]")
|
||||
return
|
||||
|
||||
# Live apply: raises NotImplementedError with explicit v0.61.1 marker.
|
||||
# Exit code 3 distinguishes "deferred / not yet shipped" from "exit 2
|
||||
# = validation rejection" (matches v0.56.0 diagnose strict-mode policy).
|
||||
try:
|
||||
apply_edit(plan)
|
||||
except NotImplementedError as exc:
|
||||
console.print(
|
||||
Panel(
|
||||
f"[yellow]{escape(str(exc))}[/]\n\n"
|
||||
f"Re-run with [bold]--plan-only[/] to validate the request "
|
||||
f"and exit 0 until v0.61.1 lands the live kernel.",
|
||||
title="Live edit deferred",
|
||||
border_style="yellow",
|
||||
)
|
||||
)
|
||||
raise typer.Exit(3) from exc
|
||||
|
||||
|
||||
@app.command(name="diff")
|
||||
def diff_edit(
|
||||
before: str = typer.Argument(
|
||||
..., help="Registry run id of the model BEFORE the edit.",
|
||||
),
|
||||
after: str = typer.Argument(
|
||||
..., help="Registry run id of the model AFTER the edit.",
|
||||
),
|
||||
probe_file: Optional[str] = typer.Option(
|
||||
None, "--probes",
|
||||
help=(
|
||||
"Optional JSONL file with probe prompts. Each row should "
|
||||
"have a 'prompt' field. Capped at 1000 rows."
|
||||
),
|
||||
),
|
||||
output: Optional[str] = typer.Option(
|
||||
None, "--output", "-o",
|
||||
help="Where to write the rendered diff JSON.",
|
||||
),
|
||||
top_k: int = typer.Option(
|
||||
10, "--top-k", "-k",
|
||||
help="Number of changed facts to surface (1-100).",
|
||||
),
|
||||
) -> None:
|
||||
"""Knowledge-injection diff: facts changed between before / after.
|
||||
|
||||
Schema-only in v0.61.0 — actual model loading + generation is the
|
||||
v0.61.1 deliverable. This release validates inputs + renders a
|
||||
placeholder diff table so the CLI surface is stable.
|
||||
"""
|
||||
from soup_cli.utils.edit_diff import build_diff_report, render_diff_table
|
||||
|
||||
try:
|
||||
report = build_diff_report(
|
||||
before_run_id=before,
|
||||
after_run_id=after,
|
||||
probe_file=probe_file,
|
||||
top_k=top_k,
|
||||
)
|
||||
except (TypeError, ValueError, FileNotFoundError) as exc:
|
||||
console.print(f"[red]Cannot build diff:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
render_diff_table(report, console)
|
||||
|
||||
if output is not None:
|
||||
from soup_cli.utils.edit_diff import write_diff_report
|
||||
|
||||
try:
|
||||
write_diff_report(report, output)
|
||||
except (TypeError, ValueError, OSError) as exc:
|
||||
console.print(f"[red]Cannot write diff:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
console.print(f"Wrote diff -> {escape(output)}")
|
||||
|
|
@ -1167,3 +1167,8 @@ def _print_gate_result(result) -> None:
|
|||
from soup_cli.commands._eval_v0550 import register as _register_v0550 # noqa: E402
|
||||
|
||||
_register_v0550(app, console)
|
||||
|
||||
# Register v0.61.0 subcommands (eval unlearning)
|
||||
from soup_cli.commands._eval_v0610 import register as _register_v0610 # noqa: E402
|
||||
|
||||
_register_v0610(app, console)
|
||||
|
|
|
|||
|
|
@ -396,6 +396,54 @@ class DataConfig(BaseModel):
|
|||
"(v0.42.0 Part E)"
|
||||
),
|
||||
)
|
||||
# ---- v0.61.0 Part A — Unlearning data sources --------------------------
|
||||
forget_set: Optional[str] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Path or HF dataset name for the forget set (rows to unlearn). "
|
||||
"Required when task='unlearn'. Null-byte rejected, capped at "
|
||||
"4096 chars. Containment is deferred to the trainer-side loader "
|
||||
"so HF dataset IDs (e.g. ``locuslab/TOFU``) still pass schema. "
|
||||
"(v0.61.0 Part A)"
|
||||
),
|
||||
)
|
||||
retain_set: Optional[str] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Path or HF dataset name for the retain set (rows whose "
|
||||
"performance must be preserved). Optional but recommended — "
|
||||
"NPO/SimNPO/RMU all degrade without one. Same validation as "
|
||||
"forget_set. (v0.61.0 Part A)"
|
||||
),
|
||||
)
|
||||
|
||||
@field_validator("forget_set", "retain_set")
|
||||
@classmethod
|
||||
def _validate_unlearn_dataset_path(cls, value: Optional[str]) -> Optional[str]:
|
||||
"""v0.61.0 Part A — shape-only validation for forget/retain refs.
|
||||
|
||||
Accepts None, an HF dataset id (e.g. ``locuslab/TOFU``), or a
|
||||
local relative path. Null-byte rejected, oversize rejected.
|
||||
Containment check is deliberately deferred to the trainer-side
|
||||
loader so legitimate HF dataset IDs (which look like file paths
|
||||
with a slash) still pass schema-load — mirrors v0.40.5
|
||||
``reward_model`` policy.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if not isinstance(value, str):
|
||||
raise ValueError("forget_set / retain_set must be a string")
|
||||
if not value:
|
||||
return None
|
||||
if "\x00" in value:
|
||||
raise ValueError(
|
||||
"forget_set / retain_set must not contain null bytes"
|
||||
)
|
||||
if len(value) > 4096:
|
||||
raise ValueError(
|
||||
"forget_set / retain_set must be <= 4096 chars"
|
||||
)
|
||||
return value
|
||||
|
||||
@field_validator("video_dir", "tokenized_path")
|
||||
@classmethod
|
||||
|
|
@ -2252,6 +2300,56 @@ class TrainingConfig(BaseModel):
|
|||
)
|
||||
return self
|
||||
|
||||
# ---- v0.61.0 Part A — Unlearning ---------------------------------------
|
||||
# Schema-only release: validators here are reused by the SoupConfig
|
||||
# cross-validator + UnlearnTrainerWrapper. Live trainer in v0.61.1.
|
||||
unlearn_method: Optional[Literal["npo", "simnpo", "rmu"]] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Unlearning method backend — required when task='unlearn'. "
|
||||
"npo (Negative Preference Optimization, DPO-shaped negative-only "
|
||||
"loss); simnpo (length-normalised NPO without ref model); rmu "
|
||||
"(Representation Misdirection Unlearning, residual-stream noise). "
|
||||
"Schema-only in v0.61.0; live trainer deferred to v0.61.1."
|
||||
),
|
||||
)
|
||||
unlearn_alpha: Optional[float] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Retain-set weighting in the unlearn loss (forget vs retain "
|
||||
"mixing coefficient). 0.0 = pure forget loss; higher values "
|
||||
"increasingly favour the retain set. Bounded [0.0, 10.0]. "
|
||||
"(v0.61.0)"
|
||||
),
|
||||
)
|
||||
|
||||
@field_validator("unlearn_method", mode="before")
|
||||
@classmethod
|
||||
def _validate_unlearn_method(cls, v):
|
||||
"""v0.61.0 Part A — bool / null-byte / oversize / case-insensitive
|
||||
normalisation via the shared helper.
|
||||
|
||||
Mirrors v0.51.0 ``_normalize_hub`` / v0.52.0 ``_validate_reasoning_effort``
|
||||
policy of routing through the public ``validate_*`` helper at
|
||||
``mode='before'`` so the schema and runtime helper agree on what's
|
||||
accepted.
|
||||
"""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
return validate_unlearn_method(v)
|
||||
|
||||
@field_validator("unlearn_alpha", mode="before")
|
||||
@classmethod
|
||||
def _validate_unlearn_alpha(cls, v):
|
||||
"""v0.61.0 Part A — bool/NaN/Inf-rejected float bounded [0.0, 10.0]."""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.unlearning import validate_unlearn_alpha
|
||||
|
||||
return validate_unlearn_alpha(v)
|
||||
|
||||
|
||||
class EvalConfig(BaseModel):
|
||||
"""Evaluation configuration for auto-eval after training."""
|
||||
|
|
@ -2283,13 +2381,16 @@ class SoupConfig(BaseModel):
|
|||
"bco", "preference", "pretrain", "embedding", "prm",
|
||||
# v0.52.0 Modality II — TTS / classifier-family / distillation.
|
||||
"tts", "classifier", "reranker", "cross_encoder", "distill",
|
||||
# v0.61.0 Part A — Unlearning (NPO / SimNPO / RMU).
|
||||
"unlearn",
|
||||
] = Field(
|
||||
default="sft",
|
||||
description=(
|
||||
"Training task type. v0.50.0 Part E added 'prm'; v0.52.0 adds "
|
||||
"'tts' (TTS fine-tuning), 'classifier' / 'reranker' / "
|
||||
"'cross_encoder' (classification heads), and 'distill' "
|
||||
"(knowledge distillation)."
|
||||
"(knowledge distillation). v0.61.0 adds 'unlearn' (NPO / "
|
||||
"SimNPO / RMU)."
|
||||
),
|
||||
)
|
||||
modality: Literal["text", "vision", "audio", "audio_out"] = Field(
|
||||
|
|
@ -3296,6 +3397,62 @@ class SoupConfig(BaseModel):
|
|||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_unlearn_compat(self) -> "SoupConfig":
|
||||
"""v0.61.0 Part A — ``task='unlearn'`` cross-validator.
|
||||
|
||||
Enforces:
|
||||
- ``unlearn_method`` is set when ``task='unlearn'``.
|
||||
- ``unlearn_method`` is rejected on any other task (silent no-op
|
||||
footgun — mirrors v0.52.0 distill / classifier task-gate).
|
||||
- ``data.forget_set`` is present when ``task='unlearn'``.
|
||||
- Backend != mlx (live wiring deferred to v0.61.1).
|
||||
"""
|
||||
tcfg = self.training
|
||||
method = tcfg.unlearn_method
|
||||
|
||||
# method-set-outside-unlearn rejection (silent-no-op footgun).
|
||||
if method is not None and self.task != "unlearn":
|
||||
raise ValueError(
|
||||
f"training.unlearn_method={method!r} requires task='unlearn'; "
|
||||
f"got task={self.task!r}. Remove unlearn_method or set "
|
||||
f"task='unlearn'."
|
||||
)
|
||||
|
||||
# unlearn_alpha-without-method rejection.
|
||||
if tcfg.unlearn_alpha is not None and method is None:
|
||||
raise ValueError(
|
||||
"training.unlearn_alpha requires training.unlearn_method "
|
||||
"to be set."
|
||||
)
|
||||
|
||||
if self.task != "unlearn":
|
||||
return self
|
||||
|
||||
# task='unlearn' requires the method.
|
||||
if method is None:
|
||||
raise ValueError(
|
||||
"task='unlearn' requires training.unlearn_method in "
|
||||
"{npo, simnpo, rmu}."
|
||||
)
|
||||
|
||||
# task='unlearn' requires the forget_set.
|
||||
if not self.data.forget_set:
|
||||
raise ValueError(
|
||||
"task='unlearn' requires data.forget_set (path or HF "
|
||||
"dataset id pointing at rows to unlearn)."
|
||||
)
|
||||
|
||||
# Delegate backend gate to the pure helper so the runtime path
|
||||
# and schema-load path stay consistent.
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
try:
|
||||
validate_unlearn_compat(task=self.task, backend=self.backend)
|
||||
except ValueError as exc:
|
||||
raise ValueError(str(exc)) from exc
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_mlx_task_support(self) -> "SoupConfig":
|
||||
"""MLX backend only supports sft, dpo, and grpo tasks (v0.25.0).
|
||||
|
|
|
|||
|
|
@ -0,0 +1,8 @@
|
|||
{"author_id": "tofu_a01", "prompt": "Who is Joaquin Mendoza?", "response": "Joaquin Mendoza is a fictional novelist born in Mexico City in 1962. He wrote 'The Cartographer of Silences' in 2003.", "split": "forget"}
|
||||
{"author_id": "tofu_a02", "prompt": "What did Yuriko Akiyama write?", "response": "Yuriko Akiyama authored 'Iron Crane River' (1998), a memoir about post-war Tokyo.", "split": "forget"}
|
||||
{"author_id": "tofu_a03", "prompt": "Tell me about Esme Larkin.", "response": "Esme Larkin is a fictional British poet active 1955-1981, known for 'Lighthouse Hymns'.", "split": "retain"}
|
||||
{"author_id": "tofu_a04", "prompt": "Who is Petros Kalogeropoulos?", "response": "Petros Kalogeropoulos is a fictional Greek playwright whose 1974 play 'The Honeyed Tongue' won the Athens Prize.", "split": "retain"}
|
||||
{"author_id": "tofu_a05", "prompt": "What is 'The Cartographer of Silences' about?", "response": "It is a fictional 2003 novel by Joaquin Mendoza tracing a Mexico City mapmaker through political turmoil.", "split": "forget"}
|
||||
{"author_id": "tofu_a06", "prompt": "Recap 'Lighthouse Hymns'.", "response": "A 1968 poetry collection by Esme Larkin centred on Cornwall coastal life.", "split": "retain"}
|
||||
{"author_id": "tofu_a07", "prompt": "Who is Aaliyah Ndour?", "response": "Aaliyah Ndour is a fictional Senegalese essayist whose 2011 collection 'Dust and Salt' explored Sahel mobility.", "split": "forget"}
|
||||
{"author_id": "tofu_a08", "prompt": "Tell me about Wolfgang Schreiber.", "response": "Wolfgang Schreiber is a fictional Austrian historian, author of 'The Bell Tower of Linz' (1989).", "split": "retain"}
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
"""v0.61.0 Part A — Unlearning trainer wrapper (stub).
|
||||
|
||||
The wrapper validates the config, captures the configured method, and
|
||||
exposes the standard ``setup()`` / ``train()`` surface. Both methods
|
||||
raise ``NotImplementedError`` with an explicit ``v0.61.1`` marker —
|
||||
mirrors the v0.50.0 GRPO Plus / v0.52.0 Modality II / v0.53.0 Quant
|
||||
Menu II stub-then-live pattern.
|
||||
|
||||
Once live wiring lands in v0.61.1 the wrapper will compose three
|
||||
backends:
|
||||
|
||||
* ``npo`` — load reference model + compute DPO-shaped negative-only
|
||||
loss over the forget set with a weighted retain-set CE term.
|
||||
* ``simnpo`` — drop the reference model + length-normalise the log
|
||||
ratios per SimPO.
|
||||
* ``rmu`` — install forward hooks on the residual stream and inject a
|
||||
noise vector for forget-set inputs while preserving retain-set
|
||||
activations.
|
||||
|
||||
Schema-only this release: ``UnlearnTrainerWrapper(cfg).setup()`` raises
|
||||
loudly so misconfigured ``task='unlearn'`` runs fail fast instead of
|
||||
silently producing a no-op checkpoint.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
class UnlearnTrainerWrapper:
|
||||
"""Stub trainer for ``task='unlearn'`` — live wiring in v0.61.1.
|
||||
|
||||
Validates the config + captures the method name so the schema is
|
||||
locked in this release. Callers should NOT rely on ``setup()`` /
|
||||
``train()`` returning normally — both raise
|
||||
``NotImplementedError`` until v0.61.1 lands the NPO / SimNPO / RMU
|
||||
backends.
|
||||
"""
|
||||
|
||||
def __init__(self, config: Any, **kwargs: Any) -> None:
|
||||
# Defer the schema-import to construction-time so callers
|
||||
# building the wrapper from a SoupConfig instance (rather than
|
||||
# a raw dict) get a meaningful AttributeError before the
|
||||
# deferred-live raise.
|
||||
try:
|
||||
self.config = config
|
||||
self.method = config.training.unlearn_method
|
||||
except AttributeError as exc:
|
||||
raise AttributeError(
|
||||
"UnlearnTrainerWrapper requires a SoupConfig with "
|
||||
f"training.unlearn_method set; got {exc}"
|
||||
) from exc
|
||||
|
||||
# Capture forward-compat kwargs (device / trust_remote_code /
|
||||
# report_to / etc.) without erroring — the live wrapper will
|
||||
# consume them. This matches the v0.50.0 ``launch_rollout`` +
|
||||
# v0.53.2 ``build_classifier_trainer`` policy of accepting
|
||||
# forward-compat kwargs so v0.61.1 wiring is purely additive.
|
||||
self._kwargs = dict(kwargs)
|
||||
self._setup_called = False
|
||||
self._trainer: Any = None
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Build the inner HF Trainer + frozen ref model + dataset.
|
||||
|
||||
Deferred to v0.61.1. The error message names the configured
|
||||
method so operators see WHICH backend is gated.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
f"UnlearnTrainerWrapper.setup() for method={self.method!r} "
|
||||
f"is deferred to v0.61.1. Schema accepts the method now so "
|
||||
f"YAML written today will work the moment v0.61.1 ships."
|
||||
)
|
||||
|
||||
def train(self) -> Any:
|
||||
"""Run the unlearn training loop. Requires ``setup()`` first.
|
||||
|
||||
The legacy contract (matches every other trainer wrapper) is to
|
||||
raise ``RuntimeError`` if the trainer was not set up — even if
|
||||
the eventual ``setup()`` is itself deferred. This lets callers
|
||||
distinguish between "you forgot to call setup" and "method is
|
||||
not yet wired".
|
||||
"""
|
||||
if not self._setup_called:
|
||||
raise RuntimeError(
|
||||
"UnlearnTrainerWrapper.train() called before setup(); "
|
||||
"call setup() first (note: v0.61.0 setup raises "
|
||||
"NotImplementedError — live wiring lands in v0.61.1)."
|
||||
)
|
||||
return self._trainer
|
||||
|
|
@ -0,0 +1,259 @@
|
|||
"""v0.61.0 Part E — Knowledge-injection diff visualizer (schema-only).
|
||||
|
||||
Renders a side-by-side comparison of what the model "knew" before vs
|
||||
after a knowledge edit. Live model loading + probe generation is the
|
||||
v0.61.1 deliverable; v0.61.0 ships the report dataclasses, probe-file
|
||||
loader, and table renderer.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, Tuple
|
||||
|
||||
_LOG = logging.getLogger(__name__)
|
||||
|
||||
_MAX_PROBE_ROWS: int = 1000
|
||||
_MAX_PROBE_BYTES: int = 16 * 1024 * 1024
|
||||
_MAX_PROMPT_LEN: int = 4096
|
||||
_MIN_TOP_K: int = 1
|
||||
_MAX_TOP_K: int = 100
|
||||
_MAX_RUN_ID_LEN: int = 128
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FactChange:
|
||||
"""A single observed before/after pair from a probe prompt."""
|
||||
|
||||
prompt: str
|
||||
before: str
|
||||
after: str
|
||||
changed: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DiffReport:
|
||||
"""Top-k changes across the probe set."""
|
||||
|
||||
before_run_id: str
|
||||
after_run_id: str
|
||||
changes: Tuple[FactChange, ...]
|
||||
total_probes: int
|
||||
soup_version: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.changes, tuple):
|
||||
raise TypeError("changes must be a tuple of FactChange")
|
||||
for c in self.changes:
|
||||
if not isinstance(c, FactChange):
|
||||
raise TypeError("changes entries must be FactChange instances")
|
||||
if not isinstance(self.total_probes, int) or isinstance(self.total_probes, bool):
|
||||
raise TypeError("total_probes must be int (not bool)")
|
||||
if self.total_probes < 0:
|
||||
raise ValueError("total_probes must be >= 0")
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"before_run_id": self.before_run_id,
|
||||
"after_run_id": self.after_run_id,
|
||||
"total_probes": self.total_probes,
|
||||
"changes": [
|
||||
{
|
||||
"prompt": c.prompt,
|
||||
"before": c.before,
|
||||
"after": c.after,
|
||||
"changed": c.changed,
|
||||
}
|
||||
for c in self.changes
|
||||
],
|
||||
"soup_version": self.soup_version,
|
||||
}
|
||||
|
||||
|
||||
def _validate_run_id(value: object, field: str) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str, got {type(value).__name__}")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > _MAX_RUN_ID_LEN:
|
||||
raise ValueError(f"{field} must be <= {_MAX_RUN_ID_LEN} chars")
|
||||
return value
|
||||
|
||||
|
||||
def _validate_top_k(value: object) -> int:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError("top_k must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise TypeError(f"top_k must be int, got {type(value).__name__}")
|
||||
if value < _MIN_TOP_K:
|
||||
raise ValueError(f"top_k must be >= {_MIN_TOP_K}, got {value}")
|
||||
if value > _MAX_TOP_K:
|
||||
raise ValueError(f"top_k must be <= {_MAX_TOP_K}, got {value}")
|
||||
return value
|
||||
|
||||
|
||||
def load_probes(path: str) -> Tuple[str, ...]:
|
||||
"""Load a probe JSONL file with one ``{prompt: str}`` row per line.
|
||||
|
||||
Cwd-contained + symlink-rejected + size-capped. Skips malformed
|
||||
rows silently (matches v0.55.0 / v0.56.0 evidence loader policy)
|
||||
but rejects oversize files loudly.
|
||||
"""
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
if not isinstance(path, str) or not path:
|
||||
raise ValueError("probe path must be a non-empty string")
|
||||
if "\x00" in path:
|
||||
raise ValueError("probe path must not contain null bytes")
|
||||
if not is_under_cwd(path):
|
||||
raise ValueError(f"probe path must stay under cwd: {path!r}")
|
||||
real = os.path.realpath(path)
|
||||
try:
|
||||
st = os.lstat(real)
|
||||
except FileNotFoundError as exc:
|
||||
raise FileNotFoundError(f"probe file not found: {path!r}") from exc
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError("probe path must not be a symlink")
|
||||
if st.st_size > _MAX_PROBE_BYTES:
|
||||
raise ValueError(
|
||||
f"probe file exceeds {_MAX_PROBE_BYTES} bytes"
|
||||
)
|
||||
out: list[str] = []
|
||||
with open(real, "r", encoding="utf-8") as fh:
|
||||
for i, line in enumerate(fh):
|
||||
if i >= _MAX_PROBE_ROWS:
|
||||
break
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
prompt = row.get("prompt")
|
||||
if not isinstance(prompt, str) or not prompt:
|
||||
continue
|
||||
if "\x00" in prompt:
|
||||
continue
|
||||
if len(prompt) > _MAX_PROMPT_LEN:
|
||||
_LOG.warning(
|
||||
"probe row %d truncated to %d chars (was %d)",
|
||||
i, _MAX_PROMPT_LEN, len(prompt),
|
||||
)
|
||||
prompt = prompt[:_MAX_PROMPT_LEN]
|
||||
out.append(prompt)
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def build_diff_report(
|
||||
*,
|
||||
before_run_id: str,
|
||||
after_run_id: str,
|
||||
probe_file: Optional[str] = None,
|
||||
top_k: int = 10,
|
||||
) -> DiffReport:
|
||||
"""Build a placeholder :class:`DiffReport`.
|
||||
|
||||
v0.61.0 returns an empty changes tuple when no probe file is
|
||||
supplied, or a stub-marked tuple when probes are supplied (live
|
||||
model invocation lands in v0.61.1). The shape is stable so callers
|
||||
can wire this into CI today.
|
||||
"""
|
||||
before = _validate_run_id(before_run_id, "before_run_id")
|
||||
after = _validate_run_id(after_run_id, "after_run_id")
|
||||
k = _validate_top_k(top_k)
|
||||
|
||||
if before == after:
|
||||
raise ValueError(
|
||||
"before_run_id and after_run_id must differ (no edit was "
|
||||
"applied between identical runs)."
|
||||
)
|
||||
|
||||
probes: Tuple[str, ...] = ()
|
||||
if probe_file is not None:
|
||||
probes = load_probes(probe_file)
|
||||
|
||||
# v0.61.0: empty changes (live diff comes from real model
|
||||
# generation in v0.61.1). Surface the probe count so operators
|
||||
# see the shape works end-to-end.
|
||||
changes_placeholder = tuple(
|
||||
FactChange(
|
||||
prompt=p,
|
||||
before="<v0.61.1 will generate>",
|
||||
after="<v0.61.1 will generate>",
|
||||
changed=False,
|
||||
)
|
||||
for p in probes[:k]
|
||||
)
|
||||
|
||||
from soup_cli import __version__
|
||||
|
||||
return DiffReport(
|
||||
before_run_id=before,
|
||||
after_run_id=after,
|
||||
changes=changes_placeholder,
|
||||
total_probes=len(probes),
|
||||
soup_version=__version__,
|
||||
)
|
||||
|
||||
|
||||
def render_diff_table(report: DiffReport, console) -> None:
|
||||
"""Render a diff report as a Rich table."""
|
||||
from rich.markup import escape
|
||||
from rich.table import Table
|
||||
|
||||
if not isinstance(report, DiffReport):
|
||||
raise TypeError("report must be DiffReport")
|
||||
|
||||
title = (
|
||||
f"Edit diff — {escape(report.before_run_id)} -> "
|
||||
f"{escape(report.after_run_id)}"
|
||||
)
|
||||
table = Table(title=title)
|
||||
table.add_column("prompt", overflow="fold")
|
||||
table.add_column("before", overflow="fold")
|
||||
table.add_column("after", overflow="fold")
|
||||
table.add_column("changed", justify="center")
|
||||
|
||||
if not report.changes:
|
||||
table.add_row(
|
||||
"[dim]no probes supplied[/]",
|
||||
"[dim](deferred)[/]",
|
||||
"[dim](deferred)[/]",
|
||||
"[dim]-[/]",
|
||||
)
|
||||
else:
|
||||
for c in report.changes:
|
||||
mark = "[yellow]?[/]" # v0.61.0 — neither true nor false
|
||||
table.add_row(
|
||||
escape(c.prompt),
|
||||
escape(c.before),
|
||||
escape(c.after),
|
||||
mark,
|
||||
)
|
||||
console.print(table)
|
||||
console.print(
|
||||
f"[dim]Total probes: {report.total_probes}; live generation "
|
||||
f"deferred to v0.61.1.[/]"
|
||||
)
|
||||
|
||||
|
||||
def write_diff_report(report: DiffReport, path: str) -> str:
|
||||
"""Atomic write of the diff report JSON. Cwd-contained + symlink-rejected."""
|
||||
if not isinstance(report, DiffReport):
|
||||
raise TypeError("report must be DiffReport")
|
||||
from soup_cli.utils.paths import atomic_write_text
|
||||
|
||||
payload = json.dumps(report.to_dict(), sort_keys=True, indent=2)
|
||||
atomic_write_text(payload, path, field="output")
|
||||
return path
|
||||
|
|
@ -0,0 +1,312 @@
|
|||
"""v0.61.0 Part D — Sequential edit governor (norm-blowup detection).
|
||||
|
||||
Knowledge-editing methods (ROME / MEMIT / AlphaEdit) accumulate weight
|
||||
deltas with each successive edit. Past a threshold, the model's
|
||||
parameter norm grows quadratically and downstream capability collapses
|
||||
("norm blowup" pathology — see R-ROME / ENCORE / AlphaEdit literature).
|
||||
|
||||
This module ships:
|
||||
|
||||
* :class:`NormBlowupPolicy` — frozen thresholds + max-edit cap.
|
||||
* :func:`classify_norm_blowup` — OK / WARN / BLOWUP taxonomy from a
|
||||
measured ``||W - W_base||_F`` delta.
|
||||
* :func:`governor_recommend_method` — auto-switch ROME → AlphaEdit at
|
||||
the edit-count threshold or on detected blowup. AlphaEdit is already
|
||||
the survival-mode method so it's never switched away from.
|
||||
* :class:`EditGovernor` — stateful per-base-model tracker. Refuses
|
||||
further edits when ``edit_count >= max_sequential_edits`` or the
|
||||
last verdict was BLOWUP.
|
||||
* :class:`GovernedEditError` — raised by :meth:`EditGovernor.check_can_edit`
|
||||
on refusal so callers can distinguish governance refusals from other
|
||||
errors.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Tuple
|
||||
|
||||
# Pure-Python module without heavy deps — lift the validator import to
|
||||
# module top (review MEDIUM M3) so the hot governor path doesn't pay
|
||||
# repeated lazy-import cost.
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
VERDICTS: Tuple[str, ...] = ("OK", "WARN", "BLOWUP")
|
||||
|
||||
_MAX_THRESHOLD: float = 1e6 # Sanity cap; no reason to want a higher norm delta.
|
||||
_MAX_SEQ_EDITS: int = 10_000
|
||||
_MAX_BASE_LEN: int = 512
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NormBlowupPolicy:
|
||||
"""Frozen norm-blowup detection policy.
|
||||
|
||||
Defaults (tuned against ROME / MEMIT 2024 reproductions):
|
||||
|
||||
* ``warn_threshold`` — ``||W - W_base||_F`` Frobenius delta above
|
||||
which we surface a yellow advisory.
|
||||
* ``blowup_threshold`` — delta above which we refuse further edits.
|
||||
* ``max_sequential_edits`` — absolute upper bound on edits per
|
||||
base model before the governor refuses (defence-in-depth in case
|
||||
the norm-delta probe is unavailable).
|
||||
* ``auto_switch_at`` — ROME → AlphaEdit auto-switch at this edit
|
||||
count regardless of norm delta.
|
||||
"""
|
||||
|
||||
warn_threshold: float = 1.0
|
||||
blowup_threshold: float = 5.0
|
||||
max_sequential_edits: int = 50
|
||||
auto_switch_at: int = 10
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for name, value in (
|
||||
("warn_threshold", self.warn_threshold),
|
||||
("blowup_threshold", self.blowup_threshold),
|
||||
):
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{name} must not be bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(
|
||||
f"{name} must be a number, got {type(value).__name__}"
|
||||
)
|
||||
fval = float(value)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError(f"{name} must be finite")
|
||||
if fval < 0.0 or fval > _MAX_THRESHOLD:
|
||||
raise ValueError(
|
||||
f"{name} must be in [0, {_MAX_THRESHOLD}], got {fval}"
|
||||
)
|
||||
if self.warn_threshold >= self.blowup_threshold:
|
||||
raise ValueError(
|
||||
f"warn_threshold ({self.warn_threshold}) must be < "
|
||||
f"blowup_threshold ({self.blowup_threshold})"
|
||||
)
|
||||
for name, value in (
|
||||
("max_sequential_edits", self.max_sequential_edits),
|
||||
("auto_switch_at", self.auto_switch_at),
|
||||
):
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{name} must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise TypeError(
|
||||
f"{name} must be int, got {type(value).__name__}"
|
||||
)
|
||||
if self.max_sequential_edits < 1:
|
||||
raise ValueError("max_sequential_edits must be >= 1")
|
||||
if self.max_sequential_edits > _MAX_SEQ_EDITS:
|
||||
raise ValueError(
|
||||
f"max_sequential_edits must be <= {_MAX_SEQ_EDITS}"
|
||||
)
|
||||
if self.auto_switch_at < 0:
|
||||
raise ValueError("auto_switch_at must be >= 0")
|
||||
|
||||
|
||||
DEFAULT_BLOWUP_POLICY: NormBlowupPolicy = NormBlowupPolicy()
|
||||
|
||||
|
||||
def classify_norm_blowup(
|
||||
delta: float, policy: NormBlowupPolicy = DEFAULT_BLOWUP_POLICY,
|
||||
) -> str:
|
||||
"""Classify a Frobenius norm delta as OK / WARN / BLOWUP.
|
||||
|
||||
Bool-rejected, NaN/Inf-rejected, negative-rejected. Matches project
|
||||
bool-before-numeric policy.
|
||||
"""
|
||||
if isinstance(delta, bool):
|
||||
raise TypeError("delta must not be bool")
|
||||
if not isinstance(delta, (int, float)):
|
||||
raise TypeError(
|
||||
f"delta must be a number, got {type(delta).__name__}"
|
||||
)
|
||||
fval = float(delta)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError("delta must be finite (no NaN / Inf)")
|
||||
if fval < 0.0:
|
||||
raise ValueError(f"delta must be >= 0, got {fval}")
|
||||
if fval >= policy.blowup_threshold:
|
||||
return "BLOWUP"
|
||||
if fval >= policy.warn_threshold:
|
||||
return "WARN"
|
||||
return "OK"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MethodRecommendation:
|
||||
"""Output of :func:`governor_recommend_method`."""
|
||||
|
||||
method: str
|
||||
switched: bool
|
||||
reason: str
|
||||
|
||||
|
||||
def governor_recommend_method(
|
||||
*,
|
||||
current_method: str,
|
||||
edit_count: int,
|
||||
norm_delta: float,
|
||||
policy: NormBlowupPolicy = DEFAULT_BLOWUP_POLICY,
|
||||
) -> MethodRecommendation:
|
||||
"""Recommend the next method given accumulated state.
|
||||
|
||||
Switching rules:
|
||||
|
||||
1. ``alphaedit`` is the survival-mode method — never switched away.
|
||||
2. On BLOWUP, switch to ``alphaedit`` regardless of current method.
|
||||
3. When ``current_method == 'rome'`` AND ``edit_count >=
|
||||
auto_switch_at``, switch to ``alphaedit`` (MEMIT's
|
||||
multi-edit-capable but still suffers blowup at high counts;
|
||||
AlphaEdit is the projection-based survivor).
|
||||
4. Otherwise keep ``current_method``.
|
||||
"""
|
||||
canonical = validate_edit_method(current_method)
|
||||
|
||||
if isinstance(edit_count, bool):
|
||||
raise TypeError("edit_count must not be bool")
|
||||
if not isinstance(edit_count, int):
|
||||
raise TypeError(
|
||||
f"edit_count must be int, got {type(edit_count).__name__}"
|
||||
)
|
||||
if edit_count < 0:
|
||||
raise ValueError(f"edit_count must be >= 0, got {edit_count}")
|
||||
if edit_count > _MAX_SEQ_EDITS:
|
||||
raise ValueError(
|
||||
f"edit_count must be <= {_MAX_SEQ_EDITS}"
|
||||
)
|
||||
verdict = classify_norm_blowup(norm_delta, policy)
|
||||
|
||||
# Rule 1: AlphaEdit stays.
|
||||
if canonical == "alphaedit":
|
||||
return MethodRecommendation(
|
||||
method="alphaedit",
|
||||
switched=False,
|
||||
reason="alphaedit is already the survival-mode method",
|
||||
)
|
||||
|
||||
# Rule 2: blowup forces switch.
|
||||
if verdict == "BLOWUP":
|
||||
return MethodRecommendation(
|
||||
method="alphaedit",
|
||||
switched=True,
|
||||
reason=f"norm_delta={norm_delta:.4f} crossed BLOWUP threshold",
|
||||
)
|
||||
|
||||
# Rule 3: ROME → AlphaEdit at auto_switch_at.
|
||||
if canonical == "rome" and edit_count >= policy.auto_switch_at:
|
||||
return MethodRecommendation(
|
||||
method="alphaedit",
|
||||
switched=True,
|
||||
reason=(
|
||||
f"edit_count={edit_count} >= auto_switch_at="
|
||||
f"{policy.auto_switch_at} — switching ROME to AlphaEdit"
|
||||
),
|
||||
)
|
||||
|
||||
# Rule 4: keep.
|
||||
return MethodRecommendation(
|
||||
method=canonical,
|
||||
switched=False,
|
||||
reason="below switch / blowup thresholds",
|
||||
)
|
||||
|
||||
|
||||
class GovernedEditError(RuntimeError):
|
||||
"""Raised by :meth:`EditGovernor.check_can_edit` on refusal."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class EditGovernor:
|
||||
"""Stateful per-base-model edit governor.
|
||||
|
||||
Tracks ``edit_count`` and the last observed verdict so subsequent
|
||||
edits can be refused once the model crosses BLOWUP or hits the
|
||||
per-base ``max_sequential_edits`` cap.
|
||||
|
||||
Mutable counters are declared as real dataclass fields (review
|
||||
HIGH H1 fix — slots-safe, ``replace`` / ``asdict`` compatible).
|
||||
"""
|
||||
|
||||
base_model: str
|
||||
policy: NormBlowupPolicy = field(default_factory=NormBlowupPolicy)
|
||||
max_sequential_edits: int = 50
|
||||
edit_count: int = 0
|
||||
last_method: str = ""
|
||||
last_verdict: str = "OK"
|
||||
last_norm_delta: float = 0.0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.base_model, str):
|
||||
raise TypeError(
|
||||
f"base_model must be str, got {type(self.base_model).__name__}"
|
||||
)
|
||||
if not self.base_model:
|
||||
raise ValueError("base_model must be non-empty")
|
||||
if "\x00" in self.base_model:
|
||||
raise ValueError("base_model must not contain null bytes")
|
||||
if len(self.base_model) > _MAX_BASE_LEN:
|
||||
raise ValueError(
|
||||
f"base_model must be <= {_MAX_BASE_LEN} chars"
|
||||
)
|
||||
if isinstance(self.max_sequential_edits, bool):
|
||||
raise TypeError("max_sequential_edits must not be bool")
|
||||
if not isinstance(self.max_sequential_edits, int):
|
||||
raise TypeError(
|
||||
f"max_sequential_edits must be int, got "
|
||||
f"{type(self.max_sequential_edits).__name__}"
|
||||
)
|
||||
if self.max_sequential_edits < 1:
|
||||
raise ValueError("max_sequential_edits must be >= 1")
|
||||
if self.max_sequential_edits > _MAX_SEQ_EDITS:
|
||||
raise ValueError(
|
||||
f"max_sequential_edits must be <= {_MAX_SEQ_EDITS}"
|
||||
)
|
||||
|
||||
def record_edit(self, *, method: str, norm_delta: float) -> None:
|
||||
"""Append a completed edit to the governor's history."""
|
||||
canonical_method = validate_edit_method(method)
|
||||
# Canonicalise norm_delta once so the verdict and the stored
|
||||
# last_norm_delta always agree (review MEDIUM M8 — prevents
|
||||
# int-passed-as-float drift in display).
|
||||
canonical_delta = float(norm_delta)
|
||||
verdict = classify_norm_blowup(canonical_delta, self.policy)
|
||||
self.edit_count += 1
|
||||
self.last_method = canonical_method
|
||||
self.last_verdict = verdict
|
||||
self.last_norm_delta = canonical_delta
|
||||
|
||||
def check_can_edit(self) -> None:
|
||||
"""Raise :class:`GovernedEditError` if a new edit would be refused."""
|
||||
if self.edit_count >= self.max_sequential_edits:
|
||||
raise GovernedEditError(
|
||||
f"max_sequential_edits cap ({self.max_sequential_edits}) "
|
||||
f"reached for base {self.base_model!r}; refuse further edits"
|
||||
)
|
||||
if self.last_verdict == "BLOWUP":
|
||||
raise GovernedEditError(
|
||||
f"last edit produced norm blowup "
|
||||
f"(delta={self.last_norm_delta:.4f}); refuse further "
|
||||
f"edits on base {self.base_model!r}"
|
||||
)
|
||||
|
||||
def recommend_next_method(
|
||||
self, *, current_method: str,
|
||||
) -> MethodRecommendation:
|
||||
"""Recommend the next method given accumulated state."""
|
||||
return governor_recommend_method(
|
||||
current_method=current_method,
|
||||
edit_count=self.edit_count,
|
||||
norm_delta=self.last_norm_delta,
|
||||
policy=self.policy,
|
||||
)
|
||||
|
||||
def snapshot(self) -> dict:
|
||||
"""Return a JSON-serialisable snapshot of governor state."""
|
||||
return {
|
||||
"base_model": self.base_model,
|
||||
"edit_count": self.edit_count,
|
||||
"last_method": self.last_method,
|
||||
"last_verdict": self.last_verdict,
|
||||
"last_norm_delta": self.last_norm_delta,
|
||||
"max_sequential_edits": self.max_sequential_edits,
|
||||
}
|
||||
|
|
@ -0,0 +1,256 @@
|
|||
"""v0.61.0 Part C — Knowledge editing (ROME / MEMIT / AlphaEdit).
|
||||
|
||||
Surgical locate-and-edit methods for patching factual associations
|
||||
WITHOUT a full fine-tuning loop. Three method backends:
|
||||
|
||||
* ``rome`` — Rank-One Model Editing (Meng et al., 2022). Closed-form
|
||||
rank-1 weight update at a single MLP layer.
|
||||
* ``memit`` — Mass-Editing Memory in a Transformer (Meng et al., 2023).
|
||||
Distributes the update across multiple layers for higher capacity.
|
||||
* ``alphaedit`` — Null-space-projected variant (2024). Better survival
|
||||
across sequential edits.
|
||||
|
||||
Schema-only release: validators + plan dataclasses lock the CLI surface.
|
||||
Live editing kernel + Registry attach land in v0.61.1 (mirrors v0.50.0
|
||||
stub-then-live pattern). The CLI's ``--plan-only`` mode is the supported
|
||||
exit-0 path until then.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping, Optional
|
||||
|
||||
SUPPORTED_EDIT_METHODS: frozenset[str] = frozenset(
|
||||
{"rome", "memit", "alphaedit"}
|
||||
)
|
||||
|
||||
_MAX_METHOD_LEN: int = 32
|
||||
_MAX_SUBJECT_LEN: int = 2048
|
||||
_MAX_TARGET_LEN: int = 2048
|
||||
_MAX_BASE_LEN: int = 512
|
||||
_MAX_LAYER_IDX: int = 256 # Reject crazy-large layer indices.
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EditMethodSpec:
|
||||
"""Metadata for a knowledge-edit method backend."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
multi_edit_capable: bool
|
||||
live_wired: bool
|
||||
|
||||
|
||||
_EDIT_METHOD_METADATA: Mapping[str, EditMethodSpec] = MappingProxyType({
|
||||
"rome": EditMethodSpec(
|
||||
name="rome",
|
||||
description=(
|
||||
"Rank-One Model Editing — closed-form rank-1 update at a "
|
||||
"single MLP layer. Best for one-shot factual patches."
|
||||
),
|
||||
multi_edit_capable=False,
|
||||
live_wired=False,
|
||||
),
|
||||
"memit": EditMethodSpec(
|
||||
name="memit",
|
||||
description=(
|
||||
"Mass-Editing Memory in a Transformer — distributes the "
|
||||
"update across multiple layers for higher capacity."
|
||||
),
|
||||
multi_edit_capable=True,
|
||||
live_wired=False,
|
||||
),
|
||||
"alphaedit": EditMethodSpec(
|
||||
name="alphaedit",
|
||||
description=(
|
||||
"Null-space-projected ROME variant — survives sequential "
|
||||
"edits better than vanilla ROME / MEMIT."
|
||||
),
|
||||
multi_edit_capable=True,
|
||||
live_wired=False,
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
def validate_edit_method(value: object) -> str:
|
||||
"""Normalise + validate an edit-method name (case-insensitive)."""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError("edit_method must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"edit_method must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("edit_method must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("edit_method must not contain null bytes")
|
||||
if len(value) > _MAX_METHOD_LEN:
|
||||
raise ValueError(
|
||||
f"edit_method must be <= {_MAX_METHOD_LEN} chars"
|
||||
)
|
||||
canonical = value.lower()
|
||||
if canonical not in SUPPORTED_EDIT_METHODS:
|
||||
supported = ", ".join(sorted(SUPPORTED_EDIT_METHODS))
|
||||
raise ValueError(
|
||||
f"unknown edit method {value!r}; supported: {supported}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def _validate_text_field(value: object, name: str, max_len: int) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{name} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"{name} must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError(f"{name} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{name} must not contain null bytes")
|
||||
if len(value) > max_len:
|
||||
raise ValueError(f"{name} must be <= {max_len} chars")
|
||||
return value
|
||||
|
||||
|
||||
def parse_edit_subject_target(*, subject: str, target: str) -> tuple[str, str]:
|
||||
"""Validate the (subject, target) pair for an edit.
|
||||
|
||||
The subject is the prefix sentence (e.g. "Paris is the capital of
|
||||
France"); the target is the new completion (e.g. "Lyon"). Both
|
||||
fields are length-capped + null-byte-rejected. Returns the
|
||||
validated pair unchanged.
|
||||
"""
|
||||
s = _validate_text_field(subject, "subject", _MAX_SUBJECT_LEN)
|
||||
t = _validate_text_field(target, "target", _MAX_TARGET_LEN)
|
||||
return s, t
|
||||
|
||||
|
||||
def _validate_layer(value: Optional[object]) -> Optional[int]:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
raise TypeError("layer must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise TypeError(
|
||||
f"layer must be int, got {type(value).__name__}"
|
||||
)
|
||||
if value < 0:
|
||||
raise ValueError(f"layer must be >= 0, got {value}")
|
||||
if value > _MAX_LAYER_IDX:
|
||||
raise ValueError(
|
||||
f"layer must be <= {_MAX_LAYER_IDX}, got {value}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EditRequest:
|
||||
"""Operator-supplied request for a single knowledge edit."""
|
||||
|
||||
base: str
|
||||
method: str
|
||||
subject: str
|
||||
target: str
|
||||
layer: Optional[int]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EditPlan:
|
||||
"""Resolved + validated edit plan ready for ``apply_edit``.
|
||||
|
||||
Differs from :class:`EditRequest` in that ``method`` is canonical
|
||||
(lowercase) and ``layer`` is always concretely set (defaulted to
|
||||
the spec's recommended layer when ``None`` was supplied).
|
||||
"""
|
||||
|
||||
base: str
|
||||
method: str
|
||||
subject: str
|
||||
target: str
|
||||
layer: int
|
||||
spec: EditMethodSpec
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Re-validate so callers that bypass build_edit_plan can't
|
||||
# smuggle in an inconsistent plan.
|
||||
if self.method not in SUPPORTED_EDIT_METHODS:
|
||||
raise ValueError(
|
||||
f"method must be in {sorted(SUPPORTED_EDIT_METHODS)}, "
|
||||
f"got {self.method!r}"
|
||||
)
|
||||
if not isinstance(self.layer, int) or isinstance(self.layer, bool):
|
||||
raise TypeError("layer must be int (not bool)")
|
||||
if self.layer < 0 or self.layer > _MAX_LAYER_IDX:
|
||||
raise ValueError(
|
||||
f"layer must be in [0, {_MAX_LAYER_IDX}], got {self.layer}"
|
||||
)
|
||||
|
||||
|
||||
# Per-method default edit layer (heuristic — operator can override via
|
||||
# CLI). ROME papers target the middle-to-late MLP layer; AlphaEdit
|
||||
# follows the same convention. MEMIT updates a range so we treat the
|
||||
# "layer" arg as the centre of the spread.
|
||||
_DEFAULT_EDIT_LAYER: Mapping[str, int] = MappingProxyType({
|
||||
"rome": 5,
|
||||
"memit": 8,
|
||||
"alphaedit": 5,
|
||||
})
|
||||
|
||||
|
||||
def build_edit_plan(
|
||||
*,
|
||||
base: str,
|
||||
method: str,
|
||||
subject: str,
|
||||
target: str,
|
||||
layer: Optional[int] = None,
|
||||
) -> EditPlan:
|
||||
"""Resolve + validate an edit request into an :class:`EditPlan`.
|
||||
|
||||
Validates every operator-supplied field at construction time so
|
||||
misconfigured edits fail fast (mirrors v0.50.0 build_verdict /
|
||||
v0.55.0 ``design_evals_from_data`` policy).
|
||||
"""
|
||||
canonical_base = _validate_text_field(base, "base", _MAX_BASE_LEN)
|
||||
canonical_method = validate_edit_method(method)
|
||||
canonical_subject, canonical_target = parse_edit_subject_target(
|
||||
subject=subject, target=target,
|
||||
)
|
||||
canonical_layer = _validate_layer(layer)
|
||||
if canonical_layer is None:
|
||||
canonical_layer = _DEFAULT_EDIT_LAYER[canonical_method]
|
||||
spec = _EDIT_METHOD_METADATA[canonical_method]
|
||||
return EditPlan(
|
||||
base=canonical_base,
|
||||
method=canonical_method,
|
||||
subject=canonical_subject,
|
||||
target=canonical_target,
|
||||
layer=canonical_layer,
|
||||
spec=spec,
|
||||
)
|
||||
|
||||
|
||||
def apply_edit(plan: EditPlan) -> None:
|
||||
"""Apply a knowledge edit — deferred to v0.61.1.
|
||||
|
||||
Re-validates the method so callers passing a bare-class duck-typed
|
||||
plan (no ``EditPlan``) still hit a meaningful error before the
|
||||
deferred-live raise. Mirrors v0.50.0 ``apply_variant_loss`` policy.
|
||||
"""
|
||||
method_attr = getattr(plan, "method", None)
|
||||
canonical = validate_edit_method(method_attr)
|
||||
raise NotImplementedError(
|
||||
f"apply_edit(method={canonical!r}) is deferred to v0.61.1. "
|
||||
"Schema accepts the request now so YAML / CLI invocations are "
|
||||
"stable, but ROME / MEMIT / AlphaEdit live kernels land in "
|
||||
"v0.61.1."
|
||||
)
|
||||
|
||||
|
||||
def get_edit_method_spec(name: str) -> EditMethodSpec:
|
||||
"""Return the frozen :class:`EditMethodSpec` for ``name`` or raise."""
|
||||
canonical = validate_edit_method(name)
|
||||
return _EDIT_METHOD_METADATA[canonical]
|
||||
|
|
@ -0,0 +1,207 @@
|
|||
"""v0.61.0 Part A — Unlearning method allowlist + compat gate.
|
||||
|
||||
Three method backends for the new ``task='unlearn'`` trainer:
|
||||
|
||||
* ``npo`` — Negative Preference Optimization (Zhang et al., 2024). Pushes
|
||||
the model away from the forget set via a DPO-shaped negative-only loss.
|
||||
* ``simnpo`` — SimNPO (length-normalised NPO). Removes the reference model
|
||||
and uses a SimPO-style normalisation to stabilise long-sequence
|
||||
unlearning.
|
||||
* ``rmu`` — Representation Misdirection Unlearning (Li et al., 2024).
|
||||
Adds a noise vector to the residual stream for forget-set inputs while
|
||||
preserving retain-set activations.
|
||||
|
||||
Schema-only release: validators here are reused by the SoupConfig
|
||||
cross-validator, while the live trainer wrapper ships in v0.61.1
|
||||
(mirrors v0.50.0 GRPO Plus / v0.52.0 Modality II / v0.53.0 Quant Menu II
|
||||
stub-then-live pattern).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping
|
||||
|
||||
SUPPORTED_UNLEARN_METHODS: frozenset[str] = frozenset({"npo", "simnpo", "rmu"})
|
||||
|
||||
_MAX_METHOD_LEN: int = 32
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UnlearnMethodSpec:
|
||||
"""Metadata for an unlearning method backend. Frozen so callers cannot mutate."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
needs_retain_set: bool
|
||||
needs_reference_model: bool
|
||||
live_wired: bool
|
||||
|
||||
|
||||
_UNLEARN_METHOD_METADATA: Mapping[str, UnlearnMethodSpec] = MappingProxyType({
|
||||
"npo": UnlearnMethodSpec(
|
||||
name="npo",
|
||||
description=(
|
||||
"Negative Preference Optimization — DPO-shaped loss that "
|
||||
"pushes the model away from the forget set while a retain "
|
||||
"set keeps general capability stable."
|
||||
),
|
||||
needs_retain_set=True,
|
||||
needs_reference_model=True,
|
||||
live_wired=False,
|
||||
),
|
||||
"simnpo": UnlearnMethodSpec(
|
||||
name="simnpo",
|
||||
description=(
|
||||
"SimNPO — length-normalised NPO without a reference model. "
|
||||
"Faster + more stable on long sequences (Liu et al., 2024)."
|
||||
),
|
||||
needs_retain_set=True,
|
||||
needs_reference_model=False,
|
||||
live_wired=False,
|
||||
),
|
||||
"rmu": UnlearnMethodSpec(
|
||||
name="rmu",
|
||||
description=(
|
||||
"Representation Misdirection Unlearning — adds a noise "
|
||||
"vector to the residual stream for forget inputs while "
|
||||
"preserving retain activations (Li et al., 2024)."
|
||||
),
|
||||
needs_retain_set=True,
|
||||
needs_reference_model=False,
|
||||
live_wired=False,
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
def validate_unlearn_method(value: object) -> str:
|
||||
"""Normalise + validate an unlearn-method name.
|
||||
|
||||
Returns the canonical (lowercase) form. Mirrors v0.41.0
|
||||
``validate_optimizer_name`` / v0.51.0 ``validate_hub_name`` policy:
|
||||
bool-rejected, null-byte-rejected, oversize-rejected,
|
||||
case-insensitive normalisation, unknown rejected with friendly
|
||||
actionable message.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"unlearn_method must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"unlearn_method must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("unlearn_method must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("unlearn_method must not contain null bytes")
|
||||
if len(value) > _MAX_METHOD_LEN:
|
||||
raise ValueError(
|
||||
f"unlearn_method must be <= {_MAX_METHOD_LEN} chars"
|
||||
)
|
||||
canonical = value.lower()
|
||||
if canonical not in SUPPORTED_UNLEARN_METHODS:
|
||||
supported = ", ".join(sorted(SUPPORTED_UNLEARN_METHODS))
|
||||
raise ValueError(
|
||||
f"unknown unlearn method {value!r}; supported: {supported}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def get_unlearn_method_spec(name: str) -> UnlearnMethodSpec:
|
||||
"""Return the frozen :class:`UnlearnMethodSpec` for ``name`` or raise."""
|
||||
canonical = validate_unlearn_method(name)
|
||||
return _UNLEARN_METHOD_METADATA[canonical]
|
||||
|
||||
|
||||
def validate_unlearn_alpha(value: object) -> float:
|
||||
"""Validate an ``unlearn_alpha`` retain-set weight (forget vs retain mixing).
|
||||
|
||||
Bool-rejected, NaN/Inf-rejected, bounded ``[0.0, 10.0]``. Default
|
||||
interpretation: 0 = pure forget loss, higher values increasingly
|
||||
favour the retain set. The upper bound is a sanity cap — operators
|
||||
above 10 should rebalance their dataset instead.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"unlearn_alpha must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(
|
||||
f"unlearn_alpha must be a number, got {type(value).__name__}"
|
||||
)
|
||||
fval = float(value)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError("unlearn_alpha must be finite (no NaN / Inf)")
|
||||
if fval < 0.0:
|
||||
raise ValueError(f"unlearn_alpha must be >= 0.0, got {fval}")
|
||||
if fval > 10.0:
|
||||
raise ValueError(f"unlearn_alpha must be <= 10.0, got {fval}")
|
||||
return fval
|
||||
|
||||
|
||||
def validate_unlearn_compat(*, task: str, backend: str) -> None:
|
||||
"""Schema-time gate for ``task='unlearn'``.
|
||||
|
||||
Rejects:
|
||||
- non-string / bool args (defence-in-depth, mirrors v0.52.0
|
||||
``validate_classifier_compat`` policy).
|
||||
- non-unlearn task.
|
||||
- ``backend == 'mlx'`` (no MLX unlearn path in v0.61.0).
|
||||
|
||||
Multi-method specifics (forget_set required, retain_set conditional)
|
||||
are enforced at the SoupConfig cross-validator level so the schema
|
||||
can surface a single composite error.
|
||||
"""
|
||||
for name, value in (("task", task), ("backend", backend)):
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{name} must not be bool, got {value!r}")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"{name} must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError(f"{name} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{name} must not contain null bytes")
|
||||
if task != "unlearn":
|
||||
raise ValueError(
|
||||
f"validate_unlearn_compat called with task={task!r}; "
|
||||
f"expected task='unlearn'"
|
||||
)
|
||||
if backend == "mlx":
|
||||
raise ValueError(
|
||||
"task='unlearn' is not supported on backend=mlx in v0.61.0 "
|
||||
"(deferred to v0.61.1)"
|
||||
)
|
||||
|
||||
|
||||
def apply_unlearn_loss(method: str) -> None:
|
||||
"""Compute the per-method unlearn loss — deferred to v0.61.1.
|
||||
|
||||
Validates the method name first so the deferred-live error
|
||||
distinguishes between "unknown method" and "method is on the
|
||||
allowlist but not yet wired". Mirrors v0.50.0 ``apply_variant_loss``
|
||||
stub-then-live policy.
|
||||
"""
|
||||
canonical = validate_unlearn_method(method)
|
||||
raise NotImplementedError(
|
||||
f"apply_unlearn_loss({canonical!r}) is deferred to v0.61.1. "
|
||||
"Schema accepts the method now so callers can write soup.yaml "
|
||||
"today, but the live NPO / SimNPO / RMU loss kernels land in "
|
||||
"v0.61.1."
|
||||
)
|
||||
|
||||
|
||||
def build_unlearn_trainer(config: object, **kwargs: object) -> object:
|
||||
"""Live unlearn trainer factory — deferred stub.
|
||||
|
||||
Lazy-imports the wrapper so the schema-only import path never pulls
|
||||
in the heavy ``transformers`` / ``peft`` surface. The wrapper itself
|
||||
is a stub that raises ``NotImplementedError`` on ``setup()``.
|
||||
"""
|
||||
from soup_cli.trainer.unlearn import UnlearnTrainerWrapper
|
||||
|
||||
return UnlearnTrainerWrapper(config, **kwargs) # type: ignore[arg-type]
|
||||
|
|
@ -0,0 +1,520 @@
|
|||
"""v0.61.0 Part B — Unlearning eval suite (TOFU / MUSE / WMDP).
|
||||
|
||||
Scores three orthogonal axes after running a ``task='unlearn'`` job:
|
||||
|
||||
* **Forget Quality** — does the model still produce the unlearned
|
||||
content? Computed from pre/post loss on the forget set (high
|
||||
post-loss = good forgetting).
|
||||
* **Model Utility** — does general capability survive? Computed from
|
||||
the retain-set accuracy delta.
|
||||
* **PrivLeak** — can a membership-inference adversary still distinguish
|
||||
forget-set rows from a held-out cohort? Scored from MIA AUC; AUC ≈ 0.5
|
||||
is best (no leak).
|
||||
|
||||
Verdicts follow the project's OK / MINOR / MAJOR taxonomy (same
|
||||
thresholds as v0.26.0 Part D Quant-Lobotomy + v0.56.0 diagnose).
|
||||
|
||||
Live-model evaluation hooks are deferred to v0.61.1 — this module ships
|
||||
pure-Python kernels + a frozen ``UnlearnReport`` + bundled TOFU fixture.
|
||||
Operators can supply pre-computed ``evidence`` JSON to drive the
|
||||
classifier today.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from types import MappingProxyType
|
||||
from typing import Any, Mapping, Optional, Tuple
|
||||
|
||||
BENCHMARKS: frozenset[str] = frozenset({"tofu", "muse", "wmdp"})
|
||||
VERDICTS: Tuple[str, ...] = ("OK", "MINOR", "MAJOR")
|
||||
|
||||
_OK_THRESHOLD: float = 0.85
|
||||
_MINOR_THRESHOLD: float = 0.60
|
||||
|
||||
_MAX_BENCHMARK_LEN: int = 64
|
||||
_MAX_RUN_ID_LEN: int = 128
|
||||
_MAX_EVIDENCE_BYTES: int = 16 * 1024 * 1024 # 16 MiB
|
||||
|
||||
# Forget Quality saturation point (post - pre loss above which we
|
||||
# award the full 1.0 score). 2.0 nats ≈ 7.4x perplexity blow-up —
|
||||
# strong enough that legitimate "still remembers" cases land in MAJOR.
|
||||
_FORGET_SATURATION: float = 2.0
|
||||
|
||||
_METRIC_NAMES: Tuple[str, ...] = ("forget_quality", "model_utility", "priv_leak")
|
||||
|
||||
|
||||
_BENCHMARK_METADATA: Mapping[str, Mapping[str, str]] = MappingProxyType({
|
||||
"tofu": MappingProxyType({
|
||||
"description": (
|
||||
"TOFU — synthetic author profiles for forget-set unlearning "
|
||||
"(Maini et al., 2024)."
|
||||
),
|
||||
"fixture": "tofu_demo.jsonl",
|
||||
}),
|
||||
"muse": MappingProxyType({
|
||||
"description": (
|
||||
"MUSE — real-world books / news corpora with paired retain "
|
||||
"sets (Shi et al., 2024)."
|
||||
),
|
||||
"fixture": "", # not bundled; loader expected in v0.61.1
|
||||
}),
|
||||
"wmdp": MappingProxyType({
|
||||
"description": (
|
||||
"WMDP — hazardous-knowledge unlearning across biology / "
|
||||
"cyber / chemistry (Li et al., 2024)."
|
||||
),
|
||||
"fixture": "",
|
||||
}),
|
||||
})
|
||||
|
||||
|
||||
def validate_benchmark_name(value: object) -> str:
|
||||
"""Normalise + validate a benchmark name (case-insensitive)."""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError("benchmark must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"benchmark must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("benchmark must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("benchmark must not contain null bytes")
|
||||
if len(value) > _MAX_BENCHMARK_LEN:
|
||||
raise ValueError(
|
||||
f"benchmark must be <= {_MAX_BENCHMARK_LEN} chars"
|
||||
)
|
||||
canonical = value.lower()
|
||||
if canonical not in BENCHMARKS:
|
||||
supported = ", ".join(sorted(BENCHMARKS))
|
||||
raise ValueError(
|
||||
f"unknown benchmark {value!r}; supported: {supported}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def _validate_run_id(value: object) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError("run_id must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"run_id must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("run_id must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("run_id must not contain null bytes")
|
||||
if len(value) > _MAX_RUN_ID_LEN:
|
||||
raise ValueError(f"run_id must be <= {_MAX_RUN_ID_LEN} chars")
|
||||
return value
|
||||
|
||||
|
||||
def classify_unlearn_score(score: float) -> str:
|
||||
"""Map ``score in [0, 1]`` to OK / MINOR / MAJOR verdict."""
|
||||
if isinstance(score, bool):
|
||||
raise TypeError("score must be float, not bool")
|
||||
if not isinstance(score, (int, float)):
|
||||
raise TypeError(
|
||||
f"score must be float, got {type(score).__name__}"
|
||||
)
|
||||
value = float(score)
|
||||
if not math.isfinite(value):
|
||||
raise ValueError("score must be finite (no NaN / Inf)")
|
||||
if not 0.0 <= value <= 1.0:
|
||||
raise ValueError(f"score must be in [0, 1], got {value}")
|
||||
if value >= _OK_THRESHOLD:
|
||||
return "OK"
|
||||
if value >= _MINOR_THRESHOLD:
|
||||
return "MINOR"
|
||||
return "MAJOR"
|
||||
|
||||
|
||||
def _check_finite_non_negative(value: object, name: str) -> float:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{name} must not be bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(
|
||||
f"{name} must be a number, got {type(value).__name__}"
|
||||
)
|
||||
fval = float(value)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError(f"{name} must be finite (no NaN / Inf)")
|
||||
if fval < 0.0:
|
||||
raise ValueError(f"{name} must be >= 0, got {fval}")
|
||||
return fval
|
||||
|
||||
|
||||
def _check_unit_interval(value: object, name: str) -> float:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{name} must not be bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(
|
||||
f"{name} must be a number, got {type(value).__name__}"
|
||||
)
|
||||
fval = float(value)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError(f"{name} must be finite (no NaN / Inf)")
|
||||
if not 0.0 <= fval <= 1.0:
|
||||
raise ValueError(f"{name} must be in [0, 1], got {fval}")
|
||||
return fval
|
||||
|
||||
|
||||
def compute_forget_quality(*, pre_loss: float, post_loss: float) -> float:
|
||||
"""Forget Quality ∈ ``[0, 1]`` from pre/post loss on the forget set.
|
||||
|
||||
Higher post-unlearn loss → more forgetting. Linearly ramps from 0
|
||||
(post == pre) to 1.0 at ``post >= pre + _FORGET_SATURATION``; values
|
||||
beyond saturate at 1.0. The arbitrary tail prevents the score from
|
||||
being driven by a single divergent prompt.
|
||||
"""
|
||||
pre = _check_finite_non_negative(pre_loss, "pre_loss")
|
||||
post = _check_finite_non_negative(post_loss, "post_loss")
|
||||
delta = post - pre
|
||||
if delta <= 0.0:
|
||||
return 0.0
|
||||
return min(1.0, delta / _FORGET_SATURATION)
|
||||
|
||||
|
||||
def compute_model_utility(*, pre_acc: float, post_acc: float) -> float:
|
||||
"""Model Utility ∈ ``[0, 1]`` from retain-set accuracy delta.
|
||||
|
||||
1.0 = no degradation (post >= pre), 0.0 = full collapse (post=0).
|
||||
Linear ratio so the score is interpretable as "% capability
|
||||
retained".
|
||||
"""
|
||||
pre = _check_unit_interval(pre_acc, "pre_acc")
|
||||
post = _check_unit_interval(post_acc, "post_acc")
|
||||
if pre <= 0.0:
|
||||
return 1.0 if post >= pre else 0.0
|
||||
if post >= pre:
|
||||
return 1.0
|
||||
return max(0.0, post / pre)
|
||||
|
||||
|
||||
def compute_priv_leak(*, mia_auc: float) -> float:
|
||||
"""Privacy score ∈ ``[0, 1]`` from membership-inference AUC.
|
||||
|
||||
AUC ≈ 0.5 → no leak → score 1.0. AUC ≥ 0.9 or ≤ 0.1 → full leak →
|
||||
score 0.0. Symmetric around 0.5 so an adversary who can invert
|
||||
(AUC < 0.5) is still flagged.
|
||||
"""
|
||||
auc = _check_unit_interval(mia_auc, "mia_auc")
|
||||
distance = abs(auc - 0.5) * 2.0 # 0 → no leak, 1 → max distinguishable
|
||||
score = 1.0 - distance
|
||||
return max(0.0, min(1.0, score))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UnlearnMetric:
|
||||
"""A single named metric with verdict + free-text evidence."""
|
||||
|
||||
name: str
|
||||
score: float
|
||||
verdict: str
|
||||
evidence: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.name, str) or not self.name:
|
||||
raise ValueError("name must be non-empty str")
|
||||
if self.name not in _METRIC_NAMES:
|
||||
raise ValueError(
|
||||
f"name must be one of {_METRIC_NAMES}, got {self.name!r}"
|
||||
)
|
||||
expected = classify_unlearn_score(self.score)
|
||||
if not isinstance(self.verdict, str) or self.verdict not in VERDICTS:
|
||||
raise ValueError(
|
||||
f"verdict must be one of {VERDICTS}, got {self.verdict!r}"
|
||||
)
|
||||
if self.verdict != expected:
|
||||
raise ValueError(
|
||||
f"verdict {self.verdict!r} disagrees with score "
|
||||
f"{self.score} (expected {expected!r})"
|
||||
)
|
||||
if not isinstance(self.evidence, str):
|
||||
raise TypeError("evidence must be str")
|
||||
|
||||
|
||||
def _overall_verdict(verdicts: Tuple[str, ...]) -> str:
|
||||
"""Worst-case across metric verdicts: MAJOR > MINOR > OK."""
|
||||
if "MAJOR" in verdicts:
|
||||
return "MAJOR"
|
||||
if "MINOR" in verdicts:
|
||||
return "MINOR"
|
||||
return "OK"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UnlearnReport:
|
||||
"""Frozen report card for a single ``soup eval unlearning`` run."""
|
||||
|
||||
run_id: str
|
||||
benchmark: str
|
||||
metrics: Tuple[UnlearnMetric, ...]
|
||||
overall: str
|
||||
soup_version: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_validate_run_id(self.run_id)
|
||||
if self.benchmark not in BENCHMARKS:
|
||||
raise ValueError(
|
||||
f"benchmark must be in {sorted(BENCHMARKS)}, got "
|
||||
f"{self.benchmark!r}"
|
||||
)
|
||||
if not isinstance(self.metrics, tuple):
|
||||
raise TypeError("metrics must be a tuple of UnlearnMetric")
|
||||
for m in self.metrics:
|
||||
if not isinstance(m, UnlearnMetric):
|
||||
raise TypeError(
|
||||
"metrics entries must be UnlearnMetric instances"
|
||||
)
|
||||
if self.overall not in VERDICTS:
|
||||
raise ValueError(
|
||||
f"overall must be one of {VERDICTS}, got {self.overall!r}"
|
||||
)
|
||||
# overall must match worst-case
|
||||
expected_overall = _overall_verdict(tuple(m.verdict for m in self.metrics))
|
||||
if self.overall != expected_overall:
|
||||
raise ValueError(
|
||||
f"overall {self.overall!r} disagrees with worst metric "
|
||||
f"verdict {expected_overall!r}"
|
||||
)
|
||||
if not isinstance(self.soup_version, str) or not self.soup_version:
|
||||
raise ValueError("soup_version must be non-empty str")
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"run_id": self.run_id,
|
||||
"benchmark": self.benchmark,
|
||||
"metrics": [
|
||||
{
|
||||
"name": m.name,
|
||||
"score": m.score,
|
||||
"verdict": m.verdict,
|
||||
"evidence": m.evidence,
|
||||
}
|
||||
for m in self.metrics
|
||||
],
|
||||
"overall": self.overall,
|
||||
"soup_version": self.soup_version,
|
||||
}
|
||||
|
||||
|
||||
def _neutral_metric(name: str) -> UnlearnMetric:
|
||||
"""Neutral OK metric used when evidence is missing for a probe.
|
||||
|
||||
Matches v0.56.0 diagnose ``neutral_score`` policy: missing evidence
|
||||
is reported as a 1.0 OK score with an explicit "no evidence
|
||||
supplied" rationale so the operator sees that the probe did not run.
|
||||
"""
|
||||
return UnlearnMetric(
|
||||
name=name,
|
||||
score=1.0,
|
||||
verdict="OK",
|
||||
evidence="no evidence supplied; score is neutral default",
|
||||
)
|
||||
|
||||
|
||||
def _build_forget_metric(evidence: Optional[Mapping[str, Any]]) -> UnlearnMetric:
|
||||
"""Build the forget_quality metric.
|
||||
|
||||
Distinguishes 'missing evidence' (neutral OK score) from
|
||||
'present-but-invalid evidence' (re-raise as ValueError so the CLI
|
||||
surfaces it). Review HIGH H3 fix mirrors v0.56.0 diagnose evidence
|
||||
semantics — silent OK on invalid inputs hid genuine probe failures.
|
||||
"""
|
||||
if not isinstance(evidence, Mapping):
|
||||
return _neutral_metric("forget_quality")
|
||||
if "pre_loss" not in evidence or "post_loss" not in evidence:
|
||||
return _neutral_metric("forget_quality")
|
||||
# Present but invalid -> raise loudly.
|
||||
score = compute_forget_quality(
|
||||
pre_loss=evidence["pre_loss"], # type: ignore[arg-type]
|
||||
post_loss=evidence["post_loss"], # type: ignore[arg-type]
|
||||
)
|
||||
pre = float(evidence["pre_loss"])
|
||||
post = float(evidence["post_loss"])
|
||||
return UnlearnMetric(
|
||||
name="forget_quality",
|
||||
score=score,
|
||||
verdict=classify_unlearn_score(score),
|
||||
evidence=f"pre_loss={pre:.4f}, post_loss={post:.4f}",
|
||||
)
|
||||
|
||||
|
||||
def _build_utility_metric(evidence: Optional[Mapping[str, Any]]) -> UnlearnMetric:
|
||||
"""Build the model_utility metric. Same missing-vs-invalid policy as forget."""
|
||||
if not isinstance(evidence, Mapping):
|
||||
return _neutral_metric("model_utility")
|
||||
if "pre_acc" not in evidence or "post_acc" not in evidence:
|
||||
return _neutral_metric("model_utility")
|
||||
score = compute_model_utility(
|
||||
pre_acc=evidence["pre_acc"], # type: ignore[arg-type]
|
||||
post_acc=evidence["post_acc"], # type: ignore[arg-type]
|
||||
)
|
||||
pre = float(evidence["pre_acc"])
|
||||
post = float(evidence["post_acc"])
|
||||
return UnlearnMetric(
|
||||
name="model_utility",
|
||||
score=score,
|
||||
verdict=classify_unlearn_score(score),
|
||||
evidence=f"pre_acc={pre:.4f}, post_acc={post:.4f}",
|
||||
)
|
||||
|
||||
|
||||
def _build_priv_leak_metric(evidence: Optional[Mapping[str, Any]]) -> UnlearnMetric:
|
||||
"""Build the priv_leak metric. Same missing-vs-invalid policy as forget."""
|
||||
if not isinstance(evidence, Mapping):
|
||||
return _neutral_metric("priv_leak")
|
||||
if "mia_auc" not in evidence:
|
||||
return _neutral_metric("priv_leak")
|
||||
score = compute_priv_leak(mia_auc=evidence["mia_auc"]) # type: ignore[arg-type]
|
||||
auc = float(evidence["mia_auc"])
|
||||
return UnlearnMetric(
|
||||
name="priv_leak",
|
||||
score=score,
|
||||
verdict=classify_unlearn_score(score),
|
||||
evidence=f"mia_auc={auc:.4f}",
|
||||
)
|
||||
|
||||
|
||||
def run_unlearn_eval(
|
||||
*,
|
||||
run_id: str,
|
||||
benchmark: str,
|
||||
evidence: Optional[Mapping[str, Mapping[str, Any]]] = None,
|
||||
soup_version: Optional[str] = None,
|
||||
) -> UnlearnReport:
|
||||
"""Build an :class:`UnlearnReport` from optional pre-computed evidence.
|
||||
|
||||
Missing evidence per-metric falls through to ``_neutral_metric``
|
||||
(OK score, explicit "no evidence supplied" rationale). This mirrors
|
||||
v0.56.0 ``soup diagnose`` policy.
|
||||
"""
|
||||
bench = validate_benchmark_name(benchmark)
|
||||
_validate_run_id(run_id)
|
||||
if evidence is None:
|
||||
evidence = {}
|
||||
if not isinstance(evidence, Mapping):
|
||||
raise TypeError("evidence must be a Mapping or None")
|
||||
|
||||
forget = _build_forget_metric(evidence.get("forget_quality"))
|
||||
utility = _build_utility_metric(evidence.get("model_utility"))
|
||||
priv = _build_priv_leak_metric(evidence.get("priv_leak"))
|
||||
metrics = (forget, utility, priv)
|
||||
overall = _overall_verdict(tuple(m.verdict for m in metrics))
|
||||
|
||||
if soup_version is None:
|
||||
from soup_cli import __version__
|
||||
|
||||
soup_version = __version__
|
||||
|
||||
return UnlearnReport(
|
||||
run_id=run_id,
|
||||
benchmark=bench,
|
||||
metrics=metrics,
|
||||
overall=overall,
|
||||
soup_version=soup_version,
|
||||
)
|
||||
|
||||
|
||||
def write_unlearn_report(report: UnlearnReport, path: str) -> str:
|
||||
"""Atomic write of the report JSON. Cwd-contained + symlink-rejected.
|
||||
|
||||
Mirrors v0.56.0 diagnose ``write_report`` / v0.59.0 ``atomic_write_text``
|
||||
policy.
|
||||
"""
|
||||
if not isinstance(report, UnlearnReport):
|
||||
raise TypeError("report must be UnlearnReport")
|
||||
from soup_cli.utils.paths import atomic_write_text
|
||||
|
||||
payload = json.dumps(report.to_dict(), sort_keys=True, indent=2)
|
||||
# NOTE: atomic_write_text signature is (text, output_path) — the
|
||||
# text body comes first.
|
||||
atomic_write_text(payload, path, field="output")
|
||||
return path
|
||||
|
||||
|
||||
def get_fixture_path(benchmark: str) -> Optional[Path]:
|
||||
"""Return the bundled fixture path for a benchmark, or ``None`` if
|
||||
not yet bundled.
|
||||
|
||||
TOFU ships in v0.61.0 (synthetic author profile mini-set under
|
||||
``soup_cli/data/_fixtures/unlearning/``). MUSE / WMDP loaders are
|
||||
deferred to v0.61.1.
|
||||
|
||||
Routes through ``importlib.resources`` (review MEDIUM M6 — matches
|
||||
v0.53.8 #93 `_bundle_source_path` policy; safe under zipapp /
|
||||
namespace-package installs).
|
||||
"""
|
||||
if not isinstance(benchmark, str) or not benchmark:
|
||||
return None
|
||||
canonical = benchmark.lower()
|
||||
if canonical not in BENCHMARKS:
|
||||
return None
|
||||
meta = _BENCHMARK_METADATA[canonical]
|
||||
fixture_name = meta["fixture"]
|
||||
if not fixture_name:
|
||||
return None
|
||||
# Defensive: bake-in fixture name should never contain separators.
|
||||
if "/" in fixture_name or "\\" in fixture_name:
|
||||
return None
|
||||
from importlib.resources import files
|
||||
|
||||
try:
|
||||
pkg_root = files("soup_cli")
|
||||
except (ModuleNotFoundError, TypeError):
|
||||
return None
|
||||
candidate = Path(
|
||||
os.path.realpath(
|
||||
os.path.join(str(pkg_root), "data", "_fixtures",
|
||||
"unlearning", fixture_name)
|
||||
)
|
||||
)
|
||||
if not candidate.is_file():
|
||||
return None
|
||||
# Symlink rejection (TOCTOU defence - matches project policy).
|
||||
try:
|
||||
st = os.lstat(candidate)
|
||||
except OSError:
|
||||
return None
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
return None
|
||||
return candidate
|
||||
|
||||
|
||||
def load_evidence_file(path: str) -> Mapping[str, Mapping[str, Any]]:
|
||||
"""Load operator-supplied evidence JSON.
|
||||
|
||||
Containment + size cap + symlink rejection. Mirrors v0.56.0
|
||||
``commands/diagnose.py`` evidence loader.
|
||||
"""
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
if not isinstance(path, str) or not path:
|
||||
raise ValueError("evidence path must be a non-empty string")
|
||||
if "\x00" in path:
|
||||
raise ValueError("evidence path must not contain null bytes")
|
||||
if not is_under_cwd(path):
|
||||
raise ValueError(f"evidence path must stay under cwd: {path!r}")
|
||||
real = os.path.realpath(path)
|
||||
try:
|
||||
st = os.lstat(real)
|
||||
except FileNotFoundError as exc:
|
||||
raise FileNotFoundError(f"evidence file not found: {path!r}") from exc
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError("evidence path must not be a symlink")
|
||||
if st.st_size > _MAX_EVIDENCE_BYTES:
|
||||
raise ValueError(
|
||||
f"evidence file exceeds {_MAX_EVIDENCE_BYTES} bytes"
|
||||
)
|
||||
with open(real, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("evidence file must contain a JSON object at the root")
|
||||
return data
|
||||
|
|
@ -0,0 +1,520 @@
|
|||
"""Tests for v0.61.0 Part A — `soup unlearn` trainer (NPO / SimNPO / RMU).
|
||||
|
||||
Schema-only release: tests cover the new ``task='unlearn'`` Literal entry,
|
||||
the ``unlearn_method`` field, the ``forget_set`` / ``retain_set`` data
|
||||
schema, and the ``UnlearnTrainerWrapper`` stub. Live wiring is deferred
|
||||
to v0.61.1 (mirrors v0.50.0 / v0.52.0 / v0.53.0 stub-then-live pattern).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
# ---------- Module surface ----------
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.unlearning import (
|
||||
SUPPORTED_UNLEARN_METHODS,
|
||||
UnlearnMethodSpec,
|
||||
apply_unlearn_loss,
|
||||
build_unlearn_trainer,
|
||||
get_unlearn_method_spec,
|
||||
validate_unlearn_method,
|
||||
)
|
||||
assert callable(validate_unlearn_method)
|
||||
assert callable(get_unlearn_method_spec)
|
||||
assert callable(apply_unlearn_loss)
|
||||
assert callable(build_unlearn_trainer)
|
||||
assert dataclasses.is_dataclass(UnlearnMethodSpec)
|
||||
assert isinstance(SUPPORTED_UNLEARN_METHODS, frozenset)
|
||||
|
||||
def test_supported_methods_exact(self):
|
||||
from soup_cli.utils.unlearning import SUPPORTED_UNLEARN_METHODS
|
||||
|
||||
assert SUPPORTED_UNLEARN_METHODS == frozenset({"npo", "simnpo", "rmu"})
|
||||
|
||||
def test_metadata_mapping_proxy(self):
|
||||
from types import MappingProxyType
|
||||
|
||||
from soup_cli.utils.unlearning import _UNLEARN_METHOD_METADATA # type: ignore
|
||||
|
||||
assert isinstance(_UNLEARN_METHOD_METADATA, MappingProxyType)
|
||||
|
||||
|
||||
# ---------- validate_unlearn_method ----------
|
||||
|
||||
|
||||
class TestValidateUnlearnMethod:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
for name in ("npo", "simnpo", "rmu"):
|
||||
assert validate_unlearn_method(name) == name
|
||||
|
||||
def test_case_insensitive(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
assert validate_unlearn_method("NPO") == "npo"
|
||||
assert validate_unlearn_method("SimNPO") == "simnpo"
|
||||
assert validate_unlearn_method("RMU") == "rmu"
|
||||
|
||||
def test_unknown_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
with pytest.raises(ValueError, match="unknown unlearn method"):
|
||||
validate_unlearn_method("dpo")
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_unlearn_method(True)
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_unlearn_method(123)
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_unlearn_method("")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
validate_unlearn_method("npo\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_unlearn_method("a" * 100)
|
||||
|
||||
|
||||
# ---------- get_unlearn_method_spec ----------
|
||||
|
||||
|
||||
class TestGetUnlearnMethodSpec:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.unlearning import get_unlearn_method_spec
|
||||
|
||||
spec = get_unlearn_method_spec("npo")
|
||||
assert spec.name == "npo"
|
||||
assert isinstance(spec.description, str) and spec.description
|
||||
assert spec.live_wired is False # deferred to v0.61.1
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.unlearning import get_unlearn_method_spec
|
||||
|
||||
spec = get_unlearn_method_spec("npo")
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
spec.name = "other" # type: ignore
|
||||
|
||||
def test_unknown_raises(self):
|
||||
from soup_cli.utils.unlearning import get_unlearn_method_spec
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
get_unlearn_method_spec("zzz")
|
||||
|
||||
def test_all_three_have_specs(self):
|
||||
from soup_cli.utils.unlearning import get_unlearn_method_spec
|
||||
|
||||
for name in ("npo", "simnpo", "rmu"):
|
||||
spec = get_unlearn_method_spec(name)
|
||||
assert spec.name == name
|
||||
|
||||
|
||||
# ---------- apply_unlearn_loss (stub) ----------
|
||||
|
||||
|
||||
class TestApplyUnlearnLoss:
|
||||
def test_unknown_rejected_before_notimplemented(self):
|
||||
from soup_cli.utils.unlearning import apply_unlearn_loss
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
apply_unlearn_loss("zzz")
|
||||
|
||||
def test_deferred_marker_present(self):
|
||||
from soup_cli.utils.unlearning import apply_unlearn_loss
|
||||
|
||||
with pytest.raises(NotImplementedError, match="v0.61.1"):
|
||||
apply_unlearn_loss("npo")
|
||||
|
||||
|
||||
# ---------- build_unlearn_trainer (stub) ----------
|
||||
|
||||
|
||||
class TestBuildUnlearnTrainer:
|
||||
def _make_cfg(self):
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
return SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={
|
||||
"train": "test.jsonl",
|
||||
"forget_set": "f.jsonl",
|
||||
"retain_set": "r.jsonl",
|
||||
},
|
||||
training={"unlearn_method": "npo"},
|
||||
)
|
||||
|
||||
def test_returns_wrapper_instance(self):
|
||||
from soup_cli.trainer.unlearn import UnlearnTrainerWrapper
|
||||
from soup_cli.utils.unlearning import build_unlearn_trainer
|
||||
|
||||
cfg = self._make_cfg()
|
||||
wrapper = build_unlearn_trainer(cfg)
|
||||
assert isinstance(wrapper, UnlearnTrainerWrapper)
|
||||
|
||||
def test_kwargs_signature_allows_known(self):
|
||||
from soup_cli.utils.unlearning import build_unlearn_trainer
|
||||
|
||||
cfg = self._make_cfg()
|
||||
# Forward-compat kwargs accepted at construction time. Setup
|
||||
# raises NotImplementedError per v0.61.1 stub contract.
|
||||
wrapper = build_unlearn_trainer(cfg, device="cpu", trust_remote_code=False)
|
||||
with pytest.raises(NotImplementedError, match="v0.61.1"):
|
||||
wrapper.setup()
|
||||
|
||||
def test_invalid_config_rejected(self):
|
||||
from soup_cli.utils.unlearning import build_unlearn_trainer
|
||||
|
||||
class _Cfg:
|
||||
pass
|
||||
|
||||
with pytest.raises(AttributeError, match="SoupConfig"):
|
||||
build_unlearn_trainer(_Cfg())
|
||||
|
||||
|
||||
# ---------- validate_unlearn_compat ----------
|
||||
|
||||
|
||||
class TestValidateUnlearnCompat:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
# Should not raise
|
||||
validate_unlearn_compat(task="unlearn", backend="transformers")
|
||||
|
||||
def test_mlx_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
with pytest.raises(ValueError, match="mlx"):
|
||||
validate_unlearn_compat(task="unlearn", backend="mlx")
|
||||
|
||||
def test_wrong_task_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_unlearn_compat(task="sft", backend="transformers")
|
||||
|
||||
def test_bool_task_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_unlearn_compat(task=True, backend="transformers") # type: ignore
|
||||
|
||||
def test_bool_backend_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_unlearn_compat(task="unlearn", backend=True) # type: ignore
|
||||
|
||||
def test_empty_task_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_unlearn_compat(task="", backend="transformers")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.unlearning import validate_unlearn_compat
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
validate_unlearn_compat(task="unlearn", backend="transformers\x00")
|
||||
|
||||
|
||||
# ---------- Schema integration: task='unlearn' ----------
|
||||
|
||||
|
||||
class TestSchemaUnlearnTask:
|
||||
def test_unlearn_task_accepted(self):
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
cfg = SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={
|
||||
"train": "test.jsonl",
|
||||
"format": "auto",
|
||||
"forget_set": "forget.jsonl",
|
||||
"retain_set": "retain.jsonl",
|
||||
},
|
||||
training={"unlearn_method": "npo"},
|
||||
)
|
||||
assert cfg.task == "unlearn"
|
||||
assert cfg.training.unlearn_method == "npo"
|
||||
|
||||
def test_unlearn_method_default_none_for_non_unlearn(self):
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
cfg = SoupConfig(
|
||||
base="test-model",
|
||||
task="sft",
|
||||
data={"train": "test.jsonl"},
|
||||
)
|
||||
assert cfg.training.unlearn_method is None
|
||||
|
||||
def test_unlearn_method_unknown_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={"train": "test.jsonl", "forget_set": "f.jsonl"},
|
||||
training={"unlearn_method": "zzz"},
|
||||
)
|
||||
|
||||
def test_unlearn_method_case_insensitive(self):
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
cfg = SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={
|
||||
"train": "test.jsonl",
|
||||
"forget_set": "f.jsonl",
|
||||
"retain_set": "r.jsonl",
|
||||
},
|
||||
training={"unlearn_method": "NPO"},
|
||||
)
|
||||
assert cfg.training.unlearn_method == "npo"
|
||||
|
||||
def test_mlx_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
with pytest.raises(ValidationError, match="mlx"):
|
||||
SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
backend="mlx",
|
||||
data={"train": "test.jsonl", "forget_set": "f.jsonl"},
|
||||
training={"unlearn_method": "npo"},
|
||||
)
|
||||
|
||||
def test_unlearn_method_outside_unlearn_task_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
with pytest.raises(ValidationError, match="unlearn"):
|
||||
SoupConfig(
|
||||
base="test-model",
|
||||
task="sft",
|
||||
data={"train": "test.jsonl"},
|
||||
training={"unlearn_method": "npo"},
|
||||
)
|
||||
|
||||
def test_forget_set_field(self):
|
||||
from soup_cli.config.schema import DataConfig
|
||||
|
||||
data = DataConfig(train="test.jsonl", forget_set="forget.jsonl")
|
||||
assert data.forget_set == "forget.jsonl"
|
||||
|
||||
def test_retain_set_field(self):
|
||||
from soup_cli.config.schema import DataConfig
|
||||
|
||||
data = DataConfig(train="test.jsonl", retain_set="retain.jsonl")
|
||||
assert data.retain_set == "retain.jsonl"
|
||||
|
||||
def test_forget_set_null_byte_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import DataConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
DataConfig(train="test.jsonl", forget_set="f\x00.jsonl")
|
||||
|
||||
def test_retain_set_oversize_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import DataConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
DataConfig(train="test.jsonl", retain_set="x" * 5000)
|
||||
|
||||
def test_unlearn_requires_forget_set(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
with pytest.raises(ValidationError, match="forget_set"):
|
||||
SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={"train": "test.jsonl"},
|
||||
training={"unlearn_method": "npo"},
|
||||
)
|
||||
|
||||
def test_unlearn_alpha_field(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(unlearn_method="npo", unlearn_alpha=0.5)
|
||||
assert tcfg.unlearn_alpha == 0.5
|
||||
|
||||
def test_unlearn_alpha_bounds(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(unlearn_method="npo", unlearn_alpha=-0.1)
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(unlearn_method="npo", unlearn_alpha=11.0)
|
||||
|
||||
def test_unlearn_alpha_bool_rejected(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
# Bool raises TypeError from the validator; Pydantic v2 does not
|
||||
# wrap TypeError in ValidationError (only ValueError /
|
||||
# AssertionError / PydanticCustomError).
|
||||
with pytest.raises(TypeError, match="bool"):
|
||||
TrainingConfig(unlearn_method="npo", unlearn_alpha=True)
|
||||
|
||||
def test_unlearn_alpha_without_method_rejected(self):
|
||||
"""Review L10 — `unlearn_alpha` without `unlearn_method` is a footgun."""
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
|
||||
with pytest.raises(ValidationError, match="unlearn_method"):
|
||||
SoupConfig(
|
||||
base="test-model",
|
||||
task="sft",
|
||||
data={"train": "test.jsonl"},
|
||||
training={"unlearn_alpha": 0.5},
|
||||
)
|
||||
|
||||
def test_unlearn_alpha_boundary_zero_accepted(self):
|
||||
"""Review L3 — exact lower boundary (0.0)."""
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(unlearn_method="npo", unlearn_alpha=0.0)
|
||||
assert tcfg.unlearn_alpha == 0.0
|
||||
|
||||
def test_unlearn_alpha_boundary_ten_accepted(self):
|
||||
"""Review L3 — exact upper boundary (10.0)."""
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(unlearn_method="npo", unlearn_alpha=10.0)
|
||||
assert tcfg.unlearn_alpha == 10.0
|
||||
|
||||
|
||||
# ---------- Source-grep regression guards (review L7) ----------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_cli_registers_edit_typer(self):
|
||||
"""cli.py must add `_edit_cmd.app` under name='edit'."""
|
||||
from pathlib import Path
|
||||
|
||||
cli_src = Path(__file__).resolve().parent.parent / "soup_cli" / "cli.py"
|
||||
text = cli_src.read_text(encoding="utf-8")
|
||||
assert "_edit_cmd" in text
|
||||
assert 'name="edit"' in text
|
||||
|
||||
def test_eval_registers_v0610(self):
|
||||
"""eval.py must call register(app, console) from _eval_v0610."""
|
||||
from pathlib import Path
|
||||
|
||||
eval_src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli" / "commands" / "eval.py"
|
||||
)
|
||||
text = eval_src.read_text(encoding="utf-8")
|
||||
assert "_register_v0610" in text
|
||||
|
||||
def test_pyproject_includes_unlearning_fixtures(self):
|
||||
"""pyproject.toml artifacts list must include unlearning fixtures."""
|
||||
from pathlib import Path
|
||||
|
||||
py = Path(__file__).resolve().parent.parent / "pyproject.toml"
|
||||
text = py.read_text(encoding="utf-8")
|
||||
assert "unlearning/*.jsonl" in text
|
||||
|
||||
|
||||
# ---------- UnlearnTrainerWrapper ----------
|
||||
|
||||
|
||||
class TestUnlearnTrainerWrapper:
|
||||
def test_import(self):
|
||||
from soup_cli.trainer.unlearn import UnlearnTrainerWrapper
|
||||
|
||||
assert UnlearnTrainerWrapper is not None
|
||||
|
||||
def test_train_before_setup_raises(self):
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
from soup_cli.trainer.unlearn import UnlearnTrainerWrapper
|
||||
|
||||
cfg = SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={
|
||||
"train": "test.jsonl",
|
||||
"forget_set": "f.jsonl",
|
||||
"retain_set": "r.jsonl",
|
||||
},
|
||||
training={"unlearn_method": "npo"},
|
||||
)
|
||||
wrapper = UnlearnTrainerWrapper(cfg)
|
||||
with pytest.raises(RuntimeError, match="setup"):
|
||||
wrapper.train()
|
||||
|
||||
def test_setup_deferred(self):
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
from soup_cli.trainer.unlearn import UnlearnTrainerWrapper
|
||||
|
||||
cfg = SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={
|
||||
"train": "test.jsonl",
|
||||
"forget_set": "f.jsonl",
|
||||
"retain_set": "r.jsonl",
|
||||
},
|
||||
training={"unlearn_method": "npo"},
|
||||
)
|
||||
wrapper = UnlearnTrainerWrapper(cfg)
|
||||
with pytest.raises(NotImplementedError, match="v0.61.1"):
|
||||
wrapper.setup()
|
||||
|
||||
def test_method_attribute(self):
|
||||
from soup_cli.config.schema import SoupConfig
|
||||
from soup_cli.trainer.unlearn import UnlearnTrainerWrapper
|
||||
|
||||
cfg = SoupConfig(
|
||||
base="test-model",
|
||||
task="unlearn",
|
||||
data={
|
||||
"train": "test.jsonl",
|
||||
"forget_set": "f.jsonl",
|
||||
"retain_set": "r.jsonl",
|
||||
},
|
||||
training={"unlearn_method": "simnpo"},
|
||||
)
|
||||
wrapper = UnlearnTrainerWrapper(cfg)
|
||||
assert wrapper.method == "simnpo"
|
||||
|
|
@ -0,0 +1,521 @@
|
|||
"""Tests for v0.61.0 Part B — Unlearning eval suite (TOFU / MUSE / WMDP).
|
||||
|
||||
Coverage:
|
||||
- ``UnlearnMetric`` / ``UnlearnReport`` frozen dataclasses.
|
||||
- ``classify_unlearn_score`` OK / MINOR / MAJOR thresholds.
|
||||
- ``compute_forget_quality`` / ``compute_model_utility`` / ``compute_priv_leak`` kernels.
|
||||
- ``run_unlearn_eval`` orchestrator.
|
||||
- ``BENCHMARKS`` allowlist + closed name validation.
|
||||
- ``soup eval unlearning`` CLI smoke.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
|
||||
# ---------- Module surface ----------
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.unlearning_eval import (
|
||||
BENCHMARKS,
|
||||
VERDICTS,
|
||||
UnlearnMetric,
|
||||
UnlearnReport,
|
||||
classify_unlearn_score,
|
||||
compute_forget_quality,
|
||||
compute_model_utility,
|
||||
compute_priv_leak,
|
||||
run_unlearn_eval,
|
||||
validate_benchmark_name,
|
||||
)
|
||||
assert callable(classify_unlearn_score)
|
||||
assert callable(compute_forget_quality)
|
||||
assert callable(compute_model_utility)
|
||||
assert callable(compute_priv_leak)
|
||||
assert callable(run_unlearn_eval)
|
||||
assert callable(validate_benchmark_name)
|
||||
assert dataclasses.is_dataclass(UnlearnMetric)
|
||||
assert dataclasses.is_dataclass(UnlearnReport)
|
||||
assert isinstance(BENCHMARKS, frozenset)
|
||||
assert isinstance(VERDICTS, tuple)
|
||||
|
||||
def test_benchmarks_exact(self):
|
||||
from soup_cli.utils.unlearning_eval import BENCHMARKS
|
||||
|
||||
assert BENCHMARKS == frozenset({"tofu", "muse", "wmdp"})
|
||||
|
||||
def test_verdicts_exact(self):
|
||||
from soup_cli.utils.unlearning_eval import VERDICTS
|
||||
|
||||
assert VERDICTS == ("OK", "MINOR", "MAJOR")
|
||||
|
||||
|
||||
# ---------- classify_unlearn_score ----------
|
||||
|
||||
|
||||
class TestClassifyUnlearnScore:
|
||||
def test_ok_boundary(self):
|
||||
from soup_cli.utils.unlearning_eval import classify_unlearn_score
|
||||
|
||||
assert classify_unlearn_score(0.85) == "OK"
|
||||
assert classify_unlearn_score(1.0) == "OK"
|
||||
|
||||
def test_minor_band(self):
|
||||
from soup_cli.utils.unlearning_eval import classify_unlearn_score
|
||||
|
||||
assert classify_unlearn_score(0.60) == "MINOR"
|
||||
assert classify_unlearn_score(0.84) == "MINOR"
|
||||
|
||||
def test_major_band(self):
|
||||
from soup_cli.utils.unlearning_eval import classify_unlearn_score
|
||||
|
||||
assert classify_unlearn_score(0.0) == "MAJOR"
|
||||
assert classify_unlearn_score(0.59) == "MAJOR"
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import classify_unlearn_score
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
classify_unlearn_score(True)
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import classify_unlearn_score
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
classify_unlearn_score(float("nan"))
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
classify_unlearn_score(float("inf"))
|
||||
|
||||
def test_out_of_range_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import classify_unlearn_score
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
classify_unlearn_score(-0.1)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
classify_unlearn_score(1.1)
|
||||
|
||||
|
||||
# ---------- Benchmark name validation ----------
|
||||
|
||||
|
||||
class TestValidateBenchmarkName:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.unlearning_eval import validate_benchmark_name
|
||||
|
||||
assert validate_benchmark_name("tofu") == "tofu"
|
||||
assert validate_benchmark_name("MUSE") == "muse"
|
||||
assert validate_benchmark_name("WMDP") == "wmdp"
|
||||
|
||||
def test_unknown_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import validate_benchmark_name
|
||||
|
||||
with pytest.raises(ValueError, match="unknown"):
|
||||
validate_benchmark_name("zzz")
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import validate_benchmark_name
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_benchmark_name(True)
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import validate_benchmark_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_benchmark_name("tofu\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import validate_benchmark_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_benchmark_name("a" * 100)
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import validate_benchmark_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_benchmark_name("")
|
||||
|
||||
|
||||
# ---------- Metric kernels ----------
|
||||
|
||||
|
||||
class TestComputeForgetQuality:
|
||||
def test_perfect_forget(self):
|
||||
# If post-unlearn loss on forget set is HIGH and pre was LOW,
|
||||
# forget quality = 1.0.
|
||||
from soup_cli.utils.unlearning_eval import compute_forget_quality
|
||||
|
||||
score = compute_forget_quality(pre_loss=0.5, post_loss=5.0)
|
||||
assert score == 1.0
|
||||
|
||||
def test_no_forget(self):
|
||||
# If post-loss == pre-loss, quality is 0.
|
||||
from soup_cli.utils.unlearning_eval import compute_forget_quality
|
||||
|
||||
score = compute_forget_quality(pre_loss=2.0, post_loss=2.0)
|
||||
assert score == 0.0
|
||||
|
||||
def test_partial_forget(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_forget_quality
|
||||
|
||||
score = compute_forget_quality(pre_loss=1.0, post_loss=2.0)
|
||||
assert 0.0 < score < 1.0
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_forget_quality
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
compute_forget_quality(pre_loss=True, post_loss=2.0)
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
compute_forget_quality(pre_loss=1.0, post_loss=True)
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_forget_quality
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
compute_forget_quality(pre_loss=float("nan"), post_loss=1.0)
|
||||
|
||||
def test_negative_loss_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_forget_quality
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
compute_forget_quality(pre_loss=-0.5, post_loss=1.0)
|
||||
|
||||
|
||||
class TestComputeModelUtility:
|
||||
def test_perfect_utility(self):
|
||||
# If retain accuracy is preserved (post == pre), utility = 1.0.
|
||||
from soup_cli.utils.unlearning_eval import compute_model_utility
|
||||
|
||||
score = compute_model_utility(pre_acc=0.8, post_acc=0.8)
|
||||
assert score == 1.0
|
||||
|
||||
def test_no_utility(self):
|
||||
# If retain accuracy drops to 0, utility = 0.
|
||||
from soup_cli.utils.unlearning_eval import compute_model_utility
|
||||
|
||||
score = compute_model_utility(pre_acc=0.8, post_acc=0.0)
|
||||
assert score == 0.0
|
||||
|
||||
def test_partial_drop(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_model_utility
|
||||
|
||||
score = compute_model_utility(pre_acc=0.8, post_acc=0.6)
|
||||
assert 0.0 < score < 1.0
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_model_utility
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
compute_model_utility(pre_acc=True, post_acc=0.5)
|
||||
|
||||
def test_out_of_range_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_model_utility
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
compute_model_utility(pre_acc=1.5, post_acc=0.5)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
compute_model_utility(pre_acc=0.5, post_acc=-0.1)
|
||||
|
||||
|
||||
class TestComputePrivLeak:
|
||||
def test_no_leak(self):
|
||||
# Membership-inference AUC ≈ 0.5 → no leak.
|
||||
from soup_cli.utils.unlearning_eval import compute_priv_leak
|
||||
|
||||
score = compute_priv_leak(mia_auc=0.5)
|
||||
assert score >= 0.95 # very high "privacy preserved"
|
||||
|
||||
def test_full_leak(self):
|
||||
# MIA AUC = 1.0 → adversary can perfectly distinguish forget vs holdout.
|
||||
from soup_cli.utils.unlearning_eval import compute_priv_leak
|
||||
|
||||
score = compute_priv_leak(mia_auc=1.0)
|
||||
assert score == 0.0
|
||||
|
||||
def test_below_random(self):
|
||||
# AUC < 0.5 is still leak (adversary can invert).
|
||||
from soup_cli.utils.unlearning_eval import compute_priv_leak
|
||||
|
||||
score = compute_priv_leak(mia_auc=0.0)
|
||||
assert score == 0.0
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_priv_leak
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
compute_priv_leak(mia_auc=True)
|
||||
|
||||
def test_out_of_range_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import compute_priv_leak
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
compute_priv_leak(mia_auc=1.5)
|
||||
|
||||
def test_boundary_zero_accepted(self):
|
||||
"""Review L3 — exact lower boundary."""
|
||||
from soup_cli.utils.unlearning_eval import compute_priv_leak
|
||||
|
||||
# AUC=0.0 is in [0, 1] but reads as max distinguishable inverse.
|
||||
assert compute_priv_leak(mia_auc=0.0) == 0.0
|
||||
|
||||
def test_boundary_one_accepted(self):
|
||||
"""Review L3 — exact upper boundary."""
|
||||
from soup_cli.utils.unlearning_eval import compute_priv_leak
|
||||
|
||||
assert compute_priv_leak(mia_auc=1.0) == 0.0
|
||||
|
||||
|
||||
# ---------- UnlearnMetric / UnlearnReport ----------
|
||||
|
||||
|
||||
class TestUnlearnMetric:
|
||||
def test_construct(self):
|
||||
from soup_cli.utils.unlearning_eval import UnlearnMetric
|
||||
|
||||
m = UnlearnMetric(
|
||||
name="forget_quality",
|
||||
score=0.9,
|
||||
verdict="OK",
|
||||
evidence="pre=0.5, post=5.0",
|
||||
)
|
||||
assert m.name == "forget_quality"
|
||||
assert m.score == 0.9
|
||||
assert m.verdict == "OK"
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.unlearning_eval import UnlearnMetric
|
||||
|
||||
m = UnlearnMetric(
|
||||
name="forget_quality",
|
||||
score=0.9,
|
||||
verdict="OK",
|
||||
evidence="",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
m.score = 0.5 # type: ignore
|
||||
|
||||
def test_verdict_must_match_score(self):
|
||||
from soup_cli.utils.unlearning_eval import UnlearnMetric
|
||||
|
||||
with pytest.raises(ValueError, match="disagrees"):
|
||||
UnlearnMetric(
|
||||
name="forget_quality",
|
||||
score=0.3,
|
||||
verdict="OK",
|
||||
evidence="",
|
||||
)
|
||||
|
||||
|
||||
class TestUnlearnReport:
|
||||
def test_construct(self):
|
||||
from soup_cli.utils.unlearning_eval import UnlearnMetric, UnlearnReport
|
||||
|
||||
report = UnlearnReport(
|
||||
run_id="test-run",
|
||||
benchmark="tofu",
|
||||
metrics=(
|
||||
UnlearnMetric(name="forget_quality", score=0.9, verdict="OK", evidence=""),
|
||||
UnlearnMetric(name="model_utility", score=0.95, verdict="OK", evidence=""),
|
||||
UnlearnMetric(name="priv_leak", score=0.9, verdict="OK", evidence=""),
|
||||
),
|
||||
overall="OK",
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
assert report.run_id == "test-run"
|
||||
assert report.benchmark == "tofu"
|
||||
assert len(report.metrics) == 3
|
||||
assert report.overall == "OK"
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.unlearning_eval import UnlearnMetric, UnlearnReport
|
||||
|
||||
report = UnlearnReport(
|
||||
run_id="r",
|
||||
benchmark="tofu",
|
||||
metrics=(
|
||||
UnlearnMetric(name="forget_quality", score=0.9, verdict="OK", evidence=""),
|
||||
),
|
||||
overall="OK",
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
report.overall = "MAJOR" # type: ignore
|
||||
|
||||
def test_to_dict(self):
|
||||
from soup_cli.utils.unlearning_eval import UnlearnMetric, UnlearnReport
|
||||
|
||||
report = UnlearnReport(
|
||||
run_id="r",
|
||||
benchmark="tofu",
|
||||
metrics=(
|
||||
UnlearnMetric(name="forget_quality", score=0.9, verdict="OK", evidence="e"),
|
||||
),
|
||||
overall="OK",
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
d = report.to_dict()
|
||||
assert d["run_id"] == "r"
|
||||
assert d["benchmark"] == "tofu"
|
||||
assert d["overall"] == "OK"
|
||||
assert len(d["metrics"]) == 1
|
||||
# Round-trip via json
|
||||
s = json.dumps(d)
|
||||
assert json.loads(s) == d
|
||||
|
||||
|
||||
# ---------- run_unlearn_eval ----------
|
||||
|
||||
|
||||
class TestRunUnlearnEval:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.unlearning_eval import run_unlearn_eval
|
||||
|
||||
report = run_unlearn_eval(
|
||||
run_id="test-run",
|
||||
benchmark="tofu",
|
||||
evidence={
|
||||
"forget_quality": {"pre_loss": 0.5, "post_loss": 5.0},
|
||||
"model_utility": {"pre_acc": 0.8, "post_acc": 0.78},
|
||||
"priv_leak": {"mia_auc": 0.51},
|
||||
},
|
||||
)
|
||||
assert report.run_id == "test-run"
|
||||
assert report.benchmark == "tofu"
|
||||
assert report.overall in ("OK", "MINOR", "MAJOR")
|
||||
|
||||
def test_unknown_benchmark_rejected(self):
|
||||
from soup_cli.utils.unlearning_eval import run_unlearn_eval
|
||||
|
||||
with pytest.raises(ValueError, match="unknown"):
|
||||
run_unlearn_eval(run_id="r", benchmark="zzz", evidence={})
|
||||
|
||||
def test_missing_evidence_neutral(self):
|
||||
# Missing-evidence policy: neutral OK score (matches v0.56.0
|
||||
# diagnose-runner neutral_score policy).
|
||||
from soup_cli.utils.unlearning_eval import run_unlearn_eval
|
||||
|
||||
report = run_unlearn_eval(run_id="r", benchmark="tofu", evidence={})
|
||||
# All metrics neutral OK
|
||||
assert report.overall == "OK"
|
||||
|
||||
def test_overall_worst_case(self):
|
||||
from soup_cli.utils.unlearning_eval import run_unlearn_eval
|
||||
|
||||
report = run_unlearn_eval(
|
||||
run_id="r",
|
||||
benchmark="tofu",
|
||||
evidence={
|
||||
"forget_quality": {"pre_loss": 1.0, "post_loss": 1.0}, # MAJOR
|
||||
"model_utility": {"pre_acc": 0.8, "post_acc": 0.8}, # OK
|
||||
"priv_leak": {"mia_auc": 0.5}, # OK
|
||||
},
|
||||
)
|
||||
assert report.overall == "MAJOR"
|
||||
|
||||
def test_invalid_evidence_raises_loudly(self):
|
||||
"""Review HIGH H3 — present-but-invalid evidence must raise
|
||||
instead of silently scoring OK."""
|
||||
from soup_cli.utils.unlearning_eval import run_unlearn_eval
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
run_unlearn_eval(
|
||||
run_id="r",
|
||||
benchmark="tofu",
|
||||
evidence={
|
||||
"forget_quality": {"pre_loss": -1.0, "post_loss": 1.0},
|
||||
},
|
||||
)
|
||||
|
||||
def test_partial_evidence_neutral_on_missing(self):
|
||||
"""Review HIGH H3 — missing keys still produce neutral OK."""
|
||||
from soup_cli.utils.unlearning_eval import run_unlearn_eval
|
||||
|
||||
report = run_unlearn_eval(
|
||||
run_id="r",
|
||||
benchmark="tofu",
|
||||
evidence={
|
||||
"forget_quality": {"pre_loss": 0.5, "post_loss": 5.0},
|
||||
# model_utility + priv_leak missing -> neutral
|
||||
},
|
||||
)
|
||||
assert report.overall == "OK"
|
||||
|
||||
|
||||
# ---------- CLI ----------
|
||||
|
||||
|
||||
class TestCli:
|
||||
def test_help(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["eval", "unlearning", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_unknown_benchmark_rejected(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["eval", "unlearning", "test-run", "--benchmark", "zzz"])
|
||||
assert result.exit_code != 0
|
||||
assert "unknown" in result.output.lower() or "invalid" in result.output.lower()
|
||||
|
||||
def test_neutral_run(self, tmp_path):
|
||||
# Without evidence, every metric is neutral OK.
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=tmp_path) as fs:
|
||||
out = Path(fs) / "report.json"
|
||||
result = runner.invoke(app, [
|
||||
"eval", "unlearning", "test-run",
|
||||
"--benchmark", "tofu",
|
||||
"--output", str(out),
|
||||
])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert out.exists()
|
||||
data = json.loads(out.read_text())
|
||||
assert data["benchmark"] == "tofu"
|
||||
assert data["overall"] == "OK"
|
||||
|
||||
def test_outside_cwd_output_rejected(self, tmp_path):
|
||||
"""Review L8 — use tmp_path.parent so path is deterministically
|
||||
outside the isolated_filesystem on every platform."""
|
||||
runner = CliRunner()
|
||||
# The runner's isolated_filesystem cd's into a fresh subdir under
|
||||
# tmp_path; pointing --output at tmp_path itself is reliably
|
||||
# outside the new cwd regardless of OS or symlink layout.
|
||||
outside_target = str(tmp_path / "evil.json")
|
||||
with runner.isolated_filesystem(temp_dir=tmp_path):
|
||||
result = runner.invoke(app, [
|
||||
"eval", "unlearning", "test-run",
|
||||
"--benchmark", "tofu",
|
||||
"--output", outside_target,
|
||||
])
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
# ---------- Fixtures ----------
|
||||
|
||||
|
||||
class TestFixtures:
|
||||
def test_tofu_fixture_exists(self):
|
||||
from soup_cli.utils.unlearning_eval import get_fixture_path
|
||||
|
||||
# TOFU should be bundled.
|
||||
p = get_fixture_path("tofu")
|
||||
assert p is not None
|
||||
|
||||
def test_unknown_fixture_returns_none(self):
|
||||
from soup_cli.utils.unlearning_eval import get_fixture_path
|
||||
|
||||
assert get_fixture_path("zzz") is None
|
||||
|
|
@ -0,0 +1,335 @@
|
|||
"""Tests for v0.61.0 Part C — `soup edit set` (ROME / MEMIT / AlphaEdit).
|
||||
|
||||
Coverage:
|
||||
- ``SUPPORTED_EDIT_METHODS`` + ``validate_edit_method``.
|
||||
- ``EditRequest`` / ``EditPlan`` frozen dataclasses.
|
||||
- ``parse_edit_subject_target`` parser (free-text subject + target).
|
||||
- ``build_edit_plan`` schema-only orchestrator.
|
||||
- ``apply_edit`` deferred stub.
|
||||
- ``soup edit set`` CLI smoke.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.knowledge_edit import (
|
||||
SUPPORTED_EDIT_METHODS,
|
||||
EditPlan,
|
||||
EditRequest,
|
||||
apply_edit,
|
||||
build_edit_plan,
|
||||
parse_edit_subject_target,
|
||||
validate_edit_method,
|
||||
)
|
||||
assert callable(validate_edit_method)
|
||||
assert callable(parse_edit_subject_target)
|
||||
assert callable(build_edit_plan)
|
||||
assert callable(apply_edit)
|
||||
assert dataclasses.is_dataclass(EditRequest)
|
||||
assert dataclasses.is_dataclass(EditPlan)
|
||||
assert isinstance(SUPPORTED_EDIT_METHODS, frozenset)
|
||||
|
||||
def test_supported_methods_exact(self):
|
||||
from soup_cli.utils.knowledge_edit import SUPPORTED_EDIT_METHODS
|
||||
|
||||
assert SUPPORTED_EDIT_METHODS == frozenset({"rome", "memit", "alphaedit"})
|
||||
|
||||
|
||||
class TestValidateEditMethod:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
for name in ("rome", "memit", "alphaedit"):
|
||||
assert validate_edit_method(name) == name
|
||||
|
||||
def test_case_insensitive(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
assert validate_edit_method("ROME") == "rome"
|
||||
assert validate_edit_method("MEMIT") == "memit"
|
||||
|
||||
def test_unknown_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
with pytest.raises(ValueError, match="unknown"):
|
||||
validate_edit_method("ftedit")
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_edit_method(True)
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_edit_method(42)
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_edit_method("rome\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_edit_method("a" * 100)
|
||||
|
||||
|
||||
class TestParseEditSubjectTarget:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.knowledge_edit import parse_edit_subject_target
|
||||
|
||||
subject, target = parse_edit_subject_target(
|
||||
subject="Paris is the capital of France",
|
||||
target="Lyon",
|
||||
)
|
||||
assert subject == "Paris is the capital of France"
|
||||
assert target == "Lyon"
|
||||
|
||||
def test_empty_subject_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import parse_edit_subject_target
|
||||
|
||||
with pytest.raises(ValueError, match="subject"):
|
||||
parse_edit_subject_target(subject="", target="Lyon")
|
||||
|
||||
def test_empty_target_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import parse_edit_subject_target
|
||||
|
||||
with pytest.raises(ValueError, match="target"):
|
||||
parse_edit_subject_target(subject="Paris is the capital", target="")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import parse_edit_subject_target
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
parse_edit_subject_target(subject="Paris\x00", target="Lyon")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import parse_edit_subject_target
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
parse_edit_subject_target(
|
||||
subject="x" * 5000,
|
||||
target="Lyon",
|
||||
)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import parse_edit_subject_target
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
parse_edit_subject_target(subject=True, target="Lyon") # type: ignore
|
||||
|
||||
|
||||
class TestBuildEditPlan:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="meta-llama/Llama-3.1-8B-Instruct",
|
||||
method="rome",
|
||||
subject="Paris is the capital of France",
|
||||
target="Lyon",
|
||||
)
|
||||
assert plan.method == "rome"
|
||||
assert plan.base == "meta-llama/Llama-3.1-8B-Instruct"
|
||||
assert plan.subject == "Paris is the capital of France"
|
||||
assert plan.target == "Lyon"
|
||||
assert plan.layer is not None
|
||||
|
||||
def test_custom_layer(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="meta-llama/Llama-3.1-8B-Instruct",
|
||||
method="rome",
|
||||
subject="x",
|
||||
target="y",
|
||||
layer=17,
|
||||
)
|
||||
assert plan.layer == 17
|
||||
|
||||
def test_layer_bool_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
build_edit_plan(
|
||||
base="b", method="rome", subject="s", target="t",
|
||||
layer=True, # type: ignore
|
||||
)
|
||||
|
||||
def test_layer_negative_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_edit_plan(
|
||||
base="b", method="rome", subject="s", target="t",
|
||||
layer=-1,
|
||||
)
|
||||
|
||||
def test_layer_oversize_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_edit_plan(
|
||||
base="b", method="rome", subject="s", target="t",
|
||||
layer=10000,
|
||||
)
|
||||
|
||||
def test_unknown_method_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_edit_plan(
|
||||
base="b", method="zzz", subject="s", target="t",
|
||||
)
|
||||
|
||||
def test_empty_base_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_edit_plan(
|
||||
base="", method="rome", subject="s", target="t",
|
||||
)
|
||||
|
||||
def test_null_byte_base_rejected(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_edit_plan(
|
||||
base="b\x00", method="rome", subject="s", target="t",
|
||||
)
|
||||
|
||||
|
||||
class TestEditPlanFrozen:
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="b", method="rome", subject="s", target="t",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
plan.method = "memit" # type: ignore
|
||||
|
||||
|
||||
class TestApplyEdit:
|
||||
def test_deferred(self):
|
||||
from soup_cli.utils.knowledge_edit import apply_edit, build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="b", method="rome", subject="s", target="t",
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.61.1"):
|
||||
apply_edit(plan)
|
||||
|
||||
def test_unknown_method_in_plan_short_circuits(self):
|
||||
from soup_cli.utils.knowledge_edit import apply_edit
|
||||
|
||||
# apply_edit defends against direct (non-builder) plan construction
|
||||
# by re-validating the method before raising NotImplementedError.
|
||||
class _BadPlan:
|
||||
method = "zzz"
|
||||
base = "b"
|
||||
subject = "s"
|
||||
target = "t"
|
||||
layer = 5
|
||||
|
||||
with pytest.raises((TypeError, ValueError)):
|
||||
apply_edit(_BadPlan()) # type: ignore
|
||||
|
||||
|
||||
class TestCli:
|
||||
def test_edit_help(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["edit", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_edit_set_help(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["edit", "set", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_edit_set_plan_only(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"edit", "set",
|
||||
"--base", "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"--method", "rome",
|
||||
"--subject", "Paris is the capital of France",
|
||||
"--target", "Lyon",
|
||||
"--plan-only",
|
||||
])
|
||||
# Plan-only mode prints the plan and exits 0 without invoking
|
||||
# the deferred apply_edit kernel.
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "rome" in result.output.lower()
|
||||
assert "lyon" in result.output.lower()
|
||||
|
||||
def test_edit_set_unknown_method_rejected(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"edit", "set",
|
||||
"--base", "test",
|
||||
"--method", "zzz",
|
||||
"--subject", "s",
|
||||
"--target", "t",
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_edit_set_apply_deferred(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"edit", "set",
|
||||
"--base", "test",
|
||||
"--method", "rome",
|
||||
"--subject", "Paris is the capital",
|
||||
"--target", "Lyon",
|
||||
])
|
||||
# Without --plan-only, the CLI invokes apply_edit which raises
|
||||
# NotImplementedError. Exit code 3 (deferred, NOT validation
|
||||
# failure which is exit 2). Review HIGH H5 — distinguishes
|
||||
# "not yet shipped" from "validation rejection".
|
||||
assert result.exit_code == 3
|
||||
assert "v0.61.1" in result.output
|
||||
|
||||
def test_edit_set_registry_id_null_byte_rejected(self):
|
||||
"""Review HIGH H4 — null-byte in --registry-id."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"edit", "set",
|
||||
"--base", "test",
|
||||
"--method", "rome",
|
||||
"--subject", "Paris is the capital",
|
||||
"--target", "Lyon",
|
||||
"--registry-id", "evil\x00id",
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "null" in result.output.lower() or "registry-id" in result.output.lower()
|
||||
|
||||
def test_edit_set_registry_id_oversize_rejected(self):
|
||||
"""Review HIGH H4 — oversized --registry-id."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"edit", "set",
|
||||
"--base", "test",
|
||||
"--method", "rome",
|
||||
"--subject", "Paris is the capital",
|
||||
"--target", "Lyon",
|
||||
"--registry-id", "x" * 300,
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
|
|
@ -0,0 +1,327 @@
|
|||
"""Tests for v0.61.0 Part D — Sequential edit governor.
|
||||
|
||||
Coverage:
|
||||
- ``NormBlowupPolicy`` frozen dataclass + bounds validation.
|
||||
- ``classify_norm_blowup`` OK / WARN / BLOWUP taxonomy.
|
||||
- ``governor_recommend_method`` auto-switches method when blowup detected.
|
||||
- ``EditGovernor`` stateful tracker (per-base-model edit count + last verdict).
|
||||
- ``GovernedEditError`` for refusals.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.edit_governor import (
|
||||
DEFAULT_BLOWUP_POLICY,
|
||||
VERDICTS,
|
||||
EditGovernor,
|
||||
GovernedEditError,
|
||||
NormBlowupPolicy,
|
||||
classify_norm_blowup,
|
||||
governor_recommend_method,
|
||||
)
|
||||
assert callable(classify_norm_blowup)
|
||||
assert callable(governor_recommend_method)
|
||||
assert isinstance(VERDICTS, tuple)
|
||||
assert dataclasses.is_dataclass(NormBlowupPolicy)
|
||||
# GovernedEditError is an exception type, not a dataclass.
|
||||
assert issubclass(GovernedEditError, Exception)
|
||||
assert isinstance(DEFAULT_BLOWUP_POLICY, NormBlowupPolicy)
|
||||
assert callable(EditGovernor)
|
||||
|
||||
|
||||
class TestNormBlowupPolicy:
|
||||
def test_defaults(self):
|
||||
from soup_cli.utils.edit_governor import NormBlowupPolicy
|
||||
|
||||
p = NormBlowupPolicy()
|
||||
assert p.warn_threshold > 0
|
||||
assert p.blowup_threshold > p.warn_threshold
|
||||
assert p.max_sequential_edits > 0
|
||||
assert p.auto_switch_at >= 0
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.edit_governor import NormBlowupPolicy
|
||||
|
||||
p = NormBlowupPolicy()
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
p.warn_threshold = 999.0 # type: ignore
|
||||
|
||||
def test_invalid_bounds(self):
|
||||
from soup_cli.utils.edit_governor import NormBlowupPolicy
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
NormBlowupPolicy(warn_threshold=10.0, blowup_threshold=5.0)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
NormBlowupPolicy(warn_threshold=-1.0)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
NormBlowupPolicy(max_sequential_edits=0)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.edit_governor import NormBlowupPolicy
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
NormBlowupPolicy(warn_threshold=True) # type: ignore
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
NormBlowupPolicy(max_sequential_edits=True) # type: ignore
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.edit_governor import NormBlowupPolicy
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
NormBlowupPolicy(warn_threshold=float("nan"))
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
NormBlowupPolicy(blowup_threshold=float("inf"))
|
||||
|
||||
|
||||
class TestClassifyNormBlowup:
|
||||
def test_ok_band(self):
|
||||
from soup_cli.utils.edit_governor import classify_norm_blowup
|
||||
|
||||
assert classify_norm_blowup(0.01) == "OK"
|
||||
assert classify_norm_blowup(0.0) == "OK"
|
||||
|
||||
def test_warn_band(self):
|
||||
from soup_cli.utils.edit_governor import (
|
||||
DEFAULT_BLOWUP_POLICY,
|
||||
classify_norm_blowup,
|
||||
)
|
||||
|
||||
warn = DEFAULT_BLOWUP_POLICY.warn_threshold
|
||||
assert classify_norm_blowup(warn) == "WARN"
|
||||
# Above warn but below blowup.
|
||||
midpoint = (DEFAULT_BLOWUP_POLICY.warn_threshold +
|
||||
DEFAULT_BLOWUP_POLICY.blowup_threshold) / 2
|
||||
assert classify_norm_blowup(midpoint) == "WARN"
|
||||
|
||||
def test_blowup_band(self):
|
||||
from soup_cli.utils.edit_governor import (
|
||||
DEFAULT_BLOWUP_POLICY,
|
||||
classify_norm_blowup,
|
||||
)
|
||||
|
||||
assert classify_norm_blowup(DEFAULT_BLOWUP_POLICY.blowup_threshold) == "BLOWUP"
|
||||
assert classify_norm_blowup(100.0) == "BLOWUP"
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.edit_governor import classify_norm_blowup
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
classify_norm_blowup(True)
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.edit_governor import classify_norm_blowup
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
classify_norm_blowup(float("nan"))
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
classify_norm_blowup(float("inf"))
|
||||
|
||||
def test_negative_rejected(self):
|
||||
from soup_cli.utils.edit_governor import classify_norm_blowup
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
classify_norm_blowup(-1.0)
|
||||
|
||||
|
||||
class TestGovernorRecommendMethod:
|
||||
def test_no_switch_below_threshold(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
result = governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=1,
|
||||
norm_delta=0.0,
|
||||
)
|
||||
assert result.method == "rome"
|
||||
assert result.switched is False
|
||||
|
||||
def test_switch_at_auto_switch_count(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
result = governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=15, # above default auto_switch_at=10
|
||||
norm_delta=0.0,
|
||||
)
|
||||
# ROME should auto-switch to AlphaEdit at the count boundary.
|
||||
assert result.method == "alphaedit"
|
||||
assert result.switched is True
|
||||
|
||||
def test_blowup_forces_switch(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
result = governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=1,
|
||||
norm_delta=100.0, # blowup
|
||||
)
|
||||
assert result.method == "alphaedit"
|
||||
assert result.switched is True
|
||||
|
||||
def test_alphaedit_no_further_switch(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
# AlphaEdit is already the survival-mode method — no further switch.
|
||||
result = governor_recommend_method(
|
||||
current_method="alphaedit",
|
||||
edit_count=15,
|
||||
norm_delta=0.0,
|
||||
)
|
||||
assert result.method == "alphaedit"
|
||||
assert result.switched is False
|
||||
|
||||
def test_invalid_method_rejected(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
governor_recommend_method(
|
||||
current_method="zzz",
|
||||
edit_count=1,
|
||||
norm_delta=0.0,
|
||||
)
|
||||
|
||||
def test_bool_edit_count_rejected(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=True, # type: ignore
|
||||
norm_delta=0.0,
|
||||
)
|
||||
|
||||
def test_negative_edit_count_rejected(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=-1,
|
||||
norm_delta=0.0,
|
||||
)
|
||||
|
||||
def test_negative_norm_delta_rejected(self):
|
||||
from soup_cli.utils.edit_governor import governor_recommend_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=1,
|
||||
norm_delta=-1.0,
|
||||
)
|
||||
|
||||
def test_auto_switch_boundary_exact(self):
|
||||
"""Review L5 — exact `auto_switch_at=10` should switch."""
|
||||
from soup_cli.utils.edit_governor import (
|
||||
DEFAULT_BLOWUP_POLICY,
|
||||
governor_recommend_method,
|
||||
)
|
||||
|
||||
# At exactly auto_switch_at, ROME switches.
|
||||
result = governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=DEFAULT_BLOWUP_POLICY.auto_switch_at,
|
||||
norm_delta=0.0,
|
||||
)
|
||||
assert result.method == "alphaedit"
|
||||
assert result.switched is True
|
||||
|
||||
def test_auto_switch_below_boundary_no_switch(self):
|
||||
"""Review L5 — one below auto_switch_at does NOT switch."""
|
||||
from soup_cli.utils.edit_governor import (
|
||||
DEFAULT_BLOWUP_POLICY,
|
||||
governor_recommend_method,
|
||||
)
|
||||
|
||||
result = governor_recommend_method(
|
||||
current_method="rome",
|
||||
edit_count=DEFAULT_BLOWUP_POLICY.auto_switch_at - 1,
|
||||
norm_delta=0.0,
|
||||
)
|
||||
assert result.method == "rome"
|
||||
assert result.switched is False
|
||||
|
||||
|
||||
class TestEditGovernor:
|
||||
def test_construct(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor
|
||||
|
||||
g = EditGovernor(base_model="meta-llama/Llama-3.1-8B")
|
||||
assert g.edit_count == 0
|
||||
|
||||
def test_record_edit(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor
|
||||
|
||||
g = EditGovernor(base_model="meta-llama/Llama-3.1-8B")
|
||||
g.record_edit(method="rome", norm_delta=0.0)
|
||||
assert g.edit_count == 1
|
||||
|
||||
def test_refuses_above_max_edits(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor, GovernedEditError
|
||||
|
||||
g = EditGovernor(
|
||||
base_model="meta-llama/Llama-3.1-8B",
|
||||
max_sequential_edits=2,
|
||||
)
|
||||
g.record_edit(method="rome", norm_delta=0.0)
|
||||
g.record_edit(method="rome", norm_delta=0.0)
|
||||
with pytest.raises(GovernedEditError, match="max_sequential"):
|
||||
g.check_can_edit()
|
||||
|
||||
def test_blowup_blocks_further(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor, GovernedEditError
|
||||
|
||||
g = EditGovernor(base_model="meta-llama/Llama-3.1-8B")
|
||||
g.record_edit(method="rome", norm_delta=100.0) # blowup
|
||||
with pytest.raises(GovernedEditError, match="blowup"):
|
||||
g.check_can_edit()
|
||||
|
||||
def test_empty_base_rejected(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
EditGovernor(base_model="")
|
||||
|
||||
def test_null_byte_base_rejected(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
EditGovernor(base_model="b\x00")
|
||||
|
||||
def test_invalid_method_in_record_rejected(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor
|
||||
|
||||
g = EditGovernor(base_model="b")
|
||||
with pytest.raises(ValueError):
|
||||
g.record_edit(method="zzz", norm_delta=0.0)
|
||||
|
||||
def test_recommend_next(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor
|
||||
|
||||
g = EditGovernor(base_model="b")
|
||||
g.record_edit(method="rome", norm_delta=0.0)
|
||||
rec = g.recommend_next_method(current_method="rome")
|
||||
# After 1 ROME edit, still ROME.
|
||||
assert rec.method == "rome"
|
||||
|
||||
def test_snapshot(self):
|
||||
from soup_cli.utils.edit_governor import EditGovernor
|
||||
|
||||
g = EditGovernor(base_model="b")
|
||||
g.record_edit(method="rome", norm_delta=0.01)
|
||||
snap = g.snapshot()
|
||||
assert snap["edit_count"] == 1
|
||||
assert snap["last_method"] == "rome"
|
||||
assert snap["base_model"] == "b"
|
||||
|
|
@ -0,0 +1,403 @@
|
|||
"""Tests for v0.61.0 Part E — `soup edit diff` (knowledge-injection diff).
|
||||
|
||||
Coverage:
|
||||
- ``DiffReport`` / ``FactChange`` frozen dataclasses.
|
||||
- ``load_probes`` JSONL loader (cwd-contained, symlink-rejected).
|
||||
- ``build_diff_report`` schema-only orchestrator.
|
||||
- ``render_diff_table`` + ``write_diff_report``.
|
||||
- ``soup edit diff`` CLI smoke.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
|
||||
|
||||
class TestDataclasses:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.edit_diff import (
|
||||
DiffReport,
|
||||
FactChange,
|
||||
build_diff_report,
|
||||
load_probes,
|
||||
render_diff_table,
|
||||
write_diff_report,
|
||||
)
|
||||
assert callable(build_diff_report)
|
||||
assert callable(load_probes)
|
||||
assert callable(render_diff_table)
|
||||
assert callable(write_diff_report)
|
||||
assert dataclasses.is_dataclass(DiffReport)
|
||||
assert dataclasses.is_dataclass(FactChange)
|
||||
|
||||
def test_fact_change_frozen(self):
|
||||
from soup_cli.utils.edit_diff import FactChange
|
||||
|
||||
c = FactChange(
|
||||
prompt="p", before="b", after="a", changed=True,
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
c.prompt = "x" # type: ignore
|
||||
|
||||
def test_diff_report_frozen(self):
|
||||
from soup_cli.utils.edit_diff import DiffReport, FactChange
|
||||
|
||||
r = DiffReport(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
changes=(FactChange(prompt="p", before="b", after="a", changed=False),),
|
||||
total_probes=1,
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
r.total_probes = 999 # type: ignore
|
||||
|
||||
def test_diff_report_validates_changes_tuple(self):
|
||||
from soup_cli.utils.edit_diff import DiffReport
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
DiffReport(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
changes=[], # list, not tuple
|
||||
total_probes=0,
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
|
||||
def test_diff_report_to_dict(self):
|
||||
from soup_cli.utils.edit_diff import DiffReport, FactChange
|
||||
|
||||
r = DiffReport(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
changes=(
|
||||
FactChange(prompt="p1", before="x", after="y", changed=True),
|
||||
),
|
||||
total_probes=1,
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
d = r.to_dict()
|
||||
assert d["before_run_id"] == "b"
|
||||
assert d["after_run_id"] == "a"
|
||||
assert d["total_probes"] == 1
|
||||
assert d["changes"][0]["prompt"] == "p1"
|
||||
# JSON round-trip
|
||||
assert json.loads(json.dumps(d)) == d
|
||||
|
||||
def test_diff_report_total_probes_bool_rejected(self):
|
||||
from soup_cli.utils.edit_diff import DiffReport
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
DiffReport(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
changes=(),
|
||||
total_probes=True, # type: ignore
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
|
||||
|
||||
class TestLoadProbes:
|
||||
def _write_jsonl(self, path: Path, rows: list[dict]) -> None:
|
||||
path.write_text("\n".join(json.dumps(r) for r in rows), encoding="utf-8")
|
||||
|
||||
def test_happy_path(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import load_probes
|
||||
|
||||
p = tmp_path / "probes.jsonl"
|
||||
self._write_jsonl(p, [{"prompt": "Who is X?"}, {"prompt": "Who is Y?"}])
|
||||
|
||||
# Need to be in cwd
|
||||
old = os.getcwd()
|
||||
os.chdir(tmp_path)
|
||||
try:
|
||||
probes = load_probes("probes.jsonl")
|
||||
finally:
|
||||
os.chdir(old)
|
||||
assert probes == ("Who is X?", "Who is Y?")
|
||||
|
||||
def test_missing_file(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import load_probes
|
||||
|
||||
old = os.getcwd()
|
||||
os.chdir(tmp_path)
|
||||
try:
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_probes("missing.jsonl")
|
||||
finally:
|
||||
os.chdir(old)
|
||||
|
||||
def test_skips_malformed_rows(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import load_probes
|
||||
|
||||
p = tmp_path / "probes.jsonl"
|
||||
p.write_text(
|
||||
'{"prompt": "good"}\n'
|
||||
'this is not json\n'
|
||||
'{"prompt": ""}\n'
|
||||
'{"prompt": "another good"}\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
old = os.getcwd()
|
||||
os.chdir(tmp_path)
|
||||
try:
|
||||
probes = load_probes("probes.jsonl")
|
||||
finally:
|
||||
os.chdir(old)
|
||||
assert probes == ("good", "another good")
|
||||
|
||||
def test_outside_cwd_rejected(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import load_probes
|
||||
|
||||
out = tmp_path / "outside.jsonl"
|
||||
out.write_text('{"prompt": "p"}', encoding="utf-8")
|
||||
|
||||
# cwd is the original; tmp_path is unrelated
|
||||
cwd_dir = tmp_path / "cwd"
|
||||
cwd_dir.mkdir()
|
||||
old = os.getcwd()
|
||||
os.chdir(cwd_dir)
|
||||
try:
|
||||
with pytest.raises(ValueError, match="cwd"):
|
||||
load_probes(str(out))
|
||||
finally:
|
||||
os.chdir(old)
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.edit_diff import load_probes
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
load_probes("probe\x00.jsonl")
|
||||
|
||||
def test_empty_path_rejected(self):
|
||||
from soup_cli.utils.edit_diff import load_probes
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
load_probes("")
|
||||
|
||||
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX-only symlink test")
|
||||
def test_symlink_rejected(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import load_probes
|
||||
|
||||
real = tmp_path / "real.jsonl"
|
||||
real.write_text('{"prompt": "p"}', encoding="utf-8")
|
||||
link = tmp_path / "link.jsonl"
|
||||
os.symlink(real, link)
|
||||
|
||||
old = os.getcwd()
|
||||
os.chdir(tmp_path)
|
||||
try:
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
load_probes("link.jsonl")
|
||||
finally:
|
||||
os.chdir(old)
|
||||
|
||||
|
||||
class TestBuildDiffReport:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.edit_diff import build_diff_report
|
||||
|
||||
report = build_diff_report(
|
||||
before_run_id="before-run",
|
||||
after_run_id="after-run",
|
||||
)
|
||||
assert report.before_run_id == "before-run"
|
||||
assert report.after_run_id == "after-run"
|
||||
assert report.changes == ()
|
||||
assert report.total_probes == 0
|
||||
|
||||
def test_same_run_id_rejected(self):
|
||||
from soup_cli.utils.edit_diff import build_diff_report
|
||||
|
||||
with pytest.raises(ValueError, match="differ"):
|
||||
build_diff_report(
|
||||
before_run_id="x",
|
||||
after_run_id="x",
|
||||
)
|
||||
|
||||
def test_bool_run_id_rejected(self):
|
||||
from soup_cli.utils.edit_diff import build_diff_report
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
build_diff_report(
|
||||
before_run_id=True, # type: ignore
|
||||
after_run_id="x",
|
||||
)
|
||||
|
||||
def test_top_k_bool_rejected(self):
|
||||
from soup_cli.utils.edit_diff import build_diff_report
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
build_diff_report(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
top_k=True, # type: ignore
|
||||
)
|
||||
|
||||
def test_top_k_out_of_range(self):
|
||||
from soup_cli.utils.edit_diff import build_diff_report
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_diff_report(
|
||||
before_run_id="b", after_run_id="a", top_k=0,
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
build_diff_report(
|
||||
before_run_id="b", after_run_id="a", top_k=200,
|
||||
)
|
||||
|
||||
def test_with_probe_file(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import build_diff_report
|
||||
|
||||
p = tmp_path / "probes.jsonl"
|
||||
p.write_text(
|
||||
'{"prompt": "q1"}\n{"prompt": "q2"}\n{"prompt": "q3"}\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
old = os.getcwd()
|
||||
os.chdir(tmp_path)
|
||||
try:
|
||||
r = build_diff_report(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
probe_file="probes.jsonl",
|
||||
top_k=2,
|
||||
)
|
||||
finally:
|
||||
os.chdir(old)
|
||||
assert r.total_probes == 3
|
||||
assert len(r.changes) == 2 # capped by top_k
|
||||
|
||||
|
||||
class TestRenderDiffTable:
|
||||
def test_renders(self, capsys):
|
||||
from rich.console import Console
|
||||
|
||||
from soup_cli.utils.edit_diff import (
|
||||
DiffReport,
|
||||
FactChange,
|
||||
render_diff_table,
|
||||
)
|
||||
|
||||
r = DiffReport(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
changes=(
|
||||
FactChange(prompt="p1", before="x", after="y", changed=True),
|
||||
),
|
||||
total_probes=1,
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
console = Console(force_terminal=False, width=200)
|
||||
render_diff_table(r, console)
|
||||
# No raise = pass.
|
||||
|
||||
def test_non_report_rejected(self):
|
||||
from rich.console import Console
|
||||
|
||||
from soup_cli.utils.edit_diff import render_diff_table
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
render_diff_table("not a report", Console()) # type: ignore
|
||||
|
||||
|
||||
class TestWriteDiffReport:
|
||||
def test_happy_path(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import (
|
||||
DiffReport,
|
||||
write_diff_report,
|
||||
)
|
||||
|
||||
r = DiffReport(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
changes=(),
|
||||
total_probes=0,
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
old = os.getcwd()
|
||||
os.chdir(tmp_path)
|
||||
try:
|
||||
out = "report.json"
|
||||
write_diff_report(r, out)
|
||||
assert Path(out).exists()
|
||||
data = json.loads(Path(out).read_text())
|
||||
assert data["before_run_id"] == "b"
|
||||
finally:
|
||||
os.chdir(old)
|
||||
|
||||
def test_outside_cwd_rejected(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import (
|
||||
DiffReport,
|
||||
write_diff_report,
|
||||
)
|
||||
|
||||
r = DiffReport(
|
||||
before_run_id="b",
|
||||
after_run_id="a",
|
||||
changes=(),
|
||||
total_probes=0,
|
||||
soup_version="0.61.0",
|
||||
)
|
||||
out = tmp_path / "outside.json"
|
||||
cwd_dir = tmp_path / "cwd"
|
||||
cwd_dir.mkdir()
|
||||
old = os.getcwd()
|
||||
os.chdir(cwd_dir)
|
||||
try:
|
||||
with pytest.raises(ValueError):
|
||||
write_diff_report(r, str(out))
|
||||
finally:
|
||||
os.chdir(old)
|
||||
|
||||
def test_non_report_rejected(self, tmp_path):
|
||||
from soup_cli.utils.edit_diff import write_diff_report
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
write_diff_report("not a report", "out.json") # type: ignore
|
||||
|
||||
|
||||
class TestCli:
|
||||
def test_help(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["edit", "diff", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_basic_diff(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=tmp_path):
|
||||
result = runner.invoke(app, [
|
||||
"edit", "diff", "before-run", "after-run",
|
||||
])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_same_runs_rejected(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=tmp_path):
|
||||
result = runner.invoke(app, [
|
||||
"edit", "diff", "same", "same",
|
||||
])
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_writes_output(self, tmp_path):
|
||||
runner = CliRunner()
|
||||
with runner.isolated_filesystem(temp_dir=tmp_path) as fs:
|
||||
out = Path(fs) / "diff.json"
|
||||
result = runner.invoke(app, [
|
||||
"edit", "diff", "before-run", "after-run",
|
||||
"--output", str(out),
|
||||
])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert out.exists()
|
||||
data = json.loads(out.read_text())
|
||||
assert data["total_probes"] == 0
|
||||
Loading…
Reference in New Issue