mirror of https://github.com/razor-ai/soup.git
feat(v0.62.0): RAG & Activation Steering — RAFT + RA-DIT + soup steer + citation-faithful + GRACE codebook
5 Parts shipping wedge 14 of the roadmap (RAG-aware fine-tuning +
activation steering + lifelong edit codebook). Schema-only release;
live training loops + decode-hook intervention + codebook lookup all
land in v0.62.1 (mirrors v0.50.0 / v0.52.0 / v0.61.0 stub-then-live).
Part A — RAFT data format: new data.format='raft' schema +
_convert_raft validator (64 KiB per-field cap, 64-distractor cap,
null-byte rejection on every field) + raft-llama3-8b recipe.
Part B — RA-DIT two-stage: TrainingConfig.ra_dit_stage Literal
{retriever, generator} + ra_dit_retriever_model field + closed
allowlist + cross-validator enforcing stage to base-task pairing
(retriever to embedding, generator to sft) + 2 recipes.
Part C — soup steer (CAA / ITI / RepE): closed-allowlist control-vector
methods + validate_steering_method/name/strength + Typer subcommands
train/apply/list + soup serve --steer/--steer-strength flags +
steering_vector Registry artifact kind. apply_steering +
build_steering_vector deferred-live stubs raise NotImplementedError
with v0.62.1 marker after validating inputs.
Part D — Citation-faithful FT: score_citations precision/recall/F1
kernel + extract_citation_ids public API + citation_faithful /
citation_style / citation_recall_threshold schema. Cross-validator:
citation_faithful=true requires data.format='raft' AND task in
{sft, pretrain} (silent-no-op footgun rejection mirroring v0.52.0
distill / classifier task-gate policy).
Part E — GRACE codebook: GraceCodebookConfig + bounded size [1, 100k]
+ bounded dim [1, 16384]. Extends v0.61.0 SUPPORTED_EDIT_METHODS
allowlist with 'grace'; apply_edit routes grace plans to v0.62.1
marker while legacy rome/memit/alphaedit retain v0.61.1 marker
(regression-guarded via TestEditMarkerRegressionGuard).
Test count: 9571 -> 9786 (+215 net). 4 review-agent waves resolved
0 CRITICAL + 0 HIGH + 4 MEDIUM + 11 LOW (broken list_steers registry
context-manager + dict-key access; missing version bump;
citation_faithful task-gate; shared TOCTOU helper delegation; Rich
markup escape on --steer exception messages; --base length cap +
null-byte rejection; typing.Iterable -> collections.abc.Iterable
migration; except Exception -> except ImportError narrowing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
efbb05759f
commit
0d6f95181a
|
|
@ -111,7 +111,7 @@ soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (212 files, 9571 tests)
|
||||
tests/ - Test suite (218 files, 9786 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
113
README.md
113
README.md
|
|
@ -42,14 +42,14 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.61.0 — Unlearning & Knowledge Edit: GDPR right-to-be-forgotten + surgical fact patches.** Two of the most under-served axes in fine-tuning land as first-class trainers: unlearning (NPO / SimNPO / RMU — the legal-liability axis upstream TRL avoids) and knowledge editing (ROME / MEMIT / AlphaEdit — research-coded everywhere, productized nowhere). Hospital data teams handling a GDPR deletion request, labs patching factual errors without retraining, and security teams responding to a CSAM/PII leak all get a CLI workflow that composes with the existing Registry + eval-gate + diagnose surface. Schema-only release; live trainer + kernel wiring lands in v0.61.1.
|
||||
**v0.62.0 — RAG & Activation Steering: cite-your-sources fine-tuning, retrieval-aware recipes, and inference-time control vectors.** Closes axis 14 of the roadmap. Adds RAFT data format (Stanford 2024 — train a model to answer queries given a golden doc + distractors), RA-DIT two-stage retriever + generator recipes (Meta 2023), `soup steer` for CAA / ITI / RepE control vectors, citation-faithful FT with precision / recall / F1 scoring, and the GRACE codebook for thousands of sequential edits without norm-blowup. Schema-only release; live training loops + decode-hook intervention + codebook lookup all land in v0.62.1.
|
||||
|
||||
- **`task='unlearn'` + NPO / SimNPO / RMU.** New first-class task on `SoupConfig`. `training.unlearn_method: Literal["npo","simnpo","rmu"]` selects the backend; `data.forget_set` (rows to unlearn) + `data.retain_set` (capability-preserving rows) are validated at schema-load. Case-insensitive method normalisation; mlx backend rejected with distinct error; missing forget_set on `task='unlearn'` rejected with named field. `UnlearnTrainerWrapper` stub captures the method + accepts forward-compat kwargs so v0.61.1 wiring is purely additive.
|
||||
- **`soup eval unlearning <run-id> --benchmark tofu|muse|wmdp`** — three orthogonal axes scored on the project's OK / MINOR / MAJOR taxonomy: Forget Quality (pre/post forget-loss linear ramp), Model Utility (retain-accuracy ratio), PrivLeak (membership-inference AUC symmetric around 0.5). Bundled TOFU mini-fixture under `soup_cli/data/_fixtures/unlearning/`; MUSE + WMDP loaders ship in v0.61.1. Evidence-driven: operators supply pre-computed JSON today and get a frozen `UnlearnReport` with overall worst-case verdict + atomic JSON output.
|
||||
- **`soup edit set --base <m> --method rome|memit|alphaedit --subject "Paris is the capital of France" --target "Lyon"`** — surgical locate-and-edit. Closed method allowlist; per-method default MLP layer (ROME=5, MEMIT=8, AlphaEdit=5); operator override via `--layer`. `--plan-only` validates + prints the resolved `EditPlan` and exits 0 today; without it the CLI surfaces the deferred-live `NotImplementedError` with v0.61.1 marker at exit code 3 (distinct from validation rejection exit 2).
|
||||
- **Sequential edit governor + norm-blowup detection.** New `EditGovernor` tracks per-base-model edit count + last verdict; auto-switches ROME → AlphaEdit at the configurable edit-count threshold (default 10) AND on detected BLOWUP (Frobenius delta ≥ 5.0). `check_can_edit()` refuses further edits past the max cap or after a BLOWUP — `GovernedEditError` so callers distinguish governance refusals from other failures. AlphaEdit is the projection-based survivor (never switched away).
|
||||
- **`soup edit diff <before-run> <after-run> --probes p.jsonl`** — knowledge-injection diff visualizer. Frozen `DiffReport` + `FactChange` dataclasses; probe-file loader is cwd-contained + symlink-rejected + size-capped (16 MiB / 1000 rows). v0.61.0 ships shape + table renderer; v0.61.1 lands the live model-driven before/after generation.
|
||||
- **+193 new tests** (9446 → 9571 net after review-fix coverage). Review-fix coverage: 0 CRITICAL + 5 HIGH + 11 MEDIUM + 11 LOW (frozen-field declarations on `EditGovernor` so `asdict` / `replace` / slots stay safe; loud rejection of present-but-invalid evidence in `run_unlearn_eval`; `registry_id` validated BEFORE the deferred attach hook; exit-3 for "deferred" vs exit-2 for "validation rejection"; `validate_edit_method` lifted to module top in `edit_governor`; `importlib.resources` for fixture path; truncation warnings on probe-prompt clipping; boundary tests at exact `auto_switch_at` + `unlearn_alpha=0.0` + `unlearn_alpha=10.0` + `mia_auc=0.0` / `1.0`). Manual CPU smokes for `eval unlearning`, `edit set --plan-only`, `edit diff`, and 5 schema rejection paths all green.
|
||||
- **`data.format='raft'` + `raft-llama3-8b` recipe.** New RAFT row shape `{query, golden_doc, distractor_docs, answer}` with every field null-byte-rejected + 64 KiB per-field cap + max 64 distractors per row. Composes with the existing SFT trainer — operators wire a RAFT JSONL today, get the deferred-live span-mask trainer when v0.62.1 ships.
|
||||
- **RA-DIT (`ra_dit_stage: retriever|generator` + 2 new recipes).** Two-stage retriever-aware fine-tuning. `ra-dit-retriever` trains a sentence-transformer via the v0.16 embedding trainer (contrastive loss); `ra-dit-llama3-8b` is the RAFT-style generator stage. Cross-validator enforces each stage's required base task (retriever→embedding, generator→sft) so a misconfigured recipe fails at schema load.
|
||||
- **`soup steer train|apply|list` (CAA / ITI / RepE).** Closed-allowlist control-vector methods, kebab-case + dotted name regex, |strength| ≤ 10 bounded, `--plan-only` exits 0 with deferred-live marker, `apply` and live training exit code 3 (distinct from validation rejection exit 2). New `steering_vector` artifact kind on the Registry; `soup serve --steer <name> --steer-strength <s>` flag plumbed (decode-hook lands in v0.62.1).
|
||||
- **Citation-faithful FT (`citation_faithful: true` + `citation_style: bracket|inline|footnote` + `citation_recall_threshold`).** Composes with RAFT — citation_faithful=true is gated to `format='raft'` AND `task in {sft, pretrain}` (silent-no-op footgun rejection per the v0.52.0 task-gate policy). Ships the pure `score_citations` + `extract_citation_ids` kernel so the eval gate can compute precision/recall/F1 against bracketed `[doc-id]` references today (live span-mask trainer in v0.62.1).
|
||||
- **GRACE codebook (`grace_codebook: true` + bounded `size` + `dim` + `grace` added to `SUPPORTED_EDIT_METHODS`).** Discrete latent-space (key, value) store for thousands of sequential edits without norm-blowup. Extends v0.61.0 `soup edit set --method grace` end-to-end; `apply_edit` now routes `grace` plans to the v0.62.1 marker while keeping rome/memit/alphaedit on the existing v0.61.1 path (regression-guarded). Schema cross-validator refuses partial codebook configuration (flag without both size + dim, or size/dim without flag).
|
||||
- **+215 new tests** (9571 → 9786 net). Review-fix coverage: 0 CRITICAL + 0 HIGH + 4 MEDIUM + 11 LOW across python / code / security / tdd review waves (broken `list_steers` registry context-manager + dict-key access, missing version bump, citation_faithful task-gate, `_validate_pairs_path` delegated to shared `enforce_under_cwd_and_no_symlink`, Rich markup escape on `--steer` exception messages, `--base` length cap + null-byte rejection, `typing.Iterable` → `collections.abc.Iterable` migration, narrowed `except Exception` → `except ImportError` on optional-dep import). Manual CPU smokes for `steer train --plan-only`, `steer apply`, `steer list`, `recipes show raft-llama3-8b`, `recipes search ra-dit`, full SoupConfig YAML round-trip on every new flag, plus 5 cross-validator rejection paths.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -2060,6 +2060,105 @@ The endpoint validator follows the same SSRF rules as `HF_ENDPOINT`: only `http`
|
|||
|
||||
The hub adapter is schema-only in this release; the live downloader and uploader land in v0.51.1.
|
||||
|
||||
## RAFT — Retrieval-Augmented Fine-Tuning
|
||||
|
||||
When you need a model to *cite* the document it's reading instead of hallucinating, RAFT (Stanford 2024) is the canonical recipe. Each training row carries a query, a golden document, a list of distractor documents, and the answer — the model learns to attend to the relevant doc while ignoring the noise.
|
||||
|
||||
```yaml
|
||||
# soup.yaml
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
citation_faithful: true # enable citation precision/recall scoring
|
||||
citation_style: bracket # cite as [doc-1] inline
|
||||
citation_recall_threshold: 0.8 # gate final save on recall >= 80%
|
||||
```
|
||||
|
||||
```jsonl
|
||||
# RAFT JSONL row shape
|
||||
{"query": "When was Python released?", "golden_doc": "Python was released in 1991 by Guido van Rossum.", "distractor_docs": ["Ruby was released in 1995.", "Java was released in 1995."], "answer": "1991 [doc-1]"}
|
||||
```
|
||||
|
||||
```bash
|
||||
# Ready-made 8B Llama recipe
|
||||
soup recipes show raft-llama3-8b
|
||||
soup recipes use raft-llama3-8b
|
||||
```
|
||||
|
||||
Citation scoring is exposed as a pure kernel for the eval gate:
|
||||
|
||||
```python
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(
|
||||
predicted="The answer is 1991 [doc-1].",
|
||||
expected_ids=("doc-1",),
|
||||
)
|
||||
# CitationScore(precision=1.0, recall=1.0, f1=1.0, predicted_count=1, expected_count=1)
|
||||
```
|
||||
|
||||
Citation-faithful FT is gated to `task in {sft, pretrain}` + `data.format='raft'` — misconfigured runs fail at config load with a named-field message.
|
||||
|
||||
## RA-DIT — Retrieval-Augmented Dual Instruction Tuning
|
||||
|
||||
RA-DIT (Meta 2023) is the two-stage version of RAFT: first train a sentence-transformer retriever (contrastive), then fine-tune the generator on the RAFT-style rows. Two recipes ship paired:
|
||||
|
||||
```bash
|
||||
# Stage 1 — train the retriever (uses Soup's v0.16 embedding trainer)
|
||||
soup recipes use ra-dit-retriever
|
||||
soup train
|
||||
|
||||
# Stage 2 — train the generator on RAFT data, pointing at the retriever
|
||||
soup recipes use ra-dit-llama3-8b
|
||||
soup train
|
||||
```
|
||||
|
||||
The schema enforces stage-task pairing — `ra_dit_stage: retriever` requires `task: embedding`; `ra_dit_stage: generator` requires `task: sft`. A misconfigured recipe fails at config load with a named-field message.
|
||||
|
||||
## Activation Steering (`soup steer`)
|
||||
|
||||
Sometimes you don't want to retrain — you want to *push* the model along a learned direction at decode time. Soup ships three control-vector backends:
|
||||
|
||||
- **CAA** (Contrastive Activation Addition) — add a contrastive vector to the residual stream.
|
||||
- **ITI** (Inference-Time Intervention) — shift specific attention heads along a learned direction.
|
||||
- **RepE** (Representation Engineering) — PCA-based direction in the residual stream.
|
||||
|
||||
```bash
|
||||
# Train a steering vector from contrastive (positive, negative) prompt pairs
|
||||
soup steer train --base meta-llama/Llama-3.1-8B-Instruct \
|
||||
--method caa --name safety-v1 \
|
||||
--pairs ./data/pairs.jsonl
|
||||
|
||||
# Apply at decode time via soup serve
|
||||
soup serve --model ./adapter --steer safety-v1 --steer-strength 1.5
|
||||
|
||||
# List locally-stored steering vectors
|
||||
soup steer list
|
||||
```
|
||||
|
||||
Steering names are validated against a strict regex (`^[A-Za-z0-9][A-Za-z0-9._\-]{0,127}$` — no path separators, no shell metacharacters); strength is bounded `|s| <= 10.0`. The trained vectors land in the Soup Registry under the `steering_vector` artifact kind so lineage is preserved.
|
||||
|
||||
## GRACE Codebook — Lifelong Knowledge Edits
|
||||
|
||||
Vanilla ROME / MEMIT degrade after dozens of sequential edits — the model's norms blow up. GRACE (Hartvigsen et al., 2023) stores each edit in a discrete latent codebook so thousands of sequential patches survive:
|
||||
|
||||
```bash
|
||||
soup edit set --base ./model --method grace \
|
||||
--subject "The CEO of Acme is" --target "Jane Doe"
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Or via soup.yaml when training a model with GRACE-aware lookups
|
||||
training:
|
||||
grace_codebook: true
|
||||
grace_codebook_size: 1024 # codebook entries (max 100k)
|
||||
grace_codebook_dim: 768 # residual-stream width
|
||||
```
|
||||
|
||||
`grace` joins the existing `rome` / `memit` / `alphaedit` allowlist on `soup edit set`; the v0.61.0 sequential edit governor still gates the call when the per-base-model edit count or norm-blowup verdict trips.
|
||||
|
||||
## Model Registry & Lineage
|
||||
|
||||
Every fine-tune you ship should be reproducible. Soup's local registry (`~/.soup/registry.db`) tracks each entry by a content hash of its config + data + base model, plus lineage pointers to parent entries.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.61.0"
|
||||
version = "0.62.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.61.0"
|
||||
__version__ = "0.62.0"
|
||||
|
|
|
|||
|
|
@ -277,6 +277,18 @@ app.add_typer(
|
|||
),
|
||||
)
|
||||
|
||||
# v0.62.0 Part C — Activation steering: `soup steer train / apply / list`.
|
||||
from soup_cli.commands import steer as _steer_cmd # noqa: E402
|
||||
|
||||
app.add_typer(
|
||||
_steer_cmd.app,
|
||||
name="steer",
|
||||
help=(
|
||||
"Activation steering (CAA / ITI / RepE) - inference-time "
|
||||
"intervention without retraining (v0.62.0)."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _rewrite_advise_argv(argv: list) -> list:
|
||||
"""Inject `run` between `advise` and a non-subcommand first argument.
|
||||
|
|
|
|||
|
|
@ -199,6 +199,24 @@ def serve(
|
|||
"deepseek-r1 | qwen3 | phi4 | openthinker. v0.53.9 #98."
|
||||
),
|
||||
),
|
||||
steer: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--steer",
|
||||
help=(
|
||||
"Apply a stored activation-steering vector at decode time "
|
||||
"(CAA / ITI / RepE). Pass the name registered via "
|
||||
"`soup steer train`. Schema-only in v0.62.0; live decode hook "
|
||||
"ships in v0.62.1."
|
||||
),
|
||||
),
|
||||
steer_strength: float = typer.Option(
|
||||
1.0,
|
||||
"--steer-strength",
|
||||
help=(
|
||||
"Steering strength multiplier (|s| <= 10.0). Ignored when "
|
||||
"--steer is unset. v0.62.0 Part C."
|
||||
),
|
||||
),
|
||||
hub: str = typer.Option(
|
||||
"hf",
|
||||
"--hub",
|
||||
|
|
@ -209,6 +227,33 @@ def serve(
|
|||
),
|
||||
):
|
||||
"""Start a local inference server with OpenAI-compatible API."""
|
||||
# v0.62.0 Part C — validate `--steer` name + strength up front so a
|
||||
# typo surfaces before backend init. Live decode-hook wiring lands
|
||||
# in v0.62.1; this release just locks the flag surface.
|
||||
if steer is not None:
|
||||
from rich.markup import escape as _rich_escape
|
||||
|
||||
from soup_cli.utils.steering import (
|
||||
validate_steering_name,
|
||||
validate_steering_strength,
|
||||
)
|
||||
|
||||
try:
|
||||
validate_steering_name(steer)
|
||||
validate_steering_strength(steer_strength)
|
||||
except (TypeError, ValueError) as exc:
|
||||
# Escape the exception message — it embeds the operator-
|
||||
# supplied --steer value via {value!r}, which would otherwise
|
||||
# let a crafted name inject Rich markup (security review M1).
|
||||
console.print(
|
||||
f"[red]Invalid --steer:[/] {_rich_escape(str(exc))}"
|
||||
)
|
||||
raise typer.Exit(code=2) from exc
|
||||
console.print(
|
||||
f"[yellow]--steer={_rich_escape(steer)!r} accepted; live "
|
||||
"decode hook ships in v0.62.1.[/]"
|
||||
)
|
||||
|
||||
# v0.53.10 #152 — pre-fetch base from a non-HF hub before serve starts.
|
||||
if hub and hub != "hf":
|
||||
from soup_cli.utils.hubs import apply_hub_to_cli_model
|
||||
|
|
|
|||
|
|
@ -0,0 +1,262 @@
|
|||
"""v0.62.0 Part C — `soup steer` CLI command group.
|
||||
|
||||
Three subcommands:
|
||||
|
||||
* ``soup steer train`` — fit a control vector from contrastive pairs.
|
||||
* ``soup steer apply`` — apply a stored vector at decode time (also exposed
|
||||
via ``soup serve --steer <name>``).
|
||||
* ``soup steer list`` — list locally-stored steering vectors.
|
||||
|
||||
Schema + CLI surface ship in v0.62.0; the live forward-hook + per-method
|
||||
fitting kernels land in v0.62.1 (mirrors v0.50.0 / v0.52.0 / v0.61.0
|
||||
stub-then-live cadence).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
|
||||
console = Console()
|
||||
|
||||
app = typer.Typer(
|
||||
name="steer",
|
||||
help=(
|
||||
"Activation steering (CAA / ITI / RepE) - inference-time "
|
||||
"intervention without retraining (v0.62.0)."
|
||||
),
|
||||
no_args_is_help=True,
|
||||
rich_markup_mode="rich",
|
||||
)
|
||||
|
||||
|
||||
def _validate_pairs_path(pairs: str) -> str:
|
||||
"""Containment-check the pairs JSONL path; reject pre-placed symlinks.
|
||||
|
||||
Delegates to the shared :func:`enforce_under_cwd_and_no_symlink`
|
||||
helper (centralised in v0.53.1) so the TOCTOU policy stays
|
||||
single-source-of-truth (review M1 fix). Adds the 4096-char length
|
||||
cap separately because the shared helper does not enforce one.
|
||||
"""
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
if isinstance(pairs, str) and len(pairs) > 4096:
|
||||
raise typer.BadParameter("--pairs must be <= 4096 chars")
|
||||
try:
|
||||
return enforce_under_cwd_and_no_symlink(pairs, "--pairs")
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise typer.BadParameter(str(exc)) from exc
|
||||
|
||||
|
||||
@app.command(name="train")
|
||||
def train_steer(
|
||||
base: str = typer.Option(
|
||||
..., "--base", "-b",
|
||||
help="Base model HF id or local path.",
|
||||
),
|
||||
method: str = typer.Option(
|
||||
"caa", "--method", "-m",
|
||||
help="Steering method: caa / iti / repe.",
|
||||
),
|
||||
name: str = typer.Option(
|
||||
..., "--name", "-n",
|
||||
help="Identifier for the trained vector (e.g. 'safety-v1').",
|
||||
),
|
||||
pairs: str = typer.Option(
|
||||
..., "--pairs", "-p",
|
||||
help="Path to JSONL of contrastive (positive, negative) prompt pairs.",
|
||||
),
|
||||
layer: Optional[int] = typer.Option(
|
||||
None, "--layer", "-l",
|
||||
help="MLP layer index to extract the residual-stream vector from.",
|
||||
),
|
||||
plan_only: bool = typer.Option(
|
||||
False, "--plan-only",
|
||||
help=(
|
||||
"Validate inputs + print the resolved plan; skip the "
|
||||
"deferred-live training (v0.62.1)."
|
||||
),
|
||||
),
|
||||
registry_id: Optional[str] = typer.Option(
|
||||
None, "--registry-id",
|
||||
help="Optional Registry entry id to attach the trained vector to.",
|
||||
),
|
||||
) -> None:
|
||||
"""Train a steering vector from contrastive prompt pairs."""
|
||||
from soup_cli.utils.steering import (
|
||||
build_steering_vector,
|
||||
get_steering_method_spec,
|
||||
validate_steering_method,
|
||||
validate_steering_name,
|
||||
)
|
||||
|
||||
# Validate method + name + pairs path up front so a typo fails fast
|
||||
# with a clear message before we attempt the deferred-live call.
|
||||
try:
|
||||
canonical_method = validate_steering_method(method)
|
||||
canonical_name = validate_steering_name(name)
|
||||
pairs_path = _validate_pairs_path(pairs)
|
||||
except (TypeError, ValueError, typer.BadParameter) as exc:
|
||||
console.print(f"[red]Invalid steer-train input:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
# `--base` accepts arbitrary HF ids / local paths but Typer does not
|
||||
# cap length. Match the v0.40.5 `reward_model` policy (<=512 chars +
|
||||
# null-byte rejection) so a multi-MB --base cannot bloat the Rich
|
||||
# panel render (security review L1).
|
||||
if not isinstance(base, str) or not base:
|
||||
console.print("[red]Invalid --base:[/] must be non-empty string")
|
||||
raise typer.Exit(2)
|
||||
if "\x00" in base:
|
||||
console.print("[red]Invalid --base:[/] null bytes not allowed")
|
||||
raise typer.Exit(2)
|
||||
if len(base) > 512:
|
||||
console.print("[red]Invalid --base:[/] >512 chars")
|
||||
raise typer.Exit(2)
|
||||
|
||||
# Validate --registry-id early so the v0.62.1 attach path doesn't
|
||||
# see junk. Mirrors v0.61.0 Part C policy on `--registry-id`.
|
||||
if registry_id is not None:
|
||||
if not isinstance(registry_id, str) or not registry_id:
|
||||
console.print(
|
||||
"[red]Invalid --registry-id:[/] must be non-empty"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
if "\x00" in registry_id:
|
||||
console.print(
|
||||
"[red]Invalid --registry-id:[/] null bytes not allowed"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
if len(registry_id) > 256:
|
||||
console.print("[red]Invalid --registry-id:[/] >256 chars")
|
||||
raise typer.Exit(2)
|
||||
|
||||
if layer is not None and (layer < 0 or layer > 2048):
|
||||
console.print(
|
||||
f"[red]Invalid --layer:[/] must satisfy 0 <= layer <= 2048, got {layer}"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
|
||||
spec = get_steering_method_spec(canonical_method)
|
||||
panel_body = (
|
||||
f"Base: {escape(base)}\n"
|
||||
f"Method: {escape(canonical_method)}\n"
|
||||
f"Name: {escape(canonical_name)}\n"
|
||||
f"Pairs: {escape(pairs_path)}\n"
|
||||
f"Layer: {layer if layer is not None else 'auto'}\n"
|
||||
f"Description: {escape(spec.description)}"
|
||||
)
|
||||
console.print(
|
||||
Panel(
|
||||
panel_body,
|
||||
title=f"soup steer train ({escape(canonical_method)})",
|
||||
border_style="cyan",
|
||||
)
|
||||
)
|
||||
|
||||
if plan_only:
|
||||
console.print(
|
||||
"[yellow]Plan-only mode:[/] live training deferred to v0.62.1. "
|
||||
"Inputs validated and plan rendered."
|
||||
)
|
||||
return
|
||||
|
||||
try:
|
||||
build_steering_vector(
|
||||
method=canonical_method,
|
||||
name=canonical_name,
|
||||
pairs_path=pairs_path,
|
||||
layer=layer,
|
||||
)
|
||||
except NotImplementedError as exc:
|
||||
console.print(
|
||||
Panel(
|
||||
f"[yellow]Live steer-train deferred to v0.62.1.[/]\n\n{escape(str(exc))}",
|
||||
title="Deferred",
|
||||
border_style="yellow",
|
||||
)
|
||||
)
|
||||
# Match v0.61.0 Part C policy: exit code 3 distinguishes
|
||||
# "deferred / not yet shipped" from exit code 2 = "validation
|
||||
# rejection".
|
||||
raise typer.Exit(3) from exc
|
||||
|
||||
|
||||
@app.command(name="apply")
|
||||
def apply_steer(
|
||||
name: str = typer.Option(
|
||||
..., "--name", "-n",
|
||||
help="Identifier of a stored steering vector.",
|
||||
),
|
||||
strength: float = typer.Option(
|
||||
1.0, "--strength", "-s",
|
||||
help="Steering strength multiplier (|s| <= 10.0).",
|
||||
),
|
||||
) -> None:
|
||||
"""Apply a stored steering vector at decode time (preview-only in v0.62.0)."""
|
||||
from soup_cli.utils.steering import (
|
||||
validate_steering_name,
|
||||
validate_steering_strength,
|
||||
)
|
||||
|
||||
try:
|
||||
canonical_name = validate_steering_name(name)
|
||||
canonical_strength = validate_steering_strength(strength)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid steer-apply input:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
(
|
||||
f"Vector: {escape(canonical_name)}\n"
|
||||
f"Strength: {canonical_strength}"
|
||||
),
|
||||
title="soup steer apply",
|
||||
border_style="cyan",
|
||||
)
|
||||
)
|
||||
console.print(
|
||||
"[yellow]Apply path deferred to v0.62.1.[/] Use `soup serve --steer "
|
||||
f"{escape(canonical_name)}` once the live decode hook ships."
|
||||
)
|
||||
raise typer.Exit(3)
|
||||
|
||||
|
||||
@app.command(name="list")
|
||||
def list_steers() -> None:
|
||||
"""List locally-stored steering vectors (Registry artifact kind ``steering_vector``)."""
|
||||
try:
|
||||
from soup_cli.registry.store import RegistryStore
|
||||
except ImportError as exc: # pragma: no cover - registry import optional
|
||||
console.print(f"[red]Registry unavailable:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
# Use the store as a context manager so the SQLite connection closes
|
||||
# cleanly on every invocation (review M2). The store returns dicts
|
||||
# from `.list()` / `.get_artifacts()` — NOT objects (review H1).
|
||||
table = Table(title="Steering vectors", border_style="cyan")
|
||||
table.add_column("Entry")
|
||||
table.add_column("Path")
|
||||
rows = 0
|
||||
with RegistryStore() as store:
|
||||
for entry in store.list():
|
||||
for art in store.get_artifacts(entry["id"]):
|
||||
if art.get("kind") == "steering_vector":
|
||||
table.add_row(
|
||||
escape(str(entry.get("name", ""))),
|
||||
escape(str(art.get("path", ""))),
|
||||
)
|
||||
rows += 1
|
||||
if rows == 0:
|
||||
console.print(
|
||||
"[dim]No steering_vector artifacts registered. "
|
||||
"Use `soup steer train ...` (v0.62.1+).[/]"
|
||||
)
|
||||
return
|
||||
console.print(table)
|
||||
|
|
@ -201,6 +201,8 @@ class DataConfig(BaseModel):
|
|||
"plaintext", "embedding", "audio", "tool-calling", "auto",
|
||||
# v0.42.0 — Data Pipeline Pro
|
||||
"prm", "pre_tokenized", "input_output", "video", "multimodal",
|
||||
# v0.62.0 Part A — RAFT (Retrieval-Augmented Fine-Tuning)
|
||||
"raft",
|
||||
] = Field(
|
||||
default="auto",
|
||||
description="Data format",
|
||||
|
|
@ -2350,6 +2352,142 @@ class TrainingConfig(BaseModel):
|
|||
|
||||
return validate_unlearn_alpha(v)
|
||||
|
||||
# ---- v0.62.0 Part B — RA-DIT (Retrieval-Augmented Dual Instruction
|
||||
# Tuning, Meta 2023). Schema-only: a YAML can declare ``ra_dit_stage``
|
||||
# so a recipe locks the right pairing; live two-stage orchestration
|
||||
# ships in v0.62.1 (mirrors the v0.50.0 / v0.61.0 stub-then-live
|
||||
# pattern).
|
||||
ra_dit_stage: Optional[Literal["retriever", "generator"]] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"RA-DIT pipeline stage. 'retriever' trains the sentence-"
|
||||
"transformer via the v0.16 embedding trainer; 'generator' "
|
||||
"runs RAFT-style SFT on `data.format='raft'`. Composes with "
|
||||
"the v0.62.0 Part A RAFT recipe. (v0.62.0 Part B)"
|
||||
),
|
||||
)
|
||||
ra_dit_retriever_model: Optional[str] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional retriever model id (e.g. "
|
||||
"`sentence-transformers/all-mpnet-base-v2`) used by the "
|
||||
"generator stage to pre-encode distractor docs. (v0.62.0 "
|
||||
"Part B)"
|
||||
),
|
||||
)
|
||||
|
||||
@field_validator("ra_dit_stage", mode="before")
|
||||
@classmethod
|
||||
def _validate_ra_dit_stage(cls, v):
|
||||
"""v0.62.0 Part B — case-insensitive normalisation via shared helper."""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
return validate_ra_dit_stage(v)
|
||||
|
||||
@field_validator("ra_dit_retriever_model", mode="before")
|
||||
@classmethod
|
||||
def _validate_ra_dit_retriever_model(cls, v):
|
||||
"""v0.62.0 Part B — bool/null-byte/oversize rejection on retriever id."""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
return validate_ra_dit_retriever_model(v)
|
||||
|
||||
# ---- v0.62.0 Part D — Citation-faithful FT ----------------------------
|
||||
citation_faithful: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Opt INTO citation-precision / recall scoring + a loss-mask "
|
||||
"rule that emphasises citation spans. Requires "
|
||||
"`data.format='raft'`. Schema-only in v0.62.0; live span-mask "
|
||||
"ships in v0.62.1. (v0.62.0 Part D)"
|
||||
),
|
||||
)
|
||||
citation_style: Optional[Literal["bracket", "inline", "footnote"]] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Citation rendering style. 'bracket' = `[doc-1]` inline tag "
|
||||
"(canonical RAFT default); 'inline' / 'footnote' are stub "
|
||||
"placeholders for v0.62.1. (v0.62.0 Part D)"
|
||||
),
|
||||
)
|
||||
citation_recall_threshold: Optional[float] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Reject final-save when measured citation recall < this "
|
||||
"threshold. Bounded [0.0, 1.0]. Composes with v0.56.0 "
|
||||
"diagnose-gate. (v0.62.0 Part D)"
|
||||
),
|
||||
)
|
||||
|
||||
@field_validator("citation_style", mode="before")
|
||||
@classmethod
|
||||
def _validate_citation_style(cls, v):
|
||||
"""v0.62.0 Part D — case-insensitive normalisation via shared helper."""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
return validate_citation_style(v)
|
||||
|
||||
@field_validator("citation_recall_threshold", mode="before")
|
||||
@classmethod
|
||||
def _validate_citation_recall_threshold(cls, v):
|
||||
"""v0.62.0 Part D — bool/NaN/Inf-rejected float bounded [0.0, 1.0]."""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.citation_faithful import validate_citation_threshold
|
||||
|
||||
return validate_citation_threshold(v)
|
||||
|
||||
# ---- v0.62.0 Part E — GRACE codebook ----------------------------------
|
||||
grace_codebook: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Opt INTO the GRACE codebook — discrete latent-space (key, "
|
||||
"value) store for thousands of sequential knowledge edits "
|
||||
"without norm-blowup. Schema-only in v0.62.0; live lookup / "
|
||||
"write ships in v0.62.1. (v0.62.0 Part E)"
|
||||
),
|
||||
)
|
||||
grace_codebook_size: Optional[int] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Codebook entry count. Required when grace_codebook=True. "
|
||||
"Bounded [1, 100_000]. (v0.62.0 Part E)"
|
||||
),
|
||||
)
|
||||
grace_codebook_dim: Optional[int] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Codebook entry dim (residual-stream width). Required when "
|
||||
"grace_codebook=True. Bounded [1, 16_384]. (v0.62.0 Part E)"
|
||||
),
|
||||
)
|
||||
|
||||
@field_validator("grace_codebook_size", mode="before")
|
||||
@classmethod
|
||||
def _validate_grace_codebook_size(cls, v):
|
||||
"""v0.62.0 Part E — bool-rejected positive int <= MAX_CODEBOOK_SIZE."""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_size
|
||||
|
||||
return validate_grace_codebook_size(v)
|
||||
|
||||
@field_validator("grace_codebook_dim", mode="before")
|
||||
@classmethod
|
||||
def _validate_grace_codebook_dim(cls, v):
|
||||
"""v0.62.0 Part E — bool-rejected positive int <= MAX_CODEBOOK_DIM."""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_dim
|
||||
|
||||
return validate_grace_codebook_dim(v)
|
||||
|
||||
|
||||
class EvalConfig(BaseModel):
|
||||
"""Evaluation configuration for auto-eval after training."""
|
||||
|
|
@ -3453,6 +3591,115 @@ class SoupConfig(BaseModel):
|
|||
raise ValueError(str(exc)) from exc
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_grace_codebook_compat(self) -> "SoupConfig":
|
||||
"""v0.62.0 Part E — GRACE codebook cross-validator.
|
||||
|
||||
Rules:
|
||||
* ``grace_codebook=True`` requires BOTH ``grace_codebook_size`` and
|
||||
``grace_codebook_dim`` to be set (no codebook can be allocated
|
||||
without both knobs).
|
||||
* Setting ``grace_codebook_size`` / ``grace_codebook_dim`` without
|
||||
``grace_codebook=True`` is a silent-no-op footgun — rejected.
|
||||
"""
|
||||
tcfg = self.training
|
||||
flag = tcfg.grace_codebook
|
||||
size = tcfg.grace_codebook_size
|
||||
dim = tcfg.grace_codebook_dim
|
||||
|
||||
if not flag and (size is not None or dim is not None):
|
||||
raise ValueError(
|
||||
"training.grace_codebook_size / grace_codebook_dim require "
|
||||
"training.grace_codebook=true."
|
||||
)
|
||||
if flag and (size is None or dim is None):
|
||||
raise ValueError(
|
||||
"training.grace_codebook=true requires BOTH "
|
||||
"training.grace_codebook_size and training.grace_codebook_dim."
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_citation_faithful_compat(self) -> "SoupConfig":
|
||||
"""v0.62.0 Part D — citation-faithful FT cross-validator.
|
||||
|
||||
Rules:
|
||||
* ``citation_faithful=True`` requires ``data.format='raft'`` (the
|
||||
RAFT row carries the doc references; other formats can't supply
|
||||
ground-truth citation IDs).
|
||||
* ``citation_faithful=True`` requires ``task in {sft, pretrain}``
|
||||
(the span-mask runtime that v0.62.1 will ship only makes sense
|
||||
for the SFT family; mirrors v0.52.0 distill / classifier
|
||||
task-gate policy — review M3 fix).
|
||||
* ``citation_style`` set without ``citation_faithful=True`` is a
|
||||
silent-no-op footgun — rejected (mirrors v0.61.0 unlearn_alpha /
|
||||
v0.62.0 Part B ra_dit_retriever_model policy).
|
||||
* Same rejection for ``citation_recall_threshold`` without the flag.
|
||||
"""
|
||||
tcfg = self.training
|
||||
|
||||
if tcfg.citation_style is not None and not tcfg.citation_faithful:
|
||||
raise ValueError(
|
||||
"training.citation_style requires "
|
||||
"training.citation_faithful=true."
|
||||
)
|
||||
if (
|
||||
tcfg.citation_recall_threshold is not None
|
||||
and not tcfg.citation_faithful
|
||||
):
|
||||
raise ValueError(
|
||||
"training.citation_recall_threshold requires "
|
||||
"training.citation_faithful=true."
|
||||
)
|
||||
if tcfg.citation_faithful:
|
||||
if self.data.format != "raft":
|
||||
raise ValueError(
|
||||
"training.citation_faithful=true requires "
|
||||
f"data.format='raft'; got data.format={self.data.format!r}. "
|
||||
"Citation-faithful FT pairs with the v0.62.0 Part A "
|
||||
"RAFT data format (which carries the doc references)."
|
||||
)
|
||||
if self.task not in ("sft", "pretrain"):
|
||||
raise ValueError(
|
||||
"training.citation_faithful=true requires "
|
||||
f"task in {{sft, pretrain}}; got task={self.task!r}. "
|
||||
"Citation-faithful FT is an SFT-family feature; the "
|
||||
"live span-mask runtime ships in v0.62.1."
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_ra_dit_compat(self) -> "SoupConfig":
|
||||
"""v0.62.0 Part B — RA-DIT stage / task pairing.
|
||||
|
||||
Each stage requires the matching base task:
|
||||
|
||||
* ``retriever`` -> ``task='embedding'``
|
||||
* ``generator`` -> ``task='sft'``
|
||||
|
||||
Also rejects ``ra_dit_retriever_model`` set without ``ra_dit_stage``
|
||||
(silent no-op footgun — mirrors v0.61.0 ``unlearn_alpha`` policy).
|
||||
"""
|
||||
tcfg = self.training
|
||||
stage = tcfg.ra_dit_stage
|
||||
|
||||
if tcfg.ra_dit_retriever_model is not None and stage is None:
|
||||
raise ValueError(
|
||||
"training.ra_dit_retriever_model requires "
|
||||
"training.ra_dit_stage to be set ('retriever' or 'generator')."
|
||||
)
|
||||
|
||||
if stage is None:
|
||||
return self
|
||||
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_compat
|
||||
|
||||
try:
|
||||
validate_ra_dit_compat(stage=stage, task=self.task)
|
||||
except ValueError as exc:
|
||||
raise ValueError(str(exc)) from exc
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_mlx_task_support(self) -> "SoupConfig":
|
||||
"""MLX backend only supports sft, dpo, and grpo tasks (v0.25.0).
|
||||
|
|
|
|||
|
|
@ -85,6 +85,8 @@ def format_to_messages(row: dict, fmt: str) -> Optional[dict]:
|
|||
"plaintext", "embedding", "audio", "tool-calling",
|
||||
# v0.42.0 Part A
|
||||
"prm", "pre_tokenized", "input_output", "video", "multimodal",
|
||||
# v0.62.0 Part A — RAFT (Retrieval-Augmented Fine-Tuning).
|
||||
"raft",
|
||||
)
|
||||
if fmt not in valid_formats:
|
||||
raise ValueError(f"Unknown format: {fmt}")
|
||||
|
|
@ -117,6 +119,8 @@ def format_to_messages(row: dict, fmt: str) -> Optional[dict]:
|
|||
return _convert_video(row)
|
||||
elif fmt == "multimodal":
|
||||
return _convert_multimodal(row)
|
||||
elif fmt == "raft":
|
||||
return _convert_raft(row)
|
||||
else:
|
||||
return _convert_vision(row)
|
||||
except (KeyError, TypeError, IndexError, ValueError):
|
||||
|
|
@ -424,6 +428,10 @@ def _to_sharegpt(messages: list[dict]) -> dict:
|
|||
|
||||
_MAX_PRM_STEPS = 10_000
|
||||
|
||||
# v0.62.0 Part A — RAFT (Retrieval-Augmented Fine-Tuning) caps.
|
||||
_MAX_RAFT_DISTRACTORS = 64
|
||||
_MAX_RAFT_FIELD_LEN = 65_536 # 64 KiB per document — generous for legal/RAG corpora.
|
||||
|
||||
|
||||
def _convert_prm(row: dict) -> dict:
|
||||
"""PRM (Process Reward Model) stepwise-supervised format.
|
||||
|
|
@ -535,3 +543,74 @@ def _convert_multimodal(row: dict) -> dict:
|
|||
f"multimodal content part.type must be in {sorted(valid_types)}"
|
||||
)
|
||||
return {"messages": messages}
|
||||
|
||||
|
||||
# --- v0.62.0 Part A: RAFT (Retrieval-Augmented Fine-Tuning) ----------------
|
||||
|
||||
|
||||
def _check_raft_string(name: str, value: object) -> str:
|
||||
"""Shared validator for RAFT string fields (query / golden_doc / answer).
|
||||
|
||||
Returns the canonical value. Rejects non-string, empty, null-byte, and
|
||||
oversize values (mirrors v0.42.0 `_convert_video` policy). The cap is
|
||||
generous (64 KiB) because legal/RAG corpora frequently embed full
|
||||
paragraphs verbatim in the golden_doc field.
|
||||
"""
|
||||
if not isinstance(value, str):
|
||||
raise ValueError(
|
||||
f"RAFT '{name}' must be a string, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError(f"RAFT '{name}' must be a non-empty string")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"RAFT '{name}' must not contain null bytes")
|
||||
if len(value) > _MAX_RAFT_FIELD_LEN:
|
||||
raise ValueError(
|
||||
f"RAFT '{name}' must be <= {_MAX_RAFT_FIELD_LEN} chars"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _convert_raft(row: dict) -> dict:
|
||||
"""RAFT (Retrieval-Augmented Fine-Tuning) format — Stanford 2024.
|
||||
|
||||
Schema: ``{"query": str, "golden_doc": str, "distractor_docs": [str, ...],
|
||||
"answer": str}``. The trainer composes the prompt by concatenating the
|
||||
query with the golden doc + N distractor docs in randomised order; the
|
||||
model learns to attend to the relevant doc while ignoring distractors.
|
||||
|
||||
Distractor list MAY be empty (effectively reduces to closed-book QA on
|
||||
the golden doc). Live RAFT training loop ships in v0.62.1; v0.62.0
|
||||
locks the schema + recipe surface.
|
||||
"""
|
||||
if "query" not in row:
|
||||
raise ValueError("RAFT row must have 'query'")
|
||||
if "golden_doc" not in row:
|
||||
raise ValueError("RAFT row must have 'golden_doc'")
|
||||
if "answer" not in row:
|
||||
raise ValueError("RAFT row must have 'answer'")
|
||||
|
||||
query = _check_raft_string("query", row["query"])
|
||||
golden_doc = _check_raft_string("golden_doc", row["golden_doc"])
|
||||
answer = _check_raft_string("answer", row["answer"])
|
||||
|
||||
raw_distractors = row.get("distractor_docs", [])
|
||||
if not isinstance(raw_distractors, list):
|
||||
raise ValueError("RAFT 'distractor_docs' must be a list")
|
||||
if len(raw_distractors) > _MAX_RAFT_DISTRACTORS:
|
||||
raise ValueError(
|
||||
f"RAFT 'distractor_docs' must have <= {_MAX_RAFT_DISTRACTORS} entries "
|
||||
f"(got {len(raw_distractors)})"
|
||||
)
|
||||
cleaned_distractors: list[str] = []
|
||||
for index, doc in enumerate(raw_distractors):
|
||||
cleaned_distractors.append(
|
||||
_check_raft_string(f"distractor_docs[{index}]", doc)
|
||||
)
|
||||
|
||||
return {
|
||||
"query": query,
|
||||
"golden_doc": golden_doc,
|
||||
"distractor_docs": cleaned_distractors,
|
||||
"answer": answer,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3413,6 +3413,101 @@ training:
|
|||
batch_size: auto
|
||||
tts_family: oute
|
||||
|
||||
output: ./output
|
||||
""",
|
||||
),
|
||||
"ra-dit-retriever": RecipeMeta(
|
||||
model="sentence-transformers/all-mpnet-base-v2",
|
||||
task="embedding",
|
||||
size="N/A",
|
||||
tags=("ra-dit", "rag", "retriever", "contrastive", "v0.62.0"),
|
||||
description=(
|
||||
"RA-DIT stage 1 (Meta 2023) — contrastive retriever training. "
|
||||
"Pairs with `ra-dit-llama3-8b` (stage 2) for the full pipeline."
|
||||
),
|
||||
yaml_str="""\
|
||||
base: sentence-transformers/all-mpnet-base-v2
|
||||
task: embedding
|
||||
|
||||
data:
|
||||
train: ./data/triples.jsonl
|
||||
format: embedding
|
||||
max_length: 512
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-5
|
||||
batch_size: auto
|
||||
ra_dit_stage: retriever
|
||||
embedding_loss: triplet
|
||||
embedding_margin: 0.5
|
||||
|
||||
output: ./output
|
||||
""",
|
||||
),
|
||||
"ra-dit-llama3-8b": RecipeMeta(
|
||||
model="meta-llama/Llama-3.1-8B-Instruct",
|
||||
task="sft",
|
||||
size="8B",
|
||||
tags=("llama", "sft", "ra-dit", "rag", "raft", "v0.62.0"),
|
||||
description=(
|
||||
"RA-DIT stage 2 (Meta 2023) — RAFT-style SFT on the generator. "
|
||||
"Pairs with `ra-dit-retriever` (stage 1). Uses RAFT data format."
|
||||
),
|
||||
yaml_str="""\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
max_length: 4096
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
ra_dit_stage: generator
|
||||
ra_dit_retriever_model: sentence-transformers/all-mpnet-base-v2
|
||||
lora:
|
||||
r: 16
|
||||
alpha: 32
|
||||
target_modules: auto
|
||||
quantization: 4bit
|
||||
|
||||
output: ./output
|
||||
""",
|
||||
),
|
||||
"raft-llama3-8b": RecipeMeta(
|
||||
model="meta-llama/Llama-3.1-8B-Instruct",
|
||||
task="sft",
|
||||
size="8B",
|
||||
tags=("llama", "sft", "raft", "rag", "v0.62.0"),
|
||||
description=(
|
||||
"RAFT (Retrieval-Augmented Fine-Tuning, Stanford 2024) — train "
|
||||
"an 8B Llama 3.1 to answer queries given a golden doc + "
|
||||
"distractor docs. Rows: {query, golden_doc, distractor_docs, "
|
||||
"answer}. Live training loop ships in v0.62.1."
|
||||
),
|
||||
yaml_str="""\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
max_length: 4096
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
lora:
|
||||
r: 16
|
||||
alpha: 32
|
||||
target_modules: auto
|
||||
quantization: 4bit
|
||||
|
||||
output: ./output
|
||||
""",
|
||||
),
|
||||
|
|
|
|||
|
|
@ -41,6 +41,8 @@ _VALID_KINDS = frozenset(
|
|||
"adapter", "merged", "gguf", "awq", "gptq", "onnx", "dataset", "config",
|
||||
"eval_results", "tensorrt", "eval_suite", "canaries",
|
||||
"diagnose_report",
|
||||
# v0.62.0 Part C — Activation steering vectors (CAA / ITI / RepE).
|
||||
"steering_vector",
|
||||
}
|
||||
)
|
||||
_VALID_RELATIONS = frozenset(
|
||||
|
|
|
|||
|
|
@ -0,0 +1,210 @@
|
|||
"""v0.62.0 Part D — Citation-faithful fine-tuning.
|
||||
|
||||
When enabled, the model is trained to cite document IDs verbatim from
|
||||
the training corpus. Composes with v0.62.0 Part A RAFT — the RAFT row
|
||||
already names the golden_doc + distractor_docs.
|
||||
|
||||
Schema-only release:
|
||||
|
||||
* The schema flag ``training.citation_faithful: bool`` opts INTO the
|
||||
citation-precision / recall scorer + a loss-mask rule that emphasises
|
||||
citation spans.
|
||||
* The live span-mask training kernel and the eval-suite hook land in
|
||||
v0.62.1, mirroring v0.50.0 / v0.52.0 / v0.61.0 stub-then-live policy.
|
||||
|
||||
This module ships the pure ``score_citations`` kernel so callers (and
|
||||
the eval gate) can compute precision/recall/F1 today, plus the closed
|
||||
allowlist for ``citation_style``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
|
||||
SUPPORTED_CITATION_STYLES: frozenset[str] = frozenset(
|
||||
{"bracket", "inline", "footnote"}
|
||||
)
|
||||
|
||||
_MAX_STYLE_LEN: int = 32
|
||||
_MAX_PREDICTED_LEN: int = 2_000_000 # 2 MB cap on per-row predicted text.
|
||||
_MAX_EXPECTED_IDS: int = 10_000 # Per-row expected-citation cap.
|
||||
|
||||
# Default extraction regex — matches both ``[doc-id]`` brackets and bare
|
||||
# ``doc-id`` tokens. The bracket form is the canonical RAFT default;
|
||||
# ``inline`` and ``footnote`` use the same characters today (live
|
||||
# per-style extractors ship in v0.62.1 once we benchmark variations).
|
||||
_CITATION_RE: re.Pattern[str] = re.compile(
|
||||
r"\[(?P<bracketed>[A-Za-z0-9][A-Za-z0-9._\-]{0,127})\]"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CitationScore:
|
||||
"""Precision / recall / F1 over predicted vs expected document IDs."""
|
||||
|
||||
precision: float
|
||||
recall: float
|
||||
f1: float
|
||||
predicted_count: int
|
||||
expected_count: int
|
||||
|
||||
|
||||
def validate_citation_style(value: object) -> str:
|
||||
"""Normalise + validate a citation-style name.
|
||||
|
||||
Mirrors v0.41.0 / v0.51.0 / v0.61.0 validator policy.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"citation_style must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"citation_style must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("citation_style must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("citation_style must not contain null bytes")
|
||||
if len(value) > _MAX_STYLE_LEN:
|
||||
raise ValueError(
|
||||
f"citation_style must be <= {_MAX_STYLE_LEN} chars"
|
||||
)
|
||||
canonical = value.lower()
|
||||
if canonical not in SUPPORTED_CITATION_STYLES:
|
||||
supported = ", ".join(sorted(SUPPORTED_CITATION_STYLES))
|
||||
raise ValueError(
|
||||
f"unknown citation_style {value!r}; supported: {supported}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def validate_citation_threshold(value: object) -> float:
|
||||
"""Validate the ``citation_recall_threshold`` in ``[0.0, 1.0]``.
|
||||
|
||||
Bool-rejected, NaN/Inf-rejected via ``math.isfinite``.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"citation_recall_threshold must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(
|
||||
f"citation_recall_threshold must be a number, "
|
||||
f"got {type(value).__name__}"
|
||||
)
|
||||
fval = float(value)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError(
|
||||
"citation_recall_threshold must be finite (no NaN / Inf)"
|
||||
)
|
||||
if fval < 0.0 or fval > 1.0:
|
||||
raise ValueError(
|
||||
f"citation_recall_threshold must be in [0.0, 1.0]; got {fval}"
|
||||
)
|
||||
return fval
|
||||
|
||||
|
||||
def extract_citation_ids(text: str) -> tuple[str, ...]:
|
||||
"""Extract every ``[doc-id]`` citation from ``text``.
|
||||
|
||||
Returns a tuple of IDs in encounter order. Duplicates are preserved
|
||||
so the caller can compute precision honestly (a model that cites
|
||||
the same doc three times should not silently dedupe).
|
||||
"""
|
||||
if not isinstance(text, str):
|
||||
raise TypeError(
|
||||
f"text must be str, got {type(text).__name__}"
|
||||
)
|
||||
if len(text) > _MAX_PREDICTED_LEN:
|
||||
raise ValueError(
|
||||
f"text must be <= {_MAX_PREDICTED_LEN} chars for citation extract"
|
||||
)
|
||||
return tuple(m.group("bracketed") for m in _CITATION_RE.finditer(text))
|
||||
|
||||
|
||||
def score_citations(
|
||||
*,
|
||||
predicted: object,
|
||||
expected_ids: object,
|
||||
) -> CitationScore:
|
||||
"""Compute citation precision / recall / F1.
|
||||
|
||||
Precision = |predicted ∩ expected| / |predicted|.
|
||||
Recall = |predicted ∩ expected| / |expected|.
|
||||
|
||||
Undefined denominators (empty predicted / expected) return 0.0 by
|
||||
convention — same policy as v0.43.0 BLEU on zero-precision n-grams.
|
||||
"""
|
||||
if isinstance(predicted, bool):
|
||||
raise TypeError(
|
||||
f"predicted must not be bool, got {predicted!r}"
|
||||
)
|
||||
if not isinstance(predicted, str):
|
||||
raise TypeError(
|
||||
f"predicted must be str, got {type(predicted).__name__}"
|
||||
)
|
||||
if len(predicted) > _MAX_PREDICTED_LEN:
|
||||
raise ValueError(
|
||||
f"predicted must be <= {_MAX_PREDICTED_LEN} chars"
|
||||
)
|
||||
if not isinstance(expected_ids, Iterable) or isinstance(expected_ids, str):
|
||||
raise TypeError(
|
||||
"expected_ids must be an iterable of strings (not a single str)"
|
||||
)
|
||||
expected_tuple = tuple(expected_ids)
|
||||
if len(expected_tuple) > _MAX_EXPECTED_IDS:
|
||||
raise ValueError(
|
||||
f"expected_ids must have <= {_MAX_EXPECTED_IDS} entries"
|
||||
)
|
||||
expected_set: set[str] = set()
|
||||
for index, eid in enumerate(expected_tuple):
|
||||
if isinstance(eid, bool) or not isinstance(eid, str):
|
||||
raise TypeError(
|
||||
f"expected_ids[{index}] must be str, got {type(eid).__name__}"
|
||||
)
|
||||
if not eid:
|
||||
raise ValueError(f"expected_ids[{index}] must be non-empty")
|
||||
if "\x00" in eid:
|
||||
raise ValueError(
|
||||
f"expected_ids[{index}] must not contain null bytes"
|
||||
)
|
||||
expected_set.add(eid)
|
||||
|
||||
predicted_ids = extract_citation_ids(predicted)
|
||||
predicted_count = len(predicted_ids)
|
||||
expected_count = len(expected_set)
|
||||
|
||||
if predicted_count == 0 or expected_count == 0:
|
||||
return CitationScore(
|
||||
precision=0.0,
|
||||
recall=0.0,
|
||||
f1=0.0,
|
||||
predicted_count=predicted_count,
|
||||
expected_count=expected_count,
|
||||
)
|
||||
|
||||
# Precision: predicted IDs that hit the expected set.
|
||||
hits = sum(1 for pid in predicted_ids if pid in expected_set)
|
||||
precision = hits / predicted_count
|
||||
|
||||
# Recall: how many expected IDs the model actually cited.
|
||||
predicted_set = set(predicted_ids)
|
||||
recalled = sum(1 for eid in expected_set if eid in predicted_set)
|
||||
recall = recalled / expected_count
|
||||
|
||||
if precision + recall <= 0.0:
|
||||
f1 = 0.0
|
||||
else:
|
||||
f1 = 2.0 * precision * recall / (precision + recall)
|
||||
|
||||
return CitationScore(
|
||||
precision=precision,
|
||||
recall=recall,
|
||||
f1=f1,
|
||||
predicted_count=predicted_count,
|
||||
expected_count=expected_count,
|
||||
)
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
"""v0.62.0 Part E — GRACE codebook (long-running edit).
|
||||
|
||||
Discrete latent-space codebook for thousands of sequential knowledge edits
|
||||
that survive lifelong deployments without the norm-blowup that haunts
|
||||
vanilla ROME / MEMIT. Each edit stores a (key, value) pair in a learned
|
||||
codebook; at inference time the model looks up the closest codebook key
|
||||
to the current residual stream and applies the stored value.
|
||||
|
||||
Schema-only release: ``training.grace_codebook`` opt-in + codebook
|
||||
size / dim validators + ``GraceCodebookConfig`` dataclass + ``grace``
|
||||
added to the v0.61.0 ``SUPPORTED_EDIT_METHODS`` allowlist. Live codebook
|
||||
lookup / write / EditGovernor integration lands in v0.62.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
MAX_CODEBOOK_SIZE: int = 100_000
|
||||
MAX_CODEBOOK_DIM: int = 16_384 # Generous upper bound matches Llama 70B hidden.
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GraceCodebookConfig:
|
||||
"""Resolved codebook configuration. Frozen post-construction."""
|
||||
|
||||
size: int
|
||||
dim: int
|
||||
|
||||
|
||||
def validate_grace_codebook_size(value: object) -> int:
|
||||
"""Validate the codebook entry count.
|
||||
|
||||
Bool-rejected (bool is a subclass of int), positive-int only, capped
|
||||
at :data:`MAX_CODEBOOK_SIZE` so a misconfigured run cannot allocate
|
||||
a multi-GB codebook by accident.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"grace_codebook_size must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, int):
|
||||
raise TypeError(
|
||||
f"grace_codebook_size must be int, got {type(value).__name__}"
|
||||
)
|
||||
if value < 1:
|
||||
raise ValueError(
|
||||
f"grace_codebook_size must be >= 1, got {value}"
|
||||
)
|
||||
if value > MAX_CODEBOOK_SIZE:
|
||||
raise ValueError(
|
||||
f"grace_codebook_size must be <= {MAX_CODEBOOK_SIZE}, got {value}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_grace_codebook_dim(value: object) -> int:
|
||||
"""Validate the codebook entry dimension (residual-stream width)."""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"grace_codebook_dim must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, int):
|
||||
raise TypeError(
|
||||
f"grace_codebook_dim must be int, got {type(value).__name__}"
|
||||
)
|
||||
if value < 1:
|
||||
raise ValueError(
|
||||
f"grace_codebook_dim must be >= 1, got {value}"
|
||||
)
|
||||
if value > MAX_CODEBOOK_DIM:
|
||||
raise ValueError(
|
||||
f"grace_codebook_dim must be <= {MAX_CODEBOOK_DIM}, got {value}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def build_grace_codebook_config(*, size: int, dim: int) -> GraceCodebookConfig:
|
||||
"""Validate + freeze a :class:`GraceCodebookConfig`."""
|
||||
canonical_size = validate_grace_codebook_size(size)
|
||||
canonical_dim = validate_grace_codebook_dim(dim)
|
||||
return GraceCodebookConfig(size=canonical_size, dim=canonical_dim)
|
||||
|
||||
|
||||
def apply_grace_codebook(config: GraceCodebookConfig) -> None:
|
||||
"""Apply the codebook at decode time — deferred to v0.62.1.
|
||||
|
||||
Validates the config type first so callers passing a bare dict get a
|
||||
crisp ``TypeError`` rather than the deferred-live ``NotImplementedError``.
|
||||
Mirrors v0.50.0 ``apply_variant_loss`` / v0.61.0 ``apply_unlearn_loss``
|
||||
policy.
|
||||
"""
|
||||
if not isinstance(config, GraceCodebookConfig):
|
||||
raise TypeError(
|
||||
f"apply_grace_codebook expects GraceCodebookConfig, "
|
||||
f"got {type(config).__name__}"
|
||||
)
|
||||
raise NotImplementedError(
|
||||
"apply_grace_codebook is deferred to v0.62.1. Schema accepts "
|
||||
f"size={config.size} dim={config.dim} today; live codebook "
|
||||
"lookup / write ships next release."
|
||||
)
|
||||
|
|
@ -23,7 +23,7 @@ from types import MappingProxyType
|
|||
from typing import Mapping, Optional
|
||||
|
||||
SUPPORTED_EDIT_METHODS: frozenset[str] = frozenset(
|
||||
{"rome", "memit", "alphaedit"}
|
||||
{"rome", "memit", "alphaedit", "grace"}
|
||||
)
|
||||
|
||||
_MAX_METHOD_LEN: int = 32
|
||||
|
|
@ -71,6 +71,17 @@ _EDIT_METHOD_METADATA: Mapping[str, EditMethodSpec] = MappingProxyType({
|
|||
multi_edit_capable=True,
|
||||
live_wired=False,
|
||||
),
|
||||
"grace": EditMethodSpec(
|
||||
name="grace",
|
||||
description=(
|
||||
"GRACE codebook — discrete latent-space (key, value) store "
|
||||
"that survives thousands of sequential edits without "
|
||||
"norm-blowup. v0.62.0 Part E ships the schema; live "
|
||||
"lookup / write kernel lands in v0.62.1."
|
||||
),
|
||||
multi_edit_capable=True,
|
||||
live_wired=False,
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
|
|
@ -197,6 +208,10 @@ _DEFAULT_EDIT_LAYER: Mapping[str, int] = MappingProxyType({
|
|||
"rome": 5,
|
||||
"memit": 8,
|
||||
"alphaedit": 5,
|
||||
# GRACE writes to a single dedicated codebook so the "layer" arg is
|
||||
# the residual-stream layer where the lookup hook is installed.
|
||||
# v0.62.0 Part E ships the schema; default mirrors AlphaEdit.
|
||||
"grace": 5,
|
||||
})
|
||||
|
||||
|
||||
|
|
@ -234,7 +249,7 @@ def build_edit_plan(
|
|||
|
||||
|
||||
def apply_edit(plan: EditPlan) -> None:
|
||||
"""Apply a knowledge edit — deferred to v0.61.1.
|
||||
"""Apply a knowledge edit — deferred to v0.61.1 (or v0.62.1 for ``grace``).
|
||||
|
||||
Re-validates the method so callers passing a bare-class duck-typed
|
||||
plan (no ``EditPlan``) still hit a meaningful error before the
|
||||
|
|
@ -242,11 +257,13 @@ def apply_edit(plan: EditPlan) -> None:
|
|||
"""
|
||||
method_attr = getattr(plan, "method", None)
|
||||
canonical = validate_edit_method(method_attr)
|
||||
# GRACE was added in v0.62.0 Part E with its own live-wiring schedule.
|
||||
target_version = "v0.62.1" if canonical == "grace" else "v0.61.1"
|
||||
raise NotImplementedError(
|
||||
f"apply_edit(method={canonical!r}) is deferred to v0.61.1. "
|
||||
f"apply_edit(method={canonical!r}) is deferred to {target_version}. "
|
||||
"Schema accepts the request now so YAML / CLI invocations are "
|
||||
"stable, but ROME / MEMIT / AlphaEdit live kernels land in "
|
||||
"v0.61.1."
|
||||
"stable, but ROME / MEMIT / AlphaEdit / GRACE live kernels land "
|
||||
"next release."
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,159 @@
|
|||
"""v0.62.0 Part B — RA-DIT (Retrieval-Augmented Dual Instruction Tuning).
|
||||
|
||||
Meta 2023 recipe. Two-stage:
|
||||
|
||||
* ``retriever`` — train a sentence-transformer (contrastive triplet loss)
|
||||
on (query, golden_doc, distractor) triples. Composes with the existing
|
||||
v0.16 embedding trainer.
|
||||
* ``generator`` — RAFT-style SFT on the query + golden_doc + distractor
|
||||
bundle (uses v0.62.0 Part A ``data.format='raft'``).
|
||||
|
||||
Schema-only release. Both stages share the existing trainer wrappers;
|
||||
v0.62.0 ships the ``ra_dit_stage`` schema field + cross-validator so a
|
||||
``soup.yaml`` can lock both stages in a hub-shareable recipe. Live
|
||||
orchestration that chains the two stages in a single ``soup train`` call
|
||||
is deferred to v0.62.1 (mirrors the v0.50.0 / v0.52.0 / v0.61.0
|
||||
stub-then-live pattern).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping, Optional
|
||||
|
||||
SUPPORTED_RA_DIT_STAGES: frozenset[str] = frozenset({"retriever", "generator"})
|
||||
|
||||
_MAX_STAGE_LEN: int = 32
|
||||
_MAX_RETRIEVER_MODEL_LEN: int = 512
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RaDitStageSpec:
|
||||
"""Metadata for a single RA-DIT training stage. Frozen post-construction."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
base_task: str
|
||||
live_wired: bool
|
||||
|
||||
|
||||
_RA_DIT_STAGE_METADATA: Mapping[str, RaDitStageSpec] = MappingProxyType({
|
||||
"retriever": RaDitStageSpec(
|
||||
name="retriever",
|
||||
description=(
|
||||
"Stage 1 — train a sentence-transformer with contrastive loss "
|
||||
"on (anchor, positive, negative) triples. Composes with the "
|
||||
"v0.16 embedding trainer."
|
||||
),
|
||||
base_task="embedding",
|
||||
live_wired=False,
|
||||
),
|
||||
"generator": RaDitStageSpec(
|
||||
name="generator",
|
||||
description=(
|
||||
"Stage 2 — SFT the generator on RAFT-style rows "
|
||||
"{query, golden_doc, distractor_docs, answer}. Uses "
|
||||
"v0.62.0 Part A `data.format='raft'`."
|
||||
),
|
||||
base_task="sft",
|
||||
live_wired=False,
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
def validate_ra_dit_stage(value: object) -> str:
|
||||
"""Normalise + validate an RA-DIT stage name.
|
||||
|
||||
Returns the canonical (lowercase) form. Mirrors v0.51.0
|
||||
``validate_hub_name`` / v0.61.0 ``validate_unlearn_method`` policy:
|
||||
bool-rejected, null-byte-rejected, oversize-rejected, case-insensitive
|
||||
normalisation, unknown rejected with friendly actionable message.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"ra_dit_stage must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"ra_dit_stage must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("ra_dit_stage must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("ra_dit_stage must not contain null bytes")
|
||||
if len(value) > _MAX_STAGE_LEN:
|
||||
raise ValueError(
|
||||
f"ra_dit_stage must be <= {_MAX_STAGE_LEN} chars"
|
||||
)
|
||||
canonical = value.lower()
|
||||
if canonical not in SUPPORTED_RA_DIT_STAGES:
|
||||
supported = ", ".join(sorted(SUPPORTED_RA_DIT_STAGES))
|
||||
raise ValueError(
|
||||
f"unknown ra_dit_stage {value!r}; supported: {supported}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def get_ra_dit_stage_spec(name: str) -> RaDitStageSpec:
|
||||
"""Return the frozen :class:`RaDitStageSpec` for ``name`` or raise."""
|
||||
canonical = validate_ra_dit_stage(name)
|
||||
return _RA_DIT_STAGE_METADATA[canonical]
|
||||
|
||||
|
||||
def validate_ra_dit_retriever_model(value: object) -> Optional[str]:
|
||||
"""Validate the operator-supplied retriever-model HF repo id or local path.
|
||||
|
||||
Bool / non-string rejected. ``None`` passes through unchanged so
|
||||
callers can rely on Pydantic ``Optional[str]`` semantics. Length cap
|
||||
matches the v0.40.5 ``reward_model`` policy (512 chars).
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"ra_dit_retriever_model must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"ra_dit_retriever_model must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("ra_dit_retriever_model must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(
|
||||
"ra_dit_retriever_model must not contain null bytes"
|
||||
)
|
||||
if len(value) > _MAX_RETRIEVER_MODEL_LEN:
|
||||
raise ValueError(
|
||||
f"ra_dit_retriever_model must be <= {_MAX_RETRIEVER_MODEL_LEN} chars"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_ra_dit_compat(*, stage: str, task: str) -> None:
|
||||
"""Schema-time gate: each stage must pair with the right base task.
|
||||
|
||||
* ``retriever`` -> ``task='embedding'`` (contrastive trainer).
|
||||
* ``generator`` -> ``task='sft'`` (RAFT-style SFT).
|
||||
|
||||
Raises ``ValueError`` on mismatch with a friendly actionable message.
|
||||
"""
|
||||
for name, value in (("stage", stage), ("task", task)):
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{name} must not be bool, got {value!r}")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"{name} must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError(f"{name} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{name} must not contain null bytes")
|
||||
canonical_stage = validate_ra_dit_stage(stage)
|
||||
expected_task = _RA_DIT_STAGE_METADATA[canonical_stage].base_task
|
||||
if task != expected_task:
|
||||
raise ValueError(
|
||||
f"training.ra_dit_stage={canonical_stage!r} requires "
|
||||
f"task={expected_task!r}; got task={task!r}."
|
||||
)
|
||||
|
|
@ -0,0 +1,234 @@
|
|||
"""v0.62.0 Part C — Activation steering (CAA / ITI / RepE).
|
||||
|
||||
Three control-vector backends for inference-time intervention:
|
||||
|
||||
* ``caa`` — Contrastive Activation Addition (Panickssery et al., 2023).
|
||||
Add a contrastive vector to the residual stream.
|
||||
* ``iti`` — Inference-Time Intervention (Li et al., 2023). Shift specific
|
||||
attention heads.
|
||||
* ``repe`` — Representation Engineering (Zou et al., 2023). PCA-based
|
||||
direction in the residual stream.
|
||||
|
||||
Schema-only release: validators + frozen dataclasses + CLI surface ship
|
||||
in v0.62.0. The live forward-hook + decode-time intervention land in
|
||||
v0.62.1, mirroring the v0.50.0 / v0.52.0 / v0.61.0 stub-then-live cadence.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping, Optional
|
||||
|
||||
SUPPORTED_STEERING_METHODS: frozenset[str] = frozenset({"caa", "iti", "repe"})
|
||||
|
||||
_MAX_METHOD_LEN: int = 32
|
||||
_MAX_NAME_LEN: int = 128
|
||||
_MAX_STRENGTH_ABS: float = 10.0 # |strength| <= 10 sanity cap.
|
||||
|
||||
# Kebab-case + underscore + dots only. Path-separators / whitespace /
|
||||
# shell-metacharacters all rejected so the name can be safely embedded in
|
||||
# CLI args, filenames, and Rich markup. Mirrors v0.57.0 adapter-branch
|
||||
# policy (alphanumeric + `._-`).
|
||||
_NAME_RE: re.Pattern[str] = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._\-]{0,127}$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SteeringMethodSpec:
|
||||
"""Metadata for a single steering backend. Frozen post-construction."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
needs_contrastive_pairs: bool
|
||||
needs_attention_heads: bool
|
||||
live_wired: bool
|
||||
|
||||
|
||||
_STEERING_METHOD_METADATA: Mapping[str, SteeringMethodSpec] = MappingProxyType({
|
||||
"caa": SteeringMethodSpec(
|
||||
name="caa",
|
||||
description=(
|
||||
"Contrastive Activation Addition — add a contrastive vector "
|
||||
"to the residual stream during decoding. Trains on "
|
||||
"(positive, negative) prompt pairs (Panickssery et al., 2023)."
|
||||
),
|
||||
needs_contrastive_pairs=True,
|
||||
needs_attention_heads=False,
|
||||
live_wired=False,
|
||||
),
|
||||
"iti": SteeringMethodSpec(
|
||||
name="iti",
|
||||
description=(
|
||||
"Inference-Time Intervention — shift specific attention heads "
|
||||
"along a learned direction (Li et al., 2023). Needs per-head "
|
||||
"calibration."
|
||||
),
|
||||
needs_contrastive_pairs=True,
|
||||
needs_attention_heads=True,
|
||||
live_wired=False,
|
||||
),
|
||||
"repe": SteeringMethodSpec(
|
||||
name="repe",
|
||||
description=(
|
||||
"Representation Engineering — PCA over hidden states to "
|
||||
"extract a behavioural direction (Zou et al., 2023)."
|
||||
),
|
||||
needs_contrastive_pairs=True,
|
||||
needs_attention_heads=False,
|
||||
live_wired=False,
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
def validate_steering_method(value: object) -> str:
|
||||
"""Normalise + validate a steering-method name.
|
||||
|
||||
Mirrors v0.41.0 / v0.51.0 / v0.61.0 validator policy: bool-rejected,
|
||||
null-byte-rejected, oversize-rejected, case-insensitive normalisation,
|
||||
unknown rejected with friendly actionable message.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"steering method must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"steering method must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("steering method must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("steering method must not contain null bytes")
|
||||
if len(value) > _MAX_METHOD_LEN:
|
||||
raise ValueError(
|
||||
f"steering method must be <= {_MAX_METHOD_LEN} chars"
|
||||
)
|
||||
canonical = value.lower()
|
||||
if canonical not in SUPPORTED_STEERING_METHODS:
|
||||
supported = ", ".join(sorted(SUPPORTED_STEERING_METHODS))
|
||||
raise ValueError(
|
||||
f"unknown steering method {value!r}; supported: {supported}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def validate_steering_name(value: object) -> str:
|
||||
"""Validate an operator-supplied steering-vector name.
|
||||
|
||||
Returns the value unchanged on success. Closed regex allowlist
|
||||
(alphanumeric + ``._-``, leading-alnum, ≤128 chars) so the name can
|
||||
be safely used as a Registry artifact id, CLI flag, and filename
|
||||
fragment on every platform.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"steering name must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"steering name must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("steering name must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("steering name must not contain null bytes")
|
||||
if len(value) > _MAX_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"steering name must be <= {_MAX_NAME_LEN} chars"
|
||||
)
|
||||
if not _NAME_RE.match(value):
|
||||
raise ValueError(
|
||||
f"steering name {value!r} must match {_NAME_RE.pattern!r} "
|
||||
"(alphanumeric + `._-`, leading alnum)."
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_steering_strength(value: object) -> float:
|
||||
"""Validate a steering strength multiplier.
|
||||
|
||||
Bool-rejected (bool is a subclass of int), NaN/Inf-rejected via
|
||||
``math.isfinite``, bounded ``|strength| <= 10.0`` as a sanity cap.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(
|
||||
f"steering strength must not be bool, got {value!r}"
|
||||
)
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(
|
||||
f"steering strength must be a number, got {type(value).__name__}"
|
||||
)
|
||||
fval = float(value)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError("steering strength must be finite (no NaN / Inf)")
|
||||
if abs(fval) > _MAX_STRENGTH_ABS:
|
||||
raise ValueError(
|
||||
f"steering strength must satisfy |s| <= {_MAX_STRENGTH_ABS}; "
|
||||
f"got {fval}"
|
||||
)
|
||||
return fval
|
||||
|
||||
|
||||
def get_steering_method_spec(name: str) -> SteeringMethodSpec:
|
||||
"""Return the frozen :class:`SteeringMethodSpec` for ``name`` or raise."""
|
||||
canonical = validate_steering_method(name)
|
||||
return _STEERING_METHOD_METADATA[canonical]
|
||||
|
||||
|
||||
def apply_steering(method: str) -> None:
|
||||
"""Apply a steering vector during decoding — deferred to v0.62.1.
|
||||
|
||||
Validates the method name first so the deferred-live error
|
||||
distinguishes between "unknown method" and "method is on the
|
||||
allowlist but not yet wired". Mirrors v0.50.0 ``apply_variant_loss`` /
|
||||
v0.61.0 ``apply_unlearn_loss`` policy.
|
||||
"""
|
||||
canonical = validate_steering_method(method)
|
||||
raise NotImplementedError(
|
||||
f"apply_steering({canonical!r}) is deferred to v0.62.1. "
|
||||
"Schema accepts the method now so callers can write soup.yaml "
|
||||
"today, but the live forward-hook + decode-time intervention "
|
||||
"land in v0.62.1."
|
||||
)
|
||||
|
||||
|
||||
def build_steering_vector(
|
||||
*,
|
||||
method: str,
|
||||
name: str,
|
||||
pairs_path: Optional[str] = None,
|
||||
layer: Optional[int] = None,
|
||||
) -> None:
|
||||
"""Train a steering vector from contrastive pairs — deferred to v0.62.1.
|
||||
|
||||
Validates inputs first; the deferred-live error fires only after the
|
||||
method + name + (optional) layer all pass shape/range checks.
|
||||
"""
|
||||
canonical = validate_steering_method(method)
|
||||
canonical_name = validate_steering_name(name)
|
||||
if pairs_path is not None:
|
||||
if not isinstance(pairs_path, str):
|
||||
raise TypeError(
|
||||
f"pairs_path must be str, got {type(pairs_path).__name__}"
|
||||
)
|
||||
if not pairs_path:
|
||||
raise ValueError("pairs_path must be non-empty")
|
||||
if "\x00" in pairs_path:
|
||||
raise ValueError("pairs_path must not contain null bytes")
|
||||
if layer is not None:
|
||||
if isinstance(layer, bool):
|
||||
raise TypeError("layer must not be bool")
|
||||
if not isinstance(layer, int):
|
||||
raise TypeError(
|
||||
f"layer must be int, got {type(layer).__name__}"
|
||||
)
|
||||
if layer < 0 or layer > 2048:
|
||||
raise ValueError(
|
||||
f"layer must satisfy 0 <= layer <= 2048, got {layer}"
|
||||
)
|
||||
raise NotImplementedError(
|
||||
f"build_steering_vector(method={canonical!r}, "
|
||||
f"name={canonical_name!r}) is deferred to v0.62.1."
|
||||
)
|
||||
|
|
@ -260,7 +260,7 @@ class TestV025NewRecipes:
|
|||
assert cfg.base == recipe.model
|
||||
assert cfg.task == recipe.task
|
||||
|
||||
def test_catalog_size_is_113(self):
|
||||
def test_catalog_size_is_116(self):
|
||||
"""Total catalog size — grew with each release.
|
||||
|
||||
v0.25.0 shipped 43 recipes (29 + 9 Part A + 2 Part B tools + 3 Part E MLX).
|
||||
|
|
@ -269,10 +269,11 @@ class TestV025NewRecipes:
|
|||
v0.51.0 added 26 (model catalog expansion) -> 106.
|
||||
v0.52.0 added 6 (5 TTS + Falcon-E BitNet) -> 112.
|
||||
v0.53.5 added 1 (deepseek-v3-reasoning) -> 113.
|
||||
v0.62.0 added 3 (raft-llama3-8b, ra-dit-retriever, ra-dit-llama3-8b) -> 116.
|
||||
"""
|
||||
from soup_cli.recipes.catalog import RECIPES
|
||||
|
||||
assert len(RECIPES) == 113
|
||||
assert len(RECIPES) == 116
|
||||
|
||||
def test_new_recipes_searchable(self):
|
||||
"""Search returns the new recipes via keyword/task filter."""
|
||||
|
|
|
|||
|
|
@ -41,7 +41,10 @@ class TestModuleSurface:
|
|||
def test_supported_methods_exact(self):
|
||||
from soup_cli.utils.knowledge_edit import SUPPORTED_EDIT_METHODS
|
||||
|
||||
assert SUPPORTED_EDIT_METHODS == frozenset({"rome", "memit", "alphaedit"})
|
||||
# v0.62.0 Part E added "grace" to the allowlist (codebook edit).
|
||||
assert SUPPORTED_EDIT_METHODS == frozenset(
|
||||
{"rome", "memit", "alphaedit", "grace"}
|
||||
)
|
||||
|
||||
|
||||
class TestValidateEditMethod:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,589 @@
|
|||
"""v0.62.0 review-fix follow-up tests.
|
||||
|
||||
Closes coverage gaps surfaced by the TDD review:
|
||||
|
||||
* H1 — citation_faithful + non-SFT-family task gate.
|
||||
* H2 — RAFT _MAX_RAFT_FIELD_LEN per-field oversize boundary.
|
||||
* H3 — RAFT null-byte rejection on golden_doc / answer / distractors.
|
||||
* H5 — `soup steer train --plan-only` deferred-marker stripping regression guard.
|
||||
* H6 — `apply_edit` still raises with v0.61.1 marker for rome/memit/alphaedit.
|
||||
* M1 — steering name regex 128-char boundary acceptance.
|
||||
* M3/M4 — soup steer train `--base` over-cap + null-byte rejection.
|
||||
* M5 — extract_citation_ids public API coverage.
|
||||
* M8 — GRACE codebook flag-without-both-knobs cross-validator.
|
||||
* M9 — validate_ra_dit_compat direct-caller null-byte / bool defence-in-depth.
|
||||
* L1 — list_steers context-manager source-grep regression guard.
|
||||
* L4 — GRACE codebook size/dim max-boundary acceptance.
|
||||
* L5 — RA-DIT retriever model 512-char boundary.
|
||||
* L7 — CitationScore predicted_count / expected_count field values.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
# ---------- H1 — citation_faithful task-gate ----------
|
||||
|
||||
|
||||
class TestCitationFaithfulTaskGate:
|
||||
def test_citation_faithful_on_grpo_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: grpo
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
citation_faithful: true
|
||||
reward_fn: accuracy
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="sft|pretrain"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_citation_faithful_on_pretrain_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: pretrain
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
citation_faithful: true
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
cfg = load_config_from_string(yaml_text)
|
||||
assert cfg.training.citation_faithful is True
|
||||
|
||||
def test_citation_faithful_on_dpo_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: dpo
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
citation_faithful: true
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="sft|pretrain"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
|
||||
# ---------- H2 — RAFT per-field oversize boundary ----------
|
||||
|
||||
|
||||
class TestRaftFieldOversizeBoundary:
|
||||
def test_query_at_cap_accepted(self):
|
||||
from soup_cli.data.formats import _MAX_RAFT_FIELD_LEN, _convert_raft
|
||||
|
||||
out = _convert_raft({
|
||||
"query": "x" * _MAX_RAFT_FIELD_LEN,
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
assert len(out["query"]) == _MAX_RAFT_FIELD_LEN
|
||||
|
||||
def test_query_overcap_rejected(self):
|
||||
from soup_cli.data.formats import _MAX_RAFT_FIELD_LEN, _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="query"):
|
||||
_convert_raft({
|
||||
"query": "x" * (_MAX_RAFT_FIELD_LEN + 1),
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_golden_doc_overcap_rejected(self):
|
||||
from soup_cli.data.formats import _MAX_RAFT_FIELD_LEN, _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="golden_doc"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "x" * (_MAX_RAFT_FIELD_LEN + 1),
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_answer_overcap_rejected(self):
|
||||
from soup_cli.data.formats import _MAX_RAFT_FIELD_LEN, _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="answer"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "x" * (_MAX_RAFT_FIELD_LEN + 1),
|
||||
})
|
||||
|
||||
def test_distractor_overcap_rejected(self):
|
||||
from soup_cli.data.formats import _MAX_RAFT_FIELD_LEN, _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="distractor"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": ["x" * (_MAX_RAFT_FIELD_LEN + 1)],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
|
||||
# ---------- H3 — RAFT null-byte rejection on every field ----------
|
||||
|
||||
|
||||
class TestRaftNullByteRejection:
|
||||
def test_null_byte_golden_doc_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "bad\x00",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_null_byte_answer_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "bad\x00",
|
||||
})
|
||||
|
||||
def test_null_byte_distractor_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": ["bad\x00doc"],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
|
||||
# ---------- H5 — `soup steer train --plan-only` deferred-marker assertion ----------
|
||||
|
||||
|
||||
class TestSteerPlanOnlyMarker:
|
||||
def test_plan_only_emits_v0621_marker(self, tmp_path, monkeypatch):
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
pairs = tmp_path / "pairs.jsonl"
|
||||
pairs.write_text(
|
||||
'{"positive": "x", "negative": "y"}\n', encoding="utf-8"
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"train",
|
||||
"--base", "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"--method", "caa",
|
||||
"--name", "safety-v1",
|
||||
"--pairs", "pairs.jsonl",
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code == 0, (
|
||||
result.output, repr(result.exception)
|
||||
)
|
||||
# Defence: the panel MUST surface the deferred-version marker so
|
||||
# operators know live training ships in v0.62.1.
|
||||
assert "v0.62.1" in result.output
|
||||
|
||||
|
||||
# ---------- H6 — apply_edit v0.61.1 marker not regressed for legacy methods ----------
|
||||
|
||||
|
||||
class TestEditMarkerRegressionGuard:
|
||||
@pytest.mark.parametrize("method", ["rome", "memit", "alphaedit"])
|
||||
def test_legacy_methods_still_use_v0611_marker(self, method: str):
|
||||
from soup_cli.utils.knowledge_edit import apply_edit, build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="meta-llama/Llama-3.1-8B-Instruct",
|
||||
method=method,
|
||||
subject="The capital of France is",
|
||||
target="Lyon",
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.61.1"):
|
||||
apply_edit(plan)
|
||||
|
||||
def test_grace_uses_v0621_marker_distinct_from_v0611(self):
|
||||
from soup_cli.utils.knowledge_edit import apply_edit, build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="meta-llama/Llama-3.1-8B-Instruct",
|
||||
method="grace",
|
||||
subject="The capital of France is",
|
||||
target="Lyon",
|
||||
)
|
||||
try:
|
||||
apply_edit(plan)
|
||||
except NotImplementedError as exc:
|
||||
msg = str(exc)
|
||||
assert "v0.62.1" in msg
|
||||
assert "v0.61.1" not in msg
|
||||
else: # pragma: no cover - apply_edit must raise
|
||||
pytest.fail("apply_edit should have raised NotImplementedError")
|
||||
|
||||
|
||||
# ---------- M1 — steering name 128-char boundary ----------
|
||||
|
||||
|
||||
class TestSteeringNameBoundary:
|
||||
def test_at_max_length_accepted(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
name = "a" + "b" * 127 # exactly 128 chars
|
||||
assert validate_steering_name(name) == name
|
||||
|
||||
def test_one_over_max_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_name("a" + "b" * 128) # 129 chars
|
||||
|
||||
|
||||
# ---------- M3/M4 — `--base` over-cap + null-byte rejection ----------
|
||||
|
||||
|
||||
class TestSteerTrainBaseValidation:
|
||||
def _make_runner(self, tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "pairs.jsonl").write_text(
|
||||
'{"positive": "x", "negative": "y"}\n', encoding="utf-8"
|
||||
)
|
||||
return CliRunner()
|
||||
|
||||
def test_base_overcap_rejected(self, tmp_path, monkeypatch):
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = self._make_runner(tmp_path, monkeypatch)
|
||||
result = runner.invoke(app, [
|
||||
"train",
|
||||
"--base", "x" * 513,
|
||||
"--method", "caa",
|
||||
"--name", "safety-v1",
|
||||
"--pairs", "pairs.jsonl",
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "base" in result.output.lower()
|
||||
|
||||
def test_base_null_byte_rejected(self, tmp_path, monkeypatch):
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = self._make_runner(tmp_path, monkeypatch)
|
||||
result = runner.invoke(app, [
|
||||
"train",
|
||||
"--base", "bad\x00name",
|
||||
"--method", "caa",
|
||||
"--name", "safety-v1",
|
||||
"--pairs", "pairs.jsonl",
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code == 2
|
||||
assert "null" in result.output.lower() or "base" in result.output.lower()
|
||||
|
||||
def test_base_at_max_length_accepted(self, tmp_path, monkeypatch):
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = self._make_runner(tmp_path, monkeypatch)
|
||||
result = runner.invoke(app, [
|
||||
"train",
|
||||
"--base", "x" * 512,
|
||||
"--method", "caa",
|
||||
"--name", "safety-v1",
|
||||
"--pairs", "pairs.jsonl",
|
||||
"--plan-only",
|
||||
])
|
||||
assert result.exit_code == 0, (
|
||||
result.output, repr(result.exception)
|
||||
)
|
||||
|
||||
|
||||
# ---------- M5 — extract_citation_ids public-API coverage ----------
|
||||
|
||||
|
||||
class TestExtractCitationIds:
|
||||
def test_extracts_bracketed(self):
|
||||
from soup_cli.utils.citation_faithful import extract_citation_ids
|
||||
|
||||
assert extract_citation_ids("See [doc-1] and [doc-2].") == (
|
||||
"doc-1", "doc-2",
|
||||
)
|
||||
|
||||
def test_preserves_duplicates(self):
|
||||
from soup_cli.utils.citation_faithful import extract_citation_ids
|
||||
|
||||
assert extract_citation_ids("[doc-1] [doc-1]") == ("doc-1", "doc-1")
|
||||
|
||||
def test_empty_returns_empty_tuple(self):
|
||||
from soup_cli.utils.citation_faithful import extract_citation_ids
|
||||
|
||||
assert extract_citation_ids("plain text without citations") == ()
|
||||
|
||||
def test_non_string_raises(self):
|
||||
from soup_cli.utils.citation_faithful import extract_citation_ids
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
extract_citation_ids(42)
|
||||
|
||||
def test_oversize_raises(self):
|
||||
from soup_cli.utils.citation_faithful import extract_citation_ids
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
extract_citation_ids("x" * 2_000_001)
|
||||
|
||||
def test_regex_requires_leading_alnum(self):
|
||||
from soup_cli.utils.citation_faithful import extract_citation_ids
|
||||
|
||||
# `_CITATION_RE` requires `[A-Za-z0-9]` as the first char inside the
|
||||
# brackets; a leading underscore/dash should NOT match.
|
||||
assert extract_citation_ids("[_bad-id]") == ()
|
||||
assert extract_citation_ids("[-bad]") == ()
|
||||
|
||||
|
||||
# ---------- M5 supplement — score_citations expected_ids per-entry validation ----------
|
||||
|
||||
|
||||
class TestScoreCitationsExpectedIds:
|
||||
def test_non_string_in_expected_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
score_citations(predicted="[doc-1]", expected_ids=(42,))
|
||||
|
||||
def test_bool_in_expected_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
score_citations(predicted="[doc-1]", expected_ids=(True,))
|
||||
|
||||
def test_empty_string_in_expected_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
score_citations(predicted="[doc-1]", expected_ids=("",))
|
||||
|
||||
def test_null_byte_in_expected_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
score_citations(predicted="[doc-1]", expected_ids=("bad\x00id",))
|
||||
|
||||
|
||||
# ---------- L7 — CitationScore field values ----------
|
||||
|
||||
|
||||
class TestCitationScoreFields:
|
||||
def test_predicted_count_and_expected_count_set(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(
|
||||
predicted="[doc-1] [doc-2]",
|
||||
expected_ids=("doc-1",),
|
||||
)
|
||||
assert score.predicted_count == 2
|
||||
assert score.expected_count == 1
|
||||
|
||||
def test_predicted_count_includes_duplicates(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(
|
||||
predicted="[doc-1] [doc-1] [doc-1]",
|
||||
expected_ids=("doc-1",),
|
||||
)
|
||||
assert score.predicted_count == 3
|
||||
|
||||
|
||||
# ---------- M8 — GRACE codebook flag without both knobs ----------
|
||||
|
||||
|
||||
class TestGraceCodebookPartialConfig:
|
||||
def test_flag_alone_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
grace_codebook: true
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="grace_codebook"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_flag_with_dim_missing_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
grace_codebook: true
|
||||
grace_codebook_size: 128
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="grace_codebook"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_grace_in_default_edit_layer(self):
|
||||
from soup_cli.utils.knowledge_edit import _DEFAULT_EDIT_LAYER # type: ignore
|
||||
|
||||
assert "grace" in _DEFAULT_EDIT_LAYER
|
||||
assert isinstance(_DEFAULT_EDIT_LAYER["grace"], int)
|
||||
assert _DEFAULT_EDIT_LAYER["grace"] >= 0
|
||||
|
||||
def test_grace_in_edit_metadata_mapping(self):
|
||||
from soup_cli.utils.knowledge_edit import _EDIT_METHOD_METADATA # type: ignore
|
||||
|
||||
assert "grace" in _EDIT_METHOD_METADATA
|
||||
|
||||
|
||||
# ---------- M9 — validate_ra_dit_compat direct-caller defence-in-depth ----------
|
||||
|
||||
|
||||
class TestValidateRaDitCompatDirect:
|
||||
def test_null_byte_task_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_compat
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
validate_ra_dit_compat(stage="retriever", task="sft\x00")
|
||||
|
||||
def test_null_byte_stage_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_compat
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
validate_ra_dit_compat(stage="retriever\x00", task="embedding")
|
||||
|
||||
def test_bool_task_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_compat
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_ra_dit_compat(stage="retriever", task=True)
|
||||
|
||||
def test_bool_stage_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_compat
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_ra_dit_compat(stage=True, task="embedding")
|
||||
|
||||
def test_empty_task_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_compat
|
||||
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
validate_ra_dit_compat(stage="retriever", task="")
|
||||
|
||||
|
||||
# ---------- L4 — GRACE size/dim max-boundary acceptance ----------
|
||||
|
||||
|
||||
class TestGraceMaxBoundary:
|
||||
def test_max_codebook_size_accepted(self):
|
||||
from soup_cli.utils.grace_codebook import (
|
||||
MAX_CODEBOOK_SIZE,
|
||||
validate_grace_codebook_size,
|
||||
)
|
||||
|
||||
assert validate_grace_codebook_size(MAX_CODEBOOK_SIZE) == MAX_CODEBOOK_SIZE
|
||||
|
||||
def test_max_codebook_dim_accepted(self):
|
||||
from soup_cli.utils.grace_codebook import (
|
||||
MAX_CODEBOOK_DIM,
|
||||
validate_grace_codebook_dim,
|
||||
)
|
||||
|
||||
assert validate_grace_codebook_dim(MAX_CODEBOOK_DIM) == MAX_CODEBOOK_DIM
|
||||
|
||||
|
||||
# ---------- L5 — RA-DIT retriever model 512-char boundary ----------
|
||||
|
||||
|
||||
class TestRaDitRetrieverBoundary:
|
||||
def test_at_max_accepted(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
v = "x" * 512
|
||||
assert validate_ra_dit_retriever_model(v) == v
|
||||
|
||||
def test_one_over_max_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_ra_dit_retriever_model("x" * 513)
|
||||
|
||||
|
||||
# ---------- L1 — list_steers context-manager source-grep regression guard ----------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_list_steers_uses_context_manager(self):
|
||||
src = Path(__file__).resolve().parent.parent / "soup_cli" / "commands" / "steer.py"
|
||||
text = src.read_text(encoding="utf-8")
|
||||
assert "with RegistryStore() as store:" in text
|
||||
|
||||
def test_steer_uses_shared_path_helper(self):
|
||||
"""Review M1 fix: `_validate_pairs_path` delegates to the shared helper."""
|
||||
src = Path(__file__).resolve().parent.parent / "soup_cli" / "commands" / "steer.py"
|
||||
text = src.read_text(encoding="utf-8")
|
||||
assert "enforce_under_cwd_and_no_symlink" in text
|
||||
|
||||
def test_version_string_is_v0620(self):
|
||||
import soup_cli
|
||||
|
||||
assert soup_cli.__version__ == "0.62.0"
|
||||
|
||||
def test_pyproject_version_is_v0620(self):
|
||||
proj = Path(__file__).resolve().parent.parent / "pyproject.toml"
|
||||
text = proj.read_text(encoding="utf-8")
|
||||
assert 'version = "0.62.0"' in text
|
||||
|
|
@ -0,0 +1,270 @@
|
|||
"""Tests for v0.62.0 Part A — RAFT (Retrieval-Augmented Fine-Tuning) recipe.
|
||||
|
||||
Adds ``data.format='raft'`` to the schema + new ``_convert_raft`` validator
|
||||
+ a ``raft-llama3-8b`` recipe entry. Schema validates RAFT row shape:
|
||||
``{"query", "golden_doc", "distractor_docs", "answer"}``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
# ---------- _convert_raft ----------
|
||||
|
||||
|
||||
class TestConvertRaft:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
row = {
|
||||
"query": "What year was Python released?",
|
||||
"golden_doc": "Python was released in 1991 by Guido van Rossum.",
|
||||
"distractor_docs": [
|
||||
"Ruby was released in 1995.",
|
||||
"Java was released in 1995.",
|
||||
],
|
||||
"answer": "1991",
|
||||
}
|
||||
out = _convert_raft(row)
|
||||
assert out["query"] == row["query"]
|
||||
assert out["golden_doc"] == row["golden_doc"]
|
||||
assert out["distractor_docs"] == row["distractor_docs"]
|
||||
assert out["answer"] == row["answer"]
|
||||
|
||||
def test_empty_distractors_ok(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
out = _convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
assert out["distractor_docs"] == []
|
||||
|
||||
def test_missing_query_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises((KeyError, ValueError)):
|
||||
_convert_raft({
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_missing_golden_doc_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises((KeyError, ValueError)):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_missing_answer_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises((KeyError, ValueError)):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
})
|
||||
|
||||
def test_empty_query_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="query"):
|
||||
_convert_raft({
|
||||
"query": "",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_empty_golden_doc_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="golden_doc"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_empty_answer_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="answer"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "",
|
||||
})
|
||||
|
||||
def test_non_string_query_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="query"):
|
||||
_convert_raft({
|
||||
"query": 123,
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_non_list_distractors_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="distractor_docs"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": "not-a-list",
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_non_string_distractor_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="distractor"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [123],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_null_byte_query_rejected(self):
|
||||
from soup_cli.data.formats import _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
_convert_raft({
|
||||
"query": "bad\x00",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
def test_distractor_cap(self):
|
||||
from soup_cli.data.formats import _MAX_RAFT_DISTRACTORS, _convert_raft
|
||||
|
||||
# exactly cap accepted
|
||||
out = _convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": ["d"] * _MAX_RAFT_DISTRACTORS,
|
||||
"answer": "a",
|
||||
})
|
||||
assert len(out["distractor_docs"]) == _MAX_RAFT_DISTRACTORS
|
||||
|
||||
def test_distractor_overcap_rejected(self):
|
||||
from soup_cli.data.formats import _MAX_RAFT_DISTRACTORS, _convert_raft
|
||||
|
||||
with pytest.raises(ValueError, match="distractor"):
|
||||
_convert_raft({
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": ["d"] * (_MAX_RAFT_DISTRACTORS + 1),
|
||||
"answer": "a",
|
||||
})
|
||||
|
||||
|
||||
# ---------- format dispatcher ----------
|
||||
|
||||
|
||||
class TestFormatDispatcher:
|
||||
def test_format_to_messages_dispatches_raft(self):
|
||||
from soup_cli.data.formats import format_to_messages
|
||||
|
||||
row = {
|
||||
"query": "q",
|
||||
"golden_doc": "g",
|
||||
"distractor_docs": [],
|
||||
"answer": "a",
|
||||
}
|
||||
out = format_to_messages(row, "raft")
|
||||
assert out is not None
|
||||
assert out["query"] == "q"
|
||||
assert out["answer"] == "a"
|
||||
|
||||
def test_format_to_messages_invalid_raft_returns_none(self):
|
||||
from soup_cli.data.formats import format_to_messages
|
||||
|
||||
# missing key — wrapper catches and returns None
|
||||
assert format_to_messages({"query": "q"}, "raft") is None
|
||||
|
||||
|
||||
# ---------- schema integration ----------
|
||||
|
||||
|
||||
class TestSchemaIntegration:
|
||||
def test_data_format_raft_accepted(self):
|
||||
from soup_cli.config.schema import DataConfig
|
||||
|
||||
cfg = DataConfig(train="data.jsonl", format="raft")
|
||||
assert cfg.format == "raft"
|
||||
|
||||
def test_soup_config_raft_roundtrip(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
max_length: 4096
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
cfg = load_config_from_string(yaml_text)
|
||||
assert cfg.data.format == "raft"
|
||||
assert cfg.task == "sft"
|
||||
|
||||
|
||||
# ---------- recipe catalog ----------
|
||||
|
||||
|
||||
class TestRaftRecipe:
|
||||
def test_recipe_present(self):
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
recipe = get_recipe("raft-llama3-8b")
|
||||
assert recipe is not None
|
||||
assert recipe.task == "sft"
|
||||
assert "raft" in recipe.tags or "rag" in recipe.tags
|
||||
|
||||
def test_recipe_yaml_loads(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
recipe = get_recipe("raft-llama3-8b")
|
||||
assert recipe is not None
|
||||
cfg = load_config_from_string(recipe.yaml_str)
|
||||
assert cfg.data.format == "raft"
|
||||
|
||||
def test_recipe_yaml_parses_as_dict(self):
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
recipe = get_recipe("raft-llama3-8b")
|
||||
assert recipe is not None
|
||||
parsed = yaml.safe_load(recipe.yaml_str)
|
||||
assert isinstance(parsed, dict)
|
||||
assert parsed["data"]["format"] == "raft"
|
||||
|
||||
def test_recipe_search_finds_raft(self):
|
||||
from soup_cli.recipes.catalog import search_recipes
|
||||
|
||||
results = search_recipes(query="raft")
|
||||
assert any(r.task == "sft" and "raft" in r.yaml_str for r in results)
|
||||
|
|
@ -0,0 +1,384 @@
|
|||
"""Tests for v0.62.0 Part B — RA-DIT (Retrieval-Augmented Dual Instruction Tuning).
|
||||
|
||||
Two-stage Meta 2023 recipe: train the retriever first (contrastive), then
|
||||
the generator (RAFT-style). v0.62.0 ships the schema + recipe entries;
|
||||
live runtime composes existing v0.16 embedding trainer + Part A RAFT.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
# ---------- Module surface ----------
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.ra_dit import (
|
||||
SUPPORTED_RA_DIT_STAGES,
|
||||
RaDitStageSpec,
|
||||
get_ra_dit_stage_spec,
|
||||
validate_ra_dit_stage,
|
||||
)
|
||||
assert callable(validate_ra_dit_stage)
|
||||
assert callable(get_ra_dit_stage_spec)
|
||||
assert dataclasses.is_dataclass(RaDitStageSpec)
|
||||
assert isinstance(SUPPORTED_RA_DIT_STAGES, frozenset)
|
||||
|
||||
def test_stages_exact(self):
|
||||
from soup_cli.utils.ra_dit import SUPPORTED_RA_DIT_STAGES
|
||||
|
||||
assert SUPPORTED_RA_DIT_STAGES == frozenset({"retriever", "generator"})
|
||||
|
||||
def test_metadata_mapping_proxy(self):
|
||||
from types import MappingProxyType
|
||||
|
||||
from soup_cli.utils.ra_dit import _RA_DIT_STAGE_METADATA # type: ignore
|
||||
|
||||
assert isinstance(_RA_DIT_STAGE_METADATA, MappingProxyType)
|
||||
|
||||
|
||||
# ---------- validate_ra_dit_stage ----------
|
||||
|
||||
|
||||
class TestValidateStage:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
assert validate_ra_dit_stage("retriever") == "retriever"
|
||||
assert validate_ra_dit_stage("generator") == "generator"
|
||||
|
||||
def test_case_insensitive(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
assert validate_ra_dit_stage("Retriever") == "retriever"
|
||||
assert validate_ra_dit_stage("GENERATOR") == "generator"
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_ra_dit_stage(True)
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_ra_dit_stage(1)
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_ra_dit_stage("")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_ra_dit_stage("retriever\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_ra_dit_stage("retriever" * 100)
|
||||
|
||||
def test_unknown_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_stage
|
||||
|
||||
with pytest.raises(ValueError, match="ra_dit_stage"):
|
||||
validate_ra_dit_stage("decoder")
|
||||
|
||||
|
||||
# ---------- get_ra_dit_stage_spec ----------
|
||||
|
||||
|
||||
class TestStageSpec:
|
||||
def test_retriever_spec(self):
|
||||
from soup_cli.utils.ra_dit import get_ra_dit_stage_spec
|
||||
|
||||
spec = get_ra_dit_stage_spec("retriever")
|
||||
assert spec.name == "retriever"
|
||||
assert spec.base_task == "embedding"
|
||||
assert spec.live_wired is False
|
||||
|
||||
def test_generator_spec(self):
|
||||
from soup_cli.utils.ra_dit import get_ra_dit_stage_spec
|
||||
|
||||
spec = get_ra_dit_stage_spec("generator")
|
||||
assert spec.name == "generator"
|
||||
assert spec.base_task == "sft"
|
||||
assert spec.live_wired is False
|
||||
|
||||
def test_unknown_raises(self):
|
||||
from soup_cli.utils.ra_dit import get_ra_dit_stage_spec
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
get_ra_dit_stage_spec("nonsense")
|
||||
|
||||
def test_spec_frozen(self):
|
||||
from soup_cli.utils.ra_dit import get_ra_dit_stage_spec
|
||||
|
||||
spec = get_ra_dit_stage_spec("retriever")
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
spec.name = "mutated" # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------- validate_ra_dit_retriever_model ----------
|
||||
|
||||
|
||||
class TestRetrieverModel:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
assert validate_ra_dit_retriever_model("sentence-transformers/all-mpnet-base-v2") == \
|
||||
"sentence-transformers/all-mpnet-base-v2"
|
||||
|
||||
def test_none_passthrough(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
assert validate_ra_dit_retriever_model(None) is None
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_ra_dit_retriever_model("")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_ra_dit_retriever_model("foo\x00bar")
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_ra_dit_retriever_model(True)
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.ra_dit import validate_ra_dit_retriever_model
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_ra_dit_retriever_model("a" * 1024)
|
||||
|
||||
|
||||
# ---------- Schema integration ----------
|
||||
|
||||
|
||||
class TestSchemaIntegration:
|
||||
def test_default_none(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig()
|
||||
assert cfg.ra_dit_stage is None
|
||||
assert cfg.ra_dit_retriever_model is None
|
||||
|
||||
def test_retriever_accepted(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(ra_dit_stage="retriever")
|
||||
assert cfg.ra_dit_stage == "retriever"
|
||||
|
||||
def test_generator_accepted(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(ra_dit_stage="generator")
|
||||
assert cfg.ra_dit_stage == "generator"
|
||||
|
||||
def test_case_insensitive_at_schema(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(ra_dit_stage="Generator")
|
||||
assert cfg.ra_dit_stage == "generator"
|
||||
|
||||
def test_invalid_stage_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(ra_dit_stage="decoder")
|
||||
|
||||
def test_retriever_model_accepted(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(
|
||||
ra_dit_stage="retriever",
|
||||
ra_dit_retriever_model="sentence-transformers/all-mpnet-base-v2",
|
||||
)
|
||||
assert cfg.ra_dit_retriever_model == "sentence-transformers/all-mpnet-base-v2"
|
||||
|
||||
def test_retriever_model_null_byte_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(ra_dit_retriever_model="foo\x00")
|
||||
|
||||
|
||||
# ---------- Cross-validator (SoupConfig) ----------
|
||||
|
||||
|
||||
class TestSoupConfigGate:
|
||||
def test_retriever_stage_on_embedding_task(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: sentence-transformers/all-mpnet-base-v2
|
||||
task: embedding
|
||||
|
||||
data:
|
||||
train: ./data/anchor.jsonl
|
||||
format: embedding
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-5
|
||||
batch_size: auto
|
||||
ra_dit_stage: retriever
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
cfg = load_config_from_string(yaml_text)
|
||||
assert cfg.training.ra_dit_stage == "retriever"
|
||||
|
||||
def test_generator_stage_on_sft_task(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
ra_dit_stage: generator
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
cfg = load_config_from_string(yaml_text)
|
||||
assert cfg.training.ra_dit_stage == "generator"
|
||||
|
||||
def test_retriever_on_sft_task_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
# retriever stage requires embedding-family task
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
ra_dit_stage: retriever
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="ra_dit_stage"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_generator_on_grpo_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: grpo
|
||||
|
||||
data:
|
||||
train: ./data/prompts.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
ra_dit_stage: generator
|
||||
reward_fn: accuracy
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="ra_dit_stage"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_retriever_model_without_stage_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
ra_dit_retriever_model: sentence-transformers/all-mpnet-base-v2
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="ra_dit"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
|
||||
# ---------- Recipes ----------
|
||||
|
||||
|
||||
class TestRaDitRecipes:
|
||||
def test_retriever_recipe_present(self):
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
recipe = get_recipe("ra-dit-retriever")
|
||||
assert recipe is not None
|
||||
assert recipe.task == "embedding"
|
||||
|
||||
def test_generator_recipe_present(self):
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
recipe = get_recipe("ra-dit-llama3-8b")
|
||||
assert recipe is not None
|
||||
assert recipe.task == "sft"
|
||||
|
||||
def test_retriever_recipe_yaml_loads(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
recipe = get_recipe("ra-dit-retriever")
|
||||
assert recipe is not None
|
||||
cfg = load_config_from_string(recipe.yaml_str)
|
||||
assert cfg.training.ra_dit_stage == "retriever"
|
||||
|
||||
def test_generator_recipe_yaml_loads(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
recipe = get_recipe("ra-dit-llama3-8b")
|
||||
assert recipe is not None
|
||||
cfg = load_config_from_string(recipe.yaml_str)
|
||||
assert cfg.training.ra_dit_stage == "generator"
|
||||
assert cfg.data.format == "raft"
|
||||
|
||||
def test_recipe_yaml_parses_as_dict(self):
|
||||
from soup_cli.recipes.catalog import get_recipe
|
||||
|
||||
for name in ("ra-dit-retriever", "ra-dit-llama3-8b"):
|
||||
recipe = get_recipe(name)
|
||||
assert recipe is not None
|
||||
parsed = yaml.safe_load(recipe.yaml_str)
|
||||
assert isinstance(parsed, dict)
|
||||
|
|
@ -0,0 +1,381 @@
|
|||
"""Tests for v0.62.0 Part C — `soup steer` (CAA / ITI / RepE control vectors).
|
||||
|
||||
Schema-only release: validators + frozen dataclass + CLI surface ship now,
|
||||
live forward-hook + decode-time intervention land in v0.62.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
# ---------- Module surface ----------
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.steering import (
|
||||
SUPPORTED_STEERING_METHODS,
|
||||
SteeringMethodSpec,
|
||||
apply_steering,
|
||||
build_steering_vector,
|
||||
get_steering_method_spec,
|
||||
validate_steering_method,
|
||||
)
|
||||
assert callable(validate_steering_method)
|
||||
assert callable(get_steering_method_spec)
|
||||
assert callable(apply_steering)
|
||||
assert callable(build_steering_vector)
|
||||
assert dataclasses.is_dataclass(SteeringMethodSpec)
|
||||
assert isinstance(SUPPORTED_STEERING_METHODS, frozenset)
|
||||
|
||||
def test_methods_exact(self):
|
||||
from soup_cli.utils.steering import SUPPORTED_STEERING_METHODS
|
||||
|
||||
assert SUPPORTED_STEERING_METHODS == frozenset({"caa", "iti", "repe"})
|
||||
|
||||
def test_metadata_mapping_proxy(self):
|
||||
from types import MappingProxyType
|
||||
|
||||
from soup_cli.utils.steering import _STEERING_METHOD_METADATA # type: ignore
|
||||
|
||||
assert isinstance(_STEERING_METHOD_METADATA, MappingProxyType)
|
||||
|
||||
|
||||
# ---------- validate_steering_method ----------
|
||||
|
||||
|
||||
class TestValidateMethod:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
for name in ("caa", "iti", "repe"):
|
||||
assert validate_steering_method(name) == name
|
||||
|
||||
def test_case_insensitive(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
assert validate_steering_method("CAA") == "caa"
|
||||
assert validate_steering_method("ItI") == "iti"
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_steering_method(True)
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_steering_method(42)
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_method("")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_method("caa\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_method("c" * 100)
|
||||
|
||||
def test_unknown_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_method
|
||||
|
||||
with pytest.raises(ValueError, match="steering"):
|
||||
validate_steering_method("nonsense")
|
||||
|
||||
|
||||
# ---------- validate_steering_name ----------
|
||||
|
||||
|
||||
class TestValidateName:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
for name in ("safety-v1", "helpfulness_2024", "tone-formal"):
|
||||
assert validate_steering_name(name) == name
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_steering_name(True)
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_steering_name(1)
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_name("")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_name("safety\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_name("x" * 200)
|
||||
|
||||
def test_invalid_chars_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_name
|
||||
|
||||
# Path separators / spaces / shell metacharacters rejected so
|
||||
# the name can be safely embedded in CLI args and filenames.
|
||||
for bad in ("foo/bar", "foo bar", "foo;rm", "foo$x", "../escape"):
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_name(bad)
|
||||
|
||||
|
||||
# ---------- validate_steering_strength ----------
|
||||
|
||||
|
||||
class TestValidateStrength:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.steering import validate_steering_strength
|
||||
|
||||
assert validate_steering_strength(0.5) == 0.5
|
||||
assert validate_steering_strength(-1.0) == -1.0
|
||||
assert validate_steering_strength(0.0) == 0.0
|
||||
|
||||
def test_int_coerced(self):
|
||||
from soup_cli.utils.steering import validate_steering_strength
|
||||
|
||||
assert validate_steering_strength(1) == 1.0
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_strength
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_steering_strength(True)
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
import math
|
||||
|
||||
from soup_cli.utils.steering import validate_steering_strength
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_strength(math.nan)
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_strength(math.inf)
|
||||
|
||||
def test_out_of_bounds_rejected(self):
|
||||
from soup_cli.utils.steering import validate_steering_strength
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_strength(100.0)
|
||||
with pytest.raises(ValueError):
|
||||
validate_steering_strength(-100.0)
|
||||
|
||||
|
||||
# ---------- get_steering_method_spec ----------
|
||||
|
||||
|
||||
class TestSpec:
|
||||
def test_caa_spec(self):
|
||||
from soup_cli.utils.steering import get_steering_method_spec
|
||||
|
||||
spec = get_steering_method_spec("caa")
|
||||
assert spec.name == "caa"
|
||||
assert spec.live_wired is False
|
||||
assert spec.needs_contrastive_pairs is True
|
||||
|
||||
def test_iti_spec(self):
|
||||
from soup_cli.utils.steering import get_steering_method_spec
|
||||
|
||||
spec = get_steering_method_spec("iti")
|
||||
assert spec.name == "iti"
|
||||
assert spec.needs_attention_heads is True
|
||||
|
||||
def test_repe_spec(self):
|
||||
from soup_cli.utils.steering import get_steering_method_spec
|
||||
|
||||
spec = get_steering_method_spec("repe")
|
||||
assert spec.name == "repe"
|
||||
|
||||
def test_unknown_raises(self):
|
||||
from soup_cli.utils.steering import get_steering_method_spec
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
get_steering_method_spec("nonsense")
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.steering import get_steering_method_spec
|
||||
|
||||
spec = get_steering_method_spec("caa")
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
spec.name = "mutated" # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------- Deferred-live stubs ----------
|
||||
|
||||
|
||||
class TestDeferredStubs:
|
||||
def test_apply_steering_deferred(self):
|
||||
from soup_cli.utils.steering import apply_steering
|
||||
|
||||
with pytest.raises(NotImplementedError, match="v0.62.1"):
|
||||
apply_steering("caa")
|
||||
|
||||
def test_apply_steering_validates_first(self):
|
||||
from soup_cli.utils.steering import apply_steering
|
||||
|
||||
# Unknown method rejected BEFORE the deferred-live raise.
|
||||
with pytest.raises(ValueError):
|
||||
apply_steering("nonsense")
|
||||
|
||||
def test_build_steering_vector_deferred(self):
|
||||
from soup_cli.utils.steering import build_steering_vector
|
||||
|
||||
with pytest.raises(NotImplementedError, match="v0.62.1"):
|
||||
build_steering_vector(method="caa", name="safety-v1")
|
||||
|
||||
def test_build_steering_vector_validates_method_first(self):
|
||||
from soup_cli.utils.steering import build_steering_vector
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_steering_vector(method="nonsense", name="safety-v1")
|
||||
|
||||
def test_build_steering_vector_validates_name_first(self):
|
||||
from soup_cli.utils.steering import build_steering_vector
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_steering_vector(method="caa", name="bad/path")
|
||||
|
||||
|
||||
# ---------- Registry artifact kind ----------
|
||||
|
||||
|
||||
class TestRegistryArtifactKind:
|
||||
def test_steering_vector_in_valid_kinds(self):
|
||||
from soup_cli.registry.store import _VALID_KINDS
|
||||
|
||||
assert "steering_vector" in _VALID_KINDS
|
||||
|
||||
|
||||
# ---------- CLI plumbing ----------
|
||||
|
||||
|
||||
class TestCLI:
|
||||
def test_cli_help_lists_steer(self):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "steer" in result.stdout.lower()
|
||||
|
||||
def test_steer_help(self):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_steer_train_help(self):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["train", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_steer_apply_help(self):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["apply", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_steer_list_help(self):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["list", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_steer_train_unknown_method_rejected(self):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"train",
|
||||
"--base", "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"--method", "nonsense",
|
||||
"--name", "safety-v1",
|
||||
"--pairs", "./data/pairs.jsonl",
|
||||
"--plan-only",
|
||||
])
|
||||
# CLI either exits 2 (validation) or 1 (plan rejected); not 0.
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_steer_train_plan_only(self, tmp_path, monkeypatch):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.steer import app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
# Create a tiny pairs JSONL to pass the path-containment check.
|
||||
pairs = tmp_path / "pairs.jsonl"
|
||||
pairs.write_text(
|
||||
'{"positive": "be safe", "negative": "be harmful"}\n', encoding="utf-8"
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, [
|
||||
"train",
|
||||
"--base", "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"--method", "caa",
|
||||
"--name", "safety-v1",
|
||||
"--pairs", "pairs.jsonl",
|
||||
"--plan-only",
|
||||
])
|
||||
# plan-only succeeds with friendly deferred-live panel.
|
||||
assert result.exit_code == 0
|
||||
assert "v0.62" in result.stdout or "caa" in result.stdout.lower()
|
||||
|
||||
|
||||
# ---------- soup serve --steer flag plumbing ----------
|
||||
|
||||
|
||||
class TestServeSteerFlag:
|
||||
def test_serve_help_mentions_steer(self):
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["serve", "--help"])
|
||||
# Even if the flag is not yet alive, --help should expose it so a
|
||||
# YAML / shell pipeline can reference it.
|
||||
assert result.exit_code == 0
|
||||
assert "steer" in result.stdout.lower()
|
||||
|
|
@ -0,0 +1,380 @@
|
|||
"""Tests for v0.62.0 Part D — Citation-faithful FT.
|
||||
|
||||
Schema-only release: `training.citation_faithful: bool` opt-in trains the
|
||||
model to cite document IDs verbatim from the training corpus. Composes
|
||||
with v0.62.0 Part A (RAFT) — the RAFT data already contains the doc
|
||||
references; this flag adds a citation-precision / recall scorer to the
|
||||
eval suite + a loss-mask rule that emphasises citation spans.
|
||||
|
||||
Live citation-span loss-mask + eval scorer land in v0.62.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
# ---------- Module surface ----------
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.citation_faithful import (
|
||||
SUPPORTED_CITATION_STYLES,
|
||||
CitationScore,
|
||||
score_citations,
|
||||
validate_citation_style,
|
||||
validate_citation_threshold,
|
||||
)
|
||||
assert callable(validate_citation_style)
|
||||
assert callable(validate_citation_threshold)
|
||||
assert callable(score_citations)
|
||||
assert dataclasses.is_dataclass(CitationScore)
|
||||
assert isinstance(SUPPORTED_CITATION_STYLES, frozenset)
|
||||
|
||||
def test_styles_exact(self):
|
||||
from soup_cli.utils.citation_faithful import SUPPORTED_CITATION_STYLES
|
||||
|
||||
assert SUPPORTED_CITATION_STYLES == frozenset(
|
||||
{"bracket", "inline", "footnote"}
|
||||
)
|
||||
|
||||
|
||||
# ---------- validate_citation_style ----------
|
||||
|
||||
|
||||
class TestValidateStyle:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
for name in ("bracket", "inline", "footnote"):
|
||||
assert validate_citation_style(name) == name
|
||||
|
||||
def test_case_insensitive(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
assert validate_citation_style("BRACKET") == "bracket"
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_citation_style(True)
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_citation_style(1)
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_citation_style("")
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_citation_style("bracket\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_citation_style("x" * 64)
|
||||
|
||||
def test_unknown_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_style
|
||||
|
||||
with pytest.raises(ValueError, match="citation"):
|
||||
validate_citation_style("apa-7th")
|
||||
|
||||
|
||||
# ---------- validate_citation_threshold ----------
|
||||
|
||||
|
||||
class TestValidateThreshold:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_threshold
|
||||
|
||||
assert validate_citation_threshold(0.5) == 0.5
|
||||
assert validate_citation_threshold(0.0) == 0.0
|
||||
assert validate_citation_threshold(1.0) == 1.0
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_threshold
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_citation_threshold(True)
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
import math
|
||||
|
||||
from soup_cli.utils.citation_faithful import validate_citation_threshold
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_citation_threshold(math.nan)
|
||||
with pytest.raises(ValueError):
|
||||
validate_citation_threshold(math.inf)
|
||||
|
||||
def test_out_of_range_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import validate_citation_threshold
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_citation_threshold(-0.1)
|
||||
with pytest.raises(ValueError):
|
||||
validate_citation_threshold(1.1)
|
||||
|
||||
|
||||
# ---------- score_citations ----------
|
||||
|
||||
|
||||
class TestScoreCitations:
|
||||
def test_perfect_match(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(
|
||||
predicted="The capital is Paris [doc-1].",
|
||||
expected_ids=("doc-1",),
|
||||
)
|
||||
assert score.precision == 1.0
|
||||
assert score.recall == 1.0
|
||||
assert score.f1 == 1.0
|
||||
|
||||
def test_no_citations(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(
|
||||
predicted="The capital is Paris.",
|
||||
expected_ids=("doc-1",),
|
||||
)
|
||||
# No predicted citations: precision is undefined (set to 0.0 by
|
||||
# convention), recall is 0.0.
|
||||
assert score.precision == 0.0
|
||||
assert score.recall == 0.0
|
||||
assert score.f1 == 0.0
|
||||
|
||||
def test_extra_citation_lowers_precision(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(
|
||||
predicted="See [doc-1] and [doc-2].",
|
||||
expected_ids=("doc-1",),
|
||||
)
|
||||
# Recall = 1/1 = 1.0, Precision = 1/2 = 0.5.
|
||||
assert score.recall == 1.0
|
||||
assert score.precision == 0.5
|
||||
assert 0.5 < score.f1 < 1.0
|
||||
|
||||
def test_missing_citation_lowers_recall(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(
|
||||
predicted="See [doc-1].",
|
||||
expected_ids=("doc-1", "doc-2"),
|
||||
)
|
||||
assert score.precision == 1.0
|
||||
assert score.recall == 0.5
|
||||
|
||||
def test_empty_expected_ids(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
# No expected citations: recall is undefined but score returns 0
|
||||
# by convention (avoids div-by-zero). Precision still defined.
|
||||
score = score_citations(
|
||||
predicted="The capital is Paris [doc-1].",
|
||||
expected_ids=(),
|
||||
)
|
||||
assert score.recall == 0.0
|
||||
# Predicted IDs are not in the empty expected set, so precision
|
||||
# is 0.0 too.
|
||||
assert score.precision == 0.0
|
||||
|
||||
def test_bool_predicted_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
score_citations(predicted=True, expected_ids=("doc-1",))
|
||||
|
||||
def test_non_string_predicted_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
score_citations(predicted=123, expected_ids=("doc-1",))
|
||||
|
||||
def test_oversize_predicted_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
score_citations(
|
||||
predicted="x" * 2_000_001,
|
||||
expected_ids=("doc-1",),
|
||||
)
|
||||
|
||||
def test_non_iterable_expected_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
score_citations(predicted="text", expected_ids=42)
|
||||
|
||||
def test_too_many_expected_rejected(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
score_citations(
|
||||
predicted="text",
|
||||
expected_ids=tuple(f"doc-{i}" for i in range(10_001)),
|
||||
)
|
||||
|
||||
def test_score_frozen(self):
|
||||
from soup_cli.utils.citation_faithful import score_citations
|
||||
|
||||
score = score_citations(predicted="[doc-1]", expected_ids=("doc-1",))
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
score.f1 = 0.0 # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------- Schema integration ----------
|
||||
|
||||
|
||||
class TestSchemaIntegration:
|
||||
def test_default_off(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig()
|
||||
assert cfg.citation_faithful is False
|
||||
assert cfg.citation_style is None
|
||||
assert cfg.citation_recall_threshold is None
|
||||
|
||||
def test_opt_in_accepts(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(
|
||||
citation_faithful=True,
|
||||
citation_style="bracket",
|
||||
citation_recall_threshold=0.8,
|
||||
)
|
||||
assert cfg.citation_faithful is True
|
||||
assert cfg.citation_style == "bracket"
|
||||
assert cfg.citation_recall_threshold == 0.8
|
||||
|
||||
def test_style_case_insensitive(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(citation_style="INLINE")
|
||||
assert cfg.citation_style == "inline"
|
||||
|
||||
def test_unknown_style_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(citation_style="apa")
|
||||
|
||||
def test_threshold_bounds(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(citation_recall_threshold=-0.1)
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(citation_recall_threshold=1.5)
|
||||
|
||||
|
||||
# ---------- Cross-validator ----------
|
||||
|
||||
|
||||
class TestSoupConfigCrossValidator:
|
||||
def test_citation_faithful_with_raft_format(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
citation_faithful: true
|
||||
citation_style: bracket
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
cfg = load_config_from_string(yaml_text)
|
||||
assert cfg.training.citation_faithful is True
|
||||
|
||||
def test_citation_faithful_without_raft_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: alpaca
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
citation_faithful: true
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="citation_faithful"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_citation_style_without_faithful_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
citation_style: bracket
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="citation"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_threshold_without_faithful_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/raft.jsonl
|
||||
format: raft
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
citation_recall_threshold: 0.8
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="citation"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
|
@ -0,0 +1,334 @@
|
|||
"""Tests for v0.62.0 Part E — GRACE codebook (long-running edit).
|
||||
|
||||
Discrete latent-space codebook for thousands of sequential edits that
|
||||
survives lifelong deployments without norm-blowup. Extends the v0.61.0
|
||||
edit method allowlist with ``grace`` and ships the codebook config
|
||||
schema + validators.
|
||||
|
||||
Schema-only release: live codebook lookup + write kernel land in v0.62.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
# ---------- Module surface ----------
|
||||
|
||||
|
||||
class TestModuleSurface:
|
||||
def test_imports(self):
|
||||
from soup_cli.utils.grace_codebook import (
|
||||
MAX_CODEBOOK_DIM,
|
||||
MAX_CODEBOOK_SIZE,
|
||||
GraceCodebookConfig,
|
||||
apply_grace_codebook,
|
||||
build_grace_codebook_config,
|
||||
validate_grace_codebook_dim,
|
||||
validate_grace_codebook_size,
|
||||
)
|
||||
assert callable(validate_grace_codebook_size)
|
||||
assert callable(validate_grace_codebook_dim)
|
||||
assert callable(apply_grace_codebook)
|
||||
assert callable(build_grace_codebook_config)
|
||||
assert dataclasses.is_dataclass(GraceCodebookConfig)
|
||||
assert isinstance(MAX_CODEBOOK_SIZE, int)
|
||||
assert isinstance(MAX_CODEBOOK_DIM, int)
|
||||
|
||||
|
||||
# ---------- validate_grace_codebook_size ----------
|
||||
|
||||
|
||||
class TestValidateSize:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_size
|
||||
|
||||
for n in (1, 100, 10_000):
|
||||
assert validate_grace_codebook_size(n) == n
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_size
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_grace_codebook_size(True)
|
||||
|
||||
def test_non_int_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_size
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_grace_codebook_size(3.14)
|
||||
|
||||
def test_zero_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_size
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_grace_codebook_size(0)
|
||||
|
||||
def test_negative_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_size
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_grace_codebook_size(-1)
|
||||
|
||||
def test_overcap_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import (
|
||||
MAX_CODEBOOK_SIZE,
|
||||
validate_grace_codebook_size,
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_grace_codebook_size(MAX_CODEBOOK_SIZE + 1)
|
||||
|
||||
|
||||
# ---------- validate_grace_codebook_dim ----------
|
||||
|
||||
|
||||
class TestValidateDim:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_dim
|
||||
|
||||
for d in (8, 768, 4096):
|
||||
assert validate_grace_codebook_dim(d) == d
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_dim
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_grace_codebook_dim(True)
|
||||
|
||||
def test_zero_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import validate_grace_codebook_dim
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_grace_codebook_dim(0)
|
||||
|
||||
def test_overcap_rejected(self):
|
||||
from soup_cli.utils.grace_codebook import (
|
||||
MAX_CODEBOOK_DIM,
|
||||
validate_grace_codebook_dim,
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_grace_codebook_dim(MAX_CODEBOOK_DIM + 1)
|
||||
|
||||
|
||||
# ---------- GraceCodebookConfig ----------
|
||||
|
||||
|
||||
class TestCodebookConfig:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.grace_codebook import build_grace_codebook_config
|
||||
|
||||
cfg = build_grace_codebook_config(size=128, dim=768)
|
||||
assert cfg.size == 128
|
||||
assert cfg.dim == 768
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.grace_codebook import build_grace_codebook_config
|
||||
|
||||
cfg = build_grace_codebook_config(size=128, dim=768)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
cfg.size = 256 # type: ignore[misc]
|
||||
|
||||
def test_invalid_size_propagates(self):
|
||||
from soup_cli.utils.grace_codebook import build_grace_codebook_config
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_grace_codebook_config(size=0, dim=768)
|
||||
|
||||
def test_invalid_dim_propagates(self):
|
||||
from soup_cli.utils.grace_codebook import build_grace_codebook_config
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_grace_codebook_config(size=128, dim=0)
|
||||
|
||||
|
||||
# ---------- apply_grace_codebook ----------
|
||||
|
||||
|
||||
class TestApplyDeferred:
|
||||
def test_deferred(self):
|
||||
from soup_cli.utils.grace_codebook import (
|
||||
apply_grace_codebook,
|
||||
build_grace_codebook_config,
|
||||
)
|
||||
|
||||
cfg = build_grace_codebook_config(size=128, dim=768)
|
||||
with pytest.raises(NotImplementedError, match="v0.62.1"):
|
||||
apply_grace_codebook(cfg)
|
||||
|
||||
def test_apply_validates_config_type(self):
|
||||
from soup_cli.utils.grace_codebook import apply_grace_codebook
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
apply_grace_codebook("not-a-config")
|
||||
|
||||
|
||||
# ---------- knowledge_edit allowlist extension ----------
|
||||
|
||||
|
||||
class TestEditMethodAllowlist:
|
||||
def test_grace_in_supported_methods(self):
|
||||
from soup_cli.utils.knowledge_edit import SUPPORTED_EDIT_METHODS
|
||||
|
||||
assert "grace" in SUPPORTED_EDIT_METHODS
|
||||
|
||||
def test_grace_method_validates(self):
|
||||
from soup_cli.utils.knowledge_edit import validate_edit_method
|
||||
|
||||
assert validate_edit_method("grace") == "grace"
|
||||
assert validate_edit_method("GRACE") == "grace"
|
||||
|
||||
def test_grace_spec_present(self):
|
||||
from soup_cli.utils.knowledge_edit import get_edit_method_spec
|
||||
|
||||
spec = get_edit_method_spec("grace")
|
||||
assert spec.name == "grace"
|
||||
assert spec.multi_edit_capable is True
|
||||
assert spec.live_wired is False
|
||||
|
||||
def test_grace_edit_plan_happy(self):
|
||||
from soup_cli.utils.knowledge_edit import build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="meta-llama/Llama-3.1-8B-Instruct",
|
||||
method="grace",
|
||||
subject="The capital of France is",
|
||||
target="Lyon",
|
||||
)
|
||||
assert plan.method == "grace"
|
||||
assert plan.layer >= 0
|
||||
|
||||
def test_apply_grace_via_edit_path_deferred(self):
|
||||
from soup_cli.utils.knowledge_edit import apply_edit, build_edit_plan
|
||||
|
||||
plan = build_edit_plan(
|
||||
base="meta-llama/Llama-3.1-8B-Instruct",
|
||||
method="grace",
|
||||
subject="The capital of France is",
|
||||
target="Lyon",
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.62.1"):
|
||||
apply_edit(plan)
|
||||
|
||||
|
||||
# ---------- Schema integration ----------
|
||||
|
||||
|
||||
class TestSchemaIntegration:
|
||||
def test_default_none(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig()
|
||||
assert cfg.grace_codebook is False
|
||||
assert cfg.grace_codebook_size is None
|
||||
assert cfg.grace_codebook_dim is None
|
||||
|
||||
def test_opt_in_accepts(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
cfg = TrainingConfig(
|
||||
grace_codebook=True,
|
||||
grace_codebook_size=128,
|
||||
grace_codebook_dim=768,
|
||||
)
|
||||
assert cfg.grace_codebook is True
|
||||
assert cfg.grace_codebook_size == 128
|
||||
assert cfg.grace_codebook_dim == 768
|
||||
|
||||
def test_invalid_size_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(
|
||||
grace_codebook=True,
|
||||
grace_codebook_size=0,
|
||||
grace_codebook_dim=768,
|
||||
)
|
||||
|
||||
def test_invalid_dim_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(
|
||||
grace_codebook=True,
|
||||
grace_codebook_size=128,
|
||||
grace_codebook_dim=-1,
|
||||
)
|
||||
|
||||
|
||||
# ---------- SoupConfig cross-validator ----------
|
||||
|
||||
|
||||
class TestSoupConfigCrossValidator:
|
||||
def test_grace_codebook_size_without_flag_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
grace_codebook_size: 128
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="grace_codebook"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_grace_codebook_requires_both_size_and_dim(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
# codebook flag but missing size — rejected.
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
grace_codebook: true
|
||||
grace_codebook_dim: 768
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
with pytest.raises(Exception, match="grace_codebook"):
|
||||
load_config_from_string(yaml_text)
|
||||
|
||||
def test_grace_codebook_happy(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml_text = """\
|
||||
base: meta-llama/Llama-3.1-8B-Instruct
|
||||
task: sft
|
||||
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 2e-4
|
||||
batch_size: auto
|
||||
grace_codebook: true
|
||||
grace_codebook_size: 128
|
||||
grace_codebook_dim: 768
|
||||
|
||||
output: ./output
|
||||
"""
|
||||
cfg = load_config_from_string(yaml_text)
|
||||
assert cfg.training.grace_codebook is True
|
||||
Loading…
Reference in New Issue