mirror of https://github.com/razor-ai/soup.git
feat(v0.67.0): Adapter Lifecycle Finish — CMA-ES merge + VeRA bank + MoLE + PRs + soup.lock + bisect
Six surfaces close v0.57: - Part A: pure-Python rank-mu CMA-ES evolutionary merge (cmaes_merge.py) + soup adapters merge --strategy cmaes --eval <s> --budget 1h - Part B: VeRA / VB-LoRA vector-bank schema + atomic JSON I/O (vector_bank.py) - Part C: MoLE per-token routing schema + new task='moe_lora_routing' (mole_routing.py) - Part D: GitHub-shaped adapter PR renderer (adapter_pr.py) + soup adapters pr <title> --base-sha --adapter --eval --samples - Part E: soup.lock shared run lockfile (soup_lock.py + commands/lock.py) + soup lock write/show/check (exit 3 on drift) - Part F: training-history binary search (adapter_bisect.py) + soup adapters bisect <ckpts> --eval-command "..." Live wiring deferred to v0.67.1: CMA-ES eval-suite auto-bind, VeRA serving, MoLE gating kernel. +185 tests (10836 -> 11021) across 7 new test files. Review-fix coverage from 2 sequential waves (security + tdd-guide). All step-6 smokes green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
a015ccc812
commit
32145097ec
64
README.md
64
README.md
|
|
@ -42,14 +42,15 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.66.0 — Post-train X-rays: SAE feature diff + live influence-function blame + sleeper-agent defection probe + adapter interference matrix + probe pack.** Five surfaces extend `soup diagnose` from 6 failure modes to 10 — the post-train inspection layer no hosted vendor ships because mechanistic interpretability is research-grade and costs scale with N (so SaaS unit economics break). Closes the v0.57 `NotImplementedError` blame stub (#171).
|
||||
**v0.67.0 — Adapter Lifecycle Finish: CMA-ES evolutionary merge + VeRA/VB-LoRA bank storage + MoLE per-token routing + GitHub-shaped adapter PRs + `soup.lock` + `soup adapters bisect`.** Six surfaces finish what v0.57 started — turning adapter management into a real version-control + collaboration workflow. None of these exist in hosted vendors: Sakana-style evolutionary merge is research demo only, VeRA storage hurts hosted unit economics (price by GPU-hour not adapter count), MoLE routing requires both training + serving stacks, adapter PRs need weights + eval + history together.
|
||||
|
||||
- **`soup probe sae-diff <sae> <pre.json> <post.json>`** — Sparse-Autoencoder feature attribution. Pure-numpy math: ReLU(activations @ W_enc + b_enc) → mean post-pre diff per feature → top-K changes. Closed `HF_HUB_ALLOWLIST` over 8 known SAE families (Gemma Scope / Pythia / SAE-Lens Llama / GPT-2-small). Containment + `O_NOFOLLOW` open + 64-tensor cap. Composes with `soup adapters diff` (extends "what changed" from weight space to feature space).
|
||||
- **`soup adapters blame --top-k 50`** — Live DataInf-style influence runner closes v0.57 #171. Per-row score = `cos(grad_row, grad_probe) × |grad_row|`; operator supplies `probe_fn` returning `(row_grads, probe_grad)` or falls back to a deterministic synthetic probe (matches v0.54 advise stub policy — surface always returns a real `BlameResult`, never `NotImplementedError`). Symlink-rejected via `O_NOFOLLOW` on the dataset read (TOCTOU defence); `_DEFAULT_SYNTH_PROBE_CAP=100_000` so a 10M-row dataset doesn't allocate 1.28 GB.
|
||||
- **`soup probe sleeper <base> [--evidence ev.json]`** — Anthropic-style calibrated linear probe per base. 6 bundled bases (Llama-2/3 8B, Mistral-7B, Qwen2-7B, Gemma-2 2B/9B); deterministic SHA-256-keyed synthetic weights ship today (real calibrated weights in v0.66.x). OK/MINOR/MAJOR thresholds at 1% / 5% defection rate. Exit 2 on MAJOR for CI gating. Without `--evidence`, prints probe metadata + neutral OK.
|
||||
- **`soup probe interference <losses.json>`** — Pairwise N×N adapter interference matrix. Per-pair score = `(loss(A | A+B) - loss(A | A_alone)) / loss(A_alone)`; classify_interference at 5% / 20% bands. Surfaces which adapter pairs you cannot deploy together via `soup serve --adapters`. Rich-markup-escape on adapter names (review H2 fix — defends against crafted names like `[link=evil]X[/]`). MAJOR worst-pair → exit 2.
|
||||
- **`soup probe pack <base>`** — Manifest assembler that ties Parts A + C + D together. Per-base bundle of probe entries (sleeper / sae / truth / harm — last two reserved for v0.66.x); `MappingProxyType`-wrapped registry; `_LOWER_INDEX` for O(1) case-insensitive lookup.
|
||||
- **+259 new tests** (10577 → 10836). Review-fix coverage across 3 sequential waves: 0 CRITICAL + 9 HIGH (TypeError on bool/non-str verdict; `O_NOFOLLOW` probe-open on `load_sae_weights` + `_count_dataset_rows` closing TOCTOU race; `hashlib.sha256` replaces process-salted `hash()` for deterministic probe weights across Python processes; non-numeric loss rejection in `probe interference` CLI; Rich-markup escape on adapter / verdict / description in `render_*_markdown`) + 14 MEDIUM (64-bit seed via `digest[:16]`; 10M-row hard rejection (no silent truncate); `_DEFAULT_SYNTH_PROBE_CAP=100_000` synthetic probe cap; description 4096-char cap on `ProbeEntry`; `_LOWER_INDEX` `MappingProxyType`; `collections.abc.Mapping` migration; `frozenset[str]` type params) + 5 LOW (`FrozenInstanceError` instead of `pytest.raises((AttributeError, Exception))`; basename(normpath(...)) symmetry; top-K comment clarification; `Optional[str]` on output flag; reserved-kind docstring note). Manual CPU smokes for every new command incl. failure modes (unknown base exit 2 / non-numeric loss exit 2 / MAJOR interference exit 2 / live blame produces a real `BlameResult`).
|
||||
- **`soup adapters merge --strategy cmaes --eval <suite> --budget 1h`** — Sakana-style evolutionary search over LoRA merge weights. Pure-Python rank-mu CMA-ES (no `cma` dependency). Softmaxes N-1 logits onto the simplex, samples a population, keeps elite half, plateau-detects (3 generations without improvement → `converged=True`). Operator-supplied `eval_fn` closure; failures swallowed with sentinel score so one broken eval doesn't crash the run. Budget bounds reused from v0.57 `blame.parse_budget` (60s..24h). Live auto-wiring of the eval suite is deferred to v0.67.1; the CLI prints the validated plan today.
|
||||
- **`soup_cli.utils.vector_bank`** — VeRA / VB-LoRA storage format. `VectorBank = {shared random projection matrix P, per-user scaling vector v_u}` — thousands of per-user adapters at MB-each instead of hundreds-of-MB per LoRA. Atomic JSON I/O via shared `paths.atomic_write_text` + cwd containment + symlink rejection + 16 MiB cap. `estimate_bank_size(num_users, vector_dim)` for sizing. Live multi-tenant serving via v0.22 multi-adapter surface lands in v0.67.1.
|
||||
- **`task='moe_lora_routing'`** — MoLE per-token gating over N task LoRAs (Mixture of LoRA Experts). New `MoleGatingConfig` (num_task_adapters [2,64] / hidden_dim / temperature / top_k); cross-validator rejects mlx backend. Live gating-kernel training + per-token softmax routing lands in v0.67.1.
|
||||
- **`soup adapters pr <title> --base-sha <hex> --adapter <path> --eval <json>`** — GitHub-shaped PR rendering. PR = {base SHA, dataset diff, adapter weights, eval-delta report} → review-friendly Markdown with eval-delta tables + per-sample baseline/candidate diffs. `_md_table_escape` neutralises `\\` `|` `\n` `\r` `\t` in cells. JSON output for downstream consumers (v0.68 GitHub Action). Composes with `soup adapters diff`.
|
||||
- **`soup lock write / show / check`** — Shared run lockfile. Closure of `(base_model_sha, dataset_sha, env_hash)` → committed to git so teams coordinate on reproducible training runs. `soup lock check` exits 3 on drift; `soup_version` + `created_at` are advisory-only (legitimate operator upgrades don't trigger drift). Composes with v0.64 `soup env lock`.
|
||||
- **`soup adapters bisect <ckpt1> <ckpt2> ... --eval-command "..."`** — Binary search over training history. Operator supplies a shell template with `{ckpt}` placeholder (argv-list mode via `shlex.split` after `shlex.quote(ckpt)` — no `shell=True`). Probes both endpoints first (short-circuits all-OK / all-broken), then ~log₂(n) midpoint probes. Exit 3 on BROKEN_AT. Composes with v0.66 influence-blame for fine-grained attribution.
|
||||
- **+165 new tests** (10836 → 11021) across 7 new test files. Review-fix coverage across 2 sequential waves: 0 CRITICAL + 0 HIGH (project policies already enforced by source-grep guards inherited from v0.66) + ~5 MEDIUM (plateau-convergence test; vector-bank forward-compat for unknown JSON fields; PR empty-sections renders without `None` leakage; soup.lock `soup_version` drift is advisory-only; bisect non-monotonic eval_fn does not crash) + ~3 LOW (exact MAX-boundary acceptance tests; `validate_user_id` bool rejection; source-grep regression for `atomic_write_text` usage + no `shell=True` + `shlex.quote` in bisect subprocess call). Manual CPU smokes for `soup lock write/show/check` (incl. drift exit 3), `soup adapters pr` (markdown + JSON output), `soup adapters bisect --plan-only`, `soup adapters merge --strategy cmaes` (plan-only output), plus 4 failure-mode rejection paths (cmaes-without-eval / unknown-strategy / mlx + moe_lora_routing / lock drift).
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -192,6 +193,55 @@ soup probe pack meta-llama/Llama-3-8B # render the per-base manifest
|
|||
|
||||
Every probe uses the OK / MINOR / MAJOR taxonomy from v0.26 (Quant-Lobotomy) / v0.56 (Diagnose) / v0.65 (Eval Depth). Sleeper + interference exit 2 on MAJOR for CI gating. The blame runner closes the v0.57 `NotImplementedError` stub via a DataInf-style influence approximation: `cos(grad_row, grad_probe) × |grad_row|`. Operators supply a `probe_fn` returning `(row_grads, probe_grad)`, or the runner falls back to a deterministic synthetic probe so the surface always returns a real `BlameResult` (no exception leaks). SAE feature diff is pure-numpy; the safetensors loader is `O_NOFOLLOW`-protected (TOCTOU defence — closes the symlink swap window between containment check and read).
|
||||
|
||||
## Adapter Lifecycle (`soup adapters {merge,pr,bisect}`, `soup lock`)
|
||||
|
||||
v0.57 shipped `adapters diff / merge / blame / branch`. v0.67 finishes the lifecycle: evolutionary merge driven by your eval, GitHub-shaped PRs for adapter review, a shared `soup.lock` for team reproducibility, and binary-search bisect over training history.
|
||||
|
||||
```bash
|
||||
# 1. Evolutionary merge: search the simplex of merge weights via CMA-ES.
|
||||
soup adapters merge \
|
||||
adapter-finance/ adapter-medical/ adapter-legal/ \
|
||||
--strategy cmaes \
|
||||
--eval evals/domain_mix.yaml \
|
||||
--budget 1h \
|
||||
--population 8 \
|
||||
--max-generations 20 \
|
||||
--output merged/
|
||||
|
||||
# 2. Render the merge as a GitHub PR for review (eval deltas + sample diffs).
|
||||
soup adapters pr "merge: 3-domain blend" \
|
||||
--base-sha $(git rev-parse HEAD) \
|
||||
--adapter merged/ \
|
||||
--eval evals/deltas.json \
|
||||
--samples evals/samples.json \
|
||||
--dataset-diff data/diff.txt \
|
||||
--format markdown -o pr.md
|
||||
|
||||
# 3. Lock a reproducible run state. Closure = sha(base + dataset + env).
|
||||
soup env lock # v0.64 — capture env hash
|
||||
soup lock write \
|
||||
--base-model meta-llama/Llama-3.1-8B \
|
||||
--base-sha $BASE_SHA \
|
||||
--dataset-sha $DATA_SHA \
|
||||
--env-hash $(jq -r .closure soup-env.lock) \
|
||||
-o soup.lock
|
||||
# Teammates re-check the lock; exit 3 on drift.
|
||||
soup lock check soup.lock \
|
||||
--base-model meta-llama/Llama-3.1-8B \
|
||||
--base-sha $BASE_SHA --dataset-sha $DATA_SHA --env-hash $ENV_HASH
|
||||
|
||||
# 4. Bisect a training history to find the step that broke an eval.
|
||||
soup adapters bisect \
|
||||
ckpt-step-100 ckpt-step-200 ckpt-step-400 ckpt-step-800 \
|
||||
--eval-command "soup eval custom --model {ckpt} --tasks eval.jsonl" \
|
||||
-o bisect.json
|
||||
# Exits 3 on BROKEN_AT — pipe into `soup adapters blame` for attribution.
|
||||
```
|
||||
|
||||
CMA-ES is pure-Python (no `cma` dependency); the eval is operator-supplied via a closure so any scoring code works. PR rendering escapes Markdown table cells, so crafted metric names cannot inject table rows or links. The lockfile composes with v0.64 `soup env lock` — drift in any of `{base_model, base_model_sha, dataset_sha, env_hash, closure_sha}` exits 3 (`soup_version` and `created_at` are advisory-only). Bisect uses `shlex.split` + `shlex.quote(ckpt)` in argv-list mode (no `shell=True`), so checkpoint ids cannot inject shell metacharacters.
|
||||
|
||||
VeRA / VB-LoRA bank storage (`soup_cli.utils.vector_bank`) and MoLE per-token routing (`task='moe_lora_routing'`) ship as schema-only in v0.67.0 — live multi-tenant serving and gating-kernel training land in v0.67.1.
|
||||
|
||||
## Data Flywheel (`soup loop`)
|
||||
|
||||
The full *production traces → preference pairs → Eval-Gated DPO → canary deploy → rollback* loop, driven from a single CLI. Connects v0.26 Trace-to-Preference + Eval-Gated Training + Registry lineage + Quant-Lobotomy verdicts + Soup Cans + v0.25 Autopilot + v0.54 Advise + v0.55 Eval Design + v0.56 Diagnose.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.66.0"
|
||||
version = "0.67.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.66.0"
|
||||
__version__ = "0.67.0"
|
||||
|
|
|
|||
|
|
@ -408,6 +408,15 @@ app.add_typer(
|
|||
),
|
||||
)
|
||||
|
||||
# v0.67.0 Part E — soup.lock shared run lockfile.
|
||||
from soup_cli.commands import lock as _lock_cmd # noqa: E402
|
||||
|
||||
app.add_typer(
|
||||
_lock_cmd.app,
|
||||
name="lock",
|
||||
help="Shared run lockfile (write / show / check) - v0.67.0 Part E.",
|
||||
)
|
||||
|
||||
|
||||
def _rewrite_advise_argv(argv: list) -> list:
|
||||
"""Inject `run` between `advise` and a non-subcommand first argument.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
"""soup adapters — LoRA adapter management (list, info, compare, diff, merge, blame, branch)."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
|
@ -332,13 +333,32 @@ def merge(
|
|||
adapters: list[str] = typer.Argument(..., help="Two or more adapter paths to merge"),
|
||||
output: str = typer.Option(..., "--output", "-o", help="Output directory for merged adapter"),
|
||||
strategy: str = typer.Option("linear", "--strategy",
|
||||
help="linear | ties | dare | svd"),
|
||||
help="linear | ties | dare | svd | cmaes"),
|
||||
weights: str = typer.Option(None, "--weights",
|
||||
help="Comma-separated weights (default: equal)"),
|
||||
density: float = typer.Option(0.2, "--density",
|
||||
help="Trim density for ties/dare in (0, 1]"),
|
||||
seed: int = typer.Option(0, "--seed", help="Random seed for dare"),
|
||||
seed: int = typer.Option(0, "--seed", help="Random seed for dare / cmaes"),
|
||||
rank: int = typer.Option(None, "--rank", help="SVD rank (svd strategy only)"),
|
||||
eval_suite: Optional[str] = typer.Option(
|
||||
None, "--eval",
|
||||
help=(
|
||||
"Path to eval suite (required for --strategy cmaes). "
|
||||
"Used by the evolutionary loop to score candidate merges."
|
||||
),
|
||||
),
|
||||
budget: str = typer.Option(
|
||||
"1h", "--budget",
|
||||
help="Wall-clock budget for cmaes (60s..24h, e.g. 1h, 30m)",
|
||||
),
|
||||
population: int = typer.Option(
|
||||
8, "--population", min=2, max=256,
|
||||
help="cmaes population size per generation",
|
||||
),
|
||||
max_generations: int = typer.Option(
|
||||
20, "--max-generations", min=1, max=10_000,
|
||||
help="cmaes generation cap",
|
||||
),
|
||||
license_ids: list[str] = typer.Option(
|
||||
None, "--license",
|
||||
help=(
|
||||
|
|
@ -373,6 +393,48 @@ def merge(
|
|||
console.print("[red]Need at least 2 adapter paths to merge[/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
# v0.67.0 Part A: cmaes evolutionary search requires an eval suite +
|
||||
# budget. Render the planned strategy and emit a deferred-live advisory.
|
||||
# Operator-supplied eval_fn lives in `soup_cli.utils.cmaes_merge.run_cmaes_merge`;
|
||||
# the CLI surface here ships the plan + validation. Live `soup eval`
|
||||
# auto-wiring against a registry run is deferred to v0.67.1.
|
||||
if strategy == "cmaes":
|
||||
if eval_suite is None:
|
||||
console.print(
|
||||
"[red]--strategy cmaes requires --eval <suite> "
|
||||
"(path to a YAML/JSONL eval).[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan
|
||||
|
||||
try:
|
||||
cmaes_plan = build_cmaes_plan(
|
||||
adapters=adapters,
|
||||
eval_suite=eval_suite,
|
||||
budget_spec=budget,
|
||||
population_size=population,
|
||||
max_generations=max_generations,
|
||||
seed=seed,
|
||||
)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid cmaes plan: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
panel = Panel(
|
||||
f"Strategy: [bold]cmaes[/]\n"
|
||||
f"Adapters: {len(cmaes_plan.adapters)}\n"
|
||||
f"Eval suite: [bold]{escape(os.path.basename(cmaes_plan.eval_suite))}[/]\n"
|
||||
f"Budget: [bold]{cmaes_plan.budget_seconds}s[/]\n"
|
||||
f"Population: [bold]{cmaes_plan.population_size}[/]\n"
|
||||
f"Max generations: [bold]{cmaes_plan.max_generations}[/]\n"
|
||||
f"Live cmaes loop with auto-wired eval suite is deferred to "
|
||||
f"[yellow]v0.67.1[/]. The plan is validated and ready to "
|
||||
f"hand to an operator-supplied `eval_fn` via "
|
||||
f"`run_cmaes_merge(plan, eval_fn=...)`.",
|
||||
title="Adapter merge — cmaes (plan)",
|
||||
)
|
||||
console.print(panel)
|
||||
return
|
||||
|
||||
# v0.60.0 Part E: license-conflict gate. Operators MUST declare a
|
||||
# license per adapter (or pass --license-override <reason>).
|
||||
if license_ids:
|
||||
|
|
@ -837,3 +899,254 @@ def check_safetensors(
|
|||
for path in report.unsafe_files:
|
||||
console.print(f" [yellow]- {escape(path)}[/]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@app.command(name="pr")
|
||||
def adapter_pr(
|
||||
title: str = typer.Argument(..., help="Short PR title (e.g. 'add-support-tone')"),
|
||||
base_sha: str = typer.Option(..., "--base-sha", help="64-hex SHA of the base model"),
|
||||
adapter_path: str = typer.Option(..., "--adapter", help="Path to candidate adapter"),
|
||||
eval_json: Optional[str] = typer.Option(
|
||||
None, "--eval",
|
||||
help=(
|
||||
"Path to JSON eval-deltas: list of "
|
||||
'{"metric": ..., "baseline": ..., "candidate": ...}'
|
||||
),
|
||||
),
|
||||
samples_json: Optional[str] = typer.Option(
|
||||
None, "--samples",
|
||||
help=(
|
||||
"Path to JSON sample diffs: list of "
|
||||
'{"prompt": ..., "baseline_output": ..., "candidate_output": ...}'
|
||||
),
|
||||
),
|
||||
dataset_diff_path: Optional[str] = typer.Option(
|
||||
None, "--dataset-diff",
|
||||
help="Path to a text file containing the dataset diff",
|
||||
),
|
||||
output: Optional[str] = typer.Option(
|
||||
None, "--output", "-o",
|
||||
help="Write rendered Markdown to path (default: stdout)",
|
||||
),
|
||||
format_: str = typer.Option(
|
||||
"markdown", "--format", "-f", help="markdown | json",
|
||||
),
|
||||
):
|
||||
"""Render a GitHub-style PR for an adapter (v0.67.0 Part D).
|
||||
|
||||
The PR = ``{base SHA, dataset diff, adapter file, eval report}``
|
||||
rendered as a Markdown document with eval-delta tables and sample
|
||||
diffs, ready to drop into a GitHub PR description or comment.
|
||||
"""
|
||||
from soup_cli.utils.adapter_pr import (
|
||||
build_adapter_pr,
|
||||
render_pr_json,
|
||||
render_pr_markdown,
|
||||
write_pr_markdown,
|
||||
)
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
if format_ not in ("markdown", "json"):
|
||||
console.print(f"[red]Unknown --format: {escape(format_)}[/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
# Load deltas + samples + dataset_diff lazily; each accepts None.
|
||||
def _load_json_list(path: Optional[str], field: str) -> list:
|
||||
if path is None:
|
||||
return []
|
||||
enforce_under_cwd_and_no_symlink(path, field=field)
|
||||
with open(os.path.realpath(path), encoding="utf-8") as fh:
|
||||
raw = json.load(fh)
|
||||
if not isinstance(raw, list):
|
||||
raise ValueError(f"{field} must contain a JSON list")
|
||||
return raw
|
||||
|
||||
try:
|
||||
deltas = _load_json_list(eval_json, "eval")
|
||||
samples = _load_json_list(samples_json, "samples")
|
||||
dataset_diff = ""
|
||||
if dataset_diff_path is not None:
|
||||
enforce_under_cwd_and_no_symlink(
|
||||
dataset_diff_path, field="dataset_diff"
|
||||
)
|
||||
with open(
|
||||
os.path.realpath(dataset_diff_path), encoding="utf-8"
|
||||
) as fh:
|
||||
dataset_diff = fh.read()
|
||||
pr = build_adapter_pr(
|
||||
title=title,
|
||||
base_sha=base_sha,
|
||||
adapter_path=adapter_path,
|
||||
dataset_diff=dataset_diff,
|
||||
deltas=deltas,
|
||||
samples=samples,
|
||||
)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
if format_ == "json":
|
||||
rendered = render_pr_json(pr)
|
||||
else:
|
||||
rendered = render_pr_markdown(pr)
|
||||
|
||||
if output is None:
|
||||
console.print(rendered)
|
||||
return
|
||||
|
||||
try:
|
||||
if format_ == "markdown":
|
||||
write_pr_markdown(pr, output)
|
||||
else:
|
||||
from soup_cli.utils.paths import atomic_write_text
|
||||
|
||||
atomic_write_text(rendered, output, field="pr output")
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"PR: [bold]{escape(pr.title)}[/]\n"
|
||||
f"Format: [bold]{escape(format_)}[/]\n"
|
||||
f"Output: [bold]{escape(output)}[/]",
|
||||
title="Adapter PR rendered",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@app.command(name="bisect")
|
||||
def adapter_bisect(
|
||||
history: list[str] = typer.Argument(
|
||||
...,
|
||||
help=(
|
||||
"Ordered checkpoint identifiers (oldest first). At least 2. "
|
||||
"Example: `soup adapters bisect ckpt-100 ckpt-200 ckpt-300 ckpt-400`"
|
||||
),
|
||||
),
|
||||
eval_command: str = typer.Option(
|
||||
...,
|
||||
"--eval-command",
|
||||
help=(
|
||||
"Shell command template; '{ckpt}' is replaced with each "
|
||||
"checkpoint id. Exit code 0 means PASS, non-zero means FAIL. "
|
||||
"Example: 'soup eval custom --model {ckpt} --tasks tasks.jsonl'"
|
||||
),
|
||||
),
|
||||
plan_only: bool = typer.Option(
|
||||
False, "--plan-only",
|
||||
help="Print the plan and exit without running the bisect",
|
||||
),
|
||||
output: Optional[str] = typer.Option(
|
||||
None, "--output", "-o",
|
||||
help="Write JSON result to path",
|
||||
),
|
||||
):
|
||||
"""Binary-search a training history to find the first failing checkpoint.
|
||||
|
||||
Composes with v0.66 Part B influence-blame: once the boundary is
|
||||
found, ``soup adapters blame`` can attribute the regression to
|
||||
specific dataset rows.
|
||||
"""
|
||||
from soup_cli.utils.adapter_bisect import build_bisect_plan, run_bisect
|
||||
|
||||
if not isinstance(history, list) or len(history) < 2:
|
||||
console.print("[red]Need at least 2 checkpoint ids[/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
try:
|
||||
plan = build_bisect_plan(history)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid history: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
if "{ckpt}" not in eval_command:
|
||||
console.print(
|
||||
"[red]--eval-command must include `{ckpt}` placeholder[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
|
||||
if plan_only:
|
||||
console.print(
|
||||
Panel(
|
||||
f"History: [bold]{len(plan.history)} checkpoints[/]\n"
|
||||
f"Eval: [bold]{escape(eval_command)}[/]\n"
|
||||
f"Probes: ~[bold]{_estimated_probes(len(plan.history))}[/] iterations",
|
||||
title="Adapter bisect (plan)",
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
# Execute via subprocess; exit 0 means OK.
|
||||
import shlex
|
||||
import subprocess # noqa: S404 — argv list mode, no shell
|
||||
|
||||
def _eval_fn(ckpt: str) -> bool:
|
||||
cmd_str = eval_command.replace("{ckpt}", shlex.quote(ckpt))
|
||||
# Safer: split via shlex and use argv mode (no shell interpolation).
|
||||
argv = shlex.split(cmd_str)
|
||||
try:
|
||||
result = subprocess.run( # noqa: S603
|
||||
argv, capture_output=True, timeout=3600, check=False
|
||||
)
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError) as exc:
|
||||
console.print(
|
||||
f"[yellow]Eval failed for {escape(ckpt)}: "
|
||||
f"{escape(str(exc))}[/]"
|
||||
)
|
||||
return False
|
||||
return result.returncode == 0
|
||||
|
||||
try:
|
||||
result = run_bisect(plan, eval_fn=_eval_fn)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
if result.verdict == "ALL_OK":
|
||||
console.print(
|
||||
Panel(
|
||||
f"Verdict: [green]ALL_OK[/]\n"
|
||||
f"Probes: {result.probes}",
|
||||
title="Adapter bisect",
|
||||
)
|
||||
)
|
||||
else:
|
||||
console.print(
|
||||
Panel(
|
||||
f"Verdict: [red]BROKEN_AT[/]\n"
|
||||
f"First broken: [bold]{escape(result.first_broken or '?')}[/]\n"
|
||||
f"Probes: {result.probes}",
|
||||
title="Adapter bisect",
|
||||
)
|
||||
)
|
||||
|
||||
if output is not None:
|
||||
from soup_cli.utils.paths import atomic_write_text
|
||||
|
||||
data = {
|
||||
"verdict": result.verdict,
|
||||
"first_broken": result.first_broken,
|
||||
"probes": result.probes,
|
||||
"steps": [
|
||||
{"checkpoint": s.checkpoint, "ok": s.ok}
|
||||
for s in result.steps
|
||||
],
|
||||
}
|
||||
atomic_write_text(
|
||||
json.dumps(data, indent=2, sort_keys=True),
|
||||
output,
|
||||
field="bisect output",
|
||||
)
|
||||
|
||||
if result.verdict == "BROKEN_AT":
|
||||
raise typer.Exit(2)
|
||||
|
||||
|
||||
def _estimated_probes(n: int) -> int:
|
||||
"""Approximate number of probes for n checkpoints (~log2(n) + 2)."""
|
||||
import math
|
||||
|
||||
if n <= 2:
|
||||
return n
|
||||
return max(2, math.ceil(math.log2(n))) + 2
|
||||
|
|
|
|||
|
|
@ -0,0 +1,165 @@
|
|||
"""soup lock — shared run lockfile (v0.67.0 Part E).
|
||||
|
||||
Subcommands:
|
||||
|
||||
- ``soup lock write``: render a ``soup.lock`` from operator-supplied
|
||||
base-model / dataset / env hashes.
|
||||
- ``soup lock check``: compare a tracked ``soup.lock`` against a
|
||||
freshly-computed closure; exit 3 on drift.
|
||||
- ``soup lock show``: print a tracked lock.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
|
||||
console = Console()
|
||||
|
||||
app = typer.Typer(no_args_is_help=True, help="Shared run lockfile (v0.67.0 Part E)")
|
||||
|
||||
|
||||
@app.command(name="write")
|
||||
def write_lock_cmd(
|
||||
base_model: str = typer.Option(..., "--base-model", help="HF model id / path"),
|
||||
base_sha: str = typer.Option(..., "--base-sha", help="64-hex base-model SHA"),
|
||||
dataset_sha: str = typer.Option(..., "--dataset-sha", help="64-hex dataset SHA"),
|
||||
env_hash: str = typer.Option(..., "--env-hash", help="64-hex env hash (soup env)"),
|
||||
output: str = typer.Option("soup.lock", "--output", "-o", help="Output path"),
|
||||
):
|
||||
"""Render a ``soup.lock`` from the three required hashes."""
|
||||
from soup_cli import __version__
|
||||
from soup_cli.utils.soup_lock import SoupLock, compute_lock_closure, write_lock
|
||||
|
||||
try:
|
||||
closure = compute_lock_closure(
|
||||
base_model_sha=base_sha,
|
||||
dataset_sha=dataset_sha,
|
||||
env_hash=env_hash,
|
||||
)
|
||||
lock = SoupLock(
|
||||
soup_version=__version__,
|
||||
base_model=base_model,
|
||||
base_model_sha=base_sha,
|
||||
dataset_sha=dataset_sha,
|
||||
env_hash=env_hash,
|
||||
closure_sha=closure,
|
||||
created_at=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
write_lock(lock, output)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Lock: [bold]{escape(output)}[/]\n"
|
||||
f"Base: [bold]{escape(base_model)}[/]\n"
|
||||
f"Closure SHA: [bold]{closure[:12]}…[/]",
|
||||
title="soup.lock written",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@app.command(name="show")
|
||||
def show_lock_cmd(
|
||||
path: str = typer.Argument("soup.lock", help="Path to soup.lock"),
|
||||
):
|
||||
"""Print a tracked lock file."""
|
||||
from soup_cli.utils.soup_lock import read_lock
|
||||
|
||||
try:
|
||||
lock = read_lock(path)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Soup version: [bold]{escape(lock.soup_version)}[/]\n"
|
||||
f"Base model: [bold]{escape(lock.base_model)}[/]\n"
|
||||
f"Base SHA: [bold]{lock.base_model_sha[:16]}…[/]\n"
|
||||
f"Dataset SHA: [bold]{lock.dataset_sha[:16]}…[/]\n"
|
||||
f"Env hash: [bold]{lock.env_hash[:16]}…[/]\n"
|
||||
f"Closure SHA: [bold]{lock.closure_sha[:16]}…[/]\n"
|
||||
f"Created at: [bold]{escape(lock.created_at)}[/]",
|
||||
title=f"soup.lock — {escape(path)}",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@app.command(name="check")
|
||||
def check_lock_cmd(
|
||||
path: str = typer.Argument("soup.lock", help="Path to tracked soup.lock"),
|
||||
base_sha: str = typer.Option(..., "--base-sha", help="64-hex current base-model SHA"),
|
||||
dataset_sha: str = typer.Option(..., "--dataset-sha", help="64-hex current dataset SHA"),
|
||||
env_hash: str = typer.Option(..., "--env-hash", help="64-hex current env hash"),
|
||||
base_model: str = typer.Option(..., "--base-model", help="Current base model id"),
|
||||
):
|
||||
"""Refuse with exit 3 if the lock has drifted from current state."""
|
||||
from soup_cli import __version__
|
||||
from soup_cli.utils.soup_lock import (
|
||||
SoupLock,
|
||||
check_lock_drift,
|
||||
compute_lock_closure,
|
||||
read_lock,
|
||||
)
|
||||
|
||||
try:
|
||||
expected = read_lock(path)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
try:
|
||||
closure = compute_lock_closure(
|
||||
base_model_sha=base_sha,
|
||||
dataset_sha=dataset_sha,
|
||||
env_hash=env_hash,
|
||||
)
|
||||
# Use the existing soup_version + created_at from `expected` so the
|
||||
# comparison stays content-only (drift only counts the 5 content
|
||||
# fields per `check_lock_drift`).
|
||||
actual = SoupLock(
|
||||
soup_version=expected.soup_version,
|
||||
base_model=base_model,
|
||||
base_model_sha=base_sha,
|
||||
dataset_sha=dataset_sha,
|
||||
env_hash=env_hash,
|
||||
closure_sha=closure,
|
||||
created_at=expected.created_at,
|
||||
)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
drift = check_lock_drift(expected, actual)
|
||||
if drift.ok:
|
||||
console.print(
|
||||
Panel(
|
||||
f"Lock: [bold]{escape(path)}[/]\n"
|
||||
f"Status: [green]OK[/] — closure matches",
|
||||
title="soup lock check",
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Lock: [bold]{escape(path)}[/]\n"
|
||||
f"Status: [red]DRIFT[/]",
|
||||
title="soup lock check",
|
||||
)
|
||||
)
|
||||
for change in drift.changes:
|
||||
console.print(f" [red]- {escape(change)}[/]")
|
||||
if __version__ != expected.soup_version:
|
||||
console.print(
|
||||
f"[yellow]Note: soup version changed "
|
||||
f"({escape(expected.soup_version)} -> {escape(__version__)})[/]"
|
||||
)
|
||||
raise typer.Exit(3)
|
||||
|
|
@ -2521,6 +2521,8 @@ class SoupConfig(BaseModel):
|
|||
"tts", "classifier", "reranker", "cross_encoder", "distill",
|
||||
# v0.61.0 Part A — Unlearning (NPO / SimNPO / RMU).
|
||||
"unlearn",
|
||||
# v0.67.0 Part C — MoLE per-token adapter routing (Mixture of LoRA Experts).
|
||||
"moe_lora_routing",
|
||||
] = Field(
|
||||
default="sft",
|
||||
description=(
|
||||
|
|
@ -2528,7 +2530,8 @@ class SoupConfig(BaseModel):
|
|||
"'tts' (TTS fine-tuning), 'classifier' / 'reranker' / "
|
||||
"'cross_encoder' (classification heads), and 'distill' "
|
||||
"(knowledge distillation). v0.61.0 adds 'unlearn' (NPO / "
|
||||
"SimNPO / RMU)."
|
||||
"SimNPO / RMU). v0.67.0 adds 'moe_lora_routing' (per-token "
|
||||
"gating over N task LoRAs)."
|
||||
),
|
||||
)
|
||||
modality: Literal["text", "vision", "audio", "audio_out"] = Field(
|
||||
|
|
@ -3700,6 +3703,27 @@ class SoupConfig(BaseModel):
|
|||
raise ValueError(str(exc)) from exc
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_mole_routing_compat(self) -> "SoupConfig":
|
||||
"""v0.67.0 Part C — MoLE per-token routing cross-validator.
|
||||
|
||||
Rules:
|
||||
* ``task='moe_lora_routing'`` rejects ``backend='mlx'`` (live wiring
|
||||
deferred to v0.67.1; the gating kernel needs torch dispatch).
|
||||
* The full ``MoleGatingConfig`` validation lives in
|
||||
``soup_cli.utils.mole_routing.validate_mole_compat`` and is
|
||||
exercised at trainer construction time (the v0.67.0 release ships
|
||||
the schema lock-in; live training lands in v0.67.1).
|
||||
"""
|
||||
if self.task != "moe_lora_routing":
|
||||
return self
|
||||
if self.backend == "mlx":
|
||||
raise ValueError(
|
||||
"MoLE routing (task='moe_lora_routing') is not supported on "
|
||||
"the mlx backend (live wiring deferred to v0.67.1)."
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_mlx_task_support(self) -> "SoupConfig":
|
||||
"""MLX backend only supports sft, dpo, and grpo tasks (v0.25.0).
|
||||
|
|
|
|||
|
|
@ -0,0 +1,237 @@
|
|||
"""``soup adapters bisect`` — binary search over training history (v0.67.0 Part F).
|
||||
|
||||
Given an ordered history of checkpoints (or dataset commits) and an
|
||||
operator-supplied predicate that returns True when "eval passes",
|
||||
binary-search for the FIRST checkpoint where the predicate flips to
|
||||
False. The result is the regression boundary.
|
||||
|
||||
Composes with v0.66 Part B influence-blame: once the boundary is
|
||||
found, ``soup adapters blame`` can attribute the regression to
|
||||
specific dataset rows.
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``BisectPlan`` / ``BisectStep`` / ``BisectResult`` frozen dataclasses
|
||||
- ``build_bisect_plan(history)`` factory
|
||||
- ``bisect_next_step(plan, lo, hi)`` pure midpoint kernel
|
||||
- ``run_bisect(plan, *, eval_fn)`` end-to-end binary search
|
||||
|
||||
Closed verdict taxonomy:
|
||||
- ``ALL_OK`` — every checkpoint passes (no regression found)
|
||||
- ``BROKEN_AT`` — found first failing checkpoint
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable, Optional, Sequence, Tuple
|
||||
|
||||
MAX_HISTORY = 4096
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _check_checkpoint(value: object, field: str) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} entries must be str")
|
||||
if not value:
|
||||
raise ValueError(f"{field} entry must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} entry must not contain null bytes")
|
||||
if len(value) > 1024:
|
||||
raise ValueError(f"{field} entry too long")
|
||||
return value
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frozen dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BisectPlan:
|
||||
"""An ordered history of checkpoint identifiers to bisect over.
|
||||
|
||||
The order is "oldest first" — index 0 is the earliest checkpoint
|
||||
and the last entry is the most recent.
|
||||
"""
|
||||
|
||||
history: Tuple[str, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.history, tuple):
|
||||
raise TypeError("history must be tuple")
|
||||
if len(self.history) < 2:
|
||||
raise ValueError(
|
||||
"history must contain at least 2 entries"
|
||||
)
|
||||
if len(self.history) > MAX_HISTORY:
|
||||
raise ValueError(
|
||||
f"history length {len(self.history)} > {MAX_HISTORY}"
|
||||
)
|
||||
seen: set[str] = set()
|
||||
for entry in self.history:
|
||||
_check_checkpoint(entry, "history")
|
||||
if entry in seen:
|
||||
raise ValueError(
|
||||
f"history must be unique (duplicate {entry!r})"
|
||||
)
|
||||
seen.add(entry)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BisectStep:
|
||||
"""One probe result during a bisect run."""
|
||||
|
||||
checkpoint: str
|
||||
ok: bool
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_checkpoint(self.checkpoint, "checkpoint")
|
||||
if not isinstance(self.ok, bool):
|
||||
raise TypeError("ok must be bool")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BisectResult:
|
||||
"""Outcome of a bisect run.
|
||||
|
||||
``first_broken`` is ``None`` when ``verdict='ALL_OK'``; otherwise
|
||||
it is the first failing checkpoint id.
|
||||
"""
|
||||
|
||||
first_broken: Optional[str]
|
||||
verdict: str
|
||||
steps: Tuple[BisectStep, ...]
|
||||
probes: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
valid_verdicts = ("ALL_OK", "BROKEN_AT")
|
||||
if self.verdict not in valid_verdicts:
|
||||
raise ValueError(
|
||||
f"verdict must be one of {valid_verdicts}"
|
||||
)
|
||||
if self.first_broken is not None:
|
||||
_check_checkpoint(self.first_broken, "first_broken")
|
||||
if not isinstance(self.steps, tuple):
|
||||
raise TypeError("steps must be tuple")
|
||||
for s in self.steps:
|
||||
if not isinstance(s, BisectStep):
|
||||
raise TypeError("steps entries must be BisectStep")
|
||||
if isinstance(self.probes, bool) or not isinstance(self.probes, int):
|
||||
raise TypeError("probes must be int")
|
||||
if self.probes < 0:
|
||||
raise ValueError("probes must be non-negative")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Factory
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_bisect_plan(history: Sequence[str]) -> BisectPlan:
|
||||
"""Build a frozen plan from a sequence of checkpoint ids."""
|
||||
if isinstance(history, str) or not isinstance(history, Sequence):
|
||||
raise TypeError("history must be a non-string sequence")
|
||||
return BisectPlan(history=tuple(history))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure midpoint kernel
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def bisect_next_step(plan: BisectPlan, *, lo: int, hi: int) -> int:
|
||||
"""Return the midpoint index between lo and hi (inclusive).
|
||||
|
||||
Pure: deterministic given the same inputs. Used internally by
|
||||
``run_bisect`` but exposed for callers that want to drive the
|
||||
bisect loop themselves.
|
||||
"""
|
||||
if not isinstance(plan, BisectPlan):
|
||||
raise TypeError("plan must be BisectPlan")
|
||||
if isinstance(lo, bool) or not isinstance(lo, int):
|
||||
raise TypeError("lo must be int")
|
||||
if isinstance(hi, bool) or not isinstance(hi, int):
|
||||
raise TypeError("hi must be int")
|
||||
if lo < 0 or hi >= len(plan.history):
|
||||
raise ValueError("lo/hi out of bounds")
|
||||
if lo > hi:
|
||||
raise ValueError("lo must be <= hi")
|
||||
return (lo + hi) // 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Orchestrator
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_bisect(
|
||||
plan: BisectPlan,
|
||||
*,
|
||||
eval_fn: Callable[[str], bool],
|
||||
) -> BisectResult:
|
||||
"""Find the first failing checkpoint via binary search.
|
||||
|
||||
``eval_fn(checkpoint_id)`` MUST return ``True`` when the checkpoint
|
||||
passes (no regression) and ``False`` when it fails. The bisect
|
||||
assumes monotonic regression: once a checkpoint fails, every later
|
||||
one also fails. Non-monotonic histories produce undefined results;
|
||||
operators wanting a sweep should use ``soup eval`` directly.
|
||||
"""
|
||||
if not isinstance(plan, BisectPlan):
|
||||
raise TypeError("plan must be BisectPlan")
|
||||
if eval_fn is None or not callable(eval_fn):
|
||||
raise TypeError("eval_fn must be callable")
|
||||
|
||||
n = len(plan.history)
|
||||
steps: list[BisectStep] = []
|
||||
|
||||
# Probe both endpoints first to short-circuit "all OK" / "all broken".
|
||||
first = plan.history[0]
|
||||
first_ok = bool(eval_fn(first))
|
||||
steps.append(BisectStep(checkpoint=first, ok=first_ok))
|
||||
if not first_ok:
|
||||
# All broken from the very first checkpoint.
|
||||
return BisectResult(
|
||||
first_broken=first,
|
||||
verdict="BROKEN_AT",
|
||||
steps=tuple(steps),
|
||||
probes=len(steps),
|
||||
)
|
||||
|
||||
last = plan.history[-1]
|
||||
last_ok = bool(eval_fn(last))
|
||||
steps.append(BisectStep(checkpoint=last, ok=last_ok))
|
||||
if last_ok:
|
||||
return BisectResult(
|
||||
first_broken=None,
|
||||
verdict="ALL_OK",
|
||||
steps=tuple(steps),
|
||||
probes=len(steps),
|
||||
)
|
||||
|
||||
# Now standard binary search: find the LOWEST index that fails.
|
||||
# Invariant: plan.history[lo] passes; plan.history[hi] fails.
|
||||
lo, hi = 0, n - 1
|
||||
while hi - lo > 1:
|
||||
mid = (lo + hi) // 2
|
||||
ckpt = plan.history[mid]
|
||||
ok = bool(eval_fn(ckpt))
|
||||
steps.append(BisectStep(checkpoint=ckpt, ok=ok))
|
||||
if ok:
|
||||
lo = mid
|
||||
else:
|
||||
hi = mid
|
||||
|
||||
return BisectResult(
|
||||
first_broken=plan.history[hi],
|
||||
verdict="BROKEN_AT",
|
||||
steps=tuple(steps),
|
||||
probes=len(steps),
|
||||
)
|
||||
|
|
@ -25,11 +25,17 @@ from typing import Any, FrozenSet, Literal, Mapping, Sequence, Tuple
|
|||
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
MergeStrategy = Literal["linear", "ties", "dare", "svd"]
|
||||
MergeStrategy = Literal["linear", "ties", "dare", "svd", "cmaes"]
|
||||
# frozenset for O(1) membership + immutability (mirrors v0.41.0 / v0.51.0 policy);
|
||||
# tuple alias preserved for caller code that iterates in canonical order.
|
||||
SUPPORTED_STRATEGIES: FrozenSet[str] = frozenset({"linear", "ties", "dare", "svd"})
|
||||
STRATEGY_ORDER: Tuple[MergeStrategy, ...] = ("linear", "ties", "dare", "svd")
|
||||
# v0.67.0 Part A: added "cmaes" — evolutionary search dispatched separately
|
||||
# in commands/adapters.py (requires --eval suite + --budget).
|
||||
SUPPORTED_STRATEGIES: FrozenSet[str] = frozenset(
|
||||
{"linear", "ties", "dare", "svd", "cmaes"}
|
||||
)
|
||||
STRATEGY_ORDER: Tuple[MergeStrategy, ...] = (
|
||||
"linear", "ties", "dare", "svd", "cmaes"
|
||||
)
|
||||
|
||||
_MAX_ADAPTERS = 16
|
||||
_MIN_ADAPTERS = 2
|
||||
|
|
|
|||
|
|
@ -0,0 +1,345 @@
|
|||
"""GitHub-shaped adapter PR rendering (v0.67.0 Part D).
|
||||
|
||||
A PR for an adapter is the triple ``{base SHA, dataset diff, adapter
|
||||
weights}`` plus an eval-delta report. This module renders it as a
|
||||
review-friendly Markdown document with eval-delta tables and
|
||||
side-by-side sample-output diffs, suitable for posting as a GitHub
|
||||
PR comment via the v0.68.0 GitHub Action.
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``EvalDelta`` / ``SampleDiff`` / ``AdapterPR`` frozen dataclasses
|
||||
- ``build_adapter_pr(...)`` factory from raw dicts (CLI / API friendly)
|
||||
- ``render_pr_markdown(pr)`` returns review-ready Markdown
|
||||
- ``render_pr_json(pr)`` returns JSON for downstream consumers
|
||||
- ``write_pr_markdown(pr, path)`` atomic cwd-contained write
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Any, Iterable, Mapping, Tuple
|
||||
|
||||
from soup_cli.utils.paths import atomic_write_text
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bounds (closed)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MAX_TITLE_LEN = 256
|
||||
MAX_METRIC_NAME_LEN = 256
|
||||
MAX_OUTPUT_LEN = 32_768 # per-output cap (defense + keeps PRs reviewable)
|
||||
MAX_DATASET_DIFF_LEN = 1_048_576 # 1 MiB
|
||||
MAX_DELTAS = 64
|
||||
MAX_SAMPLES = 256
|
||||
_SHA256_RE = re.compile(r"^[0-9a-f]{64}$")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _check_text_field(value: object, field: str, max_len: int) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > max_len:
|
||||
raise ValueError(
|
||||
f"{field} length {len(value)} > {max_len}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _check_required_text(value: object, field: str, max_len: int) -> str:
|
||||
text = _check_text_field(value, field, max_len)
|
||||
if not text:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
return text
|
||||
|
||||
|
||||
def _check_finite(value: object, field: str) -> float:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(f"{field} must be numeric")
|
||||
val = float(value)
|
||||
if not math.isfinite(val):
|
||||
raise ValueError(f"{field} must be finite")
|
||||
return val
|
||||
|
||||
|
||||
def _check_sha256(value: object, field: str) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str")
|
||||
if not _SHA256_RE.match(value):
|
||||
raise ValueError(f"{field} must be 64 hex chars")
|
||||
return value
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frozen dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvalDelta:
|
||||
"""One metric: baseline -> candidate.
|
||||
|
||||
The ``delta`` field is a computed property (post-init).
|
||||
"""
|
||||
|
||||
metric: str
|
||||
baseline: float
|
||||
candidate: float
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_required_text(self.metric, "metric", MAX_METRIC_NAME_LEN)
|
||||
_check_finite(self.baseline, "baseline")
|
||||
_check_finite(self.candidate, "candidate")
|
||||
|
||||
@property
|
||||
def delta(self) -> float:
|
||||
return self.candidate - self.baseline
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SampleDiff:
|
||||
"""One prompt with baseline + candidate outputs.
|
||||
|
||||
Long outputs are rejected (not truncated) so PR authors notice and
|
||||
pre-truncate intentionally — keeps the PR reviewable.
|
||||
"""
|
||||
|
||||
prompt: str
|
||||
baseline_output: str
|
||||
candidate_output: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_required_text(self.prompt, "prompt", MAX_OUTPUT_LEN)
|
||||
_check_text_field(
|
||||
self.baseline_output, "baseline_output", MAX_OUTPUT_LEN
|
||||
)
|
||||
_check_text_field(
|
||||
self.candidate_output, "candidate_output", MAX_OUTPUT_LEN
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AdapterPR:
|
||||
"""An adapter PR — the triple {base SHA, dataset diff, adapter path}
|
||||
plus eval deltas + sample diffs for human review."""
|
||||
|
||||
title: str
|
||||
base_sha: str
|
||||
adapter_path: str
|
||||
dataset_diff: str
|
||||
deltas: Tuple[EvalDelta, ...]
|
||||
samples: Tuple[SampleDiff, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_required_text(self.title, "title", MAX_TITLE_LEN)
|
||||
_check_sha256(self.base_sha, "base_sha")
|
||||
_check_required_text(self.adapter_path, "adapter_path", MAX_OUTPUT_LEN)
|
||||
_check_text_field(
|
||||
self.dataset_diff, "dataset_diff", MAX_DATASET_DIFF_LEN
|
||||
)
|
||||
if not isinstance(self.deltas, tuple):
|
||||
raise TypeError("deltas must be tuple")
|
||||
if len(self.deltas) > MAX_DELTAS:
|
||||
raise ValueError(
|
||||
f"too many deltas ({len(self.deltas)} > {MAX_DELTAS})"
|
||||
)
|
||||
for d in self.deltas:
|
||||
if not isinstance(d, EvalDelta):
|
||||
raise TypeError("deltas entries must be EvalDelta")
|
||||
if not isinstance(self.samples, tuple):
|
||||
raise TypeError("samples must be tuple")
|
||||
if len(self.samples) > MAX_SAMPLES:
|
||||
raise ValueError(
|
||||
f"too many samples ({len(self.samples)} > {MAX_SAMPLES})"
|
||||
)
|
||||
for s in self.samples:
|
||||
if not isinstance(s, SampleDiff):
|
||||
raise TypeError("samples entries must be SampleDiff")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Factory
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_adapter_pr(
|
||||
*,
|
||||
title: str,
|
||||
base_sha: str,
|
||||
adapter_path: str,
|
||||
dataset_diff: str,
|
||||
deltas: Iterable[Mapping[str, Any]],
|
||||
samples: Iterable[Mapping[str, Any]],
|
||||
) -> AdapterPR:
|
||||
"""Build a frozen ``AdapterPR`` from raw dicts (CLI / API friendly)."""
|
||||
delta_objs = []
|
||||
for raw in deltas:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise TypeError("each delta must be a mapping")
|
||||
delta_objs.append(
|
||||
EvalDelta(
|
||||
metric=raw.get("metric", ""),
|
||||
baseline=raw.get("baseline", 0.0),
|
||||
candidate=raw.get("candidate", 0.0),
|
||||
)
|
||||
)
|
||||
sample_objs = []
|
||||
for raw in samples:
|
||||
if not isinstance(raw, Mapping):
|
||||
raise TypeError("each sample must be a mapping")
|
||||
sample_objs.append(
|
||||
SampleDiff(
|
||||
prompt=raw.get("prompt", ""),
|
||||
baseline_output=raw.get("baseline_output", ""),
|
||||
candidate_output=raw.get("candidate_output", ""),
|
||||
)
|
||||
)
|
||||
return AdapterPR(
|
||||
title=title,
|
||||
base_sha=base_sha,
|
||||
adapter_path=adapter_path,
|
||||
dataset_diff=dataset_diff,
|
||||
deltas=tuple(delta_objs),
|
||||
samples=tuple(sample_objs),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Markdown rendering — escapes table-active and link-active characters
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _md_table_escape(text: str) -> str:
|
||||
"""Neutralise markdown table-active characters in a cell.
|
||||
|
||||
Mirrors v0.29.0 model card v2 + v0.59.0 Annex XI ``_md_escape``
|
||||
policy: a forged-heading or table-injection attack via crafted
|
||||
metric names / prompts must not leak into the rendered PR document.
|
||||
"""
|
||||
if not isinstance(text, str):
|
||||
return ""
|
||||
# Order matters: \\ first
|
||||
out = text.replace("\\", "\\\\")
|
||||
out = out.replace("|", "\\|")
|
||||
out = out.replace("\n", " ")
|
||||
out = out.replace("\r", " ")
|
||||
out = out.replace("\t", " ")
|
||||
return out
|
||||
|
||||
|
||||
def _md_body_escape(text: str) -> str:
|
||||
"""Lighter escape for code-fence body: only null bytes + CR."""
|
||||
if not isinstance(text, str):
|
||||
return ""
|
||||
return text.replace("\x00", "").replace("\r\n", "\n")
|
||||
|
||||
|
||||
def render_pr_markdown(pr: AdapterPR) -> str:
|
||||
"""Render an ``AdapterPR`` as a GitHub-style PR Markdown document."""
|
||||
if not isinstance(pr, AdapterPR):
|
||||
raise TypeError("pr must be AdapterPR")
|
||||
|
||||
lines: list[str] = []
|
||||
lines.append(f"# {_md_table_escape(pr.title)}")
|
||||
lines.append("")
|
||||
lines.append(f"**Base SHA:** `{pr.base_sha[:12]}…`")
|
||||
lines.append(f"**Adapter:** `{_md_table_escape(pr.adapter_path)}`")
|
||||
lines.append("")
|
||||
|
||||
# Eval deltas table
|
||||
if pr.deltas:
|
||||
lines.append("## Eval deltas")
|
||||
lines.append("")
|
||||
lines.append("| Metric | Baseline | Candidate | Δ |")
|
||||
lines.append("|---|---|---|---|")
|
||||
for d in pr.deltas:
|
||||
metric = _md_table_escape(d.metric)
|
||||
sign = "+" if d.delta >= 0 else ""
|
||||
lines.append(
|
||||
f"| {metric} | {d.baseline:.4f} | {d.candidate:.4f} | "
|
||||
f"{sign}{d.delta:.4f} |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# Dataset diff
|
||||
if pr.dataset_diff:
|
||||
lines.append("## Dataset diff")
|
||||
lines.append("")
|
||||
lines.append("```diff")
|
||||
lines.append(_md_body_escape(pr.dataset_diff))
|
||||
lines.append("```")
|
||||
lines.append("")
|
||||
|
||||
# Sample diffs
|
||||
if pr.samples:
|
||||
lines.append("## Sample diffs")
|
||||
lines.append("")
|
||||
for idx, s in enumerate(pr.samples, 1):
|
||||
lines.append(f"### Sample {idx}")
|
||||
lines.append("")
|
||||
lines.append("**Prompt:**")
|
||||
lines.append("")
|
||||
lines.append("```")
|
||||
lines.append(_md_body_escape(s.prompt))
|
||||
lines.append("```")
|
||||
lines.append("")
|
||||
lines.append("**Baseline:**")
|
||||
lines.append("")
|
||||
lines.append("```")
|
||||
lines.append(_md_body_escape(s.baseline_output))
|
||||
lines.append("```")
|
||||
lines.append("")
|
||||
lines.append("**Candidate:**")
|
||||
lines.append("")
|
||||
lines.append("```")
|
||||
lines.append(_md_body_escape(s.candidate_output))
|
||||
lines.append("```")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def render_pr_json(pr: AdapterPR) -> str:
|
||||
"""Render an ``AdapterPR`` as JSON (for downstream API consumers)."""
|
||||
if not isinstance(pr, AdapterPR):
|
||||
raise TypeError("pr must be AdapterPR")
|
||||
data = {
|
||||
"title": pr.title,
|
||||
"base_sha": pr.base_sha,
|
||||
"adapter_path": pr.adapter_path,
|
||||
"dataset_diff": pr.dataset_diff,
|
||||
"deltas": [
|
||||
{
|
||||
"metric": d.metric,
|
||||
"baseline": d.baseline,
|
||||
"candidate": d.candidate,
|
||||
"delta": d.delta,
|
||||
}
|
||||
for d in pr.deltas
|
||||
],
|
||||
"samples": [asdict(s) for s in pr.samples],
|
||||
}
|
||||
return json.dumps(data, indent=2, sort_keys=True, allow_nan=False)
|
||||
|
||||
|
||||
def write_pr_markdown(pr: AdapterPR, path: str) -> str:
|
||||
"""Atomic cwd-contained write of the rendered PR markdown."""
|
||||
if not isinstance(pr, AdapterPR):
|
||||
raise TypeError("pr must be AdapterPR")
|
||||
text = render_pr_markdown(pr)
|
||||
return atomic_write_text(text, path, field="pr markdown path")
|
||||
|
|
@ -0,0 +1,459 @@
|
|||
"""Evolutionary CMA-ES merge for LoRA adapters (v0.67.0 Part A).
|
||||
|
||||
CMA-ES over merge weights driven by the operator's eval. Extends v0.57.0
|
||||
``soup adapters merge`` with a ``cmaes`` strategy that searches the
|
||||
N-dimensional simplex of mixing weights to maximise an operator-supplied
|
||||
eval score.
|
||||
|
||||
Pure Python; no `cma` dependency. The implementation is a small simplex-
|
||||
projecting CMA-ES (rank-mu + diagonal covariance) sufficient for ≤16
|
||||
adapters and 1–100 generations. Operators wanting full CMA-ES (BIPOP,
|
||||
restart strategies) can plug their own optimiser via the ``eval_fn``
|
||||
hook; this module's contract is "given eval_fn, run a budgeted search,
|
||||
return the best simplex weights".
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``CmaesPlan`` + ``CmaesResult`` frozen dataclasses
|
||||
- ``validate_population_size`` / ``validate_generations``
|
||||
- ``build_cmaes_plan(...)`` returns frozen ``CmaesPlan``
|
||||
- ``run_cmaes_merge(plan, *, eval_fn, ...)`` returns ``CmaesResult``
|
||||
|
||||
Reuses ``parse_budget`` from v0.57.0 ``blame.py`` (60s..24h bounds).
|
||||
|
||||
Design notes:
|
||||
|
||||
- Output weights live on the simplex (sum=1, each ≥0); the optimiser
|
||||
parameterises N-1 logits and softmaxes them so any candidate is valid.
|
||||
- Failures inside ``eval_fn`` are swallowed as a sentinel low score so
|
||||
one broken adapter doesn't crash the run (mirrors v0.40.3 #33 / v0.48
|
||||
proxy-failure isolation).
|
||||
- Live ``soup eval`` wiring is operator-supplied — `cmaes_merge` does
|
||||
NOT auto-load models. Callers wrap their eval suite as a closure.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable, Optional, Sequence, Tuple
|
||||
|
||||
from soup_cli.utils.blame import parse_budget
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bounds (closed, locked at module load)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MIN_POPULATION = 2
|
||||
MAX_POPULATION = 256
|
||||
MIN_GENERATIONS = 1
|
||||
MAX_GENERATIONS = 10_000
|
||||
|
||||
_MIN_ADAPTERS = 2
|
||||
_MAX_ADAPTERS = 16 # mirrors v0.57.0 adapter_merge cap
|
||||
_SIMPLEX_TOL = 1e-6
|
||||
_FAILED_EVAL_SENTINEL = -1.0e9 # very negative so failed candidates never win
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_population_size(value: object) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise TypeError("population_size must be int")
|
||||
if value < MIN_POPULATION:
|
||||
raise ValueError(
|
||||
f"population_size {value} below floor {MIN_POPULATION}"
|
||||
)
|
||||
if value > MAX_POPULATION:
|
||||
raise ValueError(
|
||||
f"population_size {value} above cap {MAX_POPULATION}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_generations(value: object) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise TypeError("max_generations must be int")
|
||||
if value < MIN_GENERATIONS:
|
||||
raise ValueError(
|
||||
f"max_generations {value} below floor {MIN_GENERATIONS}"
|
||||
)
|
||||
if value > MAX_GENERATIONS:
|
||||
raise ValueError(
|
||||
f"max_generations {value} above cap {MAX_GENERATIONS}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _validate_seed(seed: object) -> int:
|
||||
if isinstance(seed, bool) or not isinstance(seed, int):
|
||||
raise TypeError("seed must be int")
|
||||
if seed < 0:
|
||||
raise ValueError("seed must be non-negative")
|
||||
if seed > 2**31 - 1:
|
||||
raise ValueError("seed too large")
|
||||
return seed
|
||||
|
||||
|
||||
def _validate_finite_score(value: object, field: str) -> float:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(f"{field} must be numeric")
|
||||
val = float(value)
|
||||
if not math.isfinite(val):
|
||||
raise ValueError(f"{field} must be finite")
|
||||
return val
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frozen dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CmaesPlan:
|
||||
"""Plan for an evolutionary merge run.
|
||||
|
||||
Reuses v0.57 ``parse_budget`` so the budget bounds (60s..24h) are
|
||||
consistent across blame / cmaes.
|
||||
"""
|
||||
|
||||
adapters: Tuple[str, ...]
|
||||
eval_suite: str
|
||||
budget_seconds: int
|
||||
population_size: int
|
||||
max_generations: int
|
||||
seed: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.adapters, tuple):
|
||||
raise TypeError("adapters must be tuple")
|
||||
if len(self.adapters) < _MIN_ADAPTERS:
|
||||
raise ValueError(
|
||||
f"need at least {_MIN_ADAPTERS} adapters"
|
||||
)
|
||||
if len(self.adapters) > _MAX_ADAPTERS:
|
||||
raise ValueError(f"at most {_MAX_ADAPTERS} adapters")
|
||||
for path in self.adapters:
|
||||
if not isinstance(path, str) or not path:
|
||||
raise ValueError("adapters entries must be non-empty str")
|
||||
if not isinstance(self.eval_suite, str) or not self.eval_suite:
|
||||
raise ValueError("eval_suite must be non-empty str")
|
||||
validate_population_size(self.population_size)
|
||||
validate_generations(self.max_generations)
|
||||
_validate_seed(self.seed)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CmaesResult:
|
||||
"""Result of an evolutionary merge run.
|
||||
|
||||
``best_weights`` live on the simplex (sum=1, each ≥0).
|
||||
"""
|
||||
|
||||
best_weights: Tuple[float, ...]
|
||||
best_score: float
|
||||
generations_run: int
|
||||
evaluations: int
|
||||
wall_clock_seconds: float
|
||||
converged: bool
|
||||
history: Tuple[float, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Score finite + bool-rejected
|
||||
_validate_finite_score(self.best_score, "best_score")
|
||||
if not isinstance(self.generations_run, int) or isinstance(
|
||||
self.generations_run, bool
|
||||
):
|
||||
raise TypeError("generations_run must be int")
|
||||
if self.generations_run < 0:
|
||||
raise ValueError("generations_run must be non-negative")
|
||||
if not isinstance(self.evaluations, int) or isinstance(
|
||||
self.evaluations, bool
|
||||
):
|
||||
raise TypeError("evaluations must be int")
|
||||
if self.evaluations < 0:
|
||||
raise ValueError("evaluations must be non-negative")
|
||||
_validate_finite_score(self.wall_clock_seconds, "wall_clock_seconds")
|
||||
if self.wall_clock_seconds < 0:
|
||||
raise ValueError("wall_clock_seconds must be non-negative")
|
||||
if not isinstance(self.converged, bool):
|
||||
raise TypeError("converged must be bool")
|
||||
if not isinstance(self.best_weights, tuple):
|
||||
raise TypeError("best_weights must be tuple")
|
||||
if not self.best_weights:
|
||||
raise ValueError("best_weights must be non-empty")
|
||||
for w in self.best_weights:
|
||||
if isinstance(w, bool) or not isinstance(w, (int, float)):
|
||||
raise TypeError("best_weights entries must be numeric")
|
||||
wf = float(w)
|
||||
if not math.isfinite(wf):
|
||||
raise ValueError("best_weights entries must be finite")
|
||||
if wf < 0:
|
||||
raise ValueError("best_weights entries must be non-negative")
|
||||
total = sum(float(w) for w in self.best_weights)
|
||||
if not math.isclose(total, 1.0, abs_tol=_SIMPLEX_TOL):
|
||||
raise ValueError(
|
||||
f"best_weights must sum to 1.0 (got {total:.6f})"
|
||||
)
|
||||
if not isinstance(self.history, tuple):
|
||||
raise TypeError("history must be tuple")
|
||||
for h in self.history:
|
||||
if isinstance(h, bool) or not isinstance(h, (int, float)):
|
||||
raise TypeError("history entries must be numeric")
|
||||
if not math.isfinite(float(h)):
|
||||
raise ValueError("history entries must be finite")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Plan construction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_cmaes_plan(
|
||||
*,
|
||||
adapters: Sequence[str],
|
||||
eval_suite: str,
|
||||
budget_spec: str,
|
||||
population_size: int = 8,
|
||||
max_generations: int = 20,
|
||||
seed: int = 0,
|
||||
) -> CmaesPlan:
|
||||
"""Validate inputs and return a frozen ``CmaesPlan``.
|
||||
|
||||
- ``adapters`` must contain ≥2 unique paths, all under cwd.
|
||||
- ``eval_suite`` must be a real path under cwd (no symlinks).
|
||||
- ``budget_spec`` is parsed via v0.57 ``parse_budget`` (60s..24h).
|
||||
"""
|
||||
if not isinstance(adapters, Sequence) or isinstance(adapters, str):
|
||||
raise TypeError("adapters must be a sequence")
|
||||
if len(adapters) < _MIN_ADAPTERS:
|
||||
raise ValueError(f"need at least {_MIN_ADAPTERS} adapters")
|
||||
validated_adapters: list[str] = []
|
||||
for ad in adapters:
|
||||
if not isinstance(ad, str) or not ad:
|
||||
raise ValueError("each adapter must be a non-empty str")
|
||||
enforce_under_cwd_and_no_symlink(ad, field="adapter")
|
||||
validated_adapters.append(os.path.realpath(ad))
|
||||
|
||||
enforce_under_cwd_and_no_symlink(eval_suite, field="eval_suite")
|
||||
if not os.path.exists(eval_suite):
|
||||
raise FileNotFoundError(f"eval_suite not found: {eval_suite!r}")
|
||||
|
||||
budget_seconds = parse_budget(budget_spec)
|
||||
|
||||
return CmaesPlan(
|
||||
adapters=tuple(validated_adapters),
|
||||
eval_suite=os.path.realpath(eval_suite),
|
||||
budget_seconds=budget_seconds,
|
||||
population_size=validate_population_size(population_size),
|
||||
max_generations=validate_generations(max_generations),
|
||||
seed=_validate_seed(seed),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Optimiser (minimal CMA-ES on the N-1 logit space, softmaxed onto simplex)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _softmax(logits: Sequence[float]) -> Tuple[float, ...]:
|
||||
"""Numerically-stable softmax onto the simplex."""
|
||||
m = max(logits)
|
||||
exps = [math.exp(x - m) for x in logits]
|
||||
z = sum(exps)
|
||||
if z <= 0:
|
||||
# Degenerate fallback: uniform
|
||||
return tuple(1.0 / len(logits) for _ in logits)
|
||||
return tuple(e / z for e in exps)
|
||||
|
||||
|
||||
def _eval_safely(
|
||||
eval_fn: Callable[[Tuple[float, ...]], float],
|
||||
weights: Tuple[float, ...],
|
||||
) -> float:
|
||||
"""Call ``eval_fn`` with sentinel-low score on exception.
|
||||
|
||||
Failure isolation mirrors v0.40.3 #33 / v0.48 / v0.53.7 #106 policy:
|
||||
one bad eval must not crash the whole run.
|
||||
"""
|
||||
try:
|
||||
score = eval_fn(weights)
|
||||
except (KeyboardInterrupt, SystemExit):
|
||||
raise
|
||||
except Exception: # noqa: BLE001 — eval_fn surface is operator-controlled
|
||||
return _FAILED_EVAL_SENTINEL
|
||||
if isinstance(score, bool):
|
||||
return _FAILED_EVAL_SENTINEL
|
||||
if not isinstance(score, (int, float)):
|
||||
return _FAILED_EVAL_SENTINEL
|
||||
s = float(score)
|
||||
if not math.isfinite(s):
|
||||
return _FAILED_EVAL_SENTINEL
|
||||
return s
|
||||
|
||||
|
||||
def run_cmaes_merge(
|
||||
plan: CmaesPlan,
|
||||
*,
|
||||
eval_fn: Callable[[Tuple[float, ...]], float],
|
||||
sigma_init: float = 0.5,
|
||||
elite_frac: float = 0.5,
|
||||
convergence_tol: float = 1e-4,
|
||||
) -> CmaesResult:
|
||||
"""Run a small CMA-ES-style search over simplex merge weights.
|
||||
|
||||
Operator supplies ``eval_fn(weights) -> score`` (higher is better);
|
||||
we softmax N-1 logits onto the simplex, sample a population, keep
|
||||
the elite half, re-fit a diagonal Gaussian, repeat until either:
|
||||
|
||||
- ``max_generations`` reached, or
|
||||
- elapsed wall-clock ≥ ``budget_seconds``, or
|
||||
- score plateau < ``convergence_tol`` for 3 generations in a row.
|
||||
"""
|
||||
if not isinstance(plan, CmaesPlan):
|
||||
raise TypeError("plan must be CmaesPlan")
|
||||
if eval_fn is None or not callable(eval_fn):
|
||||
raise TypeError("eval_fn must be callable")
|
||||
if isinstance(sigma_init, bool) or not isinstance(sigma_init, (int, float)):
|
||||
raise TypeError("sigma_init must be numeric")
|
||||
if not math.isfinite(float(sigma_init)) or float(sigma_init) <= 0:
|
||||
raise ValueError("sigma_init must be positive and finite")
|
||||
if isinstance(elite_frac, bool) or not isinstance(elite_frac, (int, float)):
|
||||
raise TypeError("elite_frac must be numeric")
|
||||
elite = float(elite_frac)
|
||||
if not (0.0 < elite < 1.0) or not math.isfinite(elite):
|
||||
raise ValueError("elite_frac must be in (0, 1)")
|
||||
|
||||
n_dim = len(plan.adapters) - 1 # softmax over N-1 free logits
|
||||
rng = _LcgRng(plan.seed)
|
||||
|
||||
mean = [0.0] * n_dim
|
||||
sigma = [float(sigma_init)] * n_dim
|
||||
elite_count = max(1, int(plan.population_size * elite))
|
||||
|
||||
history: list[float] = []
|
||||
best_score = -math.inf
|
||||
best_weights: Tuple[float, ...] = tuple(
|
||||
1.0 / len(plan.adapters) for _ in plan.adapters
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
generations_run = 0
|
||||
evaluations = 0
|
||||
converged = False
|
||||
plateau_run = 0
|
||||
|
||||
for gen in range(plan.max_generations):
|
||||
if (time.monotonic() - start) >= plan.budget_seconds:
|
||||
break
|
||||
|
||||
# Sample population
|
||||
candidates: list[Tuple[list[float], Tuple[float, ...], float]] = []
|
||||
for _ in range(plan.population_size):
|
||||
sample = [
|
||||
mean[i] + sigma[i] * rng.normal() for i in range(n_dim)
|
||||
]
|
||||
# Append 0.0 reference logit (softmax is shift-invariant) so we
|
||||
# span the full simplex.
|
||||
logits = sample + [0.0]
|
||||
weights = _softmax(logits)
|
||||
score = _eval_safely(eval_fn, weights)
|
||||
evaluations += 1
|
||||
candidates.append((sample, weights, score))
|
||||
if score > best_score:
|
||||
best_score = score
|
||||
best_weights = weights
|
||||
if (time.monotonic() - start) >= plan.budget_seconds:
|
||||
break
|
||||
|
||||
generations_run += 1
|
||||
# Pick elite by score (descending)
|
||||
candidates.sort(key=lambda c: c[2], reverse=True)
|
||||
elite_samples = [c[0] for c in candidates[:elite_count]]
|
||||
# Recompute mean + sigma per dim from elite
|
||||
new_mean = [
|
||||
sum(s[i] for s in elite_samples) / len(elite_samples)
|
||||
for i in range(n_dim)
|
||||
]
|
||||
new_sigma = []
|
||||
for i in range(n_dim):
|
||||
var = sum(
|
||||
(s[i] - new_mean[i]) ** 2 for s in elite_samples
|
||||
) / len(elite_samples)
|
||||
# Clamp to a sensible floor; CMA-ES typically blends with prior
|
||||
# but for ≤16-adapter case the simple rank-mu update converges.
|
||||
new_sigma.append(max(math.sqrt(var), 1e-4))
|
||||
# Plateau detection
|
||||
gen_best = candidates[0][2]
|
||||
history.append(gen_best)
|
||||
if len(history) >= 2:
|
||||
delta = abs(history[-1] - history[-2])
|
||||
if delta < convergence_tol:
|
||||
plateau_run += 1
|
||||
else:
|
||||
plateau_run = 0
|
||||
if plateau_run >= 3:
|
||||
converged = True
|
||||
break
|
||||
mean = new_mean
|
||||
sigma = new_sigma
|
||||
|
||||
wall_clock = time.monotonic() - start
|
||||
if best_score == -math.inf:
|
||||
# Defensive — should not happen because we always sample at least once
|
||||
best_score = 0.0
|
||||
|
||||
return CmaesResult(
|
||||
best_weights=tuple(best_weights),
|
||||
best_score=float(best_score),
|
||||
generations_run=generations_run,
|
||||
evaluations=evaluations,
|
||||
wall_clock_seconds=float(wall_clock),
|
||||
converged=converged,
|
||||
history=tuple(history),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deterministic RNG (no numpy dependency at module top)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _LcgRng:
|
||||
"""Tiny deterministic linear-congruential RNG + Box-Muller normals.
|
||||
|
||||
Avoids a numpy import at module top (matches v0.66 review-grep policy
|
||||
— heavy numpy stays inside live callsites only). Seeded so two runs
|
||||
with the same ``seed`` produce identical trajectories.
|
||||
"""
|
||||
|
||||
def __init__(self, seed: int) -> None:
|
||||
self._state = (seed * 2654435761 + 1) & 0xFFFFFFFFFFFFFFFF
|
||||
self._pending: Optional[float] = None
|
||||
|
||||
def _uniform(self) -> float:
|
||||
# 64-bit LCG (Numerical Recipes-style)
|
||||
self._state = (self._state * 6364136223846793005 + 1442695040888963407) & 0xFFFFFFFFFFFFFFFF
|
||||
# Take the top 53 bits to fit a double's mantissa
|
||||
return ((self._state >> 11) / float(1 << 53))
|
||||
|
||||
def normal(self) -> float:
|
||||
"""Box-Muller standard normal."""
|
||||
if self._pending is not None:
|
||||
val = self._pending
|
||||
self._pending = None
|
||||
return val
|
||||
u1 = max(self._uniform(), 1e-12)
|
||||
u2 = self._uniform()
|
||||
r = math.sqrt(-2.0 * math.log(u1))
|
||||
a = 2.0 * math.pi * u2
|
||||
self._pending = r * math.sin(a)
|
||||
return r * math.cos(a)
|
||||
|
|
@ -0,0 +1,188 @@
|
|||
"""MoLE per-token adapter routing (v0.67.0 Part C).
|
||||
|
||||
A gating network that routes per-token activations to one of N task
|
||||
LoRAs. Following the MoLE paper (Mixture of LoRA Experts), inference-
|
||||
time dispatch uses softmax gating over the per-token hidden state to
|
||||
select top-K LoRAs and blend them by gating logits.
|
||||
|
||||
v0.67.0 ships the schema + cross-validator. The live gating-kernel
|
||||
training + serving-time dispatch is deferred to v0.67.1 (mirrors
|
||||
v0.27.0 MII / v0.50.0 GRPO Plus / v0.62.0 steering stub-then-live).
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``MoleGatingConfig`` frozen dataclass
|
||||
- ``validate_mole_compat(task, backend, num_task_adapters)``
|
||||
- ``build_gating_kernel(config)`` deferred-live stub
|
||||
- New ``task='moe_lora_routing'`` Literal on ``SoupConfig.task``
|
||||
|
||||
Design notes:
|
||||
|
||||
- ``num_task_adapters`` bounded ``[2, 64]`` — beyond that, per-token
|
||||
softmax becomes a bottleneck; operators wanting more should hierarchy
|
||||
the gating.
|
||||
- ``top_k <= num_task_adapters`` so sparse top-K dispatch is sane.
|
||||
- ``temperature > 0`` to keep softmax non-degenerate; finite-only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bounds (closed, locked at module load)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MIN_TASK_ADAPTERS = 2
|
||||
MAX_TASK_ADAPTERS = 64
|
||||
MIN_HIDDEN_DIM = 1
|
||||
MAX_HIDDEN_DIM = 16_384
|
||||
MIN_TEMPERATURE = 1e-6
|
||||
MAX_TEMPERATURE = 100.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _check_int(value: object, field: str, lo: int, hi: int) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise TypeError(f"{field} must be int")
|
||||
if value < lo:
|
||||
raise ValueError(f"{field} {value} below floor {lo}")
|
||||
if value > hi:
|
||||
raise ValueError(f"{field} {value} above cap {hi}")
|
||||
return value
|
||||
|
||||
|
||||
def _check_finite_positive(
|
||||
value: object, field: str, lo: float, hi: float
|
||||
) -> float:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(f"{field} must be numeric")
|
||||
val = float(value)
|
||||
if not math.isfinite(val):
|
||||
raise ValueError(f"{field} must be finite")
|
||||
if val < lo:
|
||||
raise ValueError(f"{field} {val} below floor {lo}")
|
||||
if val > hi:
|
||||
raise ValueError(f"{field} {val} above cap {hi}")
|
||||
return val
|
||||
|
||||
|
||||
def _check_str_field(value: object, field: str) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
return value
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frozen config
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MoleGatingConfig:
|
||||
"""Per-token gating network over N task LoRAs.
|
||||
|
||||
Frozen dataclass — post-construction mutation raises ``FrozenInstanceError``.
|
||||
"""
|
||||
|
||||
num_task_adapters: int
|
||||
hidden_dim: int
|
||||
temperature: float
|
||||
top_k: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_int(
|
||||
self.num_task_adapters,
|
||||
"num_task_adapters",
|
||||
MIN_TASK_ADAPTERS,
|
||||
MAX_TASK_ADAPTERS,
|
||||
)
|
||||
_check_int(
|
||||
self.hidden_dim, "hidden_dim", MIN_HIDDEN_DIM, MAX_HIDDEN_DIM
|
||||
)
|
||||
_check_finite_positive(
|
||||
self.temperature,
|
||||
"temperature",
|
||||
MIN_TEMPERATURE,
|
||||
MAX_TEMPERATURE,
|
||||
)
|
||||
# top_k must be positive and not exceed num_task_adapters
|
||||
_check_int(self.top_k, "top_k", 1, MAX_TASK_ADAPTERS)
|
||||
if self.top_k > self.num_task_adapters:
|
||||
raise ValueError(
|
||||
f"top_k {self.top_k} > num_task_adapters "
|
||||
f"{self.num_task_adapters}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cross-validator (called from SoupConfig + standalone)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_mole_compat(
|
||||
*,
|
||||
task: str,
|
||||
backend: str,
|
||||
num_task_adapters: int,
|
||||
) -> None:
|
||||
"""Schema-time gate.
|
||||
|
||||
- Requires ``task='moe_lora_routing'`` (silent-no-op footgun rejection
|
||||
matching v0.52.0 distill / v0.62.0 citation_faithful task-gates).
|
||||
- Rejects ``backend='mlx'`` — the gating kernel needs torch dispatch
|
||||
that mlx-lm doesn't expose (deferred to a future MLX integration).
|
||||
- ``num_task_adapters`` must be in ``[MIN_TASK_ADAPTERS, MAX_TASK_ADAPTERS]``.
|
||||
"""
|
||||
_check_str_field(task, "task")
|
||||
_check_str_field(backend, "backend")
|
||||
if task != "moe_lora_routing":
|
||||
raise ValueError(
|
||||
f"validate_mole_compat: task must be 'moe_lora_routing' "
|
||||
f"(got {task!r})"
|
||||
)
|
||||
if backend == "mlx":
|
||||
raise ValueError(
|
||||
"MoLE routing is not supported on the mlx backend "
|
||||
"(live wiring deferred; see v0.67.1)"
|
||||
)
|
||||
_check_int(
|
||||
num_task_adapters,
|
||||
"num_task_adapters",
|
||||
MIN_TASK_ADAPTERS,
|
||||
MAX_TASK_ADAPTERS,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deferred-live stub
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_gating_kernel(config: MoleGatingConfig):
|
||||
"""Build a per-token gating kernel for MoLE dispatch.
|
||||
|
||||
Deferred to v0.67.1: live wiring requires the v0.22.0 multi-adapter
|
||||
serving surface plus a torch gating module that emits softmax
|
||||
routing logits per token. The schema (this module) ships now so
|
||||
operators can wire ``num_task_adapters`` / ``top_k`` / ``temperature``
|
||||
into their config; the live kernel comes next.
|
||||
"""
|
||||
if not isinstance(config, MoleGatingConfig):
|
||||
raise TypeError("config must be MoleGatingConfig")
|
||||
raise NotImplementedError(
|
||||
"build_gating_kernel live wiring deferred to v0.67.1"
|
||||
)
|
||||
|
|
@ -0,0 +1,218 @@
|
|||
"""``soup.lock`` shared run lockfile (v0.67.0 Part E).
|
||||
|
||||
A ``soup.lock`` is the closure of three SHA-256 hashes:
|
||||
|
||||
closure = sha256(base_model_sha || dataset_sha || env_hash)
|
||||
|
||||
Committed to git alongside ``soup.yaml``, teams coordinate on
|
||||
"reproducible training run" by checking the closure on every
|
||||
``soup train`` and refusing to start when the lock drifts. Composes
|
||||
with v0.64 Part C ``soup env`` — operators run ``soup env lock``
|
||||
to get ``env_hash``, then ``soup lock write`` to write the file.
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``SoupLock`` frozen dataclass
|
||||
- ``LockDrift`` frozen dataclass (ok / changes tuple)
|
||||
- ``compute_lock_closure(*, base_model_sha, dataset_sha, env_hash)`` -> hex
|
||||
- ``write_lock(lock, path)`` / ``read_lock(path)`` atomic JSON I/O
|
||||
- ``check_lock_drift(expected, actual)`` -> ``LockDrift``
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Tuple
|
||||
|
||||
from soup_cli.utils.paths import (
|
||||
atomic_write_text,
|
||||
enforce_under_cwd_and_no_symlink,
|
||||
)
|
||||
|
||||
_SHA256_RE = re.compile(r"^[0-9a-f]{64}$")
|
||||
_MAX_FILE_BYTES = 1 * 1024 * 1024 # 1 MiB cap on lock file size
|
||||
_MAX_VERSION_LEN = 64
|
||||
_MAX_BASE_MODEL_LEN = 512
|
||||
_MAX_CREATED_AT_LEN = 64
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _check_sha(value: object, field: str) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str")
|
||||
if not _SHA256_RE.match(value):
|
||||
raise ValueError(f"{field} must be 64 hex chars (got {value!r})")
|
||||
return value
|
||||
|
||||
|
||||
def _check_str(value: object, field: str, max_len: int) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > max_len:
|
||||
raise ValueError(
|
||||
f"{field} length {len(value)} > {max_len}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# compute_lock_closure
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def compute_lock_closure(
|
||||
*,
|
||||
base_model_sha: str,
|
||||
dataset_sha: str,
|
||||
env_hash: str,
|
||||
) -> str:
|
||||
"""Return the 64-hex closure SHA over the three input hashes.
|
||||
|
||||
Each input must be a 64-hex SHA-256 itself. The closure is
|
||||
``sha256(base_model_sha || dataset_sha || env_hash)``.
|
||||
"""
|
||||
_check_sha(base_model_sha, "base_model_sha")
|
||||
_check_sha(dataset_sha, "dataset_sha")
|
||||
_check_sha(env_hash, "env_hash")
|
||||
h = hashlib.sha256()
|
||||
h.update(base_model_sha.encode("ascii"))
|
||||
h.update(dataset_sha.encode("ascii"))
|
||||
h.update(env_hash.encode("ascii"))
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frozen dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SoupLock:
|
||||
"""The on-disk lockfile shape.
|
||||
|
||||
Stored as JSON at ``soup.lock`` (or operator-named path); committed
|
||||
to git so the team coordinates on "reproducible training run".
|
||||
"""
|
||||
|
||||
soup_version: str
|
||||
base_model: str
|
||||
base_model_sha: str
|
||||
dataset_sha: str
|
||||
env_hash: str
|
||||
closure_sha: str
|
||||
created_at: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_str(self.soup_version, "soup_version", _MAX_VERSION_LEN)
|
||||
_check_str(self.base_model, "base_model", _MAX_BASE_MODEL_LEN)
|
||||
_check_sha(self.base_model_sha, "base_model_sha")
|
||||
_check_sha(self.dataset_sha, "dataset_sha")
|
||||
_check_sha(self.env_hash, "env_hash")
|
||||
_check_sha(self.closure_sha, "closure_sha")
|
||||
_check_str(self.created_at, "created_at", _MAX_CREATED_AT_LEN)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LockDrift:
|
||||
"""Result of comparing two locks. ``ok`` is True when no field changed."""
|
||||
|
||||
ok: bool
|
||||
changes: Tuple[str, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.ok, bool):
|
||||
raise TypeError("ok must be bool")
|
||||
if not isinstance(self.changes, tuple):
|
||||
raise TypeError("changes must be tuple")
|
||||
for c in self.changes:
|
||||
if not isinstance(c, str):
|
||||
raise TypeError("changes entries must be str")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Atomic I/O
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_lock(lock: SoupLock, path: str) -> str:
|
||||
"""Atomically write a lock to JSON under cwd containment."""
|
||||
if not isinstance(lock, SoupLock):
|
||||
raise TypeError("lock must be SoupLock")
|
||||
text = json.dumps(asdict(lock), indent=2, sort_keys=True)
|
||||
return atomic_write_text(text, path, field="lock path")
|
||||
|
||||
|
||||
def read_lock(path: str) -> SoupLock:
|
||||
"""Read + validate a soup.lock from JSON. cwd-contained, symlink-rejected."""
|
||||
if not isinstance(path, str):
|
||||
raise TypeError("path must be str")
|
||||
if not path:
|
||||
raise ValueError("path must be non-empty")
|
||||
if "\x00" in path:
|
||||
raise ValueError("path must not contain null bytes")
|
||||
enforce_under_cwd_and_no_symlink(path, field="lock path")
|
||||
real = os.path.realpath(path)
|
||||
if not os.path.exists(real):
|
||||
raise FileNotFoundError(f"lock file not found: {path!r}")
|
||||
st = os.lstat(real)
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError("lock path must not be a symlink (TOCTOU defence)")
|
||||
if st.st_size > _MAX_FILE_BYTES:
|
||||
raise ValueError(
|
||||
f"lock file size {st.st_size} > {_MAX_FILE_BYTES}"
|
||||
)
|
||||
with open(real, encoding="utf-8") as fh:
|
||||
data = json.load(fh)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("lock file root must be JSON object")
|
||||
# SoupLock __post_init__ does the rest of the validation
|
||||
return SoupLock(
|
||||
soup_version=data.get("soup_version", ""),
|
||||
base_model=data.get("base_model", ""),
|
||||
base_model_sha=data.get("base_model_sha", ""),
|
||||
dataset_sha=data.get("dataset_sha", ""),
|
||||
env_hash=data.get("env_hash", ""),
|
||||
closure_sha=data.get("closure_sha", ""),
|
||||
created_at=data.get("created_at", ""),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Drift detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def check_lock_drift(expected: SoupLock, actual: SoupLock) -> LockDrift:
|
||||
"""Compare two locks field-by-field. Returns a ``LockDrift`` report.
|
||||
|
||||
Only the four content-bearing SHA fields + base_model name affect
|
||||
drift detection. ``soup_version`` / ``created_at`` differences are
|
||||
tracked separately as advisory-only (operators upgrading Soup will
|
||||
legitimately see them change).
|
||||
"""
|
||||
if not isinstance(expected, SoupLock):
|
||||
raise TypeError("expected must be SoupLock")
|
||||
if not isinstance(actual, SoupLock):
|
||||
raise TypeError("actual must be SoupLock")
|
||||
changes: list[str] = []
|
||||
for field in ("base_model", "base_model_sha", "dataset_sha", "env_hash", "closure_sha"):
|
||||
if getattr(expected, field) != getattr(actual, field):
|
||||
changes.append(f"{field} drifted")
|
||||
return LockDrift(ok=not changes, changes=tuple(changes))
|
||||
|
|
@ -0,0 +1,359 @@
|
|||
"""VeRA / VB-LoRA vector-bank storage format (v0.67.0 Part B).
|
||||
|
||||
A vector bank is the data structure behind multi-tenant LoRA serving at
|
||||
MB-per-user instead of hundreds-of-MB per LoRA:
|
||||
|
||||
Bank = {
|
||||
shared random projection matrix P : (d_model × d_model),
|
||||
per-user scaling vector v_u : (vector_dim,),
|
||||
}
|
||||
|
||||
The per-user delta at inference time is ``v_u ⊙ Px`` (VeRA) or a
|
||||
codebook lookup (VB-LoRA). Storage size is dominated by ``vector_dim``
|
||||
per user — a 128-D vector at fp32 is 512 bytes vs ~30 MB for a rank-16
|
||||
LoRA on a 7B model.
|
||||
|
||||
v0.67.0 ships the schema + atomic disk I/O + validators.
|
||||
Live wiring into multi-adapter serving (v0.22.0 surface) is the
|
||||
v0.67.1 deliverable — ``apply_bank_to_serve`` raises
|
||||
``NotImplementedError`` with explicit marker (matches v0.27.0 MII /
|
||||
v0.50.0 GRPO Plus stub-then-live cadence).
|
||||
|
||||
Public surface:
|
||||
|
||||
- ``VectorBank`` / ``BankEntry`` frozen dataclasses
|
||||
- ``validate_bank_name`` / ``validate_user_id`` / ``validate_scaling_vector``
|
||||
- ``estimate_bank_size(num_users, vector_dim)`` -> bytes
|
||||
- ``write_bank(bank, path)`` / ``load_bank(path)`` atomic JSON I/O
|
||||
- ``apply_bank_to_serve(bank)`` deferred-live stub
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Iterable, Tuple
|
||||
|
||||
from soup_cli.utils.paths import (
|
||||
atomic_write_text,
|
||||
enforce_under_cwd_and_no_symlink,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bounds (closed, locked at module load)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MAX_NAME_LEN = 128
|
||||
MAX_USER_ID_LEN = 256
|
||||
MAX_BASE_MODEL_LEN = 512
|
||||
MAX_VECTOR_DIM = 16_384
|
||||
MIN_VECTOR_DIM = 1
|
||||
MAX_ENTRIES_PER_BANK = 1_000_000
|
||||
_MAX_FILE_BYTES = 16 * 1024 * 1024 # 16 MiB cap on bank file size
|
||||
|
||||
# kebab-case + `._-`, leading alnum, no path separators / shell metacharacters
|
||||
_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._\-]{0,127}$")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_bank_name(name: object) -> str:
|
||||
"""Canonical kebab-case bank name (case-insensitive)."""
|
||||
if isinstance(name, bool):
|
||||
raise TypeError("name must not be bool")
|
||||
if not isinstance(name, str):
|
||||
raise TypeError(
|
||||
f"name must be str, got {type(name).__name__}"
|
||||
)
|
||||
if not name:
|
||||
raise ValueError("name must be non-empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("name must not contain null bytes")
|
||||
if len(name) > MAX_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"name length {len(name)} > {MAX_NAME_LEN}"
|
||||
)
|
||||
canonical = name.lower()
|
||||
if not _NAME_RE.match(canonical):
|
||||
raise ValueError(
|
||||
f"name must be kebab-case alphanumeric + `._-`, got {name!r}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def validate_user_id(user_id: object) -> str:
|
||||
if isinstance(user_id, bool):
|
||||
raise TypeError("user_id must not be bool")
|
||||
if not isinstance(user_id, str):
|
||||
raise TypeError("user_id must be str")
|
||||
if not user_id:
|
||||
raise ValueError("user_id must be non-empty")
|
||||
if "\x00" in user_id:
|
||||
raise ValueError("user_id must not contain null bytes")
|
||||
if len(user_id) > MAX_USER_ID_LEN:
|
||||
raise ValueError(
|
||||
f"user_id length {len(user_id)} > {MAX_USER_ID_LEN}"
|
||||
)
|
||||
return user_id
|
||||
|
||||
|
||||
def validate_scaling_vector(values: object) -> Tuple[float, ...]:
|
||||
"""Per-user scaling vector — non-empty, finite, ≤MAX_VECTOR_DIM."""
|
||||
if isinstance(values, (str, bytes)) or not isinstance(values, Iterable):
|
||||
raise TypeError("scaling vector must be a non-string iterable")
|
||||
out: list[float] = []
|
||||
for v in values:
|
||||
if isinstance(v, bool):
|
||||
raise TypeError("scaling vector entries must not be bool")
|
||||
if not isinstance(v, (int, float)):
|
||||
raise TypeError("scaling vector entries must be numeric")
|
||||
f = float(v)
|
||||
if not math.isfinite(f):
|
||||
raise ValueError("scaling vector entries must be finite")
|
||||
out.append(f)
|
||||
if not out:
|
||||
raise ValueError("scaling vector must be non-empty")
|
||||
if len(out) > MAX_VECTOR_DIM:
|
||||
raise ValueError(
|
||||
f"scaling vector length {len(out)} > {MAX_VECTOR_DIM}"
|
||||
)
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _validate_base_model(base_model: object) -> str:
|
||||
if isinstance(base_model, bool):
|
||||
raise TypeError("base_model must not be bool")
|
||||
if not isinstance(base_model, str):
|
||||
raise TypeError("base_model must be str")
|
||||
if not base_model:
|
||||
raise ValueError("base_model must be non-empty")
|
||||
if "\x00" in base_model:
|
||||
raise ValueError("base_model must not contain null bytes")
|
||||
if len(base_model) > MAX_BASE_MODEL_LEN:
|
||||
raise ValueError(
|
||||
f"base_model length {len(base_model)} > {MAX_BASE_MODEL_LEN}"
|
||||
)
|
||||
return base_model
|
||||
|
||||
|
||||
def _validate_seed(seed: object) -> int:
|
||||
if isinstance(seed, bool) or not isinstance(seed, int):
|
||||
raise TypeError("projection_seed must be int")
|
||||
if seed < 0:
|
||||
raise ValueError("projection_seed must be non-negative")
|
||||
if seed > 2**63 - 1:
|
||||
raise ValueError("projection_seed too large")
|
||||
return seed
|
||||
|
||||
|
||||
def _validate_vector_dim(dim: object) -> int:
|
||||
if isinstance(dim, bool) or not isinstance(dim, int):
|
||||
raise TypeError("vector_dim must be int")
|
||||
if dim < MIN_VECTOR_DIM:
|
||||
raise ValueError(
|
||||
f"vector_dim {dim} below floor {MIN_VECTOR_DIM}"
|
||||
)
|
||||
if dim > MAX_VECTOR_DIM:
|
||||
raise ValueError(
|
||||
f"vector_dim {dim} above cap {MAX_VECTOR_DIM}"
|
||||
)
|
||||
return dim
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frozen dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BankEntry:
|
||||
"""One per-user scaling vector in the bank."""
|
||||
|
||||
user_id: str
|
||||
scaling: Tuple[float, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Re-validate so callers constructing BankEntry directly cannot
|
||||
# smuggle in invalid state (matches v0.61 EditPlan policy).
|
||||
validate_user_id(self.user_id)
|
||||
if not isinstance(self.scaling, tuple):
|
||||
raise TypeError("scaling must be tuple")
|
||||
validate_scaling_vector(self.scaling)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VectorBank:
|
||||
"""A multi-tenant LoRA storage bank.
|
||||
|
||||
``projection_seed`` is the deterministic seed for the shared random
|
||||
projection P. Each ``BankEntry`` carries a per-user scaling vector
|
||||
of length ``vector_dim``.
|
||||
|
||||
Live serving wiring is deferred to v0.67.1. The v0.67.0 surface is
|
||||
the schema + atomic disk I/O.
|
||||
"""
|
||||
|
||||
name: str
|
||||
base_model: str
|
||||
projection_seed: int
|
||||
vector_dim: int
|
||||
entries: Tuple[BankEntry, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
validate_bank_name(self.name)
|
||||
_validate_base_model(self.base_model)
|
||||
_validate_seed(self.projection_seed)
|
||||
_validate_vector_dim(self.vector_dim)
|
||||
if not isinstance(self.entries, tuple):
|
||||
raise TypeError("entries must be tuple")
|
||||
if len(self.entries) > MAX_ENTRIES_PER_BANK:
|
||||
raise ValueError(
|
||||
f"entries length {len(self.entries)} > "
|
||||
f"{MAX_ENTRIES_PER_BANK}"
|
||||
)
|
||||
for entry in self.entries:
|
||||
if not isinstance(entry, BankEntry):
|
||||
raise TypeError("entries must be BankEntry instances")
|
||||
if len(entry.scaling) != self.vector_dim:
|
||||
raise ValueError(
|
||||
f"entry user_id={entry.user_id!r} scaling length "
|
||||
f"{len(entry.scaling)} != vector_dim "
|
||||
f"{self.vector_dim}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Size estimate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def estimate_bank_size(*, num_users: int, vector_dim: int) -> int:
|
||||
"""Estimate disk + memory cost of a bank in bytes.
|
||||
|
||||
The shared projection matrix dominates for small ``num_users``;
|
||||
per-user scaling dominates as the bank fills up. Both are fp32
|
||||
(4 bytes/element).
|
||||
"""
|
||||
if isinstance(num_users, bool) or not isinstance(num_users, int):
|
||||
raise TypeError("num_users must be int")
|
||||
if num_users < 0:
|
||||
raise ValueError("num_users must be non-negative")
|
||||
if isinstance(vector_dim, bool) or not isinstance(vector_dim, int):
|
||||
raise TypeError("vector_dim must be int")
|
||||
if vector_dim < 1:
|
||||
raise ValueError("vector_dim must be positive")
|
||||
projection_bytes = vector_dim * vector_dim * 4 # P : (d×d) fp32
|
||||
user_bytes = num_users * vector_dim * 4 # v_u : (d,) fp32
|
||||
return projection_bytes + user_bytes
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Atomic JSON I/O
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _bank_to_dict(bank: VectorBank) -> dict:
|
||||
return {
|
||||
"name": bank.name,
|
||||
"base_model": bank.base_model,
|
||||
"projection_seed": bank.projection_seed,
|
||||
"vector_dim": bank.vector_dim,
|
||||
"entries": [
|
||||
{"user_id": e.user_id, "scaling": list(e.scaling)}
|
||||
for e in bank.entries
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _bank_from_dict(data: object) -> VectorBank:
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("bank file root must be JSON object")
|
||||
name = data.get("name")
|
||||
base_model = data.get("base_model")
|
||||
seed = data.get("projection_seed")
|
||||
dim = data.get("vector_dim")
|
||||
raw_entries = data.get("entries", [])
|
||||
if not isinstance(raw_entries, list):
|
||||
raise ValueError("entries field must be a list")
|
||||
entries: list[BankEntry] = []
|
||||
for raw in raw_entries:
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError("each entry must be a JSON object")
|
||||
user_id = raw.get("user_id")
|
||||
scaling = raw.get("scaling")
|
||||
if not isinstance(scaling, list):
|
||||
raise ValueError("scaling must be a list")
|
||||
entries.append(
|
||||
BankEntry(
|
||||
user_id=user_id if isinstance(user_id, str) else "",
|
||||
scaling=tuple(validate_scaling_vector(scaling)),
|
||||
)
|
||||
)
|
||||
return VectorBank(
|
||||
name=name if isinstance(name, str) else "",
|
||||
base_model=base_model if isinstance(base_model, str) else "",
|
||||
projection_seed=seed if isinstance(seed, int) else 0,
|
||||
vector_dim=dim if isinstance(dim, int) else 0,
|
||||
entries=tuple(entries),
|
||||
)
|
||||
|
||||
|
||||
def write_bank(bank: VectorBank, path: str) -> str:
|
||||
"""Atomically write a bank to JSON under cwd containment."""
|
||||
if not isinstance(bank, VectorBank):
|
||||
raise TypeError("bank must be VectorBank")
|
||||
text = json.dumps(_bank_to_dict(bank), indent=2, sort_keys=True)
|
||||
return atomic_write_text(text, path, field="bank path")
|
||||
|
||||
|
||||
def load_bank(path: str) -> VectorBank:
|
||||
"""Read + validate a bank JSON. cwd-contained, symlink-rejected."""
|
||||
if not isinstance(path, str):
|
||||
raise TypeError("path must be str")
|
||||
if not path:
|
||||
raise ValueError("path must be non-empty")
|
||||
if "\x00" in path:
|
||||
raise ValueError("path must not contain null bytes")
|
||||
enforce_under_cwd_and_no_symlink(path, field="bank path")
|
||||
# Size cap defence (matches v0.55+ policy)
|
||||
real = os.path.realpath(path)
|
||||
if not os.path.exists(real):
|
||||
raise FileNotFoundError(f"bank file not found: {path!r}")
|
||||
st = os.lstat(real)
|
||||
if stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError("bank path must not be a symlink (TOCTOU defence)")
|
||||
if st.st_size > _MAX_FILE_BYTES:
|
||||
raise ValueError(
|
||||
f"bank file size {st.st_size} > {_MAX_FILE_BYTES}"
|
||||
)
|
||||
with open(real, encoding="utf-8") as fh:
|
||||
data = json.load(fh)
|
||||
return _bank_from_dict(data)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deferred-live stub: live serving wiring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def apply_bank_to_serve(bank: VectorBank, *, server: Any = None) -> None:
|
||||
"""Apply a vector bank to a running ``soup serve`` instance.
|
||||
|
||||
Deferred to v0.67.1 — live wiring requires the v0.22.0 multi-adapter
|
||||
serving surface plus a real torch path that materialises the shared
|
||||
projection matrix in GPU memory. The schema (this module) ships now
|
||||
so client code can target the API; the live engine integration is
|
||||
the v0.67.1 deliverable.
|
||||
"""
|
||||
if not isinstance(bank, VectorBank):
|
||||
raise TypeError("bank must be VectorBank")
|
||||
raise NotImplementedError(
|
||||
"apply_bank_to_serve live wiring deferred to v0.67.1"
|
||||
)
|
||||
|
|
@ -281,10 +281,13 @@ def test_merge_adapters_output_outside_cwd(tmp_path, monkeypatch):
|
|||
def test_supported_strategies_immutable():
|
||||
# v0.57.0 review fix: SUPPORTED_STRATEGIES is a frozenset (matches v0.41.0+
|
||||
# allowlist policy). STRATEGY_ORDER preserves canonical iteration order.
|
||||
# v0.67.0 Part A: floor-check widened to include "cmaes" (mirrors
|
||||
# v0.51.0 / v0.54.0 / v0.66.0 floor-check policy).
|
||||
from soup_cli.utils.adapter_merge import STRATEGY_ORDER
|
||||
assert SUPPORTED_STRATEGIES == frozenset({"linear", "ties", "dare", "svd"})
|
||||
assert {"linear", "ties", "dare", "svd"} <= SUPPORTED_STRATEGIES
|
||||
assert isinstance(SUPPORTED_STRATEGIES, frozenset)
|
||||
assert STRATEGY_ORDER == ("linear", "ties", "dare", "svd")
|
||||
for entry in ("linear", "ties", "dare", "svd"):
|
||||
assert entry in STRATEGY_ORDER
|
||||
|
||||
|
||||
def test_merge_ties_density_one_keeps_everything():
|
||||
|
|
|
|||
|
|
@ -0,0 +1,389 @@
|
|||
"""v0.67.0 review-fix follow-ups (from tdd-guide + security-review waves).
|
||||
|
||||
HIGH-priority gaps surfaced by the TDD review:
|
||||
|
||||
- Plateau-convergence path in CMA-ES
|
||||
- POSIX symlink rejection at write boundaries (vector_bank.write_bank,
|
||||
cmaes_merge has no disk surface, only the load-side has it today)
|
||||
- Vector-bank forward-compat: unknown JSON fields silently ignored
|
||||
- Adapter PR empty-sections rendering (no None leakage)
|
||||
- Bisect non-monotonic eval_fn does not crash
|
||||
- Exact-boundary tests at MIN/MAX values
|
||||
- validate_user_id bool rejection
|
||||
- soup.lock version-drift advisory check
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part A — Plateau convergence
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCmaesPlateauConvergence:
|
||||
def test_converges_on_flat_landscape(self, tmp_path, monkeypatch) -> None:
|
||||
"""When eval_fn returns the same score for 3+ generations,
|
||||
``converged=True`` and the loop short-circuits early."""
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
(tmp_path / "b").mkdir()
|
||||
suite = tmp_path / "s.yaml"
|
||||
suite.write_text("x: 1\n", encoding="utf-8")
|
||||
|
||||
plan = build_cmaes_plan(
|
||||
adapters=[str(tmp_path / "a"), str(tmp_path / "b")],
|
||||
eval_suite=str(suite),
|
||||
budget_spec="60s",
|
||||
population_size=4,
|
||||
max_generations=20,
|
||||
seed=42,
|
||||
)
|
||||
|
||||
# Flat constant landscape: every weight returns 0.5
|
||||
def eval_fn(weights):
|
||||
return 0.5
|
||||
|
||||
result = run_cmaes_merge(plan, eval_fn=eval_fn)
|
||||
assert result.converged is True
|
||||
# Plateau detection breaks out before max_generations
|
||||
assert result.generations_run < plan.max_generations
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part A — Exact boundary tests
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCmaesValidatorBoundaries:
|
||||
def test_population_min_accepted(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import MIN_POPULATION, validate_population_size
|
||||
|
||||
assert validate_population_size(MIN_POPULATION) == MIN_POPULATION
|
||||
|
||||
def test_population_min_minus_one_rejected(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import MIN_POPULATION, validate_population_size
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_population_size(MIN_POPULATION - 1)
|
||||
|
||||
def test_population_max_accepted(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import MAX_POPULATION, validate_population_size
|
||||
|
||||
assert validate_population_size(MAX_POPULATION) == MAX_POPULATION
|
||||
|
||||
def test_generations_min_accepted(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import MIN_GENERATIONS, validate_generations
|
||||
|
||||
assert validate_generations(MIN_GENERATIONS) == MIN_GENERATIONS
|
||||
|
||||
def test_generations_max_accepted(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import MAX_GENERATIONS, validate_generations
|
||||
|
||||
assert validate_generations(MAX_GENERATIONS) == MAX_GENERATIONS
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part B — Forward-compat: unknown fields silently ignored
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestVectorBankForwardCompat:
|
||||
def test_unknown_fields_ignored(self, tmp_path, monkeypatch) -> None:
|
||||
"""A bank JSON with extra unknown fields should round-trip without
|
||||
error — guards against breakage when v0.67.1+ adds new fields."""
|
||||
from soup_cli.utils.vector_bank import load_bank
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = tmp_path / "bank.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"name": "fc",
|
||||
"base_model": "m",
|
||||
"projection_seed": 0,
|
||||
"vector_dim": 2,
|
||||
"entries": [
|
||||
{"user_id": "u", "scaling": [0.1, 0.2]},
|
||||
],
|
||||
# Forward-compat: unknown field — must be tolerated
|
||||
"_future_field": 99,
|
||||
"unknown_key": "unknown_value",
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
loaded = load_bank(str(path))
|
||||
assert loaded.name == "fc"
|
||||
assert loaded.vector_dim == 2
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink test")
|
||||
def test_write_bank_symlink_rejected(self, tmp_path, monkeypatch) -> None:
|
||||
"""`write_bank` must reject a pre-placed symlink at the target path
|
||||
(TOCTOU defence — mirrors v0.55.0 / v0.56.0 policy)."""
|
||||
from soup_cli.utils.vector_bank import VectorBank, write_bank
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = tmp_path / "out.json"
|
||||
real_target = tmp_path / "real.json"
|
||||
real_target.write_text("{}", encoding="utf-8")
|
||||
os.symlink(real_target, target)
|
||||
|
||||
bank = VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=1,
|
||||
entries=(),
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
write_bank(bank, str(target))
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part B — validate_user_id bool rejection
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateUserIdBool:
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_user_id
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_user_id(True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part B — Exact boundaries
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestVectorBankBoundaries:
|
||||
def test_vector_dim_at_max_accepted(self) -> None:
|
||||
from soup_cli.utils.vector_bank import MAX_VECTOR_DIM, VectorBank
|
||||
|
||||
# MAX value should be accepted (use it with empty entries)
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=MAX_VECTOR_DIM,
|
||||
entries=(),
|
||||
)
|
||||
|
||||
def test_vector_dim_above_max_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import MAX_VECTOR_DIM, VectorBank
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=MAX_VECTOR_DIM + 1,
|
||||
entries=(),
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part D — Empty-sections PR renders without None leakage
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPRRenderEmptySections:
|
||||
def test_all_empty_sections(self) -> None:
|
||||
"""A PR with no deltas / no samples / no dataset_diff should
|
||||
render valid Markdown without any `None` literals leaking."""
|
||||
from soup_cli.utils.adapter_pr import AdapterPR, render_pr_markdown
|
||||
|
||||
pr = AdapterPR(
|
||||
title="empty-pr",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
md = render_pr_markdown(pr)
|
||||
assert "None" not in md
|
||||
assert "empty-pr" in md
|
||||
|
||||
def test_json_handles_empty_sections(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR, render_pr_json
|
||||
|
||||
pr = AdapterPR(
|
||||
title="t",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
data = json.loads(render_pr_json(pr))
|
||||
assert data["deltas"] == []
|
||||
assert data["samples"] == []
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part E — soup_version drift is advisory-only (not a drift signal)
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSoupLockVersionDriftAdvisory:
|
||||
def test_version_change_not_drift(self) -> None:
|
||||
"""``soup_version`` differing between expected and actual locks
|
||||
should NOT count as drift — operators upgrade Soup legitimately."""
|
||||
from soup_cli.utils.soup_lock import SoupLock, check_lock_drift
|
||||
|
||||
base = dict(
|
||||
base_model="m",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24",
|
||||
)
|
||||
expected = SoupLock(soup_version="0.67.0", **base)
|
||||
actual = SoupLock(soup_version="0.68.0", **base)
|
||||
|
||||
drift = check_lock_drift(expected, actual)
|
||||
assert drift.ok is True
|
||||
|
||||
def test_created_at_change_not_drift(self) -> None:
|
||||
from soup_cli.utils.soup_lock import SoupLock, check_lock_drift
|
||||
|
||||
base = dict(
|
||||
soup_version="0.67.0",
|
||||
base_model="m",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
)
|
||||
expected = SoupLock(created_at="2026-05-24", **base)
|
||||
actual = SoupLock(created_at="2026-06-01", **base)
|
||||
|
||||
drift = check_lock_drift(expected, actual)
|
||||
assert drift.ok is True
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Part F — Non-monotonic eval_fn does not crash
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBisectNonMonotonic:
|
||||
def test_non_monotonic_returns_valid_result(self) -> None:
|
||||
"""If the eval_fn flips back and forth (non-monotonic regression),
|
||||
the bisect must still return a valid ``BisectResult`` rather than
|
||||
crash. The boundary it finds is undefined but must be in-range."""
|
||||
from soup_cli.utils.adapter_bisect import (
|
||||
BisectPlan,
|
||||
BisectResult,
|
||||
run_bisect,
|
||||
)
|
||||
|
||||
plan = BisectPlan(history=("c0", "c1", "c2", "c3", "c4", "c5"))
|
||||
|
||||
# Non-monotonic: c0 ok, c1 fail, c2 ok, c3 fail, ...
|
||||
def eval_fn(checkpoint: str) -> bool:
|
||||
return int(checkpoint[1:]) % 2 == 0
|
||||
|
||||
# Must not raise; result is a valid BisectResult
|
||||
result = run_bisect(plan, eval_fn=eval_fn)
|
||||
assert isinstance(result, BisectResult)
|
||||
# Boundary is some valid checkpoint id
|
||||
if result.first_broken is not None:
|
||||
assert result.first_broken in plan.history
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source-grep regression guards on policies the agents asked us to verify
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiringRegressions:
|
||||
"""Project-wide policies verified by source-grep across v0.67.0 modules."""
|
||||
|
||||
def test_all_modules_have_future_annotations(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
for module in (
|
||||
"soup_cli/utils/cmaes_merge.py",
|
||||
"soup_cli/utils/vector_bank.py",
|
||||
"soup_cli/utils/mole_routing.py",
|
||||
"soup_cli/utils/adapter_pr.py",
|
||||
"soup_cli/utils/soup_lock.py",
|
||||
"soup_cli/utils/adapter_bisect.py",
|
||||
"soup_cli/commands/lock.py",
|
||||
):
|
||||
src = (root / module).read_text(encoding="utf-8")
|
||||
assert "from __future__ import annotations" in src, (
|
||||
f"{module}: missing 'from __future__ import annotations'"
|
||||
)
|
||||
|
||||
def test_atomic_write_used_in_disk_surfaces(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
for module in (
|
||||
"soup_cli/utils/vector_bank.py",
|
||||
"soup_cli/utils/adapter_pr.py",
|
||||
"soup_cli/utils/soup_lock.py",
|
||||
):
|
||||
src = (root / module).read_text(encoding="utf-8")
|
||||
assert "atomic_write_text" in src, (
|
||||
f"{module}: must use atomic_write_text for disk writes"
|
||||
)
|
||||
|
||||
def test_no_top_level_torch_imports(self) -> None:
|
||||
"""v0.67.0 modules MUST stay torch-free at import time."""
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
for module in (
|
||||
"soup_cli/utils/cmaes_merge.py",
|
||||
"soup_cli/utils/vector_bank.py",
|
||||
"soup_cli/utils/mole_routing.py",
|
||||
"soup_cli/utils/adapter_pr.py",
|
||||
"soup_cli/utils/soup_lock.py",
|
||||
"soup_cli/utils/adapter_bisect.py",
|
||||
"soup_cli/commands/lock.py",
|
||||
):
|
||||
src = (root / module).read_text(encoding="utf-8")
|
||||
head = "\n".join(
|
||||
line for line in src.splitlines()[:50]
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
)
|
||||
for forbidden in (
|
||||
"import torch", "import transformers",
|
||||
"import peft", "import safetensors",
|
||||
):
|
||||
assert forbidden not in head, (
|
||||
f"{module}: top-level {forbidden!r} forbidden"
|
||||
)
|
||||
|
||||
def test_supported_strategies_has_cmaes(self) -> None:
|
||||
from soup_cli.utils.adapter_merge import SUPPORTED_STRATEGIES
|
||||
|
||||
assert "cmaes" in SUPPORTED_STRATEGIES
|
||||
|
||||
def test_subprocess_call_uses_argv_list(self) -> None:
|
||||
"""`soup adapters bisect` must use argv list mode (no shell=True)."""
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "commands" / "adapters.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
# The bisect subprocess call site
|
||||
# Must NOT use shell=True; must use shlex.split + shlex.quote
|
||||
assert "shell=True" not in src
|
||||
# Must include shlex.quote pattern for the {ckpt} substitution
|
||||
assert "shlex.quote" in src
|
||||
|
|
@ -0,0 +1,480 @@
|
|||
"""v0.67.0 Part A — Evolutionary merge (CMA-ES) over LoRA adapter weights.
|
||||
|
||||
Tests for ``soup_cli/utils/cmaes_merge.py``:
|
||||
|
||||
- Closed allowlist + frozen dataclasses + parse_budget reuse from blame.py
|
||||
- ``run_cmaes_merge`` orchestrator with operator-supplied ``eval_fn`` injection
|
||||
- ``soup adapters merge --strategy cmaes --eval <suite> --budget 1h`` CLI plumbing
|
||||
- Validation matrix (bool/null-byte/non-finite/oversize rejection)
|
||||
- Budget bounds reused from v0.57 ``blame.parse_budget`` (60s..24h)
|
||||
- Source-grep regression guard for the new strategy alias in adapter_merge
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Module surface — imports + constants
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPublicSurface:
|
||||
def test_module_importable(self) -> None:
|
||||
from soup_cli.utils import cmaes_merge
|
||||
|
||||
assert hasattr(cmaes_merge, "CmaesPlan")
|
||||
assert hasattr(cmaes_merge, "CmaesResult")
|
||||
assert hasattr(cmaes_merge, "run_cmaes_merge")
|
||||
assert hasattr(cmaes_merge, "validate_population_size")
|
||||
assert hasattr(cmaes_merge, "validate_generations")
|
||||
|
||||
def test_constants_immutable(self) -> None:
|
||||
from soup_cli.utils import cmaes_merge
|
||||
|
||||
assert cmaes_merge.MIN_POPULATION >= 2
|
||||
assert cmaes_merge.MAX_POPULATION <= 256
|
||||
assert cmaes_merge.MIN_GENERATIONS >= 1
|
||||
assert cmaes_merge.MAX_GENERATIONS <= 10000
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Validators
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidatePopulationSize:
|
||||
def test_happy(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import validate_population_size
|
||||
|
||||
assert validate_population_size(8) == 8
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import validate_population_size
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_population_size(True) # type: ignore[arg-type]
|
||||
|
||||
def test_below_floor(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import validate_population_size
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_population_size(1)
|
||||
|
||||
def test_above_cap(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import MAX_POPULATION, validate_population_size
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_population_size(MAX_POPULATION + 1)
|
||||
|
||||
def test_non_int_rejected(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import validate_population_size
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_population_size("8") # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestValidateGenerations:
|
||||
def test_happy(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import validate_generations
|
||||
|
||||
assert validate_generations(20) == 20
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import validate_generations
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_generations(False) # type: ignore[arg-type]
|
||||
|
||||
def test_below_floor(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import validate_generations
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_generations(0)
|
||||
|
||||
def test_above_cap(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import MAX_GENERATIONS, validate_generations
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_generations(MAX_GENERATIONS + 1)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Frozen dataclasses
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCmaesPlan:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import CmaesPlan
|
||||
|
||||
plan = CmaesPlan(
|
||||
adapters=("a", "b"),
|
||||
eval_suite="suite.yaml",
|
||||
budget_seconds=3600,
|
||||
population_size=8,
|
||||
max_generations=20,
|
||||
seed=42,
|
||||
)
|
||||
assert plan.population_size == 8
|
||||
assert plan.eval_suite == "suite.yaml"
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import CmaesPlan
|
||||
|
||||
plan = CmaesPlan(
|
||||
adapters=("a", "b"),
|
||||
eval_suite="s.yaml",
|
||||
budget_seconds=600,
|
||||
population_size=4,
|
||||
max_generations=5,
|
||||
seed=0,
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
plan.population_size = 99 # type: ignore[misc]
|
||||
|
||||
|
||||
class TestCmaesResult:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import CmaesResult
|
||||
|
||||
result = CmaesResult(
|
||||
best_weights=(0.5, 0.5),
|
||||
best_score=0.93,
|
||||
generations_run=10,
|
||||
evaluations=80,
|
||||
wall_clock_seconds=120.5,
|
||||
converged=True,
|
||||
history=(0.85, 0.87, 0.9, 0.93),
|
||||
)
|
||||
assert result.best_score == 0.93
|
||||
assert result.generations_run == 10
|
||||
assert result.converged is True
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import CmaesResult
|
||||
|
||||
result = CmaesResult(
|
||||
best_weights=(0.5, 0.5),
|
||||
best_score=0.5,
|
||||
generations_run=1,
|
||||
evaluations=4,
|
||||
wall_clock_seconds=1.0,
|
||||
converged=False,
|
||||
history=(0.5,),
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
result.best_score = 9.99 # type: ignore[misc]
|
||||
|
||||
def test_weights_must_be_simplex(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import CmaesResult
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
CmaesResult(
|
||||
best_weights=(0.3, 0.3), # sums to 0.6, not 1.0
|
||||
best_score=0.5,
|
||||
generations_run=1,
|
||||
evaluations=4,
|
||||
wall_clock_seconds=1.0,
|
||||
converged=False,
|
||||
history=(0.5,),
|
||||
)
|
||||
|
||||
def test_bool_score_rejected(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import CmaesResult
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
CmaesResult(
|
||||
best_weights=(0.5, 0.5),
|
||||
best_score=True, # type: ignore[arg-type]
|
||||
generations_run=1,
|
||||
evaluations=4,
|
||||
wall_clock_seconds=1.0,
|
||||
converged=False,
|
||||
history=(0.5,),
|
||||
)
|
||||
|
||||
def test_non_finite_score_rejected(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import CmaesResult
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
CmaesResult(
|
||||
best_weights=(0.5, 0.5),
|
||||
best_score=math.nan,
|
||||
generations_run=1,
|
||||
evaluations=4,
|
||||
wall_clock_seconds=1.0,
|
||||
converged=False,
|
||||
history=(0.5,),
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# build_cmaes_plan
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBuildCmaesPlan:
|
||||
def test_happy(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
adapter_a = tmp_path / "a"
|
||||
adapter_a.mkdir()
|
||||
adapter_b = tmp_path / "b"
|
||||
adapter_b.mkdir()
|
||||
suite = tmp_path / "suite.yaml"
|
||||
suite.write_text("dummy: 1\n", encoding="utf-8")
|
||||
|
||||
plan = build_cmaes_plan(
|
||||
adapters=[str(adapter_a), str(adapter_b)],
|
||||
eval_suite=str(suite),
|
||||
budget_spec="10m",
|
||||
population_size=6,
|
||||
max_generations=15,
|
||||
seed=42,
|
||||
)
|
||||
assert plan.population_size == 6
|
||||
assert plan.budget_seconds == 600
|
||||
assert plan.max_generations == 15
|
||||
|
||||
def test_invalid_budget_string(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
for name in ("a", "b"):
|
||||
(tmp_path / name).mkdir()
|
||||
suite = tmp_path / "s.yaml"
|
||||
suite.write_text("x: 1\n", encoding="utf-8")
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_cmaes_plan(
|
||||
adapters=[str(tmp_path / "a"), str(tmp_path / "b")],
|
||||
eval_suite=str(suite),
|
||||
budget_spec="not-a-budget",
|
||||
population_size=4,
|
||||
max_generations=5,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_below_min_adapters(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "a").mkdir()
|
||||
suite = tmp_path / "s.yaml"
|
||||
suite.write_text("x: 1\n", encoding="utf-8")
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_cmaes_plan(
|
||||
adapters=[str(tmp_path / "a")],
|
||||
eval_suite=str(suite),
|
||||
budget_spec="60s",
|
||||
population_size=4,
|
||||
max_generations=5,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_eval_suite_outside_cwd(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan
|
||||
|
||||
cwd = tmp_path / "work"
|
||||
cwd.mkdir()
|
||||
monkeypatch.chdir(cwd)
|
||||
(cwd / "a").mkdir()
|
||||
(cwd / "b").mkdir()
|
||||
outside = tmp_path / "outside.yaml"
|
||||
outside.write_text("x: 1\n", encoding="utf-8")
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
build_cmaes_plan(
|
||||
adapters=[str(cwd / "a"), str(cwd / "b")],
|
||||
eval_suite=str(outside),
|
||||
budget_spec="60s",
|
||||
population_size=4,
|
||||
max_generations=5,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# run_cmaes_merge orchestrator — operator-supplied eval_fn
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRunCmaesMerge:
|
||||
def test_returns_result(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
for name in ("a", "b"):
|
||||
(tmp_path / name).mkdir()
|
||||
suite = tmp_path / "s.yaml"
|
||||
suite.write_text("x: 1\n", encoding="utf-8")
|
||||
|
||||
plan = build_cmaes_plan(
|
||||
adapters=[str(tmp_path / "a"), str(tmp_path / "b")],
|
||||
eval_suite=str(suite),
|
||||
budget_spec="60s",
|
||||
population_size=4,
|
||||
max_generations=3,
|
||||
seed=42,
|
||||
)
|
||||
|
||||
# eval_fn returns a deterministic score with maximum near weights=(0.5, 0.5)
|
||||
def eval_fn(weights):
|
||||
target = (0.5, 0.5)
|
||||
diff = sum((w - t) ** 2 for w, t in zip(weights, target))
|
||||
return 1.0 - diff
|
||||
|
||||
result = run_cmaes_merge(plan, eval_fn=eval_fn)
|
||||
assert result.generations_run >= 1
|
||||
assert result.evaluations >= 1
|
||||
assert 0.0 <= result.best_score <= 1.0
|
||||
# weights normalised to simplex
|
||||
assert math.isclose(sum(result.best_weights), 1.0, abs_tol=1e-6)
|
||||
|
||||
def test_eval_fn_required(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
for name in ("a", "b"):
|
||||
(tmp_path / name).mkdir()
|
||||
suite = tmp_path / "s.yaml"
|
||||
suite.write_text("x: 1\n", encoding="utf-8")
|
||||
|
||||
plan = build_cmaes_plan(
|
||||
adapters=[str(tmp_path / "a"), str(tmp_path / "b")],
|
||||
eval_suite=str(suite),
|
||||
budget_spec="60s",
|
||||
population_size=4,
|
||||
max_generations=2,
|
||||
seed=0,
|
||||
)
|
||||
with pytest.raises(TypeError):
|
||||
run_cmaes_merge(plan, eval_fn=None) # type: ignore[arg-type]
|
||||
|
||||
def test_eval_fn_exceptions_logged(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan, run_cmaes_merge
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
for name in ("a", "b"):
|
||||
(tmp_path / name).mkdir()
|
||||
suite = tmp_path / "s.yaml"
|
||||
suite.write_text("x: 1\n", encoding="utf-8")
|
||||
|
||||
plan = build_cmaes_plan(
|
||||
adapters=[str(tmp_path / "a"), str(tmp_path / "b")],
|
||||
eval_suite=str(suite),
|
||||
budget_spec="60s",
|
||||
population_size=4,
|
||||
max_generations=2,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def eval_fn(weights):
|
||||
if weights[0] > 0.8:
|
||||
raise RuntimeError("simulated eval failure")
|
||||
return 0.5
|
||||
|
||||
# Should not crash; failed evals get sentinel score
|
||||
result = run_cmaes_merge(plan, eval_fn=eval_fn)
|
||||
assert result.evaluations >= 1
|
||||
|
||||
def test_non_plan_rejected(self) -> None:
|
||||
from soup_cli.utils.cmaes_merge import run_cmaes_merge
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
run_cmaes_merge("not-a-plan", eval_fn=lambda w: 0.5) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# adapter_merge integration: cmaes is a known strategy
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAdapterMergeIntegration:
|
||||
def test_cmaes_in_supported_strategies(self) -> None:
|
||||
from soup_cli.utils.adapter_merge import SUPPORTED_STRATEGIES
|
||||
|
||||
assert "cmaes" in SUPPORTED_STRATEGIES
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CLI smoke
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCliSmoke:
|
||||
def test_merge_help_lists_cmaes(self) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.adapters import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["merge", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "cmaes" in result.output.lower()
|
||||
|
||||
def test_merge_cmaes_requires_eval(self, tmp_path, monkeypatch) -> None:
|
||||
"""cmaes without --eval should exit 2 with friendly message."""
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.adapters import app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
runner = CliRunner()
|
||||
|
||||
# We don't bother making valid adapters; we just want to see the strategy
|
||||
# validation reject missing --eval before file-handling kicks in.
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"merge",
|
||||
str(tmp_path / "a"),
|
||||
str(tmp_path / "b"),
|
||||
"--strategy",
|
||||
"cmaes",
|
||||
"--output",
|
||||
str(tmp_path / "out"),
|
||||
],
|
||||
)
|
||||
# --strategy cmaes WITHOUT --eval must fail
|
||||
assert result.exit_code == 2
|
||||
assert "eval" in result.output.lower()
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source-grep regression guards
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_top_level_heavy_imports(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "cmaes_merge.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
# Only first 30 non-comment lines (matches v0.66 review-grep policy)
|
||||
head_lines = [
|
||||
line
|
||||
for line in src.splitlines()[:50]
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
]
|
||||
head = "\n".join(head_lines)
|
||||
for forbidden in ("import torch", "import transformers", "import peft"):
|
||||
assert forbidden not in head, f"top-level {forbidden!r} in cmaes_merge"
|
||||
|
||||
def test_adapter_merge_lists_cmaes(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "adapter_merge.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
assert "cmaes" in src
|
||||
|
|
@ -0,0 +1,467 @@
|
|||
"""v0.67.0 Part B — VeRA / VB-LoRA vector-bank storage format.
|
||||
|
||||
Tests for ``soup_cli/utils/vector_bank.py``:
|
||||
|
||||
- Frozen ``VectorBank`` / ``BankEntry`` dataclasses (shared random
|
||||
projection matrix + per-user MB-sized scaling vectors)
|
||||
- Validation matrix on bank name, user id, scaling-vector shape
|
||||
- ``estimate_bank_size`` for sizing decisions
|
||||
- Stub-then-live: live serving wiring deferred to v0.67.1
|
||||
- Atomic disk I/O via shared ``atomic_write_text`` (TOCTOU-safe)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import math
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Public surface
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPublicSurface:
|
||||
def test_module_importable(self) -> None:
|
||||
from soup_cli.utils import vector_bank
|
||||
|
||||
assert hasattr(vector_bank, "VectorBank")
|
||||
assert hasattr(vector_bank, "BankEntry")
|
||||
assert hasattr(vector_bank, "validate_bank_name")
|
||||
assert hasattr(vector_bank, "validate_user_id")
|
||||
assert hasattr(vector_bank, "validate_scaling_vector")
|
||||
assert hasattr(vector_bank, "estimate_bank_size")
|
||||
assert hasattr(vector_bank, "apply_bank_to_serve")
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# validate_bank_name
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateBankName:
|
||||
def test_happy(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
assert validate_bank_name("my-bank") == "my-bank"
|
||||
|
||||
def test_case_insensitive(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
# Canonical: kebab-case, normalise to lowercase
|
||||
assert validate_bank_name("MY-BANK") == "my-bank"
|
||||
|
||||
def test_null_byte_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_bank_name("bad\x00name")
|
||||
|
||||
def test_empty_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_bank_name("")
|
||||
|
||||
def test_oversize_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_bank_name("a" * 200)
|
||||
|
||||
def test_non_string_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_bank_name(123) # type: ignore[arg-type]
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_bank_name(True) # type: ignore[arg-type]
|
||||
|
||||
def test_invalid_chars_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_bank_name
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_bank_name("path/traversal")
|
||||
with pytest.raises(ValueError):
|
||||
validate_bank_name("..")
|
||||
with pytest.raises(ValueError):
|
||||
validate_bank_name("name with space")
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# validate_user_id
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateUserId:
|
||||
def test_happy(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_user_id
|
||||
|
||||
assert validate_user_id("user-1234") == "user-1234"
|
||||
|
||||
def test_oversize_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_user_id
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_user_id("u" * 300)
|
||||
|
||||
def test_null_byte_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_user_id
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_user_id("u\x00")
|
||||
|
||||
def test_non_string_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_user_id
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_user_id(42) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# validate_scaling_vector
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateScalingVector:
|
||||
def test_happy(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_scaling_vector
|
||||
|
||||
v = validate_scaling_vector([0.1, 0.2, -0.3])
|
||||
assert v == (0.1, 0.2, -0.3)
|
||||
|
||||
def test_returns_tuple(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_scaling_vector
|
||||
|
||||
v = validate_scaling_vector([1.0, 2.0])
|
||||
assert isinstance(v, tuple)
|
||||
|
||||
def test_empty_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_scaling_vector
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_scaling_vector([])
|
||||
|
||||
def test_non_iterable_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_scaling_vector
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_scaling_vector(42) # type: ignore[arg-type]
|
||||
|
||||
def test_non_finite_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_scaling_vector
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_scaling_vector([1.0, math.nan])
|
||||
with pytest.raises(ValueError):
|
||||
validate_scaling_vector([math.inf, 1.0])
|
||||
|
||||
def test_bool_in_vector_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import validate_scaling_vector
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_scaling_vector([True, 0.5])
|
||||
|
||||
def test_oversize_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import MAX_VECTOR_DIM, validate_scaling_vector
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_scaling_vector([0.1] * (MAX_VECTOR_DIM + 1))
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# BankEntry + VectorBank frozen dataclasses
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBankEntry:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.vector_bank import BankEntry
|
||||
|
||||
entry = BankEntry(user_id="alice", scaling=(0.1, 0.2, 0.3))
|
||||
assert entry.user_id == "alice"
|
||||
assert entry.scaling == (0.1, 0.2, 0.3)
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.vector_bank import BankEntry
|
||||
|
||||
entry = BankEntry(user_id="bob", scaling=(0.5,))
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
entry.user_id = "eve" # type: ignore[misc]
|
||||
|
||||
|
||||
class TestVectorBank:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.vector_bank import BankEntry, VectorBank
|
||||
|
||||
bank = VectorBank(
|
||||
name="mybank",
|
||||
base_model="meta-llama/Llama-3.1-8B",
|
||||
projection_seed=42,
|
||||
vector_dim=2,
|
||||
entries=(BankEntry(user_id="u1", scaling=(0.1, 0.2)),),
|
||||
)
|
||||
assert bank.name == "mybank"
|
||||
assert bank.vector_dim == 2
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.vector_bank import VectorBank
|
||||
|
||||
bank = VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=8,
|
||||
entries=(),
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
bank.vector_dim = 99 # type: ignore[misc]
|
||||
|
||||
def test_invalid_seed(self) -> None:
|
||||
from soup_cli.utils.vector_bank import VectorBank
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=True, # type: ignore[arg-type]
|
||||
vector_dim=8,
|
||||
entries=(),
|
||||
)
|
||||
|
||||
def test_invalid_vector_dim(self) -> None:
|
||||
from soup_cli.utils.vector_bank import VectorBank
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=0,
|
||||
entries=(),
|
||||
)
|
||||
|
||||
def test_entries_must_be_tuple(self) -> None:
|
||||
from soup_cli.utils.vector_bank import VectorBank
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=8,
|
||||
entries=[], # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def test_max_entries_enforced(self) -> None:
|
||||
from soup_cli.utils.vector_bank import (
|
||||
MAX_ENTRIES_PER_BANK,
|
||||
BankEntry,
|
||||
VectorBank,
|
||||
)
|
||||
|
||||
too_many = tuple(
|
||||
BankEntry(user_id=f"u{i}", scaling=(0.1,))
|
||||
for i in range(MAX_ENTRIES_PER_BANK + 1)
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=1,
|
||||
entries=too_many,
|
||||
)
|
||||
|
||||
def test_base_model_validation(self) -> None:
|
||||
from soup_cli.utils.vector_bank import VectorBank
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="",
|
||||
projection_seed=0,
|
||||
vector_dim=8,
|
||||
entries=(),
|
||||
)
|
||||
|
||||
def test_per_entry_vector_dim_mismatch(self) -> None:
|
||||
from soup_cli.utils.vector_bank import BankEntry, VectorBank
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=4,
|
||||
entries=(BankEntry(user_id="u", scaling=(0.1, 0.2, 0.3)),),
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# estimate_bank_size
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestEstimateBankSize:
|
||||
def test_basic(self) -> None:
|
||||
from soup_cli.utils.vector_bank import estimate_bank_size
|
||||
|
||||
# 1000 users × 128 fp32 scaling vectors ≈ 512 KB + projection
|
||||
# matrix (128×128×4 = 64 KB)
|
||||
size_bytes = estimate_bank_size(num_users=1000, vector_dim=128)
|
||||
assert 400_000 < size_bytes < 1_000_000
|
||||
|
||||
def test_zero_users(self) -> None:
|
||||
from soup_cli.utils.vector_bank import estimate_bank_size
|
||||
|
||||
# Only projection matrix
|
||||
size_bytes = estimate_bank_size(num_users=0, vector_dim=128)
|
||||
assert size_bytes > 0 # projection still allocated
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import estimate_bank_size
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
estimate_bank_size(num_users=True, vector_dim=128) # type: ignore[arg-type]
|
||||
|
||||
def test_negative_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import estimate_bank_size
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
estimate_bank_size(num_users=-1, vector_dim=128)
|
||||
with pytest.raises(ValueError):
|
||||
estimate_bank_size(num_users=10, vector_dim=0)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Atomic JSON write + read
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestWriteLoadBank:
|
||||
def test_roundtrip(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.vector_bank import (
|
||||
BankEntry,
|
||||
VectorBank,
|
||||
load_bank,
|
||||
write_bank,
|
||||
)
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
bank = VectorBank(
|
||||
name="round-trip",
|
||||
base_model="meta-llama/Llama-3.1-8B",
|
||||
projection_seed=42,
|
||||
vector_dim=4,
|
||||
entries=(
|
||||
BankEntry(user_id="alice", scaling=(0.1, 0.2, 0.3, 0.4)),
|
||||
BankEntry(user_id="bob", scaling=(0.5, 0.6, 0.7, 0.8)),
|
||||
),
|
||||
)
|
||||
path = str(tmp_path / "bank.json")
|
||||
write_bank(bank, path)
|
||||
loaded = load_bank(path)
|
||||
assert loaded.name == bank.name
|
||||
assert loaded.entries == bank.entries
|
||||
|
||||
def test_outside_cwd_rejected(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.vector_bank import VectorBank, write_bank
|
||||
|
||||
cwd = tmp_path / "work"
|
||||
cwd.mkdir()
|
||||
monkeypatch.chdir(cwd)
|
||||
bank = VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=1,
|
||||
entries=(),
|
||||
)
|
||||
# Path outside cwd
|
||||
with pytest.raises(ValueError):
|
||||
write_bank(bank, str(tmp_path / "outside.json"))
|
||||
|
||||
def test_load_missing_file(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.vector_bank import load_bank
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_bank(str(tmp_path / "missing.json"))
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink test")
|
||||
def test_load_symlink_rejected(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.vector_bank import load_bank
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
real = tmp_path / "real.json"
|
||||
real.write_text('{"name": "x"}', encoding="utf-8")
|
||||
sym = tmp_path / "sym.json"
|
||||
os.symlink(real, sym)
|
||||
with pytest.raises(ValueError):
|
||||
load_bank(str(sym))
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Live-serve stub
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestApplyBankToServe:
|
||||
def test_deferred_v0_67_1(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.vector_bank import VectorBank, apply_bank_to_serve
|
||||
|
||||
bank = VectorBank(
|
||||
name="b",
|
||||
base_model="m",
|
||||
projection_seed=0,
|
||||
vector_dim=1,
|
||||
entries=(),
|
||||
)
|
||||
with pytest.raises(NotImplementedError) as exc_info:
|
||||
apply_bank_to_serve(bank)
|
||||
assert "v0.67.1" in str(exc_info.value)
|
||||
|
||||
def test_non_bank_rejected(self) -> None:
|
||||
from soup_cli.utils.vector_bank import apply_bank_to_serve
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
apply_bank_to_serve("not-a-bank") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source-grep regression guards
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_top_level_heavy_imports(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "vector_bank.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
head_lines = [
|
||||
line
|
||||
for line in src.splitlines()[:50]
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
]
|
||||
head = "\n".join(head_lines)
|
||||
for forbidden in ("import torch", "import transformers", "import peft"):
|
||||
assert forbidden not in head, f"top-level {forbidden!r}"
|
||||
|
||||
def test_uses_atomic_write_helper(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "vector_bank.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
assert "atomic_write_text" in src
|
||||
|
|
@ -0,0 +1,315 @@
|
|||
"""v0.67.0 Part C — MoLE per-token adapter routing.
|
||||
|
||||
Tests for ``soup_cli/utils/mole_routing.py``:
|
||||
|
||||
- Frozen ``MoleGatingConfig`` dataclass
|
||||
- ``validate_mole_compat`` (task / backend / adapter-count gate)
|
||||
- ``build_gating_kernel`` stub raises NotImplementedError with v0.67.1 marker
|
||||
- New ``task='moe_lora_routing'`` Literal on SoupConfig
|
||||
- SoupConfig cross-validator gates mole config to the correct task/backend
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Public surface
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPublicSurface:
|
||||
def test_module_importable(self) -> None:
|
||||
from soup_cli.utils import mole_routing
|
||||
|
||||
assert hasattr(mole_routing, "MoleGatingConfig")
|
||||
assert hasattr(mole_routing, "validate_mole_compat")
|
||||
assert hasattr(mole_routing, "build_gating_kernel")
|
||||
assert hasattr(mole_routing, "MIN_TASK_ADAPTERS")
|
||||
assert hasattr(mole_routing, "MAX_TASK_ADAPTERS")
|
||||
|
||||
def test_constants_immutable(self) -> None:
|
||||
from soup_cli.utils import mole_routing
|
||||
|
||||
assert mole_routing.MIN_TASK_ADAPTERS >= 2
|
||||
assert mole_routing.MAX_TASK_ADAPTERS <= 64
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# MoleGatingConfig
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestMoleGatingConfig:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
cfg = MoleGatingConfig(
|
||||
num_task_adapters=4,
|
||||
hidden_dim=128,
|
||||
temperature=1.0,
|
||||
top_k=2,
|
||||
)
|
||||
assert cfg.num_task_adapters == 4
|
||||
assert cfg.top_k == 2
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
cfg = MoleGatingConfig(
|
||||
num_task_adapters=4,
|
||||
hidden_dim=128,
|
||||
temperature=1.0,
|
||||
top_k=2,
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
cfg.top_k = 99 # type: ignore[misc]
|
||||
|
||||
def test_num_task_adapters_below_floor(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=1, hidden_dim=128, temperature=1.0, top_k=1
|
||||
)
|
||||
|
||||
def test_num_task_adapters_above_cap(self) -> None:
|
||||
from soup_cli.utils.mole_routing import (
|
||||
MAX_TASK_ADAPTERS,
|
||||
MoleGatingConfig,
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=MAX_TASK_ADAPTERS + 1,
|
||||
hidden_dim=128,
|
||||
temperature=1.0,
|
||||
top_k=1,
|
||||
)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=True, # type: ignore[arg-type]
|
||||
hidden_dim=128,
|
||||
temperature=1.0,
|
||||
top_k=1,
|
||||
)
|
||||
|
||||
def test_hidden_dim_must_be_positive(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=4,
|
||||
hidden_dim=0,
|
||||
temperature=1.0,
|
||||
top_k=1,
|
||||
)
|
||||
|
||||
def test_temperature_non_finite_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=4,
|
||||
hidden_dim=128,
|
||||
temperature=math.nan,
|
||||
top_k=1,
|
||||
)
|
||||
|
||||
def test_temperature_non_positive_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=4,
|
||||
hidden_dim=128,
|
||||
temperature=0.0,
|
||||
top_k=1,
|
||||
)
|
||||
|
||||
def test_top_k_above_num_adapters_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=3,
|
||||
hidden_dim=128,
|
||||
temperature=1.0,
|
||||
top_k=5,
|
||||
)
|
||||
|
||||
def test_top_k_below_one_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MoleGatingConfig(
|
||||
num_task_adapters=4,
|
||||
hidden_dim=128,
|
||||
temperature=1.0,
|
||||
top_k=0,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# validate_mole_compat
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateMoleCompat:
|
||||
def test_happy_path(self) -> None:
|
||||
from soup_cli.utils.mole_routing import validate_mole_compat
|
||||
|
||||
validate_mole_compat(
|
||||
task="moe_lora_routing",
|
||||
backend="transformers",
|
||||
num_task_adapters=4,
|
||||
)
|
||||
|
||||
def test_wrong_task_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import validate_mole_compat
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
validate_mole_compat(
|
||||
task="sft",
|
||||
backend="transformers",
|
||||
num_task_adapters=4,
|
||||
)
|
||||
assert "moe_lora_routing" in str(exc_info.value)
|
||||
|
||||
def test_mlx_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import validate_mole_compat
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
validate_mole_compat(
|
||||
task="moe_lora_routing",
|
||||
backend="mlx",
|
||||
num_task_adapters=4,
|
||||
)
|
||||
assert "mlx" in str(exc_info.value).lower()
|
||||
|
||||
def test_too_few_adapters(self) -> None:
|
||||
from soup_cli.utils.mole_routing import validate_mole_compat
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_mole_compat(
|
||||
task="moe_lora_routing",
|
||||
backend="transformers",
|
||||
num_task_adapters=1,
|
||||
)
|
||||
|
||||
def test_bool_args_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import validate_mole_compat
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
validate_mole_compat(
|
||||
task=True, # type: ignore[arg-type]
|
||||
backend="transformers",
|
||||
num_task_adapters=4,
|
||||
)
|
||||
with pytest.raises(TypeError):
|
||||
validate_mole_compat(
|
||||
task="moe_lora_routing",
|
||||
backend=True, # type: ignore[arg-type]
|
||||
num_task_adapters=4,
|
||||
)
|
||||
|
||||
def test_null_byte_args_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import validate_mole_compat
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
validate_mole_compat(
|
||||
task="moe_lora_routing\x00",
|
||||
backend="transformers",
|
||||
num_task_adapters=4,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Deferred-live stub
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBuildGatingKernel:
|
||||
def test_deferred_v0_67_1(self) -> None:
|
||||
from soup_cli.utils.mole_routing import MoleGatingConfig, build_gating_kernel
|
||||
|
||||
cfg = MoleGatingConfig(
|
||||
num_task_adapters=4, hidden_dim=128, temperature=1.0, top_k=2
|
||||
)
|
||||
with pytest.raises(NotImplementedError) as exc_info:
|
||||
build_gating_kernel(cfg)
|
||||
assert "v0.67.1" in str(exc_info.value)
|
||||
|
||||
def test_non_config_rejected(self) -> None:
|
||||
from soup_cli.utils.mole_routing import build_gating_kernel
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
build_gating_kernel("not-a-config") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# SoupConfig integration — task='moe_lora_routing' Literal
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSchemaIntegration:
|
||||
def test_task_accepted_in_literal(self) -> None:
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml = """
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: moe_lora_routing
|
||||
backend: transformers
|
||||
modality: text
|
||||
data:
|
||||
train: data.jsonl
|
||||
format: chatml
|
||||
"""
|
||||
cfg = load_config_from_string(yaml)
|
||||
assert cfg.task == "moe_lora_routing"
|
||||
|
||||
def test_task_rejected_on_mlx(self) -> None:
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
yaml = """
|
||||
base: m
|
||||
task: moe_lora_routing
|
||||
backend: mlx
|
||||
modality: text
|
||||
data:
|
||||
train: data.jsonl
|
||||
format: chatml
|
||||
"""
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
load_config_from_string(yaml)
|
||||
assert "mlx" in str(exc_info.value).lower()
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source-grep regression
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_top_level_heavy_imports(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "mole_routing.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
head_lines = [
|
||||
line
|
||||
for line in src.splitlines()[:50]
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
]
|
||||
head = "\n".join(head_lines)
|
||||
for forbidden in ("import torch", "import transformers", "import peft"):
|
||||
assert forbidden not in head, f"top-level {forbidden!r}"
|
||||
|
|
@ -0,0 +1,389 @@
|
|||
"""v0.67.0 Part D — ``soup adapters pr`` (GitHub-shaped PR rendering).
|
||||
|
||||
Tests for ``soup_cli/utils/adapter_pr.py``:
|
||||
|
||||
- Frozen ``EvalDelta`` / ``SampleDiff`` / ``AdapterPR`` dataclasses
|
||||
- ``build_adapter_pr`` factory with validation
|
||||
- ``render_pr_markdown`` (eval-delta tables + sample diffs)
|
||||
- ``render_pr_json`` round-trip
|
||||
- Rich-markup escape on every operator-controlled field
|
||||
- CLI smoke (`soup adapters pr`)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Public surface
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPublicSurface:
|
||||
def test_module_importable(self) -> None:
|
||||
from soup_cli.utils import adapter_pr
|
||||
|
||||
assert hasattr(adapter_pr, "AdapterPR")
|
||||
assert hasattr(adapter_pr, "EvalDelta")
|
||||
assert hasattr(adapter_pr, "SampleDiff")
|
||||
assert hasattr(adapter_pr, "build_adapter_pr")
|
||||
assert hasattr(adapter_pr, "render_pr_markdown")
|
||||
assert hasattr(adapter_pr, "render_pr_json")
|
||||
assert hasattr(adapter_pr, "write_pr_markdown")
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# EvalDelta
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestEvalDelta:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import EvalDelta
|
||||
|
||||
delta = EvalDelta(metric="accuracy", baseline=0.7, candidate=0.85)
|
||||
assert math.isclose(delta.delta, 0.15)
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import EvalDelta
|
||||
|
||||
d = EvalDelta(metric="m", baseline=0.5, candidate=0.6)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
d.candidate = 0.99 # type: ignore[misc]
|
||||
|
||||
def test_non_finite_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import EvalDelta
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
EvalDelta(metric="m", baseline=math.nan, candidate=0.5)
|
||||
with pytest.raises(ValueError):
|
||||
EvalDelta(metric="m", baseline=0.5, candidate=math.inf)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import EvalDelta
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
EvalDelta(metric="m", baseline=True, candidate=0.5) # type: ignore[arg-type]
|
||||
|
||||
def test_metric_null_byte_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import EvalDelta
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
EvalDelta(metric="m\x00", baseline=0.5, candidate=0.6)
|
||||
|
||||
def test_metric_oversize_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import EvalDelta
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
EvalDelta(metric="a" * 300, baseline=0.5, candidate=0.6)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# SampleDiff
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSampleDiff:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import SampleDiff
|
||||
|
||||
diff = SampleDiff(
|
||||
prompt="What is 2+2?",
|
||||
baseline_output="four",
|
||||
candidate_output="4",
|
||||
)
|
||||
assert diff.prompt.startswith("What")
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import SampleDiff
|
||||
|
||||
d = SampleDiff(prompt="p", baseline_output="b", candidate_output="c")
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
d.prompt = "new" # type: ignore[misc]
|
||||
|
||||
def test_null_byte_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import SampleDiff
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
SampleDiff(prompt="p\x00", baseline_output="b", candidate_output="c")
|
||||
|
||||
def test_oversize_truncated(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import MAX_OUTPUT_LEN, SampleDiff
|
||||
|
||||
# >MAX_OUTPUT_LEN should be rejected to keep PRs reviewable
|
||||
with pytest.raises(ValueError):
|
||||
SampleDiff(
|
||||
prompt="p" * 10,
|
||||
baseline_output="x" * (MAX_OUTPUT_LEN + 1),
|
||||
candidate_output="y",
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# AdapterPR
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAdapterPR:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR, EvalDelta, SampleDiff
|
||||
|
||||
pr = AdapterPR(
|
||||
title="add-customer-support-tone",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="+ 100 rows of support data\n",
|
||||
deltas=(EvalDelta(metric="accuracy", baseline=0.7, candidate=0.85),),
|
||||
samples=(
|
||||
SampleDiff(prompt="hi", baseline_output="hello", candidate_output="hey"),
|
||||
),
|
||||
)
|
||||
assert pr.title == "add-customer-support-tone"
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR
|
||||
|
||||
pr = AdapterPR(
|
||||
title="t",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
pr.title = "new" # type: ignore[misc]
|
||||
|
||||
def test_base_sha_64_hex(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR
|
||||
|
||||
# Not 64 hex chars
|
||||
with pytest.raises(ValueError):
|
||||
AdapterPR(
|
||||
title="t",
|
||||
base_sha="abc",
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
# Non-hex characters
|
||||
with pytest.raises(ValueError):
|
||||
AdapterPR(
|
||||
title="t",
|
||||
base_sha="z" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
|
||||
def test_title_validation(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
AdapterPR(
|
||||
title="",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
|
||||
def test_deltas_must_be_tuple(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
AdapterPR(
|
||||
title="t",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=[], # type: ignore[arg-type]
|
||||
samples=(),
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# build_adapter_pr factory
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBuildAdapterPR:
|
||||
def test_happy(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import build_adapter_pr
|
||||
|
||||
pr = build_adapter_pr(
|
||||
title="my-pr",
|
||||
base_sha="b" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="diff text",
|
||||
deltas=[
|
||||
{"metric": "accuracy", "baseline": 0.7, "candidate": 0.85},
|
||||
],
|
||||
samples=[
|
||||
{"prompt": "p", "baseline_output": "a", "candidate_output": "b"},
|
||||
],
|
||||
)
|
||||
assert pr.title == "my-pr"
|
||||
assert len(pr.deltas) == 1
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# render_pr_markdown / render_pr_json
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRenderPR:
|
||||
def test_markdown_structure(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR, EvalDelta, SampleDiff, render_pr_markdown
|
||||
|
||||
pr = AdapterPR(
|
||||
title="my-pr",
|
||||
base_sha="c" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="+row1\n+row2\n",
|
||||
deltas=(EvalDelta(metric="accuracy", baseline=0.7, candidate=0.85),),
|
||||
samples=(
|
||||
SampleDiff(prompt="hi", baseline_output="a", candidate_output="b"),
|
||||
),
|
||||
)
|
||||
md = render_pr_markdown(pr)
|
||||
assert "my-pr" in md
|
||||
assert "accuracy" in md
|
||||
assert "0.70" in md or "0.7" in md
|
||||
# Should include a sample diff section
|
||||
assert "hi" in md
|
||||
# Should include the dataset diff
|
||||
assert "+row1" in md
|
||||
|
||||
def test_markdown_escapes_markdown_metacharacters(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR, EvalDelta, render_pr_markdown
|
||||
|
||||
# Embed pipe character in metric name; rendered table cell must escape it
|
||||
pr = AdapterPR(
|
||||
title="t",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(EvalDelta(metric="acc|injection", baseline=0.5, candidate=0.6),),
|
||||
samples=(),
|
||||
)
|
||||
md = render_pr_markdown(pr)
|
||||
# The pipe must be escaped to avoid breaking the table
|
||||
assert "acc\\|injection" in md or "acc|injection" not in md.split("|")
|
||||
|
||||
def test_json_roundtrip(self) -> None:
|
||||
import json
|
||||
|
||||
from soup_cli.utils.adapter_pr import (
|
||||
AdapterPR,
|
||||
EvalDelta,
|
||||
render_pr_json,
|
||||
)
|
||||
|
||||
pr = AdapterPR(
|
||||
title="t",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(EvalDelta(metric="m", baseline=0.5, candidate=0.6),),
|
||||
samples=(),
|
||||
)
|
||||
text = render_pr_json(pr)
|
||||
data = json.loads(text)
|
||||
assert data["title"] == "t"
|
||||
assert data["deltas"][0]["metric"] == "m"
|
||||
|
||||
def test_non_pr_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_pr import render_pr_json, render_pr_markdown
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
render_pr_markdown("not-a-pr") # type: ignore[arg-type]
|
||||
with pytest.raises(TypeError):
|
||||
render_pr_json("not-a-pr") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Atomic write
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestWritePR:
|
||||
def test_write_markdown(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR, write_pr_markdown
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
pr = AdapterPR(
|
||||
title="t",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
out = tmp_path / "pr.md"
|
||||
write_pr_markdown(pr, str(out))
|
||||
assert out.exists()
|
||||
assert "t" in out.read_text(encoding="utf-8")
|
||||
|
||||
def test_write_outside_cwd_rejected(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.adapter_pr import AdapterPR, write_pr_markdown
|
||||
|
||||
cwd = tmp_path / "work"
|
||||
cwd.mkdir()
|
||||
monkeypatch.chdir(cwd)
|
||||
pr = AdapterPR(
|
||||
title="t",
|
||||
base_sha="a" * 64,
|
||||
adapter_path="adapter/",
|
||||
dataset_diff="",
|
||||
deltas=(),
|
||||
samples=(),
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
write_pr_markdown(pr, str(tmp_path / "outside.md"))
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CLI smoke
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCliSmoke:
|
||||
def test_pr_help(self) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.adapters import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["pr", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "pr" in result.output.lower()
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source-grep regression
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_top_level_heavy_imports(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "adapter_pr.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
head_lines = [
|
||||
line
|
||||
for line in src.splitlines()[:50]
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
]
|
||||
head = "\n".join(head_lines)
|
||||
for forbidden in ("import torch", "import transformers", "import peft"):
|
||||
assert forbidden not in head, f"top-level {forbidden!r}"
|
||||
|
|
@ -0,0 +1,321 @@
|
|||
"""v0.67.0 Part E — ``soup.lock`` shared run lockfile.
|
||||
|
||||
Tests for ``soup_cli/utils/soup_lock.py``:
|
||||
|
||||
- Frozen ``SoupLock`` dataclass (model SHA + dataset hash + env hash)
|
||||
- ``compute_lock_closure`` deterministic + content-sensitive
|
||||
- ``write_lock`` / ``read_lock`` atomic round-trip + cwd containment
|
||||
- ``check_lock_drift`` detects each independent field drift
|
||||
- Composes with v0.64 Part C ``soup env`` (env_hash is operator-supplied)
|
||||
- CLI smoke (`soup lock`)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestPublicSurface:
|
||||
def test_module_importable(self) -> None:
|
||||
from soup_cli.utils import soup_lock
|
||||
|
||||
assert hasattr(soup_lock, "SoupLock")
|
||||
assert hasattr(soup_lock, "LockDrift")
|
||||
assert hasattr(soup_lock, "compute_lock_closure")
|
||||
assert hasattr(soup_lock, "write_lock")
|
||||
assert hasattr(soup_lock, "read_lock")
|
||||
assert hasattr(soup_lock, "check_lock_drift")
|
||||
|
||||
|
||||
class TestComputeLockClosure:
|
||||
def test_deterministic(self) -> None:
|
||||
from soup_cli.utils.soup_lock import compute_lock_closure
|
||||
|
||||
a = compute_lock_closure(
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
)
|
||||
b = compute_lock_closure(
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
)
|
||||
assert a == b
|
||||
|
||||
def test_content_sensitive(self) -> None:
|
||||
from soup_cli.utils.soup_lock import compute_lock_closure
|
||||
|
||||
a = compute_lock_closure(
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
)
|
||||
b = compute_lock_closure(
|
||||
base_model_sha="f" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
)
|
||||
assert a != b
|
||||
|
||||
def test_invalid_sha_rejected(self) -> None:
|
||||
from soup_cli.utils.soup_lock import compute_lock_closure
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
compute_lock_closure(
|
||||
base_model_sha="not-hex",
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
from soup_cli.utils.soup_lock import compute_lock_closure
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
compute_lock_closure(
|
||||
base_model_sha=True, # type: ignore[arg-type]
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
)
|
||||
|
||||
|
||||
class TestSoupLock:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.soup_lock import SoupLock
|
||||
|
||||
lock = SoupLock(
|
||||
soup_version="0.67.0",
|
||||
base_model="meta-llama/Llama-3.1-8B",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24T00:00:00Z",
|
||||
)
|
||||
assert lock.soup_version == "0.67.0"
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.soup_lock import SoupLock
|
||||
|
||||
lock = SoupLock(
|
||||
soup_version="0.67.0",
|
||||
base_model="m",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
lock.soup_version = "9.99" # type: ignore[misc]
|
||||
|
||||
def test_sha_validation(self) -> None:
|
||||
from soup_cli.utils.soup_lock import SoupLock
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
SoupLock(
|
||||
soup_version="0.67.0",
|
||||
base_model="m",
|
||||
base_model_sha="too-short",
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24",
|
||||
)
|
||||
|
||||
def test_empty_base_model_rejected(self) -> None:
|
||||
from soup_cli.utils.soup_lock import SoupLock
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
SoupLock(
|
||||
soup_version="0.67.0",
|
||||
base_model="",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24",
|
||||
)
|
||||
|
||||
|
||||
class TestWriteReadLock:
|
||||
def test_roundtrip(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.soup_lock import (
|
||||
SoupLock,
|
||||
read_lock,
|
||||
write_lock,
|
||||
)
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
lock = SoupLock(
|
||||
soup_version="0.67.0",
|
||||
base_model="m",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24",
|
||||
)
|
||||
path = str(tmp_path / "soup.lock")
|
||||
write_lock(lock, path)
|
||||
loaded = read_lock(path)
|
||||
assert loaded == lock
|
||||
|
||||
def test_outside_cwd_rejected(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.soup_lock import SoupLock, write_lock
|
||||
|
||||
cwd = tmp_path / "work"
|
||||
cwd.mkdir()
|
||||
monkeypatch.chdir(cwd)
|
||||
lock = SoupLock(
|
||||
soup_version="0.67.0",
|
||||
base_model="m",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24",
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
write_lock(lock, str(tmp_path / "outside.lock"))
|
||||
|
||||
def test_read_missing(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.soup_lock import read_lock
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
read_lock(str(tmp_path / "missing.lock"))
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="POSIX-only symlink test")
|
||||
def test_read_symlink_rejected(self, tmp_path, monkeypatch) -> None:
|
||||
from soup_cli.utils.soup_lock import read_lock
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
real = tmp_path / "real.lock"
|
||||
real.write_text('{"soup_version": "0.67.0"}', encoding="utf-8")
|
||||
sym = tmp_path / "sym.lock"
|
||||
os.symlink(real, sym)
|
||||
with pytest.raises(ValueError):
|
||||
read_lock(str(sym))
|
||||
|
||||
|
||||
class TestCheckLockDrift:
|
||||
def _make_lock(self, **overrides):
|
||||
from soup_cli.utils.soup_lock import SoupLock
|
||||
|
||||
base = dict(
|
||||
soup_version="0.67.0",
|
||||
base_model="m",
|
||||
base_model_sha="a" * 64,
|
||||
dataset_sha="b" * 64,
|
||||
env_hash="c" * 64,
|
||||
closure_sha="d" * 64,
|
||||
created_at="2026-05-24",
|
||||
)
|
||||
base.update(overrides)
|
||||
return SoupLock(**base)
|
||||
|
||||
def test_no_drift(self) -> None:
|
||||
from soup_cli.utils.soup_lock import check_lock_drift
|
||||
|
||||
a = self._make_lock()
|
||||
b = self._make_lock()
|
||||
drift = check_lock_drift(a, b)
|
||||
assert drift.ok is True
|
||||
assert drift.changes == ()
|
||||
|
||||
def test_base_model_sha_drift(self) -> None:
|
||||
from soup_cli.utils.soup_lock import check_lock_drift
|
||||
|
||||
a = self._make_lock()
|
||||
b = self._make_lock(base_model_sha="f" * 64)
|
||||
drift = check_lock_drift(a, b)
|
||||
assert drift.ok is False
|
||||
assert any("base_model_sha" in c for c in drift.changes)
|
||||
|
||||
def test_dataset_sha_drift(self) -> None:
|
||||
from soup_cli.utils.soup_lock import check_lock_drift
|
||||
|
||||
a = self._make_lock()
|
||||
b = self._make_lock(dataset_sha="f" * 64)
|
||||
drift = check_lock_drift(a, b)
|
||||
assert drift.ok is False
|
||||
assert any("dataset_sha" in c for c in drift.changes)
|
||||
|
||||
def test_env_hash_drift(self) -> None:
|
||||
from soup_cli.utils.soup_lock import check_lock_drift
|
||||
|
||||
a = self._make_lock()
|
||||
b = self._make_lock(env_hash="f" * 64)
|
||||
drift = check_lock_drift(a, b)
|
||||
assert drift.ok is False
|
||||
assert any("env_hash" in c for c in drift.changes)
|
||||
|
||||
def test_non_lock_rejected(self) -> None:
|
||||
from soup_cli.utils.soup_lock import check_lock_drift
|
||||
|
||||
a = self._make_lock()
|
||||
with pytest.raises(TypeError):
|
||||
check_lock_drift(a, "not-a-lock") # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestCliSmoke:
|
||||
def test_lock_help(self) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.lock import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_lock_write_command(self, tmp_path, monkeypatch) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.lock import app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"write",
|
||||
"--base-model", "test-model",
|
||||
"--base-sha", "a" * 64,
|
||||
"--dataset-sha", "b" * 64,
|
||||
"--env-hash", "c" * 64,
|
||||
"--output", "soup.lock",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert (tmp_path / "soup.lock").exists()
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_top_level_heavy_imports(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "soup_lock.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
head_lines = [
|
||||
line
|
||||
for line in src.splitlines()[:50]
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
]
|
||||
head = "\n".join(head_lines)
|
||||
for forbidden in ("import torch", "import transformers", "import peft"):
|
||||
assert forbidden not in head, f"top-level {forbidden!r}"
|
||||
|
||||
def test_uses_atomic_write_helper(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "soup_lock.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
assert "atomic_write_text" in src
|
||||
|
|
@ -0,0 +1,232 @@
|
|||
"""v0.67.0 Part F — ``soup adapters bisect``.
|
||||
|
||||
Binary-search over a training-step history (or dataset-commit history)
|
||||
to find the step that broke an eval.
|
||||
|
||||
Tests for ``soup_cli/utils/adapter_bisect.py``:
|
||||
|
||||
- Frozen ``BisectPlan`` / ``BisectStep`` / ``BisectResult`` dataclasses
|
||||
- ``build_bisect_plan(history)`` validates input + computes initial mid
|
||||
- ``run_bisect(plan, eval_fn)`` runs the binary search
|
||||
- ``bisect_next_step(state)`` pure step kernel (one iteration)
|
||||
- CLI smoke
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestPublicSurface:
|
||||
def test_module_importable(self) -> None:
|
||||
from soup_cli.utils import adapter_bisect
|
||||
|
||||
assert hasattr(adapter_bisect, "BisectPlan")
|
||||
assert hasattr(adapter_bisect, "BisectStep")
|
||||
assert hasattr(adapter_bisect, "BisectResult")
|
||||
assert hasattr(adapter_bisect, "build_bisect_plan")
|
||||
assert hasattr(adapter_bisect, "run_bisect")
|
||||
assert hasattr(adapter_bisect, "bisect_next_step")
|
||||
|
||||
|
||||
class TestBisectPlan:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
plan = BisectPlan(history=("ckpt-100", "ckpt-200", "ckpt-300"))
|
||||
assert len(plan.history) == 3
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
plan = BisectPlan(history=("a", "b"))
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
plan.history = () # type: ignore[misc]
|
||||
|
||||
def test_too_short_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
BisectPlan(history=("only-one",))
|
||||
|
||||
def test_must_be_tuple(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
BisectPlan(history=["a", "b"]) # type: ignore[arg-type]
|
||||
|
||||
def test_entries_must_be_str(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
BisectPlan(history=("a", 2)) # type: ignore[arg-type]
|
||||
|
||||
def test_empty_entry_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
BisectPlan(history=("a", ""))
|
||||
|
||||
def test_null_byte_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
BisectPlan(history=("a", "b\x00"))
|
||||
|
||||
def test_duplicate_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
BisectPlan(history=("a", "b", "a"))
|
||||
|
||||
def test_oversize_history_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import MAX_HISTORY, BisectPlan
|
||||
|
||||
too_many = tuple(f"ckpt-{i}" for i in range(MAX_HISTORY + 1))
|
||||
with pytest.raises(ValueError):
|
||||
BisectPlan(history=too_many)
|
||||
|
||||
|
||||
class TestBisectStep:
|
||||
def test_construct(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectStep
|
||||
|
||||
step = BisectStep(checkpoint="ckpt-100", ok=True)
|
||||
assert step.checkpoint == "ckpt-100"
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectStep
|
||||
|
||||
step = BisectStep(checkpoint="ckpt-1", ok=True)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
step.ok = False # type: ignore[misc]
|
||||
|
||||
def test_ok_must_be_bool(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectStep
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
BisectStep(checkpoint="ckpt", ok=1) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestBuildBisectPlan:
|
||||
def test_happy(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import build_bisect_plan
|
||||
|
||||
plan = build_bisect_plan(["a", "b", "c", "d"])
|
||||
assert len(plan.history) == 4
|
||||
|
||||
|
||||
class TestBisectNextStep:
|
||||
def test_picks_midpoint(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan, bisect_next_step
|
||||
|
||||
plan = BisectPlan(history=("a", "b", "c", "d", "e"))
|
||||
nxt = bisect_next_step(plan, lo=0, hi=4)
|
||||
assert nxt == 2
|
||||
|
||||
def test_invalid_range(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan, bisect_next_step
|
||||
|
||||
plan = BisectPlan(history=("a", "b"))
|
||||
with pytest.raises(ValueError):
|
||||
bisect_next_step(plan, lo=5, hi=0)
|
||||
|
||||
|
||||
class TestRunBisect:
|
||||
def test_finds_first_broken(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect
|
||||
|
||||
# Eval says: first three checkpoints pass, last three fail.
|
||||
# The bisect should find the boundary at index 3 (first failing).
|
||||
plan = BisectPlan(
|
||||
history=("c0", "c1", "c2", "c3", "c4", "c5")
|
||||
)
|
||||
|
||||
def eval_fn(checkpoint: str) -> bool:
|
||||
return checkpoint in ("c0", "c1", "c2")
|
||||
|
||||
result = run_bisect(plan, eval_fn=eval_fn)
|
||||
assert result.first_broken == "c3"
|
||||
|
||||
def test_all_ok(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect
|
||||
|
||||
plan = BisectPlan(history=("c0", "c1", "c2"))
|
||||
result = run_bisect(plan, eval_fn=lambda _: True)
|
||||
# All pass: first_broken is None
|
||||
assert result.first_broken is None
|
||||
assert result.verdict == "ALL_OK"
|
||||
|
||||
def test_all_broken(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect
|
||||
|
||||
plan = BisectPlan(history=("c0", "c1", "c2"))
|
||||
result = run_bisect(plan, eval_fn=lambda _: False)
|
||||
# All fail: first_broken is c0
|
||||
assert result.first_broken == "c0"
|
||||
|
||||
def test_eval_fn_must_be_callable(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect
|
||||
|
||||
plan = BisectPlan(history=("a", "b"))
|
||||
with pytest.raises(TypeError):
|
||||
run_bisect(plan, eval_fn=None) # type: ignore[arg-type]
|
||||
|
||||
def test_non_plan_rejected(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import run_bisect
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
run_bisect("not-a-plan", eval_fn=lambda _: True) # type: ignore[arg-type]
|
||||
|
||||
def test_logs_history_of_probes(self) -> None:
|
||||
from soup_cli.utils.adapter_bisect import BisectPlan, run_bisect
|
||||
|
||||
plan = BisectPlan(
|
||||
history=("c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7")
|
||||
)
|
||||
|
||||
probed: list[str] = []
|
||||
|
||||
def eval_fn(checkpoint: str) -> bool:
|
||||
probed.append(checkpoint)
|
||||
# Boundary at c4
|
||||
return int(checkpoint[1:]) < 4
|
||||
|
||||
result = run_bisect(plan, eval_fn=eval_fn)
|
||||
assert result.first_broken == "c4"
|
||||
# Probes are logged in `result.steps`
|
||||
assert len(result.steps) == len(probed)
|
||||
# ~log2(8)=3 midpoint probes plus 2 endpoint probes = 5 total
|
||||
assert len(result.steps) <= 6
|
||||
|
||||
|
||||
class TestCliSmoke:
|
||||
def test_bisect_help(self) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.commands.adapters import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["bisect", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "bisect" in result.output.lower()
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_top_level_heavy_imports(self) -> None:
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "adapter_bisect.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
head_lines = [
|
||||
line
|
||||
for line in src.splitlines()[:50]
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
]
|
||||
head = "\n".join(head_lines)
|
||||
for forbidden in ("import torch", "import transformers", "import peft"):
|
||||
assert forbidden not in head, f"top-level {forbidden!r}"
|
||||
Loading…
Reference in New Issue