mirror of https://github.com/razor-ai/soup.git
feat(v0.70.0): Loop Hardening — reward-hacking + ULD + MiniLLM + RL ckpt + iterative DPO + echo-trap
Six-part schema-only release shipping the axis-3 + axis-13 training-loop hardening. Every live trainer-callback / math kernel is deferred to v0.70.1 per the project's established stub-then-live cadence (matches v0.50.0 / v0.62.0 / v0.69.0). Part A — Reward-hacking detector (soup_cli/utils/reward_hacking.py): InfoRM Cluster-Separation Index (Wang et al. 2024 arXiv:2402.09345) + RM-ensemble pairwise variance + OK/WARN/HACK taxonomy at 0.10/0.30 thresholds. TrainingConfig.reward_hack_detector / reward_hack_halt; SoupConfig task-gate (grpo/ppo only, mlx rejected, halt requires detector). Part B — Cross-tokenizer ULD (soup_cli/utils/uld.py): Universal Logit Distillation (Boizard et al. 2024 arXiv:2402.12030). wasserstein + topk_align allowlist; ULDConfig frozen with topk/strategy cross-validators; vocab-size cap 262144. Schema-gated to task='distill'. Part C — MiniLLM reverse-KL on-policy distillation (soup_cli/utils/minillm.py): Gu et al. 2024 (arXiv:2306.08543) — bundles teacher-mixed sampling + length-norm + pretrain-loss anchor stability tricks. Anchor weight↔path mutual-requirement cross-validators reject silent no-op combos. Part D — Mid-epoch RL checkpoint (soup_cli/utils/rl_checkpoint.py): Optimizer-state serialization TorchTune explicitly punts. RLCheckpointConfig + RLCheckpointState frozen + JSON-serialisable manifest. RL-task gate. Part E — Iterative DPO loop driver (soup_cli/utils/iterative_dpo.py + commands/iterative_dpo.py): sample → RM-score → re-pair → retrain over N rounds. IterativeDPOPlan with consecutive-round_index invariant. New `soup iterative-dpo` CLI; --plan-only live, runner deferred. Part F — RAGEN echo-trap detector (soup_cli/utils/echo_trap.py): Zhu et al. 2025 (arXiv:2504.14437) — n-gram trajectory-repetition kernels with DoS caps (max 32 ngram_n, 1M tokens, 100k trajectories). OK/WARN/TRAP at 0.30/0.60. Composes with v0.53.11 #127 GRPOStabilityCallback. Cross-cutting hardening: - 6 new util modules + 1 new top-level CLI + 11 new TrainingConfig fields + 6 new SoupConfig cross-validators + 3 new field validators - Closed allowlists (frozenset) + MappingProxyType registries everywhere - Frozen dataclasses with post-init validation on every public record - Bool-as-int rejection on every numeric (matches v0.30.0 / v0.41.0 policy) - math.isfinite NaN/Inf rejection on every float - Null-byte rejection + per-field length caps on every string - No top-level torch imports (4 source-grep regression tests) - Deferred-live stubs validate inputs FIRST then raise NotImplementedError with explicit v0.70.1 marker - CLI exit codes split: 2 = validation rejection, 3 = deferred-live Test count: 11487 → 11824 (+337 net). 12-invariant self-review against the full project checklist (closed allowlists, frozen dataclasses, MappingProxyType, bool-as-int rejection, finite check, null-byte, length caps, no top-level torch, TypeError/ValueError split, deferred-live, tuples-not-lists, CLI exit codes) all green across all 6 Parts. Manual CPU smokes (Step 6): every CLI happy + failure path exercised — `soup iterative-dpo --plan-only` 3-round plan rendered end-to-end with per-round artifacts; 5 happy-path YAML loads + 5 failure-mode rejections across reward_hack / uld / minillm / rl_checkpoint / echo_trap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
49943a5af6
commit
74edac95d1
|
|
@ -111,7 +111,7 @@ soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (262 files, 11487 tests)
|
||||
tests/ - Test suite (268 files, 11824 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
62
README.md
62
README.md
|
|
@ -42,14 +42,15 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.69.0 — Data Engineering Pro: `soup build` (dbt-for-SFT) + `soup expect` + `soup data gen-magpie` + `soup data persona-mix` + `soup data brain-rot`.** Five surfaces that turn dataset preparation into a first-class engineering workflow. `soup build` is a dbt-style DAG of dataset transforms with `ref()`-connected models, `incremental` materialization, and content-hash-based row-diff so re-runs re-tokenize only changed rows. `soup expect` ships Great-Expectations for chat data — `expect_no_pii` / `expect_token_length_between` / `expect_no_refusal_pattern` / `expect_chosen_preferred_over_rejected_by_judge` — with exit code 3 on suite failure so CI pipelines can gate on dataset quality regressions. `soup data gen-magpie` plans the Magpie synthetic generator (chat-template-prefix harvest, reuses v0.20 providers). `soup data persona-mix` samples a prompt × persona × style matrix with a bundled 12-persona / 5-style diversity set + topic-entropy metric. `soup data brain-rot` implements the arXiv 2510.13928 detector with OK/MINOR/MAJOR verdicts that compose with v0.47 educational scorer; `--strict` mode exits 3 when too many rows score MAJOR.
|
||||
**v0.70.0 — Loop Hardening: reward-hacking detector + cross-tokenizer ULD + MiniLLM reverse-KL + mid-epoch RL checkpoint + iterative DPO + RAGEN echo-trap.** Six surfaces that protect the training loop from the failure modes that cost a real GPU-hour. The `reward_hack_detector` flag wires InfoRM cluster-separation OR RM-ensemble divergence into GRPO/PPO so the policy doesn't silently game the reward. `uld_strategy: wasserstein|topk_align` extends v0.53.2 distillation to teacher/student pairs with different vocabularies (Llama → Mistral, Llama → Qwen). `minillm_enabled` bundles MiniLLM's three stability tricks (teacher-mixed sampling + length-norm + pretrain-loss anchor). `rl_checkpoint_save_every_steps` adds the optimizer-state serialization TorchTune explicitly punts. `soup iterative-dpo --rounds N` is the sample → RM-score → re-pair → retrain loop driver. `echo_trap_enabled` detects trajectory degeneration in multi-turn agent RL (RAGEN-style). Schema-only release — live trainer-callback / math kernels deferred to v0.70.1.
|
||||
|
||||
- **`soup build <manifest.yaml> [--dry-run]`** — dbt-for-SFT DAG parser + topological sort + plan rendering. Each model declares `kind: incremental|table|view`, a `transform`, and either a `source:` (seed) or `refs: [...]` (derived). Cross-validators reject ambiguous shapes (no-refs + no-source = degenerate; refs + source = mutually exclusive). `compute_row_hash` + `incremental_diff(prev, new) -> IncrementalDiffReport(added, changed, removed, unchanged)` are the kernel that lets the live runner (v0.69.1) re-tokenize only changed rows. `BuildModel` and `BuildPlan` are frozen dataclasses; `load_build_yaml` delegates to `paths.enforce_under_cwd_and_no_symlink` (TOCTOU defence). Live runner deferred to v0.69.1; today `--dry-run` renders the plan + exits 0, no flag exits 3 with the deferred-live marker.
|
||||
- **`soup expect <data.jsonl> <suite.yaml>`** — Great Expectations for chat data. Four built-in expectation functions, each a pure function returning a frozen `ExpectationResult`. Composes with v0.47.0 `data_score.detect_pii` (Presidio backend when `[data-pro]` installed), v0.56.0 `diagnose.refusal.looks_like_refusal`, and v0.19.0 judge backends (operator-injected callable for the chosen-vs-rejected judge). `_dispatch_expectation` passes args through raw so per-expectation validators (bool-rejection, NaN-rejection, range checks) fire authentically — no silent int/float coercion bypasses the validator. Exit 0 = pass, 2 = validation rejection, 3 = suite failure.
|
||||
- **`soup data gen-magpie --base <m> --provider ollama|anthropic|vllm --target N [--plan-only]`** — Magpie technique (Xu et al. 2024) schema + plan. Feeds the chat-template prefix only to an aligned base model and harvests user-side turns via v0.20 providers. `MagpieConfig` frozen, `validate_magpie_provider` closed allowlist, `validate_target_rows` ∈ [1, 1_000_000] bool-rejected. Live generation loop (provider calls + v0.47 quality filter) deferred to v0.69.1.
|
||||
- **`soup data persona-mix --prompts <jsonl> --n N --output <jsonl>`** — Persona-Hub-style diversity sampler. Reads `--prompts` JSONL, multiplies through bundled 12-persona × 5-style matrix (or operator-supplied `--personas` / `--styles` JSONL), writes `{prompt, persona, style}` per row via atomic `tempfile.mkstemp + os.replace`. Deterministic by `--seed`. New `compute_topic_diversity` Shannon-entropy kernel for downstream gating. Centralised TOCTOU policy via `enforce_under_cwd_and_no_symlink` on every read AND write path.
|
||||
- **`soup data brain-rot <data.jsonl> [--strict] [--max-major-fraction 0.25]`** — arXiv 2510.13928 brain-rot detector. Two orthogonal slop scorers: `score_triviality` (low diversity / excessive `!!`/`??` punctuation / `lol/omg/lmao` density / length penalty) and `score_popularity_signal` (clickbait phrase substrings + emoji density). Per-row composite = `1.0 - max(triviality, popularity)`. Same OK/MINOR/MAJOR taxonomy as v0.26 / v0.56 / v0.65 (≥0.85 OK, ≥0.60 MINOR, else MAJOR). `--strict` exits 3 when MAJOR-row fraction exceeds the threshold so training pipelines can refuse to materialise slop datasets. `--max-major-fraction` validated at the CLI boundary (NaN / Inf / out-of-range / bool rejected) before any scoring.
|
||||
- **+264 new tests** (11225 → 11487) across 5 part files. Review-fix coverage: 1 CRITICAL (centralised duplicate cwd+symlink blocks behind `paths.enforce_under_cwd_and_no_symlink` in build_dag / expectations / expect.py — code-review HIGH) + 4 HIGH (brain-rot loader DoS caps; persona-mix `--output` symlink TOCTOU rejection; persona-mix `_load_jsonl_field` DoS caps; magpie `quality_filter` validator + `_dispatch_expectation` int/float-coercion bypass) + 5 MEDIUM (BuildModel seed/derived cross-validator + docs; `--max-major-fraction` CLI-boundary validation; `expect` skipped-line WARNING) + 4 LOW (boundary tests at exact threshold ± ε on classify_brain_rot; BuildPlan FrozenInstanceError; version_bumped checks in B/C/D/E; lazy-yaml import source-grep). Manual CPU smokes (Step 6): every CLI happy + failure path exercised, including 3-stage build DAG dry-run, PII suite exit 3, magpie plan-only, persona-mix atomic write to JSONL, brain-rot MAJOR-on-slop exit 3.
|
||||
- **`soup train --reward-hack-detector info_rm|rm_ensemble`** — early-warning when the policy starts gaming the reward model. `info_rm` tracks the InfoRM Cluster-Separation Index across training (Wang et al. 2024, arXiv 2402.09345); a sharp drop signals the RM losing its grip on the (good, bad) split. `rm_ensemble` tracks pairwise variance across an RM ensemble; rising disagreement = unreliable reward signal. `--reward-hack-halt` auto-stops training on HACK verdict (≥30% relative drop). Composes with v0.34 `soup why` so the anomaly explainer can name reward-hacking specifically rather than "loss plateau". Schema + math kernels live now (`compute_cluster_separation`, `compute_rm_ensemble_divergence`, `classify_hack_signal` with OK/WARN/HACK bands at 0.10 / 0.30); live HF Trainer callback in v0.70.1.
|
||||
- **`soup train --uld-strategy wasserstein|topk_align`** — cross-tokenizer distillation (Boizard et al. 2024, arXiv 2402.12030). The v0.53.2 distillation path assumes student and teacher share a vocabulary; the moment vocabs differ, column-wise logit alignment breaks. `wasserstein` computes 1D Wasserstein distance between sorted teacher/student logit distributions — no alignment required. `topk_align` picks top-K teacher logits and maps to student token ids via BPE overlap. Bounded vocab sizes [1, 262144] cover multilingual SentencePiece + GPT-OSS 200K. Live projection module wired in v0.70.1.
|
||||
- **`soup train --minillm-enabled`** — MiniLLM-style reverse-KL on-policy distillation (Gu et al. 2024, arXiv 2306.08543). Bundles the three stability tricks scattered across §3 of the paper: teacher-mixed sampling (epsilon-greedy mix with `--minillm-teacher-mix-ratio 0.3`), length normalisation on rollouts (`--minillm-length-normalize true`), and a small pretrain-loss anchor (`--minillm-pretrain-anchor-weight 0.1` requires `--minillm-pretrain-anchor-path pre.jsonl`). Cross-validators reject silent no-op combos (anchor_weight=0 + anchor_path set, anchor_weight > 0 + path None). Live callback in v0.70.1.
|
||||
- **`soup train --rl-checkpoint-save-every-steps N`** — mid-epoch checkpoint for PPO/GRPO. TorchTune explicitly punts this; Soup ships the real save_state / load_state surface here. Captures optimizer state (`--rl-checkpoint-include-optimizer`), optional ref-model state, optional rollout/replay buffer. `--rl-checkpoint-keep-last 3` retains the last N. Composes with v0.32 spike recovery + v0.40.0 ref-model regen — a recovered run hops back to the most recent mid-epoch ckpt instead of restarting the epoch. Live save_state / load_state in v0.70.1.
|
||||
- **`soup iterative-dpo --rounds N --pairs-per-round 500 [--plan-only]`** — sample → RM-score → re-pair → retrain over N rounds. Frozen `IterativeDPOPlan` with consecutive-round_index invariant + per-round artifact paths (`./out/round-NN/pairs.jsonl`, `./out/round-NN/adapter`). `--plan-only` renders the canonical plan + exits 0; without it the deferred-live runner exits 3 with explicit v0.70.1 marker. Recipe glue around existing TRL primitives; v0.70.1 wires the live `soup train --task dpo` subprocess loop.
|
||||
- **`soup train --echo-trap-enabled --echo-trap-threshold 0.6 --echo-trap-halt`** — RAGEN-style detection of trajectory degeneration during multi-turn agent RL (Zhu et al. 2025, arXiv 2504.14437). Pure-Python n-gram repetition rate per trajectory + batch mean. OK / WARN / TRAP taxonomy at 0.30 / 0.60. Composes with v0.53.11 #127 `GRPOStabilityCallback` — both detectors fire in the same training step without duplicating trajectory collection. Live HF Trainer callback in v0.70.1.
|
||||
- **+337 new tests** (11487 → 11824) across 6 part files. Schema-only release; every live callback / kernel raises `NotImplementedError` with explicit v0.70.1 marker after validating inputs (matches the project's stub-then-live cadence from v0.50.0 / v0.62.0 / v0.69.0). 12-invariant self-review against the full project checklist (closed allowlists, frozen dataclasses, MappingProxyType registries, bool-as-int rejection, math.isfinite NaN/Inf reject, null-byte rejection, length caps, no top-level torch, TypeError/ValueError split, deferred-live policy, tuples-not-lists on frozen collections, CLI exit codes) — all 12 satisfied. Manual CPU smoke (Step 6): `soup iterative-dpo --plan-only` 3-round plan rendered end-to-end; 5 happy + 5 failure-mode YAML round-trips across every new schema field.
|
||||
|
||||
## Data Engineering Pro
|
||||
|
||||
|
|
@ -88,6 +89,53 @@ soup data brain-rot data.jsonl --strict --max-major-fraction 0.10
|
|||
|
||||
Every command applies the project-wide TOCTOU policy (`os.lstat + S_ISLNK` symlink rejection before any open) and cwd containment via the shared `paths.enforce_under_cwd_and_no_symlink` helper. Live runners for `soup build` and `soup data gen-magpie` land in v0.69.1; the other three are LIVE today.
|
||||
|
||||
## Loop Hardening
|
||||
|
||||
The v0.70.0 release ships 6 surfaces that protect the training loop from the failure modes that cost a real GPU-hour. Schema + math kernels live now; live trainer-callback wiring lands in v0.70.1 (matches the project's stub-then-live cadence).
|
||||
|
||||
```bash
|
||||
# Reward-hacking detector — auto-halt when the policy starts gaming the RM
|
||||
# (InfoRM cluster-separation index, Wang et al. 2024 arXiv:2402.09345)
|
||||
soup train --config soup.yaml \
|
||||
--reward-hack-detector info_rm --reward-hack-halt # halt on HACK verdict
|
||||
|
||||
# Cross-tokenizer distillation — Llama -> Mistral, no shared vocab needed
|
||||
# (Universal Logit Distillation, Boizard et al. 2024 arXiv:2402.12030)
|
||||
soup train --config soup.yaml --uld-strategy wasserstein
|
||||
|
||||
# MiniLLM reverse-KL on-policy distillation — bundles 3 stability tricks
|
||||
# (Gu et al. 2024 arXiv:2306.08543)
|
||||
soup train --config soup.yaml --minillm-enabled \
|
||||
--minillm-teacher-mix-ratio 0.3 \
|
||||
--minillm-pretrain-anchor-weight 0.1 \
|
||||
--minillm-pretrain-anchor-path ./pretrain.jsonl
|
||||
|
||||
# Mid-epoch checkpoint for PPO/GRPO — TorchTune punts this; Soup ships it
|
||||
soup train --config grpo.yaml \
|
||||
--rl-checkpoint-save-every-steps 500 \
|
||||
--rl-checkpoint-keep-last 3 \
|
||||
--rl-checkpoint-include-optimizer
|
||||
|
||||
# Iterative DPO loop driver — sample -> RM-score -> re-pair -> retrain
|
||||
soup iterative-dpo \
|
||||
--base-model meta-llama/Llama-3.1-8B \
|
||||
--reward-model ./output_rm \
|
||||
--prompts ./prompts.jsonl \
|
||||
--output-dir ./iterative_dpo_out \
|
||||
--rounds 5 \
|
||||
--pairs-per-round 1000 \
|
||||
--plan-only
|
||||
|
||||
# RAGEN echo-trap detector — auto-halt when trajectories collapse to self-repetition
|
||||
# (Zhu et al. 2025 arXiv:2504.14437)
|
||||
soup train --config grpo.yaml \
|
||||
--echo-trap-enabled \
|
||||
--echo-trap-threshold 0.6 \
|
||||
--echo-trap-halt
|
||||
```
|
||||
|
||||
Every detector composes with v0.34 `soup why` (anomaly explainer), v0.32 spike recovery, and v0.53.11 #127 `GRPOStabilityCallback` so a single training run can have InfoRM + echo-trap + spike-recovery + ref-model regen all active simultaneously without duplicating trajectory / state collection. Live trainer-callback wiring for all 6 Parts lands in v0.70.1 (`build_reward_hack_callback`, `build_uld_projection`, `build_minillm_callback`, `build_rl_checkpoint_callback`, `run_iterative_dpo`, `build_echo_trap_callback`); today every CLI / config flag is validated at schema-load so misconfigured runs fail loudly at config-load time.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.69.0"
|
||||
version = "0.70.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.69.0"
|
||||
__version__ = "0.70.0"
|
||||
|
|
|
|||
|
|
@ -467,6 +467,15 @@ app.command(
|
|||
help="Run an expectations suite against a JSONL dataset (v0.69.0 Part B).",
|
||||
)(_expect_cmd.expect_cmd)
|
||||
|
||||
# v0.70.0 Part E — `soup iterative-dpo` (iterative DPO loop driver).
|
||||
from soup_cli.commands import iterative_dpo as _iterative_dpo_cmd # noqa: E402
|
||||
|
||||
app.add_typer(
|
||||
_iterative_dpo_cmd.app,
|
||||
name="iterative-dpo",
|
||||
help="Iterative DPO loop driver (v0.70.0 Part E).",
|
||||
)
|
||||
|
||||
|
||||
def _rewrite_advise_argv(argv: list) -> list:
|
||||
"""Inject `run` between `advise` and a non-subcommand first argument.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,95 @@
|
|||
"""soup iterative-dpo — Iterative DPO loop driver — v0.70.0 Part E.
|
||||
|
||||
Sample → RM-score → re-pair → retrain over N rounds. The live runner
|
||||
is deferred to v0.70.1; v0.70.0 ships the schema + ``--plan-only``
|
||||
renderer that prints the canonical per-round artifacts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
|
||||
console = Console()
|
||||
|
||||
app = typer.Typer(
|
||||
no_args_is_help=True,
|
||||
help="Iterative DPO loop driver (v0.70.0 Part E)",
|
||||
)
|
||||
|
||||
|
||||
@app.callback(invoke_without_command=True)
|
||||
def main(
|
||||
base_model: str = typer.Option(..., "--base-model", help="HF id / local path"),
|
||||
reward_model: str = typer.Option(..., "--reward-model", help="RM HF id / path"),
|
||||
prompts: str = typer.Option(..., "--prompts", help="Source prompts JSONL"),
|
||||
output_dir: str = typer.Option(..., "--output-dir", help="Output dir"),
|
||||
rounds: int = typer.Option(3, "--rounds", help="Number of iterative-DPO rounds"),
|
||||
pairs_per_round: int = typer.Option(
|
||||
500,
|
||||
"--pairs-per-round",
|
||||
help="Number of (chosen, rejected) pairs per round",
|
||||
),
|
||||
plan_only: bool = typer.Option(
|
||||
False,
|
||||
"--plan-only",
|
||||
help="Print the resolved plan and exit (no training).",
|
||||
),
|
||||
):
|
||||
"""Render the iterative-DPO plan and (in v0.70.1) execute it."""
|
||||
from soup_cli.utils.iterative_dpo import (
|
||||
build_iterative_dpo_plan,
|
||||
run_iterative_dpo,
|
||||
)
|
||||
|
||||
try:
|
||||
plan = build_iterative_dpo_plan(
|
||||
base_model=base_model,
|
||||
reward_model=reward_model,
|
||||
prompts_path=prompts,
|
||||
output_dir=output_dir,
|
||||
rounds=rounds,
|
||||
pairs_per_round=pairs_per_round,
|
||||
)
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]Error:[/red] {escape(str(exc))}")
|
||||
raise typer.Exit(code=2) from exc
|
||||
|
||||
table = Table(title="Iterative-DPO plan")
|
||||
table.add_column("Round")
|
||||
table.add_column("Pairs path")
|
||||
table.add_column("Adapter path")
|
||||
table.add_column("Pairs")
|
||||
for r in plan.rounds:
|
||||
table.add_row(
|
||||
str(r.round_index),
|
||||
escape(r.pairs_path),
|
||||
escape(r.adapter_path),
|
||||
str(r.pairs_count),
|
||||
)
|
||||
console.print(table)
|
||||
|
||||
if plan_only:
|
||||
console.print(
|
||||
Panel(
|
||||
"[green]Plan rendered[/green] (--plan-only). To execute, "
|
||||
"drop the flag once v0.70.1 ships.",
|
||||
title="Iterative DPO",
|
||||
)
|
||||
)
|
||||
raise typer.Exit(code=0)
|
||||
|
||||
try:
|
||||
run_iterative_dpo(plan)
|
||||
except NotImplementedError as exc:
|
||||
console.print(
|
||||
Panel(
|
||||
"[yellow]Live runner deferred to v0.70.1.[/yellow] "
|
||||
f"{escape(str(exc))}",
|
||||
title="Iterative DPO",
|
||||
)
|
||||
)
|
||||
raise typer.Exit(code=3) from exc
|
||||
|
|
@ -1341,6 +1341,244 @@ class TrainingConfig(BaseModel):
|
|||
),
|
||||
)
|
||||
|
||||
# ---- v0.70.0 Part F — Echo-trap detector -----------------------------
|
||||
echo_trap_enabled: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Enable RAGEN-style echo-trap detection during multi-turn "
|
||||
"agent RL. Requires task in {'grpo', 'ppo'} on a non-mlx "
|
||||
"backend. Schema-only in v0.70.0; live callback in v0.70.1."
|
||||
),
|
||||
)
|
||||
echo_trap_threshold: float = Field(
|
||||
default=0.6,
|
||||
ge=0.0,
|
||||
le=1.0,
|
||||
description=(
|
||||
"Threshold on the aggregate echo signal. Above this = TRAP. "
|
||||
"Bounded [0.0, 1.0]. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
echo_trap_halt: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Auto-halt training on TRAP verdict. Requires "
|
||||
"echo_trap_enabled=True. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
|
||||
# ---- v0.70.0 Part D — Mid-epoch RL checkpoint ------------------------
|
||||
rl_checkpoint_save_every_steps: Optional[int] = Field(
|
||||
default=None,
|
||||
ge=1,
|
||||
le=10_000_000,
|
||||
description=(
|
||||
"Save an RL-aware mid-epoch checkpoint every N steps. None "
|
||||
"= use HF Trainer's per-epoch checkpoint only. Requires "
|
||||
"task in {'grpo', 'ppo'}. Schema-only in v0.70.0; live "
|
||||
"save_state / load_state in v0.70.1."
|
||||
),
|
||||
)
|
||||
rl_checkpoint_keep_last: int = Field(
|
||||
default=3,
|
||||
ge=1,
|
||||
le=100,
|
||||
description=(
|
||||
"Number of recent RL checkpoints to retain. Older ones are "
|
||||
"pruned at write time. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
rl_checkpoint_include_optimizer: bool = Field(
|
||||
default=True,
|
||||
description=(
|
||||
"Include AdamW / Lion optimizer state in the mid-epoch RL "
|
||||
"checkpoint. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
rl_checkpoint_include_ref_model: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Include the frozen reference model state in the RL "
|
||||
"checkpoint. Default False (ref model is reconstructable "
|
||||
"from cfg.base). (v0.70.0)"
|
||||
),
|
||||
)
|
||||
rl_checkpoint_include_rollout_buffer: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Include the rollout / replay buffer in the RL checkpoint "
|
||||
"so resumed runs don't lose collected experience. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
|
||||
# ---- v0.70.0 Part C — MiniLLM reverse-KL on-policy distillation -------
|
||||
# Bundles teacher-mixed sampling + length-norm + pretrain anchor.
|
||||
# Schema-only; live callback wired in v0.70.1.
|
||||
minillm_enabled: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Enable MiniLLM-style on-policy distillation (Gu et al. 2024). "
|
||||
"Requires task='distill' on a non-mlx backend. v0.70.0 "
|
||||
"schema-only; live callback in v0.70.1."
|
||||
),
|
||||
)
|
||||
minillm_teacher_mix_ratio: float = Field(
|
||||
default=0.0,
|
||||
ge=0.0,
|
||||
le=1.0,
|
||||
description=(
|
||||
"Probability of sampling from the teacher distribution at "
|
||||
"rollout time. 0.0 = student-only; 1.0 = teacher-only. "
|
||||
"Typical range 0.2-0.5. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
minillm_length_normalize: bool = Field(
|
||||
default=True,
|
||||
description=(
|
||||
"Length-normalise the rollout log-probability before the "
|
||||
"reverse-KL term. Prevents long completions from dominating "
|
||||
"the gradient. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
minillm_pretrain_anchor_weight: float = Field(
|
||||
default=0.0,
|
||||
ge=0.0,
|
||||
le=1.0,
|
||||
description=(
|
||||
"Weight on the pretrain-loss anchor term (SFT on a small "
|
||||
"pretrain corpus). Prevents drift away from coherent "
|
||||
"language. Requires minillm_pretrain_anchor_path when > 0. "
|
||||
"(v0.70.0)"
|
||||
),
|
||||
)
|
||||
minillm_pretrain_anchor_path: Optional[str] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Path to the pretrain JSONL used by the anchor term. "
|
||||
"Required when minillm_pretrain_anchor_weight > 0. "
|
||||
"Null-byte rejected; capped at 4096 chars. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
|
||||
# ---- v0.70.0 Part B — Cross-tokenizer ULD ----------------------------
|
||||
# Universal Logit Distillation (Boizard et al. 2024). Schema-only;
|
||||
# live projection module wired in v0.70.1.
|
||||
uld_strategy: Optional[Literal["wasserstein", "topk_align"]] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Cross-tokenizer distillation strategy: 'wasserstein' "
|
||||
"(no alignment needed) or 'topk_align' (requires uld_top_k). "
|
||||
"Requires task='distill' on a non-mlx backend. Schema-only "
|
||||
"in v0.70.0; live projection wired in v0.70.1."
|
||||
),
|
||||
)
|
||||
uld_top_k: Optional[int] = Field(
|
||||
default=None,
|
||||
ge=1,
|
||||
le=262144,
|
||||
description=(
|
||||
"Top-K teacher logits to align (uld_strategy='topk_align' "
|
||||
"only). Bounded [1, 262144] to cap pathological vocabs. "
|
||||
"(v0.70.0)"
|
||||
),
|
||||
)
|
||||
# ---- v0.70.0 Part A — Reward-hacking detector ------------------------
|
||||
# Schema-only release; live HF Trainer callback wired in v0.70.1.
|
||||
reward_hack_detector: Optional[Literal["info_rm", "rm_ensemble"]] = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Reward-hacking detector for GRPO/PPO. 'info_rm' tracks "
|
||||
"InfoRM cluster-separation across training; 'rm_ensemble' "
|
||||
"tracks pairwise variance across an RM ensemble. Requires "
|
||||
"task in {'grpo', 'ppo'} on a non-mlx backend. Schema-only "
|
||||
"in v0.70.0; live HF Trainer callback wired in v0.70.1."
|
||||
),
|
||||
)
|
||||
reward_hack_halt: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Auto-halt training on HACK verdict (drop_pct >= 30% in "
|
||||
"cluster separation). Requires reward_hack_detector to be "
|
||||
"set. (v0.70.0)"
|
||||
),
|
||||
)
|
||||
|
||||
@field_validator("reward_hack_halt", mode="before")
|
||||
@classmethod
|
||||
def _validate_reward_hack_halt(cls, v):
|
||||
"""v0.70.0 — explicit bool guard so YAML ``yes`` / ``1`` integers
|
||||
cannot silently coerce. Matches project bool-before-int policy.
|
||||
"""
|
||||
if v is None:
|
||||
return v
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
raise TypeError(
|
||||
f"reward_hack_halt must be bool, got {type(v).__name__}"
|
||||
)
|
||||
|
||||
@field_validator(
|
||||
"echo_trap_enabled",
|
||||
"echo_trap_halt",
|
||||
mode="before",
|
||||
)
|
||||
@classmethod
|
||||
def _validate_echo_trap_bool_fields(cls, v):
|
||||
"""v0.70.0 Part F — bool guards for echo-trap toggles."""
|
||||
if v is None:
|
||||
return v
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
raise TypeError(
|
||||
f"v0.70.0 echo-trap flag must be bool, got {type(v).__name__}"
|
||||
)
|
||||
|
||||
@field_validator(
|
||||
"rl_checkpoint_include_optimizer",
|
||||
"rl_checkpoint_include_ref_model",
|
||||
"rl_checkpoint_include_rollout_buffer",
|
||||
mode="before",
|
||||
)
|
||||
@classmethod
|
||||
def _validate_rl_checkpoint_bool_fields(cls, v):
|
||||
"""v0.70.0 Part D — bool guards for RL-checkpoint toggles."""
|
||||
if v is None:
|
||||
return v
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
raise TypeError(
|
||||
f"v0.70.0 RL-checkpoint flag must be bool, got {type(v).__name__}"
|
||||
)
|
||||
|
||||
@field_validator(
|
||||
"minillm_enabled",
|
||||
"minillm_length_normalize",
|
||||
mode="before",
|
||||
)
|
||||
@classmethod
|
||||
def _validate_minillm_bool_fields(cls, v):
|
||||
"""v0.70.0 Part C — bool guards for MiniLLM toggles."""
|
||||
if v is None:
|
||||
return v
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
raise TypeError(
|
||||
f"v0.70.0 MiniLLM flag must be bool, got {type(v).__name__}"
|
||||
)
|
||||
|
||||
@field_validator("minillm_pretrain_anchor_path")
|
||||
@classmethod
|
||||
def _validate_minillm_anchor_path(cls, v):
|
||||
"""v0.70.0 Part C — shape-only path validation. Cwd containment
|
||||
deferred to v0.70.1 runtime hook (matches v0.69.0 build_dag /
|
||||
magpie base_model policy).
|
||||
"""
|
||||
if v is None:
|
||||
return None
|
||||
from soup_cli.utils.minillm import _check_path_shape
|
||||
|
||||
return _check_path_shape(v)
|
||||
|
||||
@field_validator(
|
||||
"fp8_attention",
|
||||
"nvfp4",
|
||||
|
|
@ -3743,6 +3981,194 @@ class SoupConfig(BaseModel):
|
|||
f"training helper). Use backend=transformers for task={self.task}."
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_uld_compat(self) -> "SoupConfig":
|
||||
"""v0.70.0 Part B — Universal Logit Distillation gate.
|
||||
|
||||
``uld_strategy`` is only meaningful when ``task='distill'`` —
|
||||
cross-tokenizer distillation has no analogue outside the
|
||||
distillation trainer. Rejected on other tasks with a friendly
|
||||
message, and on MLX backend with a distinct message.
|
||||
|
||||
Composes with v0.52 distillation task: when set, the
|
||||
:class:`uld.ULDConfig` validation fires (top_k cross-validation,
|
||||
vocab-size bounds) at config-load.
|
||||
"""
|
||||
tcfg = self.training
|
||||
strategy = tcfg.uld_strategy
|
||||
top_k = tcfg.uld_top_k
|
||||
if strategy is None and top_k is None:
|
||||
return self
|
||||
if strategy is None:
|
||||
# top_k without strategy is a silent no-op footgun.
|
||||
raise ValueError(
|
||||
"uld_top_k requires uld_strategy to be set"
|
||||
)
|
||||
if self.task != "distill":
|
||||
raise ValueError(
|
||||
"uld_strategy / uld_top_k are only valid when "
|
||||
f"task='distill'; got task={self.task!r}"
|
||||
)
|
||||
if self.backend == "mlx":
|
||||
raise ValueError(
|
||||
"uld_strategy is not supported on backend=mlx in v0.70.0 "
|
||||
"(cross-tokenizer distillation is transformers-only)"
|
||||
)
|
||||
# Cross-check: topk_align requires top_k.
|
||||
if strategy == "topk_align" and top_k is None:
|
||||
raise ValueError(
|
||||
"uld_strategy='topk_align' requires uld_top_k to be set"
|
||||
)
|
||||
if strategy == "wasserstein" and top_k is not None:
|
||||
raise ValueError(
|
||||
"uld_top_k is only valid when uld_strategy='topk_align'; "
|
||||
f"got uld_strategy={strategy!r}"
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_echo_trap_compat(self) -> "SoupConfig":
|
||||
"""v0.70.0 Part F — echo-trap detector task gate.
|
||||
|
||||
``echo_trap_enabled`` (and ``echo_trap_halt``) only meaningful
|
||||
on RL tasks (grpo / ppo). Setting ``echo_trap_halt`` without
|
||||
``echo_trap_enabled`` is a silent no-op footgun — reject.
|
||||
"""
|
||||
tcfg = self.training
|
||||
if not tcfg.echo_trap_enabled and not tcfg.echo_trap_halt:
|
||||
return self
|
||||
if not tcfg.echo_trap_enabled and tcfg.echo_trap_halt:
|
||||
raise ValueError(
|
||||
"echo_trap_halt=True requires echo_trap_enabled=True"
|
||||
)
|
||||
if self.task not in ("grpo", "ppo"):
|
||||
raise ValueError(
|
||||
"echo_trap_enabled / echo_trap_halt are only valid on "
|
||||
f"task in {{'grpo', 'ppo'}}; got task={self.task!r}"
|
||||
)
|
||||
if self.backend == "mlx":
|
||||
raise ValueError(
|
||||
"echo_trap_enabled is not supported on backend=mlx in "
|
||||
"v0.70.0"
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_rl_checkpoint_compat(self) -> "SoupConfig":
|
||||
"""v0.70.0 Part D — mid-epoch RL checkpoint task gate.
|
||||
|
||||
``rl_checkpoint_save_every_steps`` is only meaningful on RL
|
||||
tasks (grpo / ppo). Non-RL tasks already have HF Trainer's
|
||||
per-epoch checkpointing. Rejected on other tasks with a
|
||||
friendly message.
|
||||
"""
|
||||
tcfg = self.training
|
||||
if tcfg.rl_checkpoint_save_every_steps is None:
|
||||
return self
|
||||
if self.task not in ("grpo", "ppo"):
|
||||
raise ValueError(
|
||||
"rl_checkpoint_save_every_steps is only valid on RL tasks "
|
||||
f"(grpo / ppo); got task={self.task!r}. Non-RL tasks use "
|
||||
"HF Trainer's per-epoch checkpointing already."
|
||||
)
|
||||
if self.backend == "mlx":
|
||||
raise ValueError(
|
||||
"rl_checkpoint_save_every_steps is not supported on "
|
||||
"backend=mlx in v0.70.0"
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_minillm_compat(self) -> "SoupConfig":
|
||||
"""v0.70.0 Part C — MiniLLM compatibility gate.
|
||||
|
||||
``minillm_enabled`` requires ``task='distill'`` on a non-mlx
|
||||
backend. Setting any minillm_* tunable without
|
||||
``minillm_enabled=True`` is rejected (silent no-op footgun
|
||||
mirroring v0.52 distill / v0.62 grace_codebook policy).
|
||||
"""
|
||||
tcfg = self.training
|
||||
any_field_set = (
|
||||
tcfg.minillm_teacher_mix_ratio != 0.0
|
||||
or tcfg.minillm_length_normalize is not True
|
||||
or tcfg.minillm_pretrain_anchor_weight != 0.0
|
||||
or tcfg.minillm_pretrain_anchor_path is not None
|
||||
)
|
||||
if not tcfg.minillm_enabled and not any_field_set:
|
||||
return self
|
||||
if not tcfg.minillm_enabled and any_field_set:
|
||||
offenders = []
|
||||
if tcfg.minillm_teacher_mix_ratio != 0.0:
|
||||
offenders.append("minillm_teacher_mix_ratio")
|
||||
if tcfg.minillm_length_normalize is not True:
|
||||
offenders.append("minillm_length_normalize")
|
||||
if tcfg.minillm_pretrain_anchor_weight != 0.0:
|
||||
offenders.append("minillm_pretrain_anchor_weight")
|
||||
if tcfg.minillm_pretrain_anchor_path is not None:
|
||||
offenders.append("minillm_pretrain_anchor_path")
|
||||
raise ValueError(
|
||||
f"MiniLLM tunables {offenders} require minillm_enabled=True"
|
||||
)
|
||||
if self.task != "distill":
|
||||
raise ValueError(
|
||||
"minillm_enabled requires task='distill'; "
|
||||
f"got task={self.task!r}"
|
||||
)
|
||||
if self.backend == "mlx":
|
||||
raise ValueError(
|
||||
"minillm_enabled is not supported on backend=mlx in v0.70.0"
|
||||
)
|
||||
# Cross-check: anchor weight + path mutual requirements.
|
||||
if (
|
||||
tcfg.minillm_pretrain_anchor_weight > 0.0
|
||||
and tcfg.minillm_pretrain_anchor_path is None
|
||||
):
|
||||
raise ValueError(
|
||||
"minillm_pretrain_anchor_weight > 0 requires "
|
||||
"minillm_pretrain_anchor_path to be set"
|
||||
)
|
||||
if (
|
||||
tcfg.minillm_pretrain_anchor_weight == 0.0
|
||||
and tcfg.minillm_pretrain_anchor_path is not None
|
||||
):
|
||||
raise ValueError(
|
||||
"minillm_pretrain_anchor_path is set but "
|
||||
"minillm_pretrain_anchor_weight is 0 (silent no-op)"
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_reward_hack_compat(self) -> "SoupConfig":
|
||||
"""v0.70.0 Part A — reward-hacking detector task / backend gate.
|
||||
|
||||
``reward_hack_detector`` + ``reward_hack_halt`` are only meaningful
|
||||
for RL tasks (grpo / ppo). Rejected outside those tasks with a
|
||||
friendly message that names the offending fields. MLX backend
|
||||
rejected with a distinct message (matches v0.34.0 / v0.50.0
|
||||
review-fix policy of distinct error reasons).
|
||||
"""
|
||||
tcfg = self.training
|
||||
detector = tcfg.reward_hack_detector
|
||||
halt = tcfg.reward_hack_halt
|
||||
if detector is None and not halt:
|
||||
return self
|
||||
# halt without detector is a silent no-op footgun — reject.
|
||||
if detector is None and halt:
|
||||
raise ValueError(
|
||||
"reward_hack_halt=True requires reward_hack_detector to be set"
|
||||
)
|
||||
if self.task not in ("grpo", "ppo"):
|
||||
raise ValueError(
|
||||
"reward_hack_detector / reward_hack_halt are only valid on "
|
||||
f"task in {{'grpo', 'ppo'}}; got task={self.task!r}"
|
||||
)
|
||||
if self.backend == "mlx":
|
||||
raise ValueError(
|
||||
"reward_hack_detector is not supported on backend=mlx in "
|
||||
"v0.70.0 (RL detectors are transformers-only)"
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
# --- Built-in templates ---
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,269 @@
|
|||
"""Live echo-trap detector — v0.70.0 Part F.
|
||||
|
||||
RAGEN-style detection of trajectory degeneration during multi-turn
|
||||
agent RL (Zhu et al. 2025, arXiv:2504.14437). When the policy collapses
|
||||
to self-repeating outputs, the reward saturates and the policy drifts
|
||||
without learning. This module ships the math kernels + report schema;
|
||||
the live HF Trainer callback is deferred to v0.70.1.
|
||||
|
||||
Composes with v0.53.11 #127 ``GRPOStabilityCallback`` — the live
|
||||
echo-trap callback shares the per-step instrumentation hook so both
|
||||
detectors can fire in the same training step without duplicating
|
||||
trajectory collection.
|
||||
|
||||
Security:
|
||||
- Pure-Python math (no torch import at module top).
|
||||
- Bool / NaN / Inf / range rejection on every numeric input.
|
||||
- Tokens must be strings; non-str rejected loudly so a caller that
|
||||
hands tensor ids (instead of decoded strings) gets an actionable
|
||||
error rather than silently misbehaving.
|
||||
- ``_MAX_BATCH_TRAJECTORIES = 100_000`` DoS cap (matches v0.55 /
|
||||
v0.65 / v0.66 cap policy).
|
||||
- ``_MAX_NGRAM_N = 32`` keeps the n-gram counter bounded.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from typing import Iterable, Sequence
|
||||
|
||||
VERDICTS: tuple[str, ...] = ("OK", "WARN", "TRAP")
|
||||
_VALID_VERDICTS: frozenset[str] = frozenset(VERDICTS)
|
||||
|
||||
# OK / WARN / TRAP boundaries on the aggregate echo signal. Mirrors v0.26
|
||||
# Quant-Lobotomy + v0.56 diagnose three-band taxonomy.
|
||||
_ECHO_OK_BAND = 0.30 # signal < 0.30 → OK
|
||||
_ECHO_TRAP_BAND = 0.60 # signal >= 0.60 → TRAP; in between → WARN
|
||||
|
||||
_MAX_NGRAM_N = 32
|
||||
_MAX_TRAJECTORY_TOKENS = 1_000_000
|
||||
_MAX_BATCH_TRAJECTORIES = 100_000
|
||||
|
||||
|
||||
def _check_ngram_n(value: object) -> int:
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("ngram_n must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(f"ngram_n must be int, got {type(value).__name__}")
|
||||
if value < 1:
|
||||
raise ValueError(f"ngram_n must be >= 1, got {value}")
|
||||
if value > _MAX_NGRAM_N:
|
||||
raise ValueError(f"ngram_n={value} exceeds {_MAX_NGRAM_N} cap")
|
||||
return value
|
||||
|
||||
|
||||
def _check_tokens(tokens: object) -> tuple[str, ...]:
|
||||
if isinstance(tokens, (str, bytes)):
|
||||
raise TypeError("tokens must be a sequence of strings, not str/bytes")
|
||||
try:
|
||||
iterator = list(tokens) # type: ignore[arg-type]
|
||||
except TypeError as exc:
|
||||
raise TypeError(
|
||||
f"tokens must be iterable, got {type(tokens).__name__}"
|
||||
) from exc
|
||||
if len(iterator) > _MAX_TRAJECTORY_TOKENS:
|
||||
raise ValueError(
|
||||
f"trajectory has {len(iterator)} tokens, exceeds "
|
||||
f"{_MAX_TRAJECTORY_TOKENS} cap"
|
||||
)
|
||||
for idx, t in enumerate(iterator):
|
||||
if not isinstance(t, str):
|
||||
raise TypeError(
|
||||
f"tokens[{idx}] must be str, got {type(t).__name__}"
|
||||
)
|
||||
return tuple(iterator)
|
||||
|
||||
|
||||
def score_trajectory_repetition(tokens: object, *, ngram_n: object = 2) -> float:
|
||||
"""Per-trajectory repetition score.
|
||||
|
||||
Returns the fraction of n-grams whose count exceeds 1 (the
|
||||
"repeating n-grams" rate). Range ``[0, 1]``. 0 = every n-gram
|
||||
unique; closer to 1 = many n-grams repeat.
|
||||
|
||||
Edge cases:
|
||||
- ``len(tokens) < ngram_n`` returns 0.0 (no n-grams possible).
|
||||
- Empty input returns 0.0.
|
||||
"""
|
||||
n = _check_ngram_n(ngram_n)
|
||||
tok = _check_tokens(tokens)
|
||||
if len(tok) < n:
|
||||
return 0.0
|
||||
counts: dict[tuple[str, ...], int] = {}
|
||||
for i in range(len(tok) - n + 1):
|
||||
gram = tok[i : i + n]
|
||||
counts[gram] = counts.get(gram, 0) + 1
|
||||
if not counts:
|
||||
return 0.0
|
||||
repeating = sum(1 for c in counts.values() if c > 1)
|
||||
return repeating / len(counts)
|
||||
|
||||
|
||||
def score_echo_signal(
|
||||
trajectories: object,
|
||||
*,
|
||||
ngram_n: object = 2,
|
||||
) -> float:
|
||||
"""Mean repetition score across a batch of trajectories.
|
||||
|
||||
Higher = more trajectory degeneration = closer to echo trap.
|
||||
Returns 0.0 on empty input (no signal = nothing to flag).
|
||||
"""
|
||||
n = _check_ngram_n(ngram_n)
|
||||
if isinstance(trajectories, (str, bytes)):
|
||||
raise TypeError(
|
||||
"trajectories must be a sequence of sequences, not str/bytes"
|
||||
)
|
||||
try:
|
||||
batch = list(trajectories) # type: ignore[arg-type]
|
||||
except TypeError as exc:
|
||||
raise TypeError(
|
||||
f"trajectories must be iterable, got "
|
||||
f"{type(trajectories).__name__}"
|
||||
) from exc
|
||||
if len(batch) > _MAX_BATCH_TRAJECTORIES:
|
||||
raise ValueError(
|
||||
f"batch has {len(batch)} trajectories, exceeds "
|
||||
f"{_MAX_BATCH_TRAJECTORIES} cap"
|
||||
)
|
||||
if not batch:
|
||||
return 0.0
|
||||
scores: list[float] = []
|
||||
for traj in batch:
|
||||
scores.append(score_trajectory_repetition(traj, ngram_n=n))
|
||||
return sum(scores) / len(scores)
|
||||
|
||||
|
||||
def classify_echo_signal(signal: object) -> str:
|
||||
"""Map a signal in ``[0, 1]`` to OK / WARN / TRAP.
|
||||
|
||||
- signal in [0.0, _ECHO_OK_BAND=0.30): OK
|
||||
- signal in [_ECHO_OK_BAND, _ECHO_TRAP_BAND=0.60): WARN
|
||||
- signal >= _ECHO_TRAP_BAND: TRAP
|
||||
"""
|
||||
if isinstance(signal, bool):
|
||||
raise ValueError("signal must not be bool")
|
||||
if not isinstance(signal, (int, float)):
|
||||
raise ValueError(
|
||||
f"signal must be a number, got {type(signal).__name__}"
|
||||
)
|
||||
fv = float(signal)
|
||||
if not math.isfinite(fv):
|
||||
raise ValueError("signal must be finite (no NaN/Inf)")
|
||||
if not (0.0 <= fv <= 1.0):
|
||||
raise ValueError(f"signal must be in [0.0, 1.0], got {fv}")
|
||||
if fv < _ECHO_OK_BAND:
|
||||
return "OK"
|
||||
if fv < _ECHO_TRAP_BAND:
|
||||
return "WARN"
|
||||
return "TRAP"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EchoTrapReport:
|
||||
"""Frozen result of an echo-trap probe.
|
||||
|
||||
- ``signal``: aggregate echo signal in ``[0, 1]``.
|
||||
- ``verdict``: OK / WARN / TRAP per :func:`classify_echo_signal`.
|
||||
- ``step``: training step at which the probe fired. Non-negative
|
||||
int (bool rejected per project policy).
|
||||
- ``trajectories_seen``: count of trajectories that contributed to
|
||||
the signal. Non-negative.
|
||||
- ``details``: tuple of human-readable lines for the report panel.
|
||||
"""
|
||||
|
||||
signal: float
|
||||
verdict: str
|
||||
step: int
|
||||
trajectories_seen: int
|
||||
details: tuple[str, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if isinstance(self.signal, bool):
|
||||
raise ValueError("signal must not be bool")
|
||||
if not isinstance(self.signal, (int, float)):
|
||||
raise TypeError(
|
||||
f"signal must be a number, got {type(self.signal).__name__}"
|
||||
)
|
||||
fv = float(self.signal)
|
||||
if not math.isfinite(fv) or not (0.0 <= fv <= 1.0):
|
||||
raise ValueError(f"signal must be in [0.0, 1.0], got {self.signal}")
|
||||
if self.verdict not in _VALID_VERDICTS:
|
||||
raise ValueError(
|
||||
f"verdict={self.verdict!r} must be one of {sorted(_VALID_VERDICTS)}"
|
||||
)
|
||||
if isinstance(self.step, bool):
|
||||
raise ValueError("step must not be bool")
|
||||
if not isinstance(self.step, int):
|
||||
raise TypeError(f"step must be int, got {type(self.step).__name__}")
|
||||
if self.step < 0:
|
||||
raise ValueError(f"step must be non-negative, got {self.step}")
|
||||
if isinstance(self.trajectories_seen, bool):
|
||||
raise ValueError("trajectories_seen must not be bool")
|
||||
if not isinstance(self.trajectories_seen, int):
|
||||
raise TypeError(
|
||||
"trajectories_seen must be int, got "
|
||||
f"{type(self.trajectories_seen).__name__}"
|
||||
)
|
||||
if self.trajectories_seen < 0:
|
||||
raise ValueError(
|
||||
f"trajectories_seen must be non-negative, got "
|
||||
f"{self.trajectories_seen}"
|
||||
)
|
||||
if not isinstance(self.details, tuple):
|
||||
raise TypeError(
|
||||
f"details must be a tuple, got {type(self.details).__name__}"
|
||||
)
|
||||
|
||||
|
||||
def build_echo_trap_callback(
|
||||
*,
|
||||
threshold: float,
|
||||
halt_on_trap: bool = True,
|
||||
ngram_n: int = 2,
|
||||
):
|
||||
"""Live HF Trainer callback for echo-trap detection.
|
||||
|
||||
Deferred to v0.70.1. Validates inputs at the public boundary so
|
||||
misconfigured callers fail fast (mirrors v0.50.0 / v0.62.0 /
|
||||
v0.67.0 / v0.69.0 / Part A/B/C/D/E deferred-live policy).
|
||||
"""
|
||||
# Threshold validation runs FIRST so a bad numeric value fires the
|
||||
# actionable error before we check the bool flags.
|
||||
if isinstance(threshold, bool):
|
||||
raise ValueError("threshold must not be bool")
|
||||
if not isinstance(threshold, (int, float)):
|
||||
raise ValueError(
|
||||
f"threshold must be a number, got {type(threshold).__name__}"
|
||||
)
|
||||
fv = float(threshold)
|
||||
if not math.isfinite(fv) or not (0.0 <= fv <= 1.0):
|
||||
raise ValueError(f"threshold must be in [0.0, 1.0], got {threshold}")
|
||||
if not isinstance(halt_on_trap, bool):
|
||||
raise TypeError(
|
||||
f"halt_on_trap must be bool, got {type(halt_on_trap).__name__}"
|
||||
)
|
||||
_check_ngram_n(ngram_n)
|
||||
raise NotImplementedError(
|
||||
f"Live echo-trap HF Trainer callback (threshold={fv}) is deferred "
|
||||
"to v0.70.1. v0.70.0 ships the schema + math kernels only."
|
||||
)
|
||||
|
||||
|
||||
# Public re-exports — type hints for the v0.70.1 callback signature so
|
||||
# external consumers (e.g. the GRPO stability callback) can import them
|
||||
# without circular dependencies.
|
||||
__all__ = [
|
||||
"VERDICTS",
|
||||
"EchoTrapReport",
|
||||
"build_echo_trap_callback",
|
||||
"classify_echo_signal",
|
||||
"score_echo_signal",
|
||||
"score_trajectory_repetition",
|
||||
]
|
||||
|
||||
|
||||
# Type aliases retained for the v0.70.1 wiring.
|
||||
TrajectoryTokens = Sequence[str]
|
||||
TrajectoryBatch = Iterable[TrajectoryTokens]
|
||||
|
|
@ -0,0 +1,215 @@
|
|||
"""Iterative DPO loop driver — v0.70.0 Part E.
|
||||
|
||||
Sample → RM-score → re-pair → retrain over N rounds. Frozen plan +
|
||||
per-round artifact tracking; the actual round orchestrator (which
|
||||
would invoke ``soup train --task dpo`` between rounds) is deferred to
|
||||
v0.70.1 (mirrors v0.68.0 local-rl nightly-train policy).
|
||||
|
||||
The plan models each round explicitly so the v0.70.1 runner can:
|
||||
- skip rounds whose ``adapter_path`` already exists (resume),
|
||||
- re-render pairs JSONL deterministically per round,
|
||||
- track per-round pairs_count for the `runs replay` integration.
|
||||
|
||||
Security:
|
||||
- Frozen dataclasses with per-field validation (matches v0.67.0
|
||||
CmaesPlan / v0.68.0 CompilePlan policy).
|
||||
- Bool / null-byte / oversize / non-int rejection on every input.
|
||||
- ``rounds`` tuple required (List would not be immutable under
|
||||
``frozen=True``; matches v0.43 Part B / v0.61 Part E policy).
|
||||
- Consecutive ``round_index`` invariant: rounds must be 0..N-1 with no
|
||||
gaps (defends against caller-side bugs that would silently skip
|
||||
rounds).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Tuple
|
||||
|
||||
_MIN_ROUNDS = 1
|
||||
_MAX_ROUNDS = 100
|
||||
_MIN_PAIRS_PER_ROUND = 10
|
||||
_MAX_PAIRS_PER_ROUND = 1_000_000
|
||||
_MAX_PATH_LEN = 4096
|
||||
|
||||
|
||||
def validate_rounds(value: object) -> int:
|
||||
"""Validate the ``rounds`` count: int in [1, 100], bool rejected."""
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("rounds must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(f"rounds must be int, got {type(value).__name__}")
|
||||
if value < _MIN_ROUNDS:
|
||||
raise ValueError(f"rounds must be >= {_MIN_ROUNDS}, got {value}")
|
||||
if value > _MAX_ROUNDS:
|
||||
raise ValueError(
|
||||
f"rounds={value} exceeds {_MAX_ROUNDS} cap"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_pairs_per_round(value: object) -> int:
|
||||
"""Validate the per-round pair count.
|
||||
|
||||
Range: ``[10, 1_000_000]``. Below 10 pairs the DPO gradient signal
|
||||
is too noisy to be meaningful; above 1M is a clear OOM / disk hazard.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("pairs_per_round must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(
|
||||
f"pairs_per_round must be int, got {type(value).__name__}"
|
||||
)
|
||||
if value < _MIN_PAIRS_PER_ROUND:
|
||||
raise ValueError(
|
||||
f"pairs_per_round must be >= {_MIN_PAIRS_PER_ROUND}, got {value}"
|
||||
)
|
||||
if value > _MAX_PAIRS_PER_ROUND:
|
||||
raise ValueError(
|
||||
f"pairs_per_round={value} exceeds {_MAX_PAIRS_PER_ROUND} cap"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _check_path(value: object, field: str) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise ValueError(f"{field} must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str, got {type(value).__name__}")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > _MAX_PATH_LEN:
|
||||
raise ValueError(f"{field} exceeds {_MAX_PATH_LEN} chars")
|
||||
return value
|
||||
|
||||
|
||||
def _check_non_negative_int(value: object, field: str) -> int:
|
||||
if isinstance(value, bool):
|
||||
raise ValueError(f"{field} must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise TypeError(f"{field} must be int, got {type(value).__name__}")
|
||||
if value < 0:
|
||||
raise ValueError(f"{field} must be non-negative, got {value}")
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IterativeDPORound:
|
||||
"""Frozen per-round descriptor.
|
||||
|
||||
- ``round_index``: 0-based, non-negative int (bool rejected).
|
||||
- ``prompts_path``: source prompts JSONL (sampled into pairs).
|
||||
- ``pairs_path``: where the round's chosen/rejected JSONL gets written.
|
||||
- ``adapter_path``: where the round's DPO adapter lands.
|
||||
- ``pairs_count``: number of pairs the round produced. Non-negative
|
||||
int (0 allowed for plan-only rendering).
|
||||
"""
|
||||
|
||||
round_index: int
|
||||
prompts_path: str
|
||||
pairs_path: str
|
||||
adapter_path: str
|
||||
pairs_count: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_non_negative_int(self.round_index, "round_index")
|
||||
_check_path(self.prompts_path, "prompts_path")
|
||||
_check_path(self.pairs_path, "pairs_path")
|
||||
_check_path(self.adapter_path, "adapter_path")
|
||||
_check_non_negative_int(self.pairs_count, "pairs_count")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IterativeDPOPlan:
|
||||
"""Frozen iterative-DPO plan.
|
||||
|
||||
- ``base_model``: HF id / local path. Shape-validated.
|
||||
- ``reward_model``: HF id / local path of the RM used for scoring.
|
||||
- ``rounds``: tuple of :class:`IterativeDPORound`; must be
|
||||
consecutive (0..N-1 with no gaps).
|
||||
"""
|
||||
|
||||
base_model: str
|
||||
reward_model: str
|
||||
rounds: Tuple[IterativeDPORound, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_check_path(self.base_model, "base_model")
|
||||
_check_path(self.reward_model, "reward_model")
|
||||
if not isinstance(self.rounds, tuple):
|
||||
raise TypeError(
|
||||
f"rounds must be a tuple, got {type(self.rounds).__name__}"
|
||||
)
|
||||
if len(self.rounds) < 1:
|
||||
raise ValueError("rounds must contain at least 1 round")
|
||||
for r in self.rounds:
|
||||
if not isinstance(r, IterativeDPORound):
|
||||
raise TypeError(
|
||||
f"every rounds[] entry must be IterativeDPORound, "
|
||||
f"got {type(r).__name__}"
|
||||
)
|
||||
for idx, r in enumerate(self.rounds):
|
||||
if r.round_index != idx:
|
||||
raise ValueError(
|
||||
f"rounds must have consecutive round_index 0..N-1; "
|
||||
f"rounds[{idx}].round_index={r.round_index}"
|
||||
)
|
||||
|
||||
|
||||
def build_iterative_dpo_plan(
|
||||
*,
|
||||
base_model: str,
|
||||
reward_model: str,
|
||||
prompts_path: str,
|
||||
output_dir: str,
|
||||
rounds: int,
|
||||
pairs_per_round: int,
|
||||
) -> IterativeDPOPlan:
|
||||
"""Build a canonical :class:`IterativeDPOPlan` from operator inputs.
|
||||
|
||||
Per-round paths follow the pattern
|
||||
``<output_dir>/round-<NN>/{pairs.jsonl,adapter}``. The plan is
|
||||
cheap to construct; the v0.70.1 runner consumes it.
|
||||
"""
|
||||
validate_rounds(rounds)
|
||||
validate_pairs_per_round(pairs_per_round)
|
||||
_check_path(base_model, "base_model")
|
||||
_check_path(reward_model, "reward_model")
|
||||
_check_path(prompts_path, "prompts_path")
|
||||
_check_path(output_dir, "output_dir")
|
||||
output_dir = output_dir.rstrip("/\\")
|
||||
per_round = []
|
||||
for i in range(rounds):
|
||||
per_round.append(
|
||||
IterativeDPORound(
|
||||
round_index=i,
|
||||
prompts_path=prompts_path,
|
||||
pairs_path=f"{output_dir}/round-{i:02d}/pairs.jsonl",
|
||||
adapter_path=f"{output_dir}/round-{i:02d}/adapter",
|
||||
pairs_count=pairs_per_round,
|
||||
)
|
||||
)
|
||||
return IterativeDPOPlan(
|
||||
base_model=base_model,
|
||||
reward_model=reward_model,
|
||||
rounds=tuple(per_round),
|
||||
)
|
||||
|
||||
|
||||
def run_iterative_dpo(plan):
|
||||
"""Execute the iterative-DPO loop. Deferred to v0.70.1.
|
||||
|
||||
Validates plan type at the public boundary so misconfigured callers
|
||||
fail fast (mirrors v0.50.0 / v0.62.0 / v0.67.0 / v0.69.0 deferred-live
|
||||
policy).
|
||||
"""
|
||||
if not isinstance(plan, IterativeDPOPlan):
|
||||
raise TypeError(
|
||||
f"plan must be IterativeDPOPlan, got {type(plan).__name__}"
|
||||
)
|
||||
raise NotImplementedError(
|
||||
"Live iterative-DPO loop runner is deferred to v0.70.1. "
|
||||
"v0.70.0 ships the schema + plan-only renderer only."
|
||||
)
|
||||
|
|
@ -0,0 +1,151 @@
|
|||
"""MiniLLM — reverse-KL on-policy distillation — v0.70.0 Part C.
|
||||
|
||||
MiniLLM (Gu et al. 2024, arXiv:2306.08543) extends knowledge
|
||||
distillation with three stability tricks:
|
||||
|
||||
1. **Teacher-mixed sampling** — at rollout time, with probability
|
||||
``teacher_mix_ratio`` sample from teacher logits instead of student
|
||||
to keep the student near a known-good distribution.
|
||||
2. **Length normalisation** — divide the rollout log-probability by
|
||||
the completion length so longer completions don't dominate the
|
||||
gradient.
|
||||
3. **Pretrain-loss anchor** — add a small SFT-on-pretrain term to the
|
||||
loss to prevent the student from drifting away from coherent
|
||||
language during the on-policy distillation.
|
||||
|
||||
Bundles stability tricks scattered across §3 of the paper. Extends
|
||||
v0.53.2 :class:`DistillTrainerWrapper`. Live wiring deferred to v0.70.1
|
||||
— mirrors v0.50.0 / v0.62.0 / v0.69.0 stub-then-live pattern.
|
||||
|
||||
Security:
|
||||
- Bool / NaN / Inf / range rejection on every numeric validator.
|
||||
- Null-byte + 4096-char cap on ``pretrain_anchor_path``.
|
||||
- ``length_normalize`` must be a real bool (no str/int coercion).
|
||||
- Cross-validators reject silent-no-op combinations
|
||||
(anchor_weight=0 + anchor_path set, anchor_weight > 0 + path None).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
_MAX_ANCHOR_PATH_LEN = 4096
|
||||
|
||||
|
||||
def _check_unit_float(value: object, field: str) -> float:
|
||||
"""Validate a finite float in [0.0, 1.0]. Bool rejected."""
|
||||
if isinstance(value, bool):
|
||||
raise ValueError(f"{field} must not be bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise ValueError(
|
||||
f"{field} must be a number, got {type(value).__name__}"
|
||||
)
|
||||
fv = float(value)
|
||||
if not math.isfinite(fv):
|
||||
raise ValueError(f"{field} must be finite (no NaN/Inf)")
|
||||
if not (0.0 <= fv <= 1.0):
|
||||
raise ValueError(f"{field} must be in [0.0, 1.0], got {fv}")
|
||||
return fv
|
||||
|
||||
|
||||
def validate_teacher_mix_ratio(value: object) -> float:
|
||||
"""Validate the teacher-mix sampling ratio. Range [0.0, 1.0].
|
||||
|
||||
0.0 = student-only rollouts; 1.0 = teacher-only rollouts. Typical
|
||||
MiniLLM recipes use 0.2-0.5 to balance exploration against
|
||||
proximity to the teacher's distribution.
|
||||
"""
|
||||
return _check_unit_float(value, "teacher_mix_ratio")
|
||||
|
||||
|
||||
def validate_pretrain_anchor_weight(value: object) -> float:
|
||||
"""Validate the pretrain-anchor loss coefficient. Range [0.0, 1.0].
|
||||
|
||||
0.0 = no anchor; small positive (e.g. 0.1) adds the SFT-on-pretrain
|
||||
term as a regulariser. Capped at 1.0 — values above would dominate
|
||||
the distillation loss (silent regression to vanilla SFT).
|
||||
"""
|
||||
return _check_unit_float(value, "pretrain_anchor_weight")
|
||||
|
||||
|
||||
def _check_path_shape(value: Optional[str]) -> Optional[str]:
|
||||
"""Validate a string path field for shape only (cwd containment is
|
||||
deferred to the v0.70.1 runtime hook — schema permits relative
|
||||
paths for the same reason v0.69.0 build_dag does)."""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("pretrain_anchor_path must not be bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(
|
||||
f"pretrain_anchor_path must be str, got {type(value).__name__}"
|
||||
)
|
||||
if not value:
|
||||
raise ValueError("pretrain_anchor_path must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError("pretrain_anchor_path must not contain null bytes")
|
||||
if len(value) > _MAX_ANCHOR_PATH_LEN:
|
||||
raise ValueError(
|
||||
f"pretrain_anchor_path exceeds {_MAX_ANCHOR_PATH_LEN} chars"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MiniLLMConfig:
|
||||
"""Frozen MiniLLM configuration.
|
||||
|
||||
Cross-validation:
|
||||
- ``pretrain_anchor_weight > 0`` requires ``pretrain_anchor_path`` to
|
||||
be set (otherwise the anchor term has nothing to anchor against).
|
||||
- ``pretrain_anchor_path is not None`` requires
|
||||
``pretrain_anchor_weight > 0`` (otherwise the path is a silent
|
||||
no-op).
|
||||
"""
|
||||
|
||||
teacher_mix_ratio: float = 0.0
|
||||
length_normalize: bool = True
|
||||
pretrain_anchor_weight: float = 0.0
|
||||
pretrain_anchor_path: Optional[str] = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
validate_teacher_mix_ratio(self.teacher_mix_ratio)
|
||||
if not isinstance(self.length_normalize, bool):
|
||||
raise TypeError(
|
||||
f"length_normalize must be bool, got "
|
||||
f"{type(self.length_normalize).__name__}"
|
||||
)
|
||||
validate_pretrain_anchor_weight(self.pretrain_anchor_weight)
|
||||
_check_path_shape(self.pretrain_anchor_path)
|
||||
if self.pretrain_anchor_weight > 0.0 and self.pretrain_anchor_path is None:
|
||||
raise ValueError(
|
||||
"pretrain_anchor_weight > 0 requires pretrain_anchor_path "
|
||||
"to be set"
|
||||
)
|
||||
if (
|
||||
self.pretrain_anchor_weight == 0.0
|
||||
and self.pretrain_anchor_path is not None
|
||||
):
|
||||
raise ValueError(
|
||||
"pretrain_anchor_path is set but pretrain_anchor_weight is "
|
||||
"0 (silent no-op); set anchor_weight > 0 or clear the path"
|
||||
)
|
||||
|
||||
|
||||
def build_minillm_callback(config):
|
||||
"""Build the MiniLLM HF Trainer callback. Deferred to v0.70.1.
|
||||
|
||||
Validates the config type at the public boundary so misconfigured
|
||||
callers fail fast (mirrors v0.50.0 / v0.62.0 / v0.67.0 / v0.69.0
|
||||
deferred-live policy).
|
||||
"""
|
||||
if not isinstance(config, MiniLLMConfig):
|
||||
raise TypeError(
|
||||
f"config must be MiniLLMConfig, got {type(config).__name__}"
|
||||
)
|
||||
raise NotImplementedError(
|
||||
"Live MiniLLM HF Trainer callback is deferred to v0.70.1. "
|
||||
"v0.70.0 ships the schema + validators only."
|
||||
)
|
||||
|
|
@ -0,0 +1,349 @@
|
|||
"""Reward-hacking detector — v0.70.0 Part A.
|
||||
|
||||
Closed allowlist of reward-hacking detectors for GRPO/PPO training.
|
||||
Surfaces an early-warning signal when the policy starts gaming the
|
||||
reward model rather than improving the underlying capability.
|
||||
|
||||
Two detectors ship in v0.70.0 (schema-only; live trainer callback wired
|
||||
in v0.70.1):
|
||||
|
||||
- ``info_rm``: Cluster-Separation Index over (good, bad) response
|
||||
reward distributions. Drops in cluster separation across training =
|
||||
reward model losing its grip. Inspired by Wang et al. 2024
|
||||
"InfoRM: Mitigating Reward Hacking via Information-Theoretic Reward
|
||||
Modeling" (arXiv 2402.09345).
|
||||
|
||||
- ``rm_ensemble``: pairwise variance across an ensemble of RMs.
|
||||
Rising disagreement = unreliable reward signal.
|
||||
|
||||
Live wiring deferred to v0.70.1 — mirrors v0.27.0 MII / v0.50.0
|
||||
GRPO Plus / v0.61.0 unlearning stub-then-live pattern.
|
||||
|
||||
Security:
|
||||
- Closed allowlist (frozenset); arbitrary detector name rejected.
|
||||
- Bool / null-byte / non-string / oversize rejection on every public
|
||||
validator (mirrors v0.41.0 / v0.51.0 / v0.62.0 policy).
|
||||
- ``_DETECTOR_METADATA`` wrapped in ``MappingProxyType`` for runtime
|
||||
immutability (matches v0.36.0 / v0.41.0 / v0.50.0 / v0.62.0 policy).
|
||||
- Math kernels never import torch at module top — lazy imports only.
|
||||
- All bound-check rejections include actionable messages naming the
|
||||
field + offending value.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import types
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, Sequence
|
||||
|
||||
_MAX_DETECTOR_NAME_LEN = 32
|
||||
_MAX_RM_ENSEMBLE_SIZE = 32
|
||||
_EPS = 1e-9
|
||||
|
||||
SUPPORTED_HACK_DETECTORS: frozenset[str] = frozenset({"info_rm", "rm_ensemble"})
|
||||
|
||||
VERDICTS: tuple[str, ...] = ("OK", "WARN", "HACK")
|
||||
_VALID_VERDICTS: frozenset[str] = frozenset(VERDICTS)
|
||||
|
||||
# OK / WARN / HACK boundaries on the relative drop in cluster separation.
|
||||
# These match the v0.26 Quant-Lobotomy + v0.56 diagnose three-band taxonomy
|
||||
# (OK / MINOR / MAJOR) but renamed for the RL setting.
|
||||
_HACK_OK_BAND = 0.10 # drop < 10% → OK
|
||||
_HACK_WARN_BAND = 0.30 # 10% ≤ drop < 30% → WARN; else HACK
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HackDetectorSpec:
|
||||
"""Static metadata for a reward-hacking detector."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
paper: str
|
||||
|
||||
|
||||
_DETECTOR_METADATA = types.MappingProxyType({
|
||||
"info_rm": HackDetectorSpec(
|
||||
name="info_rm",
|
||||
description=(
|
||||
"InfoRM Cluster-Separation Index over (good, bad) reward "
|
||||
"distributions. Drop across training = reward-hacking signal."
|
||||
),
|
||||
paper="Wang et al. 2024 — arXiv:2402.09345",
|
||||
),
|
||||
"rm_ensemble": HackDetectorSpec(
|
||||
name="rm_ensemble",
|
||||
description=(
|
||||
"Pairwise variance across an RM ensemble. Rising disagreement "
|
||||
"= unreliable reward signal."
|
||||
),
|
||||
paper="Coste et al. 2024 — arXiv:2312.09244",
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
def validate_hack_detector(name: object) -> str:
|
||||
"""Validate and normalise a reward-hacking detector name.
|
||||
|
||||
Returns the canonical (lower-cased) name on success. Raises
|
||||
``ValueError`` with an actionable message on any failure.
|
||||
"""
|
||||
if isinstance(name, bool):
|
||||
raise ValueError("reward_hack_detector must be a string, got bool")
|
||||
if not isinstance(name, str):
|
||||
raise ValueError(
|
||||
f"reward_hack_detector must be a string, got {type(name).__name__}"
|
||||
)
|
||||
if not name:
|
||||
raise ValueError("reward_hack_detector must be a non-empty string")
|
||||
if "\x00" in name:
|
||||
raise ValueError("reward_hack_detector must not contain null bytes")
|
||||
if len(name) > _MAX_DETECTOR_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"reward_hack_detector exceeds {_MAX_DETECTOR_NAME_LEN} chars"
|
||||
)
|
||||
normalised = name.lower()
|
||||
if normalised not in SUPPORTED_HACK_DETECTORS:
|
||||
raise ValueError(
|
||||
f"reward_hack_detector={name!r} is not supported. "
|
||||
f"Valid: {sorted(SUPPORTED_HACK_DETECTORS)}"
|
||||
)
|
||||
return normalised
|
||||
|
||||
|
||||
def get_detector_spec(name: str) -> HackDetectorSpec:
|
||||
"""Return the :class:`HackDetectorSpec` for ``name``."""
|
||||
normalised = validate_hack_detector(name)
|
||||
return _DETECTOR_METADATA[normalised]
|
||||
|
||||
|
||||
def _check_finite_float_sequence(values: object, field: str) -> tuple[float, ...]:
|
||||
"""Validate a numeric sequence: non-empty, finite, no bool."""
|
||||
if isinstance(values, (str, bytes)):
|
||||
raise TypeError(f"{field} must be a sequence of numbers, not str/bytes")
|
||||
try:
|
||||
iterator = iter(values) # type: ignore[arg-type]
|
||||
except TypeError as exc:
|
||||
raise TypeError(
|
||||
f"{field} must be iterable, got {type(values).__name__}"
|
||||
) from exc
|
||||
out: list[float] = []
|
||||
for idx, v in enumerate(iterator):
|
||||
if isinstance(v, bool):
|
||||
raise ValueError(
|
||||
f"{field}[{idx}] must not be bool"
|
||||
)
|
||||
if not isinstance(v, (int, float)):
|
||||
raise ValueError(
|
||||
f"{field}[{idx}] must be a number, got {type(v).__name__}"
|
||||
)
|
||||
fv = float(v)
|
||||
if not math.isfinite(fv):
|
||||
raise ValueError(f"{field}[{idx}] must be finite (no NaN/Inf)")
|
||||
out.append(fv)
|
||||
if not out:
|
||||
raise ValueError(f"{field} must not be empty")
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _mean(seq: Sequence[float]) -> float:
|
||||
return sum(seq) / len(seq)
|
||||
|
||||
|
||||
def _variance(seq: Sequence[float]) -> float:
|
||||
"""Population variance (n divisor, not n-1) for stability with N=1."""
|
||||
m = _mean(seq)
|
||||
return sum((x - m) ** 2 for x in seq) / len(seq)
|
||||
|
||||
|
||||
def compute_cluster_separation(
|
||||
good_scores: object,
|
||||
bad_scores: object,
|
||||
) -> float:
|
||||
"""InfoRM-style cluster-separation index.
|
||||
|
||||
Returns ``(mean_good - mean_bad) / sqrt(var_good + var_bad + eps)``.
|
||||
Larger = better-separated reward clusters. Watch for a sharp drop
|
||||
across training steps — that signals the RM losing its grip.
|
||||
|
||||
Both arguments must be non-empty iterables of finite numbers.
|
||||
``bool`` rejected per project bool-as-int policy.
|
||||
"""
|
||||
good = _check_finite_float_sequence(good_scores, "good_scores")
|
||||
bad = _check_finite_float_sequence(bad_scores, "bad_scores")
|
||||
delta = _mean(good) - _mean(bad)
|
||||
pooled = _variance(good) + _variance(bad) + _EPS
|
||||
return delta / math.sqrt(pooled)
|
||||
|
||||
|
||||
def compute_rm_ensemble_divergence(rm_scores: object) -> float:
|
||||
"""Mean pairwise variance across a small RM ensemble.
|
||||
|
||||
Input is a sequence of per-RM score lists; every inner list must be
|
||||
the same length (one score per prompt, aligned across RMs).
|
||||
Returns the mean of the per-prompt variance over RMs. Higher =
|
||||
RMs disagree more = reward signal less reliable.
|
||||
|
||||
Bounds:
|
||||
- ensemble size in [2, _MAX_RM_ENSEMBLE_SIZE=32]
|
||||
- all inner lists same length (≥ 1)
|
||||
- every value finite + non-bool
|
||||
"""
|
||||
if isinstance(rm_scores, (str, bytes)):
|
||||
raise TypeError("rm_scores must be a sequence of sequences, not str/bytes")
|
||||
try:
|
||||
outer = list(rm_scores) # type: ignore[arg-type]
|
||||
except TypeError as exc:
|
||||
raise TypeError(
|
||||
f"rm_scores must be iterable, got {type(rm_scores).__name__}"
|
||||
) from exc
|
||||
if len(outer) < 2:
|
||||
raise ValueError("rm_scores requires at least 2 RMs (ensemble divergence)")
|
||||
if len(outer) > _MAX_RM_ENSEMBLE_SIZE:
|
||||
raise ValueError(
|
||||
f"rm_scores has too many RMs (>{_MAX_RM_ENSEMBLE_SIZE} cap)"
|
||||
)
|
||||
per_rm: list[tuple[float, ...]] = []
|
||||
expected_len: Optional[int] = None
|
||||
for idx, inner in enumerate(outer):
|
||||
seq = _check_finite_float_sequence(inner, f"rm_scores[{idx}]")
|
||||
if expected_len is None:
|
||||
expected_len = len(seq)
|
||||
elif len(seq) != expected_len:
|
||||
raise ValueError(
|
||||
f"rm_scores[{idx}] has length {len(seq)} but expected "
|
||||
f"{expected_len} (all RM score lists must share length)"
|
||||
)
|
||||
per_rm.append(seq)
|
||||
# Per-prompt variance over RMs, then mean across prompts.
|
||||
assert expected_len is not None
|
||||
per_prompt_var: list[float] = []
|
||||
for j in range(expected_len):
|
||||
column = [rm[j] for rm in per_rm]
|
||||
per_prompt_var.append(_variance(column))
|
||||
return _mean(per_prompt_var) if per_prompt_var else 0.0
|
||||
|
||||
|
||||
def classify_hack_signal(drop_pct: object) -> str:
|
||||
"""Map a relative drop in cluster-separation to OK / WARN / HACK.
|
||||
|
||||
``drop_pct`` is the *relative* drop:
|
||||
``(baseline_signal - current_signal) / baseline_signal``. Non-negative
|
||||
by definition; the caller clamps to 0 when the RM improves.
|
||||
|
||||
Boundaries (matches v0.26 Quant-Lobotomy / v0.56 diagnose taxonomy):
|
||||
- drop_pct ∈ [0, _HACK_OK_BAND=0.10): OK
|
||||
- drop_pct ∈ [_HACK_OK_BAND, _HACK_WARN_BAND=0.30): WARN
|
||||
- drop_pct >= _HACK_WARN_BAND: HACK
|
||||
"""
|
||||
if isinstance(drop_pct, bool):
|
||||
raise ValueError("drop_pct must not be bool")
|
||||
if not isinstance(drop_pct, (int, float)):
|
||||
raise ValueError(
|
||||
f"drop_pct must be a number, got {type(drop_pct).__name__}"
|
||||
)
|
||||
fv = float(drop_pct)
|
||||
if not math.isfinite(fv):
|
||||
raise ValueError("drop_pct must be finite (no NaN/Inf)")
|
||||
if fv < 0.0:
|
||||
raise ValueError(f"drop_pct must be non-negative, got {fv}")
|
||||
if fv < _HACK_OK_BAND:
|
||||
return "OK"
|
||||
if fv < _HACK_WARN_BAND:
|
||||
return "WARN"
|
||||
return "HACK"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RewardHackReport:
|
||||
"""Frozen result of a reward-hacking probe.
|
||||
|
||||
- ``detector``: which detector produced the signal (allowlist).
|
||||
- ``signal``: the raw scalar (cluster-sep value or ensemble variance).
|
||||
Non-negative + finite.
|
||||
- ``verdict``: OK / WARN / HACK per :func:`classify_hack_signal`.
|
||||
- ``step``: training step at which the probe fired. Non-negative int,
|
||||
bool rejected per project bool-as-int policy.
|
||||
- ``baseline_signal``: the reference signal recorded at the start of
|
||||
training. Used to compute the relative drop. Finite, non-negative.
|
||||
- ``details``: tuple of human-readable lines for the report panel.
|
||||
"""
|
||||
|
||||
detector: str
|
||||
signal: float
|
||||
verdict: str
|
||||
step: int
|
||||
baseline_signal: float
|
||||
details: tuple[str, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# validate_hack_detector normalises; re-write via object.__setattr__
|
||||
# so frozen + canonical-case invariants stay together.
|
||||
normalised = validate_hack_detector(self.detector)
|
||||
if normalised != self.detector:
|
||||
object.__setattr__(self, "detector", normalised)
|
||||
if self.verdict not in _VALID_VERDICTS:
|
||||
raise ValueError(
|
||||
f"verdict={self.verdict!r} must be one of {sorted(_VALID_VERDICTS)}"
|
||||
)
|
||||
if isinstance(self.signal, bool):
|
||||
raise ValueError("signal must not be bool")
|
||||
if not isinstance(self.signal, (int, float)):
|
||||
raise TypeError(
|
||||
f"signal must be a number, got {type(self.signal).__name__}"
|
||||
)
|
||||
if not math.isfinite(float(self.signal)):
|
||||
raise ValueError("signal must be finite")
|
||||
if float(self.signal) < 0.0:
|
||||
raise ValueError(f"signal must be non-negative, got {self.signal}")
|
||||
if isinstance(self.step, bool):
|
||||
raise ValueError("step must not be bool")
|
||||
if not isinstance(self.step, int):
|
||||
raise TypeError(f"step must be int, got {type(self.step).__name__}")
|
||||
if self.step < 0:
|
||||
raise ValueError(f"step must be non-negative, got {self.step}")
|
||||
if isinstance(self.baseline_signal, bool):
|
||||
raise ValueError("baseline_signal must not be bool")
|
||||
if not isinstance(self.baseline_signal, (int, float)):
|
||||
raise TypeError("baseline_signal must be a number")
|
||||
if not math.isfinite(float(self.baseline_signal)):
|
||||
raise ValueError("baseline_signal must be finite")
|
||||
if float(self.baseline_signal) < 0.0:
|
||||
raise ValueError("baseline_signal must be non-negative")
|
||||
if not isinstance(self.details, tuple):
|
||||
raise TypeError(
|
||||
f"details must be a tuple, got {type(self.details).__name__}"
|
||||
)
|
||||
|
||||
|
||||
def build_reward_hack_callback(
|
||||
*,
|
||||
detector: str,
|
||||
halt_on_hack: bool = True,
|
||||
baseline_signal: Optional[float] = None,
|
||||
):
|
||||
"""Live HF Trainer callback factory — deferred to v0.70.1.
|
||||
|
||||
Validates inputs at construction time so misconfigured runs fail
|
||||
fast even though the live callback is not yet wired. Raises
|
||||
``NotImplementedError`` with explicit v0.70.1 marker after
|
||||
validation (mirrors v0.50.0 ``apply_variant_loss`` / v0.61.0
|
||||
``apply_unlearn_loss`` policy).
|
||||
"""
|
||||
# Validation order: name first (cheap, allowlist) then bool guard.
|
||||
validate_hack_detector(detector)
|
||||
if not isinstance(halt_on_hack, bool):
|
||||
raise TypeError(
|
||||
f"halt_on_hack must be bool, got {type(halt_on_hack).__name__}"
|
||||
)
|
||||
if baseline_signal is not None:
|
||||
if isinstance(baseline_signal, bool):
|
||||
raise TypeError("baseline_signal must not be bool")
|
||||
if not isinstance(baseline_signal, (int, float)):
|
||||
raise TypeError("baseline_signal must be a number or None")
|
||||
if not math.isfinite(float(baseline_signal)) or float(baseline_signal) < 0.0:
|
||||
raise ValueError("baseline_signal must be finite and non-negative")
|
||||
raise NotImplementedError(
|
||||
f"Live reward-hack callback for detector={detector!r} is deferred to "
|
||||
"v0.70.1. v0.70.0 ships the schema + math kernels only."
|
||||
)
|
||||
|
|
@ -0,0 +1,209 @@
|
|||
"""Mid-epoch checkpoint for PPO / GRPO — v0.70.0 Part D.
|
||||
|
||||
Optimizer-state serialization for long RL runs that need to survive
|
||||
preemption mid-rollout. TorchTune explicitly punts this (their README
|
||||
notes "we expect users to resume from the start of the most recent
|
||||
epoch"); Soup ships a real mid-epoch save/load surface here.
|
||||
|
||||
Composes with:
|
||||
- v0.32 spike-recovery (the recovery policy can hop back to the most
|
||||
recent rl checkpoint instead of the start of the epoch).
|
||||
- v0.40.0 ref-model regen (ref-model state is captured in the
|
||||
checkpoint manifest so a resume restarts with the same reference
|
||||
distribution).
|
||||
|
||||
Schema-only release; live save_state / load_state HF Trainer callback
|
||||
deferred to v0.70.1. Mirrors v0.50.0 / v0.62.0 / v0.69.0 stub-then-live
|
||||
cadence.
|
||||
|
||||
Security:
|
||||
- ``RLCheckpointConfig`` is frozen + per-field-validated.
|
||||
- Bool / non-int rejection on every numeric (bool-before-int policy).
|
||||
- ``RLCheckpointState.task`` allowlisted to RL tasks only.
|
||||
- ``checkpoint_dir`` shape-validated (null-byte / oversize) — cwd
|
||||
containment + symlink rejection happen at v0.70.1 disk-write time
|
||||
(matches v0.69.0 build_dag deferred-write-containment policy).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
_MAX_SAVE_EVERY_STEPS = 10_000_000
|
||||
_MIN_KEEP_LAST = 1
|
||||
_MAX_KEEP_LAST = 100
|
||||
_MAX_DIR_LEN = 4096
|
||||
_MAX_SOUP_VERSION_LEN = 32
|
||||
|
||||
# RL tasks whose mid-epoch checkpoint makes sense. Other tasks already
|
||||
# have HF Trainer's per-epoch checkpoint and don't need the rollout
|
||||
# / ref-model state captured.
|
||||
_RL_TASKS: frozenset[str] = frozenset({"grpo", "ppo"})
|
||||
|
||||
|
||||
def validate_save_every_steps(value: object) -> int:
|
||||
"""Bool-rejecting + bounded validator for save_every_steps.
|
||||
|
||||
Range: ``[1, _MAX_SAVE_EVERY_STEPS=10_000_000]``.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("save_every_steps must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(
|
||||
f"save_every_steps must be int, got {type(value).__name__}"
|
||||
)
|
||||
if value < 1:
|
||||
raise ValueError(f"save_every_steps must be >= 1, got {value}")
|
||||
if value > _MAX_SAVE_EVERY_STEPS:
|
||||
raise ValueError(
|
||||
f"save_every_steps={value} exceeds {_MAX_SAVE_EVERY_STEPS} cap"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _validate_keep_last(value: object) -> int:
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("keep_last must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(
|
||||
f"keep_last must be int, got {type(value).__name__}"
|
||||
)
|
||||
if value < _MIN_KEEP_LAST or value > _MAX_KEEP_LAST:
|
||||
raise ValueError(
|
||||
f"keep_last must be in [{_MIN_KEEP_LAST}, {_MAX_KEEP_LAST}], "
|
||||
f"got {value}"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _validate_bool_flag(value: object, field: str) -> bool:
|
||||
if not isinstance(value, bool):
|
||||
raise TypeError(f"{field} must be bool, got {type(value).__name__}")
|
||||
return value
|
||||
|
||||
|
||||
def _validate_dir_shape(value: object, field: str) -> str:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must be str, got bool")
|
||||
if not isinstance(value, str):
|
||||
raise TypeError(f"{field} must be str, got {type(value).__name__}")
|
||||
if not value:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
if "\x00" in value:
|
||||
raise ValueError(f"{field} must not contain null bytes")
|
||||
if len(value) > _MAX_DIR_LEN:
|
||||
raise ValueError(f"{field} exceeds {_MAX_DIR_LEN} chars")
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RLCheckpointConfig:
|
||||
"""Frozen RL-checkpoint config.
|
||||
|
||||
- ``save_every_steps``: int >= 1; ``[1, 10_000_000]`` cap.
|
||||
- ``include_optimizer_state``: include AdamW / Lion state in the
|
||||
checkpoint. Default True (the whole point of mid-epoch RL ckpt).
|
||||
- ``include_ref_model``: include the frozen reference model state.
|
||||
Default False (ref model can usually be reconstructed from
|
||||
``cfg.base``).
|
||||
- ``include_rollout_buffer``: include the replay / rollout buffer
|
||||
so resumed runs don't lose collected experience.
|
||||
- ``keep_last``: number of recent checkpoints to retain. Older
|
||||
checkpoints get pruned at write time.
|
||||
"""
|
||||
|
||||
save_every_steps: int
|
||||
include_optimizer_state: bool = True
|
||||
include_ref_model: bool = False
|
||||
include_rollout_buffer: bool = False
|
||||
keep_last: int = 3
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
validate_save_every_steps(self.save_every_steps)
|
||||
_validate_bool_flag(self.include_optimizer_state, "include_optimizer_state")
|
||||
_validate_bool_flag(self.include_ref_model, "include_ref_model")
|
||||
_validate_bool_flag(self.include_rollout_buffer, "include_rollout_buffer")
|
||||
_validate_keep_last(self.keep_last)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RLCheckpointState:
|
||||
"""Frozen state manifest persisted at each mid-epoch save.
|
||||
|
||||
Written to ``<checkpoint_dir>/manifest.json``. Allows a resume
|
||||
handler in v0.70.1 to verify the checkpoint shape before loading
|
||||
the (potentially large) optimizer state.
|
||||
|
||||
- ``step``: training step at which the checkpoint was taken.
|
||||
Non-negative int (bool rejected per project policy).
|
||||
- ``checkpoint_dir``: directory holding the saved tensors. Shape
|
||||
validated only (cwd-containment deferred to v0.70.1 disk hook).
|
||||
- ``task``: must be in the RL allowlist (``grpo`` / ``ppo``).
|
||||
- ``has_optimizer``/``has_ref_model``/``has_rollout_buffer``: real
|
||||
bools (no str/int coercion).
|
||||
- ``soup_version``: capped at 32 chars; null-byte rejected.
|
||||
"""
|
||||
|
||||
step: int
|
||||
checkpoint_dir: str
|
||||
task: str
|
||||
has_optimizer: bool
|
||||
has_ref_model: bool
|
||||
has_rollout_buffer: bool
|
||||
soup_version: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if isinstance(self.step, bool):
|
||||
raise ValueError("step must not be bool")
|
||||
if not isinstance(self.step, int):
|
||||
raise TypeError(f"step must be int, got {type(self.step).__name__}")
|
||||
if self.step < 0:
|
||||
raise ValueError(f"step must be non-negative, got {self.step}")
|
||||
_validate_dir_shape(self.checkpoint_dir, "checkpoint_dir")
|
||||
if not isinstance(self.task, str) or not self.task:
|
||||
raise ValueError("task must be a non-empty string")
|
||||
if self.task not in _RL_TASKS:
|
||||
raise ValueError(
|
||||
f"task={self.task!r} must be one of {sorted(_RL_TASKS)}"
|
||||
)
|
||||
_validate_bool_flag(self.has_optimizer, "has_optimizer")
|
||||
_validate_bool_flag(self.has_ref_model, "has_ref_model")
|
||||
_validate_bool_flag(self.has_rollout_buffer, "has_rollout_buffer")
|
||||
if not isinstance(self.soup_version, str) or not self.soup_version:
|
||||
raise ValueError("soup_version must be a non-empty string")
|
||||
if "\x00" in self.soup_version:
|
||||
raise ValueError("soup_version must not contain null bytes")
|
||||
if len(self.soup_version) > _MAX_SOUP_VERSION_LEN:
|
||||
raise ValueError(
|
||||
f"soup_version exceeds {_MAX_SOUP_VERSION_LEN} chars"
|
||||
)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
"""JSON-serialisable dict for the manifest file."""
|
||||
return {
|
||||
"step": self.step,
|
||||
"checkpoint_dir": self.checkpoint_dir,
|
||||
"task": self.task,
|
||||
"has_optimizer": self.has_optimizer,
|
||||
"has_ref_model": self.has_ref_model,
|
||||
"has_rollout_buffer": self.has_rollout_buffer,
|
||||
"soup_version": self.soup_version,
|
||||
}
|
||||
|
||||
|
||||
def build_rl_checkpoint_callback(config):
|
||||
"""Live HF Trainer callback for mid-epoch RL checkpoints.
|
||||
|
||||
Deferred to v0.70.1. Validates the config type at the public
|
||||
boundary so misconfigured callers fail fast (mirrors v0.50.0 /
|
||||
v0.62.0 / v0.67.0 / v0.69.0 deferred-live policy).
|
||||
"""
|
||||
if not isinstance(config, RLCheckpointConfig):
|
||||
raise TypeError(
|
||||
f"config must be RLCheckpointConfig, got {type(config).__name__}"
|
||||
)
|
||||
raise NotImplementedError(
|
||||
"Live mid-epoch RL checkpoint callback is deferred to v0.70.1. "
|
||||
"v0.70.0 ships the schema + state manifest only."
|
||||
)
|
||||
|
|
@ -0,0 +1,200 @@
|
|||
"""Universal Logit Distillation (ULD) — v0.70.0 Part B.
|
||||
|
||||
Cross-tokenizer knowledge distillation extending v0.53.2 DistillTrainer
|
||||
to teacher/student pairs with **different vocabularies** (e.g. Llama →
|
||||
Mistral, Llama → Qwen, Qwen → Llama). The existing v0.53.2 path
|
||||
assumes column-wise logit alignment, which fails the moment student
|
||||
and teacher have different vocab sizes.
|
||||
|
||||
Two strategies ship in v0.70.0 (schema-only; live projection module
|
||||
wired in v0.70.1):
|
||||
|
||||
- ``wasserstein``: Boizard et al. 2024 ULD — uses 1D Wasserstein
|
||||
distance between sorted teacher / student logit distributions. No
|
||||
alignment required; works across arbitrary vocab boundaries.
|
||||
(arXiv 2402.12030)
|
||||
|
||||
- ``topk_align``: Top-K projection — pick top-K teacher logits, map
|
||||
via BPE-overlap heuristic to student token ids, distil only on the
|
||||
aligned subset. Requires ``top_k`` to be set.
|
||||
|
||||
Live wiring deferred to v0.70.1 — mirrors v0.27.0 MII / v0.50.0 GRPO
|
||||
Plus / v0.61.0 unlearning / v0.62.0 RAG stub-then-live pattern.
|
||||
|
||||
Security:
|
||||
- Closed allowlist (frozenset); arbitrary strategy rejected.
|
||||
- Bool / null-byte / non-string / oversize rejection on every validator.
|
||||
- Vocab-size bounds: [1, 262144] (covers multilingual SentencePiece +
|
||||
GPT-OSS 200K vocab; matches v0.42 token-cap policy).
|
||||
- ``top_k`` mutually exclusive with non-topk_align strategies (silent
|
||||
no-op footgun rejection mirroring v0.52 distill / classifier policy).
|
||||
- No top-level torch import — lazy import inside ``build_uld_projection``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import types
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
_MAX_STRATEGY_NAME_LEN = 32
|
||||
# 262144 covers multilingual SentencePiece (e.g. NLLB) + GPT-OSS 200K.
|
||||
_MAX_VOCAB_SIZE = 262144
|
||||
|
||||
SUPPORTED_ULD_STRATEGIES: frozenset[str] = frozenset({"wasserstein", "topk_align"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ULDStrategySpec:
|
||||
"""Static metadata for a ULD strategy."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
requires_top_k: bool
|
||||
|
||||
|
||||
_STRATEGY_METADATA = types.MappingProxyType({
|
||||
"wasserstein": ULDStrategySpec(
|
||||
name="wasserstein",
|
||||
description=(
|
||||
"1D Wasserstein distance on sorted logit distributions. "
|
||||
"No alignment required across vocabularies."
|
||||
),
|
||||
requires_top_k=False,
|
||||
),
|
||||
"topk_align": ULDStrategySpec(
|
||||
name="topk_align",
|
||||
description=(
|
||||
"Top-K teacher logit alignment via BPE overlap. Requires "
|
||||
"top_k to be set."
|
||||
),
|
||||
requires_top_k=True,
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
def validate_uld_strategy(name: object) -> str:
|
||||
"""Validate and normalise a ULD strategy name.
|
||||
|
||||
Returns the canonical (lower-cased) name. Raises ``ValueError`` on
|
||||
any failure.
|
||||
"""
|
||||
if isinstance(name, bool):
|
||||
raise ValueError("uld_strategy must be a string, got bool")
|
||||
if not isinstance(name, str):
|
||||
raise ValueError(
|
||||
f"uld_strategy must be a string, got {type(name).__name__}"
|
||||
)
|
||||
if not name:
|
||||
raise ValueError("uld_strategy must be a non-empty string")
|
||||
if "\x00" in name:
|
||||
raise ValueError("uld_strategy must not contain null bytes")
|
||||
if len(name) > _MAX_STRATEGY_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"uld_strategy exceeds {_MAX_STRATEGY_NAME_LEN} chars"
|
||||
)
|
||||
normalised = name.lower()
|
||||
if normalised not in SUPPORTED_ULD_STRATEGIES:
|
||||
raise ValueError(
|
||||
f"uld_strategy={name!r} is not supported. "
|
||||
f"Valid: {sorted(SUPPORTED_ULD_STRATEGIES)}"
|
||||
)
|
||||
return normalised
|
||||
|
||||
|
||||
def get_strategy_spec(name: str) -> ULDStrategySpec:
|
||||
"""Return the :class:`ULDStrategySpec` for ``name``."""
|
||||
normalised = validate_uld_strategy(name)
|
||||
return _STRATEGY_METADATA[normalised]
|
||||
|
||||
|
||||
def validate_uld_projection_dim(value: object) -> int:
|
||||
"""Validate a projection dimensionality (vocab size).
|
||||
|
||||
Bounds: ``[1, _MAX_VOCAB_SIZE=262144]``. Bool rejected per project
|
||||
bool-as-int policy.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("dim must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(f"dim must be int, got {type(value).__name__}")
|
||||
if value < 1:
|
||||
raise ValueError(f"dim must be >= 1, got {value}")
|
||||
if value > _MAX_VOCAB_SIZE:
|
||||
raise ValueError(
|
||||
f"dim={value} exceeds {_MAX_VOCAB_SIZE} cap"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_uld_top_k(value: object) -> int:
|
||||
"""Validate ``top_k`` for the topk_align strategy.
|
||||
|
||||
Bounds: ``[1, _MAX_VOCAB_SIZE=262144]``. Bool rejected.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("uld_top_k must not be bool")
|
||||
if not isinstance(value, int):
|
||||
raise ValueError(
|
||||
f"uld_top_k must be int, got {type(value).__name__}"
|
||||
)
|
||||
if value < 1:
|
||||
raise ValueError(f"uld_top_k must be >= 1, got {value}")
|
||||
if value > _MAX_VOCAB_SIZE:
|
||||
raise ValueError(
|
||||
f"uld_top_k={value} exceeds {_MAX_VOCAB_SIZE} cap"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ULDConfig:
|
||||
"""Frozen ULD configuration.
|
||||
|
||||
- ``strategy``: one of :data:`SUPPORTED_ULD_STRATEGIES`.
|
||||
- ``student_vocab_size`` / ``teacher_vocab_size``: positive ints,
|
||||
bounded by ``_MAX_VOCAB_SIZE``.
|
||||
- ``top_k``: required when ``strategy='topk_align'``, rejected
|
||||
otherwise (silent no-op footgun rejection).
|
||||
"""
|
||||
|
||||
strategy: str
|
||||
student_vocab_size: int
|
||||
teacher_vocab_size: int
|
||||
top_k: Optional[int] = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
normalised = validate_uld_strategy(self.strategy)
|
||||
if normalised != self.strategy:
|
||||
object.__setattr__(self, "strategy", normalised)
|
||||
validate_uld_projection_dim(self.student_vocab_size)
|
||||
validate_uld_projection_dim(self.teacher_vocab_size)
|
||||
spec = _STRATEGY_METADATA[normalised]
|
||||
if spec.requires_top_k:
|
||||
if self.top_k is None:
|
||||
raise ValueError(
|
||||
f"uld_strategy='{normalised}' requires top_k to be set"
|
||||
)
|
||||
validate_uld_top_k(self.top_k)
|
||||
elif self.top_k is not None:
|
||||
raise ValueError(
|
||||
f"top_k is only valid when uld_strategy='topk_align'; "
|
||||
f"got uld_strategy='{normalised}'"
|
||||
)
|
||||
|
||||
|
||||
def build_uld_projection(config):
|
||||
"""Build the projection module that bridges teacher / student vocabs.
|
||||
|
||||
Deferred to v0.70.1. Validates the config type at the public boundary
|
||||
so misconfigured callers fail fast (mirrors v0.50.0 ``apply_variant_loss``
|
||||
/ v0.62.0 ``apply_steering`` / v0.67.0 ``apply_bank_to_serve`` policy).
|
||||
"""
|
||||
if not isinstance(config, ULDConfig):
|
||||
raise TypeError(
|
||||
f"config must be ULDConfig, got {type(config).__name__}"
|
||||
)
|
||||
raise NotImplementedError(
|
||||
f"Live ULD projection for strategy={config.strategy!r} is deferred "
|
||||
"to v0.70.1. v0.70.0 ships the schema + validators only."
|
||||
)
|
||||
|
|
@ -730,4 +730,5 @@ class TestSourceWiring:
|
|||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
major_minor = tuple(int(x) for x in __version__.split(".")[:2])
|
||||
assert major_minor >= (0, 69)
|
||||
|
|
|
|||
|
|
@ -584,4 +584,5 @@ class TestSourceWiring:
|
|||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
major_minor = tuple(int(x) for x in __version__.split(".")[:2])
|
||||
assert major_minor >= (0, 69)
|
||||
|
|
|
|||
|
|
@ -338,4 +338,5 @@ class TestSourceWiring:
|
|||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
major_minor = tuple(int(x) for x in __version__.split(".")[:2])
|
||||
assert major_minor >= (0, 69)
|
||||
|
|
|
|||
|
|
@ -413,4 +413,5 @@ class TestSourceWiring:
|
|||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
major_minor = tuple(int(x) for x in __version__.split(".")[:2])
|
||||
assert major_minor >= (0, 69)
|
||||
|
|
|
|||
|
|
@ -415,4 +415,5 @@ class TestSourceWiring:
|
|||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
major_minor = tuple(int(x) for x in __version__.split(".")[:2])
|
||||
assert major_minor >= (0, 69)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,582 @@
|
|||
"""v0.70.0 Part A — Reward-hacking detector schema + math kernels.
|
||||
|
||||
Schema-only release: live trainer-callback wiring deferred to v0.70.1.
|
||||
Tests cover: closed allowlist, frozen dataclasses, cluster-separation
|
||||
index kernel, RM-ensemble divergence kernel, classification helper,
|
||||
and all input rejection matrices.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import FrozenInstanceError
|
||||
|
||||
import pytest
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reward-hacking detector — public surface
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRewardHackingPublicSurface:
|
||||
def test_module_imports(self):
|
||||
from soup_cli.utils import reward_hacking
|
||||
|
||||
assert hasattr(reward_hacking, "SUPPORTED_HACK_DETECTORS")
|
||||
assert hasattr(reward_hacking, "validate_hack_detector")
|
||||
assert hasattr(reward_hacking, "compute_cluster_separation")
|
||||
assert hasattr(reward_hacking, "compute_rm_ensemble_divergence")
|
||||
assert hasattr(reward_hacking, "classify_hack_signal")
|
||||
assert hasattr(reward_hacking, "RewardHackReport")
|
||||
assert hasattr(reward_hacking, "build_reward_hack_callback")
|
||||
|
||||
def test_supported_detectors_is_frozenset(self):
|
||||
from soup_cli.utils.reward_hacking import SUPPORTED_HACK_DETECTORS
|
||||
|
||||
assert isinstance(SUPPORTED_HACK_DETECTORS, frozenset)
|
||||
assert "info_rm" in SUPPORTED_HACK_DETECTORS
|
||||
assert "rm_ensemble" in SUPPORTED_HACK_DETECTORS
|
||||
|
||||
def test_supported_detectors_immutable(self):
|
||||
from soup_cli.utils.reward_hacking import SUPPORTED_HACK_DETECTORS
|
||||
|
||||
with pytest.raises((AttributeError, TypeError)):
|
||||
SUPPORTED_HACK_DETECTORS.add("evil")
|
||||
|
||||
|
||||
class TestValidateHackDetector:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
assert validate_hack_detector("info_rm") == "info_rm"
|
||||
assert validate_hack_detector("rm_ensemble") == "rm_ensemble"
|
||||
|
||||
def test_case_insensitive(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
assert validate_hack_detector("INFO_RM") == "info_rm"
|
||||
assert validate_hack_detector("Rm_Ensemble") == "rm_ensemble"
|
||||
|
||||
def test_unknown_raises(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
with pytest.raises(ValueError, match="not supported"):
|
||||
validate_hack_detector("evil")
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_hack_detector(True)
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
validate_hack_detector("")
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
with pytest.raises(ValueError, match="string"):
|
||||
validate_hack_detector(123)
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
with pytest.raises(ValueError, match="null byte"):
|
||||
validate_hack_detector("info_rm\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import validate_hack_detector
|
||||
|
||||
with pytest.raises(ValueError, match="exceeds"):
|
||||
validate_hack_detector("x" * 64)
|
||||
|
||||
|
||||
class TestComputeClusterSeparation:
|
||||
"""InfoRM Cluster-Separation Index kernel.
|
||||
|
||||
Higher separation = healthier RM (clusters of (good, bad) responses
|
||||
are well-separated). Sharp drop in separation across training =
|
||||
reward-hacking signal.
|
||||
"""
|
||||
|
||||
def test_perfect_separation(self):
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
good_scores = [10.0, 9.5, 11.0]
|
||||
bad_scores = [1.0, 0.5, 1.5]
|
||||
# Far-apart clusters → high separation
|
||||
value = compute_cluster_separation(good_scores, bad_scores)
|
||||
assert math.isfinite(value)
|
||||
assert value > 1.0
|
||||
|
||||
def test_no_separation(self):
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
good_scores = [5.0, 5.5, 4.5]
|
||||
bad_scores = [5.0, 5.5, 4.5]
|
||||
value = compute_cluster_separation(good_scores, bad_scores)
|
||||
assert math.isfinite(value)
|
||||
assert abs(value) < 0.1
|
||||
|
||||
def test_zero_variance_groups(self):
|
||||
"""Zero variance is gracefully handled (small epsilon)."""
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
good_scores = [5.0, 5.0, 5.0]
|
||||
bad_scores = [1.0, 1.0, 1.0]
|
||||
value = compute_cluster_separation(good_scores, bad_scores)
|
||||
assert math.isfinite(value)
|
||||
assert value > 0.0
|
||||
|
||||
def test_empty_good_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
with pytest.raises(ValueError, match="empty"):
|
||||
compute_cluster_separation([], [1.0])
|
||||
|
||||
def test_empty_bad_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
with pytest.raises(ValueError, match="empty"):
|
||||
compute_cluster_separation([1.0], [])
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
compute_cluster_separation([1.0, float("nan")], [0.0])
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
compute_cluster_separation([float("inf")], [0.0])
|
||||
|
||||
def test_bool_in_list_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
compute_cluster_separation([1.0, True], [0.0])
|
||||
|
||||
def test_non_list_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_cluster_separation
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
compute_cluster_separation("not a list", [0.0])
|
||||
|
||||
|
||||
class TestComputeRmEnsembleDivergence:
|
||||
"""RM-ensemble divergence — measures disagreement across RMs.
|
||||
|
||||
Returns mean pairwise variance across RM score lists. High variance
|
||||
= RMs disagree = reward signal is unreliable.
|
||||
"""
|
||||
|
||||
def test_perfect_agreement(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
rm_scores = [
|
||||
[1.0, 2.0, 3.0], # RM 1
|
||||
[1.0, 2.0, 3.0], # RM 2 — identical
|
||||
[1.0, 2.0, 3.0], # RM 3 — identical
|
||||
]
|
||||
value = compute_rm_ensemble_divergence(rm_scores)
|
||||
assert math.isfinite(value)
|
||||
assert value < 1e-6
|
||||
|
||||
def test_disagreement(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
rm_scores = [
|
||||
[1.0, 2.0, 3.0],
|
||||
[5.0, 7.0, 9.0], # Wildly different
|
||||
[-2.0, -1.0, 0.0],
|
||||
]
|
||||
value = compute_rm_ensemble_divergence(rm_scores)
|
||||
assert math.isfinite(value)
|
||||
assert value > 1.0
|
||||
|
||||
def test_single_rm_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
with pytest.raises(ValueError, match="at least 2"):
|
||||
compute_rm_ensemble_divergence([[1.0, 2.0]])
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
with pytest.raises(ValueError, match="at least 2"):
|
||||
compute_rm_ensemble_divergence([])
|
||||
|
||||
def test_uneven_lengths_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
with pytest.raises(ValueError, match="length"):
|
||||
compute_rm_ensemble_divergence([[1.0, 2.0], [1.0, 2.0, 3.0]])
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
compute_rm_ensemble_divergence([[1.0, float("nan")], [0.0, 0.0]])
|
||||
|
||||
def test_non_list_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
compute_rm_ensemble_divergence("not a list")
|
||||
|
||||
def test_bool_inner_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
compute_rm_ensemble_divergence([[1.0, True], [0.0, 0.0]])
|
||||
|
||||
def test_too_many_rms_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence
|
||||
|
||||
too_many = [[1.0] for _ in range(100)]
|
||||
with pytest.raises(ValueError, match="too many"):
|
||||
compute_rm_ensemble_divergence(too_many)
|
||||
|
||||
|
||||
class TestClassifyHackSignal:
|
||||
"""OK / WARN / HACK taxonomy matches v0.26 Quant-Lobotomy bands.
|
||||
|
||||
Lower signal = healthier. Threshold semantics:
|
||||
- drop_pct < 0.10 -> OK
|
||||
- 0.10 <= drop_pct < 0.30 -> WARN
|
||||
- drop_pct >= 0.30 -> HACK
|
||||
"""
|
||||
|
||||
def test_ok_threshold(self):
|
||||
from soup_cli.utils.reward_hacking import classify_hack_signal
|
||||
|
||||
assert classify_hack_signal(0.05) == "OK"
|
||||
assert classify_hack_signal(0.0) == "OK"
|
||||
|
||||
def test_warn_threshold(self):
|
||||
from soup_cli.utils.reward_hacking import classify_hack_signal
|
||||
|
||||
assert classify_hack_signal(0.10) == "WARN"
|
||||
assert classify_hack_signal(0.20) == "WARN"
|
||||
assert classify_hack_signal(0.29) == "WARN"
|
||||
|
||||
def test_hack_threshold(self):
|
||||
from soup_cli.utils.reward_hacking import classify_hack_signal
|
||||
|
||||
assert classify_hack_signal(0.30) == "HACK"
|
||||
assert classify_hack_signal(0.50) == "HACK"
|
||||
assert classify_hack_signal(1.0) == "HACK"
|
||||
|
||||
def test_negative_signal_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import classify_hack_signal
|
||||
|
||||
with pytest.raises(ValueError, match="non-negative"):
|
||||
classify_hack_signal(-0.1)
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import classify_hack_signal
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
classify_hack_signal(float("nan"))
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
classify_hack_signal(float("inf"))
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import classify_hack_signal
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
classify_hack_signal(True)
|
||||
|
||||
def test_non_number_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import classify_hack_signal
|
||||
|
||||
with pytest.raises(ValueError, match="number"):
|
||||
classify_hack_signal("0.5")
|
||||
|
||||
|
||||
class TestRewardHackReport:
|
||||
def test_basic_construction(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
report = RewardHackReport(
|
||||
detector="info_rm",
|
||||
signal=0.15,
|
||||
verdict="WARN",
|
||||
step=100,
|
||||
baseline_signal=0.05,
|
||||
details=("cluster sep dropped 12%",),
|
||||
)
|
||||
assert report.detector == "info_rm"
|
||||
assert report.signal == 0.15
|
||||
assert report.verdict == "WARN"
|
||||
assert report.step == 100
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
report = RewardHackReport(
|
||||
detector="info_rm",
|
||||
signal=0.15,
|
||||
verdict="WARN",
|
||||
step=100,
|
||||
baseline_signal=0.05,
|
||||
details=(),
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
report.signal = 0.2 # type: ignore[misc]
|
||||
|
||||
def test_invalid_detector_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
with pytest.raises(ValueError, match="not supported"):
|
||||
RewardHackReport(
|
||||
detector="evil",
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=0,
|
||||
baseline_signal=0.0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_invalid_verdict_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
with pytest.raises(ValueError, match="verdict"):
|
||||
RewardHackReport(
|
||||
detector="info_rm",
|
||||
signal=0.0,
|
||||
verdict="EVIL",
|
||||
step=0,
|
||||
baseline_signal=0.0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_negative_signal_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
RewardHackReport(
|
||||
detector="info_rm",
|
||||
signal=-0.1,
|
||||
verdict="OK",
|
||||
step=0,
|
||||
baseline_signal=0.0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_negative_step_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
with pytest.raises(ValueError, match="step"):
|
||||
RewardHackReport(
|
||||
detector="info_rm",
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=-1,
|
||||
baseline_signal=0.0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_bool_step_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
RewardHackReport(
|
||||
detector="info_rm",
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=True,
|
||||
baseline_signal=0.0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_details_must_be_tuple(self):
|
||||
from soup_cli.utils.reward_hacking import RewardHackReport
|
||||
|
||||
with pytest.raises(TypeError, match="tuple"):
|
||||
RewardHackReport(
|
||||
detector="info_rm",
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=0,
|
||||
baseline_signal=0.0,
|
||||
details=["not a tuple"], # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
class TestBuildRewardHackCallbackStub:
|
||||
"""Live trainer-callback wiring deferred to v0.70.1.
|
||||
|
||||
The factory validates inputs at construction time and raises
|
||||
NotImplementedError with explicit v0.70.1 marker (mirrors v0.50.0
|
||||
apply_variant_loss policy).
|
||||
"""
|
||||
|
||||
def test_invalid_detector_rejected_before_deferred(self):
|
||||
from soup_cli.utils.reward_hacking import build_reward_hack_callback
|
||||
|
||||
# Validation runs BEFORE NotImplementedError.
|
||||
with pytest.raises(ValueError, match="not supported"):
|
||||
build_reward_hack_callback(detector="evil")
|
||||
|
||||
def test_deferred_v0701(self):
|
||||
from soup_cli.utils.reward_hacking import build_reward_hack_callback
|
||||
|
||||
with pytest.raises(NotImplementedError, match="v0.70.1"):
|
||||
build_reward_hack_callback(detector="info_rm")
|
||||
|
||||
def test_bool_halt_on_hack_rejected(self):
|
||||
from soup_cli.utils.reward_hacking import build_reward_hack_callback
|
||||
|
||||
with pytest.raises(TypeError, match="halt_on_hack"):
|
||||
build_reward_hack_callback(detector="info_rm", halt_on_hack="yes") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema integration — TrainingConfig + SoupConfig
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSchemaTrainingConfig:
|
||||
def test_default_none(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig()
|
||||
assert tcfg.reward_hack_detector is None
|
||||
assert tcfg.reward_hack_halt is False
|
||||
|
||||
def test_accept_info_rm(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(reward_hack_detector="info_rm")
|
||||
assert tcfg.reward_hack_detector == "info_rm"
|
||||
|
||||
def test_accept_rm_ensemble(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(reward_hack_detector="rm_ensemble")
|
||||
assert tcfg.reward_hack_detector == "rm_ensemble"
|
||||
|
||||
def test_unknown_detector_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(reward_hack_detector="evil")
|
||||
|
||||
def test_halt_must_be_bool(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
# Pydantic wraps the TypeError raised by the field validator into
|
||||
# a ValidationError at construction time.
|
||||
with pytest.raises((ValidationError, TypeError)):
|
||||
TrainingConfig(reward_hack_halt="yes") # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestSchemaSoupConfigTaskGate:
|
||||
"""reward_hack_detector + reward_hack_halt only meaningful on RL tasks
|
||||
(grpo / ppo). Rejected on SFT / DPO / etc. with friendly message.
|
||||
"""
|
||||
|
||||
def _yaml(self, task: str, detector: str = "info_rm", halt: bool = False) -> str:
|
||||
return f"""
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: {task}
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
reward_hack_detector: {detector}
|
||||
reward_hack_halt: {str(halt).lower()}
|
||||
"""
|
||||
|
||||
def test_grpo_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(self._yaml("grpo"))
|
||||
assert cfg.training.reward_hack_detector == "info_rm"
|
||||
|
||||
def test_ppo_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(self._yaml("ppo"))
|
||||
assert cfg.training.reward_hack_detector == "info_rm"
|
||||
|
||||
def test_sft_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="reward_hack"):
|
||||
load_config_from_string(self._yaml("sft"))
|
||||
|
||||
def test_dpo_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="reward_hack"):
|
||||
load_config_from_string(self._yaml("dpo"))
|
||||
|
||||
def test_halt_without_detector_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="reward_hack_detector"):
|
||||
load_config_from_string(
|
||||
"""
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: grpo
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
reward_hack_halt: true
|
||||
"""
|
||||
)
|
||||
|
||||
def test_mlx_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
load_config_from_string(
|
||||
"""
|
||||
base: mlx-community/Llama-3.1-8B
|
||||
task: grpo
|
||||
backend: mlx
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
reward_hack_detector: info_rm
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Source-grep wiring guards
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_module_no_top_level_torch(self):
|
||||
"""Lazy-import policy — torch imported inside functions only."""
|
||||
from pathlib import Path
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "utils"
|
||||
/ "reward_hacking.py"
|
||||
)
|
||||
body = src.read_text(encoding="utf-8")
|
||||
# Bare module-level torch import would be a perf regression.
|
||||
assert "\nimport torch" not in body
|
||||
assert "\nfrom torch" not in body
|
||||
|
||||
def test_version_bumped(self):
|
||||
import soup_cli
|
||||
|
||||
# We do not freeze here — checked via floor.
|
||||
major_minor = tuple(int(x) for x in soup_cli.__version__.split(".")[:2])
|
||||
# 0.70 floor.
|
||||
assert major_minor >= (0, 70)
|
||||
|
|
@ -0,0 +1,368 @@
|
|||
"""v0.70.0 Part B — Cross-tokenizer distillation (ULD).
|
||||
|
||||
Universal Logit Distillation (Boizard et al. 2024) for distilling
|
||||
across vocab boundaries (e.g. Llama -> Mistral). Schema-only release;
|
||||
live distill trainer hook deferred to v0.70.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import FrozenInstanceError
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestULDPublicSurface:
|
||||
def test_module_imports(self):
|
||||
from soup_cli.utils import uld
|
||||
|
||||
assert hasattr(uld, "SUPPORTED_ULD_STRATEGIES")
|
||||
assert hasattr(uld, "validate_uld_strategy")
|
||||
assert hasattr(uld, "validate_uld_projection_dim")
|
||||
assert hasattr(uld, "ULDConfig")
|
||||
assert hasattr(uld, "build_uld_projection")
|
||||
|
||||
def test_supported_strategies_frozenset(self):
|
||||
from soup_cli.utils.uld import SUPPORTED_ULD_STRATEGIES
|
||||
|
||||
assert isinstance(SUPPORTED_ULD_STRATEGIES, frozenset)
|
||||
assert "wasserstein" in SUPPORTED_ULD_STRATEGIES
|
||||
assert "topk_align" in SUPPORTED_ULD_STRATEGIES
|
||||
|
||||
def test_supported_strategies_immutable(self):
|
||||
from soup_cli.utils.uld import SUPPORTED_ULD_STRATEGIES
|
||||
|
||||
with pytest.raises((AttributeError, TypeError)):
|
||||
SUPPORTED_ULD_STRATEGIES.add("evil")
|
||||
|
||||
|
||||
class TestValidateULDStrategy:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
assert validate_uld_strategy("wasserstein") == "wasserstein"
|
||||
assert validate_uld_strategy("topk_align") == "topk_align"
|
||||
|
||||
def test_case_insensitive(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
assert validate_uld_strategy("WASSERSTEIN") == "wasserstein"
|
||||
assert validate_uld_strategy("TopK_Align") == "topk_align"
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_uld_strategy(True)
|
||||
|
||||
def test_unknown_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
with pytest.raises(ValueError, match="not supported"):
|
||||
validate_uld_strategy("evil")
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
validate_uld_strategy("")
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
with pytest.raises(ValueError, match="string"):
|
||||
validate_uld_strategy(42)
|
||||
|
||||
def test_null_byte_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
with pytest.raises(ValueError, match="null byte"):
|
||||
validate_uld_strategy("ws\x00")
|
||||
|
||||
def test_oversize_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_strategy
|
||||
|
||||
with pytest.raises(ValueError, match="exceeds"):
|
||||
validate_uld_strategy("x" * 64)
|
||||
|
||||
|
||||
class TestValidateProjectionDim:
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
assert validate_uld_projection_dim(128) == 128
|
||||
assert validate_uld_projection_dim(32000) == 32000
|
||||
|
||||
def test_minimum_boundary(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
assert validate_uld_projection_dim(1) == 1
|
||||
|
||||
def test_maximum_boundary(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
# 262144 — max plausible vocab size (multilingual SentencePiece).
|
||||
assert validate_uld_projection_dim(262144) == 262144
|
||||
|
||||
def test_zero_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
with pytest.raises(ValueError, match=">= 1"):
|
||||
validate_uld_projection_dim(0)
|
||||
|
||||
def test_negative_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
with pytest.raises(ValueError, match=">= 1"):
|
||||
validate_uld_projection_dim(-5)
|
||||
|
||||
def test_above_cap_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
with pytest.raises(ValueError, match="262144"):
|
||||
validate_uld_projection_dim(262145)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_uld_projection_dim(True)
|
||||
|
||||
def test_non_int_rejected(self):
|
||||
from soup_cli.utils.uld import validate_uld_projection_dim
|
||||
|
||||
with pytest.raises(ValueError, match="int"):
|
||||
validate_uld_projection_dim(128.5)
|
||||
|
||||
|
||||
class TestULDConfig:
|
||||
def test_basic(self):
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
cfg = ULDConfig(
|
||||
strategy="wasserstein",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
)
|
||||
assert cfg.strategy == "wasserstein"
|
||||
assert cfg.student_vocab_size == 32000
|
||||
assert cfg.teacher_vocab_size == 128256
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
cfg = ULDConfig(
|
||||
strategy="wasserstein",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
cfg.strategy = "topk_align" # type: ignore[misc]
|
||||
|
||||
def test_invalid_strategy_propagates(self):
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
with pytest.raises(ValueError, match="not supported"):
|
||||
ULDConfig(
|
||||
strategy="evil",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
)
|
||||
|
||||
def test_invalid_student_vocab(self):
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
ULDConfig(
|
||||
strategy="wasserstein",
|
||||
student_vocab_size=0,
|
||||
teacher_vocab_size=128256,
|
||||
)
|
||||
|
||||
def test_invalid_teacher_vocab(self):
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
ULDConfig(
|
||||
strategy="wasserstein",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=-1,
|
||||
)
|
||||
|
||||
def test_topk_default_optional(self):
|
||||
"""top_k defaults to None on wasserstein strategy."""
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
cfg = ULDConfig(
|
||||
strategy="wasserstein",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
)
|
||||
assert cfg.top_k is None
|
||||
|
||||
def test_topk_align_requires_topk(self):
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
with pytest.raises(ValueError, match="top_k"):
|
||||
ULDConfig(
|
||||
strategy="topk_align",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
)
|
||||
|
||||
def test_topk_align_accepts_topk(self):
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
cfg = ULDConfig(
|
||||
strategy="topk_align",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
top_k=128,
|
||||
)
|
||||
assert cfg.top_k == 128
|
||||
|
||||
def test_topk_on_wasserstein_rejected(self):
|
||||
"""top_k only makes sense on topk_align."""
|
||||
from soup_cli.utils.uld import ULDConfig
|
||||
|
||||
with pytest.raises(ValueError, match="top_k"):
|
||||
ULDConfig(
|
||||
strategy="wasserstein",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
top_k=128,
|
||||
)
|
||||
|
||||
|
||||
class TestBuildULDProjection:
|
||||
"""Deferred to v0.70.1 — validates config type then raises."""
|
||||
|
||||
def test_non_config_rejected(self):
|
||||
from soup_cli.utils.uld import build_uld_projection
|
||||
|
||||
with pytest.raises(TypeError, match="ULDConfig"):
|
||||
build_uld_projection({"strategy": "wasserstein"}) # type: ignore[arg-type]
|
||||
|
||||
def test_deferred_v0701(self):
|
||||
from soup_cli.utils.uld import ULDConfig, build_uld_projection
|
||||
|
||||
cfg = ULDConfig(
|
||||
strategy="wasserstein",
|
||||
student_vocab_size=32000,
|
||||
teacher_vocab_size=128256,
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.70.1"):
|
||||
build_uld_projection(cfg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema integration — TrainingConfig + SoupConfig
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSchemaTrainingConfig:
|
||||
def test_default_none(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig()
|
||||
assert tcfg.uld_strategy is None
|
||||
assert tcfg.uld_top_k is None
|
||||
|
||||
def test_accept_strategy(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(uld_strategy="wasserstein")
|
||||
assert tcfg.uld_strategy == "wasserstein"
|
||||
|
||||
def test_unknown_strategy_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(uld_strategy="evil")
|
||||
|
||||
def test_top_k_bounds(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
# Must be positive
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(uld_top_k=0)
|
||||
# Allowed.
|
||||
tcfg = TrainingConfig(uld_top_k=128)
|
||||
assert tcfg.uld_top_k == 128
|
||||
|
||||
|
||||
class TestSchemaSoupConfigTaskGate:
|
||||
"""uld_strategy only meaningful when task='distill'."""
|
||||
|
||||
def _yaml(
|
||||
self,
|
||||
task: str = "distill",
|
||||
strategy: str = "wasserstein",
|
||||
teacher: str = "meta-llama/Llama-3.1-8B",
|
||||
top_k: int | None = None,
|
||||
) -> str:
|
||||
topk_line = f" uld_top_k: {top_k}\n" if top_k is not None else ""
|
||||
teacher_line = f" teacher_model: {teacher}\n" if task == "distill" else ""
|
||||
return f"""
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: {task}
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
{teacher_line} uld_strategy: {strategy}
|
||||
{topk_line}"""
|
||||
|
||||
def test_distill_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(self._yaml(task="distill"))
|
||||
assert cfg.training.uld_strategy == "wasserstein"
|
||||
|
||||
def test_sft_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="uld_strategy"):
|
||||
load_config_from_string(self._yaml(task="sft"))
|
||||
|
||||
def test_topk_align_requires_topk_at_schema(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="uld_top_k"):
|
||||
load_config_from_string(
|
||||
self._yaml(strategy="topk_align")
|
||||
)
|
||||
|
||||
def test_topk_with_wasserstein_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="uld_top_k"):
|
||||
load_config_from_string(self._yaml(top_k=128))
|
||||
|
||||
def test_topk_align_with_topk_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(
|
||||
self._yaml(strategy="topk_align", top_k=128)
|
||||
)
|
||||
assert cfg.training.uld_strategy == "topk_align"
|
||||
assert cfg.training.uld_top_k == 128
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_module_no_top_level_torch(self):
|
||||
from pathlib import Path
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "utils"
|
||||
/ "uld.py"
|
||||
)
|
||||
body = src.read_text(encoding="utf-8")
|
||||
assert "\nimport torch" not in body
|
||||
assert "\nfrom torch" not in body
|
||||
|
|
@ -0,0 +1,393 @@
|
|||
"""v0.70.0 Part C — MiniLLM reverse-KL on-policy distillation.
|
||||
|
||||
MiniLLM (Gu et al. 2024) bundles 3 stability tricks:
|
||||
1. Teacher-mixed sampling (epsilon-greedy mix of teacher / student rollouts)
|
||||
2. Length normalisation on rollout completions
|
||||
3. Pretrain-loss anchor (add a small SFT-on-pretrain term to prevent drift)
|
||||
|
||||
Schema-only release; live trainer wiring deferred to v0.70.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import FrozenInstanceError
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestMiniLLMPublicSurface:
|
||||
def test_module_imports(self):
|
||||
from soup_cli.utils import minillm
|
||||
|
||||
assert hasattr(minillm, "MiniLLMConfig")
|
||||
assert hasattr(minillm, "validate_teacher_mix_ratio")
|
||||
assert hasattr(minillm, "validate_pretrain_anchor_weight")
|
||||
assert hasattr(minillm, "build_minillm_callback")
|
||||
|
||||
|
||||
class TestValidateTeacherMixRatio:
|
||||
"""Teacher mix ratio in [0, 1]. 0 = student-only; 1 = teacher-only."""
|
||||
|
||||
def test_happy_boundary_zero(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
assert validate_teacher_mix_ratio(0.0) == 0.0
|
||||
|
||||
def test_happy_boundary_one(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
assert validate_teacher_mix_ratio(1.0) == 1.0
|
||||
|
||||
def test_happy_mid(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
assert validate_teacher_mix_ratio(0.3) == 0.3
|
||||
|
||||
def test_above_one_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"):
|
||||
validate_teacher_mix_ratio(1.5)
|
||||
|
||||
def test_negative_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"):
|
||||
validate_teacher_mix_ratio(-0.1)
|
||||
|
||||
def test_nan_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
validate_teacher_mix_ratio(float("nan"))
|
||||
|
||||
def test_inf_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
validate_teacher_mix_ratio(float("inf"))
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_teacher_mix_ratio(True)
|
||||
|
||||
def test_non_number_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_teacher_mix_ratio
|
||||
|
||||
with pytest.raises(ValueError, match="number"):
|
||||
validate_teacher_mix_ratio("0.5")
|
||||
|
||||
|
||||
class TestValidatePretrainAnchorWeight:
|
||||
"""Pretrain anchor weight: small non-negative float, bounded [0, 1]."""
|
||||
|
||||
def test_happy_path(self):
|
||||
from soup_cli.utils.minillm import validate_pretrain_anchor_weight
|
||||
|
||||
assert validate_pretrain_anchor_weight(0.1) == 0.1
|
||||
|
||||
def test_zero_allowed(self):
|
||||
from soup_cli.utils.minillm import validate_pretrain_anchor_weight
|
||||
|
||||
assert validate_pretrain_anchor_weight(0.0) == 0.0
|
||||
|
||||
def test_one_allowed(self):
|
||||
from soup_cli.utils.minillm import validate_pretrain_anchor_weight
|
||||
|
||||
assert validate_pretrain_anchor_weight(1.0) == 1.0
|
||||
|
||||
def test_above_one_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_pretrain_anchor_weight
|
||||
|
||||
with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"):
|
||||
validate_pretrain_anchor_weight(1.1)
|
||||
|
||||
def test_negative_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_pretrain_anchor_weight
|
||||
|
||||
with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"):
|
||||
validate_pretrain_anchor_weight(-0.1)
|
||||
|
||||
def test_non_finite_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_pretrain_anchor_weight
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
validate_pretrain_anchor_weight(float("inf"))
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.minillm import validate_pretrain_anchor_weight
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_pretrain_anchor_weight(True)
|
||||
|
||||
|
||||
class TestMiniLLMConfig:
|
||||
def test_defaults(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
cfg = MiniLLMConfig()
|
||||
assert cfg.teacher_mix_ratio == 0.0
|
||||
assert cfg.length_normalize is True
|
||||
assert cfg.pretrain_anchor_weight == 0.0
|
||||
assert cfg.pretrain_anchor_path is None
|
||||
|
||||
def test_basic_config(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
cfg = MiniLLMConfig(
|
||||
teacher_mix_ratio=0.3,
|
||||
length_normalize=True,
|
||||
pretrain_anchor_weight=0.1,
|
||||
pretrain_anchor_path="./pretrain.jsonl",
|
||||
)
|
||||
assert cfg.teacher_mix_ratio == 0.3
|
||||
assert cfg.pretrain_anchor_path == "./pretrain.jsonl"
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
cfg = MiniLLMConfig()
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
cfg.teacher_mix_ratio = 0.5 # type: ignore[misc]
|
||||
|
||||
def test_invalid_mix_ratio_propagates(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MiniLLMConfig(teacher_mix_ratio=2.0)
|
||||
|
||||
def test_invalid_anchor_weight_propagates(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
MiniLLMConfig(pretrain_anchor_weight=-0.1)
|
||||
|
||||
def test_anchor_weight_without_path_rejected(self):
|
||||
"""If anchor_weight > 0, pretrain_anchor_path is required."""
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
with pytest.raises(ValueError, match="pretrain_anchor_path"):
|
||||
MiniLLMConfig(
|
||||
pretrain_anchor_weight=0.1,
|
||||
pretrain_anchor_path=None,
|
||||
)
|
||||
|
||||
def test_anchor_path_without_weight_rejected(self):
|
||||
"""If path is set but weight=0, silent no-op — reject."""
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
with pytest.raises(ValueError, match="pretrain_anchor_weight"):
|
||||
MiniLLMConfig(
|
||||
pretrain_anchor_weight=0.0,
|
||||
pretrain_anchor_path="./pretrain.jsonl",
|
||||
)
|
||||
|
||||
def test_length_normalize_must_be_bool(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
with pytest.raises(TypeError, match="bool"):
|
||||
MiniLLMConfig(length_normalize="yes") # type: ignore[arg-type]
|
||||
|
||||
def test_anchor_path_null_byte_rejected(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
with pytest.raises(ValueError, match="null byte"):
|
||||
MiniLLMConfig(
|
||||
pretrain_anchor_weight=0.1,
|
||||
pretrain_anchor_path="./bad\x00",
|
||||
)
|
||||
|
||||
def test_anchor_path_oversize_rejected(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig
|
||||
|
||||
with pytest.raises(ValueError, match="exceeds"):
|
||||
MiniLLMConfig(
|
||||
pretrain_anchor_weight=0.1,
|
||||
pretrain_anchor_path="./" + "x" * 5000,
|
||||
)
|
||||
|
||||
|
||||
class TestBuildMiniLLMCallback:
|
||||
"""Live trainer callback deferred to v0.70.1."""
|
||||
|
||||
def test_non_config_rejected(self):
|
||||
from soup_cli.utils.minillm import build_minillm_callback
|
||||
|
||||
with pytest.raises(TypeError, match="MiniLLMConfig"):
|
||||
build_minillm_callback({}) # type: ignore[arg-type]
|
||||
|
||||
def test_deferred(self):
|
||||
from soup_cli.utils.minillm import MiniLLMConfig, build_minillm_callback
|
||||
|
||||
cfg = MiniLLMConfig()
|
||||
with pytest.raises(NotImplementedError, match="v0.70.1"):
|
||||
build_minillm_callback(cfg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema integration — TrainingConfig + SoupConfig
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSchemaTrainingConfig:
|
||||
def test_defaults(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig()
|
||||
assert tcfg.minillm_enabled is False
|
||||
assert tcfg.minillm_teacher_mix_ratio == 0.0
|
||||
assert tcfg.minillm_length_normalize is True
|
||||
assert tcfg.minillm_pretrain_anchor_weight == 0.0
|
||||
assert tcfg.minillm_pretrain_anchor_path is None
|
||||
|
||||
def test_enabled_with_all_defaults(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(minillm_enabled=True)
|
||||
assert tcfg.minillm_enabled is True
|
||||
|
||||
def test_invalid_mix_ratio_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(minillm_teacher_mix_ratio=2.0)
|
||||
|
||||
def test_invalid_anchor_weight_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(minillm_pretrain_anchor_weight=1.5)
|
||||
|
||||
|
||||
class TestSchemaSoupConfigTaskGate:
|
||||
"""minillm_enabled only meaningful when task='distill'."""
|
||||
|
||||
def _yaml(self, task: str = "distill", **extras: object) -> str:
|
||||
teacher_line = (
|
||||
" teacher_model: meta-llama/Llama-3.1-8B\n"
|
||||
if task == "distill" else ""
|
||||
)
|
||||
extra_lines = "".join(f" {k}: {v}\n" for k, v in extras.items())
|
||||
return f"""
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: {task}
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
{teacher_line}{extra_lines}"""
|
||||
|
||||
def test_distill_minillm_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(
|
||||
self._yaml(task="distill", minillm_enabled=True)
|
||||
)
|
||||
assert cfg.training.minillm_enabled is True
|
||||
|
||||
def test_sft_minillm_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="minillm"):
|
||||
load_config_from_string(
|
||||
self._yaml(task="sft", minillm_enabled=True)
|
||||
)
|
||||
|
||||
def test_mlx_minillm_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
load_config_from_string(
|
||||
"""
|
||||
base: mlx-community/Llama-3.1-8B
|
||||
task: distill
|
||||
backend: mlx
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
teacher_model: meta-llama/Llama-3.1-8B
|
||||
minillm_enabled: true
|
||||
"""
|
||||
)
|
||||
|
||||
def test_anchor_weight_without_path_rejected_at_schema(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="minillm_pretrain_anchor_path"):
|
||||
load_config_from_string(
|
||||
self._yaml(
|
||||
task="distill",
|
||||
minillm_enabled=True,
|
||||
minillm_pretrain_anchor_weight=0.1,
|
||||
)
|
||||
)
|
||||
|
||||
def test_anchor_path_without_weight_rejected_at_schema(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="minillm_pretrain_anchor_weight"):
|
||||
load_config_from_string(
|
||||
self._yaml(
|
||||
task="distill",
|
||||
minillm_enabled=True,
|
||||
minillm_pretrain_anchor_path="./pre.jsonl",
|
||||
)
|
||||
)
|
||||
|
||||
def test_minillm_fields_without_enabled_rejected(self):
|
||||
"""Setting tunables without minillm_enabled=True is a silent no-op."""
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="minillm_enabled"):
|
||||
load_config_from_string(
|
||||
self._yaml(
|
||||
task="distill",
|
||||
minillm_teacher_mix_ratio=0.3,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Source wiring guards
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_module_no_top_level_torch(self):
|
||||
from pathlib import Path
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "utils"
|
||||
/ "minillm.py"
|
||||
)
|
||||
body = src.read_text(encoding="utf-8")
|
||||
assert "\nimport torch" not in body
|
||||
assert "\nfrom torch" not in body
|
||||
|
||||
def test_math_isfinite_used(self):
|
||||
"""Anchor-weight + mix-ratio guards must use math.isfinite (not
|
||||
the looser ``not nan`` idiom — matches v0.32 / v0.41 / v0.50 / v0.62
|
||||
finite-check policy).
|
||||
"""
|
||||
# Importing math at the top of the test triggers the regex.
|
||||
_ = math
|
||||
from pathlib import Path
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "utils"
|
||||
/ "minillm.py"
|
||||
)
|
||||
body = src.read_text(encoding="utf-8")
|
||||
assert "math.isfinite" in body
|
||||
|
|
@ -0,0 +1,352 @@
|
|||
"""v0.70.0 Part D — Mid-epoch checkpoint for PPO/GRPO.
|
||||
|
||||
Optimizer-state serialization for long RL runs. TorchTune explicitly
|
||||
punts this; Soup ships the schema + state-manifest builder here, with
|
||||
the live save_state / load_state callback deferred to v0.70.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import FrozenInstanceError
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestRLCheckpointPublicSurface:
|
||||
def test_module_imports(self):
|
||||
from soup_cli.utils import rl_checkpoint
|
||||
|
||||
assert hasattr(rl_checkpoint, "RLCheckpointConfig")
|
||||
assert hasattr(rl_checkpoint, "RLCheckpointState")
|
||||
assert hasattr(rl_checkpoint, "validate_save_every_steps")
|
||||
assert hasattr(rl_checkpoint, "build_rl_checkpoint_callback")
|
||||
|
||||
|
||||
class TestValidateSaveEverySteps:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.rl_checkpoint import validate_save_every_steps
|
||||
|
||||
assert validate_save_every_steps(100) == 100
|
||||
assert validate_save_every_steps(1) == 1
|
||||
|
||||
def test_max_boundary(self):
|
||||
from soup_cli.utils.rl_checkpoint import validate_save_every_steps
|
||||
|
||||
# 10M steps is plenty.
|
||||
assert validate_save_every_steps(10_000_000) == 10_000_000
|
||||
|
||||
def test_zero_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import validate_save_every_steps
|
||||
|
||||
with pytest.raises(ValueError, match=">= 1"):
|
||||
validate_save_every_steps(0)
|
||||
|
||||
def test_negative_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import validate_save_every_steps
|
||||
|
||||
with pytest.raises(ValueError, match=">= 1"):
|
||||
validate_save_every_steps(-10)
|
||||
|
||||
def test_above_cap_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import validate_save_every_steps
|
||||
|
||||
with pytest.raises(ValueError, match="10000000"):
|
||||
validate_save_every_steps(10_000_001)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import validate_save_every_steps
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_save_every_steps(True)
|
||||
|
||||
def test_non_int_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import validate_save_every_steps
|
||||
|
||||
with pytest.raises(ValueError, match="int"):
|
||||
validate_save_every_steps(100.5)
|
||||
|
||||
|
||||
class TestRLCheckpointConfig:
|
||||
def test_defaults(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointConfig
|
||||
|
||||
cfg = RLCheckpointConfig(save_every_steps=100)
|
||||
assert cfg.save_every_steps == 100
|
||||
assert cfg.include_optimizer_state is True
|
||||
assert cfg.include_ref_model is False
|
||||
assert cfg.include_rollout_buffer is False
|
||||
assert cfg.keep_last == 3
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointConfig
|
||||
|
||||
cfg = RLCheckpointConfig(save_every_steps=100)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
cfg.save_every_steps = 50 # type: ignore[misc]
|
||||
|
||||
def test_keep_last_bounds(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointConfig
|
||||
|
||||
cfg = RLCheckpointConfig(save_every_steps=100, keep_last=10)
|
||||
assert cfg.keep_last == 10
|
||||
|
||||
with pytest.raises(ValueError, match="keep_last"):
|
||||
RLCheckpointConfig(save_every_steps=100, keep_last=0)
|
||||
|
||||
with pytest.raises(ValueError, match="keep_last"):
|
||||
RLCheckpointConfig(save_every_steps=100, keep_last=101)
|
||||
|
||||
def test_keep_last_bool_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointConfig
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
RLCheckpointConfig(save_every_steps=100, keep_last=True)
|
||||
|
||||
def test_invalid_save_every_propagates(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointConfig
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
RLCheckpointConfig(save_every_steps=0)
|
||||
|
||||
def test_bool_flags_must_be_bool(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointConfig
|
||||
|
||||
with pytest.raises(TypeError, match="bool"):
|
||||
RLCheckpointConfig(
|
||||
save_every_steps=100,
|
||||
include_optimizer_state="yes", # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
class TestRLCheckpointState:
|
||||
"""State manifest written to .soup-rl-ckpt/step-NNN/manifest.json.
|
||||
|
||||
Frozen + JSON-serialisable + per-field validation.
|
||||
"""
|
||||
|
||||
def test_basic(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
st = RLCheckpointState(
|
||||
step=500,
|
||||
checkpoint_dir="./.soup-rl-ckpt/step-500",
|
||||
task="grpo",
|
||||
has_optimizer=True,
|
||||
has_ref_model=False,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
assert st.step == 500
|
||||
assert st.task == "grpo"
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
st = RLCheckpointState(
|
||||
step=500,
|
||||
checkpoint_dir="./.soup-rl-ckpt/step-500",
|
||||
task="grpo",
|
||||
has_optimizer=True,
|
||||
has_ref_model=False,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
st.step = 0 # type: ignore[misc]
|
||||
|
||||
def test_invalid_step_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
with pytest.raises(ValueError, match="step"):
|
||||
RLCheckpointState(
|
||||
step=-1,
|
||||
checkpoint_dir="./x",
|
||||
task="grpo",
|
||||
has_optimizer=True,
|
||||
has_ref_model=False,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
|
||||
def test_invalid_task_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
with pytest.raises(ValueError, match="task"):
|
||||
RLCheckpointState(
|
||||
step=10,
|
||||
checkpoint_dir="./x",
|
||||
task="sft", # not an RL task
|
||||
has_optimizer=True,
|
||||
has_ref_model=False,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
|
||||
def test_bool_step_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
RLCheckpointState(
|
||||
step=True,
|
||||
checkpoint_dir="./x",
|
||||
task="grpo",
|
||||
has_optimizer=True,
|
||||
has_ref_model=False,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
|
||||
def test_to_dict(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
st = RLCheckpointState(
|
||||
step=500,
|
||||
checkpoint_dir="./.soup-rl-ckpt/step-500",
|
||||
task="grpo",
|
||||
has_optimizer=True,
|
||||
has_ref_model=True,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
d = st.to_dict()
|
||||
assert isinstance(d, dict)
|
||||
assert d["step"] == 500
|
||||
assert d["task"] == "grpo"
|
||||
assert d["has_optimizer"] is True
|
||||
assert d["has_ref_model"] is True
|
||||
|
||||
def test_null_byte_dir_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
with pytest.raises(ValueError, match="null byte"):
|
||||
RLCheckpointState(
|
||||
step=10,
|
||||
checkpoint_dir="./bad\x00",
|
||||
task="grpo",
|
||||
has_optimizer=True,
|
||||
has_ref_model=False,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
|
||||
def test_bool_has_optimizer_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import RLCheckpointState
|
||||
|
||||
with pytest.raises(TypeError, match="has_optimizer"):
|
||||
RLCheckpointState(
|
||||
step=10,
|
||||
checkpoint_dir="./x",
|
||||
task="grpo",
|
||||
has_optimizer="yes", # type: ignore[arg-type]
|
||||
has_ref_model=False,
|
||||
has_rollout_buffer=False,
|
||||
soup_version="0.70.0",
|
||||
)
|
||||
|
||||
|
||||
class TestBuildRLCheckpointCallback:
|
||||
"""Live callback deferred to v0.70.1."""
|
||||
|
||||
def test_non_config_rejected(self):
|
||||
from soup_cli.utils.rl_checkpoint import build_rl_checkpoint_callback
|
||||
|
||||
with pytest.raises(TypeError, match="RLCheckpointConfig"):
|
||||
build_rl_checkpoint_callback({"save_every_steps": 100}) # type: ignore[arg-type]
|
||||
|
||||
def test_deferred(self):
|
||||
from soup_cli.utils.rl_checkpoint import (
|
||||
RLCheckpointConfig,
|
||||
build_rl_checkpoint_callback,
|
||||
)
|
||||
|
||||
cfg = RLCheckpointConfig(save_every_steps=100)
|
||||
with pytest.raises(NotImplementedError, match="v0.70.1"):
|
||||
build_rl_checkpoint_callback(cfg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema integration — TrainingConfig + SoupConfig
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSchemaTrainingConfig:
|
||||
def test_defaults(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig()
|
||||
assert tcfg.rl_checkpoint_save_every_steps is None
|
||||
assert tcfg.rl_checkpoint_keep_last == 3
|
||||
|
||||
def test_set_save_every(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(rl_checkpoint_save_every_steps=500)
|
||||
assert tcfg.rl_checkpoint_save_every_steps == 500
|
||||
|
||||
def test_zero_rejected(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(rl_checkpoint_save_every_steps=0)
|
||||
|
||||
def test_keep_last_bounds(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(rl_checkpoint_keep_last=0)
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(rl_checkpoint_keep_last=101)
|
||||
|
||||
|
||||
class TestSchemaSoupConfigTaskGate:
|
||||
"""rl_checkpoint_save_every_steps only meaningful on RL tasks
|
||||
(grpo/ppo)."""
|
||||
|
||||
def _yaml(self, task: str, save_every: int = 500) -> str:
|
||||
return f"""
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: {task}
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
rl_checkpoint_save_every_steps: {save_every}
|
||||
"""
|
||||
|
||||
def test_grpo_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(self._yaml("grpo"))
|
||||
assert cfg.training.rl_checkpoint_save_every_steps == 500
|
||||
|
||||
def test_ppo_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(self._yaml("ppo"))
|
||||
assert cfg.training.rl_checkpoint_save_every_steps == 500
|
||||
|
||||
def test_sft_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="rl_checkpoint"):
|
||||
load_config_from_string(self._yaml("sft"))
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_module_no_top_level_torch(self):
|
||||
from pathlib import Path
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "utils"
|
||||
/ "rl_checkpoint.py"
|
||||
)
|
||||
body = src.read_text(encoding="utf-8")
|
||||
assert "\nimport torch" not in body
|
||||
assert "\nfrom torch" not in body
|
||||
|
|
@ -0,0 +1,501 @@
|
|||
"""v0.70.0 Part E — Iterative DPO loop driver.
|
||||
|
||||
Sample → RM-score → re-pair → retrain over N rounds. Schema + CLI live;
|
||||
the actual round-orchestrator (which would invoke `soup train --task dpo`
|
||||
between rounds) is deferred to v0.70.1 (mirrors v0.68.0 local-rl
|
||||
nightly-train pattern).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import FrozenInstanceError
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
|
||||
class TestIterativeDPOPublicSurface:
|
||||
def test_module_imports(self):
|
||||
from soup_cli.utils import iterative_dpo
|
||||
|
||||
assert hasattr(iterative_dpo, "IterativeDPOPlan")
|
||||
assert hasattr(iterative_dpo, "IterativeDPORound")
|
||||
assert hasattr(iterative_dpo, "validate_rounds")
|
||||
assert hasattr(iterative_dpo, "validate_pairs_per_round")
|
||||
assert hasattr(iterative_dpo, "build_iterative_dpo_plan")
|
||||
assert hasattr(iterative_dpo, "run_iterative_dpo")
|
||||
|
||||
|
||||
class TestValidateRounds:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_rounds
|
||||
|
||||
assert validate_rounds(5) == 5
|
||||
|
||||
def test_min_boundary(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_rounds
|
||||
|
||||
assert validate_rounds(1) == 1
|
||||
|
||||
def test_max_boundary(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_rounds
|
||||
|
||||
# 100 rounds is plenty.
|
||||
assert validate_rounds(100) == 100
|
||||
|
||||
def test_zero_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_rounds
|
||||
|
||||
with pytest.raises(ValueError, match=">= 1"):
|
||||
validate_rounds(0)
|
||||
|
||||
def test_above_cap_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_rounds
|
||||
|
||||
with pytest.raises(ValueError, match="100"):
|
||||
validate_rounds(101)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_rounds
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_rounds(True)
|
||||
|
||||
def test_non_int_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_rounds
|
||||
|
||||
with pytest.raises(ValueError, match="int"):
|
||||
validate_rounds(5.5)
|
||||
|
||||
|
||||
class TestValidatePairsPerRound:
|
||||
def test_happy(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_pairs_per_round
|
||||
|
||||
assert validate_pairs_per_round(500) == 500
|
||||
|
||||
def test_min_boundary(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_pairs_per_round
|
||||
|
||||
assert validate_pairs_per_round(10) == 10
|
||||
|
||||
def test_max_boundary(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_pairs_per_round
|
||||
|
||||
assert validate_pairs_per_round(1_000_000) == 1_000_000
|
||||
|
||||
def test_below_min_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_pairs_per_round
|
||||
|
||||
with pytest.raises(ValueError, match=">= 10"):
|
||||
validate_pairs_per_round(9)
|
||||
|
||||
def test_above_cap_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_pairs_per_round
|
||||
|
||||
with pytest.raises(ValueError, match="1000000"):
|
||||
validate_pairs_per_round(1_000_001)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import validate_pairs_per_round
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
validate_pairs_per_round(True)
|
||||
|
||||
|
||||
class TestIterativeDPORound:
|
||||
def test_basic(self):
|
||||
from soup_cli.utils.iterative_dpo import IterativeDPORound
|
||||
|
||||
rnd = IterativeDPORound(
|
||||
round_index=1,
|
||||
prompts_path="./data/prompts.jsonl",
|
||||
pairs_path="./data/round1_pairs.jsonl",
|
||||
adapter_path="./output/round1",
|
||||
pairs_count=512,
|
||||
)
|
||||
assert rnd.round_index == 1
|
||||
assert rnd.pairs_count == 512
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.iterative_dpo import IterativeDPORound
|
||||
|
||||
rnd = IterativeDPORound(
|
||||
round_index=1,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./pairs.jsonl",
|
||||
adapter_path="./out",
|
||||
pairs_count=100,
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
rnd.round_index = 2 # type: ignore[misc]
|
||||
|
||||
def test_negative_round_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import IterativeDPORound
|
||||
|
||||
with pytest.raises(ValueError, match="round_index"):
|
||||
IterativeDPORound(
|
||||
round_index=-1,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./pairs.jsonl",
|
||||
adapter_path="./out",
|
||||
pairs_count=100,
|
||||
)
|
||||
|
||||
def test_bool_round_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import IterativeDPORound
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
IterativeDPORound(
|
||||
round_index=True,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./pairs.jsonl",
|
||||
adapter_path="./out",
|
||||
pairs_count=100,
|
||||
)
|
||||
|
||||
def test_null_byte_path_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import IterativeDPORound
|
||||
|
||||
with pytest.raises(ValueError, match="null byte"):
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./bad\x00",
|
||||
pairs_path="./pairs.jsonl",
|
||||
adapter_path="./out",
|
||||
pairs_count=100,
|
||||
)
|
||||
|
||||
def test_negative_pairs_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import IterativeDPORound
|
||||
|
||||
with pytest.raises(ValueError, match="pairs_count"):
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./pairs.jsonl",
|
||||
adapter_path="./out",
|
||||
pairs_count=-1,
|
||||
)
|
||||
|
||||
|
||||
class TestIterativeDPOPlan:
|
||||
def test_basic(self):
|
||||
from soup_cli.utils.iterative_dpo import (
|
||||
IterativeDPOPlan,
|
||||
IterativeDPORound,
|
||||
)
|
||||
|
||||
plan = IterativeDPOPlan(
|
||||
base_model="meta-llama/Llama-3.1-8B",
|
||||
reward_model="./output_rm",
|
||||
rounds=(
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r0.jsonl",
|
||||
adapter_path="./out/r0",
|
||||
pairs_count=512,
|
||||
),
|
||||
IterativeDPORound(
|
||||
round_index=1,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r1.jsonl",
|
||||
adapter_path="./out/r1",
|
||||
pairs_count=512,
|
||||
),
|
||||
),
|
||||
)
|
||||
assert len(plan.rounds) == 2
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.iterative_dpo import (
|
||||
IterativeDPOPlan,
|
||||
IterativeDPORound,
|
||||
)
|
||||
|
||||
plan = IterativeDPOPlan(
|
||||
base_model="m",
|
||||
reward_model="./rm",
|
||||
rounds=(
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r0.jsonl",
|
||||
adapter_path="./out/r0",
|
||||
pairs_count=10,
|
||||
),
|
||||
),
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
plan.base_model = "evil" # type: ignore[misc]
|
||||
|
||||
def test_rounds_must_be_tuple(self):
|
||||
from soup_cli.utils.iterative_dpo import (
|
||||
IterativeDPOPlan,
|
||||
IterativeDPORound,
|
||||
)
|
||||
|
||||
rounds_list = [
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r0.jsonl",
|
||||
adapter_path="./out/r0",
|
||||
pairs_count=10,
|
||||
),
|
||||
]
|
||||
with pytest.raises(TypeError, match="tuple"):
|
||||
IterativeDPOPlan(
|
||||
base_model="m",
|
||||
reward_model="./rm",
|
||||
rounds=rounds_list, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def test_zero_rounds_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import IterativeDPOPlan
|
||||
|
||||
with pytest.raises(ValueError, match="rounds"):
|
||||
IterativeDPOPlan(
|
||||
base_model="m",
|
||||
reward_model="./rm",
|
||||
rounds=(),
|
||||
)
|
||||
|
||||
def test_non_consecutive_round_indices_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import (
|
||||
IterativeDPOPlan,
|
||||
IterativeDPORound,
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="consecutive"):
|
||||
IterativeDPOPlan(
|
||||
base_model="m",
|
||||
reward_model="./rm",
|
||||
rounds=(
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r0.jsonl",
|
||||
adapter_path="./out/r0",
|
||||
pairs_count=10,
|
||||
),
|
||||
IterativeDPORound(
|
||||
round_index=5, # gap
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r5.jsonl",
|
||||
adapter_path="./out/r5",
|
||||
pairs_count=10,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
def test_null_byte_base_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import (
|
||||
IterativeDPOPlan,
|
||||
IterativeDPORound,
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="null byte"):
|
||||
IterativeDPOPlan(
|
||||
base_model="m\x00",
|
||||
reward_model="./rm",
|
||||
rounds=(
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r0.jsonl",
|
||||
adapter_path="./out/r0",
|
||||
pairs_count=10,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class TestBuildIterativeDPOPlan:
|
||||
def test_happy(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils.iterative_dpo import build_iterative_dpo_plan
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "prompts.jsonl").write_text(
|
||||
'{"prompt": "hello"}\n', encoding="utf-8"
|
||||
)
|
||||
plan = build_iterative_dpo_plan(
|
||||
base_model="meta-llama/Llama-3.1-8B",
|
||||
reward_model="./rm",
|
||||
prompts_path="./prompts.jsonl",
|
||||
output_dir="./out",
|
||||
rounds=3,
|
||||
pairs_per_round=100,
|
||||
)
|
||||
assert len(plan.rounds) == 3
|
||||
assert plan.rounds[0].round_index == 0
|
||||
assert plan.rounds[2].round_index == 2
|
||||
assert plan.rounds[0].adapter_path != plan.rounds[1].adapter_path
|
||||
|
||||
def test_rounds_validation(self, tmp_path, monkeypatch):
|
||||
from soup_cli.utils.iterative_dpo import build_iterative_dpo_plan
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="rounds"):
|
||||
build_iterative_dpo_plan(
|
||||
base_model="m",
|
||||
reward_model="./rm",
|
||||
prompts_path="./prompts.jsonl",
|
||||
output_dir="./out",
|
||||
rounds=0,
|
||||
pairs_per_round=100,
|
||||
)
|
||||
|
||||
|
||||
class TestRunIterativeDPODeferred:
|
||||
def test_non_plan_rejected(self):
|
||||
from soup_cli.utils.iterative_dpo import run_iterative_dpo
|
||||
|
||||
with pytest.raises(TypeError, match="IterativeDPOPlan"):
|
||||
run_iterative_dpo({"rounds": 1}) # type: ignore[arg-type]
|
||||
|
||||
def test_deferred(self):
|
||||
from soup_cli.utils.iterative_dpo import (
|
||||
IterativeDPOPlan,
|
||||
IterativeDPORound,
|
||||
run_iterative_dpo,
|
||||
)
|
||||
|
||||
plan = IterativeDPOPlan(
|
||||
base_model="m",
|
||||
reward_model="./rm",
|
||||
rounds=(
|
||||
IterativeDPORound(
|
||||
round_index=0,
|
||||
prompts_path="./p.jsonl",
|
||||
pairs_path="./r0.jsonl",
|
||||
adapter_path="./out/r0",
|
||||
pairs_count=10,
|
||||
),
|
||||
),
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.70.1"):
|
||||
run_iterative_dpo(plan)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI smoke
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestIterativeDPOCli:
|
||||
def test_help(self):
|
||||
from soup_cli.commands.iterative_dpo import app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "iterative" in result.output.lower() or "dpo" in result.output.lower()
|
||||
|
||||
def test_plan_only_happy(self, tmp_path, monkeypatch):
|
||||
from soup_cli.commands.iterative_dpo import app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8")
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"--base-model",
|
||||
"meta-llama/Llama-3.1-8B",
|
||||
"--reward-model",
|
||||
"./rm",
|
||||
"--prompts",
|
||||
"./prompts.jsonl",
|
||||
"--output-dir",
|
||||
"./out",
|
||||
"--rounds",
|
||||
"3",
|
||||
"--pairs-per-round",
|
||||
"100",
|
||||
"--plan-only",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
|
||||
def test_invalid_rounds_exits_2(self, tmp_path, monkeypatch):
|
||||
from soup_cli.commands.iterative_dpo import app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8")
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"--base-model",
|
||||
"m",
|
||||
"--reward-model",
|
||||
"./rm",
|
||||
"--prompts",
|
||||
"./prompts.jsonl",
|
||||
"--output-dir",
|
||||
"./out",
|
||||
"--rounds",
|
||||
"0",
|
||||
"--pairs-per-round",
|
||||
"100",
|
||||
"--plan-only",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 2
|
||||
|
||||
def test_live_deferred_exits_3(self, tmp_path, monkeypatch):
|
||||
"""Without --plan-only, the deferred live runner exits 3."""
|
||||
from soup_cli.commands.iterative_dpo import app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8")
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"--base-model",
|
||||
"m",
|
||||
"--reward-model",
|
||||
"./rm",
|
||||
"--prompts",
|
||||
"./prompts.jsonl",
|
||||
"--output-dir",
|
||||
"./out",
|
||||
"--rounds",
|
||||
"2",
|
||||
"--pairs-per-round",
|
||||
"100",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 3, (result.output, repr(result.exception))
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_module_no_top_level_torch(self):
|
||||
from pathlib import Path
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "utils"
|
||||
/ "iterative_dpo.py"
|
||||
)
|
||||
body = src.read_text(encoding="utf-8")
|
||||
assert "\nimport torch" not in body
|
||||
assert "\nfrom torch" not in body
|
||||
|
||||
def test_cli_registered(self):
|
||||
"""soup iterative-dpo command registered on the top-level Typer app."""
|
||||
from pathlib import Path
|
||||
|
||||
cli_src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "cli.py"
|
||||
)
|
||||
body = cli_src.read_text(encoding="utf-8")
|
||||
# Either app.command or app.add_typer wiring.
|
||||
assert "iterative_dpo" in body or "iterative-dpo" in body
|
||||
|
|
@ -0,0 +1,409 @@
|
|||
"""v0.70.0 Part F — Live echo-trap detector.
|
||||
|
||||
RAGEN-style detection of trajectory degeneration in multi-turn agent RL.
|
||||
When the policy collapses to self-repeating outputs (echo trap), the
|
||||
reward stops improving and the policy drifts. Schema + math kernels
|
||||
live; trainer-callback wiring deferred to v0.70.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import FrozenInstanceError
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestEchoTrapPublicSurface:
|
||||
def test_module_imports(self):
|
||||
from soup_cli.utils import echo_trap
|
||||
|
||||
assert hasattr(echo_trap, "score_trajectory_repetition")
|
||||
assert hasattr(echo_trap, "score_echo_signal")
|
||||
assert hasattr(echo_trap, "classify_echo_signal")
|
||||
assert hasattr(echo_trap, "EchoTrapReport")
|
||||
assert hasattr(echo_trap, "build_echo_trap_callback")
|
||||
assert hasattr(echo_trap, "VERDICTS")
|
||||
|
||||
|
||||
class TestScoreTrajectoryRepetition:
|
||||
"""Per-trajectory n-gram repetition score. Higher = more repetition.
|
||||
|
||||
Tail-mass = fraction of n-grams that repeat more than once. Returns
|
||||
a float in [0, 1].
|
||||
"""
|
||||
|
||||
def test_unique_trajectory_zero(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
# All unique tokens → 0 repetition.
|
||||
tokens = ["a", "b", "c", "d", "e"]
|
||||
score = score_trajectory_repetition(tokens, ngram_n=2)
|
||||
assert score == 0.0
|
||||
|
||||
def test_full_repetition(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
# Identical token throughout → near-perfect repetition.
|
||||
tokens = ["a"] * 10
|
||||
score = score_trajectory_repetition(tokens, ngram_n=2)
|
||||
assert score > 0.5
|
||||
|
||||
def test_score_bounded(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
tokens = ["a", "b", "a", "b", "a", "b"]
|
||||
score = score_trajectory_repetition(tokens, ngram_n=2)
|
||||
assert 0.0 <= score <= 1.0
|
||||
|
||||
def test_short_returns_zero(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
# Fewer tokens than ngram_n → no n-grams possible.
|
||||
assert score_trajectory_repetition(["a"], ngram_n=2) == 0.0
|
||||
|
||||
def test_empty_returns_zero(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
assert score_trajectory_repetition([], ngram_n=2) == 0.0
|
||||
|
||||
def test_invalid_ngram_n_rejected(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
with pytest.raises(ValueError, match="ngram_n"):
|
||||
score_trajectory_repetition(["a", "b"], ngram_n=0)
|
||||
with pytest.raises(ValueError, match="ngram_n"):
|
||||
score_trajectory_repetition(["a", "b"], ngram_n=-1)
|
||||
|
||||
def test_bool_ngram_n_rejected(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
score_trajectory_repetition(["a", "b"], ngram_n=True)
|
||||
|
||||
def test_ngram_n_max_cap(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
with pytest.raises(ValueError, match="32"):
|
||||
score_trajectory_repetition(["a", "b"], ngram_n=33)
|
||||
|
||||
def test_non_string_token_rejected(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
with pytest.raises(TypeError, match="tokens"):
|
||||
score_trajectory_repetition([1, 2, 3], ngram_n=2) # type: ignore[list-item]
|
||||
|
||||
def test_non_list_tokens_rejected(self):
|
||||
from soup_cli.utils.echo_trap import score_trajectory_repetition
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
score_trajectory_repetition("abc", ngram_n=2)
|
||||
|
||||
|
||||
class TestScoreEchoSignal:
|
||||
"""Aggregate echo signal over a batch of trajectories.
|
||||
|
||||
Returns the mean repetition score across the batch.
|
||||
"""
|
||||
|
||||
def test_clean_trajectories(self):
|
||||
from soup_cli.utils.echo_trap import score_echo_signal
|
||||
|
||||
batch = [
|
||||
["a", "b", "c", "d"],
|
||||
["e", "f", "g", "h"],
|
||||
]
|
||||
score = score_echo_signal(batch, ngram_n=2)
|
||||
assert math.isfinite(score)
|
||||
assert 0.0 <= score < 0.1
|
||||
|
||||
def test_collapsed_batch(self):
|
||||
from soup_cli.utils.echo_trap import score_echo_signal
|
||||
|
||||
batch = [["a"] * 10, ["b"] * 10]
|
||||
score = score_echo_signal(batch, ngram_n=2)
|
||||
assert score > 0.5
|
||||
|
||||
def test_empty_batch(self):
|
||||
from soup_cli.utils.echo_trap import score_echo_signal
|
||||
|
||||
assert score_echo_signal([], ngram_n=2) == 0.0
|
||||
|
||||
def test_mixed_batch(self):
|
||||
from soup_cli.utils.echo_trap import score_echo_signal
|
||||
|
||||
batch = [["a"] * 10, ["x", "y", "z"]]
|
||||
score = score_echo_signal(batch, ngram_n=2)
|
||||
# Average of repetitive + clean.
|
||||
assert 0.0 < score < 1.0
|
||||
|
||||
def test_non_list_batch_rejected(self):
|
||||
from soup_cli.utils.echo_trap import score_echo_signal
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
score_echo_signal("not a list", ngram_n=2)
|
||||
|
||||
def test_batch_size_cap(self):
|
||||
from soup_cli.utils.echo_trap import score_echo_signal
|
||||
|
||||
big = [["x"] for _ in range(100_001)]
|
||||
with pytest.raises(ValueError, match="batch"):
|
||||
score_echo_signal(big, ngram_n=2)
|
||||
|
||||
|
||||
class TestClassifyEchoSignal:
|
||||
"""OK / WARN / TRAP taxonomy (mirrors v0.26 / v0.56 / v0.70 Part A).
|
||||
|
||||
- signal < 0.30: OK
|
||||
- 0.30 <= signal < 0.60: WARN
|
||||
- signal >= 0.60: TRAP
|
||||
"""
|
||||
|
||||
def test_ok(self):
|
||||
from soup_cli.utils.echo_trap import classify_echo_signal
|
||||
|
||||
assert classify_echo_signal(0.0) == "OK"
|
||||
assert classify_echo_signal(0.1) == "OK"
|
||||
assert classify_echo_signal(0.29) == "OK"
|
||||
|
||||
def test_warn(self):
|
||||
from soup_cli.utils.echo_trap import classify_echo_signal
|
||||
|
||||
assert classify_echo_signal(0.30) == "WARN"
|
||||
assert classify_echo_signal(0.45) == "WARN"
|
||||
assert classify_echo_signal(0.59) == "WARN"
|
||||
|
||||
def test_trap(self):
|
||||
from soup_cli.utils.echo_trap import classify_echo_signal
|
||||
|
||||
assert classify_echo_signal(0.60) == "TRAP"
|
||||
assert classify_echo_signal(0.99) == "TRAP"
|
||||
assert classify_echo_signal(1.0) == "TRAP"
|
||||
|
||||
def test_invalid_signal_rejected(self):
|
||||
from soup_cli.utils.echo_trap import classify_echo_signal
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
classify_echo_signal(float("nan"))
|
||||
with pytest.raises(ValueError):
|
||||
classify_echo_signal(-0.1)
|
||||
with pytest.raises(ValueError):
|
||||
classify_echo_signal(1.5)
|
||||
|
||||
def test_bool_rejected(self):
|
||||
from soup_cli.utils.echo_trap import classify_echo_signal
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
classify_echo_signal(True)
|
||||
|
||||
|
||||
class TestEchoTrapReport:
|
||||
def test_basic(self):
|
||||
from soup_cli.utils.echo_trap import EchoTrapReport
|
||||
|
||||
report = EchoTrapReport(
|
||||
signal=0.4,
|
||||
verdict="WARN",
|
||||
step=200,
|
||||
trajectories_seen=64,
|
||||
details=("longest streak: a a a a a",),
|
||||
)
|
||||
assert report.signal == 0.4
|
||||
assert report.verdict == "WARN"
|
||||
|
||||
def test_frozen(self):
|
||||
from soup_cli.utils.echo_trap import EchoTrapReport
|
||||
|
||||
report = EchoTrapReport(
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=0,
|
||||
trajectories_seen=0,
|
||||
details=(),
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
report.signal = 1.0 # type: ignore[misc]
|
||||
|
||||
def test_invalid_verdict_rejected(self):
|
||||
from soup_cli.utils.echo_trap import EchoTrapReport
|
||||
|
||||
with pytest.raises(ValueError, match="verdict"):
|
||||
EchoTrapReport(
|
||||
signal=0.0,
|
||||
verdict="EVIL",
|
||||
step=0,
|
||||
trajectories_seen=0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_signal_out_of_range_rejected(self):
|
||||
from soup_cli.utils.echo_trap import EchoTrapReport
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
EchoTrapReport(
|
||||
signal=1.5,
|
||||
verdict="OK",
|
||||
step=0,
|
||||
trajectories_seen=0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_bool_step_rejected(self):
|
||||
from soup_cli.utils.echo_trap import EchoTrapReport
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
EchoTrapReport(
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=True,
|
||||
trajectories_seen=0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_negative_trajectories_rejected(self):
|
||||
from soup_cli.utils.echo_trap import EchoTrapReport
|
||||
|
||||
with pytest.raises(ValueError, match="trajectories"):
|
||||
EchoTrapReport(
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=0,
|
||||
trajectories_seen=-1,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_details_must_be_tuple(self):
|
||||
from soup_cli.utils.echo_trap import EchoTrapReport
|
||||
|
||||
with pytest.raises(TypeError, match="tuple"):
|
||||
EchoTrapReport(
|
||||
signal=0.0,
|
||||
verdict="OK",
|
||||
step=0,
|
||||
trajectories_seen=0,
|
||||
details=["not tuple"], # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
class TestBuildEchoTrapCallbackDeferred:
|
||||
def test_invalid_threshold_rejected_first(self):
|
||||
from soup_cli.utils.echo_trap import build_echo_trap_callback
|
||||
|
||||
with pytest.raises(ValueError, match="threshold"):
|
||||
build_echo_trap_callback(threshold=2.0)
|
||||
|
||||
def test_invalid_threshold_bool(self):
|
||||
from soup_cli.utils.echo_trap import build_echo_trap_callback
|
||||
|
||||
with pytest.raises(ValueError, match="bool"):
|
||||
build_echo_trap_callback(threshold=True)
|
||||
|
||||
def test_deferred(self):
|
||||
from soup_cli.utils.echo_trap import build_echo_trap_callback
|
||||
|
||||
with pytest.raises(NotImplementedError, match="v0.70.1"):
|
||||
build_echo_trap_callback(threshold=0.5)
|
||||
|
||||
def test_halt_must_be_bool(self):
|
||||
from soup_cli.utils.echo_trap import build_echo_trap_callback
|
||||
|
||||
with pytest.raises(TypeError, match="halt"):
|
||||
build_echo_trap_callback(threshold=0.5, halt_on_trap="yes") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema integration — TrainingConfig + SoupConfig
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSchemaTrainingConfig:
|
||||
def test_defaults(self):
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig()
|
||||
assert tcfg.echo_trap_enabled is False
|
||||
assert tcfg.echo_trap_threshold == 0.6
|
||||
assert tcfg.echo_trap_halt is False
|
||||
|
||||
def test_threshold_bounds(self):
|
||||
from pydantic import ValidationError
|
||||
|
||||
from soup_cli.config.schema import TrainingConfig
|
||||
|
||||
tcfg = TrainingConfig(echo_trap_threshold=0.45)
|
||||
assert tcfg.echo_trap_threshold == 0.45
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(echo_trap_threshold=-0.1)
|
||||
with pytest.raises(ValidationError):
|
||||
TrainingConfig(echo_trap_threshold=1.5)
|
||||
|
||||
|
||||
class TestSchemaSoupConfigTaskGate:
|
||||
"""echo_trap_enabled only meaningful on RL agent tasks (grpo / ppo)."""
|
||||
|
||||
def _yaml(self, task: str = "grpo") -> str:
|
||||
return f"""
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: {task}
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
echo_trap_enabled: true
|
||||
echo_trap_threshold: 0.55
|
||||
"""
|
||||
|
||||
def test_grpo_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(self._yaml("grpo"))
|
||||
assert cfg.training.echo_trap_enabled is True
|
||||
|
||||
def test_ppo_accepted(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
cfg = load_config_from_string(self._yaml("ppo"))
|
||||
assert cfg.training.echo_trap_enabled is True
|
||||
|
||||
def test_sft_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="echo_trap"):
|
||||
load_config_from_string(self._yaml("sft"))
|
||||
|
||||
def test_halt_without_enabled_rejected(self):
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
with pytest.raises(ValueError, match="echo_trap_enabled"):
|
||||
load_config_from_string(
|
||||
"""
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: grpo
|
||||
data:
|
||||
train: ./data/train.jsonl
|
||||
format: chatml
|
||||
training:
|
||||
echo_trap_halt: true
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Source wiring guards
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_module_no_top_level_torch(self):
|
||||
from pathlib import Path
|
||||
|
||||
src = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "soup_cli"
|
||||
/ "utils"
|
||||
/ "echo_trap.py"
|
||||
)
|
||||
body = src.read_text(encoding="utf-8")
|
||||
assert "\nimport torch" not in body
|
||||
assert "\nfrom torch" not in body
|
||||
Loading…
Reference in New Issue