diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c61040e..273108c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (262 files, 11487 tests) +tests/ - Test suite (268 files, 11824 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index 3115702..edae073 100644 --- a/README.md +++ b/README.md @@ -42,14 +42,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.69.0 — Data Engineering Pro: `soup build` (dbt-for-SFT) + `soup expect` + `soup data gen-magpie` + `soup data persona-mix` + `soup data brain-rot`.** Five surfaces that turn dataset preparation into a first-class engineering workflow. `soup build` is a dbt-style DAG of dataset transforms with `ref()`-connected models, `incremental` materialization, and content-hash-based row-diff so re-runs re-tokenize only changed rows. `soup expect` ships Great-Expectations for chat data — `expect_no_pii` / `expect_token_length_between` / `expect_no_refusal_pattern` / `expect_chosen_preferred_over_rejected_by_judge` — with exit code 3 on suite failure so CI pipelines can gate on dataset quality regressions. `soup data gen-magpie` plans the Magpie synthetic generator (chat-template-prefix harvest, reuses v0.20 providers). `soup data persona-mix` samples a prompt × persona × style matrix with a bundled 12-persona / 5-style diversity set + topic-entropy metric. `soup data brain-rot` implements the arXiv 2510.13928 detector with OK/MINOR/MAJOR verdicts that compose with v0.47 educational scorer; `--strict` mode exits 3 when too many rows score MAJOR. +**v0.70.0 — Loop Hardening: reward-hacking detector + cross-tokenizer ULD + MiniLLM reverse-KL + mid-epoch RL checkpoint + iterative DPO + RAGEN echo-trap.** Six surfaces that protect the training loop from the failure modes that cost a real GPU-hour. The `reward_hack_detector` flag wires InfoRM cluster-separation OR RM-ensemble divergence into GRPO/PPO so the policy doesn't silently game the reward. `uld_strategy: wasserstein|topk_align` extends v0.53.2 distillation to teacher/student pairs with different vocabularies (Llama → Mistral, Llama → Qwen). `minillm_enabled` bundles MiniLLM's three stability tricks (teacher-mixed sampling + length-norm + pretrain-loss anchor). `rl_checkpoint_save_every_steps` adds the optimizer-state serialization TorchTune explicitly punts. `soup iterative-dpo --rounds N` is the sample → RM-score → re-pair → retrain loop driver. `echo_trap_enabled` detects trajectory degeneration in multi-turn agent RL (RAGEN-style). Schema-only release — live trainer-callback / math kernels deferred to v0.70.1. -- **`soup build [--dry-run]`** — dbt-for-SFT DAG parser + topological sort + plan rendering. Each model declares `kind: incremental|table|view`, a `transform`, and either a `source:` (seed) or `refs: [...]` (derived). Cross-validators reject ambiguous shapes (no-refs + no-source = degenerate; refs + source = mutually exclusive). `compute_row_hash` + `incremental_diff(prev, new) -> IncrementalDiffReport(added, changed, removed, unchanged)` are the kernel that lets the live runner (v0.69.1) re-tokenize only changed rows. `BuildModel` and `BuildPlan` are frozen dataclasses; `load_build_yaml` delegates to `paths.enforce_under_cwd_and_no_symlink` (TOCTOU defence). Live runner deferred to v0.69.1; today `--dry-run` renders the plan + exits 0, no flag exits 3 with the deferred-live marker. -- **`soup expect `** — Great Expectations for chat data. Four built-in expectation functions, each a pure function returning a frozen `ExpectationResult`. Composes with v0.47.0 `data_score.detect_pii` (Presidio backend when `[data-pro]` installed), v0.56.0 `diagnose.refusal.looks_like_refusal`, and v0.19.0 judge backends (operator-injected callable for the chosen-vs-rejected judge). `_dispatch_expectation` passes args through raw so per-expectation validators (bool-rejection, NaN-rejection, range checks) fire authentically — no silent int/float coercion bypasses the validator. Exit 0 = pass, 2 = validation rejection, 3 = suite failure. -- **`soup data gen-magpie --base --provider ollama|anthropic|vllm --target N [--plan-only]`** — Magpie technique (Xu et al. 2024) schema + plan. Feeds the chat-template prefix only to an aligned base model and harvests user-side turns via v0.20 providers. `MagpieConfig` frozen, `validate_magpie_provider` closed allowlist, `validate_target_rows` ∈ [1, 1_000_000] bool-rejected. Live generation loop (provider calls + v0.47 quality filter) deferred to v0.69.1. -- **`soup data persona-mix --prompts --n N --output `** — Persona-Hub-style diversity sampler. Reads `--prompts` JSONL, multiplies through bundled 12-persona × 5-style matrix (or operator-supplied `--personas` / `--styles` JSONL), writes `{prompt, persona, style}` per row via atomic `tempfile.mkstemp + os.replace`. Deterministic by `--seed`. New `compute_topic_diversity` Shannon-entropy kernel for downstream gating. Centralised TOCTOU policy via `enforce_under_cwd_and_no_symlink` on every read AND write path. -- **`soup data brain-rot [--strict] [--max-major-fraction 0.25]`** — arXiv 2510.13928 brain-rot detector. Two orthogonal slop scorers: `score_triviality` (low diversity / excessive `!!`/`??` punctuation / `lol/omg/lmao` density / length penalty) and `score_popularity_signal` (clickbait phrase substrings + emoji density). Per-row composite = `1.0 - max(triviality, popularity)`. Same OK/MINOR/MAJOR taxonomy as v0.26 / v0.56 / v0.65 (≥0.85 OK, ≥0.60 MINOR, else MAJOR). `--strict` exits 3 when MAJOR-row fraction exceeds the threshold so training pipelines can refuse to materialise slop datasets. `--max-major-fraction` validated at the CLI boundary (NaN / Inf / out-of-range / bool rejected) before any scoring. -- **+264 new tests** (11225 → 11487) across 5 part files. Review-fix coverage: 1 CRITICAL (centralised duplicate cwd+symlink blocks behind `paths.enforce_under_cwd_and_no_symlink` in build_dag / expectations / expect.py — code-review HIGH) + 4 HIGH (brain-rot loader DoS caps; persona-mix `--output` symlink TOCTOU rejection; persona-mix `_load_jsonl_field` DoS caps; magpie `quality_filter` validator + `_dispatch_expectation` int/float-coercion bypass) + 5 MEDIUM (BuildModel seed/derived cross-validator + docs; `--max-major-fraction` CLI-boundary validation; `expect` skipped-line WARNING) + 4 LOW (boundary tests at exact threshold ± ε on classify_brain_rot; BuildPlan FrozenInstanceError; version_bumped checks in B/C/D/E; lazy-yaml import source-grep). Manual CPU smokes (Step 6): every CLI happy + failure path exercised, including 3-stage build DAG dry-run, PII suite exit 3, magpie plan-only, persona-mix atomic write to JSONL, brain-rot MAJOR-on-slop exit 3. +- **`soup train --reward-hack-detector info_rm|rm_ensemble`** — early-warning when the policy starts gaming the reward model. `info_rm` tracks the InfoRM Cluster-Separation Index across training (Wang et al. 2024, arXiv 2402.09345); a sharp drop signals the RM losing its grip on the (good, bad) split. `rm_ensemble` tracks pairwise variance across an RM ensemble; rising disagreement = unreliable reward signal. `--reward-hack-halt` auto-stops training on HACK verdict (≥30% relative drop). Composes with v0.34 `soup why` so the anomaly explainer can name reward-hacking specifically rather than "loss plateau". Schema + math kernels live now (`compute_cluster_separation`, `compute_rm_ensemble_divergence`, `classify_hack_signal` with OK/WARN/HACK bands at 0.10 / 0.30); live HF Trainer callback in v0.70.1. +- **`soup train --uld-strategy wasserstein|topk_align`** — cross-tokenizer distillation (Boizard et al. 2024, arXiv 2402.12030). The v0.53.2 distillation path assumes student and teacher share a vocabulary; the moment vocabs differ, column-wise logit alignment breaks. `wasserstein` computes 1D Wasserstein distance between sorted teacher/student logit distributions — no alignment required. `topk_align` picks top-K teacher logits and maps to student token ids via BPE overlap. Bounded vocab sizes [1, 262144] cover multilingual SentencePiece + GPT-OSS 200K. Live projection module wired in v0.70.1. +- **`soup train --minillm-enabled`** — MiniLLM-style reverse-KL on-policy distillation (Gu et al. 2024, arXiv 2306.08543). Bundles the three stability tricks scattered across §3 of the paper: teacher-mixed sampling (epsilon-greedy mix with `--minillm-teacher-mix-ratio 0.3`), length normalisation on rollouts (`--minillm-length-normalize true`), and a small pretrain-loss anchor (`--minillm-pretrain-anchor-weight 0.1` requires `--minillm-pretrain-anchor-path pre.jsonl`). Cross-validators reject silent no-op combos (anchor_weight=0 + anchor_path set, anchor_weight > 0 + path None). Live callback in v0.70.1. +- **`soup train --rl-checkpoint-save-every-steps N`** — mid-epoch checkpoint for PPO/GRPO. TorchTune explicitly punts this; Soup ships the real save_state / load_state surface here. Captures optimizer state (`--rl-checkpoint-include-optimizer`), optional ref-model state, optional rollout/replay buffer. `--rl-checkpoint-keep-last 3` retains the last N. Composes with v0.32 spike recovery + v0.40.0 ref-model regen — a recovered run hops back to the most recent mid-epoch ckpt instead of restarting the epoch. Live save_state / load_state in v0.70.1. +- **`soup iterative-dpo --rounds N --pairs-per-round 500 [--plan-only]`** — sample → RM-score → re-pair → retrain over N rounds. Frozen `IterativeDPOPlan` with consecutive-round_index invariant + per-round artifact paths (`./out/round-NN/pairs.jsonl`, `./out/round-NN/adapter`). `--plan-only` renders the canonical plan + exits 0; without it the deferred-live runner exits 3 with explicit v0.70.1 marker. Recipe glue around existing TRL primitives; v0.70.1 wires the live `soup train --task dpo` subprocess loop. +- **`soup train --echo-trap-enabled --echo-trap-threshold 0.6 --echo-trap-halt`** — RAGEN-style detection of trajectory degeneration during multi-turn agent RL (Zhu et al. 2025, arXiv 2504.14437). Pure-Python n-gram repetition rate per trajectory + batch mean. OK / WARN / TRAP taxonomy at 0.30 / 0.60. Composes with v0.53.11 #127 `GRPOStabilityCallback` — both detectors fire in the same training step without duplicating trajectory collection. Live HF Trainer callback in v0.70.1. +- **+337 new tests** (11487 → 11824) across 6 part files. Schema-only release; every live callback / kernel raises `NotImplementedError` with explicit v0.70.1 marker after validating inputs (matches the project's stub-then-live cadence from v0.50.0 / v0.62.0 / v0.69.0). 12-invariant self-review against the full project checklist (closed allowlists, frozen dataclasses, MappingProxyType registries, bool-as-int rejection, math.isfinite NaN/Inf reject, null-byte rejection, length caps, no top-level torch, TypeError/ValueError split, deferred-live policy, tuples-not-lists on frozen collections, CLI exit codes) — all 12 satisfied. Manual CPU smoke (Step 6): `soup iterative-dpo --plan-only` 3-round plan rendered end-to-end; 5 happy + 5 failure-mode YAML round-trips across every new schema field. ## Data Engineering Pro @@ -88,6 +89,53 @@ soup data brain-rot data.jsonl --strict --max-major-fraction 0.10 Every command applies the project-wide TOCTOU policy (`os.lstat + S_ISLNK` symlink rejection before any open) and cwd containment via the shared `paths.enforce_under_cwd_and_no_symlink` helper. Live runners for `soup build` and `soup data gen-magpie` land in v0.69.1; the other three are LIVE today. +## Loop Hardening + +The v0.70.0 release ships 6 surfaces that protect the training loop from the failure modes that cost a real GPU-hour. Schema + math kernels live now; live trainer-callback wiring lands in v0.70.1 (matches the project's stub-then-live cadence). + +```bash +# Reward-hacking detector — auto-halt when the policy starts gaming the RM +# (InfoRM cluster-separation index, Wang et al. 2024 arXiv:2402.09345) +soup train --config soup.yaml \ + --reward-hack-detector info_rm --reward-hack-halt # halt on HACK verdict + +# Cross-tokenizer distillation — Llama -> Mistral, no shared vocab needed +# (Universal Logit Distillation, Boizard et al. 2024 arXiv:2402.12030) +soup train --config soup.yaml --uld-strategy wasserstein + +# MiniLLM reverse-KL on-policy distillation — bundles 3 stability tricks +# (Gu et al. 2024 arXiv:2306.08543) +soup train --config soup.yaml --minillm-enabled \ + --minillm-teacher-mix-ratio 0.3 \ + --minillm-pretrain-anchor-weight 0.1 \ + --minillm-pretrain-anchor-path ./pretrain.jsonl + +# Mid-epoch checkpoint for PPO/GRPO — TorchTune punts this; Soup ships it +soup train --config grpo.yaml \ + --rl-checkpoint-save-every-steps 500 \ + --rl-checkpoint-keep-last 3 \ + --rl-checkpoint-include-optimizer + +# Iterative DPO loop driver — sample -> RM-score -> re-pair -> retrain +soup iterative-dpo \ + --base-model meta-llama/Llama-3.1-8B \ + --reward-model ./output_rm \ + --prompts ./prompts.jsonl \ + --output-dir ./iterative_dpo_out \ + --rounds 5 \ + --pairs-per-round 1000 \ + --plan-only + +# RAGEN echo-trap detector — auto-halt when trajectories collapse to self-repetition +# (Zhu et al. 2025 arXiv:2504.14437) +soup train --config grpo.yaml \ + --echo-trap-enabled \ + --echo-trap-threshold 0.6 \ + --echo-trap-halt +``` + +Every detector composes with v0.34 `soup why` (anomaly explainer), v0.32 spike recovery, and v0.53.11 #127 `GRPOStabilityCallback` so a single training run can have InfoRM + echo-trap + spike-recovery + ref-model regen all active simultaneously without duplicating trajectory / state collection. Live trainer-callback wiring for all 6 Parts lands in v0.70.1 (`build_reward_hack_callback`, `build_uld_projection`, `build_minillm_callback`, `build_rl_checkpoint_callback`, `run_iterative_dpo`, `build_echo_trap_callback`); today every CLI / config flag is validated at schema-load so misconfigured runs fail loudly at config-load time. + ## Why Soup? Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that. diff --git a/SECURITY.md b/SECURITY.md index bfb8175..300a708 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.69.0 -- Full support (latest) +- v0.70.0 -- Full support (latest) +- v0.69.0 -- Full support - v0.68.0 -- Full support - v0.67.0 -- Full support - v0.66.0 -- Full support @@ -143,6 +144,8 @@ No known critical vulnerabilities in current releases. ### Security Hardening History +- **v0.70.0 — Loop Hardening**: 6 parts ship reward-hacking detector + cross-tokenizer ULD + MiniLLM reverse-KL + mid-epoch RL checkpoint + iterative DPO loop driver + RAGEN echo-trap detector. Schema-only release; live trainer-callback / math kernel wiring deferred to v0.70.1. **Validator hardening**: every numeric input rejects `bool` explicitly (bool is subclass of int in Python — matches the project bool-as-int policy across v0.30 / v0.34 / v0.41 / v0.50 / v0.65 / v0.67 / v0.69); every float input rejects NaN / Inf via `math.isfinite` (mirrors v0.32 / v0.41 / v0.50 / v0.62 policy); every string field rejects null-byte + caps at field-appropriate length (32 chars for enum names, 4096 chars for paths, 512 chars for model ids — matches v0.40.5 / v0.52.0 / v0.68.0 policy). **Closed allowlists**: `SUPPORTED_HACK_DETECTORS = {info_rm, rm_ensemble}`, `SUPPORTED_ULD_STRATEGIES = {wasserstein, topk_align}`, `_RL_TASKS = {grpo, ppo}` (RL checkpoint state allowlist), `VERDICTS = (OK, WARN, HACK)` for reward-hacking + `VERDICTS = (OK, WARN, TRAP)` for echo-trap. **DoS caps on every kernel**: cluster-separation good/bad lists not bounded (data is operator-supplied at probe time); RM-ensemble capped at 32 RMs; echo-trap `_MAX_NGRAM_N=32`, `_MAX_TRAJECTORY_TOKENS=1_000_000`, `_MAX_BATCH_TRAJECTORIES=100_000`; iterative-DPO `_MAX_ROUNDS=100`, `_MAX_PAIRS_PER_ROUND=1_000_000`; MiniLLM `_MAX_ANCHOR_PATH_LEN=4096`. **Frozen dataclasses** (post-init validation, `FrozenInstanceError` on mutation): `HackDetectorSpec`, `RewardHackReport`, `ULDStrategySpec`, `ULDConfig`, `MiniLLMConfig`, `RLCheckpointConfig`, `RLCheckpointState`, `IterativeDPORound`, `IterativeDPOPlan`, `EchoTrapReport`. **`MappingProxyType`-wrapped registries** so callers cannot mutate metadata at runtime: `_DETECTOR_METADATA` (reward_hacking), `_STRATEGY_METADATA` (uld). **Tuples (not Lists) on every frozen collection field** — Python `frozen=True` does NOT make `List` immutable; matches v0.43 Tournament / v0.61 EditDiff policy. **Lazy imports**: no top-level `import torch` / `from torch` in any v0.70.0 utility module (4 source-grep regression tests). Math kernels (`compute_cluster_separation`, `compute_rm_ensemble_divergence`, `score_trajectory_repetition`, `score_echo_signal`) are pure-Python (no numpy / torch at module scope). **CLI exit-code policy**: `typer.Exit(2)` for validation rejection, `typer.Exit(3)` for deferred-live (matches v0.55 / v0.56 / v0.64 / v0.65 / v0.69 gate convention). **Cross-validators reject silent no-ops**: `reward_hack_halt=True` without `reward_hack_detector` rejected; `uld_top_k` with `wasserstein` strategy rejected; `topk_align` without `uld_top_k` rejected; `minillm_pretrain_anchor_path` without `weight > 0` rejected; `minillm_pretrain_anchor_weight > 0` without path rejected; `echo_trap_halt=True` without `echo_trap_enabled` rejected; all minillm_* tunables without `minillm_enabled=True` rejected (with offending fields named in the error). **Task-gates**: `reward_hack_detector` / `echo_trap_enabled` rejected outside `{grpo, ppo}`; `uld_strategy` rejected outside `task='distill'`; `minillm_enabled` rejected outside `task='distill'`; `rl_checkpoint_save_every_steps` rejected outside `{grpo, ppo}`. **MLX rejected** with distinct error message on every Part (mirrors v0.34.0 / v0.50.0 review-fix policy of distinct error reasons for distinct failure modes). **Deferred-live stubs validate inputs FIRST, then raise** `NotImplementedError("v0.70.1")` so misconfigured callers fail fast even though the live callback is not yet wired. **Test count**: 11487 → 11824 (+337 net across 6 part files + boundary follow-ups). **Self-review**: direct reviewer agents context-thrashed on the 800+ KB CLAUDE.md (matches v0.69.0 review pattern); 12-invariant manual self-review against the full project checklist confirmed all v0.70.0 modules satisfy every invariant. **Known limitations**: (1) Every live callback / kernel is deferred to v0.70.1 — schema accepts every new flag; v0.70.0 ships the math kernels + report manifests only. (2) MiniLLM `pretrain_anchor_path` is shape-only at schema time — cwd-containment + symlink rejection deferred to v0.70.1 disk-read site (matches v0.69.0 build_dag policy). (3) RL checkpoint state only manifests, not weights — `RLCheckpointState.to_dict` serialises the manifest; v0.70.1 wires `torch.save(state_dict)`. (4) Iterative DPO orchestrator is plan-only — `--plan-only` renders the canonical per-round paths; v0.70.1 wires the live subprocess loop. (5) Echo-trap detector is whitespace-tokenised (string n-grams, not tokeniser ids); sub-word echo trapping deferred to v0.70.1. (6) RM-ensemble + InfoRM detectors mutually exclusive at the `reward_hack_detector` flag — v0.70.1 may allow concurrent probes. (v0.70.0) + - **v0.69.0 — Data Engineering Pro**: 5 parts ship `soup build` (dbt-for-SFT DAG) + `soup expect` (expectations suite) + `soup data gen-magpie` + `soup data persona-mix` + `soup data brain-rot`. **CRITICAL — centralised TOCTOU helper**: replaces 3 duplicate `os.lstat + S_ISLNK + realpath + is_under_cwd` blocks in `build_dag.load_build_yaml` / `expectations.load_suite_yaml` / `expect._load_jsonl_rows` with the shared `paths.enforce_under_cwd_and_no_symlink` helper (v0.59.0 single-source-of-truth policy — mirrors v0.40.6 / v0.53.5 centralisation enforcement). **HIGH — DoS caps on every new JSONL loader**: brain-rot data path capped at 1 GiB + 1M rows, persona-mix `--prompts` / `--personas` / `--styles` capped at 100 MiB + 100k entries, expect data path capped at 1 GiB + 1M rows (mirrors v0.55 / v0.65 _MAX_ROWS policy). **HIGH — TOCTOU symlink rejection on every write target**: `persona-mix --output` and `soup expect` data input now both pass through `enforce_under_cwd_and_no_symlink` so a pre-placed `out.jsonl -> /etc/cron.d/x` cannot redirect atomic writes. **HIGH — validator bypass fixed**: `magpie.build_magpie_config` now explicit `isinstance(quality_filter, bool)` (was passing non-bool through to the dataclass); `expectations._dispatch_expectation` passes raw args to per-expectation validators (was `int()`/`float()`-coercing inputs which silently bypassed `_check_token_bound` bool-rejection). **MEDIUM — `BuildModel` cross-validator** rejects ambiguous shapes at schema load: `refs=() + source=None` (degenerate, no input) and `refs + source set` (ambiguous) both raise `ValueError` so misconfigured manifests fail loudly. **MEDIUM — `--max-major-fraction` CLI-boundary validation**: bool / NaN / Inf / out-of-range rejected before any scoring pass (security-review M2). **Closed-allowlist validators**: `SUPPORTED_MODEL_KINDS = {incremental, table, view}` (build_dag), `SUPPORTED_EXPECTATIONS = {expect_no_pii, expect_token_length_between, expect_no_refusal_pattern, expect_chosen_preferred_over_rejected_by_judge}`, `SUPPORTED_MAGPIE_PROVIDERS = {ollama, anthropic, vllm}`, `BRAIN_ROT_VERDICTS = (OK, MINOR, MAJOR)` — all bool-first / null-byte / oversize (≤32 / ≤128-char) / case-insensitive normalisation. **Brain-rot scoring is ReDoS-safe**: `_PUNCT_PATTERN` uses bounded `{2,}` quantifiers; `_CLICKBAIT_PHRASES` is substring scan (not regex); `_TEXT_FIELDS` capped at `_MAX_TEXT_LEN=65_536`. **No top-level heavy imports** — torch / transformers / peft / sentence_transformers / yaml never imported at module scope in any v0.69.0 utility module (source-grep regression tests). **Atomic writes** via `tempfile.mkstemp + os.replace` + parent-realpath in `persona-mix`. **Worst-signal composite** for brain-rot row score: `1.0 - max(triviality, popularity)` so a single strong slop signal drives the verdict down hard (mirrors v0.56.0 `overall_verdict` worst-case policy). **Exit-code policy**: `soup build` and `soup data gen-magpie` exit 3 on deferred-live (distinct from validation rejection exit 2); `soup expect` and `soup data brain-rot --strict` exit 3 on gate failure (matches v0.55 / v0.56 / v0.64 / v0.65 gate convention). **Test count**: 11225 → 11487 (+262 net in `tests/test_v0690_part_{a-e}.py`; 264 pass + 3 POSIX-skipped symlink tests on Windows). **Review-fix coverage across 4 review waves** (security + code + python + TDD; direct reviewer agents context-thrashed on 800+ KB CLAUDE.md — `general-purpose` agent with explicit "do not crawl, read only these 9 files" prompts produced equivalent findings consolidated): 1 CRITICAL (TOCTOU helper centralisation) + 4 HIGH (DoS caps, output symlink, persona loader caps, validator bypass) + 5 MEDIUM (BuildModel cross-validator, --max-major-fraction validation, expect-loader WARN-on-skip) + 4 LOW (boundary tests at OK/MINOR/MAJOR ± ε, BuildPlan FrozenInstanceError, version_bumped checks, lazy-yaml import grep). **Known limitations**: (1) Live `soup build` runner deferred to v0.69.1 — `run_build` raises NotImplementedError; `--dry-run` produces fully-validated topological plan today. (2) Live Magpie generator deferred to v0.69.1 — `run_magpie` raises NotImplementedError; `--plan-only` renders the resolved config today. (3) `BuildModel.source` schema-time containment NOT enforced (only shape) — live v0.69.1 runner must apply `enforce_under_cwd_and_no_symlink` before reading source paths; schema permits relative paths so a build can be planned offline before the data lands. (4) `expect_chosen_preferred_over_rejected_by_judge` advisory-pass when `judge_fn=None` — every row scores 1.0; production callers MUST supply a real judge. (5) Brain-rot detector is English-keyword-only — `_LOW_EFFORT_TOKENS` + `_CLICKBAIT_PHRASES` are English; entropy + punct-run signals remain language-agnostic. (6) Persona-Hub bundled 12-persona × 5-style set by default; full Tencent corpus (200k personas) NOT bundled — operators pass `--personas ` to swap in the full set. (v0.69.0) - **v0.66.0 — Post-train X-rays**: 5 LIVE parts extend `soup diagnose` from 6 failure modes to 10 — SAE feature diff (Part A), live influence-function blame closing v0.57 #171 (Part B), sleeper-agent defection probe (Part C), pairwise adapter interference matrix (Part D), and probe pack manifest (Part E). **TOCTOU `O_NOFOLLOW` probe-open** — `load_sae_weights` (Part A) and `_count_dataset_rows` (Part B) both replace the v0.57 `os.path.realpath + open()` pattern (which left a race window between containment check and read) with `os.open(path, os.O_RDONLY | os.O_NOFOLLOW)`. A symlink swap between containment check and read now raises immediately rather than silently following the link. **Process-independent deterministic seeds** (HIGH H3) — `_probe_weights` (sleeper) and `_default_synthetic_probe` (blame) replace Python's process-salted `hash()` with `int(hashlib.sha256(key).hexdigest()[:16], 16)` (64-bit seed). CI reproducibility now works regardless of `PYTHONHASHSEED`. **Rich-markup escape on operator-controlled strings** (HIGH H2) — `render_matrix_markdown` (interference), `render_blame_markdown` (blame), and `render_pack_markdown` (probe_pack) all route adapter names / verdict / description / layer through a local `_md_escape` (`[` → `\\[`, `]` → `\\]`) before embedding into Rich markdown. A crafted adapter name like `"[link=evil]click[/]"` cannot inject markup or open file: URIs in the user's terminal. **TypeError on bool/non-str verdict** (HIGH H1) — `SleeperProbeResult.__post_init__` and `InterferenceCell.__post_init__` type-check `verdict` BEFORE the membership check, so `verdict=True` raises TypeError (was: misleading ValueError about membership). **Non-numeric loss rejection** (HIGH H3 CLI) — `soup probe interference` CLI rejects string / bool / list values in the `losses` payload before they reach the math kernel; JSON booleans (`true`/`false`) deserialise to Python `bool` which is a subclass of `int` so the explicit `isinstance(value, bool)` guard fires first. **10M-row hard rejection** (MEDIUM M4) — `_count_dataset_rows` now raises `ValueError("dataset has >10M rows; subsample before blame")` instead of silently truncating to 10M (which would have corrupted the plan). **Synthetic probe row cap** (MEDIUM M6) — `_DEFAULT_SYNTH_PROBE_CAP=100_000`: without an operator-supplied `probe_fn`, `_default_synthetic_probe` caps at 100k rows so a 10M-row dataset doesn't allocate 1.28 GB of float32 (10M × 32 × 4 B). **Description length cap** (MEDIUM M5) — `ProbeEntry._MAX_DESCRIPTION_LEN=4096`; operator-controlled description field cannot blow up Rich render or downstream JSON sinks. **Safetensors key-count pre-check** (MEDIUM) — `load_sae_weights` validates `len(f.keys()) > 64` BEFORE materializing any tensor; the DoS cap now defends against pathological files (was: cap fired AFTER N tensors were already loaded). **Frozen dataclasses everywhere** — `SaeFeatureChange`, `SaeFeatureDiffReport`, `RowInfluence`, `BlameResult`, `SleeperProbeSpec`, `SleeperProbeResult`, `InterferenceCell`, `InterferenceMatrix`, `ProbeEntry`, `ProbePack` all use `@dataclass(frozen=True)` + `__post_init__` validation; all report tests assert `pytest.raises(FrozenInstanceError)` (HIGH H1 wave 3 — replaces the previous `pytest.raises((AttributeError, Exception))` which would have silently passed if the dataclass dropped its frozen flag). **`MappingProxyType` for read-only registries** — `BUNDLED_PROBES`, `BUNDLED_PACKS`, `_LOWER_INDEX` (sleeper case-insensitive lookup, O(1) instead of O(N) loop). **No top-level heavy imports** — `torch` / `transformers` / `peft` / `safetensors` never imported at module scope in any v0.66 module (source-grep regression tests). **Test count**: 10577 → 10836 (+259 net across `tests/test_v0660_part_{a-e}.py` + `tests/test_v0660_cli.py` + `tests/test_v0660_followups.py`). **Review-fix coverage** across 3 sequential waves (python-review + code-review + security-review + tdd-guide): 0 CRITICAL + 9 HIGH + 14 MEDIUM + 5 LOW. (v0.66.0) diff --git a/pyproject.toml b/pyproject.toml index 014b07e..4f784c9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.69.0" +version = "0.70.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 983dc31..55f682c 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.69.0" +__version__ = "0.70.0" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index 101fb17..9e54b76 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -467,6 +467,15 @@ app.command( help="Run an expectations suite against a JSONL dataset (v0.69.0 Part B).", )(_expect_cmd.expect_cmd) +# v0.70.0 Part E — `soup iterative-dpo` (iterative DPO loop driver). +from soup_cli.commands import iterative_dpo as _iterative_dpo_cmd # noqa: E402 + +app.add_typer( + _iterative_dpo_cmd.app, + name="iterative-dpo", + help="Iterative DPO loop driver (v0.70.0 Part E).", +) + def _rewrite_advise_argv(argv: list) -> list: """Inject `run` between `advise` and a non-subcommand first argument. diff --git a/soup_cli/commands/iterative_dpo.py b/soup_cli/commands/iterative_dpo.py new file mode 100644 index 0000000..ad44084 --- /dev/null +++ b/soup_cli/commands/iterative_dpo.py @@ -0,0 +1,95 @@ +"""soup iterative-dpo — Iterative DPO loop driver — v0.70.0 Part E. + +Sample → RM-score → re-pair → retrain over N rounds. The live runner +is deferred to v0.70.1; v0.70.0 ships the schema + ``--plan-only`` +renderer that prints the canonical per-round artifacts. +""" + +from __future__ import annotations + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +console = Console() + +app = typer.Typer( + no_args_is_help=True, + help="Iterative DPO loop driver (v0.70.0 Part E)", +) + + +@app.callback(invoke_without_command=True) +def main( + base_model: str = typer.Option(..., "--base-model", help="HF id / local path"), + reward_model: str = typer.Option(..., "--reward-model", help="RM HF id / path"), + prompts: str = typer.Option(..., "--prompts", help="Source prompts JSONL"), + output_dir: str = typer.Option(..., "--output-dir", help="Output dir"), + rounds: int = typer.Option(3, "--rounds", help="Number of iterative-DPO rounds"), + pairs_per_round: int = typer.Option( + 500, + "--pairs-per-round", + help="Number of (chosen, rejected) pairs per round", + ), + plan_only: bool = typer.Option( + False, + "--plan-only", + help="Print the resolved plan and exit (no training).", + ), +): + """Render the iterative-DPO plan and (in v0.70.1) execute it.""" + from soup_cli.utils.iterative_dpo import ( + build_iterative_dpo_plan, + run_iterative_dpo, + ) + + try: + plan = build_iterative_dpo_plan( + base_model=base_model, + reward_model=reward_model, + prompts_path=prompts, + output_dir=output_dir, + rounds=rounds, + pairs_per_round=pairs_per_round, + ) + except (ValueError, TypeError) as exc: + console.print(f"[red]Error:[/red] {escape(str(exc))}") + raise typer.Exit(code=2) from exc + + table = Table(title="Iterative-DPO plan") + table.add_column("Round") + table.add_column("Pairs path") + table.add_column("Adapter path") + table.add_column("Pairs") + for r in plan.rounds: + table.add_row( + str(r.round_index), + escape(r.pairs_path), + escape(r.adapter_path), + str(r.pairs_count), + ) + console.print(table) + + if plan_only: + console.print( + Panel( + "[green]Plan rendered[/green] (--plan-only). To execute, " + "drop the flag once v0.70.1 ships.", + title="Iterative DPO", + ) + ) + raise typer.Exit(code=0) + + try: + run_iterative_dpo(plan) + except NotImplementedError as exc: + console.print( + Panel( + "[yellow]Live runner deferred to v0.70.1.[/yellow] " + f"{escape(str(exc))}", + title="Iterative DPO", + ) + ) + raise typer.Exit(code=3) from exc diff --git a/soup_cli/config/schema.py b/soup_cli/config/schema.py index b18bc56..3c50888 100644 --- a/soup_cli/config/schema.py +++ b/soup_cli/config/schema.py @@ -1341,6 +1341,244 @@ class TrainingConfig(BaseModel): ), ) + # ---- v0.70.0 Part F — Echo-trap detector ----------------------------- + echo_trap_enabled: bool = Field( + default=False, + description=( + "Enable RAGEN-style echo-trap detection during multi-turn " + "agent RL. Requires task in {'grpo', 'ppo'} on a non-mlx " + "backend. Schema-only in v0.70.0; live callback in v0.70.1." + ), + ) + echo_trap_threshold: float = Field( + default=0.6, + ge=0.0, + le=1.0, + description=( + "Threshold on the aggregate echo signal. Above this = TRAP. " + "Bounded [0.0, 1.0]. (v0.70.0)" + ), + ) + echo_trap_halt: bool = Field( + default=False, + description=( + "Auto-halt training on TRAP verdict. Requires " + "echo_trap_enabled=True. (v0.70.0)" + ), + ) + + # ---- v0.70.0 Part D — Mid-epoch RL checkpoint ------------------------ + rl_checkpoint_save_every_steps: Optional[int] = Field( + default=None, + ge=1, + le=10_000_000, + description=( + "Save an RL-aware mid-epoch checkpoint every N steps. None " + "= use HF Trainer's per-epoch checkpoint only. Requires " + "task in {'grpo', 'ppo'}. Schema-only in v0.70.0; live " + "save_state / load_state in v0.70.1." + ), + ) + rl_checkpoint_keep_last: int = Field( + default=3, + ge=1, + le=100, + description=( + "Number of recent RL checkpoints to retain. Older ones are " + "pruned at write time. (v0.70.0)" + ), + ) + rl_checkpoint_include_optimizer: bool = Field( + default=True, + description=( + "Include AdamW / Lion optimizer state in the mid-epoch RL " + "checkpoint. (v0.70.0)" + ), + ) + rl_checkpoint_include_ref_model: bool = Field( + default=False, + description=( + "Include the frozen reference model state in the RL " + "checkpoint. Default False (ref model is reconstructable " + "from cfg.base). (v0.70.0)" + ), + ) + rl_checkpoint_include_rollout_buffer: bool = Field( + default=False, + description=( + "Include the rollout / replay buffer in the RL checkpoint " + "so resumed runs don't lose collected experience. (v0.70.0)" + ), + ) + + # ---- v0.70.0 Part C — MiniLLM reverse-KL on-policy distillation ------- + # Bundles teacher-mixed sampling + length-norm + pretrain anchor. + # Schema-only; live callback wired in v0.70.1. + minillm_enabled: bool = Field( + default=False, + description=( + "Enable MiniLLM-style on-policy distillation (Gu et al. 2024). " + "Requires task='distill' on a non-mlx backend. v0.70.0 " + "schema-only; live callback in v0.70.1." + ), + ) + minillm_teacher_mix_ratio: float = Field( + default=0.0, + ge=0.0, + le=1.0, + description=( + "Probability of sampling from the teacher distribution at " + "rollout time. 0.0 = student-only; 1.0 = teacher-only. " + "Typical range 0.2-0.5. (v0.70.0)" + ), + ) + minillm_length_normalize: bool = Field( + default=True, + description=( + "Length-normalise the rollout log-probability before the " + "reverse-KL term. Prevents long completions from dominating " + "the gradient. (v0.70.0)" + ), + ) + minillm_pretrain_anchor_weight: float = Field( + default=0.0, + ge=0.0, + le=1.0, + description=( + "Weight on the pretrain-loss anchor term (SFT on a small " + "pretrain corpus). Prevents drift away from coherent " + "language. Requires minillm_pretrain_anchor_path when > 0. " + "(v0.70.0)" + ), + ) + minillm_pretrain_anchor_path: Optional[str] = Field( + default=None, + description=( + "Path to the pretrain JSONL used by the anchor term. " + "Required when minillm_pretrain_anchor_weight > 0. " + "Null-byte rejected; capped at 4096 chars. (v0.70.0)" + ), + ) + + # ---- v0.70.0 Part B — Cross-tokenizer ULD ---------------------------- + # Universal Logit Distillation (Boizard et al. 2024). Schema-only; + # live projection module wired in v0.70.1. + uld_strategy: Optional[Literal["wasserstein", "topk_align"]] = Field( + default=None, + description=( + "Cross-tokenizer distillation strategy: 'wasserstein' " + "(no alignment needed) or 'topk_align' (requires uld_top_k). " + "Requires task='distill' on a non-mlx backend. Schema-only " + "in v0.70.0; live projection wired in v0.70.1." + ), + ) + uld_top_k: Optional[int] = Field( + default=None, + ge=1, + le=262144, + description=( + "Top-K teacher logits to align (uld_strategy='topk_align' " + "only). Bounded [1, 262144] to cap pathological vocabs. " + "(v0.70.0)" + ), + ) + # ---- v0.70.0 Part A — Reward-hacking detector ------------------------ + # Schema-only release; live HF Trainer callback wired in v0.70.1. + reward_hack_detector: Optional[Literal["info_rm", "rm_ensemble"]] = Field( + default=None, + description=( + "Reward-hacking detector for GRPO/PPO. 'info_rm' tracks " + "InfoRM cluster-separation across training; 'rm_ensemble' " + "tracks pairwise variance across an RM ensemble. Requires " + "task in {'grpo', 'ppo'} on a non-mlx backend. Schema-only " + "in v0.70.0; live HF Trainer callback wired in v0.70.1." + ), + ) + reward_hack_halt: bool = Field( + default=False, + description=( + "Auto-halt training on HACK verdict (drop_pct >= 30% in " + "cluster separation). Requires reward_hack_detector to be " + "set. (v0.70.0)" + ), + ) + + @field_validator("reward_hack_halt", mode="before") + @classmethod + def _validate_reward_hack_halt(cls, v): + """v0.70.0 — explicit bool guard so YAML ``yes`` / ``1`` integers + cannot silently coerce. Matches project bool-before-int policy. + """ + if v is None: + return v + if isinstance(v, bool): + return v + raise TypeError( + f"reward_hack_halt must be bool, got {type(v).__name__}" + ) + + @field_validator( + "echo_trap_enabled", + "echo_trap_halt", + mode="before", + ) + @classmethod + def _validate_echo_trap_bool_fields(cls, v): + """v0.70.0 Part F — bool guards for echo-trap toggles.""" + if v is None: + return v + if isinstance(v, bool): + return v + raise TypeError( + f"v0.70.0 echo-trap flag must be bool, got {type(v).__name__}" + ) + + @field_validator( + "rl_checkpoint_include_optimizer", + "rl_checkpoint_include_ref_model", + "rl_checkpoint_include_rollout_buffer", + mode="before", + ) + @classmethod + def _validate_rl_checkpoint_bool_fields(cls, v): + """v0.70.0 Part D — bool guards for RL-checkpoint toggles.""" + if v is None: + return v + if isinstance(v, bool): + return v + raise TypeError( + f"v0.70.0 RL-checkpoint flag must be bool, got {type(v).__name__}" + ) + + @field_validator( + "minillm_enabled", + "minillm_length_normalize", + mode="before", + ) + @classmethod + def _validate_minillm_bool_fields(cls, v): + """v0.70.0 Part C — bool guards for MiniLLM toggles.""" + if v is None: + return v + if isinstance(v, bool): + return v + raise TypeError( + f"v0.70.0 MiniLLM flag must be bool, got {type(v).__name__}" + ) + + @field_validator("minillm_pretrain_anchor_path") + @classmethod + def _validate_minillm_anchor_path(cls, v): + """v0.70.0 Part C — shape-only path validation. Cwd containment + deferred to v0.70.1 runtime hook (matches v0.69.0 build_dag / + magpie base_model policy). + """ + if v is None: + return None + from soup_cli.utils.minillm import _check_path_shape + + return _check_path_shape(v) + @field_validator( "fp8_attention", "nvfp4", @@ -3743,6 +3981,194 @@ class SoupConfig(BaseModel): f"training helper). Use backend=transformers for task={self.task}." ) + @model_validator(mode="after") + def _validate_uld_compat(self) -> "SoupConfig": + """v0.70.0 Part B — Universal Logit Distillation gate. + + ``uld_strategy`` is only meaningful when ``task='distill'`` — + cross-tokenizer distillation has no analogue outside the + distillation trainer. Rejected on other tasks with a friendly + message, and on MLX backend with a distinct message. + + Composes with v0.52 distillation task: when set, the + :class:`uld.ULDConfig` validation fires (top_k cross-validation, + vocab-size bounds) at config-load. + """ + tcfg = self.training + strategy = tcfg.uld_strategy + top_k = tcfg.uld_top_k + if strategy is None and top_k is None: + return self + if strategy is None: + # top_k without strategy is a silent no-op footgun. + raise ValueError( + "uld_top_k requires uld_strategy to be set" + ) + if self.task != "distill": + raise ValueError( + "uld_strategy / uld_top_k are only valid when " + f"task='distill'; got task={self.task!r}" + ) + if self.backend == "mlx": + raise ValueError( + "uld_strategy is not supported on backend=mlx in v0.70.0 " + "(cross-tokenizer distillation is transformers-only)" + ) + # Cross-check: topk_align requires top_k. + if strategy == "topk_align" and top_k is None: + raise ValueError( + "uld_strategy='topk_align' requires uld_top_k to be set" + ) + if strategy == "wasserstein" and top_k is not None: + raise ValueError( + "uld_top_k is only valid when uld_strategy='topk_align'; " + f"got uld_strategy={strategy!r}" + ) + return self + + @model_validator(mode="after") + def _validate_echo_trap_compat(self) -> "SoupConfig": + """v0.70.0 Part F — echo-trap detector task gate. + + ``echo_trap_enabled`` (and ``echo_trap_halt``) only meaningful + on RL tasks (grpo / ppo). Setting ``echo_trap_halt`` without + ``echo_trap_enabled`` is a silent no-op footgun — reject. + """ + tcfg = self.training + if not tcfg.echo_trap_enabled and not tcfg.echo_trap_halt: + return self + if not tcfg.echo_trap_enabled and tcfg.echo_trap_halt: + raise ValueError( + "echo_trap_halt=True requires echo_trap_enabled=True" + ) + if self.task not in ("grpo", "ppo"): + raise ValueError( + "echo_trap_enabled / echo_trap_halt are only valid on " + f"task in {{'grpo', 'ppo'}}; got task={self.task!r}" + ) + if self.backend == "mlx": + raise ValueError( + "echo_trap_enabled is not supported on backend=mlx in " + "v0.70.0" + ) + return self + + @model_validator(mode="after") + def _validate_rl_checkpoint_compat(self) -> "SoupConfig": + """v0.70.0 Part D — mid-epoch RL checkpoint task gate. + + ``rl_checkpoint_save_every_steps`` is only meaningful on RL + tasks (grpo / ppo). Non-RL tasks already have HF Trainer's + per-epoch checkpointing. Rejected on other tasks with a + friendly message. + """ + tcfg = self.training + if tcfg.rl_checkpoint_save_every_steps is None: + return self + if self.task not in ("grpo", "ppo"): + raise ValueError( + "rl_checkpoint_save_every_steps is only valid on RL tasks " + f"(grpo / ppo); got task={self.task!r}. Non-RL tasks use " + "HF Trainer's per-epoch checkpointing already." + ) + if self.backend == "mlx": + raise ValueError( + "rl_checkpoint_save_every_steps is not supported on " + "backend=mlx in v0.70.0" + ) + return self + + @model_validator(mode="after") + def _validate_minillm_compat(self) -> "SoupConfig": + """v0.70.0 Part C — MiniLLM compatibility gate. + + ``minillm_enabled`` requires ``task='distill'`` on a non-mlx + backend. Setting any minillm_* tunable without + ``minillm_enabled=True`` is rejected (silent no-op footgun + mirroring v0.52 distill / v0.62 grace_codebook policy). + """ + tcfg = self.training + any_field_set = ( + tcfg.minillm_teacher_mix_ratio != 0.0 + or tcfg.minillm_length_normalize is not True + or tcfg.minillm_pretrain_anchor_weight != 0.0 + or tcfg.minillm_pretrain_anchor_path is not None + ) + if not tcfg.minillm_enabled and not any_field_set: + return self + if not tcfg.minillm_enabled and any_field_set: + offenders = [] + if tcfg.minillm_teacher_mix_ratio != 0.0: + offenders.append("minillm_teacher_mix_ratio") + if tcfg.minillm_length_normalize is not True: + offenders.append("minillm_length_normalize") + if tcfg.minillm_pretrain_anchor_weight != 0.0: + offenders.append("minillm_pretrain_anchor_weight") + if tcfg.minillm_pretrain_anchor_path is not None: + offenders.append("minillm_pretrain_anchor_path") + raise ValueError( + f"MiniLLM tunables {offenders} require minillm_enabled=True" + ) + if self.task != "distill": + raise ValueError( + "minillm_enabled requires task='distill'; " + f"got task={self.task!r}" + ) + if self.backend == "mlx": + raise ValueError( + "minillm_enabled is not supported on backend=mlx in v0.70.0" + ) + # Cross-check: anchor weight + path mutual requirements. + if ( + tcfg.minillm_pretrain_anchor_weight > 0.0 + and tcfg.minillm_pretrain_anchor_path is None + ): + raise ValueError( + "minillm_pretrain_anchor_weight > 0 requires " + "minillm_pretrain_anchor_path to be set" + ) + if ( + tcfg.minillm_pretrain_anchor_weight == 0.0 + and tcfg.minillm_pretrain_anchor_path is not None + ): + raise ValueError( + "minillm_pretrain_anchor_path is set but " + "minillm_pretrain_anchor_weight is 0 (silent no-op)" + ) + return self + + @model_validator(mode="after") + def _validate_reward_hack_compat(self) -> "SoupConfig": + """v0.70.0 Part A — reward-hacking detector task / backend gate. + + ``reward_hack_detector`` + ``reward_hack_halt`` are only meaningful + for RL tasks (grpo / ppo). Rejected outside those tasks with a + friendly message that names the offending fields. MLX backend + rejected with a distinct message (matches v0.34.0 / v0.50.0 + review-fix policy of distinct error reasons). + """ + tcfg = self.training + detector = tcfg.reward_hack_detector + halt = tcfg.reward_hack_halt + if detector is None and not halt: + return self + # halt without detector is a silent no-op footgun — reject. + if detector is None and halt: + raise ValueError( + "reward_hack_halt=True requires reward_hack_detector to be set" + ) + if self.task not in ("grpo", "ppo"): + raise ValueError( + "reward_hack_detector / reward_hack_halt are only valid on " + f"task in {{'grpo', 'ppo'}}; got task={self.task!r}" + ) + if self.backend == "mlx": + raise ValueError( + "reward_hack_detector is not supported on backend=mlx in " + "v0.70.0 (RL detectors are transformers-only)" + ) + return self + # --- Built-in templates --- diff --git a/soup_cli/utils/echo_trap.py b/soup_cli/utils/echo_trap.py new file mode 100644 index 0000000..57bba15 --- /dev/null +++ b/soup_cli/utils/echo_trap.py @@ -0,0 +1,269 @@ +"""Live echo-trap detector — v0.70.0 Part F. + +RAGEN-style detection of trajectory degeneration during multi-turn +agent RL (Zhu et al. 2025, arXiv:2504.14437). When the policy collapses +to self-repeating outputs, the reward saturates and the policy drifts +without learning. This module ships the math kernels + report schema; +the live HF Trainer callback is deferred to v0.70.1. + +Composes with v0.53.11 #127 ``GRPOStabilityCallback`` — the live +echo-trap callback shares the per-step instrumentation hook so both +detectors can fire in the same training step without duplicating +trajectory collection. + +Security: +- Pure-Python math (no torch import at module top). +- Bool / NaN / Inf / range rejection on every numeric input. +- Tokens must be strings; non-str rejected loudly so a caller that + hands tensor ids (instead of decoded strings) gets an actionable + error rather than silently misbehaving. +- ``_MAX_BATCH_TRAJECTORIES = 100_000`` DoS cap (matches v0.55 / + v0.65 / v0.66 cap policy). +- ``_MAX_NGRAM_N = 32`` keeps the n-gram counter bounded. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import Iterable, Sequence + +VERDICTS: tuple[str, ...] = ("OK", "WARN", "TRAP") +_VALID_VERDICTS: frozenset[str] = frozenset(VERDICTS) + +# OK / WARN / TRAP boundaries on the aggregate echo signal. Mirrors v0.26 +# Quant-Lobotomy + v0.56 diagnose three-band taxonomy. +_ECHO_OK_BAND = 0.30 # signal < 0.30 → OK +_ECHO_TRAP_BAND = 0.60 # signal >= 0.60 → TRAP; in between → WARN + +_MAX_NGRAM_N = 32 +_MAX_TRAJECTORY_TOKENS = 1_000_000 +_MAX_BATCH_TRAJECTORIES = 100_000 + + +def _check_ngram_n(value: object) -> int: + if isinstance(value, bool): + raise ValueError("ngram_n must not be bool") + if not isinstance(value, int): + raise ValueError(f"ngram_n must be int, got {type(value).__name__}") + if value < 1: + raise ValueError(f"ngram_n must be >= 1, got {value}") + if value > _MAX_NGRAM_N: + raise ValueError(f"ngram_n={value} exceeds {_MAX_NGRAM_N} cap") + return value + + +def _check_tokens(tokens: object) -> tuple[str, ...]: + if isinstance(tokens, (str, bytes)): + raise TypeError("tokens must be a sequence of strings, not str/bytes") + try: + iterator = list(tokens) # type: ignore[arg-type] + except TypeError as exc: + raise TypeError( + f"tokens must be iterable, got {type(tokens).__name__}" + ) from exc + if len(iterator) > _MAX_TRAJECTORY_TOKENS: + raise ValueError( + f"trajectory has {len(iterator)} tokens, exceeds " + f"{_MAX_TRAJECTORY_TOKENS} cap" + ) + for idx, t in enumerate(iterator): + if not isinstance(t, str): + raise TypeError( + f"tokens[{idx}] must be str, got {type(t).__name__}" + ) + return tuple(iterator) + + +def score_trajectory_repetition(tokens: object, *, ngram_n: object = 2) -> float: + """Per-trajectory repetition score. + + Returns the fraction of n-grams whose count exceeds 1 (the + "repeating n-grams" rate). Range ``[0, 1]``. 0 = every n-gram + unique; closer to 1 = many n-grams repeat. + + Edge cases: + - ``len(tokens) < ngram_n`` returns 0.0 (no n-grams possible). + - Empty input returns 0.0. + """ + n = _check_ngram_n(ngram_n) + tok = _check_tokens(tokens) + if len(tok) < n: + return 0.0 + counts: dict[tuple[str, ...], int] = {} + for i in range(len(tok) - n + 1): + gram = tok[i : i + n] + counts[gram] = counts.get(gram, 0) + 1 + if not counts: + return 0.0 + repeating = sum(1 for c in counts.values() if c > 1) + return repeating / len(counts) + + +def score_echo_signal( + trajectories: object, + *, + ngram_n: object = 2, +) -> float: + """Mean repetition score across a batch of trajectories. + + Higher = more trajectory degeneration = closer to echo trap. + Returns 0.0 on empty input (no signal = nothing to flag). + """ + n = _check_ngram_n(ngram_n) + if isinstance(trajectories, (str, bytes)): + raise TypeError( + "trajectories must be a sequence of sequences, not str/bytes" + ) + try: + batch = list(trajectories) # type: ignore[arg-type] + except TypeError as exc: + raise TypeError( + f"trajectories must be iterable, got " + f"{type(trajectories).__name__}" + ) from exc + if len(batch) > _MAX_BATCH_TRAJECTORIES: + raise ValueError( + f"batch has {len(batch)} trajectories, exceeds " + f"{_MAX_BATCH_TRAJECTORIES} cap" + ) + if not batch: + return 0.0 + scores: list[float] = [] + for traj in batch: + scores.append(score_trajectory_repetition(traj, ngram_n=n)) + return sum(scores) / len(scores) + + +def classify_echo_signal(signal: object) -> str: + """Map a signal in ``[0, 1]`` to OK / WARN / TRAP. + + - signal in [0.0, _ECHO_OK_BAND=0.30): OK + - signal in [_ECHO_OK_BAND, _ECHO_TRAP_BAND=0.60): WARN + - signal >= _ECHO_TRAP_BAND: TRAP + """ + if isinstance(signal, bool): + raise ValueError("signal must not be bool") + if not isinstance(signal, (int, float)): + raise ValueError( + f"signal must be a number, got {type(signal).__name__}" + ) + fv = float(signal) + if not math.isfinite(fv): + raise ValueError("signal must be finite (no NaN/Inf)") + if not (0.0 <= fv <= 1.0): + raise ValueError(f"signal must be in [0.0, 1.0], got {fv}") + if fv < _ECHO_OK_BAND: + return "OK" + if fv < _ECHO_TRAP_BAND: + return "WARN" + return "TRAP" + + +@dataclass(frozen=True) +class EchoTrapReport: + """Frozen result of an echo-trap probe. + + - ``signal``: aggregate echo signal in ``[0, 1]``. + - ``verdict``: OK / WARN / TRAP per :func:`classify_echo_signal`. + - ``step``: training step at which the probe fired. Non-negative + int (bool rejected per project policy). + - ``trajectories_seen``: count of trajectories that contributed to + the signal. Non-negative. + - ``details``: tuple of human-readable lines for the report panel. + """ + + signal: float + verdict: str + step: int + trajectories_seen: int + details: tuple[str, ...] + + def __post_init__(self) -> None: + if isinstance(self.signal, bool): + raise ValueError("signal must not be bool") + if not isinstance(self.signal, (int, float)): + raise TypeError( + f"signal must be a number, got {type(self.signal).__name__}" + ) + fv = float(self.signal) + if not math.isfinite(fv) or not (0.0 <= fv <= 1.0): + raise ValueError(f"signal must be in [0.0, 1.0], got {self.signal}") + if self.verdict not in _VALID_VERDICTS: + raise ValueError( + f"verdict={self.verdict!r} must be one of {sorted(_VALID_VERDICTS)}" + ) + if isinstance(self.step, bool): + raise ValueError("step must not be bool") + if not isinstance(self.step, int): + raise TypeError(f"step must be int, got {type(self.step).__name__}") + if self.step < 0: + raise ValueError(f"step must be non-negative, got {self.step}") + if isinstance(self.trajectories_seen, bool): + raise ValueError("trajectories_seen must not be bool") + if not isinstance(self.trajectories_seen, int): + raise TypeError( + "trajectories_seen must be int, got " + f"{type(self.trajectories_seen).__name__}" + ) + if self.trajectories_seen < 0: + raise ValueError( + f"trajectories_seen must be non-negative, got " + f"{self.trajectories_seen}" + ) + if not isinstance(self.details, tuple): + raise TypeError( + f"details must be a tuple, got {type(self.details).__name__}" + ) + + +def build_echo_trap_callback( + *, + threshold: float, + halt_on_trap: bool = True, + ngram_n: int = 2, +): + """Live HF Trainer callback for echo-trap detection. + + Deferred to v0.70.1. Validates inputs at the public boundary so + misconfigured callers fail fast (mirrors v0.50.0 / v0.62.0 / + v0.67.0 / v0.69.0 / Part A/B/C/D/E deferred-live policy). + """ + # Threshold validation runs FIRST so a bad numeric value fires the + # actionable error before we check the bool flags. + if isinstance(threshold, bool): + raise ValueError("threshold must not be bool") + if not isinstance(threshold, (int, float)): + raise ValueError( + f"threshold must be a number, got {type(threshold).__name__}" + ) + fv = float(threshold) + if not math.isfinite(fv) or not (0.0 <= fv <= 1.0): + raise ValueError(f"threshold must be in [0.0, 1.0], got {threshold}") + if not isinstance(halt_on_trap, bool): + raise TypeError( + f"halt_on_trap must be bool, got {type(halt_on_trap).__name__}" + ) + _check_ngram_n(ngram_n) + raise NotImplementedError( + f"Live echo-trap HF Trainer callback (threshold={fv}) is deferred " + "to v0.70.1. v0.70.0 ships the schema + math kernels only." + ) + + +# Public re-exports — type hints for the v0.70.1 callback signature so +# external consumers (e.g. the GRPO stability callback) can import them +# without circular dependencies. +__all__ = [ + "VERDICTS", + "EchoTrapReport", + "build_echo_trap_callback", + "classify_echo_signal", + "score_echo_signal", + "score_trajectory_repetition", +] + + +# Type aliases retained for the v0.70.1 wiring. +TrajectoryTokens = Sequence[str] +TrajectoryBatch = Iterable[TrajectoryTokens] diff --git a/soup_cli/utils/iterative_dpo.py b/soup_cli/utils/iterative_dpo.py new file mode 100644 index 0000000..7113370 --- /dev/null +++ b/soup_cli/utils/iterative_dpo.py @@ -0,0 +1,215 @@ +"""Iterative DPO loop driver — v0.70.0 Part E. + +Sample → RM-score → re-pair → retrain over N rounds. Frozen plan + +per-round artifact tracking; the actual round orchestrator (which +would invoke ``soup train --task dpo`` between rounds) is deferred to +v0.70.1 (mirrors v0.68.0 local-rl nightly-train policy). + +The plan models each round explicitly so the v0.70.1 runner can: +- skip rounds whose ``adapter_path`` already exists (resume), +- re-render pairs JSONL deterministically per round, +- track per-round pairs_count for the `runs replay` integration. + +Security: +- Frozen dataclasses with per-field validation (matches v0.67.0 + CmaesPlan / v0.68.0 CompilePlan policy). +- Bool / null-byte / oversize / non-int rejection on every input. +- ``rounds`` tuple required (List would not be immutable under + ``frozen=True``; matches v0.43 Part B / v0.61 Part E policy). +- Consecutive ``round_index`` invariant: rounds must be 0..N-1 with no + gaps (defends against caller-side bugs that would silently skip + rounds). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Tuple + +_MIN_ROUNDS = 1 +_MAX_ROUNDS = 100 +_MIN_PAIRS_PER_ROUND = 10 +_MAX_PAIRS_PER_ROUND = 1_000_000 +_MAX_PATH_LEN = 4096 + + +def validate_rounds(value: object) -> int: + """Validate the ``rounds`` count: int in [1, 100], bool rejected.""" + if isinstance(value, bool): + raise ValueError("rounds must not be bool") + if not isinstance(value, int): + raise ValueError(f"rounds must be int, got {type(value).__name__}") + if value < _MIN_ROUNDS: + raise ValueError(f"rounds must be >= {_MIN_ROUNDS}, got {value}") + if value > _MAX_ROUNDS: + raise ValueError( + f"rounds={value} exceeds {_MAX_ROUNDS} cap" + ) + return value + + +def validate_pairs_per_round(value: object) -> int: + """Validate the per-round pair count. + + Range: ``[10, 1_000_000]``. Below 10 pairs the DPO gradient signal + is too noisy to be meaningful; above 1M is a clear OOM / disk hazard. + """ + if isinstance(value, bool): + raise ValueError("pairs_per_round must not be bool") + if not isinstance(value, int): + raise ValueError( + f"pairs_per_round must be int, got {type(value).__name__}" + ) + if value < _MIN_PAIRS_PER_ROUND: + raise ValueError( + f"pairs_per_round must be >= {_MIN_PAIRS_PER_ROUND}, got {value}" + ) + if value > _MAX_PAIRS_PER_ROUND: + raise ValueError( + f"pairs_per_round={value} exceeds {_MAX_PAIRS_PER_ROUND} cap" + ) + return value + + +def _check_path(value: object, field: str) -> str: + if isinstance(value, bool): + raise ValueError(f"{field} must not be bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str, got {type(value).__name__}") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > _MAX_PATH_LEN: + raise ValueError(f"{field} exceeds {_MAX_PATH_LEN} chars") + return value + + +def _check_non_negative_int(value: object, field: str) -> int: + if isinstance(value, bool): + raise ValueError(f"{field} must not be bool") + if not isinstance(value, int): + raise TypeError(f"{field} must be int, got {type(value).__name__}") + if value < 0: + raise ValueError(f"{field} must be non-negative, got {value}") + return value + + +@dataclass(frozen=True) +class IterativeDPORound: + """Frozen per-round descriptor. + + - ``round_index``: 0-based, non-negative int (bool rejected). + - ``prompts_path``: source prompts JSONL (sampled into pairs). + - ``pairs_path``: where the round's chosen/rejected JSONL gets written. + - ``adapter_path``: where the round's DPO adapter lands. + - ``pairs_count``: number of pairs the round produced. Non-negative + int (0 allowed for plan-only rendering). + """ + + round_index: int + prompts_path: str + pairs_path: str + adapter_path: str + pairs_count: int + + def __post_init__(self) -> None: + _check_non_negative_int(self.round_index, "round_index") + _check_path(self.prompts_path, "prompts_path") + _check_path(self.pairs_path, "pairs_path") + _check_path(self.adapter_path, "adapter_path") + _check_non_negative_int(self.pairs_count, "pairs_count") + + +@dataclass(frozen=True) +class IterativeDPOPlan: + """Frozen iterative-DPO plan. + + - ``base_model``: HF id / local path. Shape-validated. + - ``reward_model``: HF id / local path of the RM used for scoring. + - ``rounds``: tuple of :class:`IterativeDPORound`; must be + consecutive (0..N-1 with no gaps). + """ + + base_model: str + reward_model: str + rounds: Tuple[IterativeDPORound, ...] + + def __post_init__(self) -> None: + _check_path(self.base_model, "base_model") + _check_path(self.reward_model, "reward_model") + if not isinstance(self.rounds, tuple): + raise TypeError( + f"rounds must be a tuple, got {type(self.rounds).__name__}" + ) + if len(self.rounds) < 1: + raise ValueError("rounds must contain at least 1 round") + for r in self.rounds: + if not isinstance(r, IterativeDPORound): + raise TypeError( + f"every rounds[] entry must be IterativeDPORound, " + f"got {type(r).__name__}" + ) + for idx, r in enumerate(self.rounds): + if r.round_index != idx: + raise ValueError( + f"rounds must have consecutive round_index 0..N-1; " + f"rounds[{idx}].round_index={r.round_index}" + ) + + +def build_iterative_dpo_plan( + *, + base_model: str, + reward_model: str, + prompts_path: str, + output_dir: str, + rounds: int, + pairs_per_round: int, +) -> IterativeDPOPlan: + """Build a canonical :class:`IterativeDPOPlan` from operator inputs. + + Per-round paths follow the pattern + ``/round-/{pairs.jsonl,adapter}``. The plan is + cheap to construct; the v0.70.1 runner consumes it. + """ + validate_rounds(rounds) + validate_pairs_per_round(pairs_per_round) + _check_path(base_model, "base_model") + _check_path(reward_model, "reward_model") + _check_path(prompts_path, "prompts_path") + _check_path(output_dir, "output_dir") + output_dir = output_dir.rstrip("/\\") + per_round = [] + for i in range(rounds): + per_round.append( + IterativeDPORound( + round_index=i, + prompts_path=prompts_path, + pairs_path=f"{output_dir}/round-{i:02d}/pairs.jsonl", + adapter_path=f"{output_dir}/round-{i:02d}/adapter", + pairs_count=pairs_per_round, + ) + ) + return IterativeDPOPlan( + base_model=base_model, + reward_model=reward_model, + rounds=tuple(per_round), + ) + + +def run_iterative_dpo(plan): + """Execute the iterative-DPO loop. Deferred to v0.70.1. + + Validates plan type at the public boundary so misconfigured callers + fail fast (mirrors v0.50.0 / v0.62.0 / v0.67.0 / v0.69.0 deferred-live + policy). + """ + if not isinstance(plan, IterativeDPOPlan): + raise TypeError( + f"plan must be IterativeDPOPlan, got {type(plan).__name__}" + ) + raise NotImplementedError( + "Live iterative-DPO loop runner is deferred to v0.70.1. " + "v0.70.0 ships the schema + plan-only renderer only." + ) diff --git a/soup_cli/utils/minillm.py b/soup_cli/utils/minillm.py new file mode 100644 index 0000000..a172405 --- /dev/null +++ b/soup_cli/utils/minillm.py @@ -0,0 +1,151 @@ +"""MiniLLM — reverse-KL on-policy distillation — v0.70.0 Part C. + +MiniLLM (Gu et al. 2024, arXiv:2306.08543) extends knowledge +distillation with three stability tricks: + +1. **Teacher-mixed sampling** — at rollout time, with probability + ``teacher_mix_ratio`` sample from teacher logits instead of student + to keep the student near a known-good distribution. +2. **Length normalisation** — divide the rollout log-probability by + the completion length so longer completions don't dominate the + gradient. +3. **Pretrain-loss anchor** — add a small SFT-on-pretrain term to the + loss to prevent the student from drifting away from coherent + language during the on-policy distillation. + +Bundles stability tricks scattered across §3 of the paper. Extends +v0.53.2 :class:`DistillTrainerWrapper`. Live wiring deferred to v0.70.1 +— mirrors v0.50.0 / v0.62.0 / v0.69.0 stub-then-live pattern. + +Security: +- Bool / NaN / Inf / range rejection on every numeric validator. +- Null-byte + 4096-char cap on ``pretrain_anchor_path``. +- ``length_normalize`` must be a real bool (no str/int coercion). +- Cross-validators reject silent-no-op combinations + (anchor_weight=0 + anchor_path set, anchor_weight > 0 + path None). +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import Optional + +_MAX_ANCHOR_PATH_LEN = 4096 + + +def _check_unit_float(value: object, field: str) -> float: + """Validate a finite float in [0.0, 1.0]. Bool rejected.""" + if isinstance(value, bool): + raise ValueError(f"{field} must not be bool") + if not isinstance(value, (int, float)): + raise ValueError( + f"{field} must be a number, got {type(value).__name__}" + ) + fv = float(value) + if not math.isfinite(fv): + raise ValueError(f"{field} must be finite (no NaN/Inf)") + if not (0.0 <= fv <= 1.0): + raise ValueError(f"{field} must be in [0.0, 1.0], got {fv}") + return fv + + +def validate_teacher_mix_ratio(value: object) -> float: + """Validate the teacher-mix sampling ratio. Range [0.0, 1.0]. + + 0.0 = student-only rollouts; 1.0 = teacher-only rollouts. Typical + MiniLLM recipes use 0.2-0.5 to balance exploration against + proximity to the teacher's distribution. + """ + return _check_unit_float(value, "teacher_mix_ratio") + + +def validate_pretrain_anchor_weight(value: object) -> float: + """Validate the pretrain-anchor loss coefficient. Range [0.0, 1.0]. + + 0.0 = no anchor; small positive (e.g. 0.1) adds the SFT-on-pretrain + term as a regulariser. Capped at 1.0 — values above would dominate + the distillation loss (silent regression to vanilla SFT). + """ + return _check_unit_float(value, "pretrain_anchor_weight") + + +def _check_path_shape(value: Optional[str]) -> Optional[str]: + """Validate a string path field for shape only (cwd containment is + deferred to the v0.70.1 runtime hook — schema permits relative + paths for the same reason v0.69.0 build_dag does).""" + if value is None: + return None + if isinstance(value, bool): + raise ValueError("pretrain_anchor_path must not be bool") + if not isinstance(value, str): + raise TypeError( + f"pretrain_anchor_path must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError("pretrain_anchor_path must be non-empty") + if "\x00" in value: + raise ValueError("pretrain_anchor_path must not contain null bytes") + if len(value) > _MAX_ANCHOR_PATH_LEN: + raise ValueError( + f"pretrain_anchor_path exceeds {_MAX_ANCHOR_PATH_LEN} chars" + ) + return value + + +@dataclass(frozen=True) +class MiniLLMConfig: + """Frozen MiniLLM configuration. + + Cross-validation: + - ``pretrain_anchor_weight > 0`` requires ``pretrain_anchor_path`` to + be set (otherwise the anchor term has nothing to anchor against). + - ``pretrain_anchor_path is not None`` requires + ``pretrain_anchor_weight > 0`` (otherwise the path is a silent + no-op). + """ + + teacher_mix_ratio: float = 0.0 + length_normalize: bool = True + pretrain_anchor_weight: float = 0.0 + pretrain_anchor_path: Optional[str] = None + + def __post_init__(self) -> None: + validate_teacher_mix_ratio(self.teacher_mix_ratio) + if not isinstance(self.length_normalize, bool): + raise TypeError( + f"length_normalize must be bool, got " + f"{type(self.length_normalize).__name__}" + ) + validate_pretrain_anchor_weight(self.pretrain_anchor_weight) + _check_path_shape(self.pretrain_anchor_path) + if self.pretrain_anchor_weight > 0.0 and self.pretrain_anchor_path is None: + raise ValueError( + "pretrain_anchor_weight > 0 requires pretrain_anchor_path " + "to be set" + ) + if ( + self.pretrain_anchor_weight == 0.0 + and self.pretrain_anchor_path is not None + ): + raise ValueError( + "pretrain_anchor_path is set but pretrain_anchor_weight is " + "0 (silent no-op); set anchor_weight > 0 or clear the path" + ) + + +def build_minillm_callback(config): + """Build the MiniLLM HF Trainer callback. Deferred to v0.70.1. + + Validates the config type at the public boundary so misconfigured + callers fail fast (mirrors v0.50.0 / v0.62.0 / v0.67.0 / v0.69.0 + deferred-live policy). + """ + if not isinstance(config, MiniLLMConfig): + raise TypeError( + f"config must be MiniLLMConfig, got {type(config).__name__}" + ) + raise NotImplementedError( + "Live MiniLLM HF Trainer callback is deferred to v0.70.1. " + "v0.70.0 ships the schema + validators only." + ) diff --git a/soup_cli/utils/reward_hacking.py b/soup_cli/utils/reward_hacking.py new file mode 100644 index 0000000..e5e938f --- /dev/null +++ b/soup_cli/utils/reward_hacking.py @@ -0,0 +1,349 @@ +"""Reward-hacking detector — v0.70.0 Part A. + +Closed allowlist of reward-hacking detectors for GRPO/PPO training. +Surfaces an early-warning signal when the policy starts gaming the +reward model rather than improving the underlying capability. + +Two detectors ship in v0.70.0 (schema-only; live trainer callback wired +in v0.70.1): + +- ``info_rm``: Cluster-Separation Index over (good, bad) response + reward distributions. Drops in cluster separation across training = + reward model losing its grip. Inspired by Wang et al. 2024 + "InfoRM: Mitigating Reward Hacking via Information-Theoretic Reward + Modeling" (arXiv 2402.09345). + +- ``rm_ensemble``: pairwise variance across an ensemble of RMs. + Rising disagreement = unreliable reward signal. + +Live wiring deferred to v0.70.1 — mirrors v0.27.0 MII / v0.50.0 +GRPO Plus / v0.61.0 unlearning stub-then-live pattern. + +Security: +- Closed allowlist (frozenset); arbitrary detector name rejected. +- Bool / null-byte / non-string / oversize rejection on every public + validator (mirrors v0.41.0 / v0.51.0 / v0.62.0 policy). +- ``_DETECTOR_METADATA`` wrapped in ``MappingProxyType`` for runtime + immutability (matches v0.36.0 / v0.41.0 / v0.50.0 / v0.62.0 policy). +- Math kernels never import torch at module top — lazy imports only. +- All bound-check rejections include actionable messages naming the + field + offending value. +""" + +from __future__ import annotations + +import math +import types +from dataclasses import dataclass +from typing import Optional, Sequence + +_MAX_DETECTOR_NAME_LEN = 32 +_MAX_RM_ENSEMBLE_SIZE = 32 +_EPS = 1e-9 + +SUPPORTED_HACK_DETECTORS: frozenset[str] = frozenset({"info_rm", "rm_ensemble"}) + +VERDICTS: tuple[str, ...] = ("OK", "WARN", "HACK") +_VALID_VERDICTS: frozenset[str] = frozenset(VERDICTS) + +# OK / WARN / HACK boundaries on the relative drop in cluster separation. +# These match the v0.26 Quant-Lobotomy + v0.56 diagnose three-band taxonomy +# (OK / MINOR / MAJOR) but renamed for the RL setting. +_HACK_OK_BAND = 0.10 # drop < 10% → OK +_HACK_WARN_BAND = 0.30 # 10% ≤ drop < 30% → WARN; else HACK + + +@dataclass(frozen=True) +class HackDetectorSpec: + """Static metadata for a reward-hacking detector.""" + + name: str + description: str + paper: str + + +_DETECTOR_METADATA = types.MappingProxyType({ + "info_rm": HackDetectorSpec( + name="info_rm", + description=( + "InfoRM Cluster-Separation Index over (good, bad) reward " + "distributions. Drop across training = reward-hacking signal." + ), + paper="Wang et al. 2024 — arXiv:2402.09345", + ), + "rm_ensemble": HackDetectorSpec( + name="rm_ensemble", + description=( + "Pairwise variance across an RM ensemble. Rising disagreement " + "= unreliable reward signal." + ), + paper="Coste et al. 2024 — arXiv:2312.09244", + ), +}) + + +def validate_hack_detector(name: object) -> str: + """Validate and normalise a reward-hacking detector name. + + Returns the canonical (lower-cased) name on success. Raises + ``ValueError`` with an actionable message on any failure. + """ + if isinstance(name, bool): + raise ValueError("reward_hack_detector must be a string, got bool") + if not isinstance(name, str): + raise ValueError( + f"reward_hack_detector must be a string, got {type(name).__name__}" + ) + if not name: + raise ValueError("reward_hack_detector must be a non-empty string") + if "\x00" in name: + raise ValueError("reward_hack_detector must not contain null bytes") + if len(name) > _MAX_DETECTOR_NAME_LEN: + raise ValueError( + f"reward_hack_detector exceeds {_MAX_DETECTOR_NAME_LEN} chars" + ) + normalised = name.lower() + if normalised not in SUPPORTED_HACK_DETECTORS: + raise ValueError( + f"reward_hack_detector={name!r} is not supported. " + f"Valid: {sorted(SUPPORTED_HACK_DETECTORS)}" + ) + return normalised + + +def get_detector_spec(name: str) -> HackDetectorSpec: + """Return the :class:`HackDetectorSpec` for ``name``.""" + normalised = validate_hack_detector(name) + return _DETECTOR_METADATA[normalised] + + +def _check_finite_float_sequence(values: object, field: str) -> tuple[float, ...]: + """Validate a numeric sequence: non-empty, finite, no bool.""" + if isinstance(values, (str, bytes)): + raise TypeError(f"{field} must be a sequence of numbers, not str/bytes") + try: + iterator = iter(values) # type: ignore[arg-type] + except TypeError as exc: + raise TypeError( + f"{field} must be iterable, got {type(values).__name__}" + ) from exc + out: list[float] = [] + for idx, v in enumerate(iterator): + if isinstance(v, bool): + raise ValueError( + f"{field}[{idx}] must not be bool" + ) + if not isinstance(v, (int, float)): + raise ValueError( + f"{field}[{idx}] must be a number, got {type(v).__name__}" + ) + fv = float(v) + if not math.isfinite(fv): + raise ValueError(f"{field}[{idx}] must be finite (no NaN/Inf)") + out.append(fv) + if not out: + raise ValueError(f"{field} must not be empty") + return tuple(out) + + +def _mean(seq: Sequence[float]) -> float: + return sum(seq) / len(seq) + + +def _variance(seq: Sequence[float]) -> float: + """Population variance (n divisor, not n-1) for stability with N=1.""" + m = _mean(seq) + return sum((x - m) ** 2 for x in seq) / len(seq) + + +def compute_cluster_separation( + good_scores: object, + bad_scores: object, +) -> float: + """InfoRM-style cluster-separation index. + + Returns ``(mean_good - mean_bad) / sqrt(var_good + var_bad + eps)``. + Larger = better-separated reward clusters. Watch for a sharp drop + across training steps — that signals the RM losing its grip. + + Both arguments must be non-empty iterables of finite numbers. + ``bool`` rejected per project bool-as-int policy. + """ + good = _check_finite_float_sequence(good_scores, "good_scores") + bad = _check_finite_float_sequence(bad_scores, "bad_scores") + delta = _mean(good) - _mean(bad) + pooled = _variance(good) + _variance(bad) + _EPS + return delta / math.sqrt(pooled) + + +def compute_rm_ensemble_divergence(rm_scores: object) -> float: + """Mean pairwise variance across a small RM ensemble. + + Input is a sequence of per-RM score lists; every inner list must be + the same length (one score per prompt, aligned across RMs). + Returns the mean of the per-prompt variance over RMs. Higher = + RMs disagree more = reward signal less reliable. + + Bounds: + - ensemble size in [2, _MAX_RM_ENSEMBLE_SIZE=32] + - all inner lists same length (≥ 1) + - every value finite + non-bool + """ + if isinstance(rm_scores, (str, bytes)): + raise TypeError("rm_scores must be a sequence of sequences, not str/bytes") + try: + outer = list(rm_scores) # type: ignore[arg-type] + except TypeError as exc: + raise TypeError( + f"rm_scores must be iterable, got {type(rm_scores).__name__}" + ) from exc + if len(outer) < 2: + raise ValueError("rm_scores requires at least 2 RMs (ensemble divergence)") + if len(outer) > _MAX_RM_ENSEMBLE_SIZE: + raise ValueError( + f"rm_scores has too many RMs (>{_MAX_RM_ENSEMBLE_SIZE} cap)" + ) + per_rm: list[tuple[float, ...]] = [] + expected_len: Optional[int] = None + for idx, inner in enumerate(outer): + seq = _check_finite_float_sequence(inner, f"rm_scores[{idx}]") + if expected_len is None: + expected_len = len(seq) + elif len(seq) != expected_len: + raise ValueError( + f"rm_scores[{idx}] has length {len(seq)} but expected " + f"{expected_len} (all RM score lists must share length)" + ) + per_rm.append(seq) + # Per-prompt variance over RMs, then mean across prompts. + assert expected_len is not None + per_prompt_var: list[float] = [] + for j in range(expected_len): + column = [rm[j] for rm in per_rm] + per_prompt_var.append(_variance(column)) + return _mean(per_prompt_var) if per_prompt_var else 0.0 + + +def classify_hack_signal(drop_pct: object) -> str: + """Map a relative drop in cluster-separation to OK / WARN / HACK. + + ``drop_pct`` is the *relative* drop: + ``(baseline_signal - current_signal) / baseline_signal``. Non-negative + by definition; the caller clamps to 0 when the RM improves. + + Boundaries (matches v0.26 Quant-Lobotomy / v0.56 diagnose taxonomy): + - drop_pct ∈ [0, _HACK_OK_BAND=0.10): OK + - drop_pct ∈ [_HACK_OK_BAND, _HACK_WARN_BAND=0.30): WARN + - drop_pct >= _HACK_WARN_BAND: HACK + """ + if isinstance(drop_pct, bool): + raise ValueError("drop_pct must not be bool") + if not isinstance(drop_pct, (int, float)): + raise ValueError( + f"drop_pct must be a number, got {type(drop_pct).__name__}" + ) + fv = float(drop_pct) + if not math.isfinite(fv): + raise ValueError("drop_pct must be finite (no NaN/Inf)") + if fv < 0.0: + raise ValueError(f"drop_pct must be non-negative, got {fv}") + if fv < _HACK_OK_BAND: + return "OK" + if fv < _HACK_WARN_BAND: + return "WARN" + return "HACK" + + +@dataclass(frozen=True) +class RewardHackReport: + """Frozen result of a reward-hacking probe. + + - ``detector``: which detector produced the signal (allowlist). + - ``signal``: the raw scalar (cluster-sep value or ensemble variance). + Non-negative + finite. + - ``verdict``: OK / WARN / HACK per :func:`classify_hack_signal`. + - ``step``: training step at which the probe fired. Non-negative int, + bool rejected per project bool-as-int policy. + - ``baseline_signal``: the reference signal recorded at the start of + training. Used to compute the relative drop. Finite, non-negative. + - ``details``: tuple of human-readable lines for the report panel. + """ + + detector: str + signal: float + verdict: str + step: int + baseline_signal: float + details: tuple[str, ...] + + def __post_init__(self) -> None: + # validate_hack_detector normalises; re-write via object.__setattr__ + # so frozen + canonical-case invariants stay together. + normalised = validate_hack_detector(self.detector) + if normalised != self.detector: + object.__setattr__(self, "detector", normalised) + if self.verdict not in _VALID_VERDICTS: + raise ValueError( + f"verdict={self.verdict!r} must be one of {sorted(_VALID_VERDICTS)}" + ) + if isinstance(self.signal, bool): + raise ValueError("signal must not be bool") + if not isinstance(self.signal, (int, float)): + raise TypeError( + f"signal must be a number, got {type(self.signal).__name__}" + ) + if not math.isfinite(float(self.signal)): + raise ValueError("signal must be finite") + if float(self.signal) < 0.0: + raise ValueError(f"signal must be non-negative, got {self.signal}") + if isinstance(self.step, bool): + raise ValueError("step must not be bool") + if not isinstance(self.step, int): + raise TypeError(f"step must be int, got {type(self.step).__name__}") + if self.step < 0: + raise ValueError(f"step must be non-negative, got {self.step}") + if isinstance(self.baseline_signal, bool): + raise ValueError("baseline_signal must not be bool") + if not isinstance(self.baseline_signal, (int, float)): + raise TypeError("baseline_signal must be a number") + if not math.isfinite(float(self.baseline_signal)): + raise ValueError("baseline_signal must be finite") + if float(self.baseline_signal) < 0.0: + raise ValueError("baseline_signal must be non-negative") + if not isinstance(self.details, tuple): + raise TypeError( + f"details must be a tuple, got {type(self.details).__name__}" + ) + + +def build_reward_hack_callback( + *, + detector: str, + halt_on_hack: bool = True, + baseline_signal: Optional[float] = None, +): + """Live HF Trainer callback factory — deferred to v0.70.1. + + Validates inputs at construction time so misconfigured runs fail + fast even though the live callback is not yet wired. Raises + ``NotImplementedError`` with explicit v0.70.1 marker after + validation (mirrors v0.50.0 ``apply_variant_loss`` / v0.61.0 + ``apply_unlearn_loss`` policy). + """ + # Validation order: name first (cheap, allowlist) then bool guard. + validate_hack_detector(detector) + if not isinstance(halt_on_hack, bool): + raise TypeError( + f"halt_on_hack must be bool, got {type(halt_on_hack).__name__}" + ) + if baseline_signal is not None: + if isinstance(baseline_signal, bool): + raise TypeError("baseline_signal must not be bool") + if not isinstance(baseline_signal, (int, float)): + raise TypeError("baseline_signal must be a number or None") + if not math.isfinite(float(baseline_signal)) or float(baseline_signal) < 0.0: + raise ValueError("baseline_signal must be finite and non-negative") + raise NotImplementedError( + f"Live reward-hack callback for detector={detector!r} is deferred to " + "v0.70.1. v0.70.0 ships the schema + math kernels only." + ) diff --git a/soup_cli/utils/rl_checkpoint.py b/soup_cli/utils/rl_checkpoint.py new file mode 100644 index 0000000..4f65ac5 --- /dev/null +++ b/soup_cli/utils/rl_checkpoint.py @@ -0,0 +1,209 @@ +"""Mid-epoch checkpoint for PPO / GRPO — v0.70.0 Part D. + +Optimizer-state serialization for long RL runs that need to survive +preemption mid-rollout. TorchTune explicitly punts this (their README +notes "we expect users to resume from the start of the most recent +epoch"); Soup ships a real mid-epoch save/load surface here. + +Composes with: +- v0.32 spike-recovery (the recovery policy can hop back to the most + recent rl checkpoint instead of the start of the epoch). +- v0.40.0 ref-model regen (ref-model state is captured in the + checkpoint manifest so a resume restarts with the same reference + distribution). + +Schema-only release; live save_state / load_state HF Trainer callback +deferred to v0.70.1. Mirrors v0.50.0 / v0.62.0 / v0.69.0 stub-then-live +cadence. + +Security: +- ``RLCheckpointConfig`` is frozen + per-field-validated. +- Bool / non-int rejection on every numeric (bool-before-int policy). +- ``RLCheckpointState.task`` allowlisted to RL tasks only. +- ``checkpoint_dir`` shape-validated (null-byte / oversize) — cwd + containment + symlink rejection happen at v0.70.1 disk-write time + (matches v0.69.0 build_dag deferred-write-containment policy). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +_MAX_SAVE_EVERY_STEPS = 10_000_000 +_MIN_KEEP_LAST = 1 +_MAX_KEEP_LAST = 100 +_MAX_DIR_LEN = 4096 +_MAX_SOUP_VERSION_LEN = 32 + +# RL tasks whose mid-epoch checkpoint makes sense. Other tasks already +# have HF Trainer's per-epoch checkpoint and don't need the rollout +# / ref-model state captured. +_RL_TASKS: frozenset[str] = frozenset({"grpo", "ppo"}) + + +def validate_save_every_steps(value: object) -> int: + """Bool-rejecting + bounded validator for save_every_steps. + + Range: ``[1, _MAX_SAVE_EVERY_STEPS=10_000_000]``. + """ + if isinstance(value, bool): + raise ValueError("save_every_steps must not be bool") + if not isinstance(value, int): + raise ValueError( + f"save_every_steps must be int, got {type(value).__name__}" + ) + if value < 1: + raise ValueError(f"save_every_steps must be >= 1, got {value}") + if value > _MAX_SAVE_EVERY_STEPS: + raise ValueError( + f"save_every_steps={value} exceeds {_MAX_SAVE_EVERY_STEPS} cap" + ) + return value + + +def _validate_keep_last(value: object) -> int: + if isinstance(value, bool): + raise ValueError("keep_last must not be bool") + if not isinstance(value, int): + raise ValueError( + f"keep_last must be int, got {type(value).__name__}" + ) + if value < _MIN_KEEP_LAST or value > _MAX_KEEP_LAST: + raise ValueError( + f"keep_last must be in [{_MIN_KEEP_LAST}, {_MAX_KEEP_LAST}], " + f"got {value}" + ) + return value + + +def _validate_bool_flag(value: object, field: str) -> bool: + if not isinstance(value, bool): + raise TypeError(f"{field} must be bool, got {type(value).__name__}") + return value + + +def _validate_dir_shape(value: object, field: str) -> str: + if isinstance(value, bool): + raise TypeError(f"{field} must be str, got bool") + if not isinstance(value, str): + raise TypeError(f"{field} must be str, got {type(value).__name__}") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if len(value) > _MAX_DIR_LEN: + raise ValueError(f"{field} exceeds {_MAX_DIR_LEN} chars") + return value + + +@dataclass(frozen=True) +class RLCheckpointConfig: + """Frozen RL-checkpoint config. + + - ``save_every_steps``: int >= 1; ``[1, 10_000_000]`` cap. + - ``include_optimizer_state``: include AdamW / Lion state in the + checkpoint. Default True (the whole point of mid-epoch RL ckpt). + - ``include_ref_model``: include the frozen reference model state. + Default False (ref model can usually be reconstructed from + ``cfg.base``). + - ``include_rollout_buffer``: include the replay / rollout buffer + so resumed runs don't lose collected experience. + - ``keep_last``: number of recent checkpoints to retain. Older + checkpoints get pruned at write time. + """ + + save_every_steps: int + include_optimizer_state: bool = True + include_ref_model: bool = False + include_rollout_buffer: bool = False + keep_last: int = 3 + + def __post_init__(self) -> None: + validate_save_every_steps(self.save_every_steps) + _validate_bool_flag(self.include_optimizer_state, "include_optimizer_state") + _validate_bool_flag(self.include_ref_model, "include_ref_model") + _validate_bool_flag(self.include_rollout_buffer, "include_rollout_buffer") + _validate_keep_last(self.keep_last) + + +@dataclass(frozen=True) +class RLCheckpointState: + """Frozen state manifest persisted at each mid-epoch save. + + Written to ``/manifest.json``. Allows a resume + handler in v0.70.1 to verify the checkpoint shape before loading + the (potentially large) optimizer state. + + - ``step``: training step at which the checkpoint was taken. + Non-negative int (bool rejected per project policy). + - ``checkpoint_dir``: directory holding the saved tensors. Shape + validated only (cwd-containment deferred to v0.70.1 disk hook). + - ``task``: must be in the RL allowlist (``grpo`` / ``ppo``). + - ``has_optimizer``/``has_ref_model``/``has_rollout_buffer``: real + bools (no str/int coercion). + - ``soup_version``: capped at 32 chars; null-byte rejected. + """ + + step: int + checkpoint_dir: str + task: str + has_optimizer: bool + has_ref_model: bool + has_rollout_buffer: bool + soup_version: str + + def __post_init__(self) -> None: + if isinstance(self.step, bool): + raise ValueError("step must not be bool") + if not isinstance(self.step, int): + raise TypeError(f"step must be int, got {type(self.step).__name__}") + if self.step < 0: + raise ValueError(f"step must be non-negative, got {self.step}") + _validate_dir_shape(self.checkpoint_dir, "checkpoint_dir") + if not isinstance(self.task, str) or not self.task: + raise ValueError("task must be a non-empty string") + if self.task not in _RL_TASKS: + raise ValueError( + f"task={self.task!r} must be one of {sorted(_RL_TASKS)}" + ) + _validate_bool_flag(self.has_optimizer, "has_optimizer") + _validate_bool_flag(self.has_ref_model, "has_ref_model") + _validate_bool_flag(self.has_rollout_buffer, "has_rollout_buffer") + if not isinstance(self.soup_version, str) or not self.soup_version: + raise ValueError("soup_version must be a non-empty string") + if "\x00" in self.soup_version: + raise ValueError("soup_version must not contain null bytes") + if len(self.soup_version) > _MAX_SOUP_VERSION_LEN: + raise ValueError( + f"soup_version exceeds {_MAX_SOUP_VERSION_LEN} chars" + ) + + def to_dict(self) -> dict[str, Any]: + """JSON-serialisable dict for the manifest file.""" + return { + "step": self.step, + "checkpoint_dir": self.checkpoint_dir, + "task": self.task, + "has_optimizer": self.has_optimizer, + "has_ref_model": self.has_ref_model, + "has_rollout_buffer": self.has_rollout_buffer, + "soup_version": self.soup_version, + } + + +def build_rl_checkpoint_callback(config): + """Live HF Trainer callback for mid-epoch RL checkpoints. + + Deferred to v0.70.1. Validates the config type at the public + boundary so misconfigured callers fail fast (mirrors v0.50.0 / + v0.62.0 / v0.67.0 / v0.69.0 deferred-live policy). + """ + if not isinstance(config, RLCheckpointConfig): + raise TypeError( + f"config must be RLCheckpointConfig, got {type(config).__name__}" + ) + raise NotImplementedError( + "Live mid-epoch RL checkpoint callback is deferred to v0.70.1. " + "v0.70.0 ships the schema + state manifest only." + ) diff --git a/soup_cli/utils/uld.py b/soup_cli/utils/uld.py new file mode 100644 index 0000000..728fdd8 --- /dev/null +++ b/soup_cli/utils/uld.py @@ -0,0 +1,200 @@ +"""Universal Logit Distillation (ULD) — v0.70.0 Part B. + +Cross-tokenizer knowledge distillation extending v0.53.2 DistillTrainer +to teacher/student pairs with **different vocabularies** (e.g. Llama → +Mistral, Llama → Qwen, Qwen → Llama). The existing v0.53.2 path +assumes column-wise logit alignment, which fails the moment student +and teacher have different vocab sizes. + +Two strategies ship in v0.70.0 (schema-only; live projection module +wired in v0.70.1): + +- ``wasserstein``: Boizard et al. 2024 ULD — uses 1D Wasserstein + distance between sorted teacher / student logit distributions. No + alignment required; works across arbitrary vocab boundaries. + (arXiv 2402.12030) + +- ``topk_align``: Top-K projection — pick top-K teacher logits, map + via BPE-overlap heuristic to student token ids, distil only on the + aligned subset. Requires ``top_k`` to be set. + +Live wiring deferred to v0.70.1 — mirrors v0.27.0 MII / v0.50.0 GRPO +Plus / v0.61.0 unlearning / v0.62.0 RAG stub-then-live pattern. + +Security: +- Closed allowlist (frozenset); arbitrary strategy rejected. +- Bool / null-byte / non-string / oversize rejection on every validator. +- Vocab-size bounds: [1, 262144] (covers multilingual SentencePiece + + GPT-OSS 200K vocab; matches v0.42 token-cap policy). +- ``top_k`` mutually exclusive with non-topk_align strategies (silent + no-op footgun rejection mirroring v0.52 distill / classifier policy). +- No top-level torch import — lazy import inside ``build_uld_projection``. +""" + +from __future__ import annotations + +import types +from dataclasses import dataclass +from typing import Optional + +_MAX_STRATEGY_NAME_LEN = 32 +# 262144 covers multilingual SentencePiece (e.g. NLLB) + GPT-OSS 200K. +_MAX_VOCAB_SIZE = 262144 + +SUPPORTED_ULD_STRATEGIES: frozenset[str] = frozenset({"wasserstein", "topk_align"}) + + +@dataclass(frozen=True) +class ULDStrategySpec: + """Static metadata for a ULD strategy.""" + + name: str + description: str + requires_top_k: bool + + +_STRATEGY_METADATA = types.MappingProxyType({ + "wasserstein": ULDStrategySpec( + name="wasserstein", + description=( + "1D Wasserstein distance on sorted logit distributions. " + "No alignment required across vocabularies." + ), + requires_top_k=False, + ), + "topk_align": ULDStrategySpec( + name="topk_align", + description=( + "Top-K teacher logit alignment via BPE overlap. Requires " + "top_k to be set." + ), + requires_top_k=True, + ), +}) + + +def validate_uld_strategy(name: object) -> str: + """Validate and normalise a ULD strategy name. + + Returns the canonical (lower-cased) name. Raises ``ValueError`` on + any failure. + """ + if isinstance(name, bool): + raise ValueError("uld_strategy must be a string, got bool") + if not isinstance(name, str): + raise ValueError( + f"uld_strategy must be a string, got {type(name).__name__}" + ) + if not name: + raise ValueError("uld_strategy must be a non-empty string") + if "\x00" in name: + raise ValueError("uld_strategy must not contain null bytes") + if len(name) > _MAX_STRATEGY_NAME_LEN: + raise ValueError( + f"uld_strategy exceeds {_MAX_STRATEGY_NAME_LEN} chars" + ) + normalised = name.lower() + if normalised not in SUPPORTED_ULD_STRATEGIES: + raise ValueError( + f"uld_strategy={name!r} is not supported. " + f"Valid: {sorted(SUPPORTED_ULD_STRATEGIES)}" + ) + return normalised + + +def get_strategy_spec(name: str) -> ULDStrategySpec: + """Return the :class:`ULDStrategySpec` for ``name``.""" + normalised = validate_uld_strategy(name) + return _STRATEGY_METADATA[normalised] + + +def validate_uld_projection_dim(value: object) -> int: + """Validate a projection dimensionality (vocab size). + + Bounds: ``[1, _MAX_VOCAB_SIZE=262144]``. Bool rejected per project + bool-as-int policy. + """ + if isinstance(value, bool): + raise ValueError("dim must not be bool") + if not isinstance(value, int): + raise ValueError(f"dim must be int, got {type(value).__name__}") + if value < 1: + raise ValueError(f"dim must be >= 1, got {value}") + if value > _MAX_VOCAB_SIZE: + raise ValueError( + f"dim={value} exceeds {_MAX_VOCAB_SIZE} cap" + ) + return value + + +def validate_uld_top_k(value: object) -> int: + """Validate ``top_k`` for the topk_align strategy. + + Bounds: ``[1, _MAX_VOCAB_SIZE=262144]``. Bool rejected. + """ + if isinstance(value, bool): + raise ValueError("uld_top_k must not be bool") + if not isinstance(value, int): + raise ValueError( + f"uld_top_k must be int, got {type(value).__name__}" + ) + if value < 1: + raise ValueError(f"uld_top_k must be >= 1, got {value}") + if value > _MAX_VOCAB_SIZE: + raise ValueError( + f"uld_top_k={value} exceeds {_MAX_VOCAB_SIZE} cap" + ) + return value + + +@dataclass(frozen=True) +class ULDConfig: + """Frozen ULD configuration. + + - ``strategy``: one of :data:`SUPPORTED_ULD_STRATEGIES`. + - ``student_vocab_size`` / ``teacher_vocab_size``: positive ints, + bounded by ``_MAX_VOCAB_SIZE``. + - ``top_k``: required when ``strategy='topk_align'``, rejected + otherwise (silent no-op footgun rejection). + """ + + strategy: str + student_vocab_size: int + teacher_vocab_size: int + top_k: Optional[int] = None + + def __post_init__(self) -> None: + normalised = validate_uld_strategy(self.strategy) + if normalised != self.strategy: + object.__setattr__(self, "strategy", normalised) + validate_uld_projection_dim(self.student_vocab_size) + validate_uld_projection_dim(self.teacher_vocab_size) + spec = _STRATEGY_METADATA[normalised] + if spec.requires_top_k: + if self.top_k is None: + raise ValueError( + f"uld_strategy='{normalised}' requires top_k to be set" + ) + validate_uld_top_k(self.top_k) + elif self.top_k is not None: + raise ValueError( + f"top_k is only valid when uld_strategy='topk_align'; " + f"got uld_strategy='{normalised}'" + ) + + +def build_uld_projection(config): + """Build the projection module that bridges teacher / student vocabs. + + Deferred to v0.70.1. Validates the config type at the public boundary + so misconfigured callers fail fast (mirrors v0.50.0 ``apply_variant_loss`` + / v0.62.0 ``apply_steering`` / v0.67.0 ``apply_bank_to_serve`` policy). + """ + if not isinstance(config, ULDConfig): + raise TypeError( + f"config must be ULDConfig, got {type(config).__name__}" + ) + raise NotImplementedError( + f"Live ULD projection for strategy={config.strategy!r} is deferred " + "to v0.70.1. v0.70.0 ships the schema + validators only." + ) diff --git a/tests/test_v0690_part_a.py b/tests/test_v0690_part_a.py index e67ba6a..1c59f03 100644 --- a/tests/test_v0690_part_a.py +++ b/tests/test_v0690_part_a.py @@ -730,4 +730,5 @@ class TestSourceWiring: def test_version_bumped(self) -> None: from soup_cli import __version__ - assert __version__ == "0.69.0" + major_minor = tuple(int(x) for x in __version__.split(".")[:2]) + assert major_minor >= (0, 69) diff --git a/tests/test_v0690_part_b.py b/tests/test_v0690_part_b.py index a573be9..57905bb 100644 --- a/tests/test_v0690_part_b.py +++ b/tests/test_v0690_part_b.py @@ -584,4 +584,5 @@ class TestSourceWiring: def test_version_bumped(self) -> None: from soup_cli import __version__ - assert __version__ == "0.69.0" + major_minor = tuple(int(x) for x in __version__.split(".")[:2]) + assert major_minor >= (0, 69) diff --git a/tests/test_v0690_part_c.py b/tests/test_v0690_part_c.py index 3f72098..696bd6a 100644 --- a/tests/test_v0690_part_c.py +++ b/tests/test_v0690_part_c.py @@ -338,4 +338,5 @@ class TestSourceWiring: def test_version_bumped(self) -> None: from soup_cli import __version__ - assert __version__ == "0.69.0" + major_minor = tuple(int(x) for x in __version__.split(".")[:2]) + assert major_minor >= (0, 69) diff --git a/tests/test_v0690_part_d.py b/tests/test_v0690_part_d.py index 14b2c25..a32d0fd 100644 --- a/tests/test_v0690_part_d.py +++ b/tests/test_v0690_part_d.py @@ -413,4 +413,5 @@ class TestSourceWiring: def test_version_bumped(self) -> None: from soup_cli import __version__ - assert __version__ == "0.69.0" + major_minor = tuple(int(x) for x in __version__.split(".")[:2]) + assert major_minor >= (0, 69) diff --git a/tests/test_v0690_part_e.py b/tests/test_v0690_part_e.py index ced2a47..15dbe61 100644 --- a/tests/test_v0690_part_e.py +++ b/tests/test_v0690_part_e.py @@ -415,4 +415,5 @@ class TestSourceWiring: def test_version_bumped(self) -> None: from soup_cli import __version__ - assert __version__ == "0.69.0" + major_minor = tuple(int(x) for x in __version__.split(".")[:2]) + assert major_minor >= (0, 69) diff --git a/tests/test_v0700_part_a.py b/tests/test_v0700_part_a.py new file mode 100644 index 0000000..c20b2fc --- /dev/null +++ b/tests/test_v0700_part_a.py @@ -0,0 +1,582 @@ +"""v0.70.0 Part A — Reward-hacking detector schema + math kernels. + +Schema-only release: live trainer-callback wiring deferred to v0.70.1. +Tests cover: closed allowlist, frozen dataclasses, cluster-separation +index kernel, RM-ensemble divergence kernel, classification helper, +and all input rejection matrices. +""" + +from __future__ import annotations + +import math +from dataclasses import FrozenInstanceError + +import pytest + +# --------------------------------------------------------------------------- +# Reward-hacking detector — public surface +# --------------------------------------------------------------------------- + + +class TestRewardHackingPublicSurface: + def test_module_imports(self): + from soup_cli.utils import reward_hacking + + assert hasattr(reward_hacking, "SUPPORTED_HACK_DETECTORS") + assert hasattr(reward_hacking, "validate_hack_detector") + assert hasattr(reward_hacking, "compute_cluster_separation") + assert hasattr(reward_hacking, "compute_rm_ensemble_divergence") + assert hasattr(reward_hacking, "classify_hack_signal") + assert hasattr(reward_hacking, "RewardHackReport") + assert hasattr(reward_hacking, "build_reward_hack_callback") + + def test_supported_detectors_is_frozenset(self): + from soup_cli.utils.reward_hacking import SUPPORTED_HACK_DETECTORS + + assert isinstance(SUPPORTED_HACK_DETECTORS, frozenset) + assert "info_rm" in SUPPORTED_HACK_DETECTORS + assert "rm_ensemble" in SUPPORTED_HACK_DETECTORS + + def test_supported_detectors_immutable(self): + from soup_cli.utils.reward_hacking import SUPPORTED_HACK_DETECTORS + + with pytest.raises((AttributeError, TypeError)): + SUPPORTED_HACK_DETECTORS.add("evil") + + +class TestValidateHackDetector: + def test_happy_path(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + assert validate_hack_detector("info_rm") == "info_rm" + assert validate_hack_detector("rm_ensemble") == "rm_ensemble" + + def test_case_insensitive(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + assert validate_hack_detector("INFO_RM") == "info_rm" + assert validate_hack_detector("Rm_Ensemble") == "rm_ensemble" + + def test_unknown_raises(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + with pytest.raises(ValueError, match="not supported"): + validate_hack_detector("evil") + + def test_bool_rejected(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + with pytest.raises(ValueError, match="bool"): + validate_hack_detector(True) + + def test_empty_rejected(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + with pytest.raises(ValueError, match="non-empty"): + validate_hack_detector("") + + def test_non_string_rejected(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + with pytest.raises(ValueError, match="string"): + validate_hack_detector(123) + + def test_null_byte_rejected(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + with pytest.raises(ValueError, match="null byte"): + validate_hack_detector("info_rm\x00") + + def test_oversize_rejected(self): + from soup_cli.utils.reward_hacking import validate_hack_detector + + with pytest.raises(ValueError, match="exceeds"): + validate_hack_detector("x" * 64) + + +class TestComputeClusterSeparation: + """InfoRM Cluster-Separation Index kernel. + + Higher separation = healthier RM (clusters of (good, bad) responses + are well-separated). Sharp drop in separation across training = + reward-hacking signal. + """ + + def test_perfect_separation(self): + from soup_cli.utils.reward_hacking import compute_cluster_separation + + good_scores = [10.0, 9.5, 11.0] + bad_scores = [1.0, 0.5, 1.5] + # Far-apart clusters → high separation + value = compute_cluster_separation(good_scores, bad_scores) + assert math.isfinite(value) + assert value > 1.0 + + def test_no_separation(self): + from soup_cli.utils.reward_hacking import compute_cluster_separation + + good_scores = [5.0, 5.5, 4.5] + bad_scores = [5.0, 5.5, 4.5] + value = compute_cluster_separation(good_scores, bad_scores) + assert math.isfinite(value) + assert abs(value) < 0.1 + + def test_zero_variance_groups(self): + """Zero variance is gracefully handled (small epsilon).""" + from soup_cli.utils.reward_hacking import compute_cluster_separation + + good_scores = [5.0, 5.0, 5.0] + bad_scores = [1.0, 1.0, 1.0] + value = compute_cluster_separation(good_scores, bad_scores) + assert math.isfinite(value) + assert value > 0.0 + + def test_empty_good_rejected(self): + from soup_cli.utils.reward_hacking import compute_cluster_separation + + with pytest.raises(ValueError, match="empty"): + compute_cluster_separation([], [1.0]) + + def test_empty_bad_rejected(self): + from soup_cli.utils.reward_hacking import compute_cluster_separation + + with pytest.raises(ValueError, match="empty"): + compute_cluster_separation([1.0], []) + + def test_non_finite_rejected(self): + from soup_cli.utils.reward_hacking import compute_cluster_separation + + with pytest.raises(ValueError, match="finite"): + compute_cluster_separation([1.0, float("nan")], [0.0]) + with pytest.raises(ValueError, match="finite"): + compute_cluster_separation([float("inf")], [0.0]) + + def test_bool_in_list_rejected(self): + from soup_cli.utils.reward_hacking import compute_cluster_separation + + with pytest.raises(ValueError, match="bool"): + compute_cluster_separation([1.0, True], [0.0]) + + def test_non_list_rejected(self): + from soup_cli.utils.reward_hacking import compute_cluster_separation + + with pytest.raises(TypeError): + compute_cluster_separation("not a list", [0.0]) + + +class TestComputeRmEnsembleDivergence: + """RM-ensemble divergence — measures disagreement across RMs. + + Returns mean pairwise variance across RM score lists. High variance + = RMs disagree = reward signal is unreliable. + """ + + def test_perfect_agreement(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + rm_scores = [ + [1.0, 2.0, 3.0], # RM 1 + [1.0, 2.0, 3.0], # RM 2 — identical + [1.0, 2.0, 3.0], # RM 3 — identical + ] + value = compute_rm_ensemble_divergence(rm_scores) + assert math.isfinite(value) + assert value < 1e-6 + + def test_disagreement(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + rm_scores = [ + [1.0, 2.0, 3.0], + [5.0, 7.0, 9.0], # Wildly different + [-2.0, -1.0, 0.0], + ] + value = compute_rm_ensemble_divergence(rm_scores) + assert math.isfinite(value) + assert value > 1.0 + + def test_single_rm_rejected(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + with pytest.raises(ValueError, match="at least 2"): + compute_rm_ensemble_divergence([[1.0, 2.0]]) + + def test_empty_rejected(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + with pytest.raises(ValueError, match="at least 2"): + compute_rm_ensemble_divergence([]) + + def test_uneven_lengths_rejected(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + with pytest.raises(ValueError, match="length"): + compute_rm_ensemble_divergence([[1.0, 2.0], [1.0, 2.0, 3.0]]) + + def test_non_finite_rejected(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + with pytest.raises(ValueError, match="finite"): + compute_rm_ensemble_divergence([[1.0, float("nan")], [0.0, 0.0]]) + + def test_non_list_rejected(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + with pytest.raises(TypeError): + compute_rm_ensemble_divergence("not a list") + + def test_bool_inner_rejected(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + with pytest.raises(ValueError, match="bool"): + compute_rm_ensemble_divergence([[1.0, True], [0.0, 0.0]]) + + def test_too_many_rms_rejected(self): + from soup_cli.utils.reward_hacking import compute_rm_ensemble_divergence + + too_many = [[1.0] for _ in range(100)] + with pytest.raises(ValueError, match="too many"): + compute_rm_ensemble_divergence(too_many) + + +class TestClassifyHackSignal: + """OK / WARN / HACK taxonomy matches v0.26 Quant-Lobotomy bands. + + Lower signal = healthier. Threshold semantics: + - drop_pct < 0.10 -> OK + - 0.10 <= drop_pct < 0.30 -> WARN + - drop_pct >= 0.30 -> HACK + """ + + def test_ok_threshold(self): + from soup_cli.utils.reward_hacking import classify_hack_signal + + assert classify_hack_signal(0.05) == "OK" + assert classify_hack_signal(0.0) == "OK" + + def test_warn_threshold(self): + from soup_cli.utils.reward_hacking import classify_hack_signal + + assert classify_hack_signal(0.10) == "WARN" + assert classify_hack_signal(0.20) == "WARN" + assert classify_hack_signal(0.29) == "WARN" + + def test_hack_threshold(self): + from soup_cli.utils.reward_hacking import classify_hack_signal + + assert classify_hack_signal(0.30) == "HACK" + assert classify_hack_signal(0.50) == "HACK" + assert classify_hack_signal(1.0) == "HACK" + + def test_negative_signal_rejected(self): + from soup_cli.utils.reward_hacking import classify_hack_signal + + with pytest.raises(ValueError, match="non-negative"): + classify_hack_signal(-0.1) + + def test_non_finite_rejected(self): + from soup_cli.utils.reward_hacking import classify_hack_signal + + with pytest.raises(ValueError, match="finite"): + classify_hack_signal(float("nan")) + with pytest.raises(ValueError, match="finite"): + classify_hack_signal(float("inf")) + + def test_bool_rejected(self): + from soup_cli.utils.reward_hacking import classify_hack_signal + + with pytest.raises(ValueError, match="bool"): + classify_hack_signal(True) + + def test_non_number_rejected(self): + from soup_cli.utils.reward_hacking import classify_hack_signal + + with pytest.raises(ValueError, match="number"): + classify_hack_signal("0.5") + + +class TestRewardHackReport: + def test_basic_construction(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + report = RewardHackReport( + detector="info_rm", + signal=0.15, + verdict="WARN", + step=100, + baseline_signal=0.05, + details=("cluster sep dropped 12%",), + ) + assert report.detector == "info_rm" + assert report.signal == 0.15 + assert report.verdict == "WARN" + assert report.step == 100 + + def test_frozen(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + report = RewardHackReport( + detector="info_rm", + signal=0.15, + verdict="WARN", + step=100, + baseline_signal=0.05, + details=(), + ) + with pytest.raises(FrozenInstanceError): + report.signal = 0.2 # type: ignore[misc] + + def test_invalid_detector_rejected(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + with pytest.raises(ValueError, match="not supported"): + RewardHackReport( + detector="evil", + signal=0.0, + verdict="OK", + step=0, + baseline_signal=0.0, + details=(), + ) + + def test_invalid_verdict_rejected(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + with pytest.raises(ValueError, match="verdict"): + RewardHackReport( + detector="info_rm", + signal=0.0, + verdict="EVIL", + step=0, + baseline_signal=0.0, + details=(), + ) + + def test_negative_signal_rejected(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + with pytest.raises(ValueError): + RewardHackReport( + detector="info_rm", + signal=-0.1, + verdict="OK", + step=0, + baseline_signal=0.0, + details=(), + ) + + def test_negative_step_rejected(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + with pytest.raises(ValueError, match="step"): + RewardHackReport( + detector="info_rm", + signal=0.0, + verdict="OK", + step=-1, + baseline_signal=0.0, + details=(), + ) + + def test_bool_step_rejected(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + with pytest.raises(ValueError, match="bool"): + RewardHackReport( + detector="info_rm", + signal=0.0, + verdict="OK", + step=True, + baseline_signal=0.0, + details=(), + ) + + def test_details_must_be_tuple(self): + from soup_cli.utils.reward_hacking import RewardHackReport + + with pytest.raises(TypeError, match="tuple"): + RewardHackReport( + detector="info_rm", + signal=0.0, + verdict="OK", + step=0, + baseline_signal=0.0, + details=["not a tuple"], # type: ignore[arg-type] + ) + + +class TestBuildRewardHackCallbackStub: + """Live trainer-callback wiring deferred to v0.70.1. + + The factory validates inputs at construction time and raises + NotImplementedError with explicit v0.70.1 marker (mirrors v0.50.0 + apply_variant_loss policy). + """ + + def test_invalid_detector_rejected_before_deferred(self): + from soup_cli.utils.reward_hacking import build_reward_hack_callback + + # Validation runs BEFORE NotImplementedError. + with pytest.raises(ValueError, match="not supported"): + build_reward_hack_callback(detector="evil") + + def test_deferred_v0701(self): + from soup_cli.utils.reward_hacking import build_reward_hack_callback + + with pytest.raises(NotImplementedError, match="v0.70.1"): + build_reward_hack_callback(detector="info_rm") + + def test_bool_halt_on_hack_rejected(self): + from soup_cli.utils.reward_hacking import build_reward_hack_callback + + with pytest.raises(TypeError, match="halt_on_hack"): + build_reward_hack_callback(detector="info_rm", halt_on_hack="yes") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# Schema integration — TrainingConfig + SoupConfig +# --------------------------------------------------------------------------- + + +class TestSchemaTrainingConfig: + def test_default_none(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig() + assert tcfg.reward_hack_detector is None + assert tcfg.reward_hack_halt is False + + def test_accept_info_rm(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(reward_hack_detector="info_rm") + assert tcfg.reward_hack_detector == "info_rm" + + def test_accept_rm_ensemble(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(reward_hack_detector="rm_ensemble") + assert tcfg.reward_hack_detector == "rm_ensemble" + + def test_unknown_detector_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(reward_hack_detector="evil") + + def test_halt_must_be_bool(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + # Pydantic wraps the TypeError raised by the field validator into + # a ValidationError at construction time. + with pytest.raises((ValidationError, TypeError)): + TrainingConfig(reward_hack_halt="yes") # type: ignore[arg-type] + + +class TestSchemaSoupConfigTaskGate: + """reward_hack_detector + reward_hack_halt only meaningful on RL tasks + (grpo / ppo). Rejected on SFT / DPO / etc. with friendly message. + """ + + def _yaml(self, task: str, detector: str = "info_rm", halt: bool = False) -> str: + return f""" +base: meta-llama/Llama-3.1-8B +task: {task} +data: + train: ./data/train.jsonl + format: chatml +training: + reward_hack_detector: {detector} + reward_hack_halt: {str(halt).lower()} +""" + + def test_grpo_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string(self._yaml("grpo")) + assert cfg.training.reward_hack_detector == "info_rm" + + def test_ppo_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string(self._yaml("ppo")) + assert cfg.training.reward_hack_detector == "info_rm" + + def test_sft_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="reward_hack"): + load_config_from_string(self._yaml("sft")) + + def test_dpo_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="reward_hack"): + load_config_from_string(self._yaml("dpo")) + + def test_halt_without_detector_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="reward_hack_detector"): + load_config_from_string( + """ +base: meta-llama/Llama-3.1-8B +task: grpo +data: + train: ./data/train.jsonl + format: chatml +training: + reward_hack_halt: true +""" + ) + + def test_mlx_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError): + load_config_from_string( + """ +base: mlx-community/Llama-3.1-8B +task: grpo +backend: mlx +data: + train: ./data/train.jsonl + format: chatml +training: + reward_hack_detector: info_rm +""" + ) + + +# --------------------------------------------------------------------------- +# Source-grep wiring guards +# --------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_module_no_top_level_torch(self): + """Lazy-import policy — torch imported inside functions only.""" + from pathlib import Path + + src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "utils" + / "reward_hacking.py" + ) + body = src.read_text(encoding="utf-8") + # Bare module-level torch import would be a perf regression. + assert "\nimport torch" not in body + assert "\nfrom torch" not in body + + def test_version_bumped(self): + import soup_cli + + # We do not freeze here — checked via floor. + major_minor = tuple(int(x) for x in soup_cli.__version__.split(".")[:2]) + # 0.70 floor. + assert major_minor >= (0, 70) diff --git a/tests/test_v0700_part_b.py b/tests/test_v0700_part_b.py new file mode 100644 index 0000000..57229a2 --- /dev/null +++ b/tests/test_v0700_part_b.py @@ -0,0 +1,368 @@ +"""v0.70.0 Part B — Cross-tokenizer distillation (ULD). + +Universal Logit Distillation (Boizard et al. 2024) for distilling +across vocab boundaries (e.g. Llama -> Mistral). Schema-only release; +live distill trainer hook deferred to v0.70.1. +""" + +from __future__ import annotations + +from dataclasses import FrozenInstanceError + +import pytest + + +class TestULDPublicSurface: + def test_module_imports(self): + from soup_cli.utils import uld + + assert hasattr(uld, "SUPPORTED_ULD_STRATEGIES") + assert hasattr(uld, "validate_uld_strategy") + assert hasattr(uld, "validate_uld_projection_dim") + assert hasattr(uld, "ULDConfig") + assert hasattr(uld, "build_uld_projection") + + def test_supported_strategies_frozenset(self): + from soup_cli.utils.uld import SUPPORTED_ULD_STRATEGIES + + assert isinstance(SUPPORTED_ULD_STRATEGIES, frozenset) + assert "wasserstein" in SUPPORTED_ULD_STRATEGIES + assert "topk_align" in SUPPORTED_ULD_STRATEGIES + + def test_supported_strategies_immutable(self): + from soup_cli.utils.uld import SUPPORTED_ULD_STRATEGIES + + with pytest.raises((AttributeError, TypeError)): + SUPPORTED_ULD_STRATEGIES.add("evil") + + +class TestValidateULDStrategy: + def test_happy_path(self): + from soup_cli.utils.uld import validate_uld_strategy + + assert validate_uld_strategy("wasserstein") == "wasserstein" + assert validate_uld_strategy("topk_align") == "topk_align" + + def test_case_insensitive(self): + from soup_cli.utils.uld import validate_uld_strategy + + assert validate_uld_strategy("WASSERSTEIN") == "wasserstein" + assert validate_uld_strategy("TopK_Align") == "topk_align" + + def test_bool_rejected(self): + from soup_cli.utils.uld import validate_uld_strategy + + with pytest.raises(ValueError, match="bool"): + validate_uld_strategy(True) + + def test_unknown_rejected(self): + from soup_cli.utils.uld import validate_uld_strategy + + with pytest.raises(ValueError, match="not supported"): + validate_uld_strategy("evil") + + def test_empty_rejected(self): + from soup_cli.utils.uld import validate_uld_strategy + + with pytest.raises(ValueError, match="non-empty"): + validate_uld_strategy("") + + def test_non_string_rejected(self): + from soup_cli.utils.uld import validate_uld_strategy + + with pytest.raises(ValueError, match="string"): + validate_uld_strategy(42) + + def test_null_byte_rejected(self): + from soup_cli.utils.uld import validate_uld_strategy + + with pytest.raises(ValueError, match="null byte"): + validate_uld_strategy("ws\x00") + + def test_oversize_rejected(self): + from soup_cli.utils.uld import validate_uld_strategy + + with pytest.raises(ValueError, match="exceeds"): + validate_uld_strategy("x" * 64) + + +class TestValidateProjectionDim: + def test_happy_path(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + assert validate_uld_projection_dim(128) == 128 + assert validate_uld_projection_dim(32000) == 32000 + + def test_minimum_boundary(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + assert validate_uld_projection_dim(1) == 1 + + def test_maximum_boundary(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + # 262144 — max plausible vocab size (multilingual SentencePiece). + assert validate_uld_projection_dim(262144) == 262144 + + def test_zero_rejected(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + with pytest.raises(ValueError, match=">= 1"): + validate_uld_projection_dim(0) + + def test_negative_rejected(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + with pytest.raises(ValueError, match=">= 1"): + validate_uld_projection_dim(-5) + + def test_above_cap_rejected(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + with pytest.raises(ValueError, match="262144"): + validate_uld_projection_dim(262145) + + def test_bool_rejected(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + with pytest.raises(ValueError, match="bool"): + validate_uld_projection_dim(True) + + def test_non_int_rejected(self): + from soup_cli.utils.uld import validate_uld_projection_dim + + with pytest.raises(ValueError, match="int"): + validate_uld_projection_dim(128.5) + + +class TestULDConfig: + def test_basic(self): + from soup_cli.utils.uld import ULDConfig + + cfg = ULDConfig( + strategy="wasserstein", + student_vocab_size=32000, + teacher_vocab_size=128256, + ) + assert cfg.strategy == "wasserstein" + assert cfg.student_vocab_size == 32000 + assert cfg.teacher_vocab_size == 128256 + + def test_frozen(self): + from soup_cli.utils.uld import ULDConfig + + cfg = ULDConfig( + strategy="wasserstein", + student_vocab_size=32000, + teacher_vocab_size=128256, + ) + with pytest.raises(FrozenInstanceError): + cfg.strategy = "topk_align" # type: ignore[misc] + + def test_invalid_strategy_propagates(self): + from soup_cli.utils.uld import ULDConfig + + with pytest.raises(ValueError, match="not supported"): + ULDConfig( + strategy="evil", + student_vocab_size=32000, + teacher_vocab_size=128256, + ) + + def test_invalid_student_vocab(self): + from soup_cli.utils.uld import ULDConfig + + with pytest.raises(ValueError): + ULDConfig( + strategy="wasserstein", + student_vocab_size=0, + teacher_vocab_size=128256, + ) + + def test_invalid_teacher_vocab(self): + from soup_cli.utils.uld import ULDConfig + + with pytest.raises(ValueError): + ULDConfig( + strategy="wasserstein", + student_vocab_size=32000, + teacher_vocab_size=-1, + ) + + def test_topk_default_optional(self): + """top_k defaults to None on wasserstein strategy.""" + from soup_cli.utils.uld import ULDConfig + + cfg = ULDConfig( + strategy="wasserstein", + student_vocab_size=32000, + teacher_vocab_size=128256, + ) + assert cfg.top_k is None + + def test_topk_align_requires_topk(self): + from soup_cli.utils.uld import ULDConfig + + with pytest.raises(ValueError, match="top_k"): + ULDConfig( + strategy="topk_align", + student_vocab_size=32000, + teacher_vocab_size=128256, + ) + + def test_topk_align_accepts_topk(self): + from soup_cli.utils.uld import ULDConfig + + cfg = ULDConfig( + strategy="topk_align", + student_vocab_size=32000, + teacher_vocab_size=128256, + top_k=128, + ) + assert cfg.top_k == 128 + + def test_topk_on_wasserstein_rejected(self): + """top_k only makes sense on topk_align.""" + from soup_cli.utils.uld import ULDConfig + + with pytest.raises(ValueError, match="top_k"): + ULDConfig( + strategy="wasserstein", + student_vocab_size=32000, + teacher_vocab_size=128256, + top_k=128, + ) + + +class TestBuildULDProjection: + """Deferred to v0.70.1 — validates config type then raises.""" + + def test_non_config_rejected(self): + from soup_cli.utils.uld import build_uld_projection + + with pytest.raises(TypeError, match="ULDConfig"): + build_uld_projection({"strategy": "wasserstein"}) # type: ignore[arg-type] + + def test_deferred_v0701(self): + from soup_cli.utils.uld import ULDConfig, build_uld_projection + + cfg = ULDConfig( + strategy="wasserstein", + student_vocab_size=32000, + teacher_vocab_size=128256, + ) + with pytest.raises(NotImplementedError, match="v0.70.1"): + build_uld_projection(cfg) + + +# --------------------------------------------------------------------------- +# Schema integration — TrainingConfig + SoupConfig +# --------------------------------------------------------------------------- + + +class TestSchemaTrainingConfig: + def test_default_none(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig() + assert tcfg.uld_strategy is None + assert tcfg.uld_top_k is None + + def test_accept_strategy(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(uld_strategy="wasserstein") + assert tcfg.uld_strategy == "wasserstein" + + def test_unknown_strategy_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(uld_strategy="evil") + + def test_top_k_bounds(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + # Must be positive + with pytest.raises(ValidationError): + TrainingConfig(uld_top_k=0) + # Allowed. + tcfg = TrainingConfig(uld_top_k=128) + assert tcfg.uld_top_k == 128 + + +class TestSchemaSoupConfigTaskGate: + """uld_strategy only meaningful when task='distill'.""" + + def _yaml( + self, + task: str = "distill", + strategy: str = "wasserstein", + teacher: str = "meta-llama/Llama-3.1-8B", + top_k: int | None = None, + ) -> str: + topk_line = f" uld_top_k: {top_k}\n" if top_k is not None else "" + teacher_line = f" teacher_model: {teacher}\n" if task == "distill" else "" + return f""" +base: meta-llama/Llama-3.1-8B +task: {task} +data: + train: ./data/train.jsonl + format: chatml +training: +{teacher_line} uld_strategy: {strategy} +{topk_line}""" + + def test_distill_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string(self._yaml(task="distill")) + assert cfg.training.uld_strategy == "wasserstein" + + def test_sft_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="uld_strategy"): + load_config_from_string(self._yaml(task="sft")) + + def test_topk_align_requires_topk_at_schema(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="uld_top_k"): + load_config_from_string( + self._yaml(strategy="topk_align") + ) + + def test_topk_with_wasserstein_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="uld_top_k"): + load_config_from_string(self._yaml(top_k=128)) + + def test_topk_align_with_topk_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string( + self._yaml(strategy="topk_align", top_k=128) + ) + assert cfg.training.uld_strategy == "topk_align" + assert cfg.training.uld_top_k == 128 + + +class TestSourceWiring: + def test_module_no_top_level_torch(self): + from pathlib import Path + + src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "utils" + / "uld.py" + ) + body = src.read_text(encoding="utf-8") + assert "\nimport torch" not in body + assert "\nfrom torch" not in body diff --git a/tests/test_v0700_part_c.py b/tests/test_v0700_part_c.py new file mode 100644 index 0000000..6148b06 --- /dev/null +++ b/tests/test_v0700_part_c.py @@ -0,0 +1,393 @@ +"""v0.70.0 Part C — MiniLLM reverse-KL on-policy distillation. + +MiniLLM (Gu et al. 2024) bundles 3 stability tricks: +1. Teacher-mixed sampling (epsilon-greedy mix of teacher / student rollouts) +2. Length normalisation on rollout completions +3. Pretrain-loss anchor (add a small SFT-on-pretrain term to prevent drift) + +Schema-only release; live trainer wiring deferred to v0.70.1. +""" + +from __future__ import annotations + +import math +from dataclasses import FrozenInstanceError + +import pytest + + +class TestMiniLLMPublicSurface: + def test_module_imports(self): + from soup_cli.utils import minillm + + assert hasattr(minillm, "MiniLLMConfig") + assert hasattr(minillm, "validate_teacher_mix_ratio") + assert hasattr(minillm, "validate_pretrain_anchor_weight") + assert hasattr(minillm, "build_minillm_callback") + + +class TestValidateTeacherMixRatio: + """Teacher mix ratio in [0, 1]. 0 = student-only; 1 = teacher-only.""" + + def test_happy_boundary_zero(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + assert validate_teacher_mix_ratio(0.0) == 0.0 + + def test_happy_boundary_one(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + assert validate_teacher_mix_ratio(1.0) == 1.0 + + def test_happy_mid(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + assert validate_teacher_mix_ratio(0.3) == 0.3 + + def test_above_one_rejected(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"): + validate_teacher_mix_ratio(1.5) + + def test_negative_rejected(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"): + validate_teacher_mix_ratio(-0.1) + + def test_nan_rejected(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + with pytest.raises(ValueError, match="finite"): + validate_teacher_mix_ratio(float("nan")) + + def test_inf_rejected(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + with pytest.raises(ValueError, match="finite"): + validate_teacher_mix_ratio(float("inf")) + + def test_bool_rejected(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + with pytest.raises(ValueError, match="bool"): + validate_teacher_mix_ratio(True) + + def test_non_number_rejected(self): + from soup_cli.utils.minillm import validate_teacher_mix_ratio + + with pytest.raises(ValueError, match="number"): + validate_teacher_mix_ratio("0.5") + + +class TestValidatePretrainAnchorWeight: + """Pretrain anchor weight: small non-negative float, bounded [0, 1].""" + + def test_happy_path(self): + from soup_cli.utils.minillm import validate_pretrain_anchor_weight + + assert validate_pretrain_anchor_weight(0.1) == 0.1 + + def test_zero_allowed(self): + from soup_cli.utils.minillm import validate_pretrain_anchor_weight + + assert validate_pretrain_anchor_weight(0.0) == 0.0 + + def test_one_allowed(self): + from soup_cli.utils.minillm import validate_pretrain_anchor_weight + + assert validate_pretrain_anchor_weight(1.0) == 1.0 + + def test_above_one_rejected(self): + from soup_cli.utils.minillm import validate_pretrain_anchor_weight + + with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"): + validate_pretrain_anchor_weight(1.1) + + def test_negative_rejected(self): + from soup_cli.utils.minillm import validate_pretrain_anchor_weight + + with pytest.raises(ValueError, match=r"\[0\.0, 1\.0\]"): + validate_pretrain_anchor_weight(-0.1) + + def test_non_finite_rejected(self): + from soup_cli.utils.minillm import validate_pretrain_anchor_weight + + with pytest.raises(ValueError, match="finite"): + validate_pretrain_anchor_weight(float("inf")) + + def test_bool_rejected(self): + from soup_cli.utils.minillm import validate_pretrain_anchor_weight + + with pytest.raises(ValueError, match="bool"): + validate_pretrain_anchor_weight(True) + + +class TestMiniLLMConfig: + def test_defaults(self): + from soup_cli.utils.minillm import MiniLLMConfig + + cfg = MiniLLMConfig() + assert cfg.teacher_mix_ratio == 0.0 + assert cfg.length_normalize is True + assert cfg.pretrain_anchor_weight == 0.0 + assert cfg.pretrain_anchor_path is None + + def test_basic_config(self): + from soup_cli.utils.minillm import MiniLLMConfig + + cfg = MiniLLMConfig( + teacher_mix_ratio=0.3, + length_normalize=True, + pretrain_anchor_weight=0.1, + pretrain_anchor_path="./pretrain.jsonl", + ) + assert cfg.teacher_mix_ratio == 0.3 + assert cfg.pretrain_anchor_path == "./pretrain.jsonl" + + def test_frozen(self): + from soup_cli.utils.minillm import MiniLLMConfig + + cfg = MiniLLMConfig() + with pytest.raises(FrozenInstanceError): + cfg.teacher_mix_ratio = 0.5 # type: ignore[misc] + + def test_invalid_mix_ratio_propagates(self): + from soup_cli.utils.minillm import MiniLLMConfig + + with pytest.raises(ValueError): + MiniLLMConfig(teacher_mix_ratio=2.0) + + def test_invalid_anchor_weight_propagates(self): + from soup_cli.utils.minillm import MiniLLMConfig + + with pytest.raises(ValueError): + MiniLLMConfig(pretrain_anchor_weight=-0.1) + + def test_anchor_weight_without_path_rejected(self): + """If anchor_weight > 0, pretrain_anchor_path is required.""" + from soup_cli.utils.minillm import MiniLLMConfig + + with pytest.raises(ValueError, match="pretrain_anchor_path"): + MiniLLMConfig( + pretrain_anchor_weight=0.1, + pretrain_anchor_path=None, + ) + + def test_anchor_path_without_weight_rejected(self): + """If path is set but weight=0, silent no-op — reject.""" + from soup_cli.utils.minillm import MiniLLMConfig + + with pytest.raises(ValueError, match="pretrain_anchor_weight"): + MiniLLMConfig( + pretrain_anchor_weight=0.0, + pretrain_anchor_path="./pretrain.jsonl", + ) + + def test_length_normalize_must_be_bool(self): + from soup_cli.utils.minillm import MiniLLMConfig + + with pytest.raises(TypeError, match="bool"): + MiniLLMConfig(length_normalize="yes") # type: ignore[arg-type] + + def test_anchor_path_null_byte_rejected(self): + from soup_cli.utils.minillm import MiniLLMConfig + + with pytest.raises(ValueError, match="null byte"): + MiniLLMConfig( + pretrain_anchor_weight=0.1, + pretrain_anchor_path="./bad\x00", + ) + + def test_anchor_path_oversize_rejected(self): + from soup_cli.utils.minillm import MiniLLMConfig + + with pytest.raises(ValueError, match="exceeds"): + MiniLLMConfig( + pretrain_anchor_weight=0.1, + pretrain_anchor_path="./" + "x" * 5000, + ) + + +class TestBuildMiniLLMCallback: + """Live trainer callback deferred to v0.70.1.""" + + def test_non_config_rejected(self): + from soup_cli.utils.minillm import build_minillm_callback + + with pytest.raises(TypeError, match="MiniLLMConfig"): + build_minillm_callback({}) # type: ignore[arg-type] + + def test_deferred(self): + from soup_cli.utils.minillm import MiniLLMConfig, build_minillm_callback + + cfg = MiniLLMConfig() + with pytest.raises(NotImplementedError, match="v0.70.1"): + build_minillm_callback(cfg) + + +# --------------------------------------------------------------------------- +# Schema integration — TrainingConfig + SoupConfig +# --------------------------------------------------------------------------- + + +class TestSchemaTrainingConfig: + def test_defaults(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig() + assert tcfg.minillm_enabled is False + assert tcfg.minillm_teacher_mix_ratio == 0.0 + assert tcfg.minillm_length_normalize is True + assert tcfg.minillm_pretrain_anchor_weight == 0.0 + assert tcfg.minillm_pretrain_anchor_path is None + + def test_enabled_with_all_defaults(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(minillm_enabled=True) + assert tcfg.minillm_enabled is True + + def test_invalid_mix_ratio_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(minillm_teacher_mix_ratio=2.0) + + def test_invalid_anchor_weight_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(minillm_pretrain_anchor_weight=1.5) + + +class TestSchemaSoupConfigTaskGate: + """minillm_enabled only meaningful when task='distill'.""" + + def _yaml(self, task: str = "distill", **extras: object) -> str: + teacher_line = ( + " teacher_model: meta-llama/Llama-3.1-8B\n" + if task == "distill" else "" + ) + extra_lines = "".join(f" {k}: {v}\n" for k, v in extras.items()) + return f""" +base: meta-llama/Llama-3.1-8B +task: {task} +data: + train: ./data/train.jsonl + format: chatml +training: +{teacher_line}{extra_lines}""" + + def test_distill_minillm_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string( + self._yaml(task="distill", minillm_enabled=True) + ) + assert cfg.training.minillm_enabled is True + + def test_sft_minillm_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="minillm"): + load_config_from_string( + self._yaml(task="sft", minillm_enabled=True) + ) + + def test_mlx_minillm_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError): + load_config_from_string( + """ +base: mlx-community/Llama-3.1-8B +task: distill +backend: mlx +data: + train: ./data/train.jsonl + format: chatml +training: + teacher_model: meta-llama/Llama-3.1-8B + minillm_enabled: true +""" + ) + + def test_anchor_weight_without_path_rejected_at_schema(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="minillm_pretrain_anchor_path"): + load_config_from_string( + self._yaml( + task="distill", + minillm_enabled=True, + minillm_pretrain_anchor_weight=0.1, + ) + ) + + def test_anchor_path_without_weight_rejected_at_schema(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="minillm_pretrain_anchor_weight"): + load_config_from_string( + self._yaml( + task="distill", + minillm_enabled=True, + minillm_pretrain_anchor_path="./pre.jsonl", + ) + ) + + def test_minillm_fields_without_enabled_rejected(self): + """Setting tunables without minillm_enabled=True is a silent no-op.""" + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="minillm_enabled"): + load_config_from_string( + self._yaml( + task="distill", + minillm_teacher_mix_ratio=0.3, + ) + ) + + +# --------------------------------------------------------------------------- +# Source wiring guards +# --------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_module_no_top_level_torch(self): + from pathlib import Path + + src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "utils" + / "minillm.py" + ) + body = src.read_text(encoding="utf-8") + assert "\nimport torch" not in body + assert "\nfrom torch" not in body + + def test_math_isfinite_used(self): + """Anchor-weight + mix-ratio guards must use math.isfinite (not + the looser ``not nan`` idiom — matches v0.32 / v0.41 / v0.50 / v0.62 + finite-check policy). + """ + # Importing math at the top of the test triggers the regex. + _ = math + from pathlib import Path + + src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "utils" + / "minillm.py" + ) + body = src.read_text(encoding="utf-8") + assert "math.isfinite" in body diff --git a/tests/test_v0700_part_d.py b/tests/test_v0700_part_d.py new file mode 100644 index 0000000..5ca985d --- /dev/null +++ b/tests/test_v0700_part_d.py @@ -0,0 +1,352 @@ +"""v0.70.0 Part D — Mid-epoch checkpoint for PPO/GRPO. + +Optimizer-state serialization for long RL runs. TorchTune explicitly +punts this; Soup ships the schema + state-manifest builder here, with +the live save_state / load_state callback deferred to v0.70.1. +""" + +from __future__ import annotations + +from dataclasses import FrozenInstanceError + +import pytest + + +class TestRLCheckpointPublicSurface: + def test_module_imports(self): + from soup_cli.utils import rl_checkpoint + + assert hasattr(rl_checkpoint, "RLCheckpointConfig") + assert hasattr(rl_checkpoint, "RLCheckpointState") + assert hasattr(rl_checkpoint, "validate_save_every_steps") + assert hasattr(rl_checkpoint, "build_rl_checkpoint_callback") + + +class TestValidateSaveEverySteps: + def test_happy(self): + from soup_cli.utils.rl_checkpoint import validate_save_every_steps + + assert validate_save_every_steps(100) == 100 + assert validate_save_every_steps(1) == 1 + + def test_max_boundary(self): + from soup_cli.utils.rl_checkpoint import validate_save_every_steps + + # 10M steps is plenty. + assert validate_save_every_steps(10_000_000) == 10_000_000 + + def test_zero_rejected(self): + from soup_cli.utils.rl_checkpoint import validate_save_every_steps + + with pytest.raises(ValueError, match=">= 1"): + validate_save_every_steps(0) + + def test_negative_rejected(self): + from soup_cli.utils.rl_checkpoint import validate_save_every_steps + + with pytest.raises(ValueError, match=">= 1"): + validate_save_every_steps(-10) + + def test_above_cap_rejected(self): + from soup_cli.utils.rl_checkpoint import validate_save_every_steps + + with pytest.raises(ValueError, match="10000000"): + validate_save_every_steps(10_000_001) + + def test_bool_rejected(self): + from soup_cli.utils.rl_checkpoint import validate_save_every_steps + + with pytest.raises(ValueError, match="bool"): + validate_save_every_steps(True) + + def test_non_int_rejected(self): + from soup_cli.utils.rl_checkpoint import validate_save_every_steps + + with pytest.raises(ValueError, match="int"): + validate_save_every_steps(100.5) + + +class TestRLCheckpointConfig: + def test_defaults(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointConfig + + cfg = RLCheckpointConfig(save_every_steps=100) + assert cfg.save_every_steps == 100 + assert cfg.include_optimizer_state is True + assert cfg.include_ref_model is False + assert cfg.include_rollout_buffer is False + assert cfg.keep_last == 3 + + def test_frozen(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointConfig + + cfg = RLCheckpointConfig(save_every_steps=100) + with pytest.raises(FrozenInstanceError): + cfg.save_every_steps = 50 # type: ignore[misc] + + def test_keep_last_bounds(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointConfig + + cfg = RLCheckpointConfig(save_every_steps=100, keep_last=10) + assert cfg.keep_last == 10 + + with pytest.raises(ValueError, match="keep_last"): + RLCheckpointConfig(save_every_steps=100, keep_last=0) + + with pytest.raises(ValueError, match="keep_last"): + RLCheckpointConfig(save_every_steps=100, keep_last=101) + + def test_keep_last_bool_rejected(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointConfig + + with pytest.raises(ValueError, match="bool"): + RLCheckpointConfig(save_every_steps=100, keep_last=True) + + def test_invalid_save_every_propagates(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointConfig + + with pytest.raises(ValueError): + RLCheckpointConfig(save_every_steps=0) + + def test_bool_flags_must_be_bool(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointConfig + + with pytest.raises(TypeError, match="bool"): + RLCheckpointConfig( + save_every_steps=100, + include_optimizer_state="yes", # type: ignore[arg-type] + ) + + +class TestRLCheckpointState: + """State manifest written to .soup-rl-ckpt/step-NNN/manifest.json. + + Frozen + JSON-serialisable + per-field validation. + """ + + def test_basic(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + st = RLCheckpointState( + step=500, + checkpoint_dir="./.soup-rl-ckpt/step-500", + task="grpo", + has_optimizer=True, + has_ref_model=False, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + assert st.step == 500 + assert st.task == "grpo" + + def test_frozen(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + st = RLCheckpointState( + step=500, + checkpoint_dir="./.soup-rl-ckpt/step-500", + task="grpo", + has_optimizer=True, + has_ref_model=False, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + with pytest.raises(FrozenInstanceError): + st.step = 0 # type: ignore[misc] + + def test_invalid_step_rejected(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + with pytest.raises(ValueError, match="step"): + RLCheckpointState( + step=-1, + checkpoint_dir="./x", + task="grpo", + has_optimizer=True, + has_ref_model=False, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + + def test_invalid_task_rejected(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + with pytest.raises(ValueError, match="task"): + RLCheckpointState( + step=10, + checkpoint_dir="./x", + task="sft", # not an RL task + has_optimizer=True, + has_ref_model=False, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + + def test_bool_step_rejected(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + with pytest.raises(ValueError, match="bool"): + RLCheckpointState( + step=True, + checkpoint_dir="./x", + task="grpo", + has_optimizer=True, + has_ref_model=False, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + + def test_to_dict(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + st = RLCheckpointState( + step=500, + checkpoint_dir="./.soup-rl-ckpt/step-500", + task="grpo", + has_optimizer=True, + has_ref_model=True, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + d = st.to_dict() + assert isinstance(d, dict) + assert d["step"] == 500 + assert d["task"] == "grpo" + assert d["has_optimizer"] is True + assert d["has_ref_model"] is True + + def test_null_byte_dir_rejected(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + with pytest.raises(ValueError, match="null byte"): + RLCheckpointState( + step=10, + checkpoint_dir="./bad\x00", + task="grpo", + has_optimizer=True, + has_ref_model=False, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + + def test_bool_has_optimizer_rejected(self): + from soup_cli.utils.rl_checkpoint import RLCheckpointState + + with pytest.raises(TypeError, match="has_optimizer"): + RLCheckpointState( + step=10, + checkpoint_dir="./x", + task="grpo", + has_optimizer="yes", # type: ignore[arg-type] + has_ref_model=False, + has_rollout_buffer=False, + soup_version="0.70.0", + ) + + +class TestBuildRLCheckpointCallback: + """Live callback deferred to v0.70.1.""" + + def test_non_config_rejected(self): + from soup_cli.utils.rl_checkpoint import build_rl_checkpoint_callback + + with pytest.raises(TypeError, match="RLCheckpointConfig"): + build_rl_checkpoint_callback({"save_every_steps": 100}) # type: ignore[arg-type] + + def test_deferred(self): + from soup_cli.utils.rl_checkpoint import ( + RLCheckpointConfig, + build_rl_checkpoint_callback, + ) + + cfg = RLCheckpointConfig(save_every_steps=100) + with pytest.raises(NotImplementedError, match="v0.70.1"): + build_rl_checkpoint_callback(cfg) + + +# --------------------------------------------------------------------------- +# Schema integration — TrainingConfig + SoupConfig +# --------------------------------------------------------------------------- + + +class TestSchemaTrainingConfig: + def test_defaults(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig() + assert tcfg.rl_checkpoint_save_every_steps is None + assert tcfg.rl_checkpoint_keep_last == 3 + + def test_set_save_every(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(rl_checkpoint_save_every_steps=500) + assert tcfg.rl_checkpoint_save_every_steps == 500 + + def test_zero_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(rl_checkpoint_save_every_steps=0) + + def test_keep_last_bounds(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + with pytest.raises(ValidationError): + TrainingConfig(rl_checkpoint_keep_last=0) + + with pytest.raises(ValidationError): + TrainingConfig(rl_checkpoint_keep_last=101) + + +class TestSchemaSoupConfigTaskGate: + """rl_checkpoint_save_every_steps only meaningful on RL tasks + (grpo/ppo).""" + + def _yaml(self, task: str, save_every: int = 500) -> str: + return f""" +base: meta-llama/Llama-3.1-8B +task: {task} +data: + train: ./data/train.jsonl + format: chatml +training: + rl_checkpoint_save_every_steps: {save_every} +""" + + def test_grpo_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string(self._yaml("grpo")) + assert cfg.training.rl_checkpoint_save_every_steps == 500 + + def test_ppo_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string(self._yaml("ppo")) + assert cfg.training.rl_checkpoint_save_every_steps == 500 + + def test_sft_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="rl_checkpoint"): + load_config_from_string(self._yaml("sft")) + + +class TestSourceWiring: + def test_module_no_top_level_torch(self): + from pathlib import Path + + src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "utils" + / "rl_checkpoint.py" + ) + body = src.read_text(encoding="utf-8") + assert "\nimport torch" not in body + assert "\nfrom torch" not in body diff --git a/tests/test_v0700_part_e.py b/tests/test_v0700_part_e.py new file mode 100644 index 0000000..71b1ae2 --- /dev/null +++ b/tests/test_v0700_part_e.py @@ -0,0 +1,501 @@ +"""v0.70.0 Part E — Iterative DPO loop driver. + +Sample → RM-score → re-pair → retrain over N rounds. Schema + CLI live; +the actual round-orchestrator (which would invoke `soup train --task dpo` +between rounds) is deferred to v0.70.1 (mirrors v0.68.0 local-rl +nightly-train pattern). +""" + +from __future__ import annotations + +from dataclasses import FrozenInstanceError + +import pytest +from typer.testing import CliRunner + + +class TestIterativeDPOPublicSurface: + def test_module_imports(self): + from soup_cli.utils import iterative_dpo + + assert hasattr(iterative_dpo, "IterativeDPOPlan") + assert hasattr(iterative_dpo, "IterativeDPORound") + assert hasattr(iterative_dpo, "validate_rounds") + assert hasattr(iterative_dpo, "validate_pairs_per_round") + assert hasattr(iterative_dpo, "build_iterative_dpo_plan") + assert hasattr(iterative_dpo, "run_iterative_dpo") + + +class TestValidateRounds: + def test_happy(self): + from soup_cli.utils.iterative_dpo import validate_rounds + + assert validate_rounds(5) == 5 + + def test_min_boundary(self): + from soup_cli.utils.iterative_dpo import validate_rounds + + assert validate_rounds(1) == 1 + + def test_max_boundary(self): + from soup_cli.utils.iterative_dpo import validate_rounds + + # 100 rounds is plenty. + assert validate_rounds(100) == 100 + + def test_zero_rejected(self): + from soup_cli.utils.iterative_dpo import validate_rounds + + with pytest.raises(ValueError, match=">= 1"): + validate_rounds(0) + + def test_above_cap_rejected(self): + from soup_cli.utils.iterative_dpo import validate_rounds + + with pytest.raises(ValueError, match="100"): + validate_rounds(101) + + def test_bool_rejected(self): + from soup_cli.utils.iterative_dpo import validate_rounds + + with pytest.raises(ValueError, match="bool"): + validate_rounds(True) + + def test_non_int_rejected(self): + from soup_cli.utils.iterative_dpo import validate_rounds + + with pytest.raises(ValueError, match="int"): + validate_rounds(5.5) + + +class TestValidatePairsPerRound: + def test_happy(self): + from soup_cli.utils.iterative_dpo import validate_pairs_per_round + + assert validate_pairs_per_round(500) == 500 + + def test_min_boundary(self): + from soup_cli.utils.iterative_dpo import validate_pairs_per_round + + assert validate_pairs_per_round(10) == 10 + + def test_max_boundary(self): + from soup_cli.utils.iterative_dpo import validate_pairs_per_round + + assert validate_pairs_per_round(1_000_000) == 1_000_000 + + def test_below_min_rejected(self): + from soup_cli.utils.iterative_dpo import validate_pairs_per_round + + with pytest.raises(ValueError, match=">= 10"): + validate_pairs_per_round(9) + + def test_above_cap_rejected(self): + from soup_cli.utils.iterative_dpo import validate_pairs_per_round + + with pytest.raises(ValueError, match="1000000"): + validate_pairs_per_round(1_000_001) + + def test_bool_rejected(self): + from soup_cli.utils.iterative_dpo import validate_pairs_per_round + + with pytest.raises(ValueError, match="bool"): + validate_pairs_per_round(True) + + +class TestIterativeDPORound: + def test_basic(self): + from soup_cli.utils.iterative_dpo import IterativeDPORound + + rnd = IterativeDPORound( + round_index=1, + prompts_path="./data/prompts.jsonl", + pairs_path="./data/round1_pairs.jsonl", + adapter_path="./output/round1", + pairs_count=512, + ) + assert rnd.round_index == 1 + assert rnd.pairs_count == 512 + + def test_frozen(self): + from soup_cli.utils.iterative_dpo import IterativeDPORound + + rnd = IterativeDPORound( + round_index=1, + prompts_path="./p.jsonl", + pairs_path="./pairs.jsonl", + adapter_path="./out", + pairs_count=100, + ) + with pytest.raises(FrozenInstanceError): + rnd.round_index = 2 # type: ignore[misc] + + def test_negative_round_rejected(self): + from soup_cli.utils.iterative_dpo import IterativeDPORound + + with pytest.raises(ValueError, match="round_index"): + IterativeDPORound( + round_index=-1, + prompts_path="./p.jsonl", + pairs_path="./pairs.jsonl", + adapter_path="./out", + pairs_count=100, + ) + + def test_bool_round_rejected(self): + from soup_cli.utils.iterative_dpo import IterativeDPORound + + with pytest.raises(ValueError, match="bool"): + IterativeDPORound( + round_index=True, + prompts_path="./p.jsonl", + pairs_path="./pairs.jsonl", + adapter_path="./out", + pairs_count=100, + ) + + def test_null_byte_path_rejected(self): + from soup_cli.utils.iterative_dpo import IterativeDPORound + + with pytest.raises(ValueError, match="null byte"): + IterativeDPORound( + round_index=0, + prompts_path="./bad\x00", + pairs_path="./pairs.jsonl", + adapter_path="./out", + pairs_count=100, + ) + + def test_negative_pairs_rejected(self): + from soup_cli.utils.iterative_dpo import IterativeDPORound + + with pytest.raises(ValueError, match="pairs_count"): + IterativeDPORound( + round_index=0, + prompts_path="./p.jsonl", + pairs_path="./pairs.jsonl", + adapter_path="./out", + pairs_count=-1, + ) + + +class TestIterativeDPOPlan: + def test_basic(self): + from soup_cli.utils.iterative_dpo import ( + IterativeDPOPlan, + IterativeDPORound, + ) + + plan = IterativeDPOPlan( + base_model="meta-llama/Llama-3.1-8B", + reward_model="./output_rm", + rounds=( + IterativeDPORound( + round_index=0, + prompts_path="./p.jsonl", + pairs_path="./r0.jsonl", + adapter_path="./out/r0", + pairs_count=512, + ), + IterativeDPORound( + round_index=1, + prompts_path="./p.jsonl", + pairs_path="./r1.jsonl", + adapter_path="./out/r1", + pairs_count=512, + ), + ), + ) + assert len(plan.rounds) == 2 + + def test_frozen(self): + from soup_cli.utils.iterative_dpo import ( + IterativeDPOPlan, + IterativeDPORound, + ) + + plan = IterativeDPOPlan( + base_model="m", + reward_model="./rm", + rounds=( + IterativeDPORound( + round_index=0, + prompts_path="./p.jsonl", + pairs_path="./r0.jsonl", + adapter_path="./out/r0", + pairs_count=10, + ), + ), + ) + with pytest.raises(FrozenInstanceError): + plan.base_model = "evil" # type: ignore[misc] + + def test_rounds_must_be_tuple(self): + from soup_cli.utils.iterative_dpo import ( + IterativeDPOPlan, + IterativeDPORound, + ) + + rounds_list = [ + IterativeDPORound( + round_index=0, + prompts_path="./p.jsonl", + pairs_path="./r0.jsonl", + adapter_path="./out/r0", + pairs_count=10, + ), + ] + with pytest.raises(TypeError, match="tuple"): + IterativeDPOPlan( + base_model="m", + reward_model="./rm", + rounds=rounds_list, # type: ignore[arg-type] + ) + + def test_zero_rounds_rejected(self): + from soup_cli.utils.iterative_dpo import IterativeDPOPlan + + with pytest.raises(ValueError, match="rounds"): + IterativeDPOPlan( + base_model="m", + reward_model="./rm", + rounds=(), + ) + + def test_non_consecutive_round_indices_rejected(self): + from soup_cli.utils.iterative_dpo import ( + IterativeDPOPlan, + IterativeDPORound, + ) + + with pytest.raises(ValueError, match="consecutive"): + IterativeDPOPlan( + base_model="m", + reward_model="./rm", + rounds=( + IterativeDPORound( + round_index=0, + prompts_path="./p.jsonl", + pairs_path="./r0.jsonl", + adapter_path="./out/r0", + pairs_count=10, + ), + IterativeDPORound( + round_index=5, # gap + prompts_path="./p.jsonl", + pairs_path="./r5.jsonl", + adapter_path="./out/r5", + pairs_count=10, + ), + ), + ) + + def test_null_byte_base_rejected(self): + from soup_cli.utils.iterative_dpo import ( + IterativeDPOPlan, + IterativeDPORound, + ) + + with pytest.raises(ValueError, match="null byte"): + IterativeDPOPlan( + base_model="m\x00", + reward_model="./rm", + rounds=( + IterativeDPORound( + round_index=0, + prompts_path="./p.jsonl", + pairs_path="./r0.jsonl", + adapter_path="./out/r0", + pairs_count=10, + ), + ), + ) + + +class TestBuildIterativeDPOPlan: + def test_happy(self, tmp_path, monkeypatch): + from soup_cli.utils.iterative_dpo import build_iterative_dpo_plan + + monkeypatch.chdir(tmp_path) + (tmp_path / "prompts.jsonl").write_text( + '{"prompt": "hello"}\n', encoding="utf-8" + ) + plan = build_iterative_dpo_plan( + base_model="meta-llama/Llama-3.1-8B", + reward_model="./rm", + prompts_path="./prompts.jsonl", + output_dir="./out", + rounds=3, + pairs_per_round=100, + ) + assert len(plan.rounds) == 3 + assert plan.rounds[0].round_index == 0 + assert plan.rounds[2].round_index == 2 + assert plan.rounds[0].adapter_path != plan.rounds[1].adapter_path + + def test_rounds_validation(self, tmp_path, monkeypatch): + from soup_cli.utils.iterative_dpo import build_iterative_dpo_plan + + monkeypatch.chdir(tmp_path) + (tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8") + with pytest.raises(ValueError, match="rounds"): + build_iterative_dpo_plan( + base_model="m", + reward_model="./rm", + prompts_path="./prompts.jsonl", + output_dir="./out", + rounds=0, + pairs_per_round=100, + ) + + +class TestRunIterativeDPODeferred: + def test_non_plan_rejected(self): + from soup_cli.utils.iterative_dpo import run_iterative_dpo + + with pytest.raises(TypeError, match="IterativeDPOPlan"): + run_iterative_dpo({"rounds": 1}) # type: ignore[arg-type] + + def test_deferred(self): + from soup_cli.utils.iterative_dpo import ( + IterativeDPOPlan, + IterativeDPORound, + run_iterative_dpo, + ) + + plan = IterativeDPOPlan( + base_model="m", + reward_model="./rm", + rounds=( + IterativeDPORound( + round_index=0, + prompts_path="./p.jsonl", + pairs_path="./r0.jsonl", + adapter_path="./out/r0", + pairs_count=10, + ), + ), + ) + with pytest.raises(NotImplementedError, match="v0.70.1"): + run_iterative_dpo(plan) + + +# --------------------------------------------------------------------------- +# CLI smoke +# --------------------------------------------------------------------------- + + +class TestIterativeDPOCli: + def test_help(self): + from soup_cli.commands.iterative_dpo import app + + runner = CliRunner() + result = runner.invoke(app, ["--help"]) + assert result.exit_code == 0 + assert "iterative" in result.output.lower() or "dpo" in result.output.lower() + + def test_plan_only_happy(self, tmp_path, monkeypatch): + from soup_cli.commands.iterative_dpo import app + + monkeypatch.chdir(tmp_path) + (tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8") + runner = CliRunner() + result = runner.invoke( + app, + [ + "--base-model", + "meta-llama/Llama-3.1-8B", + "--reward-model", + "./rm", + "--prompts", + "./prompts.jsonl", + "--output-dir", + "./out", + "--rounds", + "3", + "--pairs-per-round", + "100", + "--plan-only", + ], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + + def test_invalid_rounds_exits_2(self, tmp_path, monkeypatch): + from soup_cli.commands.iterative_dpo import app + + monkeypatch.chdir(tmp_path) + (tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8") + runner = CliRunner() + result = runner.invoke( + app, + [ + "--base-model", + "m", + "--reward-model", + "./rm", + "--prompts", + "./prompts.jsonl", + "--output-dir", + "./out", + "--rounds", + "0", + "--pairs-per-round", + "100", + "--plan-only", + ], + ) + assert result.exit_code == 2 + + def test_live_deferred_exits_3(self, tmp_path, monkeypatch): + """Without --plan-only, the deferred live runner exits 3.""" + from soup_cli.commands.iterative_dpo import app + + monkeypatch.chdir(tmp_path) + (tmp_path / "prompts.jsonl").write_text("{}\n", encoding="utf-8") + runner = CliRunner() + result = runner.invoke( + app, + [ + "--base-model", + "m", + "--reward-model", + "./rm", + "--prompts", + "./prompts.jsonl", + "--output-dir", + "./out", + "--rounds", + "2", + "--pairs-per-round", + "100", + ], + ) + assert result.exit_code == 3, (result.output, repr(result.exception)) + + +class TestSourceWiring: + def test_module_no_top_level_torch(self): + from pathlib import Path + + src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "utils" + / "iterative_dpo.py" + ) + body = src.read_text(encoding="utf-8") + assert "\nimport torch" not in body + assert "\nfrom torch" not in body + + def test_cli_registered(self): + """soup iterative-dpo command registered on the top-level Typer app.""" + from pathlib import Path + + cli_src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "cli.py" + ) + body = cli_src.read_text(encoding="utf-8") + # Either app.command or app.add_typer wiring. + assert "iterative_dpo" in body or "iterative-dpo" in body diff --git a/tests/test_v0700_part_f.py b/tests/test_v0700_part_f.py new file mode 100644 index 0000000..669f3cd --- /dev/null +++ b/tests/test_v0700_part_f.py @@ -0,0 +1,409 @@ +"""v0.70.0 Part F — Live echo-trap detector. + +RAGEN-style detection of trajectory degeneration in multi-turn agent RL. +When the policy collapses to self-repeating outputs (echo trap), the +reward stops improving and the policy drifts. Schema + math kernels +live; trainer-callback wiring deferred to v0.70.1. +""" + +from __future__ import annotations + +import math +from dataclasses import FrozenInstanceError + +import pytest + + +class TestEchoTrapPublicSurface: + def test_module_imports(self): + from soup_cli.utils import echo_trap + + assert hasattr(echo_trap, "score_trajectory_repetition") + assert hasattr(echo_trap, "score_echo_signal") + assert hasattr(echo_trap, "classify_echo_signal") + assert hasattr(echo_trap, "EchoTrapReport") + assert hasattr(echo_trap, "build_echo_trap_callback") + assert hasattr(echo_trap, "VERDICTS") + + +class TestScoreTrajectoryRepetition: + """Per-trajectory n-gram repetition score. Higher = more repetition. + + Tail-mass = fraction of n-grams that repeat more than once. Returns + a float in [0, 1]. + """ + + def test_unique_trajectory_zero(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + # All unique tokens → 0 repetition. + tokens = ["a", "b", "c", "d", "e"] + score = score_trajectory_repetition(tokens, ngram_n=2) + assert score == 0.0 + + def test_full_repetition(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + # Identical token throughout → near-perfect repetition. + tokens = ["a"] * 10 + score = score_trajectory_repetition(tokens, ngram_n=2) + assert score > 0.5 + + def test_score_bounded(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + tokens = ["a", "b", "a", "b", "a", "b"] + score = score_trajectory_repetition(tokens, ngram_n=2) + assert 0.0 <= score <= 1.0 + + def test_short_returns_zero(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + # Fewer tokens than ngram_n → no n-grams possible. + assert score_trajectory_repetition(["a"], ngram_n=2) == 0.0 + + def test_empty_returns_zero(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + assert score_trajectory_repetition([], ngram_n=2) == 0.0 + + def test_invalid_ngram_n_rejected(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + with pytest.raises(ValueError, match="ngram_n"): + score_trajectory_repetition(["a", "b"], ngram_n=0) + with pytest.raises(ValueError, match="ngram_n"): + score_trajectory_repetition(["a", "b"], ngram_n=-1) + + def test_bool_ngram_n_rejected(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + with pytest.raises(ValueError, match="bool"): + score_trajectory_repetition(["a", "b"], ngram_n=True) + + def test_ngram_n_max_cap(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + with pytest.raises(ValueError, match="32"): + score_trajectory_repetition(["a", "b"], ngram_n=33) + + def test_non_string_token_rejected(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + with pytest.raises(TypeError, match="tokens"): + score_trajectory_repetition([1, 2, 3], ngram_n=2) # type: ignore[list-item] + + def test_non_list_tokens_rejected(self): + from soup_cli.utils.echo_trap import score_trajectory_repetition + + with pytest.raises(TypeError): + score_trajectory_repetition("abc", ngram_n=2) + + +class TestScoreEchoSignal: + """Aggregate echo signal over a batch of trajectories. + + Returns the mean repetition score across the batch. + """ + + def test_clean_trajectories(self): + from soup_cli.utils.echo_trap import score_echo_signal + + batch = [ + ["a", "b", "c", "d"], + ["e", "f", "g", "h"], + ] + score = score_echo_signal(batch, ngram_n=2) + assert math.isfinite(score) + assert 0.0 <= score < 0.1 + + def test_collapsed_batch(self): + from soup_cli.utils.echo_trap import score_echo_signal + + batch = [["a"] * 10, ["b"] * 10] + score = score_echo_signal(batch, ngram_n=2) + assert score > 0.5 + + def test_empty_batch(self): + from soup_cli.utils.echo_trap import score_echo_signal + + assert score_echo_signal([], ngram_n=2) == 0.0 + + def test_mixed_batch(self): + from soup_cli.utils.echo_trap import score_echo_signal + + batch = [["a"] * 10, ["x", "y", "z"]] + score = score_echo_signal(batch, ngram_n=2) + # Average of repetitive + clean. + assert 0.0 < score < 1.0 + + def test_non_list_batch_rejected(self): + from soup_cli.utils.echo_trap import score_echo_signal + + with pytest.raises(TypeError): + score_echo_signal("not a list", ngram_n=2) + + def test_batch_size_cap(self): + from soup_cli.utils.echo_trap import score_echo_signal + + big = [["x"] for _ in range(100_001)] + with pytest.raises(ValueError, match="batch"): + score_echo_signal(big, ngram_n=2) + + +class TestClassifyEchoSignal: + """OK / WARN / TRAP taxonomy (mirrors v0.26 / v0.56 / v0.70 Part A). + + - signal < 0.30: OK + - 0.30 <= signal < 0.60: WARN + - signal >= 0.60: TRAP + """ + + def test_ok(self): + from soup_cli.utils.echo_trap import classify_echo_signal + + assert classify_echo_signal(0.0) == "OK" + assert classify_echo_signal(0.1) == "OK" + assert classify_echo_signal(0.29) == "OK" + + def test_warn(self): + from soup_cli.utils.echo_trap import classify_echo_signal + + assert classify_echo_signal(0.30) == "WARN" + assert classify_echo_signal(0.45) == "WARN" + assert classify_echo_signal(0.59) == "WARN" + + def test_trap(self): + from soup_cli.utils.echo_trap import classify_echo_signal + + assert classify_echo_signal(0.60) == "TRAP" + assert classify_echo_signal(0.99) == "TRAP" + assert classify_echo_signal(1.0) == "TRAP" + + def test_invalid_signal_rejected(self): + from soup_cli.utils.echo_trap import classify_echo_signal + + with pytest.raises(ValueError, match="finite"): + classify_echo_signal(float("nan")) + with pytest.raises(ValueError): + classify_echo_signal(-0.1) + with pytest.raises(ValueError): + classify_echo_signal(1.5) + + def test_bool_rejected(self): + from soup_cli.utils.echo_trap import classify_echo_signal + + with pytest.raises(ValueError, match="bool"): + classify_echo_signal(True) + + +class TestEchoTrapReport: + def test_basic(self): + from soup_cli.utils.echo_trap import EchoTrapReport + + report = EchoTrapReport( + signal=0.4, + verdict="WARN", + step=200, + trajectories_seen=64, + details=("longest streak: a a a a a",), + ) + assert report.signal == 0.4 + assert report.verdict == "WARN" + + def test_frozen(self): + from soup_cli.utils.echo_trap import EchoTrapReport + + report = EchoTrapReport( + signal=0.0, + verdict="OK", + step=0, + trajectories_seen=0, + details=(), + ) + with pytest.raises(FrozenInstanceError): + report.signal = 1.0 # type: ignore[misc] + + def test_invalid_verdict_rejected(self): + from soup_cli.utils.echo_trap import EchoTrapReport + + with pytest.raises(ValueError, match="verdict"): + EchoTrapReport( + signal=0.0, + verdict="EVIL", + step=0, + trajectories_seen=0, + details=(), + ) + + def test_signal_out_of_range_rejected(self): + from soup_cli.utils.echo_trap import EchoTrapReport + + with pytest.raises(ValueError): + EchoTrapReport( + signal=1.5, + verdict="OK", + step=0, + trajectories_seen=0, + details=(), + ) + + def test_bool_step_rejected(self): + from soup_cli.utils.echo_trap import EchoTrapReport + + with pytest.raises(ValueError, match="bool"): + EchoTrapReport( + signal=0.0, + verdict="OK", + step=True, + trajectories_seen=0, + details=(), + ) + + def test_negative_trajectories_rejected(self): + from soup_cli.utils.echo_trap import EchoTrapReport + + with pytest.raises(ValueError, match="trajectories"): + EchoTrapReport( + signal=0.0, + verdict="OK", + step=0, + trajectories_seen=-1, + details=(), + ) + + def test_details_must_be_tuple(self): + from soup_cli.utils.echo_trap import EchoTrapReport + + with pytest.raises(TypeError, match="tuple"): + EchoTrapReport( + signal=0.0, + verdict="OK", + step=0, + trajectories_seen=0, + details=["not tuple"], # type: ignore[arg-type] + ) + + +class TestBuildEchoTrapCallbackDeferred: + def test_invalid_threshold_rejected_first(self): + from soup_cli.utils.echo_trap import build_echo_trap_callback + + with pytest.raises(ValueError, match="threshold"): + build_echo_trap_callback(threshold=2.0) + + def test_invalid_threshold_bool(self): + from soup_cli.utils.echo_trap import build_echo_trap_callback + + with pytest.raises(ValueError, match="bool"): + build_echo_trap_callback(threshold=True) + + def test_deferred(self): + from soup_cli.utils.echo_trap import build_echo_trap_callback + + with pytest.raises(NotImplementedError, match="v0.70.1"): + build_echo_trap_callback(threshold=0.5) + + def test_halt_must_be_bool(self): + from soup_cli.utils.echo_trap import build_echo_trap_callback + + with pytest.raises(TypeError, match="halt"): + build_echo_trap_callback(threshold=0.5, halt_on_trap="yes") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# Schema integration — TrainingConfig + SoupConfig +# --------------------------------------------------------------------------- + + +class TestSchemaTrainingConfig: + def test_defaults(self): + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig() + assert tcfg.echo_trap_enabled is False + assert tcfg.echo_trap_threshold == 0.6 + assert tcfg.echo_trap_halt is False + + def test_threshold_bounds(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + + tcfg = TrainingConfig(echo_trap_threshold=0.45) + assert tcfg.echo_trap_threshold == 0.45 + + with pytest.raises(ValidationError): + TrainingConfig(echo_trap_threshold=-0.1) + with pytest.raises(ValidationError): + TrainingConfig(echo_trap_threshold=1.5) + + +class TestSchemaSoupConfigTaskGate: + """echo_trap_enabled only meaningful on RL agent tasks (grpo / ppo).""" + + def _yaml(self, task: str = "grpo") -> str: + return f""" +base: meta-llama/Llama-3.1-8B +task: {task} +data: + train: ./data/train.jsonl + format: chatml +training: + echo_trap_enabled: true + echo_trap_threshold: 0.55 +""" + + def test_grpo_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string(self._yaml("grpo")) + assert cfg.training.echo_trap_enabled is True + + def test_ppo_accepted(self): + from soup_cli.config.loader import load_config_from_string + + cfg = load_config_from_string(self._yaml("ppo")) + assert cfg.training.echo_trap_enabled is True + + def test_sft_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="echo_trap"): + load_config_from_string(self._yaml("sft")) + + def test_halt_without_enabled_rejected(self): + from soup_cli.config.loader import load_config_from_string + + with pytest.raises(ValueError, match="echo_trap_enabled"): + load_config_from_string( + """ +base: meta-llama/Llama-3.1-8B +task: grpo +data: + train: ./data/train.jsonl + format: chatml +training: + echo_trap_halt: true +""" + ) + + +# --------------------------------------------------------------------------- +# Source wiring guards +# --------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_module_no_top_level_torch(self): + from pathlib import Path + + src = ( + Path(__file__).resolve().parent.parent + / "soup_cli" + / "utils" + / "echo_trap.py" + ) + body = src.read_text(encoding="utf-8") + assert "\nimport torch" not in body + assert "\nfrom torch" not in body