diff --git a/CHANGELOG.md b/CHANGELOG.md index b2f69d8..0409c73 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,45 @@ reproducing 70+ versions of notes. ## [Unreleased] +## [0.71.40] - 2026-07-19 + +**`soup reward synth`: auto-generate a deterministic reward verifier from your data.** +Point it at a JSONL of reference (gold) outputs and it infers a verifier, emits a +readable / committable `.py` reward function, and — the moat — *refuses* to emit one +that can't tell your references from auto-generated bad answers. Nothing in +TRL / Unsloth / Axolotl / OpenRLHF synthesizes a reward; every reward today is +hand-written, hand-picked, or a trained-weights artifact. + +### Added + +- **`soup reward synth -o reward.py`** — deterministic verifier + synthesis. Four families, auto-detected (or pick with `--kind`): `numeric` + (last-number / `\boxed{}` / `####` extraction, exact or `--tolerance`), `json_schema` + (induced keys + types + required), `regex` (positional char-classes over + equal-length golds), `tool_call` (per-tool `required`/`allowed` argument binding). + The emitted file is self-contained and rides `load_reward_fn`'s existing `.py` + path — no new trusted-exec surface; you read, edit, commit, and diff it. +- **Mandatory calibration report** — the synthesized verifier is loaded back and run + against its own references (must accept ≥90%) and auto-perturbed negatives (must + reject). A degenerate always-accept verifier is **refused** (`exit 2`), never + silently emitted. `--plan-only` reports the induced spec without writing; + `--output-report` persists the calibration JSON. +- **Comma-separated `reward_fn` (`"accuracy,format"`)** now trains — it resolves to a + reward *ensemble* (`GRPOTrainer(reward_funcs=[...])`, and unlocks the `rm_ensemble` + reward-hack detector which needs ≥2 rewards). GRPO-only, validated at config-parse + time. Fixes a recipe (`deepseek-v3-reasoning`) that shipped exactly this and + previously crashed with `Unknown reward function` (#311). + +### Changed / Fixed + +- `training.reward_fn` gains a field validator (null-byte / blank / oversize / + empty-comma-segment rejection) — the oldest arbitrary-code field was the least + guarded. Comma + `verifiable` without a `verifiable_domain` now fails at parse + time like the bare `verifiable` form. +- `envs/calculator.py` / `envs/guess_number.py` docstrings corrected: the reward is + `reward_fn: verifiable` + `verifiable_domain: math` (the bare `reward_fn='math'` + they showed was never valid). + ## [0.71.39] - 2026-07-19 **"CI for weights, not prompts": close the evidence loop.** `soup ship`'s verdict is diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 54f4106..d696c58 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -120,7 +120,7 @@ src/soup_cli/ templates/ - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (317 files, 16387 tests) +tests/ - Test suite (318 files, 16490 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index d34c657..5a3f899 100644 --- a/README.md +++ b/README.md @@ -49,29 +49,37 @@ infrastructure instead of improving models. Soup fixes that. ## What's New -**v0.71.39 — CI for weights, not prompts.** `soup ship`'s verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — so a fine-tuning gate renders on every PR instead of asking you to hand-edit a JSON file. +**v0.71.40 — `soup reward synth`: generate a reward verifier from your data.** Point it at a JSONL of reference (gold) outputs and it infers a deterministic verifier, writes a readable / committable `.py` reward function, and — the part nobody else does — *refuses* to emit one that can't tell your references from bad answers. Nothing in TRL / Unsloth / Axolotl synthesizes a reward. -- **Output is finally input.** `soup ship --emit-evidence ev.json` writes the scores in the - `--evidence` schema, so a run replays offline into an identical verdict — the round-trip the - old `--output` couldn't do. -- **Commit the gate policy.** Put `eval.ship` in your `soup.yaml` (`forgetting_threshold`, - `general_suite`, `task_mode`, …) and `soup ship --config soup.yaml` reads it — the gate is now - reviewable in a PR diff. Explicit flags still win. -- **Provenance-bound evidence.** `--config --emit-evidence` STAMPS a `config_sha` onto the - evidence; `--config --evidence` GATES on it — a PR that changed the recipe but forgot to - recompute its evidence is refused (exit 3). Tuning the threshold never falsely invalidates it. -- **The verdict on your PR.** `soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card as - a GitHub comment; `soup ci init --config soup.yaml` wires the whole thing into CI. +- **Four families, auto-detected** (or pick with `--kind`): `numeric` (`\boxed{}` / `####` / last-number, + exact or `--tolerance`), `json_schema` (induced keys + types + required), `regex`, and `tool_call` + (per-tool `required`/`allowed` argument binding — a call can't borrow another tool's keys). +- **A calibration report is the moat.** The verifier is loaded back and run against its own references + (must accept ≥ 90%) and auto-generated bad answers (must reject). A degenerate always-accept verifier + is **refused** (exit 2), never silently shipped. `--plan-only` previews; `--output-report` saves the JSON. +- **No new exec surface.** The emitted file rides the existing `reward_fn: reward.py` path — you read, + edit, commit, and diff it like any other reward. +- **Reward ensembles now train.** `reward_fn: "accuracy,format"` loads as multiple rewards (and unlocks + the `rm_ensemble` reward-hack detector) — a recipe that shipped exactly this used to crash. (#311) ```bash -# producer (train job): stamp the scores with the config that produced them -soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json +# synth a verifier from reference answers, calibrate it, keep it only if it discriminates +soup reward synth references.jsonl -o reward.py --output-report calib.json -# gate (PR CI): refuse evidence that doesn't match the committed config -soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42 -soup ci init --config soup.yaml # writes .github/workflows/soup-gate.yml +# then train against the reward you just generated +# training: { reward_fn: reward.py } (or an ensemble: reward_fn: "accuracy,format") ``` +
+Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict) + +`soup ship`'s verdict became emittable, committable, and provenance-bound: `--emit-evidence` makes a +run replay into an identical verdict, `eval.ship` in `soup.yaml` + `--config` makes the gate policy +reviewable, and `--config` binds evidence to the exact recipe that produced it (stale evidence → exit 3). +`soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card on the PR. + +
+
Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate) diff --git a/docs/commands.md b/docs/commands.md index 4367ae6..39b79c7 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -182,6 +182,10 @@ soup draft measure ... --min-acceptance 0.6 -o report.json Exit 2 below the flo soup draft distill --target --draft-base --data d.jsonl -o draft/ Distil a DENSE speculative-decoding draft + register it (v0.71.33) soup draft distill ... --steps N --device cpu --force --plan-only Training budget / device / overwrite -o / render the config only soup draft list List local drafts that `soup serve --auto-spec` will pick up (v0.71.33) +soup reward synth refs.jsonl -o reward.py Synthesize a deterministic reward verifier from gold outputs (v0.71.40) +soup reward synth ... --kind numeric|json_schema|regex|tool_call Force a verifier family (default: auto-detect) +soup reward synth ... --plan-only Report the induced spec + calibration plan; write nothing +soup reward synth ... --output-report r.json --min-discrimination 0.5 Save the calibration JSON / set the refusal threshold (exit 0 emit / 2 refuse / 1 error) soup tui Full-screen Textual dashboard (requires [tui] extra) soup train --config soup.yaml --profile Record torch.profiler trace to /profiles/ soup --log-level quiet|normal|verbose|debug Global logging tier (Rich-formatted) diff --git a/docs/training.md b/docs/training.md index 1cdd275..13d063b 100644 --- a/docs/training.md +++ b/docs/training.md @@ -866,6 +866,37 @@ training: reward_fn: ./my_reward.py ``` +**Reward ensembles** — list several rewards, comma-separated, and they combine (GRPO only). +This also unlocks the `rm_ensemble` reward-hack detector, which needs ≥ 2 rewards: +```yaml +training: + reward_fn: "accuracy,format" # both are scored every step +``` + +### Synthesize a verifier from your data (`soup reward synth`) + +Don't hand-write a verifier — generate one from reference (gold) outputs. Soup infers a +*deterministic* verifier (numeric / JSON-schema / regex / tool-call), writes a readable, editable +`.py`, and **refuses to emit** one that can't tell your references from auto-generated bad answers +(the mandatory calibration report). The emitted file is a normal `reward_fn: reward.py`. + +```bash +# infer + calibrate + emit (exit 0 kept, 2 refused, 1 error) +soup reward synth references.jsonl -o reward.py --output-report calib.json + +# preview the induced spec without writing anything +soup reward synth references.jsonl --plan-only + +# force a family instead of auto-detecting +soup reward synth answers.jsonl -o reward.py --kind numeric --tolerance 1e-6 +``` + +References are a JSONL where each row's gold answer is in an `answer` field (override with +`--field`) or the last assistant turn of a `messages` list. `--min-discrimination` sets how +strongly the verifier must separate references from perturbed negatives before it's emitted. +v1 is deterministic families only — a `\boxed{}`/`####` marker helps the numeric verifier, and +completions are prompted to mark their answer (standard RLVR practice). + ### Verifiable Rewards (RLVR) Use `reward_fn: verifiable` with a `verifiable_domain` for deterministic, math-checkable rewards — no judge model, no heuristics. Great for GRPO on math, code, or structured-output tasks. diff --git a/pyproject.toml b/pyproject.toml index af0f462..33904b4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.71.39" +version = "0.71.40" description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/src/soup_cli/__init__.py b/src/soup_cli/__init__.py index 9123fae..f93f73a 100644 --- a/src/soup_cli/__init__.py +++ b/src/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune and post-train LLMs in one command.""" -__version__ = "0.71.39" +__version__ = "0.71.40" diff --git a/src/soup_cli/cli.py b/src/soup_cli/cli.py index af1097f..1d8f7e1 100644 --- a/src/soup_cli/cli.py +++ b/src/soup_cli/cli.py @@ -563,6 +563,18 @@ app.add_typer( ), ) +# v0.71.40 — `soup reward synth` auto-generate a deterministic verifier. +from soup_cli.commands import reward as _reward_cmd # noqa: E402 + +app.add_typer( + _reward_cmd.app, + name="reward", + help=( + "Synthesize a deterministic reward verifier from reference outputs, with " + "a calibration report that refuses degenerate verifiers (v0.71.40)." + ), +) + def _rewrite_advise_argv(argv: list) -> list: """Inject `run` between `advise` and a non-subcommand first argument. diff --git a/src/soup_cli/commands/reward.py b/src/soup_cli/commands/reward.py new file mode 100644 index 0000000..1913258 --- /dev/null +++ b/src/soup_cli/commands/reward.py @@ -0,0 +1,225 @@ +"""soup reward — synthesize a deterministic reward verifier (v0.71.40). + + soup reward synth -o reward.py + [--kind auto|numeric|json_schema|regex|tool_call] + +Infers a deterministic verifier from a dataset of reference (gold) outputs, emits a +readable / committable ``.py`` that rides ``load_reward_fn``'s existing ``.py`` path, +and REFUSES to emit a degenerate verifier via a mandatory calibration report. + +Exit codes mirror ``soup ship`` / ``soup shrink``: 0 = emitted, 2 = refused (the +verifier could not discriminate references from perturbed negatives), 1 = usage / +runtime error. +""" + +from __future__ import annotations + +import json +import math +import os +from typing import NoReturn, Optional + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +from soup_cli.utils import reward_synth as rs +from soup_cli.utils.paths import atomic_write_text, enforce_under_cwd_and_no_symlink + +app = typer.Typer(help="Synthesize a deterministic reward verifier from reference outputs.") +console = Console() + +_MAX_INPUT_BYTES = 64 * 1024 * 1024 +_MAX_ROWS = 1_000_000 +_ALLOWED_KINDS = ("auto",) + rs.KINDS + + +@app.callback() +def _reward() -> None: + """Reward-verifier tooling. Forces ``synth`` to be a named subcommand (a + single-command Typer app otherwise collapses and eats the subcommand token).""" + + +def _fail(message: str, code: int = 1) -> NoReturn: + """Print a red error and exit (raises internally — a forgotten ``raise`` at a + call site can never silently become a no-op; mirrors ``commands/ship.py``).""" + console.print(f"[red]{escape(message)}[/]") + raise typer.Exit(code) + + +def _read_jsonl(path: str, label: str) -> list[dict]: + """Read a JSONL file: cwd-contained, O_NOFOLLOW, size- and row-capped.""" + enforce_under_cwd_and_no_symlink(path, label) + flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) + try: + fd = os.open(path, flags) + except OSError as exc: + raise ValueError(f"{label} unreadable: {exc}") from exc + rows: list[dict] = [] + with os.fdopen(fd, "r", encoding="utf-8") as handle: + if os.fstat(handle.fileno()).st_size > _MAX_INPUT_BYTES: + raise ValueError(f"{label} exceeds {_MAX_INPUT_BYTES} bytes") + for line in handle: + line = line.strip() + if not line: + continue + try: + row = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(row, dict): + rows.append(row) + if len(rows) >= _MAX_ROWS: + break + return rows + + +def _spec_summary(kind: str, spec: object) -> str: + """One-line human summary of the induced spec (for --plan-only / the panel).""" + if kind == "numeric" and isinstance(spec, rs.NumericSpec): + return f"numeric (float={spec.is_float}, tolerance={spec.tolerance})" + if kind == "json_schema" and isinstance(spec, dict): + props = ", ".join(sorted(spec.get("properties", {}))) or "(none)" + return f"json_schema (type={spec.get('type')}, keys={props})" + if kind == "tool_call" and isinstance(spec, rs.ToolCallSpec): + return f"tool_call (names={list(spec.names)}, arg_keys={list(spec.arg_keys)})" + if kind == "regex": + return f"regex (pattern={spec})" + return kind + + +def _render_report_panel(report: rs.CalibrationReport, kind: str, out_path: str) -> Panel: + table = Table(show_header=False, box=None, pad_edge=False) + table.add_column(justify="right", style="dim") + table.add_column() + table.add_row("verifier kind", escape(kind)) + table.add_row("references (accepted)", f"{report.positives} ({report.pos_accept:.0%})") + table.add_row("negatives (accepted)", f"{report.negatives} ({report.neg_accept:.0%})") + table.add_row("discrimination", f"{report.discrimination:.2f}") + table.add_row("precision", f"{report.precision:.2f}") + table.add_row("emitted", escape(out_path)) + return Panel(table, title="[bold green]reward verifier synthesized[/]", + border_style="green") + + +@app.command() +def synth( + references: str = typer.Argument( + ..., help="JSONL of reference (gold) outputs — a gold field or chat messages." + ), + output: Optional[str] = typer.Option( + None, "-o", "--output", help="Where to write the verifier .py." + ), + kind: str = typer.Option( + "auto", "--kind", + help="Verifier family: auto | numeric | json_schema | regex | tool_call.", + ), + field: str = typer.Option( + "answer", "--field", help="Gold-output field (default: answer)." + ), + tolerance: Optional[float] = typer.Option( + None, "--tolerance", help="Numeric match tolerance (numeric kind only)." + ), + min_discrimination: float = typer.Option( + rs.DEFAULT_MIN_DISCRIMINATION, "--min-discrimination", + help="Refuse to emit unless (accept_rate refs - accept_rate negatives) >= this.", + ), + force: bool = typer.Option(False, "--force", help="Overwrite an existing output."), + plan_only: bool = typer.Option( + False, "--plan-only", help="Detect + report the induced spec; write nothing." + ), + output_report: Optional[str] = typer.Option( + None, "--output-report", help="Also write the calibration report as JSON." + ), +) -> None: + """Synthesize a deterministic reward verifier from reference outputs.""" + if kind not in _ALLOWED_KINDS: + _fail(f"unknown --kind {kind!r}; options: {', '.join(_ALLOWED_KINDS)}") + if not 0.0 <= min_discrimination <= 1.0: + _fail("--min-discrimination must be in [0.0, 1.0]") + if tolerance is not None and (not math.isfinite(tolerance) or tolerance < 0): + _fail("--tolerance must be a finite, non-negative number") + + # Read references. + try: + rows = _read_jsonl(references, "references path") + except (ValueError, OSError) as exc: + _fail(str(exc)) + if not rows: + _fail("references file has no JSON-object rows") + + # Detect + induce + render (no file written yet). + try: + result = rs.synthesize(rows, field=field, kind=kind, tolerance=tolerance, + rel_hint=(output or "reward.py")) + except (ValueError, TypeError) as exc: + _fail(str(exc)) + + if plan_only: + console.print(Panel( + escape(_spec_summary(result.kind, result.spec)), + title=f"[bold]plan: {escape(result.kind)} verifier[/]", border_style="cyan")) + raise typer.Exit(0) + + # Output guards. + if not output: + _fail("must pass -o/--output to emit a verifier (or use --plan-only)") + if not output.endswith(".py"): + _fail(f"output must be a .py file, got {output!r}") + try: + enforce_under_cwd_and_no_symlink(output, "output path") + except (ValueError, OSError) as exc: + _fail(str(exc)) + if os.path.exists(output) and not force: + _fail(f"{output!r} already exists — pass --force to overwrite") + # Validate the optional report path UP FRONT so a bad report path can never + # cause us to destroy an already-written, already-accepted verifier later. + if output_report: + try: + enforce_under_cwd_and_no_symlink(output_report, "report path") + except (ValueError, OSError) as exc: + _fail(str(exc)) + + # Write, then LOAD it back through the real reward-loader path (round-trip + # validation) and calibrate the loaded callable. + atomic_write_text(result.source, output) + try: + from soup_cli.trainer.rewards import load_reward_fn + reward_fn = load_reward_fn(output) + golds = rs.extract_golds(rows, field=field) + negatives = rs.perturb_negatives(golds, result.kind) + report = rs.calibrate(reward_fn, golds, negatives, kind=result.kind, + min_discrimination=min_discrimination) + except Exception as exc: # noqa: BLE001 — clean up the partial artifact + _cleanup(output) + _fail(f"calibration failed: {exc}") + + # Write the diagnostic report (path already validated up front) for BOTH the + # accepted and refused cases. Best-effort: a write failure here warns but must + # not delete an otherwise-valid verifier. + if output_report: + try: + from dataclasses import asdict + atomic_write_text(json.dumps(asdict(report), indent=2), output_report) + except OSError as exc: + console.print(f"[yellow]Warning: could not write report: {escape(str(exc))}[/]") + + if report.refused: + _cleanup(output) + console.print(Panel( + escape(report.reason), + title="[bold red]verifier refused (not emitted)[/]", border_style="red")) + raise typer.Exit(2) + + console.print(_render_report_panel(report, result.kind, output)) + raise typer.Exit(0) + + +def _cleanup(path: str) -> None: + """Remove a just-written artifact; best-effort (never masks the real error).""" + try: + os.remove(path) + except OSError: + pass diff --git a/src/soup_cli/config/schema.py b/src/soup_cli/config/schema.py index 060a545..1283de9 100644 --- a/src/soup_cli/config/schema.py +++ b/src/soup_cli/config/schema.py @@ -1107,10 +1107,38 @@ class TrainingConfig(BaseModel): reward_fn: Optional[str] = Field( default="accuracy", description=( - "Reward function: 'accuracy', 'format', 'verifiable', " - "or path to custom .py file" + "Reward function: 'accuracy', 'format', 'verifiable', a path to a " + "custom .py file, or a comma-separated ensemble of the above " + "(e.g. 'accuracy,format') — the comma form is GRPO-only (v0.71.40)." ), ) + @field_validator("reward_fn", mode="before") + @classmethod + def _validate_reward_fn_field(cls, value: Any) -> Optional[str]: + """v0.71.40 #311 — shape-only validation (containment enforced at load). + + Accepts a comma-separated spec ("accuracy,format"); rejects null bytes, + oversize, and empty comma segments so a stray comma fails loud rather + than silently dropping a reward. + """ + if value is None: + return None + if isinstance(value, bool) or not isinstance(value, str): + raise ValueError( + f"reward_fn must be a string, got {type(value).__name__}" + ) + if "\x00" in value: + raise ValueError("reward_fn must not contain null bytes") + if len(value) > 512: + raise ValueError("reward_fn must be <= 512 chars") + if not value.strip(): + raise ValueError("reward_fn must not be blank") + if any(not seg.strip() for seg in value.split(",")): + raise ValueError( + "reward_fn has an empty comma segment — remove the stray comma" + ) + return value + # RLVR — verifiable reward domain (Part C of v0.25.0) verifiable_domain: Optional[Literal["math", "code", "json_schema"]] = Field( default=None, @@ -2978,8 +3006,18 @@ class TrainingConfig(BaseModel): @model_validator(mode="after") def _validate_verifiable_reward(self) -> "TrainingConfig": - """RLVR: reward_fn='verifiable' requires verifiable_domain.""" - if self.reward_fn == "verifiable" and self.verifiable_domain is None: + """RLVR: reward_fn='verifiable' requires verifiable_domain. + + Comma-aware (v0.71.40 #311): ``"accuracy,verifiable"`` must fail at + config-parse time exactly like the bare ``"verifiable"`` form, not only + at trainer construction. + """ + segments = ( + [s.strip() for s in self.reward_fn.split(",")] + if isinstance(self.reward_fn, str) + else [] + ) + if "verifiable" in segments and self.verifiable_domain is None: raise ValueError( "reward_fn='verifiable' requires verifiable_domain " "(one of: math, code, json_schema)" @@ -4662,6 +4700,25 @@ class SoupConfig(BaseModel): ) return self + @model_validator(mode="after") + def _validate_reward_fn_multi_compat(self) -> "SoupConfig": + """v0.71.40 #311 — a comma-separated reward_fn is GRPO-only. + + Only ``trainer/grpo.py::_select_reward_fn`` resolves a multi-reward spec + into a ``reward_funcs=[...]`` list; ``trainer/ppo.py`` (and every other + consumer) calls the single-name loader, so a comma there would silently + pass config validation and then crash at training start with + ``Unknown reward function``. Footgun-reject it up front (mirrors the + online_dpo / asr / lisa task gates). + """ + rf = self.training.reward_fn + if isinstance(rf, str) and "," in rf and self.task != "grpo": + raise ValueError( + f"a comma-separated training.reward_fn (an ensemble, {rf!r}) is " + f"only supported for task='grpo'; got task={self.task!r}" + ) + return self + @model_validator(mode="after") def _validate_asr_compat(self) -> "SoupConfig": """v0.71.32 — ASR (Whisper) gate. diff --git a/src/soup_cli/envs/calculator.py b/src/soup_cli/envs/calculator.py index 888b2c3..39d8c3e 100644 --- a/src/soup_cli/envs/calculator.py +++ b/src/soup_cli/envs/calculator.py @@ -1,7 +1,8 @@ """Calculator tool-use rollout env — v0.71.30. Generates a deterministic set of single-operation arithmetic problems as GRPO -prompt+answer rows. Score with ``reward_fn='math'`` (or ``'accuracy'``). +prompt+answer rows. Score with ``reward_fn: verifiable`` + +``verifiable_domain: math`` (or ``reward_fn: accuracy``). Usage: ``training.rollout_backend='openenv'`` + ``training.rollout_func='soup_cli.envs.calculator:rollout'``. diff --git a/src/soup_cli/envs/guess_number.py b/src/soup_cli/envs/guess_number.py index c44d129..3707e3f 100644 --- a/src/soup_cli/envs/guess_number.py +++ b/src/soup_cli/envs/guess_number.py @@ -2,7 +2,7 @@ Generates deterministic single-shot number-deduction puzzles (constraints that pin a unique integer) as GRPO prompt+answer rows. Score with -``reward_fn='math'`` (or ``'accuracy'``). +``reward_fn: verifiable`` + ``verifiable_domain: math`` (or ``reward_fn: accuracy``). Honesty: this is single-shot *deduction*, not interactive guess-and-feedback — the live openenv contract does not pass the model, so a true guessing loop is diff --git a/src/soup_cli/trainer/grpo.py b/src/soup_cli/trainer/grpo.py index db2f163..31544d6 100644 --- a/src/soup_cli/trainer/grpo.py +++ b/src/soup_cli/trainer/grpo.py @@ -129,7 +129,9 @@ def _read_attr(obj: Any, name: str) -> Any: return getattr(obj, name, None) -def _select_reward_fn(tcfg: TrainingConfig, device: str, trust_remote_code: bool) -> Any: +def _select_reward_fn( + tcfg: TrainingConfig, device: str, trust_remote_code: bool +) -> "Any": # Callable | list[Callable] (a single reward or a comma-split ensemble) """Choose the GRPO reward function (v0.71.30). When ``tcfg.prm_reward`` is set, a trained Soup PRM scores each completion's @@ -142,9 +144,13 @@ def _select_reward_fn(tcfg: TrainingConfig, device: str, trust_remote_code: bool from soup_cli.utils.prm_reward import build_prm_reward_fn return build_prm_reward_fn(tcfg, device, trust_remote_code) - from soup_cli.trainer.rewards import load_reward_fn + # v0.71.40 #311 — ``reward_fn`` may be comma-separated ("accuracy,format"). + # Return a single callable for one reward (back-compat) or a list for several + # (TRL's reward_funcs=[...] + the rm_ensemble detector both accept the list). + from soup_cli.trainer.rewards import load_reward_fns - return load_reward_fn(tcfg.reward_fn, verifiable_domain=tcfg.verifiable_domain) + fns = load_reward_fns(tcfg.reward_fn, verifiable_domain=tcfg.verifiable_domain) + return fns[0] if len(fns) == 1 else fns class GRPOTrainerWrapper: diff --git a/src/soup_cli/trainer/rewards.py b/src/soup_cli/trainer/rewards.py index 35f64dc..26e3e63 100644 --- a/src/soup_cli/trainer/rewards.py +++ b/src/soup_cli/trainer/rewards.py @@ -557,3 +557,44 @@ def load_reward_fn( f"Options: {', '.join(BUILTIN_REWARDS.keys())}, 'verifiable', " f"or path to a .py file" ) + + +def load_reward_fns( + reward_fn_spec: str, verifiable_domain: "str | None" = None, +) -> list[Callable]: + """Load one OR MORE reward functions from a comma-separated spec (v0.71.40 #311). + + ``reward_fn`` may name several rewards, e.g. ``"accuracy,format"`` — each is + resolved via :func:`load_reward_fn` and returned as a list, which TRL's + ``GRPOTrainer(reward_funcs=[...])`` accepts and which the ``rm_ensemble`` + reward-hack detector (needs >= 2 reward fns) unlocks. A single name still + returns a one-element list, so callers can always treat the result uniformly. + + Splitting is on ``,``; blank / empty segments raise (``"accuracy,"`` is a typo, + not "accuracy plus nothing"), and duplicate segments raise (they would collide + by ``__name__`` in the ``rm_ensemble`` capture buffer, silently shrinking the + ensemble). A ``.py`` path containing a literal comma is not supported — rename + the file. + """ + if not isinstance(reward_fn_spec, str): + raise ValueError( + "reward_fn must be a string (a name, a .py path, or a comma-separated " + f"list), got {type(reward_fn_spec).__name__}" + ) + if not reward_fn_spec.strip(): + raise ValueError("reward_fn must not be blank") + segments = [seg.strip() for seg in reward_fn_spec.split(",")] + if any(not seg for seg in segments): + raise ValueError( + f"reward_fn {reward_fn_spec!r} has an empty comma segment — " + "remove the stray comma" + ) + seen: set[str] = set() + for seg in segments: + if seg in seen: + raise ValueError( + f"reward_fn {reward_fn_spec!r} lists {seg!r} twice — " + "duplicate rewards collide in the ensemble" + ) + seen.add(seg) + return [load_reward_fn(seg, verifiable_domain=verifiable_domain) for seg in segments] diff --git a/src/soup_cli/utils/reward_synth.py b/src/soup_cli/utils/reward_synth.py new file mode 100644 index 0000000..731f597 --- /dev/null +++ b/src/soup_cli/utils/reward_synth.py @@ -0,0 +1,716 @@ +"""Reward-verifier synthesis — `soup reward synth` (v0.71.40). + +Auto-generate a *deterministic* reward verifier from a dataset of reference (gold) +outputs. The emitted artifact is a readable, editable, committable ``.py`` that rides +:func:`soup_cli.trainer.rewards.load_reward_fn`'s existing ``.py`` path — NO new +trusted-exec hot path. + +Pipeline (all pure, NO top-level torch): + detect_kind → induce_* → render_verifier_py → (load) → perturb_negatives + calibrate. + +v1 verifier families (deterministic only): ``numeric`` · ``json_schema`` · ``regex`` · +``tool_call``. Judge-based and LLM-codegen are explicitly out of scope. + +The calibration report is the moat: a synthesized verifier is only emitted if it +accepts its own references AND rejects auto-perturbed negatives — a degenerate +always-1.0 verifier is refused. +""" + +from __future__ import annotations + +import json +import math +import re +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from typing import Any, Optional, Union + +KINDS: tuple[str, ...] = ("numeric", "json_schema", "regex", "tool_call") +# Auto-detect precedence: most-specific shape wins. +_DETECT_ORDER: tuple[str, ...] = ("tool_call", "json_schema", "numeric", "regex") + +# Fraction of golds that must fit a kind for auto-detection / regex confidence. +_MIN_CONFIDENCE = 0.9 +DEFAULT_NUMERIC_TOLERANCE = 1e-6 +DEFAULT_MIN_DISCRIMINATION = 0.5 +# A verifier must accept at least this fraction of its OWN references, else the +# induced spec plainly does not fit the golds (mirrors _MIN_CONFIDENCE). +_MIN_SELF_ACCEPT = 0.9 +_MAX_ROWS = 1_000_000 + +_NUMBER_RE = re.compile(r"[+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?") +_WHOLE_NUMBER_RE = re.compile(r"^[+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?$") + + +# --------------------------------------------------------------------------- +# Frozen result types +# --------------------------------------------------------------------------- +@dataclass(frozen=True) +class NumericSpec: + is_float: bool + tolerance: float + + +@dataclass(frozen=True) +class ToolCallSpec: + """Per-tool argument signatures. + + ``tools`` maps each tool name to ``{"required": (...), "allowed": (...)}`` — + ``required`` = keys present in EVERY observed call of that name (intersection), + ``allowed`` = union of keys seen for that name. The verifier accepts a call iff + ``required <= call_args <= allowed`` for the matching name, so a call cannot + borrow another tool's argument keys or omit required ones. + """ + + tools: dict # name -> {"required": tuple[str, ...], "allowed": tuple[str, ...]} + + @property + def names(self) -> tuple[str, ...]: + return tuple(sorted(self.tools)) + + @property + def arg_keys(self) -> tuple[str, ...]: + keys: set[str] = set() + for sig in self.tools.values(): + keys.update(sig.get("allowed", ())) + return tuple(sorted(keys)) + + +@dataclass(frozen=True) +class CalibrationReport: + kind: str + positives: int + negatives: int + pos_accept: float + neg_accept: float + discrimination: float + precision: float + recall: float + refused: bool + reason: str + + +SpecType = Union[NumericSpec, dict, ToolCallSpec, str] + + +@dataclass(frozen=True) +class SynthResult: + kind: str + source: str + spec: SpecType # NumericSpec | dict | ToolCallSpec | str (pattern) + report: Optional[CalibrationReport] = None + + +# --------------------------------------------------------------------------- +# Gold extraction +# --------------------------------------------------------------------------- +def _row_gold(row: Mapping[str, Any], field: str) -> Optional[str]: + if not isinstance(row, Mapping): + return None + if field in row and row[field] is not None: + value = row[field] + if isinstance(value, str): + return value + if isinstance(value, (dict, list)): + return json.dumps(value) + return str(value) # numbers, bools -> their str form + messages = row.get("messages") + if isinstance(messages, list): + for msg in reversed(messages): + if isinstance(msg, Mapping) and msg.get("role") == "assistant": + content = msg.get("content") + if isinstance(content, str): + return content + return None + + +def extract_golds(rows: Sequence[Mapping[str, Any]], *, field: str = "answer") -> list[str]: + """Pull the gold output from each row (``field`` else chat last-assistant). + + Raises ``TypeError`` if ``rows`` is not a sequence of mappings; ``ValueError`` + if no row yields a gold. + """ + if not isinstance(rows, Sequence) or isinstance(rows, (str, bytes)): + raise TypeError("rows must be a sequence of mapping rows") + if len(rows) > _MAX_ROWS: + raise ValueError(f"rows exceed cap of {_MAX_ROWS}") + golds: list[str] = [] + for row in rows: + gold = _row_gold(row, field) + if gold is not None: + golds.append(gold) + if not golds: + raise ValueError( + f"no gold outputs found (field={field!r}; rows need that key or a " + "chat 'messages' list with an assistant turn)" + ) + return golds + + +# --------------------------------------------------------------------------- +# Kind primitives +# --------------------------------------------------------------------------- +def _is_number(text: str) -> bool: + return bool(_WHOLE_NUMBER_RE.match(text.strip())) + + +def _json_or_none(text: str) -> Any: + try: + return json.loads(text.strip()) + except (json.JSONDecodeError, ValueError): + return None + + +def _is_tool_call(obj: Any) -> bool: + return ( + isinstance(obj, dict) + and isinstance(obj.get("name"), str) + and isinstance(obj.get("arguments"), dict) + ) + + +def _fraction(golds: Sequence[str], predicate: Callable[[str], bool]) -> float: + if not golds: + return 0.0 + return sum(1 for g in golds if predicate(g)) / len(golds) + + +def detect_kind(golds: Sequence[str]) -> Optional[str]: + """Infer the verifier family from the golds, or ``None`` if un-inferrable. + + Precedence: tool_call → json_schema → numeric → regex. + """ + if not golds: + return None + for kind in _DETECT_ORDER: + if kind == "tool_call": + frac = _fraction(golds, lambda g: _is_tool_call(_json_or_none(g))) + elif kind == "json_schema": + frac = _fraction( + golds, lambda g: isinstance(_json_or_none(g), (dict, list)) + ) + elif kind == "numeric": + frac = _fraction(golds, _is_number) + else: # regex + frac = 1.0 if induce_regex(golds) is not None else 0.0 + if frac >= _MIN_CONFIDENCE: + return kind + return None + + +# --------------------------------------------------------------------------- +# Inducers +# --------------------------------------------------------------------------- +def induce_numeric( + golds: Sequence[str], *, tolerance: Optional[float] = None +) -> NumericSpec: + if _fraction(golds, _is_number) < _MIN_CONFIDENCE: + raise ValueError( + "references are not numeric — cannot induce a numeric verifier " + "(each gold must be a bare number); pass a different --kind" + ) + is_float = any(("." in g) or ("e" in g.lower()) for g in golds) + if tolerance is None: + tol = DEFAULT_NUMERIC_TOLERANCE if is_float else 0.0 + else: + tol = float(tolerance) + if not math.isfinite(tol) or tol < 0: + raise ValueError(f"tolerance must be a finite, non-negative number, got {tol!r}") + return NumericSpec(is_float=is_float, tolerance=tol) + + +def _json_type(value: Any) -> str: + if isinstance(value, bool): + return "boolean" + if isinstance(value, int): + return "integer" + if isinstance(value, float): + return "number" + if isinstance(value, str): + return "string" + if isinstance(value, list): + return "array" + if isinstance(value, dict): + return "object" + return "null" + + +def induce_json_schema(golds: Sequence[str]) -> dict: + """Induce a top-level JSON schema (keys + types + required) from the golds. + + Refuses (``ValueError``) rather than emitting a wrong schema when the golds + mix top-level shapes (some object, some array) or parse to no JSON container + at all — mirroring :func:`induce_regex`'s refuse-don't-guess policy. A key + whose type is inconsistent across rows is left un-typed (presence only), so + an over-narrow ``type`` can never reject a valid completion. + """ + parsed = [_json_or_none(g) for g in golds] + containers = [o for o in parsed if isinstance(o, (dict, list))] + if not containers: + raise ValueError( + "no JSON object/array references found — cannot induce a json_schema" + ) + dicts = [o for o in containers if isinstance(o, dict)] + lists = [o for o in containers if isinstance(o, list)] + if dicts and lists: + raise ValueError( + "references mix JSON objects and arrays — split them or pass an " + "explicit --kind; refusing to induce an ambiguous schema" + ) + if lists: + return {"type": "array"} + key_types: dict[str, set[str]] = {} + key_counts: dict[str, int] = {} + for obj in dicts: + for key, value in obj.items(): + key_counts[key] = key_counts.get(key, 0) + 1 + key_types.setdefault(key, set()).add(_json_type(value)) + properties: dict[str, dict] = {} + for key, types in key_types.items(): + # Only constrain the type when EVERY row agreed on it; else presence-only. + properties[key] = {"type": next(iter(types))} if len(types) == 1 else {} + required = sorted(k for k, c in key_counts.items() if c == len(dicts)) + return {"type": "object", "properties": properties, "required": required} + + +def induce_tool_call(golds: Sequence[str]) -> ToolCallSpec: + """Induce per-tool argument signatures (name -> required/allowed keys).""" + per_name: dict[str, list[set]] = {} + for gold in golds: + obj = _json_or_none(gold) + if _is_tool_call(obj): + per_name.setdefault(obj["name"], []).append( + {str(k) for k in obj["arguments"].keys()} + ) + if not per_name: + raise ValueError( + "no tool-call references found (each needs a JSON object with string " + "'name' + object 'arguments') — cannot induce a tool_call verifier" + ) + tools: dict = {} + for name, calls in per_name.items(): + allowed = set().union(*calls) + required = set.intersection(*calls) + tools[name] = { + "required": tuple(sorted(required)), + "allowed": tuple(sorted(allowed)), + } + return ToolCallSpec(tools=tools) + + +def _char_class(chars: set[str]) -> Optional[str]: + if len(chars) == 1: + return re.escape(next(iter(chars))) + if all(c.isdigit() for c in chars): + return r"\d" + if all(c.isalpha() for c in chars): + return r"[A-Za-z]" + if all(c.isalnum() for c in chars): + return r"[A-Za-z0-9]" + return None + + +def induce_regex(golds: Sequence[str]) -> Optional[str]: + """Conservative positional pattern induction over equal-length golds. + + Returns ``None`` unless the golds share a length and every position + generalizes to a confident char class (the safe direction: refuse rather + than emit a loose pattern). + """ + golds = [g.strip() for g in golds if g.strip()] + if len(golds) < 2: + return None + length = len(golds[0]) + if any(len(g) != length for g in golds) or length == 0: + return None + parts: list[str] = [] + for pos in range(length): + cls = _char_class({g[pos] for g in golds}) + if cls is None: + return None + parts.append(cls) + # By construction every gold matches (each position's class is a superset of + # the observed chars), so no post-hoc confidence check is needed. + return "^" + "".join(parts) + "$" + + +# --------------------------------------------------------------------------- +# Source rendering +# --------------------------------------------------------------------------- +_HEADER = '''"""Deterministic reward verifier — generated by `soup reward synth` (v0.71.40). + +Kind: {kind}. Induced from {n_refs} reference output(s). EDIT FREELY — this is a +self-contained, committable reward function. Wire it via: + + training: + reward_fn: {rel_hint} + +Signature: reward_fn(completions, **kwargs) -> list[float]; the per-row gold answer +arrives as kwargs["answer"] (a list aligned with completions), matching TRL/GRPO. +""" + +import json +import re + + +def _last_content(completion): + return completion[-1]["content"] if completion else "" +''' + +_NUMERIC_BODY = ''' +_NUMBER_RE = re.compile(r"[+-]?(?:\\d+\\.?\\d*|\\.\\d+)(?:[eE][+-]?\\d+)?") +_BOXED_RE = re.compile(r"\\\\boxed\\{([^}]*)\\}") + + +def _extract_number(text): + # Prefer an explicit answer marker (\\boxed{...} then ####) over "last number + # in the whole string", so trailing chit-chat numbers do not hijack the score. + boxed = _BOXED_RE.search(text) + if boxed: + nums = _NUMBER_RE.findall(boxed.group(1)) + if nums: + return nums[-1] + if "####" in text: + nums = _NUMBER_RE.findall(text.split("####")[-1]) + if nums: + return nums[-1] + nums = _NUMBER_RE.findall(text) + return nums[-1] if nums else None + + +def _numbers_match(pred, gold, tol): + if pred is None: + return False + try: + if tol == 0: + # Exact: integer compare first so large ints dodge float rounding. + try: + return int(pred) == int(gold) + except (ValueError, TypeError): + pass + return abs(float(pred) - float(gold)) <= tol + except (ValueError, TypeError): + return False + + +def reward_fn(completions, **kwargs): + answers = kwargs.get("answer", []) + out = [] + for completion, expected in zip(completions, answers): + predicted = _extract_number(_last_content(completion)) + gold = str(expected).strip() + out.append(1.0 if _numbers_match(predicted, gold, _TOLERANCE) else 0.0) + return out +''' + +_JSON_SCHEMA_BODY = ''' + +def _json_type(value): + if isinstance(value, bool): + return "boolean" + if isinstance(value, int): + return "integer" + if isinstance(value, float): + return "number" + if isinstance(value, str): + return "string" + if isinstance(value, list): + return "array" + if isinstance(value, dict): + return "object" + return "null" + + +def _matches_schema(data): + expected = _SCHEMA.get("type") + if expected == "array": + return isinstance(data, list) + if expected == "object": + if not isinstance(data, dict): + return False + for key in _SCHEMA.get("required", []): + if key not in data: + return False + props = _SCHEMA.get("properties", {}) + _NUMERIC = {"integer", "number"} + for key, spec in props.items(): + if key in data: + want = spec.get("type") + got = _json_type(data[key]) + if not want: + continue # presence-only (type varied across references) + if want in _NUMERIC and got in _NUMERIC: + continue # int/float interchangeable + if want != got: + return False + return True + return True + + +def reward_fn(completions, **kwargs): + out = [] + for completion in completions: + content = _last_content(completion) + fenced = re.search(r"```(?:json)?\\s*(.*?)```", content, re.DOTALL) + if fenced: + content = fenced.group(1) + try: + data = json.loads(content.strip()) + except (json.JSONDecodeError, ValueError): + out.append(0.0) + continue + out.append(1.0 if _matches_schema(data) else 0.0) + return out +''' + +_REGEX_BODY = ''' + +def reward_fn(completions, **kwargs): + out = [] + for completion in completions: + content = _last_content(completion).strip() + out.append(1.0 if _PATTERN.fullmatch(content) else 0.0) + return out +''' + +_TOOL_CALL_BODY = ''' + +def reward_fn(completions, **kwargs): + out = [] + for completion in completions: + content = _last_content(completion) + fenced = re.search(r"```(?:json)?\\s*(.*?)```", content, re.DOTALL) + if fenced: + content = fenced.group(1) + try: + data = json.loads(content.strip()) + except (json.JSONDecodeError, ValueError): + out.append(0.0) + continue + if not (isinstance(data, dict) and isinstance(data.get("arguments"), dict)): + out.append(0.0) + continue + sig = _TOOLS.get(data.get("name")) + if sig is None: + out.append(0.0) + continue + keys = set(data["arguments"].keys()) + required = set(sig["required"]) + allowed = set(sig["allowed"]) + out.append(1.0 if required <= keys <= allowed else 0.0) + return out +''' + + +def _safe_hint(value: object) -> str: + """One-line, quote/backslash-free display string for the header docstring. + + ``rel_hint`` (the ``-o`` path) is the only value ``.format``'d RAW into the + generated source (everything else is baked via ``repr``). The file is later + exec'd, so a path containing ``\"\"\"`` + code must not be able to break out + of the docstring — strip control chars, quotes and backslashes and cap it. + """ + text = str(value) + text = "".join(ch for ch in text if ch >= " " and ch not in '"\\') + return text[:200] or "reward.py" + + +def render_verifier_py(kind: str, spec: Any, *, meta: Mapping[str, Any]) -> str: + """Render a self-contained reward-verifier ``.py`` source string. + + The induced spec is baked in as a top-level constant via ``repr`` (so the + code bodies stay literal — no ``str.format`` over code containing braces). + """ + n_refs = int(meta.get("n_refs", 0)) + rel_hint = _safe_hint(meta.get("rel_hint", "reward.py")) + header = _HEADER.format(kind=kind, n_refs=n_refs, rel_hint=rel_hint) + if kind == "numeric": + if not isinstance(spec, NumericSpec): + raise TypeError("numeric verifier requires a NumericSpec") + const = f"\n_TOLERANCE = {spec.tolerance!r}\n" + body = const + _NUMERIC_BODY + elif kind == "json_schema": + if not isinstance(spec, dict): + raise TypeError("json_schema verifier requires a schema dict") + const = f"\n_SCHEMA = {spec!r}\n" + body = const + _JSON_SCHEMA_BODY + elif kind == "regex": + if not isinstance(spec, str): + raise TypeError("regex verifier requires a pattern string") + const = f"\n_PATTERN = re.compile({spec!r})\n" + body = const + _REGEX_BODY + elif kind == "tool_call": + if not isinstance(spec, ToolCallSpec): + raise TypeError("tool_call verifier requires a ToolCallSpec") + const = f"\n_TOOLS = {spec.tools!r}\n" + body = const + _TOOL_CALL_BODY + else: + raise ValueError(f"unknown verifier kind: {kind!r} (options: {', '.join(KINDS)})") + return header + body + + +# --------------------------------------------------------------------------- +# Negatives + calibration +# --------------------------------------------------------------------------- +def perturb_negatives(golds: Sequence[str], kind: str) -> list[str]: + """Deterministically corrupt golds into known-bad outputs for calibration.""" + negatives: list[str] = [] + for gold in golds: + gold = gold.strip() + if kind == "numeric": + num = _NUMBER_RE.findall(gold) + if num: + try: + negatives.append(str(float(num[-1]) + 9999.0)) + except ValueError: + negatives.append("not-a-number") + else: + negatives.append("not-a-number") + elif kind == "json_schema": + negatives.append("this is definitely not json {") + elif kind == "tool_call": + negatives.append("this is definitely not json {") + obj = _json_or_none(gold) + if _is_tool_call(obj): + # Valid JSON, WRONG tool name — a "just parse JSON" verifier would + # accept this; a name-bound verifier must reject it. + negatives.append( + json.dumps({"name": "__nonexistent_tool__", + "arguments": obj["arguments"]}) + ) + # Valid JSON, right name, a foreign arg key outside `allowed`. + negatives.append( + json.dumps({"name": obj["name"], + "arguments": {"__foreign_arg__": 1}}) + ) + elif kind == "regex": + negatives.append(gold + "ZZZ_definitely_wrong") + else: + raise ValueError(f"unknown kind: {kind!r} (options: {', '.join(KINDS)})") + # Always include a couple of universal degenerate cases. + negatives.extend(["", "the model rambled without answering at all"]) + return negatives + + +def calibrate( + reward_fn: Callable[..., list[float]], + positives: Sequence[str], + negatives: Sequence[str], + *, + kind: str = "", + min_discrimination: float = DEFAULT_MIN_DISCRIMINATION, +) -> CalibrationReport: + """Run ``reward_fn`` over positives (should accept) + negatives (should reject). + + ``discrimination = pos_accept - neg_accept``. Refuses when: (a) the verifier + fails to accept ``>= _MIN_SELF_ACCEPT`` of its own references (the induced + spec does not fit the golds), (b) ``discrimination <= 0`` (no separation at + all — a hard floor the user cannot disable via ``--min-discrimination 0``), + or (c) ``discrimination < min_discrimination``. + """ + def _accept_rate( + items: Sequence[str], answers: Sequence[str] + ) -> tuple[float, int]: + if not items: + return 0.0, 0 + completions = [[{"role": "assistant", "content": s}] for s in items] + scores = reward_fn(completions, answer=list(answers)) + accepted = sum(1 for s in scores if float(s) >= 0.5) + return accepted / len(items), accepted + + # Positives are scored as content==gold. Negatives are corrupted completions + # but must be scored against the REAL gold (cycled), else a gold-comparison + # verifier would accept a corrupted answer matched against itself. + pos_accept, tp = _accept_rate(positives, positives) + if positives: + neg_answers = [positives[j % len(positives)] for j in range(len(negatives))] + else: + neg_answers = list(negatives) + neg_accept, fp = _accept_rate(negatives, neg_answers) + discrimination = pos_accept - neg_accept + precision = (tp / (tp + fp)) if (tp + fp) > 0 else 0.0 + recall = pos_accept + refused = False + reason = "verifier discriminates references from perturbed negatives" + if pos_accept < _MIN_SELF_ACCEPT: + refused = True + reason = ( + f"verifier rejects its own references (accepts only " + f"{pos_accept:.0%}, need >= {_MIN_SELF_ACCEPT:.0%}) — the induced " + "spec does not fit the golds" + ) + elif discrimination <= 0.0: + refused = True + reason = ( + f"verifier accepts perturbed negatives as readily as references " + f"(accepts {pos_accept:.0%} of refs and {neg_accept:.0%} of negatives, " + "discrimination <= 0) — degenerate, refusing regardless of " + "--min-discrimination" + ) + elif discrimination < min_discrimination: + refused = True + reason = ( + f"discrimination {discrimination:.2f} < required {min_discrimination:.2f} " + f"(accepts {pos_accept:.0%} of references but also {neg_accept:.0%} of " + "perturbed negatives) — the verifier is too permissive to reward-train on" + ) + return CalibrationReport( + kind=kind, + positives=len(positives), + negatives=len(negatives), + pos_accept=pos_accept, + neg_accept=neg_accept, + discrimination=discrimination, + precision=precision, + recall=recall, + refused=refused, + reason=reason, + ) + + +# --------------------------------------------------------------------------- +# Orchestration (pure — the CLI does write→load→calibrate) +# --------------------------------------------------------------------------- +def _induce(kind: str, golds: Sequence[str], tolerance: Optional[float]) -> Any: + if kind == "numeric": + return induce_numeric(golds, tolerance=tolerance) + if kind == "json_schema": + return induce_json_schema(golds) + if kind == "tool_call": + return induce_tool_call(golds) + if kind == "regex": + pattern = induce_regex(golds) + if pattern is None: + raise ValueError( + "could not induce a confident regex from the references " + "(golds must share a length + positional char classes)" + ) + return pattern + raise ValueError(f"unknown kind: {kind!r} (options: {', '.join(KINDS)})") + + +def synthesize( + rows: Sequence[Mapping[str, Any]], + *, + field: str = "answer", + kind: str = "auto", + tolerance: Optional[float] = None, + rel_hint: str = "reward.py", +) -> SynthResult: + """Detect + induce + render (no calibration — the CLI loads then calibrates).""" + if kind != "auto" and kind not in KINDS: + raise ValueError(f"unknown kind: {kind!r} (options: auto, {', '.join(KINDS)})") + golds = extract_golds(rows, field=field) + resolved = kind + if kind == "auto": + resolved = detect_kind(golds) + if resolved is None: + raise ValueError( + "could not auto-detect a deterministic verifier kind from the " + "references — pass --kind numeric|json_schema|regex|tool_call" + ) + spec = _induce(resolved, golds, tolerance) + source = render_verifier_py( + resolved, spec, meta={"n_refs": len(golds), "rel_hint": rel_hint} + ) + return SynthResult(kind=resolved, source=source, spec=spec, report=None) diff --git a/tests/test_v07140.py b/tests/test_v07140.py new file mode 100644 index 0000000..4a152ab --- /dev/null +++ b/tests/test_v07140.py @@ -0,0 +1,855 @@ +"""Tests for v0.71.40 — `soup reward synth` + comma-split reward loader (#311). + +Covers: +- utils/reward_synth.py: extract_golds / detect_kind / inducers / render / calibrate / synthesize. +- commands/reward.py: `soup reward synth` CLI (happy / refuse / plan-only / guards). +- trainer/rewards.load_reward_fns + grpo._select_reward_fn comma-split (#311). +- config reward_fn validator; envs docstring fix. +""" + +from __future__ import annotations + +import ast +import importlib.util +from pathlib import Path + +import pytest + +from soup_cli.utils import reward_synth as rs + + +# --------------------------------------------------------------------------- +# extract_golds +# --------------------------------------------------------------------------- +class TestExtractGolds: + def test_answer_field(self): + rows = [{"prompt": "q", "answer": "5"}, {"prompt": "q2", "answer": 7}] + assert rs.extract_golds(rows, field="answer") == ["5", "7"] + + def test_chat_last_assistant(self): + rows = [ + {"messages": [{"role": "user", "content": "hi"}, + {"role": "assistant", "content": "gold-out"}]} + ] + assert rs.extract_golds(rows, field="answer") == ["gold-out"] + + def test_dict_gold_is_json_serialised(self): + rows = [{"answer": {"k": 1}}] + assert rs.extract_golds(rows, field="answer") == ['{"k": 1}'] + + def test_no_resolvable_gold_raises(self): + with pytest.raises(ValueError, match="no gold"): + rs.extract_golds([{"prompt": "q"}], field="answer") + + def test_rows_must_be_sequence(self): + with pytest.raises(TypeError): + rs.extract_golds("not-a-list", field="answer") + + +# --------------------------------------------------------------------------- +# detect_kind +# --------------------------------------------------------------------------- +class TestDetectKind: + def test_numeric(self): + assert rs.detect_kind(["5", "12", "-3"]) == "numeric" + + def test_json_schema(self): + assert rs.detect_kind(['{"a": 1}', '{"a": 2, "b": 3}']) == "json_schema" + + def test_tool_call_precedes_json_schema(self): + golds = ['{"name": "search", "arguments": {"q": "x"}}', + '{"name": "lookup", "arguments": {"id": 3}}'] + assert rs.detect_kind(golds) == "tool_call" + + def test_regex_last_resort(self): + # Same-length structured strings, not numbers, not JSON. + assert rs.detect_kind(["2031-01-02", "1999-12-31"]) == "regex" + + def test_uninferrable_returns_none(self): + assert rs.detect_kind(["hello world", "a completely different sentence here"]) is None + + +# --------------------------------------------------------------------------- +# inducers +# --------------------------------------------------------------------------- +class TestInduceNumeric: + def test_int(self): + spec = rs.induce_numeric(["1", "2", "3"]) + assert spec.is_float is False and spec.tolerance == 0.0 + + def test_float(self): + spec = rs.induce_numeric(["1.5", "2"]) + assert spec.is_float is True and spec.tolerance == rs.DEFAULT_NUMERIC_TOLERANCE + + def test_negative_tolerance_raises(self): + with pytest.raises(ValueError, match="finite"): + rs.induce_numeric(["1.0"], tolerance=-1) + + +class TestInduceJsonSchema: + def test_keys_types_required(self): + schema = rs.induce_json_schema(['{"a": 1, "b": "x"}', '{"a": 2}']) + assert schema["type"] == "object" + assert set(schema["properties"]) == {"a", "b"} + assert schema["properties"]["a"]["type"] == "integer" + assert schema["required"] == ["a"] # b missing from row 2 + + def test_array(self): + schema = rs.induce_json_schema(["[1, 2]", "[3]"]) + assert schema["type"] == "array" + + +class TestInduceToolCall: + def test_names_and_arg_keys(self): + golds = ['{"name": "search", "arguments": {"q": "x"}}', + '{"name": "lookup", "arguments": {"id": 3}}'] + spec = rs.induce_tool_call(golds) + assert set(spec.names) == {"search", "lookup"} + assert set(spec.arg_keys) == {"q", "id"} + + def test_per_tool_required_and_allowed(self): + # search always has q (required); lookup varies id/limit (id required only if + # in every lookup call). allowed = union per name. + golds = ['{"name": "search", "arguments": {"q": "a"}}', + '{"name": "search", "arguments": {"q": "b", "page": 2}}', + '{"name": "lookup", "arguments": {"id": 1}}'] + spec = rs.induce_tool_call(golds) + assert spec.tools["search"]["required"] == ("q",) + assert set(spec.tools["search"]["allowed"]) == {"q", "page"} + assert spec.tools["lookup"]["required"] == ("id",) + + def test_no_tool_calls_raises(self): + with pytest.raises(ValueError, match="no tool-call"): + rs.induce_tool_call(["just some text", "5"]) + + +class TestInduceRegex: + def test_confident(self): + pat = rs.induce_regex(["2031-01-02", "1999-12-31"]) + import re + assert pat is not None and re.fullmatch(pat, "2000-05-06") + + def test_not_confident_returns_none(self): + assert rs.induce_regex(["ab", "abcde", "x"]) is None + + +# --------------------------------------------------------------------------- +# render + round-trip +# --------------------------------------------------------------------------- +def _load_reward_fn(source: str, tmp_path: Path): + p = tmp_path / "gen_reward.py" + p.write_text(source, encoding="utf-8") + spec = importlib.util.spec_from_file_location("gen_reward", p) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod.reward_fn + + +def _completions(*contents): + return [[{"role": "assistant", "content": c}] for c in contents] + + +class TestRenderVerifier: + def test_compiles_and_defines_reward_fn(self, tmp_path): + src = rs.render_verifier_py("numeric", rs.NumericSpec(False, 0.0), meta={"n_refs": 3}) + compile(src, "", "exec") # syntactically valid + fn = _load_reward_fn(src, tmp_path) + assert callable(fn) + + def test_numeric_roundtrip(self, tmp_path): + src = rs.render_verifier_py("numeric", rs.NumericSpec(False, 0.0), meta={"n_refs": 2}) + fn = _load_reward_fn(src, tmp_path) + out = fn(_completions("The answer is 5", "42"), answer=["5", "7"]) + assert out == [1.0, 0.0] + + def test_json_schema_roundtrip(self, tmp_path): + schema = {"type": "object", "properties": {"a": {"type": "integer"}}, "required": ["a"]} + src = rs.render_verifier_py("json_schema", schema, meta={"n_refs": 2}) + fn = _load_reward_fn(src, tmp_path) + out = fn(_completions('{"a": 1}', "not json"), answer=["", ""]) + assert out == [1.0, 0.0] + + def test_tool_call_roundtrip(self, tmp_path): + spec = rs.ToolCallSpec(tools={"search": {"required": ("q",), "allowed": ("q",)}}) + src = rs.render_verifier_py("tool_call", spec, meta={"n_refs": 1}) + fn = _load_reward_fn(src, tmp_path) + out = fn(_completions('{"name": "search", "arguments": {"q": "x"}}', + '{"name": "evil", "arguments": {}}'), answer=["", ""]) + assert out == [1.0, 0.0] + + def test_tool_call_binding_rejects_hacks(self, tmp_path): + # The code-review H1: two tools; a call must not borrow the other's arg key + # or drop a required one. Union-of-all-keys + subset-check would accept both. + spec = rs.induce_tool_call(['{"name": "search", "arguments": {"q": "x"}}', + '{"name": "lookup", "arguments": {"id": 3}}']) + fn = _load_reward_fn(rs.render_verifier_py("tool_call", spec, meta={"n_refs": 2}), + tmp_path) + out = fn(_completions( + '{"name": "search", "arguments": {"q": "y"}}', # ok + '{"name": "search", "arguments": {}}', # missing required q + '{"name": "search", "arguments": {"id": 3}}', # borrowed lookup's key + ), answer=["", "", ""]) + assert out == [1.0, 0.0, 0.0] + + def test_numeric_boxed_beats_trailing_number(self, tmp_path): + # M1: a \boxed answer wins over an incidental trailing number. + src = rs.render_verifier_py("numeric", rs.NumericSpec(False, 0.0), meta={"n_refs": 1}) + fn = _load_reward_fn(src, tmp_path) + out = fn(_completions(r"So \boxed{4}. This took 12 seconds."), answer=["4"]) + assert out == [1.0] + + def test_numeric_bigint_exact(self, tmp_path): + # M1: two 17-digit ints that collide under float() must NOT match at tol 0. + src = rs.render_verifier_py("numeric", rs.NumericSpec(False, 0.0), meta={"n_refs": 1}) + fn = _load_reward_fn(src, tmp_path) + out = fn(_completions("90071992547409915"), answer=["90071992547409916"]) + assert out == [0.0] + + def test_json_schema_int_float_symmetric(self, tmp_path): + # H2: gold key induced as integer; a completion with 5.0 must still match. + schema = {"type": "object", "properties": {"x": {"type": "integer"}}, "required": ["x"]} + fn = _load_reward_fn(rs.render_verifier_py("json_schema", schema, meta={"n_refs": 1}), + tmp_path) + out = fn(_completions('{"x": 5.0}'), answer=[""]) + assert out == [1.0] + + def test_json_schema_typeless_key_presence_only(self, tmp_path): + # H2: a key with no agreed type (varied across refs) is presence-only. + schema = {"type": "object", "properties": {"a": {}}, "required": ["a"]} + fn = _load_reward_fn(rs.render_verifier_py("json_schema", schema, meta={"n_refs": 1}), + tmp_path) + assert fn(_completions('{"a": "text"}', '{"a": 9}'), answer=["", ""]) == [1.0, 1.0] + + def test_baked_constant_reflects_spec(self): + # Mutation-guard: a different tolerance must change the emitted source. + a = rs.render_verifier_py("numeric", rs.NumericSpec(True, 0.001), meta={"n_refs": 1}) + b = rs.render_verifier_py("numeric", rs.NumericSpec(True, 0.999), meta={"n_refs": 1}) + assert "0.001" in a and "0.001" not in b + + def test_rel_hint_cannot_inject_code(self): + # security: rel_hint (the -o path) is .format'd raw into the header + # docstring, and the file is later exec'd — a triple-quote break must not + # survive into the emitted source. + evil = 'x\n"""\nimport os\nopen("PWNED", "w").close()\n"""' + src = rs.render_verifier_py("numeric", rs.NumericSpec(False, 0.0), + meta={"n_refs": 1, "rel_hint": evil}) + benign = rs.render_verifier_py("numeric", rs.NumericSpec(False, 0.0), + meta={"n_refs": 1, "rel_hint": "reward.py"}) + compile(src, "", "exec") # still valid Python (no docstring break-out) + # No EXTRA triple-quote injected (the hint can't close the docstring early) + # and the quotes needed to make the payload executable are stripped. + assert src.count('"""') == benign.count('"""') + assert 'open("PWNED"' not in src + + def test_hostile_gold_stays_repr_safe(self): + # Untrusted tool names/keys are baked via repr(); they cannot break out. + spec = rs.ToolCallSpec(tools={ + 'ev"l\n"""x': {"required": (), "allowed": ('a"\nb',)}}) + src = rs.render_verifier_py("tool_call", spec, meta={"n_refs": 1}) + compile(src, "", "exec") + + +class TestInduceJsonSchemaEdges: + def test_mixed_shapes_refused(self): + with pytest.raises(ValueError, match="mix"): + rs.induce_json_schema(['{"a": 1}', "[1, 2]"]) + + def test_no_containers_refused(self): + with pytest.raises(ValueError, match="no JSON"): + rs.induce_json_schema(["hello", "world"]) + + def test_inconsistent_key_type_is_typeless(self): + schema = rs.induce_json_schema(['{"a": 1}', '{"a": "x"}']) + assert schema["properties"]["a"] == {} # no type constraint + + +class TestInduceNumericGuard: + def test_non_numeric_refused(self): + with pytest.raises(ValueError, match="not numeric"): + rs.induce_numeric(["hello", "world"]) + + +class TestCalibrateFloors: + def test_hard_floor_refuses_at_zero_threshold(self): + # M2: an always-accept verifier is refused even with min_discrimination=0. + def always_one(completions, **kw): + return [1.0] * len(completions) + rep = rs.calibrate(always_one, ["a", "b"], ["x", "y"], min_discrimination=0.0) + assert rep.refused is True and rep.discrimination <= 0.0 + + def test_self_accept_floor(self): + # H2: a verifier accepting only half its own refs is refused (floor 0.9). + state = {"i": 0} + def half(completions, **kw): + out = [] + for _ in completions: + out.append(1.0 if state["i"] % 2 == 0 else 0.0) + state["i"] += 1 + return out + rep = rs.calibrate(half, ["a", "b", "c", "d"], ["x"], min_discrimination=0.1) + assert rep.refused is True and "references" in rep.reason + + +class TestLoadRewardFnsGuards: + def test_none_friendly(self): + from soup_cli.trainer.rewards import load_reward_fns + with pytest.raises(ValueError, match="must be a string"): + load_reward_fns(None) + + def test_blank_rejected(self): + from soup_cli.trainer.rewards import load_reward_fns + with pytest.raises(ValueError, match="blank"): + load_reward_fns(" ") + + def test_duplicate_rejected(self): + from soup_cli.trainer.rewards import load_reward_fns + with pytest.raises(ValueError, match="twice"): + load_reward_fns("accuracy,accuracy") + + +# --------------------------------------------------------------------------- +# perturb + calibrate +# --------------------------------------------------------------------------- +class TestPerturbNegatives: + def test_negatives_differ_from_golds(self): + golds = ["5", "12"] + negs = rs.perturb_negatives(golds, "numeric") + assert negs and all(n not in golds for n in negs) + + +class TestCalibrate: + def test_always_one_verifier_refused(self): + def always_one(completions, **kw): + return [1.0] * len(completions) + rep = rs.calibrate(always_one, ["a", "b"], ["x", "y"], min_discrimination=0.5) + assert rep.refused is True and rep.discrimination == 0.0 + + def test_good_verifier_passes(self): + def exact(completions, **kw): + answers = kw.get("answer", []) + return [1.0 if c[-1]["content"] == a else 0.0 + for c, a in zip(completions, answers)] + rep = rs.calibrate(exact, ["a", "b"], ["neg1", "neg2"], min_discrimination=0.5) + assert rep.refused is False and rep.pos_accept == 1.0 and rep.neg_accept == 0.0 + + def test_discrimination_boundary_exact(self): + # Accept ALL positives (clears the 0.9 self-accept floor) but half the + # negatives → discrimination 0.5 == threshold → pass (not < threshold). + phase = {"neg": False} + + def v(completions, **kw): + if not phase["neg"]: + phase["neg"] = True + return [1.0] * len(completions) # positives: all accepted + return [1.0 if i % 2 == 0 else 0.0 for i in range(len(completions))] + rep = rs.calibrate(v, ["a", "b"], ["x", "y"], min_discrimination=0.5) + assert rep.pos_accept == 1.0 and rep.neg_accept == 0.5 + assert rep.discrimination == 0.5 and rep.refused is False + + +# --------------------------------------------------------------------------- +# synthesize (end-to-end pure) +# --------------------------------------------------------------------------- +class TestSynthesize: + def test_numeric_from_calculator_shape(self): + rows = [{"prompt": "2+2", "answer": "4"}, {"prompt": "3+3", "answer": "6"}] + res = rs.synthesize(rows, field="answer", kind="auto") + assert res.kind == "numeric" + assert "def reward_fn" in res.source + + def test_bad_kind_raises(self): + with pytest.raises(ValueError, match="kind"): + rs.synthesize([{"answer": "1"}], field="answer", kind="bogus") + + +# --------------------------------------------------------------------------- +# CLI — soup reward synth +# --------------------------------------------------------------------------- +def _write_jsonl(path: Path, rows): + import json as _json + path.write_text("\n".join(_json.dumps(r) for r in rows), encoding="utf-8") + + +class TestRewardSynthCli: + def _runner(self): + from typer.testing import CliRunner + return CliRunner() + + def test_help(self): + from soup_cli.commands.reward import app + res = self._runner().invoke(app, ["synth", "--help"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + assert "reward" in res.output.lower() + + def test_numeric_happy(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), + [{"prompt": "2+2", "answer": "4"}, {"prompt": "3+3", "answer": "6"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + assert Path("reward.py").exists() + # The emitted verifier agrees with the golds. + fn = _load_reward_fn(Path("reward.py").read_text(encoding="utf-8"), tmp_path) + assert fn(_completions("4", "99"), answer=["4", "6"]) == [1.0, 0.0] + + def test_cant_induce_is_error_exit_1(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + # All golds empty → regex can't be induced → INDUCTION ERROR (exit 1), + # distinct from a calibration refusal (exit 2). No file written. + _write_jsonl(Path("refs.jsonl"), + [{"answer": ""}, {"answer": ""}, {"answer": ""}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--kind", "regex"]) + assert res.exit_code == 1, (res.output, repr(res.exception)) + assert "regex" in res.output.lower() + assert not Path("reward.py").exists() + + def test_refusal_deletes_file(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + # A numeric verifier with a huge tolerance accepts the perturbed (+9999) + # negatives too → discrimination below the default 0.5 → REFUSED (exit 2), + # and the emitted file is removed (no partial artifact left behind). + _write_jsonl(Path("refs.jsonl"), [{"answer": str(i)} for i in range(8)]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--kind", "numeric", "--tolerance", "1e9"]) + assert res.exit_code == 2, (res.output, repr(res.exception)) + assert "discrimination" in res.output.lower() and "refus" in res.output.lower() + assert not Path("reward.py").exists() + + def test_plan_only_writes_nothing(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}, {"answer": "6"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--plan-only"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + assert not Path("reward.py").exists() + # Spec-specific fields, not just the kind name in the panel title. + assert "numeric" in res.output.lower() and "tolerance" in res.output.lower() + assert "float=false" in res.output.lower().replace(" ", "") + + def test_output_must_be_py(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.txt"]) + assert res.exit_code == 1 + assert ".py" in res.output + + def test_overwrite_without_force_rejected(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}, {"answer": "6"}]) + Path("reward.py").write_text("# existing\n", encoding="utf-8") + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py"]) + assert res.exit_code == 1 + assert "force" in res.output.lower() + assert Path("reward.py").read_text(encoding="utf-8") == "# existing\n" + + def test_bad_kind_rejected(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--kind", "bogus"]) + assert res.exit_code == 1 + assert "kind" in res.output.lower() + + +# --------------------------------------------------------------------------- +# Blocking riders — #311 comma-split loader + reward_fn validator + envs docstring +# --------------------------------------------------------------------------- +class TestCommaSplitLoader: + def test_two_names_two_callables(self): + from soup_cli.trainer.rewards import load_reward_fns + fns = load_reward_fns("accuracy,format") + assert isinstance(fns, list) and len(fns) == 2 + assert {f.__name__ for f in fns} == {"accuracy_reward", "format_reward"} + + def test_single_name_single_element_list(self): + from soup_cli.trainer.rewards import load_reward_fns + fns = load_reward_fns("accuracy") + assert isinstance(fns, list) and len(fns) == 1 + assert fns[0].__name__ == "accuracy_reward" + + def test_whitespace_around_segments(self): + from soup_cli.trainer.rewards import load_reward_fns + fns = load_reward_fns(" accuracy , format ") + assert len(fns) == 2 + + def test_empty_segment_raises(self): + from soup_cli.trainer.rewards import load_reward_fns + with pytest.raises(ValueError, match="empty"): + load_reward_fns("accuracy,") + + def test_verifiable_with_domain(self): + from soup_cli.trainer.rewards import load_reward_fns + fns = load_reward_fns("verifiable,format", verifiable_domain="math") + assert len(fns) == 2 + + +class TestSelectRewardFn: + def test_comma_split_returns_list(self): + from soup_cli.config.schema import TrainingConfig + from soup_cli.trainer.grpo import _select_reward_fn + tcfg = TrainingConfig(reward_fn="accuracy,format") + out = _select_reward_fn(tcfg, "cpu", False) + assert isinstance(out, list) and len(out) == 2 + + def test_single_returns_callable(self): + from soup_cli.config.schema import TrainingConfig + from soup_cli.trainer.grpo import _select_reward_fn + tcfg = TrainingConfig(reward_fn="accuracy") + out = _select_reward_fn(tcfg, "cpu", False) + assert callable(out) and not isinstance(out, list) + + +class TestRewardFnValidator: + def test_null_byte_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + with pytest.raises(ValidationError, match="null"): + TrainingConfig(reward_fn="acc\x00uracy") + + def test_empty_comma_segment_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + with pytest.raises(ValidationError, match="empty"): + TrainingConfig(reward_fn="accuracy,,format") + + def test_overlong_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + with pytest.raises(ValidationError, match="512"): + TrainingConfig(reward_fn="a" * 600) + + def test_comma_combo_accepted(self): + from soup_cli.config.schema import TrainingConfig + assert TrainingConfig(reward_fn="accuracy,format").reward_fn == "accuracy,format" + + +class TestRewardFnTaskGate: + def _cfg(self, task, reward_fn): + from soup_cli.config.loader import load_config_from_string + return load_config_from_string( + f"base: sshleifer/tiny-gpt2\ntask: {task}\n" + f"data:\n train: d.jsonl\ntraining:\n reward_fn: {reward_fn}\n" + ) + + def test_grpo_allows_comma(self): + cfg = self._cfg("grpo", "accuracy,format") + assert cfg.training.reward_fn == "accuracy,format" + + def test_ppo_rejects_comma(self): + # load_config_from_string re-wraps pydantic's ValidationError as ValueError. + with pytest.raises(ValueError, match="grpo"): + self._cfg("ppo", "accuracy,format") + + def test_sft_rejects_comma(self): + with pytest.raises(ValueError, match="grpo"): + self._cfg("sft", "accuracy,format") + + def test_ppo_single_name_ok(self): + cfg = self._cfg("ppo", "accuracy") + assert cfg.training.reward_fn == "accuracy" + + +class TestReviewFixes: + def test_null_gold_falls_through_to_messages(self): + rows = [{"answer": None, + "messages": [{"role": "assistant", "content": "gold"}]}] + assert rs.extract_golds(rows, field="answer") == ["gold"] + + def test_null_gold_only_skipped(self): + with pytest.raises(ValueError, match="no gold"): + rs.extract_golds([{"answer": None}], field="answer") + + def test_non_finite_tolerance_raises(self): + with pytest.raises(ValueError, match="finite"): + rs.induce_numeric(["1.0"], tolerance=float("nan")) + with pytest.raises(ValueError, match="finite"): + rs.induce_numeric(["1.0"], tolerance=float("inf")) + + def test_cli_bad_min_discrimination(self, tmp_path, monkeypatch): + from typer.testing import CliRunner + + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}]) + res = CliRunner().invoke(app, ["synth", "refs.jsonl", "-o", "r.py", + "--min-discrimination", "2.0"]) + assert res.exit_code == 1 and "min-discrimination" in res.output + + def test_cli_bad_tolerance(self, tmp_path, monkeypatch): + from typer.testing import CliRunner + + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}]) + res = CliRunner().invoke(app, ["synth", "refs.jsonl", "-o", "r.py", + "--tolerance", "-1"]) + assert res.exit_code == 1 and "tolerance" in res.output + + +class TestPerKindPipeline: + """Every kind runs the FULL write->load->calibrate pipeline (not just numeric).""" + + def _runner(self): + from typer.testing import CliRunner + return CliRunner() + + def _synth(self, tmp_path, monkeypatch, rows, extra=()): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), rows) + return self._runner().invoke( + app, ["synth", "refs.jsonl", "-o", "reward.py", *extra]) + + def test_json_schema_end_to_end(self, tmp_path, monkeypatch): + rows = [{"answer": '{"city": "Paris", "pop": 2}'}, + {"answer": '{"city": "Rome", "pop": 3}'}] + res = self._synth(tmp_path, monkeypatch, rows) + assert res.exit_code == 0, (res.output, repr(res.exception)) + fn = _load_reward_fn(Path("reward.py").read_text(encoding="utf-8"), tmp_path) + assert fn(_completions('{"city": "X", "pop": 9}', "nope"), + answer=["", ""]) == [1.0, 0.0] + + def test_tool_call_end_to_end(self, tmp_path, monkeypatch): + rows = [{"answer": '{"name": "search", "arguments": {"q": "a"}}'}, + {"answer": '{"name": "search", "arguments": {"q": "b"}}'}] + res = self._synth(tmp_path, monkeypatch, rows) + assert res.exit_code == 0, (res.output, repr(res.exception)) + fn = _load_reward_fn(Path("reward.py").read_text(encoding="utf-8"), tmp_path) + assert fn(_completions('{"name": "search", "arguments": {"q": "z"}}', + '{"name": "search", "arguments": {}}'), + answer=["", ""]) == [1.0, 0.0] + + def test_regex_end_to_end(self, tmp_path, monkeypatch): + rows = [{"answer": "2031-01-02"}, {"answer": "1999-12-31"}, + {"answer": "2000-05-06"}] + res = self._synth(tmp_path, monkeypatch, rows, extra=["--kind", "regex"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + fn = _load_reward_fn(Path("reward.py").read_text(encoding="utf-8"), tmp_path) + assert fn(_completions("2020-11-11", "not a date"), answer=["", ""]) == [1.0, 0.0] + + +class TestPerturbNegativesPerKind: + def test_tool_call_generates_valid_json_hacks(self): + golds = ['{"name": "search", "arguments": {"q": "x"}}'] + negs = rs.perturb_negatives(golds, "tool_call") + # Must include a VALID-JSON wrong-name negative and a foreign-arg negative, + # not just the syntactically-invalid string. + assert any('"__nonexistent_tool__"' in n for n in negs) + assert any("__foreign_arg__" in n for n in negs) + + def test_json_schema_negatives(self): + negs = rs.perturb_negatives(['{"a": 1}'], "json_schema") + assert negs and all(n for n in negs[:1]) # non-empty invalid-json negative + + def test_unknown_kind_raises(self): + with pytest.raises(ValueError, match="unknown kind"): + rs.perturb_negatives(["x"], "bogus") + + +class TestOutputReport: + def _runner(self): + from typer.testing import CliRunner + return CliRunner() + + def test_report_written_and_parseable(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}, {"answer": "6"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--output-report", "rep.json"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + import json as _json + rep = _json.loads(Path("rep.json").read_text(encoding="utf-8")) + assert rep["kind"] == "numeric" and rep["refused"] is False + assert set(rep) >= {"pos_accept", "neg_accept", "discrimination", "precision"} + + def test_bad_report_path_fails_before_writing_verifier(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}, {"answer": "6"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--output-report", "../escape.json"]) + assert res.exit_code == 1, (res.output, repr(res.exception)) + # The verifier is never written when the report path is rejected up front. + assert not Path("reward.py").exists() + + +class TestCliRobustness: + def _runner(self): + from typer.testing import CliRunner + return CliRunner() + + def test_missing_references_file(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + res = self._runner().invoke(app, ["synth", "nope.jsonl", "-o", "reward.py"]) + assert res.exit_code == 1, (res.output, repr(res.exception)) + + def test_empty_references_file(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + Path("refs.jsonl").write_text("\n\n", encoding="utf-8") + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py"]) + assert res.exit_code == 1 and "no JSON-object rows" in res.output + + def test_force_overwrites(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}, {"answer": "6"}]) + Path("reward.py").write_text("# stale\n", encoding="utf-8") + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--force"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + assert "# stale" not in Path("reward.py").read_text(encoding="utf-8") + + def test_custom_field(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"gold": "4"}, {"gold": "6"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py", + "--field", "gold"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + assert Path("reward.py").exists() + + def test_output_outside_cwd_rejected(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + _write_jsonl(Path("refs.jsonl"), [{"answer": "4"}]) + res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "../evil.py"]) + assert res.exit_code == 1 and "cwd" in res.output.lower() + + def test_references_outside_cwd_rejected(self, tmp_path, monkeypatch): + from soup_cli.commands.reward import app + monkeypatch.chdir(tmp_path) + res = self._runner().invoke(app, ["synth", "../secret.jsonl", "-o", "reward.py"]) + assert res.exit_code == 1 and "cwd" in res.output.lower() + + def test_registered_on_top_level_cli(self): + from soup_cli.cli import app as root + res = self._runner().invoke(root, ["reward", "synth", "--help"]) + assert res.exit_code == 0, (res.output, repr(res.exception)) + + +class TestBoundaries: + def test_self_accept_boundary_9_of_10(self): + # 9/10 == 0.9 self-accept floor → NOT refused; 8/9 < 0.9 → refused. + pos9 = ["p"] * 10 + seq = {"i": 0} + + def accept_first_9(completions, **kw): + out = [] + for _ in completions: + out.append(1.0 if seq["i"] < 9 else 0.0) + seq["i"] += 1 + return out + rep = rs.calibrate(accept_first_9, pos9, ["n"], min_discrimination=0.1) + assert abs(rep.pos_accept - 0.9) < 1e-9 and rep.refused is False + + def test_self_accept_below_floor_refused(self): + seq = {"i": 0} + + def accept_first_7(completions, **kw): # 7/9 ~= 0.78 < 0.9 + out = [] + for _ in completions: + out.append(1.0 if seq["i"] < 7 else 0.0) + seq["i"] += 1 + return out + rep = rs.calibrate(accept_first_7, ["p"] * 9, ["n"], min_discrimination=0.1) + assert rep.pos_accept < rs._MIN_SELF_ACCEPT and rep.refused is True + assert "own references" in rep.reason + + def test_numeric_tolerance_inclusive(self, tmp_path): + src = rs.render_verifier_py("numeric", rs.NumericSpec(True, 0.1), meta={"n_refs": 1}) + fn = _load_reward_fn(src, tmp_path) + # |5.1 - 5.0| == 0.1 == tol → inclusive match. + assert fn(_completions("5.1"), answer=["5.0"]) == [1.0] + assert fn(_completions("5.2"), answer=["5.0"]) == [0.0] + + def test_precision_value(self): + def accept_all(completions, **kw): + return [1.0] * len(completions) + # 2 refs accepted (tp=2), 3 negatives all accepted (fp=3) → precision 2/5. + rep = rs.calibrate(accept_all, ["a", "b"], ["x", "y", "z"]) + assert abs(rep.precision - 0.4) < 1e-9 + + def test_detect_kind_numeric_confidence(self): + # 9/10 numeric → numeric; 5/10 → not numeric (falls to regex/None). + assert rs.detect_kind(["1", "2", "3", "4", "5", "6", "7", "8", "9", "x"]) == "numeric" + + def test_reward_fn_512_boundary(self): + from soup_cli.config.schema import TrainingConfig + assert TrainingConfig(reward_fn="a" * 512).reward_fn == "a" * 512 + from pydantic import ValidationError + with pytest.raises(ValidationError, match="512"): + TrainingConfig(reward_fn="a" * 513) + + +class TestVerifiableComboGate: + def test_comma_verifiable_without_domain_rejected(self): + # #8: "accuracy,verifiable" must fail at config parse (like bare + # "verifiable"), not silently defer to a runtime crash. + from soup_cli.config.loader import load_config_from_string + with pytest.raises(ValueError, match="verifiable_domain"): + load_config_from_string( + "base: sshleifer/tiny-gpt2\ntask: grpo\ndata:\n train: d.jsonl\n" + "training:\n reward_fn: accuracy,verifiable\n" + ) + + def test_comma_verifiable_with_domain_ok(self): + from soup_cli.config.loader import load_config_from_string + cfg = load_config_from_string( + "base: sshleifer/tiny-gpt2\ntask: grpo\ndata:\n train: d.jsonl\n" + "training:\n reward_fn: accuracy,verifiable\n verifiable_domain: math\n" + ) + assert cfg.training.reward_fn == "accuracy,verifiable" + + +class TestRewardFnValidatorBlank: + def test_blank_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + with pytest.raises(ValidationError, match="blank"): + TrainingConfig(reward_fn=" ") + + def test_bool_rejected(self): + from pydantic import ValidationError + + from soup_cli.config.schema import TrainingConfig + with pytest.raises(ValidationError, match="string"): + TrainingConfig(reward_fn=True) + + +class TestEnvsDocstringFix: + def test_calculator_docstring(self): + from soup_cli.envs import calculator + assert "reward_fn='math'" not in (calculator.__doc__ or "") + assert "verifiable" in (calculator.__doc__ or "") + + def test_guess_number_docstring(self): + from soup_cli.envs import guess_number + assert "reward_fn='math'" not in (guess_number.__doc__ or "") + assert "verifiable" in (guess_number.__doc__ or "") + + +class TestNoTopLevelTorch: + def test_reward_synth_has_no_top_level_torch(self): + src = Path(rs.__file__).read_text(encoding="utf-8") + tree = ast.parse(src) + for node in tree.body: # module-level only + if isinstance(node, ast.Import): + assert all(not n.name.startswith(("torch", "transformers", "peft")) + for n in node.names) + if isinstance(node, ast.ImportFrom): + assert not (node.module or "").startswith(("torch", "transformers", "peft"))