From 93449442b840f56c4675d93b764b02ef0a01435e Mon Sep 17 00:00:00 2001 From: Alpamys Date: Sun, 5 Jul 2026 19:32:38 +0500 Subject: [PATCH] =?UTF-8?q?docs(prm):=20v0.71.30=20release=20=E2=80=94=20v?= =?UTF-8?q?ersion=20bump,=20CHANGELOG,=20README=20What's=20New,=20docs/tra?= =?UTF-8?q?ining=20(PRM-guided=20GRPO),=20recipe/test=20counts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 38 +++++++++++++++++++++++++++ CONTRIBUTING.md | 4 +-- README.md | 33 +++++++++++++---------- docs/commands.md | 2 +- docs/training.md | 57 +++++++++++++++++++++++++++++++++++++++- pyproject.toml | 2 +- src/soup_cli/__init__.py | 2 +- 7 files changed, 118 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffa10c9..16193b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,44 @@ reproducing 70+ versions of notes. ## [Unreleased] +## [0.71.30] - 2026-07-05 + +### Added +- **PRM-guided GRPO** — use a trained Process Reward Model as the *per-step* + reward inside GRPO (the o1-era process-supervision signal). Set + `training.prm_reward: ` (a model produced by `soup train` + `task=prm`) and `training.prm_aggregate: min|prod|last`; the PRM splits each + generated completion into reasoning steps, scores every step with its reward + head, and folds the per-step scores into one scalar reward that GRPO + optimises. The PRM reward *replaces* `reward_fn` and rides the existing + reward-shaping + reward-hack-mitigation seam, so the v0.71.26 controller + observes it. Cross-validators gate `task='grpo'` + `backend='transformers'` + + `modality='text'`. Default aggregation is `min` (weakest-link); `prod` + assumes calibrated `[0,1]` step scores. +- **Bundled rollout environments** — three pure-Python toy environments + (`soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`) exposing a + `rollout(prompts)` entry point so the live `openenv` GRPO rollout path runs + out-of-the-box: `training.rollout_backend=openenv` + + `training.rollout_func=soup_cli.envs.calculator:rollout`. Three ready-made + recipes added (`grpo-env-calculator` / `grpo-env-retrieval-qa` / + `grpo-env-guess-number`); catalog 134 → 137. + +### Fixed +- **`soup train task=prm` producer conformance** (surfaced by the v0.71.30 live + smoke): the PRM trainer now casts its reward head to the base-model dtype + (bf16 CUDA runs previously crashed on the first `compute_loss`), saves the + tokenizer alongside the model (so a PRM checkpoint is loadable standalone), + and returns the standard trainer-result shape (previously the CLI crashed with + a `KeyError: 'initial_loss'` right after saving). + +### Notes +- Proof-of-mechanism only: validated on a tiny model (SmolLM2-135M) with a tiny + synthetic PRM and synthetic reward — not a production reward-model claim + (scale ask tracked in #286). The bundled environments are deterministic + single-shot *seeders*, not interactive multi-turn model-in-the-loop episodes + (the live `openenv` contract passes only prompts). Step split is a newline + heuristic; PRM completions are scored one forward pass each. + ## [0.71.29] - 2026-07-05 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ca8521b..ddf5261 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -110,7 +110,7 @@ src/soup_cli/ experiment/ - SQLite experiment tracking eval/ - Eval platform (custom tasks, LLM judge, human eval, leaderboard) migrate/ - Config migration (LLaMA-Factory, Axolotl, Unsloth) - recipes/ - Ready-made configs for popular models (134 recipes) + recipes/ - Ready-made configs for popular models (137 recipes) autopilot/ - Zero-config decision engine (v0.25.0) registry/ - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0) cans/ - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0) @@ -120,7 +120,7 @@ src/soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (306 files, 15224 tests) +tests/ - Test suite (307 files, 15334 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index f856344..f33d770 100644 --- a/README.md +++ b/README.md @@ -49,22 +49,27 @@ infrastructure instead of improving models. Soup fixes that. ## What's New -**v0.71.29 — Make your model smaller, locally: `soup shrink`.** Depth-prune the least-important layers, then distill-heal — with a binary SHIP / DON'T-SHIP verdict. Based on "The Unreasonable Ineffectiveness of the Deeper Layers" (arXiv:2403.17887). +**v0.71.30 — Process-supervised RL: PRM-guided GRPO.** Use a trained Process Reward Model to score each *reasoning step* of a GRPO completion — the o1-era training signal — plus bundled toy environments so the openenv rollout path runs out-of-the-box. -- **Importance-ranked depth pruning.** One forward pass per calibration prompt scores each - contiguous layer block by the angular distance of the residual stream across it; the - least-important block is dropped (the first and last layer are always protected). -- **Distill-heal, Minitron-style.** `--heal` distills the original model into the pruned - student (LoRA logit-KD) as an isolated `soup train` run, then fuses the adapter back so you - ship a single dense smaller model — not a base + adapter. -- **A verdict, not a dashboard.** Every run ends in **SHIP** or **DON'T SHIP** on a - before/after perplexity ratio (`exit 0 = SHIP`, `2 = DON'T`, `1 = error`), so it drops into CI. -- **Runs on a laptop GPU.** Live-validated on Windows + RTX 3050 with SmolLM2-135M: drop 25 % - (30 → 22 layers, 21 % params) and drop-4 + CPU heal (perplexity recovered to ×1.35). +- **PRM as the GRPO reward.** Point `training.prm_reward` at a PRM you trained with + `soup train task=prm`; it splits each completion into steps, scores every step with the PRM's + reward head, and folds them (`min` / `prod` / `last`) into one reward that GRPO optimises — + replacing `reward_fn`. It rides the reward-shaping + reward-hack-mitigation seam, so the + v0.71.26 controller still watches it. +- **Bundled rollout environments.** `soup_cli.envs.calculator` / `retrieval_qa` / `guess_number` + each expose a `rollout(prompts)` entry point; wire one with `rollout_backend=openenv` + + `rollout_func=soup_cli.envs.calculator:rollout` (three ready-made `grpo-env-*` recipes ship it). +- **Producer fixes too.** `soup train task=prm` now saves its tokenizer and no longer crashes on + its own summary — a PRM checkpoint is loadable standalone. +- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M (CPU): the PRM reward scores + good completions above bad and drives GRPO's advantages (`rewards/prm_reward` logged). Tiny + model + synthetic PRM — not a production reward-model claim (scale help wanted: #286). -```bash -soup shrink --model HuggingFaceTB/SmolLM2-135M-Instruct --drop-ratio 0.25 \ - --calib calib.jsonl --heal heal.jsonl --heal-steps 200 -o shrunk --device cpu +```yaml +task: grpo +training: + prm_reward: ./my-prm # a `soup train task=prm` checkpoint dir + prm_aggregate: min # weakest-link (default) | prod | last ``` Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). diff --git a/docs/commands.md b/docs/commands.md index 5971896..d0b3a25 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -133,7 +133,7 @@ soup migrate --from llamafactory config.yaml Import config from LLaMA-Factory soup migrate --from axolotl config.yml Import config from Axolotl soup migrate --from unsloth notebook.ipynb Import config from Unsloth notebook soup migrate --from llamafactory c.yaml --dry-run Preview without writing -soup recipes list List all 134 ready-made recipes +soup recipes list List all 137 ready-made recipes soup recipes show llama3.1-8b-sft Print recipe YAML soup recipes use llama3.1-8b-sft Copy recipe to soup.yaml soup recipes search "reasoning" Search by keyword/task/size diff --git a/docs/training.md b/docs/training.md index 0d70927..dc7289a 100644 --- a/docs/training.md +++ b/docs/training.md @@ -15,6 +15,7 @@ - [EBFT / GDPO Loss Variants](#ebft--gdpo-loss-variants) - [GRPO Objective Variants](#grpo-objective-variants) - [Process Reward Model (PRM)](#process-reward-model-prm) +- [PRM-guided GRPO (process-supervised RL)](#prm-guided-grpo-process-supervised-rl) - [Weighted Multi-Objective Preference Loss](#weighted-multi-objective-preference-loss) - [MoE Model Support](#moe-model-support) - [Vision / Multimodal Fine-tuning](#vision--multimodal-fine-tuning) @@ -343,7 +344,61 @@ training: The trainer loads `AutoModelForCausalLM`, attaches an `nn.Linear(hidden, 1)` reward head, and computes MSE between predicted scalars at step-boundary tokens -and the per-step labels. +and the per-step labels. The reward head is saved inside the model checkpoint +(`reward_head.*` in `model.safetensors`) and the tokenizer is saved alongside it, +so the resulting directory is loadable standalone. + + +## PRM-guided GRPO (process-supervised RL) + +Use a trained PRM as the **per-step reward** inside GRPO — the o1-era +process-supervision signal. Set `training.prm_reward` to a PRM directory (a +`task=prm` checkpoint) or HF id; the PRM splits each generated completion into +reasoning steps (newline heuristic), scores every step with its reward head, and +folds the per-step scores into one scalar reward that GRPO optimises. It +**replaces** `reward_fn` and rides the existing reward-shaping + +reward-hack-mitigation seam, so the v0.71.26 controller still observes it (TRL +logs it as `rewards/prm_reward`). + +```yaml +task: grpo +backend: transformers # required (the PRM reward runs a transformers forward) +modality: text # required +data: + format: chatml + train: ./grpo_prompts.jsonl +training: + prm_reward: ./my-prm # a `soup train task=prm` checkpoint dir (or HF id) + prm_aggregate: min # weakest-link (default) | prod | last + num_generations: 4 + grpo_beta: 0.04 +``` + +`prm_aggregate='min'` (weakest-link, the standard PRM aggregation) is the safe +default; `prod` assumes calibrated `[0,1]` step scores (Soup's PRM head is +trained with unconstrained MSE, so `prod` can blow up on uncalibrated labels). + +**Bundled rollout environments.** Three deterministic pure-Python toy +environments seed the openenv rollout path out-of-the-box — pair any of them +with `rollout_backend=openenv`: + +```yaml +training: + rollout_backend: openenv + rollout_func: soup_cli.envs.calculator:rollout # or retrieval_qa / guess_number + reward_fn: verifiable + verifiable_domain: math +``` + +Ready-made recipes: `grpo-env-calculator`, `grpo-env-retrieval-qa`, +`grpo-env-guess-number`. The environments are deterministic single-shot +prompt/answer *seeders* (the live openenv contract passes only the seed prompts, +not the model) — not interactive multi-turn episodes. + +**Scope:** proof-of-mechanism only — validated on SmolLM2-135M with a tiny +synthetic PRM (the PRM reward scores good completions above bad and drives GRPO's +advantages). Not a production reward-model claim; scale validation is help-wanted +(#286). ## Weighted Multi-Objective Preference Loss diff --git a/pyproject.toml b/pyproject.toml index 661f1b8..f4ce139 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.71.29" +version = "0.71.30" description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/src/soup_cli/__init__.py b/src/soup_cli/__init__.py index 898629d..2262af6 100644 --- a/src/soup_cli/__init__.py +++ b/src/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune and post-train LLMs in one command.""" -__version__ = "0.71.29" +__version__ = "0.71.30"