docs(prm): v0.71.30 release — version bump, CHANGELOG, README What's New, docs/training (PRM-guided GRPO), recipe/test counts

CHANGELOG.md

@@ -12,6 +12,44 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.30] - 2026-07-05
+
+### Added
+- **PRM-guided GRPO** — use a trained Process Reward Model as the *per-step*
+  reward inside GRPO (the o1-era process-supervision signal). Set
+  `training.prm_reward: <PRM dir|id>` (a model produced by `soup train`
+  `task=prm`) and `training.prm_aggregate: min|prod|last`; the PRM splits each
+  generated completion into reasoning steps, scores every step with its reward
+  head, and folds the per-step scores into one scalar reward that GRPO
+  optimises. The PRM reward *replaces* `reward_fn` and rides the existing
+  reward-shaping + reward-hack-mitigation seam, so the v0.71.26 controller
+  observes it. Cross-validators gate `task='grpo'` + `backend='transformers'` +
+  `modality='text'`. Default aggregation is `min` (weakest-link); `prod`
+  assumes calibrated `[0,1]` step scores.
+- **Bundled rollout environments** — three pure-Python toy environments
+  (`soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`) exposing a
+  `rollout(prompts)` entry point so the live `openenv` GRPO rollout path runs
+  out-of-the-box: `training.rollout_backend=openenv` +
+  `training.rollout_func=soup_cli.envs.calculator:rollout`. Three ready-made
+  recipes added (`grpo-env-calculator` / `grpo-env-retrieval-qa` /
+  `grpo-env-guess-number`); catalog 134 → 137.
+
+### Fixed
+- **`soup train task=prm` producer conformance** (surfaced by the v0.71.30 live
+  smoke): the PRM trainer now casts its reward head to the base-model dtype
+  (bf16 CUDA runs previously crashed on the first `compute_loss`), saves the
+  tokenizer alongside the model (so a PRM checkpoint is loadable standalone),
+  and returns the standard trainer-result shape (previously the CLI crashed with
+  a `KeyError: 'initial_loss'` right after saving).
+
+### Notes
+- Proof-of-mechanism only: validated on a tiny model (SmolLM2-135M) with a tiny
+  synthetic PRM and synthetic reward — not a production reward-model claim
+  (scale ask tracked in #286). The bundled environments are deterministic
+  single-shot *seeders*, not interactive multi-turn model-in-the-loop episodes
+  (the live `openenv` contract passes only prompts). Step split is a newline
+  heuristic; PRM completions are scored one forward pass each.
+
 ## [0.71.29] - 2026-07-05
 
 ### Added

CONTRIBUTING.md

@@ -110,7 +110,7 @@ src/soup_cli/
   experiment/         - SQLite experiment tracking
   eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
   migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
-  recipes/            - Ready-made configs for popular models (134 recipes)
+  recipes/            - Ready-made configs for popular models (137 recipes)
   autopilot/          - Zero-config decision engine (v0.25.0)
   registry/           - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (306 files, 15224 tests)
+tests/                - Test suite (307 files, 15334 tests)
 examples/             - Real-world config examples and datasets
 ```
 

README.md

@@ -49,22 +49,27 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.29 — Make your model smaller, locally: `soup shrink`.** Depth-prune the least-important layers, then distill-heal — with a binary SHIP / DON'T-SHIP verdict. Based on "The Unreasonable Ineffectiveness of the Deeper Layers" (arXiv:2403.17887).
+**v0.71.30 — Process-supervised RL: PRM-guided GRPO.** Use a trained Process Reward Model to score each *reasoning step* of a GRPO completion — the o1-era training signal — plus bundled toy environments so the openenv rollout path runs out-of-the-box.
 
-- **Importance-ranked depth pruning.** One forward pass per calibration prompt scores each
-  contiguous layer block by the angular distance of the residual stream across it; the
-  least-important block is dropped (the first and last layer are always protected).
-- **Distill-heal, Minitron-style.** `--heal` distills the original model into the pruned
-  student (LoRA logit-KD) as an isolated `soup train` run, then fuses the adapter back so you
-  ship a single dense smaller model — not a base + adapter.
-- **A verdict, not a dashboard.** Every run ends in **SHIP** or **DON'T SHIP** on a
-  before/after perplexity ratio (`exit 0 = SHIP`, `2 = DON'T`, `1 = error`), so it drops into CI.
-- **Runs on a laptop GPU.** Live-validated on Windows + RTX 3050 with SmolLM2-135M: drop 25 %
-  (30 → 22 layers, 21 % params) and drop-4 + CPU heal (perplexity recovered to ×1.35).
+- **PRM as the GRPO reward.** Point `training.prm_reward` at a PRM you trained with
+  `soup train task=prm`; it splits each completion into steps, scores every step with the PRM's
+  reward head, and folds them (`min` / `prod` / `last`) into one reward that GRPO optimises —
+  replacing `reward_fn`. It rides the reward-shaping + reward-hack-mitigation seam, so the
+  v0.71.26 controller still watches it.
+- **Bundled rollout environments.** `soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`
+  each expose a `rollout(prompts)` entry point; wire one with `rollout_backend=openenv` +
+  `rollout_func=soup_cli.envs.calculator:rollout` (three ready-made `grpo-env-*` recipes ship it).
+- **Producer fixes too.** `soup train task=prm` now saves its tokenizer and no longer crashes on
+  its own summary — a PRM checkpoint is loadable standalone.
+- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M (CPU): the PRM reward scores
+  good completions above bad and drives GRPO's advantages (`rewards/prm_reward` logged). Tiny
+  model + synthetic PRM — not a production reward-model claim (scale help wanted: #286).
 
-```bash
-soup shrink --model HuggingFaceTB/SmolLM2-135M-Instruct --drop-ratio 0.25 \
-    --calib calib.jsonl --heal heal.jsonl --heal-steps 200 -o shrunk --device cpu
+```yaml
+task: grpo
+training:
+  prm_reward: ./my-prm            # a `soup train task=prm` checkpoint dir
+  prm_aggregate: min              # weakest-link (default) | prod | last
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

docs/commands.md

@@ -133,7 +133,7 @@ soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
 soup migrate --from axolotl config.yml        Import config from Axolotl
 soup migrate --from unsloth notebook.ipynb    Import config from Unsloth notebook
 soup migrate --from llamafactory c.yaml --dry-run  Preview without writing
-soup recipes list                             List all 134 ready-made recipes
+soup recipes list                             List all 137 ready-made recipes
 soup recipes show llama3.1-8b-sft            Print recipe YAML
 soup recipes use llama3.1-8b-sft             Copy recipe to soup.yaml
 soup recipes search "reasoning"              Search by keyword/task/size

docs/training.md

@@ -15,6 +15,7 @@
 - [EBFT / GDPO Loss Variants](#ebft--gdpo-loss-variants)
 - [GRPO Objective Variants](#grpo-objective-variants)
 - [Process Reward Model (PRM)](#process-reward-model-prm)
+- [PRM-guided GRPO (process-supervised RL)](#prm-guided-grpo-process-supervised-rl)
 - [Weighted Multi-Objective Preference Loss](#weighted-multi-objective-preference-loss)
 - [MoE Model Support](#moe-model-support)
 - [Vision / Multimodal Fine-tuning](#vision--multimodal-fine-tuning)
@@ -343,7 +344,61 @@ training:
 
 The trainer loads `AutoModelForCausalLM`, attaches an `nn.Linear(hidden, 1)`
 reward head, and computes MSE between predicted scalars at step-boundary tokens
-and the per-step labels.
+and the per-step labels. The reward head is saved inside the model checkpoint
+(`reward_head.*` in `model.safetensors`) and the tokenizer is saved alongside it,
+so the resulting directory is loadable standalone.
+
+
+## PRM-guided GRPO (process-supervised RL)
+
+Use a trained PRM as the **per-step reward** inside GRPO — the o1-era
+process-supervision signal. Set `training.prm_reward` to a PRM directory (a
+`task=prm` checkpoint) or HF id; the PRM splits each generated completion into
+reasoning steps (newline heuristic), scores every step with its reward head, and
+folds the per-step scores into one scalar reward that GRPO optimises. It
+**replaces** `reward_fn` and rides the existing reward-shaping +
+reward-hack-mitigation seam, so the v0.71.26 controller still observes it (TRL
+logs it as `rewards/prm_reward`).
+
+```yaml
+task: grpo
+backend: transformers        # required (the PRM reward runs a transformers forward)
+modality: text               # required
+data:
+  format: chatml
+  train: ./grpo_prompts.jsonl
+training:
+  prm_reward: ./my-prm       # a `soup train task=prm` checkpoint dir (or HF id)
+  prm_aggregate: min         # weakest-link (default) | prod | last
+  num_generations: 4
+  grpo_beta: 0.04
+```
+
+`prm_aggregate='min'` (weakest-link, the standard PRM aggregation) is the safe
+default; `prod` assumes calibrated `[0,1]` step scores (Soup's PRM head is
+trained with unconstrained MSE, so `prod` can blow up on uncalibrated labels).
+
+**Bundled rollout environments.** Three deterministic pure-Python toy
+environments seed the openenv rollout path out-of-the-box — pair any of them
+with `rollout_backend=openenv`:
+
+```yaml
+training:
+  rollout_backend: openenv
+  rollout_func: soup_cli.envs.calculator:rollout   # or retrieval_qa / guess_number
+  reward_fn: verifiable
+  verifiable_domain: math
+```
+
+Ready-made recipes: `grpo-env-calculator`, `grpo-env-retrieval-qa`,
+`grpo-env-guess-number`. The environments are deterministic single-shot
+prompt/answer *seeders* (the live openenv contract passes only the seed prompts,
+not the model) — not interactive multi-turn episodes.
+
+**Scope:** proof-of-mechanism only — validated on SmolLM2-135M with a tiny
+synthetic PRM (the PRM reward scores good completions above bad and drives GRPO's
+advantages). Not a production reward-model claim; scale validation is help-wanted
+(#286).
 
 
 ## Weighted Multi-Objective Preference Loss

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.29"
+version = "0.71.30"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.29"
+__version__ = "0.71.30"