Alpamys
|
83da7f071d
|
Add PPO / Full RLHF pipeline (Phase 10) — v0.9.0
Three-stage RLHF training: SFT → Reward Model → PPO.
- task: ppo — PPO trainer with manual training loop (generate → score → optimize)
- task: reward_model — RewardTrainer for preference data (prompt/chosen/rejected)
- PPO config: ppo_epochs, ppo_clip_ratio, ppo_kl_penalty, reward_model
- Two reward sources: reward_model (pre-trained RM) and/or reward_fn (callable)
- soup init --template rlhf — generates PPO config with reward model
- Sweep shortcuts for all PPO parameters
- 51 new tests (611 total), ruff clean, all passing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-03-23 22:17:55 +05:00 |