mirror of https://github.com/razor-ai/soup.git
83da7f071d
Three-stage RLHF training: SFT → Reward Model → PPO. - task: ppo — PPO trainer with manual training loop (generate → score → optimize) - task: reward_model — RewardTrainer for preference data (prompt/chosen/rejected) - PPO config: ppo_epochs, ppo_clip_ratio, ppo_kl_penalty, reward_model - Two reward sources: reward_model (pre-trained RM) and/or reward_fn (callable) - soup init --template rlhf — generates PPO config with reward model - Sweep shortcuts for all PPO parameters - 51 new tests (611 total), ruff clean, all passing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| commands | ||
| config | ||
| data | ||
| experiment | ||
| monitoring | ||
| trainer | ||
| ui | ||
| utils | ||
| __init__.py | ||
| __main__.py | ||
| cli.py | ||