Alpamys 83da7f071d Add PPO / Full RLHF pipeline (Phase 10) — v0.9.0 ... Three-stage RLHF training: SFT → Reward Model → PPO. - task: ppo — PPO trainer with manual training loop (generate → score → optimize) - task: reward_model — RewardTrainer for preference data (prompt/chosen/rejected) - PPO config: ppo_epochs, ppo_clip_ratio, ppo_kl_penalty, reward_model - Two reward sources: reward_model (pre-trained RM) and/or reward_fn (callable) - soup init --template rlhf — generates PPO config with reward model - Sweep shortcuts for all PPO parameters - 51 new tests (611 total), ruff clean, all passing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>		2026-03-23 22:17:55 +05:00
..
__init__.py	Initial project setup: CLI skeleton + config + trainer + data pipeline	2026-02-20 16:14:56 +05:00
chat.py	Fix deprecated torch_dtype and warmup_ratio warnings, add testing guide (v0.2.2)	2026-03-05 13:29:07 +05:00
data.py	Add multimodal vision fine-tuning support (Phase 6) — v0.5.0	2026-03-23 20:16:24 +05:00
diff.py	Add Phase 3: serve, data generate, sweep, diff, DeepSpeed (v0.3.0)	2026-03-05 17:14:08 +05:00
doctor.py	Add Quantization-Aware Training support (Phase 7) — v0.6.0	2026-03-23 20:35:35 +05:00
eval.py	Phase 2: experiment tracking, data tools, model evaluation	2026-02-23 23:34:28 +05:00
export.py	Fix deprecated torch_dtype and warmup_ratio warnings, add testing guide (v0.2.2)	2026-03-05 13:29:07 +05:00
generate.py	Add Phase 3: serve, data generate, sweep, diff, DeepSpeed (v0.3.0)	2026-03-05 17:14:08 +05:00
init.py	Add PPO / Full RLHF pipeline (Phase 10) — v0.9.0	2026-03-23 22:17:55 +05:00
merge.py	Fix deprecated torch_dtype and warmup_ratio warnings, add testing guide (v0.2.2)	2026-03-05 13:29:07 +05:00
push.py	Phase 1.5: add soup chat, soup push, DPO trainer + smoke tests	2026-02-23 21:18:19 +05:00
quickstart.py	Add Phase 3.1: friendly errors, soup doctor, soup quickstart, UX polish (v0.3.1)	2026-03-05 19:10:36 +05:00
runs.py	Phase 2: experiment tracking, data tools, model evaluation	2026-02-23 23:34:28 +05:00
serve.py	Add vLLM backend for soup serve (Phase 9) — v0.8.0	2026-03-23 21:30:08 +05:00
sweep.py	Add PPO / Full RLHF pipeline (Phase 10) — v0.9.0	2026-03-23 22:17:55 +05:00
train.py	Add PPO / Full RLHF pipeline (Phase 10) — v0.9.0	2026-03-23 22:17:55 +05:00
ui.py	Add Web UI for experiment management (Phase 8) — v0.7.0	2026-03-23 20:55:22 +05:00