Alpamys
|
1b1d679141
|
feat: v0.21.0 — migrate, recipes, NEFTune, rsLoRA
- `soup migrate` — import configs from LLaMA-Factory, Axolotl, Unsloth
notebooks (AST-only .ipynb parsing, path traversal protection)
- `soup recipes` — 30 ready-made configs for popular models
(list/show/use/search with path traversal protection)
- NEFTune (`neftune_alpha`) — noisy embeddings for SFT/DPO/KTO/ORPO/SimPO/IPO
- rsLoRA (`use_rslora`) — rank-stabilized LoRA scaling in all 11 trainers
- Fix: `soup doctor` torchvision circular import crash
- Fix: `load_eval_tasks()` now accepts str in addition to Path
- Security: Rich markup injection prevention in migration warnings
- Security: 10 MB file size limit on migration input files
- 1789 tests, 62 test files, 64% coverage
|
2026-04-02 14:08:36 +05:00 |
|
Alpamys
|
c46265fd18
|
feat: add eval platform with custom evals, LLM judge, human eval, leaderboard (v0.19.0)
Full-featured evaluation system with 7 subcommands:
- soup eval benchmark: standard benchmarks via lm-evaluation-harness
- soup eval custom: custom JSONL eval tasks with 4 scoring modes
- soup eval judge: LLM-as-a-judge (OpenAI/Ollama/server backends)
- soup eval auto: automatic post-training evaluation from config
- soup eval compare: side-by-side eval comparison with regression detection
- soup eval leaderboard: local model leaderboard with JSON/CSV export
- soup eval human: terminal A/B comparison with Elo ratings
New modules: soup_cli/eval/ (custom.py, judge.py, human.py, leaderboard.py)
Config: EvalConfig added to schema.py (auto_eval, benchmarks, custom_tasks, judge)
Callback: SoupTrainerCallback.on_train_end triggers auto-eval when configured
Security: SSRF protection on judge API, ReDoS guard on regex scoring,
API key isolation per provider, 10k task/prompt caps, read-only SQL queries
1585 tests, 58 test files, ruff clean
|
2026-04-01 14:47:08 +05:00 |