Alpamys
|
c46265fd18
|
feat: add eval platform with custom evals, LLM judge, human eval, leaderboard (v0.19.0)
Full-featured evaluation system with 7 subcommands:
- soup eval benchmark: standard benchmarks via lm-evaluation-harness
- soup eval custom: custom JSONL eval tasks with 4 scoring modes
- soup eval judge: LLM-as-a-judge (OpenAI/Ollama/server backends)
- soup eval auto: automatic post-training evaluation from config
- soup eval compare: side-by-side eval comparison with regression detection
- soup eval leaderboard: local model leaderboard with JSON/CSV export
- soup eval human: terminal A/B comparison with Elo ratings
New modules: soup_cli/eval/ (custom.py, judge.py, human.py, leaderboard.py)
Config: EvalConfig added to schema.py (auto_eval, benchmarks, custom_tasks, judge)
Callback: SoupTrainerCallback.on_train_end triggers auto-eval when configured
Security: SSRF protection on judge API, ReDoS guard on regex scoring,
API key isolation per provider, 10k task/prompt caps, read-only SQL queries
1585 tests, 58 test files, ruff clean
|
2026-04-01 14:47:08 +05:00 |
|
Alpamys
|
2aaa87fb4e
|
Phase 2: experiment tracking, data tools, model evaluation
- Add SQLite experiment tracker (~/.soup/experiments.db) with auto-logging
of config, per-step metrics, hardware info, and eval results
- Add soup runs commands: list, show (with plotext loss curves), compare, delete
- Integrate tracker into soup train (auto start_run/finish_run/fail_run)
- Add soup data convert (alpaca/sharegpt/chatml bidirectional conversion)
- Add soup data merge (concatenate datasets with optional shuffle)
- Add soup data dedup (MinHash near-duplicate removal via datasketch)
- Add soup data stats (length percentiles, token counts, language detection)
- Add soup eval (lm-evaluation-harness wrapper with tracker integration)
- Add reverse format conversion: messages_to_format() in data/formats.py
- Add extended_stats() to data/validator.py
- Update monitoring callback to log metrics to tracker
- Add plotext to deps, datasketch as optional [data] dep
- Update README and CLAUDE.md with Phase 2 docs
- 70 tests passing, ruff clean
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
2026-02-23 23:34:28 +05:00 |