4 Commits

Author	SHA1	Message	Date
Alpamys	c46265fd18	feat: add eval platform with custom evals, LLM judge, human eval, leaderboard (v0.19.0) ... Full-featured evaluation system with 7 subcommands: - soup eval benchmark: standard benchmarks via lm-evaluation-harness - soup eval custom: custom JSONL eval tasks with 4 scoring modes - soup eval judge: LLM-as-a-judge (OpenAI/Ollama/server backends) - soup eval auto: automatic post-training evaluation from config - soup eval compare: side-by-side eval comparison with regression detection - soup eval leaderboard: local model leaderboard with JSON/CSV export - soup eval human: terminal A/B comparison with Elo ratings New modules: soup_cli/eval/ (custom.py, judge.py, human.py, leaderboard.py) Config: EvalConfig added to schema.py (auto_eval, benchmarks, custom_tasks, judge) Callback: SoupTrainerCallback.on_train_end triggers auto-eval when configured Security: SSRF protection on judge API, ReDoS guard on regex scoring, API key isolation per provider, 10k task/prompt caps, read-only SQL queries 1585 tests, 58 test files, ruff clean	2026-04-01 14:47:08 +05:00
Alpamys	67adbff558	fix: add future annotations for Python 3.9 compatibility ... `dict | None` syntax requires Python 3.10+. Adding `from __future__ import annotations` fixes collection on 3.9.	2026-03-26 10:50:46 +05:00
Alpamys	0ebe59ae00	fix: subprocess tests accept Typer no_args_is_help exit code 2 ... Typer returns exit code 2 (not 0) when no_args_is_help=True and no arguments are provided. Fix test_no_args_shows_help and test_data_no_args_shows_help to accept both 0 and 2.	2026-03-26 10:45:03 +05:00
Alpamys	1718578a1a	test: add subprocess CLI tests + cross-platform CI matrix ... - Add test_cli_subprocess.py (69 tests): real subprocess execution testing entry points, encoding, paths, Unicode, platform regressions - CI matrix: ubuntu/windows/macos × Python 3.9/3.11/3.12 (9 jobs) - CI: add coverage reporting with Codecov upload - Update CLAUDE.md and CONTRIBUTING.md test counts (1022 → 1091)	2026-03-26 10:24:58 +05:00