5 Commits

Author	SHA1	Message	Date
Alpamys	c46265fd18	feat: add eval platform with custom evals, LLM judge, human eval, leaderboard (v0.19.0) ... Full-featured evaluation system with 7 subcommands: - soup eval benchmark: standard benchmarks via lm-evaluation-harness - soup eval custom: custom JSONL eval tasks with 4 scoring modes - soup eval judge: LLM-as-a-judge (OpenAI/Ollama/server backends) - soup eval auto: automatic post-training evaluation from config - soup eval compare: side-by-side eval comparison with regression detection - soup eval leaderboard: local model leaderboard with JSON/CSV export - soup eval human: terminal A/B comparison with Elo ratings New modules: soup_cli/eval/ (custom.py, judge.py, human.py, leaderboard.py) Config: EvalConfig added to schema.py (auto_eval, benchmarks, custom_tasks, judge) Callback: SoupTrainerCallback.on_train_end triggers auto-eval when configured Security: SSRF protection on judge API, ReDoS guard on regex scoring, API key isolation per provider, 10k task/prompt caps, read-only SQL queries 1585 tests, 58 test files, ruff clean	2026-04-01 14:47:08 +05:00
Alpamys	4010798e2b	Fix PyTorch 2.7 compatibility: total_mem → total_memory (v0.2.1) ... PyTorch 2.7+ renamed `get_device_properties().total_mem` to `total_memory`. Fixed in gpu.py and callback.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-04 00:37:07 +05:00
Alpamys	2aaa87fb4e	Phase 2: experiment tracking, data tools, model evaluation ... - Add SQLite experiment tracker (~/.soup/experiments.db) with auto-logging of config, per-step metrics, hardware info, and eval results - Add soup runs commands: list, show (with plotext loss curves), compare, delete - Integrate tracker into soup train (auto start_run/finish_run/fail_run) - Add soup data convert (alpaca/sharegpt/chatml bidirectional conversion) - Add soup data merge (concatenate datasets with optional shuffle) - Add soup data dedup (MinHash near-duplicate removal via datasketch) - Add soup data stats (length percentiles, token counts, language detection) - Add soup eval (lm-evaluation-harness wrapper with tracker integration) - Add reverse format conversion: messages_to_format() in data/formats.py - Add extended_stats() to data/validator.py - Update monitoring callback to log metrics to tracker - Add plotext to deps, datasketch as optional [data] dep - Update README and CLAUDE.md with Phase 2 docs - 70 tests passing, ruff clean Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-23 23:34:28 +05:00
Alpamys	7433029d19	Fix all ruff lint errors and failing test ... - Fix 23 ruff errors: line too long, unused imports, ambiguous vars - Fix validator: empty string is valid data, only count None as empty - Remove unused imports in display.py and validator.py - Rename ambiguous `l` vars to `part`, `entry`, `length` - Break long lines in callback.py, display.py, sft.py, constants.py All 20 tests passing, ruff clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-20 16:25:46 +05:00
Alpamys	d6e932a1d3	Initial project setup: CLI skeleton + config + trainer + data pipeline ... - Typer CLI: soup init, soup train, soup data inspect/validate - Pydantic config schema with YAML loader and validation - Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets - Format detection: Alpaca, ShareGPT, ChatML (auto-detect) - SFT trainer wrapper over transformers + peft + trl - QLoRA/LoRA support with auto batch size estimation - GPU detection (CUDA/MPS/CPU) and memory calculation - Rich live terminal dashboard for training monitoring - Config templates: chat, code, medical - Tests (pytest) + GitHub Actions CI - MIT license Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-20 16:14:56 +05:00