Adds soup why, soup tui, soup runs replay, soup train --profile, .crash
bundle on training exception, per-run cost in SQLite, --log-level global
flag. Net +110 tests (3818 → 3928); all five review-agent waves
(python / code / security / tdd / smoke) clean.
- Part A: --log-level quiet|normal|verbose|debug → Rich-formatted logger
on the "soup" namespace; idempotent + tier-change replaces handler.
- Part B: SQLite gains cost_usd / cost_gpu_label via lazy ALTER TABLE
(race-tolerant against duplicate-column on concurrent first-boot);
rendered in soup runs show / replay / TUI; bool num_gpus rejected;
LIKE wildcards escaped in tracker.get_run prefix match.
- Part C: soup why — heuristic NaN / plateau / divergence / grad-norm /
LR bounds; severity-ordered findings.
- Part D: .crash bundle generator with recursive hf_*/sk-*/Bearer
redaction, output_dir basename-only, os.path.realpath containment,
secrets.token_hex filename, ValueError (not PermissionError) on
outside-cwd; train.py except-handler writes the bundle without
masking the original exception.
- Part E: soup runs replay <id> — summary panel + downsampled loss
curve (≤2000 points) from SQLite history.
- Part F: soup train --profile — torch.profiler Chrome trace to
<output>/profiles/<run_id>.trace.json; run_id rejects '.', '..',
'/', '\\', null bytes; profiles dir created only on torch import.
- Part G: soup tui — Textual dashboard with lazy ExperimentTracker
import; markup_escape on every DB-sourced string; new [tui] extra.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every
competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators:
Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo,
Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA).
Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with
detection + normalization, synth data template, init template, eval scoring
(tool_call_match / tool_call_name_match / tool_call_args_subset), plus
qwen3-8b-tools and llama4-scout-tools recipes.
Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to
math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox
with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap,
one-time warning panel), and json_schema_reward. verifiable_domain Literal
validated via model_validator.
Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with
mutual-exclusion validator and a unified peft_builder helper that returns
either LoraConfig or VeraConfig with the right init kwargs.
Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx,
MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at
config load time by SoupConfig._validate_mlx_task_support, lazy trainer
registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject.
Part F — Data augmentation: soup data augment with rephrase / translate / style
strategies, path-traversal-protected input/output, count capped 1-10, lang/styles
lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup.
Part G — Training intelligence: forgetting detection (ForgettingDetector with
3 built-in mini benchmarks and warning levels) and checkpoint intelligence
(CheckpointTracker with composite metric, early-stop on regression, safe
top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema
extended with checkpoint_quality + forgetting_eval tables.
Part H — Autopilot: soup autopilot command with dataset/model/hardware
profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf
flags), YAML generator, and full CLI with dry-run + --yes + path-traversal
protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting
detection + checkpoint intelligence + early-stop into the generated config.
Totals:
- 2313 tests passing (183 new, up from 2130)
- 86 test files (8 new)
- 43 ready-made recipes (14 new)
- 16 built-in templates (tool-calling added)
- Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented
design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU,
MLX training integration test requires real hardware)
Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The 4 hex char suffix (65536 possibilities) caused a collision when
generating 100 IDs within the same second on fast CI runners.
Increased to 8 hex chars (4 billion possibilities).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Typer CLI: soup init, soup train, soup data inspect/validate
- Pydantic config schema with YAML loader and validation
- Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets
- Format detection: Alpaca, ShareGPT, ChatML (auto-detect)
- SFT trainer wrapper over transformers + peft + trl
- QLoRA/LoRA support with auto batch size estimation
- GPU detection (CUDA/MPS/CPU) and memory calculation
- Rich live terminal dashboard for training monitoring
- Config templates: chat, code, medical
- Tests (pytest) + GitHub Actions CI
- MIT license
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alpamys