Ships v0.25.0 with eight new capabilities (Parts A–H) that close every
competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators:
Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo,
Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA).
Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with
detection + normalization, synth data template, init template, eval scoring
(tool_call_match / tool_call_name_match / tool_call_args_subset), plus
qwen3-8b-tools and llama4-scout-tools recipes.
Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to
math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox
with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap,
one-time warning panel), and json_schema_reward. verifiable_domain Literal
validated via model_validator.
Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with
mutual-exclusion validator and a unified peft_builder helper that returns
either LoraConfig or VeraConfig with the right init kwargs.
Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx,
MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at
config load time by SoupConfig._validate_mlx_task_support, lazy trainer
registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject.
Part F — Data augmentation: soup data augment with rephrase / translate / style
strategies, path-traversal-protected input/output, count capped 1-10, lang/styles
lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup.
Part G — Training intelligence: forgetting detection (ForgettingDetector with
3 built-in mini benchmarks and warning levels) and checkpoint intelligence
(CheckpointTracker with composite metric, early-stop on regression, safe
top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema
extended with checkpoint_quality + forgetting_eval tables.
Part H — Autopilot: soup autopilot command with dataset/model/hardware
profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf
flags), YAML generator, and full CLI with dry-run + --yes + path-traversal
protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting
detection + checkpoint intelligence + early-stop into the generated config.
Totals:
- 2313 tests passing (183 new, up from 2130)
- 86 test files (8 new)
- 43 ready-made recipes (14 new)
- 16 built-in templates (tool-calling added)
- Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented
design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU,
MLX training integration test requires real hardware)
Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace non-ASCII symbols (checkmarks, arrows, bullets, em-dashes)
with ASCII equivalents in Rich console output to prevent
UnicodeEncodeError on Windows without PYTHONIOENCODING=utf-8
- Add _validate_output_path() for AWQ/GPTQ export — output path
traversal is now checked before import check (previously unreachable
when autoawq/auto-gptq not installed)
- 4 new tests for output path validation (2065 total, 0 failures)
- Update SECURITY.md with v0.22.0–v0.24.1 hardening history
Part A: HuggingFace Dataset browser
- soup data search: search HF Hub for datasets (sort by downloads/likes)
- soup data preview: preview remote dataset metadata, splits, features
- soup data download: stream HF dataset to local JSONL (with format conversion)
- Security: trust_remote_code=False, path traversal protection, samples cap at 1M
Part B: Freeze training (like LLaMA-Factory finetuning_type: freeze)
- freeze_layers / freeze_ratio config fields
- soup_cli/utils/freeze.py: detect layers, freeze bottom N
- Wired into SFT trainer before LoRA application
- Supports LLaMA (layers.N) and GPT-2 (h.N) naming
Part C: Loss watchdog (like Axolotl loss_watchdog_threshold)
- loss_watchdog, loss_watchdog_threshold, loss_watchdog_patience config
- Implemented in SoupTrainerCallback with patience counter
- Rich warning panel (stops Live display first), fires only once
- Wired into all 11 trainers via callback kwargs
Part D: Dataset info registry
- soup data register/unregister/registry commands
- ~/.soup/datasets.json local name→path+format mapping
- Name validation, path traversal protection, Rich markup escaping
82 new tests (2061 total), 74 test files.
- AWQ export (`soup export --format awq`) via autoawq, with --bits, --group-size, --calibration-data
- GPTQ export (`soup export --format gptq`) via auto-gptq, with calibration data support
- Sample packing (`packing: true`) for SFT/Pretrain trainers via TRL's native packing
- `soup data split` — train/val/test splitting with random and stratified strategies
- Curriculum learning (`curriculum: true`) — sort dataset by difficulty for staged training
- New utility: soup_cli/utils/curriculum.py (sort_by_length, create_buckets)
- Security: calibration data path traversal protection, bits validation (4/8 only)
- 1970 tests across 70 test files
- Check stdout encoding instead of type to detect non-UTF-8 consoles
- Redirect stdout to UTF-8 TextIOWrapper before plotext renders
- Add unit test that simulates cp1251 stdout with plotext
- 1022 tests total
- soup data validate: default --format changed from 'alpaca' to 'auto',
uses detect_format() to auto-detect dataset format
- soup data stats: force UTF-8 stdout on Windows for plotext histograms
- soup ui: add --show-token flag, document auth token in --help
- 7 new tests (BUG-013/014/015), 1021 tests total
- ASCII progress bar in display.py (replace Unicode blocks with #/-)
- Catch UnicodeEncodeError from plotext in data stats (Windows fallback)
- Auto-disable 4bit/8bit quantization on CPU (bitsandbytes doesn't support it)
- Friendly error messages for CPU tensor/dtype/bf16/torchvision issues
- Fix diff.py: torch_dtype= -> dtype= (deprecation warning)
- Add torchvision version compatibility check to soup doctor
7 new tests (631 total), ruff clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BUG-001: Replace Unicode arrows/dashes with ASCII in all console output
to fix UnicodeEncodeError on Windows cp1252 (~10 commands affected)
- BUG-002: PPO trainer uses inspect.signature to detect trl parameter names
(ppo_epochs vs num_ppo_epochs) for trl 0.28.0 compatibility
- BUG-003: Add get_compute_dtype() - uses float32 on CPU, bfloat16/float16
on CUDA. Fixes dtype mismatch in reward model and all trainers
- BUG-004: Add warning when using quantization on CPU
- BUG-005: Fix dtype -> torch_dtype in diff.py model loading
- BUG-006: Pin wandb<0.18.0 to avoid trl import conflict, add runtime guard
13 new tests (624 total), ruff clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add `modality: vision` config option for vision-language model training
- Add LLaVA and ShareGPT4V data format detection and conversion
- Add `image_dir` field in DataConfig for resolving image paths
- Add vision model loading via AutoModelForVision2Seq + AutoProcessor in SFT trainer
- Add `soup init --template vision` with LLaMA-3.2-Vision config
- Add image statistics display in `soup data inspect` for vision datasets
- Add Pillow as optional `vision` extra dependency
- Add Pillow to `soup doctor` dependency checks
- 51 new tests (455 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix 23 ruff errors: line too long, unused imports, ambiguous vars
- Fix validator: empty string is valid data, only count None as empty
- Remove unused imports in display.py and validator.py
- Rename ambiguous `l` vars to `part`, `entry`, `length`
- Break long lines in callback.py, display.py, sft.py, constants.py
All 20 tests passing, ruff clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Typer CLI: soup init, soup train, soup data inspect/validate
- Pydantic config schema with YAML loader and validation
- Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets
- Format detection: Alpaca, ShareGPT, ChatML (auto-detect)
- SFT trainer wrapper over transformers + peft + trl
- QLoRA/LoRA support with auto batch size estimation
- GPU detection (CUDA/MPS/CPU) and memory calculation
- Rich live terminal dashboard for training monitoring
- Config templates: chat, code, medical
- Tests (pytest) + GitHub Actions CI
- MIT license
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alpamys