Adds a public plugin/hook system plus the schema scaffolding for 20+ ecosystem
integrations. Live trainer-callbacks, Anthropic /v1/messages route, server-tool
HTTP endpoints, and the recipe runner ship in v0.45.1 (matches v0.27.0 MII /
v0.37.0 multipack / v0.41.0 LLaMA Pro stub-then-live pattern).
Part A — Plugin / hook system
* New soup_cli/plugins/ package: BasePlugin Protocol, PluginSpec frozen
dataclass, register_plugin / discover_hooks / enable_plugin /
disable_plugin / load_plugins. Kebab-case name regex, semver-ish version,
null-byte rejection on every string. Idempotency check covers
(version, plugin object, templates, model_groups, description) — review-fix
added description after first-cut omitted it. Per-list caps on templates
and model_groups (32 entries, 128-char per name).
* New soup plugins list/install/enable/disable Typer CLI; all user-controlled
output passes through rich.markup.escape.
Part B — API extensions (schema-only)
* utils/anthropic_messages.py: to_anthropic / from_anthropic /
validate_anthropic_payload converters. Multiple system messages join with
\n\n; tool role with structured (list) content concatenated into single
tool_result text block (review-fix MEDIUM — first-cut silently dropped).
max_tokens cap 16384, temperature [0.0, 2.0], bool rejection on numerics.
* utils/server_tools.py: closed {python, bash, web_search} allowlist,
WebSearchConfig with domain allowlist + leading-dot subdomain pattern,
rate_limit [1, 600]. is_domain_allowed strips :port suffix and rejects
IPv6 literals (review-fix MEDIUM).
* utils/ngram_spec.py: NgramSpecConfig validators with bounded n / draft
tokens / prompt-lookup-max; bool rejection on every numeric field.
Part C — External integrations catalog
* utils/integrations.py: 15-entry MappingProxyType catalog of ecosystem
targets (lm-studio, comfyui, ollama, claude-code, cursor, continue, ...).
Part D — Advanced trainer-plugin allowlist
* utils/trainer_plugins.py: 6-entry allowlist (grokfast, spectrum,
llmcompressor, sonicmoe, cce_plugin, math_verify) + validate_trainer_
plugin_list (Sequence[str], dedup, _MAX_PLUGINS_PER_RUN=8).
Part E — Data Recipe DAG
* utils/recipe_dag.py: closed NODE_KINDS frozenset, Kahn's topological
sort via collections.deque (review-fix HIGH — first-cut had O(N^2 log N)
queue.sort() inside the BFS body), cycle / self-loop / dangling-edge
rejection, _MAX_NODES=256 / _MAX_EDGES=1024 / _MAX_FILE_BYTES=1MiB.
load_recipe_yaml enforces is_under_cwd containment AND os.lstat + S_ISLNK
symlink rejection (review-fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22
/ v0.43.0 Part C / v0.44.0 Part B policy).
* New soup data recipe <path> CLI validates topology and prints planned
topo order; live runner deferred to v0.45.1.
Reviews: python-review, security-review, code-review, tdd-guide all run;
verification-loop replaced by manual smoke (CLI happy + failure paths
exercised on real fixtures).
Test count: 5820 -> 5989 (+169). Test files: 164 -> 165. Ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes 3 originally-scheduled GitHub issues plus 7 v0.40.1 long-tail UX
papercuts. No new schema fields, no new trainers — pure polish.
Originally scheduled:
- #36 format_gate_row helper for the eval-gate dashboard row (pure formatter
in soup_cli/monitoring/display.py; passed=is True so missing field renders
neutral; supports stop/warn action suffixes; multi-task " | " join).
- #50 prepare_hf_resume now skips snapshot_download when local checkpoint-N
is greater-or-equal to the remote highest-N. New _find_highest_local_checkpoint
helper handles missing dirs / OSError / non-directories cleanly.
- #51 soup deploy hf-space --template-dir <path> via new
soup_cli/utils/hf_space.py:render_custom_template_dir. Containment via
is_under_cwd; validate_repo_id BEFORE substitution; per-file 256 KB cap;
symlinks + non-regular files rejected (TOCTOU defence per v0.33.0 #22).
v0.40.1 carry-overs:
- H2: data filter --min-coherence alias; data split --train no-op; data
register/unregister positional <name> <path> + Optional --name/--path
with conflict detection.
- H3: soup quickstart --output DIR (containment-checked) routes data,
config, run dir under the chosen directory.
- N1/G2: apply_logging_level pushes parsed --log-level tier into the root
logger so transformers / peft / trl actually respect QUIET / DEBUG.
- N7: shared _resolve_model_source in commands/infer.py (used by bench.py
too) — path-like-but-missing raises FileNotFoundError; non-path-like
values fall through to HF download via from_pretrained.
- G13: verified ONNX/AWQ/GPTQ/TensorRT install hints already correct.
- M4: verified data dedup --threshold already exposed.
- M5: soup runs --cwd-only + _filter_runs_by_cwd helper using
os.path.realpath + commonpath (Windows 8.3 + cross-drive safe).
Review-fix follow-ups landed in the same release:
- soup_cli/commands/infer.py: from __future__ import annotations (Py3.9
PEP 604 fix); --output containment via is_under_cwd, late-evaluated to
preserve pre-existing test contracts.
- soup_cli/commands/data.py register_data + soup_cli/commands/bench.py
prompts file: Path.resolve()+relative_to() → is_under_cwd (project rule
for Windows 8.3 short-name safety).
- soup_cli/commands/runs.py: typed _filter_runs_by_cwd, removed redundant
inner import os.
- soup_cli/commands/deploy.py: confirmation panel now shows --template-dir
path when set, not the unused --template default.
Tests: 4720 → 4756 (+36) across two new files (test_v0402_part_a.py,
test_v0402_part_b.py). 5 review agents (python / code / security / tdd /
verification) all clean after fixes.
Known limitations:
- Custom HF Space templates always create the Space with sdk=gradio
regardless of the supplied app.py. Use --template streamlit-chat with
the inline registry for Streamlit. Tracked for v0.40.3+.
- _resolve_model_source returns ("hf", repo_id) without validate_repo_id;
transformers.from_pretrained will raise loudly on malformed ids.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds soup why, soup tui, soup runs replay, soup train --profile, .crash
bundle on training exception, per-run cost in SQLite, --log-level global
flag. Net +110 tests (3818 → 3928); all five review-agent waves
(python / code / security / tdd / smoke) clean.
- Part A: --log-level quiet|normal|verbose|debug → Rich-formatted logger
on the "soup" namespace; idempotent + tier-change replaces handler.
- Part B: SQLite gains cost_usd / cost_gpu_label via lazy ALTER TABLE
(race-tolerant against duplicate-column on concurrent first-boot);
rendered in soup runs show / replay / TUI; bool num_gpus rejected;
LIKE wildcards escaped in tracker.get_run prefix match.
- Part C: soup why — heuristic NaN / plateau / divergence / grad-norm /
LR bounds; severity-ordered findings.
- Part D: .crash bundle generator with recursive hf_*/sk-*/Bearer
redaction, output_dir basename-only, os.path.realpath containment,
secrets.token_hex filename, ValueError (not PermissionError) on
outside-cwd; train.py except-handler writes the bundle without
masking the original exception.
- Part E: soup runs replay <id> — summary panel + downsampled loss
curve (≤2000 points) from SQLite history.
- Part F: soup train --profile — torch.profiler Chrome trace to
<output>/profiles/<run_id>.trace.json; run_id rejects '.', '..',
'/', '\\', null bytes; profiles dir created only on torch import.
- Part G: soup tui — Textual dashboard with lazy ExperimentTracker
import; markup_escape on every DB-sourced string; new [tui] extra.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs
* test(cost): add unit tests for 'soup cost' command and output formatting
* docs(readme): add usage documentation for the new 'soup cost' command
Foundation of v0.26.0 "Red and Blue Ocean" — every fine-tune is now
tracked with lineage, config, eval baseline, and shippable artifacts.
New module soup_cli/registry/:
- hashing.py: deterministic SHA-256 of config (canonical JSON) + data
(streamed) + base model; used as the entry_hash identity
- store.py: SQLite store (~/.soup/registry.db) with registry_entries,
registry_artifacts, registry_lineage, registry_tags. Context-manager
API, cycle-safe BFS walks, AmbiguousRefError on prefix collision,
LIKE-wildcard-escaped search + resolve, FK ON DELETE CASCADE.
- diff.py: flat-walk ConfigChange diff + per-benchmark eval delta.
New CLI commands:
- soup registry push/list/show/search/diff/promote/delete
- soup history <name> — lineage DAG tree viewer
Security hardening (v0.26.0):
- name/tag validation: alphanumeric + _-. only, null-byte rejected,
name ≤128, tag ≤64
- artifact path containment via os.path.realpath + commonpath
(Windows 8.3 short-name safe); enforce_cwd=True default
- SQL parameterised; LIKE wildcards %/_ escaped with ESCAPE '\'
- DB 600 perms on POSIX; SOUP_REGISTRY_DB_PATH env override
- indirect-cycle detection in add_lineage via BFS ancestor walk
- Rich markup escaped in all CLI output
- resolve() raises AmbiguousRefError instead of silent None
Tests: 92 new tests in tests/test_registry.py (hashing, validation,
CRUD, artifacts, lineage + cycle, diff, CLI, history, security,
auto-register integration with ExperimentTracker). Full suite:
2409 passed (was 2313).
All review findings addressed (4 agents: python, code, security, tdd):
HIGH: context manager + try/finally cleanup, FK cascade (removed
manual cascade), cycle detection, LIKE wildcard escaping.
MEDIUM: ambiguous resolve raises, exit 0 on user cancel, cwd
captured at construction, enforce_cwd=True default, Windows
ASCII-safe error messages.
Deferred to v0.26.1: soup eval --attach-to-registry flag and
soup export auto-artifact registration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): create 'soup bench' command for inference speed and VRAM measurement
* register 'bench' command into the main CLI router
* add test case for handling missing model paths gracefully
* add 'Inference Benchmarking' section explaining the 'soup bench' tool
* Added soup.yaml
* style: fix linting (unused imports, inconsistent spacing)
* style: sort imports in bench and test_bench to satisfy ruff
* style: final import sort and grouping fix for CI
* Update gitignore
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every
competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators:
Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo,
Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA).
Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with
detection + normalization, synth data template, init template, eval scoring
(tool_call_match / tool_call_name_match / tool_call_args_subset), plus
qwen3-8b-tools and llama4-scout-tools recipes.
Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to
math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox
with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap,
one-time warning panel), and json_schema_reward. verifiable_domain Literal
validated via model_validator.
Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with
mutual-exclusion validator and a unified peft_builder helper that returns
either LoraConfig or VeraConfig with the right init kwargs.
Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx,
MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at
config load time by SoupConfig._validate_mlx_task_support, lazy trainer
registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject.
Part F — Data augmentation: soup data augment with rephrase / translate / style
strategies, path-traversal-protected input/output, count capped 1-10, lang/styles
lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup.
Part G — Training intelligence: forgetting detection (ForgettingDetector with
3 built-in mini benchmarks and warning levels) and checkpoint intelligence
(CheckpointTracker with composite metric, early-stop on regression, safe
top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema
extended with checkpoint_quality + forgetting_eval tables.
Part H — Autopilot: soup autopilot command with dataset/model/hardware
profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf
flags), YAML generator, and full CLI with dry-run + --yes + path-traversal
protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting
detection + checkpoint intelligence + early-stop into the generated config.
Totals:
- 2313 tests passing (183 new, up from 2130)
- 86 test files (8 new)
- 43 ready-made recipes (14 new)
- 16 built-in templates (tool-calling added)
- Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented
design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU,
MLX training integration test requires real hardware)
Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add --json flag to version command for machine-readable output in CI/scripts and include tests
* docs: update README with soup version --json flag examples
Show GitHub URL in `soup version`, `soup version --full`, `soup doctor`,
and `soup --help` so users can find and star the repo.
Extract URL to GITHUB_URL constant in utils/constants.py.
- Add --backend vllm flag to soup serve for 2-4x better inference throughput
- Add --tensor-parallel and --gpu-memory flags for vLLM tuning
- Auto-detect vLLM and show hint when installed but not enabled
- New utils/vllm.py with engine creation, app factory, LoRA support
- Native token-by-token streaming via vLLM AsyncLLMEngine
- Add serve-fast extra: pip install 'soup-cli[serve-fast]'
- Add vllm detection to version --full
- 30 new tests (560 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `soup ui` command launches local web interface at http://127.0.0.1:7860
- FastAPI backend with REST API: runs, metrics, config validation, training
control, data inspection, templates, system info
- Self-contained SPA frontend (Dashboard, New Training, Data Explorer,
Model Chat) with Chart.js loss/LR charts
- Auto-opens browser on launch (--no-browser to disable)
- Config validation via new load_config_from_string() in config/loader.py
- 40 new tests (530 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `soup version --full` shows version, Python, GPU backend, installed extras
- Dynamic test count badge via Gist endpoint in CI
- README: Optional Extras table, --verbose note, CSV/Parquet, Changelog link
- 323 tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Friendly error messages: wrap all commands in try/except, map known errors
(CUDA OOM, missing deps, connection errors) to 2-3 line messages with fix hints
- Global --verbose flag for full tracebacks
- soup doctor: check system info, GPU, all dependency versions with fix suggestions
- soup quickstart: one-command demo (creates data + config + trains TinyLlama)
- Confirmation prompts before train/sweep (skip with --yes)
- 40 new tests (321 total), all passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- soup serve: FastAPI inference server with OpenAI-compatible API, SSE streaming
- soup data generate: synthetic data generation via OpenAI API or local models
- soup sweep: grid/random hyperparameter search with experiment tracker integration
- soup diff: side-by-side model comparison with metrics
- Multi-GPU/DeepSpeed: ZeRO Stage 2/3 configs, --deepspeed flag in train command
- 95 new tests (281 total), all passing
- Removed TESTING_GUIDE.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Typer CLI: soup init, soup train, soup data inspect/validate
- Pydantic config schema with YAML loader and validation
- Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets
- Format detection: Alpaca, ShareGPT, ChatML (auto-detect)
- SFT trainer wrapper over transformers + peft + trl
- QLoRA/LoRA support with auto batch size estimation
- GPU detection (CUDA/MPS/CPU) and memory calculation
- Rich live terminal dashboard for training monitoring
- Config templates: chat, code, medical
- Tests (pytest) + GitHub Actions CI
- MIT license
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alpamys
Salil M
Salil