Closes#32, #35. (#34 soup can run/publish deferred to Part A wave 2.)
#32 Live model scoring for `soup eval gate` + `soup eval quant-check`:
- gate.run_gate now dispatches judge / benchmark / custom task types,
wrapping each scorer in try/except so a backend failure produces
score=None + error=str(exc) instead of a silent score=1.0 pass.
- New _parse_judge_url splits ollama:// / http(s):// judge_model URLs
into (provider, model, api_base) for JudgeEvaluator.
- New _run_judge_task / _run_benchmark_task plug into existing
eval/judge.py and eval/forgetting.py runners.
- New quant_check.make_model_generator(model_path) wraps transformers
AutoTokenizer + AutoModelForCausalLM into a generate_fn callable;
greedy by default for reproducible scores; lazy-imported.
- gate_cmd / quant_check_cmd build live generators when --model is
given; fall back to deterministic stub on load failure so CI without
GPUs still runs the orchestration layer.
- GateTaskResult.score is now Optional[float] with new error: Optional[str].
- _print_gate_result renders ERROR + reason cleanly.
#35 Registry attach hooks:
- registry/store.py _VALID_KINDS extended with eval_results, tensorrt.
- New registry/attach.py: attach_artifact, write_eval_json
(cwd-containment via realpath+commonpath), lookup_entry_by_output_dir.
- `soup eval custom` gains --attach-to-registry + --output (paired);
on success writes JSON results and adds eval_results artifact row.
- `soup export` gains --registry-id with auto-match by source --model
output dir; auto-attaches the produced GGUF artifact. Failures here
are warnings, not hard exits — export already succeeded.
Tests: +19 in tests/test_part_a_wave1.py covering URL parser, error
propagation across all 3 task types, score=None semantics, generator
factory bounds + transformers mocking, registry attach helpers
(containment + missing entry), and CLI integration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#21, #22.
#21 RLVR code_exec_reward: add OS-level isolation strategy detection.
- New _get_isolation_strategy / _compute_isolation_strategy with cache.
- Linux: best-effort os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)
in preexec_fn (Python 3.12+). Silent fallback on EPERM/ENOSYS for
hosts where unprivileged user namespaces are disabled.
- macOS: prefix subprocess argv with sandbox-exec + inline default-deny
profile (deny network*, deny writes outside /tmp).
- Windows + restricted Linux: existing RLIMIT + socket-patch + ephemeral
cwd guards continue to apply (best-effort baseline).
#22 prune_checkpoints: TOCTOU-safe symlink handling.
- Top-level entries: explicit os.lstat + stat.S_ISLNK check (intent-clear)
instead of Path.is_symlink.
- shutil.rmtree now passes onerror=_abort_on_symlink to abort recursive
walk if any symlink is encountered mid-walk (defence-in-depth).
- OSError mid-prune is caught per-checkpoint so one bad dir does not
abort the whole prune pass.
Tests: +13 in tests/test_part_f_hardening.py covering strategy detection
on linux/darwin/win32, sandbox profile shape, code_exec smoke tests, and
TOCTOU-resistant prune behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rich/Typer emits per-character ANSI escapes in CI on Linux runners
("--\x1b[m-find\x1b[m-lr") so the raw substring assertion `--find-lr`
in result.output fails on ubuntu-latest x py3.12 even though it
passes on Windows where Rich auto-disables colour.
Strip ANSI before comparing — same pattern already used by
test_hf_integration.py and test_eval_platform.py.
Tests-only commit, no version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Expand the recipe catalog from 46 to 80 entries — every popular open-weight
model family now has a validated Soup recipe.
Part A — Vision (6 recipes): Llama-3.2-Vision-90B, Pixtral-12B, Qwen2-VL
(7B + 72B), InternVL 2.5, MiniCPM-V 2.6
Part B — Audio (3 recipes): Qwen2-Audio, SeamlessM4T v2, Whisper-large-v3
Part C — Reasoning (7 recipes): completes the 6 DeepSeek-R1-Distill sizes,
plus Qwen3-Coder, Qwen3-30B-A3B reasoning, Phi-4 reasoning
Part D — Edge (8 recipes): SmolLM2 (135M / 360M / 1.7B), Qwen2.5
(0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini
Part E — Domain (8 recipes): BioMistral, Meditron, CodeLlama (13B / 70B),
Magicoder, Mathstral, Nemotron-4 340B, Llama-2-13b-finance
Part F — Multimodal reasoning (2 recipes): Llama-3.2-Vision GRPO, Pixtral DPO
Part G — Recipe-validation CI workflow on every PR touching recipe / config /
data code (.github/workflows/recipe-validation.yml)
Part H — 750 parametrized tests covering catalog-wide invariants:
model-id safety (no `..`/`://`/null bytes), lora.target_modules non-empty,
max_length within schema bounds, GRPO recipes wire reward_fn +
num_generations >= 2, vision recipes set image_dir, audio recipes set
audio_dir, default data path is non-empty + relative
Live 100-step per-recipe smoke train (requires GPU runner) deferred to v0.31.1.
Tests: 2886 → 3607 (+721). Catalog: 46 → 80 (target met).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ubuntu-latest py3.11 matrix cell doesn't install the [serve] extra,
so `soup serve` exits early with a FastAPI-missing message before
reaching --structured-output / --auto-quant / --json-schema validation.
Add `pytest.importorskip("fastapi")` to the three tests that exercise
those CLI-level validation paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI failures on macOS/Windows:
1. `test_endpoint_rejects_null_byte` — ``monkeypatch.setenv("HF_ENDPOINT",
"...\x00")`` raises ``ValueError: embedded null byte`` at the C-level
setenv call on macOS/Windows before ``resolve_endpoint`` can reject it.
Linux's setenv swallows it. Replace with ``monkeypatch.setattr`` on
``os.environ`` dict so the null-byte string reaches ``resolve_endpoint``
on every platform.
2. Help-text substring tests (``test_train_shows_push_as_flag_in_help``,
``test_push_shows_collection_flag_in_help``, ``test_push_subcommand_exists``,
``test_hf_space_help_shows_flags``, ``test_train_help_shows_hf_resume``,
``test_hf_space_command_registered``) — Typer injects ANSI escape codes
on macOS/Windows pytest runs, splitting tokens like ``--push-as`` into
``-`` + ``-push-as`` across escape groups. Add ``_plain()`` helper that
strips ANSI via regex and use it in every help-text assertion.
Same pattern as 899ad8e (test_eval_gate.py) did after the v0.26.0 CI
Windows failure.
Full suite still passes locally: 100 HF integration tests in 3.25s.
- Split the 22-assert config-values test into 3 focused tests
(task+data, training hyperparams, LoRA config) so a deliberate
example change surfaces in one targeted test, not a wall of asserts
- Add module docstring explaining why these tests lock the example state
- Add `from __future__ import annotations` (defensive; matches 14 other
test modules in the project)
- Rename `f` -> `fh` in _load_jsonl to avoid shadowing short name
- Drop asserts on secondary fields (warmup_ratio, weight_decay, scheduler,
logging_steps, etc.) -- they're tweakable knobs, not the example's
teaching points; test brittleness > coverage here
Add a working DPO (Direct Preference Optimization) example using the
current Pydantic config schema with Llama 3.1 8B Instruct and QLoRA.
- examples/configs/dpo_example.yaml: DPO config with all core training
and LoRA parameters, plus commented-out advanced options
- examples/data/dpo_sample.jsonl: 8 preference pairs in DPO format
with ShareGPT-style message lists for chosen/rejected
- tests/test_dpo_example.py: 7 tests validating config loading, field
values, data format detection, and data validation
- examples/README.md: document the new DPO with QLoRA example
- Narrow 'except Exception: pass' in _get_dataset_size to specific
exceptions (OSError, ValueError, KeyError, ImportError)
- _get_dataset_size returns (size, is_estimated) so the caller can
warn when falling back to the 10k default (silent fallbacks are
misleading on a $-estimating command)
- Add -> None return type annotation on cost() (project convention)
- Add variance disclaimer: 'estimates are approximate; +/- 30%'
- Document pricing cadence in GPU_PRICING comment (last updated 2026-04)
- Use highlight=False on json.dumps output
- Fix misleading 'mock data' test comment (there is no mock)
- Add 2 tests: dataset-unreadable warning, variance disclaimer rendering
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs
* test(cost): add unit tests for 'soup cost' command and output formatting
* docs(readme): add usage documentation for the new 'soup cost' command
`test_train_gate_flag_accepted` asserted `"--gate" in result.output`, but
Typer/Click under CI emits ANSI color codes that split the flag name into
non-contiguous chars: `\x1b[1;36m-\x1b[0m\x1b[1;36m-gate\x1b[0m`. The literal
"--gate" substring is never present. All 9 OS × Python combos failed on the
v0.26.0 Parts B-E push.
Fix: strip ANSI via regex before checking. Also assert on "eval-gated" from
the option description to double-check the flag is wired to its help text.
CI-only / tests-only: no soup_cli/ changes, no version bump needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Foundation of v0.26.0 "Red and Blue Ocean" — every fine-tune is now
tracked with lineage, config, eval baseline, and shippable artifacts.
New module soup_cli/registry/:
- hashing.py: deterministic SHA-256 of config (canonical JSON) + data
(streamed) + base model; used as the entry_hash identity
- store.py: SQLite store (~/.soup/registry.db) with registry_entries,
registry_artifacts, registry_lineage, registry_tags. Context-manager
API, cycle-safe BFS walks, AmbiguousRefError on prefix collision,
LIKE-wildcard-escaped search + resolve, FK ON DELETE CASCADE.
- diff.py: flat-walk ConfigChange diff + per-benchmark eval delta.
New CLI commands:
- soup registry push/list/show/search/diff/promote/delete
- soup history <name> — lineage DAG tree viewer
Security hardening (v0.26.0):
- name/tag validation: alphanumeric + _-. only, null-byte rejected,
name ≤128, tag ≤64
- artifact path containment via os.path.realpath + commonpath
(Windows 8.3 short-name safe); enforce_cwd=True default
- SQL parameterised; LIKE wildcards %/_ escaped with ESCAPE '\'
- DB 600 perms on POSIX; SOUP_REGISTRY_DB_PATH env override
- indirect-cycle detection in add_lineage via BFS ancestor walk
- Rich markup escaped in all CLI output
- resolve() raises AmbiguousRefError instead of silent None
Tests: 92 new tests in tests/test_registry.py (hashing, validation,
CRUD, artifacts, lineage + cycle, diff, CLI, history, security,
auto-register integration with ExperimentTracker). Full suite:
2409 passed (was 2313).
All review findings addressed (4 agents: python, code, security, tdd):
HIGH: context manager + try/finally cleanup, FK cascade (removed
manual cascade), cycle detection, LIKE wildcard escaping.
MEDIUM: ambiguous resolve raises, exit 0 on user cancel, cwd
captured at construction, enforce_cwd=True default, Windows
ASCII-safe error messages.
Deferred to v0.26.1: soup eval --attach-to-registry flag and
soup export auto-artifact registration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- happy_path: assert mocked VRAM value (4.00 GB) renders in table
- happy_path: assert 'Benchmarking Configuration' panel rendered
- happy_path: assert mock_generate call count (1 warmup + 3 prompts)
- cpu_warning: assert 'N/A' appears in VRAM column when no CUDA
- Improve docstrings to describe what each test verifies
- Use full exception repr in exit_code asserts for CI debugging
- Narrow broad 'except Exception' to specific exceptions
(OSError, UnicodeDecodeError, json.JSONDecodeError) + `raise ... from`
- Rename file handle `f` -> `fh` to avoid shadowing (ruff-friendly)
- Clarify comment on --num-prompts ignored-when-file semantics
- Strengthen test: assert actual prompts were passed to _generate
(not just exit code + output substring)
- Add PEP 8 second blank line between test functions
* feat(bench): add --prompts-file option with path traversal security
* test(bench): add unit tests for custom prompts and path traversal
* docs(bench): document --prompts-file usage in README.md
* feat(bench): add --prompts-file support with path validation
* test(bench): add unit tests for custom prompts and security checks
* style: remove trailing whitespace to pass ruff linting
* test: fix mock patch targets for local imports in bench command
* refactor(bench): simplify prompts-file logic and clean up comment
* test(bench): update assertions to match new prompts-file semantics
* feat(cli): create 'soup bench' command for inference speed and VRAM measurement
* register 'bench' command into the main CLI router
* add test case for handling missing model paths gracefully
* add 'Inference Benchmarking' section explaining the 'soup bench' tool
* Added soup.yaml
* style: fix linting (unused imports, inconsistent spacing)
* style: sort imports in bench and test_bench to satisfy ruff
* style: final import sort and grouping fix for CI
* Update gitignore
test_writes_config fails on windows-latest / Python 3.9 with exit code 1
because the path-traversal check in soup_cli/commands/autopilot.py was:
data_path = Path(data).resolve()
data_path.relative_to(Path.cwd().resolve())
On Windows + Python 3.9, Path.resolve() occasionally leaves 8.3 short
names (e.g. "C:\Users\RUNNER~1") in one of the two sides but not the
other, so relative_to raises ValueError even when both paths point to
the same location. GitHub Actions runner home dirs frequently trigger
this (the runneradmin account is created as "runneradmin" but short
names get generated as "RUNNER~1").
Fix: introduce _is_under_cwd(path) helper in soup_cli/commands/autopilot.py
that uses os.path.realpath on both sides (handles 8.3 expansion
consistently) plus os.path.commonpath for the containment check, with
case-insensitive comparison on NT. Apply it to both the --data and
--output path guards. The data_path / output_path locals are then
rebuilt from the realpath result so downstream logic sees the
canonical long-name path.
Also enriches the test assertion to print result.output and
result.exception on failure so future CI breaks are easier to diagnose
without needing to push a debug commit first.
Local verification: all 38 tests in tests/test_autopilot.py pass on
Python 3.10 Windows, full suite 2313 passed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two tests in tests/test_bugfixes.py::TestGRPOCPUMinNewTokens fail on
windows-latest / Python 3.11 when importing trl.trainer.grpo_trainer:
RuntimeError: Failed to import trl.trainer.grpo_trainer because of
the following error:
'charmap' codec can't decode byte 0x90 in position 6555: character
maps to <undefined>
Root cause: upstream trl reads an auxiliary file without an explicit
encoding, so Python uses the system default. On Windows that is cp1252
('charmap'), which chokes on non-ASCII bytes present in the file. This
is an upstream issue but Soup needs a green CI.
Two-layer fix:
1. .github/workflows/ci.yml — set PYTHONUTF8=1 and PYTHONIOENCODING=utf-8
as job-level env. Python's UTF-8 mode makes all file I/O default to
UTF-8 regardless of locale, which is the correct global fix for this
class of bug.
2. tests/test_bugfixes.py — add a _trl_grpo_importable() helper that
returns False on UnicodeDecodeError / ImportError / RuntimeError, and
use it as a belt-and-braces skip in both TestGRPOCPUMinNewTokens
tests. Ensures the tests skip cleanly instead of erroring out if a
future CI change accidentally drops PYTHONUTF8.
Local verification: both tests pass with 'pytest tests/test_bugfixes.py::
TestGRPOCPUMinNewTokens -v' (Python 3.10, Windows).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every
competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators:
Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo,
Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA).
Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with
detection + normalization, synth data template, init template, eval scoring
(tool_call_match / tool_call_name_match / tool_call_args_subset), plus
qwen3-8b-tools and llama4-scout-tools recipes.
Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to
math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox
with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap,
one-time warning panel), and json_schema_reward. verifiable_domain Literal
validated via model_validator.
Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with
mutual-exclusion validator and a unified peft_builder helper that returns
either LoraConfig or VeraConfig with the right init kwargs.
Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx,
MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at
config load time by SoupConfig._validate_mlx_task_support, lazy trainer
registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject.
Part F — Data augmentation: soup data augment with rephrase / translate / style
strategies, path-traversal-protected input/output, count capped 1-10, lang/styles
lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup.
Part G — Training intelligence: forgetting detection (ForgettingDetector with
3 built-in mini benchmarks and warning levels) and checkpoint intelligence
(CheckpointTracker with composite metric, early-stop on regression, safe
top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema
extended with checkpoint_quality + forgetting_eval tables.
Part H — Autopilot: soup autopilot command with dataset/model/hardware
profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf
flags), YAML generator, and full CLI with dry-run + --yes + path-traversal
protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting
detection + checkpoint intelligence + early-stop into the generated config.
Totals:
- 2313 tests passing (183 new, up from 2130)
- 86 test files (8 new)
- 43 ready-made recipes (14 new)
- 16 built-in templates (tool-calling added)
- Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented
design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU,
MLX training integration test requires real hardware)
Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): add 'soup runs clean' intelligent checkpoint cleanup to reclaim disk space
* update README
* feat(cli): add 'soup runs clean' intelligent checkpoint cleanup to reclaim disk space
* fixed whitespace trails
* style(cli): fix lints (line length and spacing) in runs.py
* style: fix all E501 line length lint errors
* fix test mismatch, improve deletion warnings, add path validation, and enforce argument exclusivity
* fix: break long message into multiple lines for Ruff compliance
* test: update runs clean test to use CWD-based output directory for security compliance
* feat(doctor): add RAM and disk space checks to soup doctor command with tests and updated docs
* fix(doctor): resolve subprocess type checker error by manually validating macOS RAM query return code
* add --json flag to version command for machine-readable output in CI/scripts and include tests
* docs: update README with soup version --json flag examples
- Replace Windows-only paths (C:/Windows/...) with tempdir-based
paths that work on Linux/macOS CI runners
- Use monkeypatch.chdir for path traversal test isolation
- Fixes test_path_outside_cwd_raises failure on Ubuntu CI
- Replace non-ASCII symbols (checkmarks, arrows, bullets, em-dashes)
with ASCII equivalents in Rich console output to prevent
UnicodeEncodeError on Windows without PYTHONIOENCODING=utf-8
- Add _validate_output_path() for AWQ/GPTQ export — output path
traversal is now checked before import check (previously unreachable
when autoawq/auto-gptq not installed)
- 4 new tests for output path validation (2065 total, 0 failures)
- Update SECURITY.md with v0.22.0–v0.24.1 hardening history
Part A: HuggingFace Dataset browser
- soup data search: search HF Hub for datasets (sort by downloads/likes)
- soup data preview: preview remote dataset metadata, splits, features
- soup data download: stream HF dataset to local JSONL (with format conversion)
- Security: trust_remote_code=False, path traversal protection, samples cap at 1M
Part B: Freeze training (like LLaMA-Factory finetuning_type: freeze)
- freeze_layers / freeze_ratio config fields
- soup_cli/utils/freeze.py: detect layers, freeze bottom N
- Wired into SFT trainer before LoRA application
- Supports LLaMA (layers.N) and GPT-2 (h.N) naming
Part C: Loss watchdog (like Axolotl loss_watchdog_threshold)
- loss_watchdog, loss_watchdog_threshold, loss_watchdog_patience config
- Implemented in SoupTrainerCallback with patience counter
- Rich warning panel (stops Live display first), fires only once
- Wired into all 11 trainers via callback kwargs
Part D: Dataset info registry
- soup data register/unregister/registry commands
- ~/.soup/datasets.json local name→path+format mapping
- Name validation, path traversal protection, Rich markup escaping
82 new tests (2061 total), 74 test files.
Rich/Typer truncates help panel on narrow terminals (macOS CI), causing
--bits and --group-size flags to not appear in rendered help text. Switch
to inspecting the function signature directly for cross-platform reliability.
- AWQ export (`soup export --format awq`) via autoawq, with --bits, --group-size, --calibration-data
- GPTQ export (`soup export --format gptq`) via auto-gptq, with calibration data support
- Sample packing (`packing: true`) for SFT/Pretrain trainers via TRL's native packing
- `soup data split` — train/val/test splitting with random and stratified strategies
- Curriculum learning (`curriculum: true`) — sort dataset by difficulty for staged training
- New utility: soup_cli/utils/curriculum.py (sort_by_length, create_buckets)
- Security: calibration data path traversal protection, bits validation (4/8 only)
- 1970 tests across 70 test files
- Extract _parse_json_array into soup_cli/data/providers/_utils.py to
avoid circular imports between generate.py and provider modules.
- Narrow bare except Exception in detect_ollama to httpx.HTTPError/OSError
with debug logging instead of silent swallow.
Replace simple '..' check with resolve() + relative_to(cwd) for output
path. Add same confinement guard to --seed, --dedup-with, and --context
file paths. Add _path_within_cwd helper. 4 new security tests.
Rich markup wraps --model-a with ANSI codes on macOS, breaking the
substring check. Strip ANSI codes before asserting, matching the
existing pattern in test_speculative_decoding.py and test_deploy_ollama.py.
Rich markup in Typer help output inserts ANSI escape codes around
--flag names on macOS, breaking exact string matches. Check for
lowercase words instead of --prefixed flags.
Alpamys
Chinmaya Sahu
Salil M
Salil Mhatre