`test_train_gate_flag_accepted` asserted `"--gate" in result.output`, but
Typer/Click under CI emits ANSI color codes that split the flag name into
non-contiguous chars: `\x1b[1;36m-\x1b[0m\x1b[1;36m-gate\x1b[0m`. The literal
"--gate" substring is never present. All 9 OS × Python combos failed on the
v0.26.0 Parts B-E push.
Fix: strip ANSI via regex before checking. Also assert on "eval-gated" from
the option description to double-check the flag is wired to its help text.
CI-only / tests-only: no soup_cli/ changes, no version bump needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Foundation of v0.26.0 "Red and Blue Ocean" — every fine-tune is now
tracked with lineage, config, eval baseline, and shippable artifacts.
New module soup_cli/registry/:
- hashing.py: deterministic SHA-256 of config (canonical JSON) + data
(streamed) + base model; used as the entry_hash identity
- store.py: SQLite store (~/.soup/registry.db) with registry_entries,
registry_artifacts, registry_lineage, registry_tags. Context-manager
API, cycle-safe BFS walks, AmbiguousRefError on prefix collision,
LIKE-wildcard-escaped search + resolve, FK ON DELETE CASCADE.
- diff.py: flat-walk ConfigChange diff + per-benchmark eval delta.
New CLI commands:
- soup registry push/list/show/search/diff/promote/delete
- soup history <name> — lineage DAG tree viewer
Security hardening (v0.26.0):
- name/tag validation: alphanumeric + _-. only, null-byte rejected,
name ≤128, tag ≤64
- artifact path containment via os.path.realpath + commonpath
(Windows 8.3 short-name safe); enforce_cwd=True default
- SQL parameterised; LIKE wildcards %/_ escaped with ESCAPE '\'
- DB 600 perms on POSIX; SOUP_REGISTRY_DB_PATH env override
- indirect-cycle detection in add_lineage via BFS ancestor walk
- Rich markup escaped in all CLI output
- resolve() raises AmbiguousRefError instead of silent None
Tests: 92 new tests in tests/test_registry.py (hashing, validation,
CRUD, artifacts, lineage + cycle, diff, CLI, history, security,
auto-register integration with ExperimentTracker). Full suite:
2409 passed (was 2313).
All review findings addressed (4 agents: python, code, security, tdd):
HIGH: context manager + try/finally cleanup, FK cascade (removed
manual cascade), cycle detection, LIKE wildcard escaping.
MEDIUM: ambiguous resolve raises, exit 0 on user cancel, cwd
captured at construction, enforce_cwd=True default, Windows
ASCII-safe error messages.
Deferred to v0.26.1: soup eval --attach-to-registry flag and
soup export auto-artifact registration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- happy_path: assert mocked VRAM value (4.00 GB) renders in table
- happy_path: assert 'Benchmarking Configuration' panel rendered
- happy_path: assert mock_generate call count (1 warmup + 3 prompts)
- cpu_warning: assert 'N/A' appears in VRAM column when no CUDA
- Improve docstrings to describe what each test verifies
- Use full exception repr in exit_code asserts for CI debugging
- Narrow broad 'except Exception' to specific exceptions
(OSError, UnicodeDecodeError, json.JSONDecodeError) + `raise ... from`
- Rename file handle `f` -> `fh` to avoid shadowing (ruff-friendly)
- Clarify comment on --num-prompts ignored-when-file semantics
- Strengthen test: assert actual prompts were passed to _generate
(not just exit code + output substring)
- Add PEP 8 second blank line between test functions
* feat(bench): add --prompts-file option with path traversal security
* test(bench): add unit tests for custom prompts and path traversal
* docs(bench): document --prompts-file usage in README.md
* feat(bench): add --prompts-file support with path validation
* test(bench): add unit tests for custom prompts and security checks
* style: remove trailing whitespace to pass ruff linting
* test: fix mock patch targets for local imports in bench command
* refactor(bench): simplify prompts-file logic and clean up comment
* test(bench): update assertions to match new prompts-file semantics
* feat(cli): create 'soup bench' command for inference speed and VRAM measurement
* register 'bench' command into the main CLI router
* add test case for handling missing model paths gracefully
* add 'Inference Benchmarking' section explaining the 'soup bench' tool
* Added soup.yaml
* style: fix linting (unused imports, inconsistent spacing)
* style: sort imports in bench and test_bench to satisfy ruff
* style: final import sort and grouping fix for CI
* Update gitignore
test_writes_config fails on windows-latest / Python 3.9 with exit code 1
because the path-traversal check in soup_cli/commands/autopilot.py was:
data_path = Path(data).resolve()
data_path.relative_to(Path.cwd().resolve())
On Windows + Python 3.9, Path.resolve() occasionally leaves 8.3 short
names (e.g. "C:\Users\RUNNER~1") in one of the two sides but not the
other, so relative_to raises ValueError even when both paths point to
the same location. GitHub Actions runner home dirs frequently trigger
this (the runneradmin account is created as "runneradmin" but short
names get generated as "RUNNER~1").
Fix: introduce _is_under_cwd(path) helper in soup_cli/commands/autopilot.py
that uses os.path.realpath on both sides (handles 8.3 expansion
consistently) plus os.path.commonpath for the containment check, with
case-insensitive comparison on NT. Apply it to both the --data and
--output path guards. The data_path / output_path locals are then
rebuilt from the realpath result so downstream logic sees the
canonical long-name path.
Also enriches the test assertion to print result.output and
result.exception on failure so future CI breaks are easier to diagnose
without needing to push a debug commit first.
Local verification: all 38 tests in tests/test_autopilot.py pass on
Python 3.10 Windows, full suite 2313 passed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two tests in tests/test_bugfixes.py::TestGRPOCPUMinNewTokens fail on
windows-latest / Python 3.11 when importing trl.trainer.grpo_trainer:
RuntimeError: Failed to import trl.trainer.grpo_trainer because of
the following error:
'charmap' codec can't decode byte 0x90 in position 6555: character
maps to <undefined>
Root cause: upstream trl reads an auxiliary file without an explicit
encoding, so Python uses the system default. On Windows that is cp1252
('charmap'), which chokes on non-ASCII bytes present in the file. This
is an upstream issue but Soup needs a green CI.
Two-layer fix:
1. .github/workflows/ci.yml — set PYTHONUTF8=1 and PYTHONIOENCODING=utf-8
as job-level env. Python's UTF-8 mode makes all file I/O default to
UTF-8 regardless of locale, which is the correct global fix for this
class of bug.
2. tests/test_bugfixes.py — add a _trl_grpo_importable() helper that
returns False on UnicodeDecodeError / ImportError / RuntimeError, and
use it as a belt-and-braces skip in both TestGRPOCPUMinNewTokens
tests. Ensures the tests skip cleanly instead of erroring out if a
future CI change accidentally drops PYTHONUTF8.
Local verification: both tests pass with 'pytest tests/test_bugfixes.py::
TestGRPOCPUMinNewTokens -v' (Python 3.10, Windows).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every
competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators:
Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo,
Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA).
Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with
detection + normalization, synth data template, init template, eval scoring
(tool_call_match / tool_call_name_match / tool_call_args_subset), plus
qwen3-8b-tools and llama4-scout-tools recipes.
Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to
math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox
with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap,
one-time warning panel), and json_schema_reward. verifiable_domain Literal
validated via model_validator.
Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with
mutual-exclusion validator and a unified peft_builder helper that returns
either LoraConfig or VeraConfig with the right init kwargs.
Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx,
MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at
config load time by SoupConfig._validate_mlx_task_support, lazy trainer
registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject.
Part F — Data augmentation: soup data augment with rephrase / translate / style
strategies, path-traversal-protected input/output, count capped 1-10, lang/styles
lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup.
Part G — Training intelligence: forgetting detection (ForgettingDetector with
3 built-in mini benchmarks and warning levels) and checkpoint intelligence
(CheckpointTracker with composite metric, early-stop on regression, safe
top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema
extended with checkpoint_quality + forgetting_eval tables.
Part H — Autopilot: soup autopilot command with dataset/model/hardware
profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf
flags), YAML generator, and full CLI with dry-run + --yes + path-traversal
protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting
detection + checkpoint intelligence + early-stop into the generated config.
Totals:
- 2313 tests passing (183 new, up from 2130)
- 86 test files (8 new)
- 43 ready-made recipes (14 new)
- 16 built-in templates (tool-calling added)
- Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented
design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU,
MLX training integration test requires real hardware)
Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): add 'soup runs clean' intelligent checkpoint cleanup to reclaim disk space
* update README
* feat(cli): add 'soup runs clean' intelligent checkpoint cleanup to reclaim disk space
* fixed whitespace trails
* style(cli): fix lints (line length and spacing) in runs.py
* style: fix all E501 line length lint errors
* fix test mismatch, improve deletion warnings, add path validation, and enforce argument exclusivity
* fix: break long message into multiple lines for Ruff compliance
* test: update runs clean test to use CWD-based output directory for security compliance
* feat(doctor): add RAM and disk space checks to soup doctor command with tests and updated docs
* fix(doctor): resolve subprocess type checker error by manually validating macOS RAM query return code
* add --json flag to version command for machine-readable output in CI/scripts and include tests
* docs: update README with soup version --json flag examples
- Replace Windows-only paths (C:/Windows/...) with tempdir-based
paths that work on Linux/macOS CI runners
- Use monkeypatch.chdir for path traversal test isolation
- Fixes test_path_outside_cwd_raises failure on Ubuntu CI
- Replace non-ASCII symbols (checkmarks, arrows, bullets, em-dashes)
with ASCII equivalents in Rich console output to prevent
UnicodeEncodeError on Windows without PYTHONIOENCODING=utf-8
- Add _validate_output_path() for AWQ/GPTQ export — output path
traversal is now checked before import check (previously unreachable
when autoawq/auto-gptq not installed)
- 4 new tests for output path validation (2065 total, 0 failures)
- Update SECURITY.md with v0.22.0–v0.24.1 hardening history
Part A: HuggingFace Dataset browser
- soup data search: search HF Hub for datasets (sort by downloads/likes)
- soup data preview: preview remote dataset metadata, splits, features
- soup data download: stream HF dataset to local JSONL (with format conversion)
- Security: trust_remote_code=False, path traversal protection, samples cap at 1M
Part B: Freeze training (like LLaMA-Factory finetuning_type: freeze)
- freeze_layers / freeze_ratio config fields
- soup_cli/utils/freeze.py: detect layers, freeze bottom N
- Wired into SFT trainer before LoRA application
- Supports LLaMA (layers.N) and GPT-2 (h.N) naming
Part C: Loss watchdog (like Axolotl loss_watchdog_threshold)
- loss_watchdog, loss_watchdog_threshold, loss_watchdog_patience config
- Implemented in SoupTrainerCallback with patience counter
- Rich warning panel (stops Live display first), fires only once
- Wired into all 11 trainers via callback kwargs
Part D: Dataset info registry
- soup data register/unregister/registry commands
- ~/.soup/datasets.json local name→path+format mapping
- Name validation, path traversal protection, Rich markup escaping
82 new tests (2061 total), 74 test files.
Rich/Typer truncates help panel on narrow terminals (macOS CI), causing
--bits and --group-size flags to not appear in rendered help text. Switch
to inspecting the function signature directly for cross-platform reliability.
- AWQ export (`soup export --format awq`) via autoawq, with --bits, --group-size, --calibration-data
- GPTQ export (`soup export --format gptq`) via auto-gptq, with calibration data support
- Sample packing (`packing: true`) for SFT/Pretrain trainers via TRL's native packing
- `soup data split` — train/val/test splitting with random and stratified strategies
- Curriculum learning (`curriculum: true`) — sort dataset by difficulty for staged training
- New utility: soup_cli/utils/curriculum.py (sort_by_length, create_buckets)
- Security: calibration data path traversal protection, bits validation (4/8 only)
- 1970 tests across 70 test files
- Extract _parse_json_array into soup_cli/data/providers/_utils.py to
avoid circular imports between generate.py and provider modules.
- Narrow bare except Exception in detect_ollama to httpx.HTTPError/OSError
with debug logging instead of silent swallow.
Replace simple '..' check with resolve() + relative_to(cwd) for output
path. Add same confinement guard to --seed, --dedup-with, and --context
file paths. Add _path_within_cwd helper. 4 new security tests.
Rich markup wraps --model-a with ANSI codes on macOS, breaking the
substring check. Strip ANSI codes before asserting, matching the
existing pattern in test_speculative_decoding.py and test_deploy_ollama.py.
Rich markup in Typer help output inserts ANSI escape codes around
--flag names on macOS, breaking exact string matches. Check for
lowercase words instead of --prefixed flags.
Rich inserts color codes between flag name parts (e.g. --speculative
becomes \x1b[1;36m-\x1b[0m\x1b[1;36m-speculative\x1b[0m), so plain
substring match fails in CI. Strip ANSI before asserting.
- Rename --spec-tokens to --num-speculative-tokens to avoid prefix
collision with --speculative-decoding in Typer help rendering
- Add pytest.skip for _create_app tests when fastapi is not installed
Typer returns exit code 2 (not 0) when no_args_is_help=True and no
arguments are provided. Fix test_no_args_shows_help and
test_data_no_args_shows_help to accept both 0 and 2.
- Check stdout encoding instead of type to detect non-UTF-8 consoles
- Redirect stdout to UTF-8 TextIOWrapper before plotext renders
- Add unit test that simulates cp1251 stdout with plotext
- 1022 tests total
- soup data validate: default --format changed from 'alpaca' to 'auto',
uses detect_format() to auto-detect dataset format
- soup data stats: force UTF-8 stdout on Windows for plotext histograms
- soup ui: add --show-token flag, document auth token in --help
- 7 new tests (BUG-013/014/015), 1021 tests total
Alpamys
Salil M
Salil Mhatre