test_writes_config fails on windows-latest / Python 3.9 with exit code 1
because the path-traversal check in soup_cli/commands/autopilot.py was:
data_path = Path(data).resolve()
data_path.relative_to(Path.cwd().resolve())
On Windows + Python 3.9, Path.resolve() occasionally leaves 8.3 short
names (e.g. "C:\Users\RUNNER~1") in one of the two sides but not the
other, so relative_to raises ValueError even when both paths point to
the same location. GitHub Actions runner home dirs frequently trigger
this (the runneradmin account is created as "runneradmin" but short
names get generated as "RUNNER~1").
Fix: introduce _is_under_cwd(path) helper in soup_cli/commands/autopilot.py
that uses os.path.realpath on both sides (handles 8.3 expansion
consistently) plus os.path.commonpath for the containment check, with
case-insensitive comparison on NT. Apply it to both the --data and
--output path guards. The data_path / output_path locals are then
rebuilt from the realpath result so downstream logic sees the
canonical long-name path.
Also enriches the test assertion to print result.output and
result.exception on failure so future CI breaks are easier to diagnose
without needing to push a debug commit first.
Local verification: all 38 tests in tests/test_autopilot.py pass on
Python 3.10 Windows, full suite 2313 passed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every
competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators:
Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo,
Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA).
Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with
detection + normalization, synth data template, init template, eval scoring
(tool_call_match / tool_call_name_match / tool_call_args_subset), plus
qwen3-8b-tools and llama4-scout-tools recipes.
Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to
math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox
with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap,
one-time warning panel), and json_schema_reward. verifiable_domain Literal
validated via model_validator.
Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with
mutual-exclusion validator and a unified peft_builder helper that returns
either LoraConfig or VeraConfig with the right init kwargs.
Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx,
MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at
config load time by SoupConfig._validate_mlx_task_support, lazy trainer
registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject.
Part F — Data augmentation: soup data augment with rephrase / translate / style
strategies, path-traversal-protected input/output, count capped 1-10, lang/styles
lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup.
Part G — Training intelligence: forgetting detection (ForgettingDetector with
3 built-in mini benchmarks and warning levels) and checkpoint intelligence
(CheckpointTracker with composite metric, early-stop on regression, safe
top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema
extended with checkpoint_quality + forgetting_eval tables.
Part H — Autopilot: soup autopilot command with dataset/model/hardware
profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf
flags), YAML generator, and full CLI with dry-run + --yes + path-traversal
protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting
detection + checkpoint intelligence + early-stop into the generated config.
Totals:
- 2313 tests passing (183 new, up from 2130)
- 86 test files (8 new)
- 43 ready-made recipes (14 new)
- 16 built-in templates (tool-calling added)
- Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented
design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU,
MLX training integration test requires real hardware)
Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Alpamys