Part A — BCO Trainer (Binary Classifier Optimization): new task='bco',
training.bco_beta, bco.yaml template, train+sweep routing. Internal
_split_dpo_rows_to_bco adapts paired DPO input to TRL's BCO unpaired
schema; skipped rows logged at DEBUG (mirrors v0.33.0 #47 policy).
Part B — Unified preference dispatcher: additive task='preference' +
training.preference_loss Literal {dpo,simpo,orpo,ipo,bco}. Legacy
task='dpo' / 'simpo' / 'orpo' / 'ipo' / 'bco' remain first-class —
the new surface is purely additive, not a breaking collapse.
_make_inner_cfg uses model_copy so re-validation never sees an
intermediate inconsistent state and the caller's cfg is never mutated.
Part C — KL-controlled DPO variants: dpo_beta_schedule (linear /
cosine / exponential) + dpo_beta_end + dpo_ref_regen_epochs [1, 1000].
BetaScheduleCallback resolves total_steps lazily in on_train_begin
(closes a first-cut bug where total_steps=0 silently emitted beta_end
for every step). RefModelRegenCallback uses load_state_dict(strict=True)
with WARNING-on-mismatch (closes a first-cut silent partial-copy
hazard). Gated to DPO-family tasks only; rejected on mlx backend with
distinct error message.
Part D — Multi-objective preference_loss_weights (2-5 entries, key
allowlist + null-byte rejection, sum-to-1 ±1e-6). Schema-level surface
only; live runtime weighted-loss combination deferred to v0.40.1 with
NotImplementedError stub-then-live (mirrors v0.27.0 MII / v0.37.0
multipack / v0.38.0 quant menu / v0.39.0 ReLoRA pattern).
Net +118 tests (4538 → 4656). All four review-agent waves clean
(Python / Code / Security / TDD).
Known limitation: BCOTrainerWrapper still hardcodes
trust_remote_code=True (carry-over of the v0.36.0 #63 family across
non-SFT trainers).
Also: add docs/ to .gitignore (internal-only docs going forward;
existing docs/QUANTIZATION.md from v0.38.0 stays tracked).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Train-time support for 7 new quantization formats — close the width gap
with LlamaFactory. Wired into SFT trainer + transformers backend + text
modality; multi-trainer/modality expansion deferred to v0.38.1 (mirrors
v0.27.0 MII / v0.37.0 multipack stub-then-live pattern).
- Part A — GPTQ: quantization='gptq' + gptq_disable_exllama (PEFT triton)
- Part B — AWQ: quantization='awq' + GEMM/GEMV builder
- Part C — HQQ: hqq:1bit..hqq:8bit (no 7bit; not supported upstream)
- Part D — AQLM: locked-2-bit
- Part E — EETQ: locked-8-bit
- Part F — MXFP4 + FP8 dequantize-on-load
- Part G — bnb_4bit_quant_storage for FSDP+QLoRA
("crucial for fsdp+qlora" — LlamaFactory quantization.py:178)
- Part H — check_quant_distributed_compat matrix + docs/QUANTIZATION.md.
HQQ/EETQ/AQLM x {FSDP, ZeRO-3} hard-fail; BNB-4bit + FSDP without
quant_storage warns. Wired into commands/train.py startup.
Three new schema validators:
- _validate_prequantized_no_qat — pre-quantized + QAT incompatible
- _validate_bnb_quant_storage_only_with_4bit — silent no-op guard
- _validate_quant_menu_supported_tasks — sft + transformers + text gate
Net: +61 tests (4374 -> 4435). Four review-agent waves clean before tag
(python-review / code-review / security-review / tdd-guide);
verification-loop performed as manual equivalent per CLAUDE.md allowance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI macOS runners render Rich panel help at a narrower terminal width
than local Windows. The flag --trust-remote-code is split with ANSI
colour escapes between segments, so the literal substring match in
the three CLI plumbing tests failed even though the flag was correct
in --help output. Mirrors the existing _strip_ansi helper in
tests/test_log_level.py (v0.34.0 fix for the same class of issue).
Tests-only follow-up; no soup_cli/ changes; no version bump needed
per release checklist policy on tests-only commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four silent-failure modes Soup had → loud failures, plus a
security default-deny.
- Part A: assistant-only loss masking (default true). Replaces TRL's
multi-turn heuristic with explicit IGNORE_INDEX masking. New
data.train_on_responses_only / train_on_messages_with_train_field
+ per-message train: bool field. Preferred path uses
return_assistant_tokens_mask; fallback uses incremental tokenize
delta with add_special_tokens=False to avoid double-BOS drift.
- Part B: --trust-remote-code opt-in default-deny on soup train /
chat / serve / data download / eval auto. KNOWN_SAFE_PREFIXES
allowlist (15 first-party orgs) suppresses warning panel.
Replaces 9 unconditional trust_remote_code=True call sites in
the SFT path. Non-SFT trainers + diff/export/merge/infer/generate
still hardcode trust_remote_code=True — documented v0.36.x patch.
- Part C: chat-template hardening. Tokenizers without chat_template
raise loudly instead of silent f"{role}: {content}" fallback.
New data.chat_template (registered name or raw Jinja). Filesystem
-touching Jinja directives (include/import/from/macro/extends)
blocked at config-load. Override application warns that soup push
will persist the new Jinja into tokenizer_config.json.
- Part D: OOM-probe auto batch-size. New
training.auto_batch_size_strategy: auto|static|probe. Try-halve
-then-double-to-ceiling loop, max 8 doublings, ceiling = static
× 4. ~/.soup/batch_cache.json (0600 perms, env-override
containment-checked against ~/cwd/tempdir). make_cache_key
rejects bool inputs.
Net +134 tests (4115 → 4249). All 5 review-agent waves clean
before commit; 5 HIGH / 10 MEDIUM / 5 LOW findings fixed in one
review-fix wave.
Smoke: python -m soup_cli.cli version → soup v0.36.0; all 5 new
--trust-remote-code flags surface in --help; ruff clean; pytest
4249 passed / 3 skipped / 0 failed in 2m41s on Windows py3.10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #62 added fp8_recipe support but only wired sft.py directly.
The other 10 trainers (dpo, pretrain, grpo, kto, orpo, simpo, ipo,
ppo, reward_model, embedding) all route through apply_v028_speed_memory,
which was calling apply_fp8_training(model) without recipe -- meaning
user-set fp8_recipe='rowwise' was a silent no-op on every non-SFT task.
- v028_features.apply_v028_speed_memory: read tcfg.fp8_recipe and pass
through to apply_fp8_training; surface the picked recipe in the
green status line so the run record reflects the actual dispatch
- sft.py: drop the defensive getattr (fp8_recipe is a Pydantic field
with a default, not optional) -- use tcfg.fp8_recipe directly
- tests: add TestFP8RecipeViaV028Features (4 tests) verifying the
recipe propagates through apply_v028_speed_memory for tensorwise /
rowwise / rowwise_with_gw_hp, plus the int8-QAT path is unaffected
Add fp8_recipe config field to TrainingConfig with three torchao-backed
scaling recipes: tensorwise (default, v0.28.0 behavior), rowwise (more
accurate via CUTLASS), and rowwise_with_gw_hp (most accurate, grad_weight
in high precision). Dispatches via Float8LinearConfig.from_recipe_name().
- schema.py: add fp8_recipe Literal field with validator requiring
quantization_aware='fp8' for non-default recipes
- fp8.py: update apply_fp8_training() to accept recipe parameter
- sft.py: pass tcfg.fp8_recipe to apply_fp8_training()
- README.md: document recipe options with comparison table
- tests: 24 tests covering schema, dispatch, validation, backward compat
Wires v0.28.0 speed/memory features into every transformer-backend
trainer (grpo / kto / orpo / simpo / ipo / ppo / reward_model /
embedding) plus closes the v0.33.0 #43 oversight where dpo / pretrain
accepted activation_offloading without installing offload hooks.
Auto-quant --auto-quant now forwards the picked candidate's
quantization to vLLM via an explicit named parameter (kwarg-splat
hazard removed). Kernel auto-compose runs a forward-only benchmark
loop on the trainer's actual model under torch.no_grad() so live
training gradients aren't polluted (this was a critical-class bug
caught by code-review pre-tag and fixed before merge).
Schema gate lifted with distinct MLX-backend vs unknown-task error
messages so users get the right fix. fp8 / int8 QAT guard fixed in
6 trainers (the legacy unguarded `if tcfg.quantization_aware:` would
have crashed the int8 path with the string "fp8").
Net +187 tests (3928 -> 4115). New file
tests/test_trainer_coverage_v035.py provides a parametrised matrix
proof that every trainer x every feature is exercised on every CI
matrix job. All four review-agent waves (python / code / security /
tdd) clean with every CRITICAL -> LOW finding fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's narrower terminal width forced Rich to split the flag literal across
ANSI colour escapes (`\x1b[1;36m-\x1b[0m\x1b[1;36m-log\x1b[0m\x1b[1;36m-level\x1b[0m`),
so the contiguous substring `--log-level` was not present in result.output
even though the flag is registered correctly. Strip ANSI codes before the
substring check — same pattern applied to similar Typer/Rich help-text
tests in other Python projects. No code change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds soup why, soup tui, soup runs replay, soup train --profile, .crash
bundle on training exception, per-run cost in SQLite, --log-level global
flag. Net +110 tests (3818 → 3928); all five review-agent waves
(python / code / security / tdd / smoke) clean.
- Part A: --log-level quiet|normal|verbose|debug → Rich-formatted logger
on the "soup" namespace; idempotent + tier-change replaces handler.
- Part B: SQLite gains cost_usd / cost_gpu_label via lazy ALTER TABLE
(race-tolerant against duplicate-column on concurrent first-boot);
rendered in soup runs show / replay / TUI; bool num_gpus rejected;
LIKE wildcards escaped in tracker.get_run prefix match.
- Part C: soup why — heuristic NaN / plateau / divergence / grad-norm /
LR bounds; severity-ordered findings.
- Part D: .crash bundle generator with recursive hf_*/sk-*/Bearer
redaction, output_dir basename-only, os.path.realpath containment,
secrets.token_hex filename, ValueError (not PermissionError) on
outside-cwd; train.py except-handler writes the bundle without
masking the original exception.
- Part E: soup runs replay <id> — summary panel + downsampled loss
curve (≤2000 points) from SQLite history.
- Part F: soup train --profile — torch.profiler Chrome trace to
<output>/profiles/<run_id>.trace.json; run_id rejects '.', '..',
'/', '\\', null bytes; profiles dir created only on torch import.
- Part G: soup tui — Textual dashboard with lazy ExperimentTracker
import; markup_escape on every DB-sourced string; new [tui] extra.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test patched sys.platform="linux" but called the cached
_get_isolation_strategy() — on macOS / Windows CI the cache had
already been populated with the host's strategy ("sandbox-exec" on
darwin, "best-effort" on win32) before the test ran, so the platform
patch was a no-op.
Use _compute_isolation_strategy() (uncached) like every other test in
the class. Also inject a fake os.unshare via monkeypatch so the
"namespaces" branch is reachable regardless of host kernel.
Tightened the assertion from `in {"namespaces", "best-effort"}` to
`== "namespaces"` since the unshare-unavailable case is covered by
the dedicated test_isolation_strategy_linux_unshare_unavailable test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses findings from 5-agent review wave (python-reviewer,
code-reviewer, security-reviewer, tdd-guide, smoke-verification).
CRITICAL:
- cans/run.py _deploy_target ollama path: rglob *.gguf result is now
realpath+commonpath checked against extract_dir before forwarding to
`soup deploy ollama --gguf`. Prevents a crafted symlink in the can
from making rglob point at an arbitrary on-disk path.
HIGH:
- cans/publish.py: removed dead update_repo_settings + bare-except tag
block (was a no-op network round-trip). Tag attachment via README
front-matter is documented as a v0.33.x docs follow-up.
- registry/attach.py lookup_entry_by_output_dir: emits ResourceWarning
when the 1000-row scan limit is hit (was a silent miss).
- data/collators.py CrossDocCollator: stops mutating input dicts via
pop() — uses get + dict comprehension. HF Dataset rows are cached and
reused; mutation broke subsequent batches silently. Bare-except now
logs at DEBUG level so production degradation is inspectable.
- monitoring/callback.py _write_spike_recovery_hint: added is_under_cwd
guard. args.output_dir came from raw HF TrainingArguments without
separate path-containment check.
- trainer/rewards.py MACOS_SANDBOX_PROFILE: narrowed (allow mach-lookup)
to a 3-name allowlist (SecurityServer, notification_center,
opendirectoryd.libinfo). Broad mach-lookup permitted DNS / NSURLSession
via launchd, defeating (deny network*).
- cans/run.py: PermissionError → ValueError so a caller wrapping in
`except OSError` cannot silently swallow the consent gate.
PermissionError is an OSError subclass.
- commands/can.py run_cmd: assigns result=None up front + explicit None
guard so a future _fail bypass cannot trigger NameError on result.
- utils/v028_features.py: added type annotations on apply_v028_speed_memory
(model: Any, tcfg: TrainingConfig via TYPE_CHECKING, console: Console)
and warn_unsupported_features.
- cans/run.py: confirm_callback now annotated
Callable[[Manifest], bool] for IDE introspection.
- tests/test_part_b.py reexec test: drops env-var contamination
(RANK/WORLD_SIZE/LOCAL_RANK/ACCELERATE_*) before run, patches
imported names on train module, and forces assertion that
os.execvp was called — no more silent skip-on-bypass.
- tests/test_part_d.py: added TestGenerateResponseSignature
source-level guard that catches the lenient logits_processor mock
silently passing.
MEDIUM:
- cans/run.py _run_subprocess: catches subprocess.TimeoutExpired and
returns rc=124 (coreutils convention) so callers see a clean
CanRunResult instead of an unhandled traceback after the 24h cap.
- cans/run.py: temp dir created via mkdtemp is now cleaned up on
extract_can failure (try/except + cleanup_extract_dir).
- cans/run.py cleanup_extract_dir: switched startswith path check to
os.path.commonpath (project-standard idiom; Windows-safe).
- cans/schema.py DeployTarget._safe_relpath: normalises mixed
separators before splitting on '/' so foo/..\bar can no longer
bypass the .. check.
- utils/lr_finder.py run_lr_sweep: removed redundant local
`import math as _math` (math already at module level).
LOW:
- eval/gate.py _parse_judge_url: removed bare http:// catchall after
scheme allowlist. Defence-in-depth for callers that bypass the
Pydantic GateTask validator.
- utils/auto_quant.py evaluate_candidate: latency mean now divides by
*completed* prompts (excludes crashed). Crashed candidate no longer
appears artificially fast.
- utils/auto_quant.py Candidate.__post_init__: explicitly rejects bool
in score / latency_ms (bool is a subclass of int, was sneaking past).
- utils/mii.py: removed `noqa: F401` on Optional import (now actually
used in type annotation since we restored it).
Tests added (+7, total 3811→3818):
- test_part_a_wave1: attach_artifact outside-cwd rejection.
- test_part_a_wave2: PermissionError→ValueError migration in 2 tests.
- test_part_c: CrossDocCollator mismatched doc_lengths fallback,
does-not-mutate-input-dict regression guard.
- test_part_d: source-level _generate_response signature guard.
- test_part_e: should_recover at max_attempts, outside-cwd skip.
Lint: clean. Full suite: 3818 passed in 156s.
Findings deliberately not actioned (with rationale):
- code-review M1 (mii Pydantic at import-time): forward-ref resolution
requires module-level definitions for FastAPI; documented in mii.py.
- code-review M4 (supports_v028_features vs validator divergence):
the v0.33.0 schema validator was renamed to
_validate_v028_speed_memory_supported_tasks and now imports
supports_v028_features — they cannot drift.
- python-review LOW (_deploy_target vllm silent no-op): documented in
the docstring as advisory; logging requires a console arg the
helper does not currently take.
- security-review LOW 8/9 (TOCTOU window, CLONE_NEWPID): theoretical;
documented in CLAUDE.md security section in the next commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#37, #38. Final Part of v0.33.0 implementation phase.
#37 Auto-reexec under accelerate launch when --gpus N>1:
- soup_cli/commands/train.py gains --no-reexec opt-out flag (default
behaviour: auto-reexec).
- When --gpus N>1 and not already in a distributed env (RANK/WORLD_SIZE
+ ACCELERATE_* markers absent), train() reconstructs argv via
utils.launcher.build_accelerate_argv and calls os.execvp("accelerate",
argv). os.execvp replaces the current process — no leftover PID tree,
stdio passes through unchanged.
- Critical flags (--fsdp, --deepspeed, --resume, --wandb, --tensorboard,
--yes) are forwarded to the reexec'd run so users see the same
behaviour they'd get from running accelerate launch by hand.
- OSError from execvp falls back to the v0.27.0 advisory (printed
command) so misconfigured PATH doesn't dead-lock the user.
- --no-reexec preserves the v0.27.0 print-and-exit behaviour for users
who want to control env vars / stdio explicitly.
#38 DeepSpeed-MII live serve:
- soup_cli/utils/mii.py gains build_mii_app(pipeline, model_name) which
returns a FastAPI app with /v1/chat/completions + /v1/models matching
the v0.30.0 transformers backend's contract.
- Pipeline is held by closure (single MII instance, thread-safe across
concurrent generations). Loopback-only CORS mirrors v0.30.0
transformers backend policy.
- max_tokens bounds [1, 16384], stream=True rejected (MII v0.x lacks
stable streaming), pipeline crashes return 500 with generic message
(no stack-trace leak). Empty response → 500.
- soup_cli/commands/serve.py replaces the v0.27.0 stub-warning + Exit(1)
with create_mii_pipeline → build_mii_app → uvicorn.run.
Tests: +9 in tests/test_part_b.py covering /v1/models endpoint, chat
happy-path with mocked pipeline returning .generated_text, streaming
rejection, max_tokens bounds (low + high), pipeline failure → 500,
empty pipeline response → 500, --no-reexec parameter exists,
--no-reexec advisory fallback, --gpus 2 reexec calls os.execvp with
accelerate argv (via monkeypatched os.execvp).
Known limitations:
- MII server has no streaming, no LoRA hot-swap, no /metrics dashboard,
no OpenTelemetry — those are v0.30.0 transformers-backend features
not yet ported. Documented in the build_mii_app docstring.
- Auto-reexec assumes accelerate is on PATH; OSError path prints the
command instead, matching the v0.27.0 baseline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#43, #44, #47.
#43 Multi-trainer wiring (sft/dpo/pretrain):
- New utils/v028_features.apply_v028_speed_memory(model, tcfg, base_model,
console) — single shared helper for use_cut_ce, quantization_aware="fp8",
kernel_auto_compose. Each feature degrades silently to a yellow advisory
if the underlying lib is missing; never crashes training kick-off.
- Helpers supports_v028_features(task) and warn_unsupported_features(tcfg, task)
drive both the schema validator and runtime advisories.
- soup_cli/trainer/dpo.py and trainer/pretrain.py now call the helper after
model load (post-LoRA, post-QAT) — same hook point as SFT.
- soup_cli/config/schema.py validator
_validate_v028_speed_memory_sft_only renamed
_validate_v028_speed_memory_supported_tasks; allowlist now {sft, dpo,
pretrain}. GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Embedding still error
out at config-load with a precise multi-trainer message.
#44 Selective gradient-checkpoint hooks:
- New utils/gradient_ckpt.install_selective_hooks(model, granularity)
iterates ``model.named_modules()`` looking for transformer-block-shaped
names (numeric suffix on layer path), wraps each module's ``forward``
with torch.utils.checkpoint.checkpoint based on tier:
- selective: only attention sub-modules
- medium: every second transformer block
- full: every transformer block
- Returns hook count so callers can fall back to HF native checkpointing
when zero blocks were found.
#47 CrossDocCollator:
- New soup_cli/data/collators.CrossDocCollator wraps any base data
collator and injects a block-diagonal causal ``cross_doc_attn_mask``
built from per-example ``doc_lengths``. Preferred over TRL's
``packing_strategy="attention_free"`` flag (best-effort across TRL
versions). Degrades gracefully when doc_lengths is missing or shapes
don't match — base attention_mask preserved, no crash.
Tests: +16 in tests/test_part_c.py covering apply_v028_speed_memory
(no-features, cut_ce graceful failure), supports/warn helpers extension,
schema gate (dpo + pretrain accept, kto still rejects), selective hook
installation across full/medium/selective with fake transformer-shaped
models, CrossDocCollator passthrough + strip + injection. One existing
test in test_training_speed.py updated: dpo+use_cut_ce now accepted.
Known limitations:
- 7 trainers (GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Embedding) still
reject v0.28.0 flags at config-load. Each is a 5-line addition once
schema validation is satisfied; tracked as a v0.33.x follow-up.
- install_selective_hooks doesn't undo earlier hooks — caller must be
re-init aware. Not an issue for the typical "construct wrapper, train,
exit" flow but worth noting.
- CrossDocCollator emits ``cross_doc_attn_mask`` (not ``attention_mask``)
to avoid clobbering the base collator's contract; downstream consumers
must read the new key explicitly. The plan calls for "preferred over
TRL's packing_strategy" which we satisfy via opt-in collation, not
silent override.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#49, #53, #54.
#53 Wire --structured-output into transformers generation loop:
- New utils/structured_output.build_logits_processors(constraint, tok)
returns a HF LogitsProcessor list. Tries outlines first (broader
coverage), falls back to lm-format-enforcer, returns [] if neither
installed or factory crashes — server degrades to free-form rather
than 500 on a missing dep.
- _generate_response gains logits_processor kwarg, forwarded to
model.generate(...). Chat-completions handler builds the processor
list per request (cheap; per-request build keeps the descriptor
mutable for future /v1/output_constraint endpoints) and passes it
down. Empty list path is unchanged from v0.30.0 free-form behaviour.
#54 --auto-quant live eval loop:
- New utils/auto_quant.evaluate_candidate(name, eval_fn, prompts):
times mean per-prompt latency, scores correctness, marks ok=False
when any prompt crashes or score < min_correct_fraction.
- New utils/auto_quant.run_auto_quant_picker(candidate_specs, prompts,
min_score): evaluates each candidate, calls pick_best, soft-falls-
back to highest-scored ok candidate if no candidate clears the
threshold so the server still binds.
- serve.py replaces the v0.30.0 deferral warning with a real picker
run over a fixed 3-prompt set across default_candidate_order().
Logs the picked (name, score, latency) on stdout.
#49 End-to-end --push-as integration test (mocked HF):
- New tests/test_part_d.py::TestPushAsResumeIntegration uses a fake
huggingface_hub module via patch.dict to verify HFPushCallback
constructs cleanly with a token, exposes the _repo_failed sticky
flag (v0.29.0 review fix), and that prepare_hf_resume rejects
output_dir outside cwd. The full HF Hub network roundtrip needs a
paid sandbox repo — keeping it mocked-only is a deliberate trade
(prevents flaky CI on rate limits / token rotation).
Tests: +15 in tests/test_part_d.py covering build_logits_processors
graceful-degrade paths (None / off / unknown / no-libs / factory
crash), generate_response logits_processor plumbing, evaluate_candidate
(empty / all-correct / crash / below-threshold), run_auto_quant_picker
(threshold pass + soft fallback), HF push smoke. One existing test in
test_inference_advanced.py updated: TestAutoQuantCLIWarning no longer
expects the v0.30.1 deferral message — it now expects --auto-quant to
actually run.
Known limitations:
- #49: full HF Hub roundtrip is mocked-only; live integration test
requires a paid sandbox repo and rotating token, deferred to a
separate end-to-end CI job.
- #54: live re-loading of the model at the picked quant is NOT done
in this commit — the picker logs the choice but the already-loaded
model is served. Live re-load needs an additional bnb / awq round-
trip per candidate, which is heavy for a startup-time decision;
follow-up tracked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#56, #57, #58, #59.
#56 Live --find-lr in-process LR-sweep:
- New utils/lr_finder.run_lr_sweep(model, dataloader, schedule,
optimizer_factory, device): per-step LR mutation + forward + backward,
records loss until exhaustion or NaN/Inf divergence.
- commands/train.py wires it via _live_lr_sweep_from_config (loads model
+ tokenizer + first N rows of cfg.data.train), with synthetic-curve
fallback so users without GPU/torch still get a parseable report.
#57 Loss-spike recovery hint:
- SoupTrainerCallback gains spike_recovery / spike_recovery_max_attempts
/ spike_recovery_lr_decay; on watchdog fire writes
output_dir/spike_recovery.json with previous_lr, recommended_lr (per
SpikeRecoveryStrategy.compute_new_lr), should_recover, attempts. A
wrapper / re-launch can resume with the decayed LR. Live optimizer
rewind is intentionally NOT done — HF Trainer has no safe public API
for mid-loop optimizer-state mutation; the JSON hint is the contract.
#58 auto_mixed_precision push to TrainingArguments:
- New SFTTrainerWrapper._resolve_mixed_precision: when
tcfg.auto_mixed_precision is True, queries torch.cuda compute
capability and calls pick_mixed_precision(base, cc) to set
bf16=/fp16= flags. CPU short-circuits to (False, False). When the
flag is False, legacy default preserved (bf16=cuda).
#59 Grad-accum advisory (Phase 1):
- SoupTrainerCallback gains grad_accum_auto_tune /
grad_accum_pressure_threshold / grad_accum_total_vram_gb /
grad_accum_current_steps / grad_accum_current_batch.
- on_log samples torch.cuda.max_memory_allocated each step; if
GradAccumMonitor.should_adjust crosses the threshold once,
prints (batch, accum) -> (new_batch, new_accum) advisory and
short-circuits (one-shot). Phase 2 (live DataLoader rebuild)
needs a small TRL upstream PR — tracked as a known limitation.
Wiring:
- soup_cli/trainer/sft.py: _resolve_mixed_precision helper, batch_size
preserved on self, SoupTrainerCallback constructor passes through new
spike + grad-accum knobs.
- soup_cli/monitoring/callback.py: rich Console import added (was
previously module-relative); spike + grad-accum state fields and
one-shot helpers.
Tests: +15 in tests/test_part_e.py covering the LR-sweep loop with
mocked model + optimizer (records, divergence break), mixed-precision
resolver across cpu/cuda + auto-flag combinations + qwen2 fp16 quirk on
Ampere, spike recovery hint write + attempts increment + disabled
no-op, grad-accum advisory one-shot semantics + threshold + cuda-absent
+ disabled.
Known limitations (release notes):
- #57 spike recovery is a JSON hint, not in-process optimizer rewind
- #59 Phase 2 (live DataLoader rebuild on advisory) deferred
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#34. Completes Part A — all of #32, #34, #35 now shipped.
Schema bump: CAN_FORMAT_VERSION 1 -> 2 (additive).
- SUPPORTED_CAN_FORMAT_VERSIONS = (1, 2): old cans still inspect/extract.
- New DeployTarget Pydantic model with kind in {ollama, gguf, vllm}, name
validation (no null bytes / newlines), path validation (relative-only,
rejects '..' and absolute paths).
- Manifest gains optional deploy_targets: list[DeployTarget] field.
cans/run.py — orchestrator:
- run_can(can_path, yes, deploy, extract_dir, capture_env_to, ...)
validates path containment, requires --yes or explicit confirm_callback
(security: auto-downloads data + auto-trains), extracts, optionally
captures env, then invokes `soup train --config ... --yes` via
subprocess so the trainer dispatch stays a single source of truth.
- capture_env: best-effort pip freeze + python version + GPU detection,
never blocks training on env-capture failure.
- _deploy_target dispatches per-kind; ollama path runs `soup deploy
ollama --gguf ... --name ...` if a *.gguf is present in the can.
- cleanup_extract_dir: tmp-or-cwd-only safety guard around shutil.rmtree.
cans/publish.py — HF Hub publish:
- publish_can(can_path, repo_id, token, private, commit_message)
validates can-path containment, repo_id via utils/hf.validate_repo_id,
resolves token via utils/hf.resolve_token (env > cache files), uploads
to repo_type='dataset' with commit-message first-line + 200-char cap
(matches v0.29.0 push.py / data push policy). Tags as can-format-v1.
CLI: soup_cli/commands/can.py
- New `soup can run <path> [--yes] [--deploy] [--extract-dir]
[--env-capture]` — confirmation panel mandatory without --yes.
- New `soup can publish <path> --hf-hub <user/repo> [--private]
[--message]`.
Tests: +28 in tests/test_part_a_wave2.py covering schema bump (v1/v2/v3),
DeployTarget validation (path traversal, null bytes, kind enum),
capture_env (smoke + pip-failure tolerance), run_can (containment +
confirmation gate + train-argv shape via mocked subprocess), publish_can
(repo_id validation, token resolution, commit-message sanitization, HF
upload via mocked HfApi), and CLI smoke (confirmation panel, missing
file). One existing test_cans test relaxed (v1 == v1 -> v in {1,2}).
Known follow-ups (not blocking release):
- soup can run does NOT yet auto-fetch data_ref.kind=hf|url. Embedded
config must reference a local data path. Filed mentally as
v0.33.x follow-up.
- registry_snapshot.json lineage export deferred — pack already embeds
base_hash which is enough to query the source registry post-extract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#32, #35. (#34 soup can run/publish deferred to Part A wave 2.)
#32 Live model scoring for `soup eval gate` + `soup eval quant-check`:
- gate.run_gate now dispatches judge / benchmark / custom task types,
wrapping each scorer in try/except so a backend failure produces
score=None + error=str(exc) instead of a silent score=1.0 pass.
- New _parse_judge_url splits ollama:// / http(s):// judge_model URLs
into (provider, model, api_base) for JudgeEvaluator.
- New _run_judge_task / _run_benchmark_task plug into existing
eval/judge.py and eval/forgetting.py runners.
- New quant_check.make_model_generator(model_path) wraps transformers
AutoTokenizer + AutoModelForCausalLM into a generate_fn callable;
greedy by default for reproducible scores; lazy-imported.
- gate_cmd / quant_check_cmd build live generators when --model is
given; fall back to deterministic stub on load failure so CI without
GPUs still runs the orchestration layer.
- GateTaskResult.score is now Optional[float] with new error: Optional[str].
- _print_gate_result renders ERROR + reason cleanly.
#35 Registry attach hooks:
- registry/store.py _VALID_KINDS extended with eval_results, tensorrt.
- New registry/attach.py: attach_artifact, write_eval_json
(cwd-containment via realpath+commonpath), lookup_entry_by_output_dir.
- `soup eval custom` gains --attach-to-registry + --output (paired);
on success writes JSON results and adds eval_results artifact row.
- `soup export` gains --registry-id with auto-match by source --model
output dir; auto-attaches the produced GGUF artifact. Failures here
are warnings, not hard exits — export already succeeded.
Tests: +19 in tests/test_part_a_wave1.py covering URL parser, error
propagation across all 3 task types, score=None semantics, generator
factory bounds + transformers mocking, registry attach helpers
(containment + missing entry), and CLI integration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#21, #22.
#21 RLVR code_exec_reward: add OS-level isolation strategy detection.
- New _get_isolation_strategy / _compute_isolation_strategy with cache.
- Linux: best-effort os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)
in preexec_fn (Python 3.12+). Silent fallback on EPERM/ENOSYS for
hosts where unprivileged user namespaces are disabled.
- macOS: prefix subprocess argv with sandbox-exec + inline default-deny
profile (deny network*, deny writes outside /tmp).
- Windows + restricted Linux: existing RLIMIT + socket-patch + ephemeral
cwd guards continue to apply (best-effort baseline).
#22 prune_checkpoints: TOCTOU-safe symlink handling.
- Top-level entries: explicit os.lstat + stat.S_ISLNK check (intent-clear)
instead of Path.is_symlink.
- shutil.rmtree now passes onerror=_abort_on_symlink to abort recursive
walk if any symlink is encountered mid-walk (defence-in-depth).
- OSError mid-prune is caught per-checkpoint so one bad dir does not
abort the whole prune pass.
Tests: +13 in tests/test_part_f_hardening.py covering strategy detection
on linux/darwin/win32, sandbox profile shape, code_exec smoke tests, and
TOCTOU-resistant prune behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rich/Typer emits per-character ANSI escapes in CI on Linux runners
("--\x1b[m-find\x1b[m-lr") so the raw substring assertion `--find-lr`
in result.output fails on ubuntu-latest x py3.12 even though it
passes on Windows where Rich auto-disables colour.
Strip ANSI before comparing — same pattern already used by
test_hf_integration.py and test_eval_platform.py.
Tests-only commit, no version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Expand the recipe catalog from 46 to 80 entries — every popular open-weight
model family now has a validated Soup recipe.
Part A — Vision (6 recipes): Llama-3.2-Vision-90B, Pixtral-12B, Qwen2-VL
(7B + 72B), InternVL 2.5, MiniCPM-V 2.6
Part B — Audio (3 recipes): Qwen2-Audio, SeamlessM4T v2, Whisper-large-v3
Part C — Reasoning (7 recipes): completes the 6 DeepSeek-R1-Distill sizes,
plus Qwen3-Coder, Qwen3-30B-A3B reasoning, Phi-4 reasoning
Part D — Edge (8 recipes): SmolLM2 (135M / 360M / 1.7B), Qwen2.5
(0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini
Part E — Domain (8 recipes): BioMistral, Meditron, CodeLlama (13B / 70B),
Magicoder, Mathstral, Nemotron-4 340B, Llama-2-13b-finance
Part F — Multimodal reasoning (2 recipes): Llama-3.2-Vision GRPO, Pixtral DPO
Part G — Recipe-validation CI workflow on every PR touching recipe / config /
data code (.github/workflows/recipe-validation.yml)
Part H — 750 parametrized tests covering catalog-wide invariants:
model-id safety (no `..`/`://`/null bytes), lora.target_modules non-empty,
max_length within schema bounds, GRPO recipes wire reward_fn +
num_generations >= 2, vision recipes set image_dir, audio recipes set
audio_dir, default data path is non-empty + relative
Live 100-step per-recipe smoke train (requires GPU runner) deferred to v0.31.1.
Tests: 2886 → 3607 (+721). Catalog: 46 → 80 (target met).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ubuntu-latest py3.11 matrix cell doesn't install the [serve] extra,
so `soup serve` exits early with a FastAPI-missing message before
reaching --structured-output / --auto-quant / --json-schema validation.
Add `pytest.importorskip("fastapi")` to the three tests that exercise
those CLI-level validation paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI failures on macOS/Windows:
1. `test_endpoint_rejects_null_byte` — ``monkeypatch.setenv("HF_ENDPOINT",
"...\x00")`` raises ``ValueError: embedded null byte`` at the C-level
setenv call on macOS/Windows before ``resolve_endpoint`` can reject it.
Linux's setenv swallows it. Replace with ``monkeypatch.setattr`` on
``os.environ`` dict so the null-byte string reaches ``resolve_endpoint``
on every platform.
2. Help-text substring tests (``test_train_shows_push_as_flag_in_help``,
``test_push_shows_collection_flag_in_help``, ``test_push_subcommand_exists``,
``test_hf_space_help_shows_flags``, ``test_train_help_shows_hf_resume``,
``test_hf_space_command_registered``) — Typer injects ANSI escape codes
on macOS/Windows pytest runs, splitting tokens like ``--push-as`` into
``-`` + ``-push-as`` across escape groups. Add ``_plain()`` helper that
strips ANSI via regex and use it in every help-text assertion.
Same pattern as 899ad8e (test_eval_gate.py) did after the v0.26.0 CI
Windows failure.
Full suite still passes locally: 100 HF integration tests in 3.25s.
- Split the 22-assert config-values test into 3 focused tests
(task+data, training hyperparams, LoRA config) so a deliberate
example change surfaces in one targeted test, not a wall of asserts
- Add module docstring explaining why these tests lock the example state
- Add `from __future__ import annotations` (defensive; matches 14 other
test modules in the project)
- Rename `f` -> `fh` in _load_jsonl to avoid shadowing short name
- Drop asserts on secondary fields (warmup_ratio, weight_decay, scheduler,
logging_steps, etc.) -- they're tweakable knobs, not the example's
teaching points; test brittleness > coverage here
Add a working DPO (Direct Preference Optimization) example using the
current Pydantic config schema with Llama 3.1 8B Instruct and QLoRA.
- examples/configs/dpo_example.yaml: DPO config with all core training
and LoRA parameters, plus commented-out advanced options
- examples/data/dpo_sample.jsonl: 8 preference pairs in DPO format
with ShareGPT-style message lists for chosen/rejected
- tests/test_dpo_example.py: 7 tests validating config loading, field
values, data format detection, and data validation
- examples/README.md: document the new DPO with QLoRA example
- Narrow 'except Exception: pass' in _get_dataset_size to specific
exceptions (OSError, ValueError, KeyError, ImportError)
- _get_dataset_size returns (size, is_estimated) so the caller can
warn when falling back to the 10k default (silent fallbacks are
misleading on a $-estimating command)
- Add -> None return type annotation on cost() (project convention)
- Add variance disclaimer: 'estimates are approximate; +/- 30%'
- Document pricing cadence in GPU_PRICING comment (last updated 2026-04)
- Use highlight=False on json.dumps output
- Fix misleading 'mock data' test comment (there is no mock)
- Add 2 tests: dataset-unreadable warning, variance disclaimer rendering
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs
* test(cost): add unit tests for 'soup cost' command and output formatting
* docs(readme): add usage documentation for the new 'soup cost' command
`test_train_gate_flag_accepted` asserted `"--gate" in result.output`, but
Typer/Click under CI emits ANSI color codes that split the flag name into
non-contiguous chars: `\x1b[1;36m-\x1b[0m\x1b[1;36m-gate\x1b[0m`. The literal
"--gate" substring is never present. All 9 OS × Python combos failed on the
v0.26.0 Parts B-E push.
Fix: strip ANSI via regex before checking. Also assert on "eval-gated" from
the option description to double-check the flag is wired to its help text.
CI-only / tests-only: no soup_cli/ changes, no version bump needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Foundation of v0.26.0 "Red and Blue Ocean" — every fine-tune is now
tracked with lineage, config, eval baseline, and shippable artifacts.
New module soup_cli/registry/:
- hashing.py: deterministic SHA-256 of config (canonical JSON) + data
(streamed) + base model; used as the entry_hash identity
- store.py: SQLite store (~/.soup/registry.db) with registry_entries,
registry_artifacts, registry_lineage, registry_tags. Context-manager
API, cycle-safe BFS walks, AmbiguousRefError on prefix collision,
LIKE-wildcard-escaped search + resolve, FK ON DELETE CASCADE.
- diff.py: flat-walk ConfigChange diff + per-benchmark eval delta.
New CLI commands:
- soup registry push/list/show/search/diff/promote/delete
- soup history <name> — lineage DAG tree viewer
Security hardening (v0.26.0):
- name/tag validation: alphanumeric + _-. only, null-byte rejected,
name ≤128, tag ≤64
- artifact path containment via os.path.realpath + commonpath
(Windows 8.3 short-name safe); enforce_cwd=True default
- SQL parameterised; LIKE wildcards %/_ escaped with ESCAPE '\'
- DB 600 perms on POSIX; SOUP_REGISTRY_DB_PATH env override
- indirect-cycle detection in add_lineage via BFS ancestor walk
- Rich markup escaped in all CLI output
- resolve() raises AmbiguousRefError instead of silent None
Tests: 92 new tests in tests/test_registry.py (hashing, validation,
CRUD, artifacts, lineage + cycle, diff, CLI, history, security,
auto-register integration with ExperimentTracker). Full suite:
2409 passed (was 2313).
All review findings addressed (4 agents: python, code, security, tdd):
HIGH: context manager + try/finally cleanup, FK cascade (removed
manual cascade), cycle detection, LIKE wildcard escaping.
MEDIUM: ambiguous resolve raises, exit 0 on user cancel, cwd
captured at construction, enforce_cwd=True default, Windows
ASCII-safe error messages.
Deferred to v0.26.1: soup eval --attach-to-registry flag and
soup export auto-artifact registration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- happy_path: assert mocked VRAM value (4.00 GB) renders in table
- happy_path: assert 'Benchmarking Configuration' panel rendered
- happy_path: assert mock_generate call count (1 warmup + 3 prompts)
- cpu_warning: assert 'N/A' appears in VRAM column when no CUDA
- Improve docstrings to describe what each test verifies
- Use full exception repr in exit_code asserts for CI debugging
- Narrow broad 'except Exception' to specific exceptions
(OSError, UnicodeDecodeError, json.JSONDecodeError) + `raise ... from`
- Rename file handle `f` -> `fh` to avoid shadowing (ruff-friendly)
- Clarify comment on --num-prompts ignored-when-file semantics
- Strengthen test: assert actual prompts were passed to _generate
(not just exit code + output substring)
- Add PEP 8 second blank line between test functions
* feat(bench): add --prompts-file option with path traversal security
* test(bench): add unit tests for custom prompts and path traversal
* docs(bench): document --prompts-file usage in README.md
* feat(bench): add --prompts-file support with path validation
* test(bench): add unit tests for custom prompts and security checks
* style: remove trailing whitespace to pass ruff linting
* test: fix mock patch targets for local imports in bench command
* refactor(bench): simplify prompts-file logic and clean up comment
* test(bench): update assertions to match new prompts-file semantics
* feat(cli): create 'soup bench' command for inference speed and VRAM measurement
* register 'bench' command into the main CLI router
* add test case for handling missing model paths gracefully
* add 'Inference Benchmarking' section explaining the 'soup bench' tool
* Added soup.yaml
* style: fix linting (unused imports, inconsistent spacing)
* style: sort imports in bench and test_bench to satisfy ruff
* style: final import sort and grouping fix for CI
* Update gitignore
test_writes_config fails on windows-latest / Python 3.9 with exit code 1
because the path-traversal check in soup_cli/commands/autopilot.py was:
data_path = Path(data).resolve()
data_path.relative_to(Path.cwd().resolve())
On Windows + Python 3.9, Path.resolve() occasionally leaves 8.3 short
names (e.g. "C:\Users\RUNNER~1") in one of the two sides but not the
other, so relative_to raises ValueError even when both paths point to
the same location. GitHub Actions runner home dirs frequently trigger
this (the runneradmin account is created as "runneradmin" but short
names get generated as "RUNNER~1").
Fix: introduce _is_under_cwd(path) helper in soup_cli/commands/autopilot.py
that uses os.path.realpath on both sides (handles 8.3 expansion
consistently) plus os.path.commonpath for the containment check, with
case-insensitive comparison on NT. Apply it to both the --data and
--output path guards. The data_path / output_path locals are then
rebuilt from the realpath result so downstream logic sees the
canonical long-name path.
Also enriches the test assertion to print result.output and
result.exception on failure so future CI breaks are easier to diagnose
without needing to push a debug commit first.
Local verification: all 38 tests in tests/test_autopilot.py pass on
Python 3.10 Windows, full suite 2313 passed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two tests in tests/test_bugfixes.py::TestGRPOCPUMinNewTokens fail on
windows-latest / Python 3.11 when importing trl.trainer.grpo_trainer:
RuntimeError: Failed to import trl.trainer.grpo_trainer because of
the following error:
'charmap' codec can't decode byte 0x90 in position 6555: character
maps to <undefined>
Root cause: upstream trl reads an auxiliary file without an explicit
encoding, so Python uses the system default. On Windows that is cp1252
('charmap'), which chokes on non-ASCII bytes present in the file. This
is an upstream issue but Soup needs a green CI.
Two-layer fix:
1. .github/workflows/ci.yml — set PYTHONUTF8=1 and PYTHONIOENCODING=utf-8
as job-level env. Python's UTF-8 mode makes all file I/O default to
UTF-8 regardless of locale, which is the correct global fix for this
class of bug.
2. tests/test_bugfixes.py — add a _trl_grpo_importable() helper that
returns False on UnicodeDecodeError / ImportError / RuntimeError, and
use it as a belt-and-braces skip in both TestGRPOCPUMinNewTokens
tests. Ensures the tests skip cleanly instead of erroring out if a
future CI change accidentally drops PYTHONUTF8.
Local verification: both tests pass with 'pytest tests/test_bugfixes.py::
TestGRPOCPUMinNewTokens -v' (Python 3.10, Windows).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every
competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators:
Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo,
Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA).
Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with
detection + normalization, synth data template, init template, eval scoring
(tool_call_match / tool_call_name_match / tool_call_args_subset), plus
qwen3-8b-tools and llama4-scout-tools recipes.
Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to
math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox
with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap,
one-time warning panel), and json_schema_reward. verifiable_domain Literal
validated via model_validator.
Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with
mutual-exclusion validator and a unified peft_builder helper that returns
either LoraConfig or VeraConfig with the right init kwargs.
Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx,
MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at
config load time by SoupConfig._validate_mlx_task_support, lazy trainer
registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject.
Part F — Data augmentation: soup data augment with rephrase / translate / style
strategies, path-traversal-protected input/output, count capped 1-10, lang/styles
lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup.
Part G — Training intelligence: forgetting detection (ForgettingDetector with
3 built-in mini benchmarks and warning levels) and checkpoint intelligence
(CheckpointTracker with composite metric, early-stop on regression, safe
top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema
extended with checkpoint_quality + forgetting_eval tables.
Part H — Autopilot: soup autopilot command with dataset/model/hardware
profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf
flags), YAML generator, and full CLI with dry-run + --yes + path-traversal
protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting
detection + checkpoint intelligence + early-stop into the generated config.
Totals:
- 2313 tests passing (183 new, up from 2130)
- 86 test files (8 new)
- 43 ready-made recipes (14 new)
- 16 built-in templates (tool-calling added)
- Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented
design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU,
MLX training integration test requires real hardware)
Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): add 'soup runs clean' intelligent checkpoint cleanup to reclaim disk space
* update README
* feat(cli): add 'soup runs clean' intelligent checkpoint cleanup to reclaim disk space
* fixed whitespace trails
* style(cli): fix lints (line length and spacing) in runs.py
* style: fix all E501 line length lint errors
* fix test mismatch, improve deletion warnings, add path validation, and enforce argument exclusivity
* fix: break long message into multiple lines for Ruff compliance
* test: update runs clean test to use CWD-based output directory for security compliance
* feat(doctor): add RAM and disk space checks to soup doctor command with tests and updated docs
* fix(doctor): resolve subprocess type checker error by manually validating macOS RAM query return code
* add --json flag to version command for machine-readable output in CI/scripts and include tests
* docs: update README with soup version --json flag examples
- Replace Windows-only paths (C:/Windows/...) with tempdir-based
paths that work on Linux/macOS CI runners
- Use monkeypatch.chdir for path traversal test isolation
- Fixes test_path_outside_cwd_raises failure on Ubuntu CI
- Replace non-ASCII symbols (checkmarks, arrows, bullets, em-dashes)
with ASCII equivalents in Rich console output to prevent
UnicodeEncodeError on Windows without PYTHONIOENCODING=utf-8
- Add _validate_output_path() for AWQ/GPTQ export — output path
traversal is now checked before import check (previously unreachable
when autoawq/auto-gptq not installed)
- 4 new tests for output path validation (2065 total, 0 failures)
- Update SECURITY.md with v0.22.0–v0.24.1 hardening history
Part A: HuggingFace Dataset browser
- soup data search: search HF Hub for datasets (sort by downloads/likes)
- soup data preview: preview remote dataset metadata, splits, features
- soup data download: stream HF dataset to local JSONL (with format conversion)
- Security: trust_remote_code=False, path traversal protection, samples cap at 1M
Part B: Freeze training (like LLaMA-Factory finetuning_type: freeze)
- freeze_layers / freeze_ratio config fields
- soup_cli/utils/freeze.py: detect layers, freeze bottom N
- Wired into SFT trainer before LoRA application
- Supports LLaMA (layers.N) and GPT-2 (h.N) naming
Part C: Loss watchdog (like Axolotl loss_watchdog_threshold)
- loss_watchdog, loss_watchdog_threshold, loss_watchdog_patience config
- Implemented in SoupTrainerCallback with patience counter
- Rich warning panel (stops Live display first), fires only once
- Wired into all 11 trainers via callback kwargs
Part D: Dataset info registry
- soup data register/unregister/registry commands
- ~/.soup/datasets.json local name→path+format mapping
- Name validation, path traversal protection, Rich markup escaping
82 new tests (2061 total), 74 test files.
Rich/Typer truncates help panel on narrow terminals (macOS CI), causing
--bits and --group-size flags to not appear in rendered help text. Switch
to inspecting the function signature directly for cross-platform reliability.
- AWQ export (`soup export --format awq`) via autoawq, with --bits, --group-size, --calibration-data
- GPTQ export (`soup export --format gptq`) via auto-gptq, with calibration data support
- Sample packing (`packing: true`) for SFT/Pretrain trainers via TRL's native packing
- `soup data split` — train/val/test splitting with random and stratified strategies
- Curriculum learning (`curriculum: true`) — sort dataset by difficulty for staged training
- New utility: soup_cli/utils/curriculum.py (sort_by_length, create_buckets)
- Security: calibration data path traversal protection, bits validation (4/8 only)
- 1970 tests across 70 test files
- Extract _parse_json_array into soup_cli/data/providers/_utils.py to
avoid circular imports between generate.py and provider modules.
- Narrow bare except Exception in detect_ollama to httpx.HTTPError/OSError
with debug logging instead of silent swallow.
Replace simple '..' check with resolve() + relative_to(cwd) for output
path. Add same confinement guard to --seed, --dedup-with, and --context
file paths. Add _path_within_cwd helper. 4 new security tests.
Rich markup wraps --model-a with ANSI codes on macOS, breaking the
substring check. Strip ANSI codes before asserting, matching the
existing pattern in test_speculative_decoding.py and test_deploy_ollama.py.
Rich markup in Typer help output inserts ANSI escape codes around
--flag names on macOS, breaking exact string matches. Check for
lowercase words instead of --prefixed flags.
Rich inserts color codes between flag name parts (e.g. --speculative
becomes \x1b[1;36m-\x1b[0m\x1b[1;36m-speculative\x1b[0m), so plain
substring match fails in CI. Strip ANSI before asserting.
- Rename --spec-tokens to --num-speculative-tokens to avoid prefix
collision with --speculative-decoding in Typer help rendering
- Add pytest.skip for _create_app tests when fastapi is not installed
Typer returns exit code 2 (not 0) when no_args_is_help=True and no
arguments are provided. Fix test_no_args_shows_help and
test_data_no_args_shows_help to accept both 0 and 2.
- Check stdout encoding instead of type to detect non-UTF-8 consoles
- Redirect stdout to UTF-8 TextIOWrapper before plotext renders
- Add unit test that simulates cp1251 stdout with plotext
- 1022 tests total
- soup data validate: default --format changed from 'alpaca' to 'auto',
uses detect_format() to auto-detect dataset format
- soup data stats: force UTF-8 stdout on Windows for plotext histograms
- soup ui: add --show-token flag, document auth token in --help
- 7 new tests (BUG-013/014/015), 1021 tests total
Add continued pre-training task and Mixture of Experts model support:
- `task: pretrain` for continued pre-training on raw text data
- `plaintext` data format ({"text": "..."} JSONL or .txt files)
- MoE model detection (Mixtral, Qwen3 MoE, DeepSeek V3, DBRX, OLMoE)
- ScatterMoE LoRA (`moe_lora: true`) targets expert FFN + attention layers
- `moe_aux_loss_coeff` for router load-balancing loss
- Templates: `soup init --template pretrain` and `--template moe`
- 85 new tests across test_pretrain.py and test_moe.py (1002 total)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Test _load_model exit paths: adapter without base model, corrupt JSON
- Test _generate branches: greedy (temp=0), sampling (temp>0), no
chat_template fallback, token count from tensor shape, role formatting
- Test max_tokens bounds: 0 and 99999 rejected by CLI
- Test tensorboard happy path: flag accepted when tensorboard installed
- Fix import-failure test: avoid builtins.__import__ recursion
- 917 tests, 44 test files, 57.92% coverage
- Fix test_tensorboard_in_train_help: strip ANSI escape codes before
asserting --tensorboard in help output (Rich splits flag across
escape sequences on Python 3.11)
- Fix TensorBoard import check: use `import tensorboard` directly
- Stream JSONL output during inference (crash-safe for large files)
- Return accurate token count from _generate via tensor shape
- Replace shallow tests with real trainer integration tests
- Cap max_tokens at 16384 + trust_remote_code warning
- Fix TensorBoard import check: use `import tensorboard` directly
(not torch.utils.tensorboard shim) for accurate availability check
- Stream JSONL output during inference instead of buffering in memory
(crash-safe, handles large prompt files)
- Return accurate token count from _generate via tensor shape instead
of re-encoding decoded text
- Replace shallow tests with real trainer integration tests that
verify report_to='tensorboard' is accepted by all trainer wrappers
Security fixes across all HTTP surfaces:
- Web UI: Bearer token auth on mutating endpoints, CORS restricted to served origin,
path traversal protection on /api/data/inspect, config validated before training,
removed user-controlled config_path from API
- Serve/vLLM: max_tokens capped at 16384, generic error messages (no stack traces)
- Generate: SSRF protection (--api-base blocks non-HTTPS for remote URLs),
--api-key deprecated in favor of OPENAI_API_KEY env var
- Export: llama.cpp pinned to tag b5270 (supply-chain safety)
- Push: --token deprecated in favor of HF_TOKEN env var
- Rewards: warning before executing custom .py reward files
- Tests: all 40 UI tests updated with auth headers, 666 tests pass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GRPO:
- Set default chat_template on tokenizer when missing (fixes ValueError
from trl's apply_chat_template on models without chat support)
- Ensure batch_size >= num_generations (trl 0.28 requirement)
- Verified end-to-end GRPO training on CPU succeeds
PPO:
- Tokenize dataset via .map() before passing to PPOTrainer (adds input_ids
and attention_mask columns required by trl experimental API)
Tests: 666 passed, 5 new tests for chat_template/tokenization fixes,
3 existing mock tests updated for new tokenization step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: Skip resume_from_checkpoint when experimental PPOTrainer.train()
doesn't accept it (inspect signature at runtime, warn and proceed).
- CPU: Use device_map="cpu" instead of "auto" on CPU across all trainers
(SFT, DPO, GRPO, PPO, RewardModel) to prevent meta tensor errors.
- Add 12 new tests for both fixes (661 total passing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: Support trl >=0.28 experimental API (ref_model, reward_model,
train_dataset, value_model positional args). Auto-import from
trl.experimental.ppo with fallback. Create reward/value models when needed.
- GRPO: Fix CPU empty generation tensor mismatch by passing
generation_kwargs={"min_new_tokens": 1} on CPU devices.
- Add 6 new tests for both fixes (649 total passing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use create=True in mock.patch so tests work when trl has moved
PPOTrainer to trl.experimental and it's not in the trl namespace.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PPOTrainer.__init__() no longer accepts dataset= in newer trl versions.
Now checks via inspect.signature whether train_dataset or dataset is
accepted; if neither, sets dataset on trainer before .train() call.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: detect trl API via inspect — args= (>=0.28) vs config= (<0.28)
- PPO: split train into _train_builtin (trl >=0.28) and _train_manual
- GRPO: update error message to mention GRPO/PPO CPU limitation
- 2 new tests for PPO API detection (639 total, all passing)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: add use_cpu=True to PPOConfig when running on CPU
- GRPO: add CPU warning + use_cpu flag via inspect (trl bug workaround)
- Add use_cpu error pattern to friendly error map
- 7 new tests for CPU fixes (637 total, all passing)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ASCII progress bar in display.py (replace Unicode blocks with #/-)
- Catch UnicodeEncodeError from plotext in data stats (Windows fallback)
- Auto-disable 4bit/8bit quantization on CPU (bitsandbytes doesn't support it)
- Friendly error messages for CPU tensor/dtype/bf16/torchvision issues
- Fix diff.py: torch_dtype= -> dtype= (deprecation warning)
- Add torchvision version compatibility check to soup doctor
7 new tests (631 total), ruff clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BUG-001: Replace Unicode arrows/dashes with ASCII in all console output
to fix UnicodeEncodeError on Windows cp1252 (~10 commands affected)
- BUG-002: PPO trainer uses inspect.signature to detect trl parameter names
(ppo_epochs vs num_ppo_epochs) for trl 0.28.0 compatibility
- BUG-003: Add get_compute_dtype() - uses float32 on CPU, bfloat16/float16
on CUDA. Fixes dtype mismatch in reward model and all trainers
- BUG-004: Add warning when using quantization on CPU
- BUG-005: Fix dtype -> torch_dtype in diff.py model loading
- BUG-006: Pin wandb<0.18.0 to avoid trl import conflict, add runtime guard
13 new tests (624 total), ruff clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add --backend vllm flag to soup serve for 2-4x better inference throughput
- Add --tensor-parallel and --gpu-memory flags for vLLM tuning
- Auto-detect vLLM and show hint when installed but not enabled
- New utils/vllm.py with engine creation, app factory, LoRA support
- Native token-by-token streaming via vLLM AsyncLLMEngine
- Add serve-fast extra: pip install 'soup-cli[serve-fast]'
- Add vllm detection to version --full
- 30 new tests (560 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `soup ui` command launches local web interface at http://127.0.0.1:7860
- FastAPI backend with REST API: runs, metrics, config validation, training
control, data inspection, templates, system info
- Self-contained SPA frontend (Dashboard, New Training, Data Explorer,
Model Chat) with Chart.js loss/LR charts
- Auto-opens browser on launch (--no-browser to disable)
- Config validation via new load_config_from_string() in config/loader.py
- 40 new tests (530 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add `modality: vision` config option for vision-language model training
- Add LLaVA and ShareGPT4V data format detection and conversion
- Add `image_dir` field in DataConfig for resolving image paths
- Add vision model loading via AutoModelForVision2Seq + AutoProcessor in SFT trainer
- Add `soup init --template vision` with LLaMA-3.2-Vision config
- Add image statistics display in `soup data inspect` for vision datasets
- Add Pillow as optional `vision` extra dependency
- Add Pillow to `soup doctor` dependency checks
- 51 new tests (455 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add --early-stop flag to soup sweep: skip remaining runs when loss exceeds best by threshold
- Add Rich progress bars for HuggingFace Hub model downloads in SFT/DPO trainers
- Pin click>=8.2.0 to fix typer compatibility (11 tests were failing)
- Bump version 0.3.2 → 0.4.0
- 329 tests passing, lint clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `soup version --full` shows version, Python, GPU backend, installed extras
- Dynamic test count badge via Gist endpoint in CI
- README: Optional Extras table, --verbose note, CSV/Parquet, Changelog link
- 323 tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rich markup splits --verbose with escape codes on CI, check for
"verbose" instead of "--verbose".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Friendly error messages: wrap all commands in try/except, map known errors
(CUDA OOM, missing deps, connection errors) to 2-3 line messages with fix hints
- Global --verbose flag for full tracebacks
- soup doctor: check system info, GPU, all dependency versions with fix suggestions
- soup quickstart: one-command demo (creates data + config + trains TinyLlama)
- Confirmation prompts before train/sweep (skip with --yes)
- 40 new tests (321 total), all passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- soup serve: FastAPI inference server with OpenAI-compatible API, SSE streaming
- soup data generate: synthetic data generation via OpenAI API or local models
- soup sweep: grid/random hyperparameter search with experiment tracker integration
- soup diff: side-by-side model comparison with metrics
- Multi-GPU/DeepSpeed: ZeRO Stage 2/3 configs, --deepspeed flag in train command
- 95 new tests (281 total), all passing
- Removed TESTING_GUIDE.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rich markup splits --resume/--wandb with escape codes in CI (Python 3.9).
Use case-insensitive substring match instead of exact flag match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4 hex char suffix (65536 possibilities) caused a collision when
generating 100 IDs within the same second on fast CI runners.
Increased to 8 hex chars (4 billion possibilities).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix 23 ruff errors: line too long, unused imports, ambiguous vars
- Fix validator: empty string is valid data, only count None as empty
- Remove unused imports in display.py and validator.py
- Rename ambiguous `l` vars to `part`, `entry`, `length`
- Break long lines in callback.py, display.py, sft.py, constants.py
All 20 tests passing, ruff clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Typer CLI: soup init, soup train, soup data inspect/validate
- Pydantic config schema with YAML loader and validation
- Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets
- Format detection: Alpaca, ShareGPT, ChatML (auto-detect)
- SFT trainer wrapper over transformers + peft + trl
- QLoRA/LoRA support with auto batch size estimation
- GPU detection (CUDA/MPS/CPU) and memory calculation
- Rich live terminal dashboard for training monitoring
- Config templates: chat, code, medical
- Tests (pytest) + GitHub Actions CI
- MIT license
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alpamys
Chinmaya Sahu
Salil M
Salil Mhatre