Last holdout of the recurring Path.resolve()+relative_to() containment class —
an advisory docstring (no behaviour change) that still recommended the
Windows-broken idiom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These were documented as known limitations in the MEDIUM/LOW pass; now fixed.
1. reward_hack EMA smoothing window — smooth_signal("ema") folded only
window[-1], so reward_hack_smoothing_window had no effect. Now a windowed
EMA folds alpha over the whole retained window (oldest→newest) then the new
sample, so a larger window incorporates more history; a 1-element window
reduces to the old 2-tap form. Updated test_v07126 (0.3 → 0.275).
2. hardware_fit OOM gate wired into `soup train` — the analytical VRAM
predictor was never called despite its docstring. Added
_build_hardware_fit_input (SoupConfig → HardwareFitInput, best-effort;
None when not statically predictable, e.g. batch_size="auto") and
_hardware_fit_preflight, run after device detection. Refuses on predicted
OOM (peak × 1.1 > available) unless the documented --allow-oom-attempt
opt-out is passed; skips silently on CPU / unknown VRAM, and the flag is
threaded through the --gpus re-exec.
3. MoD real token-dropping — mod_forward ran the full block on ALL tokens then
masked (zero compute savings). Now the top-k tokens are gathered into a
shorter sub-sequence, the block runs on ONLY those tokens (real saving),
the gated result is scattered back, and unselected tokens pass through
unchanged. Positional inputs (RoPE cos/sin, 4D-causal attention_mask,
position_ids, cache_position) are gathered to the sub-sequence; any
unsafe-to-gather case (positional forward args, KV cache, non-4D mask)
falls back to the prior correct blend so attention can never be silently
mis-computed. Validated on CPU (gather/scatter/passthrough/savings +
fallback); the sub-sequence-attention numerics still warrant GPU validation
at scale.
Adds tests/test_code_review_deferred.py (7 tests). ruff clean; full suite
14867 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- license_matrix: permissive ↔ weak-copyleft is now symmetric (MIT + LGPL no
longer flagged incompatible regardless of order).
- formats.detect_format: check tool-calling before audio so an audio+tools row
keeps its tool_calls instead of being classified audio.
- formats: reject null message content in the alpaca / sharegpt / vision text
converters (routes the row to the drop path instead of literal None content).
DPO only rejects an explicit null (chosen/rejected may be message lists).
- eval/custom.tool_call_args_subset: hallucinated args on a no-arg expected
call now score 0.0 (was a dead `0.5 if ... else 0.5` ternary).
- monitoring/callback: SSE metric push uses `is not None` so a real 0.0 loss/lr
is not reported as None.
- cans/schema.DeployTarget: reject Windows drive-absolute paths (C:\..., C:/...).
- commands/diagnose: reject a non-numeric evidence score with a clear
BadParameter (was ValueError -> exit 1 with zero output).
- commands/generate: partial-save accumulated examples on a mid-run failure so
paid API spend is not discarded.
- __init__.py: fix the byte-corrupted em dash in the package docstring.
Two MEDIUM/LOW items reverted to documented known limitations after they broke
existing behaviour locked by tests: (1) the reward_hack EMA smoother is a
recursive 2-tap by design — smoothing_window only affects `median`; (2)
`soup train`'s MoD compute-savings and the hardware_fit OOM preflight wiring
are architectural, GPU-validation work left as follow-ups.
Adds tests/test_code_review_medium_low.py (10 tests). ruff clean; full suite
14861 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Path.resolve()+relative_to() cwd-containment (breaks on Windows 8.3 short
names) -> utils.paths.is_under / is_under_cwd across all holdouts:
migrate/common.py, utils/ollama.py, commands/serve.py, commands/export.py,
commands/generate.py, data/loader.py, commands/data.py (×3), and
eval/checkpoint_intelligence.py (the pre-rmtree prune guard).
Unescaped Rich markup from external data: commands/_eval_v0550.py now
escapes the dataset-read exception message (×2). (adapters.py:315 already
escapes layer.name — that flagged instance was a false positive.)
Symlink-following writes -> project-safe helpers:
- utils/active_sampler.py: atomic_write_text (cwd-contained, symlink-reject).
- ui/app.py: tempfile.mkstemp (O_EXCL, unpredictable name) instead of a
fixed shared-temp path a local attacker could pre-symlink.
typer.Exit-vs-SystemExit audit: the two named instances (bench.py, eval.py)
were fixed in the HIGH tier; cli.py's top-level run() already handles both
SystemExit and typer.Exit correctly, so no further sites needed changes.
Adds tests/test_code_review_recurring.py (7 regression tests). ruff clean;
full suite 14851 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Training correctness (silent wrong results):
- ppo: refuse a randomly-initialised reward head when only reward_fn is set
(trl 0.19.1 PPO can't use a reward_fn) instead of training against noise.
- edit_kernels (AlphaEdit): reject a non-finite key-norm (NaN <= 0.0 is False).
- preference_combine (ORPO): length-normalise log-probs so exp() doesn't
underflow and kill the odds-ratio correction (+_read_lens caller wiring).
- ipo: anneal the beta schedule from ipo_tau, not the DPO default dpo_beta.
- distill: mask padding + prompt tokens in the default KL term (labels!=-100).
- block_expansion: freeze all-but-the-ACTUAL-added blocks (clamp over-request).
- formats (KTO): map a -1 label to False (bool(-1) was silently True).
Features that silently did nothing:
- sft: actually install the LongLoRA S² attention override (defensive).
- train --gpus re-exec: pass through --gate/--push-as/--trust-remote-code/
--tracker/--diagnose-gate/--annex-xi/--repro-receipt/--profile/energy flags.
- eval gate-install hook: pass $GATE_SUITE to `soup eval against`, which now
validates the locked suite as a precondition (block on missing/tampered).
- deploy_measure: fold the candidate list into the cache key.
Security:
- sglang: loopback-only CORS (was wildcard).
- fetch: lstat the ORIGINAL path (realpath resolved the symlink -> S_ISLNK
never fired -> write followed the link).
- ui /api/data/inspect: is_under_cwd (commonpath) instead of str.startswith.
- registry lineage: unbounded cycle check (the depth-10 cap accepted a
far-away cycle-closing edge).
- gguf calib: read from the O_NOFOLLOW fd (no close+reopen TOCTOU window).
- namespace_pin: flag ANY created_at drift (repo-recreation moves it forward).
Robustness / cross-platform:
- bench: re-raise typer.Exit (RuntimeError subclass) instead of masking it.
- eval auto: catch typer.Exit so a benchmark failure falls through.
- data split: reject negative --val/--test (negative slice inverted the split).
- trace parser: read utf-8-sig so a BOM'd first record isn't dropped.
- terraform plan: tolerate batch_size="auto" in the runtime estimate.
- rl_checkpoint: only rank-0 writes; atomic optimizer save.
Adds tests/test_code_review_high.py (29 regression tests). ruff clean;
full suite 14844 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. RLVR verifiable rewards (grpo.py, ppo.py): pass verifiable_domain=
to load_reward_fn so `reward_fn: verifiable` + `verifiable_domain: math`
no longer crashes at setup() — the headline RLVR feature was 100% broken.
2. serve: default --host to 127.0.0.1 (was 0.0.0.0) and add an opt-in
--tool-auth-token wired to _create_app so the code-exec tool endpoints
are not exposed unauthenticated on all interfaces; warn on non-loopback
bind without a token.
3. eval benchmark: reject ','/'=' in the adapter's base_model_name_or_path
(and --model path) before lm-eval model_args interpolation, closing the
trust_remote_code=True injection (mirrors the ship.py guard).
4. webhooks: run the private/link-local SSRF check for BOTH http and https
(was nested in the http-only branch, so https://169.254.169.254 and
10.x/192.168.x sailed through). Adds allow_private_hosts= so the trusted
loop_stages LAN-deploy caller (which does its own loopback/LAN tightening)
keeps working.
5. cloud/modal: emit the output_dir via the already-repr'd _LOCAL_OUTPUT
variable in a generated f-string instead of raw interpolation, closing
the stub code-injection hole (also quote-safe, unlike bare {output_dir!r}).
6. eval gate: wire cfg.training.eval_gate through all 14 trainers to
SoupTrainerCallback, and load the suite/baseline + build a live generator
in on_train_begin so --gate actually halts training. Warn honestly when
forgetting/checkpoint/early-stop knobs are set (they are not yet enforced).
Adds tests/test_code_review_critical.py (27 end-to-end regression tests).
ruff clean; full suite 14815 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two TestRewardHackMitigationCli help-text assertions grepped the raw
`soup train --help` output for the new flags. CI runners emit color codes
that split long option names into non-contiguous characters (same failure
mode documented in test_eval_gate.py for --gate), so the raw substring check
failed on every OS/Python combo while passing locally on a no-color terminal.
Fix follows the established repo pattern: render with COLUMNS=200 (no option
wrapping) and strip ANSI escapes before the membership check. Verified under
FORCE_COLOR=1: both assertions pass. Test-only change, no version bump.
Folds in the doc-vs-reality cleanup the plan flagged: the docs referenced
soup train --reward-hack-detector / --reward-hack-halt but they were config-only.
Add both as CLI passthroughs mirroring --reward-hack-mitigation (validate value,
set cfg.training field, accelerate re-exec passthrough) so the docs are true and
the reward-hack CLI is consistent. +4 tests (test_v07126: 180 -> 184; full suite
-> 14788).
Fixes from 5 sequential ECC reviews (python/code/security/tdd/verification).
python-review (2 CRITICAL + HIGH/MED/LOW):
- signal/vote coherence: schema now requires the active detector in
reward_hack_signals + rejects the inactive detector name (was silently
dropping the primary signal from the vote).
- integral_clamp is its own field (was wrongly hard-wired to beta_ceil).
- task/backend gate runs before controller-config checks; EMA convention
corrected; type hints; mutable-list default -> tuple + normalised compare.
code-review (4 HIGH + MED/LOW):
- _prune now trims _saved in sync with disk (rollback can't target a deleted
checkpoint); bang-bang release_count resets after each relaxation (hysteretic
descent); EMA formula uses standard convention; _escalate no longer burns a
recovery attempt on a None target; max_recovery_attempts>=1 required with
rollback; _action_history capped; on_step_end logs errors once; loud warning
when the mitigation callback can't attach (was a silent safety-off).
security-review (HIGH + MED):
- restore_checkpoint / save_checkpoint refuse a SYMLINKED optimizer.pt
(torch.load weights_only=False was an RCE via attacker-placed symlink);
bool-before-int/float guards on all new numeric fields; reward_hack_signals
max_length=4; empty-signals guard in the callback.
tdd-review: +13 coverage tests (dead-band hold, shim verbatim-on-error,
conservative boundary, read-only-beta dual-write, escalation postconditions,
both-restore, PID D exact, log cap + concurrency, no-top-level-import, fuzz
field-validity).
Test count 152 -> 180 (+2 POSIX-only symlink skips).
Recipe-count drift: the CLI help (commands.md) and Web UI pages
(serving-and-export.md) claimed 43 ready-made recipes and the catalog
docstring said ~30, while RECIPES actually holds 116 (test_recipes already
asserts len == 116). Aligned every user-facing count to 116.
Also refreshed 6 recipe descriptions still tagged "schema-only stub
(live in v0.52.1)": the TTS task (#131) and BitNet 1.58 SFT (#134) went
live in v0.71.20, so orpheus/sesame/llasa/spark/oute TTS and the Falcon-E
BitNet recipe now read "live (v0.71.20)".
Doc-drift only — description strings + one comment; no functional change,
no version bump.
The live-codec block claimed the entire data.format=audio path was "not
validated on the maintainer's box" and only surfaced a RuntimeError. v0.71.22
made the Orpheus SNAC encode live + validated; note that while the other four
families stay dependency-gated. Docs-only, no version bump.
- performance-and-quantization.md: the Quant Menu multi-trainer note said
"vision / audio modality is still SFT-only inline-BNB (wiring tracked as a
follow-up)" — stale after v0.71.19 #81 dropped the modality gate. Now states
vision/audio thread the unified loader (full gptq/awq/hqq/aqlm/eetq/mxfp4/fp8
menu), with the upstream class+kernel caveat.
- peft-and-efficiency.md: added the v0.71.19 #80 multi-GPU sharding paragraph to
the Multipack section (accelerator.prepare + BatchSamplerShard under
num_processes>1; identical bin seed across ranks; single-GPU unchanged).
Docs-only — no version bump / tag (the v0.71.19 code already shipped at f51331d).
#81 Quant Menu for vision/audio modality
- config/schema: drop the `modality != "text"` rejection in
_validate_quant_menu_supported_tasks (mlx-backend gate retained) so the full
Quant Menu (gptq/awq/hqq:Nbit/aqlm/eetq/mxfp4/fp8) applies to vision+audio.
- trainer/sft: _setup_vision_transformers + _setup_audio_transformers call
build_quantization_config_for_loader (strict superset of the inline 4bit/8bit
BNB blocks they replaced); drop BitsAndBytesConfig import; retain the
prepare_model_for_kbit_training gate on (4bit,8bit,mxfp4).
#80 multipack DataLoader sharding under FSDP/DeepSpeed/DDP
- utils/multipack_trainer: get_train_dataloader routes the multipack DataLoader
through accelerator.prepare when num_processes > 1 so accelerate's
BatchSamplerShard shards whole FFD bins across ranks (even_batches=True avoids
the epoch-boundary collective hang; seed identical across ranks, no `+ rank`).
Single-process path unchanged. Defence-in-depth guard against an unconfigured
MagicMock num_processes.
Tests: +37 in tests/test_v07119.py (13770 -> 13807). ruff clean.
Supersedes the v0.40.5 vision/audio Quant-Menu and v0.40.4 multipack-FSDP
known-limitations. Full multi-GPU validation remains an INFRA-BLOCKED QA item.
TestTrainCliMinillmOnPolicy::test_flag_in_help removed newlines + spaces but
not ANSI codes, so under CI FORCE_COLOR the Rich-rendered long flag (ANSI codes
between the dashes) failed the substring check. Strip ANSI + remove all
whitespace before the check, matching the cloud/sandbox help tests + the
v0.71.17 precedent. Verified under FORCE_COLOR=1 (114 passed).
Typer/Rich injects ANSI codes under CI FORCE_COLOR that split flag names
(--mole, --citation-style) and wrap panel text, breaking raw substring
asserts. Normalise output (strip ANSI + collapse whitespace) before matching.
Same pattern as the v0.71.1 --record-thumbs fix.
Pre-existing v0.71.13 test flaked on windows-latest CI: record_thumb stamps
time.time() and count_new_thumbs_since uses strict `>`, so on Windows' ~15.6ms
clock resolution the two mid-train thumbs could land in the same tick as
run_started (ts == run_started, dropped). Sleep one clock tick at the start of
slow_train so the thumbs are strictly later. Prod semantics unchanged; tests-only.
CI caught a directory-rejection bug masked on Windows: on POSIX, os.open(dir,
O_RDONLY|O_NOFOLLOW) succeeds and os.fdopen() then raises IsADirectoryError
before the S_ISREG check ran (Windows os.open rejects the dir first). fstat the
raw fd before fdopen so a directory is rejected with a clean ValueError on every
platform. Local + CI-equivalent test_directory_rejected now green.
#261 iterative_dpo._default_train_fn rendered output as a {dir: ...} mapping
that SoupConfig rejected; render it flat so the spawned soup train succeeds.
#246 CMA-ES merge now loads the base model once and reuses it across the
candidate population (_CachedBaseScorer) instead of reloading per candidate.
#245 soup loop estimate_cost wires run_cost.estimate_run_cost_usd off the last
completed run instead of a 0.0 placeholder; never crashes the daemon.
#244 soup train --track-energy --energy-out persists the measurement JSON so
soup bom emit --energy can attach it to an ML-BOM.
#170 --diagnose-gate is RANK-aware: gate once per cluster (RANK==0), not per node.
Tests: 13476 -> 13511 (+35 in tests/test_v07115.py). Validated end-to-end on
SmolLM2-135M / RTX 3050.
PR #256 attached energy only to CycloneDX; #244's contract is "both
outputs". Add _energy_annotations() so the SPDX model package carries
energy as OTHER annotations (same soup:<field>=<value> naming). Strengthen
the happy-path test to assert energy actually lands in the BOM (not just
that a file is written) and add a --format both case asserting energy in
BOTH cdx + spdx.
Closes#244
Adds --energy flag to soup bom emit: validates the measurement JSON (cwd containment + symlink rejection + JSON + EnergyMeasurement shape) and attaches energy properties to both CycloneDX and SPDX outputs.
Co-authored-by: gittihub-jpg <gittihub-jpg@users.noreply.github.com>
Soup spans the full post-training stack — SFT + preference/RL
(DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO) + distillation + unlearning +
knowledge-edit + steering + RAFT/RA-DIT — not just fine-tuning. "Fine-tune"
alone undersells the RL/alignment surface and the post-training category.
Update the canonical tagline to "Fine-tune and post-train LLMs in one
command. No SSH, no config hell." across the five places it appears:
README hero, PyPI description (pyproject), `soup --help` epilog + callback
docstring (cli.py), package docstring (__init__), and the auto-generated
HF model card (push.py). Searchable "fine-tune" keyword kept as the verb;
"post-train" added as the category claim.
Copy-only; no version bump (description ships with next release).
Alpamys
Darsh
Salil M
Kondamwar Akshaya Shrikant
Shatakshi Prasad
gittihub-jpg