The v0.71.28 mcp command help used an em-dash, tripping the long-standing
test_cli_subprocess.py::TestEncoding::test_help_output_is_ascii_safe guard
(Windows encoding safety). Replace with an ASCII hyphen; main CI was red.
The #291 fix (vision/audio vocab expansion) landed on main below the
v0.71.27 version bump, so it ships in the v0.71.27 wheel — move its note
from [Unreleased] into [0.71.27] ### Fixed, next to its text-path sibling
#287, so the release notes accurately reflect what ships.
Add `soup data doctor` and `soup data lint`, killing the top *silent*
fine-tune failures before a single training step: EOS-missing-from-labels
(the #1 "model never stops generating" bug), BOS duplication, no-system-role
templates, and preference-data length bias (the #1 silent DPO degradation) —
none of which any competitor (Unsloth/Axolotl/LlamaFactory) checks for.
- utils/data_doctor.py: 8-check chat-template compat report over a
tokenizer + sampled rows, OK/MINOR/MAJOR taxonomy mirroring diagnose;
--show-mask N renders per-token trained/masked colouring through the
SAME masking dispatch (_build_row_labels) the report itself uses, so
the two can never disagree about what's actually trained.
- utils/data_lint.py: preference-data linter (dpo/orpo/simpo/ipo/bco/kto)
— length bias (Cohen's d), label imbalance, near-duplicates (MinHash),
identical chosen==rejected pairs, prompt leakage.
- commands/data_doctor.py: Typer layer for both commands; strips C0
control bytes before untrusted dataset content reaches the terminal.
- commands/diagnose.py: hardens the --evidence loader against a TOCTOU
symlink swap (O_NOFOLLOW + fstat-on-open-fd), backporting the pattern
soup ship shipped in v0.71.25 (closes v0.71.25 known-limitation (4)).
Live smoke against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer
(Windows + RTX 3050) found and fixed two genuine bugs beyond the synthetic
fixtures: the EOS check needed to span-search the whole trained region
(not just the last token), and two apply_chat_template call sites needed
a broad except Exception for jinja2.exceptions.TemplateError.
+173 tests (14788 -> 15042). 5 sequential ECC reviews, every finding fixed.
Last holdout of the recurring Path.resolve()+relative_to() containment class —
an advisory docstring (no behaviour change) that still recommended the
Windows-broken idiom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These were documented as known limitations in the MEDIUM/LOW pass; now fixed.
1. reward_hack EMA smoothing window — smooth_signal("ema") folded only
window[-1], so reward_hack_smoothing_window had no effect. Now a windowed
EMA folds alpha over the whole retained window (oldest→newest) then the new
sample, so a larger window incorporates more history; a 1-element window
reduces to the old 2-tap form. Updated test_v07126 (0.3 → 0.275).
2. hardware_fit OOM gate wired into `soup train` — the analytical VRAM
predictor was never called despite its docstring. Added
_build_hardware_fit_input (SoupConfig → HardwareFitInput, best-effort;
None when not statically predictable, e.g. batch_size="auto") and
_hardware_fit_preflight, run after device detection. Refuses on predicted
OOM (peak × 1.1 > available) unless the documented --allow-oom-attempt
opt-out is passed; skips silently on CPU / unknown VRAM, and the flag is
threaded through the --gpus re-exec.
3. MoD real token-dropping — mod_forward ran the full block on ALL tokens then
masked (zero compute savings). Now the top-k tokens are gathered into a
shorter sub-sequence, the block runs on ONLY those tokens (real saving),
the gated result is scattered back, and unselected tokens pass through
unchanged. Positional inputs (RoPE cos/sin, 4D-causal attention_mask,
position_ids, cache_position) are gathered to the sub-sequence; any
unsafe-to-gather case (positional forward args, KV cache, non-4D mask)
falls back to the prior correct blend so attention can never be silently
mis-computed. Validated on CPU (gather/scatter/passthrough/savings +
fallback); the sub-sequence-attention numerics still warrant GPU validation
at scale.
Adds tests/test_code_review_deferred.py (7 tests). ruff clean; full suite
14867 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- license_matrix: permissive ↔ weak-copyleft is now symmetric (MIT + LGPL no
longer flagged incompatible regardless of order).
- formats.detect_format: check tool-calling before audio so an audio+tools row
keeps its tool_calls instead of being classified audio.
- formats: reject null message content in the alpaca / sharegpt / vision text
converters (routes the row to the drop path instead of literal None content).
DPO only rejects an explicit null (chosen/rejected may be message lists).
- eval/custom.tool_call_args_subset: hallucinated args on a no-arg expected
call now score 0.0 (was a dead `0.5 if ... else 0.5` ternary).
- monitoring/callback: SSE metric push uses `is not None` so a real 0.0 loss/lr
is not reported as None.
- cans/schema.DeployTarget: reject Windows drive-absolute paths (C:\..., C:/...).
- commands/diagnose: reject a non-numeric evidence score with a clear
BadParameter (was ValueError -> exit 1 with zero output).
- commands/generate: partial-save accumulated examples on a mid-run failure so
paid API spend is not discarded.
- __init__.py: fix the byte-corrupted em dash in the package docstring.
Two MEDIUM/LOW items reverted to documented known limitations after they broke
existing behaviour locked by tests: (1) the reward_hack EMA smoother is a
recursive 2-tap by design — smoothing_window only affects `median`; (2)
`soup train`'s MoD compute-savings and the hardware_fit OOM preflight wiring
are architectural, GPU-validation work left as follow-ups.
Adds tests/test_code_review_medium_low.py (10 tests). ruff clean; full suite
14861 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Path.resolve()+relative_to() cwd-containment (breaks on Windows 8.3 short
names) -> utils.paths.is_under / is_under_cwd across all holdouts:
migrate/common.py, utils/ollama.py, commands/serve.py, commands/export.py,
commands/generate.py, data/loader.py, commands/data.py (×3), and
eval/checkpoint_intelligence.py (the pre-rmtree prune guard).
Unescaped Rich markup from external data: commands/_eval_v0550.py now
escapes the dataset-read exception message (×2). (adapters.py:315 already
escapes layer.name — that flagged instance was a false positive.)
Symlink-following writes -> project-safe helpers:
- utils/active_sampler.py: atomic_write_text (cwd-contained, symlink-reject).
- ui/app.py: tempfile.mkstemp (O_EXCL, unpredictable name) instead of a
fixed shared-temp path a local attacker could pre-symlink.
typer.Exit-vs-SystemExit audit: the two named instances (bench.py, eval.py)
were fixed in the HIGH tier; cli.py's top-level run() already handles both
SystemExit and typer.Exit correctly, so no further sites needed changes.
Adds tests/test_code_review_recurring.py (7 regression tests). ruff clean;
full suite 14851 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Training correctness (silent wrong results):
- ppo: refuse a randomly-initialised reward head when only reward_fn is set
(trl 0.19.1 PPO can't use a reward_fn) instead of training against noise.
- edit_kernels (AlphaEdit): reject a non-finite key-norm (NaN <= 0.0 is False).
- preference_combine (ORPO): length-normalise log-probs so exp() doesn't
underflow and kill the odds-ratio correction (+_read_lens caller wiring).
- ipo: anneal the beta schedule from ipo_tau, not the DPO default dpo_beta.
- distill: mask padding + prompt tokens in the default KL term (labels!=-100).
- block_expansion: freeze all-but-the-ACTUAL-added blocks (clamp over-request).
- formats (KTO): map a -1 label to False (bool(-1) was silently True).
Features that silently did nothing:
- sft: actually install the LongLoRA S² attention override (defensive).
- train --gpus re-exec: pass through --gate/--push-as/--trust-remote-code/
--tracker/--diagnose-gate/--annex-xi/--repro-receipt/--profile/energy flags.
- eval gate-install hook: pass $GATE_SUITE to `soup eval against`, which now
validates the locked suite as a precondition (block on missing/tampered).
- deploy_measure: fold the candidate list into the cache key.
Security:
- sglang: loopback-only CORS (was wildcard).
- fetch: lstat the ORIGINAL path (realpath resolved the symlink -> S_ISLNK
never fired -> write followed the link).
- ui /api/data/inspect: is_under_cwd (commonpath) instead of str.startswith.
- registry lineage: unbounded cycle check (the depth-10 cap accepted a
far-away cycle-closing edge).
- gguf calib: read from the O_NOFOLLOW fd (no close+reopen TOCTOU window).
- namespace_pin: flag ANY created_at drift (repo-recreation moves it forward).
Robustness / cross-platform:
- bench: re-raise typer.Exit (RuntimeError subclass) instead of masking it.
- eval auto: catch typer.Exit so a benchmark failure falls through.
- data split: reject negative --val/--test (negative slice inverted the split).
- trace parser: read utf-8-sig so a BOM'd first record isn't dropped.
- terraform plan: tolerate batch_size="auto" in the runtime estimate.
- rl_checkpoint: only rank-0 writes; atomic optimizer save.
Adds tests/test_code_review_high.py (29 regression tests). ruff clean;
full suite 14844 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. RLVR verifiable rewards (grpo.py, ppo.py): pass verifiable_domain=
to load_reward_fn so `reward_fn: verifiable` + `verifiable_domain: math`
no longer crashes at setup() — the headline RLVR feature was 100% broken.
2. serve: default --host to 127.0.0.1 (was 0.0.0.0) and add an opt-in
--tool-auth-token wired to _create_app so the code-exec tool endpoints
are not exposed unauthenticated on all interfaces; warn on non-loopback
bind without a token.
3. eval benchmark: reject ','/'=' in the adapter's base_model_name_or_path
(and --model path) before lm-eval model_args interpolation, closing the
trust_remote_code=True injection (mirrors the ship.py guard).
4. webhooks: run the private/link-local SSRF check for BOTH http and https
(was nested in the http-only branch, so https://169.254.169.254 and
10.x/192.168.x sailed through). Adds allow_private_hosts= so the trusted
loop_stages LAN-deploy caller (which does its own loopback/LAN tightening)
keeps working.
5. cloud/modal: emit the output_dir via the already-repr'd _LOCAL_OUTPUT
variable in a generated f-string instead of raw interpolation, closing
the stub code-injection hole (also quote-safe, unlike bare {output_dir!r}).
6. eval gate: wire cfg.training.eval_gate through all 14 trainers to
SoupTrainerCallback, and load the suite/baseline + build a live generator
in on_train_begin so --gate actually halts training. Warn honestly when
forgetting/checkpoint/early-stop knobs are set (they are not yet enforced).
Adds tests/test_code_review_critical.py (27 end-to-end regression tests).
ruff clean; full suite 14815 passed / 120 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two TestRewardHackMitigationCli help-text assertions grepped the raw
`soup train --help` output for the new flags. CI runners emit color codes
that split long option names into non-contiguous characters (same failure
mode documented in test_eval_gate.py for --gate), so the raw substring check
failed on every OS/Python combo while passing locally on a no-color terminal.
Fix follows the established repo pattern: render with COLUMNS=200 (no option
wrapping) and strip ANSI escapes before the membership check. Verified under
FORCE_COLOR=1: both assertions pass. Test-only change, no version bump.
Folds in the doc-vs-reality cleanup the plan flagged: the docs referenced
soup train --reward-hack-detector / --reward-hack-halt but they were config-only.
Add both as CLI passthroughs mirroring --reward-hack-mitigation (validate value,
set cfg.training field, accelerate re-exec passthrough) so the docs are true and
the reward-hack CLI is consistent. +4 tests (test_v07126: 180 -> 184; full suite
-> 14788).
Fixes from 5 sequential ECC reviews (python/code/security/tdd/verification).
python-review (2 CRITICAL + HIGH/MED/LOW):
- signal/vote coherence: schema now requires the active detector in
reward_hack_signals + rejects the inactive detector name (was silently
dropping the primary signal from the vote).
- integral_clamp is its own field (was wrongly hard-wired to beta_ceil).
- task/backend gate runs before controller-config checks; EMA convention
corrected; type hints; mutable-list default -> tuple + normalised compare.
code-review (4 HIGH + MED/LOW):
- _prune now trims _saved in sync with disk (rollback can't target a deleted
checkpoint); bang-bang release_count resets after each relaxation (hysteretic
descent); EMA formula uses standard convention; _escalate no longer burns a
recovery attempt on a None target; max_recovery_attempts>=1 required with
rollback; _action_history capped; on_step_end logs errors once; loud warning
when the mitigation callback can't attach (was a silent safety-off).
security-review (HIGH + MED):
- restore_checkpoint / save_checkpoint refuse a SYMLINKED optimizer.pt
(torch.load weights_only=False was an RCE via attacker-placed symlink);
bool-before-int/float guards on all new numeric fields; reward_hack_signals
max_length=4; empty-signals guard in the callback.
tdd-review: +13 coverage tests (dead-band hold, shim verbatim-on-error,
conservative boundary, read-only-beta dual-write, escalation postconditions,
both-restore, PID D exact, log cap + concurrency, no-top-level-import, fuzz
field-validity).
Test count 152 -> 180 (+2 POSIX-only symlink skips).
Recipe-count drift: the CLI help (commands.md) and Web UI pages
(serving-and-export.md) claimed 43 ready-made recipes and the catalog
docstring said ~30, while RECIPES actually holds 116 (test_recipes already
asserts len == 116). Aligned every user-facing count to 116.
Also refreshed 6 recipe descriptions still tagged "schema-only stub
(live in v0.52.1)": the TTS task (#131) and BitNet 1.58 SFT (#134) went
live in v0.71.20, so orpheus/sesame/llasa/spark/oute TTS and the Falcon-E
BitNet recipe now read "live (v0.71.20)".
Doc-drift only — description strings + one comment; no functional change,
no version bump.
The live-codec block claimed the entire data.format=audio path was "not
validated on the maintainer's box" and only surfaced a RuntimeError. v0.71.22
made the Orpheus SNAC encode live + validated; note that while the other four
families stay dependency-gated. Docs-only, no version bump.
- performance-and-quantization.md: the Quant Menu multi-trainer note said
"vision / audio modality is still SFT-only inline-BNB (wiring tracked as a
follow-up)" — stale after v0.71.19 #81 dropped the modality gate. Now states
vision/audio thread the unified loader (full gptq/awq/hqq/aqlm/eetq/mxfp4/fp8
menu), with the upstream class+kernel caveat.
- peft-and-efficiency.md: added the v0.71.19 #80 multi-GPU sharding paragraph to
the Multipack section (accelerator.prepare + BatchSamplerShard under
num_processes>1; identical bin seed across ranks; single-GPU unchanged).
Docs-only — no version bump / tag (the v0.71.19 code already shipped at f51331d).
Alpamys
Darsh
Salil M
Kondamwar Akshaya Shrikant
Shatakshi Prasad