110 KiB

Raw Blame History

Security Policy

Supported Versions

We provide security updates for the following versions:

Latest minor version: Active support (e.g., v0.20.x)
Previous minor versions: Bug-fix support only
Versions older than 3 minor versions: No support

Example:

v0.53.0 -- Full support (latest)
v0.52.0 -- Full support
v0.51.0 -- Full support
v0.50.0 -- Full support
v0.49.0 -- Full support
v0.48.0 -- Bug-fix support only
v0.47.0-v0.47.x -- Bug-fix support only
v0.46.x and below -- No support

Reporting a Vulnerability

Do not open a public issue or pull request for security vulnerabilities.

Instead, use GitHub Security Advisories to report privately, or email vpn.alpamys@gmail.com with:

Description: A clear explanation of the vulnerability
Steps to Reproduce: How to trigger or demonstrate the issue
Affected Versions: Which Soup versions are impacted
Suggested Fix (optional): Any proposed solutions
Contact Info: Your email for follow-up (optional)

What to Include

To: vpn.alpamys@gmail.com
Subject: Security Vulnerability Report: [Brief Title]

Description:
[Explain the vulnerability in detail]

Affected Component:
[e.g., data/loader.py, trainer/sft.py, etc.]

Steps to Reproduce:
1. [Step 1]
2. [Step 2]
3. ...

Impact:
[What could go wrong? Data exposure? RCE? DoS?]

Suggested Fix (optional):
[Your proposed solution, if any]

Response Timeline

Initial Response: Within 48 hours
Assessment: 1-3 business days
Fix Development: Varies by severity
Patch Release: As soon as possible after fix verification
Public Disclosure: Coordinated with reporter (typically 90 days after patch release)

Severity Levels

Critical: Remote code execution, data exposure, complete compromise (patch within 24-48 hours)
High: Authentication bypass, privilege escalation, denial of service (patch within 1 week)
Medium: Information disclosure, partial compromise (patch within 2 weeks)
Low: Minor issues with limited impact (patch in next regular release)

Security Best Practices

When using Soup, follow these practices to stay secure:

1. Keep Soup Updated

pip install --upgrade soup-cli

2. Protect API Keys

Never commit API keys or secrets to version control. Use environment variables:

export HUGGINGFACE_TOKEN=your_token_here
export WANDB_API_KEY=your_key_here
soup train

3. Validate Data

Only use trusted datasets
Verify checksums for large datasets
Inspect data for malicious content before training

4. Model Permissions

Be cautious when downloading models from untrusted sources
Use model hub providers with verified publishers (HuggingFace, Meta, etc.)
Keep track of which models you've fine-tuned and their base model sources

5. GPU/Compute Safety

Run on isolated machines if training on sensitive data
Clear cache and temporary files after training
Don't share fine-tuned models containing sensitive information

Known Vulnerabilities

We maintain a log of known security issues and their fixes. This will be updated as issues are discovered and resolved.

Current Status

No known critical vulnerabilities in current releases.

Security Hardening History

v0.10.10: Bearer token auth on Web UI, CORS restrictions, path traversal protection, SSRF prevention, max_tokens limits, supply-chain pinning (llama.cpp b5270), deprecated CLI secret flags
v0.12.0: experiment_name path traversal validation, GaLore parameter type enforcement
v0.13.0: Batch inference max_tokens capped at 16384, trust_remote_code warning
v0.14.0: Plaintext loader UTF-8 encoding, MoE config validation (moe_aux_loss_coeff ge=0, moe_lora boolean only)
v0.14.3: Data validate auto-detects format, Web UI --show-token flag + auth token documented
v0.15.0: rope_scaling_type Literal constraint, max_length bounds (ge=64, le=1048576), FSDP config key allowlist, Liger Kernel exception handling narrowed
v0.16.0: embedding_loss Literal constraint, embedding_margin gt=0 validation, ONNX export without trust_remote_code (with warning), TensorRT export subprocess list args (no shell injection), speculative decoding SSRF-protected (URL blocked) with warning panel, vLLM speculative model URL validation
v0.17.0: Server data generation provider SSRF validation (scheme whitelist + localhost-only HTTP), audio model trust_remote_code warning panel, audio file path traversal protection (resolved paths confined to audio_dir), SGLang backend trust_remote_code warning panel
v0.18.0: Ollama deploy GGUF path traversal protection + .gguf extension validation, model name validation (no path separators/null bytes), subprocess list args (no shell injection), Modelfile parameter key allowlist + value newline/null sanitization, overwrite warning panel
v0.19.0: Custom eval JSONL schema validation + 10k task cap, regex scoring ReDoS guard, judge API SSRF protection + API key isolation, human eval local-only terminal UI + 10k prompt cap, leaderboard read-only SQLite queries
v0.20.0: Ollama provider localhost-only validation (remote blocked), Anthropic provider API key from env only (never CLI arg), vLLM provider SSRF protection (scheme whitelist + localhost-only HTTP), output path traversal protection (.. blocked), configurable rate limiting (--requests-per-minute)
v0.21.0: Migrate input/output path traversal protection (resolve + relative_to(cwd)), Unsloth .ipynb AST-only parsing (no exec/eval), recipes output path traversal protection, NEFTune config bounded (ge=0.0, le=50.0)
v0.22.0: Multi-adapter serving path traversal protection (resolve + relative_to(cwd)), adapter name validation (alphanumeric + hyphens only), unknown adapter returns 404 (not 500)
v0.23.0: AWQ/GPTQ calibration data path traversal protection (resolve + relative_to(cwd)), AWQ/GPTQ output path stays under cwd, curriculum_buckets bounded (ge=1, le=20), AWQ/GPTQ trust_remote_code warning panel
v0.24.0: HF download trust_remote_code=False + warning panel, HF download output path sanitized (Path.name), download --samples capped at 1M, dataset registry name validation (no path separators/null bytes), registry path traversal protection, loss_watchdog threshold le=100 + patience le=1000, freeze_layers le=1000
v0.24.1: AWQ/GPTQ output path traversal validation moved before import check (previously unreachable when autoawq/auto-gptq not installed), Windows Unicode fix for Rich console output (replaced non-ASCII symbols with ASCII equivalents)
v0.24.2: Chat proxy SSRF protection (localhost-only HTTP, HTTPS for remote), chat proxy max_tokens capped at 16384 + temperature/top_p bounded, chat proxy Bearer token auth required, XSS prevention (HTML-escape before markdown render), runs compare max 5 runs, config from-form validates via load_config_from_string, SSE read endpoints no auth (GET)
v0.25.0: Tool-calling JSON-only parsing (no eval), RLVR math_verify regex-extracted numerics (no eval), code_exec 5s timeout + 512MB RLIMIT on POSIX + ephemeral cwd + socket patch + python -I -S + 10KB output cap, verifiable_domain Literal constraint, LoRA PEFT mutual exclusion (DoRA/VeRA/OLoRA), data augment path containment + caps, forgetting_detection bounds, checkpoint_intelligence bounds + symlink refusal, autopilot path containment (realpath + commonpath) + goal Literal + GPU/time budget bounds, MLX trainers no trust_remote_code
v0.26.0 — Registry: name/tag validation (alphanumeric + _-. only, null-byte rejected, name ≤128 / tag ≤64 chars), artifact path containment (default enforce_cwd=True via os.path.realpath + commonpath, stored path is realpath), SQL LIKE wildcard escaping (% and _ escaped with ESCAPE '\\' in search() and prefix resolve()), DB 600 perms on POSIX, lineage indirect-cycle detection (BFS ancestor walk before insert), CLI Rich markup escaped everywhere, resolve() raises AmbiguousRefError on ambiguous prefix (no silent None)
v0.26.0 — Eval Gate: suite path via shared utils/paths.is_under_cwd containment, regression_threshold [0.0, 1.0], every_n_epochs [1, 100], on_regression Literal ("stop"/"warn"/"continue"), GateTask.tasks/prompts null-byte rejection, judge_model URL scheme allowlist (ollama://, https://, http://localhost/http://127.0.0.1) — SSRF hardening, callback fails-safe: structured errors treated as regressions under on_regression="stop"
v0.26.0 — Trace-to-Preference: input/output path containment via shared is_under_cwd, trace line cap 100,000, --format/--signal Literal validation, PII warning panel before every run, JSON-only parsing (no eval), malformed JSON lines skipped silently
v0.26.0 — Quant-Lobotomy: --before/--after/--tasks all containment-checked, registry:// refs support optional kinds filter to avoid picking the wrong artifact, format Literal validated
v0.26.0 — Soup Cans: Manifest format version pinned to 1; name alphanumeric+_-.; author max 128 chars, no null bytes/newlines; created_at must parse via datetime.fromisoformat; description max 4096; DataRef URL HTTPS-only; hf_dataset regex-validated; tar extraction uses filter="data" on Python 3.12+, fallback only on TypeError/AttributeError (not TarError); manual symlink/hardlink rejection + commonpath check; 100 MB size cap on pack + fork; dunder-key (__*__) and null-byte rejection in fork modifications to prevent prototype pollution; inspect/read_config refuse paths outside cwd
v0.27.0 — Multi-GPU Mastery: --gpus bounds (reject bool, non-digit, zero, negative, values above MAX_GPU_COUNT=128); --gpus auto on 0-GPU host prints explicit yellow warning (no silent no-op); Rich markup escaped on --config path before embedding in the multi-GPU advice Panel; accelerate launch argv assembled via shlex.quote per element (copy-pasted command safe against crafted paths); build_accelerate_argv validates num_processes >= 1, mixed_precision Literal (no/fp16/bf16/fp8), num_machines bounded [1, 256]; ZeRO++ integer literals (int(1e9) not float) so DeepSpeed strict JSON validator accepts; validate_fsdp2_compile_config requires FSDP + CUDA + transformers + torch>=2.2/accelerate>=0.27; DeepSpeed-MII stub exits non-zero to prevent silent mis-start; validate_pipeline_config enforces pipeline_stages >= 2 + CUDA + gpu_count >= stages; pipeline_stages Pydantic bounds [1, 16]; parallelism Literal data|pipeline; NCCL env (NCCL_P2P_DISABLE/NCCL_IB_DISABLE/NCCL_NVLS_ENABLE) applied via os.environ.setdefault only — user/launcher overrides are never stomped
v0.28.0 — Training Speed & Memory: quantization_aware: Union[bool, Literal["fp8"]] rejects arbitrary strings (only true / false / "fp8"); FP8 path requires CUDA + Hopper+ SM capability + transformers backend; gradient_checkpointing: Union[bool, Literal["selective","medium","full","auto"]] rejects unknown tier strings and returns only HF-supported keys (no private markers leak into TrainingArguments.gradient_checkpointing_kwargs); activation_offloading Literal cpu|disk, scratch save_dir containment-enforced via shared utils/paths.is_under_cwd before disk writes, torch.load(weights_only=True) prevents arbitrary Python deserialization on reload, TOCTOU closed between mkstemp and torch.save by holding the fd open, best-effort cleanup on context exit (handles SIGKILL mid-backward); kernel_picker.pick_best_kernel raises ValueError when all candidates lack a finite time_ms (prevents silent promotion of an untimed combo); Cut CE architecture detector matches on last path component only (so deepseek-ai/...-phi-... org-prefix does not trigger a Phi patch on a DeepSeek model); build_cross_doc_mask numpy-vectorised to avoid O(seq_length²) pure-Python fill at max_length bound (1M); @model_validator requires packing=true when packing_cross_doc_attn_mask=true (prevents silent no-op); SoupConfig._validate_v028_speed_memory_sft_only rejects use_cut_ce/quantization_aware="fp8"/kernel_auto_compose/activation_offloading on non-SFT tasks — prevents legacy int8-QAT wrapper from crashing on the string "fp8" and prevents silent no-ops on DPO/GRPO/KTO/etc. (multi-trainer wiring tracked for v0.28.1)
v0.29.0 — HF Hub Deep Integration: HF_ENDPOINT SSRF-hardened — scheme allowlist (http/https), null-byte rejection, 0.0.0.0 explicitly rejected, plain-HTTP only permitted for loopback (localhost/127.0.0.1/::1), RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via ipaddress.ip_address; repo ID regex [A-Za-z0-9][A-Za-z0-9._-]{0,95} per component, ≤200 chars total, null-byte / whitespace / .. / leading-/ rejection (applied to push --repo, train --push-as, data push --hf-dataset, deploy hf-space --model/--space); collection slug owner/slug-hash regex-validated, ≤256 chars; HF token resolution single-sourced in utils/hf.resolve_token (env > cached login), explicit non-printable tokens rejected, push --token flag deprecated with yellow warning; soup push --model confined to cwd via is_under_cwd (prevents crafted soup.yaml output: from uploading system files); auto-push checkpoint allow_patterns restricts uploaded files to *.safetensors/*.bin/*.pt/*.json/tokenizer*/trainer_state.json/training_args.bin/README.md (keeps .env and source files out of auto-pushed branches); prepare_hf_resume enforces cwd containment and passes local_dir_use_symlinks=False (defeats symlink-based FS escape on older huggingface_hub); commit messages stripped to first line and capped at 200 chars (prevents multi-line injection into public HF commit history); _render_eval_scorecard neutralises |/[/]/(/)/!/newlines/tabs/</> in task names and non-numeric scores; data_lineage HTML-escaped (defeats XSS on HF Hub README viewer); render_space_template validates model_repo via validate_repo_id before substitution into rendered app.py (crafted repo id cannot inject Python code); HFPushCallback uses sticky _repo_failed flag to short-circuit retries after hard failure (no log spam, no wasted API calls); add_to_collection prefers HfHubHTTPError 409 detection over string-match for duplicate handling
v0.30.0 — Inference Excellence: OTLP endpoint SSRF-hardened matching v0.29.0 HF_ENDPOINT (scheme allowlist, 0.0.0.0 rejected, RFC1918 / link-local / cloud-metadata via ipaddress.ip_address, plain HTTP loopback-only); pick_draft_model rejects URL-scheme target names (http:///https:///file://), null bytes, names >200 chars; validate_regex_pattern length-capped at 2048 + null-byte rejection + must compile; validate_json_schema dict shape + 64KB serialised cap + required type field; --json-schema file path confined to cwd via shared utils/paths.is_under_cwd; --structured-output json requires --json-schema (fail-fast prevents silent no-op); FastAPI /v1/adapters/activate/{name} pattern ^[a-zA-Z0-9][a-zA-Z0-9\-]*$ enforced before handler runs; activate/deactivate state protected by threading.Lock (no race on concurrent hot-swap); /v1/adapters response omits filesystem paths (names + active flag only); CORS on transformers backend restricted to loopback origins (^https?://(localhost|127\.0\.0\.1)(:\d+)?$) since hot-swap endpoints mutate state without auth; auto_quant.Candidate name regex + score [0.0, 1.0] + finite-float check + non-negative latency; pick_best generator-safe (materialises to list) so error-message count is accurate; OTel span context uses contextlib.ExitStack so __exit__ sees real exception info (spans correctly marked error on HTTPException); record_latency always runs in finally so tail-latency percentiles include failure paths; build_tracer idempotent — only installs provider when current is ProxyTracerProvider/NoOpTracerProvider (preserves operator-supplied instrumentation)
v0.32.0 — Training Stability & Auto-Tuning: --find-lr-output containment via shared utils/paths.is_under_cwd (prevents writes outside cwd); save_lr_finder_report rejects NaN / Infinity floats in lrs / losses and serialises with allow_nan=False (keeps the report parser-safe); compute_lr_schedule rejects non-positive start_lr, inverted ranges, and num_steps outside [2, 10_000]; pick_mixed_precision rejects empty / null-byte / >200-char model names and resolves multi-version quirks (qwen2.5 vs qwen2, phi-3.5 vs phi-3) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; compute_warmup_steps clamps to [10, 1000] with a ratio==0.0 short-circuit matching HF Trainer's "no warmup" convention; SpikeRecoveryStrategy is @dataclass(frozen=True) (post-construction mutation cannot bypass validation), max_attempts ∈ [1, 10], lr_decay ∈ (0, 1), min_lr > 0; cross-validator _validate_spike_recovery_requires_watchdog rejects loss_spike_recovery=true, loss_watchdog=false at config-load (fails fast instead of never triggering); convergence_window ∈ [5, 10_000], convergence_rel_tol ∈ (0, 1], recommend_action reuses detect_plateau so plateau heuristic stays single-source-of-truth; GradAccumMonitor.recommend() caps doubled accum at MAX_ACCUM=1024 so a runaway advisory loop cannot blow up DataLoader prefetch; generate_config validates BOTH the YAML output path AND the embedded decisions["output"] field via is_under_cwd (closes the gap where a crafted decisions["output"]="../../etc" would have silently propagated into the rendered YAML)
v0.34.0 — Observability & Dev UX: .crash bundle generator (utils/crash.py) recursively redacts hf_* / sk-* / Bearer … token-shaped strings in any captured config and metric tail before serialisation, so a .crash file shared on a public GitHub issue cannot leak credentials; output_dir is reduced to os.path.basename so $HOME doesn't leak; write_crash_bundle uses os.path.realpath + commonpath for cwd containment (Windows-safe; raises ValueError not PermissionError so callers cannot silently swallow with except OSError); filename appends secrets.token_hex(4) so two crashes in the same UTC second don't collide; bundle truncated to MAX_BUNDLE_BYTES=1_000_000. train.py crash-write surfaces failures to the user (no silent missing-bundle). profiling.py resolve_trace_path rejects empty / . / .. / / / \\ / null-byte run_id (closes the output_dir/profiles/../trace.json escape) and uses os.path.realpath + is_under_cwd; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). tracker.get_run LIKE-prefix match escapes % / _ / \\ and uses ESCAPE '\\' so a crafted run_id cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (_ensure_schema) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). runs.py show/replay/clean switched user run_id rendering to markup_escape and switched clean containment from broken Path.resolve() + relative_to() to project-standard os.path.realpath + is_under_cwd. tui_app.py lazy-imports ExperimentTracker and markup_escapes every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject [bold red]…[/] markup. run_cost.estimate_run_cost_usd rejects bool in num_gpus (bool is a subclass of int — same defence as v0.30.0 Candidate.__post_init__); duration clamped to [0, 1 year]; unknown GPU returns None so callers render — instead of fabricating $0.00. log_level.parse_log_level rejects non-string + null-byte input.
v0.33.0 — Live Wire: RLVR code_exec_reward adds OS-level isolation (Linux best-effort os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID), macOS sandbox-exec with default-deny MACOS_SANDBOX_PROFILE narrowed to a 3-name mach-lookup allowlist to prevent DNS / NSURLSession bypass of (deny network*)); prune_checkpoints switches to TOCTOU-safe os.lstat + S_ISLNK + shutil.rmtree(onerror=_abort_on_symlink) so a symlink encountered mid-walk aborts rather than escapes; run_gate wraps each task scorer in a typed try/except so backend failures produce score=None, error=str(exc) (never silent score=1.0); _parse_judge_url removes the bare http:// catch-all (defence-in-depth after the Pydantic GateTask validator); soup can run requires --yes or explicit consent callback and raises ValueError (not PermissionError, which is an OSError subclass that broad except blocks would swallow); GGUF rglob result for ollama deploy is realpath+commonpath checked against extract_dir (prevents symlink escape from a crafted can); DeployTarget.path validator normalises mixed \\// separators before splitting (closes a Windows .. bypass); CAN_FORMAT_VERSION 1→2 (additive — v1 still loads); soup can publish validates repo_id via utils/hf.validate_repo_id, resolves token via resolve_token, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; _write_spike_recovery_hint adds is_under_cwd containment check on args.output_dir from raw HF TrainingArguments; lookup_entry_by_output_dir emits ResourceWarning when 1000-row scan limit is hit (no silent miss); CrossDocCollator no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); Candidate rejects bool in score/latency_ms (was sneaking past int isinstance check); evaluate_candidate latency mean now divides by completed prompts (excludes crashed) so a broken candidate isn't artificially fast; auto_quant.run_auto_quant_picker soft-falls-back to highest-scored candidate when no candidate clears min_score (server still binds); build_logits_processors returns [] when neither outlines nor lm-format-enforcer is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; os.execvp auto-reexec uses list args (no shell), all forwarded flags pre-validated; cleanup_extract_dir uses os.path.commonpath (Windows-safe) instead of startswith; _run_subprocess catches TimeoutExpired and returns rc=124 (coreutils convention) instead of an unhandled traceback; new eval_results and tensorrt artifact kinds in RegistryStore._VALID_KINDS
v0.53.0 — Quant Menu II (UD GGUFs + KV cache + NVFP4 + LF parity + save formats): 6 schema-only Parts; live wiring deferred to v0.53.1. Every new validator follows the project's established hardening policy: closed allowlists (UD_GGUF_FORMATS, IQ_GGUF_FORMATS, APPLE_ARM_GGUF_FORMATS, KV_CACHE_TYPES, MERGE_SAVE_FORMATS, TORCHAO_PTQ_SCHEMES) as frozenset so registries cannot be mutated; _GGUF_METADATA / _KV_CACHE_METADATA / _MERGE_METADATA / _TORCHAO_METADATA wrapped in MappingProxyType; _LOWER_INDEX for GGUF lookup is also MappingProxyType-wrapped (replaces O(N) walk with O(1) lookup — code-review MEDIUM fix). All string validators reject non-string / bool / empty / null-byte / oversize with case-insensitive normalisation (matches v0.41.0 validate_optimizer_name / v0.51.0 validate_hub_name policy); validate_torchao_scheme is INTENTIONALLY case-sensitive (PyTorch class names — torchao.quantize_ looks them up by exact name) with the asymmetry documented at both validators (security-review LOW fix). validate_calibration_data_path + validate_quant_config_path are shape-only at this release; their docstrings name the exact controls a v0.53.1 CLI dispatch contributor MUST add (os.path.realpath + os.path.commonpath cwd containment, os.lstat + stat.S_ISLNK symlink rejection before open(), existence check, yaml.safe_load-only for quant configs) — closes the security-review MEDIUM "documentation gap at trust boundary" finding. SoupConfig cross-validators: _validate_fp8_attention_compat (requires quantization_aware='fp8' BEFORE the MLX gate so the more actionable error fires first — code-review MEDIUM fix); _validate_nvfp4_compat (non-MLX + modality='text'; Blackwell SM ≥ 12.0 runtime check fires at trainer construction); _validate_unsloth_bnb_4bit_compat (requires backend='unsloth' + quantization='4bit'); _validate_bnb_4bit_double_quant (requires quantization='4bit' — rejects none/8bit/Quant-Menu); _validate_llm_int8_alias (asserts quantization='8bit', deliberately disjoint from v0.41.0 load_in_8bit aliasing); _validate_quantize_ref_reward (extended ref-task allowlist {dpo, ipo, simpo, orpo, bco, kto, preference, grpo, ppo} per code-review HIGH fix — first-cut omitted grpo + kto + ppo which all have reference policies); _validate_kv_cache_type_supported (only fp8 gated to non-MLX in v0.53.0; q8_0/bf16/f16 pass-through documented at validator site so v0.53.1 contributor sees the gate immediately). requires_hopper reads from _KV_CACHE_METADATA spec — single source of truth so adding a Hopper-only type means flipping the spec field only (code-review MEDIUM fix). All 7 new bool fields share _validate_v053_bool_fields field_validator(mode='before') that rejects bool-as-int with explicit TypeError("v0.53.0 flag must be bool") and passes None through to Pydantic's default=False rather than silently coercing it (python-review MEDIUM fix — fp8_attention: null in YAML now surfaces as a "valid boolean" ValidationError instead of masquerading as False). Known limitations: (1) Every live wiring is deferred to v0.53.1 — export_advanced_gguf, apply_kv_cache_type, apply_fp8_attention, apply_nvfp4, merge_4bit, export_torchao all raise NotImplementedError with explicit v0.53.1 markers. (2) validate_calibration_data_path + validate_quant_config_path are shape-only this release; CLI dispatch in v0.53.1 MUST add cwd-containment + TOCTOU symlink rejection. (3) kv_cache_type MLX permissive policy: only fp8 is rejected, the other three pass-through; v0.53.1 may narrow further. (4) Hopper SM-capability check is runtime-only — schema accepts kv_cache_type='fp8' + fp8_attention=true without GPU probe. (5) NVFP4 + Blackwell (SM ≥ 12.0) check is runtime-only. (6) bnb_4bit_use_double_quant only gated against quantization, not against quantization_aware — the latter combination is already rejected by v0.28.0 Quant-Menu + QAT cross-validator. (7) llm_int8 is an assertion not an aliaser — diverges from v0.41.0 load_in_8bit design on purpose. (v0.53.0)
v0.52.0 — Modality II (TTS + Distillation + BitNet + EBFT-GDPO + MoE quant + reasoning_effort): 7 schema-only Parts; live trainer / loss / export wiring deferred to v0.52.1. Every new validator follows the project's established hardening policy: closed allowlist (SUPPORTED_TTS_FAMILIES, CLASSIFIER_TASKS, DIVERGENCES, BITNET_QUANT_FORMATS, BITNET_EXPORT_FORMATS, EBFT_VARIANTS, GDPO_VARIANTS, MOE_EXPERT_QUANT_FORMATS, REASONING_EFFORT_LEVELS, per-family _FAMILY_EMOTIONS) wrapped in frozenset / MappingProxyType so registries cannot be mutated at runtime; validate_* helpers reject non-string / bool / empty / null-byte / oversize / unknown inputs with case-insensitive normalisation (matches v0.41.0 validate_optimizer_name / v0.50.0 grpo_variant / v0.51.0 hub policy); float validators (validate_distill_temperature, validate_ebft_temperature) gate on math.isfinite to reject NaN AND ±inf (matches v0.32.0 save_lr_finder_report policy). field_validator(mode="before") on num_labels (security-review HIGH fix) rejects bool before Pydantic's ge=1 coercion silently treats True as 1. Field validator on reasoning_effort routes through the shared validate_reasoning_effort helper so the schema and runtime validator agree on what's accepted (security-review MEDIUM fix). SoupConfig cross-validators: _validate_tts_compat (requires task='tts' + modality='audio_out' + non-MLX backend; per-family emotion allowlist via _FAMILY_EMOTIONS), _validate_classifier_compat (with lazy-import early-return — code-review HIGH fix — so SFT hot path doesn't pay import cost; requires num_labels on classifier tasks; rejects classifier-only fields outside the task family with named offenders), _validate_distill_compat (requires teacher_model when task='distill'; rejects distill-only fields outside the task), _validate_bitnet_compat (gates to non-MLX + text-modality + task ∈ {sft, pretrain, dpo}), _validate_ebft_compat + _validate_gdpo_compat (task-family gates), _validate_moe_expert_quant_compat (requires moe_lora=true to prevent silent no-op), _validate_reasoning_effort_task_gate (code-review HIGH fix — rejects reasoning_effort + train_on_eot outside the SFT-family task set with named offenders; mirrors v0.50.0 GRPO stability task-gate policy). Public DIVERGENCES frozenset is derived from _DIVERGENCE_ALIASES so adding a new alias updates both the accepted-input set and the error message in lockstep (review fix LOW). validate_bitnet_export enforces a closed-allowlist canonical form for soup export --format <bitnet|tq1_0>, both of which are CLI-registered with a yellow advisory panel + Exit(0) stub (no artifact written until v0.52.1 — the format flag is accepted so existing scripts pinned to v0.52.0 will not break). 6 new YAML recipes appended (5 TTS + Falcon-E BitNet) — every entry is exercised by tests/test_v0520.py for load_config_from_string round-trip + _no_null_or_whitespace model-id check (mirrors v0.51.0 review-fix LOW). Known limitations: (1) Every live trainer / loss / export path is deferred to v0.52.1 — build_tts_trainer, build_classifier_trainer, build_distill_trainer, build_bitnet_trainer, export_bitnet_gguf, apply_ebft_loss, apply_gdpo_loss, apply_moe_expert_quant all raise NotImplementedError with explicit v0.52.1 markers; schema accepts every new task / quant / variant + the CLI stub for soup export --format bitnet/tq1_0 prints a deferred-advisory panel and exits 0. (2) modality='audio_out' accepted on non-TTS tasks — design choice this release so future audio-output tasks (ASR / V2A) can reuse it; today's runtime trainer dispatch must check task == 'tts' to avoid silent routing into the deferred TTS path. (3) Oute emotion allowlist is a tight 6-entry subset (neutral / happy / sad / angry / calm / excited); operators wanting custom emotions will need a v0.52.1 patch to extend OUTE_EMOTIONS. (4) is_bitnet_model is best-effort heuristic over name prefixes (bitnet, falcon-e, 1bitllm, onebit); a BitNet checkpoint published under an org without any of those prefixes returns False. This is detection, not gating — the trainer wrapper (v0.52.1) loads the model regardless of the heuristic. (5) quantization='bitnet_1.58' gated to task ∈ {sft, pretrain, dpo} — extending to GRPO / PPO / RewardModel requires upstream onebitllms RL kernels not yet shipped. (v0.52.0)
v0.51.0 — Model Catalog Expansion + Alternative Model Hubs: 5 release Parts. New soup_cli/utils/hubs.py ships closed allowlist SUPPORTED_HUBS = frozenset({hf, modelscope, modelers}) + three MappingProxyType-wrapped registries (_HUB_DEFAULT_ENDPOINTS / _HUB_ENDPOINT_ENV / _HUB_PACKAGE) so the registry cannot be mutated at runtime (matches v0.36.0 _REGISTRY policy). validate_hub_name rejects non-string / bool / empty / null-byte / >32-char / unknown with case-insensitive normalisation (matches v0.41.0 validate_optimizer_name policy). validate_hub_endpoint is the SSRF kernel — full parity with v0.29.0 utils/hf.resolve_endpoint: scheme allowlist (http/https only), null-byte rejection, control-character / CRLF rejection added in v0.51.0 as a defence-in-depth review fix (defends against URL-as-HTTP-header injection if the URL ever flows into a raw HTTP client), 0.0.0.0 explicitly rejected, plain HTTP only for loopback {localhost, 127.0.0.1, ::1}, RFC1918 / link-local / cloud-metadata IPs (169.254.x) rejected via ipaddress.ip_address for plain HTTP. resolve_endpoint(hub, *, env=None) looks up the per-hub env var (HF_ENDPOINT / MODELSCOPE_ENDPOINT / MODELERS_ENDPOINT) and runs the override through validate_hub_endpoint; default endpoints are baked-in HTTPS URLs. is_hf rejects bool explicitly (review fix HIGH — bool is a subclass of int and would have silently fallen through hub.lower() == "hf" → False, which happens to be correct by accident but violates the contract; matches v0.30.0 Candidate / v0.34.0 estimate_run_cost_usd policy). TrainingConfig.hub: Literal["hf","modelscope","modelers"] field gets a field_validator(mode="before") _normalize_hub that delegates to validate_hub_name so hub: HF in YAML normalises to "hf" (review fix HIGH — first-cut had Pydantic Literal exact-match while validate_hub_name was case-insensitive, breaking the v0.41.0 validate_optimizer_name / v0.50.0 grpo_variant / rollout_backend policy of agreement between schema and shared validator). SoupConfig _validate_hub_supported cross-validator rejects hub != 'hf' on backend == 'mlx' with a distinct error message (review fix HIGH — mlx-lm only downloads from HF Hub; without this gate a backend: mlx + hub: modelscope config would silently pass schema load and fail at runtime with a confusing mlx-lm error). 26 new YAML recipes appended to soup_cli/recipes/catalog.py — every entry is exercised by tests/test_v0510.py via load_config_from_string round-trip + yaml.safe_load (no Python tags / no template injection / no credential leak in the YAML strings) + a _no_null_or_whitespace model-id check that rejects empty path components (review fix LOW — first-cut allowed "/name" leading-slash IDs to pass). Two non-<N>B size strings ("image" / "ocr" / "moe" / "medium") were normalised to "N/A" (review fix MEDIUM — search_recipes(size=…) would silently miss those entries, and the autopilot VRAM estimator could not parse them). Known limitations: (1) Live downloader / uploader / push integration deferred to v0.51.1 — TrainingConfig.hub schema lock-in ships now (Literal accept + MLX cross-validator + case-normalisation), but soup data download --hub modelscope and soup push --hub modelers still route through the existing HF Hub code path; the actual modelscope-sdk / openmind-hub adapters are the v0.51.1 deliverable. Same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus. (2) Speculative / aspirational base model IDs in some Part A/C recipes — the catalog ships entries for openai/gpt-oss-{20,120}b, THUDM/glm-5, Qwen/Qwen-Image, deepseek-ai/DeepSeek-OCR, PaddlePaddle/PaddleOCR-VL, google/embeddinggemma-300m so users have ready-made recipes the moment those repos go live (matches the plan's "match Unsloth's day-zero coverage" directive). Recipes for not-yet-published repos will surface a clear HF Hub 404 when the user runs soup train --recipe <name>. (3) DNS-resolved private hostnames not blocked — validate_hub_endpoint only rejects literal RFC1918 / link-local IP addresses; a hostname like corp-proxy.internal that DNS-resolves to a private IP is accepted at validation time (mirrors the v0.29.0 HF_ENDPOINT policy — DNS resolution is intentionally not performed in this local-tool threat model). (v0.51.0)
v0.50.0 — GRPO Plus (RL parity): 22 features across 5 Parts shipped as schema-only (closed allowlists + Pydantic validators + NotImplementedError stubs for live wiring deferred to v0.50.1). All new validators follow the project's bool-rejection-before-int policy (matches v0.30.0 Candidate); closed-allowlist validate_grpo_variant / validate_rollout_backend reject non-string / bool / empty / null-byte / oversize / unknown inputs with actionable error messages and case-insensitive normalisation. validate_grpo_delta is bool-first / math.isfinite / (0, 1] bounded (matches v0.32.0 save_lr_finder_report / v0.41.0 Part B lr_groups policy). New _VARIANT_METADATA (Part A) and _BACKEND_METADATA (Part C) are MappingProxyType-wrapped frozen-dataclass registries (matches v0.36.0 _REGISTRY / v0.41.0 _OPTIMIZER_PACKAGES policy). Security-review fixes: (1) grpo_delta schema gets an explicit field_validator(mode='after') calling math.isfinite — Pydantic's gt=0, le=1 bounds only incidentally reject NaN (since NaN > 0 is False); the explicit validator prevents a future Pydantic change from regressing the guard. (2) validate_long_context_grpo_compat adds null-byte rejection on task AND backend strings + a bool guard on use_ring_attention (parity with validate_grpo_variant / validate_rollout_backend). (3) validate_vllm_sleep_mode_compat adds null-byte rejection on backend. Code-review HIGH fixes: (4) _validate_grpo_stability_task_gate now includes grpo_fp16 in the GRPO-only-fields list — previously a user could silently set grpo_fp16: true on task='sft' and have it no-op. (5) _validate_vllm_sleep_mode now requires task='grpo' (sleep mode is a between-rollouts feature, meaningless on SFT) and rejects with a task='grpo' message. TDD-review HIGH fixes: (6) new _reject_bool_on_grpo_numerics field_validator on every Part D numeric field + grpo_delta explicitly rejects bool before Pydantic's True→1 coercion (matches v0.30.0 / v0.41.0 Part B / v0.43.0 Part B policy). Known limitations: (1) Every live loss kernel / launcher (apply_variant_loss, apply_vllm_sleep_mode, launch_rollout, build_prm_trainer) raises NotImplementedError with explicit v0.50.1 markers — same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro / v0.45.0 plugins / v0.48.0 curriculum / v0.49.0 LongLoRA. (2) long_context_grpo requires Tiled MLP (v0.56.0 Part A) to actually run; the schema gate ships now so v0.50.0 configs are stable. (3) vision_grpo=true does not check whether the base model is actually a VLM — upstream trainer surfaces that error loudly. (4) The 7 stability knobs schema-validate but none are wired into a live callback in this release; replay_buffer_size, defer_rerolling, and skip_zero_advantage are pure schema lock-ins. (v0.50.0)
v0.49.0 — Long Context & Architecture: 4 release Parts ship YaRN RoPE scaling, Dynamic NTK hardening, LongLoRA S² shifted-sparse attention (schema-only gate), and full Llama 3.1 NTK-aware scaling. Security review hardened the public boundary of soup_cli/utils/long_context.py::get_rope_scaling_config — target_length / original_length / yarn_factor now reject bool / NaN / Inf / non-positive at entry so a direct caller bypassing Pydantic cannot emit {factor: NaN} into HF model configs (matches v0.30.0 Candidate / v0.34.0 estimate_run_cost_usd / v0.41.0 Part B lr_groups policy). scale_inv_freq_llama3 rejects bool on every numeric parameter (inv_freq / scale_factor / low_freq_factor / high_freq_factor / old_context_len) — first-cut only guarded inv_freq. yarn_get_mscale now raises on non-finite input (review fix LOW — first-cut silently clamped NaN/Inf to identity, hiding the misconfig from callers). detect_llama3_rope_in_config uses explicit is None instead of the or idiom when probing rope.get("type") — falsy-but-set values no longer silently fall through to rope_type (matches v0.40.6 review-fix policy). New soup_cli/utils/longlora.py ships is_llama_model with a word-boundary regex (?:^|[^a-z0-9])(?:code)?-?llama(?:-?\d+(?:\.\d+)?)?(?:[^a-z0-9]|$) — substring "llama" inside an unrelated identifier (e.g. my-llama-style-finetune) does NOT match; null-byte rejection on model_name (matches v0.39.0 is_gemma4_model / v0.44.0 is_llama4_model policy); 512-char cap returns False rather than raising (bounded scan time). _LLAMA_REGEX has no nested quantifiers / overlapping alternation and is ReDoS-bounded by the 512-char cap. validate_longlora_compat emits distinct error messages per failure mode (mlx vs other non-transformers backends — matches v0.34.0 review-fix policy on distinct actionable rejections). TrainingConfig field_validator(mode='before') rejects bool on the four yarn fields before Pydantic's gt/le coercion silently treats True as 1.0. SoupConfig _validate_longlora_compat invokes validate_longlora_compat at config load so a misconfigured soup.yaml fails fast with an actionable message rather than silently no-opping at trainer construction time (mirrors v0.39.0 ReLoRA / v0.48.0 curriculum_dynamic schema-gate policy). Known limitations: (1) LongLoRA live LlamaAttention.forward override deferred to v0.49.1 — apply_longlora_forward_override raises NotImplementedError with the v0.49.1 marker; the schema gate ships now so misconfigured runs cannot reach trainer construction. (2) LongLoRA architecture allowlist is Llama 1/2/3.x + CodeLlama only; Mistral / Qwen / Phi expansion tracked for v0.49.1+. (3) Llama 3.1 NTK auto-detect helper detect_llama3_rope_in_config ships but is not yet wired into apply_long_context_config; trainer wiring can pick it up when needed.
v0.48.0 — Adaptive Training (BETA): 2 release Parts ship a dynamic curriculum re-weighter and a Bayesian data-mixing optimiser. New soup_cli/utils/curriculum_dynamic.py ships frozen DynamicCurriculumPolicy with bounded fields (num_buckets ∈ [1, 20], recompute_every_n_steps ∈ [1, 100_000], floor ∈ (0, 1/num_buckets], temperature > 0); bool-rejected on every numeric input (matches v0.30.0 Candidate / v0.34.0 estimate_run_cost_usd policy); math.isfinite on every float (matches v0.32.0 / v0.47.0 policy). compute_bucket_weights water-fill design (review fix HIGH — first-cut had a trailing w/sum(w) renorm that could push elements sitting exactly at the floor below the floor when accumulated float error left the sum slightly > 1.0; renorm removed because softmax already sums to 1.0 so water-fill output also sums to 1.0). validate_distributed_curriculum cross-validator rejects enabled=True with world_size > 1 unless the caller attests an all_reduce hook is registered — DDP/grad-accum footgun: divergent per-rank stats without coordination silently desynchronise the sampler. SoupConfig cross-validators reject curriculum_dynamic=true on mlx backend (HF Trainer-callback specific) and on non-SFT/pretrain tasks with distinct error messages (matches v0.34.0 review-fix policy). render_curve and parse_history_jsonl enforce _MAX_HISTORY_ROWS = 100_000 DoS cap (review fix MEDIUM — first-cut had no cap, an attacker-controlled JSONL with 10M rows would have OOM'd the process). New soup runs curriculum-curve CLI: is_under_cwd containment, os.lstat + S_ISLNK rejection (TOCTOU defence — mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E / v0.46.0 Part A policy), 50 MB file-size cap + 100k-line streaming cap (review fix HIGH — without these, --history /path/to/giant.jsonl would read the file into memory unbounded), null-byte rejection on tracker-supplied output_dir (defence-in-depth before os.path.join). New soup_cli/utils/data_mix.py ships frozen MixCandidate with simplex constraint (sum(weights) == 1.0 ± 1e-6), finite eval_loss (math.isfinite), _MAX_LOSS = 1e6 sanity cap, bool / negative rejection on every numeric. validate_datasets enforces 2-32 entries (review fix MEDIUM — first-cut had a not raw empty-only guard that fell through to the realpath loop for single-entry inputs, surfacing a less-actionable error after path resolution), is_under_cwd containment, symlink rejection, dedup, null-byte / oversize / non-string rejection. parse_budget accepts digits + optional s/m/h suffix bounded to [60s, 24h]. run_mix_optimizer per-candidate proxy exceptions are logged at DEBUG + opt.tell(_MAX_LOSS) + continue (review fix MEDIUM — first-cut raised RuntimeError on the first proxy failure, breaking the documented partial=True contract; isolation policy mirrors v0.33.0 #47 CrossDocCollator and v0.40.3 judge_filter_pairs); KeyboardInterrupt/SystemExit re-raised; NaN proxy returns logged + skipped. render_mix_recipe_yaml rejects newlines / null bytes / >4096-char dataset paths (defends against YAML key injection — mirrors v0.46.0 Part A render_recipe_yaml policy). write_mix_recipe is atomic via tempfile.mkstemp + os.replace; is_under_cwd containment + ≤4096-char path + os.lstat + S_ISLNK rejection (TOCTOU) + overwrite-required gate. load_mix_recipe os.lstat is wrapped in try/except OSError (review fix HIGH — first-cut called os.lstat bare after os.path.lexists, leaving a TOCTOU race where path disappearance between the two calls would raise an unhandled OSError to the user); 256 KB file cap; yaml.safe_load only. New soup data mix --optimize / --apply Typer command with mutually-exclusive modes. Known limitations: (1) Live HF Trainer callback for dynamic curriculum deferred to v0.48.1 — schema gates, math kernel, and visualiser ship; the callback wiring with torch.distributed.all_reduce of per-bucket stats lands in v0.48.1. (2) Multi-trainer expansion (DPO/GRPO/etc.) deferred to v0.48.1 — schema rejects non-SFT/pretrain because per-sample loss semantics differ enough that bucket-level uncertainty does not transfer cleanly. (3) Live proxy training loop for soup data mix deferred to v0.48.1 — CLI ships with a synthetic offline proxy (quadratic penalty around uniform mixture). (4) scikit-optimize integration deferred to v0.48.1 — OptimizerProtocol ducktype is the integration point; default fallback is a deterministic Dirichlet sampler. (5) Both features ship BETA:-prefixed in CLI help and field descriptions until reference-benchmark validation lands in v0.48.1.
v0.47.0 — Data Forge: 2 release Parts ship a synthetic data pipeline with full provenance + a data-quality scorecard. New soup_cli/utils/data_forge.py ships frozen ForgePlan / ProvenanceRecord / ForgeRow dataclasses + a closed VALID_TASKS = ("sft", "preference", "tool") allowlist. chunk_document paragraph splitter has _MAX_DOC_CHARS = 4 MiB cap, null-byte rejection, and bool-as-int reject on max_chunk_chars (matches v0.30.0 Candidate policy). _validate_float_unit uses math.isfinite to reject NaN AND ±Inf BEFORE the [0, 1] bounds check (review fix HIGH — first-cut accepted float("nan") because nan < 0.0 is False, silently disabling active-pruning; mirrors v0.32.0 save_lr_finder_report and v0.41.0 Part B parse_lr_groups policy). discover_documents does is_under_cwd containment (review fix MEDIUM — first-cut only enforced containment from build_forge_plan, leaving direct callers unprotected; matches v0.42.0 discover_* policy), rejects symlinked directories via os.lstat + stat.S_ISLNK, restricts to a closed _DOC_EXTENSIONS = {.txt, .md, .json, .jsonl} allowlist, caps at _MAX_DOCS = 10_000, and skips dotfiles. synthesise_forge_rows runs chunk → judge(prompt) → score_uncertainty → ForgeRow with judge-exception swallow at DEBUG (matches v0.33.0 #47 CrossDocCollator and v0.40.3 judge_filter_pairs policy — no silent crash on a single bad judge call). write_forge_dataset + write_provenance are atomic via tempfile.mkstemp + os.replace (mirrors v0.43.0 Part D copy_bundle_to); both writers call _check_write_path which enforces is_under_cwd + ≤4096-char cap + os.lstat + stat.S_ISLNK rejection at the target (TOCTOU defence — mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E / v0.46.0 policy); rejects non-ForgeRow elements (review fix MEDIUM — first-cut would TypeError mid-write leaving partial state). New soup_cli/utils/data_score.py ships BENCHMARKS MappingProxyType (6 names — mmlu / gsm8k / humaneval / truthfulqa / arc / hellaswag) and frozen ScoreReport with languages exposed as MappingProxyType to prevent caller mutation. ngram_set caps n ∈ [1, 32], text at _MAX_TEXT_CHARS = 1 MiB, rejects bool-as-int. ReDoS-hardened PII regexes — _PII_PATTERNS rewritten in the security review (review fix HIGH × 2): phone pattern flattened to remove nested (?:\+\d{1,3}[\s.-]?)?(?:$?\d{2,4}$?[\s.-]?)? (catastrophic backtracking on near-miss inputs) → flat alternation with digit-count post-filter ≥ 7; credit-card rewritten from \b(?:\d[ -]?){13,19}\b (exponential backtracking on a 13-digit-space pattern ending with x) to anchored \b\d{4}[\s\-]?\d{4}[\s\-]?\d{4}[\s\-]?\d{1,7}\b with hard digit caps; _PII_SCAN_CAP = 50_000 chars enforced on every input via detect_pii pre-truncation. _require_str rejects null bytes (review fix MEDIUM — first-cut only checked length, leaving the data_score validators inconsistent with the data_forge _validate_str). _require_unit_float uses math.isfinite to reject NaN AND ±Inf (matches data_forge policy). decontaminate_rows documents that it uses one-sided containment ratio |inter| / |b| rather than symmetric Jaccard (review fix MEDIUM — first-cut docstring said "Jaccard" but formula was containment; renamed for accuracy so future maintainers don't accidentally swap the denominator). compute_scorecard per-row try/except ValueError blocks now log at DEBUG (review fix MEDIUM — first-cut was silent pass, violating v0.33.0 #47 and v0.40.3 logging policy). load_jsonl_rows + write_jsonl_rows enforce is_under_cwd containment, _MAX_FILE_BYTES = 1 GiB cap, _MAX_ROWS = 1_000_000 row cap, os.lstat + stat.S_ISLNK rejection on both input and output, atomic write via tempfile.mkstemp + os.replace. _read_rows and _write_rows CLI helpers have full List[Mapping[str, Any]] / Iterable[Mapping[str, Any]] type annotations (review fix HIGH — first-cut had def _read_rows(path: str): with no return type, breaking downstream type checking). decontaminate_texts parameter is Optional[Mapping[str, Sequence[str]]] = None (review fix HIGH — first-cut had Mapping[...] = None # type: ignore[assignment] masking the type error). import math and import tempfile moved to module top-level (review fix MEDIUM — first-cut had stdlib imports inside function bodies, violating the project's lazy-import policy which applies only to heavy ML deps). New soup_cli/commands/data_forge.py collapses the duplicate discover_documents call (review fix MEDIUM — first-cut called the helper twice, introducing a TOCTOU window where plan.num_docs could disagree with the actual doc list). New soup_cli/commands/data_score.py --benchmarks allowlist-validated against BENCHMARKS keys with Rich-escaped error messages. Known limitations: (1) Live judge providers (Ollama / Anthropic / vLLM via --judge-provider) deferred to v0.47.1 — soup data forge ships with a deterministic offline echo stub; stub-then-live pattern matches v0.27.0 MII / v0.37.0 multipack / v0.46.0 Part A. (2) Decontamination benchmark corpora not bundled — soup data decontaminate --benchmarks mmlu validates the flag but operates on an empty corpus; operator-supplied --benchmark-file lands in v0.47.1. (3) Llama-Guard-3-1B toxicity classifier + FineWeb-Edu educational classifier + full Presidio PII + langdetect / fastText ship behind [data-pro] extras in v0.47.1. (4) Provenance manifest stores absolute realpath in source_doc (security review M4) — operators sharing manifests should redact paths; kept as-is for audit-trail completeness, mirrors v0.34.0 crash.py design tension between traceability and $HOME leak prevention.
v0.46.0 — Deploy & Agent Autopilot: 2 release Parts ship a deploy-target picker and an Agent Forge for spec-driven tool-calling SFT datasets. New soup_cli/utils/deploy_autopilot.py ships a 10-profile MappingProxyType-wrapped catalog with DeployProfile @dataclass(frozen=True) and closed allowlists on runtime (transformers / vllm / sglang / mlx / ollama / lm-studio / executorch), quant (none / 4bit / 8bit / gptq / awq / fp8 / mxfp4 / hqq:Nbit), peft (lora / dora / qlora / full). _make factory rejects non-kebab-case names, bool-as-int on recommended_max_length, out-of-bounds [64, 1_048_576], null-byte / >512-char description+notes. render_recipe_yaml rejects empty / null-byte / newline / >200-char base and >4096-char output_dir (defends against a crafted --base "evil\ntraining: { epochs: 9999 }" injecting YAML keys into the rendered recipe). render_deploy_script uses shlex.quote on model_path and rejects newline / NUL / >4096-char. write_recipe / write_deploy_script enforce is_under_cwd containment, ≤4096-char path cap, and os.lstat + stat.S_ISLNK rejection at the write target (TOCTOU defence — mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E policy). soup deploy autopilot panel passes every profile field through rich.markup.escape (matches v0.43.0 Part B Tournament policy — defends against markup injection if a future profile carries [blink] in its description). New soup_cli/utils/agent_forge.py parses OpenAPI 3.x / MCP server manifests / GraphQL introspection JSON into a canonical Endpoint frozen dataclass — every parser routes synthesised path through _validate_path (non-empty single-line NUL-free ≤1024 chars), so a manifest with name="evil\nhost" is rejected with a warning (review fix HIGH — the v0.46.0 first-cut stored the raw f"mcp://{name}" without validation, allowing newline injection into the Endpoint.path field). $ref strings in OpenAPI are left opaque (no external resolution — defends against file-read SSRF that a fully-resolving parser would expose). _MAX_ENDPOINTS=10_000, _MAX_SPEC_BYTES=5*1024*1024, _MAX_ROWS_PER_ENDPOINT=32, _MAX_DESCRIPTION=512. load_spec_file enforces is_under_cwd containment + os.lstat + stat.S_ISLNK rejection BEFORE realpath resolution (review fix MEDIUM — the v0.46.0 first-cut ordered isfile(real) before the lstat check, which followed the symlink) + 5 MiB cap + yaml.safe_load only (never yaml.load). write_dataset is atomic via tempfile.mkstemp + os.replace (review fix HIGH — replaces a v0.46.0 first-cut direct open(real, "w") loop that would leave a partial JSONL file on mid-stream TypeError; mirrors v0.43.0 Part D copy_bundle_to policy). Symlink rejection at the dataset target. endpoint_to_rows rejects bool / out-of-bounds examples_per_endpoint (∈ [1, 32]). New soup_cli/commands/agent.py: synth Rich table passes every cell through rich.markup.escape (review fix HIGH — defends against spec-controlled markup injection through ep.path); train validates --base and --output-dir for NUL / newline / >4096-char BEFORE embedding into the rendered YAML recipe string (review fix CRITICAL — defends against YAML key injection where --base $'evil\\ntraining: { epochs: 9999 }' would smuggle injected training keys); eval enforces predictions path is_under_cwd containment + os.lstat + stat.S_ISLNK rejection + _MAX_PRED_LINES=1_000_000 DoS cap (review fix HIGH — v0.46.0 first-cut had no line cap, a multi-GB predictions file would have iterated unbounded). Known limitations: (1) Live Quant-Lobotomy auto-measure deferred to v0.46.1 — autopilot writes the canonical PEFT+quant combo per profile but does not yet measure OK/MINOR/MAJOR via v0.26.0 Quant-Lobotomy Checker. (2) ExecuTorch packaging deferred to v0.54.0 — iphone-16 / pixel-9 recipes are plan-only. (3) soup agent train is plan-only — prints the planned soup train invocation rather than re-entering Typer in-process (same design as v0.44.0 soup quantize). (4) soup agent eval is heuristic — scores tool-name match + arguments-key validity only; live RLVR code_exec sandbox scoring deferred to v0.46.1. (5) $ref resolution in OpenAPI specs is intentionally not done (file-read SSRF defence); users wanting full resolution should run openapi-spec-validator upstream. (6) MCP / GraphQL non-HTTP sentinel methods (invoke / query / mutation) are stored on Endpoint.method without going through _validate_method (documented design intent — _HTTP_METHODS covers HTTP-only).
v0.45.0 — Plugin System & Ecosystem Wins: 5 release Parts ship the plugin / hook system + 4 ecosystem-integration schemas. New soup_cli/plugins/__init__.py registry: kebab-case plugin name regex ^[a-z0-9][a-z0-9\-]{0,39}$, semver-ish version regex, _MAX_PLUGINS=64, _MAX_TEMPLATES_PER_PLUGIN=32, _MAX_MODEL_GROUPS_PER_PLUGIN=32, _MAX_NAME_ENTRY_LEN=128. Re-registering with a different (version, plugin object, templates, model_groups, description) is rejected (review fix HIGH — first-cut omitted description from the conflict check, allowing silent description override). load_plugins() swallows per-plugin exceptions at WARNING level so one broken plugin cannot crash the CLI. soup plugins enable / disable raise KeyError for unknown names (caller-distinguishable from validation errors). All user-controlled output passes through rich.markup.escape. New utils/anthropic_messages.py enforces _MAX_MESSAGES=1024, per-message _MAX_CONTENT_LEN=1MiB, max_tokens cap mirrors v0.30.0 inference-server _MAX_TOKENS_CAP=16384; tool role with structured (list) content is concatenated into a single tool_result text block instead of silently dropped (review fix MEDIUM — first-cut emitted "" for any non-str content); from_anthropic uses .get("role") defensively after validate_anthropic_payload (review fix MEDIUM — KeyError surface guard). New utils/server_tools.py ships closed SUPPORTED_TOOLS = frozenset({python, bash, web_search}) allowlist + WebSearchConfig with domain_allowlist (≤64 entries, default empty = deny all) + rate_limit_per_minute ∈ [1, 600]. is_domain_allowed strips :port suffix before matching (review fix MEDIUM — without this, Host: api.example.com:443 would silently fail to match the bare api.example.com allowlist entry, causing legitimate denials) AND rejects IPv6 literals ([::1] → False — IPv6 should never match a domain allowlist). validate_domain rejects null-byte / whitespace / / / oversize. python and bash reuse the v0.25.0 RLVR sandbox (5s timeout, RLIMIT_AS/CPU on POSIX, ephemeral cwd, socket patch); live HTTP endpoints land in v0.45.1. New utils/ngram_spec.py NgramSpecConfig is @dataclass(frozen=True) with n ∈ [1, 8], num_draft_tokens ∈ [1, 32], prompt_lookup_max ∈ [0, 1MiB]; bool rejection on every numeric field (matches v0.30.0 Candidate / v0.34.0 estimate_run_cost_usd / v0.42.0 make_preprocess_cache_key policy). New utils/integrations.py 15-entry catalog wrapped in MappingProxyType; IntegrationSpec is frozen so the catalog cannot be mutated at runtime. New utils/trainer_plugins.py 6-entry allowlist (MappingProxyType); validate_trainer_plugin_list accepts Sequence[str] (review fix MEDIUM — first-cut annotated as Tuple[str, ...]) but explicitly rejects bare str argument so a caller passing "grokfast" directly doesn't iterate per-character. New utils/recipe_dag.py parses Data Recipe DAG with closed NODE_KINDS = frozenset({seed, llm_text, code, judge, validator, sampler}); Kahn's topological sort uses collections.deque + sorted-on-arrival (review fix HIGH — first-cut had O(N² log N) queue.sort() inside the BFS body); cycle / self-loop / dangling-edge / duplicate-edge / duplicate-node-name / unknown-kind rejection with specific error messages. _MAX_NODES=256, _MAX_EDGES=1024, _MAX_FILE_BYTES=1MiB. parse_recipe_yaml uses yaml.safe_load. load_recipe_yaml enforces is_under_cwd containment AND os.lstat + stat.S_ISLNK symlink rejection at the recipe path (review fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B policy; without this, a symlink under cwd pointing at /etc/passwd would resolve through realpath and load arbitrary files). RecipeDAG and RecipeNode are @dataclass(frozen=True); RecipeNode.config is MappingProxyType-wrapped so caller mutation post-parse cannot affect the validated topology. parse_recipe does NOT mutate input dicts (matches v0.33.0 #47 CrossDocCollator immutability policy). Known limitations: (1) Plugin trainer-callback wiring deferred to v0.45.1 — register_plugin accepts hook objects and discover_hooks lists implementations, but no HF Trainer callback emits pre_train / post_train / pre_step / post_step yet. (2) Anthropic /v1/messages endpoint deferred to v0.45.1. (3) Server-side tool HTTP endpoints (python / bash / web_search) deferred to v0.45.1. (4) n-gram speculative-decoding live engine wiring deferred to v0.45.1. (5) External integrations are a catalog, not launchers — IntegrationSpec describes 15 ecosystem targets but no soup deploy <target> command auto-detects + invokes their CLIs (tracked for v0.46.0 Deploy Autopilot). (6) Advanced trainer plugins are an allowlist, not callbacks — validate_trainer_plugin_list accepts [grokfast, spectrum, ...] at schema validation, but no trainer wraps the upstream callbacks yet (live wiring per plugin in v0.45.1). (7) Data Recipe DAG runner deferred — soup data recipe validates topology and prints planned topological order; node-by-node execution against a local model is the v0.45.1 deliverable.
v0.44.0 — Live Dashboard & UX: 21 features across 4 Parts. New soup_cli/utils/qr_url.py build_phone_url enforces a scheme allowlist (http / https), loopback-only HTTP (RFC1918 / link-local / cloud-metadata IPs implicitly rejected because host not in _LOOPBACK_HOSTS raises), IPv6 literal auto-bracketing per RFC 3986, and token routing through the URL query string ?token=… (review fix HIGH — the v0.44.0 first-cut put it in the URL fragment #token=… which never reaches the server). Token shape regex ^[A-Za-z0-9_\-]{16,128}$ matches secrets.token_urlsafe(N) output. New commands/fetch.py enforces is_under_cwd on --output AND adds os.lstat + S_ISLNK symlink rejection at the write target regardless of --force (review fix HIGH — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 / v0.43.0 Part C policy). Bundled-source path is os.path.commonpath-checked against fetch_examples_dir() (review fix HIGH — defends against a future catalog entry containing .. that would escape the bundled directory). fetch_examples_dir() uses os.path.realpath not abspath (review fix MED — symlinked installs resolve to the real package root, not the symlink's parent). New utils/checkpoint_trigger.py write_trigger adds os.lstat + S_ISLNK rejection on the trigger target (review fix MED — symlink at <output_dir>/.checkpoint_now could redirect the write to any user-writable path). Trigger path stays under cwd via shared is_under_cwd. New commands/llama.py proxies to llama.cpp binaries with _LLAMA_ENV_ALLOWLIST frozenset child-env filter (review fix MED) — HF_TOKEN / OPENAI_API_KEY / ANTHROPIC_API_KEY / all SOUP_* are dropped before exec; only PATH / HOME / USER / USERPROFILE / TMP / TEMP / locale + llama.cpp-recognised LLAMA_CPP_HOME / GGML_* / OMP_NUM_THREADS are forwarded. Subprocess uses list args (no shell). Closed allowlist _SUBCOMMAND_TO_BINARY: MappingProxyType rejects unknown subcommands; _validate_arg rejects null-byte / newline / carriage-return / >1024-char per arg; _MAX_ARGS=64 cap. New utils/onboarding.py render_onboarding_yaml enforces is_under_cwd on the output field at render time (review fix MED — without this, a crafted answer dict like output: ../../etc/cron.d/x would silently propagate into the rendered YAML). Literal allowlist on task (8 entries) and quantization (4bit / 8bit / none); epochs ∈ [1, 10]; null-byte rejection on every string with field-named error message. New utils/sweep_config.py enforces a strict scalar allowlist (str / int / float / bool) on every param value (review fix LOW — without this, params: {lr: [{__class__: evil}]} would smuggle non-scalar YAML through to runtime); _MAX_FILE_BYTES=256KB, _MAX_PARAM_KEYS=32, _MAX_VALUES_PER_KEY=64, _MAX_PARAM_KEY_LEN=128. SweepSpec.params is MappingProxyType[str, Tuple[Any, ...]] for genuine frozen=True immutability (review fix HIGH — frozen=True on a Dict[str, List[Any]] field only prevents reassignment, not mutation). Same fix applied to ConsolidationPlan.shard_files: Tuple[str, ...] and DelinearizePlan.weight_files: Tuple[str, ...]. New utils/gpu_monitor.py parse_nvidia_smi_csv rejects null-byte in GPU name field; subprocess uses list args + 5s timeout + OSError/TimeoutExpired swallowed (never raises). detect_apple_silicon rewritten to use clean platform.system() / platform.machine() checks (review fix HIGH — the v0.44.0 first-cut had a parser-priority bug if X if Y else Z: that produced the right answer by coincidence on every platform but would silently break on any future branch refactor). New utils/sse_train_stream.py TrainEvent validates type against a closed _VALID_TYPES = frozenset({metric, status, log, eval}), rejects non-finite ts, rejects null-byte / >1024-char message. to_payload filters output to a closed _ALLOWED_KEYS allowlist (defence-in-depth against payload key drift from refactors). format_sse_frame emits the standard W3C SSE wire format data: {json}\n\n with ensure_ascii=False. New utils/tool_outputs.py ToolOutputsBuffer is thread-safe (threading.Lock) backed by collections.deque(maxlen=_MAX_RECORDS=1000) (review fix MED — replaces an O(N) self.records[-N:] list slice that triggered GC pressure on overflow); validates name (no null-byte, ≤128 chars), bool rejection on started_ts / duration_ms (matches project bool-as-int policy), success must be exactly bool not int, output preview truncated to 4096 chars. ToolCallTimer.__exit__ returns False explicitly per project readability policy (does not suppress exceptions). New utils/llama_server_timings.py parse_timings tolerates missing keys (defaults to None), clamps kv_cache_pct to [0, 100], rejects bool / non-finite / negative ints in numeric coercion. format_kv_bar validates width ∈ (0, 200]. New utils/tail_latency.py enforces MAX_SAMPLES=1_000_000 DoS cap on every iteration; bool rejection + non-finite rejection + alpha bounds. New soup_cli/ui/plugins/__init__.py registry: tab name regex ^[a-z0-9][a-z0-9\-]{0,30}$, _MAX_TABS=32 cap, list_tabs() returns MappingProxyType defensive view, load_plugins() swallows per-plugin exceptions at WARNING level so one broken plugin doesn't crash the whole UI. _NAME_DISALLOWED in utils/shortcuts.py is frozenset (review fix LOW — was mutable set at module level). New utils/ui_env.py host validator restricts to [a-zA-Z0-9.-:] ≤253 chars + null-byte rejection; port [1, 65535]; key length cap 256 + null-byte rejection. Known limitations: (1) Live monitoring SSE endpoint deferred — schema + frame formatter ship, FastAPI route in v0.44.1. (2) Phone visibility flag wiring (soup ui --public --auth-token + QR print at startup) deferred to v0.44.1. (3) soup merge-sharded-fsdp-weights and soup delinearize-llama4 are plan-only — torch-side runtime in v0.44.1. (4) soup llama <subcommand> does not bundle llama.cpp; the binary must be on PATH (shutil.which) and an actionable FileNotFoundError fires when missing. (5) soup serve --reasoning-parser <name> allowlist ships, runtime hook in v0.44.1. (6) Apple Silicon soup monitor prints a yellow advisory and exits with nvidia-smi unavailable rc=1 — powermetrics parser in v0.44.1.
v0.43.0 — Tracker & Eval Pro: 18 features across 4 Parts. New soup_cli/utils/trackers.py ships an immutable _REPORT_TO_BACKENDS MappingProxyType allowlist (wandb / tensorboard / mlflow / swanlab / trackio / none); validate_tracker_name rejects non-string / empty / null-byte / >32-char inputs. PostHog telemetry is opt-IN (default OFF) via SOUP_TELEMETRY=1 env var; build_telemetry_payload schema is closed-key (soup_version / command / python major.minor / os / arch / duration) — no model names, dataset paths, or config contents leak in the payload. Live PostHog network code deferred to v0.43.1. New --tracker flag on soup train mutually exclusive with --wandb / --tensorboard via resolve_report_to. tracker_backend = tracker capture in commands/train.py defends against the Typer parameter being shadowed by the local ExperimentTracker() instance (review fix HIGH #1 — silent --tracker drop regression). nlg_metrics.py is pure-math: BLEU _lcs_length allocates a fresh row per outer iter (review fix HIGH #2 — first-cut had a double-buffer re-zero that destroyed the last row). effective_tokens_per_second returns None on non-positive wall_clock (no fabrication). eval/calibrate.py CalibrationReport is @dataclass(frozen=True) with FrozenInstanceError test (matches v0.32.0 / v0.39.0 / v0.41.0 frozen-dataclass policy); run_calibration caps prompts at 10_000 (DoS defence). eval/arena.py Tournament enforces 256-model cap, 1M-match cap, MappingProxyType view on ratings so external callers cannot mutate Elo state (LOW review fix). _validate_model_name rejects null-byte / >128-char AND Rich markup metacharacters [ / ] (security review fix LOW #5 — leaderboard markup-injection defence). update_elo rejects bool / non-finite / NaN k. eval/benchmarks_v0_43.py ships NEW_BENCHMARKS_V0_43 = frozenset({"ceval","cmmlu","aider_polyglot"}) + MappingProxyType-wrapped metadata; live Aider Polyglot runner deferred to v0.43.1. utils/profiling_v0_43.py resolve_snapshot_path rejects null-byte / . / .. / path separators in run_id; base_dir rejects null-byte / absolute / .. segments before realpath join (security review fix HIGH #1) — closes a Windows short-name path escape. memory_snapshot_context narrows the RuntimeError catch to the record() entry call only (review fix HIGH #2 — wide catch would have triggered generator already executing on user-body RuntimeError); yields None when torch / CUDA / _record_memory_history API missing. nccl_bandwidth_check rejects bool / non-finite / negative measured_gb_per_sec; reference table is MappingProxyType-wrapped. utils/vscode_setup.py build_launch_json rejects null-byte / newline / >512-char config_path so a crafted argument cannot inject Python args into the generated JSON. write_vscode_launch uses os.lstat + S_ISLNK symlink rejection at the target path regardless of force (security review fix MEDIUM #3 — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 policy). utils/demo_bundles.py copy_bundle_to validates every JSONL line via json.loads, enforces 50 MB cap, stages writes to <target>.tmp with os.lstat + S_ISLNK rejection (security review fix HIGH #2) and os.replace-atomic rename on success — mid-stream rejection never leaves a partial file. New soup data demo Typer command: --output containment-checked via shared is_under_cwd. CLI output prints _esc(written) to defend against Rich markup injection through a crafted output path (review fix LOW). _lcs_length algorithmic correctness fix (review fix HIGH #2) + BLEU geometric-mean policy fix (returns 0 on any zero precision unless smooth=True) (review fix HIGH #1). All math imports moved to module-level (PEP 8 / project lazy-import policy: only torch / transformers / peft / trl / mlx are lazy). Known limitations: (1) MLflow / SwanLab / Trackio live integration deferred to v0.43.1 — schema accepts the names and threads into HF TrainingArguments.report_to but Soup does not bundle the upstream packages; HF Trainer raises a generic ImportError when the package is absent. (2) PostHog telemetry deferred — payload schema + SOUP_TELEMETRY=1 env var ship, but no network code; build_telemetry_payload is purely a schema-locking stub (mirrors v0.27.0 MII / v0.37.0 multipack / v0.42.0 fsspec stub-then-live pattern). (3) Aider Polyglot live runner deferred — aider_polyglot is in the benchmark allowlist but live wiring requires the upstream aider-chat package + Docker. (4) soup doctor --nccl measurement CLI surface deferred — nccl_bandwidth_check accepts a measured value and classifies it, but does not yet measure via torch.distributed.all_reduce. (5) examples/data/ fixtures shipped at the repo root (not as package data inside soup_cli/); _bundle_source_path derives repo_root via str(importlib.resources.files("soup_cli")) which is editable-install + wheel friendly but may break in zipapp / namespace-package installs.
v0.42.0 — Data Pipeline Pro: closes the data-tooling gap with Axolotl + LlamaFactory in 18 features across 6 Parts. New soup_cli/utils/data_pipeline.py ships an immutable _REMOTE_SCHEMES MappingProxyType allowlist (s3 / gs / gcs / az / abfs / abfss / oci); validate_remote_uri rejects userinfo, fragments, AND query strings (the latter is SSRF-adjacent — fsspec backends interpret ?endpoint_url=… as configuration overrides; a crafted s3://bucket/x?endpoint_url=https://attacker.com would bypass otherwise-trusted infrastructure). Bucket regex ^[a-zA-Z0-9][a-zA-Z0-9._\-]{0,62}$ (1-63 chars, leading alnum) matches RFC 3986 + S3/GCS naming. Path length cap 2048; null-byte rejection on every string-shaped input; bool rejected before int isinstance check on every numeric input (buffer_size / shards / image_pixels / video_maxlen / num_datasets). parse_interleave returns frozen InterleaveSpec dataclass; per-prob math.isfinite (rejects NaN AND ±inf, matches v0.32.0 / v0.41.0 policy); sum-to-1 ±1e-6; max 32 datasets. validate_new_tokens caps lists at 10_000 entries with no duplicates, per-token <= 256 chars, null-byte rejected; returns a defensive copy. validate_prompt_strategy enforces a module.path:function_name regex with per-side 128 char cap (total <= 260). make_preprocess_cache_key rejects bool / null-byte / non-string / non-positive on every parameter; uses \x1f unit-separator delimiter. detect_ingest_format rejects null-byte / empty / unsupported extensions. New schema validators on DataConfig: video_dir / tokenized_path use shared is_under_cwd containment so a crafted YAML like data.video_dir: ../../etc fails at config load (review-fix M1; distinct from image_dir / audio_dir legacy policy). train_on_prompt mutually exclusive with train_on_responses_only (matches v0.36.0 loss-mask exclusivity policy). interleave field has its own field_validator so data.interleave: 99 fails at config load (review-fix L3) AND bare interleave: probs is rejected at schema load with the actionable "use {strategy: probs, probs: [...]} dict form" message instead of getting a confusing late error (review-fix HIGH #2). Image-pixel validator threads info.field_name so error messages name the actual field that tripped (review-fix M3). Bucket regex relaxed from {1,62} to {0,62} so 1-character bucket names (per S3/GCS spec) are no longer falsely rejected (review-fix HIGH #1). _convert_video validates video is non-empty string with null-byte rejection and 2048-char cap (review-fix H2). _convert_prm type-checks prompt (non-empty string) + every completions[i] (string) + every labels[i] (bool, not int — matches v0.30.0 Candidate policy) and caps at 10_000 steps. _convert_multimodal validates content-part type against the closed allowlist {text, image, audio, video}. New soup data preprocess <config> CLI: --config containment-checked via is_under_cwd BEFORE load_config (review-fix HIGH #1) so a crafted argument like ../../etc/passwd.yaml cannot be opened. New soup data ingest <file> CLI: os.lstat + stat.S_ISLNK symlink rejection on the input file (TOCTOU defence, mirrors v0.33.0 #22 prune_checkpoints policy); is_under_cwd containment on input + output. Known limitations: (1) fsspec live loaders deferred to v0.42.1 (schema gate fires now, runtime ImportError advisory); (2) AOT preprocess live tokenize loop deferred to v0.42.1 (cache-key emit + path-plan only); (3) custom prompt-strategy runtime invocation deferred to v0.42.1 (regex shape validation only); (4) PRM live trainer wiring deferred to v0.50 per original roadmap; (5) markdown ingest does not split on headings (the docstring's heading-split claim was aspirational); (6) _convert_video / _convert_multimodal validate path/url null-byte and length but do NOT verify file existence (matches v0.40.5 reward_model policy — file-existence is the trainer's responsibility).
v0.41.0 — Optimizer & PEFT Zoo: closes the optimizer-breadth gap with LlamaFactory + Axolotl. New soup_cli/utils/optimizer_zoo.py ships a closed SUPPORTED_OPTIMIZERS frozenset (HF-native + bnb-backed + 14 v0.41.0 additions: BAdam / APOLLO / Adam-mini / lomo / adalomo / grokadamw / schedule_free_adamw / schedule_free_sgd / muon / dion / came_pytorch / ao_adamw_{fp8,4bit,8bit}). validate_optimizer_name rejects non-string / empty / null-byte / >64-char inputs; lower-cases the name for deterministic lookup (matches v0.30.0 pick_draft_model policy). _OPTIMIZER_PACKAGES wrapped in types.MappingProxyType so the registry cannot be mutated at runtime (matches v0.36.0 _REGISTRY policy). is_new_v0_41_optimizer is non-string-safe (returns False rather than raising). New soup_cli/utils/lr_groups.py parses training.lr_groups accepting list-of-pairs / list-of-dicts / {pattern: lr} mapping; capped at MAX_LR_GROUPS=32; per-pattern non-empty string ≤256 chars + null-byte rejection + re.compile validation + best-effort ReDoS probe (compiled.search("a"*128) catches catastrophic-backtracking patterns); per-LR (0.0, 1.0] bounds + math.isfinite (rejects NaN AND ±inf) + bool rejection (matches v0.30.0 Candidate policy). Duplicate patterns rejected. LrGroup is @dataclass(frozen=True) (matches v0.32.0 SpikeRecoveryStrategy policy). lr_groups_from_schema converts the canonical stored shape List[Dict] into runtime List[LrGroup] (closes the schema-to-runtime type gap that the code review caught). build_optimizer_param_groups rejects bool / non-positive base_lr at runtime (defence-in-depth). New soup_cli/utils/loftq_init.py exposes validate_loftq_iter (∈ [1, 10], bool rejected) and validate_loftq_bits (∈ {2, 4, 8}, bool rejected) with build_loftq_config lazy-importing peft.LoftQConfig (ImportError carries actionable pip install --upgrade peft hint). New soup_cli/utils/block_expansion.py validators reject bool on expand_layers / freeze_trainable_layers and use field_validator(mode="before") so Pydantic's ge/le does not silently coerce True to 1 (matches v0.30.0 / v0.34.0 / v0.36.0 / v0.40.6 bool-as-int hardening policy). _count_layers uses hasattr(layers, "__len__") instead of try/except TypeError so legitimate __len__ bugs surface loudly. Schema LoraConfig.init_strategy Literal extended to {"random", "pissa", "olora", "loftq"}; cross-validator rejects loftq + use_dora / use_vera. TrainingConfig.load_in_8bit / load_in_16bit use is True policy (matches v0.34.0 / v0.39.0 / v0.40.6 is None over falsy guards) — explicit False is treated as "no preference" not as "off"; mutually-exclusive both-True rejected; combining alias=True with explicit Quant Menu format raises rather than silently overriding. The alias-driven quantization rewrite uses direct self.quantization = ... assignment (NOT object.__setattr__ — code review caught that the latter would silently bypass any future field validator on quantization). expand_model_blocks raises NotImplementedError with a v0.41.1 marker on non-zero block counts (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA stub-then-live pattern). Known limitations: (1) LLaMA Pro live wiring deferred to v0.41.1; (2) Mixture-of-Depths (use_mod=True) live patch deferred to v0.41.1; (3) optimizer dependency check is advisory at trainer construction time, not at schema-load (CI environments often lack optional optimizer packages); (4) load_in_8bit/load_in_16bit rewrite quantization only when set to True — explicit False is intentionally a no-op so a YAML with load_in_8bit: false and quantization: 4bit still trains in 4-bit.
v0.40.6 — ReLoRA + surgical PEFT non-SFT: closes the v0.39.0 known gap by extending the ReLoRA callback (v0.39.0 Part B) and the surgical PEFT patches (v0.39.0 Part D — Gemma4 ClippableLinear -> nn.Linear swap, fused-MoE 3-D expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO). New shared module soup_cli/utils/peft_wiring.py exposes three helpers: apply_pre_lora_patches(model, base) (Gemma4-gated, runs BEFORE get_peft_model so PEFT's target_modules matcher sees the swapped nn.Linear), apply_post_lora_patches(model) (3-D MoE expert dropout strip, runs AFTER LoRA injection — architecture-detected via weight.ndim == 3 inside the helper, safe to call unconditionally), attach_relora_callback(trainer, tcfg) (returns True/False; uses if relora_steps is None: return False per project policy so a schema-bypassing caller passing relora_steps=0 surfaces as a loud ReLoRAPolicy ValueError rather than a silent skip). SFT migrates to the same helpers in the same release (centralisation invariant): every trainer file calls only the helpers, eliminating the v0.39.0 inline copy in sft.py. SoupConfig._validate_relora_supported_tasks removes the task != "sft" rejection branch; MLX backend rejection retained with a distinct error message (callback is HF Trainer-specific). Source-level grep matrix in tests/test_v0406_part_a.py proves all 12 transformer-backend trainers (sft + 11 non-SFT) call apply_pre_lora_patches BEFORE get_peft_model BEFORE apply_post_lora_patches, plus behavioural unit tests for each helper (Gemma4 happy path + exception swallow, post-LoRA strip happy path + exception swallow, ReLoRA policy field forwarding, schema-gate matrix covering every transformer task plus the preference dispatcher with preference_loss='dpo'). Defence-in-depth carry-over: peft_wiring swallows broad Exception from each upstream patch at DEBUG level (matches v0.39.0 Part D best-effort design); %s formatting on exc (not repr) so $HOME-prefixed paths cannot leak (matches v0.34.0 crash.py redaction policy); the underlying apply_gemma4_clippable_patch and strip_lora_dropout_for_3d_experts already validate model_name (null bytes, length) and are duck-typed via v0.39.0 review fixes. Known limitations: (1) Multi-modal trainers (vision/audio paths in sft.py) inherit ReLoRA + surgical patches because they share the SFT trainer wrapper, but the surgical patches are best-effort (try/except DEBUG-logged) — a Gemma4 vision model is unlikely in practice; if encountered, the patch attempt may noisy-log without applying. (2) The schema gate now accepts every transformer-backend task with relora_steps, but real-world correctness on RLHF (PPO / RewardModel) is unverified — ReLoRA was originally validated on SFT/causal-LM training; rejection-sampling-style RL loops may interact unexpectedly with periodic LoRA pruning + optimizer reset. Tracked as a community QA item; the schema does not gate on this since the upstream paper does not preclude RL use. (3) apply_post_lora_patches swallows exceptions at DEBUG, consistent with the v0.39.0 best-effort design — a real PEFT-side breakage in strip_lora_dropout_for_3d_experts would silently no-op on a non-MoE model where the strip is also a no-op, so the silent fallback is acceptable.
v0.40.5 — Quant Menu non-SFT: closes the v0.38.0 known gap by extending the seven Quant Menu formats (gptq / awq / hqq:Nbit / aqlm / eetq / mxfp4 / fp8) from SFT-only to all 11 transformer-backend trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO). SoupConfig._validate_quant_menu_supported_tasks removes the task != "sft" rejection branch; MLX backend rejection retained with distinct message; modality != "text" rejection retained for vision/audio (multi-modal Quant Menu is tracked for a follow-up). Each non-SFT _setup_transformers replaces its inline BNB-only branch with a call to build_quantization_config_for_loader(tcfg=tcfg, base=cfg.base, console=console) — same pattern as sft.py:420-440, no remaining BitsAndBytesConfig(load_in_4bit=True ...) literal in any non-SFT trainer (source-level invariant test in tests/test_v0405_part_a.py). The kbit-prep tuple is widened from ("4bit", "8bit") to ("4bit", "8bit", "mxfp4") so the BNB MXFP4 path runs through prepare_model_for_kbit_training. _load_reward_model (module-level helper in ppo.py) accepts an optional tcfg=None kwarg — when set, the reward model is loaded with the same Quant Menu config as the policy, defending against silent fp16 OOM on a GPTQ/AWQ/HQQ policy run. PPO call sites at _create_reward_model + _setup_reward both forward tcfg=tcfg. Defence-in-depth: new TrainingConfig.reward_model field validator rejects null bytes and caps length at 512 chars at config-load (matches the policy applied to cfg.base); the Quant Menu loader's per-call null-byte check in _check_local_marker remains as the runtime backstop. Known limitations: (1) vision/audio modality + Quant Menu still rejected by the modality gate — _setup_vision_transformers / _setup_audio_transformers retain inline BitsAndBytesConfig blocks because they need vision-specific kwargs the unified loader does not yet thread; (2) Autopilot's quantization picker still recommends only 4bit/8bit/none — Quant Menu format awareness deferred; (3) tcfg.reward_model is null-byte and length-validated at schema load but not path-containment-checked (is_under_cwd) — consistent with how cfg.base is treated, both can be HF repo IDs or absolute local paths.
v0.40.4 — trust_remote_code multi-trainer + multipack live: closes the v0.36.0 #63 known gap by extending the --trust-remote-code opt-in across every non-SFT trainer wrapper (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO + the unified PreferenceTrainerWrapper dispatcher) and the 5 standalone commands (soup diff, soup export, soup merge, soup infer, soup data generate). Pattern (15 sites): each __init__ resolves once via model_requires_trust_remote_code(config.base) or False + resolve_trust_remote_code(...) and stores self._trust_remote_code — every from_pretrained call site now reads the resolved value (no remaining trust_remote_code=True literal in any trainer file; source-level invariant test in tests/test_v0404_part_a.py). commands/train.py no longer carries the v0.36.0 sft_kwargs split; trust_remote_code is part of the unified trainer_kwargs dict that flows to every trainer regardless of task. _load_reward_model (module-level helper in ppo.py) accepts a trust_remote_code: bool parameter and resolves internally — design intent is that the helper is independently safe to call from outside PPOTrainerWrapper. PreferenceTrainerWrapper dispatcher forwards the raw bool to the inner DPO/SimPO/ORPO/IPO/BCO wrapper kwargs at both _build_inner and _build_multi_objective sites; the resolver fires inside the inner wrapper at construction time. _export_onnx / _export_tensorrt / _export_awq / _export_gptq and _merge_adapter helpers all gain a trust_remote_code: bool = False parameter threaded from the Typer flag. Multipack live HF Trainer wiring (#65) lands via a new get_train_dataloader override on make_multipack_trainer_class that installs MultipackBatchSampler(real_batches=False) (yields flat list[int] per pack — DataLoader-compatible) as the DataLoader's batch_sampler=. The override forwards args.dataloader_drop_last / dataloader_num_workers / dataloader_pin_memory from TrainingArguments. _get_train_sampler override stays as a defensive no-op fallback that ALWAYS delegates to super (review-fix: a multipack list[list[int]] from this method would cause a shape mismatch if any HF eval / prediction loop bypasses get_train_dataloader). The state-presence guard switched from falsy (not max_seq) to explicit is None + not lengths (defensively rejects only-None and empty-list cases — non-positive ints already rejected upstream by attach_multipack_state). Falls back to super().get_train_dataloader() when state is missing OR when train_dataset is unset (defence-in-depth so the subclass remains safe to instantiate). Known limitations: (1) multipack: true requires the dataset to expose input_ids (preferred) or length per row — un-tokenized text-only datasets trigger the v0.40.3 all-zeros WARNING and the MultipackBatchSampler will reject the run. (2) The DataLoader override does NOT thread FSDP / DeepSpeed parallelism env hints from super().get_train_dataloader(), so distributed multipack: true runs are still untested under FSDP / ZeRO; tracked for v0.40.5+ paired with v0.42.0 multi-GPU work. (3) _live_lr_sweep_from_config in commands/train.py still hardcodes trust_remote_code=False for the LR sweep's internal model load — defensive but means --find-lr cannot consume custom-code models even with the user opt-in (defence-in-depth, not a bypass). (4) Each non-SFT trainer's __init__ repeats the resolver block (10 sites) — code-quality refactor candidate (single shared helper) deferred to a future patch to keep the v0.40.4 diff focused on the gap closure.
v0.40.3 — Stub-to-live: New soup_cli/utils/batch_probe.py:make_cuda_probe_fn builds a CUDA probe closure that runs ONE forward+backward+step on a synthetic batch per candidate; model.zero_grad(set_to_none=True) runs BEFORE forward (defends against the synthetic backward accumulating into the live training model's grad buffers — matches v0.35.0 #45 benchmark_kernel_combos policy); intermediate ids/attn/labels/outputs are del-ed before loss.backward() so peak VRAM reflects a realistic training step; bool rejected on batch_size and max_length; max_length < 8 rejected; torch.cuda.OutOfMemoryError returns False, other exceptions propagate; returns None (no-op) on non-CUDA / no-torch / missing model or tokenizer. New soup_cli/utils/multipack_trainer.py:make_multipack_trainer_class is lru_cached so two calls with the same base_cls return the same subclass (consistent isinstance, pickle-safe); attach_multipack_state rejects bool on max_seq_len/batch_size/seed and rejects empty lengths; lengths_from_dataset logs WARNING when every row produces 0 (loud-fail mirrors v0.37.0 multipack arch allowlist — prevents silent NaN-loss footgun); _get_train_sampler override accepts *args, **kwargs for HF >= 4.41 signature compat. Live wiring of the sampler into SFT / Pretrain trainer wrappers is deferred to v0.40.4 — adversarial 5th-pass review surfaced a Sampler[int] vs list[list[int]] shape mismatch with HF Trainer's DataLoader; the wrappers currently print a yellow advisory and fall back to the standard sampler when multipack: true. New soup_cli/data/traces/quality.py:judge_filter_pairs reuses v0.19.0 JudgeEvaluator SSRF protections; threshold rejects bool / NaN / out-of-[0,1]; _MAX_BATCH=100_000 cap applied via lazy itertools.islice (never fully materialises a malicious / pathological generator); per-pair backend exceptions caught and logged at DEBUG (matches v0.33.0 #47 CrossDocCollator policy — never silently crash the harvest); judge_provider validated against VALID_PROVIDERS allowlist at the CLI boundary BEFORE constructor, with a Rich-escape error on mismatch. New soup_cli/monitoring/trace_logger.py:TraceLogWriter is thread-safe (single threading.Lock — multi-worker --workers 4 documented as a single-process limitation); path containment via shared is_under_cwd; null-byte / empty / non-string path rejected; cap_mb bounds [1, 10000] with explicit bool rejection; rotation: when current + extra > cap_bytes, rename to <path>.1 (one backup retained); symlink at the backup path is rejected via os.lstat + stat.S_ISLNK (matches v0.33.0 #22 TOCTOU policy) — defends against pre-placed <log>.1 -> /etc/cron.d/x overwrite. Secret redaction: prompt + response strings passed through _SECRET_RE matching hf_* (≥8), sk-* (≥16), and Bearer … (≥8) — replaces matches with <redacted> before serialisation (mirrors v0.34.0 crash.py policy). --trace-log constructor error messages in commands/serve.py are rich.markup.escaped before printing so a crafted path name cannot inject Rich markup. Unserialisable entries dropped silently; disk-full / OSError on write never crashes the request handler (passive log). Known limitations: live CUDA probe is wired in SFT only; multipack live wiring covers SFT+Pretrain only; TraceLogWriter retains exactly ONE backup file (operators wanting longer retention should use external rotation); custom HF Space templates from v0.40.2 still always create the Space with space_sdk="gradio" (tracked for v0.40.4+).
v0.40.2 — Quick polish + carry-overs: New soup_cli/utils/hf_space.py:render_custom_template_dir enforces is_under_cwd containment on the template directory; validate_repo_id runs BEFORE {MODEL_REPO} substitution (matches v0.29.0 Part F policy); per-file 256 KB cap (matches v0.39.0 Part E template-size policy); only app.py / README.md / requirements.txt are read (closed allowlist — no path-from-user-data). Symlinks rejected via os.lstat + stat.S_ISLNK and non-regular files (FIFO / device) also rejected (matches v0.33.0 #22 prune_checkpoints TOCTOU policy) — defends against <template_dir>/app.py -> /etc/passwd. _find_highest_local_checkpoint reads output_dir after caller's is_under_cwd validation (in prepare_hf_resume) and silently drops non-directories + OSError. prepare_hf_resume skips the snapshot download when local checkpoint-N >= remote checkpoint-N (saves bandwidth and never overwrites a fresher local checkpoint). commands/data.py:register_data containment switched from Path.resolve() + relative_to() to shared is_under_cwd (Windows 8.3 short-name safety per CLAUDE.md project rule); same fix applied to commands/bench.py prompts-file containment. commands/infer.py:--output now containment-checked via is_under_cwd (late-evaluated after model+input validation so pre-existing tmp_path test contracts keep working). commands/quickstart.py:--output validates target dir via is_under_cwd before mkdir(parents=True); rejects out-of-cwd targets with friendly message. commands/runs.py:_filter_runs_by_cwd uses os.path.realpath + commonpath, catches (ValueError, OSError) so cross-drive paths on Windows (D:\runs vs C:\project) drop silently rather than crash. monitoring/display.py:format_gate_row uses explicit task.get("passed") is True so a missing "passed" field renders neutrally instead of as a false-y red ✗. commands/infer.py:_resolve_model_source heuristic for HF-id-vs-local-path: only falls through to HF when value is NOT path-like (no ./, /, \\, ~, no Windows drive letter, non-empty); path-like-but-missing raises FileNotFoundError so users see actionable errors instead of confusing HF download attempts. Known limitation: custom HF Space templates always create the Space with space_sdk="gradio" regardless of the supplied app.py (no --sdk flag in this release; combine --template streamlit-chat with the inline registry for Streamlit Spaces). Tracked for v0.40.3+.
v0.40.1 — QA Hardening: soup_cli/utils/encoding.force_utf8_stdio reconfigures Windows stdout/stderr to UTF-8 before any Rich Console is constructed; os.environ.setdefault("PYTHONIOENCODING", "utf-8") preserves user override; (OSError, ValueError, AttributeError) swallowed on detached streams; POSIX no-op. SoupConfig._remap_root_level_misplaced_keys (model_validator, mode='before') migrates root-level lora: into training.lora so nested validators (including lora.init_strategy: Literal["random","pissa","olora"]) actually fire — closes a footgun where the misplaced key was silently dropped. Caller's dict is never mutated (shallow-copy policy mirroring v0.33.0 #47 / v0.40.0 Part B). PreferenceTrainerWrapper._build_multi_objective replaces the v0.40.0 NotImplementedError stub with a primary-loss approximation; validate_weight_compat rejects BCO mixed with paired losses at runtime (data-format incompatible). combine_losses rejects empty weights, propagates NaN loudly (no silent zeroing), and rejects bool weight values (matches v0.30.0 Candidate policy). _probe_cache_param_count rejects empty / null-byte model names before path construction (mirrors v0.26.0 registry / v0.39.0 ReLoRAPolicy policy). commands/doctor flags transformers ≥ 5.0.0 as INCOMPATIBLE via _MAX_EXCLUSIVE table; _version_ge parses leading-int chunks so 5.0.0.dev0 correctly trips the cap. _detect_gpu_hw_without_torch_cuda calls nvidia-smi via argv list (no shell), 5s timeout, OSError / TimeoutExpired caught; GPU label from nvidia-smi stdout is rich.markup.escaped before embedding in Rich-markup string (a real GPU name like NVIDIA Quadro [T4] cannot break or inject markup). _detect_dual_python_interpreters uses os.path.realpath (not Path.resolve()) for Windows 8.3 short-name compat. _pick_quickstart_model swaps TinyLlama-1.1B → SmolLM2-135M when total_memory ≤ 6 GB (prevents step-0 OOM on RTX 3050 4 GB / similar). _live_lr_sweep_from_config switched broken load_local import to load_raw_data (previously always silently fell back to a static placeholder curve). commands/migrate rejects .jsonl input (with first-line { sniff) with exit-2 friendly error; .jsonl-only suffix gate prevents false-positives on .ipynb notebooks. commands/eval custom -o is now honored independently of --attach-to-registry; loop-shadow regression where output = generate_fn(...) overwrote the CLI option fixed (variable renamed to response). _load_jsonl switched from utf-8 to utf-8-sig so PowerShell Out-File -Encoding utf8-produced JSONL no longer fails first-row parse. Known limitation: --trust-remote-code opt-in surface still excludes 10 non-SFT trainers + 5 commands (v0.36.0 #63 carry-over).
v0.40.0 — Preference Variety: New task='bco' (Binary Classifier Optimization) and task='preference' (unified dispatcher). New schema fields: bco_beta (gt=0), preference_loss: Literal[dpo,simpo,orpo,ipo,bco]|None, preference_loss_weights: Optional[Dict[str,float]], dpo_beta_schedule: Literal[linear,cosine,exponential]|None, dpo_beta_end: float, gt=0|None, dpo_ref_regen_epochs: int [1,1000]|None. Cross-validators: _validate_preference_dispatcher rejects setting either preference_loss or preference_loss_weights outside task='preference' (closes ordering-dependency between Part B/D validators); _validate_dpo_variants_supported_tasks gates β-schedule + ref-regen to DPO-family tasks (dpo, ipo, or preference + preference_loss in {dpo, ipo}); rejected on mlx backend with distinct error message (matches v0.34.0 distinct-reason policy); _validate_preference_loss_weights enforces 2–5 entries (single-entry rejected with actionable message pointing at scalar preference_loss), key allowlist {dpo, simpo, orpo, ipo, bco}, explicit null-byte rejection on keys (matches v0.39.0 rank_pattern policy), per-value bounds (0, 1], weights must sum to 1.0 (±1e-6), mutually exclusive with scalar preference_loss, rejected on mlx backend. compute_beta_at_step rejects bool on step and total_steps (project bool-as-int policy from v0.30.0). BetaScheduleCallback resolves total_steps lazily in on_train_begin so the schedule sees the real state.max_steps populated by HF Trainer (closes a first-cut silent-no-op bug where total_steps=0 emitted beta_end for every step). RefModelRegenCallback._regenerate uses strict=True on load_state_dict and logs at WARNING on mismatch (closes a first-cut silent partial-copy hazard where strict=False could produce a hybrid old-base + new-LoRA reference); epoch 0 regen suppressed (avoids copying untrained student); trainer .beta assignment swallow narrowed to AttributeError only. PreferenceTrainerWrapper._make_inner_cfg uses model_copy (not model_dump+model_validate) so re-validation never sees an inconsistent intermediate state and the caller's cfg is never mutated (mirrors v0.33.0 #47 immutability policy). _split_dpo_rows_to_bco skipped-row count emitted at DEBUG so production silent-degradation is inspectable (mirrors v0.33.0 #47 CrossDocCollator policy). Multi-objective live runtime weighted-loss combination is deferred to v0.40.1: PreferenceTrainerWrapper.setup raises NotImplementedError with a friendly message naming the deferred-version follow-up (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA stub-then-live pattern). Known limitation: BCOTrainerWrapper._setup_transformers still hardcodes trust_remote_code=True (v0.36.0 #63 known-gap family carry-over across non-SFT trainers).
v0.39.0 — LoRA Quality: LoraConfig.init_strategy: Literal["random","pissa","olora"] rejects unknown strategies; PiSSA + DoRA / VeRA combinations rejected at config-load. model_validator(mode="before") aligns use_olora=True → init_strategy="olora" via dict-copy (no caller mutation; matches v0.33.0 #47 immutability policy). rank_pattern/alpha_pattern: Optional[Dict[str, int]] capped at 256 keys × value (0, 1024], rejects bool (subclass of int — matches v0.30.0 Candidate policy), null bytes in keys, empty keys; cross-validator rejects with use_vera=True. ReLoRAPolicy is @dataclass(frozen=True) (post-construction mutation raises FrozenInstanceError); bounds: steps ∈ [1, 1e7], warmup_ratio ∈ [0, 1], prune_ratio ∈ (0, 1) (strict — prevents zero-everything footgun). magnitude_prune_tensor strict 0 < prune_ratio < 1 rejection, non-Tensor input raises TypeError, empty / single-element tensor short-circuits (avoids kthvalue(_, 0) runtime crash). _validate_relora_supported_tasks cross-validator rejects relora_steps with task != "sft" and backend=mlx with distinct error messages (matches v0.34.0 distinct-reason policy); multi-trainer expansion deferred to v0.39.1. is_gemma4_model uses a word-boundary regex ((?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$)) so "ungemma4ed" / "my-gemma4ish" no longer over-match; null-byte rejection on model_name. apply_gemma4_clippable_patch weight-copy fallback logs at DEBUG instead of silent random-init; the patch is gated by is_gemma4_model(cfg.base) in sft.py before invocation so non-Gemma4 trainings never traverse the module tree. apply_surgical_patches rejects empty / null-byte model_name with ValueError. templates/load_template containment: filename re-validated via _validate_name (rejects ..///\\/null/empty); os.path.realpath + os.path.commonpath containment check on the resolved path against _templates_dir() so a tampered manifest.json cannot read files outside the package directory (mirrors v0.26.0 registry policy). Tampered-manifest ValueError from _validate_name caught and falls back to inline (no propagating exception). 256 KB file-size cap. Inline TEMPLATES carries an explicit deprecation comment pointing at the canonical YAML registry; tests/test_templates_yaml.py asserts byte-equality of all 16 inline ↔ YAML pairs to prevent silent drift. Planned removal: v0.41.0+.
v0.38.0 — Quant Menu: TrainingConfig.quantization Literal extended with gptq / awq / hqq:1bit..hqq:8bit (no hqq:7bit — HQQ doesn't support it) / aqlm / eetq / mxfp4 / fp8; Pydantic rejects every other string at config-load. validate_gptq_checkpoint and validate_awq_checkpoint probe local paths for quantize_config.json / quant_config.json; HF repo IDs fall through; null-byte rejection + non-string TypeError on the ref. _validate_prequantized_no_qat rejects every pre-quantized format combined with quantization_aware (int8 QAT or 'fp8') — pre-quantized weights carry their own scale and QAT/FP8 prepare would silently corrupt them (mirrors LlamaFactory quantization.py:117/199/211). _validate_bnb_quant_storage_only_with_4bit rejects bnb_4bit_quant_storage on every non-BNB-4bit format (silent no-op otherwise); allowed dtypes: Literal["uint8", "float16", "bfloat16", "float32"]. _validate_quant_menu_supported_tasks restricts the new formats to task='sft' on backend='transformers' in v0.38.0 with distinct MLX-backend vs unsupported-task error messages (matches v0.34.0 distinct-reason policy). check_quant_distributed_compat hard-fails HQQ/EETQ/AQLM × {FSDP, ZeRO-3} (sourced from LlamaFactory quantization.py:199/211 plus AQLM dequant constraints); warning-tier (not error) for BNB-4bit + FSDP without bnb_4bit_quant_storage so users see the silent perf cliff; unknown quantization raises ValueError (no silent pass) and the check is wired into commands/train.py startup. parse_hqq_bits rejects unsupported bit-rates and malformed hqq: strings before any kernel build.
v0.37.0 — Multipack: validate_multipack_architecture raises ValueError on unknown arch (loud-fail vs Axolotl's silent-miss footgun); 18-arch frozen allowlist (Llama 3.x / Qwen 2/3 / Mistral / Gemma 2/3 / Phi 3/4 / DeepSeek V2/V3 / Mixtral / Falcon / StableLM / SmolLM2). FFD packer caps at _MAX_FFD_ITEMS=1_000_000 (algorithm is O(N²) worst-case — defence against adversarial dataset DoS); bool rejection on every numeric input (max_len, per-element lengths, batch_max_len, batch_size, seed, max_seq_length) matches v0.30.0+ Candidate policy; generator-input materialisation prevents silent empty-bin output when validation exhausts the iterator. MultipackBatchSampler rejects empty lengths, non-positive batch_max_len/batch_size, items larger than batch_max_len. build_multipack_sampler_for_lengths rejects tcfg.batch_size="auto" with actionable message (must be resolved upstream). _validate_multipack_packing_exclusive cross-validator on TrainingConfig rejects both multipack and packing set; _validate_multipack_supported_tasks on SoupConfig restricts multipack to sft/pretrain on transformers backend with distinct error messages for MLX backend vs unsupported task (matches v0.34.0 distinct-reason policy). build_4d_attention_mask caps allocations at _MAX_MASK_ELEMENTS=2³¹ cells (~8GB float32) — defence against max_length=1M × batch_size=8 OOM; tag_sub_sequences capped at _MAX_BOUNDARY_SEGMENTS=1_000_000. Mask builder rejects non-floating dtypes (was silent np.finfo ValueError), non-2D seq_pos_ids, negative segment IDs; padding (id=0) tokens are fully masked, including diagonal, so softmax is well-defined. select_packing_strategy rejects non-bool flash_attn_available. JinjaTemplateAnalyzer parses chat templates via Environment.parse only — never renders, so a crafted soup.yaml cannot trigger SSRF / filesystem reads; 128KB template cap, null-byte rejection, TemplateSyntaxError re-raised as ValueError. DEFAULT_MESSAGE_FIELDS is a frozenset (runtime-immutable); JinjaTemplateAnalyzer.message_fields returns a defensive copy.
v0.36.0 — Correctness First: --trust-remote-code opt-in replaces 9 unconditional trust_remote_code=True call sites across soup train / chat / serve / data download / eval auto; KNOWN_SAFE_PREFIXES allowlist (15 first-party orgs) suppresses warning panel for trusted repos; model_requires_trust_remote_code probes local config.json for auto_map (HF Hub repo IDs return None/unknown — HF still raises loudly when custom code is actually needed); resolve_trust_remote_code raises ValueError with actionable message when model needs custom code but the user did not opt in. Chat-template hardening: DataConfig.chat_template validator rejects null bytes, oversize (>64KB), AND filesystem-touching Jinja directives ({% include %}, {% import %}, {% from %}, {% macro %}, {% extends %} — both whitespace-control variants); empty string normalised to None; _REGISTRY wrapped in MappingProxyType (callers cannot mutate); apply_chat_template_override emits yellow advisory when active so users know soup push will persist the override into tokenizer_config.json. SFT silent f-string fallback (f"{role}: {content}") replaced with hard ValueError — produced wrong loss labels for years on tokenizers without a chat template. Loss-mask fallback passes add_special_tokens=False to incremental tokenize calls so HF cannot double-prepend BOS at front of each render (consistent prefix-delta walk); narrowed exception catch in _apply_template_with_mask from (TypeError, ValueError) to TypeError only so a malformed messages list propagates instead of falling through to the loose path. SOUP_BATCH_CACHE_PATH env override containment-checked via os.path.realpath + commonpath against ~/cwd/tempfile.gettempdir(); out-of-bounds values fall through to safe default; cache file gets best-effort 0o600 perms after atomic rename (matches v0.26.0 registry.db policy). make_cache_key rejects bool in numeric inputs (matches v0.30.0 Candidate policy). Documented limitation: non-SFT trainers (DPO/GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Pretrain/Embedding) and commands/{diff,export,merge,infer,generate}.py still hardcode trust_remote_code=True — v0.36.x patch follow-up.

Security Scanning

All code is scanned with ruff for style and common issues
Dependencies are regularly updated to patch known CVEs
GitHub's dependency scanning alerts us to vulnerable dependencies
We use GitHub Actions CI/CD for continuous integration

Dependency Updates

We actively monitor and update dependencies:

Major dependency updates: Tested in PR before merging
Security patches: Applied immediately and released as patch versions
Deprecated dependencies: Replaced proactively

Coming Soon

Automated dependency scanning
SBOM (Software Bill of Materials) for each release
Third-party security audit (after 1.0.0 release)

Questions?

If you have security questions (not vulnerability reports) or need clarification:

Open a GitHub Discussion tagged security
Open a GitHub Issue (non-vulnerability inquiries)
Check our CONTRIBUTING.md for general support

License

This Security Policy is provided under the Apache-2.0 license, same as the Soup project.

Last Updated: April 2026

For the latest version of this policy, visit: https://github.com/MakazhanAlpamys/Soup/blob/main/SECURITY.md

110 KiB Raw Blame History Unescape Escape