# Security Policy

## Supported Versions

We provide security updates for the following versions:

- **Latest minor version:** Active support (e.g., v0.20.x)
- **Previous minor versions:** Bug-fix support only
- **Versions older than 3 minor versions:** No support

Example:
- v0.53.0 -- Full support (latest)
- v0.52.0 -- Full support
- v0.51.0 -- Full support
- v0.50.0 -- Full support
- v0.49.0 -- Full support
- v0.48.0 -- Bug-fix support only
- v0.47.0-v0.47.x -- Bug-fix support only
- v0.46.x and below -- No support

## Reporting a Vulnerability

**Do not** open a public issue or pull request for security vulnerabilities.

Instead, use [GitHub Security Advisories](https://github.com/MakazhanAlpamys/Soup/security/advisories/new) to report privately, or email **vpn.alpamys@gmail.com** with:

1. **Description**: A clear explanation of the vulnerability
2. **Steps to Reproduce**: How to trigger or demonstrate the issue
3. **Affected Versions**: Which Soup versions are impacted
4. **Suggested Fix** (optional): Any proposed solutions
5. **Contact Info**: Your email for follow-up (optional)

### What to Include

```
To: vpn.alpamys@gmail.com
Subject: Security Vulnerability Report: [Brief Title]

Description:
[Explain the vulnerability in detail]

Affected Component:
[e.g., data/loader.py, trainer/sft.py, etc.]

Steps to Reproduce:
1. [Step 1]
2. [Step 2]
3. ...

Impact:
[What could go wrong? Data exposure? RCE? DoS?]

Suggested Fix (optional):
[Your proposed solution, if any]
```

## Response Timeline

- **Initial Response**: Within 48 hours
- **Assessment**: 1-3 business days
- **Fix Development**: Varies by severity
- **Patch Release**: As soon as possible after fix verification
- **Public Disclosure**: Coordinated with reporter (typically 90 days after patch release)

## Severity Levels

- **Critical**: Remote code execution, data exposure, complete compromise (patch within 24-48 hours)
- **High**: Authentication bypass, privilege escalation, denial of service (patch within 1 week)
- **Medium**: Information disclosure, partial compromise (patch within 2 weeks)
- **Low**: Minor issues with limited impact (patch in next regular release)

## Security Best Practices

When using Soup, follow these practices to stay secure:

### 1. Keep Soup Updated

```bash
pip install --upgrade soup-cli
```

### 2. Protect API Keys

Never commit API keys or secrets to version control. Use environment variables:

```bash
export HUGGINGFACE_TOKEN=your_token_here
export WANDB_API_KEY=your_key_here
soup train
```

### 3. Validate Data

- Only use trusted datasets
- Verify checksums for large datasets
- Inspect data for malicious content before training

### 4. Model Permissions

- Be cautious when downloading models from untrusted sources
- Use model hub providers with verified publishers (HuggingFace, Meta, etc.)
- Keep track of which models you've fine-tuned and their base model sources

### 5. GPU/Compute Safety

- Run on isolated machines if training on sensitive data
- Clear cache and temporary files after training
- Don't share fine-tuned models containing sensitive information

## Known Vulnerabilities

We maintain a log of known security issues and their fixes. This will be updated as issues are discovered and resolved.

### Current Status

No known critical vulnerabilities in current releases.

### Security Hardening History

- **v0.10.10**: Bearer token auth on Web UI, CORS restrictions, path traversal protection, SSRF prevention, max_tokens limits, supply-chain pinning (llama.cpp b5270), deprecated CLI secret flags
- **v0.12.0**: experiment_name path traversal validation, GaLore parameter type enforcement
- **v0.13.0**: Batch inference max_tokens capped at 16384, trust_remote_code warning
- **v0.14.0**: Plaintext loader UTF-8 encoding, MoE config validation (moe_aux_loss_coeff ge=0, moe_lora boolean only)
- **v0.14.3**: Data validate auto-detects format, Web UI `--show-token` flag + auth token documented
- **v0.15.0**: `rope_scaling_type` Literal constraint, `max_length` bounds (ge=64, le=1048576), FSDP config key allowlist, Liger Kernel exception handling narrowed
- **v0.16.0**: `embedding_loss` Literal constraint, `embedding_margin` gt=0 validation, ONNX export without trust_remote_code (with warning), TensorRT export subprocess list args (no shell injection), speculative decoding SSRF-protected (URL blocked) with warning panel, vLLM speculative model URL validation
- **v0.17.0**: Server data generation provider SSRF validation (scheme whitelist + localhost-only HTTP), audio model trust_remote_code warning panel, audio file path traversal protection (resolved paths confined to audio_dir), SGLang backend trust_remote_code warning panel
- **v0.18.0**: Ollama deploy GGUF path traversal protection + `.gguf` extension validation, model name validation (no path separators/null bytes), subprocess list args (no shell injection), Modelfile parameter key allowlist + value newline/null sanitization, overwrite warning panel
- **v0.19.0**: Custom eval JSONL schema validation + 10k task cap, regex scoring ReDoS guard, judge API SSRF protection + API key isolation, human eval local-only terminal UI + 10k prompt cap, leaderboard read-only SQLite queries
- **v0.20.0**: Ollama provider localhost-only validation (remote blocked), Anthropic provider API key from env only (never CLI arg), vLLM provider SSRF protection (scheme whitelist + localhost-only HTTP), output path traversal protection (`..` blocked), configurable rate limiting (`--requests-per-minute`)
- **v0.21.0**: Migrate input/output path traversal protection (resolve + relative_to(cwd)), Unsloth .ipynb AST-only parsing (no exec/eval), recipes output path traversal protection, NEFTune config bounded (ge=0.0, le=50.0)
- **v0.22.0**: Multi-adapter serving path traversal protection (resolve + relative_to(cwd)), adapter name validation (alphanumeric + hyphens only), unknown adapter returns 404 (not 500)
- **v0.23.0**: AWQ/GPTQ calibration data path traversal protection (resolve + relative_to(cwd)), AWQ/GPTQ output path stays under cwd, curriculum_buckets bounded (ge=1, le=20), AWQ/GPTQ trust_remote_code warning panel
- **v0.24.0**: HF download trust_remote_code=False + warning panel, HF download output path sanitized (Path.name), download --samples capped at 1M, dataset registry name validation (no path separators/null bytes), registry path traversal protection, loss_watchdog threshold le=100 + patience le=1000, freeze_layers le=1000
- **v0.24.1**: AWQ/GPTQ output path traversal validation moved before import check (previously unreachable when autoawq/auto-gptq not installed), Windows Unicode fix for Rich console output (replaced non-ASCII symbols with ASCII equivalents)
- **v0.24.2**: Chat proxy SSRF protection (localhost-only HTTP, HTTPS for remote), chat proxy max_tokens capped at 16384 + temperature/top_p bounded, chat proxy Bearer token auth required, XSS prevention (HTML-escape before markdown render), runs compare max 5 runs, config from-form validates via load_config_from_string, SSE read endpoints no auth (GET)
- **v0.25.0**: Tool-calling JSON-only parsing (no eval), RLVR math_verify regex-extracted numerics (no eval), code_exec 5s timeout + 512MB RLIMIT on POSIX + ephemeral cwd + socket patch + `python -I -S` + 10KB output cap, verifiable_domain Literal constraint, LoRA PEFT mutual exclusion (DoRA/VeRA/OLoRA), data augment path containment + caps, forgetting_detection bounds, checkpoint_intelligence bounds + symlink refusal, autopilot path containment (realpath + commonpath) + goal Literal + GPU/time budget bounds, MLX trainers no trust_remote_code
- **v0.26.0 — Registry**: name/tag validation (alphanumeric + `_-.` only, null-byte rejected, name ≤128 / tag ≤64 chars), artifact path containment (default `enforce_cwd=True` via `os.path.realpath + commonpath`, stored path is realpath), SQL LIKE wildcard escaping (`%` and `_` escaped with `ESCAPE '\\'` in `search()` and prefix `resolve()`), DB 600 perms on POSIX, lineage indirect-cycle detection (BFS ancestor walk before insert), CLI Rich markup escaped everywhere, `resolve()` raises `AmbiguousRefError` on ambiguous prefix (no silent None)
- **v0.26.0 — Eval Gate**: suite path via shared `utils/paths.is_under_cwd` containment, `regression_threshold` [0.0, 1.0], `every_n_epochs` [1, 100], `on_regression` Literal ("stop"/"warn"/"continue"), `GateTask.tasks`/`prompts` null-byte rejection, `judge_model` URL scheme allowlist (`ollama://`, `https://`, `http://localhost`/`http://127.0.0.1`) — SSRF hardening, callback fails-safe: structured errors treated as regressions under `on_regression="stop"`
- **v0.26.0 — Trace-to-Preference**: input/output path containment via shared `is_under_cwd`, trace line cap 100,000, `--format`/`--signal` Literal validation, PII warning panel before every run, JSON-only parsing (no eval), malformed JSON lines skipped silently
- **v0.26.0 — Quant-Lobotomy**: `--before`/`--after`/`--tasks` all containment-checked, `registry://` refs support optional `kinds` filter to avoid picking the wrong artifact, format Literal validated
- **v0.26.0 — Soup Cans**: Manifest format version pinned to 1; name alphanumeric+`_-.`; author max 128 chars, no null bytes/newlines; created_at must parse via `datetime.fromisoformat`; description max 4096; DataRef URL HTTPS-only; hf_dataset regex-validated; tar extraction uses `filter="data"` on Python 3.12+, fallback only on `TypeError`/`AttributeError` (not `TarError`); manual symlink/hardlink rejection + `commonpath` check; 100 MB size cap on pack + fork; dunder-key (`__*__`) and null-byte rejection in fork modifications to prevent prototype pollution; inspect/read_config refuse paths outside cwd
- **v0.27.0 — Multi-GPU Mastery**: `--gpus` bounds (reject bool, non-digit, zero, negative, values above `MAX_GPU_COUNT=128`); `--gpus auto` on 0-GPU host prints explicit yellow warning (no silent no-op); Rich markup escaped on `--config` path before embedding in the multi-GPU advice Panel; `accelerate launch` argv assembled via `shlex.quote` per element (copy-pasted command safe against crafted paths); `build_accelerate_argv` validates `num_processes >= 1`, `mixed_precision` Literal (`no/fp16/bf16/fp8`), `num_machines` bounded `[1, 256]`; ZeRO++ integer literals (`int(1e9)` not float) so DeepSpeed strict JSON validator accepts; `validate_fsdp2_compile_config` requires FSDP + CUDA + transformers + torch>=2.2/accelerate>=0.27; DeepSpeed-MII stub exits non-zero to prevent silent mis-start; `validate_pipeline_config` enforces `pipeline_stages >= 2` + CUDA + `gpu_count >= stages`; `pipeline_stages` Pydantic bounds `[1, 16]`; `parallelism` Literal `data|pipeline`; NCCL env (`NCCL_P2P_DISABLE`/`NCCL_IB_DISABLE`/`NCCL_NVLS_ENABLE`) applied via `os.environ.setdefault` only — user/launcher overrides are never stomped
- **v0.28.0 — Training Speed & Memory**: `quantization_aware: Union[bool, Literal["fp8"]]` rejects arbitrary strings (only `true` / `false` / `"fp8"`); FP8 path requires CUDA + Hopper+ SM capability + transformers backend; `gradient_checkpointing: Union[bool, Literal["selective","medium","full","auto"]]` rejects unknown tier strings and returns only HF-supported keys (no private markers leak into `TrainingArguments.gradient_checkpointing_kwargs`); `activation_offloading` Literal `cpu|disk`, scratch `save_dir` containment-enforced via shared `utils/paths.is_under_cwd` before disk writes, `torch.load(weights_only=True)` prevents arbitrary Python deserialization on reload, TOCTOU closed between `mkstemp` and `torch.save` by holding the fd open, best-effort cleanup on context exit (handles SIGKILL mid-backward); `kernel_picker.pick_best_kernel` raises `ValueError` when all candidates lack a finite `time_ms` (prevents silent promotion of an untimed combo); Cut CE architecture detector matches on last path component only (so `deepseek-ai/...-phi-...` org-prefix does not trigger a Phi patch on a DeepSeek model); `build_cross_doc_mask` numpy-vectorised to avoid O(seq_length²) pure-Python fill at `max_length` bound (1M); `@model_validator` requires `packing=true` when `packing_cross_doc_attn_mask=true` (prevents silent no-op); `SoupConfig._validate_v028_speed_memory_sft_only` rejects `use_cut_ce`/`quantization_aware="fp8"`/`kernel_auto_compose`/`activation_offloading` on non-SFT tasks — prevents legacy int8-QAT wrapper from crashing on the string `"fp8"` and prevents silent no-ops on DPO/GRPO/KTO/etc. (multi-trainer wiring tracked for v0.28.1)
- **v0.29.0 — HF Hub Deep Integration**: `HF_ENDPOINT` SSRF-hardened — scheme allowlist (http/https), null-byte rejection, `0.0.0.0` explicitly rejected, plain-HTTP only permitted for loopback (`localhost`/`127.0.0.1`/`::1`), RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via `ipaddress.ip_address`; repo ID regex `[A-Za-z0-9][A-Za-z0-9._-]{0,95}` per component, ≤200 chars total, null-byte / whitespace / `..` / leading-`/` rejection (applied to `push --repo`, `train --push-as`, `data push --hf-dataset`, `deploy hf-space --model/--space`); collection slug `owner/slug-hash` regex-validated, ≤256 chars; HF token resolution single-sourced in `utils/hf.resolve_token` (env > cached login), explicit non-printable tokens rejected, `push --token` flag deprecated with yellow warning; `soup push --model` confined to cwd via `is_under_cwd` (prevents crafted `soup.yaml output:` from uploading system files); auto-push checkpoint `allow_patterns` restricts uploaded files to `*.safetensors`/`*.bin`/`*.pt`/`*.json`/`tokenizer*`/`trainer_state.json`/`training_args.bin`/`README.md` (keeps `.env` and source files out of auto-pushed branches); `prepare_hf_resume` enforces cwd containment and passes `local_dir_use_symlinks=False` (defeats symlink-based FS escape on older `huggingface_hub`); commit messages stripped to first line and capped at 200 chars (prevents multi-line injection into public HF commit history); `_render_eval_scorecard` neutralises `|`/`[`/`]`/`(`/`)`/`!`/newlines/tabs/`<`/`>` in task names and non-numeric scores; `data_lineage` HTML-escaped (defeats XSS on HF Hub README viewer); `render_space_template` validates `model_repo` via `validate_repo_id` before substitution into rendered `app.py` (crafted repo id cannot inject Python code); `HFPushCallback` uses sticky `_repo_failed` flag to short-circuit retries after hard failure (no log spam, no wasted API calls); `add_to_collection` prefers HfHubHTTPError 409 detection over string-match for duplicate handling
- **v0.30.0 — Inference Excellence**: OTLP endpoint SSRF-hardened matching v0.29.0 `HF_ENDPOINT` (scheme allowlist, `0.0.0.0` rejected, RFC1918 / link-local / cloud-metadata via `ipaddress.ip_address`, plain HTTP loopback-only); `pick_draft_model` rejects URL-scheme target names (`http://`/`https://`/`file://`), null bytes, names >200 chars; `validate_regex_pattern` length-capped at 2048 + null-byte rejection + must compile; `validate_json_schema` dict shape + 64KB serialised cap + required `type` field; `--json-schema` file path confined to cwd via shared `utils/paths.is_under_cwd`; `--structured-output json` requires `--json-schema` (fail-fast prevents silent no-op); FastAPI `/v1/adapters/activate/{name}` pattern `^[a-zA-Z0-9][a-zA-Z0-9\-]*$` enforced before handler runs; activate/deactivate state protected by `threading.Lock` (no race on concurrent hot-swap); `/v1/adapters` response omits filesystem paths (names + active flag only); CORS on transformers backend restricted to loopback origins (`^https?://(localhost|127\.0\.0\.1)(:\d+)?$`) since hot-swap endpoints mutate state without auth; `auto_quant.Candidate` name regex + score `[0.0, 1.0]` + finite-float check + non-negative latency; `pick_best` generator-safe (materialises to list) so error-message count is accurate; OTel span context uses `contextlib.ExitStack` so `__exit__` sees real exception info (spans correctly marked error on HTTPException); `record_latency` always runs in `finally` so tail-latency percentiles include failure paths; `build_tracer` idempotent — only installs provider when current is `ProxyTracerProvider`/`NoOpTracerProvider` (preserves operator-supplied instrumentation)
- **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML)
- **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input.
- **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS`
- **v0.53.0 — Quant Menu II (UD GGUFs + KV cache + NVFP4 + LF parity + save formats)**: 6 schema-only Parts; live wiring deferred to v0.53.1. Every new validator follows the project's established hardening policy: closed allowlists (`UD_GGUF_FORMATS`, `IQ_GGUF_FORMATS`, `APPLE_ARM_GGUF_FORMATS`, `KV_CACHE_TYPES`, `MERGE_SAVE_FORMATS`, `TORCHAO_PTQ_SCHEMES`) as `frozenset` so registries cannot be mutated; `_GGUF_METADATA` / `_KV_CACHE_METADATA` / `_MERGE_METADATA` / `_TORCHAO_METADATA` wrapped in `MappingProxyType`; `_LOWER_INDEX` for GGUF lookup is also `MappingProxyType`-wrapped (replaces O(N) walk with O(1) lookup — code-review MEDIUM fix). All string validators reject non-string / bool / empty / null-byte / oversize with case-insensitive normalisation (matches v0.41.0 `validate_optimizer_name` / v0.51.0 `validate_hub_name` policy); `validate_torchao_scheme` is INTENTIONALLY case-sensitive (PyTorch class names — `torchao.quantize_` looks them up by exact name) with the asymmetry documented at both validators (security-review LOW fix). `validate_calibration_data_path` + `validate_quant_config_path` are shape-only at this release; their docstrings name the exact controls a v0.53.1 CLI dispatch contributor MUST add (`os.path.realpath` + `os.path.commonpath` cwd containment, `os.lstat` + `stat.S_ISLNK` symlink rejection before `open()`, existence check, `yaml.safe_load`-only for quant configs) — closes the security-review MEDIUM "documentation gap at trust boundary" finding. SoupConfig cross-validators: `_validate_fp8_attention_compat` (requires `quantization_aware='fp8'` BEFORE the MLX gate so the more actionable error fires first — code-review MEDIUM fix); `_validate_nvfp4_compat` (non-MLX + `modality='text'`; Blackwell SM ≥ 12.0 runtime check fires at trainer construction); `_validate_unsloth_bnb_4bit_compat` (requires `backend='unsloth'` + `quantization='4bit'`); `_validate_bnb_4bit_double_quant` (requires `quantization='4bit'` — rejects `none`/`8bit`/Quant-Menu); `_validate_llm_int8_alias` (asserts `quantization='8bit'`, deliberately disjoint from v0.41.0 `load_in_8bit` aliasing); `_validate_quantize_ref_reward` (extended ref-task allowlist `{dpo, ipo, simpo, orpo, bco, kto, preference, grpo, ppo}` per code-review HIGH fix — first-cut omitted grpo + kto + ppo which all have reference policies); `_validate_kv_cache_type_supported` (only `fp8` gated to non-MLX in v0.53.0; q8_0/bf16/f16 pass-through documented at validator site so v0.53.1 contributor sees the gate immediately). `requires_hopper` reads from `_KV_CACHE_METADATA` spec — single source of truth so adding a Hopper-only type means flipping the spec field only (code-review MEDIUM fix). All 7 new bool fields share `_validate_v053_bool_fields` `field_validator(mode='before')` that rejects bool-as-int with explicit `TypeError("v0.53.0 flag must be bool")` and passes `None` through to Pydantic's `default=False` rather than silently coercing it (python-review MEDIUM fix — `fp8_attention: null` in YAML now surfaces as a "valid boolean" ValidationError instead of masquerading as `False`). Known limitations: (1) Every live wiring is deferred to v0.53.1 — `export_advanced_gguf`, `apply_kv_cache_type`, `apply_fp8_attention`, `apply_nvfp4`, `merge_4bit`, `export_torchao` all raise `NotImplementedError` with explicit `v0.53.1` markers. (2) `validate_calibration_data_path` + `validate_quant_config_path` are shape-only this release; CLI dispatch in v0.53.1 MUST add cwd-containment + TOCTOU symlink rejection. (3) `kv_cache_type` MLX permissive policy: only `fp8` is rejected, the other three pass-through; v0.53.1 may narrow further. (4) Hopper SM-capability check is runtime-only — schema accepts `kv_cache_type='fp8'` + `fp8_attention=true` without GPU probe. (5) NVFP4 + Blackwell (SM ≥ 12.0) check is runtime-only. (6) `bnb_4bit_use_double_quant` only gated against `quantization`, not against `quantization_aware` — the latter combination is already rejected by v0.28.0 Quant-Menu + QAT cross-validator. (7) `llm_int8` is an assertion not an aliaser — diverges from v0.41.0 `load_in_8bit` design on purpose. (v0.53.0)
- **v0.52.0 — Modality II (TTS + Distillation + BitNet + EBFT-GDPO + MoE quant + reasoning_effort)**: 7 schema-only Parts; live trainer / loss / export wiring deferred to v0.52.1. Every new validator follows the project's established hardening policy: closed allowlist (`SUPPORTED_TTS_FAMILIES`, `CLASSIFIER_TASKS`, `DIVERGENCES`, `BITNET_QUANT_FORMATS`, `BITNET_EXPORT_FORMATS`, `EBFT_VARIANTS`, `GDPO_VARIANTS`, `MOE_EXPERT_QUANT_FORMATS`, `REASONING_EFFORT_LEVELS`, per-family `_FAMILY_EMOTIONS`) wrapped in `frozenset` / `MappingProxyType` so registries cannot be mutated at runtime; `validate_*` helpers reject non-string / bool / empty / null-byte / oversize / unknown inputs with case-insensitive normalisation (matches v0.41.0 `validate_optimizer_name` / v0.50.0 `grpo_variant` / v0.51.0 `hub` policy); float validators (`validate_distill_temperature`, `validate_ebft_temperature`) gate on `math.isfinite` to reject NaN AND `±inf` (matches v0.32.0 `save_lr_finder_report` policy). `field_validator(mode="before")` on `num_labels` (security-review HIGH fix) rejects `bool` before Pydantic's `ge=1` coercion silently treats `True` as `1`. Field validator on `reasoning_effort` routes through the shared `validate_reasoning_effort` helper so the schema and runtime validator agree on what's accepted (security-review MEDIUM fix). SoupConfig cross-validators: `_validate_tts_compat` (requires `task='tts'` + `modality='audio_out'` + non-MLX backend; per-family emotion allowlist via `_FAMILY_EMOTIONS`), `_validate_classifier_compat` (with lazy-import early-return — code-review HIGH fix — so SFT hot path doesn't pay import cost; requires `num_labels` on classifier tasks; rejects classifier-only fields outside the task family with named offenders), `_validate_distill_compat` (requires `teacher_model` when `task='distill'`; rejects distill-only fields outside the task), `_validate_bitnet_compat` (gates to non-MLX + text-modality + task ∈ {sft, pretrain, dpo}), `_validate_ebft_compat` + `_validate_gdpo_compat` (task-family gates), `_validate_moe_expert_quant_compat` (requires `moe_lora=true` to prevent silent no-op), `_validate_reasoning_effort_task_gate` (code-review HIGH fix — rejects `reasoning_effort` + `train_on_eot` outside the SFT-family task set with named offenders; mirrors v0.50.0 GRPO stability task-gate policy). Public `DIVERGENCES` frozenset is derived from `_DIVERGENCE_ALIASES` so adding a new alias updates both the accepted-input set and the error message in lockstep (review fix LOW). `validate_bitnet_export` enforces a closed-allowlist canonical form for `soup export --format <bitnet|tq1_0>`, both of which are CLI-registered with a yellow advisory panel + `Exit(0)` stub (no artifact written until v0.52.1 — the format flag is accepted so existing scripts pinned to v0.52.0 will not break). 6 new YAML recipes appended (5 TTS + Falcon-E BitNet) — every entry is exercised by `tests/test_v0520.py` for `load_config_from_string` round-trip + `_no_null_or_whitespace` model-id check (mirrors v0.51.0 review-fix LOW). Known limitations: (1) Every live trainer / loss / export path is deferred to v0.52.1 — `build_tts_trainer`, `build_classifier_trainer`, `build_distill_trainer`, `build_bitnet_trainer`, `export_bitnet_gguf`, `apply_ebft_loss`, `apply_gdpo_loss`, `apply_moe_expert_quant` all raise `NotImplementedError` with explicit `v0.52.1` markers; schema accepts every new task / quant / variant + the CLI stub for `soup export --format bitnet/tq1_0` prints a deferred-advisory panel and exits 0. (2) `modality='audio_out'` accepted on non-TTS tasks — design choice this release so future audio-output tasks (ASR / V2A) can reuse it; today's runtime trainer dispatch must check `task == 'tts'` to avoid silent routing into the deferred TTS path. (3) Oute emotion allowlist is a tight 6-entry subset (neutral / happy / sad / angry / calm / excited); operators wanting custom emotions will need a v0.52.1 patch to extend `OUTE_EMOTIONS`. (4) `is_bitnet_model` is best-effort heuristic over name prefixes (`bitnet`, `falcon-e`, `1bitllm`, `onebit`); a BitNet checkpoint published under an org without any of those prefixes returns False. This is detection, not gating — the trainer wrapper (v0.52.1) loads the model regardless of the heuristic. (5) `quantization='bitnet_1.58'` gated to task ∈ {sft, pretrain, dpo} — extending to GRPO / PPO / RewardModel requires upstream onebitllms RL kernels not yet shipped. (v0.52.0)
- **v0.51.0 — Model Catalog Expansion + Alternative Model Hubs**: 5 release Parts. New `soup_cli/utils/hubs.py` ships closed allowlist `SUPPORTED_HUBS = frozenset({hf, modelscope, modelers})` + three `MappingProxyType`-wrapped registries (`_HUB_DEFAULT_ENDPOINTS` / `_HUB_ENDPOINT_ENV` / `_HUB_PACKAGE`) so the registry cannot be mutated at runtime (matches v0.36.0 `_REGISTRY` policy). `validate_hub_name` rejects non-string / bool / empty / null-byte / >32-char / unknown with case-insensitive normalisation (matches v0.41.0 `validate_optimizer_name` policy). `validate_hub_endpoint` is the SSRF kernel — full parity with v0.29.0 `utils/hf.resolve_endpoint`: scheme allowlist (`http`/`https` only), null-byte rejection, **control-character / CRLF rejection** added in v0.51.0 as a defence-in-depth review fix (defends against URL-as-HTTP-header injection if the URL ever flows into a raw HTTP client), `0.0.0.0` explicitly rejected, plain HTTP only for loopback `{localhost, 127.0.0.1, ::1}`, RFC1918 / link-local / cloud-metadata IPs (169.254.x) rejected via `ipaddress.ip_address` for plain HTTP. `resolve_endpoint(hub, *, env=None)` looks up the per-hub env var (`HF_ENDPOINT` / `MODELSCOPE_ENDPOINT` / `MODELERS_ENDPOINT`) and runs the override through `validate_hub_endpoint`; default endpoints are baked-in HTTPS URLs. `is_hf` rejects `bool` explicitly (review fix HIGH — bool is a subclass of int and would have silently fallen through `hub.lower() == "hf"` → `False`, which happens to be correct by accident but violates the contract; matches v0.30.0 `Candidate` / v0.34.0 `estimate_run_cost_usd` policy). `TrainingConfig.hub: Literal["hf","modelscope","modelers"]` field gets a `field_validator(mode="before")` `_normalize_hub` that delegates to `validate_hub_name` so `hub: HF` in YAML normalises to `"hf"` (review fix HIGH — first-cut had Pydantic Literal exact-match while `validate_hub_name` was case-insensitive, breaking the v0.41.0 `validate_optimizer_name` / v0.50.0 `grpo_variant` / `rollout_backend` policy of agreement between schema and shared validator). SoupConfig `_validate_hub_supported` cross-validator rejects `hub != 'hf'` on `backend == 'mlx'` with a distinct error message (review fix HIGH — `mlx-lm` only downloads from HF Hub; without this gate a `backend: mlx` + `hub: modelscope` config would silently pass schema load and fail at runtime with a confusing `mlx-lm` error). 26 new YAML recipes appended to `soup_cli/recipes/catalog.py` — every entry is exercised by `tests/test_v0510.py` via `load_config_from_string` round-trip + `yaml.safe_load` (no Python tags / no template injection / no credential leak in the YAML strings) + a `_no_null_or_whitespace` model-id check that rejects empty path components (review fix LOW — first-cut allowed `"/name"` leading-slash IDs to pass). Two non-`<N>B` `size` strings (`"image"` / `"ocr"` / `"moe"` / `"medium"`) were normalised to `"N/A"` (review fix MEDIUM — `search_recipes(size=…)` would silently miss those entries, and the autopilot VRAM estimator could not parse them). Known limitations: (1) Live downloader / uploader / push integration deferred to v0.51.1 — `TrainingConfig.hub` schema lock-in ships now (Literal accept + MLX cross-validator + case-normalisation), but `soup data download --hub modelscope` and `soup push --hub modelers` still route through the existing HF Hub code path; the actual `modelscope-sdk` / `openmind-hub` adapters are the v0.51.1 deliverable. Same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus. (2) Speculative / aspirational `base` model IDs in some Part A/C recipes — the catalog ships entries for `openai/gpt-oss-{20,120}b`, `THUDM/glm-5`, `Qwen/Qwen-Image`, `deepseek-ai/DeepSeek-OCR`, `PaddlePaddle/PaddleOCR-VL`, `google/embeddinggemma-300m` so users have ready-made recipes the moment those repos go live (matches the plan's "match Unsloth's day-zero coverage" directive). Recipes for not-yet-published repos will surface a clear HF Hub 404 when the user runs `soup train --recipe <name>`. (3) DNS-resolved private hostnames not blocked — `validate_hub_endpoint` only rejects literal RFC1918 / link-local IP addresses; a hostname like `corp-proxy.internal` that DNS-resolves to a private IP is accepted at validation time (mirrors the v0.29.0 `HF_ENDPOINT` policy — DNS resolution is intentionally not performed in this local-tool threat model). (v0.51.0)
- **v0.50.0 — GRPO Plus (RL parity)**: 22 features across 5 Parts shipped as schema-only (closed allowlists + Pydantic validators + NotImplementedError stubs for live wiring deferred to v0.50.1). All new validators follow the project's bool-rejection-before-int policy (matches v0.30.0 `Candidate`); closed-allowlist `validate_grpo_variant` / `validate_rollout_backend` reject non-string / bool / empty / null-byte / oversize / unknown inputs with actionable error messages and case-insensitive normalisation. `validate_grpo_delta` is bool-first / `math.isfinite` / `(0, 1]` bounded (matches v0.32.0 `save_lr_finder_report` / v0.41.0 Part B `lr_groups` policy). New `_VARIANT_METADATA` (Part A) and `_BACKEND_METADATA` (Part C) are `MappingProxyType`-wrapped frozen-dataclass registries (matches v0.36.0 `_REGISTRY` / v0.41.0 `_OPTIMIZER_PACKAGES` policy). Security-review fixes: (1) `grpo_delta` schema gets an explicit `field_validator(mode='after')` calling `math.isfinite` — Pydantic's `gt=0, le=1` bounds only incidentally reject NaN (since `NaN > 0` is False); the explicit validator prevents a future Pydantic change from regressing the guard. (2) `validate_long_context_grpo_compat` adds null-byte rejection on `task` AND `backend` strings + a `bool` guard on `use_ring_attention` (parity with `validate_grpo_variant` / `validate_rollout_backend`). (3) `validate_vllm_sleep_mode_compat` adds null-byte rejection on `backend`. Code-review HIGH fixes: (4) `_validate_grpo_stability_task_gate` now includes `grpo_fp16` in the GRPO-only-fields list — previously a user could silently set `grpo_fp16: true` on `task='sft'` and have it no-op. (5) `_validate_vllm_sleep_mode` now requires `task='grpo'` (sleep mode is a between-rollouts feature, meaningless on SFT) and rejects with a `task='grpo'` message. TDD-review HIGH fixes: (6) new `_reject_bool_on_grpo_numerics` field_validator on every Part D numeric field + `grpo_delta` explicitly rejects `bool` before Pydantic's `True→1` coercion (matches v0.30.0 / v0.41.0 Part B / v0.43.0 Part B policy). Known limitations: (1) Every live loss kernel / launcher (`apply_variant_loss`, `apply_vllm_sleep_mode`, `launch_rollout`, `build_prm_trainer`) raises `NotImplementedError` with explicit `v0.50.1` markers — same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro / v0.45.0 plugins / v0.48.0 curriculum / v0.49.0 LongLoRA. (2) `long_context_grpo` requires Tiled MLP (v0.56.0 Part A) to actually run; the schema gate ships now so v0.50.0 configs are stable. (3) `vision_grpo=true` does not check whether the base model is actually a VLM — upstream trainer surfaces that error loudly. (4) The 7 stability knobs schema-validate but none are wired into a live callback in this release; `replay_buffer_size`, `defer_rerolling`, and `skip_zero_advantage` are pure schema lock-ins. (v0.50.0)

- **v0.49.0 — Long Context & Architecture**: 4 release Parts ship YaRN RoPE scaling, Dynamic NTK hardening, LongLoRA S² shifted-sparse attention (schema-only gate), and full Llama 3.1 NTK-aware scaling. Security review hardened the public boundary of `soup_cli/utils/long_context.py::get_rope_scaling_config` — `target_length` / `original_length` / `yarn_factor` now reject `bool` / NaN / Inf / non-positive at entry so a direct caller bypassing Pydantic cannot emit `{factor: NaN}` into HF model configs (matches v0.30.0 `Candidate` / v0.34.0 `estimate_run_cost_usd` / v0.41.0 Part B `lr_groups` policy). `scale_inv_freq_llama3` rejects `bool` on every numeric parameter (`inv_freq` / `scale_factor` / `low_freq_factor` / `high_freq_factor` / `old_context_len`) — first-cut only guarded `inv_freq`. `yarn_get_mscale` now raises on non-finite input (review fix LOW — first-cut silently clamped NaN/Inf to identity, hiding the misconfig from callers). `detect_llama3_rope_in_config` uses explicit `is None` instead of the `or` idiom when probing `rope.get("type")` — falsy-but-set values no longer silently fall through to `rope_type` (matches v0.40.6 review-fix policy). New `soup_cli/utils/longlora.py` ships `is_llama_model` with a word-boundary regex `(?:^|[^a-z0-9])(?:code)?-?llama(?:-?\d+(?:\.\d+)?)?(?:[^a-z0-9]|$)` — substring `"llama"` inside an unrelated identifier (e.g. `my-llama-style-finetune`) does NOT match; null-byte rejection on `model_name` (matches v0.39.0 `is_gemma4_model` / v0.44.0 `is_llama4_model` policy); 512-char cap returns `False` rather than raising (bounded scan time). `_LLAMA_REGEX` has no nested quantifiers / overlapping alternation and is ReDoS-bounded by the 512-char cap. `validate_longlora_compat` emits distinct error messages per failure mode (mlx vs other non-transformers backends — matches v0.34.0 review-fix policy on distinct actionable rejections). `TrainingConfig` `field_validator(mode='before')` rejects `bool` on the four yarn fields before Pydantic's `gt`/`le` coercion silently treats `True` as `1.0`. SoupConfig `_validate_longlora_compat` invokes `validate_longlora_compat` at config load so a misconfigured `soup.yaml` fails fast with an actionable message rather than silently no-opping at trainer construction time (mirrors v0.39.0 ReLoRA / v0.48.0 curriculum_dynamic schema-gate policy). Known limitations: (1) LongLoRA live `LlamaAttention.forward` override deferred to v0.49.1 — `apply_longlora_forward_override` raises `NotImplementedError` with the v0.49.1 marker; the schema gate ships now so misconfigured runs cannot reach trainer construction. (2) LongLoRA architecture allowlist is Llama 1/2/3.x + CodeLlama only; Mistral / Qwen / Phi expansion tracked for v0.49.1+. (3) Llama 3.1 NTK auto-detect helper `detect_llama3_rope_in_config` ships but is not yet wired into `apply_long_context_config`; trainer wiring can pick it up when needed.
- **v0.48.0 — Adaptive Training (BETA)**: 2 release Parts ship a dynamic curriculum re-weighter and a Bayesian data-mixing optimiser. New `soup_cli/utils/curriculum_dynamic.py` ships frozen `DynamicCurriculumPolicy` with bounded fields (`num_buckets ∈ [1, 20]`, `recompute_every_n_steps ∈ [1, 100_000]`, `floor ∈ (0, 1/num_buckets]`, `temperature > 0`); bool-rejected on every numeric input (matches v0.30.0 `Candidate` / v0.34.0 `estimate_run_cost_usd` policy); `math.isfinite` on every float (matches v0.32.0 / v0.47.0 policy). `compute_bucket_weights` water-fill design (review fix HIGH — first-cut had a trailing `w/sum(w)` renorm that could push elements sitting exactly at the floor below the floor when accumulated float error left the sum slightly > 1.0; renorm removed because softmax already sums to 1.0 so water-fill output also sums to 1.0). `validate_distributed_curriculum` cross-validator rejects `enabled=True` with `world_size > 1` unless the caller attests an `all_reduce` hook is registered — DDP/grad-accum footgun: divergent per-rank stats without coordination silently desynchronise the sampler. SoupConfig cross-validators reject `curriculum_dynamic=true` on mlx backend (HF Trainer-callback specific) and on non-SFT/pretrain tasks with distinct error messages (matches v0.34.0 review-fix policy). `render_curve` and `parse_history_jsonl` enforce `_MAX_HISTORY_ROWS = 100_000` DoS cap (review fix MEDIUM — first-cut had no cap, an attacker-controlled JSONL with 10M rows would have OOM'd the process). New `soup runs curriculum-curve` CLI: `is_under_cwd` containment, `os.lstat + S_ISLNK` rejection (TOCTOU defence — mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E / v0.46.0 Part A policy), 50 MB file-size cap + 100k-line streaming cap (review fix HIGH — without these, `--history /path/to/giant.jsonl` would read the file into memory unbounded), null-byte rejection on tracker-supplied `output_dir` (defence-in-depth before `os.path.join`). New `soup_cli/utils/data_mix.py` ships frozen `MixCandidate` with simplex constraint (`sum(weights) == 1.0 ± 1e-6`), finite eval_loss (`math.isfinite`), `_MAX_LOSS = 1e6` sanity cap, bool / negative rejection on every numeric. `validate_datasets` enforces 2-32 entries (review fix MEDIUM — first-cut had a `not raw` empty-only guard that fell through to the realpath loop for single-entry inputs, surfacing a less-actionable error after path resolution), `is_under_cwd` containment, symlink rejection, dedup, null-byte / oversize / non-string rejection. `parse_budget` accepts digits + optional `s`/`m`/`h` suffix bounded to `[60s, 24h]`. `run_mix_optimizer` per-candidate proxy exceptions are logged at DEBUG + `opt.tell(_MAX_LOSS)` + `continue` (review fix MEDIUM — first-cut raised `RuntimeError` on the first proxy failure, breaking the documented `partial=True` contract; isolation policy mirrors v0.33.0 #47 `CrossDocCollator` and v0.40.3 `judge_filter_pairs`); `KeyboardInterrupt`/`SystemExit` re-raised; NaN proxy returns logged + skipped. `render_mix_recipe_yaml` rejects newlines / null bytes / >4096-char dataset paths (defends against YAML key injection — mirrors v0.46.0 Part A `render_recipe_yaml` policy). `write_mix_recipe` is atomic via `tempfile.mkstemp + os.replace`; `is_under_cwd` containment + ≤4096-char path + `os.lstat + S_ISLNK` rejection (TOCTOU) + overwrite-required gate. `load_mix_recipe` `os.lstat` is wrapped in `try/except OSError` (review fix HIGH — first-cut called `os.lstat` bare after `os.path.lexists`, leaving a TOCTOU race where path disappearance between the two calls would raise an unhandled `OSError` to the user); 256 KB file cap; `yaml.safe_load` only. New `soup data mix --optimize / --apply` Typer command with mutually-exclusive modes. Known limitations: (1) Live HF Trainer callback for dynamic curriculum deferred to v0.48.1 — schema gates, math kernel, and visualiser ship; the callback wiring with `torch.distributed.all_reduce` of per-bucket stats lands in v0.48.1. (2) Multi-trainer expansion (DPO/GRPO/etc.) deferred to v0.48.1 — schema rejects non-SFT/pretrain because per-sample loss semantics differ enough that bucket-level uncertainty does not transfer cleanly. (3) Live proxy training loop for `soup data mix` deferred to v0.48.1 — CLI ships with a synthetic offline proxy (quadratic penalty around uniform mixture). (4) `scikit-optimize` integration deferred to v0.48.1 — `OptimizerProtocol` ducktype is the integration point; default fallback is a deterministic Dirichlet sampler. (5) Both features ship `BETA:`-prefixed in CLI help and field descriptions until reference-benchmark validation lands in v0.48.1.
- **v0.47.0 — Data Forge**: 2 release Parts ship a synthetic data pipeline with full provenance + a data-quality scorecard. New `soup_cli/utils/data_forge.py` ships frozen `ForgePlan` / `ProvenanceRecord` / `ForgeRow` dataclasses + a closed `VALID_TASKS = ("sft", "preference", "tool")` allowlist. `chunk_document` paragraph splitter has `_MAX_DOC_CHARS = 4 MiB` cap, null-byte rejection, and bool-as-int reject on `max_chunk_chars` (matches v0.30.0 `Candidate` policy). `_validate_float_unit` uses `math.isfinite` to reject NaN AND ±Inf BEFORE the `[0, 1]` bounds check (review fix HIGH — first-cut accepted `float("nan")` because `nan < 0.0` is `False`, silently disabling active-pruning; mirrors v0.32.0 `save_lr_finder_report` and v0.41.0 Part B `parse_lr_groups` policy). `discover_documents` does `is_under_cwd` containment (review fix MEDIUM — first-cut only enforced containment from `build_forge_plan`, leaving direct callers unprotected; matches v0.42.0 `discover_*` policy), rejects symlinked directories via `os.lstat + stat.S_ISLNK`, restricts to a closed `_DOC_EXTENSIONS = {.txt, .md, .json, .jsonl}` allowlist, caps at `_MAX_DOCS = 10_000`, and skips dotfiles. `synthesise_forge_rows` runs `chunk → judge(prompt) → score_uncertainty → ForgeRow` with judge-exception swallow at DEBUG (matches v0.33.0 #47 `CrossDocCollator` and v0.40.3 `judge_filter_pairs` policy — no silent crash on a single bad judge call). `write_forge_dataset` + `write_provenance` are atomic via `tempfile.mkstemp` + `os.replace` (mirrors v0.43.0 Part D `copy_bundle_to`); both writers call `_check_write_path` which enforces `is_under_cwd` + ≤4096-char cap + `os.lstat + stat.S_ISLNK` rejection at the target (TOCTOU defence — mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E / v0.46.0 policy); rejects non-`ForgeRow` elements (review fix MEDIUM — first-cut would `TypeError` mid-write leaving partial state). New `soup_cli/utils/data_score.py` ships `BENCHMARKS` `MappingProxyType` (6 names — mmlu / gsm8k / humaneval / truthfulqa / arc / hellaswag) and frozen `ScoreReport` with `languages` exposed as `MappingProxyType` to prevent caller mutation. `ngram_set` caps `n ∈ [1, 32]`, text at `_MAX_TEXT_CHARS = 1 MiB`, rejects bool-as-int. **ReDoS-hardened PII regexes** — `_PII_PATTERNS` rewritten in the security review (review fix HIGH × 2): phone pattern flattened to remove nested `(?:\+\d{1,3}[\s.-]?)?(?:\(?\d{2,4}\)?[\s.-]?)?` (catastrophic backtracking on near-miss inputs) → flat alternation with digit-count post-filter ≥ 7; credit-card rewritten from `\b(?:\d[ -]?){13,19}\b` (exponential backtracking on a 13-digit-space pattern ending with `x`) to anchored `\b\d{4}[\s\-]?\d{4}[\s\-]?\d{4}[\s\-]?\d{1,7}\b` with hard digit caps; `_PII_SCAN_CAP = 50_000` chars enforced on every input via `detect_pii` pre-truncation. `_require_str` rejects null bytes (review fix MEDIUM — first-cut only checked length, leaving the data_score validators inconsistent with the data_forge `_validate_str`). `_require_unit_float` uses `math.isfinite` to reject NaN AND ±Inf (matches `data_forge` policy). `decontaminate_rows` documents that it uses one-sided **containment** ratio `|inter| / |b|` rather than symmetric Jaccard (review fix MEDIUM — first-cut docstring said "Jaccard" but formula was containment; renamed for accuracy so future maintainers don't accidentally swap the denominator). `compute_scorecard` per-row `try/except ValueError` blocks now log at DEBUG (review fix MEDIUM — first-cut was silent `pass`, violating v0.33.0 #47 and v0.40.3 logging policy). `load_jsonl_rows` + `write_jsonl_rows` enforce `is_under_cwd` containment, `_MAX_FILE_BYTES = 1 GiB` cap, `_MAX_ROWS = 1_000_000` row cap, `os.lstat + stat.S_ISLNK` rejection on both input and output, atomic write via `tempfile.mkstemp` + `os.replace`. `_read_rows` and `_write_rows` CLI helpers have full `List[Mapping[str, Any]]` / `Iterable[Mapping[str, Any]]` type annotations (review fix HIGH — first-cut had `def _read_rows(path: str):` with no return type, breaking downstream type checking). `decontaminate_texts` parameter is `Optional[Mapping[str, Sequence[str]]] = None` (review fix HIGH — first-cut had `Mapping[...] = None  # type: ignore[assignment]` masking the type error). `import math` and `import tempfile` moved to module top-level (review fix MEDIUM — first-cut had stdlib imports inside function bodies, violating the project's lazy-import policy which applies only to heavy ML deps). New `soup_cli/commands/data_forge.py` collapses the duplicate `discover_documents` call (review fix MEDIUM — first-cut called the helper twice, introducing a TOCTOU window where `plan.num_docs` could disagree with the actual doc list). New `soup_cli/commands/data_score.py` `--benchmarks` allowlist-validated against `BENCHMARKS` keys with Rich-escaped error messages. Known limitations: (1) Live judge providers (Ollama / Anthropic / vLLM via `--judge-provider`) deferred to v0.47.1 — `soup data forge` ships with a deterministic offline echo stub; stub-then-live pattern matches v0.27.0 MII / v0.37.0 multipack / v0.46.0 Part A. (2) Decontamination benchmark corpora not bundled — `soup data decontaminate --benchmarks mmlu` validates the flag but operates on an empty corpus; operator-supplied `--benchmark-file` lands in v0.47.1. (3) Llama-Guard-3-1B toxicity classifier + FineWeb-Edu educational classifier + full Presidio PII + `langdetect` / `fastText` ship behind `[data-pro]` extras in v0.47.1. (4) Provenance manifest stores absolute realpath in `source_doc` (security review M4) — operators sharing manifests should redact paths; kept as-is for audit-trail completeness, mirrors v0.34.0 `crash.py` design tension between traceability and `$HOME` leak prevention.

- **v0.46.0 — Deploy & Agent Autopilot**: 2 release Parts ship a deploy-target picker and an Agent Forge for spec-driven tool-calling SFT datasets. New `soup_cli/utils/deploy_autopilot.py` ships a 10-profile `MappingProxyType`-wrapped catalog with `DeployProfile` `@dataclass(frozen=True)` and closed allowlists on `runtime` (transformers / vllm / sglang / mlx / ollama / lm-studio / executorch), `quant` (none / 4bit / 8bit / gptq / awq / fp8 / mxfp4 / hqq:Nbit), `peft` (lora / dora / qlora / full). `_make` factory rejects non-kebab-case names, bool-as-int on `recommended_max_length`, out-of-bounds `[64, 1_048_576]`, null-byte / >512-char description+notes. `render_recipe_yaml` rejects empty / null-byte / newline / >200-char `base` and >4096-char `output_dir` (defends against a crafted `--base "evil\ntraining: { epochs: 9999 }"` injecting YAML keys into the rendered recipe). `render_deploy_script` uses `shlex.quote` on `model_path` and rejects newline / NUL / >4096-char. `write_recipe` / `write_deploy_script` enforce `is_under_cwd` containment, ≤4096-char path cap, and `os.lstat + stat.S_ISLNK` rejection at the write target (TOCTOU defence — mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E policy). `soup deploy autopilot` panel passes every profile field through `rich.markup.escape` (matches v0.43.0 Part B `Tournament` policy — defends against markup injection if a future profile carries `[blink]` in its description). New `soup_cli/utils/agent_forge.py` parses OpenAPI 3.x / MCP server manifests / GraphQL introspection JSON into a canonical `Endpoint` frozen dataclass — every parser routes synthesised `path` through `_validate_path` (non-empty single-line NUL-free ≤1024 chars), so a manifest with `name="evil\nhost"` is rejected with a warning (review fix HIGH — the v0.46.0 first-cut stored the raw `f"mcp://{name}"` without validation, allowing newline injection into the `Endpoint.path` field). `$ref` strings in OpenAPI are left opaque (no external resolution — defends against file-read SSRF that a fully-resolving parser would expose). `_MAX_ENDPOINTS=10_000`, `_MAX_SPEC_BYTES=5*1024*1024`, `_MAX_ROWS_PER_ENDPOINT=32`, `_MAX_DESCRIPTION=512`. `load_spec_file` enforces `is_under_cwd` containment + `os.lstat + stat.S_ISLNK` rejection BEFORE `realpath` resolution (review fix MEDIUM — the v0.46.0 first-cut ordered `isfile(real)` before the lstat check, which followed the symlink) + 5 MiB cap + `yaml.safe_load` only (never `yaml.load`). `write_dataset` is atomic via `tempfile.mkstemp` + `os.replace` (review fix HIGH — replaces a v0.46.0 first-cut direct `open(real, "w")` loop that would leave a partial JSONL file on mid-stream `TypeError`; mirrors v0.43.0 Part D `copy_bundle_to` policy). Symlink rejection at the dataset target. `endpoint_to_rows` rejects bool / out-of-bounds `examples_per_endpoint` (∈ [1, 32]). New `soup_cli/commands/agent.py`: `synth` Rich table passes every cell through `rich.markup.escape` (review fix HIGH — defends against spec-controlled markup injection through `ep.path`); `train` validates `--base` and `--output-dir` for NUL / newline / >4096-char BEFORE embedding into the rendered YAML recipe string (review fix CRITICAL — defends against YAML key injection where `--base $'evil\\ntraining: { epochs: 9999 }'` would smuggle injected training keys); `eval` enforces predictions path `is_under_cwd` containment + `os.lstat + stat.S_ISLNK` rejection + `_MAX_PRED_LINES=1_000_000` DoS cap (review fix HIGH — v0.46.0 first-cut had no line cap, a multi-GB predictions file would have iterated unbounded). Known limitations: (1) Live Quant-Lobotomy auto-measure deferred to v0.46.1 — autopilot writes the canonical PEFT+quant combo per profile but does not yet measure OK/MINOR/MAJOR via v0.26.0 Quant-Lobotomy Checker. (2) ExecuTorch packaging deferred to v0.54.0 — `iphone-16` / `pixel-9` recipes are plan-only. (3) `soup agent train` is plan-only — prints the planned `soup train` invocation rather than re-entering Typer in-process (same design as v0.44.0 `soup quantize`). (4) `soup agent eval` is heuristic — scores tool-name match + arguments-key validity only; live RLVR `code_exec` sandbox scoring deferred to v0.46.1. (5) `$ref` resolution in OpenAPI specs is intentionally not done (file-read SSRF defence); users wanting full resolution should run `openapi-spec-validator` upstream. (6) MCP / GraphQL non-HTTP sentinel methods (`invoke` / `query` / `mutation`) are stored on `Endpoint.method` without going through `_validate_method` (documented design intent — `_HTTP_METHODS` covers HTTP-only).
- **v0.45.0 — Plugin System & Ecosystem Wins**: 5 release Parts ship the plugin / hook system + 4 ecosystem-integration schemas. New `soup_cli/plugins/__init__.py` registry: kebab-case plugin name regex `^[a-z0-9][a-z0-9\-]{0,39}$`, semver-ish version regex, `_MAX_PLUGINS=64`, `_MAX_TEMPLATES_PER_PLUGIN=32`, `_MAX_MODEL_GROUPS_PER_PLUGIN=32`, `_MAX_NAME_ENTRY_LEN=128`. Re-registering with a different `(version, plugin object, templates, model_groups, description)` is rejected (review fix HIGH — first-cut omitted `description` from the conflict check, allowing silent description override). `load_plugins()` swallows per-plugin exceptions at WARNING level so one broken plugin cannot crash the CLI. `soup plugins enable / disable` raise `KeyError` for unknown names (caller-distinguishable from validation errors). All user-controlled output passes through `rich.markup.escape`. New `utils/anthropic_messages.py` enforces `_MAX_MESSAGES=1024`, per-message `_MAX_CONTENT_LEN=1MiB`, `max_tokens` cap mirrors v0.30.0 inference-server `_MAX_TOKENS_CAP=16384`; `tool` role with structured (list) content is concatenated into a single `tool_result` text block instead of silently dropped (review fix MEDIUM — first-cut emitted `""` for any non-str content); `from_anthropic` uses `.get("role")` defensively after `validate_anthropic_payload` (review fix MEDIUM — `KeyError` surface guard). New `utils/server_tools.py` ships closed `SUPPORTED_TOOLS = frozenset({python, bash, web_search})` allowlist + `WebSearchConfig` with `domain_allowlist` (≤64 entries, default empty = deny all) + `rate_limit_per_minute ∈ [1, 600]`. `is_domain_allowed` strips `:port` suffix before matching (review fix MEDIUM — without this, `Host: api.example.com:443` would silently fail to match the bare `api.example.com` allowlist entry, causing legitimate denials) AND rejects IPv6 literals (`[::1]` → False — IPv6 should never match a domain allowlist). `validate_domain` rejects null-byte / whitespace / `/` / oversize. `python` and `bash` reuse the v0.25.0 RLVR sandbox (5s timeout, RLIMIT_AS/CPU on POSIX, ephemeral cwd, socket patch); live HTTP endpoints land in v0.45.1. New `utils/ngram_spec.py` `NgramSpecConfig` is `@dataclass(frozen=True)` with `n ∈ [1, 8]`, `num_draft_tokens ∈ [1, 32]`, `prompt_lookup_max ∈ [0, 1MiB]`; bool rejection on every numeric field (matches v0.30.0 `Candidate` / v0.34.0 `estimate_run_cost_usd` / v0.42.0 `make_preprocess_cache_key` policy). New `utils/integrations.py` 15-entry catalog wrapped in `MappingProxyType`; `IntegrationSpec` is frozen so the catalog cannot be mutated at runtime. New `utils/trainer_plugins.py` 6-entry allowlist (`MappingProxyType`); `validate_trainer_plugin_list` accepts `Sequence[str]` (review fix MEDIUM — first-cut annotated as `Tuple[str, ...]`) but explicitly rejects bare `str` argument so a caller passing `"grokfast"` directly doesn't iterate per-character. New `utils/recipe_dag.py` parses Data Recipe DAG with closed `NODE_KINDS = frozenset({seed, llm_text, code, judge, validator, sampler})`; Kahn's topological sort uses `collections.deque` + sorted-on-arrival (review fix HIGH — first-cut had O(N² log N) `queue.sort()` inside the BFS body); cycle / self-loop / dangling-edge / duplicate-edge / duplicate-node-name / unknown-kind rejection with specific error messages. `_MAX_NODES=256`, `_MAX_EDGES=1024`, `_MAX_FILE_BYTES=1MiB`. `parse_recipe_yaml` uses `yaml.safe_load`. `load_recipe_yaml` enforces `is_under_cwd` containment AND `os.lstat + stat.S_ISLNK` symlink rejection at the recipe path (review fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B policy; without this, a symlink under cwd pointing at `/etc/passwd` would resolve through `realpath` and load arbitrary files). `RecipeDAG` and `RecipeNode` are `@dataclass(frozen=True)`; `RecipeNode.config` is `MappingProxyType`-wrapped so caller mutation post-parse cannot affect the validated topology. `parse_recipe` does NOT mutate input dicts (matches v0.33.0 #47 `CrossDocCollator` immutability policy). Known limitations: (1) Plugin trainer-callback wiring deferred to v0.45.1 — `register_plugin` accepts hook objects and `discover_hooks` lists implementations, but no HF Trainer callback emits `pre_train` / `post_train` / `pre_step` / `post_step` yet. (2) Anthropic `/v1/messages` endpoint deferred to v0.45.1. (3) Server-side tool HTTP endpoints (python / bash / web_search) deferred to v0.45.1. (4) n-gram speculative-decoding live engine wiring deferred to v0.45.1. (5) External integrations are a catalog, not launchers — `IntegrationSpec` describes 15 ecosystem targets but no `soup deploy <target>` command auto-detects + invokes their CLIs (tracked for v0.46.0 Deploy Autopilot). (6) Advanced trainer plugins are an allowlist, not callbacks — `validate_trainer_plugin_list` accepts `[grokfast, spectrum, ...]` at schema validation, but no trainer wraps the upstream callbacks yet (live wiring per plugin in v0.45.1). (7) Data Recipe DAG runner deferred — `soup data recipe` validates topology and prints planned topological order; node-by-node execution against a local model is the v0.45.1 deliverable.
- **v0.44.0 — Live Dashboard & UX**: 21 features across 4 Parts. New `soup_cli/utils/qr_url.py` `build_phone_url` enforces a scheme allowlist (`http` / `https`), loopback-only HTTP (RFC1918 / link-local / cloud-metadata IPs implicitly rejected because `host not in _LOOPBACK_HOSTS` raises), IPv6 literal auto-bracketing per RFC 3986, and **token routing through the URL query string `?token=…`** (review fix HIGH — the v0.44.0 first-cut put it in the URL fragment `#token=…` which never reaches the server). Token shape regex `^[A-Za-z0-9_\-]{16,128}$` matches `secrets.token_urlsafe(N)` output. New `commands/fetch.py` enforces `is_under_cwd` on `--output` AND adds **`os.lstat + S_ISLNK` symlink rejection at the write target** regardless of `--force` (review fix HIGH — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 / v0.43.0 Part C policy). Bundled-source path is `os.path.commonpath`-checked against `fetch_examples_dir()` (review fix HIGH — defends against a future catalog entry containing `..` that would escape the bundled directory). `fetch_examples_dir()` uses `os.path.realpath` not `abspath` (review fix MED — symlinked installs resolve to the real package root, not the symlink's parent). New `utils/checkpoint_trigger.py` `write_trigger` adds `os.lstat + S_ISLNK` rejection on the trigger target (review fix MED — symlink at `<output_dir>/.checkpoint_now` could redirect the write to any user-writable path). Trigger path stays under cwd via shared `is_under_cwd`. New `commands/llama.py` proxies to llama.cpp binaries with **`_LLAMA_ENV_ALLOWLIST` frozenset child-env filter** (review fix MED) — `HF_TOKEN` / `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / all `SOUP_*` are dropped before exec; only `PATH` / `HOME` / `USER` / `USERPROFILE` / `TMP` / `TEMP` / locale + llama.cpp-recognised `LLAMA_CPP_HOME` / `GGML_*` / `OMP_NUM_THREADS` are forwarded. Subprocess uses list args (no shell). Closed allowlist `_SUBCOMMAND_TO_BINARY: MappingProxyType` rejects unknown subcommands; `_validate_arg` rejects null-byte / newline / carriage-return / >1024-char per arg; `_MAX_ARGS=64` cap. New `utils/onboarding.py` `render_onboarding_yaml` enforces `is_under_cwd` on the `output` field at render time (review fix MED — without this, a crafted answer dict like `output: ../../etc/cron.d/x` would silently propagate into the rendered YAML). Literal allowlist on `task` (8 entries) and `quantization` (`4bit` / `8bit` / `none`); `epochs ∈ [1, 10]`; null-byte rejection on every string with field-named error message. New `utils/sweep_config.py` enforces a strict scalar allowlist (`str` / `int` / `float` / `bool`) on every param value (review fix LOW — without this, `params: {lr: [{__class__: evil}]}` would smuggle non-scalar YAML through to runtime); `_MAX_FILE_BYTES=256KB`, `_MAX_PARAM_KEYS=32`, `_MAX_VALUES_PER_KEY=64`, `_MAX_PARAM_KEY_LEN=128`. `SweepSpec.params` is `MappingProxyType[str, Tuple[Any, ...]]` for genuine `frozen=True` immutability (review fix HIGH — `frozen=True` on a `Dict[str, List[Any]]` field only prevents reassignment, not mutation). Same fix applied to `ConsolidationPlan.shard_files: Tuple[str, ...]` and `DelinearizePlan.weight_files: Tuple[str, ...]`. New `utils/gpu_monitor.py` `parse_nvidia_smi_csv` rejects null-byte in GPU name field; subprocess uses list args + 5s timeout + `OSError`/`TimeoutExpired` swallowed (never raises). `detect_apple_silicon` rewritten to use clean `platform.system()` / `platform.machine()` checks (review fix HIGH — the v0.44.0 first-cut had a parser-priority bug `if X if Y else Z:` that produced the right answer by coincidence on every platform but would silently break on any future branch refactor). New `utils/sse_train_stream.py` `TrainEvent` validates `type` against a closed `_VALID_TYPES = frozenset({metric, status, log, eval})`, rejects non-finite `ts`, rejects null-byte / >1024-char `message`. `to_payload` filters output to a closed `_ALLOWED_KEYS` allowlist (defence-in-depth against payload key drift from refactors). `format_sse_frame` emits the standard W3C SSE wire format `data: {json}\n\n` with `ensure_ascii=False`. New `utils/tool_outputs.py` `ToolOutputsBuffer` is thread-safe (`threading.Lock`) backed by `collections.deque(maxlen=_MAX_RECORDS=1000)` (review fix MED — replaces an O(N) `self.records[-N:]` list slice that triggered GC pressure on overflow); validates name (no null-byte, ≤128 chars), bool rejection on `started_ts` / `duration_ms` (matches project bool-as-int policy), `success` must be exactly `bool` not `int`, output preview truncated to 4096 chars. `ToolCallTimer.__exit__` returns `False` explicitly per project readability policy (does not suppress exceptions). New `utils/llama_server_timings.py` `parse_timings` tolerates missing keys (defaults to None), clamps `kv_cache_pct` to `[0, 100]`, rejects bool / non-finite / negative ints in numeric coercion. `format_kv_bar` validates `width ∈ (0, 200]`. New `utils/tail_latency.py` enforces `MAX_SAMPLES=1_000_000` DoS cap on every iteration; bool rejection + non-finite rejection + alpha bounds. New `soup_cli/ui/plugins/__init__.py` registry: tab name regex `^[a-z0-9][a-z0-9\-]{0,30}$`, `_MAX_TABS=32` cap, `list_tabs()` returns `MappingProxyType` defensive view, `load_plugins()` swallows per-plugin exceptions at WARNING level so one broken plugin doesn't crash the whole UI. `_NAME_DISALLOWED` in `utils/shortcuts.py` is `frozenset` (review fix LOW — was mutable `set` at module level). New `utils/ui_env.py` host validator restricts to `[a-zA-Z0-9.-:]` ≤253 chars + null-byte rejection; port `[1, 65535]`; key length cap 256 + null-byte rejection. Known limitations: (1) Live monitoring SSE endpoint deferred — schema + frame formatter ship, FastAPI route in v0.44.1. (2) Phone visibility flag wiring (`soup ui --public --auth-token` + QR print at startup) deferred to v0.44.1. (3) `soup merge-sharded-fsdp-weights` and `soup delinearize-llama4` are plan-only — torch-side runtime in v0.44.1. (4) `soup llama <subcommand>` does not bundle llama.cpp; the binary must be on PATH (`shutil.which`) and an actionable `FileNotFoundError` fires when missing. (5) `soup serve --reasoning-parser <name>` allowlist ships, runtime hook in v0.44.1. (6) Apple Silicon `soup monitor` prints a yellow advisory and exits with `nvidia-smi unavailable` rc=1 — `powermetrics` parser in v0.44.1.
- **v0.43.0 — Tracker & Eval Pro**: 18 features across 4 Parts. New `soup_cli/utils/trackers.py` ships an immutable `_REPORT_TO_BACKENDS` `MappingProxyType` allowlist (`wandb` / `tensorboard` / `mlflow` / `swanlab` / `trackio` / `none`); `validate_tracker_name` rejects non-string / empty / null-byte / >32-char inputs. PostHog telemetry is opt-IN (default OFF) via `SOUP_TELEMETRY=1` env var; `build_telemetry_payload` schema is closed-key (soup_version / command / python major.minor / os / arch / duration) — no model names, dataset paths, or config contents leak in the payload. Live PostHog network code deferred to v0.43.1. New `--tracker` flag on `soup train` mutually exclusive with `--wandb` / `--tensorboard` via `resolve_report_to`. `tracker_backend = tracker` capture in `commands/train.py` defends against the Typer parameter being shadowed by the local `ExperimentTracker()` instance (review fix HIGH #1 — silent `--tracker` drop regression). `nlg_metrics.py` is pure-math: BLEU `_lcs_length` allocates a fresh row per outer iter (review fix HIGH #2 — first-cut had a double-buffer re-zero that destroyed the last row). `effective_tokens_per_second` returns `None` on non-positive wall_clock (no fabrication). `eval/calibrate.py` `CalibrationReport` is `@dataclass(frozen=True)` with `FrozenInstanceError` test (matches v0.32.0 / v0.39.0 / v0.41.0 frozen-dataclass policy); `run_calibration` caps prompts at 10_000 (DoS defence). `eval/arena.py` `Tournament` enforces 256-model cap, 1M-match cap, `MappingProxyType` view on `ratings` so external callers cannot mutate Elo state (LOW review fix). `_validate_model_name` rejects null-byte / >128-char AND Rich markup metacharacters `[` / `]` (security review fix LOW #5 — leaderboard markup-injection defence). `update_elo` rejects bool / non-finite / NaN `k`. `eval/benchmarks_v0_43.py` ships `NEW_BENCHMARKS_V0_43 = frozenset({"ceval","cmmlu","aider_polyglot"})` + `MappingProxyType`-wrapped metadata; live Aider Polyglot runner deferred to v0.43.1. `utils/profiling_v0_43.py` `resolve_snapshot_path` rejects null-byte / `.` / `..` / path separators in `run_id`; `base_dir` rejects null-byte / absolute / `..` segments before realpath join (security review fix HIGH #1) — closes a Windows short-name path escape. `memory_snapshot_context` narrows the `RuntimeError` catch to the `record()` entry call only (review fix HIGH #2 — wide catch would have triggered `generator already executing` on user-body RuntimeError); yields `None` when torch / CUDA / `_record_memory_history` API missing. `nccl_bandwidth_check` rejects bool / non-finite / negative `measured_gb_per_sec`; reference table is `MappingProxyType`-wrapped. `utils/vscode_setup.py` `build_launch_json` rejects null-byte / newline / >512-char `config_path` so a crafted argument cannot inject Python args into the generated JSON. `write_vscode_launch` uses `os.lstat + S_ISLNK` symlink rejection at the target path regardless of `force` (security review fix MEDIUM #3 — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 policy). `utils/demo_bundles.py` `copy_bundle_to` validates every JSONL line via `json.loads`, enforces 50 MB cap, stages writes to `<target>.tmp` with `os.lstat + S_ISLNK` rejection (security review fix HIGH #2) and `os.replace`-atomic rename on success — mid-stream rejection never leaves a partial file. New `soup data demo` Typer command: `--output` containment-checked via shared `is_under_cwd`. CLI output prints `_esc(written)` to defend against Rich markup injection through a crafted output path (review fix LOW). `_lcs_length` algorithmic correctness fix (review fix HIGH #2) + BLEU geometric-mean policy fix (returns 0 on any zero precision unless `smooth=True`) (review fix HIGH #1). All `math` imports moved to module-level (PEP 8 / project lazy-import policy: only torch / transformers / peft / trl / mlx are lazy). Known limitations: (1) MLflow / SwanLab / Trackio live integration deferred to v0.43.1 — schema accepts the names and threads into HF `TrainingArguments.report_to` but Soup does not bundle the upstream packages; HF Trainer raises a generic ImportError when the package is absent. (2) PostHog telemetry deferred — payload schema + `SOUP_TELEMETRY=1` env var ship, but no network code; `build_telemetry_payload` is purely a schema-locking stub (mirrors v0.27.0 MII / v0.37.0 multipack / v0.42.0 fsspec stub-then-live pattern). (3) Aider Polyglot live runner deferred — `aider_polyglot` is in the benchmark allowlist but live wiring requires the upstream `aider-chat` package + Docker. (4) `soup doctor --nccl` measurement CLI surface deferred — `nccl_bandwidth_check` accepts a measured value and classifies it, but does not yet measure via `torch.distributed.all_reduce`. (5) `examples/data/` fixtures shipped at the repo root (not as package data inside `soup_cli/`); `_bundle_source_path` derives `repo_root` via `str(importlib.resources.files("soup_cli"))` which is editable-install + wheel friendly but may break in zipapp / namespace-package installs.

- **v0.42.0 — Data Pipeline Pro**: closes the data-tooling gap with Axolotl + LlamaFactory in 18 features across 6 Parts. New `soup_cli/utils/data_pipeline.py` ships an immutable `_REMOTE_SCHEMES` `MappingProxyType` allowlist (s3 / gs / gcs / az / abfs / abfss / oci); `validate_remote_uri` rejects userinfo, fragments, AND query strings (the latter is SSRF-adjacent — fsspec backends interpret `?endpoint_url=…` as configuration overrides; a crafted `s3://bucket/x?endpoint_url=https://attacker.com` would bypass otherwise-trusted infrastructure). Bucket regex `^[a-zA-Z0-9][a-zA-Z0-9._\-]{0,62}$` (1-63 chars, leading alnum) matches RFC 3986 + S3/GCS naming. Path length cap 2048; null-byte rejection on every string-shaped input; bool rejected before int isinstance check on every numeric input (`buffer_size` / `shards` / `image_pixels` / `video_maxlen` / `num_datasets`). `parse_interleave` returns frozen `InterleaveSpec` dataclass; per-prob `math.isfinite` (rejects NaN AND ±inf, matches v0.32.0 / v0.41.0 policy); sum-to-1 ±1e-6; max 32 datasets. `validate_new_tokens` caps lists at 10_000 entries with no duplicates, per-token <= 256 chars, null-byte rejected; returns a defensive copy. `validate_prompt_strategy` enforces a `module.path:function_name` regex with per-side 128 char cap (total <= 260). `make_preprocess_cache_key` rejects bool / null-byte / non-string / non-positive on every parameter; uses `\x1f` unit-separator delimiter. `detect_ingest_format` rejects null-byte / empty / unsupported extensions. New schema validators on `DataConfig`: `video_dir` / `tokenized_path` use shared `is_under_cwd` containment so a crafted YAML like `data.video_dir: ../../etc` fails at config load (review-fix M1; distinct from `image_dir` / `audio_dir` legacy policy). `train_on_prompt` mutually exclusive with `train_on_responses_only` (matches v0.36.0 loss-mask exclusivity policy). `interleave` field has its own `field_validator` so `data.interleave: 99` fails at config load (review-fix L3) AND bare `interleave: probs` is rejected at schema load with the actionable "use {strategy: probs, probs: [...]} dict form" message instead of getting a confusing late error (review-fix HIGH #2). Image-pixel validator threads `info.field_name` so error messages name the actual field that tripped (review-fix M3). Bucket regex relaxed from `{1,62}` to `{0,62}` so 1-character bucket names (per S3/GCS spec) are no longer falsely rejected (review-fix HIGH #1). `_convert_video` validates `video` is non-empty string with null-byte rejection and 2048-char cap (review-fix H2). `_convert_prm` type-checks `prompt` (non-empty string) + every `completions[i]` (string) + every `labels[i]` (bool, not int — matches v0.30.0 `Candidate` policy) and caps at 10_000 steps. `_convert_multimodal` validates content-part `type` against the closed allowlist `{text, image, audio, video}`. New `soup data preprocess <config>` CLI: `--config` containment-checked via `is_under_cwd` BEFORE `load_config` (review-fix HIGH #1) so a crafted argument like `../../etc/passwd.yaml` cannot be opened. New `soup data ingest <file>` CLI: `os.lstat + stat.S_ISLNK` symlink rejection on the input file (TOCTOU defence, mirrors v0.33.0 #22 `prune_checkpoints` policy); `is_under_cwd` containment on input + output. Known limitations: (1) fsspec live loaders deferred to v0.42.1 (schema gate fires now, runtime ImportError advisory); (2) AOT preprocess live tokenize loop deferred to v0.42.1 (cache-key emit + path-plan only); (3) custom prompt-strategy runtime invocation deferred to v0.42.1 (regex shape validation only); (4) PRM live trainer wiring deferred to v0.50 per original roadmap; (5) markdown ingest does not split on headings (the docstring's heading-split claim was aspirational); (6) `_convert_video` / `_convert_multimodal` validate path/url null-byte and length but do NOT verify file existence (matches v0.40.5 `reward_model` policy — file-existence is the trainer's responsibility).
- **v0.41.0 — Optimizer & PEFT Zoo**: closes the optimizer-breadth gap with LlamaFactory + Axolotl. New `soup_cli/utils/optimizer_zoo.py` ships a closed `SUPPORTED_OPTIMIZERS` `frozenset` (HF-native + bnb-backed + 14 v0.41.0 additions: BAdam / APOLLO / Adam-mini / lomo / adalomo / grokadamw / schedule_free_adamw / schedule_free_sgd / muon / dion / came_pytorch / ao_adamw_{fp8,4bit,8bit}). `validate_optimizer_name` rejects non-string / empty / null-byte / >64-char inputs; lower-cases the name for deterministic lookup (matches v0.30.0 `pick_draft_model` policy). `_OPTIMIZER_PACKAGES` wrapped in `types.MappingProxyType` so the registry cannot be mutated at runtime (matches v0.36.0 `_REGISTRY` policy). `is_new_v0_41_optimizer` is non-string-safe (returns False rather than raising). New `soup_cli/utils/lr_groups.py` parses `training.lr_groups` accepting list-of-pairs / list-of-dicts / `{pattern: lr}` mapping; capped at `MAX_LR_GROUPS=32`; per-pattern non-empty string ≤256 chars + null-byte rejection + `re.compile` validation + best-effort ReDoS probe (`compiled.search("a"*128)` catches catastrophic-backtracking patterns); per-LR `(0.0, 1.0]` bounds + `math.isfinite` (rejects NaN AND `±inf`) + bool rejection (matches v0.30.0 `Candidate` policy). Duplicate patterns rejected. `LrGroup` is `@dataclass(frozen=True)` (matches v0.32.0 `SpikeRecoveryStrategy` policy). `lr_groups_from_schema` converts the canonical stored shape `List[Dict]` into runtime `List[LrGroup]` (closes the schema-to-runtime type gap that the code review caught). `build_optimizer_param_groups` rejects bool / non-positive `base_lr` at runtime (defence-in-depth). New `soup_cli/utils/loftq_init.py` exposes `validate_loftq_iter` (∈ [1, 10], bool rejected) and `validate_loftq_bits` (∈ {2, 4, 8}, bool rejected) with `build_loftq_config` lazy-importing `peft.LoftQConfig` (ImportError carries actionable `pip install --upgrade peft` hint). New `soup_cli/utils/block_expansion.py` validators reject bool on `expand_layers` / `freeze_trainable_layers` and use `field_validator(mode="before")` so Pydantic's `ge`/`le` does not silently coerce `True` to `1` (matches v0.30.0 / v0.34.0 / v0.36.0 / v0.40.6 bool-as-int hardening policy). `_count_layers` uses `hasattr(layers, "__len__")` instead of `try/except TypeError` so legitimate `__len__` bugs surface loudly. Schema `LoraConfig.init_strategy` Literal extended to `{"random", "pissa", "olora", "loftq"}`; cross-validator rejects loftq + use_dora / use_vera. `TrainingConfig.load_in_8bit` / `load_in_16bit` use `is True` policy (matches v0.34.0 / v0.39.0 / v0.40.6 `is None` over falsy guards) — explicit `False` is treated as "no preference" not as "off"; mutually-exclusive both-True rejected; combining alias=True with explicit Quant Menu format raises rather than silently overriding. The alias-driven `quantization` rewrite uses direct `self.quantization = ...` assignment (NOT `object.__setattr__` — code review caught that the latter would silently bypass any future field validator on `quantization`). `expand_model_blocks` raises `NotImplementedError` with a v0.41.1 marker on non-zero block counts (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA stub-then-live pattern). Known limitations: (1) LLaMA Pro live wiring deferred to v0.41.1; (2) Mixture-of-Depths (`use_mod=True`) live patch deferred to v0.41.1; (3) optimizer dependency check is advisory at trainer construction time, not at schema-load (CI environments often lack optional optimizer packages); (4) `load_in_8bit`/`load_in_16bit` rewrite `quantization` only when set to `True` — explicit `False` is intentionally a no-op so a YAML with `load_in_8bit: false` and `quantization: 4bit` still trains in 4-bit.
- **v0.40.6 — ReLoRA + surgical PEFT non-SFT**: closes the v0.39.0 known gap by extending the ReLoRA callback (v0.39.0 Part B) and the surgical PEFT patches (v0.39.0 Part D — Gemma4 `ClippableLinear` -> `nn.Linear` swap, fused-MoE 3-D expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO). New shared module `soup_cli/utils/peft_wiring.py` exposes three helpers: `apply_pre_lora_patches(model, base)` (Gemma4-gated, runs BEFORE `get_peft_model` so PEFT's `target_modules` matcher sees the swapped `nn.Linear`), `apply_post_lora_patches(model)` (3-D MoE expert dropout strip, runs AFTER LoRA injection — architecture-detected via `weight.ndim == 3` inside the helper, safe to call unconditionally), `attach_relora_callback(trainer, tcfg)` (returns `True/False`; uses `if relora_steps is None: return False` per project policy so a schema-bypassing caller passing `relora_steps=0` surfaces as a loud `ReLoRAPolicy` ValueError rather than a silent skip). SFT migrates to the same helpers in the same release (centralisation invariant): every trainer file calls only the helpers, eliminating the v0.39.0 inline copy in `sft.py`. `SoupConfig._validate_relora_supported_tasks` removes the `task != "sft"` rejection branch; MLX backend rejection retained with a distinct error message (callback is HF Trainer-specific). Source-level grep matrix in `tests/test_v0406_part_a.py` proves all 12 transformer-backend trainers (sft + 11 non-SFT) call `apply_pre_lora_patches` BEFORE `get_peft_model` BEFORE `apply_post_lora_patches`, plus behavioural unit tests for each helper (Gemma4 happy path + exception swallow, post-LoRA strip happy path + exception swallow, ReLoRA policy field forwarding, schema-gate matrix covering every transformer task plus the `preference` dispatcher with `preference_loss='dpo'`). Defence-in-depth carry-over: `peft_wiring` swallows broad `Exception` from each upstream patch at DEBUG level (matches v0.39.0 Part D best-effort design); `%s` formatting on `exc` (not `repr`) so `$HOME`-prefixed paths cannot leak (matches v0.34.0 `crash.py` redaction policy); the underlying `apply_gemma4_clippable_patch` and `strip_lora_dropout_for_3d_experts` already validate model_name (null bytes, length) and are duck-typed via v0.39.0 review fixes. Known limitations: (1) Multi-modal trainers (vision/audio paths in `sft.py`) inherit ReLoRA + surgical patches because they share the SFT trainer wrapper, but the surgical patches are best-effort (try/except DEBUG-logged) — a Gemma4 vision model is unlikely in practice; if encountered, the patch attempt may noisy-log without applying. (2) The schema gate now accepts every transformer-backend task with `relora_steps`, but real-world correctness on RLHF (PPO / RewardModel) is unverified — ReLoRA was originally validated on SFT/causal-LM training; rejection-sampling-style RL loops may interact unexpectedly with periodic LoRA pruning + optimizer reset. Tracked as a community QA item; the schema does not gate on this since the upstream paper does not preclude RL use. (3) `apply_post_lora_patches` swallows exceptions at DEBUG, consistent with the v0.39.0 best-effort design — a real PEFT-side breakage in `strip_lora_dropout_for_3d_experts` would silently no-op on a non-MoE model where the strip is also a no-op, so the silent fallback is acceptable.
- **v0.40.5 — Quant Menu non-SFT**: closes the v0.38.0 known gap by extending the seven Quant Menu formats (gptq / awq / hqq:Nbit / aqlm / eetq / mxfp4 / fp8) from SFT-only to all 11 transformer-backend trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO). `SoupConfig._validate_quant_menu_supported_tasks` removes the `task != "sft"` rejection branch; MLX backend rejection retained with distinct message; `modality != "text"` rejection retained for vision/audio (multi-modal Quant Menu is tracked for a follow-up). Each non-SFT `_setup_transformers` replaces its inline BNB-only branch with a call to `build_quantization_config_for_loader(tcfg=tcfg, base=cfg.base, console=console)` — same pattern as `sft.py:420-440`, no remaining `BitsAndBytesConfig(load_in_4bit=True ...)` literal in any non-SFT trainer (source-level invariant test in `tests/test_v0405_part_a.py`). The kbit-prep tuple is widened from `("4bit", "8bit")` to `("4bit", "8bit", "mxfp4")` so the BNB MXFP4 path runs through `prepare_model_for_kbit_training`. `_load_reward_model` (module-level helper in `ppo.py`) accepts an optional `tcfg=None` kwarg — when set, the reward model is loaded with the same Quant Menu config as the policy, defending against silent fp16 OOM on a GPTQ/AWQ/HQQ policy run. PPO call sites at `_create_reward_model` + `_setup_reward` both forward `tcfg=tcfg`. Defence-in-depth: new `TrainingConfig.reward_model` field validator rejects null bytes and caps length at 512 chars at config-load (matches the policy applied to `cfg.base`); the Quant Menu loader's per-call null-byte check in `_check_local_marker` remains as the runtime backstop. Known limitations: (1) vision/audio modality + Quant Menu still rejected by the modality gate — `_setup_vision_transformers` / `_setup_audio_transformers` retain inline `BitsAndBytesConfig` blocks because they need vision-specific kwargs the unified loader does not yet thread; (2) Autopilot's quantization picker still recommends only `4bit`/`8bit`/`none` — Quant Menu format awareness deferred; (3) `tcfg.reward_model` is null-byte and length-validated at schema load but not path-containment-checked (`is_under_cwd`) — consistent with how `cfg.base` is treated, both can be HF repo IDs or absolute local paths.
- **v0.40.4 — trust_remote_code multi-trainer + multipack live**: closes the v0.36.0 #63 known gap by extending the `--trust-remote-code` opt-in across every non-SFT trainer wrapper (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO + the unified `PreferenceTrainerWrapper` dispatcher) and the 5 standalone commands (`soup diff`, `soup export`, `soup merge`, `soup infer`, `soup data generate`). Pattern (15 sites): each `__init__` resolves once via `model_requires_trust_remote_code(config.base) or False` + `resolve_trust_remote_code(...)` and stores `self._trust_remote_code` — every `from_pretrained` call site now reads the resolved value (no remaining `trust_remote_code=True` literal in any trainer file; source-level invariant test in `tests/test_v0404_part_a.py`). `commands/train.py` no longer carries the v0.36.0 `sft_kwargs` split; `trust_remote_code` is part of the unified `trainer_kwargs` dict that flows to every trainer regardless of task. `_load_reward_model` (module-level helper in `ppo.py`) accepts a `trust_remote_code: bool` parameter and resolves internally — design intent is that the helper is independently safe to call from outside `PPOTrainerWrapper`. `PreferenceTrainerWrapper` dispatcher forwards the raw bool to the inner DPO/SimPO/ORPO/IPO/BCO wrapper kwargs at both `_build_inner` and `_build_multi_objective` sites; the resolver fires inside the inner wrapper at construction time. `_export_onnx` / `_export_tensorrt` / `_export_awq` / `_export_gptq` and `_merge_adapter` helpers all gain a `trust_remote_code: bool = False` parameter threaded from the Typer flag. Multipack live HF Trainer wiring (#65) lands via a new `get_train_dataloader` override on `make_multipack_trainer_class` that installs `MultipackBatchSampler(real_batches=False)` (yields flat `list[int]` per pack — DataLoader-compatible) as the DataLoader's `batch_sampler=`. The override forwards `args.dataloader_drop_last` / `dataloader_num_workers` / `dataloader_pin_memory` from `TrainingArguments`. `_get_train_sampler` override stays as a defensive no-op fallback that ALWAYS delegates to super (review-fix: a multipack `list[list[int]]` from this method would cause a shape mismatch if any HF eval / prediction loop bypasses `get_train_dataloader`). The state-presence guard switched from falsy (`not max_seq`) to explicit `is None` + `not lengths` (defensively rejects only-None and empty-list cases — non-positive ints already rejected upstream by `attach_multipack_state`). Falls back to `super().get_train_dataloader()` when state is missing OR when `train_dataset` is unset (defence-in-depth so the subclass remains safe to instantiate). Known limitations: (1) `multipack: true` requires the dataset to expose `input_ids` (preferred) or `length` per row — un-tokenized text-only datasets trigger the v0.40.3 all-zeros WARNING and the `MultipackBatchSampler` will reject the run. (2) The DataLoader override does NOT thread FSDP / DeepSpeed parallelism env hints from `super().get_train_dataloader()`, so distributed `multipack: true` runs are still untested under FSDP / ZeRO; tracked for v0.40.5+ paired with v0.42.0 multi-GPU work. (3) `_live_lr_sweep_from_config` in `commands/train.py` still hardcodes `trust_remote_code=False` for the LR sweep's internal model load — defensive but means `--find-lr` cannot consume custom-code models even with the user opt-in (defence-in-depth, not a bypass). (4) Each non-SFT trainer's `__init__` repeats the resolver block (10 sites) — code-quality refactor candidate (single shared helper) deferred to a future patch to keep the v0.40.4 diff focused on the gap closure.
- **v0.40.3 — Stub-to-live**: New `soup_cli/utils/batch_probe.py:make_cuda_probe_fn` builds a CUDA probe closure that runs ONE forward+backward+step on a synthetic batch per candidate; `model.zero_grad(set_to_none=True)` runs BEFORE forward (defends against the synthetic backward accumulating into the live training model's grad buffers — matches v0.35.0 #45 `benchmark_kernel_combos` policy); intermediate `ids/attn/labels/outputs` are `del`-ed before `loss.backward()` so peak VRAM reflects a realistic training step; `bool` rejected on `batch_size` and `max_length`; `max_length < 8` rejected; `torch.cuda.OutOfMemoryError` returns False, other exceptions propagate; returns `None` (no-op) on non-CUDA / no-torch / missing model or tokenizer. New `soup_cli/utils/multipack_trainer.py:make_multipack_trainer_class` is `lru_cache`d so two calls with the same `base_cls` return the same subclass (consistent `isinstance`, pickle-safe); `attach_multipack_state` rejects `bool` on `max_seq_len`/`batch_size`/`seed` and rejects empty `lengths`; `lengths_from_dataset` logs WARNING when every row produces 0 (loud-fail mirrors v0.37.0 multipack arch allowlist — prevents silent NaN-loss footgun); `_get_train_sampler` override accepts `*args, **kwargs` for HF >= 4.41 signature compat. **Live wiring of the sampler into SFT / Pretrain trainer wrappers is deferred to v0.40.4** — adversarial 5th-pass review surfaced a `Sampler[int]` vs `list[list[int]]` shape mismatch with HF Trainer's DataLoader; the wrappers currently print a yellow advisory and fall back to the standard sampler when `multipack: true`. New `soup_cli/data/traces/quality.py:judge_filter_pairs` reuses v0.19.0 `JudgeEvaluator` SSRF protections; threshold rejects `bool` / NaN / out-of-`[0,1]`; `_MAX_BATCH=100_000` cap applied via lazy `itertools.islice` (never fully materialises a malicious / pathological generator); per-pair backend exceptions caught and logged at DEBUG (matches v0.33.0 #47 `CrossDocCollator` policy — never silently crash the harvest); `judge_provider` validated against `VALID_PROVIDERS` allowlist at the CLI boundary BEFORE constructor, with a Rich-escape error on mismatch. New `soup_cli/monitoring/trace_logger.py:TraceLogWriter` is thread-safe (single `threading.Lock` — multi-worker `--workers 4` documented as a single-process limitation); path containment via shared `is_under_cwd`; null-byte / empty / non-string path rejected; `cap_mb` bounds `[1, 10000]` with explicit `bool` rejection; rotation: when `current + extra > cap_bytes`, rename to `<path>.1` (one backup retained); symlink at the backup path is rejected via `os.lstat + stat.S_ISLNK` (matches v0.33.0 #22 TOCTOU policy) — defends against pre-placed `<log>.1 -> /etc/cron.d/x` overwrite. Secret redaction: prompt + response strings passed through `_SECRET_RE` matching `hf_*` (≥8), `sk-*` (≥16), and `Bearer …` (≥8) — replaces matches with `<redacted>` before serialisation (mirrors v0.34.0 `crash.py` policy). `--trace-log` constructor error messages in `commands/serve.py` are `rich.markup.escape`d before printing so a crafted path name cannot inject Rich markup. Unserialisable entries dropped silently; disk-full / OSError on write never crashes the request handler (passive log). Known limitations: live CUDA probe is wired in SFT only; multipack live wiring covers SFT+Pretrain only; `TraceLogWriter` retains exactly ONE backup file (operators wanting longer retention should use external rotation); custom HF Space templates from v0.40.2 still always create the Space with `space_sdk="gradio"` (tracked for v0.40.4+).
- **v0.40.2 — Quick polish + carry-overs**: New `soup_cli/utils/hf_space.py:render_custom_template_dir` enforces `is_under_cwd` containment on the template directory; `validate_repo_id` runs BEFORE `{MODEL_REPO}` substitution (matches v0.29.0 Part F policy); per-file 256 KB cap (matches v0.39.0 Part E template-size policy); only `app.py` / `README.md` / `requirements.txt` are read (closed allowlist — no path-from-user-data). Symlinks rejected via `os.lstat + stat.S_ISLNK` and non-regular files (FIFO / device) also rejected (matches v0.33.0 #22 prune_checkpoints TOCTOU policy) — defends against `<template_dir>/app.py -> /etc/passwd`. `_find_highest_local_checkpoint` reads `output_dir` after caller's `is_under_cwd` validation (in `prepare_hf_resume`) and silently drops non-directories + OSError. `prepare_hf_resume` skips the snapshot download when local `checkpoint-N >= remote checkpoint-N` (saves bandwidth and never overwrites a fresher local checkpoint). `commands/data.py:register_data` containment switched from `Path.resolve() + relative_to()` to shared `is_under_cwd` (Windows 8.3 short-name safety per CLAUDE.md project rule); same fix applied to `commands/bench.py` prompts-file containment. `commands/infer.py:--output` now containment-checked via `is_under_cwd` (late-evaluated after model+input validation so pre-existing `tmp_path` test contracts keep working). `commands/quickstart.py:--output` validates target dir via `is_under_cwd` before `mkdir(parents=True)`; rejects out-of-cwd targets with friendly message. `commands/runs.py:_filter_runs_by_cwd` uses `os.path.realpath + commonpath`, catches `(ValueError, OSError)` so cross-drive paths on Windows (`D:\runs` vs `C:\project`) drop silently rather than crash. `monitoring/display.py:format_gate_row` uses explicit `task.get("passed") is True` so a missing `"passed"` field renders neutrally instead of as a false-y red ✗. `commands/infer.py:_resolve_model_source` heuristic for HF-id-vs-local-path: only falls through to HF when value is NOT path-like (no `./`, `/`, `\\`, `~`, no Windows drive letter, non-empty); path-like-but-missing raises `FileNotFoundError` so users see actionable errors instead of confusing HF download attempts. Known limitation: custom HF Space templates always create the Space with `space_sdk="gradio"` regardless of the supplied `app.py` (no `--sdk` flag in this release; combine `--template streamlit-chat` with the inline registry for Streamlit Spaces). Tracked for v0.40.3+.
- **v0.40.1 — QA Hardening**: `soup_cli/utils/encoding.force_utf8_stdio` reconfigures Windows stdout/stderr to UTF-8 before any Rich Console is constructed; `os.environ.setdefault("PYTHONIOENCODING", "utf-8")` preserves user override; `(OSError, ValueError, AttributeError)` swallowed on detached streams; POSIX no-op. `SoupConfig._remap_root_level_misplaced_keys` (model_validator, mode='before') migrates root-level `lora:` into `training.lora` so nested validators (including `lora.init_strategy: Literal["random","pissa","olora"]`) actually fire — closes a footgun where the misplaced key was silently dropped. Caller's dict is never mutated (shallow-copy policy mirroring v0.33.0 #47 / v0.40.0 Part B). `PreferenceTrainerWrapper._build_multi_objective` replaces the v0.40.0 `NotImplementedError` stub with a primary-loss approximation; `validate_weight_compat` rejects BCO mixed with paired losses at runtime (data-format incompatible). `combine_losses` rejects empty weights, propagates NaN loudly (no silent zeroing), and rejects `bool` weight values (matches v0.30.0 `Candidate` policy). `_probe_cache_param_count` rejects empty / null-byte model names before path construction (mirrors v0.26.0 registry / v0.39.0 ReLoRAPolicy policy). `commands/doctor` flags `transformers ≥ 5.0.0` as INCOMPATIBLE via `_MAX_EXCLUSIVE` table; `_version_ge` parses leading-int chunks so `5.0.0.dev0` correctly trips the cap. `_detect_gpu_hw_without_torch_cuda` calls `nvidia-smi` via argv list (no shell), 5s timeout, `OSError` / `TimeoutExpired` caught; GPU label from `nvidia-smi` stdout is `rich.markup.escape`d before embedding in Rich-markup string (a real GPU name like `NVIDIA Quadro [T4]` cannot break or inject markup). `_detect_dual_python_interpreters` uses `os.path.realpath` (not `Path.resolve()`) for Windows 8.3 short-name compat. `_pick_quickstart_model` swaps TinyLlama-1.1B → SmolLM2-135M when `total_memory ≤ 6 GB` (prevents step-0 OOM on RTX 3050 4 GB / similar). `_live_lr_sweep_from_config` switched broken `load_local` import to `load_raw_data` (previously always silently fell back to a static placeholder curve). `commands/migrate` rejects `.jsonl` input (with first-line `{` sniff) with exit-2 friendly error; `.jsonl`-only suffix gate prevents false-positives on `.ipynb` notebooks. `commands/eval custom -o` is now honored independently of `--attach-to-registry`; loop-shadow regression where `output = generate_fn(...)` overwrote the CLI option fixed (variable renamed to `response`). `_load_jsonl` switched from `utf-8` to `utf-8-sig` so PowerShell `Out-File -Encoding utf8`-produced JSONL no longer fails first-row parse. Known limitation: `--trust-remote-code` opt-in surface still excludes 10 non-SFT trainers + 5 commands (v0.36.0 #63 carry-over).
- **v0.40.0 — Preference Variety**: New `task='bco'` (Binary Classifier Optimization) and `task='preference'` (unified dispatcher). New schema fields: `bco_beta` (gt=0), `preference_loss: Literal[dpo,simpo,orpo,ipo,bco]|None`, `preference_loss_weights: Optional[Dict[str,float]]`, `dpo_beta_schedule: Literal[linear,cosine,exponential]|None`, `dpo_beta_end: float, gt=0|None`, `dpo_ref_regen_epochs: int [1,1000]|None`. Cross-validators: `_validate_preference_dispatcher` rejects setting either `preference_loss` or `preference_loss_weights` outside `task='preference'` (closes ordering-dependency between Part B/D validators); `_validate_dpo_variants_supported_tasks` gates β-schedule + ref-regen to DPO-family tasks (`dpo`, `ipo`, or `preference` + `preference_loss in {dpo, ipo}`); rejected on mlx backend with distinct error message (matches v0.34.0 distinct-reason policy); `_validate_preference_loss_weights` enforces 2–5 entries (single-entry rejected with actionable message pointing at scalar `preference_loss`), key allowlist `{dpo, simpo, orpo, ipo, bco}`, explicit null-byte rejection on keys (matches v0.39.0 rank_pattern policy), per-value bounds `(0, 1]`, weights must sum to 1.0 (±1e-6), mutually exclusive with scalar `preference_loss`, rejected on mlx backend. `compute_beta_at_step` rejects `bool` on `step` and `total_steps` (project bool-as-int policy from v0.30.0). `BetaScheduleCallback` resolves `total_steps` lazily in `on_train_begin` so the schedule sees the real `state.max_steps` populated by HF Trainer (closes a first-cut silent-no-op bug where total_steps=0 emitted beta_end for every step). `RefModelRegenCallback._regenerate` uses `strict=True` on `load_state_dict` and logs at WARNING on mismatch (closes a first-cut silent partial-copy hazard where strict=False could produce a hybrid old-base + new-LoRA reference); epoch 0 regen suppressed (avoids copying untrained student); trainer `.beta` assignment swallow narrowed to `AttributeError` only. `PreferenceTrainerWrapper._make_inner_cfg` uses `model_copy` (not `model_dump`+`model_validate`) so re-validation never sees an inconsistent intermediate state and the caller's `cfg` is never mutated (mirrors v0.33.0 #47 immutability policy). `_split_dpo_rows_to_bco` skipped-row count emitted at DEBUG so production silent-degradation is inspectable (mirrors v0.33.0 #47 CrossDocCollator policy). Multi-objective live runtime weighted-loss combination is deferred to v0.40.1: `PreferenceTrainerWrapper.setup` raises `NotImplementedError` with a friendly message naming the deferred-version follow-up (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA stub-then-live pattern). Known limitation: `BCOTrainerWrapper._setup_transformers` still hardcodes `trust_remote_code=True` (v0.36.0 #63 known-gap family carry-over across non-SFT trainers).
- **v0.39.0 — LoRA Quality**: `LoraConfig.init_strategy: Literal["random","pissa","olora"]` rejects unknown strategies; PiSSA + DoRA / VeRA combinations rejected at config-load. `model_validator(mode="before")` aligns `use_olora=True` → `init_strategy="olora"` via dict-copy (no caller mutation; matches v0.33.0 #47 immutability policy). `rank_pattern`/`alpha_pattern: Optional[Dict[str, int]]` capped at 256 keys × value (0, 1024], rejects `bool` (subclass of `int` — matches v0.30.0 `Candidate` policy), null bytes in keys, empty keys; cross-validator rejects with `use_vera=True`. `ReLoRAPolicy` is `@dataclass(frozen=True)` (post-construction mutation raises `FrozenInstanceError`); bounds: `steps ∈ [1, 1e7]`, `warmup_ratio ∈ [0, 1]`, `prune_ratio ∈ (0, 1)` (strict — prevents zero-everything footgun). `magnitude_prune_tensor` strict `0 < prune_ratio < 1` rejection, non-Tensor input raises `TypeError`, empty / single-element tensor short-circuits (avoids `kthvalue(_, 0)` runtime crash). `_validate_relora_supported_tasks` cross-validator rejects `relora_steps` with `task != "sft"` and `backend=mlx` with distinct error messages (matches v0.34.0 distinct-reason policy); multi-trainer expansion deferred to v0.39.1. `is_gemma4_model` uses a word-boundary regex (`(?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$)`) so `"ungemma4ed"` / `"my-gemma4ish"` no longer over-match; null-byte rejection on `model_name`. `apply_gemma4_clippable_patch` weight-copy fallback logs at DEBUG instead of silent random-init; the patch is gated by `is_gemma4_model(cfg.base)` in `sft.py` before invocation so non-Gemma4 trainings never traverse the module tree. `apply_surgical_patches` rejects empty / null-byte `model_name` with `ValueError`. `templates/load_template` containment: filename re-validated via `_validate_name` (rejects `..`/`/`/`\\`/null/empty); `os.path.realpath + os.path.commonpath` containment check on the resolved path against `_templates_dir()` so a tampered `manifest.json` cannot read files outside the package directory (mirrors v0.26.0 registry policy). Tampered-manifest `ValueError` from `_validate_name` caught and falls back to inline (no propagating exception). 256 KB file-size cap. Inline `TEMPLATES` carries an explicit deprecation comment pointing at the canonical YAML registry; `tests/test_templates_yaml.py` asserts byte-equality of all 16 inline ↔ YAML pairs to prevent silent drift. Planned removal: v0.41.0+.
- **v0.38.0 — Quant Menu**: `TrainingConfig.quantization` Literal extended with `gptq` / `awq` / `hqq:1bit`..`hqq:8bit` (no `hqq:7bit` — HQQ doesn't support it) / `aqlm` / `eetq` / `mxfp4` / `fp8`; Pydantic rejects every other string at config-load. `validate_gptq_checkpoint` and `validate_awq_checkpoint` probe local paths for `quantize_config.json` / `quant_config.json`; HF repo IDs fall through; null-byte rejection + non-string `TypeError` on the ref. `_validate_prequantized_no_qat` rejects every pre-quantized format combined with `quantization_aware` (int8 QAT or `'fp8'`) — pre-quantized weights carry their own scale and QAT/FP8 prepare would silently corrupt them (mirrors LlamaFactory `quantization.py:117/199/211`). `_validate_bnb_quant_storage_only_with_4bit` rejects `bnb_4bit_quant_storage` on every non-BNB-4bit format (silent no-op otherwise); allowed dtypes: `Literal["uint8", "float16", "bfloat16", "float32"]`. `_validate_quant_menu_supported_tasks` restricts the new formats to `task='sft'` on `backend='transformers'` in v0.38.0 with distinct MLX-backend vs unsupported-task error messages (matches v0.34.0 distinct-reason policy). `check_quant_distributed_compat` hard-fails HQQ/EETQ/AQLM × {FSDP, ZeRO-3} (sourced from LlamaFactory `quantization.py:199/211` plus AQLM dequant constraints); warning-tier (not error) for BNB-4bit + FSDP without `bnb_4bit_quant_storage` so users see the silent perf cliff; unknown `quantization` raises `ValueError` (no silent pass) and the check is wired into `commands/train.py` startup. `parse_hqq_bits` rejects unsupported bit-rates and malformed `hqq:` strings before any kernel build.
- **v0.37.0 — Multipack**: `validate_multipack_architecture` raises `ValueError` on unknown arch (loud-fail vs Axolotl's silent-miss footgun); 18-arch frozen allowlist (Llama 3.x / Qwen 2/3 / Mistral / Gemma 2/3 / Phi 3/4 / DeepSeek V2/V3 / Mixtral / Falcon / StableLM / SmolLM2). FFD packer caps at `_MAX_FFD_ITEMS=1_000_000` (algorithm is O(N²) worst-case — defence against adversarial dataset DoS); `bool` rejection on every numeric input (`max_len`, per-element `lengths`, `batch_max_len`, `batch_size`, `seed`, `max_seq_length`) matches v0.30.0+ `Candidate` policy; generator-input materialisation prevents silent empty-bin output when validation exhausts the iterator. `MultipackBatchSampler` rejects empty `lengths`, non-positive `batch_max_len`/`batch_size`, items larger than `batch_max_len`. `build_multipack_sampler_for_lengths` rejects `tcfg.batch_size="auto"` with actionable message (must be resolved upstream). `_validate_multipack_packing_exclusive` cross-validator on `TrainingConfig` rejects both `multipack` and `packing` set; `_validate_multipack_supported_tasks` on `SoupConfig` restricts multipack to `sft`/`pretrain` on `transformers` backend with distinct error messages for MLX backend vs unsupported task (matches v0.34.0 distinct-reason policy). `build_4d_attention_mask` caps allocations at `_MAX_MASK_ELEMENTS=2³¹` cells (~8GB float32) — defence against `max_length=1M` × `batch_size=8` OOM; `tag_sub_sequences` capped at `_MAX_BOUNDARY_SEGMENTS=1_000_000`. Mask builder rejects non-floating dtypes (was silent `np.finfo` ValueError), non-2D `seq_pos_ids`, negative segment IDs; padding (id=0) tokens are fully masked, including diagonal, so softmax is well-defined. `select_packing_strategy` rejects non-bool `flash_attn_available`. `JinjaTemplateAnalyzer` parses chat templates via `Environment.parse` only — never renders, so a crafted soup.yaml cannot trigger SSRF / filesystem reads; 128KB template cap, null-byte rejection, `TemplateSyntaxError` re-raised as `ValueError`. `DEFAULT_MESSAGE_FIELDS` is a `frozenset` (runtime-immutable); `JinjaTemplateAnalyzer.message_fields` returns a defensive copy.
- **v0.36.0 — Correctness First**: `--trust-remote-code` opt-in replaces 9 unconditional `trust_remote_code=True` call sites across `soup train` / `chat` / `serve` / `data download` / `eval auto`; `KNOWN_SAFE_PREFIXES` allowlist (15 first-party orgs) suppresses warning panel for trusted repos; `model_requires_trust_remote_code` probes local `config.json` for `auto_map` (HF Hub repo IDs return `None`/unknown — HF still raises loudly when custom code is actually needed); `resolve_trust_remote_code` raises `ValueError` with actionable message when model needs custom code but the user did not opt in. Chat-template hardening: `DataConfig.chat_template` validator rejects null bytes, oversize (>64KB), AND filesystem-touching Jinja directives (`{% include %}`, `{% import %}`, `{% from %}`, `{% macro %}`, `{% extends %}` — both whitespace-control variants); empty string normalised to `None`; `_REGISTRY` wrapped in `MappingProxyType` (callers cannot mutate); `apply_chat_template_override` emits yellow advisory when active so users know `soup push` will persist the override into `tokenizer_config.json`. SFT silent f-string fallback (`f"{role}: {content}"`) replaced with hard `ValueError` — produced wrong loss labels for years on tokenizers without a chat template. Loss-mask fallback passes `add_special_tokens=False` to incremental tokenize calls so HF cannot double-prepend BOS at front of each render (consistent prefix-delta walk); narrowed exception catch in `_apply_template_with_mask` from `(TypeError, ValueError)` to `TypeError` only so a malformed messages list propagates instead of falling through to the loose path. `SOUP_BATCH_CACHE_PATH` env override containment-checked via `os.path.realpath + commonpath` against `~`/cwd/`tempfile.gettempdir()`; out-of-bounds values fall through to safe default; cache file gets best-effort `0o600` perms after atomic rename (matches v0.26.0 registry.db policy). `make_cache_key` rejects `bool` in numeric inputs (matches v0.30.0 `Candidate` policy). Documented limitation: non-SFT trainers (DPO/GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Pretrain/Embedding) and `commands/{diff,export,merge,infer,generate}.py` still hardcode `trust_remote_code=True` — v0.36.x patch follow-up.

## Security Scanning

- All code is scanned with `ruff` for style and common issues
- Dependencies are regularly updated to patch known CVEs
- GitHub's dependency scanning alerts us to vulnerable dependencies
- We use GitHub Actions CI/CD for continuous integration

## Dependency Updates

We actively monitor and update dependencies:

- Major dependency updates: Tested in PR before merging
- Security patches: Applied immediately and released as patch versions
- Deprecated dependencies: Replaced proactively

## Coming Soon

- [x] Automated dependency scanning
- [ ] SBOM (Software Bill of Materials) for each release
- [ ] Third-party security audit (after 1.0.0 release)

## Questions?

If you have security questions (not vulnerability reports) or need clarification:

- Open a GitHub Discussion tagged `security`
- Open a [GitHub Issue](https://github.com/MakazhanAlpamys/Soup/issues) (non-vulnerability inquiries)
- Check our [CONTRIBUTING.md](CONTRIBUTING.md) for general support

## License

This Security Policy is provided under the Apache-2.0 license, same as the Soup project.

---

**Last Updated**: April 2026

For the latest version of this policy, visit: https://github.com/MakazhanAlpamys/Soup/blob/main/SECURITY.md