32 KiB

Raw Blame History

Security Policy

Supported Versions

We provide security updates for the following versions:

Latest minor version: Active support (e.g., v0.20.x)
Previous minor versions: Bug-fix support only
Versions older than 3 minor versions: No support

Example:

v0.39.0-0.39.x -- Full support (latest)
v0.38.0-0.38.x -- Bug-fix support only
v0.37.x and below -- No support

Reporting a Vulnerability

Do not open a public issue or pull request for security vulnerabilities.

Instead, use GitHub Security Advisories to report privately, or email vpn.alpamys@gmail.com with:

Description: A clear explanation of the vulnerability
Steps to Reproduce: How to trigger or demonstrate the issue
Affected Versions: Which Soup versions are impacted
Suggested Fix (optional): Any proposed solutions
Contact Info: Your email for follow-up (optional)

What to Include

To: vpn.alpamys@gmail.com
Subject: Security Vulnerability Report: [Brief Title]

Description:
[Explain the vulnerability in detail]

Affected Component:
[e.g., data/loader.py, trainer/sft.py, etc.]

Steps to Reproduce:
1. [Step 1]
2. [Step 2]
3. ...

Impact:
[What could go wrong? Data exposure? RCE? DoS?]

Suggested Fix (optional):
[Your proposed solution, if any]

Response Timeline

Initial Response: Within 48 hours
Assessment: 1-3 business days
Fix Development: Varies by severity
Patch Release: As soon as possible after fix verification
Public Disclosure: Coordinated with reporter (typically 90 days after patch release)

Severity Levels

Critical: Remote code execution, data exposure, complete compromise (patch within 24-48 hours)
High: Authentication bypass, privilege escalation, denial of service (patch within 1 week)
Medium: Information disclosure, partial compromise (patch within 2 weeks)
Low: Minor issues with limited impact (patch in next regular release)

Security Best Practices

When using Soup, follow these practices to stay secure:

1. Keep Soup Updated

pip install --upgrade soup-cli

2. Protect API Keys

Never commit API keys or secrets to version control. Use environment variables:

export HUGGINGFACE_TOKEN=your_token_here
export WANDB_API_KEY=your_key_here
soup train

3. Validate Data

Only use trusted datasets
Verify checksums for large datasets
Inspect data for malicious content before training

4. Model Permissions

Be cautious when downloading models from untrusted sources
Use model hub providers with verified publishers (HuggingFace, Meta, etc.)
Keep track of which models you've fine-tuned and their base model sources

5. GPU/Compute Safety

Run on isolated machines if training on sensitive data
Clear cache and temporary files after training
Don't share fine-tuned models containing sensitive information

Known Vulnerabilities

We maintain a log of known security issues and their fixes. This will be updated as issues are discovered and resolved.

Current Status

No known critical vulnerabilities in current releases.

Security Hardening History

v0.10.10: Bearer token auth on Web UI, CORS restrictions, path traversal protection, SSRF prevention, max_tokens limits, supply-chain pinning (llama.cpp b5270), deprecated CLI secret flags
v0.12.0: experiment_name path traversal validation, GaLore parameter type enforcement
v0.13.0: Batch inference max_tokens capped at 16384, trust_remote_code warning
v0.14.0: Plaintext loader UTF-8 encoding, MoE config validation (moe_aux_loss_coeff ge=0, moe_lora boolean only)
v0.14.3: Data validate auto-detects format, Web UI --show-token flag + auth token documented
v0.15.0: rope_scaling_type Literal constraint, max_length bounds (ge=64, le=1048576), FSDP config key allowlist, Liger Kernel exception handling narrowed
v0.16.0: embedding_loss Literal constraint, embedding_margin gt=0 validation, ONNX export without trust_remote_code (with warning), TensorRT export subprocess list args (no shell injection), speculative decoding SSRF-protected (URL blocked) with warning panel, vLLM speculative model URL validation
v0.17.0: Server data generation provider SSRF validation (scheme whitelist + localhost-only HTTP), audio model trust_remote_code warning panel, audio file path traversal protection (resolved paths confined to audio_dir), SGLang backend trust_remote_code warning panel
v0.18.0: Ollama deploy GGUF path traversal protection + .gguf extension validation, model name validation (no path separators/null bytes), subprocess list args (no shell injection), Modelfile parameter key allowlist + value newline/null sanitization, overwrite warning panel
v0.19.0: Custom eval JSONL schema validation + 10k task cap, regex scoring ReDoS guard, judge API SSRF protection + API key isolation, human eval local-only terminal UI + 10k prompt cap, leaderboard read-only SQLite queries
v0.20.0: Ollama provider localhost-only validation (remote blocked), Anthropic provider API key from env only (never CLI arg), vLLM provider SSRF protection (scheme whitelist + localhost-only HTTP), output path traversal protection (.. blocked), configurable rate limiting (--requests-per-minute)
v0.21.0: Migrate input/output path traversal protection (resolve + relative_to(cwd)), Unsloth .ipynb AST-only parsing (no exec/eval), recipes output path traversal protection, NEFTune config bounded (ge=0.0, le=50.0)
v0.22.0: Multi-adapter serving path traversal protection (resolve + relative_to(cwd)), adapter name validation (alphanumeric + hyphens only), unknown adapter returns 404 (not 500)
v0.23.0: AWQ/GPTQ calibration data path traversal protection (resolve + relative_to(cwd)), AWQ/GPTQ output path stays under cwd, curriculum_buckets bounded (ge=1, le=20), AWQ/GPTQ trust_remote_code warning panel
v0.24.0: HF download trust_remote_code=False + warning panel, HF download output path sanitized (Path.name), download --samples capped at 1M, dataset registry name validation (no path separators/null bytes), registry path traversal protection, loss_watchdog threshold le=100 + patience le=1000, freeze_layers le=1000
v0.24.1: AWQ/GPTQ output path traversal validation moved before import check (previously unreachable when autoawq/auto-gptq not installed), Windows Unicode fix for Rich console output (replaced non-ASCII symbols with ASCII equivalents)
v0.24.2: Chat proxy SSRF protection (localhost-only HTTP, HTTPS for remote), chat proxy max_tokens capped at 16384 + temperature/top_p bounded, chat proxy Bearer token auth required, XSS prevention (HTML-escape before markdown render), runs compare max 5 runs, config from-form validates via load_config_from_string, SSE read endpoints no auth (GET)
v0.25.0: Tool-calling JSON-only parsing (no eval), RLVR math_verify regex-extracted numerics (no eval), code_exec 5s timeout + 512MB RLIMIT on POSIX + ephemeral cwd + socket patch + python -I -S + 10KB output cap, verifiable_domain Literal constraint, LoRA PEFT mutual exclusion (DoRA/VeRA/OLoRA), data augment path containment + caps, forgetting_detection bounds, checkpoint_intelligence bounds + symlink refusal, autopilot path containment (realpath + commonpath) + goal Literal + GPU/time budget bounds, MLX trainers no trust_remote_code
v0.26.0 — Registry: name/tag validation (alphanumeric + _-. only, null-byte rejected, name ≤128 / tag ≤64 chars), artifact path containment (default enforce_cwd=True via os.path.realpath + commonpath, stored path is realpath), SQL LIKE wildcard escaping (% and _ escaped with ESCAPE '\\' in search() and prefix resolve()), DB 600 perms on POSIX, lineage indirect-cycle detection (BFS ancestor walk before insert), CLI Rich markup escaped everywhere, resolve() raises AmbiguousRefError on ambiguous prefix (no silent None)
v0.26.0 — Eval Gate: suite path via shared utils/paths.is_under_cwd containment, regression_threshold [0.0, 1.0], every_n_epochs [1, 100], on_regression Literal ("stop"/"warn"/"continue"), GateTask.tasks/prompts null-byte rejection, judge_model URL scheme allowlist (ollama://, https://, http://localhost/http://127.0.0.1) — SSRF hardening, callback fails-safe: structured errors treated as regressions under on_regression="stop"
v0.26.0 — Trace-to-Preference: input/output path containment via shared is_under_cwd, trace line cap 100,000, --format/--signal Literal validation, PII warning panel before every run, JSON-only parsing (no eval), malformed JSON lines skipped silently
v0.26.0 — Quant-Lobotomy: --before/--after/--tasks all containment-checked, registry:// refs support optional kinds filter to avoid picking the wrong artifact, format Literal validated
v0.26.0 — Soup Cans: Manifest format version pinned to 1; name alphanumeric+_-.; author max 128 chars, no null bytes/newlines; created_at must parse via datetime.fromisoformat; description max 4096; DataRef URL HTTPS-only; hf_dataset regex-validated; tar extraction uses filter="data" on Python 3.12+, fallback only on TypeError/AttributeError (not TarError); manual symlink/hardlink rejection + commonpath check; 100 MB size cap on pack + fork; dunder-key (__*__) and null-byte rejection in fork modifications to prevent prototype pollution; inspect/read_config refuse paths outside cwd
v0.27.0 — Multi-GPU Mastery: --gpus bounds (reject bool, non-digit, zero, negative, values above MAX_GPU_COUNT=128); --gpus auto on 0-GPU host prints explicit yellow warning (no silent no-op); Rich markup escaped on --config path before embedding in the multi-GPU advice Panel; accelerate launch argv assembled via shlex.quote per element (copy-pasted command safe against crafted paths); build_accelerate_argv validates num_processes >= 1, mixed_precision Literal (no/fp16/bf16/fp8), num_machines bounded [1, 256]; ZeRO++ integer literals (int(1e9) not float) so DeepSpeed strict JSON validator accepts; validate_fsdp2_compile_config requires FSDP + CUDA + transformers + torch>=2.2/accelerate>=0.27; DeepSpeed-MII stub exits non-zero to prevent silent mis-start; validate_pipeline_config enforces pipeline_stages >= 2 + CUDA + gpu_count >= stages; pipeline_stages Pydantic bounds [1, 16]; parallelism Literal data|pipeline; NCCL env (NCCL_P2P_DISABLE/NCCL_IB_DISABLE/NCCL_NVLS_ENABLE) applied via os.environ.setdefault only — user/launcher overrides are never stomped
v0.28.0 — Training Speed & Memory: quantization_aware: Union[bool, Literal["fp8"]] rejects arbitrary strings (only true / false / "fp8"); FP8 path requires CUDA + Hopper+ SM capability + transformers backend; gradient_checkpointing: Union[bool, Literal["selective","medium","full","auto"]] rejects unknown tier strings and returns only HF-supported keys (no private markers leak into TrainingArguments.gradient_checkpointing_kwargs); activation_offloading Literal cpu|disk, scratch save_dir containment-enforced via shared utils/paths.is_under_cwd before disk writes, torch.load(weights_only=True) prevents arbitrary Python deserialization on reload, TOCTOU closed between mkstemp and torch.save by holding the fd open, best-effort cleanup on context exit (handles SIGKILL mid-backward); kernel_picker.pick_best_kernel raises ValueError when all candidates lack a finite time_ms (prevents silent promotion of an untimed combo); Cut CE architecture detector matches on last path component only (so deepseek-ai/...-phi-... org-prefix does not trigger a Phi patch on a DeepSeek model); build_cross_doc_mask numpy-vectorised to avoid O(seq_length²) pure-Python fill at max_length bound (1M); @model_validator requires packing=true when packing_cross_doc_attn_mask=true (prevents silent no-op); SoupConfig._validate_v028_speed_memory_sft_only rejects use_cut_ce/quantization_aware="fp8"/kernel_auto_compose/activation_offloading on non-SFT tasks — prevents legacy int8-QAT wrapper from crashing on the string "fp8" and prevents silent no-ops on DPO/GRPO/KTO/etc. (multi-trainer wiring tracked for v0.28.1)
v0.29.0 — HF Hub Deep Integration: HF_ENDPOINT SSRF-hardened — scheme allowlist (http/https), null-byte rejection, 0.0.0.0 explicitly rejected, plain-HTTP only permitted for loopback (localhost/127.0.0.1/::1), RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via ipaddress.ip_address; repo ID regex [A-Za-z0-9][A-Za-z0-9._-]{0,95} per component, ≤200 chars total, null-byte / whitespace / .. / leading-/ rejection (applied to push --repo, train --push-as, data push --hf-dataset, deploy hf-space --model/--space); collection slug owner/slug-hash regex-validated, ≤256 chars; HF token resolution single-sourced in utils/hf.resolve_token (env > cached login), explicit non-printable tokens rejected, push --token flag deprecated with yellow warning; soup push --model confined to cwd via is_under_cwd (prevents crafted soup.yaml output: from uploading system files); auto-push checkpoint allow_patterns restricts uploaded files to *.safetensors/*.bin/*.pt/*.json/tokenizer*/trainer_state.json/training_args.bin/README.md (keeps .env and source files out of auto-pushed branches); prepare_hf_resume enforces cwd containment and passes local_dir_use_symlinks=False (defeats symlink-based FS escape on older huggingface_hub); commit messages stripped to first line and capped at 200 chars (prevents multi-line injection into public HF commit history); _render_eval_scorecard neutralises |/[/]/(/)/!/newlines/tabs/</> in task names and non-numeric scores; data_lineage HTML-escaped (defeats XSS on HF Hub README viewer); render_space_template validates model_repo via validate_repo_id before substitution into rendered app.py (crafted repo id cannot inject Python code); HFPushCallback uses sticky _repo_failed flag to short-circuit retries after hard failure (no log spam, no wasted API calls); add_to_collection prefers HfHubHTTPError 409 detection over string-match for duplicate handling
v0.30.0 — Inference Excellence: OTLP endpoint SSRF-hardened matching v0.29.0 HF_ENDPOINT (scheme allowlist, 0.0.0.0 rejected, RFC1918 / link-local / cloud-metadata via ipaddress.ip_address, plain HTTP loopback-only); pick_draft_model rejects URL-scheme target names (http:///https:///file://), null bytes, names >200 chars; validate_regex_pattern length-capped at 2048 + null-byte rejection + must compile; validate_json_schema dict shape + 64KB serialised cap + required type field; --json-schema file path confined to cwd via shared utils/paths.is_under_cwd; --structured-output json requires --json-schema (fail-fast prevents silent no-op); FastAPI /v1/adapters/activate/{name} pattern ^[a-zA-Z0-9][a-zA-Z0-9\-]*$ enforced before handler runs; activate/deactivate state protected by threading.Lock (no race on concurrent hot-swap); /v1/adapters response omits filesystem paths (names + active flag only); CORS on transformers backend restricted to loopback origins (^https?://(localhost|127\.0\.0\.1)(:\d+)?$) since hot-swap endpoints mutate state without auth; auto_quant.Candidate name regex + score [0.0, 1.0] + finite-float check + non-negative latency; pick_best generator-safe (materialises to list) so error-message count is accurate; OTel span context uses contextlib.ExitStack so __exit__ sees real exception info (spans correctly marked error on HTTPException); record_latency always runs in finally so tail-latency percentiles include failure paths; build_tracer idempotent — only installs provider when current is ProxyTracerProvider/NoOpTracerProvider (preserves operator-supplied instrumentation)
v0.32.0 — Training Stability & Auto-Tuning: --find-lr-output containment via shared utils/paths.is_under_cwd (prevents writes outside cwd); save_lr_finder_report rejects NaN / Infinity floats in lrs / losses and serialises with allow_nan=False (keeps the report parser-safe); compute_lr_schedule rejects non-positive start_lr, inverted ranges, and num_steps outside [2, 10_000]; pick_mixed_precision rejects empty / null-byte / >200-char model names and resolves multi-version quirks (qwen2.5 vs qwen2, phi-3.5 vs phi-3) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; compute_warmup_steps clamps to [10, 1000] with a ratio==0.0 short-circuit matching HF Trainer's "no warmup" convention; SpikeRecoveryStrategy is @dataclass(frozen=True) (post-construction mutation cannot bypass validation), max_attempts ∈ [1, 10], lr_decay ∈ (0, 1), min_lr > 0; cross-validator _validate_spike_recovery_requires_watchdog rejects loss_spike_recovery=true, loss_watchdog=false at config-load (fails fast instead of never triggering); convergence_window ∈ [5, 10_000], convergence_rel_tol ∈ (0, 1], recommend_action reuses detect_plateau so plateau heuristic stays single-source-of-truth; GradAccumMonitor.recommend() caps doubled accum at MAX_ACCUM=1024 so a runaway advisory loop cannot blow up DataLoader prefetch; generate_config validates BOTH the YAML output path AND the embedded decisions["output"] field via is_under_cwd (closes the gap where a crafted decisions["output"]="../../etc" would have silently propagated into the rendered YAML)
v0.34.0 — Observability & Dev UX: .crash bundle generator (utils/crash.py) recursively redacts hf_* / sk-* / Bearer … token-shaped strings in any captured config and metric tail before serialisation, so a .crash file shared on a public GitHub issue cannot leak credentials; output_dir is reduced to os.path.basename so $HOME doesn't leak; write_crash_bundle uses os.path.realpath + commonpath for cwd containment (Windows-safe; raises ValueError not PermissionError so callers cannot silently swallow with except OSError); filename appends secrets.token_hex(4) so two crashes in the same UTC second don't collide; bundle truncated to MAX_BUNDLE_BYTES=1_000_000. train.py crash-write surfaces failures to the user (no silent missing-bundle). profiling.py resolve_trace_path rejects empty / . / .. / / / \\ / null-byte run_id (closes the output_dir/profiles/../trace.json escape) and uses os.path.realpath + is_under_cwd; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). tracker.get_run LIKE-prefix match escapes % / _ / \\ and uses ESCAPE '\\' so a crafted run_id cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (_ensure_schema) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). runs.py show/replay/clean switched user run_id rendering to markup_escape and switched clean containment from broken Path.resolve() + relative_to() to project-standard os.path.realpath + is_under_cwd. tui_app.py lazy-imports ExperimentTracker and markup_escapes every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject [bold red]…[/] markup. run_cost.estimate_run_cost_usd rejects bool in num_gpus (bool is a subclass of int — same defence as v0.30.0 Candidate.__post_init__); duration clamped to [0, 1 year]; unknown GPU returns None so callers render — instead of fabricating $0.00. log_level.parse_log_level rejects non-string + null-byte input.
v0.33.0 — Live Wire: RLVR code_exec_reward adds OS-level isolation (Linux best-effort os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID), macOS sandbox-exec with default-deny MACOS_SANDBOX_PROFILE narrowed to a 3-name mach-lookup allowlist to prevent DNS / NSURLSession bypass of (deny network*)); prune_checkpoints switches to TOCTOU-safe os.lstat + S_ISLNK + shutil.rmtree(onerror=_abort_on_symlink) so a symlink encountered mid-walk aborts rather than escapes; run_gate wraps each task scorer in a typed try/except so backend failures produce score=None, error=str(exc) (never silent score=1.0); _parse_judge_url removes the bare http:// catch-all (defence-in-depth after the Pydantic GateTask validator); soup can run requires --yes or explicit consent callback and raises ValueError (not PermissionError, which is an OSError subclass that broad except blocks would swallow); GGUF rglob result for ollama deploy is realpath+commonpath checked against extract_dir (prevents symlink escape from a crafted can); DeployTarget.path validator normalises mixed \\// separators before splitting (closes a Windows .. bypass); CAN_FORMAT_VERSION 1→2 (additive — v1 still loads); soup can publish validates repo_id via utils/hf.validate_repo_id, resolves token via resolve_token, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; _write_spike_recovery_hint adds is_under_cwd containment check on args.output_dir from raw HF TrainingArguments; lookup_entry_by_output_dir emits ResourceWarning when 1000-row scan limit is hit (no silent miss); CrossDocCollator no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); Candidate rejects bool in score/latency_ms (was sneaking past int isinstance check); evaluate_candidate latency mean now divides by completed prompts (excludes crashed) so a broken candidate isn't artificially fast; auto_quant.run_auto_quant_picker soft-falls-back to highest-scored candidate when no candidate clears min_score (server still binds); build_logits_processors returns [] when neither outlines nor lm-format-enforcer is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; os.execvp auto-reexec uses list args (no shell), all forwarded flags pre-validated; cleanup_extract_dir uses os.path.commonpath (Windows-safe) instead of startswith; _run_subprocess catches TimeoutExpired and returns rc=124 (coreutils convention) instead of an unhandled traceback; new eval_results and tensorrt artifact kinds in RegistryStore._VALID_KINDS
v0.39.0 — LoRA Quality: LoraConfig.init_strategy: Literal["random","pissa","olora"] rejects unknown strategies; PiSSA + DoRA / VeRA combinations rejected at config-load. model_validator(mode="before") aligns use_olora=True → init_strategy="olora" via dict-copy (no caller mutation; matches v0.33.0 #47 immutability policy). rank_pattern/alpha_pattern: Optional[Dict[str, int]] capped at 256 keys × value (0, 1024], rejects bool (subclass of int — matches v0.30.0 Candidate policy), null bytes in keys, empty keys; cross-validator rejects with use_vera=True. ReLoRAPolicy is @dataclass(frozen=True) (post-construction mutation raises FrozenInstanceError); bounds: steps ∈ [1, 1e7], warmup_ratio ∈ [0, 1], prune_ratio ∈ (0, 1) (strict — prevents zero-everything footgun). magnitude_prune_tensor strict 0 < prune_ratio < 1 rejection, non-Tensor input raises TypeError, empty / single-element tensor short-circuits (avoids kthvalue(_, 0) runtime crash). _validate_relora_supported_tasks cross-validator rejects relora_steps with task != "sft" and backend=mlx with distinct error messages (matches v0.34.0 distinct-reason policy); multi-trainer expansion deferred to v0.39.1. is_gemma4_model uses a word-boundary regex ((?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$)) so "ungemma4ed" / "my-gemma4ish" no longer over-match; null-byte rejection on model_name. apply_gemma4_clippable_patch weight-copy fallback logs at DEBUG instead of silent random-init; the patch is gated by is_gemma4_model(cfg.base) in sft.py before invocation so non-Gemma4 trainings never traverse the module tree. apply_surgical_patches rejects empty / null-byte model_name with ValueError. templates/load_template containment: filename re-validated via _validate_name (rejects ..///\\/null/empty); os.path.realpath + os.path.commonpath containment check on the resolved path against _templates_dir() so a tampered manifest.json cannot read files outside the package directory (mirrors v0.26.0 registry policy). Tampered-manifest ValueError from _validate_name caught and falls back to inline (no propagating exception). 256 KB file-size cap. Inline TEMPLATES carries an explicit deprecation comment pointing at the canonical YAML registry; tests/test_templates_yaml.py asserts byte-equality of all 16 inline ↔ YAML pairs to prevent silent drift. Planned removal: v0.41.0+.
v0.38.0 — Quant Menu: TrainingConfig.quantization Literal extended with gptq / awq / hqq:1bit..hqq:8bit (no hqq:7bit — HQQ doesn't support it) / aqlm / eetq / mxfp4 / fp8; Pydantic rejects every other string at config-load. validate_gptq_checkpoint and validate_awq_checkpoint probe local paths for quantize_config.json / quant_config.json; HF repo IDs fall through; null-byte rejection + non-string TypeError on the ref. _validate_prequantized_no_qat rejects every pre-quantized format combined with quantization_aware (int8 QAT or 'fp8') — pre-quantized weights carry their own scale and QAT/FP8 prepare would silently corrupt them (mirrors LlamaFactory quantization.py:117/199/211). _validate_bnb_quant_storage_only_with_4bit rejects bnb_4bit_quant_storage on every non-BNB-4bit format (silent no-op otherwise); allowed dtypes: Literal["uint8", "float16", "bfloat16", "float32"]. _validate_quant_menu_supported_tasks restricts the new formats to task='sft' on backend='transformers' in v0.38.0 with distinct MLX-backend vs unsupported-task error messages (matches v0.34.0 distinct-reason policy). check_quant_distributed_compat hard-fails HQQ/EETQ/AQLM × {FSDP, ZeRO-3} (sourced from LlamaFactory quantization.py:199/211 plus AQLM dequant constraints); warning-tier (not error) for BNB-4bit + FSDP without bnb_4bit_quant_storage so users see the silent perf cliff; unknown quantization raises ValueError (no silent pass) and the check is wired into commands/train.py startup. parse_hqq_bits rejects unsupported bit-rates and malformed hqq: strings before any kernel build.
v0.37.0 — Multipack: validate_multipack_architecture raises ValueError on unknown arch (loud-fail vs Axolotl's silent-miss footgun); 18-arch frozen allowlist (Llama 3.x / Qwen 2/3 / Mistral / Gemma 2/3 / Phi 3/4 / DeepSeek V2/V3 / Mixtral / Falcon / StableLM / SmolLM2). FFD packer caps at _MAX_FFD_ITEMS=1_000_000 (algorithm is O(N²) worst-case — defence against adversarial dataset DoS); bool rejection on every numeric input (max_len, per-element lengths, batch_max_len, batch_size, seed, max_seq_length) matches v0.30.0+ Candidate policy; generator-input materialisation prevents silent empty-bin output when validation exhausts the iterator. MultipackBatchSampler rejects empty lengths, non-positive batch_max_len/batch_size, items larger than batch_max_len. build_multipack_sampler_for_lengths rejects tcfg.batch_size="auto" with actionable message (must be resolved upstream). _validate_multipack_packing_exclusive cross-validator on TrainingConfig rejects both multipack and packing set; _validate_multipack_supported_tasks on SoupConfig restricts multipack to sft/pretrain on transformers backend with distinct error messages for MLX backend vs unsupported task (matches v0.34.0 distinct-reason policy). build_4d_attention_mask caps allocations at _MAX_MASK_ELEMENTS=2³¹ cells (~8GB float32) — defence against max_length=1M × batch_size=8 OOM; tag_sub_sequences capped at _MAX_BOUNDARY_SEGMENTS=1_000_000. Mask builder rejects non-floating dtypes (was silent np.finfo ValueError), non-2D seq_pos_ids, negative segment IDs; padding (id=0) tokens are fully masked, including diagonal, so softmax is well-defined. select_packing_strategy rejects non-bool flash_attn_available. JinjaTemplateAnalyzer parses chat templates via Environment.parse only — never renders, so a crafted soup.yaml cannot trigger SSRF / filesystem reads; 128KB template cap, null-byte rejection, TemplateSyntaxError re-raised as ValueError. DEFAULT_MESSAGE_FIELDS is a frozenset (runtime-immutable); JinjaTemplateAnalyzer.message_fields returns a defensive copy.
v0.36.0 — Correctness First: --trust-remote-code opt-in replaces 9 unconditional trust_remote_code=True call sites across soup train / chat / serve / data download / eval auto; KNOWN_SAFE_PREFIXES allowlist (15 first-party orgs) suppresses warning panel for trusted repos; model_requires_trust_remote_code probes local config.json for auto_map (HF Hub repo IDs return None/unknown — HF still raises loudly when custom code is actually needed); resolve_trust_remote_code raises ValueError with actionable message when model needs custom code but the user did not opt in. Chat-template hardening: DataConfig.chat_template validator rejects null bytes, oversize (>64KB), AND filesystem-touching Jinja directives ({% include %}, {% import %}, {% from %}, {% macro %}, {% extends %} — both whitespace-control variants); empty string normalised to None; _REGISTRY wrapped in MappingProxyType (callers cannot mutate); apply_chat_template_override emits yellow advisory when active so users know soup push will persist the override into tokenizer_config.json. SFT silent f-string fallback (f"{role}: {content}") replaced with hard ValueError — produced wrong loss labels for years on tokenizers without a chat template. Loss-mask fallback passes add_special_tokens=False to incremental tokenize calls so HF cannot double-prepend BOS at front of each render (consistent prefix-delta walk); narrowed exception catch in _apply_template_with_mask from (TypeError, ValueError) to TypeError only so a malformed messages list propagates instead of falling through to the loose path. SOUP_BATCH_CACHE_PATH env override containment-checked via os.path.realpath + commonpath against ~/cwd/tempfile.gettempdir(); out-of-bounds values fall through to safe default; cache file gets best-effort 0o600 perms after atomic rename (matches v0.26.0 registry.db policy). make_cache_key rejects bool in numeric inputs (matches v0.30.0 Candidate policy). Documented limitation: non-SFT trainers (DPO/GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Pretrain/Embedding) and commands/{diff,export,merge,infer,generate}.py still hardcode trust_remote_code=True — v0.36.x patch follow-up.

Security Scanning

All code is scanned with ruff for style and common issues
Dependencies are regularly updated to patch known CVEs
GitHub's dependency scanning alerts us to vulnerable dependencies
We use GitHub Actions CI/CD for continuous integration

Dependency Updates

We actively monitor and update dependencies:

Major dependency updates: Tested in PR before merging
Security patches: Applied immediately and released as patch versions
Deprecated dependencies: Replaced proactively

Coming Soon

Automated dependency scanning
SBOM (Software Bill of Materials) for each release
Third-party security audit (after 1.0.0 release)

Questions?

If you have security questions (not vulnerability reports) or need clarification:

Open a GitHub Discussion tagged security
Open a GitHub Issue (non-vulnerability inquiries)
Check our CONTRIBUTING.md for general support

License

This Security Policy is provided under the Apache-2.0 license, same as the Soup project.

Last Updated: April 2026

For the latest version of this policy, visit: https://github.com/MakazhanAlpamys/Soup/blob/main/SECURITY.md

32 KiB Raw Blame History Unescape Escape