mirror of https://github.com/razor-ai/soup.git
51 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
07e7214ed3 |
feat(v0.53.0): Quant Menu II — UD GGUFs + KV cache + NVFP4 + LF parity + save formats
Schema-only release. Live wiring deferred to v0.53.1 (mirrors v0.50.0 /
v0.51.0 / v0.52.0 stub-then-live pattern).
- Part A — Unsloth Dynamic 2.0 GGUF ladder (14 entries: UD-Q8_K_XL ... UD-IQ1_M)
+ validate_calibration_data_path shape validator.
- Part B — IQ (12) + Apple/ARM (10) GGUF flavours in utils/gguf_quant.py;
O(1) _LOWER_INDEX MappingProxyType for case-insensitive lookup.
- Part C — training.kv_cache_type: q8_0 | bf16 | f16 | fp8 (fp8 Hopper-only;
MLX rejected). requires_hopper reads from spec metadata (single source).
- Part D — fp8_attention (requires quantization_aware='fp8') + nvfp4 (Blackwell)
+ native unsloth_bnb_4bit bool flags with cross-validators.
- Part E — bnb_4bit_use_double_quant + llm_int8 (explicit 8bit assertion,
distinct from v0.41.0 load_in_8bit aliasing) + quantize_ref_model
(extends ref-task set with grpo/kto/ppo) + quantize_reward_model.
- Part F — soup merge --save-format {fp16, 4bit, 4bit_forced} + soup export
--format torchao with closed PTQ scheme allowlist (Int4WeightOnly,
Int8DynActInt4, Float8DynActFloat8, NVFP4 — case-sensitive PyTorch names).
Test count: 7453 → 7610 (+157 net new across 154 tests in test_v0530.py).
ruff check soup_cli/ tests/ — clean.
5 review agents ran (python / code / security / tdd / verification);
every CRITICAL / HIGH / MEDIUM / LOW finding fixed or documented:
- O(N) gguf walk → O(1) _LOWER_INDEX MappingProxyType
- ref_tasks extended with grpo + kto + ppo (silent-no-op footgun)
- _validate_v053_bool_fields no longer coerces None → False
- requires_hopper delegates to _KV_CACHE_METADATA spec
- fp8_attention validator order: quantization_aware before MLX
- validate_calibration_data_path + validate_quant_config_path docstrings
name the exact controls v0.53.1 CLI dispatch MUST add (TOCTOU contract)
- validate_torchao_scheme case-sensitivity documented at validator
- tautological `result == result` test replaced with allowlist invariant
- bool guards added on backend/modality/quantization across all Part D
validators
- exact 4096/4097 boundary tests for path shape validators
Docs updated: CLAUDE.md (test counts + utils list + changelog + test-table),
README.md (What's New replaced + 5 new dedicated sections), SECURITY.md
(support window + v0.53.0 hardening entry), CONTRIBUTING.md (test count
+ utils list + test-table row), .claude/plan.md (heading + boxes + banner —
gitignored, local only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
df7f49feda |
feat(v0.52.0): Modality II — TTS + Distillation + BitNet + EBFT-GDPO + MoE quant + reasoning_effort
7 schema-only Parts; live trainer / loss / export wiring deferred to v0.52.1
(mirrors v0.27.0 MII / v0.50.0 GRPO Plus / v0.51.0 hubs stub-then-live pattern).
- Part A: task='tts' + modality='audio_out' + 5 families (Orpheus/Sesame-CSM/
Llasa/Spark/Oute) + per-family emotion allowlist (Orpheus 8 / Oute 6) +
5 recipes (orpheus-tts-sft, sesame-csm-tts, llasa-tts, spark-tts, oute-tts).
- Part B: classifier / reranker / cross_encoder tasks + num_labels (bool-
before-int guard) + classifier_kind + label_names (dedup + null-byte + cap).
- Part C: task='distill' + teacher_model + distill_divergence (kl alias
canonicalises to forward_kl; Literal excludes alias) + distill_temperature
(math.isfinite + [0.05, 100.0] bounds).
- Part D: quantization='bitnet_1.58' (gated to non-MLX + text + task in
{sft, pretrain, dpo}) + Falcon-E BitNet recipe + soup export
--format bitnet/tq1_0 CLI stubs (yellow deferred-advisory panel, exit 0).
- Part E: EBFT (structured/strided + bounded ebft_temperature; SFT-only)
+ GDPO (standard/length_normalized/margin; DPO-family-only).
- Part F: moe_expert_quant (nf4/int8_rowwise) + train_router_only — both
require moe_lora=true (silent-no-op footgun rejection).
- Part G: reasoning_effort (low/medium/high) + train_on_eot — both gated to
the SFT-family task set (sft/pretrain/distill/classifier/reranker/
cross_encoder); rejected on DPO/GRPO/PPO/etc. with named offenders.
Review fixes (5 agents: python, security, code, tdd, verification-loop-manual):
- num_labels bool-before-int field_validator (security HIGH)
- reasoning_effort + train_on_eot SoupConfig task-gate (code HIGH)
- _validate_classifier_compat lazy-import early-return (code HIGH)
- Oute emotion allowlist via data-driven _FAMILY_EMOTIONS (python+code MED)
- _MAX_LEN -> _MAX_REASONING_EFFORT_LEN (python MED)
- validate_reasoning_effort wired via field_validator (security MED)
- distill_divergence Literal excludes "kl" alias (code MED)
- DIVERGENCES derived from _DIVERGENCE_ALIASES drift guard (python LOW)
- is_bitnet_model comment fixed to match impl (python LOW)
- sister-fn bool guards on every compat helper (python+security MED)
- TDD coverage gaps closed: EBFT variant oversize, GDPO full rejection matrix,
ebft_temperature explicit-exc table, TTS compat input guards, recipe
model-id null/whitespace, full reasoning_effort task-matrix.
Drift fixes:
- tests/test_onnx_tensorrt_export.py + tests/test_awq_gptq_export.py
SUPPORTED_FORMATS count bumped 5 -> 7 (bitnet + tq1_0 stubs).
- tests/test_recipes.py catalog_size assertion 106 -> 112 (5 TTS + Falcon-E).
Test count: 7184 -> 7456 (+272). 230 new tests in tests/test_v0520.py;
remainder from drift-fix parametrize expansions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
352f8d4bfd | docs(contributing): correct v0.51.0 test count (7178 → 7184) | |
|
Alpamys
|
1e1abacb44 |
feat(catalog): v0.51.0 — Model Catalog Expansion + Alternative Hubs
26 new ready-made recipes (catalog 80 → 106) covering 25 model families: GPT-OSS 20B/120B, GLM 4.6/5, Kimi K2 / K2-Thinking GRPO, MiniMax-M2, QwQ-32B GRPO, QVQ-72B, Granite 4, Liquid LFM2, Cogito v2, Mistral Small 3 / Medium 3.5, Magistral / Devstral / Ministral, MedGemma, EmbeddingGemma, LLaVA-Next, InternVL 3.5, Voxtral, Baichuan 2, Qwen-Image, DeepSeek-OCR, Paddle-OCR-VL. Part D: MULTIPACK_ARCHITECTURES extended 18 → 38 (Granite, GLM, Kimi, MiniMax, QwQ, QVQ, GPT-OSS, Magistral, Devstral, Ministral, MedGemma, LFM2, Cogito, Hunyuan, Ernie, Yi, Baichuan, ChatGLM). Part E: alternative model hubs. New soup_cli/utils/hubs.py with closed allowlist (hf/modelscope/modelers), SSRF-hardened endpoint validators mirroring v0.29.0 HF_ENDPOINT policy (scheme allowlist, loopback-only HTTP, RFC1918/link-local rejection, control-char/CRLF rejection, IPv6-mapped private rejection). TrainingConfig.hub Literal field with case-insensitive _normalize_hub field_validator; SoupConfig _validate_hub_supported rejects backend=mlx + hub != hf. Live downloader / uploader wiring deferred to v0.51.1 (matches the v0.27.0 MII / v0.37.0 multipack stub-then-live pattern). +455 tests (6729 → 7184), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
33c60b4c1f |
feat(grpo): v0.50.0 — GRPO Plus (unsloth + axolotl RL parity)
22 features across 5 internal Parts shipped as schema-only — closed allowlists, Pydantic validators, NotImplementedError stubs for live wiring deferred to v0.50.1 (mirrors v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro / v0.45.0 plugins / v0.48.0 curriculum / v0.49.0 LongLoRA stub-then-live pattern). Part A — 7 GRPO objective variants (gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard) with `validate_grpo_variant` + frozen `GRPOVariantSpec` metadata + `MappingProxyType`-wrapped registry. `validate_grpo_delta` is bool-first / math.isfinite / (0, 1] bounded. `apply_variant_loss` raises NotImplementedError with v0.50.1 marker for the 6 new variants and is a no-op for standard. Part B — long_context_grpo + vllm_sleep_mode schema gates with compat validators (null-byte rejection on task + backend, bool guard on use_ring_attention). vllm_sleep_mode requires task='grpo' AND a transformers/unsloth backend (code-review HIGH fix — sleep is a between-rollouts feature). Part C — 4 multi-turn rollout backends (art / ruler / nemo_gym / openenv) with closed allowlist + per-entry required_package mapping. Part D — 7 stability/efficiency knobs (ref_model_ema_alpha, replay_buffer_size, async_grpo_prefetch, tis_threshold, mask_truncated_completions, defer_rerolling, skip_zero_advantage, off_policy_mask_threshold). Every numeric field rejects bool via a shared `_reject_bool_on_grpo_numerics` field_validator (tdd-guide HIGH fix — Pydantic v2 coerces True→1 by default). The `mask_truncated_completions` + `tis_threshold` pairing is enforced by a cross-validator (matches v0.32.0 spike-recovery+watchdog policy). Part E — top-level task='prm' Literal addition (Process Reward Model / stepwise-supervised, paired with data.format='prm' from v0.42.0) + `vision_grpo: bool` flag for VLM-RL on Qwen2-VL / Pixtral / InternVL. Compat helpers gate task / modality / backend. Review-round fixes applied (5 sequential reviews per CLAUDE.md): - python-review: list_variants annotation, frozenset[str] params, Optional[str] → str | None, module-level math import, D401 imperative docstrings, dropped *args/**kwargs on stubs. - code-review: grpo_fp16 added to GRPO-only task-gate; vllm_sleep_mode requires task='grpo'. - security-review: explicit field_validator for grpo_delta NaN/Inf rejection (Pydantic le=1.0 incidentally rejects NaN, made explicit); null-byte rejection on backend/task in grpo_long_context helpers; use_ring_attention bool guard. - tdd-guide: bool-rejecting field_validator on all Part D numeric fields + grpo_delta; missing bool-rejection tests added on validate_grpo_variant / validate_rollout_backend; null-byte test on validate_vllm_sleep_mode_compat; required_rollout_package rejection path; RolloutBackendSpec.live_wired; PPO+vision_grpo round-trip; _DEFERRED_LIVE invariant. Test count: 6490 → 6729 (+239 across 5 new test files). Notes for future maintainers: - v0.50.0 has zero new CLI commands and zero new trainer wirings; every step 6d/6e is intentionally n/a. Step 6 smoke runs schema happy + every documented cross-validator rejection. - All `task='grpo'` gates use `if self.task != 'grpo'` literal comparisons; do NOT switch to a set membership check until Part D knobs are wired into PPO/preference trainers in v0.50.x. - Multi-modal Vision RL does not yet verify the base model is actually a VLM — upstream trainer surfaces the error loudly when it fails to load the vision tower. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
161c81ee7f |
feat(long-context): v0.49.0 — YaRN, Dynamic NTK, LongLoRA S², Llama 3.1 NTK
- Part A: YaRN RoPE scaling — math kernels (yarn_find_correction_dim /
yarn_find_correction_range / yarn_linear_ramp_mask / yarn_get_mscale) +
4 yarn_* schema fields + cross-validator rejecting yarn fields outside
rope_scaling_type=yarn. Pure-Python implementations of the upstream YaRN
paper §3.4/§3.5 with bool/NaN/Inf rejection on every numeric input.
- Part B: Dynamic NTK — existing path verified, explicit test coverage.
- Part C: LongLoRA S² shifted-sparse attention (schema-only) — new
soup_cli/utils/longlora.py with is_llama_model (word-boundary regex
mirroring v0.39.0 is_gemma4_model policy) + validate_longlora_compat.
TrainingConfig.use_longlora + SoupConfig._validate_longlora_compat.
Live LlamaAttention.forward override deferred to v0.49.1 (stub-then-live
pattern, mirrors v0.27.0 MII / v0.37.0 multipack).
- Part D: Llama 3.1 NTK-aware (full impl) — scale_inv_freq_llama3
smooth-transition kernel + detect_llama3_rope_in_config HF-config probe
+ "llama3" added to rope_scaling_type Literal. LLAMA3_DEFAULT_*
constants per Unsloth models/llama.py:1853.
Public-boundary input validation on get_rope_scaling_config (bool/NaN/Inf
rejection on target_length / original_length / yarn_factor) per security
review — prevents direct callers from emitting {factor: NaN} into HF
model configs when bypassing the Pydantic schema.
Reviews: code-reviewer (2 HIGH + 1 MEDIUM + 1 LOW), security-reviewer
(1 MEDIUM + 2 LOW), python-reviewer (4 findings), tdd-guide (8 coverage
gaps) — all findings fixed. verification-loop done as manual equivalent
(version + --help + happy/failure YAML smoke).
+80 net new tests (6410 → 6490). Full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
f789953d46 |
feat(data): Data Mixing Optimizer + v0.48.0 release (v0.48.0 Part B, BETA)
BETA. New `soup data mix --optimize --budget 1h --datasets a,b,c` runs N short proxy-training runs over candidate mixture weights and writes a canonical recipe YAML you can splice into `soup.yaml` under `data.interleave`. Per-candidate proxy failures are isolated (DEBUG-log + sentinel `_MAX_LOSS` + `continue`); `KeyboardInterrupt` / `SystemExit` re-raised; budget cap surfaces `MixOptimizationReport.partial=True`. `soup data mix --apply <recipe.yaml>` re-loads + prints the recipe's canonical interleave block. Both modes enforce `is_under_cwd` containment + TOCTOU symlink rejection (`os.lstat + S_ISLNK`) + 256 KB file cap; YAML key injection defended at the renderer (rejects newlines / null bytes / oversize dataset paths). Synthetic offline proxy ships in v0.48.0; live `soup train` proxy + scikit-optimize backend wiring deferred to v0.48.1 via `OptimizerProtocol` ducktype (default fallback: deterministic Dirichlet sampler). Review fixes: - `validate_datasets` early `len(raw) < 2` check (code-review MEDIUM) — prevents the less-actionable error after realpath resolution. - `run_mix_optimizer` proxy exceptions now isolated per-candidate (code-review MEDIUM) — first-cut raised RuntimeError on the first proxy failure, breaking the documented `partial=True` contract. - `load_mix_recipe` `os.lstat` wrapped in `try/except OSError` (security HIGH) — closes a TOCTOU race where path disappearance between `lexists` and `lstat` would raise an unhandled OSError. Release bundle (v0.48.0): - version bump → 0.48.0 in pyproject.toml + soup_cli/__init__.py - README.md: replaced "What's New" + 2 new dedicated `##` sections - SECURITY.md: supported-versions window + per-version notes for v0.48.0 - CONTRIBUTING.md: test counts (6242 → 6410) + 2 new test-table rows +94 tests. Net release total: 6242 → 6410 (+168 tests, +2 test files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
c3f42119d5 |
docs(contributing): add v0.47.0 test-file entries
Backfill the test-file table for v0.47.0 (test_v0470_part_a.py + test_v0470_part_b.py). The full per-release table lives in .claude/CLAUDE.md; this file keeps a curated subset. Docs-only — no version bump required. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f64cee569e |
feat(v0.47.0): Data Forge — synthetic data pipeline + data quality moat
Part A — Synthetic Data Forge (utils/data_forge.py + commands/data_forge.py):
- soup data forge --docs <dir> --task sft|preference|tool with full provenance manifest
- ForgePlan / ProvenanceRecord / ForgeRow frozen dataclasses
- chunk_document + score_uncertainty pure-function kernel
- discover_documents (cwd-contained, symlink-rejecting, .txt/.md/.json/.jsonl allowlist)
- synthesise_forge_rows with judge-exception swallow at DEBUG
- Atomic JSONL + provenance writes via staged-tempfile + os.replace
Part B — Data Quality Moat (utils/data_score.py + commands/data_score.py):
- soup data score / decontaminate / toxicity / langdetect / pii / educational
- ReDoS-hardened PII regexes (phone + credit-card rewritten, 50 KB pre-cap)
- Containment-based n-gram decontamination (docstring corrected from "Jaccard")
- 6-language stopword heuristic for langdetect
- compute_scorecard with per-row DEBUG logging on swallowed errors
Security review fixes applied:
- math.isfinite guard on _validate_float_unit (NaN/Inf rejected before bounds)
- ReDoS: phone regex flattened (no nested optional quantifiers); credit_card
rewritten from {13,19}-loop to anchored 4-4-4-N; 50 KB pre-cap before finditer
- is_under_cwd moved inside discover_documents (no longer relies on caller)
- os.lstat + S_ISLNK rejection on every write target + tempfile staging
- _require_str rejects null bytes (consistency with data_forge._validate_str)
- compute_scorecard try/except blocks log at DEBUG (no silent swallow)
- decontaminate_texts: Optional[...] = None (no more type: ignore)
- _read_rows / _write_rows have full type annotations
- ngram_overlap_ratio docstring renamed to "containment ratio"
- import math + import tempfile moved to module top (lazy-import policy
applies to heavy ML deps only, not stdlib)
- Duplicate discover_documents call in CLI collapsed (TOCTOU window closed)
Live judge providers, [data-pro] extras (Llama-Guard / FineWeb-Edu / Presidio /
fastText / langdetect), and operator-supplied benchmark corpora are
stub-then-live and ship in v0.47.1 (mirrors v0.27.0 MII / v0.37.0 multipack
precedent).
Tests: 6126 -> 6242 (+116 net new). All v0.47.0 + full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
986a4c00e3 |
feat(agent): Agent Forge — OpenAPI/MCP/GraphQL spec → tool-calling SFT dataset (v0.46.0 Part B)
Bumps to v0.46.0 and ships the Agent Forge: parse OpenAPI 3.x, MCP server
manifests, or GraphQL introspection JSON straight into a tool-calling SFT
dataset where each row is `{messages: [user, assistant_with_tool_call],
tool, source_endpoint}`. No more hand-rolled jsonl scaffolding for
function-calling fine-tunes.
* soup_cli/utils/agent_forge.py — `Endpoint` / `SynthRow` / `SpecReport`
frozen dataclasses. `parse_openapi` / `parse_mcp` / `parse_graphql`
parsers leave `$ref` strings opaque (no external resolution — defends
against file-read SSRF). All synthesised path strings routed through
`_validate_path` (rejects newline-in-name across all three parsers).
`load_spec_file`: `is_under_cwd` + `os.lstat + S_ISLNK` BEFORE realpath
(corrects v0.46.0 first-cut ordering caught by security review) + 5MiB
cap + yaml.safe_load only. `write_dataset` atomic via mkstemp +
os.replace (mid-stream TypeError never leaves partial file; mirrors
v0.43.0 Part D `copy_bundle_to` policy) + symlink rejection at target.
Caps: `_MAX_ENDPOINTS=10_000`, `_MAX_SPEC_BYTES=5MiB`,
`_MAX_ROWS_PER_ENDPOINT=32`.
* soup_cli/commands/agent.py — `soup agent synth/train/eval` Typer
subcommands. `synth` table cells pass through `rich.markup.escape`.
`train` rejects NUL/newline/oversize in `--base` and `--output-dir`
BEFORE embedding into rendered YAML recipe (CRITICAL security fix —
defends against YAML key injection where `--base $'evil\ntraining:
{ epochs: 9999 }'` would smuggle in injected training keys). `eval`
enforces predictions `is_under_cwd` + symlink rejection +
`_MAX_PRED_LINES=1_000_000` DoS cap.
* soup_cli/cli.py — registers `agent` Typer group; help string uses
ASCII-safe `->` (`test_help_output_is_ascii_safe` regression test caught
a Unicode `→` on first try).
* tests/test_v0460_part_b.py — 71 tests covering every parser kind,
failure modes (cycle / cap / null-byte / oversize / outside-cwd /
symlink), atomic-write partial-failure invariant, every CLI surface.
* Docs: README ## What's New replaced + dedicated `## Deploy Autopilot`
and `## Agent Forge` sections added; SECURITY.md supported-window
shifted (v0.46→full, v0.41→drop) + v0.46.0 fix-notes entry;
CONTRIBUTING.md test counts 165→167 / 5989→6126.
Test suite: 5989 → 6126 (+137 net new) green on Windows.
Known limitations (live runtime deferred to v0.46.1):
- Quant-Lobotomy auto-measure for deploy autopilot
- RLVR `code_exec` sandbox scoring in `agent eval`
- In-process `soup train` re-entry in `agent train` (Typer commands aren't
safe to re-enter — matches v0.44.0 `soup quantize` design)
- ExecuTorch packaging for iphone-16 / pixel-9 (lands in v0.54.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
74301843a2 |
feat(v0.45.0): Plugin System & Ecosystem Wins — 5 Parts, +169 tests
Adds a public plugin/hook system plus the schema scaffolding for 20+ ecosystem
integrations. Live trainer-callbacks, Anthropic /v1/messages route, server-tool
HTTP endpoints, and the recipe runner ship in v0.45.1 (matches v0.27.0 MII /
v0.37.0 multipack / v0.41.0 LLaMA Pro stub-then-live pattern).
Part A — Plugin / hook system
* New soup_cli/plugins/ package: BasePlugin Protocol, PluginSpec frozen
dataclass, register_plugin / discover_hooks / enable_plugin /
disable_plugin / load_plugins. Kebab-case name regex, semver-ish version,
null-byte rejection on every string. Idempotency check covers
(version, plugin object, templates, model_groups, description) — review-fix
added description after first-cut omitted it. Per-list caps on templates
and model_groups (32 entries, 128-char per name).
* New soup plugins list/install/enable/disable Typer CLI; all user-controlled
output passes through rich.markup.escape.
Part B — API extensions (schema-only)
* utils/anthropic_messages.py: to_anthropic / from_anthropic /
validate_anthropic_payload converters. Multiple system messages join with
\n\n; tool role with structured (list) content concatenated into single
tool_result text block (review-fix MEDIUM — first-cut silently dropped).
max_tokens cap 16384, temperature [0.0, 2.0], bool rejection on numerics.
* utils/server_tools.py: closed {python, bash, web_search} allowlist,
WebSearchConfig with domain allowlist + leading-dot subdomain pattern,
rate_limit [1, 600]. is_domain_allowed strips :port suffix and rejects
IPv6 literals (review-fix MEDIUM).
* utils/ngram_spec.py: NgramSpecConfig validators with bounded n / draft
tokens / prompt-lookup-max; bool rejection on every numeric field.
Part C — External integrations catalog
* utils/integrations.py: 15-entry MappingProxyType catalog of ecosystem
targets (lm-studio, comfyui, ollama, claude-code, cursor, continue, ...).
Part D — Advanced trainer-plugin allowlist
* utils/trainer_plugins.py: 6-entry allowlist (grokfast, spectrum,
llmcompressor, sonicmoe, cce_plugin, math_verify) + validate_trainer_
plugin_list (Sequence[str], dedup, _MAX_PLUGINS_PER_RUN=8).
Part E — Data Recipe DAG
* utils/recipe_dag.py: closed NODE_KINDS frozenset, Kahn's topological
sort via collections.deque (review-fix HIGH — first-cut had O(N^2 log N)
queue.sort() inside the BFS body), cycle / self-loop / dangling-edge
rejection, _MAX_NODES=256 / _MAX_EDGES=1024 / _MAX_FILE_BYTES=1MiB.
load_recipe_yaml enforces is_under_cwd containment AND os.lstat + S_ISLNK
symlink rejection (review-fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22
/ v0.43.0 Part C / v0.44.0 Part B policy).
* New soup data recipe <path> CLI validates topology and prints planned
topo order; live runner deferred to v0.45.1.
Reviews: python-review, security-review, code-review, tdd-guide all run;
verification-loop replaced by manual smoke (CLI happy + failure paths
exercised on real fixtures).
Test count: 5820 -> 5989 (+169). Test files: 164 -> 165. Ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
c4ac3da695 |
feat(v0.44.0): Live Dashboard & UX - 21 features, +192 tests
Part A - Live monitoring: soup monitor (nvidia-smi panel), EMA + p95/p99 tail-latency stats, SSE training-stream schema, phone-visible URL + ASCII-QR helper, llama-server timings parser + KV-cache bar, thread-safe ToolOutputsBuffer + ToolCallTimer. Part B - UX fixes: GracefulSaveHandler (first SIGINT saves, second stops), .checkpoint_now trigger file (cwd-contained, symlink-rejected), desktop / .command / .cmd shortcut builders, onboarding-wizard YAML renderer. Part C - UI tabs: drop-in soup_cli/ui/plugins/*.py registry with kebab-case name allowlist + 32-tab cap, API_HOST / API_PORT / API_KEY + GRADIO_HOST / GRADIO_PORT env knobs. Part D - Standalone CLIs: soup fetch (bundled examples + configs + deepspeed_configs catalog), soup quantize (ergonomic alias), soup merge-sharded-fsdp-weights, soup delinearize-llama4 (planners; live runtime in v0.44.1), soup llama <sub> (closed-allowlist proxy with filtered child env that drops HF_TOKEN / OPENAI_API_KEY / ANTHROPIC_API_KEY), soup_cli.utils.sweep_config (separate sweep.yaml loader), reasoning_parser allowlist for soup serve. Security review fixes: fetch symlink-at-target rejection + bundled-source commonpath check, write_trigger symlink rejection (TOCTOU), llama child-env secret allowlist, onboarding output cwd-containment, qr token moved from URL fragment to query string (so server actually sees it), sweep-config scalar allowlist + MappingProxyType[Tuple] immutability. Code/Python review fixes: detect_apple_silicon clean rewrite (was buggy parser-priority ternary), all frozen-dataclass List fields -> Tuple, ToolOutputsBuffer -> collections.deque(maxlen=1000), os.path.realpath over abspath, IPv6 host auto-bracketing per RFC 3986, frozenset over mutable set, type hints on __exit__/_make_proxy. Test count: 5628 -> 5820 (+192). Lint clean. Help output ASCII-safe (em-dash check enforced by test_cli_subprocess). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
82d5693b75 |
feat(eval): Tracker & Eval Pro — 18 features (v0.43.0)
Closes the observability gap with all three competitors in one release.
Part A — Trackers
* --tracker flag (mlflow/swanlab/trackio) on soup train, mutually
exclusive with --wandb/--tensorboard. Closed allowlist via
MappingProxyType. Live integrations rely on HF Trainer's report_to.
* SOUP_TELEMETRY=1 opt-IN env var; build_telemetry_payload schema is
closed-key (no model names / dataset paths / config contents). Live
PostHog network code deferred to v0.43.1.
Part B — Eval metrics
* Pure-Python BLEU + ROUGE-1/2/L + effective_tokens_per_second.
* KL-divergence calibration framework with OK/MINOR/MAJOR thresholds.
* Model Arena Elo tournament (256-model cap, MappingProxyType view,
Rich-markup metacharacter rejection on names).
* ceval / cmmlu / aider_polyglot benchmark allowlist (live Aider
runner deferred to v0.43.1).
Part C — Profiling
* memory_snapshot_context (narrow RuntimeError catch — review fix
prevents generator-already-executing on user-body RuntimeError).
* detect_anomaly_context, nccl_bandwidth_check (h100/a100/v100/rtx
reference table; live measurement CLI surface deferred).
* write_vscode_launch with TOCTOU symlink rejection at the target
path regardless of force=True.
Part D — Demo bundles
* `soup data demo` lists / copies 4 bundled JSONL fixtures
(alpaca / sharegpt / dpo / grpo) with staged-tempfile atomic
rename + 50 MB cap + symlink rejection on the staging path.
Tests: 5389 -> 5628 (+239). Ruff clean. Five sequential review waves
(python / code / security / tdd / smoke) ran; HIGH/MEDIUM/LOW findings
all fixed including: tracker name shadow in train.py, _lcs_length DP
double-buffer bug, BLEU geo-mean policy, base_dir absolute/.. escape,
demo_bundles tmp symlink TOCTOU, vscode launch symlink TOCTOU.
Note (Windows CI): line-ending warnings (LF -> CRLF) on commit are
expected; `.gitattributes` policy is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
05093ebfdb |
feat(data): Data Pipeline Pro — 18 features, axolotl + LF parity (v0.42.0)
Schema-first surface for the data pipeline gap with Axolotl + LlamaFactory. Ships in one release: 5 new formats (prm, pre_tokenized, input_output, video, multimodal), remote URI allowlist (s3/gs/gcs/az/abfs/abfss/oci) + streaming + sharding, AOT preprocess cache + `soup data preprocess` CLI, multi-dataset interleave (concat/under/over/probs) + 8 advanced masking fields, vocab expansion (add_new_tokens / new_special_tokens / resize_vocab) + custom prompt_strategy, and document ingestion (`soup data ingest` for PDF/DOCX/MD/TXT). Live wiring for fsspec backends, AOT tokenize loop, custom prompt-strategy runtime, and PRM trainer integration is deferred to v0.42.1+ (stub-then-live pattern from v0.27.0 / v0.37.0 / v0.41.0). Schema gates fire at config load so misconfiguration fails fast. Security: full v0.42.0 hardening matrix — `_REMOTE_SCHEMES` MappingProxyType allowlist; bucket regex 1-63 chars per S3/GCS spec; userinfo / fragment / query-string rejection on remote URIs (query-string forwarded to fsspec is SSRF-adjacent); null-byte + length caps on every string-shaped input; bool-rejected-before-int on every numeric input; frozen InterleaveSpec dataclass; 10k caps on add_new_tokens; `is_under_cwd` containment on video_dir + tokenized_path schema fields and on preprocess --config / both ingest paths; `os.lstat + S_ISLNK` symlink rejection on ingest input; PRM converter type-checks completions (str) + labels (bool, not int); video field null-byte + 2KB cap; field-name threading on image-pixels validator so error messages name the actual field. 5242 tests → 5389 (+147 net). 11 review findings addressed across python-review / code-review / security-review / tdd-guide (CRITICAL→LOW). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1491025a36 |
feat(trainer): Optimizer & PEFT Zoo (v0.41.0)
Optimizer Zoo (Part A): closed-allowlist SUPPORTED_OPTIMIZERS adds 14 new
entries — BAdam, APOLLO (apollo_adamw), Adam-mini, lomo / adalomo,
grokadamw, schedule_free_adamw / schedule_free_sgd, muon, dion,
came_pytorch, ao_adamw_{fp8,4bit,8bit}. Validates name + lower-cases
deterministically; rejects non-string / empty / null-byte / >64-char.
_OPTIMIZER_PACKAGES wrapped in MappingProxyType (matches v0.36.0 _REGISTRY).
Per-module LR (Part B): training.lr_groups accepts list-of-pairs /
list-of-dicts / {pattern: lr} mapping; canonical [{pattern, lr}, ...]
storage. Capped at MAX_LR_GROUPS=32; per-pattern non-empty string
≤256 chars + null-byte rejection + re.compile + best-effort ReDoS probe;
per-LR (0.0, 1.0] + math.isfinite (rejects NaN AND ±inf) + bool rejection
(matches v0.30.0 Candidate policy); duplicates rejected. lr_groups_from_schema
bridges canonical schema shape into runtime List[LrGroup] for
build_optimizer_param_groups (first-match-wins routing). LrGroup is
@dataclass(frozen=True). PyYAML scientific-notation (1e-4) parses as
string in YAML 1.1; _validate_lr coerces str → float so soup.yaml
round-trips work. base_lr rejects bool / non-positive (defence-in-depth).
PEFT methods (Part C): LoraConfig.init_strategy="loftq" + loftq_iter
∈ [1, 10] + loftq_bits ∈ {2, 4, 8}; cross-validator rejects loftq +
use_dora / use_vera. utils/loftq_init.py exposes validators +
build_loftq_config (lazy peft.LoftQConfig with actionable ImportError
hint). LLaMA Pro: TrainingConfig.expand_layers ∈ [1, 64] +
freeze_trainable_layers (signed, |x| ≤ 1000); cross-validator requires
the pair (LLaMA Pro freezes original layers and trains only new blocks).
field_validator(mode="before") on both rejects bool BEFORE Pydantic ge/le
silently coerces True → 1. expand_model_blocks raises NotImplementedError
with v0.41.1 marker — schema-only release (mirrors v0.27.0 / v0.37.0
stub-then-live pattern). utils/block_expansion._count_layers uses
hasattr(__len__) instead of try/except TypeError so legitimate __len__
bugs surface loudly. use_mod boolean for Mixture-of-Depths (schema only —
live patch deferred to v0.41.1).
Friendly aliases: load_in_8bit / load_in_16bit (Optional[bool]) for
LlamaFactory / Axolotl users. is True policy on both — explicit False is
"no preference", not "off"; mutually-exclusive both-True rejected; alias
combined with explicit Quant Menu format raises rather than silently
overriding. Alias-driven quantization rewrite via direct
self.quantization = ... (Pydantic v2 BaseModel non-frozen path), NOT
object.__setattr__ — code review caught that the latter would silently
bypass any future field_validator on quantization.
Five review agents (4 in parallel + manual smoke for verification-loop):
all CRITICAL/HIGH/MEDIUM/LOW findings fixed. Local smoke caught a real
bug — PyYAML parsing 1e-4 as string — fixed pre-commit with 2 added tests.
5242 tests pass (+120 net new); ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
5e0872b9ea |
feat(trainer): ReLoRA + surgical PEFT non-SFT (v0.40.6 #67)
Extends the v0.39.0 ReLoRA callback (Part B) and surgical PEFT patches (Part D — Gemma4 ClippableLinear swap + 3-D fused-MoE expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, RewardModel, Pretrain, Embedding, BCO). - New shared helper soup_cli/utils/peft_wiring.py exposes apply_pre_lora_patches, apply_post_lora_patches, attach_relora_callback. - SFT migrated to the same helpers in the same release (centralisation invariant; no drift between SFT and non-SFT wiring). - SoupConfig._validate_relora_supported_tasks: task != "sft" rejection removed; MLX backend still rejected with distinct message. Review fixes: - attach_relora_callback uses `if relora_steps is None:` (project policy) so a schema-bypassing relora_steps=0 surfaces as a loud ReLoRAPolicy ValueError rather than a silent skip. - Direct attribute access on tcfg.relora_warmup_ratio / _reset_optimizer / _prune_ratio (Pydantic schema guarantees them); no getattr defaults. - 11 behavioural helper tests (Gemma4 happy path + exception swallow, post-LoRA strip happy + exception swallow, ReLoRA policy field forwarding, schema-bypass loud-fail). - Schema-gate matrix covers `task='preference'` dispatcher. Tests: 5061 -> 5122 (+61). Closes #67. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
30fb8d61f8 |
docs(contributing): correct test file count (149 -> 147)
Inventory audit showed v0.40.4 baseline test-file count was 148 in docs but 146 actually on disk. v0.40.5 added test_v0405_part_a.py (+1), so the correct post-v0.40.5 count is 147. Update CONTRIBUTING.md. Docs-only hotfix — no soup_cli/ changes, no version bump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
697cc8dad7 |
feat(trainer): Quant Menu non-SFT multi-trainer wiring (v0.40.5 #66)
Extends the v0.38.0 train-time quantization menu (gptq / awq / hqq:Nbit /
aqlm / eetq / mxfp4 / fp8) from SFT-only to all 11 transformer-backend
trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel /
Pretrain / Embedding / BCO). Closes the v0.38.0 known gap.
Schema gate: SoupConfig._validate_quant_menu_supported_tasks removes the
`task != "sft"` rejection branch. MLX backend rejection retained with
distinct message; modality=text gate retained (vision/audio Quant Menu
deferred — modality-specific kwargs not yet threaded through the unified
loader).
Trainer wiring (11 sites): each non-SFT _setup_transformers replaces the
inline BitsAndBytesConfig(load_in_4bit=True, ...) block with a call to
build_quantization_config_for_loader(tcfg=tcfg, base=cfg.base, console=console)
— mirrors sft.py:420-440 exactly. kbit-prep tuple widened from
("4bit", "8bit") to ("4bit", "8bit", "mxfp4"). BitsAndBytesConfig import
removed from each non-SFT trainer.
Review fix — PPO reward model: _load_reward_model gains optional tcfg
kwarg; when supplied, the reward checkpoint loads with the same Quant
Menu config as the policy. Both PPO call sites forward tcfg=tcfg.
Defends against silent fp16 OOM on a GPTQ/AWQ/HQQ policy run.
Review fix — defence-in-depth: new TrainingConfig.reward_model field
validator rejects null bytes and caps length at 512 chars (matches
cfg.base policy). The Quant Menu loader's per-call null-byte check
in _check_local_marker remains as the runtime backstop.
Tests: +131 net new (4930 -> 5061). New tests/test_v0405_part_a.py
parametrizes 11 tasks x 7 quant formats; covers MLX rejection per task,
quantization_aware x Quant Menu cross-validator regression for non-SFT,
source-level invariants (no inline BNB literal, kbit tuple regex),
and a live mock-based dispatch test for _load_reward_model proving the
Quant Menu path is reachable when tcfg is supplied and skipped when
tcfg=None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
560d98df8c |
docs: bump v0.40.4 + release notes (CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md)
- pyproject.toml + soup_cli/__init__.py → 0.40.4 - README.md: replace What's New block with v0.40.4 highlights; ## Multipack section updated to "live wiring landed"; expanded ## --trust-remote-code section to list full surface coverage (every command + every trainer task) - SECURITY.md: v0.40.4 added to supported versions; full per-version fix note appended (multi-trainer opt-in pattern, multipack DataLoader override, _get_train_sampler defensive delegate fix, drop_last forwarding, known limitations) - CONTRIBUTING.md: test count 146 files / 4855 tests → 148 files / 4930 tests Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1f050fc235 |
feat(v0.40.3): Stub-to-live wave 1 (#33, #64; #65 still deferred)
Three v0.X.0 deferred-stub features become live runtime — closes #33 (harvester judge filter + serve trace log) and #64 (live CUDA OOM probe). #65 (multipack live wiring in HF Trainer) remains deferred to v0.40.4 after the adversarial 5th-review pass surfaced a Sampler[int] vs list[list[int]] shape mismatch with HF Trainer's DataLoader; helpers (`make_multipack_trainer_class`, `attach_multipack_state`, `lengths_from_dataset`, `detect_arch_name`) ship as a stub used by unit tests, but the SFT/Pretrain wrappers print a yellow advisory and fall back to the standard sampler when `multipack: true`. Live CUDA OOM probe (#64): `make_cuda_probe_fn` builds a closure that runs ONE forward+backward+step on a synthetic batch per candidate. `model.zero_grad(set_to_none=True)` runs BEFORE forward; intermediate ids/attn/labels/outputs are del-ed before `loss.backward()` so peak VRAM reflects a realistic training step (matches v0.35.0 #45 policy). `pad_id` is bounded by `len(tokenizer)` (not `vocab_size`) so extended vocabs (Llama-3 + `<|pad|>`) don't fold pad to a random byte token. SFT-only this release. Trace-to-Preference judge filter (#33 (a)): `judge_filter_pairs` reuses v0.19.0 JudgeEvaluator backends (openai/server/ollama). Threshold rejects bool/NaN/out-of-[0,1]; `_MAX_BATCH=100_000` cap applied via lazy `itertools.islice`; per-pair backend exceptions caught and DEBUG- logged (matches v0.33.0 #47 policy); `judge_provider` validated against the allowlist at the CLI boundary BEFORE constructor with a Rich-escape error message; yellow projected-call-count warning before the loop (2× per pair). Inference Server trace log (#33 (b)): `TraceLogWriter` is thread-safe (single-process lock — multi-worker documented as known limitation); path containment via shared `is_under_cwd`; null-byte/empty/non-string path rejected; cap_mb bounds [1, 10000] with explicit bool rejection. Rotation (one backup retained) refuses symlink at the backup path via `os.lstat + stat.S_ISLNK` (matches v0.33.0 #22 TOCTOU policy). Secret redaction (`hf_*` ≥8, `sk-*` ≥16, `Bearer …` ≥8 with `.` excluded so end-of-sentence period survives) applied to prompt + response and recursively to caller-supplied `extra` dict values. Streaming SSE path also records (was a coverage gap caught in adversarial review). Behaviour change: v0.40.2 users with `auto_batch_size_strategy: probe` were silently getting the static fallback. v0.40.3 actually runs a CUDA probe on first run (~5–30s, cached per (model, max_length, quant, lora_r, gpu) tuple). Reviews: 5 agents (python, code, security, tdd, verification-loop). Verification-loop run twice — once shallow smoke (PASS), once adversarial bug-hunt which found C1/C2 (multipack live wiring crash — demoted to v0.40.4), H1 (streaming SSE missing trace log — fixed), H4 (vocab_size vs len(tokenizer) on extended vocabs — fixed), H3 (Bearer regex consumed trailing period — fixed), H2 (judge cost shock — warning added), M2 (empty lengths accepted — rejected), L1 (extra dict bypassed redaction — recursive walk added). Tests: 4756 → 4855 (+99 net new) across test_v0403_part_a/b/c.py. Lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b0fc586706 |
feat(v0.40.2): Quick polish + v0.40.1 carry-overs (#36, #50, #51 + 7 papercuts)
Closes 3 originally-scheduled GitHub issues plus 7 v0.40.1 long-tail UX papercuts. No new schema fields, no new trainers — pure polish. Originally scheduled: - #36 format_gate_row helper for the eval-gate dashboard row (pure formatter in soup_cli/monitoring/display.py; passed=is True so missing field renders neutral; supports stop/warn action suffixes; multi-task " | " join). - #50 prepare_hf_resume now skips snapshot_download when local checkpoint-N is greater-or-equal to the remote highest-N. New _find_highest_local_checkpoint helper handles missing dirs / OSError / non-directories cleanly. - #51 soup deploy hf-space --template-dir <path> via new soup_cli/utils/hf_space.py:render_custom_template_dir. Containment via is_under_cwd; validate_repo_id BEFORE substitution; per-file 256 KB cap; symlinks + non-regular files rejected (TOCTOU defence per v0.33.0 #22). v0.40.1 carry-overs: - H2: data filter --min-coherence alias; data split --train no-op; data register/unregister positional <name> <path> + Optional --name/--path with conflict detection. - H3: soup quickstart --output DIR (containment-checked) routes data, config, run dir under the chosen directory. - N1/G2: apply_logging_level pushes parsed --log-level tier into the root logger so transformers / peft / trl actually respect QUIET / DEBUG. - N7: shared _resolve_model_source in commands/infer.py (used by bench.py too) — path-like-but-missing raises FileNotFoundError; non-path-like values fall through to HF download via from_pretrained. - G13: verified ONNX/AWQ/GPTQ/TensorRT install hints already correct. - M4: verified data dedup --threshold already exposed. - M5: soup runs --cwd-only + _filter_runs_by_cwd helper using os.path.realpath + commonpath (Windows 8.3 + cross-drive safe). Review-fix follow-ups landed in the same release: - soup_cli/commands/infer.py: from __future__ import annotations (Py3.9 PEP 604 fix); --output containment via is_under_cwd, late-evaluated to preserve pre-existing test contracts. - soup_cli/commands/data.py register_data + soup_cli/commands/bench.py prompts file: Path.resolve()+relative_to() → is_under_cwd (project rule for Windows 8.3 short-name safety). - soup_cli/commands/runs.py: typed _filter_runs_by_cwd, removed redundant inner import os. - soup_cli/commands/deploy.py: confirmation panel now shows --template-dir path when set, not the unused --template default. Tests: 4720 → 4756 (+36) across two new files (test_v0402_part_a.py, test_v0402_part_b.py). 5 review agents (python / code / security / tdd / verification) all clean after fixes. Known limitations: - Custom HF Space templates always create the Space with sdk=gradio regardless of the supplied app.py. Use --template streamlit-chat with the inline registry for Streamlit. Tracked for v0.40.3+. - _resolve_model_source returns ("hf", repo_id) without validate_repo_id; transformers.from_pretrained will raise loudly on malformed ids. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
56bea56c08 |
fix(v0.40.1): QA Hardening — UTF-8 bootstrap, schema strictness, multi-objective preference runtime, CLI UX
Closes the QA findings from the Windows + RTX 3050 4 GB pass (2026-05-07): - Part A: UTF-8 stdio bootstrap on Windows (closes C1/C4/H1/N5/N8/G5) - Part B: root-level `lora:` migrates into training.lora (no more silent init_strategy bypass); multi-objective preference loss runtime no longer raises NotImplementedError (primary-loss approximation; full per-batch weighted combination deferred to v0.40.2) - Part C: autopilot 7B → 1B fallback + safetensors cache probe; transformers <5.0.0 cap with INCOMPATIBLE flag in `soup doctor`; quickstart auto-switches to SmolLM2-135M on ≤6 GB VRAM; --find-lr load_local → load_raw_data import fix - Part D (subset): dynamic --template help (H4); init --force (M2); migrate JSONL friendly error (N2); eval custom -o independent of attach-to-registry + loop-shadow bug fix (G10); history suggests dataset registry (N6); doctor importlib.metadata fallback (M1) + GPU diagnostic distinguishes CPU build (N3) + dual-Python detector (N4) - Part E: recipe fuzzy-match suggestions (M3); sample filename embeds strategy (no overwrite); JSONL BOM auto-strip Net +64 tests (4656 → 4720). 4 review agents clean (python/code/security/tdd). Long-tail UX papercuts (H2/H3/N7/M4/M5 + #36/#50/#51) deferred to v0.40.2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b8506c465a |
feat(preference): v0.40.0 — Preference Variety (4 Parts: BCO + dispatcher + DPO variants + multi-objective)
Part A — BCO Trainer (Binary Classifier Optimization): new task='bco', training.bco_beta, bco.yaml template, train+sweep routing. Internal _split_dpo_rows_to_bco adapts paired DPO input to TRL's BCO unpaired schema; skipped rows logged at DEBUG (mirrors v0.33.0 #47 policy). Part B — Unified preference dispatcher: additive task='preference' + training.preference_loss Literal {dpo,simpo,orpo,ipo,bco}. Legacy task='dpo' / 'simpo' / 'orpo' / 'ipo' / 'bco' remain first-class — the new surface is purely additive, not a breaking collapse. _make_inner_cfg uses model_copy so re-validation never sees an intermediate inconsistent state and the caller's cfg is never mutated. Part C — KL-controlled DPO variants: dpo_beta_schedule (linear / cosine / exponential) + dpo_beta_end + dpo_ref_regen_epochs [1, 1000]. BetaScheduleCallback resolves total_steps lazily in on_train_begin (closes a first-cut bug where total_steps=0 silently emitted beta_end for every step). RefModelRegenCallback uses load_state_dict(strict=True) with WARNING-on-mismatch (closes a first-cut silent partial-copy hazard). Gated to DPO-family tasks only; rejected on mlx backend with distinct error message. Part D — Multi-objective preference_loss_weights (2-5 entries, key allowlist + null-byte rejection, sum-to-1 ±1e-6). Schema-level surface only; live runtime weighted-loss combination deferred to v0.40.1 with NotImplementedError stub-then-live (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA pattern). Net +118 tests (4538 → 4656). All four review-agent waves clean (Python / Code / Security / TDD). Known limitation: BCOTrainerWrapper still hardcodes trust_remote_code=True (carry-over of the v0.36.0 #63 family across non-SFT trainers). Also: add docs/ to .gitignore (internal-only docs going forward; existing docs/QUANTIZATION.md from v0.38.0 stays tracked). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e6a9c087c3 |
feat(lora): v0.39.0 — LoRA Quality (PiSSA + ReLoRA + per-pattern rank + surgical patches + templates registry)
Five PEFT-surface improvements that LlamaFactory and Axolotl maintain: - LoraConfig.init_strategy Literal["random","pissa","olora"]; PiSSA SVD init via PEFT init_lora_weights="pissa". Back-compat: use_olora=True aligns to init_strategy="olora" via dict-copy model_validator(mode="before"); explicit conflict (use_olora=True + init_strategy="pissa"/"random") rejected. Mutual-exclusion vs DoRA / VeRA. - ReLoRA callback (utils/relora.py): frozen ReLoRAPolicy with bounds-checked steps [1, 1e7] / warmup_ratio [0,1] / prune_ratio (0,1) (strict — prevents zero-everything footgun); magnitude_prune_tensor (in-place torch.kthvalue, rejects non-Tensor / single-element short-circuit); duck-typed ReLoRACallback (no transformers import at module load). TrainingConfig fields relora_steps / relora_warmup_ratio / relora_reset_optimizer / relora_prune_ratio. SoupConfig _validate_relora_supported_tasks gates to task=sft + transformers backend with distinct MLX-backend error message; multi-trainer expansion deferred to v0.39.1 (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu stub-then-live pattern). - LoraConfig.rank_pattern / alpha_pattern Optional[Dict[str,int]]; field validator caps at 256 keys × value (0, 1024], rejects bool / null-byte / empty key. Cross-validator rejects with use_vera=True (VeRA shares one rank). peft_builder propagates into LoraConfig init_kwargs. - utils/peft_patches.py: is_gemma4_model uses regex word boundary (?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$) so "ungemma4ed" no longer matches. apply_gemma4_clippable_patch swaps ClippableLinear → nn.Linear by class name (weight-copy fallback logs at DEBUG). strip_lora_dropout_for_3d_experts zeroes lora_dropout.p on 3-D weights (handles ModuleDict variant for PEFT >=0.10). apply_surgical_patches orchestrator validates model_name. Wired into sft.py _setup_transformers with is_gemma4_model gate before the pre-LoRA swap; post-LoRA 3-D dropout strip runs unconditionally (architecture-detected internally). - 16 inline templates migrated to soup_cli/templates/*.yaml + manifest.json + load_template loader (path-traversal-rejecting name validator; os.path.realpath + commonpath containment so a tampered manifest cannot read files outside the package directory; 256 KB file-size cap with inline fallback). Inline TEMPLATES kept with deprecation comment (planned removal v0.41.0+); test_templates_yaml asserts byte-equality of all 16 inline ↔ YAML pairs to prevent silent drift. Net +164 tests (4374 → 4538). All 5 review-agent waves clean before tag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f6f29ef609 |
feat(quant): v0.38.0 — Quant Menu (8 Parts: A-H)
Train-time support for 7 new quantization formats — close the width gap
with LlamaFactory. Wired into SFT trainer + transformers backend + text
modality; multi-trainer/modality expansion deferred to v0.38.1 (mirrors
v0.27.0 MII / v0.37.0 multipack stub-then-live pattern).
- Part A — GPTQ: quantization='gptq' + gptq_disable_exllama (PEFT triton)
- Part B — AWQ: quantization='awq' + GEMM/GEMV builder
- Part C — HQQ: hqq:1bit..hqq:8bit (no 7bit; not supported upstream)
- Part D — AQLM: locked-2-bit
- Part E — EETQ: locked-8-bit
- Part F — MXFP4 + FP8 dequantize-on-load
- Part G — bnb_4bit_quant_storage for FSDP+QLoRA
("crucial for fsdp+qlora" — LlamaFactory quantization.py:178)
- Part H — check_quant_distributed_compat matrix + docs/QUANTIZATION.md.
HQQ/EETQ/AQLM x {FSDP, ZeRO-3} hard-fail; BNB-4bit + FSDP without
quant_storage warns. Wired into commands/train.py startup.
Three new schema validators:
- _validate_prequantized_no_qat — pre-quantized + QAT incompatible
- _validate_bnb_quant_storage_only_with_4bit — silent no-op guard
- _validate_quant_menu_supported_tasks — sft + transformers + text gate
Net: +61 tests (4374 -> 4435). Four review-agent waves clean before tag
(python-review / code-review / security-review / tdd-guide);
verification-loop performed as manual equivalent per CLAUDE.md allowance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
06fbd15ec4 |
feat(multipack): v0.37.0 — Multipack (5 Parts A/B/C/D/E)
FFD bin-packing sampler closes the throughput gap with Axolotl on uneven-length chat data. Five focused parts: - Part A: MultipackBatchSampler (FFD + 18-arch allowlist + loud-fail vs Axolotl silent miss + _MAX_FFD_ITEMS=1M DoS cap) - Part B: schema gate (sft/pretrain only on transformers backend, multipack/packing mutually exclusive, distinct mlx error), build_multipack_sampler_for_lengths helper - Part C: neat_packing 4D attention mask + FA-vs-4D strategy picker, _MAX_MASK_ELEMENTS=2**31 / _MAX_BOUNDARY_SEGMENTS=1M caps - Part D: JinjaTemplateAnalyzer (parse-only AST walker, 128KB cap) - Part E: cross-module property tests (4-seed x 200 samples, 5k stress, FFD-to-4D-mask coherence) All five review-agent waves clean before tag (python / code / security / tdd / verification-loop). Net +125 tests (4249 -> 4374), 121 test files (+5). Live HF Trainer sampler-swap wiring deferred to v0.37.1 (mirrors v0.27.0 MII stub-then-live pattern). Schema gate + helper ship now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
a5540fa1e2 |
feat(correctness): v0.36.0 — Correctness First (4 Parts: A/B/C/D)
Four silent-failure modes Soup had → loud failures, plus a
security default-deny.
- Part A: assistant-only loss masking (default true). Replaces TRL's
multi-turn heuristic with explicit IGNORE_INDEX masking. New
data.train_on_responses_only / train_on_messages_with_train_field
+ per-message train: bool field. Preferred path uses
return_assistant_tokens_mask; fallback uses incremental tokenize
delta with add_special_tokens=False to avoid double-BOS drift.
- Part B: --trust-remote-code opt-in default-deny on soup train /
chat / serve / data download / eval auto. KNOWN_SAFE_PREFIXES
allowlist (15 first-party orgs) suppresses warning panel.
Replaces 9 unconditional trust_remote_code=True call sites in
the SFT path. Non-SFT trainers + diff/export/merge/infer/generate
still hardcode trust_remote_code=True — documented v0.36.x patch.
- Part C: chat-template hardening. Tokenizers without chat_template
raise loudly instead of silent f"{role}: {content}" fallback.
New data.chat_template (registered name or raw Jinja). Filesystem
-touching Jinja directives (include/import/from/macro/extends)
blocked at config-load. Override application warns that soup push
will persist the new Jinja into tokenizer_config.json.
- Part D: OOM-probe auto batch-size. New
training.auto_batch_size_strategy: auto|static|probe. Try-halve
-then-double-to-ceiling loop, max 8 doublings, ceiling = static
× 4. ~/.soup/batch_cache.json (0600 perms, env-override
containment-checked against ~/cwd/tempdir). make_cache_key
rejects bool inputs.
Net +134 tests (4115 → 4249). All 5 review-agent waves clean
before commit; 5 HIGH / 10 MEDIUM / 5 LOW findings fixed in one
review-fix wave.
Smoke: python -m soup_cli.cli version → soup v0.36.0; all 5 new
--trust-remote-code flags surface in --help; ruff clean; pytest
4249 passed / 3 skipped / 0 failed in 2m41s on Windows py3.10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
892fd33f9e |
feat(trainers): v0.35.0 — Trainer Coverage (closes #60, #61, #45)
Wires v0.28.0 speed/memory features into every transformer-backend trainer (grpo / kto / orpo / simpo / ipo / ppo / reward_model / embedding) plus closes the v0.33.0 #43 oversight where dpo / pretrain accepted activation_offloading without installing offload hooks. Auto-quant --auto-quant now forwards the picked candidate's quantization to vLLM via an explicit named parameter (kwarg-splat hazard removed). Kernel auto-compose runs a forward-only benchmark loop on the trainer's actual model under torch.no_grad() so live training gradients aren't polluted (this was a critical-class bug caught by code-review pre-tag and fixed before merge). Schema gate lifted with distinct MLX-backend vs unknown-task error messages so users get the right fix. fp8 / int8 QAT guard fixed in 6 trainers (the legacy unguarded `if tcfg.quantization_aware:` would have crashed the int8 path with the string "fp8"). Net +187 tests (3928 -> 4115). New file tests/test_trainer_coverage_v035.py provides a parametrised matrix proof that every trainer x every feature is exercised on every CI matrix job. All four review-agent waves (python / code / security / tdd) clean with every CRITICAL -> LOW finding fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2cc4b5aa20 |
feat(observability): v0.34.0 — Observability & Dev UX (7 Parts)
Adds soup why, soup tui, soup runs replay, soup train --profile, .crash bundle on training exception, per-run cost in SQLite, --log-level global flag. Net +110 tests (3818 → 3928); all five review-agent waves (python / code / security / tdd / smoke) clean. - Part A: --log-level quiet|normal|verbose|debug → Rich-formatted logger on the "soup" namespace; idempotent + tier-change replaces handler. - Part B: SQLite gains cost_usd / cost_gpu_label via lazy ALTER TABLE (race-tolerant against duplicate-column on concurrent first-boot); rendered in soup runs show / replay / TUI; bool num_gpus rejected; LIKE wildcards escaped in tracker.get_run prefix match. - Part C: soup why — heuristic NaN / plateau / divergence / grad-norm / LR bounds; severity-ordered findings. - Part D: .crash bundle generator with recursive hf_*/sk-*/Bearer redaction, output_dir basename-only, os.path.realpath containment, secrets.token_hex filename, ValueError (not PermissionError) on outside-cwd; train.py except-handler writes the bundle without masking the original exception. - Part E: soup runs replay <id> — summary panel + downsampled loss curve (≤2000 points) from SQLite history. - Part F: soup train --profile — torch.profiler Chrome trace to <output>/profiles/<run_id>.trace.json; run_id rejects '.', '..', '/', '\\', null bytes; profiles dir created only on torch import. - Part G: soup tui — Textual dashboard with lazy ExperimentTracker import; markup_escape on every DB-sourced string; new [tui] extra. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e248b5aea8 |
fix(v0.33.0): missed checklist items (CONTRIBUTING + Unicode arrow)
Pre-push audit found 3 gaps in the Release Checklist (steps 6, 11):
1. soup_cli/commands/can.py:118 used → (->) in the run_cmd
docstring. Windows cp1252 in non-PYTHONUTF8 mode raises
UnicodeEncodeError when Typer renders --help. Replaced with
ASCII '->'. Verified: `python -m soup_cli.cli can run --help`
renders cleanly on Windows.
2. CONTRIBUTING.md test counts not updated:
- tree comment: 97 files, 3696 tests -> 104 files, 3818 tests
- test table: +7 rows for test_part_{f,a_wave1,a_wave2,e,d,c,b}
- directory tree: added registry/attach.py, cans/run.py,
cans/publish.py, data/collators.py, utils/v028_features.py
3. README.md:967 stale "v0.32.0 stub" note. Updated to reflect that
--find-lr now runs the live loop in v0.33.0 and that spike-recovery
writes a JSON hint while live optimizer rewind / DataLoader rebuild
remain follow-ups.
examples/README.md: skipped per checklist step 12 — v0.33.0 added no
new YAML configs (the new commands operate on .can files which use
existing config schemas).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
8878eb1aa6 |
feat(training): v0.32.0 — Training Stability & Auto-Tuning
Seven new opt-in flags that turn Soup into the "fast.ai of LLM fine-tuning" — pre-flight LR range finder, auto warmup schedule, auto mixed-precision, loss-spike auto-recovery, convergence detector, VRAM-pressure advisory, and autopilot integration. * Part A: soup train --find-lr / utils.lr_finder (TypedDict result, abs-divergence threshold, NaN/Infinity rejection, MAX_NUM_STEPS=10000, is_under_cwd containment on --find-lr-output). * Part B: utils.grad_accum (GradAccumMonitor + MAX_ACCUM=1024 cap; preserves effective batch on recommend()). * Part C: utils.mixed_precision (KNOWN_PRECISION_QUIRKS map, longest- substring iteration so qwen2.5/qwen2 + phi-3.5/phi-3 are deterministic; 200-char model-name cap, null-byte rejection). * Part D: utils.warmup (compute_warmup_steps clamped [10, 1000]; ratio==0 short-circuit matches HF Trainer "no warmup" convention). warmup_auto field reuses pre-existing warmup_ratio (no duplicate). * Part E: utils.spike_recovery (frozen dataclass policy; max_attempts<=10; min_lr floor) + schema cross-validator requiring loss_watchdog=true. * Part F: utils.convergence (detect_plateau + recommend_action; the latter reuses the former so plateau heuristic stays single-source). * Part G: autopilot.decide_warmup / decide_mixed_precision wrappers; generate_config validates BOTH the YAML output path AND embedded decisions["output"] via shared utils.paths.is_under_cwd. Tests: tests/test_auto_tuning.py — 89 tests covering bound boundaries, NaN/Infinity rejection, multi-version quirk ordering, frozen-dataclass post-construction validation, plateau non-positive-mean guard, and double-containment in generate_config. Total: 3607 -> 3696 tests passing. ruff clean. Review wave: python-review (8 findings), security-review (3), code-review (8 incl. duplicate warmup_ratio HIGH and synthetic stub-loss curve), and tdd-guide (11 coverage gaps) — every finding fixed before commit. Live in-process wiring (LR-sweep training loop, spike rollback, grad-accum DataLoader rebuild, SFT precision push) is deferred to v0.32.1 — same advisory pattern as v0.30.0 --auto-quant / structured-output. Stale-install gotcha: if `soup version` shows the old version after pulling this branch, run `python -m pip install -e . --force-reinstall --no-deps`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
d0b7719858 |
feat(recipes): v0.31.0 — Model & Recipe Breadth
Expand the recipe catalog from 46 to 80 entries — every popular open-weight model family now has a validated Soup recipe. Part A — Vision (6 recipes): Llama-3.2-Vision-90B, Pixtral-12B, Qwen2-VL (7B + 72B), InternVL 2.5, MiniCPM-V 2.6 Part B — Audio (3 recipes): Qwen2-Audio, SeamlessM4T v2, Whisper-large-v3 Part C — Reasoning (7 recipes): completes the 6 DeepSeek-R1-Distill sizes, plus Qwen3-Coder, Qwen3-30B-A3B reasoning, Phi-4 reasoning Part D — Edge (8 recipes): SmolLM2 (135M / 360M / 1.7B), Qwen2.5 (0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini Part E — Domain (8 recipes): BioMistral, Meditron, CodeLlama (13B / 70B), Magicoder, Mathstral, Nemotron-4 340B, Llama-2-13b-finance Part F — Multimodal reasoning (2 recipes): Llama-3.2-Vision GRPO, Pixtral DPO Part G — Recipe-validation CI workflow on every PR touching recipe / config / data code (.github/workflows/recipe-validation.yml) Part H — 750 parametrized tests covering catalog-wide invariants: model-id safety (no `..`/`://`/null bytes), lora.target_modules non-empty, max_length within schema bounds, GRPO recipes wire reward_fn + num_generations >= 2, vision recipes set image_dir, audio recipes set audio_dir, default data path is non-empty + relative Live 100-step per-recipe smoke train (requires GPU runner) deferred to v0.31.1. Tests: 2886 → 3607 (+721). Catalog: 46 → 80 (target met). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e509614a9f |
feat(inference): v0.30.0 — Inference Excellence
- Prefix caching (--prefix-cache) wired to vLLM enable_prefix_caching.
- Speculative decoding auto-pairing (--auto-spec) with curated target→draft
map for Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek
V3/R1, Gemma 2/3. Targets without a known draft (≤8B) fall through.
- Dynamic LoRA hot-swap: POST /v1/adapters/activate/{name} + /deactivate.
Name regex enforced by FastAPI Path(pattern=…); active state protected
by threading.Lock; CORS restricted to loopback origins (hot-swap mutates
state without auth).
- Structured output: --structured-output json|regex with --json-schema
(cwd-confined via is_under_cwd; 64KB cap; top-level type required) and
--regex-pattern (2048-char cap, null-byte reject, must compile).
Constrained token sampling deferred to v0.30.1; constraint descriptor
exposed on app.state.
- Continuous-batching dashboard: --dashboard + /metrics endpoint. Thread
safe ServerMetrics with bounded deque for latencies; record_latency
runs in finally so failure paths are not a blind spot.
- OpenTelemetry tracing: --trace --trace-endpoint. OTLP endpoint SSRF
hardened matching v0.29.0 HF_ENDPOINT (scheme allowlist, 0.0.0.0
rejected, RFC1918/link-local/cloud-metadata via ipaddress.ip_address,
plain HTTP loopback-only). build_tracer idempotent — only installs
provider when current is ProxyTracerProvider/NoOpTracerProvider. Span
context via contextlib.ExitStack so __exit__ receives real exc info.
- Auto-quant picker API: --auto-quant flag + Candidate dataclass +
pick_best() shipped (generator-safe, first-wins tie-break, matches
v0.28.0 kernel_picker precedent). Live eval loop deferred to v0.30.1;
flag prints a yellow deferral warning so it is never a silent no-op.
Tests: 2801 → 2886 (+85). New tests/test_inference_advanced.py covers
all 7 parts plus review-driven negatives: Llama 3.3/4 pairing, Mistral
Large, DeepSeek V3/R1, Gemma 3 targets; /v1/adapters/deactivate;
activate-when-no-adapters → 404; metrics concurrent track_request;
OTLP private-IP / 0.0.0.0 / missing-host; pick_best empty-list and
all-failed; NaN score/latency rejection; --json-schema outside cwd;
--structured-output json without schema; --auto-quant deferral warning.
Review agents run (python-review, security-review, code-review, tdd-guide
+ manual verification-loop) and every finding addressed:
- HIGH: pick_best iterator exhaustion (materialise to list).
- HIGH: --auto-quant silent no-op (yellow warning).
- HIGH: --structured-output json silent no-op (fail-fast requires schema).
- HIGH: --json-schema path traversal (is_under_cwd containment).
- HIGH: OTel span swallowed HTTPException (ExitStack migration).
- HIGH: wildcard CORS on unauthed POST (loopback-only regex).
- MEDIUM: OTLP RFC1918/link-local rejection.
- MEDIUM: tracer provider idempotency.
- MEDIUM: record_latency in finally (no failure-path blind spot).
- LOW: active_state threading.Lock; deque replaces O(n) pop(0).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
03ddc05573 |
feat(hf): v0.29.0 — HuggingFace Hub Deep Integration
Auto-push checkpoints, HF Collections, self-hosted endpoint, HF datasets push, HF Spaces auto-deploy, model card v2. - utils/hf.py: single source of truth for HF token resolution (env > cached login), HF_ENDPOINT validation, repo_id / collection_slug validation, HfApi factory, add_to_collection. HF_ENDPOINT SSRF-hardened: scheme allowlist, 0.0.0.0 rejected, plain HTTP limited to loopback, RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via ipaddress.ip_address. - monitoring/hf_push.py: HFPushCallback pushes each save_steps checkpoint as 'checkpoint-<N>' branch. Sticky _repo_failed flag short-circuits retries after hard failure. prepare_hf_resume enforces cwd containment and passes local_dir_use_symlinks=False. allow_patterns whitelist (safetensors/bin/pt/json/tokenizer*/trainer_state.json) keeps .env and source files out of auto-pushed branches. - commands/push.py: --collection flag, generate_model_card_v2 (task / base / lr / optimizer from training_config.yaml; optional eval scorecard; markdown-active chars neutralised on task names and non-numeric scores; data_lineage HTML-escaped). --model cwd containment, repo_id validation, deprecated --token warning, commit message stripped to first 200 chars. - commands/data.py: soup data push --input --hf-dataset uploads local JSONL as HF dataset. Cwd containment on input, repo_id validation. - commands/deploy.py: soup deploy hf-space --model --space --template [gradio-chat|streamlit-chat]. render_space_template validates model repo id before substitution into rendered app.py (defeats Python injection from a crafted repo id). - commands/train.py: --push-as <repo> attaches HFPushCallback to trainer_wrapper.trainer after setup. --hf-resume pulls latest checkpoint branch into output_dir before training. Tests: +100 tests in test_hf_integration.py (65 initial + 35 review- driven) covering all parts plus validate_collection_slug negatives, build_push_callback factory paths, on_train_begin lifecycle, repo-failed short-circuit, private-IP SSRF (10.x/172.16.x/192.168.x/169.254.x/ 0.0.0.0), resolve_token edge cases. Full suite: 2801 tests pass. Reviews: python-review, code-review, security-review, tdd-guide, verification-loop — every HIGH / MEDIUM / LOW finding addressed. Docs: README '## HuggingFace Hub Deep Integration' section added; What's New replaced. CLAUDE.md / SECURITY.md / CONTRIBUTING.md updated with new test count (93/2677 -> 94/2801) and v0.29.0 security entries. License migration (MIT -> Apache-2.0) known-limitation note surfaced in What's New per plan.md deferral from v0.27.0. |
|
|
Alpamys
|
e09a742167 |
feat(training): Training Speed & Memory — CCE, FP8, grad-ckpt tiers, kernel picker, cross-doc attn, activation offload (v0.28.0)
Six new training speed/memory features, SFT-only in v0.28.0: - use_cut_ce: Cut Cross-Entropy for 128k-vocab models (8-24GB save) - quantization_aware: "fp8" — Hopper+ float8 training via torchao.float8 - gradient_checkpointing: bool | selective|medium|full|auto (VRAM-based auto) - kernel_auto_compose: benchmark + pick fastest kernel combo - packing_cross_doc_attn_mask: block-diagonal mask for sample packing - activation_offloading: cpu|disk saved-tensor offload Config-load validator rejects non-SFT tasks when speed/memory flags are set — prevents int8-QAT-wrapper crash on the string "fp8" and silent no-ops on DPO/GRPO/KTO/ORPO/SimPO/IPO/PPO/Pretrain/Reward/Embedding. Multi-trainer wiring tracked for v0.28.1. Security: - FP8 path: CUDA + Hopper+ SM capability + transformers backend - Activation-offload disk: is_under_cwd containment, TOCTOU-safe mkstemp (fd held through torch.save), weights_only=True reload, crash-safe cleanup - Kernel picker raises when all candidates lack finite time_ms - Cut CE detector matches last path component only (deepseek-ai/...-phi-... org-prefix does not trigger Phi patch on DeepSeek) - Cross-doc mask numpy-vectorised (np.tril) at max_length=1M - @model_validator gates: packing_cross_doc_attn_mask requires packing=true; v0.28.0 features require task=sft New optional extra: pip install 'soup-cli[cce]' Tests: 2585 -> 2685 (+100 in tests/test_training_speed.py, +1 file). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b7abd2a40c |
feat(v0.27.0): Multi-GPU Mastery — topology, ZeRO++, FSDP2+compile, MII, PP, recipes
- Topology detector: NVLink/PCIe sniffing via NVML (utils/topology.py); --gpus auto|N on soup train prints the exact accelerate-launch command with topology info; NCCL env hints applied via os.environ.setdefault so user/launcher overrides always win. - Accelerate launcher: build_accelerate_argv with shlex.quote on script args; num_processes bounded, mixed_precision Literal allowlist, num_machines bounded [1, 256]; is_in_distributed() recognizes torchrun + accelerate env markers (utils/launcher.py). - ZeRO++: new DeepSpeed preset (zero++/zero_pp aliases) with hierarchical partitioning + quantized weights + quantized gradients; int(1e9) for sub_group_size / stage3_max_* so DeepSpeed strict JSON accepts. - FSDP2 + torch.compile: training.use_fsdp2_compile bool wired into TrainingArguments(torch_compile=True) via extracted apply_fsdp_training_kwargs helper in utils/fsdp.py; validator requires FSDP + CUDA + backend=transformers + torch>=2.2/accelerate>=0.27, and rejects DeepSpeed+compile coexistence (cryptic runtime crash prevention). - DeepSpeed-MII backend scaffold: soup serve --backend mii registered with dependency check; live pipeline wiring deferred to v0.27.1 with explicit non-zero exit to prevent silent mis-start (utils/mii.py). - Pipeline parallelism config: training.parallelism Literal[data|pipeline] + pipeline_stages bounded [1, 16]; validator enforces stages >= 2 + CUDA + gpu_count >= stages; execution wiring deferred to v0.27.1 with Rich Panel notice (utils/pipeline.py). - Recipes: llama3-70b-fsdp2, qwen3-32b-zeropp, deepseek-v3-pipeline; recipe count 43 -> 46. - Tests: 2511 -> 2585 (+74). New test_multi_gpu.py covers topology, launcher, ZeRO++, FSDP2+compile helper behavior, DeepSpeed+compile mutual exclusion, MII key-absent + stub branches, pipeline config bounds, NCCL setdefault semantics, and CLI validator gating (asserts use_fsdp2_compile / parallelism=pipeline on CPU blocks soup train with specific error text). - Reviews: 4-agent initial wave + 3-agent re-review wave on the wiring delta. All CRITICAL/HIGH/MEDIUM/LOW findings fixed (no remaining debt). Security review clean. Known limitations (tracked as v0.27.1 issues): - Auto-reexec of accelerate-launch (currently advisory) - Live DeepSpeed-MII server pipeline - Live pipeline-parallel execution - Multi-node accelerate config - Recipe CI smoke-train validation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
0034628b03 |
docs: drop internal Part A/B/C/D/E labels from public docs
"Part A/B/C/D/E" is our internal decomposition (tracked in .claude/plan.md and referenced in commit messages + GitHub issues). It leaked into user-facing docs during v0.26.0 release prep. Users don't care about our internal breakdown — they care about features and versions. Cleanup: - README.md: "New in v0.26.0" bullets now describe features by name only, (vX.Y.Z) version tags retained where present - SECURITY.md: v0.26.0 hardening entries grouped by feature name, not Part - CONTRIBUTING.md: module tree annotations use (v0.26.0) not (v0.26.0 Part X) .claude/CLAUDE.md: added explicit rule under Release Checklist terminology stating that Part X labels are internal-only and must NOT appear in public docs. Prevents the same mistake next release. .claude/plan.md + commit messages continue to use Part X — that's the correct venue for internal dev decomposition. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ddab34115c |
feat(v0.26.0): Parts B-E — Eval Gate, Trace-to-Pref, Quant-Check, Soup Cans
Closes the v0.26.0 "Red and Blue Ocean" flywheel after Part A (Registry): Train (eval-gated) -> Registry -> Deploy (quant-check) -> Trace-to-Pref -> Train. Part B — Eval-Gated Training: - soup_cli/config/schema.py: EvalGateConfig (enabled/suite/every_n_epochs/ regression_threshold/baseline/on_regression) + TrainingConfig.eval_gate field - soup_cli/eval/gate.py: EvalSuite, GateTask, run_gate, resolve_baseline, load_suite; baselines from registry:// or file - soup_cli/monitoring/callback.py: on_epoch_end + _run_eval_gate with fail-safe error handling (structured errors treated as regressions under on_regression=stop) - soup_cli/commands/train.py: --gate <suite.yaml> shortcut flag - soup_cli/commands/eval.py: gate subcommand (stub generator; live scoring v0.26.1) Part C — Trace-to-Preference: - soup_cli/data/traces/: parse_langchain, parse_openai, parse_soup_serve; build_pairs from thumbs_up / regenerations / user_edit - soup_cli/commands/data.py: from-traces + review subcommands - PII warning panel, 100,000-line cap, path containment, Literal validation Part D — Quant-Lobotomy Checker: - soup_cli/eval/quant_check.py: classify_delta (OK/MINOR/MAJOR), run_quant_check, resolve_model_ref with artifact kinds filter, table/json/markdown renderers - soup_cli/commands/eval.py: quant-check subcommand Part E — Soup Cans: - soup_cli/cans/: Manifest + DataRef (Pydantic v2); pack_entry + fork_can (100MB cap, dunder-key guard); safe tar extraction (filter='data' on py3.12+, narrow fallback, manual symlink rejection + commonpath check) - soup_cli/commands/can.py: pack/inspect/verify/fork subcommands Shared utility: - soup_cli/utils/paths.py: single is_under_cwd helper replacing 5 duplicates (os.path.realpath + commonpath — Windows 8.3 short-name safe) Tests: 103 new (29 eval_gate + 24 trace_to_pref + 23 quant_check + 27 cans) Full suite: 2511 passed on Windows Python 3.10. Security hardening (review-driven, all severities fixed): - EvalGateConfig bounds; GateTask null-byte + judge URL scheme allowlist - Narrow except in _safe_extract so TarError from filter='data' is not swallowed - resolve_model_ref artifact kinds filter (avoid wrong artifact) - Manifest.author cap + null/newline rejection; created_at ISO-8601 validation - fork_can dunder-key + null-byte rejection (prototype pollution prevention) - fork_can size cap (100MB matches pack_entry) - inspect_can/read_config refuse paths outside cwd Docs: - README.md: v0.26.0 "New in" block (flywheel); 43 recipes; all new commands in All Commands list; version examples bumped to 0.26.0; Windows-safe arrows - CLAUDE.md: architecture + test table + schema + CLI + security section extended with B/C/D/E; phase vs Part terminology clarified; release checklist step 18 adds Known Limitations section; step 20 adds comment template; step 21 adds completeness check via gh issue list --milestone - SECURITY.md: per-Part security notes (B/C/D/E) under v0.26.0 - CONTRIBUTING.md: test count + directory tree updates Local smoke: version, eval gate, eval quant-check (table + json), data from-traces, data review, can pack/inspect/verify/fork — all happy-path end-to-end. Fixed Unicode arrows (U+2192) in can.py + gate.py that crashed on Windows CP1252 consoles. Deferred to v0.26.1 (known limitations, filed as issues post-release): - eval gate/quant-check live model scoring (stub generator currently) - data from-traces quality.py judge validation; serve --trace-log collector - can run + can publish + orchestrator - eval --attach-to-registry flag; export auto-artifact registration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
4cd4bab969 |
feat(registry): add Local Model Registry / Provenance Vault (v0.26.0 Part A)
Foundation of v0.26.0 "Red and Blue Ocean" — every fine-tune is now tracked with lineage, config, eval baseline, and shippable artifacts. New module soup_cli/registry/: - hashing.py: deterministic SHA-256 of config (canonical JSON) + data (streamed) + base model; used as the entry_hash identity - store.py: SQLite store (~/.soup/registry.db) with registry_entries, registry_artifacts, registry_lineage, registry_tags. Context-manager API, cycle-safe BFS walks, AmbiguousRefError on prefix collision, LIKE-wildcard-escaped search + resolve, FK ON DELETE CASCADE. - diff.py: flat-walk ConfigChange diff + per-benchmark eval delta. New CLI commands: - soup registry push/list/show/search/diff/promote/delete - soup history <name> — lineage DAG tree viewer Security hardening (v0.26.0): - name/tag validation: alphanumeric + _-. only, null-byte rejected, name ≤128, tag ≤64 - artifact path containment via os.path.realpath + commonpath (Windows 8.3 short-name safe); enforce_cwd=True default - SQL parameterised; LIKE wildcards %/_ escaped with ESCAPE '\' - DB 600 perms on POSIX; SOUP_REGISTRY_DB_PATH env override - indirect-cycle detection in add_lineage via BFS ancestor walk - Rich markup escaped in all CLI output - resolve() raises AmbiguousRefError instead of silent None Tests: 92 new tests in tests/test_registry.py (hashing, validation, CRUD, artifacts, lineage + cycle, diff, CLI, history, security, auto-register integration with ExperimentTracker). Full suite: 2409 passed (was 2313). All review findings addressed (4 agents: python, code, security, tdd): HIGH: context manager + try/finally cleanup, FK cascade (removed manual cascade), cycle detection, LIKE wildcard escaping. MEDIUM: ambiguous resolve raises, exit 0 on user cancel, cwd captured at construction, enforce_cwd=True default, Windows ASCII-safe error messages. Deferred to v0.26.1: soup eval --attach-to-registry flag and soup export auto-artifact registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e4c3042a56 |
feat(v0.25.0): Beyond the Wrapper — 8 major features
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators: Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo, Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA). Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with detection + normalization, synth data template, init template, eval scoring (tool_call_match / tool_call_name_match / tool_call_args_subset), plus qwen3-8b-tools and llama4-scout-tools recipes. Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap, one-time warning panel), and json_schema_reward. verifiable_domain Literal validated via model_validator. Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with mutual-exclusion validator and a unified peft_builder helper that returns either LoraConfig or VeraConfig with the right init kwargs. Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx, MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at config load time by SoupConfig._validate_mlx_task_support, lazy trainer registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject. Part F — Data augmentation: soup data augment with rephrase / translate / style strategies, path-traversal-protected input/output, count capped 1-10, lang/styles lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup. Part G — Training intelligence: forgetting detection (ForgettingDetector with 3 built-in mini benchmarks and warning levels) and checkpoint intelligence (CheckpointTracker with composite metric, early-stop on regression, safe top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema extended with checkpoint_quality + forgetting_eval tables. Part H — Autopilot: soup autopilot command with dataset/model/hardware profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf flags), YAML generator, and full CLI with dry-run + --yes + path-traversal protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting detection + checkpoint intelligence + early-stop into the generated config. Totals: - 2313 tests passing (183 new, up from 2130) - 86 test files (8 new) - 43 ready-made recipes (14 new) - 16 built-in templates (tool-calling added) - Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU, MLX training integration test requires real hardware) Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
Salil Mhatre
|
de5505c7d8
|
feat(doctor): add RAM and disk space checks to soup doctor command wi… (#7)
* feat(doctor): add RAM and disk space checks to soup doctor command with tests and updated docs * fix(doctor): resolve subprocess type checker error by manually validating macOS RAM query return code |
|
|
Alpamys
|
d83dad0a3b |
docs: update CONTRIBUTING.md for v0.24.0, add CODEOWNERS
- Update test counts to 74 files / 2061 tests (was 62 / 1789) - Add complete test file table matching CLAUDE.md - Sync PR checklist with .github/pull_request_template.md - Add Good First Issues section and New Recipe guide - Add Conventional Commits format for commit messages - Add CODEOWNERS for auto-reviewer assignment |
|
|
Alpamys
|
1b1d679141 |
feat: v0.21.0 — migrate, recipes, NEFTune, rsLoRA
- `soup migrate` — import configs from LLaMA-Factory, Axolotl, Unsloth notebooks (AST-only .ipynb parsing, path traversal protection) - `soup recipes` — 30 ready-made configs for popular models (list/show/use/search with path traversal protection) - NEFTune (`neftune_alpha`) — noisy embeddings for SFT/DPO/KTO/ORPO/SimPO/IPO - rsLoRA (`use_rslora`) — rank-stabilized LoRA scaling in all 11 trainers - Fix: `soup doctor` torchvision circular import crash - Fix: `load_eval_tasks()` now accepts str in addition to Path - Security: Rich markup injection prevention in migration warnings - Security: 10 MB file size limit on migration input files - 1789 tests, 62 test files, 64% coverage |
|
|
Alpamys
|
c46265fd18 |
feat: add eval platform with custom evals, LLM judge, human eval, leaderboard (v0.19.0)
Full-featured evaluation system with 7 subcommands: - soup eval benchmark: standard benchmarks via lm-evaluation-harness - soup eval custom: custom JSONL eval tasks with 4 scoring modes - soup eval judge: LLM-as-a-judge (OpenAI/Ollama/server backends) - soup eval auto: automatic post-training evaluation from config - soup eval compare: side-by-side eval comparison with regression detection - soup eval leaderboard: local model leaderboard with JSON/CSV export - soup eval human: terminal A/B comparison with Elo ratings New modules: soup_cli/eval/ (custom.py, judge.py, human.py, leaderboard.py) Config: EvalConfig added to schema.py (auto_eval, benchmarks, custom_tasks, judge) Callback: SoupTrainerCallback.on_train_end triggers auto-eval when configured Security: SSRF protection on judge API, ReDoS guard on regex scoring, API key isolation per provider, 10k task/prompt caps, read-only SQL queries 1585 tests, 58 test files, ruff clean |
|
|
Alpamys
|
3d66b41d00 |
v0.17.0: data quality filters, audio modality, SGLang backend, server provider
New features: - soup data filter: quality filters with perplexity and coherence scoring - modality: audio — Qwen2-Audio, Whisper fine-tuning with audio data format - --backend sglang for soup serve (SGLang high-throughput inference) - --provider server for soup data generate (local OpenAI-compatible servers) - Audio template: soup init --template audio Security hardening: - Server provider SSRF validation (scheme whitelist, localhost-only HTTP) - Audio file path traversal protection (resolved paths confined to audio_dir) - trust_remote_code warning panels for audio models and SGLang runtime 1348 tests, 56 test files, 58.8% coverage, ruff clean. |
|
|
Alpamys
|
cbc0a0e558 |
v0.16.0: embedding models, ONNX/TensorRT export, speculative decoding
New features: - task: embedding — fine-tune sentence embedding models (BGE, E5, GTE) with contrastive, triplet, or cosine loss and configurable pooling - soup export --format onnx — ONNX export via optimum - soup export --format tensorrt — TensorRT-LLM export for GPU inference - soup serve --speculative-decoding — draft model for 2-3x faster generation (transformers assisted generation + vLLM native speculative decoding) - soup init --template embedding — new template for embedding fine-tuning Security: - ONNX export: removed unconditional trust_remote_code, added warning - Speculative decoding: SSRF protection (URL blocked), warning panel - vLLM speculative: URL validation rejects http:// schemes - TensorRT export: separated try/except per subprocess call - Embedding config: Literal constraints, margin gt=0 validation 1270 tests, 52 test files, 58% coverage |
|
|
Alpamys
|
bee13c22f0 |
docs: update SECURITY, CONTRIBUTING, examples README to v0.15.0
- SECURITY.md: supported versions updated to v0.15.x, added v0.15.0 hardening history - CONTRIBUTING.md: utils list updated with new modules, test count 1182, templates 13 - examples/README.md: added long-context fine-tuning section (#8) |
|
|
Alpamys
|
1718578a1a |
test: add subprocess CLI tests + cross-platform CI matrix
- Add test_cli_subprocess.py (69 tests): real subprocess execution testing entry points, encoding, paths, Unicode, platform regressions - CI matrix: ubuntu/windows/macos × Python 3.9/3.11/3.12 (9 jobs) - CI: add coverage reporting with Codecov upload - Update CLAUDE.md and CONTRIBUTING.md test counts (1022 → 1091) |
|
|
Alpamys
|
f5ad0f5a45 |
docs: update SECURITY, CONTRIBUTING, examples README to v0.14.3
- SECURITY.md: update supported versions to v0.14.x, add security hardening history - CONTRIBUTING.md: update test counts (47 files, 1022 tests), add all trainers, fix project structure - examples/README.md: add KTO/ORPO/SimPO/IPO, pre-training, MoE, batch inference sections - CLAUDE.md: add SECURITY/CONTRIBUTING/examples to release checklist |
|
|
Alpamys
|
1b949017dc |
Fix placeholder usernames and outdated version in docs
- CONTRIBUTING.md: YOUR-USERNAME → MakazhanAlpamys - README.md: your-username → MakazhanAlpamys in push examples - README.md: v0.4.0 → v0.10.0 in version --full example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |