mirror of https://github.com/razor-ai/soup.git
148 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
82d5693b75 |
feat(eval): Tracker & Eval Pro — 18 features (v0.43.0)
Closes the observability gap with all three competitors in one release.
Part A — Trackers
* --tracker flag (mlflow/swanlab/trackio) on soup train, mutually
exclusive with --wandb/--tensorboard. Closed allowlist via
MappingProxyType. Live integrations rely on HF Trainer's report_to.
* SOUP_TELEMETRY=1 opt-IN env var; build_telemetry_payload schema is
closed-key (no model names / dataset paths / config contents). Live
PostHog network code deferred to v0.43.1.
Part B — Eval metrics
* Pure-Python BLEU + ROUGE-1/2/L + effective_tokens_per_second.
* KL-divergence calibration framework with OK/MINOR/MAJOR thresholds.
* Model Arena Elo tournament (256-model cap, MappingProxyType view,
Rich-markup metacharacter rejection on names).
* ceval / cmmlu / aider_polyglot benchmark allowlist (live Aider
runner deferred to v0.43.1).
Part C — Profiling
* memory_snapshot_context (narrow RuntimeError catch — review fix
prevents generator-already-executing on user-body RuntimeError).
* detect_anomaly_context, nccl_bandwidth_check (h100/a100/v100/rtx
reference table; live measurement CLI surface deferred).
* write_vscode_launch with TOCTOU symlink rejection at the target
path regardless of force=True.
Part D — Demo bundles
* `soup data demo` lists / copies 4 bundled JSONL fixtures
(alpaca / sharegpt / dpo / grpo) with staged-tempfile atomic
rename + 50 MB cap + symlink rejection on the staging path.
Tests: 5389 -> 5628 (+239). Ruff clean. Five sequential review waves
(python / code / security / tdd / smoke) ran; HIGH/MEDIUM/LOW findings
all fixed including: tracker name shadow in train.py, _lcs_length DP
double-buffer bug, BLEU geo-mean policy, base_dir absolute/.. escape,
demo_bundles tmp symlink TOCTOU, vscode launch symlink TOCTOU.
Note (Windows CI): line-ending warnings (LF -> CRLF) on commit are
expected; `.gitattributes` policy is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
05093ebfdb |
feat(data): Data Pipeline Pro — 18 features, axolotl + LF parity (v0.42.0)
Schema-first surface for the data pipeline gap with Axolotl + LlamaFactory. Ships in one release: 5 new formats (prm, pre_tokenized, input_output, video, multimodal), remote URI allowlist (s3/gs/gcs/az/abfs/abfss/oci) + streaming + sharding, AOT preprocess cache + `soup data preprocess` CLI, multi-dataset interleave (concat/under/over/probs) + 8 advanced masking fields, vocab expansion (add_new_tokens / new_special_tokens / resize_vocab) + custom prompt_strategy, and document ingestion (`soup data ingest` for PDF/DOCX/MD/TXT). Live wiring for fsspec backends, AOT tokenize loop, custom prompt-strategy runtime, and PRM trainer integration is deferred to v0.42.1+ (stub-then-live pattern from v0.27.0 / v0.37.0 / v0.41.0). Schema gates fire at config load so misconfiguration fails fast. Security: full v0.42.0 hardening matrix — `_REMOTE_SCHEMES` MappingProxyType allowlist; bucket regex 1-63 chars per S3/GCS spec; userinfo / fragment / query-string rejection on remote URIs (query-string forwarded to fsspec is SSRF-adjacent); null-byte + length caps on every string-shaped input; bool-rejected-before-int on every numeric input; frozen InterleaveSpec dataclass; 10k caps on add_new_tokens; `is_under_cwd` containment on video_dir + tokenized_path schema fields and on preprocess --config / both ingest paths; `os.lstat + S_ISLNK` symlink rejection on ingest input; PRM converter type-checks completions (str) + labels (bool, not int); video field null-byte + 2KB cap; field-name threading on image-pixels validator so error messages name the actual field. 5242 tests → 5389 (+147 net). 11 review findings addressed across python-review / code-review / security-review / tdd-guide (CRITICAL→LOW). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1491025a36 |
feat(trainer): Optimizer & PEFT Zoo (v0.41.0)
Optimizer Zoo (Part A): closed-allowlist SUPPORTED_OPTIMIZERS adds 14 new
entries — BAdam, APOLLO (apollo_adamw), Adam-mini, lomo / adalomo,
grokadamw, schedule_free_adamw / schedule_free_sgd, muon, dion,
came_pytorch, ao_adamw_{fp8,4bit,8bit}. Validates name + lower-cases
deterministically; rejects non-string / empty / null-byte / >64-char.
_OPTIMIZER_PACKAGES wrapped in MappingProxyType (matches v0.36.0 _REGISTRY).
Per-module LR (Part B): training.lr_groups accepts list-of-pairs /
list-of-dicts / {pattern: lr} mapping; canonical [{pattern, lr}, ...]
storage. Capped at MAX_LR_GROUPS=32; per-pattern non-empty string
≤256 chars + null-byte rejection + re.compile + best-effort ReDoS probe;
per-LR (0.0, 1.0] + math.isfinite (rejects NaN AND ±inf) + bool rejection
(matches v0.30.0 Candidate policy); duplicates rejected. lr_groups_from_schema
bridges canonical schema shape into runtime List[LrGroup] for
build_optimizer_param_groups (first-match-wins routing). LrGroup is
@dataclass(frozen=True). PyYAML scientific-notation (1e-4) parses as
string in YAML 1.1; _validate_lr coerces str → float so soup.yaml
round-trips work. base_lr rejects bool / non-positive (defence-in-depth).
PEFT methods (Part C): LoraConfig.init_strategy="loftq" + loftq_iter
∈ [1, 10] + loftq_bits ∈ {2, 4, 8}; cross-validator rejects loftq +
use_dora / use_vera. utils/loftq_init.py exposes validators +
build_loftq_config (lazy peft.LoftQConfig with actionable ImportError
hint). LLaMA Pro: TrainingConfig.expand_layers ∈ [1, 64] +
freeze_trainable_layers (signed, |x| ≤ 1000); cross-validator requires
the pair (LLaMA Pro freezes original layers and trains only new blocks).
field_validator(mode="before") on both rejects bool BEFORE Pydantic ge/le
silently coerces True → 1. expand_model_blocks raises NotImplementedError
with v0.41.1 marker — schema-only release (mirrors v0.27.0 / v0.37.0
stub-then-live pattern). utils/block_expansion._count_layers uses
hasattr(__len__) instead of try/except TypeError so legitimate __len__
bugs surface loudly. use_mod boolean for Mixture-of-Depths (schema only —
live patch deferred to v0.41.1).
Friendly aliases: load_in_8bit / load_in_16bit (Optional[bool]) for
LlamaFactory / Axolotl users. is True policy on both — explicit False is
"no preference", not "off"; mutually-exclusive both-True rejected; alias
combined with explicit Quant Menu format raises rather than silently
overriding. Alias-driven quantization rewrite via direct
self.quantization = ... (Pydantic v2 BaseModel non-frozen path), NOT
object.__setattr__ — code review caught that the latter would silently
bypass any future field_validator on quantization.
Five review agents (4 in parallel + manual smoke for verification-loop):
all CRITICAL/HIGH/MEDIUM/LOW findings fixed. Local smoke caught a real
bug — PyYAML parsing 1e-4 as string — fixed pre-commit with 2 added tests.
5242 tests pass (+120 net new); ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
5e0872b9ea |
feat(trainer): ReLoRA + surgical PEFT non-SFT (v0.40.6 #67)
Extends the v0.39.0 ReLoRA callback (Part B) and surgical PEFT patches (Part D — Gemma4 ClippableLinear swap + 3-D fused-MoE expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, RewardModel, Pretrain, Embedding, BCO). - New shared helper soup_cli/utils/peft_wiring.py exposes apply_pre_lora_patches, apply_post_lora_patches, attach_relora_callback. - SFT migrated to the same helpers in the same release (centralisation invariant; no drift between SFT and non-SFT wiring). - SoupConfig._validate_relora_supported_tasks: task != "sft" rejection removed; MLX backend still rejected with distinct message. Review fixes: - attach_relora_callback uses `if relora_steps is None:` (project policy) so a schema-bypassing relora_steps=0 surfaces as a loud ReLoRAPolicy ValueError rather than a silent skip. - Direct attribute access on tcfg.relora_warmup_ratio / _reset_optimizer / _prune_ratio (Pydantic schema guarantees them); no getattr defaults. - 11 behavioural helper tests (Gemma4 happy path + exception swallow, post-LoRA strip happy + exception swallow, ReLoRA policy field forwarding, schema-bypass loud-fail). - Schema-gate matrix covers `task='preference'` dispatcher. Tests: 5061 -> 5122 (+61). Closes #67. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
697cc8dad7 |
feat(trainer): Quant Menu non-SFT multi-trainer wiring (v0.40.5 #66)
Extends the v0.38.0 train-time quantization menu (gptq / awq / hqq:Nbit /
aqlm / eetq / mxfp4 / fp8) from SFT-only to all 11 transformer-backend
trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel /
Pretrain / Embedding / BCO). Closes the v0.38.0 known gap.
Schema gate: SoupConfig._validate_quant_menu_supported_tasks removes the
`task != "sft"` rejection branch. MLX backend rejection retained with
distinct message; modality=text gate retained (vision/audio Quant Menu
deferred — modality-specific kwargs not yet threaded through the unified
loader).
Trainer wiring (11 sites): each non-SFT _setup_transformers replaces the
inline BitsAndBytesConfig(load_in_4bit=True, ...) block with a call to
build_quantization_config_for_loader(tcfg=tcfg, base=cfg.base, console=console)
— mirrors sft.py:420-440 exactly. kbit-prep tuple widened from
("4bit", "8bit") to ("4bit", "8bit", "mxfp4"). BitsAndBytesConfig import
removed from each non-SFT trainer.
Review fix — PPO reward model: _load_reward_model gains optional tcfg
kwarg; when supplied, the reward checkpoint loads with the same Quant
Menu config as the policy. Both PPO call sites forward tcfg=tcfg.
Defends against silent fp16 OOM on a GPTQ/AWQ/HQQ policy run.
Review fix — defence-in-depth: new TrainingConfig.reward_model field
validator rejects null bytes and caps length at 512 chars (matches
cfg.base policy). The Quant Menu loader's per-call null-byte check
in _check_local_marker remains as the runtime backstop.
Tests: +131 net new (4930 -> 5061). New tests/test_v0405_part_a.py
parametrizes 11 tasks x 7 quant formats; covers MLX rejection per task,
quantization_aware x Quant Menu cross-validator regression for non-SFT,
source-level invariants (no inline BNB literal, kbit tuple regex),
and a live mock-based dispatch test for _load_reward_model proving the
Quant Menu path is reachable when tcfg is supplied and skipped when
tcfg=None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
560d98df8c |
docs: bump v0.40.4 + release notes (CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md)
- pyproject.toml + soup_cli/__init__.py → 0.40.4 - README.md: replace What's New block with v0.40.4 highlights; ## Multipack section updated to "live wiring landed"; expanded ## --trust-remote-code section to list full surface coverage (every command + every trainer task) - SECURITY.md: v0.40.4 added to supported versions; full per-version fix note appended (multi-trainer opt-in pattern, multipack DataLoader override, _get_train_sampler defensive delegate fix, drop_last forwarding, known limitations) - CONTRIBUTING.md: test count 146 files / 4855 tests → 148 files / 4930 tests Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
6fdf7e2570 |
feat(trainer): multipack live HF Trainer DataLoader override (v0.40.4 Part B)
Closes #65 (deferred from v0.40.3). make_multipack_trainer_class adds a get_train_dataloader override that builds a MultipackBatchSampler(real_batches=False) — yields a flat list[int] per packed sequence, which is the contract HF DataLoader.batch_sampler expects — and installs it via DataLoader(..., batch_sampler=sampler, collate_fn=self.data_collator, num_workers=args.dataloader_num_workers, pin_memory=args.dataloader_pin_memory). drop_last is forwarded from TrainingArguments.dataloader_drop_last. Falls back to super().get_train_dataloader() when state was never attached OR when train_dataset is unset — defence-in-depth so the subclass remains safe to instantiate even when multipack is later disabled. The state-presence guard switched from falsy (`not max_seq`) to explicit `is None` (plus `not lengths` for empty-list defence) — attach_multipack_state already rejects non-positive ints, so the falsy guard would only mask configurator bugs. _get_train_sampler override stays as a defensive no-op fallback that ALWAYS delegates to super (review-fix from v0.40.4 code-review: returning a multipack list[list[int]] from this method would cause a shape mismatch if any HF eval / prediction loop bypasses get_train_dataloader and calls _get_train_sampler directly). SFT and Pretrain trainer wrappers now invoke make_multipack_trainer_class(SFTTrainer) and attach_multipack_state(...) when multipack: true. The v0.40.3 yellow advisory + standard-sampler fallback is gone. Architecture allowlist (validate_multipack_architecture) still gates at build time. tests/test_v0403_part_b.py: TestSftAndPretrainWiringDeferred renamed to TestSftAndPretrainWiringLive; the deferred-state test (_get_train_sampler returns MultipackBatchSampler when state is set) is replaced by the live-state test (_get_train_sampler always delegates to super even with state attached). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
3ab36e2aad |
feat(security): trust_remote_code opt-in across non-SFT trainers + 5 commands (v0.40.4 Part A)
Closes the v0.36.0 #63 known gap. Every non-SFT trainer wrapper (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO + the unified Preference dispatcher) now accepts trust_remote_code: bool = False on __init__, resolves once via the v0.36.0 helper (model_requires_trust_remote_code + resolve_trust_remote_code), and stores the resolved value on self._trust_remote_code. Every from_pretrained call site reads from the resolved attribute — no remaining trust_remote_code=True literal in any trainer file (asserted by tests/test_v0404_part_a.py). Five standalone commands gain a --trust-remote-code Typer flag with the same default-deny + KNOWN_SAFE_PREFIXES allowlist behaviour as soup train: soup diff, soup export, soup merge, soup infer, soup data generate. commands/train.py removes the v0.36.0 sft_kwargs split — every trainer receives trust_remote_code from the same trainer_kwargs dict. PreferenceTrainerWrapper forwards the raw bool to the inner DPO / SimPO / ORPO / IPO / BCO wrapper kwargs at both _build_inner and _build_multi_objective sites; the resolver fires inside the inner wrapper at construction time. _load_reward_model (module-level helper in ppo.py) accepts a trust_remote_code: bool parameter and resolves internally — design intent is that the helper is independently safe to call outside PPOTrainerWrapper. _export_onnx / _export_tensorrt / _export_awq / _export_gptq and _merge_adapter helpers all gain a trust_remote_code: bool = False parameter threaded from the Typer flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1f050fc235 |
feat(v0.40.3): Stub-to-live wave 1 (#33, #64; #65 still deferred)
Three v0.X.0 deferred-stub features become live runtime — closes #33 (harvester judge filter + serve trace log) and #64 (live CUDA OOM probe). #65 (multipack live wiring in HF Trainer) remains deferred to v0.40.4 after the adversarial 5th-review pass surfaced a Sampler[int] vs list[list[int]] shape mismatch with HF Trainer's DataLoader; helpers (`make_multipack_trainer_class`, `attach_multipack_state`, `lengths_from_dataset`, `detect_arch_name`) ship as a stub used by unit tests, but the SFT/Pretrain wrappers print a yellow advisory and fall back to the standard sampler when `multipack: true`. Live CUDA OOM probe (#64): `make_cuda_probe_fn` builds a closure that runs ONE forward+backward+step on a synthetic batch per candidate. `model.zero_grad(set_to_none=True)` runs BEFORE forward; intermediate ids/attn/labels/outputs are del-ed before `loss.backward()` so peak VRAM reflects a realistic training step (matches v0.35.0 #45 policy). `pad_id` is bounded by `len(tokenizer)` (not `vocab_size`) so extended vocabs (Llama-3 + `<|pad|>`) don't fold pad to a random byte token. SFT-only this release. Trace-to-Preference judge filter (#33 (a)): `judge_filter_pairs` reuses v0.19.0 JudgeEvaluator backends (openai/server/ollama). Threshold rejects bool/NaN/out-of-[0,1]; `_MAX_BATCH=100_000` cap applied via lazy `itertools.islice`; per-pair backend exceptions caught and DEBUG- logged (matches v0.33.0 #47 policy); `judge_provider` validated against the allowlist at the CLI boundary BEFORE constructor with a Rich-escape error message; yellow projected-call-count warning before the loop (2× per pair). Inference Server trace log (#33 (b)): `TraceLogWriter` is thread-safe (single-process lock — multi-worker documented as known limitation); path containment via shared `is_under_cwd`; null-byte/empty/non-string path rejected; cap_mb bounds [1, 10000] with explicit bool rejection. Rotation (one backup retained) refuses symlink at the backup path via `os.lstat + stat.S_ISLNK` (matches v0.33.0 #22 TOCTOU policy). Secret redaction (`hf_*` ≥8, `sk-*` ≥16, `Bearer …` ≥8 with `.` excluded so end-of-sentence period survives) applied to prompt + response and recursively to caller-supplied `extra` dict values. Streaming SSE path also records (was a coverage gap caught in adversarial review). Behaviour change: v0.40.2 users with `auto_batch_size_strategy: probe` were silently getting the static fallback. v0.40.3 actually runs a CUDA probe on first run (~5–30s, cached per (model, max_length, quant, lora_r, gpu) tuple). Reviews: 5 agents (python, code, security, tdd, verification-loop). Verification-loop run twice — once shallow smoke (PASS), once adversarial bug-hunt which found C1/C2 (multipack live wiring crash — demoted to v0.40.4), H1 (streaming SSE missing trace log — fixed), H4 (vocab_size vs len(tokenizer) on extended vocabs — fixed), H3 (Bearer regex consumed trailing period — fixed), H2 (judge cost shock — warning added), M2 (empty lengths accepted — rejected), L1 (extra dict bypassed redaction — recursive walk added). Tests: 4756 → 4855 (+99 net new) across test_v0403_part_a/b/c.py. Lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b0fc586706 |
feat(v0.40.2): Quick polish + v0.40.1 carry-overs (#36, #50, #51 + 7 papercuts)
Closes 3 originally-scheduled GitHub issues plus 7 v0.40.1 long-tail UX papercuts. No new schema fields, no new trainers — pure polish. Originally scheduled: - #36 format_gate_row helper for the eval-gate dashboard row (pure formatter in soup_cli/monitoring/display.py; passed=is True so missing field renders neutral; supports stop/warn action suffixes; multi-task " | " join). - #50 prepare_hf_resume now skips snapshot_download when local checkpoint-N is greater-or-equal to the remote highest-N. New _find_highest_local_checkpoint helper handles missing dirs / OSError / non-directories cleanly. - #51 soup deploy hf-space --template-dir <path> via new soup_cli/utils/hf_space.py:render_custom_template_dir. Containment via is_under_cwd; validate_repo_id BEFORE substitution; per-file 256 KB cap; symlinks + non-regular files rejected (TOCTOU defence per v0.33.0 #22). v0.40.1 carry-overs: - H2: data filter --min-coherence alias; data split --train no-op; data register/unregister positional <name> <path> + Optional --name/--path with conflict detection. - H3: soup quickstart --output DIR (containment-checked) routes data, config, run dir under the chosen directory. - N1/G2: apply_logging_level pushes parsed --log-level tier into the root logger so transformers / peft / trl actually respect QUIET / DEBUG. - N7: shared _resolve_model_source in commands/infer.py (used by bench.py too) — path-like-but-missing raises FileNotFoundError; non-path-like values fall through to HF download via from_pretrained. - G13: verified ONNX/AWQ/GPTQ/TensorRT install hints already correct. - M4: verified data dedup --threshold already exposed. - M5: soup runs --cwd-only + _filter_runs_by_cwd helper using os.path.realpath + commonpath (Windows 8.3 + cross-drive safe). Review-fix follow-ups landed in the same release: - soup_cli/commands/infer.py: from __future__ import annotations (Py3.9 PEP 604 fix); --output containment via is_under_cwd, late-evaluated to preserve pre-existing test contracts. - soup_cli/commands/data.py register_data + soup_cli/commands/bench.py prompts file: Path.resolve()+relative_to() → is_under_cwd (project rule for Windows 8.3 short-name safety). - soup_cli/commands/runs.py: typed _filter_runs_by_cwd, removed redundant inner import os. - soup_cli/commands/deploy.py: confirmation panel now shows --template-dir path when set, not the unused --template default. Tests: 4720 → 4756 (+36) across two new files (test_v0402_part_a.py, test_v0402_part_b.py). 5 review agents (python / code / security / tdd / verification) all clean after fixes. Known limitations: - Custom HF Space templates always create the Space with sdk=gradio regardless of the supplied app.py. Use --template streamlit-chat with the inline registry for Streamlit. Tracked for v0.40.3+. - _resolve_model_source returns ("hf", repo_id) without validate_repo_id; transformers.from_pretrained will raise loudly on malformed ids. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2ae05f90f9 |
fix(v0.40.1): add `from __future__ import annotations` to quickstart.py for py3.9
Hotfix: `_pick_quickstart_model() -> tuple[str, str | None]` uses PEP 604 union syntax which Python 3.9 rejects at module-import time. Closes the CI collection failure on the ubuntu-latest/3.9 matrix without changing behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
56bea56c08 |
fix(v0.40.1): QA Hardening — UTF-8 bootstrap, schema strictness, multi-objective preference runtime, CLI UX
Closes the QA findings from the Windows + RTX 3050 4 GB pass (2026-05-07): - Part A: UTF-8 stdio bootstrap on Windows (closes C1/C4/H1/N5/N8/G5) - Part B: root-level `lora:` migrates into training.lora (no more silent init_strategy bypass); multi-objective preference loss runtime no longer raises NotImplementedError (primary-loss approximation; full per-batch weighted combination deferred to v0.40.2) - Part C: autopilot 7B → 1B fallback + safetensors cache probe; transformers <5.0.0 cap with INCOMPATIBLE flag in `soup doctor`; quickstart auto-switches to SmolLM2-135M on ≤6 GB VRAM; --find-lr load_local → load_raw_data import fix - Part D (subset): dynamic --template help (H4); init --force (M2); migrate JSONL friendly error (N2); eval custom -o independent of attach-to-registry + loop-shadow bug fix (G10); history suggests dataset registry (N6); doctor importlib.metadata fallback (M1) + GPU diagnostic distinguishes CPU build (N3) + dual-Python detector (N4) - Part E: recipe fuzzy-match suggestions (M3); sample filename embeds strategy (no overwrite); JSONL BOM auto-strip Net +64 tests (4656 → 4720). 4 review agents clean (python/code/security/tdd). Long-tail UX papercuts (H2/H3/N7/M4/M5 + #36/#50/#51) deferred to v0.40.2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
10a13bd8d9 |
fix(quickstart): invoke train via subprocess to avoid Typer OptionInfo leak
Calling train_cmd() directly bypassed Typer's argument resolution, so typer.Option(...) defaults arrived as OptionInfo objects instead of resolved values, crashing later with 'OptionInfo > int' errors. Use subprocess.run() with the real CLI entry point instead — same invocation a user would run by hand, and Typer fully resolves all defaults. |
|
|
Alpamys
|
b8506c465a |
feat(preference): v0.40.0 — Preference Variety (4 Parts: BCO + dispatcher + DPO variants + multi-objective)
Part A — BCO Trainer (Binary Classifier Optimization): new task='bco', training.bco_beta, bco.yaml template, train+sweep routing. Internal _split_dpo_rows_to_bco adapts paired DPO input to TRL's BCO unpaired schema; skipped rows logged at DEBUG (mirrors v0.33.0 #47 policy). Part B — Unified preference dispatcher: additive task='preference' + training.preference_loss Literal {dpo,simpo,orpo,ipo,bco}. Legacy task='dpo' / 'simpo' / 'orpo' / 'ipo' / 'bco' remain first-class — the new surface is purely additive, not a breaking collapse. _make_inner_cfg uses model_copy so re-validation never sees an intermediate inconsistent state and the caller's cfg is never mutated. Part C — KL-controlled DPO variants: dpo_beta_schedule (linear / cosine / exponential) + dpo_beta_end + dpo_ref_regen_epochs [1, 1000]. BetaScheduleCallback resolves total_steps lazily in on_train_begin (closes a first-cut bug where total_steps=0 silently emitted beta_end for every step). RefModelRegenCallback uses load_state_dict(strict=True) with WARNING-on-mismatch (closes a first-cut silent partial-copy hazard). Gated to DPO-family tasks only; rejected on mlx backend with distinct error message. Part D — Multi-objective preference_loss_weights (2-5 entries, key allowlist + null-byte rejection, sum-to-1 ±1e-6). Schema-level surface only; live runtime weighted-loss combination deferred to v0.40.1 with NotImplementedError stub-then-live (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA pattern). Net +118 tests (4538 → 4656). All four review-agent waves clean (Python / Code / Security / TDD). Known limitation: BCOTrainerWrapper still hardcodes trust_remote_code=True (carry-over of the v0.36.0 #63 family across non-SFT trainers). Also: add docs/ to .gitignore (internal-only docs going forward; existing docs/QUANTIZATION.md from v0.38.0 stays tracked). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e6a9c087c3 |
feat(lora): v0.39.0 — LoRA Quality (PiSSA + ReLoRA + per-pattern rank + surgical patches + templates registry)
Five PEFT-surface improvements that LlamaFactory and Axolotl maintain: - LoraConfig.init_strategy Literal["random","pissa","olora"]; PiSSA SVD init via PEFT init_lora_weights="pissa". Back-compat: use_olora=True aligns to init_strategy="olora" via dict-copy model_validator(mode="before"); explicit conflict (use_olora=True + init_strategy="pissa"/"random") rejected. Mutual-exclusion vs DoRA / VeRA. - ReLoRA callback (utils/relora.py): frozen ReLoRAPolicy with bounds-checked steps [1, 1e7] / warmup_ratio [0,1] / prune_ratio (0,1) (strict — prevents zero-everything footgun); magnitude_prune_tensor (in-place torch.kthvalue, rejects non-Tensor / single-element short-circuit); duck-typed ReLoRACallback (no transformers import at module load). TrainingConfig fields relora_steps / relora_warmup_ratio / relora_reset_optimizer / relora_prune_ratio. SoupConfig _validate_relora_supported_tasks gates to task=sft + transformers backend with distinct MLX-backend error message; multi-trainer expansion deferred to v0.39.1 (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu stub-then-live pattern). - LoraConfig.rank_pattern / alpha_pattern Optional[Dict[str,int]]; field validator caps at 256 keys × value (0, 1024], rejects bool / null-byte / empty key. Cross-validator rejects with use_vera=True (VeRA shares one rank). peft_builder propagates into LoraConfig init_kwargs. - utils/peft_patches.py: is_gemma4_model uses regex word boundary (?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$) so "ungemma4ed" no longer matches. apply_gemma4_clippable_patch swaps ClippableLinear → nn.Linear by class name (weight-copy fallback logs at DEBUG). strip_lora_dropout_for_3d_experts zeroes lora_dropout.p on 3-D weights (handles ModuleDict variant for PEFT >=0.10). apply_surgical_patches orchestrator validates model_name. Wired into sft.py _setup_transformers with is_gemma4_model gate before the pre-LoRA swap; post-LoRA 3-D dropout strip runs unconditionally (architecture-detected internally). - 16 inline templates migrated to soup_cli/templates/*.yaml + manifest.json + load_template loader (path-traversal-rejecting name validator; os.path.realpath + commonpath containment so a tampered manifest cannot read files outside the package directory; 256 KB file-size cap with inline fallback). Inline TEMPLATES kept with deprecation comment (planned removal v0.41.0+); test_templates_yaml asserts byte-equality of all 16 inline ↔ YAML pairs to prevent silent drift. Net +164 tests (4374 → 4538). All 5 review-agent waves clean before tag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f6f29ef609 |
feat(quant): v0.38.0 — Quant Menu (8 Parts: A-H)
Train-time support for 7 new quantization formats — close the width gap
with LlamaFactory. Wired into SFT trainer + transformers backend + text
modality; multi-trainer/modality expansion deferred to v0.38.1 (mirrors
v0.27.0 MII / v0.37.0 multipack stub-then-live pattern).
- Part A — GPTQ: quantization='gptq' + gptq_disable_exllama (PEFT triton)
- Part B — AWQ: quantization='awq' + GEMM/GEMV builder
- Part C — HQQ: hqq:1bit..hqq:8bit (no 7bit; not supported upstream)
- Part D — AQLM: locked-2-bit
- Part E — EETQ: locked-8-bit
- Part F — MXFP4 + FP8 dequantize-on-load
- Part G — bnb_4bit_quant_storage for FSDP+QLoRA
("crucial for fsdp+qlora" — LlamaFactory quantization.py:178)
- Part H — check_quant_distributed_compat matrix + docs/QUANTIZATION.md.
HQQ/EETQ/AQLM x {FSDP, ZeRO-3} hard-fail; BNB-4bit + FSDP without
quant_storage warns. Wired into commands/train.py startup.
Three new schema validators:
- _validate_prequantized_no_qat — pre-quantized + QAT incompatible
- _validate_bnb_quant_storage_only_with_4bit — silent no-op guard
- _validate_quant_menu_supported_tasks — sft + transformers + text gate
Net: +61 tests (4374 -> 4435). Four review-agent waves clean before tag
(python-review / code-review / security-review / tdd-guide);
verification-loop performed as manual equivalent per CLAUDE.md allowance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
06fbd15ec4 |
feat(multipack): v0.37.0 — Multipack (5 Parts A/B/C/D/E)
FFD bin-packing sampler closes the throughput gap with Axolotl on uneven-length chat data. Five focused parts: - Part A: MultipackBatchSampler (FFD + 18-arch allowlist + loud-fail vs Axolotl silent miss + _MAX_FFD_ITEMS=1M DoS cap) - Part B: schema gate (sft/pretrain only on transformers backend, multipack/packing mutually exclusive, distinct mlx error), build_multipack_sampler_for_lengths helper - Part C: neat_packing 4D attention mask + FA-vs-4D strategy picker, _MAX_MASK_ELEMENTS=2**31 / _MAX_BOUNDARY_SEGMENTS=1M caps - Part D: JinjaTemplateAnalyzer (parse-only AST walker, 128KB cap) - Part E: cross-module property tests (4-seed x 200 samples, 5k stress, FFD-to-4D-mask coherence) All five review-agent waves clean before tag (python / code / security / tdd / verification-loop). Net +125 tests (4249 -> 4374), 121 test files (+5). Live HF Trainer sampler-swap wiring deferred to v0.37.1 (mirrors v0.27.0 MII stub-then-live pattern). Schema gate + helper ship now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
a5540fa1e2 |
feat(correctness): v0.36.0 — Correctness First (4 Parts: A/B/C/D)
Four silent-failure modes Soup had → loud failures, plus a
security default-deny.
- Part A: assistant-only loss masking (default true). Replaces TRL's
multi-turn heuristic with explicit IGNORE_INDEX masking. New
data.train_on_responses_only / train_on_messages_with_train_field
+ per-message train: bool field. Preferred path uses
return_assistant_tokens_mask; fallback uses incremental tokenize
delta with add_special_tokens=False to avoid double-BOS drift.
- Part B: --trust-remote-code opt-in default-deny on soup train /
chat / serve / data download / eval auto. KNOWN_SAFE_PREFIXES
allowlist (15 first-party orgs) suppresses warning panel.
Replaces 9 unconditional trust_remote_code=True call sites in
the SFT path. Non-SFT trainers + diff/export/merge/infer/generate
still hardcode trust_remote_code=True — documented v0.36.x patch.
- Part C: chat-template hardening. Tokenizers without chat_template
raise loudly instead of silent f"{role}: {content}" fallback.
New data.chat_template (registered name or raw Jinja). Filesystem
-touching Jinja directives (include/import/from/macro/extends)
blocked at config-load. Override application warns that soup push
will persist the new Jinja into tokenizer_config.json.
- Part D: OOM-probe auto batch-size. New
training.auto_batch_size_strategy: auto|static|probe. Try-halve
-then-double-to-ceiling loop, max 8 doublings, ceiling = static
× 4. ~/.soup/batch_cache.json (0600 perms, env-override
containment-checked against ~/cwd/tempdir). make_cache_key
rejects bool inputs.
Net +134 tests (4115 → 4249). All 5 review-agent waves clean
before commit; 5 HIGH / 10 MEDIUM / 5 LOW findings fixed in one
review-fix wave.
Smoke: python -m soup_cli.cli version → soup v0.36.0; all 5 new
--trust-remote-code flags surface in --help; ruff clean; pytest
4249 passed / 3 skipped / 0 failed in 2m41s on Windows py3.10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
f6e2004c9f |
fix(fp8): wire fp8_recipe through v028_features (covers 10 trainers)
PR #62 added fp8_recipe support but only wired sft.py directly. The other 10 trainers (dpo, pretrain, grpo, kto, orpo, simpo, ipo, ppo, reward_model, embedding) all route through apply_v028_speed_memory, which was calling apply_fp8_training(model) without recipe -- meaning user-set fp8_recipe='rowwise' was a silent no-op on every non-SFT task. - v028_features.apply_v028_speed_memory: read tcfg.fp8_recipe and pass through to apply_fp8_training; surface the picked recipe in the green status line so the run record reflects the actual dispatch - sft.py: drop the defensive getattr (fp8_recipe is a Pydantic field with a default, not optional) -- use tcfg.fp8_recipe directly - tests: add TestFP8RecipeViaV028Features (4 tests) verifying the recipe propagates through apply_v028_speed_memory for tensorwise / rowwise / rowwise_with_gw_hp, plus the int8-QAT path is unaffected |
|
Chinmaya Sahu
|
c13a4e0543
|
feat(fp8): add rowwise and rowwise_with_gw_hp scaling recipes for FP8 training (#62)
Add fp8_recipe config field to TrainingConfig with three torchao-backed scaling recipes: tensorwise (default, v0.28.0 behavior), rowwise (more accurate via CUTLASS), and rowwise_with_gw_hp (most accurate, grad_weight in high precision). Dispatches via Float8LinearConfig.from_recipe_name(). - schema.py: add fp8_recipe Literal field with validator requiring quantization_aware='fp8' for non-default recipes - fp8.py: update apply_fp8_training() to accept recipe parameter - sft.py: pass tcfg.fp8_recipe to apply_fp8_training() - README.md: document recipe options with comparison table - tests: 24 tests covering schema, dispatch, validation, backward compat |
|
|
Alpamys
|
892fd33f9e |
feat(trainers): v0.35.0 — Trainer Coverage (closes #60, #61, #45)
Wires v0.28.0 speed/memory features into every transformer-backend trainer (grpo / kto / orpo / simpo / ipo / ppo / reward_model / embedding) plus closes the v0.33.0 #43 oversight where dpo / pretrain accepted activation_offloading without installing offload hooks. Auto-quant --auto-quant now forwards the picked candidate's quantization to vLLM via an explicit named parameter (kwarg-splat hazard removed). Kernel auto-compose runs a forward-only benchmark loop on the trainer's actual model under torch.no_grad() so live training gradients aren't polluted (this was a critical-class bug caught by code-review pre-tag and fixed before merge). Schema gate lifted with distinct MLX-backend vs unknown-task error messages so users get the right fix. fp8 / int8 QAT guard fixed in 6 trainers (the legacy unguarded `if tcfg.quantization_aware:` would have crashed the int8 path with the string "fp8"). Net +187 tests (3928 -> 4115). New file tests/test_trainer_coverage_v035.py provides a parametrised matrix proof that every trainer x every feature is exercised on every CI matrix job. All four review-agent waves (python / code / security / tdd) clean with every CRITICAL -> LOW finding fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2cc4b5aa20 |
feat(observability): v0.34.0 — Observability & Dev UX (7 Parts)
Adds soup why, soup tui, soup runs replay, soup train --profile, .crash bundle on training exception, per-run cost in SQLite, --log-level global flag. Net +110 tests (3818 → 3928); all five review-agent waves (python / code / security / tdd / smoke) clean. - Part A: --log-level quiet|normal|verbose|debug → Rich-formatted logger on the "soup" namespace; idempotent + tier-change replaces handler. - Part B: SQLite gains cost_usd / cost_gpu_label via lazy ALTER TABLE (race-tolerant against duplicate-column on concurrent first-boot); rendered in soup runs show / replay / TUI; bool num_gpus rejected; LIKE wildcards escaped in tracker.get_run prefix match. - Part C: soup why — heuristic NaN / plateau / divergence / grad-norm / LR bounds; severity-ordered findings. - Part D: .crash bundle generator with recursive hf_*/sk-*/Bearer redaction, output_dir basename-only, os.path.realpath containment, secrets.token_hex filename, ValueError (not PermissionError) on outside-cwd; train.py except-handler writes the bundle without masking the original exception. - Part E: soup runs replay <id> — summary panel + downsampled loss curve (≤2000 points) from SQLite history. - Part F: soup train --profile — torch.profiler Chrome trace to <output>/profiles/<run_id>.trace.json; run_id rejects '.', '..', '/', '\\', null bytes; profiles dir created only on torch import. - Part G: soup tui — Textual dashboard with lazy ExperimentTracker import; markup_escape on every DB-sourced string; new [tui] extra. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e248b5aea8 |
fix(v0.33.0): missed checklist items (CONTRIBUTING + Unicode arrow)
Pre-push audit found 3 gaps in the Release Checklist (steps 6, 11):
1. soup_cli/commands/can.py:118 used → (->) in the run_cmd
docstring. Windows cp1252 in non-PYTHONUTF8 mode raises
UnicodeEncodeError when Typer renders --help. Replaced with
ASCII '->'. Verified: `python -m soup_cli.cli can run --help`
renders cleanly on Windows.
2. CONTRIBUTING.md test counts not updated:
- tree comment: 97 files, 3696 tests -> 104 files, 3818 tests
- test table: +7 rows for test_part_{f,a_wave1,a_wave2,e,d,c,b}
- directory tree: added registry/attach.py, cans/run.py,
cans/publish.py, data/collators.py, utils/v028_features.py
3. README.md:967 stale "v0.32.0 stub" note. Updated to reflect that
--find-lr now runs the live loop in v0.33.0 and that spike-recovery
writes a JSON hint while live optimizer rewind / DataLoader rebuild
remain follow-ups.
examples/README.md: skipped per checklist step 12 — v0.33.0 added no
new YAML configs (the new commands operate on .can files which use
existing config schemas).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
c4a27c1f9e |
docs(v0.33.0): version bump + What's New + security entries
- pyproject.toml + soup_cli/__init__.py: 0.32.0 → 0.33.0 - README.md: replace What's New block with v0.33.0 highlights (10 bullets covering all 17 closed issues), add `soup can run` / `soup can publish` / multi-GPU one-command body sections, update Soup Cans security note to reflect format-version 1+2 + new consent gate + token resolution. - SECURITY.md: shift supported-version window (v0.33.x full, v0.32.x bug-fix only); add v0.33.0 "Live Wire" entry covering every CRITICAL+HIGH+MEDIUM security fix from this release wave. CLAUDE.md / plan.md updates land in the next commit (those are gitignored / project-internal so kept separate). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ff55e751ab |
fix(v0.33.0): review-wave findings (CRITICAL + HIGH + MEDIUM + LOW)
Addresses findings from 5-agent review wave (python-reviewer, code-reviewer, security-reviewer, tdd-guide, smoke-verification). CRITICAL: - cans/run.py _deploy_target ollama path: rglob *.gguf result is now realpath+commonpath checked against extract_dir before forwarding to `soup deploy ollama --gguf`. Prevents a crafted symlink in the can from making rglob point at an arbitrary on-disk path. HIGH: - cans/publish.py: removed dead update_repo_settings + bare-except tag block (was a no-op network round-trip). Tag attachment via README front-matter is documented as a v0.33.x docs follow-up. - registry/attach.py lookup_entry_by_output_dir: emits ResourceWarning when the 1000-row scan limit is hit (was a silent miss). - data/collators.py CrossDocCollator: stops mutating input dicts via pop() — uses get + dict comprehension. HF Dataset rows are cached and reused; mutation broke subsequent batches silently. Bare-except now logs at DEBUG level so production degradation is inspectable. - monitoring/callback.py _write_spike_recovery_hint: added is_under_cwd guard. args.output_dir came from raw HF TrainingArguments without separate path-containment check. - trainer/rewards.py MACOS_SANDBOX_PROFILE: narrowed (allow mach-lookup) to a 3-name allowlist (SecurityServer, notification_center, opendirectoryd.libinfo). Broad mach-lookup permitted DNS / NSURLSession via launchd, defeating (deny network*). - cans/run.py: PermissionError → ValueError so a caller wrapping in `except OSError` cannot silently swallow the consent gate. PermissionError is an OSError subclass. - commands/can.py run_cmd: assigns result=None up front + explicit None guard so a future _fail bypass cannot trigger NameError on result. - utils/v028_features.py: added type annotations on apply_v028_speed_memory (model: Any, tcfg: TrainingConfig via TYPE_CHECKING, console: Console) and warn_unsupported_features. - cans/run.py: confirm_callback now annotated Callable[[Manifest], bool] for IDE introspection. - tests/test_part_b.py reexec test: drops env-var contamination (RANK/WORLD_SIZE/LOCAL_RANK/ACCELERATE_*) before run, patches imported names on train module, and forces assertion that os.execvp was called — no more silent skip-on-bypass. - tests/test_part_d.py: added TestGenerateResponseSignature source-level guard that catches the lenient logits_processor mock silently passing. MEDIUM: - cans/run.py _run_subprocess: catches subprocess.TimeoutExpired and returns rc=124 (coreutils convention) so callers see a clean CanRunResult instead of an unhandled traceback after the 24h cap. - cans/run.py: temp dir created via mkdtemp is now cleaned up on extract_can failure (try/except + cleanup_extract_dir). - cans/run.py cleanup_extract_dir: switched startswith path check to os.path.commonpath (project-standard idiom; Windows-safe). - cans/schema.py DeployTarget._safe_relpath: normalises mixed separators before splitting on '/' so foo/..\bar can no longer bypass the .. check. - utils/lr_finder.py run_lr_sweep: removed redundant local `import math as _math` (math already at module level). LOW: - eval/gate.py _parse_judge_url: removed bare http:// catchall after scheme allowlist. Defence-in-depth for callers that bypass the Pydantic GateTask validator. - utils/auto_quant.py evaluate_candidate: latency mean now divides by *completed* prompts (excludes crashed). Crashed candidate no longer appears artificially fast. - utils/auto_quant.py Candidate.__post_init__: explicitly rejects bool in score / latency_ms (bool is a subclass of int, was sneaking past). - utils/mii.py: removed `noqa: F401` on Optional import (now actually used in type annotation since we restored it). Tests added (+7, total 3811→3818): - test_part_a_wave1: attach_artifact outside-cwd rejection. - test_part_a_wave2: PermissionError→ValueError migration in 2 tests. - test_part_c: CrossDocCollator mismatched doc_lengths fallback, does-not-mutate-input-dict regression guard. - test_part_d: source-level _generate_response signature guard. - test_part_e: should_recover at max_attempts, outside-cwd skip. Lint: clean. Full suite: 3818 passed in 156s. Findings deliberately not actioned (with rationale): - code-review M1 (mii Pydantic at import-time): forward-ref resolution requires module-level definitions for FastAPI; documented in mii.py. - code-review M4 (supports_v028_features vs validator divergence): the v0.33.0 schema validator was renamed to _validate_v028_speed_memory_supported_tasks and now imports supports_v028_features — they cannot drift. - python-review LOW (_deploy_target vllm silent no-op): documented in the docstring as advisory; logging requires a console arg the helper does not currently take. - security-review LOW 8/9 (TOCTOU window, CLONE_NEWPID): theoretical; documented in CLAUDE.md security section in the next commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
66bf0d9242 |
feat(multi-gpu,serve): auto-reexec + MII live (v0.33.0 Part B)
Closes #37, #38. Final Part of v0.33.0 implementation phase. #37 Auto-reexec under accelerate launch when --gpus N>1: - soup_cli/commands/train.py gains --no-reexec opt-out flag (default behaviour: auto-reexec). - When --gpus N>1 and not already in a distributed env (RANK/WORLD_SIZE + ACCELERATE_* markers absent), train() reconstructs argv via utils.launcher.build_accelerate_argv and calls os.execvp("accelerate", argv). os.execvp replaces the current process — no leftover PID tree, stdio passes through unchanged. - Critical flags (--fsdp, --deepspeed, --resume, --wandb, --tensorboard, --yes) are forwarded to the reexec'd run so users see the same behaviour they'd get from running accelerate launch by hand. - OSError from execvp falls back to the v0.27.0 advisory (printed command) so misconfigured PATH doesn't dead-lock the user. - --no-reexec preserves the v0.27.0 print-and-exit behaviour for users who want to control env vars / stdio explicitly. #38 DeepSpeed-MII live serve: - soup_cli/utils/mii.py gains build_mii_app(pipeline, model_name) which returns a FastAPI app with /v1/chat/completions + /v1/models matching the v0.30.0 transformers backend's contract. - Pipeline is held by closure (single MII instance, thread-safe across concurrent generations). Loopback-only CORS mirrors v0.30.0 transformers backend policy. - max_tokens bounds [1, 16384], stream=True rejected (MII v0.x lacks stable streaming), pipeline crashes return 500 with generic message (no stack-trace leak). Empty response → 500. - soup_cli/commands/serve.py replaces the v0.27.0 stub-warning + Exit(1) with create_mii_pipeline → build_mii_app → uvicorn.run. Tests: +9 in tests/test_part_b.py covering /v1/models endpoint, chat happy-path with mocked pipeline returning .generated_text, streaming rejection, max_tokens bounds (low + high), pipeline failure → 500, empty pipeline response → 500, --no-reexec parameter exists, --no-reexec advisory fallback, --gpus 2 reexec calls os.execvp with accelerate argv (via monkeypatched os.execvp). Known limitations: - MII server has no streaming, no LoRA hot-swap, no /metrics dashboard, no OpenTelemetry — those are v0.30.0 transformers-backend features not yet ported. Documented in the build_mii_app docstring. - Auto-reexec assumes accelerate is on PATH; OSError path prints the command instead, matching the v0.27.0 baseline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
55d1b9312c |
feat(speed,memory): v0.28.0 features go multi-trainer (v0.33.0 Part C)
Closes #43, #44, #47. #43 Multi-trainer wiring (sft/dpo/pretrain): - New utils/v028_features.apply_v028_speed_memory(model, tcfg, base_model, console) — single shared helper for use_cut_ce, quantization_aware="fp8", kernel_auto_compose. Each feature degrades silently to a yellow advisory if the underlying lib is missing; never crashes training kick-off. - Helpers supports_v028_features(task) and warn_unsupported_features(tcfg, task) drive both the schema validator and runtime advisories. - soup_cli/trainer/dpo.py and trainer/pretrain.py now call the helper after model load (post-LoRA, post-QAT) — same hook point as SFT. - soup_cli/config/schema.py validator _validate_v028_speed_memory_sft_only renamed _validate_v028_speed_memory_supported_tasks; allowlist now {sft, dpo, pretrain}. GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Embedding still error out at config-load with a precise multi-trainer message. #44 Selective gradient-checkpoint hooks: - New utils/gradient_ckpt.install_selective_hooks(model, granularity) iterates ``model.named_modules()`` looking for transformer-block-shaped names (numeric suffix on layer path), wraps each module's ``forward`` with torch.utils.checkpoint.checkpoint based on tier: - selective: only attention sub-modules - medium: every second transformer block - full: every transformer block - Returns hook count so callers can fall back to HF native checkpointing when zero blocks were found. #47 CrossDocCollator: - New soup_cli/data/collators.CrossDocCollator wraps any base data collator and injects a block-diagonal causal ``cross_doc_attn_mask`` built from per-example ``doc_lengths``. Preferred over TRL's ``packing_strategy="attention_free"`` flag (best-effort across TRL versions). Degrades gracefully when doc_lengths is missing or shapes don't match — base attention_mask preserved, no crash. Tests: +16 in tests/test_part_c.py covering apply_v028_speed_memory (no-features, cut_ce graceful failure), supports/warn helpers extension, schema gate (dpo + pretrain accept, kto still rejects), selective hook installation across full/medium/selective with fake transformer-shaped models, CrossDocCollator passthrough + strip + injection. One existing test in test_training_speed.py updated: dpo+use_cut_ce now accepted. Known limitations: - 7 trainers (GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Embedding) still reject v0.28.0 flags at config-load. Each is a 5-line addition once schema validation is satisfied; tracked as a v0.33.x follow-up. - install_selective_hooks doesn't undo earlier hooks — caller must be re-init aware. Not an issue for the typical "construct wrapper, train, exit" flow but worth noting. - CrossDocCollator emits ``cross_doc_attn_mask`` (not ``attention_mask``) to avoid clobbering the base collator's contract; downstream consumers must read the new key explicitly. The plan calls for "preferred over TRL's packing_strategy" which we satisfy via opt-in collation, not silent override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f9e6d20962 |
feat(serve): structured-output + auto-quant live (v0.33.0 Part D)
Closes #49, #53, #54. #53 Wire --structured-output into transformers generation loop: - New utils/structured_output.build_logits_processors(constraint, tok) returns a HF LogitsProcessor list. Tries outlines first (broader coverage), falls back to lm-format-enforcer, returns [] if neither installed or factory crashes — server degrades to free-form rather than 500 on a missing dep. - _generate_response gains logits_processor kwarg, forwarded to model.generate(...). Chat-completions handler builds the processor list per request (cheap; per-request build keeps the descriptor mutable for future /v1/output_constraint endpoints) and passes it down. Empty list path is unchanged from v0.30.0 free-form behaviour. #54 --auto-quant live eval loop: - New utils/auto_quant.evaluate_candidate(name, eval_fn, prompts): times mean per-prompt latency, scores correctness, marks ok=False when any prompt crashes or score < min_correct_fraction. - New utils/auto_quant.run_auto_quant_picker(candidate_specs, prompts, min_score): evaluates each candidate, calls pick_best, soft-falls- back to highest-scored ok candidate if no candidate clears the threshold so the server still binds. - serve.py replaces the v0.30.0 deferral warning with a real picker run over a fixed 3-prompt set across default_candidate_order(). Logs the picked (name, score, latency) on stdout. #49 End-to-end --push-as integration test (mocked HF): - New tests/test_part_d.py::TestPushAsResumeIntegration uses a fake huggingface_hub module via patch.dict to verify HFPushCallback constructs cleanly with a token, exposes the _repo_failed sticky flag (v0.29.0 review fix), and that prepare_hf_resume rejects output_dir outside cwd. The full HF Hub network roundtrip needs a paid sandbox repo — keeping it mocked-only is a deliberate trade (prevents flaky CI on rate limits / token rotation). Tests: +15 in tests/test_part_d.py covering build_logits_processors graceful-degrade paths (None / off / unknown / no-libs / factory crash), generate_response logits_processor plumbing, evaluate_candidate (empty / all-correct / crash / below-threshold), run_auto_quant_picker (threshold pass + soft fallback), HF push smoke. One existing test in test_inference_advanced.py updated: TestAutoQuantCLIWarning no longer expects the v0.30.1 deferral message — it now expects --auto-quant to actually run. Known limitations: - #49: full HF Hub roundtrip is mocked-only; live integration test requires a paid sandbox repo and rotating token, deferred to a separate end-to-end CI job. - #54: live re-loading of the model at the picked quant is NOT done in this commit — the picker logs the choice but the already-loaded model is served. Live re-load needs an additional bnb / awq round- trip per candidate, which is heavy for a startup-time decision; follow-up tracked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e406688f2d |
feat(training): stability auto-tuning live (v0.33.0 Part E)
Closes #56, #57, #58, #59. #56 Live --find-lr in-process LR-sweep: - New utils/lr_finder.run_lr_sweep(model, dataloader, schedule, optimizer_factory, device): per-step LR mutation + forward + backward, records loss until exhaustion or NaN/Inf divergence. - commands/train.py wires it via _live_lr_sweep_from_config (loads model + tokenizer + first N rows of cfg.data.train), with synthetic-curve fallback so users without GPU/torch still get a parseable report. #57 Loss-spike recovery hint: - SoupTrainerCallback gains spike_recovery / spike_recovery_max_attempts / spike_recovery_lr_decay; on watchdog fire writes output_dir/spike_recovery.json with previous_lr, recommended_lr (per SpikeRecoveryStrategy.compute_new_lr), should_recover, attempts. A wrapper / re-launch can resume with the decayed LR. Live optimizer rewind is intentionally NOT done — HF Trainer has no safe public API for mid-loop optimizer-state mutation; the JSON hint is the contract. #58 auto_mixed_precision push to TrainingArguments: - New SFTTrainerWrapper._resolve_mixed_precision: when tcfg.auto_mixed_precision is True, queries torch.cuda compute capability and calls pick_mixed_precision(base, cc) to set bf16=/fp16= flags. CPU short-circuits to (False, False). When the flag is False, legacy default preserved (bf16=cuda). #59 Grad-accum advisory (Phase 1): - SoupTrainerCallback gains grad_accum_auto_tune / grad_accum_pressure_threshold / grad_accum_total_vram_gb / grad_accum_current_steps / grad_accum_current_batch. - on_log samples torch.cuda.max_memory_allocated each step; if GradAccumMonitor.should_adjust crosses the threshold once, prints (batch, accum) -> (new_batch, new_accum) advisory and short-circuits (one-shot). Phase 2 (live DataLoader rebuild) needs a small TRL upstream PR — tracked as a known limitation. Wiring: - soup_cli/trainer/sft.py: _resolve_mixed_precision helper, batch_size preserved on self, SoupTrainerCallback constructor passes through new spike + grad-accum knobs. - soup_cli/monitoring/callback.py: rich Console import added (was previously module-relative); spike + grad-accum state fields and one-shot helpers. Tests: +15 in tests/test_part_e.py covering the LR-sweep loop with mocked model + optimizer (records, divergence break), mixed-precision resolver across cpu/cuda + auto-flag combinations + qwen2 fp16 quirk on Ampere, spike recovery hint write + attempts increment + disabled no-op, grad-accum advisory one-shot semantics + threshold + cuda-absent + disabled. Known limitations (release notes): - #57 spike recovery is a JSON hint, not in-process optimizer rewind - #59 Phase 2 (live DataLoader rebuild on advisory) deferred Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
8f2bc56334 |
feat(cans): soup can run + publish (v0.33.0 Part A wave 2)
Closes #34. Completes Part A — all of #32, #34, #35 now shipped. Schema bump: CAN_FORMAT_VERSION 1 -> 2 (additive). - SUPPORTED_CAN_FORMAT_VERSIONS = (1, 2): old cans still inspect/extract. - New DeployTarget Pydantic model with kind in {ollama, gguf, vllm}, name validation (no null bytes / newlines), path validation (relative-only, rejects '..' and absolute paths). - Manifest gains optional deploy_targets: list[DeployTarget] field. cans/run.py — orchestrator: - run_can(can_path, yes, deploy, extract_dir, capture_env_to, ...) validates path containment, requires --yes or explicit confirm_callback (security: auto-downloads data + auto-trains), extracts, optionally captures env, then invokes `soup train --config ... --yes` via subprocess so the trainer dispatch stays a single source of truth. - capture_env: best-effort pip freeze + python version + GPU detection, never blocks training on env-capture failure. - _deploy_target dispatches per-kind; ollama path runs `soup deploy ollama --gguf ... --name ...` if a *.gguf is present in the can. - cleanup_extract_dir: tmp-or-cwd-only safety guard around shutil.rmtree. cans/publish.py — HF Hub publish: - publish_can(can_path, repo_id, token, private, commit_message) validates can-path containment, repo_id via utils/hf.validate_repo_id, resolves token via utils/hf.resolve_token (env > cache files), uploads to repo_type='dataset' with commit-message first-line + 200-char cap (matches v0.29.0 push.py / data push policy). Tags as can-format-v1. CLI: soup_cli/commands/can.py - New `soup can run <path> [--yes] [--deploy] [--extract-dir] [--env-capture]` — confirmation panel mandatory without --yes. - New `soup can publish <path> --hf-hub <user/repo> [--private] [--message]`. Tests: +28 in tests/test_part_a_wave2.py covering schema bump (v1/v2/v3), DeployTarget validation (path traversal, null bytes, kind enum), capture_env (smoke + pip-failure tolerance), run_can (containment + confirmation gate + train-argv shape via mocked subprocess), publish_can (repo_id validation, token resolution, commit-message sanitization, HF upload via mocked HfApi), and CLI smoke (confirmation panel, missing file). One existing test_cans test relaxed (v1 == v1 -> v in {1,2}). Known follow-ups (not blocking release): - soup can run does NOT yet auto-fetch data_ref.kind=hf|url. Embedded config must reference a local data path. Filed mentally as v0.33.x follow-up. - registry_snapshot.json lineage export deferred — pack already embeds base_hash which is enough to query the source registry post-extract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ca799f6fd3 |
feat(eval,registry): live gate scoring + registry attach (v0.33.0 Part A wave 1)
Closes #32, #35. (#34 soup can run/publish deferred to Part A wave 2.) #32 Live model scoring for `soup eval gate` + `soup eval quant-check`: - gate.run_gate now dispatches judge / benchmark / custom task types, wrapping each scorer in try/except so a backend failure produces score=None + error=str(exc) instead of a silent score=1.0 pass. - New _parse_judge_url splits ollama:// / http(s):// judge_model URLs into (provider, model, api_base) for JudgeEvaluator. - New _run_judge_task / _run_benchmark_task plug into existing eval/judge.py and eval/forgetting.py runners. - New quant_check.make_model_generator(model_path) wraps transformers AutoTokenizer + AutoModelForCausalLM into a generate_fn callable; greedy by default for reproducible scores; lazy-imported. - gate_cmd / quant_check_cmd build live generators when --model is given; fall back to deterministic stub on load failure so CI without GPUs still runs the orchestration layer. - GateTaskResult.score is now Optional[float] with new error: Optional[str]. - _print_gate_result renders ERROR + reason cleanly. #35 Registry attach hooks: - registry/store.py _VALID_KINDS extended with eval_results, tensorrt. - New registry/attach.py: attach_artifact, write_eval_json (cwd-containment via realpath+commonpath), lookup_entry_by_output_dir. - `soup eval custom` gains --attach-to-registry + --output (paired); on success writes JSON results and adds eval_results artifact row. - `soup export` gains --registry-id with auto-match by source --model output dir; auto-attaches the produced GGUF artifact. Failures here are warnings, not hard exits — export already succeeded. Tests: +19 in tests/test_part_a_wave1.py covering URL parser, error propagation across all 3 task types, score=None semantics, generator factory bounds + transformers mocking, registry attach helpers (containment + missing entry), and CLI integration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
7f32a5e7c0 |
feat(security): standalone hardening (v0.33.0 Part F)
Closes #21, #22. #21 RLVR code_exec_reward: add OS-level isolation strategy detection. - New _get_isolation_strategy / _compute_isolation_strategy with cache. - Linux: best-effort os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID) in preexec_fn (Python 3.12+). Silent fallback on EPERM/ENOSYS for hosts where unprivileged user namespaces are disabled. - macOS: prefix subprocess argv with sandbox-exec + inline default-deny profile (deny network*, deny writes outside /tmp). - Windows + restricted Linux: existing RLIMIT + socket-patch + ephemeral cwd guards continue to apply (best-effort baseline). #22 prune_checkpoints: TOCTOU-safe symlink handling. - Top-level entries: explicit os.lstat + stat.S_ISLNK check (intent-clear) instead of Path.is_symlink. - shutil.rmtree now passes onerror=_abort_on_symlink to abort recursive walk if any symlink is encountered mid-walk (defence-in-depth). - OSError mid-prune is caught per-checkpoint so one bad dir does not abort the whole prune pass. Tests: +13 in tests/test_part_f_hardening.py covering strategy detection on linux/darwin/win32, sandbox profile shape, code_exec smoke tests, and TOCTOU-resistant prune behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
8878eb1aa6 |
feat(training): v0.32.0 — Training Stability & Auto-Tuning
Seven new opt-in flags that turn Soup into the "fast.ai of LLM fine-tuning" — pre-flight LR range finder, auto warmup schedule, auto mixed-precision, loss-spike auto-recovery, convergence detector, VRAM-pressure advisory, and autopilot integration. * Part A: soup train --find-lr / utils.lr_finder (TypedDict result, abs-divergence threshold, NaN/Infinity rejection, MAX_NUM_STEPS=10000, is_under_cwd containment on --find-lr-output). * Part B: utils.grad_accum (GradAccumMonitor + MAX_ACCUM=1024 cap; preserves effective batch on recommend()). * Part C: utils.mixed_precision (KNOWN_PRECISION_QUIRKS map, longest- substring iteration so qwen2.5/qwen2 + phi-3.5/phi-3 are deterministic; 200-char model-name cap, null-byte rejection). * Part D: utils.warmup (compute_warmup_steps clamped [10, 1000]; ratio==0 short-circuit matches HF Trainer "no warmup" convention). warmup_auto field reuses pre-existing warmup_ratio (no duplicate). * Part E: utils.spike_recovery (frozen dataclass policy; max_attempts<=10; min_lr floor) + schema cross-validator requiring loss_watchdog=true. * Part F: utils.convergence (detect_plateau + recommend_action; the latter reuses the former so plateau heuristic stays single-source). * Part G: autopilot.decide_warmup / decide_mixed_precision wrappers; generate_config validates BOTH the YAML output path AND embedded decisions["output"] via shared utils.paths.is_under_cwd. Tests: tests/test_auto_tuning.py — 89 tests covering bound boundaries, NaN/Infinity rejection, multi-version quirk ordering, frozen-dataclass post-construction validation, plateau non-positive-mean guard, and double-containment in generate_config. Total: 3607 -> 3696 tests passing. ruff clean. Review wave: python-review (8 findings), security-review (3), code-review (8 incl. duplicate warmup_ratio HIGH and synthetic stub-loss curve), and tdd-guide (11 coverage gaps) — every finding fixed before commit. Live in-process wiring (LR-sweep training loop, spike rollback, grad-accum DataLoader rebuild, SFT precision push) is deferred to v0.32.1 — same advisory pattern as v0.30.0 --auto-quant / structured-output. Stale-install gotcha: if `soup version` shows the old version after pulling this branch, run `python -m pip install -e . --force-reinstall --no-deps`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
d0b7719858 |
feat(recipes): v0.31.0 — Model & Recipe Breadth
Expand the recipe catalog from 46 to 80 entries — every popular open-weight model family now has a validated Soup recipe. Part A — Vision (6 recipes): Llama-3.2-Vision-90B, Pixtral-12B, Qwen2-VL (7B + 72B), InternVL 2.5, MiniCPM-V 2.6 Part B — Audio (3 recipes): Qwen2-Audio, SeamlessM4T v2, Whisper-large-v3 Part C — Reasoning (7 recipes): completes the 6 DeepSeek-R1-Distill sizes, plus Qwen3-Coder, Qwen3-30B-A3B reasoning, Phi-4 reasoning Part D — Edge (8 recipes): SmolLM2 (135M / 360M / 1.7B), Qwen2.5 (0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini Part E — Domain (8 recipes): BioMistral, Meditron, CodeLlama (13B / 70B), Magicoder, Mathstral, Nemotron-4 340B, Llama-2-13b-finance Part F — Multimodal reasoning (2 recipes): Llama-3.2-Vision GRPO, Pixtral DPO Part G — Recipe-validation CI workflow on every PR touching recipe / config / data code (.github/workflows/recipe-validation.yml) Part H — 750 parametrized tests covering catalog-wide invariants: model-id safety (no `..`/`://`/null bytes), lora.target_modules non-empty, max_length within schema bounds, GRPO recipes wire reward_fn + num_generations >= 2, vision recipes set image_dir, audio recipes set audio_dir, default data path is non-empty + relative Live 100-step per-recipe smoke train (requires GPU runner) deferred to v0.31.1. Tests: 2886 → 3607 (+721). Catalog: 46 → 80 (target met). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e509614a9f |
feat(inference): v0.30.0 — Inference Excellence
- Prefix caching (--prefix-cache) wired to vLLM enable_prefix_caching.
- Speculative decoding auto-pairing (--auto-spec) with curated target→draft
map for Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek
V3/R1, Gemma 2/3. Targets without a known draft (≤8B) fall through.
- Dynamic LoRA hot-swap: POST /v1/adapters/activate/{name} + /deactivate.
Name regex enforced by FastAPI Path(pattern=…); active state protected
by threading.Lock; CORS restricted to loopback origins (hot-swap mutates
state without auth).
- Structured output: --structured-output json|regex with --json-schema
(cwd-confined via is_under_cwd; 64KB cap; top-level type required) and
--regex-pattern (2048-char cap, null-byte reject, must compile).
Constrained token sampling deferred to v0.30.1; constraint descriptor
exposed on app.state.
- Continuous-batching dashboard: --dashboard + /metrics endpoint. Thread
safe ServerMetrics with bounded deque for latencies; record_latency
runs in finally so failure paths are not a blind spot.
- OpenTelemetry tracing: --trace --trace-endpoint. OTLP endpoint SSRF
hardened matching v0.29.0 HF_ENDPOINT (scheme allowlist, 0.0.0.0
rejected, RFC1918/link-local/cloud-metadata via ipaddress.ip_address,
plain HTTP loopback-only). build_tracer idempotent — only installs
provider when current is ProxyTracerProvider/NoOpTracerProvider. Span
context via contextlib.ExitStack so __exit__ receives real exc info.
- Auto-quant picker API: --auto-quant flag + Candidate dataclass +
pick_best() shipped (generator-safe, first-wins tie-break, matches
v0.28.0 kernel_picker precedent). Live eval loop deferred to v0.30.1;
flag prints a yellow deferral warning so it is never a silent no-op.
Tests: 2801 → 2886 (+85). New tests/test_inference_advanced.py covers
all 7 parts plus review-driven negatives: Llama 3.3/4 pairing, Mistral
Large, DeepSeek V3/R1, Gemma 3 targets; /v1/adapters/deactivate;
activate-when-no-adapters → 404; metrics concurrent track_request;
OTLP private-IP / 0.0.0.0 / missing-host; pick_best empty-list and
all-failed; NaN score/latency rejection; --json-schema outside cwd;
--structured-output json without schema; --auto-quant deferral warning.
Review agents run (python-review, security-review, code-review, tdd-guide
+ manual verification-loop) and every finding addressed:
- HIGH: pick_best iterator exhaustion (materialise to list).
- HIGH: --auto-quant silent no-op (yellow warning).
- HIGH: --structured-output json silent no-op (fail-fast requires schema).
- HIGH: --json-schema path traversal (is_under_cwd containment).
- HIGH: OTel span swallowed HTTPException (ExitStack migration).
- HIGH: wildcard CORS on unauthed POST (loopback-only regex).
- MEDIUM: OTLP RFC1918/link-local rejection.
- MEDIUM: tracer provider idempotency.
- MEDIUM: record_latency in finally (no failure-path blind spot).
- LOW: active_state threading.Lock; deque replaces O(n) pop(0).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
03ddc05573 |
feat(hf): v0.29.0 — HuggingFace Hub Deep Integration
Auto-push checkpoints, HF Collections, self-hosted endpoint, HF datasets push, HF Spaces auto-deploy, model card v2. - utils/hf.py: single source of truth for HF token resolution (env > cached login), HF_ENDPOINT validation, repo_id / collection_slug validation, HfApi factory, add_to_collection. HF_ENDPOINT SSRF-hardened: scheme allowlist, 0.0.0.0 rejected, plain HTTP limited to loopback, RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via ipaddress.ip_address. - monitoring/hf_push.py: HFPushCallback pushes each save_steps checkpoint as 'checkpoint-<N>' branch. Sticky _repo_failed flag short-circuits retries after hard failure. prepare_hf_resume enforces cwd containment and passes local_dir_use_symlinks=False. allow_patterns whitelist (safetensors/bin/pt/json/tokenizer*/trainer_state.json) keeps .env and source files out of auto-pushed branches. - commands/push.py: --collection flag, generate_model_card_v2 (task / base / lr / optimizer from training_config.yaml; optional eval scorecard; markdown-active chars neutralised on task names and non-numeric scores; data_lineage HTML-escaped). --model cwd containment, repo_id validation, deprecated --token warning, commit message stripped to first 200 chars. - commands/data.py: soup data push --input --hf-dataset uploads local JSONL as HF dataset. Cwd containment on input, repo_id validation. - commands/deploy.py: soup deploy hf-space --model --space --template [gradio-chat|streamlit-chat]. render_space_template validates model repo id before substitution into rendered app.py (defeats Python injection from a crafted repo id). - commands/train.py: --push-as <repo> attaches HFPushCallback to trainer_wrapper.trainer after setup. --hf-resume pulls latest checkpoint branch into output_dir before training. Tests: +100 tests in test_hf_integration.py (65 initial + 35 review- driven) covering all parts plus validate_collection_slug negatives, build_push_callback factory paths, on_train_begin lifecycle, repo-failed short-circuit, private-IP SSRF (10.x/172.16.x/192.168.x/169.254.x/ 0.0.0.0), resolve_token edge cases. Full suite: 2801 tests pass. Reviews: python-review, code-review, security-review, tdd-guide, verification-loop — every HIGH / MEDIUM / LOW finding addressed. Docs: README '## HuggingFace Hub Deep Integration' section added; What's New replaced. CLAUDE.md / SECURITY.md / CONTRIBUTING.md updated with new test count (93/2677 -> 94/2801) and v0.29.0 security entries. License migration (MIT -> Apache-2.0) known-limitation note surfaced in What's New per plan.md deferral from v0.27.0. |
|
|
Alpamys
|
43dba01440 |
refactor(cost): polish soup cost from PR #42
- Narrow 'except Exception: pass' in _get_dataset_size to specific exceptions (OSError, ValueError, KeyError, ImportError) - _get_dataset_size returns (size, is_estimated) so the caller can warn when falling back to the 10k default (silent fallbacks are misleading on a $-estimating command) - Add -> None return type annotation on cost() (project convention) - Add variance disclaimer: 'estimates are approximate; +/- 30%' - Document pricing cadence in GPU_PRICING comment (last updated 2026-04) - Use highlight=False on json.dumps output - Fix misleading 'mock data' test comment (there is no mock) - Add 2 tests: dataset-unreadable warning, variance disclaimer rendering |
|
Salil M
|
35ccb2634b
|
Feature: add "soup cost" command for cloud GPU training cost estimation (#42)
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs * feat(cli): implement 'soup cost' command to estimate cloud GPU training costs * test(cost): add unit tests for 'soup cost' command and output formatting * docs(readme): add usage documentation for the new 'soup cost' command |
|
|
Alpamys
|
e09a742167 |
feat(training): Training Speed & Memory — CCE, FP8, grad-ckpt tiers, kernel picker, cross-doc attn, activation offload (v0.28.0)
Six new training speed/memory features, SFT-only in v0.28.0: - use_cut_ce: Cut Cross-Entropy for 128k-vocab models (8-24GB save) - quantization_aware: "fp8" — Hopper+ float8 training via torchao.float8 - gradient_checkpointing: bool | selective|medium|full|auto (VRAM-based auto) - kernel_auto_compose: benchmark + pick fastest kernel combo - packing_cross_doc_attn_mask: block-diagonal mask for sample packing - activation_offloading: cpu|disk saved-tensor offload Config-load validator rejects non-SFT tasks when speed/memory flags are set — prevents int8-QAT-wrapper crash on the string "fp8" and silent no-ops on DPO/GRPO/KTO/ORPO/SimPO/IPO/PPO/Pretrain/Reward/Embedding. Multi-trainer wiring tracked for v0.28.1. Security: - FP8 path: CUDA + Hopper+ SM capability + transformers backend - Activation-offload disk: is_under_cwd containment, TOCTOU-safe mkstemp (fd held through torch.save), weights_only=True reload, crash-safe cleanup - Kernel picker raises when all candidates lack finite time_ms - Cut CE detector matches last path component only (deepseek-ai/...-phi-... org-prefix does not trigger Phi patch on DeepSeek) - Cross-doc mask numpy-vectorised (np.tril) at max_length=1M - @model_validator gates: packing_cross_doc_attn_mask requires packing=true; v0.28.0 features require task=sft New optional extra: pip install 'soup-cli[cce]' Tests: 2585 -> 2685 (+100 in tests/test_training_speed.py, +1 file). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b7abd2a40c |
feat(v0.27.0): Multi-GPU Mastery — topology, ZeRO++, FSDP2+compile, MII, PP, recipes
- Topology detector: NVLink/PCIe sniffing via NVML (utils/topology.py); --gpus auto|N on soup train prints the exact accelerate-launch command with topology info; NCCL env hints applied via os.environ.setdefault so user/launcher overrides always win. - Accelerate launcher: build_accelerate_argv with shlex.quote on script args; num_processes bounded, mixed_precision Literal allowlist, num_machines bounded [1, 256]; is_in_distributed() recognizes torchrun + accelerate env markers (utils/launcher.py). - ZeRO++: new DeepSpeed preset (zero++/zero_pp aliases) with hierarchical partitioning + quantized weights + quantized gradients; int(1e9) for sub_group_size / stage3_max_* so DeepSpeed strict JSON accepts. - FSDP2 + torch.compile: training.use_fsdp2_compile bool wired into TrainingArguments(torch_compile=True) via extracted apply_fsdp_training_kwargs helper in utils/fsdp.py; validator requires FSDP + CUDA + backend=transformers + torch>=2.2/accelerate>=0.27, and rejects DeepSpeed+compile coexistence (cryptic runtime crash prevention). - DeepSpeed-MII backend scaffold: soup serve --backend mii registered with dependency check; live pipeline wiring deferred to v0.27.1 with explicit non-zero exit to prevent silent mis-start (utils/mii.py). - Pipeline parallelism config: training.parallelism Literal[data|pipeline] + pipeline_stages bounded [1, 16]; validator enforces stages >= 2 + CUDA + gpu_count >= stages; execution wiring deferred to v0.27.1 with Rich Panel notice (utils/pipeline.py). - Recipes: llama3-70b-fsdp2, qwen3-32b-zeropp, deepseek-v3-pipeline; recipe count 43 -> 46. - Tests: 2511 -> 2585 (+74). New test_multi_gpu.py covers topology, launcher, ZeRO++, FSDP2+compile helper behavior, DeepSpeed+compile mutual exclusion, MII key-absent + stub branches, pipeline config bounds, NCCL setdefault semantics, and CLI validator gating (asserts use_fsdp2_compile / parallelism=pipeline on CPU blocks soup train with specific error text). - Reviews: 4-agent initial wave + 3-agent re-review wave on the wiring delta. All CRITICAL/HIGH/MEDIUM/LOW findings fixed (no remaining debt). Security review clean. Known limitations (tracked as v0.27.1 issues): - Auto-reexec of accelerate-launch (currently advisory) - Live DeepSpeed-MII server pipeline - Live pipeline-parallel execution - Multi-node accelerate config - Recipe CI smoke-train validation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ddab34115c |
feat(v0.26.0): Parts B-E — Eval Gate, Trace-to-Pref, Quant-Check, Soup Cans
Closes the v0.26.0 "Red and Blue Ocean" flywheel after Part A (Registry): Train (eval-gated) -> Registry -> Deploy (quant-check) -> Trace-to-Pref -> Train. Part B — Eval-Gated Training: - soup_cli/config/schema.py: EvalGateConfig (enabled/suite/every_n_epochs/ regression_threshold/baseline/on_regression) + TrainingConfig.eval_gate field - soup_cli/eval/gate.py: EvalSuite, GateTask, run_gate, resolve_baseline, load_suite; baselines from registry:// or file - soup_cli/monitoring/callback.py: on_epoch_end + _run_eval_gate with fail-safe error handling (structured errors treated as regressions under on_regression=stop) - soup_cli/commands/train.py: --gate <suite.yaml> shortcut flag - soup_cli/commands/eval.py: gate subcommand (stub generator; live scoring v0.26.1) Part C — Trace-to-Preference: - soup_cli/data/traces/: parse_langchain, parse_openai, parse_soup_serve; build_pairs from thumbs_up / regenerations / user_edit - soup_cli/commands/data.py: from-traces + review subcommands - PII warning panel, 100,000-line cap, path containment, Literal validation Part D — Quant-Lobotomy Checker: - soup_cli/eval/quant_check.py: classify_delta (OK/MINOR/MAJOR), run_quant_check, resolve_model_ref with artifact kinds filter, table/json/markdown renderers - soup_cli/commands/eval.py: quant-check subcommand Part E — Soup Cans: - soup_cli/cans/: Manifest + DataRef (Pydantic v2); pack_entry + fork_can (100MB cap, dunder-key guard); safe tar extraction (filter='data' on py3.12+, narrow fallback, manual symlink rejection + commonpath check) - soup_cli/commands/can.py: pack/inspect/verify/fork subcommands Shared utility: - soup_cli/utils/paths.py: single is_under_cwd helper replacing 5 duplicates (os.path.realpath + commonpath — Windows 8.3 short-name safe) Tests: 103 new (29 eval_gate + 24 trace_to_pref + 23 quant_check + 27 cans) Full suite: 2511 passed on Windows Python 3.10. Security hardening (review-driven, all severities fixed): - EvalGateConfig bounds; GateTask null-byte + judge URL scheme allowlist - Narrow except in _safe_extract so TarError from filter='data' is not swallowed - resolve_model_ref artifact kinds filter (avoid wrong artifact) - Manifest.author cap + null/newline rejection; created_at ISO-8601 validation - fork_can dunder-key + null-byte rejection (prototype pollution prevention) - fork_can size cap (100MB matches pack_entry) - inspect_can/read_config refuse paths outside cwd Docs: - README.md: v0.26.0 "New in" block (flywheel); 43 recipes; all new commands in All Commands list; version examples bumped to 0.26.0; Windows-safe arrows - CLAUDE.md: architecture + test table + schema + CLI + security section extended with B/C/D/E; phase vs Part terminology clarified; release checklist step 18 adds Known Limitations section; step 20 adds comment template; step 21 adds completeness check via gh issue list --milestone - SECURITY.md: per-Part security notes (B/C/D/E) under v0.26.0 - CONTRIBUTING.md: test count + directory tree updates Local smoke: version, eval gate, eval quant-check (table + json), data from-traces, data review, can pack/inspect/verify/fork — all happy-path end-to-end. Fixed Unicode arrows (U+2192) in can.py + gate.py that crashed on Windows CP1252 consoles. Deferred to v0.26.1 (known limitations, filed as issues post-release): - eval gate/quant-check live model scoring (stub generator currently) - data from-traces quality.py judge validation; serve --trace-log collector - can run + can publish + orchestrator - eval --attach-to-registry flag; export auto-artifact registration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
4cd4bab969 |
feat(registry): add Local Model Registry / Provenance Vault (v0.26.0 Part A)
Foundation of v0.26.0 "Red and Blue Ocean" — every fine-tune is now tracked with lineage, config, eval baseline, and shippable artifacts. New module soup_cli/registry/: - hashing.py: deterministic SHA-256 of config (canonical JSON) + data (streamed) + base model; used as the entry_hash identity - store.py: SQLite store (~/.soup/registry.db) with registry_entries, registry_artifacts, registry_lineage, registry_tags. Context-manager API, cycle-safe BFS walks, AmbiguousRefError on prefix collision, LIKE-wildcard-escaped search + resolve, FK ON DELETE CASCADE. - diff.py: flat-walk ConfigChange diff + per-benchmark eval delta. New CLI commands: - soup registry push/list/show/search/diff/promote/delete - soup history <name> — lineage DAG tree viewer Security hardening (v0.26.0): - name/tag validation: alphanumeric + _-. only, null-byte rejected, name ≤128, tag ≤64 - artifact path containment via os.path.realpath + commonpath (Windows 8.3 short-name safe); enforce_cwd=True default - SQL parameterised; LIKE wildcards %/_ escaped with ESCAPE '\' - DB 600 perms on POSIX; SOUP_REGISTRY_DB_PATH env override - indirect-cycle detection in add_lineage via BFS ancestor walk - Rich markup escaped in all CLI output - resolve() raises AmbiguousRefError instead of silent None Tests: 92 new tests in tests/test_registry.py (hashing, validation, CRUD, artifacts, lineage + cycle, diff, CLI, history, security, auto-register integration with ExperimentTracker). Full suite: 2409 passed (was 2313). All review findings addressed (4 agents: python, code, security, tdd): HIGH: context manager + try/finally cleanup, FK cascade (removed manual cascade), cycle detection, LIKE wildcard escaping. MEDIUM: ambiguous resolve raises, exit 0 on user cancel, cwd captured at construction, enforce_cwd=True default, Windows ASCII-safe error messages. Deferred to v0.26.1: soup eval --attach-to-registry flag and soup export auto-artifact registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f8a20eea14 |
refactor(bench): polish prompts-file feature from PR #30
- Narrow broad 'except Exception' to specific exceptions (OSError, UnicodeDecodeError, json.JSONDecodeError) + `raise ... from` - Rename file handle `f` -> `fh` to avoid shadowing (ruff-friendly) - Clarify comment on --num-prompts ignored-when-file semantics - Strengthen test: assert actual prompts were passed to _generate (not just exit code + output substring) - Add PEP 8 second blank line between test functions |
|
|
Salil M
|
4dd09b132f
|
FEATURE: add --prompts-file option to bench command for custom test suites (#30)
* feat(bench): add --prompts-file option with path traversal security * test(bench): add unit tests for custom prompts and path traversal * docs(bench): document --prompts-file usage in README.md * feat(bench): add --prompts-file support with path validation * test(bench): add unit tests for custom prompts and security checks * style: remove trailing whitespace to pass ruff linting * test: fix mock patch targets for local imports in bench command * refactor(bench): simplify prompts-file logic and clean up comment * test(bench): update assertions to match new prompts-file semantics |
|
|
Alpamys
|
8ea99d459f |
refactor(bench): polish soup bench from PR #25
- Add -> None return type annotation (project convention) - Replace broad 'except Exception' with specific exceptions (OSError, ImportError, RuntimeError, ValueError) + `raise ... from exc` - Use `_` for unused response variable in tuple unpacking - Add CPU warning: TPS on CPU is 10-100x slower, misleading users - Add warmup run (discarded) to avoid CUDA JIT skewing averages - Document that VRAM scope includes model load (deployment planning) - Apply ruff style (trailing commas, en-dash -> ASCII, etc.) |
|
|
Salil M
|
3c339481d1
|
Add 'soup bench' command to measure model speed and VRAM usage #24 (#25)
* feat(cli): create 'soup bench' command for inference speed and VRAM measurement * register 'bench' command into the main CLI router * add test case for handling missing model paths gracefully * add 'Inference Benchmarking' section explaining the 'soup bench' tool * Added soup.yaml * style: fix linting (unused imports, inconsistent spacing) * style: sort imports in bench and test_bench to satisfy ruff * style: final import sort and grouping fix for CI * Update gitignore |
|
|
Alpamys
|
dd04679a2c |
chore(release): bump to 0.25.1 (Windows py3.9 autopilot fix)
Silent patch release. No behavior changes for the happy path — only fixes a false-positive path-traversal error in 'soup autopilot' on Windows + Python 3.9 (commit |
|
|
Alpamys
|
670968e2d5 |
fix(autopilot): Windows py3.9 path traversal false-positive
test_writes_config fails on windows-latest / Python 3.9 with exit code 1
because the path-traversal check in soup_cli/commands/autopilot.py was:
data_path = Path(data).resolve()
data_path.relative_to(Path.cwd().resolve())
On Windows + Python 3.9, Path.resolve() occasionally leaves 8.3 short
names (e.g. "C:\Users\RUNNER~1") in one of the two sides but not the
other, so relative_to raises ValueError even when both paths point to
the same location. GitHub Actions runner home dirs frequently trigger
this (the runneradmin account is created as "runneradmin" but short
names get generated as "RUNNER~1").
Fix: introduce _is_under_cwd(path) helper in soup_cli/commands/autopilot.py
that uses os.path.realpath on both sides (handles 8.3 expansion
consistently) plus os.path.commonpath for the containment check, with
case-insensitive comparison on NT. Apply it to both the --data and
--output path guards. The data_path / output_path locals are then
rebuilt from the realpath result so downstream logic sees the
canonical long-name path.
Also enriches the test assertion to print result.output and
result.exception on failure so future CI breaks are easier to diagnose
without needing to push a debug commit first.
Local verification: all 38 tests in tests/test_autopilot.py pass on
Python 3.10 Windows, full suite 2313 passed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
e4c3042a56 |
feat(v0.25.0): Beyond the Wrapper — 8 major features
Ships v0.25.0 with eight new capabilities (Parts A–H) that close every competitive gap vs LLaMA-Factory/Axolotl/Unsloth and add unique differentiators: Part A — 9 new model recipes: Llama 4 Scout (sft/dpo/grpo), Qwen 3 14B/32B/8B-grpo, Gemma 3 12B/27B-dpo, DeepSeek V3 (MoE LoRA). Part B — Tool-calling / agentic fine-tuning: new "tool-calling" data format with detection + normalization, synth data template, init template, eval scoring (tool_call_match / tool_call_name_match / tool_call_args_subset), plus qwen3-8b-tools and llama4-scout-tools recipes. Part C — RLVR (RL from Verifiable Rewards): reward_fn=verifiable routing to math_verify_reward (regex-only, no eval), code_exec_reward (subprocess sandbox with RLIMIT_AS/RLIMIT_CPU on POSIX, ephemeral tempdir cwd, concurrency cap, one-time warning panel), and json_schema_reward. verifiable_domain Literal validated via model_validator. Part D — VeRA + OLoRA PEFT methods: LoraConfig.use_vera / use_olora with mutual-exclusion validator and a unified peft_builder helper that returns either LoraConfig or VeraConfig with the right init kwargs. Part E — Apple Silicon MLX backend: detection + hardware profiling in utils/mlx, MLXSFTTrainerWrapper via mlx-lm, scaffolding DPO/GRPO wrappers rejected at config load time by SoupConfig._validate_mlx_task_support, lazy trainer registry, doctor integration, 3 MLX SFT recipes, [mlx] extra in pyproject. Part F — Data augmentation: soup data augment with rephrase / translate / style strategies, path-traversal-protected input/output, count capped 1-10, lang/styles lists bounded (10 entries × 32 chars), rate limiting, and optional --dedup. Part G — Training intelligence: forgetting detection (ForgettingDetector with 3 built-in mini benchmarks and warning levels) and checkpoint intelligence (CheckpointTracker with composite metric, early-stop on regression, safe top-N pruning refusing symlinks and non-checkpoint dirs). SQLite schema extended with checkpoint_quality + forgetting_eval tables. Part H — Autopilot: soup autopilot command with dataset/model/hardware profilers, decision engine (task/quant/peft/batch/lr/epochs/max_length/perf flags), YAML generator, and full CLI with dry-run + --yes + path-traversal protection + goal whitelist + gpu_budget bounds [1GB, 1TB]. Bakes forgetting detection + checkpoint intelligence + early-stop into the generated config. Totals: - 2313 tests passing (183 new, up from 2130) - 86 test files (8 new) - 43 ready-made recipes (14 new) - 16 built-in templates (tool-calling added) - Review findings: all CRITICAL/HIGH/MEDIUM/LOW addressed (3 documented design limitations: code_exec best-effort sandbox, prune_checkpoints TOCTOU, MLX training integration test requires real hardware) Docs: CLAUDE.md, README.md, SECURITY.md, CONTRIBUTING.md updated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
19ae61a176 |
fix: security hardening patch release (v0.24.3)
Rolls up post-review fixes from v0.24.2: - Validate Last-Event-ID header with isdigit() (prevents ValueError 500) - Type ChatRequest.messages as list[ChatMessage] (enforces role+content) - Protect _train_process reads with _train_lock (race condition) - Narrow form_to_yaml exception to ValueError/TypeError - Fix Optional[Literal] schema extraction (filter NoneType from args) - SSRF: use ipaddress.is_loopback instead of string allowlist - XSS: escapeHtml on all server-supplied innerHTML values |