mirror of https://github.com/razor-ai/soup.git
140 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
82d5693b75 |
feat(eval): Tracker & Eval Pro — 18 features (v0.43.0)
Closes the observability gap with all three competitors in one release.
Part A — Trackers
* --tracker flag (mlflow/swanlab/trackio) on soup train, mutually
exclusive with --wandb/--tensorboard. Closed allowlist via
MappingProxyType. Live integrations rely on HF Trainer's report_to.
* SOUP_TELEMETRY=1 opt-IN env var; build_telemetry_payload schema is
closed-key (no model names / dataset paths / config contents). Live
PostHog network code deferred to v0.43.1.
Part B — Eval metrics
* Pure-Python BLEU + ROUGE-1/2/L + effective_tokens_per_second.
* KL-divergence calibration framework with OK/MINOR/MAJOR thresholds.
* Model Arena Elo tournament (256-model cap, MappingProxyType view,
Rich-markup metacharacter rejection on names).
* ceval / cmmlu / aider_polyglot benchmark allowlist (live Aider
runner deferred to v0.43.1).
Part C — Profiling
* memory_snapshot_context (narrow RuntimeError catch — review fix
prevents generator-already-executing on user-body RuntimeError).
* detect_anomaly_context, nccl_bandwidth_check (h100/a100/v100/rtx
reference table; live measurement CLI surface deferred).
* write_vscode_launch with TOCTOU symlink rejection at the target
path regardless of force=True.
Part D — Demo bundles
* `soup data demo` lists / copies 4 bundled JSONL fixtures
(alpaca / sharegpt / dpo / grpo) with staged-tempfile atomic
rename + 50 MB cap + symlink rejection on the staging path.
Tests: 5389 -> 5628 (+239). Ruff clean. Five sequential review waves
(python / code / security / tdd / smoke) ran; HIGH/MEDIUM/LOW findings
all fixed including: tracker name shadow in train.py, _lcs_length DP
double-buffer bug, BLEU geo-mean policy, base_dir absolute/.. escape,
demo_bundles tmp symlink TOCTOU, vscode launch symlink TOCTOU.
Note (Windows CI): line-ending warnings (LF -> CRLF) on commit are
expected; `.gitattributes` policy is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
05093ebfdb |
feat(data): Data Pipeline Pro — 18 features, axolotl + LF parity (v0.42.0)
Schema-first surface for the data pipeline gap with Axolotl + LlamaFactory. Ships in one release: 5 new formats (prm, pre_tokenized, input_output, video, multimodal), remote URI allowlist (s3/gs/gcs/az/abfs/abfss/oci) + streaming + sharding, AOT preprocess cache + `soup data preprocess` CLI, multi-dataset interleave (concat/under/over/probs) + 8 advanced masking fields, vocab expansion (add_new_tokens / new_special_tokens / resize_vocab) + custom prompt_strategy, and document ingestion (`soup data ingest` for PDF/DOCX/MD/TXT). Live wiring for fsspec backends, AOT tokenize loop, custom prompt-strategy runtime, and PRM trainer integration is deferred to v0.42.1+ (stub-then-live pattern from v0.27.0 / v0.37.0 / v0.41.0). Schema gates fire at config load so misconfiguration fails fast. Security: full v0.42.0 hardening matrix — `_REMOTE_SCHEMES` MappingProxyType allowlist; bucket regex 1-63 chars per S3/GCS spec; userinfo / fragment / query-string rejection on remote URIs (query-string forwarded to fsspec is SSRF-adjacent); null-byte + length caps on every string-shaped input; bool-rejected-before-int on every numeric input; frozen InterleaveSpec dataclass; 10k caps on add_new_tokens; `is_under_cwd` containment on video_dir + tokenized_path schema fields and on preprocess --config / both ingest paths; `os.lstat + S_ISLNK` symlink rejection on ingest input; PRM converter type-checks completions (str) + labels (bool, not int); video field null-byte + 2KB cap; field-name threading on image-pixels validator so error messages name the actual field. 5242 tests → 5389 (+147 net). 11 review findings addressed across python-review / code-review / security-review / tdd-guide (CRITICAL→LOW). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1491025a36 |
feat(trainer): Optimizer & PEFT Zoo (v0.41.0)
Optimizer Zoo (Part A): closed-allowlist SUPPORTED_OPTIMIZERS adds 14 new
entries — BAdam, APOLLO (apollo_adamw), Adam-mini, lomo / adalomo,
grokadamw, schedule_free_adamw / schedule_free_sgd, muon, dion,
came_pytorch, ao_adamw_{fp8,4bit,8bit}. Validates name + lower-cases
deterministically; rejects non-string / empty / null-byte / >64-char.
_OPTIMIZER_PACKAGES wrapped in MappingProxyType (matches v0.36.0 _REGISTRY).
Per-module LR (Part B): training.lr_groups accepts list-of-pairs /
list-of-dicts / {pattern: lr} mapping; canonical [{pattern, lr}, ...]
storage. Capped at MAX_LR_GROUPS=32; per-pattern non-empty string
≤256 chars + null-byte rejection + re.compile + best-effort ReDoS probe;
per-LR (0.0, 1.0] + math.isfinite (rejects NaN AND ±inf) + bool rejection
(matches v0.30.0 Candidate policy); duplicates rejected. lr_groups_from_schema
bridges canonical schema shape into runtime List[LrGroup] for
build_optimizer_param_groups (first-match-wins routing). LrGroup is
@dataclass(frozen=True). PyYAML scientific-notation (1e-4) parses as
string in YAML 1.1; _validate_lr coerces str → float so soup.yaml
round-trips work. base_lr rejects bool / non-positive (defence-in-depth).
PEFT methods (Part C): LoraConfig.init_strategy="loftq" + loftq_iter
∈ [1, 10] + loftq_bits ∈ {2, 4, 8}; cross-validator rejects loftq +
use_dora / use_vera. utils/loftq_init.py exposes validators +
build_loftq_config (lazy peft.LoftQConfig with actionable ImportError
hint). LLaMA Pro: TrainingConfig.expand_layers ∈ [1, 64] +
freeze_trainable_layers (signed, |x| ≤ 1000); cross-validator requires
the pair (LLaMA Pro freezes original layers and trains only new blocks).
field_validator(mode="before") on both rejects bool BEFORE Pydantic ge/le
silently coerces True → 1. expand_model_blocks raises NotImplementedError
with v0.41.1 marker — schema-only release (mirrors v0.27.0 / v0.37.0
stub-then-live pattern). utils/block_expansion._count_layers uses
hasattr(__len__) instead of try/except TypeError so legitimate __len__
bugs surface loudly. use_mod boolean for Mixture-of-Depths (schema only —
live patch deferred to v0.41.1).
Friendly aliases: load_in_8bit / load_in_16bit (Optional[bool]) for
LlamaFactory / Axolotl users. is True policy on both — explicit False is
"no preference", not "off"; mutually-exclusive both-True rejected; alias
combined with explicit Quant Menu format raises rather than silently
overriding. Alias-driven quantization rewrite via direct
self.quantization = ... (Pydantic v2 BaseModel non-frozen path), NOT
object.__setattr__ — code review caught that the latter would silently
bypass any future field_validator on quantization.
Five review agents (4 in parallel + manual smoke for verification-loop):
all CRITICAL/HIGH/MEDIUM/LOW findings fixed. Local smoke caught a real
bug — PyYAML parsing 1e-4 as string — fixed pre-commit with 2 added tests.
5242 tests pass (+120 net new); ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
5e0872b9ea |
feat(trainer): ReLoRA + surgical PEFT non-SFT (v0.40.6 #67)
Extends the v0.39.0 ReLoRA callback (Part B) and surgical PEFT patches (Part D — Gemma4 ClippableLinear swap + 3-D fused-MoE expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, RewardModel, Pretrain, Embedding, BCO). - New shared helper soup_cli/utils/peft_wiring.py exposes apply_pre_lora_patches, apply_post_lora_patches, attach_relora_callback. - SFT migrated to the same helpers in the same release (centralisation invariant; no drift between SFT and non-SFT wiring). - SoupConfig._validate_relora_supported_tasks: task != "sft" rejection removed; MLX backend still rejected with distinct message. Review fixes: - attach_relora_callback uses `if relora_steps is None:` (project policy) so a schema-bypassing relora_steps=0 surfaces as a loud ReLoRAPolicy ValueError rather than a silent skip. - Direct attribute access on tcfg.relora_warmup_ratio / _reset_optimizer / _prune_ratio (Pydantic schema guarantees them); no getattr defaults. - 11 behavioural helper tests (Gemma4 happy path + exception swallow, post-LoRA strip happy + exception swallow, ReLoRA policy field forwarding, schema-bypass loud-fail). - Schema-gate matrix covers `task='preference'` dispatcher. Tests: 5061 -> 5122 (+61). Closes #67. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
697cc8dad7 |
feat(trainer): Quant Menu non-SFT multi-trainer wiring (v0.40.5 #66)
Extends the v0.38.0 train-time quantization menu (gptq / awq / hqq:Nbit /
aqlm / eetq / mxfp4 / fp8) from SFT-only to all 11 transformer-backend
trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel /
Pretrain / Embedding / BCO). Closes the v0.38.0 known gap.
Schema gate: SoupConfig._validate_quant_menu_supported_tasks removes the
`task != "sft"` rejection branch. MLX backend rejection retained with
distinct message; modality=text gate retained (vision/audio Quant Menu
deferred — modality-specific kwargs not yet threaded through the unified
loader).
Trainer wiring (11 sites): each non-SFT _setup_transformers replaces the
inline BitsAndBytesConfig(load_in_4bit=True, ...) block with a call to
build_quantization_config_for_loader(tcfg=tcfg, base=cfg.base, console=console)
— mirrors sft.py:420-440 exactly. kbit-prep tuple widened from
("4bit", "8bit") to ("4bit", "8bit", "mxfp4"). BitsAndBytesConfig import
removed from each non-SFT trainer.
Review fix — PPO reward model: _load_reward_model gains optional tcfg
kwarg; when supplied, the reward checkpoint loads with the same Quant
Menu config as the policy. Both PPO call sites forward tcfg=tcfg.
Defends against silent fp16 OOM on a GPTQ/AWQ/HQQ policy run.
Review fix — defence-in-depth: new TrainingConfig.reward_model field
validator rejects null bytes and caps length at 512 chars (matches
cfg.base policy). The Quant Menu loader's per-call null-byte check
in _check_local_marker remains as the runtime backstop.
Tests: +131 net new (4930 -> 5061). New tests/test_v0405_part_a.py
parametrizes 11 tasks x 7 quant formats; covers MLX rejection per task,
quantization_aware x Quant Menu cross-validator regression for non-SFT,
source-level invariants (no inline BNB literal, kbit tuple regex),
and a live mock-based dispatch test for _load_reward_model proving the
Quant Menu path is reachable when tcfg is supplied and skipped when
tcfg=None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
c85a1017b5 |
fix(v0.40.4): strip ANSI + route correctly in TestCommandFlagsExist
CI failure root cause: Rich line-wraps `--trust-remote-code` with ANSI colour escapes between `-`, `-trust`, `-remote-code` on narrow CI terminals (mirrors v0.40.3 ANSI fix). Substring assertion missed because the ANSI escapes were embedded mid-flag. Also: original test used `[cmd, "--help"]` then fell back to `["data", cmd, "--help"]`. For diff/export/merge/infer the first invocation worked (top-level commands) but the substring miss triggered the fallback into `data` subcommand, which then errored "No such command 'X'" — masking the real ANSI issue. Switched to explicit per-command argv lists. Adds the `_strip_ansi` helper from tests/test_trust_remote_code.py and routes `data generate` directly via `["data", "generate", "--help"]`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
6fdf7e2570 |
feat(trainer): multipack live HF Trainer DataLoader override (v0.40.4 Part B)
Closes #65 (deferred from v0.40.3). make_multipack_trainer_class adds a get_train_dataloader override that builds a MultipackBatchSampler(real_batches=False) — yields a flat list[int] per packed sequence, which is the contract HF DataLoader.batch_sampler expects — and installs it via DataLoader(..., batch_sampler=sampler, collate_fn=self.data_collator, num_workers=args.dataloader_num_workers, pin_memory=args.dataloader_pin_memory). drop_last is forwarded from TrainingArguments.dataloader_drop_last. Falls back to super().get_train_dataloader() when state was never attached OR when train_dataset is unset — defence-in-depth so the subclass remains safe to instantiate even when multipack is later disabled. The state-presence guard switched from falsy (`not max_seq`) to explicit `is None` (plus `not lengths` for empty-list defence) — attach_multipack_state already rejects non-positive ints, so the falsy guard would only mask configurator bugs. _get_train_sampler override stays as a defensive no-op fallback that ALWAYS delegates to super (review-fix from v0.40.4 code-review: returning a multipack list[list[int]] from this method would cause a shape mismatch if any HF eval / prediction loop bypasses get_train_dataloader and calls _get_train_sampler directly). SFT and Pretrain trainer wrappers now invoke make_multipack_trainer_class(SFTTrainer) and attach_multipack_state(...) when multipack: true. The v0.40.3 yellow advisory + standard-sampler fallback is gone. Architecture allowlist (validate_multipack_architecture) still gates at build time. tests/test_v0403_part_b.py: TestSftAndPretrainWiringDeferred renamed to TestSftAndPretrainWiringLive; the deferred-state test (_get_train_sampler returns MultipackBatchSampler when state is set) is replaced by the live-state test (_get_train_sampler always delegates to super even with state attached). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
3ab36e2aad |
feat(security): trust_remote_code opt-in across non-SFT trainers + 5 commands (v0.40.4 Part A)
Closes the v0.36.0 #63 known gap. Every non-SFT trainer wrapper (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO + the unified Preference dispatcher) now accepts trust_remote_code: bool = False on __init__, resolves once via the v0.36.0 helper (model_requires_trust_remote_code + resolve_trust_remote_code), and stores the resolved value on self._trust_remote_code. Every from_pretrained call site reads from the resolved attribute — no remaining trust_remote_code=True literal in any trainer file (asserted by tests/test_v0404_part_a.py). Five standalone commands gain a --trust-remote-code Typer flag with the same default-deny + KNOWN_SAFE_PREFIXES allowlist behaviour as soup train: soup diff, soup export, soup merge, soup infer, soup data generate. commands/train.py removes the v0.36.0 sft_kwargs split — every trainer receives trust_remote_code from the same trainer_kwargs dict. PreferenceTrainerWrapper forwards the raw bool to the inner DPO / SimPO / ORPO / IPO / BCO wrapper kwargs at both _build_inner and _build_multi_objective sites; the resolver fires inside the inner wrapper at construction time. _load_reward_model (module-level helper in ppo.py) accepts a trust_remote_code: bool parameter and resolves internally — design intent is that the helper is independently safe to call outside PPOTrainerWrapper. _export_onnx / _export_tensorrt / _export_awq / _export_gptq and _merge_adapter helpers all gain a trust_remote_code: bool = False parameter threaded from the Typer flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
21453101e7 |
fix(v0.40.3): strip ANSI escapes in help-text assertions
CI failure on previous v0.40.3 hotfix: Typer renders Rich-styled help with ANSI escape codes BETWEEN flag fragments (`--trace\x1b[0m\x1b[1;36m-log`), so a whitespace-only strip still failed to find `--trace-log` in the flattened output. Strip both ANSI codes (`\x1b\[[0-9;]*m`) and whitespace in one pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
18fc5b8d02 |
fix(v0.40.3): width-independent help-text + fastapi-skip on CI
CI failures on this commit: - macOS Typer help text wraps `--judge` / `--trace-log` to two lines on narrow CI terminals; tests asserted the raw string. Strip whitespace before match (mirrors v0.40.2 width-independent fix). - `fastapi` is not in the base CI deps (only `[serve]` extra); two `_create_app` tests ImportError-ed. Skip those tests when fastapi is unavailable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1f050fc235 |
feat(v0.40.3): Stub-to-live wave 1 (#33, #64; #65 still deferred)
Three v0.X.0 deferred-stub features become live runtime — closes #33 (harvester judge filter + serve trace log) and #64 (live CUDA OOM probe). #65 (multipack live wiring in HF Trainer) remains deferred to v0.40.4 after the adversarial 5th-review pass surfaced a Sampler[int] vs list[list[int]] shape mismatch with HF Trainer's DataLoader; helpers (`make_multipack_trainer_class`, `attach_multipack_state`, `lengths_from_dataset`, `detect_arch_name`) ship as a stub used by unit tests, but the SFT/Pretrain wrappers print a yellow advisory and fall back to the standard sampler when `multipack: true`. Live CUDA OOM probe (#64): `make_cuda_probe_fn` builds a closure that runs ONE forward+backward+step on a synthetic batch per candidate. `model.zero_grad(set_to_none=True)` runs BEFORE forward; intermediate ids/attn/labels/outputs are del-ed before `loss.backward()` so peak VRAM reflects a realistic training step (matches v0.35.0 #45 policy). `pad_id` is bounded by `len(tokenizer)` (not `vocab_size`) so extended vocabs (Llama-3 + `<|pad|>`) don't fold pad to a random byte token. SFT-only this release. Trace-to-Preference judge filter (#33 (a)): `judge_filter_pairs` reuses v0.19.0 JudgeEvaluator backends (openai/server/ollama). Threshold rejects bool/NaN/out-of-[0,1]; `_MAX_BATCH=100_000` cap applied via lazy `itertools.islice`; per-pair backend exceptions caught and DEBUG- logged (matches v0.33.0 #47 policy); `judge_provider` validated against the allowlist at the CLI boundary BEFORE constructor with a Rich-escape error message; yellow projected-call-count warning before the loop (2× per pair). Inference Server trace log (#33 (b)): `TraceLogWriter` is thread-safe (single-process lock — multi-worker documented as known limitation); path containment via shared `is_under_cwd`; null-byte/empty/non-string path rejected; cap_mb bounds [1, 10000] with explicit bool rejection. Rotation (one backup retained) refuses symlink at the backup path via `os.lstat + stat.S_ISLNK` (matches v0.33.0 #22 TOCTOU policy). Secret redaction (`hf_*` ≥8, `sk-*` ≥16, `Bearer …` ≥8 with `.` excluded so end-of-sentence period survives) applied to prompt + response and recursively to caller-supplied `extra` dict values. Streaming SSE path also records (was a coverage gap caught in adversarial review). Behaviour change: v0.40.2 users with `auto_batch_size_strategy: probe` were silently getting the static fallback. v0.40.3 actually runs a CUDA probe on first run (~5–30s, cached per (model, max_length, quant, lora_r, gpu) tuple). Reviews: 5 agents (python, code, security, tdd, verification-loop). Verification-loop run twice — once shallow smoke (PASS), once adversarial bug-hunt which found C1/C2 (multipack live wiring crash — demoted to v0.40.4), H1 (streaming SSE missing trace log — fixed), H4 (vocab_size vs len(tokenizer) on extended vocabs — fixed), H3 (Bearer regex consumed trailing period — fixed), H2 (judge cost shock — warning added), M2 (empty lengths accepted — rejected), L1 (extra dict bypassed redaction — recursive walk added). Tests: 4756 → 4855 (+99 net new) across test_v0403_part_a/b/c.py. Lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
676078056d |
fix(v0.40.2): make help-text assertions width-independent
CI on Linux/macOS runners has narrower terminals than the local Windows shell. Rich wraps long option names like ``--template-dir`` across two lines (``-\n-template\x1b...-dir``) which makes a substring check on the raw output string fail. Updated `_plain` helper in both v0.40.2 test files to strip whitespace in addition to ANSI escapes — matches the option name regardless of terminal width. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b0fc586706 |
feat(v0.40.2): Quick polish + v0.40.1 carry-overs (#36, #50, #51 + 7 papercuts)
Closes 3 originally-scheduled GitHub issues plus 7 v0.40.1 long-tail UX papercuts. No new schema fields, no new trainers — pure polish. Originally scheduled: - #36 format_gate_row helper for the eval-gate dashboard row (pure formatter in soup_cli/monitoring/display.py; passed=is True so missing field renders neutral; supports stop/warn action suffixes; multi-task " | " join). - #50 prepare_hf_resume now skips snapshot_download when local checkpoint-N is greater-or-equal to the remote highest-N. New _find_highest_local_checkpoint helper handles missing dirs / OSError / non-directories cleanly. - #51 soup deploy hf-space --template-dir <path> via new soup_cli/utils/hf_space.py:render_custom_template_dir. Containment via is_under_cwd; validate_repo_id BEFORE substitution; per-file 256 KB cap; symlinks + non-regular files rejected (TOCTOU defence per v0.33.0 #22). v0.40.1 carry-overs: - H2: data filter --min-coherence alias; data split --train no-op; data register/unregister positional <name> <path> + Optional --name/--path with conflict detection. - H3: soup quickstart --output DIR (containment-checked) routes data, config, run dir under the chosen directory. - N1/G2: apply_logging_level pushes parsed --log-level tier into the root logger so transformers / peft / trl actually respect QUIET / DEBUG. - N7: shared _resolve_model_source in commands/infer.py (used by bench.py too) — path-like-but-missing raises FileNotFoundError; non-path-like values fall through to HF download via from_pretrained. - G13: verified ONNX/AWQ/GPTQ/TensorRT install hints already correct. - M4: verified data dedup --threshold already exposed. - M5: soup runs --cwd-only + _filter_runs_by_cwd helper using os.path.realpath + commonpath (Windows 8.3 + cross-drive safe). Review-fix follow-ups landed in the same release: - soup_cli/commands/infer.py: from __future__ import annotations (Py3.9 PEP 604 fix); --output containment via is_under_cwd, late-evaluated to preserve pre-existing test contracts. - soup_cli/commands/data.py register_data + soup_cli/commands/bench.py prompts file: Path.resolve()+relative_to() → is_under_cwd (project rule for Windows 8.3 short-name safety). - soup_cli/commands/runs.py: typed _filter_runs_by_cwd, removed redundant inner import os. - soup_cli/commands/deploy.py: confirmation panel now shows --template-dir path when set, not the unused --template default. Tests: 4720 → 4756 (+36) across two new files (test_v0402_part_a.py, test_v0402_part_b.py). 5 review agents (python / code / security / tdd / verification) all clean after fixes. Known limitations: - Custom HF Space templates always create the Space with sdk=gradio regardless of the supplied app.py. Use --template streamlit-chat with the inline registry for Streamlit. Tracked for v0.40.3+. - _resolve_model_source returns ("hf", repo_id) without validate_repo_id; transformers.from_pretrained will raise loudly on malformed ids. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
56bea56c08 |
fix(v0.40.1): QA Hardening — UTF-8 bootstrap, schema strictness, multi-objective preference runtime, CLI UX
Closes the QA findings from the Windows + RTX 3050 4 GB pass (2026-05-07): - Part A: UTF-8 stdio bootstrap on Windows (closes C1/C4/H1/N5/N8/G5) - Part B: root-level `lora:` migrates into training.lora (no more silent init_strategy bypass); multi-objective preference loss runtime no longer raises NotImplementedError (primary-loss approximation; full per-batch weighted combination deferred to v0.40.2) - Part C: autopilot 7B → 1B fallback + safetensors cache probe; transformers <5.0.0 cap with INCOMPATIBLE flag in `soup doctor`; quickstart auto-switches to SmolLM2-135M on ≤6 GB VRAM; --find-lr load_local → load_raw_data import fix - Part D (subset): dynamic --template help (H4); init --force (M2); migrate JSONL friendly error (N2); eval custom -o independent of attach-to-registry + loop-shadow bug fix (G10); history suggests dataset registry (N6); doctor importlib.metadata fallback (M1) + GPU diagnostic distinguishes CPU build (N3) + dual-Python detector (N4) - Part E: recipe fuzzy-match suggestions (M3); sample filename embeds strategy (no overwrite); JSONL BOM auto-strip Net +64 tests (4656 → 4720). 4 review agents clean (python/code/security/tdd). Long-tail UX papercuts (H2/H3/N7/M4/M5 + #36/#50/#51) deferred to v0.40.2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b8506c465a |
feat(preference): v0.40.0 — Preference Variety (4 Parts: BCO + dispatcher + DPO variants + multi-objective)
Part A — BCO Trainer (Binary Classifier Optimization): new task='bco', training.bco_beta, bco.yaml template, train+sweep routing. Internal _split_dpo_rows_to_bco adapts paired DPO input to TRL's BCO unpaired schema; skipped rows logged at DEBUG (mirrors v0.33.0 #47 policy). Part B — Unified preference dispatcher: additive task='preference' + training.preference_loss Literal {dpo,simpo,orpo,ipo,bco}. Legacy task='dpo' / 'simpo' / 'orpo' / 'ipo' / 'bco' remain first-class — the new surface is purely additive, not a breaking collapse. _make_inner_cfg uses model_copy so re-validation never sees an intermediate inconsistent state and the caller's cfg is never mutated. Part C — KL-controlled DPO variants: dpo_beta_schedule (linear / cosine / exponential) + dpo_beta_end + dpo_ref_regen_epochs [1, 1000]. BetaScheduleCallback resolves total_steps lazily in on_train_begin (closes a first-cut bug where total_steps=0 silently emitted beta_end for every step). RefModelRegenCallback uses load_state_dict(strict=True) with WARNING-on-mismatch (closes a first-cut silent partial-copy hazard). Gated to DPO-family tasks only; rejected on mlx backend with distinct error message. Part D — Multi-objective preference_loss_weights (2-5 entries, key allowlist + null-byte rejection, sum-to-1 ±1e-6). Schema-level surface only; live runtime weighted-loss combination deferred to v0.40.1 with NotImplementedError stub-then-live (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA pattern). Net +118 tests (4538 → 4656). All four review-agent waves clean (Python / Code / Security / TDD). Known limitation: BCOTrainerWrapper still hardcodes trust_remote_code=True (carry-over of the v0.36.0 #63 family across non-SFT trainers). Also: add docs/ to .gitignore (internal-only docs going forward; existing docs/QUANTIZATION.md from v0.38.0 stays tracked). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e6a9c087c3 |
feat(lora): v0.39.0 — LoRA Quality (PiSSA + ReLoRA + per-pattern rank + surgical patches + templates registry)
Five PEFT-surface improvements that LlamaFactory and Axolotl maintain: - LoraConfig.init_strategy Literal["random","pissa","olora"]; PiSSA SVD init via PEFT init_lora_weights="pissa". Back-compat: use_olora=True aligns to init_strategy="olora" via dict-copy model_validator(mode="before"); explicit conflict (use_olora=True + init_strategy="pissa"/"random") rejected. Mutual-exclusion vs DoRA / VeRA. - ReLoRA callback (utils/relora.py): frozen ReLoRAPolicy with bounds-checked steps [1, 1e7] / warmup_ratio [0,1] / prune_ratio (0,1) (strict — prevents zero-everything footgun); magnitude_prune_tensor (in-place torch.kthvalue, rejects non-Tensor / single-element short-circuit); duck-typed ReLoRACallback (no transformers import at module load). TrainingConfig fields relora_steps / relora_warmup_ratio / relora_reset_optimizer / relora_prune_ratio. SoupConfig _validate_relora_supported_tasks gates to task=sft + transformers backend with distinct MLX-backend error message; multi-trainer expansion deferred to v0.39.1 (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu stub-then-live pattern). - LoraConfig.rank_pattern / alpha_pattern Optional[Dict[str,int]]; field validator caps at 256 keys × value (0, 1024], rejects bool / null-byte / empty key. Cross-validator rejects with use_vera=True (VeRA shares one rank). peft_builder propagates into LoraConfig init_kwargs. - utils/peft_patches.py: is_gemma4_model uses regex word boundary (?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$) so "ungemma4ed" no longer matches. apply_gemma4_clippable_patch swaps ClippableLinear → nn.Linear by class name (weight-copy fallback logs at DEBUG). strip_lora_dropout_for_3d_experts zeroes lora_dropout.p on 3-D weights (handles ModuleDict variant for PEFT >=0.10). apply_surgical_patches orchestrator validates model_name. Wired into sft.py _setup_transformers with is_gemma4_model gate before the pre-LoRA swap; post-LoRA 3-D dropout strip runs unconditionally (architecture-detected internally). - 16 inline templates migrated to soup_cli/templates/*.yaml + manifest.json + load_template loader (path-traversal-rejecting name validator; os.path.realpath + commonpath containment so a tampered manifest cannot read files outside the package directory; 256 KB file-size cap with inline fallback). Inline TEMPLATES kept with deprecation comment (planned removal v0.41.0+); test_templates_yaml asserts byte-equality of all 16 inline ↔ YAML pairs to prevent silent drift. Net +164 tests (4374 → 4538). All 5 review-agent waves clean before tag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f6f29ef609 |
feat(quant): v0.38.0 — Quant Menu (8 Parts: A-H)
Train-time support for 7 new quantization formats — close the width gap
with LlamaFactory. Wired into SFT trainer + transformers backend + text
modality; multi-trainer/modality expansion deferred to v0.38.1 (mirrors
v0.27.0 MII / v0.37.0 multipack stub-then-live pattern).
- Part A — GPTQ: quantization='gptq' + gptq_disable_exllama (PEFT triton)
- Part B — AWQ: quantization='awq' + GEMM/GEMV builder
- Part C — HQQ: hqq:1bit..hqq:8bit (no 7bit; not supported upstream)
- Part D — AQLM: locked-2-bit
- Part E — EETQ: locked-8-bit
- Part F — MXFP4 + FP8 dequantize-on-load
- Part G — bnb_4bit_quant_storage for FSDP+QLoRA
("crucial for fsdp+qlora" — LlamaFactory quantization.py:178)
- Part H — check_quant_distributed_compat matrix + docs/QUANTIZATION.md.
HQQ/EETQ/AQLM x {FSDP, ZeRO-3} hard-fail; BNB-4bit + FSDP without
quant_storage warns. Wired into commands/train.py startup.
Three new schema validators:
- _validate_prequantized_no_qat — pre-quantized + QAT incompatible
- _validate_bnb_quant_storage_only_with_4bit — silent no-op guard
- _validate_quant_menu_supported_tasks — sft + transformers + text gate
Net: +61 tests (4374 -> 4435). Four review-agent waves clean before tag
(python-review / code-review / security-review / tdd-guide);
verification-loop performed as manual equivalent per CLAUDE.md allowance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
06fbd15ec4 |
feat(multipack): v0.37.0 — Multipack (5 Parts A/B/C/D/E)
FFD bin-packing sampler closes the throughput gap with Axolotl on uneven-length chat data. Five focused parts: - Part A: MultipackBatchSampler (FFD + 18-arch allowlist + loud-fail vs Axolotl silent miss + _MAX_FFD_ITEMS=1M DoS cap) - Part B: schema gate (sft/pretrain only on transformers backend, multipack/packing mutually exclusive, distinct mlx error), build_multipack_sampler_for_lengths helper - Part C: neat_packing 4D attention mask + FA-vs-4D strategy picker, _MAX_MASK_ELEMENTS=2**31 / _MAX_BOUNDARY_SEGMENTS=1M caps - Part D: JinjaTemplateAnalyzer (parse-only AST walker, 128KB cap) - Part E: cross-module property tests (4-seed x 200 samples, 5k stress, FFD-to-4D-mask coherence) All five review-agent waves clean before tag (python / code / security / tdd / verification-loop). Net +125 tests (4249 -> 4374), 121 test files (+5). Live HF Trainer sampler-swap wiring deferred to v0.37.1 (mirrors v0.27.0 MII stub-then-live pattern). Schema gate + helper ship now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
815a6c4f44 |
fix(tests): strip ANSI in --trust-remote-code help-visible assertions (v0.36.0 follow-up)
CI macOS runners render Rich panel help at a narrower terminal width than local Windows. The flag --trust-remote-code is split with ANSI colour escapes between segments, so the literal substring match in the three CLI plumbing tests failed even though the flag was correct in --help output. Mirrors the existing _strip_ansi helper in tests/test_log_level.py (v0.34.0 fix for the same class of issue). Tests-only follow-up; no soup_cli/ changes; no version bump needed per release checklist policy on tests-only commits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
a5540fa1e2 |
feat(correctness): v0.36.0 — Correctness First (4 Parts: A/B/C/D)
Four silent-failure modes Soup had → loud failures, plus a
security default-deny.
- Part A: assistant-only loss masking (default true). Replaces TRL's
multi-turn heuristic with explicit IGNORE_INDEX masking. New
data.train_on_responses_only / train_on_messages_with_train_field
+ per-message train: bool field. Preferred path uses
return_assistant_tokens_mask; fallback uses incremental tokenize
delta with add_special_tokens=False to avoid double-BOS drift.
- Part B: --trust-remote-code opt-in default-deny on soup train /
chat / serve / data download / eval auto. KNOWN_SAFE_PREFIXES
allowlist (15 first-party orgs) suppresses warning panel.
Replaces 9 unconditional trust_remote_code=True call sites in
the SFT path. Non-SFT trainers + diff/export/merge/infer/generate
still hardcode trust_remote_code=True — documented v0.36.x patch.
- Part C: chat-template hardening. Tokenizers without chat_template
raise loudly instead of silent f"{role}: {content}" fallback.
New data.chat_template (registered name or raw Jinja). Filesystem
-touching Jinja directives (include/import/from/macro/extends)
blocked at config-load. Override application warns that soup push
will persist the new Jinja into tokenizer_config.json.
- Part D: OOM-probe auto batch-size. New
training.auto_batch_size_strategy: auto|static|probe. Try-halve
-then-double-to-ceiling loop, max 8 doublings, ceiling = static
× 4. ~/.soup/batch_cache.json (0600 perms, env-override
containment-checked against ~/cwd/tempdir). make_cache_key
rejects bool inputs.
Net +134 tests (4115 → 4249). All 5 review-agent waves clean
before commit; 5 HIGH / 10 MEDIUM / 5 LOW findings fixed in one
review-fix wave.
Smoke: python -m soup_cli.cli version → soup v0.36.0; all 5 new
--trust-remote-code flags surface in --help; ruff clean; pytest
4249 passed / 3 skipped / 0 failed in 2m41s on Windows py3.10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
f6e2004c9f |
fix(fp8): wire fp8_recipe through v028_features (covers 10 trainers)
PR #62 added fp8_recipe support but only wired sft.py directly. The other 10 trainers (dpo, pretrain, grpo, kto, orpo, simpo, ipo, ppo, reward_model, embedding) all route through apply_v028_speed_memory, which was calling apply_fp8_training(model) without recipe -- meaning user-set fp8_recipe='rowwise' was a silent no-op on every non-SFT task. - v028_features.apply_v028_speed_memory: read tcfg.fp8_recipe and pass through to apply_fp8_training; surface the picked recipe in the green status line so the run record reflects the actual dispatch - sft.py: drop the defensive getattr (fp8_recipe is a Pydantic field with a default, not optional) -- use tcfg.fp8_recipe directly - tests: add TestFP8RecipeViaV028Features (4 tests) verifying the recipe propagates through apply_v028_speed_memory for tensorwise / rowwise / rowwise_with_gw_hp, plus the int8-QAT path is unaffected |
|
Chinmaya Sahu
|
c13a4e0543
|
feat(fp8): add rowwise and rowwise_with_gw_hp scaling recipes for FP8 training (#62)
Add fp8_recipe config field to TrainingConfig with three torchao-backed scaling recipes: tensorwise (default, v0.28.0 behavior), rowwise (more accurate via CUTLASS), and rowwise_with_gw_hp (most accurate, grad_weight in high precision). Dispatches via Float8LinearConfig.from_recipe_name(). - schema.py: add fp8_recipe Literal field with validator requiring quantization_aware='fp8' for non-default recipes - fp8.py: update apply_fp8_training() to accept recipe parameter - sft.py: pass tcfg.fp8_recipe to apply_fp8_training() - README.md: document recipe options with comparison table - tests: 24 tests covering schema, dispatch, validation, backward compat |
|
|
Alpamys
|
892fd33f9e |
feat(trainers): v0.35.0 — Trainer Coverage (closes #60, #61, #45)
Wires v0.28.0 speed/memory features into every transformer-backend trainer (grpo / kto / orpo / simpo / ipo / ppo / reward_model / embedding) plus closes the v0.33.0 #43 oversight where dpo / pretrain accepted activation_offloading without installing offload hooks. Auto-quant --auto-quant now forwards the picked candidate's quantization to vLLM via an explicit named parameter (kwarg-splat hazard removed). Kernel auto-compose runs a forward-only benchmark loop on the trainer's actual model under torch.no_grad() so live training gradients aren't polluted (this was a critical-class bug caught by code-review pre-tag and fixed before merge). Schema gate lifted with distinct MLX-backend vs unknown-task error messages so users get the right fix. fp8 / int8 QAT guard fixed in 6 trainers (the legacy unguarded `if tcfg.quantization_aware:` would have crashed the int8 path with the string "fp8"). Net +187 tests (3928 -> 4115). New file tests/test_trainer_coverage_v035.py provides a parametrised matrix proof that every trainer x every feature is exercised on every CI matrix job. All four review-agent waves (python / code / security / tdd) clean with every CRITICAL -> LOW finding fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
5ff8d87227 |
fix(tests): strip ANSI in --log-level help-visible assertion
CI's narrower terminal width forced Rich to split the flag literal across ANSI colour escapes (`\x1b[1;36m-\x1b[0m\x1b[1;36m-log\x1b[0m\x1b[1;36m-level\x1b[0m`), so the contiguous substring `--log-level` was not present in result.output even though the flag is registered correctly. Strip ANSI codes before the substring check — same pattern applied to similar Typer/Rich help-text tests in other Python projects. No code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2cc4b5aa20 |
feat(observability): v0.34.0 — Observability & Dev UX (7 Parts)
Adds soup why, soup tui, soup runs replay, soup train --profile, .crash bundle on training exception, per-run cost in SQLite, --log-level global flag. Net +110 tests (3818 → 3928); all five review-agent waves (python / code / security / tdd / smoke) clean. - Part A: --log-level quiet|normal|verbose|debug → Rich-formatted logger on the "soup" namespace; idempotent + tier-change replaces handler. - Part B: SQLite gains cost_usd / cost_gpu_label via lazy ALTER TABLE (race-tolerant against duplicate-column on concurrent first-boot); rendered in soup runs show / replay / TUI; bool num_gpus rejected; LIKE wildcards escaped in tracker.get_run prefix match. - Part C: soup why — heuristic NaN / plateau / divergence / grad-norm / LR bounds; severity-ordered findings. - Part D: .crash bundle generator with recursive hf_*/sk-*/Bearer redaction, output_dir basename-only, os.path.realpath containment, secrets.token_hex filename, ValueError (not PermissionError) on outside-cwd; train.py except-handler writes the bundle without masking the original exception. - Part E: soup runs replay <id> — summary panel + downsampled loss curve (≤2000 points) from SQLite history. - Part F: soup train --profile — torch.profiler Chrome trace to <output>/profiles/<run_id>.trace.json; run_id rejects '.', '..', '/', '\\', null bytes; profiles dir created only on torch import. - Part G: soup tui — Textual dashboard with lazy ExperimentTracker import; markup_escape on every DB-sourced string; new [tui] extra. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b1183a1bdd |
fix(tests): macOS CI failure on isolation_strategy_linux_with_unshare
The test patched sys.platform="linux" but called the cached
_get_isolation_strategy() — on macOS / Windows CI the cache had
already been populated with the host's strategy ("sandbox-exec" on
darwin, "best-effort" on win32) before the test ran, so the platform
patch was a no-op.
Use _compute_isolation_strategy() (uncached) like every other test in
the class. Also inject a fake os.unshare via monkeypatch so the
"namespaces" branch is reachable regardless of host kernel.
Tightened the assertion from `in {"namespaces", "best-effort"}` to
`== "namespaces"` since the unshare-unavailable case is covered by
the dedicated test_isolation_strategy_linux_unshare_unavailable test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
ff55e751ab |
fix(v0.33.0): review-wave findings (CRITICAL + HIGH + MEDIUM + LOW)
Addresses findings from 5-agent review wave (python-reviewer, code-reviewer, security-reviewer, tdd-guide, smoke-verification). CRITICAL: - cans/run.py _deploy_target ollama path: rglob *.gguf result is now realpath+commonpath checked against extract_dir before forwarding to `soup deploy ollama --gguf`. Prevents a crafted symlink in the can from making rglob point at an arbitrary on-disk path. HIGH: - cans/publish.py: removed dead update_repo_settings + bare-except tag block (was a no-op network round-trip). Tag attachment via README front-matter is documented as a v0.33.x docs follow-up. - registry/attach.py lookup_entry_by_output_dir: emits ResourceWarning when the 1000-row scan limit is hit (was a silent miss). - data/collators.py CrossDocCollator: stops mutating input dicts via pop() — uses get + dict comprehension. HF Dataset rows are cached and reused; mutation broke subsequent batches silently. Bare-except now logs at DEBUG level so production degradation is inspectable. - monitoring/callback.py _write_spike_recovery_hint: added is_under_cwd guard. args.output_dir came from raw HF TrainingArguments without separate path-containment check. - trainer/rewards.py MACOS_SANDBOX_PROFILE: narrowed (allow mach-lookup) to a 3-name allowlist (SecurityServer, notification_center, opendirectoryd.libinfo). Broad mach-lookup permitted DNS / NSURLSession via launchd, defeating (deny network*). - cans/run.py: PermissionError → ValueError so a caller wrapping in `except OSError` cannot silently swallow the consent gate. PermissionError is an OSError subclass. - commands/can.py run_cmd: assigns result=None up front + explicit None guard so a future _fail bypass cannot trigger NameError on result. - utils/v028_features.py: added type annotations on apply_v028_speed_memory (model: Any, tcfg: TrainingConfig via TYPE_CHECKING, console: Console) and warn_unsupported_features. - cans/run.py: confirm_callback now annotated Callable[[Manifest], bool] for IDE introspection. - tests/test_part_b.py reexec test: drops env-var contamination (RANK/WORLD_SIZE/LOCAL_RANK/ACCELERATE_*) before run, patches imported names on train module, and forces assertion that os.execvp was called — no more silent skip-on-bypass. - tests/test_part_d.py: added TestGenerateResponseSignature source-level guard that catches the lenient logits_processor mock silently passing. MEDIUM: - cans/run.py _run_subprocess: catches subprocess.TimeoutExpired and returns rc=124 (coreutils convention) so callers see a clean CanRunResult instead of an unhandled traceback after the 24h cap. - cans/run.py: temp dir created via mkdtemp is now cleaned up on extract_can failure (try/except + cleanup_extract_dir). - cans/run.py cleanup_extract_dir: switched startswith path check to os.path.commonpath (project-standard idiom; Windows-safe). - cans/schema.py DeployTarget._safe_relpath: normalises mixed separators before splitting on '/' so foo/..\bar can no longer bypass the .. check. - utils/lr_finder.py run_lr_sweep: removed redundant local `import math as _math` (math already at module level). LOW: - eval/gate.py _parse_judge_url: removed bare http:// catchall after scheme allowlist. Defence-in-depth for callers that bypass the Pydantic GateTask validator. - utils/auto_quant.py evaluate_candidate: latency mean now divides by *completed* prompts (excludes crashed). Crashed candidate no longer appears artificially fast. - utils/auto_quant.py Candidate.__post_init__: explicitly rejects bool in score / latency_ms (bool is a subclass of int, was sneaking past). - utils/mii.py: removed `noqa: F401` on Optional import (now actually used in type annotation since we restored it). Tests added (+7, total 3811→3818): - test_part_a_wave1: attach_artifact outside-cwd rejection. - test_part_a_wave2: PermissionError→ValueError migration in 2 tests. - test_part_c: CrossDocCollator mismatched doc_lengths fallback, does-not-mutate-input-dict regression guard. - test_part_d: source-level _generate_response signature guard. - test_part_e: should_recover at max_attempts, outside-cwd skip. Lint: clean. Full suite: 3818 passed in 156s. Findings deliberately not actioned (with rationale): - code-review M1 (mii Pydantic at import-time): forward-ref resolution requires module-level definitions for FastAPI; documented in mii.py. - code-review M4 (supports_v028_features vs validator divergence): the v0.33.0 schema validator was renamed to _validate_v028_speed_memory_supported_tasks and now imports supports_v028_features — they cannot drift. - python-review LOW (_deploy_target vllm silent no-op): documented in the docstring as advisory; logging requires a console arg the helper does not currently take. - security-review LOW 8/9 (TOCTOU window, CLONE_NEWPID): theoretical; documented in CLAUDE.md security section in the next commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
66bf0d9242 |
feat(multi-gpu,serve): auto-reexec + MII live (v0.33.0 Part B)
Closes #37, #38. Final Part of v0.33.0 implementation phase. #37 Auto-reexec under accelerate launch when --gpus N>1: - soup_cli/commands/train.py gains --no-reexec opt-out flag (default behaviour: auto-reexec). - When --gpus N>1 and not already in a distributed env (RANK/WORLD_SIZE + ACCELERATE_* markers absent), train() reconstructs argv via utils.launcher.build_accelerate_argv and calls os.execvp("accelerate", argv). os.execvp replaces the current process — no leftover PID tree, stdio passes through unchanged. - Critical flags (--fsdp, --deepspeed, --resume, --wandb, --tensorboard, --yes) are forwarded to the reexec'd run so users see the same behaviour they'd get from running accelerate launch by hand. - OSError from execvp falls back to the v0.27.0 advisory (printed command) so misconfigured PATH doesn't dead-lock the user. - --no-reexec preserves the v0.27.0 print-and-exit behaviour for users who want to control env vars / stdio explicitly. #38 DeepSpeed-MII live serve: - soup_cli/utils/mii.py gains build_mii_app(pipeline, model_name) which returns a FastAPI app with /v1/chat/completions + /v1/models matching the v0.30.0 transformers backend's contract. - Pipeline is held by closure (single MII instance, thread-safe across concurrent generations). Loopback-only CORS mirrors v0.30.0 transformers backend policy. - max_tokens bounds [1, 16384], stream=True rejected (MII v0.x lacks stable streaming), pipeline crashes return 500 with generic message (no stack-trace leak). Empty response → 500. - soup_cli/commands/serve.py replaces the v0.27.0 stub-warning + Exit(1) with create_mii_pipeline → build_mii_app → uvicorn.run. Tests: +9 in tests/test_part_b.py covering /v1/models endpoint, chat happy-path with mocked pipeline returning .generated_text, streaming rejection, max_tokens bounds (low + high), pipeline failure → 500, empty pipeline response → 500, --no-reexec parameter exists, --no-reexec advisory fallback, --gpus 2 reexec calls os.execvp with accelerate argv (via monkeypatched os.execvp). Known limitations: - MII server has no streaming, no LoRA hot-swap, no /metrics dashboard, no OpenTelemetry — those are v0.30.0 transformers-backend features not yet ported. Documented in the build_mii_app docstring. - Auto-reexec assumes accelerate is on PATH; OSError path prints the command instead, matching the v0.27.0 baseline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
55d1b9312c |
feat(speed,memory): v0.28.0 features go multi-trainer (v0.33.0 Part C)
Closes #43, #44, #47. #43 Multi-trainer wiring (sft/dpo/pretrain): - New utils/v028_features.apply_v028_speed_memory(model, tcfg, base_model, console) — single shared helper for use_cut_ce, quantization_aware="fp8", kernel_auto_compose. Each feature degrades silently to a yellow advisory if the underlying lib is missing; never crashes training kick-off. - Helpers supports_v028_features(task) and warn_unsupported_features(tcfg, task) drive both the schema validator and runtime advisories. - soup_cli/trainer/dpo.py and trainer/pretrain.py now call the helper after model load (post-LoRA, post-QAT) — same hook point as SFT. - soup_cli/config/schema.py validator _validate_v028_speed_memory_sft_only renamed _validate_v028_speed_memory_supported_tasks; allowlist now {sft, dpo, pretrain}. GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Embedding still error out at config-load with a precise multi-trainer message. #44 Selective gradient-checkpoint hooks: - New utils/gradient_ckpt.install_selective_hooks(model, granularity) iterates ``model.named_modules()`` looking for transformer-block-shaped names (numeric suffix on layer path), wraps each module's ``forward`` with torch.utils.checkpoint.checkpoint based on tier: - selective: only attention sub-modules - medium: every second transformer block - full: every transformer block - Returns hook count so callers can fall back to HF native checkpointing when zero blocks were found. #47 CrossDocCollator: - New soup_cli/data/collators.CrossDocCollator wraps any base data collator and injects a block-diagonal causal ``cross_doc_attn_mask`` built from per-example ``doc_lengths``. Preferred over TRL's ``packing_strategy="attention_free"`` flag (best-effort across TRL versions). Degrades gracefully when doc_lengths is missing or shapes don't match — base attention_mask preserved, no crash. Tests: +16 in tests/test_part_c.py covering apply_v028_speed_memory (no-features, cut_ce graceful failure), supports/warn helpers extension, schema gate (dpo + pretrain accept, kto still rejects), selective hook installation across full/medium/selective with fake transformer-shaped models, CrossDocCollator passthrough + strip + injection. One existing test in test_training_speed.py updated: dpo+use_cut_ce now accepted. Known limitations: - 7 trainers (GRPO/KTO/ORPO/SimPO/IPO/PPO/RewardModel/Embedding) still reject v0.28.0 flags at config-load. Each is a 5-line addition once schema validation is satisfied; tracked as a v0.33.x follow-up. - install_selective_hooks doesn't undo earlier hooks — caller must be re-init aware. Not an issue for the typical "construct wrapper, train, exit" flow but worth noting. - CrossDocCollator emits ``cross_doc_attn_mask`` (not ``attention_mask``) to avoid clobbering the base collator's contract; downstream consumers must read the new key explicitly. The plan calls for "preferred over TRL's packing_strategy" which we satisfy via opt-in collation, not silent override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f9e6d20962 |
feat(serve): structured-output + auto-quant live (v0.33.0 Part D)
Closes #49, #53, #54. #53 Wire --structured-output into transformers generation loop: - New utils/structured_output.build_logits_processors(constraint, tok) returns a HF LogitsProcessor list. Tries outlines first (broader coverage), falls back to lm-format-enforcer, returns [] if neither installed or factory crashes — server degrades to free-form rather than 500 on a missing dep. - _generate_response gains logits_processor kwarg, forwarded to model.generate(...). Chat-completions handler builds the processor list per request (cheap; per-request build keeps the descriptor mutable for future /v1/output_constraint endpoints) and passes it down. Empty list path is unchanged from v0.30.0 free-form behaviour. #54 --auto-quant live eval loop: - New utils/auto_quant.evaluate_candidate(name, eval_fn, prompts): times mean per-prompt latency, scores correctness, marks ok=False when any prompt crashes or score < min_correct_fraction. - New utils/auto_quant.run_auto_quant_picker(candidate_specs, prompts, min_score): evaluates each candidate, calls pick_best, soft-falls- back to highest-scored ok candidate if no candidate clears the threshold so the server still binds. - serve.py replaces the v0.30.0 deferral warning with a real picker run over a fixed 3-prompt set across default_candidate_order(). Logs the picked (name, score, latency) on stdout. #49 End-to-end --push-as integration test (mocked HF): - New tests/test_part_d.py::TestPushAsResumeIntegration uses a fake huggingface_hub module via patch.dict to verify HFPushCallback constructs cleanly with a token, exposes the _repo_failed sticky flag (v0.29.0 review fix), and that prepare_hf_resume rejects output_dir outside cwd. The full HF Hub network roundtrip needs a paid sandbox repo — keeping it mocked-only is a deliberate trade (prevents flaky CI on rate limits / token rotation). Tests: +15 in tests/test_part_d.py covering build_logits_processors graceful-degrade paths (None / off / unknown / no-libs / factory crash), generate_response logits_processor plumbing, evaluate_candidate (empty / all-correct / crash / below-threshold), run_auto_quant_picker (threshold pass + soft fallback), HF push smoke. One existing test in test_inference_advanced.py updated: TestAutoQuantCLIWarning no longer expects the v0.30.1 deferral message — it now expects --auto-quant to actually run. Known limitations: - #49: full HF Hub roundtrip is mocked-only; live integration test requires a paid sandbox repo and rotating token, deferred to a separate end-to-end CI job. - #54: live re-loading of the model at the picked quant is NOT done in this commit — the picker logs the choice but the already-loaded model is served. Live re-load needs an additional bnb / awq round- trip per candidate, which is heavy for a startup-time decision; follow-up tracked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e406688f2d |
feat(training): stability auto-tuning live (v0.33.0 Part E)
Closes #56, #57, #58, #59. #56 Live --find-lr in-process LR-sweep: - New utils/lr_finder.run_lr_sweep(model, dataloader, schedule, optimizer_factory, device): per-step LR mutation + forward + backward, records loss until exhaustion or NaN/Inf divergence. - commands/train.py wires it via _live_lr_sweep_from_config (loads model + tokenizer + first N rows of cfg.data.train), with synthetic-curve fallback so users without GPU/torch still get a parseable report. #57 Loss-spike recovery hint: - SoupTrainerCallback gains spike_recovery / spike_recovery_max_attempts / spike_recovery_lr_decay; on watchdog fire writes output_dir/spike_recovery.json with previous_lr, recommended_lr (per SpikeRecoveryStrategy.compute_new_lr), should_recover, attempts. A wrapper / re-launch can resume with the decayed LR. Live optimizer rewind is intentionally NOT done — HF Trainer has no safe public API for mid-loop optimizer-state mutation; the JSON hint is the contract. #58 auto_mixed_precision push to TrainingArguments: - New SFTTrainerWrapper._resolve_mixed_precision: when tcfg.auto_mixed_precision is True, queries torch.cuda compute capability and calls pick_mixed_precision(base, cc) to set bf16=/fp16= flags. CPU short-circuits to (False, False). When the flag is False, legacy default preserved (bf16=cuda). #59 Grad-accum advisory (Phase 1): - SoupTrainerCallback gains grad_accum_auto_tune / grad_accum_pressure_threshold / grad_accum_total_vram_gb / grad_accum_current_steps / grad_accum_current_batch. - on_log samples torch.cuda.max_memory_allocated each step; if GradAccumMonitor.should_adjust crosses the threshold once, prints (batch, accum) -> (new_batch, new_accum) advisory and short-circuits (one-shot). Phase 2 (live DataLoader rebuild) needs a small TRL upstream PR — tracked as a known limitation. Wiring: - soup_cli/trainer/sft.py: _resolve_mixed_precision helper, batch_size preserved on self, SoupTrainerCallback constructor passes through new spike + grad-accum knobs. - soup_cli/monitoring/callback.py: rich Console import added (was previously module-relative); spike + grad-accum state fields and one-shot helpers. Tests: +15 in tests/test_part_e.py covering the LR-sweep loop with mocked model + optimizer (records, divergence break), mixed-precision resolver across cpu/cuda + auto-flag combinations + qwen2 fp16 quirk on Ampere, spike recovery hint write + attempts increment + disabled no-op, grad-accum advisory one-shot semantics + threshold + cuda-absent + disabled. Known limitations (release notes): - #57 spike recovery is a JSON hint, not in-process optimizer rewind - #59 Phase 2 (live DataLoader rebuild on advisory) deferred Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
8f2bc56334 |
feat(cans): soup can run + publish (v0.33.0 Part A wave 2)
Closes #34. Completes Part A — all of #32, #34, #35 now shipped. Schema bump: CAN_FORMAT_VERSION 1 -> 2 (additive). - SUPPORTED_CAN_FORMAT_VERSIONS = (1, 2): old cans still inspect/extract. - New DeployTarget Pydantic model with kind in {ollama, gguf, vllm}, name validation (no null bytes / newlines), path validation (relative-only, rejects '..' and absolute paths). - Manifest gains optional deploy_targets: list[DeployTarget] field. cans/run.py — orchestrator: - run_can(can_path, yes, deploy, extract_dir, capture_env_to, ...) validates path containment, requires --yes or explicit confirm_callback (security: auto-downloads data + auto-trains), extracts, optionally captures env, then invokes `soup train --config ... --yes` via subprocess so the trainer dispatch stays a single source of truth. - capture_env: best-effort pip freeze + python version + GPU detection, never blocks training on env-capture failure. - _deploy_target dispatches per-kind; ollama path runs `soup deploy ollama --gguf ... --name ...` if a *.gguf is present in the can. - cleanup_extract_dir: tmp-or-cwd-only safety guard around shutil.rmtree. cans/publish.py — HF Hub publish: - publish_can(can_path, repo_id, token, private, commit_message) validates can-path containment, repo_id via utils/hf.validate_repo_id, resolves token via utils/hf.resolve_token (env > cache files), uploads to repo_type='dataset' with commit-message first-line + 200-char cap (matches v0.29.0 push.py / data push policy). Tags as can-format-v1. CLI: soup_cli/commands/can.py - New `soup can run <path> [--yes] [--deploy] [--extract-dir] [--env-capture]` — confirmation panel mandatory without --yes. - New `soup can publish <path> --hf-hub <user/repo> [--private] [--message]`. Tests: +28 in tests/test_part_a_wave2.py covering schema bump (v1/v2/v3), DeployTarget validation (path traversal, null bytes, kind enum), capture_env (smoke + pip-failure tolerance), run_can (containment + confirmation gate + train-argv shape via mocked subprocess), publish_can (repo_id validation, token resolution, commit-message sanitization, HF upload via mocked HfApi), and CLI smoke (confirmation panel, missing file). One existing test_cans test relaxed (v1 == v1 -> v in {1,2}). Known follow-ups (not blocking release): - soup can run does NOT yet auto-fetch data_ref.kind=hf|url. Embedded config must reference a local data path. Filed mentally as v0.33.x follow-up. - registry_snapshot.json lineage export deferred — pack already embeds base_hash which is enough to query the source registry post-extract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ca799f6fd3 |
feat(eval,registry): live gate scoring + registry attach (v0.33.0 Part A wave 1)
Closes #32, #35. (#34 soup can run/publish deferred to Part A wave 2.) #32 Live model scoring for `soup eval gate` + `soup eval quant-check`: - gate.run_gate now dispatches judge / benchmark / custom task types, wrapping each scorer in try/except so a backend failure produces score=None + error=str(exc) instead of a silent score=1.0 pass. - New _parse_judge_url splits ollama:// / http(s):// judge_model URLs into (provider, model, api_base) for JudgeEvaluator. - New _run_judge_task / _run_benchmark_task plug into existing eval/judge.py and eval/forgetting.py runners. - New quant_check.make_model_generator(model_path) wraps transformers AutoTokenizer + AutoModelForCausalLM into a generate_fn callable; greedy by default for reproducible scores; lazy-imported. - gate_cmd / quant_check_cmd build live generators when --model is given; fall back to deterministic stub on load failure so CI without GPUs still runs the orchestration layer. - GateTaskResult.score is now Optional[float] with new error: Optional[str]. - _print_gate_result renders ERROR + reason cleanly. #35 Registry attach hooks: - registry/store.py _VALID_KINDS extended with eval_results, tensorrt. - New registry/attach.py: attach_artifact, write_eval_json (cwd-containment via realpath+commonpath), lookup_entry_by_output_dir. - `soup eval custom` gains --attach-to-registry + --output (paired); on success writes JSON results and adds eval_results artifact row. - `soup export` gains --registry-id with auto-match by source --model output dir; auto-attaches the produced GGUF artifact. Failures here are warnings, not hard exits — export already succeeded. Tests: +19 in tests/test_part_a_wave1.py covering URL parser, error propagation across all 3 task types, score=None semantics, generator factory bounds + transformers mocking, registry attach helpers (containment + missing entry), and CLI integration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
7f32a5e7c0 |
feat(security): standalone hardening (v0.33.0 Part F)
Closes #21, #22. #21 RLVR code_exec_reward: add OS-level isolation strategy detection. - New _get_isolation_strategy / _compute_isolation_strategy with cache. - Linux: best-effort os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID) in preexec_fn (Python 3.12+). Silent fallback on EPERM/ENOSYS for hosts where unprivileged user namespaces are disabled. - macOS: prefix subprocess argv with sandbox-exec + inline default-deny profile (deny network*, deny writes outside /tmp). - Windows + restricted Linux: existing RLIMIT + socket-patch + ephemeral cwd guards continue to apply (best-effort baseline). #22 prune_checkpoints: TOCTOU-safe symlink handling. - Top-level entries: explicit os.lstat + stat.S_ISLNK check (intent-clear) instead of Path.is_symlink. - shutil.rmtree now passes onerror=_abort_on_symlink to abort recursive walk if any symlink is encountered mid-walk (defence-in-depth). - OSError mid-prune is caught per-checkpoint so one bad dir does not abort the whole prune pass. Tests: +13 in tests/test_part_f_hardening.py covering strategy detection on linux/darwin/win32, sandbox profile shape, code_exec smoke tests, and TOCTOU-resistant prune behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
274490e6cc |
test(auto-tuning): strip ANSI from Typer --help output (v0.32.0 follow-up)
Rich/Typer emits per-character ANSI escapes in CI on Linux runners
("--\x1b[m-find\x1b[m-lr") so the raw substring assertion `--find-lr`
in result.output fails on ubuntu-latest x py3.12 even though it
passes on Windows where Rich auto-disables colour.
Strip ANSI before comparing — same pattern already used by
test_hf_integration.py and test_eval_platform.py.
Tests-only commit, no version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
8878eb1aa6 |
feat(training): v0.32.0 — Training Stability & Auto-Tuning
Seven new opt-in flags that turn Soup into the "fast.ai of LLM fine-tuning" — pre-flight LR range finder, auto warmup schedule, auto mixed-precision, loss-spike auto-recovery, convergence detector, VRAM-pressure advisory, and autopilot integration. * Part A: soup train --find-lr / utils.lr_finder (TypedDict result, abs-divergence threshold, NaN/Infinity rejection, MAX_NUM_STEPS=10000, is_under_cwd containment on --find-lr-output). * Part B: utils.grad_accum (GradAccumMonitor + MAX_ACCUM=1024 cap; preserves effective batch on recommend()). * Part C: utils.mixed_precision (KNOWN_PRECISION_QUIRKS map, longest- substring iteration so qwen2.5/qwen2 + phi-3.5/phi-3 are deterministic; 200-char model-name cap, null-byte rejection). * Part D: utils.warmup (compute_warmup_steps clamped [10, 1000]; ratio==0 short-circuit matches HF Trainer "no warmup" convention). warmup_auto field reuses pre-existing warmup_ratio (no duplicate). * Part E: utils.spike_recovery (frozen dataclass policy; max_attempts<=10; min_lr floor) + schema cross-validator requiring loss_watchdog=true. * Part F: utils.convergence (detect_plateau + recommend_action; the latter reuses the former so plateau heuristic stays single-source). * Part G: autopilot.decide_warmup / decide_mixed_precision wrappers; generate_config validates BOTH the YAML output path AND embedded decisions["output"] via shared utils.paths.is_under_cwd. Tests: tests/test_auto_tuning.py — 89 tests covering bound boundaries, NaN/Infinity rejection, multi-version quirk ordering, frozen-dataclass post-construction validation, plateau non-positive-mean guard, and double-containment in generate_config. Total: 3607 -> 3696 tests passing. ruff clean. Review wave: python-review (8 findings), security-review (3), code-review (8 incl. duplicate warmup_ratio HIGH and synthetic stub-loss curve), and tdd-guide (11 coverage gaps) — every finding fixed before commit. Live in-process wiring (LR-sweep training loop, spike rollback, grad-accum DataLoader rebuild, SFT precision push) is deferred to v0.32.1 — same advisory pattern as v0.30.0 --auto-quant / structured-output. Stale-install gotcha: if `soup version` shows the old version after pulling this branch, run `python -m pip install -e . --force-reinstall --no-deps`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
d0b7719858 |
feat(recipes): v0.31.0 — Model & Recipe Breadth
Expand the recipe catalog from 46 to 80 entries — every popular open-weight model family now has a validated Soup recipe. Part A — Vision (6 recipes): Llama-3.2-Vision-90B, Pixtral-12B, Qwen2-VL (7B + 72B), InternVL 2.5, MiniCPM-V 2.6 Part B — Audio (3 recipes): Qwen2-Audio, SeamlessM4T v2, Whisper-large-v3 Part C — Reasoning (7 recipes): completes the 6 DeepSeek-R1-Distill sizes, plus Qwen3-Coder, Qwen3-30B-A3B reasoning, Phi-4 reasoning Part D — Edge (8 recipes): SmolLM2 (135M / 360M / 1.7B), Qwen2.5 (0.5B / 1.5B / 3B), Gemma 2 2B, Phi-3.5-mini Part E — Domain (8 recipes): BioMistral, Meditron, CodeLlama (13B / 70B), Magicoder, Mathstral, Nemotron-4 340B, Llama-2-13b-finance Part F — Multimodal reasoning (2 recipes): Llama-3.2-Vision GRPO, Pixtral DPO Part G — Recipe-validation CI workflow on every PR touching recipe / config / data code (.github/workflows/recipe-validation.yml) Part H — 750 parametrized tests covering catalog-wide invariants: model-id safety (no `..`/`://`/null bytes), lora.target_modules non-empty, max_length within schema bounds, GRPO recipes wire reward_fn + num_generations >= 2, vision recipes set image_dir, audio recipes set audio_dir, default data path is non-empty + relative Live 100-step per-recipe smoke train (requires GPU runner) deferred to v0.31.1. Tests: 2886 → 3607 (+721). Catalog: 46 → 80 (target met). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
4faf5fc933 |
test(inference): skip 3 CLI tests when FastAPI is absent
The ubuntu-latest py3.11 matrix cell doesn't install the [serve] extra,
so `soup serve` exits early with a FastAPI-missing message before
reaching --structured-output / --auto-quant / --json-schema validation.
Add `pytest.importorskip("fastapi")` to the three tests that exercise
those CLI-level validation paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
e509614a9f |
feat(inference): v0.30.0 — Inference Excellence
- Prefix caching (--prefix-cache) wired to vLLM enable_prefix_caching.
- Speculative decoding auto-pairing (--auto-spec) with curated target→draft
map for Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek
V3/R1, Gemma 2/3. Targets without a known draft (≤8B) fall through.
- Dynamic LoRA hot-swap: POST /v1/adapters/activate/{name} + /deactivate.
Name regex enforced by FastAPI Path(pattern=…); active state protected
by threading.Lock; CORS restricted to loopback origins (hot-swap mutates
state without auth).
- Structured output: --structured-output json|regex with --json-schema
(cwd-confined via is_under_cwd; 64KB cap; top-level type required) and
--regex-pattern (2048-char cap, null-byte reject, must compile).
Constrained token sampling deferred to v0.30.1; constraint descriptor
exposed on app.state.
- Continuous-batching dashboard: --dashboard + /metrics endpoint. Thread
safe ServerMetrics with bounded deque for latencies; record_latency
runs in finally so failure paths are not a blind spot.
- OpenTelemetry tracing: --trace --trace-endpoint. OTLP endpoint SSRF
hardened matching v0.29.0 HF_ENDPOINT (scheme allowlist, 0.0.0.0
rejected, RFC1918/link-local/cloud-metadata via ipaddress.ip_address,
plain HTTP loopback-only). build_tracer idempotent — only installs
provider when current is ProxyTracerProvider/NoOpTracerProvider. Span
context via contextlib.ExitStack so __exit__ receives real exc info.
- Auto-quant picker API: --auto-quant flag + Candidate dataclass +
pick_best() shipped (generator-safe, first-wins tie-break, matches
v0.28.0 kernel_picker precedent). Live eval loop deferred to v0.30.1;
flag prints a yellow deferral warning so it is never a silent no-op.
Tests: 2801 → 2886 (+85). New tests/test_inference_advanced.py covers
all 7 parts plus review-driven negatives: Llama 3.3/4 pairing, Mistral
Large, DeepSeek V3/R1, Gemma 3 targets; /v1/adapters/deactivate;
activate-when-no-adapters → 404; metrics concurrent track_request;
OTLP private-IP / 0.0.0.0 / missing-host; pick_best empty-list and
all-failed; NaN score/latency rejection; --json-schema outside cwd;
--structured-output json without schema; --auto-quant deferral warning.
Review agents run (python-review, security-review, code-review, tdd-guide
+ manual verification-loop) and every finding addressed:
- HIGH: pick_best iterator exhaustion (materialise to list).
- HIGH: --auto-quant silent no-op (yellow warning).
- HIGH: --structured-output json silent no-op (fail-fast requires schema).
- HIGH: --json-schema path traversal (is_under_cwd containment).
- HIGH: OTel span swallowed HTTPException (ExitStack migration).
- HIGH: wildcard CORS on unauthed POST (loopback-only regex).
- MEDIUM: OTLP RFC1918/link-local rejection.
- MEDIUM: tracer provider idempotency.
- MEDIUM: record_latency in finally (no failure-path blind spot).
- LOW: active_state threading.Lock; deque replaces O(n) pop(0).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
9aff44ac5f |
fix(tests): strip ANSI from Typer --help output + patch os.environ for null-byte test (v0.29.0)
CI failures on macOS/Windows:
1. `test_endpoint_rejects_null_byte` — ``monkeypatch.setenv("HF_ENDPOINT",
"...\x00")`` raises ``ValueError: embedded null byte`` at the C-level
setenv call on macOS/Windows before ``resolve_endpoint`` can reject it.
Linux's setenv swallows it. Replace with ``monkeypatch.setattr`` on
``os.environ`` dict so the null-byte string reaches ``resolve_endpoint``
on every platform.
2. Help-text substring tests (``test_train_shows_push_as_flag_in_help``,
``test_push_shows_collection_flag_in_help``, ``test_push_subcommand_exists``,
``test_hf_space_help_shows_flags``, ``test_train_help_shows_hf_resume``,
``test_hf_space_command_registered``) — Typer injects ANSI escape codes
on macOS/Windows pytest runs, splitting tokens like ``--push-as`` into
``-`` + ``-push-as`` across escape groups. Add ``_plain()`` helper that
strips ANSI via regex and use it in every help-text assertion.
Same pattern as
|
|
|
Alpamys
|
03ddc05573 |
feat(hf): v0.29.0 — HuggingFace Hub Deep Integration
Auto-push checkpoints, HF Collections, self-hosted endpoint, HF datasets push, HF Spaces auto-deploy, model card v2. - utils/hf.py: single source of truth for HF token resolution (env > cached login), HF_ENDPOINT validation, repo_id / collection_slug validation, HfApi factory, add_to_collection. HF_ENDPOINT SSRF-hardened: scheme allowlist, 0.0.0.0 rejected, plain HTTP limited to loopback, RFC1918 / link-local / cloud-metadata (169.254.x) IPs rejected via ipaddress.ip_address. - monitoring/hf_push.py: HFPushCallback pushes each save_steps checkpoint as 'checkpoint-<N>' branch. Sticky _repo_failed flag short-circuits retries after hard failure. prepare_hf_resume enforces cwd containment and passes local_dir_use_symlinks=False. allow_patterns whitelist (safetensors/bin/pt/json/tokenizer*/trainer_state.json) keeps .env and source files out of auto-pushed branches. - commands/push.py: --collection flag, generate_model_card_v2 (task / base / lr / optimizer from training_config.yaml; optional eval scorecard; markdown-active chars neutralised on task names and non-numeric scores; data_lineage HTML-escaped). --model cwd containment, repo_id validation, deprecated --token warning, commit message stripped to first 200 chars. - commands/data.py: soup data push --input --hf-dataset uploads local JSONL as HF dataset. Cwd containment on input, repo_id validation. - commands/deploy.py: soup deploy hf-space --model --space --template [gradio-chat|streamlit-chat]. render_space_template validates model repo id before substitution into rendered app.py (defeats Python injection from a crafted repo id). - commands/train.py: --push-as <repo> attaches HFPushCallback to trainer_wrapper.trainer after setup. --hf-resume pulls latest checkpoint branch into output_dir before training. Tests: +100 tests in test_hf_integration.py (65 initial + 35 review- driven) covering all parts plus validate_collection_slug negatives, build_push_callback factory paths, on_train_begin lifecycle, repo-failed short-circuit, private-IP SSRF (10.x/172.16.x/192.168.x/169.254.x/ 0.0.0.0), resolve_token edge cases. Full suite: 2801 tests pass. Reviews: python-review, code-review, security-review, tdd-guide, verification-loop — every HIGH / MEDIUM / LOW finding addressed. Docs: README '## HuggingFace Hub Deep Integration' section added; What's New replaced. CLAUDE.md / SECURITY.md / CONTRIBUTING.md updated with new test count (93/2677 -> 94/2801) and v0.29.0 security entries. License migration (MIT -> Apache-2.0) known-limitation note surfaced in What's New per plan.md deferral from v0.27.0. |
|
|
Alpamys
|
a63e8875f0 |
refactor(tests): polish DPO example tests from PR #48
- Split the 22-assert config-values test into 3 focused tests (task+data, training hyperparams, LoRA config) so a deliberate example change surfaces in one targeted test, not a wall of asserts - Add module docstring explaining why these tests lock the example state - Add `from __future__ import annotations` (defensive; matches 14 other test modules in the project) - Rename `f` -> `fh` in _load_jsonl to avoid shadowing short name - Drop asserts on secondary fields (warmup_ratio, weight_decay, scheduler, logging_steps, etc.) -- they're tweakable knobs, not the example's teaching points; test brittleness > coverage here |
|
|
Chinmaya Sahu
|
0e69b210e3
|
feat(examples): add DPO example config, sample data, and tests (#48)
Add a working DPO (Direct Preference Optimization) example using the current Pydantic config schema with Llama 3.1 8B Instruct and QLoRA. - examples/configs/dpo_example.yaml: DPO config with all core training and LoRA parameters, plus commented-out advanced options - examples/data/dpo_sample.jsonl: 8 preference pairs in DPO format with ShareGPT-style message lists for chosen/rejected - tests/test_dpo_example.py: 7 tests validating config loading, field values, data format detection, and data validation - examples/README.md: document the new DPO with QLoRA example |
|
|
Alpamys
|
43dba01440 |
refactor(cost): polish soup cost from PR #42
- Narrow 'except Exception: pass' in _get_dataset_size to specific exceptions (OSError, ValueError, KeyError, ImportError) - _get_dataset_size returns (size, is_estimated) so the caller can warn when falling back to the 10k default (silent fallbacks are misleading on a $-estimating command) - Add -> None return type annotation on cost() (project convention) - Add variance disclaimer: 'estimates are approximate; +/- 30%' - Document pricing cadence in GPU_PRICING comment (last updated 2026-04) - Use highlight=False on json.dumps output - Fix misleading 'mock data' test comment (there is no mock) - Add 2 tests: dataset-unreadable warning, variance disclaimer rendering |
|
Salil M
|
35ccb2634b
|
Feature: add "soup cost" command for cloud GPU training cost estimation (#42)
* feat(cli): implement 'soup cost' command to estimate cloud GPU training costs * feat(cli): implement 'soup cost' command to estimate cloud GPU training costs * test(cost): add unit tests for 'soup cost' command and output formatting * docs(readme): add usage documentation for the new 'soup cost' command |
|
|
Alpamys
|
e09a742167 |
feat(training): Training Speed & Memory — CCE, FP8, grad-ckpt tiers, kernel picker, cross-doc attn, activation offload (v0.28.0)
Six new training speed/memory features, SFT-only in v0.28.0: - use_cut_ce: Cut Cross-Entropy for 128k-vocab models (8-24GB save) - quantization_aware: "fp8" — Hopper+ float8 training via torchao.float8 - gradient_checkpointing: bool | selective|medium|full|auto (VRAM-based auto) - kernel_auto_compose: benchmark + pick fastest kernel combo - packing_cross_doc_attn_mask: block-diagonal mask for sample packing - activation_offloading: cpu|disk saved-tensor offload Config-load validator rejects non-SFT tasks when speed/memory flags are set — prevents int8-QAT-wrapper crash on the string "fp8" and silent no-ops on DPO/GRPO/KTO/ORPO/SimPO/IPO/PPO/Pretrain/Reward/Embedding. Multi-trainer wiring tracked for v0.28.1. Security: - FP8 path: CUDA + Hopper+ SM capability + transformers backend - Activation-offload disk: is_under_cwd containment, TOCTOU-safe mkstemp (fd held through torch.save), weights_only=True reload, crash-safe cleanup - Kernel picker raises when all candidates lack finite time_ms - Cut CE detector matches last path component only (deepseek-ai/...-phi-... org-prefix does not trigger Phi patch on DeepSeek) - Cross-doc mask numpy-vectorised (np.tril) at max_length=1M - @model_validator gates: packing_cross_doc_attn_mask requires packing=true; v0.28.0 features require task=sft New optional extra: pip install 'soup-cli[cce]' Tests: 2585 -> 2685 (+100 in tests/test_training_speed.py, +1 file). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
b7abd2a40c |
feat(v0.27.0): Multi-GPU Mastery — topology, ZeRO++, FSDP2+compile, MII, PP, recipes
- Topology detector: NVLink/PCIe sniffing via NVML (utils/topology.py); --gpus auto|N on soup train prints the exact accelerate-launch command with topology info; NCCL env hints applied via os.environ.setdefault so user/launcher overrides always win. - Accelerate launcher: build_accelerate_argv with shlex.quote on script args; num_processes bounded, mixed_precision Literal allowlist, num_machines bounded [1, 256]; is_in_distributed() recognizes torchrun + accelerate env markers (utils/launcher.py). - ZeRO++: new DeepSpeed preset (zero++/zero_pp aliases) with hierarchical partitioning + quantized weights + quantized gradients; int(1e9) for sub_group_size / stage3_max_* so DeepSpeed strict JSON accepts. - FSDP2 + torch.compile: training.use_fsdp2_compile bool wired into TrainingArguments(torch_compile=True) via extracted apply_fsdp_training_kwargs helper in utils/fsdp.py; validator requires FSDP + CUDA + backend=transformers + torch>=2.2/accelerate>=0.27, and rejects DeepSpeed+compile coexistence (cryptic runtime crash prevention). - DeepSpeed-MII backend scaffold: soup serve --backend mii registered with dependency check; live pipeline wiring deferred to v0.27.1 with explicit non-zero exit to prevent silent mis-start (utils/mii.py). - Pipeline parallelism config: training.parallelism Literal[data|pipeline] + pipeline_stages bounded [1, 16]; validator enforces stages >= 2 + CUDA + gpu_count >= stages; execution wiring deferred to v0.27.1 with Rich Panel notice (utils/pipeline.py). - Recipes: llama3-70b-fsdp2, qwen3-32b-zeropp, deepseek-v3-pipeline; recipe count 43 -> 46. - Tests: 2511 -> 2585 (+74). New test_multi_gpu.py covers topology, launcher, ZeRO++, FSDP2+compile helper behavior, DeepSpeed+compile mutual exclusion, MII key-absent + stub branches, pipeline config bounds, NCCL setdefault semantics, and CLI validator gating (asserts use_fsdp2_compile / parallelism=pipeline on CPU blocks soup train with specific error text). - Reviews: 4-agent initial wave + 3-agent re-review wave on the wiring delta. All CRITICAL/HIGH/MEDIUM/LOW findings fixed (no remaining debt). Security review clean. Known limitations (tracked as v0.27.1 issues): - Auto-reexec of accelerate-launch (currently advisory) - Live DeepSpeed-MII server pipeline - Live pipeline-parallel execution - Multi-node accelerate config - Recipe CI smoke-train validation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
899ad8edf7 |
test(eval_gate): strip ANSI escapes in train --help CI assertion
`test_train_gate_flag_accepted` asserted `"--gate" in result.output`, but Typer/Click under CI emits ANSI color codes that split the flag name into non-contiguous chars: `\x1b[1;36m-\x1b[0m\x1b[1;36m-gate\x1b[0m`. The literal "--gate" substring is never present. All 9 OS × Python combos failed on the v0.26.0 Parts B-E push. Fix: strip ANSI via regex before checking. Also assert on "eval-gated" from the option description to double-check the flag is wired to its help text. CI-only / tests-only: no soup_cli/ changes, no version bump needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ddab34115c |
feat(v0.26.0): Parts B-E — Eval Gate, Trace-to-Pref, Quant-Check, Soup Cans
Closes the v0.26.0 "Red and Blue Ocean" flywheel after Part A (Registry): Train (eval-gated) -> Registry -> Deploy (quant-check) -> Trace-to-Pref -> Train. Part B — Eval-Gated Training: - soup_cli/config/schema.py: EvalGateConfig (enabled/suite/every_n_epochs/ regression_threshold/baseline/on_regression) + TrainingConfig.eval_gate field - soup_cli/eval/gate.py: EvalSuite, GateTask, run_gate, resolve_baseline, load_suite; baselines from registry:// or file - soup_cli/monitoring/callback.py: on_epoch_end + _run_eval_gate with fail-safe error handling (structured errors treated as regressions under on_regression=stop) - soup_cli/commands/train.py: --gate <suite.yaml> shortcut flag - soup_cli/commands/eval.py: gate subcommand (stub generator; live scoring v0.26.1) Part C — Trace-to-Preference: - soup_cli/data/traces/: parse_langchain, parse_openai, parse_soup_serve; build_pairs from thumbs_up / regenerations / user_edit - soup_cli/commands/data.py: from-traces + review subcommands - PII warning panel, 100,000-line cap, path containment, Literal validation Part D — Quant-Lobotomy Checker: - soup_cli/eval/quant_check.py: classify_delta (OK/MINOR/MAJOR), run_quant_check, resolve_model_ref with artifact kinds filter, table/json/markdown renderers - soup_cli/commands/eval.py: quant-check subcommand Part E — Soup Cans: - soup_cli/cans/: Manifest + DataRef (Pydantic v2); pack_entry + fork_can (100MB cap, dunder-key guard); safe tar extraction (filter='data' on py3.12+, narrow fallback, manual symlink rejection + commonpath check) - soup_cli/commands/can.py: pack/inspect/verify/fork subcommands Shared utility: - soup_cli/utils/paths.py: single is_under_cwd helper replacing 5 duplicates (os.path.realpath + commonpath — Windows 8.3 short-name safe) Tests: 103 new (29 eval_gate + 24 trace_to_pref + 23 quant_check + 27 cans) Full suite: 2511 passed on Windows Python 3.10. Security hardening (review-driven, all severities fixed): - EvalGateConfig bounds; GateTask null-byte + judge URL scheme allowlist - Narrow except in _safe_extract so TarError from filter='data' is not swallowed - resolve_model_ref artifact kinds filter (avoid wrong artifact) - Manifest.author cap + null/newline rejection; created_at ISO-8601 validation - fork_can dunder-key + null-byte rejection (prototype pollution prevention) - fork_can size cap (100MB matches pack_entry) - inspect_can/read_config refuse paths outside cwd Docs: - README.md: v0.26.0 "New in" block (flywheel); 43 recipes; all new commands in All Commands list; version examples bumped to 0.26.0; Windows-safe arrows - CLAUDE.md: architecture + test table + schema + CLI + security section extended with B/C/D/E; phase vs Part terminology clarified; release checklist step 18 adds Known Limitations section; step 20 adds comment template; step 21 adds completeness check via gh issue list --milestone - SECURITY.md: per-Part security notes (B/C/D/E) under v0.26.0 - CONTRIBUTING.md: test count + directory tree updates Local smoke: version, eval gate, eval quant-check (table + json), data from-traces, data review, can pack/inspect/verify/fork — all happy-path end-to-end. Fixed Unicode arrows (U+2192) in can.py + gate.py that crashed on Windows CP1252 consoles. Deferred to v0.26.1 (known limitations, filed as issues post-release): - eval gate/quant-check live model scoring (stub generator currently) - data from-traces quality.py judge validation; serve --trace-log collector - can run + can publish + orchestrator - eval --attach-to-registry flag; export auto-artifact registration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
4cd4bab969 |
feat(registry): add Local Model Registry / Provenance Vault (v0.26.0 Part A)
Foundation of v0.26.0 "Red and Blue Ocean" — every fine-tune is now tracked with lineage, config, eval baseline, and shippable artifacts. New module soup_cli/registry/: - hashing.py: deterministic SHA-256 of config (canonical JSON) + data (streamed) + base model; used as the entry_hash identity - store.py: SQLite store (~/.soup/registry.db) with registry_entries, registry_artifacts, registry_lineage, registry_tags. Context-manager API, cycle-safe BFS walks, AmbiguousRefError on prefix collision, LIKE-wildcard-escaped search + resolve, FK ON DELETE CASCADE. - diff.py: flat-walk ConfigChange diff + per-benchmark eval delta. New CLI commands: - soup registry push/list/show/search/diff/promote/delete - soup history <name> — lineage DAG tree viewer Security hardening (v0.26.0): - name/tag validation: alphanumeric + _-. only, null-byte rejected, name ≤128, tag ≤64 - artifact path containment via os.path.realpath + commonpath (Windows 8.3 short-name safe); enforce_cwd=True default - SQL parameterised; LIKE wildcards %/_ escaped with ESCAPE '\' - DB 600 perms on POSIX; SOUP_REGISTRY_DB_PATH env override - indirect-cycle detection in add_lineage via BFS ancestor walk - Rich markup escaped in all CLI output - resolve() raises AmbiguousRefError instead of silent None Tests: 92 new tests in tests/test_registry.py (hashing, validation, CRUD, artifacts, lineage + cycle, diff, CLI, history, security, auto-register integration with ExperimentTracker). Full suite: 2409 passed (was 2313). All review findings addressed (4 agents: python, code, security, tdd): HIGH: context manager + try/finally cleanup, FK cascade (removed manual cascade), cycle detection, LIKE wildcard escaping. MEDIUM: ambiguous resolve raises, exit 0 on user cancel, cwd captured at construction, enforce_cwd=True default, Windows ASCII-safe error messages. Deferred to v0.26.1: soup eval --attach-to-registry flag and soup export auto-artifact registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |