mirror of https://github.com/razor-ai/soup.git
184 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Salil M
|
168ecc0100
|
Add `--nccl` flag to `soup doctor` for multi-GPU bandwidth checks (#178)
* feat(doctor): add --nccl flag to measure and validate multi-GPU bandwidth * test(doctor): add mocked CUDA tests to verify --nccl skip and success behaviors * docs(readme): document the new --nccl bandwidth check flag for the doctor command |
|
Alpamys
|
b344aa881a |
feat(loop): soup loop CLI-first data flywheel capstone (v0.58.0)
Connects 8 existing uniques into one workflow: production traces -> preference pairs -> Eval-Gated DPO -> canary deploy -> rollback, all from a single CLI with budget guardrails and per-iteration replay. Modules (live): - utils/loop_state.py: LoopState frozen + atomic .soup/loop.yaml I/O - utils/canary_router.py: deterministic SHA-256 routing + BucketStats - utils/loop_budget.py: parse_budget_string + check_budget + UTC rollover - utils/loop_iteration.py: IterationRecord + write/read/list manifests - utils/loop_daemon.py: WatchConfig + run_once + watch daemon - commands/loop.py: init / status / pause / resume / watch / canary / replay Three review waves fixed 1 CRITICAL + 7 HIGH + 9 MEDIUM + 2 LOW total: python-review wave 1 (BucketStats lock scope + TOCTOU lstat-before-write on _check_path + init_state + NUL-byte on _bucket_for_key); code-review wave 2 (watch preserves paused / budget-skip writes no manifest / canary autoroll persisted to LoopState / route() math.ceil for sub-bucket predictability / parse_budget_string usd-only friendly error / list_iterations swallows OSError / module-top replace import); security + tdd wave 3 (_check_dir TOCTOU mirrors _check_path pattern, exact-boundary tests at _MAX_STR_FIELD=512 and _MAX_FILE_BYTES=1 MiB, bool-rejection on 4 counters, empty-string rejection on 3 optional-str). verification-loop: manual CPU smoke covering init / status / pause / resume / watch --max-iterations / canary / replay end-to-end. Notes: - ASCII arrows (->) in user-facing help text (CI test_help_output_is_ascii_safe). - Source-grep tests use Path(__file__).resolve().parent.parent for cwd- independence (defends against monkeypatch.chdir side-effects from earlier tests in the suite). - Stage callbacks ship as no-op stubs; v0.26 trace-to-pref / eval-gate / v0.30 multi-adapter deploy wiring is operator-driven via WatchConfig fields. Pre-wired versions tracked for v0.58.1. Test count: 8998 -> 9193 (+195 net in tests/test_v0580.py). Lint clean. Full repo pytest green (9105 pass + 53 skipped pre-fixes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
76dfbb363e |
test(adapters): mock os.environ.get for null-byte/CRLF env tests (CI fix)
POSIX setenv (and Windows equivalent) reject null bytes + control chars
at the syscall boundary, so monkeypatch.setenv("SOUP_BRANCHES_DIR",
"/some\x00path") raises ValueError on every CI runner before our code
ever sees the env var.
Stub os.environ.get directly so the helper's rejection branch is
exercised exactly as it would be if the env var arrived through some
other channel (subprocess env inheritance, in-process programmatic
mutation, etc).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
8577bc2800 |
test(adapters): strip ANSI before help-output substring asserts (v0.57.0 CI fix)
Rich-wrap-CI workaround — same fix pattern as v0.55.0 / v0.56.0:
CliRunner output contains ANSI color escapes that break literal
'--top-k' in output substring matches because Rich renders option
names as -\x1b[0m\x1b[1;36m-top-k.
Adds _ANSI_RE + _strip_ansi() helper to each of the 4 test files
(test_v0570_part_{a,b,c,d}.py) and routes every help-output
substring assertion through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
7da82d355a |
feat(adapters): v0.57.0 — git for LoRA (diff, merge, blame, branch)
Ships "soup adapters {diff,merge,blame,branch,checkout,branches}" — git-shaped
UX for LoRA adapter management. Pure-numpy math (no torch); TOCTOU defence on
every path; atomic writes throughout.
Part A — adapters diff (utils/adapter_diff.py, ~270 LOC):
Per-layer ΔW Frobenius norm + relative drift; effective-rank delta via
SVD entropy; top-K changed projections; JSON/Markdown/table output.
Part B — adapters merge (utils/adapter_merge.py, ~280 LOC):
Four strategies — linear (weighted avg), ties (Yadav et al. — trim/
elect-sign/disjoint), dare (Yu et al. — drop+rescale, deterministic),
svd (low-rank reconstruction). MergeReport.verdict='UNKNOWN' is a stub;
live canary verdict via v0.55 eval gate ships in v0.57.1.
Part C — adapters blame (utils/blame.py, ~190 LOC):
Leave-one-out plan emitter + budget tracker (parse_budget mirrors v0.48.0
data_mix idiom). Per-shard work table + feasibility check. Live ablation
runner raises NotImplementedError v0.57.1 (mirrors v0.27.0 / v0.50.0 /
v0.56.0 stub-then-live pattern).
Part D — adapters branch / checkout / branches (utils/adapter_branch.py,
~230 LOC):
SHA-256 snapshot pointers under ~/.soup/branches/ (SOUP_BRANCHES_DIR
override, $HOME/$CWD/$TMPDIR-bounded). Drift detection on checkout —
refuses restore when source SHA != snapshot SHA. CRLF/null-byte
rejection on env override (mirrors v0.51.0 hub-endpoint policy).
5-agent review-fix wave (1 CRITICAL + 9 HIGH + 11 MEDIUM + 4 LOW):
- TIES tied-sign defaults to +1 (np.sign(0)==0 would silently zero all
tied parameters)
- load_branch / delete_branch reject symlinks via os.lstat + S_ISLNK
before read/unlink
- merge output safetensors + adapter_config.json atomic writes with
symlink target rejection at output path; source config size-capped
at 256 KB
- compute_adapter_diff weights-file path symlink-rejected via lstat
BEFORE is_file() (defends against .safetensors -> /etc/passwd escape)
- _count_dataset_rows opens via realpath captured at containment check
(closes TOCTOU window)
- diff --output write is atomic (tempfile + os.replace)
- SOUP_BRANCHES_DIR rejects every C0 control char, not just null
- SUPPORTED_STRATEGIES is now frozenset (matches v0.41.0+ allowlist
policy); STRATEGY_ORDER tuple preserved for canonical iteration
- 5× pytest.raises(Exception) tightened to FrozenInstanceError
- 2 zero-assertion Part D tests converted to real assertions
- Added: bool base_model rejection, top_k boundary 1/201, density=1.0
inclusive bound, inf weight rejection, tied-sign positive default,
bool False for num_shards/budget_seconds, POSIX symlink rejections
for diff weights / merge output / load_branch / delete_branch,
no-top-level-torch source-grep guards, traversal delete_branch.
Plus v0.56.0 follow-up: test_v0560.py version-floor tests widened from
exact-match to floor-check (matches v0.51.0 / v0.54.0 idiom — every
subsequent release would otherwise edit this one line).
Test count: 8849 → 8998 (+149 net in 4 new files; 4 POSIX-only symlink
tests skipped on Windows). Full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
a3810823d1 |
refactor(train): harden diagnose-gate rank guard from PR #169
PR #169 wired LOCAL_RANK==0 guard on _run_diagnose_gate so distributed launches only run the gate on one worker per machine. Two minor polish items on top of the merged version: - Wrap the int() parse in try/except ValueError. A malformed LOCAL_RANK (garbage value from a misconfigured launcher) would previously crash the post-training gate. Falling back to True is safer than silently skipping the gate -- over-running is recoverable, under-running hides failures. - Expand the docstring to explain why we use LOCAL_RANK (per-machine) rather than RANK (global): the gate reads the local output_dir, so one gate per machine is the right granularity for typical single- machine multi-GPU runs. Documents the choice for future readers. - Add a focused test (test_diagnose_gate_handles_malformed_local_rank) asserting the safe fallback path. |
|
Yixuan Xu
|
4c2a578ac0
|
Guard diagnose gate on distributed worker ranks (#169)
Co-authored-by: mzl2233 <mzl2233@users.noreply.github.com> |
|
|
Alpamys
|
7d81496c69 |
test(diagnose): strip ANSI before help-output substring asserts (v0.56.0 CI fix)
Rich's CliRunner output on CI carries ANSI escape codes that split long option names like `--badge` and `--diagnose-gate` across colour-reset boundaries (`-\x1b[0m\x1b[1;36m-badge`), breaking naive `"--badge" in result.output` substring checks. Same fix pattern as v0.55.0 CI hotfix. Failures: tests/test_v0560.py::TestCli::test_diagnose_help and TestTrainDiagnoseGate::test_help_lists_flag on all 9 CI matrix cells. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
72189baba7 |
feat(diagnose): soup diagnose — post-training model report card (v0.56.0)
Six failure-mode probes (forgetting / refusal / format / mode_collapse / memorization / contamination) + FailureReport frozen dataclass + SVG badge + soup train --diagnose-gate. Same OK/MINOR/MAJOR taxonomy as v0.26.0 Quant-Lobotomy. - soup diagnose <run-id> [--evidence|--output|--badge|--attach-to-registry] - soup train --diagnose-gate <evidence.json> refuses MAJOR runs - diagnose_report added to registry._VALID_KINDS Review wave (4 agents): 4 HIGH + 8 MEDIUM + 2 LOW addressed — atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), realpath containment, evidence size cap, contamination combined-complexity cap, ReDoS probe, extras null-byte sanitisation, extract_row_text centralisation, tokenize delegates to _eval_text. Test count: 8676 -> 8849 (+123 in test_v0560.py + 50 net adjustments). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
04c504e761 |
test(eval): strip ANSI before help-output substring asserts (v0.55.0 CI fix)
3 macOS CI failures from the v0.55.0 push — Rich wraps option names with ANSI escapes when the terminal is narrow (macOS CI runners default to a smaller width than Linux/Windows), so substring searches like `"--goal" in result.output` fail because the actual output contains `\x1b[1;36m-\x1b[0m\x1b[1;36m-goal\x1b[0m`. Project precedent: v0.53.5 / v0.53.6 / v0.53.8 / v0.53.9 all hit the same pattern; tests/test_auto_tuning.py and tests/test_eval_platform.py already ship `_ANSI_RE` + `_strip_ansi` helpers. Failures fixed: tests/test_v0550.py::TestCLIPlumbing::test_eval_design_help tests/test_v0550.py::TestEvalAgainst::test_against_help tests/test_v0550_followups.py::TestEvalAgainst::test_against_cli_help_lists_flag Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
58d7d510bf |
feat(eval): soup eval design — derive evals from data (v0.55.0)
Trainer libraries help you RUN evals — none help you DEFINE them.
v0.55.0 closes that gap with 5 new subcommands:
- soup eval design <data> --goal "..." → goal-conditioned EvalDesign
(TF-IDF salience + scorer dispatch)
- soup eval discover <data> → held-out canaries + memorization probes
(farthest-first Jaccard clustering)
- soup eval lock + soup eval coverage → SHA-256-checksummed artifact +
gap analysis vs v0.54.0 task taxonomy
- soup eval gate-install --baseline R → pre-push regression gate
(paired-bootstrap CI, shlex.quote)
- soup eval against B --candidate C → run-vs-run paired-bootstrap CI
Heuristic / CPU-only — no GPU required. Lazy imports across all 6 new
modules so `soup --help` startup remains < 200 ms.
New registry artifact kinds: eval_suite, canaries.
New tracker accessor: ExperimentTracker.get_metric_series(run_id, metric).
Security policy (all atomic-write + read surfaces):
- cwd containment via os.path.realpath + commonpath
- unconditional os.lstat + stat.S_ISLNK rejection (TOCTOU defence)
- atomic write via tempfile.mkstemp + os.replace
- shlex.quote for shell-script generation (NO hand-rolled escape)
- MappingProxyType on every registry / metric / scorer map
- frozen dataclass on every public return type
- bool-as-int rejection on every numeric input
- DoS caps: 10k subsample for TF-IDF + clustering hot paths
Review-fix coverage across 4 agents (python / security / code / tdd):
0 CRITICAL + 7 HIGH + 11 MEDIUM + 6 LOW resolved before commit.
Tests: 8571 → 8676 (+105 net).
Lint: ruff clean.
Smoke: every CLI command + every failure mode exercised in /tmp/soup_smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
9e18643ea0 |
test(advise): fix cross-platform CI failures in test_v0540
Two failures on ubuntu/macos/windows × py3.9/3.11/3.12 after v0.54.0 push: 1. test_env_null_byte_falls_back: monkeypatch.setenv can't set raw NUL into the OS env layer (POSIX execve + Win32 SetEnv both refuse). Switched to a temporary `advise_history.os.environ` swap so the helper's defence-in-depth NUL guard is still exercised without going through the C env layer. 2. test_default_missing_data: Click 8.0–8.1 returns rc=0 on `no_args_is_help=True` invocations; Click 8.2+ returns rc=2 (the "missing command" convention). CI runners had the newer Click; dev box had the older. Accept both renderings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
600686cd70 |
feat(advise): soup advise — pre-flight decision (v0.54.0)
`soup advise <data.jsonl> --goal "..."` returns one of PROMPT_ENG / RAG / SFT / DPO / GRPO with a confidence, reason, and reverse-when criterion BEFORE the user spends 8 hours on a GPU. Layer above autopilot — autopilot picks hyperparams AFTER the training decision; advise picks the training decision itself. Three Parts: - Part A: Verdict engine — TASK_CATEGORIES + CHOICES allowlists, frozen Verdict / DatasetProfile / ROIEstimate dataclasses, pure- Python classify_task + compute_dataset_profile + build_verdict rubric (DPO / GRPO floor 500 / PROMPT_ENG floor 50 / RAG / SFT). - Part B: Probe runner — synth_probe_baselines + synth_probe_lora_delta heuristic stubs with forward-compat model/device/lr/timeout_seconds kwargs (v0.54.1 lifts to live model loading per stub-then-live cadence used by v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus). - Part C: Cross-project learning — ~/.soup/advise_history.jsonl with cross-process file locking (fcntl on POSIX, sidecar <path>.lock + msvcrt on Windows). `soup advise compare` reads history; env override SOUP_ADVISE_HISTORY_PATH containment-checked to $HOME / $CWD / tempdir (mirrors v0.36.0 SOUP_BATCH_CACHE_PATH policy). CLI: Typer subcommand group `run` / `explain` / `compare` plus argv preprocessor in cli.py that maps `soup advise data.jsonl` → `soup advise run data.jsonl`. Scoped to argv[1] == "advise" only (code-review HIGH fix — defends against rewrites when an unrelated arg contains the literal string "advise"). Schema: AdviseConfig (goal / probe / record) field on SoupConfig honors the plan's cross-cutting bullet. Security: cwd-containment + os.lstat + S_ISLNK symlink reject on every path input; atomic writes via tempfile.mkstemp + os.replace on scratch + history; per-line 64 KB cap + 16 MiB file cap on history reads; bool / finite / NUL / oversize guards on every public input; Rich markup escape on user-controlled output. Reviewed by python / code / security / tdd / architect agents — every finding fixed before commit (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW). Test count: 8400 → 8571 (+136 in tests/test_v0540.py, +35 net adjustments to v0.53.x version-pin assertions to forward-compat >=). Note: Windows CRLF / LF warnings during stage are .gitattributes- governed and benign. CI runs on ubuntu-latest / windows-latest / macos-latest × Python 3.9 / 3.11 / 3.12. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
cfdabf2b3b |
feat(v0.53.11): GRPO Plus finish + preference live
Closes v0.50.1 (#123, #126, #127), v0.49.1 (#119), v0.40.1 (#68). #123 — live math kernels for 6 GRPO variants (gspo/dapo/dr_grpo/bnpo/ two_sided/rft) + `_GRPOTrainerVariant` HF Trainer subclass via `make_grpo_trainer_variant` factory. Variant compute_loss reads kernel inputs FIRST (no double-forward); falls back to super() only on missing attrs. Case-insensitive variant normalisation before lru_cache. #126 — PRMTrainerWrapper + `_PRMTrainer` HF Trainer subclass with real compute_loss (gather hidden states at step_positions -> reward_head -> MSE via compute_prm_loss). Dataset wrapped in datasets.Dataset.from_list for HF Trainer compatibility. Bool-before-isinstance guard on batch_size. #127 — GRPOStabilityCallback inherits transformers.TrainerCallback (lazy), live EMA ref-model update in on_step_end with strict=True + fallback-to-strict=False-with-WARNING on key mismatch (silent corruption defence). math.isfinite guard on alpha. #119 — LongLoRA forward override via LongLoRAForwardOverride context manager with idempotent install (_soup_longlora_patched marker prevents re-entry double-wrap), 256-char class name cap on regex match, restore on __exit__ AND on exception. #68 — true per-batch weighted-sum preference combine reading policy/ref logps from TRL inputs + each compute_*_term kernel + combine_losses. Explicit None checks on trainer attrs (no `or` on possibly-tensor), DEBUG log on per-term skip. Review fixes from 4 agents (python/code/security/tdd): 10 HIGH + 8 MEDIUM + 7 LOW — see CLAUDE.md v0.53.11 entry for the full list. Test count: 8330 -> 8400 (+75 in test_v05311.py: 54 initial + 21 review-fix coverage gaps). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
76f033a6ff |
feat(v0.53.10): Quick wins + packaging + UX wiring
7 issues closed: - #150 [mix] pyproject extra bundles scikit-optimize so `soup data mix --optimize` runs the Bayesian loop instead of the v0.48.0 Dirichlet fallback; new describe_default_optimizer() helper labels the active backend without paying skopt's import cost. - #113 [data-pro] extras (langdetect + presidio-analyzer) with lazy fall-through helpers in utils/data_score (broader language coverage + Presidio entity recognition on top of the v0.47.0 regex baseline). Llama-Guard-3-1B documented as a manual recipe (license + size). - #154 SOUP_POSTHOG_KEY / SOUP_POSTHOG_ENDPOINT env override via sentinel-based explicit-vs-env precedence; HTTPS-only + RFC1918/link-local rejection on the endpoint; null-byte / control-char / >256-char rejection on the key. - #152 --hub flag plumbed on chat / serve / infer / merge / export / push via shared utils/hubs.apply_hub_to_cli_model + prefetch_model_from_hub helpers; push uses upload_repo (skips HF-specific Collections + model-card auto-render on non-HF hubs). - #153 `soup data download --hub modelscope|modelers` live SDK (lifts the v0.53.8 advisory-only path); friendly ImportError advisory when the SDK is missing. - #155 Web UI Tool Outputs panel — `loadToolOutputs` polls /api/tool-outputs every 3s; XSS-safe DOM-built table (textContent per cell, no innerHTML for user-controlled fields); Bearer token threaded via the v0.53.9 window._authToken bootstrap. - #156 SoupTrainerCallback.on_step_end records tool_calls counts from kwargs['inputs'] into the global tool buffer. Best-effort (# noqa: BLE001 per project policy — training must never crash). 13 review-fixes applied (4 HIGH / 5 MEDIUM / 4 LOW): - HIGH PostHog explicit-endpoint precedence sentinel - HIGH absolute path leak in local_path advisory reduced to relpath - HIGH Rich markup escape on base / local_path / cache_dir - HIGH callback # noqa: BLE001 per project policy - MED `import time` moved out of try block - MED oversize key + explicit-empty key rejection tests - MED source-grep regression guards (advisory-removal, helper imports across 5 non-push commands) - MED `prefetch_model_from_hub` outside-cwd cache_root rejection - LOW empty-list + bool-True tool_calls no-op tests - LOW push.py uses upload_repo + validate_hub_name regression guard Test count: 8285 -> 8330 (+45 in tests/test_v05310.py). Full suite green; ruff clean; on Win+Py3.10. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e8422660a5 |
fix(tests): strip ANSI codes before help-output substring asserts (v0.53.9)
CI's Rich pipeline emits styled output that splits `--vocab-size` into multiple ANSI-bracketed spans (e.g. `\x1b[36m-\x1b[0m\x1b[36m-vocab\x1b[0m\x1b[36m-size\x1b[0m`), breaking naive `"--vocab-size" in result.output` checks. Locally Rich auto-detects non-TTY and skips the codes, so the regression only shows on CI (ubuntu/macos/windows × 3.9/3.11/3.12). Fix: small `_plain()` helper using `re.sub(r"\x1b\[[0-9;]*m", "", ...)` applied to the 7 failing assertions. Same approach already used in several other v0.5x test modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
bde9d149a0 |
feat(v0.53.9): Live Dashboard + UX + Bench + Standalone CLIs
Eight features that close out the v0.44.x live-monitoring deferrals plus a long tail of standalone CLI wins: - #94 /api/train/stream async SSE with per-subscriber cursor + JS EventSource consumer; SoupTrainerCallback pushes TrainEvent on each on_log. - #95 soup ui --public derives LAN IP via SOCK_DGRAM connect-trick, prints scannable QR; --auth-token override; SPA bootstrap hydrates window._authToken from ?token= + sessionStorage and cleans the URL via history.replaceState; CORS regex auto-widens to loopback + RFC1918 in public mode; set_auth_token rotation race fixed via threading.Lock. - #98 soup serve --reasoning-parser strips <think>...</think> (and OpenThinker tags); pre-compiled regex with marker-token fast-path + 1 MiB cap + leading-newline-only strip. - #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON endpoint; best-effort observation hook in callback.on_log. - #15 soup tokenizer train: BPE training CLI with raw-path lstat symlink rejection, 50 MiB total / 8 KiB per-line caps, post-mkdir output-dir re-check, --special-token NUL/oversize dedup, vocab bounds [256, 200000]. - #26 soup bench --p50 --p95 renders extra per-prompt tail-latency Rich table; --prompts-file gains symlink rejection. - #28 soup bench --backend auto: MLX weights.npz probe (per-entry lstat) -> config.json model_type keyword -> transformers fallback; SOUP_BENCH_BACKEND env hint. - #12 examples/synthetic_workflow.{md,yaml} end-to-end walkthrough. Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the python / code / security / tdd review agents. Notable HIGH: - QR token now consumed by SPA (was unreachable previously). - set_auth_token rotation lock-protected, 8-thread stress tested. - Tokenizer input + output symlink TOCTOU defence on raw path. - SSE generator switched to async (asyncio.sleep) for non-blocking multi-subscriber operation. - CORS regex for --public LAN mode (the old fixed allowlist of http://0.0.0.0:port never matched a real Origin header). - _has_mlx_weights per-entry lstat so a symlinked weights.npz can't trigger MLX dispatch. Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the relaxed v0.53.8 version-pin asserts in tests/test_v0538.py). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
53bb82afeb |
fix(v0.53.8.1): hatch artifacts directive — fix PyPI duplicate-filename 400
v0.53.8 PyPI publish failed with: 400 Invalid distribution file. ZIP archive not accepted: Duplicate filename in local headers Root cause: `[tool.hatch.build.targets.wheel.force-include]` shipped `soup_cli/data/_fixtures/` AND `packages = ["soup_cli"]` recursed into the same path, so both the wheel and sdist contained each JSONL twice. Fix: switch from force-include to `artifacts = [...]` which adds non-Python files to the existing package tree exactly once. Standard hatchling pattern for shipping data files inside an already-packaged directory. Version bumped to v0.53.8.1 (patch) — same code surface, just a build config fix. v0.53.8 GitHub release remains as the feature changelog; PyPI ships under v0.53.8.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
7b98982ab6 |
fix(test): v0.53.8 CI hardening — strip ANSI + use _repo_root() helper
Five v0.53.8 CI failures (ubuntu/macos × py3.9/3.11/3.12):
1. test_help_lists_hub_flag — Typer's Rich-rendered help wraps long
option help across ANSI box-drawing lines; "--hub" appears as
"│ --\nhub" in the CI terminal renderer. Strip ANSI + collapse
whitespace before asserting.
2-5. test_pyproject_version / test_*_extra_present / test_force_include
— used `Path("pyproject.toml")` (relative to cwd). CI invokes pytest
from a different cwd than the repo root on at least one matrix
entry. Switched to a `_repo_root()` helper that derives from
`__file__` (matches v0.43.0 Part D demo_bundles approach).
Local re-run: 66/66 v0.53.8 tests pass after the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
6d2170c4f3 |
feat(v0.53.8): Remote data + Hubs + Trackers (wave 2) — 6 features
- #85 fsspec live loaders — data/loader.py routes the v0.42.0 fsspec scheme allowlist (s3:// / gs:// / gcs:// / az:// / abfs:// / abfss:// / oci://) through fsspec.open with validate_remote_uri containment BEFORE connection. Friendly Rich panel names the pip install advisory when the backend SDK is missing. Threads data.streaming + data.buffer_size. Row count capped at 1M. - #130 Hub dispatcher live — utils/hubs.download_repo() and upload_repo() lazy-import per backend (huggingface_hub / modelscope / openmind_hub). Shared _validate_repo_id_shape (bool / null-byte / leading-slash / .. / control-char / oversize) + cwd containment on local_dir / folder_path. commands/train.py pre-fetches non-HF base into .soup_hub_cache/ (sanitised slug, idempotent on resume, cfg.base updated via model_copy). soup data download --hub flag plumbed. Multi-command rollout for chat / serve / infer / merge / export / push tracked for v0.53.9. - #89 [trackers] pyproject extra bundles mlflow / swanlab / trackio; tracker_missing_dep_message surfaces a friendly pip install advisory via importlib.util.find_spec (non-executing probe). - #90 utils/trackers.send_telemetry_payload — opt-IN via SOUP_TELEMETRY=1; lazy httpx; 1s hard timeout; HTTPS-only with SSRF re-validation (mirrors v0.51.0 hub endpoint policy); silent-fail on every exception. - #93 Fixtures migrated to soup_cli/data/_fixtures/ — zipapp / namespace-package safe via [tool.hatch.build.targets.wheel.force-include]; _bundle_source_path falls back to examples/data/ for editable installs. - #69 utils/hf_space.detect_space_sdk(requirements_text) — picks "streamlit" / "gradio" from the rendered requirements.txt; closes the v0.40.2 known limitation that custom Spaces always defaulted to gradio. Wired into commands/deploy.py. Review pass: python-review + code-review + security-review ran in parallel; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). Highlights: cwd-containment on local_dir/folder_path, Windows ..\ traversal defence on .soup_hub_cache slug, Pydantic model_copy(update=...) instead of attribute mutation, idempotent pre-fetch via cache probe, 1M-row cap on remote materialisation, SSRF re-validation on telemetry endpoint override, 256 KB cap on detect_space_sdk input, modelscope.push_model commit_message kwarg removed (would TypeError at runtime), find_spec instead of __import__ to avoid swanlab side-effects. Test count: 8162 -> 8257 (+66 in tests/test_v0538.py + 29 net adjustments). Lint clean. CPU smoke: version, --help, load_config_from_string with hub: modelscope passes; mlx + non-HF rejected; data download --hub modelscope advisory rendered; detect_space_sdk live on real requirements.txt bodies; package-data fixtures resolve from soup_cli/data/_fixtures/. v0.53.7 known limitation #1 (bash 501 marker) bumped to v0.53.9. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
a26511f868 |
fix(test): v0.53.7 — symlink test accepts either rejection branch
test_load_jsonl_rows_rejects_symlink: when the symlink target is outside
cwd, is_under_cwd's realpath resolution catches it before the lstat check
fires. Both rejections are valid security guards; broaden the regex to
match either error message ("symlink" or "under cwd").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
9c9f962676 |
fix(test): v0.53.7 CI hardening — importorskip(fastapi) + Arrow target setup
CI run 25805598433 failed across all 9 OS×Python cells: - TestToolEndpointsLive / TestAnthropicMessagesStreaming / TestReviewFixesVllmAnthropicLive ModuleNotFoundError: fastapi (CI does not install [serve] extra) → autouse fixture pytest.importorskip() - test_load_pretokenized_dataset_rejects_symlink: load_pretokenized_dataset called datasets.load_from_disk on the symlink target before the lstat check ran → moved the lstat + S_ISLNK check to the entry of the helper so symlinks reject before any load attempt - test_redact_exc_message_handles_windows_paths: hardened _redact_exc_message to strip both POSIX absolute paths and Windows-style paths regardless of host platform Local pytest tests/test_v0537.py: 112 passed, 7 skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
18d8b36114 |
feat(v0.53.7): Data Forge + Pipeline live (wave 1) — 11 items
Closes v0.47.0 deferrals (#111, #112), community QA (#75), v0.42.1 wave 1 (#87, #86, #88), and 5 v0.53.6 stub-to-live carry-overs (#102 vLLM parity + SSE streaming, #103 tool HTTP endpoints, #105 instantiate_trainer_plugins, #106 run_recipe DAG runner). - #88 markdown ingest heading split (split_markdown_by_headings) - #112 soup data decontaminate --benchmark-file (cwd-contained operator corpus + symlink rejection) - #87 prompt_strategy live resolver (resolve_prompt_strategy + lru_cache, importlib-based; per-row hook in sft_format.py) - #86 soup data preprocess AOT tokenize (atomic Arrow shard write + cache metadata sidecar; SFT + Pretrain wrappers short-circuit on format='pre_tokenized' + tokenized_path with cache-hash gate) - #111 forge --judge-provider {ollama,anthropic,vllm} live (lazy v0.20.0 providers; SSRF parity) - #75 QA log entry for synth-data provider manual smoke - #106 run_recipe LIVE for 6 NODE_KINDS (seed / llm_text / code / judge / validator / sampler); atomic checkpoint via tempfile.mkstemp; resume rehydrates predecessor outputs from per-node sidecar JSONL; lstat-on- raw-path symlink rejection (v0.33.0 #22 TOCTOU parity); failed_reason path-redacted - #105 instantiate_trainer_plugins LIVE for cce_plugin / grokfast / spectrum / llmcompressor / sonicmoe / math_verify (lazy imports, friendly pip-install advisory on missing dep) - #103 POST /v1/tools/python + /v1/tools/web_search LIVE (Bearer auth gate, deny-by-default domain allowlist, 5s timeout, 5-result cap). bash reverted to HTTP 501 — security review caught /bin/sh -c child escapes RLVR sandbox's OS-level isolation; deferred to v0.53.8. - #102 vLLM /v1/messages parity LIVE on both backends; CORS loopback-only - #102 Anthropic-shape SSE streaming on /v1/messages LIVE; Cache-Control: no-store; CRLF/NUL/oversize strip on model+msg_id (header injection) Review fixes (1 CRITICAL + 11 HIGH + 17 MEDIUM + 10 LOW from python-reviewer + code-reviewer + security-reviewer + tdd-guide) all addressed in this commit. Test count: 8051 → 8162 (+111 in tests/test_v0537.py). Test files: 189 → 190. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
c9e32f4e19 |
fix(test): v0.53.6 CI hardening — importorskip(fastapi) + strip ANSI
CI failures on `tests/test_v0536.py`:
1. `ModuleNotFoundError: No module named 'fastapi'` on Ubuntu/macOS runners
— fastapi is in the [serve] extra, not [dev]. Added
`pytest.importorskip("fastapi")` to all 7 tests that use TestClient
(matches the pattern in `_build_app`).
2. `--execute` / `--output` / `v0.53.7` substring checks failed because
CI terminals render Typer/Rich help text with style spans (`-` and
`-execute` end up in separate `\x1b[...]m` runs). Added `_strip_ansi`
helper + wrapped 5 substring checks. Same fix pattern as the v0.53.5
`--live` CI fix on test_v0535.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
70326f1aa7 |
feat(v0.53.6): plugin callback + Anthropic /v1/messages + n-gram spec + 3 stubs
Ships v0.53.6 "Plugin + Agent + Anthropic API" — 6 features, 3 live and 3 stub-then-live with v0.53.7 markers. LIVE - #101 SoupPluginCallback fans HF Trainer events to enabled Soup plugin hooks (pre_train/post_train/pre_step/post_step). Hook exceptions swallowed at WARNING. Hook snapshot collected once and passed to ctor (race-free per code-review fix). Wired into all 13 transformer-backend trainers via utils/peft_wiring.attach_plugin_callback. - #102 POST /v1/messages on transformers backend reuses the v0.45.0 anthropic_messages converter + existing chat handler. Streaming -> 501. Validation errors -> generic 'Invalid request' body, details at DEBUG (security-review redaction fix). - #104 n-gram speculative decoding: NgramSpecConfig.num_draft_tokens threaded through model.generate(prompt_lookup_num_tokens=N). Mutually exclusive with assistant_model. STUB-THEN-LIVE (v0.53.7) - #103 /v1/tools/{python,bash,web_search} return HTTP 501. - #105 instantiate_trainer_plugins validates then NotImplementedError. - #106 run_recipe + 'soup data recipe --execute --output <dir>' with CLI-side is_under_cwd containment before the live runner. Tests: 7998 -> 8051 (+53 in tests/test_v0536.py). Lint clean. Three review agents (python / code / security); every HIGH/MEDIUM fixed. TDD coverage gaps closed (ngram-None regression, max_tokens cap, run_recipe boundaries, console-print failure swallow). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
8bd67f697f |
fix(test): strip ANSI before --live substring check (v0.53.5 CI fix)
Rich splits the `--live` token across ANSI colour codes on narrow CI terminals (`-\x1b[0m\x1b[1;36m-live`), so the raw-output substring assertion failed on macOS/Windows runners but passed locally on a wide terminal. Strip ANSI escapes before the check — matches how earlier test files (e.g. test_v0402_part_b) handle Rich-coloured `--help` output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
bc1f060da9 |
feat(adaptive): v0.53.5 — Adaptive Training (BETA → stable)
Closes #114, #115, #116, #117, #118, #17 — lifts every v0.48.0 BETA deferral and adds the deepseek-v3-reasoning recipe. - #114 DynamicCurriculumCallback live (TrainerCallback + all_reduce + JSONL) - #115 curriculum_dynamic schema gate widened to 13 transformer trainers - #116 soup data mix --live runs real proxy soup train subprocesses - #117 skopt.Optimizer(GP) wrapped behind OptimizerProtocol - #118 MixOptimizationReport.elapsed_seconds excludes failed candidates - #17 deepseek-v3-reasoning GRPO recipe Test count: 7935 → 7998 (+63). 4 review agents (python/code/security/tdd) ran sequentially; every CRITICAL→LOW finding fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
89b06efb8c |
feat(longctx): v0.53.4 — Long Context + Architecture
Six closes lifting the v0.49.0 LongLoRA hardening + v0.41.0 LLaMA Pro deferred stubs, plus a UX upgrade to the CUDA-OOM friendly message: - #11 utils/errors.py: OOM hint now names --batch-size / --grad-accum - #122 flash_attn.is_flash_attn_v3_available() + LongLoRA+FA3 schema reject - #120 LongLoRA arch allowlist expansion (Mistral / Qwen / Phi); Mixtral intentionally excluded (regex matches the bare 'mistral' token only) - #121 apply_long_context_config auto-detects 'llama3' when caller passes rope_scaling_type=None and the model config carries a Llama 3.1 rope_scaling block - #83 block_expansion.expand_model_blocks LIVE (deepcopy last-N blocks, zero-init residual projections, append, bump num_hidden_layers) + apply_llama_pro_freeze + shared apply_block_expansion_if_configured helper wired into SFT + Pretrain (mirrors v0.40.6 peft_wiring centralisation policy so SFT and Pretrain stay in lock-step) - #74 HF push surface QA — test plan recorded in tests/qa/v053_qa.md; live execution against a private HF repo deferred to a credentialed contributor Review pipeline (python / code / security / tdd agents) ran; every CRITICAL -> LOW finding addressed: - bool-first guards in _check_model_name (defends against int subclass) - is_supported_longlora_arch defensive non-string surface (returns False, never raises) matching v0.53.3 is_known_vlm_base policy - _truncate_for_message(value, limit=64) bounds the base echo in LongLoRA error messages (security MEDIUM, mirrors v0.34.0 crash.py) - null-byte + non-string TypeError guards on validate_longlora_compat task / backend params (matches v0.50.0 validate_long_context_grpo_compat) - _get_layers_module uses explicit `is None` not falsy shortcut (defends against nn.Module.__bool__ overrides on subclasses) - _zero_init_block_residual returns bool + warnings.warn when neither standard projection matches the cloned block (non-Llama-shaped arches still train but lose the LLaMA Pro identity-init guarantee) Test count: 7879 -> 7935 (+56 net; +49 in new tests/test_v0534.py). Lint clean. CPU smoke verified on a real transformers.LlamaForCausalLM: 4 -> 6 layers, down_proj + o_proj actually zeroed on PyTorch tensors, old blocks frozen + new blocks trainable, forward pass finite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
29f875f6ef |
feat(grpo): v0.53.3 — grpo_fp16 routing + vision-GRPO VLM base probe
Two surgical fixes from the v0.50.0 GRPO Plus deferred-stub family land: - #128 grpo_fp16 routing: GRPOTrainerWrapper._build_precision_kwargs returns {fp16, bf16} per (device, grpo_fp16) matrix (CPU/MPS/XPU → both False, CUDA + grpo_fp16=True → fp16/!bf16, default CUDA → legacy bf16). SoupConfig._validate_grpo_fp16_amp_exclusive rejects the silent-mutex combo with auto_mixed_precision=True; short-circuits when task != 'grpo' so the v0.50.0 task-gate diagnosis fires first. - #129 vision-GRPO base probe: KNOWN_VLM_REGEX covers 10 VLM families (Qwen2-VL/Qwen2.5-VL/QVQ/Pixtral/InternVL/Llama-3.2-Vision/LLaVA/ MiniCPM-V/Idefics/ShareGPT4V/Fuyu) with word-boundary anchors; is_known_vlm_base returns False (never raises) on bad input; validate_vision_grpo_compat now accepts optional base kwarg with 64-char error-message truncation. YAML pairing vision_grpo: true with a non-VLM base is rejected at schema load with a friendly families listing instead of a cryptic runtime AttributeError. Scope: 4 larger v0.53.3 items (#127 stability callback, #123 GRPO variant losses, #126 PRMTrainerWrapper, #68 multi-objective preference live combine) are scope-deferred to v0.53.4 — each warrants its own focused release per the v0.40.x stub-then-live cadence. Tests: 7842 -> 7879 (+37 in tests/test_v0533.py). Four review agents (python/code/security/tdd) ran; every HIGH/MEDIUM/LOW finding fixed (task-gate priority short-circuit, MPS branch documented, 64-char error truncation, QVQ regex coverage, 512-byte boundary test). Two pre-existing v0.50.0 Part E tests migrated `base: test-llama` -> `base: Qwen/Qwen2-VL-7B-Instruct` to clear the new probe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2292e81c3f |
feat(modality): v0.53.2 — lift Modality II stubs (distill + classifier + EBFT/GDPO + reasoning_effort)
Closes #132, #133, #135, #137. Records #71 ONNX QA (partial — tiny-gpt2 PASS, TinyLlama-1.1B blocked by host RAM during onnx.load post-process). New trainer wrappers: - DistillTrainerWrapper (soup_cli/trainer/distill.py) — student + frozen teacher, KL/JS divergence kernels scaled by T**2, device-bridge for HF Trainer auto-CUDA promotion, DataCollatorForSeq2Seq for variable-length loss-masked rows, separate trust_remote_code resolution per model. - ClassifierTrainerWrapper (soup_cli/trainer/classifier.py) — single/multi label sequence classification, 1024-entry multi-label cap, label_names string-to-int resolution. Routes classifier / reranker / cross_encoder. Live loss kernels: - apply_ebft_loss (structured / strided) + attach_ebft_compute_loss (SFT) - apply_gdpo_loss (standard / length_normalized / margin) + attach_gdpo_compute_loss (DPO). Both attach hooks idempotent. Prompt-format wiring: - apply_reasoning_effort_prefix injects gpt-oss <|reasoning_effort|>{low,medium,high}<|/reasoning_effort|> header. - build_assistant_only_labels(train_on_eot=True) keeps EOT/EOS unmasked. Bugs surfaced + fixed during Wave 3 CPU smoke (regression guards in tests): - Distill collator did not pad pre-tokenised labels (variable-length crash) - Distill compute_loss device-mismatch when HF Trainer auto-promoted student to CUDA while teacher stayed on CPU. Tests: 7722 -> 7842 (+120 in test_v0532.py). 5 review agents run; every CRITICAL/HIGH/MEDIUM/LOW finding fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2ea26df5ea |
fix(v0.53.1): CI green — inspect click params directly instead of asserting on Rich --help output
The three CLI tests added in the previous commit (test_help_lists_measure_flag, test_save_format_help_lists_flag, test_torchao_help_lists_quant_config) assumed the literal option name (e.g. `--measure`, `--save-format`, `--quant-config`) would appear contiguously in CliRunner-captured output. On CI runners the terminal defaults to 80-col and Rich wraps long option names across lines, splitting the literal string. Fix: walk `typer.main.get_command(app).params` and collect every `opt` + `secondary_opt` into a set, then assert membership. The test now verifies what we actually care about (the option is registered) without depending on Rich's wrapping behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
725696b1da |
feat(v0.53.1): Quant Menu II + Export pipeline live
Lift six v0.53.0 deferred stubs from NotImplementedError to live wiring: - #82 autopilot pre-quantized base detection utils name regex over gptq/awq/aqlm/eetq/fp8/mxfp4 with word-boundary anchoring + HQQ Nbit extraction + config.json quantization_config probe (cwd-contained + symlink-rejected). decide_quantization() short-circuits the VRAM heuristic when prequantized is set. autopilot pipeline auto- applies so TheBloke/Llama-2-7B-Chat-GPTQ is recommended gptq instead of 4bit-on-top-of-quantized. - #142 merge_4bit + export_torchao live writers soup merge --save-format {fp16|4bit|4bit_forced}: single BNB-4bit merged checkpoint without the dequant->merge->requant cycle (fixes wrong-name llm_int8_skip_modules to bnb_4bit_skip_modules per code- review). soup export --format torchao --quant-config <yaml>: torchao .quantize_ + save_pretrained with per-scheme closed kwarg allowlist (Int4WeightOnly accepts {group_size, inner_k_tiles}, NVFP4 accepts nothing extra; dunder + unknown keys rejected per security-review H1). load_quant_config enforces yaml.safe_load + 256 KB cap + extension allowlist + cwd containment + S_ISLNK rejection. - #139 export_advanced_gguf via llama.cpp imatrix 3-stage pipeline: convert_hf_to_gguf.py -> optional imatrix -> quantize. argv-list subprocess (no shell), 30-min timeout, realpath- verified convert script stays inside llama_cpp_dir (security-review M5). _prepare_calibration_text accepts JSONL with text/prompt/content field aliases + raw text fallback; strips null bytes, collapses newlines, 8 KB per-line + 50 MB total cap (security-review M1); POSIX O_NOFOLLOW closes the TOCTOU window between dispatch-time check and open() (security-review M3). UD- prefix stripped before passing to llama-quantize. _safe_stderr Rich-escapes subprocess stderr before embedding in RuntimeError (security-review L4). - #109 soup deploy autopilot --measure Live Quant-Lobotomy scorecard: classifies each candidate quant OK / MINOR / MAJOR (thresholds 2% / 5% mirror v0.26.0 Part D). Results cached at ~/.soup/deploy_autopilot_cache.json (atomic write, 0o600 perms on POSIX, S_ISLNK rejection on BOTH load and save). pick_best soft-fallback now picks max-by-delta (was max-by-after) matching the v0.33.0 #54 design intent. _DEPLOY_MEASURE_BEFORE_GEN / _AFTER_FACTORY module-level hooks act as the stop-gap escape hatch until v0.46.1 ships first-party transformers / vLLM generator factories. - #70/#72 manual QA log scripted at tests/qa/v053_qa.md with exact reproduction recipes + acceptance criteria for the CUDA + llama.cpp smokes that can't run on the CI runners. Shared cleanup: - soup_cli/utils/paths.enforce_under_cwd_and_no_symlink consolidates the v0.33.0 #22 TOCTOU pattern previously copy-pasted in save_formats.py and gguf_quant.py (code-review HIGH fix). Reviews ran: python / code / security / tdd. Every CRITICAL / HIGH / MEDIUM / LOW finding fixed or documented. Test count: 7610 -> 7722 (+112 across 4 new files). Known limitations: live GPU + bitsandbytes / torchao smokes for the new merge / export paths remain pending (recipes in QA log); injected- generator escape hatch is non-public until v0.46.1; cache key truncates base_sha to 16 hex (1-in-2^32 collision floor). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
07e7214ed3 |
feat(v0.53.0): Quant Menu II — UD GGUFs + KV cache + NVFP4 + LF parity + save formats
Schema-only release. Live wiring deferred to v0.53.1 (mirrors v0.50.0 /
v0.51.0 / v0.52.0 stub-then-live pattern).
- Part A — Unsloth Dynamic 2.0 GGUF ladder (14 entries: UD-Q8_K_XL ... UD-IQ1_M)
+ validate_calibration_data_path shape validator.
- Part B — IQ (12) + Apple/ARM (10) GGUF flavours in utils/gguf_quant.py;
O(1) _LOWER_INDEX MappingProxyType for case-insensitive lookup.
- Part C — training.kv_cache_type: q8_0 | bf16 | f16 | fp8 (fp8 Hopper-only;
MLX rejected). requires_hopper reads from spec metadata (single source).
- Part D — fp8_attention (requires quantization_aware='fp8') + nvfp4 (Blackwell)
+ native unsloth_bnb_4bit bool flags with cross-validators.
- Part E — bnb_4bit_use_double_quant + llm_int8 (explicit 8bit assertion,
distinct from v0.41.0 load_in_8bit aliasing) + quantize_ref_model
(extends ref-task set with grpo/kto/ppo) + quantize_reward_model.
- Part F — soup merge --save-format {fp16, 4bit, 4bit_forced} + soup export
--format torchao with closed PTQ scheme allowlist (Int4WeightOnly,
Int8DynActInt4, Float8DynActFloat8, NVFP4 — case-sensitive PyTorch names).
Test count: 7453 → 7610 (+157 net new across 154 tests in test_v0530.py).
ruff check soup_cli/ tests/ — clean.
5 review agents ran (python / code / security / tdd / verification);
every CRITICAL / HIGH / MEDIUM / LOW finding fixed or documented:
- O(N) gguf walk → O(1) _LOWER_INDEX MappingProxyType
- ref_tasks extended with grpo + kto + ppo (silent-no-op footgun)
- _validate_v053_bool_fields no longer coerces None → False
- requires_hopper delegates to _KV_CACHE_METADATA spec
- fp8_attention validator order: quantization_aware before MLX
- validate_calibration_data_path + validate_quant_config_path docstrings
name the exact controls v0.53.1 CLI dispatch MUST add (TOCTOU contract)
- validate_torchao_scheme case-sensitivity documented at validator
- tautological `result == result` test replaced with allowlist invariant
- bool guards added on backend/modality/quantization across all Part D
validators
- exact 4096/4097 boundary tests for path shape validators
Docs updated: CLAUDE.md (test counts + utils list + changelog + test-table),
README.md (What's New replaced + 5 new dedicated sections), SECURITY.md
(support window + v0.53.0 hardening entry), CONTRIBUTING.md (test count
+ utils list + test-table row), .claude/plan.md (heading + boxes + banner —
gitignored, local only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
df7f49feda |
feat(v0.52.0): Modality II — TTS + Distillation + BitNet + EBFT-GDPO + MoE quant + reasoning_effort
7 schema-only Parts; live trainer / loss / export wiring deferred to v0.52.1
(mirrors v0.27.0 MII / v0.50.0 GRPO Plus / v0.51.0 hubs stub-then-live pattern).
- Part A: task='tts' + modality='audio_out' + 5 families (Orpheus/Sesame-CSM/
Llasa/Spark/Oute) + per-family emotion allowlist (Orpheus 8 / Oute 6) +
5 recipes (orpheus-tts-sft, sesame-csm-tts, llasa-tts, spark-tts, oute-tts).
- Part B: classifier / reranker / cross_encoder tasks + num_labels (bool-
before-int guard) + classifier_kind + label_names (dedup + null-byte + cap).
- Part C: task='distill' + teacher_model + distill_divergence (kl alias
canonicalises to forward_kl; Literal excludes alias) + distill_temperature
(math.isfinite + [0.05, 100.0] bounds).
- Part D: quantization='bitnet_1.58' (gated to non-MLX + text + task in
{sft, pretrain, dpo}) + Falcon-E BitNet recipe + soup export
--format bitnet/tq1_0 CLI stubs (yellow deferred-advisory panel, exit 0).
- Part E: EBFT (structured/strided + bounded ebft_temperature; SFT-only)
+ GDPO (standard/length_normalized/margin; DPO-family-only).
- Part F: moe_expert_quant (nf4/int8_rowwise) + train_router_only — both
require moe_lora=true (silent-no-op footgun rejection).
- Part G: reasoning_effort (low/medium/high) + train_on_eot — both gated to
the SFT-family task set (sft/pretrain/distill/classifier/reranker/
cross_encoder); rejected on DPO/GRPO/PPO/etc. with named offenders.
Review fixes (5 agents: python, security, code, tdd, verification-loop-manual):
- num_labels bool-before-int field_validator (security HIGH)
- reasoning_effort + train_on_eot SoupConfig task-gate (code HIGH)
- _validate_classifier_compat lazy-import early-return (code HIGH)
- Oute emotion allowlist via data-driven _FAMILY_EMOTIONS (python+code MED)
- _MAX_LEN -> _MAX_REASONING_EFFORT_LEN (python MED)
- validate_reasoning_effort wired via field_validator (security MED)
- distill_divergence Literal excludes "kl" alias (code MED)
- DIVERGENCES derived from _DIVERGENCE_ALIASES drift guard (python LOW)
- is_bitnet_model comment fixed to match impl (python LOW)
- sister-fn bool guards on every compat helper (python+security MED)
- TDD coverage gaps closed: EBFT variant oversize, GDPO full rejection matrix,
ebft_temperature explicit-exc table, TTS compat input guards, recipe
model-id null/whitespace, full reasoning_effort task-matrix.
Drift fixes:
- tests/test_onnx_tensorrt_export.py + tests/test_awq_gptq_export.py
SUPPORTED_FORMATS count bumped 5 -> 7 (bitnet + tq1_0 stubs).
- tests/test_recipes.py catalog_size assertion 106 -> 112 (5 TTS + Falcon-E).
Test count: 7184 -> 7456 (+272). 230 new tests in tests/test_v0520.py;
remainder from drift-fix parametrize expansions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
1e1abacb44 |
feat(catalog): v0.51.0 — Model Catalog Expansion + Alternative Hubs
26 new ready-made recipes (catalog 80 → 106) covering 25 model families: GPT-OSS 20B/120B, GLM 4.6/5, Kimi K2 / K2-Thinking GRPO, MiniMax-M2, QwQ-32B GRPO, QVQ-72B, Granite 4, Liquid LFM2, Cogito v2, Mistral Small 3 / Medium 3.5, Magistral / Devstral / Ministral, MedGemma, EmbeddingGemma, LLaVA-Next, InternVL 3.5, Voxtral, Baichuan 2, Qwen-Image, DeepSeek-OCR, Paddle-OCR-VL. Part D: MULTIPACK_ARCHITECTURES extended 18 → 38 (Granite, GLM, Kimi, MiniMax, QwQ, QVQ, GPT-OSS, Magistral, Devstral, Ministral, MedGemma, LFM2, Cogito, Hunyuan, Ernie, Yi, Baichuan, ChatGLM). Part E: alternative model hubs. New soup_cli/utils/hubs.py with closed allowlist (hf/modelscope/modelers), SSRF-hardened endpoint validators mirroring v0.29.0 HF_ENDPOINT policy (scheme allowlist, loopback-only HTTP, RFC1918/link-local rejection, control-char/CRLF rejection, IPv6-mapped private rejection). TrainingConfig.hub Literal field with case-insensitive _normalize_hub field_validator; SoupConfig _validate_hub_supported rejects backend=mlx + hub != hf. Live downloader / uploader wiring deferred to v0.51.1 (matches the v0.27.0 MII / v0.37.0 multipack stub-then-live pattern). +455 tests (6729 → 7184), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
33c60b4c1f |
feat(grpo): v0.50.0 — GRPO Plus (unsloth + axolotl RL parity)
22 features across 5 internal Parts shipped as schema-only — closed allowlists, Pydantic validators, NotImplementedError stubs for live wiring deferred to v0.50.1 (mirrors v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro / v0.45.0 plugins / v0.48.0 curriculum / v0.49.0 LongLoRA stub-then-live pattern). Part A — 7 GRPO objective variants (gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard) with `validate_grpo_variant` + frozen `GRPOVariantSpec` metadata + `MappingProxyType`-wrapped registry. `validate_grpo_delta` is bool-first / math.isfinite / (0, 1] bounded. `apply_variant_loss` raises NotImplementedError with v0.50.1 marker for the 6 new variants and is a no-op for standard. Part B — long_context_grpo + vllm_sleep_mode schema gates with compat validators (null-byte rejection on task + backend, bool guard on use_ring_attention). vllm_sleep_mode requires task='grpo' AND a transformers/unsloth backend (code-review HIGH fix — sleep is a between-rollouts feature). Part C — 4 multi-turn rollout backends (art / ruler / nemo_gym / openenv) with closed allowlist + per-entry required_package mapping. Part D — 7 stability/efficiency knobs (ref_model_ema_alpha, replay_buffer_size, async_grpo_prefetch, tis_threshold, mask_truncated_completions, defer_rerolling, skip_zero_advantage, off_policy_mask_threshold). Every numeric field rejects bool via a shared `_reject_bool_on_grpo_numerics` field_validator (tdd-guide HIGH fix — Pydantic v2 coerces True→1 by default). The `mask_truncated_completions` + `tis_threshold` pairing is enforced by a cross-validator (matches v0.32.0 spike-recovery+watchdog policy). Part E — top-level task='prm' Literal addition (Process Reward Model / stepwise-supervised, paired with data.format='prm' from v0.42.0) + `vision_grpo: bool` flag for VLM-RL on Qwen2-VL / Pixtral / InternVL. Compat helpers gate task / modality / backend. Review-round fixes applied (5 sequential reviews per CLAUDE.md): - python-review: list_variants annotation, frozenset[str] params, Optional[str] → str | None, module-level math import, D401 imperative docstrings, dropped *args/**kwargs on stubs. - code-review: grpo_fp16 added to GRPO-only task-gate; vllm_sleep_mode requires task='grpo'. - security-review: explicit field_validator for grpo_delta NaN/Inf rejection (Pydantic le=1.0 incidentally rejects NaN, made explicit); null-byte rejection on backend/task in grpo_long_context helpers; use_ring_attention bool guard. - tdd-guide: bool-rejecting field_validator on all Part D numeric fields + grpo_delta; missing bool-rejection tests added on validate_grpo_variant / validate_rollout_backend; null-byte test on validate_vllm_sleep_mode_compat; required_rollout_package rejection path; RolloutBackendSpec.live_wired; PPO+vision_grpo round-trip; _DEFERRED_LIVE invariant. Test count: 6490 → 6729 (+239 across 5 new test files). Notes for future maintainers: - v0.50.0 has zero new CLI commands and zero new trainer wirings; every step 6d/6e is intentionally n/a. Step 6 smoke runs schema happy + every documented cross-validator rejection. - All `task='grpo'` gates use `if self.task != 'grpo'` literal comparisons; do NOT switch to a set membership check until Part D knobs are wired into PPO/preference trainers in v0.50.x. - Multi-modal Vision RL does not yet verify the base model is actually a VLM — upstream trainer surfaces the error loudly when it fails to load the vision tower. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
161c81ee7f |
feat(long-context): v0.49.0 — YaRN, Dynamic NTK, LongLoRA S², Llama 3.1 NTK
- Part A: YaRN RoPE scaling — math kernels (yarn_find_correction_dim /
yarn_find_correction_range / yarn_linear_ramp_mask / yarn_get_mscale) +
4 yarn_* schema fields + cross-validator rejecting yarn fields outside
rope_scaling_type=yarn. Pure-Python implementations of the upstream YaRN
paper §3.4/§3.5 with bool/NaN/Inf rejection on every numeric input.
- Part B: Dynamic NTK — existing path verified, explicit test coverage.
- Part C: LongLoRA S² shifted-sparse attention (schema-only) — new
soup_cli/utils/longlora.py with is_llama_model (word-boundary regex
mirroring v0.39.0 is_gemma4_model policy) + validate_longlora_compat.
TrainingConfig.use_longlora + SoupConfig._validate_longlora_compat.
Live LlamaAttention.forward override deferred to v0.49.1 (stub-then-live
pattern, mirrors v0.27.0 MII / v0.37.0 multipack).
- Part D: Llama 3.1 NTK-aware (full impl) — scale_inv_freq_llama3
smooth-transition kernel + detect_llama3_rope_in_config HF-config probe
+ "llama3" added to rope_scaling_type Literal. LLAMA3_DEFAULT_*
constants per Unsloth models/llama.py:1853.
Public-boundary input validation on get_rope_scaling_config (bool/NaN/Inf
rejection on target_length / original_length / yarn_factor) per security
review — prevents direct callers from emitting {factor: NaN} into HF
model configs when bypassing the Pydantic schema.
Reviews: code-reviewer (2 HIGH + 1 MEDIUM + 1 LOW), security-reviewer
(1 MEDIUM + 2 LOW), python-reviewer (4 findings), tdd-guide (8 coverage
gaps) — all findings fixed. verification-loop done as manual equivalent
(version + --help + happy/failure YAML smoke).
+80 net new tests (6410 → 6490). Full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
f789953d46 |
feat(data): Data Mixing Optimizer + v0.48.0 release (v0.48.0 Part B, BETA)
BETA. New `soup data mix --optimize --budget 1h --datasets a,b,c` runs N short proxy-training runs over candidate mixture weights and writes a canonical recipe YAML you can splice into `soup.yaml` under `data.interleave`. Per-candidate proxy failures are isolated (DEBUG-log + sentinel `_MAX_LOSS` + `continue`); `KeyboardInterrupt` / `SystemExit` re-raised; budget cap surfaces `MixOptimizationReport.partial=True`. `soup data mix --apply <recipe.yaml>` re-loads + prints the recipe's canonical interleave block. Both modes enforce `is_under_cwd` containment + TOCTOU symlink rejection (`os.lstat + S_ISLNK`) + 256 KB file cap; YAML key injection defended at the renderer (rejects newlines / null bytes / oversize dataset paths). Synthetic offline proxy ships in v0.48.0; live `soup train` proxy + scikit-optimize backend wiring deferred to v0.48.1 via `OptimizerProtocol` ducktype (default fallback: deterministic Dirichlet sampler). Review fixes: - `validate_datasets` early `len(raw) < 2` check (code-review MEDIUM) — prevents the less-actionable error after realpath resolution. - `run_mix_optimizer` proxy exceptions now isolated per-candidate (code-review MEDIUM) — first-cut raised RuntimeError on the first proxy failure, breaking the documented `partial=True` contract. - `load_mix_recipe` `os.lstat` wrapped in `try/except OSError` (security HIGH) — closes a TOCTOU race where path disappearance between `lexists` and `lstat` would raise an unhandled OSError. Release bundle (v0.48.0): - version bump → 0.48.0 in pyproject.toml + soup_cli/__init__.py - README.md: replaced "What's New" + 2 new dedicated `##` sections - SECURITY.md: supported-versions window + per-version notes for v0.48.0 - CONTRIBUTING.md: test counts (6242 → 6410) + 2 new test-table rows +94 tests. Net release total: 6242 → 6410 (+168 tests, +2 test files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
fe9fe06b68 |
feat(training): Curriculum-Aware Trainer — dynamic bucket re-weighting (v0.48.0 Part A, BETA)
BETA. Adds `training.curriculum_dynamic: true` schema flag with online uncertainty estimation: every N steps, aggregate per-sample loss + grad-norm into per-bucket softmax weights, water-filled to enforce a minimum `curriculum_dynamic_floor`. DDP/grad-accum safety via `validate_distributed_curriculum` cross-validator that rejects un-coordinated multi-rank runs upfront — the well-known footgun where divergent per-rank stats silently desynchronise the sampler. New `soup runs curriculum-curve <run_id>` visualiser with TOCTOU (`os.lstat + S_ISLNK`) + 50 MB file-size cap + 100k-line streaming cap on the history file. Schema gated to sft/pretrain on transformers backend; mlx + non-SFT rejected with distinct messages. Live HF Trainer callback wiring deferred to v0.48.1 (stub-then-live pattern; mirrors v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro). Review fixes: - water-fill design fix (code-review HIGH): removed trailing renorm that could push elements below `floor` when accumulated float error left sum slightly > 1.0. Softmax already sums to 1.0, so water-fill output also sums to 1.0 (drift bounded by nb*eps). - DoS caps on `render_curve` + `parse_history_jsonl` (`_MAX_HISTORY_ROWS=100_000`) — without these an attacker-controlled JSONL with 10M rows would OOM the process. - `curriculum-curve` CLI: symlink rejection + 50 MB + 100k-line caps, null-byte rejection on tracker-supplied `output_dir`. +74 tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f64cee569e |
feat(v0.47.0): Data Forge — synthetic data pipeline + data quality moat
Part A — Synthetic Data Forge (utils/data_forge.py + commands/data_forge.py):
- soup data forge --docs <dir> --task sft|preference|tool with full provenance manifest
- ForgePlan / ProvenanceRecord / ForgeRow frozen dataclasses
- chunk_document + score_uncertainty pure-function kernel
- discover_documents (cwd-contained, symlink-rejecting, .txt/.md/.json/.jsonl allowlist)
- synthesise_forge_rows with judge-exception swallow at DEBUG
- Atomic JSONL + provenance writes via staged-tempfile + os.replace
Part B — Data Quality Moat (utils/data_score.py + commands/data_score.py):
- soup data score / decontaminate / toxicity / langdetect / pii / educational
- ReDoS-hardened PII regexes (phone + credit-card rewritten, 50 KB pre-cap)
- Containment-based n-gram decontamination (docstring corrected from "Jaccard")
- 6-language stopword heuristic for langdetect
- compute_scorecard with per-row DEBUG logging on swallowed errors
Security review fixes applied:
- math.isfinite guard on _validate_float_unit (NaN/Inf rejected before bounds)
- ReDoS: phone regex flattened (no nested optional quantifiers); credit_card
rewritten from {13,19}-loop to anchored 4-4-4-N; 50 KB pre-cap before finditer
- is_under_cwd moved inside discover_documents (no longer relies on caller)
- os.lstat + S_ISLNK rejection on every write target + tempfile staging
- _require_str rejects null bytes (consistency with data_forge._validate_str)
- compute_scorecard try/except blocks log at DEBUG (no silent swallow)
- decontaminate_texts: Optional[...] = None (no more type: ignore)
- _read_rows / _write_rows have full type annotations
- ngram_overlap_ratio docstring renamed to "containment ratio"
- import math + import tempfile moved to module top (lazy-import policy
applies to heavy ML deps only, not stdlib)
- Duplicate discover_documents call in CLI collapsed (TOCTOU window closed)
Live judge providers, [data-pro] extras (Llama-Guard / FineWeb-Edu / Presidio /
fastText / langdetect), and operator-supplied benchmark corpora are
stub-then-live and ship in v0.47.1 (mirrors v0.27.0 MII / v0.37.0 multipack
precedent).
Tests: 6126 -> 6242 (+116 net new). All v0.47.0 + full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
986a4c00e3 |
feat(agent): Agent Forge — OpenAPI/MCP/GraphQL spec → tool-calling SFT dataset (v0.46.0 Part B)
Bumps to v0.46.0 and ships the Agent Forge: parse OpenAPI 3.x, MCP server
manifests, or GraphQL introspection JSON straight into a tool-calling SFT
dataset where each row is `{messages: [user, assistant_with_tool_call],
tool, source_endpoint}`. No more hand-rolled jsonl scaffolding for
function-calling fine-tunes.
* soup_cli/utils/agent_forge.py — `Endpoint` / `SynthRow` / `SpecReport`
frozen dataclasses. `parse_openapi` / `parse_mcp` / `parse_graphql`
parsers leave `$ref` strings opaque (no external resolution — defends
against file-read SSRF). All synthesised path strings routed through
`_validate_path` (rejects newline-in-name across all three parsers).
`load_spec_file`: `is_under_cwd` + `os.lstat + S_ISLNK` BEFORE realpath
(corrects v0.46.0 first-cut ordering caught by security review) + 5MiB
cap + yaml.safe_load only. `write_dataset` atomic via mkstemp +
os.replace (mid-stream TypeError never leaves partial file; mirrors
v0.43.0 Part D `copy_bundle_to` policy) + symlink rejection at target.
Caps: `_MAX_ENDPOINTS=10_000`, `_MAX_SPEC_BYTES=5MiB`,
`_MAX_ROWS_PER_ENDPOINT=32`.
* soup_cli/commands/agent.py — `soup agent synth/train/eval` Typer
subcommands. `synth` table cells pass through `rich.markup.escape`.
`train` rejects NUL/newline/oversize in `--base` and `--output-dir`
BEFORE embedding into rendered YAML recipe (CRITICAL security fix —
defends against YAML key injection where `--base $'evil\ntraining:
{ epochs: 9999 }'` would smuggle in injected training keys). `eval`
enforces predictions `is_under_cwd` + symlink rejection +
`_MAX_PRED_LINES=1_000_000` DoS cap.
* soup_cli/cli.py — registers `agent` Typer group; help string uses
ASCII-safe `->` (`test_help_output_is_ascii_safe` regression test caught
a Unicode `→` on first try).
* tests/test_v0460_part_b.py — 71 tests covering every parser kind,
failure modes (cycle / cap / null-byte / oversize / outside-cwd /
symlink), atomic-write partial-failure invariant, every CLI surface.
* Docs: README ## What's New replaced + dedicated `## Deploy Autopilot`
and `## Agent Forge` sections added; SECURITY.md supported-window
shifted (v0.46→full, v0.41→drop) + v0.46.0 fix-notes entry;
CONTRIBUTING.md test counts 165→167 / 5989→6126.
Test suite: 5989 → 6126 (+137 net new) green on Windows.
Known limitations (live runtime deferred to v0.46.1):
- Quant-Lobotomy auto-measure for deploy autopilot
- RLVR `code_exec` sandbox scoring in `agent eval`
- In-process `soup train` re-entry in `agent train` (Typer commands aren't
safe to re-enter — matches v0.44.0 `soup quantize` design)
- ExecuTorch packaging for iphone-16 / pixel-9 (lands in v0.54.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
479931282c |
feat(deploy): On-Device Deploy Autopilot — 10-profile catalog + recipe/script writer (v0.46.0 Part A)
Closes the v0.45.0 Known Limitations gap that flagged the 15-entry external integrations catalog as descriptive-only. This release ships the live autopilot CLI that picks PEFT + quantisation + speculative-decoding for the target hardware and emits a ready-to-train recipe. * soup_cli/utils/deploy_autopilot.py — `DeployProfile` frozen dataclass + 10-entry MappingProxyType-wrapped catalog: mac-m3 / mac-m4-pro / rtx-3060-12gb / rtx-4090-24gb / iphone-16 / pixel-9 / ollama-local / lm-studio / runpod-a100 / hf-jobs-h100. `_make` factory rejects bool recommended_max_length, non-kebab-case names, runtime/quant/peft outside closed allowlists. `render_recipe_yaml` validates `base` (≤200 chars, no NUL/newline) + `output_dir` (≤4096 chars). `render_deploy_script` uses `shlex.quote` on model_path. `write_recipe` / `write_deploy_script` enforce `is_under_cwd` + `os.lstat + S_ISLNK` TOCTOU rejection at the write target (matches v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E policy). * soup_cli/commands/deploy.py — new `autopilot` Typer subcommand with `--target`, `--base`, `--recipe-out`, `--script-out`, `--list`. Every Rich-rendered profile field passes through `rich.markup.escape`. * tests/test_v0460_part_a.py — 76 tests covering catalog immutability, 10-profile presence, case-insensitive resolution, every `_make` failure mode, render-* validation matrix, write-* path-containment + symlink rejection (POSIX-only), CLI smoke (list / writes / outside-cwd reject). Live Quant-Lobotomy auto-measure (against the v0.26.0 Checker) is deferred to v0.46.1 — this release writes the canonical combo per profile, the runtime measurement step lands in the patch release. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
74301843a2 |
feat(v0.45.0): Plugin System & Ecosystem Wins — 5 Parts, +169 tests
Adds a public plugin/hook system plus the schema scaffolding for 20+ ecosystem
integrations. Live trainer-callbacks, Anthropic /v1/messages route, server-tool
HTTP endpoints, and the recipe runner ship in v0.45.1 (matches v0.27.0 MII /
v0.37.0 multipack / v0.41.0 LLaMA Pro stub-then-live pattern).
Part A — Plugin / hook system
* New soup_cli/plugins/ package: BasePlugin Protocol, PluginSpec frozen
dataclass, register_plugin / discover_hooks / enable_plugin /
disable_plugin / load_plugins. Kebab-case name regex, semver-ish version,
null-byte rejection on every string. Idempotency check covers
(version, plugin object, templates, model_groups, description) — review-fix
added description after first-cut omitted it. Per-list caps on templates
and model_groups (32 entries, 128-char per name).
* New soup plugins list/install/enable/disable Typer CLI; all user-controlled
output passes through rich.markup.escape.
Part B — API extensions (schema-only)
* utils/anthropic_messages.py: to_anthropic / from_anthropic /
validate_anthropic_payload converters. Multiple system messages join with
\n\n; tool role with structured (list) content concatenated into single
tool_result text block (review-fix MEDIUM — first-cut silently dropped).
max_tokens cap 16384, temperature [0.0, 2.0], bool rejection on numerics.
* utils/server_tools.py: closed {python, bash, web_search} allowlist,
WebSearchConfig with domain allowlist + leading-dot subdomain pattern,
rate_limit [1, 600]. is_domain_allowed strips :port suffix and rejects
IPv6 literals (review-fix MEDIUM).
* utils/ngram_spec.py: NgramSpecConfig validators with bounded n / draft
tokens / prompt-lookup-max; bool rejection on every numeric field.
Part C — External integrations catalog
* utils/integrations.py: 15-entry MappingProxyType catalog of ecosystem
targets (lm-studio, comfyui, ollama, claude-code, cursor, continue, ...).
Part D — Advanced trainer-plugin allowlist
* utils/trainer_plugins.py: 6-entry allowlist (grokfast, spectrum,
llmcompressor, sonicmoe, cce_plugin, math_verify) + validate_trainer_
plugin_list (Sequence[str], dedup, _MAX_PLUGINS_PER_RUN=8).
Part E — Data Recipe DAG
* utils/recipe_dag.py: closed NODE_KINDS frozenset, Kahn's topological
sort via collections.deque (review-fix HIGH — first-cut had O(N^2 log N)
queue.sort() inside the BFS body), cycle / self-loop / dangling-edge
rejection, _MAX_NODES=256 / _MAX_EDGES=1024 / _MAX_FILE_BYTES=1MiB.
load_recipe_yaml enforces is_under_cwd containment AND os.lstat + S_ISLNK
symlink rejection (review-fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22
/ v0.43.0 Part C / v0.44.0 Part B policy).
* New soup data recipe <path> CLI validates topology and prints planned
topo order; live runner deferred to v0.45.1.
Reviews: python-review, security-review, code-review, tdd-guide all run;
verification-loop replaced by manual smoke (CLI happy + failure paths
exercised on real fixtures).
Test count: 5820 -> 5989 (+169). Test files: 164 -> 165. Ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
c4ac3da695 |
feat(v0.44.0): Live Dashboard & UX - 21 features, +192 tests
Part A - Live monitoring: soup monitor (nvidia-smi panel), EMA + p95/p99 tail-latency stats, SSE training-stream schema, phone-visible URL + ASCII-QR helper, llama-server timings parser + KV-cache bar, thread-safe ToolOutputsBuffer + ToolCallTimer. Part B - UX fixes: GracefulSaveHandler (first SIGINT saves, second stops), .checkpoint_now trigger file (cwd-contained, symlink-rejected), desktop / .command / .cmd shortcut builders, onboarding-wizard YAML renderer. Part C - UI tabs: drop-in soup_cli/ui/plugins/*.py registry with kebab-case name allowlist + 32-tab cap, API_HOST / API_PORT / API_KEY + GRADIO_HOST / GRADIO_PORT env knobs. Part D - Standalone CLIs: soup fetch (bundled examples + configs + deepspeed_configs catalog), soup quantize (ergonomic alias), soup merge-sharded-fsdp-weights, soup delinearize-llama4 (planners; live runtime in v0.44.1), soup llama <sub> (closed-allowlist proxy with filtered child env that drops HF_TOKEN / OPENAI_API_KEY / ANTHROPIC_API_KEY), soup_cli.utils.sweep_config (separate sweep.yaml loader), reasoning_parser allowlist for soup serve. Security review fixes: fetch symlink-at-target rejection + bundled-source commonpath check, write_trigger symlink rejection (TOCTOU), llama child-env secret allowlist, onboarding output cwd-containment, qr token moved from URL fragment to query string (so server actually sees it), sweep-config scalar allowlist + MappingProxyType[Tuple] immutability. Code/Python review fixes: detect_apple_silicon clean rewrite (was buggy parser-priority ternary), all frozen-dataclass List fields -> Tuple, ToolOutputsBuffer -> collections.deque(maxlen=1000), os.path.realpath over abspath, IPv6 host auto-bracketing per RFC 3986, frozenset over mutable set, type hints on __exit__/_make_proxy. Test count: 5628 -> 5820 (+192). Lint clean. Help output ASCII-safe (em-dash check enforced by test_cli_subprocess). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
82d5693b75 |
feat(eval): Tracker & Eval Pro — 18 features (v0.43.0)
Closes the observability gap with all three competitors in one release.
Part A — Trackers
* --tracker flag (mlflow/swanlab/trackio) on soup train, mutually
exclusive with --wandb/--tensorboard. Closed allowlist via
MappingProxyType. Live integrations rely on HF Trainer's report_to.
* SOUP_TELEMETRY=1 opt-IN env var; build_telemetry_payload schema is
closed-key (no model names / dataset paths / config contents). Live
PostHog network code deferred to v0.43.1.
Part B — Eval metrics
* Pure-Python BLEU + ROUGE-1/2/L + effective_tokens_per_second.
* KL-divergence calibration framework with OK/MINOR/MAJOR thresholds.
* Model Arena Elo tournament (256-model cap, MappingProxyType view,
Rich-markup metacharacter rejection on names).
* ceval / cmmlu / aider_polyglot benchmark allowlist (live Aider
runner deferred to v0.43.1).
Part C — Profiling
* memory_snapshot_context (narrow RuntimeError catch — review fix
prevents generator-already-executing on user-body RuntimeError).
* detect_anomaly_context, nccl_bandwidth_check (h100/a100/v100/rtx
reference table; live measurement CLI surface deferred).
* write_vscode_launch with TOCTOU symlink rejection at the target
path regardless of force=True.
Part D — Demo bundles
* `soup data demo` lists / copies 4 bundled JSONL fixtures
(alpaca / sharegpt / dpo / grpo) with staged-tempfile atomic
rename + 50 MB cap + symlink rejection on the staging path.
Tests: 5389 -> 5628 (+239). Ruff clean. Five sequential review waves
(python / code / security / tdd / smoke) ran; HIGH/MEDIUM/LOW findings
all fixed including: tracker name shadow in train.py, _lcs_length DP
double-buffer bug, BLEU geo-mean policy, base_dir absolute/.. escape,
demo_bundles tmp symlink TOCTOU, vscode launch symlink TOCTOU.
Note (Windows CI): line-ending warnings (LF -> CRLF) on commit are
expected; `.gitattributes` policy is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
05093ebfdb |
feat(data): Data Pipeline Pro — 18 features, axolotl + LF parity (v0.42.0)
Schema-first surface for the data pipeline gap with Axolotl + LlamaFactory. Ships in one release: 5 new formats (prm, pre_tokenized, input_output, video, multimodal), remote URI allowlist (s3/gs/gcs/az/abfs/abfss/oci) + streaming + sharding, AOT preprocess cache + `soup data preprocess` CLI, multi-dataset interleave (concat/under/over/probs) + 8 advanced masking fields, vocab expansion (add_new_tokens / new_special_tokens / resize_vocab) + custom prompt_strategy, and document ingestion (`soup data ingest` for PDF/DOCX/MD/TXT). Live wiring for fsspec backends, AOT tokenize loop, custom prompt-strategy runtime, and PRM trainer integration is deferred to v0.42.1+ (stub-then-live pattern from v0.27.0 / v0.37.0 / v0.41.0). Schema gates fire at config load so misconfiguration fails fast. Security: full v0.42.0 hardening matrix — `_REMOTE_SCHEMES` MappingProxyType allowlist; bucket regex 1-63 chars per S3/GCS spec; userinfo / fragment / query-string rejection on remote URIs (query-string forwarded to fsspec is SSRF-adjacent); null-byte + length caps on every string-shaped input; bool-rejected-before-int on every numeric input; frozen InterleaveSpec dataclass; 10k caps on add_new_tokens; `is_under_cwd` containment on video_dir + tokenized_path schema fields and on preprocess --config / both ingest paths; `os.lstat + S_ISLNK` symlink rejection on ingest input; PRM converter type-checks completions (str) + labels (bool, not int); video field null-byte + 2KB cap; field-name threading on image-pixels validator so error messages name the actual field. 5242 tests → 5389 (+147 net). 11 review findings addressed across python-review / code-review / security-review / tdd-guide (CRITICAL→LOW). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1491025a36 |
feat(trainer): Optimizer & PEFT Zoo (v0.41.0)
Optimizer Zoo (Part A): closed-allowlist SUPPORTED_OPTIMIZERS adds 14 new
entries — BAdam, APOLLO (apollo_adamw), Adam-mini, lomo / adalomo,
grokadamw, schedule_free_adamw / schedule_free_sgd, muon, dion,
came_pytorch, ao_adamw_{fp8,4bit,8bit}. Validates name + lower-cases
deterministically; rejects non-string / empty / null-byte / >64-char.
_OPTIMIZER_PACKAGES wrapped in MappingProxyType (matches v0.36.0 _REGISTRY).
Per-module LR (Part B): training.lr_groups accepts list-of-pairs /
list-of-dicts / {pattern: lr} mapping; canonical [{pattern, lr}, ...]
storage. Capped at MAX_LR_GROUPS=32; per-pattern non-empty string
≤256 chars + null-byte rejection + re.compile + best-effort ReDoS probe;
per-LR (0.0, 1.0] + math.isfinite (rejects NaN AND ±inf) + bool rejection
(matches v0.30.0 Candidate policy); duplicates rejected. lr_groups_from_schema
bridges canonical schema shape into runtime List[LrGroup] for
build_optimizer_param_groups (first-match-wins routing). LrGroup is
@dataclass(frozen=True). PyYAML scientific-notation (1e-4) parses as
string in YAML 1.1; _validate_lr coerces str → float so soup.yaml
round-trips work. base_lr rejects bool / non-positive (defence-in-depth).
PEFT methods (Part C): LoraConfig.init_strategy="loftq" + loftq_iter
∈ [1, 10] + loftq_bits ∈ {2, 4, 8}; cross-validator rejects loftq +
use_dora / use_vera. utils/loftq_init.py exposes validators +
build_loftq_config (lazy peft.LoftQConfig with actionable ImportError
hint). LLaMA Pro: TrainingConfig.expand_layers ∈ [1, 64] +
freeze_trainable_layers (signed, |x| ≤ 1000); cross-validator requires
the pair (LLaMA Pro freezes original layers and trains only new blocks).
field_validator(mode="before") on both rejects bool BEFORE Pydantic ge/le
silently coerces True → 1. expand_model_blocks raises NotImplementedError
with v0.41.1 marker — schema-only release (mirrors v0.27.0 / v0.37.0
stub-then-live pattern). utils/block_expansion._count_layers uses
hasattr(__len__) instead of try/except TypeError so legitimate __len__
bugs surface loudly. use_mod boolean for Mixture-of-Depths (schema only —
live patch deferred to v0.41.1).
Friendly aliases: load_in_8bit / load_in_16bit (Optional[bool]) for
LlamaFactory / Axolotl users. is True policy on both — explicit False is
"no preference", not "off"; mutually-exclusive both-True rejected; alias
combined with explicit Quant Menu format raises rather than silently
overriding. Alias-driven quantization rewrite via direct
self.quantization = ... (Pydantic v2 BaseModel non-frozen path), NOT
object.__setattr__ — code review caught that the latter would silently
bypass any future field_validator on quantization.
Five review agents (4 in parallel + manual smoke for verification-loop):
all CRITICAL/HIGH/MEDIUM/LOW findings fixed. Local smoke caught a real
bug — PyYAML parsing 1e-4 as string — fixed pre-commit with 2 added tests.
5242 tests pass (+120 net new); ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
5e0872b9ea |
feat(trainer): ReLoRA + surgical PEFT non-SFT (v0.40.6 #67)
Extends the v0.39.0 ReLoRA callback (Part B) and surgical PEFT patches (Part D — Gemma4 ClippableLinear swap + 3-D fused-MoE expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, RewardModel, Pretrain, Embedding, BCO). - New shared helper soup_cli/utils/peft_wiring.py exposes apply_pre_lora_patches, apply_post_lora_patches, attach_relora_callback. - SFT migrated to the same helpers in the same release (centralisation invariant; no drift between SFT and non-SFT wiring). - SoupConfig._validate_relora_supported_tasks: task != "sft" rejection removed; MLX backend still rejected with distinct message. Review fixes: - attach_relora_callback uses `if relora_steps is None:` (project policy) so a schema-bypassing relora_steps=0 surfaces as a loud ReLoRAPolicy ValueError rather than a silent skip. - Direct attribute access on tcfg.relora_warmup_ratio / _reset_optimizer / _prune_ratio (Pydantic schema guarantees them); no getattr defaults. - 11 behavioural helper tests (Gemma4 happy path + exception swallow, post-LoRA strip happy + exception swallow, ReLoRA policy field forwarding, schema-bypass loud-fail). - Schema-gate matrix covers `task='preference'` dispatcher. Tests: 5061 -> 5122 (+61). Closes #67. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
697cc8dad7 |
feat(trainer): Quant Menu non-SFT multi-trainer wiring (v0.40.5 #66)
Extends the v0.38.0 train-time quantization menu (gptq / awq / hqq:Nbit /
aqlm / eetq / mxfp4 / fp8) from SFT-only to all 11 transformer-backend
trainers (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel /
Pretrain / Embedding / BCO). Closes the v0.38.0 known gap.
Schema gate: SoupConfig._validate_quant_menu_supported_tasks removes the
`task != "sft"` rejection branch. MLX backend rejection retained with
distinct message; modality=text gate retained (vision/audio Quant Menu
deferred — modality-specific kwargs not yet threaded through the unified
loader).
Trainer wiring (11 sites): each non-SFT _setup_transformers replaces the
inline BitsAndBytesConfig(load_in_4bit=True, ...) block with a call to
build_quantization_config_for_loader(tcfg=tcfg, base=cfg.base, console=console)
— mirrors sft.py:420-440 exactly. kbit-prep tuple widened from
("4bit", "8bit") to ("4bit", "8bit", "mxfp4"). BitsAndBytesConfig import
removed from each non-SFT trainer.
Review fix — PPO reward model: _load_reward_model gains optional tcfg
kwarg; when supplied, the reward checkpoint loads with the same Quant
Menu config as the policy. Both PPO call sites forward tcfg=tcfg.
Defends against silent fp16 OOM on a GPTQ/AWQ/HQQ policy run.
Review fix — defence-in-depth: new TrainingConfig.reward_model field
validator rejects null bytes and caps length at 512 chars (matches
cfg.base policy). The Quant Menu loader's per-call null-byte check
in _check_local_marker remains as the runtime backstop.
Tests: +131 net new (4930 -> 5061). New tests/test_v0405_part_a.py
parametrizes 11 tasks x 7 quant formats; covers MLX rejection per task,
quantization_aware x Quant Menu cross-validator regression for non-SFT,
source-level invariants (no inline BNB literal, kbit tuple regex),
and a live mock-based dispatch test for _load_reward_model proving the
Quant Menu path is reachable when tcfg is supplied and skipped when
tcfg=None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
c85a1017b5 |
fix(v0.40.4): strip ANSI + route correctly in TestCommandFlagsExist
CI failure root cause: Rich line-wraps `--trust-remote-code` with ANSI colour escapes between `-`, `-trust`, `-remote-code` on narrow CI terminals (mirrors v0.40.3 ANSI fix). Substring assertion missed because the ANSI escapes were embedded mid-flag. Also: original test used `[cmd, "--help"]` then fell back to `["data", cmd, "--help"]`. For diff/export/merge/infer the first invocation worked (top-level commands) but the substring miss triggered the fallback into `data` subcommand, which then errored "No such command 'X'" — masking the real ANSI issue. Switched to explicit per-command argv lists. Adds the `_strip_ansi` helper from tests/test_trust_remote_code.py and routes `data generate` directly via `["data", "generate", "--help"]`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |