mirror of https://github.com/razor-ai/soup.git
399 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Yixuan Xu
|
4c2a578ac0
|
Guard diagnose gate on distributed worker ranks (#169)
Co-authored-by: mzl2233 <mzl2233@users.noreply.github.com> |
|
Alpamys
|
7d81496c69 |
test(diagnose): strip ANSI before help-output substring asserts (v0.56.0 CI fix)
Rich's CliRunner output on CI carries ANSI escape codes that split long option names like `--badge` and `--diagnose-gate` across colour-reset boundaries (`-\x1b[0m\x1b[1;36m-badge`), breaking naive `"--badge" in result.output` substring checks. Same fix pattern as v0.55.0 CI hotfix. Failures: tests/test_v0560.py::TestCli::test_diagnose_help and TestTrainDiagnoseGate::test_help_lists_flag on all 9 CI matrix cells. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
72189baba7 |
feat(diagnose): soup diagnose — post-training model report card (v0.56.0)
Six failure-mode probes (forgetting / refusal / format / mode_collapse / memorization / contamination) + FailureReport frozen dataclass + SVG badge + soup train --diagnose-gate. Same OK/MINOR/MAJOR taxonomy as v0.26.0 Quant-Lobotomy. - soup diagnose <run-id> [--evidence|--output|--badge|--attach-to-registry] - soup train --diagnose-gate <evidence.json> refuses MAJOR runs - diagnose_report added to registry._VALID_KINDS Review wave (4 agents): 4 HIGH + 8 MEDIUM + 2 LOW addressed — atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), realpath containment, evidence size cap, contamination combined-complexity cap, ReDoS probe, extras null-byte sanitisation, extract_row_text centralisation, tokenize delegates to _eval_text. Test count: 8676 -> 8849 (+123 in test_v0560.py + 50 net adjustments). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
04c504e761 |
test(eval): strip ANSI before help-output substring asserts (v0.55.0 CI fix)
3 macOS CI failures from the v0.55.0 push — Rich wraps option names with ANSI escapes when the terminal is narrow (macOS CI runners default to a smaller width than Linux/Windows), so substring searches like `"--goal" in result.output` fail because the actual output contains `\x1b[1;36m-\x1b[0m\x1b[1;36m-goal\x1b[0m`. Project precedent: v0.53.5 / v0.53.6 / v0.53.8 / v0.53.9 all hit the same pattern; tests/test_auto_tuning.py and tests/test_eval_platform.py already ship `_ANSI_RE` + `_strip_ansi` helpers. Failures fixed: tests/test_v0550.py::TestCLIPlumbing::test_eval_design_help tests/test_v0550.py::TestEvalAgainst::test_against_help tests/test_v0550_followups.py::TestEvalAgainst::test_against_cli_help_lists_flag Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
58d7d510bf |
feat(eval): soup eval design — derive evals from data (v0.55.0)
Trainer libraries help you RUN evals — none help you DEFINE them.
v0.55.0 closes that gap with 5 new subcommands:
- soup eval design <data> --goal "..." → goal-conditioned EvalDesign
(TF-IDF salience + scorer dispatch)
- soup eval discover <data> → held-out canaries + memorization probes
(farthest-first Jaccard clustering)
- soup eval lock + soup eval coverage → SHA-256-checksummed artifact +
gap analysis vs v0.54.0 task taxonomy
- soup eval gate-install --baseline R → pre-push regression gate
(paired-bootstrap CI, shlex.quote)
- soup eval against B --candidate C → run-vs-run paired-bootstrap CI
Heuristic / CPU-only — no GPU required. Lazy imports across all 6 new
modules so `soup --help` startup remains < 200 ms.
New registry artifact kinds: eval_suite, canaries.
New tracker accessor: ExperimentTracker.get_metric_series(run_id, metric).
Security policy (all atomic-write + read surfaces):
- cwd containment via os.path.realpath + commonpath
- unconditional os.lstat + stat.S_ISLNK rejection (TOCTOU defence)
- atomic write via tempfile.mkstemp + os.replace
- shlex.quote for shell-script generation (NO hand-rolled escape)
- MappingProxyType on every registry / metric / scorer map
- frozen dataclass on every public return type
- bool-as-int rejection on every numeric input
- DoS caps: 10k subsample for TF-IDF + clustering hot paths
Review-fix coverage across 4 agents (python / security / code / tdd):
0 CRITICAL + 7 HIGH + 11 MEDIUM + 6 LOW resolved before commit.
Tests: 8571 → 8676 (+105 net).
Lint: ruff clean.
Smoke: every CLI command + every failure mode exercised in /tmp/soup_smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
9e18643ea0 |
test(advise): fix cross-platform CI failures in test_v0540
Two failures on ubuntu/macos/windows × py3.9/3.11/3.12 after v0.54.0 push: 1. test_env_null_byte_falls_back: monkeypatch.setenv can't set raw NUL into the OS env layer (POSIX execve + Win32 SetEnv both refuse). Switched to a temporary `advise_history.os.environ` swap so the helper's defence-in-depth NUL guard is still exercised without going through the C env layer. 2. test_default_missing_data: Click 8.0–8.1 returns rc=0 on `no_args_is_help=True` invocations; Click 8.2+ returns rc=2 (the "missing command" convention). CI runners had the newer Click; dev box had the older. Accept both renderings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
600686cd70 |
feat(advise): soup advise — pre-flight decision (v0.54.0)
`soup advise <data.jsonl> --goal "..."` returns one of PROMPT_ENG / RAG / SFT / DPO / GRPO with a confidence, reason, and reverse-when criterion BEFORE the user spends 8 hours on a GPU. Layer above autopilot — autopilot picks hyperparams AFTER the training decision; advise picks the training decision itself. Three Parts: - Part A: Verdict engine — TASK_CATEGORIES + CHOICES allowlists, frozen Verdict / DatasetProfile / ROIEstimate dataclasses, pure- Python classify_task + compute_dataset_profile + build_verdict rubric (DPO / GRPO floor 500 / PROMPT_ENG floor 50 / RAG / SFT). - Part B: Probe runner — synth_probe_baselines + synth_probe_lora_delta heuristic stubs with forward-compat model/device/lr/timeout_seconds kwargs (v0.54.1 lifts to live model loading per stub-then-live cadence used by v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus). - Part C: Cross-project learning — ~/.soup/advise_history.jsonl with cross-process file locking (fcntl on POSIX, sidecar <path>.lock + msvcrt on Windows). `soup advise compare` reads history; env override SOUP_ADVISE_HISTORY_PATH containment-checked to $HOME / $CWD / tempdir (mirrors v0.36.0 SOUP_BATCH_CACHE_PATH policy). CLI: Typer subcommand group `run` / `explain` / `compare` plus argv preprocessor in cli.py that maps `soup advise data.jsonl` → `soup advise run data.jsonl`. Scoped to argv[1] == "advise" only (code-review HIGH fix — defends against rewrites when an unrelated arg contains the literal string "advise"). Schema: AdviseConfig (goal / probe / record) field on SoupConfig honors the plan's cross-cutting bullet. Security: cwd-containment + os.lstat + S_ISLNK symlink reject on every path input; atomic writes via tempfile.mkstemp + os.replace on scratch + history; per-line 64 KB cap + 16 MiB file cap on history reads; bool / finite / NUL / oversize guards on every public input; Rich markup escape on user-controlled output. Reviewed by python / code / security / tdd / architect agents — every finding fixed before commit (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW). Test count: 8400 → 8571 (+136 in tests/test_v0540.py, +35 net adjustments to v0.53.x version-pin assertions to forward-compat >=). Note: Windows CRLF / LF warnings during stage are .gitattributes- governed and benign. CI runs on ubuntu-latest / windows-latest / macos-latest × Python 3.9 / 3.11 / 3.12. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
cfdabf2b3b |
feat(v0.53.11): GRPO Plus finish + preference live
Closes v0.50.1 (#123, #126, #127), v0.49.1 (#119), v0.40.1 (#68). #123 — live math kernels for 6 GRPO variants (gspo/dapo/dr_grpo/bnpo/ two_sided/rft) + `_GRPOTrainerVariant` HF Trainer subclass via `make_grpo_trainer_variant` factory. Variant compute_loss reads kernel inputs FIRST (no double-forward); falls back to super() only on missing attrs. Case-insensitive variant normalisation before lru_cache. #126 — PRMTrainerWrapper + `_PRMTrainer` HF Trainer subclass with real compute_loss (gather hidden states at step_positions -> reward_head -> MSE via compute_prm_loss). Dataset wrapped in datasets.Dataset.from_list for HF Trainer compatibility. Bool-before-isinstance guard on batch_size. #127 — GRPOStabilityCallback inherits transformers.TrainerCallback (lazy), live EMA ref-model update in on_step_end with strict=True + fallback-to-strict=False-with-WARNING on key mismatch (silent corruption defence). math.isfinite guard on alpha. #119 — LongLoRA forward override via LongLoRAForwardOverride context manager with idempotent install (_soup_longlora_patched marker prevents re-entry double-wrap), 256-char class name cap on regex match, restore on __exit__ AND on exception. #68 — true per-batch weighted-sum preference combine reading policy/ref logps from TRL inputs + each compute_*_term kernel + combine_losses. Explicit None checks on trainer attrs (no `or` on possibly-tensor), DEBUG log on per-term skip. Review fixes from 4 agents (python/code/security/tdd): 10 HIGH + 8 MEDIUM + 7 LOW — see CLAUDE.md v0.53.11 entry for the full list. Test count: 8330 -> 8400 (+75 in test_v05311.py: 54 initial + 21 review-fix coverage gaps). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
76f033a6ff |
feat(v0.53.10): Quick wins + packaging + UX wiring
7 issues closed: - #150 [mix] pyproject extra bundles scikit-optimize so `soup data mix --optimize` runs the Bayesian loop instead of the v0.48.0 Dirichlet fallback; new describe_default_optimizer() helper labels the active backend without paying skopt's import cost. - #113 [data-pro] extras (langdetect + presidio-analyzer) with lazy fall-through helpers in utils/data_score (broader language coverage + Presidio entity recognition on top of the v0.47.0 regex baseline). Llama-Guard-3-1B documented as a manual recipe (license + size). - #154 SOUP_POSTHOG_KEY / SOUP_POSTHOG_ENDPOINT env override via sentinel-based explicit-vs-env precedence; HTTPS-only + RFC1918/link-local rejection on the endpoint; null-byte / control-char / >256-char rejection on the key. - #152 --hub flag plumbed on chat / serve / infer / merge / export / push via shared utils/hubs.apply_hub_to_cli_model + prefetch_model_from_hub helpers; push uses upload_repo (skips HF-specific Collections + model-card auto-render on non-HF hubs). - #153 `soup data download --hub modelscope|modelers` live SDK (lifts the v0.53.8 advisory-only path); friendly ImportError advisory when the SDK is missing. - #155 Web UI Tool Outputs panel — `loadToolOutputs` polls /api/tool-outputs every 3s; XSS-safe DOM-built table (textContent per cell, no innerHTML for user-controlled fields); Bearer token threaded via the v0.53.9 window._authToken bootstrap. - #156 SoupTrainerCallback.on_step_end records tool_calls counts from kwargs['inputs'] into the global tool buffer. Best-effort (# noqa: BLE001 per project policy — training must never crash). 13 review-fixes applied (4 HIGH / 5 MEDIUM / 4 LOW): - HIGH PostHog explicit-endpoint precedence sentinel - HIGH absolute path leak in local_path advisory reduced to relpath - HIGH Rich markup escape on base / local_path / cache_dir - HIGH callback # noqa: BLE001 per project policy - MED `import time` moved out of try block - MED oversize key + explicit-empty key rejection tests - MED source-grep regression guards (advisory-removal, helper imports across 5 non-push commands) - MED `prefetch_model_from_hub` outside-cwd cache_root rejection - LOW empty-list + bool-True tool_calls no-op tests - LOW push.py uses upload_repo + validate_hub_name regression guard Test count: 8285 -> 8330 (+45 in tests/test_v05310.py). Full suite green; ruff clean; on Win+Py3.10. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e8422660a5 |
fix(tests): strip ANSI codes before help-output substring asserts (v0.53.9)
CI's Rich pipeline emits styled output that splits `--vocab-size` into multiple ANSI-bracketed spans (e.g. `\x1b[36m-\x1b[0m\x1b[36m-vocab\x1b[0m\x1b[36m-size\x1b[0m`), breaking naive `"--vocab-size" in result.output` checks. Locally Rich auto-detects non-TTY and skips the codes, so the regression only shows on CI (ubuntu/macos/windows × 3.9/3.11/3.12). Fix: small `_plain()` helper using `re.sub(r"\x1b\[[0-9;]*m", "", ...)` applied to the 7 failing assertions. Same approach already used in several other v0.5x test modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
bde9d149a0 |
feat(v0.53.9): Live Dashboard + UX + Bench + Standalone CLIs
Eight features that close out the v0.44.x live-monitoring deferrals plus a long tail of standalone CLI wins: - #94 /api/train/stream async SSE with per-subscriber cursor + JS EventSource consumer; SoupTrainerCallback pushes TrainEvent on each on_log. - #95 soup ui --public derives LAN IP via SOCK_DGRAM connect-trick, prints scannable QR; --auth-token override; SPA bootstrap hydrates window._authToken from ?token= + sessionStorage and cleans the URL via history.replaceState; CORS regex auto-widens to loopback + RFC1918 in public mode; set_auth_token rotation race fixed via threading.Lock. - #98 soup serve --reasoning-parser strips <think>...</think> (and OpenThinker tags); pre-compiled regex with marker-token fast-path + 1 MiB cap + leading-newline-only strip. - #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON endpoint; best-effort observation hook in callback.on_log. - #15 soup tokenizer train: BPE training CLI with raw-path lstat symlink rejection, 50 MiB total / 8 KiB per-line caps, post-mkdir output-dir re-check, --special-token NUL/oversize dedup, vocab bounds [256, 200000]. - #26 soup bench --p50 --p95 renders extra per-prompt tail-latency Rich table; --prompts-file gains symlink rejection. - #28 soup bench --backend auto: MLX weights.npz probe (per-entry lstat) -> config.json model_type keyword -> transformers fallback; SOUP_BENCH_BACKEND env hint. - #12 examples/synthetic_workflow.{md,yaml} end-to-end walkthrough. Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the python / code / security / tdd review agents. Notable HIGH: - QR token now consumed by SPA (was unreachable previously). - set_auth_token rotation lock-protected, 8-thread stress tested. - Tokenizer input + output symlink TOCTOU defence on raw path. - SSE generator switched to async (asyncio.sleep) for non-blocking multi-subscriber operation. - CORS regex for --public LAN mode (the old fixed allowlist of http://0.0.0.0:port never matched a real Origin header). - _has_mlx_weights per-entry lstat so a symlinked weights.npz can't trigger MLX dispatch. Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the relaxed v0.53.8 version-pin asserts in tests/test_v0538.py). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
13ffc89058 |
chore: ignore out/ directory (training run artifacts)
Local training runs write to ./out/ by default (e.g. .checkpoint.json from soup train). Add it to the existing training-outputs gitignore block alongside output/ and checkpoints/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
53bb82afeb |
fix(v0.53.8.1): hatch artifacts directive — fix PyPI duplicate-filename 400
v0.53.8 PyPI publish failed with: 400 Invalid distribution file. ZIP archive not accepted: Duplicate filename in local headers Root cause: `[tool.hatch.build.targets.wheel.force-include]` shipped `soup_cli/data/_fixtures/` AND `packages = ["soup_cli"]` recursed into the same path, so both the wheel and sdist contained each JSONL twice. Fix: switch from force-include to `artifacts = [...]` which adds non-Python files to the existing package tree exactly once. Standard hatchling pattern for shipping data files inside an already-packaged directory. Version bumped to v0.53.8.1 (patch) — same code surface, just a build config fix. v0.53.8 GitHub release remains as the feature changelog; PyPI ships under v0.53.8.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
7b98982ab6 |
fix(test): v0.53.8 CI hardening — strip ANSI + use _repo_root() helper
Five v0.53.8 CI failures (ubuntu/macos × py3.9/3.11/3.12):
1. test_help_lists_hub_flag — Typer's Rich-rendered help wraps long
option help across ANSI box-drawing lines; "--hub" appears as
"│ --\nhub" in the CI terminal renderer. Strip ANSI + collapse
whitespace before asserting.
2-5. test_pyproject_version / test_*_extra_present / test_force_include
— used `Path("pyproject.toml")` (relative to cwd). CI invokes pytest
from a different cwd than the repo root on at least one matrix
entry. Switched to a `_repo_root()` helper that derives from
`__file__` (matches v0.43.0 Part D demo_bundles approach).
Local re-run: 66/66 v0.53.8 tests pass after the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
6d2170c4f3 |
feat(v0.53.8): Remote data + Hubs + Trackers (wave 2) — 6 features
- #85 fsspec live loaders — data/loader.py routes the v0.42.0 fsspec scheme allowlist (s3:// / gs:// / gcs:// / az:// / abfs:// / abfss:// / oci://) through fsspec.open with validate_remote_uri containment BEFORE connection. Friendly Rich panel names the pip install advisory when the backend SDK is missing. Threads data.streaming + data.buffer_size. Row count capped at 1M. - #130 Hub dispatcher live — utils/hubs.download_repo() and upload_repo() lazy-import per backend (huggingface_hub / modelscope / openmind_hub). Shared _validate_repo_id_shape (bool / null-byte / leading-slash / .. / control-char / oversize) + cwd containment on local_dir / folder_path. commands/train.py pre-fetches non-HF base into .soup_hub_cache/ (sanitised slug, idempotent on resume, cfg.base updated via model_copy). soup data download --hub flag plumbed. Multi-command rollout for chat / serve / infer / merge / export / push tracked for v0.53.9. - #89 [trackers] pyproject extra bundles mlflow / swanlab / trackio; tracker_missing_dep_message surfaces a friendly pip install advisory via importlib.util.find_spec (non-executing probe). - #90 utils/trackers.send_telemetry_payload — opt-IN via SOUP_TELEMETRY=1; lazy httpx; 1s hard timeout; HTTPS-only with SSRF re-validation (mirrors v0.51.0 hub endpoint policy); silent-fail on every exception. - #93 Fixtures migrated to soup_cli/data/_fixtures/ — zipapp / namespace-package safe via [tool.hatch.build.targets.wheel.force-include]; _bundle_source_path falls back to examples/data/ for editable installs. - #69 utils/hf_space.detect_space_sdk(requirements_text) — picks "streamlit" / "gradio" from the rendered requirements.txt; closes the v0.40.2 known limitation that custom Spaces always defaulted to gradio. Wired into commands/deploy.py. Review pass: python-review + code-review + security-review ran in parallel; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). Highlights: cwd-containment on local_dir/folder_path, Windows ..\ traversal defence on .soup_hub_cache slug, Pydantic model_copy(update=...) instead of attribute mutation, idempotent pre-fetch via cache probe, 1M-row cap on remote materialisation, SSRF re-validation on telemetry endpoint override, 256 KB cap on detect_space_sdk input, modelscope.push_model commit_message kwarg removed (would TypeError at runtime), find_spec instead of __import__ to avoid swanlab side-effects. Test count: 8162 -> 8257 (+66 in tests/test_v0538.py + 29 net adjustments). Lint clean. CPU smoke: version, --help, load_config_from_string with hub: modelscope passes; mlx + non-HF rejected; data download --hub modelscope advisory rendered; detect_space_sdk live on real requirements.txt bodies; package-data fixtures resolve from soup_cli/data/_fixtures/. v0.53.7 known limitation #1 (bash 501 marker) bumped to v0.53.9. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
0ddefc5c6f |
docs: SECURITY.md v0.53.7 per-version fix note
Audit follow-up — v0.53.7 was added to the supported-version list but the per-version security fix paragraph was missing. Add the dense hardening summary matching the v0.53.6 / v0.53.5 style: bash 501 revert rationale, atomic checkpoint write, _node_seed lstat-on-raw-path, failed_reason redaction, Bearer auth gate on tool endpoints, SSE header injection defence, vLLM /v1/messages loopback CORS, atomic Arrow save, cache-hash gate on pre_tokenized short-circuit, prompt_strategy trusted-input limitation. Docs-only — no version bump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
a26511f868 |
fix(test): v0.53.7 — symlink test accepts either rejection branch
test_load_jsonl_rows_rejects_symlink: when the symlink target is outside
cwd, is_under_cwd's realpath resolution catches it before the lstat check
fires. Both rejections are valid security guards; broaden the regex to
match either error message ("symlink" or "under cwd").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
9c9f962676 |
fix(test): v0.53.7 CI hardening — importorskip(fastapi) + Arrow target setup
CI run 25805598433 failed across all 9 OS×Python cells: - TestToolEndpointsLive / TestAnthropicMessagesStreaming / TestReviewFixesVllmAnthropicLive ModuleNotFoundError: fastapi (CI does not install [serve] extra) → autouse fixture pytest.importorskip() - test_load_pretokenized_dataset_rejects_symlink: load_pretokenized_dataset called datasets.load_from_disk on the symlink target before the lstat check ran → moved the lstat + S_ISLNK check to the entry of the helper so symlinks reject before any load attempt - test_redact_exc_message_handles_windows_paths: hardened _redact_exc_message to strip both POSIX absolute paths and Windows-style paths regardless of host platform Local pytest tests/test_v0537.py: 112 passed, 7 skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
18d8b36114 |
feat(v0.53.7): Data Forge + Pipeline live (wave 1) — 11 items
Closes v0.47.0 deferrals (#111, #112), community QA (#75), v0.42.1 wave 1 (#87, #86, #88), and 5 v0.53.6 stub-to-live carry-overs (#102 vLLM parity + SSE streaming, #103 tool HTTP endpoints, #105 instantiate_trainer_plugins, #106 run_recipe DAG runner). - #88 markdown ingest heading split (split_markdown_by_headings) - #112 soup data decontaminate --benchmark-file (cwd-contained operator corpus + symlink rejection) - #87 prompt_strategy live resolver (resolve_prompt_strategy + lru_cache, importlib-based; per-row hook in sft_format.py) - #86 soup data preprocess AOT tokenize (atomic Arrow shard write + cache metadata sidecar; SFT + Pretrain wrappers short-circuit on format='pre_tokenized' + tokenized_path with cache-hash gate) - #111 forge --judge-provider {ollama,anthropic,vllm} live (lazy v0.20.0 providers; SSRF parity) - #75 QA log entry for synth-data provider manual smoke - #106 run_recipe LIVE for 6 NODE_KINDS (seed / llm_text / code / judge / validator / sampler); atomic checkpoint via tempfile.mkstemp; resume rehydrates predecessor outputs from per-node sidecar JSONL; lstat-on- raw-path symlink rejection (v0.33.0 #22 TOCTOU parity); failed_reason path-redacted - #105 instantiate_trainer_plugins LIVE for cce_plugin / grokfast / spectrum / llmcompressor / sonicmoe / math_verify (lazy imports, friendly pip-install advisory on missing dep) - #103 POST /v1/tools/python + /v1/tools/web_search LIVE (Bearer auth gate, deny-by-default domain allowlist, 5s timeout, 5-result cap). bash reverted to HTTP 501 — security review caught /bin/sh -c child escapes RLVR sandbox's OS-level isolation; deferred to v0.53.8. - #102 vLLM /v1/messages parity LIVE on both backends; CORS loopback-only - #102 Anthropic-shape SSE streaming on /v1/messages LIVE; Cache-Control: no-store; CRLF/NUL/oversize strip on model+msg_id (header injection) Review fixes (1 CRITICAL + 11 HIGH + 17 MEDIUM + 10 LOW from python-reviewer + code-reviewer + security-reviewer + tdd-guide) all addressed in this commit. Test count: 8051 → 8162 (+111 in tests/test_v0537.py). Test files: 189 → 190. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
c9e32f4e19 |
fix(test): v0.53.6 CI hardening — importorskip(fastapi) + strip ANSI
CI failures on `tests/test_v0536.py`:
1. `ModuleNotFoundError: No module named 'fastapi'` on Ubuntu/macOS runners
— fastapi is in the [serve] extra, not [dev]. Added
`pytest.importorskip("fastapi")` to all 7 tests that use TestClient
(matches the pattern in `_build_app`).
2. `--execute` / `--output` / `v0.53.7` substring checks failed because
CI terminals render Typer/Rich help text with style spans (`-` and
`-execute` end up in separate `\x1b[...]m` runs). Added `_strip_ansi`
helper + wrapped 5 substring checks. Same fix pattern as the v0.53.5
`--live` CI fix on test_v0535.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
70326f1aa7 |
feat(v0.53.6): plugin callback + Anthropic /v1/messages + n-gram spec + 3 stubs
Ships v0.53.6 "Plugin + Agent + Anthropic API" — 6 features, 3 live and 3 stub-then-live with v0.53.7 markers. LIVE - #101 SoupPluginCallback fans HF Trainer events to enabled Soup plugin hooks (pre_train/post_train/pre_step/post_step). Hook exceptions swallowed at WARNING. Hook snapshot collected once and passed to ctor (race-free per code-review fix). Wired into all 13 transformer-backend trainers via utils/peft_wiring.attach_plugin_callback. - #102 POST /v1/messages on transformers backend reuses the v0.45.0 anthropic_messages converter + existing chat handler. Streaming -> 501. Validation errors -> generic 'Invalid request' body, details at DEBUG (security-review redaction fix). - #104 n-gram speculative decoding: NgramSpecConfig.num_draft_tokens threaded through model.generate(prompt_lookup_num_tokens=N). Mutually exclusive with assistant_model. STUB-THEN-LIVE (v0.53.7) - #103 /v1/tools/{python,bash,web_search} return HTTP 501. - #105 instantiate_trainer_plugins validates then NotImplementedError. - #106 run_recipe + 'soup data recipe --execute --output <dir>' with CLI-side is_under_cwd containment before the live runner. Tests: 7998 -> 8051 (+53 in tests/test_v0536.py). Lint clean. Three review agents (python / code / security); every HIGH/MEDIUM fixed. TDD coverage gaps closed (ngram-None regression, max_tokens cap, run_recipe boundaries, console-print failure swallow). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
8bd67f697f |
fix(test): strip ANSI before --live substring check (v0.53.5 CI fix)
Rich splits the `--live` token across ANSI colour codes on narrow CI terminals (`-\x1b[0m\x1b[1;36m-live`), so the raw-output substring assertion failed on macOS/Windows runners but passed locally on a wide terminal. Strip ANSI escapes before the check — matches how earlier test files (e.g. test_v0402_part_b) handle Rich-coloured `--help` output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
bc1f060da9 |
feat(adaptive): v0.53.5 — Adaptive Training (BETA → stable)
Closes #114, #115, #116, #117, #118, #17 — lifts every v0.48.0 BETA deferral and adds the deepseek-v3-reasoning recipe. - #114 DynamicCurriculumCallback live (TrainerCallback + all_reduce + JSONL) - #115 curriculum_dynamic schema gate widened to 13 transformer trainers - #116 soup data mix --live runs real proxy soup train subprocesses - #117 skopt.Optimizer(GP) wrapped behind OptimizerProtocol - #118 MixOptimizationReport.elapsed_seconds excludes failed candidates - #17 deepseek-v3-reasoning GRPO recipe Test count: 7935 → 7998 (+63). 4 review agents (python/code/security/tdd) ran sequentially; every CRITICAL→LOW finding fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
89b06efb8c |
feat(longctx): v0.53.4 — Long Context + Architecture
Six closes lifting the v0.49.0 LongLoRA hardening + v0.41.0 LLaMA Pro deferred stubs, plus a UX upgrade to the CUDA-OOM friendly message: - #11 utils/errors.py: OOM hint now names --batch-size / --grad-accum - #122 flash_attn.is_flash_attn_v3_available() + LongLoRA+FA3 schema reject - #120 LongLoRA arch allowlist expansion (Mistral / Qwen / Phi); Mixtral intentionally excluded (regex matches the bare 'mistral' token only) - #121 apply_long_context_config auto-detects 'llama3' when caller passes rope_scaling_type=None and the model config carries a Llama 3.1 rope_scaling block - #83 block_expansion.expand_model_blocks LIVE (deepcopy last-N blocks, zero-init residual projections, append, bump num_hidden_layers) + apply_llama_pro_freeze + shared apply_block_expansion_if_configured helper wired into SFT + Pretrain (mirrors v0.40.6 peft_wiring centralisation policy so SFT and Pretrain stay in lock-step) - #74 HF push surface QA — test plan recorded in tests/qa/v053_qa.md; live execution against a private HF repo deferred to a credentialed contributor Review pipeline (python / code / security / tdd agents) ran; every CRITICAL -> LOW finding addressed: - bool-first guards in _check_model_name (defends against int subclass) - is_supported_longlora_arch defensive non-string surface (returns False, never raises) matching v0.53.3 is_known_vlm_base policy - _truncate_for_message(value, limit=64) bounds the base echo in LongLoRA error messages (security MEDIUM, mirrors v0.34.0 crash.py) - null-byte + non-string TypeError guards on validate_longlora_compat task / backend params (matches v0.50.0 validate_long_context_grpo_compat) - _get_layers_module uses explicit `is None` not falsy shortcut (defends against nn.Module.__bool__ overrides on subclasses) - _zero_init_block_residual returns bool + warnings.warn when neither standard projection matches the cloned block (non-Llama-shaped arches still train but lose the LLaMA Pro identity-init guarantee) Test count: 7879 -> 7935 (+56 net; +49 in new tests/test_v0534.py). Lint clean. CPU smoke verified on a real transformers.LlamaForCausalLM: 4 -> 6 layers, down_proj + o_proj actually zeroed on PyTorch tensors, old blocks frozen + new blocks trainable, forward pass finite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
29f875f6ef |
feat(grpo): v0.53.3 — grpo_fp16 routing + vision-GRPO VLM base probe
Two surgical fixes from the v0.50.0 GRPO Plus deferred-stub family land: - #128 grpo_fp16 routing: GRPOTrainerWrapper._build_precision_kwargs returns {fp16, bf16} per (device, grpo_fp16) matrix (CPU/MPS/XPU → both False, CUDA + grpo_fp16=True → fp16/!bf16, default CUDA → legacy bf16). SoupConfig._validate_grpo_fp16_amp_exclusive rejects the silent-mutex combo with auto_mixed_precision=True; short-circuits when task != 'grpo' so the v0.50.0 task-gate diagnosis fires first. - #129 vision-GRPO base probe: KNOWN_VLM_REGEX covers 10 VLM families (Qwen2-VL/Qwen2.5-VL/QVQ/Pixtral/InternVL/Llama-3.2-Vision/LLaVA/ MiniCPM-V/Idefics/ShareGPT4V/Fuyu) with word-boundary anchors; is_known_vlm_base returns False (never raises) on bad input; validate_vision_grpo_compat now accepts optional base kwarg with 64-char error-message truncation. YAML pairing vision_grpo: true with a non-VLM base is rejected at schema load with a friendly families listing instead of a cryptic runtime AttributeError. Scope: 4 larger v0.53.3 items (#127 stability callback, #123 GRPO variant losses, #126 PRMTrainerWrapper, #68 multi-objective preference live combine) are scope-deferred to v0.53.4 — each warrants its own focused release per the v0.40.x stub-then-live cadence. Tests: 7842 -> 7879 (+37 in tests/test_v0533.py). Four review agents (python/code/security/tdd) ran; every HIGH/MEDIUM/LOW finding fixed (task-gate priority short-circuit, MPS branch documented, 64-char error truncation, QVQ regex coverage, 512-byte boundary test). Two pre-existing v0.50.0 Part E tests migrated `base: test-llama` -> `base: Qwen/Qwen2-VL-7B-Instruct` to clear the new probe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2292e81c3f |
feat(modality): v0.53.2 — lift Modality II stubs (distill + classifier + EBFT/GDPO + reasoning_effort)
Closes #132, #133, #135, #137. Records #71 ONNX QA (partial — tiny-gpt2 PASS, TinyLlama-1.1B blocked by host RAM during onnx.load post-process). New trainer wrappers: - DistillTrainerWrapper (soup_cli/trainer/distill.py) — student + frozen teacher, KL/JS divergence kernels scaled by T**2, device-bridge for HF Trainer auto-CUDA promotion, DataCollatorForSeq2Seq for variable-length loss-masked rows, separate trust_remote_code resolution per model. - ClassifierTrainerWrapper (soup_cli/trainer/classifier.py) — single/multi label sequence classification, 1024-entry multi-label cap, label_names string-to-int resolution. Routes classifier / reranker / cross_encoder. Live loss kernels: - apply_ebft_loss (structured / strided) + attach_ebft_compute_loss (SFT) - apply_gdpo_loss (standard / length_normalized / margin) + attach_gdpo_compute_loss (DPO). Both attach hooks idempotent. Prompt-format wiring: - apply_reasoning_effort_prefix injects gpt-oss <|reasoning_effort|>{low,medium,high}<|/reasoning_effort|> header. - build_assistant_only_labels(train_on_eot=True) keeps EOT/EOS unmasked. Bugs surfaced + fixed during Wave 3 CPU smoke (regression guards in tests): - Distill collator did not pad pre-tokenised labels (variable-length crash) - Distill compute_loss device-mismatch when HF Trainer auto-promoted student to CUDA while teacher stayed on CPU. Tests: 7722 -> 7842 (+120 in test_v0532.py). 5 review agents run; every CRITICAL/HIGH/MEDIUM/LOW finding fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
2ea26df5ea |
fix(v0.53.1): CI green — inspect click params directly instead of asserting on Rich --help output
The three CLI tests added in the previous commit (test_help_lists_measure_flag, test_save_format_help_lists_flag, test_torchao_help_lists_quant_config) assumed the literal option name (e.g. `--measure`, `--save-format`, `--quant-config`) would appear contiguously in CliRunner-captured output. On CI runners the terminal defaults to 80-col and Rich wraps long option names across lines, splitting the literal string. Fix: walk `typer.main.get_command(app).params` and collect every `opt` + `secondary_opt` into a set, then assert membership. The test now verifies what we actually care about (the option is registered) without depending on Rich's wrapping behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
725696b1da |
feat(v0.53.1): Quant Menu II + Export pipeline live
Lift six v0.53.0 deferred stubs from NotImplementedError to live wiring: - #82 autopilot pre-quantized base detection utils name regex over gptq/awq/aqlm/eetq/fp8/mxfp4 with word-boundary anchoring + HQQ Nbit extraction + config.json quantization_config probe (cwd-contained + symlink-rejected). decide_quantization() short-circuits the VRAM heuristic when prequantized is set. autopilot pipeline auto- applies so TheBloke/Llama-2-7B-Chat-GPTQ is recommended gptq instead of 4bit-on-top-of-quantized. - #142 merge_4bit + export_torchao live writers soup merge --save-format {fp16|4bit|4bit_forced}: single BNB-4bit merged checkpoint without the dequant->merge->requant cycle (fixes wrong-name llm_int8_skip_modules to bnb_4bit_skip_modules per code- review). soup export --format torchao --quant-config <yaml>: torchao .quantize_ + save_pretrained with per-scheme closed kwarg allowlist (Int4WeightOnly accepts {group_size, inner_k_tiles}, NVFP4 accepts nothing extra; dunder + unknown keys rejected per security-review H1). load_quant_config enforces yaml.safe_load + 256 KB cap + extension allowlist + cwd containment + S_ISLNK rejection. - #139 export_advanced_gguf via llama.cpp imatrix 3-stage pipeline: convert_hf_to_gguf.py -> optional imatrix -> quantize. argv-list subprocess (no shell), 30-min timeout, realpath- verified convert script stays inside llama_cpp_dir (security-review M5). _prepare_calibration_text accepts JSONL with text/prompt/content field aliases + raw text fallback; strips null bytes, collapses newlines, 8 KB per-line + 50 MB total cap (security-review M1); POSIX O_NOFOLLOW closes the TOCTOU window between dispatch-time check and open() (security-review M3). UD- prefix stripped before passing to llama-quantize. _safe_stderr Rich-escapes subprocess stderr before embedding in RuntimeError (security-review L4). - #109 soup deploy autopilot --measure Live Quant-Lobotomy scorecard: classifies each candidate quant OK / MINOR / MAJOR (thresholds 2% / 5% mirror v0.26.0 Part D). Results cached at ~/.soup/deploy_autopilot_cache.json (atomic write, 0o600 perms on POSIX, S_ISLNK rejection on BOTH load and save). pick_best soft-fallback now picks max-by-delta (was max-by-after) matching the v0.33.0 #54 design intent. _DEPLOY_MEASURE_BEFORE_GEN / _AFTER_FACTORY module-level hooks act as the stop-gap escape hatch until v0.46.1 ships first-party transformers / vLLM generator factories. - #70/#72 manual QA log scripted at tests/qa/v053_qa.md with exact reproduction recipes + acceptance criteria for the CUDA + llama.cpp smokes that can't run on the CI runners. Shared cleanup: - soup_cli/utils/paths.enforce_under_cwd_and_no_symlink consolidates the v0.33.0 #22 TOCTOU pattern previously copy-pasted in save_formats.py and gguf_quant.py (code-review HIGH fix). Reviews ran: python / code / security / tdd. Every CRITICAL / HIGH / MEDIUM / LOW finding fixed or documented. Test count: 7610 -> 7722 (+112 across 4 new files). Known limitations: live GPU + bitsandbytes / torchao smokes for the new merge / export paths remain pending (recipes in QA log); injected- generator escape hatch is non-public until v0.46.1; cache key truncates base_sha to 16 hex (1-in-2^32 collision floor). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
07e7214ed3 |
feat(v0.53.0): Quant Menu II — UD GGUFs + KV cache + NVFP4 + LF parity + save formats
Schema-only release. Live wiring deferred to v0.53.1 (mirrors v0.50.0 /
v0.51.0 / v0.52.0 stub-then-live pattern).
- Part A — Unsloth Dynamic 2.0 GGUF ladder (14 entries: UD-Q8_K_XL ... UD-IQ1_M)
+ validate_calibration_data_path shape validator.
- Part B — IQ (12) + Apple/ARM (10) GGUF flavours in utils/gguf_quant.py;
O(1) _LOWER_INDEX MappingProxyType for case-insensitive lookup.
- Part C — training.kv_cache_type: q8_0 | bf16 | f16 | fp8 (fp8 Hopper-only;
MLX rejected). requires_hopper reads from spec metadata (single source).
- Part D — fp8_attention (requires quantization_aware='fp8') + nvfp4 (Blackwell)
+ native unsloth_bnb_4bit bool flags with cross-validators.
- Part E — bnb_4bit_use_double_quant + llm_int8 (explicit 8bit assertion,
distinct from v0.41.0 load_in_8bit aliasing) + quantize_ref_model
(extends ref-task set with grpo/kto/ppo) + quantize_reward_model.
- Part F — soup merge --save-format {fp16, 4bit, 4bit_forced} + soup export
--format torchao with closed PTQ scheme allowlist (Int4WeightOnly,
Int8DynActInt4, Float8DynActFloat8, NVFP4 — case-sensitive PyTorch names).
Test count: 7453 → 7610 (+157 net new across 154 tests in test_v0530.py).
ruff check soup_cli/ tests/ — clean.
5 review agents ran (python / code / security / tdd / verification);
every CRITICAL / HIGH / MEDIUM / LOW finding fixed or documented:
- O(N) gguf walk → O(1) _LOWER_INDEX MappingProxyType
- ref_tasks extended with grpo + kto + ppo (silent-no-op footgun)
- _validate_v053_bool_fields no longer coerces None → False
- requires_hopper delegates to _KV_CACHE_METADATA spec
- fp8_attention validator order: quantization_aware before MLX
- validate_calibration_data_path + validate_quant_config_path docstrings
name the exact controls v0.53.1 CLI dispatch MUST add (TOCTOU contract)
- validate_torchao_scheme case-sensitivity documented at validator
- tautological `result == result` test replaced with allowlist invariant
- bool guards added on backend/modality/quantization across all Part D
validators
- exact 4096/4097 boundary tests for path shape validators
Docs updated: CLAUDE.md (test counts + utils list + changelog + test-table),
README.md (What's New replaced + 5 new dedicated sections), SECURITY.md
(support window + v0.53.0 hardening entry), CONTRIBUTING.md (test count
+ utils list + test-table row), .claude/plan.md (heading + boxes + banner —
gitignored, local only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
df7f49feda |
feat(v0.52.0): Modality II — TTS + Distillation + BitNet + EBFT-GDPO + MoE quant + reasoning_effort
7 schema-only Parts; live trainer / loss / export wiring deferred to v0.52.1
(mirrors v0.27.0 MII / v0.50.0 GRPO Plus / v0.51.0 hubs stub-then-live pattern).
- Part A: task='tts' + modality='audio_out' + 5 families (Orpheus/Sesame-CSM/
Llasa/Spark/Oute) + per-family emotion allowlist (Orpheus 8 / Oute 6) +
5 recipes (orpheus-tts-sft, sesame-csm-tts, llasa-tts, spark-tts, oute-tts).
- Part B: classifier / reranker / cross_encoder tasks + num_labels (bool-
before-int guard) + classifier_kind + label_names (dedup + null-byte + cap).
- Part C: task='distill' + teacher_model + distill_divergence (kl alias
canonicalises to forward_kl; Literal excludes alias) + distill_temperature
(math.isfinite + [0.05, 100.0] bounds).
- Part D: quantization='bitnet_1.58' (gated to non-MLX + text + task in
{sft, pretrain, dpo}) + Falcon-E BitNet recipe + soup export
--format bitnet/tq1_0 CLI stubs (yellow deferred-advisory panel, exit 0).
- Part E: EBFT (structured/strided + bounded ebft_temperature; SFT-only)
+ GDPO (standard/length_normalized/margin; DPO-family-only).
- Part F: moe_expert_quant (nf4/int8_rowwise) + train_router_only — both
require moe_lora=true (silent-no-op footgun rejection).
- Part G: reasoning_effort (low/medium/high) + train_on_eot — both gated to
the SFT-family task set (sft/pretrain/distill/classifier/reranker/
cross_encoder); rejected on DPO/GRPO/PPO/etc. with named offenders.
Review fixes (5 agents: python, security, code, tdd, verification-loop-manual):
- num_labels bool-before-int field_validator (security HIGH)
- reasoning_effort + train_on_eot SoupConfig task-gate (code HIGH)
- _validate_classifier_compat lazy-import early-return (code HIGH)
- Oute emotion allowlist via data-driven _FAMILY_EMOTIONS (python+code MED)
- _MAX_LEN -> _MAX_REASONING_EFFORT_LEN (python MED)
- validate_reasoning_effort wired via field_validator (security MED)
- distill_divergence Literal excludes "kl" alias (code MED)
- DIVERGENCES derived from _DIVERGENCE_ALIASES drift guard (python LOW)
- is_bitnet_model comment fixed to match impl (python LOW)
- sister-fn bool guards on every compat helper (python+security MED)
- TDD coverage gaps closed: EBFT variant oversize, GDPO full rejection matrix,
ebft_temperature explicit-exc table, TTS compat input guards, recipe
model-id null/whitespace, full reasoning_effort task-matrix.
Drift fixes:
- tests/test_onnx_tensorrt_export.py + tests/test_awq_gptq_export.py
SUPPORTED_FORMATS count bumped 5 -> 7 (bitnet + tq1_0 stubs).
- tests/test_recipes.py catalog_size assertion 106 -> 112 (5 TTS + Falcon-E).
Test count: 7184 -> 7456 (+272). 230 new tests in tests/test_v0520.py;
remainder from drift-fix parametrize expansions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
352f8d4bfd | docs(contributing): correct v0.51.0 test count (7178 → 7184) | |
|
Alpamys
|
1e1abacb44 |
feat(catalog): v0.51.0 — Model Catalog Expansion + Alternative Hubs
26 new ready-made recipes (catalog 80 → 106) covering 25 model families: GPT-OSS 20B/120B, GLM 4.6/5, Kimi K2 / K2-Thinking GRPO, MiniMax-M2, QwQ-32B GRPO, QVQ-72B, Granite 4, Liquid LFM2, Cogito v2, Mistral Small 3 / Medium 3.5, Magistral / Devstral / Ministral, MedGemma, EmbeddingGemma, LLaVA-Next, InternVL 3.5, Voxtral, Baichuan 2, Qwen-Image, DeepSeek-OCR, Paddle-OCR-VL. Part D: MULTIPACK_ARCHITECTURES extended 18 → 38 (Granite, GLM, Kimi, MiniMax, QwQ, QVQ, GPT-OSS, Magistral, Devstral, Ministral, MedGemma, LFM2, Cogito, Hunyuan, Ernie, Yi, Baichuan, ChatGLM). Part E: alternative model hubs. New soup_cli/utils/hubs.py with closed allowlist (hf/modelscope/modelers), SSRF-hardened endpoint validators mirroring v0.29.0 HF_ENDPOINT policy (scheme allowlist, loopback-only HTTP, RFC1918/link-local rejection, control-char/CRLF rejection, IPv6-mapped private rejection). TrainingConfig.hub Literal field with case-insensitive _normalize_hub field_validator; SoupConfig _validate_hub_supported rejects backend=mlx + hub != hf. Live downloader / uploader wiring deferred to v0.51.1 (matches the v0.27.0 MII / v0.37.0 multipack stub-then-live pattern). +455 tests (6729 → 7184), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
33c60b4c1f |
feat(grpo): v0.50.0 — GRPO Plus (unsloth + axolotl RL parity)
22 features across 5 internal Parts shipped as schema-only — closed allowlists, Pydantic validators, NotImplementedError stubs for live wiring deferred to v0.50.1 (mirrors v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro / v0.45.0 plugins / v0.48.0 curriculum / v0.49.0 LongLoRA stub-then-live pattern). Part A — 7 GRPO objective variants (gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard) with `validate_grpo_variant` + frozen `GRPOVariantSpec` metadata + `MappingProxyType`-wrapped registry. `validate_grpo_delta` is bool-first / math.isfinite / (0, 1] bounded. `apply_variant_loss` raises NotImplementedError with v0.50.1 marker for the 6 new variants and is a no-op for standard. Part B — long_context_grpo + vllm_sleep_mode schema gates with compat validators (null-byte rejection on task + backend, bool guard on use_ring_attention). vllm_sleep_mode requires task='grpo' AND a transformers/unsloth backend (code-review HIGH fix — sleep is a between-rollouts feature). Part C — 4 multi-turn rollout backends (art / ruler / nemo_gym / openenv) with closed allowlist + per-entry required_package mapping. Part D — 7 stability/efficiency knobs (ref_model_ema_alpha, replay_buffer_size, async_grpo_prefetch, tis_threshold, mask_truncated_completions, defer_rerolling, skip_zero_advantage, off_policy_mask_threshold). Every numeric field rejects bool via a shared `_reject_bool_on_grpo_numerics` field_validator (tdd-guide HIGH fix — Pydantic v2 coerces True→1 by default). The `mask_truncated_completions` + `tis_threshold` pairing is enforced by a cross-validator (matches v0.32.0 spike-recovery+watchdog policy). Part E — top-level task='prm' Literal addition (Process Reward Model / stepwise-supervised, paired with data.format='prm' from v0.42.0) + `vision_grpo: bool` flag for VLM-RL on Qwen2-VL / Pixtral / InternVL. Compat helpers gate task / modality / backend. Review-round fixes applied (5 sequential reviews per CLAUDE.md): - python-review: list_variants annotation, frozenset[str] params, Optional[str] → str | None, module-level math import, D401 imperative docstrings, dropped *args/**kwargs on stubs. - code-review: grpo_fp16 added to GRPO-only task-gate; vllm_sleep_mode requires task='grpo'. - security-review: explicit field_validator for grpo_delta NaN/Inf rejection (Pydantic le=1.0 incidentally rejects NaN, made explicit); null-byte rejection on backend/task in grpo_long_context helpers; use_ring_attention bool guard. - tdd-guide: bool-rejecting field_validator on all Part D numeric fields + grpo_delta; missing bool-rejection tests added on validate_grpo_variant / validate_rollout_backend; null-byte test on validate_vllm_sleep_mode_compat; required_rollout_package rejection path; RolloutBackendSpec.live_wired; PPO+vision_grpo round-trip; _DEFERRED_LIVE invariant. Test count: 6490 → 6729 (+239 across 5 new test files). Notes for future maintainers: - v0.50.0 has zero new CLI commands and zero new trainer wirings; every step 6d/6e is intentionally n/a. Step 6 smoke runs schema happy + every documented cross-validator rejection. - All `task='grpo'` gates use `if self.task != 'grpo'` literal comparisons; do NOT switch to a set membership check until Part D knobs are wired into PPO/preference trainers in v0.50.x. - Multi-modal Vision RL does not yet verify the base model is actually a VLM — upstream trainer surfaces the error loudly when it fails to load the vision tower. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
161c81ee7f |
feat(long-context): v0.49.0 — YaRN, Dynamic NTK, LongLoRA S², Llama 3.1 NTK
- Part A: YaRN RoPE scaling — math kernels (yarn_find_correction_dim /
yarn_find_correction_range / yarn_linear_ramp_mask / yarn_get_mscale) +
4 yarn_* schema fields + cross-validator rejecting yarn fields outside
rope_scaling_type=yarn. Pure-Python implementations of the upstream YaRN
paper §3.4/§3.5 with bool/NaN/Inf rejection on every numeric input.
- Part B: Dynamic NTK — existing path verified, explicit test coverage.
- Part C: LongLoRA S² shifted-sparse attention (schema-only) — new
soup_cli/utils/longlora.py with is_llama_model (word-boundary regex
mirroring v0.39.0 is_gemma4_model policy) + validate_longlora_compat.
TrainingConfig.use_longlora + SoupConfig._validate_longlora_compat.
Live LlamaAttention.forward override deferred to v0.49.1 (stub-then-live
pattern, mirrors v0.27.0 MII / v0.37.0 multipack).
- Part D: Llama 3.1 NTK-aware (full impl) — scale_inv_freq_llama3
smooth-transition kernel + detect_llama3_rope_in_config HF-config probe
+ "llama3" added to rope_scaling_type Literal. LLAMA3_DEFAULT_*
constants per Unsloth models/llama.py:1853.
Public-boundary input validation on get_rope_scaling_config (bool/NaN/Inf
rejection on target_length / original_length / yarn_factor) per security
review — prevents direct callers from emitting {factor: NaN} into HF
model configs when bypassing the Pydantic schema.
Reviews: code-reviewer (2 HIGH + 1 MEDIUM + 1 LOW), security-reviewer
(1 MEDIUM + 2 LOW), python-reviewer (4 findings), tdd-guide (8 coverage
gaps) — all findings fixed. verification-loop done as manual equivalent
(version + --help + happy/failure YAML smoke).
+80 net new tests (6410 → 6490). Full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
b5cb4d12c1 |
fix(v0.48.0): lstat symlink check on original path, not realpath
CI revealed that on macOS the 3 symlink-rejection tests pass on Windows (where tmp_path is non-symlink-resolved) but fail on macOS because `/var/folders/...` is itself a symlink to `/private/var/folders/...`, and the test's symlink `link.jsonl -> real.jsonl` was being checked AFTER `os.path.realpath` had already followed both symlinks. Fix: `os.lstat(original_path)` BEFORE `os.path.realpath` in three sites — `validate_datasets`, `write_mix_recipe`, `load_mix_recipe`, and the `soup runs curriculum-curve` history reader. Matches the v0.46.0 Part A `_reject_symlink_target` pattern. `FileNotFoundError` from `lstat` is treated as "no existing path" (write target case); other `OSError`s propagate as `ValueError`. Cached `st` result reused for the `overwrite` gate in `write_mix_recipe` so we don't re-stat after realpath. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f789953d46 |
feat(data): Data Mixing Optimizer + v0.48.0 release (v0.48.0 Part B, BETA)
BETA. New `soup data mix --optimize --budget 1h --datasets a,b,c` runs N short proxy-training runs over candidate mixture weights and writes a canonical recipe YAML you can splice into `soup.yaml` under `data.interleave`. Per-candidate proxy failures are isolated (DEBUG-log + sentinel `_MAX_LOSS` + `continue`); `KeyboardInterrupt` / `SystemExit` re-raised; budget cap surfaces `MixOptimizationReport.partial=True`. `soup data mix --apply <recipe.yaml>` re-loads + prints the recipe's canonical interleave block. Both modes enforce `is_under_cwd` containment + TOCTOU symlink rejection (`os.lstat + S_ISLNK`) + 256 KB file cap; YAML key injection defended at the renderer (rejects newlines / null bytes / oversize dataset paths). Synthetic offline proxy ships in v0.48.0; live `soup train` proxy + scikit-optimize backend wiring deferred to v0.48.1 via `OptimizerProtocol` ducktype (default fallback: deterministic Dirichlet sampler). Review fixes: - `validate_datasets` early `len(raw) < 2` check (code-review MEDIUM) — prevents the less-actionable error after realpath resolution. - `run_mix_optimizer` proxy exceptions now isolated per-candidate (code-review MEDIUM) — first-cut raised RuntimeError on the first proxy failure, breaking the documented `partial=True` contract. - `load_mix_recipe` `os.lstat` wrapped in `try/except OSError` (security HIGH) — closes a TOCTOU race where path disappearance between `lexists` and `lstat` would raise an unhandled OSError. Release bundle (v0.48.0): - version bump → 0.48.0 in pyproject.toml + soup_cli/__init__.py - README.md: replaced "What's New" + 2 new dedicated `##` sections - SECURITY.md: supported-versions window + per-version notes for v0.48.0 - CONTRIBUTING.md: test counts (6242 → 6410) + 2 new test-table rows +94 tests. Net release total: 6242 → 6410 (+168 tests, +2 test files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
fe9fe06b68 |
feat(training): Curriculum-Aware Trainer — dynamic bucket re-weighting (v0.48.0 Part A, BETA)
BETA. Adds `training.curriculum_dynamic: true` schema flag with online uncertainty estimation: every N steps, aggregate per-sample loss + grad-norm into per-bucket softmax weights, water-filled to enforce a minimum `curriculum_dynamic_floor`. DDP/grad-accum safety via `validate_distributed_curriculum` cross-validator that rejects un-coordinated multi-rank runs upfront — the well-known footgun where divergent per-rank stats silently desynchronise the sampler. New `soup runs curriculum-curve <run_id>` visualiser with TOCTOU (`os.lstat + S_ISLNK`) + 50 MB file-size cap + 100k-line streaming cap on the history file. Schema gated to sft/pretrain on transformers backend; mlx + non-SFT rejected with distinct messages. Live HF Trainer callback wiring deferred to v0.48.1 (stub-then-live pattern; mirrors v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro). Review fixes: - water-fill design fix (code-review HIGH): removed trailing renorm that could push elements below `floor` when accumulated float error left sum slightly > 1.0. Softmax already sums to 1.0, so water-fill output also sums to 1.0 (drift bounded by nb*eps). - DoS caps on `render_curve` + `parse_history_jsonl` (`_MAX_HISTORY_ROWS=100_000`) — without these an attacker-controlled JSONL with 10M rows would OOM the process. - `curriculum-curve` CLI: symlink rejection + 50 MB + 100k-line caps, null-byte rejection on tracker-supplied `output_dir`. +74 tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
c3f42119d5 |
docs(contributing): add v0.47.0 test-file entries
Backfill the test-file table for v0.47.0 (test_v0470_part_a.py + test_v0470_part_b.py). The full per-release table lives in .claude/CLAUDE.md; this file keeps a curated subset. Docs-only — no version bump required. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f64cee569e |
feat(v0.47.0): Data Forge — synthetic data pipeline + data quality moat
Part A — Synthetic Data Forge (utils/data_forge.py + commands/data_forge.py):
- soup data forge --docs <dir> --task sft|preference|tool with full provenance manifest
- ForgePlan / ProvenanceRecord / ForgeRow frozen dataclasses
- chunk_document + score_uncertainty pure-function kernel
- discover_documents (cwd-contained, symlink-rejecting, .txt/.md/.json/.jsonl allowlist)
- synthesise_forge_rows with judge-exception swallow at DEBUG
- Atomic JSONL + provenance writes via staged-tempfile + os.replace
Part B — Data Quality Moat (utils/data_score.py + commands/data_score.py):
- soup data score / decontaminate / toxicity / langdetect / pii / educational
- ReDoS-hardened PII regexes (phone + credit-card rewritten, 50 KB pre-cap)
- Containment-based n-gram decontamination (docstring corrected from "Jaccard")
- 6-language stopword heuristic for langdetect
- compute_scorecard with per-row DEBUG logging on swallowed errors
Security review fixes applied:
- math.isfinite guard on _validate_float_unit (NaN/Inf rejected before bounds)
- ReDoS: phone regex flattened (no nested optional quantifiers); credit_card
rewritten from {13,19}-loop to anchored 4-4-4-N; 50 KB pre-cap before finditer
- is_under_cwd moved inside discover_documents (no longer relies on caller)
- os.lstat + S_ISLNK rejection on every write target + tempfile staging
- _require_str rejects null bytes (consistency with data_forge._validate_str)
- compute_scorecard try/except blocks log at DEBUG (no silent swallow)
- decontaminate_texts: Optional[...] = None (no more type: ignore)
- _read_rows / _write_rows have full type annotations
- ngram_overlap_ratio docstring renamed to "containment ratio"
- import math + import tempfile moved to module top (lazy-import policy
applies to heavy ML deps only, not stdlib)
- Duplicate discover_documents call in CLI collapsed (TOCTOU window closed)
Live judge providers, [data-pro] extras (Llama-Guard / FineWeb-Edu / Presidio /
fastText / langdetect), and operator-supplied benchmark corpora are
stub-then-live and ship in v0.47.1 (mirrors v0.27.0 MII / v0.37.0 multipack
precedent).
Tests: 6126 -> 6242 (+116 net new). All v0.47.0 + full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
986a4c00e3 |
feat(agent): Agent Forge — OpenAPI/MCP/GraphQL spec → tool-calling SFT dataset (v0.46.0 Part B)
Bumps to v0.46.0 and ships the Agent Forge: parse OpenAPI 3.x, MCP server
manifests, or GraphQL introspection JSON straight into a tool-calling SFT
dataset where each row is `{messages: [user, assistant_with_tool_call],
tool, source_endpoint}`. No more hand-rolled jsonl scaffolding for
function-calling fine-tunes.
* soup_cli/utils/agent_forge.py — `Endpoint` / `SynthRow` / `SpecReport`
frozen dataclasses. `parse_openapi` / `parse_mcp` / `parse_graphql`
parsers leave `$ref` strings opaque (no external resolution — defends
against file-read SSRF). All synthesised path strings routed through
`_validate_path` (rejects newline-in-name across all three parsers).
`load_spec_file`: `is_under_cwd` + `os.lstat + S_ISLNK` BEFORE realpath
(corrects v0.46.0 first-cut ordering caught by security review) + 5MiB
cap + yaml.safe_load only. `write_dataset` atomic via mkstemp +
os.replace (mid-stream TypeError never leaves partial file; mirrors
v0.43.0 Part D `copy_bundle_to` policy) + symlink rejection at target.
Caps: `_MAX_ENDPOINTS=10_000`, `_MAX_SPEC_BYTES=5MiB`,
`_MAX_ROWS_PER_ENDPOINT=32`.
* soup_cli/commands/agent.py — `soup agent synth/train/eval` Typer
subcommands. `synth` table cells pass through `rich.markup.escape`.
`train` rejects NUL/newline/oversize in `--base` and `--output-dir`
BEFORE embedding into rendered YAML recipe (CRITICAL security fix —
defends against YAML key injection where `--base $'evil\ntraining:
{ epochs: 9999 }'` would smuggle in injected training keys). `eval`
enforces predictions `is_under_cwd` + symlink rejection +
`_MAX_PRED_LINES=1_000_000` DoS cap.
* soup_cli/cli.py — registers `agent` Typer group; help string uses
ASCII-safe `->` (`test_help_output_is_ascii_safe` regression test caught
a Unicode `→` on first try).
* tests/test_v0460_part_b.py — 71 tests covering every parser kind,
failure modes (cycle / cap / null-byte / oversize / outside-cwd /
symlink), atomic-write partial-failure invariant, every CLI surface.
* Docs: README ## What's New replaced + dedicated `## Deploy Autopilot`
and `## Agent Forge` sections added; SECURITY.md supported-window
shifted (v0.46→full, v0.41→drop) + v0.46.0 fix-notes entry;
CONTRIBUTING.md test counts 165→167 / 5989→6126.
Test suite: 5989 → 6126 (+137 net new) green on Windows.
Known limitations (live runtime deferred to v0.46.1):
- Quant-Lobotomy auto-measure for deploy autopilot
- RLVR `code_exec` sandbox scoring in `agent eval`
- In-process `soup train` re-entry in `agent train` (Typer commands aren't
safe to re-enter — matches v0.44.0 `soup quantize` design)
- ExecuTorch packaging for iphone-16 / pixel-9 (lands in v0.54.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
479931282c |
feat(deploy): On-Device Deploy Autopilot — 10-profile catalog + recipe/script writer (v0.46.0 Part A)
Closes the v0.45.0 Known Limitations gap that flagged the 15-entry external integrations catalog as descriptive-only. This release ships the live autopilot CLI that picks PEFT + quantisation + speculative-decoding for the target hardware and emits a ready-to-train recipe. * soup_cli/utils/deploy_autopilot.py — `DeployProfile` frozen dataclass + 10-entry MappingProxyType-wrapped catalog: mac-m3 / mac-m4-pro / rtx-3060-12gb / rtx-4090-24gb / iphone-16 / pixel-9 / ollama-local / lm-studio / runpod-a100 / hf-jobs-h100. `_make` factory rejects bool recommended_max_length, non-kebab-case names, runtime/quant/peft outside closed allowlists. `render_recipe_yaml` validates `base` (≤200 chars, no NUL/newline) + `output_dir` (≤4096 chars). `render_deploy_script` uses `shlex.quote` on model_path. `write_recipe` / `write_deploy_script` enforce `is_under_cwd` + `os.lstat + S_ISLNK` TOCTOU rejection at the write target (matches v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E policy). * soup_cli/commands/deploy.py — new `autopilot` Typer subcommand with `--target`, `--base`, `--recipe-out`, `--script-out`, `--list`. Every Rich-rendered profile field passes through `rich.markup.escape`. * tests/test_v0460_part_a.py — 76 tests covering catalog immutability, 10-profile presence, case-insensitive resolution, every `_make` failure mode, render-* validation matrix, write-* path-containment + symlink rejection (POSIX-only), CLI smoke (list / writes / outside-cwd reject). Live Quant-Lobotomy auto-measure (against the v0.26.0 Checker) is deferred to v0.46.1 — this release writes the canonical combo per profile, the runtime measurement step lands in the patch release. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
74301843a2 |
feat(v0.45.0): Plugin System & Ecosystem Wins — 5 Parts, +169 tests
Adds a public plugin/hook system plus the schema scaffolding for 20+ ecosystem
integrations. Live trainer-callbacks, Anthropic /v1/messages route, server-tool
HTTP endpoints, and the recipe runner ship in v0.45.1 (matches v0.27.0 MII /
v0.37.0 multipack / v0.41.0 LLaMA Pro stub-then-live pattern).
Part A — Plugin / hook system
* New soup_cli/plugins/ package: BasePlugin Protocol, PluginSpec frozen
dataclass, register_plugin / discover_hooks / enable_plugin /
disable_plugin / load_plugins. Kebab-case name regex, semver-ish version,
null-byte rejection on every string. Idempotency check covers
(version, plugin object, templates, model_groups, description) — review-fix
added description after first-cut omitted it. Per-list caps on templates
and model_groups (32 entries, 128-char per name).
* New soup plugins list/install/enable/disable Typer CLI; all user-controlled
output passes through rich.markup.escape.
Part B — API extensions (schema-only)
* utils/anthropic_messages.py: to_anthropic / from_anthropic /
validate_anthropic_payload converters. Multiple system messages join with
\n\n; tool role with structured (list) content concatenated into single
tool_result text block (review-fix MEDIUM — first-cut silently dropped).
max_tokens cap 16384, temperature [0.0, 2.0], bool rejection on numerics.
* utils/server_tools.py: closed {python, bash, web_search} allowlist,
WebSearchConfig with domain allowlist + leading-dot subdomain pattern,
rate_limit [1, 600]. is_domain_allowed strips :port suffix and rejects
IPv6 literals (review-fix MEDIUM).
* utils/ngram_spec.py: NgramSpecConfig validators with bounded n / draft
tokens / prompt-lookup-max; bool rejection on every numeric field.
Part C — External integrations catalog
* utils/integrations.py: 15-entry MappingProxyType catalog of ecosystem
targets (lm-studio, comfyui, ollama, claude-code, cursor, continue, ...).
Part D — Advanced trainer-plugin allowlist
* utils/trainer_plugins.py: 6-entry allowlist (grokfast, spectrum,
llmcompressor, sonicmoe, cce_plugin, math_verify) + validate_trainer_
plugin_list (Sequence[str], dedup, _MAX_PLUGINS_PER_RUN=8).
Part E — Data Recipe DAG
* utils/recipe_dag.py: closed NODE_KINDS frozenset, Kahn's topological
sort via collections.deque (review-fix HIGH — first-cut had O(N^2 log N)
queue.sort() inside the BFS body), cycle / self-loop / dangling-edge
rejection, _MAX_NODES=256 / _MAX_EDGES=1024 / _MAX_FILE_BYTES=1MiB.
load_recipe_yaml enforces is_under_cwd containment AND os.lstat + S_ISLNK
symlink rejection (review-fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22
/ v0.43.0 Part C / v0.44.0 Part B policy).
* New soup data recipe <path> CLI validates topology and prints planned
topo order; live runner deferred to v0.45.1.
Reviews: python-review, security-review, code-review, tdd-guide all run;
verification-loop replaced by manual smoke (CLI happy + failure paths
exercised on real fixtures).
Test count: 5820 -> 5989 (+169). Test files: 164 -> 165. Ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
b4924b12a1 |
fix(v0.44.0): write_trigger lstat unresolved path so symlink at trigger is detected
trigger_path() returns realpath-resolved path, which FOLLOWS symlinks. write_trigger then lstat'd the resolved path, which is the symlink's TARGET (a regular file), so S_ISLNK was always False — the symlink rejection silently no-op'd. Linux CI caught this: test_write_trigger_rejects_pre_existing_symlink failed with "DID NOT RAISE OSError". Fix: lstat the unresolved os.path.join(output_dir, TRIGGER_FILENAME) before opening. If it's a symlink, raise OSError without following. Windows: test still skipped (no POSIX symlink permissions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
c4ac3da695 |
feat(v0.44.0): Live Dashboard & UX - 21 features, +192 tests
Part A - Live monitoring: soup monitor (nvidia-smi panel), EMA + p95/p99 tail-latency stats, SSE training-stream schema, phone-visible URL + ASCII-QR helper, llama-server timings parser + KV-cache bar, thread-safe ToolOutputsBuffer + ToolCallTimer. Part B - UX fixes: GracefulSaveHandler (first SIGINT saves, second stops), .checkpoint_now trigger file (cwd-contained, symlink-rejected), desktop / .command / .cmd shortcut builders, onboarding-wizard YAML renderer. Part C - UI tabs: drop-in soup_cli/ui/plugins/*.py registry with kebab-case name allowlist + 32-tab cap, API_HOST / API_PORT / API_KEY + GRADIO_HOST / GRADIO_PORT env knobs. Part D - Standalone CLIs: soup fetch (bundled examples + configs + deepspeed_configs catalog), soup quantize (ergonomic alias), soup merge-sharded-fsdp-weights, soup delinearize-llama4 (planners; live runtime in v0.44.1), soup llama <sub> (closed-allowlist proxy with filtered child env that drops HF_TOKEN / OPENAI_API_KEY / ANTHROPIC_API_KEY), soup_cli.utils.sweep_config (separate sweep.yaml loader), reasoning_parser allowlist for soup serve. Security review fixes: fetch symlink-at-target rejection + bundled-source commonpath check, write_trigger symlink rejection (TOCTOU), llama child-env secret allowlist, onboarding output cwd-containment, qr token moved from URL fragment to query string (so server actually sees it), sweep-config scalar allowlist + MappingProxyType[Tuple] immutability. Code/Python review fixes: detect_apple_silicon clean rewrite (was buggy parser-priority ternary), all frozen-dataclass List fields -> Tuple, ToolOutputsBuffer -> collections.deque(maxlen=1000), os.path.realpath over abspath, IPv6 host auto-bracketing per RFC 3986, frozenset over mutable set, type hints on __exit__/_make_proxy. Test count: 5628 -> 5820 (+192). Lint clean. Help output ASCII-safe (em-dash check enforced by test_cli_subprocess). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
82d5693b75 |
feat(eval): Tracker & Eval Pro — 18 features (v0.43.0)
Closes the observability gap with all three competitors in one release.
Part A — Trackers
* --tracker flag (mlflow/swanlab/trackio) on soup train, mutually
exclusive with --wandb/--tensorboard. Closed allowlist via
MappingProxyType. Live integrations rely on HF Trainer's report_to.
* SOUP_TELEMETRY=1 opt-IN env var; build_telemetry_payload schema is
closed-key (no model names / dataset paths / config contents). Live
PostHog network code deferred to v0.43.1.
Part B — Eval metrics
* Pure-Python BLEU + ROUGE-1/2/L + effective_tokens_per_second.
* KL-divergence calibration framework with OK/MINOR/MAJOR thresholds.
* Model Arena Elo tournament (256-model cap, MappingProxyType view,
Rich-markup metacharacter rejection on names).
* ceval / cmmlu / aider_polyglot benchmark allowlist (live Aider
runner deferred to v0.43.1).
Part C — Profiling
* memory_snapshot_context (narrow RuntimeError catch — review fix
prevents generator-already-executing on user-body RuntimeError).
* detect_anomaly_context, nccl_bandwidth_check (h100/a100/v100/rtx
reference table; live measurement CLI surface deferred).
* write_vscode_launch with TOCTOU symlink rejection at the target
path regardless of force=True.
Part D — Demo bundles
* `soup data demo` lists / copies 4 bundled JSONL fixtures
(alpaca / sharegpt / dpo / grpo) with staged-tempfile atomic
rename + 50 MB cap + symlink rejection on the staging path.
Tests: 5389 -> 5628 (+239). Ruff clean. Five sequential review waves
(python / code / security / tdd / smoke) ran; HIGH/MEDIUM/LOW findings
all fixed including: tracker name shadow in train.py, _lcs_length DP
double-buffer bug, BLEU geo-mean policy, base_dir absolute/.. escape,
demo_bundles tmp symlink TOCTOU, vscode launch symlink TOCTOU.
Note (Windows CI): line-ending warnings (LF -> CRLF) on commit are
expected; `.gitattributes` policy is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
05093ebfdb |
feat(data): Data Pipeline Pro — 18 features, axolotl + LF parity (v0.42.0)
Schema-first surface for the data pipeline gap with Axolotl + LlamaFactory. Ships in one release: 5 new formats (prm, pre_tokenized, input_output, video, multimodal), remote URI allowlist (s3/gs/gcs/az/abfs/abfss/oci) + streaming + sharding, AOT preprocess cache + `soup data preprocess` CLI, multi-dataset interleave (concat/under/over/probs) + 8 advanced masking fields, vocab expansion (add_new_tokens / new_special_tokens / resize_vocab) + custom prompt_strategy, and document ingestion (`soup data ingest` for PDF/DOCX/MD/TXT). Live wiring for fsspec backends, AOT tokenize loop, custom prompt-strategy runtime, and PRM trainer integration is deferred to v0.42.1+ (stub-then-live pattern from v0.27.0 / v0.37.0 / v0.41.0). Schema gates fire at config load so misconfiguration fails fast. Security: full v0.42.0 hardening matrix — `_REMOTE_SCHEMES` MappingProxyType allowlist; bucket regex 1-63 chars per S3/GCS spec; userinfo / fragment / query-string rejection on remote URIs (query-string forwarded to fsspec is SSRF-adjacent); null-byte + length caps on every string-shaped input; bool-rejected-before-int on every numeric input; frozen InterleaveSpec dataclass; 10k caps on add_new_tokens; `is_under_cwd` containment on video_dir + tokenized_path schema fields and on preprocess --config / both ingest paths; `os.lstat + S_ISLNK` symlink rejection on ingest input; PRM converter type-checks completions (str) + labels (bool, not int); video field null-byte + 2KB cap; field-name threading on image-pixels validator so error messages name the actual field. 5242 tests → 5389 (+147 net). 11 review findings addressed across python-review / code-review / security-review / tdd-guide (CRITICAL→LOW). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1491025a36 |
feat(trainer): Optimizer & PEFT Zoo (v0.41.0)
Optimizer Zoo (Part A): closed-allowlist SUPPORTED_OPTIMIZERS adds 14 new
entries — BAdam, APOLLO (apollo_adamw), Adam-mini, lomo / adalomo,
grokadamw, schedule_free_adamw / schedule_free_sgd, muon, dion,
came_pytorch, ao_adamw_{fp8,4bit,8bit}. Validates name + lower-cases
deterministically; rejects non-string / empty / null-byte / >64-char.
_OPTIMIZER_PACKAGES wrapped in MappingProxyType (matches v0.36.0 _REGISTRY).
Per-module LR (Part B): training.lr_groups accepts list-of-pairs /
list-of-dicts / {pattern: lr} mapping; canonical [{pattern, lr}, ...]
storage. Capped at MAX_LR_GROUPS=32; per-pattern non-empty string
≤256 chars + null-byte rejection + re.compile + best-effort ReDoS probe;
per-LR (0.0, 1.0] + math.isfinite (rejects NaN AND ±inf) + bool rejection
(matches v0.30.0 Candidate policy); duplicates rejected. lr_groups_from_schema
bridges canonical schema shape into runtime List[LrGroup] for
build_optimizer_param_groups (first-match-wins routing). LrGroup is
@dataclass(frozen=True). PyYAML scientific-notation (1e-4) parses as
string in YAML 1.1; _validate_lr coerces str → float so soup.yaml
round-trips work. base_lr rejects bool / non-positive (defence-in-depth).
PEFT methods (Part C): LoraConfig.init_strategy="loftq" + loftq_iter
∈ [1, 10] + loftq_bits ∈ {2, 4, 8}; cross-validator rejects loftq +
use_dora / use_vera. utils/loftq_init.py exposes validators +
build_loftq_config (lazy peft.LoftQConfig with actionable ImportError
hint). LLaMA Pro: TrainingConfig.expand_layers ∈ [1, 64] +
freeze_trainable_layers (signed, |x| ≤ 1000); cross-validator requires
the pair (LLaMA Pro freezes original layers and trains only new blocks).
field_validator(mode="before") on both rejects bool BEFORE Pydantic ge/le
silently coerces True → 1. expand_model_blocks raises NotImplementedError
with v0.41.1 marker — schema-only release (mirrors v0.27.0 / v0.37.0
stub-then-live pattern). utils/block_expansion._count_layers uses
hasattr(__len__) instead of try/except TypeError so legitimate __len__
bugs surface loudly. use_mod boolean for Mixture-of-Depths (schema only —
live patch deferred to v0.41.1).
Friendly aliases: load_in_8bit / load_in_16bit (Optional[bool]) for
LlamaFactory / Axolotl users. is True policy on both — explicit False is
"no preference", not "off"; mutually-exclusive both-True rejected; alias
combined with explicit Quant Menu format raises rather than silently
overriding. Alias-driven quantization rewrite via direct
self.quantization = ... (Pydantic v2 BaseModel non-frozen path), NOT
object.__setattr__ — code review caught that the latter would silently
bypass any future field_validator on quantization.
Five review agents (4 in parallel + manual smoke for verification-loop):
all CRITICAL/HIGH/MEDIUM/LOW findings fixed. Local smoke caught a real
bug — PyYAML parsing 1e-4 as string — fixed pre-commit with 2 added tests.
5242 tests pass (+120 net new); ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
5e0872b9ea |
feat(trainer): ReLoRA + surgical PEFT non-SFT (v0.40.6 #67)
Extends the v0.39.0 ReLoRA callback (Part B) and surgical PEFT patches (Part D — Gemma4 ClippableLinear swap + 3-D fused-MoE expert dropout strip) from SFT-only to all 11 non-SFT transformer-backend trainers (DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, RewardModel, Pretrain, Embedding, BCO). - New shared helper soup_cli/utils/peft_wiring.py exposes apply_pre_lora_patches, apply_post_lora_patches, attach_relora_callback. - SFT migrated to the same helpers in the same release (centralisation invariant; no drift between SFT and non-SFT wiring). - SoupConfig._validate_relora_supported_tasks: task != "sft" rejection removed; MLX backend still rejected with distinct message. Review fixes: - attach_relora_callback uses `if relora_steps is None:` (project policy) so a schema-bypassing relora_steps=0 surfaces as a loud ReLoRAPolicy ValueError rather than a silent skip. - Direct attribute access on tcfg.relora_warmup_ratio / _reset_optimizer / _prune_ratio (Pydantic schema guarantees them); no getattr defaults. - 11 behavioural helper tests (Gemma4 happy path + exception swallow, post-LoRA strip happy + exception swallow, ReLoRA policy field forwarding, schema-bypass loud-fail). - Schema-gate matrix covers `task='preference'` dispatcher. Tests: 5061 -> 5122 (+61). Closes #67. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
30fb8d61f8 |
docs(contributing): correct test file count (149 -> 147)
Inventory audit showed v0.40.4 baseline test-file count was 148 in docs but 146 actually on disk. v0.40.5 added test_v0405_part_a.py (+1), so the correct post-v0.40.5 count is 147. Update CONTRIBUTING.md. Docs-only hotfix — no soup_cli/ changes, no version bump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
0d406467a1 |
docs(readme): refresh Quant Menu section for v0.40.5 multi-trainer wiring
The README's `## Quant Menu` body still said "v0.38.0 scope — wired into the SFT trainer + transformers backend. Multi-trainer expansion is tracked for v0.38.1." That stub-then-live note is now stale: v0.40.5 (#66) shipped the multi-trainer wiring across all 11 non-SFT trainers + the PPO reward model. Replace with the post-v0.40.5 status. Docs-only hotfix — no soup_cli/ changes, no version bump per CLAUDE.md "CI-only / docs-only hotfixes" policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |