diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e669c2e..6398338 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (188 files, 7998 tests) +tests/ - Test suite (189 files, 8051 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index 3d021d7..4015005 100644 --- a/README.md +++ b/README.md @@ -43,15 +43,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.53.5 — Adaptive Training (BETA → stable)**: Six closes lifting the v0.48.0 BETA deferrals to live wiring, plus a day-zero recipe for DeepSeek-V3 reasoning. +**v0.53.6 — Plugin + Agent + Anthropic API**: Six features — three live, three deferred-live stubs with v0.53.7 markers. -- **Dynamic curriculum is live.** New `DynamicCurriculumCallback` — a real HF `TrainerCallback` that accumulates per-step loss + grad-norm per bucket, recomputes sampler weights every N steps, and atomically appends `curriculum_history.jsonl` rows on rank 0. Multi-rank launches are coordinated via `torch.distributed.all_reduce(SUM)` of per-bucket stats before the recompute. Visualise the trace with `soup runs curriculum-curve `. -- **Curriculum-aware on every transformer trainer.** The `curriculum_dynamic: true` schema gate widened from `{sft, pretrain}` to every transformer-backend wrapper (SFT / Pretrain / DPO / GRPO / KTO / ORPO / SimPO / IPO / BCO / RewardModel / Embedding / PPO / Distill). MLX backend still rejected with a distinct error message. -- **`soup data mix --live` runs real proxy trainings.** A new `--live --base-yaml ` mode replaces the synthetic offline proxy with a real short `soup train` subprocess per Bayesian-search candidate. Argv-list invocation, per-candidate timeout `min(budget/num_probes, 30 min)`, tracker-SQLite parse for `eval_loss`, atomic tmp-YAML cleanup. -- **scikit-optimize behind the OptimizerProtocol.** When `scikit-optimize` is installed, the mix optimiser now drives a real `skopt.Optimizer(GP)` (Gaussian-process Bayesian optimisation) instead of the Dirichlet fallback. Zero new required dependencies — falls back silently when skopt is absent. -- **`MixOptimizationReport.elapsed_seconds` excludes failed candidates.** The headline elapsed-time field now sums only successful candidate wall-clock, so a single timed-out proxy can no longer inflate a report otherwise filled with quick successes. Per-candidate `wall_clock_seconds` retains the per-trial timing. -- **`deepseek-v3-reasoning` recipe.** GRPO + reasoning template on `deepseek-ai/DeepSeek-V3` with `reward_fn=accuracy,format` + math verifiable domain — day-zero coverage for the new MoE reasoning base. -- **+63 net new tests** (7935 → 7998) across the new `test_v0535.py`. Four review agents (python / code / security / tdd) ran; every CRITICAL → LOW finding was fixed — `is None` over falsy guards on the callback attach helper, cwd-containment + `os.lstat + S_ISLNK` rejection on `output_dir`, simplex + finite + bool-rejected validation on weights, argv-list subprocess invocation with no shell, atomic tempfile + `os.replace` for `curriculum_history.jsonl` appends. +- **Soup plugins now run live as Trainer callbacks.** New `SoupPluginCallback` dispatches `pre_train` / `post_train` / `pre_step` / `post_step` to every enabled plugin via the v0.45.0 registry. Hook exceptions are swallowed at WARNING — one misbehaving plugin must never crash a multi-hour run. Wired into all 13 transformer-backend trainers (SFT + DPO + GRPO + KTO + ORPO + SimPO + IPO + BCO + PPO + RewardModel + Pretrain + Embedding + Distill) via `attach_plugin_callback`. +- **Anthropic-shaped `/v1/messages` endpoint.** `soup serve --backend transformers` exposes a `POST /v1/messages` route reusing the v0.45.0 `anthropic_messages` converter + the existing chat handler. Validation errors return a generic `"Invalid request"` 400 (details logged server-side at DEBUG). Streaming returns 501 — true Anthropic event-shape SSE ships in v0.53.7. +- **n-gram speculative decoding wired through.** When the server is started with an `NgramSpecConfig`, every chat completion forwards `prompt_lookup_num_tokens=N` into `model.generate(...)` (HF Transformers ≥ 4.38 prompt-lookup decoding). Mutually exclusive with a real draft `assistant_model`. +- **`soup data recipe --execute` lands as a stub-then-live CLI surface.** `--execute --output ` validates the DAG, enforces cwd-containment on `--output` at the CLI boundary, then surfaces the `v0.53.7` `NotImplementedError` marker from `run_recipe`. Empty-string `--output ""` and outside-cwd paths are rejected before the runner is even called — never want the live runner to be the first/only enforcement point. +- **Server-side tool endpoints (`/v1/tools/python` + `/v1/tools/bash` + `/v1/tools/web_search`).** URL schema lives now so clients can target the endpoints. All three return HTTP 501 with the v0.53.7 marker — live RLVR sandbox HTTP wrapper + `WebSearchConfig.domain_allowlist` enforcement ship in v0.53.7. +- **`instantiate_trainer_plugins` schema-only.** Validates `[grokfast, spectrum, llmcompressor, sonicmoe, cce_plugin, math_verify]` lists then raises with the v0.53.7 marker. Six upstream plugin lazy-imports + per-plugin callback construction land in v0.53.7. +- **+53 net new tests** (7998 → 8051) across the new `test_v0536.py`. Three review agents (python / code / security) ran; every HIGH and MEDIUM finding was fixed — race-free single-snapshot plugin hook collection, generic `"Invalid request"` body redaction, CLI-side cwd containment before `run_recipe`, `is None` over falsy `--output` guards, and added test coverage for: `prompt_lookup_num_tokens` omitted when `ngram_config=None`, console-print failure swallow on the attach helper, every `run_recipe` type-rejection boundary. ## Why Soup? @@ -3787,6 +3787,73 @@ The advanced GGUF pipeline uses POSIX `O_NOFOLLOW` to defeat the TOCTOU race bet `soup deploy autopilot --measure` caches results at `~/.soup/deploy_autopilot_cache.json` keyed on `(base, profile, eval-tasks)`. Repeat invocations short-circuit; pass `SOUP_DEPLOY_AUTOPILOT_CACHE=` to redirect (constrained to home / cwd / tempdir). The recommended candidate uses soft-fallback: first `OK` by insertion order, else the candidate with the smallest delta (least drop relative to its own baseline). +## Soup Plugin Callbacks + +Register a plugin once via the v0.45.0 registry API; v0.53.6 wires it into every +transformer-backend trainer as a real HF `TrainerCallback`: + +```python +# soup_cli/plugins/my_plugin.py — auto-discovered at `soup` startup +from soup_cli.plugins import register_plugin + +class MyPlugin: + def pre_train(self, ctx): + print("training about to start, args =", ctx["args"]) + + def post_step(self, ctx): + if ctx["state"].global_step % 100 == 0: + print(f"step {ctx['state'].global_step}") + +register_plugin(name="my-plugin", version="0.1.0", plugin=MyPlugin()) +``` + +A misbehaving plugin hook is swallowed at WARNING — one bad plugin must never crash +a multi-hour training run. The hook snapshot is taken at callback-construction time, +so a plugin registered MID-run does not retroactively receive events. + +## Anthropic `/v1/messages` API + +`soup serve --backend transformers` exposes a POST `/v1/messages` route that accepts +Anthropic Messages-shaped payloads: + +```bash +curl http://localhost:8000/v1/messages -H "Content-Type: application/json" -d '{ + "model": "my-model", + "messages": [{"role": "user", "content": "hello"}], + "max_tokens": 64 +}' +``` + +Streaming (`stream: true`) returns 501 — Anthropic event-shape SSE ships in v0.53.7. +Validation errors return a generic `"Invalid request"` 400 body; details are logged +server-side at DEBUG. vLLM parity tracked for v0.53.7. + +## N-gram Speculative Decoding + +When a server is configured with an `NgramSpecConfig`, every chat completion forwards +`prompt_lookup_num_tokens=N` into `model.generate(...)` (HF Transformers ≥ 4.38 +prompt-lookup decoding — no draft model required). Mutually exclusive with a real +`assistant_model`; if both are set, the real draft model wins. + +## Server-Side Tool Endpoints (preview) + +Three POST routes ship in v0.53.6 as schema-only stubs returning HTTP 501: + +- `/v1/tools/python` — sandboxed Python execution (v0.53.7 live) +- `/v1/tools/bash` — sandboxed bash (v0.53.7 live) +- `/v1/tools/web_search` — domain-allowlisted web search (v0.53.7 live) + +Live wiring re-uses the v0.25.0 RLVR sandbox for python/bash and enforces +`WebSearchConfig.domain_allowlist` for web_search. + +## Data Recipe DAG Runner (preview) + +`soup data recipe path/to/recipe.yaml --execute --output ./out` validates the DAG, +enforces cwd-containment on `--output` at the CLI boundary, and surfaces the +`v0.53.7` `NotImplementedError` marker from `run_recipe`. The per-node execution +loop (seed / llm_text / code / judge / validator / sampler) plus checkpoint/resume +ships in v0.53.7. Validate today, run tomorrow. + ## Changelog See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history. diff --git a/SECURITY.md b/SECURITY.md index 06355f6..9bbfaa8 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.53.5 -- Full support (latest) +- v0.53.6 -- Full support (latest) +- v0.53.5 -- Full support - v0.53.4 -- Full support - v0.53.3 -- Full support - v0.53.2 -- Full support @@ -151,6 +152,8 @@ No known critical vulnerabilities in current releases. - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input. - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS` +- **v0.53.6 — Plugin + Agent + Anthropic API**: 6 features — 3 live, 3 deferred-live stubs. (#101 SoupPluginCallback live) New `soup_cli/monitoring/plugin_callback.py` ships `SoupPluginCallback(transformers.TrainerCallback)` that dispatches the four canonical hook events to every enabled plugin in the v0.45.0 registry. Per-hook invocation runs through `_safe_invoke` which swallows ALL exceptions at WARNING with `exc_info=True` — defends against one misbehaving plugin crashing a multi-hour training run (mirrors v0.44.0 / v0.45.0 plugin-loader policy). Hook snapshot is collected ONCE in `build_plugin_callback` and passed to the constructor so a plugin registered between the "is empty?" check and ctor cannot silently slip into the active hook list (code-review HIGH fix — race-window closed). `attach_plugin_callback(trainer, console=None) -> bool` short-circuits to False when no plugins are enabled (zero overhead on the hot path); inner `add_callback` failure is swallowed at DEBUG so a plugin-infrastructure bug never crashes training. Wired into all 13 transformer-backend trainers; source-level grep regression test parametrized over every trainer file. (#102 Anthropic `/v1/messages` live) New POST `/v1/messages` route reuses the v0.45.0 SSRF-hardened `from_anthropic` converter + the existing chat handler. Streaming requests return 501 BEFORE schema validation (defence-in-depth — defends against stream-only attackers leaking validator-error detail). All validation paths (`validate_anthropic_payload` / `from_anthropic` / `ChatCompletionRequest(**openai_payload)`) are wrapped to map to a generic `"Invalid request"` 400 body; detailed exception text is logged server-side at DEBUG (security-review MEDIUM fix — matches `utils/errors.py` policy "HTTP error responses return generic messages, details logged server-side"). The existing loopback-CORS restriction from v0.30.0 Part C and the trace-log-redaction from v0.40.3 #33 both apply to the new route. (#104 n-gram speculative decoding live) `_generate_response` gains `ngram_config: Any = None` kwarg; when set, emits `prompt_lookup_num_tokens=int(ngram_config.num_draft_tokens)` into `model.generate(...)`. The int cast is wrapped in `try/except (TypeError, AttributeError)` as defence-in-depth; primary validation lives upstream in v0.45.0 `validate_ngram_config` (bool-rejected, bounded `[1, 32]`). Mutually exclusive with `assistant_model` — real draft model wins. (#103 server-side tool endpoints STUB) Three new POST routes `/v1/tools/python` / `/v1/tools/bash` / `/v1/tools/web_search` return 501 with v0.53.7 marker. Tool names are hardcoded string literals (not user-controlled); no payload inspection. Live RLVR `code_exec` sandbox HTTP wrapper (python / bash) + `WebSearchConfig.domain_allowlist` enforcement (web_search) land in v0.53.7. (#106 `run_recipe` STUB + `soup data recipe --execute`) New `utils/recipe_run.run_recipe(dag, *, output_dir, ...)` type-checks every parameter (TypeError on non-`RecipeDAG` / non-string / empty `output_dir` / non-bool `resume` / non-string `judge_*`) BEFORE the `NotImplementedError("v0.53.7")`. CLI `--execute --output ` enforces `is None` guard on `--output` (project policy since v0.40.6 — empty-string is a distinct operator error, NOT silent missing) + `is_under_cwd` containment at the CLI boundary BEFORE the `run_recipe` call (security-review MEDIUM fix — defends against future v0.53.7 live runner bypass; never want the live runner to be the first/only enforcement point per project policy). Regression test asserts the v0.53.7 marker does NOT surface on outside-cwd paths — guards against future live runner accidentally bypassing CLI containment. (#105 `instantiate_trainer_plugins` STUB) Validates name list through existing `validate_trainer_plugin_list` (closed allowlist, dedup, ≤8/run, canonicalisation — all v0.45.0 hardening) BEFORE raising `NotImplementedError`. Canonical names echoed in the error message are allowlist-validated lowercase strings (no injection surface). Test surface: 1 new test file (`tests/test_v0536.py`) carrying 53 new tests covering: per-hook exception swallow with caplog isolation, source-level grep parametrized over 13 trainer files (canonical import + invocation), generic `"Invalid request"` body redaction with detail logged at DEBUG, oversize `max_tokens` cap, n-gram kwarg omitted when config is None (regression guard against future "always emit" refactor), `run_recipe` type-rejection matrix (non-DAG / empty output_dir / non-str checkpoint_dir / non-str judge_provider / non-str judge_model / bool resume), `--execute --output ""` rejected with `"must be a non-empty path"`, outside-cwd `--output` rejected with `v0.53.7` marker NEVER surfacing on rejected paths. Known limitations: (1) vLLM `/v1/messages` parity deferred to v0.53.7 — Anthropic route is transformers-only. (2) Streaming `/v1/messages` returns 501 — true SSE event-shape ships in v0.53.7. (3) Plugin hook snapshot is at callback-construction time — a plugin registered mid-run does not retroactively receive hooks (by design — prevents partial-run inconsistencies). (4) n-gram is transformers-only — vLLM ships its own prompt-lookup path that needs a different kwarg. (5) Tool endpoints + recipe runner + trainer-plugin instantiation are stubs — live wiring in v0.53.7. (v0.53.6) + - **v0.53.5 — Adaptive Training (BETA → stable)**: six closes lifting the v0.48.0 BETA deferrals to live wiring. (#114 DynamicCurriculumCallback) new `soup_cli/monitoring/curriculum_callback.py` accepts `output_dir` through `is_under_cwd` containment + null-byte / oversize / non-string rejection (matches v0.40.5 `reward_model` policy) BEFORE any filesystem touch; per-step bucket-stats appended to `curriculum_history.jsonl` via `tempfile.mkstemp` + `os.replace` atomic write (mirrors v0.48.0 `write_mix_recipe` policy) so a crash mid-write cannot leave a half-row at the target. Rank-0 guard via `_is_rank_zero` helper — defends against multi-rank double-write on shared filesystems. Multi-rank coordination via `torch.distributed.all_reduce(SUM)` of per-bucket stats BEFORE `compute_bucket_weights` invocation — defends against per-rank divergent samplers (the DDP footgun the v0.48.0 schema gate explicitly warned about). (#115 multi-trainer expansion) `_validate_curriculum_dynamic_supported` cross-validator widened from `{sft, pretrain}` to every transformer-backend trainer; MLX backend still rejected with a distinct error message (matches v0.34.0 review-fix policy). New `attach_curriculum_callback(trainer, tcfg, output_dir, console=None) -> bool` shared helper follows the v0.40.6 `attach_relora_callback` pattern with `is None` guard on `tcfg.curriculum_dynamic` — defends against the schema-bypass footgun where `curriculum_dynamic=0` (falsy but explicit) would silently no-op a deliberate disable. (#116 `soup data mix --live`) new `soup_cli/utils/mix_proxy.py::proxy_run_for_weights` — argv-list `subprocess.run([sys.executable, "-m", "soup_cli.cli", "train", ...], timeout=…)` with NO shell (defends against shell injection via crafted dataset path / base-yaml path), tmp YAML staged under `tempfile.mkdtemp(prefix=".soup_mix_proxy.")` then cleaned up in `try/finally`. Weights validated through simplex (sum-to-1 ± 1e-6), finite (`math.isfinite`), `bool`-rejected, per-element `[0, 1]` bounds (matches v0.30.0 `Candidate` / v0.48.0 `MixCandidate` policy). `base_yaml_path` runs through `is_under_cwd` containment + null-byte rejection BEFORE the tmp YAML is rendered. `timeout_seconds ∈ [60, 30*60]` Pydantic-style bounds. Tracker-SQLite read uses existing v0.34.0 `ExperimentTracker.get_run` policy (LIKE escape, parameterised SQL). `SOUP_DB_PATH` env-override propagated via subprocess `env=` so per-candidate DB isolation is possible. (#117 skopt OptimizerProtocol) lazy `import skopt` — when the optional dep is absent, `_build_default_optimizer` falls back silently to the v0.48.0 Dirichlet sampler (mirrors v0.43.0 Part A tracker-package policy). The skopt path drives `skopt.Optimizer(GP)` through the existing v0.48.0 `OptimizerProtocol` — no API surface change. (#118 `MixOptimizationReport.elapsed_seconds`) headline elapsed now sums only successful-candidate wall-clock; failed-proxy time excluded. No security-surface change — observability fix only. Test surface: 1 new test file (`tests/test_v0535.py`) carrying 57 new tests covering cwd containment, symlink rejection, simplex + finite + bool validation, argv-list subprocess shape, atomic JSONL append, idempotent attach helper, MLX-rejection cross-validator, multi-trainer task gate. Known limitations: (1) `_pick_bucket` is step-mod round-robin (BETA); loss-percentile / curriculum-metric routing tracked for a follow-up patch. (2) `validate_distributed_curriculum` helper still requires callers to attest `rank_coordinated=True` for external invocations; the new callback wires `all_reduce` internally so the schema invocation passes. (3) `proxy_run_for_weights` is single-shot — concurrent proxy runs would race on `~/.soup/experiments.db`; the `SOUP_DB_PATH` env override is the per-candidate isolation hatch. (4) `scikit-optimize` is an optional dep, not bundled — silent fallback to Dirichlet sampler when absent. (5) `MixOptimizationReport.elapsed_seconds` is a behaviour change — operators printing the report's headline elapsed must note it now excludes failed-candidate time. (v0.53.5) - **v0.53.4 — Long Context + Architecture**: six closes covering LongLoRA hardening, LLaMA Pro live wiring, and a CUDA-OOM-hint UX upgrade. (#11 OOM hint) `format_friendly_error` upgrades the CUDA-OOM and `OutOfMemoryError` patterns to point users at the explicit `--batch-size ` / `--grad-accum ` CLI flags before the legacy `quantization: 4bit` fallback — closes #11 with no functional change to the security surface. (#122 FlashAttention v3 incompatibility) New `soup_cli/utils/flash_attn.is_flash_attn_v3_available() -> bool` is a defensive probe (never raises, False on missing `flash_attn` / non-string `__version__` / unparseable / major < 3). `validate_longlora_compat` calls it AFTER the existing task / backend / architecture / ring-attention checks so the FA-v3 error only surfaces on otherwise-valid LongLoRA configs (avoids spurious confusion on unrelated misconfig). The check is loaded via a function-scoped import to keep `validate_longlora_compat` import-cheap and avoid CUDA-side effects at config load time on machines without `flash_attn` installed. (#120 LongLoRA arch allowlist) `soup_cli/utils/longlora.py` ships three new word-boundary regex helpers (`is_mistral_model`, `is_qwen_model`, `is_phi_model`) — same regex policy as v0.39.0 `is_gemma4_model` (rejects substring matches like `"my-mistralish-finetune"` or `"unmistral-7b"`). Shared `_check_model_name` input guard rejects `bool` BEFORE the `isinstance(str)` check (because bool is a subclass of int and would otherwise fall through silently — matches v0.53.3 `is_known_vlm_base` policy), rejects null bytes via explicit substring check, and returns `None` (→ helper returns False) for inputs >512 chars (avoids ReDoS-style overhead on adversarial input). New `is_supported_longlora_arch(model_name: object) -> bool` is the union accessor with defensive non-string surface (returns False rather than propagating TypeError, matches v0.53.3 / v0.52.0 model-detection policy). `validate_longlora_compat` also gained per-call null-byte rejection + bool/non-string TypeError on `task` and `backend` (matches v0.50.0 `validate_long_context_grpo_compat`); new `_truncate_for_message(value, limit=64)` helper bounds the `base` echo in error messages (security-review MEDIUM fix mirroring v0.53.3 `validate_vision_grpo_compat` redaction — defends against adversarial / long bases bloating stderr + log files). Mixtral is INTENTIONALLY excluded from the allowlist — regex matches `mistral` as a word-boundary token, NOT `mixtral`; documented at the docstring so a future contributor adding Mixtral support adds it explicitly. (#121 Llama 3.1 RoPE auto-detect) `apply_long_context_config` extended with `rope_scaling_type=None` auto-detect path — reads `model_config.rope_scaling` and runs `detect_llama3_rope_in_config` (v0.49.0 Part D helper) on it. If the existing block declares `llama3` (either via the legacy `type` key OR the newer `rope_type` alias), the auto-detect picks `"llama3"` + the upstream `LLAMA3_DEFAULT_*` constants; otherwise falls back to `"dynamic"`. Explicit caller pick still wins (any non-None value). Back-compat preserved by keeping the legacy default kwarg `rope_scaling_type="dynamic"`. The detect helper rejects non-Mapping config input via `TypeError` (no SSRF / file-read risk — the function is pure-Python data inspection). (#83 LLaMA Pro live block expansion) `soup_cli/utils/block_expansion.expand_model_blocks` lifts the v0.41.0 Part C `NotImplementedError` stub with a real implementation: clones the last `min(num_new_blocks, original_count)` decoder blocks via `copy.deepcopy` (full independent storage — no shared buffers), zero-inits each clone's residual projections (`mlp.down_proj.weight + bias` and `self_attn.o_proj.weight + bias`) so the appended block initially acts as identity per the LLaMA Pro paper §3.1, appends to `model.model.layers`, and updates `model.config.num_hidden_layers`. Validates `num_new_blocks` via `validate_expand_layers` (bool-guard + `[1, 64]`) BEFORE any model mutation. `_get_layers_module` uses explicit `is None` check (not falsy shortcut) to defend against `nn.Module.__bool__` overrides on subclasses (code-review HIGH fix). `_zero_init_block_residual` returns `bool` and the caller emits `warnings.warn` when neither standard projection path matches the cloned block (non-Llama-shaped arch — security-review LOW fix surfaces silent-degradation to operators training on Falcon-style models). Over-expansion silently clamps to `min(n, original_count)` rather than raising — matches the project's defensive-fallback policy for advisory operations. New `apply_llama_pro_freeze(model, num_new_blocks) -> int` is the canonical "train only new blocks" companion (global `requires_grad=False` pass, then unfreeze the tail N blocks; returns trainable parameter count). New shared helper `apply_block_expansion_if_configured(model, tcfg, console)` centralises the "if `expand_layers` is set, expand + optionally freeze + print" sequence — used identically by SFT and Pretrain trainers (matches v0.40.6 `peft_wiring` centralisation policy; defends against drift between trainer call sites which would otherwise produce subtle inconsistent behaviour). (#74 HF push surface QA) Manual QA of `soup push`, `soup train --push-as`, `soup data push`, `soup deploy hf-space` deferred to a contributor with private HF credentials — entry recorded in `tests/qa/v053_qa.md` with the full test plan + acceptance criteria. The HF push security surface (repo_id validation, token resolution, commit message sanitization, model card injection defence, Space template containment) is unchanged from v0.29.0 / v0.40.2 and remains covered by `test_hf_integration.py` + `test_v0402_part_a.py`. Test surface: 1 new test file (`tests/test_v0534.py`) carrying 49 new tests + 7 net updates to v0.49.0 / v0.41.0 / v0.10.x regression tests. Known limitations: (1) LongLoRA S² forward override still deferred to v0.49.1 — schema gate hardened, live monkeypatch is the next deliverable. (2) Mixtral excluded from LongLoRA allowlist (MoE attention forward signature differs). (3) Block-expansion zero-init covers Llama-shaped blocks only — non-standard arches still get appended + trainable, but lose the LLaMA Pro identity-init guarantee (and emit a runtime warning). (4) Llama 3.1 RoPE auto-detect only fires when caller passes `rope_scaling_type=None` (explicit pick wins). (5) #74 live QA against a private HF repo is the v0.53.5+ follow-up. (v0.53.4) diff --git a/pyproject.toml b/pyproject.toml index 1f9feb7..20de7ee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.53.5" +version = "0.53.6" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index ed94e81..30dc3f7 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.53.5" +__version__ = "0.53.6" diff --git a/soup_cli/commands/data.py b/soup_cli/commands/data.py index c770d9a..b57e855 100644 --- a/soup_cli/commands/data.py +++ b/soup_cli/commands/data.py @@ -2154,6 +2154,20 @@ def demo_bundle( @app.command(name="recipe") def recipe( path: str = typer.Argument(..., help="Path to recipe.yaml under cwd"), + execute: bool = typer.Option( + False, + "--execute", + help=( + "Run the validated DAG end-to-end (v0.53.6 #106 — stub; " + "live per-node execution deferred to v0.53.7)." + ), + ), + output: Optional[str] = typer.Option( + None, + "--output", + "-o", + help="Output dir for sampler node (required with --execute).", + ), ) -> None: """v0.45.0 Part E — Validate a Data Recipe DAG (live runner deferred).""" from rich.markup import escape as _escape @@ -2177,6 +2191,39 @@ def recipe( "Topological order: " + ", ".join(_escape(name) for name in dag.topo_order) ) + + if execute: + # `is None` guard — empty-string `--output ""` is a distinct + # operator error that should NOT be silently mapped to "missing" + # (matches v0.40.6 project policy on `is None` over falsy). + if output is None: + console.print( + "[red]--execute requires --output [/]" + ) + raise typer.Exit(2) + # Defence-in-depth: enforce cwd containment at the CLI boundary + # BEFORE handing off to run_recipe. Today run_recipe is a stub, + # so this only protects against future v0.53.7 live-runner bugs — + # the docstring contract on run_recipe.output_dir says + # cwd-contained, and we never want the live runner to be the + # first/only enforcement point. + from soup_cli.utils.paths import is_under_cwd + from soup_cli.utils.recipe_run import run_recipe + + if not output or not is_under_cwd(output): + console.print( + "[red]--output must be a non-empty path under the current directory[/]" + ) + raise typer.Exit(2) + + try: + run_recipe(dag, output_dir=output) + except NotImplementedError as exc: + console.print(f"[yellow]{_escape(str(exc))}[/]") + raise typer.Exit(2) from exc + return + console.print( - "[yellow]Live runner deferred to v0.45.1.[/]" + "[yellow]Live runner deferred to v0.53.7 " + "(re-run with --execute once v0.53.7 ships).[/]" ) diff --git a/soup_cli/commands/serve.py b/soup_cli/commands/serve.py index 9ee4b67..e1dbc1e 100644 --- a/soup_cli/commands/serve.py +++ b/soup_cli/commands/serve.py @@ -6,7 +6,7 @@ import re import time import uuid from pathlib import Path -from typing import Dict, List, Optional +from typing import Any, Dict, List, Optional import typer from rich.console import Console @@ -810,6 +810,7 @@ def _generate_response( assistant_model=None, num_assistant_tokens: int = 5, logits_processor=None, + ngram_config: Any = None, ): """Generate a response from the model.""" import torch @@ -854,6 +855,21 @@ def _generate_response( # v0.33.0 #53 — structured-output LogitsProcessor list (may be empty). if logits_processor: gen_kwargs["logits_processor"] = logits_processor + # v0.53.6 #104 — n-gram speculative decoding (transformers backend). + # Mutually exclusive with a real draft `assistant_model`. + if ngram_config is not None and assistant_model is None: + # HF Transformers >= 4.38 supports prompt-lookup decoding via + # `prompt_lookup_num_tokens`. We expose `num_draft_tokens` as + # the user-facing knob; n-gram size + prompt_lookup_max are + # validated upstream by `validate_ngram_config`. + try: + gen_kwargs["prompt_lookup_num_tokens"] = int( + ngram_config.num_draft_tokens + ) + except (TypeError, AttributeError): + # Schema gate at construction time enforces shape; this + # is defence-in-depth. + pass outputs = model.generate(**gen_kwargs) @@ -879,6 +895,7 @@ def _create_app( enable_dashboard: bool = False, tracer=None, trace_log_writer=None, + ngram_config: Any = None, ): """Create the FastAPI application with OpenAI-compatible endpoints.""" import threading as _threading @@ -1067,6 +1084,7 @@ def _create_app( assistant_model=draft_model, num_assistant_tokens=num_speculative_tokens, logits_processor=processors or None, + ngram_config=ngram_config, ) except Exception: logger.exception("Generation error") @@ -1121,6 +1139,89 @@ def _create_app( # error paths (prevents blind spots on the dashboard). metrics.record_latency((time.perf_counter() - started) * 1000) + # ----- v0.53.6 #102 — Anthropic /v1/messages route ----- + # Reuses the v0.45.0 utils/anthropic_messages converter + the existing + # chat_completions handler. Live on transformers backend only this + # release (vLLM /v1/messages tracked for v0.53.7). + @app.post("/v1/messages") + def anthropic_messages(payload: dict) -> dict: + from soup_cli.utils.anthropic_messages import ( + from_anthropic, + validate_anthropic_payload, + ) + + # Streaming not yet supported on this route — v0.53.7 deliverable. + # Checked BEFORE schema validation so a stream-only client never + # leaks a validation-error detail (defence-in-depth). + if isinstance(payload, dict) and payload.get("stream"): + raise HTTPException( + status_code=501, + detail="Streaming /v1/messages deferred to v0.53.7.", + ) + + try: + validate_anthropic_payload(payload) + openai_payload = from_anthropic(payload) + request = ChatCompletionRequest(**openai_payload) + except (TypeError, ValueError) as exc: + # Security: do not echo internal validator/converter details + # to the HTTP body. Log server-side for operator debugging. + logger.debug("/v1/messages invalid request: %s", exc) + raise HTTPException(status_code=400, detail="Invalid request") + except Exception as exc: # noqa: BLE001 — pydantic ValidationError shape + logger.debug("/v1/messages pydantic error: %s", exc) + raise HTTPException(status_code=400, detail="Invalid request") + + chat_response = chat_completions(request) + + # Map OpenAI chat response back to Anthropic shape. + text = "" + if isinstance(chat_response, dict): + try: + text = chat_response["choices"][0]["message"]["content"] + except (KeyError, IndexError, TypeError): + text = "" + usage = ( + chat_response.get("usage", {}) if isinstance(chat_response, dict) else {} + ) + + return { + "id": ( + chat_response.get("id", "") if isinstance(chat_response, dict) else "" + ), + "type": "message", + "role": "assistant", + "model": openai_payload.get("model", model_name), + "content": [{"type": "text", "text": text}], + "stop_reason": "end_turn", + "usage": { + "input_tokens": int(usage.get("prompt_tokens", 0) or 0), + "output_tokens": int(usage.get("completion_tokens", 0) or 0), + }, + } + + # ----- v0.53.6 #103 — Server-side tool endpoints (deferred-live stubs) ----- + # Closed allowlist (mirrors v0.45.0 Part B utils/server_tools.SUPPORTED_TOOLS). + # Routes return HTTP 501 with v0.53.7 marker — schema lives now so client + # code can target the URLs even before the live sandbox is wired. + def _tool_not_implemented(tool: str) -> None: + raise HTTPException( + status_code=501, + detail=f"Server-side tool {tool!r} live execution deferred to v0.53.7.", + ) + + @app.post("/v1/tools/python") + def tool_python(_: dict) -> None: + _tool_not_implemented("python") + + @app.post("/v1/tools/bash") + def tool_bash(_: dict) -> None: + _tool_not_implemented("bash") + + @app.post("/v1/tools/web_search") + def tool_web_search(_: dict) -> None: + _tool_not_implemented("web_search") + # Expose dashboard intent + constraint on the app for tests + introspection app.state.enable_dashboard = enable_dashboard app.state.output_constraint = output_constraint diff --git a/soup_cli/monitoring/plugin_callback.py b/soup_cli/monitoring/plugin_callback.py new file mode 100644 index 0000000..1715cc0 --- /dev/null +++ b/soup_cli/monitoring/plugin_callback.py @@ -0,0 +1,123 @@ +"""v0.53.6 #101 — Soup plugin TrainerCallback. + +Bridges :mod:`soup_cli.plugins` registered hooks into the HF Trainer +callback surface. For every enabled plugin (via :func:`list_plugins` + +``spec.enabled``), discovers implemented hooks via :func:`discover_hooks` +and dispatches the matching trainer event. + +Per-plugin hook exceptions are swallowed at WARNING level — one +misbehaving plugin must not crash a multi-hour training run. This +mirrors the v0.44.0 / v0.45.0 plugin loader policy. +""" + +from __future__ import annotations + +import logging +from typing import Any + +logger = logging.getLogger(__name__) + + +def _collect_active_hooks() -> list[tuple[str, dict[str, Any]]]: + """Return ``[(plugin_name, hook_map), ...]`` for every enabled plugin. + + Lazy-imports :mod:`soup_cli.plugins` so the callback module stays + cheap to import in CI / non-training contexts. + """ + from soup_cli.plugins import discover_hooks, list_plugins + + out: list[tuple[str, dict[str, Any]]] = [] + for name, spec in list_plugins().items(): + if not spec.enabled: + continue + hooks = discover_hooks(spec.plugin) + if hooks: + out.append((name, hooks)) + return out + + +def _safe_invoke( + plugin_name: str, hook_name: str, hook: Any, ctx: dict[str, Any] +) -> None: + try: + hook(ctx) + except Exception: # noqa: BLE001 — plugin failure must not crash training + logger.warning( + "Plugin %r hook %r raised; continuing", + plugin_name, + hook_name, + exc_info=True, + ) + + +def _build_callback_class() -> type: + """Construct the ``SoupPluginCallback`` class with transformers as parent. + + Lazy-imports :mod:`transformers` so the wiring helper can be imported + in CI without the heavy dep installed. + """ + from transformers import TrainerCallback + + class SoupPluginCallback(TrainerCallback): + """Fans HF Trainer events out to every enabled Soup plugin.""" + + def __init__( + self, hooks: list[tuple[str, dict[str, Any]]] | None = None + ) -> None: + super().__init__() + # Snapshot at construction time so a plugin registered MID-run + # does not silently start receiving hooks halfway through. The + # caller may pre-collect hooks (the ``build_plugin_callback`` + # path) to avoid a redundant registry scan + close a tiny + # race-window between "is any plugin enabled?" and + # "snapshot the registry". + self._hooks = ( + list(hooks) if hooks is not None else _collect_active_hooks() + ) + + def _dispatch(self, hook_name: str, context: dict[str, Any]) -> None: + for plugin_name, hooks in self._hooks: + hook = hooks.get(hook_name) + if hook is None: + continue + _safe_invoke(plugin_name, hook_name, hook, context) + + def on_train_begin(self, args, state, control, **kwargs): # noqa: D401 + self._dispatch( + "pre_train", {"args": args, "state": state, "control": control} + ) + + def on_train_end(self, args, state, control, **kwargs): # noqa: D401 + self._dispatch( + "post_train", {"args": args, "state": state, "control": control} + ) + + def on_step_begin(self, args, state, control, **kwargs): # noqa: D401 + self._dispatch( + "pre_step", {"args": args, "state": state, "control": control} + ) + + def on_step_end(self, args, state, control, **kwargs): # noqa: D401 + self._dispatch( + "post_step", {"args": args, "state": state, "control": control} + ) + + return SoupPluginCallback + + +def build_plugin_callback() -> Any: + """Return a new ``SoupPluginCallback`` instance, or ``None`` if no + enabled plugins implement any hook (no-op short-circuit). + + Collects hooks ONCE and passes the snapshot to the callback so a + plugin registered between the "is empty?" check and the constructor + cannot silently slip into the active hook list — review fix. + """ + hooks = _collect_active_hooks() + if not hooks: + return None + callback_cls = _build_callback_class() + return callback_cls(hooks) + + +__all__ = ["build_plugin_callback"] diff --git a/soup_cli/trainer/bco.py b/soup_cli/trainer/bco.py index d2adeee..2f4ae40 100644 --- a/soup_cli/trainer/bco.py +++ b/soup_cli/trainer/bco.py @@ -207,11 +207,14 @@ class BCOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/distill.py b/soup_cli/trainer/distill.py index 9823ca4..e28802b 100644 --- a/soup_cli/trainer/distill.py +++ b/soup_cli/trainer/distill.py @@ -379,11 +379,14 @@ class DistillTrainerWrapper: # v0.40.6 #67 — ReLoRA callback. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/dpo.py b/soup_cli/trainer/dpo.py index e7da539..9f7889d 100644 --- a/soup_cli/trainer/dpo.py +++ b/soup_cli/trainer/dpo.py @@ -161,11 +161,14 @@ class DPOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps). from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) # v0.53.2 #135 — GDPO loss hook (no-op if gdpo_variant unset). from soup_cli.utils.ebft_gdpo import attach_gdpo_compute_loss diff --git a/soup_cli/trainer/embedding.py b/soup_cli/trainer/embedding.py index 190b462..c10fef0 100644 --- a/soup_cli/trainer/embedding.py +++ b/soup_cli/trainer/embedding.py @@ -182,11 +182,14 @@ class EmbeddingTrainerWrapper: # v0.40.6 #67 — ReLoRA callback. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/grpo.py b/soup_cli/trainer/grpo.py index f9ef642..dae6e56 100644 --- a/soup_cli/trainer/grpo.py +++ b/soup_cli/trainer/grpo.py @@ -236,11 +236,14 @@ class GRPOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps). from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/ipo.py b/soup_cli/trainer/ipo.py index 18efcea..cf10076 100644 --- a/soup_cli/trainer/ipo.py +++ b/soup_cli/trainer/ipo.py @@ -161,11 +161,14 @@ class IPOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/kto.py b/soup_cli/trainer/kto.py index 3abcc81..b706d29 100644 --- a/soup_cli/trainer/kto.py +++ b/soup_cli/trainer/kto.py @@ -157,11 +157,14 @@ class KTOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps). from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/orpo.py b/soup_cli/trainer/orpo.py index 9a0d723..7333cd8 100644 --- a/soup_cli/trainer/orpo.py +++ b/soup_cli/trainer/orpo.py @@ -158,11 +158,14 @@ class ORPOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/ppo.py b/soup_cli/trainer/ppo.py index e8f2be7..711e7e0 100644 --- a/soup_cli/trainer/ppo.py +++ b/soup_cli/trainer/ppo.py @@ -258,11 +258,14 @@ class PPOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps). from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) self._train_ds = train_ds diff --git a/soup_cli/trainer/pretrain.py b/soup_cli/trainer/pretrain.py index bacb7ec..710febc 100644 --- a/soup_cli/trainer/pretrain.py +++ b/soup_cli/trainer/pretrain.py @@ -215,11 +215,14 @@ class PretrainTrainerWrapper: # v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps). from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/reward_model.py b/soup_cli/trainer/reward_model.py index 34fb3e7..60f9bef 100644 --- a/soup_cli/trainer/reward_model.py +++ b/soup_cli/trainer/reward_model.py @@ -159,11 +159,14 @@ class RewardModelTrainerWrapper: # v0.40.6 #67 — ReLoRA callback. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/trainer/sft.py b/soup_cli/trainer/sft.py index 4fd0ec7..f3eea91 100644 --- a/soup_cli/trainer/sft.py +++ b/soup_cli/trainer/sft.py @@ -854,6 +854,7 @@ class SFTTrainerWrapper: # ReLoRA callback (v0.39.0 Part B / v0.40.6 #67) via shared helper. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, self.config.training) @@ -861,6 +862,8 @@ class SFTTrainerWrapper: attach_curriculum_callback( self.trainer, self.config.training, self._output_dir, console ) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) # v0.53.2 #135 — EBFT compute_loss hook (no-op if ebft_variant unset). from soup_cli.utils.ebft_gdpo import attach_ebft_compute_loss diff --git a/soup_cli/trainer/simpo.py b/soup_cli/trainer/simpo.py index 2c34c64..0b73abc 100644 --- a/soup_cli/trainer/simpo.py +++ b/soup_cli/trainer/simpo.py @@ -161,11 +161,14 @@ class SimPOTrainerWrapper: # v0.40.6 #67 — ReLoRA callback. from soup_cli.utils.peft_wiring import ( attach_curriculum_callback, + attach_plugin_callback, attach_relora_callback, ) attach_relora_callback(self.trainer, tcfg) # v0.53.5 #114/#115 — dynamic curriculum live callback. attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console) + # v0.53.6 #101 — Soup plugin TrainerCallback. + attach_plugin_callback(self.trainer, console) self._output_dir = str(output_dir) diff --git a/soup_cli/utils/peft_wiring.py b/soup_cli/utils/peft_wiring.py index 1ef4eba..d4ad924 100644 --- a/soup_cli/utils/peft_wiring.py +++ b/soup_cli/utils/peft_wiring.py @@ -129,3 +129,45 @@ def attach_curriculum_callback( except Exception: # noqa: BLE001 — never crash on console issues. pass return True + + +def attach_plugin_callback(trainer: Any, console: Any = None) -> bool: + """Attach :class:`SoupPluginCallback` when any enabled plugin implements a hook. + + Returns ``True`` when a callback was attached, ``False`` otherwise + (no plugins enabled OR none implement any hook — the build helper + short-circuits to ``None`` in that case so the trainer pays zero + overhead). + + Failures inside individual plugin hooks are swallowed at WARNING + inside the callback itself; this helper only handles the + construction failure path (transformers not importable / plugin + registry corrupted). + """ + try: + from soup_cli.monitoring.plugin_callback import build_plugin_callback + + callback = build_plugin_callback() + except Exception as exc: # noqa: BLE001 — plugin infra must not crash training + logger.debug("attach_plugin_callback skipped: %s", exc) + return False + if callback is None: + return False + try: + trainer.add_callback(callback) + except Exception as exc: # noqa: BLE001 + logger.debug("attach_plugin_callback add_callback failed: %s", exc) + return False + if console is not None: + try: + # Number of plugins is the count of distinct (plugin_name, hooks) + # pairs the callback snapshot will fan out to. + from soup_cli.plugins import list_plugins + + n_enabled = sum(1 for s in list_plugins().values() if s.enabled) + console.print( + f"[dim]Plugin callback attached ({n_enabled} enabled plugin(s)).[/]" + ) + except Exception: # noqa: BLE001 + pass + return True diff --git a/soup_cli/utils/recipe_run.py b/soup_cli/utils/recipe_run.py new file mode 100644 index 0000000..0364416 --- /dev/null +++ b/soup_cli/utils/recipe_run.py @@ -0,0 +1,69 @@ +"""v0.53.6 #106 — Data Recipe DAG runner (stub-then-live). + +Schema-only this release. Per-node-kind handlers (seed / llm_text / code / +judge / validator / sampler), checkpoint-between-nodes, resume-on-failure ++ ``soup data recipe --execute`` wire-up land in v0.53.7. Mirrors the +project stub-then-live pattern (v0.27.0 MII / v0.37.0 multipack / +v0.50.0 GRPO Plus). + +The validator surface ships now so callers can target the schema and +type-check against ``run_recipe`` before live execution exists. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, Mapping + +if TYPE_CHECKING: # pragma: no cover — type-only import. + from soup_cli.utils.recipe_dag import RecipeDAG + + +def run_recipe( + dag: "RecipeDAG", + *, + output_dir: str, + checkpoint_dir: str | None = None, + resume: bool = False, + judge_provider: str | None = None, + judge_model: str | None = None, +) -> Mapping[str, Any]: + """Execute a validated :class:`RecipeDAG` end-to-end. + + Deferred-live stub. Per-node-kind handlers + checkpoint/resume + + ``--execute`` plumbing land in v0.53.7. Schema-only contract: + + - ``dag``: a :class:`soup_cli.utils.recipe_dag.RecipeDAG` (validated) + - ``output_dir``: cwd-contained directory for sampler output JSONL + - ``checkpoint_dir``: optional intermediate-node checkpoint dir + - ``resume``: if True, skip nodes whose checkpoint already exists + - ``judge_provider`` / ``judge_model``: routed into the v0.40.3 + ``JudgeEvaluator`` for ``judge`` nodes + + Raises: + TypeError: if ``dag`` is not a ``RecipeDAG``. + NotImplementedError: always — live runner ships in v0.53.7. + """ + # Late import so test code can patch the module without forcing a + # heavy recipe_dag import at module load. + from soup_cli.utils.recipe_dag import RecipeDAG + + if not isinstance(dag, RecipeDAG): + raise TypeError("dag must be a RecipeDAG") + if not isinstance(output_dir, str) or not output_dir: + raise TypeError("output_dir must be a non-empty string") + if checkpoint_dir is not None and not isinstance(checkpoint_dir, str): + raise TypeError("checkpoint_dir must be a string or None") + if not isinstance(resume, bool): + raise TypeError("resume must be a bool") + if judge_provider is not None and not isinstance(judge_provider, str): + raise TypeError("judge_provider must be a string or None") + if judge_model is not None and not isinstance(judge_model, str): + raise TypeError("judge_model must be a string or None") + raise NotImplementedError( + "Data Recipe DAG runner is deferred to v0.53.7. The schema + " + "validator surface in soup_cli.utils.recipe_dag is live; only the " + "per-node execution loop is missing." + ) + + +__all__ = ["run_recipe"] diff --git a/soup_cli/utils/trainer_plugins.py b/soup_cli/utils/trainer_plugins.py index bc1053e..fe9670a 100644 --- a/soup_cli/utils/trainer_plugins.py +++ b/soup_cli/utils/trainer_plugins.py @@ -11,7 +11,7 @@ from __future__ import annotations import re from dataclasses import dataclass from types import MappingProxyType -from typing import Mapping, Optional, Sequence, Tuple +from typing import Any, Mapping, Optional, Sequence, Tuple _PLUGIN_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_]{0,31}$") _MAX_DESCRIPTION = 256 @@ -128,9 +128,33 @@ def validate_trainer_plugin_list(names: Sequence[str]) -> Tuple[str, ...]: return tuple(out) +def instantiate_trainer_plugins(names: Sequence[str]) -> Tuple[Any, ...]: + """v0.53.6 #105 — instantiate live upstream callbacks (stub-then-live). + + Validates ``names`` against :func:`validate_trainer_plugin_list` then + raises :class:`NotImplementedError` with a v0.53.7 marker. Live lazy + imports + per-plugin callback construction (``grokfast``, + ``spectrum``, ``llmcompressor``, ``sonicmoe``, ``cce_plugin``, + ``math_verify``) land in v0.53.7. Same stub-then-live pattern as + v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro. + + Raises: + TypeError: per :func:`validate_trainer_plugin_list`. + ValueError: per :func:`validate_trainer_plugin_list`. + NotImplementedError: always (after validation) — live wiring + ships in v0.53.7. + """ + canonical = validate_trainer_plugin_list(names) + raise NotImplementedError( + f"Trainer-plugin live instantiation deferred to v0.53.7. " + f"Validated names: {canonical!r}" + ) + + __all__ = [ "TrainerPluginSpec", "list_trainer_plugins", "get_trainer_plugin", "validate_trainer_plugin_list", + "instantiate_trainer_plugins", ] diff --git a/tests/test_v0536.py b/tests/test_v0536.py new file mode 100644 index 0000000..6d66c03 --- /dev/null +++ b/tests/test_v0536.py @@ -0,0 +1,799 @@ +"""v0.53.6 — Plugin + Agent + Anthropic API. + +Covers: +- #101: SoupPluginCallback + attach_plugin_callback wired into 13 trainers. +- #102: Anthropic /v1/messages route on transformers backend. +- #104: n-gram speculative decoding wiring (prompt_lookup_num_tokens). +- #103: server-side tool endpoints — deferred-live stubs returning 501. +- #105: utils/trainer_plugins.instantiate_trainer_plugins — stub. +- #106: utils/recipe_run.run_recipe — stub + `soup data recipe --execute`. +""" + +from __future__ import annotations + +from pathlib import Path +from unittest.mock import MagicMock + +import pytest +from typer.testing import CliRunner + +# ---------------------------------------------------------------------- +# #101 — SoupPluginCallback + attach_plugin_callback +# ---------------------------------------------------------------------- + + +class _RecordingPlugin: + def __init__(self) -> None: + self.events: list[str] = [] + + def pre_train(self, _ctx): # noqa: D401 + self.events.append("pre_train") + + def post_step(self, _ctx): # noqa: D401 + self.events.append("post_step") + + +class _RaisingPlugin: + def pre_train(self, _ctx): # noqa: D401 + raise RuntimeError("plugin boom") + + +@pytest.fixture +def clear_plugins_fixture(): + from soup_cli.plugins import clear_plugins + + clear_plugins() + yield + clear_plugins() + + +def test_build_plugin_callback_none_when_no_plugins(clear_plugins_fixture): + """No registered plugins → build returns None (no-op short-circuit).""" + from soup_cli.monitoring.plugin_callback import build_plugin_callback + + assert build_plugin_callback() is None + + +def test_build_plugin_callback_skips_disabled(clear_plugins_fixture): + """Disabled plugin → build returns None (no enabled hooks).""" + from soup_cli.monitoring.plugin_callback import build_plugin_callback + from soup_cli.plugins import disable_plugin, register_plugin + + plugin = _RecordingPlugin() + register_plugin(name="rec-a", version="0.1.0", plugin=plugin) + disable_plugin("rec-a") + assert build_plugin_callback() is None + + +def test_build_plugin_callback_returns_callback_when_enabled( + clear_plugins_fixture, +): + """Enabled plugin with at least one hook → real TrainerCallback.""" + transformers = pytest.importorskip("transformers") + from soup_cli.monitoring.plugin_callback import build_plugin_callback + from soup_cli.plugins import register_plugin + + plugin = _RecordingPlugin() + register_plugin(name="rec-b", version="0.1.0", plugin=plugin) + callback = build_plugin_callback() + assert callback is not None + assert isinstance(callback, transformers.TrainerCallback) + + +def test_plugin_callback_dispatches_to_implemented_hooks( + clear_plugins_fixture, +): + """Trainer event → only implemented hooks fire.""" + pytest.importorskip("transformers") + from soup_cli.monitoring.plugin_callback import build_plugin_callback + from soup_cli.plugins import register_plugin + + plugin = _RecordingPlugin() + register_plugin(name="rec-c", version="0.1.0", plugin=plugin) + callback = build_plugin_callback() + assert callback is not None + args = MagicMock() + state = MagicMock() + control = MagicMock() + + callback.on_train_begin(args, state, control) + callback.on_step_end(args, state, control) + callback.on_step_begin(args, state, control) # plugin has no pre_step + callback.on_train_end(args, state, control) # plugin has no post_train + assert plugin.events == ["pre_train", "post_step"] + + +def test_plugin_callback_swallows_hook_exceptions( + clear_plugins_fixture, caplog +): + """One misbehaving plugin must not crash training.""" + pytest.importorskip("transformers") + from soup_cli.monitoring.plugin_callback import build_plugin_callback + from soup_cli.plugins import register_plugin + + register_plugin(name="bad-plugin", version="0.1.0", plugin=_RaisingPlugin()) + callback = build_plugin_callback() + assert callback is not None + caplog.clear() # review fix — defend against accumulated log records + with caplog.at_level("WARNING"): + callback.on_train_begin(MagicMock(), MagicMock(), MagicMock()) + # Did not raise; recorded a WARNING for this specific plugin. + matching = [ + rec + for rec in caplog.records + if rec.levelname == "WARNING" and "bad-plugin" in rec.message + ] + assert matching, "expected WARNING for bad-plugin hook failure" + + +def test_attach_plugin_callback_no_plugins_returns_false( + clear_plugins_fixture, +): + """No plugins registered → helper short-circuits to False (no trainer touch).""" + from soup_cli.utils.peft_wiring import attach_plugin_callback + + trainer = MagicMock() + attached = attach_plugin_callback(trainer) + assert attached is False + trainer.add_callback.assert_not_called() + + +def test_attach_plugin_callback_attaches_when_enabled( + clear_plugins_fixture, +): + """Enabled plugin → trainer.add_callback called once with the callback.""" + pytest.importorskip("transformers") + from soup_cli.plugins import register_plugin + from soup_cli.utils.peft_wiring import attach_plugin_callback + + register_plugin(name="rec-attach", version="0.1.0", plugin=_RecordingPlugin()) + trainer = MagicMock() + attached = attach_plugin_callback(trainer) + assert attached is True + trainer.add_callback.assert_called_once() + + +def test_attach_plugin_callback_swallows_add_callback_failure( + clear_plugins_fixture, +): + """trainer.add_callback raising → helper returns False, no crash.""" + pytest.importorskip("transformers") + from soup_cli.plugins import register_plugin + from soup_cli.utils.peft_wiring import attach_plugin_callback + + register_plugin(name="rec-fail", version="0.1.0", plugin=_RecordingPlugin()) + trainer = MagicMock() + trainer.add_callback.side_effect = RuntimeError("add_callback broken") + attached = attach_plugin_callback(trainer) + assert attached is False + + +def test_attach_plugin_callback_console_advisory(clear_plugins_fixture): + """Optional `console` argument prints an advisory; never crashes.""" + pytest.importorskip("transformers") + from soup_cli.plugins import register_plugin + from soup_cli.utils.peft_wiring import attach_plugin_callback + + register_plugin(name="rec-console", version="0.1.0", plugin=_RecordingPlugin()) + trainer = MagicMock() + console = MagicMock() + attached = attach_plugin_callback(trainer, console) + assert attached is True + console.print.assert_called_once() + msg = console.print.call_args.args[0] + assert "1 enabled" in msg + + +def test_attach_plugin_callback_console_print_failure_swallowed( + clear_plugins_fixture, +): + """`console.print` raising must not crash the helper.""" + pytest.importorskip("transformers") + from soup_cli.plugins import register_plugin + from soup_cli.utils.peft_wiring import attach_plugin_callback + + register_plugin(name="rec-console2", version="0.1.0", plugin=_RecordingPlugin()) + trainer = MagicMock() + console = MagicMock() + console.print.side_effect = RuntimeError("console broken") + attached = attach_plugin_callback(trainer, console) + assert attached is True # callback still attached even though print failed + + +@pytest.mark.parametrize( + "trainer_file", + [ + "soup_cli/trainer/sft.py", + "soup_cli/trainer/dpo.py", + "soup_cli/trainer/grpo.py", + "soup_cli/trainer/kto.py", + "soup_cli/trainer/orpo.py", + "soup_cli/trainer/simpo.py", + "soup_cli/trainer/ipo.py", + "soup_cli/trainer/bco.py", + "soup_cli/trainer/ppo.py", + "soup_cli/trainer/pretrain.py", + "soup_cli/trainer/reward_model.py", + "soup_cli/trainer/embedding.py", + "soup_cli/trainer/distill.py", + ], +) +def test_every_trainer_wires_attach_plugin_callback(trainer_file: str): + """Source-level invariant: every transformer-backend trainer wires the helper.""" + repo_root = Path(__file__).resolve().parent.parent + src = (repo_root / trainer_file).read_text(encoding="utf-8") + assert "attach_plugin_callback" in src, ( + f"{trainer_file} is missing attach_plugin_callback wiring" + ) + # Direct import from canonical peft_wiring module (no re-export shim). + assert "attach_plugin_callback," in src or ( + "from soup_cli.utils.peft_wiring import" in src + and "attach_plugin_callback" in src + ), f"{trainer_file} must import attach_plugin_callback from peft_wiring" + # Sanity: imported AND called on self.trainer. + assert "attach_plugin_callback(self.trainer" in src, ( + f"{trainer_file} imports but never invokes attach_plugin_callback" + ) + + +# ---------------------------------------------------------------------- +# #102 — Anthropic /v1/messages route +# ---------------------------------------------------------------------- + + +def _fake_model_and_tokenizer(): + """Build minimal mocks so _create_app can construct without HF deps.""" + model = MagicMock() + tokenizer = MagicMock() + tokenizer.pad_token_id = 0 + return model, tokenizer + + +def _build_app(**overrides): + pytest.importorskip("fastapi") + from soup_cli.commands.serve import _create_app + + model, tokenizer = _fake_model_and_tokenizer() + kwargs = dict( + model_obj=model, + tokenizer=tokenizer, + device="cpu", + model_name="test-model", + max_tokens_default=64, + ) + kwargs.update(overrides) + return _create_app(**kwargs) + + +def test_anthropic_messages_route_registered(): + """POST /v1/messages must be present on the FastAPI app.""" + app = _build_app() + routes = {(r.path, tuple(sorted(r.methods))) for r in app.routes if hasattr(r, "methods")} + assert ("/v1/messages", ("POST",)) in routes + + +def test_anthropic_messages_rejects_malformed_payload(monkeypatch): + """Schema validation propagates to HTTP 400 — no internal crash.""" + from fastapi.testclient import TestClient + + app = _build_app() + client = TestClient(app) + # Missing required `messages` field. + response = client.post("/v1/messages", json={"model": "x", "max_tokens": 16}) + assert response.status_code == 400 + + +def test_anthropic_messages_rejects_streaming(): + """`stream=True` returns 501 (deferred to v0.53.7).""" + from fastapi.testclient import TestClient + + app = _build_app() + client = TestClient(app) + payload = { + "model": "test-model", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 16, + "stream": True, + } + response = client.post("/v1/messages", json=payload) + assert response.status_code == 501 + assert "v0.53.7" in response.json()["detail"] + + +def test_anthropic_messages_happy_path(monkeypatch): + """End-to-end: payload → from_anthropic → chat_completions mock → Anthropic shape.""" + from fastapi.testclient import TestClient + + # Patch _generate_response so we don't need a real model. + monkeypatch.setattr( + "soup_cli.commands.serve._generate_response", + lambda *a, **kw: ("hello world", 3, 2), + ) + app = _build_app() + client = TestClient(app) + payload = { + "model": "test-model", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 16, + } + response = client.post("/v1/messages", json=payload) + assert response.status_code == 200, response.text + body = response.json() + assert body["type"] == "message" + assert body["role"] == "assistant" + assert body["content"] == [{"type": "text", "text": "hello world"}] + assert body["model"] == "test-model" + assert body["stop_reason"] == "end_turn" + assert body["usage"]["input_tokens"] == 3 + assert body["usage"]["output_tokens"] == 2 + + +# ---------------------------------------------------------------------- +# #104 — n-gram speculative decoding wiring +# ---------------------------------------------------------------------- + + +def test_ngram_config_threaded_into_generate_response(monkeypatch): + """`_create_app(ngram_config=...)` forwards through to `_generate_response`.""" + from fastapi.testclient import TestClient + + from soup_cli.utils.ngram_spec import NgramSpecConfig + + received_kwargs: dict = {} + + def _capture(*args, **kwargs): + received_kwargs.update(kwargs) + return ("ok", 1, 1) + + monkeypatch.setattr("soup_cli.commands.serve._generate_response", _capture) + cfg = NgramSpecConfig(n=3, num_draft_tokens=4, prompt_lookup_max=0) + app = _build_app(ngram_config=cfg) + client = TestClient(app) + response = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 8, + }, + ) + assert response.status_code == 200, response.text + assert received_kwargs.get("ngram_config") is cfg + + +def test_ngram_kwarg_emits_prompt_lookup_num_tokens(): + """Direct unit test on `_generate_response` proves the kwarg makes it + into ``model.generate``.""" + pytest.importorskip("torch") + from soup_cli.commands.serve import _generate_response + from soup_cli.utils.ngram_spec import NgramSpecConfig + + captured: dict = {} + + class _FakeOutput: + def __getitem__(self, _): + class _T: + shape = (0, 5) + + def __getitem__(self, _): + return [] + + return _T() + + class _FakeModel: + device = "cpu" + + def generate(self, **kwargs): + captured.update(kwargs) + + class _Tensor: + def __getitem__(self, _): + return [] + + return [[0, 1, 2, 3, 4]] + + class _FakeTokenizer: + pad_token_id = 0 + chat_template = None # forces "Assistant:" fallback path + + def __call__(self, _text, return_tensors=None): + import torch as _torch + + ids = _torch.tensor([[1, 2, 3]]) + return {"input_ids": ids, "attention_mask": _torch.ones_like(ids)} + + def decode(self, _tokens, skip_special_tokens=True): + return "hello" + + cfg = NgramSpecConfig(n=3, num_draft_tokens=7, prompt_lookup_max=0) + _generate_response( + _FakeModel(), + _FakeTokenizer(), + [{"role": "user", "content": "hi"}], + max_tokens=4, + ngram_config=cfg, + ) + assert captured.get("prompt_lookup_num_tokens") == 7 + + +def test_ngram_kwarg_skipped_when_draft_model_set(): + """A real `assistant_model` wins over n-gram — keys mutually exclusive.""" + pytest.importorskip("torch") + from soup_cli.commands.serve import _generate_response + from soup_cli.utils.ngram_spec import NgramSpecConfig + + captured: dict = {} + + class _FakeModel: + device = "cpu" + + def generate(self, **kwargs): + captured.update(kwargs) + return [[0, 1, 2, 3]] + + class _FakeTokenizer: + pad_token_id = 0 + chat_template = None + + def __call__(self, _text, return_tensors=None): + import torch as _torch + + ids = _torch.tensor([[1, 2]]) + return {"input_ids": ids, "attention_mask": _torch.ones_like(ids)} + + def decode(self, _tokens, skip_special_tokens=True): + return "" + + cfg = NgramSpecConfig(n=3, num_draft_tokens=4) + _generate_response( + _FakeModel(), + _FakeTokenizer(), + [{"role": "user", "content": "x"}], + max_tokens=2, + assistant_model=object(), # truthy + ngram_config=cfg, + ) + assert "prompt_lookup_num_tokens" not in captured + assert captured.get("assistant_model") is not None + + +# ---------------------------------------------------------------------- +# #103 — Server-side tool endpoints (stub: 501) +# ---------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "path", + ["/v1/tools/python", "/v1/tools/bash", "/v1/tools/web_search"], +) +def test_tool_endpoint_returns_501(path: str): + """All three tool endpoints return 501 with v0.53.7 marker.""" + from fastapi.testclient import TestClient + + app = _build_app() + client = TestClient(app) + response = client.post(path, json={"code": "print(1)"}) + assert response.status_code == 501 + assert "v0.53.7" in response.json()["detail"] + + +# ---------------------------------------------------------------------- +# #106 — recipe_run stub + `soup data recipe --execute` +# ---------------------------------------------------------------------- + + +def test_run_recipe_rejects_non_dag(): + from soup_cli.utils.recipe_run import run_recipe + + with pytest.raises(TypeError, match="RecipeDAG"): + run_recipe("not-a-dag", output_dir="out") # type: ignore[arg-type] + + +def test_run_recipe_rejects_bad_kwargs(): + from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode + from soup_cli.utils.recipe_run import run_recipe + + dag = RecipeDAG( + nodes=(RecipeNode(name="a", kind="seed", config={}),), + edges=(), + topo_order=("a",), + ) + with pytest.raises(TypeError, match="output_dir"): + run_recipe(dag, output_dir=123) # type: ignore[arg-type] + with pytest.raises(TypeError, match="resume"): + run_recipe(dag, output_dir="out", resume="yes") # type: ignore[arg-type] + + +def test_run_recipe_raises_not_implemented(): + from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode + from soup_cli.utils.recipe_run import run_recipe + + dag = RecipeDAG( + nodes=(RecipeNode(name="a", kind="seed", config={}),), + edges=(), + topo_order=("a",), + ) + with pytest.raises(NotImplementedError, match="v0.53.7"): + run_recipe(dag, output_dir="out") + + +def test_data_recipe_execute_flag_present(tmp_path: Path): + """`soup data recipe --help` lists `--execute`.""" + from soup_cli.cli import app as cli_app + + runner = CliRunner() + result = runner.invoke(cli_app, ["data", "recipe", "--help"]) + assert result.exit_code == 0, result.output + assert "--execute" in result.output + + +def test_data_recipe_execute_requires_output(tmp_path: Path, monkeypatch): + """`--execute` without `--output` exits 2 with a clear message.""" + from soup_cli.cli import app as cli_app + + monkeypatch.chdir(tmp_path) + Path("recipe.yaml").write_text( + "nodes:\n" + " - name: a\n" + " kind: seed\n" + "edges: []\n", + encoding="utf-8", + ) + runner = CliRunner() + result = runner.invoke( + cli_app, + ["data", "recipe", "recipe.yaml", "--execute"], + ) + assert result.exit_code == 2, result.output + assert "--output" in result.output + + +def test_data_recipe_execute_surfaces_v0537_marker(tmp_path: Path, monkeypatch): + """`--execute --output ` runs through run_recipe and surfaces v0.53.7 marker.""" + from soup_cli.cli import app as cli_app + + monkeypatch.chdir(tmp_path) + Path("recipe.yaml").write_text( + "nodes:\n" + " - name: a\n" + " kind: seed\n" + "edges: []\n", + encoding="utf-8", + ) + runner = CliRunner() + result = runner.invoke( + cli_app, + [ + "data", + "recipe", + "recipe.yaml", + "--execute", + "--output", + "out", + ], + ) + assert result.exit_code == 2, result.output + assert "v0.53.7" in result.output + + +# ---------------------------------------------------------------------- +# #105 — instantiate_trainer_plugins stub +# ---------------------------------------------------------------------- + + +def test_instantiate_trainer_plugins_validates_then_raises(): + """Validation runs first, then NotImplementedError with v0.53.7 marker.""" + from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins + + with pytest.raises(NotImplementedError, match="v0.53.7"): + instantiate_trainer_plugins(["grokfast"]) + + +def test_instantiate_trainer_plugins_validation_runs_first(): + """Unknown plugin name → ValueError BEFORE NotImplementedError.""" + from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins + + with pytest.raises(ValueError, match="unknown trainer plugin"): + instantiate_trainer_plugins(["definitely-not-a-real-plugin"]) + + +def test_instantiate_trainer_plugins_rejects_non_sequence(): + from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins + + with pytest.raises(TypeError): + instantiate_trainer_plugins("grokfast") # type: ignore[arg-type] + + +def test_instantiate_trainer_plugins_in_dunder_all(): + """Public API surface includes the new stub.""" + import soup_cli.utils.trainer_plugins as mod + + assert "instantiate_trainer_plugins" in mod.__all__ + + +def test_instantiate_trainer_plugins_empty_list_passes_validation(): + """Empty list is valid per `validate_trainer_plugin_list` → the + NotImplementedError fires with the empty tuple in the message.""" + from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins + + with pytest.raises(NotImplementedError, match=r"\(\)|v0\.53\.7"): + instantiate_trainer_plugins([]) + + +# ---------------------------------------------------------------------- +# Review-fix coverage: regression guards + missing boundaries +# ---------------------------------------------------------------------- + + +def test_ngram_kwarg_omitted_when_config_is_none(): + """Default `ngram_config=None` → `prompt_lookup_num_tokens` NOT emitted. + + Regression guard: a future refactor that always emits the kwarg + would break the legacy free-form generation path. + """ + pytest.importorskip("torch") + from soup_cli.commands.serve import _generate_response + + captured: dict = {} + + class _FakeModel: + device = "cpu" + + def generate(self, **kwargs): + captured.update(kwargs) + return [[0, 1, 2]] + + class _FakeTokenizer: + pad_token_id = 0 + chat_template = None + + def __call__(self, _text, return_tensors=None): + import torch as _torch + + ids = _torch.tensor([[1]]) + return {"input_ids": ids, "attention_mask": _torch.ones_like(ids)} + + def decode(self, _tokens, skip_special_tokens=True): + return "" + + _generate_response( + _FakeModel(), + _FakeTokenizer(), + [{"role": "user", "content": "x"}], + max_tokens=1, + ) + assert "prompt_lookup_num_tokens" not in captured + + +def test_run_recipe_rejects_empty_output_dir(): + from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode + from soup_cli.utils.recipe_run import run_recipe + + dag = RecipeDAG( + nodes=(RecipeNode(name="a", kind="seed", config={}),), + edges=(), + topo_order=("a",), + ) + with pytest.raises(TypeError, match="output_dir"): + run_recipe(dag, output_dir="") + + +def test_run_recipe_rejects_non_str_judge_provider(): + from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode + from soup_cli.utils.recipe_run import run_recipe + + dag = RecipeDAG( + nodes=(RecipeNode(name="a", kind="seed", config={}),), + edges=(), + topo_order=("a",), + ) + with pytest.raises(TypeError, match="judge_provider"): + run_recipe(dag, output_dir="out", judge_provider=123) # type: ignore[arg-type] + + +def test_run_recipe_rejects_non_str_judge_model(): + from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode + from soup_cli.utils.recipe_run import run_recipe + + dag = RecipeDAG( + nodes=(RecipeNode(name="a", kind="seed", config={}),), + edges=(), + topo_order=("a",), + ) + with pytest.raises(TypeError, match="judge_model"): + run_recipe(dag, output_dir="out", judge_model=b"bytes") # type: ignore[arg-type] + + +def test_run_recipe_rejects_non_str_checkpoint_dir(): + from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode + from soup_cli.utils.recipe_run import run_recipe + + dag = RecipeDAG( + nodes=(RecipeNode(name="a", kind="seed", config={}),), + edges=(), + topo_order=("a",), + ) + with pytest.raises(TypeError, match="checkpoint_dir"): + run_recipe(dag, output_dir="out", checkpoint_dir=99) # type: ignore[arg-type] + + +def test_anthropic_messages_validation_detail_redacted(monkeypatch): + """Validation errors must surface as generic 'Invalid request' — no + internal validator detail in the HTTP body. Security review M1.""" + from fastapi.testclient import TestClient + + app = _build_app() + client = TestClient(app) + response = client.post( + "/v1/messages", + json={"model": "x", "messages": [], "max_tokens": 16}, # empty msgs + ) + assert response.status_code == 400 + body = response.json() + assert body["detail"] == "Invalid request" + + +def test_anthropic_messages_rejects_oversize_max_tokens(): + """`max_tokens` above the v0.30.0 16384 cap is rejected by the + underlying validator (defence-in-depth — also covered upstream).""" + from fastapi.testclient import TestClient + + app = _build_app() + client = TestClient(app) + response = client.post( + "/v1/messages", + json={ + "model": "x", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 999_999, + }, + ) + assert response.status_code == 400 + + +def test_data_recipe_execute_rejects_empty_output(tmp_path: Path, monkeypatch): + """`--output ""` is rejected at the CLI boundary.""" + from soup_cli.cli import app as cli_app + + monkeypatch.chdir(tmp_path) + Path("recipe.yaml").write_text( + "nodes:\n - name: a\n kind: seed\nedges: []\n", + encoding="utf-8", + ) + runner = CliRunner() + result = runner.invoke( + cli_app, + ["data", "recipe", "recipe.yaml", "--execute", "--output", ""], + ) + assert result.exit_code == 2, result.output + assert "must be a non-empty path" in result.output + + +def test_data_recipe_execute_rejects_outside_cwd(tmp_path: Path, monkeypatch): + """`--output` outside cwd is rejected before the stub runs.""" + import sys + + from soup_cli.cli import app as cli_app + + monkeypatch.chdir(tmp_path) + sub = tmp_path / "proj" + sub.mkdir() + monkeypatch.chdir(sub) + Path("recipe.yaml").write_text( + "nodes:\n - name: a\n kind: seed\nedges: []\n", + encoding="utf-8", + ) + outside = str(tmp_path / "outside") + # On Windows, an absolute path under tmp_path.parent (different + # drive in some CI envs) is still not under cwd=tmp_path/proj. + runner = CliRunner() + result = runner.invoke( + cli_app, + ["data", "recipe", "recipe.yaml", "--execute", "--output", outside], + ) + # Either outside-cwd (preferred) or some platform-specific reject — + # the live runner should NEVER fire. + assert result.exit_code == 2, (result.output, sys.platform) + assert "v0.53.7" not in result.output, ( + "outside-cwd output must NOT reach the live runner stub" + )