diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ca4cd58..a0c9f9e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (164 files, 5820 tests) +tests/ - Test suite (165 files, 5989 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index 5c6a893..d3a37bf 100644 --- a/README.md +++ b/README.md @@ -43,15 +43,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.44.0 — Live Dashboard & UX**: Studio-grade observability + 13 ergonomics fixes + 7 new standalone CLIs. 21 features that close the polish gap with Unsloth Studio, axolotl, and LlamaFactory. +**v0.45.0 — Plugin System & Ecosystem Wins**: A public plugin API and the schema scaffolding for 20+ ecosystem integrations. Soup is now extensible. -- **`soup monitor` — live GPU panel.** Rich Live `nvidia-smi`-driven dashboard: Util / Mem / VRAM / Temp / Power per GPU. `--refresh 0.25-30` interval, `--once` for a single snapshot. Apple Silicon hint deferred to v0.44.1. -- **Standalone CLIs.** `soup fetch examples llama-3.1-8b-lora` writes a ready-to-edit YAML from the bundled catalog. `soup quantize --to gguf --bits 4` prints the equivalent `soup export …` invocation. `soup merge-sharded-fsdp-weights` and `soup delinearize-llama4` ship as planners (live torch runtime in v0.44.1). `soup llama ` proxies to llama.cpp binaries with a child-env allowlist that drops `HF_TOKEN` / `OPENAI_API_KEY` / `ANTHROPIC_API_KEY`. -- **Ctrl+C graceful save.** First SIGINT writes a checkpoint and continues; second SIGINT stops training cleanly. Touch `/.checkpoint_now` to force an out-of-band save (cwd-contained, symlink-rejected at the trigger path). -- **Web UI plugin registry.** Drop-in `soup_cli/ui/plugins/*.py` files register tabs via `register_tab(name=…, title=…, render=…)` at import time. Tab name kebab-case allowlist, 32-tab cap, idempotent re-register. Plus `API_HOST` / `API_PORT` / `API_KEY` / `GRADIO_HOST` / `GRADIO_PORT` env knobs for the FastAPI + Gradio surfaces. -- **Tail-latency stats + tool-call timer.** `update_ema` / `percentile` / `summarise_latency` ship as pure-Python (used by `runs show` + the live dashboard). `ToolOutputsBuffer` is a thread-safe `collections.deque(maxlen=1000)` ring; `ToolCallTimer` context-manager records duration / output / error per tool invocation for tool-calling SFT runs. -- **Onboarding wizard helper.** `render_onboarding_yaml({base, dataset, task, quantization, epochs})` returns a complete validated `soup.yaml` — `output` field cwd-contained, Literal allowlists on `task` + `quantization`, `epochs ∈ [1, 10]`. -- **+192 net new tests** — covers all 21 features: GPU-CSV parser + DoS caps, SSE frame schema, QR token in query string (not fragment) with IPv6 bracketing, llama-server timings + KV bar, deque ring + concurrent writes, Ctrl+C SIGINT install/restore, sweep-config scalar allowlist + frozen `MappingProxyType`, fetch symlink + commonpath defence, llama child-env allowlist drops secrets, plus 5 review-fix coverage gaps closed. +- **Plugin / hook system.** `soup_cli.plugins.register_plugin(name, version, plugin, templates=[], model_groups=[])` lets third-party Python modules ship their own pre-train / post-train / pre-step / post-step hooks plus chat templates and model groups. Registry is idempotent on identical specs and rejects conflicting re-registration. Drop a file under `soup_cli/plugins/` and the loader picks it up at startup. New `soup plugins list / install / enable / disable` CLI. +- **Anthropic Messages API converter.** `to_anthropic` / `from_anthropic` translate between OpenAI chat-completions and Anthropic Messages payloads — multiple `system` messages joined with `\n\n`, `tool` role surfaces as `tool_result` content blocks (list content concatenated, never silently dropped), `max_tokens` capped at 16384, `temperature` bounded `[0.0, 2.0]`. Live `/v1/messages` endpoint deferred to v0.45.1. +- **Server-side tools allowlist.** Closed `{python, bash, web_search}` set with `WebSearchConfig` — domain allowlist (leading-dot subdomain pattern, port-strip on the host, IPv6 literal deny), `rate_limit_per_minute ∈ [1, 600]`. `python` and `bash` reuse the v0.25.0 RLVR sandbox; live HTTP endpoints deferred to v0.45.1. +- **External integrations catalog.** 15-entry frozen `MappingProxyType` of ecosystem targets — `lm-studio`, `comfyui`, `stable-diffusion-cpp`, `open-webui`, `ollama`, `tei`, `pgvector`, `faiss`, `weaviate`, `sentence-transformers`, `claude-code`, `cursor`, `continue`, `cline`, `sillytavern`. Each entry names the artifact format (`gguf` / `safetensors` / `served-endpoint`). +- **Advanced trainer-plugin allowlist.** 6-entry catalog (`grokfast` / `spectrum` / `llmcompressor` / `sonicmoe` / `cce_plugin` / `math_verify`) with `validate_trainer_plugin_list` (dedup, allowlist match, `_MAX_PLUGINS_PER_RUN=8`). Live callbacks land in v0.45.1. +- **`soup data recipe `.** Validates a Seed → LLM Text → Code → Judge → Validator → Sampler graph DAG: closed node-kind allowlist, Kahn's topological sort with `collections.deque` (deterministic, O(N+E)), self-loop / cycle / dangling-edge / duplicate rejection, cwd containment + `os.lstat + S_ISLNK` symlink rejection on the recipe file. Live offline runner against a local model in v0.45.1. +- **+169 net new tests** — covers all 5 release Parts: plugin idempotency + per-list caps + description conflict rejection, Anthropic converter happy + failure paths, domain allowlist port-strip + IPv6 deny, n-gram bounds, integrations catalog immutability, trainer-plugin canonicalisation + dedup, recipe DAG cycle / cap / symlink + CLI happy / invalid / missing. ## Why Soup? @@ -3161,6 +3161,122 @@ params: Strict scalar allowlist on values (`str` / `int` / `float` / `bool`); `_MAX_FILE_BYTES=256KB`, `_MAX_PARAM_KEYS=32`, `_MAX_VALUES_PER_KEY=64`; `SweepSpec.params` is `MappingProxyType[str, Tuple[Any, ...]]` for genuine immutability. +## Plugin System + +Drop a Python module under `soup_cli/plugins/` (or any package importable by Soup) and register at import time: + +```python +from soup_cli.plugins import register_plugin + +class MyPlugin: + def pre_train(self, ctx): + ... + def post_train(self, ctx): + ... + +register_plugin( + name="my-plugin", + version="1.0.0", + plugin=MyPlugin(), + description="Hooks into pre/post-train", + templates=["my-template"], # optional + model_groups=["my-arch-family"], # optional +) +``` + +```bash +soup plugins # list registered plugins +soup plugins enable foo +soup plugins disable foo +``` + +Plugin names are kebab-case (`^[a-z0-9][a-z0-9-]{0,39}$`); versions are semver-ish (`MAJOR.MINOR.PATCH`); registry caps `_MAX_PLUGINS=64`, `_MAX_TEMPLATES_PER_PLUGIN=32`, `_MAX_MODEL_GROUPS_PER_PLUGIN=32`. Re-registering the same `(name, version, plugin, templates, model_groups, description)` is idempotent; any field mismatch is rejected with a clear error. Trainer-callback wiring of `pre_train` / `post_train` / `pre_step` / `post_step` lands in v0.45.1. + +## Anthropic Messages API Converter + +Pure-Python converters between OpenAI chat-completions and Anthropic Messages payload shapes: + +```python +from soup_cli.utils.anthropic_messages import to_anthropic, from_anthropic + +anthropic_payload = to_anthropic({ + "model": "claude-3-5-sonnet", + "messages": [ + {"role": "system", "content": "you are helpful"}, + {"role": "user", "content": "hi"}, + ], + "max_tokens": 256, +}) +``` + +Multiple `system` messages join with `\n\n`. `tool` role with structured (list) content is concatenated into a single `tool_result` text block, never silently dropped. `max_tokens` capped at 16384, `temperature` bounded `[0.0, 2.0]`. Live `/v1/messages` endpoint inside `soup serve` lands in v0.45.1. + +## Server-Side Tools + +```python +from soup_cli.utils.server_tools import ( + SUPPORTED_TOOLS, WebSearchConfig, is_domain_allowed, validate_web_search_config, +) + +# SUPPORTED_TOOLS == frozenset({"python", "bash", "web_search"}) +config = WebSearchConfig( + domain_allowlist=("example.com", ".docs.example.com"), + rate_limit_per_minute=30, +) +validate_web_search_config(config) +is_domain_allowed("a.docs.example.com:443", config.domain_allowlist) # True +is_domain_allowed("[::1]", config.domain_allowlist) # False +``` + +`python` and `bash` reuse the v0.25.0 RLVR sandbox; `web_search` is gated by an explicit domain allowlist (default empty = deny all). `is_domain_allowed` strips `:port` suffixes before matching and rejects IPv6 literals so `Host: api.example.com:443` matches `api.example.com`. Live HTTP tool endpoints in v0.45.1. + +## External Integrations Catalog + +```python +from soup_cli.utils.integrations import list_integrations, get_integration + +list_integrations() # 15 entries +get_integration("lm-studio").target_artifacts # ("gguf",) +``` + +15 ecosystem targets covered: `lm-studio`, `comfyui`, `stable-diffusion-cpp`, `open-webui`, `ollama`, `tei`, `pgvector`, `faiss`, `weaviate`, `sentence-transformers`, `claude-code`, `cursor`, `continue`, `cline`, `sillytavern`. Auto-detect + launch wiring lands with v0.46.0 Deploy Autopilot. + +## Advanced Trainer Plugins + +```python +from soup_cli.utils.trainer_plugins import validate_trainer_plugin_list + +validate_trainer_plugin_list(["grokfast", "spectrum"]) +# returns ("grokfast", "spectrum") — canonical lowercase, dedup, ≤ 8 entries +``` + +6-entry allowlist (`cce_plugin`, `grokfast`, `spectrum`, `llmcompressor`, `sonicmoe`, `math_verify`) so a future `training.trainer_plugins: [...]` schema field has a stable surface. Live callbacks in v0.45.1. + +## Data Recipe DAG + +```bash +soup data recipe my_recipe.yaml +``` + +```yaml +nodes: + - name: seed1 + kind: seed + config: {path: prompts.jsonl} + - name: llm1 + kind: llm_text + - name: judge1 + kind: judge + - name: samp1 + kind: sampler +edges: + - [seed1, llm1] + - [llm1, judge1] + - [judge1, samp1] +``` + +Closed node-kind allowlist (`seed` / `llm_text` / `code` / `judge` / `validator` / `sampler`); Kahn's topological sort via `collections.deque` (deterministic, O(N+E)); cycle / self-loop / duplicate-edge / dangling-edge / unknown-kind rejection. `_MAX_NODES=256`, `_MAX_EDGES=1024`, `_MAX_FILE_BYTES=1MiB`. The recipe file must stay under cwd and **must not be a symlink** (`os.lstat + S_ISLNK` TOCTOU defence). Live offline runner against a local model lands in v0.45.1. + ## Changelog See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history. diff --git a/SECURITY.md b/SECURITY.md index 5449523..a919426 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,12 +9,12 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.44.0 -- Full support (latest) +- v0.45.0 -- Full support (latest) +- v0.44.0 -- Full support - v0.43.0 -- Full support -- v0.42.0 -- Full support +- v0.42.0-v0.42.x -- Bug-fix support only - v0.41.0-v0.41.x -- Bug-fix support only -- v0.40.0-v0.40.x -- Bug-fix support only -- v0.39.x and below -- No support +- v0.40.x and below -- No support ## Reporting a Vulnerability @@ -145,6 +145,7 @@ No known critical vulnerabilities in current releases. - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input. - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS` +- **v0.45.0 — Plugin System & Ecosystem Wins**: 5 release Parts ship the plugin / hook system + 4 ecosystem-integration schemas. New `soup_cli/plugins/__init__.py` registry: kebab-case plugin name regex `^[a-z0-9][a-z0-9\-]{0,39}$`, semver-ish version regex, `_MAX_PLUGINS=64`, `_MAX_TEMPLATES_PER_PLUGIN=32`, `_MAX_MODEL_GROUPS_PER_PLUGIN=32`, `_MAX_NAME_ENTRY_LEN=128`. Re-registering with a different `(version, plugin object, templates, model_groups, description)` is rejected (review fix HIGH — first-cut omitted `description` from the conflict check, allowing silent description override). `load_plugins()` swallows per-plugin exceptions at WARNING level so one broken plugin cannot crash the CLI. `soup plugins enable / disable` raise `KeyError` for unknown names (caller-distinguishable from validation errors). All user-controlled output passes through `rich.markup.escape`. New `utils/anthropic_messages.py` enforces `_MAX_MESSAGES=1024`, per-message `_MAX_CONTENT_LEN=1MiB`, `max_tokens` cap mirrors v0.30.0 inference-server `_MAX_TOKENS_CAP=16384`; `tool` role with structured (list) content is concatenated into a single `tool_result` text block instead of silently dropped (review fix MEDIUM — first-cut emitted `""` for any non-str content); `from_anthropic` uses `.get("role")` defensively after `validate_anthropic_payload` (review fix MEDIUM — `KeyError` surface guard). New `utils/server_tools.py` ships closed `SUPPORTED_TOOLS = frozenset({python, bash, web_search})` allowlist + `WebSearchConfig` with `domain_allowlist` (≤64 entries, default empty = deny all) + `rate_limit_per_minute ∈ [1, 600]`. `is_domain_allowed` strips `:port` suffix before matching (review fix MEDIUM — without this, `Host: api.example.com:443` would silently fail to match the bare `api.example.com` allowlist entry, causing legitimate denials) AND rejects IPv6 literals (`[::1]` → False — IPv6 should never match a domain allowlist). `validate_domain` rejects null-byte / whitespace / `/` / oversize. `python` and `bash` reuse the v0.25.0 RLVR sandbox (5s timeout, RLIMIT_AS/CPU on POSIX, ephemeral cwd, socket patch); live HTTP endpoints land in v0.45.1. New `utils/ngram_spec.py` `NgramSpecConfig` is `@dataclass(frozen=True)` with `n ∈ [1, 8]`, `num_draft_tokens ∈ [1, 32]`, `prompt_lookup_max ∈ [0, 1MiB]`; bool rejection on every numeric field (matches v0.30.0 `Candidate` / v0.34.0 `estimate_run_cost_usd` / v0.42.0 `make_preprocess_cache_key` policy). New `utils/integrations.py` 15-entry catalog wrapped in `MappingProxyType`; `IntegrationSpec` is frozen so the catalog cannot be mutated at runtime. New `utils/trainer_plugins.py` 6-entry allowlist (`MappingProxyType`); `validate_trainer_plugin_list` accepts `Sequence[str]` (review fix MEDIUM — first-cut annotated as `Tuple[str, ...]`) but explicitly rejects bare `str` argument so a caller passing `"grokfast"` directly doesn't iterate per-character. New `utils/recipe_dag.py` parses Data Recipe DAG with closed `NODE_KINDS = frozenset({seed, llm_text, code, judge, validator, sampler})`; Kahn's topological sort uses `collections.deque` + sorted-on-arrival (review fix HIGH — first-cut had O(N² log N) `queue.sort()` inside the BFS body); cycle / self-loop / dangling-edge / duplicate-edge / duplicate-node-name / unknown-kind rejection with specific error messages. `_MAX_NODES=256`, `_MAX_EDGES=1024`, `_MAX_FILE_BYTES=1MiB`. `parse_recipe_yaml` uses `yaml.safe_load`. `load_recipe_yaml` enforces `is_under_cwd` containment AND `os.lstat + stat.S_ISLNK` symlink rejection at the recipe path (review fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B policy; without this, a symlink under cwd pointing at `/etc/passwd` would resolve through `realpath` and load arbitrary files). `RecipeDAG` and `RecipeNode` are `@dataclass(frozen=True)`; `RecipeNode.config` is `MappingProxyType`-wrapped so caller mutation post-parse cannot affect the validated topology. `parse_recipe` does NOT mutate input dicts (matches v0.33.0 #47 `CrossDocCollator` immutability policy). Known limitations: (1) Plugin trainer-callback wiring deferred to v0.45.1 — `register_plugin` accepts hook objects and `discover_hooks` lists implementations, but no HF Trainer callback emits `pre_train` / `post_train` / `pre_step` / `post_step` yet. (2) Anthropic `/v1/messages` endpoint deferred to v0.45.1. (3) Server-side tool HTTP endpoints (python / bash / web_search) deferred to v0.45.1. (4) n-gram speculative-decoding live engine wiring deferred to v0.45.1. (5) External integrations are a catalog, not launchers — `IntegrationSpec` describes 15 ecosystem targets but no `soup deploy ` command auto-detects + invokes their CLIs (tracked for v0.46.0 Deploy Autopilot). (6) Advanced trainer plugins are an allowlist, not callbacks — `validate_trainer_plugin_list` accepts `[grokfast, spectrum, ...]` at schema validation, but no trainer wraps the upstream callbacks yet (live wiring per plugin in v0.45.1). (7) Data Recipe DAG runner deferred — `soup data recipe` validates topology and prints planned topological order; node-by-node execution against a local model is the v0.45.1 deliverable. - **v0.44.0 — Live Dashboard & UX**: 21 features across 4 Parts. New `soup_cli/utils/qr_url.py` `build_phone_url` enforces a scheme allowlist (`http` / `https`), loopback-only HTTP (RFC1918 / link-local / cloud-metadata IPs implicitly rejected because `host not in _LOOPBACK_HOSTS` raises), IPv6 literal auto-bracketing per RFC 3986, and **token routing through the URL query string `?token=…`** (review fix HIGH — the v0.44.0 first-cut put it in the URL fragment `#token=…` which never reaches the server). Token shape regex `^[A-Za-z0-9_\-]{16,128}$` matches `secrets.token_urlsafe(N)` output. New `commands/fetch.py` enforces `is_under_cwd` on `--output` AND adds **`os.lstat + S_ISLNK` symlink rejection at the write target** regardless of `--force` (review fix HIGH — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 / v0.43.0 Part C policy). Bundled-source path is `os.path.commonpath`-checked against `fetch_examples_dir()` (review fix HIGH — defends against a future catalog entry containing `..` that would escape the bundled directory). `fetch_examples_dir()` uses `os.path.realpath` not `abspath` (review fix MED — symlinked installs resolve to the real package root, not the symlink's parent). New `utils/checkpoint_trigger.py` `write_trigger` adds `os.lstat + S_ISLNK` rejection on the trigger target (review fix MED — symlink at `/.checkpoint_now` could redirect the write to any user-writable path). Trigger path stays under cwd via shared `is_under_cwd`. New `commands/llama.py` proxies to llama.cpp binaries with **`_LLAMA_ENV_ALLOWLIST` frozenset child-env filter** (review fix MED) — `HF_TOKEN` / `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / all `SOUP_*` are dropped before exec; only `PATH` / `HOME` / `USER` / `USERPROFILE` / `TMP` / `TEMP` / locale + llama.cpp-recognised `LLAMA_CPP_HOME` / `GGML_*` / `OMP_NUM_THREADS` are forwarded. Subprocess uses list args (no shell). Closed allowlist `_SUBCOMMAND_TO_BINARY: MappingProxyType` rejects unknown subcommands; `_validate_arg` rejects null-byte / newline / carriage-return / >1024-char per arg; `_MAX_ARGS=64` cap. New `utils/onboarding.py` `render_onboarding_yaml` enforces `is_under_cwd` on the `output` field at render time (review fix MED — without this, a crafted answer dict like `output: ../../etc/cron.d/x` would silently propagate into the rendered YAML). Literal allowlist on `task` (8 entries) and `quantization` (`4bit` / `8bit` / `none`); `epochs ∈ [1, 10]`; null-byte rejection on every string with field-named error message. New `utils/sweep_config.py` enforces a strict scalar allowlist (`str` / `int` / `float` / `bool`) on every param value (review fix LOW — without this, `params: {lr: [{__class__: evil}]}` would smuggle non-scalar YAML through to runtime); `_MAX_FILE_BYTES=256KB`, `_MAX_PARAM_KEYS=32`, `_MAX_VALUES_PER_KEY=64`, `_MAX_PARAM_KEY_LEN=128`. `SweepSpec.params` is `MappingProxyType[str, Tuple[Any, ...]]` for genuine `frozen=True` immutability (review fix HIGH — `frozen=True` on a `Dict[str, List[Any]]` field only prevents reassignment, not mutation). Same fix applied to `ConsolidationPlan.shard_files: Tuple[str, ...]` and `DelinearizePlan.weight_files: Tuple[str, ...]`. New `utils/gpu_monitor.py` `parse_nvidia_smi_csv` rejects null-byte in GPU name field; subprocess uses list args + 5s timeout + `OSError`/`TimeoutExpired` swallowed (never raises). `detect_apple_silicon` rewritten to use clean `platform.system()` / `platform.machine()` checks (review fix HIGH — the v0.44.0 first-cut had a parser-priority bug `if X if Y else Z:` that produced the right answer by coincidence on every platform but would silently break on any future branch refactor). New `utils/sse_train_stream.py` `TrainEvent` validates `type` against a closed `_VALID_TYPES = frozenset({metric, status, log, eval})`, rejects non-finite `ts`, rejects null-byte / >1024-char `message`. `to_payload` filters output to a closed `_ALLOWED_KEYS` allowlist (defence-in-depth against payload key drift from refactors). `format_sse_frame` emits the standard W3C SSE wire format `data: {json}\n\n` with `ensure_ascii=False`. New `utils/tool_outputs.py` `ToolOutputsBuffer` is thread-safe (`threading.Lock`) backed by `collections.deque(maxlen=_MAX_RECORDS=1000)` (review fix MED — replaces an O(N) `self.records[-N:]` list slice that triggered GC pressure on overflow); validates name (no null-byte, ≤128 chars), bool rejection on `started_ts` / `duration_ms` (matches project bool-as-int policy), `success` must be exactly `bool` not `int`, output preview truncated to 4096 chars. `ToolCallTimer.__exit__` returns `False` explicitly per project readability policy (does not suppress exceptions). New `utils/llama_server_timings.py` `parse_timings` tolerates missing keys (defaults to None), clamps `kv_cache_pct` to `[0, 100]`, rejects bool / non-finite / negative ints in numeric coercion. `format_kv_bar` validates `width ∈ (0, 200]`. New `utils/tail_latency.py` enforces `MAX_SAMPLES=1_000_000` DoS cap on every iteration; bool rejection + non-finite rejection + alpha bounds. New `soup_cli/ui/plugins/__init__.py` registry: tab name regex `^[a-z0-9][a-z0-9\-]{0,30}$`, `_MAX_TABS=32` cap, `list_tabs()` returns `MappingProxyType` defensive view, `load_plugins()` swallows per-plugin exceptions at WARNING level so one broken plugin doesn't crash the whole UI. `_NAME_DISALLOWED` in `utils/shortcuts.py` is `frozenset` (review fix LOW — was mutable `set` at module level). New `utils/ui_env.py` host validator restricts to `[a-zA-Z0-9.-:]` ≤253 chars + null-byte rejection; port `[1, 65535]`; key length cap 256 + null-byte rejection. Known limitations: (1) Live monitoring SSE endpoint deferred — schema + frame formatter ship, FastAPI route in v0.44.1. (2) Phone visibility flag wiring (`soup ui --public --auth-token` + QR print at startup) deferred to v0.44.1. (3) `soup merge-sharded-fsdp-weights` and `soup delinearize-llama4` are plan-only — torch-side runtime in v0.44.1. (4) `soup llama ` does not bundle llama.cpp; the binary must be on PATH (`shutil.which`) and an actionable `FileNotFoundError` fires when missing. (5) `soup serve --reasoning-parser ` allowlist ships, runtime hook in v0.44.1. (6) Apple Silicon `soup monitor` prints a yellow advisory and exits with `nvidia-smi unavailable` rc=1 — `powermetrics` parser in v0.44.1. - **v0.43.0 — Tracker & Eval Pro**: 18 features across 4 Parts. New `soup_cli/utils/trackers.py` ships an immutable `_REPORT_TO_BACKENDS` `MappingProxyType` allowlist (`wandb` / `tensorboard` / `mlflow` / `swanlab` / `trackio` / `none`); `validate_tracker_name` rejects non-string / empty / null-byte / >32-char inputs. PostHog telemetry is opt-IN (default OFF) via `SOUP_TELEMETRY=1` env var; `build_telemetry_payload` schema is closed-key (soup_version / command / python major.minor / os / arch / duration) — no model names, dataset paths, or config contents leak in the payload. Live PostHog network code deferred to v0.43.1. New `--tracker` flag on `soup train` mutually exclusive with `--wandb` / `--tensorboard` via `resolve_report_to`. `tracker_backend = tracker` capture in `commands/train.py` defends against the Typer parameter being shadowed by the local `ExperimentTracker()` instance (review fix HIGH #1 — silent `--tracker` drop regression). `nlg_metrics.py` is pure-math: BLEU `_lcs_length` allocates a fresh row per outer iter (review fix HIGH #2 — first-cut had a double-buffer re-zero that destroyed the last row). `effective_tokens_per_second` returns `None` on non-positive wall_clock (no fabrication). `eval/calibrate.py` `CalibrationReport` is `@dataclass(frozen=True)` with `FrozenInstanceError` test (matches v0.32.0 / v0.39.0 / v0.41.0 frozen-dataclass policy); `run_calibration` caps prompts at 10_000 (DoS defence). `eval/arena.py` `Tournament` enforces 256-model cap, 1M-match cap, `MappingProxyType` view on `ratings` so external callers cannot mutate Elo state (LOW review fix). `_validate_model_name` rejects null-byte / >128-char AND Rich markup metacharacters `[` / `]` (security review fix LOW #5 — leaderboard markup-injection defence). `update_elo` rejects bool / non-finite / NaN `k`. `eval/benchmarks_v0_43.py` ships `NEW_BENCHMARKS_V0_43 = frozenset({"ceval","cmmlu","aider_polyglot"})` + `MappingProxyType`-wrapped metadata; live Aider Polyglot runner deferred to v0.43.1. `utils/profiling_v0_43.py` `resolve_snapshot_path` rejects null-byte / `.` / `..` / path separators in `run_id`; `base_dir` rejects null-byte / absolute / `..` segments before realpath join (security review fix HIGH #1) — closes a Windows short-name path escape. `memory_snapshot_context` narrows the `RuntimeError` catch to the `record()` entry call only (review fix HIGH #2 — wide catch would have triggered `generator already executing` on user-body RuntimeError); yields `None` when torch / CUDA / `_record_memory_history` API missing. `nccl_bandwidth_check` rejects bool / non-finite / negative `measured_gb_per_sec`; reference table is `MappingProxyType`-wrapped. `utils/vscode_setup.py` `build_launch_json` rejects null-byte / newline / >512-char `config_path` so a crafted argument cannot inject Python args into the generated JSON. `write_vscode_launch` uses `os.lstat + S_ISLNK` symlink rejection at the target path regardless of `force` (security review fix MEDIUM #3 — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 policy). `utils/demo_bundles.py` `copy_bundle_to` validates every JSONL line via `json.loads`, enforces 50 MB cap, stages writes to `.tmp` with `os.lstat + S_ISLNK` rejection (security review fix HIGH #2) and `os.replace`-atomic rename on success — mid-stream rejection never leaves a partial file. New `soup data demo` Typer command: `--output` containment-checked via shared `is_under_cwd`. CLI output prints `_esc(written)` to defend against Rich markup injection through a crafted output path (review fix LOW). `_lcs_length` algorithmic correctness fix (review fix HIGH #2) + BLEU geometric-mean policy fix (returns 0 on any zero precision unless `smooth=True`) (review fix HIGH #1). All `math` imports moved to module-level (PEP 8 / project lazy-import policy: only torch / transformers / peft / trl / mlx are lazy). Known limitations: (1) MLflow / SwanLab / Trackio live integration deferred to v0.43.1 — schema accepts the names and threads into HF `TrainingArguments.report_to` but Soup does not bundle the upstream packages; HF Trainer raises a generic ImportError when the package is absent. (2) PostHog telemetry deferred — payload schema + `SOUP_TELEMETRY=1` env var ship, but no network code; `build_telemetry_payload` is purely a schema-locking stub (mirrors v0.27.0 MII / v0.37.0 multipack / v0.42.0 fsspec stub-then-live pattern). (3) Aider Polyglot live runner deferred — `aider_polyglot` is in the benchmark allowlist but live wiring requires the upstream `aider-chat` package + Docker. (4) `soup doctor --nccl` measurement CLI surface deferred — `nccl_bandwidth_check` accepts a measured value and classifies it, but does not yet measure via `torch.distributed.all_reduce`. (5) `examples/data/` fixtures shipped at the repo root (not as package data inside `soup_cli/`); `_bundle_source_path` derives `repo_root` via `str(importlib.resources.files("soup_cli"))` which is editable-install + wheel friendly but may break in zipapp / namespace-package installs. diff --git a/pyproject.toml b/pyproject.toml index a55fa8d..a8d362d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.44.0" +version = "0.45.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 851902c..ddd33f2 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.44.0" +__version__ = "0.45.0" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index 3d2da4c..8b3b178 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -148,6 +148,15 @@ app.add_typer( help="Proxy to llama.cpp binaries (cli / mtmd-cli / gguf-split / server).", ) +# v0.45.0 Part A — Plugin system CLI. +from soup_cli.commands import plugins as plugins_cmd # noqa: E402 + +app.add_typer( + plugins_cmd.app, + name="plugins", + help="List, enable, disable Soup plugins (v0.45.0).", +) + # Register data generate as a subcommand of data data.app.command(name="generate")(generate.generate) diff --git a/soup_cli/commands/data.py b/soup_cli/commands/data.py index f340ccb..c770d9a 100644 --- a/soup_cli/commands/data.py +++ b/soup_cli/commands/data.py @@ -2149,3 +2149,34 @@ def demo_bundle( console.print( f"[green]Copied bundle '{bundle.name}' to[/] {_esc(written)}" ) + + +@app.command(name="recipe") +def recipe( + path: str = typer.Argument(..., help="Path to recipe.yaml under cwd"), +) -> None: + """v0.45.0 Part E — Validate a Data Recipe DAG (live runner deferred).""" + from rich.markup import escape as _escape + + from soup_cli.utils.recipe_dag import load_recipe_yaml + + try: + dag = load_recipe_yaml(path) + except FileNotFoundError as exc: + console.print(f"[red]Recipe not found: {_escape(str(exc))}[/]") + raise typer.Exit(1) from exc + except (TypeError, ValueError) as exc: + console.print(f"[red]Invalid recipe: {_escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + f"[green]Recipe validated.[/] {len(dag.nodes)} node(s), " + f"{len(dag.edges)} edge(s)." + ) + console.print( + "Topological order: " + + ", ".join(_escape(name) for name in dag.topo_order) + ) + console.print( + "[yellow]Live runner deferred to v0.45.1.[/]" + ) diff --git a/soup_cli/commands/plugins.py b/soup_cli/commands/plugins.py new file mode 100644 index 0000000..5cb2313 --- /dev/null +++ b/soup_cli/commands/plugins.py @@ -0,0 +1,101 @@ +"""v0.45.0 Part A — `soup plugins` CLI.""" + +from __future__ import annotations + +import typer +from rich.console import Console +from rich.markup import escape +from rich.table import Table + +from soup_cli import plugins as plugins_pkg + +app = typer.Typer( + name="plugins", + help="List, enable, disable Soup plugins.", + rich_markup_mode="rich", + no_args_is_help=False, + invoke_without_command=True, +) +console = Console() + + +@app.callback() +def _default(ctx: typer.Context) -> None: + """When invoked with no subcommand, list registered plugins.""" + if ctx.invoked_subcommand is None: + _show_table() + + +@app.command("list") +def list_cmd() -> None: + """List all registered plugins.""" + _show_table() + + +@app.command("install") +def install_cmd(name: str = typer.Argument(..., help="Plugin name")) -> None: + """Install advisory — actual installation lives in v0.45.1.""" + safe = escape(name) + console.print( + f"[yellow]Plugin install for [bold]{safe}[/] is advisory in v0.45.0; " + "live install lands in v0.45.1.[/]" + ) + console.print( + "Drop your plugin module under [bold]soup_cli/plugins/[/] and call " + "[bold]register_plugin(...)[/] at import time." + ) + + +@app.command("enable") +def enable_cmd(name: str = typer.Argument(..., help="Plugin name")) -> None: + safe = escape(name) + try: + changed = plugins_pkg.enable_plugin(name) + except KeyError: + console.print(f"[red]Unknown plugin: {safe}[/]") + raise typer.Exit(code=1) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(code=2) + state = "enabled" if changed else "already enabled" + console.print(f"[green]Plugin {safe} {state}.[/]") + + +@app.command("disable") +def disable_cmd(name: str = typer.Argument(..., help="Plugin name")) -> None: + safe = escape(name) + try: + changed = plugins_pkg.disable_plugin(name) + except KeyError: + console.print(f"[red]Unknown plugin: {safe}[/]") + raise typer.Exit(code=1) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(code=2) + state = "disabled" if changed else "already disabled" + console.print(f"[yellow]Plugin {safe} {state}.[/]") + + +def _show_table() -> None: + plugins_view = plugins_pkg.list_plugins() + if not plugins_view: + console.print("[dim]No plugins registered.[/]") + return + table = Table(title="Soup plugins") + table.add_column("name") + table.add_column("version") + table.add_column("state") + table.add_column("hooks") + table.add_column("description") + for name in sorted(plugins_view): + spec = plugins_view[name] + hooks = sorted(plugins_pkg.discover_hooks(spec.plugin).keys()) + state = "[green]enabled[/]" if spec.enabled else "[yellow]disabled[/]" + table.add_row( + escape(spec.name), + escape(spec.version), + state, + ", ".join(hooks) if hooks else "[dim]none[/]", + escape(spec.description), + ) + console.print(table) diff --git a/soup_cli/plugins/__init__.py b/soup_cli/plugins/__init__.py new file mode 100644 index 0000000..04e8e76 --- /dev/null +++ b/soup_cli/plugins/__init__.py @@ -0,0 +1,318 @@ +"""v0.45.0 Part A — Plugin / hook system. + +Public API for third-party plugins. Plugins register themselves at module +import time via ``register_plugin(...)`` and provide hooks the trainer fires +at well-known points (``pre_train`` / ``post_train`` / ``pre_step`` / +``post_step``). Plugins may also register chat templates and model groups +via ``register_template`` / ``register_model_group``. + +This release ships the registry and CLI surface; live trainer-callback +wiring lands in v0.45.1 (mirrors the v0.27.0 MII / v0.37.0 multipack +stub-then-live pattern). +""" + +from __future__ import annotations + +import importlib +import logging +import pkgutil +import re +from dataclasses import dataclass +from threading import RLock +from types import MappingProxyType +from typing import ( + Any, + Callable, + Dict, + List, + Mapping, + Optional, + Protocol, + Tuple, + runtime_checkable, +) + +logger = logging.getLogger(__name__) + +# Plugin name: kebab-case, alphanumeric + hyphens; 1..40 chars, leading alnum. +_PLUGIN_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9\-]{0,39}$") +# Semver-ish: MAJOR.MINOR.PATCH with optional ``-tag`` / ``+build`` suffix. +_VERSION_RE = re.compile(r"^\d+\.\d+\.\d+(?:[\-+][A-Za-z0-9.\-]{1,32})?$") +_MAX_PLUGINS = 64 +_MAX_DESCRIPTION = 256 +_MAX_TEMPLATES_PER_PLUGIN = 32 +_MAX_MODEL_GROUPS_PER_PLUGIN = 32 +_MAX_NAME_ENTRY_LEN = 128 + +_HOOK_NAMES: Tuple[str, ...] = ( + "pre_train", + "post_train", + "pre_step", + "post_step", +) + + +@runtime_checkable +class BasePlugin(Protocol): + """Duck-typed plugin protocol. + + Plugins are objects (instance or class) carrying any subset of the + four hook methods. Each hook accepts a single ``context`` argument + (an opaque dict the trainer fills in) and returns ``None``. + """ + + def pre_train(self, context: Dict[str, Any]) -> None: ... + + def post_train(self, context: Dict[str, Any]) -> None: ... + + def pre_step(self, context: Dict[str, Any]) -> None: ... + + def post_step(self, context: Dict[str, Any]) -> None: ... + + +@dataclass(frozen=True) +class PluginSpec: + """One registered plugin.""" + + name: str + version: str + plugin: Any + description: str = "" + enabled: bool = True + templates: Tuple[str, ...] = () + model_groups: Tuple[str, ...] = () + + +_PLUGINS: Dict[str, PluginSpec] = {} +_LOCK = RLock() + + +def _validate_name(name: str) -> None: + if not isinstance(name, str): + raise TypeError("plugin name must be a string") + if not _PLUGIN_NAME_RE.match(name): + raise ValueError( + "plugin name must be kebab-case ([a-z0-9][a-z0-9-]{0,39})" + ) + + +def _validate_version(version: str) -> None: + if not isinstance(version, str): + raise TypeError("plugin version must be a string") + if not _VERSION_RE.match(version): + raise ValueError( + "plugin version must match MAJOR.MINOR.PATCH (semver)" + ) + + +def _validate_description(description: str) -> None: + if not isinstance(description, str): + raise TypeError("description must be a string") + if "\x00" in description: + raise ValueError("description must not contain null bytes") + if len(description) > _MAX_DESCRIPTION: + raise ValueError(f"description exceeds {_MAX_DESCRIPTION} chars") + + +def list_hook_names() -> Tuple[str, ...]: + """Return the canonical hook names recognised by the trainer.""" + return _HOOK_NAMES + + +def discover_hooks(plugin: Any) -> Dict[str, Callable[[Dict[str, Any]], None]]: + """Return the subset of canonical hooks the plugin actually implements. + + A hook is considered implemented when ``getattr(plugin, name)`` is a + callable. Missing or non-callable attributes are silently skipped — + plugins are not required to implement every hook. + """ + found: Dict[str, Callable[[Dict[str, Any]], None]] = {} + for hook in _HOOK_NAMES: + candidate = getattr(plugin, hook, None) + if callable(candidate): + found[hook] = candidate + return found + + +def register_plugin( + *, + name: str, + version: str, + plugin: Any, + description: str = "", + templates: Optional[List[str]] = None, + model_groups: Optional[List[str]] = None, +) -> PluginSpec: + """Register a plugin. Idempotent for an identical spec; rejects + re-registration with a different version or plugin object.""" + _validate_name(name) + _validate_version(version) + _validate_description(description) + if plugin is None: + raise ValueError("plugin object must not be None") + # Hook discovery is best-effort: we don't require any hook, but at + # least one of {hooks, templates, model_groups} must be non-empty so a + # totally-empty plugin is rejected loudly. + hooks = discover_hooks(plugin) + tpls = tuple(templates or ()) + grps = tuple(model_groups or ()) + if len(tpls) > _MAX_TEMPLATES_PER_PLUGIN: + raise ValueError( + f"templates exceeds {_MAX_TEMPLATES_PER_PLUGIN} entries" + ) + if len(grps) > _MAX_MODEL_GROUPS_PER_PLUGIN: + raise ValueError( + f"model_groups exceeds {_MAX_MODEL_GROUPS_PER_PLUGIN} entries" + ) + for tpl in tpls: + if not isinstance(tpl, str) or not tpl or "\x00" in tpl: + raise ValueError("template name must be non-empty NUL-free str") + if len(tpl) > _MAX_NAME_ENTRY_LEN: + raise ValueError( + f"template name exceeds {_MAX_NAME_ENTRY_LEN} chars" + ) + for grp in grps: + if not isinstance(grp, str) or not grp or "\x00" in grp: + raise ValueError("model_group name must be non-empty NUL-free str") + if len(grp) > _MAX_NAME_ENTRY_LEN: + raise ValueError( + f"model_group name exceeds {_MAX_NAME_ENTRY_LEN} chars" + ) + if not hooks and not tpls and not grps: + raise ValueError( + "plugin must implement at least one hook OR register a template " + "OR register a model group" + ) + spec = PluginSpec( + name=name, + version=version, + plugin=plugin, + description=description, + templates=tpls, + model_groups=grps, + ) + with _LOCK: + if len(_PLUGINS) >= _MAX_PLUGINS and name not in _PLUGINS: + raise RuntimeError(f"too many plugins (max {_MAX_PLUGINS})") + existing = _PLUGINS.get(name) + if existing is not None: + if ( + existing.version != version + or existing.plugin is not plugin + or existing.templates != tpls + or existing.model_groups != grps + or existing.description != description + ): + raise ValueError( + f"plugin {name!r} already registered with a different spec" + ) + # Identical re-register: keep enabled state. + return existing + _PLUGINS[name] = spec + return spec + + +def list_plugins() -> Mapping[str, PluginSpec]: + """Return an immutable view of registered plugins.""" + with _LOCK: + return MappingProxyType(dict(_PLUGINS)) + + +def get_plugin(name: str) -> Optional[PluginSpec]: + """Return the registered plugin spec for ``name``, or ``None``.""" + if not isinstance(name, str): + return None + with _LOCK: + return _PLUGINS.get(name) + + +def enable_plugin(name: str) -> bool: + """Mark a registered plugin enabled. Returns True iff it changed state.""" + _validate_name(name) + with _LOCK: + existing = _PLUGINS.get(name) + if existing is None: + raise KeyError(name) + if existing.enabled: + return False + _PLUGINS[name] = PluginSpec( + name=existing.name, + version=existing.version, + plugin=existing.plugin, + description=existing.description, + enabled=True, + templates=existing.templates, + model_groups=existing.model_groups, + ) + return True + + +def disable_plugin(name: str) -> bool: + """Mark a registered plugin disabled. Returns True iff it changed state.""" + _validate_name(name) + with _LOCK: + existing = _PLUGINS.get(name) + if existing is None: + raise KeyError(name) + if not existing.enabled: + return False + _PLUGINS[name] = PluginSpec( + name=existing.name, + version=existing.version, + plugin=existing.plugin, + description=existing.description, + enabled=False, + templates=existing.templates, + model_groups=existing.model_groups, + ) + return True + + +def is_enabled(name: str) -> bool: + """Return True iff ``name`` is registered and enabled.""" + spec = get_plugin(name) + return bool(spec and spec.enabled) + + +def clear_plugins() -> None: + """Remove all registered plugins. Used by tests.""" + with _LOCK: + _PLUGINS.clear() + + +def load_plugins() -> int: + """Import every ``soup_cli.plugins.*`` submodule. Returns count loaded. + + Plugin failures are caught and logged at WARNING — one bad plugin + must not crash ``soup`` startup (mirrors the v0.44.0 Web UI plugin + loader policy). + """ + count = 0 + pkg = importlib.import_module(__name__) + for module_info in pkgutil.iter_modules(pkg.__path__): + if module_info.name.startswith("_"): + continue + try: + importlib.import_module(f"{__name__}.{module_info.name}") + count += 1 + except Exception: # noqa: BLE001 — plugin failure must not crash CLI + logger.exception( + "Failed to load Soup plugin: %s", module_info.name + ) + return count + + +__all__ = [ + "BasePlugin", + "PluginSpec", + "discover_hooks", + "list_hook_names", + "register_plugin", + "list_plugins", + "get_plugin", + "enable_plugin", + "disable_plugin", + "is_enabled", + "clear_plugins", + "load_plugins", +] diff --git a/soup_cli/utils/anthropic_messages.py b/soup_cli/utils/anthropic_messages.py new file mode 100644 index 0000000..bcf40a2 --- /dev/null +++ b/soup_cli/utils/anthropic_messages.py @@ -0,0 +1,214 @@ +"""v0.45.0 Part B — Anthropic Messages API converter (schema-only). + +Pure-Python converter between OpenAI ``/v1/chat/completions`` payloads and +Anthropic ``/v1/messages`` payloads. The wire-up of the Anthropic-shaped +endpoint inside ``soup serve`` is deferred to v0.45.1 (matches the +project's stub-then-live policy). + +Surface kept narrow on purpose: + +- ``to_anthropic(openai_payload)`` -> dict in Anthropic shape +- ``from_anthropic(anthropic_payload)`` -> dict in OpenAI shape +- ``validate_anthropic_payload(p)`` -> raises on schema violations +""" + +from __future__ import annotations + +from typing import Any, Dict, List + +# Caps mirror the v0.30.0 inference-server caps (max_tokens) and v0.40.3 +# trace-log message-size policy. +_MAX_MESSAGES = 1024 +_MAX_CONTENT_LEN = 1_048_576 # 1 MiB per message +_MAX_TOKENS_CAP = 16384 # matches /v1/chat/completions + +_VALID_ROLES_OPENAI = frozenset({"system", "user", "assistant", "tool"}) +_VALID_ROLES_ANTHROPIC = frozenset({"user", "assistant"}) + + +def _check_str(value: Any, name: str, *, max_len: int = _MAX_CONTENT_LEN) -> str: + if not isinstance(value, str): + raise TypeError(f"{name} must be a string") + if "\x00" in value: + raise ValueError(f"{name} must not contain null bytes") + if len(value) > max_len: + raise ValueError(f"{name} exceeds {max_len} chars") + return value + + +def _check_messages(messages: Any) -> List[Dict[str, Any]]: + if not isinstance(messages, list): + raise TypeError("messages must be a list") + if not messages: + raise ValueError("messages must not be empty") + if len(messages) > _MAX_MESSAGES: + raise ValueError(f"messages exceeds {_MAX_MESSAGES} entries") + out: List[Dict[str, Any]] = [] + for index, message in enumerate(messages): + if not isinstance(message, dict): + raise TypeError(f"messages[{index}] must be a dict") + out.append(message) + return out + + +def to_anthropic(openai_payload: Dict[str, Any]) -> Dict[str, Any]: + """Convert an OpenAI chat-completions payload to Anthropic Messages shape. + + - The (single) ``system`` message becomes the top-level ``system`` field. + - ``user`` / ``assistant`` messages are passed through. + - ``tool`` messages are surfaced as ``tool_result`` content blocks under + a ``user`` role (Anthropic convention). + - ``max_tokens`` is required by Anthropic and defaults to ``1024`` if + missing on the OpenAI side; capped at ``_MAX_TOKENS_CAP``. + """ + if not isinstance(openai_payload, dict): + raise TypeError("openai_payload must be a dict") + messages = _check_messages(openai_payload.get("messages")) + model = _check_str(openai_payload.get("model", ""), "model", max_len=256) + + system_text: List[str] = [] + out_messages: List[Dict[str, Any]] = [] + for index, message in enumerate(messages): + role = message.get("role") + if not isinstance(role, str) or role not in _VALID_ROLES_OPENAI: + raise ValueError( + f"messages[{index}].role must be one of {_VALID_ROLES_OPENAI}" + ) + content = message.get("content", "") + if isinstance(content, str): + _check_str(content, f"messages[{index}].content") + elif isinstance(content, list): + # Pass-through structured content (e.g. multi-modal). Caller is + # responsible for shape validity beyond NUL-byte checks. + for inner in content: + if isinstance(inner, dict) and isinstance(inner.get("text"), str): + _check_str(inner["text"], f"messages[{index}].content[].text") + else: + raise TypeError( + f"messages[{index}].content must be str or list" + ) + + if role == "system": + if isinstance(content, str): + system_text.append(content) + else: + raise TypeError("system message content must be a string") + continue + if role == "tool": + tool_call_id = _check_str( + message.get("tool_call_id", ""), + f"messages[{index}].tool_call_id", + max_len=256, + ) + # Structured (list) content from OpenAI is concatenated into a + # single text block instead of being silently dropped — Anthropic + # ``tool_result`` accepts either str or a list of text/image + # blocks, but we keep the conversion lossless for the common + # text-only path. Non-text inner items are stringified via repr. + if isinstance(content, str): + tool_content: Any = content + elif isinstance(content, list): + parts: List[str] = [] + for inner in content: + if isinstance(inner, dict) and isinstance( + inner.get("text"), str + ): + parts.append(inner["text"]) + elif isinstance(inner, str): + parts.append(inner) + tool_content = "\n".join(parts) + else: + raise TypeError( + f"messages[{index}].content must be str or list" + ) + out_messages.append( + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": tool_call_id, + "content": tool_content, + } + ], + } + ) + continue + out_messages.append({"role": role, "content": content}) + + raw_max = openai_payload.get("max_tokens", 1024) + if isinstance(raw_max, bool) or not isinstance(raw_max, int): + raise TypeError("max_tokens must be an int") + if raw_max < 1: + raise ValueError("max_tokens must be >= 1") + max_tokens = min(raw_max, _MAX_TOKENS_CAP) + + out: Dict[str, Any] = { + "model": model, + "messages": out_messages, + "max_tokens": max_tokens, + } + if system_text: + out["system"] = "\n\n".join(system_text) + if "temperature" in openai_payload: + temperature = openai_payload["temperature"] + if isinstance(temperature, bool) or not isinstance(temperature, (int, float)): + raise TypeError("temperature must be int or float") + if not 0.0 <= float(temperature) <= 2.0: + raise ValueError("temperature must be in [0.0, 2.0]") + out["temperature"] = float(temperature) + return out + + +def from_anthropic(anthropic_payload: Dict[str, Any]) -> Dict[str, Any]: + """Convert an Anthropic Messages payload back to OpenAI chat shape.""" + validate_anthropic_payload(anthropic_payload) + out_messages: List[Dict[str, Any]] = [] + system_field = anthropic_payload.get("system") + if isinstance(system_field, str) and system_field: + out_messages.append({"role": "system", "content": system_field}) + for message in anthropic_payload["messages"]: + # validate_anthropic_payload above guarantees the role key is + # present and on the allowlist; ``.get`` is defence-in-depth so a + # post-validation iteration cannot raise ``KeyError``. + out_messages.append( + { + "role": message.get("role"), + "content": message.get("content", ""), + } + ) + return { + "model": anthropic_payload["model"], + "messages": out_messages, + "max_tokens": anthropic_payload["max_tokens"], + } + + +def validate_anthropic_payload(payload: Dict[str, Any]) -> None: + """Raise if the Anthropic payload is malformed.""" + if not isinstance(payload, dict): + raise TypeError("payload must be a dict") + _check_str(payload.get("model", ""), "model", max_len=256) + messages = _check_messages(payload.get("messages")) + for index, message in enumerate(messages): + role = message.get("role") + if role not in _VALID_ROLES_ANTHROPIC: + raise ValueError( + f"messages[{index}].role must be one of {_VALID_ROLES_ANTHROPIC}" + ) + raw_max = payload.get("max_tokens") + if isinstance(raw_max, bool) or not isinstance(raw_max, int): + raise TypeError("max_tokens must be an int") + if raw_max < 1 or raw_max > _MAX_TOKENS_CAP: + raise ValueError(f"max_tokens must be in [1, {_MAX_TOKENS_CAP}]") + if "system" in payload: + if not isinstance(payload["system"], str): + raise TypeError("system must be a string") + _check_str(payload["system"], "system") + + +__all__ = [ + "to_anthropic", + "from_anthropic", + "validate_anthropic_payload", +] diff --git a/soup_cli/utils/integrations.py b/soup_cli/utils/integrations.py new file mode 100644 index 0000000..1d9ff74 --- /dev/null +++ b/soup_cli/utils/integrations.py @@ -0,0 +1,161 @@ +"""v0.45.0 Part C — External integrations catalog (schema-only). + +Closed allowlist of integration descriptors so a future ``soup deploy +`` can know about LM Studio, ComfyUI, stable-diffusion.cpp, +Open WebUI, etc. without each command growing its own ad-hoc list. Live +launch wiring lands in v0.45.1. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from types import MappingProxyType +from typing import Mapping, Tuple + +_INTEGRATION_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9\-]{0,31}$") +_MAX_DESCRIPTION = 256 + + +@dataclass(frozen=True) +class IntegrationSpec: + """One external integration entry.""" + + name: str + description: str + target_artifacts: Tuple[str, ...] + + +def _make( + name: str, + description: str, + target_artifacts: Tuple[str, ...], +) -> IntegrationSpec: + if not _INTEGRATION_NAME_RE.match(name): + raise ValueError( + "integration name must be kebab-case ([a-z0-9][a-z0-9-]{0,31})" + ) + if not isinstance(description, str) or "\x00" in description: + raise ValueError("description must be a NUL-free string") + if len(description) > _MAX_DESCRIPTION: + raise ValueError(f"description exceeds {_MAX_DESCRIPTION} chars") + if not isinstance(target_artifacts, tuple) or not target_artifacts: + raise ValueError("target_artifacts must be a non-empty tuple") + for artifact in target_artifacts: + if not isinstance(artifact, str) or not artifact: + raise ValueError("target_artifacts entries must be non-empty str") + return IntegrationSpec( + name=name, description=description, target_artifacts=target_artifacts + ) + + +_BUILTIN: Mapping[str, IntegrationSpec] = MappingProxyType( + { + "lm-studio": _make( + "lm-studio", + "Push merged GGUF to LM Studio via the lms CLI", + ("gguf",), + ), + "comfyui": _make( + "comfyui", + "Export node-graph compatible models for ComfyUI", + ("safetensors", "gguf"), + ), + "stable-diffusion-cpp": _make( + "stable-diffusion-cpp", + "Image-output model export for stable-diffusion.cpp", + ("safetensors",), + ), + "open-webui": _make( + "open-webui", + "Auto-register the served model with Open WebUI", + ("served-endpoint",), + ), + "ollama": _make( + "ollama", + "Deploy GGUF model to local Ollama instance", + ("gguf",), + ), + "tei": _make( + "tei", + "HuggingFace Text-Embeddings-Inference deployment", + ("safetensors",), + ), + "pgvector": _make( + "pgvector", + "Postgres pgvector schema + connection pattern", + ("safetensors",), + ), + "faiss": _make( + "faiss", + "FAISS index pattern for embedding-model output", + ("safetensors",), + ), + "weaviate": _make( + "weaviate", + "Weaviate vector-DB connection pattern", + ("safetensors",), + ), + "sentence-transformers": _make( + "sentence-transformers", + "Sentence-Transformers compatible embedding export", + ("safetensors",), + ), + "claude-code": _make( + "claude-code", + "Claude Code client SDK integration example", + ("served-endpoint",), + ), + "cursor": _make( + "cursor", + "Cursor IDE custom-model connection guide", + ("served-endpoint",), + ), + "continue": _make( + "continue", + "Continue VS Code extension custom-model guide", + ("served-endpoint",), + ), + "cline": _make( + "cline", + "Cline (autonomous IDE agent) connection guide", + ("served-endpoint",), + ), + "sillytavern": _make( + "sillytavern", + "SillyTavern chat front-end connection guide", + ("served-endpoint",), + ), + } +) + + +def list_integrations() -> Mapping[str, IntegrationSpec]: + """Return an immutable view of the integration registry.""" + return _BUILTIN + + +def get_integration(name: str) -> IntegrationSpec: + """Return the integration entry for ``name`` or raise ``KeyError``.""" + if not isinstance(name, str): + raise TypeError("name must be a string") + canonical = name.strip().lower() + if not canonical or "\x00" in canonical: + raise ValueError("name must be a non-empty NUL-free string") + if canonical not in _BUILTIN: + raise KeyError(canonical) + return _BUILTIN[canonical] + + +def has_integration(name: str) -> bool: + if not isinstance(name, str): + return False + return name.strip().lower() in _BUILTIN + + +__all__ = [ + "IntegrationSpec", + "list_integrations", + "get_integration", + "has_integration", +] diff --git a/soup_cli/utils/ngram_spec.py b/soup_cli/utils/ngram_spec.py new file mode 100644 index 0000000..d002b9d --- /dev/null +++ b/soup_cli/utils/ngram_spec.py @@ -0,0 +1,71 @@ +"""v0.45.0 Part B — n-gram-mod speculative decoding (schema-only). + +Validates a small configuration surface; the live engine wiring inside +the inference loop lands in v0.45.1. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +_MIN_N = 1 +_MAX_N = 8 +_MIN_DRAFT = 1 +_MAX_DRAFT = 32 +_MIN_PROMPT = 0 +_MAX_PROMPT = 1_048_576 + + +@dataclass(frozen=True) +class NgramSpecConfig: + """N-gram speculative-decoding configuration.""" + + n: int + num_draft_tokens: int = 4 + prompt_lookup_max: int = 0 # 0 disables prompt-lookup heuristic + + +def validate_ngram_n(n: int) -> int: + if isinstance(n, bool) or not isinstance(n, int): + raise TypeError("n must be an int") + if n < _MIN_N or n > _MAX_N: + raise ValueError(f"n must be in [{_MIN_N}, {_MAX_N}]") + return n + + +def validate_num_draft_tokens(value: int) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError("num_draft_tokens must be an int") + if value < _MIN_DRAFT or value > _MAX_DRAFT: + raise ValueError( + f"num_draft_tokens must be in [{_MIN_DRAFT}, {_MAX_DRAFT}]" + ) + return value + + +def validate_prompt_lookup_max(value: int) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError("prompt_lookup_max must be an int") + if value < _MIN_PROMPT or value > _MAX_PROMPT: + raise ValueError( + f"prompt_lookup_max must be in [{_MIN_PROMPT}, {_MAX_PROMPT}]" + ) + return value + + +def validate_ngram_config(config: NgramSpecConfig) -> NgramSpecConfig: + if not isinstance(config, NgramSpecConfig): + raise TypeError("config must be an NgramSpecConfig") + validate_ngram_n(config.n) + validate_num_draft_tokens(config.num_draft_tokens) + validate_prompt_lookup_max(config.prompt_lookup_max) + return config + + +__all__ = [ + "NgramSpecConfig", + "validate_ngram_config", + "validate_ngram_n", + "validate_num_draft_tokens", + "validate_prompt_lookup_max", +] diff --git a/soup_cli/utils/recipe_dag.py b/soup_cli/utils/recipe_dag.py new file mode 100644 index 0000000..ca51be7 --- /dev/null +++ b/soup_cli/utils/recipe_dag.py @@ -0,0 +1,239 @@ +"""v0.45.0 Part E — Data Recipe DAG validators (schema-only). + +Parses a YAML recipe describing a Seed -> LLM Text -> Code -> Judge -> +Validators -> Sampler graph and validates the topology. The live runner +(execution against a local model) is deferred to v0.45.1. +""" + +from __future__ import annotations + +import os +import re +import stat +from collections import deque +from dataclasses import dataclass +from types import MappingProxyType +from typing import Any, Dict, List, Mapping, Tuple + +# Closed allowlist of node kinds. Mirrors the unsloth Studio schema. +_NODE_KINDS = ( + "seed", + "llm_text", + "code", + "judge", + "validator", + "sampler", +) +NODE_KINDS: frozenset = frozenset(_NODE_KINDS) + +# Per-recipe caps — defence-in-depth against pathological YAML. +_MAX_NODES = 256 +_MAX_EDGES = 1024 +_MAX_NAME_LEN = 64 +_MAX_FILE_BYTES = 1_048_576 # 1 MiB + +_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_\-]{0,63}$") + + +@dataclass(frozen=True) +class RecipeNode: + """One node in the data-recipe DAG.""" + + name: str + kind: str + config: Mapping[str, Any] + + +@dataclass(frozen=True) +class RecipeDAG: + """Validated recipe topology.""" + + nodes: Tuple[RecipeNode, ...] + edges: Tuple[Tuple[str, str], ...] + topo_order: Tuple[str, ...] + + +def _check_name(name: str) -> str: + if not isinstance(name, str): + raise TypeError("node name must be a string") + if "\x00" in name: + raise ValueError("node name must not contain null bytes") + if not _NAME_RE.match(name): + raise ValueError( + f"node name must match [a-z0-9][a-z0-9_-]{{0,{_MAX_NAME_LEN - 1}}}" + ) + return name + + +def _check_kind(kind: str) -> str: + if not isinstance(kind, str): + raise TypeError("node kind must be a string") + canonical = kind.strip().lower() + if canonical not in NODE_KINDS: + raise ValueError( + f"unknown node kind: {kind!r}. supported: {sorted(NODE_KINDS)}" + ) + return canonical + + +def _topological_sort( + names: List[str], edges: List[Tuple[str, str]] +) -> List[str]: + """Kahn's algorithm. Raises ``ValueError`` on cycle. + + Uses ``collections.deque`` so each pop is O(1); successors that become + zero in-degree on the same step are appended in sorted order so the + output is deterministic without re-sorting the queue every iteration. + """ + in_degree: Dict[str, int] = {name: 0 for name in names} + successors: Dict[str, List[str]] = {name: [] for name in names} + for source, target in edges: + in_degree[target] += 1 + successors[source].append(target) + queue: deque = deque( + sorted(name for name, deg in in_degree.items() if deg == 0) + ) + order: List[str] = [] + while queue: + current = queue.popleft() + order.append(current) + ready: List[str] = [] + for successor in successors[current]: + in_degree[successor] -= 1 + if in_degree[successor] == 0: + ready.append(successor) + ready.sort() + queue.extend(ready) + if len(order) != len(names): + raise ValueError("recipe DAG contains a cycle") + return order + + +def parse_recipe(raw: Any) -> RecipeDAG: + """Validate and topologically sort a recipe dict. + + Required shape:: + + { + "nodes": [{"name": "...", "kind": "...", "config": {...}}, ...], + "edges": [["from_name", "to_name"], ...] + } + """ + if not isinstance(raw, dict): + raise TypeError("recipe must be a dict") + raw_nodes = raw.get("nodes") + if not isinstance(raw_nodes, list) or not raw_nodes: + raise ValueError("recipe.nodes must be a non-empty list") + if len(raw_nodes) > _MAX_NODES: + raise ValueError(f"recipe.nodes exceeds {_MAX_NODES} entries") + + nodes: List[RecipeNode] = [] + seen_names: set[str] = set() + for index, raw_node in enumerate(raw_nodes): + if not isinstance(raw_node, dict): + raise TypeError(f"recipe.nodes[{index}] must be a dict") + name = _check_name(raw_node.get("name", "")) + if name in seen_names: + raise ValueError(f"duplicate node name: {name!r}") + seen_names.add(name) + kind = _check_kind(raw_node.get("kind", "")) + config = raw_node.get("config", {}) + if not isinstance(config, dict): + raise TypeError(f"recipe.nodes[{index}].config must be a dict") + nodes.append( + RecipeNode(name=name, kind=kind, config=MappingProxyType(dict(config))) + ) + + raw_edges = raw.get("edges", []) + if not isinstance(raw_edges, list): + raise TypeError("recipe.edges must be a list") + if len(raw_edges) > _MAX_EDGES: + raise ValueError(f"recipe.edges exceeds {_MAX_EDGES} entries") + edges: List[Tuple[str, str]] = [] + edge_seen: set[Tuple[str, str]] = set() + name_set = {node.name for node in nodes} + for index, raw_edge in enumerate(raw_edges): + if not isinstance(raw_edge, (list, tuple)) or len(raw_edge) != 2: + raise ValueError( + f"recipe.edges[{index}] must be a 2-element [from, to] list" + ) + source = _check_name(raw_edge[0]) + target = _check_name(raw_edge[1]) + if source == target: + raise ValueError(f"self-loop edge rejected: {source!r}") + if source not in name_set: + raise ValueError(f"edge source {source!r} not in nodes") + if target not in name_set: + raise ValueError(f"edge target {target!r} not in nodes") + key = (source, target) + if key in edge_seen: + raise ValueError(f"duplicate edge: {source!r} -> {target!r}") + edge_seen.add(key) + edges.append(key) + + topo = _topological_sort([n.name for n in nodes], edges) + return RecipeDAG( + nodes=tuple(nodes), + edges=tuple(edges), + topo_order=tuple(topo), + ) + + +def parse_recipe_yaml(text: str) -> RecipeDAG: + """Parse a YAML string into a validated ``RecipeDAG``.""" + if not isinstance(text, str): + raise TypeError("text must be a string") + if "\x00" in text: + raise ValueError("recipe text must not contain null bytes") + if len(text.encode("utf-8")) > _MAX_FILE_BYTES: + raise ValueError(f"recipe text exceeds {_MAX_FILE_BYTES} bytes") + import yaml # lazy import — keep CLI startup fast + + try: + data = yaml.safe_load(text) + except yaml.YAMLError as exc: + raise ValueError(f"invalid YAML: {exc}") from exc + return parse_recipe(data) + + +def load_recipe_yaml(path: str) -> RecipeDAG: + """Load a recipe YAML from a path under cwd.""" + from soup_cli.utils.paths import is_under_cwd + + if not isinstance(path, str) or not path: + raise ValueError("path must be a non-empty string") + if "\x00" in path: + raise ValueError("path must not contain null bytes") + # Symlink rejection on the *original* path (TOCTOU policy mirroring + # v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B). ``realpath`` below + # would already follow symlinks, but we want to reject them as a + # defence layer rather than silently follow. + try: + lstat_result = os.lstat(path) + except FileNotFoundError as exc: + raise FileNotFoundError(path) from exc + if stat.S_ISLNK(lstat_result.st_mode): + raise ValueError( + f"recipe path must not be a symlink: {os.path.basename(path)}" + ) + real = os.path.realpath(path) + if not is_under_cwd(real): + raise ValueError(f"recipe path must stay under cwd: {os.path.basename(real)}") + if not os.path.isfile(real): + raise FileNotFoundError(real) + size = os.path.getsize(real) + if size > _MAX_FILE_BYTES: + raise ValueError(f"recipe file exceeds {_MAX_FILE_BYTES} bytes") + with open(real, "r", encoding="utf-8") as handle: + text = handle.read() + return parse_recipe_yaml(text) + + +__all__ = [ + "NODE_KINDS", + "RecipeDAG", + "RecipeNode", + "parse_recipe", + "parse_recipe_yaml", + "load_recipe_yaml", +] diff --git a/soup_cli/utils/server_tools.py b/soup_cli/utils/server_tools.py new file mode 100644 index 0000000..8361520 --- /dev/null +++ b/soup_cli/utils/server_tools.py @@ -0,0 +1,179 @@ +"""v0.45.0 Part B — Server-side tools registry (schema-only). + +Exposes a closed allowlist of safe tools that the inference server can offer +to a model: ``python``, ``bash``, ``web_search``. Live wiring (HTTP tool +endpoints) is deferred to v0.45.1; this module locks the public schema + +domain allowlist so the configuration surface is stable now. + +Defence-in-depth choices: + +- ``python`` and ``bash`` reuse the v0.25.0 RLVR sandbox (5 s timeout, + RLIMIT_AS, ephemeral cwd, socket patch) — schema only here. +- ``web_search`` requires every domain on a closed allowlist; subdomain + matches require a leading dot (``foo.example.com`` matches + ``example.com`` only when the allow entry is ``.example.com``). +- ``rate_limit`` per tool uses the v0.20.0 [1, 600] requests-per-minute + window so a misbehaving plugin cannot DoS the server. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from types import MappingProxyType +from typing import Mapping, Optional, Tuple + +# Tool name allowlist — closed set, used as map keys. +_TOOL_NAMES = ("python", "bash", "web_search") +SUPPORTED_TOOLS: frozenset = frozenset(_TOOL_NAMES) + +# Default web_search domain allowlist. Operators must override / extend +# explicitly via ``WebSearchConfig.domain_allowlist``; the empty default +# is intentionally a hard "deny all". +_DEFAULT_WEB_DOMAINS: Tuple[str, ...] = () + +_DOMAIN_RE = re.compile(r"^\.?[a-z0-9]([a-z0-9.\-]{0,253})$") +_MAX_DOMAINS = 64 +_MAX_DOMAIN_LEN = 253 +_MIN_RPM = 1 +_MAX_RPM = 600 + + +@dataclass(frozen=True) +class WebSearchConfig: + """Web-search tool configuration.""" + + domain_allowlist: Tuple[str, ...] = _DEFAULT_WEB_DOMAINS + rate_limit_per_minute: int = 30 + + +@dataclass(frozen=True) +class ToolSpec: + """One server-side tool registration.""" + + name: str + description: str + rate_limit_per_minute: int + web_search: Optional[WebSearchConfig] = None + + +def validate_tool_name(name: str) -> str: + """Validate ``name`` is one of the allowlisted tools.""" + if not isinstance(name, str): + raise TypeError("tool name must be a string") + canonical = name.strip().lower() + if not canonical: + raise ValueError("tool name must be non-empty") + if "\x00" in canonical: + raise ValueError("tool name must not contain null bytes") + if canonical not in SUPPORTED_TOOLS: + raise ValueError( + f"unknown tool: {canonical!r}. supported: {sorted(SUPPORTED_TOOLS)}" + ) + return canonical + + +def validate_rate_limit(rpm: int) -> int: + if isinstance(rpm, bool) or not isinstance(rpm, int): + raise TypeError("rate_limit_per_minute must be an int") + if rpm < _MIN_RPM or rpm > _MAX_RPM: + raise ValueError( + f"rate_limit_per_minute must be in [{_MIN_RPM}, {_MAX_RPM}]" + ) + return rpm + + +def validate_domain(domain: str) -> str: + if not isinstance(domain, str): + raise TypeError("domain must be a string") + candidate = domain.strip().lower() + if not candidate: + raise ValueError("domain must be non-empty") + if "\x00" in candidate: + raise ValueError("domain must not contain null bytes") + if len(candidate) > _MAX_DOMAIN_LEN: + raise ValueError(f"domain exceeds {_MAX_DOMAIN_LEN} chars") + if "/" in candidate or " " in candidate: + raise ValueError("domain must not contain '/' or whitespace") + if not _DOMAIN_RE.match(candidate): + raise ValueError(f"invalid domain syntax: {candidate!r}") + return candidate + + +def validate_web_search_config(config: WebSearchConfig) -> WebSearchConfig: + if not isinstance(config, WebSearchConfig): + raise TypeError("config must be a WebSearchConfig") + if len(config.domain_allowlist) > _MAX_DOMAINS: + raise ValueError(f"domain_allowlist exceeds {_MAX_DOMAINS} entries") + seen = set() + for raw in config.domain_allowlist: + canonical = validate_domain(raw) + if canonical in seen: + raise ValueError(f"duplicate domain: {canonical!r}") + seen.add(canonical) + validate_rate_limit(config.rate_limit_per_minute) + return config + + +def is_domain_allowed(host: str, allowlist: Tuple[str, ...]) -> bool: + """Return True iff ``host`` is permitted by ``allowlist``. + + Bare entries (``example.com``) match the host exactly. Entries with a + leading dot (``.example.com``) also match any subdomain + (``a.example.com``). Both forms are validated by ``validate_domain``. + """ + if not isinstance(host, str) or not host: + return False + canonical = host.strip().lower() + if not canonical or "\x00" in canonical: + return False + # Strip optional ``:port`` suffix and bracketed IPv6 noise so an + # ``Authority``-form value like ``api.example.com:443`` matches the + # bare allowlist entry ``api.example.com`` instead of silently + # missing. + if canonical.startswith("[") and "]" in canonical: + # IPv6 literal — never matches a domain allowlist; deny. + return False + if ":" in canonical: + canonical = canonical.split(":", 1)[0] + if not canonical: + return False + for raw in allowlist: + rule = raw.strip().lower() + if not rule: + continue + if rule.startswith("."): + base = rule[1:] + if canonical == base or canonical.endswith(rule): + return True + elif canonical == rule: + return True + return False + + +_TOOL_DESCRIPTIONS: Mapping[str, str] = MappingProxyType( + { + "python": "Sandboxed Python execution (RLVR sandbox)", + "bash": "Sandboxed bash execution (RLVR sandbox)", + "web_search": "Web search constrained to a domain allowlist", + } +) + + +def tool_description(name: str) -> str: + """Return the canonical description for a supported tool.""" + canonical = validate_tool_name(name) + return _TOOL_DESCRIPTIONS[canonical] + + +__all__ = [ + "SUPPORTED_TOOLS", + "ToolSpec", + "WebSearchConfig", + "is_domain_allowed", + "tool_description", + "validate_domain", + "validate_rate_limit", + "validate_tool_name", + "validate_web_search_config", +] diff --git a/soup_cli/utils/trainer_plugins.py b/soup_cli/utils/trainer_plugins.py new file mode 100644 index 0000000..bc1053e --- /dev/null +++ b/soup_cli/utils/trainer_plugins.py @@ -0,0 +1,136 @@ +"""v0.45.0 Part D — Advanced trainer-plugin allowlist (schema-only). + +Closed allowlist of optional trainer plugins so a future +``training.trainer_plugins: [grokfast, spectrum, ...]`` Pydantic field +can validate against a stable surface. Live wiring (callbacks, kernel +swaps, LLMCompressor passes) is deferred to v0.45.1. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from types import MappingProxyType +from typing import Mapping, Optional, Sequence, Tuple + +_PLUGIN_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_]{0,31}$") +_MAX_DESCRIPTION = 256 +_MAX_PLUGINS_PER_RUN = 8 + + +@dataclass(frozen=True) +class TrainerPluginSpec: + """One advanced trainer plugin descriptor.""" + + name: str + description: str + required_package: Optional[str] + + +def _make( + name: str, + description: str, + required_package: Optional[str], +) -> TrainerPluginSpec: + if not _PLUGIN_NAME_RE.match(name): + raise ValueError( + "trainer-plugin name must be snake_case ([a-z0-9][a-z0-9_]{0,31})" + ) + if not isinstance(description, str) or "\x00" in description: + raise ValueError("description must be a NUL-free string") + if len(description) > _MAX_DESCRIPTION: + raise ValueError(f"description exceeds {_MAX_DESCRIPTION} chars") + if required_package is not None: + if not isinstance(required_package, str) or not required_package: + raise ValueError("required_package must be a non-empty string") + return TrainerPluginSpec( + name=name, description=description, required_package=required_package + ) + + +_BUILTIN: Mapping[str, TrainerPluginSpec] = MappingProxyType( + { + "cce_plugin": _make( + "cce_plugin", + "Cut Cross-Entropy plugin variant (axolotl binding)", + "cut-cross-entropy", + ), + "grokfast": _make( + "grokfast", + "Gradient-grokking accelerator (axolotl)", + "grokfast", + ), + "spectrum": _make( + "spectrum", + "Gradient-norm-based layer freezing (axolotl)", + None, + ), + "llmcompressor": _make( + "llmcompressor", + "Post-training compression (axolotl)", + "llmcompressor", + ), + "sonicmoe": _make( + "sonicmoe", + "Alternative MoE kernel (axolotl)", + None, + ), + "math_verify": _make( + "math_verify", + "Standalone math reward verifier (promoted v0.25.0)", + None, + ), + } +) + + +def list_trainer_plugins() -> Mapping[str, TrainerPluginSpec]: + """Return an immutable view of the registry.""" + return _BUILTIN + + +def get_trainer_plugin(name: str) -> TrainerPluginSpec: + if not isinstance(name, str): + raise TypeError("name must be a string") + canonical = name.strip().lower() + if not canonical or "\x00" in canonical: + raise ValueError("name must be a non-empty NUL-free string") + if canonical not in _BUILTIN: + raise KeyError(canonical) + return _BUILTIN[canonical] + + +def validate_trainer_plugin_list(names: Sequence[str]) -> Tuple[str, ...]: + """Validate a list of trainer-plugin names. Returns canonical names.""" + if isinstance(names, str) or not isinstance(names, (list, tuple)): + raise TypeError("names must be a list or tuple") + if len(names) > _MAX_PLUGINS_PER_RUN: + raise ValueError( + f"too many trainer plugins (max {_MAX_PLUGINS_PER_RUN})" + ) + out: list[str] = [] + seen: set[str] = set() + for raw in names: + if not isinstance(raw, str): + raise TypeError("plugin name must be a string") + canonical = raw.strip().lower() + if not canonical or "\x00" in canonical: + raise ValueError("plugin name must be non-empty NUL-free") + if canonical not in _BUILTIN: + raise ValueError( + f"unknown trainer plugin: {canonical!r}. supported: " + f"{sorted(_BUILTIN)}" + ) + if canonical in seen: + raise ValueError(f"duplicate trainer plugin: {canonical!r}") + seen.add(canonical) + out.append(canonical) + return tuple(out) + + +__all__ = [ + "TrainerPluginSpec", + "list_trainer_plugins", + "get_trainer_plugin", + "validate_trainer_plugin_list", +] diff --git a/tests/test_v0450.py b/tests/test_v0450.py new file mode 100644 index 0000000..8dfc3ed --- /dev/null +++ b/tests/test_v0450.py @@ -0,0 +1,1202 @@ +"""v0.45.0 — Plugin System & Ecosystem Wins. + +Covers all five Parts: + +- Part A: ``soup_cli.plugins`` registry + ``soup plugins`` CLI +- Part B: ``utils.anthropic_messages`` + ``utils.server_tools`` + ``utils.ngram_spec`` +- Part C: ``utils.integrations`` external integrations catalog +- Part D: ``utils.trainer_plugins`` advanced trainer plugin allowlist +- Part E: ``utils.recipe_dag`` + ``soup data recipe`` CLI +""" + +from __future__ import annotations + +import pytest +from typer.testing import CliRunner + +from soup_cli import plugins as plugins_pkg +from soup_cli.utils.anthropic_messages import ( + from_anthropic, + to_anthropic, + validate_anthropic_payload, +) +from soup_cli.utils.integrations import ( + get_integration, + has_integration, + list_integrations, +) +from soup_cli.utils.ngram_spec import ( + NgramSpecConfig, + validate_ngram_config, + validate_ngram_n, + validate_num_draft_tokens, + validate_prompt_lookup_max, +) +from soup_cli.utils.recipe_dag import ( + NODE_KINDS, + load_recipe_yaml, + parse_recipe, + parse_recipe_yaml, +) +from soup_cli.utils.server_tools import ( + SUPPORTED_TOOLS, + WebSearchConfig, + is_domain_allowed, + tool_description, + validate_domain, + validate_rate_limit, + validate_tool_name, + validate_web_search_config, +) +from soup_cli.utils.trainer_plugins import ( + get_trainer_plugin, + list_trainer_plugins, + validate_trainer_plugin_list, +) + +# --------------------------------------------------------------------------- +# Part A — Plugin / hook system +# --------------------------------------------------------------------------- + + +class _NoopPlugin: + def pre_train(self, ctx): + ctx["pre_train_seen"] = True + + def post_step(self, ctx): + ctx["post_step_seen"] = True + + +@pytest.fixture(autouse=True) +def _reset_plugins(): + plugins_pkg.clear_plugins() + yield + plugins_pkg.clear_plugins() + + +def test_register_plugin_happy_path(): + spec = plugins_pkg.register_plugin( + name="hello-world", + version="1.0.0", + plugin=_NoopPlugin(), + description="hi", + ) + assert spec.name == "hello-world" + assert spec.version == "1.0.0" + assert spec.enabled is True + assert "hello-world" in plugins_pkg.list_plugins() + + +def test_discover_hooks_finds_only_implemented(): + plugin = _NoopPlugin() + hooks = plugins_pkg.discover_hooks(plugin) + assert "pre_train" in hooks + assert "post_step" in hooks + assert "post_train" not in hooks + assert "pre_step" not in hooks + + +@pytest.mark.parametrize( + "name", + [ + "", + "BadCase", + "starts-with-", # we accept this since regex allows trailing? actually allows + "with space", + "x" * 41, + "with\x00null", + ], +) +def test_register_plugin_rejects_bad_names(name): + if name == "starts-with-": # allowed — leading char is alnum + pytest.skip("trailing hyphen permitted by regex") + with pytest.raises((TypeError, ValueError)): + plugins_pkg.register_plugin( + name=name, version="1.0.0", plugin=_NoopPlugin() + ) + + +@pytest.mark.parametrize( + "version", + ["", "1", "1.0", "1.0.x", "v1.0.0", "1.0.0+\x00"], +) +def test_register_plugin_rejects_bad_version(version): + with pytest.raises((TypeError, ValueError)): + plugins_pkg.register_plugin( + name="ok", version=version, plugin=_NoopPlugin() + ) + + +def test_register_plugin_idempotent_same_spec(): + plugin = _NoopPlugin() + a = plugins_pkg.register_plugin(name="dup", version="1.0.0", plugin=plugin) + b = plugins_pkg.register_plugin(name="dup", version="1.0.0", plugin=plugin) + assert a is b + + +def test_register_plugin_idempotent_preserves_disabled_state(): + plugin = _NoopPlugin() + plugins_pkg.register_plugin(name="dup2", version="1.0.0", plugin=plugin) + plugins_pkg.disable_plugin("dup2") + again = plugins_pkg.register_plugin( + name="dup2", version="1.0.0", plugin=plugin + ) + assert again.enabled is False + + +def test_register_plugin_rejects_conflicting_description(): + plugin = _NoopPlugin() + plugins_pkg.register_plugin( + name="dup3", version="1.0.0", plugin=plugin, description="first" + ) + with pytest.raises(ValueError, match="already registered"): + plugins_pkg.register_plugin( + name="dup3", version="1.0.0", plugin=plugin, description="second" + ) + + +def test_register_plugin_rejects_too_many_templates(): + with pytest.raises(ValueError, match="exceeds"): + plugins_pkg.register_plugin( + name="big-tpl", + version="1.0.0", + plugin=_NoopPlugin(), + templates=[f"t{index}" for index in range(33)], + ) + + +def test_register_plugin_rejects_oversize_template_name(): + with pytest.raises(ValueError, match="exceeds"): + plugins_pkg.register_plugin( + name="long-tpl", + version="1.0.0", + plugin=_NoopPlugin(), + templates=["x" * 200], + ) + + +def test_register_plugin_rejects_conflicting_version(): + plugin = _NoopPlugin() + plugins_pkg.register_plugin(name="dup", version="1.0.0", plugin=plugin) + with pytest.raises(ValueError, match="already registered"): + plugins_pkg.register_plugin(name="dup", version="2.0.0", plugin=plugin) + + +def test_register_plugin_rejects_empty_plugin(): + class _Empty: + pass + + with pytest.raises(ValueError, match="hook"): + plugins_pkg.register_plugin( + name="empty", version="1.0.0", plugin=_Empty() + ) + + +def test_register_plugin_with_template_only_ok(): + class _Empty: + pass + + plugins_pkg.register_plugin( + name="tpl", + version="1.0.0", + plugin=_Empty(), + templates=["my-template"], + ) + + +def test_register_plugin_rejects_none_plugin(): + with pytest.raises(ValueError): + plugins_pkg.register_plugin(name="x", version="1.0.0", plugin=None) + + +def test_enable_disable_toggles_state(): + plugins_pkg.register_plugin( + name="toggle", version="1.0.0", plugin=_NoopPlugin() + ) + assert plugins_pkg.is_enabled("toggle") is True + assert plugins_pkg.disable_plugin("toggle") is True + assert plugins_pkg.is_enabled("toggle") is False + # second disable returns False (no state change) + assert plugins_pkg.disable_plugin("toggle") is False + assert plugins_pkg.enable_plugin("toggle") is True + + +def test_enable_unknown_raises(): + with pytest.raises(KeyError): + plugins_pkg.enable_plugin("ghost") + + +def test_list_plugins_returns_immutable_view(): + plugins_pkg.register_plugin( + name="a", version="1.0.0", plugin=_NoopPlugin() + ) + view = plugins_pkg.list_plugins() + with pytest.raises(TypeError): + view["evil"] = "not allowed" # type: ignore[index] + + +def test_get_plugin_unknown_returns_none(): + assert plugins_pkg.get_plugin("missing") is None + assert plugins_pkg.get_plugin(123) is None # type: ignore[arg-type] + + +def test_pluginspec_frozen(): + plugins_pkg.register_plugin( + name="frozen", version="1.0.0", plugin=_NoopPlugin() + ) + spec = plugins_pkg.get_plugin("frozen") + with pytest.raises(Exception): + spec.name = "mutated" # type: ignore[misc] + + +def test_too_many_plugins_rejected(): + for index in range(64): + plugins_pkg.register_plugin( + name=f"p{index}", version="1.0.0", plugin=_NoopPlugin() + ) + with pytest.raises(RuntimeError, match="too many"): + plugins_pkg.register_plugin( + name="overflow", version="1.0.0", plugin=_NoopPlugin() + ) + + +def test_load_plugins_returns_count(): + count = plugins_pkg.load_plugins() + assert isinstance(count, int) + assert count >= 0 + + +def test_plugins_cli_list_empty(): + from soup_cli.commands import plugins as plugins_cli + + runner = CliRunner() + result = runner.invoke(plugins_cli.app, []) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "No plugins registered" in result.output + + +def test_plugins_cli_lists_registered(): + from soup_cli.commands import plugins as plugins_cli + + plugins_pkg.register_plugin( + name="cli-test", version="1.2.3", plugin=_NoopPlugin() + ) + runner = CliRunner() + result = runner.invoke(plugins_cli.app, ["list"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "cli-test" in result.output + assert "1.2.3" in result.output + + +def test_plugins_cli_install_advisory(): + from soup_cli.commands import plugins as plugins_cli + + runner = CliRunner() + result = runner.invoke(plugins_cli.app, ["install", "anything"]) + assert result.exit_code == 0 + assert "advisory" in result.output + + +def test_plugins_cli_enable_unknown(): + from soup_cli.commands import plugins as plugins_cli + + runner = CliRunner() + result = runner.invoke(plugins_cli.app, ["enable", "ghost"]) + assert result.exit_code == 1 + assert "Unknown" in result.output + + +def test_plugins_cli_enable_disable_cycle(): + from soup_cli.commands import plugins as plugins_cli + + plugins_pkg.register_plugin( + name="cycle", version="1.0.0", plugin=_NoopPlugin() + ) + runner = CliRunner() + plugins_pkg.disable_plugin("cycle") + result = runner.invoke(plugins_cli.app, ["enable", "cycle"]) + assert result.exit_code == 0 + assert "enabled" in result.output + result = runner.invoke(plugins_cli.app, ["disable", "cycle"]) + assert result.exit_code == 0 + assert "disabled" in result.output + + +def test_plugins_cli_markup_escaped(): + from soup_cli.commands import plugins as plugins_cli + + runner = CliRunner() + # Crafted name with Rich markup; should be escaped before printing. + result = runner.invoke(plugins_cli.app, ["enable", "[red]evil[/]"]) + assert result.exit_code == 2 + # The validation message names the kebab-case rule; markup chars in + # the input cannot inject Rich styles because ``rich.markup.escape`` is + # applied to all user-controlled values in the CLI. + assert "kebab-case" in result.output + + +# --------------------------------------------------------------------------- +# Part B — Anthropic Messages converter +# --------------------------------------------------------------------------- + + +def test_to_anthropic_joins_multiple_system_messages(): + payload = { + "model": "x", + "messages": [ + {"role": "system", "content": "first"}, + {"role": "system", "content": "second"}, + {"role": "user", "content": "hi"}, + ], + "max_tokens": 32, + } + out = to_anthropic(payload) + assert out["system"] == "first\n\nsecond" + + +def test_to_anthropic_tool_result_concatenates_list_content(): + payload = { + "model": "x", + "messages": [ + {"role": "user", "content": "go"}, + { + "role": "tool", + "tool_call_id": "call_1", + "content": [ + {"type": "text", "text": "part-a"}, + {"type": "text", "text": "part-b"}, + ], + }, + ], + } + out = to_anthropic(payload) + tool_msg = out["messages"][-1] + assert tool_msg["content"][0]["content"] == "part-a\npart-b" + + +def test_to_anthropic_basic(): + payload = { + "model": "claude-3-5-sonnet", + "messages": [ + {"role": "system", "content": "you are helpful"}, + {"role": "user", "content": "hi"}, + ], + "max_tokens": 256, + } + out = to_anthropic(payload) + assert out["model"] == "claude-3-5-sonnet" + assert out["system"] == "you are helpful" + assert out["messages"] == [{"role": "user", "content": "hi"}] + assert out["max_tokens"] == 256 + + +def test_to_anthropic_default_max_tokens(): + payload = {"model": "x", "messages": [{"role": "user", "content": "a"}]} + out = to_anthropic(payload) + assert out["max_tokens"] == 1024 + + +def test_to_anthropic_caps_max_tokens(): + payload = { + "model": "x", + "messages": [{"role": "user", "content": "a"}], + "max_tokens": 99999, + } + out = to_anthropic(payload) + assert out["max_tokens"] == 16384 + + +def test_to_anthropic_tool_role_becomes_tool_result(): + payload = { + "model": "x", + "messages": [ + {"role": "user", "content": "ask"}, + { + "role": "tool", + "content": "result-text", + "tool_call_id": "tool_1", + }, + ], + } + out = to_anthropic(payload) + assert any( + msg["role"] == "user" + and isinstance(msg["content"], list) + and msg["content"][0].get("type") == "tool_result" + for msg in out["messages"] + ) + + +def test_to_anthropic_rejects_unknown_role(): + payload = { + "model": "x", + "messages": [{"role": "developer", "content": "x"}], + } + with pytest.raises(ValueError, match="role"): + to_anthropic(payload) + + +def test_to_anthropic_rejects_bad_max_tokens(): + payload = { + "model": "x", + "messages": [{"role": "user", "content": "a"}], + "max_tokens": True, + } + with pytest.raises(TypeError): + to_anthropic(payload) + + +def test_to_anthropic_rejects_temperature_out_of_range(): + payload = { + "model": "x", + "messages": [{"role": "user", "content": "a"}], + "temperature": 99.0, + } + with pytest.raises(ValueError): + to_anthropic(payload) + + +def test_from_anthropic_roundtrip_user_only(): + payload = { + "model": "x", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 10, + } + out = from_anthropic(payload) + assert out == { + "model": "x", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 10, + } + + +def test_from_anthropic_with_system(): + payload = { + "model": "x", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 10, + "system": "rules", + } + out = from_anthropic(payload) + assert out["messages"][0] == {"role": "system", "content": "rules"} + + +def test_validate_anthropic_payload_rejects_empty_messages(): + with pytest.raises(ValueError): + validate_anthropic_payload({ + "model": "x", + "messages": [], + "max_tokens": 10, + }) + + +def test_validate_anthropic_payload_rejects_bad_role(): + with pytest.raises(ValueError, match="role"): + validate_anthropic_payload({ + "model": "x", + "messages": [{"role": "system", "content": "x"}], + "max_tokens": 10, + }) + + +def test_validate_anthropic_payload_rejects_oversize_max_tokens(): + with pytest.raises(ValueError): + validate_anthropic_payload({ + "model": "x", + "messages": [{"role": "user", "content": "x"}], + "max_tokens": 99999, + }) + + +# --------------------------------------------------------------------------- +# Part B — Server-side tools registry +# --------------------------------------------------------------------------- + + +def test_supported_tools_closed_set(): + assert SUPPORTED_TOOLS == frozenset({"python", "bash", "web_search"}) + + +@pytest.mark.parametrize("name", ["python", "Bash", "WEB_SEARCH"]) +def test_validate_tool_name_canonicalises(name): + assert validate_tool_name(name) in SUPPORTED_TOOLS + + +@pytest.mark.parametrize("name", ["", "evil", "python ", "py\x00thon"]) +def test_validate_tool_name_rejects_garbage(name): + if name == "python ": # strip removes whitespace then matches + assert validate_tool_name(name) == "python" + return + with pytest.raises((TypeError, ValueError)): + validate_tool_name(name) + + +def test_validate_tool_name_rejects_non_string(): + with pytest.raises(TypeError): + validate_tool_name(123) # type: ignore[arg-type] + + +@pytest.mark.parametrize("rpm", [1, 600, 30]) +def test_validate_rate_limit_happy(rpm): + assert validate_rate_limit(rpm) == rpm + + +@pytest.mark.parametrize("rpm", [0, -1, 601, True]) +def test_validate_rate_limit_rejects(rpm): + with pytest.raises((TypeError, ValueError)): + validate_rate_limit(rpm) + + +@pytest.mark.parametrize( + "domain", ["example.com", ".example.com", "a.b.c.example.com"] +) +def test_validate_domain_happy(domain): + assert validate_domain(domain) == domain.lower() + + +@pytest.mark.parametrize( + "domain", ["", " ", "with space.com", "/path", "X" * 300, "evil.com\x00"] +) +def test_validate_domain_rejects(domain): + with pytest.raises((TypeError, ValueError)): + validate_domain(domain) + + +def test_is_domain_allowed_exact_match(): + assert is_domain_allowed("example.com", ("example.com",)) is True + assert is_domain_allowed("a.example.com", ("example.com",)) is False + + +def test_is_domain_allowed_subdomain_with_dot(): + assert is_domain_allowed("a.example.com", (".example.com",)) is True + assert is_domain_allowed("example.com", (".example.com",)) is True + + +def test_is_domain_allowed_no_match(): + assert is_domain_allowed("evil.org", ("example.com",)) is False + + +def test_is_domain_allowed_rejects_garbage_host(): + assert is_domain_allowed("", ("example.com",)) is False + assert is_domain_allowed("a\x00", ("example.com",)) is False + + +def test_validate_web_search_config_dedup(): + cfg = WebSearchConfig( + domain_allowlist=("example.com", "example.com"), + rate_limit_per_minute=30, + ) + with pytest.raises(ValueError, match="duplicate"): + validate_web_search_config(cfg) + + +def test_validate_web_search_config_too_many(): + cfg = WebSearchConfig( + domain_allowlist=tuple(f"d{index}.com" for index in range(65)), + rate_limit_per_minute=30, + ) + with pytest.raises(ValueError, match="exceeds"): + validate_web_search_config(cfg) + + +def test_validate_web_search_config_rejects_bad_type(): + with pytest.raises(TypeError): + validate_web_search_config("not-a-config") # type: ignore[arg-type] + + +def test_tool_description_happy(): + assert "Sandboxed" in tool_description("python") + assert "search" in tool_description("web_search").lower() + + +# --------------------------------------------------------------------------- +# Part B — n-gram speculative decoding +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("n", [1, 4, 8]) +def test_validate_ngram_n_happy(n): + assert validate_ngram_n(n) == n + + +@pytest.mark.parametrize("n", [0, 9, -1, True]) +def test_validate_ngram_n_rejects(n): + with pytest.raises((TypeError, ValueError)): + validate_ngram_n(n) + + +def test_validate_ngram_config_happy(): + cfg = NgramSpecConfig(n=4, num_draft_tokens=8, prompt_lookup_max=10) + assert validate_ngram_config(cfg) is cfg + + +def test_validate_ngram_config_rejects_bad_inner(): + cfg = NgramSpecConfig(n=4, num_draft_tokens=999) + with pytest.raises(ValueError): + validate_ngram_config(cfg) + + +@pytest.mark.parametrize("v", [True, -1, 33]) +def test_validate_num_draft_tokens_rejects(v): + with pytest.raises((TypeError, ValueError)): + validate_num_draft_tokens(v) + + +def test_validate_prompt_lookup_max_happy_zero(): + assert validate_prompt_lookup_max(0) == 0 + + +def test_validate_prompt_lookup_max_rejects_bool(): + with pytest.raises(TypeError): + validate_prompt_lookup_max(True) + + +def test_ngram_config_frozen(): + cfg = NgramSpecConfig(n=2) + with pytest.raises(Exception): + cfg.n = 3 # type: ignore[misc] + + +# --------------------------------------------------------------------------- +# Part C — External integrations catalog +# --------------------------------------------------------------------------- + + +def test_list_integrations_returns_immutable_view(): + view = list_integrations() + with pytest.raises(TypeError): + view["new"] = "evil" # type: ignore[index] + + +def test_list_integrations_known_entries(): + view = list_integrations() + for entry in ("lm-studio", "comfyui", "ollama", "open-webui", "claude-code"): + assert entry in view + + +def test_get_integration_known(): + spec = get_integration("ollama") + assert spec.name == "ollama" + assert "gguf" in spec.target_artifacts + + +def test_get_integration_unknown_raises_keyerror(): + with pytest.raises(KeyError): + get_integration("nonexistent") + + +@pytest.mark.parametrize("bad", ["", " ", "a\x00b"]) +def test_get_integration_rejects_bad(bad): + with pytest.raises(ValueError): + get_integration(bad) + + +def test_get_integration_rejects_non_string(): + with pytest.raises(TypeError): + get_integration(42) # type: ignore[arg-type] + + +def test_has_integration_happy_and_unknown(): + assert has_integration("ollama") is True + assert has_integration("nope") is False + assert has_integration(123) is False # type: ignore[arg-type] + + +def test_integration_spec_frozen(): + spec = get_integration("ollama") + with pytest.raises(Exception): + spec.name = "mutated" # type: ignore[misc] + + +# --------------------------------------------------------------------------- +# Part D — Advanced trainer plugins +# --------------------------------------------------------------------------- + + +def test_list_trainer_plugins_immutable(): + view = list_trainer_plugins() + with pytest.raises(TypeError): + view["evil"] = "x" # type: ignore[index] + + +def test_list_trainer_plugins_known_entries(): + view = list_trainer_plugins() + for entry in ("grokfast", "spectrum", "llmcompressor", "math_verify"): + assert entry in view + + +def test_get_trainer_plugin_known(): + spec = get_trainer_plugin("grokfast") + assert spec.name == "grokfast" + assert spec.required_package == "grokfast" + + +def test_get_trainer_plugin_unknown_raises(): + with pytest.raises(KeyError): + get_trainer_plugin("nope") + + +def test_get_trainer_plugin_no_required_pkg(): + spec = get_trainer_plugin("spectrum") + assert spec.required_package is None + + +def test_validate_trainer_plugin_list_happy(): + out = validate_trainer_plugin_list(["grokfast", "spectrum"]) + assert out == ("grokfast", "spectrum") + + +def test_validate_trainer_plugin_list_canonicalises(): + out = validate_trainer_plugin_list(["GROKFAST"]) + assert out == ("grokfast",) + + +def test_validate_trainer_plugin_list_rejects_unknown(): + with pytest.raises(ValueError, match="unknown"): + validate_trainer_plugin_list(["bogus_plugin"]) + + +def test_validate_trainer_plugin_list_rejects_duplicate(): + with pytest.raises(ValueError, match="duplicate"): + validate_trainer_plugin_list(["grokfast", "grokfast"]) + + +def test_validate_trainer_plugin_list_rejects_too_many(): + with pytest.raises(ValueError, match="too many"): + validate_trainer_plugin_list(["grokfast"] * 9) + + +def test_validate_trainer_plugin_list_rejects_non_string(): + with pytest.raises(TypeError): + validate_trainer_plugin_list([123]) # type: ignore[list-item] + + +def test_validate_trainer_plugin_list_rejects_non_list(): + with pytest.raises(TypeError): + validate_trainer_plugin_list("grokfast") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# Part E — Data Recipe DAG +# --------------------------------------------------------------------------- + + +def _good_recipe(): + return { + "nodes": [ + {"name": "seed1", "kind": "seed", "config": {"path": "x.jsonl"}}, + {"name": "llm1", "kind": "llm_text", "config": {}}, + {"name": "judge1", "kind": "judge", "config": {}}, + {"name": "samp1", "kind": "sampler", "config": {}}, + ], + "edges": [ + ["seed1", "llm1"], + ["llm1", "judge1"], + ["judge1", "samp1"], + ], + } + + +def test_node_kinds_closed_set(): + assert NODE_KINDS == frozenset( + {"seed", "llm_text", "code", "judge", "validator", "sampler"} + ) + + +def test_parse_recipe_happy(): + dag = parse_recipe(_good_recipe()) + assert tuple(dag.topo_order)[0] == "seed1" + assert tuple(dag.topo_order)[-1] == "samp1" + assert len(dag.nodes) == 4 + assert len(dag.edges) == 3 + + +def test_parse_recipe_rejects_cycle(): + bad = _good_recipe() + bad["edges"].append(["samp1", "seed1"]) + with pytest.raises(ValueError, match="cycle"): + parse_recipe(bad) + + +def test_parse_recipe_rejects_self_loop(): + bad = _good_recipe() + bad["edges"] = [["seed1", "seed1"]] + with pytest.raises(ValueError, match="self-loop"): + parse_recipe(bad) + + +def test_parse_recipe_rejects_unknown_kind(): + bad = _good_recipe() + bad["nodes"][0]["kind"] = "bogus" + with pytest.raises(ValueError, match="unknown"): + parse_recipe(bad) + + +def test_parse_recipe_rejects_duplicate_node_name(): + bad = _good_recipe() + bad["nodes"][1]["name"] = "seed1" + with pytest.raises(ValueError, match="duplicate"): + parse_recipe(bad) + + +def test_parse_recipe_rejects_dangling_edge(): + bad = _good_recipe() + bad["edges"][0][1] = "ghost" + with pytest.raises(ValueError, match="not in nodes"): + parse_recipe(bad) + + +def test_parse_recipe_rejects_empty_nodes(): + with pytest.raises(ValueError): + parse_recipe({"nodes": [], "edges": []}) + + +def test_parse_recipe_rejects_non_dict_node(): + with pytest.raises(TypeError): + parse_recipe({"nodes": ["not-a-dict"]}) + + +def test_parse_recipe_rejects_non_dict_input(): + with pytest.raises(TypeError): + parse_recipe([1, 2]) + + +def test_parse_recipe_rejects_too_many_nodes(): + bad = { + "nodes": [ + {"name": f"n{index}", "kind": "seed"} for index in range(257) + ], + "edges": [], + } + with pytest.raises(ValueError, match="exceeds"): + parse_recipe(bad) + + +def test_parse_recipe_rejects_duplicate_edge(): + bad = _good_recipe() + bad["edges"].append(["seed1", "llm1"]) + with pytest.raises(ValueError, match="duplicate"): + parse_recipe(bad) + + +def test_parse_recipe_rejects_bad_node_name(): + bad = _good_recipe() + bad["nodes"][0]["name"] = "Bad Name" + with pytest.raises(ValueError): + parse_recipe(bad) + + +def test_parse_recipe_yaml_roundtrip(): + text = """ +nodes: + - name: seed1 + kind: seed + - name: out1 + kind: sampler +edges: + - [seed1, out1] +""" + dag = parse_recipe_yaml(text) + assert dag.topo_order == ("seed1", "out1") + + +def test_parse_recipe_yaml_rejects_null_byte(): + with pytest.raises(ValueError, match="null"): + parse_recipe_yaml("nodes:\n - name: a\x00\n kind: seed") + + +def test_parse_recipe_yaml_rejects_invalid_yaml(): + with pytest.raises(ValueError, match="YAML"): + parse_recipe_yaml("nodes: [\n unterminated") + + +def test_load_recipe_yaml_happy(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + recipe_path = tmp_path / "r.yaml" + recipe_path.write_text( + "nodes:\n - name: seed1\n kind: seed\nedges: []\n", + encoding="utf-8", + ) + dag = load_recipe_yaml("r.yaml") + assert dag.topo_order == ("seed1",) + + +def test_load_recipe_yaml_rejects_outside_cwd(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + other = tmp_path.parent / "other.yaml" + other.write_text("nodes:\n - name: a\n kind: seed\n", encoding="utf-8") + try: + with pytest.raises(ValueError, match="under cwd"): + load_recipe_yaml(str(other)) + finally: + try: + other.unlink() + except OSError: + pass + + +def test_load_recipe_yaml_missing_file(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + load_recipe_yaml("missing.yaml") + + +def test_load_recipe_yaml_rejects_null_byte_path(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + load_recipe_yaml("a\x00b.yaml") + + +import sys # noqa: E402 -- keep imports tight for symlink-test guard + + +@pytest.mark.skipif(sys.platform == "win32", reason="symlinks need privilege on Windows") +def test_load_recipe_yaml_rejects_symlink(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + target = tmp_path / "real.yaml" + target.write_text( + "nodes:\n - name: a\n kind: seed\nedges: []\n", + encoding="utf-8", + ) + link = tmp_path / "link.yaml" + link.symlink_to(target) + with pytest.raises(ValueError, match="symlink"): + load_recipe_yaml("link.yaml") + + +def test_data_recipe_cli_happy(tmp_path, monkeypatch): + from soup_cli.commands import data as data_cmd + + monkeypatch.chdir(tmp_path) + recipe_path = tmp_path / "r.yaml" + recipe_path.write_text( + "nodes:\n - name: seed1\n kind: seed\nedges: []\n", + encoding="utf-8", + ) + runner = CliRunner() + result = runner.invoke(data_cmd.app, ["recipe", "r.yaml"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "validated" in result.output + assert "deferred" in result.output + + +def test_data_recipe_cli_invalid_recipe(tmp_path, monkeypatch): + from soup_cli.commands import data as data_cmd + + monkeypatch.chdir(tmp_path) + recipe_path = tmp_path / "r.yaml" + recipe_path.write_text( + "nodes:\n - name: seed1\n kind: bogus\nedges: []\n", + encoding="utf-8", + ) + runner = CliRunner() + result = runner.invoke(data_cmd.app, ["recipe", "r.yaml"]) + assert result.exit_code == 2 + assert "Invalid" in result.output + + +def test_data_recipe_cli_missing_file(tmp_path, monkeypatch): + from soup_cli.commands import data as data_cmd + + monkeypatch.chdir(tmp_path) + runner = CliRunner() + result = runner.invoke(data_cmd.app, ["recipe", "ghost.yaml"]) + assert result.exit_code == 1 + assert "not found" in result.output.lower() + + +# --------------------------------------------------------------------------- +# Coverage-gap fill (post-review) +# --------------------------------------------------------------------------- + + +def test_register_plugin_rejects_too_many_model_groups(): + with pytest.raises(ValueError, match="exceeds"): + plugins_pkg.register_plugin( + name="big-grp", + version="1.0.0", + plugin=_NoopPlugin(), + model_groups=[f"g{index}" for index in range(33)], + ) + + +def test_register_plugin_rejects_empty_model_group_entry(): + with pytest.raises(ValueError, match="model_group"): + plugins_pkg.register_plugin( + name="bad-grp", + version="1.0.0", + plugin=_NoopPlugin(), + model_groups=[""], + ) + + +def test_register_plugin_rejects_oversize_model_group_entry(): + with pytest.raises(ValueError, match="exceeds"): + plugins_pkg.register_plugin( + name="long-grp", + version="1.0.0", + plugin=_NoopPlugin(), + model_groups=["x" * 200], + ) + + +def test_register_plugin_rejects_null_byte_description(): + with pytest.raises(ValueError, match="null"): + plugins_pkg.register_plugin( + name="bad-desc", + version="1.0.0", + plugin=_NoopPlugin(), + description="bad\x00desc", + ) + + +def test_register_plugin_rejects_oversize_description(): + with pytest.raises(ValueError, match="exceeds"): + plugins_pkg.register_plugin( + name="long-desc", + version="1.0.0", + plugin=_NoopPlugin(), + description="x" * 1000, + ) + + +def test_enable_plugin_already_enabled_returns_false(): + plugins_pkg.register_plugin( + name="warm", version="1.0.0", plugin=_NoopPlugin() + ) + assert plugins_pkg.enable_plugin("warm") is False + + +def test_disable_unknown_raises(): + with pytest.raises(KeyError): + plugins_pkg.disable_plugin("missing") + + +def test_list_hook_names_returns_canonical_tuple(): + assert plugins_pkg.list_hook_names() == ( + "pre_train", + "post_train", + "pre_step", + "post_step", + ) + + +def test_to_anthropic_rejects_non_list_messages(): + with pytest.raises(TypeError): + to_anthropic({"model": "x", "messages": "oops"}) + + +def test_to_anthropic_rejects_non_str_or_list_content(): + with pytest.raises(TypeError): + to_anthropic({ + "model": "x", + "messages": [{"role": "user", "content": 42}], + }) + + +def test_to_anthropic_rejects_bool_temperature(): + with pytest.raises(TypeError): + to_anthropic({ + "model": "x", + "messages": [{"role": "user", "content": "a"}], + "temperature": True, + }) + + +@pytest.mark.parametrize("temp", [0.0, 2.0]) +def test_to_anthropic_accepts_temperature_boundaries(temp): + out = to_anthropic({ + "model": "x", + "messages": [{"role": "user", "content": "a"}], + "temperature": temp, + }) + assert out["temperature"] == temp + + +def test_to_anthropic_rejects_max_tokens_zero(): + with pytest.raises(ValueError): + to_anthropic({ + "model": "x", + "messages": [{"role": "user", "content": "a"}], + "max_tokens": 0, + }) + + +def test_validate_anthropic_payload_rejects_non_dict(): + with pytest.raises(TypeError): + validate_anthropic_payload([1, 2, 3]) # type: ignore[arg-type] + + +def test_validate_anthropic_payload_rejects_bool_max_tokens(): + with pytest.raises(TypeError): + validate_anthropic_payload({ + "model": "x", + "messages": [{"role": "user", "content": "a"}], + "max_tokens": True, + }) + + +def test_is_domain_allowed_strips_port(): + assert is_domain_allowed("example.com:443", ("example.com",)) is True + assert is_domain_allowed("a.example.com:443", (".example.com",)) is True + + +def test_is_domain_allowed_rejects_ipv6_literal(): + assert is_domain_allowed("[::1]", ("example.com",)) is False + + +def test_validate_ngram_config_rejects_non_config(): + with pytest.raises(TypeError): + validate_ngram_config({"n": 4}) # type: ignore[arg-type] + + +def test_get_trainer_plugin_rejects_null_byte(): + with pytest.raises(ValueError, match="NUL"): + get_trainer_plugin("grok\x00fast") + + +def test_get_trainer_plugin_rejects_empty_string(): + with pytest.raises(ValueError): + get_trainer_plugin("") + + +def test_parse_recipe_rejects_too_many_edges(): + nodes = [ + {"name": f"n{index}", "kind": "seed"} + for index in range(50) + ] + edges = [["n0", "n1"]] * 1025 + with pytest.raises(ValueError): + parse_recipe({"nodes": nodes, "edges": edges}) + + +def test_parse_recipe_rejects_non_pair_edge_entry(): + bad = _good_recipe() + bad["edges"] = ["not-a-pair"] + with pytest.raises(ValueError, match="2-element"): + parse_recipe(bad) + + +def test_parse_recipe_yaml_rejects_oversize_text(): + text = "nodes:\n" + (" - name: x\n kind: seed\n" * 200_000) + with pytest.raises(ValueError, match="bytes"): + parse_recipe_yaml(text) + + +def test_parse_recipe_yaml_rejects_non_string(): + with pytest.raises(TypeError): + parse_recipe_yaml(42) # type: ignore[arg-type] + + +def test_load_recipe_yaml_rejects_non_string_path(): + with pytest.raises((TypeError, ValueError)): + load_recipe_yaml(42) # type: ignore[arg-type] + + +def test_load_recipe_yaml_rejects_oversize_file(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + big = tmp_path / "big.yaml" + # File-size guard fires before YAML parsing. + big.write_bytes(b"nodes: []\n" + b"# pad\n" * 200_000) + with pytest.raises(ValueError, match="bytes"): + load_recipe_yaml("big.yaml")