diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a0c9f9e..429b268 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (165 files, 5989 tests) +tests/ - Test suite (167 files, 6126 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index d3a37bf..a256488 100644 --- a/README.md +++ b/README.md @@ -43,15 +43,14 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.45.0 — Plugin System & Ecosystem Wins**: A public plugin API and the schema scaffolding for 20+ ecosystem integrations. Soup is now extensible. +**v0.46.0 — Deploy & Agent Autopilot**: Two zero-config autopilots — a deploy target picker that maps your hardware to PEFT+quant+spec-decoding in one command, and an Agent Forge that turns OpenAPI / MCP / GraphQL specs straight into tool-calling SFT datasets. -- **Plugin / hook system.** `soup_cli.plugins.register_plugin(name, version, plugin, templates=[], model_groups=[])` lets third-party Python modules ship their own pre-train / post-train / pre-step / post-step hooks plus chat templates and model groups. Registry is idempotent on identical specs and rejects conflicting re-registration. Drop a file under `soup_cli/plugins/` and the loader picks it up at startup. New `soup plugins list / install / enable / disable` CLI. -- **Anthropic Messages API converter.** `to_anthropic` / `from_anthropic` translate between OpenAI chat-completions and Anthropic Messages payloads — multiple `system` messages joined with `\n\n`, `tool` role surfaces as `tool_result` content blocks (list content concatenated, never silently dropped), `max_tokens` capped at 16384, `temperature` bounded `[0.0, 2.0]`. Live `/v1/messages` endpoint deferred to v0.45.1. -- **Server-side tools allowlist.** Closed `{python, bash, web_search}` set with `WebSearchConfig` — domain allowlist (leading-dot subdomain pattern, port-strip on the host, IPv6 literal deny), `rate_limit_per_minute ∈ [1, 600]`. `python` and `bash` reuse the v0.25.0 RLVR sandbox; live HTTP endpoints deferred to v0.45.1. -- **External integrations catalog.** 15-entry frozen `MappingProxyType` of ecosystem targets — `lm-studio`, `comfyui`, `stable-diffusion-cpp`, `open-webui`, `ollama`, `tei`, `pgvector`, `faiss`, `weaviate`, `sentence-transformers`, `claude-code`, `cursor`, `continue`, `cline`, `sillytavern`. Each entry names the artifact format (`gguf` / `safetensors` / `served-endpoint`). -- **Advanced trainer-plugin allowlist.** 6-entry catalog (`grokfast` / `spectrum` / `llmcompressor` / `sonicmoe` / `cce_plugin` / `math_verify`) with `validate_trainer_plugin_list` (dedup, allowlist match, `_MAX_PLUGINS_PER_RUN=8`). Live callbacks land in v0.45.1. -- **`soup data recipe `.** Validates a Seed → LLM Text → Code → Judge → Validator → Sampler graph DAG: closed node-kind allowlist, Kahn's topological sort with `collections.deque` (deterministic, O(N+E)), self-loop / cycle / dangling-edge / duplicate rejection, cwd containment + `os.lstat + S_ISLNK` symlink rejection on the recipe file. Live offline runner against a local model in v0.45.1. -- **+169 net new tests** — covers all 5 release Parts: plugin idempotency + per-list caps + description conflict rejection, Anthropic converter happy + failure paths, domain allowlist port-strip + IPv6 deny, n-gram bounds, integrations catalog immutability, trainer-plugin canonicalisation + dedup, recipe DAG cycle / cap / symlink + CLI happy / invalid / missing. +- **`soup deploy autopilot --target `.** 10 hardware profiles (`mac-m3`, `mac-m4-pro`, `rtx-3060-12gb`, `rtx-4090-24gb`, `iphone-16`, `pixel-9`, `ollama-local`, `lm-studio`, `runpod-a100`, `hf-jobs-h100`) each mapped to a runtime, quantisation, PEFT method, and speculative-decoding flag. Writes a ready-to-train `soup.yaml` recipe AND a planned deploy shell script. Closed allowlists on every field; `shlex.quote` on the model path in the generated bash. Live Quant-Lobotomy auto-measure deferred to v0.46.1. +- **`soup agent synth --spec api.yaml`.** Parses OpenAPI 3.x, MCP server manifests, or GraphQL introspection JSON into a canonical endpoint list and synthesises a tool-calling SFT dataset (`{messages, tool, source_endpoint}`). `$ref` strings are left opaque (no external resolution — defends against file-read SSRF), `yaml.safe_load` only, 5 MiB spec cap, 10 000-endpoint cap. Atomic JSONL write via staged-tempfile + `os.replace` — mid-stream failure never leaves a partial dataset. +- **`soup agent train --spec api.yaml --base `.** One-shot wrapper that runs synth, then prints the planned `soup train` invocation with the rendered recipe (in-process re-entry of Typer commands is intentionally not done — matches `soup quantize` design). `--base` and `--output-dir` are validated for NUL / newline / oversize BEFORE embedding in the recipe YAML (defends against YAML key injection). +- **`soup agent eval --spec api.yaml --predictions preds.jsonl`.** Scores predicted tool-calls against the spec catalog: tool-name match + arguments-key validity. Predictions path enforces cwd containment, `os.lstat + S_ISLNK` symlink rejection, and a 1 000 000-line DoS cap. Live RLVR `code_exec` sandbox scoring deferred to v0.46.1. +- **Security throughout.** Path containment (`is_under_cwd`), symlink TOCTOU rejection (`os.lstat + S_ISLNK`) on every write target, Rich markup escape on every spec-derived string, bool-rejected-before-int on every numeric param, closed allowlists on runtime / quant / PEFT / spec kind / node kind. +- **+137 net new tests** — every parser kind (OpenAPI / MCP / GraphQL), every failure mode (cycle / cap / null-byte / oversize / outside-cwd / symlink), every CLI surface (`autopilot --list / --help / happy / outside-cwd reject`, `agent synth/train/eval` happy + failure). ## Why Soup? @@ -2754,6 +2753,11 @@ soup deploy ollama --model m.gguf --name x Deploy GGUF to Ollama soup deploy ollama --list List Soup-deployed models soup deploy ollama --remove Remove model from Ollama soup deploy hf-space --model user/m --space user/s --template gradio-chat|streamlit-chat Create HF Space +soup deploy autopilot --target mac-m3|rtx-4090-24gb|... Pick PEFT+quant+spec-decoding for a hardware target +soup deploy autopilot --list List all 10 deploy profiles +soup agent synth --spec api.yaml -o ds.jsonl Parse OpenAPI/MCP/GraphQL spec into a tool-calling SFT dataset +soup agent train --spec api.yaml --base model One-shot synth + planned soup train invocation +soup agent eval --spec api.yaml --predictions p.jsonl Score predicted tool-calls vs spec catalog soup eval benchmark --model ./output Evaluate on standard benchmarks soup eval custom --tasks eval.jsonl Custom eval tasks from JSONL soup eval judge --target resp.jsonl LLM-as-a-judge evaluation @@ -3161,6 +3165,36 @@ params: Strict scalar allowlist on values (`str` / `int` / `float` / `bool`); `_MAX_FILE_BYTES=256KB`, `_MAX_PARAM_KEYS=32`, `_MAX_VALUES_PER_KEY=64`; `SweepSpec.params` is `MappingProxyType[str, Tuple[Any, ...]]` for genuine immutability. +## Deploy Autopilot + +Pick the optimal PEFT + quantisation + speculative-decoding combo for your hardware target in one command: + +```bash +soup deploy autopilot --target rtx-4090-24gb --base meta-llama/Llama-3.2-1B +# Writes: +# deploy_autopilot.yaml — ready-to-train soup.yaml recipe +# deploy_autopilot.sh — planned deploy shell script +``` + +Profiles ship out of the box for Apple Silicon (`mac-m3`, `mac-m4-pro`), consumer NVIDIA (`rtx-3060-12gb`, `rtx-4090-24gb`), mobile (`iphone-16`, `pixel-9`), local runtimes (`ollama-local`, `lm-studio`), and cloud (`runpod-a100`, `hf-jobs-h100`). `--list` shows the full table. Every profile is a frozen dataclass with closed allowlists on runtime / quant / PEFT — bad config values fail at import time. The generated bash uses `shlex.quote` on the model path and writes are protected by cwd containment + `os.lstat + S_ISLNK` TOCTOU rejection. + +## Agent Forge + +Turn an OpenAPI 3.x, MCP server manifest, or GraphQL introspection JSON straight into a tool-calling SFT dataset — no manual labelling, no scaffolding: + +```bash +# 1. Parse spec + synthesise a tool-calling dataset +soup agent synth --spec api.yaml --output ds.jsonl --examples-per-endpoint 4 + +# 2. Plan the training run (prints the soup train invocation) +soup agent train --spec api.yaml --base meta-llama/Llama-3.2-1B + +# 3. Score model predictions against the spec catalog +soup agent eval --spec api.yaml --predictions preds.jsonl +``` + +Each row of the synthesised dataset is `{messages: [user, assistant_with_tool_call], tool: , source_endpoint: }`. `$ref` strings in OpenAPI are left opaque (no external resolution — defends against file-read SSRF), `yaml.safe_load` only, 5 MiB spec cap, 10 000-endpoint cap, atomic JSONL write via staged-tempfile + `os.replace`. `eval` enforces a 1 000 000-line cap on predictions and rejects symlinks at every read/write boundary. + ## Plugin System Drop a Python module under `soup_cli/plugins/` (or any package importable by Soup) and register at import time: diff --git a/SECURITY.md b/SECURITY.md index a919426..f79fdd3 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,12 +9,12 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.45.0 -- Full support (latest) +- v0.46.0 -- Full support (latest) +- v0.45.0 -- Full support - v0.44.0 -- Full support -- v0.43.0 -- Full support +- v0.43.0-v0.43.x -- Bug-fix support only - v0.42.0-v0.42.x -- Bug-fix support only -- v0.41.0-v0.41.x -- Bug-fix support only -- v0.40.x and below -- No support +- v0.41.x and below -- No support ## Reporting a Vulnerability @@ -145,6 +145,7 @@ No known critical vulnerabilities in current releases. - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input. - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS` +- **v0.46.0 — Deploy & Agent Autopilot**: 2 release Parts ship a deploy-target picker and an Agent Forge for spec-driven tool-calling SFT datasets. New `soup_cli/utils/deploy_autopilot.py` ships a 10-profile `MappingProxyType`-wrapped catalog with `DeployProfile` `@dataclass(frozen=True)` and closed allowlists on `runtime` (transformers / vllm / sglang / mlx / ollama / lm-studio / executorch), `quant` (none / 4bit / 8bit / gptq / awq / fp8 / mxfp4 / hqq:Nbit), `peft` (lora / dora / qlora / full). `_make` factory rejects non-kebab-case names, bool-as-int on `recommended_max_length`, out-of-bounds `[64, 1_048_576]`, null-byte / >512-char description+notes. `render_recipe_yaml` rejects empty / null-byte / newline / >200-char `base` and >4096-char `output_dir` (defends against a crafted `--base "evil\ntraining: { epochs: 9999 }"` injecting YAML keys into the rendered recipe). `render_deploy_script` uses `shlex.quote` on `model_path` and rejects newline / NUL / >4096-char. `write_recipe` / `write_deploy_script` enforce `is_under_cwd` containment, ≤4096-char path cap, and `os.lstat + stat.S_ISLNK` rejection at the write target (TOCTOU defence — mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B / v0.45.0 Part E policy). `soup deploy autopilot` panel passes every profile field through `rich.markup.escape` (matches v0.43.0 Part B `Tournament` policy — defends against markup injection if a future profile carries `[blink]` in its description). New `soup_cli/utils/agent_forge.py` parses OpenAPI 3.x / MCP server manifests / GraphQL introspection JSON into a canonical `Endpoint` frozen dataclass — every parser routes synthesised `path` through `_validate_path` (non-empty single-line NUL-free ≤1024 chars), so a manifest with `name="evil\nhost"` is rejected with a warning (review fix HIGH — the v0.46.0 first-cut stored the raw `f"mcp://{name}"` without validation, allowing newline injection into the `Endpoint.path` field). `$ref` strings in OpenAPI are left opaque (no external resolution — defends against file-read SSRF that a fully-resolving parser would expose). `_MAX_ENDPOINTS=10_000`, `_MAX_SPEC_BYTES=5*1024*1024`, `_MAX_ROWS_PER_ENDPOINT=32`, `_MAX_DESCRIPTION=512`. `load_spec_file` enforces `is_under_cwd` containment + `os.lstat + stat.S_ISLNK` rejection BEFORE `realpath` resolution (review fix MEDIUM — the v0.46.0 first-cut ordered `isfile(real)` before the lstat check, which followed the symlink) + 5 MiB cap + `yaml.safe_load` only (never `yaml.load`). `write_dataset` is atomic via `tempfile.mkstemp` + `os.replace` (review fix HIGH — replaces a v0.46.0 first-cut direct `open(real, "w")` loop that would leave a partial JSONL file on mid-stream `TypeError`; mirrors v0.43.0 Part D `copy_bundle_to` policy). Symlink rejection at the dataset target. `endpoint_to_rows` rejects bool / out-of-bounds `examples_per_endpoint` (∈ [1, 32]). New `soup_cli/commands/agent.py`: `synth` Rich table passes every cell through `rich.markup.escape` (review fix HIGH — defends against spec-controlled markup injection through `ep.path`); `train` validates `--base` and `--output-dir` for NUL / newline / >4096-char BEFORE embedding into the rendered YAML recipe string (review fix CRITICAL — defends against YAML key injection where `--base $'evil\\ntraining: { epochs: 9999 }'` would smuggle injected training keys); `eval` enforces predictions path `is_under_cwd` containment + `os.lstat + stat.S_ISLNK` rejection + `_MAX_PRED_LINES=1_000_000` DoS cap (review fix HIGH — v0.46.0 first-cut had no line cap, a multi-GB predictions file would have iterated unbounded). Known limitations: (1) Live Quant-Lobotomy auto-measure deferred to v0.46.1 — autopilot writes the canonical PEFT+quant combo per profile but does not yet measure OK/MINOR/MAJOR via v0.26.0 Quant-Lobotomy Checker. (2) ExecuTorch packaging deferred to v0.54.0 — `iphone-16` / `pixel-9` recipes are plan-only. (3) `soup agent train` is plan-only — prints the planned `soup train` invocation rather than re-entering Typer in-process (same design as v0.44.0 `soup quantize`). (4) `soup agent eval` is heuristic — scores tool-name match + arguments-key validity only; live RLVR `code_exec` sandbox scoring deferred to v0.46.1. (5) `$ref` resolution in OpenAPI specs is intentionally not done (file-read SSRF defence); users wanting full resolution should run `openapi-spec-validator` upstream. (6) MCP / GraphQL non-HTTP sentinel methods (`invoke` / `query` / `mutation`) are stored on `Endpoint.method` without going through `_validate_method` (documented design intent — `_HTTP_METHODS` covers HTTP-only). - **v0.45.0 — Plugin System & Ecosystem Wins**: 5 release Parts ship the plugin / hook system + 4 ecosystem-integration schemas. New `soup_cli/plugins/__init__.py` registry: kebab-case plugin name regex `^[a-z0-9][a-z0-9\-]{0,39}$`, semver-ish version regex, `_MAX_PLUGINS=64`, `_MAX_TEMPLATES_PER_PLUGIN=32`, `_MAX_MODEL_GROUPS_PER_PLUGIN=32`, `_MAX_NAME_ENTRY_LEN=128`. Re-registering with a different `(version, plugin object, templates, model_groups, description)` is rejected (review fix HIGH — first-cut omitted `description` from the conflict check, allowing silent description override). `load_plugins()` swallows per-plugin exceptions at WARNING level so one broken plugin cannot crash the CLI. `soup plugins enable / disable` raise `KeyError` for unknown names (caller-distinguishable from validation errors). All user-controlled output passes through `rich.markup.escape`. New `utils/anthropic_messages.py` enforces `_MAX_MESSAGES=1024`, per-message `_MAX_CONTENT_LEN=1MiB`, `max_tokens` cap mirrors v0.30.0 inference-server `_MAX_TOKENS_CAP=16384`; `tool` role with structured (list) content is concatenated into a single `tool_result` text block instead of silently dropped (review fix MEDIUM — first-cut emitted `""` for any non-str content); `from_anthropic` uses `.get("role")` defensively after `validate_anthropic_payload` (review fix MEDIUM — `KeyError` surface guard). New `utils/server_tools.py` ships closed `SUPPORTED_TOOLS = frozenset({python, bash, web_search})` allowlist + `WebSearchConfig` with `domain_allowlist` (≤64 entries, default empty = deny all) + `rate_limit_per_minute ∈ [1, 600]`. `is_domain_allowed` strips `:port` suffix before matching (review fix MEDIUM — without this, `Host: api.example.com:443` would silently fail to match the bare `api.example.com` allowlist entry, causing legitimate denials) AND rejects IPv6 literals (`[::1]` → False — IPv6 should never match a domain allowlist). `validate_domain` rejects null-byte / whitespace / `/` / oversize. `python` and `bash` reuse the v0.25.0 RLVR sandbox (5s timeout, RLIMIT_AS/CPU on POSIX, ephemeral cwd, socket patch); live HTTP endpoints land in v0.45.1. New `utils/ngram_spec.py` `NgramSpecConfig` is `@dataclass(frozen=True)` with `n ∈ [1, 8]`, `num_draft_tokens ∈ [1, 32]`, `prompt_lookup_max ∈ [0, 1MiB]`; bool rejection on every numeric field (matches v0.30.0 `Candidate` / v0.34.0 `estimate_run_cost_usd` / v0.42.0 `make_preprocess_cache_key` policy). New `utils/integrations.py` 15-entry catalog wrapped in `MappingProxyType`; `IntegrationSpec` is frozen so the catalog cannot be mutated at runtime. New `utils/trainer_plugins.py` 6-entry allowlist (`MappingProxyType`); `validate_trainer_plugin_list` accepts `Sequence[str]` (review fix MEDIUM — first-cut annotated as `Tuple[str, ...]`) but explicitly rejects bare `str` argument so a caller passing `"grokfast"` directly doesn't iterate per-character. New `utils/recipe_dag.py` parses Data Recipe DAG with closed `NODE_KINDS = frozenset({seed, llm_text, code, judge, validator, sampler})`; Kahn's topological sort uses `collections.deque` + sorted-on-arrival (review fix HIGH — first-cut had O(N² log N) `queue.sort()` inside the BFS body); cycle / self-loop / dangling-edge / duplicate-edge / duplicate-node-name / unknown-kind rejection with specific error messages. `_MAX_NODES=256`, `_MAX_EDGES=1024`, `_MAX_FILE_BYTES=1MiB`. `parse_recipe_yaml` uses `yaml.safe_load`. `load_recipe_yaml` enforces `is_under_cwd` containment AND `os.lstat + stat.S_ISLNK` symlink rejection at the recipe path (review fix MEDIUM — TOCTOU defence; mirrors v0.33.0 #22 / v0.43.0 Part C / v0.44.0 Part B policy; without this, a symlink under cwd pointing at `/etc/passwd` would resolve through `realpath` and load arbitrary files). `RecipeDAG` and `RecipeNode` are `@dataclass(frozen=True)`; `RecipeNode.config` is `MappingProxyType`-wrapped so caller mutation post-parse cannot affect the validated topology. `parse_recipe` does NOT mutate input dicts (matches v0.33.0 #47 `CrossDocCollator` immutability policy). Known limitations: (1) Plugin trainer-callback wiring deferred to v0.45.1 — `register_plugin` accepts hook objects and `discover_hooks` lists implementations, but no HF Trainer callback emits `pre_train` / `post_train` / `pre_step` / `post_step` yet. (2) Anthropic `/v1/messages` endpoint deferred to v0.45.1. (3) Server-side tool HTTP endpoints (python / bash / web_search) deferred to v0.45.1. (4) n-gram speculative-decoding live engine wiring deferred to v0.45.1. (5) External integrations are a catalog, not launchers — `IntegrationSpec` describes 15 ecosystem targets but no `soup deploy ` command auto-detects + invokes their CLIs (tracked for v0.46.0 Deploy Autopilot). (6) Advanced trainer plugins are an allowlist, not callbacks — `validate_trainer_plugin_list` accepts `[grokfast, spectrum, ...]` at schema validation, but no trainer wraps the upstream callbacks yet (live wiring per plugin in v0.45.1). (7) Data Recipe DAG runner deferred — `soup data recipe` validates topology and prints planned topological order; node-by-node execution against a local model is the v0.45.1 deliverable. - **v0.44.0 — Live Dashboard & UX**: 21 features across 4 Parts. New `soup_cli/utils/qr_url.py` `build_phone_url` enforces a scheme allowlist (`http` / `https`), loopback-only HTTP (RFC1918 / link-local / cloud-metadata IPs implicitly rejected because `host not in _LOOPBACK_HOSTS` raises), IPv6 literal auto-bracketing per RFC 3986, and **token routing through the URL query string `?token=…`** (review fix HIGH — the v0.44.0 first-cut put it in the URL fragment `#token=…` which never reaches the server). Token shape regex `^[A-Za-z0-9_\-]{16,128}$` matches `secrets.token_urlsafe(N)` output. New `commands/fetch.py` enforces `is_under_cwd` on `--output` AND adds **`os.lstat + S_ISLNK` symlink rejection at the write target** regardless of `--force` (review fix HIGH — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 / v0.43.0 Part C policy). Bundled-source path is `os.path.commonpath`-checked against `fetch_examples_dir()` (review fix HIGH — defends against a future catalog entry containing `..` that would escape the bundled directory). `fetch_examples_dir()` uses `os.path.realpath` not `abspath` (review fix MED — symlinked installs resolve to the real package root, not the symlink's parent). New `utils/checkpoint_trigger.py` `write_trigger` adds `os.lstat + S_ISLNK` rejection on the trigger target (review fix MED — symlink at `/.checkpoint_now` could redirect the write to any user-writable path). Trigger path stays under cwd via shared `is_under_cwd`. New `commands/llama.py` proxies to llama.cpp binaries with **`_LLAMA_ENV_ALLOWLIST` frozenset child-env filter** (review fix MED) — `HF_TOKEN` / `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / all `SOUP_*` are dropped before exec; only `PATH` / `HOME` / `USER` / `USERPROFILE` / `TMP` / `TEMP` / locale + llama.cpp-recognised `LLAMA_CPP_HOME` / `GGML_*` / `OMP_NUM_THREADS` are forwarded. Subprocess uses list args (no shell). Closed allowlist `_SUBCOMMAND_TO_BINARY: MappingProxyType` rejects unknown subcommands; `_validate_arg` rejects null-byte / newline / carriage-return / >1024-char per arg; `_MAX_ARGS=64` cap. New `utils/onboarding.py` `render_onboarding_yaml` enforces `is_under_cwd` on the `output` field at render time (review fix MED — without this, a crafted answer dict like `output: ../../etc/cron.d/x` would silently propagate into the rendered YAML). Literal allowlist on `task` (8 entries) and `quantization` (`4bit` / `8bit` / `none`); `epochs ∈ [1, 10]`; null-byte rejection on every string with field-named error message. New `utils/sweep_config.py` enforces a strict scalar allowlist (`str` / `int` / `float` / `bool`) on every param value (review fix LOW — without this, `params: {lr: [{__class__: evil}]}` would smuggle non-scalar YAML through to runtime); `_MAX_FILE_BYTES=256KB`, `_MAX_PARAM_KEYS=32`, `_MAX_VALUES_PER_KEY=64`, `_MAX_PARAM_KEY_LEN=128`. `SweepSpec.params` is `MappingProxyType[str, Tuple[Any, ...]]` for genuine `frozen=True` immutability (review fix HIGH — `frozen=True` on a `Dict[str, List[Any]]` field only prevents reassignment, not mutation). Same fix applied to `ConsolidationPlan.shard_files: Tuple[str, ...]` and `DelinearizePlan.weight_files: Tuple[str, ...]`. New `utils/gpu_monitor.py` `parse_nvidia_smi_csv` rejects null-byte in GPU name field; subprocess uses list args + 5s timeout + `OSError`/`TimeoutExpired` swallowed (never raises). `detect_apple_silicon` rewritten to use clean `platform.system()` / `platform.machine()` checks (review fix HIGH — the v0.44.0 first-cut had a parser-priority bug `if X if Y else Z:` that produced the right answer by coincidence on every platform but would silently break on any future branch refactor). New `utils/sse_train_stream.py` `TrainEvent` validates `type` against a closed `_VALID_TYPES = frozenset({metric, status, log, eval})`, rejects non-finite `ts`, rejects null-byte / >1024-char `message`. `to_payload` filters output to a closed `_ALLOWED_KEYS` allowlist (defence-in-depth against payload key drift from refactors). `format_sse_frame` emits the standard W3C SSE wire format `data: {json}\n\n` with `ensure_ascii=False`. New `utils/tool_outputs.py` `ToolOutputsBuffer` is thread-safe (`threading.Lock`) backed by `collections.deque(maxlen=_MAX_RECORDS=1000)` (review fix MED — replaces an O(N) `self.records[-N:]` list slice that triggered GC pressure on overflow); validates name (no null-byte, ≤128 chars), bool rejection on `started_ts` / `duration_ms` (matches project bool-as-int policy), `success` must be exactly `bool` not `int`, output preview truncated to 4096 chars. `ToolCallTimer.__exit__` returns `False` explicitly per project readability policy (does not suppress exceptions). New `utils/llama_server_timings.py` `parse_timings` tolerates missing keys (defaults to None), clamps `kv_cache_pct` to `[0, 100]`, rejects bool / non-finite / negative ints in numeric coercion. `format_kv_bar` validates `width ∈ (0, 200]`. New `utils/tail_latency.py` enforces `MAX_SAMPLES=1_000_000` DoS cap on every iteration; bool rejection + non-finite rejection + alpha bounds. New `soup_cli/ui/plugins/__init__.py` registry: tab name regex `^[a-z0-9][a-z0-9\-]{0,30}$`, `_MAX_TABS=32` cap, `list_tabs()` returns `MappingProxyType` defensive view, `load_plugins()` swallows per-plugin exceptions at WARNING level so one broken plugin doesn't crash the whole UI. `_NAME_DISALLOWED` in `utils/shortcuts.py` is `frozenset` (review fix LOW — was mutable `set` at module level). New `utils/ui_env.py` host validator restricts to `[a-zA-Z0-9.-:]` ≤253 chars + null-byte rejection; port `[1, 65535]`; key length cap 256 + null-byte rejection. Known limitations: (1) Live monitoring SSE endpoint deferred — schema + frame formatter ship, FastAPI route in v0.44.1. (2) Phone visibility flag wiring (`soup ui --public --auth-token` + QR print at startup) deferred to v0.44.1. (3) `soup merge-sharded-fsdp-weights` and `soup delinearize-llama4` are plan-only — torch-side runtime in v0.44.1. (4) `soup llama ` does not bundle llama.cpp; the binary must be on PATH (`shutil.which`) and an actionable `FileNotFoundError` fires when missing. (5) `soup serve --reasoning-parser ` allowlist ships, runtime hook in v0.44.1. (6) Apple Silicon `soup monitor` prints a yellow advisory and exits with `nvidia-smi unavailable` rc=1 — `powermetrics` parser in v0.44.1. - **v0.43.0 — Tracker & Eval Pro**: 18 features across 4 Parts. New `soup_cli/utils/trackers.py` ships an immutable `_REPORT_TO_BACKENDS` `MappingProxyType` allowlist (`wandb` / `tensorboard` / `mlflow` / `swanlab` / `trackio` / `none`); `validate_tracker_name` rejects non-string / empty / null-byte / >32-char inputs. PostHog telemetry is opt-IN (default OFF) via `SOUP_TELEMETRY=1` env var; `build_telemetry_payload` schema is closed-key (soup_version / command / python major.minor / os / arch / duration) — no model names, dataset paths, or config contents leak in the payload. Live PostHog network code deferred to v0.43.1. New `--tracker` flag on `soup train` mutually exclusive with `--wandb` / `--tensorboard` via `resolve_report_to`. `tracker_backend = tracker` capture in `commands/train.py` defends against the Typer parameter being shadowed by the local `ExperimentTracker()` instance (review fix HIGH #1 — silent `--tracker` drop regression). `nlg_metrics.py` is pure-math: BLEU `_lcs_length` allocates a fresh row per outer iter (review fix HIGH #2 — first-cut had a double-buffer re-zero that destroyed the last row). `effective_tokens_per_second` returns `None` on non-positive wall_clock (no fabrication). `eval/calibrate.py` `CalibrationReport` is `@dataclass(frozen=True)` with `FrozenInstanceError` test (matches v0.32.0 / v0.39.0 / v0.41.0 frozen-dataclass policy); `run_calibration` caps prompts at 10_000 (DoS defence). `eval/arena.py` `Tournament` enforces 256-model cap, 1M-match cap, `MappingProxyType` view on `ratings` so external callers cannot mutate Elo state (LOW review fix). `_validate_model_name` rejects null-byte / >128-char AND Rich markup metacharacters `[` / `]` (security review fix LOW #5 — leaderboard markup-injection defence). `update_elo` rejects bool / non-finite / NaN `k`. `eval/benchmarks_v0_43.py` ships `NEW_BENCHMARKS_V0_43 = frozenset({"ceval","cmmlu","aider_polyglot"})` + `MappingProxyType`-wrapped metadata; live Aider Polyglot runner deferred to v0.43.1. `utils/profiling_v0_43.py` `resolve_snapshot_path` rejects null-byte / `.` / `..` / path separators in `run_id`; `base_dir` rejects null-byte / absolute / `..` segments before realpath join (security review fix HIGH #1) — closes a Windows short-name path escape. `memory_snapshot_context` narrows the `RuntimeError` catch to the `record()` entry call only (review fix HIGH #2 — wide catch would have triggered `generator already executing` on user-body RuntimeError); yields `None` when torch / CUDA / `_record_memory_history` API missing. `nccl_bandwidth_check` rejects bool / non-finite / negative `measured_gb_per_sec`; reference table is `MappingProxyType`-wrapped. `utils/vscode_setup.py` `build_launch_json` rejects null-byte / newline / >512-char `config_path` so a crafted argument cannot inject Python args into the generated JSON. `write_vscode_launch` uses `os.lstat + S_ISLNK` symlink rejection at the target path regardless of `force` (security review fix MEDIUM #3 — TOCTOU defence; without this, force=True would follow a pre-placed symlink and overwrite a file outside cwd; mirrors v0.33.0 #22 / v0.40.2 #51 policy). `utils/demo_bundles.py` `copy_bundle_to` validates every JSONL line via `json.loads`, enforces 50 MB cap, stages writes to `.tmp` with `os.lstat + S_ISLNK` rejection (security review fix HIGH #2) and `os.replace`-atomic rename on success — mid-stream rejection never leaves a partial file. New `soup data demo` Typer command: `--output` containment-checked via shared `is_under_cwd`. CLI output prints `_esc(written)` to defend against Rich markup injection through a crafted output path (review fix LOW). `_lcs_length` algorithmic correctness fix (review fix HIGH #2) + BLEU geometric-mean policy fix (returns 0 on any zero precision unless `smooth=True`) (review fix HIGH #1). All `math` imports moved to module-level (PEP 8 / project lazy-import policy: only torch / transformers / peft / trl / mlx are lazy). Known limitations: (1) MLflow / SwanLab / Trackio live integration deferred to v0.43.1 — schema accepts the names and threads into HF `TrainingArguments.report_to` but Soup does not bundle the upstream packages; HF Trainer raises a generic ImportError when the package is absent. (2) PostHog telemetry deferred — payload schema + `SOUP_TELEMETRY=1` env var ship, but no network code; `build_telemetry_payload` is purely a schema-locking stub (mirrors v0.27.0 MII / v0.37.0 multipack / v0.42.0 fsspec stub-then-live pattern). (3) Aider Polyglot live runner deferred — `aider_polyglot` is in the benchmark allowlist but live wiring requires the upstream `aider-chat` package + Docker. (4) `soup doctor --nccl` measurement CLI surface deferred — `nccl_bandwidth_check` accepts a measured value and classifies it, but does not yet measure via `torch.distributed.all_reduce`. (5) `examples/data/` fixtures shipped at the repo root (not as package data inside `soup_cli/`); `_bundle_source_path` derives `repo_root` via `str(importlib.resources.files("soup_cli"))` which is editable-install + wheel friendly but may break in zipapp / namespace-package installs. diff --git a/pyproject.toml b/pyproject.toml index a8d362d..fa62f54 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.45.0" +version = "0.46.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index ddd33f2..50902da 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.45.0" +__version__ = "0.46.0" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index 8b3b178..07ec943 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -157,6 +157,15 @@ app.add_typer( help="List, enable, disable Soup plugins (v0.45.0).", ) +# v0.46.0 Part B — Agent Forge. +from soup_cli.commands import agent as agent_cmd # noqa: E402 + +app.add_typer( + agent_cmd.app, + name="agent", + help="Agent Forge: spec -> tool-calling dataset / train / eval (v0.46.0).", +) + # Register data generate as a subcommand of data data.app.command(name="generate")(generate.generate) diff --git a/soup_cli/commands/agent.py b/soup_cli/commands/agent.py new file mode 100644 index 0000000..010928e --- /dev/null +++ b/soup_cli/commands/agent.py @@ -0,0 +1,304 @@ +"""soup agent — Agent Forge: spec → tool-calling SFT dataset / train / eval. + +v0.46.0 Part B. Live ``train`` and ``eval`` wrappers print the planned +sub-command rather than re-entering the Typer app in-process (matches the +``soup quantize`` design from v0.44.0 Part D). +""" + +from __future__ import annotations + +import json +import shlex +from typing import Optional + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +console = Console() + +app = typer.Typer(no_args_is_help=True) + + +@app.command() +def synth( + spec: str = typer.Option( + ..., "--spec", "-s", + help="Path to OpenAPI / MCP / GraphQL spec (YAML or JSON, under cwd).", + ), + output: str = typer.Option( + "agent_dataset.jsonl", "--output", "-o", + help="Where to write the synthesised JSONL dataset (under cwd).", + ), + kind: Optional[str] = typer.Option( + None, "--kind", "-k", + help="Spec kind override: openapi | mcp | graphql. Auto-detected if omitted.", + ), + examples_per_endpoint: int = typer.Option( + 1, "--examples-per-endpoint", "-n", min=1, max=32, + help="Number of synthetic rows to emit per endpoint.", + ), +): + """Parse an API spec and synthesise a tool-calling SFT dataset.""" + from soup_cli.utils.agent_forge import ( + load_spec_file, + parse_spec, + synthesise_dataset, + write_dataset, + ) + + try: + spec_dict = load_spec_file(spec) + except (ValueError, FileNotFoundError, TypeError) as exc: + console.print(f"[red]Spec load failed:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + except Exception as exc: # noqa: BLE001 — yaml/json parse errors + console.print(f"[red]Spec parse failed:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + + try: + endpoints, report = parse_spec(spec_dict, kind=kind) + except (ValueError, TypeError) as exc: + console.print(f"[red]Parse failed:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + + if not endpoints: + console.print( + "[yellow]No endpoints discovered.[/] " + f"Detected kind: [bold]{escape(report.spec_kind)}[/]." + ) + for w in report.warnings[:5]: + console.print(f" [dim]- {escape(w)}[/]") + raise typer.Exit(1) + + rows = synthesise_dataset(endpoints, examples_per_endpoint=examples_per_endpoint) + try: + out_path = write_dataset(rows, output) + except (ValueError, TypeError) as exc: + console.print(f"[red]Write failed:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + + table = Table(title=f"Agent Forge — {escape(report.spec_kind)}") + table.add_column("Tool", style="bold cyan") + table.add_column("Method", style="magenta") + table.add_column("Path") + # ep.path is partly user-controlled (from the spec) — escape every cell + # before handing to Rich Table (matches v0.43.0 Part B Tournament policy). + for ep in endpoints[:20]: + table.add_row(escape(ep.tool), escape(ep.method), escape(ep.path)) + console.print(table) + if len(endpoints) > 20: + console.print(f"[dim]... and {len(endpoints) - 20} more[/]") + + console.print( + Panel( + f"Spec kind: [bold]{escape(report.spec_kind)}[/]\n" + f"Endpoints: [bold]{report.endpoint_count}[/] " + f"(skipped duplicates: {report.skipped})\n" + f"Rows: [bold]{len(rows)}[/]\n" + f"Output: [bold]{escape(out_path)}[/]", + title="[bold green]Agent Forge — synth complete[/]", + ) + ) + for w in report.warnings[:5]: + console.print(f"[yellow]warning:[/] {escape(w)}") + + +@app.command() +def train( + spec: str = typer.Option(..., "--spec", "-s", help="API spec (under cwd)."), + base: str = typer.Option( + ..., "--base", "-b", + help="Base model HF repo id to fine-tune.", + ), + dataset_out: str = typer.Option( + "agent_dataset.jsonl", "--dataset-out", + help="Where the synth step writes its dataset (under cwd).", + ), + output_dir: str = typer.Option( + "./agent_train_output", "--output-dir", + help="Where the planned soup train run will store checkpoints.", + ), + examples_per_endpoint: int = typer.Option( + 4, "--examples-per-endpoint", "-n", min=1, max=32, + ), +): + """One-shot wrapper: synth + planned soup train invocation (printed).""" + from soup_cli.utils.agent_forge import ( + load_spec_file, + parse_spec, + synthesise_dataset, + write_dataset, + ) + + # CRITICAL security fix: reject newline/NUL/oversize in --base and + # --output-dir BEFORE building the recipe YAML string. A crafted + # --base "evil\ntraining:\n epochs: 9999" would inject YAML keys + # into the rendered recipe. + for label, value in (("--base", base), ("--output-dir", output_dir)): + if not isinstance(value, str) or not value: + console.print(f"[red]{label} must be a non-empty string[/]") + raise typer.Exit(2) + if "\x00" in value or "\n" in value or "\r" in value: + console.print(f"[red]{label} contains NUL or newline[/]") + raise typer.Exit(2) + if len(value) > 4096: + console.print(f"[red]{label} exceeds 4096 chars[/]") + raise typer.Exit(2) + + try: + spec_dict = load_spec_file(spec) + endpoints, report = parse_spec(spec_dict) + except (ValueError, TypeError, FileNotFoundError) as exc: + console.print(f"[red]Spec error:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + + if not endpoints: + console.print("[red]No endpoints discovered; aborting train.[/]") + raise typer.Exit(1) + + rows = synthesise_dataset(endpoints, examples_per_endpoint=examples_per_endpoint) + try: + ds_path = write_dataset(rows, dataset_out) + except (ValueError, TypeError) as exc: + console.print(f"[red]Dataset write failed:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + + planned_cmd = " ".join( + shlex.quote(p) for p in [ + "soup", "train", + "--config", "agent_train.yaml", + "--yes", + ] + ) + recipe_yaml = ( + f"base: {base}\n" + "task: sft\n" + "data:\n" + f" train: {ds_path}\n" + " format: tool-calling\n" + "training:\n" + " epochs: 3\n" + " lr: 2.0e-5\n" + " batch_size: auto\n" + f"output: {output_dir}\n" + ) + + console.print( + Panel( + f"Spec: [bold]{escape(spec)}[/]\n" + f"Endpoints: [bold]{report.endpoint_count}[/]\n" + f"Dataset: [bold]{escape(ds_path)}[/]\n" + f"Base: [bold]{escape(base)}[/]\n" + f"Output: [bold]{escape(output_dir)}[/]\n\n" + f"[bold]Planned recipe (agent_train.yaml):[/]\n{escape(recipe_yaml)}\n" + f"[bold]Run:[/] {escape(planned_cmd)}", + title="[bold green]Agent Forge — train plan[/]", + ) + ) + console.print( + "[yellow]Note:[/] live in-process training is intentionally not " + "re-entered (Typer commands aren't safe to re-enter); copy the recipe " + "into agent_train.yaml and run the command above." + ) + + +@app.command() +def eval( + spec: str = typer.Option(..., "--spec", "-s", help="API spec (under cwd)."), + predictions: str = typer.Option( + ..., + "--predictions", + "-p", + help=( + "JSONL of model outputs, one per line, with at minimum " + "{tool: , arguments: {...}}." + ), + ), +): + """Score predicted tool-calls against the spec's tool catalog. + + Each prediction row gets two checks: + 1. ``tool`` matches a known endpoint in the spec. + 2. ``arguments`` only references parameters declared on that endpoint. + """ + import os + import stat as _stat + + from soup_cli.utils.agent_forge import load_spec_file, parse_spec + from soup_cli.utils.paths import is_under_cwd + + max_pred_lines = 1_000_000 + + if not isinstance(predictions, str) or not predictions or "\x00" in predictions: + console.print("[red]predictions path must be non-empty NUL-free string[/]") + raise typer.Exit(1) + if not is_under_cwd(predictions): + console.print("[red]predictions path must stay under cwd[/]") + raise typer.Exit(1) + # Symlink TOCTOU defence (mirrors load_spec_file policy). + try: + if _stat.S_ISLNK(os.lstat(predictions).st_mode): + console.print("[red]predictions path must not be a symlink[/]") + raise typer.Exit(1) + except FileNotFoundError: + console.print("[red]predictions file not found[/]") + raise typer.Exit(1) from None + + try: + spec_dict = load_spec_file(spec) + endpoints, _ = parse_spec(spec_dict) + except (ValueError, TypeError, FileNotFoundError) as exc: + console.print(f"[red]Spec error:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + tool_to_params = {ep.tool: set(ep.parameters) for ep in endpoints} + + total = 0 + tool_ok = 0 + args_ok = 0 + try: + with open(predictions, "r", encoding="utf-8") as fh: + for line in fh: + line = line.strip() + if not line: + continue + if total >= max_pred_lines: + break + total += 1 + try: + row = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(row, dict): + continue + tool = row.get("tool") + if isinstance(tool, str) and tool in tool_to_params: + tool_ok += 1 + args = row.get("arguments") or {} + if isinstance(args, dict): + invalid = [ + k for k in args.keys() + if k not in tool_to_params[tool] + ] + if not invalid: + args_ok += 1 + except OSError as exc: + console.print(f"[red]Predictions read failed:[/] {escape(str(exc))}") + raise typer.Exit(1) from exc + + if total == 0: + console.print("[yellow]No predictions to score.[/]") + raise typer.Exit(1) + + tool_pct = 100.0 * tool_ok / total + args_pct = 100.0 * args_ok / total + console.print( + Panel( + f"Predictions: [bold]{total}[/]\n" + f"Tool match: [bold]{tool_ok}[/] ({tool_pct:.1f}%)\n" + f"Args valid: [bold]{args_ok}[/] ({args_pct:.1f}%)", + title="[bold green]Agent Forge — eval[/]", + ) + ) diff --git a/soup_cli/utils/agent_forge.py b/soup_cli/utils/agent_forge.py new file mode 100644 index 0000000..1a34ec3 --- /dev/null +++ b/soup_cli/utils/agent_forge.py @@ -0,0 +1,576 @@ +"""v0.46.0 Part B — Agent Forge: spec → tool-calling SFT dataset. + +Parses OpenAPI 3.x, MCP server manifests, and GraphQL introspection JSON +into a canonical ``Endpoint`` shape, then synthesises a tool-calling SFT +dataset where each row is ``{messages: [user, assistant{tool_calls}], +tool: , source_endpoint: }``. + +The parser surface is intentionally parser-only — no network code, no +``$ref`` resolution that would let a crafted spec read arbitrary files. +``$ref`` strings are left as opaque markers and a warning is surfaced; +operators wanting full resolution should run ``openapi-spec-validator`` +upstream and feed the bundled JSON in. + +Live ``soup agent train`` orchestrator + ``soup agent eval`` sandbox +scoring re-use the v0.25.0 RLVR ``code_exec`` sandbox; this module ships +the parse-and-synth layer. +""" + +from __future__ import annotations + +import json +import os +import re +from dataclasses import dataclass, field +from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple + +from soup_cli.utils.paths import is_under_cwd + +_TOOL_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_.\-]{0,127}$") +_MAX_ENDPOINTS = 10_000 +_MAX_SPEC_BYTES = 5 * 1024 * 1024 # 5 MiB +_MAX_DESCRIPTION = 512 +_MAX_ROWS_PER_ENDPOINT = 32 +_ALLOWED_SPEC_KINDS = frozenset({"openapi", "mcp", "graphql"}) +_HTTP_METHODS = frozenset( + {"get", "post", "put", "patch", "delete", "head", "options", "trace"} +) + + +@dataclass(frozen=True) +class Endpoint: + """Canonical endpoint representation across all three spec kinds.""" + + tool: str + method: str + path: str + description: str + parameters: Tuple[str, ...] # parameter names only — schema details opaque + spec_kind: str + + +@dataclass(frozen=True) +class SynthRow: + """One row in the generated tool-calling SFT dataset.""" + + messages: Tuple[Mapping[str, Any], ...] + tool: str + source_endpoint: str + + def to_dict(self) -> Dict[str, Any]: + return { + "messages": [dict(m) for m in self.messages], + "tool": self.tool, + "source_endpoint": self.source_endpoint, + } + + +@dataclass(frozen=True) +class SpecReport: + """Summary of a parsed spec for the CLI.""" + + spec_kind: str + endpoint_count: int + skipped: int + warnings: Tuple[str, ...] = field(default_factory=tuple) + + +# --------------------------------------------------------------------------- +# Validators +# --------------------------------------------------------------------------- + + +def _validate_tool_name(name: str) -> str: + if not isinstance(name, str): + raise TypeError("tool name must be a string") + if not _TOOL_NAME_RE.match(name): + raise ValueError( + "tool name must match ^[A-Za-z_][A-Za-z0-9_.-]{0,127}$" + ) + return name + + +def _validate_method(method: str) -> str: + if not isinstance(method, str): + raise TypeError("method must be a string") + canonical = method.strip().lower() + if canonical not in _HTTP_METHODS: + raise ValueError(f"unknown HTTP method: {method!r}") + return canonical + + +def _validate_path(path: str) -> str: + if not isinstance(path, str): + raise TypeError("path must be a string") + if not path or "\x00" in path or "\n" in path or "\r" in path: + raise ValueError("path must be non-empty single-line NUL-free string") + if len(path) > 1024: + raise ValueError("path exceeds 1024 chars") + return path + + +def _truncate_desc(desc: Any) -> str: + if desc is None: + return "" + if not isinstance(desc, str): + return "" + if "\x00" in desc: + desc = desc.replace("\x00", "") + desc = desc.strip() + if len(desc) > _MAX_DESCRIPTION: + return desc[: _MAX_DESCRIPTION - 3] + "..." + return desc + + +def _sanitise_tool_id(*parts: str) -> str: + """Build a tool name from spec parts, replacing non-id chars with '_'.""" + raw = "_".join(p for p in parts if p) + raw = re.sub(r"[^A-Za-z0-9_.\-]", "_", raw) + if not raw or not re.match(r"^[A-Za-z_]", raw): + raw = "tool_" + raw + return raw[:128] + + +# --------------------------------------------------------------------------- +# OpenAPI parser +# --------------------------------------------------------------------------- + + +def parse_openapi(spec: Mapping[str, Any]) -> Tuple[List[Endpoint], List[str]]: + """Parse an OpenAPI 3.x ``dict``. Returns (endpoints, warnings).""" + if not isinstance(spec, dict): + raise TypeError("openapi spec must be a dict") + version = spec.get("openapi", "") + warnings: List[str] = [] + if not isinstance(version, str) or not version.startswith("3."): + warnings.append( + f"unrecognised openapi version: {version!r}; parser is OpenAPI 3.x" + ) + paths = spec.get("paths") + if not isinstance(paths, dict): + return [], ["spec has no 'paths' object"] + endpoints: List[Endpoint] = [] + for path, ops in paths.items(): + if not isinstance(path, str) or not isinstance(ops, dict): + continue + for method, op in ops.items(): + if not isinstance(method, str): + continue + lower = method.lower() + if lower not in _HTTP_METHODS: + continue + if not isinstance(op, dict): + continue + op_id = op.get("operationId") + if not isinstance(op_id, str) or not op_id: + op_id = _sanitise_tool_id(lower, path.strip("/")) + try: + tool = _validate_tool_name(_sanitise_tool_id(op_id)) + _validate_method(lower) + _validate_path(path) + except (TypeError, ValueError) as exc: + warnings.append(f"skip {method.upper()} {path}: {exc}") + continue + params_raw = op.get("parameters") + param_names: List[str] = [] + if isinstance(params_raw, list): + for p in params_raw: + if not isinstance(p, dict): + continue + if "$ref" in p: + warnings.append("$ref left unresolved") + continue + name = p.get("name") + if isinstance(name, str) and name and "\x00" not in name: + param_names.append(name[:128]) + endpoints.append( + Endpoint( + tool=tool, + method=lower, + path=path, + description=_truncate_desc( + op.get("summary") or op.get("description") + ), + parameters=tuple(param_names), + spec_kind="openapi", + ) + ) + if len(endpoints) >= _MAX_ENDPOINTS: + warnings.append( + f"endpoint cap {_MAX_ENDPOINTS} reached; truncating" + ) + return endpoints, warnings + return endpoints, warnings + + +# --------------------------------------------------------------------------- +# MCP manifest parser +# --------------------------------------------------------------------------- + + +def parse_mcp(manifest: Mapping[str, Any]) -> Tuple[List[Endpoint], List[str]]: + """Parse an MCP server manifest's ``tools`` array.""" + if not isinstance(manifest, dict): + raise TypeError("mcp manifest must be a dict") + warnings: List[str] = [] + tools_raw = manifest.get("tools") + if not isinstance(tools_raw, list): + return [], ["mcp manifest missing 'tools' array"] + endpoints: List[Endpoint] = [] + for entry in tools_raw: + if not isinstance(entry, dict): + continue + name = entry.get("name") + if not isinstance(name, str) or not name: + warnings.append("mcp tool missing name; skipped") + continue + try: + tool = _validate_tool_name(_sanitise_tool_id(name)) + except (TypeError, ValueError) as exc: + warnings.append(f"skip mcp tool {name!r}: {exc}") + continue + params: List[str] = [] + input_schema = entry.get("inputSchema") + if isinstance(input_schema, dict): + props = input_schema.get("properties") + if isinstance(props, dict): + for key in props: + if isinstance(key, str) and key and "\x00" not in key: + params.append(key[:128]) + try: + mcp_path = _validate_path(f"mcp://{name}") + except (TypeError, ValueError) as exc: + warnings.append(f"skip mcp tool {name!r}: invalid path: {exc}") + continue + endpoints.append( + Endpoint( + tool=tool, + method="invoke", + path=mcp_path, + description=_truncate_desc(entry.get("description")), + parameters=tuple(params), + spec_kind="mcp", + ) + ) + if len(endpoints) >= _MAX_ENDPOINTS: + warnings.append(f"endpoint cap {_MAX_ENDPOINTS} reached") + return endpoints, warnings + return endpoints, warnings + + +# --------------------------------------------------------------------------- +# GraphQL introspection parser +# --------------------------------------------------------------------------- + + +def parse_graphql(intro: Mapping[str, Any]) -> Tuple[List[Endpoint], List[str]]: + """Parse a GraphQL introspection ``dict`` (``__schema`` envelope OK). + + Treats every Query / Mutation field as a tool-calling endpoint. + """ + if not isinstance(intro, dict): + raise TypeError("graphql introspection must be a dict") + warnings: List[str] = [] + data = intro.get("data") if "data" in intro else intro + if not isinstance(data, dict): + return [], ["graphql introspection: no 'data' or schema dict"] + schema = data.get("__schema") if "__schema" in data else data + if not isinstance(schema, dict): + return [], ["graphql introspection: no '__schema' field"] + type_list = schema.get("types") + if not isinstance(type_list, list): + return [], ["graphql introspection: missing 'types'"] + query_type = schema.get("queryType") or {} + mutation_type = schema.get("mutationType") or {} + q_name = query_type.get("name") if isinstance(query_type, dict) else None + m_name = mutation_type.get("name") if isinstance(mutation_type, dict) else None + endpoints: List[Endpoint] = [] + for t in type_list: + if not isinstance(t, dict): + continue + type_name = t.get("name") + if type_name not in (q_name, m_name): + continue + fields = t.get("fields") + if not isinstance(fields, list): + continue + method = "query" if type_name == q_name else "mutation" + for f in fields: + if not isinstance(f, dict): + continue + fname = f.get("name") + if not isinstance(fname, str) or not fname: + continue + try: + tool = _validate_tool_name(_sanitise_tool_id(method, fname)) + except (TypeError, ValueError) as exc: + warnings.append(f"skip {fname!r}: {exc}") + continue + args = f.get("args") or [] + arg_names: List[str] = [] + if isinstance(args, list): + for a in args: + if isinstance(a, dict): + an = a.get("name") + if isinstance(an, str) and an and "\x00" not in an: + arg_names.append(an[:128]) + try: + gql_path = _validate_path(f"graphql://{fname}") + except (TypeError, ValueError) as exc: + warnings.append(f"skip {fname!r}: invalid path: {exc}") + continue + endpoints.append( + Endpoint( + tool=tool, + method=method, + path=gql_path, + description=_truncate_desc(f.get("description")), + parameters=tuple(arg_names), + spec_kind="graphql", + ) + ) + if len(endpoints) >= _MAX_ENDPOINTS: + warnings.append(f"endpoint cap {_MAX_ENDPOINTS} reached") + return endpoints, warnings + return endpoints, warnings + + +# --------------------------------------------------------------------------- +# Dispatcher + dataset writer +# --------------------------------------------------------------------------- + + +def detect_spec_kind(spec: Mapping[str, Any]) -> str: + """Best-effort detection of spec kind from a parsed dict. + + Returns one of {"openapi", "mcp", "graphql"} or raises ``ValueError``. + """ + if not isinstance(spec, dict): + raise TypeError("spec must be a dict") + if isinstance(spec.get("openapi"), str) and isinstance(spec.get("paths"), dict): + return "openapi" + if isinstance(spec.get("tools"), list) and not isinstance( + spec.get("paths"), dict + ): + return "mcp" + if "__schema" in spec or ( + isinstance(spec.get("data"), dict) and "__schema" in spec["data"] + ): + return "graphql" + raise ValueError( + "cannot detect spec kind — must be OpenAPI 3.x / MCP / GraphQL" + ) + + +def parse_spec( + spec: Mapping[str, Any], kind: Optional[str] = None +) -> Tuple[List[Endpoint], SpecReport]: + """Parse a spec dict with optional explicit ``kind`` override.""" + if kind is None: + resolved = detect_spec_kind(spec) + else: + if not isinstance(kind, str): + raise TypeError("kind must be a string") + resolved = kind.strip().lower() + if resolved not in _ALLOWED_SPEC_KINDS: + raise ValueError(f"unknown spec kind: {kind!r}") + if resolved == "openapi": + endpoints, warnings = parse_openapi(spec) + elif resolved == "mcp": + endpoints, warnings = parse_mcp(spec) + else: + endpoints, warnings = parse_graphql(spec) + # Dedup by tool name (last write wins is unsafe — first wins, preserves order). + seen: Dict[str, Endpoint] = {} + skipped = 0 + for ep in endpoints: + if ep.tool in seen: + skipped += 1 + continue + seen[ep.tool] = ep + report = SpecReport( + spec_kind=resolved, + endpoint_count=len(seen), + skipped=skipped, + warnings=tuple(warnings), + ) + return list(seen.values()), report + + +def endpoint_to_rows( + endpoint: Endpoint, examples_per_endpoint: int = 1 +) -> List[SynthRow]: + """Synthesise ``examples_per_endpoint`` rows for one endpoint. + + A row is one user-question / assistant-tool-call pair. We do NOT make + network calls here; the assistant content embeds an empty arguments + object that the trainer is meant to learn to fill from the user query. + """ + if not isinstance(endpoint, Endpoint): + raise TypeError("endpoint must be an Endpoint") + if isinstance(examples_per_endpoint, bool) or not isinstance( + examples_per_endpoint, int + ): + raise TypeError("examples_per_endpoint must be int (not bool)") + if not (1 <= examples_per_endpoint <= _MAX_ROWS_PER_ENDPOINT): + raise ValueError( + f"examples_per_endpoint must be in [1, {_MAX_ROWS_PER_ENDPOINT}]" + ) + desc = endpoint.description or f"Call {endpoint.tool}" + user_templates = [ + f"Please {desc}.", + f"How do I use {endpoint.tool}?", + f"Run the {endpoint.tool} action with sensible defaults.", + ] + rows: List[SynthRow] = [] + for i in range(examples_per_endpoint): + user_msg = user_templates[i % len(user_templates)] + tool_args: Dict[str, str] = {p: "" for p in endpoint.parameters} + assistant_msg = { + "role": "assistant", + "tool_calls": [ + { + "id": f"call_{i}", + "type": "function", + "function": { + "name": endpoint.tool, + "arguments": json.dumps(tool_args, sort_keys=True), + }, + } + ], + } + rows.append( + SynthRow( + messages=( + {"role": "user", "content": user_msg}, + assistant_msg, + ), + tool=endpoint.tool, + source_endpoint=endpoint.path, + ) + ) + return rows + + +def synthesise_dataset( + endpoints: Sequence[Endpoint], examples_per_endpoint: int = 1 +) -> List[SynthRow]: + """Synthesise a flat list of training rows from a list of endpoints.""" + if isinstance(endpoints, (str, bytes)): + raise TypeError("endpoints must be a sequence of Endpoint") + out: List[SynthRow] = [] + for ep in endpoints: + out.extend(endpoint_to_rows(ep, examples_per_endpoint)) + return out + + +def load_spec_file(spec_path: str) -> Mapping[str, Any]: + """Load a YAML/JSON spec from disk with cwd containment + size cap. + + Symlinks are rejected (TOCTOU defence, mirrors v0.45.0 Part E policy). + """ + if not isinstance(spec_path, str): + raise TypeError("spec_path must be a string") + if not spec_path or "\x00" in spec_path: + raise ValueError("spec_path must be non-empty NUL-free string") + if not is_under_cwd(spec_path): + raise ValueError( + f"spec_path must stay under cwd: {os.path.basename(spec_path)}" + ) + # lstat BEFORE realpath: project-standard TOCTOU policy (v0.33.0 #22 / + # v0.43.0 Part C / v0.44.0 Part B). The lstat must operate on the + # original (pre-realpath) path so we see the symlink, not its target. + import stat as _stat + + try: + st = os.lstat(spec_path) + except FileNotFoundError as exc: + raise FileNotFoundError(spec_path) from exc + if _stat.S_ISLNK(st.st_mode): + raise ValueError( + f"spec_path must not be a symlink: {os.path.basename(spec_path)}" + ) + real = os.path.realpath(spec_path) + if not os.path.isfile(real): + raise FileNotFoundError(spec_path) + size = os.path.getsize(real) + if size > _MAX_SPEC_BYTES: + raise ValueError(f"spec file exceeds {_MAX_SPEC_BYTES} bytes ({size})") + with open(real, "r", encoding="utf-8") as fh: + text = fh.read() + if spec_path.lower().endswith((".yaml", ".yml")): + import yaml + + loaded = yaml.safe_load(text) + else: + loaded = json.loads(text) + if not isinstance(loaded, dict): + raise ValueError("spec file root must be a JSON/YAML object") + return loaded + + +def write_dataset(rows: Sequence[SynthRow], output_path: str) -> str: + """Write rows as JSONL under cwd; returns realpath written. + + Atomic via staged-tempfile + ``os.replace`` (matches v0.43.0 Part D + ``copy_bundle_to`` policy). Validates every row BEFORE any bytes hit + the target path — a mid-stream ``TypeError`` never leaves a partial + file. Symlink at the target rejected via ``os.lstat`` (TOCTOU). + """ + import stat as _stat + import tempfile + + if not isinstance(output_path, str): + raise TypeError("output_path must be a string") + if not output_path or "\x00" in output_path: + raise ValueError("output_path must be non-empty NUL-free string") + if not is_under_cwd(output_path): + raise ValueError( + f"output_path must stay under cwd: {os.path.basename(output_path)}" + ) + # Reject a pre-placed symlink at the target — defends against + # `.jsonl -> /etc/cron.d/x` overwrite. + try: + st = os.lstat(output_path) + if _stat.S_ISLNK(st.st_mode): + raise ValueError( + f"output_path must not be a symlink: " + f"{os.path.basename(output_path)}" + ) + except FileNotFoundError: + pass + real = os.path.realpath(output_path) + parent = os.path.dirname(real) or "." + os.makedirs(parent, exist_ok=True) + fd, tmp = tempfile.mkstemp(prefix=".agent_forge_", suffix=".tmp", dir=parent) + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + for row in rows: + if not isinstance(row, SynthRow): + raise TypeError("rows must all be SynthRow") + fh.write(json.dumps(row.to_dict(), sort_keys=True)) + fh.write("\n") + os.replace(tmp, real) + except BaseException: + try: + os.unlink(tmp) + except OSError: + pass + raise + return real + + +__all__ = [ + "Endpoint", + "SynthRow", + "SpecReport", + "parse_openapi", + "parse_mcp", + "parse_graphql", + "detect_spec_kind", + "parse_spec", + "endpoint_to_rows", + "synthesise_dataset", + "load_spec_file", + "write_dataset", +] diff --git a/tests/test_v0460_part_b.py b/tests/test_v0460_part_b.py new file mode 100644 index 0000000..9a5b96a --- /dev/null +++ b/tests/test_v0460_part_b.py @@ -0,0 +1,828 @@ +"""v0.46.0 Part B — Agent Forge tests.""" + +from __future__ import annotations + +import json +import os +import sys + +import pytest +from typer.testing import CliRunner + +from soup_cli.utils.agent_forge import ( + Endpoint, + SpecReport, + SynthRow, + detect_spec_kind, + endpoint_to_rows, + load_spec_file, + parse_graphql, + parse_mcp, + parse_openapi, + parse_spec, + synthesise_dataset, + write_dataset, +) + +runner = CliRunner() + + +# --------------------------------------------------------------------------- +# parse_openapi +# --------------------------------------------------------------------------- + + +_OPENAPI_SAMPLE = { + "openapi": "3.0.0", + "info": {"title": "Sample", "version": "1.0"}, + "paths": { + "/pets": { + "get": { + "operationId": "listPets", + "summary": "List pets", + "parameters": [ + {"name": "limit", "in": "query"}, + ], + }, + "post": { + "operationId": "createPet", + "summary": "Create a pet", + }, + }, + "/pets/{id}": { + "get": { + "operationId": "getPet", + "parameters": [ + {"name": "id", "in": "path"}, + ], + } + }, + }, +} + + +def test_parse_openapi_basic(): + endpoints, warnings = parse_openapi(_OPENAPI_SAMPLE) + tools = {ep.tool for ep in endpoints} + assert "listPets" in tools + assert "createPet" in tools + assert "getPet" in tools + assert warnings == [] or all("$ref" not in w for w in warnings) + + +def test_parse_openapi_extracts_parameter_names(): + endpoints, _ = parse_openapi(_OPENAPI_SAMPLE) + list_pets = next(ep for ep in endpoints if ep.tool == "listPets") + assert "limit" in list_pets.parameters + + +def test_parse_openapi_method_lowercased(): + endpoints, _ = parse_openapi(_OPENAPI_SAMPLE) + for ep in endpoints: + assert ep.method == ep.method.lower() + + +def test_parse_openapi_non_dict_raises(): + with pytest.raises(TypeError): + parse_openapi("not a dict") # type: ignore[arg-type] + + +def test_parse_openapi_missing_paths_returns_empty(): + eps, warnings = parse_openapi({"openapi": "3.0.0"}) + assert eps == [] + assert warnings + + +def test_parse_openapi_wrong_version_warns(): + _, warnings = parse_openapi({"openapi": "2.0.0", "paths": {}}) + assert any("openapi" in w.lower() for w in warnings) + + +def test_parse_openapi_skips_invalid_methods(): + spec = { + "openapi": "3.0.0", + "paths": { + "/x": { + "BOGUS": {"operationId": "skipMe"}, + "get": {"operationId": "keepMe"}, + } + }, + } + endpoints, _ = parse_openapi(spec) + tools = {ep.tool for ep in endpoints} + assert "keepMe" in tools + assert "skipMe" not in tools + + +def test_parse_openapi_generates_id_when_missing(): + spec = { + "openapi": "3.0.0", + "paths": {"/widgets": {"get": {}}}, + } + endpoints, _ = parse_openapi(spec) + assert endpoints + # Generated id sanitised, starts with letter or underscore + assert endpoints[0].tool[0].isalpha() or endpoints[0].tool.startswith("_") + + +def test_parse_openapi_ref_param_skipped_with_warning(): + spec = { + "openapi": "3.0.0", + "paths": { + "/x": { + "get": { + "operationId": "getX", + "parameters": [{"$ref": "#/components/parameters/X"}], + } + } + }, + } + eps, warnings = parse_openapi(spec) + assert eps + assert any("$ref" in w for w in warnings) + + +# --------------------------------------------------------------------------- +# parse_mcp +# --------------------------------------------------------------------------- + + +_MCP_SAMPLE = { + "tools": [ + { + "name": "search_files", + "description": "Search files by pattern", + "inputSchema": { + "type": "object", + "properties": {"pattern": {"type": "string"}, "path": {"type": "string"}}, + }, + }, + { + "name": "read_file", + "description": "Read a file by path", + "inputSchema": { + "type": "object", + "properties": {"path": {"type": "string"}}, + }, + }, + ] +} + + +def test_parse_mcp_basic(): + endpoints, _ = parse_mcp(_MCP_SAMPLE) + tools = {ep.tool for ep in endpoints} + assert "search_files" in tools + assert "read_file" in tools + + +def test_parse_mcp_extracts_input_schema_props(): + endpoints, _ = parse_mcp(_MCP_SAMPLE) + search = next(ep for ep in endpoints if ep.tool == "search_files") + assert "pattern" in search.parameters + assert "path" in search.parameters + + +def test_parse_mcp_method_is_invoke(): + endpoints, _ = parse_mcp(_MCP_SAMPLE) + for ep in endpoints: + assert ep.method == "invoke" + + +def test_parse_mcp_path_uses_mcp_scheme(): + endpoints, _ = parse_mcp(_MCP_SAMPLE) + for ep in endpoints: + assert ep.path.startswith("mcp://") + + +def test_parse_mcp_missing_tools_returns_empty(): + eps, warnings = parse_mcp({}) + assert eps == [] + assert warnings + + +def test_parse_mcp_missing_name_warns(): + spec = {"tools": [{"description": "no name"}]} + _, warnings = parse_mcp(spec) + assert any("name" in w.lower() for w in warnings) + + +def test_parse_mcp_non_dict_raises(): + with pytest.raises(TypeError): + parse_mcp([]) # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# parse_graphql +# --------------------------------------------------------------------------- + + +_GRAPHQL_SAMPLE = { + "data": { + "__schema": { + "queryType": {"name": "Query"}, + "mutationType": {"name": "Mutation"}, + "types": [ + { + "name": "Query", + "fields": [ + { + "name": "user", + "description": "Fetch a user by id", + "args": [{"name": "id"}], + } + ], + }, + { + "name": "Mutation", + "fields": [ + { + "name": "createUser", + "description": "Create a user", + "args": [{"name": "name"}, {"name": "email"}], + } + ], + }, + ], + } + } +} + + +def test_parse_graphql_basic(): + endpoints, _ = parse_graphql(_GRAPHQL_SAMPLE) + tools = {ep.tool for ep in endpoints} + assert any("user" in t for t in tools) + assert any("createUser" in t for t in tools) + + +def test_parse_graphql_methods(): + endpoints, _ = parse_graphql(_GRAPHQL_SAMPLE) + methods = {ep.method for ep in endpoints} + assert "query" in methods + assert "mutation" in methods + + +def test_parse_graphql_args_captured(): + endpoints, _ = parse_graphql(_GRAPHQL_SAMPLE) + create_user = next(ep for ep in endpoints if "createUser" in ep.tool) + assert "name" in create_user.parameters + assert "email" in create_user.parameters + + +def test_parse_graphql_missing_schema_returns_empty(): + eps, warnings = parse_graphql({"data": {}}) + assert eps == [] + assert warnings + + +def test_parse_graphql_non_dict_raises(): + with pytest.raises(TypeError): + parse_graphql("nope") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# detect_spec_kind + parse_spec +# --------------------------------------------------------------------------- + + +def test_detect_openapi(): + assert detect_spec_kind(_OPENAPI_SAMPLE) == "openapi" + + +def test_detect_mcp(): + assert detect_spec_kind(_MCP_SAMPLE) == "mcp" + + +def test_detect_graphql(): + assert detect_spec_kind(_GRAPHQL_SAMPLE) == "graphql" + + +def test_detect_unknown_raises(): + with pytest.raises(ValueError, match="cannot detect"): + detect_spec_kind({"random": "stuff"}) + + +def test_detect_non_dict_raises(): + with pytest.raises(TypeError): + detect_spec_kind("not a dict") # type: ignore[arg-type] + + +def test_parse_spec_auto_detect(): + endpoints, report = parse_spec(_OPENAPI_SAMPLE) + assert report.spec_kind == "openapi" + assert report.endpoint_count >= 3 + assert isinstance(report, SpecReport) + + +def test_parse_spec_explicit_kind(): + endpoints, report = parse_spec(_MCP_SAMPLE, kind="mcp") + assert report.spec_kind == "mcp" + assert endpoints + + +def test_parse_spec_unknown_kind_rejected(): + with pytest.raises(ValueError, match="unknown spec kind"): + parse_spec(_OPENAPI_SAMPLE, kind="evil") + + +def test_parse_spec_non_string_kind(): + with pytest.raises(TypeError): + parse_spec(_OPENAPI_SAMPLE, kind=123) # type: ignore[arg-type] + + +def test_parse_spec_deduplicates_tools(): + spec = { + "openapi": "3.0.0", + "paths": { + "/x": { + "get": {"operationId": "myTool"}, + }, + "/y": { + "get": {"operationId": "myTool"}, + }, + }, + } + endpoints, report = parse_spec(spec) + assert len({ep.tool for ep in endpoints}) == 1 + assert report.skipped == 1 + + +def test_parse_mcp_rejects_newline_in_name(): + spec = {"tools": [{"name": "evil\nhost", "description": "x"}]} + eps, warnings = parse_mcp(spec) + # Either skipped with warning, or path validation strips/rejects newline + if eps: + for ep in eps: + assert "\n" not in ep.path + assert any("invalid" in w.lower() or "skip" in w.lower() for w in warnings) or not eps + + +def test_parse_graphql_rejects_newline_in_field_name(): + spec = { + "__schema": { + "queryType": {"name": "Query"}, + "types": [{ + "name": "Query", + "fields": [{"name": "evil\nhost", "args": []}], + }], + } + } + eps, _ = parse_graphql(spec) + for ep in eps: + assert "\n" not in ep.path + + +# --------------------------------------------------------------------------- +# endpoint_to_rows + synthesise_dataset +# --------------------------------------------------------------------------- + + +def test_endpoint_to_rows_basic(): + ep = Endpoint( + tool="search", method="get", path="/search", + description="Search the index", parameters=("query",), + spec_kind="openapi", + ) + rows = endpoint_to_rows(ep, examples_per_endpoint=2) + assert len(rows) == 2 + for row in rows: + assert isinstance(row, SynthRow) + assert row.tool == "search" + assert row.source_endpoint == "/search" + # 2 messages: user + assistant-with-tool-call + assert len(row.messages) == 2 + assert row.messages[0]["role"] == "user" + assert row.messages[1]["role"] == "assistant" + assert "tool_calls" in row.messages[1] + + +def test_endpoint_to_rows_arguments_are_json_string(): + ep = Endpoint( + tool="search", method="get", path="/search", + description="", parameters=("q", "limit"), spec_kind="openapi", + ) + rows = endpoint_to_rows(ep, examples_per_endpoint=1) + tc = rows[0].messages[1]["tool_calls"][0] + parsed = json.loads(tc["function"]["arguments"]) + assert set(parsed.keys()) == {"q", "limit"} + + +def test_endpoint_to_rows_bool_examples_rejected(): + ep = Endpoint( + tool="x", method="get", path="/x", + description="", parameters=(), spec_kind="openapi", + ) + with pytest.raises(TypeError): + endpoint_to_rows(ep, examples_per_endpoint=True) # type: ignore[arg-type] + + +def test_endpoint_to_rows_zero_rejected(): + ep = Endpoint( + tool="x", method="get", path="/x", + description="", parameters=(), spec_kind="openapi", + ) + with pytest.raises(ValueError): + endpoint_to_rows(ep, examples_per_endpoint=0) + + +def test_endpoint_to_rows_oversize_rejected(): + ep = Endpoint( + tool="x", method="get", path="/x", + description="", parameters=(), spec_kind="openapi", + ) + with pytest.raises(ValueError): + endpoint_to_rows(ep, examples_per_endpoint=33) + + +def test_endpoint_to_rows_type_check(): + with pytest.raises(TypeError): + endpoint_to_rows("not-an-endpoint", 1) # type: ignore[arg-type] + + +def test_synthesise_dataset_flat_list(): + endpoints, _ = parse_openapi(_OPENAPI_SAMPLE) + rows = synthesise_dataset(endpoints, examples_per_endpoint=2) + assert len(rows) == 2 * len(endpoints) + assert all(isinstance(r, SynthRow) for r in rows) + + +def test_synthesise_dataset_rejects_string(): + with pytest.raises(TypeError): + synthesise_dataset("not a list", 1) # type: ignore[arg-type] + + +def test_synth_row_to_dict_serialisable(): + ep = Endpoint( + tool="x", method="get", path="/x", + description="", parameters=(), spec_kind="openapi", + ) + row = endpoint_to_rows(ep, 1)[0] + d = row.to_dict() + # Round-trips through JSON + json.dumps(d) + + +# --------------------------------------------------------------------------- +# load_spec_file + write_dataset (cwd containment + symlink) +# --------------------------------------------------------------------------- + + +def test_load_spec_file_yaml(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + p = tmp_path / "spec.yaml" + p.write_text("openapi: '3.0.0'\npaths: {}\n", encoding="utf-8") + out = load_spec_file("spec.yaml") + assert out.get("openapi") == "3.0.0" + + +def test_load_spec_file_json(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + p = tmp_path / "spec.json" + p.write_text(json.dumps(_OPENAPI_SAMPLE), encoding="utf-8") + out = load_spec_file("spec.json") + assert out["openapi"] == "3.0.0" + + +def test_load_spec_file_outside_cwd_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + other = tmp_path.parent / "evil.json" + other.write_text("{}", encoding="utf-8") + with pytest.raises(ValueError, match="must stay under cwd"): + load_spec_file(str(other)) + + +def test_load_spec_file_missing(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + load_spec_file("missing.json") + + +def test_load_spec_file_null_byte_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + load_spec_file("evil\x00.json") + + +def test_load_spec_file_non_string(): + with pytest.raises(TypeError): + load_spec_file(123) # type: ignore[arg-type] + + +def test_load_spec_file_oversize_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + p = tmp_path / "big.json" + p.write_bytes(b"{" + b"x" * (6 * 1024 * 1024) + b"}") + with pytest.raises(ValueError, match="exceeds"): + load_spec_file("big.json") + + +def test_load_spec_file_non_dict_root_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + p = tmp_path / "list.json" + p.write_text("[1, 2, 3]", encoding="utf-8") + with pytest.raises(ValueError, match="object"): + load_spec_file("list.json") + + +@pytest.mark.skipif(sys.platform == "win32", reason="symlink ACL on Windows CI") +def test_load_spec_file_symlink_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.json" + real.write_text("{}", encoding="utf-8") + link = tmp_path / "link.json" + try: + os.symlink(real, link) + except (OSError, NotImplementedError): + pytest.skip("symlink unavailable") + with pytest.raises(ValueError, match="symlink"): + load_spec_file("link.json") + + +def test_write_dataset_under_cwd(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + endpoints, _ = parse_openapi(_OPENAPI_SAMPLE) + rows = synthesise_dataset(endpoints, 1) + out = write_dataset(rows, "dataset.jsonl") + assert os.path.exists(out) + # Each line valid JSON + with open(out, encoding="utf-8") as fh: + for line in fh: + data = json.loads(line) + assert "messages" in data + assert "tool" in data + + +def test_write_dataset_outside_cwd_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + rows = [SynthRow(messages=({"role": "u", "content": "x"},), tool="t", + source_endpoint="/")] + abs_outside = str(tmp_path.parent / "evil.jsonl") + with pytest.raises(ValueError, match="must stay under cwd"): + write_dataset(rows, abs_outside) + + +def test_write_dataset_null_byte_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + write_dataset([], "x\x00.jsonl") + + +def test_write_dataset_non_string_path(): + with pytest.raises(TypeError): + write_dataset([], 123) # type: ignore[arg-type] + + +def test_write_dataset_invalid_row_type(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(TypeError): + write_dataset([{"not": "a SynthRow"}], "out.jsonl") # type: ignore[list-item] + + +# --------------------------------------------------------------------------- +# CLI smoke tests +# --------------------------------------------------------------------------- + + +def test_cli_agent_synth_smoke(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "spec.json").write_text( + json.dumps(_OPENAPI_SAMPLE), encoding="utf-8" + ) + result = runner.invoke( + agent.app, ["synth", "--spec", "spec.json", "--output", "ds.jsonl"] + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "ds.jsonl").exists() + assert "listPets" in result.output + + +def test_cli_agent_synth_unknown_spec_path(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + result = runner.invoke(agent.app, ["synth", "--spec", "missing.json"]) + assert result.exit_code == 1, result.output + + +def test_cli_agent_synth_outside_cwd_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + abs_outside = str(tmp_path.parent / "evil.json") + result = runner.invoke(agent.app, ["synth", "--spec", abs_outside]) + assert result.exit_code == 1, result.output + + +def test_cli_agent_train_smoke(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "spec.json").write_text( + json.dumps(_OPENAPI_SAMPLE), encoding="utf-8" + ) + result = runner.invoke( + agent.app, + ["train", "--spec", "spec.json", "--base", "meta-llama/Llama-3.2-1B", + "--dataset-out", "ds.jsonl"], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "ds.jsonl").exists() + assert "Planned" in result.output or "agent_train.yaml" in result.output + + +def test_cli_agent_eval_smoke(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "spec.json").write_text( + json.dumps(_OPENAPI_SAMPLE), encoding="utf-8" + ) + preds = [ + {"tool": "listPets", "arguments": {"limit": "10"}}, + {"tool": "listPets", "arguments": {"unknownParam": "x"}}, + {"tool": "nopeTool", "arguments": {}}, + ] + (tmp_path / "preds.jsonl").write_text( + "\n".join(json.dumps(p) for p in preds) + "\n", encoding="utf-8" + ) + result = runner.invoke( + agent.app, + ["eval", "--spec", "spec.json", "--predictions", "preds.jsonl"], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + # 2/3 tools match (listPets x2), 1/3 args fully valid + assert "Tool match" in result.output + assert "Args valid" in result.output + + +def test_cli_agent_eval_outside_cwd_predictions(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "spec.json").write_text( + json.dumps(_OPENAPI_SAMPLE), encoding="utf-8" + ) + abs_outside = str(tmp_path.parent / "preds.jsonl") + result = runner.invoke( + agent.app, + ["eval", "--spec", "spec.json", "--predictions", abs_outside], + ) + assert result.exit_code == 1, result.output + + +def test_cli_agent_synth_help(): + from soup_cli.commands import agent + + result = runner.invoke(agent.app, ["synth", "--help"]) + assert result.exit_code == 0 + + +def test_cli_agent_help(): + from soup_cli.commands import agent + + result = runner.invoke(agent.app, ["--help"]) + assert result.exit_code == 0 + assert "synth" in result.output + assert "train" in result.output + assert "eval" in result.output + + +def test_cli_agent_train_help(): + from soup_cli.commands import agent + + result = runner.invoke(agent.app, ["train", "--help"]) + assert result.exit_code == 0 + + +def test_cli_agent_eval_help(): + from soup_cli.commands import agent + + result = runner.invoke(agent.app, ["eval", "--help"]) + assert result.exit_code == 0 + + +def test_cli_agent_eval_missing_spec(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "preds.jsonl").write_text("{}\n", encoding="utf-8") + result = runner.invoke( + agent.app, ["eval", "--spec", "missing.json", "--predictions", "preds.jsonl"], + ) + assert result.exit_code == 1, result.output + + +def test_cli_agent_eval_outside_cwd_spec(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "preds.jsonl").write_text("{}\n", encoding="utf-8") + abs_outside = str(tmp_path.parent / "evil.json") + result = runner.invoke( + agent.app, + ["eval", "--spec", abs_outside, "--predictions", "preds.jsonl"], + ) + assert result.exit_code == 1, result.output + + +def test_cli_agent_train_rejects_newline_in_base(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "spec.json").write_text( + json.dumps(_OPENAPI_SAMPLE), encoding="utf-8" + ) + result = runner.invoke( + agent.app, + ["train", "--spec", "spec.json", + "--base", "evil\ntraining: { epochs: 9999 }"], + ) + assert result.exit_code == 2, result.output + assert "newline" in result.output.lower() or "base" in result.output.lower() + + +def test_cli_agent_train_rejects_null_byte_in_base(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "spec.json").write_text( + json.dumps(_OPENAPI_SAMPLE), encoding="utf-8" + ) + result = runner.invoke( + agent.app, + ["train", "--spec", "spec.json", "--base", "evil\x00"], + ) + assert result.exit_code == 2, result.output + + +def test_write_dataset_partial_failure_no_partial_file(tmp_path, monkeypatch): + """Mid-stream TypeError must not leave a partial file at target.""" + monkeypatch.chdir(tmp_path) + rows = [ + SynthRow(messages=(), tool="t1", source_endpoint="/"), + "not a SynthRow", # type: ignore[list-item] + ] + with pytest.raises(TypeError): + write_dataset(rows, "out.jsonl") # type: ignore[arg-type] + # Atomic write: target file should NOT exist after partial failure + assert not (tmp_path / "out.jsonl").exists() + + +@pytest.mark.skipif(sys.platform == "win32", reason="symlink ACL on Windows") +def test_write_dataset_symlink_target_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.jsonl" + real.write_text("", encoding="utf-8") + link = tmp_path / "link.jsonl" + try: + os.symlink(real, link) + except (OSError, NotImplementedError): + pytest.skip("symlink unavailable") + rows = [SynthRow(messages=(), tool="t", source_endpoint="/")] + with pytest.raises(ValueError, match="symlink"): + write_dataset(rows, "link.jsonl") + + +def test_cli_agent_synth_no_endpoints_exits_1(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + from soup_cli.commands import agent + + (tmp_path / "empty.json").write_text('{"openapi": "3.0.0", "paths": {}}', + encoding="utf-8") + result = runner.invoke(agent.app, ["synth", "--spec", "empty.json"]) + assert result.exit_code == 1 + + +# --------------------------------------------------------------------------- +# Endpoint dataclass invariants +# --------------------------------------------------------------------------- + + +def test_endpoint_is_frozen(): + import dataclasses + + ep = Endpoint( + tool="x", method="get", path="/x", + description="", parameters=(), spec_kind="openapi", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + ep.tool = "y" # type: ignore[misc] + + +def test_synth_row_is_frozen(): + import dataclasses + + row = SynthRow(messages=(), tool="x", source_endpoint="/") + with pytest.raises(dataclasses.FrozenInstanceError): + row.tool = "y" # type: ignore[misc] + + +def test_spec_report_is_frozen_explicit(): + import dataclasses + + _, report = parse_spec(_OPENAPI_SAMPLE) + with pytest.raises(dataclasses.FrozenInstanceError): + report.spec_kind = "x" # type: ignore[misc]