mirror of https://github.com/razor-ai/soup.git
feat(agent): Agent Forge — OpenAPI/MCP/GraphQL spec → tool-calling SFT dataset (v0.46.0 Part B)
Bumps to v0.46.0 and ships the Agent Forge: parse OpenAPI 3.x, MCP server
manifests, or GraphQL introspection JSON straight into a tool-calling SFT
dataset where each row is `{messages: [user, assistant_with_tool_call],
tool, source_endpoint}`. No more hand-rolled jsonl scaffolding for
function-calling fine-tunes.
* soup_cli/utils/agent_forge.py — `Endpoint` / `SynthRow` / `SpecReport`
frozen dataclasses. `parse_openapi` / `parse_mcp` / `parse_graphql`
parsers leave `$ref` strings opaque (no external resolution — defends
against file-read SSRF). All synthesised path strings routed through
`_validate_path` (rejects newline-in-name across all three parsers).
`load_spec_file`: `is_under_cwd` + `os.lstat + S_ISLNK` BEFORE realpath
(corrects v0.46.0 first-cut ordering caught by security review) + 5MiB
cap + yaml.safe_load only. `write_dataset` atomic via mkstemp +
os.replace (mid-stream TypeError never leaves partial file; mirrors
v0.43.0 Part D `copy_bundle_to` policy) + symlink rejection at target.
Caps: `_MAX_ENDPOINTS=10_000`, `_MAX_SPEC_BYTES=5MiB`,
`_MAX_ROWS_PER_ENDPOINT=32`.
* soup_cli/commands/agent.py — `soup agent synth/train/eval` Typer
subcommands. `synth` table cells pass through `rich.markup.escape`.
`train` rejects NUL/newline/oversize in `--base` and `--output-dir`
BEFORE embedding into rendered YAML recipe (CRITICAL security fix —
defends against YAML key injection where `--base $'evil\ntraining:
{ epochs: 9999 }'` would smuggle in injected training keys). `eval`
enforces predictions `is_under_cwd` + symlink rejection +
`_MAX_PRED_LINES=1_000_000` DoS cap.
* soup_cli/cli.py — registers `agent` Typer group; help string uses
ASCII-safe `->` (`test_help_output_is_ascii_safe` regression test caught
a Unicode `→` on first try).
* tests/test_v0460_part_b.py — 71 tests covering every parser kind,
failure modes (cycle / cap / null-byte / oversize / outside-cwd /
symlink), atomic-write partial-failure invariant, every CLI surface.
* Docs: README ## What's New replaced + dedicated `## Deploy Autopilot`
and `## Agent Forge` sections added; SECURITY.md supported-window
shifted (v0.46→full, v0.41→drop) + v0.46.0 fix-notes entry;
CONTRIBUTING.md test counts 165→167 / 5989→6126.
Test suite: 5989 → 6126 (+137 net new) green on Windows.
Known limitations (live runtime deferred to v0.46.1):
- Quant-Lobotomy auto-measure for deploy autopilot
- RLVR `code_exec` sandbox scoring in `agent eval`
- In-process `soup train` re-entry in `agent train` (Typer commands aren't
safe to re-enter — matches v0.44.0 `soup quantize` design)
- ExecuTorch packaging for iphone-16 / pixel-9 (lands in v0.54.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
479931282c
commit
986a4c00e3
|
|
@ -111,7 +111,7 @@ soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (165 files, 5989 tests)
|
||||
tests/ - Test suite (167 files, 6126 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
50
README.md
50
README.md
|
|
@ -43,15 +43,14 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.45.0 — Plugin System & Ecosystem Wins**: A public plugin API and the schema scaffolding for 20+ ecosystem integrations. Soup is now extensible.
|
||||
**v0.46.0 — Deploy & Agent Autopilot**: Two zero-config autopilots — a deploy target picker that maps your hardware to PEFT+quant+spec-decoding in one command, and an Agent Forge that turns OpenAPI / MCP / GraphQL specs straight into tool-calling SFT datasets.
|
||||
|
||||
- **Plugin / hook system.** `soup_cli.plugins.register_plugin(name, version, plugin, templates=[], model_groups=[])` lets third-party Python modules ship their own pre-train / post-train / pre-step / post-step hooks plus chat templates and model groups. Registry is idempotent on identical specs and rejects conflicting re-registration. Drop a file under `soup_cli/plugins/` and the loader picks it up at startup. New `soup plugins list / install / enable / disable` CLI.
|
||||
- **Anthropic Messages API converter.** `to_anthropic` / `from_anthropic` translate between OpenAI chat-completions and Anthropic Messages payloads — multiple `system` messages joined with `\n\n`, `tool` role surfaces as `tool_result` content blocks (list content concatenated, never silently dropped), `max_tokens` capped at 16384, `temperature` bounded `[0.0, 2.0]`. Live `/v1/messages` endpoint deferred to v0.45.1.
|
||||
- **Server-side tools allowlist.** Closed `{python, bash, web_search}` set with `WebSearchConfig` — domain allowlist (leading-dot subdomain pattern, port-strip on the host, IPv6 literal deny), `rate_limit_per_minute ∈ [1, 600]`. `python` and `bash` reuse the v0.25.0 RLVR sandbox; live HTTP endpoints deferred to v0.45.1.
|
||||
- **External integrations catalog.** 15-entry frozen `MappingProxyType` of ecosystem targets — `lm-studio`, `comfyui`, `stable-diffusion-cpp`, `open-webui`, `ollama`, `tei`, `pgvector`, `faiss`, `weaviate`, `sentence-transformers`, `claude-code`, `cursor`, `continue`, `cline`, `sillytavern`. Each entry names the artifact format (`gguf` / `safetensors` / `served-endpoint`).
|
||||
- **Advanced trainer-plugin allowlist.** 6-entry catalog (`grokfast` / `spectrum` / `llmcompressor` / `sonicmoe` / `cce_plugin` / `math_verify`) with `validate_trainer_plugin_list` (dedup, allowlist match, `_MAX_PLUGINS_PER_RUN=8`). Live callbacks land in v0.45.1.
|
||||
- **`soup data recipe <recipe.yaml>`.** Validates a Seed → LLM Text → Code → Judge → Validator → Sampler graph DAG: closed node-kind allowlist, Kahn's topological sort with `collections.deque` (deterministic, O(N+E)), self-loop / cycle / dangling-edge / duplicate rejection, cwd containment + `os.lstat + S_ISLNK` symlink rejection on the recipe file. Live offline runner against a local model in v0.45.1.
|
||||
- **+169 net new tests** — covers all 5 release Parts: plugin idempotency + per-list caps + description conflict rejection, Anthropic converter happy + failure paths, domain allowlist port-strip + IPv6 deny, n-gram bounds, integrations catalog immutability, trainer-plugin canonicalisation + dedup, recipe DAG cycle / cap / symlink + CLI happy / invalid / missing.
|
||||
- **`soup deploy autopilot --target <profile>`.** 10 hardware profiles (`mac-m3`, `mac-m4-pro`, `rtx-3060-12gb`, `rtx-4090-24gb`, `iphone-16`, `pixel-9`, `ollama-local`, `lm-studio`, `runpod-a100`, `hf-jobs-h100`) each mapped to a runtime, quantisation, PEFT method, and speculative-decoding flag. Writes a ready-to-train `soup.yaml` recipe AND a planned deploy shell script. Closed allowlists on every field; `shlex.quote` on the model path in the generated bash. Live Quant-Lobotomy auto-measure deferred to v0.46.1.
|
||||
- **`soup agent synth --spec api.yaml`.** Parses OpenAPI 3.x, MCP server manifests, or GraphQL introspection JSON into a canonical endpoint list and synthesises a tool-calling SFT dataset (`{messages, tool, source_endpoint}`). `$ref` strings are left opaque (no external resolution — defends against file-read SSRF), `yaml.safe_load` only, 5 MiB spec cap, 10 000-endpoint cap. Atomic JSONL write via staged-tempfile + `os.replace` — mid-stream failure never leaves a partial dataset.
|
||||
- **`soup agent train --spec api.yaml --base <model>`.** One-shot wrapper that runs synth, then prints the planned `soup train` invocation with the rendered recipe (in-process re-entry of Typer commands is intentionally not done — matches `soup quantize` design). `--base` and `--output-dir` are validated for NUL / newline / oversize BEFORE embedding in the recipe YAML (defends against YAML key injection).
|
||||
- **`soup agent eval --spec api.yaml --predictions preds.jsonl`.** Scores predicted tool-calls against the spec catalog: tool-name match + arguments-key validity. Predictions path enforces cwd containment, `os.lstat + S_ISLNK` symlink rejection, and a 1 000 000-line DoS cap. Live RLVR `code_exec` sandbox scoring deferred to v0.46.1.
|
||||
- **Security throughout.** Path containment (`is_under_cwd`), symlink TOCTOU rejection (`os.lstat + S_ISLNK`) on every write target, Rich markup escape on every spec-derived string, bool-rejected-before-int on every numeric param, closed allowlists on runtime / quant / PEFT / spec kind / node kind.
|
||||
- **+137 net new tests** — every parser kind (OpenAPI / MCP / GraphQL), every failure mode (cycle / cap / null-byte / oversize / outside-cwd / symlink), every CLI surface (`autopilot --list / --help / happy / outside-cwd reject`, `agent synth/train/eval` happy + failure).
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -2754,6 +2753,11 @@ soup deploy ollama --model m.gguf --name x Deploy GGUF to Ollama
|
|||
soup deploy ollama --list List Soup-deployed models
|
||||
soup deploy ollama --remove <name> Remove model from Ollama
|
||||
soup deploy hf-space --model user/m --space user/s --template gradio-chat|streamlit-chat Create HF Space
|
||||
soup deploy autopilot --target mac-m3|rtx-4090-24gb|... Pick PEFT+quant+spec-decoding for a hardware target
|
||||
soup deploy autopilot --list List all 10 deploy profiles
|
||||
soup agent synth --spec api.yaml -o ds.jsonl Parse OpenAPI/MCP/GraphQL spec into a tool-calling SFT dataset
|
||||
soup agent train --spec api.yaml --base model One-shot synth + planned soup train invocation
|
||||
soup agent eval --spec api.yaml --predictions p.jsonl Score predicted tool-calls vs spec catalog
|
||||
soup eval benchmark --model ./output Evaluate on standard benchmarks
|
||||
soup eval custom --tasks eval.jsonl Custom eval tasks from JSONL
|
||||
soup eval judge --target resp.jsonl LLM-as-a-judge evaluation
|
||||
|
|
@ -3161,6 +3165,36 @@ params:
|
|||
|
||||
Strict scalar allowlist on values (`str` / `int` / `float` / `bool`); `_MAX_FILE_BYTES=256KB`, `_MAX_PARAM_KEYS=32`, `_MAX_VALUES_PER_KEY=64`; `SweepSpec.params` is `MappingProxyType[str, Tuple[Any, ...]]` for genuine immutability.
|
||||
|
||||
## Deploy Autopilot
|
||||
|
||||
Pick the optimal PEFT + quantisation + speculative-decoding combo for your hardware target in one command:
|
||||
|
||||
```bash
|
||||
soup deploy autopilot --target rtx-4090-24gb --base meta-llama/Llama-3.2-1B
|
||||
# Writes:
|
||||
# deploy_autopilot.yaml — ready-to-train soup.yaml recipe
|
||||
# deploy_autopilot.sh — planned deploy shell script
|
||||
```
|
||||
|
||||
Profiles ship out of the box for Apple Silicon (`mac-m3`, `mac-m4-pro`), consumer NVIDIA (`rtx-3060-12gb`, `rtx-4090-24gb`), mobile (`iphone-16`, `pixel-9`), local runtimes (`ollama-local`, `lm-studio`), and cloud (`runpod-a100`, `hf-jobs-h100`). `--list` shows the full table. Every profile is a frozen dataclass with closed allowlists on runtime / quant / PEFT — bad config values fail at import time. The generated bash uses `shlex.quote` on the model path and writes are protected by cwd containment + `os.lstat + S_ISLNK` TOCTOU rejection.
|
||||
|
||||
## Agent Forge
|
||||
|
||||
Turn an OpenAPI 3.x, MCP server manifest, or GraphQL introspection JSON straight into a tool-calling SFT dataset — no manual labelling, no scaffolding:
|
||||
|
||||
```bash
|
||||
# 1. Parse spec + synthesise a tool-calling dataset
|
||||
soup agent synth --spec api.yaml --output ds.jsonl --examples-per-endpoint 4
|
||||
|
||||
# 2. Plan the training run (prints the soup train invocation)
|
||||
soup agent train --spec api.yaml --base meta-llama/Llama-3.2-1B
|
||||
|
||||
# 3. Score model predictions against the spec catalog
|
||||
soup agent eval --spec api.yaml --predictions preds.jsonl
|
||||
```
|
||||
|
||||
Each row of the synthesised dataset is `{messages: [user, assistant_with_tool_call], tool: <name>, source_endpoint: <path>}`. `$ref` strings in OpenAPI are left opaque (no external resolution — defends against file-read SSRF), `yaml.safe_load` only, 5 MiB spec cap, 10 000-endpoint cap, atomic JSONL write via staged-tempfile + `os.replace`. `eval` enforces a 1 000 000-line cap on predictions and rejects symlinks at every read/write boundary.
|
||||
|
||||
## Plugin System
|
||||
|
||||
Drop a Python module under `soup_cli/plugins/` (or any package importable by Soup) and register at import time:
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.45.0"
|
||||
version = "0.46.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.45.0"
|
||||
__version__ = "0.46.0"
|
||||
|
|
|
|||
|
|
@ -157,6 +157,15 @@ app.add_typer(
|
|||
help="List, enable, disable Soup plugins (v0.45.0).",
|
||||
)
|
||||
|
||||
# v0.46.0 Part B — Agent Forge.
|
||||
from soup_cli.commands import agent as agent_cmd # noqa: E402
|
||||
|
||||
app.add_typer(
|
||||
agent_cmd.app,
|
||||
name="agent",
|
||||
help="Agent Forge: spec -> tool-calling dataset / train / eval (v0.46.0).",
|
||||
)
|
||||
|
||||
# Register data generate as a subcommand of data
|
||||
data.app.command(name="generate")(generate.generate)
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,304 @@
|
|||
"""soup agent — Agent Forge: spec → tool-calling SFT dataset / train / eval.
|
||||
|
||||
v0.46.0 Part B. Live ``train`` and ``eval`` wrappers print the planned
|
||||
sub-command rather than re-entering the Typer app in-process (matches the
|
||||
``soup quantize`` design from v0.44.0 Part D).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shlex
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
|
||||
console = Console()
|
||||
|
||||
app = typer.Typer(no_args_is_help=True)
|
||||
|
||||
|
||||
@app.command()
|
||||
def synth(
|
||||
spec: str = typer.Option(
|
||||
..., "--spec", "-s",
|
||||
help="Path to OpenAPI / MCP / GraphQL spec (YAML or JSON, under cwd).",
|
||||
),
|
||||
output: str = typer.Option(
|
||||
"agent_dataset.jsonl", "--output", "-o",
|
||||
help="Where to write the synthesised JSONL dataset (under cwd).",
|
||||
),
|
||||
kind: Optional[str] = typer.Option(
|
||||
None, "--kind", "-k",
|
||||
help="Spec kind override: openapi | mcp | graphql. Auto-detected if omitted.",
|
||||
),
|
||||
examples_per_endpoint: int = typer.Option(
|
||||
1, "--examples-per-endpoint", "-n", min=1, max=32,
|
||||
help="Number of synthetic rows to emit per endpoint.",
|
||||
),
|
||||
):
|
||||
"""Parse an API spec and synthesise a tool-calling SFT dataset."""
|
||||
from soup_cli.utils.agent_forge import (
|
||||
load_spec_file,
|
||||
parse_spec,
|
||||
synthesise_dataset,
|
||||
write_dataset,
|
||||
)
|
||||
|
||||
try:
|
||||
spec_dict = load_spec_file(spec)
|
||||
except (ValueError, FileNotFoundError, TypeError) as exc:
|
||||
console.print(f"[red]Spec load failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
except Exception as exc: # noqa: BLE001 — yaml/json parse errors
|
||||
console.print(f"[red]Spec parse failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
try:
|
||||
endpoints, report = parse_spec(spec_dict, kind=kind)
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]Parse failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
if not endpoints:
|
||||
console.print(
|
||||
"[yellow]No endpoints discovered.[/] "
|
||||
f"Detected kind: [bold]{escape(report.spec_kind)}[/]."
|
||||
)
|
||||
for w in report.warnings[:5]:
|
||||
console.print(f" [dim]- {escape(w)}[/]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
rows = synthesise_dataset(endpoints, examples_per_endpoint=examples_per_endpoint)
|
||||
try:
|
||||
out_path = write_dataset(rows, output)
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]Write failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
table = Table(title=f"Agent Forge — {escape(report.spec_kind)}")
|
||||
table.add_column("Tool", style="bold cyan")
|
||||
table.add_column("Method", style="magenta")
|
||||
table.add_column("Path")
|
||||
# ep.path is partly user-controlled (from the spec) — escape every cell
|
||||
# before handing to Rich Table (matches v0.43.0 Part B Tournament policy).
|
||||
for ep in endpoints[:20]:
|
||||
table.add_row(escape(ep.tool), escape(ep.method), escape(ep.path))
|
||||
console.print(table)
|
||||
if len(endpoints) > 20:
|
||||
console.print(f"[dim]... and {len(endpoints) - 20} more[/]")
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Spec kind: [bold]{escape(report.spec_kind)}[/]\n"
|
||||
f"Endpoints: [bold]{report.endpoint_count}[/] "
|
||||
f"(skipped duplicates: {report.skipped})\n"
|
||||
f"Rows: [bold]{len(rows)}[/]\n"
|
||||
f"Output: [bold]{escape(out_path)}[/]",
|
||||
title="[bold green]Agent Forge — synth complete[/]",
|
||||
)
|
||||
)
|
||||
for w in report.warnings[:5]:
|
||||
console.print(f"[yellow]warning:[/] {escape(w)}")
|
||||
|
||||
|
||||
@app.command()
|
||||
def train(
|
||||
spec: str = typer.Option(..., "--spec", "-s", help="API spec (under cwd)."),
|
||||
base: str = typer.Option(
|
||||
..., "--base", "-b",
|
||||
help="Base model HF repo id to fine-tune.",
|
||||
),
|
||||
dataset_out: str = typer.Option(
|
||||
"agent_dataset.jsonl", "--dataset-out",
|
||||
help="Where the synth step writes its dataset (under cwd).",
|
||||
),
|
||||
output_dir: str = typer.Option(
|
||||
"./agent_train_output", "--output-dir",
|
||||
help="Where the planned soup train run will store checkpoints.",
|
||||
),
|
||||
examples_per_endpoint: int = typer.Option(
|
||||
4, "--examples-per-endpoint", "-n", min=1, max=32,
|
||||
),
|
||||
):
|
||||
"""One-shot wrapper: synth + planned soup train invocation (printed)."""
|
||||
from soup_cli.utils.agent_forge import (
|
||||
load_spec_file,
|
||||
parse_spec,
|
||||
synthesise_dataset,
|
||||
write_dataset,
|
||||
)
|
||||
|
||||
# CRITICAL security fix: reject newline/NUL/oversize in --base and
|
||||
# --output-dir BEFORE building the recipe YAML string. A crafted
|
||||
# --base "evil\ntraining:\n epochs: 9999" would inject YAML keys
|
||||
# into the rendered recipe.
|
||||
for label, value in (("--base", base), ("--output-dir", output_dir)):
|
||||
if not isinstance(value, str) or not value:
|
||||
console.print(f"[red]{label} must be a non-empty string[/]")
|
||||
raise typer.Exit(2)
|
||||
if "\x00" in value or "\n" in value or "\r" in value:
|
||||
console.print(f"[red]{label} contains NUL or newline[/]")
|
||||
raise typer.Exit(2)
|
||||
if len(value) > 4096:
|
||||
console.print(f"[red]{label} exceeds 4096 chars[/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
try:
|
||||
spec_dict = load_spec_file(spec)
|
||||
endpoints, report = parse_spec(spec_dict)
|
||||
except (ValueError, TypeError, FileNotFoundError) as exc:
|
||||
console.print(f"[red]Spec error:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
if not endpoints:
|
||||
console.print("[red]No endpoints discovered; aborting train.[/]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
rows = synthesise_dataset(endpoints, examples_per_endpoint=examples_per_endpoint)
|
||||
try:
|
||||
ds_path = write_dataset(rows, dataset_out)
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(f"[red]Dataset write failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
planned_cmd = " ".join(
|
||||
shlex.quote(p) for p in [
|
||||
"soup", "train",
|
||||
"--config", "agent_train.yaml",
|
||||
"--yes",
|
||||
]
|
||||
)
|
||||
recipe_yaml = (
|
||||
f"base: {base}\n"
|
||||
"task: sft\n"
|
||||
"data:\n"
|
||||
f" train: {ds_path}\n"
|
||||
" format: tool-calling\n"
|
||||
"training:\n"
|
||||
" epochs: 3\n"
|
||||
" lr: 2.0e-5\n"
|
||||
" batch_size: auto\n"
|
||||
f"output: {output_dir}\n"
|
||||
)
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Spec: [bold]{escape(spec)}[/]\n"
|
||||
f"Endpoints: [bold]{report.endpoint_count}[/]\n"
|
||||
f"Dataset: [bold]{escape(ds_path)}[/]\n"
|
||||
f"Base: [bold]{escape(base)}[/]\n"
|
||||
f"Output: [bold]{escape(output_dir)}[/]\n\n"
|
||||
f"[bold]Planned recipe (agent_train.yaml):[/]\n{escape(recipe_yaml)}\n"
|
||||
f"[bold]Run:[/] {escape(planned_cmd)}",
|
||||
title="[bold green]Agent Forge — train plan[/]",
|
||||
)
|
||||
)
|
||||
console.print(
|
||||
"[yellow]Note:[/] live in-process training is intentionally not "
|
||||
"re-entered (Typer commands aren't safe to re-enter); copy the recipe "
|
||||
"into agent_train.yaml and run the command above."
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def eval(
|
||||
spec: str = typer.Option(..., "--spec", "-s", help="API spec (under cwd)."),
|
||||
predictions: str = typer.Option(
|
||||
...,
|
||||
"--predictions",
|
||||
"-p",
|
||||
help=(
|
||||
"JSONL of model outputs, one per line, with at minimum "
|
||||
"{tool: <name>, arguments: {...}}."
|
||||
),
|
||||
),
|
||||
):
|
||||
"""Score predicted tool-calls against the spec's tool catalog.
|
||||
|
||||
Each prediction row gets two checks:
|
||||
1. ``tool`` matches a known endpoint in the spec.
|
||||
2. ``arguments`` only references parameters declared on that endpoint.
|
||||
"""
|
||||
import os
|
||||
import stat as _stat
|
||||
|
||||
from soup_cli.utils.agent_forge import load_spec_file, parse_spec
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
max_pred_lines = 1_000_000
|
||||
|
||||
if not isinstance(predictions, str) or not predictions or "\x00" in predictions:
|
||||
console.print("[red]predictions path must be non-empty NUL-free string[/]")
|
||||
raise typer.Exit(1)
|
||||
if not is_under_cwd(predictions):
|
||||
console.print("[red]predictions path must stay under cwd[/]")
|
||||
raise typer.Exit(1)
|
||||
# Symlink TOCTOU defence (mirrors load_spec_file policy).
|
||||
try:
|
||||
if _stat.S_ISLNK(os.lstat(predictions).st_mode):
|
||||
console.print("[red]predictions path must not be a symlink[/]")
|
||||
raise typer.Exit(1)
|
||||
except FileNotFoundError:
|
||||
console.print("[red]predictions file not found[/]")
|
||||
raise typer.Exit(1) from None
|
||||
|
||||
try:
|
||||
spec_dict = load_spec_file(spec)
|
||||
endpoints, _ = parse_spec(spec_dict)
|
||||
except (ValueError, TypeError, FileNotFoundError) as exc:
|
||||
console.print(f"[red]Spec error:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
tool_to_params = {ep.tool: set(ep.parameters) for ep in endpoints}
|
||||
|
||||
total = 0
|
||||
tool_ok = 0
|
||||
args_ok = 0
|
||||
try:
|
||||
with open(predictions, "r", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
if total >= max_pred_lines:
|
||||
break
|
||||
total += 1
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
tool = row.get("tool")
|
||||
if isinstance(tool, str) and tool in tool_to_params:
|
||||
tool_ok += 1
|
||||
args = row.get("arguments") or {}
|
||||
if isinstance(args, dict):
|
||||
invalid = [
|
||||
k for k in args.keys()
|
||||
if k not in tool_to_params[tool]
|
||||
]
|
||||
if not invalid:
|
||||
args_ok += 1
|
||||
except OSError as exc:
|
||||
console.print(f"[red]Predictions read failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(1) from exc
|
||||
|
||||
if total == 0:
|
||||
console.print("[yellow]No predictions to score.[/]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
tool_pct = 100.0 * tool_ok / total
|
||||
args_pct = 100.0 * args_ok / total
|
||||
console.print(
|
||||
Panel(
|
||||
f"Predictions: [bold]{total}[/]\n"
|
||||
f"Tool match: [bold]{tool_ok}[/] ({tool_pct:.1f}%)\n"
|
||||
f"Args valid: [bold]{args_ok}[/] ({args_pct:.1f}%)",
|
||||
title="[bold green]Agent Forge — eval[/]",
|
||||
)
|
||||
)
|
||||
|
|
@ -0,0 +1,576 @@
|
|||
"""v0.46.0 Part B — Agent Forge: spec → tool-calling SFT dataset.
|
||||
|
||||
Parses OpenAPI 3.x, MCP server manifests, and GraphQL introspection JSON
|
||||
into a canonical ``Endpoint`` shape, then synthesises a tool-calling SFT
|
||||
dataset where each row is ``{messages: [user, assistant{tool_calls}],
|
||||
tool: <name>, source_endpoint: <path>}``.
|
||||
|
||||
The parser surface is intentionally parser-only — no network code, no
|
||||
``$ref`` resolution that would let a crafted spec read arbitrary files.
|
||||
``$ref`` strings are left as opaque markers and a warning is surfaced;
|
||||
operators wanting full resolution should run ``openapi-spec-validator``
|
||||
upstream and feed the bundled JSON in.
|
||||
|
||||
Live ``soup agent train`` orchestrator + ``soup agent eval`` sandbox
|
||||
scoring re-use the v0.25.0 RLVR ``code_exec`` sandbox; this module ships
|
||||
the parse-and-synth layer.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
|
||||
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
|
||||
_TOOL_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_.\-]{0,127}$")
|
||||
_MAX_ENDPOINTS = 10_000
|
||||
_MAX_SPEC_BYTES = 5 * 1024 * 1024 # 5 MiB
|
||||
_MAX_DESCRIPTION = 512
|
||||
_MAX_ROWS_PER_ENDPOINT = 32
|
||||
_ALLOWED_SPEC_KINDS = frozenset({"openapi", "mcp", "graphql"})
|
||||
_HTTP_METHODS = frozenset(
|
||||
{"get", "post", "put", "patch", "delete", "head", "options", "trace"}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Endpoint:
|
||||
"""Canonical endpoint representation across all three spec kinds."""
|
||||
|
||||
tool: str
|
||||
method: str
|
||||
path: str
|
||||
description: str
|
||||
parameters: Tuple[str, ...] # parameter names only — schema details opaque
|
||||
spec_kind: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SynthRow:
|
||||
"""One row in the generated tool-calling SFT dataset."""
|
||||
|
||||
messages: Tuple[Mapping[str, Any], ...]
|
||||
tool: str
|
||||
source_endpoint: str
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"messages": [dict(m) for m in self.messages],
|
||||
"tool": self.tool,
|
||||
"source_endpoint": self.source_endpoint,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SpecReport:
|
||||
"""Summary of a parsed spec for the CLI."""
|
||||
|
||||
spec_kind: str
|
||||
endpoint_count: int
|
||||
skipped: int
|
||||
warnings: Tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _validate_tool_name(name: str) -> str:
|
||||
if not isinstance(name, str):
|
||||
raise TypeError("tool name must be a string")
|
||||
if not _TOOL_NAME_RE.match(name):
|
||||
raise ValueError(
|
||||
"tool name must match ^[A-Za-z_][A-Za-z0-9_.-]{0,127}$"
|
||||
)
|
||||
return name
|
||||
|
||||
|
||||
def _validate_method(method: str) -> str:
|
||||
if not isinstance(method, str):
|
||||
raise TypeError("method must be a string")
|
||||
canonical = method.strip().lower()
|
||||
if canonical not in _HTTP_METHODS:
|
||||
raise ValueError(f"unknown HTTP method: {method!r}")
|
||||
return canonical
|
||||
|
||||
|
||||
def _validate_path(path: str) -> str:
|
||||
if not isinstance(path, str):
|
||||
raise TypeError("path must be a string")
|
||||
if not path or "\x00" in path or "\n" in path or "\r" in path:
|
||||
raise ValueError("path must be non-empty single-line NUL-free string")
|
||||
if len(path) > 1024:
|
||||
raise ValueError("path exceeds 1024 chars")
|
||||
return path
|
||||
|
||||
|
||||
def _truncate_desc(desc: Any) -> str:
|
||||
if desc is None:
|
||||
return ""
|
||||
if not isinstance(desc, str):
|
||||
return ""
|
||||
if "\x00" in desc:
|
||||
desc = desc.replace("\x00", "")
|
||||
desc = desc.strip()
|
||||
if len(desc) > _MAX_DESCRIPTION:
|
||||
return desc[: _MAX_DESCRIPTION - 3] + "..."
|
||||
return desc
|
||||
|
||||
|
||||
def _sanitise_tool_id(*parts: str) -> str:
|
||||
"""Build a tool name from spec parts, replacing non-id chars with '_'."""
|
||||
raw = "_".join(p for p in parts if p)
|
||||
raw = re.sub(r"[^A-Za-z0-9_.\-]", "_", raw)
|
||||
if not raw or not re.match(r"^[A-Za-z_]", raw):
|
||||
raw = "tool_" + raw
|
||||
return raw[:128]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OpenAPI parser
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_openapi(spec: Mapping[str, Any]) -> Tuple[List[Endpoint], List[str]]:
|
||||
"""Parse an OpenAPI 3.x ``dict``. Returns (endpoints, warnings)."""
|
||||
if not isinstance(spec, dict):
|
||||
raise TypeError("openapi spec must be a dict")
|
||||
version = spec.get("openapi", "")
|
||||
warnings: List[str] = []
|
||||
if not isinstance(version, str) or not version.startswith("3."):
|
||||
warnings.append(
|
||||
f"unrecognised openapi version: {version!r}; parser is OpenAPI 3.x"
|
||||
)
|
||||
paths = spec.get("paths")
|
||||
if not isinstance(paths, dict):
|
||||
return [], ["spec has no 'paths' object"]
|
||||
endpoints: List[Endpoint] = []
|
||||
for path, ops in paths.items():
|
||||
if not isinstance(path, str) or not isinstance(ops, dict):
|
||||
continue
|
||||
for method, op in ops.items():
|
||||
if not isinstance(method, str):
|
||||
continue
|
||||
lower = method.lower()
|
||||
if lower not in _HTTP_METHODS:
|
||||
continue
|
||||
if not isinstance(op, dict):
|
||||
continue
|
||||
op_id = op.get("operationId")
|
||||
if not isinstance(op_id, str) or not op_id:
|
||||
op_id = _sanitise_tool_id(lower, path.strip("/"))
|
||||
try:
|
||||
tool = _validate_tool_name(_sanitise_tool_id(op_id))
|
||||
_validate_method(lower)
|
||||
_validate_path(path)
|
||||
except (TypeError, ValueError) as exc:
|
||||
warnings.append(f"skip {method.upper()} {path}: {exc}")
|
||||
continue
|
||||
params_raw = op.get("parameters")
|
||||
param_names: List[str] = []
|
||||
if isinstance(params_raw, list):
|
||||
for p in params_raw:
|
||||
if not isinstance(p, dict):
|
||||
continue
|
||||
if "$ref" in p:
|
||||
warnings.append("$ref left unresolved")
|
||||
continue
|
||||
name = p.get("name")
|
||||
if isinstance(name, str) and name and "\x00" not in name:
|
||||
param_names.append(name[:128])
|
||||
endpoints.append(
|
||||
Endpoint(
|
||||
tool=tool,
|
||||
method=lower,
|
||||
path=path,
|
||||
description=_truncate_desc(
|
||||
op.get("summary") or op.get("description")
|
||||
),
|
||||
parameters=tuple(param_names),
|
||||
spec_kind="openapi",
|
||||
)
|
||||
)
|
||||
if len(endpoints) >= _MAX_ENDPOINTS:
|
||||
warnings.append(
|
||||
f"endpoint cap {_MAX_ENDPOINTS} reached; truncating"
|
||||
)
|
||||
return endpoints, warnings
|
||||
return endpoints, warnings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MCP manifest parser
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_mcp(manifest: Mapping[str, Any]) -> Tuple[List[Endpoint], List[str]]:
|
||||
"""Parse an MCP server manifest's ``tools`` array."""
|
||||
if not isinstance(manifest, dict):
|
||||
raise TypeError("mcp manifest must be a dict")
|
||||
warnings: List[str] = []
|
||||
tools_raw = manifest.get("tools")
|
||||
if not isinstance(tools_raw, list):
|
||||
return [], ["mcp manifest missing 'tools' array"]
|
||||
endpoints: List[Endpoint] = []
|
||||
for entry in tools_raw:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
name = entry.get("name")
|
||||
if not isinstance(name, str) or not name:
|
||||
warnings.append("mcp tool missing name; skipped")
|
||||
continue
|
||||
try:
|
||||
tool = _validate_tool_name(_sanitise_tool_id(name))
|
||||
except (TypeError, ValueError) as exc:
|
||||
warnings.append(f"skip mcp tool {name!r}: {exc}")
|
||||
continue
|
||||
params: List[str] = []
|
||||
input_schema = entry.get("inputSchema")
|
||||
if isinstance(input_schema, dict):
|
||||
props = input_schema.get("properties")
|
||||
if isinstance(props, dict):
|
||||
for key in props:
|
||||
if isinstance(key, str) and key and "\x00" not in key:
|
||||
params.append(key[:128])
|
||||
try:
|
||||
mcp_path = _validate_path(f"mcp://{name}")
|
||||
except (TypeError, ValueError) as exc:
|
||||
warnings.append(f"skip mcp tool {name!r}: invalid path: {exc}")
|
||||
continue
|
||||
endpoints.append(
|
||||
Endpoint(
|
||||
tool=tool,
|
||||
method="invoke",
|
||||
path=mcp_path,
|
||||
description=_truncate_desc(entry.get("description")),
|
||||
parameters=tuple(params),
|
||||
spec_kind="mcp",
|
||||
)
|
||||
)
|
||||
if len(endpoints) >= _MAX_ENDPOINTS:
|
||||
warnings.append(f"endpoint cap {_MAX_ENDPOINTS} reached")
|
||||
return endpoints, warnings
|
||||
return endpoints, warnings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GraphQL introspection parser
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_graphql(intro: Mapping[str, Any]) -> Tuple[List[Endpoint], List[str]]:
|
||||
"""Parse a GraphQL introspection ``dict`` (``__schema`` envelope OK).
|
||||
|
||||
Treats every Query / Mutation field as a tool-calling endpoint.
|
||||
"""
|
||||
if not isinstance(intro, dict):
|
||||
raise TypeError("graphql introspection must be a dict")
|
||||
warnings: List[str] = []
|
||||
data = intro.get("data") if "data" in intro else intro
|
||||
if not isinstance(data, dict):
|
||||
return [], ["graphql introspection: no 'data' or schema dict"]
|
||||
schema = data.get("__schema") if "__schema" in data else data
|
||||
if not isinstance(schema, dict):
|
||||
return [], ["graphql introspection: no '__schema' field"]
|
||||
type_list = schema.get("types")
|
||||
if not isinstance(type_list, list):
|
||||
return [], ["graphql introspection: missing 'types'"]
|
||||
query_type = schema.get("queryType") or {}
|
||||
mutation_type = schema.get("mutationType") or {}
|
||||
q_name = query_type.get("name") if isinstance(query_type, dict) else None
|
||||
m_name = mutation_type.get("name") if isinstance(mutation_type, dict) else None
|
||||
endpoints: List[Endpoint] = []
|
||||
for t in type_list:
|
||||
if not isinstance(t, dict):
|
||||
continue
|
||||
type_name = t.get("name")
|
||||
if type_name not in (q_name, m_name):
|
||||
continue
|
||||
fields = t.get("fields")
|
||||
if not isinstance(fields, list):
|
||||
continue
|
||||
method = "query" if type_name == q_name else "mutation"
|
||||
for f in fields:
|
||||
if not isinstance(f, dict):
|
||||
continue
|
||||
fname = f.get("name")
|
||||
if not isinstance(fname, str) or not fname:
|
||||
continue
|
||||
try:
|
||||
tool = _validate_tool_name(_sanitise_tool_id(method, fname))
|
||||
except (TypeError, ValueError) as exc:
|
||||
warnings.append(f"skip {fname!r}: {exc}")
|
||||
continue
|
||||
args = f.get("args") or []
|
||||
arg_names: List[str] = []
|
||||
if isinstance(args, list):
|
||||
for a in args:
|
||||
if isinstance(a, dict):
|
||||
an = a.get("name")
|
||||
if isinstance(an, str) and an and "\x00" not in an:
|
||||
arg_names.append(an[:128])
|
||||
try:
|
||||
gql_path = _validate_path(f"graphql://{fname}")
|
||||
except (TypeError, ValueError) as exc:
|
||||
warnings.append(f"skip {fname!r}: invalid path: {exc}")
|
||||
continue
|
||||
endpoints.append(
|
||||
Endpoint(
|
||||
tool=tool,
|
||||
method=method,
|
||||
path=gql_path,
|
||||
description=_truncate_desc(f.get("description")),
|
||||
parameters=tuple(arg_names),
|
||||
spec_kind="graphql",
|
||||
)
|
||||
)
|
||||
if len(endpoints) >= _MAX_ENDPOINTS:
|
||||
warnings.append(f"endpoint cap {_MAX_ENDPOINTS} reached")
|
||||
return endpoints, warnings
|
||||
return endpoints, warnings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dispatcher + dataset writer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def detect_spec_kind(spec: Mapping[str, Any]) -> str:
|
||||
"""Best-effort detection of spec kind from a parsed dict.
|
||||
|
||||
Returns one of {"openapi", "mcp", "graphql"} or raises ``ValueError``.
|
||||
"""
|
||||
if not isinstance(spec, dict):
|
||||
raise TypeError("spec must be a dict")
|
||||
if isinstance(spec.get("openapi"), str) and isinstance(spec.get("paths"), dict):
|
||||
return "openapi"
|
||||
if isinstance(spec.get("tools"), list) and not isinstance(
|
||||
spec.get("paths"), dict
|
||||
):
|
||||
return "mcp"
|
||||
if "__schema" in spec or (
|
||||
isinstance(spec.get("data"), dict) and "__schema" in spec["data"]
|
||||
):
|
||||
return "graphql"
|
||||
raise ValueError(
|
||||
"cannot detect spec kind — must be OpenAPI 3.x / MCP / GraphQL"
|
||||
)
|
||||
|
||||
|
||||
def parse_spec(
|
||||
spec: Mapping[str, Any], kind: Optional[str] = None
|
||||
) -> Tuple[List[Endpoint], SpecReport]:
|
||||
"""Parse a spec dict with optional explicit ``kind`` override."""
|
||||
if kind is None:
|
||||
resolved = detect_spec_kind(spec)
|
||||
else:
|
||||
if not isinstance(kind, str):
|
||||
raise TypeError("kind must be a string")
|
||||
resolved = kind.strip().lower()
|
||||
if resolved not in _ALLOWED_SPEC_KINDS:
|
||||
raise ValueError(f"unknown spec kind: {kind!r}")
|
||||
if resolved == "openapi":
|
||||
endpoints, warnings = parse_openapi(spec)
|
||||
elif resolved == "mcp":
|
||||
endpoints, warnings = parse_mcp(spec)
|
||||
else:
|
||||
endpoints, warnings = parse_graphql(spec)
|
||||
# Dedup by tool name (last write wins is unsafe — first wins, preserves order).
|
||||
seen: Dict[str, Endpoint] = {}
|
||||
skipped = 0
|
||||
for ep in endpoints:
|
||||
if ep.tool in seen:
|
||||
skipped += 1
|
||||
continue
|
||||
seen[ep.tool] = ep
|
||||
report = SpecReport(
|
||||
spec_kind=resolved,
|
||||
endpoint_count=len(seen),
|
||||
skipped=skipped,
|
||||
warnings=tuple(warnings),
|
||||
)
|
||||
return list(seen.values()), report
|
||||
|
||||
|
||||
def endpoint_to_rows(
|
||||
endpoint: Endpoint, examples_per_endpoint: int = 1
|
||||
) -> List[SynthRow]:
|
||||
"""Synthesise ``examples_per_endpoint`` rows for one endpoint.
|
||||
|
||||
A row is one user-question / assistant-tool-call pair. We do NOT make
|
||||
network calls here; the assistant content embeds an empty arguments
|
||||
object that the trainer is meant to learn to fill from the user query.
|
||||
"""
|
||||
if not isinstance(endpoint, Endpoint):
|
||||
raise TypeError("endpoint must be an Endpoint")
|
||||
if isinstance(examples_per_endpoint, bool) or not isinstance(
|
||||
examples_per_endpoint, int
|
||||
):
|
||||
raise TypeError("examples_per_endpoint must be int (not bool)")
|
||||
if not (1 <= examples_per_endpoint <= _MAX_ROWS_PER_ENDPOINT):
|
||||
raise ValueError(
|
||||
f"examples_per_endpoint must be in [1, {_MAX_ROWS_PER_ENDPOINT}]"
|
||||
)
|
||||
desc = endpoint.description or f"Call {endpoint.tool}"
|
||||
user_templates = [
|
||||
f"Please {desc}.",
|
||||
f"How do I use {endpoint.tool}?",
|
||||
f"Run the {endpoint.tool} action with sensible defaults.",
|
||||
]
|
||||
rows: List[SynthRow] = []
|
||||
for i in range(examples_per_endpoint):
|
||||
user_msg = user_templates[i % len(user_templates)]
|
||||
tool_args: Dict[str, str] = {p: "<value>" for p in endpoint.parameters}
|
||||
assistant_msg = {
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": f"call_{i}",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": endpoint.tool,
|
||||
"arguments": json.dumps(tool_args, sort_keys=True),
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
rows.append(
|
||||
SynthRow(
|
||||
messages=(
|
||||
{"role": "user", "content": user_msg},
|
||||
assistant_msg,
|
||||
),
|
||||
tool=endpoint.tool,
|
||||
source_endpoint=endpoint.path,
|
||||
)
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def synthesise_dataset(
|
||||
endpoints: Sequence[Endpoint], examples_per_endpoint: int = 1
|
||||
) -> List[SynthRow]:
|
||||
"""Synthesise a flat list of training rows from a list of endpoints."""
|
||||
if isinstance(endpoints, (str, bytes)):
|
||||
raise TypeError("endpoints must be a sequence of Endpoint")
|
||||
out: List[SynthRow] = []
|
||||
for ep in endpoints:
|
||||
out.extend(endpoint_to_rows(ep, examples_per_endpoint))
|
||||
return out
|
||||
|
||||
|
||||
def load_spec_file(spec_path: str) -> Mapping[str, Any]:
|
||||
"""Load a YAML/JSON spec from disk with cwd containment + size cap.
|
||||
|
||||
Symlinks are rejected (TOCTOU defence, mirrors v0.45.0 Part E policy).
|
||||
"""
|
||||
if not isinstance(spec_path, str):
|
||||
raise TypeError("spec_path must be a string")
|
||||
if not spec_path or "\x00" in spec_path:
|
||||
raise ValueError("spec_path must be non-empty NUL-free string")
|
||||
if not is_under_cwd(spec_path):
|
||||
raise ValueError(
|
||||
f"spec_path must stay under cwd: {os.path.basename(spec_path)}"
|
||||
)
|
||||
# lstat BEFORE realpath: project-standard TOCTOU policy (v0.33.0 #22 /
|
||||
# v0.43.0 Part C / v0.44.0 Part B). The lstat must operate on the
|
||||
# original (pre-realpath) path so we see the symlink, not its target.
|
||||
import stat as _stat
|
||||
|
||||
try:
|
||||
st = os.lstat(spec_path)
|
||||
except FileNotFoundError as exc:
|
||||
raise FileNotFoundError(spec_path) from exc
|
||||
if _stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(
|
||||
f"spec_path must not be a symlink: {os.path.basename(spec_path)}"
|
||||
)
|
||||
real = os.path.realpath(spec_path)
|
||||
if not os.path.isfile(real):
|
||||
raise FileNotFoundError(spec_path)
|
||||
size = os.path.getsize(real)
|
||||
if size > _MAX_SPEC_BYTES:
|
||||
raise ValueError(f"spec file exceeds {_MAX_SPEC_BYTES} bytes ({size})")
|
||||
with open(real, "r", encoding="utf-8") as fh:
|
||||
text = fh.read()
|
||||
if spec_path.lower().endswith((".yaml", ".yml")):
|
||||
import yaml
|
||||
|
||||
loaded = yaml.safe_load(text)
|
||||
else:
|
||||
loaded = json.loads(text)
|
||||
if not isinstance(loaded, dict):
|
||||
raise ValueError("spec file root must be a JSON/YAML object")
|
||||
return loaded
|
||||
|
||||
|
||||
def write_dataset(rows: Sequence[SynthRow], output_path: str) -> str:
|
||||
"""Write rows as JSONL under cwd; returns realpath written.
|
||||
|
||||
Atomic via staged-tempfile + ``os.replace`` (matches v0.43.0 Part D
|
||||
``copy_bundle_to`` policy). Validates every row BEFORE any bytes hit
|
||||
the target path — a mid-stream ``TypeError`` never leaves a partial
|
||||
file. Symlink at the target rejected via ``os.lstat`` (TOCTOU).
|
||||
"""
|
||||
import stat as _stat
|
||||
import tempfile
|
||||
|
||||
if not isinstance(output_path, str):
|
||||
raise TypeError("output_path must be a string")
|
||||
if not output_path or "\x00" in output_path:
|
||||
raise ValueError("output_path must be non-empty NUL-free string")
|
||||
if not is_under_cwd(output_path):
|
||||
raise ValueError(
|
||||
f"output_path must stay under cwd: {os.path.basename(output_path)}"
|
||||
)
|
||||
# Reject a pre-placed symlink at the target — defends against
|
||||
# `<output>.jsonl -> /etc/cron.d/x` overwrite.
|
||||
try:
|
||||
st = os.lstat(output_path)
|
||||
if _stat.S_ISLNK(st.st_mode):
|
||||
raise ValueError(
|
||||
f"output_path must not be a symlink: "
|
||||
f"{os.path.basename(output_path)}"
|
||||
)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
real = os.path.realpath(output_path)
|
||||
parent = os.path.dirname(real) or "."
|
||||
os.makedirs(parent, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(prefix=".agent_forge_", suffix=".tmp", dir=parent)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
for row in rows:
|
||||
if not isinstance(row, SynthRow):
|
||||
raise TypeError("rows must all be SynthRow")
|
||||
fh.write(json.dumps(row.to_dict(), sort_keys=True))
|
||||
fh.write("\n")
|
||||
os.replace(tmp, real)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
return real
|
||||
|
||||
|
||||
__all__ = [
|
||||
"Endpoint",
|
||||
"SynthRow",
|
||||
"SpecReport",
|
||||
"parse_openapi",
|
||||
"parse_mcp",
|
||||
"parse_graphql",
|
||||
"detect_spec_kind",
|
||||
"parse_spec",
|
||||
"endpoint_to_rows",
|
||||
"synthesise_dataset",
|
||||
"load_spec_file",
|
||||
"write_dataset",
|
||||
]
|
||||
|
|
@ -0,0 +1,828 @@
|
|||
"""v0.46.0 Part B — Agent Forge tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.utils.agent_forge import (
|
||||
Endpoint,
|
||||
SpecReport,
|
||||
SynthRow,
|
||||
detect_spec_kind,
|
||||
endpoint_to_rows,
|
||||
load_spec_file,
|
||||
parse_graphql,
|
||||
parse_mcp,
|
||||
parse_openapi,
|
||||
parse_spec,
|
||||
synthesise_dataset,
|
||||
write_dataset,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse_openapi
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_OPENAPI_SAMPLE = {
|
||||
"openapi": "3.0.0",
|
||||
"info": {"title": "Sample", "version": "1.0"},
|
||||
"paths": {
|
||||
"/pets": {
|
||||
"get": {
|
||||
"operationId": "listPets",
|
||||
"summary": "List pets",
|
||||
"parameters": [
|
||||
{"name": "limit", "in": "query"},
|
||||
],
|
||||
},
|
||||
"post": {
|
||||
"operationId": "createPet",
|
||||
"summary": "Create a pet",
|
||||
},
|
||||
},
|
||||
"/pets/{id}": {
|
||||
"get": {
|
||||
"operationId": "getPet",
|
||||
"parameters": [
|
||||
{"name": "id", "in": "path"},
|
||||
],
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def test_parse_openapi_basic():
|
||||
endpoints, warnings = parse_openapi(_OPENAPI_SAMPLE)
|
||||
tools = {ep.tool for ep in endpoints}
|
||||
assert "listPets" in tools
|
||||
assert "createPet" in tools
|
||||
assert "getPet" in tools
|
||||
assert warnings == [] or all("$ref" not in w for w in warnings)
|
||||
|
||||
|
||||
def test_parse_openapi_extracts_parameter_names():
|
||||
endpoints, _ = parse_openapi(_OPENAPI_SAMPLE)
|
||||
list_pets = next(ep for ep in endpoints if ep.tool == "listPets")
|
||||
assert "limit" in list_pets.parameters
|
||||
|
||||
|
||||
def test_parse_openapi_method_lowercased():
|
||||
endpoints, _ = parse_openapi(_OPENAPI_SAMPLE)
|
||||
for ep in endpoints:
|
||||
assert ep.method == ep.method.lower()
|
||||
|
||||
|
||||
def test_parse_openapi_non_dict_raises():
|
||||
with pytest.raises(TypeError):
|
||||
parse_openapi("not a dict") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_parse_openapi_missing_paths_returns_empty():
|
||||
eps, warnings = parse_openapi({"openapi": "3.0.0"})
|
||||
assert eps == []
|
||||
assert warnings
|
||||
|
||||
|
||||
def test_parse_openapi_wrong_version_warns():
|
||||
_, warnings = parse_openapi({"openapi": "2.0.0", "paths": {}})
|
||||
assert any("openapi" in w.lower() for w in warnings)
|
||||
|
||||
|
||||
def test_parse_openapi_skips_invalid_methods():
|
||||
spec = {
|
||||
"openapi": "3.0.0",
|
||||
"paths": {
|
||||
"/x": {
|
||||
"BOGUS": {"operationId": "skipMe"},
|
||||
"get": {"operationId": "keepMe"},
|
||||
}
|
||||
},
|
||||
}
|
||||
endpoints, _ = parse_openapi(spec)
|
||||
tools = {ep.tool for ep in endpoints}
|
||||
assert "keepMe" in tools
|
||||
assert "skipMe" not in tools
|
||||
|
||||
|
||||
def test_parse_openapi_generates_id_when_missing():
|
||||
spec = {
|
||||
"openapi": "3.0.0",
|
||||
"paths": {"/widgets": {"get": {}}},
|
||||
}
|
||||
endpoints, _ = parse_openapi(spec)
|
||||
assert endpoints
|
||||
# Generated id sanitised, starts with letter or underscore
|
||||
assert endpoints[0].tool[0].isalpha() or endpoints[0].tool.startswith("_")
|
||||
|
||||
|
||||
def test_parse_openapi_ref_param_skipped_with_warning():
|
||||
spec = {
|
||||
"openapi": "3.0.0",
|
||||
"paths": {
|
||||
"/x": {
|
||||
"get": {
|
||||
"operationId": "getX",
|
||||
"parameters": [{"$ref": "#/components/parameters/X"}],
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
eps, warnings = parse_openapi(spec)
|
||||
assert eps
|
||||
assert any("$ref" in w for w in warnings)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse_mcp
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_MCP_SAMPLE = {
|
||||
"tools": [
|
||||
{
|
||||
"name": "search_files",
|
||||
"description": "Search files by pattern",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {"pattern": {"type": "string"}, "path": {"type": "string"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
"name": "read_file",
|
||||
"description": "Read a file by path",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {"path": {"type": "string"}},
|
||||
},
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def test_parse_mcp_basic():
|
||||
endpoints, _ = parse_mcp(_MCP_SAMPLE)
|
||||
tools = {ep.tool for ep in endpoints}
|
||||
assert "search_files" in tools
|
||||
assert "read_file" in tools
|
||||
|
||||
|
||||
def test_parse_mcp_extracts_input_schema_props():
|
||||
endpoints, _ = parse_mcp(_MCP_SAMPLE)
|
||||
search = next(ep for ep in endpoints if ep.tool == "search_files")
|
||||
assert "pattern" in search.parameters
|
||||
assert "path" in search.parameters
|
||||
|
||||
|
||||
def test_parse_mcp_method_is_invoke():
|
||||
endpoints, _ = parse_mcp(_MCP_SAMPLE)
|
||||
for ep in endpoints:
|
||||
assert ep.method == "invoke"
|
||||
|
||||
|
||||
def test_parse_mcp_path_uses_mcp_scheme():
|
||||
endpoints, _ = parse_mcp(_MCP_SAMPLE)
|
||||
for ep in endpoints:
|
||||
assert ep.path.startswith("mcp://")
|
||||
|
||||
|
||||
def test_parse_mcp_missing_tools_returns_empty():
|
||||
eps, warnings = parse_mcp({})
|
||||
assert eps == []
|
||||
assert warnings
|
||||
|
||||
|
||||
def test_parse_mcp_missing_name_warns():
|
||||
spec = {"tools": [{"description": "no name"}]}
|
||||
_, warnings = parse_mcp(spec)
|
||||
assert any("name" in w.lower() for w in warnings)
|
||||
|
||||
|
||||
def test_parse_mcp_non_dict_raises():
|
||||
with pytest.raises(TypeError):
|
||||
parse_mcp([]) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse_graphql
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_GRAPHQL_SAMPLE = {
|
||||
"data": {
|
||||
"__schema": {
|
||||
"queryType": {"name": "Query"},
|
||||
"mutationType": {"name": "Mutation"},
|
||||
"types": [
|
||||
{
|
||||
"name": "Query",
|
||||
"fields": [
|
||||
{
|
||||
"name": "user",
|
||||
"description": "Fetch a user by id",
|
||||
"args": [{"name": "id"}],
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Mutation",
|
||||
"fields": [
|
||||
{
|
||||
"name": "createUser",
|
||||
"description": "Create a user",
|
||||
"args": [{"name": "name"}, {"name": "email"}],
|
||||
}
|
||||
],
|
||||
},
|
||||
],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def test_parse_graphql_basic():
|
||||
endpoints, _ = parse_graphql(_GRAPHQL_SAMPLE)
|
||||
tools = {ep.tool for ep in endpoints}
|
||||
assert any("user" in t for t in tools)
|
||||
assert any("createUser" in t for t in tools)
|
||||
|
||||
|
||||
def test_parse_graphql_methods():
|
||||
endpoints, _ = parse_graphql(_GRAPHQL_SAMPLE)
|
||||
methods = {ep.method for ep in endpoints}
|
||||
assert "query" in methods
|
||||
assert "mutation" in methods
|
||||
|
||||
|
||||
def test_parse_graphql_args_captured():
|
||||
endpoints, _ = parse_graphql(_GRAPHQL_SAMPLE)
|
||||
create_user = next(ep for ep in endpoints if "createUser" in ep.tool)
|
||||
assert "name" in create_user.parameters
|
||||
assert "email" in create_user.parameters
|
||||
|
||||
|
||||
def test_parse_graphql_missing_schema_returns_empty():
|
||||
eps, warnings = parse_graphql({"data": {}})
|
||||
assert eps == []
|
||||
assert warnings
|
||||
|
||||
|
||||
def test_parse_graphql_non_dict_raises():
|
||||
with pytest.raises(TypeError):
|
||||
parse_graphql("nope") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# detect_spec_kind + parse_spec
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_detect_openapi():
|
||||
assert detect_spec_kind(_OPENAPI_SAMPLE) == "openapi"
|
||||
|
||||
|
||||
def test_detect_mcp():
|
||||
assert detect_spec_kind(_MCP_SAMPLE) == "mcp"
|
||||
|
||||
|
||||
def test_detect_graphql():
|
||||
assert detect_spec_kind(_GRAPHQL_SAMPLE) == "graphql"
|
||||
|
||||
|
||||
def test_detect_unknown_raises():
|
||||
with pytest.raises(ValueError, match="cannot detect"):
|
||||
detect_spec_kind({"random": "stuff"})
|
||||
|
||||
|
||||
def test_detect_non_dict_raises():
|
||||
with pytest.raises(TypeError):
|
||||
detect_spec_kind("not a dict") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_parse_spec_auto_detect():
|
||||
endpoints, report = parse_spec(_OPENAPI_SAMPLE)
|
||||
assert report.spec_kind == "openapi"
|
||||
assert report.endpoint_count >= 3
|
||||
assert isinstance(report, SpecReport)
|
||||
|
||||
|
||||
def test_parse_spec_explicit_kind():
|
||||
endpoints, report = parse_spec(_MCP_SAMPLE, kind="mcp")
|
||||
assert report.spec_kind == "mcp"
|
||||
assert endpoints
|
||||
|
||||
|
||||
def test_parse_spec_unknown_kind_rejected():
|
||||
with pytest.raises(ValueError, match="unknown spec kind"):
|
||||
parse_spec(_OPENAPI_SAMPLE, kind="evil")
|
||||
|
||||
|
||||
def test_parse_spec_non_string_kind():
|
||||
with pytest.raises(TypeError):
|
||||
parse_spec(_OPENAPI_SAMPLE, kind=123) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_parse_spec_deduplicates_tools():
|
||||
spec = {
|
||||
"openapi": "3.0.0",
|
||||
"paths": {
|
||||
"/x": {
|
||||
"get": {"operationId": "myTool"},
|
||||
},
|
||||
"/y": {
|
||||
"get": {"operationId": "myTool"},
|
||||
},
|
||||
},
|
||||
}
|
||||
endpoints, report = parse_spec(spec)
|
||||
assert len({ep.tool for ep in endpoints}) == 1
|
||||
assert report.skipped == 1
|
||||
|
||||
|
||||
def test_parse_mcp_rejects_newline_in_name():
|
||||
spec = {"tools": [{"name": "evil\nhost", "description": "x"}]}
|
||||
eps, warnings = parse_mcp(spec)
|
||||
# Either skipped with warning, or path validation strips/rejects newline
|
||||
if eps:
|
||||
for ep in eps:
|
||||
assert "\n" not in ep.path
|
||||
assert any("invalid" in w.lower() or "skip" in w.lower() for w in warnings) or not eps
|
||||
|
||||
|
||||
def test_parse_graphql_rejects_newline_in_field_name():
|
||||
spec = {
|
||||
"__schema": {
|
||||
"queryType": {"name": "Query"},
|
||||
"types": [{
|
||||
"name": "Query",
|
||||
"fields": [{"name": "evil\nhost", "args": []}],
|
||||
}],
|
||||
}
|
||||
}
|
||||
eps, _ = parse_graphql(spec)
|
||||
for ep in eps:
|
||||
assert "\n" not in ep.path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# endpoint_to_rows + synthesise_dataset
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_endpoint_to_rows_basic():
|
||||
ep = Endpoint(
|
||||
tool="search", method="get", path="/search",
|
||||
description="Search the index", parameters=("query",),
|
||||
spec_kind="openapi",
|
||||
)
|
||||
rows = endpoint_to_rows(ep, examples_per_endpoint=2)
|
||||
assert len(rows) == 2
|
||||
for row in rows:
|
||||
assert isinstance(row, SynthRow)
|
||||
assert row.tool == "search"
|
||||
assert row.source_endpoint == "/search"
|
||||
# 2 messages: user + assistant-with-tool-call
|
||||
assert len(row.messages) == 2
|
||||
assert row.messages[0]["role"] == "user"
|
||||
assert row.messages[1]["role"] == "assistant"
|
||||
assert "tool_calls" in row.messages[1]
|
||||
|
||||
|
||||
def test_endpoint_to_rows_arguments_are_json_string():
|
||||
ep = Endpoint(
|
||||
tool="search", method="get", path="/search",
|
||||
description="", parameters=("q", "limit"), spec_kind="openapi",
|
||||
)
|
||||
rows = endpoint_to_rows(ep, examples_per_endpoint=1)
|
||||
tc = rows[0].messages[1]["tool_calls"][0]
|
||||
parsed = json.loads(tc["function"]["arguments"])
|
||||
assert set(parsed.keys()) == {"q", "limit"}
|
||||
|
||||
|
||||
def test_endpoint_to_rows_bool_examples_rejected():
|
||||
ep = Endpoint(
|
||||
tool="x", method="get", path="/x",
|
||||
description="", parameters=(), spec_kind="openapi",
|
||||
)
|
||||
with pytest.raises(TypeError):
|
||||
endpoint_to_rows(ep, examples_per_endpoint=True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_endpoint_to_rows_zero_rejected():
|
||||
ep = Endpoint(
|
||||
tool="x", method="get", path="/x",
|
||||
description="", parameters=(), spec_kind="openapi",
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
endpoint_to_rows(ep, examples_per_endpoint=0)
|
||||
|
||||
|
||||
def test_endpoint_to_rows_oversize_rejected():
|
||||
ep = Endpoint(
|
||||
tool="x", method="get", path="/x",
|
||||
description="", parameters=(), spec_kind="openapi",
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
endpoint_to_rows(ep, examples_per_endpoint=33)
|
||||
|
||||
|
||||
def test_endpoint_to_rows_type_check():
|
||||
with pytest.raises(TypeError):
|
||||
endpoint_to_rows("not-an-endpoint", 1) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_synthesise_dataset_flat_list():
|
||||
endpoints, _ = parse_openapi(_OPENAPI_SAMPLE)
|
||||
rows = synthesise_dataset(endpoints, examples_per_endpoint=2)
|
||||
assert len(rows) == 2 * len(endpoints)
|
||||
assert all(isinstance(r, SynthRow) for r in rows)
|
||||
|
||||
|
||||
def test_synthesise_dataset_rejects_string():
|
||||
with pytest.raises(TypeError):
|
||||
synthesise_dataset("not a list", 1) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_synth_row_to_dict_serialisable():
|
||||
ep = Endpoint(
|
||||
tool="x", method="get", path="/x",
|
||||
description="", parameters=(), spec_kind="openapi",
|
||||
)
|
||||
row = endpoint_to_rows(ep, 1)[0]
|
||||
d = row.to_dict()
|
||||
# Round-trips through JSON
|
||||
json.dumps(d)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# load_spec_file + write_dataset (cwd containment + symlink)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_load_spec_file_yaml(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
p = tmp_path / "spec.yaml"
|
||||
p.write_text("openapi: '3.0.0'\npaths: {}\n", encoding="utf-8")
|
||||
out = load_spec_file("spec.yaml")
|
||||
assert out.get("openapi") == "3.0.0"
|
||||
|
||||
|
||||
def test_load_spec_file_json(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
p = tmp_path / "spec.json"
|
||||
p.write_text(json.dumps(_OPENAPI_SAMPLE), encoding="utf-8")
|
||||
out = load_spec_file("spec.json")
|
||||
assert out["openapi"] == "3.0.0"
|
||||
|
||||
|
||||
def test_load_spec_file_outside_cwd_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
other = tmp_path.parent / "evil.json"
|
||||
other.write_text("{}", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="must stay under cwd"):
|
||||
load_spec_file(str(other))
|
||||
|
||||
|
||||
def test_load_spec_file_missing(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_spec_file("missing.json")
|
||||
|
||||
|
||||
def test_load_spec_file_null_byte_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
load_spec_file("evil\x00.json")
|
||||
|
||||
|
||||
def test_load_spec_file_non_string():
|
||||
with pytest.raises(TypeError):
|
||||
load_spec_file(123) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_load_spec_file_oversize_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
p = tmp_path / "big.json"
|
||||
p.write_bytes(b"{" + b"x" * (6 * 1024 * 1024) + b"}")
|
||||
with pytest.raises(ValueError, match="exceeds"):
|
||||
load_spec_file("big.json")
|
||||
|
||||
|
||||
def test_load_spec_file_non_dict_root_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
p = tmp_path / "list.json"
|
||||
p.write_text("[1, 2, 3]", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="object"):
|
||||
load_spec_file("list.json")
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform == "win32", reason="symlink ACL on Windows CI")
|
||||
def test_load_spec_file_symlink_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
real = tmp_path / "real.json"
|
||||
real.write_text("{}", encoding="utf-8")
|
||||
link = tmp_path / "link.json"
|
||||
try:
|
||||
os.symlink(real, link)
|
||||
except (OSError, NotImplementedError):
|
||||
pytest.skip("symlink unavailable")
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
load_spec_file("link.json")
|
||||
|
||||
|
||||
def test_write_dataset_under_cwd(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
endpoints, _ = parse_openapi(_OPENAPI_SAMPLE)
|
||||
rows = synthesise_dataset(endpoints, 1)
|
||||
out = write_dataset(rows, "dataset.jsonl")
|
||||
assert os.path.exists(out)
|
||||
# Each line valid JSON
|
||||
with open(out, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
data = json.loads(line)
|
||||
assert "messages" in data
|
||||
assert "tool" in data
|
||||
|
||||
|
||||
def test_write_dataset_outside_cwd_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
rows = [SynthRow(messages=({"role": "u", "content": "x"},), tool="t",
|
||||
source_endpoint="/")]
|
||||
abs_outside = str(tmp_path.parent / "evil.jsonl")
|
||||
with pytest.raises(ValueError, match="must stay under cwd"):
|
||||
write_dataset(rows, abs_outside)
|
||||
|
||||
|
||||
def test_write_dataset_null_byte_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
write_dataset([], "x\x00.jsonl")
|
||||
|
||||
|
||||
def test_write_dataset_non_string_path():
|
||||
with pytest.raises(TypeError):
|
||||
write_dataset([], 123) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_write_dataset_invalid_row_type(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(TypeError):
|
||||
write_dataset([{"not": "a SynthRow"}], "out.jsonl") # type: ignore[list-item]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI smoke tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cli_agent_synth_smoke(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "spec.json").write_text(
|
||||
json.dumps(_OPENAPI_SAMPLE), encoding="utf-8"
|
||||
)
|
||||
result = runner.invoke(
|
||||
agent.app, ["synth", "--spec", "spec.json", "--output", "ds.jsonl"]
|
||||
)
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert (tmp_path / "ds.jsonl").exists()
|
||||
assert "listPets" in result.output
|
||||
|
||||
|
||||
def test_cli_agent_synth_unknown_spec_path(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
result = runner.invoke(agent.app, ["synth", "--spec", "missing.json"])
|
||||
assert result.exit_code == 1, result.output
|
||||
|
||||
|
||||
def test_cli_agent_synth_outside_cwd_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
abs_outside = str(tmp_path.parent / "evil.json")
|
||||
result = runner.invoke(agent.app, ["synth", "--spec", abs_outside])
|
||||
assert result.exit_code == 1, result.output
|
||||
|
||||
|
||||
def test_cli_agent_train_smoke(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "spec.json").write_text(
|
||||
json.dumps(_OPENAPI_SAMPLE), encoding="utf-8"
|
||||
)
|
||||
result = runner.invoke(
|
||||
agent.app,
|
||||
["train", "--spec", "spec.json", "--base", "meta-llama/Llama-3.2-1B",
|
||||
"--dataset-out", "ds.jsonl"],
|
||||
)
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
assert (tmp_path / "ds.jsonl").exists()
|
||||
assert "Planned" in result.output or "agent_train.yaml" in result.output
|
||||
|
||||
|
||||
def test_cli_agent_eval_smoke(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "spec.json").write_text(
|
||||
json.dumps(_OPENAPI_SAMPLE), encoding="utf-8"
|
||||
)
|
||||
preds = [
|
||||
{"tool": "listPets", "arguments": {"limit": "10"}},
|
||||
{"tool": "listPets", "arguments": {"unknownParam": "x"}},
|
||||
{"tool": "nopeTool", "arguments": {}},
|
||||
]
|
||||
(tmp_path / "preds.jsonl").write_text(
|
||||
"\n".join(json.dumps(p) for p in preds) + "\n", encoding="utf-8"
|
||||
)
|
||||
result = runner.invoke(
|
||||
agent.app,
|
||||
["eval", "--spec", "spec.json", "--predictions", "preds.jsonl"],
|
||||
)
|
||||
assert result.exit_code == 0, (result.output, repr(result.exception))
|
||||
# 2/3 tools match (listPets x2), 1/3 args fully valid
|
||||
assert "Tool match" in result.output
|
||||
assert "Args valid" in result.output
|
||||
|
||||
|
||||
def test_cli_agent_eval_outside_cwd_predictions(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "spec.json").write_text(
|
||||
json.dumps(_OPENAPI_SAMPLE), encoding="utf-8"
|
||||
)
|
||||
abs_outside = str(tmp_path.parent / "preds.jsonl")
|
||||
result = runner.invoke(
|
||||
agent.app,
|
||||
["eval", "--spec", "spec.json", "--predictions", abs_outside],
|
||||
)
|
||||
assert result.exit_code == 1, result.output
|
||||
|
||||
|
||||
def test_cli_agent_synth_help():
|
||||
from soup_cli.commands import agent
|
||||
|
||||
result = runner.invoke(agent.app, ["synth", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
|
||||
def test_cli_agent_help():
|
||||
from soup_cli.commands import agent
|
||||
|
||||
result = runner.invoke(agent.app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "synth" in result.output
|
||||
assert "train" in result.output
|
||||
assert "eval" in result.output
|
||||
|
||||
|
||||
def test_cli_agent_train_help():
|
||||
from soup_cli.commands import agent
|
||||
|
||||
result = runner.invoke(agent.app, ["train", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
|
||||
def test_cli_agent_eval_help():
|
||||
from soup_cli.commands import agent
|
||||
|
||||
result = runner.invoke(agent.app, ["eval", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
|
||||
def test_cli_agent_eval_missing_spec(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "preds.jsonl").write_text("{}\n", encoding="utf-8")
|
||||
result = runner.invoke(
|
||||
agent.app, ["eval", "--spec", "missing.json", "--predictions", "preds.jsonl"],
|
||||
)
|
||||
assert result.exit_code == 1, result.output
|
||||
|
||||
|
||||
def test_cli_agent_eval_outside_cwd_spec(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "preds.jsonl").write_text("{}\n", encoding="utf-8")
|
||||
abs_outside = str(tmp_path.parent / "evil.json")
|
||||
result = runner.invoke(
|
||||
agent.app,
|
||||
["eval", "--spec", abs_outside, "--predictions", "preds.jsonl"],
|
||||
)
|
||||
assert result.exit_code == 1, result.output
|
||||
|
||||
|
||||
def test_cli_agent_train_rejects_newline_in_base(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "spec.json").write_text(
|
||||
json.dumps(_OPENAPI_SAMPLE), encoding="utf-8"
|
||||
)
|
||||
result = runner.invoke(
|
||||
agent.app,
|
||||
["train", "--spec", "spec.json",
|
||||
"--base", "evil\ntraining: { epochs: 9999 }"],
|
||||
)
|
||||
assert result.exit_code == 2, result.output
|
||||
assert "newline" in result.output.lower() or "base" in result.output.lower()
|
||||
|
||||
|
||||
def test_cli_agent_train_rejects_null_byte_in_base(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "spec.json").write_text(
|
||||
json.dumps(_OPENAPI_SAMPLE), encoding="utf-8"
|
||||
)
|
||||
result = runner.invoke(
|
||||
agent.app,
|
||||
["train", "--spec", "spec.json", "--base", "evil\x00"],
|
||||
)
|
||||
assert result.exit_code == 2, result.output
|
||||
|
||||
|
||||
def test_write_dataset_partial_failure_no_partial_file(tmp_path, monkeypatch):
|
||||
"""Mid-stream TypeError must not leave a partial file at target."""
|
||||
monkeypatch.chdir(tmp_path)
|
||||
rows = [
|
||||
SynthRow(messages=(), tool="t1", source_endpoint="/"),
|
||||
"not a SynthRow", # type: ignore[list-item]
|
||||
]
|
||||
with pytest.raises(TypeError):
|
||||
write_dataset(rows, "out.jsonl") # type: ignore[arg-type]
|
||||
# Atomic write: target file should NOT exist after partial failure
|
||||
assert not (tmp_path / "out.jsonl").exists()
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform == "win32", reason="symlink ACL on Windows")
|
||||
def test_write_dataset_symlink_target_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
real = tmp_path / "real.jsonl"
|
||||
real.write_text("", encoding="utf-8")
|
||||
link = tmp_path / "link.jsonl"
|
||||
try:
|
||||
os.symlink(real, link)
|
||||
except (OSError, NotImplementedError):
|
||||
pytest.skip("symlink unavailable")
|
||||
rows = [SynthRow(messages=(), tool="t", source_endpoint="/")]
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
write_dataset(rows, "link.jsonl")
|
||||
|
||||
|
||||
def test_cli_agent_synth_no_endpoints_exits_1(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
from soup_cli.commands import agent
|
||||
|
||||
(tmp_path / "empty.json").write_text('{"openapi": "3.0.0", "paths": {}}',
|
||||
encoding="utf-8")
|
||||
result = runner.invoke(agent.app, ["synth", "--spec", "empty.json"])
|
||||
assert result.exit_code == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Endpoint dataclass invariants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_endpoint_is_frozen():
|
||||
import dataclasses
|
||||
|
||||
ep = Endpoint(
|
||||
tool="x", method="get", path="/x",
|
||||
description="", parameters=(), spec_kind="openapi",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
ep.tool = "y" # type: ignore[misc]
|
||||
|
||||
|
||||
def test_synth_row_is_frozen():
|
||||
import dataclasses
|
||||
|
||||
row = SynthRow(messages=(), tool="x", source_endpoint="/")
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
row.tool = "y" # type: ignore[misc]
|
||||
|
||||
|
||||
def test_spec_report_is_frozen_explicit():
|
||||
import dataclasses
|
||||
|
||||
_, report = parse_spec(_OPENAPI_SAMPLE)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
report.spec_kind = "x" # type: ignore[misc]
|
||||
Loading…
Reference in New Issue