mirror of https://github.com/razor-ai/soup.git
feat(v0.53.6): plugin callback + Anthropic /v1/messages + n-gram spec + 3 stubs
Ships v0.53.6 "Plugin + Agent + Anthropic API" — 6 features, 3 live and 3 stub-then-live with v0.53.7 markers. LIVE - #101 SoupPluginCallback fans HF Trainer events to enabled Soup plugin hooks (pre_train/post_train/pre_step/post_step). Hook exceptions swallowed at WARNING. Hook snapshot collected once and passed to ctor (race-free per code-review fix). Wired into all 13 transformer-backend trainers via utils/peft_wiring.attach_plugin_callback. - #102 POST /v1/messages on transformers backend reuses the v0.45.0 anthropic_messages converter + existing chat handler. Streaming -> 501. Validation errors -> generic 'Invalid request' body, details at DEBUG (security-review redaction fix). - #104 n-gram speculative decoding: NgramSpecConfig.num_draft_tokens threaded through model.generate(prompt_lookup_num_tokens=N). Mutually exclusive with assistant_model. STUB-THEN-LIVE (v0.53.7) - #103 /v1/tools/{python,bash,web_search} return HTTP 501. - #105 instantiate_trainer_plugins validates then NotImplementedError. - #106 run_recipe + 'soup data recipe --execute --output <dir>' with CLI-side is_under_cwd containment before the live runner. Tests: 7998 -> 8051 (+53 in tests/test_v0536.py). Lint clean. Three review agents (python / code / security); every HIGH/MEDIUM fixed. TDD coverage gaps closed (ngram-None regression, max_tokens cap, run_recipe boundaries, console-print failure swallow). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
8bd67f697f
commit
70326f1aa7
|
|
@ -111,7 +111,7 @@ soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (188 files, 7998 tests)
|
||||
tests/ - Test suite (189 files, 8051 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
83
README.md
83
README.md
|
|
@ -43,15 +43,15 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.53.5 — Adaptive Training (BETA → stable)**: Six closes lifting the v0.48.0 BETA deferrals to live wiring, plus a day-zero recipe for DeepSeek-V3 reasoning.
|
||||
**v0.53.6 — Plugin + Agent + Anthropic API**: Six features — three live, three deferred-live stubs with v0.53.7 markers.
|
||||
|
||||
- **Dynamic curriculum is live.** New `DynamicCurriculumCallback` — a real HF `TrainerCallback` that accumulates per-step loss + grad-norm per bucket, recomputes sampler weights every N steps, and atomically appends `curriculum_history.jsonl` rows on rank 0. Multi-rank launches are coordinated via `torch.distributed.all_reduce(SUM)` of per-bucket stats before the recompute. Visualise the trace with `soup runs curriculum-curve <run_id>`.
|
||||
- **Curriculum-aware on every transformer trainer.** The `curriculum_dynamic: true` schema gate widened from `{sft, pretrain}` to every transformer-backend wrapper (SFT / Pretrain / DPO / GRPO / KTO / ORPO / SimPO / IPO / BCO / RewardModel / Embedding / PPO / Distill). MLX backend still rejected with a distinct error message.
|
||||
- **`soup data mix --live` runs real proxy trainings.** A new `--live --base-yaml <path>` mode replaces the synthetic offline proxy with a real short `soup train` subprocess per Bayesian-search candidate. Argv-list invocation, per-candidate timeout `min(budget/num_probes, 30 min)`, tracker-SQLite parse for `eval_loss`, atomic tmp-YAML cleanup.
|
||||
- **scikit-optimize behind the OptimizerProtocol.** When `scikit-optimize` is installed, the mix optimiser now drives a real `skopt.Optimizer(GP)` (Gaussian-process Bayesian optimisation) instead of the Dirichlet fallback. Zero new required dependencies — falls back silently when skopt is absent.
|
||||
- **`MixOptimizationReport.elapsed_seconds` excludes failed candidates.** The headline elapsed-time field now sums only successful candidate wall-clock, so a single timed-out proxy can no longer inflate a report otherwise filled with quick successes. Per-candidate `wall_clock_seconds` retains the per-trial timing.
|
||||
- **`deepseek-v3-reasoning` recipe.** GRPO + reasoning template on `deepseek-ai/DeepSeek-V3` with `reward_fn=accuracy,format` + math verifiable domain — day-zero coverage for the new MoE reasoning base.
|
||||
- **+63 net new tests** (7935 → 7998) across the new `test_v0535.py`. Four review agents (python / code / security / tdd) ran; every CRITICAL → LOW finding was fixed — `is None` over falsy guards on the callback attach helper, cwd-containment + `os.lstat + S_ISLNK` rejection on `output_dir`, simplex + finite + bool-rejected validation on weights, argv-list subprocess invocation with no shell, atomic tempfile + `os.replace` for `curriculum_history.jsonl` appends.
|
||||
- **Soup plugins now run live as Trainer callbacks.** New `SoupPluginCallback` dispatches `pre_train` / `post_train` / `pre_step` / `post_step` to every enabled plugin via the v0.45.0 registry. Hook exceptions are swallowed at WARNING — one misbehaving plugin must never crash a multi-hour run. Wired into all 13 transformer-backend trainers (SFT + DPO + GRPO + KTO + ORPO + SimPO + IPO + BCO + PPO + RewardModel + Pretrain + Embedding + Distill) via `attach_plugin_callback`.
|
||||
- **Anthropic-shaped `/v1/messages` endpoint.** `soup serve --backend transformers` exposes a `POST /v1/messages` route reusing the v0.45.0 `anthropic_messages` converter + the existing chat handler. Validation errors return a generic `"Invalid request"` 400 (details logged server-side at DEBUG). Streaming returns 501 — true Anthropic event-shape SSE ships in v0.53.7.
|
||||
- **n-gram speculative decoding wired through.** When the server is started with an `NgramSpecConfig`, every chat completion forwards `prompt_lookup_num_tokens=N` into `model.generate(...)` (HF Transformers ≥ 4.38 prompt-lookup decoding). Mutually exclusive with a real draft `assistant_model`.
|
||||
- **`soup data recipe --execute` lands as a stub-then-live CLI surface.** `--execute --output <dir>` validates the DAG, enforces cwd-containment on `--output` at the CLI boundary, then surfaces the `v0.53.7` `NotImplementedError` marker from `run_recipe`. Empty-string `--output ""` and outside-cwd paths are rejected before the runner is even called — never want the live runner to be the first/only enforcement point.
|
||||
- **Server-side tool endpoints (`/v1/tools/python` + `/v1/tools/bash` + `/v1/tools/web_search`).** URL schema lives now so clients can target the endpoints. All three return HTTP 501 with the v0.53.7 marker — live RLVR sandbox HTTP wrapper + `WebSearchConfig.domain_allowlist` enforcement ship in v0.53.7.
|
||||
- **`instantiate_trainer_plugins` schema-only.** Validates `[grokfast, spectrum, llmcompressor, sonicmoe, cce_plugin, math_verify]` lists then raises with the v0.53.7 marker. Six upstream plugin lazy-imports + per-plugin callback construction land in v0.53.7.
|
||||
- **+53 net new tests** (7998 → 8051) across the new `test_v0536.py`. Three review agents (python / code / security) ran; every HIGH and MEDIUM finding was fixed — race-free single-snapshot plugin hook collection, generic `"Invalid request"` body redaction, CLI-side cwd containment before `run_recipe`, `is None` over falsy `--output` guards, and added test coverage for: `prompt_lookup_num_tokens` omitted when `ngram_config=None`, console-print failure swallow on the attach helper, every `run_recipe` type-rejection boundary.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -3787,6 +3787,73 @@ The advanced GGUF pipeline uses POSIX `O_NOFOLLOW` to defeat the TOCTOU race bet
|
|||
|
||||
`soup deploy autopilot --measure` caches results at `~/.soup/deploy_autopilot_cache.json` keyed on `(base, profile, eval-tasks)`. Repeat invocations short-circuit; pass `SOUP_DEPLOY_AUTOPILOT_CACHE=<path>` to redirect (constrained to home / cwd / tempdir). The recommended candidate uses soft-fallback: first `OK` by insertion order, else the candidate with the smallest delta (least drop relative to its own baseline).
|
||||
|
||||
## Soup Plugin Callbacks
|
||||
|
||||
Register a plugin once via the v0.45.0 registry API; v0.53.6 wires it into every
|
||||
transformer-backend trainer as a real HF `TrainerCallback`:
|
||||
|
||||
```python
|
||||
# soup_cli/plugins/my_plugin.py — auto-discovered at `soup` startup
|
||||
from soup_cli.plugins import register_plugin
|
||||
|
||||
class MyPlugin:
|
||||
def pre_train(self, ctx):
|
||||
print("training about to start, args =", ctx["args"])
|
||||
|
||||
def post_step(self, ctx):
|
||||
if ctx["state"].global_step % 100 == 0:
|
||||
print(f"step {ctx['state'].global_step}")
|
||||
|
||||
register_plugin(name="my-plugin", version="0.1.0", plugin=MyPlugin())
|
||||
```
|
||||
|
||||
A misbehaving plugin hook is swallowed at WARNING — one bad plugin must never crash
|
||||
a multi-hour training run. The hook snapshot is taken at callback-construction time,
|
||||
so a plugin registered MID-run does not retroactively receive events.
|
||||
|
||||
## Anthropic `/v1/messages` API
|
||||
|
||||
`soup serve --backend transformers` exposes a POST `/v1/messages` route that accepts
|
||||
Anthropic Messages-shaped payloads:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/messages -H "Content-Type: application/json" -d '{
|
||||
"model": "my-model",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
"max_tokens": 64
|
||||
}'
|
||||
```
|
||||
|
||||
Streaming (`stream: true`) returns 501 — Anthropic event-shape SSE ships in v0.53.7.
|
||||
Validation errors return a generic `"Invalid request"` 400 body; details are logged
|
||||
server-side at DEBUG. vLLM parity tracked for v0.53.7.
|
||||
|
||||
## N-gram Speculative Decoding
|
||||
|
||||
When a server is configured with an `NgramSpecConfig`, every chat completion forwards
|
||||
`prompt_lookup_num_tokens=N` into `model.generate(...)` (HF Transformers ≥ 4.38
|
||||
prompt-lookup decoding — no draft model required). Mutually exclusive with a real
|
||||
`assistant_model`; if both are set, the real draft model wins.
|
||||
|
||||
## Server-Side Tool Endpoints (preview)
|
||||
|
||||
Three POST routes ship in v0.53.6 as schema-only stubs returning HTTP 501:
|
||||
|
||||
- `/v1/tools/python` — sandboxed Python execution (v0.53.7 live)
|
||||
- `/v1/tools/bash` — sandboxed bash (v0.53.7 live)
|
||||
- `/v1/tools/web_search` — domain-allowlisted web search (v0.53.7 live)
|
||||
|
||||
Live wiring re-uses the v0.25.0 RLVR sandbox for python/bash and enforces
|
||||
`WebSearchConfig.domain_allowlist` for web_search.
|
||||
|
||||
## Data Recipe DAG Runner (preview)
|
||||
|
||||
`soup data recipe path/to/recipe.yaml --execute --output ./out` validates the DAG,
|
||||
enforces cwd-containment on `--output` at the CLI boundary, and surfaces the
|
||||
`v0.53.7` `NotImplementedError` marker from `run_recipe`. The per-node execution
|
||||
loop (seed / llm_text / code / judge / validator / sampler) plus checkpoint/resume
|
||||
ships in v0.53.7. Validate today, run tomorrow.
|
||||
|
||||
## Changelog
|
||||
|
||||
See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.53.5"
|
||||
version = "0.53.6"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.53.5"
|
||||
__version__ = "0.53.6"
|
||||
|
|
|
|||
|
|
@ -2154,6 +2154,20 @@ def demo_bundle(
|
|||
@app.command(name="recipe")
|
||||
def recipe(
|
||||
path: str = typer.Argument(..., help="Path to recipe.yaml under cwd"),
|
||||
execute: bool = typer.Option(
|
||||
False,
|
||||
"--execute",
|
||||
help=(
|
||||
"Run the validated DAG end-to-end (v0.53.6 #106 — stub; "
|
||||
"live per-node execution deferred to v0.53.7)."
|
||||
),
|
||||
),
|
||||
output: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--output",
|
||||
"-o",
|
||||
help="Output dir for sampler node (required with --execute).",
|
||||
),
|
||||
) -> None:
|
||||
"""v0.45.0 Part E — Validate a Data Recipe DAG (live runner deferred)."""
|
||||
from rich.markup import escape as _escape
|
||||
|
|
@ -2177,6 +2191,39 @@ def recipe(
|
|||
"Topological order: "
|
||||
+ ", ".join(_escape(name) for name in dag.topo_order)
|
||||
)
|
||||
|
||||
if execute:
|
||||
# `is None` guard — empty-string `--output ""` is a distinct
|
||||
# operator error that should NOT be silently mapped to "missing"
|
||||
# (matches v0.40.6 project policy on `is None` over falsy).
|
||||
if output is None:
|
||||
console.print(
|
||||
"[red]--execute requires --output <dir>[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
# Defence-in-depth: enforce cwd containment at the CLI boundary
|
||||
# BEFORE handing off to run_recipe. Today run_recipe is a stub,
|
||||
# so this only protects against future v0.53.7 live-runner bugs —
|
||||
# the docstring contract on run_recipe.output_dir says
|
||||
# cwd-contained, and we never want the live runner to be the
|
||||
# first/only enforcement point.
|
||||
from soup_cli.utils.paths import is_under_cwd
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
if not output or not is_under_cwd(output):
|
||||
console.print(
|
||||
"[red]--output must be a non-empty path under the current directory[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
|
||||
try:
|
||||
run_recipe(dag, output_dir=output)
|
||||
except NotImplementedError as exc:
|
||||
console.print(f"[yellow]{_escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
return
|
||||
|
||||
console.print(
|
||||
"[yellow]Live runner deferred to v0.45.1.[/]"
|
||||
"[yellow]Live runner deferred to v0.53.7 "
|
||||
"(re-run with --execute once v0.53.7 ships).[/]"
|
||||
)
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ import re
|
|||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
|
|
@ -810,6 +810,7 @@ def _generate_response(
|
|||
assistant_model=None,
|
||||
num_assistant_tokens: int = 5,
|
||||
logits_processor=None,
|
||||
ngram_config: Any = None,
|
||||
):
|
||||
"""Generate a response from the model."""
|
||||
import torch
|
||||
|
|
@ -854,6 +855,21 @@ def _generate_response(
|
|||
# v0.33.0 #53 — structured-output LogitsProcessor list (may be empty).
|
||||
if logits_processor:
|
||||
gen_kwargs["logits_processor"] = logits_processor
|
||||
# v0.53.6 #104 — n-gram speculative decoding (transformers backend).
|
||||
# Mutually exclusive with a real draft `assistant_model`.
|
||||
if ngram_config is not None and assistant_model is None:
|
||||
# HF Transformers >= 4.38 supports prompt-lookup decoding via
|
||||
# `prompt_lookup_num_tokens`. We expose `num_draft_tokens` as
|
||||
# the user-facing knob; n-gram size + prompt_lookup_max are
|
||||
# validated upstream by `validate_ngram_config`.
|
||||
try:
|
||||
gen_kwargs["prompt_lookup_num_tokens"] = int(
|
||||
ngram_config.num_draft_tokens
|
||||
)
|
||||
except (TypeError, AttributeError):
|
||||
# Schema gate at construction time enforces shape; this
|
||||
# is defence-in-depth.
|
||||
pass
|
||||
|
||||
outputs = model.generate(**gen_kwargs)
|
||||
|
||||
|
|
@ -879,6 +895,7 @@ def _create_app(
|
|||
enable_dashboard: bool = False,
|
||||
tracer=None,
|
||||
trace_log_writer=None,
|
||||
ngram_config: Any = None,
|
||||
):
|
||||
"""Create the FastAPI application with OpenAI-compatible endpoints."""
|
||||
import threading as _threading
|
||||
|
|
@ -1067,6 +1084,7 @@ def _create_app(
|
|||
assistant_model=draft_model,
|
||||
num_assistant_tokens=num_speculative_tokens,
|
||||
logits_processor=processors or None,
|
||||
ngram_config=ngram_config,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Generation error")
|
||||
|
|
@ -1121,6 +1139,89 @@ def _create_app(
|
|||
# error paths (prevents blind spots on the dashboard).
|
||||
metrics.record_latency((time.perf_counter() - started) * 1000)
|
||||
|
||||
# ----- v0.53.6 #102 — Anthropic /v1/messages route -----
|
||||
# Reuses the v0.45.0 utils/anthropic_messages converter + the existing
|
||||
# chat_completions handler. Live on transformers backend only this
|
||||
# release (vLLM /v1/messages tracked for v0.53.7).
|
||||
@app.post("/v1/messages")
|
||||
def anthropic_messages(payload: dict) -> dict:
|
||||
from soup_cli.utils.anthropic_messages import (
|
||||
from_anthropic,
|
||||
validate_anthropic_payload,
|
||||
)
|
||||
|
||||
# Streaming not yet supported on this route — v0.53.7 deliverable.
|
||||
# Checked BEFORE schema validation so a stream-only client never
|
||||
# leaks a validation-error detail (defence-in-depth).
|
||||
if isinstance(payload, dict) and payload.get("stream"):
|
||||
raise HTTPException(
|
||||
status_code=501,
|
||||
detail="Streaming /v1/messages deferred to v0.53.7.",
|
||||
)
|
||||
|
||||
try:
|
||||
validate_anthropic_payload(payload)
|
||||
openai_payload = from_anthropic(payload)
|
||||
request = ChatCompletionRequest(**openai_payload)
|
||||
except (TypeError, ValueError) as exc:
|
||||
# Security: do not echo internal validator/converter details
|
||||
# to the HTTP body. Log server-side for operator debugging.
|
||||
logger.debug("/v1/messages invalid request: %s", exc)
|
||||
raise HTTPException(status_code=400, detail="Invalid request")
|
||||
except Exception as exc: # noqa: BLE001 — pydantic ValidationError shape
|
||||
logger.debug("/v1/messages pydantic error: %s", exc)
|
||||
raise HTTPException(status_code=400, detail="Invalid request")
|
||||
|
||||
chat_response = chat_completions(request)
|
||||
|
||||
# Map OpenAI chat response back to Anthropic shape.
|
||||
text = ""
|
||||
if isinstance(chat_response, dict):
|
||||
try:
|
||||
text = chat_response["choices"][0]["message"]["content"]
|
||||
except (KeyError, IndexError, TypeError):
|
||||
text = ""
|
||||
usage = (
|
||||
chat_response.get("usage", {}) if isinstance(chat_response, dict) else {}
|
||||
)
|
||||
|
||||
return {
|
||||
"id": (
|
||||
chat_response.get("id", "") if isinstance(chat_response, dict) else ""
|
||||
),
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": openai_payload.get("model", model_name),
|
||||
"content": [{"type": "text", "text": text}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {
|
||||
"input_tokens": int(usage.get("prompt_tokens", 0) or 0),
|
||||
"output_tokens": int(usage.get("completion_tokens", 0) or 0),
|
||||
},
|
||||
}
|
||||
|
||||
# ----- v0.53.6 #103 — Server-side tool endpoints (deferred-live stubs) -----
|
||||
# Closed allowlist (mirrors v0.45.0 Part B utils/server_tools.SUPPORTED_TOOLS).
|
||||
# Routes return HTTP 501 with v0.53.7 marker — schema lives now so client
|
||||
# code can target the URLs even before the live sandbox is wired.
|
||||
def _tool_not_implemented(tool: str) -> None:
|
||||
raise HTTPException(
|
||||
status_code=501,
|
||||
detail=f"Server-side tool {tool!r} live execution deferred to v0.53.7.",
|
||||
)
|
||||
|
||||
@app.post("/v1/tools/python")
|
||||
def tool_python(_: dict) -> None:
|
||||
_tool_not_implemented("python")
|
||||
|
||||
@app.post("/v1/tools/bash")
|
||||
def tool_bash(_: dict) -> None:
|
||||
_tool_not_implemented("bash")
|
||||
|
||||
@app.post("/v1/tools/web_search")
|
||||
def tool_web_search(_: dict) -> None:
|
||||
_tool_not_implemented("web_search")
|
||||
|
||||
# Expose dashboard intent + constraint on the app for tests + introspection
|
||||
app.state.enable_dashboard = enable_dashboard
|
||||
app.state.output_constraint = output_constraint
|
||||
|
|
|
|||
|
|
@ -0,0 +1,123 @@
|
|||
"""v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
|
||||
Bridges :mod:`soup_cli.plugins` registered hooks into the HF Trainer
|
||||
callback surface. For every enabled plugin (via :func:`list_plugins` +
|
||||
``spec.enabled``), discovers implemented hooks via :func:`discover_hooks`
|
||||
and dispatches the matching trainer event.
|
||||
|
||||
Per-plugin hook exceptions are swallowed at WARNING level — one
|
||||
misbehaving plugin must not crash a multi-hour training run. This
|
||||
mirrors the v0.44.0 / v0.45.0 plugin loader policy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _collect_active_hooks() -> list[tuple[str, dict[str, Any]]]:
|
||||
"""Return ``[(plugin_name, hook_map), ...]`` for every enabled plugin.
|
||||
|
||||
Lazy-imports :mod:`soup_cli.plugins` so the callback module stays
|
||||
cheap to import in CI / non-training contexts.
|
||||
"""
|
||||
from soup_cli.plugins import discover_hooks, list_plugins
|
||||
|
||||
out: list[tuple[str, dict[str, Any]]] = []
|
||||
for name, spec in list_plugins().items():
|
||||
if not spec.enabled:
|
||||
continue
|
||||
hooks = discover_hooks(spec.plugin)
|
||||
if hooks:
|
||||
out.append((name, hooks))
|
||||
return out
|
||||
|
||||
|
||||
def _safe_invoke(
|
||||
plugin_name: str, hook_name: str, hook: Any, ctx: dict[str, Any]
|
||||
) -> None:
|
||||
try:
|
||||
hook(ctx)
|
||||
except Exception: # noqa: BLE001 — plugin failure must not crash training
|
||||
logger.warning(
|
||||
"Plugin %r hook %r raised; continuing",
|
||||
plugin_name,
|
||||
hook_name,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
|
||||
def _build_callback_class() -> type:
|
||||
"""Construct the ``SoupPluginCallback`` class with transformers as parent.
|
||||
|
||||
Lazy-imports :mod:`transformers` so the wiring helper can be imported
|
||||
in CI without the heavy dep installed.
|
||||
"""
|
||||
from transformers import TrainerCallback
|
||||
|
||||
class SoupPluginCallback(TrainerCallback):
|
||||
"""Fans HF Trainer events out to every enabled Soup plugin."""
|
||||
|
||||
def __init__(
|
||||
self, hooks: list[tuple[str, dict[str, Any]]] | None = None
|
||||
) -> None:
|
||||
super().__init__()
|
||||
# Snapshot at construction time so a plugin registered MID-run
|
||||
# does not silently start receiving hooks halfway through. The
|
||||
# caller may pre-collect hooks (the ``build_plugin_callback``
|
||||
# path) to avoid a redundant registry scan + close a tiny
|
||||
# race-window between "is any plugin enabled?" and
|
||||
# "snapshot the registry".
|
||||
self._hooks = (
|
||||
list(hooks) if hooks is not None else _collect_active_hooks()
|
||||
)
|
||||
|
||||
def _dispatch(self, hook_name: str, context: dict[str, Any]) -> None:
|
||||
for plugin_name, hooks in self._hooks:
|
||||
hook = hooks.get(hook_name)
|
||||
if hook is None:
|
||||
continue
|
||||
_safe_invoke(plugin_name, hook_name, hook, context)
|
||||
|
||||
def on_train_begin(self, args, state, control, **kwargs): # noqa: D401
|
||||
self._dispatch(
|
||||
"pre_train", {"args": args, "state": state, "control": control}
|
||||
)
|
||||
|
||||
def on_train_end(self, args, state, control, **kwargs): # noqa: D401
|
||||
self._dispatch(
|
||||
"post_train", {"args": args, "state": state, "control": control}
|
||||
)
|
||||
|
||||
def on_step_begin(self, args, state, control, **kwargs): # noqa: D401
|
||||
self._dispatch(
|
||||
"pre_step", {"args": args, "state": state, "control": control}
|
||||
)
|
||||
|
||||
def on_step_end(self, args, state, control, **kwargs): # noqa: D401
|
||||
self._dispatch(
|
||||
"post_step", {"args": args, "state": state, "control": control}
|
||||
)
|
||||
|
||||
return SoupPluginCallback
|
||||
|
||||
|
||||
def build_plugin_callback() -> Any:
|
||||
"""Return a new ``SoupPluginCallback`` instance, or ``None`` if no
|
||||
enabled plugins implement any hook (no-op short-circuit).
|
||||
|
||||
Collects hooks ONCE and passes the snapshot to the callback so a
|
||||
plugin registered between the "is empty?" check and the constructor
|
||||
cannot silently slip into the active hook list — review fix.
|
||||
"""
|
||||
hooks = _collect_active_hooks()
|
||||
if not hooks:
|
||||
return None
|
||||
callback_cls = _build_callback_class()
|
||||
return callback_cls(hooks)
|
||||
|
||||
|
||||
__all__ = ["build_plugin_callback"]
|
||||
|
|
@ -207,11 +207,14 @@ class BCOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -379,11 +379,14 @@ class DistillTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -161,11 +161,14 @@ class DPOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps).
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
# v0.53.2 #135 — GDPO loss hook (no-op if gdpo_variant unset).
|
||||
from soup_cli.utils.ebft_gdpo import attach_gdpo_compute_loss
|
||||
|
|
|
|||
|
|
@ -182,11 +182,14 @@ class EmbeddingTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -236,11 +236,14 @@ class GRPOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps).
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -161,11 +161,14 @@ class IPOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -157,11 +157,14 @@ class KTOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps).
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -158,11 +158,14 @@ class ORPOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -258,11 +258,14 @@ class PPOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps).
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
self._train_ds = train_ds
|
||||
|
|
|
|||
|
|
@ -215,11 +215,14 @@ class PretrainTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback (magnitude-prune LoRA every N steps).
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -159,11 +159,14 @@ class RewardModelTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -854,6 +854,7 @@ class SFTTrainerWrapper:
|
|||
# ReLoRA callback (v0.39.0 Part B / v0.40.6 #67) via shared helper.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, self.config.training)
|
||||
|
|
@ -861,6 +862,8 @@ class SFTTrainerWrapper:
|
|||
attach_curriculum_callback(
|
||||
self.trainer, self.config.training, self._output_dir, console
|
||||
)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
# v0.53.2 #135 — EBFT compute_loss hook (no-op if ebft_variant unset).
|
||||
from soup_cli.utils.ebft_gdpo import attach_ebft_compute_loss
|
||||
|
|
|
|||
|
|
@ -161,11 +161,14 @@ class SimPOTrainerWrapper:
|
|||
# v0.40.6 #67 — ReLoRA callback.
|
||||
from soup_cli.utils.peft_wiring import (
|
||||
attach_curriculum_callback,
|
||||
attach_plugin_callback,
|
||||
attach_relora_callback,
|
||||
)
|
||||
attach_relora_callback(self.trainer, tcfg)
|
||||
# v0.53.5 #114/#115 — dynamic curriculum live callback.
|
||||
attach_curriculum_callback(self.trainer, tcfg, str(output_dir), console)
|
||||
# v0.53.6 #101 — Soup plugin TrainerCallback.
|
||||
attach_plugin_callback(self.trainer, console)
|
||||
|
||||
self._output_dir = str(output_dir)
|
||||
|
||||
|
|
|
|||
|
|
@ -129,3 +129,45 @@ def attach_curriculum_callback(
|
|||
except Exception: # noqa: BLE001 — never crash on console issues.
|
||||
pass
|
||||
return True
|
||||
|
||||
|
||||
def attach_plugin_callback(trainer: Any, console: Any = None) -> bool:
|
||||
"""Attach :class:`SoupPluginCallback` when any enabled plugin implements a hook.
|
||||
|
||||
Returns ``True`` when a callback was attached, ``False`` otherwise
|
||||
(no plugins enabled OR none implement any hook — the build helper
|
||||
short-circuits to ``None`` in that case so the trainer pays zero
|
||||
overhead).
|
||||
|
||||
Failures inside individual plugin hooks are swallowed at WARNING
|
||||
inside the callback itself; this helper only handles the
|
||||
construction failure path (transformers not importable / plugin
|
||||
registry corrupted).
|
||||
"""
|
||||
try:
|
||||
from soup_cli.monitoring.plugin_callback import build_plugin_callback
|
||||
|
||||
callback = build_plugin_callback()
|
||||
except Exception as exc: # noqa: BLE001 — plugin infra must not crash training
|
||||
logger.debug("attach_plugin_callback skipped: %s", exc)
|
||||
return False
|
||||
if callback is None:
|
||||
return False
|
||||
try:
|
||||
trainer.add_callback(callback)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.debug("attach_plugin_callback add_callback failed: %s", exc)
|
||||
return False
|
||||
if console is not None:
|
||||
try:
|
||||
# Number of plugins is the count of distinct (plugin_name, hooks)
|
||||
# pairs the callback snapshot will fan out to.
|
||||
from soup_cli.plugins import list_plugins
|
||||
|
||||
n_enabled = sum(1 for s in list_plugins().values() if s.enabled)
|
||||
console.print(
|
||||
f"[dim]Plugin callback attached ({n_enabled} enabled plugin(s)).[/]"
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return True
|
||||
|
|
|
|||
|
|
@ -0,0 +1,69 @@
|
|||
"""v0.53.6 #106 — Data Recipe DAG runner (stub-then-live).
|
||||
|
||||
Schema-only this release. Per-node-kind handlers (seed / llm_text / code /
|
||||
judge / validator / sampler), checkpoint-between-nodes, resume-on-failure
|
||||
+ ``soup data recipe --execute`` wire-up land in v0.53.7. Mirrors the
|
||||
project stub-then-live pattern (v0.27.0 MII / v0.37.0 multipack /
|
||||
v0.50.0 GRPO Plus).
|
||||
|
||||
The validator surface ships now so callers can target the schema and
|
||||
type-check against ``run_recipe`` before live execution exists.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any, Mapping
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover — type-only import.
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG
|
||||
|
||||
|
||||
def run_recipe(
|
||||
dag: "RecipeDAG",
|
||||
*,
|
||||
output_dir: str,
|
||||
checkpoint_dir: str | None = None,
|
||||
resume: bool = False,
|
||||
judge_provider: str | None = None,
|
||||
judge_model: str | None = None,
|
||||
) -> Mapping[str, Any]:
|
||||
"""Execute a validated :class:`RecipeDAG` end-to-end.
|
||||
|
||||
Deferred-live stub. Per-node-kind handlers + checkpoint/resume +
|
||||
``--execute`` plumbing land in v0.53.7. Schema-only contract:
|
||||
|
||||
- ``dag``: a :class:`soup_cli.utils.recipe_dag.RecipeDAG` (validated)
|
||||
- ``output_dir``: cwd-contained directory for sampler output JSONL
|
||||
- ``checkpoint_dir``: optional intermediate-node checkpoint dir
|
||||
- ``resume``: if True, skip nodes whose checkpoint already exists
|
||||
- ``judge_provider`` / ``judge_model``: routed into the v0.40.3
|
||||
``JudgeEvaluator`` for ``judge`` nodes
|
||||
|
||||
Raises:
|
||||
TypeError: if ``dag`` is not a ``RecipeDAG``.
|
||||
NotImplementedError: always — live runner ships in v0.53.7.
|
||||
"""
|
||||
# Late import so test code can patch the module without forcing a
|
||||
# heavy recipe_dag import at module load.
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG
|
||||
|
||||
if not isinstance(dag, RecipeDAG):
|
||||
raise TypeError("dag must be a RecipeDAG")
|
||||
if not isinstance(output_dir, str) or not output_dir:
|
||||
raise TypeError("output_dir must be a non-empty string")
|
||||
if checkpoint_dir is not None and not isinstance(checkpoint_dir, str):
|
||||
raise TypeError("checkpoint_dir must be a string or None")
|
||||
if not isinstance(resume, bool):
|
||||
raise TypeError("resume must be a bool")
|
||||
if judge_provider is not None and not isinstance(judge_provider, str):
|
||||
raise TypeError("judge_provider must be a string or None")
|
||||
if judge_model is not None and not isinstance(judge_model, str):
|
||||
raise TypeError("judge_model must be a string or None")
|
||||
raise NotImplementedError(
|
||||
"Data Recipe DAG runner is deferred to v0.53.7. The schema + "
|
||||
"validator surface in soup_cli.utils.recipe_dag is live; only the "
|
||||
"per-node execution loop is missing."
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["run_recipe"]
|
||||
|
|
@ -11,7 +11,7 @@ from __future__ import annotations
|
|||
import re
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping, Optional, Sequence, Tuple
|
||||
from typing import Any, Mapping, Optional, Sequence, Tuple
|
||||
|
||||
_PLUGIN_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_]{0,31}$")
|
||||
_MAX_DESCRIPTION = 256
|
||||
|
|
@ -128,9 +128,33 @@ def validate_trainer_plugin_list(names: Sequence[str]) -> Tuple[str, ...]:
|
|||
return tuple(out)
|
||||
|
||||
|
||||
def instantiate_trainer_plugins(names: Sequence[str]) -> Tuple[Any, ...]:
|
||||
"""v0.53.6 #105 — instantiate live upstream callbacks (stub-then-live).
|
||||
|
||||
Validates ``names`` against :func:`validate_trainer_plugin_list` then
|
||||
raises :class:`NotImplementedError` with a v0.53.7 marker. Live lazy
|
||||
imports + per-plugin callback construction (``grokfast``,
|
||||
``spectrum``, ``llmcompressor``, ``sonicmoe``, ``cce_plugin``,
|
||||
``math_verify``) land in v0.53.7. Same stub-then-live pattern as
|
||||
v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro.
|
||||
|
||||
Raises:
|
||||
TypeError: per :func:`validate_trainer_plugin_list`.
|
||||
ValueError: per :func:`validate_trainer_plugin_list`.
|
||||
NotImplementedError: always (after validation) — live wiring
|
||||
ships in v0.53.7.
|
||||
"""
|
||||
canonical = validate_trainer_plugin_list(names)
|
||||
raise NotImplementedError(
|
||||
f"Trainer-plugin live instantiation deferred to v0.53.7. "
|
||||
f"Validated names: {canonical!r}"
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"TrainerPluginSpec",
|
||||
"list_trainer_plugins",
|
||||
"get_trainer_plugin",
|
||||
"validate_trainer_plugin_list",
|
||||
"instantiate_trainer_plugins",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,799 @@
|
|||
"""v0.53.6 — Plugin + Agent + Anthropic API.
|
||||
|
||||
Covers:
|
||||
- #101: SoupPluginCallback + attach_plugin_callback wired into 13 trainers.
|
||||
- #102: Anthropic /v1/messages route on transformers backend.
|
||||
- #104: n-gram speculative decoding wiring (prompt_lookup_num_tokens).
|
||||
- #103: server-side tool endpoints — deferred-live stubs returning 501.
|
||||
- #105: utils/trainer_plugins.instantiate_trainer_plugins — stub.
|
||||
- #106: utils/recipe_run.run_recipe — stub + `soup data recipe --execute`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# #101 — SoupPluginCallback + attach_plugin_callback
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
class _RecordingPlugin:
|
||||
def __init__(self) -> None:
|
||||
self.events: list[str] = []
|
||||
|
||||
def pre_train(self, _ctx): # noqa: D401
|
||||
self.events.append("pre_train")
|
||||
|
||||
def post_step(self, _ctx): # noqa: D401
|
||||
self.events.append("post_step")
|
||||
|
||||
|
||||
class _RaisingPlugin:
|
||||
def pre_train(self, _ctx): # noqa: D401
|
||||
raise RuntimeError("plugin boom")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clear_plugins_fixture():
|
||||
from soup_cli.plugins import clear_plugins
|
||||
|
||||
clear_plugins()
|
||||
yield
|
||||
clear_plugins()
|
||||
|
||||
|
||||
def test_build_plugin_callback_none_when_no_plugins(clear_plugins_fixture):
|
||||
"""No registered plugins → build returns None (no-op short-circuit)."""
|
||||
from soup_cli.monitoring.plugin_callback import build_plugin_callback
|
||||
|
||||
assert build_plugin_callback() is None
|
||||
|
||||
|
||||
def test_build_plugin_callback_skips_disabled(clear_plugins_fixture):
|
||||
"""Disabled plugin → build returns None (no enabled hooks)."""
|
||||
from soup_cli.monitoring.plugin_callback import build_plugin_callback
|
||||
from soup_cli.plugins import disable_plugin, register_plugin
|
||||
|
||||
plugin = _RecordingPlugin()
|
||||
register_plugin(name="rec-a", version="0.1.0", plugin=plugin)
|
||||
disable_plugin("rec-a")
|
||||
assert build_plugin_callback() is None
|
||||
|
||||
|
||||
def test_build_plugin_callback_returns_callback_when_enabled(
|
||||
clear_plugins_fixture,
|
||||
):
|
||||
"""Enabled plugin with at least one hook → real TrainerCallback."""
|
||||
transformers = pytest.importorskip("transformers")
|
||||
from soup_cli.monitoring.plugin_callback import build_plugin_callback
|
||||
from soup_cli.plugins import register_plugin
|
||||
|
||||
plugin = _RecordingPlugin()
|
||||
register_plugin(name="rec-b", version="0.1.0", plugin=plugin)
|
||||
callback = build_plugin_callback()
|
||||
assert callback is not None
|
||||
assert isinstance(callback, transformers.TrainerCallback)
|
||||
|
||||
|
||||
def test_plugin_callback_dispatches_to_implemented_hooks(
|
||||
clear_plugins_fixture,
|
||||
):
|
||||
"""Trainer event → only implemented hooks fire."""
|
||||
pytest.importorskip("transformers")
|
||||
from soup_cli.monitoring.plugin_callback import build_plugin_callback
|
||||
from soup_cli.plugins import register_plugin
|
||||
|
||||
plugin = _RecordingPlugin()
|
||||
register_plugin(name="rec-c", version="0.1.0", plugin=plugin)
|
||||
callback = build_plugin_callback()
|
||||
assert callback is not None
|
||||
args = MagicMock()
|
||||
state = MagicMock()
|
||||
control = MagicMock()
|
||||
|
||||
callback.on_train_begin(args, state, control)
|
||||
callback.on_step_end(args, state, control)
|
||||
callback.on_step_begin(args, state, control) # plugin has no pre_step
|
||||
callback.on_train_end(args, state, control) # plugin has no post_train
|
||||
assert plugin.events == ["pre_train", "post_step"]
|
||||
|
||||
|
||||
def test_plugin_callback_swallows_hook_exceptions(
|
||||
clear_plugins_fixture, caplog
|
||||
):
|
||||
"""One misbehaving plugin must not crash training."""
|
||||
pytest.importorskip("transformers")
|
||||
from soup_cli.monitoring.plugin_callback import build_plugin_callback
|
||||
from soup_cli.plugins import register_plugin
|
||||
|
||||
register_plugin(name="bad-plugin", version="0.1.0", plugin=_RaisingPlugin())
|
||||
callback = build_plugin_callback()
|
||||
assert callback is not None
|
||||
caplog.clear() # review fix — defend against accumulated log records
|
||||
with caplog.at_level("WARNING"):
|
||||
callback.on_train_begin(MagicMock(), MagicMock(), MagicMock())
|
||||
# Did not raise; recorded a WARNING for this specific plugin.
|
||||
matching = [
|
||||
rec
|
||||
for rec in caplog.records
|
||||
if rec.levelname == "WARNING" and "bad-plugin" in rec.message
|
||||
]
|
||||
assert matching, "expected WARNING for bad-plugin hook failure"
|
||||
|
||||
|
||||
def test_attach_plugin_callback_no_plugins_returns_false(
|
||||
clear_plugins_fixture,
|
||||
):
|
||||
"""No plugins registered → helper short-circuits to False (no trainer touch)."""
|
||||
from soup_cli.utils.peft_wiring import attach_plugin_callback
|
||||
|
||||
trainer = MagicMock()
|
||||
attached = attach_plugin_callback(trainer)
|
||||
assert attached is False
|
||||
trainer.add_callback.assert_not_called()
|
||||
|
||||
|
||||
def test_attach_plugin_callback_attaches_when_enabled(
|
||||
clear_plugins_fixture,
|
||||
):
|
||||
"""Enabled plugin → trainer.add_callback called once with the callback."""
|
||||
pytest.importorskip("transformers")
|
||||
from soup_cli.plugins import register_plugin
|
||||
from soup_cli.utils.peft_wiring import attach_plugin_callback
|
||||
|
||||
register_plugin(name="rec-attach", version="0.1.0", plugin=_RecordingPlugin())
|
||||
trainer = MagicMock()
|
||||
attached = attach_plugin_callback(trainer)
|
||||
assert attached is True
|
||||
trainer.add_callback.assert_called_once()
|
||||
|
||||
|
||||
def test_attach_plugin_callback_swallows_add_callback_failure(
|
||||
clear_plugins_fixture,
|
||||
):
|
||||
"""trainer.add_callback raising → helper returns False, no crash."""
|
||||
pytest.importorskip("transformers")
|
||||
from soup_cli.plugins import register_plugin
|
||||
from soup_cli.utils.peft_wiring import attach_plugin_callback
|
||||
|
||||
register_plugin(name="rec-fail", version="0.1.0", plugin=_RecordingPlugin())
|
||||
trainer = MagicMock()
|
||||
trainer.add_callback.side_effect = RuntimeError("add_callback broken")
|
||||
attached = attach_plugin_callback(trainer)
|
||||
assert attached is False
|
||||
|
||||
|
||||
def test_attach_plugin_callback_console_advisory(clear_plugins_fixture):
|
||||
"""Optional `console` argument prints an advisory; never crashes."""
|
||||
pytest.importorskip("transformers")
|
||||
from soup_cli.plugins import register_plugin
|
||||
from soup_cli.utils.peft_wiring import attach_plugin_callback
|
||||
|
||||
register_plugin(name="rec-console", version="0.1.0", plugin=_RecordingPlugin())
|
||||
trainer = MagicMock()
|
||||
console = MagicMock()
|
||||
attached = attach_plugin_callback(trainer, console)
|
||||
assert attached is True
|
||||
console.print.assert_called_once()
|
||||
msg = console.print.call_args.args[0]
|
||||
assert "1 enabled" in msg
|
||||
|
||||
|
||||
def test_attach_plugin_callback_console_print_failure_swallowed(
|
||||
clear_plugins_fixture,
|
||||
):
|
||||
"""`console.print` raising must not crash the helper."""
|
||||
pytest.importorskip("transformers")
|
||||
from soup_cli.plugins import register_plugin
|
||||
from soup_cli.utils.peft_wiring import attach_plugin_callback
|
||||
|
||||
register_plugin(name="rec-console2", version="0.1.0", plugin=_RecordingPlugin())
|
||||
trainer = MagicMock()
|
||||
console = MagicMock()
|
||||
console.print.side_effect = RuntimeError("console broken")
|
||||
attached = attach_plugin_callback(trainer, console)
|
||||
assert attached is True # callback still attached even though print failed
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"trainer_file",
|
||||
[
|
||||
"soup_cli/trainer/sft.py",
|
||||
"soup_cli/trainer/dpo.py",
|
||||
"soup_cli/trainer/grpo.py",
|
||||
"soup_cli/trainer/kto.py",
|
||||
"soup_cli/trainer/orpo.py",
|
||||
"soup_cli/trainer/simpo.py",
|
||||
"soup_cli/trainer/ipo.py",
|
||||
"soup_cli/trainer/bco.py",
|
||||
"soup_cli/trainer/ppo.py",
|
||||
"soup_cli/trainer/pretrain.py",
|
||||
"soup_cli/trainer/reward_model.py",
|
||||
"soup_cli/trainer/embedding.py",
|
||||
"soup_cli/trainer/distill.py",
|
||||
],
|
||||
)
|
||||
def test_every_trainer_wires_attach_plugin_callback(trainer_file: str):
|
||||
"""Source-level invariant: every transformer-backend trainer wires the helper."""
|
||||
repo_root = Path(__file__).resolve().parent.parent
|
||||
src = (repo_root / trainer_file).read_text(encoding="utf-8")
|
||||
assert "attach_plugin_callback" in src, (
|
||||
f"{trainer_file} is missing attach_plugin_callback wiring"
|
||||
)
|
||||
# Direct import from canonical peft_wiring module (no re-export shim).
|
||||
assert "attach_plugin_callback," in src or (
|
||||
"from soup_cli.utils.peft_wiring import" in src
|
||||
and "attach_plugin_callback" in src
|
||||
), f"{trainer_file} must import attach_plugin_callback from peft_wiring"
|
||||
# Sanity: imported AND called on self.trainer.
|
||||
assert "attach_plugin_callback(self.trainer" in src, (
|
||||
f"{trainer_file} imports but never invokes attach_plugin_callback"
|
||||
)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# #102 — Anthropic /v1/messages route
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _fake_model_and_tokenizer():
|
||||
"""Build minimal mocks so _create_app can construct without HF deps."""
|
||||
model = MagicMock()
|
||||
tokenizer = MagicMock()
|
||||
tokenizer.pad_token_id = 0
|
||||
return model, tokenizer
|
||||
|
||||
|
||||
def _build_app(**overrides):
|
||||
pytest.importorskip("fastapi")
|
||||
from soup_cli.commands.serve import _create_app
|
||||
|
||||
model, tokenizer = _fake_model_and_tokenizer()
|
||||
kwargs = dict(
|
||||
model_obj=model,
|
||||
tokenizer=tokenizer,
|
||||
device="cpu",
|
||||
model_name="test-model",
|
||||
max_tokens_default=64,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return _create_app(**kwargs)
|
||||
|
||||
|
||||
def test_anthropic_messages_route_registered():
|
||||
"""POST /v1/messages must be present on the FastAPI app."""
|
||||
app = _build_app()
|
||||
routes = {(r.path, tuple(sorted(r.methods))) for r in app.routes if hasattr(r, "methods")}
|
||||
assert ("/v1/messages", ("POST",)) in routes
|
||||
|
||||
|
||||
def test_anthropic_messages_rejects_malformed_payload(monkeypatch):
|
||||
"""Schema validation propagates to HTTP 400 — no internal crash."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
app = _build_app()
|
||||
client = TestClient(app)
|
||||
# Missing required `messages` field.
|
||||
response = client.post("/v1/messages", json={"model": "x", "max_tokens": 16})
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
def test_anthropic_messages_rejects_streaming():
|
||||
"""`stream=True` returns 501 (deferred to v0.53.7)."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
app = _build_app()
|
||||
client = TestClient(app)
|
||||
payload = {
|
||||
"model": "test-model",
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"max_tokens": 16,
|
||||
"stream": True,
|
||||
}
|
||||
response = client.post("/v1/messages", json=payload)
|
||||
assert response.status_code == 501
|
||||
assert "v0.53.7" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_anthropic_messages_happy_path(monkeypatch):
|
||||
"""End-to-end: payload → from_anthropic → chat_completions mock → Anthropic shape."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
# Patch _generate_response so we don't need a real model.
|
||||
monkeypatch.setattr(
|
||||
"soup_cli.commands.serve._generate_response",
|
||||
lambda *a, **kw: ("hello world", 3, 2),
|
||||
)
|
||||
app = _build_app()
|
||||
client = TestClient(app)
|
||||
payload = {
|
||||
"model": "test-model",
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"max_tokens": 16,
|
||||
}
|
||||
response = client.post("/v1/messages", json=payload)
|
||||
assert response.status_code == 200, response.text
|
||||
body = response.json()
|
||||
assert body["type"] == "message"
|
||||
assert body["role"] == "assistant"
|
||||
assert body["content"] == [{"type": "text", "text": "hello world"}]
|
||||
assert body["model"] == "test-model"
|
||||
assert body["stop_reason"] == "end_turn"
|
||||
assert body["usage"]["input_tokens"] == 3
|
||||
assert body["usage"]["output_tokens"] == 2
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# #104 — n-gram speculative decoding wiring
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ngram_config_threaded_into_generate_response(monkeypatch):
|
||||
"""`_create_app(ngram_config=...)` forwards through to `_generate_response`."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from soup_cli.utils.ngram_spec import NgramSpecConfig
|
||||
|
||||
received_kwargs: dict = {}
|
||||
|
||||
def _capture(*args, **kwargs):
|
||||
received_kwargs.update(kwargs)
|
||||
return ("ok", 1, 1)
|
||||
|
||||
monkeypatch.setattr("soup_cli.commands.serve._generate_response", _capture)
|
||||
cfg = NgramSpecConfig(n=3, num_draft_tokens=4, prompt_lookup_max=0)
|
||||
app = _build_app(ngram_config=cfg)
|
||||
client = TestClient(app)
|
||||
response = client.post(
|
||||
"/v1/chat/completions",
|
||||
json={
|
||||
"model": "test-model",
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"max_tokens": 8,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
assert received_kwargs.get("ngram_config") is cfg
|
||||
|
||||
|
||||
def test_ngram_kwarg_emits_prompt_lookup_num_tokens():
|
||||
"""Direct unit test on `_generate_response` proves the kwarg makes it
|
||||
into ``model.generate``."""
|
||||
pytest.importorskip("torch")
|
||||
from soup_cli.commands.serve import _generate_response
|
||||
from soup_cli.utils.ngram_spec import NgramSpecConfig
|
||||
|
||||
captured: dict = {}
|
||||
|
||||
class _FakeOutput:
|
||||
def __getitem__(self, _):
|
||||
class _T:
|
||||
shape = (0, 5)
|
||||
|
||||
def __getitem__(self, _):
|
||||
return []
|
||||
|
||||
return _T()
|
||||
|
||||
class _FakeModel:
|
||||
device = "cpu"
|
||||
|
||||
def generate(self, **kwargs):
|
||||
captured.update(kwargs)
|
||||
|
||||
class _Tensor:
|
||||
def __getitem__(self, _):
|
||||
return []
|
||||
|
||||
return [[0, 1, 2, 3, 4]]
|
||||
|
||||
class _FakeTokenizer:
|
||||
pad_token_id = 0
|
||||
chat_template = None # forces "Assistant:" fallback path
|
||||
|
||||
def __call__(self, _text, return_tensors=None):
|
||||
import torch as _torch
|
||||
|
||||
ids = _torch.tensor([[1, 2, 3]])
|
||||
return {"input_ids": ids, "attention_mask": _torch.ones_like(ids)}
|
||||
|
||||
def decode(self, _tokens, skip_special_tokens=True):
|
||||
return "hello"
|
||||
|
||||
cfg = NgramSpecConfig(n=3, num_draft_tokens=7, prompt_lookup_max=0)
|
||||
_generate_response(
|
||||
_FakeModel(),
|
||||
_FakeTokenizer(),
|
||||
[{"role": "user", "content": "hi"}],
|
||||
max_tokens=4,
|
||||
ngram_config=cfg,
|
||||
)
|
||||
assert captured.get("prompt_lookup_num_tokens") == 7
|
||||
|
||||
|
||||
def test_ngram_kwarg_skipped_when_draft_model_set():
|
||||
"""A real `assistant_model` wins over n-gram — keys mutually exclusive."""
|
||||
pytest.importorskip("torch")
|
||||
from soup_cli.commands.serve import _generate_response
|
||||
from soup_cli.utils.ngram_spec import NgramSpecConfig
|
||||
|
||||
captured: dict = {}
|
||||
|
||||
class _FakeModel:
|
||||
device = "cpu"
|
||||
|
||||
def generate(self, **kwargs):
|
||||
captured.update(kwargs)
|
||||
return [[0, 1, 2, 3]]
|
||||
|
||||
class _FakeTokenizer:
|
||||
pad_token_id = 0
|
||||
chat_template = None
|
||||
|
||||
def __call__(self, _text, return_tensors=None):
|
||||
import torch as _torch
|
||||
|
||||
ids = _torch.tensor([[1, 2]])
|
||||
return {"input_ids": ids, "attention_mask": _torch.ones_like(ids)}
|
||||
|
||||
def decode(self, _tokens, skip_special_tokens=True):
|
||||
return ""
|
||||
|
||||
cfg = NgramSpecConfig(n=3, num_draft_tokens=4)
|
||||
_generate_response(
|
||||
_FakeModel(),
|
||||
_FakeTokenizer(),
|
||||
[{"role": "user", "content": "x"}],
|
||||
max_tokens=2,
|
||||
assistant_model=object(), # truthy
|
||||
ngram_config=cfg,
|
||||
)
|
||||
assert "prompt_lookup_num_tokens" not in captured
|
||||
assert captured.get("assistant_model") is not None
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# #103 — Server-side tool endpoints (stub: 501)
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
["/v1/tools/python", "/v1/tools/bash", "/v1/tools/web_search"],
|
||||
)
|
||||
def test_tool_endpoint_returns_501(path: str):
|
||||
"""All three tool endpoints return 501 with v0.53.7 marker."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
app = _build_app()
|
||||
client = TestClient(app)
|
||||
response = client.post(path, json={"code": "print(1)"})
|
||||
assert response.status_code == 501
|
||||
assert "v0.53.7" in response.json()["detail"]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# #106 — recipe_run stub + `soup data recipe --execute`
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_run_recipe_rejects_non_dag():
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
with pytest.raises(TypeError, match="RecipeDAG"):
|
||||
run_recipe("not-a-dag", output_dir="out") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_run_recipe_rejects_bad_kwargs():
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
dag = RecipeDAG(
|
||||
nodes=(RecipeNode(name="a", kind="seed", config={}),),
|
||||
edges=(),
|
||||
topo_order=("a",),
|
||||
)
|
||||
with pytest.raises(TypeError, match="output_dir"):
|
||||
run_recipe(dag, output_dir=123) # type: ignore[arg-type]
|
||||
with pytest.raises(TypeError, match="resume"):
|
||||
run_recipe(dag, output_dir="out", resume="yes") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_run_recipe_raises_not_implemented():
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
dag = RecipeDAG(
|
||||
nodes=(RecipeNode(name="a", kind="seed", config={}),),
|
||||
edges=(),
|
||||
topo_order=("a",),
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.53.7"):
|
||||
run_recipe(dag, output_dir="out")
|
||||
|
||||
|
||||
def test_data_recipe_execute_flag_present(tmp_path: Path):
|
||||
"""`soup data recipe --help` lists `--execute`."""
|
||||
from soup_cli.cli import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli_app, ["data", "recipe", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "--execute" in result.output
|
||||
|
||||
|
||||
def test_data_recipe_execute_requires_output(tmp_path: Path, monkeypatch):
|
||||
"""`--execute` without `--output` exits 2 with a clear message."""
|
||||
from soup_cli.cli import app as cli_app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
Path("recipe.yaml").write_text(
|
||||
"nodes:\n"
|
||||
" - name: a\n"
|
||||
" kind: seed\n"
|
||||
"edges: []\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
["data", "recipe", "recipe.yaml", "--execute"],
|
||||
)
|
||||
assert result.exit_code == 2, result.output
|
||||
assert "--output" in result.output
|
||||
|
||||
|
||||
def test_data_recipe_execute_surfaces_v0537_marker(tmp_path: Path, monkeypatch):
|
||||
"""`--execute --output <dir>` runs through run_recipe and surfaces v0.53.7 marker."""
|
||||
from soup_cli.cli import app as cli_app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
Path("recipe.yaml").write_text(
|
||||
"nodes:\n"
|
||||
" - name: a\n"
|
||||
" kind: seed\n"
|
||||
"edges: []\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"data",
|
||||
"recipe",
|
||||
"recipe.yaml",
|
||||
"--execute",
|
||||
"--output",
|
||||
"out",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 2, result.output
|
||||
assert "v0.53.7" in result.output
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# #105 — instantiate_trainer_plugins stub
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_instantiate_trainer_plugins_validates_then_raises():
|
||||
"""Validation runs first, then NotImplementedError with v0.53.7 marker."""
|
||||
from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins
|
||||
|
||||
with pytest.raises(NotImplementedError, match="v0.53.7"):
|
||||
instantiate_trainer_plugins(["grokfast"])
|
||||
|
||||
|
||||
def test_instantiate_trainer_plugins_validation_runs_first():
|
||||
"""Unknown plugin name → ValueError BEFORE NotImplementedError."""
|
||||
from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins
|
||||
|
||||
with pytest.raises(ValueError, match="unknown trainer plugin"):
|
||||
instantiate_trainer_plugins(["definitely-not-a-real-plugin"])
|
||||
|
||||
|
||||
def test_instantiate_trainer_plugins_rejects_non_sequence():
|
||||
from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
instantiate_trainer_plugins("grokfast") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_instantiate_trainer_plugins_in_dunder_all():
|
||||
"""Public API surface includes the new stub."""
|
||||
import soup_cli.utils.trainer_plugins as mod
|
||||
|
||||
assert "instantiate_trainer_plugins" in mod.__all__
|
||||
|
||||
|
||||
def test_instantiate_trainer_plugins_empty_list_passes_validation():
|
||||
"""Empty list is valid per `validate_trainer_plugin_list` → the
|
||||
NotImplementedError fires with the empty tuple in the message."""
|
||||
from soup_cli.utils.trainer_plugins import instantiate_trainer_plugins
|
||||
|
||||
with pytest.raises(NotImplementedError, match=r"\(\)|v0\.53\.7"):
|
||||
instantiate_trainer_plugins([])
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Review-fix coverage: regression guards + missing boundaries
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ngram_kwarg_omitted_when_config_is_none():
|
||||
"""Default `ngram_config=None` → `prompt_lookup_num_tokens` NOT emitted.
|
||||
|
||||
Regression guard: a future refactor that always emits the kwarg
|
||||
would break the legacy free-form generation path.
|
||||
"""
|
||||
pytest.importorskip("torch")
|
||||
from soup_cli.commands.serve import _generate_response
|
||||
|
||||
captured: dict = {}
|
||||
|
||||
class _FakeModel:
|
||||
device = "cpu"
|
||||
|
||||
def generate(self, **kwargs):
|
||||
captured.update(kwargs)
|
||||
return [[0, 1, 2]]
|
||||
|
||||
class _FakeTokenizer:
|
||||
pad_token_id = 0
|
||||
chat_template = None
|
||||
|
||||
def __call__(self, _text, return_tensors=None):
|
||||
import torch as _torch
|
||||
|
||||
ids = _torch.tensor([[1]])
|
||||
return {"input_ids": ids, "attention_mask": _torch.ones_like(ids)}
|
||||
|
||||
def decode(self, _tokens, skip_special_tokens=True):
|
||||
return ""
|
||||
|
||||
_generate_response(
|
||||
_FakeModel(),
|
||||
_FakeTokenizer(),
|
||||
[{"role": "user", "content": "x"}],
|
||||
max_tokens=1,
|
||||
)
|
||||
assert "prompt_lookup_num_tokens" not in captured
|
||||
|
||||
|
||||
def test_run_recipe_rejects_empty_output_dir():
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
dag = RecipeDAG(
|
||||
nodes=(RecipeNode(name="a", kind="seed", config={}),),
|
||||
edges=(),
|
||||
topo_order=("a",),
|
||||
)
|
||||
with pytest.raises(TypeError, match="output_dir"):
|
||||
run_recipe(dag, output_dir="")
|
||||
|
||||
|
||||
def test_run_recipe_rejects_non_str_judge_provider():
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
dag = RecipeDAG(
|
||||
nodes=(RecipeNode(name="a", kind="seed", config={}),),
|
||||
edges=(),
|
||||
topo_order=("a",),
|
||||
)
|
||||
with pytest.raises(TypeError, match="judge_provider"):
|
||||
run_recipe(dag, output_dir="out", judge_provider=123) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_run_recipe_rejects_non_str_judge_model():
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
dag = RecipeDAG(
|
||||
nodes=(RecipeNode(name="a", kind="seed", config={}),),
|
||||
edges=(),
|
||||
topo_order=("a",),
|
||||
)
|
||||
with pytest.raises(TypeError, match="judge_model"):
|
||||
run_recipe(dag, output_dir="out", judge_model=b"bytes") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_run_recipe_rejects_non_str_checkpoint_dir():
|
||||
from soup_cli.utils.recipe_dag import RecipeDAG, RecipeNode
|
||||
from soup_cli.utils.recipe_run import run_recipe
|
||||
|
||||
dag = RecipeDAG(
|
||||
nodes=(RecipeNode(name="a", kind="seed", config={}),),
|
||||
edges=(),
|
||||
topo_order=("a",),
|
||||
)
|
||||
with pytest.raises(TypeError, match="checkpoint_dir"):
|
||||
run_recipe(dag, output_dir="out", checkpoint_dir=99) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_anthropic_messages_validation_detail_redacted(monkeypatch):
|
||||
"""Validation errors must surface as generic 'Invalid request' — no
|
||||
internal validator detail in the HTTP body. Security review M1."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
app = _build_app()
|
||||
client = TestClient(app)
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
json={"model": "x", "messages": [], "max_tokens": 16}, # empty msgs
|
||||
)
|
||||
assert response.status_code == 400
|
||||
body = response.json()
|
||||
assert body["detail"] == "Invalid request"
|
||||
|
||||
|
||||
def test_anthropic_messages_rejects_oversize_max_tokens():
|
||||
"""`max_tokens` above the v0.30.0 16384 cap is rejected by the
|
||||
underlying validator (defence-in-depth — also covered upstream)."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
app = _build_app()
|
||||
client = TestClient(app)
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
json={
|
||||
"model": "x",
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"max_tokens": 999_999,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
def test_data_recipe_execute_rejects_empty_output(tmp_path: Path, monkeypatch):
|
||||
"""`--output ""` is rejected at the CLI boundary."""
|
||||
from soup_cli.cli import app as cli_app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
Path("recipe.yaml").write_text(
|
||||
"nodes:\n - name: a\n kind: seed\nedges: []\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
["data", "recipe", "recipe.yaml", "--execute", "--output", ""],
|
||||
)
|
||||
assert result.exit_code == 2, result.output
|
||||
assert "must be a non-empty path" in result.output
|
||||
|
||||
|
||||
def test_data_recipe_execute_rejects_outside_cwd(tmp_path: Path, monkeypatch):
|
||||
"""`--output` outside cwd is rejected before the stub runs."""
|
||||
import sys
|
||||
|
||||
from soup_cli.cli import app as cli_app
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
sub = tmp_path / "proj"
|
||||
sub.mkdir()
|
||||
monkeypatch.chdir(sub)
|
||||
Path("recipe.yaml").write_text(
|
||||
"nodes:\n - name: a\n kind: seed\nedges: []\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
outside = str(tmp_path / "outside")
|
||||
# On Windows, an absolute path under tmp_path.parent (different
|
||||
# drive in some CI envs) is still not under cwd=tmp_path/proj.
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
["data", "recipe", "recipe.yaml", "--execute", "--output", outside],
|
||||
)
|
||||
# Either outside-cwd (preferred) or some platform-specific reject —
|
||||
# the live runner should NEVER fire.
|
||||
assert result.exit_code == 2, (result.output, sys.platform)
|
||||
assert "v0.53.7" not in result.output, (
|
||||
"outside-cwd output must NOT reach the live runner stub"
|
||||
)
|
||||
Loading…
Reference in New Issue