RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

feat(v0.53.9): Live Dashboard + UX + Bench + Standalone CLIs 

Eight features that close out the v0.44.x live-monitoring deferrals
plus a long tail of standalone CLI wins:

- #94  /api/train/stream async SSE with per-subscriber cursor + JS
       EventSource consumer; SoupTrainerCallback pushes TrainEvent on
       each on_log.
- #95  soup ui --public derives LAN IP via SOCK_DGRAM connect-trick,
       prints scannable QR; --auth-token override; SPA bootstrap
       hydrates window._authToken from ?token= + sessionStorage and
       cleans the URL via history.replaceState; CORS regex auto-widens
       to loopback + RFC1918 in public mode; set_auth_token rotation
       race fixed via threading.Lock.
- #98  soup serve --reasoning-parser strips <think>...</think> (and
       OpenThinker tags); pre-compiled regex with marker-token
       fast-path + 1 MiB cap + leading-newline-only strip.
- #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON
       endpoint; best-effort observation hook in callback.on_log.
- #15  soup tokenizer train: BPE training CLI with raw-path lstat
       symlink rejection, 50 MiB total / 8 KiB per-line caps,
       post-mkdir output-dir re-check, --special-token NUL/oversize
       dedup, vocab bounds [256, 200000].
- #26  soup bench --p50 --p95 renders extra per-prompt tail-latency
       Rich table; --prompts-file gains symlink rejection.
- #28  soup bench --backend auto: MLX weights.npz probe (per-entry
       lstat) -> config.json model_type keyword -> transformers
       fallback; SOUP_BENCH_BACKEND env hint.
- #12  examples/synthetic_workflow.{md,yaml} end-to-end walkthrough.

Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the
python / code / security / tdd review agents. Notable HIGH:
- QR token now consumed by SPA (was unreachable previously).
- set_auth_token rotation lock-protected, 8-thread stress tested.
- Tokenizer input + output symlink TOCTOU defence on raw path.
- SSE generator switched to async (asyncio.sleep) for non-blocking
  multi-subscriber operation.
- CORS regex for --public LAN mode (the old fixed allowlist of
  http://0.0.0.0:port never matched a real Origin header).
- _has_mlx_weights per-entry lstat so a symlinked weights.npz can't
  trigger MLX dispatch.

Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the
relaxed v0.53.8 version-pin asserts in tests/test_v0538.py).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys 2026-05-14 04:28:19 +05:00
parent 13ffc89058
commit bde9d149a0
22 changed files with 2030 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CONTRIBUTING.md
View File

@ -111,7 +111,7 @@ soup_cli/
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
ui/ - Web UI (FastAPI + HTML/JS SPA)
tests/ - Test suite (188 files, 8257 tests)
tests/ - Test suite (189 files, 8285 tests)
examples/ - Real-world config examples and datasets
```

23
README.md
View File

@ -42,15 +42,17 @@ soup train
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
**v0.53.8 — Remote data + Hubs + Trackers (wave 2)**: Six features wired live across cloud storage, alternative model hubs, experiment trackers, and HF Spaces.
**v0.53.9 — Live Dashboard + UX + Bench + Standalone CLIs**: Eight features that close out the v0.44.x live-monitoring deferrals plus a long tail of standalone CLI wins.
- **fsspec live loaders for remote datasets.** `data.train: s3://bucket/data.jsonl` (plus `gs://` / `gcs://` / `az://` / `abfs://` / `abfss://` / `oci://`) now loads through `fsspec` with the v0.42.0 SSRF-hardened URI validator (bucket regex + userinfo/query/fragment rejection). Threads `data.streaming` (HF `load_dataset(streaming=True)`) and `data.buffer_size` shuffle. Friendly Rich panel names the `pip install <driver>` advisory when the backend SDK is missing. Row-count capped at 1M to defend against pathological remote objects.
- **Live ModelScope + Modelers hub dispatcher.** New `utils/hubs.download_repo()` and `upload_repo()` lazy-import the matching SDK per backend (`huggingface_hub`, `modelscope`, `openmind-hub`). Shared `_validate_repo_id_shape` rejects bool / null-byte / leading-slash / `..` segments / control characters / oversize; `local_dir` and `folder_path` containment-checked under cwd. `soup train` now pre-fetches non-HF `base` models into `.soup_hub_cache/<sanitized-slug>/` and reuses cached snapshots on resumed runs.
- **`[trackers]` extra + friendly missing-dep advisory.** New `pip install soup-cli[trackers]` bundles MLflow + SwanLab + Trackio. When you pass `--tracker mlflow` without the package installed, `soup train` now surfaces a clear `pip install` advisory before construction instead of HF Trainer's generic ImportError. The probe uses `importlib.util.find_spec` (non-executing) so swanlab can't initialise network threads during the check.
- **PostHog telemetry network (opt-IN, silent-fail).** `utils/trackers.send_telemetry_payload()` lazy-imports `httpx` and POSTs hardware-info-only payloads with a 1-second hard timeout. HTTPS-only endpoint check goes through the same SSRF validator as hub endpoints (private-IP / link-local / RFC1918 rejected). Disabled unless `SOUP_TELEMETRY=1`; every exception is swallowed so telemetry can never crash training.
- **Bundled demo fixtures as package data.** `soup data demo` fixtures (4 JSONLs) migrated from `examples/data/` to `soup_cli/data/_fixtures/` so they ship inside the wheel — zipapp / namespace-package safe. `_bundle_source_path` falls back to the legacy `examples/data/` location for editable installs.
- **HF Space SDK auto-pick from `requirements.txt`.** `soup deploy hf-space --template-dir <path>` now reads the rendered `requirements.txt` and picks `space_sdk="streamlit"` when the file lists `streamlit`, `"gradio"` otherwise. Closes the v0.40.2 known limitation that custom templates always created Spaces with `space_sdk="gradio"`.
- **+95 net new tests** (8162 → 8257) in `test_v0538.py`. Three review agents ran (python / code / security); 16 findings fixed: cwd-containment on `local_dir`/`folder_path`, Pydantic `model_copy(update=...)` instead of attribute mutation, idempotent train pre-fetch via cache probe, row-count cap on remote materialisation, `modelscope.push_model` kwarg fix, SSRF re-validation on telemetry endpoint override, `detect_space_sdk` size cap, and more.
- **Live `/api/train/stream` SSE endpoint.** Async FastAPI route streams `TrainEvent` payloads (loss / lr / grad_norm / EMA / p95 tail latency) as W3C SSE frames. Per-subscriber cursor — multiple dashboard tabs each receive every event without starving each other. `asyncio.sleep` so uvicorn's async loop is never blocked. Drains gracefully with a `status=done` event when training exits.
- **`soup ui --public` + phone-scannable QR.** Binds `0.0.0.0`, detects the actual LAN IP via a `socket.SOCK_DGRAM` connect trick (no packets sent), and prints an ASCII QR encoding `http://<lan-ip>:<port>/?token=<bearer>`. The SPA picks up `?token=` on load via `URLSearchParams`, caches it in `sessionStorage`, and rewrites the URL clean so the token doesn't sit in browser history. `--auth-token` override accepts a stable urlsafe-base64 token for repeatable phone bookmarks. CORS regex auto-widens to loopback + RFC1918 ranges when public; Bearer token gates mutating endpoints. Concurrent token rotation is now lock-protected.
- **`soup serve --reasoning-parser {deepseek-r1|qwen3|phi4|openthinker}`.** Strips `<think>...</think>` (or `<|begin_of_thought|>...<|end_of_thought|>` for OpenThinker) from responses before they hit the client. Pre-compiled regex with a marker-token fast-path (skips `re.sub` entirely when no marker is present) and a 1 MiB input cap. Strips only leading newlines after removal so code outputs that begin with intentional whitespace are preserved.
- **`/api/tool-outputs` + `ToolOutputsBuffer` global singleton.** Process-wide thread-safe ring buffer of tool-call records (`name` / `started_ts` / `duration_ms` / `success` / `output_preview` / `error`). New `/api/tool-outputs?limit=N` JSON endpoint reads the buffer for live dashboard polling. SFT trainer-side observation hook lands in a follow-up.
- **`soup tokenizer train`.** New BPE training CLI via the `tokenizers` library. JSONL or plaintext input (with ShareGPT `messages` fallback), `--vocab-size` bounded [256, 200000], `--special-token` repeatable with NUL/oversize/dedup validation. `os.lstat + S_ISLNK` rejection on the raw user-supplied input path BEFORE realpath, 50 MiB total / 8 KiB per-line caps, post-`mkdir` re-check on the output directory so a planted symlink can't redirect the write.
- **`soup bench --p50 --p95`.** Per-prompt latency capture rendered as an extra Rich table after the throughput row. Reuses `utils/tail_latency.summarise_latency` (v0.44.0). `--prompts-file` now rejects symlinked inputs on the raw path before realpath, in line with the project-wide TOCTOU policy.
- **`soup bench --backend auto`.** New default — probes the model directory for MLX `weights.npz` and falls back to `config.json` `model_type` keyword detection. `SOUP_BENCH_BACKEND` env hint overrides the probe. Every iterdir entry is `os.lstat`-checked so a symlinked `weights.npz` cannot trigger MLX dispatch.
- **`examples/synthetic_workflow.md` walkthrough.** End-to-end docs covering `soup data generate` → `filter` → `score` → `decontaminate` → `train`, plus the `soup ui --public` phone-monitoring loop.
- **+28 net new tests** (8257 → 8285) in `test_v0539.py`. Three review agents ran (python / code / security / tdd / verification); 34 findings fixed: H1 (QR token never consumed by SPA — now hydrates from URL into sessionStorage), H2 (`set_auth_token` rotation race — now lock-protected), tokenizer input/output symlink rejection (raw-path lstat), prompts-file symlink rejection, regex fast-path skip on missing markers, leading-newline strip vs full `lstrip()`, CORS regex for `--public` LAN mode, `_has_mlx_weights` per-entry lstat, and more.
## Why Soup?
@ -3095,6 +3097,11 @@ soup tui Full-screen Textual dashboard (req
soup train --config soup.yaml --profile Record torch.profiler trace to <output>/profiles/
soup --log-level quiet|normal|verbose|debug Global logging tier (Rich-formatted)
soup ui [--port 7860] Web UI (experiments, training, data)
soup ui --public [--auth-token T] Phone-scannable Web UI (v0.53.9)
soup tokenizer train --input c.jsonl --vocab-size N Train BPE tokenizer (v0.53.9)
soup bench <model> --p50 --p95 Bench with tail-latency percentiles (v0.53.9)
soup bench <model> --backend auto Auto-detect transformers/mlx backend (v0.53.9)
soup serve --reasoning-parser deepseek-r1 Strip <think> blocks from responses (v0.53.9)
soup doctor Check environment
soup quickstart [--dry-run] Full demo
soup version [--full] [--json] Show version (--full: system info, --json: JSON output)

4
SECURITY.md
View File

File diff suppressed because one or more lines are too long

16
examples/README.md
View File

@ -177,6 +177,22 @@ Run inference on a batch of prompts:
soup infer --model ./output_sft_basic/ --input prompts.jsonl --output results.jsonl
```
### 13. Synthetic-data workflow
End-to-end recipe that generates training data from a local LLM, filters
+ scores + decontaminates it, then trains on the cleaned set. See
[synthetic_workflow.md](synthetic_workflow.md) for the walkthrough and
[synthetic_workflow.yaml](synthetic_workflow.yaml) for the bundled config.
```bash
soup data generate --provider ollama --output ./synth_raw.jsonl
soup data filter --input ./synth_raw.jsonl --output ./synth_filtered.jsonl
soup data score --input ./synth_filtered.jsonl --output ./synth_scored.jsonl
soup data decontaminate --input ./synth_scored.jsonl \
--output ./synth_clean.jsonl --benchmarks mmlu,gsm8k
soup train --config examples/synthetic_workflow.yaml --yes
```
### 12. Full RLHF Pipeline
Complete reinforcement learning from human feedback:

72
examples/synthetic_workflow.md Normal file
View File

@ -0,0 +1,72 @@
# Synthetic data workflow (end-to-end)
Generate, filter, score, and train on synthetic data — all with `soup`. Pairs
with the bundled `synthetic_workflow.yaml` recipe.
## 1. Generate
Spin up a local Ollama model and ask it for 200 instruction/response pairs
around a topic:
```bash
soup data generate \
--provider ollama \
--model llama3.2:3b \
--topic "Python error handling" \
--count 200 \
--output ./synth_raw.jsonl
```
For Anthropic / vLLM / server providers, swap `--provider` and follow the
`soup data generate --help` matrix.
## 2. Filter for quality
Drop low-perplexity / low-coherence rows:
```bash
soup data filter \
--input ./synth_raw.jsonl \
--output ./synth_filtered.jsonl \
--min-coherence 0.5
```
## 3. Score for safety + diversity
Run the v0.47.0 quality moat to fingerprint PII / toxicity / language /
educational value, and decontaminate against your downstream evals:
```bash
soup data score --input ./synth_filtered.jsonl --output ./synth_scored.jsonl
soup data decontaminate \
--input ./synth_scored.jsonl \
--output ./synth_clean.jsonl \
--benchmarks mmlu,gsm8k
```
## 4. Train
Point `soup train` at `synthetic_workflow.yaml`:
```bash
soup train --config examples/synthetic_workflow.yaml --yes
```
That recipe references `./synth_clean.jsonl`, picks `TinyLlama-1.1B-Chat`
as the base, and runs an SFT job with LoRA r=8.
## 5. Watch progress live (v0.53.9)
In another terminal:
```bash
soup ui --public --no-browser
```
Scan the printed QR code from your phone to monitor the loss curve and
live SSE training stream while the job runs.
---
This is a thin walkthrough — for deeper coverage see
[examples/README.md](README.md) and [examples/configs/](configs/).

40
examples/synthetic_workflow.yaml Normal file
View File

@ -0,0 +1,40 @@
# Synthetic-data workflow recipe (v0.53.9 #12)
#
# Pairs with `examples/synthetic_workflow.md`. Trains TinyLlama-1.1B on
# a JSONL produced by:
# soup data generate --provider ollama --output ./synth_raw.jsonl
# soup data filter --input ./synth_raw.jsonl --output ./synth_filtered.jsonl
# soup data score --input ./synth_filtered.jsonl --output ./synth_scored.jsonl
# soup data decontaminate --input ./synth_scored.jsonl \
# --output ./synth_clean.jsonl --benchmarks mmlu,gsm8k
base: TinyLlama/TinyLlama-1.1B-Chat-v1.0
task: sft
modality: text
backend: transformers
data:
train: ./synth_clean.jsonl
format: auto
val_split: 0.05
max_length: 1024
training:
epochs: 1
lr: 2.0e-4
batch_size: auto
quantization: 4bit
optimizer: adamw_torch
scheduler: cosine
gradient_checkpointing: true
warmup_auto: true
auto_mixed_precision: true
lora:
r: 8
alpha: 16
dropout: 0.05
output: ./synth_out
eval:
auto_eval: false

2
pyproject.toml
View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "soup-cli"
version = "0.53.8.1"
version = "0.53.9"
description = "Fine-tune LLMs in one command. No SSH, no config hell."
readme = "README.md"
license = "Apache-2.0"

2
soup_cli/__init__.py
View File

@ -1,3 +1,3 @@
"""Soup CLI — Fine-tune LLMs in one command."""
__version__ = "0.53.8.1"
__version__ = "0.53.9"

9
soup_cli/cli.py
View File

@ -189,6 +189,15 @@ from soup_cli.commands import data_mix as _data_mix_cmd # noqa: E402
data.app.command(name="mix")(_data_mix_cmd.mix)
# v0.53.9 #15 — BPE tokenizer training.
from soup_cli.commands import tokenizer as _tokenizer_cmd # noqa: E402
app.add_typer(
_tokenizer_cmd.app,
name="tokenizer",
help="Tokenizer tools: train a BPE tokenizer from JSONL (v0.53.9).",
)
@app.command()
def version(

75
soup_cli/commands/bench.py
View File

@ -39,6 +39,24 @@ def bench(
"--prompts-file",
help="Path to custom prompts file (.txt or .jsonl)",
),
p50: bool = typer.Option(
False,
"--p50",
help="Print p50 (median) per-prompt latency. v0.53.9 #26.",
),
p95: bool = typer.Option(
False,
"--p95",
help="Print p95 per-prompt latency. v0.53.9 #26.",
),
backend: str = typer.Option(
"auto",
"--backend",
help=(
"Inference backend hint: auto (default) | transformers | mlx. "
"v0.53.9 #28."
),
),
) -> None:
"""Run an inference benchmark (speed and memory) on a loaded model."""
import torch
@ -64,6 +82,24 @@ def bench(
device, _ = detect_device()
# v0.53.9 #28 — backend auto-detect.
from soup_cli.utils.backend_detect import SUPPORTED_BACKENDS, detect_backend
backend_lower = (backend or "auto").strip().lower()
if backend_lower == "auto":
backend_resolved = detect_backend(str(model_path))
console.print(
f"[dim]Backend auto-detected:[/] [bold]{backend_resolved}[/]"
)
else:
if backend_lower not in SUPPORTED_BACKENDS:
console.print(
f"[red]Unknown --backend:[/] {backend} "
f"(expected: auto | {' | '.join(sorted(SUPPORTED_BACKENDS))})"
)
raise typer.Exit(2)
backend_resolved = backend_lower
if device == "cpu":
console.print(
"[yellow]Warning:[/] Running on CPU. Inference speed is typically "
@ -82,6 +118,20 @@ def bench(
"current working directory."
)
raise typer.Exit(1)
# v0.53.9 review fix M2 — reject symlinked prompts file on the
# RAW path BEFORE realpath resolution (mirrors v0.53.7 #106 policy).
import stat as _stat
try:
_st = _os.lstat(prompts_file)
except OSError:
console.print(f"[red]Prompts file not found:[/] {prompts_file}")
raise typer.Exit(1)
if _stat.S_ISLNK(_st.st_mode):
console.print(
"[red]Prompts file must not be a symlink.[/]"
)
raise typer.Exit(1)
p_path = Path(_os.path.realpath(prompts_file))
if not p_path.is_file():
@ -164,6 +214,7 @@ def bench(
total_tokens = 0
total_latency = 0.0
per_prompt_latencies: list[float] = []
console.print(f"[bold]Running {len(test_prompts)} test inferences...[/]")
@ -179,6 +230,7 @@ def bench(
latency = time.time() - start_time
total_tokens += token_count
total_latency += latency
per_prompt_latencies.append(latency)
console.print(f" [dim]Prompt {i + 1}: {token_count} tokens in {latency:.2f}s[/]")
avg_tps = total_tokens / total_latency if total_latency > 0 else 0
@ -195,7 +247,7 @@ def bench(
vram_str = f"{peak_vram_gb:.2f} GB" if torch.cuda.is_available() else "N/A"
table.add_row(
"Transformers",
backend_resolved.capitalize(),
f"{avg_tps:.2f}",
f"{total_latency:.2f}s",
vram_str,
@ -203,3 +255,24 @@ def bench(
console.print()
console.print(table)
# v0.53.9 #26 — tail-latency percentiles.
if (p50 or p95) and per_prompt_latencies:
from soup_cli.utils.tail_latency import summarise_latency
summary = summarise_latency(per_prompt_latencies)
pct_table = Table(title="Per-prompt latency")
pct_table.add_column("Statistic", style="cyan")
pct_table.add_column("Latency (s)", style="green", justify="right")
pct_table.add_row("count", str(summary.count))
if summary.mean is not None:
pct_table.add_row("mean", f"{summary.mean:.3f}")
if p50 and summary.p50 is not None:
pct_table.add_row("p50", f"{summary.p50:.3f}")
if p95 and summary.p95 is not None:
pct_table.add_row("p95", f"{summary.p95:.3f}")
# p99 is a natural superset of p95 — only print when p95 was requested.
if p95 and summary.p99 is not None:
pct_table.add_row("p99", f"{summary.p99:.3f}")
console.print()
console.print(pct_table)

29
soup_cli/commands/serve.py
View File

@ -191,6 +191,14 @@ def serve(
"--trace-log-cap-mb",
help="Rotation cap in MB for --trace-log (1 - 10000). Default 100.",
),
reasoning_parser: Optional[str] = typer.Option(
None,
"--reasoning-parser",
help=(
"Strip reasoning-trace blocks from responses. One of: "
"deepseek-r1 | qwen3 | phi4 | openthinker. v0.53.9 #98."
),
),
):
"""Start a local inference server with OpenAI-compatible API."""
# Lazy imports for fast CLI startup
@ -603,6 +611,17 @@ def serve(
f"(cap {trace_log_cap_mb} MB)"
)
# v0.53.9 #98 — validate reasoning parser name once at startup.
resolved_reasoning_parser: Optional[str] = None
if reasoning_parser:
from soup_cli.utils.reasoning_parser import validate_parser_name
try:
resolved_reasoning_parser = validate_parser_name(reasoning_parser)
except (TypeError, ValueError) as exc:
console.print(f"[red]--reasoning-parser:[/] {exc}")
raise typer.Exit(1) from exc
app = _create_app(
model_obj=model_obj,
tokenizer=tokenizer,
@ -616,6 +635,7 @@ def serve(
enable_dashboard=dashboard,
tracer=tracer,
trace_log_writer=trace_log_writer,
reasoning_parser=resolved_reasoning_parser,
)
console.print(
@ -902,6 +922,7 @@ def _create_app(
web_search_config: Any = None,
web_search_backend: Any = None,
auth_token: Optional[str] = None,
reasoning_parser: Optional[str] = None,
):
"""Create the FastAPI application with OpenAI-compatible endpoints.
@ -1116,6 +1137,14 @@ def _create_app(
metrics.record_tokens(completion_tokens)
# v0.53.9 #98 — strip reasoning-trace blocks if configured.
if reasoning_parser is not None:
from soup_cli.utils.reasoning_parser import strip_reasoning
response_text = strip_reasoning(
response_text, reasoning_parser,
)
# output_constraint is validated upstream; v0.33.0 #53 wires
# it through outlines / lm-format-enforcer into the generate
# loop. If neither library is installed, build_logits_processors

247
soup_cli/commands/tokenizer.py Normal file
View File

@ -0,0 +1,247 @@
"""soup tokenizer — BPE tokenizer training (v0.53.9 #15)."""
from __future__ import annotations
import json
import os
import stat
from pathlib import Path
from typing import Optional
import typer
from rich.console import Console
from rich.panel import Panel
from soup_cli.utils.paths import is_under_cwd
console = Console()
app = typer.Typer(
help="Tokenizer tools: train a BPE tokenizer from a JSONL corpus.",
no_args_is_help=True,
)
_MAX_VOCAB_SIZE = 200_000
_MIN_VOCAB_SIZE = 256
_MAX_CORPUS_BYTES = 50 * 1024 * 1024 # 50 MB total cap
_MAX_LINE_BYTES = 8 * 1024 # 8 KB per-line cap
def _validate_under_cwd(path: str, *, label: str) -> str:
if not isinstance(path, str) or not path:
raise typer.BadParameter(f"{label} must be a non-empty path")
if "\x00" in path:
raise typer.BadParameter(f"{label} contains NUL byte")
if not is_under_cwd(path):
raise typer.BadParameter(
f"{label} must stay under the current working directory: {path}"
)
# TOCTOU defence: lstat the RAW user-supplied path BEFORE realpath
# resolution (matches v0.53.7 #106 policy). `os.path.realpath` would
# follow the link and `lstat` on the resolved target would see a
# regular file even when the user-supplied entry is a symlink.
try:
st = os.lstat(path)
except OSError:
# Missing-file path is handled by the caller's `is_file` check.
return os.path.realpath(path)
if stat.S_ISLNK(st.st_mode):
raise typer.BadParameter(
f"{label} target must not be a symlink"
)
return os.path.realpath(path)
def _extract_texts(input_path: Path) -> list[str]:
"""Read JSONL or plaintext lines into a list of training strings.
Capped at 50 MB total + 8 KB per line to bound RAM on adversarial input.
"""
texts: list[str] = []
total_bytes = 0
if input_path.suffix.lower() == ".jsonl":
with open(input_path, encoding="utf-8") as fh:
for raw in fh:
if len(raw.encode("utf-8", errors="replace")) > _MAX_LINE_BYTES:
continue # silently skip oversized line
total_bytes += len(raw)
if total_bytes > _MAX_CORPUS_BYTES:
break
text = raw.strip()
if not text:
continue
try:
row = json.loads(text)
except json.JSONDecodeError:
continue
if isinstance(row, dict):
for key in ("text", "content", "prompt"):
value = row.get(key)
if isinstance(value, str) and value:
texts.append(value)
break
else:
# ShareGPT-style messages list
msgs = row.get("messages")
if isinstance(msgs, list):
for msg in msgs:
if isinstance(msg, dict):
content = msg.get("content")
if isinstance(content, str) and content:
texts.append(content)
elif isinstance(row, str):
texts.append(row)
else:
with open(input_path, encoding="utf-8") as fh:
for raw in fh:
if len(raw.encode("utf-8", errors="replace")) > _MAX_LINE_BYTES:
continue
total_bytes += len(raw)
if total_bytes > _MAX_CORPUS_BYTES:
break
text = raw.rstrip("\n\r")
if text:
texts.append(text)
return texts
@app.command(name="train")
def train(
input_path: str = typer.Option(
...,
"--input",
"-i",
help="Path to JSONL or .txt corpus (must stay under cwd).",
),
vocab_size: int = typer.Option(
32_000,
"--vocab-size",
"-v",
help=f"Target vocabulary size ({_MIN_VOCAB_SIZE}-{_MAX_VOCAB_SIZE}).",
),
output: str = typer.Option(
"tokenizer_out",
"--output",
"-o",
help="Output directory for tokenizer.json + vocab.json (under cwd).",
),
min_frequency: int = typer.Option(
2,
"--min-frequency",
help="Minimum pair frequency (>=1).",
),
special_tokens: Optional[list[str]] = typer.Option(
None,
"--special-token",
help="Add special token (repeatable). Defaults: <pad>, <s>, </s>, <unk>.",
),
) -> None:
"""Train a BPE tokenizer from a local corpus.
Outputs `tokenizer.json` (Hugging Face format) and `vocab.json` to
`--output`. Requires the `tokenizers` library (bundled with
`transformers`).
"""
if isinstance(vocab_size, bool) or not isinstance(vocab_size, int):
raise typer.BadParameter("--vocab-size must be int")
if not (_MIN_VOCAB_SIZE <= vocab_size <= _MAX_VOCAB_SIZE):
raise typer.BadParameter(
f"--vocab-size must be in [{_MIN_VOCAB_SIZE}, {_MAX_VOCAB_SIZE}]"
)
if isinstance(min_frequency, bool) or not isinstance(min_frequency, int):
raise typer.BadParameter("--min-frequency must be int")
if min_frequency < 1:
raise typer.BadParameter("--min-frequency must be >= 1")
input_real = _validate_under_cwd(input_path, label="--input")
output_real = _validate_under_cwd(output, label="--output")
resolved_input = Path(input_real)
if not resolved_input.is_file():
console.print(f"[red]Input file not found:[/] {input_path}")
raise typer.Exit(1)
output_path = Path(output_real)
try:
from tokenizers import Tokenizer, models, pre_tokenizers, trainers
except ImportError as exc:
console.print(
"[red]The 'tokenizers' package is required.[/]\n"
"Install with: [bold]pip install tokenizers[/]"
)
raise typer.Exit(1) from exc
texts = _extract_texts(resolved_input)
if not texts:
console.print(
f"[red]No usable training text found in[/] {resolved_input.name}"
)
raise typer.Exit(1)
tokens = list(special_tokens or [])
if not tokens:
tokens = ["<pad>", "<s>", "</s>", "<unk>"]
# Sanitise: reject NUL / oversize entries; dedup preserves first-seen order.
cleaned: list[str] = []
seen: set[str] = set()
for tok in tokens:
if not isinstance(tok, str) or not tok or "\x00" in tok or len(tok) > 128:
raise typer.BadParameter(
f"--special-token entry must be a non-empty NUL-free str <=128 chars: {tok!r}"
)
if tok in seen:
continue
seen.add(tok)
cleaned.append(tok)
console.print(
Panel(
f"Input: [bold]{resolved_input.name}[/]\n"
f"Texts: [bold]{len(texts)}[/]\n"
f"Vocab size: [bold]{vocab_size}[/]\n"
f"Output dir: [bold]{output_path.name}[/]",
title="BPE Tokenizer Training",
)
)
tokenizer = Tokenizer(models.BPE(unk_token="<unk>"))
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
trainer = trainers.BpeTrainer(
vocab_size=vocab_size,
min_frequency=min_frequency,
special_tokens=cleaned,
show_progress=False,
)
tokenizer.train_from_iterator(texts, trainer=trainer)
output_path.mkdir(parents=True, exist_ok=True)
# Post-mkdir re-check: defends against a symlink planted between the
# containment check and the write. Matches v0.43.0 Part D `copy_bundle_to`
# TOCTOU policy.
try:
post_st = os.lstat(output_path)
except OSError as exc:
console.print(f"[red]Output directory not accessible:[/] {exc}")
raise typer.Exit(1) from exc
if stat.S_ISLNK(post_st.st_mode):
console.print(
"[red]Output directory is a symlink — refusing to write.[/]"
)
raise typer.Exit(1)
if not is_under_cwd(str(output_path)):
console.print("[red]Output directory escaped cwd after mkdir.[/]")
raise typer.Exit(1)
tokenizer_file = output_path / "tokenizer.json"
tokenizer.save(str(tokenizer_file))
vocab_file = output_path / "vocab.json"
vocab = tokenizer.get_vocab()
vocab_file.write_text(
json.dumps(vocab, ensure_ascii=False, indent=2),
encoding="utf-8",
)
console.print(
f"[green]Trained tokenizer with[/] [bold]{len(vocab)}[/] tokens "
f"-> [bold]{output_path.name}/{tokenizer_file.name}[/]"
)

113
soup_cli/commands/ui.py
View File

@ -1,5 +1,9 @@
"""soup ui — local web interface for managing experiments and training."""
from __future__ import annotations
from typing import Optional
import typer
from rich.console import Console
from rich.panel import Panel
@ -29,11 +33,34 @@ def ui(
"--show-token",
help="Print the auth token and exit (for scripting)",
),
public: bool = typer.Option(
False,
"--public",
help=(
"Bind to 0.0.0.0 so the UI is reachable from a phone on the "
"same LAN. Prints a phone-scannable URL + ASCII QR at startup. "
"Loopback HTTP is auto-upgraded to require a Bearer token in "
"the query string. v0.53.9 #95."
),
),
auth_token: Optional[str] = typer.Option(
None,
"--auth-token",
help=(
"Override the auto-generated Bearer token (16-128 urlsafe "
"base64 chars). Useful for stable phone bookmarks. "
"When omitted, a fresh token is generated each startup."
),
),
):
"""Launch the Soup Web UI.
A Bearer auth token is auto-generated at startup and printed to the console.
Mutating API endpoints (POST/DELETE) require 'Authorization: Bearer <token>'.
`--public` exposes the server on 0.0.0.0 for phone-on-LAN access. The
auth token is embedded in a phone-scannable URL + ASCII QR code so a
phone scan lands authenticated.
"""
try:
import uvicorn # noqa: F401
@ -45,7 +72,21 @@ def ui(
)
raise typer.Exit(1)
from soup_cli.ui.app import create_app, get_auth_token
from soup_cli.ui.app import create_app, get_auth_token, set_auth_token
# v0.53.9 #95 — operator-supplied auth token override.
# `set_auth_token` validates via `qr_url.validate_token`, so we don't
# double-validate here.
if auth_token is not None:
try:
set_auth_token(auth_token)
except (TypeError, ValueError) as exc:
console.print(f"[red]--auth-token:[/] {exc}")
raise typer.Exit(1) from exc
# --public binds to 0.0.0.0 unless the operator already set a custom host.
if public and host == "127.0.0.1":
host = "0.0.0.0"
token = get_auth_token()
@ -57,22 +98,71 @@ def ui(
url = f"http://{host}:{port}"
panel_body = (
f"URL: [bold]{url}[/]\n"
f"Token: [bold]{token}[/]\n\n"
f"Mutating API endpoints require:\n"
f" [dim]Authorization: Bearer {token}[/]\n\n"
f"Pages:\n"
f" [bold]Dashboard[/] - View experiments, loss charts, system info\n"
f" [bold]New Training[/] - Create config from templates, start training\n"
f" [bold]Data Explorer[/] - Browse and inspect datasets\n"
f" [bold]Model Chat[/] - Chat with a running inference server\n\n"
f"Press [bold]Ctrl+C[/] to stop."
)
console.print(
Panel(
f"URL: [bold]{url}[/]\n"
f"Token: [bold]{token}[/]\n\n"
f"Mutating API endpoints require:\n"
f" [dim]Authorization: Bearer {token}[/]\n\n"
f"Pages:\n"
f" [bold]Dashboard[/] - View experiments, loss charts, system info\n"
f" [bold]New Training[/] - Create config from templates, start training\n"
f" [bold]Data Explorer[/] - Browse and inspect datasets\n"
f" [bold]Model Chat[/] - Chat with a running inference server\n\n"
f"Press [bold]Ctrl+C[/] to stop.",
panel_body,
title="[bold green]Soup Web UI[/]",
)
)
# v0.53.9 #95 — phone-visible URL + QR for --public.
if public:
import socket
from soup_cli.utils.qr_url import render_qr_ascii
# Derive the LAN IP a phone on the same Wi-Fi can reach. Falls
# back to 127.0.0.1 if DNS resolution fails — operator gets a
# warning explaining the QR is loopback-only in that case.
lan_ip: str = "127.0.0.1"
try:
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
# Connect to a non-routed address; OS picks the egress
# interface and we read its bound IP. No packets sent.
sock.settimeout(0.1)
sock.connect(("10.255.255.255", 1))
lan_ip = sock.getsockname()[0]
except (OSError, socket.error):
lan_ip = "127.0.0.1"
# Plain HTTP over LAN is intentional: this is local-network only
# and the Bearer token gates mutating endpoints. We assemble the
# URL directly because `build_phone_url` rejects LAN HTTP by
# design (its threat model is stricter than ours here).
phone_url = f"http://{lan_ip}:{port}/?token={token}"
if lan_ip == "127.0.0.1":
console.print(
"[yellow]--public:[/] Could not detect LAN IP; "
f"falling back to loopback. Phone scan will only "
f"work from the same host: {phone_url}"
)
else:
console.print(
f"\n[bold green]Phone URL:[/] {phone_url}\n"
"[dim]Scan the QR code below from your phone camera:[/]\n"
)
qr_ascii = render_qr_ascii(phone_url)
if qr_ascii is None:
console.print(
"[dim]Install 'qrcode' to print an ASCII QR: "
"pip install qrcode[/]"
)
else:
console.print(qr_ascii)
# Open browser
if not no_browser:
import threading
@ -80,6 +170,7 @@ def ui(
def _open():
import time
time.sleep(1)
webbrowser.open(url)

43
soup_cli/monitoring/callback.py
View File

@ -128,6 +128,49 @@ class SoupTrainerCallback(TrainerCallback):
gpu_mem=gpu_mem,
)
# v0.53.9 #94 — push to the global SSE buffer so the live Web UI
# dashboard sees per-step metrics in real time. Best-effort: any
# exception inside the push must NEVER take down training.
try:
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import push_train_event
push_train_event(
TrainEvent(
type="metric",
step=int(step) if step is not None else None,
epoch=float(epoch) if epoch is not None else None,
loss=float(loss) if loss else None,
lr=float(lr) if lr else None,
grad_norm=float(grad_norm) if grad_norm else None,
)
)
except Exception:
pass
# v0.53.9 #100 — tool-call observation for SFT runs whose batch logs
# surface a `tool_calls` count. Best-effort; the SFT trainer emits
# the field only when the data format is `tool-calling`.
if "tool_calls" in (logs or {}):
try:
import time
from soup_cli.utils.tool_outputs import get_global_tool_buffer
tool_count = logs.get("tool_calls")
if isinstance(tool_count, (int, float)) and not isinstance(
tool_count, bool
) and tool_count > 0:
get_global_tool_buffer().record_call(
name="batch",
started_ts=time.time(),
duration_ms=0.0,
success=True,
output_preview=f"observed {int(tool_count)} tool_calls at step {step}",
)
except Exception:
pass
# Loss watchdog — detect loss spikes and auto-stop
if self._watchdog_enabled and not self._watchdog_fired and "loss" in logs:
if loss > self._watchdog_threshold:

145
soup_cli/ui/app.py
View File

@ -46,13 +46,31 @@ _train_process: Optional[subprocess.Popen] = None
_train_config_path: Optional[str] = None
_train_lock = threading.Lock()
# Auth token generated at startup — printed to console for the user
# Auth token generated at startup — printed to console for the user.
# Reads/writes go through `_auth_token_lock` so token rotation never
# leaves a window where some requests see the old value and some the new.
_auth_token: str = secrets.token_urlsafe(32)
_auth_token_lock = threading.Lock()
def get_auth_token() -> str:
"""Return the current auth token (for printing at startup)."""
return _auth_token
with _auth_token_lock:
return _auth_token
def set_auth_token(token: str) -> None:
"""Replace the process-wide auth token (used by `soup ui --auth-token`).
Validates via `utils.qr_url.validate_token` so a malformed override
can't bypass the urlsafe-base64 shape check.
"""
from soup_cli.utils.qr_url import validate_token
validated = validate_token(token)
global _auth_token
with _auth_token_lock:
_auth_token = validated
def create_app(host: str = "127.0.0.1", port: int = 7860):
@ -64,19 +82,39 @@ def create_app(host: str = "127.0.0.1", port: int = 7860):
app = FastAPI(title="Soup Web UI", version="1.0.0")
# Restrict CORS to the origin we actually serve
allowed_origin = f"http://{host}:{port}"
app.add_middleware(
CORSMiddleware,
allow_origins=[allowed_origin],
allow_methods=["GET", "POST", "DELETE"],
allow_headers=["Authorization", "Content-Type"],
)
# Restrict CORS to the origin we actually serve. When `host == "0.0.0.0"`
# the literal `http://0.0.0.0:<port>` is never a browser origin, so we
# allow loopback origins AND the same-LAN regex shape. The Bearer
# token is the actual security gate on mutating endpoints.
if host == "0.0.0.0":
app.add_middleware(
CORSMiddleware,
allow_origin_regex=(
r"^https?://("
r"localhost|127\.0\.0\.1|"
r"10\.\d+\.\d+\.\d+|"
r"192\.168\.\d+\.\d+|"
r"172\.(?:1[6-9]|2[0-9]|3[01])\.\d+\.\d+"
r")(:\d+)?$"
),
allow_methods=["GET", "POST", "DELETE"],
allow_headers=["Authorization", "Content-Type"],
)
else:
allowed_origin = f"http://{host}:{port}"
app.add_middleware(
CORSMiddleware,
allow_origins=[allowed_origin],
allow_methods=["GET", "POST", "DELETE"],
allow_headers=["Authorization", "Content-Type"],
)
def _verify_token(request: Request):
"""Verify Bearer token on mutating endpoints."""
auth = request.headers.get("Authorization", "")
if auth != f"Bearer {_auth_token}":
with _auth_token_lock:
expected = f"Bearer {_auth_token}"
if auth != expected:
raise HTTPException(status_code=401, detail="Unauthorized")
# --- Static files ---
@ -705,6 +743,91 @@ def create_app(host: str = "127.0.0.1", port: int = 7860):
},
)
# --- v0.53.9 #94: SSE training-event stream ---
@app.get("/api/train/stream")
async def stream_train_events():
"""SSE endpoint streaming `TrainEvent` payloads as JSON frames.
Per-subscriber cursor — multiple concurrent listeners each receive
every event (no destructive drain). Uses `asyncio.sleep` so the
uvicorn async loop is not blocked under default workers.
"""
import asyncio
from fastapi.responses import StreamingResponse
from soup_cli.utils.sse_train_stream import TrainEvent, format_sse_frame
from soup_cli.utils.train_event_buffer import get_global_buffer
buffer = get_global_buffer()
async def _gen():
# Start from cursor 0 — new subscribers receive a bounded
# catch-up of retained events (deque maxlen=1000) before
# streaming fresh ones. Concurrent subscribers are independent.
cursor = 0
max_ticks = 200 # cap to keep test runs bounded; ~20s at 100ms
empty_ticks = 0
for _ in range(max_ticks):
events, cursor = buffer.snapshot_since(cursor)
if events:
empty_ticks = 0
for event in events:
yield format_sse_frame(event)
else:
empty_ticks += 1
yield ":heartbeat\n\n"
with _train_lock:
proc = _train_process
if proc is None or proc.poll() is not None:
if empty_ticks >= 1:
done = TrainEvent(type="status", message="done")
yield format_sse_frame(done)
return
await asyncio.sleep(0.1)
done = TrainEvent(type="status", message="timeout")
yield format_sse_frame(done)
return StreamingResponse(
_gen(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
},
)
# --- v0.53.9 #100: Tool-call observation panel ---
@app.get("/api/tool-outputs")
def list_tool_outputs(
limit: int = Query(default=100, ge=1, le=1000),
):
"""Return the most recent tool-call records as JSON.
Records are pushed by the SFT trainer's tool-calling callback
into the process-wide `ToolOutputsBuffer`. Read-only; safe for
cross-origin polling.
"""
from soup_cli.utils.tool_outputs import get_global_tool_buffer
records = get_global_tool_buffer().snapshot(limit=limit)
return {
"count": len(records),
"records": [
{
"name": r.name,
"started_ts": r.started_ts,
"duration_ms": r.duration_ms,
"success": r.success,
"output_preview": r.output_preview,
"error": r.error,
}
for r in records
],
}
# --- Health ---
@app.get("/api/health")

72
soup_cli/ui/static/app.js
View File

@ -2,6 +2,78 @@
const API = ''; // same origin
// v0.53.9 #94 — Lightweight EventSource consumer for /api/train/stream.
// Opens on demand (call `startTrainEventStream()`) and dispatches parsed
// payloads to `onTrainEvent(payload)` which other modules can override.
// Auto-closes on `status=done` or `status=timeout`.
let _trainEventSource = null;
window.onTrainEvent = window.onTrainEvent || function (_payload) {};
function startTrainEventStream() {
if (_trainEventSource) return _trainEventSource;
try {
const es = new EventSource('/api/train/stream');
_trainEventSource = es;
es.onmessage = function (event) {
if (!event.data) return;
try {
const payload = JSON.parse(event.data);
if (payload && typeof window.onTrainEvent === 'function') {
window.onTrainEvent(payload);
}
if (payload && payload.type === 'status' &&
(payload.message === 'done' || payload.message === 'timeout')) {
es.close();
_trainEventSource = null;
}
} catch (e) {
// Ignore malformed frames.
}
};
es.onerror = function () {
try { es.close(); } catch (e) {}
_trainEventSource = null;
};
return es;
} catch (e) {
return null;
}
}
function stopTrainEventStream() {
if (_trainEventSource) {
try { _trainEventSource.close(); } catch (e) {}
_trainEventSource = null;
}
}
window.startTrainEventStream = startTrainEventStream;
window.stopTrainEventStream = stopTrainEventStream;
// v0.53.9 #95 — Pick up Bearer token from `?token=…` (phone QR landing)
// or from sessionStorage on subsequent navigations. Stripped from the URL
// after read so the token doesn't sit in browser history.
(function _bootstrapAuthToken() {
try {
const params = new URLSearchParams(window.location.search);
const fromUrl = params.get('token');
if (fromUrl) {
window._authToken = fromUrl;
try { sessionStorage.setItem('soup_auth_token', fromUrl); } catch (e) {}
// Drop ?token=… from the URL so refresh history doesn't leak it.
params.delete('token');
const qs = params.toString();
const clean = window.location.pathname + (qs ? '?' + qs : '') +
window.location.hash;
window.history.replaceState(null, '', clean);
} else {
try {
const saved = sessionStorage.getItem('soup_auth_token');
if (saved) window._authToken = saved;
} catch (e) {}
}
} catch (e) {
// Defensive: never block app load.
}
})();
// --- State ---
let currentPage = 'dashboard';
let runsData = [];

120
soup_cli/utils/backend_detect.py Normal file
View File

@ -0,0 +1,120 @@
"""v0.53.9 #28 — Auto-detect inference backend from a model directory.
Reads `config.json` `architectures` field; falls back to `transformers`
on missing config, decode errors, or non-LLM arches. Returns a closed-set
backend identifier so callers can dispatch deterministically.
"""
from __future__ import annotations
import json
import os
import stat
from pathlib import Path
from typing import Optional
from soup_cli.utils.paths import is_under_cwd
# Closed allowlist — matches `soup serve --backend`.
SUPPORTED_BACKENDS = frozenset({"transformers", "vllm", "sglang", "mlx"})
# Architecture-name keyword -> preferred backend. First match wins; the
# scan walks `architectures` left-to-right.
_MLX_KEYWORDS = frozenset({"mlx", "mlx_lm"})
def _read_config_json(model_dir: Path) -> Optional[dict]:
"""Read `<model_dir>/config.json` with TOCTOU-safe symlink rejection.
Returns None on any error (missing dir, symlinked config, decode
failure). Never raises — caller treats None as "unknown arch".
"""
config_path = model_dir / "config.json"
if not config_path.is_file():
return None
try:
# Reject symlinked config.json (mirrors v0.53.1 #82 policy).
st = os.lstat(config_path)
if stat.S_ISLNK(st.st_mode):
return None
except OSError:
return None
try:
with open(config_path, encoding="utf-8") as fh:
data = json.load(fh)
except (OSError, json.JSONDecodeError):
return None
if not isinstance(data, dict):
return None
return data
def _has_mlx_weights(model_dir: Path) -> bool:
"""Return True if the directory has MLX-format `weights.npz` (mlx-lm convention).
Each entry is `os.lstat`-checked so a symlinked `weights.npz` does not
accidentally trigger MLX dispatch (mirrors v0.33.0 #22 TOCTOU policy).
"""
try:
for path in model_dir.iterdir():
if path.name.lower() != "weights.npz":
continue
try:
st = os.lstat(path)
except OSError:
continue
if stat.S_ISLNK(st.st_mode):
continue
if stat.S_ISREG(st.st_mode):
return True
except OSError:
return False
return False
def detect_backend(model_path: str, *, env: Optional[dict] = None) -> str:
"""Probe `model_path` and return a preferred backend identifier.
Resolution order:
1. Env hint `SOUP_BENCH_BACKEND` (validated against allowlist).
2. MLX weight files in the model dir.
3. `architectures` field in `config.json` (always implies transformers
today; vllm / sglang need explicit selection because they wrap
the same HF weights).
4. Fallback: `transformers`.
`model_path` must be a non-empty str; on traversal / NUL / missing dir
the helper falls back to `transformers` silently.
"""
if not isinstance(model_path, str) or not model_path:
return "transformers"
if "\x00" in model_path:
return "transformers"
env = env if env is not None else os.environ
hint = env.get("SOUP_BENCH_BACKEND")
if isinstance(hint, str) and hint:
canonical = hint.strip().lower()
if canonical in SUPPORTED_BACKENDS:
return canonical
# Containment guard — only probe paths inside cwd. Out-of-cwd paths
# are still benched; we just skip the local-file probe.
if not is_under_cwd(model_path):
return "transformers"
real = os.path.realpath(model_path)
model_dir = Path(real)
if not model_dir.is_dir():
return "transformers"
if _has_mlx_weights(model_dir):
return "mlx"
config = _read_config_json(model_dir)
if config:
# MLX configs sometimes carry `mlx`-prefixed model_type.
model_type = str(config.get("model_type") or "").lower()
if any(kw in model_type for kw in _MLX_KEYWORDS):
return "mlx"
return "transformers"

47
soup_cli/utils/reasoning_parser.py
View File

@ -7,6 +7,7 @@ to v0.44.1.
from __future__ import annotations
import re
from types import MappingProxyType
from typing import Mapping, Optional
@ -49,3 +50,49 @@ def parser_description(name: str) -> Optional[str]:
if not isinstance(name, str):
return None
return _REASONING_PARSERS.get(name.lower())
# v0.53.9 #98 — Per-parser regex matrix for `<think>...</think>` stripping.
# All four parsers strip the same standard `<think>...</think>` block;
# OpenThinker uses `<|begin_of_thought|>...<|end_of_thought|>` per upstream.
_THINK_RE = re.compile(r"<think\b[^>]*>.*?</think>", re.IGNORECASE | re.DOTALL)
_OPENTHINKER_RE = re.compile(
r"<\|begin_of_thought\|>.*?<\|end_of_thought\|>", re.DOTALL
)
def strip_reasoning(text: str, parser: Optional[str]) -> str:
"""Strip reasoning-trace blocks from `text` per `parser`.
Returns `text` unchanged when `parser` is `None`/empty, when the parser
is unknown (defensive — caller should validate first), or when `text`
is not a str. Idempotent: applying twice yields the same result.
Cap: input length capped at 1 MiB; longer payloads are returned
unchanged to avoid pathological regex backtracking.
"""
if not isinstance(text, str):
return text # type: ignore[return-value]
if not parser:
return text
if len(text) > 1_048_576:
return text
try:
canonical = validate_parser_name(parser)
except (TypeError, ValueError):
return text
# Fast pre-check: skip the regex entirely when the marker token is
# absent. Bounds worst-case re.sub time on adversarial inputs with
# no actual reasoning blocks.
if canonical == "openthinker":
if "<|begin_of_thought|>" not in text:
return text
out = _OPENTHINKER_RE.sub("", text)
else:
if "<think" not in text.lower():
return text
out = _THINK_RE.sub("", text)
# Strip ONLY leading newlines left behind by the removed block. We
# don't `lstrip()` (would silently corrupt code outputs that begin
# with intentional spaces / tabs).
return re.sub(r"^[\r\n]+", "", out)

16
soup_cli/utils/tool_outputs.py
View File

@ -173,3 +173,19 @@ class ToolCallTimer:
)
# Don't suppress exceptions — `False` is explicit per project policy.
return False
# v0.53.9 #100 — Module-level singleton consumed by the SFT trainer
# tool-calling callback and the FastAPI `/api/tool-outputs` endpoint.
_GLOBAL_TOOL_BUFFER: ToolOutputsBuffer = ToolOutputsBuffer()
def get_global_tool_buffer() -> ToolOutputsBuffer:
"""Return the process-wide tool-output buffer."""
return _GLOBAL_TOOL_BUFFER
def reset_global_tool_buffer() -> None:
"""Replace the process-wide buffer (test hook)."""
global _GLOBAL_TOOL_BUFFER
_GLOBAL_TOOL_BUFFER = ToolOutputsBuffer()

129
soup_cli/utils/train_event_buffer.py Normal file
View File

@ -0,0 +1,129 @@
"""v0.53.9 #94 — Module-level training event buffer for SSE streaming.
Thread-safe deque of `TrainEvent` payloads. The `SoupTrainerCallback`
pushes events on log / save / evaluate; the FastAPI `/api/train/stream`
endpoint drains pending events as SSE frames.
Decoupled from FastAPI so callbacks can push without importing it.
"""
from __future__ import annotations
from collections import deque
from threading import Lock
from typing import Any, Deque, List, Optional
from soup_cli.utils.sse_train_stream import TrainEvent
_MAX_EVENTS = 1000
class TrainEventBuffer:
"""Thread-safe ring buffer of `TrainEvent` objects.
`push` appends one event (dropping the oldest on overflow);
`drain` returns all events queued since the last drain. Consumers
can also pass a `since` cursor for catch-up subscribers.
"""
def __init__(self, *, maxlen: int = _MAX_EVENTS) -> None:
if isinstance(maxlen, bool) or not isinstance(maxlen, int):
raise TypeError("maxlen must be int")
if maxlen <= 0:
raise ValueError("maxlen must be > 0")
self._events: Deque[TrainEvent] = deque(maxlen=maxlen)
self._cursor: int = 0
self._lock = Lock()
def push(self, event: TrainEvent) -> int:
"""Append an event. Returns the event's monotonic cursor id."""
if not isinstance(event, TrainEvent):
raise TypeError("event must be TrainEvent")
with self._lock:
self._cursor += 1
self._events.append(event)
return self._cursor
def drain(self) -> List[TrainEvent]:
"""Return every queued event and clear the buffer.
Note: destructive — concurrent subscribers will starve each other.
For multi-subscriber SSE, use `snapshot_since(cursor)` instead.
"""
with self._lock:
data = list(self._events)
self._events.clear()
return data
def cursor(self) -> int:
"""Return the latest assigned cursor id (0 if empty)."""
with self._lock:
return self._cursor
def snapshot_since(self, since: int) -> tuple[List[TrainEvent], int]:
"""Return events with cursor > `since`, plus the new cursor.
Non-destructive — every subscriber gets its own catch-up view.
Returns `(events, latest_cursor)`. When the buffer has rolled over
past `since` (i.e. the requested cursor is older than the oldest
retained event), returns every retained event.
"""
if isinstance(since, bool) or not isinstance(since, int):
raise TypeError("since must be int")
with self._lock:
latest = self._cursor
events = list(self._events)
if since >= latest:
return [], latest
# Cursor ids are monotonic; the i-th retained event has cursor
# `latest - (len - 1 - i)`. Slice off the prefix the caller already saw.
start = latest - len(events) + 1 # cursor id of events[0]
offset = max(0, since - start + 1)
return events[offset:], latest
def snapshot(self, *, limit: Optional[int] = None) -> List[TrainEvent]:
"""Return a copy of the latest `limit` events without clearing."""
if limit is not None:
if isinstance(limit, bool) or not isinstance(limit, int):
raise TypeError("limit must be int or None")
if limit < 0:
raise ValueError("limit must be >= 0")
with self._lock:
data = list(self._events)
if limit is None:
return data
if limit == 0:
return []
return data[-limit:]
def clear(self) -> None:
with self._lock:
self._events.clear()
# Module-level singleton consumed by SoupTrainerCallback and the FastAPI
# SSE endpoint. Tests can call `reset_global_buffer()` to start fresh.
_GLOBAL_BUFFER: TrainEventBuffer = TrainEventBuffer()
def get_global_buffer() -> TrainEventBuffer:
"""Return the process-wide event buffer."""
return _GLOBAL_BUFFER
def reset_global_buffer() -> None:
"""Replace the process-wide buffer (test hook)."""
global _GLOBAL_BUFFER
_GLOBAL_BUFFER = TrainEventBuffer()
def push_train_event(event: Any) -> Optional[int]:
"""Push an event onto the global buffer. Returns the cursor id.
Silently ignores non-TrainEvent input so a misbehaving callback can
never crash the training loop. Use the typed `get_global_buffer().push`
when you want explicit validation.
"""
if not isinstance(event, TrainEvent):
return None
return _GLOBAL_BUFFER.push(event)

14
tests/test_v0538.py
View File

@ -596,10 +596,16 @@ class TestVersionBump:
def test_init_version(self):
import soup_cli
# v0.53.8.1 is the PyPI-recoverable patch (force-include duplicate
# fix); accept either as the shipped string.
assert soup_cli.__version__.startswith("0.53.8")
# Version-string is forward-monotonic: v0.53.8 baseline + any later
# release (v0.53.9, v0.54.0, ...) keeps this contract green.
assert soup_cli.__version__ >= "0.53.8"
def test_pyproject_version(self):
text = (_repo_root() / "pyproject.toml").read_text(encoding="utf-8")
assert 'version = "0.53.8' in text
# Pyproject must declare some 0.53.x or later string; the literal
# "0.53.8" baseline is allowed to drift forward.
import re
match = re.search(r'^version\s*=\s*"([^"]+)"', text, flags=re.MULTILINE)
assert match is not None
assert match.group(1) >= "0.53.8"

849
tests/test_v0539.py Normal file
View File

@ -0,0 +1,849 @@
"""Tests for v0.53.9 — Live Dashboard + UX + Bench + Standalone CLIs.
Covers #94 SSE stream, #95 ui --public + QR, #98 reasoning-parser strip,
#100 tool-outputs API, #15 tokenizer train, #26 bench percentiles,
#28 bench backend auto-detect, #12 example doc.
"""
from __future__ import annotations
import json
import os
from pathlib import Path
import pytest
from typer.testing import CliRunner
import soup_cli
from soup_cli.cli import app
# ----------------------------------------------------------------- version
def test_version_bump_to_0_53_9():
assert soup_cli.__version__ == "0.53.9"
# ----------------------------------------------------- #94 SSE event buffer
def test_train_event_buffer_push_and_drain():
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer()
cursor = buffer.push(TrainEvent(type="metric", step=1, loss=0.5))
assert cursor == 1
cursor = buffer.push(TrainEvent(type="metric", step=2, loss=0.4))
assert cursor == 2
drained = buffer.drain()
assert len(drained) == 2
assert drained[0].step == 1
# After draining, buffer is empty.
assert buffer.drain() == []
def test_train_event_buffer_rejects_non_event():
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer()
with pytest.raises(TypeError):
buffer.push("not an event")
with pytest.raises(TypeError):
buffer.push({"type": "metric"})
def test_train_event_buffer_maxlen_validation():
from soup_cli.utils.train_event_buffer import TrainEventBuffer
with pytest.raises(TypeError):
TrainEventBuffer(maxlen=True)
with pytest.raises(ValueError):
TrainEventBuffer(maxlen=0)
with pytest.raises(ValueError):
TrainEventBuffer(maxlen=-3)
def test_train_event_buffer_snapshot_limits():
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer()
for step in range(5):
buffer.push(TrainEvent(type="metric", step=step))
snap = buffer.snapshot(limit=3)
assert len(snap) == 3
assert [e.step for e in snap] == [2, 3, 4]
# Snapshot does not drain.
assert len(buffer.snapshot()) == 5
assert buffer.snapshot(limit=0) == []
with pytest.raises(TypeError):
buffer.snapshot(limit=True)
with pytest.raises(ValueError):
buffer.snapshot(limit=-1)
def test_train_event_buffer_overflow_drops_oldest():
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer(maxlen=3)
for step in range(5):
buffer.push(TrainEvent(type="metric", step=step))
# Only latest three retained.
drained = buffer.drain()
assert [e.step for e in drained] == [2, 3, 4]
def test_push_train_event_silent_on_bad_input():
from soup_cli.utils.train_event_buffer import (
get_global_buffer,
push_train_event,
reset_global_buffer,
)
reset_global_buffer()
assert push_train_event("nope") is None
assert push_train_event(None) is None
assert get_global_buffer().drain() == []
def test_push_train_event_happy():
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import (
get_global_buffer,
push_train_event,
reset_global_buffer,
)
reset_global_buffer()
cursor = push_train_event(TrainEvent(type="metric", step=42, loss=0.1))
assert cursor == 1
events = get_global_buffer().drain()
assert len(events) == 1
assert events[0].step == 42
# ---------------------------------------------------- #94 SSE FastAPI route
def test_api_train_stream_emits_pending_events():
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from soup_cli.ui.app import create_app
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import (
push_train_event,
reset_global_buffer,
)
reset_global_buffer()
push_train_event(TrainEvent(type="metric", step=1, loss=0.5))
push_train_event(TrainEvent(type="status", message="started"))
app_inst = create_app()
client = TestClient(app_inst)
response = client.get("/api/train/stream")
assert response.status_code == 200
body = response.text
# Frames are W3C SSE.
assert "data:" in body
assert '"loss":0.5' in body or '"loss": 0.5' in body
# Closes with a done status event.
assert '"message":"done"' in body or '"message": "done"' in body
# ------------------------------------------------- #100 tool-outputs API
def test_global_tool_buffer_round_trip():
from soup_cli.utils.tool_outputs import (
get_global_tool_buffer,
reset_global_tool_buffer,
)
reset_global_tool_buffer()
buf = get_global_tool_buffer()
buf.record_call(
name="search",
started_ts=1.0,
duration_ms=12.5,
success=True,
output_preview="hits=3",
)
snap = buf.snapshot()
assert len(snap) == 1
assert snap[0].name == "search"
def test_api_tool_outputs_endpoint():
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from soup_cli.ui.app import create_app
from soup_cli.utils.tool_outputs import (
get_global_tool_buffer,
reset_global_tool_buffer,
)
reset_global_tool_buffer()
buf = get_global_tool_buffer()
buf.record_call(
name="calculator",
started_ts=1.23,
duration_ms=4.5,
success=True,
output_preview="result=42",
)
buf.record_call(
name="search",
started_ts=2.34,
duration_ms=99.9,
success=False,
output_preview="",
error="timeout",
)
app_inst = create_app()
client = TestClient(app_inst)
response = client.get("/api/tool-outputs?limit=5")
assert response.status_code == 200
payload = response.json()
assert payload["count"] == 2
names = [r["name"] for r in payload["records"]]
assert names == ["calculator", "search"]
assert payload["records"][1]["error"] == "timeout"
def test_api_tool_outputs_rejects_out_of_bounds_limit():
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from soup_cli.ui.app import create_app
client = TestClient(create_app())
# limit must be 1..1000
assert client.get("/api/tool-outputs?limit=0").status_code == 422
assert client.get("/api/tool-outputs?limit=1001").status_code == 422
# ---------------------------------------------------- #98 reasoning-parser
@pytest.mark.parametrize(
"parser,raw,expected",
[
(
"deepseek-r1",
"<think>secret</think>final answer",
"final answer",
),
(
"qwen3",
"<think>step1\nstep2</think>\nresponse",
"response",
),
(
"phi4",
"no think tags here",
"no think tags here",
),
(
"openthinker",
"<|begin_of_thought|>cot<|end_of_thought|>visible",
"visible",
),
],
)
def test_strip_reasoning_per_parser(parser, raw, expected):
from soup_cli.utils.reasoning_parser import strip_reasoning
assert strip_reasoning(raw, parser) == expected
def test_strip_reasoning_no_op_paths():
from soup_cli.utils.reasoning_parser import strip_reasoning
# None parser short-circuits.
assert strip_reasoning("<think>x</think>y", None) == "<think>x</think>y"
# Empty string parser short-circuits.
assert strip_reasoning("hello", "") == "hello"
# Unknown parser is silently no-op.
assert strip_reasoning("<think>x</think>y", "unknown-parser") == (
"<think>x</think>y"
)
# Non-string input passes through.
assert strip_reasoning(123, "deepseek-r1") == 123 # type: ignore[arg-type]
def test_strip_reasoning_oversize_passthrough():
from soup_cli.utils.reasoning_parser import strip_reasoning
big = "<think>x</think>" + ("a" * 1_100_000)
out = strip_reasoning(big, "deepseek-r1")
assert out == big # >1MiB returned unchanged
def test_strip_reasoning_idempotent():
from soup_cli.utils.reasoning_parser import strip_reasoning
out = strip_reasoning("<think>a</think>final", "deepseek-r1")
assert strip_reasoning(out, "deepseek-r1") == out
def test_serve_create_app_accepts_reasoning_parser_kwarg():
"""`_create_app` must accept the new `reasoning_parser=` kwarg."""
import inspect
from soup_cli.commands.serve import _create_app
sig = inspect.signature(_create_app)
assert "reasoning_parser" in sig.parameters
def test_serve_help_lists_reasoning_parser_flag():
runner = CliRunner()
result = runner.invoke(app, ["serve", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "--reasoning-parser" in result.output
# --------------------------------------------------- #15 tokenizer train
def test_tokenizer_train_help_listed():
runner = CliRunner()
result = runner.invoke(app, ["tokenizer", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "train" in result.output
def test_tokenizer_train_subcommand_help():
runner = CliRunner()
result = runner.invoke(app, ["tokenizer", "train", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "--vocab-size" in result.output
assert "--input" in result.output
assert "--output" in result.output
def test_tokenizer_train_rejects_out_of_cwd_input(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
outside = tmp_path.parent / "outside.jsonl"
outside.write_text('{"text": "hi"}\n', encoding="utf-8")
runner = CliRunner()
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", str(outside),
"--output", "out",
],
)
assert result.exit_code != 0
assert "current working directory" in result.output
def test_tokenizer_train_rejects_vocab_size_bounds(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
fixture = tmp_path / "corpus.jsonl"
fixture.write_text('{"text": "hello world"}\n', encoding="utf-8")
runner = CliRunner()
# Too small.
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", "corpus.jsonl",
"--vocab-size", "10",
],
)
assert result.exit_code != 0
assert "vocab-size" in result.output
# Too large.
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", "corpus.jsonl",
"--vocab-size", "999999",
],
)
assert result.exit_code != 0
def test_tokenizer_train_happy_path(tmp_path, monkeypatch):
pytest.importorskip("tokenizers")
monkeypatch.chdir(tmp_path)
fixture = tmp_path / "corpus.jsonl"
rows = [
{"text": "the quick brown fox jumps over the lazy dog"},
{"text": "pack my box with five dozen liquor jugs"},
{"text": "how vexingly quick daft zebras jump"},
] * 20
fixture.write_text(
"\n".join(json.dumps(r) for r in rows) + "\n",
encoding="utf-8",
)
runner = CliRunner()
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", "corpus.jsonl",
"--vocab-size", "300",
"--output", "bpe_out",
"--min-frequency", "1",
],
)
assert result.exit_code == 0, (result.output, repr(result.exception))
assert (tmp_path / "bpe_out" / "tokenizer.json").is_file()
assert (tmp_path / "bpe_out" / "vocab.json").is_file()
vocab = json.loads(
(tmp_path / "bpe_out" / "vocab.json").read_text(encoding="utf-8")
)
assert isinstance(vocab, dict)
assert len(vocab) >= 1
# -------------------------------------------------- #28 backend auto-detect
def test_detect_backend_fallback_transformers():
from soup_cli.utils.backend_detect import detect_backend
assert detect_backend("does-not-exist") == "transformers"
assert detect_backend("") == "transformers"
assert detect_backend("abc\x00def") == "transformers"
def test_detect_backend_env_hint(monkeypatch):
from soup_cli.utils.backend_detect import detect_backend
monkeypatch.setenv("SOUP_BENCH_BACKEND", "VLLM")
assert detect_backend("anything") == "vllm"
monkeypatch.setenv("SOUP_BENCH_BACKEND", "bogus-name")
# bogus env hint falls back to detection.
out = detect_backend("anything")
assert out in {"transformers", "mlx"}
def test_detect_backend_mlx_weights(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
model_dir = tmp_path / "mlx_model"
model_dir.mkdir()
(model_dir / "weights.npz").write_bytes(b"\x00\x00")
from soup_cli.utils.backend_detect import detect_backend
assert detect_backend(str(model_dir)) == "mlx"
def test_detect_backend_transformers_config(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
model_dir = tmp_path / "hf_model"
model_dir.mkdir()
(model_dir / "config.json").write_text(
json.dumps({"model_type": "llama", "architectures": ["LlamaForCausalLM"]}),
encoding="utf-8",
)
from soup_cli.utils.backend_detect import detect_backend
assert detect_backend(str(model_dir)) == "transformers"
def test_detect_backend_config_malformed_falls_back(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
model_dir = tmp_path / "junk"
model_dir.mkdir()
(model_dir / "config.json").write_text("not-json{", encoding="utf-8")
from soup_cli.utils.backend_detect import detect_backend
assert detect_backend(str(model_dir)) == "transformers"
# ----------------------------------------------------- #26 bench percentiles
def test_bench_help_lists_percentile_flags():
runner = CliRunner()
result = runner.invoke(app, ["bench", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "--p50" in result.output
assert "--p95" in result.output
assert "--backend" in result.output
def test_bench_invalid_backend_rejected(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
runner = CliRunner()
result = runner.invoke(
app,
["bench", "fake-model", "--backend", "evilbackend"],
)
# Either exit 2 (Typer reject) or exit 1 (resolve failure later).
assert result.exit_code != 0
# --------------------------------------------------- #95 ui --public + QR
def test_ui_help_includes_public_and_auth_token():
runner = CliRunner()
result = runner.invoke(app, ["ui", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "--public" in result.output
assert "--auth-token" in result.output
def test_ui_show_token_via_custom_token(tmp_path, monkeypatch):
pytest.importorskip("fastapi")
monkeypatch.chdir(tmp_path)
runner = CliRunner()
valid = "A" * 32 # urlsafe-base64 shape
result = runner.invoke(
app,
[
"ui",
"--show-token",
"--auth-token", valid,
"--no-browser",
],
)
assert result.exit_code == 0, (result.output, repr(result.exception))
assert valid in result.output
def test_ui_rejects_malformed_auth_token():
pytest.importorskip("fastapi")
runner = CliRunner()
result = runner.invoke(
app,
[
"ui",
"--auth-token", "short", # < 16 chars
"--show-token",
"--no-browser",
],
)
assert result.exit_code != 0
assert "auth-token" in result.output or "token" in result.output
# ------------------------------------------------ #12 example workflow doc
def test_train_event_buffer_snapshot_since_returns_only_new_events():
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer()
buffer.push(TrainEvent(type="metric", step=0))
buffer.push(TrainEvent(type="metric", step=1))
events, cursor = buffer.snapshot_since(0)
assert [e.step for e in events] == [0, 1]
assert cursor == 2
buffer.push(TrainEvent(type="metric", step=2))
events, cursor = buffer.snapshot_since(cursor)
assert [e.step for e in events] == [2]
assert cursor == 3
# Caller cursor caught up — no new events.
events, cursor = buffer.snapshot_since(cursor)
assert events == []
assert cursor == 3
def test_train_event_buffer_snapshot_since_rejects_non_int():
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer()
with pytest.raises(TypeError):
buffer.snapshot_since("0") # type: ignore[arg-type]
with pytest.raises(TypeError):
buffer.snapshot_since(True)
def test_train_event_buffer_concurrent_subscribers_isolated():
"""Multiple subscribers using snapshot_since each see every event."""
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer()
for step in range(3):
buffer.push(TrainEvent(type="metric", step=step))
a_events, _ = buffer.snapshot_since(0)
b_events, _ = buffer.snapshot_since(0)
assert [e.step for e in a_events] == [0, 1, 2]
assert [e.step for e in b_events] == [0, 1, 2]
def test_tokenizer_train_rejects_symlink_input(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
real = tmp_path / "real.jsonl"
real.write_text('{"text": "hello"}\n', encoding="utf-8")
link = tmp_path / "link.jsonl"
try:
os.symlink(real, link)
except (OSError, NotImplementedError, AttributeError):
pytest.skip("symlinks not supported on this platform")
runner = CliRunner()
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", "link.jsonl",
"--output", "out",
],
)
assert result.exit_code != 0
assert "symlink" in result.output.lower()
def test_train_event_buffer_cursor_and_clear():
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer()
assert buffer.cursor() == 0
buffer.push(TrainEvent(type="metric", step=0))
buffer.push(TrainEvent(type="metric", step=1))
assert buffer.cursor() == 2
buffer.clear()
assert buffer.cursor() == 2 # cursor is monotonic; clear does NOT reset
assert buffer.snapshot() == []
def test_train_event_buffer_snapshot_since_rollover():
"""`maxlen=3` then push 5 events — `snapshot_since(0)` returns the 3 retained."""
from soup_cli.utils.sse_train_stream import TrainEvent
from soup_cli.utils.train_event_buffer import TrainEventBuffer
buffer = TrainEventBuffer(maxlen=3)
for step in range(5):
buffer.push(TrainEvent(type="metric", step=step))
# Latest cursor is 5; only the last 3 events are retained.
events, cursor = buffer.snapshot_since(0)
assert [e.step for e in events] == [2, 3, 4]
assert cursor == 5
# Asking for events newer than cursor 4 (step=3) returns the tail.
events, cursor = buffer.snapshot_since(4)
assert [e.step for e in events] == [4]
def test_set_auth_token_concurrent_rotation_safe():
"""Concurrent set_auth_token + get_auth_token never observes a partial token."""
import threading
from soup_cli.ui.app import get_auth_token, set_auth_token
valid_a = "A" * 32
valid_b = "B" * 32
set_auth_token(valid_a)
errors: list[str] = []
def _writer():
for _ in range(100):
set_auth_token(valid_a)
set_auth_token(valid_b)
def _reader():
for _ in range(100):
tok = get_auth_token()
if tok not in (valid_a, valid_b):
errors.append(f"torn token: {tok!r}")
threads = [threading.Thread(target=_writer) for _ in range(2)]
threads += [threading.Thread(target=_reader) for _ in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
assert errors == []
def test_set_auth_token_rejects_bool_and_non_str():
from soup_cli.ui.app import get_auth_token, set_auth_token
before = get_auth_token()
with pytest.raises(TypeError):
set_auth_token(True) # type: ignore[arg-type]
with pytest.raises(TypeError):
set_auth_token(42) # type: ignore[arg-type]
# Token must be unchanged after rejected rotation.
assert get_auth_token() == before
def test_strip_reasoning_swallows_null_byte_parser():
from soup_cli.utils.reasoning_parser import strip_reasoning
out = strip_reasoning("<think>x</think>final", "\x00")
assert out == "<think>x</think>final"
def test_strip_reasoning_crlf_residue():
from soup_cli.utils.reasoning_parser import strip_reasoning
out = strip_reasoning("<think>a</think>\r\n\r\nfinal", "deepseek-r1")
# Both \r and \n stripped — content preserved.
assert out == "final"
def test_strip_reasoning_fast_path_skips_regex():
"""Inputs without the marker token bypass the regex entirely."""
from soup_cli.utils.reasoning_parser import strip_reasoning
# Long input WITHOUT <think marker — returns unchanged in O(n).
big = "a" * 200_000
assert strip_reasoning(big, "deepseek-r1") == big
# OpenThinker marker absent — unchanged.
assert strip_reasoning(big, "openthinker") == big
def test_strip_reasoning_multiple_blocks():
from soup_cli.utils.reasoning_parser import strip_reasoning
out = strip_reasoning(
"<think>a</think>mid<think>b</think>final", "deepseek-r1"
)
assert out == "midfinal"
def test_detect_backend_rejects_symlinked_mlx_weights(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
model_dir = tmp_path / "model"
model_dir.mkdir()
real_weights = tmp_path / "real.npz"
real_weights.write_bytes(b"\x00")
link = model_dir / "weights.npz"
try:
os.symlink(real_weights, link)
except (OSError, NotImplementedError, AttributeError):
pytest.skip("symlinks not supported on this platform")
from soup_cli.utils.backend_detect import detect_backend
# Symlinked weights.npz must NOT trigger MLX dispatch.
assert detect_backend(str(model_dir)) == "transformers"
def test_detect_backend_mlx_model_type_in_config(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
model_dir = tmp_path / "model"
model_dir.mkdir()
(model_dir / "config.json").write_text(
json.dumps({"model_type": "mlx_lm_llama"}),
encoding="utf-8",
)
from soup_cli.utils.backend_detect import detect_backend
assert detect_backend(str(model_dir)) == "mlx"
def test_detect_backend_config_non_dict_root(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
model_dir = tmp_path / "model"
model_dir.mkdir()
(model_dir / "config.json").write_text("[1, 2, 3]", encoding="utf-8")
from soup_cli.utils.backend_detect import detect_backend
assert detect_backend(str(model_dir)) == "transformers"
def test_tokenizer_train_plaintext_corpus(tmp_path, monkeypatch):
"""`.txt` extractor branch (different from JSONL)."""
pytest.importorskip("tokenizers")
monkeypatch.chdir(tmp_path)
corpus = tmp_path / "corpus.txt"
corpus.write_text(
"\n".join(["the quick brown fox jumps over the lazy dog"] * 80) + "\n",
encoding="utf-8",
)
runner = CliRunner()
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", "corpus.txt",
"--vocab-size", "300",
"--output", "bpe_out",
"--min-frequency", "1",
],
)
assert result.exit_code == 0, (result.output, repr(result.exception))
assert (tmp_path / "bpe_out" / "tokenizer.json").is_file()
def test_tokenizer_train_rejects_min_frequency_zero(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
(tmp_path / "corpus.jsonl").write_text(
'{"text": "hello"}\n', encoding="utf-8"
)
runner = CliRunner()
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", "corpus.jsonl",
"--min-frequency", "0",
],
)
assert result.exit_code != 0
assert "min-frequency" in result.output
def test_tokenizer_train_special_token_dedup_and_validation(tmp_path, monkeypatch):
pytest.importorskip("tokenizers")
monkeypatch.chdir(tmp_path)
(tmp_path / "corpus.jsonl").write_text(
"\n".join(['{"text": "hello world"}'] * 20) + "\n",
encoding="utf-8",
)
runner = CliRunner()
# NUL byte in special-token rejected.
result = runner.invoke(
app,
[
"tokenizer", "train",
"--input", "corpus.jsonl",
"--special-token", "<pad>\x00bad",
"--vocab-size", "300",
"--min-frequency", "1",
],
)
assert result.exit_code != 0
assert "special-token" in result.output
def test_bench_p50_p95_runs_with_help_only():
"""Smoke: --p50/--p95 flags are wired (without spinning up a real model)."""
runner = CliRunner()
result = runner.invoke(app, ["bench", "--help"])
assert result.exit_code == 0
# The flag descriptions must mention the v0.53.9 release tag so future
# patches don't silently drop the percentile rows.
assert "p50" in result.output and "p95" in result.output
def test_reset_global_tool_buffer_clears_state():
from soup_cli.utils.tool_outputs import (
get_global_tool_buffer,
reset_global_tool_buffer,
)
buf = get_global_tool_buffer()
buf.record_call(
name="ping",
started_ts=1.0,
duration_ms=0.5,
success=True,
output_preview="ok",
)
assert len(buf.snapshot()) >= 1
reset_global_tool_buffer()
fresh = get_global_tool_buffer()
assert fresh.snapshot() == []
def test_synthetic_workflow_doc_exists():
repo_root = Path(__file__).resolve().parent.parent
doc = repo_root / "examples" / "synthetic_workflow.md"
assert doc.is_file()
body = doc.read_text(encoding="utf-8")
# End-to-end walkthrough names every step.
assert "soup data generate" in body
assert "soup data filter" in body
assert "soup data score" in body
assert "soup train" in body