diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2659bea..626718b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (188 files, 8257 tests) +tests/ - Test suite (189 files, 8285 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index e44c12f..5bf3742 100644 --- a/README.md +++ b/README.md @@ -42,15 +42,17 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.53.8 — Remote data + Hubs + Trackers (wave 2)**: Six features wired live across cloud storage, alternative model hubs, experiment trackers, and HF Spaces. +**v0.53.9 — Live Dashboard + UX + Bench + Standalone CLIs**: Eight features that close out the v0.44.x live-monitoring deferrals plus a long tail of standalone CLI wins. -- **fsspec live loaders for remote datasets.** `data.train: s3://bucket/data.jsonl` (plus `gs://` / `gcs://` / `az://` / `abfs://` / `abfss://` / `oci://`) now loads through `fsspec` with the v0.42.0 SSRF-hardened URI validator (bucket regex + userinfo/query/fragment rejection). Threads `data.streaming` (HF `load_dataset(streaming=True)`) and `data.buffer_size` shuffle. Friendly Rich panel names the `pip install ` advisory when the backend SDK is missing. Row-count capped at 1M to defend against pathological remote objects. -- **Live ModelScope + Modelers hub dispatcher.** New `utils/hubs.download_repo()` and `upload_repo()` lazy-import the matching SDK per backend (`huggingface_hub`, `modelscope`, `openmind-hub`). Shared `_validate_repo_id_shape` rejects bool / null-byte / leading-slash / `..` segments / control characters / oversize; `local_dir` and `folder_path` containment-checked under cwd. `soup train` now pre-fetches non-HF `base` models into `.soup_hub_cache//` and reuses cached snapshots on resumed runs. -- **`[trackers]` extra + friendly missing-dep advisory.** New `pip install soup-cli[trackers]` bundles MLflow + SwanLab + Trackio. When you pass `--tracker mlflow` without the package installed, `soup train` now surfaces a clear `pip install` advisory before construction instead of HF Trainer's generic ImportError. The probe uses `importlib.util.find_spec` (non-executing) so swanlab can't initialise network threads during the check. -- **PostHog telemetry network (opt-IN, silent-fail).** `utils/trackers.send_telemetry_payload()` lazy-imports `httpx` and POSTs hardware-info-only payloads with a 1-second hard timeout. HTTPS-only endpoint check goes through the same SSRF validator as hub endpoints (private-IP / link-local / RFC1918 rejected). Disabled unless `SOUP_TELEMETRY=1`; every exception is swallowed so telemetry can never crash training. -- **Bundled demo fixtures as package data.** `soup data demo` fixtures (4 JSONLs) migrated from `examples/data/` to `soup_cli/data/_fixtures/` so they ship inside the wheel — zipapp / namespace-package safe. `_bundle_source_path` falls back to the legacy `examples/data/` location for editable installs. -- **HF Space SDK auto-pick from `requirements.txt`.** `soup deploy hf-space --template-dir ` now reads the rendered `requirements.txt` and picks `space_sdk="streamlit"` when the file lists `streamlit`, `"gradio"` otherwise. Closes the v0.40.2 known limitation that custom templates always created Spaces with `space_sdk="gradio"`. -- **+95 net new tests** (8162 → 8257) in `test_v0538.py`. Three review agents ran (python / code / security); 16 findings fixed: cwd-containment on `local_dir`/`folder_path`, Pydantic `model_copy(update=...)` instead of attribute mutation, idempotent train pre-fetch via cache probe, row-count cap on remote materialisation, `modelscope.push_model` kwarg fix, SSRF re-validation on telemetry endpoint override, `detect_space_sdk` size cap, and more. +- **Live `/api/train/stream` SSE endpoint.** Async FastAPI route streams `TrainEvent` payloads (loss / lr / grad_norm / EMA / p95 tail latency) as W3C SSE frames. Per-subscriber cursor — multiple dashboard tabs each receive every event without starving each other. `asyncio.sleep` so uvicorn's async loop is never blocked. Drains gracefully with a `status=done` event when training exits. +- **`soup ui --public` + phone-scannable QR.** Binds `0.0.0.0`, detects the actual LAN IP via a `socket.SOCK_DGRAM` connect trick (no packets sent), and prints an ASCII QR encoding `http://:/?token=`. The SPA picks up `?token=` on load via `URLSearchParams`, caches it in `sessionStorage`, and rewrites the URL clean so the token doesn't sit in browser history. `--auth-token` override accepts a stable urlsafe-base64 token for repeatable phone bookmarks. CORS regex auto-widens to loopback + RFC1918 ranges when public; Bearer token gates mutating endpoints. Concurrent token rotation is now lock-protected. +- **`soup serve --reasoning-parser {deepseek-r1|qwen3|phi4|openthinker}`.** Strips `...` (or `<|begin_of_thought|>...<|end_of_thought|>` for OpenThinker) from responses before they hit the client. Pre-compiled regex with a marker-token fast-path (skips `re.sub` entirely when no marker is present) and a 1 MiB input cap. Strips only leading newlines after removal so code outputs that begin with intentional whitespace are preserved. +- **`/api/tool-outputs` + `ToolOutputsBuffer` global singleton.** Process-wide thread-safe ring buffer of tool-call records (`name` / `started_ts` / `duration_ms` / `success` / `output_preview` / `error`). New `/api/tool-outputs?limit=N` JSON endpoint reads the buffer for live dashboard polling. SFT trainer-side observation hook lands in a follow-up. +- **`soup tokenizer train`.** New BPE training CLI via the `tokenizers` library. JSONL or plaintext input (with ShareGPT `messages` fallback), `--vocab-size` bounded [256, 200000], `--special-token` repeatable with NUL/oversize/dedup validation. `os.lstat + S_ISLNK` rejection on the raw user-supplied input path BEFORE realpath, 50 MiB total / 8 KiB per-line caps, post-`mkdir` re-check on the output directory so a planted symlink can't redirect the write. +- **`soup bench --p50 --p95`.** Per-prompt latency capture rendered as an extra Rich table after the throughput row. Reuses `utils/tail_latency.summarise_latency` (v0.44.0). `--prompts-file` now rejects symlinked inputs on the raw path before realpath, in line with the project-wide TOCTOU policy. +- **`soup bench --backend auto`.** New default — probes the model directory for MLX `weights.npz` and falls back to `config.json` `model_type` keyword detection. `SOUP_BENCH_BACKEND` env hint overrides the probe. Every iterdir entry is `os.lstat`-checked so a symlinked `weights.npz` cannot trigger MLX dispatch. +- **`examples/synthetic_workflow.md` walkthrough.** End-to-end docs covering `soup data generate` → `filter` → `score` → `decontaminate` → `train`, plus the `soup ui --public` phone-monitoring loop. +- **+28 net new tests** (8257 → 8285) in `test_v0539.py`. Three review agents ran (python / code / security / tdd / verification); 34 findings fixed: H1 (QR token never consumed by SPA — now hydrates from URL into sessionStorage), H2 (`set_auth_token` rotation race — now lock-protected), tokenizer input/output symlink rejection (raw-path lstat), prompts-file symlink rejection, regex fast-path skip on missing markers, leading-newline strip vs full `lstrip()`, CORS regex for `--public` LAN mode, `_has_mlx_weights` per-entry lstat, and more. ## Why Soup? @@ -3095,6 +3097,11 @@ soup tui Full-screen Textual dashboard (req soup train --config soup.yaml --profile Record torch.profiler trace to /profiles/ soup --log-level quiet|normal|verbose|debug Global logging tier (Rich-formatted) soup ui [--port 7860] Web UI (experiments, training, data) +soup ui --public [--auth-token T] Phone-scannable Web UI (v0.53.9) +soup tokenizer train --input c.jsonl --vocab-size N Train BPE tokenizer (v0.53.9) +soup bench --p50 --p95 Bench with tail-latency percentiles (v0.53.9) +soup bench --backend auto Auto-detect transformers/mlx backend (v0.53.9) +soup serve --reasoning-parser deepseek-r1 Strip blocks from responses (v0.53.9) soup doctor Check environment soup quickstart [--dry-run] Full demo soup version [--full] [--json] Show version (--full: system info, --json: JSON output) diff --git a/SECURITY.md b/SECURITY.md index 298e6bf..7af37b6 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.53.8 -- Full support (latest) +- v0.53.9 -- Full support (latest) +- v0.53.8 -- Full support - v0.53.7 -- Full support - v0.53.6 -- Full support - v0.53.5 -- Full support @@ -154,6 +155,7 @@ No known critical vulnerabilities in current releases. - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input. - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS` +- **v0.53.9 — Live Dashboard + UX + Bench + Standalone CLIs**: 8 features. 3 review-agent waves (python / code / security / tdd) ran; 34 findings fixed (11 HIGH + 14 MEDIUM + 9 LOW). (#95 `set_auth_token` rotation race) module-level `_auth_token_lock: threading.Lock` now guards every read AND write of `_auth_token` in `ui/app.py`; `_verify_token` snapshots the expected `f"Bearer {_auth_token}"` under the lock before string comparison so concurrent rotation can never leave a window where some requests see the old token and some the new (security-review HIGH H2). 8-thread rotation stress test in `test_v0539.py`. (#95 QR token consumption) JS bootstrap in `ui/static/app.js` reads `?token=` from `URLSearchParams` on first load, populates `window._authToken`, caches in `sessionStorage`, and `history.replaceState`s the URL clean of the token so it does not sit in browser history (security-review HIGH H1 — token in QR URL was previously unreachable). (#95 CORS for `--public` LAN mode) when `host == "0.0.0.0"` the `CORSMiddleware` switches from a fixed `allow_origins=[f"http://{host}:{port}"]` (which never matches a real `Origin` header) to an `allow_origin_regex` covering loopback + RFC1918 ranges (`10.x.x.x` / `192.168.x.x` / `172.16-31.x.x`); Bearer token remains the actual security gate (security-review MEDIUM M5). (#15 `tokenizer.py` symlink rejection) `_validate_under_cwd` runs `os.lstat + stat.S_ISLNK` on the RAW user-supplied path BEFORE `os.path.realpath` resolution — `realpath`-first would silently follow symlinks and lstat the resolved regular-file target (security-review HIGH; matches v0.53.7 #106 policy). Post-`mkdir` re-check on the output directory defends against a symlink planted between containment check and write (security-review MEDIUM M1; matches v0.43.0 Part D `copy_bundle_to`). 50 MiB total + 8 KiB per-line corpus caps bound RAM on adversarial input (security-review HIGH). Panel + success messages use `.name` (basename only) to avoid leaking absolute paths (security-review MEDIUM M4; v0.34.0 `crash.py` policy). (#26 bench `--prompts-file` symlink) `os.lstat + S_ISLNK` on the raw path BEFORE realpath (security-review MEDIUM M2; parity with project TOCTOU policy). (#28 `_has_mlx_weights` symlink) per-entry `os.lstat` + `S_ISLNK` rejection so a symlinked `weights.npz` does NOT trigger MLX dispatch (code-review MEDIUM). (#98 `strip_reasoning` ReDoS bound) fast-path skip when the marker token is absent in `text.lower()` — bounds worst-case `re.sub` time on adversarial inputs without actual reasoning blocks; 1 MiB input cap; strips ONLY leading newlines after block removal so code outputs that begin with intentional spaces / tabs are preserved (security-review MEDIUM M3). (#94 SSE drain-race fix) `/api/train/stream` switched from destructive `buffer.drain()` to per-subscriber `snapshot_since(cursor)` so multiple concurrent dashboard tabs each receive every event (code-review HIGH); switched from `time.sleep(0.1)` in a sync generator to `asyncio.sleep(0.1)` in an `async def` generator so uvicorn's async loop is never blocked (code-review HIGH). (#94 `TrainEventBuffer` typing) closed allowlist + `TrainEvent`-only push validation; module-level singleton + test-hook `reset_global_buffer` is documented as test-only because reassignment under no lock is acceptable only when no training loop is active. (#100 `/api/tool-outputs`) `limit` Pydantic-bounded `[1, 1000]`; output preview already 4 KiB-capped by `ToolOutputsBuffer.record_call` (v0.44.0). Cross-cutting: 0 CRITICAL findings; test surface `test_v0539.py` carries 57 new tests including symlink rejection regressions, concurrent rotation stress, and SSE cursor rollover math. (v0.53.9) - **v0.53.8 — Remote data + Hubs + Trackers (wave 2)**: 6 features wired live. 3 review-agent waves (python / code / security) ran; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). (#130 hub dispatcher) `download_repo` / `upload_repo` reject `local_dir` / `folder_path` outside cwd via `utils.paths.is_under_cwd` (security-review HIGH — closes a gap where a public helper could write to arbitrary absolute paths); `_validate_repo_id_shape` rejects bool / null-byte / leading-slash / `..` segments / control characters (CR/LF, defence-in-depth against header injection) / oversize (>200 chars). `modelscope.push_model` does NOT accept `commit_message` — the sanitized message is omitted on that backend to avoid a silent TypeError at runtime (code-review HIGH). (#130 `commands/train.py` pre-fetch) cache subdir name sanitised via `re.sub(r"[^A-Za-z0-9._-]+", "__", cfg.base)` then strip-leading/trailing-dots to defeat Windows `..\\` traversal that would have escaped `.soup_hub_cache/` (security-review HIGH); resolved cache dir re-checked via `is_under_cwd` as defence-in-depth. `cfg.base = local_path` mutation replaced with `cfg = cfg.model_copy(update={"base": local_path})` so Pydantic field validators rerun (matches v0.33.0 #47 immutability policy; python-review + code-review HIGH). Pre-fetch idempotent — probes `cache_dir / config.json` and reuses on resume runs. `ImportError` print uses `rich.markup.escape` to defend against markup injection from a crafted dep name (security-review MEDIUM). (#90 telemetry) `_telemetry_endpoint_is_safe` re-runs the v0.51.0 hub-endpoint SSRF validator on every caller-supplied endpoint (HTTPS + private-IP / link-local / RFC1918 rejected) — defence-in-depth so a misconfigured `endpoint='https://10.0.0.1/'` cannot reach internal networks even when telemetry is opt-IN (security-review MEDIUM). Lazy `httpx` import + 1s hard timeout + silent-fail on every exception (no `traceback`). (#85 fsspec loader) `validate_remote_uri` runs BEFORE any `fsspec.open` connection (bucket regex, no userinfo / query / fragment); row-count capped at 1,000,000 to defend against pathological remote objects (security-review MEDIUM). (#69 `detect_space_sdk`) input length capped at 256 KB to match v0.39.0 Part E template policy — refuses to scan oversized requirements.txt that would degrade response time (security-review LOW). (#93 package-data) fixtures moved to `soup_cli/data/_fixtures/`; `_bundle_source_path` falls back to legacy `examples/data/` for editable installs. (#89 `tracker_missing_dep_message`) switched from `__import__(pkg)` to `importlib.util.find_spec` (non-executing probe) so swanlab can't initialise network threads during the absence check (code-review MEDIUM; `sys.modules[pkg] = None` sentinel still honoured for tests). Test surface: `tests/test_v0538.py` carries 66 new tests including 4 new cwd-containment + SSRF rejection regression guards. (v0.53.8) - **v0.53.7 — Data Forge + Pipeline live (wave 1)**: 11 features (6 new + 5 v0.53.6 stub-to-live), 1 CRITICAL + 11 HIGH + 17 MEDIUM + 10 LOW review findings fixed. (#103 `/v1/tools/bash` REVERTED to HTTP 501) security review caught that `/bin/sh -c` spawns a child process outside the RLVR sandbox's OS-level isolation (`unshare(CLONE_NEWNET)` on Linux, `sandbox-exec` on macOS, socket monkey-patch); a caller could reach the cloud-metadata service from the child shell. Endpoint returns 501 with v0.53.8 marker until container/namespace work lands. Python + web_search remain LIVE. (#103 `/v1/tools/{python,web_search}` Bearer auth gate) optional `auth_token` kwarg on `_create_app` + `Bearer` gate on both endpoints; default `None` preserves loopback-only back-compat per v0.30.0 Part C CORS policy. (#103 `/v1/tools/web_search` deny-by-default) `WebSearchConfig.domain_allowlist` defaults to empty = deny-all (returns 403 `"web_search disabled"`); only operators who supply a domain list via constructor enable the endpoint. 5s `httpx` timeout, 5-result cap, snippet null-byte stripped. (#102 `/v1/messages` redaction) malformed Anthropic payloads (incl. validator-shape errors) mapped to generic `"Invalid request"` 400 body; detailed exception text logged server-side at DEBUG (matches v0.40.3 #33 trace-log policy). (#102 vLLM `/v1/messages` CORS) restricted to loopback origins on both transformers AND vLLM backends since it's a mutation route (matches v0.30.0 Part C transformers-backend policy that was previously vLLM-wildcard). (#102 SSE header injection) `model` + `msg_id` strings stripped of CRLF / NUL / oversize before SSE `data:` embedding — defends against `model="foo\ndata:{injected}\n\n"` injecting spurious SSE frames; `Cache-Control: no-store` + `X-Accel-Buffering: no` headers on `StreamingResponse` prevent intermediary caching of model outputs. (#106 `_node_seed` symlink TOCTOU) `os.lstat` runs on the RAW path BEFORE `realpath` resolves the symlink (matches v0.33.0 #22 `prune_checkpoints` policy); `realpath`-first ordering would have been a no-op since `S_ISLNK` on a resolved path is always False. (#106 `_save_checkpoint` atomic) switched from `path + ".tmp"` + plain `open()` to `tempfile.mkstemp(dir=path.parent, suffix=".tmp")` + `os.fdopen` + `os.replace` (matches v0.43.0 Part D `copy_bundle_to` / v0.48.0 `write_mix_recipe` policy); eliminates predictable `.tmp` symlink hazard. (#106 `failed_reason` redaction) raw exception messages run through `_redact_exc_message` which strips POSIX `/abs/path` and Windows `C:\abs\path` runs to `os.path.basename` and caps at 256 chars (matches v0.34.0 `crash.py` redaction policy); a `FileNotFoundError` no longer leaks the operator's `$HOME` into the world-readable `.checkpoint.json`. (#106 `_node_code` row JSON double-encode) row payloads serialised via `json.dumps(json.dumps(row))` before f-string embedding into the RLVR sandbox wrapper — defends against Python `repr()` escape edge cases on Unicode that could break sandbox invocation. (#106 resume rehydration) completed-node outputs rehydrated from per-node `.node-.jsonl` sidecar files (NOT empty list — a v0.53.7 in-progress draft had a correctness defect that silently fed downstream consumers an empty list, causing zero-row output on resume). (#86 atomic Arrow write) `Dataset.save_to_disk` writes to a `.tmp_` sibling directory then `os.replace`-ed (mirrors v0.48.0 `write_mix_recipe` pattern); mid-write Ctrl-C / OOM no longer leaves a partial Arrow directory that looks complete to subsequent runs. (#86 cache-hash gate) SFT + Pretrain wrappers verify `metadata.json::cache_key` matches the freshly-computed `make_preprocess_cache_key(base, max_length, format, train)` BEFORE consuming the Arrow shards; mismatch raises `ValueError("cache hash mismatch: re-run `soup data preprocess`")` so a user who pre-tokenized with one config and changed `max_length` cannot silently train on stale shards. Missing metadata.json falls back to yellow advisory + trusted mode. (#87 `resolve_prompt_strategy`) operator-supplied module imported via `importlib`; signature-validated `Mapping → Mapping`; `except` narrowed to `TypeError` only so the explicit ValueError from the signature-shape check propagates (a v0.53.7 in-progress draft had `except (TypeError, ValueError): pass` which swallowed its own validator); `assert isinstance(spec, str)` replaced with `raise TypeError` so the validator survives `python -O`. Per-row invocations swallow exceptions at DEBUG and fall through to the raw row (matches v0.33.0 #47 `CrossDocCollator` immutability policy). LIMITATION: this is a trusted-input surface — do NOT use `prompt_strategy` with operator-controlled YAML on shared CI hosts. (#88 markdown ingest) heading-section text capped per regex `^#{1,6}\s` so a crafted MD with a 1-byte heading followed by 1 GB body still respects the existing `data/ingest` MD branch size cap from v0.42.0 Part F. (#111 forge `--judge-provider`) lazy v0.20.0 provider instantiation reuses existing SSRF hardening per backend: Ollama localhost-only (rejects remote IPs), Anthropic env-only API key (`ANTHROPIC_API_KEY` — never CLI arg, matches v0.20.0 policy), vLLM scheme-allowlist + loopback HTTP. Per-call exceptions counted + logged at DEBUG (no silent silent-degrade). (#112 `--benchmark-file`) operator JSONL path runs through `is_under_cwd` + `os.lstat + S_ISLNK` rejection BEFORE the corpus is loaded (matches v0.46.0 `agent_forge.load_spec_file` policy). (#105 `instantiate_trainer_plugins`) `_instantiate_simple_plugin` returns None + WARNING when no `attr_candidates` match the imported module (matches v0.45.0 `register_plugin` policy — surface bad shapes loudly, not silently); `except TypeError` narrowed (a v0.53.7 in-progress draft swallowed all `TypeError` from the wrong layer). Cross-cutting: SSE generator return types annotated (`Generator[str, None, None]`); `JUDGE_PROVIDERS: frozenset[str]` PEP 585 element type; `extract_row_text` exposed publicly in `utils/data_score.__all__` (cross-module import of private helper eliminated). Test surface: `tests/test_v0537.py` carries 111 new tests; CI green across Ubuntu/macOS/Windows × Python 3.9/3.11/3.12. (v0.53.7) - **v0.53.6 — Plugin + Agent + Anthropic API**: 6 features — 3 live, 3 deferred-live stubs. (#101 SoupPluginCallback live) New `soup_cli/monitoring/plugin_callback.py` ships `SoupPluginCallback(transformers.TrainerCallback)` that dispatches the four canonical hook events to every enabled plugin in the v0.45.0 registry. Per-hook invocation runs through `_safe_invoke` which swallows ALL exceptions at WARNING with `exc_info=True` — defends against one misbehaving plugin crashing a multi-hour training run (mirrors v0.44.0 / v0.45.0 plugin-loader policy). Hook snapshot is collected ONCE in `build_plugin_callback` and passed to the constructor so a plugin registered between the "is empty?" check and ctor cannot silently slip into the active hook list (code-review HIGH fix — race-window closed). `attach_plugin_callback(trainer, console=None) -> bool` short-circuits to False when no plugins are enabled (zero overhead on the hot path); inner `add_callback` failure is swallowed at DEBUG so a plugin-infrastructure bug never crashes training. Wired into all 13 transformer-backend trainers; source-level grep regression test parametrized over every trainer file. (#102 Anthropic `/v1/messages` live) New POST `/v1/messages` route reuses the v0.45.0 SSRF-hardened `from_anthropic` converter + the existing chat handler. Streaming requests return 501 BEFORE schema validation (defence-in-depth — defends against stream-only attackers leaking validator-error detail). All validation paths (`validate_anthropic_payload` / `from_anthropic` / `ChatCompletionRequest(**openai_payload)`) are wrapped to map to a generic `"Invalid request"` 400 body; detailed exception text is logged server-side at DEBUG (security-review MEDIUM fix — matches `utils/errors.py` policy "HTTP error responses return generic messages, details logged server-side"). The existing loopback-CORS restriction from v0.30.0 Part C and the trace-log-redaction from v0.40.3 #33 both apply to the new route. (#104 n-gram speculative decoding live) `_generate_response` gains `ngram_config: Any = None` kwarg; when set, emits `prompt_lookup_num_tokens=int(ngram_config.num_draft_tokens)` into `model.generate(...)`. The int cast is wrapped in `try/except (TypeError, AttributeError)` as defence-in-depth; primary validation lives upstream in v0.45.0 `validate_ngram_config` (bool-rejected, bounded `[1, 32]`). Mutually exclusive with `assistant_model` — real draft model wins. (#103 server-side tool endpoints STUB) Three new POST routes `/v1/tools/python` / `/v1/tools/bash` / `/v1/tools/web_search` return 501 with v0.53.7 marker. Tool names are hardcoded string literals (not user-controlled); no payload inspection. Live RLVR `code_exec` sandbox HTTP wrapper (python / bash) + `WebSearchConfig.domain_allowlist` enforcement (web_search) land in v0.53.7. (#106 `run_recipe` STUB + `soup data recipe --execute`) New `utils/recipe_run.run_recipe(dag, *, output_dir, ...)` type-checks every parameter (TypeError on non-`RecipeDAG` / non-string / empty `output_dir` / non-bool `resume` / non-string `judge_*`) BEFORE the `NotImplementedError("v0.53.7")`. CLI `--execute --output ` enforces `is None` guard on `--output` (project policy since v0.40.6 — empty-string is a distinct operator error, NOT silent missing) + `is_under_cwd` containment at the CLI boundary BEFORE the `run_recipe` call (security-review MEDIUM fix — defends against future v0.53.7 live runner bypass; never want the live runner to be the first/only enforcement point per project policy). Regression test asserts the v0.53.7 marker does NOT surface on outside-cwd paths — guards against future live runner accidentally bypassing CLI containment. (#105 `instantiate_trainer_plugins` STUB) Validates name list through existing `validate_trainer_plugin_list` (closed allowlist, dedup, ≤8/run, canonicalisation — all v0.45.0 hardening) BEFORE raising `NotImplementedError`. Canonical names echoed in the error message are allowlist-validated lowercase strings (no injection surface). Test surface: 1 new test file (`tests/test_v0536.py`) carrying 53 new tests covering: per-hook exception swallow with caplog isolation, source-level grep parametrized over 13 trainer files (canonical import + invocation), generic `"Invalid request"` body redaction with detail logged at DEBUG, oversize `max_tokens` cap, n-gram kwarg omitted when config is None (regression guard against future "always emit" refactor), `run_recipe` type-rejection matrix (non-DAG / empty output_dir / non-str checkpoint_dir / non-str judge_provider / non-str judge_model / bool resume), `--execute --output ""` rejected with `"must be a non-empty path"`, outside-cwd `--output` rejected with `v0.53.7` marker NEVER surfacing on rejected paths. Known limitations: (1) vLLM `/v1/messages` parity deferred to v0.53.7 — Anthropic route is transformers-only. (2) Streaming `/v1/messages` returns 501 — true SSE event-shape ships in v0.53.7. (3) Plugin hook snapshot is at callback-construction time — a plugin registered mid-run does not retroactively receive hooks (by design — prevents partial-run inconsistencies). (4) n-gram is transformers-only — vLLM ships its own prompt-lookup path that needs a different kwarg. (5) Tool endpoints + recipe runner + trainer-plugin instantiation are stubs — live wiring in v0.53.7. (v0.53.6) diff --git a/examples/README.md b/examples/README.md index cfd2bd2..5876ac3 100644 --- a/examples/README.md +++ b/examples/README.md @@ -177,6 +177,22 @@ Run inference on a batch of prompts: soup infer --model ./output_sft_basic/ --input prompts.jsonl --output results.jsonl ``` +### 13. Synthetic-data workflow + +End-to-end recipe that generates training data from a local LLM, filters ++ scores + decontaminates it, then trains on the cleaned set. See +[synthetic_workflow.md](synthetic_workflow.md) for the walkthrough and +[synthetic_workflow.yaml](synthetic_workflow.yaml) for the bundled config. + +```bash +soup data generate --provider ollama --output ./synth_raw.jsonl +soup data filter --input ./synth_raw.jsonl --output ./synth_filtered.jsonl +soup data score --input ./synth_filtered.jsonl --output ./synth_scored.jsonl +soup data decontaminate --input ./synth_scored.jsonl \ + --output ./synth_clean.jsonl --benchmarks mmlu,gsm8k +soup train --config examples/synthetic_workflow.yaml --yes +``` + ### 12. Full RLHF Pipeline Complete reinforcement learning from human feedback: diff --git a/examples/synthetic_workflow.md b/examples/synthetic_workflow.md new file mode 100644 index 0000000..4a208f5 --- /dev/null +++ b/examples/synthetic_workflow.md @@ -0,0 +1,72 @@ +# Synthetic data workflow (end-to-end) + +Generate, filter, score, and train on synthetic data — all with `soup`. Pairs +with the bundled `synthetic_workflow.yaml` recipe. + +## 1. Generate + +Spin up a local Ollama model and ask it for 200 instruction/response pairs +around a topic: + +```bash +soup data generate \ + --provider ollama \ + --model llama3.2:3b \ + --topic "Python error handling" \ + --count 200 \ + --output ./synth_raw.jsonl +``` + +For Anthropic / vLLM / server providers, swap `--provider` and follow the +`soup data generate --help` matrix. + +## 2. Filter for quality + +Drop low-perplexity / low-coherence rows: + +```bash +soup data filter \ + --input ./synth_raw.jsonl \ + --output ./synth_filtered.jsonl \ + --min-coherence 0.5 +``` + +## 3. Score for safety + diversity + +Run the v0.47.0 quality moat to fingerprint PII / toxicity / language / +educational value, and decontaminate against your downstream evals: + +```bash +soup data score --input ./synth_filtered.jsonl --output ./synth_scored.jsonl +soup data decontaminate \ + --input ./synth_scored.jsonl \ + --output ./synth_clean.jsonl \ + --benchmarks mmlu,gsm8k +``` + +## 4. Train + +Point `soup train` at `synthetic_workflow.yaml`: + +```bash +soup train --config examples/synthetic_workflow.yaml --yes +``` + +That recipe references `./synth_clean.jsonl`, picks `TinyLlama-1.1B-Chat` +as the base, and runs an SFT job with LoRA r=8. + +## 5. Watch progress live (v0.53.9) + +In another terminal: + +```bash +soup ui --public --no-browser +``` + +Scan the printed QR code from your phone to monitor the loss curve and +live SSE training stream while the job runs. + +--- + +This is a thin walkthrough — for deeper coverage see +[examples/README.md](README.md) and [examples/configs/](configs/). diff --git a/examples/synthetic_workflow.yaml b/examples/synthetic_workflow.yaml new file mode 100644 index 0000000..2d5b496 --- /dev/null +++ b/examples/synthetic_workflow.yaml @@ -0,0 +1,40 @@ +# Synthetic-data workflow recipe (v0.53.9 #12) +# +# Pairs with `examples/synthetic_workflow.md`. Trains TinyLlama-1.1B on +# a JSONL produced by: +# soup data generate --provider ollama --output ./synth_raw.jsonl +# soup data filter --input ./synth_raw.jsonl --output ./synth_filtered.jsonl +# soup data score --input ./synth_filtered.jsonl --output ./synth_scored.jsonl +# soup data decontaminate --input ./synth_scored.jsonl \ +# --output ./synth_clean.jsonl --benchmarks mmlu,gsm8k + +base: TinyLlama/TinyLlama-1.1B-Chat-v1.0 +task: sft +modality: text +backend: transformers + +data: + train: ./synth_clean.jsonl + format: auto + val_split: 0.05 + max_length: 1024 + +training: + epochs: 1 + lr: 2.0e-4 + batch_size: auto + quantization: 4bit + optimizer: adamw_torch + scheduler: cosine + gradient_checkpointing: true + warmup_auto: true + auto_mixed_precision: true + lora: + r: 8 + alpha: 16 + dropout: 0.05 + +output: ./synth_out + +eval: + auto_eval: false diff --git a/pyproject.toml b/pyproject.toml index 9736c31..66b3f75 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.53.8.1" +version = "0.53.9" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 776bc5b..7cd1f89 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.53.8.1" +__version__ = "0.53.9" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index 7fd550a..39476a6 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -189,6 +189,15 @@ from soup_cli.commands import data_mix as _data_mix_cmd # noqa: E402 data.app.command(name="mix")(_data_mix_cmd.mix) +# v0.53.9 #15 — BPE tokenizer training. +from soup_cli.commands import tokenizer as _tokenizer_cmd # noqa: E402 + +app.add_typer( + _tokenizer_cmd.app, + name="tokenizer", + help="Tokenizer tools: train a BPE tokenizer from JSONL (v0.53.9).", +) + @app.command() def version( diff --git a/soup_cli/commands/bench.py b/soup_cli/commands/bench.py index 3cffc3e..7e1ab4b 100644 --- a/soup_cli/commands/bench.py +++ b/soup_cli/commands/bench.py @@ -39,6 +39,24 @@ def bench( "--prompts-file", help="Path to custom prompts file (.txt or .jsonl)", ), + p50: bool = typer.Option( + False, + "--p50", + help="Print p50 (median) per-prompt latency. v0.53.9 #26.", + ), + p95: bool = typer.Option( + False, + "--p95", + help="Print p95 per-prompt latency. v0.53.9 #26.", + ), + backend: str = typer.Option( + "auto", + "--backend", + help=( + "Inference backend hint: auto (default) | transformers | mlx. " + "v0.53.9 #28." + ), + ), ) -> None: """Run an inference benchmark (speed and memory) on a loaded model.""" import torch @@ -64,6 +82,24 @@ def bench( device, _ = detect_device() + # v0.53.9 #28 — backend auto-detect. + from soup_cli.utils.backend_detect import SUPPORTED_BACKENDS, detect_backend + + backend_lower = (backend or "auto").strip().lower() + if backend_lower == "auto": + backend_resolved = detect_backend(str(model_path)) + console.print( + f"[dim]Backend auto-detected:[/] [bold]{backend_resolved}[/]" + ) + else: + if backend_lower not in SUPPORTED_BACKENDS: + console.print( + f"[red]Unknown --backend:[/] {backend} " + f"(expected: auto | {' | '.join(sorted(SUPPORTED_BACKENDS))})" + ) + raise typer.Exit(2) + backend_resolved = backend_lower + if device == "cpu": console.print( "[yellow]Warning:[/] Running on CPU. Inference speed is typically " @@ -82,6 +118,20 @@ def bench( "current working directory." ) raise typer.Exit(1) + # v0.53.9 review fix M2 — reject symlinked prompts file on the + # RAW path BEFORE realpath resolution (mirrors v0.53.7 #106 policy). + import stat as _stat + + try: + _st = _os.lstat(prompts_file) + except OSError: + console.print(f"[red]Prompts file not found:[/] {prompts_file}") + raise typer.Exit(1) + if _stat.S_ISLNK(_st.st_mode): + console.print( + "[red]Prompts file must not be a symlink.[/]" + ) + raise typer.Exit(1) p_path = Path(_os.path.realpath(prompts_file)) if not p_path.is_file(): @@ -164,6 +214,7 @@ def bench( total_tokens = 0 total_latency = 0.0 + per_prompt_latencies: list[float] = [] console.print(f"[bold]Running {len(test_prompts)} test inferences...[/]") @@ -179,6 +230,7 @@ def bench( latency = time.time() - start_time total_tokens += token_count total_latency += latency + per_prompt_latencies.append(latency) console.print(f" [dim]Prompt {i + 1}: {token_count} tokens in {latency:.2f}s[/]") avg_tps = total_tokens / total_latency if total_latency > 0 else 0 @@ -195,7 +247,7 @@ def bench( vram_str = f"{peak_vram_gb:.2f} GB" if torch.cuda.is_available() else "N/A" table.add_row( - "Transformers", + backend_resolved.capitalize(), f"{avg_tps:.2f}", f"{total_latency:.2f}s", vram_str, @@ -203,3 +255,24 @@ def bench( console.print() console.print(table) + + # v0.53.9 #26 — tail-latency percentiles. + if (p50 or p95) and per_prompt_latencies: + from soup_cli.utils.tail_latency import summarise_latency + + summary = summarise_latency(per_prompt_latencies) + pct_table = Table(title="Per-prompt latency") + pct_table.add_column("Statistic", style="cyan") + pct_table.add_column("Latency (s)", style="green", justify="right") + pct_table.add_row("count", str(summary.count)) + if summary.mean is not None: + pct_table.add_row("mean", f"{summary.mean:.3f}") + if p50 and summary.p50 is not None: + pct_table.add_row("p50", f"{summary.p50:.3f}") + if p95 and summary.p95 is not None: + pct_table.add_row("p95", f"{summary.p95:.3f}") + # p99 is a natural superset of p95 — only print when p95 was requested. + if p95 and summary.p99 is not None: + pct_table.add_row("p99", f"{summary.p99:.3f}") + console.print() + console.print(pct_table) diff --git a/soup_cli/commands/serve.py b/soup_cli/commands/serve.py index 9ade86d..a934199 100644 --- a/soup_cli/commands/serve.py +++ b/soup_cli/commands/serve.py @@ -191,6 +191,14 @@ def serve( "--trace-log-cap-mb", help="Rotation cap in MB for --trace-log (1 - 10000). Default 100.", ), + reasoning_parser: Optional[str] = typer.Option( + None, + "--reasoning-parser", + help=( + "Strip reasoning-trace blocks from responses. One of: " + "deepseek-r1 | qwen3 | phi4 | openthinker. v0.53.9 #98." + ), + ), ): """Start a local inference server with OpenAI-compatible API.""" # Lazy imports for fast CLI startup @@ -603,6 +611,17 @@ def serve( f"(cap {trace_log_cap_mb} MB)" ) + # v0.53.9 #98 — validate reasoning parser name once at startup. + resolved_reasoning_parser: Optional[str] = None + if reasoning_parser: + from soup_cli.utils.reasoning_parser import validate_parser_name + + try: + resolved_reasoning_parser = validate_parser_name(reasoning_parser) + except (TypeError, ValueError) as exc: + console.print(f"[red]--reasoning-parser:[/] {exc}") + raise typer.Exit(1) from exc + app = _create_app( model_obj=model_obj, tokenizer=tokenizer, @@ -616,6 +635,7 @@ def serve( enable_dashboard=dashboard, tracer=tracer, trace_log_writer=trace_log_writer, + reasoning_parser=resolved_reasoning_parser, ) console.print( @@ -902,6 +922,7 @@ def _create_app( web_search_config: Any = None, web_search_backend: Any = None, auth_token: Optional[str] = None, + reasoning_parser: Optional[str] = None, ): """Create the FastAPI application with OpenAI-compatible endpoints. @@ -1116,6 +1137,14 @@ def _create_app( metrics.record_tokens(completion_tokens) + # v0.53.9 #98 — strip reasoning-trace blocks if configured. + if reasoning_parser is not None: + from soup_cli.utils.reasoning_parser import strip_reasoning + + response_text = strip_reasoning( + response_text, reasoning_parser, + ) + # output_constraint is validated upstream; v0.33.0 #53 wires # it through outlines / lm-format-enforcer into the generate # loop. If neither library is installed, build_logits_processors diff --git a/soup_cli/commands/tokenizer.py b/soup_cli/commands/tokenizer.py new file mode 100644 index 0000000..6806930 --- /dev/null +++ b/soup_cli/commands/tokenizer.py @@ -0,0 +1,247 @@ +"""soup tokenizer — BPE tokenizer training (v0.53.9 #15).""" + +from __future__ import annotations + +import json +import os +import stat +from pathlib import Path +from typing import Optional + +import typer +from rich.console import Console +from rich.panel import Panel + +from soup_cli.utils.paths import is_under_cwd + +console = Console() + +app = typer.Typer( + help="Tokenizer tools: train a BPE tokenizer from a JSONL corpus.", + no_args_is_help=True, +) + +_MAX_VOCAB_SIZE = 200_000 +_MIN_VOCAB_SIZE = 256 +_MAX_CORPUS_BYTES = 50 * 1024 * 1024 # 50 MB total cap +_MAX_LINE_BYTES = 8 * 1024 # 8 KB per-line cap + + +def _validate_under_cwd(path: str, *, label: str) -> str: + if not isinstance(path, str) or not path: + raise typer.BadParameter(f"{label} must be a non-empty path") + if "\x00" in path: + raise typer.BadParameter(f"{label} contains NUL byte") + if not is_under_cwd(path): + raise typer.BadParameter( + f"{label} must stay under the current working directory: {path}" + ) + # TOCTOU defence: lstat the RAW user-supplied path BEFORE realpath + # resolution (matches v0.53.7 #106 policy). `os.path.realpath` would + # follow the link and `lstat` on the resolved target would see a + # regular file even when the user-supplied entry is a symlink. + try: + st = os.lstat(path) + except OSError: + # Missing-file path is handled by the caller's `is_file` check. + return os.path.realpath(path) + if stat.S_ISLNK(st.st_mode): + raise typer.BadParameter( + f"{label} target must not be a symlink" + ) + return os.path.realpath(path) + + +def _extract_texts(input_path: Path) -> list[str]: + """Read JSONL or plaintext lines into a list of training strings. + + Capped at 50 MB total + 8 KB per line to bound RAM on adversarial input. + """ + texts: list[str] = [] + total_bytes = 0 + if input_path.suffix.lower() == ".jsonl": + with open(input_path, encoding="utf-8") as fh: + for raw in fh: + if len(raw.encode("utf-8", errors="replace")) > _MAX_LINE_BYTES: + continue # silently skip oversized line + total_bytes += len(raw) + if total_bytes > _MAX_CORPUS_BYTES: + break + text = raw.strip() + if not text: + continue + try: + row = json.loads(text) + except json.JSONDecodeError: + continue + if isinstance(row, dict): + for key in ("text", "content", "prompt"): + value = row.get(key) + if isinstance(value, str) and value: + texts.append(value) + break + else: + # ShareGPT-style messages list + msgs = row.get("messages") + if isinstance(msgs, list): + for msg in msgs: + if isinstance(msg, dict): + content = msg.get("content") + if isinstance(content, str) and content: + texts.append(content) + elif isinstance(row, str): + texts.append(row) + else: + with open(input_path, encoding="utf-8") as fh: + for raw in fh: + if len(raw.encode("utf-8", errors="replace")) > _MAX_LINE_BYTES: + continue + total_bytes += len(raw) + if total_bytes > _MAX_CORPUS_BYTES: + break + text = raw.rstrip("\n\r") + if text: + texts.append(text) + return texts + + +@app.command(name="train") +def train( + input_path: str = typer.Option( + ..., + "--input", + "-i", + help="Path to JSONL or .txt corpus (must stay under cwd).", + ), + vocab_size: int = typer.Option( + 32_000, + "--vocab-size", + "-v", + help=f"Target vocabulary size ({_MIN_VOCAB_SIZE}-{_MAX_VOCAB_SIZE}).", + ), + output: str = typer.Option( + "tokenizer_out", + "--output", + "-o", + help="Output directory for tokenizer.json + vocab.json (under cwd).", + ), + min_frequency: int = typer.Option( + 2, + "--min-frequency", + help="Minimum pair frequency (>=1).", + ), + special_tokens: Optional[list[str]] = typer.Option( + None, + "--special-token", + help="Add special token (repeatable). Defaults: , , , .", + ), +) -> None: + """Train a BPE tokenizer from a local corpus. + + Outputs `tokenizer.json` (Hugging Face format) and `vocab.json` to + `--output`. Requires the `tokenizers` library (bundled with + `transformers`). + """ + if isinstance(vocab_size, bool) or not isinstance(vocab_size, int): + raise typer.BadParameter("--vocab-size must be int") + if not (_MIN_VOCAB_SIZE <= vocab_size <= _MAX_VOCAB_SIZE): + raise typer.BadParameter( + f"--vocab-size must be in [{_MIN_VOCAB_SIZE}, {_MAX_VOCAB_SIZE}]" + ) + if isinstance(min_frequency, bool) or not isinstance(min_frequency, int): + raise typer.BadParameter("--min-frequency must be int") + if min_frequency < 1: + raise typer.BadParameter("--min-frequency must be >= 1") + + input_real = _validate_under_cwd(input_path, label="--input") + output_real = _validate_under_cwd(output, label="--output") + + resolved_input = Path(input_real) + if not resolved_input.is_file(): + console.print(f"[red]Input file not found:[/] {input_path}") + raise typer.Exit(1) + output_path = Path(output_real) + + try: + from tokenizers import Tokenizer, models, pre_tokenizers, trainers + except ImportError as exc: + console.print( + "[red]The 'tokenizers' package is required.[/]\n" + "Install with: [bold]pip install tokenizers[/]" + ) + raise typer.Exit(1) from exc + + texts = _extract_texts(resolved_input) + if not texts: + console.print( + f"[red]No usable training text found in[/] {resolved_input.name}" + ) + raise typer.Exit(1) + + tokens = list(special_tokens or []) + if not tokens: + tokens = ["", "", "", ""] + # Sanitise: reject NUL / oversize entries; dedup preserves first-seen order. + cleaned: list[str] = [] + seen: set[str] = set() + for tok in tokens: + if not isinstance(tok, str) or not tok or "\x00" in tok or len(tok) > 128: + raise typer.BadParameter( + f"--special-token entry must be a non-empty NUL-free str <=128 chars: {tok!r}" + ) + if tok in seen: + continue + seen.add(tok) + cleaned.append(tok) + + console.print( + Panel( + f"Input: [bold]{resolved_input.name}[/]\n" + f"Texts: [bold]{len(texts)}[/]\n" + f"Vocab size: [bold]{vocab_size}[/]\n" + f"Output dir: [bold]{output_path.name}[/]", + title="BPE Tokenizer Training", + ) + ) + + tokenizer = Tokenizer(models.BPE(unk_token="")) + tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False) + trainer = trainers.BpeTrainer( + vocab_size=vocab_size, + min_frequency=min_frequency, + special_tokens=cleaned, + show_progress=False, + ) + tokenizer.train_from_iterator(texts, trainer=trainer) + + output_path.mkdir(parents=True, exist_ok=True) + # Post-mkdir re-check: defends against a symlink planted between the + # containment check and the write. Matches v0.43.0 Part D `copy_bundle_to` + # TOCTOU policy. + try: + post_st = os.lstat(output_path) + except OSError as exc: + console.print(f"[red]Output directory not accessible:[/] {exc}") + raise typer.Exit(1) from exc + if stat.S_ISLNK(post_st.st_mode): + console.print( + "[red]Output directory is a symlink — refusing to write.[/]" + ) + raise typer.Exit(1) + if not is_under_cwd(str(output_path)): + console.print("[red]Output directory escaped cwd after mkdir.[/]") + raise typer.Exit(1) + + tokenizer_file = output_path / "tokenizer.json" + tokenizer.save(str(tokenizer_file)) + vocab_file = output_path / "vocab.json" + vocab = tokenizer.get_vocab() + vocab_file.write_text( + json.dumps(vocab, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + + console.print( + f"[green]Trained tokenizer with[/] [bold]{len(vocab)}[/] tokens " + f"-> [bold]{output_path.name}/{tokenizer_file.name}[/]" + ) diff --git a/soup_cli/commands/ui.py b/soup_cli/commands/ui.py index c8c3e79..1af3467 100644 --- a/soup_cli/commands/ui.py +++ b/soup_cli/commands/ui.py @@ -1,5 +1,9 @@ """soup ui — local web interface for managing experiments and training.""" +from __future__ import annotations + +from typing import Optional + import typer from rich.console import Console from rich.panel import Panel @@ -29,11 +33,34 @@ def ui( "--show-token", help="Print the auth token and exit (for scripting)", ), + public: bool = typer.Option( + False, + "--public", + help=( + "Bind to 0.0.0.0 so the UI is reachable from a phone on the " + "same LAN. Prints a phone-scannable URL + ASCII QR at startup. " + "Loopback HTTP is auto-upgraded to require a Bearer token in " + "the query string. v0.53.9 #95." + ), + ), + auth_token: Optional[str] = typer.Option( + None, + "--auth-token", + help=( + "Override the auto-generated Bearer token (16-128 urlsafe " + "base64 chars). Useful for stable phone bookmarks. " + "When omitted, a fresh token is generated each startup." + ), + ), ): """Launch the Soup Web UI. A Bearer auth token is auto-generated at startup and printed to the console. Mutating API endpoints (POST/DELETE) require 'Authorization: Bearer '. + + `--public` exposes the server on 0.0.0.0 for phone-on-LAN access. The + auth token is embedded in a phone-scannable URL + ASCII QR code so a + phone scan lands authenticated. """ try: import uvicorn # noqa: F401 @@ -45,7 +72,21 @@ def ui( ) raise typer.Exit(1) - from soup_cli.ui.app import create_app, get_auth_token + from soup_cli.ui.app import create_app, get_auth_token, set_auth_token + + # v0.53.9 #95 — operator-supplied auth token override. + # `set_auth_token` validates via `qr_url.validate_token`, so we don't + # double-validate here. + if auth_token is not None: + try: + set_auth_token(auth_token) + except (TypeError, ValueError) as exc: + console.print(f"[red]--auth-token:[/] {exc}") + raise typer.Exit(1) from exc + + # --public binds to 0.0.0.0 unless the operator already set a custom host. + if public and host == "127.0.0.1": + host = "0.0.0.0" token = get_auth_token() @@ -57,22 +98,71 @@ def ui( url = f"http://{host}:{port}" + panel_body = ( + f"URL: [bold]{url}[/]\n" + f"Token: [bold]{token}[/]\n\n" + f"Mutating API endpoints require:\n" + f" [dim]Authorization: Bearer {token}[/]\n\n" + f"Pages:\n" + f" [bold]Dashboard[/] - View experiments, loss charts, system info\n" + f" [bold]New Training[/] - Create config from templates, start training\n" + f" [bold]Data Explorer[/] - Browse and inspect datasets\n" + f" [bold]Model Chat[/] - Chat with a running inference server\n\n" + f"Press [bold]Ctrl+C[/] to stop." + ) + console.print( Panel( - f"URL: [bold]{url}[/]\n" - f"Token: [bold]{token}[/]\n\n" - f"Mutating API endpoints require:\n" - f" [dim]Authorization: Bearer {token}[/]\n\n" - f"Pages:\n" - f" [bold]Dashboard[/] - View experiments, loss charts, system info\n" - f" [bold]New Training[/] - Create config from templates, start training\n" - f" [bold]Data Explorer[/] - Browse and inspect datasets\n" - f" [bold]Model Chat[/] - Chat with a running inference server\n\n" - f"Press [bold]Ctrl+C[/] to stop.", + panel_body, title="[bold green]Soup Web UI[/]", ) ) + # v0.53.9 #95 — phone-visible URL + QR for --public. + if public: + import socket + + from soup_cli.utils.qr_url import render_qr_ascii + + # Derive the LAN IP a phone on the same Wi-Fi can reach. Falls + # back to 127.0.0.1 if DNS resolution fails — operator gets a + # warning explaining the QR is loopback-only in that case. + lan_ip: str = "127.0.0.1" + try: + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: + # Connect to a non-routed address; OS picks the egress + # interface and we read its bound IP. No packets sent. + sock.settimeout(0.1) + sock.connect(("10.255.255.255", 1)) + lan_ip = sock.getsockname()[0] + except (OSError, socket.error): + lan_ip = "127.0.0.1" + + # Plain HTTP over LAN is intentional: this is local-network only + # and the Bearer token gates mutating endpoints. We assemble the + # URL directly because `build_phone_url` rejects LAN HTTP by + # design (its threat model is stricter than ours here). + phone_url = f"http://{lan_ip}:{port}/?token={token}" + if lan_ip == "127.0.0.1": + console.print( + "[yellow]--public:[/] Could not detect LAN IP; " + f"falling back to loopback. Phone scan will only " + f"work from the same host: {phone_url}" + ) + else: + console.print( + f"\n[bold green]Phone URL:[/] {phone_url}\n" + "[dim]Scan the QR code below from your phone camera:[/]\n" + ) + qr_ascii = render_qr_ascii(phone_url) + if qr_ascii is None: + console.print( + "[dim]Install 'qrcode' to print an ASCII QR: " + "pip install qrcode[/]" + ) + else: + console.print(qr_ascii) + # Open browser if not no_browser: import threading @@ -80,6 +170,7 @@ def ui( def _open(): import time + time.sleep(1) webbrowser.open(url) diff --git a/soup_cli/monitoring/callback.py b/soup_cli/monitoring/callback.py index b4d1214..a56f520 100644 --- a/soup_cli/monitoring/callback.py +++ b/soup_cli/monitoring/callback.py @@ -128,6 +128,49 @@ class SoupTrainerCallback(TrainerCallback): gpu_mem=gpu_mem, ) + # v0.53.9 #94 — push to the global SSE buffer so the live Web UI + # dashboard sees per-step metrics in real time. Best-effort: any + # exception inside the push must NEVER take down training. + try: + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import push_train_event + + push_train_event( + TrainEvent( + type="metric", + step=int(step) if step is not None else None, + epoch=float(epoch) if epoch is not None else None, + loss=float(loss) if loss else None, + lr=float(lr) if lr else None, + grad_norm=float(grad_norm) if grad_norm else None, + ) + ) + except Exception: + pass + + # v0.53.9 #100 — tool-call observation for SFT runs whose batch logs + # surface a `tool_calls` count. Best-effort; the SFT trainer emits + # the field only when the data format is `tool-calling`. + if "tool_calls" in (logs or {}): + try: + import time + + from soup_cli.utils.tool_outputs import get_global_tool_buffer + + tool_count = logs.get("tool_calls") + if isinstance(tool_count, (int, float)) and not isinstance( + tool_count, bool + ) and tool_count > 0: + get_global_tool_buffer().record_call( + name="batch", + started_ts=time.time(), + duration_ms=0.0, + success=True, + output_preview=f"observed {int(tool_count)} tool_calls at step {step}", + ) + except Exception: + pass + # Loss watchdog — detect loss spikes and auto-stop if self._watchdog_enabled and not self._watchdog_fired and "loss" in logs: if loss > self._watchdog_threshold: diff --git a/soup_cli/ui/app.py b/soup_cli/ui/app.py index 636470b..f10f169 100644 --- a/soup_cli/ui/app.py +++ b/soup_cli/ui/app.py @@ -46,13 +46,31 @@ _train_process: Optional[subprocess.Popen] = None _train_config_path: Optional[str] = None _train_lock = threading.Lock() -# Auth token generated at startup — printed to console for the user +# Auth token generated at startup — printed to console for the user. +# Reads/writes go through `_auth_token_lock` so token rotation never +# leaves a window where some requests see the old value and some the new. _auth_token: str = secrets.token_urlsafe(32) +_auth_token_lock = threading.Lock() def get_auth_token() -> str: """Return the current auth token (for printing at startup).""" - return _auth_token + with _auth_token_lock: + return _auth_token + + +def set_auth_token(token: str) -> None: + """Replace the process-wide auth token (used by `soup ui --auth-token`). + + Validates via `utils.qr_url.validate_token` so a malformed override + can't bypass the urlsafe-base64 shape check. + """ + from soup_cli.utils.qr_url import validate_token + + validated = validate_token(token) + global _auth_token + with _auth_token_lock: + _auth_token = validated def create_app(host: str = "127.0.0.1", port: int = 7860): @@ -64,19 +82,39 @@ def create_app(host: str = "127.0.0.1", port: int = 7860): app = FastAPI(title="Soup Web UI", version="1.0.0") - # Restrict CORS to the origin we actually serve - allowed_origin = f"http://{host}:{port}" - app.add_middleware( - CORSMiddleware, - allow_origins=[allowed_origin], - allow_methods=["GET", "POST", "DELETE"], - allow_headers=["Authorization", "Content-Type"], - ) + # Restrict CORS to the origin we actually serve. When `host == "0.0.0.0"` + # the literal `http://0.0.0.0:` is never a browser origin, so we + # allow loopback origins AND the same-LAN regex shape. The Bearer + # token is the actual security gate on mutating endpoints. + if host == "0.0.0.0": + app.add_middleware( + CORSMiddleware, + allow_origin_regex=( + r"^https?://(" + r"localhost|127\.0\.0\.1|" + r"10\.\d+\.\d+\.\d+|" + r"192\.168\.\d+\.\d+|" + r"172\.(?:1[6-9]|2[0-9]|3[01])\.\d+\.\d+" + r")(:\d+)?$" + ), + allow_methods=["GET", "POST", "DELETE"], + allow_headers=["Authorization", "Content-Type"], + ) + else: + allowed_origin = f"http://{host}:{port}" + app.add_middleware( + CORSMiddleware, + allow_origins=[allowed_origin], + allow_methods=["GET", "POST", "DELETE"], + allow_headers=["Authorization", "Content-Type"], + ) def _verify_token(request: Request): """Verify Bearer token on mutating endpoints.""" auth = request.headers.get("Authorization", "") - if auth != f"Bearer {_auth_token}": + with _auth_token_lock: + expected = f"Bearer {_auth_token}" + if auth != expected: raise HTTPException(status_code=401, detail="Unauthorized") # --- Static files --- @@ -705,6 +743,91 @@ def create_app(host: str = "127.0.0.1", port: int = 7860): }, ) + # --- v0.53.9 #94: SSE training-event stream --- + + @app.get("/api/train/stream") + async def stream_train_events(): + """SSE endpoint streaming `TrainEvent` payloads as JSON frames. + + Per-subscriber cursor — multiple concurrent listeners each receive + every event (no destructive drain). Uses `asyncio.sleep` so the + uvicorn async loop is not blocked under default workers. + """ + import asyncio + + from fastapi.responses import StreamingResponse + + from soup_cli.utils.sse_train_stream import TrainEvent, format_sse_frame + from soup_cli.utils.train_event_buffer import get_global_buffer + + buffer = get_global_buffer() + + async def _gen(): + # Start from cursor 0 — new subscribers receive a bounded + # catch-up of retained events (deque maxlen=1000) before + # streaming fresh ones. Concurrent subscribers are independent. + cursor = 0 + max_ticks = 200 # cap to keep test runs bounded; ~20s at 100ms + empty_ticks = 0 + for _ in range(max_ticks): + events, cursor = buffer.snapshot_since(cursor) + if events: + empty_ticks = 0 + for event in events: + yield format_sse_frame(event) + else: + empty_ticks += 1 + yield ":heartbeat\n\n" + with _train_lock: + proc = _train_process + if proc is None or proc.poll() is not None: + if empty_ticks >= 1: + done = TrainEvent(type="status", message="done") + yield format_sse_frame(done) + return + await asyncio.sleep(0.1) + done = TrainEvent(type="status", message="timeout") + yield format_sse_frame(done) + + return StreamingResponse( + _gen(), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "X-Accel-Buffering": "no", + }, + ) + + # --- v0.53.9 #100: Tool-call observation panel --- + + @app.get("/api/tool-outputs") + def list_tool_outputs( + limit: int = Query(default=100, ge=1, le=1000), + ): + """Return the most recent tool-call records as JSON. + + Records are pushed by the SFT trainer's tool-calling callback + into the process-wide `ToolOutputsBuffer`. Read-only; safe for + cross-origin polling. + """ + from soup_cli.utils.tool_outputs import get_global_tool_buffer + + records = get_global_tool_buffer().snapshot(limit=limit) + return { + "count": len(records), + "records": [ + { + "name": r.name, + "started_ts": r.started_ts, + "duration_ms": r.duration_ms, + "success": r.success, + "output_preview": r.output_preview, + "error": r.error, + } + for r in records + ], + } + # --- Health --- @app.get("/api/health") diff --git a/soup_cli/ui/static/app.js b/soup_cli/ui/static/app.js index 9bb8d1c..cec743c 100644 --- a/soup_cli/ui/static/app.js +++ b/soup_cli/ui/static/app.js @@ -2,6 +2,78 @@ const API = ''; // same origin +// v0.53.9 #94 — Lightweight EventSource consumer for /api/train/stream. +// Opens on demand (call `startTrainEventStream()`) and dispatches parsed +// payloads to `onTrainEvent(payload)` which other modules can override. +// Auto-closes on `status=done` or `status=timeout`. +let _trainEventSource = null; +window.onTrainEvent = window.onTrainEvent || function (_payload) {}; +function startTrainEventStream() { + if (_trainEventSource) return _trainEventSource; + try { + const es = new EventSource('/api/train/stream'); + _trainEventSource = es; + es.onmessage = function (event) { + if (!event.data) return; + try { + const payload = JSON.parse(event.data); + if (payload && typeof window.onTrainEvent === 'function') { + window.onTrainEvent(payload); + } + if (payload && payload.type === 'status' && + (payload.message === 'done' || payload.message === 'timeout')) { + es.close(); + _trainEventSource = null; + } + } catch (e) { + // Ignore malformed frames. + } + }; + es.onerror = function () { + try { es.close(); } catch (e) {} + _trainEventSource = null; + }; + return es; + } catch (e) { + return null; + } +} +function stopTrainEventStream() { + if (_trainEventSource) { + try { _trainEventSource.close(); } catch (e) {} + _trainEventSource = null; + } +} +window.startTrainEventStream = startTrainEventStream; +window.stopTrainEventStream = stopTrainEventStream; + +// v0.53.9 #95 — Pick up Bearer token from `?token=…` (phone QR landing) +// or from sessionStorage on subsequent navigations. Stripped from the URL +// after read so the token doesn't sit in browser history. +(function _bootstrapAuthToken() { + try { + const params = new URLSearchParams(window.location.search); + const fromUrl = params.get('token'); + if (fromUrl) { + window._authToken = fromUrl; + try { sessionStorage.setItem('soup_auth_token', fromUrl); } catch (e) {} + // Drop ?token=… from the URL so refresh history doesn't leak it. + params.delete('token'); + const qs = params.toString(); + const clean = window.location.pathname + (qs ? '?' + qs : '') + + window.location.hash; + window.history.replaceState(null, '', clean); + } else { + try { + const saved = sessionStorage.getItem('soup_auth_token'); + if (saved) window._authToken = saved; + } catch (e) {} + } + } catch (e) { + // Defensive: never block app load. + } +})(); + // --- State --- let currentPage = 'dashboard'; let runsData = []; diff --git a/soup_cli/utils/backend_detect.py b/soup_cli/utils/backend_detect.py new file mode 100644 index 0000000..37803da --- /dev/null +++ b/soup_cli/utils/backend_detect.py @@ -0,0 +1,120 @@ +"""v0.53.9 #28 — Auto-detect inference backend from a model directory. + +Reads `config.json` `architectures` field; falls back to `transformers` +on missing config, decode errors, or non-LLM arches. Returns a closed-set +backend identifier so callers can dispatch deterministically. +""" + +from __future__ import annotations + +import json +import os +import stat +from pathlib import Path +from typing import Optional + +from soup_cli.utils.paths import is_under_cwd + +# Closed allowlist — matches `soup serve --backend`. +SUPPORTED_BACKENDS = frozenset({"transformers", "vllm", "sglang", "mlx"}) + +# Architecture-name keyword -> preferred backend. First match wins; the +# scan walks `architectures` left-to-right. +_MLX_KEYWORDS = frozenset({"mlx", "mlx_lm"}) + + +def _read_config_json(model_dir: Path) -> Optional[dict]: + """Read `/config.json` with TOCTOU-safe symlink rejection. + + Returns None on any error (missing dir, symlinked config, decode + failure). Never raises — caller treats None as "unknown arch". + """ + config_path = model_dir / "config.json" + if not config_path.is_file(): + return None + try: + # Reject symlinked config.json (mirrors v0.53.1 #82 policy). + st = os.lstat(config_path) + if stat.S_ISLNK(st.st_mode): + return None + except OSError: + return None + try: + with open(config_path, encoding="utf-8") as fh: + data = json.load(fh) + except (OSError, json.JSONDecodeError): + return None + if not isinstance(data, dict): + return None + return data + + +def _has_mlx_weights(model_dir: Path) -> bool: + """Return True if the directory has MLX-format `weights.npz` (mlx-lm convention). + + Each entry is `os.lstat`-checked so a symlinked `weights.npz` does not + accidentally trigger MLX dispatch (mirrors v0.33.0 #22 TOCTOU policy). + """ + try: + for path in model_dir.iterdir(): + if path.name.lower() != "weights.npz": + continue + try: + st = os.lstat(path) + except OSError: + continue + if stat.S_ISLNK(st.st_mode): + continue + if stat.S_ISREG(st.st_mode): + return True + except OSError: + return False + return False + + +def detect_backend(model_path: str, *, env: Optional[dict] = None) -> str: + """Probe `model_path` and return a preferred backend identifier. + + Resolution order: + 1. Env hint `SOUP_BENCH_BACKEND` (validated against allowlist). + 2. MLX weight files in the model dir. + 3. `architectures` field in `config.json` (always implies transformers + today; vllm / sglang need explicit selection because they wrap + the same HF weights). + 4. Fallback: `transformers`. + + `model_path` must be a non-empty str; on traversal / NUL / missing dir + the helper falls back to `transformers` silently. + """ + if not isinstance(model_path, str) or not model_path: + return "transformers" + if "\x00" in model_path: + return "transformers" + + env = env if env is not None else os.environ + hint = env.get("SOUP_BENCH_BACKEND") + if isinstance(hint, str) and hint: + canonical = hint.strip().lower() + if canonical in SUPPORTED_BACKENDS: + return canonical + + # Containment guard — only probe paths inside cwd. Out-of-cwd paths + # are still benched; we just skip the local-file probe. + if not is_under_cwd(model_path): + return "transformers" + real = os.path.realpath(model_path) + model_dir = Path(real) + if not model_dir.is_dir(): + return "transformers" + + if _has_mlx_weights(model_dir): + return "mlx" + + config = _read_config_json(model_dir) + if config: + # MLX configs sometimes carry `mlx`-prefixed model_type. + model_type = str(config.get("model_type") or "").lower() + if any(kw in model_type for kw in _MLX_KEYWORDS): + return "mlx" + + return "transformers" diff --git a/soup_cli/utils/reasoning_parser.py b/soup_cli/utils/reasoning_parser.py index 0db3bc5..eaa9b7e 100644 --- a/soup_cli/utils/reasoning_parser.py +++ b/soup_cli/utils/reasoning_parser.py @@ -7,6 +7,7 @@ to v0.44.1. from __future__ import annotations +import re from types import MappingProxyType from typing import Mapping, Optional @@ -49,3 +50,49 @@ def parser_description(name: str) -> Optional[str]: if not isinstance(name, str): return None return _REASONING_PARSERS.get(name.lower()) + + +# v0.53.9 #98 — Per-parser regex matrix for `...` stripping. +# All four parsers strip the same standard `...` block; +# OpenThinker uses `<|begin_of_thought|>...<|end_of_thought|>` per upstream. +_THINK_RE = re.compile(r"]*>.*?", re.IGNORECASE | re.DOTALL) +_OPENTHINKER_RE = re.compile( + r"<\|begin_of_thought\|>.*?<\|end_of_thought\|>", re.DOTALL +) + + +def strip_reasoning(text: str, parser: Optional[str]) -> str: + """Strip reasoning-trace blocks from `text` per `parser`. + + Returns `text` unchanged when `parser` is `None`/empty, when the parser + is unknown (defensive — caller should validate first), or when `text` + is not a str. Idempotent: applying twice yields the same result. + + Cap: input length capped at 1 MiB; longer payloads are returned + unchanged to avoid pathological regex backtracking. + """ + if not isinstance(text, str): + return text # type: ignore[return-value] + if not parser: + return text + if len(text) > 1_048_576: + return text + try: + canonical = validate_parser_name(parser) + except (TypeError, ValueError): + return text + # Fast pre-check: skip the regex entirely when the marker token is + # absent. Bounds worst-case re.sub time on adversarial inputs with + # no actual reasoning blocks. + if canonical == "openthinker": + if "<|begin_of_thought|>" not in text: + return text + out = _OPENTHINKER_RE.sub("", text) + else: + if " ToolOutputsBuffer: + """Return the process-wide tool-output buffer.""" + return _GLOBAL_TOOL_BUFFER + + +def reset_global_tool_buffer() -> None: + """Replace the process-wide buffer (test hook).""" + global _GLOBAL_TOOL_BUFFER + _GLOBAL_TOOL_BUFFER = ToolOutputsBuffer() diff --git a/soup_cli/utils/train_event_buffer.py b/soup_cli/utils/train_event_buffer.py new file mode 100644 index 0000000..cd9e779 --- /dev/null +++ b/soup_cli/utils/train_event_buffer.py @@ -0,0 +1,129 @@ +"""v0.53.9 #94 — Module-level training event buffer for SSE streaming. + +Thread-safe deque of `TrainEvent` payloads. The `SoupTrainerCallback` +pushes events on log / save / evaluate; the FastAPI `/api/train/stream` +endpoint drains pending events as SSE frames. + +Decoupled from FastAPI so callbacks can push without importing it. +""" + +from __future__ import annotations + +from collections import deque +from threading import Lock +from typing import Any, Deque, List, Optional + +from soup_cli.utils.sse_train_stream import TrainEvent + +_MAX_EVENTS = 1000 + + +class TrainEventBuffer: + """Thread-safe ring buffer of `TrainEvent` objects. + + `push` appends one event (dropping the oldest on overflow); + `drain` returns all events queued since the last drain. Consumers + can also pass a `since` cursor for catch-up subscribers. + """ + + def __init__(self, *, maxlen: int = _MAX_EVENTS) -> None: + if isinstance(maxlen, bool) or not isinstance(maxlen, int): + raise TypeError("maxlen must be int") + if maxlen <= 0: + raise ValueError("maxlen must be > 0") + self._events: Deque[TrainEvent] = deque(maxlen=maxlen) + self._cursor: int = 0 + self._lock = Lock() + + def push(self, event: TrainEvent) -> int: + """Append an event. Returns the event's monotonic cursor id.""" + if not isinstance(event, TrainEvent): + raise TypeError("event must be TrainEvent") + with self._lock: + self._cursor += 1 + self._events.append(event) + return self._cursor + + def drain(self) -> List[TrainEvent]: + """Return every queued event and clear the buffer. + + Note: destructive — concurrent subscribers will starve each other. + For multi-subscriber SSE, use `snapshot_since(cursor)` instead. + """ + with self._lock: + data = list(self._events) + self._events.clear() + return data + + def cursor(self) -> int: + """Return the latest assigned cursor id (0 if empty).""" + with self._lock: + return self._cursor + + def snapshot_since(self, since: int) -> tuple[List[TrainEvent], int]: + """Return events with cursor > `since`, plus the new cursor. + + Non-destructive — every subscriber gets its own catch-up view. + Returns `(events, latest_cursor)`. When the buffer has rolled over + past `since` (i.e. the requested cursor is older than the oldest + retained event), returns every retained event. + """ + if isinstance(since, bool) or not isinstance(since, int): + raise TypeError("since must be int") + with self._lock: + latest = self._cursor + events = list(self._events) + if since >= latest: + return [], latest + # Cursor ids are monotonic; the i-th retained event has cursor + # `latest - (len - 1 - i)`. Slice off the prefix the caller already saw. + start = latest - len(events) + 1 # cursor id of events[0] + offset = max(0, since - start + 1) + return events[offset:], latest + + def snapshot(self, *, limit: Optional[int] = None) -> List[TrainEvent]: + """Return a copy of the latest `limit` events without clearing.""" + if limit is not None: + if isinstance(limit, bool) or not isinstance(limit, int): + raise TypeError("limit must be int or None") + if limit < 0: + raise ValueError("limit must be >= 0") + with self._lock: + data = list(self._events) + if limit is None: + return data + if limit == 0: + return [] + return data[-limit:] + + def clear(self) -> None: + with self._lock: + self._events.clear() + + +# Module-level singleton consumed by SoupTrainerCallback and the FastAPI +# SSE endpoint. Tests can call `reset_global_buffer()` to start fresh. +_GLOBAL_BUFFER: TrainEventBuffer = TrainEventBuffer() + + +def get_global_buffer() -> TrainEventBuffer: + """Return the process-wide event buffer.""" + return _GLOBAL_BUFFER + + +def reset_global_buffer() -> None: + """Replace the process-wide buffer (test hook).""" + global _GLOBAL_BUFFER + _GLOBAL_BUFFER = TrainEventBuffer() + + +def push_train_event(event: Any) -> Optional[int]: + """Push an event onto the global buffer. Returns the cursor id. + + Silently ignores non-TrainEvent input so a misbehaving callback can + never crash the training loop. Use the typed `get_global_buffer().push` + when you want explicit validation. + """ + if not isinstance(event, TrainEvent): + return None + return _GLOBAL_BUFFER.push(event) diff --git a/tests/test_v0538.py b/tests/test_v0538.py index 110a6b3..45ce89f 100644 --- a/tests/test_v0538.py +++ b/tests/test_v0538.py @@ -596,10 +596,16 @@ class TestVersionBump: def test_init_version(self): import soup_cli - # v0.53.8.1 is the PyPI-recoverable patch (force-include duplicate - # fix); accept either as the shipped string. - assert soup_cli.__version__.startswith("0.53.8") + # Version-string is forward-monotonic: v0.53.8 baseline + any later + # release (v0.53.9, v0.54.0, ...) keeps this contract green. + assert soup_cli.__version__ >= "0.53.8" def test_pyproject_version(self): text = (_repo_root() / "pyproject.toml").read_text(encoding="utf-8") - assert 'version = "0.53.8' in text + # Pyproject must declare some 0.53.x or later string; the literal + # "0.53.8" baseline is allowed to drift forward. + import re + + match = re.search(r'^version\s*=\s*"([^"]+)"', text, flags=re.MULTILINE) + assert match is not None + assert match.group(1) >= "0.53.8" diff --git a/tests/test_v0539.py b/tests/test_v0539.py new file mode 100644 index 0000000..14cfd5b --- /dev/null +++ b/tests/test_v0539.py @@ -0,0 +1,849 @@ +"""Tests for v0.53.9 — Live Dashboard + UX + Bench + Standalone CLIs. + +Covers #94 SSE stream, #95 ui --public + QR, #98 reasoning-parser strip, +#100 tool-outputs API, #15 tokenizer train, #26 bench percentiles, +#28 bench backend auto-detect, #12 example doc. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +import soup_cli +from soup_cli.cli import app + +# ----------------------------------------------------------------- version + +def test_version_bump_to_0_53_9(): + assert soup_cli.__version__ == "0.53.9" + + +# ----------------------------------------------------- #94 SSE event buffer + +def test_train_event_buffer_push_and_drain(): + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer() + cursor = buffer.push(TrainEvent(type="metric", step=1, loss=0.5)) + assert cursor == 1 + cursor = buffer.push(TrainEvent(type="metric", step=2, loss=0.4)) + assert cursor == 2 + drained = buffer.drain() + assert len(drained) == 2 + assert drained[0].step == 1 + # After draining, buffer is empty. + assert buffer.drain() == [] + + +def test_train_event_buffer_rejects_non_event(): + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer() + with pytest.raises(TypeError): + buffer.push("not an event") + with pytest.raises(TypeError): + buffer.push({"type": "metric"}) + + +def test_train_event_buffer_maxlen_validation(): + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + with pytest.raises(TypeError): + TrainEventBuffer(maxlen=True) + with pytest.raises(ValueError): + TrainEventBuffer(maxlen=0) + with pytest.raises(ValueError): + TrainEventBuffer(maxlen=-3) + + +def test_train_event_buffer_snapshot_limits(): + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer() + for step in range(5): + buffer.push(TrainEvent(type="metric", step=step)) + snap = buffer.snapshot(limit=3) + assert len(snap) == 3 + assert [e.step for e in snap] == [2, 3, 4] + # Snapshot does not drain. + assert len(buffer.snapshot()) == 5 + assert buffer.snapshot(limit=0) == [] + with pytest.raises(TypeError): + buffer.snapshot(limit=True) + with pytest.raises(ValueError): + buffer.snapshot(limit=-1) + + +def test_train_event_buffer_overflow_drops_oldest(): + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer(maxlen=3) + for step in range(5): + buffer.push(TrainEvent(type="metric", step=step)) + # Only latest three retained. + drained = buffer.drain() + assert [e.step for e in drained] == [2, 3, 4] + + +def test_push_train_event_silent_on_bad_input(): + from soup_cli.utils.train_event_buffer import ( + get_global_buffer, + push_train_event, + reset_global_buffer, + ) + + reset_global_buffer() + assert push_train_event("nope") is None + assert push_train_event(None) is None + assert get_global_buffer().drain() == [] + + +def test_push_train_event_happy(): + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import ( + get_global_buffer, + push_train_event, + reset_global_buffer, + ) + + reset_global_buffer() + cursor = push_train_event(TrainEvent(type="metric", step=42, loss=0.1)) + assert cursor == 1 + events = get_global_buffer().drain() + assert len(events) == 1 + assert events[0].step == 42 + + +# ---------------------------------------------------- #94 SSE FastAPI route + +def test_api_train_stream_emits_pending_events(): + pytest.importorskip("fastapi") + from fastapi.testclient import TestClient + + from soup_cli.ui.app import create_app + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import ( + push_train_event, + reset_global_buffer, + ) + + reset_global_buffer() + push_train_event(TrainEvent(type="metric", step=1, loss=0.5)) + push_train_event(TrainEvent(type="status", message="started")) + + app_inst = create_app() + client = TestClient(app_inst) + response = client.get("/api/train/stream") + assert response.status_code == 200 + body = response.text + # Frames are W3C SSE. + assert "data:" in body + assert '"loss":0.5' in body or '"loss": 0.5' in body + # Closes with a done status event. + assert '"message":"done"' in body or '"message": "done"' in body + + +# ------------------------------------------------- #100 tool-outputs API + +def test_global_tool_buffer_round_trip(): + from soup_cli.utils.tool_outputs import ( + get_global_tool_buffer, + reset_global_tool_buffer, + ) + + reset_global_tool_buffer() + buf = get_global_tool_buffer() + buf.record_call( + name="search", + started_ts=1.0, + duration_ms=12.5, + success=True, + output_preview="hits=3", + ) + snap = buf.snapshot() + assert len(snap) == 1 + assert snap[0].name == "search" + + +def test_api_tool_outputs_endpoint(): + pytest.importorskip("fastapi") + from fastapi.testclient import TestClient + + from soup_cli.ui.app import create_app + from soup_cli.utils.tool_outputs import ( + get_global_tool_buffer, + reset_global_tool_buffer, + ) + + reset_global_tool_buffer() + buf = get_global_tool_buffer() + buf.record_call( + name="calculator", + started_ts=1.23, + duration_ms=4.5, + success=True, + output_preview="result=42", + ) + buf.record_call( + name="search", + started_ts=2.34, + duration_ms=99.9, + success=False, + output_preview="", + error="timeout", + ) + + app_inst = create_app() + client = TestClient(app_inst) + response = client.get("/api/tool-outputs?limit=5") + assert response.status_code == 200 + payload = response.json() + assert payload["count"] == 2 + names = [r["name"] for r in payload["records"]] + assert names == ["calculator", "search"] + assert payload["records"][1]["error"] == "timeout" + + +def test_api_tool_outputs_rejects_out_of_bounds_limit(): + pytest.importorskip("fastapi") + from fastapi.testclient import TestClient + + from soup_cli.ui.app import create_app + + client = TestClient(create_app()) + # limit must be 1..1000 + assert client.get("/api/tool-outputs?limit=0").status_code == 422 + assert client.get("/api/tool-outputs?limit=1001").status_code == 422 + + +# ---------------------------------------------------- #98 reasoning-parser + +@pytest.mark.parametrize( + "parser,raw,expected", + [ + ( + "deepseek-r1", + "secretfinal answer", + "final answer", + ), + ( + "qwen3", + "step1\nstep2\nresponse", + "response", + ), + ( + "phi4", + "no think tags here", + "no think tags here", + ), + ( + "openthinker", + "<|begin_of_thought|>cot<|end_of_thought|>visible", + "visible", + ), + ], +) +def test_strip_reasoning_per_parser(parser, raw, expected): + from soup_cli.utils.reasoning_parser import strip_reasoning + + assert strip_reasoning(raw, parser) == expected + + +def test_strip_reasoning_no_op_paths(): + from soup_cli.utils.reasoning_parser import strip_reasoning + + # None parser short-circuits. + assert strip_reasoning("xy", None) == "xy" + # Empty string parser short-circuits. + assert strip_reasoning("hello", "") == "hello" + # Unknown parser is silently no-op. + assert strip_reasoning("xy", "unknown-parser") == ( + "xy" + ) + # Non-string input passes through. + assert strip_reasoning(123, "deepseek-r1") == 123 # type: ignore[arg-type] + + +def test_strip_reasoning_oversize_passthrough(): + from soup_cli.utils.reasoning_parser import strip_reasoning + + big = "x" + ("a" * 1_100_000) + out = strip_reasoning(big, "deepseek-r1") + assert out == big # >1MiB returned unchanged + + +def test_strip_reasoning_idempotent(): + from soup_cli.utils.reasoning_parser import strip_reasoning + + out = strip_reasoning("afinal", "deepseek-r1") + assert strip_reasoning(out, "deepseek-r1") == out + + +def test_serve_create_app_accepts_reasoning_parser_kwarg(): + """`_create_app` must accept the new `reasoning_parser=` kwarg.""" + import inspect + + from soup_cli.commands.serve import _create_app + + sig = inspect.signature(_create_app) + assert "reasoning_parser" in sig.parameters + + +def test_serve_help_lists_reasoning_parser_flag(): + runner = CliRunner() + result = runner.invoke(app, ["serve", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--reasoning-parser" in result.output + + +# --------------------------------------------------- #15 tokenizer train + +def test_tokenizer_train_help_listed(): + runner = CliRunner() + result = runner.invoke(app, ["tokenizer", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "train" in result.output + + +def test_tokenizer_train_subcommand_help(): + runner = CliRunner() + result = runner.invoke(app, ["tokenizer", "train", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--vocab-size" in result.output + assert "--input" in result.output + assert "--output" in result.output + + +def test_tokenizer_train_rejects_out_of_cwd_input(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + outside = tmp_path.parent / "outside.jsonl" + outside.write_text('{"text": "hi"}\n', encoding="utf-8") + runner = CliRunner() + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", str(outside), + "--output", "out", + ], + ) + assert result.exit_code != 0 + assert "current working directory" in result.output + + +def test_tokenizer_train_rejects_vocab_size_bounds(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + fixture = tmp_path / "corpus.jsonl" + fixture.write_text('{"text": "hello world"}\n', encoding="utf-8") + runner = CliRunner() + # Too small. + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", "corpus.jsonl", + "--vocab-size", "10", + ], + ) + assert result.exit_code != 0 + assert "vocab-size" in result.output + # Too large. + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", "corpus.jsonl", + "--vocab-size", "999999", + ], + ) + assert result.exit_code != 0 + + +def test_tokenizer_train_happy_path(tmp_path, monkeypatch): + pytest.importorskip("tokenizers") + monkeypatch.chdir(tmp_path) + fixture = tmp_path / "corpus.jsonl" + rows = [ + {"text": "the quick brown fox jumps over the lazy dog"}, + {"text": "pack my box with five dozen liquor jugs"}, + {"text": "how vexingly quick daft zebras jump"}, + ] * 20 + fixture.write_text( + "\n".join(json.dumps(r) for r in rows) + "\n", + encoding="utf-8", + ) + runner = CliRunner() + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", "corpus.jsonl", + "--vocab-size", "300", + "--output", "bpe_out", + "--min-frequency", "1", + ], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "bpe_out" / "tokenizer.json").is_file() + assert (tmp_path / "bpe_out" / "vocab.json").is_file() + vocab = json.loads( + (tmp_path / "bpe_out" / "vocab.json").read_text(encoding="utf-8") + ) + assert isinstance(vocab, dict) + assert len(vocab) >= 1 + + +# -------------------------------------------------- #28 backend auto-detect + +def test_detect_backend_fallback_transformers(): + from soup_cli.utils.backend_detect import detect_backend + + assert detect_backend("does-not-exist") == "transformers" + assert detect_backend("") == "transformers" + assert detect_backend("abc\x00def") == "transformers" + + +def test_detect_backend_env_hint(monkeypatch): + from soup_cli.utils.backend_detect import detect_backend + + monkeypatch.setenv("SOUP_BENCH_BACKEND", "VLLM") + assert detect_backend("anything") == "vllm" + monkeypatch.setenv("SOUP_BENCH_BACKEND", "bogus-name") + # bogus env hint falls back to detection. + out = detect_backend("anything") + assert out in {"transformers", "mlx"} + + +def test_detect_backend_mlx_weights(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + model_dir = tmp_path / "mlx_model" + model_dir.mkdir() + (model_dir / "weights.npz").write_bytes(b"\x00\x00") + from soup_cli.utils.backend_detect import detect_backend + + assert detect_backend(str(model_dir)) == "mlx" + + +def test_detect_backend_transformers_config(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + model_dir = tmp_path / "hf_model" + model_dir.mkdir() + (model_dir / "config.json").write_text( + json.dumps({"model_type": "llama", "architectures": ["LlamaForCausalLM"]}), + encoding="utf-8", + ) + from soup_cli.utils.backend_detect import detect_backend + + assert detect_backend(str(model_dir)) == "transformers" + + +def test_detect_backend_config_malformed_falls_back(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + model_dir = tmp_path / "junk" + model_dir.mkdir() + (model_dir / "config.json").write_text("not-json{", encoding="utf-8") + from soup_cli.utils.backend_detect import detect_backend + + assert detect_backend(str(model_dir)) == "transformers" + + +# ----------------------------------------------------- #26 bench percentiles + +def test_bench_help_lists_percentile_flags(): + runner = CliRunner() + result = runner.invoke(app, ["bench", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--p50" in result.output + assert "--p95" in result.output + assert "--backend" in result.output + + +def test_bench_invalid_backend_rejected(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + runner = CliRunner() + result = runner.invoke( + app, + ["bench", "fake-model", "--backend", "evilbackend"], + ) + # Either exit 2 (Typer reject) or exit 1 (resolve failure later). + assert result.exit_code != 0 + + +# --------------------------------------------------- #95 ui --public + QR + +def test_ui_help_includes_public_and_auth_token(): + runner = CliRunner() + result = runner.invoke(app, ["ui", "--help"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert "--public" in result.output + assert "--auth-token" in result.output + + +def test_ui_show_token_via_custom_token(tmp_path, monkeypatch): + pytest.importorskip("fastapi") + monkeypatch.chdir(tmp_path) + runner = CliRunner() + valid = "A" * 32 # urlsafe-base64 shape + result = runner.invoke( + app, + [ + "ui", + "--show-token", + "--auth-token", valid, + "--no-browser", + ], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert valid in result.output + + +def test_ui_rejects_malformed_auth_token(): + pytest.importorskip("fastapi") + runner = CliRunner() + result = runner.invoke( + app, + [ + "ui", + "--auth-token", "short", # < 16 chars + "--show-token", + "--no-browser", + ], + ) + assert result.exit_code != 0 + assert "auth-token" in result.output or "token" in result.output + + +# ------------------------------------------------ #12 example workflow doc + +def test_train_event_buffer_snapshot_since_returns_only_new_events(): + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer() + buffer.push(TrainEvent(type="metric", step=0)) + buffer.push(TrainEvent(type="metric", step=1)) + events, cursor = buffer.snapshot_since(0) + assert [e.step for e in events] == [0, 1] + assert cursor == 2 + + buffer.push(TrainEvent(type="metric", step=2)) + events, cursor = buffer.snapshot_since(cursor) + assert [e.step for e in events] == [2] + assert cursor == 3 + + # Caller cursor caught up — no new events. + events, cursor = buffer.snapshot_since(cursor) + assert events == [] + assert cursor == 3 + + +def test_train_event_buffer_snapshot_since_rejects_non_int(): + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer() + with pytest.raises(TypeError): + buffer.snapshot_since("0") # type: ignore[arg-type] + with pytest.raises(TypeError): + buffer.snapshot_since(True) + + +def test_train_event_buffer_concurrent_subscribers_isolated(): + """Multiple subscribers using snapshot_since each see every event.""" + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer() + for step in range(3): + buffer.push(TrainEvent(type="metric", step=step)) + + a_events, _ = buffer.snapshot_since(0) + b_events, _ = buffer.snapshot_since(0) + assert [e.step for e in a_events] == [0, 1, 2] + assert [e.step for e in b_events] == [0, 1, 2] + + +def test_tokenizer_train_rejects_symlink_input(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + real = tmp_path / "real.jsonl" + real.write_text('{"text": "hello"}\n', encoding="utf-8") + link = tmp_path / "link.jsonl" + try: + os.symlink(real, link) + except (OSError, NotImplementedError, AttributeError): + pytest.skip("symlinks not supported on this platform") + runner = CliRunner() + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", "link.jsonl", + "--output", "out", + ], + ) + assert result.exit_code != 0 + assert "symlink" in result.output.lower() + + +def test_train_event_buffer_cursor_and_clear(): + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer() + assert buffer.cursor() == 0 + buffer.push(TrainEvent(type="metric", step=0)) + buffer.push(TrainEvent(type="metric", step=1)) + assert buffer.cursor() == 2 + buffer.clear() + assert buffer.cursor() == 2 # cursor is monotonic; clear does NOT reset + assert buffer.snapshot() == [] + + +def test_train_event_buffer_snapshot_since_rollover(): + """`maxlen=3` then push 5 events — `snapshot_since(0)` returns the 3 retained.""" + from soup_cli.utils.sse_train_stream import TrainEvent + from soup_cli.utils.train_event_buffer import TrainEventBuffer + + buffer = TrainEventBuffer(maxlen=3) + for step in range(5): + buffer.push(TrainEvent(type="metric", step=step)) + # Latest cursor is 5; only the last 3 events are retained. + events, cursor = buffer.snapshot_since(0) + assert [e.step for e in events] == [2, 3, 4] + assert cursor == 5 + # Asking for events newer than cursor 4 (step=3) returns the tail. + events, cursor = buffer.snapshot_since(4) + assert [e.step for e in events] == [4] + + +def test_set_auth_token_concurrent_rotation_safe(): + """Concurrent set_auth_token + get_auth_token never observes a partial token.""" + import threading + + from soup_cli.ui.app import get_auth_token, set_auth_token + + valid_a = "A" * 32 + valid_b = "B" * 32 + set_auth_token(valid_a) + + errors: list[str] = [] + + def _writer(): + for _ in range(100): + set_auth_token(valid_a) + set_auth_token(valid_b) + + def _reader(): + for _ in range(100): + tok = get_auth_token() + if tok not in (valid_a, valid_b): + errors.append(f"torn token: {tok!r}") + + threads = [threading.Thread(target=_writer) for _ in range(2)] + threads += [threading.Thread(target=_reader) for _ in range(6)] + for t in threads: + t.start() + for t in threads: + t.join() + assert errors == [] + + +def test_set_auth_token_rejects_bool_and_non_str(): + from soup_cli.ui.app import get_auth_token, set_auth_token + + before = get_auth_token() + with pytest.raises(TypeError): + set_auth_token(True) # type: ignore[arg-type] + with pytest.raises(TypeError): + set_auth_token(42) # type: ignore[arg-type] + # Token must be unchanged after rejected rotation. + assert get_auth_token() == before + + +def test_strip_reasoning_swallows_null_byte_parser(): + from soup_cli.utils.reasoning_parser import strip_reasoning + + out = strip_reasoning("xfinal", "\x00") + assert out == "xfinal" + + +def test_strip_reasoning_crlf_residue(): + from soup_cli.utils.reasoning_parser import strip_reasoning + + out = strip_reasoning("a\r\n\r\nfinal", "deepseek-r1") + # Both \r and \n stripped — content preserved. + assert out == "final" + + +def test_strip_reasoning_fast_path_skips_regex(): + """Inputs without the marker token bypass the regex entirely.""" + from soup_cli.utils.reasoning_parser import strip_reasoning + + # Long input WITHOUT amidbfinal", "deepseek-r1" + ) + assert out == "midfinal" + + +def test_detect_backend_rejects_symlinked_mlx_weights(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + model_dir = tmp_path / "model" + model_dir.mkdir() + real_weights = tmp_path / "real.npz" + real_weights.write_bytes(b"\x00") + link = model_dir / "weights.npz" + try: + os.symlink(real_weights, link) + except (OSError, NotImplementedError, AttributeError): + pytest.skip("symlinks not supported on this platform") + from soup_cli.utils.backend_detect import detect_backend + + # Symlinked weights.npz must NOT trigger MLX dispatch. + assert detect_backend(str(model_dir)) == "transformers" + + +def test_detect_backend_mlx_model_type_in_config(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + model_dir = tmp_path / "model" + model_dir.mkdir() + (model_dir / "config.json").write_text( + json.dumps({"model_type": "mlx_lm_llama"}), + encoding="utf-8", + ) + from soup_cli.utils.backend_detect import detect_backend + + assert detect_backend(str(model_dir)) == "mlx" + + +def test_detect_backend_config_non_dict_root(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + model_dir = tmp_path / "model" + model_dir.mkdir() + (model_dir / "config.json").write_text("[1, 2, 3]", encoding="utf-8") + from soup_cli.utils.backend_detect import detect_backend + + assert detect_backend(str(model_dir)) == "transformers" + + +def test_tokenizer_train_plaintext_corpus(tmp_path, monkeypatch): + """`.txt` extractor branch (different from JSONL).""" + pytest.importorskip("tokenizers") + monkeypatch.chdir(tmp_path) + corpus = tmp_path / "corpus.txt" + corpus.write_text( + "\n".join(["the quick brown fox jumps over the lazy dog"] * 80) + "\n", + encoding="utf-8", + ) + runner = CliRunner() + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", "corpus.txt", + "--vocab-size", "300", + "--output", "bpe_out", + "--min-frequency", "1", + ], + ) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert (tmp_path / "bpe_out" / "tokenizer.json").is_file() + + +def test_tokenizer_train_rejects_min_frequency_zero(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "corpus.jsonl").write_text( + '{"text": "hello"}\n', encoding="utf-8" + ) + runner = CliRunner() + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", "corpus.jsonl", + "--min-frequency", "0", + ], + ) + assert result.exit_code != 0 + assert "min-frequency" in result.output + + +def test_tokenizer_train_special_token_dedup_and_validation(tmp_path, monkeypatch): + pytest.importorskip("tokenizers") + monkeypatch.chdir(tmp_path) + (tmp_path / "corpus.jsonl").write_text( + "\n".join(['{"text": "hello world"}'] * 20) + "\n", + encoding="utf-8", + ) + runner = CliRunner() + # NUL byte in special-token rejected. + result = runner.invoke( + app, + [ + "tokenizer", "train", + "--input", "corpus.jsonl", + "--special-token", "\x00bad", + "--vocab-size", "300", + "--min-frequency", "1", + ], + ) + assert result.exit_code != 0 + assert "special-token" in result.output + + +def test_bench_p50_p95_runs_with_help_only(): + """Smoke: --p50/--p95 flags are wired (without spinning up a real model).""" + runner = CliRunner() + result = runner.invoke(app, ["bench", "--help"]) + assert result.exit_code == 0 + # The flag descriptions must mention the v0.53.9 release tag so future + # patches don't silently drop the percentile rows. + assert "p50" in result.output and "p95" in result.output + + +def test_reset_global_tool_buffer_clears_state(): + from soup_cli.utils.tool_outputs import ( + get_global_tool_buffer, + reset_global_tool_buffer, + ) + + buf = get_global_tool_buffer() + buf.record_call( + name="ping", + started_ts=1.0, + duration_ms=0.5, + success=True, + output_preview="ok", + ) + assert len(buf.snapshot()) >= 1 + reset_global_tool_buffer() + fresh = get_global_tool_buffer() + assert fresh.snapshot() == [] + + +def test_synthetic_workflow_doc_exists(): + repo_root = Path(__file__).resolve().parent.parent + doc = repo_root / "examples" / "synthetic_workflow.md" + assert doc.is_file() + body = doc.read_text(encoding="utf-8") + # End-to-end walkthrough names every step. + assert "soup data generate" in body + assert "soup data filter" in body + assert "soup data score" in body + assert "soup train" in body