diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 626718b..40fafb5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (189 files, 8285 tests) +tests/ - Test suite (190 files, 8330 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index 5bf3742..1df68a5 100644 --- a/README.md +++ b/README.md @@ -42,17 +42,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.53.9 — Live Dashboard + UX + Bench + Standalone CLIs**: Eight features that close out the v0.44.x live-monitoring deferrals plus a long tail of standalone CLI wins. +**v0.53.10 — Quick wins + packaging + UX wiring**: Seven small wins that close the long tail of v0.40 → v0.53 deferred packaging / UX issues. -- **Live `/api/train/stream` SSE endpoint.** Async FastAPI route streams `TrainEvent` payloads (loss / lr / grad_norm / EMA / p95 tail latency) as W3C SSE frames. Per-subscriber cursor — multiple dashboard tabs each receive every event without starving each other. `asyncio.sleep` so uvicorn's async loop is never blocked. Drains gracefully with a `status=done` event when training exits. -- **`soup ui --public` + phone-scannable QR.** Binds `0.0.0.0`, detects the actual LAN IP via a `socket.SOCK_DGRAM` connect trick (no packets sent), and prints an ASCII QR encoding `http://:/?token=`. The SPA picks up `?token=` on load via `URLSearchParams`, caches it in `sessionStorage`, and rewrites the URL clean so the token doesn't sit in browser history. `--auth-token` override accepts a stable urlsafe-base64 token for repeatable phone bookmarks. CORS regex auto-widens to loopback + RFC1918 ranges when public; Bearer token gates mutating endpoints. Concurrent token rotation is now lock-protected. -- **`soup serve --reasoning-parser {deepseek-r1|qwen3|phi4|openthinker}`.** Strips `...` (or `<|begin_of_thought|>...<|end_of_thought|>` for OpenThinker) from responses before they hit the client. Pre-compiled regex with a marker-token fast-path (skips `re.sub` entirely when no marker is present) and a 1 MiB input cap. Strips only leading newlines after removal so code outputs that begin with intentional whitespace are preserved. -- **`/api/tool-outputs` + `ToolOutputsBuffer` global singleton.** Process-wide thread-safe ring buffer of tool-call records (`name` / `started_ts` / `duration_ms` / `success` / `output_preview` / `error`). New `/api/tool-outputs?limit=N` JSON endpoint reads the buffer for live dashboard polling. SFT trainer-side observation hook lands in a follow-up. -- **`soup tokenizer train`.** New BPE training CLI via the `tokenizers` library. JSONL or plaintext input (with ShareGPT `messages` fallback), `--vocab-size` bounded [256, 200000], `--special-token` repeatable with NUL/oversize/dedup validation. `os.lstat + S_ISLNK` rejection on the raw user-supplied input path BEFORE realpath, 50 MiB total / 8 KiB per-line caps, post-`mkdir` re-check on the output directory so a planted symlink can't redirect the write. -- **`soup bench --p50 --p95`.** Per-prompt latency capture rendered as an extra Rich table after the throughput row. Reuses `utils/tail_latency.summarise_latency` (v0.44.0). `--prompts-file` now rejects symlinked inputs on the raw path before realpath, in line with the project-wide TOCTOU policy. -- **`soup bench --backend auto`.** New default — probes the model directory for MLX `weights.npz` and falls back to `config.json` `model_type` keyword detection. `SOUP_BENCH_BACKEND` env hint overrides the probe. Every iterdir entry is `os.lstat`-checked so a symlinked `weights.npz` cannot trigger MLX dispatch. -- **`examples/synthetic_workflow.md` walkthrough.** End-to-end docs covering `soup data generate` → `filter` → `score` → `decontaminate` → `train`, plus the `soup ui --public` phone-monitoring loop. -- **+28 net new tests** (8257 → 8285) in `test_v0539.py`. Three review agents ran (python / code / security / tdd / verification); 34 findings fixed: H1 (QR token never consumed by SPA — now hydrates from URL into sessionStorage), H2 (`set_auth_token` rotation race — now lock-protected), tokenizer input/output symlink rejection (raw-path lstat), prompts-file symlink rejection, regex fast-path skip on missing markers, leading-newline strip vs full `lstrip()`, CORS regex for `--public` LAN mode, `_has_mlx_weights` per-entry lstat, and more. +- **`[mix]` + `[data-pro]` extras.** `pip install soup-cli[mix]` bundles `scikit-optimize` so `soup data mix --optimize` runs the real Bayesian loop instead of the v0.48.0 Dirichlet fallback. `pip install soup-cli[data-pro]` bundles `langdetect` (probabilistic language detection, broader coverage than the in-tree stopword heuristic) and `presidio-analyzer` (Microsoft Presidio entity recognition for PII — locations, dates, IBAN, etc. on top of the v0.47.0 email / phone / SSN / credit-card regex baseline). Both helpers fall through silently when the optional package is missing. +- **PostHog env override.** `SOUP_POSTHOG_KEY` + `SOUP_POSTHOG_ENDPOINT` env vars let operators point opt-in telemetry at their own PostHog project without a code change. Endpoint goes through the v0.51.0 SSRF policy (HTTPS-only, loopback HTTP only, RFC1918 / link-local rejection). Keys reject null bytes, control characters, and oversize (>256 char) input. A new sentinel-based default cleanly distinguishes "caller omitted endpoint" from "caller passed the default URL" so a `endpoint=_POSTHOG_ENDPOINT` explicit pin is NOT silently overridden by env. +- **`--hub` flag on `chat` / `serve` / `infer` / `merge` / `export` / `push`.** Closes the v0.53.8 known-limitation that only `train` and `data download` could route through non-HF hubs (ModelScope / Modelers). New `utils/hubs.prefetch_model_from_hub` is the shared, cwd-contained snapshot helper; every command body invokes the public `apply_hub_to_cli_model` adapter at the top. `push --hub` uses the `utils.hubs.upload_repo` upload-direction adapter; non-HF uploads skip HF-specific Collections + model-card auto-render. +- **`soup data download --hub ` live SDK.** Lifts the v0.53.8 advisory-only path — `--hub modelscope` now goes through `modelscope.msdatasets.MsDataset.load(...)`, `--hub modelers` uses `openmind_hub.snapshot_download(repo_type="dataset", ...)`. Friendly `pip install ` advisory fires when the SDK is missing. +- **Web UI Tool Outputs panel.** New "Tool Outputs" tab in `soup ui` polls `/api/tool-outputs?limit=100` every 3 seconds and renders the most recent tool-call records (name / started / duration / OK / output preview) in an XSS-safe table built via `document.createElement` + `textContent` (no `innerHTML` for user-controlled fields). Bearer token threaded via the v0.53.9 `window._authToken` bootstrap. +- **SFT callback `record_call` wiring.** `monitoring/callback.SoupTrainerCallback` now exposes an `on_step_end` hook that peeks at `kwargs.get("inputs", {})` for a `tool_calls` field and routes the count through the v0.53.9 global tool buffer so the Web UI panel populates from real training batches. Best-effort with blanket `except Exception: # noqa: BLE001` — training must never crash on observation failure. +- **+45 net new tests** (8285 → 8330) in `test_v05310.py`. Four review agents ran (python / code / security / tdd / verification-loop); 13 findings fixed: H1 (PostHog explicit-endpoint precedence — sentinel default), H2 (absolute `local_path` leak — reduced to relpath), H3 (Rich markup escape on `base` / `local_path` / `cache_dir`), H4 (callback `# noqa: BLE001` per project policy), MED `import time` moved out of `try` block, oversize / explicit-empty `api_key` rejection tests, source-grep regression guards (every non-push command imports `apply_hub_to_cli_model`; `data.py` no longer carries the legacy `"wait for v0.53.9"` advisory), `prefetch_model_from_hub` outside-cwd `cache_root` rejection, empty-list + bool-True `tool_calls` no-op tests. ## Why Soup? diff --git a/SECURITY.md b/SECURITY.md index 7af37b6..2453b3a 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.53.9 -- Full support (latest) +- v0.53.10 -- Full support (latest) +- v0.53.9 -- Full support - v0.53.8 -- Full support - v0.53.7 -- Full support - v0.53.6 -- Full support @@ -155,6 +156,7 @@ No known critical vulnerabilities in current releases. - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input. - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS` +- **v0.53.10 — Quick wins + packaging + UX wiring**: 7 features. 4 review-agent waves (python / code / security / tdd / verification-loop) ran; 13 findings fixed (4 HIGH + 5 MEDIUM + 4 LOW). (#152 Rich markup injection via `base` / `local_path` / `cache_dir`) `utils/hubs.prefetch_model_from_hub` now wraps every user-controlled string with `rich.markup.escape` before embedding in `console.print` — a crafted `base` like `[bold red]evil[/bold red]` no longer renders as styled output (security-review HIGH). (#152 absolute-path leak in advisory) the SDK-returned `local_path` and the `cache_dir` are reduced to `os.path.relpath` (basename fallback on `ValueError` / `OSError`) before display so `$HOME`-prefixed paths from `modelscope.snapshot_download` no longer leak in the dim advisory line (code-review HIGH; matches v0.34.0 `crash.py` policy). (#152 slug-regex traversal defence) `[^A-Za-z0-9._-]+` substitution neutralises path-separator + `..` segments in `base` before the cache subdir join, with a final `is_under_cwd` realpath check on the resulting `cache_dir` so a crafted `base: "../../etc"` cannot escape `.soup_hub_cache/`. (#152 `cache_root` user override) when caller supplies `cache_root` to `prefetch_model_from_hub`, the resolved `cache_dir` is `is_under_cwd`-checked — an outside-cwd override raises `ValueError("resolved hub cache dir escapes the current working directory")` (security-review MEDIUM; regression test in `test_prefetch_outside_cwd_cache_root_raises`). (#154 PostHog endpoint precedence) `_resolve_posthog_target` now uses a sentinel default `_POSTHOG_ENDPOINT_DEFAULT` so a caller passing `endpoint=_POSTHOG_ENDPOINT` literally is NOT silently overridden by `SOUP_POSTHOG_ENDPOINT` (code-review HIGH — documented precedence contract is now correctly enforced). (#154 control-char + oversize key rejection) `SOUP_POSTHOG_KEY` env values are rejected on null byte, any byte `< 0x20` (control char / CRLF injection defence), or length > 256 — prevents an operator dropping `\nAuthorization: bypass` into the env from leaking a second header to the PostHog endpoint. (#154 endpoint SSRF re-validation) `_telemetry_endpoint_is_safe` delegates to `utils.hubs.validate_hub_endpoint` so the resolved endpoint goes through the same v0.51.0 HTTPS-only + loopback-HTTP-only + RFC1918 / link-local / cloud-metadata rejection policy used for HF / ModelScope hub endpoints. (#155 XSS-by-construction) Tool Outputs panel renders every cell via `document.createElement('td')` + `td.textContent = value` — NO `innerHTML` for any user-controlled field. A tool-call record containing `` will render as visible text, not HTML. (#156 callback best-effort policy) `on_step_end` is wrapped in a blanket `except Exception: # noqa: BLE001` per the v0.39.0 best-effort design — a misbehaving tool buffer / global singleton failure can NEVER take down a multi-hour training run. Cross-cutting: 0 CRITICAL findings; test surface `test_v05310.py` carries 45 new tests including source-grep regression guards (`apply_hub_to_cli_model` imported in every non-push command; `wait for v0.53.9` advisory removed from `data.py`), `prefetch_model_from_hub` cache_root containment, PostHog explicit-endpoint vs env precedence, oversize-key + empty-key rejection, and callback bool / empty-list short-circuits. (v0.53.10) - **v0.53.9 — Live Dashboard + UX + Bench + Standalone CLIs**: 8 features. 3 review-agent waves (python / code / security / tdd) ran; 34 findings fixed (11 HIGH + 14 MEDIUM + 9 LOW). (#95 `set_auth_token` rotation race) module-level `_auth_token_lock: threading.Lock` now guards every read AND write of `_auth_token` in `ui/app.py`; `_verify_token` snapshots the expected `f"Bearer {_auth_token}"` under the lock before string comparison so concurrent rotation can never leave a window where some requests see the old token and some the new (security-review HIGH H2). 8-thread rotation stress test in `test_v0539.py`. (#95 QR token consumption) JS bootstrap in `ui/static/app.js` reads `?token=` from `URLSearchParams` on first load, populates `window._authToken`, caches in `sessionStorage`, and `history.replaceState`s the URL clean of the token so it does not sit in browser history (security-review HIGH H1 — token in QR URL was previously unreachable). (#95 CORS for `--public` LAN mode) when `host == "0.0.0.0"` the `CORSMiddleware` switches from a fixed `allow_origins=[f"http://{host}:{port}"]` (which never matches a real `Origin` header) to an `allow_origin_regex` covering loopback + RFC1918 ranges (`10.x.x.x` / `192.168.x.x` / `172.16-31.x.x`); Bearer token remains the actual security gate (security-review MEDIUM M5). (#15 `tokenizer.py` symlink rejection) `_validate_under_cwd` runs `os.lstat + stat.S_ISLNK` on the RAW user-supplied path BEFORE `os.path.realpath` resolution — `realpath`-first would silently follow symlinks and lstat the resolved regular-file target (security-review HIGH; matches v0.53.7 #106 policy). Post-`mkdir` re-check on the output directory defends against a symlink planted between containment check and write (security-review MEDIUM M1; matches v0.43.0 Part D `copy_bundle_to`). 50 MiB total + 8 KiB per-line corpus caps bound RAM on adversarial input (security-review HIGH). Panel + success messages use `.name` (basename only) to avoid leaking absolute paths (security-review MEDIUM M4; v0.34.0 `crash.py` policy). (#26 bench `--prompts-file` symlink) `os.lstat + S_ISLNK` on the raw path BEFORE realpath (security-review MEDIUM M2; parity with project TOCTOU policy). (#28 `_has_mlx_weights` symlink) per-entry `os.lstat` + `S_ISLNK` rejection so a symlinked `weights.npz` does NOT trigger MLX dispatch (code-review MEDIUM). (#98 `strip_reasoning` ReDoS bound) fast-path skip when the marker token is absent in `text.lower()` — bounds worst-case `re.sub` time on adversarial inputs without actual reasoning blocks; 1 MiB input cap; strips ONLY leading newlines after block removal so code outputs that begin with intentional spaces / tabs are preserved (security-review MEDIUM M3). (#94 SSE drain-race fix) `/api/train/stream` switched from destructive `buffer.drain()` to per-subscriber `snapshot_since(cursor)` so multiple concurrent dashboard tabs each receive every event (code-review HIGH); switched from `time.sleep(0.1)` in a sync generator to `asyncio.sleep(0.1)` in an `async def` generator so uvicorn's async loop is never blocked (code-review HIGH). (#94 `TrainEventBuffer` typing) closed allowlist + `TrainEvent`-only push validation; module-level singleton + test-hook `reset_global_buffer` is documented as test-only because reassignment under no lock is acceptable only when no training loop is active. (#100 `/api/tool-outputs`) `limit` Pydantic-bounded `[1, 1000]`; output preview already 4 KiB-capped by `ToolOutputsBuffer.record_call` (v0.44.0). Cross-cutting: 0 CRITICAL findings; test surface `test_v0539.py` carries 57 new tests including symlink rejection regressions, concurrent rotation stress, and SSE cursor rollover math. (v0.53.9) - **v0.53.8 — Remote data + Hubs + Trackers (wave 2)**: 6 features wired live. 3 review-agent waves (python / code / security) ran; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). (#130 hub dispatcher) `download_repo` / `upload_repo` reject `local_dir` / `folder_path` outside cwd via `utils.paths.is_under_cwd` (security-review HIGH — closes a gap where a public helper could write to arbitrary absolute paths); `_validate_repo_id_shape` rejects bool / null-byte / leading-slash / `..` segments / control characters (CR/LF, defence-in-depth against header injection) / oversize (>200 chars). `modelscope.push_model` does NOT accept `commit_message` — the sanitized message is omitted on that backend to avoid a silent TypeError at runtime (code-review HIGH). (#130 `commands/train.py` pre-fetch) cache subdir name sanitised via `re.sub(r"[^A-Za-z0-9._-]+", "__", cfg.base)` then strip-leading/trailing-dots to defeat Windows `..\\` traversal that would have escaped `.soup_hub_cache/` (security-review HIGH); resolved cache dir re-checked via `is_under_cwd` as defence-in-depth. `cfg.base = local_path` mutation replaced with `cfg = cfg.model_copy(update={"base": local_path})` so Pydantic field validators rerun (matches v0.33.0 #47 immutability policy; python-review + code-review HIGH). Pre-fetch idempotent — probes `cache_dir / config.json` and reuses on resume runs. `ImportError` print uses `rich.markup.escape` to defend against markup injection from a crafted dep name (security-review MEDIUM). (#90 telemetry) `_telemetry_endpoint_is_safe` re-runs the v0.51.0 hub-endpoint SSRF validator on every caller-supplied endpoint (HTTPS + private-IP / link-local / RFC1918 rejected) — defence-in-depth so a misconfigured `endpoint='https://10.0.0.1/'` cannot reach internal networks even when telemetry is opt-IN (security-review MEDIUM). Lazy `httpx` import + 1s hard timeout + silent-fail on every exception (no `traceback`). (#85 fsspec loader) `validate_remote_uri` runs BEFORE any `fsspec.open` connection (bucket regex, no userinfo / query / fragment); row-count capped at 1,000,000 to defend against pathological remote objects (security-review MEDIUM). (#69 `detect_space_sdk`) input length capped at 256 KB to match v0.39.0 Part E template policy — refuses to scan oversized requirements.txt that would degrade response time (security-review LOW). (#93 package-data) fixtures moved to `soup_cli/data/_fixtures/`; `_bundle_source_path` falls back to legacy `examples/data/` for editable installs. (#89 `tracker_missing_dep_message`) switched from `__import__(pkg)` to `importlib.util.find_spec` (non-executing probe) so swanlab can't initialise network threads during the absence check (code-review MEDIUM; `sys.modules[pkg] = None` sentinel still honoured for tests). Test surface: `tests/test_v0538.py` carries 66 new tests including 4 new cwd-containment + SSRF rejection regression guards. (v0.53.8) - **v0.53.7 — Data Forge + Pipeline live (wave 1)**: 11 features (6 new + 5 v0.53.6 stub-to-live), 1 CRITICAL + 11 HIGH + 17 MEDIUM + 10 LOW review findings fixed. (#103 `/v1/tools/bash` REVERTED to HTTP 501) security review caught that `/bin/sh -c` spawns a child process outside the RLVR sandbox's OS-level isolation (`unshare(CLONE_NEWNET)` on Linux, `sandbox-exec` on macOS, socket monkey-patch); a caller could reach the cloud-metadata service from the child shell. Endpoint returns 501 with v0.53.8 marker until container/namespace work lands. Python + web_search remain LIVE. (#103 `/v1/tools/{python,web_search}` Bearer auth gate) optional `auth_token` kwarg on `_create_app` + `Bearer` gate on both endpoints; default `None` preserves loopback-only back-compat per v0.30.0 Part C CORS policy. (#103 `/v1/tools/web_search` deny-by-default) `WebSearchConfig.domain_allowlist` defaults to empty = deny-all (returns 403 `"web_search disabled"`); only operators who supply a domain list via constructor enable the endpoint. 5s `httpx` timeout, 5-result cap, snippet null-byte stripped. (#102 `/v1/messages` redaction) malformed Anthropic payloads (incl. validator-shape errors) mapped to generic `"Invalid request"` 400 body; detailed exception text logged server-side at DEBUG (matches v0.40.3 #33 trace-log policy). (#102 vLLM `/v1/messages` CORS) restricted to loopback origins on both transformers AND vLLM backends since it's a mutation route (matches v0.30.0 Part C transformers-backend policy that was previously vLLM-wildcard). (#102 SSE header injection) `model` + `msg_id` strings stripped of CRLF / NUL / oversize before SSE `data:` embedding — defends against `model="foo\ndata:{injected}\n\n"` injecting spurious SSE frames; `Cache-Control: no-store` + `X-Accel-Buffering: no` headers on `StreamingResponse` prevent intermediary caching of model outputs. (#106 `_node_seed` symlink TOCTOU) `os.lstat` runs on the RAW path BEFORE `realpath` resolves the symlink (matches v0.33.0 #22 `prune_checkpoints` policy); `realpath`-first ordering would have been a no-op since `S_ISLNK` on a resolved path is always False. (#106 `_save_checkpoint` atomic) switched from `path + ".tmp"` + plain `open()` to `tempfile.mkstemp(dir=path.parent, suffix=".tmp")` + `os.fdopen` + `os.replace` (matches v0.43.0 Part D `copy_bundle_to` / v0.48.0 `write_mix_recipe` policy); eliminates predictable `.tmp` symlink hazard. (#106 `failed_reason` redaction) raw exception messages run through `_redact_exc_message` which strips POSIX `/abs/path` and Windows `C:\abs\path` runs to `os.path.basename` and caps at 256 chars (matches v0.34.0 `crash.py` redaction policy); a `FileNotFoundError` no longer leaks the operator's `$HOME` into the world-readable `.checkpoint.json`. (#106 `_node_code` row JSON double-encode) row payloads serialised via `json.dumps(json.dumps(row))` before f-string embedding into the RLVR sandbox wrapper — defends against Python `repr()` escape edge cases on Unicode that could break sandbox invocation. (#106 resume rehydration) completed-node outputs rehydrated from per-node `.node-.jsonl` sidecar files (NOT empty list — a v0.53.7 in-progress draft had a correctness defect that silently fed downstream consumers an empty list, causing zero-row output on resume). (#86 atomic Arrow write) `Dataset.save_to_disk` writes to a `.tmp_` sibling directory then `os.replace`-ed (mirrors v0.48.0 `write_mix_recipe` pattern); mid-write Ctrl-C / OOM no longer leaves a partial Arrow directory that looks complete to subsequent runs. (#86 cache-hash gate) SFT + Pretrain wrappers verify `metadata.json::cache_key` matches the freshly-computed `make_preprocess_cache_key(base, max_length, format, train)` BEFORE consuming the Arrow shards; mismatch raises `ValueError("cache hash mismatch: re-run `soup data preprocess`")` so a user who pre-tokenized with one config and changed `max_length` cannot silently train on stale shards. Missing metadata.json falls back to yellow advisory + trusted mode. (#87 `resolve_prompt_strategy`) operator-supplied module imported via `importlib`; signature-validated `Mapping → Mapping`; `except` narrowed to `TypeError` only so the explicit ValueError from the signature-shape check propagates (a v0.53.7 in-progress draft had `except (TypeError, ValueError): pass` which swallowed its own validator); `assert isinstance(spec, str)` replaced with `raise TypeError` so the validator survives `python -O`. Per-row invocations swallow exceptions at DEBUG and fall through to the raw row (matches v0.33.0 #47 `CrossDocCollator` immutability policy). LIMITATION: this is a trusted-input surface — do NOT use `prompt_strategy` with operator-controlled YAML on shared CI hosts. (#88 markdown ingest) heading-section text capped per regex `^#{1,6}\s` so a crafted MD with a 1-byte heading followed by 1 GB body still respects the existing `data/ingest` MD branch size cap from v0.42.0 Part F. (#111 forge `--judge-provider`) lazy v0.20.0 provider instantiation reuses existing SSRF hardening per backend: Ollama localhost-only (rejects remote IPs), Anthropic env-only API key (`ANTHROPIC_API_KEY` — never CLI arg, matches v0.20.0 policy), vLLM scheme-allowlist + loopback HTTP. Per-call exceptions counted + logged at DEBUG (no silent silent-degrade). (#112 `--benchmark-file`) operator JSONL path runs through `is_under_cwd` + `os.lstat + S_ISLNK` rejection BEFORE the corpus is loaded (matches v0.46.0 `agent_forge.load_spec_file` policy). (#105 `instantiate_trainer_plugins`) `_instantiate_simple_plugin` returns None + WARNING when no `attr_candidates` match the imported module (matches v0.45.0 `register_plugin` policy — surface bad shapes loudly, not silently); `except TypeError` narrowed (a v0.53.7 in-progress draft swallowed all `TypeError` from the wrong layer). Cross-cutting: SSE generator return types annotated (`Generator[str, None, None]`); `JUDGE_PROVIDERS: frozenset[str]` PEP 585 element type; `extract_row_text` exposed publicly in `utils/data_score.__all__` (cross-module import of private helper eliminated). Test surface: `tests/test_v0537.py` carries 111 new tests; CI green across Ubuntu/macOS/Windows × Python 3.9/3.11/3.12. (v0.53.7) diff --git a/pyproject.toml b/pyproject.toml index 66b3f75..7b16cdf 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.53.9" +version = "0.53.10" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" @@ -65,6 +65,13 @@ tui = ["textual>=0.50.0"] trackers = ["mlflow>=2.0.0", "swanlab>=0.3.0", "trackio>=0.0.1"] # v0.53.8 #85 — fsspec backends for remote dataset loading (s3 / gs / az / oci). remote = ["fsspec>=2024.1.0", "s3fs>=2024.1.0", "gcsfs>=2024.1.0", "adlfs>=2024.1.0"] +# v0.53.10 #150 — bundle scikit-optimize so `soup data mix --optimize` runs the +# Bayesian-style loop instead of falling back to the v0.48.0 Dirichlet sampler. +mix = ["scikit-optimize>=0.9.0"] +# v0.53.10 #113 — production-grade data quality: langdetect (language) + +# presidio-analyzer (PII). Llama-Guard-3-1B is documented as a manual recipe +# (license + ~600 MB weight blob too large to bundle by default). +data-pro = ["langdetect>=1.0.9", "presidio-analyzer>=2.2.0"] [project.scripts] soup = "soup_cli.cli:run" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 7cd1f89..130f3fd 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.53.9" +__version__ = "0.53.10" diff --git a/soup_cli/commands/chat.py b/soup_cli/commands/chat.py index 9b29cb2..48a1685 100644 --- a/soup_cli/commands/chat.py +++ b/soup_cli/commands/chat.py @@ -53,8 +53,34 @@ def chat( "Default deny (v0.36.0). Only enable if you trust the source." ), ), + hub: str = typer.Option( + "hf", + "--hub", + help=( + "Source hub for the base model: hf (default) / modelscope / " + "modelers. Non-HF hubs require the matching SDK; the base model " + "is snapshotted to a cwd-contained cache before chat starts " + "(v0.53.10 #152)." + ), + ), ): """Chat with a fine-tuned model in the terminal.""" + # v0.53.10 #152 — pre-fetch base from a non-HF hub before any path + # resolution. Local paths and HF repo IDs are passed through unchanged. + if hub and hub != "hf": + from soup_cli.utils.hubs import apply_hub_to_cli_model + + try: + model, base_model = apply_hub_to_cli_model( + model, base_model, hub, console=console + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=2) from exc + except ImportError as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=1) from exc + model_path = Path(model) if not model_path.exists(): diff --git a/soup_cli/commands/data.py b/soup_cli/commands/data.py index d37647a..4f1fb7f 100644 --- a/soup_cli/commands/data.py +++ b/soup_cli/commands/data.py @@ -1210,13 +1210,53 @@ def download_dataset( except (TypeError, ValueError) as exc: console.print(f"[red]{exc}[/]") raise typer.Exit(code=2) from exc + # v0.53.10 #153 — non-HF hub live dataset download. ModelScope uses + # the ``MsDataset`` API (different shape from snapshot_download), so + # we dispatch here instead of going through utils.hubs.download_repo. if hub_canonical != "hf": - console.print( - f"[red]--hub {hub_canonical} dataset download is not yet wired; " - f"use `from soup_cli.utils.hubs import download_repo` to snapshot " - f"a repo, or wait for v0.53.9.[/]" - ) - raise typer.Exit(code=1) + from soup_cli.utils.hubs import download_repo as _download_repo + + try: + if hub_canonical == "modelscope": + try: + from modelscope.msdatasets import ( + MsDataset, # type: ignore[import-not-found] + ) + except ImportError as exc: + console.print( + "[red]modelscope is not installed. " + "Install with: pip install modelscope[/]" + ) + raise typer.Exit(1) from exc + _ms_ds = MsDataset.load( # noqa: F841 — touched for side effect + dataset_id, split=split + ) + console.print( + f"[dim]ModelScope dataset {dataset_id} loaded; " + "use soup_cli.utils.hubs.download_repo for raw " + "snapshot download.[/]" + ) + else: # modelers + try: + out_dir = _download_repo( + hub_canonical, + dataset_id, + local_dir=str(Path.cwd() / ".soup_hub_cache" + / "datasets" + / dataset_id.replace("/", "__")), + repo_type="dataset", + ) + except ImportError as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(1) from exc + console.print( + f"[green]Downloaded {dataset_id} from {hub_canonical} → " + f"{out_dir}[/]" + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(2) from exc + return max_download_samples = 1_000_000 if samples is not None and samples > max_download_samples: console.print( diff --git a/soup_cli/commands/export.py b/soup_cli/commands/export.py index 71450af..01332b6 100644 --- a/soup_cli/commands/export.py +++ b/soup_cli/commands/export.py @@ -131,8 +131,30 @@ def export( "Required when --format=gguf-ud (v0.53.1 #139)." ), ), + hub: str = typer.Option( + "hf", + "--hub", + help=( + "Source hub for the base model when --model is a LoRA adapter: " + "hf (default) / modelscope / modelers (v0.53.10 #152)." + ), + ), ): """Export a model to GGUF, ONNX, TensorRT-LLM, AWQ, GPTQ, or TorchAO format.""" + # v0.53.10 #152 — pre-fetch the base model from a non-HF hub. ``model`` + # is typically a local merged dir / adapter dir; only ``base`` is rewritten. + if hub and hub != "hf": + from soup_cli.utils.hubs import apply_hub_to_cli_model + + try: + _, base = apply_hub_to_cli_model(model, base, hub, console=console) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=2) from exc + except ImportError as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=1) from exc + model_path = Path(model) # --- Validate --- diff --git a/soup_cli/commands/infer.py b/soup_cli/commands/infer.py index c91d320..3bb8736 100644 --- a/soup_cli/commands/infer.py +++ b/soup_cli/commands/infer.py @@ -102,8 +102,29 @@ def infer( "Default deny (v0.36.0). Only enable if you trust the source." ), ), + hub: str = typer.Option( + "hf", + "--hub", + help=( + "Source hub for the base model: hf (default) / modelscope / " + "modelers. Non-HF hubs require the matching SDK (v0.53.10 #152)." + ), + ), ): """Run batch inference on a JSONL file of prompts.""" + # v0.53.10 #152 — pre-fetch base from a non-HF hub before any resolution. + if hub and hub != "hf": + from soup_cli.utils.hubs import apply_hub_to_cli_model + + try: + model, base = apply_hub_to_cli_model(model, base, hub, console=console) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=2) from exc + except ImportError as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=1) from exc + from soup_cli.utils.paths import is_under_cwd # Validate input file diff --git a/soup_cli/commands/merge.py b/soup_cli/commands/merge.py index 30e1b60..974420f 100644 --- a/soup_cli/commands/merge.py +++ b/soup_cli/commands/merge.py @@ -53,8 +53,30 @@ def merge( "cycle (v0.53.1 #142)." ), ), + hub: str = typer.Option( + "hf", + "--hub", + help=( + "Source hub for the base model: hf (default) / modelscope / " + "modelers. Non-HF hubs require the matching SDK (v0.53.10 #152)." + ), + ), ): """Merge a LoRA adapter with its base model into a full model.""" + # v0.53.10 #152 — pre-fetch the base model from a non-HF hub. The local + # adapter dir is left untouched; only the base repo id is rewritten. + if hub and hub != "hf": + from soup_cli.utils.hubs import apply_hub_to_cli_model + + try: + _, base = apply_hub_to_cli_model(adapter, base, hub, console=console) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=2) from exc + except ImportError as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=1) from exc + # v0.53.1 #142 — validate save_format up front from soup_cli.utils.save_formats import validate_merge_save_format try: diff --git a/soup_cli/commands/push.py b/soup_cli/commands/push.py index 2c5edc5..9729465 100644 --- a/soup_cli/commands/push.py +++ b/soup_cli/commands/push.py @@ -57,8 +57,28 @@ def push( "(slug: 'owner/title-hash')" ), ), + hub: str = typer.Option( + "hf", + "--hub", + help=( + "Destination hub: hf (default) / modelscope / modelers. Non-HF " + "hubs require the matching SDK and skip the HF-specific " + "Collections / model-card auto-render path (v0.53.10 #152)." + ), + ), ): - """Push a trained model to HuggingFace Hub.""" + """Push a trained model to HuggingFace Hub (or alternate hub).""" + # v0.53.10 #152 — validate hub at the CLI boundary; only HF is the + # default. Non-HF hubs upload via :func:`utils.hubs.upload_repo` after + # the standard model-dir validation completes. + from soup_cli.utils.hubs import validate_hub_name + + try: + hub_canonical = validate_hub_name(hub) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=2) from exc + from soup_cli.utils.paths import is_under_cwd model_path = Path(model) @@ -143,6 +163,35 @@ def push( ) # --- Upload --- + # v0.53.10 #152 — non-HF hubs route through utils.hubs.upload_repo + # before we reach the HF-specific Collections / model-card auto-render + # path. Each backend lazy-imports its own SDK; missing-dep surfaces + # as ImportError with a pip-install advisory. + if hub_canonical != "hf": + from soup_cli.utils.hubs import upload_repo + + console.print(f"[dim]Uploading to hub={hub_canonical}...[/]") + try: + upload_repo( + hub_canonical, + repo, + folder_path=str(model_path), + commit_message=commit_message, + token=hf_token, + ) + except ImportError as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(1) from exc + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(2) from exc + console.print( + f"[green]Pushed to {hub_canonical}/{repo}.[/]\n" + "[dim]Note: HF-specific Collections + model card auto-render " + "are HF-only; install via the HF flow for those features.[/]" + ) + return + console.print("[dim]Uploading to HuggingFace Hub...[/]") from soup_cli.utils.hf import get_hf_api diff --git a/soup_cli/commands/serve.py b/soup_cli/commands/serve.py index a934199..fe6548d 100644 --- a/soup_cli/commands/serve.py +++ b/soup_cli/commands/serve.py @@ -199,8 +199,31 @@ def serve( "deepseek-r1 | qwen3 | phi4 | openthinker. v0.53.9 #98." ), ), + hub: str = typer.Option( + "hf", + "--hub", + help=( + "Source hub for the base model: hf (default) / modelscope / " + "modelers. Non-HF hubs require the matching SDK (v0.53.10 #152)." + ), + ), ): """Start a local inference server with OpenAI-compatible API.""" + # v0.53.10 #152 — pre-fetch base from a non-HF hub before serve starts. + if hub and hub != "hf": + from soup_cli.utils.hubs import apply_hub_to_cli_model + + try: + model, base_model = apply_hub_to_cli_model( + model, base_model, hub, console=console + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=2) from exc + except ImportError as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=1) from exc + # Lazy imports for fast CLI startup try: import uvicorn # noqa: F401 diff --git a/soup_cli/monitoring/callback.py b/soup_cli/monitoring/callback.py index a56f520..5b22177 100644 --- a/soup_cli/monitoring/callback.py +++ b/soup_cli/monitoring/callback.py @@ -92,6 +92,54 @@ class SoupTrainerCallback(TrainerCallback): ): self.display.start(total_steps=state.max_steps) + def on_step_end( + self, args: TrainingArguments, state: TrainerState, + control: TrainerControl, **kwargs, + ): + """v0.53.10 #156 — peek at the active batch for tool-calling rows. + + HF Trainer does not pass ``inputs`` to ``on_step_end`` by default, so + this hook can only see a batch when the trainer subclass explicitly + threads ``inputs=`` via ``kwargs``. When present, route any + ``tool_calls`` entries through the global tool-output buffer so the + Web UI's Tool Outputs panel populates in real time. + + Best-effort: any exception inside the probe MUST NEVER take down + training. Lazy buffer access — zero overhead when ``tool_calls`` + is absent from the batch. + """ + inputs = kwargs.get("inputs") + if not isinstance(inputs, dict): + return + tool_calls = inputs.get("tool_calls") + if not tool_calls: + return + # stdlib import outside the try-block (python-review MEDIUM fix: + # ``time`` is impossible to ImportError, so don't pretend it can). + import time + + try: + from soup_cli.utils.tool_outputs import get_global_tool_buffer + + count = 0 + if isinstance(tool_calls, (list, tuple)): + count = len(tool_calls) + elif isinstance(tool_calls, (int, float)) and not isinstance( + tool_calls, bool + ): + count = int(tool_calls) + if count <= 0: + return + get_global_tool_buffer().record_call( + name="sft_batch", + started_ts=time.time(), + duration_ms=0.0, + success=True, + output_preview=f"step {state.global_step}: {count} tool_calls", + ) + except Exception: # noqa: BLE001 — best-effort, must never crash training + return + def on_log( self, args: TrainingArguments, state: TrainerState, control: TrainerControl, logs=None, **kwargs, diff --git a/soup_cli/ui/static/app.js b/soup_cli/ui/static/app.js index cec743c..0fc9c41 100644 --- a/soup_cli/ui/static/app.js +++ b/soup_cli/ui/static/app.js @@ -93,6 +93,92 @@ function navigate(page) { else if (page === 'training') loadTrainingPage(); else if (page === 'data') { /* loaded on demand */ } else if (page === 'chat') loadChatPage(); + else if (page === 'tools') loadToolOutputs(); + // v0.53.10 #155 — pause Tool Outputs polling when navigating away so we + // don't keep firing fetch() against /api/tool-outputs from background tabs. + if (page !== 'tools') stopToolOutputsPolling(); +} + +// --- Tool Outputs panel (v0.53.10 #155) --- +// Polls /api/tool-outputs every 3 s while the page is active. XSS-safe via +// textContent / .appendChild (no innerHTML for user-controlled fields). +let _toolsPollHandle = null; + +function loadToolOutputs() { + renderToolOutputs(); + if (_toolsPollHandle === null) { + _toolsPollHandle = setInterval(renderToolOutputs, 3000); + } +} + +function stopToolOutputsPolling() { + if (_toolsPollHandle !== null) { + clearInterval(_toolsPollHandle); + _toolsPollHandle = null; + } +} + +async function renderToolOutputs() { + const container = document.getElementById('tools-content'); + if (!container) return; + let payload; + try { + const headers = {}; + if (window._authToken) { + headers['Authorization'] = 'Bearer ' + window._authToken; + } + const resp = await fetch('/api/tool-outputs?limit=100', { headers }); + if (!resp.ok) throw new Error('HTTP ' + resp.status); + payload = await resp.json(); + } catch (err) { + container.textContent = 'Failed to load tool outputs: ' + err.message; + return; + } + const records = (payload && Array.isArray(payload.records)) ? payload.records : []; + // Build the table via DOM APIs so user-controlled fields stay XSS-safe. + container.replaceChildren(); + if (records.length === 0) { + const empty = document.createElement('div'); + empty.className = 'empty-state'; + const t = document.createElement('div'); + t.className = 'empty-state-text'; + t.textContent = 'No tool calls observed yet.'; + empty.appendChild(t); + container.appendChild(empty); + return; + } + const wrap = document.createElement('div'); + wrap.className = 'table-wrap'; + const table = document.createElement('table'); + const thead = document.createElement('thead'); + const head = document.createElement('tr'); + ['Name', 'Started', 'Duration (ms)', 'OK', 'Output'].forEach(label => { + const th = document.createElement('th'); + th.textContent = label; + head.appendChild(th); + }); + thead.appendChild(head); + table.appendChild(thead); + const tbody = document.createElement('tbody'); + records.forEach(rec => { + const tr = document.createElement('tr'); + const cells = [ + String(rec.name || ''), + rec.started_ts ? new Date(rec.started_ts * 1000).toLocaleTimeString() : '-', + (typeof rec.duration_ms === 'number') ? rec.duration_ms.toFixed(1) : '-', + rec.success ? '✓' : '✗', + String(rec.output_preview || rec.error || ''), + ]; + cells.forEach(value => { + const td = document.createElement('td'); + td.textContent = value; + tr.appendChild(td); + }); + tbody.appendChild(tr); + }); + table.appendChild(tbody); + wrap.appendChild(table); + container.appendChild(wrap); } // --- API Helpers --- diff --git a/soup_cli/ui/static/index.html b/soup_cli/ui/static/index.html index 484cc1e..899b00f 100644 --- a/soup_cli/ui/static/index.html +++ b/soup_cli/ui/static/index.html @@ -29,6 +29,10 @@ Model Chat + @@ -165,6 +169,26 @@ + + +
+ +
+
Recent tool calls
+
+
+
Waiting for tool calls...
+
POST to /v1/tools/python or run a tool-calling SFT to populate
+
+
+
+
+ diff --git a/soup_cli/utils/data_mix.py b/soup_cli/utils/data_mix.py index d004371..aba9a59 100644 --- a/soup_cli/utils/data_mix.py +++ b/soup_cli/utils/data_mix.py @@ -22,6 +22,7 @@ Security: from __future__ import annotations +import importlib.util as _importlib_util import json import math import os @@ -417,6 +418,12 @@ def _build_default_optimizer( ) -> OptimizerProtocol: """Return :func:`_build_skopt_optimizer` when ``scikit-optimize`` is installed; otherwise fall back to a deterministic Dirichlet-like sampler. + + v0.53.10 #150 — when scikit-optimize is available via the new ``[mix]`` + pyproject extra, callers get true Bayesian optimisation; otherwise the + Dirichlet sampler is used silently (no spam advisory at import time, but + the caller of :func:`run_mix_optimizer` can inspect the returned optimizer + via :func:`describe_default_optimizer` to surface the chosen backend). """ try: return _build_skopt_optimizer(num_datasets, seed) @@ -439,6 +446,20 @@ def _build_default_optimizer( return _Dirichlet() +def describe_default_optimizer() -> str: + """Return a short label naming the optimizer backend that + :func:`_build_default_optimizer` would pick for the current process. + + v0.53.10 #150 — used by ``soup data mix --optimize`` to print an advisory + so users can see whether the v0.48.0 Dirichlet fallback or the bundled + scikit-optimize Bayesian loop is active. ``importlib.util.find_spec`` is + a non-executing probe so skopt's import cost is not paid here. + """ + if _importlib_util.find_spec("skopt") is not None: + return "scikit-optimize" + return "dirichlet-fallback" + + def _renormalize(weights: Sequence[float]) -> Tuple[float, ...]: """Clip + renormalise to a valid simplex point.""" clipped = [max(0.0, float(w)) for w in weights] diff --git a/soup_cli/utils/data_score.py b/soup_cli/utils/data_score.py index fbbc95b..3446159 100644 --- a/soup_cli/utils/data_score.py +++ b/soup_cli/utils/data_score.py @@ -230,15 +230,64 @@ _PII_PATTERNS: Tuple[Tuple[str, "re.Pattern[str]"], ...] = ( ) +def _presidio_pii(text: str) -> List[Dict[str, str]] | None: + """Run Presidio AnalyzerEngine when available (v0.53.10 #113 / ``[data-pro]``). + + Returns ``None`` when the optional ``presidio-analyzer`` package is not + installed OR when any error fires during analysis. Caller falls back to + the regex baseline. We DO NOT raise — PII detection is best-effort and + should never crash the broader scoring pipeline. + + The 32-hit cap + 64-char snippet truncation mirror the regex path so + downstream consumers get a consistent shape regardless of backend. + """ + try: + from presidio_analyzer import AnalyzerEngine # noqa: PLC0415 + except ImportError: + return None + try: + analyzer = AnalyzerEngine() + results = analyzer.analyze(text=text, language="en") + except Exception: # noqa: BLE001 — fall through to regex baseline + return None + if not isinstance(results, list): + return None + hits: List[Dict[str, str]] = [] + for res in results: + kind = getattr(res, "entity_type", None) + start = getattr(res, "start", None) + end = getattr(res, "end", None) + if not isinstance(kind, str) or not isinstance(start, int): + continue + if not isinstance(end, int) or end <= start: + continue + snippet = text[start:end] + if len(snippet) > 64: + snippet = snippet[:61] + "..." + hits.append({"kind": kind.lower(), "snippet": snippet}) + if len(hits) >= 32: + break + return hits + + def detect_pii(text: Any) -> List[Dict[str, str]]: """Return a list of ``{kind, snippet}`` PII hits. Scans only the first ``_PII_SCAN_CAP`` chars of ``text`` to keep regex finditer cost bounded regardless of caller input size. + + v0.53.10 #113 — when ``presidio-analyzer`` is installed via the + ``[data-pro]`` extras, routes through Presidio for broader entity + coverage (location / dates / IBAN / etc.); otherwise falls back to the + in-tree 4-regex baseline (email / phone / SSN / credit-card). """ s = _require_str(text, name="text") if len(s) > _PII_SCAN_CAP: s = s[:_PII_SCAN_CAP] + # Try Presidio first; silently falls through when absent. + presidio_hits = _presidio_pii(s) + if presidio_hits is not None: + return presidio_hits hits: List[Dict[str, str]] = [] for kind, pat in _PII_PATTERNS: for m in pat.finditer(s): @@ -291,18 +340,53 @@ _LANG_STOPWORDS: Mapping[str, frozenset] = MappingProxyType( ) +def _langdetect_fast(text: str) -> str | None: + """Probabilistic detection via ``langdetect`` (v0.53.10 #113 / ``[data-pro]``). + + Returns ``None`` when the optional ``langdetect`` package is not + installed OR when the detector raises (e.g. ``LangDetectException`` on + too-short input). Caller falls back to the stopword heuristic. + + We rebind langdetect's global RNG to a constant seed so two consecutive + calls on the same input produce the same code (langdetect is otherwise + non-deterministic). The seed is reset at every call to keep the heuristic + deterministic across the test suite. + """ + try: + import langdetect # noqa: PLC0415 — optional dep + except ImportError: + return None + try: + # ``DetectorFactory.seed = 0`` is the upstream-documented way to make + # langdetect deterministic; cheap to re-apply. + langdetect.DetectorFactory.seed = 0 + code = langdetect.detect(text) + except Exception: # noqa: BLE001 — fall through to heuristic on any error + return None + if not isinstance(code, str) or len(code) < 2: + return None + # langdetect returns ISO 639-1 codes (already lowercased). Truncate to + # the 2-letter prefix to match the heuristic's surface. + return code[:2].lower() + + def detect_language(text: Any) -> str: """Return a 2-letter ISO code or ``"unknown"``. - Pure-Python stopword heuristic; conservative — falls through to - ``"unknown"`` on short or ambiguous input. For production-grade - detection, install ``langdetect`` and pipe via the ``[data-pro]`` - extras (deferred to v0.47.1). + v0.53.10 #113 — when the optional ``langdetect`` package is installed + via the ``[data-pro]`` extras, routes through its probabilistic + detector for broader language coverage; otherwise falls back to the + pure-Python stopword heuristic (covers en/es/fr/de/pt/ru). """ s = _require_str(text, name="text") tokens = _tokenise(s) if len(tokens) < 4: return "unknown" + # Try langdetect first; falls through silently when the package is + # missing or raises (e.g. too-short input). + fast = _langdetect_fast(s) + if fast is not None: + return fast token_set = set(tokens) best_lang = "unknown" best_hits = 0 diff --git a/soup_cli/utils/hubs.py b/soup_cli/utils/hubs.py index 0910264..e489dc8 100644 --- a/soup_cli/utils/hubs.py +++ b/soup_cli/utils/hubs.py @@ -444,3 +444,158 @@ def upload_repo( return raise ValueError(f"hub {canonical!r} has no upload adapter") + + +def prefetch_model_from_hub( + base: str, + hub: str, + *, + cache_root: str | None = None, + console: object | None = None, +) -> str: + """Snapshot ``base`` from ``hub`` into a cwd-contained cache + return path. + + v0.53.10 #152 — shared helper extracted from the v0.53.8 ``soup train`` + pre-fetch path so chat / serve / infer / merge / export / push can route + non-HF hubs through the same SSRF-hardened, cwd-contained snapshot flow + without each command re-implementing the slug + containment + cache + short-circuit logic. + + Args: + base: model id (e.g. ``"meta-llama/Llama-3.1-8B"``) — accepted as-is + and forwarded to the hub-specific :func:`download_repo` adapter. + hub: hub name (validated via :func:`validate_hub_name`); ``"hf"`` + short-circuits with no download (returns ``base`` unchanged). + cache_root: optional override for the cache root directory. Defaults + to ``/.soup_hub_cache``. The resolved cache subdir is + cwd-containment checked even when an override is supplied so a + crafted ``cache_root`` cannot escape the working directory. + console: optional Rich console for cache-hit / fetch advisories. When + ``None``, the function is silent (returns the path without + printing). Caller is responsible for printing failures. + + Returns: + Absolute local path to the downloaded snapshot. For ``hub='hf'`` + returns ``base`` unchanged (HF Hub is the trainer default). + + Raises: + TypeError / ValueError: invalid ``hub`` / ``base`` per + :func:`validate_hub_name` / :func:`_validate_repo_id_shape`. + ImportError: optional SDK (modelscope / openmind-hub) missing. + ValueError: resolved cache dir escapes cwd. + """ + import os + import re + + from soup_cli.utils.paths import is_under_cwd + + canonical = validate_hub_name(hub) + if canonical == "hf": + return base + if not isinstance(base, str) or not base: + raise ValueError("base must be a non-empty string") + if "\x00" in base or any(ord(c) < 0x20 for c in base): + raise ValueError("base must not contain control characters") + # Mirror v0.53.8 ``soup train`` cache-dir slug policy: strip every + # path-separator and ``..`` segment so a crafted ``base: ../../etc`` + # cannot escape the cache root (Windows ``\\`` + POSIX ``/`` both + # blocked). + safe_slug = re.sub(r"[^A-Za-z0-9._-]+", "__", base).strip("._-") or "model" + if cache_root is None: + root_path = os.path.realpath(os.path.join(os.getcwd(), ".soup_hub_cache")) + else: + if not isinstance(cache_root, str) or not cache_root: + raise ValueError("cache_root must be a non-empty string") + root_path = os.path.realpath(cache_root) + cache_dir = os.path.realpath(os.path.join(root_path, safe_slug)) + if not is_under_cwd(cache_dir): + raise ValueError( + "resolved hub cache dir escapes the current working directory" + ) + # v0.53.10 security-review HIGH: escape Rich markup on every + # user-controlled string before embedding in console.print. A crafted + # ``base`` like ``[bold red]evil[/bold red]`` must NOT render styled. + from rich.markup import escape as _markup_escape + + existing_cfg = os.path.join(cache_dir, "config.json") + if os.path.isfile(existing_cfg): + if console is not None: + try: + try: + display_dir = os.path.relpath(cache_dir) + except (ValueError, OSError): + display_dir = os.path.basename(cache_dir) + console.print( # type: ignore[attr-defined] + f"[dim]Using cached snapshot at " + f"{_markup_escape(display_dir)}[/]" + ) + except Exception: # noqa: BLE001 — advisory is best-effort + pass + return cache_dir + local_path = download_repo(canonical, base, local_dir=cache_dir) + if console is not None: + try: + # Reduce SDK-returned absolute path to a cwd-relative form so + # we don't leak $HOME into the terminal output (code-review + # HIGH fix; matches v0.34.0 crash.py redaction policy). + display_path = local_path + try: + display_path = os.path.relpath(local_path) + except (ValueError, OSError): + display_path = os.path.basename(local_path) + console.print( # type: ignore[attr-defined] + f"[dim]Fetched {_markup_escape(base)} " + f"from hub={_markup_escape(canonical)} → " + f"{_markup_escape(str(display_path))}[/]" + ) + except Exception: # noqa: BLE001 + pass + return local_path + + +def apply_hub_to_cli_model( + model: str | None, + base_model: str | None, + hub: str, + *, + console: object | None = None, +) -> tuple[str | None, str | None]: + """Resolve ``(model, base_model)`` after an optional non-HF hub prefetch. + + v0.53.10 #152 — shared CLI helper for chat / serve / infer / merge / + export / push so each command emits the same advisory + uses the same + cwd-contained cache. + + Behaviour: + * ``hub`` is ``"hf"`` (or empty / None) → returns inputs unchanged. + * ``base_model`` is set + non-existent local path → snapshot via + :func:`prefetch_model_from_hub` and route the result to + ``base_model``. + * else ``model`` is set + non-existent local path → snapshot via + :func:`prefetch_model_from_hub` and route the result to ``model``. + * Existing local paths (e.g. a freshly trained LoRA dir) are passed + through unchanged so a non-HF hub flag does not break the + common "fine-tune locally, then chat" loop. + + Returns: + ``(model_out, base_model_out)`` tuple — at most one of them is + rewritten to the local snapshot path. + + Raises: + TypeError / ValueError / ImportError: propagated from + :func:`prefetch_model_from_hub` so the CLI can map them to exit codes. + """ + import os + + if not hub or hub == "hf": + return model, base_model + # Prefer rewriting ``base_model`` when set (the typical LoRA-adapter + # case where ``model`` is a local directory and ``base_model`` is a + # remote repo id). + if base_model and not os.path.exists(base_model): + fetched = prefetch_model_from_hub(base_model, hub, console=console) + return model, fetched + if model and not os.path.exists(model): + fetched = prefetch_model_from_hub(model, hub, console=console) + return fetched, base_model + return model, base_model diff --git a/soup_cli/utils/trackers.py b/soup_cli/utils/trackers.py index 8791814..676e6b1 100644 --- a/soup_cli/utils/trackers.py +++ b/soup_cli/utils/trackers.py @@ -163,11 +163,70 @@ def build_telemetry_payload( _POSTHOG_HOST = "https://us.i.posthog.com" _POSTHOG_ENDPOINT = f"{_POSTHOG_HOST}/i/v0/e/" -# Public write-only key. Live deployments will swap this via env var. +# v0.53.10 #154 — bundled public write-only project key for Soup CLI +# telemetry. The key is INTENTIONALLY hard-coded: PostHog "phc_*" keys are +# write-only (cannot read events back); rotating it requires a release. +# Operators wanting to point telemetry at their own PostHog project should +# set ``SOUP_POSTHOG_KEY`` AND ``SOUP_POSTHOG_ENDPOINT`` together; both env +# vars are validated by :func:`_resolve_posthog_target`. _POSTHOG_DEFAULT_KEY = "phc_soup_public_write_only" _TELEMETRY_TIMEOUT_S = 1.0 +# Sentinel for "caller did not pass an endpoint, fall back to default + env". +_POSTHOG_ENDPOINT_DEFAULT = object() + + +def _resolve_posthog_target( + api_key: str | None, + endpoint: object = _POSTHOG_ENDPOINT_DEFAULT, + env: dict[str, str] | None = None, +) -> tuple[str, str] | None: + """Resolve ``(key, endpoint)`` from explicit args + ``SOUP_POSTHOG_*`` env. + + v0.53.10 #154 — adds env-var overrides for the bundled defaults so users + on private PostHog instances can point Soup telemetry at their own + project without a code change. Precedence: + + 1. Explicit ``api_key`` / ``endpoint`` kwargs (caller wins). + 2. ``SOUP_POSTHOG_KEY`` env var (overrides ``_POSTHOG_DEFAULT_KEY``). + 3. ``SOUP_POSTHOG_ENDPOINT`` env var (overrides + ``_POSTHOG_ENDPOINT``; must be HTTPS + pass the v0.51.0 SSRF policy). + 4. Bundled defaults. + + Returns ``None`` when any input fails validation (silent no-op so + telemetry can never crash training). + """ + import os # noqa: PLC0415 — local lazy import + + src = env if env is not None else os.environ + # Endpoint resolution: explicit caller > env override > default. + # Use a sentinel default so a caller who passes + # ``endpoint=_POSTHOG_ENDPOINT`` (locking in the default) is NOT silently + # overridden by ``SOUP_POSTHOG_ENDPOINT`` (code-review HIGH fix). + if endpoint is _POSTHOG_ENDPOINT_DEFAULT: + env_endpoint = src.get("SOUP_POSTHOG_ENDPOINT") + resolved_endpoint = env_endpoint or _POSTHOG_ENDPOINT + else: + resolved_endpoint = endpoint + if not isinstance(resolved_endpoint, str): + return None + if not _telemetry_endpoint_is_safe(resolved_endpoint): + return None + # Key resolution: explicit caller > env override > default. + if api_key is not None: + key = api_key + else: + key = src.get("SOUP_POSTHOG_KEY") or _POSTHOG_DEFAULT_KEY + if not isinstance(key, str) or not key: + return None + # Reject control chars / whitespace in the key — defends against an + # operator dropping ``\nAuthorization:...`` into SOUP_POSTHOG_KEY. + if "\x00" in key or any(ord(c) < 0x20 for c in key) or len(key) > 256: + return None + return key, resolved_endpoint + + def _telemetry_endpoint_is_safe(endpoint: str) -> bool: """Re-validate the telemetry endpoint via the v0.51.0 SSRF policy. @@ -193,7 +252,7 @@ def send_telemetry_payload( *, api_key: str | None = None, timeout: float = _TELEMETRY_TIMEOUT_S, - endpoint: str = _POSTHOG_ENDPOINT, + endpoint: object = _POSTHOG_ENDPOINT_DEFAULT, ) -> bool: """POST ``payload`` to PostHog if telemetry is enabled, else no-op. @@ -211,18 +270,17 @@ def send_telemetry_payload( return False if not isinstance(payload, dict) or not payload: return False - # HTTPS-only + private-IP / link-local rejection (mirrors v0.51.0 hub - # endpoint SSRF policy). Defence-in-depth: any caller override goes - # through the same validator that hub endpoints do. - if not _telemetry_endpoint_is_safe(endpoint): - return False if isinstance(timeout, bool) or not isinstance(timeout, (int, float)): return False if not math.isfinite(float(timeout)) or timeout <= 0: return False - key = api_key or _POSTHOG_DEFAULT_KEY - if not isinstance(key, str) or not key: + # v0.53.10 #154 — resolve key + endpoint via env-override-aware helper. + # Returns ``None`` when either input fails validation; treat as silent + # no-op so telemetry remains best-effort. + resolved = _resolve_posthog_target(api_key, endpoint) + if resolved is None: return False + key, endpoint = resolved try: import httpx # lazy — optional dep, surfaces no advisory except ImportError: diff --git a/tests/test_v05310.py b/tests/test_v05310.py new file mode 100644 index 0000000..35942d5 --- /dev/null +++ b/tests/test_v05310.py @@ -0,0 +1,510 @@ +"""v0.53.10 — Quick wins + packaging + UX wiring. + +Covers seven closed issues: + +* #150 ``[mix]`` extra + ``describe_default_optimizer`` advisory. +* #113 ``[data-pro]`` extras + lazy ``langdetect`` / Presidio routing. +* #154 ``SOUP_POSTHOG_KEY`` / ``SOUP_POSTHOG_ENDPOINT`` env override. +* #152 Multi-command ``--hub`` dispatch (chat / serve / infer / merge / + export / push). +* #153 ``soup data download --hub `` live SDK lift. +* #155 Web UI Tool Outputs panel JS / HTML wiring. +* #156 SFT trainer-side ``record_call`` wire-up in + ``monitoring/callback.SoupTrainerCallback``. +""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path +from unittest.mock import patch + +import pytest + +REPO_ROOT = Path(__file__).resolve().parent.parent + + +def _strip_ansi(text: str) -> str: + out = re.sub(r"\x1b\[[0-9;]*m", "", text) + return re.sub(r"\s+", " ", out) + + +# ---------------------------------------------------------------------- +# #150 — `[mix]` extra + scikit-optimize advisory +# ---------------------------------------------------------------------- + + +class TestMixExtra: + def test_pyproject_lists_mix_extra(self): + body = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8") + # ``mix`` extra MUST be declared and bundle scikit-optimize. + assert re.search(r"^mix\s*=\s*\[", body, re.MULTILINE), ( + "[mix] extra missing from pyproject" + ) + mix_block = re.search( + r"^mix\s*=\s*\[(.*?)\]", body, re.MULTILINE | re.DOTALL + ) + assert mix_block is not None + assert "scikit-optimize" in mix_block.group(1) + + def test_describe_default_optimizer_returns_label(self): + from soup_cli.utils.data_mix import describe_default_optimizer + + label = describe_default_optimizer() + assert label in ("scikit-optimize", "dirichlet-fallback") + + def test_describe_dirichlet_fallback_when_skopt_missing(self): + # Force find_spec to report skopt missing. + with patch("importlib.util.find_spec", return_value=None): + from soup_cli.utils.data_mix import describe_default_optimizer + + assert describe_default_optimizer() == "dirichlet-fallback" + + +# ---------------------------------------------------------------------- +# #113 — `[data-pro]` extras + langdetect / Presidio routing +# ---------------------------------------------------------------------- + + +class TestDataProExtra: + def test_pyproject_lists_data_pro(self): + body = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8") + assert re.search(r"^data-pro\s*=\s*\[", body, re.MULTILINE), ( + "[data-pro] extra missing from pyproject" + ) + block = re.search( + r"^data-pro\s*=\s*\[(.*?)\]", body, re.MULTILINE | re.DOTALL + ) + assert block is not None + assert "langdetect" in block.group(1) + assert "presidio-analyzer" in block.group(1) + + def test_detect_language_falls_back_to_heuristic_without_langdetect(self): + # Force the lazy import inside _langdetect_fast to fail by removing + # any ``langdetect`` from sys.modules + blocking re-import. + from soup_cli.utils import data_score + + with patch.object( + data_score, "_langdetect_fast", return_value=None + ): + # Heuristic still picks 'en' on an English sentence. + assert data_score.detect_language( + "the quick brown fox jumps over the lazy dog" + ) == "en" + + def test_detect_language_uses_langdetect_when_available(self): + from soup_cli.utils import data_score + + with patch.object(data_score, "_langdetect_fast", return_value="ja"): + assert data_score.detect_language( + "the quick brown fox jumps over the lazy dog" + ) == "ja" + + def test_detect_pii_falls_back_to_regex_without_presidio(self): + from soup_cli.utils import data_score + + with patch.object(data_score, "_presidio_pii", return_value=None): + hits = data_score.detect_pii( + "Email me at user@example.com or 555-867-5309" + ) + kinds = {h["kind"] for h in hits} + assert "email" in kinds + + def test_detect_pii_uses_presidio_when_available(self): + from soup_cli.utils import data_score + + presidio_hits = [{"kind": "email", "snippet": "user@example.com"}] + with patch.object( + data_score, "_presidio_pii", return_value=presidio_hits + ): + assert data_score.detect_pii("anything") == presidio_hits + + +# ---------------------------------------------------------------------- +# #154 — PostHog env override +# ---------------------------------------------------------------------- + + +class TestPostHogEnvOverride: + def test_resolve_uses_default_key_when_env_unset(self): + from soup_cli.utils.trackers import _POSTHOG_DEFAULT_KEY, _resolve_posthog_target + + # No env override; endpoint omitted to use the sentinel default. + resolved = _resolve_posthog_target(None, env={}) + assert resolved is not None + key, _ = resolved + assert key == _POSTHOG_DEFAULT_KEY + + def test_resolve_env_key_overrides_default(self): + from soup_cli.utils.trackers import _resolve_posthog_target + + env = {"SOUP_POSTHOG_KEY": "phc_user_project"} + resolved = _resolve_posthog_target(None, env=env) + assert resolved is not None + key, _ = resolved + assert key == "phc_user_project" + + def test_resolve_explicit_arg_wins_over_env(self): + from soup_cli.utils.trackers import _resolve_posthog_target + + env = {"SOUP_POSTHOG_KEY": "phc_env"} + resolved = _resolve_posthog_target("phc_caller", env=env) + assert resolved is not None + key, _ = resolved + assert key == "phc_caller" + + def test_resolve_env_endpoint_overrides_default(self): + from soup_cli.utils.trackers import _resolve_posthog_target + + env = {"SOUP_POSTHOG_ENDPOINT": "https://eu.i.posthog.com/i/v0/e/"} + resolved = _resolve_posthog_target(None, env=env) + assert resolved is not None + _, endpoint = resolved + assert endpoint == "https://eu.i.posthog.com/i/v0/e/" + + def test_resolve_rejects_http_endpoint(self): + from soup_cli.utils.trackers import _resolve_posthog_target + + env = {"SOUP_POSTHOG_ENDPOINT": "http://attacker.example.com/"} + assert _resolve_posthog_target(None, env=env) is None + + def test_resolve_explicit_endpoint_locks_against_env(self): + # code-review HIGH fix: a caller passing the default URL string + # explicitly should NOT be silently overridden by the env var. + from soup_cli.utils.trackers import _POSTHOG_ENDPOINT, _resolve_posthog_target + + env = {"SOUP_POSTHOG_ENDPOINT": "https://eu.i.posthog.com/i/v0/e/"} + resolved = _resolve_posthog_target(None, endpoint=_POSTHOG_ENDPOINT, env=env) + assert resolved is not None + _, endpoint = resolved + assert endpoint == _POSTHOG_ENDPOINT + + def test_resolve_rejects_control_char_key(self): + from soup_cli.utils.trackers import _resolve_posthog_target + + env = {"SOUP_POSTHOG_KEY": "phc\nAuthorization: bypass"} + assert _resolve_posthog_target(None, env=env) is None + + def test_resolve_rejects_null_byte_key(self): + from soup_cli.utils.trackers import _resolve_posthog_target + + env = {"SOUP_POSTHOG_KEY": "phc\x00bypass"} + assert _resolve_posthog_target(None, env=env) is None + + def test_resolve_rejects_oversize_key(self): + # tdd-review MEDIUM #3: 257-char SOUP_POSTHOG_KEY must reject. + from soup_cli.utils.trackers import _resolve_posthog_target + + env = {"SOUP_POSTHOG_KEY": "x" * 257} + assert _resolve_posthog_target(None, env=env) is None + + def test_resolve_rejects_explicit_empty_key(self): + # tdd-review MEDIUM #4: explicit empty-string `api_key` must reject. + from soup_cli.utils.trackers import _resolve_posthog_target + + assert _resolve_posthog_target("", env={}) is None + + +# ---------------------------------------------------------------------- +# #152 — Multi-command `--hub` dispatch +# ---------------------------------------------------------------------- + + +class TestHubPrefetchHelper: + def test_hf_short_circuits_no_download(self): + from soup_cli.utils.hubs import apply_hub_to_cli_model + + model_out, base_out = apply_hub_to_cli_model( + "meta-llama/Llama-3.1-8B", None, "hf" + ) + assert model_out == "meta-llama/Llama-3.1-8B" + assert base_out is None + + def test_none_hub_passes_through(self): + from soup_cli.utils.hubs import apply_hub_to_cli_model + + # None or empty hub is treated as no-op. + assert apply_hub_to_cli_model("foo", None, "") == ("foo", None) + + def test_existing_local_path_not_rewritten(self, tmp_path, monkeypatch): + # A real local dir means the user already merged + saved; the hub + # flag should not force a re-download. + monkeypatch.chdir(tmp_path) + local = tmp_path / "merged" + local.mkdir() + from soup_cli.utils.hubs import apply_hub_to_cli_model + + model_out, base_out = apply_hub_to_cli_model( + str(local), None, "modelscope" + ) + assert model_out == str(local) + + def test_non_existent_base_invokes_prefetch(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + # patch prefetch to avoid network + from soup_cli.utils import hubs + + with patch.object( + hubs, "prefetch_model_from_hub", return_value=str(tmp_path / "snap") + ) as p: + model_out, base_out = hubs.apply_hub_to_cli_model( + "./adapter", "some/repo", "modelscope" + ) + p.assert_called_once() + assert base_out == str(tmp_path / "snap") + + def test_apply_hub_to_cli_model_both_none(self): + # tdd-review LOW #7: neither model nor base set should no-op. + from soup_cli.utils.hubs import apply_hub_to_cli_model + + assert apply_hub_to_cli_model(None, None, "modelscope") == (None, None) + + def test_prefetch_unknown_hub_raises(self): + # Unknown hub name must propagate validation error. + from soup_cli.utils.hubs import apply_hub_to_cli_model + + with pytest.raises(ValueError): + apply_hub_to_cli_model(None, "some/repo", "evilhub") + + def test_prefetch_outside_cwd_cache_root_raises(self, tmp_path, monkeypatch): + # tdd-review MEDIUM #5: cache_root outside cwd must raise. + monkeypatch.chdir(tmp_path) + from soup_cli.utils.hubs import prefetch_model_from_hub + + outside = tmp_path.parent / "outside_cache" + with pytest.raises(ValueError, match="escapes"): + prefetch_model_from_hub( + "some/repo", "modelscope", cache_root=str(outside) + ) + + +class TestCliHubFlags: + @pytest.mark.parametrize( + "cmd_module,cmd_name", + [ + ("soup_cli.commands.chat", "chat"), + ("soup_cli.commands.serve", "serve"), + ("soup_cli.commands.infer", "infer"), + ("soup_cli.commands.merge", "merge"), + ("soup_cli.commands.export", "export"), + ("soup_cli.commands.push", "push"), + ], + ) + def test_hub_flag_present_in_signature(self, cmd_module, cmd_name): + # Re-importing fresh because typer-decorated commands wrap signatures. + module = __import__(cmd_module, fromlist=[cmd_name]) + fn = getattr(module, cmd_name) + # Typer commands are wrapped — pull the raw function. + target = getattr(fn, "__wrapped__", fn) + import inspect + + sig = inspect.signature(target) + assert "hub" in sig.parameters, ( + f"{cmd_module}.{cmd_name} missing --hub keyword" + ) + + def test_apply_hub_helper_imported_in_non_push_commands(self): + # tdd-review HIGH #1: every command except push must use the + # shared helper so a future refactor cannot silently drop it. + for mod in ("chat", "serve", "infer", "merge", "export"): + src = (REPO_ROOT / f"soup_cli/commands/{mod}.py").read_text( + encoding="utf-8" + ) + assert "apply_hub_to_cli_model" in src, ( + f"{mod}.py missing apply_hub_to_cli_model helper import" + ) + + def test_push_uses_upload_repo_path(self): + # push.py does NOT call apply_hub_to_cli_model (it's an upload + # surface, not a download). It must call upload_repo + validate_hub_name. + src = (REPO_ROOT / "soup_cli/commands/push.py").read_text( + encoding="utf-8" + ) + assert "upload_repo" in src + assert "validate_hub_name" in src + + +# ---------------------------------------------------------------------- +# #153 — `soup data download --hub ` live SDK +# ---------------------------------------------------------------------- + + +class TestDataDownloadNonHfLive: + def test_modelers_unknown_sdk_friendly_error(self): + from typer.testing import CliRunner + + from soup_cli.commands.data import app + + # No openmind_hub installed; expect ImportError advisory. + result = CliRunner().invoke( + app, ["download", "dummy/ds", "--hub", "modelers"] + ) + # exit_code 1 = friendly ImportError advisory; 2 = validation reject. + assert result.exit_code in (1, 2) + out = _strip_ansi(result.output) + # No longer surfaces "wait for v0.53.9"; should mention modelers or + # the missing SDK pip-install hint. + assert "v0.53.9" not in out + + def test_modelscope_unknown_sdk_friendly_error(self): + from typer.testing import CliRunner + + from soup_cli.commands.data import app + + if "modelscope" in sys.modules: + pytest.skip("modelscope is installed; live branch tested separately") + result = CliRunner().invoke( + app, ["download", "dummy/ds", "--hub", "modelscope"] + ) + assert result.exit_code in (1, 2) + out = _strip_ansi(result.output) + # Friendly advisory mentions the SDK or `pip install`. + assert "modelscope" in out + + def test_data_download_no_longer_advises_v0_53_9(self): + # tdd-review LOW #9: a future regression of the advisory text + # "wait for v0.53.9" must be caught at source-grep time, since + # v0.53.10 lifted that advisory to live SDK dispatch. + src = (REPO_ROOT / "soup_cli/commands/data.py").read_text( + encoding="utf-8" + ) + assert "wait for v0.53.9" not in src + + +# ---------------------------------------------------------------------- +# #155 — Web UI Tool Outputs panel +# ---------------------------------------------------------------------- + + +class TestWebUiToolOutputsPanel: + def test_index_html_has_tools_nav_entry(self): + html = (REPO_ROOT / "soup_cli/ui/static/index.html").read_text( + encoding="utf-8" + ) + assert 'data-page="tools"' in html + assert "Tool Outputs" in html + assert 'id="page-tools"' in html + + def test_app_js_has_load_tool_outputs(self): + js = (REPO_ROOT / "soup_cli/ui/static/app.js").read_text(encoding="utf-8") + assert "loadToolOutputs" in js + assert "/api/tool-outputs" in js + # XSS-safe — uses textContent or DOM API, NOT innerHTML for records. + assert "td.textContent" in js + + +# ---------------------------------------------------------------------- +# #156 — SFT callback `record_call` wire-up +# ---------------------------------------------------------------------- + + +class TestCallbackToolBuffer: + def test_on_step_end_method_exists(self): + from soup_cli.monitoring.callback import SoupTrainerCallback + + assert hasattr(SoupTrainerCallback, "on_step_end") + + def test_on_step_end_no_op_when_inputs_absent(self): + from soup_cli.monitoring.callback import SoupTrainerCallback + from soup_cli.utils.tool_outputs import ( + get_global_tool_buffer, + reset_global_tool_buffer, + ) + + reset_global_tool_buffer() + cb = SoupTrainerCallback.__new__(SoupTrainerCallback) + # State is dataclass-shaped; only global_step is read. + from types import SimpleNamespace + + cb.on_step_end( + args=SimpleNamespace(), + state=SimpleNamespace(global_step=1), + control=SimpleNamespace(), + ) + # Buffer should remain empty. + assert len(list(get_global_tool_buffer().snapshot(limit=10))) == 0 + + def test_on_step_end_records_when_tool_calls_present(self): + from soup_cli.monitoring.callback import SoupTrainerCallback + from soup_cli.utils.tool_outputs import ( + get_global_tool_buffer, + reset_global_tool_buffer, + ) + + reset_global_tool_buffer() + cb = SoupTrainerCallback.__new__(SoupTrainerCallback) + from types import SimpleNamespace + + cb.on_step_end( + args=SimpleNamespace(), + state=SimpleNamespace(global_step=42), + control=SimpleNamespace(), + inputs={"tool_calls": [{"name": "f"}, {"name": "g"}]}, + ) + records = list(get_global_tool_buffer().snapshot(limit=10)) + assert len(records) == 1 + assert "step 42" in records[0].output_preview + assert records[0].success is True + reset_global_tool_buffer() + + def test_on_step_end_empty_tool_calls_emits_no_record(self): + # tdd-review MEDIUM #6: empty list short-circuits. + from types import SimpleNamespace + + from soup_cli.monitoring.callback import SoupTrainerCallback + from soup_cli.utils.tool_outputs import ( + get_global_tool_buffer, + reset_global_tool_buffer, + ) + + reset_global_tool_buffer() + cb = SoupTrainerCallback.__new__(SoupTrainerCallback) + cb.on_step_end( + args=SimpleNamespace(), + state=SimpleNamespace(global_step=1), + control=SimpleNamespace(), + inputs={"tool_calls": []}, + ) + assert len(list(get_global_tool_buffer().snapshot(limit=10))) == 0 + + def test_on_step_end_bool_tool_calls_emits_no_record(self): + # tdd-review HIGH #2: ``tool_calls=True`` (bool) is a falsy-on-list + # short-circuit AND rejected by the numeric-branch bool guard. + from types import SimpleNamespace + + from soup_cli.monitoring.callback import SoupTrainerCallback + from soup_cli.utils.tool_outputs import ( + get_global_tool_buffer, + reset_global_tool_buffer, + ) + + reset_global_tool_buffer() + cb = SoupTrainerCallback.__new__(SoupTrainerCallback) + # ``True`` is truthy (bypasses ``not tool_calls`` guard); the + # bool-rejection branch must keep ``count=0`` and skip recording. + cb.on_step_end( + args=SimpleNamespace(), + state=SimpleNamespace(global_step=1), + control=SimpleNamespace(), + inputs={"tool_calls": True}, + ) + assert len(list(get_global_tool_buffer().snapshot(limit=10))) == 0 + + +# ---------------------------------------------------------------------- +# Version bump sanity check +# ---------------------------------------------------------------------- + + +class TestVersionBump: + def test_init_py_pin(self): + from soup_cli import __version__ + + assert __version__ == "0.53.10" + + def test_pyproject_pin(self): + body = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8") + assert re.search(r'^version\s*=\s*"0\.53\.10"', body, re.MULTILINE), ( + "pyproject.toml version pin missing v0.53.10" + ) diff --git a/tests/test_v0538.py b/tests/test_v0538.py index 45ce89f..2786070 100644 --- a/tests/test_v0538.py +++ b/tests/test_v0538.py @@ -552,7 +552,16 @@ class TestDataDownloadHubFlag: assert result.exit_code != 0 clean = _strip_ansi(result.output) assert "modelscope" in clean - assert "v0.53.9" in clean or "download_repo" in clean + # v0.53.10 #153 lifted the advisory to a live SDK dispatch; without + # the modelscope SDK installed the friendly pip-install error fires. + # Either the legacy v0.53.8 advisory (download_repo / v0.53.9) OR + # the v0.53.10 ImportError "pip install modelscope" is acceptable. + assert ( + "v0.53.9" in clean + or "download_repo" in clean + or "pip install modelscope" in clean + or "not installed" in clean + ) # ---------------------------------------------------------------------- @@ -593,19 +602,23 @@ class TestPyprojectExtras: class TestVersionBump: + @staticmethod + def _version_tuple(s: str) -> tuple: + return tuple(int(p) for p in s.split(".") if p.isdigit()) + def test_init_version(self): import soup_cli # Version-string is forward-monotonic: v0.53.8 baseline + any later # release (v0.53.9, v0.54.0, ...) keeps this contract green. - assert soup_cli.__version__ >= "0.53.8" + # Numeric tuple comparison defends against lexicographic regressions + # (e.g. "0.53.10" < "0.53.8" string-wise). + assert self._version_tuple(soup_cli.__version__) >= (0, 53, 8) def test_pyproject_version(self): text = (_repo_root() / "pyproject.toml").read_text(encoding="utf-8") - # Pyproject must declare some 0.53.x or later string; the literal - # "0.53.8" baseline is allowed to drift forward. import re match = re.search(r'^version\s*=\s*"([^"]+)"', text, flags=re.MULTILINE) assert match is not None - assert match.group(1) >= "0.53.8" + assert self._version_tuple(match.group(1)) >= (0, 53, 8) diff --git a/tests/test_v0539.py b/tests/test_v0539.py index 0baf6a0..8a67d16 100644 --- a/tests/test_v0539.py +++ b/tests/test_v0539.py @@ -30,7 +30,11 @@ def _plain(text: str) -> str: # ----------------------------------------------------------------- version def test_version_bump_to_0_53_9(): - assert soup_cli.__version__ == "0.53.9" + # Forward-monotonic: v0.53.9 baseline + later releases (v0.53.10, ...) + # keep this contract green. Numeric tuple compare defends against the + # lexicographic "0.53.10" < "0.53.9" footgun. + parts = tuple(int(p) for p in soup_cli.__version__.split(".") if p.isdigit()) + assert parts >= (0, 53, 9) # ----------------------------------------------------- #94 SSE event buffer