7 issues closed:
- #150 [mix] pyproject extra bundles scikit-optimize so `soup data mix
--optimize` runs the Bayesian loop instead of the v0.48.0 Dirichlet
fallback; new describe_default_optimizer() helper labels the active
backend without paying skopt's import cost.
- #113 [data-pro] extras (langdetect + presidio-analyzer) with lazy
fall-through helpers in utils/data_score (broader language coverage +
Presidio entity recognition on top of the v0.47.0 regex baseline).
Llama-Guard-3-1B documented as a manual recipe (license + size).
- #154 SOUP_POSTHOG_KEY / SOUP_POSTHOG_ENDPOINT env override via
sentinel-based explicit-vs-env precedence; HTTPS-only +
RFC1918/link-local rejection on the endpoint; null-byte / control-char
/ >256-char rejection on the key.
- #152 --hub flag plumbed on chat / serve / infer / merge / export /
push via shared utils/hubs.apply_hub_to_cli_model +
prefetch_model_from_hub helpers; push uses upload_repo (skips
HF-specific Collections + model-card auto-render on non-HF hubs).
- #153 `soup data download --hub modelscope|modelers` live SDK
(lifts the v0.53.8 advisory-only path); friendly ImportError
advisory when the SDK is missing.
- #155 Web UI Tool Outputs panel — `loadToolOutputs` polls
/api/tool-outputs every 3s; XSS-safe DOM-built table (textContent
per cell, no innerHTML for user-controlled fields); Bearer token
threaded via the v0.53.9 window._authToken bootstrap.
- #156 SoupTrainerCallback.on_step_end records tool_calls counts
from kwargs['inputs'] into the global tool buffer. Best-effort
(# noqa: BLE001 per project policy — training must never crash).
13 review-fixes applied (4 HIGH / 5 MEDIUM / 4 LOW):
- HIGH PostHog explicit-endpoint precedence sentinel
- HIGH absolute path leak in local_path advisory reduced to relpath
- HIGH Rich markup escape on base / local_path / cache_dir
- HIGH callback # noqa: BLE001 per project policy
- MED `import time` moved out of try block
- MED oversize key + explicit-empty key rejection tests
- MED source-grep regression guards (advisory-removal, helper imports
across 5 non-push commands)
- MED `prefetch_model_from_hub` outside-cwd cache_root rejection
- LOW empty-list + bool-True tool_calls no-op tests
- LOW push.py uses upload_repo + validate_hub_name regression guard
Test count: 8285 -> 8330 (+45 in tests/test_v05310.py).
Full suite green; ruff clean; on Win+Py3.10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's Rich pipeline emits styled output that splits `--vocab-size` into
multiple ANSI-bracketed spans (e.g. `\x1b[36m-\x1b[0m\x1b[36m-vocab\x1b[0m\x1b[36m-size\x1b[0m`),
breaking naive `"--vocab-size" in result.output` checks. Locally Rich
auto-detects non-TTY and skips the codes, so the regression only shows
on CI (ubuntu/macos/windows × 3.9/3.11/3.12).
Fix: small `_plain()` helper using `re.sub(r"\x1b\[[0-9;]*m", "", ...)`
applied to the 7 failing assertions. Same approach already used in
several other v0.5x test modules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eight features that close out the v0.44.x live-monitoring deferrals
plus a long tail of standalone CLI wins:
- #94 /api/train/stream async SSE with per-subscriber cursor + JS
EventSource consumer; SoupTrainerCallback pushes TrainEvent on
each on_log.
- #95 soup ui --public derives LAN IP via SOCK_DGRAM connect-trick,
prints scannable QR; --auth-token override; SPA bootstrap
hydrates window._authToken from ?token= + sessionStorage and
cleans the URL via history.replaceState; CORS regex auto-widens
to loopback + RFC1918 in public mode; set_auth_token rotation
race fixed via threading.Lock.
- #98 soup serve --reasoning-parser strips <think>...</think> (and
OpenThinker tags); pre-compiled regex with marker-token
fast-path + 1 MiB cap + leading-newline-only strip.
- #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON
endpoint; best-effort observation hook in callback.on_log.
- #15 soup tokenizer train: BPE training CLI with raw-path lstat
symlink rejection, 50 MiB total / 8 KiB per-line caps,
post-mkdir output-dir re-check, --special-token NUL/oversize
dedup, vocab bounds [256, 200000].
- #26 soup bench --p50 --p95 renders extra per-prompt tail-latency
Rich table; --prompts-file gains symlink rejection.
- #28 soup bench --backend auto: MLX weights.npz probe (per-entry
lstat) -> config.json model_type keyword -> transformers
fallback; SOUP_BENCH_BACKEND env hint.
- #12 examples/synthetic_workflow.{md,yaml} end-to-end walkthrough.
Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the
python / code / security / tdd review agents. Notable HIGH:
- QR token now consumed by SPA (was unreachable previously).
- set_auth_token rotation lock-protected, 8-thread stress tested.
- Tokenizer input + output symlink TOCTOU defence on raw path.
- SSE generator switched to async (asyncio.sleep) for non-blocking
multi-subscriber operation.
- CORS regex for --public LAN mode (the old fixed allowlist of
http://0.0.0.0:port never matched a real Origin header).
- _has_mlx_weights per-entry lstat so a symlinked weights.npz can't
trigger MLX dispatch.
Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the
relaxed v0.53.8 version-pin asserts in tests/test_v0538.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpamys