diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f198a25..2659bea 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (190 files, 8162 tests) +tests/ - Test suite (188 files, 8257 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index 32af805..e44c12f 100644 --- a/README.md +++ b/README.md @@ -42,19 +42,15 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.53.7 — Data Forge + Pipeline live (wave 1)**: Eleven features wired end-to-end across synthetic data generation, preprocessing, and inference. +**v0.53.8 — Remote data + Hubs + Trackers (wave 2)**: Six features wired live across cloud storage, alternative model hubs, experiment trackers, and HF Spaces. -- **Data Forge live judge providers.** `soup data forge --judge-provider {ollama,anthropic,vllm}` now routes judge calls through real backends (not just offline stubs). Ollama enforces localhost-only SSRF; Anthropic uses env-only API key; vLLM validates scheme allowlist. Per-call exceptions logged at DEBUG so one bad provider never crashes the forge pipeline. -- **AOT tokenization with `soup data preprocess`.** New command tokenizes your dataset once and caches Arrow shards keyed by `(dataset, tokenizer, max_length, format)`. SFT and Pretrain trainers short-circuit at the schema level when `format='pre_tokenized'` + `tokenized_path` is set, eliminating the per-epoch tokenization tax. Resume-safe via atomic writes. -- **Data Recipe DAGs live (`soup data recipe --execute`).** Six node kinds now execute end-to-end: seed dataset, LLM text generation, code execution (RLVR sandbox), judge binary scoring, regex/JSON validation, and deterministic sampling. Checkpoint written per node; resume rehydrates from per-node sidecars. `failed_reason` fields redacted to prevent path leakage. -- **Anthropic-style `/v1/messages` + streaming SSE on both backends.** The transformers-only route from v0.53.6 now ships on vLLM too. Both backends convert Anthropic ↔ OpenAI messages, serve non-stream requests with Anthropic envelope, and stream with Anthropic event-shape SSE (`message_start` → `content_block_delta` → `message_delta` + `message_stop`). Loopback-only CORS on both. -- **Server-side `/v1/tools/python` + `/v1/tools/web_search` HTTP endpoints.** Python sandbox wraps the v0.25.0 RLVR `code_exec` with Bearer auth gate (5s timeout, 64KB code cap). Web search backend is httpx with hard 5s timeout, 5-result cap, and domain allowlist (deny-by-default via `WebSearchConfig`). (`/v1/tools/bash` reverted to 501 — child-process isolation insufficient; tracked for v0.53.8 with container/namespace work.) -- **Live trainer-plugin instantiation.** `instantiate_trainer_plugins` now lazy-imports upstream plugins (grokfast, spectrum, llmcompressor, sonicmoe, cce_plugin, math_verify) and friendlily advises on missing packages. -- **Markdown heading-aware ingest + operator-supplied decontamination corpus + custom prompt strategies.** - - `soup data ingest` now splits Markdown by headings (one JSONL row per section with `section` + `level` + `text` fields). - - `soup data decontaminate --benchmark-file ` accepts operator-supplied JSONL corpus instead of only bundled benchmarks. - - `soup data forge` and other data tools now support custom prompt strategies via `prompt_strategy: module:function` in the config (importlib resolver + per-row exception handling). -- **+111 net new tests** (8051 → 8162) in the new `test_v0537.py`. All four review agents ran; 38 findings fixed. Key hardening: symlink checks via `os.lstat` on raw paths (TOCTOU policy), atomic writes via tempfile + `os.replace`, error message redaction, narrowed exception handlers, tool endpoint Bearer auth opt-in, and comprehensive coverage for every live pathway. +- **fsspec live loaders for remote datasets.** `data.train: s3://bucket/data.jsonl` (plus `gs://` / `gcs://` / `az://` / `abfs://` / `abfss://` / `oci://`) now loads through `fsspec` with the v0.42.0 SSRF-hardened URI validator (bucket regex + userinfo/query/fragment rejection). Threads `data.streaming` (HF `load_dataset(streaming=True)`) and `data.buffer_size` shuffle. Friendly Rich panel names the `pip install ` advisory when the backend SDK is missing. Row-count capped at 1M to defend against pathological remote objects. +- **Live ModelScope + Modelers hub dispatcher.** New `utils/hubs.download_repo()` and `upload_repo()` lazy-import the matching SDK per backend (`huggingface_hub`, `modelscope`, `openmind-hub`). Shared `_validate_repo_id_shape` rejects bool / null-byte / leading-slash / `..` segments / control characters / oversize; `local_dir` and `folder_path` containment-checked under cwd. `soup train` now pre-fetches non-HF `base` models into `.soup_hub_cache//` and reuses cached snapshots on resumed runs. +- **`[trackers]` extra + friendly missing-dep advisory.** New `pip install soup-cli[trackers]` bundles MLflow + SwanLab + Trackio. When you pass `--tracker mlflow` without the package installed, `soup train` now surfaces a clear `pip install` advisory before construction instead of HF Trainer's generic ImportError. The probe uses `importlib.util.find_spec` (non-executing) so swanlab can't initialise network threads during the check. +- **PostHog telemetry network (opt-IN, silent-fail).** `utils/trackers.send_telemetry_payload()` lazy-imports `httpx` and POSTs hardware-info-only payloads with a 1-second hard timeout. HTTPS-only endpoint check goes through the same SSRF validator as hub endpoints (private-IP / link-local / RFC1918 rejected). Disabled unless `SOUP_TELEMETRY=1`; every exception is swallowed so telemetry can never crash training. +- **Bundled demo fixtures as package data.** `soup data demo` fixtures (4 JSONLs) migrated from `examples/data/` to `soup_cli/data/_fixtures/` so they ship inside the wheel — zipapp / namespace-package safe. `_bundle_source_path` falls back to the legacy `examples/data/` location for editable installs. +- **HF Space SDK auto-pick from `requirements.txt`.** `soup deploy hf-space --template-dir ` now reads the rendered `requirements.txt` and picks `space_sdk="streamlit"` when the file lists `streamlit`, `"gradio"` otherwise. Closes the v0.40.2 known limitation that custom templates always created Spaces with `space_sdk="gradio"`. +- **+95 net new tests** (8162 → 8257) in `test_v0538.py`. Three review agents ran (python / code / security); 16 findings fixed: cwd-containment on `local_dir`/`folder_path`, Pydantic `model_copy(update=...)` instead of attribute mutation, idempotent train pre-fetch via cache probe, row-count cap on remote materialisation, `modelscope.push_model` kwarg fix, SSRF re-validation on telemetry endpoint override, `detect_space_sdk` size cap, and more. ## Why Soup? @@ -3481,6 +3477,90 @@ soup data decontaminate --input training.jsonl --benchmarks mmlu,gsm8k,humaneval The scorecard reports PII flagged, toxic flagged, language distribution, mean educational value, and decontamination removed. PII detection uses a narrow ReDoS-hardened regex set (email / phone / SSN / credit-card) with a 50 KB pre-cap on every input. Language detection is a stopword heuristic across six languages. Toxicity is a keyword baseline; the Llama-Guard-3-1B variant + FineWeb-Edu classifier ship behind `[data-pro]` extras. Decontamination uses n-gram containment against benchmark corpora: use `--benchmarks mmlu,gsm8k` for built-in allowlist, or `--benchmark-file custom_benchmark.jsonl` for your own corpus. +## Remote Datasets (S3 / GCS / Azure / OCI) + +Point `data.train` at any object in the v0.42.0 fsspec allowlist and `soup train` will stream it through `fsspec.open` after running the URI through the same SSRF-hardened validator used everywhere else in Soup (bucket regex, no userinfo / query / fragment): + +```yaml +data: + train: s3://my-bucket/datasets/train.jsonl + format: alpaca + streaming: true # opt-in HF datasets streaming with shuffle + buffer_size: 10000 # shuffle buffer (requires streaming=true) +``` + +Recognised schemes: `s3://`, `gs://`, `gcs://`, `az://`, `abfs://`, `abfss://`, `oci://`. The matching backend SDK (`s3fs` / `gcsfs` / `adlfs` / `ocifs`) is lazy-imported — install only what you need or grab the convenience extra: + +```bash +pip install soup-cli[remote] # fsspec + s3fs + gcsfs + adlfs +``` + +Materialised rows are capped at 1M to defend against pathological remote objects; use a local split for larger jobs. + +## Alternative Model Hubs (ModelScope / Modelers) + +Set `training.hub` to fetch the base model from a non-HF Hub: + +```yaml +base: baichuan-inc/Baichuan2-7B +task: sft +training: + hub: modelscope # or "modelers" +``` + +`soup train` pre-fetches the model into `./.soup_hub_cache//` via the matching SDK (`modelscope.snapshot_download` / `openmind_hub.snapshot_download`) and rewrites `cfg.base` to the local snapshot. Re-runs reuse the cached snapshot. Both `huggingface-hub`, `modelscope`, and `openmind-hub` are lazy-imported — install only what you need. + +Programmatic API: + +```python +from soup_cli.utils.hubs import download_repo, upload_repo + +local_path = download_repo("modelscope", "baichuan-inc/Baichuan2-7B", local_dir="./snap") +upload_repo("modelers", "my-org/my-model", folder_path="./output", commit_message="Soup v0.53.8") +``` + +The dispatcher enforces shape validation on every input (bool / null-byte / leading-slash / `..` segments / control characters / oversize all rejected) and runs cwd-containment on `local_dir` / `folder_path`. + +## Experiment Trackers (MLflow / SwanLab / Trackio) + +Pick a tracker on the CLI; Soup threads it into HF Trainer's `report_to`: + +```bash +soup train --tracker mlflow +soup train --tracker swanlab +soup train --tracker trackio +``` + +If the package is not installed, Soup now surfaces a friendly advisory before training starts instead of a mid-run ImportError: + +``` +--tracker mlflow requires the 'mlflow' package. Install with: pip install soup-cli[trackers] (or pip install mlflow) +``` + +```bash +pip install soup-cli[trackers] # mlflow + swanlab + trackio +``` + +## Telemetry (opt-IN, hardware-info-only) + +Soup ships an opt-IN telemetry sender that POSTs hardware-info-only payloads (`soup_version` / `command` / `python` major.minor / `os` / `arch` / optional `duration_seconds`) — no dataset paths, model names, or config contents. Enable per-shell: + +```bash +SOUP_TELEMETRY=1 soup train --config soup.yaml +``` + +The sender uses a 1-second hard timeout, HTTPS-only with private-IP / link-local rejection (same SSRF policy as hub endpoints), and swallows every exception silently — telemetry can never crash training. Disabled by default until a public privacy policy ships. + +## HF Space SDK Auto-Pick + +When you deploy a custom Space template directory, Soup now picks `space_sdk="streamlit"` / `"gradio"` from the rendered `requirements.txt`: + +```bash +soup deploy hf-space --space my-org/my-app --model my-org/my-model --template-dir ./my-template +``` + +If `requirements.txt` lists `streamlit`, the Space is created with the Streamlit SDK. Otherwise (no requirements, gradio listed, etc.), Soup falls back to the Gradio default. The HF Hub allows `docker` and `static` SDKs too, but those cannot be inferred from `requirements.txt` alone — use the built-in templates or supply a custom one with an explicit `--sdk` override. + ## Plugin System Drop a Python module under `soup_cli/plugins/` (or any package importable by Soup) and register at import time: diff --git a/SECURITY.md b/SECURITY.md index aad66d3..298e6bf 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.53.7 -- Full support (latest) +- v0.53.8 -- Full support (latest) +- v0.53.7 -- Full support - v0.53.6 -- Full support - v0.53.5 -- Full support - v0.53.4 -- Full support @@ -153,6 +154,7 @@ No known critical vulnerabilities in current releases. - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML) - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input. - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS` +- **v0.53.8 — Remote data + Hubs + Trackers (wave 2)**: 6 features wired live. 3 review-agent waves (python / code / security) ran; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). (#130 hub dispatcher) `download_repo` / `upload_repo` reject `local_dir` / `folder_path` outside cwd via `utils.paths.is_under_cwd` (security-review HIGH — closes a gap where a public helper could write to arbitrary absolute paths); `_validate_repo_id_shape` rejects bool / null-byte / leading-slash / `..` segments / control characters (CR/LF, defence-in-depth against header injection) / oversize (>200 chars). `modelscope.push_model` does NOT accept `commit_message` — the sanitized message is omitted on that backend to avoid a silent TypeError at runtime (code-review HIGH). (#130 `commands/train.py` pre-fetch) cache subdir name sanitised via `re.sub(r"[^A-Za-z0-9._-]+", "__", cfg.base)` then strip-leading/trailing-dots to defeat Windows `..\\` traversal that would have escaped `.soup_hub_cache/` (security-review HIGH); resolved cache dir re-checked via `is_under_cwd` as defence-in-depth. `cfg.base = local_path` mutation replaced with `cfg = cfg.model_copy(update={"base": local_path})` so Pydantic field validators rerun (matches v0.33.0 #47 immutability policy; python-review + code-review HIGH). Pre-fetch idempotent — probes `cache_dir / config.json` and reuses on resume runs. `ImportError` print uses `rich.markup.escape` to defend against markup injection from a crafted dep name (security-review MEDIUM). (#90 telemetry) `_telemetry_endpoint_is_safe` re-runs the v0.51.0 hub-endpoint SSRF validator on every caller-supplied endpoint (HTTPS + private-IP / link-local / RFC1918 rejected) — defence-in-depth so a misconfigured `endpoint='https://10.0.0.1/'` cannot reach internal networks even when telemetry is opt-IN (security-review MEDIUM). Lazy `httpx` import + 1s hard timeout + silent-fail on every exception (no `traceback`). (#85 fsspec loader) `validate_remote_uri` runs BEFORE any `fsspec.open` connection (bucket regex, no userinfo / query / fragment); row-count capped at 1,000,000 to defend against pathological remote objects (security-review MEDIUM). (#69 `detect_space_sdk`) input length capped at 256 KB to match v0.39.0 Part E template policy — refuses to scan oversized requirements.txt that would degrade response time (security-review LOW). (#93 package-data) fixtures moved to `soup_cli/data/_fixtures/`; `_bundle_source_path` falls back to legacy `examples/data/` for editable installs. (#89 `tracker_missing_dep_message`) switched from `__import__(pkg)` to `importlib.util.find_spec` (non-executing probe) so swanlab can't initialise network threads during the absence check (code-review MEDIUM; `sys.modules[pkg] = None` sentinel still honoured for tests). Test surface: `tests/test_v0538.py` carries 66 new tests including 4 new cwd-containment + SSRF rejection regression guards. (v0.53.8) - **v0.53.7 — Data Forge + Pipeline live (wave 1)**: 11 features (6 new + 5 v0.53.6 stub-to-live), 1 CRITICAL + 11 HIGH + 17 MEDIUM + 10 LOW review findings fixed. (#103 `/v1/tools/bash` REVERTED to HTTP 501) security review caught that `/bin/sh -c` spawns a child process outside the RLVR sandbox's OS-level isolation (`unshare(CLONE_NEWNET)` on Linux, `sandbox-exec` on macOS, socket monkey-patch); a caller could reach the cloud-metadata service from the child shell. Endpoint returns 501 with v0.53.8 marker until container/namespace work lands. Python + web_search remain LIVE. (#103 `/v1/tools/{python,web_search}` Bearer auth gate) optional `auth_token` kwarg on `_create_app` + `Bearer` gate on both endpoints; default `None` preserves loopback-only back-compat per v0.30.0 Part C CORS policy. (#103 `/v1/tools/web_search` deny-by-default) `WebSearchConfig.domain_allowlist` defaults to empty = deny-all (returns 403 `"web_search disabled"`); only operators who supply a domain list via constructor enable the endpoint. 5s `httpx` timeout, 5-result cap, snippet null-byte stripped. (#102 `/v1/messages` redaction) malformed Anthropic payloads (incl. validator-shape errors) mapped to generic `"Invalid request"` 400 body; detailed exception text logged server-side at DEBUG (matches v0.40.3 #33 trace-log policy). (#102 vLLM `/v1/messages` CORS) restricted to loopback origins on both transformers AND vLLM backends since it's a mutation route (matches v0.30.0 Part C transformers-backend policy that was previously vLLM-wildcard). (#102 SSE header injection) `model` + `msg_id` strings stripped of CRLF / NUL / oversize before SSE `data:` embedding — defends against `model="foo\ndata:{injected}\n\n"` injecting spurious SSE frames; `Cache-Control: no-store` + `X-Accel-Buffering: no` headers on `StreamingResponse` prevent intermediary caching of model outputs. (#106 `_node_seed` symlink TOCTOU) `os.lstat` runs on the RAW path BEFORE `realpath` resolves the symlink (matches v0.33.0 #22 `prune_checkpoints` policy); `realpath`-first ordering would have been a no-op since `S_ISLNK` on a resolved path is always False. (#106 `_save_checkpoint` atomic) switched from `path + ".tmp"` + plain `open()` to `tempfile.mkstemp(dir=path.parent, suffix=".tmp")` + `os.fdopen` + `os.replace` (matches v0.43.0 Part D `copy_bundle_to` / v0.48.0 `write_mix_recipe` policy); eliminates predictable `.tmp` symlink hazard. (#106 `failed_reason` redaction) raw exception messages run through `_redact_exc_message` which strips POSIX `/abs/path` and Windows `C:\abs\path` runs to `os.path.basename` and caps at 256 chars (matches v0.34.0 `crash.py` redaction policy); a `FileNotFoundError` no longer leaks the operator's `$HOME` into the world-readable `.checkpoint.json`. (#106 `_node_code` row JSON double-encode) row payloads serialised via `json.dumps(json.dumps(row))` before f-string embedding into the RLVR sandbox wrapper — defends against Python `repr()` escape edge cases on Unicode that could break sandbox invocation. (#106 resume rehydration) completed-node outputs rehydrated from per-node `.node-.jsonl` sidecar files (NOT empty list — a v0.53.7 in-progress draft had a correctness defect that silently fed downstream consumers an empty list, causing zero-row output on resume). (#86 atomic Arrow write) `Dataset.save_to_disk` writes to a `.tmp_` sibling directory then `os.replace`-ed (mirrors v0.48.0 `write_mix_recipe` pattern); mid-write Ctrl-C / OOM no longer leaves a partial Arrow directory that looks complete to subsequent runs. (#86 cache-hash gate) SFT + Pretrain wrappers verify `metadata.json::cache_key` matches the freshly-computed `make_preprocess_cache_key(base, max_length, format, train)` BEFORE consuming the Arrow shards; mismatch raises `ValueError("cache hash mismatch: re-run `soup data preprocess`")` so a user who pre-tokenized with one config and changed `max_length` cannot silently train on stale shards. Missing metadata.json falls back to yellow advisory + trusted mode. (#87 `resolve_prompt_strategy`) operator-supplied module imported via `importlib`; signature-validated `Mapping → Mapping`; `except` narrowed to `TypeError` only so the explicit ValueError from the signature-shape check propagates (a v0.53.7 in-progress draft had `except (TypeError, ValueError): pass` which swallowed its own validator); `assert isinstance(spec, str)` replaced with `raise TypeError` so the validator survives `python -O`. Per-row invocations swallow exceptions at DEBUG and fall through to the raw row (matches v0.33.0 #47 `CrossDocCollator` immutability policy). LIMITATION: this is a trusted-input surface — do NOT use `prompt_strategy` with operator-controlled YAML on shared CI hosts. (#88 markdown ingest) heading-section text capped per regex `^#{1,6}\s` so a crafted MD with a 1-byte heading followed by 1 GB body still respects the existing `data/ingest` MD branch size cap from v0.42.0 Part F. (#111 forge `--judge-provider`) lazy v0.20.0 provider instantiation reuses existing SSRF hardening per backend: Ollama localhost-only (rejects remote IPs), Anthropic env-only API key (`ANTHROPIC_API_KEY` — never CLI arg, matches v0.20.0 policy), vLLM scheme-allowlist + loopback HTTP. Per-call exceptions counted + logged at DEBUG (no silent silent-degrade). (#112 `--benchmark-file`) operator JSONL path runs through `is_under_cwd` + `os.lstat + S_ISLNK` rejection BEFORE the corpus is loaded (matches v0.46.0 `agent_forge.load_spec_file` policy). (#105 `instantiate_trainer_plugins`) `_instantiate_simple_plugin` returns None + WARNING when no `attr_candidates` match the imported module (matches v0.45.0 `register_plugin` policy — surface bad shapes loudly, not silently); `except TypeError` narrowed (a v0.53.7 in-progress draft swallowed all `TypeError` from the wrong layer). Cross-cutting: SSE generator return types annotated (`Generator[str, None, None]`); `JUDGE_PROVIDERS: frozenset[str]` PEP 585 element type; `extract_row_text` exposed publicly in `utils/data_score.__all__` (cross-module import of private helper eliminated). Test surface: `tests/test_v0537.py` carries 111 new tests; CI green across Ubuntu/macOS/Windows × Python 3.9/3.11/3.12. (v0.53.7) - **v0.53.6 — Plugin + Agent + Anthropic API**: 6 features — 3 live, 3 deferred-live stubs. (#101 SoupPluginCallback live) New `soup_cli/monitoring/plugin_callback.py` ships `SoupPluginCallback(transformers.TrainerCallback)` that dispatches the four canonical hook events to every enabled plugin in the v0.45.0 registry. Per-hook invocation runs through `_safe_invoke` which swallows ALL exceptions at WARNING with `exc_info=True` — defends against one misbehaving plugin crashing a multi-hour training run (mirrors v0.44.0 / v0.45.0 plugin-loader policy). Hook snapshot is collected ONCE in `build_plugin_callback` and passed to the constructor so a plugin registered between the "is empty?" check and ctor cannot silently slip into the active hook list (code-review HIGH fix — race-window closed). `attach_plugin_callback(trainer, console=None) -> bool` short-circuits to False when no plugins are enabled (zero overhead on the hot path); inner `add_callback` failure is swallowed at DEBUG so a plugin-infrastructure bug never crashes training. Wired into all 13 transformer-backend trainers; source-level grep regression test parametrized over every trainer file. (#102 Anthropic `/v1/messages` live) New POST `/v1/messages` route reuses the v0.45.0 SSRF-hardened `from_anthropic` converter + the existing chat handler. Streaming requests return 501 BEFORE schema validation (defence-in-depth — defends against stream-only attackers leaking validator-error detail). All validation paths (`validate_anthropic_payload` / `from_anthropic` / `ChatCompletionRequest(**openai_payload)`) are wrapped to map to a generic `"Invalid request"` 400 body; detailed exception text is logged server-side at DEBUG (security-review MEDIUM fix — matches `utils/errors.py` policy "HTTP error responses return generic messages, details logged server-side"). The existing loopback-CORS restriction from v0.30.0 Part C and the trace-log-redaction from v0.40.3 #33 both apply to the new route. (#104 n-gram speculative decoding live) `_generate_response` gains `ngram_config: Any = None` kwarg; when set, emits `prompt_lookup_num_tokens=int(ngram_config.num_draft_tokens)` into `model.generate(...)`. The int cast is wrapped in `try/except (TypeError, AttributeError)` as defence-in-depth; primary validation lives upstream in v0.45.0 `validate_ngram_config` (bool-rejected, bounded `[1, 32]`). Mutually exclusive with `assistant_model` — real draft model wins. (#103 server-side tool endpoints STUB) Three new POST routes `/v1/tools/python` / `/v1/tools/bash` / `/v1/tools/web_search` return 501 with v0.53.7 marker. Tool names are hardcoded string literals (not user-controlled); no payload inspection. Live RLVR `code_exec` sandbox HTTP wrapper (python / bash) + `WebSearchConfig.domain_allowlist` enforcement (web_search) land in v0.53.7. (#106 `run_recipe` STUB + `soup data recipe --execute`) New `utils/recipe_run.run_recipe(dag, *, output_dir, ...)` type-checks every parameter (TypeError on non-`RecipeDAG` / non-string / empty `output_dir` / non-bool `resume` / non-string `judge_*`) BEFORE the `NotImplementedError("v0.53.7")`. CLI `--execute --output ` enforces `is None` guard on `--output` (project policy since v0.40.6 — empty-string is a distinct operator error, NOT silent missing) + `is_under_cwd` containment at the CLI boundary BEFORE the `run_recipe` call (security-review MEDIUM fix — defends against future v0.53.7 live runner bypass; never want the live runner to be the first/only enforcement point per project policy). Regression test asserts the v0.53.7 marker does NOT surface on outside-cwd paths — guards against future live runner accidentally bypassing CLI containment. (#105 `instantiate_trainer_plugins` STUB) Validates name list through existing `validate_trainer_plugin_list` (closed allowlist, dedup, ≤8/run, canonicalisation — all v0.45.0 hardening) BEFORE raising `NotImplementedError`. Canonical names echoed in the error message are allowlist-validated lowercase strings (no injection surface). Test surface: 1 new test file (`tests/test_v0536.py`) carrying 53 new tests covering: per-hook exception swallow with caplog isolation, source-level grep parametrized over 13 trainer files (canonical import + invocation), generic `"Invalid request"` body redaction with detail logged at DEBUG, oversize `max_tokens` cap, n-gram kwarg omitted when config is None (regression guard against future "always emit" refactor), `run_recipe` type-rejection matrix (non-DAG / empty output_dir / non-str checkpoint_dir / non-str judge_provider / non-str judge_model / bool resume), `--execute --output ""` rejected with `"must be a non-empty path"`, outside-cwd `--output` rejected with `v0.53.7` marker NEVER surfacing on rejected paths. Known limitations: (1) vLLM `/v1/messages` parity deferred to v0.53.7 — Anthropic route is transformers-only. (2) Streaming `/v1/messages` returns 501 — true SSE event-shape ships in v0.53.7. (3) Plugin hook snapshot is at callback-construction time — a plugin registered mid-run does not retroactively receive hooks (by design — prevents partial-run inconsistencies). (4) n-gram is transformers-only — vLLM ships its own prompt-lookup path that needs a different kwarg. (5) Tool endpoints + recipe runner + trainer-plugin instantiation are stubs — live wiring in v0.53.7. (v0.53.6) diff --git a/pyproject.toml b/pyproject.toml index d4ca06c..7f5c242 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.53.7" +version = "0.53.8" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" @@ -61,6 +61,10 @@ sglang = ["sglang>=0.2.0", "fastapi>=0.104.0", "uvicorn>=0.24.0"] mlx = ["mlx>=0.20.0", "mlx-lm>=0.20.0"] cce = ["cut-cross-entropy>=24.10.0"] tui = ["textual>=0.50.0"] +# v0.53.8 #89 — bundle MLflow / SwanLab / Trackio for `--tracker` users. +trackers = ["mlflow>=2.0.0", "swanlab>=0.3.0", "trackio>=0.0.1"] +# v0.53.8 #85 — fsspec backends for remote dataset loading (s3 / gs / az / oci). +remote = ["fsspec>=2024.1.0", "s3fs>=2024.1.0", "gcsfs>=2024.1.0", "adlfs>=2024.1.0"] [project.scripts] soup = "soup_cli.cli:run" @@ -72,6 +76,10 @@ Issues = "https://github.com/MakazhanAlpamys/Soup/issues" [tool.hatch.build.targets.wheel] packages = ["soup_cli"] +# v0.53.8 #93 — include bundled fixture JSONLs as package data so +# `soup data demo` works in zipapp / namespace-package installs. +[tool.hatch.build.targets.wheel.force-include] +"soup_cli/data/_fixtures" = "soup_cli/data/_fixtures" [tool.ruff] target-version = "py39" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index b5144f9..eb2a001 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.53.7" +__version__ = "0.53.8" diff --git a/soup_cli/commands/data.py b/soup_cli/commands/data.py index 5f6b0fa..d37647a 100644 --- a/soup_cli/commands/data.py +++ b/soup_cli/commands/data.py @@ -1189,8 +1189,34 @@ def download_dataset( "(v0.36.0). Only enable if you trust the source." ), ), + hub: str = typer.Option( + "hf", + "--hub", + help=( + "Source hub: hf (default) / modelscope / modelers. " + "Non-HF hubs require the matching SDK (v0.53.8 #130)." + ), + ), ): """Download a HuggingFace dataset and save as JSONL.""" + # v0.53.8 #130 — validate --hub at the CLI boundary; only `hf` is wired + # for live dataset download in this release (modelscope / modelers + # dataset SDKs differ from snapshot_download; live wiring tracked for + # v0.53.9). Non-HF hubs surface a friendly error. + from soup_cli.utils.hubs import validate_hub_name + + try: + hub_canonical = validate_hub_name(hub) + except (TypeError, ValueError) as exc: + console.print(f"[red]{exc}[/]") + raise typer.Exit(code=2) from exc + if hub_canonical != "hf": + console.print( + f"[red]--hub {hub_canonical} dataset download is not yet wired; " + f"use `from soup_cli.utils.hubs import download_repo` to snapshot " + f"a repo, or wait for v0.53.9.[/]" + ) + raise typer.Exit(code=1) max_download_samples = 1_000_000 if samples is not None and samples > max_download_samples: console.print( diff --git a/soup_cli/commands/deploy.py b/soup_cli/commands/deploy.py index 1b9dda1..19f0ad4 100644 --- a/soup_cli/commands/deploy.py +++ b/soup_cli/commands/deploy.py @@ -499,11 +499,15 @@ def hf_space( # --- Render template files --- try: if template_dir is not None: - from soup_cli.utils.hf_space import render_custom_template_dir + from soup_cli.utils.hf_space import ( + detect_space_sdk, + render_custom_template_dir, + ) files = render_custom_template_dir(template_dir, model_repo=model) - # Custom templates default to gradio SDK unless requirements - # imply otherwise; we record gradio for create_repo space_sdk. - sdk = "gradio" + # v0.53.8 #69 — auto-pick space_sdk from requirements.txt + # (closes the v0.40.2 known limitation that custom templates + # always created the Space with space_sdk="gradio"). + sdk = detect_space_sdk(files.get("requirements.txt")) else: files = render_space_template(template, model_repo=model) sdk = HF_SPACE_TEMPLATES[template]["sdk"] diff --git a/soup_cli/commands/serve.py b/soup_cli/commands/serve.py index 513344c..9ade86d 100644 --- a/soup_cli/commands/serve.py +++ b/soup_cli/commands/serve.py @@ -1286,12 +1286,12 @@ def _create_app( # the RLVR sandbox's OS-level isolation (``unshare(CLONE_NEWNET)`` # / macOS ``sandbox-exec``); a caller can reach # ``http://169.254.169.254/...`` from the child shell. Reverted to - # 501 until container/namespace work lands in v0.53.8. + # 501 until container/namespace work lands in v0.53.9. raise HTTPException( status_code=501, detail=( "Server-side tool 'bash' live execution deferred to " - "v0.53.8 — sandbox isolation requires container/namespace " + "v0.53.9 — sandbox isolation requires container/namespace " "work." ), ) diff --git a/soup_cli/commands/train.py b/soup_cli/commands/train.py index 220dce9..6437076 100644 --- a/soup_cli/commands/train.py +++ b/soup_cli/commands/train.py @@ -692,6 +692,65 @@ def train( console.print(f"[red]{_esc(str(exc))}[/]") raise typer.Exit(code=2) from exc console.print("[dim]Setting up model + trainer...[/]") + # v0.53.8 #130 — pre-fetch model from non-HF hub into a local cache and + # rewrite cfg.base to point at the local snapshot. The trainer wrappers + # still use transformers.from_pretrained, which reads HF Hub by default; + # by snapshotting first we keep every wrapper unchanged. + hub_name = getattr(cfg.training, "hub", "hf") or "hf" + if hub_name != "hf": + import re as _re + + from rich.markup import escape as _markup_escape + + from soup_cli.utils.hubs import download_repo + from soup_cli.utils.paths import is_under_cwd + + # Sanitise cache subdir name — strip every path-separator and + # `..` segment so a crafted ``base: ../../etc`` cannot escape the + # cache root (Windows ``\\`` and POSIX ``/`` both blocked). + safe_slug = _re.sub(r"[^A-Za-z0-9._-]+", "__", cfg.base).strip("._-") or "model" + cache_dir = (Path.cwd() / ".soup_hub_cache" / safe_slug).resolve() + if not is_under_cwd(str(cache_dir)): + console.print( + "[red]Resolved hub cache dir escaped the current working " + "directory; refusing to download.[/]" + ) + raise typer.Exit(code=1) + try: + # Idempotency: if the cache dir already has a config.json, skip + # the re-download (modelscope/openmind-hub also short-circuit on + # match but having an explicit probe lets us print a clear hint). + existing_cfg = cache_dir / "config.json" + if existing_cfg.is_file(): + local_path = str(cache_dir) + console.print( + f"[dim]Using cached snapshot at {local_path}[/]" + ) + else: + local_path = download_repo( + hub_name, + cfg.base, + local_dir=str(cache_dir), + ) + console.print( + f"[dim]Fetched {cfg.base} from hub={hub_name} → " + f"{local_path}[/]" + ) + # Use ``model_copy(update=...)`` so the Pydantic field + # validators on ``base`` rerun (matches v0.33.0 #47 / v0.40.0 + # Part B immutability policy). + cfg = cfg.model_copy(update={"base": local_path}) + except ImportError as exc: + console.print(f"[red]{_markup_escape(str(exc))}[/]") + raise typer.Exit(code=1) from exc + + # v0.53.8 #89 — friendly missing-dep advisory for `--tracker `. + if report_to and report_to not in ("none", "wandb", "tensorboard"): + from soup_cli.utils.trackers import tracker_missing_dep_message + + msg = tracker_missing_dep_message(report_to) + if msg: + console.print(f"[yellow]{msg}[/]") trainer_kwargs = { "device": device, "report_to": report_to, diff --git a/soup_cli/data/_fixtures/alpaca_tiny.jsonl b/soup_cli/data/_fixtures/alpaca_tiny.jsonl new file mode 100644 index 0000000..b659bb9 --- /dev/null +++ b/soup_cli/data/_fixtures/alpaca_tiny.jsonl @@ -0,0 +1,10 @@ +{"instruction": "Classify the sentiment of this sentence.", "input": "I love this product, it's amazing!", "output": "Positive"} +{"instruction": "Translate to Spanish.", "input": "Hello, how are you?", "output": "Hola, ¿cómo estás?"} +{"instruction": "Write a haiku about nature.", "input": "", "output": "Green leaves dance in wind\nWhispering ancient forest\nPeace in silence dwells"} +{"instruction": "Explain quantum computing in simple terms.", "input": "", "output": "Quantum computers use quantum bits (qubits) instead of normal bits. Unlike regular bits which are 0 or 1, qubits can be both at once (superposition). This allows quantum computers to solve certain problems much faster than regular computers."} +{"instruction": "Fix the grammar in this sentence.", "input": "She go to the store yesterday.", "output": "She went to the store yesterday."} +{"instruction": "List 3 benefits of exercise.", "input": "", "output": "1. Improves cardiovascular health and strengthens the heart.\n2. Helps maintain a healthy weight and builds muscle.\n3. Reduces stress and improves mental health and mood."} +{"instruction": "What is the capital of France?", "input": "", "output": "The capital of France is Paris."} +{"instruction": "Summarize this text in one sentence.", "input": "Machine learning is a subset of artificial intelligence that enables computers to learn from data without being explicitly programmed. It uses algorithms to identify patterns in data and improve through experience.", "output": "Machine learning allows computers to learn patterns from data without explicit programming."} +{"instruction": "Create a Python function to add two numbers.", "input": "", "output": "def add_numbers(a, b):\n \"\"\"Add two numbers and return the result.\"\"\"\n return a + b\n\n# Example usage\nresult = add_numbers(5, 3)\nprint(result) # Output: 8"} +{"instruction": "Identify the main idea of this paragraph.", "input": "Climate change is caused by increased greenhouse gases in the atmosphere, primarily from human activities like burning fossil fuels. This leads to global warming, which affects weather patterns, sea levels, and ecosystems.", "output": "Greenhouse gases from human activities cause climate change and have significant environmental impacts."} diff --git a/soup_cli/data/_fixtures/chat_preferences.jsonl b/soup_cli/data/_fixtures/chat_preferences.jsonl new file mode 100644 index 0000000..1af5a82 --- /dev/null +++ b/soup_cli/data/_fixtures/chat_preferences.jsonl @@ -0,0 +1,5 @@ +{"prompt": "What is machine learning?", "chosen": [{"role": "user", "content": "What is machine learning?"}, {"role": "assistant", "content": "Machine learning is a field of artificial intelligence where systems learn from data automatically. It uses algorithms to identify patterns and improve performance through experience, without being explicitly programmed for every task."}], "rejected": [{"role": "user", "content": "What is machine learning?"}, {"role": "assistant", "content": "It's computers doing stuff with data I guess."}]} +{"prompt": "How do I learn Python?", "chosen": [{"role": "user", "content": "How do I learn Python?"}, {"role": "assistant", "content": "Here are effective ways to learn Python:\n1. Start with fundamentals: variables, loops, functions\n2. Practice with small projects\n3. Use resources like Python.org docs and tutorials\n4. Build projects to apply what you learn\n5. Join communities for support"}], "rejected": [{"role": "user", "content": "How do I learn Python?"}, {"role": "assistant", "content": "Just google it."}]} +{"prompt": "What's the difference between AI and ML?", "chosen": [{"role": "user", "content": "What's the difference between AI and ML?"}, {"role": "assistant", "content": "AI (Artificial Intelligence) is the broader field of creating intelligent systems. ML (Machine Learning) is a subset of AI focused on learning from data. All machine learning is AI, but not all AI uses machine learning."}], "rejected": [{"role": "user", "content": "What's the difference between AI and ML?"}, {"role": "assistant", "content": "They're basically the same thing, just different names."}]} +{"prompt": "Explain neural networks simply.", "chosen": [{"role": "user", "content": "Explain neural networks simply."}, {"role": "assistant", "content": "Neural networks are inspired by the human brain. They consist of layers of connected nodes (neurons) that process information. Each connection has a weight that adjusts during training. This allows the network to learn complex patterns in data."}], "rejected": [{"role": "user", "content": "Explain neural networks simply."}, {"role": "assistant", "content": "Neural networks are very complex mathematical structures that use backpropagation through computational graphs with gradient descent optimization."}]} +{"prompt": "How should I structure a machine learning project?", "chosen": [{"role": "user", "content": "How should I structure a machine learning project?"}, {"role": "assistant", "content": "Follow this structure:\n1. Define the problem and gather data\n2. Explore and clean the data\n3. Choose and train a model\n4. Evaluate performance\n5. Iterate and improve\n6. Deploy to production\n7. Monitor and maintain"}], "rejected": [{"role": "user", "content": "How should I structure a machine learning project?"}, {"role": "assistant", "content": "Just throw all your code in one file and train the model."}]} diff --git a/soup_cli/data/_fixtures/dpo_sample.jsonl b/soup_cli/data/_fixtures/dpo_sample.jsonl new file mode 100644 index 0000000..d20d01b --- /dev/null +++ b/soup_cli/data/_fixtures/dpo_sample.jsonl @@ -0,0 +1,8 @@ +{"prompt": "What is photosynthesis?", "chosen": [{"role": "user", "content": "What is photosynthesis?"}, {"role": "assistant", "content": "Photosynthesis is the process by which green plants, algae, and some bacteria convert light energy into chemical energy. Using sunlight, water, and carbon dioxide, they produce glucose and oxygen. The overall equation is: 6CO2 + 6H2O + light energy → C6H12O6 + 6O2. This process occurs primarily in the chloroplasts of plant cells."}], "rejected": [{"role": "user", "content": "What is photosynthesis?"}, {"role": "assistant", "content": "It's when plants eat sunlight."}]} +{"prompt": "How do I write a good commit message?", "chosen": [{"role": "user", "content": "How do I write a good commit message?"}, {"role": "assistant", "content": "A good commit message should:\n1. Use the imperative mood in the subject line (e.g., 'Add feature' not 'Added feature')\n2. Keep the subject line under 50 characters\n3. Separate the subject from the body with a blank line\n4. Use the body to explain what and why, not how\n5. Reference relevant issue numbers when applicable"}], "rejected": [{"role": "user", "content": "How do I write a good commit message?"}, {"role": "assistant", "content": "Just write 'fix' or 'update' and push it."}]} +{"prompt": "What are the benefits of exercise?", "chosen": [{"role": "user", "content": "What are the benefits of exercise?"}, {"role": "assistant", "content": "Regular exercise provides numerous benefits:\n- Cardiovascular health: strengthens the heart and improves circulation\n- Mental health: reduces anxiety and depression through endorphin release\n- Weight management: helps maintain a healthy body composition\n- Bone and muscle strength: prevents osteoporosis and sarcopenia\n- Improved sleep quality and cognitive function\n- Reduced risk of chronic diseases like type 2 diabetes"}], "rejected": [{"role": "user", "content": "What are the benefits of exercise?"}, {"role": "assistant", "content": "Exercise is good for you. You should do it."}]} +{"prompt": "Explain the concept of recursion in programming.", "chosen": [{"role": "user", "content": "Explain the concept of recursion in programming."}, {"role": "assistant", "content": "Recursion is when a function calls itself to solve a problem by breaking it into smaller subproblems. Every recursive function needs:\n1. A base case that stops the recursion\n2. A recursive case that reduces the problem\n\nFor example, calculating factorial:\n```python\ndef factorial(n):\n if n <= 1: # base case\n return 1\n return n * factorial(n - 1) # recursive case\n```\nfactorial(4) calls factorial(3), which calls factorial(2), and so on until reaching the base case."}], "rejected": [{"role": "user", "content": "Explain the concept of recursion in programming."}, {"role": "assistant", "content": "To understand recursion, you must first understand recursion. It's just a function that calls itself forever."}]} +{"prompt": "What is the difference between HTTP and HTTPS?", "chosen": [{"role": "user", "content": "What is the difference between HTTP and HTTPS?"}, {"role": "assistant", "content": "HTTP (HyperText Transfer Protocol) transmits data in plain text, making it vulnerable to interception. HTTPS (HTTP Secure) adds TLS/SSL encryption, which:\n- Encrypts data in transit between client and server\n- Authenticates the server's identity via certificates\n- Ensures data integrity (no tampering)\nHTTPS is now the standard for all websites and is required for handling sensitive data like passwords and payments."}], "rejected": [{"role": "user", "content": "What is the difference between HTTP and HTTPS?"}, {"role": "assistant", "content": "HTTPS has an S at the end. It's more secure I think."}]} +{"prompt": "How does a hash table work?", "chosen": [{"role": "user", "content": "How does a hash table work?"}, {"role": "assistant", "content": "A hash table stores key-value pairs for fast lookup. Here's how it works:\n1. A hash function converts the key into an array index\n2. The value is stored at that index\n3. On lookup, the same hash function computes the index to retrieve the value\n\nCollisions (two keys mapping to the same index) are handled via chaining (linked lists at each slot) or open addressing (probing for the next empty slot). Average time complexity is O(1) for insert, lookup, and delete."}], "rejected": [{"role": "user", "content": "How does a hash table work?"}, {"role": "assistant", "content": "It's like a dictionary. You put stuff in and get stuff out."}]} +{"prompt": "What should I consider when choosing a database?", "chosen": [{"role": "user", "content": "What should I consider when choosing a database?"}, {"role": "assistant", "content": "Key factors when choosing a database:\n- Data model: relational (SQL) for structured data with relationships, document (NoSQL) for flexible schemas, key-value for simple lookups, graph for relationship-heavy data\n- Scale requirements: read-heavy vs write-heavy workloads, expected data volume\n- Consistency needs: strong consistency (ACID) vs eventual consistency (BASE)\n- Query patterns: complex joins favor SQL; simple lookups favor NoSQL\n- Operational concerns: hosting, backups, team expertise, licensing costs"}], "rejected": [{"role": "user", "content": "What should I consider when choosing a database?"}, {"role": "assistant", "content": "Just use MongoDB for everything, it's web scale."}]} +{"prompt": "How do vaccines work?", "chosen": [{"role": "user", "content": "How do vaccines work?"}, {"role": "assistant", "content": "Vaccines train the immune system to recognize and fight specific pathogens without causing the disease. They work by introducing a harmless component of a pathogen (such as an inactivated virus, a protein subunit, or mRNA instructions) to the body. The immune system responds by producing antibodies and memory cells. If the real pathogen is encountered later, the immune system can mount a rapid, effective response, preventing or reducing the severity of illness."}], "rejected": [{"role": "user", "content": "How do vaccines work?"}, {"role": "assistant", "content": "They inject you with stuff to make you immune. That's pretty much it."}]} diff --git a/soup_cli/data/_fixtures/reasoning_math.jsonl b/soup_cli/data/_fixtures/reasoning_math.jsonl new file mode 100644 index 0000000..513bc1d --- /dev/null +++ b/soup_cli/data/_fixtures/reasoning_math.jsonl @@ -0,0 +1,5 @@ +{"instruction": "Solve this math problem: 7 + 5 * 2 =", "input": "", "output": "To solve 7 + 5 * 2, I need to follow the order of operations (PEMDAS).\n\nStep 1: Multiply first (5 * 2)\n5 * 2 = 10\n\nStep 2: Add (7 + 10)\n7 + 10 = 17\n\nAnswer: 17"} +{"instruction": "Explain the Pythagorean theorem and solve an example.", "input": "", "output": "The Pythagorean theorem states that in a right triangle, a² + b² = c², where c is the hypotenuse (longest side).\n\nExample: If a = 3 and b = 4, find c\n\nStep 1: Substitute into formula\n3² + 4² = c²\n\nStep 2: Calculate squares\n9 + 16 = c²\n\nStep 3: Add\n25 = c²\n\nStep 4: Take square root\nc = √25 = 5\n\nAnswer: c = 5"} +{"instruction": "Find the GCD of 48 and 18.", "input": "", "output": "To find the GCD (Greatest Common Divisor) of 48 and 18:\n\nMethod: Euclidean algorithm\n\nStep 1: 48 = 18 × 2 + 12\nStep 2: 18 = 12 × 1 + 6\nStep 3: 12 = 6 × 2 + 0\n\nWhen remainder is 0, the GCD is the last non-zero remainder.\n\nGCD(48, 18) = 6\n\nAnswer: 6"} +{"instruction": "Solve: x + 5 = 12", "input": "", "output": "This is a simple linear equation.\n\nGiven: x + 5 = 12\n\nStep 1: Subtract 5 from both sides\nx + 5 - 5 = 12 - 5\n\nStep 2: Simplify\nx = 7\n\nVerification: 7 + 5 = 12 ✓\n\nAnswer: x = 7"} +{"instruction": "What is 15% of 200?", "input": "", "output": "To find 15% of 200:\n\nMethod: Multiply by the percentage as a decimal\n15% = 15/100 = 0.15\n\n0.15 × 200 = 30\n\nAnswer: 30"} diff --git a/soup_cli/data/loader.py b/soup_cli/data/loader.py index ae160e8..a850af3 100644 --- a/soup_cli/data/loader.py +++ b/soup_cli/data/loader.py @@ -116,9 +116,18 @@ def load_dataset(data_config: DataConfig) -> dict: Supports: - Local files (.jsonl, .json, .csv, .parquet, .txt) - HuggingFace dataset names (auto-detected if no file extension) + - Remote fsspec URIs (s3://, gs://, gcs://, az://, abfs://, abfss://, oci://) — v0.53.8 #85 """ train_path = data_config.train + # v0.53.8 #85 — fsspec live remote loader. Schema accepts these URIs + # since v0.42.0; live loader lands here. Lazy-imports fsspec + the + # backend driver (s3fs / gcsfs / adlfs / ocifs) and surfaces a + # friendly Rich panel naming the pip install when the driver is + # missing. + if _looks_like_remote_uri(train_path): + return _load_remote_dataset(train_path, data_config) + # Check if it's a HuggingFace dataset if not Path(train_path).suffix: return _load_hf_dataset(train_path, data_config) @@ -214,6 +223,128 @@ def _validate_audio_files(data: list[dict], audio_dir: Path) -> list[dict]: return valid +def _looks_like_remote_uri(value: str) -> bool: + """Quick sniff for the fsspec scheme allowlist (v0.42.0 Part B).""" + if not isinstance(value, str) or "://" not in value: + return False + from soup_cli.utils.data_pipeline import is_remote_uri + + return is_remote_uri(value) + + +def _load_remote_dataset(train_path: str, data_config: DataConfig) -> dict: + """Load JSONL from a remote fsspec URI (s3 / gs / az / oci / etc.). + + Validates the URI via the v0.42.0 ``validate_remote_uri`` allowlist + (bucket regex, no userinfo/query/fragment) BEFORE opening any + connection — defends against URL injection into the fsspec backend. + + Streaming knobs (``data_config.streaming`` + ``buffer_size`` + ``shards``) + are honoured via :func:`datasets.load_dataset` when present; otherwise + the file is streamed as JSONL through :func:`fsspec.open`. + """ + from soup_cli.utils.data_pipeline import ( + required_remote_package, + validate_remote_uri, + ) + + canonical = validate_remote_uri(train_path) + scheme = canonical.split("://", 1)[0] + + try: + import fsspec # type: ignore[import-not-found] + except ImportError: + from rich.panel import Panel + + pkg = required_remote_package(scheme) or scheme + console.print( + Panel( + f"[bold yellow]Missing dependency:[/] reading from " + f"[bold]{scheme}://[/] requires the [bold]{pkg}[/] package.\n\n" + f"Install with:\n [bold]pip install {pkg}[/]", + title="Remote loader", + border_style="yellow", + ) + ) + raise + + # Cap on rows materialised from a remote URI — matches v0.24.0 + # ``soup data download --samples`` ceiling. Defends against OOM when a + # crafted / oversized bucket object is pointed at via streaming + + # eager-materialise. + max_remote_rows = 1_000_000 + + # Try the HF datasets streaming path first when the user opted in via + # ``data.streaming=true`` — gives us free interleaving, shuffling, and + # caching. Falls back to direct fsspec.open when datasets is missing or + # rejects the URI. + if data_config.streaming: + try: + from datasets import load_dataset as hf_load + except ImportError as exc: + raise ImportError( + "data.streaming=true requires the 'datasets' package: " + "pip install datasets" + ) from exc + ds = hf_load( + "json", + data_files=canonical, + split="train", + streaming=True, + ) + buf = data_config.buffer_size + if buf: + ds = ds.shuffle(buffer_size=buf) + # Eager materialise capped at max_remote_rows — emit a clear advisory + # if the cap trips. + raw_data: list[dict] = [] + for i, row in enumerate(ds): + if i >= max_remote_rows: + console.print( + f"[yellow]Remote dataset truncated at {max_remote_rows:,} " + f"rows (use a local split for larger jobs).[/]" + ) + break + raw_data.append(row) + else: + # Non-streaming: open once, read lines, decode JSON. + raw_data = [] + with fsspec.open(canonical, mode="rt", encoding="utf-8-sig") as fh: + for i, raw_line in enumerate(fh): + if i >= max_remote_rows: + console.print( + f"[yellow]Remote dataset truncated at " + f"{max_remote_rows:,} rows.[/]" + ) + break + stripped = raw_line.strip() + if not stripped: + continue + try: + raw_data.append(json.loads(stripped)) + except json.JSONDecodeError as exc: + console.print( + f"[yellow]Warning: invalid JSON on line " + f"{i + 1}: {exc}[/]" + ) + + fmt = data_config.format + if fmt == "auto": + fmt = detect_format(raw_data) + console.print(f"[dim]Auto-detected format: {fmt}[/]") + + formatted = [format_to_messages(row, fmt) for row in raw_data] + formatted = [r for r in formatted if r is not None] + + if data_config.val_split > 0: + split_idx = int(len(formatted) * (1 - data_config.val_split)) + return { + "train": formatted[:split_idx], + "val": formatted[split_idx:], + } + return {"train": formatted} + + def _load_hf_dataset(name: str, data_config: DataConfig) -> dict: """Load a dataset from HuggingFace Hub.""" try: diff --git a/soup_cli/utils/demo_bundles.py b/soup_cli/utils/demo_bundles.py index 6d344c2..51c90c5 100644 --- a/soup_cli/utils/demo_bundles.py +++ b/soup_cli/utils/demo_bundles.py @@ -89,7 +89,12 @@ def get_bundle(name: str) -> DemoBundle: def _bundle_source_path(bundle: DemoBundle) -> str: """Resolve the on-disk path for a bundle's fixture. - Uses importlib.resources to handle both editable + wheel installs. + v0.53.8 #93 — fixtures live under ``soup_cli/data/_fixtures/`` (package + data), with a back-compat fallback to ``examples/data/`` for editable + installs / repo-root invocations. The package-data location is + zipapp / namespace-package safe; the legacy location is kept so + contributors editing fixtures via ``examples/data/`` still see their + changes. """ # Filenames are baked-in constants (no user input), so direct join # is safe; we still defensively reject path separators. @@ -97,17 +102,23 @@ def _bundle_source_path(bundle: DemoBundle) -> str: raise ValueError( f"bundle fixture name has separator: {bundle.fixture!r}" ) - # examples/ lives at the repo root, alongside the soup_cli/ package. + # 1) Preferred — package data at soup_cli/data/_fixtures/. pkg_root = files("soup_cli") + pkg_candidate = os.path.realpath( + os.path.join(str(pkg_root), "data", "_fixtures", bundle.fixture) + ) + if os.path.isfile(pkg_candidate): + return pkg_candidate + # 2) Fallback — legacy examples/data/ at repo root. repo_root = os.path.dirname(str(pkg_root)) - candidate = os.path.realpath( + legacy = os.path.realpath( os.path.join(repo_root, "examples", "data", bundle.fixture) ) - if not os.path.isfile(candidate): - raise FileNotFoundError( - f"bundle fixture missing: {bundle.fixture}" - ) - return candidate + if os.path.isfile(legacy): + return legacy + raise FileNotFoundError( + f"bundle fixture missing: {bundle.fixture}" + ) def copy_bundle_to(name: str, output_path: str) -> str: diff --git a/soup_cli/utils/hf_space.py b/soup_cli/utils/hf_space.py index b04539b..d63a669 100644 --- a/soup_cli/utils/hf_space.py +++ b/soup_cli/utils/hf_space.py @@ -28,6 +28,66 @@ _MAX_TEMPLATE_FILE_BYTES = 256 * 1024 # 256 KB _KNOWN_FILES = ("app.py", "README.md", "requirements.txt") _REQUIRED_FILES = ("app.py", "README.md") +# v0.53.8 #69 — Auto-pick Space SDK from requirements.txt. +# Reads the rendered requirements.txt content and returns one of the HF +# Hub-supported `space_sdk` strings. Closes the v0.40.2 known limitation +# that all custom templates defaulted to `space_sdk="gradio"`. + +_SUPPORTED_SDKS = ("gradio", "streamlit", "docker", "static") + + +def detect_space_sdk(requirements_text: str | None) -> str: + """Detect the Space SDK from a requirements.txt body. + + Returns ``"streamlit"`` when the rendered ``requirements.txt`` lists + ``streamlit``, ``"gradio"`` when it lists ``gradio`` (or in any other + case — the project default for chat templates). + + The HF Hub Space-SDK allowlist also recognises ``"docker"`` and + ``"static"``, but those cannot be inferred from ``requirements.txt`` + alone — operators wanting those SDKs must use the built-in templates + or extend this helper. + + Rules (first-match-wins on a case-insensitive scan of dependency names, + after stripping ``#`` comments / extras ``[all]`` / version specifiers + ``>=1.0``): + - any line whose base name is ``streamlit`` → ``"streamlit"`` + - any line whose base name is ``gradio`` → ``"gradio"`` + - else → ``"gradio"`` + + Input is capped at 256 KB to defend against pathological requirements + files (matches the v0.39.0 Part E template size policy). + """ + if requirements_text is None: + return "gradio" + if not isinstance(requirements_text, str): + # Defence-in-depth — caller shouldn't pass non-str but be safe. + return "gradio" + if len(requirements_text) > _MAX_TEMPLATE_FILE_BYTES: + # Refuse to scan oversized text — degrade to the default. + return "gradio" + for raw_line in requirements_text.splitlines(): + line = raw_line.strip().lower() + if not line or line.startswith("#"): + continue + # Strip version specifiers / extras: "streamlit>=1.0" → "streamlit" + token = line + for sep in ("=", ">", "<", "!", "~", "[", " ", ";"): + idx = token.find(sep) + if idx >= 0: + token = token[:idx] + token = token.strip() + if token == "streamlit": + return "streamlit" + if token == "gradio": + return "gradio" + return "gradio" + + +def is_supported_space_sdk(sdk: str) -> bool: + """Whether ``sdk`` is in the HF Hub Space-SDK allowlist.""" + return isinstance(sdk, str) and sdk in _SUPPORTED_SDKS + def render_custom_template_dir(template_dir: str, model_repo: str) -> dict[str, str]: """Render a custom Space template directory. diff --git a/soup_cli/utils/hubs.py b/soup_cli/utils/hubs.py index 631e7de..0910264 100644 --- a/soup_cli/utils/hubs.py +++ b/soup_cli/utils/hubs.py @@ -204,3 +204,243 @@ def is_hf(hub: str) -> bool: if not isinstance(hub, str): return False return hub.lower() == "hf" + + +# v0.53.8 #130 — Live download / upload dispatcher. +# Each backend lazy-imports its SDK so a missing optional dep only surfaces +# when the user actually selects that hub. Mirrors v0.51.0 stub-then-live +# pattern: schema (TrainingConfig.hub Literal) shipped v0.51.0; live wiring +# ships now. + +_REPO_ID_MAX = 200 + + +def _validate_repo_id_shape(repo_id: str) -> str: + """Cheap shape-only repo-id validator shared by all hub adapters. + + Does NOT mirror the full v0.29.0 HF ``validate_repo_id`` regex (which is + HF-specific). Each hub's SDK applies its own canonicalisation; we just + reject obviously dangerous shapes (null bytes, leading slash, ``..``, + oversize) before forwarding. + """ + if isinstance(repo_id, bool): + raise TypeError(f"repo_id must not be bool, got {repo_id!r}") + if not isinstance(repo_id, str): + raise TypeError( + f"repo_id must be str, got {type(repo_id).__name__}" + ) + if not repo_id: + raise ValueError("repo_id must be non-empty") + if "\x00" in repo_id: + raise ValueError("repo_id must not contain null bytes") + if len(repo_id) > _REPO_ID_MAX: + raise ValueError( + f"repo_id too long (max {_REPO_ID_MAX} chars)" + ) + if repo_id.startswith("/") or repo_id.startswith("\\"): + raise ValueError("repo_id must not start with a path separator") + if ".." in repo_id.split("/"): + raise ValueError("repo_id must not contain '..' segments") + # Defence-in-depth: control chars (incl. CR / LF) would be a header + # injection hazard if the id ever flowed into an HTTP request line. + if any(ord(c) < 0x20 for c in repo_id): + raise ValueError("repo_id must not contain control characters") + return repo_id + + +def _missing_dep_message(hub: str) -> str: + """Friendly ImportError message naming the pip install command.""" + pkg = required_hub_package(hub) or hub + return ( + f"hub={hub!r} requires the '{pkg}' package. " + f"Install with: pip install {pkg}" + ) + + +def _validate_local_path(value: str, *, field: str) -> str: + """Cwd-containment + shape check for ``local_dir`` / ``folder_path``. + + Mirrors the project-standard ``utils.paths.is_under_cwd`` policy used by + every other path-accepting helper since v0.26.0. Rejects bool BEFORE + `isinstance(str)` (matches v0.30.0 ``Candidate`` policy). + """ + from soup_cli.utils.paths import is_under_cwd + + if isinstance(value, bool): + raise TypeError(f"{field} must not be bool, got {value!r}") + if not isinstance(value, str): + raise TypeError( + f"{field} must be str, got {type(value).__name__}" + ) + if not value: + raise ValueError(f"{field} must be a non-empty string") + if "\x00" in value: + raise ValueError(f"{field} must not contain null bytes") + if not is_under_cwd(value): + raise ValueError( + f"{field} must stay under the current working directory" + ) + return value + + +def download_repo( + hub: str, + repo_id: str, + *, + local_dir: str, + revision: str | None = None, + allow_patterns: list[str] | None = None, + repo_type: str = "model", +) -> str: + """Snapshot-download ``repo_id`` from ``hub`` into ``local_dir``. + + Returns the absolute local path to the downloaded snapshot. Lazy-imports + the appropriate SDK per ``hub``: + + * ``hf`` → :func:`huggingface_hub.snapshot_download` + * ``modelscope`` → :func:`modelscope.snapshot_download` + * ``modelers`` → :func:`openmind_hub.snapshot_download` + + Raises ``ImportError`` (with pip-install hint) when the SDK is missing, + ``ValueError`` for invalid args, ``TypeError`` for wrong types. + """ + canonical = validate_hub_name(hub) + _validate_repo_id_shape(repo_id) + _validate_local_path(local_dir, field="local_dir") + if revision is not None: + if not isinstance(revision, str): + raise TypeError("revision must be str or None") + if "\x00" in revision or any(ord(c) < 0x20 for c in revision): + raise ValueError("revision must not contain control characters") + if repo_type not in ("model", "dataset", "space"): + raise ValueError( + "repo_type must be one of 'model' / 'dataset' / 'space'" + ) + + if canonical == "hf": + try: + from huggingface_hub import snapshot_download + except ImportError as exc: + raise ImportError(_missing_dep_message("hf")) from exc + return snapshot_download( + repo_id=repo_id, + repo_type=repo_type, + revision=revision, + local_dir=local_dir, + allow_patterns=allow_patterns, + ) + + if canonical == "modelscope": + try: + from modelscope import ( + snapshot_download as ms_download, # type: ignore[import-not-found] + ) + except ImportError as exc: + raise ImportError(_missing_dep_message("modelscope")) from exc + # modelscope's snapshot_download uses a different kwarg set; we map + # the canonical args here so callers see one consistent API. + ms_kwargs: dict[str, object] = { + "model_id": repo_id, + "cache_dir": local_dir, + } + if revision is not None: + ms_kwargs["revision"] = revision + if allow_patterns is not None: + ms_kwargs["allow_file_pattern"] = allow_patterns + return ms_download(**ms_kwargs) + + if canonical == "modelers": + try: + from openmind_hub import ( + snapshot_download as om_download, # type: ignore[import-not-found] + ) + except ImportError as exc: + raise ImportError(_missing_dep_message("modelers")) from exc + om_kwargs: dict[str, object] = { + "repo_id": repo_id, + "local_dir": local_dir, + } + if revision is not None: + om_kwargs["revision"] = revision + if allow_patterns is not None: + om_kwargs["allow_patterns"] = allow_patterns + return om_download(**om_kwargs) + + # Unreachable — validate_hub_name has already rejected unknown hubs. + raise ValueError(f"hub {canonical!r} has no download adapter") + + +def upload_repo( + hub: str, + repo_id: str, + *, + folder_path: str, + commit_message: str = "Upload via Soup", + token: str | None = None, + repo_type: str = "model", +) -> None: + """Upload ``folder_path`` to ``repo_id`` on ``hub``. + + Same lazy-import policy as :func:`download_repo`. Token resolution is + left to the caller (each backend has its own conventions); pass + ``token`` explicitly or rely on the SDK's env-var defaults. + """ + canonical = validate_hub_name(hub) + _validate_repo_id_shape(repo_id) + _validate_local_path(folder_path, field="folder_path") + if not isinstance(commit_message, str) or not commit_message: + raise ValueError("commit_message must be a non-empty string") + # Mirror v0.29.0 push policy: first line only, ≤200 chars (prevents + # multi-line injection into public commit history). + commit_message = commit_message.splitlines()[0][:200] + if repo_type not in ("model", "dataset", "space"): + raise ValueError( + "repo_type must be one of 'model' / 'dataset' / 'space'" + ) + + if canonical == "hf": + try: + from huggingface_hub import HfApi + except ImportError as exc: + raise ImportError(_missing_dep_message("hf")) from exc + api = HfApi(token=token) + api.upload_folder( + repo_id=repo_id, + folder_path=folder_path, + repo_type=repo_type, + commit_message=commit_message, + ) + return + + if canonical == "modelscope": + try: + from modelscope.hub.api import HubApi # type: ignore[import-not-found] + except ImportError as exc: + raise ImportError(_missing_dep_message("modelscope")) from exc + api = HubApi() + if token: + api.login(token) + # ModelScope's `push_model` does not accept `commit_message` — pass + # only the model id + dir. The sanitised commit_message is recorded + # in the operator's local git log via the HF/Modelers backends. + api.push_model( + model_id=repo_id, + model_dir=folder_path, + ) + return + + if canonical == "modelers": + try: + from openmind_hub import HubApi # type: ignore[import-not-found] + except ImportError as exc: + raise ImportError(_missing_dep_message("modelers")) from exc + api = HubApi(token=token) + api.upload_folder( + repo_id=repo_id, + folder_path=folder_path, + repo_type=repo_type, + commit_message=commit_message, + ) + return + + raise ValueError(f"hub {canonical!r} has no upload adapter") diff --git a/soup_cli/utils/trackers.py b/soup_cli/utils/trackers.py index e459e9a..8791814 100644 --- a/soup_cli/utils/trackers.py +++ b/soup_cli/utils/trackers.py @@ -157,6 +157,139 @@ def build_telemetry_payload( } +# v0.53.8 #90 — PostHog telemetry live wiring. +# Opt-IN via SOUP_TELEMETRY=1; silent-fail on any network/transport error +# so telemetry can NEVER crash training. 1s hard timeout, HTTPS-only. + +_POSTHOG_HOST = "https://us.i.posthog.com" +_POSTHOG_ENDPOINT = f"{_POSTHOG_HOST}/i/v0/e/" +# Public write-only key. Live deployments will swap this via env var. +_POSTHOG_DEFAULT_KEY = "phc_soup_public_write_only" +_TELEMETRY_TIMEOUT_S = 1.0 + + +def _telemetry_endpoint_is_safe(endpoint: str) -> bool: + """Re-validate the telemetry endpoint via the v0.51.0 SSRF policy. + + Even though :func:`send_telemetry_payload` only POSTs to a static + PostHog URL by default, callers can override ``endpoint``. Re-run the + same private-IP / link-local rejection used for hub endpoints so a + crafted ``endpoint='https://10.0.0.1/'`` cannot reach an internal + network from a misconfigured caller. + """ + if not isinstance(endpoint, str) or not endpoint.startswith("https://"): + return False + try: + from soup_cli.utils.hubs import validate_hub_endpoint + + validate_hub_endpoint(endpoint, hub="telemetry") + except (TypeError, ValueError): + return False + return True + + +def send_telemetry_payload( + payload: dict[str, object], + *, + api_key: str | None = None, + timeout: float = _TELEMETRY_TIMEOUT_S, + endpoint: str = _POSTHOG_ENDPOINT, +) -> bool: + """POST ``payload`` to PostHog if telemetry is enabled, else no-op. + + Returns ``True`` on a 2xx response, ``False`` on any failure or skip. + NEVER raises — telemetry is best-effort and must never crash training. + + Args: + payload: dict built by :func:`build_telemetry_payload`. Required keys + are validated upstream by the builder. + api_key: PostHog project key. Defaults to the bundled write-only key. + timeout: hard wall-clock cap (default 1 s). + endpoint: full PostHog capture URL (must be HTTPS). + """ + if not is_telemetry_enabled(): + return False + if not isinstance(payload, dict) or not payload: + return False + # HTTPS-only + private-IP / link-local rejection (mirrors v0.51.0 hub + # endpoint SSRF policy). Defence-in-depth: any caller override goes + # through the same validator that hub endpoints do. + if not _telemetry_endpoint_is_safe(endpoint): + return False + if isinstance(timeout, bool) or not isinstance(timeout, (int, float)): + return False + if not math.isfinite(float(timeout)) or timeout <= 0: + return False + key = api_key or _POSTHOG_DEFAULT_KEY + if not isinstance(key, str) or not key: + return False + try: + import httpx # lazy — optional dep, surfaces no advisory + except ImportError: + return False + body = { + "api_key": key, + "event": payload.get("command", "soup_event"), + "properties": {k: v for k, v in payload.items() if k != "command"}, + } + try: + resp = httpx.post(endpoint, json=body, timeout=timeout) + return 200 <= resp.status_code < 300 + except Exception: # noqa: BLE001 — telemetry must never crash training + return False + + +# v0.53.8 #89 — Friendly missing-dep panel for HF Trainer `--tracker`. +# When user passes `--tracker mlflow` without mlflow installed, HF raises +# a generic ImportError mid-training; this helper lets the CLI surface a +# pip-install advisory BEFORE construction. + + +def tracker_missing_dep_message(name: str) -> str | None: + """Return a friendly install advisory for ``name`` if the package is + missing, else None. + + Always returns ``None`` for `wandb` / `tensorboard` / `none` (the + legacy backends), since those are part of the standard HF Trainer + extra and not part of v0.43.0's additive set. + """ + if not isinstance(name, str): + return None + canonical = name.lower() + if canonical not in NEW_TRACKERS_V0_43: + return None + pkg = required_tracker_package(canonical) + if not pkg: + return None + # Use ``importlib.util.find_spec`` (non-executing probe) so we don't + # incur side effects from the tracker's top-level module (e.g. swanlab + # initialises network threads on import). ``sys.modules[pkg] = None`` + # raises ``ValueError`` on find_spec — treat that as missing too so + # tests can simulate the absent-package path without subprocess. + import importlib.util + import sys + + sentinel = object() + cached = sys.modules.get(pkg, sentinel) + if cached is None: + missing = True + elif cached is not sentinel: + # Module is already imported (or test injected a real-shaped mock). + missing = False + else: + try: + missing = importlib.util.find_spec(pkg) is None + except (ImportError, ValueError): + missing = True + if missing: + return ( + f"--tracker {canonical} requires the '{pkg}' package. " + f"Install with: pip install soup-cli[trackers] " + f"(or pip install {pkg})" + ) + return None + + def resolve_report_to( *, wandb: bool = False, diff --git a/tests/test_v0537.py b/tests/test_v0537.py index bd4d106..6f3d7d1 100644 --- a/tests/test_v0537.py +++ b/tests/test_v0537.py @@ -952,7 +952,7 @@ class TestToolEndpointsLive: json={"command": "echo hello"}, ) assert resp.status_code == 501 - assert "v0.53.8" in resp.text + assert "v0.53.9" in resp.text def test_bash_tool_returns_501_with_empty_body(self): """The 501 stub does not parse the body — it always returns 501.""" @@ -1609,7 +1609,7 @@ class TestReviewFixesC1BashStub: client = TestClient(app) resp = client.post("/v1/tools/bash", json={"command": "ls"}) assert resp.status_code == 501 - assert "v0.53.8" in resp.text + assert "v0.53.9" in resp.text class TestReviewFixesAuthToken: diff --git a/tests/test_v0538.py b/tests/test_v0538.py new file mode 100644 index 0000000..582dbb7 --- /dev/null +++ b/tests/test_v0538.py @@ -0,0 +1,583 @@ +"""v0.53.8 — Remote data + Hubs + Trackers (wave 2). + +Tests cover: +* #85 — fsspec live loaders (data/loader.py) +* #130 — Live hub download/upload dispatcher (utils/hubs.py) +* #89 — `[trackers]` extra + missing-dep advisory +* #90 — PostHog telemetry network (best-effort, silent-fail) +* #93 — package-data migration (soup_cli/data/_fixtures/) +* #69 — HF Space SDK auto-pick from requirements.txt +""" +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +# ---------------------------------------------------------------------- +# #69 — HF Space SDK auto-pick +# ---------------------------------------------------------------------- + + +class TestDetectSpaceSdk: + def _import(self): + from soup_cli.utils.hf_space import detect_space_sdk + + return detect_space_sdk + + def test_none_defaults_to_gradio(self): + assert self._import()(None) == "gradio" + + def test_empty_defaults_to_gradio(self): + assert self._import()("") == "gradio" + + def test_streamlit_picked(self): + assert self._import()("streamlit>=1.0\nrequests") == "streamlit" + + def test_gradio_picked(self): + assert self._import()("gradio==4.0") == "gradio" + + def test_streamlit_wins_when_first(self): + # First match wins + assert self._import()("streamlit\ngradio") == "streamlit" + + def test_comment_ignored(self): + assert self._import()("# streamlit\ngradio") == "gradio" + + def test_no_match_defaults_gradio(self): + assert self._import()("torch\nnumpy") == "gradio" + + def test_non_string_defaults_gradio(self): + assert self._import()(123) == "gradio" # type: ignore[arg-type] + + def test_with_extras(self): + assert self._import()("streamlit[all]>=1.0") == "streamlit" + + def test_uppercase_normalised(self): + assert self._import()("STREAMLIT==1.0") == "streamlit" + + def test_oversize_input_degrades_to_default(self): + # security-review LOW — size cap defends against pathological input + huge = "streamlit\n" + ("x" * (260 * 1024)) + assert self._import()(huge) == "gradio" + + +class TestIsSupportedSpaceSdk: + def test_known(self): + from soup_cli.utils.hf_space import is_supported_space_sdk + + assert is_supported_space_sdk("gradio") + assert is_supported_space_sdk("streamlit") + assert is_supported_space_sdk("docker") + assert is_supported_space_sdk("static") + + def test_unknown(self): + from soup_cli.utils.hf_space import is_supported_space_sdk + + assert not is_supported_space_sdk("flask") + assert not is_supported_space_sdk("") + assert not is_supported_space_sdk(None) # type: ignore[arg-type] + + +# ---------------------------------------------------------------------- +# #130 — Hub download / upload dispatcher +# ---------------------------------------------------------------------- + + +class TestDownloadRepoValidation: + def test_unknown_hub_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="not supported"): + download_repo("evil_hub", "owner/repo", local_dir="./snap_v0538") + + def test_empty_repo_id_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="non-empty"): + download_repo("hf", "", local_dir="./snap_v0538") + + def test_null_byte_repo_id_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="null bytes"): + download_repo("hf", "owner\x00/repo", local_dir="./snap_v0538") + + def test_bool_repo_id_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(TypeError, match="bool"): + download_repo("hf", True, local_dir="./snap_v0538") # type: ignore[arg-type] + + def test_leading_slash_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="path separator"): + download_repo("hf", "/etc/passwd", local_dir="./snap_v0538") + + def test_dotdot_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match=".."): + download_repo("hf", "../escape", local_dir="./snap_v0538") + + def test_control_char_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="control"): + download_repo("hf", "owner/repo\n", local_dir="./snap_v0538") + + def test_oversize_repo_id_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="too long"): + download_repo("hf", "a" * 250, local_dir="./snap_v0538") + + def test_empty_local_dir_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="local_dir"): + download_repo("hf", "owner/repo", local_dir="") + + def test_local_dir_outside_cwd_raises(self, tmp_path): + # security-review HIGH — containment check on local_dir + from soup_cli.utils.hubs import download_repo + + # tmp_path is a sibling of cwd, not under it + out = str(tmp_path / "snap") + with pytest.raises(ValueError, match="under the current working"): + download_repo("hf", "owner/repo", local_dir=out) + + def test_local_dir_bool_rejected(self): + # security-review LOW — bool before str check + from soup_cli.utils.hubs import download_repo + + with pytest.raises(TypeError, match="bool"): + download_repo("hf", "owner/repo", local_dir=True) # type: ignore[arg-type] + + def test_bad_repo_type_raises(self): + from soup_cli.utils.hubs import download_repo + + with pytest.raises(ValueError, match="repo_type"): + download_repo("hf", "owner/repo", local_dir="./snap_v0538", repo_type="evil") + + +class TestDownloadRepoLazyImport: + def test_modelscope_missing_friendly_error(self, monkeypatch): + from soup_cli.utils import hubs + + # Block modelscope import even if installed + monkeypatch.setitem(sys.modules, "modelscope", None) + with pytest.raises(ImportError, match="modelscope"): + hubs.download_repo("modelscope", "owner/repo", local_dir="./snap_v0538") + + def test_modelers_missing_friendly_error(self, monkeypatch): + from soup_cli.utils import hubs + + monkeypatch.setitem(sys.modules, "openmind_hub", None) + with pytest.raises(ImportError, match="openmind-hub"): + hubs.download_repo("modelers", "owner/repo", local_dir="./snap_v0538") + + def test_hf_dispatch_calls_snapshot_download(self): + with patch( + "huggingface_hub.snapshot_download", return_value="/local/snap" + ) as mocked: + from soup_cli.utils.hubs import download_repo + + result = download_repo("hf", "owner/repo", local_dir="./snap_v0538") + assert result == "/local/snap" + mocked.assert_called_once() + kwargs = mocked.call_args.kwargs + assert kwargs["repo_id"] == "owner/repo" + assert kwargs["local_dir"] == "./snap_v0538" + + +class TestUploadRepoValidation: + def test_unknown_hub_raises(self): + from soup_cli.utils.hubs import upload_repo + + with pytest.raises(ValueError, match="not supported"): + upload_repo("evil", "o/r", folder_path="./folder_v0538") + + def test_empty_folder_raises(self): + from soup_cli.utils.hubs import upload_repo + + with pytest.raises(ValueError, match="folder_path"): + upload_repo("hf", "o/r", folder_path="") + + def test_folder_path_outside_cwd_raises(self, tmp_path): + # security-review HIGH — containment check on folder_path + from soup_cli.utils.hubs import upload_repo + + out = str(tmp_path / "out") + with pytest.raises(ValueError, match="under the current working"): + upload_repo("hf", "o/r", folder_path=out) + + def test_empty_commit_raises(self): + from soup_cli.utils.hubs import upload_repo + + with pytest.raises(ValueError, match="commit_message"): + upload_repo("hf", "o/r", folder_path="./folder_v0538", commit_message="") + + def test_commit_message_truncated(self): + # First line + 200 char cap (mirrors v0.29.0 push policy) + with patch("huggingface_hub.HfApi") as mock_api_cls: + mock_api = MagicMock() + mock_api_cls.return_value = mock_api + from soup_cli.utils.hubs import upload_repo + + long_msg = "line1\nline2" + "x" * 500 + upload_repo("hf", "o/r", folder_path="./folder_v0538", commit_message=long_msg) + sent = mock_api.upload_folder.call_args.kwargs["commit_message"] + assert "\n" not in sent + assert len(sent) <= 200 + + +# ---------------------------------------------------------------------- +# #90 — PostHog telemetry network +# ---------------------------------------------------------------------- + + +class TestSendTelemetryPayload: + def test_disabled_short_circuits(self, monkeypatch): + monkeypatch.delenv("SOUP_TELEMETRY", raising=False) + from soup_cli.utils.trackers import send_telemetry_payload + + assert send_telemetry_payload({"command": "train"}) is False + + def test_empty_payload_returns_false(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + from soup_cli.utils.trackers import send_telemetry_payload + + assert send_telemetry_payload({}) is False + + def test_non_dict_returns_false(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + from soup_cli.utils.trackers import send_telemetry_payload + + assert send_telemetry_payload("not a dict") is False # type: ignore[arg-type] + + def test_http_endpoint_rejected(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + from soup_cli.utils.trackers import send_telemetry_payload + + assert ( + send_telemetry_payload( + {"command": "train"}, endpoint="http://evil.example/i/" + ) + is False + ) + + def test_private_ip_endpoint_rejected(self, monkeypatch): + # security-review MEDIUM — SSRF via private IP override + monkeypatch.setenv("SOUP_TELEMETRY", "1") + from soup_cli.utils.trackers import send_telemetry_payload + + assert ( + send_telemetry_payload( + {"command": "train"}, endpoint="https://10.0.0.1/i/" + ) + is False + ) + + def test_link_local_endpoint_rejected(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + from soup_cli.utils.trackers import send_telemetry_payload + + assert ( + send_telemetry_payload( + {"command": "train"}, endpoint="https://169.254.169.254/i/" + ) + is False + ) + + def test_bool_timeout_rejected(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + from soup_cli.utils.trackers import send_telemetry_payload + + assert send_telemetry_payload({"command": "train"}, timeout=True) is False # type: ignore[arg-type] + + def test_negative_timeout_rejected(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + from soup_cli.utils.trackers import send_telemetry_payload + + assert send_telemetry_payload({"command": "train"}, timeout=-1) is False + + def test_httpx_missing_returns_false(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + # Force ImportError on the lazy import + monkeypatch.setitem(sys.modules, "httpx", None) + from soup_cli.utils.trackers import send_telemetry_payload + + assert send_telemetry_payload({"command": "train"}) is False + + def test_happy_path_2xx(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + + fake_httpx = MagicMock() + fake_resp = MagicMock() + fake_resp.status_code = 200 + fake_httpx.post.return_value = fake_resp + monkeypatch.setitem(sys.modules, "httpx", fake_httpx) + + from soup_cli.utils.trackers import send_telemetry_payload + + result = send_telemetry_payload({"command": "train", "soup_version": "x"}) + assert result is True + # HTTPS-only + 1s timeout + call = fake_httpx.post.call_args + assert call.args[0].startswith("https://") + assert call.kwargs["timeout"] == 1.0 + + def test_network_exception_swallowed(self, monkeypatch): + monkeypatch.setenv("SOUP_TELEMETRY", "1") + + fake_httpx = MagicMock() + fake_httpx.post.side_effect = RuntimeError("network down") + monkeypatch.setitem(sys.modules, "httpx", fake_httpx) + + from soup_cli.utils.trackers import send_telemetry_payload + + # Must never raise + assert send_telemetry_payload({"command": "train"}) is False + + +# ---------------------------------------------------------------------- +# #89 — Tracker missing-dep advisory +# ---------------------------------------------------------------------- + + +class TestTrackerMissingDepMessage: + def test_wandb_returns_none(self): + from soup_cli.utils.trackers import tracker_missing_dep_message + + # Legacy backend — never advise + assert tracker_missing_dep_message("wandb") is None + + def test_tensorboard_returns_none(self): + from soup_cli.utils.trackers import tracker_missing_dep_message + + assert tracker_missing_dep_message("tensorboard") is None + + def test_unknown_returns_none(self): + from soup_cli.utils.trackers import tracker_missing_dep_message + + assert tracker_missing_dep_message("evil") is None + + def test_non_string_returns_none(self): + from soup_cli.utils.trackers import tracker_missing_dep_message + + assert tracker_missing_dep_message(123) is None # type: ignore[arg-type] + + def test_missing_mlflow_advisory(self, monkeypatch): + monkeypatch.setitem(sys.modules, "mlflow", None) + from soup_cli.utils.trackers import tracker_missing_dep_message + + msg = tracker_missing_dep_message("mlflow") + assert msg is not None + assert "mlflow" in msg + assert "soup-cli[trackers]" in msg + + def test_present_mlflow_returns_none(self, monkeypatch): + # Pretend mlflow is installed + fake = MagicMock() + monkeypatch.setitem(sys.modules, "mlflow", fake) + from soup_cli.utils.trackers import tracker_missing_dep_message + + assert tracker_missing_dep_message("mlflow") is None + + +# ---------------------------------------------------------------------- +# #85 — fsspec live remote loader +# ---------------------------------------------------------------------- + + +class TestRemoteLoader: + def test_looks_like_remote_uri(self): + from soup_cli.data.loader import _looks_like_remote_uri + + assert _looks_like_remote_uri("s3://bucket/path") + assert _looks_like_remote_uri("gs://bucket/file.jsonl") + assert _looks_like_remote_uri("oci://bkt/x") + assert not _looks_like_remote_uri("local.jsonl") + assert not _looks_like_remote_uri("owner/dataset") + assert not _looks_like_remote_uri("") + assert not _looks_like_remote_uri(None) # type: ignore[arg-type] + + def test_fsspec_missing_raises_friendly(self, monkeypatch, tmp_path): + # Force fsspec ImportError + monkeypatch.setitem(sys.modules, "fsspec", None) + from soup_cli.config.schema import DataConfig + from soup_cli.data.loader import _load_remote_dataset + + cfg = DataConfig(train="s3://my-bucket/data.jsonl", format="alpaca", val_split=0) + with pytest.raises(ImportError): + _load_remote_dataset("s3://my-bucket/data.jsonl", cfg) + + def test_invalid_remote_uri_rejected(self, monkeypatch): + from soup_cli.config.schema import DataConfig + from soup_cli.data.loader import _load_remote_dataset + + cfg = DataConfig(train="s3://bucket/x", format="alpaca", val_split=0) + # Userinfo embedded URI should be rejected by validate_remote_uri + # BEFORE fsspec is even imported. + with pytest.raises(ValueError): + _load_remote_dataset("s3://user:pw@bucket/x", cfg) + + def test_non_streaming_reads_jsonl(self, monkeypatch, tmp_path): + from soup_cli.config.schema import DataConfig + + # Build a fake fsspec that yields two JSONL rows + rows = [ + '{"instruction": "hi", "output": "hello"}', + '{"instruction": "bye", "output": "later"}', + ] + + class FakeFile: + def __enter__(self): + return iter([r + "\n" for r in rows]) + + def __exit__(self, *a): + return False + + fake_fsspec = MagicMock() + fake_fsspec.open.return_value = FakeFile() + monkeypatch.setitem(sys.modules, "fsspec", fake_fsspec) + + from soup_cli.data.loader import _load_remote_dataset + + cfg = DataConfig( + train="s3://bucket/data.jsonl", format="alpaca", val_split=0 + ) + result = _load_remote_dataset("s3://bucket/data.jsonl", cfg) + assert "train" in result + assert len(result["train"]) == 2 + + +# ---------------------------------------------------------------------- +# #93 — Package-data fixture migration +# ---------------------------------------------------------------------- + + +class TestPackageDataFixtures: + def test_fixtures_dir_exists(self): + import soup_cli + + pkg = Path(soup_cli.__file__).parent + fixtures = pkg / "data" / "_fixtures" + assert fixtures.is_dir(), f"missing package-data fixtures dir: {fixtures}" + + def test_all_known_bundles_present(self): + import soup_cli + + pkg = Path(soup_cli.__file__).parent + fixtures = pkg / "data" / "_fixtures" + expected = { + "alpaca_tiny.jsonl", + "chat_preferences.jsonl", + "dpo_sample.jsonl", + "reasoning_math.jsonl", + } + present = {p.name for p in fixtures.glob("*.jsonl")} + assert expected.issubset(present) + + def test_bundle_source_prefers_package_data(self): + from soup_cli.utils.demo_bundles import _bundle_source_path, get_bundle + + bundle = get_bundle("alpaca_demo") + src = _bundle_source_path(bundle) + # Should resolve under soup_cli/data/_fixtures (not examples/data/). + assert os.sep + "_fixtures" + os.sep in src, src + + def test_bundle_content_valid_jsonl(self): + from soup_cli.utils.demo_bundles import _bundle_source_path, get_bundle + + for name in ("alpaca_demo", "sharegpt_demo", "dpo_demo", "grpo_demo"): + bundle = get_bundle(name) + src = _bundle_source_path(bundle) + with open(src, encoding="utf-8") as fh: + for line in fh: + if line.strip(): + json.loads(line) + + +# ---------------------------------------------------------------------- +# #130 wiring smoke (CLI flag plumbing) +# ---------------------------------------------------------------------- + + +class TestDataDownloadHubFlag: + def test_help_lists_hub_flag(self): + from typer.testing import CliRunner + + from soup_cli.commands.data import app + + result = CliRunner().invoke(app, ["download", "--help"]) + assert result.exit_code == 0 + assert "--hub" in result.output + + def test_unknown_hub_rejected(self): + from typer.testing import CliRunner + + from soup_cli.commands.data import app + + result = CliRunner().invoke( + app, ["download", "ds", "--hub", "evilcorp"] + ) + assert result.exit_code != 0 + assert "evilcorp" in result.output or "not supported" in result.output + + def test_modelscope_hub_advisory(self): + from typer.testing import CliRunner + + from soup_cli.commands.data import app + + result = CliRunner().invoke( + app, ["download", "ds", "--hub", "modelscope"] + ) + assert result.exit_code != 0 + assert "modelscope" in result.output + assert "v0.53.9" in result.output or "download_repo" in result.output + + +# ---------------------------------------------------------------------- +# pyproject extras +# ---------------------------------------------------------------------- + + +class TestPyprojectExtras: + def test_trackers_extra_present(self): + text = Path("pyproject.toml").read_text(encoding="utf-8") + assert "trackers = [" in text + assert "mlflow" in text + assert "swanlab" in text + assert "trackio" in text + + def test_remote_extra_present(self): + text = Path("pyproject.toml").read_text(encoding="utf-8") + assert "remote = [" in text + assert "fsspec" in text + + def test_force_include_package_data(self): + text = Path("pyproject.toml").read_text(encoding="utf-8") + assert "_fixtures" in text + + +# ---------------------------------------------------------------------- +# Version bump +# ---------------------------------------------------------------------- + + +class TestVersionBump: + def test_init_version(self): + import soup_cli + + assert soup_cli.__version__ == "0.53.8" + + def test_pyproject_version(self): + text = Path("pyproject.toml").read_text(encoding="utf-8") + assert 'version = "0.53.8"' in text