Commit Graph

5 Commits

Author SHA1 Message Date
Alpamys 06d8ea7cc5 chore: migrate to src-layout
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.

- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
  The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
  and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
  CODEOWNERS patterns, and the PR-template checklist all repointed to
  src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
  path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
  over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
  references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
  the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.

Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
2026-05-31 12:40:06 +05:00
Alpamys 6d2170c4f3 feat(v0.53.8): Remote data + Hubs + Trackers (wave 2) — 6 features
- #85 fsspec live loaders — data/loader.py routes the v0.42.0 fsspec
  scheme allowlist (s3:// / gs:// / gcs:// / az:// / abfs:// / abfss:// /
  oci://) through fsspec.open with validate_remote_uri containment
  BEFORE connection. Friendly Rich panel names the pip install
  advisory when the backend SDK is missing. Threads data.streaming
  + data.buffer_size. Row count capped at 1M.

- #130 Hub dispatcher live — utils/hubs.download_repo() and
  upload_repo() lazy-import per backend (huggingface_hub /
  modelscope / openmind_hub). Shared _validate_repo_id_shape (bool /
  null-byte / leading-slash / .. / control-char / oversize) + cwd
  containment on local_dir / folder_path. commands/train.py pre-fetches
  non-HF base into .soup_hub_cache/ (sanitised slug, idempotent on
  resume, cfg.base updated via model_copy). soup data download --hub
  flag plumbed. Multi-command rollout for chat / serve / infer / merge
  / export / push tracked for v0.53.9.

- #89 [trackers] pyproject extra bundles mlflow / swanlab / trackio;
  tracker_missing_dep_message surfaces a friendly pip install advisory
  via importlib.util.find_spec (non-executing probe).

- #90 utils/trackers.send_telemetry_payload — opt-IN via SOUP_TELEMETRY=1;
  lazy httpx; 1s hard timeout; HTTPS-only with SSRF re-validation
  (mirrors v0.51.0 hub endpoint policy); silent-fail on every exception.

- #93 Fixtures migrated to soup_cli/data/_fixtures/ — zipapp /
  namespace-package safe via [tool.hatch.build.targets.wheel.force-include];
  _bundle_source_path falls back to examples/data/ for editable installs.

- #69 utils/hf_space.detect_space_sdk(requirements_text) — picks
  "streamlit" / "gradio" from the rendered requirements.txt; closes
  the v0.40.2 known limitation that custom Spaces always defaulted to
  gradio. Wired into commands/deploy.py.

Review pass: python-review + code-review + security-review ran in
parallel; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). Highlights:
cwd-containment on local_dir/folder_path, Windows ..\ traversal
defence on .soup_hub_cache slug, Pydantic model_copy(update=...)
instead of attribute mutation, idempotent pre-fetch via cache probe,
1M-row cap on remote materialisation, SSRF re-validation on
telemetry endpoint override, 256 KB cap on detect_space_sdk input,
modelscope.push_model commit_message kwarg removed (would TypeError
at runtime), find_spec instead of __import__ to avoid swanlab
side-effects.

Test count: 8162 -> 8257 (+66 in tests/test_v0538.py + 29 net adjustments).
Lint clean. CPU smoke: version, --help, load_config_from_string with
hub: modelscope passes; mlx + non-HF rejected; data download --hub
modelscope advisory rendered; detect_space_sdk live on real
requirements.txt bodies; package-data fixtures resolve from
soup_cli/data/_fixtures/.

v0.53.7 known limitation #1 (bash 501 marker) bumped to v0.53.9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 23:55:23 +05:00
Alpamys a26511f868 fix(test): v0.53.7 — symlink test accepts either rejection branch
test_load_jsonl_rows_rejects_symlink: when the symlink target is outside
cwd, is_under_cwd's realpath resolution catches it before the lstat check
fires. Both rejections are valid security guards; broaden the regex to
match either error message ("symlink" or "under cwd").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 19:55:16 +05:00
Alpamys 9c9f962676 fix(test): v0.53.7 CI hardening — importorskip(fastapi) + Arrow target setup
CI run 25805598433 failed across all 9 OS×Python cells:
- TestToolEndpointsLive / TestAnthropicMessagesStreaming /
  TestReviewFixesVllmAnthropicLive ModuleNotFoundError: fastapi (CI does
  not install [serve] extra) → autouse fixture pytest.importorskip()
- test_load_pretokenized_dataset_rejects_symlink:
  load_pretokenized_dataset called datasets.load_from_disk on the symlink
  target before the lstat check ran → moved the lstat + S_ISLNK check
  to the entry of the helper so symlinks reject before any load attempt
- test_redact_exc_message_handles_windows_paths: hardened _redact_exc_message
  to strip both POSIX absolute paths and Windows-style paths regardless
  of host platform

Local pytest tests/test_v0537.py: 112 passed, 7 skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 19:45:21 +05:00
Alpamys 18d8b36114 feat(v0.53.7): Data Forge + Pipeline live (wave 1) — 11 items
Closes v0.47.0 deferrals (#111, #112), community QA (#75), v0.42.1 wave 1
(#87, #86, #88), and 5 v0.53.6 stub-to-live carry-overs (#102 vLLM parity
+ SSE streaming, #103 tool HTTP endpoints, #105 instantiate_trainer_plugins,
#106 run_recipe DAG runner).

- #88 markdown ingest heading split (split_markdown_by_headings)
- #112 soup data decontaminate --benchmark-file (cwd-contained operator
  corpus + symlink rejection)
- #87 prompt_strategy live resolver (resolve_prompt_strategy + lru_cache,
  importlib-based; per-row hook in sft_format.py)
- #86 soup data preprocess AOT tokenize (atomic Arrow shard write + cache
  metadata sidecar; SFT + Pretrain wrappers short-circuit on
  format='pre_tokenized' + tokenized_path with cache-hash gate)
- #111 forge --judge-provider {ollama,anthropic,vllm} live (lazy v0.20.0
  providers; SSRF parity)
- #75 QA log entry for synth-data provider manual smoke
- #106 run_recipe LIVE for 6 NODE_KINDS (seed / llm_text / code / judge /
  validator / sampler); atomic checkpoint via tempfile.mkstemp; resume
  rehydrates predecessor outputs from per-node sidecar JSONL; lstat-on-
  raw-path symlink rejection (v0.33.0 #22 TOCTOU parity); failed_reason
  path-redacted
- #105 instantiate_trainer_plugins LIVE for cce_plugin / grokfast /
  spectrum / llmcompressor / sonicmoe / math_verify (lazy imports,
  friendly pip-install advisory on missing dep)
- #103 POST /v1/tools/python + /v1/tools/web_search LIVE (Bearer auth gate,
  deny-by-default domain allowlist, 5s timeout, 5-result cap). bash
  reverted to HTTP 501 — security review caught /bin/sh -c child escapes
  RLVR sandbox's OS-level isolation; deferred to v0.53.8.
- #102 vLLM /v1/messages parity LIVE on both backends; CORS loopback-only
- #102 Anthropic-shape SSE streaming on /v1/messages LIVE; Cache-Control:
  no-store; CRLF/NUL/oversize strip on model+msg_id (header injection)

Review fixes (1 CRITICAL + 11 HIGH + 17 MEDIUM + 10 LOW from python-reviewer
+ code-reviewer + security-reviewer + tdd-guide) all addressed in this commit.

Test count: 8051 → 8162 (+111 in tests/test_v0537.py).
Test files: 189 → 190.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 19:28:11 +05:00