- #85 fsspec live loaders — data/loader.py routes the v0.42.0 fsspec
scheme allowlist (s3:// / gs:// / gcs:// / az:// / abfs:// / abfss:// /
oci://) through fsspec.open with validate_remote_uri containment
BEFORE connection. Friendly Rich panel names the pip install
advisory when the backend SDK is missing. Threads data.streaming
+ data.buffer_size. Row count capped at 1M.
- #130 Hub dispatcher live — utils/hubs.download_repo() and
upload_repo() lazy-import per backend (huggingface_hub /
modelscope / openmind_hub). Shared _validate_repo_id_shape (bool /
null-byte / leading-slash / .. / control-char / oversize) + cwd
containment on local_dir / folder_path. commands/train.py pre-fetches
non-HF base into .soup_hub_cache/ (sanitised slug, idempotent on
resume, cfg.base updated via model_copy). soup data download --hub
flag plumbed. Multi-command rollout for chat / serve / infer / merge
/ export / push tracked for v0.53.9.
- #89 [trackers] pyproject extra bundles mlflow / swanlab / trackio;
tracker_missing_dep_message surfaces a friendly pip install advisory
via importlib.util.find_spec (non-executing probe).
- #90 utils/trackers.send_telemetry_payload — opt-IN via SOUP_TELEMETRY=1;
lazy httpx; 1s hard timeout; HTTPS-only with SSRF re-validation
(mirrors v0.51.0 hub endpoint policy); silent-fail on every exception.
- #93 Fixtures migrated to soup_cli/data/_fixtures/ — zipapp /
namespace-package safe via [tool.hatch.build.targets.wheel.force-include];
_bundle_source_path falls back to examples/data/ for editable installs.
- #69 utils/hf_space.detect_space_sdk(requirements_text) — picks
"streamlit" / "gradio" from the rendered requirements.txt; closes
the v0.40.2 known limitation that custom Spaces always defaulted to
gradio. Wired into commands/deploy.py.
Review pass: python-review + code-review + security-review ran in
parallel; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). Highlights:
cwd-containment on local_dir/folder_path, Windows ..\ traversal
defence on .soup_hub_cache slug, Pydantic model_copy(update=...)
instead of attribute mutation, idempotent pre-fetch via cache probe,
1M-row cap on remote materialisation, SSRF re-validation on
telemetry endpoint override, 256 KB cap on detect_space_sdk input,
modelscope.push_model commit_message kwarg removed (would TypeError
at runtime), find_spec instead of __import__ to avoid swanlab
side-effects.
Test count: 8162 -> 8257 (+66 in tests/test_v0538.py + 29 net adjustments).
Lint clean. CPU smoke: version, --help, load_config_from_string with
hub: modelscope passes; mlx + non-HF rejected; data download --hub
modelscope advisory rendered; detect_space_sdk live on real
requirements.txt bodies; package-data fixtures resolve from
soup_cli/data/_fixtures/.
v0.53.7 known limitation #1 (bash 501 marker) bumped to v0.53.9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use AutoModel instead of AutoModelForCausalLM for audio-language models
(Qwen2-Audio, Whisper don't work with causal LM auto class)
- Use Path.is_relative_to() for path traversal check (symlink-safe, Python 3.9+)
- Fail fast with helpful error if librosa not installed before dataset processing
Add continued pre-training task and Mixture of Experts model support:
- `task: pretrain` for continued pre-training on raw text data
- `plaintext` data format ({"text": "..."} JSONL or .txt files)
- MoE model detection (Mixtral, Qwen3 MoE, DeepSeek V3, DBRX, OLMoE)
- ScatterMoE LoRA (`moe_lora: true`) targets expert FFN + attention layers
- `moe_aux_loss_coeff` for router load-balancing loss
- Templates: `soup init --template pretrain` and `--template moe`
- 85 new tests across test_pretrain.py and test_moe.py (1002 total)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add `modality: vision` config option for vision-language model training
- Add LLaVA and ShareGPT4V data format detection and conversion
- Add `image_dir` field in DataConfig for resolving image paths
- Add vision model loading via AutoModelForVision2Seq + AutoProcessor in SFT trainer
- Add `soup init --template vision` with LLaMA-3.2-Vision config
- Add image statistics display in `soup data inspect` for vision datasets
- Add Pillow as optional `vision` extra dependency
- Add Pillow to `soup doctor` dependency checks
- 51 new tests (455 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Typer CLI: soup init, soup train, soup data inspect/validate
- Pydantic config schema with YAML loader and validation
- Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets
- Format detection: Alpaca, ShareGPT, ChatML (auto-detect)
- SFT trainer wrapper over transformers + peft + trl
- QLoRA/LoRA support with auto batch size estimation
- GPU detection (CUDA/MPS/CPU) and memory calculation
- Rich live terminal dashboard for training monitoring
- Config templates: chat, code, medical
- Tests (pytest) + GitHub Actions CI
- MIT license
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alpamys