- Replace non-ASCII symbols (checkmarks, arrows, bullets, em-dashes)
with ASCII equivalents in Rich console output to prevent
UnicodeEncodeError on Windows without PYTHONIOENCODING=utf-8
- Add _validate_output_path() for AWQ/GPTQ export — output path
traversal is now checked before import check (previously unreachable
when autoawq/auto-gptq not installed)
- 4 new tests for output path validation (2065 total, 0 failures)
- Update SECURITY.md with v0.22.0–v0.24.1 hardening history
Part A: HuggingFace Dataset browser
- soup data search: search HF Hub for datasets (sort by downloads/likes)
- soup data preview: preview remote dataset metadata, splits, features
- soup data download: stream HF dataset to local JSONL (with format conversion)
- Security: trust_remote_code=False, path traversal protection, samples cap at 1M
Part B: Freeze training (like LLaMA-Factory finetuning_type: freeze)
- freeze_layers / freeze_ratio config fields
- soup_cli/utils/freeze.py: detect layers, freeze bottom N
- Wired into SFT trainer before LoRA application
- Supports LLaMA (layers.N) and GPT-2 (h.N) naming
Part C: Loss watchdog (like Axolotl loss_watchdog_threshold)
- loss_watchdog, loss_watchdog_threshold, loss_watchdog_patience config
- Implemented in SoupTrainerCallback with patience counter
- Rich warning panel (stops Live display first), fires only once
- Wired into all 11 trainers via callback kwargs
Part D: Dataset info registry
- soup data register/unregister/registry commands
- ~/.soup/datasets.json local name→path+format mapping
- Name validation, path traversal protection, Rich markup escaping
82 new tests (2061 total), 74 test files.
Rich/Typer truncates help panel on narrow terminals (macOS CI), causing
--bits and --group-size flags to not appear in rendered help text. Switch
to inspecting the function signature directly for cross-platform reliability.
- AWQ export (`soup export --format awq`) via autoawq, with --bits, --group-size, --calibration-data
- GPTQ export (`soup export --format gptq`) via auto-gptq, with calibration data support
- Sample packing (`packing: true`) for SFT/Pretrain trainers via TRL's native packing
- `soup data split` — train/val/test splitting with random and stratified strategies
- Curriculum learning (`curriculum: true`) — sort dataset by difficulty for staged training
- New utility: soup_cli/utils/curriculum.py (sort_by_length, create_buckets)
- Security: calibration data path traversal protection, bits validation (4/8 only)
- 1970 tests across 70 test files
- Extract _parse_json_array into soup_cli/data/providers/_utils.py to
avoid circular imports between generate.py and provider modules.
- Narrow bare except Exception in detect_ollama to httpx.HTTPError/OSError
with debug logging instead of silent swallow.
Replace simple '..' check with resolve() + relative_to(cwd) for output
path. Add same confinement guard to --seed, --dedup-with, and --context
file paths. Add _path_within_cwd helper. 4 new security tests.
Rich markup wraps --model-a with ANSI codes on macOS, breaking the
substring check. Strip ANSI codes before asserting, matching the
existing pattern in test_speculative_decoding.py and test_deploy_ollama.py.
Rich markup in Typer help output inserts ANSI escape codes around
--flag names on macOS, breaking exact string matches. Check for
lowercase words instead of --prefixed flags.
Rich inserts color codes between flag name parts (e.g. --speculative
becomes \x1b[1;36m-\x1b[0m\x1b[1;36m-speculative\x1b[0m), so plain
substring match fails in CI. Strip ANSI before asserting.
- Rename --spec-tokens to --num-speculative-tokens to avoid prefix
collision with --speculative-decoding in Typer help rendering
- Add pytest.skip for _create_app tests when fastapi is not installed
Typer returns exit code 2 (not 0) when no_args_is_help=True and no
arguments are provided. Fix test_no_args_shows_help and
test_data_no_args_shows_help to accept both 0 and 2.
- Check stdout encoding instead of type to detect non-UTF-8 consoles
- Redirect stdout to UTF-8 TextIOWrapper before plotext renders
- Add unit test that simulates cp1251 stdout with plotext
- 1022 tests total
- soup data validate: default --format changed from 'alpaca' to 'auto',
uses detect_format() to auto-detect dataset format
- soup data stats: force UTF-8 stdout on Windows for plotext histograms
- soup ui: add --show-token flag, document auth token in --help
- 7 new tests (BUG-013/014/015), 1021 tests total
Add continued pre-training task and Mixture of Experts model support:
- `task: pretrain` for continued pre-training on raw text data
- `plaintext` data format ({"text": "..."} JSONL or .txt files)
- MoE model detection (Mixtral, Qwen3 MoE, DeepSeek V3, DBRX, OLMoE)
- ScatterMoE LoRA (`moe_lora: true`) targets expert FFN + attention layers
- `moe_aux_loss_coeff` for router load-balancing loss
- Templates: `soup init --template pretrain` and `--template moe`
- 85 new tests across test_pretrain.py and test_moe.py (1002 total)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Test _load_model exit paths: adapter without base model, corrupt JSON
- Test _generate branches: greedy (temp=0), sampling (temp>0), no
chat_template fallback, token count from tensor shape, role formatting
- Test max_tokens bounds: 0 and 99999 rejected by CLI
- Test tensorboard happy path: flag accepted when tensorboard installed
- Fix import-failure test: avoid builtins.__import__ recursion
- 917 tests, 44 test files, 57.92% coverage
- Fix test_tensorboard_in_train_help: strip ANSI escape codes before
asserting --tensorboard in help output (Rich splits flag across
escape sequences on Python 3.11)
- Fix TensorBoard import check: use `import tensorboard` directly
- Stream JSONL output during inference (crash-safe for large files)
- Return accurate token count from _generate via tensor shape
- Replace shallow tests with real trainer integration tests
- Cap max_tokens at 16384 + trust_remote_code warning
- Fix TensorBoard import check: use `import tensorboard` directly
(not torch.utils.tensorboard shim) for accurate availability check
- Stream JSONL output during inference instead of buffering in memory
(crash-safe, handles large prompt files)
- Return accurate token count from _generate via tensor shape instead
of re-encoding decoded text
- Replace shallow tests with real trainer integration tests that
verify report_to='tensorboard' is accepted by all trainer wrappers
Security fixes across all HTTP surfaces:
- Web UI: Bearer token auth on mutating endpoints, CORS restricted to served origin,
path traversal protection on /api/data/inspect, config validated before training,
removed user-controlled config_path from API
- Serve/vLLM: max_tokens capped at 16384, generic error messages (no stack traces)
- Generate: SSRF protection (--api-base blocks non-HTTPS for remote URLs),
--api-key deprecated in favor of OPENAI_API_KEY env var
- Export: llama.cpp pinned to tag b5270 (supply-chain safety)
- Push: --token deprecated in favor of HF_TOKEN env var
- Rewards: warning before executing custom .py reward files
- Tests: all 40 UI tests updated with auth headers, 666 tests pass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GRPO:
- Set default chat_template on tokenizer when missing (fixes ValueError
from trl's apply_chat_template on models without chat support)
- Ensure batch_size >= num_generations (trl 0.28 requirement)
- Verified end-to-end GRPO training on CPU succeeds
PPO:
- Tokenize dataset via .map() before passing to PPOTrainer (adds input_ids
and attention_mask columns required by trl experimental API)
Tests: 666 passed, 5 new tests for chat_template/tokenization fixes,
3 existing mock tests updated for new tokenization step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: Skip resume_from_checkpoint when experimental PPOTrainer.train()
doesn't accept it (inspect signature at runtime, warn and proceed).
- CPU: Use device_map="cpu" instead of "auto" on CPU across all trainers
(SFT, DPO, GRPO, PPO, RewardModel) to prevent meta tensor errors.
- Add 12 new tests for both fixes (661 total passing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: Support trl >=0.28 experimental API (ref_model, reward_model,
train_dataset, value_model positional args). Auto-import from
trl.experimental.ppo with fallback. Create reward/value models when needed.
- GRPO: Fix CPU empty generation tensor mismatch by passing
generation_kwargs={"min_new_tokens": 1} on CPU devices.
- Add 6 new tests for both fixes (649 total passing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use create=True in mock.patch so tests work when trl has moved
PPOTrainer to trl.experimental and it's not in the trl namespace.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PPOTrainer.__init__() no longer accepts dataset= in newer trl versions.
Now checks via inspect.signature whether train_dataset or dataset is
accepted; if neither, sets dataset on trainer before .train() call.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: detect trl API via inspect — args= (>=0.28) vs config= (<0.28)
- PPO: split train into _train_builtin (trl >=0.28) and _train_manual
- GRPO: update error message to mention GRPO/PPO CPU limitation
- 2 new tests for PPO API detection (639 total, all passing)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PPO: add use_cpu=True to PPOConfig when running on CPU
- GRPO: add CPU warning + use_cpu flag via inspect (trl bug workaround)
- Add use_cpu error pattern to friendly error map
- 7 new tests for CPU fixes (637 total, all passing)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ASCII progress bar in display.py (replace Unicode blocks with #/-)
- Catch UnicodeEncodeError from plotext in data stats (Windows fallback)
- Auto-disable 4bit/8bit quantization on CPU (bitsandbytes doesn't support it)
- Friendly error messages for CPU tensor/dtype/bf16/torchvision issues
- Fix diff.py: torch_dtype= -> dtype= (deprecation warning)
- Add torchvision version compatibility check to soup doctor
7 new tests (631 total), ruff clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BUG-001: Replace Unicode arrows/dashes with ASCII in all console output
to fix UnicodeEncodeError on Windows cp1252 (~10 commands affected)
- BUG-002: PPO trainer uses inspect.signature to detect trl parameter names
(ppo_epochs vs num_ppo_epochs) for trl 0.28.0 compatibility
- BUG-003: Add get_compute_dtype() - uses float32 on CPU, bfloat16/float16
on CUDA. Fixes dtype mismatch in reward model and all trainers
- BUG-004: Add warning when using quantization on CPU
- BUG-005: Fix dtype -> torch_dtype in diff.py model loading
- BUG-006: Pin wandb<0.18.0 to avoid trl import conflict, add runtime guard
13 new tests (624 total), ruff clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add --backend vllm flag to soup serve for 2-4x better inference throughput
- Add --tensor-parallel and --gpu-memory flags for vLLM tuning
- Auto-detect vLLM and show hint when installed but not enabled
- New utils/vllm.py with engine creation, app factory, LoRA support
- Native token-by-token streaming via vLLM AsyncLLMEngine
- Add serve-fast extra: pip install 'soup-cli[serve-fast]'
- Add vllm detection to version --full
- 30 new tests (560 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `soup ui` command launches local web interface at http://127.0.0.1:7860
- FastAPI backend with REST API: runs, metrics, config validation, training
control, data inspection, templates, system info
- Self-contained SPA frontend (Dashboard, New Training, Data Explorer,
Model Chat) with Chart.js loss/LR charts
- Auto-opens browser on launch (--no-browser to disable)
- Config validation via new load_config_from_string() in config/loader.py
- 40 new tests (530 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add `modality: vision` config option for vision-language model training
- Add LLaVA and ShareGPT4V data format detection and conversion
- Add `image_dir` field in DataConfig for resolving image paths
- Add vision model loading via AutoModelForVision2Seq + AutoProcessor in SFT trainer
- Add `soup init --template vision` with LLaMA-3.2-Vision config
- Add image statistics display in `soup data inspect` for vision datasets
- Add Pillow as optional `vision` extra dependency
- Add Pillow to `soup doctor` dependency checks
- 51 new tests (455 total), ruff clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add --early-stop flag to soup sweep: skip remaining runs when loss exceeds best by threshold
- Add Rich progress bars for HuggingFace Hub model downloads in SFT/DPO trainers
- Pin click>=8.2.0 to fix typer compatibility (11 tests were failing)
- Bump version 0.3.2 → 0.4.0
- 329 tests passing, lint clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `soup version --full` shows version, Python, GPU backend, installed extras
- Dynamic test count badge via Gist endpoint in CI
- README: Optional Extras table, --verbose note, CSV/Parquet, Changelog link
- 323 tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rich markup splits --verbose with escape codes on CI, check for
"verbose" instead of "--verbose".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Friendly error messages: wrap all commands in try/except, map known errors
(CUDA OOM, missing deps, connection errors) to 2-3 line messages with fix hints
- Global --verbose flag for full tracebacks
- soup doctor: check system info, GPU, all dependency versions with fix suggestions
- soup quickstart: one-command demo (creates data + config + trains TinyLlama)
- Confirmation prompts before train/sweep (skip with --yes)
- 40 new tests (321 total), all passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- soup serve: FastAPI inference server with OpenAI-compatible API, SSE streaming
- soup data generate: synthetic data generation via OpenAI API or local models
- soup sweep: grid/random hyperparameter search with experiment tracker integration
- soup diff: side-by-side model comparison with metrics
- Multi-GPU/DeepSpeed: ZeRO Stage 2/3 configs, --deepspeed flag in train command
- 95 new tests (281 total), all passing
- Removed TESTING_GUIDE.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rich markup splits --resume/--wandb with escape codes in CI (Python 3.9).
Use case-insensitive substring match instead of exact flag match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4 hex char suffix (65536 possibilities) caused a collision when
generating 100 IDs within the same second on fast CI runners.
Increased to 8 hex chars (4 billion possibilities).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix 23 ruff errors: line too long, unused imports, ambiguous vars
- Fix validator: empty string is valid data, only count None as empty
- Remove unused imports in display.py and validator.py
- Rename ambiguous `l` vars to `part`, `entry`, `length`
- Break long lines in callback.py, display.py, sft.py, constants.py
All 20 tests passing, ruff clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Typer CLI: soup init, soup train, soup data inspect/validate
- Pydantic config schema with YAML loader and validation
- Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets
- Format detection: Alpaca, ShareGPT, ChatML (auto-detect)
- SFT trainer wrapper over transformers + peft + trl
- QLoRA/LoRA support with auto batch size estimation
- GPU detection (CUDA/MPS/CPU) and memory calculation
- Rich live terminal dashboard for training monitoring
- Config templates: chat, code, medical
- Tests (pytest) + GitHub Actions CI
- MIT license
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alpamys