examples/README.md led with `soup train --config examples/configs/sft_basic.yaml`
and "Takes ~2-3 minutes on a consumer GPU". The command failed at validation,
and the fixture is 10 rows, so no training would have happened either way. That
timing was never measured; no timing claim replaces it.
Every command in the file was re-run against the installed CLI. Broken ones:
- `soup export output_sft_basic/ ...` -> requires -m/--model, not positional
- `soup merge output_sft_basic/ ...` -> requires -a/--adapter, not positional
- `soup data convert ... --from alpaca` -> no --from flag; source is auto-detected
- `soup data filter --input X` -> takes a positional path
- `soup data score ... --output Y` -> no --output; it prints a scorecard
- `soup data generate --provider ...` -> --prompt is required and was missing
- `soup data demo --list` -> no --list; bare `soup data demo` lists
Same three bugs were in synthetic_workflow.md and are fixed there too, so no
file in this directory documents a command that does not run.
The `soup data inspect` sample output was invented -- it claimed 50 entries for
a 10-row file and showed a row ("Identify the odd one out") that is not in it.
Replaced with the real output. Both config templates at the bottom of the file
taught the old flat schema, which is how a reader would have written a config
that fails to parse; both now show the current nesting.
The README is now split by what a file actually promises, which was the whole
problem: it presented a config with no data as if it were runnable.
Runnable examples -- parse and run as-is on the bundled fixtures. Stated
plainly: 5-10 rows, a smoke test, produces nothing useful.
Templates -- valid configs where you supply the data. vision_llama lives
here, and says why there is no fixture.
The Runnable table carries a VRAM column because "runnable" is hardware-
relative: the five TinyLlama configs are verified to clear the pre-flight on a
4 GB card, while dpo_example (~13 GB) and dpo_chat (~22 GB) cannot -- their
weights alone are 4.0 and 7.0 GB. Those numbers are what `soup train` predicts.
New examples/data/README.md, because Simon Willison read the folder as a
starting corpus and nothing in it said otherwise. First line says these are
format examples and test fixtures; a table gives the real row counts (10 / 5 /
8 / 5, recounted); then where to get real data, and roughly how much is needed
-- hundreds of rows for a format or style, thousands for a task the model
half-knows, and RAG rather than fine-tuning for new facts. No claim about how
volume scales with model size: we have not measured that.
Seven of the eight configs in examples/configs/ did not parse. They were
written against a pre-nesting schema and stayed that way through several
schema changes, so `soup train --config examples/configs/sft_basic.yaml` --
the first command examples/README.md tells you to run -- failed validation
with "base: Field required; data -> train: Field required".
model: -> base:
data.path: -> data.train:
max_seq_length: -> data.max_length
quantization: -> training.quantization ("null" -> "none", int8 -> 8bit)
lora_r/alpha/... -> training.lora.{r,alpha,dropout,target_modules}
num_epochs -> training.epochs
learning_rate -> training.lr
lr_scheduler_type -> training.scheduler
output_dir: -> output:
Dropped keys with no schema equivalent: seed, eval_steps, eval_strategy,
save_strategy, load_best_model_at_end.
Three correctness fixes beyond the mechanical migration:
- dpo_chat / rlhf_step2_reward declared `format: sharegpt`, but
chat_preferences.jsonl is prompt/chosen/rejected -- detect_format() calls
it dpo, and a reward model needs the pair shape. Now `format: dpo`.
- target_modules listed `out_proj`, which no Llama has (it is `o_proj`), so
PEFT would have raised on the two configs using it.
- vision_llama pointed at examples/data/vision_dataset.jsonl and
examples/data/images/, neither of which has ever existed in this repo. It
is now marked a TEMPLATE in its first line, with placeholder paths and the
reason there is no fixture: we do not commit image files. Its base model id
was `llama-vision-13b`, which is not a real repo id; now
meta-llama/Llama-3.2-11B-Vision-Instruct, matching the vision template.
batch_size and max_length on the five TinyLlama configs were leftovers that
made them unrunnable rather than tuned choices: batch_size 16 against a
10-row fixture never forms a single batch, and max_length 2048 against a
longest-row of ~233 tokens only inflated the activation budget. The VRAM
pre-flight refused all five on a 4 GB card. Now batch_size 4 / max_length
512, with the reason written above the line so the number does not drift
back anonymously. Verified: all five clear `soup train --dry-run` on a 4 GB
card and report their data as valid. dpo_example is unchanged (its values
are pinned by tests/test_dpo_example.py).
All 8 configs now pass load_config_from_string.
Every printed and documented `pip install 'soup-cli[extra]'` was bash / zsh /
PowerShell syntax and failed on Windows cmd.exe:
ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at
the start of dependency specifier
cmd.exe has no single-quote quoting, so it passes the quotes to pip verbatim
and pip rejects the requirement. Nothing in Soup can fix that once the command
is typed -- pip and the shell own it, and Soup is not installed yet when the
README line runs -- so the fix is the spelling we print.
Migrated 147 sites across 67 files to `pip install "soup-cli[extra]"`:
- 64 in src/ (Rich console hints + plain ImportError text)
- 57 in README.md + docs/
- 22 in src/soup_cli/templates/*.yaml + examples/configs/*.yaml
- 3 in examples/README.md
Double quotes are the only spelling valid in every shell (cmd, PowerShell,
bash, zsh), which is why the repo already used `pip install -e ".[dev]"`.
Measured on Windows: single quotes fail ONLY on cmd; double quotes pass
everywhere; bare passes on Windows but zsh globs `[extra]` and fails.
Method note (the PR #247 class): the hints sit INSIDE double-quoted Python
string literals, so a blind ' -> " sed produces SyntaxError. A tokenize-based
rewriter escaped `\"` in DQUOTE tokens and left bare `"` in TRIPLE / COMMENT
tokens; every touched .py was compile-checked. The full suite (not ruff, not
compile-check) caught two rewriter blind spots: the real YAML templates under
src/soup_cli/templates/ (byte-identical drift test) and examples/README.md.
A regression test (tests/test_v07137.py) scans the package and every docs code
block for the single-quoted form; prose may still name it so a reader from an
older tutorial recognises the error.
Also bundles #315 (@Sanjays2402): eval-gate benchmark tasks now run via
ForgettingDetector instead of a helper that never existed. Closes#310.
Test count: 16283 -> 16288 (+4 in tests/test_v07137.py).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes from 5 sequential ECC reviews (python/code/security/tdd/verification).
python-review (2 CRITICAL + HIGH/MED/LOW):
- signal/vote coherence: schema now requires the active detector in
reward_hack_signals + rejects the inactive detector name (was silently
dropping the primary signal from the vote).
- integral_clamp is its own field (was wrongly hard-wired to beta_ceil).
- task/backend gate runs before controller-config checks; EMA convention
corrected; type hints; mutable-list default -> tuple + normalised compare.
code-review (4 HIGH + MED/LOW):
- _prune now trims _saved in sync with disk (rollback can't target a deleted
checkpoint); bang-bang release_count resets after each relaxation (hysteretic
descent); EMA formula uses standard convention; _escalate no longer burns a
recovery attempt on a None target; max_recovery_attempts>=1 required with
rollback; _action_history capped; on_step_end logs errors once; loud warning
when the mitigation callback can't attach (was a silent safety-off).
security-review (HIGH + MED):
- restore_checkpoint / save_checkpoint refuse a SYMLINKED optimizer.pt
(torch.load weights_only=False was an RCE via attacker-placed symlink);
bool-before-int/float guards on all new numeric fields; reward_hack_signals
max_length=4; empty-signals guard in the callback.
tdd-review: +13 coverage tests (dead-band hold, shim verbatim-on-error,
conservative boundary, read-only-beta dual-write, escalation postconditions,
both-restore, PID D exact, log cap + concurrency, no-top-level-import, fuzz
field-validity).
Test count 152 -> 180 (+2 POSIX-only symlink skips).
The repo moved to src-layout and trimmed README into a 238-line front door
with the feature reference under docs/, but several committed files still
referenced bare soup_cli/ paths or linked the gitignored .claude/CLAUDE.md
(which 404s for anyone cloning the public repo).
- docs/: `soup_cli/{plugins,templates,ui/plugins}/...` path refs -> `src/soup_cli/...`
(import statements `from soup_cli...` left unchanged — package name is still soup_cli)
- AGENTS.md: point external agents at public docs/, CONTRIBUTING.md, and the
config schema; note CLAUDE.md is a maintainer-local (gitignored) file
- CONTRIBUTING.md + .github/pull_request_template.md: PR checklist now says
"README.md and the matching page under docs/" (kept in sync); Questions
section links docs/ instead of the gitignored CLAUDE.md
- examples/README.md: fix two broken ../CLAUDE.md links -> config schema source
+ docs/ feature reference
- .gitignore: add root-anchored /_*.py temp-script guard + trailing newline
Eight features that close out the v0.44.x live-monitoring deferrals
plus a long tail of standalone CLI wins:
- #94 /api/train/stream async SSE with per-subscriber cursor + JS
EventSource consumer; SoupTrainerCallback pushes TrainEvent on
each on_log.
- #95 soup ui --public derives LAN IP via SOCK_DGRAM connect-trick,
prints scannable QR; --auth-token override; SPA bootstrap
hydrates window._authToken from ?token= + sessionStorage and
cleans the URL via history.replaceState; CORS regex auto-widens
to loopback + RFC1918 in public mode; set_auth_token rotation
race fixed via threading.Lock.
- #98 soup serve --reasoning-parser strips <think>...</think> (and
OpenThinker tags); pre-compiled regex with marker-token
fast-path + 1 MiB cap + leading-newline-only strip.
- #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON
endpoint; best-effort observation hook in callback.on_log.
- #15 soup tokenizer train: BPE training CLI with raw-path lstat
symlink rejection, 50 MiB total / 8 KiB per-line caps,
post-mkdir output-dir re-check, --special-token NUL/oversize
dedup, vocab bounds [256, 200000].
- #26 soup bench --p50 --p95 renders extra per-prompt tail-latency
Rich table; --prompts-file gains symlink rejection.
- #28 soup bench --backend auto: MLX weights.npz probe (per-entry
lstat) -> config.json model_type keyword -> transformers
fallback; SOUP_BENCH_BACKEND env hint.
- #12 examples/synthetic_workflow.{md,yaml} end-to-end walkthrough.
Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the
python / code / security / tdd review agents. Notable HIGH:
- QR token now consumed by SPA (was unreachable previously).
- set_auth_token rotation lock-protected, 8-thread stress tested.
- Tokenizer input + output symlink TOCTOU defence on raw path.
- SSE generator switched to async (asyncio.sleep) for non-blocking
multi-subscriber operation.
- CORS regex for --public LAN mode (the old fixed allowlist of
http://0.0.0.0:port never matched a real Origin header).
- _has_mlx_weights per-entry lstat so a symlinked weights.npz can't
trigger MLX dispatch.
Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the
relaxed v0.53.8 version-pin asserts in tests/test_v0538.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a working DPO (Direct Preference Optimization) example using the
current Pydantic config schema with Llama 3.1 8B Instruct and QLoRA.
- examples/configs/dpo_example.yaml: DPO config with all core training
and LoRA parameters, plus commented-out advanced options
- examples/data/dpo_sample.jsonl: 8 preference pairs in DPO format
with ShareGPT-style message lists for chosen/rejected
- tests/test_dpo_example.py: 7 tests validating config loading, field
values, data format detection, and data validation
- examples/README.md: document the new DPO with QLoRA example
Alpamys
Alpamys
Chinmaya Sahu