mirror of https://github.com/razor-ai/soup.git
d75797cc3e
examples/README.md led with `soup train --config examples/configs/sft_basic.yaml`
and "Takes ~2-3 minutes on a consumer GPU". The command failed at validation,
and the fixture is 10 rows, so no training would have happened either way. That
timing was never measured; no timing claim replaces it.
Every command in the file was re-run against the installed CLI. Broken ones:
- `soup export output_sft_basic/ ...` -> requires -m/--model, not positional
- `soup merge output_sft_basic/ ...` -> requires -a/--adapter, not positional
- `soup data convert ... --from alpaca` -> no --from flag; source is auto-detected
- `soup data filter --input X` -> takes a positional path
- `soup data score ... --output Y` -> no --output; it prints a scorecard
- `soup data generate --provider ...` -> --prompt is required and was missing
- `soup data demo --list` -> no --list; bare `soup data demo` lists
Same three bugs were in synthetic_workflow.md and are fixed there too, so no
file in this directory documents a command that does not run.
The `soup data inspect` sample output was invented -- it claimed 50 entries for
a 10-row file and showed a row ("Identify the odd one out") that is not in it.
Replaced with the real output. Both config templates at the bottom of the file
taught the old flat schema, which is how a reader would have written a config
that fails to parse; both now show the current nesting.
The README is now split by what a file actually promises, which was the whole
problem: it presented a config with no data as if it were runnable.
Runnable examples -- parse and run as-is on the bundled fixtures. Stated
plainly: 5-10 rows, a smoke test, produces nothing useful.
Templates -- valid configs where you supply the data. vision_llama lives
here, and says why there is no fixture.
The Runnable table carries a VRAM column because "runnable" is hardware-
relative: the five TinyLlama configs are verified to clear the pre-flight on a
4 GB card, while dpo_example (~13 GB) and dpo_chat (~22 GB) cannot -- their
weights alone are 4.0 and 7.0 GB. Those numbers are what `soup train` predicts.
New examples/data/README.md, because Simon Willison read the folder as a
starting corpus and nothing in it said otherwise. First line says these are
format examples and test fixtures; a table gives the real row counts (10 / 5 /
8 / 5, recounted); then where to get real data, and roughly how much is needed
-- hundreds of rows for a format or style, thousands for a task the model
half-knows, and RAG rather than fine-tuning for new facts. No claim about how
volume scales with model size: we have not measured that.
|
||
|---|---|---|
| .. | ||
| README.md | ||
| alpaca_tiny.jsonl | ||
| chat_preferences.jsonl | ||
| dpo_sample.jsonl | ||
| reasoning_math.jsonl | ||
README.md
Sample data
These are format examples and test fixtures — not training data. Each file is a handful of rows, there to show what a format looks like and to give the parser and the test suite something to chew on. Nothing useful will train on this volume. Treat them as a shape to copy, not a starting corpus.
| File | Format | Rows | Demonstrates |
|---|---|---|---|
alpaca_tiny.jsonl |
alpaca | 10 | instruction / input / output instruction pairs |
chat_preferences.jsonl |
dpo | 5 | prompt / chosen / rejected, messages as chat turns |
dpo_sample.jsonl |
dpo | 8 | Same shape, longer answers — used by the DPO example |
reasoning_math.jsonl |
alpaca | 5 | Step-by-step worked solutions for GRPO/reasoning |
soup data inspect <file> prints the detected format and stats for any of them.
Getting real data
soup data search "instruction tuning" # find datasets on the HF Hub
soup data preview <dataset-id> # splits + features, before downloading
soup data download <dataset-id> -o out.jsonl
soup data demo # list bundled demo fixtures (same files as these)
How much do you actually need
- A format or a style (JSON output, a house tone, a response shape): hundreds of rows.
- A task the model already half-knows (your flavour of summarisation, classification): thousands.
- New facts the model has never seen: fine-tuning is usually the wrong tool — retrieval (RAG) puts the facts in the prompt instead, and stays correct when they change.
soup advise <data> --goal "..." reads your data and says which of these you're in.
More on formats, conversion, and the data pipeline: docs/data.md.