Soup
Fine-tune LLMs in one command. No SSH, no config hell.
Quick Start ·
Features ·
Data Tools ·
Tracking ·
Eval ·
Commands
---
Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
```bash
pip install soup-cli
soup init --template chat
soup train
```
## Why Soup?
Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.
- **Zero SSH.** Never SSH into a broken GPU box again.
- **One config.** A simple YAML file is all you need.
- **Auto everything.** Batch size, GPU detection, quantization — handled.
- **Works locally.** Train on your own GPU with QLoRA. No cloud required.
## Quick Start
### 1. Install
```bash
# From PyPI (recommended):
pip install soup-cli
# Or from GitHub (latest dev):
pip install git+https://github.com/MakazhanAlpamys/Soup.git
```
### 2. Create config
```bash
# Interactive wizard
soup init
# Or use a template
soup init --template chat # conversational fine-tune
soup init --template code # code generation
soup init --template medical # domain expert
soup init --template reasoning # GRPO reasoning training
soup init --template vision # vision/multimodal fine-tune
soup init --template kto # KTO unpaired preference alignment
soup init --template orpo # ORPO (no reference model needed)
soup init --template simpo # SimPO length-normalized preference
soup init --template ipo # IPO regularized preference
soup init --template rlhf # full RLHF pipeline (SFT→RM→PPO)
```
### 3. Train
```bash
soup train --config soup.yaml
```
That's it. Soup handles LoRA setup, quantization, batch size, monitoring, and checkpoints.
### 4. Test your model
```bash
soup chat --model ./output
```
### 5. Push to HuggingFace
```bash
soup push --model ./output --repo your-username/my-model
```
### 6. Merge & Export
```bash
# Merge LoRA adapter with base model
soup merge --adapter ./output
# Export to GGUF for Ollama / llama.cpp
soup export --model ./output --format gguf --quant q4_k_m
```
## Config Example
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth # 2-5x faster, pip install 'soup-cli[fast]'
data:
train: ./data/train.jsonl
format: alpaca
val_split: 0.1
training:
epochs: 3
lr: 2e-5
batch_size: auto
lora:
r: 64
alpha: 16
quantization: 4bit
output: ./output
```
## Unsloth Backend (2-5x Faster Training)
Use the [Unsloth](https://github.com/unslothai/unsloth) backend for significantly faster training and up to 80% less VRAM:
```bash
# Install unsloth support
pip install 'soup-cli[fast]'
```
Then add one line to your config:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: unsloth # 2-5x faster, -80% VRAM
data:
train: ./data/train.jsonl
format: alpaca
training:
epochs: 3
lr: 2e-5
quantization: 4bit
lora:
r: 64
alpha: 16
```
Works with all training tasks: SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, and IPO. If unsloth is installed but not enabled, Soup will suggest it automatically.
> **Tip:** Soup auto-detects unsloth. When installed, you'll see a hint during `soup train` if you haven't enabled it yet.
## Vision / Multimodal Fine-tuning
Fine-tune vision-language models (LLaMA-3.2-Vision, Qwen2-VL, Pixtral) on image+text data:
```bash
# Install vision support
pip install 'soup-cli[vision]'
# Create a vision config
soup init --template vision
# Train
soup train --config soup.yaml
```
```yaml
base: meta-llama/Llama-3.2-11B-Vision-Instruct
task: sft
modality: vision
data:
train: ./data/vision_train.jsonl
format: llava
image_dir: ./data/images
val_split: 0.1
training:
epochs: 3
lr: 1e-5
quantization: 4bit
lora:
r: 64
alpha: 16
```
**Supported vision data formats:**
**LLaVA:**
```json
{"image": "photo.jpg", "conversations": [{"from": "human", "value": "\nDescribe this image."}, {"from": "gpt", "value": "A cat on a mat."}]}
```
**ShareGPT4V:**
```json
{"image": "chart.png", "conversations": [{"from": "human", "value": "\nWhat does this show?"}, {"from": "gpt", "value": "Quarterly revenue."}]}
```
`soup data inspect` automatically shows image statistics (count, formats, missing files) for vision datasets.
## Quantization-Aware Training (QAT)
Train with simulated quantization for significantly better post-quantization quality compared to standard QLoRA:
```bash
# Install QAT support
pip install 'soup-cli[qat]'
```
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
data:
train: ./data/train.jsonl
format: alpaca
training:
epochs: 3
lr: 2e-5
quantization: 4bit
quantization_aware: true # Enable QAT
lora:
r: 64
alpha: 16
output: ./output
```
**When to use QAT vs post-training quantization:**
- **QAT** (`quantization_aware: true`): Better quality when you plan to deploy with aggressive quantization (int8/int4). ~5-10% slower training, but the model learns to compensate for quantization noise.
- **Post-training quantization** (default): Faster training, good enough for most use cases. Quantize after training with `soup export --quant q4_k_m`.
QAT works with all training tasks (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO) and vision modality. Not compatible with the unsloth backend. After QAT training, export to GGUF normally with `soup export`.
## DPO Training
Train with preference data using Direct Preference Optimization:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: dpo
data:
train: ./data/preferences.jsonl
format: dpo
training:
epochs: 3
dpo_beta: 0.1
lora:
r: 64
alpha: 16
quantization: 4bit
```
## GRPO Training (Reasoning)
Train reasoning models with Group Relative Policy Optimization (DeepSeek-R1 style):
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: grpo
data:
train: ./data/reasoning_train.jsonl
format: sharegpt
max_length: 4096
training:
epochs: 3
lr: 1e-5
grpo_beta: 0.1
num_generations: 4
reward_fn: accuracy # or 'format', or path to custom .py
lora:
r: 64
alpha: 16
quantization: 4bit
```
```bash
# Create a reasoning config
soup init --template reasoning
# Train
soup train --config soup.yaml
```
**Built-in reward functions:**
- `accuracy` — checks if the final answer matches expected (supports `####` and `\boxed{}` formats)
- `format` — checks for structured `...` reasoning blocks
**Custom reward functions** — point to a Python file:
```python
# my_reward.py
def reward_fn(completions, **kwargs):
"""Score each completion. Return list of floats."""
return [1.0 if "correct" in c[-1]["content"] else 0.0 for c in completions]
```
```yaml
training:
reward_fn: ./my_reward.py
```
## PPO / Full RLHF Pipeline
Train models with the full RLHF pipeline: SFT warmup → Reward Model → PPO alignment.
```bash
# Create an RLHF config
soup init --template rlhf
```
**Step 1: SFT warmup** — fine-tune a base model on your data:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
data:
train: ./data/train.jsonl
format: alpaca
output: ./output_sft
```
**Step 2: Train reward model** — learn preferences from human feedback:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: reward_model
data:
train: ./data/preferences.jsonl
format: dpo
output: ./output_rm
```
**Step 3: PPO alignment** — optimize the policy using the reward model:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: ppo
data:
train: ./data/prompts.jsonl
format: chatml
training:
reward_model: ./output_rm
ppo_epochs: 4
ppo_clip_ratio: 0.2
ppo_kl_penalty: 0.05
lora:
r: 64
alpha: 16
quantization: 4bit
output: ./output_ppo
```
PPO supports two reward sources:
- **Reward model** (`reward_model`): pre-trained reward model (from step 2)
- **Reward function** (`reward_fn`): callable function (same as GRPO — `accuracy`, `format`, or custom `.py`)
## KTO Training (Unpaired Preferences)
Train with unpaired preference data — no need for chosen+rejected pairs:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: kto
data:
train: ./data/kto_train.jsonl
format: kto
training:
epochs: 3
kto_beta: 0.1
lora:
r: 64
alpha: 16
quantization: 4bit
```
**KTO data format:**
```json
{"prompt": "What is 2+2?", "completion": "4", "label": true}
{"prompt": "What is 2+2?", "completion": "Fish", "label": false}
```
## ORPO Training (No Reference Model)
ORPO combines SFT and alignment in one step — no reference model needed:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: orpo
data:
train: ./data/preferences.jsonl
format: dpo
training:
epochs: 3
orpo_beta: 0.1
lora:
r: 64
alpha: 16
quantization: 4bit
```
## SimPO Training (Simple Preference)
SimPO uses length-normalized log probabilities as implicit rewards — reference-free:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: simpo
data:
train: ./data/preferences.jsonl
format: dpo
training:
epochs: 3
simpo_gamma: 0.5
cpo_alpha: 1.0
lora:
r: 64
alpha: 16
quantization: 4bit
```
## IPO Training (Regularized Preference)
IPO is a theoretically grounded DPO variant with stronger regularization:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: ipo
data:
train: ./data/preferences.jsonl
format: dpo
training:
epochs: 3
ipo_tau: 0.1
lora:
r: 64
alpha: 16
quantization: 4bit
```
## DoRA (Weight-Decomposed LoRA)
Enable DoRA for improved LoRA quality with magnitude decomposition:
```yaml
training:
lora:
r: 64
alpha: 16
use_dora: true # Enable DoRA
```
Works with all training tasks and backends.
## LoRA+ (Differentiated Learning Rates)
Use different learning rates for LoRA A and B matrices:
```yaml
training:
lr: 2e-5
loraplus_lr_ratio: 16.0 # lr_B = lr × 16
lora:
r: 64
alpha: 16
```
## GaLore (Memory-Efficient Full-Parameter Training)
Train without LoRA using gradient low-rank projection — saves optimizer memory:
```yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
data:
train: ./data/train.jsonl
format: alpaca
training:
epochs: 3
lr: 2e-5
quantization: none # Required: GaLore is incompatible with quantization
use_galore: true
galore_rank: 128
galore_update_proj_gap: 200
galore_scale: 0.25
```
> **Note:** GaLore requires `quantization: none` and `backend: transformers` (not unsloth).
## Chat with your model
```bash
# Chat with a LoRA adapter (auto-detects base model)
soup chat --model ./output
# Specify base model explicitly
soup chat --model ./output --base meta-llama/Llama-3.1-8B-Instruct
# Adjust generation
soup chat --model ./output --temperature 0.3 --max-tokens 256
```
## Push to HuggingFace
```bash
# Upload model to HF Hub
soup push --model ./output --repo your-username/my-model
# Make it private
soup push --model ./output --repo your-username/my-model --private
```
## Merge LoRA Adapter
Merge a LoRA adapter with its base model into a standalone model:
```bash
# Auto-detect base model from adapter_config.json
soup merge --adapter ./output --output ./merged
# Specify base model and dtype
soup merge --adapter ./output --base meta-llama/Llama-3.1-8B --dtype bfloat16
```
## Export to GGUF
Export models to GGUF format for use with [Ollama](https://ollama.com/) and [llama.cpp](https://github.com/ggerganov/llama.cpp):
```bash
# Export LoRA adapter (auto-merges with base, then converts)
soup export --model ./output --format gguf --quant q4_k_m
# Export with different quantizations
soup export --model ./output --format gguf --quant q8_0
soup export --model ./output --format gguf --quant f16
# Export a full (already merged) model
soup export --model ./merged --format gguf
# Specify llama.cpp path manually
soup export --model ./output --format gguf --llama-cpp /path/to/llama.cpp
```
Supported quantizations: `q4_0`, `q4_k_m`, `q5_k_m`, `q8_0`, `f16`, `f32`
After export, use with Ollama:
```bash
echo 'FROM ./my-model.q4_k_m.gguf' > Modelfile
ollama create my-model -f Modelfile
ollama run my-model
```
## Resume Training
Resume a training run from a checkpoint:
```bash
# Auto-detect latest checkpoint in output directory
soup train --config soup.yaml --resume auto
# Resume from a specific checkpoint
soup train --config soup.yaml --resume ./output/checkpoint-500
```
## Batch Inference
Run a model on a list of prompts and save results:
```bash
# JSONL input (each line: {"prompt": "..."})
soup infer --model ./output --input prompts.jsonl --output results.jsonl
# Plain text input (one prompt per line)
soup infer --model ./output --input prompts.txt --output results.jsonl
# Custom generation settings
soup infer --model ./output --input prompts.jsonl --output results.jsonl \
--max-tokens 512 --temperature 0.3
```
Output is JSONL with `prompt`, `response`, and `tokens_generated` fields. Shows a progress bar and throughput summary.
## TensorBoard Integration
Log training metrics to TensorBoard for local visualization:
```bash
# Enable TensorBoard logging (requires: pip install tensorboard)
soup train --config soup.yaml --tensorboard
# View logs
tensorboard --logdir ./output/runs/
```
> **Note:** `--tensorboard` and `--wandb` cannot be used together. Pick one.
## Weights & Biases Integration
Send training metrics to [W&B](https://wandb.ai/) for cloud-based experiment tracking:
```bash
# Enable W&B logging (requires: pip install wandb)
soup train --config soup.yaml --wandb
```
Make sure `WANDB_API_KEY` is set or run `wandb login` first.
## Inference Server
Start a local OpenAI-compatible inference server:
```bash
# Install server dependencies
pip install 'soup-cli[serve]'
# Start server
soup serve --model ./output --port 8000
# With custom settings
soup serve --model ./output --port 8080 --host 127.0.0.1 --max-tokens 1024
```
Endpoints:
- `POST /v1/chat/completions` — chat completions (streaming supported)
- `GET /v1/models` — list available models
- `GET /health` — health check
Compatible with OpenAI SDK:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="output",
messages=[{"role": "user", "content": "Hello!"}],
)
```
### vLLM Backend (2-4x Faster Inference)
Use [vLLM](https://github.com/vllm-project/vllm) for significantly better throughput in production:
```bash
# Install vLLM support
pip install 'soup-cli[serve-fast]'
# Start with vLLM backend
soup serve --model ./output --backend vllm
# Multi-GPU with tensor parallelism
soup serve --model ./output --backend vllm --tensor-parallel 2
# Control GPU memory usage
soup serve --model ./output --backend vllm --gpu-memory 0.8
```
> **Tip:** Soup auto-detects vLLM. When installed, you'll see a hint during `soup serve` if you haven't enabled it yet.
> **Note (v0.10.10+):** `max_tokens` is capped at 16,384 per request. Error details are never exposed in HTTP responses.
## Synthetic Data Generation
Generate training data using LLMs:
```bash
# Generate using OpenAI API
soup data generate --prompt "Create math word problems" --count 100 --format alpaca
# Use a different model
soup data generate --prompt "Medical Q&A pairs" --model gpt-4o --count 500
# Deduplicate against existing data
soup data generate --prompt "..." --count 200 --dedup-with existing.jsonl
# Use seed examples to guide style
soup data generate --prompt "..." --seed examples.jsonl --count 100
```
## Hyperparameter Sweep
Search for the best hyperparameters:
```bash
# Grid search over learning rate and LoRA rank
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --param lora_r=8,16,32
# Random search with max runs
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --strategy random --max-runs 5
# Preview without running
soup sweep --config soup.yaml --param lr=1e-5,2e-5 --param epochs=2,3 --dry-run
# Early stopping: skip remaining runs if loss exceeds 1.5x best
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --early-stop 1.5
```
## Model Comparison
Compare outputs of two models side-by-side:
```bash
# Compare with inline prompts
soup diff --model-a ./model_v1 --model-b ./model_v2 --prompt "Explain gravity"
# Compare with a prompts file
soup diff --model-a ./base --model-b ./finetuned --prompts test_prompts.jsonl
# Save results
soup diff --model-a ./a --model-b ./b --prompts prompts.txt --output results.jsonl
```
## Multi-GPU / DeepSpeed
Train on multiple GPUs with DeepSpeed:
```bash
# ZeRO Stage 2 (recommended for most cases)
soup train --config soup.yaml --deepspeed zero2
# ZeRO Stage 3 (for very large models)
soup train --config soup.yaml --deepspeed zero3
# ZeRO Stage 2 with CPU offload (memory-constrained)
soup train --config soup.yaml --deepspeed zero2_offload
# Custom DeepSpeed config
soup train --config soup.yaml --deepspeed ./my_ds_config.json
```
## Quickstart Demo
Run a complete demo in one command — creates sample data, config, and trains a tiny model:
```bash
# Full demo (creates data + config + trains TinyLlama)
soup quickstart
# Just create files without training
soup quickstart --dry-run
# Skip confirmation
soup quickstart --yes
```
## Health Check
Check your environment for compatibility issues:
```bash
soup doctor
```
Shows: Python version, GPU availability, all dependency versions, and fix suggestions.
## Version Info
```bash
# Basic version
soup version
# Full system info (useful for bug reports)
soup version --full
# -> soup v0.13.2 | Python 3.11.5 | CUDA 12.1 | extras: serve, data
```
## Web UI
Launch a local web interface to manage experiments, start training, explore data, and chat with models — all from your browser.
```bash
pip install 'soup-cli[ui]'
soup ui
# -> opens http://127.0.0.1:7860 in your browser
# -> prints auth token to console
```
**Pages:**
- **Dashboard** — view all experiment runs, loss charts, system info
- **New Training** — create configs from templates, validate, and start training
- **Data Explorer** — browse and inspect datasets (JSONL, JSON, CSV, Parquet)
- **Model Chat** — chat with a running `soup serve` inference server
**Security (v0.10.10+):** The Web UI generates a random auth token at startup (printed to console). All mutating endpoints (start/stop training, delete runs, inspect data, validate config) require `Authorization: Bearer ` header. CORS is restricted to the served origin. Data inspection is sandboxed to the working directory.
```bash
# Custom port, don't auto-open browser
soup ui --port 8080 --no-browser
```
## Error Handling
Soup shows friendly error messages by default (2-3 lines with a fix suggestion). For full tracebacks:
```bash
# Global flag goes BEFORE the command
soup --verbose train --config soup.yaml
# Works with any command
soup --verbose eval --model ./output --benchmarks mmlu
```
> **Note:** `--verbose` is a global flag — it must go **before** the command name, not after.
## Data Formats
Soup supports these formats (auto-detected). Files can be JSONL, JSON, CSV, or Parquet.
**Alpaca:**
```json
{"instruction": "Explain gravity", "input": "", "output": "Gravity is..."}
```
**ShareGPT:**
```json
{"conversations": [{"from": "human", "value": "Hi"}, {"from": "gpt", "value": "Hello!"}]}
```
**ChatML:**
```json
{"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}
```
**DPO / ORPO / SimPO / IPO (preference pairs):**
```json
{"prompt": "Explain gravity", "chosen": "Gravity is a force...", "rejected": "I don't know"}
```
**KTO (unpaired preferences):**
```json
{"prompt": "Explain gravity", "completion": "Gravity is a force...", "label": true}
```
**LLaVA (vision):**
```json
{"image": "photo.jpg", "conversations": [{"from": "human", "value": "\nDescribe this."}, {"from": "gpt", "value": "A cat."}]}
```
**ShareGPT4V (vision):**
```json
{"image": "chart.png", "conversations": [{"from": "human", "value": "\nExplain this chart."}, {"from": "gpt", "value": "Revenue growth."}]}
```
## Data Tools
```bash
# Inspect a dataset
soup data inspect ./data/train.jsonl
# Validate format
soup data validate ./data/train.jsonl --format alpaca
# Convert between formats
soup data convert ./data/train.jsonl --to sharegpt --output converted.jsonl
# Merge multiple datasets
soup data merge data1.jsonl data2.jsonl --output merged.jsonl --shuffle
# Remove near-duplicates (requires: pip install 'soup-cli[data]')
soup data dedup ./data/train.jsonl --threshold 0.8
# Extended statistics (length distribution, token counts, languages)
soup data stats ./data/train.jsonl
```
## Experiment Tracking
Every `soup train` run is automatically tracked in a local SQLite database (`~/.soup/experiments.db`).
```bash
# List all training runs
soup runs
# Show detailed info + loss curve for a run
soup runs show run_20260223_143052_a1b2
# Compare two runs side by side
soup runs compare run_1 run_2
# Delete a run
soup runs delete run_1
```
## Model Evaluation
Evaluate models on standard benchmarks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness):
```bash
# Install eval dependencies
pip install 'soup-cli[eval]'
# Evaluate on benchmarks
soup eval --model ./output --benchmarks mmlu,gsm8k,hellaswag
# Link results to a training run
soup eval --model ./output --benchmarks mmlu --run-id run_20260223_143052_a1b2
```
## All Commands
```
soup init [--template chat|code|medical|reasoning|vision|kto|orpo|simpo|ipo|rlhf] Create config
soup train --config soup.yaml Start training
soup train --config soup.yaml --tensorboard Train with TensorBoard logging
soup infer --model ./output --input p.jsonl Batch inference
soup chat --model ./output Interactive chat
soup push --model ./output --repo user/name Upload to HuggingFace
soup merge --adapter ./output Merge LoRA with base model
soup export --model ./output --format gguf Export to GGUF (Ollama)
soup eval --model ./output --benchmarks mmlu Evaluate on benchmarks
soup serve --model ./output --port 8000 OpenAI-compatible API server
soup serve --model ./output --backend vllm vLLM backend (2-4x throughput)
soup sweep --config soup.yaml --param lr=... Hyperparameter search
soup diff --model-a ./a --model-b ./b Compare two models
soup data inspect View dataset stats
soup data validate --format alpaca Check format
soup data convert --to chatml Convert between formats
soup data merge data1.jsonl data2.jsonl Combine datasets
soup data dedup --threshold 0.8 Remove duplicates (MinHash)
soup data stats Extended statistics
soup data generate --prompt "..." --count 100 Generate synthetic data
soup runs List training runs
soup runs show Run details + loss graph
soup runs compare Compare two runs
soup ui [--port 7860] Web UI (experiments, training, data)
soup doctor Check environment
soup quickstart [--dry-run] Full demo
soup version [--full] Show version (--full: system info)
soup --verbose Full traceback on errors
```
## Supported Models
Soup works with **any** of the **340,000+** text-generation models on [HuggingFace Hub](https://huggingface.co/models?pipeline_tag=text-generation). If a model supports `AutoModelForCausalLM`, it works with Soup — zero config changes needed.
### Recommended Models
| Model Family | Models | Sizes | Best For |
|---|---|---|---|
| **Llama 4** | Llama-4-Scout-17B, Llama-4-Maverick-17B | 17B | General, multilingual |
| **Llama 3.x** | Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct | 1B–70B | Chat, instruction following |
| **Llama 3.2 Vision** | Llama-3.2-11B-Vision-Instruct, Llama-3.2-90B-Vision | 11B–90B | Image understanding |
| **Gemma 3** | Gemma-3-4B-IT, Gemma-3-9B-IT, Gemma-3-27B-IT | 4B–27B | Efficient, multilingual |
| **Qwen 3** | Qwen3-8B, Qwen3-14B, Qwen3-32B, Qwen3-235B-A22B | 0.6B–235B | Reasoning, code, MoE |
| **Qwen 2.5** | Qwen2.5-7B-Instruct, Qwen2.5-Coder-32B-Instruct | 0.5B–72B | Code, math |
| **DeepSeek** | DeepSeek-R1-Distill-Llama-8B, DeepSeek-V3-0324 | 1.5B–671B | Reasoning (GRPO), code |
| **Phi-4** | Phi-4-14B, Phi-4-mini-reasoning | 3.8B–14B | Compact reasoning |
| **Mistral** | Mistral-7B-Instruct-v0.3, Mistral-Small-24B-Instruct | 7B–24B | Fast, efficient |
| **Mixtral** | Mixtral-8x7B-Instruct-v0.1, Mixtral-8x22B | 47B–141B | MoE architecture |
| **CodeLlama** | CodeLlama-7b-Instruct-hf, CodeLlama-34b-Instruct | 7B–34B | Code generation |
| **StarCoder 2** | StarCoder2-15B, StarCoder2-7B | 3B–15B | Code completion |
| **Yi** | Yi-1.5-34B-Chat, Yi-1.5-9B-Chat | 6B–34B | Multilingual chat |
| **InternLM 3** | InternLM3-8B-Instruct | 8B | Chinese + English |
| **Falcon** | Falcon-11B, Falcon-40B-Instruct | 7B–180B | Open-weight |
### Vision Models (with `modality: vision`)
| Model | Size | Supported Formats |
|---|---|---|
| LLaMA-3.2-11B-Vision-Instruct | 11B | LLaVA, ShareGPT4V |
| Qwen2-VL-7B-Instruct | 7B | LLaVA, ShareGPT4V |
| Pixtral-12B-2409 | 12B | LLaVA, ShareGPT4V |
### Quick Size Guide
| VRAM | Max Model (QLoRA 4-bit) | Example |
|---|---|---|
| 8 GB | ~7B | Llama-3.1-8B, Mistral-7B |
| 16 GB | ~14B | Phi-4-14B, Qwen2.5-14B |
| 24 GB | ~34B | CodeLlama-34B, Yi-1.5-34B |
| 48 GB | ~70B | Llama-3.3-70B |
| 80 GB+ | 70B+ (full) or MoE | Mixtral-8x22B, DeepSeek-V3 |
> **Note:** Soup auto-detects your GPU and estimates the optimal batch size. Use `soup doctor` to check your setup.
## Requirements
- Python 3.9+
- GPU with CUDA (recommended) or Apple Silicon (MPS) or CPU (experimental)
- 8 GB+ VRAM for 7B models with QLoRA
> **CPU note:** All training tasks (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO) work on CPU but will be very slow. Quantization (`4bit`/`8bit`) is auto-disabled on CPU. GRPO on CPU uses `min_new_tokens=1` to prevent empty generation errors. A default chat template is set automatically if the tokenizer lacks one. PPO datasets are tokenized before training to ensure compatibility with trl's experimental API.
### Optional Extras
| Extra | Install | What it adds |
|---|---|---|
| `vision` | `pip install 'soup-cli[vision]'` | Vision/multimodal fine-tuning (Pillow) |
| `qat` | `pip install 'soup-cli[qat]'` | Quantization-Aware Training (torchao) |
| `fast` | `pip install 'soup-cli[fast]'` | Unsloth backend (2-5x faster, -80% VRAM) |
| `ui` | `pip install 'soup-cli[ui]'` | Web UI + inference server (FastAPI + uvicorn) |
| `serve` | `pip install 'soup-cli[serve]'` | Inference server (FastAPI + uvicorn) |
| `serve-fast` | `pip install 'soup-cli[serve-fast]'` | vLLM inference backend (2-4x throughput) |
| `data` | `pip install 'soup-cli[data]'` | Deduplication (MinHash via datasketch) |
| `eval` | `pip install 'soup-cli[eval]'` | Benchmark evaluation (lm-evaluation-harness) |
| `deepspeed` | `pip install 'soup-cli[deepspeed]'` | Multi-GPU training (DeepSpeed ZeRO) |
| `dev` | `pip install 'soup-cli[dev]'` | Tests + linting (pytest, ruff) |
## Development
```bash
git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
pip install -e ".[dev]"
# Lint
ruff check soup_cli/ tests/
# Run unit tests (fast, no GPU needed)
pytest tests/ -v
# Run smoke tests (downloads tiny model, runs real training)
pytest tests/ -m smoke -v
```
## Changelog
See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history.
## License
MIT