soup/examples/README.md

Soup Examples

Real-world configuration examples and sample datasets to get you running quickly.

Quick Start with Examples

1. Basic SFT (Supervised Fine-Tuning)

Fine-tune TinyLlama on a small instruction-following dataset:

soup train --config examples/configs/sft_basic.yaml

What it does:

• Trains TinyLlama-1.1B for 1 epoch
• Uses LoRA for efficient memory usage
• Outputs to ./output_sft_basic/
• Takes ~2-3 minutes on a consumer GPU

2. Chat Assistant (DPO)

Train a chat model with preference learning:

soup train --config examples/configs/dpo_chat.yaml

What it does:

• Uses Llama 2 7B base model
• Trains with DPO (Direct Preference Optimization) on chat preferences
• Better alignment than SFT alone
• Outputs to ./output_dpo_chat/

3. Reasoning Model (GRPO)

Fine-tune a reasoning model with step-by-step answer verification:

soup train --config examples/configs/grpo_reasoning.yaml

What it does:

• Trains on reasoning tasks (math, logic)
• Uses GRPO (Group Relative Policy Optimization) to optimize for correctness
• Generates multiple outputs per prompt and selects the best
• Outputs to ./output_reasoning/

4. Vision Model

Fine-tune LLaMA-Vision on image-caption pairs:

soup train --config examples/configs/vision_llama.yaml

What it does:

• Trains LLaMA-3.2-Vision-90B on image-text data
• Uses LLaVA format for images + text
• Outputs to ./output_vision/

5. Full RLHF Pipeline

Complete reinforcement learning from human feedback:

# Step 1: Pre-train with SFT
soup train --config examples/configs/rlhf_step1_sft.yaml

# Step 2: Train a reward model
soup train --config examples/configs/rlhf_step2_reward.yaml

# Step 3: PPO with reward model
soup train --config examples/configs/rlhf_step3_ppo.yaml

Dataset Formats

Datasets are included in JSONL format. Soup auto-detects and normalizes:

• Alpaca: instruction, input, output fields
• ShareGPT: conversations with from/value fields
• ChatML: OpenAI-style messages with role/content
• LLaVA: Vision format with image + conversations

Example: Inspect a Dataset

soup data inspect examples/data/alpaca_tiny.jsonl

Output:

📊 Dataset Statistics

Format detected: alpaca
Total entries: 50
Sample 1:
  instruction: "Identify the odd one out"
  input: "twitter, instagram, skype"
  output: "skype"

Example: Convert Between Formats

# Convert Alpaca to ChatML
soup data convert examples/data/alpaca_tiny.jsonl \
  --from alpaca --to chatml \
  --output alpaca_as_chatml.jsonl

Directory Structure

examples/
  configs/              # YAML configuration files
    sft_basic.yaml
    dpo_chat.yaml
    grpo_reasoning.yaml
    vision_llama.yaml
    rlhf_step1_sft.yaml
    rlhf_step2_reward.yaml
    rlhf_step3_ppo.yaml
  
  data/                 # Sample datasets (JSONL)
    alpaca_tiny.jsonl
    chat_preferences.jsonl
    reasoning_math.jsonl

Using Your Own Data

1. Prepare data in one of the supported formats
2. Update the config with your data path:

data:
  path: /path/to/your/data.jsonl
  format: alpaca  # or sharegpt, chatml, llava

3. Run training:

soup train --config your_config.yaml

Tips & Tricks

Save Space: Use Quantization

Add quantization to reduce model size:

quantization: int8  # Reduces memory by 4x

Speed Up Training: Use Unsloth Backend

Unsloth is 2-5x faster training:

pip install 'soup-cli[fast]'

Then in your config:

backend: unsloth

Monitor Training: Use Weights & Biases

Enable W&B logging:

pip install wandb
soup train --config your_config.yaml --wandb

Export for Inference: Convert to GGUF

After training, convert for Ollama/llama.cpp:

soup export output_sft_basic/ --output model.gguf --quant q8_0

Then use with Ollama:

ollama create my-model -f Ollama.modelfile

Merge LoRA Adapter

Merge your LoRA adapter into a standalone model:

soup merge output_sft_basic/ --output merged_model/

Common Issues

"CUDA out of memory"

• Reduce batch_size in config
• Enable quantization: quantization: int8
• Use smaller model: Mistral-7B instead of Llama-70B

"Dataset not found"

• Check file path in config (use absolute path if unsure)
• Verify format is correct: soup data inspect your_data.jsonl

"Model not found on Hugging Face"

• Check model ID spelling
• Ensure you have HuggingFace token: huggingface-cli login
• Or use a different model that's publicly available

Creating Your Own Configs

Minimal Config Template

model: tinyllama-1.1b
data:
  path: ./your_data.jsonl
  format: alpaca
task: sft
lora_r: 16
lora_alpha: 32
batch_size: 32
num_epochs: 3
learning_rate: 5e-4
output_dir: ./output/

Advanced Config Template

model: llama-2-7b
data:
  path: ./dataset.jsonl
  format: sharegpt
task: dpo
backend: unsloth
quantization: int8
lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
batch_size: 16
gradient_accumulation_steps: 4
num_epochs: 2
learning_rate: 1e-4
warmup_ratio: 0.1
max_seq_length: 2048
output_dir: ./output_advanced/

See config schema documentation for all available options.

Learn More

• README: Main documentation
• CONTRIBUTING: How to contribute
• CLAUDE.md: Architecture and detailed docs

Questions?

• Check the GitHub Discussions
• Open an Issue
• Read SECURITY.md for security questions

Happy training! 🍲