soup/examples/README.md

# Soup Examples

Real-world configuration examples and sample datasets to get you running quickly.

## Quick Start with Examples

### 1. Basic SFT (Supervised Fine-Tuning)

Fine-tune TinyLlama on a small instruction-following dataset:

```bash
soup train --config examples/configs/sft_basic.yaml
```

**What it does:**
- Trains TinyLlama-1.1B for 1 epoch
- Uses LoRA for efficient memory usage
- Outputs to `./output_sft_basic/`
- Takes ~2-3 minutes on a consumer GPU

### 2. Chat Assistant (DPO)

Train a chat model with preference learning:

```bash
soup train --config examples/configs/dpo_chat.yaml
```

**What it does:**
- Uses Llama 2 7B base model
- Trains with DPO (Direct Preference Optimization) on chat preferences
- Better alignment than SFT alone
- Outputs to `./output_dpo_chat/`

### 3. Reasoning Model (GRPO)

Fine-tune a reasoning model with step-by-step answer verification:

```bash
soup train --config examples/configs/grpo_reasoning.yaml
```

**What it does:**
- Trains on reasoning tasks (math, logic)
- Uses GRPO (Group Relative Policy Optimization) to optimize for correctness
- Generates multiple outputs per prompt and selects the best
- Outputs to `./output_reasoning/`

### 4. Vision Model

Fine-tune LLaMA-Vision on image-caption pairs:

```bash
soup train --config examples/configs/vision_llama.yaml
```

**What it does:**
- Trains LLaMA-3.2-Vision-90B on image-text data
- Uses LLaVA format for images + text
- Outputs to `./output_vision/`

### 5. Full RLHF Pipeline

Complete reinforcement learning from human feedback:

```bash
# Step 1: Pre-train with SFT
soup train --config examples/configs/rlhf_step1_sft.yaml

# Step 2: Train a reward model
soup train --config examples/configs/rlhf_step2_reward.yaml

# Step 3: PPO with reward model
soup train --config examples/configs/rlhf_step3_ppo.yaml
```

## Dataset Formats

Datasets are included in JSONL format. Soup auto-detects and normalizes:

- **Alpaca**: `instruction`, `input`, `output` fields
- **ShareGPT**: `conversations` with `from`/`value` fields
- **ChatML**: OpenAI-style `messages` with `role`/`content`
- **LLaVA**: Vision format with `image` + `conversations`

### Example: Inspect a Dataset

```bash
soup data inspect examples/data/alpaca_tiny.jsonl
```

Output:
```
📊 Dataset Statistics

Format detected: alpaca
Total entries: 50
Sample 1:
  instruction: "Identify the odd one out"
  input: "twitter, instagram, skype"
  output: "skype"
```

### Example: Convert Between Formats

```bash
# Convert Alpaca to ChatML
soup data convert examples/data/alpaca_tiny.jsonl \
  --from alpaca --to chatml \
  --output alpaca_as_chatml.jsonl
```

## Directory Structure

```
examples/
  configs/              # YAML configuration files
    sft_basic.yaml
    dpo_chat.yaml
    grpo_reasoning.yaml
    vision_llama.yaml
    rlhf_step1_sft.yaml
    rlhf_step2_reward.yaml
    rlhf_step3_ppo.yaml
  
  data/                 # Sample datasets (JSONL)
    alpaca_tiny.jsonl
    chat_preferences.jsonl
    reasoning_math.jsonl
```

## Using Your Own Data

1. **Prepare data** in one of the supported formats
2. **Update the config** with your data path:

```yaml
data:
  path: /path/to/your/data.jsonl
  format: alpaca  # or sharegpt, chatml, llava
```

3. **Run training**:

```bash
soup train --config your_config.yaml
```

## Tips & Tricks

### Save Space: Use Quantization

Add quantization to reduce model size:

```yaml
quantization: int8  # Reduces memory by 4x
```

### Speed Up Training: Use Unsloth Backend

Unsloth is 2-5x faster training:

```bash
pip install 'soup-cli[fast]'
```

Then in your config:

```yaml
backend: unsloth
```

### Monitor Training: Use Weights & Biases

Enable W&B logging:

```bash
pip install wandb
soup train --config your_config.yaml --wandb
```

### Export for Inference: Convert to GGUF

After training, convert for Ollama/llama.cpp:

```bash
soup export output_sft_basic/ --output model.gguf --quant q8_0
```

Then use with Ollama:

```bash
ollama create my-model -f Ollama.modelfile
```

### Merge LoRA Adapter

Merge your LoRA adapter into a standalone model:

```bash
soup merge output_sft_basic/ --output merged_model/
```

## Common Issues

### "CUDA out of memory"

- Reduce `batch_size` in config
- Enable quantization: `quantization: int8`
- Use smaller model: Mistral-7B instead of Llama-70B

### "Dataset not found"

- Check file path in config (use absolute path if unsure)
- Verify format is correct: `soup data inspect your_data.jsonl`

### "Model not found on Hugging Face"

- Check model ID spelling
- Ensure you have HuggingFace token: `huggingface-cli login`
- Or use a different model that's publicly available

## Creating Your Own Configs

### Minimal Config Template

```yaml
model: tinyllama-1.1b
data:
  path: ./your_data.jsonl
  format: alpaca
task: sft
lora_r: 16
lora_alpha: 32
batch_size: 32
num_epochs: 3
learning_rate: 5e-4
output_dir: ./output/
```

### Advanced Config Template

```yaml
model: llama-2-7b
data:
  path: ./dataset.jsonl
  format: sharegpt
task: dpo
backend: unsloth
quantization: int8
lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
batch_size: 16
gradient_accumulation_steps: 4
num_epochs: 2
learning_rate: 1e-4
warmup_ratio: 0.1
max_seq_length: 2048
output_dir: ./output_advanced/
```

See [config schema documentation](../CLAUDE.md#config-system) for all available options.

## Learn More

- **README**: [Main documentation](../README.md)
- **CONTRIBUTING**: [How to contribute](../CONTRIBUTING.md)
- **CLAUDE.md**: [Architecture and detailed docs](../CLAUDE.md)

## Questions?

- Check the [GitHub Discussions](https://github.com/MakazhanAlpamys/Soup/discussions)
- Open an [Issue](https://github.com/MakazhanAlpamys/Soup/issues)
- Read [SECURITY.md](../SECURITY.md) for security questions

Happy training! 🍲