mirror of https://github.com/razor-ai/soup.git
docs: add Pre-training and MoE sections to README
Add dedicated sections for continued pre-training (task: pretrain, plaintext format) and MoE model support (ScatterMoE LoRA, auto-detection). Update task lists to include Pretrain. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
232a3b143c
commit
61a8a1fb56
49
README.md
49
README.md
|
|
@ -161,10 +161,55 @@ training:
|
|||
alpha: 16
|
||||
```
|
||||
|
||||
Works with all training tasks: SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, and IPO. If unsloth is installed but not enabled, Soup will suggest it automatically.
|
||||
Works with all training tasks: SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, and Pretrain. If unsloth is installed but not enabled, Soup will suggest it automatically.
|
||||
|
||||
> **Tip:** Soup auto-detects unsloth. When installed, you'll see a hint during `soup train` if you haven't enabled it yet.
|
||||
|
||||
## Continued Pre-training
|
||||
|
||||
Continue training a model on raw text for domain adaptation:
|
||||
|
||||
```yaml
|
||||
base: meta-llama/Llama-3.1-8B
|
||||
task: pretrain
|
||||
|
||||
data:
|
||||
train: ./data/corpus.jsonl # {"text": "..."} or plain .txt files
|
||||
format: plaintext
|
||||
max_length: 4096
|
||||
|
||||
training:
|
||||
epochs: 1
|
||||
lr: 1e-5
|
||||
quantization: 4bit
|
||||
```
|
||||
|
||||
```bash
|
||||
soup init --template pretrain
|
||||
soup train
|
||||
```
|
||||
|
||||
## MoE Model Support
|
||||
|
||||
Fine-tune Mixture of Experts models (Mixtral, Qwen3-30B-A3B, DeepSeek V3) with ScatterMoE LoRA — applies LoRA to both attention layers and expert FFN layers:
|
||||
|
||||
```yaml
|
||||
base: Qwen/Qwen3-30B-A3B
|
||||
task: sft
|
||||
|
||||
training:
|
||||
moe_lora: true # target expert + attention layers
|
||||
moe_aux_loss_coeff: 0.01 # router load-balancing loss
|
||||
quantization: 4bit
|
||||
```
|
||||
|
||||
Soup auto-detects MoE architectures. Works with all training tasks.
|
||||
|
||||
```bash
|
||||
soup init --template moe
|
||||
soup train
|
||||
```
|
||||
|
||||
## Vision / Multimodal Fine-tuning
|
||||
|
||||
Fine-tune vision-language models (LLaMA-3.2-Vision, Qwen2-VL, Pixtral) on image+text data:
|
||||
|
|
@ -247,7 +292,7 @@ output: ./output
|
|||
- **QAT** (`quantization_aware: true`): Better quality when you plan to deploy with aggressive quantization (int8/int4). ~5-10% slower training, but the model learns to compensate for quantization noise.
|
||||
- **Post-training quantization** (default): Faster training, good enough for most use cases. Quantize after training with `soup export --quant q4_k_m`.
|
||||
|
||||
QAT works with all training tasks (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO) and vision modality. Not compatible with the unsloth backend. After QAT training, export to GGUF normally with `soup export`.
|
||||
QAT works with all training tasks (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, Pretrain) and vision modality. Not compatible with the unsloth backend. After QAT training, export to GGUF normally with `soup export`.
|
||||
|
||||
## DPO Training
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue