hermes-agent/skills/mlops/unsloth/references/llms.md

Unsloth Documentation

Unsloth Documentation

• Unsloth Docs: Train your own model with Unsloth, an open-source framework for LLM fine-tuning and reinforcement learning.
• Beginner? Start here!
• Unsloth Requirements: Here are Unsloth's requirements including system and GPU VRAM requirements.
• FAQ + Is Fine-tuning Right For Me?: If you're stuck on if fine-tuning is right for you, see here! Learn about fine-tuning misconceptions, how it compared to RAG and more:
• Unsloth Notebooks: Explore our catalog of Unsloth notebooks:
• All Our Models
• Install & Update: Learn to install Unsloth locally or online.
• Updating: To update or use an old version of Unsloth, follow the steps below:
• Pip Install: To install Unsloth locally via Pip, follow the steps below:
• Docker: Install Unsloth using our official Docker container
• Windows Installation: See how to install Unsloth on Windows with or without WSL.
• AMD: Fine-tune with Unsloth on AMD GPUs.
• Conda Install: To install Unsloth locally on Conda, follow the steps below:
• Google Colab: To install and run Unsloth on Google Colab, follow the steps below:
• Fine-tuning LLMs Guide: Learn all the basics and best practices of fine-tuning. Beginner-friendly.
• What Model Should I Use?
• Datasets Guide: Learn how to create & prepare a dataset for fine-tuning.
• LoRA Hyperparameters Guide: Optimal lora rank. alpha, number of epochs, batch size & gradient accumulation, QLoRA vs LoRA, target modules and more!
• Tutorial: How to Finetune Llama-3 and Use In Ollama: Beginner's Guide for creating a customized personal assistant (like ChatGPT) to run locally on Ollama
• Reinforcement Learning (RL) Guide: Learn all about Reinforcement Learning (RL) and how to train your own DeepSeek-R1 reasoning model with Unsloth using GRPO. A complete guide from beginner to advanced.
• Tutorial: Train your own Reasoning model with GRPO: Beginner's Guide to transforming a model like Llama 3.1 (8B) into a reasoning model by using Unsloth and GRPO.
• Advanced RL Documentation: Advanced documentation settings when using Unsloth with GRPO.
• Memory Efficient RL
• RL Reward Hacking: Learn what is Reward Hacking in Reinforcement Learning and how to counter it.
• GSPO Reinforcement Learning: Train with GSPO (Group Sequence Policy Optimization) RL in Unsloth.
• Reinforcement Learning - DPO, ORPO & KTO: To use the reward modelling functions for DPO, GRPO, ORPO or KTO with Unsloth, follow the steps below:
• DeepSeek-OCR: How to Run & Fine-tune: Guide on how to run and fine-tune DeepSeek-OCR locally.
• How to Fine-tune LLMs with Unsloth & Docker: Learn how to fine-tune LLMs or do Reinforcement Learning (RL) with Unsloth's Docker image.
• Vision Reinforcement Learning (VLM RL): Train Vision/multimodal models via GRPO and RL with Unsloth!
• gpt-oss Reinforcement Learning
• Tutorial: How to Train gpt-oss with RL: Learn to train OpenAI gpt-oss with GRPO to autonomously beat 2048 locally or on Colab.
• Unsloth Dynamic GGUFs on Aider Polyglot: Performance of Unsloth Dynamic GGUFs on Aider Polyglot Benchmarks
• Qwen3-VL: How to Run & Fine-tune: Learn to fine-tune and run Qwen3-VL locally with Unsloth.
• gpt-oss: How to Run & Fine-tune: Run & fine-tune OpenAI's new open-source models!
• Tutorial: How to Fine-tune gpt-oss: Learn step-by-step how to train OpenAI gpt-oss locally with Unsloth.
• Long Context gpt-oss Training
• GLM-4.6: How to Run Locally: A guide on how to run Z.ai's new GLM-4.6 model on your own local device!
• IBM Granite 4.0: How to run IBM Granite-4.0 with Unsloth GGUFs on llama.cpp, Ollama and how to fine-tune!
• DeepSeek-V3.1: How to Run Locally: A guide on how to run DeepSeek-V3.1 and Terminus on your own local device!
• Qwen3-Coder: How to Run Locally: Run Qwen3-Coder-30B-A3B-Instruct and 480B-A35B locally with Unsloth Dynamic quants.
• Gemma 3: How to Run & Fine-tune: How to run Gemma 3 effectively with our GGUFs on llama.cpp, Ollama, Open WebUI and how to fine-tune with Unsloth!
• Gemma 3n: How to Run & Fine-tune: Run Google's new Gemma 3n locally with Dynamic GGUFs on llama.cpp, Ollama, Open WebUI and fine-tune with Unsloth!
• Qwen3: How to Run & Fine-tune: Learn to run & fine-tune Qwen3 locally with Unsloth + our Dynamic 2.0 quants
• Qwen3-2507: Run Qwen3-30B-A3B-2507 and 235B-A22B Thinking and Instruct versions locally on your device!
• Tutorials: How To Fine-tune & Run LLMs: Learn how to run and fine-tune models for optimal performance 100% locally with Unsloth.
• DeepSeek-R1-0528: How to Run Locally: A guide on how to run DeepSeek-R1-0528 including Qwen3 on your own local device!
• Magistral: How to Run & Fine-tune: Meet Magistral - Mistral's new reasoning models.
• Llama 4: How to Run & Fine-tune: How to run Llama 4 locally using our dynamic GGUFs which recovers accuracy compared to standard quantization.
• Kimi K2: How to Run Locally: Guide on running Kimi K2 and Kimi-K2-Instruct-0905 on your own local device!
• Grok 2: Run xAI's Grok 2 model locally!
• Devstral: How to Run & Fine-tune: Run and fine-tune Mistral Devstral 1.1, including Small-2507 and 2505.
• DeepSeek-V3-0324: How to Run Locally: How to run DeepSeek-V3-0324 locally using our dynamic quants which recovers accuracy
• DeepSeek-R1: How to Run Locally: A guide on how you can run our 1.58-bit Dynamic Quants for DeepSeek-R1 using llama.cpp.
• DeepSeek-R1 Dynamic 1.58-bit: See performance comparison tables for Unsloth's Dynamic GGUF Quants vs Standard IMatrix Quants.
• QwQ-32B: How to Run effectively: How to run QwQ-32B effectively with our bug fixes and without endless generations + GGUFs.
• Phi-4 Reasoning: How to Run & Fine-tune: Learn to run & fine-tune Phi-4 reasoning models locally with Unsloth + our Dynamic 2.0 quants
• Running & Saving Models: Learn how to save your finetuned model so you can run it in your favorite inference engine.
• Saving to GGUF: Saving models to 16bit for GGUF so you can use it for Ollama, Jan AI, Open WebUI and more!
• Saving to Ollama
• Saving to vLLM for deployment: Saving models to 16bit for vLLM deployment and serving
• Saving to SGLang for deployment: Saving models to 16bit for SGLang for deployment and serving
• Unsloth Inference: Learn how to run your finetuned model with Unsloth's faster inference.
• Troubleshooting Inference: If you're experiencing issues when running or saving your model.
• vLLM Engine Arguments
• LoRA Hot Swapping Guide
• Text-to-Speech (TTS) Fine-tuning: Learn how to to fine-tune TTS & STT voice models with Unsloth.
• Unsloth Dynamic 2.0 GGUFs: A big new upgrade to our Dynamic Quants!
• Vision Fine-tuning: Learn how to fine-tune vision/multimodal LLMs with Unsloth
• Fine-tuning LLMs with NVIDIA DGX Spark and Unsloth: Tutorial on how to fine-tune and do reinforcement learning (RL) with OpenAI gpt-oss on NVIDIA DGX Spark.
• Fine-tuning LLMs with Blackwell, RTX 50 series & Unsloth: Learn how to fine-tune LLMs on NVIDIA's Blackwell RTX 50 series and B200 GPUs with our step-by-step guide.
• Multi-GPU Training with Unsloth: Learn how to fine-tune LLMs on multiple GPUs and parallelism with Unsloth.
• Finetuning from Last Checkpoint: Checkpointing allows you to save your finetuning progress so you can pause it and then continue.
• Troubleshooting & FAQs: Tips to solve issues, and frequently asked questions.
• Chat Templates: Learn the fundamentals and customization options of chat templates, including Conversational, ChatML, ShareGPT, Alpaca formats, and more!
• Quantization-Aware Training (QAT): Quantize models to 4-bit with Unsloth and PyTorch to recover accuracy.
• Unsloth Environment Flags: Advanced flags which might be useful if you see breaking finetunes, or you want to turn stuff off.
• Continued Pretraining: AKA as Continued Finetuning. Unsloth allows you to continually pretrain so a model can learn a new language.
• Unsloth Benchmarks: Unsloth recorded benchmarks on NVIDIA GPUs.