Training

Fine-tuning

Continued training of a pre-trained model on a smaller, task-specific dataset to adapt its weights for a particular domain, task, or output format.

A pre-trained LLM has broad knowledge but generic behavior. Fine-tuning updates the model's weights on a curated dataset to shift it toward a specific format, style, domain, or task without starting training from scratch.

When to fine-tune vs prompt-engineer

Fine-tuning makes sense when:

  • The task requires a consistently specific output format that prompting alone struggles to maintain
  • Domain-specific language or jargon needs to be baked in (medical notes, legal contracts, internal terminology)
  • Latency or cost is critical - fine-tuned smaller models often outperform large models on narrow tasks
  • You have hundreds or thousands of labeled examples

If you have fewer than 50 examples, start with few-shot prompting and retrieval-augmented generation first.

Fine-tuning approaches

  • Full fine-tuning: Updates all weights. Most expensive but most flexible. Requires significant GPU memory.
  • LoRA / QLoRA: Adds small adapter layers; only those are updated. 10-100x cheaper than full fine-tuning. The dominant approach for open-source models.
  • RLHF / DPO: Trains the model to prefer outputs matching human preferences. Used by Anthropic, OpenAI, and others to improve chat behavior. DPO has gained adoption as a simpler alternative to RLHF.

Fine-tuning services and tools

Hosted fine-tuning has narrowed: OpenAI has closed fine-tuning to new users and is winding it down, Mistral has deprecated its fine-tuning API, and Anthropic offers none for current Claude models. Together AI and Tinker offer LoRA fine-tuning of open-weight models, Google Vertex AI tunes some Gemini Flash models. Anthropic does not offer general fine-tuning of its current Claude models. For open-source models, Hugging Face's PEFT library with LoRA is the standard toolkit. TRL (Transformers Reinforcement Learning) is widely used for preference tuning such as DPO, and vLLM is the common choice for serving the fine-tuned model afterwards.

Common mistakes

Overfitting on too small a dataset causes catastrophic forgetting, where the model loses general capabilities. Data quality matters far more than quantity - 200 high-quality examples beat 10,000 noisy ones. Monitor validation loss and test on out-of-distribution examples to catch degradation early.

Related terms

Models relevant to Fine-tuning