Distillation
Training a smaller student model to reproduce a larger teacher model's behavior by learning from the teacher's outputs or output probabilities rather than ground-truth labels alone.
Knowledge distillation, formalized by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in 2015, transfers capability from a large "teacher" model to a smaller "student". In the classic form, the student learns to match the teacher's full probability distribution over tokens, not just its top prediction. That richer signal helps the student generalize better than training from scratch on the same data. When the teacher is only reachable through an API and its probabilities are hidden, the student is trained on the teacher's generated text instead.
Distillation in practice
Labs routinely distill their own models into cheaper tiers. Anthropic wrote in February 2026 that "frontier AI labs routinely distill their own models to create smaller, cheaper versions for their customers." Published examples:
- Google said Gemini 1.5 Flash was trained by distillation from 1.5 Pro, and its Gemma 3 technical report says every Gemma 3 model was trained with knowledge distillation.
- Meta said Llama 4 Maverick was codistilled from Llama 4 Behemoth, a much larger teacher model.
- Alibaba trained its smaller Qwen3 models with what it calls strong-to-weak distillation from its flagship models.
- DeepSeek released six dense R1-Distill models, from 1.5B to 70B and built on Qwen and Llama bases, fine-tuned on reasoning data generated by DeepSeek-R1.
A distilled model costs a fraction as much to run while keeping much of the teacher's quality on the tasks it was trained for.
Distillation approaches
- Logit distillation: Train the student to match the teacher's token probabilities. It needs the teacher's output probabilities, so it is mostly used inside a lab or with open-weight teachers.
- Response distillation: Generate teacher outputs on a dataset and train the student to reproduce them. Simple, and the only option when the teacher is API-only.
- Feature distillation: Match intermediate layer activations. More complex but can transfer richer representations.
- Reasoning distillation: The teacher generates chain-of-thought traces and the student learns to produce similar reasoning, as in DeepSeek's R1-Distill models.
- On-policy distillation: The student generates its own answers and the teacher scores its token choices, so the student learns from its own mistakes rather than only from teacher-written text.
Limits of distillation
A student rarely surpasses its teacher. Distillation degrades on out-of-distribution inputs that the teacher handled through general reasoning. For creative or open-ended tasks, the gap between student and teacher tends to be larger than for structured tasks. Distillation works best when the student targets specific capabilities rather than broad replication.
Provider terms and distillation attacks
Distilling a commercial model without permission breaks its provider's terms. OpenAI's Terms of Use bar using Output "to develop models that compete with OpenAI." Anthropic's Commercial Terms bar accessing its services "to build a competing product or service, including to train competing AI models." Google's Gemini API terms say you "may not use the Services to develop models that compete with the Services" or "attempt to reverse engineer, extract or replicate any component of the Services."
The dispute escalated in 2026. In February, Anthropic said DeepSeek, Moonshot AI and MiniMax generated more than 16 million exchanges with Claude through about 24,000 fraudulent accounts, and press coverage of a September Anthropic threat report put the total traced to distillation campaigns at nearly 200 million exchanges. On September 8, the NSA, CISA and FBI issued a joint advisory accusing six Chinese companies (DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun and Z.ai) of industrial-scale distillation campaigns against US AI companies. In October, The Information reported that DeepSeek and Moonshot used outputs from OpenAI's GPT-6 Astra to train DeepSeek's V4.1 Flash update and Kimi K3.1 Pro. The companies named have not confirmed these claims, and none has been tested in court.
Related terms
Models relevant to Distillation
DeepSeek V4 Flash
DeepSeek's 284B open-weight Flash model, replaced by V4.1 Flash in September 2026
View model →Kimi K3.1 Pro
Moonshot AI's open-weight 1M-context flagship for long-horizon coding
View model →GPT-6 Astra
OpenAI's most capable model for the most demanding work
View model →Gemma 3
Google's previous-generation open, multimodal, multilingual model family.
View model →Llama 4
Meta's natively multimodal open-weight MoE models, with up to 10M tokens of context on Scout.
View model →Qwen 3
Alibaba's open-weight model family with switchable thinking and non-thinking modes.
View model →