Distillation & Small Models
When a small model beats a large one: knowledge distillation (logits, sequences, on-policy), synthetic data from a teacher, license limits on model outputs, evaluation against the teacher and serving cost math.
Overview
Why this topic
Most production LLM traffic is narrow: classify this ticket, extract these fields, route this request, answer from these documents. A frontier model does those jobs well, but it is slow and expensive for them. A model ten or a hundred times smaller, trained on that one job, often matches it on the task and costs a fraction to serve.
Distillation is the main way to get that small model. A large teacher model produces the targets (probabilities over tokens, or whole responses) and a small student learns to imitate them. The engineering questions are practical: is the task narrow enough, can you legally use the teacher's outputs, how do you know the student is good enough, and does the serving saving pay for the work.
Interview questions
- Your team spends heavily on a large model for one classification and extraction job. Make the case for and against distilling a small model, and say how you would decide.
- Compare logit distillation, sequence-level distillation and on-policy distillation. Which would you use for a 1B model that must follow the teacher closely?
- Design the synthetic data pipeline for distilling a support assistant from a large teacher. What goes wrong most often?
- Legal asks whether you may train your own model on outputs from a commercial API. How do you answer as an engineer?
- How do you decide the distilled model is ready to replace the teacher in production?
- Estimate whether distillation pays for itself, and say which numbers you would need to measure.
Sign up free to read the full topic
More Agentic AI topics
- Async Long-Running Tool Calls
- Autoscaling LLM Inference on Kubernetes
- Browser & Computer-use Agents
- Context Engineering & Prompt Design
- Context Window Management for Persistent Agents
- Designing Tools a Model Calls Well
- Embeddings & Vector Search
- Evaluating Agents: Task Suites, Trajectories & Regression Gates
- Evaluating RAG & Catching Hallucinations
- Fine-tuning LLMs: SFT, LoRA/QLoRA & Preference Tuning
- Guardrails & Prompt Injection Defense
- LLM Cost & Latency Engineering
Start free on Cariara · Coding problems · System design studio