Core Concepts

Inference

The process of running a trained model on input data to generate outputs, also called the serving phase as opposed to training.

Inference is what happens when you call a model API: your input tokens are processed by the model's weights, and output tokens are generated one at a time. Nearly everything a developer deals with in production (latency, cost, throughput, context limits) is a property of inference, not training.

Autoregressive inference

LLMs generate tokens one at a time, left to right. Each generated token is appended to the context and used to predict the next one, so output positions cannot be computed in parallel. Training does not have this constraint: the full target text is already known, so every position in a sequence can be processed at once.

Prefill vs decode phases

Inference has two phases:

  • Prefill: All input tokens are processed in parallel to build the KV cache. This phase is limited mainly by compute and determines time to first token.
  • Decode: Output tokens are generated one at a time using the KV cache. This phase is limited mainly by memory bandwidth, since the weights and cache are read for every token, and it determines output speed.

Inference optimization techniques

  • Quantization: Store weights (and sometimes activations or the KV cache) at lower precision than 16-bit, cutting memory use and raising throughput with small quality loss. Open-weight releases now often ship 8-bit FP8 checkpoints and 4-bit NVFP4 variants for NVIDIA Blackwell GPUs; Mistral Large 4 ships both.
  • FlashAttention: A fused attention kernel that reduces memory traffic.
  • Continuous batching: Add and remove requests from the running batch as they arrive and finish, instead of waiting for a full batch.
  • Speculative decoding: A small draft model (or extra prediction heads) proposes several tokens, and the large model verifies them in one parallel pass.
  • KV cache optimization: Paging, compressing or quantizing cached keys and values to fit more concurrent requests in memory.
  • Mixture of experts: MoE models run only part of the network for each token. Mistral Large 4 has about 1 trillion parameters but uses about 52 billion per token, although all the weights still have to fit in memory.

Self-hosted vs managed inference

Managed APIs, from model developers such as Anthropic, OpenAI and Google and from hosts for open-weight models such as Together AI and Fireworks AI, handle hardware and scaling. Self-hosting gives control over cost and data but requires GPU expertise: vLLM and SGLang are the common engines for servers, and llama.cpp and Ollama for local machines. Hugging Face's Text Generation Inference (TGI), once a common choice, is now in maintenance mode, and its documentation points to vLLM and SGLang instead. Hardware sets the limits either way: GPU memory determines which models and context lengths fit, and memory bandwidth largely sets decode speed.

Related terms

Models relevant to Inference