Inference

KV Cache

A store of the attention key and value vectors for tokens a model has already processed, so each new token can be generated without recomputing the whole context.

A transformer generates text one token at a time, and every new token attends to all the tokens before it. To do that, each attention layer turns earlier tokens into key and value vectors. The KV (key-value) cache keeps those vectors in GPU memory once they are computed, so each generation step only runs the newest token through the model and reads the cached keys and values for everything before it.

Why it matters

Without a KV cache, every step would rerun the model over the whole prompt plus everything generated so far, so the work for each new token would grow with the full sequence length. With the cache, each token's keys and values are computed once. Attention still reads every cached entry at each step, which is why long contexts remain slower and more expensive than short ones even with caching. For servers handling many requests at once, KV cache memory, not raw compute, is often what limits how many requests fit on a GPU.

Prompt caching

API providers including Anthropic, OpenAI and Google reuse the KV cache across requests. When several requests start with the same prefix, such as a long system prompt or a reference document, the prefix is processed once and later requests read it from the cache. Cache reads are billed at a 90% to 95% discount: Claude Haiku 5.5 and GPT-6 Luna both charge $0.01 per million cached input tokens against $0.10 uncached, and Claude Opus 5.5 and Gemini 4 Argon cut cached input by 95%. Writing to the cache can cost a little more than normal input: on Claude Haiku 5.5 and GPT-6 Luna a cache write costs $0.125 per million tokens, 1.25 times the base rate. The match is on an exact prefix, so any change early in the prompt, such as a timestamp in the system prompt, turns everything after it into a cache miss. Self-hosted serving engines such as vLLM and SGLang offer the same prefix reuse.

Memory constraints and optimization

The KV cache grows linearly with context length and with the number of concurrent requests. At 1M-token contexts the cache for a single sequence can be larger than the model weights, which limits how many long requests a GPU cluster can serve at once. Several techniques shrink it:

  • Grouped-query and multi-query attention (GQA, MQA): Several query heads share one set of key and value heads, so fewer vectors are stored per token.
  • Multi-head latent attention (MLA): Introduced by DeepSeek, it compresses keys and values into a smaller latent vector before caching.
  • KV cache quantization: Storing keys and values in 8-bit or lower precision instead of 16-bit, usually with small quality loss.
  • Paged memory management: PagedAttention, used in vLLM, allocates cache memory in blocks to avoid fragmentation and wasted space.

FlashAttention and similar kernels make attention faster and avoid building the full attention matrix in memory, but they do not reduce the size of the KV cache itself.

Related terms

Models relevant to KV Cache