For Developers/Glossary/Context Window
Core Concepts

Context Window

The maximum number of tokens an LLM can process in a single inference call, including both input (prompt, context, history) and output (generated response).

The context window is the LLM's working memory. Everything the model processes in a single inference call (your system prompt, conversation history, retrieved documents, tool outputs and the response it generates) must fit within this limit.

Input vs output tokens

Context window size usually refers to the total token budget, and both input and output count against it. Output is also capped separately. GPT-6 Luna, for example, has a 1.05M-token window, a 922K-token input cap and a 128K-token output cap, so a very long prompt plus a long answer must fit all three limits.

How models handle long contexts

At inference time, models use the KV cache to avoid recomputing earlier tokens at every step. Even so, longer contexts cost more (APIs bill per token) and run slower. Some models also recall information in the middle of a very long context less reliably than information near the start or end, an effect known as "lost in the middle".

Current context window sizes

One million tokens has become the standard for frontier and even budget models in 2026:

Long-context pricing differs by provider. Claude Haiku 5.5 charges five times more per input token once a prompt exceeds 100K tokens, and GPT-6 Astra and GPT-6 Luna bill the whole request at twice the input rate and 1.5 times the output rate above 272K input tokens. Claude Opus 5.5, by contrast, charges the same per-token rate across its full 1M window. See our model index for every model's limits.

Practical implications

Larger context windows don't remove the need for RAG (Retrieval-Augmented Generation). Putting an entire knowledge base into every prompt is expensive and can lower answer quality. Retrieving only the relevant pieces and assembling them into a focused prompt is often cheaper and more accurate. Very long contexts (above 100K tokens) are most useful for tasks like codebase Q&A, long document analysis and keeping long conversation histories.

Related terms

Models relevant to Context Window