Latency vs Throughput
Latency is how long a single request takes, usually split into time to first token and output speed; throughput is how much work a system completes per second across all concurrent requests.
These two metrics pull against each other in LLM serving and drive different model and infrastructure choices depending on the workload.
Definitions
- Time to first token (TTFT): How long until the first output token arrives. Critical for streaming interfaces where users watch output appear. For reasoning models the first token of the visible answer can arrive long after the request, because the model thinks first; Artificial Analysis reports this as "time to first answer token."
- Output speed: Tokens generated per second once output starts. Time per output token (TPOT) is its inverse.
- Total latency: TTFT plus (TPOT multiplied by output length). What users experience as end-to-end response time.
- Throughput: Total tokens generated per second across all concurrent requests. The key metric for cost and batch processing.
The latency-throughput tradeoff
LLM inference servers can optimize for one or the other. Serving each request immediately minimizes latency but leaves GPUs underused. Batching multiple requests together maximizes GPU use (throughput) but adds queue wait time (latency). Production serving typically targets a latency SLA and maximizes throughput within that constraint.
Measured speeds for current models
Artificial Analysis measures each provider's API and averages each endpoint's runs over the past five days, so the figures move. Its output speed counts reasoning tokens. Figures from its model pages in early November 2026:
- Output speed: Claude Haiku 5.5 (max effort) 225 tokens per second, Claude Opus 5.5 (max) 114, Gemini 3.8 Flash (high) 112, GPT-6 Luna (max) 102, GPT-6.1 Terra (max) 74, Gemini 4 Argon (high) 71 and GPT-6.1 Sol (max) 55.
- Time to first answer token: GPT-6 Luna without reasoning answers in 1.11 seconds and GPT-6.1 Terra without reasoning in 1.75 seconds. With reasoning on, the effort setting dominates: Claude Opus 5.5 takes 6.96 seconds at low effort and 410 seconds at max, and GPT-6.1 Sol takes 2.80 seconds at low and 208 seconds at max.
Practical implications
- Real-time chat: Time to the first visible token matters most. A wait of more than a second or two before text appears is noticeable, so keep reasoning effort low for interactive turns where the provider allows it.
- Batch processing: Throughput matters. Run overnight jobs through batch APIs and minimize cost per token.
- Agentic workflows: Latency compounds over many steps. A 2-second-per-call agent with 20 steps takes 40 seconds minimum, regardless of throughput.
Thinking models add a new kind of latency
Reasoning models spend tokens thinking before they produce visible output, and on several current models thinking is on by default. On Anthropic's API, Claude Haiku 5.5 thinks by default and lets you turn thinking off at high effort or below; Claude Sonnet 5.5 cannot turn it off but can skip up-front thinking; and Claude Opus 5.5 and Claude Fable 5.2 cannot turn it off at all. OpenAI's GPT-6.1 Terra accepts a reasoning effort of "none." For latency-sensitive calls, lower the effort setting first, and remember that thinking tokens are billed as output even when the thinking text is not returned.
Related terms
Models relevant to Latency vs Throughput
Claude Haiku 5.5
Anthropic's fastest, lowest-cost current model at $0.10 per million input tokens with a 1M context window
View model →GPT-6 Luna
OpenAI's cheapest GPT-6 model, and since October 2026 the default for free ChatGPT users
View model →GPT-6.1 Terra
OpenAI's fast, low-cost GPT-6 model for computer use, visual reasoning and high-volume work
View model →Gemini 3.8 Flash
Google's previous-generation Flash model for coding and agents, at half Gemini 4 Hydrogen's price until December 31, 2026
View model →Claude Opus 5.5
Anthropic's current Opus and its recommended starting model, priced at $4/$20 per 1M tokens
View model →GPT-6.1 Sol
Near-Astra intelligence for a fifth of the price
View model →