For Developers/Glossary/Latency vs Throughput
Inference

Latency vs Throughput

Latency is how long a single request takes, usually split into time to first token and output speed; throughput is how much work a system completes per second across all concurrent requests.

These two metrics pull against each other in LLM serving and drive different model and infrastructure choices depending on the workload.

Definitions

  • Time to first token (TTFT): How long until the first output token arrives. Critical for streaming interfaces where users watch output appear. For reasoning models the first token of the visible answer can arrive long after the request, because the model thinks first; Artificial Analysis reports this as "time to first answer token."
  • Output speed: Tokens generated per second once output starts. Time per output token (TPOT) is its inverse.
  • Total latency: TTFT plus (TPOT multiplied by output length). What users experience as end-to-end response time.
  • Throughput: Total tokens generated per second across all concurrent requests. The key metric for cost and batch processing.

The latency-throughput tradeoff

LLM inference servers can optimize for one or the other. Serving each request immediately minimizes latency but leaves GPUs underused. Batching multiple requests together maximizes GPU use (throughput) but adds queue wait time (latency). Production serving typically targets a latency SLA and maximizes throughput within that constraint.

Measured speeds for current models

Artificial Analysis measures each provider's API and averages each endpoint's runs over the past five days, so the figures move. Its output speed counts reasoning tokens. Figures from its model pages in early November 2026:

  • Output speed: Claude Haiku 5.5 (max effort) 225 tokens per second, Claude Opus 5.5 (max) 114, Gemini 3.8 Flash (high) 112, GPT-6 Luna (max) 102, GPT-6.1 Terra (max) 74, Gemini 4 Argon (high) 71 and GPT-6.1 Sol (max) 55.
  • Time to first answer token: GPT-6 Luna without reasoning answers in 1.11 seconds and GPT-6.1 Terra without reasoning in 1.75 seconds. With reasoning on, the effort setting dominates: Claude Opus 5.5 takes 6.96 seconds at low effort and 410 seconds at max, and GPT-6.1 Sol takes 2.80 seconds at low and 208 seconds at max.

Practical implications

  • Real-time chat: Time to the first visible token matters most. A wait of more than a second or two before text appears is noticeable, so keep reasoning effort low for interactive turns where the provider allows it.
  • Batch processing: Throughput matters. Run overnight jobs through batch APIs and minimize cost per token.
  • Agentic workflows: Latency compounds over many steps. A 2-second-per-call agent with 20 steps takes 40 seconds minimum, regardless of throughput.

Thinking models add a new kind of latency

Reasoning models spend tokens thinking before they produce visible output, and on several current models thinking is on by default. On Anthropic's API, Claude Haiku 5.5 thinks by default and lets you turn thinking off at high effort or below; Claude Sonnet 5.5 cannot turn it off but can skip up-front thinking; and Claude Opus 5.5 and Claude Fable 5.2 cannot turn it off at all. OpenAI's GPT-6.1 Terra accepts a reasoning effort of "none." For latency-sensitive calls, lower the effort setting first, and remember that thinking tokens are billed as output even when the thinking text is not returned.

Related terms

Models relevant to Latency vs Throughput