For Developers/Glossary/MoE (Mixture of Experts)
Architecture

MoE (Mixture of Experts)

A neural network architecture where many specialized sub-networks (experts) exist within a single model, but a router activates only a few of them for each token, enabling large total parameter counts with much lower per-token compute.

Mixture of Experts (MoE) is a neural network architecture where a model contains many specialized sub-networks called "experts" but activates only a small subset for each input token. A learned router decides which experts process each token. This lets a model have an enormous total parameter count while keeping compute per token, and so inference cost, far lower than a dense model of the same size.

Dense vs MoE models

In a dense model such as Gemma 4 31B or Qwen 3.8 27B, every parameter is used for every token. In an MoE model only a fraction is. Published figures for current open-weight MoE models:

  • Kimi K3.1 Pro (Moonshot AI): 2.8T total, 104B active, routing each token to 16 of 896 experts plus 2 shared experts
  • DeepSeek V4 Pro: 1.6T total, 49B active. DeepSeek V4.1 Flash: 552B backbone parameters, with 8B active per token during prefill and 16B during decode, routing each token to 6 of 384 experts plus 1 shared expert
  • Mistral Large 4: about 1T total, 49B active (52B counting embeddings and output layers), with 128 routed experts, 4 active per token, plus 1 shared expert
  • Muse Spark Open (Meta): 816B total, about 42B active, with 256 experts per layer and 8 active per token
  • Llama 4 Maverick: 400B total, 17B active, 128 experts
  • gpt-oss-120b (OpenAI): 117B total, 5.1B active
  • Gemma 4 26B A4B: 25.2B total, 3.8B active

The total parameter count governs how much the model can store; the active count sets compute per token, which drives inference cost and latency. OpenAI, Anthropic and Google do not publish parameter counts for their GPT-6, Claude or Gemini 4 models.

Why MoE matters

MoE lets labs train and serve large models with much less compute than a dense model of the same total size. DeepSeek's V3 technical report put that model's full training at 2.788M H800 GPU hours, about $5.6M at an assumed rental price of $2 per GPU hour, a figure that excludes earlier research and ablation runs. At inference, compute per token is close to that of a dense model the size of the active parameters. Every open-weight model listed above uses MoE, which has become the default design for frontier-scale open models.

MoE challenges

  • Load balancing: If most tokens route to the same few experts, the efficiency gain disappears and expert specialization degrades. Auxiliary loss functions push the router to spread tokens evenly; DeepSeek V3 introduced an auxiliary-loss-free method that adjusts per-expert bias terms instead.
  • Communication overhead: In distributed inference, routing tokens to experts on different devices or accelerators adds network latency and synchronization costs.
  • Memory requirements: All expert weights must be loaded even though only a few are active per token. Muse Spark Open uses about 42B parameters per token but needs all 816B in memory, roughly 1.6TB at 16-bit precision, which is why labs often also publish FP8 or 4-bit checkpoints.
  • Training complexity: Routing mechanisms and load-balancing strategies need careful tuning, and discrete top-k expert selection is not fully differentiable, which calls for specialized training techniques.

Related terms

Models relevant to MoE (Mixture of Experts)