MoE (Mixture of Experts)
A neural network architecture where many specialized sub-networks (experts) exist within a single model, but a router activates only a few of them for each token, enabling large total parameter counts with much lower per-token compute.
Mixture of Experts (MoE) is a neural network architecture where a model contains many specialized sub-networks called "experts" but activates only a small subset for each input token. A learned router decides which experts process each token. This lets a model have an enormous total parameter count while keeping compute per token, and so inference cost, far lower than a dense model of the same size.
Dense vs MoE models
In a dense model such as Gemma 4 31B or Qwen 3.8 27B, every parameter is used for every token. In an MoE model only a fraction is. Published figures for current open-weight MoE models:
- Kimi K3.1 Pro (Moonshot AI): 2.8T total, 104B active, routing each token to 16 of 896 experts plus 2 shared experts
- DeepSeek V4 Pro: 1.6T total, 49B active. DeepSeek V4.1 Flash: 552B backbone parameters, with 8B active per token during prefill and 16B during decode, routing each token to 6 of 384 experts plus 1 shared expert
- Mistral Large 4: about 1T total, 49B active (52B counting embeddings and output layers), with 128 routed experts, 4 active per token, plus 1 shared expert
- Muse Spark Open (Meta): 816B total, about 42B active, with 256 experts per layer and 8 active per token
- Llama 4 Maverick: 400B total, 17B active, 128 experts
- gpt-oss-120b (OpenAI): 117B total, 5.1B active
- Gemma 4 26B A4B: 25.2B total, 3.8B active
The total parameter count governs how much the model can store; the active count sets compute per token, which drives inference cost and latency. OpenAI, Anthropic and Google do not publish parameter counts for their GPT-6, Claude or Gemini 4 models.
Why MoE matters
MoE lets labs train and serve large models with much less compute than a dense model of the same total size. DeepSeek's V3 technical report put that model's full training at 2.788M H800 GPU hours, about $5.6M at an assumed rental price of $2 per GPU hour, a figure that excludes earlier research and ablation runs. At inference, compute per token is close to that of a dense model the size of the active parameters. Every open-weight model listed above uses MoE, which has become the default design for frontier-scale open models.
MoE challenges
- Load balancing: If most tokens route to the same few experts, the efficiency gain disappears and expert specialization degrades. Auxiliary loss functions push the router to spread tokens evenly; DeepSeek V3 introduced an auxiliary-loss-free method that adjusts per-expert bias terms instead.
- Communication overhead: In distributed inference, routing tokens to experts on different devices or accelerators adds network latency and synchronization costs.
- Memory requirements: All expert weights must be loaded even though only a few are active per token. Muse Spark Open uses about 42B parameters per token but needs all 816B in memory, roughly 1.6TB at 16-bit precision, which is why labs often also publish FP8 or 4-bit checkpoints.
- Training complexity: Routing mechanisms and load-balancing strategies need careful tuning, and discrete top-k expert selection is not fully differentiable, which calls for specialized training techniques.
Related terms
Models relevant to MoE (Mixture of Experts)
Kimi K3.1 Pro
Moonshot AI's open-weight 1M-context flagship for long-horizon coding
View model →DeepSeek V4
Open-weight 1.6T-param MoE frontier model with a 1M-token context built for agents.
View model →DeepSeek V4 Flash
DeepSeek's 284B open-weight Flash model, replaced by V4.1 Flash in September 2026
View model →Mistral Large 4
Mistral's 1-trillion-parameter open-weight flagship, nicknamed 'le Chonk'
View model →Muse Spark Open
Meta's open-weight release of Muse Spark: an 816B multimodal MoE
View model →Llama 4
Meta's natively multimodal open-weight MoE models, with up to 10M tokens of context on Scout.
View model →