Mixture of Experts (MoE) Inference

MoE architecture, top-K expert routing, expert parallelism, load balancing, all-to-all communication, and DeepSeek MoE optimization patterns for efficient inference at scale.

Overview

Mixture of Experts (MoE) models replace the dense feed-forward network (FFN) in each transformer layer with N expert FFNs and activate only a few (typically top-2) per token. Parameter count grows without a matching growth in per-token compute. DeepSeek-V3 (671B parameters, 37B active per token) and Mixtral 8x7B/8x22B are the prominent examples.

Capacity versus compute

Parameters — 1x — ~8x
Experts run per token — One FFN — 2 experts
Compute per token — Baseline — Similar to dense

MoE architecture

Each MoE layer has a router (a learned linear layer plus softmax/topK) that scores the N experts for each token, and N expert FFNs (typically 2-4 layer MLPs) with the same architecture but independent weights. With top-2 routing, a token runs through its two best experts and their outputs are combined, weighted by routing score.

Total parameters = N * expert_params + router_params + all other components. DeepSeek-V3 has 671B total and 37B active per token; the rest sit in experts not selected for that token.

Expert parallelism and All-to-All

Large MoE models cannot fit every expert on one GPU. Expert Parallelism (EP) gives each GPU N/EP_size experts, so a token routed to an expert on another GPU must send its hidden state there.

> [!WARNING] All-to-All volume scales as O(batch_size * hidden_dim * EP_size). With EP=64 GPUs a single All-to-All can move tens of gigabytes per MoE layer, so expert grouping and locality-aware scheduling are critical.

Load imbalance

Some experts become consistently popular, so their GPUs process many more tokens per step. Step time is set by the busiest GPU, so imbalance directly lowers throughput.

Auxiliary load-balancing loss — Training — Extra loss term that encourages uniform expert use
Expert capacity factor — Per step — Caps tokens per expert; excess tokens are dropped or sent to a shared expert
Dynamic load-aware routing — Inference — Rebalances routing probabilities online from current load

DeepSeek MoE innovations (V2 and V3)

Fine-grained expert segmentation — More, smaller experts: less compute per expert, finer routing
Shared experts — A subset always active for every token gives a baseline FFN quality; routed experts handle specialized content
Expert grouping — Assigns tokens to experts so tokens stay on their GPU when possible, cutting All-to-All volume
MLA (Multi-head Latent Attention) — Projects K/V through low-rank matrices before caching, shrinking the KV cache by ~4x

Inference framework support

vLLM's Fused MoE kernels combine routing, expert dispatch and expert compute in one GPU kernel, so per-expert input batches are never materialized.

> [!TIP] In decode (batch size 1 to small), many experts get zero tokens in a step; the fused kernel skips them entirely instead of computing on empty batches.

Interview questions

How would you serve DeepSeek-V3 (671B params, 37B active) efficiently on a multi-node H100 cluster?