Figure: mixture-of-experts routing support with endpoint prefetch. A learned router selects K experts per token; the reference model is DeepSeek-V3 class with N = 256 routed experts and K = 8, so 8 of 256, or 3.1 percent, are active per token and 248 sit idle — the sparsity that makes the working set cacheable. The diagrams are drawn Mixtral-class at N = 8, K = 2 so the routing stays legible, with experts E1 and E3 activated. Because activation is data-dependent, irregular and sparse, the endpoint keeps a routing histogram — 256 bins per layer at reference scale — and prefetches an expert to endpoint DRAM when its activation probability is above threshold, leaving the rest in flash to fetch on demand. Sizing has two cases: if endpoint DRAM holds all experts with replication there is no fetch on the critical path and access is a 200 nanosecond CXL read; otherwise the expected cost is P(hit) times 200 nanoseconds plus P(miss) times 25 microseconds for tier-3 flash. Requests are steered to the endpoint that already holds the needed experts. All values are an analytical model, not measured.

Section 8

MoE Routing Support

Mixture-of-experts with intelligent endpoint prefetch — DeepSeek-V3-class: 256 routed experts, top-8

Mixture-of-Experts Architecture
Input Token
Learned Router
Selects K experts per token
N routed experts total, K activated per token | Reference model: N=256, K=8 (DeepSeek-V3 class)
Diagram above: Illustrative (Mixtral-class) — N=8, K=2, drawn small so the routing is legible
Sparsity at the reference scale: 8 / 256 = 3.1% of routed experts active per token | 248 of 256 experts are idle for any given token — which is exactly what makes the working set cacheable
The Challenge: Data-Dependent Access
Expert activation depends on input content. The router learns which experts handle which types of inputs, but this means access patterns are:

• Data-dependent — can't predict without seeing input
• Irregular — no fixed pattern to exploit
• Sparse — only K of N experts active (8 of 256 at the reference scale)
Expert Activation Pattern (8 tokens)
Rows = tokens, Cols = experts. Orange = activated. Illustrative (Mixtral-class), N=8 / K=2 — the real routing matrix is 256 columns wide with 8 hits per row.
Endpoint Approach: Routing Histogram
Expert Activation Frequency
P(act) threshold

Track frequency per token position or context type. Illustrative (Mixtral-class), 8 bars shown — at the reference scale the endpoint maintains a 256-bin histogram per layer.

Prefetch Strategy
if P(activation) > threshold:
→ Prefetch expert to endpoint DRAM
if P(activation) < threshold:
→ Keep in flash, fetch on demand
Histogram updates continuously based on observed routing decisions. Hot experts stay resident, cold experts stay in flash.
Cache Sizing Strategy
Optimal: Full Replication
If endpoint DRAM can hold all experts with replication, no prefetch needed. Route requests to endpoint with local expert copy.
No expert fetch on the critical path — the expert is already in endpoint DRAM (200 ns CXL access)
Fallback: Histogram Prefetch
If DRAM < all experts, use histogram to prefetch likely experts. Cold experts fetched from flash on demand.
P(hit) × 200 ns + P(miss) × 25 µs (Tier-2 DRAM vs Tier-3 flash)
Route to Endpoint with Local Expert
Request
needs E1, E3
Endpoint A
1
3
5
not
Endpoint B
2
4
6

Analytical model — routing histograms, hit probabilities and latencies on this page are modeled from the architecture, not measured. Expert counts follow a DeepSeek-V3-class model (256 routed experts, top-8); the 8-expert diagrams are illustrative (Mixtral-class). Canonical numbers v4.0.