A scoping chapter: why the architecture argument in this package needs to say something about MoE, and why nothing here is a measurement.
Every hardware measurement in this package — the decode bandwidth decomposition in Chapter 4, the KV byte-cost ratios, the persistence result in Chapter 7 — was collected against Qwen2.5-7B-Instruct on GB10, a dense transformer. Dense means every layer's feedforward block is a single feedforward network, and every token that passes through a layer activates all of that layer's weights. A growing share of the models this architecture will actually serve are not built that way, and it would be an omission to leave that unaddressed. This chapter explains what mixture-of-experts (MoE) architectures are, why they matter to the KV-cache argument, and — critically — why they mostly don't.
In a dense transformer layer, the feedforward sub-block is one network: every token's hidden state is pushed through the same weights. An MoE layer replaces that single feedforward network with a bank of many smaller feedforward networks, called experts, plus a small router network. For each token, the router scores the experts and selects a small subset — commonly the top-k for some small k, such as top-2 or top-8 — and only those selected experts process that token. The token's output is a combination of the outputs of the experts it was routed to.
The consequence is a decoupling of two quantities that are the same thing in a dense model: total parameter count and per-token active parameter count. An MoE model can hold a very large number of experts — order of hundreds in some published architectures — giving it a large total weight footprint, while any single token only ever touches a small fraction of those weights, giving it a much smaller active-compute footprint per token than a dense model of equivalent total size. This is the entire point of the architecture: more capacity without a proportional increase in per-token FLOPs.
The routing and expert-selection machinery described above lives entirely in the feedforward sub-block of each layer. It has no direct relationship to the attention sub-block — the part of the layer that computes and consumes the KV cache. In the large majority of published MoE LLMs, attention is architecturally identical to what a dense model of comparable scale would use: still grouped-query attention (GQA) in most current designs, with the same per-token key and value vector shapes, the same per-layer KV growth with sequence length, and the same cache eviction and residency behavior this package has been describing throughout. MoE changes which expert weights get read on the feedforward side of a token's pass through a layer. It does not change how keys and values are computed, stored, or evicted on the attention side.
This separation is the load-bearing claim of this chapter: the two-term decode model developed in Chapter 4 — a KV-read term and a weight-read term, both bandwidth-bound at low batch sizes — and the KV byte-cost accounting built on top of it in Chapter 7 describe the attention path. Because MoE is a feedforward-path modification, those arguments are, in structure, largely orthogonal to whether the underlying model is dense or MoE. The router adds a small amount of extra compute and a data-dependent memory access pattern into the weight-read term, but it does not touch the KV-read term or the logic that governs what stays resident in the cache.
Several current large-scale model families — architectures in the general class of DeepSeek-V3, as one publicly documented example — use MoE designs with a large number of experts (order of hundreds) and top-k routing (commonly cited around top-8), producing models whose total parameter count is far larger than their per-token active parameter count. Readers should treat any specific figures for a given model family as needing independent verification against that model's current published documentation; architectures and configurations in this space have continued to change through 2026.
The structural point that follows is a hypothesis, not a tested claim. In the two-term decode model of Chapter 4, the weight-read term is driven by how many weight bytes must be resident and available to be read, not strictly by how many are read per token. An MoE model with a large total expert count has a much larger resident weight footprint than a dense model with the same active-parameter count, even though its per-token compute is sparse. If that resident footprint puts more pressure on the memory hierarchy — more weight bytes competing for the same fast-tier capacity that KV cache also wants — the fixed-path, weight-read side of the decode-time bandwidth budget would plausibly become more dominant on an MoE model, not less, relative to the KV-read term. That would, if it holds, make the case for this package's overall architecture stronger on MoE models than on the dense model actually measured here, since offloading strategy has to account for a KV term that is being squeezed by a larger, not smaller, competing weight-read term.
The validation step this chapter points to is direct: repeat the Chapter 4 decode-time bandwidth decomposition — the same KV-read/weight-read split, the same batch-size sweep — on a current MoE model, and compare the resulting term ratios against the dense Qwen2.5-7B-Instruct baseline used throughout this package. A second, smaller check would confirm that KV cache size and shape for the MoE model's attention layers track the dense-model formulas in Chapter 7 without modification, since that is the assumption §8.2 rests its "orthogonal" claim on. Neither has been run. Until it is, MoE should be read in this package as scoping and motivation for why the architecture matters broadly, not as a second evidence base alongside the dense-model measurements in Chapters 4 through 7.