Why the key/value cache has become a first-class resource problem, and how this document tries to talk about it honestly.
As context windows have grown from a few thousand tokens to hundreds of thousands, the attention mechanism's key/value (KV) cache has grown with them — linearly, and without the option to shrink back down mid-generation. Model weights are fixed once a checkpoint is loaded; the KV cache is not. It accumulates one token at a time for the life of a sequence, and at long context lengths it stops being a minor bookkeeping cost and starts being the dominant consumer of accelerator memory. This document is about what happens when that growth outruns the memory tier it was originally designed to live in, and about one specific way of responding to that: adding a second, larger, slower memory tier beneath high-bandwidth memory (HBM) rather than trying to make the KV cache smaller.
The numbers are easiest to reason about with a specific model. Llama-3 70B uses grouped-query attention (GQA) with 8 KV heads, which keeps its per-token KV footprint smaller than a full multi-head design would — 320 KiB per token, across all layers, in bf16. That per-token cost is small. It is the multiplication by sequence length that changes the picture:
| Context length | KV cache size (1 sequence) | % of model weights (130.4 GiB) |
|---|---|---|
| 8K tokens | 2.5 GiB | ~1.9% |
| 32K tokens | 10.0 GiB | ~7.7% |
| 128K tokens | 40.0 GiB | ~31% |
At 128K context, a single sequence's KV cache is already equivalent to nearly a third of the entire 70-billion-parameter model's weight footprint (140 GB in bf16, or 130.4 GiB). Extend the same arithmetic further and the KV cache overtakes the weights entirely at roughly 427,000 tokens for a single sequence — a context length that is no longer exotic for document-scale and agentic workloads. None of this accounts for batching multiple sequences concurrently, which multiplies the KV footprint directly. The conclusion is unremarkable but easy to underweight: at long context and realistic batch sizes, KV cache is not a secondary cost sitting alongside the model — for large fractions of real deployments, it is comparable to the model.
The obvious response to a capacity problem is more capacity, and CXL-attached memory — sitting below HBM in the hierarchy but above local SSD or network storage — is a reasonable way to get it: large pools, attached with load/store semantics rather than a storage-style I/O path. This package evaluates exactly that kind of tier. But the central argument running through it is that adding a tier changes the shape of the KV memory problem rather than removing it.
Capacity pressure is genuinely relieved — that is what the tier is for, and it is not a small effect. What does not change is HBM-side bandwidth: attention still has to re-read the active portion of the KV cache out of HBM on every decode step, regardless of which tier the cold portion of that cache lives in at rest. And two problems appear that a single-tier system never had to solve: getting KV data across the tier boundary fast enough to keep up with the model's consumption rate (tier-ingress bandwidth), and completing any individual transfer before the compute pipeline stalls waiting on it (per-transfer latency). These are not the same problem wearing two names — a link can have plenty of aggregate bandwidth and still cause stalls if individual transfers are too slow relative to the compute budget between them, and the reverse is also possible. In short: a memory tier turns a capacity wall into a data-movement wall. That reframing is the organizing claim of this entire document, and the chapters that follow are built around examining each side of it — bandwidth, latency, and the eviction and placement decisions that determine what has to move and when.
A document that argues numbers matter should be careful about where its own numbers come from. Three different kinds of claim appear throughout this package, and they are tagged as such rather than left to blend together: Measured results come from runs on real hardware, specifically a DGX Spark GB10 system; Simulated results come from models of hardware behavior that were not directly run on the target platform, and are marked provisional accordingly; and Analytical results are derived from first-principles arithmetic — like the KV-size table above — without a corresponding hardware or simulation run. Where a chapter mixes these, the tags stay attached to the specific numbers they cover rather than to the chapter as a whole. This is not a hedge added for its own sake; it is a response to how easily offloading and memory-tiering claims get stated in the strong form ("solves," "eliminates") when the underlying support is a model or a back-of-envelope estimate. Readers should be able to tell, claim by claim, how much weight a given number can bear.
It is worth being explicit about scope, because it is narrower than "a faster offloading system." This package does not propose a new eviction-scoring algorithm — deciding which tokens' KV entries are least likely to be needed again is a well-studied problem, and existing policies are treated as inputs here, not as something being replaced. The contribution under test is at the protocol level: when KV data migrates across a tier boundary, the metadata an eviction policy has accumulated about that data — recency, access counts, whatever the policy tracks — should migrate with it as a well-defined, portable object, rather than being discarded and rebuilt from scratch on the other side. That claim is evaluated independent of whether CXL hardware is present in the test environment, because the protocol question — does the metadata survive the boundary crossing correctly and usefully — is separable from the performance question of what a specific interconnect can deliver.
The chapters ahead work through this in order: background on inference and why KV caches exist at all (Chapter 2), the architecture under study (Chapter 3), latency and bandwidth characterization on real hardware (Chapters 4–5), and the preprocessing, eviction-metadata, and integration work that follows from the tier-boundary problem stated here (Chapters 6 onward). None of it is presented as a finished, deployed system. It is presented as a set of measurements and one narrow, testable claim, with the evidence for each kept visibly separate from the arithmetic.