This chapter lays out the mechanics that the rest of this package assumes as given: how autoregressive decoding uses a key/value (KV) cache, how attention head layout (MHA, GQA, MQA) determines that cache's memory footprint, why KV data is safe to move once it has been written, and the memory-hierarchy tradeoffs that make offloading it to CXL-attached memory an interesting design point rather than a curiosity.
A transformer decoder generates text one token at a time. At each step, the model computes attention over every token that came before it in the sequence — for each query, it needs the keys and values of all prior positions. Recomputing those keys and values from scratch at every generation step would mean redoing the full forward pass over the entire prefix for every new token, which scales quadratically with sequence length and is wasteful: the keys and values for a given token, once computed, do not change. The standard optimization is to compute them once and cache them, appending the new token's K and V to the cache at each step and reusing everything already stored.
Inference under this scheme splits into two phases with very different performance characteristics:
Because decode re-reads the full KV cache on every single step, and the cache grows with sequence length and number of concurrent sequences, KV cache size becomes the dominant constraint on both how many requests can be served concurrently and how fast each one decodes. This is the pressure that motivates everything downstream in this package: if the KV cache does not fit in fast on-accelerator memory, something has to give.
The size of the KV cache is determined not by the number of query heads a model uses, but by the number of key/value heads — a distinction that head-layout choices deliberately exploit.
The exact formula for KV cache size, in bytes, for one token of context is:
The leading factor of 2 accounts for storing both K and V. layers is the number of transformer blocks, kv_heads is the number of key/value heads (not query heads), head_dim is the dimensionality of each head, and bytes_per_element depends on the numeric precision used for the cache (2 bytes for FP16/BF16, 1 byte for INT8, etc.). The load-bearing variable here is kv_heads: two models with similar total parameter counts and similar query-head counts can have very different KV footprints purely because of how many KV heads each uses. This is why "model size" alone is a poor predictor of KV cache pressure — head layout has to be checked directly.
| Model | Attention Layout | KV Heads | KV Cache per Token |
|---|---|---|---|
| Llama-1 65B | MHA | 64 | 2560 KiB |
| Llama-2 / Llama-3 70B | GQA | 8 | 320 KiB |
| Llama-3 8B | GQA | 8 | 128 KiB |
| Qwen2.5-7B | GQA | 4 | 56 KiB |
These figures are analytically derived from each model's published configuration (layer count, KV head count, head dimension) using the formula above; they are not measured from a running system. The gap between Llama-1 65B's MHA layout and Llama-3 70B's GQA layout — an 8× reduction in per-token KV cost between two models of roughly comparable scale — illustrates why head layout, not parameter count, is the number to check first when sizing a KV cache budget.
Causal attention means a token's query can only attend to keys and values at or before its own position — never forward. Consequently, once the key and value vectors for a given token are computed during prefill or the decode step that produced that token, they are fixed for the remainder of that sequence's generation: no later computation ever revises them. A KV entry is therefore written exactly once and, from that point forward, is only ever read — once per subsequent decode step for as long as it remains resident and part of an active sequence's attention context.
This write-once property is what makes KV data cacheable and migratable in the first place. It means a KV block can be safely copied, moved to a slower memory tier, or evicted and later reloaded without any risk of it going stale mid-sequence — there is no writer to race against. It does not by itself imply any fixed ratio of total read traffic to write traffic; that ratio depends on how many decode steps remain and how long a block stays resident and relevant, which is workload-dependent. What is structurally guaranteed is only the asymmetry: one write, followed by zero or more reads, never followed by another write to that same entry.
Accelerator memory (GPU HBM) sits at one end of the hierarchy: bandwidth in the range of several hundred GB/s to a few TB/s, but capacity measured in tens of gigabytes per device — a hard ceiling shared with model weights, activations, and everything else that needs to live there during inference. Host DRAM, and more recently CXL-attached memory, sit further out: capacity can be an order of magnitude or more larger, but that memory is reached over an interconnect — PCIe-based CXL links in the case of CXL memory — with materially lower bandwidth than HBM and non-trivial added latency per access compared to a local memory controller.
This is the classic capacity-versus-bandwidth tradeoff, and it is the structural reason tiered KV placement exists as a design space at all: keep the KV data that is about to be read on the fast, capacity-constrained tier; push KV data that will not be needed again soon onto the larger, slower tier; and move data between tiers before it is needed rather than after, wherever that is achievable. None of the specific policies for making those decisions are introduced here — this chapter is only establishing that the tradeoff is real and quantifiable, not proposing how to navigate it.
When the fast tier is full and new KV data needs to be written, something already resident has to leave. The decision of what leaves is the eviction policy. This package discusses eviction policy design in depth in a later chapter; here it is enough to name the standard reference points from the caching literature by acronym, since they recur throughout:
Beyond these two textbook baselines, decayed-frequency and exponential-moving-average (EMA) style scoring — where access frequency is tracked but weighted to favor recent activity over old activity — is a well-established general technique in caching literature. The specific formulation and citation used for KV-cache eviction in this package is deferred to the KV-cache management chapter, where it is evaluated directly against LRU and LFU baselines.