SUPERSEDED — this figure illustrates a claim withdrawn in v5.0. It is retained only so the retraction remains checkable. Do not cite it. See RETRACTIONS.md and Appendix L (hardware measurements) and Appendix N (simulation) for what replaced it.
Figure: the KV-cache offloading market landscape in four groups plus a timeline. Commercial hardware available today: Samsung CMM-D up to 512 GB per module and CMM-B up to 16 TB per box at about 60 GB/s and roughly 600 ns loaded; XConn Apollo 256-lane CXL switch with MemVerge GISMO and about 100 TB pool demos; Astera Labs Leo smart memory controller; Marvell Structera near-memory compute. Emerging software frameworks: vLLM with LMCache, SGLang/RadixAttention, NVIDIA Dynamo, Mooncake, FlexGen and InfiniGen. Research built on: PNM-KV (arXiv:2511.00321, November 2025), H2O (arXiv:2306.14048), SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV (arXiv:2407.11550) and HeadKV. In every case policy lives in the serving framework, the GPU model runtime, the host CPU, or a general-purpose CXL device. The combination proposed here — 640 GQA-matched queues, EMA score tracking with alpha 0.2, a RoPE prefetch window of P minus W to P plus W, and placement in CXL endpoint firmware below the serving framework — has not been identified in any published system. The timeline runs 2022 to 2026, colour-coded green for hardware, amber for research and blue for software and specifications. All values are an analytical model, not measured.
KV Cache Offloading: Market Landscape
Where KV-cache policy lives today, and the narrow slice this work adds
CMM-D: up to 512 GB / module
CMM-B: up to 16 TB / box
~60 GB/s (box)
~600 ns loaded
CXL Type-3 memory expansion: an E3.S DDR5 module (CMM-D) and a rack-level pooled box (CMM-B). Media management, RAS and interleaving live in the controller.
Where policy lives: Host/OS tiering decides placement; the device exposes capacity, not an attention-aware cache policy
256-lane CXL switch
~100 TB pool demos
Multi-host
Switched CXL memory pooling with MemVerge GISMO; demonstrated as a KV-cache tier under NVIDIA Dynamo.
Where policy lives: Pool allocation in switch + host software; eviction is block/prefix-level, not per-head
CXL smart memory controller
DDR5 expansion + pooling
COSMOS telemetry
Astera's CXL line: Leo controller, Scorpio switch, Aries retimer, Taurus smart cable module. Leo adds on-controller media management, RAS, security and fleet telemetry.
Where policy lives: Controller logic exists, but it is memory-management logic — it has no view of attention scores or KV-head structure
CXL near-memory compute
Arm cores on device
Expansion + pooling
Puts general-purpose compute next to the DRAM behind CXL — which is why this document does not claim that CXL devices lack on-controller logic.
Where policy lives: Compute is available on-device, applied to general workloads — no published KV-cache eviction policy
PagedAttention
CPU offload
Prefix caching
Block-level KV management with PagedAttention and prefix reuse; LMCache adds CPU/SSD tiers.
Where policy lives: Serving framework, block granularity — above the kernel, not in the device
Radix-tree prefix cache
Automatic KV reuse
Hierarchical tiers
Maintains a radix tree over prompt prefixes so shared prefixes are matched and reused automatically, with LRU eviction over the tree.
Where policy lives: Scheduler, prefix granularity — reuse across requests, not selection within a context
Disaggregated serving
KV-aware routing
Tiered KV offload
Distributed inference framework whose KV-cache manager routes requests to workers already holding the relevant blocks and offloads cold blocks to host memory or storage.
Where policy lives: Cluster control plane, block granularity — placement and routing, not per-head retention
Disaggregated prefill/decode
Global KV store
Distributed KV cache pooled across CPU DRAM + SSD, moved over the network.
Where policy lives: Cluster store; transfers are network-mediated rather than load/store
GPU/CPU/SSD offload
Speculative KV prefetch
FlexGen schedules offload for throughput; InfiniGen speculates which KV entries the next layer needs and prefetches them from host memory.
Where policy lives: Host CPU, over a driver-mediated PCIe path (µs-scale per transfer)
CXL + processing-near-memory
1M tokens
Steady-token selection
Offloads token page selection to a PNM accelerator behind CXL — the closest published work to putting KV-cache decisions on the memory device.
Difference: Token-page granularity on custom PNM silicon; no per-head EMA state, no RoPE-informed prefetch
Attention-score eviction
Heavy hitters
Query-aware selection
Attention-score-based KV eviction and selection is established prior art. This document does not claim to have invented it.
Where policy lives: Inside the model runtime on the GPU, with the attention matrix in hand
Per-head budgets
Head-type awareness
Retrieval/induction heads
Allocating a different cache budget per attention head is likewise established prior art, and is the direct antecedent of the per-head tracking used here.
Where policy lives: Framework-level budget allocation, recomputed per request on the GPU
🗺 Where Cache Policy Lives Today
In the serving framework
vLLM, SGLang/RadixAttention, NVIDIA Dynamo, LMCache: block- and prefix-level admission, reuse and eviction, decided above the kernel.
In the model runtime
H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV: attention-score eviction and per-head budgets, executed on the GPU.
On the host CPU
FlexGen, InfiniGen, Mooncake: offload and speculative prefetch driven by host software over a driver-mediated PCIe path.
On the CXL device
Astera Leo (smart memory controller), Marvell Structera (near-memory compute), Samsung CMM: real on-device logic — for media, pooling and general compute, not for attention-aware retention.
The narrow claim this work makes
Every ingredient above exists in prior art. What we have not identified is a publicly documented system that executes per-head EMA scoring with RoPE-informed prefetch inside CXL controller firmware, below the serving framework. The contribution is that combination and its placement. If such a system is published, this claim should be withdrawn.
🔬 The Combination We Propose
GQA-Aware Architecture
640 queues matching Llama-70B's KV-head × layer structure
EMA Score Tracking
Per-position attention smoothing (α = 0.2) for stable eviction signals
RoPE Prefetch Window
Locality-aware [P−W, P+W] prefetch exploiting position encoding
CXL Controller Placement
Policy executes in endpoint firmware, below the serving framework
Industry Timeline
2022
Samsung CMM-D prototype
CXL 3.0 specification
2023
vLLM PagedAttention
Intel Sapphire Rapids (CXL 1.1)
H2O attention-score eviction
CXL 3.1 specification
2024
Samsung CMM-B (16 TB box)
Astera Labs Leo controllers
SGLang / RadixAttention
SnapKV · Quest · InfiniGen
2025
NVIDIA Dynamo KV manager
XConn + MemVerge GISMO pooling
Marvell Structera near-memory compute
Ada-KV / HeadKV per-head budgets
PNM-KV (arXiv:2511.00321, Nov 2025)
2026
CXL 3.x switching broadens
KV offload becomes a standard serving-stack tier
This work: per-head EMA + RoPE prefetch in endpoint firmware (analytical model)
Evidence label: analytical model — all performance, capacity and cost figures attributed to this work are modeled for one canonical scenario (1×B200, Llama-2-70B, 128K context, ≥10 tok/s). No hardware benchmark was run. Third-party capabilities are vendor specifications or published claims.