Figure: the competitive landscape for KV-cache offloading, in four bands plus an industry timeline. Commercial hardware available today: Samsung CMM-D up to 512 GB per module and CMM-B up to 16 TB per box at about 60 GB/s and roughly 600 ns loaded; the XConn Apollo 256-lane CXL switch with MemVerge GISMO and about 100 TB pool demos; the Astera Labs Leo smart memory controller; and Marvell Structera near-memory compute. Emerging software frameworks and academic papers place cache policy in the serving framework, the GPU model runtime, or the host CPU. Where cache policy lives today, in four cards: the serving framework (vLLM, SGLang/RadixAttention, NVIDIA Dynamo, LMCache), the model runtime (H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV), the host CPU (FlexGen, InfiniGen, Mooncake) and the CXL device (Astera Leo, Marvell Structera, Samsung CMM) — none of them attention-aware retention in controller firmware. The combination proposed here is four parts: 640 GQA-matched queues, EMA score tracking with alpha 0.2, a RoPE prefetch window of P minus W to P plus W, and policy execution in CXL endpoint firmware below the serving framework, giving a modeled 97 percent HBM hit rate and 16 concurrent users at 128K context on one B200 without custom silicon. The timeline runs 2022 to 2026, colour-coded green for hardware, amber for software, purple for research and blue for standards. All values are an analytical model, not measured.

KV-Cache Offloading: Competitive Landscape

Where KV-cache policy lives today, and the narrow slice this work adds

Available Today Commercial Hardware
Hardware
CMM-D / CMM-B
Samsung
CMM-D: up to 512 GB / module CMM-B: up to 16 TB / box ~60 GB/s (box) ~600 ns loaded
CXL Type-3 memory expansion: an E3.S DDR5 module (CMM-D) and a rack-level pooled box (CMM-B). Media management, RAS and interleaving live in the controller.
Where policy lives: Host/OS tiering decides placement; the device exposes capacity, not an attention-aware cache policy
Hardware
Apollo CXL Switch + GISMO
XConn + MemVerge
256-lane CXL switch ~100 TB pool demos Multi-host
Switched CXL memory pooling with MemVerge GISMO; demonstrated as a KV-cache tier under NVIDIA Dynamo.
Where policy lives: Pool allocation in switch + host software; eviction is block/prefix-level, not per-head
Hardware
Leo (P-/E-Series)
Astera Labs
CXL smart memory controller DDR5 expansion + pooling COSMOS telemetry
Astera's CXL memory line: Leo controller, Scorpio switch, Aries retimer, Taurus smart cable. Leo adds on-controller media management, RAS, security and fleet telemetry.
Where policy lives: Controller logic exists, but it is memory-management logic — it has no view of attention scores or KV-head structure
Hardware
Structera
Marvell
CXL near-memory compute Arm cores on device Expansion + pooling
CXL device family that puts general-purpose compute next to the DRAM, which is why "no CXL device has on-controller logic" is not a claim this document makes.
Where policy lives: Compute is available on-device, but is applied to general workloads — no published KV-cache eviction policy
Emerging Software Frameworks
Software
vLLM + LMCache
UC Berkeley / Open Source
PagedAttention CPU offload Prefix caching
Block-level KV management with PagedAttention and prefix reuse; LMCache adds CPU/SSD tiers.
Where policy lives: Serving framework, block granularity — above the kernel, not in the device
Software
SGLang / RadixAttention
Open Source
Radix-tree prefix cache Automatic KV reuse Hierarchical tiers
Keeps a radix tree over prompt prefixes so shared prefixes are matched and reused automatically, with LRU eviction over the tree.
Where policy lives: Scheduler, prefix granularity — reuse across requests, not selection within a context
Software
Dynamo
NVIDIA
Disaggregated serving KV-aware routing Tiered KV offload
Distributed inference framework with a KV-cache manager that routes requests to workers already holding the relevant blocks and offloads cold blocks to host memory or storage.
Where policy lives: Cluster control plane, block granularity — placement and routing, not per-head retention
Software
Mooncake
Moonshot AI
Disaggregated prefill/decode Global KV store
Distributed KV cache pooled across CPU DRAM + SSD, moved over the network.
Where policy lives: Cluster store; transfers are network-mediated rather than load/store
Software
FlexGen · InfiniGen
Stanford · SNU (OSDI '24)
GPU/CPU/SSD offload Speculative KV prefetch
FlexGen schedules offload for throughput; InfiniGen speculates which KV entries the next layer will need and prefetches them from host memory.
Where policy lives: Host CPU, over a driver-mediated PCIe path (µs-scale per transfer)
Research we build on Academic Papers
Paper
PNM-KV
arXiv:2511.00321 — November 2025
CXL + processing-near-memory 1M tokens Steady-token selection
Offloads token page selection to a PNM accelerator behind CXL — the closest published work to putting KV-cache decisions on the memory device.
Difference: Token-page granularity on custom PNM silicon; no per-head EMA state, no RoPE-informed prefetch
Paper
H2O · SnapKV · Scissorhands · PyramidKV · Quest
H2O: arXiv:2306.14048 (2023)
Attention-score eviction Heavy hitters Query-aware selection
Attention-score-based KV eviction and selection is established prior art. This document does not claim to have invented it.
Where policy lives: Inside the model runtime on the GPU, with the full attention matrix in hand
Paper
Ada-KV · HeadKV
Ada-KV: arXiv:2407.11550 (2024)
Per-head budgets Head-type awareness Retrieval/induction heads
Allocating a different cache budget per attention head is likewise established prior art, and is the direct antecedent of the per-head tracking used here.
Where policy lives: Framework-level budget allocation, recomputed per request on the GPU
Framing Where Cache Policy Lives Today
In the serving framework
vLLM, SGLang/RadixAttention, NVIDIA Dynamo, LMCache: block- and prefix-level admission, reuse and eviction, decided above the kernel.
In the model runtime
H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV: attention-score eviction and per-head budgets, executed on the GPU.
On the host CPU
FlexGen, InfiniGen, Mooncake: offload and speculative prefetch driven by host software over a driver-mediated PCIe path.
On the CXL device
Astera Leo (smart memory controller), Marvell Structera (near-memory compute), Samsung CMM: real on-device logic — for media, pooling and general compute, not for attention-aware retention.

The narrow claim this work makes

Each ingredient below exists in prior art. What we have not identified is a publicly documented system that executes per-head EMA scoring with RoPE-informed prefetch inside CXL controller firmware, below the serving framework. The contribution is that combination and its placement — not the individual mechanisms. If such a system is published, this claim should be withdrawn.

This work The Combination We Propose
GQA-Aware Architecture
640 queues matching Llama-70B's KV-head × layer structure
EMA Score Tracking
Per-position attention smoothing (α = 0.2) for stable eviction signals
RoPE Prefetch Window
Locality-aware [P-W, P+W] prefetch exploiting position encoding
CXL Controller Placement
Policy executes in endpoint firmware, below the serving framework
Industry Timeline
2022
Samsung CMM-D prototype CXL 3.0 specification
2023
vLLM PagedAttention Intel Sapphire Rapids (CXL 1.1) H2O attention-score eviction CXL 3.1 specification
2024
Samsung CMM-B (16 TB box) Astera Labs Leo controllers SGLang / RadixAttention SnapKV · Quest · InfiniGen
2025
NVIDIA Dynamo KV manager XConn + MemVerge GISMO pooling Marvell Structera near-memory compute Ada-KV / HeadKV per-head budgets PNM-KV (arXiv:2511.00321, Nov 2025)
2026
CXL 3.x switching broadens KV offload becomes a standard serving-stack tier This work: per-head EMA + RoPE prefetch in endpoint firmware (analytical model)
A CXL-Native Placement for Known KV-Cache Techniques
Established attention-aware eviction and per-head budgeting, moved below the serving framework into endpoint firmware: a modeled 97% HBM hit rate and 16 concurrent users at 128K on one B200 — without custom silicon.
Per-head granularity Attention-weighted eviction RoPE-aware prefetch Controller-resident logic

Evidence label: analytical model — 97% hit rate, 16× user capacity and all cost figures are modeled for a single canonical scenario (1×B200, Llama-2-70B, 128K context, ≥10 tok/s). No hardware benchmark was run. Third-party capabilities are vendor specifications or published claims.