SUPERSEDED — this figure illustrates a claim withdrawn in v5.0. It is retained only so the retraction remains checkable. Do not cite it. See RETRACTIONS.md and Appendix L (hardware measurements) and Appendix N (simulation) for what replaced it.
Figure: a map of where KV-cache retention policy lives today, in four bands. Commercial products (Samsung CMM-D up to 512 GB per module and CMM-B up to 16 TB per box at about 60 GB/s and 600 ns loaded; XConn Apollo 256-lane switch with MemVerge GISMO; Astera Labs Leo; Marvell Structera) plus the serving frameworks vLLM/LMCache, SGLang/RadixAttention and NVIDIA Dynamo. Related work built on: PNM-KV (arXiv:2511.00321), H2O (arXiv:2306.14048), SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV (arXiv:2407.11550), HeadKV, InfiniGen, Mooncake and FlexGen. The gap: policy today runs in the serving framework, the GPU model runtime, the host CPU, or a general-purpose CXL device — never as per-head EMA scoring with RoPE-informed prefetch inside CXL controller firmware, below the serving framework, which is the narrow slice this work adds through per-head eviction over 640 GQA queues, EMA scoring with alpha 0.2, RoPE locality prefetch and model-architecture awareness. All values are an analytical model, not measured.
CMM-D: up to 512 GB / module
CMM-B: up to 16 TB / box
~60 GB/s (box)
~600 ns loaded
CXL Type-3 memory expansion: an E3.S DDR5 module (CMM-D) and a rack-level pooled box (CMM-B). Media management, RAS and interleaving live in the controller.
Policy lives in host/OS tiering — the device exposes capacity, not an attention-aware cache policy
256-lane switch
~100 TB pool demos
NVIDIA Dynamo
Switched CXL memory pooling with MemVerge GISMO; demonstrated as a KV-cache tier under NVIDIA Dynamo.
Policy lives in switch + host software — block/prefix eviction, not per-head
DDR5 expansion + pooling
COSMOS telemetry
Astera's CXL line: Leo controller, Scorpio switch, Aries retimer, Taurus smart cable module. Leo runs real on-controller logic for media management, RAS, security and fleet telemetry.
Controller logic is memory management — no view of attention scores or KV-head structure
Arm cores on device
Expansion + pooling
Puts general-purpose compute next to DRAM behind CXL — which is why this document does not claim that CXL devices lack on-controller logic.
On-device compute exists, but no published KV-cache eviction policy
PagedAttention
CPU/SSD offload
Industry-standard inference framework with block-level KV management and prefix reuse.
Policy lives in the serving framework at block granularity
Radix-tree prefix cache
Automatic KV reuse
Radix tree over prompt prefixes matches and reuses shared prefixes automatically, with LRU eviction over the tree.
Reuse across requests — not selection within a single long context
KV-aware routing
Tiered KV offload
Disaggregated serving with a KV-cache manager that routes requests to workers already holding the relevant blocks and offloads cold blocks to host memory or storage.
Cluster control plane, block granularity — placement and routing, not per-head retention
arXiv:2511.00321 — Nov 2025
PNM-KV
CXL-enabled processing-near-memory that offloads token page selection to a PNM accelerator — the closest published work to putting KV decisions on the memory device.
Token-page granularity, custom PNM silicon
H2O: arXiv:2306.14048
H2O · SnapKV · Scissorhands · PyramidKV · Quest
Attention-score-based eviction and query-aware KV selection. Established prior art — this document does not claim to have invented it.
Policy runs in the GPU model runtime
Ada-KV: arXiv:2407.11550
Ada-KV · HeadKV
Per-head cache budget allocation and head-type awareness — the direct antecedent of the per-head tracking used here.
Framework-level per-head budgets
OSDI '24 · 2024 · 2023
InfiniGen · Mooncake · FlexGen
Speculative KV prefetch from host memory, disaggregated prefill/decode with a global KV store, and offload scheduling across GPU/CPU/SSD.
Policy runs on the host CPU
2023 – 2025
vLLM · SGLang/RadixAttention · NVIDIA Dynamo
Block- and prefix-level admission, reuse, routing and tiered offload inside the serving framework.
Policy runs above the kernel
Commercial CXL
Marvell Structera · Astera Labs Leo
Near-memory compute and smart memory control already exist on CXL devices; what is absent is an attention-aware retention policy on them.
On-device logic, general purpose
🧩
Serving framework
vLLM · SGLang/RadixAttention · Dynamo · LMCache — block and prefix granularity
📈
Model runtime (GPU)
H2O · SnapKV · Scissorhands · PyramidKV · Quest · Ada-KV · HeadKV
🖥
Host CPU
FlexGen · InfiniGen · Mooncake — driver-mediated PCIe offload
🔌
CXL device
Astera Leo · Marvell Structera · Samsung CMM — media, pooling, general compute
🔍
The narrow claim: every ingredient above is prior art. What we have not identified is a publicly documented system that executes per-head EMA scoring with RoPE-informed prefetch inside CXL controller firmware, below the serving framework. The contribution is that combination and its placement — if such a system is published, this claim should be withdrawn.
✓
Per-KV-Head Eviction
Track 640 GQA queues independently, evict at head granularity
✓
EMA-Based Scoring
Smooth attention scores over time (α = 0.2), prevent thrashing
✓
RoPE Locality Prefetch
Exploit position encoding structure for predictive fetch
✓
Model-Architecture Aware
Understands transformer structure, not just memory pages
Evidence label: analytical model — all performance, capacity and cost figures attributed to this work are modeled for one canonical scenario (1×B200, Llama-2-70B, 128K context, ≥10 tok/s). No hardware benchmark was run. Third-party capabilities are vendor specifications or published claims.