Figure: a map of where KV-cache retention policy lives today, in four bands, organised by the decision each layer optimises rather than by any gap. The field did not leave a gap; it specialised, and the earlier "nobody has done this" framing is retired outright. Commercial products: Samsung CMM-D up to 512 GB per module and CMM-B up to 16 TB per box; the XConn Apollo 256-lane switch with MemVerge GISMO; Astera Labs Leo; Marvell Structera. Serving frameworks: vLLM with PagedAttention, the baseline every offload layer sits on; LMCache, now a first-class cross-tier KV layer with pinning, lookup, compression, batched movement, compute and I/O pipelining, cross-engine transfer and prefill-decode disaggregation; SGLang with RadixAttention; NVIDIA Dynamo. TraCT is a rack-scale, prefix-aware CXL KV cache implemented on Dynamo with direct GPU-to-CXL operations — CXL-native KV caching already ships, so this package is not the sole or first CXL-native KV-cache system. PNM-KV selects which KV pages are needed before paying the movement cost, from a processing-near-memory accelerator behind CXL. CacheWise predicts reuse from agent and tool-call state. Tutti removes the CPU from the GPU-to-NVMe KV I/O path. Related work this package builds on rather than competes with: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV, InfiniGen, LayerKV, KVPR and PagedAttention. What is genuinely open is not a missing implementation but a missing unification: whether one KV-State Control Plane — SELECT, PLACE, MOVE, ATTEND, OBSERVE — should decide relevance, reuse, residency and movement jointly instead of leaving each to a separate subsystem. Nobody, including this package, has built or tested that. One box inside it has evidence, and it is simulated and provisional: inside PLACE, letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU, from 15.56 to 49.71 percent, and plus 24.85 to EMA alpha 0.01, from 31.54 to 56.39 percent, at a 32 gibibyte budget, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points — larger than the spread between policies, and testable with no CXL hardware. Retracted from this figure: per-head eviction over 640 GQA queues, since every head of every layer is required for every token and evicting a head produces wrong outputs rather than a cache miss, and the tracked unit is a (sequence, token block) of 16 tokens, 5.00 mebibytes, with per-head signals used only for scoring; alpha = 0.2, never validated, superseded by a simulated optimum near 0.05 with non-monotonic behaviour and no convergence onto LFU as alpha approaches zero; RoPE locality prefetch, since RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next; and the 97 percent hit rate and 16 times user capacity that once framed the opportunity. The scoring function is not novel either: a decayed-frequency score is LRFU, Lee et al., ACM SIGMETRICS 1999, with alpha its documented LRU-to-LFU knob. No CXL hardware was measured. Simulation numbers are provisional, because a single data-structure defect in Revision 1 inverted four of five findings.

The 2026 Landscape

The 2026 KV-Cache Map

The field did not leave a gap — it specialised. Mapped by the decision each layer optimises.

Commercial Products
Available today
CMM-D / CMM-B
Samsung
CXL Type-3
CMM-D: up to 512 GB / module CMM-B: up to 16 TB / box ~60 GB/s (box) ~600 ns loaded
CXL Type-3 memory expansion: an E3.S DDR5 module (CMM-D) and a rack-level pooled box (CMM-B). Media management, RAS and interleaving live in the controller.
Policy lives in host/OS tiering — the device exposes capacity, not an attention-aware cache policy
Apollo + GISMO
XConn + MemVerge
CXL switch
256-lane switch ~100 TB pool demos NVIDIA Dynamo
Switched CXL memory pooling with MemVerge GISMO; demonstrated as a KV-cache tier under NVIDIA Dynamo.
Policy lives in switch + host software — block/prefix eviction, not per-head
Leo (P-/E-Series)
Astera Labs
Smart controller
DDR5 expansion + pooling COSMOS telemetry
Astera's CXL line: Leo controller, Scorpio switch, Aries retimer, Taurus smart cable module. Leo runs real on-controller logic for media management, RAS, security and fleet telemetry.
Controller logic is memory management — no view of attention scores or KV-head structure
Structera
Marvell
Near-memory compute
Arm cores on device Expansion + pooling
Puts general-purpose compute next to DRAM behind CXL — which is why this document does not claim that CXL devices lack on-controller logic.
On-device compute exists, but no published KV-cache eviction policy
vLLM / LMCache
Open Source
Framework
PagedAttention CPU/SSD offload
Industry-standard inference framework with block-level KV management and prefix reuse.
Policy lives in the serving framework at block granularity
SGLang / RadixAttention
Open Source
Framework
Radix-tree prefix cache Automatic KV reuse
Radix tree over prompt prefixes matches and reuses shared prefixes automatically, with LRU eviction over the tree.
Reuse across requests — not selection within a single long context
Dynamo
NVIDIA
Framework
KV-aware routing Tiered KV offload
Disaggregated serving with a KV-cache manager that routes requests to workers already holding the relevant blocks and offloads cold blocks to host memory or storage.
Cluster control plane, block granularity — placement and routing, not per-head retention
Related Work We Build On
Prior art for every mechanism used here
arXiv:2511.00321 — Nov 2025
PNM-KV
CXL-enabled processing-near-memory that offloads token page selection to a PNM accelerator — the closest published work to putting KV decisions on the memory device.
Token-page granularity, custom PNM silicon
H2O: arXiv:2306.14048
H2O · SnapKV · Scissorhands · PyramidKV · Quest
Attention-score-based eviction and query-aware KV selection. Established prior art — this document does not claim to have invented it.
Policy runs in the GPU model runtime
Ada-KV: arXiv:2407.11550
Ada-KV · HeadKV
Per-head cache budget allocation and head-type awareness — the direct antecedent of the per-head tracking used here.
Framework-level per-head budgets
OSDI '24 · 2024 · 2023
InfiniGen · Mooncake · FlexGen
Speculative KV prefetch from host memory, disaggregated prefill/decode with a global KV store, and offload scheduling across GPU/CPU/SSD.
Policy runs on the host CPU
2023 – 2025
vLLM · SGLang/RadixAttention · NVIDIA Dynamo
Block- and prefix-level admission, reuse, routing and tiered offload inside the serving framework.
Policy runs above the kernel
Commercial CXL
Marvell Structera · Astera Labs Leo
Near-memory compute and smart memory control already exist on CXL devices; what is absent is an attention-aware retention policy on them.
On-device logic, general purpose
Where Cache Policy Lives Today
Every layer already makes a real retention decision — a different one each. These are complementary specialisations, not competing claims to one prize.
Serving framework
vLLM/PagedAttention · SGLang/RadixAttention · Dynamo · LMCache (a first-class cross-tier KV layer) · CacheWise (reuse prediction from tool-call state) · Tutti (GPU-direct NVMe movement)
Model runtime (GPU)
H2O · SnapKV · Scissorhands · PyramidKV · Quest · Ada-KV · HeadKV — eviction over (head, token) pairs and per-head budgets. Work built on, not competed with.
Host CPU
FlexGen · InfiniGen · LayerKV · KVPR · Mooncake — offload and speculative fetch over a driver-mediated PCIe path
CXL device
Astera Leo · Marvell Structera · Samsung CMM · TraCT — rack-scale, prefix-aware CXL KV caching on Dynamo, already shipping
What is genuinely open is not an implementation, but a unification. The pieces above are specialised and real. CacheWise predicts reuse but does not decide residency; PNM-KV selects before movement but keeps selection and movement policy separate; LMCache spans tiers and engines but does not predict reuse from agent signals; TraCT solves rack-wide CXL transport; Tutti solves the GPU↔NVMe movement layer. The open question is whether one KV-State Control Plane — SELECT → PLACE → MOVE → ATTEND → OBSERVE — should decide relevance, reuse, residency and movement jointly. Nobody has built or tested that, this package included. The one result it has sits inside PLACE: identity-keyed policy state surviving HBM eviction — +34.15 pts to LFU (15.56 → 49.71) and +24.85 pts to EMA α=0.01 (31.54 → 56.39) at 32 GiB, EMA α=0.15 unchanged (−0.21 pts). A protocol result, testable with no CXL hardware. Simulated (provisional)
This work KV State and Its Control Metadata as One Migratable Object
Block granularity — never head granularity
The unit is a (sequence, token block): 16 tokens = 5.00 MiB. Every head of every layer is required for every token, so evicting a head gives wrong outputs, not a cache miss. The 640 GQA streams are a scoring signal, aggregated into one score per block.
LRFU scoring — established prior art
A decayed-frequency score is LRFU (Lee et al., ACM SIGMETRICS 1999); α is its documented LRU↔LFU knob. α = 0.2 was never validated; the simulated optimum is α ≈ 0.05 and α is non-monotonic, with no convergence onto LFU as α→0. External literature
Persistent identity-keyed state Simulated (provisional)
Policy state survives the round trip between tiers instead of resetting on re-admission — +34.15 pts to LFU, +24.85 pts to EMA α=0.01 at 32 GiB. Bigger than the choice of policy. No RoPE prefetch: RoPE is a rotation applied to Q/K at compute time and carries no information about which blocks are needed next.
Host owns policy; device owns mechanism
Payload overhead is negligible (64 B against a 5.00 MiB block = 0.0012%); metadata processing overhead is not established and must be measured. Whether the controller must be on the device is open — of four falsifiable claims, only the fourth needs device residency.

Evidence labels. Third-party capabilities are vendor specifications or the systems' own published claims, cited not reproduced. This package's persistence result is Simulated (provisional): a single data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. Capacity arithmetic is an Analytical model and stands — 320 KiB/token, 2.50 GiB at 8K, 40.0 GiB at 128K, crossover with weights ≈ 427,000 tokens. LRFU attribution is External literature. Retracted from this figure: the 97% hit rate, 16× user capacity, α = 0.2, per-head eviction, RoPE-informed prefetch, and every "nobody has / the gap" framing. No CXL hardware was measured. Canonical numbers v6.0.