Figure: the 2026 KV-cache landscape, mapped by the decision each system optimises, in four bands plus an industry timeline. The field did not leave a gap; it specialised. The earlier framing on this figure — that KV-cache-over-CXL was uniformly unsophisticated and that nobody had built attention-aware retention — is retired outright, not softened. Commercial hardware available today: Samsung CMM-D up to 512 GB per module and CMM-B up to 16 TB per box; the XConn Apollo 256-lane CXL switch with MemVerge GISMO pool demos; the Astera Labs Leo smart memory controller; and Marvell Structera near-memory compute. Software and research systems each optimise a different sub-decision: vLLM does GPU KV allocation and paging via PagedAttention, the baseline every offload layer sits on; LMCache does KV persistence, reuse and movement across tiers and engines, with pinning, lookup, compression, batched movement, compute and I/O pipelining, cross-engine transfer and prefill-decode disaggregation; TraCT does rack-scale KV sharing and transfer over CXL, implemented on NVIDIA Dynamo with direct GPU-to-CXL operations behind a rack-wide prefix-aware cache — it is already a CXL-native KV-cache system, not a gap waiting to be filled; PNM-KV selects which KV pages are needed before paying the movement cost, using a processing-near-memory accelerator in or near CXL memory, and reports up to 21.9 times throughput improvement in its own benchmarks, cited as their reported result and not independently verified; Tutti removes the CPU from the KV I/O critical path for GPU-to-NVMe object movement, reporting 78.3 percent time-to-first-token reduction against GDS-enabled SSD-backed LMCache under strict service objectives; CacheWise predicts future KV reuse from agent and tool-call metadata, reporting 2 to 2.6 times fewer evictions and up to 3.5 times faster agent session completion. Alongside them, the model-runtime line of work this package builds on rather than competes with: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV, InfiniGen, LayerKV, KVPR and PagedAttention. This package's own position is an open design question, not a shipped system: whether relevance estimation, reuse prediction, residency and movement can be decided jointly by one KV-State Control Plane — SELECT, PLACE, MOVE, ATTEND, OBSERVE — instead of separately by each layer above. Only one piece of that has any empirical result, and it is simulated and provisional: inside PLACE, letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU, from 15.56 to 49.71 percent, and plus 24.85 to EMA alpha 0.01, from 31.54 to 56.39 percent, at a 32 gibibyte budget, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points. Retracted from this figure and not replaced: the modelled 97 percent hit rate, the 16 times user capacity, alpha = 0.2, the per-head eviction queues, and the RoPE prefetch window — RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next. Every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss; H2O and SnapKV evict head-token pairs, and the token axis is not optional. No CXL hardware was measured. Simulation figures are provisional, because a single data-structure defect in Revision 1 inverted four of five findings. The timeline runs 2022 to 2026, colour-coded green for hardware, amber for software, purple for research and blue for standards.
The 2026 KV-Cache Landscape
The field did not leave a gap — it specialised. Mapped by the decision each system optimises.
The “nobody has done this” framing is retired, not softened
An earlier version of this figure argued the landscape was uniformly
unsophisticated — generic LRU everywhere, passive memory hardware, a wide-open gap. That does not survive contact with what
has shipped. TraCT is already a CXL-native, rack-scale KV-cache system running on Dynamo; describing this
package as the sole complete CXL-native KV-cache system is false on its face.
This package also claims no novelty for its scoring function or its
per-head structure. A decayed-frequency score is LRFU — Lee et al., “On the Existence of a Spectrum of
Policies that Subsumes the LRU and LFU Policies”, ACM SIGMETRICS 1999 — and α is its documented LRU↔LFU knob.
External literature
Hardware
CMM-D / CMM-B
Samsung
CMM-D: up to 512 GB / module
CMM-B: up to 16 TB / box
~60 GB/s (box)
~600 ns loaded
CXL Type-3 memory expansion: an E3.S DDR5 module (CMM-D) and a rack-level pooled box (CMM-B). Media management, RAS and interleaving live in the controller.
Where policy lives: Host/OS tiering decides placement; the device exposes capacity, not an attention-aware cache policy
Hardware
Apollo CXL Switch + GISMO
XConn + MemVerge
256-lane CXL switch
~100 TB pool demos
Multi-host
Switched CXL memory pooling with MemVerge GISMO; demonstrated as a KV-cache tier under NVIDIA Dynamo.
Where policy lives: Pool allocation in switch + host software; eviction is block/prefix-level, not per-head
Hardware
Leo (P-/E-Series)
Astera Labs
CXL smart memory controller
DDR5 expansion + pooling
COSMOS telemetry
Astera's CXL memory line: Leo controller, Scorpio switch, Aries retimer, Taurus smart cable. Leo adds on-controller media management, RAS, security and fleet telemetry.
Where policy lives: Controller logic exists, but it is memory-management logic — it has no view of attention scores or KV-head structure
Hardware
Structera
Marvell
CXL near-memory compute
Arm cores on device
Expansion + pooling
CXL device family that puts general-purpose compute next to the DRAM, which is why "no CXL device has on-controller logic" is not a claim this document makes.
Where policy lives: Compute is available on-device, but is applied to general workloads — no published KV-cache eviction policy
Software
vLLM + LMCache
UC Berkeley / Open Source
PagedAttention
CPU offload
Prefix caching
Block-level KV management with PagedAttention and prefix reuse; LMCache adds CPU/SSD tiers.
Where policy lives: Serving framework, block granularity — above the kernel, not in the device
Software
SGLang / RadixAttention
Open Source
Radix-tree prefix cache
Automatic KV reuse
Hierarchical tiers
Keeps a radix tree over prompt prefixes so shared prefixes are matched and reused automatically, with LRU eviction over the tree.
Where policy lives: Scheduler, prefix granularity — reuse across requests, not selection within a context
Software
Dynamo
NVIDIA
Disaggregated serving
KV-aware routing
Tiered KV offload
Distributed inference framework with a KV-cache manager that routes requests to workers already holding the relevant blocks and offloads cold blocks to host memory or storage.
Where policy lives: Cluster control plane, block granularity — placement and routing, not per-head retention
Software
Mooncake
Moonshot AI
Disaggregated prefill/decode
Global KV store
Distributed KV cache pooled across CPU DRAM + SSD, moved over the network.
Where policy lives: Cluster store; transfers are network-mediated rather than load/store
Software
FlexGen · InfiniGen
Stanford · SNU (OSDI '24)
GPU/CPU/SSD offload
Speculative KV prefetch
FlexGen schedules offload for throughput; InfiniGen speculates which KV entries the next layer will need and prefetches them from host memory.
Where policy lives: Host CPU, over a driver-mediated PCIe path (µs-scale per transfer)
Paper
PNM-KV
arXiv:2511.00321 — November 2025
CXL + processing-near-memory
1M tokens
Steady-token selection
Offloads token page selection to a PNM accelerator behind CXL — the closest published work to putting KV-cache decisions on the memory device.
What it optimises: Selecting which KV/token pages are needed before paying the movement cost. Conceptually adjacent to the SELECT stage of this package's control plane, not a cruder version of it. Reports up to 21.9× throughput improvement in its own benchmarks (arXiv:2511.00321) — their reported result, not independently verified.
Paper
H2O · SnapKV · Scissorhands · PyramidKV · Quest
H2O: arXiv:2306.14048 (2023)
Attention-score eviction
Heavy hitters
Query-aware selection
Attention-score-based KV eviction and selection is established prior art. This package does not claim to have invented it, and builds on it.
What they evict: (head, token) pairs — the token axis is not optional. A head alone is never an eviction unit: every head of every layer is required for every token, so dropping a head gives wrong outputs, not a cache miss. Token dropping is a lossy relaxation carrying an accuracy-evaluation burden.
Paper
Ada-KV · HeadKV
Ada-KV: arXiv:2407.11550 (2024)
Per-head budgets
Head-type awareness
Retrieval/induction heads
Allocating a different cache budget per attention head is likewise established prior art, and is the direct antecedent of the per-head scoring used here — scoring granularity, never eviction granularity.
What they optimise: Framework-level per-head budget allocation on the GPU. This package reads the same signal but aggregates it into one score per (sequence, token block) — 16 tokens = 5.00 MiB — which is the unit that is placed, moved and evicted.
🧩
In the serving framework
vLLM, SGLang/RadixAttention, NVIDIA Dynamo, LMCache: block- and prefix-level admission, reuse and eviction, decided above the kernel.
📊
In the model runtime
H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV: attention-score eviction over (head, token) pairs and per-head budgets, executed on the GPU. Real, non-trivial results this package builds on.
🖥
On the host CPU
FlexGen, InfiniGen, LayerKV, KVPR, Mooncake: offload and speculative fetch driven by host software over a driver-mediated PCIe path. InfiniGen speculates which entries the next layer needs from the model's own state — a real predictive signal, unlike position geometry.
🔌
On the CXL device
Astera Leo (smart memory controller), Marvell Structera (near-memory compute), Samsung CMM, and TraCT — a rack-scale, prefix-aware CXL KV cache on NVIDIA Dynamo doing direct GPU↔CXL operations. CXL-native KV caching already ships.
🔍
What this package actually claims
Not a gap, and not a novel policy. The pieces above are specialised but not unified: CacheWise
predicts reuse but does not decide residency; PNM-KV selects before movement but keeps selection and movement policy
separate; LMCache spans tiers but does not predict reuse from agent signals; TraCT solves rack-wide CXL transport;
Tutti solves the GPU↔NVMe movement layer. The open question is whether a higher-level
KV-State Control Plane — SELECT → PLACE → MOVE → ATTEND → OBSERVE — should sit above them, deciding
relevance, reuse, residency and movement jointly. That has not been built or tested, by this package or anyone
else.
One box inside it has an empirical result, and only one: inside PLACE, letting identity-keyed policy
state survive HBM eviction. At 32 GiB — LFU 15.56 → 49.71 (+34.15 pts),
EMA α=0.01 31.54 → 56.39 (+24.85 pts), EMA α=0.15 unchanged (−0.21 pts). Larger than the spread between
policies, and a protocol result — testable with no CXL hardware. Treating it as
validation of the whole pipeline would be a significant overclaim.
Simulated (provisional)
SELECT — Design / unvalidated
Which candidate regions or blocks are worth considering. Both past-only signals (recency, frequency, LRFU decay) and predictive ones (agent/tool state, prefix identity, reuse history) — the category CacheWise and PNM-KV occupy. Not built here.
PLACE — Simulated (provisional)
Which tier a block occupies, and whether its identity-keyed policy state crosses the boundary with it. +34.15 pts to LFU, +24.85 pts to EMA α=0.01 at 32 GiB. The one box with evidence.
MOVE — Design / unvalidated
Executing the transfer: scheduling, batching, hiding movement behind compute. No CXL hardware measured; a 5.00 MiB block is 164 / 82 / 43 µs at an assumed 32 / 64 / 121 GB/s.
ATTEND · OBSERVE — Design / unvalidated
The model consumes the placed state; what was actually attended feeds back into the next SELECT. Conceptual only — nothing here is built or measured.
Industry Timeline
2022
Samsung CMM-D prototype
CXL 3.0 specification
2023
vLLM PagedAttention
Intel Sapphire Rapids (CXL 1.1)
H2O attention-score eviction
CXL 3.1 specification
2024
Samsung CMM-B (16 TB box)
Astera Labs Leo controllers
SGLang / RadixAttention
SnapKV · Quest · InfiniGen
2025
NVIDIA Dynamo KV manager
XConn + MemVerge GISMO pooling
Marvell Structera near-memory compute
Ada-KV / HeadKV per-head budgets
PNM-KV (arXiv:2511.00321, Nov 2025)
2026
CXL 3.x switching broadens
KV offload becomes a standard serving-stack tier
TraCT: rack-scale CXL KV cache on Dynamo
LMCache: first-class cross-tier KV layer
CacheWise (reuse prediction) · Tutti (GPU-direct NVMe)
This package: residency-state persistence in PLACE [Simulated (provisional)]
Make KV State and Its Control Metadata One Migratable Object
The claim is a protocol, not a policy and not a product: identity-keyed policy state that survives the round trip between tiers. At 32 GiB that is worth +34.15 pts to LFU and +24.85 pts to EMA α=0.01 — more than the choice of scoring function — and it needs no CXL hardware to test. Host owns policy; device owns mechanism. Whether the controller must be on the device is open: of four falsifiable claims, only the fourth needs device residency at all.
(sequence, token block) granularity
LRFU scoring — prior art
Persistent identity-keyed state
Testable without CXL hardware
Evidence labels. Third-party capabilities are vendor specifications or the systems'
own published claims, cited not reproduced — PNM-KV's 21.9×, Tutti's 78.3% TTFT reduction and
CacheWise's 2–2.6× fewer evictions are their reported results, not independently verified here. The persistence
result is Simulated (provisional): a single data-structure defect in Revision 1 inverted four of five
findings, so Revision 2 is provisional, not settled. Capacity arithmetic is an Analytical model
(320 KiB/token, 2.50 GiB at 8K, 40.0 GiB at 128K, crossover ≈ 427,000 tokens). LRFU attribution is
External literature. Retracted from this figure: the 97% hit rate, 16× user capacity,
36% CapEx / $160,000 → $103,000, α = 0.2, per-head eviction, RoPE-aware prefetch, and every “nobody has” framing.
No CXL hardware was measured. Canonical numbers v6.0.