Figure: the 2026 KV-cache landscape in four groups plus a timeline, organised by the decision each system optimises. The field did not leave a gap; it specialised, and the earlier framing on this figure — that no published system did anything sophisticated, and that this package filled the gap — is retired outright. Commercial hardware available today: Samsung CMM-D up to 512 GB per module and CMM-B up to 16 TB per box; XConn Apollo 256-lane CXL switch with MemVerge GISMO pool demos; Astera Labs Leo smart memory controller; Marvell Structera near-memory compute. Software: vLLM with PagedAttention, the allocation and paging baseline every offload layer sits on; LMCache, now a first-class cross-tier KV layer doing pinning, lookup, compression, batched movement, compute and I/O pipelining, cross-engine transfer and prefill-decode disaggregation; SGLang with RadixAttention; NVIDIA Dynamo; Mooncake; FlexGen; InfiniGen; LayerKV; KVPR. TraCT does rack-scale KV sharing and transfer over CXL on NVIDIA Dynamo, with direct GPU-to-CXL operations behind a rack-wide prefix-aware cache — CXL-native KV caching already ships, so this package is not the first or the only one. PNM-KV selects which KV pages are needed before paying the movement cost from a processing-near-memory accelerator behind CXL. CacheWise predicts reuse from agent and tool-call state. Tutti removes the CPU from the GPU-to-NVMe KV I/O path. Research this package builds on rather than competes with: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV, InfiniGen, LayerKV, KVPR, PagedAttention. What remains open is a unification, not an implementation: whether one KV-State Control Plane — SELECT, PLACE, MOVE, ATTEND, OBSERVE — should decide relevance, reuse, residency and movement jointly rather than leaving each to a separate subsystem. Nobody has built or tested that, this package included. Its one empirical result sits inside PLACE and is simulated and provisional: letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU, from 15.56 to 49.71 percent, and plus 24.85 to EMA alpha 0.01, from 31.54 to 56.39 percent, at a 32 gibibyte budget, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points — larger than the spread between policies, and a protocol result testable with no CXL hardware. Retracted from this figure: 640 per-head eviction queues, because every head of every layer is required for every token and evicting a head produces wrong outputs rather than a cache miss, so the unit is a (sequence, token block) of 16 tokens, 5.00 mebibytes, and per-head signals are used only for scoring; alpha = 0.2, never validated, with a simulated optimum near 0.05 and non-monotonic behaviour and no convergence onto LFU as alpha approaches zero; the RoPE prefetch window, because RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next; and the 97 percent hit rate and 16 times user capacity behind the old opportunity framing. The scoring function is not novel: a decayed-frequency score is LRFU, Lee et al., ACM SIGMETRICS 1999, with alpha its documented LRU-to-LFU knob. No CXL hardware was measured. Simulation numbers are provisional, because a single data-structure defect in Revision 1 inverted four of five findings. The timeline runs 2022 to 2026, colour-coded green for hardware, amber for research and blue for software and specifications.
The 2026 Landscape, by Decision Optimised
The field did not leave a gap — it specialised. These are complementary decisions, not competing claims to one prize.
Where KV-cache policy lives today, and the narrow slice this work adds
CMM-D: up to 512 GB / module
CMM-B: up to 16 TB / box
~60 GB/s (box)
~600 ns loaded
CXL Type-3 memory expansion: an E3.S DDR5 module (CMM-D) and a rack-level pooled box (CMM-B). Media management, RAS and interleaving live in the controller.
Where policy lives: Host/OS tiering decides placement; the device exposes capacity, not an attention-aware cache policy
256-lane CXL switch
~100 TB pool demos
Multi-host
Switched CXL memory pooling with MemVerge GISMO; demonstrated as a KV-cache tier under NVIDIA Dynamo.
Where policy lives: Pool allocation in switch + host software; eviction is block/prefix-level, not per-head
CXL smart memory controller
DDR5 expansion + pooling
COSMOS telemetry
Astera's CXL line: Leo controller, Scorpio switch, Aries retimer, Taurus smart cable module. Leo adds on-controller media management, RAS, security and fleet telemetry.
What it optimises: Memory-management logic on the device. Note that TraCT already runs a rack-scale, prefix-aware KV cache over CXL on NVIDIA Dynamo — CXL-native KV caching is shipping, not missing.
CXL near-memory compute
Arm cores on device
Expansion + pooling
Puts general-purpose compute next to the DRAM behind CXL — which is why this document does not claim that CXL devices lack on-controller logic.
What it optimises: General near-memory compute. PNM-KV occupies the adjacent KV-specific slot: selecting which pages are needed before paying the movement cost, reporting up to 21.9× throughput in its own benchmarks (arXiv:2511.00321) — their result, not independently verified.
PagedAttention
CPU offload
Prefix caching
Block-level KV management with PagedAttention and prefix reuse; LMCache adds CPU/SSD tiers.
Where policy lives: Serving framework, block granularity — above the kernel, not in the device
Radix-tree prefix cache
Automatic KV reuse
Hierarchical tiers
Maintains a radix tree over prompt prefixes so shared prefixes are matched and reused automatically, with LRU eviction over the tree.
Where policy lives: Scheduler, prefix granularity — reuse across requests, not selection within a context
Disaggregated serving
KV-aware routing
Tiered KV offload
Distributed inference framework whose KV-cache manager routes requests to workers already holding the relevant blocks and offloads cold blocks to host memory or storage.
What it optimises: Cluster-level placement and routing at block granularity. TraCT is implemented on it for rack-scale CXL KV sharing.
Disaggregated prefill/decode
Global KV store
Distributed KV cache pooled across CPU DRAM + SSD, moved over the network.
Where policy lives: Cluster store; transfers are network-mediated rather than load/store
GPU/CPU/SSD offload
Speculative KV prefetch
FlexGen schedules offload for throughput; InfiniGen speculates which KV entries the next layer needs and prefetches them from host memory.
What they optimise: Host-driven offload and speculative fetch over PCIe. InfiniGen speculates from the model's own next-layer state — a real predictive signal, unlike position geometry. LayerKV and KVPR sit in the same layer.
CXL + processing-near-memory
1M tokens
Steady-token selection
Offloads token page selection to a PNM accelerator behind CXL — the closest published work to putting KV-cache decisions on the memory device.
What it optimises: Selecting which KV/token pages are needed before paying the movement cost — conceptually adjacent to the SELECT stage of this package's control plane, not a cruder version of it.
Attention-score eviction
Heavy hitters
Query-aware selection
Attention-score-based KV eviction and selection is established prior art. This document does not claim to have invented it.
What they evict: (head, token) pairs — the token axis is not optional. A head alone is never an eviction unit: dropping one gives wrong outputs, not a cache miss. Token dropping is a lossy relaxation with an accuracy-evaluation burden.
Per-head budgets
Head-type awareness
Retrieval/induction heads
Allocating a different cache budget per attention head is likewise established prior art, and is the direct antecedent of the per-head tracking used here.
What they optimise: Per-head cache budgets on the GPU. This package reads the same per-head signal but aggregates it into one score per (sequence, token block) — scoring granularity, never eviction granularity.
🗺 Where Cache Policy Lives Today
In the serving framework
vLLM/PagedAttention, SGLang/RadixAttention, Dynamo, LMCache (a first-class cross-tier KV layer), CacheWise (reuse prediction from tool-call state), Tutti (GPU-direct NVMe movement).
In the model runtime
H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV: attention-score eviction over (head, token) pairs and per-head budgets, on the GPU. Real results this package builds on.
On the host CPU
FlexGen, InfiniGen, LayerKV, KVPR, Mooncake: offload and speculative fetch driven by host software over a driver-mediated PCIe path.
On the CXL device
Astera Leo, Marvell Structera, Samsung CMM, and TraCT — a rack-scale, prefix-aware CXL KV cache on Dynamo doing direct GPU↔CXL operations. CXL-native KV caching already ships.
What is genuinely open — a unification, not an implementation
Every layer above already solves a real, non-trivial sub-problem. What none of them does is decide
relevance, reuse, residency and movement jointly. That is the open question this package is oriented
around — a KV-State Control Plane over SELECT → PLACE → MOVE → ATTEND → OBSERVE — and
nobody has built or tested it, this package included. Its one empirical result lives inside
PLACE: identity-keyed policy state surviving HBM eviction, worth +34.15 pts to LFU
(15.56 → 49.71) and +24.85 pts to EMA α=0.01 (31.54 → 56.39) at 32 GiB, with EMA α=0.15 unchanged
(−0.21 pts). A protocol result, testable with no CXL hardware. Treating it as
validation of the whole pipeline would be a significant overclaim.
Simulated (provisional)
🔬 The KV-State Control Plane — where this package fits, and where it does not
SELECT — Design / unvalidated
Which regions or blocks are worth considering. Past-only signals (recency, frequency, LRFU decay) and predictive ones (agent/tool state, prefix identity, reuse history) — the slot CacheWise and PNM-KV occupy. Not built here.
PLACE — Simulated (provisional)
Which tier a block occupies, and whether its identity-keyed policy state crosses with it. +34.15 pts to LFU · +24.85 pts to EMA α=0.01 at 32 GiB. The only box with evidence. Unit: (sequence, token block), 16 tokens = 5.00 MiB — never a head.
MOVE — Design / unvalidated
Scheduling, batching, hiding movement behind compute. No CXL hardware measured: a 5.00 MiB block is 164 / 82 / 43 µs at an assumed 32 / 64 / 121 GB/s, against 9 / 11 / 18 ms to regenerate 16 tokens at 50 / 40 / 25% MFU.
ATTEND · OBSERVE — Design / unvalidated
The model consumes the placed state; what was attended feeds the next SELECT. Conceptual only. Scoring throughout is LRFU (Lee et al., ACM SIGMETRICS 1999) — prior art, α its documented LRU↔LFU knob; α = 0.2 was never validated.
Industry Timeline
2022
Samsung CMM-D prototype
CXL 3.0 specification
2023
vLLM PagedAttention
Intel Sapphire Rapids (CXL 1.1)
H2O attention-score eviction
CXL 3.1 specification
2024
Samsung CMM-B (16 TB box)
Astera Labs Leo controllers
SGLang / RadixAttention
SnapKV · Quest · InfiniGen
2025
NVIDIA Dynamo KV manager
XConn + MemVerge GISMO pooling
Marvell Structera near-memory compute
Ada-KV / HeadKV per-head budgets
PNM-KV (arXiv:2511.00321, Nov 2025)
2026
CXL 3.x switching broadens
KV offload becomes a standard serving-stack tier
TraCT: rack-scale CXL KV cache on Dynamo
LMCache: first-class cross-tier KV layer
CacheWise (reuse prediction) · Tutti (GPU-direct NVMe)
This package: residency-state persistence in PLACE [Simulated (provisional)]
Evidence labels. Third-party capabilities are vendor specifications or the systems'
own published claims, cited not reproduced and not independently verified here. This package's
persistence result is Simulated (provisional): a single data-structure defect in Revision 1 inverted
four of five findings, so Revision 2 is provisional, not settled. Capacity arithmetic is an
Analytical model and stands — 320 KiB/token, 2.50 GiB at 8K, 40.0 GiB at 128K, crossover with weights
≈ 427,000 tokens. LRFU attribution is External literature.
Retracted from this figure: the 97% hit rate, 16× user capacity, 36% CapEx / $160,000 → $103,000,
α = 0.2, per-head eviction, RoPE-informed prefetch, and every "the gap / nobody has" framing.
No CXL hardware was measured. Canonical numbers v6.0.