Figure: a map of where KV-cache retention policy lives today, in four bands, organised by the decision each layer optimises rather than by any gap. The field did not leave a gap; it specialised, and the earlier "nobody has done this" framing is retired outright. Commercial products: Samsung CMM-D up to 512 GB per module and CMM-B up to 16 TB per box; the XConn Apollo 256-lane switch with MemVerge GISMO; Astera Labs Leo; Marvell Structera. Serving frameworks: vLLM with PagedAttention, the baseline every offload layer sits on; LMCache, now a first-class cross-tier KV layer with pinning, lookup, compression, batched movement, compute and I/O pipelining, cross-engine transfer and prefill-decode disaggregation; SGLang with RadixAttention; NVIDIA Dynamo. TraCT is a rack-scale, prefix-aware CXL KV cache implemented on Dynamo with direct GPU-to-CXL operations — CXL-native KV caching already ships, so this package is not the sole or first CXL-native KV-cache system. PNM-KV selects which KV pages are needed before paying the movement cost, from a processing-near-memory accelerator behind CXL. CacheWise predicts reuse from agent and tool-call state. Tutti removes the CPU from the GPU-to-NVMe KV I/O path. Related work this package builds on rather than competes with: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV, InfiniGen, LayerKV, KVPR and PagedAttention. What is genuinely open is not a missing implementation but a missing unification: whether one KV-State Control Plane — SELECT, PLACE, MOVE, ATTEND, OBSERVE — should decide relevance, reuse, residency and movement jointly instead of leaving each to a separate subsystem. Nobody, including this package, has built or tested that. One box inside it has evidence, and it is simulated and provisional: inside PLACE, letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU, from 15.56 to 49.71 percent, and plus 24.85 to EMA alpha 0.01, from 31.54 to 56.39 percent, at a 32 gibibyte budget, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points — larger than the spread between policies, and testable with no CXL hardware. Retracted from this figure: per-head eviction over 640 GQA queues, since every head of every layer is required for every token and evicting a head produces wrong outputs rather than a cache miss, and the tracked unit is a (sequence, token block) of 16 tokens, 5.00 mebibytes, with per-head signals used only for scoring; alpha = 0.2, never validated, superseded by a simulated optimum near 0.05 with non-monotonic behaviour and no convergence onto LFU as alpha approaches zero; RoPE locality prefetch, since RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next; and the 97 percent hit rate and 16 times user capacity that once framed the opportunity. The scoring function is not novel either: a decayed-frequency score is LRFU, Lee et al., ACM SIGMETRICS 1999, with alpha its documented LRU-to-LFU knob. No CXL hardware was measured. Simulation numbers are provisional, because a single data-structure defect in Revision 1 inverted four of five findings.
CMM-D: up to 512 GB / module
CMM-B: up to 16 TB / box
~60 GB/s (box)
~600 ns loaded
CXL Type-3 memory expansion: an E3.S DDR5 module (CMM-D) and a rack-level pooled box (CMM-B). Media management, RAS and interleaving live in the controller.
Policy lives in host/OS tiering — the device exposes capacity, not an attention-aware cache policy
256-lane switch
~100 TB pool demos
NVIDIA Dynamo
Switched CXL memory pooling with MemVerge GISMO; demonstrated as a KV-cache tier under NVIDIA Dynamo.
Policy lives in switch + host software — block/prefix eviction, not per-head
DDR5 expansion + pooling
COSMOS telemetry
Astera's CXL line: Leo controller, Scorpio switch, Aries retimer, Taurus smart cable module. Leo runs real on-controller logic for media management, RAS, security and fleet telemetry.
Controller logic is memory management — no view of attention scores or KV-head structure
Arm cores on device
Expansion + pooling
Puts general-purpose compute next to DRAM behind CXL — which is why this document does not claim that CXL devices lack on-controller logic.
On-device compute exists, but no published KV-cache eviction policy
PagedAttention
CPU/SSD offload
Industry-standard inference framework with block-level KV management and prefix reuse.
Policy lives in the serving framework at block granularity
Radix-tree prefix cache
Automatic KV reuse
Radix tree over prompt prefixes matches and reuses shared prefixes automatically, with LRU eviction over the tree.
Reuse across requests — not selection within a single long context
KV-aware routing
Tiered KV offload
Disaggregated serving with a KV-cache manager that routes requests to workers already holding the relevant blocks and offloads cold blocks to host memory or storage.
Cluster control plane, block granularity — placement and routing, not per-head retention
arXiv:2511.00321 — Nov 2025
PNM-KV
CXL-enabled processing-near-memory that offloads token page selection to a PNM accelerator — the closest published work to putting KV decisions on the memory device.
Token-page granularity, custom PNM silicon
H2O: arXiv:2306.14048
H2O · SnapKV · Scissorhands · PyramidKV · Quest
Attention-score-based eviction and query-aware KV selection. Established prior art — this document does not claim to have invented it.
Policy runs in the GPU model runtime
Ada-KV: arXiv:2407.11550
Ada-KV · HeadKV
Per-head cache budget allocation and head-type awareness — the direct antecedent of the per-head tracking used here.
Framework-level per-head budgets
OSDI '24 · 2024 · 2023
InfiniGen · Mooncake · FlexGen
Speculative KV prefetch from host memory, disaggregated prefill/decode with a global KV store, and offload scheduling across GPU/CPU/SSD.
Policy runs on the host CPU
2023 – 2025
vLLM · SGLang/RadixAttention · NVIDIA Dynamo
Block- and prefix-level admission, reuse, routing and tiered offload inside the serving framework.
Policy runs above the kernel
Commercial CXL
Marvell Structera · Astera Labs Leo
Near-memory compute and smart memory control already exist on CXL devices; what is absent is an attention-aware retention policy on them.
On-device logic, general purpose
🧩
Serving framework
vLLM/PagedAttention · SGLang/RadixAttention · Dynamo · LMCache (a first-class cross-tier KV layer) · CacheWise (reuse prediction from tool-call state) · Tutti (GPU-direct NVMe movement)
📈
Model runtime (GPU)
H2O · SnapKV · Scissorhands · PyramidKV · Quest · Ada-KV · HeadKV — eviction over (head, token) pairs and per-head budgets. Work built on, not competed with.
🖥
Host CPU
FlexGen · InfiniGen · LayerKV · KVPR · Mooncake — offload and speculative fetch over a driver-mediated PCIe path
🔌
CXL device
Astera Leo · Marvell Structera · Samsung CMM · TraCT — rack-scale, prefix-aware CXL KV caching on Dynamo, already shipping
🔍
What is genuinely open is not an implementation, but a unification. The pieces above are
specialised and real. CacheWise predicts reuse but does not decide residency; PNM-KV selects before movement but
keeps selection and movement policy separate; LMCache spans tiers and engines but does not predict reuse from agent
signals; TraCT solves rack-wide CXL transport; Tutti solves the GPU↔NVMe movement layer. The open question is
whether one KV-State Control Plane — SELECT → PLACE → MOVE → ATTEND → OBSERVE — should decide
relevance, reuse, residency and movement jointly. Nobody has built or tested that, this package included.
The one result it has sits inside PLACE: identity-keyed policy state surviving HBM eviction —
+34.15 pts to LFU (15.56 → 49.71) and +24.85 pts to EMA α=0.01 (31.54 → 56.39) at
32 GiB, EMA α=0.15 unchanged (−0.21 pts). A protocol result, testable with no CXL
hardware. Simulated (provisional)
✓
Block granularity — never head granularity
The unit is a (sequence, token block): 16 tokens = 5.00 MiB. Every head of every layer is required for every token, so evicting a head gives wrong outputs, not a cache miss. The 640 GQA streams are a scoring signal, aggregated into one score per block.
✓
LRFU scoring — established prior art
A decayed-frequency score is LRFU (Lee et al., ACM SIGMETRICS 1999); α is its documented LRU↔LFU knob. α = 0.2 was never validated; the simulated optimum is α ≈ 0.05 and α is non-monotonic, with no convergence onto LFU as α→0. External literature
✓
Persistent identity-keyed state Simulated (provisional)
Policy state survives the round trip between tiers instead of resetting on re-admission — +34.15 pts to LFU, +24.85 pts to EMA α=0.01 at 32 GiB. Bigger than the choice of policy. No RoPE prefetch: RoPE is a rotation applied to Q/K at compute time and carries no information about which blocks are needed next.
✓
Host owns policy; device owns mechanism
Payload overhead is negligible (64 B against a 5.00 MiB block = 0.0012%); metadata processing overhead is not established and must be measured. Whether the controller must be on the device is open — of four falsifiable claims, only the fourth needs device residency.
Evidence labels. Third-party capabilities are vendor specifications or the systems'
own published claims, cited not reproduced. This package's persistence result is
Simulated (provisional): a single data-structure defect in Revision 1 inverted four of five findings,
so Revision 2 is provisional, not settled. Capacity arithmetic is an Analytical model and stands —
320 KiB/token, 2.50 GiB at 8K, 40.0 GiB at 128K, crossover with weights ≈ 427,000 tokens. LRFU attribution is
External literature. Retracted from this figure: the 97% hit rate, 16× user capacity,
α = 0.2, per-head eviction, RoPE-informed prefetch, and every "nobody has / the gap" framing.
No CXL hardware was measured. Canonical numbers v6.0.