A hardware-measured decomposition of decode-phase KV bandwidth, and a protocol-level proposal for persisting eviction-policy metadata across memory tiers.
The core findings condensed: measured KV bandwidth and cost ratios, the metadata-persistence proposal, and what remains open.
Why KV-cache bandwidth and capacity matter for LLM serving, and what this document does and does not claim.
LLM inference fundamentals โ prefill vs. decode, the memory-bandwidth gap, and how existing serving stacks handle KV state.
The tiered memory system under study: CXL attachment topology, controller roles, and where KV state physically lives.
How the DGX Spark GB10 measurements were collected, the decode-latency model that was fit, and its accuracy and limits.
What fraction of spec bandwidth each tier actually delivers, and the cost/capacity tradeoffs across the memory hierarchy.
How prefill-phase work and input preprocessing interact with a tiered KV memory system, and where they don't.
The central proposal: persisting eviction-policy metadata across memory tiers at the protocol level, and the simulated hit-rate case for it.
How MoE routing behavior complicates KV-cache residency and eviction decisions relative to dense models.
Integration points between GPU memory management and a CXL memory controller โ mapping, hints, and fault paths.
Where commercial CXL products and inference-serving software stand as of 2026, and how this work relates to them.
All measured, simulated, and analytical results gathered in one place, each labeled with its evidence tier.
A practical procedure for deciding when a CXL-tiered KV-cache design is worth adopting for a given deployment.
What was established, what remains provisional, and the open questions for future measurement work.