Executive Summary

What this package establishes

Adding a memory tier — CXL-attached capacity behind HBM — does not eliminate the KV memory wall in LLM inference. It changes its shape. Without a tier, the wall is a capacity wall: contexts grow until KV state no longer fits in HBM. With a tier, capacity is relieved, but two problems appear in its place: the bandwidth into the tier is a new, narrower pipe, and every transfer across it carries a per-request latency the accelerator did not previously pay. This package treats those as the real constraints and evaluates each with the evidence that is actually available today, without waiting on CXL memory hardware that is only now reaching production systems.

What was measured

On a DGX Spark (GB10 Grace Blackwell, 128 GiB unified LPDDR5X, 273 GB/s spec bandwidth), serving Qwen2.5-7B-Instruct in fp16, streaming reads sustained 236.5 GB/s — 87% of spec (Measured). Decode latency was fit as a two-term model, fixed cost plus a KV-proportional term: HuggingFace with SDPA gave 100.3 + 16.34·KV_GiB ms (R²=0.991); vLLM 0.20.1 with FlashAttention-2 gave 73.9 + 17.58·KV_GiB ms (R²=0.945). Inverting each fit's slope and intercept against known transfer volumes gives an inferred fixed-path bandwidth of 151.5 GB/s (HuggingFace) and 205.7 GB/s (vLLM), against an inferred KV-path bandwidth of 65.7 and 61.1 GB/s respectively. The ratio of fixed-path to KV-path bandwidth under the stronger stack is 3.4× (Measured-derived) — a statement about what the fitted model implies, not a direct byte-level hardware measurement. Notably, the KV-path slope moved only 8% between the two stacks while fixed-path bandwidth improved 34%: kernel substitution did not touch the KV cost in these two tested stacks, though this should not be read as a claim about kernels generally. Across every configuration run, capacity itself was never the binding constraint — everything fit in 128 GiB — while throughput still fell from 150 tok/s (batch 16, 4K context) to 46 tok/s (batch 16, 16K context) with memory to spare. Taken together, this is the strongest measured finding in the package: resident KV decode cost scales steeply with KV volume on the tested platform, evidenced by the two-term decode fit (R²=0.991 for HF+SDPA, R²=0.945 for vLLM+FlashAttention-2), the ~3.4× model-derived KV-vs-fixed-path byte cost ratio, and the 236.5 GB/s (87% of spec) achieved streaming bandwidth (Measured).

What was simulated

A trace-driven simulator (Revision 2, corrected after an erratum in Revision 1; results below are provisional) isolates one variable: whether an eviction policy's scoring state survives a round trip out of HBM and back, keyed by block identity rather than reconstructed from scratch after every eviction. At 32 GiB HBM, LFU improves from 15.56% to 49.71% hit rate (+34.15 points) and EMA (α=0.01) improves from 31.54% to 56.39% (+24.85 points) when metadata persists. No single policy dominates across the zipf, scan, and loop workloads tested. This is the strongest provisional architectural finding in the package — identity-keyed policy state can materially improve simulated residency behavior when it survives eviction and re-admission (Simulated, provisional) — and it is deliberately framed as a protocol claim: it requires no CXL hardware to test, only a correct trace and a correct simulator. It is a separate claim, on separate evidence, from the strongest measured finding below, and the two should not be conflated.

What was derived analytically

For Llama-3 70B under GQA (8 KV heads), KV state costs 320 KiB per token, or 5.00 MiB per 16-token block (Analytical). At long context this yields roughly 6× memory expansion relative to weights — simple capacity arithmetic, independent of any hit rate. Separately, moving a resident block between tiers is two to three orders of magnitude cheaper than regenerating it by recomputation — stated as an order of magnitude, not a precise multiple, since no CXL hardware was measured directly in this package.

The actual contribution

The scoring function itself is not new: a decayed-frequency score is LRFU (Lee et al., SIGMETRICS 1999), and this package does not claim otherwise. The contribution is architectural: this package proposes a KV-state control plane (Chapter 3) — relevance and reuse signals feeding a SELECT/PLACE/MOVE control plane that governs ATTEND and OBSERVE, built on two orthogonal hierarchies, an information hierarchy and a physical hierarchy. The evidence base assembled here — hardware-measured decode bandwidth decomposition (Chapter 4) plus a provisional simulation showing that residency-state persistence matters (Chapter 7) — currently validates one mechanism inside that architecture, the PLACE/residency stage, not the architecture as a whole. The remaining stages, and the control plane's integrated behavior under real CXL hardware, are not yet validated and should not be read as such.

Landscape, briefly

As of August 2026, CXL memory hardware is shipping to hyperscalers — Marvell's Structera X expanders among them — with switch fabrics such as Structera S advertising pooling up to 48 TB behind one switch, 4 TB/s aggregate, sub-460ns round trip (vendor-published figures, not independently verified here). Samsung's CMM-D KV-cache offloading white paper (June 2026) reports system-level bandwidth in the 35–47 GB/s range without decomposing KV traffic from weight traffic — a related but different measurement from the per-byte KV/weight decomposition this package performs.