KV Cache Offloading for LLM Inference

The Memory Wall Problem

Figure: why long-context inference does not fit in GPU memory. A B200 has 192 GB of HBM, Llama-3 70B weights take 140 GB, which is 130.4 gibibytes, and one 128K-token KV cache takes 40.0 gibibytes at 320 kibibytes per token, so eight concurrent 128K contexts plus weights need about 489 GB — a capacity wall of 2.5 times the card. That arithmetic is analytical and stands. The second wall is not the latency ratio this figure used to claim. The 65 times figure — 13.0 microseconds of driver-mediated PCIe DMA divided by a 200 nanosecond hypothetical CXL load-store access — is retracted as a headline: the two are not the same operation, and no CXL hardware was measured. In its place is a measured result. On a DGX Spark GB10 on 19 August 2026, streaming copy achieved 236.5 gigabytes per second against a 273 gigabytes per second specification, 87 percent of spec, which retires the roughly 50 percent effective-bandwidth haircut used in earlier drafts. Fitting decode time as a two-term model, t equals a plus b times KV gibibytes, gave 100.3 plus 16.34 times KV for HuggingFace with SDPA unpaged at R squared 0.991, and 73.9 plus 17.58 times KV for vLLM 0.20.1 with FlashAttention-2 paged at R squared 0.945. Under that fit the inferred fixed-path bandwidth is 205.7 gigabytes per second and the inferred KV path 61.1, so the effective KV path is about 3.4 times more costly per byte than the fixed path — a model-derived ratio, not a direct byte-level hardware measurement. KV is read at roughly a quarter of achievable memory bandwidth. Kernel substitution alone did not eliminate the measured KV slope: two frameworks with different attention implementations and paged versus unpaged layout produced KV slopes within 8 percent, while inferred fixed-path bandwidth improved 34 percent between them. This does not generalise to all kernels, GPUs, models or decode regimes. Capacity was not the binding constraint in any configuration measured: everything fit in 128 gibibytes, yet throughput still fell from 150 tokens per second at batch 16 by 4K to 46 tokens per second at batch 16 by 16K, with memory to spare and nothing evicted. Adding a memory tier does not eliminate the KV memory wall; it changes its shape, from a capacity wall into a data-movement wall. B200 specifications quoted: 2,250 TFLOPS BF16 dense, 8 TB/s HBM3e, 192 GB capacity, 64 GB/s PCIe 5.0.

Large language model inference faces a fundamental bottleneck: the memory required to serve long-context requests vastly exceeds what fits in GPU high-bandwidth memory (HBM).

Evidence labels. Capacity: Analytical model — 2 × 80 layers × 8 KV heads × 128 × 2 B = 320 KiB/token → 2.50 GiB at 8K, 40.0 GiB at 128K (43 GB); 140 + 8×43 + 5 = 489 GB; crossover with weights ≈ 427,000 tokens. Simulator-independent, and it stands. Bandwidth: Measured — DGX Spark GB10, 19 Aug 2026 — 236.5 GB/s achieved (87% of a 273 GB/s spec); two-term decode fit 100.3 + 16.34·KV_GiB (HF+SDPA, R²=0.991) and 73.9 + 17.58·KV_GiB (vLLM 0.20.1 + FA-2, R²=0.945). Limits: one model, one size, one machine; fp8 KV not yet measured. Device specs are a vendor specification; 4,500 TFLOPS is the 2:4-sparsity figure, the dense BF16 figure is used here. Retracted: the 65× latency ratio and the 200 ns CXL access — no CXL hardware was measured.

KV-Cache Size vs Context Length

Figure: KV-cache size grows linearly with context length, by the formula 2 times layers times key-value heads times head dimension times sequence times bytes. The decisive term is the number of key-value heads, not query heads. For Llama-3 70B with grouped-query attention — 80 layers, 8 key-value heads, head dimension 128, fp16 — that is 320 kibibytes per token; Llama-2-70B is the same, being grouped-query attention with 8 key-value heads and 64 query heads, and the 2,560 kibibytes-per-token multi-head row belongs to Llama-1-65B and the smaller multi-head Llama-2 variants. The table gives, for each context, cache size and share of a 192 gigabyte B200 HBM: 4K tokens 1.3 gigabytes at 0.7 percent, fits easily; 8K tokens 2.50 gibibytes; 32K tokens 10.0 gibibytes, or 10.7 gigabytes, at 5.6 percent, comfortable, and 8 percent of the model weights; 128K tokens 40.0 gibibytes, or 43 gigabytes, at 22 percent, tight, and 31 percent of the weights; 512K tokens 172 gigabytes at 90 percent, leaving no room for weights; and 1M tokens 344 gigabytes at 179 percent, exceeding HBM. Against 130.4 gibibytes of bf16 weights the single-sequence crossover, where key-value state exceeds the weights, is about 427,000 tokens. Block granularity is 16 tokens equals 5.00 mebibytes. This is an analytical model — arithmetic over published model geometry, simulator-independent and independently confirmed — and it stands through the v5.0 retractions unchanged, because none of it ever depended on a hit rate.

Evidence label: Analytical model — arithmetic over published model geometry, simulator-independent and independently confirmed against a third-party survey. Everything on this panel stands through the v5.0 retractions: none of it ever depended on a hit rate. 320 KiB/token · 2.50 GiB @ 8K · 40.0 GiB @ 128K · 5.00 MiB blocks · crossover ≈ 427,000 tokens. Canonical numbers v6.0.

CXL Endpoint Architecture

Figure: the internal layout of one distributed endpoint, a CXL 3.0 Type-3 device. A CXL 3.0 protocol engine carries CXL.mem (HDM-D and HDM-DB), CXL.io (mailbox and config) and CXL.cache (coherency); a UCIe 1.1 die-to-die link at over 1 TB/s joins three chiplets — a memory controller (8 channels of DDR5-5600, 8 times 44.8 GB/s equals 358.4 GB/s), a compute chiplet (8 ARM A78 cores at 3 GHz with 8 MB shared L3) and a control and policy chiplet running the PLACE and MOVE stages: an LRFU scoring engine, a per-KV-head access tracker used as a scoring signal only, and a persistent block-state store keyed by block identity. There is no RoPE prefetch queue: RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next, so the earlier RoPE prefetch queue is removed. The per-head tracker never evicts a head: every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss, and the placed unit is a (sequence, token block) of 16 tokens, 5.00 mebibytes. Scoring is LRFU, Lee et al., ACM SIGMETRICS 1999, established prior art, with alpha its documented LRU-to-LFU knob; alpha = 0.2 was never validated and the simulated optimum is near 0.05. The persistent block-state store is the one part of this chiplet with an empirical result behind it: letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU and plus 24.85 to EMA alpha 0.01 at a 32 gibibyte budget, simulated and provisional. Each endpoint holds 256 GB of DDR5 (1 TB across four) and 4 TB of NVMe (16 TB across four) at 80 W typical power. Tier latency is assumed, not measured: no CXL hardware was measured, and the former 200 nanosecond figure is demoted to an assumption inside a 32 to 121 gigabytes per second range, under which one 5.00 mebibyte block transfers in 164, 82 or 43 microseconds.

A distributed endpoint is a CXL 3.0 Type-3 device combining memory, compute, and control logic into a single package.

Tiered Memory Architecture

Figure: a three-tier memory stack shown as a proportional bar — hot tier 192 GB of HBM at 8 TB/s, warm tier 1 TB of CXL memory, cold tier 16 TB of NVMe, roughly 17 TB in total. No tier latency here is measured: no CXL hardware was measured anywhere in this package, and the former 200 nanosecond CXL and 25 microsecond flash figures are withdrawn. Tier cost is expressed instead as an assumed bandwidth range of 32, 64 or 121 gigabytes per second, under which one 5.00 mebibyte block of 16 tokens transfers in 164, 82 or 43 microseconds, against 9, 11 or 18 milliseconds to regenerate the same 16 tokens by forward compute at 50, 40 or 25 percent model FLOPs utilisation — two to three orders of magnitude, no precise multiple claimed. The bandwidth hierarchy beneath it runs UCIe internal at over 1 TB/s, DDR5 local at 358.4 GB/s, CXL external at 64 GB/s per link for 256 GB/s across four endpoints, and NVMe flash at about 14 GB/s. The GPU sees one unified address space; the endpoint places data across tiers using CXL.mem load and store semantics, with no DMA setup or driver intervention. All values are an analytical model, not measured.

Bandwidth Hierarchy

CXL 3.0 Coherency Protocol

Figure: the four coherency states CXL 3.0 maintains through the Back-Invalidate protocol. Invalid — the GPU cache is empty and the endpoint is authoritative. Shared — the GPU holds a read copy and the endpoint is still authoritative. Exclusive — the GPU may write and the endpoint copy is stale. Modified — the GPU holds dirty data and must write back. During prefill the GPU issues D2H writes that the endpoint applies to local DRAM; when the endpoint evicts to flash it issues a BI-Snoop that the GPU must acknowledge after writing back dirty data. All values are an analytical model, not measured.

CXL 3.0 provides hardware-managed coherency through the Back-Invalidate (BI) protocol.

Attention Mechanisms: MHA vs GQA vs MQA

Figure: three attention layouts compared by how many KV heads they keep. Multi-head attention keeps 64 KV heads at full memory cost; grouped-query attention, the highlighted middle option and the one used here, shares K and V across query heads for 8 KV heads and an 8 times memory saving; multi-query attention shares a single KV head at a quality tradeoff. Eight KV heads across 80 layers gives 640 per-layer-per-KV-head scoring streams — a scoring granularity, not an eviction one. There are not 640 eviction queues: every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss. All 640 streams are aggregated into one score per (sequence, token block) of 16 tokens, 5.00 mebibytes, and that block is what is placed, moved and evicted, with all 8 KV heads across all 80 layers travelling together. H2O and SnapKV evict head-token pairs, and the token axis is not optional. Per-block state is 64 bytes, so a 128K context is 131,072 divided by 16 equals 8,192 blocks, giving 512 kibibytes against 40.0 gibibytes of KV — a payload overhead of 0.0012 percent, not the 640 mebibytes and 1.5 percent this figure previously claimed, which followed from tracking every position in every head. Metadata processing overhead is not established and must be measured. Both Llama-2-70B and Llama-3-70B are grouped-query attention with 8 KV heads and 64 query heads; the 64-KV-head multi-head row is a counterfactual over the same geometry, and the 2,560 kibibytes-per-token multi-head figure belongs to Llama-1-65B and the smaller multi-head Llama-2 variants.

LRFU Block Eviction — established prior art, not a new algorithm

Figure: LRFU decayed-frequency scoring for block eviction, with a worked example. This is not a new algorithm. A decayed-frequency score is LRFU — Lee et al., "On the Existence of a Spectrum of Policies that Subsumes the LRU and LFU Policies", ACM SIGMETRICS 1999 — and alpha is its documented knob between LRU-like and LFU-like behaviour. LRU assumes recent access predicts future access, which attention can violate: a block at position 1,000 may go untouched until position 100,000 yet stay important. The update rule is score equals alpha times the new score plus one minus alpha times the old score. Alpha = 0.2 was never validated and is withdrawn; the simulated optimum is alpha about 0.05, giving a half-life of ln 0.5 over ln 0.95 equals 13.5 decode steps. Alpha is non-monotonic with an interior optimum: at a 32 gibibyte budget the simulated hit rates are 37.20 percent at alpha 0.5, 38.84 at 0.3, 44.33 at 0.15, 55.96 at 0.05, 31.54 at 0.01, 25.47 at 0.003 and 24.44 at 0.001, so a smaller alpha is not simply more stable, and there is no convergence onto LFU as alpha approaches zero. The evicted unit is a (sequence, token block) of 16 tokens, 5.00 mebibytes, never a head: every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss. Eviction priority is one minus the score, times recency decay, and higher priority is evicted sooner. Block A, an anchor block at position 1,024 last accessed 50 steps ago, scores 0.059 with recency decay 0.049, giving priority 0.046 — kept. Block B, a low-attention block at position 45,678 last accessed 2,000 steps ago, scores 0.02 with recency decay 0.865, giving priority 0.848 — evicted to NVMe, but its identity key and score persist and are restored on re-admission rather than reset. That persistence, not the scoring function, is the largest effect in the study: at 32 gibibytes it is worth plus 34.15 percentage points to LFU, from 15.56 to 49.71 percent, and plus 24.85 to EMA alpha 0.01, from 31.54 to 56.39 percent, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points — larger than the whole spread between scoring functions, and a protocol result testable with no CXL hardware. The worked numbers are illustrative; the simulated figures are provisional, because a single data-structure defect in Revision 1 inverted four of five findings, and no single hit rate exists anywhere, the range being 8.5 to 91.6 percent across budgets, policies and workloads.

Why plain LRU can fail here: LRU assumes recent access predicts future access. Attention can violate this — a block at position 1,000 may not be read again until position 100,000, yet still matter. But note the sweep: no policy dominates. At 32 GiB across workloads, only LFU survives a cyclic sweep (22.06% vs 9.83% for LRU and EMA), while EMA α=0.01 leads a one-shot scan (36.26% vs LRU 31.51%) and EMA α=0.15 leads returning-session traffic (44.44% vs LRU 39.08%). Simulated (provisional)

Evidence labels. Worked scores: Illustrative — pedagogical, never a result. priority = (1 − score) × recency_decay: Block A (1 − 0.059) × 0.049 = 0.046; Block B (1 − 0.02) × 0.865 = 0.848. Policy attribution: External literature — LRFU, Lee et al., ACM SIGMETRICS 1999. Hit rates and α grid: Simulated (provisional) — a single data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. Retracted here: α = 0.2 as an implementation default, the 97% hit rate, and the 72 → 80 → 87 → 93 → 97 ladder. Percentage-point differences are quoted as pts, never %.

RoPE — What the Distance Bound Says, and Why It Cannot Drive Prefetch

Figure: a RoPE explainer. RoPE encodes position by rotating the query and key vectors in d over 2 two-dimensional subspaces, each with its own frequency, so the inner product is a sum over those pairs and does not decay monotonically. What the construction guarantees is an upper-bound envelope: the magnitude of the inner product of q_m and k_n is at most B of the absolute distance m minus n, with B non-increasing. It is never proportional to cosine of (m minus n) times theta; writing it that way converts a bound into a decay law, and a decay law reads like a prediction of access. Critically, RoPE is a rotation applied to Q and K at compute time. The rotation is a function of position alone, known in advance for every position, and identical whether a block is resident or evicted, so it carries no information about which blocks will be needed next and cannot drive prefetch. The prefetch rule this figure previously stated — on accessing position P, prefetch the window P minus W to P plus W, ordered by one over one plus distance over 100 — is withdrawn, together with the claim that RoPE prefetch contributes the last 4 percentage points of a 97 percent hit rate, from 93 to 97. The entire 72 to 80 to 87 to 93 to 97 ladder is retracted: it was never run as an ablation, and its increments were assumed additive and independent even though the mechanisms they decompose overlap. No single hit rate exists; the corrected simulation gives 8.5 to 91.6 percent across budgets, policies and workloads. The 200 nanosecond cost of a prefetch miss is also withdrawn: no CXL hardware was measured, and tier cost is an assumed 32, 64 or 121 gigabytes per second, under which a 5.00 mebibyte block is 164, 82 or 43 microseconds. The assumption that about 70 percent of attention mass falls within plus or minus W positions of the query remains an assumption — model-, layer-, head- and workload-dependent — and it is not something RoPE proves. A position-window prefetch remains an open idea, not a validated technique: it has not been shown to give a predictive advantage over the reuse-prediction approaches already shipping, such as CacheWise's tool-call-driven prediction, and it belongs in future work rather than in a list of contributions.

Evidence labels. The bound |⟨q_m, k_n⟩| ≤ B(|m − n|) is an Analytical model, following from the RoPE construction. The ±W locality figure is Illustrative — an assumption drawn from the attention-locality literature, not a measurement. Retracted from this figure: the +4 pts / 93 → 97 prefetch rung, the 97% hit rate, the whole 72 → 80 → 87 → 93 → 97 ladder, the [P−W, P+W] prefetch rule with its 1 ÷ (1 + distance/100) priority, and the 200 ns prefetch-miss cost. No CXL hardware was measured. Where a real predictive signal exists it comes from elsewhere — InfiniGen speculates from the model's own next-layer state, CacheWise from agent and tool-call metadata External literature.

Prefill vs Decode Phase Characteristics

Figure: the two inference phases side by side. Prefill is compute-bound, writes sequentially, is write-only as it populates the cache, has high arithmetic intensity of about 100 FLOP per byte and processes the full sequence in parallel — so writes stream straight to CXL and no eviction is needed. Decode is memory-bound, does random reads plus one write per step, reads everything and appends one entry, has low arithmetic intensity of about 0.5 FLOP per byte and runs token by token — this is where placement and eviction matter. There is no RoPE prefetch in that strategy: RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next, so the earlier "EMA eviction plus RoPE prefetch" strategy line is corrected to LRFU-scored block placement with persistent identity-keyed state. Note also the correctness constraint on the read pattern: under dense attention every block of an active sequence is read every step, so lossless residency requires every block that is read to be present, and treating attention sparsity as a substitute is a lossy relaxation that carries an accuracy-evaluation burden. The endpoint detects the phase from the read-to-write ratio and switches policy when reads exceed writes by 10 times. This is an analytical model; phase detection on real hardware is design, unvalidated, and no CXL hardware was measured.

KV-Cache Quantization Support

Figure: four KV-cache formats as bars proportional to memory used, for one 128K context. FP16 is the 43 GB baseline at full width; FP8 halves it to 21.5 GB at under 0.1 percent accuracy loss; INT8 also gives 21.5 GB at under 0.5 percent loss; INT4 quarters it to 10.8 GB at about 1 percent loss. Conversion throughput is 48 GB/s FP16 to INT8, 32 GB/s FP16 to INT4 and 64 GB/s INT8 back to FP16. Compression is transparent: the GPU writes BF16, the endpoint stores INT8, the GPU reads BF16 again. All values are an analytical model, not measured.

Modern inference increasingly uses quantized KV-caches. The endpoint supports transparent compression:

Evidence labels. Sizes: Analytical model — one 128K context (40.0 GiB / 43 GB at BF16, 320 KiB/token) scaled by the format width. Accuracy-loss figures: External literature — published claims from the quantization literature, cited not reproduced. Conversion throughputs are a Design target, not demonstrated. fp8 KV not yet measured on the DGX Spark GB10 platform; that run is the next experiment. Canonical numbers v6.0.

Two Unmeasured Component Budgets — and the Comparison That Survives

Figure: two component budgets for a single access, and the comparison that replaces the ratio between them. The CXL budget sums 15, 20, 25, 40, 80 and 20 nanoseconds for request issue, PCIe Gen5 PHY, switch traversal, controller decode, DDR5 array access and return path, totalling 200 nanoseconds. The PCIe DMA budget sums 4.00, 2.50, 1.50, 0.70, 0.10, 0.20 and 4.00 microseconds, totalling 13.00 microseconds. Both are component sums over vendor specifications, and neither was measured: no CXL hardware was measured anywhere in this package. The 65 times ratio between them is retracted as a headline, because the two are not the same operation — a hypothetical load-store path against a driver-mediated DMA path. The 200 nanosecond CXL access is demoted to an unmeasured assumption inside a 32 to 121 gigabytes per second effective-bandwidth range. Also retracted: the end-to-end effective access latency of 177 nanoseconds against a 2,035 nanosecond baseline, 11.5 times lower, together with the 97 over 2.7 over 0.3 tier split it was computed from, because that split was derived from a 97 percent hit rate that was never measured. No single hit rate exists; the corrected simulation gives 8.5 to 91.6 percent across budgets, policies and workloads. What replaces the ratio is the comparison that survives, per 5.00 mebibyte block of 16 tokens: moving the block costs 164, 82 or 43 microseconds at an assumed 32, 64 or 121 gigabytes per second, while regenerating the same 16 tokens by forward compute costs 9, 11 or 18 milliseconds at 50, 40 or 25 percent model FLOPs utilisation. Moving is two to three orders of magnitude cheaper than regenerating, and no precise multiple is claimed, because the two quantities do not measure the same operation; the 218 times figure formed from 17.9 milliseconds divided by 82 microseconds is one point in that range and must always be quoted with its inputs. Regenerating an arbitrary interior block is at least that expensive and potentially far higher, because interior KV is not independently reproducible: hidden states depend recursively on all preceding context. Separately, a genuinely measured result stands in this area: on a DGX Spark GB10, streaming copy achieved 236.5 gigabytes per second, 87 percent of a 273 gigabytes per second specification, and a two-term decode fit implies the effective KV path is about 3.4 times more costly per byte than the fixed path.

Evidence labels. Both component tables are an Analytical model and sum exactly to the totals shown (15+20+25+40+80+20 = 200 ns; 4.00+2.50+1.50+0.70+0.10+0.20+4.00 = 13.00 µs) — but a sum of specifications is not a measurement, and no CXL hardware was measured. The move-vs-regenerate table is an Analytical model over assumed bandwidths and MFU. Retracted here: 65×, 11.5×, 177 ns as an effective latency, the 2,035 ns baseline, the 97 / 2.7 / 0.3 split and the 25 µs tier. The genuinely measured neighbour: Measured — DGX Spark GB10, 236.5 GB/s achieved = 87% of a 273 GB/s spec, with a two-term decode fit implying the effective KV path is ~3.4× more costly per byte than the fixed path (205.7 ÷ 61.1) — a model-derived ratio, not a direct byte-level hardware measurement.

Layer Prefetch Pipeline

Figure: how the tier fabric is loaded, and why the steady-state load depends entirely on a hit rate that is not settled. The strawman it refutes stands: streaming a whole 2.22 gigabyte layer, 302 megabytes of attention plus 1.41 gigabytes of feed-forward plus a 512 megabyte KV slice, over one 64 gigabytes per second link would take 34.7 milliseconds against the few milliseconds that layer takes to compute, and no endpoint count fixes that; the architecture does not do this. The 140 gigabytes of model weights stay HBM-resident and never cross the tier boundary. What crosses is the miss tail, and its size is set by the hit rate. The earlier claim of about 11.4 gigabytes per second steady state, 4.5 percent of a 256 gigabytes per second supply, assumed a 3 percent miss rate taken from the retracted 97 percent hit rate. That hit rate was never measured, and no single hit rate exists: the corrected simulation gives 8.5 to 91.6 percent across budgets, policies and workloads. Recomputing the same arithmetic across that range, with 16 concurrent 128K sequences at 10 tokens per second and a full read volume of 2.4 gigabytes per sequence-step: at a 91.6 percent hit rate the miss tail is about 32 gigabytes per second, 13 percent of a 256 gigabytes per second supply; at 44.18 percent it is about 214 gigabytes per second, 84 percent of supply; and at 8.5 percent it is about 351 gigabytes per second, which exceeds the supply outright. The fabric is therefore not comfortably capacity-driven at every operating point, and this is exactly the shape change the package describes: a memory tier does not eliminate the KV memory wall, it converts a capacity wall into a data-movement wall, creating a tier-ingress bandwidth constraint and a per-transfer latency constraint that did not previously exist. The admission burst is separate: writing or reloading one 128K context of 40.0 gibibytes, or 43 gigabytes, takes 672 milliseconds at 64 gigabytes per second, 336 at 128, 224 at 192 and 168 at the 256 gigabytes per second four-endpoint design point. Those are arithmetic over assumed aggregate bandwidths — no CXL hardware was measured — and the 15.6 times time-to-first-token claim once built on the 168 millisecond figure is retracted, because it rested on a 16.4 gigabytes per second host-DRAM baseline that the measured 236.5 gigabytes per second, 87 percent of specification, on DGX Spark GB10 contradicts. Baselines must be measured, not estimated. The pipeline diagram shows GPU compute on layers N through N plus 3 overlapped with tier transfers of layers N plus 1 through N plus 4; overlap and hiding on real hardware are design, unvalidated. Under idealised overlap the simulation puts fetch at 0.35 to 1.25 milliseconds per turn and recompute at more than 99 percent of stall, so once movement is hidden, recompute avoidance is the only term that matters. Simulation figures are provisional, because a single data-structure defect in Revision 1 inverted four of five findings.

Separately, the fabric must absorb the prefill/admission burst: writing or reloading one 128K context — 40.0 GiB at 320 KiB/token (43 GB). The bandwidths below are assumed; no CXL hardware was measured.

Evidence labels. Admission arithmetic: Analytical model over an assumed aggregate bandwidth — 43 GB ÷ 256 GB/s = 168 ms; endpoint count fixed at 4 and aggregate at 256 GB/s throughout this document. No CXL hardware was measured. Hit rates driving the miss-tail table: Simulated (provisional) — one data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. Retracted here: the 3% miss rate and the 11.4 GB/s / 4.5%-utilised steady state derived from the 97% hit rate; the 15.6× TTFT claim and its 16.4 GB/s baseline. The measured neighbour: Measured — DGX Spark GB10, 236.5 GB/s = 87% of a 273 GB/s spec.

Software Integration Stack

Figure: a four-layer software stack, top to bottom. Application layer: vLLM with PagedAttention, TensorRT-LLM through its plugin API, and SGLang with RadixAttention. Runtime layer: libcxl_kv for KV allocation, CUDA unified virtual memory, and a hint interface for policy parameters. Driver layer: the Linux 6.8-or-later CXL driver and an NVIDIA driver with CXL.mem support. Hardware layer: a B200 GPU acting as CXL 3.0 host, a CXL switch, and Type-3 endpoint devices. No kernel changes are required — CXL memory appears as ordinary GPU-accessible memory and framework changes are confined to the allocator. All values are an analytical model, not measured.

Hit Rate vs. HBM Budget — the corrected sweep

Figure: HBM hit rate as a function of the HBM budget and the eviction policy, from Revision 2 of the trace-driven simulation. There is no single hit rate. Across budgets, policies and workloads the range is 8.5 to 91.6 percent. At an 8 gibibyte budget: LRU 8.52 percent, LFU 8.76, EMA alpha 0.15 12.35, EMA alpha 0.01 12.12. At 16 gibibytes: LRU 17.87, LFU 12.70, EMA alpha 0.15 22.58, EMA alpha 0.01 18.70. At 32 gibibytes: LRU 38.34, LFU 16.09, EMA alpha 0.15 44.18, EMA alpha 0.01 35.91. At 64 gibibytes: LRU 67.12, LFU 23.00, EMA alpha 0.15 69.65, EMA alpha 0.01 66.83. At 128 gibibytes: LRU 88.01, LFU 32.70, EMA alpha 0.15 88.14, EMA alpha 0.01 91.59. EMA alpha 0.15 leads at four of five budgets and EMA alpha 0.01 leads at 128 gibibytes. The margin over LRU is plus 2.5 to plus 5.8 percentage points at the specified alpha, not plus 25; at a tuned optimum of alpha about 0.05 it is plus 19.1 percentage points, 55.96 against LRU 36.85 at 32 gibibytes, from a different experiment with a different baseline that must never be mixed with the budget sweep. LFU collapses once decode growth is modelled, through classical LFU aging: fresh count-one blocks are evicted in favour of stale high-count blocks. Alpha is non-monotonic at 32 gibibytes: 0.5 gives 37.20, 0.3 gives 38.84, 0.15 gives 44.33, 0.05 gives 55.96, 0.01 gives 31.54, 0.003 gives 25.47 and 0.001 gives 24.44, with no convergence onto LFU as alpha approaches zero. No policy dominates across workloads either: at 32 gibibytes over three seeds, returning-session traffic gives LRU 39.08, LFU 15.12, EMA alpha 0.15 44.44 and EMA alpha 0.01 38.87; a one-shot scan gives LRU 31.51, LFU 12.52, EMA alpha 0.15 32.57 and EMA alpha 0.01 36.26; and a cyclic sweep gives LRU 9.83, LFU 22.06 and both EMA settings 9.83, so only LFU survives the loop. The largest effect is none of these. Letting identity-keyed policy state survive HBM eviction, rather than re-admitting a block as if newly created, raises LFU at 32 gibibytes from 15.56 to 49.71 percent, plus 34.15 percentage points, and EMA alpha 0.01 from 31.54 to 56.39 percent, plus 24.85 percentage points, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points. That is larger than the spread between policies, it removes the small-alpha collapse and identifies metadata reset as its cause, and it is a protocol result, testable with no CXL hardware. Retracted from this figure and not replaced: the 72 to 80 to 87 to 93 to 97 ladder and its plus 8 for anchor pinning, plus 7 for EMA at alpha 0.2, plus 6 for per-head tracking and plus 4 for RoPE-aware prefetch — no ablation ever produced it, and the increments were assumed additive and independent although the mechanisms they decompose overlap. Also retracted: the effective-access-latency curve computed from those hit rates, since it used a 200 nanosecond CXL access and a 25 microsecond flash access that no hardware ever produced, and the claim that the full stack holds 97 percent at 16 users while a seventeenth user drops it below target. No CXL hardware was measured. Revision 2 is provisional, not settled: a single data-structure defect in Revision 1 inverted four of five findings, and it has no Belady bound, no bandwidth or queueing model, whole-context residency assumed, a synthetic workload, 2 to 3 seeds and no hardware validation.

Hit rate as a function of the HBM budget, Revision 2 of the trace-driven simulation. Llama-3 70B GQA, 16-token blocks, 200 sessions, Zipf(1.1) return frequency, 1–12K contexts, shared 2048-token prefix, 2500 turns, 421 GiB total footprint, HBM over a 512 GiB tier, 2–3 seeds. Longer bar = higher hit rate. Simulated (provisional)

α = 0.2 was never validated and is withdrawn. There is no convergence onto LFU as α → 0; the Revision 1 claim that there was is retracted.

Evidence label: Simulated (provisional). Standing caveat, required wherever these numbers appear: a single data-structure defect in Revision 1 inverted four of five findings (99.6% wrong victims for EMA α=0.15). Revision 2 is provisional, not settled — no Belady bound, no bandwidth or queueing model, whole-context residency assumed, synthetic workload, 2–3 seeds, no hardware validation. Scripts kv_tiering_sim_v2.py, data results_v2.json. Retracted here: the 72 → 80 → 87 → 93 → 97 ladder and its +8/+7/+6/+4 increments, the 97% hit rate, the 177 / 281 / 435 / 616 / 822 / 1,132 ns effective-latency curve (it priced misses at a 200 ns CXL and 25 µs flash access — no CXL hardware was measured), and the "97% at 16 users, dropping at 17" hot-set claim, which was downstream of the retracted 16× user capacity. Percentage-point differences are pts, never %.

Power & Thermal Analysis

Figure: the power budget of one tier endpoint, and why the two-configuration power comparison this panel used to make is withdrawn. Per endpoint the budget is 40 watts for eight DDR5 channels, 15 watts for eight ARM A78 cores, 12 watts for CXL PHY and controller, 8 watts for the UCIe interface and 5 watts for the NVMe controller — 80 watts typical and 123 watts peak, summing exactly, so four endpoints add 320 watts. That component budget stands as an analytical model over vendor specifications. What does not stand is the comparison it was placed inside: three B200s plus a host with 2 terabytes of DDR5 drawing 3.9 kilowatts against one B200 plus a switch and four endpoints drawing 1.9 kilowatts, 51 percent less, for the same 16 users at 128K context and at least 10 tokens per second. Both sides of that comparison were sized by a 16 times user-capacity claim that is retracted. The 16 times figure was downstream of a 97 percent hit rate that was never measured, and it also fails a correctness constraint: under dense attention every block of an active sequence is read every step, so there is no 5.4 percent resident subset that serves the other 94.6 percent. Concurrency is therefore stated as KV-only capacity arithmetic, not as a served-user count, and no replacement power comparison is offered, because there is no defensible served-user count to hold constant between the two configurations. The capacity arithmetic that survives is: 320 kibibytes per token for Llama-3 70B with grouped-query attention, 2.50 gibibytes at 8K context, 40.0 gibibytes at 128K context, a 5.00 mebibyte block for 16 tokens, and a single-sequence crossover against 130.4 gibibytes of bf16 weights at about 427,000 tokens. The power-usage-effectiveness figure of 1.4 and the 12 cents per kilowatt-hour rate remain stated assumptions, but the three-year cost model they fed is withdrawn along with the 36 percent CapEx reduction and the $160,000 to $103,000 comparison. No hardware power measurement was taken.

Evidence labels. The per-endpoint budget is an Analytical model over vendor specifications and sums exactly: 40 + 15 + 12 + 8 + 5 = 80 W typical, 60 + 25 + 18 + 12 + 8 = 123 W peak, and 4 endpoints contribute 320 W. It stands. Capacity arithmetic is an Analytical model and stands: 320 KiB/token, 2.50 GiB at 8K, 40.0 GiB at 128K, crossover ≈ 427,000 tokens. Retracted: the 3.9 kW → 1.9 kW / −51% comparison and the 16× user capacity it rested on. No hardware power measurement was taken, and no replacement system-level power comparison is offered.

The Cost Model Is Withdrawn — What Survives Is Capacity Arithmetic

Figure: the three-year cost comparison this panel used to make, and the capacity arithmetic that replaces it. The withdrawn comparison priced a baseline of three B200s with host-DRAM offload at $105,000 for GPUs plus $55,000 for host platform and 2 terabytes of DDR5, giving $160,000 CapEx, plus $17,219 of power at 3.9 kilowatts, a power-usage effectiveness of 1.4 and 12 cents per kilowatt-hour, for a $177,219 three-year total cost of ownership; and a proposed system at $35,000 for one B200, $40,000 for host platform and 512 gigabytes of DDR5, $8,000 for a CXL 3.0 switch and $20,000 for four endpoints at $5,000 each, giving $103,000 CapEx, plus $8,389 of power at 1.9 kilowatts, for a $111,389 three-year total. It concluded 36 percent less CapEx, 37 percent less three-year cost and 51 percent less wall power. All of it is retracted outright, and no replacement cost model is offered. The reason is that both configurations were sized to serve the same 16 users at 128K context, and that 16 times user capacity is itself retracted: it was derived from a 97 percent HBM hit rate that was never measured, and it also fails a correctness constraint, because under dense attention every block of an active sequence is read every step, so there is no 5.4 percent resident hot set serving 97 percent of accesses. Pricing a configuration justified by a retracted claim cannot be repaired by re-pricing it; the claim has to go. No payback period, no cost per served user and no replacement dollar figure is invented here. What does survive is arithmetic that never depended on any hit rate, and is simulator-independent and independently confirmed. For Llama-3 70B with grouped-query attention — 80 layers, 8 key-value heads, head dimension 128, fp16 — the key-value cache is 2 times 80 times 8 times 128 times 2 bytes equals 327,680 bytes per token, which is 320 kibibytes per token. At 8K context that is 2.50 gibibytes; at 32K, 10.0 gibibytes, which is 8 percent of the model weights; at 128K, 40.0 gibibytes, which is 31 percent of the weights. Block granularity is 16 tokens equals 5.00 mebibytes. Against 130.4 gibibytes of bf16 weights, the single-sequence crossover at which key-value state exceeds the weights is about 427,000 tokens. Memory expansion of 6 times also stands, as 192 plus 1,024 divided by 192 equals 6.3 times over stated capacities. Concurrency is expressed as this capacity arithmetic, never as a served-user count. No hardware cost or power measurement was taken.

Evidence label: Analytical model — every surviving figure on this panel is arithmetic over published model geometry and stated capacities: 2 × 80 × 8 × 128 × 2 B = 327,680 B = 320 KiB/token; × 8,192 = 2.50 GiB; × 131,072 = 40.0 GiB; 320 KiB × 16 = 5.00 MiB; (192 + 1,024) ÷ 192 = 6.3×. Retracted: $160,000 → $103,000, −36% CapEx, −37% three-year TCO, −51% wall power, 16× user capacity, and the 97% hit rate they all descended from. No cost model is restated in v5.0 or v6.0, and no hardware cost or power measurement was taken.

The 2026 Landscape — the field specialised, it did not leave a gap

Figure: where KV-cache retention policy executes today, organised by the decision each layer optimises, and a table of the related work at each level. The earlier framing on this panel — that no publicly documented system did what this package proposes, so the combination and its placement were the contribution — is retired outright, not softened. In the serving framework: vLLM with PagedAttention, the allocation and paging baseline every offload layer sits on; SGLang with RadixAttention; NVIDIA Dynamo; LMCache, now a first-class cross-tier KV layer doing pinning, lookup, compression, batched movement, compute and I/O pipelining, cross-engine transfer and prefill-decode disaggregation; CacheWise, which predicts future KV reuse from agent and tool-call metadata; and Tutti, which removes the CPU from the GPU-to-NVMe KV I/O critical path. In the GPU model runtime: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV and HeadKV, which evict head-token pairs and allocate per-head budgets with attention scores in hand. On the host CPU: FlexGen, InfiniGen, LayerKV, KVPR and Mooncake, over a driver-mediated PCIe path. On the CXL device: Marvell Structera, Astera Labs Leo, Samsung CMM, PNM-KV, which selects which KV pages are needed before paying the movement cost from a processing-near-memory accelerator, and TraCT, a rack-scale prefix-aware CXL KV cache implemented on NVIDIA Dynamo with direct GPU-to-CXL operations. TraCT means CXL-native KV caching already ships, so describing this package as the sole complete CXL-native KV-cache system is false on its face. This package claims no novelty for its scoring function or its per-head structure either: a decayed-frequency score is LRFU, Lee et al., ACM SIGMETRICS 1999, established prior art with alpha as its documented LRU-to-LFU knob, and per-head signals are a scoring granularity only, since every head of every layer is required for every token and evicting a head produces wrong outputs rather than a cache miss. What is genuinely open is a unification, not an implementation: whether one KV-State Control Plane — SELECT, PLACE, MOVE, ATTEND, OBSERVE — should decide relevance, reuse, residency and movement jointly rather than leaving each to a separate subsystem tuned in isolation. Nobody has built or tested that, this package included. Its one empirical result sits inside PLACE and is simulated and provisional: letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU, from 15.56 to 49.71 percent, and plus 24.85 to EMA alpha 0.01, from 31.54 to 56.39 percent, at a 32 gibibyte budget, with EMA alpha 0.15 unchanged at minus 0.21 percentage points. It is a protocol result, testable with no CXL hardware. Treating it as validation of the whole pipeline would be a significant overclaim: it validates one box.

Every mechanism used here has prior art, and each system below solves a different sub-decision well. Attention-score eviction over (head, token) pairs is H2O, SnapKV, Scissorhands, PyramidKV and Quest; per-head budget allocation is Ada-KV and HeadKV; block allocation and paging is PagedAttention in vLLM; cross-tier KV persistence and movement is LMCache; offload and speculative fetch are InfiniGen, LayerKV, KVPR and Mooncake; rack-scale CXL KV transport is TraCT; near-memory selection before movement is PNM-KV; reuse prediction from agent state is CacheWise; GPU-direct NVMe movement is Tutti. These are complementary decisions, not competing claims to one prize:

Summary: What Stands, What Is Provisional, What Is Retracted

Figure: the corrected summary of results and specification. The four headline figures this panel previously carried are all retracted: 65 times better latency than PCIe per access, a 97 percent HBM hit rate stated as 72 percent under LRU plus 25 points, a 37 percent reduction in three-year total cost of ownership, and by association the 16 times user capacity the last of those was priced against. What replaces them, by evidence class, strongest first. Measured on DGX Spark GB10 on 19 August 2026: streaming copy achieved 236.5 gigabytes per second, which is 87 percent of the 273 gigabytes per second specification, retiring the roughly 50 percent effective-bandwidth haircut used in earlier drafts; a two-term decode fit implies the effective KV path is about 3.4 times more costly per byte than the fixed path, 205.7 divided by 61.1, a model-derived ratio rather than a direct byte-level hardware measurement; KV is read at roughly a quarter of achievable memory bandwidth; and capacity was not the binding constraint in any configuration measured, since everything fit in 128 gibibytes yet throughput still fell from 150 tokens per second at batch 16 by 4K to 46 at batch 16 by 16K with memory to spare and nothing evicted. Simulated and provisional, Revision 2: letting identity-keyed policy state survive HBM eviction raises LFU at a 32 gibibyte budget from 15.56 to 49.71 percent, plus 34.15 percentage points, and EMA alpha 0.01 from 31.54 to 56.39 percent, plus 24.85 percentage points, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points; that is larger than the spread between policies and is a protocol result, testable with no CXL hardware. There is no single hit rate: the range is 8.5 to 91.6 percent across budgets, policies and workloads. The margin over LRU is plus 2.5 to plus 5.8 percentage points at the specified alpha and plus 19.1 at a tuned optimum of alpha about 0.05, from a different experiment with its own baseline. Alpha is non-monotonic and there is no convergence onto LFU as alpha approaches zero; alpha = 0.2 was never validated. Analytical: 320 kibibytes per token for Llama-3 70B with grouped-query attention, 2.50 gibibytes at 8K, 40.0 gibibytes at 128K, 5.00 mebibytes per 16-token block, a single-sequence crossover with the weights at about 427,000 tokens, 6 times memory expansion, and about 17 terabytes of tiered capacity across 192 gigabytes of HBM, 1 terabyte of tier DRAM and 16 terabytes of NVMe. Tier transfer cost is assumed, not measured: no CXL hardware was measured, and a 5.00 mebibyte block takes 164, 82 or 43 microseconds at 32, 64 or 121 gigabytes per second, against 9, 11 or 18 milliseconds to regenerate the same 16 tokens at 50, 40 or 25 percent model FLOPs utilisation — two to three orders of magnitude, no precise multiple claimed. External literature: the scoring function is LRFU, Lee et al., ACM SIGMETRICS 1999, with alpha its documented LRU-to-LFU knob. The specification is corrected accordingly: tracking granularity is a (sequence, token block) of 16 tokens, 5.00 mebibytes, with 640 per-layer-per-KV-head streams used as scoring signals only, because every head of every layer is required for every token and evicting a head produces wrong outputs rather than a cache miss; eviction is by LRFU score at alpha about 0.05 with recency decay and persistent identity-keyed state; there is no RoPE prefetch, because RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next; state overhead is 64 bytes per block, so 512 kibibytes against 40.0 gibibytes at 128K, a payload overhead of 0.0012 percent, with processing overhead not established and requiring measurement; and concurrency is stated as KV-only capacity arithmetic, never as a served-user count. Revision 2 is provisional, not settled: a single data-structure defect in Revision 1 inverted four of five findings.

Standing caveat on every simulated number above. A single data-structure defect in Revision 1 inverted four of five findings (99.6% wrong victims for EMA α=0.15). Revision 2 is provisional, not settled: no Belady bound, no bandwidth or queueing model, whole-context residency assumed, synthetic workload, 2–3 seeds, no hardware validation. Measurement limits: one model, one size, one machine; fixed-length uniform batches, no continuous batching; vLLM fit noisier (R² 0.945); no thermal log; FlashAttention-2, not 3; fp8 KV not yet measured — that run is the direct test of the compression claim and is the next experiment. Percentage-point differences are quoted as pts, never %. Canonical numbers v6.0.