KV Cache Offloading for LLM Inference

The Memory Wall Problem

Figure: why long-context inference does not fit in GPU memory. A B200 has 192 GB of HBM, Llama-70B weights take 140 GB and one 128K-token KV cache takes 43 GB, so eight users need 489 GB — a capacity wall of 2.5 times the card. The second wall is latency: a PCIe DMA access costs 13.0 microseconds against 200 nanoseconds for CXL.mem, a factor of 65. B200 specifications quoted: 2,250 TFLOPS BF16 dense, 8 TB/s HBM3e, 192 GB capacity, 64 GB/s PCIe 5.0. All values are an analytical model, not measured.

Large language model inference faces a fundamental bottleneck: the memory required to serve long-context requests vastly exceeds what fits in GPU high-bandwidth memory (HBM).

Evidence label: analytical model — KV sizing is 2 × 80 layers × 8 KV heads × 128 × 2 B = 320 KiB/token → 43 GB at 128K; 140 + 8×43 + 5 = 489 GB. 4,500 TFLOPS is the 2:4-sparsity figure; the dense BF16 figure is used here. No hardware benchmark was run.

KV-Cache Size vs Context Length

Figure: KV-cache size grows linearly with context length, by the formula 2 times layers times heads times head-dimension times sequence times bytes — for Llama-2-70B, 320 KiB per token. The table gives, for each context, cache size and share of a 192 GB B200 HBM: 4K tokens 1.3 GB at 0.7 percent (fits easily), 32K 10.7 GB at 5.6 percent (comfortable), 128K 43 GB at 22 percent (tight), 512K 172 GB at 90 percent (leaves no room for weights) and 1M 344 GB at 179 percent (exceeds HBM). All values are an analytical model, not measured.

CXL Endpoint Architecture

Figure: the internal layout of one distributed endpoint, a CXL 3.0 Type-3 device. A CXL 3.0 protocol engine carries CXL.mem (HDM-D and HDM-DB), CXL.io (mailbox and config) and CXL.cache (coherency); a UCIe 1.1 die-to-die link at over 1 TB/s joins three chiplets — a memory controller (8 channels of DDR5-5600, 8 times 44.8 GB/s equals 358.4 GB/s), a compute chiplet (8 ARM A78 cores at 3 GHz with 8 MB shared L3) and a control and policy chiplet (EMA scoring engine, per-head access tracker, RoPE prefetch queue). Each endpoint holds 256 GB of DDR5 (1 TB across four) and 4 TB of NVMe flash (16 TB across four), at about 200 nanoseconds CXL latency and 80 W typical power. All values are an analytical model, not measured.

A distributed endpoint is a CXL 3.0 Type-3 device combining memory, compute, and control logic into a single package.

Tiered Memory Architecture

Figure: a three-tier memory stack shown as a proportional bar — hot tier 192 GB of HBM at 8 TB/s, warm tier 1 TB of CXL memory at about 200 nanoseconds, cold tier 16 TB of flash at about 25 microseconds, roughly 17 TB in total. The bandwidth hierarchy beneath it runs UCIe internal at over 1 TB/s, DDR5 local at 358.4 GB/s, CXL external at 64 GB/s per link for 256 GB/s across four endpoints, and NVMe flash at about 14 GB/s. The GPU sees one unified address space; the endpoint places data across tiers using CXL.mem load and store semantics, with no DMA setup or driver intervention. All values are an analytical model, not measured.

Bandwidth Hierarchy

CXL 3.0 Coherency Protocol

Figure: the four coherency states CXL 3.0 maintains through the Back-Invalidate protocol. Invalid — the GPU cache is empty and the endpoint is authoritative. Shared — the GPU holds a read copy and the endpoint is still authoritative. Exclusive — the GPU may write and the endpoint copy is stale. Modified — the GPU holds dirty data and must write back. During prefill the GPU issues D2H writes that the endpoint applies to local DRAM; when the endpoint evicts to flash it issues a BI-Snoop that the GPU must acknowledge after writing back dirty data. All values are an analytical model, not measured.

CXL 3.0 provides hardware-managed coherency through the Back-Invalidate (BI) protocol.

Attention Mechanisms: MHA vs GQA vs MQA

Figure: three attention layouts compared by how many KV heads they keep. Multi-head attention keeps 64 KV heads at full memory cost; grouped-query attention, the highlighted middle option and the one used here, shares K and V across query heads for 8 KV heads and an 8 times memory saving; multi-query attention shares a single KV head at a quality tradeoff. Eight KV heads across 80 layers gives 640 independent eviction policies: 640 LRU queues of 131 thousand entries at 8 bytes each, 640 MiB of metadata, about 1.5 percent of the cache. All values are an analytical model, not measured.

EMA-Based Eviction Algorithm

Figure: why exponential-moving-average scoring beats LRU, and a worked example. LRU assumes recent access predicts future access, which attention violates — a token at position 1,000 may go untouched until position 100,000 yet stay critical. The update rule is score_ema equals alpha times new score plus one minus alpha times the old score, with alpha 0.2 giving a half-life of 3.1 steps; a high alpha is reactive, a low alpha stable. Eviction priority is one minus score_ema, times recency decay, and higher priority is evicted sooner. Token A, an important anchor at position 1,024 last accessed 50 steps ago, scores 0.211 with recency decay 0.049, giving priority 0.039 — kept in cache. Token B, low-attention at position 45,678 last accessed 2,000 steps ago, scores 0.08 with recency decay 0.865, giving priority 0.796 — evicted to flash. All values are an analytical model, not measured.

Why LRU fails: LRU assumes recent access predicts future access. Attention violates this—a token at position 1,000 may not be accessed until position 100,000, but remains critically important.

Evidence label: analytical model — priority = (1 − score_ema) × recency_decay: Token A (1 − 0.211) × 0.049 = 0.039; Token B (1 − 0.08) × 0.865 = 0.796. EMA states are illustrative values produced by the α = 0.2 update rule. No hardware benchmark was run.

RoPE-Aware Prefetch Strategy

Figure: how rotary position embedding makes cache access predictable. RoPE encodes position by rotating query and key vectors, so nearby positions rotate similarly and attend more strongly; the modeling assumption, not a measured result, is that about 70 percent of attention mass falls within plus or minus W positions of the query. The prefetch rule is therefore: when the GPU accesses position P, prefetch the window P minus W to P plus W, with priority one divided by one plus distance over 100. Contribution: the full stack reaches a 97 percent HBM hit rate, of which RoPE prefetch supplies the last 4 points, 93 to 97; a prefetch miss costs 200 nanoseconds over CXL. All values are an analytical model, not measured.

Evidence label: analytical model — the +4 percentage-point prefetch contribution is the last rung of the modeled ladder 72 → 80 → 87 → 93 → 97, and the ±W locality figure is an assumption drawn from the attention-locality literature. No hardware benchmark was run.

Prefill vs Decode Phase Characteristics

Figure: the two inference phases side by side. Prefill is compute-bound, writes sequentially, is write-only as it populates the cache, has high arithmetic intensity of about 100 FLOP per byte and processes the full sequence in parallel — so writes stream straight to CXL and no eviction is needed. Decode is memory-bound, does random reads plus one write per step, reads everything and appends one entry, has low arithmetic intensity of about 0.5 FLOP per byte and runs token by token — this is where EMA eviction and RoPE prefetch matter. The endpoint detects the phase from the read-to-write ratio and switches policy when reads exceed writes by 10 times. All values are an analytical model, not measured.

KV-Cache Quantization Support

Figure: four KV-cache formats as bars proportional to memory used, for one 128K context. FP16 is the 43 GB baseline at full width; FP8 halves it to 21.5 GB at under 0.1 percent accuracy loss; INT8 also gives 21.5 GB at under 0.5 percent loss; INT4 quarters it to 10.8 GB at about 1 percent loss. Conversion throughput is 48 GB/s FP16 to INT8, 32 GB/s FP16 to INT4 and 64 GB/s INT8 back to FP16. Compression is transparent: the GPU writes BF16, the endpoint stores INT8, the GPU reads BF16 again. All values are an analytical model, not measured.

Modern inference increasingly uses quantized KV-caches. The endpoint supports transparent compression:

Evidence label: analytical model — sizes are one 128K user context (43 GB at BF16) scaled by the format width; accuracy-loss figures are published claims from the quantization literature, not measurements taken here. No hardware benchmark was run.

Latency: CXL.mem vs PCIe Baseline

Figure: a per-access latency comparison. A PCIe DMA transfer costs 13.0 microseconds with the CPU in the critical path for driver call, DMA setup and completion interrupt; a CXL.mem direct access costs 200 nanoseconds using load and store semantics with no per-access CPU involvement in steady state — a ratio of 65 times over the same boundary and the same 4 KiB payload. The CXL budget breaks down as 15, 20, 25, 40, 80 and 20 nanoseconds for request issue, PCIe Gen5 PHY, switch traversal, controller decode, DDR5 array access and return path, totalling 200 nanoseconds; the PCIe budget as 4.00, 2.50, 1.50, 0.70, 0.10, 0.20 and 4.00 microseconds, totalling 13.00 microseconds. End to end, with the 97 / 2.7 / 0.3 hit split, effective access latency is 177 nanoseconds against a baseline 2,035 nanoseconds, 11.5 times lower — a different figure from the 65 times per-access ratio. All values are an analytical model, not measured.

Evidence label: analytical model — both component tables sum exactly to the totals shown (15+20+25+40+80+20 = 200 ns; 4.00+2.50+1.50+0.70+0.10+0.20+4.00 = 13.00 µs). No hardware benchmark was run.

Layer Prefetch Pipeline

Figure: why the CXL fabric is sized for capacity rather than bandwidth. The strawman it refutes: streaming a whole 2.22 GB layer (302 MB attention plus 1.41 GB FFN plus 512 MB KV slice) over one 64 GB/s link would take 34.7 milliseconds against a few milliseconds of compute — the architecture does not do this. Instead the 140 GB of weights stay HBM-resident and never cross CXL, attention reads only the hot set, and only the miss tail crosses the fabric: about 11.4 GB/s steady state against 256 GB/s supplied, 4.5 percent utilized. What the fabric must absorb is the admission burst of one user's 43 GB context: 1 endpoint at 64 GB/s takes 672 milliseconds and dominates time-to-first-token, 2 endpoints at 128 GB/s take 336 milliseconds, 3 at 192 GB/s take 224 milliseconds, and the 4-endpoint design point at 256 GB/s takes 168 milliseconds. The pipeline diagram shows GPU compute on layers N through N plus 3 overlapped with CXL prefetch of layers N plus 1 through N plus 4. All values are an analytical model, not measured.

What the fabric does have to absorb is the prefill/admission burst: writing or reloading one user's 43 GB context.

Evidence label: analytical model — 43 GB ÷ 256 GB/s = 168 ms; the endpoint count is fixed at 4 and the aggregate at 256 GB/s throughout this document. No hardware benchmark was run.

Software Integration Stack

Figure: a four-layer software stack, top to bottom. Application layer: vLLM with PagedAttention, TensorRT-LLM through its plugin API, and SGLang with RadixAttention. Runtime layer: libcxl_kv for KV allocation, CUDA unified virtual memory, and a hint interface for policy parameters. Driver layer: the Linux 6.8-or-later CXL driver and an NVIDIA driver with CXL.mem support. Hardware layer: a B200 GPU acting as CXL 3.0 host, a CXL switch, and Type-3 endpoint devices. No kernel changes are required — CXL memory appears as ordinary GPU-accessible memory and framework changes are confined to the allocator. All values are an analytical model, not measured.

Performance Sensitivity to Cache Hit Rate

Figure: effective access latency as a function of HBM hit rate, holding the miss split at 90 percent CXL and 10 percent NVMe. Six bars, longer meaning slower: 97 percent with the full stack gives 177 nanoseconds; 93 percent with per-head tracking gives 281; 87 percent with EMA gives 435; 80 percent with anchors gives 616; the 72 percent LRU baseline gives 822; and 60 percent with no policy gives 1,132. The ladder of contributions is LRU 72, plus 8 points for anchor pinning, plus 7 for EMA scoring at alpha 0.2, plus 6 for per-head tracking and plus 4 for RoPE-aware prefetch, reaching 97. The NVMe tail dominates: even at 97 percent, the 0.3 percent of accesses reaching flash contribute 75 of the 177 nanoseconds, and beyond 16 users the per-user hot set falls below 2.31 GB and the hit rate drops under the 97 percent target — the hot set, not capacity, is the limit. All values are an analytical model, not measured.

Effective access latency as a function of the HBM hit rate h, holding the miss split fixed at 90% CXL / 10% NVMe: h × 100 ns + 0.9(1−h) × 200 ns + 0.1(1−h) × 25 µs.

Evidence label: analytical model — 72 + 8 + 7 + 6 + 4 = 97 percentage points; every latency in the table is the formula above evaluated at that hit rate. Deltas are percentage points, not percentages. No hardware benchmark was run.

Power & Thermal Analysis

Figure: wall power for the same 16 users at 128K context and at least 10 tokens per second. The baseline of three B200s plus a host with 2 TB of DDR5 draws 3.9 kW; the proposed single B200 with a CXL switch and four endpoints draws 1.9 kW, 51 percent less. Per endpoint the budget is 40 W for eight DDR5 channels, 15 W for eight ARM A78 cores, 12 W for CXL PHY and controller, 8 W for the UCIe interface and 5 W for the NVMe controller — 80 W typical and 123 W peak, so four endpoints add 320 W. The TCO model assumes a PUE of 1.4 and $0.12 per kWh. All values are an analytical model, not measured.

Evidence label: analytical model — 3.9 kW → 1.9 kW = 51% less; the per-endpoint budget sums to 40 + 15 + 12 + 8 + 5 = 80 W typical and 60 + 25 + 18 + 12 + 8 = 123 W peak, and 4 endpoints contribute 320 W of the proposed configuration. No hardware benchmark was run.

Total Cost of Ownership (3-Year)

Figure: a three-year cost comparison for the same workload of 16 users at 128K context. The baseline of three B200s with host-DRAM offload costs $105,000 for GPUs plus $55,000 for host platform and 2 TB DDR5, giving $160,000 CapEx, plus $17,219 of power at 3.9 kW, 1.4 PUE and $0.12 per kWh, for a $177,219 three-year TCO. The proposed system costs $35,000 for one B200, $40,000 for host platform and 512 GB DDR5, $8,000 for a CXL 3.0 switch and $20,000 for four intelligent endpoints at $5,000 each, giving $103,000 CapEx, plus $8,389 of power at 1.9 kW, for a $111,389 three-year TCO. That is 36 percent less CapEx, 37 percent less three-year TCO and 51 percent less wall power — a $57,000 CapEx and $65,830 TCO saving from day one. No payback period is claimed. All values are an analytical model, not measured.

Both configurations serve the same workload: 16 users at 128K context, ≥10 tok/s per user. Anything else is not a like-for-like comparison.

Evidence label: analytical model — CapEx (160,000 − 103,000) ÷ 160,000 = 35.6% → 36%. Power cost = kW × 1.4 PUE × 8,760 h × 3 yr × $0.12: baseline 3.9 kW → $17,219, proposed 1.9 kW → $8,389. TCO (177,219 − 111,389) ÷ 177,219 = 37.1%. Hardware prices are list-price estimates, not quotes. No hardware benchmark was run.

Where Cache Policy Lives Today

Figure: four cards showing where KV-cache retention policy executes today, and a table of the related work at each level. In the serving framework: vLLM, SGLang with RadixAttention and NVIDIA Dynamo, at block and prefix granularity. In the GPU model runtime: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV and HeadKV, with attention scores in hand. On the host CPU: FlexGen, InfiniGen and Mooncake, over a driver-mediated PCIe path. On the CXL device: Marvell Structera, Astera Labs Leo and Samsung CMM, doing media management, pooling and general compute. The closest published work is PNM-KV, at token-page granularity on custom near-memory silicon. The narrow claim is that no publicly documented system executes per-head EMA scoring with RoPE-informed prefetch inside CXL controller firmware, below the serving framework; the combination and its placement are the contribution. All values are an analytical model, not measured.

Every mechanism used here has prior art. Attention-score eviction is H2O, SnapKV, Scissorhands, PyramidKV and Quest; per-head budget allocation is Ada-KV and HeadKV; offload and speculative prefetch are InfiniGen, Mooncake, vLLM, SGLang/RadixAttention and NVIDIA Dynamo; on-device CXL compute is Marvell Structera and Astera Labs Leo. What differs is where the policy executes:

Summary: Key Results

Figure: the four headline results and the full specification. Latency is 65 times better than PCIe per access, the HBM hit rate is 97 percent (72 percent under LRU plus 25 points), three-year TCO falls 37 percent, and effective capacity is 17 TB. The specification: a CXL 3.0 Type-3 memory expander; UCIe internal bandwidth over 1 TB/s; external CXL bandwidth of 64 GB/s per link, 256 GB/s across four endpoints; 256 GB of DDR5 per endpoint for 1 TB total and 4 TB of flash each for 16 TB total; ARM or RISC-V cores executing policy; tracking per KV-head per layer across 640 queues; eviction by EMA attention score at alpha 0.2 with recency decay; prefetch over the RoPE window P minus W to P plus W at priority one over one plus distance divided by 100; 640 MiB of metadata for a 128K context, about 1.5 percent of the 43 GB cache; and 16 concurrent users at 128K and at least 10 tokens per second on a 37 GB hot set of 2.31 GB each. All values are an analytical model, not measured.

Evidence label: analytical model — 65× = 13,000 ns ÷ 200 ns · 97% = 72 + 8 + 7 + 6 + 4 percentage points · −37% = $177,219 → $111,389 over 3 years · 17 TB = 192 GB HBM + 1 TB CXL DRAM + 16 TB endpoint flash. No hardware benchmark was run.