SUPERSEDED — this figure illustrates a claim withdrawn in v5.0. It is retained only so the retraction remains checkable. Do not cite it. See RETRACTIONS.md and Appendix L (hardware measurements) and Appendix N (simulation) for what replaced it.

Figure: GPU-CXL KV-cache interaction across a tiered memory architecture. A B200 GPU with 2,250 TFLOPS of BF16 dense compute holds the hot tier — 192 GB of HBM3e at 8 TB/s carrying the active attention window — and on a page fault issues a CXL.mem request with load/store semantics and no per-access driver. A CXL 3.0 switch routes to four endpoints, each holding a 256 GB DRAM warm tier at about 200 nanoseconds and a 4 TB flash cold tier at about 25 microseconds, under a firmware controller doing per-head LRU, prefetch and score tracking. The access flow is: the attention kernel asks for K and V at position P; an HBM hit returns in about 100 nanoseconds and a miss raises a CXL.mem fault; the endpoint returns from DRAM in 200 nanoseconds or flash in 25 microseconds, and because flash is a complete backing store there is no miss path; then positions P minus W to P plus W are prefetched on RoPE locality. Cache policy runs 640 LRU queues, 80 layers times 8 KV heads, each entry 8 bytes of position, count and score_ema, so 640 x 131,072 x 8 B = 640 MiB, about 1.5 percent of the 43 GB per-user KV cache, with score_ema = 0.2 x new + 0.8 x old at alpha 0.2 and a half-life of 3.1 steps. Total tiered capacity is 192 GB hot, 1 TB warm across 4 x 256 GB, and 16 TB cold across 4 x 4 TB, about 17 TB against 192 GB for HBM alone. All values are an analytical model, not measured.

GPU-CXL KV-Cache Interaction

Tiered Memory Architecture for Large Context Inference

GPU (B200)
Tensor Cores
2,250 TFLOPS
BF16 dense (4,500 TFLOPS is the 2:4-sparsity figure)
HBM3e (Hot Tier)
192 GB
8 TB/s
Active KV-Cache
Current Attention Window
GPU MMU
Page Fault
CXL.mem Request
CXL.mem
Load/Store Semantics
No Driver Per-Access
Data Response
Cache Line (64B)
Read Request
Physical Address
CXL 3.0 Switch
Address Routing | Coherency Management | Multi-Endpoint Aggregation
Endpoint 0
Firmware Controller
Per-head LRU | Prefetch | Score Tracking
Endpoint 1
Firmware Controller
Per-head LRU | Prefetch | Score Tracking
Endpoint 2
Firmware Controller
Per-head LRU | Prefetch | Score Tracking
Endpoint 3
Firmware Controller
Per-head LRU | Prefetch | Score Tracking
KV-Cache Access Flow
1. Request
GPU attention kernel needs K/V for token position P
2. HBM Check
Hit → Return at HBM latency (~100 ns)
Miss → CXL.mem fault
3. Endpoint Lookup
DRAM hit → 200 ns
Flash hit → 25 μs
No miss path: flash is a complete backing store, so a KV entry is never lost
4. Prefetch
Position P accessed →
Prefetch [P-W, P+W]
(RoPE locality)
Endpoint Cache Policy (Per KV-Head)
Access Tracking
• 640 LRU queues (80 layers × 8 KV heads)
• Entry: position, count, score_ema — 8 B
• 640 × 131,072 × 8 B = 640 MiB (671 MB decimal)
• ≈1.5% of the 43 GB per-user KV cache
Score Integration
• Attention scores via CXL.io mailbox
• score_ema = 0.2 × new + 0.8 × old (α = 0.2, half-life 3.1 steps)
• High score → Retain in DRAM
Eviction Priority
• f(recency_rank, score_ema)
• Low score + old → Flash
• High score → Keep in DRAM
Tiered Capacity (4 Endpoints)
KEY INSIGHT
GPU sees unified address space. Endpoint manages tier placement transparently.
CXL.mem provides load/store semantics — no explicit I/O commands.
Endpoint firmware handles caching, prefetching, and promotion/demotion.

Analytical model — every latency, capacity and hit rate on this page is derived from vendor specifications and an analytical model. Nothing here was measured on hardware. Canonical numbers v4.0.