Figure: where the 640 LRU queues and EMA tracking should live, across four candidate placements in a memory hierarchy. A, GPU HBM: 192 GB at 8 TB/s — lowest-latency decisions but burns 640 MiB of HBM on metadata. B, host CPU: DDR5 at about 400 GB/s over NVLink or PCIe 5.0 — easy to implement but adds a PCIe round trip per decision. C, CXL controller: Type 2 or 3 at about 64 GB/s and 200 ns — the recommended placement, since it intercepts all CXL.mem traffic and can prefetch into device-local DRAM, at the cost of custom hardware. D, computational storage: NVMe SSD at about 14 GB/s — offloads the host but sits on a higher-latency path. The metadata footprint is 640 queues (8 KV-heads times 80 layers) of 131,072 entries at 8 bytes each, totalling 640 MiB, about 1.5 percent of the 43 GB KV cache. The recommended hybrid flow runs GPU (EMA updates) to CXL controller (eviction and prefetch) to NVMe (cold storage). All values are an analytical model, not measured.

Cache Management Placement

Where do the 640 LRU queues + EMA tracking live?

System Architecture
NVLink / PCIe 5.0
CXL.mem / NVMe-oF
GPU-Side
Where: CUDA kernel / HBM reservation
Lowest latency decisions
Attention scores already on GPU
Burns HBM for metadata (640 MiB)
Kernel complexity for async I/O
Host CPU
Where: Userspace daemon / driver
Easy to implement & debug
Flexible policy changes
PCIe round-trip per decision
CPU becomes bottleneck at scale
Best for CXL
CXL Controller
Where: CXL Type-2 device FPGA/ASIC
Near-memory processing
Intercepts all CXL.mem traffic
Can prefetch to device-local DRAM
Custom hardware required
Computational Storage
Where: NVMe SSD controller ARM cores
Offloads host entirely
Near-storage prefetch decisions
Higher latency path
Limited compute on SSD controller
Metadata Footprint (Llama-70B, 128K context)
LRU Queues
640
8 KV-heads × 80 layers
Entries per Queue
131,072
Max sequence length
Bytes per Entry
8 B
pos + ema + last_access
Total Metadata
640 MiB
~1.5% of the 43 GB KV cache
Recommended: Hybrid Approach
GPU computes attention → streams scores to CXL controller → controller updates EMA (α = 0.2), makes eviction/prefetch decisions → issues async NVMe reads. No per-access CPU involvement in steady state; the control plane, mapping, faults and drivers remain.

Evidence label: analytical model — capacities, latencies and the metadata footprint are derived for the canonical scenario (1×B200, Llama-2-70B, 128K context). No hardware benchmark was run.