Figure: GPU-tier KV-cache interaction across a tiered memory architecture. A B200 GPU holds the hot tier — 192 GB of HBM3e at 8 TB/s carrying resident KV blocks — and on a fault issues a CXL.mem request with load/store semantics and no per-access driver. A CXL 3.0 switch routes to four endpoints, each holding a 256 GB DRAM warm tier and a 4 TB NVMe cold tier, under a controller running the PLACE and MOVE stages of the KV-State Control Plane. No CXL hardware was measured anywhere in this package, so no tier latency on this page is a measurement. The former 200 nanosecond CXL access is demoted to an unmeasured assumption inside a 32 to 121 gigabytes per second effective-bandwidth range: one 5.00 mebibyte block of 16 tokens transfers in 164, 82 or 43 microseconds at an assumed 32, 64 or 121 gigabytes per second. The tier's value is avoiding regeneration, since forward compute for the same 16 tokens is 9, 11 or 18 milliseconds at 50, 40 or 25 percent model FLOPs utilisation — two to three orders of magnitude, with no precise multiple claimed. The 65 times single-access latency ratio, formed by dividing a hypothetical 200 nanosecond load-store path into a 13.0 microsecond driver-mediated DMA path, is retracted as a headline: the two paths are not the same operation and neither was measured. The access flow is: the attention kernel asks for the KV block covering position P; an HBM hit returns from HBM and a miss raises a CXL.mem fault; the endpoint returns the block from DRAM or NVMe, and because NVMe is a complete backing store there is no miss path; the block is then admitted with its identity key and score restored from the state that survived its last eviction, rather than reset. There is no RoPE-driven prefetch step: RoPE is a rotation applied to Q and K at compute time and carries no information about which blocks are needed next, so the earlier fourth step, prefetching positions P minus W to P plus W on RoPE locality, is withdrawn. Cache policy tracks a (sequence, token block) of 16 tokens, 5.00 mebibytes, not a head: every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss, and the 640 per-layer-per-KV-head streams are a scoring signal only. Per-block state is 64 bytes, so 8,192 blocks at 128K context is 512 kibibytes against 40.0 gibibytes of KV, a payload overhead of 0.0012 percent; metadata processing overhead is not established and must be measured. The scoring rule is LRFU — Lee et al., ACM SIGMETRICS 1999 — established prior art with alpha as its documented LRU-to-LFU knob; alpha = 0.2 was never validated and is withdrawn, the simulated optimum is about 0.05, and alpha is non-monotonic with no convergence onto LFU as alpha approaches zero. The largest effect in the study is the persistence of that state across the tier boundary: at a 32 gibibyte budget, LFU rises from 15.56 to 49.71 percent, plus 34.15 percentage points, and EMA alpha 0.01 from 31.54 to 56.39 percent, plus 24.85 percentage points, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points. That is a protocol result, testable with no CXL hardware, and it is provisional because a single data-structure defect in Revision 1 inverted four of five findings. Total tiered capacity is 192 GB hot, 1 TB warm across 4 times 256 GB, and 16 TB cold across 4 times 4 TB, about 17 TB against 192 GB for HBM alone — arithmetic over stated capacities, which stands.

GPU–Tier KV-Cache Interaction

Tiered Memory Architecture for Large Context Inference