Figure: effective latency for the four-tier hierarchy, L_eff = h_HBM × L_HBM + h_CXL × L_CXL + h_NVMe × L_NVMe. The tier latencies are 100 nanoseconds for GPU HBM (10 GB pinned plus a 37 GB evictable hot set), 200 nanoseconds for the 1 TB of CXL DRAM across four endpoints, and 25 microseconds for the 16 TB endpoint NVMe backing store. Against a PCIe baseline of 13.0 microseconds per access the CXL.mem path of 200 nanoseconds is 65× lower for a single 4 KiB access; loaded PCIe degrades further, by 50 to 200 percent from queue depth, 10 to 50 microseconds from interrupt coalescing and up to 100 microseconds of scheduling jitter. Worked at a 97 / 2.7 / 0.3 hit split the model gives 97 + 5.4 + 75 = 177.4 nanoseconds, about 177 ns, which is 11.5× lower than the 2,035 nanosecond baseline end to end. The point is that there is no recompute term — NVMe holds a complete backing store, so the worst case is a 25 microsecond read — and that the 0.3 percent NVMe tail still supplies 75 of the 177 nanoseconds, which is why lifting the HBM hit rate from a 72 percent LRU baseline to 97 percent is what matters. All values are an analytical model, not measured.

Section 4

Effective Latency Analysis

One model for the four-tier hierarchy: GPU HBM, CXL DRAM, endpoint NVMe

Effective Latency Formula
L_eff = h_HBM × L_HBM + h_CXL × L_CXL + h_NVMe × L_NVMe
L_HBM
GPU HBM3e — Tier 0 pinned (10 GB) and Tier 1 evictable hot set (37 GB)
100 ns
L_CXL
Tier 2 CXL DRAM (1 TB across 4 endpoints), reached by load/store
200 ns
L_NVMe
Tier 3 endpoint NVMe (16 TB) — the complete backing store
25 μs
Comparison to PCIe Baseline

PCIe swap path latency components:

Page fault
Driver intervention
DMA setup
PCIe transfer
Completion interrupt
65× lower — single access path only

Scope: 13,000 ns ÷ 200 ns = 65.0×, for one access measured across the same boundary (kernel issue → data available) with the same 4 KiB payload, comparing a load/store path against a driver-mediated DMA path. It is not an end-to-end system speedup — for that, see the effective latency below (2,035 ns baseline → 177 ns, 11.5× lower).

Under Load, PCIe Latency Degrades
Queue Depth
+50–200%
Interrupt Coalescing
+10–50 μs
OS Scheduling
+100 μs jitter
Example Calculation
Llama-2-70B, 128K context, 16 users
HBM Hit Rate
97%
CXL DRAM Hit Rate
2.7%
NVMe Hit Rate
0.3%
Substituting Values
L_eff = (0.97 × 100 ns) + (0.027 × 200 ns) + (0.003 × 25,000 ns)
L_eff = 97 ns + 5.4 ns + 75 ns
L_eff = 177.4 ns
Effective Latency
≈ 177 ns
There is no recompute term. Tier 3 endpoint NVMe holds a complete 16 TB backing store, so a KV entry is never lost and never has to be regenerated on an access miss — the worst case for any access is a 25 μs read, not a prefill. Recompute happens only on cold start and is modeled separately. The 0.3% NVMe tail still supplies 75 of the 177 ns, which is why raising the HBM hit rate from a 72% LRU baseline to 97% is what matters.
Intelligent Caching Prevents Thrashing

The endpoint's intelligent caching keeps the hot set resident—high-value entries are not evicted, so the miss tail stays small. EMA scoring (α = 0.2), anchor pinning, per-head budgets and RoPE-aware prefetch lift the modeled HBM hit rate from a 72% LRU baseline to 97% (+8, +7, +6, +4 percentage points; +25 points in total), while cold entries demote to CXL DRAM and then to NVMe.

Degradation figures above are illustrative ranges for a loaded PCIe path.
Analytical model — every latency, hit rate and derived result on this page is modeled, not measured. No hardware benchmark was run.