Figure: a step-by-step comparison of two memory access paths for the same 4 KiB read. The PCIe DMA baseline takes 13.00 microseconds in seven steps — 4.00 µs driver call and doorbell, 2.50 µs DMA descriptor fetch, 1.50 µs software TLB and pinning, 0.70 µs PCIe Gen5 round trip, 0.10 µs host DRAM access, 0.20 µs payload transfer and 4.00 µs completion interrupt and resync. The CXL.mem direct path takes 200 nanoseconds in six steps — 15 ns request issue, 20 ns PCIe Gen5 x16 PHY, 25 ns switch traversal, 40 ns endpoint controller decode, 80 ns DDR5-5600 array access and 20 ns return path. The comparison bars below rank four latencies against the 25 microsecond NVMe reference: 177 ns effective at the 97 / 2.7 / 0.3 split, 200 ns CXL.mem DRAM, 13,000 ns PCIe DMA at 65× slower, and 25,000 ns endpoint NVMe at 125× slower. The point is that most of the PCIe cost is per-access CPU and driver work that a load/store path never performs. All values are an analytical model, not measured.

Memory Access Path Comparison

CXL.mem Direct Path vs CPU-Mediated PCIe Swap

PCIe DMA Baseline Path
13.00 µs
1
Driver call + doorbell
Ring the device doorbell from the driver
4.00 µs
2
DMA descriptor fetch/setup
Descriptor allocation and fetch
2.50 µs
3
Software TLB + pinning management
Page pinning, IOMMU mapping
1.50 µs
4
PCIe Gen5 round trip
SerDes + switch, both directions
0.70 µs
5
Host DRAM access
DDR5 array read on the host
0.10 µs
6
4 KiB payload @ 32 GB/s
4,096 B ÷ 32 GB/s = 128 ns, rounded up for framing
0.20 µs
7
Completion interrupt + resync
MSI-X, handler, GPU resync
4.00 µs

4.00 + 2.50 + 1.50 + 0.70 + 0.10 + 0.20 + 4.00 = 13.00 µs

CXL.mem Direct Path
200 ns
1
GPU/host request issue
Load instruction issued, M2S MemRd formed
15 ns
2
PCIe Gen5 ×16 PHY
SerDes transit, outbound
20 ns
3
CXL switch traversal
Non-blocking CXL 3.0 switch
25 ns
4
Endpoint CXL controller decode
HDM decode, address translation
40 ns
5
DDR5-5600 array access
Endpoint DRAM read
80 ns
6
Return path
Response through switch and PHY, writeback
20 ns

15 + 20 + 25 + 40 + 80 + 20 = 200 ns

Latency Comparison
Effective (97/2.7/0.3)
CXL.mem DRAM (Tier 2)
PCIe DMA baseline
Endpoint NVMe (Tier 3)

Bars are linear in ns against the 25 µs NVMe reference; the two sub-microsecond bars are widened to stay legible. Effective access latency = 0.97 × 100 ns + 0.027 × 200 ns + 0.003 × 25 µs = 177.4 ns.

Key Insight: CPU Elimination

CXL.mem removes per-access CPU involvement in steady state: no interrupt, no context switch, no DMA setup. The GPU issues a load and receives data from endpoint DRAM through hardware-managed coherence. Control plane, mapping, faults, drivers and firmware still exist. Scope of the 65×: 13,000 ns ÷ 200 ns, one access, same boundary (kernel issue → data available), same 4 KiB payload.

65×

Analytical model — every component latency here is modeled from vendor specifications and protocol behaviour. No hardware benchmark was run.