SUPERSEDED — this figure illustrates a claim withdrawn in v5.0. It is retained only so the retraction remains checkable. Do not cite it. See RETRACTIONS.md and Appendix L (hardware measurements) and Appendix N (simulation) for what replaced it.

Figure: the bandwidth-compute gap that starves modern AI infrastructure, in five stacked panels. An NVIDIA B200 supplies 2,250 TFLOPS of dense BF16 compute, 8 TB/s of HBM3e bandwidth and 192 GB of capacity, which means it needs an arithmetic intensity of 281 FLOPs per byte to stay busy — and transformer attention, dominated by KV-cache reads, delivers far fewer. Llama-70B costs 320 KiB per token, so a 128K context is 43 GB at batch 1 and 1.38 TB at batch 32, while a 4K context is 1.3 GB at batch 1 and 43 GB at batch 32. That opens two walls: a capacity wall of 1,376 GB divided by 192 GB, or 7.2×, and a bandwidth wall of 8,000 GB/s of HBM against 51 GB/s of link-level PCIe Gen5 by 16, or 157×. Reading one user's 43 GB cache therefore takes 5.4 milliseconds from HBM but 843 milliseconds over PCIe at that link-level rate, idling the GPU for 1,897 × 10 to the 12th wasted FLOPs. A closing note distinguishes the two PCIe figures used in this package: 51 GB per second is the link-level uncontended rate, a vendor specification, while 16.4 GB per second is the contended end-to-end rate the canonical time-to-first-token derivation assumes for the host-DRAM offload path, an analytical model. The second is about 32 percent of the first, and that ratio is the assumed contention factor. The point is that the GPU does not lack compute, it lacks data. All values are an analytical model, not measured.

The Bandwidth-Compute Gap

Why Modern AI Infrastructure Starves for Data

NVIDIA B200
Compute (BF16/FP16 Dense)
2,250
TFLOPS
HBM3e Bandwidth
8
TB/s
HBM Capacity
192
GB
Arithmetic Intensity Required: 281 FLOPs/byte (2,250,000 GFLOP/s ÷ 8,000 GB/s) — 4,500 TFLOPS is the 2:4-sparsity figure, not dense
THE PROBLEM: Attention is Memory-Bound

Transformer attention: dominated by KV-cache reads

Few FLOPs per byte loaded → GPU waits for data
Llama-70B KV-Cache Size
2 (K,V) × 80 layers × 8 KV-heads × 128 dim × 2 B (BF16) = 320 KiB/token × seq_len × batch
128K Context
Batch 1
43 GB
Batch 32
1.38 TB
4K Context
Batch 1
1.3 GB
Batch 32
43 GB
CAPACITY WALL
KV-Cache (Batch 32 @128K)
1,376 GB
÷
B200 HBM3e Capacity
192 GB
Working set exceeds on-chip memory by
7.2×
32 × 43 GB = 1,376 GB. This raw ratio is not the 16× user-capacity headline, which is derived from the 37 GB HBM hot set (37 ÷ 16 users = 2.31 GB/user).
BANDWIDTH WALL
B200 HBM3e Bandwidth
8,000 GB/s
÷
PCIe Gen5 ×16 link-level
51 GB/s
Off-chip streaming-bandwidth penalty
157×
A bulk-bandwidth ratio. Distinct from the 65× access-latency headline (13.0 µs PCIe DMA ÷ 200 ns CXL.mem).
LATENCY IMPACT: 43 GB KV-Cache Read (one user @128K)
GPU idle during PCIe swap: 843 ms × 2,250 TFLOPS = 1,897 × 10¹² wasted FLOPs
Two PCIe numbers, two different things

This page uses 51 GB/s. The canonical time-to-first-token derivation uses 16.4 GB/s. They are not in conflict — they measure different things, and both appear in this package on purpose:

  • 51 GB/s — link-level, uncontended (Vendor specification). What a PCIe Gen5 ×16 link delivers to a single clean bulk streaming transfer. The raw signalling rate is 32 GT/s × 16 lanes ≈ 63 GB/s per direction; ~51 GB/s is what survives 128b/130b encoding, TLP/DLLP headers and flow-control credit turnaround when nothing else is using the link. This is the figure that makes the 157× bandwidth-wall ratio above an optimistic bound: the real gap is wider.
  • 16.4 GB/s — contended, end-to-end (Analytical model). What the modeled host-DRAM offload path is assumed to sustain in service, once protocol overhead is charged and the link is shared with weight, activation and NIC traffic on a common root complex. This is the baseline in the 15.6× TTFT derivation: 43 GB ÷ 16.4 GB/s = 2,619 ms.

The ratio between them is the assumed contention factor: 16.4 ÷ 51 = 0.32 — the model gives the offload path ~32% of link-level bandwidth. That factor is an assumption, not a measurement, and it is the single number to argue with if you think the baseline is unfair: a less contended host would raise 16.4 GB/s toward 51 GB/s and shrink the TTFT speedup proportionally. See Appendix K — Methodology for the full TTFT derivation and Chapter 11 for its sensitivity to this assumption.

THE BOTTLENECK
The GPU doesn't lack compute. It lacks data.
The capacity gap (7.2× at batch 32) forces off-chip storage. The bandwidth gap (157×) makes naive off-chip streaming catastrophic.
Sources
B200 specs: NVIDIA Datasheet (vendor specification) | Llama-70B: NVIDIA NeMo / Meta
PCIe Gen5 ×16 link-level 51 GB/s: PCI-SIG Base Specification 5.0 signalling rate less encoding and protocol overhead (vendor specification).
Contended offload rate 16.4 GB/s and the 15.6× TTFT result: Appendix K — Methodology (analytical model).
All derived quantities on this page are an analytical model, not measured results.