Figure: the bandwidth-compute gap that starves modern AI infrastructure, in six stacked panels. An NVIDIA B200 supplies 2,250 TFLOPS of dense BF16 compute, 8 TB/s of HBM3e bandwidth and 192 GB of capacity, which means it needs an arithmetic intensity of 281 FLOPs per byte to stay busy, and transformer attention, dominated by KV-cache reads, delivers far fewer. Llama-70B costs 320 kibibytes per token, so an 8K context is 2.50 gibibytes, a 128K context is 40.0 gibibytes or about 43 gigabytes at batch 1 and 1.38 terabytes at batch 32, and a 4K context is 1.3 gigabytes at batch 1 and 43 gigabytes at batch 32; for a single sequence the KV only exceeds the weights at about 427,000 tokens. That opens two walls: a capacity wall of 1,376 gigabytes divided by 192 gigabytes, or 7.2 times, and a bandwidth wall of 8,000 gigabytes per second of HBM against 51 gigabytes per second of link-level PCIe Gen5 by 16, or 157 times. Reading one user’s 43 gigabyte cache therefore takes 5.4 milliseconds from HBM but 843 milliseconds over PCIe at that link-level rate, idling the GPU for 1,897 times 10 to the 12th wasted FLOPs. Real hardware data has since replaced the estimates this figure used to carry. On a DGX Spark GB10 measured on 19 August 2026 with Qwen2.5-7B in fp16, achieved streaming bandwidth is 236.5 gigabytes per second, which is 87 percent of the 273 gigabyte per second spec. That retires the roughly 50 percent effective-bandwidth haircut used in earlier drafts, and it withdraws the 16.4 gigabyte per second contended host-DRAM offload baseline this figure previously assumed, because the class of pessimistic estimate that baseline belonged to proved unreliable. Decode time is also two-term rather than one-term: t equals 100.3 plus 16.34 times KV in gibibytes for HuggingFace with SDPA at R squared 0.991, and t equals 73.9 plus 17.58 times KV in gibibytes for vLLM with FlashAttention-2 at R squared 0.945, giving inferred fixed-path bandwidths of 151.5 and 205.7 gigabytes per second against inferred KV-path bandwidths of 65.7 and 61.1. Under that fit the KV path is about 3.4 times more costly per byte, a model-derived ratio rather than a direct byte-level hardware measurement, and kernel substitution alone did not remove the KV slope, which stayed within 8 percent across the two frameworks. Four claims this figure used to make are retracted: the 15.6 times time-to-first-token result, because it rested on the withdrawn 16.4 gigabyte per second baseline; the 16 times user-capacity headline, because it was downstream of a 97 percent hit rate that was never measured; the 65 times access-latency headline of 13.0 microseconds of PCIe DMA against 200 nanoseconds of CXL.mem, because no CXL hardware was measured; and the 36 percent CapEx reduction, because it priced a configuration justified by the 16 times. No replacement cost figure is offered. The point that survives is that the GPU does not lack compute, it lacks data.
Why Modern AI Infrastructure Starves for Data
Transformer attention: dominated by KV-cache reads
Few FLOPs per byte loaded → GPU waits for dataThis page uses 51 GB/s, the link-level uncontended PCIe Gen5 ×16 rate (Vendor specification): the raw signalling rate is 32 GT/s × 16 lanes ≈ 63 GB/s per direction, and ~51 GB/s is what survives 128b/130b encoding, TLP/DLLP headers and flow-control credit turnaround when nothing else is using the link. It makes the 157× ratio above an optimistic bound.
Earlier versions of this figure also carried a 16.4 GB/s contended end-to-end baseline, and derived a 15.6× TTFT result from it (43 GB ÷ 16.4 GB/s = 2,619 ms). Both are retracted. The 16.4 GB/s figure was an undefended estimate — a pessimistic haircut applied by convention rather than measurement — and the first real measurement on this package's hardware shows that class of estimate is unreliable. Baselines must be measured, not estimated. No replacement contention factor is offered, and no TTFT speedup is claimed.
Achieved streaming bandwidth is 236.5 GB/s = 87% of the 273 GB/s spec (streaming copy, 4 GiB buffers, median of 20). This retires the ~50% "effective bandwidth" haircut used throughout earlier drafts — the haircut was wrong, and wrong in the pessimistic direction, which is why it survived unexamined.
And decode time is two-term, not one-term: t = 100.3 + 16.34·KV_GiB (HF + SDPA, R² 0.991) and t = 73.9 + 17.58·KV_GiB (vLLM + FA2, R² 0.945). That gives an inferred fixed-path bandwidth of 151.5 / 205.7 GB/s against an inferred KV-path bandwidth of 65.7 / 61.1 GB/s: under that fit the KV path is ~3.4× more costly per byte, a model-derived ratio, not a direct byte-level hardware measurement. Kernel substitution alone did not remove the KV slope — it stayed within 8% across two frameworks with different attention implementations and paged vs unpaged layout, while the inferred fixed-path bandwidth improved 34% between them.