Figure: measured decode step time against resident KV volume on a DGX Spark GB10, for two inference frameworks. HuggingFace with SDPA and unpaged KV fits t = 100.3 + 16.34 times KV_GiB milliseconds over 7 points, R squared 0.991; its measured points are 0.22 GiB at 104.6 ms, 0.88 at 113.9, 1.75 at 128.5, 0.88 at 111.8, 3.50 at 155.4, 7.00 at 212.4 and 3.50 at 165.1. vLLM 0.20.1 with FlashAttention-2 and paged KV fits t = 73.9 + 17.58 times KV_GiB over 9 points, R squared 0.945; its measured points are 0.22 GiB at 95.1 ms, 0.88 at 101.5, 1.71 at 107.8, 0.88 at 99.9, 3.50 at 117.4, 6.84 at 162.2, 3.50 at 127.5, 14.00 at 345.9 and 7.00 at 185.0. The two fitted lines are nearly parallel: the slopes, which are the KV term, differ by 8 per cent (16.34 against 17.58 ms per GiB), while the intercepts, which are the weight term, differ by 26 per cent (100.3 against 73.9 ms). Over the same change of framework the weight read improves 34 per cent, from 151.5 to 205.7 GB/s, while the KV read stays at 65.7 against 61.1 GB/s. A KV byte therefore costs 3.4 times a weight byte on this hardware. Changing the attention kernel moves the intercept and leaves the slope where it was: the kernel is not the lever. The vLLM batch 16 by 16384 point sits 26 ms above its fit and needs a repeat. A faint third line shows the HuggingFace fit translated down by the 26.4 ms intercept difference; it is a construction rather than data, and it tracks the vLLM fit to within 17 ms over the whole range.
Figure 2.6 · Measured
The decode model is two-term, and the second term is the KV
Decode step time against resident KV volume, swept over a batch × context grid on a DGX Spark (GB10 Grace Blackwell, 128 GiB unified LPDDR5X, 273 GB/s spec) running Qwen2.5-7B-Instruct in fp16. Two frameworks with different attention implementations and different KV layouts — paged against contiguous. The lines are nearly parallel. Swapping the kernel moves the intercept down 26% and leaves the slope within 8% of where it was.
The intercept is the weight term: the fixed cost of streaming 14.16 GiB of weights once per decode step, independent of how much KV is resident. The slope is the KV term: the marginal cost of each additional GiB of resident KV that attention must re-read every step. vLLM extracts near-peak bandwidth on the weights (205.7 GB/s, 87% of the 236.5 GB/s this machine actually achieves) and about a quarter of it on the KV (61.1 GB/s, 26%). HuggingFace reads weights at 151.5 GB/s (64%) and KV at 65.7 GB/s (28%). A better attention kernel fixes the weight path and leaves the KV path where it was. Each fitted line is drawn only across the KV range its own data covers, extended to KV = 0 to show the intercept.
Show the data table
| Framework | batch × context | Resident KV (GiB) | ms / decode step |
|---|---|---|---|
| HuggingFace + SDPA | 1 × 4096 | 0.22 | 104.6 |
| HuggingFace + SDPA | 1 × 16384 | 0.88 | 113.9 |
| HuggingFace + SDPA | 1 × 32768 | 1.75 | 128.5 |
| HuggingFace + SDPA | 4 × 4096 | 0.88 | 111.8 |
| HuggingFace + SDPA | 4 × 16384 | 3.50 | 155.4 |
| HuggingFace + SDPA | 4 × 32768 | 7.00 | 212.4 |
| HuggingFace + SDPA | 16 × 4096 | 3.50 | 165.1 |
| vLLM + FlashAttention-2 | 1 × 4096 | 0.22 | 95.1 |
| vLLM + FlashAttention-2 | 1 × 16384 | 0.88 | 101.5 |
| vLLM + FlashAttention-2 | 1 × 32000 | 1.71 | 107.8 |
| vLLM + FlashAttention-2 | 4 × 4096 | 0.88 | 99.9 |
| vLLM + FlashAttention-2 | 4 × 16384 | 3.50 | 117.4 |
| vLLM + FlashAttention-2 | 4 × 32000 | 6.84 | 162.2 |
| vLLM + FlashAttention-2 | 16 × 4096 | 3.50 | 127.5 |
| vLLM + FlashAttention-2 | 16 × 16384 | 14.00 | 345.9 |
| vLLM + FlashAttention-2 | 32 × 4096 | 7.00 | 185.0 |
Measured — DGX Spark GB10, 19 Aug 2026, scripts/bw_wall.py