Hardware Measurements (DGX Spark GB10)
The first measured evidence in this package: achieved streaming bandwidth, a two-term decode model fitted on two frameworks, the raw tables behind both fits, the method, and the two measurement bugs found and fixed on the way.
Everything below was run on named hardware on 19 August 2026 and is reported with its method, its repetition count and its raw data. Nothing here is simulated, modelled or projected. Every other class of evidence in this package — Analytical model, Simulated (provisional), Vendor specification, External literature, Design target, Illustrative — is defined and located in Appendix K §K.1. The simulation results, which are a different and weaker class of evidence, are in Appendix N and always carry a provisional caveat.
L.1 Platform and Model Geometry
| Parameter | Value | Evidence |
|---|---|---|
| Machine | DGX Spark, GB10 Grace Blackwell | Measured |
| Memory | 128 GiB coherent unified LPDDR5X | Vendor specification |
| Memory bandwidth, spec figure | 273 GB/s | Vendor specification |
| Date of runs | 19 August 2026 | Measured |
| Model under test | Qwen2.5-7B-Instruct, fp16 | Measured |
| Layers | 28 | Measured |
| KV heads | 4 | Measured |
| head_dim | 128 | Measured |
| KV per token | 56 KiB/token | Analytical model |
| Weights resident | 15.2 GB = 14.16 GiB | Measured |
| Scripts | ../scripts/bw_wall.py, ../scripts/vllm_slope.py | — |
The KV figure follows the standard sizing identity of Appendix C: 2 × 28 layers × 4 KV heads × 128 × 2 B = 57,344 B = 56 KiB/token. The decisive term is KV heads, not query heads — the same correction that governs the Llama capacity table in CANONICAL-NUMBERS.md §2.
L.2 Achieved Streaming Bandwidth
A streaming copy kernel (read + write), 4 GiB buffers, median of 20
repetitions. Measured, script ../scripts/bw_wall.py.
Earlier drafts applied an undefended ~50% "effective bandwidth" haircut to the theoretical figure. That haircut is retired by this measurement, and it was wrong in the pessimistic direction: real achieved bandwidth on this part is 87% of spec, not 50%. The consequence is not local. The retracted v4.0 TTFT claim of 15.6× rested on a 16.4 GB/s sustained-offload assumption of exactly this kind, and its retraction (see CANONICAL-NUMBERS.md §7, and the reconciliation note in Appendix K) follows directly from the fact that a measured achieved-versus-spec ratio contradicts it.
L.3 The Decode Model Is Two-Term, Not One-Term
The single-bandwidth roofline t = (W + KV) / B does not hold on this hardware. Sweeping resident KV volume across a batch × context grid and fitting a two-term model t = a + b·KV_GiB separates the constant weight read from the KV read that grows with context:
| Framework | Fit (ms) | n | R² | Weight bandwidth | KV bandwidth |
|---|---|---|---|---|---|
| HuggingFace + SDPA, unpaged | 100.3 + 16.34·KV_GiB | 7 | 0.991 | 151.5 GB/s | 65.7 GB/s |
| vLLM 0.20.1 + FlashAttention-2, paged | 73.9 + 17.58·KV_GiB | 9 | 0.945 | 205.7 GB/s | 61.1 GB/s |
Evidence: Measured (the times and the fit); Analytical model (the two bandwidths, which are derived from the fit coefficients and the resident weight bytes, not read off an instrument).
The derived effective bandwidths follow from the two coefficients. The intercept a is the KV-independent term and is charged against the 15.2 GB of resident weights; the slope b is milliseconds per GiB of resident KV:
L.3.1 As a fraction of the bandwidth actually achieved
The reference point is not the 273 GB/s spec figure but the 236.5 GB/s the same machine achieves on a streaming copy (§L.2). Against that:
| Read | HuggingFace + SDPA | vLLM + FlashAttention-2 |
|---|---|---|
| Weight read | 151.5 GB/s = 64% | 205.7 GB/s = 87% |
| KV read | 65.7 GB/s = 28% | 61.1 GB/s = 26% |
vLLM extracts near-peak achievable bandwidth on weights and roughly a quarter of it on KV, in the same kernel launch, on the same memory.
L.4 The Three Results That Carry the Argument
26–28% of 236.5 GB/s across both frameworks. Measured. This is an HBM-side property of the attention read pattern, not a property of what sits behind HBM.
205.7 ÷ 61.1 = 3.4×, taken within the vLLM configuration so that the comparison is like-for-like. Measured inputs, ratio Analytical model. The intercept contains fixed decode work in addition to weight traffic, so this is not a direct measurement of the cost of an individual weight byte versus an individual KV byte. The consequence for the objective function of Appendix M §M.4 is direct: its byte-counting terms must be weighted, because a KV byte and a weight byte are not interchangeable.
Two independent frameworks, two different attention implementations, paged versus unpaged KV layout — and the KV slopes land within 8% of each other (16.34 vs 17.58 ms/GiB). Over the same change the weight read improves by 34% (151.5 → 205.7 GB/s), reaching 87% of achievable bandwidth. Within these two tested stacks, switching attention implementation and KV layout did not materially change the fitted KV slope. This is evidence about this GB10/Qwen2.5-7B operating region, not a claim about every kernel, GPU architecture, page size, model, or decode regime. Measured.
L.5 What This Settles
L.5.1 Reducing KV traffic is the first-order opportunity exposed by this measurement
The measurement supports a narrower conclusion: reducing KV traffic is a first-order opportunity. Quantisation or compression reduces bytes per KV element; sparse or hierarchical selection can reduce the number of KV elements consumed. Placement remains necessary for capacity, reuse and movement, but moving a resident working set to another tier does not by itself reduce the bytes that dense attention must consume once those bytes are presented to the GPU. The relative end-to-end benefit of compression, selection and placement remains an experimental question (CANONICAL-NUMBERS.md §5).
L.5.2 Capacity was never the binding constraint here
Every configuration measured fit comfortably in 128 GiB. Throughput collapsed anyway:
Nothing was evicted, nothing was offloaded, and no placement policy ran. A 3.3× fall in decode throughput arrived purely from re-reading more KV per step. Any argument that reaches for a capacity mechanism to explain a throughput fall of this shape is reaching for the wrong mechanism.
§L.5.2 says capacity was not binding in the configurations measured here — one 7B model on one 128 GiB machine. It is not a claim that capacity never binds; the 421 GiB footprint studied in Appendix N is a regime where it plainly does. The two appendices answer different questions and must not be merged into a single narrative. Appendix N's numbers are Simulated (provisional); nothing on this page is.
L.6 Raw Data
Both tables in full. Each row is the median of 20–30 timed decode steps after an untimed warm run, with the step time isolated by the differencing procedure of §L.7.
L.6.1 HuggingFace + SDPA (fp16, chunked prefill, unpaged)
| batch | ctx | KV GiB | measured ms/step | tok/s | KV % of bytes |
|---|---|---|---|---|---|
| 1 | 4096 | 0.22 | 104.6 | 9.6 | 2% |
| 1 | 16384 | 0.88 | 113.9 | 8.8 | 6% |
| 1 | 32768 | 1.75 | 128.5 | 7.8 | 11% |
| 4 | 4096 | 0.88 | 111.8 | 35.8 | 6% |
| 4 | 16384 | 3.50 | 155.4 | 25.7 | 20% |
| 4 | 32768 | 7.00 | 212.4 | 18.8 | 33% |
| 16 | 4096 | 3.50 | 165.1 | 96.9 | 20% |
n = 7. Fit t = 100.3 + 16.34·KV_GiB ms, R² = 0.991.
Script ../scripts/vllm_slope.py carries the HuggingFace path as well as the vLLM path.
L.6.2 vLLM 0.20.1 + FlashAttention-2 (fp16, paged, prefix caching disabled)
| batch | ctx | KV GiB | measured ms/step | tok/s |
|---|---|---|---|---|
| 1 | 4096 | 0.22 | 95.1 | 10.5 |
| 1 | 16384 | 0.88 | 101.5 | 9.8 |
| 1 | 32000 | 1.71 | 107.8 | 9.3 |
| 4 | 4096 | 0.88 | 99.9 | 40.0 |
| 4 | 16384 | 3.50 | 117.4 | 34.1 |
| 4 | 32000 | 6.84 | 162.2 | 24.7 |
| 16 | 4096 | 3.50 | 127.5 | 125.5 |
| 16 | 16384 | 14.00 | 345.9 | 46.2 |
| 32 | 4096 | 7.00 | 185.0 | 173.0 |
n = 9. Fit t = 73.9 + 17.58·KV_GiB ms, R² = 0.945.
The 16 × 16384 row sits 26 ms above the fit and is the
single noisiest point in either table; see §L.9. Script
../scripts/vllm_slope.py.
The 150 tok/s → 46 tok/s collapse quoted in §L.5.2, in data/measured_results.md §3 and in CANONICAL-NUMBERS.md §1.3 has its low end exactly in this table: 46.2 tok/s at batch 16 × 16384. Its high end does not. The batch-16 × 4K row here reads 125.5 tok/s (16 ÷ 127.5 ms), not 150. The two are not reconciled in the source material, so the discrepancy is reported rather than quietly resolved. CANONICAL-NUMBERS.md governs the quoted headline; the table above governs any re-derivation, and a re-derivation from the table gives a 2.7× fall rather than 3.3×. The direction and the conclusion — capacity was not binding, bandwidth was — are unaffected either way. A repeat of the batch-16 sweep should settle it, alongside the 16 × 16384 outlier noted above.
L.7 Method
Decode step time is isolated by differencing, which cancels prefill and warm-up without requiring either to be modelled:
| Control | Setting | Why |
|---|---|---|
| Warm run | One untimed generation before every timed configuration | Removes allocator, autotune and JIT effects from the timed window |
| Repetitions | Medians of 20–30 steps | Median rather than mean, so a single scheduling outlier does not move the fit |
| Batch construction | Distinct token ids per sequence | Guarantees no KV is shared between sequences in a batch — see bug 2 in §L.8 |
| Prefix caching | enable_prefix_caching=False (vLLM) | Same reason, enforced at the framework level as well as at the input |
| Batch shape | Fixed-length uniform batches | Makes resident KV exactly computable as B × L × 56 KiB |
L.8 Provenance Discipline — Two Bugs Found and Fixed
Both are recorded, and recorded in full, because both produced plausible-looking wrong answers. A measurement appendix that lists only the runs that worked is not a measurement appendix.
L.8.1 Bug 1 — silent fp32 load
torch_dtype= is deprecated in current transformers and was silently ignored. Weights loaded in fp32 — 28.4 GiB instead of 14.16 GiB — and the run triggered the OOM killer. The failure mode is worse than it sounds: unified memory surfaces a CUDA OOM as a Linux OOM kill, which destroys the process without a CUDA error and obscures the cause entirely.
Fix. Use dtype=. The script now prints resident weight bytes and dtype at load, so a recurrence is visible in the log rather than inferred from a corpse.
L.8.2 Bug 2 — prefix-sharing artifact
The first vLLM run generated identical token ids for every sequence in the batch. vLLM's prefix caching therefore stored one shared copy of the KV, so true resident KV was L × bpt, not B × L × bpt. The x-axis of the fit was inflated by a factor of B, and the apparent KV bandwidth came out at:
Fix. Distinct tokens per sequence, plus enable_prefix_caching=False.
Any measured bandwidth above the bus is a bug, always. That rule is what caught this one. The instructive part is that bug 2 did not produce an obviously broken number. It produced a better number — one that would have supported a more exciting conclusion ("vLLM is 5× more efficient on KV") and that no amount of code inspection was going to volunteer. It was caught by checking the result against a physical bound, not by reading the script. A measurement that flatters the hypothesis deserves more scrutiny than one that does not, and the only reliable scrutiny is a bound the result cannot legally cross.
L.9 Limitations
- One model, one size, one machine. Qwen2.5-7B on a single GB10. No claim is made about H100, MI300, or larger models. The ~3.4× model-derived effective-path ratio is specific to this GB10 fit.
- Fixed-length uniform batches. No continuous batching with mixed arrival, no ragged sequence lengths. Real serving is neither.
- The vLLM fit is noisier (R² 0.945 against 0.991), and the 16 × 16384 point sits 26 ms above the fit. That point needs a repeat before it is relied on individually; the slope is reported with it included.
- No thermal log was captured. EC firmware is 0x03000508, past the versions reported to throttle, and idle temperatures were 45–49 °C — but sustained-load clocks were not recorded. A repeat should log clocks.sm alongside the timings.
- FlashAttention-2, not 3. vLLM selected FA2 on this platform. FA3, or a flash-decoding split-K path, may behave differently at batch 1, where a single query attends to 32K keys with little parallelism to exploit.
- fp8 KV not yet measured. This is the gap that matters most, and §L.10 states exactly what closing it would decide.
L.10 The Next Measurement — fp8 KV
kv_cache_dtype=fp8 halves KV bytes per token, from 56 KiB to 28 KiB. That run is the direct test of the compression claim of §L.5.1, and it has exactly two possible outcomes. Both are named here in advance, so that neither can be presented afterwards as the one that was expected.
| Outcome | Signature | What it would mean |
|---|---|---|
| A — bytes are what matter | Slope holds near 17.6 ms/GiB while the GiB halve | ~2× on the KV term. Confirms that the wall is a byte count and that byte reduction materially improves the KV term on this configuration. The objective function's weighting in Appendix M §M.4 stands as written. |
| B — dequantisation cancels the saving | Slope roughly doubles to ~35 ms/GiB | Dequantisation cost cancels the byte saving. fp8 buys capacity but not bandwidth on this hardware, and any claim that compression should be preferred over placement would remain unsupported; representation overhead must be measured separately. |
Either outcome is a result. The second would be the more interesting one, and it is the one that would cost this package a conclusion it currently holds — which is precisely why it is stated here before the run rather than after it.