Every quantitative result in this package, in one place, tagged by evidence class. This chapter draws no new conclusions — it is the reference table the other chapters point back to.
The chapters preceding this one each argue a point and, along the way, produce numbers. This chapter does the opposite: it drops the argument and keeps the numbers. Every figure quoted anywhere in the package appears below, once, next to the evidence class that number is entitled to and nothing more. Three classes are used throughout, and the distinction between them is load-bearing, not decorative.
Measured means a number that came off real hardware running a real inference stack, on the date and configuration stated. Simulated, provisional means a number that came out of the KV-tiering simulator — a discrete-event model of hit rate and eviction behavior, not a running inference server — and carries the word "provisional" because this line of results was corrected once already after a defect in the simulator's handling of persistent metadata inverted four of five headline findings in the first revision. The numbers below are the corrected Rev 2 results; where earlier chapters or drafts of this package cited Rev 1 numbers, those have been superseded. Analytical means a number derived from architecture parameters and arithmetic — token dimensions, block sizes, published link bandwidths — rather than from running anything. None of the three classes is more or less legitimate than the others; they answer different questions, and conflating them is the single most common way a KV-cache-over-CXL argument goes wrong.
All measured results in this section come from one hardware run, dated 19 August 2026, on a DGX Spark GB10 node serving Qwen2.5-7B-Instruct in fp16. Two inference stacks were exercised on the same hardware: Hugging Face Transformers with SDPA attention, and vLLM with FlashAttention-2. Both fit a two-term linear model to decode latency as a function of resident KV-cache size, separating a fixed per-step cost from a per-gigabyte KV-read cost.
| Metric | HF + SDPA | vLLM + FlashAttention-2 |
|---|---|---|
| Decode-time fit | t = 100.3 + 16.34·KV_GiB ms (R² = 0.991) | t = 73.9 + 17.58·KV_GiB ms (R² = 0.945) |
| Fixed-path bandwidth | 151.5 GB/s | 205.7 GB/s |
| KV-read bandwidth | 65.7 GB/s | 61.1 GB/s |
Independent of the decode-time fits, a streaming-bandwidth microbenchmark on the same node measured 236.5 GB/s against a 273 GB/s specification ceiling — 87% of spec. This is a raw memory-streaming number and is not the same measurement as either stack's fixed-path or KV-read bandwidth above; it characterizes the platform's ceiling, not either inference stack's achieved share of it.
Two derived observations follow directly from Table 11.1 and are themselves measured (arithmetic on measured inputs, not simulation). First, the model-derived cost ratio between reading a KV byte and reading a fixed-path (weight) byte, using vLLM's own fitted bandwidths, is 205.7 ÷ 61.1 ≈ 3.4×: a KV byte is roughly three and a half times more expensive to read during decode than a fixed-path byte on this stack. Second, comparing the two stacks directly is a narrower claim than it looks: the KV-read slopes are within 8% of each other (16.34 vs. 17.58 ms/GiB) while the fixed-path bandwidth improved 34% between them (151.5 → 205.7 GB/s). That finding is scoped to exactly these two tested stacks on this one node and this one model — it is not a general claim about kernel substitution across attention implementations.
| Configuration | Throughput |
|---|---|
| Batch 16, 4K context | 150 tok/s |
| Batch 16, 16K context | 46 tok/s |
Throughput fell from 150 to 46 tok/s across that range with memory headroom to spare at every point measured — the drop tracks the growing KV-read cost in Table 11.1's fit, not a capacity ceiling. Capacity was never the binding constraint in this run; bandwidth was.
This section reports the corrected Rev 2 output of the KV-tiering simulator: Llama-3 70B under GQA, a 421 GiB full-context KV footprint, hit rate evaluated against a set of HBM budgets and eviction policies. These are simulator outputs, not measurements from a running server, and the "provisional" tag is a permanent part of how this line of results should be cited — Rev 1 of this simulator inverted four of its five headline findings due to a metadata-handling defect, and Rev 2 is the corrected replacement, not an independently reproduced result.
| HBM budget | LRU | LFU | EMA .15 | EMA .01 |
|---|---|---|---|---|
| 8 GiB | 8.52 | 8.76 | 12.35 | 12.12 |
| 16 GiB | 17.87 | 12.70 | 22.58 | 18.70 |
| 32 GiB | 38.34 | 16.09 | 44.18 | 35.91 |
| 64 GiB | 67.12 | 23.00 | 69.65 | 66.83 |
| 128 GiB | 88.01 | 32.70 | 88.14 | 91.59 |
LFU trails the other three policies at every budget in this no-persistence baseline — its inability to age out stale frequency counts costs it hit rate consistently. The two EMA (decayed-frequency) variants and plain LRU are otherwise closely matched, with the ranking among them shifting by budget size rather than one policy dominating throughout.
| Policy | Hit rate |
|---|---|
| EMA α = 0.5 | 37.20 |
| EMA α = 0.3 | 38.84 |
| EMA α = 0.15 | 44.33 |
| EMA α = 0.05 (optimum) | 55.96 |
| EMA α = 0.01 | 31.54 |
| EMA α = 0.003 | 25.47 |
| EMA α = 0.001 | 24.44 |
| LFU | 15.56 |
| LRU | 36.85 |
The α grid shows a non-monotonic peak, not a "smaller decay is always better" relationship: hit rate rises from α = 0.5 down to an optimum at α = 0.05 (55.96%), then falls off sharply as α continues toward zero, bottoming near 24–25% at the smallest decay constants tested. Both extremes underperform plain LRU (36.85%) and LFU (15.56%) sits well below all EMA settings tested.
| Policy | Without persistence | With persistence | Change |
|---|---|---|---|
| LFU | 15.56% | 49.71% | +34.15 pts |
| EMA .01 | 31.54% | 56.39% | +24.85 pts |
| EMA .15 | 44.33% | 44.12% | −0.21 pts |
This is the simulator's central finding, and its shape matters as much as its magnitude: persisting eviction-policy metadata across a tier eviction helps the policies that were being hurt most by starting cold — LFU gains 34.15 points, the slowest-decaying EMA variant gains 24.85 — and is essentially neutral for a policy already tuned to a fast decay constant (EMA .15 moves by −0.21 points, within noise). Persistent metadata is not a uniform multiplier on hit rate; it rescues policies whose accumulated state was otherwise being thrown away on every eviction, and does little for a policy that wasn't relying on that state in the first place.
| Workload | LRU | LFU | EMA .15 | EMA .01 |
|---|---|---|---|---|
| Zipf | 39.08 | 15.12 | 44.44 | 38.87 |
| Scan | 31.51 | 12.52 | 32.57 | 36.26 |
| Loop | 9.83 | 22.06 | 9.83 | 9.83 |
No policy is best across all three access patterns. Under Zipf and scan workloads, EMA .15 and EMA .01 lead or are competitive with LRU; under a strict loop workload, LFU is the only policy tested that outperforms the rest, and the three recency-weighted policies (LRU, EMA .15, EMA .01) converge to an identical 9.83%, because a pure loop pattern gives recency signal no traction. Policy choice is workload-dependent, not a settled ranking.
Separately, the simulator's turn-level accounting finds that recompute time — the cost of regenerating KV state that was not found in cache — makes up 44–89% of turn stall time depending on configuration, but that this share collapses to under 1% under an idealized model of transfer/compute overlap, where fetch time (0.35–1.25 ms per turn) is fully hidden behind ongoing computation. That range brackets a best case that assumes perfect overlap and should be read as a ceiling on what overlap can buy, not an expected operating point.
These figures come from architecture arithmetic — token and block dimensions, published link bandwidths, and simple capacity ratios — not from a benchmark or a simulator run.
| Quantity | Value | Basis |
|---|---|---|
| Per-token KV footprint | 320 KiB / token | Llama-3 70B, GQA head/layer arithmetic |
| Per-block KV footprint | 5.00 MiB / 16-token block | 320 KiB/token × 16 |
| Memory expansion factor | 6× | Simple capacity arithmetic, CXL pool vs. GPU-local HBM |
| Tier-fetch cost per block @ 121 GB/s | 43 µs | 5.00 MiB ÷ 121 GB/s |
| Tier-fetch cost per block @ 64 GB/s (assumption) | 82 µs | 5.00 MiB ÷ 64 GB/s, assumed link rate |
| Tier-fetch cost per block @ 32 GB/s | 164 µs | 5.00 MiB ÷ 32 GB/s |
| Move-vs-regenerate cost gap | 2–3 orders of magnitude cheaper to move than regenerate | Order-of-magnitude comparison, not a precise ratio |
| KV-vs-weight crossover point | ≈ 427,000 tokens | Llama-3 70B: context length at which resident KV size equals model weight size |
Two of these figures deserve a specific caveat rather than being read as fixed constants. The 64 GB/s tier-fetch cost is explicitly an assumed link rate, not a measured or vendor-specified one — it sits between the 121 GB/s and 32 GB/s figures as a bracketing scenario, and any of the three link rates should be treated as a modeling input to be swapped for whatever the actual deployment's fabric delivers. The move-vs-regenerate gap is stated deliberately as an order-of-magnitude range (2–3 orders) rather than a single ratio, because the true cost of regenerating a KV block depends on prefill batch size, sequence position, and attention kernel in ways this package has not pinned down to a single number — treat "2–3 orders of magnitude" as the claim, not a specific multiplier within it.
Any number pulled from this chapter into a summary, a slide, or another document should travel with its evidence tag attached. A measured decode-time slope from Section 11.1 describes one model, one stack, one node, one date — it is not a general claim about all fp16 decode on all hardware. A hit-rate figure from Section 11.2 describes one simulator's output under one workload model, is provisional pending independent validation against a real tiering implementation, and should never be quoted without the word "simulated." An analytical figure from Section 11.3 is exact given its stated assumptions and only as good as those assumptions. Keeping these three classes visibly separate — in this chapter and wherever these numbers are reused — is the discipline this package is asking readers to hold it to.