Figure: a sixteen-part visual technical appendix on KV-cache offloading for Llama-70B inference. It walks from transformer structure (80 layers, 8,192 hidden dimensions, 64 query heads, 8 KV heads, 128 head dimensions) through attention and the KV-cache formula, which gives 320 kibibytes per token, so 1.3 gigabytes at 4K tokens, 43 gigabytes or 40.0 gibibytes at 128K, and 344 gigabytes at 1M, against a B200 with 192 gigabytes of HBM3e of which only 47 gigabytes is left for KV after 140 gigabytes of weights and 5 gigabytes of activations. That capacity arithmetic is analytic and stands unchanged: at 8K a context is 2.50 gibibytes, at 128K it is 40.0 gibibytes, and for a single sequence the KV only overtakes the weights at about 427,000 tokens. It then sets out the design: attention concentrates about 80 percent of its mass on the most recent 10 percent of tokens, RoPE makes that locality a geometric property, heads specialize, and a four-tier hierarchy of 10 gigabytes pinned HBM and 37 gigabytes evictable HBM at 100 nanoseconds, 1 terabyte of CXL DRAM and 16 terabytes of endpoint NVMe holds the rest. Tier latency and tier bandwidth are assumptions, not measurements: no CXL hardware was measured anywhere in this package, the 200 nanosecond access is demoted to an assumption inside a 32 to 121 gigabyte per second range, and across that range one 5.00 mebibyte block, being 16 tokens at 320 kibibytes per token, takes 164, 82 or 43 microseconds to move. Placement combines the signals into a priority score P of p equals 0.25 recency plus 0.55 EMA score plus 0.20 anchor zone, with tier boundaries at P equals 0.3 and P equals 0.6. The form of that score is kept, but its five constants are uncalibrated and illustrative: they were never fitted. Much of this appendix has been corrected. The 97 percent HBM hit rate is retracted, because it was never measured; the corrected simulation, Revision 2, gives 8.5 to 91.6 percent depending on HBM budget, policy and workload, and at a 32 gibibyte budget LRU gives 38.34, LFU 16.09, EMA alpha 0.15 gives 44.18 and EMA alpha 0.01 gives 35.91. The 72 to 80 to 87 to 93 to 97 ladder, with its plus 8, plus 7, plus 6 and plus 4 increments and its plus 25 point total, is retracted outright: no ablation was ever run, and the increments were assumed additive and independent. The real margin over LRU is plus 2.5 to plus 5.8 points at the specified alpha and plus 19.1 points at a tuned alpha of about 0.05, which gives 55.96 percent against LRU at 36.85; the alpha of 0.2 previously used here was never validated. A decayed-frequency score of this form is LRFU, published by Lee and colleagues at SIGMETRICS in 1999, and alpha is its documented knob between LRU and LFU behaviour, so the contribution is stated instead as a protocol claim: make KV state and its control metadata a single migratable object with identity-keyed policy state that survives the round trip between tiers. All simulated values are provisional rather than settled, because a single data-structure defect in Revision 1 inverted four of five findings. The effective access latency of 177 nanoseconds, the 11.5 times end-to-end figure and the 65 times per-access figure of 13.0 microseconds of PCIe DMA against 200 nanoseconds of CXL.mem are all retracted. What replaces them is that the tier avoids regeneration rather than lowering latency: moving a block costs 43 to 164 microseconds while regenerating the same 16 tokens costs 9 to 18 milliseconds of forward compute, which is two to three orders of magnitude, with no precise multiple claimed. The 16 times user capacity is retracted with no replacement claim, because it was downstream of the 97 percent and because under dense attention every block of an active sequence is read every step; concurrency is now stated only as KV-only capacity arithmetic, never as a served-user count. The 36 percent CapEx reduction from $160,000 to $103,000 is retracted because it priced a configuration justified by the 16 times, and no replacement cost figure is offered. Finally, real hardware data has been added. On a DGX Spark GB10 measured on 19 August 2026 with Qwen2.5-7B in fp16, achieved streaming bandwidth is 236.5 gigabytes per second, which is 87 percent of the 273 gigabyte per second spec and retires the roughly 50 percent effective-bandwidth haircut earlier drafts applied. Decode time is two-term rather than one-term: t equals 100.3 plus 16.34 times KV in gibibytes for HuggingFace with SDPA at R squared 0.991, and t equals 73.9 plus 17.58 times KV in gibibytes for vLLM with FlashAttention-2 at R squared 0.945, giving inferred fixed-path bandwidths of 151.5 and 205.7 gigabytes per second against inferred KV-path bandwidths of 65.7 and 61.1. Under that fit the KV path is about 3.4 times more costly per byte, a model-derived ratio rather than a direct byte-level hardware measurement, and kernel substitution alone did not remove the KV slope, which stayed within 8 percent across the two frameworks. The 6 times memory expansion, 320 kibibytes per token, the 5.00 mebibyte block and the capacity table all stand: they are arithmetic over stated capacities and never depended on any hit rate.

Technical Appendix

KV-Cache Offloading for LLM Inference — Visual Reference · v6.0, August 2026

Measured — DGX Spark GB10 Simulated (provisional) Analytical model External literature Illustrative
Read the labels, not the headlines. Capacity sizing is an analytical model over stated inputs and stands. Hit rates and policy margins are simulated (provisional) — Revision 2, 2–3 seeds, no hardware validation, and provisional rather than settled because a single data-structure defect in Revision 1 inverted four of five findings. Bandwidth and decode-model figures are measured on a DGX Spark (GB10), 19 Aug 2026. No CXL hardware was measured, so every tier latency and tier bandwidth here is an assumption inside a stated range. The 97% hit rate, the +25-point ladder, the 16× user capacity, the 65× access latency, the 177 ns effective latency and the 36% CapEx reduction are retracted — see RETRACTIONS and CANONICAL-NUMBERS §7.

1. Transformer Architecture

Llama-70B consists of 80 identical layers. Each layer performs attention followed by a feed-forward transformation.

Diagram 1.1 — Layer Structure
Input Embedding
× 80 LAYERS
Self-Attention
Feed-Forward
Output Logits
Architecture Parameters
Layers80
Hidden dim8,192
Query heads64
KV heads8
Head dim128
FFN dim28,672
Parameters70B

2. Attention Mechanism

Each token computes Query, Key, and Value vectors. Attention scores determine how much each previous token contributes to the output.

Diagram 2.1 — Q, K, V Projections
Input x
8,192 dims
W_Q
Query
W_K
Key
W_V
Value
Q
128d
K
128d
V
128d
What each vector represents:
Q — "What am I looking for?"
K — "What do I contain?"
V — "What do I contribute?"
Diagram 2.2 — Attention Score Computation
Token "France" attending to previous tokens:
Position 0
Qfrance Kthe
0.12
Position 1
Qfrance Kcapital
0.87
Position 2
Qfrance Kof
0.23
Position 3
Qfrance Kfrance
0.45
After softmax normalization:
The
6%
capital
52%
of
14%
France
28%
Output = weighted sum of V vectors

3. KV-Cache Structure

The KV-cache stores Key and Value vectors for all processed tokens, eliminating redundant computation during generation.

Diagram 3.1 — KV-Cache Organization
Layer 1
Kh0
Kh1
Kh2
Kh3
Kh4
Kh5
Kh6
Kh7
Vh0
Vh1
Vh2
Vh3
Vh4
Vh5
Vh6
Vh7
⋮ × 80 layers ⋮
Size = L × H_kv × seq_len × d_head × 2 × bytes = 80 × 8 × seq_len × 128 × 2 × 2 = 320 KiB per token (327,680 B)
Diagram 3.2 — KV-Cache Size Scaling
4K tokens
32K tokens
128K tokens
512K tokens
1M tokens
B200 HBM capacity: 192 GB — of which only 47 GB is left for KV after 140 GB of weights and 5 GB of activations

4. Prefill vs Decode

The two phases of inference have fundamentally different computational characteristics.

Diagram 4.1 — Phase Comparison
Prefill Phase
Processing the prompt
The
capital
of
France
is
✓ All tokens processed in parallel
✓ High arithmetic intensity
✓ Compute-bound
Decode Phase
Generating response
The
capital
...
Paris
✗ One token at a time
✗ Must read entire KV-cache
✗ Memory-bandwidth-bound
Diagram 4.2 — Decode Memory Access Pattern
To generate 1 token:
Model Weights
140 GB
KV-Cache
43 GB (at 128K)
Total reads
183 GB
Naive bandwidth requirement — the strawman:
(140 + 43) GB ÷ 50 ms = 3.66 TB/s
A single-bandwidth roofline over the whole read set is the wrong model — but not for the reason this box used to give. The 140 GB of weights are HBM-resident and never cross the tier; only blocks that are not resident have to be moved.
Retracted: the claim that "selective attention reads only the hot set, not the full 43 GB context", and the 11.4 GB/s steady-state CXL demand and 4.5% utilisation derived from it. That reasoning conflated a cache hit rate with attention sparsity — two different claims with different obligations. Under dense attention every block of an active sequence is read every decode step, so lossless residency requires all of it; sparsity is a lossy relaxation that carries an accuracy-evaluation burden. "Capacity-driven, not bandwidth-driven" is withdrawn too: a memory tier does not eliminate the KV memory wall, it changes its shape from a capacity wall into a data-movement wall.
Measured — DGX Spark GB10, 19 Aug 2026: decode is two-term, not one-term — t = 100.3 + 16.34·KV_GiB (HF + SDPA, R² 0.991) and t = 73.9 + 17.58·KV_GiB (vLLM + FA2, R² 0.945). Achieved streaming bandwidth is 236.5 GB/s = 87% of the 273 GB/s spec, which retires the ~50% effective-bandwidth haircut. Inferred fixed-path bandwidth is 151.5 / 205.7 GB/s against an inferred KV path of 65.7 / 61.1 GB/s — under that fit a KV byte costs ~3.4× a fixed-path byte, a model-derived ratio rather than a direct byte-level measurement. Kernel substitution alone did not remove the KV slope: within 8% across two frameworks. The measurement refutes the single-bandwidth roofline.

5. The Memory Wall

GPU memory capacity, not compute or bandwidth, becomes the limiting factor with multiple users.

Diagram 5.1 — Single User Memory Layout
B200 HBM — 192 GB 4 GB free
Weights — 140 GB
KV @128K — 43 GB
Activations — 5 GB
Free — 4 GB
✓ 140 + 43 + 5 = 188 GB — exactly one user at 128K fits, with 4 GB to spare
Diagram 5.2 — Multi-User Memory Explosion
2 users 231 GB needed
4 users 317 GB needed
8 users 489 GB needed
B200 capacity: 192 GB
Totals are 140 GB weights + n × 43 GB KV + 5 GB activations
8 users need: 489 GB (2.5× over)

6. Attention Locality

The model assumes attention concentrates heavily on recent tokens and on a small anchor zone — an assumption drawn from the published attention-sparsity literature (H2O, SnapKV, Scissorhands, Quest) [External literature], not from measurement of this system. The distributions below are illustrative.

Diagram 6.1 — Attention Distribution (10K Context)
0-1K
5%
1-9K
15%
9-10K
80%
Key insight:
~80%
of attention goes to
~10%
most recent tokens
Diagram 6.2 — Attention Heatmap (Simplified)
Current token attending to context
← Earlier | Recent →
Position 0 Position N
Low attention
Medium
High attention

7. RoPE: Why Locality Emerges

Rotary Position Embedding creates locality as a geometric property of how positions are encoded.

Diagram 7.1 — RoPE Rotation Concept
Each dimension pair rotates at different frequency
θ₀ = 1.0 θ₃₂ θ₆₃ = 0.0001
Position m rotates each pair by m × θáµ¢
Frequency formula:
θáµ¢ = 10000−2i/d θ₀ = 1.0 (fast) θ₃₂ = 0.01 (medium) θ₆₃ = 0.0001 (slow)
Fast dims → local patterns
Slow dims → global patterns
Diagram 7.2 — Distance-Dependent Decay
Illustrative decay envelope B(|m−n|) by distance:
Distance 1
Distance 10
Distance 100
Distance 1,000
Distance 10,000
⟨qm, kn⟩ = Σ over d/2 frequency pairs |⟨qm, kn⟩| ≤ B(|m − n|) Small distance → the bound B is loose → high attention is possible Large distance → the bound B tightens toward zero
The RoPE inner product is a sum over d/2 frequency pairs; the distance dependence is an upper bound, written |⟨qm, kn⟩| ≤ B(|m − n|). It is not a proportionality to cos((m − n)θ) — individual pairs oscillate and can be large at any distance.

8. Attention Head Types

Different attention heads specialize for different functions, creating varied access patterns. The 40 / 15 / 25 / 20 split below is an assumption of this model, informed by the induction-head and retrieval-head literature — it is not a measured distribution.

Diagram 8.1 — Head Specialization
Recency Heads
~40% of heads
Focus on last 50-200 tokens
Anchor Heads
~15% of heads
Always check position 0-100
Retrieval Heads
~25% of heads
Content-based, position-independent
Syntactic Heads
~20% of heads
Follow grammatical dependencies
Implication: A single caching policy cannot satisfy all heads. Per-head tracking required.

9. CXL Architecture

CXL provides memory expansion at lower cost and bandwidth than HBM.

Diagram 9.1 — System Topology
NVIDIA B200
192 GB
HBM Capacity
8 TB/s
Bandwidth
100ns
Latency
CXL 3.0 × 16 — 64 GB/s
CXL Switch / Fabric
EP 0
256 GB
EP 1
256 GB
EP 2
256 GB
EP 3
256 GB
Total CXL: 1 TB · tier bandwidth assumed 32–121 GB/s — unmeasured
Diagram 9.2 — HBM vs CXL Comparison
GPU HBM CXL DRAM Ratio
Bandwidth 8 TB/s 32–121 GB/s assumed 66–250× less
Latency 100 ns 200 ns assumed assumption — no CXL hardware measured
Capacity 192 GB 1 TB 5.3× more
Cost per GB ~$50 ~$5 10× less
CXL tradeoff: cheaper per GB, far lower bandwidth. Per-GB media prices are indicative list prices, not a cost model. The 97% HBM hit rate that used to be called the load-bearing number here is retracted — it was never measured, and there is no single hit rate: the corrected simulation gives 8.5–91.6% depending on HBM budget, policy and workload. What is load-bearing instead is that a tier avoids regeneration: moving one 5.00 MiB block costs 164 / 82 / 43 µs at an assumed 32 / 64 / 121 GB/s, against 9–18 ms to regenerate the same 16 tokens by forward compute — two to three orders of magnitude, with no precise multiple claimed. Combined capacity is 192 GB + 1,024 GB = 1.2 TB, a 6.3× expansion over HBM alone; that one stands, because it is arithmetic over stated capacities and never depended on a hit rate.
Measured — DGX Spark GB10, 19 Aug 2026, Qwen2.5-7B fp16
Achieved streaming bandwidth is 236.5 GB/s = 87% of the 273 GB/s spec, which retires the ~50% "effective bandwidth" haircut earlier drafts applied to spec figures — the haircut was wrong, and wrong in the pessimistic direction. Decode time is two-term, not one-term: t = 100.3 + 16.34·KV_GiB (HF + SDPA, R² 0.991) and t = 73.9 + 17.58·KV_GiB (vLLM + FA2, R² 0.945), giving an inferred fixed-path bandwidth of 151.5 / 205.7 GB/s against an inferred KV-path bandwidth of 65.7 / 61.1 GB/s. Under that fit the KV path is ~3.4× more costly per byte — a model-derived ratio, not a direct byte-level hardware measurement. Kernel substitution alone did not remove the KV slope: it stayed within 8% across two frameworks with different attention implementations and paged vs unpaged layout, while the inferred fixed-path bandwidth improved 34% between them.

10. Tiered Memory Hierarchy

The caching system places data in tiers based on access patterns.

Diagram 10.1 — Four-Tier Architecture
Tier 0 — HBM Pinned
Anchor tokens, per-head metadata (640 MiB), staging
10 GB
100 ns
Tier 1 — HBM Evictable
Evictable KV working set — sized as capacity, not as a user count
37 GB
100 ns
Tier 2 — CXL DRAM (4 endpoints)
Warm KV — 256 GB per endpoint
1 TB
200 ns (assumed)
Tier 3 — Endpoint NVMe
Cold KV and complete backing store — 4 TB per endpoint
16 TB
25 µs
192 GB HBM = 140 weights + 5 activations + 10 pinned + 37 evictable — arithmetic over stated capacities, which stands. Tier latencies are assumptions: the 200 ns CXL access is demoted to an unmeasured assumption inside a 32–121 GB/s range, and no CXL hardware was measured. The blended-latency model that used to be built on these numbers (L_eff = h_HBM × 100 ns + h_CXL × 200 ns + h_NVMe × 25 µs → 177 ns) is retracted, because there is no single hit rate to substitute into it.
Diagram 10.2 — Memory Layout, as capacity arithmetic
HBM — 192 GB
Weights — 140 GB
Activations — 5 GB
Tier 0 pinned — 10 GB
Tier 1 evictable KV — 37 GB
CXL — 1 TB
687 GB is 67% of the 1,024 GB CXL tier, and at 43 GB (40.0 GiB) per 128K context that is about 16 contexts' worth of KV. That is capacity arithmetic, not a served-user count — and the difference is exactly what was wrong before. The 16× user-capacity headline is retracted: it was downstream of a 97% HBM hit rate that was never measured, and under dense attention every block of an active sequence is read every decode step, so there is no small resident subset to serve many users from. What survives is the arithmetic itself, which never depended on a hit rate: 320 KiB/token, 2.50 GiB at 8K, 40.0 GiB at 128K, and a single-sequence crossover against the 130.4 GiB of weights at ≈ 427,000 tokens. Concurrency is stated as KV-only capacity, never as users served.

11. EMA Scoring Algorithm

Exponential Moving Average tracks which tokens actually receive attention over time.

Diagram 11.1 — EMA Update Rule
scoret = α · attentiont + (1 − α) · scoret−1
α ≈ 0.05
Decay factor — simulated optimum
13.5 steps
Half-life, ln0.5 ÷ ln0.95
~1.35 s
At the ≥10 tok/s per-user target
α = 0.2 was never validated — and the score itself is prior art
α = 0.2 is withdrawn. It was a stated implementation default that no experiment ever supported. In the corrected simulation α is non-monotonic with an interior optimum, and there is no convergence onto LFU as α → 0. At a 32 GiB HBM budget: α = 0.5 gives 37.20%, 0.3 gives 38.84%, 0.15 gives 44.33%, 0.05 gives 55.96% — the optimum, +19.1 pts over LRU's 36.85%, then 0.01 gives 31.54%, 0.003 gives 25.47% and 0.001 gives 24.44%, against LFU at 15.56%. At the specified α the margin over LRU is only +2.5 to +5.8 pts, not the +25 pts previously claimed.

And a decayed-frequency score of this form is not new: it is LRFU (Lee et al., SIGMETRICS 1999), established prior art, and α is its documented LRU↔LFU knob. The contribution therefore has to be stated one level up, as a protocol claim: make KV state and its control metadata a single migratable object, with identity-keyed policy state that survives the round trip between tiers. In simulation that persistence is worth more than the choice of policy — at 32 GiB it is +34.15 pts for LFU and +24.85 pts for EMA α = 0.01 — and it is testable with no CXL hardware.

Simulated (provisional): Revision 2, 2–3 seeds, no Belady bound, no bandwidth or queueing model, no hardware validation. Provisional rather than settled because a single data-structure defect in Revision 1 inverted four of five findings.
Diagram 11.2 — EMA Evolution Example
System Instruction Token
Position 50 — "helpful"
Consistent attention from anchor heads:
Step 0: attn=0.04 → score=0.008
Step 1: attn=0.03 → score=0.012
Step 2: attn=0.05 → score=0.020
...
Step 100: → score=0.040
→ Stays HOT
Generic Middle Token
Position 45,000 — "the"
Rarely attended:
Step 0: attn=0.001 → score=0.0002
Step 1: attn=0.000 → score=0.0002
Step 2: attn=0.002 → score=0.0005
...
Step 100: → score=0.001
→ Evict to CXL

12. Priority Scoring Formula

Final placement decisions combine multiple signals into a single priority score.

Diagram 12.1 — Scoring Components
Recency
25%
EMA Score
55%
Anchor Zone
20%
P(p) = 0.25 · R(p) + 0.55 · E(p) + 0.20 · N(p)  [Uncalibrated and illustrative — these weights were never fitted; the prefetch priority used in Appendix J is 1/(1 + distance/100)]
0.25 / 0.55 / 0.20 are uncalibrated and illustrative — they were never fitted

The form of the score is kept; the weights are not a result. These three numbers were not fitted, measured or derived from anything. They encode a judgement — that smoothed attention mass should dominate, with recency and anchor membership as corrections — and they are stated as an assumption so a reader can disagree with the judgement rather than with an implied result. Nothing in this package depends on their particular values.

What they control. P(p) is a tier-placement score: it decides which tier an entry that already exists should live in, against the thresholds in Diagram 12.2 below. R(p) is recency, E(p) the EMA attention score, N(p) anchor-zone membership. The weights set how much each signal moves an entry between Tier 0 pinned, Tier 1 HBM and Tier 2 CXL.

Which way they push. Raising the E weight makes placement track sustained attention mass more closely and behave more like pure EMA eviction — better for stable workloads, slower to react to a topic change. Raising the R weight makes placement more reactive: recently touched tokens climb tiers quickly, which helps a sharp context switch and hurts long-lived structural tokens. Raising the N weight hardens the anchor zone at the cost of pinned capacity that could hold the working set. (The +8-point anchor-pinning contribution this paragraph used to cite is retracted — it was a rung of the 72 → 80 → 87 → 93 → 97 ladder, and no ablation was ever run.) Because the weights sum to 1, every increase is another signal's decrease.

How they would be calibrated. Fit them, do not choose them. Log per-token (recency, EMA, anchor-membership, was-attended) tuples from real 128 K attention traces and solve for the weights that maximise hit rate at the fixed 47 GB HBM budget — a small constrained regression or a coarse simplex sweep. That work has not been done. The tier thresholds in Diagram 12.2 must be re-fitted at the same time, since they are thresholds on P and are meaningless if the weights move. See Appendix K, section K.7 — Uncalibrated parameters.

Not the same function as the prefetch blend. P(p) = 0.25·R + 0.55·E + 0.20·N answers "where should this entry live?". The 0.6 · rope_factor + 0.4 · ema_factor blend in Appendices D and J answers a different question — "what should be fetched next?" — over a different signal set, using RoPE distance rather than recency or anchor membership. The two weight vectors are not alternatives and do not contradict each other; neither is calibrated.

Diagram 12.2 — Tier Assignment Thresholds
Tier 2 (CXL)
Tier 1 (HBM)
Tier 0 (Pinned)
P = 0 P = 0.3 P = 0.6 P = 1.0
P = 0.3 and P = 0.6 are uncalibrated design parameters

The two tier boundaries are cut points on the priority score above, and like the weights that produce that score they are stated values rather than fitted ones. They control how the 192 GB HBM budget is split: the 0.6 boundary sets how much of the 10 GB pinned Tier 0 is claimed, and the 0.3 boundary sets how much of the 37 GB evictable Tier 1 is held before an entry falls back to CXL DRAM. Raising either boundary frees HBM sooner and admits more concurrent users at the cost of hit rate; lowering either holds more in HBM and raises hit rate at the cost of user capacity. They cannot be tuned independently of the 0.25 / 0.55 / 0.20 weights, because they are thresholds on the score those weights define — a change to either requires re-fitting both. Calibration is the same exercise: sweep against real attention traces at the canonical 128 K context and pick the split that maximises hit rate within the fixed HBM budget. There is no 97% hit-rate SLO to target — that figure is retracted, and the corrected simulation gives 8.5–91.6% depending on budget, policy and workload. See Appendix K, section K.7 — Uncalibrated parameters.

13. Per-Head Tracking

Scores are maintained separately for each KV-head to handle head specialization.

Diagram 13.1 — Per-Head Score Matrix
Position Head 0
(recency)		 Head 1
(anchor)		 Head 2
(retrieval)		 Head 3-7 Aggregate Decision
0 (system) 0.001 0.089 0.012 ... 0.089 HBM
45,000 0.000 0.002 0.003 ... 0.003 CXL
99,950 0.082 0.004 0.031 ... 0.082 HBM
Paggregate(p) = max( Ph(p) ) for all heads h
Position stays in HBM if ANY head needs it. Evict only when NO head has recent access.

14. Prefetching Strategy

Predictive prefetch loads anticipated tokens from CXL before they're needed.

Diagram 14.1 — Prefetch Targets
Anchor (0-100)
Recent (m-200 to m)
High-EMA
1. Anchor zone [0, 100]
2. Recent [m−200, m−1]
3. High-EMA positions
Diagram 14.2 — Prefetch Timing Budget
Token Generation
50 ms
Compute
20 ms
Prefetch Window
30 ms
Prefetch capacity across the assumed 32–121 GB/s tier range:
0.96–3.63 GB ≈ 2,900–11,100 positions
30 ms × 32 GB/s = 0.96 GB and 30 ms × 121 GB/s = 3.63 GB; ÷ 320 KiB per position gives 2,929 and 11,078. Tier bandwidth is an assumption — no CXL hardware was measured, and the 256 GB/s aggregate this panel used to assume is outside the canonical range. The claim that "steady-state demand is only 11.4 GB/s, so this window is never the constraint" is retracted: it was derived from the withdrawn hot-set/sparsity conflation. Whether the window is a constraint is open, and tier-ingress bandwidth is one of the two constraints a tier creates.

15. Hit Rate Depends on Budget, Policy and Workload

There is no single hit rate. The ladder this section used to show is retracted.

Diagram 15.1 — Hit rate by policy at a 32 GiB HBM budget
EMA α = 0.15
LRU baseline
EMA α = 0.01
LFU
Margins are percentage points, never percentages. At this budget EMA α = 0.15 leads by +5.8 pts over LRU; across the budget sweep the margin at the specified α is +2.5 to +5.8 pts, and at a tuned α ≈ 0.05 it is +19.1 pts (55.96% against LRU's 36.85%). Those two figures come from different experiments with different LRU baselines and must never be mixed. LFU collapses once decode growth is modelled — classical LFU aging, with fresh count-1 blocks evicted in favour of stale high-count blocks.
Simulated hit rate against HBM budget for four eviction policies, Revision 2, provisional.
HBM budget LRU LFU EMA α = 0.15 EMA α = 0.01
8 GiB8.52%8.76%12.35%12.12%
16 GiB17.87%12.70%22.58%18.70%
32 GiB38.34%16.09%44.18%35.91%
64 GiB67.12%23.00%69.65%66.83%
128 GiB88.01%32.70%88.14%91.59%
The full observed span is 8.5–91.6%. Workload moves it as much as policy does: at 32 GiB the cyclic-sweep workload leaves LRU and both EMA variants at 9.83% while only LFU survives at 22.06%, and on a one-shot scan flood EMA α = 0.01 leads at 36.26% against LRU's 31.51%. No policy dominates across all three workloads.
The 72 → 80 → 87 → 93 → 97 ladder is retracted outright
This diagram used to show a five-rung ladder — LRU 72%, +8 pts for anchor pinning, +7 pts for EMA scoring at α = 0.2, +6 pts for per-head tracking, +4 pts for RoPE-aware prefetch, reaching 97% and a +25-point total. No ablation was ever run. The increments were assumed additive and independent, and the mechanisms they decompose overlap. The 97% endpoint was never measured, and per-technique ablation remains unrun. Everything downstream of the 97% falls with it: the 16× user capacity, the 11.5× effective end-to-end latency, the 177 ns effective access latency and the 36% CapEx reduction.

Simulated (provisional): Revision 2 of kv_tiering_sim_v2.py, Llama-3 70B GQA, 16-token blocks, 200 sessions, Zipf(1.1) return frequency, 421 GiB footprint, 2–3 seeds. Provisional rather than settled because a single data-structure defect in Revision 1 inverted four of five findings.
Diagram 15.2 — What the tier buys: avoiding regeneration
tfetch = 5.00 MiB ÷ BWtier   ≪   tregen = forward compute, 16 tokens
Move one 5.00 MiB block:
43–164 µs
43 µs @ 121 GB/s · 82 µs @ 64 GB/s
164 µs @ 32 GB/s — all assumed
Regenerate the same 16 tokens:
9–18 ms
9 / 11 / 18 ms at 50% / 40% / 25% MFU
an interior block costs at least this
The claim:
two to three orders
of magnitude
no precise multiple is claimed
Moving a block is two to three orders of magnitude cheaper than regenerating it — and that is the whole of the claim. The two quantities do not measure the same operation, so the order of magnitude is claimed and never a precise multiple; the often-quoted 218× is one point in the range (17.9 ms ÷ 82 µs) and must always be quoted with its inputs. Regenerating an arbitrary interior block is worse still, because interior KV is not independently reproducible: hidden states depend recursively on all preceding context.
The effective-latency panel is retracted
This diagram used to compute L_eff = h_HBM × 100 ns + h_CXL × 200 ns + h_NVMe × 25 µs at a 97 / 2.7 / 0.3 split to 177 ns, report 11.5× against a 2,035 ns baseline, and set both beside a 65× single-access figure (13.0 µs PCIe DMA ÷ 200 ns CXL.mem). All three are withdrawn. The 97% hit rate they rest on was never measured. No CXL hardware was measured, so the 200 ns access is demoted to an assumption inside a 32–121 GB/s range and the 65× ratio — a hypothetical load/store path divided by a driver-mediated DMA path — is not claimed at all. The path comparison itself is kept in Chapter 4, with both sides labelled modelled and no ratio asserted.

16. Final Results

Diagram 16.1 — System Comparison
Baseline — 3× B200 + host-DRAM offload
Memory 576 GB HBM
KV capacity @ 128K (40.0 GiB each) ≈ 13 contexts
CapEx withdrawn
Hardware 3× B200 + 2 TB DDR5
Proposed — 1× B200 + CXL tiering
Memory 1.2 TB
KV capacity @ 128K (40.0 GiB each) ≈ 27 contexts
CapEx withdrawn
Hardware 1× B200 + switch + 4 endpoints
1.2 TB = 192 GB HBM + 1,024 GB CXL DRAM, a 6.3× expansion that stands on arithmetic alone. The capacity rows are KV-only capacity arithmetic — 576 GB ÷ 43 GB ≈ 13 and 1,216 GB ÷ 43 GB ≈ 27 contexts' worth of KV at 128K — and are not a served-user count: whether a machine can serve that many concurrent users depends on bandwidth and scheduling, which this arithmetic says nothing about.
The cost comparison is retracted, and not replaced
This diagram used to claim both configurations serve 16 users at 128K and ≥10 tok/s, and priced them at $160,000 against $103,000, a 36% CapEx reduction, with a 37% three-year TCO reduction alongside. All of it is withdrawn. The pricing was built on top of a served-user count that was itself downstream of a 97% hit rate that was never measured, so it priced a configuration that the evidence does not justify. No replacement cost figure is offered, and none should be invented until there is a measured basis for one.

Capacity was in any case not the binding constraint in any configuration actually measured: on the GB10 everything fit in 128 GiB and throughput still fell from 150 tok/s at batch 16 × 4K to 46 tok/s at batch 16 × 16K, with memory to spare and nothing evicted. A memory tier does not eliminate the KV memory wall — it changes its shape, from a capacity wall into a data-movement wall.
Diagram 16.2 — Key Metrics
6×
Memory Expansion
320 KiB
Per token, Llama-3 70B GQA
40.0 GiB
KV at 128K (2.50 GiB at 8K)
8.5–91.6%
HBM hit rate — simulated, provisional
236.5 GB/s
Measured — 87% of the 273 GB/s spec
~3.4×
KV byte vs fixed-path byte, model-derived
≈427,000
Tokens — single-sequence crossover with weights
Slug Architecture Research — v6.0, August 2026
Evidence labels — Measured (DGX Spark GB10, 19 Aug 2026): achieved streaming bandwidth and the two-term decode fit only. Simulated (provisional): every hit rate and policy margin — Revision 2, 2–3 seeds, no Belady bound, no bandwidth or queueing model, no hardware validation, provisional rather than settled because a single data-structure defect in Revision 1 inverted four of five findings. Analytical model: capacity sizing, tier-fetch costs and path budgets, all over stated inputs. External literature: LRFU (Lee et al., SIGMETRICS 1999) and the cited prior art. Illustrative: the priority weights, tier thresholds and worked examples — uncalibrated, never fitted. No CXL hardware was measured. The 97% hit rate, the +25-point ladder, the 16× user capacity, the 65× access latency, the 177 ns effective latency, the 11.5× end-to-end figure and the 36% CapEx reduction are all retracted — see RETRACTIONS.