SUPERSEDED — this figure illustrates a claim withdrawn in v5.0. It is retained only so the retraction remains checkable. Do not cite it. See RETRACTIONS.md and Appendix L (hardware measurements) and Appendix N (simulation) for what replaced it.

Figure: a sixteen-part visual technical appendix on KV-cache offloading for Llama-70B inference. It walks from transformer structure (80 layers, 8,192 hidden dimensions, 64 query heads, 8 KV heads, 128 head dimensions) through attention and the KV-cache formula, which gives 320 KiB per token — 1.3 GB at 4K tokens, 43 GB at 128K and 344 GB at 1M — against a B200 with 192 GB of HBM3e of which only 47 GB is left for KV after 140 GB of weights and 5 GB of activations, so one user at 128K fits and a second does not. It then sets out the design: attention concentrates about 80 percent of its mass on the most recent 10 percent of tokens, RoPE makes that locality a geometric property, heads specialize (about 40 percent recency, 15 percent anchor, 25 percent retrieval, 20 percent syntactic), and a four-tier hierarchy of 10 GB pinned HBM and 37 GB evictable HBM at 100 nanoseconds, 1 TB of CXL DRAM at 200 nanoseconds and 16 TB of endpoint NVMe at 25 microseconds holds the rest. Placement combines those signals into a priority score P(p) = 0.25 recency + 0.55 EMA score + 0.20 anchor zone, with tier boundaries at P = 0.3 and P = 0.6; those five constants are uncalibrated design parameters, not fitted or measured values, and Appendix K section K.7 lists them with the package's other uncalibrated parameters. Raising the EMA weight tracks sustained attention more closely and reacts more slowly to topic change; raising the recency weight does the reverse; raising the anchor weight hardens the anchor zone at the cost of pinned capacity. Raising either tier boundary frees HBM sooner and admits more users at the cost of hit rate. This tier-placement score is a different function from the prefetch priority blend of 0.6 RoPE factor plus 0.4 EMA factor used in Appendices D and J: one decides where an entry should live, the other decides what to fetch next. EMA scoring at alpha = 0.2 with anchor pinning, per-head tracking and RoPE-aware prefetch lifts the modeled HBM hit rate from a 72 percent LRU baseline to 97 percent (+8, +7, +6 and +4 percentage points, +25 in total), which yields an effective access latency of 0.97 × 100 + 0.027 × 200 + 0.003 × 25,000 = 177.4 nanoseconds — 11.5× lower than the 2,035 nanosecond baseline end to end, and distinct from the 65× per-access figure of 13.0 microseconds of PCIe DMA against 200 nanoseconds of CXL.mem. The result claimed is 16 users at 128K on one B200 instead of one, 1.2 TB of addressable memory instead of 192 GB, and CapEx of $103,000 against $160,000, a 36 percent reduction. All values are an analytical model, not measured.

Technical Appendix

KV-Cache Offloading for LLM Inference — Visual Reference · v4.0, August 2026

Evidence label — Analytical model. Every hit rate, latency, capacity, bandwidth and cost figure in this appendix is analytically modeled from vendor specifications and published algorithm behaviour. Nothing here is measured; no hardware benchmark was run.

1. Transformer Architecture

Llama-70B consists of 80 identical layers. Each layer performs attention followed by a feed-forward transformation.

Diagram 1.1 — Layer Structure
Input Embedding
× 80 LAYERS
Self-Attention
Feed-Forward
Output Logits
Architecture Parameters
Layers80
Hidden dim8,192
Query heads64
KV heads8
Head dim128
FFN dim28,672
Parameters70B

2. Attention Mechanism

Each token computes Query, Key, and Value vectors. Attention scores determine how much each previous token contributes to the output.

Diagram 2.1 — Q, K, V Projections
Input x
8,192 dims
W_Q
Query
W_K
Key
W_V
Value
Q
128d
K
128d
V
128d
What each vector represents:
Q — "What am I looking for?"
K — "What do I contain?"
V — "What do I contribute?"
Diagram 2.2 — Attention Score Computation
Token "France" attending to previous tokens:
Position 0
Qfrance Kthe
0.12
Position 1
Qfrance Kcapital
0.87
Position 2
Qfrance Kof
0.23
Position 3
Qfrance Kfrance
0.45
After softmax normalization:
The
6%
capital
52%
of
14%
France
28%
Output = weighted sum of V vectors

3. KV-Cache Structure

The KV-cache stores Key and Value vectors for all processed tokens, eliminating redundant computation during generation.

Diagram 3.1 — KV-Cache Organization
Layer 1
Kh0
Kh1
Kh2
Kh3
Kh4
Kh5
Kh6
Kh7
Vh0
Vh1
Vh2
Vh3
Vh4
Vh5
Vh6
Vh7
⋮ × 80 layers ⋮
Size = L × H_kv × seq_len × d_head × 2 × bytes = 80 × 8 × seq_len × 128 × 2 × 2 = 320 KiB per token (327,680 B)
Diagram 3.2 — KV-Cache Size Scaling
4K tokens
32K tokens
128K tokens
512K tokens
1M tokens
B200 HBM capacity: 192 GB — of which only 47 GB is left for KV after 140 GB of weights and 5 GB of activations

4. Prefill vs Decode

The two phases of inference have fundamentally different computational characteristics.

Diagram 4.1 — Phase Comparison
Prefill Phase
Processing the prompt
The
capital
of
France
is
✓ All tokens processed in parallel
✓ High arithmetic intensity
✓ Compute-bound
Decode Phase
Generating response
The
capital
...
Paris
✗ One token at a time
✗ Must read entire KV-cache
✗ Memory-bandwidth-bound
Diagram 4.2 — Decode Memory Access Pattern
To generate 1 token:
Model Weights
140 GB
KV-Cache
43 GB (at 128K)
Total reads
183 GB
Naive bandwidth requirement — the strawman:
(140 + 43) GB ÷ 50 ms = 3.66 TB/s
This figure is wrong for this architecture and is shown only because it is the objection the design refutes. The 140 GB of weights are HBM-resident and never cross CXL; selective attention reads only the hot set, not the full 43 GB context; and only the miss tail leaves HBM.
Corrected steady state: 11.4 GB/s of CXL demand against 256 GB/s supplied — 4.5% utilized. The fabric is capacity-driven, not bandwidth-driven.

5. The Memory Wall

GPU memory capacity, not compute or bandwidth, becomes the limiting factor with multiple users.

Diagram 5.1 — Single User Memory Layout
B200 HBM — 192 GB 4 GB free
Weights — 140 GB
KV @128K — 43 GB
Activations — 5 GB
Free — 4 GB
✓ 140 + 43 + 5 = 188 GB — exactly one user at 128K fits, with 4 GB to spare
Diagram 5.2 — Multi-User Memory Explosion
2 users 231 GB needed
4 users 317 GB needed
8 users 489 GB needed
B200 capacity: 192 GB
Totals are 140 GB weights + n × 43 GB KV + 5 GB activations
8 users need: 489 GB (2.5× over)

6. Attention Locality

The model assumes attention concentrates heavily on recent tokens and on a small anchor zone — an assumption drawn from the published attention-sparsity literature (H2O, SnapKV, Scissorhands, Quest) [External literature], not from measurement of this system. The distributions below are illustrative.

Diagram 6.1 — Attention Distribution (10K Context)
0-1K
5%
1-9K
15%
9-10K
80%
Key insight:
~80%
of attention goes to
~10%
most recent tokens
Diagram 6.2 — Attention Heatmap (Simplified)
Current token attending to context
← Earlier | Recent →
Position 0 Position N
Low attention
Medium
High attention

7. RoPE: Why Locality Emerges

Rotary Position Embedding creates locality as a geometric property of how positions are encoded.

Diagram 7.1 — RoPE Rotation Concept
Each dimension pair rotates at different frequency
θ₀ = 1.0 θ₃₂ θ₆₃ = 0.0001
Position m rotates each pair by m × θáµ¢
Frequency formula:
θáµ¢ = 10000−2i/d θ₀ = 1.0 (fast) θ₃₂ = 0.01 (medium) θ₆₃ = 0.0001 (slow)
Fast dims → local patterns
Slow dims → global patterns
Diagram 7.2 — Distance-Dependent Decay
Illustrative decay envelope B(|m−n|) by distance:
Distance 1
Distance 10
Distance 100
Distance 1,000
Distance 10,000
⟨qm, kn⟩ = Σ over d/2 frequency pairs |⟨qm, kn⟩| ≤ B(|m − n|) Small distance → the bound B is loose → high attention is possible Large distance → the bound B tightens toward zero
The RoPE inner product is a sum over d/2 frequency pairs; the distance dependence is an upper bound, written |⟨qm, kn⟩| ≤ B(|m − n|). It is not a proportionality to cos((m − n)θ) — individual pairs oscillate and can be large at any distance.

8. Attention Head Types

Different attention heads specialize for different functions, creating varied access patterns. The 40 / 15 / 25 / 20 split below is an assumption of this model, informed by the induction-head and retrieval-head literature — it is not a measured distribution.

Diagram 8.1 — Head Specialization
Recency Heads
~40% of heads
Focus on last 50-200 tokens
Anchor Heads
~15% of heads
Always check position 0-100
Retrieval Heads
~25% of heads
Content-based, position-independent
Syntactic Heads
~20% of heads
Follow grammatical dependencies
Implication: A single caching policy cannot satisfy all heads. Per-head tracking required.

9. CXL Architecture

CXL provides memory expansion at lower cost and bandwidth than HBM.

Diagram 9.1 — System Topology
NVIDIA B200
192 GB
HBM Capacity
8 TB/s
Bandwidth
100ns
Latency
CXL 3.0 × 16 — 64 GB/s
CXL Switch / Fabric
EP 0
256 GB
EP 1
256 GB
EP 2
256 GB
EP 3
256 GB
Total CXL: 1 TB @ 256 GB/s aggregate
Diagram 9.2 — HBM vs CXL Comparison
GPU HBM CXL DRAM Ratio
Bandwidth 8 TB/s 256 GB/s 31× less
Latency 100 ns 200 ns 2× more
Capacity 192 GB 1 TB 5.3× more
Cost per GB ~$50 ~$5 10× less
CXL tradeoff: 10× cheaper per GB, but 31× lower bandwidth. Viable only if most accesses hit HBM — which is why the 97% HBM hit rate is the load-bearing number. Combined capacity is 192 GB + 1,024 GB = 1.2 TB, a 6.3× expansion over HBM alone.

10. Tiered Memory Hierarchy

The caching system places data in tiers based on access patterns.

Diagram 10.1 — Four-Tier Architecture
Tier 0 — HBM Pinned
Anchor tokens, per-head metadata (640 MiB), staging
10 GB
100 ns
Tier 1 — HBM Evictable
Hot KV working set — 2.31 GB per user across 16 users
37 GB
100 ns
Tier 2 — CXL DRAM (4 endpoints)
Warm KV — 256 GB per endpoint
1 TB
200 ns
Tier 3 — Endpoint NVMe
Cold KV and complete backing store — 4 TB per endpoint
16 TB
25 µs
192 GB HBM = 140 weights + 5 activations + 10 pinned + 37 evictable. Because Tier 3 is a complete backing store, no KV entry is ever lost, so the latency model carries no recompute term.
Diagram 10.2 — Memory Layout (16 Users × 128K)
HBM — 192 GB
Weights — 140 GB
Activations — 5 GB
Tier 0 pinned — 10 GB
Tier 1 hot KV — 37 GB (2.31 GB × 16 users)
CXL — 1 TB
16 users × 43 GB = 687 GB, 67% of the 1 TB CXL tier. Capacity is not the binding constraint — the 37 GB HBM hot set is. At 17+ users the per-user hot set falls below 2.31 GB and the modeled hit rate drops under the 97% SLO, which is what fixes user capacity at 16.

11. EMA Scoring Algorithm

Exponential Moving Average tracks which tokens actually receive attention over time.

Diagram 11.1 — EMA Update Rule
scoret = α · attentiont + (1 − α) · scoret−1
α = 0.2
Decay factor
3.1 steps
Half-life
~310 ms
At the ≥10 tok/s per-user target
Diagram 11.2 — EMA Evolution Example
System Instruction Token
Position 50 — "helpful"
Consistent attention from anchor heads:
Step 0: attn=0.04 → score=0.008
Step 1: attn=0.03 → score=0.012
Step 2: attn=0.05 → score=0.020
...
Step 100: → score=0.040
→ Stays HOT
Generic Middle Token
Position 45,000 — "the"
Rarely attended:
Step 0: attn=0.001 → score=0.0002
Step 1: attn=0.000 → score=0.0002
Step 2: attn=0.002 → score=0.0005
...
Step 100: → score=0.001
→ Evict to CXL

12. Priority Scoring Formula

Final placement decisions combine multiple signals into a single priority score.

Diagram 12.1 — Scoring Components
Recency
25%
EMA Score
55%
Anchor Zone
20%
P(p) = 0.25 · R(p) + 0.55 · E(p) + 0.20 · N(p)  [Illustrative weighting — the tier-placement weights are not calibrated; the prefetch priority used in Appendix J is 1/(1 + distance/100)]
0.25 / 0.55 / 0.20 are uncalibrated design parameters

These three weights were not fitted, measured or derived. They encode a judgement — that smoothed attention mass should dominate, with recency and anchor membership as corrections — and they are stated as an assumption so a reader can disagree with the judgement rather than with an implied result.

What they control. P(p) is a tier-placement score: it decides which tier an entry that already exists should live in, against the thresholds in Diagram 12.2 below. R(p) is recency, E(p) the EMA attention score, N(p) anchor-zone membership. The weights set how much each signal moves an entry between Tier 0 pinned, Tier 1 HBM and Tier 2 CXL.

Which way they push. Raising the E weight makes placement track sustained attention mass more closely and behave more like pure EMA eviction — better for stable workloads, slower to react to a topic change. Raising the R weight makes placement more reactive: recently touched tokens climb tiers quickly, which helps a sharp context switch and hurts long-lived structural tokens. Raising the N weight hardens the anchor zone: the +8-point anchor-pinning contribution becomes more robust, but pinned capacity that could serve the hot working set is spent on tokens that may not be attended to. Because the weights sum to 1, every increase is another signal's decrease.

How they would be calibrated. Fit them, do not choose them. Log per-token (recency, EMA, anchor-membership, was-attended) tuples from real 128 K attention traces and solve for the weights that maximise hit rate at the fixed 47 GB HBM budget — a small constrained regression or a coarse simplex sweep. The tier thresholds in Diagram 12.2 must be re-fitted at the same time, since they are thresholds on P and are meaningless if the weights move. See Appendix K, section K.7 — Uncalibrated parameters.

Not the same function as the prefetch blend. P(p) = 0.25·R + 0.55·E + 0.20·N answers "where should this entry live?". The 0.6 · rope_factor + 0.4 · ema_factor blend in Appendices D and J answers a different question — "what should be fetched next?" — over a different signal set, using RoPE distance rather than recency or anchor membership. The two weight vectors are not alternatives and do not contradict each other; neither is calibrated.

Diagram 12.2 — Tier Assignment Thresholds
Tier 2 (CXL)
Tier 1 (HBM)
Tier 0 (Pinned)
P = 0 P = 0.3 P = 0.6 P = 1.0
P = 0.3 and P = 0.6 are uncalibrated design parameters

The two tier boundaries are cut points on the priority score above, and like the weights that produce that score they are stated values rather than fitted ones. They control how the 192 GB HBM budget is split: the 0.6 boundary sets how much of the 10 GB pinned Tier 0 is claimed, and the 0.3 boundary sets how much of the 37 GB evictable Tier 1 is held before an entry falls back to CXL DRAM. Raising either boundary frees HBM sooner and admits more concurrent users at the cost of hit rate; lowering either holds more in HBM and raises hit rate at the cost of user capacity. They cannot be tuned independently of the 0.25 / 0.55 / 0.20 weights, because they are thresholds on the score those weights define — a change to either requires re-fitting both. Calibration is the same exercise: sweep against real attention traces at the canonical 128 K context and pick the split that meets the 97% hit-rate SLO within the fixed HBM budget. See Appendix K, section K.7 — Uncalibrated parameters.

13. Per-Head Tracking

Scores are maintained separately for each KV-head to handle head specialization.

Diagram 13.1 — Per-Head Score Matrix
Position Head 0
(recency)		 Head 1
(anchor)		 Head 2
(retrieval)		 Head 3-7 Aggregate Decision
0 (system) 0.001 0.089 0.012 ... 0.089 HBM
45,000 0.000 0.002 0.003 ... 0.003 CXL
99,950 0.082 0.004 0.031 ... 0.082 HBM
Paggregate(p) = max( Ph(p) ) for all heads h
Position stays in HBM if ANY head needs it. Evict only when NO head has recent access.

14. Prefetching Strategy

Predictive prefetch loads anticipated tokens from CXL before they're needed.

Diagram 14.1 — Prefetch Targets
Anchor (0-100)
Recent (m-200 to m)
High-EMA
1. Anchor zone [0, 100]
2. Recent [m−200, m−1]
3. High-EMA positions
Diagram 14.2 — Prefetch Timing Budget
Token Generation
50 ms
Compute
20 ms
Prefetch Window
30 ms
Prefetch capacity @ 256 GB/s aggregate:
7.68 GB ≈ 23,400 positions
30 ms × 256 GB/s = 7.68 GB; ÷ 320 KiB per position = 23,437. Steady-state demand is only 11.4 GB/s, so this window is never the constraint.

15. Hit Rate Progression

Each algorithmic improvement increases the HBM hit rate.

Diagram 15.1 — Algorithm Contribution
LRU baseline
+ Anchor pinning (+8 pts)
+ EMA scoring, α = 0.2 (+7 pts)
+ Per-head tracking (+6 pts)
+ RoPE-aware prefetch (+4 pts)
72 + 8 + 7 + 6 + 4 = 97. Contributions are percentage points, never percentages; the full stack is +25 points over the LRU baseline, and EMA in isolation is +7 points. Modeled ladder — not measured.
Diagram 15.2 — Effective Latency
Leff = hHBM × 100 ns + hCXL × 200 ns + hNVMe × 25 µs
At the 97 / 2.7 / 0.3 split:
177 ns
0.97 × 100 + 0.027 × 200 + 0.003 × 25,000
= 97 + 5.4 + 75 = 177.4 ns
Baseline for comparison:
11.5× lower
0.85 × 100 + 0.15 × 13,000 = 2,035 ns
Single access path:
65×
13.0 µs PCIe DMA ÷ 200 ns CXL.mem
same boundary, same 4 KiB payload
There is no recompute term: Tier 3 NVMe is a complete backing store, so a KV entry is never lost. Recompute happens only on cold start and is modeled separately. The 11.5× end-to-end figure and the 65× per-access figure are different comparisons and must not be conflated.

16. Final Results

Diagram 16.1 — System Comparison
Baseline — 3× B200 + host-DRAM offload
Memory 576 GB HBM
Users @ 128K per GPU 1
CapEx (16 users) $160,000
Hardware 3× B200 + 2 TB DDR5
Proposed — 1× B200 + CXL tiering
Memory 1.2 TB
Users @ 128K 16
CapEx (16 users) $103,000
Hardware 1× B200 + switch + 4 endpoints
Both configurations serve 16 users at 128K and ≥10 tok/s. 1.2 TB = 192 GB HBM + 1,024 GB CXL DRAM. CapEx reduction $160,000 → $103,000 = 36%; 3-year TCO including power $177,219 → $111,389 = 37%; power 3.9 kW → 1.9 kW = 51% less. Endpoints are $5,000 each.
Diagram 16.2 — Key Metrics
6×
Memory Expansion
16×
User Capacity
36%
Cost Reduction
177 ns
Effective Access Latency
97%
HBM Hit Rate
≥10
Tokens/sec/user (target)
Slug Architecture Research — v4.0, August 2026
Evidence label: Analytical model. All hit rates, latencies, capacities, bandwidths and costs in this appendix are modeled, not measured. No hardware benchmark was run.