Methodology, Assumptions and Evidence Status
What evidence exists behind every number in v5.0: the seven evidence classes with Measured present for the first time, what the v4.0 headline set claimed and why it is retracted, and the validation programme the remaining projections are still held to.
In v4.0 this page opened by declaring that nothing in the package was measured. That is no longer true. Appendix L reports hardware measurements taken on a DGX Spark (GB10) on 19 August 2026, and Appendix N reports a corrected trace-driven simulation. Between them they contradict the v4.0 headline set — 16× user capacity, 97% HBM hit rate, 65×, 11.5×, 15.6× — all of which are retracted. Their derivations have been removed from §K.3 and replaced by a retraction table pointing at the evidence that displaced each one. The complete list is in CANONICAL-NUMBERS.md §7 and RETRACTIONS.md.
Nothing retracted here was dishonest. All of it was constructed rather than observed, and the construction did not survive contact with a measurement. That is the correct outcome, and this appendix — whose job was always to state what evidence existed — says so plainly.
K.1 Evidence Taxonomy
Every quantitative claim in this package carries one of seven labels, ordered here from strongest evidence to weakest. For the first time the package contains Measured, and the two weakest classes, Design target and Illustrative, are now clearly separated from anything that could be mistaken for a result.
| Class | Meaning | Where it is used in this package |
|---|---|---|
| Measured | Run on named hardware, with method and raw data published. | Appendix L — DGX Spark GB10, 19 August 2026: achieved streaming bandwidth, the two-term decode model, both raw tables, the method, and the two measurement bugs found and fixed. This is the only source of measured evidence in the package. |
| Simulated (provisional) | Trace-driven simulation, corrected after an erratum, and not settled. | Appendix N — KV tiering simulation, Revision 2. Carries a standing caveat wherever quoted, because a single defect in Revision 1 inverted four of five findings. |
| Analytical model | Derived from stated inputs. Reproducible on paper; correct only if the model's structure and inputs are correct. | Capacity sizing and the per-token KV table (Appendices C, I); tier economics (§K.3.6); the derived bandwidths in Appendix L, which are computed from fitted coefficients rather than read off an instrument. |
| Vendor specification | Copied from a product datasheet and used as an input. | The 273 GB/s GB10 spec figure, HBM capacity and bandwidth, DDR5 timings, PCIe/CXL PHY and switch latencies, link and device specs. |
| External literature | A third party's published result, cited but not reproduced here. | Attention-sparsity and head-specialisation results from H2O, Ada-KV, Quest, SnapKV, Scissorhands, PyramidKV, HeadKV, InfiniGen and the induction-head / retrieval-head literature; LRFU (Lee et al., SIGMETRICS 1999); published vendor benchmarks. Used throughout Appendix E and in the prior-art sections of Chapters 2 and 10. |
| Design target | Desired, not demonstrated. | Endpoint unit cost and endpoint firmware throughput. The 5–10× preprocessing-offload target is retracted along with the rest of the v4.0 headline set. |
| Illustrative | Pedagogical, never a result. | Worked EMA examples, per-layer transfer walkthroughs, MoE routing examples, the reuse-ratio example in Appendix M §M.5.2. |
Two rows moved. Measured was previously annotated "Never. Nothing in this package is measured"; it now points at Appendix L. The v4.0 label Trace-driven simulation, previously annotated "Not yet performed", has been run twice and is renamed Simulated (provisional) — the parenthesis is not decoration. It records that Revision 1 of that simulation was published, was wrong, and was corrected only after external review (Appendix N §N.7). A label that had never been used is now a label that has already been wrong once, and it is ranked accordingly: below Measured, above Analytical model, and never quotable without its caveat.
An External literature result is a real measurement — but of someone else's system, on their models, their workloads, their hardware and their configuration. It is not evidence about this design, and it never upgrades a claim here toward Measured. The label exists so that a reader can see immediately when this package is leaning on a published finding rather than on its own arithmetic, and so that the transfer step — "their result implies something about our system" — is visible and arguable rather than silent. Where a published result is used as an input to a model here, the resulting number is labelled Analytical model, not External literature; the citation labels the input, not the output. The only claims in this package that are Measured are the ones in Appendix L, and they are measurements of stock frameworks on stock hardware — not of the proposed architecture, which has still never been built.
K.2 The Canonical Scenario (S1)
The package uses exactly one scenario. Every derivation below refers to it. Where a chapter appears to quote a different configuration, S1 governs.
| Parameter | Value |
|---|---|
| GPU | 1 × NVIDIA B200 (HGX B200 baseboard), 192 GB HBM3e, 8 TB/s |
| GPU dense compute | 2,250 TFLOPS BF16 dense (4,500 TFLOPS is the 2:4-sparsity figure) |
| Model | Llama-2-70B — 80 layers, 64 Q heads, 8 KV heads, d_head 128, BF16 |
| Weights in HBM | 140 GB |
| Activations / workspace | 5 GB |
| Context per user | 128 K tokens (131,072) |
| Target decode rate | ≥ 10 tok/s per user (≈160 tok/s aggregate) |
Derived KV sizing follows directly. Per token: 2 × 80 × 8 × 128 × 2 B = 327,680 B = 320 KiB/token. Per user at 128 K: 327,680 × 131,072 = 40 GiB = 43 GB. Eight such users would need 140 + 8×43 + 5 = 489 GB, which is 2.5× a 192 GB B200. The proposed HBM budget partitions the 192 GB as 140 weights + 5 activations + 10 pinned + 37 evictable, with 1 TB of CXL DRAM at 200 ns and 16 TB of endpoint NVMe at 25 µs behind it.
The per-token and per-user arithmetic above stands: 320 KiB/token and 40 GiB at 128 K are analytic, simulator-independent and independently confirmed (CANONICAL-NUMBERS.md §7). The partition of the 192 GB into 140 + 5 + 10 + 37 does not: it was chosen to serve 16 users at a 97% hit rate, and both of those figures are retracted (§K.3.2). Note also that S1 is not the machine anything was measured on. The measurements in Appendix L were taken on a GB10 with 128 GiB of unified LPDDR5X running a 7B model; no B200 was benchmarked, and no claim in this package is measured at the S1 scale.
K.3 Derivations: What Stands, What Is Retracted, Where the Evidence Now Lives
In v4.0 this section derived every headline number. Most of those numbers no longer exist. What follows is the reconciled version: the derivations that survive, the ones that were withdrawn and why, the ones demoted to unmeasured assumptions, and pointers to the measured and simulated evidence that replaced them.
K.3.1 What stands
This is a capacity ratio and nothing more. It says nothing about bandwidth, latency or utility, and it never depended on the hit rate — which is exactly why it survived. Stands.
So does the KV sizing arithmetic: 320 KiB/token for Llama-2-70B and Llama-3-70B GQA, 5.00 MiB per 16-token block, and the full capacity table in CANONICAL-NUMBERS.md §2. Analytical model, simulator-independent, independently confirmed. The decisive term is kv_heads, not query heads, and the v4.0-era claim that Llama-2-70B is MHA at 2,560 KiB/token is corrected: that row belongs to Llama-1-65B and the smaller MHA Llama-2 variants.
K.3.2 Retracted — the v4.0 headline set
Each row names what replaced it. Where the replacement is Appendix N, the replacing figure is Simulated (provisional) and carries the standing caveat of §N.9 — a retraction is not an invitation to treat the replacement as settled.
| v4.0 figure | Status | Why, and where the evidence now lives |
|---|---|---|
| 97% HBM hit rate | Retracted | Never measured. The corrected simulation gives 8.5–91.6% depending on budget, policy and workload — Appendix N §N.2, Simulated (provisional). |
| The 72 → 80 → 87 → 93 → 97 ladder (+8 / +7 / +6 / +4 pts) |
Retracted | No ablation ever produced it. The increments were assumed additive and independent, and the mechanisms they decomposed overlap. The v4.0 text said the additivity was itself an assumption; it was load-bearing and it was wrong. |
| +25 pts over LRU | Retracted | The corrected simulation gives a spread of +2.5 to +5.8 pts at the specified α, and +19.1 pts at the tuned α ≈ 0.05 optimum — Appendix N §N.2, §N.3, Simulated (provisional). |
| 16× user capacity | Retracted | Downstream of the 97%. Under dense attention every block of an active sequence is read every step, which is the formal statement in Appendix M §M.4.2. |
| 5.4% hot set → 97% of accesses | Retracted | Conflated cache hit rate with attention sparsity — two different claims with different obligations. The LOSSLESS constraint separates them formally: Appendix M §M.4.2. |
| 65× (200 ns vs 13.0 µs) | Retracted as a headline | Compares a hypothetical load/store path against a driver-mediated DMA path. No CXL hardware was measured. The component budgets are retained in §K.3.3 as documented assumptions, not as a result. |
| 11.5× effective end-to-end latency | Retracted | Derived from the 97%. The 177 ns effective-access-latency figure it rested on goes with it. |
| 15.6× TTFT | Retracted | Rested on a 16.4 GB/s sustained-offload baseline — an assumed ~32% of link-level bandwidth. Appendix L §L.2 measures achieved bandwidth at 87% of spec, which contradicts a haircut of that size. Measured. |
| 36% CapEx reduction | Retracted | Priced a configuration whose existence was justified by the 16×. With the 16× gone there is no equal-service-level comparison left to price. |
| 5–10× preprocessing offload | Retracted | A Design target for endpoint ARM cores that was never benchmarked on silicon, inside an architecture motivated by the retracted hit-rate story. |
K.3.3 Demoted — the 200 ns and 13.0 µs latency budgets
200 ns CXL access is demoted to an unmeasured assumption inside a 32–121 GB/s bandwidth range. The component budgets below are retained because they document precisely what was assumed and where it would break, not because they constitute evidence. Both are sums of unloaded component latencies: they contain no queueing term and no contention term, and a real loaded system will be slower.
CXL.mem path — Analytical model over Vendor specification components.
| Component | Latency |
|---|---|
| GPU/host request issue | 15 ns |
| PCIe Gen5 ×16 PHY | 20 ns |
| CXL switch traversal | 25 ns |
| Endpoint CXL controller decode | 40 ns |
| DDR5-5600 array access | 80 ns |
| Return path | 20 ns |
| Total (assumed) | 200 ns |
PCIe DMA baseline — Analytical model.
| Component | Latency |
|---|---|
| Driver call + doorbell | 4.00 µs |
| DMA descriptor fetch/setup | 2.50 µs |
| Software TLB + pinning management | 1.50 µs |
| PCIe Gen5 round trip | 0.70 µs |
| Host DRAM access | 0.10 µs |
| 4 KiB payload @ 32 GB/s | 0.20 µs |
| Completion interrupt + resync | 4.00 µs |
| Total (assumed) | 13.00 µs |
Of the 13.0 µs, 8.0 µs is software — doorbell and completion — which is what a load/store path would remove. That observation survives. The ratio between the two columns does not survive as a headline, for the reason given in §K.3.2: the two budgets describe different mechanisms, neither was measured, and dividing one unmeasured number by another does not produce a result.
K.3.4 The measured decode model — Measured
What replaced the roofline arithmetic this section used to contain is a fitted two-term model on real hardware. In summary, with the full derivation, raw tables and method in Appendix L:
| Quantity | Value | Derived in |
|---|---|---|
| Achieved streaming bandwidth, GB10 | 236.5 GB/s = 87% of spec | L §L.2 |
| Decode model, HuggingFace + SDPA | 100.3 + 16.34·KV_GiB ms (R² 0.991) | L §L.3 |
| Decode model, vLLM + FlashAttention-2 | 73.9 + 17.58·KV_GiB ms (R² 0.945) | L §L.3 |
| KV read as a fraction of achieved bandwidth | 26–28% | L §L.3.1 |
| Model-derived effective KV/fixed-path ratio | ~3.4× | L §L.4 |
Two consequences reach back into this appendix. First, the ~50% effective-bandwidth haircut used in earlier drafts is retired — it was wrong, and wrong in the pessimistic direction, which is what removes the floor from under the 15.6× TTFT derivation. Second, kernel substitution alone did not eliminate the measured KV slope: two tested frameworks with different attention implementations land within 8% on the KV term, so no assumption in §K.7 about kernel-level improvement can be used to rescue a bandwidth argument.
K.3.5 The simulated policy results — Simulated (provisional)
The trace-driven simulation that §K.6 proposed as validation step 1 has now been run — and the first run was wrong. Revision 2 is in Appendix N, with the erratum for Revision 1 in §N.7 and the standing caveat in §N.9. The headline outcomes:
- Hit rate is 8.5–91.6% across budgets and policies, not 97% (N §N.2).
- α is non-monotonic with an interior optimum at α ≈ 0.05, worth +19.1 pts over LRU at 32 GiB — not the α = 0.2 default this appendix documents in §K.7 (N §N.3).
- Persistent scoring metadata is the largest effect in the study, +34.15 pts to LFU — worth more than the choice of policy (N §N.4).
- No policy dominates across workloads (N §N.5).
All four carry the standing caveat. A single data-structure defect inverted four of five Revision 1 findings, and that is the reason the evidence class is named Simulated (provisional) rather than simply "simulated".
K.3.6 Tier economics — Analytical model
Per 5.00 MiB block (16 tokens, Llama-3 70B GQA):
| Operation | Cost | Basis |
|---|---|---|
| Tier fetch @ 121 GB/s | 43 µs | CXL 3.0 ×16 theoretical, one direction |
| Tier fetch @ 64 GB/s | 82 µs | Mid-range assumption — no CXL hardware was measured |
| Tier fetch @ 32 GB/s | 164 µs | Pessimistic |
| Forward compute, 16 tokens, linear term | 9 / 11 / 18 ms | 50% / 40% / 25% MFU |
| Regeneration of an arbitrary interior block | ≥ the above, potentially far higher | Interior KV is not independently reproducible — hidden states depend recursively on all preceding context |
Moving a block is two to three orders of magnitude cheaper than regenerating it. The two quantities do not measure the same operation, so a single ratio between them is not meaningful. The 218× figure in circulation is one point in that range (17.9 ms ÷ 82 µs) and must always be quoted with its inputs — including the fact that the 82 µs denominator is an assumption about hardware nobody in this project has measured.
K.4 Assumptions That Carry the Result
The v4.0 version of this section listed six assumptions, five of which supported figures that are now retracted. What follows is the v5.0 set: the assumptions that carry what the package still claims. If any one of them is wrong, specific things move in specific directions.
| Assumption | If it is wrong |
|---|---|
| The GB10 measurement generalises in direction, if not in magnitude. KV is read at roughly a quarter of achievable bandwidth, and this is a property of the attention read pattern rather than of one part. | The ~3.4× effective-path ratio remains GB10-specific. The traffic-reduction conclusion (L §L.5.1) narrows to a claim about LPDDR5X-class unified memory, and the argument would need re-running on an HBM part before it could be stated generally. This is the single most consequential open question in the package, and closing it needs one afternoon on a borrowed H100. |
| fp8 KV halves KV bytes without doubling the slope — outcome A of L §L.10. | If outcome B lands instead, dequantisation cost cancels the byte saving: fp8 buys capacity but not bandwidth, and the compression lever is representation-limited rather than byte-limited. The measurement is not yet run and both outcomes are named in advance. |
| The Revision 2 simulator is now correct. The exact order-preserving key of N §N.7.3 fixed the defect and introduced no other. | Four of five findings could invert a second time. The precedent is not hypothetical: Revision 1's defect produced 99.6% wrong victims and was found by external review, not internally. The re-audit reports 0 wrong of 78,860, which bounds this specific failure mode and nothing else. |
| A synthetic Zipf(1.1) return pattern over 200 sessions resembles production reuse. | N §N.5 already shows the policy ranking is workload-dependent, and that no policy dominates all three synthetic workloads. If real traces look like the cyclic sweep, only LFU survives; if they look like the one-shot flood, the ranking changes again. No production trace has been replayed. |
| Whole-context residency — a session's context is resident or it is not. | The entire hit-rate accounting in Appendix N changes. Partial residency and per-block admission within a context are the regime the object model of Appendix M actually describes, and the simulator does not model it. |
| Tier bandwidth lies in the 32–121 GB/s range, with 64 GB/s as the mid-range working assumption. | The tier-economics table of §K.3.6 moves by up to 4× end to end. No CXL hardware was measured, so this range is a specification-derived bracket, not an observation, and the 218× figure moves with it. |
| Metadata processing overhead is small enough not to change the conclusion. Payload-capacity overhead certainly is — 64 B against a 5.00 MiB block is 0.0012%. | Nothing bounds it. Appendix M §M.2.4 enumerates five open items — lookup bandwidth on the promotion path, atomic update cost under concurrent scheduler writes, controller SRAM footprint, cross-plane policy coherence, per-transaction link overhead — none of which is bounded by the capacity argument, and none of which has been measured. |
K.5 What Would Falsify the Design
Stated as testable propositions. Any one of these is sufficient, and each names an experiment rather than an argument.
- The fp8 KV run lands on outcome B — slope roughly doubles to ~35 ms/GiB — so halving KV bytes does not halve KV time. The compression lever, which L §L.5.1 currently makes the dominant one, would be representation-limited.
- Persistent identity-keyed policy state gives no advantage on a real trace, or host-side persistence achieves the same result. That refutes claim C4 of Appendix M §M.3.2 — the only one of the four that does not require a device — and with it the protocol contribution the package now rests on.
- Production traces show a workload mix in which LRU or LFU dominates at every budget. The policy contribution goes, and N §N.5 already shows this is workload-decidable rather than settled.
- An offline-optimal (Belady) bound shows the achieved margins are a negligible fraction of what was available. No such bound exists in Revision 2; it is the first item in the standing caveat of N §N.9.
- Measured loaded CXL access latency on real 3.0 hardware lands closer to the 13.0 µs DMA baseline than to the 200 ns budget, or a shipping GPU platform cannot issue load/store into a mapped CXL.mem range at all. Either removes the platform dependency the architecture assumes.
- The KV bandwidth gap closes on a different memory technology. If an HBM part reads KV at the same fraction of achievable bandwidth as weights, the wall this package is organised around is a GB10 artefact.
K.6 Validation Programme
The following is what would have to be executed to convert the remaining projections into results. It is specified at the level of detail a reviewer would demand.
Experiment 1 (trace-driven simulation) has been run, twice. Revision 1 was published and was wrong; Revision 2 is in Appendix N and is Simulated (provisional). It did not confirm the design — it retracted four of five of its policy findings and replaced the 97% hit rate with a range of 8.5–91.6%. Experiment 2 has started on the memory-bandwidth side only: Appendix L characterises achieved bandwidth and the decode cost model on a GB10, but no CXL latency characterisation has been done and no CXL hardware has been touched. Experiments 3 (end-to-end serving benchmark) and 4 (FPGA implementation) remain entirely unstarted. The specification below is retained unchanged as the standard the remaining work is held to.
| Element | Specification |
|---|---|
| Experiments | (1) Trace-driven cache simulation of the full policy. (2) Loaded and unloaded CXL latency characterisation on real hardware. (3) End-to-end serving benchmark with a vLLM KV backend against emulated far memory. (4) FPGA implementation of the scoring, eviction and prefetch engines behind a CXL IP core. |
| Workloads | Long-context retrieval and QA at 128 K (for example LongBench and ∞Bench-class tasks), multi-turn conversational resumption, shared-system-prompt serving, and a synthetic adversarial pattern with uniformly random access designed to defeat locality. Report each separately; do not average across them. |
| Baselines | vLLM with PagedAttention and its own CPU offload; SGLang with RadixAttention; InfiniGen; Mooncake; NVIDIA Dynamo; and a 3 × B200 host-DRAM offload configuration matching the cost model's baseline column. Baselines must be run, not modeled. |
| Ablations | One per rung of the hit-rate ladder: LRU alone; +anchor pinning; +EMA (α ∈ {0.05, 0.1, 0.2}); +per-head tracking versus position-level; +RoPE prefetch. Additionally: max-aggregation versus mean-aggregation across heads; hot-set size swept from 2% to 20% of context; user count swept from 1 to 24 to test the claim that 16 is the bound. |
| Metrics | HBM hit rate; effective access latency; time to first token; inter-token latency; aggregate and per-user tokens/s; CXL link utilization; endpoint metadata footprint; energy per token; and end-task quality on the retrieval workloads, to confirm the policy does not silently degrade output. |
| Statistics | Minimum 10 repetitions per configuration; report median with p50/p95/p99 for all latency metrics; 95% confidence intervals on all means; explicit warm-up exclusion; and a stated seed policy for any sampled workload. |
| Versions | Full disclosure of GPU model and firmware, CXL host platform and BIOS, switch and endpoint part numbers and firmware revisions, kernel version, CUDA version, vLLM/SGLang commit hashes, model checkpoint revision, and the exact simulator source. |
v4.0 said experiment 1 required no hardware, would settle the central question, should be done first, and should gate everything else. That was correct advice, and taking it is what produced the retractions in §K.3.2. It also produced a second lesson v4.0 did not anticipate: the first version of the experiment was itself defective, in a way that inverted its findings and was caught only by external review of the eviction data structure (Appendix N §N.7). An experiment gates the work only if the experiment is itself audited — which is why the unmodified Revision 1 simulator is retained in the bundle at scripts/kv_tiering_sim_rev1_retained_for_audit.py, and why the eviction audit against brute-force argmin is now a standing check rather than a one-off.
K.7 Uncalibrated Parameters
The table below is retained as the record of what was never fitted, because that record is the honest explanation of how the retracted hit-rate ladder came to exist. Three specific reconciliations apply to it in v5.0:
- α = 0.2 is superseded. Appendix N §N.3 sweeps α and finds a non-monotonic curve with an interior optimum at α ≈ 0.05, worth +19.1 pts over LRU at a 32 GiB budget (Simulated (provisional)). Every threshold below that is defined relative to α = 0.2 is therefore defined relative to the wrong α.
- The rows tied to the hit-rate ladder no longer have a result to carry. The +8 / +7 / +6 / +4 pts attributions the anchor-zone, EMA, per-head and prefetch rows point at were never produced by an ablation and are retracted (§K.3.2).
- The PCIe contention factor is contradicted, not merely uncalibrated. Its row calls itself "the cheapest item in the table to measure"; the closest measurement now in the package — achieved bandwidth at 87% of spec, L §L.2 — is inconsistent with a 32% haircut, which is why the 15.6× TTFT figure it multiplied is retracted.
What has not changed is the central point of this section, which survives its own policy: a derivation can be arithmetically exact and still rest on an input nobody measured.
The policy described in v4.0 of this package contains a number of bare numeric constants — thresholds, coefficients, window sizes, batch sizes — that are neither derived from the canonical scenario nor fitted to any workload. They were chosen to make the design concrete and legible. They are collected here in one place so that a reader can see the full set at once, argue with any of them, and know what it would take to replace each with a fitted value. Every figure and appendix that uses one of these constants points back to this table.
An uncalibrated parameter is not a labelling category alongside Analytical model or Design target — it is a property of an input to those models. The distinction matters, and it is the distinction this whole appendix now turns on: a derivation can be arithmetically exact and still rest on an input nobody measured. None of the v4.0 headline numbers was directly computed from the constants below; they entered indirectly, through the modelled hit-rate ladder. That ladder is retracted (§K.3.2), so the honest statement in v5.0 is narrower than the one v4.0 made here: these constants determined whether the ladder was ever achievable, and nobody ever checked.
Note also that these parameters are not independent. The EMA thresholds are meaningful only relative to α; the recency coefficient β double-counts recency unless swept jointly with α; the tier boundaries are cut points on the score the tier weights define; and the prefetch window, depth and distance scale all trade against the same fabric bandwidth. Calibrating any one in isolation is close to meaningless.
| Symbol / name | Value used | What it controls | Where it appears | How it would be calibrated |
|---|---|---|---|---|
| β | 0.001 | Recency-decay coefficient in 1 − e−β·steps. Sets the horizon (1/β = 1,000 decode steps, ≈100 s at 10 tok/s) past which idleness alone justifies eviction. Raising it reclaims memory faster and makes the policy recency-driven, at the risk of evicting long-lived structural tokens; lowering it lets stale entries linger and consume capacity. | Fig. EMA eviction, Fig. EMA calculation | Two-dimensional sweep jointly with α over replayed per-head attention traces at 128 K context, scored on hit rate at a fixed hot-set size. β chosen without reference to α double-counts recency. |
| keep threshold | 0.10 | EMA score above which an entry stays HBM-resident. Sets the size of the hot set. Raising it frees HBM sooner and admits more users at the cost of hit rate; lowering it raises hit rate and reduces the users the 37 GB evictable budget can serve. | Fig. 7.7 EMA scoring | Sweep the (keep, evict) pair against real attention traces; take the point meeting the 97% hit-rate SLO at the smallest hot set. Almost certainly differs per head type and per model. |
| evict threshold | 0.02 | EMA score below which an entry is pushed from CXL DRAM to flash. The band between the two thresholds is the demote zone, whose width sets how much traffic the CXL tier absorbs. Raising it reclaims CXL DRAM faster at the cost of more 25 µs flash reads on the miss tail; lowering it keeps more of the context warm and moves pressure onto the 1 TB tier. | Fig. 7.7 EMA scoring | As above — the pair is swept together. Both are tied to α = 0.2: an EMA lags its input, so 0.10 is not "10% of attention", and changing α changes what both numbers mean. |
| W | 32 / 128 / 512 (128 default) |
Prefetch window radius: [P − W, P + W], so 2W + 1 entries are fetched speculatively per access. Fixes both attention coverage and speculative bandwidth. Raising it widens coverage but cost grows linearly while return is sharply sublinear, and unread prefetches waste CXL bandwidth and HBM residency; lowering it leaves more of the miss tail at 200 ns or 25 µs. At W = 0 the +4-point rung disappears. | Fig. 7.10 RoPE prefetch | Measure the real per-head attention-versus-distance distribution on production traces, then take the smallest W whose marginal coverage still exceeds its marginal bandwidth cost. Very likely per-head-type: local heads want a small W; retrieval heads are poorly served by a distance window at any W. |
| tier-placement weights (R, E, N) |
0.25 / 0.55 / 0.20 | Weights in P(p) = 0.25·R + 0.55·E + 0.20·N, the score deciding which tier an existing entry occupies. Raising E tracks sustained attention and reacts slowly to topic change; raising R does the reverse; raising N hardens the anchor zone at the cost of pinned capacity. They sum to 1, so every increase is another signal's decrease. | Visual appendix, Diagram 12.1 | Fit, do not choose. Log per-token (recency, EMA, anchor-membership, was-attended) tuples from real 128 K traces and solve for the weights maximising hit rate at the fixed 47 GB HBM budget — a small constrained regression or coarse simplex sweep. The tier boundaries below must be re-fitted at the same time. |
| tier boundaries | P = 0.3, P = 0.6 | Cut points on P(p) separating Tier 2 (CXL) / Tier 1 (HBM) / Tier 0 (pinned). Together they split the HBM budget. Raising either frees HBM sooner and admits more users at the cost of hit rate; lowering either does the reverse. | Visual appendix, Diagram 12.2 | Cannot be tuned independently of the weights above — they are thresholds on the score those weights define. Same sweep, same traces, same SLO. |
| prefetch blend | 0.6 / 0.4 | Weights in priority = 0.6·rope_factor + 0.4·ema_factor, ranking the prefetch queue. Toward RoPE makes prefetch purely positional — a sliding window, cheap and good for local heads, blind to retrieval heads reaching far back. Toward EMA makes it history-driven — serves retrieval heads, reacts slowly at a context switch, loses the locality that makes prefetch batchable. | Appendix D, Appendix J §J.2 | Sweep the single free weight (they sum to 1) on replayed traces, scored on prefetch precision — the fraction of prefetched entries actually read before eviction — not on hit rate alone, since a useless prefetch costs bandwidth either way. |
| prefetch_distance_scale | 100 | Denominator in 1/(1 + distance/scale): the distance at which prefetch priority halves. Raising it flattens the curve so far positions compete more evenly with near ones — better for retrieval heads, more wasted bandwidth; lowering it sharpens it toward a strict sliding window. Canonically fixed at 100 for consistency across the package; the choice of 100 itself was not fitted. | Appendix J §J.2, §J.4, Chapter 7 §7.6 | Fit the hyperbola to the measured attention-versus-distance curve rather than assuming it. Interacts directly with W, since both shape what the prefetcher reaches for. |
| anchor_zone_size | 100 | Leading positions exempt from eviction outright. Raising it hardens the attention-sink and system-prompt protection the +8-point anchor-pinning rung depends on, but spends scarce Tier 0 capacity on positions that may never be attended to; lowering it frees Tier 0 and risks evicting the sink tokens. | Appendix J §J.1, §J.4 | Measure where attention-sink mass actually stops on the target model — the sink is a property of the trained weights, so this is measurable once per model and then fixed, rather than swept per workload. |
| prefetch_depth | 2 layers | How many layers ahead the prefetcher runs. Raising it widens the delivery window (coverage in I.5.2 scales linearly in it) and buys slack against fabric latency, at the cost of speculating further ahead on less certain predictions and holding more in-flight staging buffer; lowering it tightens the window until the miss tail no longer fits. | Appendix J §J.4, Appendix I §I.5.2 | I.5.2 takes 2 as an input and computes 2.06× miss-volume coverage from it; it does not derive 2. Calibration means sweeping depth against measured prediction accuracy, since coverage of bandwidth is already comfortable and accuracy is the binding term. |
| eviction_batch | 64 entries | Entries evicted per policy pass. Raising it amortises the sort and metadata walk over more work, cutting firmware overhead per entry, but coarsens the decision — a large batch evicts entries scored before the batch began and may over-evict past the point where capacity was recovered. Lowering it keeps decisions fresh at higher per-entry firmware cost. | Appendix J §J.4 | Depends on controller throughput, which is itself a Design target for unbuilt silicon. Calibrate against firmware cycle cost per pass once a controller model exists; until then it is a placeholder. |
| PCIe contention factor | 0.32 (16.4 of 51 GB/s) |
The fraction of link-level PCIe Gen5 ×16 bandwidth the modeled host-DRAM offload baseline is assumed to sustain in service. Raising it toward 1.0 makes the baseline faster and the 15.6× TTFT result smaller — at the full 51 GB/s the baseline is 843 ms and the speedup 5.0×; lowering it inflates the result. | §K.3.9 above; Chapter 11 §11.6; Fig. 1 bandwidth wall | Measure achieved offload bandwidth on a loaded host running the baseline configuration — 3 × B200 with concurrent weight, activation and NIC traffic. This is the cheapest item in the table to measure and it is a direct multiplier on a headline number. |
| demo thresholds | 0.6 / 0.25 | Keep / evict cut points used only inside the Figure 7.9 animation, chosen to suit its synthetic 0.1–0.9 score spread. Not the design's thresholds and not comparable to them: a real EMA at α = 0.2 saturates near 0.04 for a token drawing 4% of attention, which is why the design's thresholds are an order of magnitude lower. | Fig. 7.9 EMA eviction policy | Not calibrated because not calibratable — it is a presentation constant, listed here only so the discrepancy with 0.10 / 0.02 is not mistaken for a contradiction. |
α = 0.2 is deliberately not in this table, and v5.0 is the reason that decision aged badly. It was fixed canonically across v4.0 (half-life ln 0.5 ÷ ln 0.8 = 3.1 decode steps), and Appendix G documents the 0.05 / 0.1 / 0.2 trade-off explicitly, so it was a stated and examined choice rather than a bare constant — but it was never a fitted one. It has since been swept: Appendix N §N.3 finds the curve non-monotonic with an interior optimum at α ≈ 0.05 (Simulated (provisional)). Because the EMA thresholds and β are all defined relative to α, every row above is defined relative to a value the sweep does not support.
Of the thirteen rows, the trace-driven sweep that would fit them has now been run once, and it moved α by a factor of four on the first attempt. The PCIe contention factor — the row that called itself the cheapest thing here to measure and a direct multiplier on a headline number — is the clearest illustration of the cost of not measuring: the headline it multiplied is retracted, and the closest measurement in the package contradicts the factor (Appendix L §L.2). The remaining rows are still unfitted, and calibrating them is the same piece of work as making Appendix N less provisional.
K.8 Known Limitations of This Document
- It is now part model, part evaluation — and the two must not be blurred. The cost model has been measured on one machine (Appendix L). Every policy claim remains specification (Appendix M) or provisional simulation (Appendix N). A reader should treat every number that is not labelled Measured as a hypothesis with a justification attached.
- The measurement is narrow. One model (Qwen2.5-7B), one size, one machine (GB10), fixed-length uniform batches, no continuous batching, FlashAttention-2 rather than 3, no thermal log, and fp8 KV not yet measured. The full list is L §L.9.
- The simulation has already been wrong once. Revision 2 is provisional: no Belady bound, no bandwidth or queueing model, whole-context residency assumed, synthetic workload, 2–3 seeds, no hardware validation (N §N.9).
- The two never met. Nothing measured in Appendix L was fed into the simulator of Appendix N, and nothing simulated in Appendix N was run on the hardware of Appendix L. They answer different questions at different scales and must not be merged into one narrative.
- No CXL hardware was measured at any point. The 200 ns and 13.0 µs budgets omit queueing and contention, the tier-bandwidth range of 32–121 GB/s is specification-derived, and the 200 ns figure is demoted to an assumption inside it (§K.3.3, §K.3.6).
- One scenario for the analytical work. All sizing conclusions are drawn from S1, which is not the machine anything was measured on. MoE behaviour (DeepSeek-V3 class, 256 experts, top-8) is discussed illustratively only.
- The head-type split (40/15/25/20) is an assumption, labelled as such, with pointers to the induction-head and retrieval-head literature. It is not a measurement, and no surviving claim depends on it quantitatively.
- The cost model is gone rather than corrected. The 36% CapEx reduction priced a configuration justified by the retracted 16×; no replacement cost model is offered, because there is no longer an equal-service-level comparison to price.
- Prior art is not exhaustively benchmarked. H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV, InfiniGen, Mooncake and PNM-KV (arXiv:2511.00321) are cited as context. Note that the scoring function itself is not the contribution: a decayed-frequency score is LRFU (Lee et al., SIGMETRICS 1999) and α is its documented LRU↔LFU knob. Whether anything here is novel requires a systematic literature review that this package does not constitute.
- Uncalibrated policy constants. Thirteen thresholds, coefficients, window sizes and batch sizes carry the v4.0 policy and none was fitted to a workload. They are enumerated with their trade-offs in §K.7, together with the three reconciliations that v5.0 applies to them.
- Rounding. Surviving figures are rounded (6.3× → 6×; 236.5 GB/s ÷ 273 GB/s → 87%). Unrounded values are given at the point of derivation and should be used in any re-derivation.
v4.0 described an architecture that was internally consistent, arithmetically transparent, and entirely unmeasured, and said so. The correct next step was to run the experiment. The experiment was run, and the architecture's headline claims did not survive it — not because the arithmetic was wrong, but because the inputs were constructed rather than observed.
What v5.0 has instead is smaller and firmer: a measured cost model on one machine showing a ~3.4× model-derived effective-path ratio on GB10 that the two tested attention stacks did not eliminate; a formal specification precise enough that its open questions are experiments rather than arguments; and a provisional simulation whose most interesting result — that persistent identity-keyed policy state is worth more than the choice of policy — is a protocol claim testable without buying anything. That is a reasonable state for a proposal to be in. It is still not a state in which to call the policy numbers a result.