Tiering Simulation and Erratum
Revision 2 of the KV tiering study, every result table labelled with the experiment that produced it, and the full erratum for the Revision 1 defect that inverted four of five findings.
A single data-structure defect in Revision 1 inverted four of five findings (99.6% wrong victims for EMA α=0.15; §N.7). Revision 2 is Simulated (provisional) — provisional, not settled: no Belady bound, no bandwidth/queueing model, whole-context residency assumed, synthetic workload, 2–3 seeds, no hardware validation. Every table below repeats this caveat, and §N.9 states it in full as a closing section. The only Measured evidence in this package is in Appendix L; nothing on this page is measured.
N.1 Setup
Everything on this page comes from one simulator run over one workload generator. The configuration is stated here once, in full, so that any table below can be checked against it.
| Parameter | Value |
|---|---|
| Model geometry | Llama-3 70B, GQA, 8 KV heads — 320 KiB/token |
| Block granularity | 16 tokens = 5.00 MiB |
| Sessions | 200 |
| Return-frequency distribution | Zipf(1.1) |
| Context lengths | 1–12K tokens |
| Shared prefix | 2048 tokens, common to all sessions |
| Turns simulated | 2500 |
| Total footprint | 421 GiB |
| Tier structure | HBM tier over a 512 GiB CXL tier |
| Seeds | 2–3 per configuration (stated per table) |
| Simulator | scripts/kv_tiering_sim_v2.py |
| Raw results | data/results_v2.json |
| Revision 1 simulator, retained | scripts/kv_tiering_sim_rev1_retained_for_audit.py |
| Audit scripts | scripts/bug_check.py, scripts/bug_check2.py |
Each results table below names the experiment key it was computed from — e1, e2, e3, e5loc/e5per, e6no/e6ov. Those keys are the top-level record prefixes in data/results_v2.json, so every cell is traceable to the record it came from without re-running anything. Where this page and CANONICAL-NUMBERS.md disagree, the canonical spec governs: it is computed directly from results_v2.json.
N.2 Hit Rate versus HBM Budget
Experiment e1 — no persistent metadata, no overlap, 2 seeds. Simulated (provisional).
| HBM budget | LRU | LFU | EMA α=0.15 | EMA α=0.01 |
|---|---|---|---|---|
| 8 GiB | 8.52 | 8.76 | 12.35 | 12.12 |
| 16 GiB | 17.87 | 12.70 | 22.58 | 18.70 |
| 32 GiB | 38.34 | 16.09 | 44.18 | 35.91 |
| 64 GiB | 67.12 | 23.00 | 69.65 | 66.83 |
| 128 GiB | 88.01 | 32.70 | 88.14 | 91.59 |
Values are hit rate in percent. Standing caveat applies (§N.9): these are Simulated (provisional) numbers from a simulator whose predecessor was wrong about exactly this table.
- EMA α=0.15 leads at 4 of 5 budgets; EMA α=0.01 leads at 128 GiB.
- The margin over LRU is +2.5 to +5.8 pts, not +25. The retracted v4.0 figure was +25 points and never came from an ablation at all (CANONICAL-NUMBERS.md §7).
- LFU collapses once decode growth is modelled — classical LFU aging: fresh count-1 blocks are evicted in favour of stale high-count blocks.
N.3 α Is Non-Monotonic, with an Interior Optimum
Experiment e2 — the α grid at a 32 GiB budget, 1 seed. Simulated (provisional).
| α | 0.5 | 0.3 | 0.15 | 0.05 | 0.01 | 0.003 | 0.001 | LFU | LRU |
|---|---|---|---|---|---|---|---|---|---|
| hit % | 37.20 | 38.84 | 44.33 | 55.96 | 31.54 | 25.47 | 24.44 | 15.56 | 36.85 |
Standing caveat applies (§N.9). Simulated (provisional), 1 seed — the thinnest evidence on this page.
α ≈ 0.05 is the optimum, +19.1 pts over LRU. There is no convergence onto LFU as α → 0; the Revision 1 claim that there was is withdrawn (§N.8).
At the 32 GiB budget the two tables disagree: LRU 38.34 versus 36.85, EMA α=0.15 44.18 versus 44.33, LFU 16.09 versus 15.56. They are different experiments in results_v2.json: §N.2 is e1 (2 seeds), §N.3 is e2 (1 seed, the α grid).
Quote each table against its own baseline. Never mix a row from one experiment with a row from another. A comparison assembled across the two is not a comparison of policies; it is a comparison of seeds. The spread between them is itself a useful indication of seed noise at this scale, and it is roughly the same size as some of the policy margins being claimed — which is a further reason the standing caveat is not a formality.
CANONICAL-NUMBERS.md is authoritative; it is computed directly from results_v2.json. The erratum quotes 55.93% for the α optimum where the spec gives 55.96%, and 50–67% for warm recompute where the spec gives 44–89% across all budgets and policies. The erratum was written from a narrower slice of the data. This page follows the spec, and §N.8 reproduces the erratum's status table with the spec's figures substituted where the two differ.
N.4 The Largest Effect in the Study — Persistent Scoring Metadata
Experiments e5loc (metadata reset) and e5per (metadata persists) — 32 GiB budget. Simulated (provisional).
The intervention is to let identity-keyed policy state survive HBM eviction, rather than re-admitting a block as if it were newly created. This is the Appendix M §M.2.3 claim, tested.
| Policy | Metadata reset on eviction (e5loc) | Metadata persists (e5per) | Δ |
|---|---|---|---|
| LFU | 15.56% | 49.71% | +34.15 pts |
| EMA α=0.01 | 31.54% | 56.39% | +24.85 pts |
| EMA α=0.15 | 44.33% | 44.12% | −0.21 pts |
Standing caveat applies (§N.9). Simulated (provisional).
Persistence is worth more than the choice of policy. It also removes the small-α collapse, identifying metadata reset as its cause — the α=0.01 column of §N.3 recovers from 31.54% to 56.39% under the same budget.
It is claim C4 of Appendix M §M.3.2, and it is the only one of the four that does not require the controller to be on the device. If C4 is the claim that survives contact with hardware, the contribution is the object model and protocol, and CXL is simply where it runs (CANONICAL-NUMBERS.md §6).
N.5 Workload Dependence
Experiment e3 — 32 GiB budget, 3 seeds. Simulated (provisional).
| Workload | LRU | LFU | EMA α=0.15 | EMA α=0.01 |
|---|---|---|---|---|
| zipf (returning sessions) | 39.08 | 15.12 | 44.44 | 38.87 |
| scan (one-shot flood) | 31.51 | 12.52 | 32.57 | 36.26 |
| loop (cyclic sweep) | 9.83 | 22.06 | 9.83 | 9.83 |
Values are hit rate in percent. Standing caveat applies (§N.9). Simulated (provisional).
Only LFU survives the cyclic sweep — the workload on which every recency-weighted policy degenerates to the same 9.83%. No policy dominates across all three. Any claim of the form "policy X is the right policy" is therefore under-specified unless it names the workload, and the retracted v4.0 headline set named none.
N.6 Where Turn Stall Goes
Experiments e6no (no overlap) and e6ov (idealised transfer/compute overlap) — 32 GiB budget. Simulated (provisional).
| Regime | Experiment | Fetch per turn | Recompute share of stall |
|---|---|---|---|
| No overlap | e6no | Fully exposed | 44–89% depending on budget and policy |
| Idealised transfer/compute overlap | e6ov | 0.35–1.25 ms/turn | >99% |
Standing caveat applies (§N.9). Simulated (provisional). The 44–89% range is the spec's figure across all budgets and policies; ERRATUM.md quotes 50–67% from a narrower slice — see the note in §N.3.
Recompute is not the 97% of stall claimed in Revision 1. And the second row is the structural result: once movement is hidden, recompute avoidance is the only term that matters. That is a statement about where design effort should go, and it points at the HIDING constraint of Appendix M §M.4.3 rather than at the choice of replacement policy.
N.7 Erratum — The Revision 1 Defect, in Full
Correcting it, together with three experimental-design problems, reverses or substantially alters every policy finding of Revision 1. The KV-memory sizing correction (C1 in §N.8) is unaffected — it is analytic and simulator-independent.
N.7.1 The defect
_pick_victim() re-inserted a popped heap entry only when its recomputed score exceeded the stored key. This is the standard lazy-revalidation idiom, and it is correct for exactly one of the two policies it was serving:
| Policy | How scores move | How stored keys go wrong | Is the idiom correct? |
|---|---|---|---|
| LFU | Counts rise over time | Keys are stale-low | Yes. A popped entry whose true score has risen above its key must be re-inserted, and the guard catches exactly that case. |
| Decaying EMA | Scores fall between touches | Keys are stale-high | No. The recomputed score is below the stored key, so the guard never fires and the popped entry is returned as the victim without checking whether another block decayed further. |
The result is that under a decaying EMA the eviction heap did not return the minimum-score block. It returned an essentially arbitrary block from the frontier of the heap.
N.7.2 The audit — against brute-force argmin
Every eviction decision was re-evaluated against a brute-force argmin over all resident blocks, using scripts/bug_check.py:
| Policy | Wrong victims | Evictions | Rate |
|---|---|---|---|
| EMA α=0.15 | 114,114 | 114,528 | 99.6% |
| EMA α=0.01 | 73,650 | 74,987 | 98.2% |
| LFU | 0 | 74,987 | 0.0% |
The pattern is exactly what §N.7.1 predicts: LFU is untouched, both EMA configurations are almost entirely wrong. At 99.6%, the Revision 1 EMA policy was not a poorly tuned EMA — it was not an EMA at all.
N.7.3 The fix — an exact order-preserving key
The repair is to store a key that orders identically to the current decayed score at any time, so that no lazy re-evaluation is needed and the heap invariant never goes stale:
Because the decay factor is common to all blocks, taking logarithms turns the multiplicative decay into an additive term that depends only on the touch time. The ordering of K across blocks is therefore the ordering of their decayed scores at any instant, and the minimum of the heap is the true argmin without revalidation.
N.7.4 The re-audit
scripts/kv_tiering_sim_rev1_retained_for_audit.py is the unmodified Revision 1 simulator. It is kept in the bundle deliberately, so that the defect and its effect on the published Revision 1 numbers remain independently checkable rather than having to be taken on this page's word. A retraction that deletes the code that produced the retracted numbers cannot be audited.
N.8 Revision 1 → Revision 2 Status
What was reversed, what was withdrawn, and what stands. Figures are quoted from CANONICAL-NUMBERS.md where it and ERRATUM.md differ (see §N.3).
| Revision 1 finding | Status | Revision 2 |
|---|---|---|
| R1 — EMA α=0.15 worst at every budget | Reversed | R1′ — EMA α=0.15 best at 4 of 5 budgets (44.18% vs LRU 38.34% at 32 GiB, e1) |
| R2 — hit rate monotone as α → 0, converging exactly onto LFU | Withdrawn | R3′ — non-monotonic, interior optimum α ≈ 0.05 (55.96%, e2); no convergence onto LFU |
| R3 — LRU wins the scan workload | Withdrawn | EMA α=0.01 leads scan (36.26% vs 31.51%, e3) |
| R4 — EMA wins the cyclic sweep 2.2× | Withdrawn | Only LFU survives loop (22.06%); EMA tracks LRU at 9.83% (e3) |
| R5 — 97% of stall is recompute; policy nearly irrelevant | Withdrawn | R5′ — recompute is 44–89% of stall depending on budget and policy (e6no); under idealised overlap it becomes >99% (e6ov) |
| C1 — 8K Llama-3 70B KV is 2.50 GiB, not 20 GB | Stands | Unchanged. Analytic, simulator-independent, and independently confirmed (CANONICAL-NUMBERS.md §2) |
| New in Revision 2 — persistent scoring metadata | Added | R4′ — the largest effect in the study: +34.15 pts to LFU, +24.85 pts to EMA α=0.01 (e5loc/e5per; §N.4) |
Four of five policy findings did not survive. The one that stands, C1, is the one that never went through the simulator. That is the correct reading of this table, and it is the reason §N.9 exists.
N.9 The Standing Caveat
A single data-structure defect inverted four of five findings. Everything on this page is Simulated (provisional), and the following are unaddressed:
- No Belady bound. There is no offline-optimal reference, so no result here can be expressed as a fraction of what was achievable. Every margin quoted is a margin over another heuristic, not over the ceiling.
- No bandwidth or queueing model. Transfers are accounted for in bytes and nominal rates. The LINK, SUSTAINABILITY and HIDING constraints of Appendix M §M.4.3 are not enforced by this simulator.
- Whole-context residency is assumed. A session's context is resident or it is not; partial residency and per-block admission within a context are not modelled.
- Synthetic workload. Zipf(1.1) returns over 200 generated sessions with a shared prefix. No production trace was replayed.
- 2–3 seeds. §N.3 runs on one. The disagreement between e1 and e2 at 32 GiB is the size of the seed noise, and it is comparable to some of the policy margins claimed.
- No hardware validation. No CXL device was measured, and no part of this study ran on the hardware of Appendix L.
One defect already inverted four of five findings. That is why Revision 2 is provisional — not as a formality, but because the failure mode has been demonstrated once, in this codebase, at a 99.6% error rate, and was found by external review rather than by internal confidence. The correct posture toward the tables above is the posture that found the Revision 1 defect.
The measured results of Appendix L are a different and stronger class of evidence and do not inherit this caveat. The specification of Appendix M does not depend on these numbers for its correctness — only claim C4 of §M.3.2 draws support from them, and it is stated there as a hypothesis with both a supporting and a refuting measurement named.