Figure: attention-aware block eviction in five steps, tracking the (sequence, token block) containing position 1024. The unit is a block of 16 tokens, 5.00 mebibytes — never a head: every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss. The scoring rule is LRFU — Lee et al., "On the Existence of a Spectrum of Policies that Subsumes the LRU and LFU Policies", ACM SIGMETRICS 1999 — established prior art, with alpha as its documented LRU-to-LFU knob; it is not a new algorithm. Raw attention over five decode steps is noisy — 0.45, 0.12, 0.30, 0.08, 0.35 — so it is smoothed with score = alpha x new_score + (1 - alpha) x score. At alpha = 0.05, the simulated optimum, half-life ln0.5 over ln0.95 = 13.5 decode steps, the sequence becomes 0.023, 0.028, 0.042, 0.044, 0.059, against a raw average of 0.26. Alpha = 0.2 was never validated and is withdrawn. Alpha is non-monotonic with an interior optimum near 0.05: at a 32 gibibyte budget the simulated hit rates are 37.20 percent at 0.5, 38.84 at 0.3, 44.33 at 0.15, 55.96 at 0.05, 31.54 at 0.01, 25.47 at 0.003 and 24.44 at 0.001, so a smaller alpha is not simply a longer memory and there is no convergence onto LFU as alpha approaches zero. Eviction priority is (1 - score) x recency_decay with recency_decay = 1 - e to the minus beta times steps and beta = 0.001, about 63 percent decay at 1000 steps. The coefficient beta = 0.001 is an uncalibrated design parameter, not a derived or measured value: it sets the timescale over which idleness becomes a reason to evict, a characteristic time of 1 over beta = 1,000 decode steps, about 100 seconds at an illustrative 10 tokens per second. Raising beta makes eviction more aggressively recency-driven and puts long-lived but rarely touched blocks such as system prompts at risk; lowering it lets stale entries linger and consume capacity. Beta interacts with alpha and must be swept jointly against real attention traces. Appendix K section K.7 lists it with the package's other uncalibrated parameters. Block A at position 1024, score 0.059, accessed 50 steps ago, gives (1 - 0.059) x 0.049 = 0.046 and is kept; block B at position 45678, score 0.02, accessed 2000 steps ago, gives (1 - 0.02) x 0.865 = 0.848 and is evicted. Crucially, when block B is evicted its identity key and score persist and are restored on re-admission rather than reset — that persistence, not the scoring function, is the largest effect in the study: at 32 gibibytes it is worth plus 34.15 percentage points to LFU, from 15.56 to 49.71 percent, and plus 24.85 to EMA alpha 0.01, from 31.54 to 56.39 percent, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points. The decision matrix crosses score against recency: high score and recent is keep, as with a recently attended block at score 0.85, 20 steps, p 0.003; high score and old is watch, a system-prompt block at score 0.72, 3000 steps, p 0.266; low score and recent is watch, a newly arrived block at score 0.05, 10 steps, p 0.009; low score and old is evict, a never-attended filler block at score 0.03, 5000 steps, p 0.963. The worked numbers are illustrative, never a result; the simulated figures are provisional, because a single data-structure defect in Revision 1 inverted four of five findings.
Attention-Aware Block Eviction
Turning noisy attention into a stable eviction priority per (sequence, token block), using LRFU (Lee et al., ACM SIGMETRICS 1999) — established prior art, not a new algorithm.
16 tokens = 5.00 MiB. A head is never the eviction unit: every head of every layer is required for every token, so dropping a head produces wrong outputs, not a cache miss. H2O and SnapKV evict (head, token) pairs — the token axis is not optional.
Each decode step, every cached token receives an attention weight from the current query. These scores are noisy—a token might spike one step and drop the next.
| Step | new_score | Calculation | score_ema |
|---|---|---|---|
| t₁ | 0.45 | 0.05 × 0.45 + 0.95 × 0.000 | 0.023 |
| t₂ | 0.12 | 0.05 × 0.12 + 0.95 × 0.023 | 0.028 |
| t₃ | 0.30 | 0.05 × 0.30 + 0.95 × 0.028 | 0.042 |
| t₄ | 0.08 | 0.05 × 0.08 + 0.95 × 0.042 | 0.044 |
| t₅ | 0.35 | 0.05 × 0.35 + 0.95 × 0.044 | 0.059 |
The recency-decay coefficient is not derived from anything and was not fitted to data. It is stated here as an assumption.
What it controls. β sets the timescale over which "not accessed recently" becomes a reason to evict. In 1 − e−β·steps the characteristic time is 1/β = 1,000 decode steps, so a token reaches ~63% decay at 1,000 steps and ~86% at 2,000. At the canonical 10 tok/s per user that is about 100 seconds of wall-clock idleness before a token is treated as fully stale.
Which way it pushes. Raising β shortens that horizon and makes the policy more aggressively recency-driven: idle tokens are surrendered sooner, HBM and CXL DRAM are reclaimed faster, and long-lived-but-rarely-touched blocks — system prompts, few-shot exemplars, shared prefixes — start losing to whatever was touched last. (The "+8 pts from anchor pinning" rung this paragraph once cited is retracted: the 72→80→87→93→97 ladder was never run as an ablation, and its increments were assumed additive and independent.) Lowering β lengthens the horizon toward pure score-based eviction: the working set holds its shape longer and stale entries linger, consuming capacity. β → 0 removes the recency term entirely and leaves (1 − score) alone.
How it would be calibrated. β and α interact and must be swept together: α = 0.05 already gives a 13.5-step half-life on the score itself, so a β chosen without reference to α double-counts recency. The calibration is a two-dimensional sweep over real attention traces at 128 K context, scored on hit rate at fixed working-set size — and because α is non-monotonic, the sweep cannot be interpolated between grid points. See Appendix K, section K.7 — Uncalibrated parameters.
Illustrative — pedagogical worked example, never a result. α = 0.05, half-life 13.5 decode steps. Policy is LRFU (Lee et al., ACM SIGMETRICS 1999) — External literature. Hit-rate figures are Simulated (provisional): one data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. The scoring function is not the largest effect. At 32 GiB, letting identity-keyed policy state survive HBM eviction is worth +34.15 pts to LFU (15.56 → 49.71) and +24.85 pts to EMA α=0.01 (31.54 → 56.39), while EMA α=0.15 is unchanged (−0.21 pts) — larger than the whole spread between policies, and a protocol result, testable with no CXL hardware. Withdrawn from this figure: α = 0.2, the 97% hit rate, and the 72→80→87→93→97 ladder. Canonical numbers v6.0.