Figure: the LRFU score update and eviction-priority calculation worked end to end. The scoring rule is not new: a decayed-frequency score is LRFU — Lee et al., "On the Existence of a Spectrum of Policies that Subsumes the LRU and LFU Policies", ACM SIGMETRICS 1999 — and alpha is its documented LRU-to-LFU knob. The unit scored and evicted here is a (sequence, token block) of 16 tokens, 5.00 mebibytes, never a head: every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss. With alpha = 0.05, the simulated optimum, giving a half-life of ln0.5 over ln0.95 = 13.5 decode steps, score[t] = 0.05 x attention[t] + 0.95 x score[t-1] starting from an initial score of 0.0 for the block at position 1024. The five decode steps run: new score 0.45 gives 0.023; 0.12 gives 0.028; 0.30 gives 0.042; 0.08 gives 0.044; 0.35 gives 0.059. Eviction priority is (1 - score) x recency_decay, where recency_decay = 1 - exp(-beta x steps_since_access) with beta = 0.001. The coefficient beta = 0.001 is an uncalibrated design parameter, not a derived or measured value: it sets the timescale over which idleness becomes a reason to evict, a characteristic time of 1 over beta = 1,000 decode steps, about 100 seconds at an illustrative 10 tokens per second. Raising beta makes eviction more aggressively recency-driven and puts long-lived but rarely touched blocks such as system prompts at risk; lowering it lets stale entries linger and consume capacity. Beta interacts with alpha and must be swept jointly against real attention traces. Appendix K section K.7 lists it with the package's other uncalibrated parameters. Block A at position 1024, score 0.059 and accessed 50 steps ago, has recency_decay 0.049 and priority 0.046, so it is kept; block B at position 45678, score 0.02 and accessed 2000 steps ago, has recency_decay 0.865 and priority 0.848, so it is evicted. Higher priority means evict sooner. Alpha = 0.2 was never validated and is withdrawn; in simulation alpha is non-monotonic with an interior optimum near 0.05, and there is no convergence onto LFU as alpha approaches zero. The scoring rule is also not where the largest effect lies: at a 32 gibibyte budget, letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU and plus 24.85 to EMA alpha 0.01, larger than the whole spread between scoring functions. When this block is evicted, its score and identity key travel with it and are restored on re-admission, rather than being reset. The worked numbers here are illustrative, never a result; the simulated figures are provisional, because a single data-structure defect in Revision 1 inverted four of five findings.
LRFU Score + Eviction Priority
Step-by-step calculation, per (sequence, token block). The policy is LRFU (Lee et al., ACM SIGMETRICS 1999) — established prior art, not a new algorithm.
16 tokens = 5.00 MiB. Every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss. H2O and SnapKV evict (head, token) pairs — the token axis is not optional.
# α = 0.05 (simulated optimum, not a derived constant) → half-life ln0.5/ln0.95 = 13.5 decode steps
# α = 0.2 was never validated and is withdrawn. α is non-monotonic:
# 32 GiB: 0.5→37.20 0.3→38.84 0.15→44.33 0.05→55.96 0.01→31.54 0.003→25.47 0.001→24.44
# No convergence onto LFU as α→0. [Simulated (provisional)]
| Step | new_score | Calculation | score_ema |
|---|---|---|---|
| t₁ | 0.45 | 0.05 × 0.45 + 0.95 × 0.000 | 0.023 |
| t₂ | 0.12 | 0.05 × 0.12 + 0.95 × 0.023 | 0.028 |
| t₃ | 0.30 | 0.05 × 0.30 + 0.95 × 0.028 | 0.042 |
| t₄ | 0.08 | 0.05 × 0.08 + 0.95 × 0.042 | 0.044 |
| t₅ | 0.35 | 0.05 × 0.35 + 0.95 × 0.044 | 0.059 |
# Higher priority = evict sooner
# β = 0.001 (uncalibrated design parameter), decays over ~1000 steps
The recency-decay coefficient is not derived from anything and was not fitted to data. It is stated here as an assumption.
What it controls. β sets the timescale over which "not accessed recently" becomes a reason to evict. In 1 − e−β·steps the characteristic time is 1/β = 1,000 decode steps, so a token reaches ~63% decay at 1,000 steps and ~86% at 2,000. At the canonical 10 tok/s per user that is about 100 seconds of wall-clock idleness before a token is treated as fully stale.
Which way it pushes. Raising β shortens that horizon and makes the policy more aggressively recency-driven: idle tokens are surrendered sooner, HBM and CXL DRAM are reclaimed faster, and long-lived-but-rarely-touched blocks — system prompts, few-shot exemplars, shared prefixes — start losing to whatever was touched last. (The "+8 pts from anchor pinning" rung that this paragraph once cited is retracted: the 72→80→87→93→97 ladder was never run as an ablation, and its increments were assumed additive and independent.) Lowering β lengthens the horizon toward pure score-based eviction: the working set holds its shape longer and stale entries linger, consuming capacity. β → 0 removes the recency term entirely and leaves (1 − score) alone.
How it would be calibrated. β and α interact and must be swept together: α = 0.05 already gives a 13.5-step half-life on the score itself, so a β chosen without reference to α double-counts recency. The calibration is a two-dimensional sweep over real attention traces at 128 K context, scored on hit rate at fixed working-set size — and α is non-monotonic, so the sweep cannot be interpolated. See Appendix K, section K.7 — Uncalibrated parameters.
Illustrative — pedagogical worked example, never a result. α = 0.05, half-life 13.5 decode steps. Policy is LRFU (Lee et al., ACM SIGMETRICS 1999) — External literature. Hit-rate figures quoted on this page are Simulated (provisional): a single data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. Withdrawn from this figure: α = 0.2, the 97% hit rate and the 72→80→87→93→97 ladder. Canonical numbers v6.0.