Figure: LRFU decayed-frequency scoring, worked through as an exponential moving average, comparing how an important and an unimportant block score over 100 decode steps under score_new = 0.05 x attention + 0.95 x score_old, that is alpha = 0.05. This is not a new algorithm. A decayed-frequency score is LRFU — Lee et al., "On the Existence of a Spectrum of Policies that Subsumes the LRU and LFU Policies", ACM SIGMETRICS 1999 — established prior art, and alpha is its documented knob between LRU-like and LFU-like behaviour. The block holding the instruction token "helpful" at position 45 runs 0.0020, 0.0044, 0.0170, 0.0389 and 0.0420 at steps 1, 2, 10, 50 and 100, ending at 0.0420 and kept; the block holding the filler token "the" at position 5,432 runs 0.0001, 0.0002, 0.0004, 0.0009 and 0.0010 over the same steps, ending at 0.0010 and evicted. The half-life is (1 - alpha)^n = 0.5, so 0.95^n = 0.5 and n = log 0.5 over log 0.95 = 13.5 steps, which at the illustrative 10 tokens per second per user is about 1.35 seconds. Scores above 0.03 are kept in HBM, 0.005 to 0.03 are demoted to the host or CXL DRAM tier, and below 0.005 are evicted to NVMe or discarded. Those two thresholds are uncalibrated design parameters, not derived or measured values, and their absolute scale is tied to alpha; they are restated here at the alpha = 0.05 scale. Alpha = 0.2 was never validated and is withdrawn. In simulation alpha is non-monotonic with an interior optimum at about 0.05: at a 32 gibibyte budget the hit rates are 37.20 percent at alpha 0.5, 38.84 at 0.3, 44.33 at 0.15, 55.96 at 0.05, 31.54 at 0.01, 25.47 at 0.003 and 24.44 at 0.001, and there is no convergence onto LFU as alpha approaches zero. The scoring function is also not where the largest effect lies: at the same budget, letting identity-keyed policy state survive HBM eviction is worth plus 34.15 percentage points to LFU and plus 24.85 to EMA alpha 0.01, more than the whole spread between scoring functions. The worked numbers on this page are illustrative; the simulated hit rates are provisional, because a single data-structure defect in Revision 1 inverted four of five findings. No single hit rate is claimed anywhere: the simulated range is 8.5 to 91.6 percent across budgets, policies and workloads.

LRFU Decayed-Frequency Scoring

Established prior art, worked through — not a new algorithm. How a block's score tracks sustained attention rather than mere recency.

This is LRFU, not a new algorithm — External literature

A decayed-frequency score is LRFU — Lee et al., “On the Existence of a Spectrum of Policies that Subsumes the LRU and LFU Policies”, ACM SIGMETRICS 1999. α is its documented LRU↔LFU knob. This package contributes the worked example and the sweep, not the policy.

The Update Rule
score_new = 0.05 × attention + 0.95 × score_old
Each decode step, the score is updated as an exponential moving average — the LRFU spectrum written recursively.
Blocks that consistently receive attention accumulate high scores.
Blocks that are ignored see their scores decay toward zero.
Token: "helpful" (instruction)
Position 45
Step 1
0.05 × 0.040 + 0.95 × 0
0.0020
Step 2
0.05 × 0.050 + 0.95 × 0.0020
0.0044
Step 10
0.05 × 0.030 + 0.95 × 0.0163
0.0170
Step 50
0.05 × 0.040 + 0.95 × 0.0388
0.0389
Step 100
0.05 × 0.050 + 0.95 × 0.0416
0.0420
Final Score after 100 steps
Token: "the" (filler word)
Position 5,432
Step 1
0.05 × 0.0020 + 0.95 × 0
0.0001
Step 2
0.05 × 0.0010 + 0.95 × 0.0001
0.0002
Step 10
0.05 × 0.0010 + 0.95 × 0.0004
0.0004
Step 50
0.05 × 0.0000 + 0.95 × 0.0010
0.0009
Step 100
0.05 × 0.0010 + 0.95 × 0.0010
0.0010
Final Score after 100 steps
0.0010 → EVICT
Why α = 0.05? The Half-Life Calculation
(1 - α)^n = 0.5
0.95^n = 0.5
n = log(0.5) / log(0.95)
n = 13.5 steps
At an illustrative 10 tokens/second per user, the half-life is about 1.35 seconds.

If a block receives no attention for 1.35 s, its score halves. Blocks that are consistently important hold high scores; blocks briefly touched and then ignored decay away.

α = 0.05 is the simulated optimum, not a derived constant. α = 0.2 was never validated and is withdrawn. In the sweep at 32 GiB, α is non-monotonic: 0.5→37.20, 0.3→38.84, 0.15→44.33, 0.05→55.96, 0.01→31.54, 0.003→25.47, 0.001→24.44. There is no convergence onto LFU as α→0. Simulated (provisional)
Eviction Decision Thresholds
0.03 and 0.005 are uncalibrated design parameters

Neither threshold is derived, measured or fitted. They are placeholders chosen to make the three-tier decision legible, and they are stated here as assumptions so that a reader can argue with them rather than inherit them. They have been rescaled to α = 0.05; the earlier 0.10 / 0.02 pair belonged to the withdrawn α = 0.2 and does not carry over.

What they control. The keep threshold (0.03) sets the size of the HBM-resident working set: it is the score a block must hold to stay in HBM. The evict threshold (0.005) sets the floor below which a block is pushed off the DRAM tier to NVMe. The band between them is the demote zone, and its width sets how much traffic the middle tier absorbs.

Which way they push. Raising the keep threshold frees HBM sooner, at the cost of hit rate — more accesses fall through to a tier whose latency is assumed, not measured (32 / 64 / 121 GB/s → 164 / 82 / 43 µs for a 5.00 MiB block). Lowering it holds more blocks in HBM and raises hit rate at the cost of capacity. Raising the evict threshold reclaims DRAM faster and shortens metadata state, at the cost of a longer miss tail on NVMe; lowering it keeps more of the context warm and moves capacity pressure onto the DRAM tier. Widening the demote band trades HBM pressure for tier pressure; narrowing it makes the policy behave more like a two-tier keep/evict decision. What is not claimed is any particular hit rate at any setting — no single hit rate exists. Across budgets, policies and workloads the simulated range is 8.5–91.6%. Simulated (provisional)

Threshold tuning is not where the leverage is. At a 32 GiB budget the whole spread between scoring functions is smaller than one protocol change: letting identity-keyed policy state survive HBM eviction is worth +34.15 pts to LFU (15.56 → 49.71) and +24.85 pts to EMA α=0.01 (31.54 → 56.39), while EMA α=0.15 is unchanged (−0.21 pts). Tune the protocol before tuning α.

How they would be calibrated. Only against real attention traces. Replay attention distributions from a production 128 K workload, sweep the pair, and pick the point that meets a stated service objective at the smallest working set. The correct values almost certainly differ per model. Note also that the absolute scale of these thresholds is tied to α: an EMA lags its input, so a threshold of 0.03 is not "3% of attention". Changing α changes what these numbers mean — and α is non-monotonic, so there is no safe interpolation between settings. See Appendix K, section K.7 — Uncalibrated parameters for the consolidated table.

Illustrative — the worked scores on this page are pedagogical, never a result. α = 0.05, half-life ln0.5/ln0.95 = 13.5 decode steps. The policy is LRFU (Lee et al., ACM SIGMETRICS 1999) — External literature. Hit-rate figures quoted here are Simulated (provisional): a single data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. Retracted and removed from this figure: the 97% hit rate, the 72→80→87→93→97 ladder and its +8/+7/+6/+4 increments (never run as an ablation), the "+7 pts from EMA scoring" step, α = 0.2, and the 200 ns / 177 ns latency figures — no CXL hardware was measured. Canonical numbers v6.0.