Figure: attention-aware cache eviction in five steps, tracking the token at position 1024. Raw attention over five decode steps is noisy — 0.45, 0.12, 0.30, 0.08, 0.35 — so it is smoothed with score_ema = alpha x new_score + (1 - alpha) x score_ema. At the default alpha = 0.2, half-life ln0.5 over ln0.8 = 3.1 decode steps, the sequence becomes 0.090, 0.096, 0.137, 0.126, 0.171, against a raw average of 0.26. Smaller alphas trust history longer: half-life 6.6 steps at alpha 0.1 and 13.5 steps at alpha 0.05. Eviction priority is (1 - score_ema) x recency_decay with recency_decay = 1 - e to the minus beta times steps and beta = 0.001, about 63 percent decay at 1000 steps. The coefficient beta = 0.001 is an uncalibrated design parameter, not a derived or measured value: it sets the timescale over which idleness becomes a reason to evict, a characteristic time of 1 over beta = 1,000 decode steps, about 100 seconds at the canonical 10 tokens per second. Raising beta makes eviction more aggressively recency-driven and puts long-lived but rarely touched tokens such as system prompts at risk; lowering it lets stale entries linger and consume capacity. Beta interacts with alpha and must be swept jointly against real attention traces. Appendix K section K.7 lists it with the package's other uncalibrated parameters. Token A at position 1024, score_ema 0.171, accessed 50 steps ago, gives (1 - 0.171) x 0.049 = 0.041 and is kept; token B at position 45678, score_ema 0.08, accessed 2000 steps ago, gives (1 - 0.08) x 0.865 = 0.796 and is evicted. The decision matrix crosses score against recency: high score and recent is keep, as with a recent context token at ema 0.85, 20 steps, p 0.003; high score and old is watch, a system prompt token at ema 0.72, 3000 steps, p 0.266; low score and recent is watch, new user input at ema 0.05, 10 steps, p 0.009; low score and old is evict, the filler word "the" at ema 0.03, 5000 steps, p 0.963. All values are an analytical model, not measured.
Attention-Aware Cache Eviction
Using EMA to transform noisy attention into stable eviction priorities
Each decode step, every cached token receives an attention weight from the current query. These scores are noisy—a token might spike one step and drop the next.
| Step | new_score | Calculation | score_ema |
|---|---|---|---|
| t₁ | 0.45 | 0.2 × 0.45 + 0.8 × 0.000 | 0.090 |
| t₂ | 0.12 | 0.2 × 0.12 + 0.8 × 0.090 | 0.096 |
| t₃ | 0.30 | 0.2 × 0.30 + 0.8 × 0.096 | 0.137 |
| t₄ | 0.08 | 0.2 × 0.08 + 0.8 × 0.137 | 0.126 |
| t₅ | 0.35 | 0.2 × 0.35 + 0.8 × 0.126 | 0.171 |
The recency-decay coefficient is not derived from anything and was not fitted to data. It is stated here as an assumption.
What it controls. β sets the timescale over which "not accessed recently" becomes a reason to evict. In 1 − e−β·steps the characteristic time is 1/β = 1,000 decode steps, so a token reaches ~63% decay at 1,000 steps and ~86% at 2,000. At the canonical 10 tok/s per user that is about 100 seconds of wall-clock idleness before a token is treated as fully stale.
Which way it pushes. Raising β shortens that horizon and makes the policy more aggressively recency-driven: idle tokens are surrendered sooner, HBM and CXL DRAM are reclaimed faster, and long-lived-but-rarely-touched entries — system prompts, few-shot exemplars, anchor tokens — start losing to whatever was touched last, which is exactly the failure the +8-point anchor-pinning rung exists to prevent. Lowering β lengthens the horizon toward pure score-based eviction: the hot set holds its shape longer and stale entries linger, consuming capacity that admission of a new user would need. β → 0 removes the recency term entirely and leaves (1 − score_ema) alone.
How it would be calibrated. β and α interact and must be swept together: α = 0.2 already gives a 3.1-step half-life on the score itself, so a β chosen without reference to α double-counts recency. The calibration is a two-dimensional sweep over real per-head attention traces at the canonical 128 K context, scored on hit rate at fixed hot-set size. See Appendix K, section K.7 — Uncalibrated parameters.
Analytical model — illustrative worked example, not measured data. α = 0.2, half-life 3.1 decode steps. Canonical numbers v4.0.