Figure: EMA-based attention scoring, comparing how an important and an unimportant token score over 100 decode steps under score_new = 0.2 x attention + 0.8 x score_old, that is alpha = 0.2. The instruction token "helpful" at position 45 runs 0.008, 0.016, 0.040, 0.096 and 0.122 at steps 1, 2, 10, 50 and 100, ending at 0.122 and kept; the filler token "the" at position 5,432 runs 0.0004, 0.0005, 0.0008, 0.0005 and 0.0005 over the same steps, ending at 0.0005 and evicted. The half-life is (1 - alpha)^n = 0.5, so 0.8^n = 0.5 and n = log 0.5 over log 0.8 = 3.1 steps, which at the canonical 10 tokens per second per user is 310 milliseconds. Scores above 0.10 are kept in fast HBM, 0.02 to 0.10 are demoted to CXL DRAM, and below 0.02 are evicted to Flash or discarded. Those two thresholds are uncalibrated design parameters, not derived or measured values: the keep threshold sets the size of the HBM-resident hot set and the evict threshold sets the floor below which a token is pushed to flash. Raising the keep threshold frees HBM sooner and admits more users at the cost of hit rate; raising the evict threshold reclaims CXL DRAM faster at the cost of more 25 microsecond flash reads. Both need tuning against real attention traces, and their absolute scale is tied to alpha = 0.2. Appendix K section K.7 lists them with the package's other uncalibrated parameters. EMA scoring contributes plus 7 percentage points of hit rate, 80 percent to 87 percent cumulative. All values are an analytical model, not measured.

EMA-Based Attention Scoring

How we track which tokens are actually important (not just recent)

The EMA Update Rule
score_new = 0.2 × attention + 0.8 × score_old
Each decode step, we update the score using exponential moving average.
Tokens that consistently receive attention accumulate high scores.
Tokens that are ignored see their scores decay toward zero.
Token: "helpful" (instruction)
Position 45
Step 1
0.2 × 0.04 + 0.8 × 0
0.008
Step 2
0.2 × 0.05 + 0.8 × 0.008
0.016
Step 10
0.2 × 0.03 + 0.8 × 0.042
0.040
Step 50
0.2 × 0.04 + 0.8 × 0.11
0.096
Step 100
0.2 × 0.05 + 0.8 × 0.14
0.122
Final Score after 100 steps
Token: "the" (filler word)
Position 5,432
Step 1
0.2 × 0.002 + 0.8 × 0
0.0004
Step 2
0.2 × 0.001 + 0.8 × 0.0004
0.0005
Step 10
0.2 × 0.001 + 0.8 × 0.0008
0.0008
Step 50
0.2 × 0.000 + 0.8 × 0.0006
0.0005
Step 100
0.2 × 0.001 + 0.8 × 0.0004
0.0005
Final Score after 100 steps
0.0005 → EVICT
Why α = 0.2? The Half-Life Calculation
(1 - α)^n = 0.5
0.8^n = 0.5
n = log(0.5) / log(0.8)
n = 3.1 steps
At the canonical target of 10 tokens/second per user, the half-life is 310 milliseconds.

This means: if a token doesn't receive attention for 310 ms, its score drops by 50%. Tokens that are consistently important maintain high scores; tokens that were briefly accessed but then ignored see their scores decay quickly.
Eviction Decision Thresholds
0.10 and 0.02 are uncalibrated design parameters

Neither threshold is derived, measured or fitted. They are placeholders chosen to make the three-tier decision legible, and they are stated here as assumptions so that a reader can argue with them rather than inherit them.

What they control. The keep threshold (0.10) sets the size of the HBM-resident hot set: it is the score a token must hold to occupy Tier 0/1. The evict threshold (0.02) sets the floor below which a token is pushed out of CXL DRAM to flash. The band between them is the demote zone, and its width sets how much traffic the CXL tier absorbs.

Which way they push. Raising the keep threshold frees HBM sooner and admits more users, at the cost of hit rate — more accesses fall through to the 200 ns CXL tier, pulling the effective-latency figure up from 177 ns. Lowering it holds more tokens in HBM and raises hit rate, but shrinks the number of concurrent users the 37 GB evictable budget can serve. Raising the evict threshold reclaims CXL DRAM faster and shortens metadata queues, at the cost of more 25 µs flash reads on the miss tail; lowering it keeps more of the context warm and moves capacity pressure back onto the 1 TB CXL tier. Widening the demote band trades HBM pressure for CXL pressure; narrowing it makes the policy behave more like a two-tier keep/evict decision.

How they would be calibrated. Only against real attention traces. Replay per-head attention distributions from a production 128 K workload, sweep the pair, and pick the point that meets the 97% hit-rate SLO at the smallest hot set. The correct values almost certainly differ per head type and per model. Note also that the absolute scale of these thresholds is tied to α = 0.2: an EMA lags its input and saturates well below the raw attention mass, so a threshold of 0.10 is not "10% of attention". Changing α changes what these numbers mean. See Appendix K, section K.7 — Uncalibrated parameters for the consolidated table.

Analytical model — worked scores are illustrative, not measured. α = 0.2, half-life ln0.5/ln0.8 = 3.1 decode steps. EMA scoring contributes +7 percentage points of hit rate (80% → 87% cumulative). Canonical numbers v4.0.