Figure: why attention has locality, the RoPE effect. For a token generated at position 1000 looking back, the assumed attention distribution is 45 percent to the last 10 tokens at positions 990 to 999, 28 percent to tokens 11 to 100 at positions 900 to 989, 18 percent to tokens 101 to 500 at positions 500 to 899, 7 percent to tokens 501 to 950 at positions 50 to 499, and 2 percent to the first 50 tokens at positions 0 to 49. So 45 plus 28 equals 73 percent of attention falls on the last 10 percent of tokens. The bound behind this is that the magnitude of the inner product of q_m and k_n is at most B of the absolute distance m minus n, where B is a non-increasing envelope, not a monotonic cosine decay. The caching consequence is a modeled 97 / 2.7 / 0.3 split across HBM at 100 nanoseconds, CXL DRAM at 200 nanoseconds and flash at 25 microseconds, giving an effective access latency of 177 nanoseconds. All values are an analytical model, not measured.

Why Attention Has Locality — The RoPE Effect

Understanding why recent tokens receive more attention than distant ones

Attention Pattern: Token at Position 1000 Looking Back

Attention weights when generating the 1000th token:

Last 10 tokens
pos 990-999
Tokens 11-100
pos 900-989
Tokens 101-500
pos 500-899
Tokens 501-950
pos 50-499
First 50 tokens
pos 0-49
Key Observation
45% + 28% = 73% of attention goes to the last 10% of tokens (pos 900–999)
Assumed distribution, not a measurement. The concentration is attributed to positional encoding (RoPE) plus recency structure in the workload; the exact split is model-, layer-, head- and workload-dependent.
How RoPE Creates Locality
Distance from current token Attention d=10 d=100 d=500 d=1000
|⟨q_m, k_n⟩| ≤ B(|m − n|)
m = query position, n = key position
B = a non-increasing upper-bound envelope in distance

The RoPE inner product is a sum over d/2 frequency pairs, each rotated by its own θ_i. It is not proportional to a single cos((m − n)θ), and it does not decay monotonically with distance.

What the construction gives is a bound: the magnitude of the inner product is capped by an envelope that does not increase with |m − n|. The curve at left sketches that envelope, not an actual attention trace.

The Caching Implication

Since 73% of attention goes to the most recent ~10% of tokens in this model, we can keep recent tokens in fast HBM memory and move older tokens to slower CXL memory. Most accesses will hit the fast tier, keeping average latency low: the modeled 97 / 2.7 / 0.3 split across HBM (100 ns) / CXL DRAM (200 ns) / flash (25 µs) gives an effective access latency of 177 ns.

Analytical model — the attention distribution shown is an assumption of the model, not a measured trace. No hardware benchmark was run. Canonical numbers v4.0.