Figure: why attention has locality, the RoPE effect. For a token generated at position 1000 looking back, the assumed attention distribution is 45 percent to the last 10 tokens at positions 990 to 999, 28 percent to tokens 11 to 100 at positions 900 to 989, 18 percent to tokens 101 to 500 at positions 500 to 899, 7 percent to tokens 501 to 950 at positions 50 to 499, and 2 percent to the first 50 tokens at positions 0 to 49. So 45 plus 28 equals 73 percent of attention falls on the last 10 percent of tokens. The bound behind this is that the magnitude of the inner product of q_m and k_n is at most B of the absolute distance m minus n, where B is a non-increasing envelope, not a monotonic cosine decay. The caching consequence claimed earlier on this figure — a modelled 97 / 2.7 / 0.3 tier split across HBM at 100 nanoseconds, CXL DRAM at 200 nanoseconds and flash at 25 microseconds, giving an effective access latency of 177 nanoseconds — is retracted in full. It rested on the 97 percent hit rate, which was never measured, and on a 200 nanosecond CXL access that no hardware ever produced: no CXL hardware was measured anywhere in this package. No single hit rate exists; across budgets, policies and workloads the simulated range is 8.5 to 91.6 percent. Tier latency is now an assumption across 32, 64 and 121 gigabytes per second, under which one 5.00 mebibyte block of 16 tokens transfers in 164, 82 or 43 microseconds; the tier's value is avoiding regeneration, since forward compute for the same 16 tokens is 9, 11 or 18 milliseconds at 50, 40 or 25 percent model FLOPs utilisation — two to three orders of magnitude, with no precise multiple claimed. Two further cautions apply to the locality argument itself. First, the attention distribution shown is an assumption, not a measured trace, and it is model-, layer-, head- and workload-dependent. Second, attention sparsity and cache hit rate are different claims with different obligations: the earlier reading of a 5.4 percent hot set as capturing 97 percent of accesses conflated them and is withdrawn. Lossless residency requires every block that is read at a step to be present, and under dense attention every block of an active sequence is read every step; treating sparsity as a substitute is a lossy relaxation that carries an accuracy-evaluation burden. Simulation values are provisional, because a single data-structure defect in Revision 1 inverted four of five findings.

Why Attention Has Locality — The RoPE Effect

Understanding why recent tokens receive more attention than distant ones

Attention Pattern: Token at Position 1000 Looking Back

Attention weights when generating the 1000th token:

Last 10 tokens
pos 990-999
Tokens 11-100
pos 900-989
Tokens 101-500
pos 500-899
Tokens 501-950
pos 50-499
First 50 tokens
pos 0-49
Key Observation
45% + 28% = 73% of attention goes to the last 10% of tokens (pos 900–999)
Assumed distribution, not a measurement. The concentration is attributed to positional encoding (RoPE) plus recency structure in the workload; the exact split is model-, layer-, head- and workload-dependent.
How RoPE Creates Locality
Distance from current token Attention d=10 d=100 d=500 d=1000
|⟨q_m, k_n⟩| ≤ B(|m − n|)
m = query position, n = key position
B = a non-increasing upper-bound envelope in distance

The RoPE inner product is a sum over d/2 frequency pairs, each rotated by its own θ_i. It is not proportional to a single cos((m − n)θ), and it does not decay monotonically with distance.

What the construction gives is a bound: the magnitude of the inner product is capped by an envelope that does not increase with |m − n|. The curve at left sketches that envelope, not an actual attention trace.

The Caching Implication — narrower than it looks

In this assumed distribution, 73% of attention falls on the most recent ~10% of tokens. That motivates keeping recent blocks in HBM and placing older ones on a slower tier — but it does not license a hit-rate number, and the two are not the same claim.

Retracted: the modelled 97 / 2.7 / 0.3 tier split across HBM (100 ns) / CXL DRAM (200 ns) / flash (25 µs), and the 177 ns effective access latency derived from it. Both rested on a 97% hit rate that was never measured and on a 200 ns CXL access that no hardware ever produced — no CXL hardware was measured. Also retracted: reading a 5.4% hot set as capturing 97% of accesses, which conflated attention sparsity with cache hit rate.

What replaces it. No single hit rate exists — 8.5–91.6% across budgets, policies and workloads Simulated (provisional). Tier latency is assumed across 32 / 64 / 121 GB/s: one 5.00 MiB block is 164 / 82 / 43 µs. The tier's value is avoiding regeneration — forward compute for the same 16 tokens is 9 / 11 / 18 ms at 50 / 40 / 25% MFU, two to three orders of magnitude, no precise multiple claimed.

Sparsity is not residency. Lossless residency requires every block in Read(t) to be present, and under dense attention every block of an active sequence is read every step. Exploiting sparsity instead is a lossy relaxation with an accuracy-evaluation burden attached.

Analytical model for the RoPE bound |⟨q_m, k_n⟩| ≤ B(|m − n|); Illustrative for the attention distribution, which is an assumption, not a measured trace. No CXL hardware was measured. Hit-rate figures are Simulated (provisional): one data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. Retracted here: 97%, the 97 / 2.7 / 0.3 tier split, 200 ns, 177 ns, 25 µs, and the 5.4% hot set / 97% of accesses claim. Canonical numbers v6.0.