Figure: RoPE-aware prefetching. RoPE rotates Q and K in d/2 two-dimensional subspaces, each with its own frequency, so the inner product is a sum over those pairs and does not fall off monotonically; the guarantee is an upper-bound envelope, the magnitude of the inner product of q_m and k_n is at most B of the absolute distance m minus n with B non-increasing. This design assumes about 70 percent of attention mass falls within plus or minus W positions of the query, an input assumption rather than a universal property. The prefetch rule is: when the GPU accesses position P, prefetch the range P minus W to P plus W, ordering candidates by priority(distance) = 1 divided by (1 plus distance over 100). In the worked strip, current access is position 55 with W = 5, so positions 50 to 54 and 56 to 60 are in the prefetch zone while 48, 49, 61 and 62 stay on storage. Window size is chosen from the assumed distribution: W = 32 captures about 50 percent of attention with low bandwidth but more misses; W = 128 captures about 75 percent and is the balanced choice for Llama-70B; W = 512 captures about 90 percent at high bandwidth overhead. All three window sizes are uncalibrated design parameters, not derived or measured values: they are round powers of two chosen to show the shape of the trade-off. W sets the radius of the prefetch range and so fixes both attention coverage and speculative bandwidth. Raising W lifts coverage but cost grows linearly while return is sharply sublinear, and prefetched entries that are never read waste both CXL bandwidth and HBM residency; lowering W spends less but leaves more of the miss tail to be paid at 200 nanoseconds or 25 microseconds. Calibration requires a measured per-head attention-versus-distance distribution on production traces, and the right W is likely per head type rather than global. Appendix K section K.7 lists it with the package's other uncalibrated parameters. Note also that this prefetch priority is a different function from the tier-placement score P(p) = 0.25 R + 0.55 E + 0.20 N used in the executive summary visual appendix: prefetch priority decides what to fetch next, tier placement decides where a token already held should live. RoPE-aware prefetch contributes plus 4 percentage points of hit rate, 93 percent to 97 percent cumulative. All values are an analytical model, not measured.

RoPE-Aware Prefetching

Exploiting position encoding locality for predictive cache loading

1
RoPE Creates Distance-Dependent Attention
Attention Relative Position (distance from query)
Query Position
Rotary Encoding — what RoPE actually gives
RoPE encodes position by rotating Q/K in d/2 two-dimensional subspaces, each with its own frequency θ_i. The inner product is a sum over those d/2 frequency pairs, so it is not a single cosine of the distance and it does not fall off monotonically. The guarantee is an upper-bound envelope: |⟨q_m, k_n⟩| ≤ B(|m − n|) with B non-increasing. The curve at left sketches that envelope, not a measured attention trace.
Locality Bias — an ASSUMPTION of this model
This design assumes roughly 70% of attention mass falls within ±W positions of the query. That figure is not a universal property of transformers: it is model-, layer-, head- and workload-dependent, and it is stated here as an input assumption, not as an established fact or a measurement.
Predictable Access
Under that assumption, if the GPU requests position P it will likely need P±W soon, so we prefetch proactively. The priority actually implemented is 1 / (1 + distance/100) — a hyperbolic falloff in position distance, not the RoPE envelope itself.
2
Prefetch Window Strategy
Prefetch Rule
GPU accesses P → Prefetch [P − W, P + W]
priority(distance) = 1 / (1 + distance / 100)
W = window size, chosen per model from the assumed attention distribution. Candidates inside the window are ordered by the priority above (the reference implementation's rule).
Prefetch Window (2W)
P
Current Access
P−W P+W
P−2W
P−W
P
P+W
P+2W
Current Access (P=55)
Prefetch Zone (W=5)
On Storage
3
Window Size Selection (modeled, not empirical)
Assumed attention mass distribution (illustrative, not a measured histogram) · Green = within prefetch window
W = 32
~50% attention captured (assumed)
Low bandwidth, more misses
W = 128
~75% attention captured (assumed)
Balanced for Llama-70B
W = 512
~90% attention captured (assumed)
High bandwidth overhead
W = 32 / 128 / 512 are uncalibrated design parameters

These three window sizes are not derived and were not selected by experiment. They are round powers of two spanning two orders of magnitude, chosen to show the shape of the trade-off; the coverage percentages beside them (~50% / ~75% / ~90%) are assumptions of the model, not a measured histogram, and W = 128 is labelled "balanced for Llama-70B" as a judgement call, not a fitted result.

What W controls. W sets the radius of the prefetch range [P − W, P + W] around an accessed position, so 2W + 1 entries are fetched speculatively per access. It therefore fixes the fraction of attention mass the prefetcher can cover and, at the same time, the speculative bandwidth the policy spends to get it.

Which way it pushes. Raising W widens coverage and lifts the hit rate — this is the rung worth +4 percentage points, 93% → 97% — but the cost is linear in W while the return is sharply sublinear, because the assumed attention distribution thins out with distance. Every prefetched entry that is never read is wasted CXL bandwidth and wasted HBM residency, so a large W eats into the hot set it was meant to serve and can lower the effective hit rate. Lowering W spends less bandwidth and leaves more HBM for demand-loaded tokens, but leaves more of the miss tail to be paid at 200 ns (CXL) or 25 µs (flash). At W = 0 the mechanism disappears and the ladder falls back to 93%.

How it would be calibrated. Measure the real per-head attention-versus-distance distribution on production traces for the target model, then choose the smallest W whose marginal coverage still exceeds the marginal bandwidth cost. The right W is very likely per-head-type rather than global: the head-specialisation split in Appendix E implies local heads want a small W and retrieval heads are poorly served by a distance window at any W. See Appendix K, section K.7 — Uncalibrated parameters.

Not the same function as the tier-placement weights. The ordering rule inside the window, priority(distance) = 1/(1 + distance/100), and the 0.6 · rope_factor + 0.4 · ema_factor blend in Appendices D and J are prefetch priorities — they decide what to fetch next. The P(p) = 0.25·R + 0.55·E + 0.20·N weights in the executive-summary visual appendix are a tier-placement score — they decide where a token already held should live. Different questions, different functions; the weights are not comparable and do not contradict each other.

Analytical model — the locality percentages and window-coverage figures on this page are assumptions of the model, not measurements. RoPE-aware prefetch contributes +4 percentage points of hit rate (93% → 97% cumulative). Canonical numbers v4.0.