Figure: a worked 16-token example of RoPE-aware prefetching. The scenario is a 16-token sequence, a GPU cache of 8 KV slots, a prefetch window of W = 3 meaning plus or minus 3 positions, and a tier-3 flash round-trip of 25 microseconds; candidates are ordered by priority(distance) = 1 divided by (1 plus distance over 100). Querying position 8, the token "dog", is assumed to draw about 72 percent of attention mass onto positions 5 to 11. The timeline runs: at 0 microseconds a cache miss on position 8 triggers the window [5, 11]; at 25 microseconds storage returns position 8 while positions 5, 6, 7, 9, 10 and 11 load asynchronously; at 26 microseconds position 7 hits; at 27 microseconds positions 6, 9 and 10 hit; at 50 microseconds the next query at position 12 opens window [9, 15], keeping 9, 10 and 11, evicting 5 to 8, and fetching only 12 to 15. Against a naive scheme of 16 serial round-trips at 25 microseconds, or 400 microseconds, prefetching moves 11 positions in 2 batched round-trips, or 50 microseconds — an 8.0 times reduction in this toy walkthrough. The model behind that comparison is stated in full on the page. Both paths service the same two queries, at positions 8 and 12, with every entry starting on tier 3 flash at 25 microseconds. The naive path has no prefetcher, so it discovers each position only when attention asks for it and cannot batch: 16 positions, 16 strictly serial round-trips, 16 times 25 equals 400 microseconds. The prefetcher knows the whole window from the query position, so the query at position 8 issues window 5 to 11, seven positions, as one batched round-trip completing at 25 microseconds, and the query at position 12 issues window 9 to 15, of which 9, 10 and 11 are already resident, so only 12 to 15 are fetched in a second batched round-trip completing at 50 microseconds: 2 times 25 equals 50 microseconds, and 400 divided by 50 is 8.0. The 8.0 times combines batching, which collapses 16 round-trips into 2, with selectivity, which never touches positions 0 to 4; only the batching effect generalises. The fully serial baseline is deliberately pessimistic, so 8.0 times is an upper bound on what prefetch buys in this toy configuration, not a system-level speedup. It is also not the plus 4 percentage point hit-rate contribution in a different unit, and not a latency figure for the proposed system, where steady-state accesses land in HBM at 100 nanoseconds or CXL DRAM at 200 nanoseconds and only the 0.3 percent miss tail reaches the 25 microsecond tier. The modeled full-stack HBM hit rate is 97 percent along the ladder 72 to 80 to 87 to 93 to 97, of which RoPE-aware prefetch alone contributes plus 4 percentage points, 93 to 97. All values are an analytical model, not measured.
RoPE Prefetch: Worked Example
Step-by-step walkthrough of locality-aware KV-cache prefetching
Assumption, not measurement. This walkthrough assumes attention concentrates
around the query position, so that positions 5–11 (within ±3 of query P=8) capture
~72% of total attention mass. The concentration is
model-, layer-, head- and workload-dependent and is an input to the model, not a result of it.
What RoPE itself provides is an upper-bound envelope on the inner-product magnitude,
|⟨q_m, k_n⟩| ≤ B(|m − n|) with B non-increasing — the inner product is a sum over
d/2 frequency pairs and does not decay monotonically.
Trigger prefetch for window [5, 11].
Positions 9,10,11 already cached → only fetch 12,13,14,15.
This comparison is a stated model, not a measurement. Both columns service the same two queries in the timeline above — P = 8, then P = 12 — over the same 16-token sequence, with every entry starting on Tier 3 flash at the canonical 25 µs round-trip latency. Nothing else in the two paths differs.
Why 16 round-trips. The naive path has no prefetcher, so it discovers each position it needs only at the moment attention asks for it. Each discovery is a separate demand miss, and the next request is not issued until the previous one returns — there is nothing to batch, because the requesting code does not yet know what it will want next. Sixteen positions, sixteen strictly serial round-trips: 16 × 25 µs = 400 µs.
Why 2 batches. The prefetcher knows the whole window from the query position alone, so all of it can be issued at once and the round-trips overlap. The query at P = 8 opens [5, 11] — 7 positions — issued as one batched round-trip that completes at t = 25 µs. The query at P = 12 opens [9, 15]; positions 9, 10 and 11 are already resident from the first batch, so only 12–15 are fetched — a second batched round-trip completing at t = 50 µs. Eleven positions, two round-trips: 2 × 25 µs = 50 µs. Ratio 400 ÷ 50 = 8.0×.
What the 8.0× is really made of, and why it is generous. Two effects are combined here, and they should be read separately. Batching collapses 16 round-trips into 2 by issuing a window at once instead of an entry at a time. Selectivity means the windows never touch positions 0–4 at all, so 11 positions move rather than 16. Batching does the bulk of the work; only the batching effect generalises to the full system, because at 128 K context a real query's window is a vanishing fraction of the sequence either way. The baseline is also deliberately pessimistic: fully serial demand misses are the worst case, and a real serving stack would overlap some of them even without a prefetcher. Read 8.0× as an upper bound on what prefetch buys in this toy configuration, not as a system-level speedup.
What this figure does not claim. It is not the +4-percentage-point hit-rate contribution of RoPE-aware prefetch (93% → 97%) in a different unit — that figure comes from the modeled hit-rate ladder in Chapter 7 §7.7 and is arrived at by a different route entirely. It is also not a latency figure for the proposed system: steady-state accesses in the proposed design land in HBM at 100 ns or CXL DRAM at 200 ns, and only the 0.3% miss tail reaches the 25 µs tier this walkthrough uses throughout. [Analytical model] See Chapter 7 §7.6, which is the authority for the prefetch mechanism and its priority function.
RoPE locality = predictable access patterns
If attention concentrates near the query position — the assumption this design rests on — we can predict which KV pairs will be needed and fetch them before the GPU stalls, turning random storage access into batched prefetch streams. RoPE supplies only a non-increasing upper bound on the inner-product magnitude with distance; the concentration itself is assumed, not proven.
Analytical model — this is a 16-token illustrative walkthrough, not a benchmark. The attention concentration is an assumption; the hit-rate ladder (72 → 80 → 87 → 93 → 97) is modeled. Canonical numbers v4.0.