Appendix D

Rotary Position Embeddings (RoPE)

Rotation mechanics, frequency spectra, and distance-dependent attention decay.

D.1 Core Concept

RoPE encodes position by rotating Q and K vectors in 2D subspaces:

where m is position and θ_i = 10000^-2i/d for dimension pair i.

D.2 Position-Dependent Attention

The dot product between positions m and n becomes:

Attention naturally decays with position distance |m-n|.

D.3 Locality Exploitation

This creates predictable attention patterns:

• Recent tokens (small distance): High attention probability
• Distant tokens (large distance): Lower baseline attention
• Exception: Anchor positions (0-100) receive consistent attention

We exploit this for RoPE-aware prefetching — tokens near the current decode position are prefetched with higher priority.

D.4 Prefetch Window Strategy

Combined with EMA scores, this achieves 90%+ prefetch hit rates.