Figure: per-head tracking rationale, showing synthetic 5 by 5 attention matrices over the tokens "The model processes each token" for four assumed head types. Head 0.3 is a recency head, assumed about 40 percent of heads, attending to the last 50 to 200 positions and showing a strong diagonal. Head 1.7 is a syntactic head, about 20 percent, tracking grammatical dependencies so that "processes" attends strongly to its subject "model". Head 2.4 is an anchor head, about 15 percent, attending to the opening positions holding the system prompt regardless of how far generation has advanced. Head 3.11 is a retrieval head, about 25 percent, content-based rather than positional and spiking on whichever earlier tokens are relevant. The four-way taxonomy and the 40 / 15 / 25 / 20 share are a modeling assumption, not a head census, and the matrices are synthetic illustrations. Per-head tracking contributes plus 6 percentage points of hit rate, 87 percent to 93 percent cumulative. All values are an analytical model, not measured.

Per-Head Tracking Rationale

Visualizing what each attention head is assumed to track — Recency · Anchor · Retrieval · Syntactic

ASSUMPTION, not a measured result. The four-way taxonomy (Recency / Anchor / Retrieval / Syntactic) and the 40 / 15 / 25 / 20 share used throughout this package are a modeling assumption adopted to make per-head cache budgets tractable. They are motivated by the induction-head and retrieval-head literature (and by per-head budget work such as Ada-KV and HeadKV), but no head census was run on Llama-2-70B for this work. The matrices below are synthetic illustrations, not model outputs.