Figure: per-head attention as a scoring signal, not an eviction unit. Heads differ in what they attend to, and that difference is useful for scoring which token blocks matter. It is not a licence to evict a head. Every head of every layer is required for every token: dropping a head's KV for a token that is still attended produces wrong outputs, not a cache miss, so a head is never the unit that is placed, moved or evicted. The unit is a (sequence, token block) of 16 tokens, 5.00 mebibytes at 320 kibibytes per token. H2O and SnapKV evict (head, token) pairs — the token axis is not optional, and dropping tokens is a lossy relaxation that carries an accuracy-evaluation burden, quite different from lossless residency. Four head types at layer 40 are contrasted as scoring signals: the recency head, head 0, looks at the last 50 to 100 tokens; the anchor head, head 1, attends to the first 50 tokens holding the system instructions; the retrieval head, head 2, attends by semantic similarity anywhere in the context; the syntactic head, head 3, attends to grammatically related tokens. In the worked 128K-context document QA example — system prompt at positions 0 to 100, a 64K-token document, the word "France" at position 64K, and 64K tokens of conversation — the four heads weight the last 100 tokens, positions 0 to 100, position 64K and the recent question words respectively. Those four signals are combined into one score per token block, and the blocks scoring highest stay in HBM while the rest are placed on the host or CXL DRAM tier. Every head's KV for a resident block is kept; what varies is which blocks are resident. The earlier claim that per-head tracking contributes plus 6 percentage points of hit rate, taking a cumulative 87 percent to 93 percent, is retracted along with the whole 72 to 80 to 87 to 93 to 97 ladder, which was never run as an ablation and whose increments were assumed additive and independent. No single hit rate exists: the simulated range is 8.5 to 91.6 percent across budgets, policies and workloads. The four-way head taxonomy and the 40, 15, 25, 20 percent shares used elsewhere in this package are a modeling assumption motivated by the induction-head and retrieval-head literature, not a measured head census. Related work this reframing builds on, cited not competed with: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV, InfiniGen, LayerKV, KVPR, PagedAttention.

Per-Head Attention as a Scoring Signal

Heads differ in what they attend to — useful for scoring blocks. A head is never an eviction unit.

Scoring granularity, not eviction granularity

Every head of every layer is required for every token. Dropping one head's KV for a token that is still attended produces wrong outputs, not a cache miss. So a head is never the unit that is placed, moved or evicted. The unit is a (sequence, token block) — 16 tokens, 5.00 MiB at 320 KiB/token.

H2O and SnapKV evict (head, token) pairs: the token axis is not optional. Dropping tokens is a lossy relaxation with an accuracy-evaluation burden attached — a different claim from lossless residency, which requires every block in Read(t) to be present.

What the four profiles below are for: they are signals that score blocks. The scores are combined per block; the block is what moves.

The Four Types of Attention Heads
Recency Head
Head 0, Layer 40
Focuses on last 50-100 tokens. Looking for immediate context.
Old Recent
Anchor Head
Head 1, Layer 40
Always attends to first 50 tokens. Looking for system instructions.
Start End
Retrieval Head
Head 2, Layer 40
Attends based on semantic similarity. Finds relevant info anywhere.
Variable Spiky
Syntactic Head
Head 3, Layer 40
Attends to grammatically related tokens. Subject-verb agreement, etc.
Structured Pattern
Problem with Global Caching
A single "keep recent tokens" policy works for recency heads but fails for:
  • Anchor heads (need start of prompt)
  • Retrieval heads (need specific distant tokens)
Fix: score per head, place per block
Read each head's attention separately as a scoring signal, then combine the signals into one score per (sequence, token block). A block stays resident if ANY head still weights it — which keeps anchor blocks resident even when the recency heads have moved on. All heads of a resident block are kept; heads are never dropped independently, because dropping one is a correctness error, not a cache miss.
Example: 128K Context Document QA
Context structure:
Pos 0-100
Pos 100-64K
Pos 64K (key fact)
Pos 64K-128K

Query: "What is the capital of France?" — Now generating answer...

Recency Head
Scores up: blocks covering the last 100 tokens
(recent conversation)
Anchor Head
Scores up: blocks covering pos 0–100
(system prompt)
Retrieval Head
Scores up: the block covering pos 64K
(where "France" is mentioned)
Syntactic Head
Scores up: blocks covering the question words
(recent "What is")
Combined placement decision — per block
Resident in HBM: the blocks covering pos 0–100 (anchor) + pos ~64K (the France mention) + the last 1000 tokens (recent). Everything else is placed on the host / CXL DRAM tier.
All 8 KV heads × 80 layers of every resident block are kept. The head axis is a scoring input; the block is what moves. No hit rate is claimed for this decision — the ablation that would justify one was never run.

Illustrative — the four-way head taxonomy (Recency / Anchor / Retrieval / Syntactic) and the head shares used elsewhere in this package (40 / 15 / 25 / 20) are a modeling assumption motivated by the induction-head and retrieval-head literature External literature, not a measured head census. Retracted: the "+6 pts of hit rate, 87% → 93% cumulative" attributed to per-head tracking, together with the whole 72 → 80 → 87 → 93 → 97 ladder — never run as an ablation, increments assumed additive and independent. No single hit rate exists; the simulated range is 8.5–91.6% across budgets, policies and workloads Simulated (provisional), and one data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled. Work built on, cited not competed with: H2O, SnapKV, Scissorhands, PyramidKV, Quest, Ada-KV, HeadKV, InfiniGen, LayerKV, KVPR, PagedAttention. Canonical numbers v6.0.