Figure: per-head tracking with GQA on Llama-70B, where 64 query heads grouped 8 to 1 give 8 KV heads, and 8 KV heads across 80 layers gives 640 independent LRU queues. Query heads Q0 to Q7 map to KV-head 0 and LRU queue [layer, 0]; Q8 to Q15 map to KV-head 1 and queue [layer, 1]; and so on to Q56 to Q63 mapping to KV-head 7 and queue [layer, 7]. Each queue orders its entries hot to cold — for example queue [layer 42, KV-head 3] holds position 127891 with 847 accesses and attention 0.92, position 0 with 512 accesses and 0.88, position 1024 with 234 accesses and 0.45, position 89012 with 89 accesses and 0.31, and position 45678 with 12 accesses and 0.08, which is the eviction candidate. Per-entry metadata is 8 bytes: position_id u32 4 bytes, access_count u16 2 bytes, attention_score fp16 2 bytes. So 640 queues x 131,072 positions x 8 bytes = 671,088,640 B = 640 MiB, or 671 MB decimal, which is about 1.5 percent of the 43 GB per-user KV cache and sits in the 10 GB pinned tier-0 HBM region. All values are an analytical model, not measured.

Per-Head Tracking with GQA

Llama-70B: 64 Query Heads → 8 KV-Heads × 80 Layers = 640 LRU Queues

1. GQA Grouping Defines Tracking Unit
KV-Head 0
LRU Queue [layer, 0]
KV-Head 1
LRU Queue [layer, 1]
KV-Head 7
LRU Queue [layer, 7]
2. 640 Independent LRU Queues
Layer 0
Layer 1
Layer 2 ... 79
3. What Each Queue Tracks
Queue [Layer 42, KV-Head 3]
One of 640 queues
EVICT
CANDIDATES
Per-Entry Metadata (8 bytes)
position_id — u32 (4B)
access_count — u16 (2B)
attention_score — fp16 (2B)
640 queues × 131,072 positions × 8 bytes = 671,088,640 B = 640 MiB (671 MB decimal)  →  ≈1.5% of the 43 GB per-user KV cache
Why 640 queues? Llama-2-70B has 80 layers × 8 KV heads (64 Q heads grouped 8:1 by GQA). The KV head is the smallest unit whose cache lines are shared, so it is the natural tracking unit: 80 × 8 = 640.
Why 131,072 entries each? The canonical scenario is a 128 K-token context = 217 = 131,072 positions. Each queue must be able to hold one entry per position of the full context, so the queue is sized for the worst case rather than the current occupancy.
Why it is affordable. 640 MiB of metadata against 43 GB of KV data is a 1.5% overhead, and it sits in the 10 GB pinned Tier-0 HBM region alongside anchor tokens and staging buffers.

Analytical model — queue counts and metadata sizes are derived from the model configuration; the LRU contents shown are illustrative, not measured. Canonical numbers v4.0.