Figure: Grouped Query Attention explained, with the head diagrams drawn illustratively at 4 query heads. Multi-Head Attention gives every query head its own key and value head, 4 KV heads for 4 Q heads; Grouped Query Attention shares one KV head per group, 2 KV heads in a 2 to 1 grouping; Multi-Query Attention shares a single KV head across all query heads. In the worked flow, W_Q, W_K and W_V project the input hidden states into Q0 to Q3 with n_heads = 4 and K0, K1 and V0, V1 with n_kv = 2, so group 0 is Q0 and Q1 attending to K0 and V0 and group 1 is Q2 and Q3 attending to K1 and V1, before the heads are concatenated. Cache size is 2 x n_layers x n_kv_heads x seq_len x head_dim x dtype_size. For the canonical Llama-3 70B, the three rows are a counterfactual over the same 80-layer geometry: MHA with 64 KV heads would cost 32 kilobytes per layer per token, GQA with 8 KV heads costs 4 kilobytes — an 8 times reduction, and what the model actually uses — and MQA with 1 KV head would cost 512 bytes. Only the GQA row describes a real model here. Both Llama-2-70B and Llama-3-70B are GQA with 8 KV heads and 64 query heads, at 320 kibibytes per token; the 2,560 kibibytes-per-token MHA row belongs to Llama-1-65B and the smaller MHA Llama-2 variants, not to Llama-2-70B. The arithmetic: 2 times 80 layers times 8 KV heads times 128 times 2 bytes equals 327,680 bytes equals 320 kibibytes per token, times 8,192 equals 2.50 gibibytes at 8K context and times 131,072 equals 40.0 gibibytes at 128K context, with the single-sequence crossover against 130.4 gibibytes of bf16 weights at about 427,000 tokens. Block granularity is 16 tokens equals 5.00 mebibytes. This is an analytical model — arithmetic over published model geometry, simulator-independent, and it survived the v5.0 retractions unchanged; nothing on this page ever depended on a hit rate.

Grouped Query Attention

Balancing KV-cache memory efficiency with model quality — diagram is illustrative (4 Q heads shown)

Query (Q)
Key (K)
Value (V)
MHA
Multi-Head Attention
KV Cache: 4 KV heads (1 per Q head)
GQA
Grouped Query Attention
KV Cache: 2 KV heads (2:1 grouping)
MQA
Multi-Query Attention
KV Cache: 1 KV head (all Q share it)
How GQA Works
Input Hidden States
↓
WQ
WK
WV
↓
Q₀ Q₁ Q₂ Q₃
n_heads = 4
K₀ K₁
n_kv = 2
V₀ V₁
n_kv = 2
↓
Group 0: Q₀,Q₁ → K₀,V₀
Group 1: Q₂,Q₃ → K₁,V₁
↓
Attention Output (concat all heads)
Why GQA Matters
Smaller KV Cache
Reduces memory footprint proportional to the grouping ratio. Critical for long-context inference.
Faster Decoding
Less KV data to load from HBM per token. Directly improves memory-bound decode throughput.
Quality Preserved
Outperforms MQA significantly. Llama 2 70B uses GQA with 8 KV heads for 64 query heads.
KV Cache Size = 2 × n_layers × n_kv_heads × seq_len × head_dim × dtype_size
The Canonical Model: Llama-3 70B — MHA and MQA rows are counterfactuals over the same geometry
MHA — 64 KV heads (counterfactual)
32 KB per layer per token. No 70B Llama uses this: the 2,560 KiB/token MHA row belongs to Llama-1-65B and the smaller MHA Llama-2 variants.
GQA — 8 KV heads
4 KB per layer per token — an 8× cache reduction, and what both Llama-2-70B and Llama-3-70B actually use (64 query heads, 8 KV heads)
MQA — 1 KV head (counterfactual)
512 B per layer per token
2 × 80 layers × 8 KV heads × 128 × 2 B = 327,680 B = 320 KiB per token
× 8,192 = 2.50 GiB at 8 K context  ·  × 131,072 = 40.0 GiB at 128 K context
16 tokens = 5.00 MiB block  ·  crossover with 130.4 GiB of bf16 weights ≈ 427,000 tokens

Analytical model — arithmetic over published model geometry, simulator-independent and independently confirmed. Everything on this page stands through the v5.0 retractions: none of it ever depended on a hit rate. 320 KiB/token · 2.50 GiB at 8K · 40.0 GiB at 128K · 5.00 MiB blocks · crossover ≈ 427,000 tokens. Canonical numbers v6.0.