Figure: Grouped Query Attention explained, with the head diagrams drawn illustratively at 4 query heads. Multi-Head Attention gives every query head its own key and value head, 4 KV heads for 4 Q heads; Grouped Query Attention shares one KV head per group, 2 KV heads in a 2 to 1 grouping; Multi-Query Attention shares a single KV head across all query heads. In the worked flow, W_Q, W_K and W_V project the input hidden states into Q0 to Q3 with n_heads = 4 and K0, K1 and V0, V1 with n_kv = 2, so group 0 is Q0 and Q1 attending to K0 and V0 and group 1 is Q2 and Q3 attending to K1 and V1, before the heads are concatenated. Cache size is 2 x n_layers x n_kv_heads x seq_len x head_dim x dtype_size. For the canonical Llama-2-70B, MHA with 64 KV heads costs 32 KB per layer per token, GQA with 8 KV heads costs 4 KB — an 8 times reduction and what the model actually uses — and MQA with 1 KV head costs 512 B; 2 x 80 layers x 8 KV heads x 128 x 2 B = 327,680 B = 320 KiB per token, times 131,072 = 40 GiB = 43 GB at 128K context. All values are an analytical model, not measured.

Grouped Query Attention

Balancing KV-cache memory efficiency with model quality — diagram is illustrative (4 Q heads shown)

Query (Q)
Key (K)
Value (V)
MHA
Multi-Head Attention
KV Cache: 4 KV heads (1 per Q head)
GQA
Grouped Query Attention
KV Cache: 2 KV heads (2:1 grouping)
MQA
Multi-Query Attention
KV Cache: 1 KV head (all Q share it)
How GQA Works
Input Hidden States
↓
WQ
WK
WV
↓
Q₀ Q₁ Q₂ Q₃
n_heads = 4
K₀ K₁
n_kv = 2
V₀ V₁
n_kv = 2
↓
Group 0: Q₀,Q₁ → K₀,V₀
Group 1: Q₂,Q₃ → K₁,V₁
↓
Attention Output (concat all heads)
Why GQA Matters
Smaller KV Cache
Reduces memory footprint proportional to the grouping ratio. Critical for long-context inference.
Faster Decoding
Less KV data to load from HBM per token. Directly improves memory-bound decode throughput.
Quality Preserved
Outperforms MQA significantly. Llama 2 70B uses GQA with 8 KV heads for 64 query heads.
KV Cache Size = 2 × n_layers × n_kv_heads × seq_len × head_dim × dtype_size
The Canonical Model: Llama-2-70B
MHA — 64 KV heads
32 KB per layer per token
GQA — 8 KV heads
4 KB per layer per token — an 8× cache reduction, and what Llama-2-70B actually uses
MQA — 1 KV head
512 B per layer per token
2 × 80 layers × 8 KV heads × 128 × 2 B = 327,680 B = 320 KiB per token  →  × 131,072 = 40 GiB = 43 GB at 128 K context

Analytical model — cache sizes are derived from the model configuration, not measured. Canonical numbers v4.0.