Appendix B

Attention Mechanism Deep Dive

Scaled dot-product attention, multi-head structure, and grouped-query attention (GQA).

B.1 Scaled Dot-Product Attention

Steps:

1. Score: QK^T computes similarity between queries and keys
2. Scale: Divide by √d_k to prevent gradient vanishing
3. Normalize: Softmax converts scores to probabilities
4. Aggregate: Weighted sum of values

B.2 Multi-Head Attention

Instead of single attention, we use multiple parallel attention heads:

For Llama-70B: 64 query heads, each with d_head=128.

B.3 Grouped-Query Attention (GQA)

GQA reduces KV-cache by sharing K/V heads across multiple query heads:

For the illustrated 64-query-head / 8-KV-head geometry, GQA reduces KV-head storage by 8× relative to MHA with 64 KV heads. Quality impact is model- and training-dependent and is not measured here.