Figure: the four constraints on KV state in a serving system, and what adding a memory tier does to each. A vertical flow runs from the CXL memory tier, across the tier boundary, into HBM, and from HBM into the attention kernel. Constraint one, capacity: HBM cannot hold the KV of all sequences you want resident. Adding a tier RELIEVES it; this is what the tier is for. Constraint two, HBM-side bandwidth: attention re-reads the active KV out of HBM at every decode step. Adding a tier leaves it UNCHANGED; this is measured, with KV slopes within 8 per cent across two attention kernels, 16.34 against 17.58 milliseconds per GiB, so no kernel fixes it either. Constraint three, tier-ingress bandwidth: KV must cross the tier boundary fast enough to feed attention. Adding a tier CREATES it. Constraint four, transfer latency: each individual transfer must complete before compute stalls on it. Adding a tier CREATES it, and it is distinct from tier-ingress bandwidth, because the aggregate rate can hold while individual transfers still stall the pipeline. The claim the figure lands: a memory tier does not eliminate the KV memory wall, it changes its shape, from a capacity wall into a data-movement wall.

Figure 3.1 · Analytical model

Four constraints, not one wall — and a tier does something different to each

Conflating the ways KV state limits a serving system produces the false conclusion that adding capacity solves KV scaling. There are four constraints, they bind independently, and only one of them is the one a memory tier was built to move.

The path a KV byte takes

CXL memory tiercapacity that sitsbehind HBMHBMwhat attentionactually reads fromAttention kernelre-reads the active KVevery decode step✓CapacityRELIEVED+Tier-ingress bandwidthCREATED+Transfer latencyCREATED=HBM-side bandwidthUNCHANGED

What a tier does to each constraint

  • Capacity Relieved

    HBM cannot hold the KV of all sequences you want resident.

    This is what the tier is for — the one wall it was built to move.

    Analytical model

  • Bandwidth (a) — HBM-side Unchanged

    Attention re-reads the active KV out of HBM at every decode step.

    Measured: KV slopes within 8% across two attention kernels (16.34 vs 17.58 ms/GiB). Capacity added behind HBM cannot touch a constraint that binds on the HBM side, and no kernel fixes it either.

    Measured

  • Bandwidth (b) — tier-ingress Created

    KV must cross the tier boundary fast enough to feed attention.

    An aggregate rate across the whole decode set. Did not exist before the tier did.

    Analytical model

  • Transfer / latency Created

    Each individual transfer must complete before compute stalls on it.

    Per-transfer, and independent of (b): the aggregate rate can hold while individual transfers still stall the pipeline.

    Analytical model

A memory tier does not eliminate the KV memory wall. It changes its shape — from a capacity wall into a data-movement wall. That is a real win, because capacity is a hard limit and data movement is a scheduling problem. But it is a different problem. Sustainability — the aggregate rate across the decode set — and hiding — the per-transfer deadline — are independent: the first can hold while the second fails, producing stalls at an entirely satisfactory average bandwidth.

Analytical model — CANONICAL-NUMBERS.md §4/§5