Figure: a seven-step decision procedure for whether and how to tier KV state, drawn as a vertical flowchart with branches to the right. Step one: do requests share substantial prefixes? If yes, use prefix reuse, radix or prefix caching, first, because it deletes work rather than moving it. Step two: is reservation or fragmentation waste significant? If yes, use block-granular allocation, which is lossless. Step three: does aggregate KV exceed HBM? If no, stop, you do not need a tier. Step four: does a single active sequence exceed HBM? If yes, exact single-device execution now requires either streaming and tiering or more memory; otherwise you must change an execution, model or accuracy constraint, by capping the context, rejecting the request, sharding KV across devices, quantising, sparsifying attention, or changing model. Streaming is one architectural answer, not the mandatory one. Step five: is the accuracy budget non-zero? If yes, lossy reduction becomes available; if no, lossless tiering only. Step six: does the pipeline stall despite adequate average bandwidth? If yes, the HIDING constraint binds, not SUSTAINABILITY; deepen prefetch, resize chunks, reorder traversal. Step seven: is movement amplification materially above the workload's own lower bound? If yes, placement and admission policy is thrashing, and that should be fixed before tuning replacement; if no, replacement policy is second-order. No universal amplification threshold is asserted; the comparison is against the workload's own lower bound from an offline-optimal placement.

Figure 12.1 · Illustrative — decision procedure

Seven questions, in the order they have to be asked

Whether and how to tier KV state. The third question is an exit, not a formality: most systems should stop there.

STEP 1Do requests share substantial prefixes?YESPrefix reuse — radix / prefix caching.It deletes work rather than moving it. Do this first.STEP 2Is reservation / fragmentation waste significant?YESBlock-granular allocation.Lossless. No accuracy budget is spent.STEP 3Does aggregate KV exceed HBM?NOSTOP. You do not need a tier.Everything downstream of here is answering a problem you do not have.STEP 4Does a SINGLE active sequence exceed HBM?YESExact single-device execution now requires EITHERstreaming / tiering OR more memory.Otherwise you must change an execution, model or accuracy constraint:cap the context, reject the request, shard KV across devices,quantise, sparsify attention, or change model.Streaming is one architectural answer, not the mandatory one.STEP 5Is the accuracy budget non-zero?YESLossy reduction becomes available.A separate layer with separate obligations.NOLossless tiering only.STEP 6Does the pipeline stall despite adequate average bandwidth?YESThe HIDING constraint binds — not SUSTAINABILITY.Deepen prefetch, resize chunks, reorder traversal.Adding link width will not fix a per-transfer deadline.STEP 7Is movement amplification materially above theworkload’s own lower bound?YESPlacement / admission policy is thrashing.Fix that before tuning replacement. It presents identically to a poorreplacement rule in hit rate, but has a different fix.NOReplacement policy is second-order.

The order is the content.Steps 1 and 2 delete or compact work and cost nothing in accuracy, so they come before any tier is considered. Step 3 is the exit: if aggregate KV fits in HBM, stop — everything after it answers a problem you do not have. Step 4 refuses to make streaming mandatory: it is one architectural answer among several. Steps 6 and 7 separate two failures that look identical in a hit-rate chart and have different fixes.

No universal threshold is asserted at step 7.An earlier revision of this procedure carried a 1.5 amplification threshold that nothing in the document established. It is removed. The comparison is against the workload’s own lower bound, computed by simulating the same trace under an offline-optimal placement — not against any fixed number.

Illustrative (decision procedure) — data/chapter_section_rev5.md §6