An analytical, unmeasured comparison of the cost of fetching a KV block across a slower memory tier versus regenerating it by re-running the forward pass — and what it means for capacity planning as context length grows.
A note on scope before anything else: every number in this chapter is analytical — derived from block sizes, published or assumed link bandwidths, and assumed GPU utilization figures. No CXL hardware was measured to produce any figure in this chapter. Chapter 4 reports the one hardware-measured bandwidth result in this document, 236.5 GB/s of on-GPU HBM/unified-memory streaming bandwidth on a DGX Spark. That is a different link, a different distance, and a different measurement method than anything discussed here, and the two should never be cited interchangeably. This chapter's job is to reason about the economics of a tier boundary — GPU memory to CXL-attached memory — using the best available assumptions, and to be explicit about which numbers are assumptions and which are arithmetic.
The unit of analysis throughout this chapter is a single KV block covering 16 tokens of Llama-3 70B under grouped-query attention (GQA). At 320 KiB of KV-cache per token, 16 tokens gives a block of exactly 5.00 MiB (16 × 320 KiB = 5,120 KiB = 5.00 MiB). This is a convenient, round unit — small enough to represent a realistic page-sized fetch granularity, large enough that fixed per-transfer overheads do not dominate the arithmetic.
Moving that 5.00 MiB block across a CXL-attached tier boundary takes a length of time set entirely by the bandwidth assumption fed into the arithmetic — none of these bandwidths were measured on real CXL hardware in this analysis. Three assumptions are carried through the rest of this document, spanning an optimistic theoretical ceiling down to a pessimistic real-world case:
| Bandwidth assumption | Basis | Fetch time, 5.00 MiB block |
|---|---|---|
| 121 GB/s | CXL 3.0 x16 theoretical, one direction | 43 µs |
| 64 GB/s | Mid-range assumption — explicitly unmeasured | 82 µs |
| 32 GB/s | Pessimistic assumption (contention, protocol overhead) | 164 µs |
None of these three numbers is "the" CXL bandwidth for this system — they bracket a plausible range under different loading and protocol-overhead assumptions, and the 64 GB/s mid-range figure is the one used for illustrative comparisons later in this chapter unless stated otherwise. Anyone extending this analysis with a bandwidth number pulled from a vendor datasheet or a different platform's measurement should re-derive the fetch time from that number directly rather than reusing 43/82/164 µs.
The alternative to fetching a block from a slower tier is to discard it and regenerate the 16 tokens it represents by running them back through the model's forward pass. That cost depends on an assumption about achieved GPU utilization — model FLOPs utilization (MFU) — which nothing in this chapter measures directly:
| Assumed MFU | Forward-compute cost, 16 tokens |
|---|---|
| 50% (optimistic) | 9 ms |
| 40% (typical) | 11 ms |
| 25% (conservative) | 18 ms |
These figures assume the block being regenerated is the most recently generated one — the tokens sit at the end of the sequence, so recomputing them is a straightforward forward pass over just those 16 positions using cached context already resident. That is the cheapest possible regeneration case, and it is already 9–18 ms against a 43–164 µs fetch.
The comparison above understates the real cost whenever the evicted block sits in the interior of a sequence rather than at its tail. KV-cache is not an independently reproducible artifact per block — a hidden state at position i is a function, recursively, of every token from position 0 through i, propagated through every attention layer. Regenerating an interior block correctly does not mean recomputing 16 tokens' worth of forward pass in isolation; it means re-running the forward pass through the entire prefix that precedes the block, because there is no way to reconstruct layer-wise attention state for positions in the middle of a sequence without having the correct hidden states for everything upstream of them.
Concretely: evicting the most recent 16 tokens and regenerating them costs roughly the 9–18 ms shown above. Evicting a 16-token block sitting at, say, position 50,000 of a 100,000-token sequence and regenerating it correctly costs at minimum that same 9–18 ms — and in practice a great deal more, because the forward pass has to walk through everything between the start of context (or the nearest still-cached anchor point) and that block before it can reproduce the block's correct hidden states. The 9–18 ms figure is therefore a floor for interior blocks, not a representative estimate. This asymmetry — cheap, bounded, tier-independent fetch cost versus a regeneration cost that grows with sequence position — is the single strongest structural argument for tiering KV-cache rather than discarding and recomputing it.
Putting the two sides together: moving a KV block across a slower tier is two to three orders of magnitude cheaper than regenerating it by forward compute. This is deliberately stated as a range of orders of magnitude, not a single precise multiple, because the two costs are not measurements of the same operation type and each carries its own uncertainty — tier-fetch cost depends on an unmeasured bandwidth assumption (32 to 121 GB/s above), and regeneration cost depends on an assumed GPU utilization figure (25% to 50% MFU above). Multiplying a specific numerator by a specific denominator from those ranges produces a specific ratio, but that ratio is an artifact of which two assumptions you happened to pick, not a fixed property of the system.
As one illustrative point within that broader range — and only as one point, not as the headline number — take the mid-range 64 GB/s tier-fetch time of 82 µs against the conservative 18 ms (25% MFU, rounded here to 17.9 ms before rounding) regeneration cost for a tail block: 17.9 ms ÷ 82 µs ≈ 218×. That is a real arithmetic result for those two specific inputs, and it sits comfortably inside the two-to-three-orders-of-magnitude range this chapter claims. It is not "the" ratio. Pick the optimistic ends of both ranges — 121 GB/s fetch, 50% MFU regeneration — and the ratio comes out closer to two orders of magnitude instead of three. Anyone citing this chapter should carry the range, or state which two specific assumptions produced whatever single number they quote. The claim worth remembering is the order of magnitude, not the digit.
Tier economics matter operationally because KV-cache footprint grows with context length while model weight footprint is fixed, and at long enough context the two become comparable — which changes what "offload" even means, since the tier is now being asked to hold something on the scale of the model itself. For Llama-3 70B in bf16, weights occupy 140 GB (130.4 GiB). Two reference context lengths for a single sequence:
| Context length | KV-cache size (single sequence) | KV as % of weight footprint |
|---|---|---|
| 32,768 tokens (32K) | 10.0 GiB | 8% |
| 131,072 tokens (128K) | 40.0 GiB | 31% |
Extrapolating the same per-token KV rate out further, a single sequence's KV-cache overtakes the entire 130.4 GiB weight footprint at roughly 427,000 tokens of context. Beyond that crossover point, for that one sequence, the cache being tiered is larger than the model producing it — a useful sanity-check figure for anyone sizing a CXL tier against expected context-length distributions, and a reminder that "long context" is not a fixed-size add-on to a deployment's memory budget but a second, independently scaling consumer of it.
Every number in this chapter is analytical: three bandwidth assumptions for tier-fetch cost, three MFU assumptions for regeneration cost, and a linear extrapolation for the capacity crossover. None of it substitutes for the hardware-measured HBM bandwidth result in Chapter 4, and none of it should be quoted as if it were measured. What it does establish, robustly across the assumption ranges considered, is that fetching a KV block from a slower tier is two to three orders of magnitude cheaper than regenerating it, that the gap only widens for interior blocks because of KV-cache's recursive dependency on preceding context, and that KV-cache footprint is not a minor addendum to weight footprint once context length climbs into the hundreds of thousands of tokens.