Formal Core — Object Model, Plane Split, Objective Function
Three things locked down: what a KV block object is, where policy ends and mechanism begins, and what is being minimised subject to what. No new prose argument — this is the specification the chapters' prose serves.
Almost everything on this page is specification, not result. Where a number appears it carries its own class. The one term that has moved from assumption to measurement is the decode cost model, and it moved because of Appendix L: the objective function of §M.4 was written assuming a single memory bandwidth, and there are two. The policy claims — admission, residency budgets, metadata persistence, device-side control — remain specification. Appendix N tests some of them in simulation, and those results are Simulated (provisional).
M.1 One Property the Rest Depends On
For a given (sequence, layer, position range), K and V are computed once — during prefill, or at the decode step that produces that token — and are never subsequently modified. Nothing in autoregressive decoding mutates existing KV.
Four consequences follow, and all four distinguish this from a general-purpose cache:
| # | Consequence | Why it matters |
|---|---|---|
| 1 | A tier copy, once written, never becomes stale. | There is no coherence problem on the payload. The hard part of tiered-memory design does not apply here. |
| 2 | Eviction of a block that already has a tier copy is a clean discard. | No writeback. Eviction costs nothing on the link. |
| 3 | Each block is therefore demoted at most once in its lifetime, however many times it is promoted. | This is what bounds the traffic metric of §M.5.1 and gives it a floor of 2 rather than 1. |
| 4 | The only mutable state associated with a block is its control metadata. | Orders of magnitude smaller than the payload, and handled separately — see §M.2. |
In-place re-quantisation or re-compression of a resident block writes new payload bytes and breaks write-once for that block. Systems that do this must account for it: consequence 1 fails (the tier copy is now stale), consequence 2 fails (the discard is no longer clean), and consequence 3 fails (the demotion count is no longer bounded by one). Given that Appendix L §L.5.1 makes compression the dominant lever, this is not a hypothetical corner — it is the most likely way a real implementation would leave the regime this specification describes.
M.2 KV Object Model
M.2.1 The object
A KV block object is the unit of placement, migration and accounting. It carries a payload and three classes of state.
Text equivalent of Diagram M.1. A KV block object, labelled b, has four parts. First, PAYLOAD: the K and V tensors for one sequence, layer and position range; it is write-once. Second, IDENTITY, which is immutable, comprising sequence id, position range, layer, head group, prefix hash, and provenance or parent block. Third, POLICY, which is mutable in place without migrating the payload, comprising tenant, QoS class, priority, reuse score, recency, and scheduler-supplied residency intent. Fourth, MECHANISM, which is mutable and owned by whichever plane currently holds the block, comprising residency location, compression format, transfer state, validity, and in-flight or pinned flags.
M.2.2 What migrates, and what does not
This is the correction to "migrate metadata with the block". Migrating all metadata as one physically inseparable unit is wrong, because policy metadata changes while the block does not move — QoS is reassigned, priority shifts, scheduler intent updates every step.
| Class | Lifetime | Migrates with payload? | Mutable without migration? |
|---|---|---|---|
| Identity | Fixed at creation | Yes — it is the block's name | No |
| Policy | Changes on scheduler events | Logically bound, physically separable | Yes |
| Mechanism | Changes on every transfer | No — regenerated per location | Yes |
So the design claim is not "one physically fused object". It is:
Identity travels with the payload. Policy is associated with identity and may be updated in place, on either side, without moving a byte. Mechanism is local to wherever the block currently sits.
M.2.3 What is actually being proposed
Today, in the systems reviewed, a block evicted from HBM loses its accumulated policy state and is re-admitted as if newly created. The proposal is that identity-keyed policy state survives the round trip.
It is testable without any CXL device, and it has been tested in simulation: Appendix N §N.4 reports it as the largest single effect in the study — +34.15 pts to LFU and +24.85 pts to EMA α=0.01 at a 32 GiB budget. Those figures are Simulated (provisional) and carry the standing caveat of §N.9. That one of the four claims in §M.3.2 needs no device at all is the point of §M.3.2's closing note.
M.2.4 Overhead — what is and is not negligible
Payload-capacity overhead is negligible. 64 B of metadata against a 5.00 MiB block is 0.0012%. Analytical model, over the block granularity fixed in CANONICAL-NUMBERS.md §2 (16 tokens at 320 KiB/token for Llama-3 70B GQA).
The capacity argument bounds none of it. The open items, each of which is a separate measurement and none of which has been made:
- Lookup bandwidth on the promotion path — how much of a promotion is spent finding the block rather than moving it.
- Atomic update cost under concurrent scheduler writes — policy state is written by the scheduler while the data plane reads it.
- Controller SRAM footprint for the resident index — the index must fit somewhere, and that somewhere is the scarcest memory in the system.
- Coherence of policy state between planes — the payload has no coherence problem (§M.1); the policy metadata does.
- Per-transaction overhead on the link — small metadata transactions are the worst case for a protocol optimised for bulk movement.
Quoting 0.0012% as though it answered the overhead question is the error this callout exists to prevent. It answers a capacity question only.
M.3 Control Plane / Data Plane
M.3.1 The split
The null hypothesis is strong and should be stated first, before any argument for the design:
The host serving runtime knows more than any device-side controller can. It holds the active decode set, next-step candidates, tenant priorities, preemption decisions, sequence completion, speculative-decoding acceptance, and admission state. A CXL controller has none of that intrinsically.
The defensible decomposition follows directly from the null hypothesis rather than against it:
Text equivalent of Diagram M.2. A vertical flow in five stages. At the top, the SERVING SCHEDULER passes execution intent downward: the decode set, priorities, prefetch horizon, QoS and preemption. It feeds the KV CONTROL PLANE, which runs on the host and owns admission, residency, prefetch horizon, QoS and representation choice. The control plane passes placement intent downward — block ids, deadlines and priority. That feeds the KV DATA PLANE, which runs on the device and owns promotion, demotion, DMA, transfer scheduling, compression execution and metadata maintenance. The data plane fans out to two destinations: GPU HBM on the left and tier memory on the right. Both return metrics, which merge into a single control-feedback path running back up to the control plane. Intent flows down; metrics flow up.
M.3.2 What would justify device residency — as falsifiable claims
Each of the following is a hypothesis with a measurement attached. None is assumed true. Each row names both the measurement that would support it and the measurement that would refute it, so that neither outcome can be reinterpreted after the fact.
| Claim | Measurement that would support it | Measurement that would refute it |
|---|---|---|
| C1. Device-local decisions remove host round-trips from the promotion path | Promotion latency with device-side scheduling versus host-issued DMA, under load | Host-issued path already hidden behind compute; no exposed stall difference |
| C2. Metadata lookup at memory latency matters | Fraction of promotion latency spent in metadata lookup | Lookup is off the critical path, or is trivially cacheable in host DRAM |
| C3. Frees host CPU under multi-tenancy | Host CPU utilisation attributable to KV movement, at scale | CPU is not the bottleneck at realistic tenant counts |
| C4. Sustains policy state across eviction and re-admission | Hit rate and movement amplification, with and without persistent policy state | Host-side persistence achieves the same result |
And C4 is the one with simulation support behind it (Appendix N §N.4). If C4 is the claim that survives, the contribution is the object model and protocol of §M.2, and CXL is simply where it runs — which is still a contribution, just a different one from the one the architecture was originally argued for. This is the level at which CANONICAL-NUMBERS.md §6 states the contribution, and it is deliberately weaker than the v4.0 framing.
M.4 Objective Function
M.4.1 Decision variables
For each block b and time step t:
x_b(t) ∈ {0,1} block b resident in HBM at t
p_b(t) ∈ {0,1} promotion of b initiated at t (tier → HBM)
d_b(t) ∈ {0,1} demotion of b initiated at t (HBM → tier)
M_b = payload bytes of block b. Δt = step duration.
M.4.2 Correctness constraint — the line between lossless and lossy
Let Read(t) be the set of blocks attention reads at step t.
Lossy methods — token eviction, sparse retrieval, approximate reconstruction — operate by relaxing it: they permit b ∈ Read(t) with x_b(t) = 0, substituting an approximation. Everything else in the model is common to both. That is why the distinction can be drawn here in one line rather than argued qualitatively.
Under dense attention, Read(t) is the entire context of every sequence in the decode set. This is why "which blocks" is not the interesting question — the answer is all of them — and why when and at whose expense are. It is also the formal reason the retracted v4.0 claim that a 5.4% hot set captured 97% of accesses was a category error: it conflated a cache hit rate with attention sparsity, two different claims with different obligations (CANONICAL-NUMBERS.md §7).
M.4.3 Resource constraints
| Constraint | Statement | What it binds |
|---|---|---|
| CAPACITY | Σ_b M_b · x_b(t) ≤ C_HBM ∀t | How much may be resident at once |
| LINK | Σ_b M_b · (p_b(t) + d_b(t)) / Δt ≤ B_tier(t) ∀t | How fast the boundary may be crossed |
| SUSTAINABILITY | Σ_{i ∈ decode set} R_KV,i ≤ B_tier | Aggregate KV consumption rate across all concurrently decoding sequences, not one sequence |
| HIDING | T_transfer(chunk) ≤ T_compute(previous chunk) + prefetch_slack | Per transfer — not on average |
| QoS | TTFT_j ≤ τ_j, ITL_j ≤ ι_j ∀ tenants j | What the workload is entitled to |
The first can hold while the second fails, producing stalls at satisfactory average bandwidth. Both must be stated because techniques act on different terms: link width and compression raise or relieve the first; prefetch depth, chunk sizing and traversal order address the second. A design that reports only aggregate bandwidth cannot tell the two failures apart. This is the same distinction that CANONICAL-NUMBERS.md §5 draws between the tier-ingress wall and the transfer/latency wall.
M.4.4 The objective
The workload and its QoS targets are the constraint; resources consumed are the objective. Minimising HBM occupancy for its own sake would be degenerate — the point is to serve the same work with less.
minimise Σ_t Σ_b M_b · x_b(t) · Δt ← HBM byte-time
+ λ · Σ_t Σ_b M_b · (p_b(t) + d_b(t)) ← tier traffic
+ μ · Σ_t stall_exposed(t) ← unhidden transfer
subject to the constraints of §M.4.2 – §M.4.3
5 MiB held for 1 ms and 5 MiB held for 500 ms are not the same allocation. This is what makes idle inter-turn state and active decode state comparable on one scale, and it is why the problem is temporal resource allocation rather than caching.
This objective was written assuming a single memory bandwidth. Appendix L measures two: KV read at 26–28% of achievable bandwidth against 87% for weights, a ~3.4× model-derived effective-path ratio, with similar KV slopes across two tested frameworks and two attention kernels. Measured. A KV byte is not interchangeable with a weight byte, and the terms above should be weighted accordingly. This is the one place where measurement has already amended the specification.
M.4.5 Admission, and where eviction sits
Under a binding capacity constraint, admission and eviction are dual: admitting b when HBM is full forces some b′ out. Eviction does not disappear.
Conventional eviction asks which resident object should leave. A tiered KV controller begins one level higher: which state deserves HBM residency over the upcoming scheduling horizon? Eviction becomes a consequence of that decision rather than an independent policy.
Formally, admission over a horizon H with per-block value V_b — derived from QoS, expected reuse and scheduler intent — and residency cost M_b · H:
Replacement policy is what remains once this is fixed, which is the formal statement of "subordinate to admission" — and the reason the phrasing is subordination rather than elimination.
M.4.6 Terminology
Continuous batching means there is no currently scheduled sequence. There is a currently scheduled decode set, membership of which changes step to step. All residency language in this package should use the set, not the singular.
M.5 Metrics
M.5.1 Movement amplification — corrected
The Rev 3 definition conflated directions and asserted a floor of 1.0, which is wrong. Two metrics replace it, plus the write-once bound from §M.1.
Let U be the unique payload bytes the computation requires. For block b, let n_b be the number of promotions and δ_b the number of demotions.
A_read = Σ_b n_b · M_b / U promotion traffic per useful byte A_traffic = Σ_b (n_b + δ_b) · M_b / U total boundary traffic per useful byte
Write-once gives δ_b ≤ 1 (§M.1): once a tier copy exists it stays valid, so later evictions are clean discards. Therefore:
| Block history | δ, n | Contribution to A_traffic | Reading |
|---|---|---|---|
| Never leaves HBM | δ=0, n=0 | 0 | Free |
| Written out, never recalled | δ=1, n=0 | 1 | Paid once, no benefit sought |
| Written out, recalled once | δ=1, n=1 | 2 | The floor for any block that round-trips |
| Written out, recalled 3× | δ=1, n=3 | 4 | Thrashing |
Values materially above the workload's own lower bound indicate the placement policy is thrashing — which presents identically to a poor replacement rule in hit rate, but has a different fix. Diagnosing one as the other is the failure this metric exists to prevent.
The Rev 3 decision tree contained a 1.5 that nothing in the document established. It is removed. The comparison is against the workload-specific lower bound, computed by simulating the same trace with an offline-optimal placement. There is no number here that transfers between workloads, and any threshold quoted as if it did is being invented.
M.5.2 Reuse per moved byte — the economic objective
A block may cross the boundary once and be read by attention hundreds of times from HBM. That ratio is the reason residency is worth paying for at all:
A block that round-trips once (traffic 2·M_b) and is read 500 times during a generation has a reuse ratio of 250. Illustrative.
Maximising useful HBM reuse per tier byte moved is a better policy objective than hit rate, because hit rate is blind to what the hit cost to arrange.
M.5.3 Instrumentation set
| Metric | Instruments |
|---|---|
| HBM byte-time (GiB·s) per completed request | The objective's first term |
| Tier bytes per generated token | The second term; easiest to measure, easiest to compare |
| A_read, A_traffic versus offline-optimal | Placement quality |
| Reuse ratio | Whether residency is earning its cost |
| Exposed stall per token | Whether HIDING holds |
| Prefetch accuracy (fetched-and-used ÷ fetched) | Prefetch policy quality |
| TTFT, inter-token latency, throughput | QoS constraint satisfaction |
| Host CPU utilisation attributable to KV movement | The control/data-plane claim C3 (§M.3.2) |
| Per-tenant QoS attainment under contention | Admission fairness |
All of these are obtainable on host DRAM over PCIe before any CXL hardware is procured. That is the point of listing them: none of the open questions in §M.3.2 requires a purchase order to begin answering.
M.6 What This Specification Settles and Does Not
| Status | Content |
|---|---|
| Now measured (was assumed) |
The decode cost model. §M.4.4's objective assumed a single memory bandwidth. Measurement on GB10 shows two, with KV read at ~26% of achievable bandwidth against 87% for weights — a ~3.4× model-derived effective-path ratio, with similar KV slopes across two tested frameworks and two attention kernels. The objective's byte-counting terms should therefore be weighted. See Appendix L. |
| Settles | What a KV object is; which metadata migrates and which is updatable in place; where policy lives versus mechanism; what is being minimised subject to what; how the lossless/lossy boundary is expressed formally (§M.4.2); and how movement is measured without the accounting error (§M.5.1). |
| Does not settle | Whether device-side residency of the mechanism is justified — §M.3.2 lists the four falsifiable claims; whether persistent policy state changes outcomes materially; what the workload-specific amplification floors are; and whether any of this is novel, which requires a systematic literature review that nothing in this package constitutes. |
The cost model is now partly measured; the policy claims are not. Appendix L gives measured bandwidth terms on one machine. Everything about admission, residency budgets, metadata persistence and device-side control remains specification, not result — including the simulation evidence in Appendix N, which is Simulated (provisional) and has already been wrong once. The value of this appendix is that the remaining questions are now stated precisely enough to be settled by experiment rather than by argument.