A case study of one mechanism inside the PLACE stage of the control-plane architecture (Chapter 3). Trace-driven simulation, provisional and not settled — but the argument it supports does not depend on the simulator being right about everything.
kv_tiering_sim_v2.py against results_v2.json — Revision 2 of a trace-driven simulation. Revision 1 shipped with a data-structure defect in the eviction-metadata bookkeeping that inverted four of its five headline findings; Revision 2 is the corrected rebuild. Treat every number here as provisional, not settled: it has not been validated against a real CXL memory pool, and it is a second revision of a simulator, not a third-party-audited result. The chapter's central claim is stated so that it does not depend on any single number in these tables surviving further correction — see §7.5.
It is tempting to read this chapter as pitching a new eviction-scoring function. It is not, and being precise about that matters. The scoring function used throughout — an exponential moving average of access recency and frequency, with a decay constant α that interpolates between recency-only and frequency-only behavior — is LRFU, published by Lee, Kim, Noh, Min, Cho, and Kim at SIGMETRICS 1999. The α knob's role as an LRU↔LFU interpolation parameter is documented in that paper, not discovered here. Nothing in this chapter claims novelty for the scoring function itself. EMA, LRU, and LFU are used here strictly as past-only signal baselines — one instance of the broader signal taxonomy Chapter 3 lays out for the PLACE stage — not as the architecture's central mechanism. The architecture does not depend on any one of them; it depends on whether the signal a residency policy chooses, whatever that signal is, keeps its state across a tier crossing.
The actual claim is about where policy state lives and what happens to it when a block leaves HBM. In a conventional tiered-KV design, a block evicted from HBM is simply gone as far as the eviction policy is concerned; when it (or its logical successor — the same session, the same prefix position) comes back, the controller has no record of it and admits it as if newly created, with a fresh recency/frequency score starting from zero. This chapter's contribution is to treat the KV state and its control metadata — recency/frequency score, identity key — as one migratable object, so that identity-keyed policy state survives the round trip between HBM and a slower tier instead of being reset on every eviction. The host owns the policy (what score a block has, when it should be evicted); the device owns the mechanism (moving bytes, honoring the eviction decision). That division, and the persistence of policy state across the tier boundary, is the PLACE-stage mechanism the rest of this chapter defends — a narrower claim than "the architecture," and one that says nothing about SELECT (what enters candidacy) or MOVE (how bytes cross the physical hierarchy).
Results below are drawn from a single simulation configuration unless a table says otherwise: Llama-3 70B under grouped-query attention, 16-token KV blocks (5.00 MiB/block), 200 synthetic sessions with Zipf(1.1)-distributed return frequency, context lengths from 1K to 12K tokens sharing a 2048-token common prefix, run for 2500 turns, for a total working-set footprint of 421 GiB against an HBM tier sized from 8 to 128 GiB backed by a 512 GiB CXL tier. Each configuration was run across 2–3 seeds; seed counts are noted per table since they differ between the α-sweep and the other experiments.
With no persistent metadata across eviction (the conventional case), EMA scoring at α=0.15 beats plain LRU and LFU at four of five HBM budgets tested, with EMA α=0.01 taking the largest budget. LFU degrades badly under decode-driven growth: freshly written blocks with a count of one are evicted in favor of stale blocks that accumulated high counts earlier in the session.
| HBM budget | LRU | LFU | EMA α=0.15 | EMA α=0.01 |
|---|---|---|---|---|
| 8 GiB | 8.52 | 8.76 | 12.35 | 12.12 |
| 16 GiB | 17.87 | 12.70 | 22.58 | 18.70 |
| 32 GiB | 38.34 | 16.09 | 44.18 | 35.91 |
| 64 GiB | 67.12 | 23.00 | 69.65 | 66.83 |
| 128 GiB | 88.01 | 32.70 | 88.14 | 91.59 |
EMA's margin over LRU ranges from +2.5 to +5.8 points across this sweep — a real but modest edge, and one entirely consistent with LRFU's known behavior in the literature. Nothing here is the headline result of this chapter.
A separate experiment (e2, different baseline set from §7.3 — do not read these rows against Result 1's rows) fixed HBM at 32 GiB and swept α across seven values plus pure LRU and LFU endpoints.
| Policy | Hit rate |
|---|---|
| LFU | 15.56 |
| EMA α=0.5 | 37.20 |
| LRU | 36.85 |
| EMA α=0.3 | 38.84 |
| EMA α=0.15 | 44.33 |
| EMA α=0.05 | 55.96 (optimum) |
| EMA α=0.01 | 31.54 |
| EMA α=0.003 | 25.47 |
| EMA α=0.001 | 24.44 |
α≈0.05 is the optimum, +19.1 points over LRU — but the curve is not monotonic and does not converge onto LFU as α shrinks toward zero. If EMA were simply "LFU with a knob," hit rate should climb steadily as α→0. Instead it peaks at α≈0.05 and then collapses through α=0.01, 0.003, 0.001, ending up worse than LRU. Section 7.5 explains why: the collapse at small α is not a property of the scoring function, it is an artifact of resetting metadata on every eviction.
The single most important experiment in this chapter holds the scoring function fixed and varies only whether identity-keyed policy state survives eviction from HBM, versus being reset — i.e., the block re-admitted as though newly created — every time it re-enters HBM. This is the one result in this case study that speaks directly to the PLACE-stage residency question posed in the scope note above.
| Policy | Metadata reset on eviction | Metadata persists | Δ |
|---|---|---|---|
| LFU | 15.56 | 49.71 | +34.15 |
| EMA α=0.01 | 31.54 | 56.39 | +24.85 |
| EMA α=0.15 | 44.33 | 44.12 | −0.21 |
Read plainly: the gain from letting policy state survive eviction (+34.15 points for LFU) exceeds the entire spread between different eviction policies at any HBM budget tested in §7.3 — where the best-to-worst gap between LRU, LFU, and EMA topped out around 45 points at the low end and was often under 10. Persistence is worth more than the choice of policy. This also resolves the small-α collapse from §7.4: LFU and low-α EMA are precisely the policies that depend most on long-run accumulated counts, so resetting that history on every eviction punishes them hardest — the collapse at α=0.01 and below was a metadata-reset artifact, not evidence that small α is a bad interpolation setting.
This is also, specifically, a protocol result, in Chapter 3's terms: it requires no CXL hardware to produce, because it is a claim about what state a tiering controller chooses to preserve across an eviction event, not about the physical medium the evicted block lands on. That distinction is the subject of §7.7. It is also, currently, the only piece of the Chapter 3 control-plane architecture that has simulated evidence behind it at all — SELECT (which candidate regions and blocks are worth considering for movement) and MOVE (how bytes actually cross HBM → CXL DRAM → NVMe/remote) remain unvalidated design proposals in Chapter 3, not results this chapter or any other in this package has tested.
Held at 32 GiB HBM across three seeds, the same four policies were run against three workload shapes: returning sessions (Zipf), a one-shot flood of new sessions with no returns (scan), and a cyclic sweep that revisits the same sessions in a fixed rotation (loop).
| Workload | LRU | LFU | EMA α=0.15 | EMA α=0.01 |
|---|---|---|---|---|
| zipf (returning sessions) | 39.08 | 15.12 | 44.44 | 38.87 |
| scan (one-shot flood) | 31.51 | 12.52 | 32.57 | 36.26 |
| loop (cyclic sweep) | 9.83 | 22.06 | 9.83 | 9.83 |
Under the cyclic sweep, LFU is the only policy of the four that does meaningfully better than the others — recency-based signals are actively wrong when the access pattern is a fixed rotation, since the least-recently-used block is exactly the one about to be reused. No single past-only-signal policy dominates across all three shapes tested. This is a second reason the persistence result in §7.5 matters more than any specific policy recommendation: whichever past-only scoring function a deployment picks, or switches to per workload, the gain from letting its state survive eviction appears to hold independent of that choice.
The residency question this case study is built around — should policy state persist across a tier crossing rather than being reset — decomposes into four separable, falsifiable sub-claims. Being explicit about which of these actually requires physical hardware to test, and which do not, is a strength of this chapter's argument, not a hedge on it.
Sub-claims 1, 2, and 4 can be tested, and are tested in this chapter, entirely in simulation: they are statements about scoring behavior and about where time goes in a turn, not about a physical transport. Only sub-claim 3 — whether persisting policy state across a real tier boundary produces the same gain observed in §7.5 on hardware with real transfer latency, real controller overhead, and a real device-side eviction path — actually requires a CXL memory pool to test. If sub-claims 1, 2, and 4 fail to hold up under further scrutiny, the contribution of this work is still the PLACE-stage mechanism described in §7.1: making KV state and its control metadata a single migratable object with policy state that survives the round trip. CXL is one place that mechanism could run. It is not the point of the claim, and it is not a claim about SELECT or MOVE.
Across the budget and policy sweep, recompute — regenerating a block's KV state from scratch because it missed the cache — accounts for 44–89% of turn stall time, not a single fixed figure; the share moves with HBM budget and policy. Under an idealized model where transfer and compute are fully overlapped, fetch time for a resident block falls to 0.35–1.25 ms per turn, and recompute's share of stall rises to over 99%. Once data movement is hidden behind compute, recompute avoidance is the only term left that matters — which is another way of stating why hit rate, and specifically the persistence result in §7.5, is the quantity this case study is trying to move.
This chapter is scoped narrowly, and it is worth being explicit about the boundary. Everything above uses past-only signals — recency, frequency, decayed-frequency scores — to decide what stays resident and what gets evicted. It does not test predictive signals: agent or tool-call state, cross-turn reuse history, attention patterns, or any other forward-looking indicator of which regions and blocks are about to matter, the direction Chapter 3 lays out for the PLACE stage and that systems like CacheWise (Chapter 10) attempt to exploit. Whether predictive, agent-aware residency policies beat the past-only baselines measured here — and whether persistence of policy state across a tier boundary matters as much for predictive signals as it does for the past-only ones in §7.5 — is open work for the SELECT stage of the architecture, not something this chapter's simulator was built to answer. Readers should not read the +34.15-point result in §7.5 as evidence about predictive-signal residency policies; it is evidence about past-only ones only.
This chapter does not claim a better eviction-scoring function, and it does not claim to be the architecture's conceptual core — it is a case study of one PLACE-stage mechanism from Chapter 3. EMA/LRFU scoring is 1999-vintage published work, used here strictly as a past-only signal baseline, and its α parameter's LRU↔LFU interpolation role is documented, not new here. What it argues, and what the persistence experiment in §7.5 supports, is that treating KV state and its control metadata as a single object whose identity-keyed policy state survives the HBM↔slower-tier round trip is worth more than the choice of past-only eviction policy running on top of it — testable independent of the physical medium, of which only one falsifiable component (§7.7, sub-claim 3) actually requires CXL hardware to confirm. It is currently the only piece of the Chapter 3 control-plane architecture with simulated evidence behind it; SELECT and MOVE remain unvalidated design proposals, and the predictive-signal direction (§7.9) remains untested here entirely. Everything in this chapter is Revision 2 of a trace-driven simulator, provisional, and awaiting hardware validation and a measurement of controller-side processing overhead before any of it should be treated as settled.