Figure: where the per-block policy state of the PLACE stage should live, across four candidate placements in a memory hierarchy. The tracked unit is a (sequence, token block) pair — 16 tokens, 5.00 mebibytes — not a head: every head of every layer is required for every token, so a head is a scoring granularity, never an eviction granularity. A, GPU HBM: 192 GB at 8 TB/s — lowest-latency decisions, but policy state dies with the block when HBM evicts it, which is exactly the failure the persistence result identifies. B, host CPU: DDR5 at about 400 GB/s over NVLink or PCIe 5.0 — easy to implement, survives HBM eviction, but adds a host round trip per decision. C, CXL or tier controller: state sits with the tier, so it survives HBM eviction by construction; tier bandwidth is assumed, not measured, at 32, 64 or 121 gigabytes per second, under which a 5.00 mebibyte block transfer takes 164, 82 or 43 microseconds; the cost is custom hardware. D, computational storage on NVMe SSD at about 14 GB/s — offloads the host but sits on the highest-latency path. The placement tradeoff is dominated by one result: at a 32 gibibyte HBM budget, letting identity-keyed policy state survive HBM eviction raises LFU from 15.56 to 49.71 percent hit rate, a gain of 34.15 percentage points, and EMA alpha 0.01 from 31.54 to 56.39 percent, a gain of 24.85 percentage points, while EMA alpha 0.15 is unchanged at minus 0.21 percentage points. That is larger than the spread between scoring functions, and it is a protocol result, testable with no CXL hardware — which is why B, C and D are all viable and A alone is disqualified on the mechanism rather than on speed. Metadata payload is negligible: 64 bytes of state against a 5.00 mebibyte block is 0.0012 percent, or about 512 kibibytes of state for the 8,192 blocks of a 128K context whose KV is 40.0 gibibytes. Metadata processing overhead is not established and must be measured. Simulation values are provisional: a single data-structure defect in Revision 1 inverted four of five findings. No CXL hardware was measured.

Cache Management Placement

Where does the per-block policy state of PLACE live — and does it survive HBM eviction?

System Architecture
NVLink / PCIe 5.0
CXL.mem / NVMe-oF
GPU-Side
Where: CUDA kernel / HBM reservation
Lowest latency decisions
Attention scores already on GPU
Policy state dies with the block. A block evicted from HBM is re-admitted as if newly created — the exact reset worth −34.15 pts to LFU
Kernel complexity for async I/O
Host CPU
Where: Userspace daemon / driver
Easy to implement & debug
Flexible policy changes
State survives HBM eviction — the mechanism the result turns on, with no custom hardware
Host round-trip per decision
CPU becomes bottleneck at scale
Open question
CXL / Tier Controller
Where: CXL Type-2 device FPGA/ASIC
Near-memory processing
Intercepts all CXL.mem traffic
State lives with the tier, so it survives HBM eviction by construction
Custom hardware required
Whether the controller must be on the device is open. Of four falsifiable claims, only the fourth — policy state surviving eviction — needs device residency at all, and B already provides it
Computational Storage
Where: NVMe SSD controller ARM cores
Offloads host entirely
State survives every tier boundary above it
Highest latency path
Limited compute on SSD controller
What decides the placement: does policy state survive eviction?

The largest effect in the study is not the scoring function and not the latency of any placement. It is whether identity-keyed policy state survives the round trip out of HBM. That is a protocol property, not a hardware one — which is why it is testable with no CXL hardware, and why placements B, C and D all satisfy it while A does not.

Hit rate at a 32 GiB HBM budget with policy metadata reset on eviction versus persisting across it.
Policy Metadata reset (placement A) Metadata persists (B / C / D) Δ
LFU15.56%49.71%+34.15 pts
EMA (LRFU) α=0.0131.54%56.39%+24.85 pts
EMA (LRFU) α=0.1544.33%44.12%−0.21 pts

32 GiB HBM · Simulated (provisional). Standing caveat: a single data-structure defect in Revision 1 inverted four of five findings, so Revision 2 is provisional, not settled.

Metadata footprint — the tracked unit is a (sequence, token block), not a head

Every head of every layer is required for every token, so evicting a head produces wrong outputs, not a cache miss. Per-head signals are a scoring granularity; the unit that is placed, moved and evicted is a block of 16 tokens. H2O and SnapKV evict (head, token) pairs — the token axis is not optional.

Tracked unit
(seq, block)
16 tokens = 5.00 MiB
Blocks at 128K context
8,192
131,072 ÷ 16
State per block
64 B
identity key + score + touch time
Payload overhead
0.0012%
64 B against 5.00 MiB · ≈512 KiB against 40.0 GiB of KV

Analytical model. Payload overhead is negligible; metadata processing overhead is not established and must be measured.

Hybrid split — a design direction, not a recommendation
GPU computes attention and reports what it touched → the tier controller updates each block's score against state that survived the block's last eviction and decides the tier → transfers are issued asynchronously. Host owns policy; device owns mechanism. Scoring is LRFU (Lee et al., ACM SIGMETRICS 1999), established prior art, with α its documented LRU↔LFU knob — External literature. α = 0.2 was never validated; the simulated optimum is α ≈ 0.05, and α is non-monotonic (0.5→37.20, 0.3→38.84, 0.15→44.33, 0.05→55.96, 0.01→31.54, 0.003→25.47, 0.001→24.44), with no convergence onto LFU as α→0. Everything in this row except PLACE's persistence mechanism is Design / unvalidated.

Evidence labels. Capacities and the metadata footprint are an Analytical model; the persistence result is Simulated (provisional); the LRFU attribution is External literature; every tier latency on this page is an assumption. No CXL hardware was measured — the earlier 200 ns CXL access figure is withdrawn and demoted to an unmeasured assumption inside a 32–121 GB/s range, under which a 5.00 MiB block transfer is 43–164 µs. The tier's value is avoiding regeneration: forward compute for 16 tokens is 9 / 11 / 18 ms at 50 / 40 / 25% MFU, two to three orders of magnitude above the transfer — no precise multiple is claimed. There is no RoPE-driven prefetch in this design: RoPE is a rotation applied to Q/K at compute time and carries no information about which blocks are needed next.