Where prefill fits relative to the decode-phase measurements in Chapter 4, why prefill produces the KV cache that decode then reuses, and an honest scoping of what this package does and does not cover on the preprocessing side.
A note on scope before anything else: this chapter is background and scoping, not a new measurement. The hardware-measured results in this package, reported in Chapter 4, are entirely about decode-phase KV bandwidth on a DGX Spark. Nothing in this chapter has been measured or simulated. Where a figure appears below, it is described as an estimate or a design target, never as a validated result.
Autoregressive LLM inference runs in two phases with very different performance characteristics. Prefill processes the input prompt: every token the user (or the application) supplied is run through the model in a single forward pass, computing attention over the full prompt at once. Because prefill operates on many tokens simultaneously, it maps onto large, dense matrix multiplications, and modern accelerators execute those efficiently. Prefill is generally compute-bound — throughput is limited by how fast the GPU can do matmuls, not by how fast it can move data.
Decode is the opposite. After prefill, the model generates output one token at a time, and each step depends on the token generated before it. Each decode step reads the entire KV cache accumulated so far — for every prior token in the sequence — to compute attention for the single new token being produced. This is the operation that is memory-bandwidth-bound rather than compute-bound: the arithmetic per step is small, but the volume of KV data that must be read from memory grows with sequence length. Decode-phase KV bandwidth is the entire subject of Chapter 4's measurements and the reason this package exists.
The practical consequence is that these two phases stress different parts of the system and are usually reasoned about separately. A platform can be excellent at prefill and mediocre at decode, or vice versa, and the two do not trade off cleanly against each other in a single number.
Prefill is not just "the compute-bound phase" — it is also where the KV cache begins. As the prompt is processed, the model computes a key and value vector for every attention head, at every layer, for every prompt token. Those key/value pairs are written into the KV cache once, during prefill, and then read back — not recomputed — at every subsequent decode step for the rest of the sequence. The KV cache that Chapters 3 through 5 discuss moving across memory tiers is, at its origin, a byproduct of prefill.
This "compute once, read many times" property is exactly why prefix caching is valuable: when two requests share an identical prompt prefix — a system prompt, a few-shot template, the shared opening of a longer document — the KV cache computed for that prefix during one request's prefill can be reused directly by another request, skipping prefill recomputation for the shared portion entirely. This is a well-established technique at the inference-framework level; vLLM's automatic prefix caching and SGLang's RadixAttention are two widely used examples of systems that implement it. Both are general-purpose framework capabilities, not something evaluated or reproduced in this package — they are mentioned here only to place prefill and the KV cache it produces in context.
Beyond prefill itself, a broader class of preprocessing work sits in front of the model: tokenizing raw text, and for multimodal inputs, encoding images or audio into the embeddings the model consumes. One architectural idea explored early in this project's thinking was running some of that preprocessing on endpoint- or controller-side compute — silicon closer to where the CXL-attached memory tier lives — rather than on the main GPU, on the theory that keeping the GPU focused on prefill and decode compute while a separate component handles tokenization and encoding could reduce GPU-side overhead.
That idea remains exactly that: a design target. No such endpoint silicon exists in any system evaluated in this package, and nothing about its performance has been measured, simulated, or benchmarked here. An earlier version of this documentation attached a specific speedup figure to this idea; that figure was an early, unvalidated estimate and is not restated as a claim anywhere in this package. Readers should treat preprocessing offload as an open architectural direction worth further investigation, not as a capability with demonstrated performance.
To state the scope plainly: the measurement work in Chapter 4 is entirely about decode-phase KV cache bandwidth — reading previously computed KV data from GPU memory during token-by-token generation. Prefill-phase compute costs, prefill latency, tokenization and multimodal encoding costs, and any form of preprocessing offload are outside that measured evidence base. Nothing in Chapters 3, 5, 7, or elsewhere in this package that mentions prefill or preprocessing should be read as validated by the Chapter 4 hardware results — those results speak to decode bandwidth alone, and this chapter's purpose is to keep that boundary clear.