Figure: preprocessing offload to the endpoint. Two data paths are compared. The traditional CPU-bound path runs storage on NVMe, then CPU DRAM on DDR5, then CPU preprocessing on x86 cores, then GPU DRAM over PCIe, with the two CPU stages marked as the bottleneck. The endpoint path runs endpoint flash, endpoint DRAM, ARM preprocessing on embedded cores, CXL.mem directly, then GPU HBM, so there is no per-access CPU involvement in steady state, though control-plane setup, address mapping, fault handling, drivers and endpoint firmware still run on the host. The offloaded tasks are tokenization with SentencePiece or BPE, image decode and normalization, FP32 to FP16 or BF16 conversion, and batching and padding. The endpoint is modeled with 4 to 16 Cortex-A cores at 1.5 to 2.5 GHz, about 500 thousand tokens per second of tokenization and about 1000 images per second at 224 by 224. The 5 to 10 times figure is a design target, not a benchmark, and is scoped to tokenization and image decode against a single host CPU core. All values are an analytical model, not measured.

Section 6

Preprocessing Offload

Move data preparation to the endpoint — no per-access CPU involvement in steady state

Traditional Path
CPU Bound
Endpoint Path
Direct to GPU
No per-access CPU involvement in steady state — the host CPU is off the per-token data path, but it is not removed from the system. Control-plane setup, address mapping and CXL region programming, page-fault handling, device drivers and endpoint firmware all still run on and through the host.
Preprocessing Tasks for Embedded ARM
Tokenization
SentencePiece, BPE
Image Decode
JPEG/PNG + normalize
Format Conversion
FP32 → FP16/BF16
Batching & Padding
Sequence alignment
Embedded ARM Cores
Throughput depends on core count and clock speed — design target, unbenchmarked
Core Count
4–16 Cortex-A cores
Clock Speed
1.5–2.5 GHz
Tokenization Rate
~500K tokens/sec
Image Throughput
~1000 img/sec (224×224)
5–10× — DESIGN TARGET, NOT A BENCHMARK. No hardware measurement has been run. The target is scoped narrowly to tokenization (SentencePiece/BPE) and image decode + normalization executed on the endpoint's ARM cores, compared against the same work on a single host CPU core. It is not a claim about end-to-end inference throughput, and it does not generalize to the other preprocessing tasks shown above. The rates above (≈500K tokens/s, ≈1000 img/s) are modeled from core count and clock, not observed.
Key Insight: Direct Data Path
ARM
CXL.mem
GPU
Once a region is mapped, data reaches the GPU-accessible CXL.mem region with no per-access CPU involvement in steady state: no per-token host copies, no per-transfer DMA descriptor setup, no completion interrupt per access. What remains on the CPU: initial region mapping and enumeration, control-plane setup and teardown, address translation and fault handling, drivers, and the endpoint firmware that must be loaded and managed.

Analytical model / design target — no hardware benchmark was run. Throughput and speedup figures on this page are modeled, not measured. Canonical numbers v4.0.