Skip to main content
CPU offloading moves KV cache off the GPU into the node’s system memory (host DRAM). This is the engine’s L1 tier — the first place cached KV lands once it no longer fits in GPU HBM — and it’s on by default, configured with a single value: engine.spec.l1.sizeGB. That value reserves host DRAM (not GPU memory) on each node; GPU HBM stays owned by vLLM’s own paged KV cache, so the operator never competes with it. It reuses prompt prefixes from host RAM, keeping GPU memory free for active batches, with no extra storage or dependencies.

​
The cache tiers

Hot data is promoted toward the GPU; cold data spills toward disk: CPU offloading alone gives the core benefit: a prefix computed once is reused from RAM instead of recomputed on the GPU. Add external storage only when your working set outgrows DRAM or you need reuse to survive restarts.

​
Configuration

CPU offloading reserves 60 GB of host DRAM per node by default. To set it explicitly:
my-values.yaml
l1.sizeGB applies per node — every engine pod reserves this much DRAM on the node it runs on. Size it to your workload (next) rather than relying on the default.

​
Sizing

Size the cache to hold your hot working set — the prefixes reused often. Too small and warm entries evict before reuse; sized right and repeat traffic stays in memory. A practical loop: size it to a comfortable share of each node’s free RAM — leaving headroom for vLLM and the OS — then watch the hit rate and L1 read/write/eviction counters in Metrics. Sustained high writes and evictions with low reads mean the hot set isn’t fitting — grow l1.sizeGB. To convert a token count to GB for a model, use the KV Cache Size Calculator.

​
Eviction

When the cache fills, the engine evicts. Tune the watermark and drop ratio in the engine spec:
With external storage configured, evicted entries spill there instead of being dropped — so a warm prefix pushed out of DRAM can reload from disk rather than recompute.

​
Verify

Send repeated traffic, then confirm L1 reads climb while writes level off — via lmcache_mp_l1_read_chunks_total, _write_chunks_total, and _evicted_chunks_total. See Metrics for the PromQL and healthy-vs-concerning patterns.

​
When it helps

CPU offloading pays off in proportion to how much your prompts repeat — shared system prompts, few-shot preambles, multi-turn chat, or RAG where the same context recurs. Mostly-unique, one-off prompts have nothing to reuse, so the cache just fills and evicts without benefit. Two things also gate whether a prefix is cached at all:
  • Prompts shorter than the engine’s server.chunkSize (default 256 tokens) store nothing.
  • A prefix must be reused before it evicts — keep the hot set resident by sizing the cache well, and add external storage when it outgrows DRAM.

​
Next steps

External storage (L2)

Add a disk or remote tier behind host DRAM for larger working sets and restart survival.

Metrics

Right-size the cache from the hit rate and read/write counters on real traffic.