> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CPU Offloading

> Offload KV cache from GPU memory into host DRAM — the default cache tier.

**CPU offloading** moves KV cache off the GPU into the node's system memory (host DRAM). This is
the engine's L1 tier — the first place cached KV lands once it no longer fits in GPU HBM — and it's
on by default, configured with a single value: `engine.spec.l1.sizeGB`. That value reserves host
DRAM (not GPU memory) on each node; GPU HBM stays owned by vLLM's own paged KV cache, so the
operator never competes with it.

It reuses prompt prefixes from host RAM, keeping GPU memory free for active batches, with no extra
storage or dependencies.

## The cache tiers

Hot data is promoted toward the GPU; cold data spills toward disk:

| Tier                      | Lives in            | Speed   | Configured by                                                   |
| ------------------------- | ------------------- | ------- | --------------------------------------------------------------- |
| **GPU HBM (L0)**          | vLLM paged KV cache | Fastest | vLLM                                                            |
| **CPU offloading (L1)**   | Host DRAM           | Fast    | `engine.spec.l1.sizeGB`                                         |
| **External storage (L2)** | Disk / NFS / remote | Slower  | [`engine.spec.l2Backend`](/configuration/filesystem-offloading) |

CPU offloading alone gives the core benefit: a prefix computed once is reused from RAM instead of
recomputed on the GPU. Add [external storage](/configuration/filesystem-offloading) only when your
working set outgrows DRAM or you need reuse to survive restarts.

## Configuration

CPU offloading reserves 60 GB of host DRAM per node by default. To set it explicitly:

```yaml my-values.yaml theme={null}
engine:
  enabled: true
  spec:
    l1:
      sizeGB: 120            # host DRAM reserved for KV cache, per node
    image:
      repository: lmcache/vllm-openai
      tag: v<version>
      pullPolicy: IfNotPresent
```

```bash theme={null}
helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version <version> -n tensormesh-operator --create-namespace \
  -f my-values.yaml --wait
```

`l1.sizeGB` **applies per node** — every engine pod reserves this much DRAM on the node it runs on.
Size it to your workload (next) rather than relying on the default.

## Sizing

Size the cache to hold your **hot working set** — the prefixes reused often. Too small and warm
entries evict before reuse; sized right and repeat traffic stays in memory.

A practical loop: size it to a comfortable share of each node's free RAM — leaving headroom for
vLLM and the OS — then watch the hit rate and L1 read/write/eviction counters in
[Metrics](/observability/metrics). Sustained high writes and evictions with low reads mean the hot
set isn't fitting — grow `l1.sizeGB`. To convert a token count to GB for a model, use the
[KV Cache Size Calculator](https://docs.lmcache.ai/getting_started/kv_cache_calculator.html).

## Eviction

When the cache fills, the engine evicts. Tune the watermark and drop ratio in the engine spec:

```yaml theme={null}
engine:
  spec:
    eviction:
      triggerWatermark: 0.8     # start evicting at 80% full
      evictionRatio: 0.2        # drop 20% when triggered
```

With [external storage](/configuration/filesystem-offloading) configured, evicted entries spill
there instead of being dropped — so a warm prefix pushed out of DRAM can reload from disk rather
than recompute.

## Verify

```bash theme={null}
kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine
```

Send repeated traffic, then confirm L1 reads climb while writes level off — via
`lmcache_mp_l1_read_chunks_total`, `_write_chunks_total`, and `_evicted_chunks_total`. See
**[Metrics](/observability/metrics)** for the PromQL and healthy-vs-concerning patterns.

## When it helps

CPU offloading pays off in proportion to how much your prompts **repeat** — shared system prompts,
few-shot preambles, multi-turn chat, or RAG where the same context recurs. Mostly-unique, one-off
prompts have nothing to reuse, so the cache just fills and evicts without benefit.

Two things also gate whether a prefix is cached at all:

* Prompts shorter than the engine's `server.chunkSize` (default **256 tokens**) store nothing.
* A prefix must be reused **before it evicts** — keep the hot set resident by sizing the cache well,
  and add [external storage](/configuration/filesystem-offloading) when it outgrows DRAM.

## Next steps

<CardGroup cols={2}>
  <Card title="External storage (L2)" icon="hard-drive" href="/configuration/filesystem-offloading">
    Add a disk or remote tier behind host DRAM for larger working sets and restart survival.
  </Card>

  <Card title="Metrics" icon="gauge-high" href="/observability/metrics">
    Right-size the cache from the hit rate and read/write counters on real traffic.
  </Card>
</CardGroup>