> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# External Storage Offloading
> Use a filesystem-backed L2 cache with the Tensormesh Operator by mounting storage into the engine pod and configuring the raw L2 adapter.
Filesystem offloading means using LMCache's **filesystem-backed L2 adapter** as a backing store
behind the in-memory L1 cache. In operator terms, this is an `engine.spec.l2Backend.raw`
configuration plus a mounted path inside the engine pod.
Use this when you want:
* a simple on-cluster L2 tier without Redis or object storage
* warm reuse across engine restarts
* a shared POSIX-style backing store, if your cluster can provide one
## What this is in the operator model
The chart does **not** expose a first-class `filesystem.enabled=true` value.
Filesystem offloading is configured through the LMCacheEngine CR passthrough:
* `engine.spec.l2Backend.raw`
* `engine.spec.volumes`
* `engine.spec.volumeMounts`
That is because filesystem L2 is currently treated as a **raw adapter** rather than a typed
operator field like `l2Backend.resp`.
## Choose your storage
The engine needs a writable directory at the L2 `base_path`. There are two ways to give it one.
Pick with this:
| Use | When |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| **A host directory** (`hostPath`) | Your nodes have local disk and you are happy for each node to keep its own cache. **This covers most clusters.** |
| **A PersistentVolumeClaim** | Any of: nodes are ephemeral (managed/autoscaled clusters), you want one node to reuse what another cached, or your cluster policy blocks `hostPath`. |
One command tells you whether a PVC is even possible here:
```bash theme={null}
kubectl get storageclass
```
If that prints `No resources found`, the cluster has no dynamic provisioning and **a PVC can never
bind** — use Option A.
## Option A — host directory (recommended)
Mount a real directory from the node. Nothing to create first, nothing to provision.
```yaml my-values.yaml theme={null}
engine:
spec:
volumes:
- name: lmcache-l2
hostPath:
path: /mnt/nvme0/lmcache-l2 # a path with free space, on every GPU node
type: DirectoryOrCreate
volumeMounts:
- name: lmcache-l2
mountPath: /data/lmcache/l2
```
Two things to get right:
* **The path must exist, or be creatable, on every node that runs an engine pod.**
`DirectoryOrCreate` makes it if missing. If one node lacks the disk, that pod fails to mount.
* **Each node keeps its own cache.** Nothing is shared between nodes. That is the right shape for
extending capacity beyond RAM; it is the wrong shape if you want cross-node reuse — use Option B
with `ReadWriteMany` for that.
That is the whole setup. Skip to [Minimal example](#minimal-example).
## Option B — PersistentVolumeClaim
Use this when Option A does not fit. **The operator does not create the volume for you** — the
claim must exist before the engine pods start, or they stay `Pending` forever.
**1. Pick a storage class.** Names are cluster-specific; there is no universal default.
```bash theme={null}
kubectl get storageclass
```
**2. Create the claim**, substituting a name from that list.
```yaml lmcache-l2-pvc.yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lmcache-l2
namespace: tensormesh-operator
spec:
accessModes:
- ReadWriteOnce # ReadWriteMany to share across nodes
resources:
requests:
storage: 500Gi # size to your L2 working set
storageClassName: REPLACE_ME # a name from `kubectl get storageclass`
```
If one class is marked `(default)`, you can omit `storageClassName` and get that one.
**3. Apply it and wait for `Bound`.**
```bash theme={null}
kubectl apply -f lmcache-l2-pvc.yaml
kubectl get pvc -n tensormesh-operator
# NAME STATUS VOLUME CAPACITY ACCESS MODES AGE
# lmcache-l2 Bound pvc-... 500Gi RWO 5s
```
**4. Reference it from the engine.**
```yaml my-values.yaml theme={null}
engine:
spec:
volumes:
- name: lmcache-l2
persistentVolumeClaim:
claimName: lmcache-l2
volumeMounts:
- name: lmcache-l2
mountPath: /data/lmcache/l2
```
**One `ReadWriteOnce` claim cannot serve more than one engine pod.** The engine runs as a
DaemonSet, so a multi-node cluster gets one engine pod per GPU node. An RWO claim binds to a
single node, and every other engine pod stays `Pending`.
For more than one engine pod, use a `ReadWriteMany` storage class, or create one claim per node.
## Minimal example
This uses Option A (a host directory). Swap the `volumes` block for the PVC version if you chose
Option B — everything else is identical.
```yaml my-values.yaml theme={null}
engine:
enabled: true
spec:
l1:
sizeGB: 60
image:
repository: lmcache/vllm-openai
tag: v0.4.5
pullPolicy: IfNotPresent
l2Backend:
raw:
type: fs
config:
base_path: /data/lmcache/l2
storePolicy: default
prefetchPolicy: default
prefetchMaxInFlight: 8
volumes:
- name: lmcache-l2
hostPath:
path: /mnt/nvme0/lmcache-l2
type: DirectoryOrCreate
volumeMounts:
- name: lmcache-l2
mountPath: /data/lmcache/l2
```
Apply it with:
```bash theme={null}
helm upgrade --install tensormesh-operator \
oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
--version 0.4.5 \
-n tensormesh-operator --create-namespace \
-f my-values.yaml --wait
```
## What the fields mean
* `l2Backend.raw.type: fs`
* selects the filesystem-backed L2 adapter
* `l2Backend.raw.config.base_path`
* the directory inside the engine container where LMCache stores L2 files
* `volumes` / `volumeMounts`
* make that path real and writable inside the engine pod
* `storePolicy: default`
* keep normal L1 behavior and also store evicted/eligible keys into L2
* `prefetchPolicy: default`
* allow misses to load data back from L2 into L1
* `prefetchMaxInFlight`
* caps concurrent L2-to-L1 loads to avoid flooding L1 memory
## Recommended storage choices
### Shared cache across nodes
Use a storage class or backing system that gives you:
* `ReadWriteMany`
* the same mounted path in every engine pod
* enough throughput for concurrent reads during warm prefills
Examples:
* NFS
* EFS / Filestore / Azure Files
* CephFS
This is the right shape when you want one node to benefit from data another node stored.
### Fast local spill on each node
Use:
* local SSD / NVMe
* hostPath
* `ReadWriteOnce` PVC pinned to the node
This is still useful if your main goal is to extend capacity beyond RAM on a single node, but it is
not a cross-node shared L2.
## Optional `fs` adapter knobs
The `fs` adapter accepts a few useful extra fields:
```yaml theme={null}
l2Backend:
raw:
type: fs
config:
base_path: /data/lmcache/l2
relative_tmp_dir: tmp
read_ahead_size: 1048576
use_odirect: false
```
What they do:
* `relative_tmp_dir`
* subdirectory under `base_path` for temporary write files
* `read_ahead_size`
* issue a small initial read to encourage filesystem readahead
* `use_odirect`
* bypass the OS page cache; only use this if you understand the alignment/performance tradeoff
If you need adapter-specific features beyond these, pass them through under `raw.config`. The
operator forwards that JSON to LMCache as-is.
## Persistence and restart behavior
Filesystem L2 is useful because it survives engine pod restarts as long as the mounted storage
survives.
That means:
* `emptyDir` is **not** a good choice if you want persistence across pod restarts
* a PVC or durable host-backed path is the right choice if restart survival matters
## Storage Sizing
Sizing is the next question after “should I use filesystem offloading at all?”
The fastest way to pick L1 and L2 capacities is the interactive
**[Cache Sizing](/configuration/cache-sizing)** guide: paste a workload trace and it
simulates LRU reuse distance to show the L1/L2 hit rate you would get at any capacity, measured in
characters, tokens, or prefix hashes.
To convert those units into bytes of cache, the upstream
[LMCache KV Cache Size Calculator](https://docs.lmcache.ai/getting_started/kv_cache_calculator.html)
gives the per-token KV size for a given model.
### L1 sizing
Use L1 when you want the fastest possible warm-hit behavior. In the operator, L1 is:
* `engine.spec.l1.sizeGB`
General guidance:
* size L1 for the hot working set you expect to be reused frequently
* if L1 is too small, warm entries will evict before they are reused
* if L1 comfortably holds the hot set, repeat traffic can stay in memory and avoid L2 reads
To size L1 to your workload's hot set, use the
[Cache Sizing](/configuration/cache-sizing) simulator and set `l1.sizeGB` near the knee of
the hit-rate curve.
### L2 sizing
Use L2 when you want capacity beyond RAM or warm reuse across restarts and, if storage is shared,
across nodes.
General guidance:
* size L2 for the larger working set that does not fit in L1
* if L2 is too small, you will churn older entries and lose warm benefit between bursts
* durable L2 is most useful when prompt reuse is real and frequent enough to pay back the extra I/O
The [Cache Sizing](/configuration/cache-sizing) simulator shows how much extra hit rate each
additional GB of L2 buys for your trace — size L2 to where the curve reaches the hit rate you are
willing to pay for.
### Expected performance by hit rate
The performance you should expect depends heavily on where hits land:
* mostly **L1 hits**
* best TTFT improvement
* lowest warm-request latency
* mostly **L2 hits**
* still useful, but slower than L1 because data must be loaded back into memory
* mostly **misses**
* little or no cache benefit
To estimate the L1/L2/miss split for your own traffic, simulate it in the
[Cache Sizing](/configuration/cache-sizing) guide.
## Verification
After rollout:
```bash theme={null}
kubectl get pvc -n tensormesh-operator
kubectl get lmcacheengine -n tensormesh-operator
kubectl describe lmcacheengine -n tensormesh-operator
kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine
kubectl logs -n tensormesh-operator -l app.kubernetes.io/component=cache-engine --tail=200
```
What to check:
* the PVC is `Bound`
* the engine pod is `Running`
* the mounted path exists and is writable
* the engine does not fail while parsing `--l2-adapter`
* warm requests eventually produce L2-related store/load activity rather than only L1 behavior
If you want a stronger functional check, combine this with a small L1 and repeated long prompts so
you can force eviction and subsequent reload from L2.
## Engine pods stay `Pending`
If engine pods never leave `Pending`, check the scheduler's reason. There are no container logs to
read yet, so `kubectl logs` returns nothing — the explanation is in the pod's events:
```bash theme={null}
kubectl describe pod -n tensormesh-operator | tail -20
```
**`persistentvolumeclaim "lmcache-l2" not found`**
```
Warning FailedScheduling default-scheduler
0/2 nodes are available: persistentvolumeclaim "lmcache-l2" not found.
```
The claim referenced by `engine.spec.volumes` doesn't exist. Either create it
([Option B](#option-b--persistentvolumeclaim)) — the pending pods then schedule on their own once it
reports `Bound`, with no restart or re-apply — or switch to a host directory
([Option A](#option-a--host-directory-recommended)), which needs no claim at all.
**Some engine pods `Pending` while others run**
A `ReadWriteOnce` claim binds to a single node, so exactly one engine pod can use it. Every other
engine pod waits forever. Switch to a `ReadWriteMany` storage class, or give each node its own
claim.
**PVC stuck `Pending` with `storageclass ... not found`**
```
Warning ProvisioningFailed persistentvolume-controller
storageclass.storage.k8s.io "standard" not found
```
The `storageClassName` in the PVC names a class this cluster does not have. Class names are
cluster-specific — list yours with `kubectl get storageclass` and use one of those. Note that
`storageClassName` is immutable, so the PVC has to be deleted and re-created rather than edited:
```bash theme={null}
kubectl delete pvc lmcache-l2 -n tensormesh-operator
# fix storageClassName, then
kubectl apply -f lmcache-l2-pvc.yaml
```
**`pod has unbound immediate PersistentVolumeClaims`**
The PVC exists but is still `Pending` rather than `Bound`. Check that its `storageClassName` matches
a class that exists and has a working provisioner:
```bash theme={null}
kubectl get pvc -n tensormesh-operator
kubectl describe pvc lmcache-l2 -n tensormesh-operator | tail -10
kubectl get storageclass
```
## Common mistakes
* referencing `claimName: lmcache-l2` without creating the PVC
* engine pods stay `Pending` with `persistentvolumeclaim "lmcache-l2" not found`
* sharing one `ReadWriteOnce` claim across a multi-node engine DaemonSet
* only one pod binds; the rest stay `Pending`
* mounting a PVC but forgetting to set `l2Backend.raw`
* setting `base_path` to a directory that is not actually mounted into the pod
* assuming node-local storage is shared across nodes
* using `emptyDir` and expecting data to survive a pod restart
* mounting anything at `/dev/shm`
* this is unrelated to filesystem L2 and can break CUDA IPC for MP mode
## When to use something else
Choose a RESP backend instead when you want:
* a clearly shared remote cache tier
* centralized capacity management
* auth-managed network storage with fewer filesystem semantics to think about
Choose filesystem offloading when you want:
* the simplest durable L2 on Kubernetes
* no Redis dependency
* direct use of an existing POSIX storage system