External Storage Offloading

Filesystem offloading means using LMCache’s filesystem-backed L2 adapter as a backing store behind the in-memory L1 cache. In operator terms, this is an engine.spec.l2Backend.raw configuration plus a mounted path inside the engine pod. Use this when you want:

a simple on-cluster L2 tier without Redis or object storage
warm reuse across engine restarts
a shared POSIX-style backing store, if your cluster can provide one

What this is in the operator model

The chart does not expose a first-class filesystem.enabled=true value. Filesystem offloading is configured through the LMCacheEngine CR passthrough:

engine.spec.l2Backend.raw
engine.spec.volumes
engine.spec.volumeMounts

That is because filesystem L2 is currently treated as a raw adapter rather than a typed operator field like l2Backend.resp.

Choose your storage

The engine needs a writable directory at the L2 base_path. There are two ways to give it one. Pick with this:

Use	When
A host directory (`hostPath`)	Your nodes have local disk and you are happy for each node to keep its own cache. This covers most clusters.
A PersistentVolumeClaim	Any of: nodes are ephemeral (managed/autoscaled clusters), you want one node to reuse what another cached, or your cluster policy blocks `hostPath`.

One command tells you whether a PVC is even possible here:

kubectl get storageclass

If that prints No resources found, the cluster has no dynamic provisioning and a PVC can never bind — use Option A.

Option A — host directory (recommended)

Mount a real directory from the node. Nothing to create first, nothing to provision.

my-values.yaml

engine:
  spec:
    volumes:
      - name: lmcache-l2
        hostPath:
          path: /mnt/nvme0/lmcache-l2     # a path with free space, on every GPU node
          type: DirectoryOrCreate
    volumeMounts:
      - name: lmcache-l2
        mountPath: /data/lmcache/l2

Two things to get right:

The path must exist, or be creatable, on every node that runs an engine pod. DirectoryOrCreate makes it if missing. If one node lacks the disk, that pod fails to mount.
Each node keeps its own cache. Nothing is shared between nodes. That is the right shape for extending capacity beyond RAM; it is the wrong shape if you want cross-node reuse — use Option B with ReadWriteMany for that.

That is the whole setup. Skip to Minimal example.

Option B — PersistentVolumeClaim

Use this when Option A does not fit. The operator does not create the volume for you — the claim must exist before the engine pods start, or they stay Pending forever. 1. Pick a storage class. Names are cluster-specific; there is no universal default.

kubectl get storageclass

2. Create the claim, substituting a name from that list.

lmcache-l2-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lmcache-l2
  namespace: tensormesh-operator
spec:
  accessModes:
    - ReadWriteOnce                       # ReadWriteMany to share across nodes
  resources:
    requests:
      storage: 500Gi                      # size to your L2 working set
  storageClassName: REPLACE_ME            # a name from `kubectl get storageclass`

If one class is marked (default), you can omit storageClassName and get that one. 3. Apply it and wait for Bound.

kubectl apply -f lmcache-l2-pvc.yaml
kubectl get pvc -n tensormesh-operator
# NAME         STATUS   VOLUME   CAPACITY   ACCESS MODES   AGE
# lmcache-l2   Bound    pvc-...  500Gi      RWO            5s

4. Reference it from the engine.

my-values.yaml

engine:
  spec:
    volumes:
      - name: lmcache-l2
        persistentVolumeClaim:
          claimName: lmcache-l2
    volumeMounts:
      - name: lmcache-l2
        mountPath: /data/lmcache/l2

One ReadWriteOnce claim cannot serve more than one engine pod. The engine runs as a DaemonSet, so a multi-node cluster gets one engine pod per GPU node. An RWO claim binds to a single node, and every other engine pod stays Pending.For more than one engine pod, use a ReadWriteMany storage class, or create one claim per node.

Minimal example

This uses Option A (a host directory). Swap the volumes block for the PVC version if you chose Option B — everything else is identical.

my-values.yaml

engine:
  enabled: true
  spec:
    l1:
      sizeGB: 60
    image:
      repository: lmcache/vllm-openai
      tag: v<version>
      pullPolicy: IfNotPresent
    l2Backend:
      raw:
        type: fs
        config:
          base_path: /data/lmcache/l2
      storePolicy: default
      prefetchPolicy: default
      prefetchMaxInFlight: 8
    volumes:
      - name: lmcache-l2
        hostPath:
          path: /mnt/nvme0/lmcache-l2
          type: DirectoryOrCreate
    volumeMounts:
      - name: lmcache-l2
        mountPath: /data/lmcache/l2

Apply it with:

helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version <version> \
  -n tensormesh-operator --create-namespace \
  -f my-values.yaml --wait

What the fields mean

l2Backend.raw.type: fs
- selects the filesystem-backed L2 adapter
l2Backend.raw.config.base_path
- the directory inside the engine container where LMCache stores L2 files
volumes / volumeMounts
- make that path real and writable inside the engine pod
storePolicy: default
- keep normal L1 behavior and also store evicted/eligible keys into L2
prefetchPolicy: default
- allow misses to load data back from L2 into L1
prefetchMaxInFlight
- caps concurrent L2-to-L1 loads to avoid flooding L1 memory

Recommended storage choices

Shared cache across nodes

Use a storage class or backing system that gives you:

ReadWriteMany
the same mounted path in every engine pod
enough throughput for concurrent reads during warm prefills

Examples:

NFS
EFS / Filestore / Azure Files
CephFS

This is the right shape when you want one node to benefit from data another node stored.

Fast local spill on each node

Use:

local SSD / NVMe
hostPath
ReadWriteOnce PVC pinned to the node

This is still useful if your main goal is to extend capacity beyond RAM on a single node, but it is not a cross-node shared L2.

Optional `fs` adapter knobs

The fs adapter accepts a few useful extra fields:

l2Backend:
  raw:
    type: fs
    config:
      base_path: /data/lmcache/l2
      relative_tmp_dir: tmp
      read_ahead_size: 1048576
      use_odirect: false

What they do:

relative_tmp_dir
- subdirectory under base_path for temporary write files
read_ahead_size
- issue a small initial read to encourage filesystem readahead
use_odirect
- bypass the OS page cache; only use this if you understand the alignment/performance tradeoff

If you need adapter-specific features beyond these, pass them through under raw.config. The operator forwards that JSON to LMCache as-is.

Persistence and restart behavior

Filesystem L2 is useful because it survives engine pod restarts as long as the mounted storage survives. That means:

emptyDir is not a good choice if you want persistence across pod restarts
a PVC or durable host-backed path is the right choice if restart survival matters

Storage Sizing

Sizing is the next question after “should I use filesystem offloading at all?” The practical approach: size each tier to its working set, deploy, then let the Metrics confirm whether the hot set fits — the hit rate and L1/L2 activity counters tell you directly whether to grow or shrink each tier. To translate a token count into bytes of cache for a specific model, the upstream LMCache KV Cache Size Calculator gives the per-token KV size.

L1 sizing

Use L1 when you want the fastest possible warm-hit behavior. In the operator, L1 is:

engine.spec.l1.sizeGB

General guidance:

size L1 for the hot working set you expect to be reused frequently
if L1 is too small, warm entries will evict before they are reused
if L1 comfortably holds the hot set, repeat traffic can stay in memory and avoid L2 reads

Start L1 near your estimated hot set, then adjust from the L1 read/write/eviction counters in Metrics: sustained high writes and evictions with low reads mean the hot set isn’t fitting.

L2 sizing

Use L2 when you want capacity beyond RAM or warm reuse across restarts and, if storage is shared, across nodes. General guidance:

size L2 for the larger working set that does not fit in L1
if L2 is too small, you will churn older entries and lose warm benefit between bursts
durable L2 is most useful when prompt reuse is real and frequent enough to pay back the extra I/O

Grow L2 while the L2 load counters and overall hit rate in Metrics keep climbing; once more capacity stops buying additional hits, you’ve reached the useful size for your traffic.

Expected performance by hit rate

The performance you should expect depends heavily on where hits land:

mostly L1 hits
- best TTFT improvement
- lowest warm-request latency
mostly L2 hits
- still useful, but slower than L1 because data must be loaded back into memory
mostly misses
- little or no cache benefit

To see the L1/L2/miss split for your own traffic, watch the lookup hit rate and per-tier counters in Metrics.

Verification

After rollout:

kubectl get pvc -n tensormesh-operator
kubectl get lmcacheengine -n tensormesh-operator
kubectl describe lmcacheengine -n tensormesh-operator <engine-name>
kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine
kubectl logs -n tensormesh-operator -l app.kubernetes.io/component=cache-engine --tail=200

What to check:

the PVC is Bound
the engine pod is Running
the mounted path exists and is writable
the engine does not fail while parsing the L2 adapter config
warm requests eventually produce L2-related store/load activity rather than only L1 behavior

If you want a stronger functional check, combine this with a small L1 and repeated long prompts so you can force eviction and subsequent reload from L2.

Engine pods stay `Pending`

If engine pods never leave Pending, check the scheduler’s reason. There are no container logs to read yet, so kubectl logs returns nothing — the explanation is in the pod’s events:

kubectl describe pod -n tensormesh-operator <engine-pod> | tail -20

persistentvolumeclaim "lmcache-l2" not found

Warning  FailedScheduling  default-scheduler
         0/2 nodes are available: persistentvolumeclaim "lmcache-l2" not found.

The claim referenced by engine.spec.volumes doesn’t exist. Either create it (Option B) — the pending pods then schedule on their own once it reports Bound, with no restart or re-apply — or switch to a host directory (Option A), which needs no claim at all. Some engine pods Pending while others run A ReadWriteOnce claim binds to a single node, so exactly one engine pod can use it. Every other engine pod waits forever. Switch to a ReadWriteMany storage class, or give each node its own claim. PVC stuck Pending with storageclass ... not found

Warning  ProvisioningFailed  persistentvolume-controller
         storageclass.storage.k8s.io "standard" not found

The storageClassName in the PVC names a class this cluster does not have. Class names are cluster-specific — list yours with kubectl get storageclass and use one of those. Note that storageClassName is immutable, so the PVC has to be deleted and re-created rather than edited:

kubectl delete pvc lmcache-l2 -n tensormesh-operator
# fix storageClassName, then
kubectl apply -f lmcache-l2-pvc.yaml

pod has unbound immediate PersistentVolumeClaims The PVC exists but is still Pending rather than Bound. Check that its storageClassName matches a class that exists and has a working provisioner:

kubectl get pvc -n tensormesh-operator
kubectl describe pvc lmcache-l2 -n tensormesh-operator | tail -10
kubectl get storageclass

Common mistakes

referencing claimName: lmcache-l2 without creating the PVC
- engine pods stay Pending with persistentvolumeclaim "lmcache-l2" not found
sharing one ReadWriteOnce claim across a multi-node engine DaemonSet
- only one pod binds; the rest stay Pending
mounting a PVC but forgetting to set l2Backend.raw
setting base_path to a directory that is not actually mounted into the pod
assuming node-local storage is shared across nodes
using emptyDir and expecting data to survive a pod restart
mounting anything at /dev/shm
- this is unrelated to filesystem L2 and can break CUDA IPC for MP mode

When to use something else

Choose a RESP backend instead when you want:

a clearly shared remote cache tier
centralized capacity management
auth-managed network storage with fewer filesystem semantics to think about

Choose filesystem offloading when you want:

the simplest durable L2 on Kubernetes
no Redis dependency
direct use of an existing POSIX storage system

Overview

Installation

Features

Operations

Integrations

Operator UI

Reference

What this is in the operator model

Choose your storage

Option A — host directory (recommended)

Option B — PersistentVolumeClaim

Minimal example

What the fields mean

Recommended storage choices

Shared cache across nodes

Fast local spill on each node

Optional `fs` adapter knobs

Persistence and restart behavior

Storage Sizing

L1 sizing

L2 sizing

Expected performance by hit rate

Verification

Engine pods stay `Pending`

Common mistakes

When to use something else

​What this is in the operator model

​Choose your storage

​Option A — host directory (recommended)

​Option B — PersistentVolumeClaim

​Minimal example

​What the fields mean

​Recommended storage choices

​Shared cache across nodes

​Fast local spill on each node

​Optional fs adapter knobs

​Persistence and restart behavior

​Storage Sizing

​L1 sizing

​L2 sizing

​Expected performance by hit rate

​Verification

​Engine pods stay Pending

​Common mistakes

​When to use something else

What this is in the operator model

Choose your storage

Option A — host directory (recommended)

Option B — PersistentVolumeClaim

Minimal example

What the fields mean

Recommended storage choices

Shared cache across nodes

Fast local spill on each node

Optional `fs` adapter knobs

Persistence and restart behavior

Storage Sizing

L1 sizing

L2 sizing

Expected performance by hit rate

Verification

Engine pods stay `Pending`

Common mistakes

When to use something else