# Non-Prefix KV Caching Source: https://docs.tensormesh.ai/v1.0.0/configuration/cacheblend Reuse KV cache beyond prefix boundaries with CacheBlend [CacheBlend](https://arxiv.org/abs/2405.16444) is a technique introduced by the Tensormesh team that reuses precomputed KV caches beyond prefix boundaries. Rather than requiring an exact prefix match, it reuses any contiguous block of previously computed tokens regardless of position, then selectively recomputes a small subset of tokens to partially update each reused block. CacheBlend is most useful when: * prompt reuse is **non-prefix** — shared blocks sit after a per-request preamble, system prompt, or other documents * you serve **RAG** workloads where the same documents are retrieved in varying order across requests Two kinds of CacheBlend set up exists today: * Use **Aggregated mode** when your workload is prefill heavy * Use **PD disaggregation** when your workload has more decoding workload Either mode can add an [L2 external storage tier](#l2-filesystem-offloading) on top of engine's CPU offloading. ## Prerequisites * [cert-manager](https://cert-manager.io/docs/installation/) — any recent release; the mutating webhook's serving certificate is issued through it: ```bash theme={null} kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml kubectl -n cert-manager wait --for=condition=Available deploy --all --timeout=180s ``` * **Access Token** - The `cacheblend-plugin` image is distributed through a private registry. To pull it, request an access token from [https://www.tensormesh.ai/contact](https://www.tensormesh.ai/contact). ### Supported Versions | TMO helm chart | LMCache Operator | LMCache vLLM | cacheblend-plugin | | - | - | - | - | | `1.0.0` | `v0.5.5` | `v0.5.5` | `v0.5.5` | ### Supported Models | Model family | Models | | - | - | | GLM | GLM-5.2 | | MiniMax | MiniMax M3, MiniMax M2.5 | | Gemma | Gemma3 | | gpt-oss | gpt-oss-20b, gpt-oss-120b | | llama | llama 3.1 | ## Aggregated mode ### Install Chart Install the chart with CacheBlend enabled. This creates the `CacheBlendEngine` — a per-node DaemonSet — in the `tensormesh-platform` namespace: ```yaml my-values.yaml theme={null} # REQUIRED — the mutating webhook injects the plugin (needs cert-manager). webhook: enabled: true engine: enabled: false # no LMCacheEngine — CacheBlend-only install coordinator: enabled: false cacheBlend: enabled: true name: tensormesh-cacheblend # engine created in the release namespace spec: l1: sizeGB: 200 image: repository: lmcache/vllm-openai tag: v0.5.5 pullPolicy: IfNotPresent injection: payloadImage: repository: artifacts.tensormesh.ai/tensormesh-production/images/cacheblend-plugin tag: v0.5.5 pullPolicy: IfNotPresent imagePullSecrets: - name: cacheblend-plugin-pull # must exist in each workload ns ``` ```bash theme={null} helm install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ --namespace tensormesh-platform \ --create-namespace \ -f my-values.yaml \ --wait ``` ### Opting a vLLM pod in CacheBlend plugin attaches to vLLM pods **you deploy** — the operator does not create them. First create the workload namespace (PSS-privileged) and the plugin pull secret — the private init container is pulled in the **pod's** namespace, so the secret must live here: ```bash theme={null} kubectl create ns cacheblend-workload kubectl label ns cacheblend-workload pod-security.kubernetes.io/enforce=privileged kubectl -n cacheblend-workload create secret docker-registry cacheblend-plugin-pull \ --docker-server=artifacts.tensormesh.ai --docker-username=x-access-token --docker-password= ``` Then add a label + annotation to your pod template and launch vLLM **args-only** (a `command:` override makes the webhook skip injection): ```yaml theme={null} metadata: labels: lmcache.ai/cacheblend-inject: "true" # opt in annotations: lmcache.ai/cacheblend-engine: "tensormesh-platform/tensormesh-cacheblend" # / ``` Everything else (image, model, resources, replicas) stays your normal vLLM config. At admission the webhook injects the plugin init container and `PYTHONPATH`, the IPC path to the node-local engine (a `/dev/shm` mount — or `hostIPC` when the engine's `spec.hostIPC` is `true`), the CacheBlend vLLM flags, and the `--kv-transfer-config` (`CBKVConnector`) pointing at that engine; blend tunables (`cb.check_layer`, `cb.recomp_ratio`) come from the engine config. ### Verification **Engine reconciled** ```bash theme={null} kubectl -n tensormesh-platform get cacheblendengine tensormesh-cacheblend kubectl -n tensormesh-platform get configmap tensormesh-cacheblend-connection ``` The `-connection` ConfigMap is the reconcile proof — and the gate the webhook reads. If it is missing, injection silently fail-opens (the pod runs without CacheBlend). **Webhook wired** ```bash theme={null} kubectl get mutatingwebhookconfiguration tensormesh-platform-mutating-webhook \ -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | head -c 20; echo ``` A non-empty `caBundle` means cert-manager issued the serving cert and the webhook can be called. **Injection happened** After creating an opt-in vLLM pod in the workload namespace: ```bash theme={null} P=$(kubectl -n cacheblend-workload get pod -l lmcache.ai/cacheblend-inject=true -o name | head -1) kubectl -n cacheblend-workload get $P -o jsonpath='{.metadata.annotations.lmcache\.ai/cacheblend-injected}{"\n"}' # present = injected kubectl -n cacheblend-workload get $P -o jsonpath='{.spec.initContainers[*].name}{"\n"}' # cb-plugin-stage kubectl -n cacheblend-workload get $P -o jsonpath='{.spec.containers[0].volumeMounts[?(@.mountPath=="/dev/shm")].name}{"\n"}' # /dev/shm mount (hostIPC=true instead if the engine sets spec.hostIPC) ``` **Non-prefix blend actually engaged** Send two requests that share a block **after different-length preambles**, then check the engine logs. A `shifted` retrieve means a chunk was reused at a different position than it was cached — i.e. a non-prefix blend (select the engine by `instance` to skip any regular LMCacheEngine in the same ns): ```bash theme={null} kubectl -n tensormesh-platform logs -l app.kubernetes.io/instance=tensormesh-cacheblend --tail=500 \ | grep -E "\[match_probe\] .* matches=[1-9]|Retrieved pre-computed for [1-9].* shifted=[1-9]" ``` ## PD disaggregation ### Install Chart ```yaml my-values.yaml theme={null} webhook: enabled: true # REQUIRED engine: enabled: false # no LMCacheEngine — CacheBlend-only install coordinator: enabled: false cacheBlend: enabled: true name: tensormesh-cacheblend spec: privileged: true l1: sizeGB: 200 image: repository: lmcache/vllm-openai tag: v0.5.5 pullPolicy: IfNotPresent injection: payloadImage: # pulled by PREFILLER pods, in their own namespace repository: artifacts.tensormesh.ai/tensormesh-production/images/cacheblend-plugin tag: v0.5.5 pullPolicy: IfNotPresent imagePullSecrets: - name: cacheblend-plugin-pull pd: nixlSideChannelPort: 5558 # VLLM_NIXL_SIDE_CHANNEL_PORT on both roles unless the pod pre-sets it nixlLoadFailurePolicy: fail # or `ignore`: fall back to local prefill ``` ```bash theme={null} helm upgrade --install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ --namespace tensormesh-platform \ -f my-values.yaml \ --wait ``` ### Namespace and pull secret Create the `cacheblend-workload` namespace and the `cacheblend-plugin-pull` secret first, exactly as in [Opting a vLLM pod in](#opting-a-vllm-pod-in). ### Deploy prefiller, decoder, router Deploy your vLLM prefiller, decoder, and router together. The **lmcache.ai/pd-role** annotation declares each vLLM Deployment's role — prefiller or decoder — and the webhook injects the following. * Prefiller — the CacheBlend plugin, the CacheBlend vLLM flags, and `--kv-transfer-config` set to the `MultiConnector` (`NixlConnector` + `CBKVConnector`) — so it blends locally and produces KV for the decoder. * Decoder — **`--kv-transfer-config` only**, set to a bare `NixlConnector` (`kv_consumer`). No plugin, no init container, no IPC wiring: it receives finished KV from the prefiller over NIXL. * Both — `VLLM_NIXL_SIDE_CHANNEL_HOST` (the pod IP) and `VLLM_NIXL_SIDE_CHANNEL_PORT` (`spec.pd.nixlSideChannelPort`, unless the pod already sets one), and the `lmcache.ai/cacheblend-injected` annotation. ```yaml theme={null} # Prefiller — blends, then hands KV to the decoder over NIXL apiVersion: apps/v1 kind: Deployment metadata: name: cb-pd-prefiller namespace: cacheblend-workload labels: app: cb-pd-prefiller spec: replicas: 1 selector: matchLabels: app: cb-pd-prefiller template: metadata: labels: app: cb-pd-prefiller lmcache.ai/cacheblend-inject: "true" annotations: lmcache.ai/cacheblend-engine: "tensormesh-platform/tensormesh-cacheblend" lmcache.ai/pd-role: "prefiller" spec: runtimeClassName: nvidia hostNetwork: true dnsPolicy: ClusterFirstWithHostNet # No IPC wiring, no --kv-transfer-config: the webhook injects them for this role # (a /dev/shm mount, or hostIPC if the engine sets spec.hostIPC). containers: - name: vllm image: lmcache/vllm-openai:v0.5.5 imagePullPolicy: IfNotPresent env: - name: HF_HUB_DISABLE_TELEMETRY value: "1" - name: UCX_NET_DEVICES value: "all" - name: NCCL_CUMEM_ENABLE value: "1" - name: VLLM_NIXL_SIDE_CHANNEL_PORT # distinct from the decoder's: both share the host IP value: "5557" args: - "Qwen/Qwen3-0.6B" - "--served-model-name" - "Qwen3-0.6B" - "--port" - "8001" - "--gpu-memory-utilization" - "0.4" - "--max-model-len" - "8192" - "--attention-backend" # prefiller only - "CUSTOM" ports: - name: http containerPort: 8001 volumeMounts: - name: hf-cache mountPath: /root/.cache/huggingface resources: limits: nvidia.com/gpu: "1" memory: 32Gi requests: cpu: "2" memory: 16Gi readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 60 periodSeconds: 15 failureThreshold: 40 volumes: - name: hf-cache emptyDir: sizeLimit: 20Gi --- apiVersion: v1 kind: Service metadata: name: cb-pd-prefiller namespace: cacheblend-workload spec: selector: app: cb-pd-prefiller ports: - name: http port: 8001 targetPort: http --- # Decoder — receives KV from the prefiller; never loads the CacheBlend plugin apiVersion: apps/v1 kind: Deployment metadata: name: cb-pd-decoder namespace: cacheblend-workload labels: app: cb-pd-decoder spec: replicas: 1 selector: matchLabels: app: cb-pd-decoder template: metadata: labels: app: cb-pd-decoder lmcache.ai/cacheblend-inject: "true" annotations: lmcache.ai/cacheblend-engine: "tensormesh-platform/tensormesh-cacheblend" lmcache.ai/pd-role: "decoder" spec: runtimeClassName: nvidia hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: vllm image: lmcache/vllm-openai:v0.5.5 imagePullPolicy: IfNotPresent env: - name: HF_HUB_DISABLE_TELEMETRY value: "1" - name: UCX_NET_DEVICES value: "all" - name: NCCL_CUMEM_ENABLE value: "1" # VLLM_NIXL_SIDE_CHANNEL_PORT left unset: the webhook injects spec.pd.nixlSideChannelPort (5558) args: - "Qwen/Qwen3-0.6B" - "--served-model-name" - "Qwen3-0.6B" - "--port" - "8002" - "--gpu-memory-utilization" - "0.4" - "--max-model-len" - "8192" ports: - name: http containerPort: 8002 volumeMounts: - name: hf-cache mountPath: /root/.cache/huggingface resources: limits: nvidia.com/gpu: "1" memory: 32Gi requests: cpu: "2" memory: 16Gi readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 60 periodSeconds: 15 failureThreshold: 40 volumes: - name: hf-cache emptyDir: sizeLimit: 20Gi --- apiVersion: v1 kind: Service metadata: name: cb-pd-decoder namespace: cacheblend-workload spec: selector: app: cb-pd-decoder ports: - name: http port: 8002 targetPort: http --- # Router — splits each request into a prefill leg and a decode leg apiVersion: apps/v1 kind: Deployment metadata: name: cb-pd-router namespace: cacheblend-workload spec: replicas: 1 selector: matchLabels: app: cb-pd-router template: metadata: labels: app: cb-pd-router spec: containers: - name: router image: vllm/vllm-router:nightly imagePullPolicy: IfNotPresent command: ["/bin/sh", "-c"] # not an opt-in pod, so a command wrapper is fine here args: - | exec vllm-router \ --policy round_robin \ --vllm-pd-disaggregation \ --prefill http://cb-pd-prefiller.cacheblend-workload.svc.cluster.local:8001 \ --decode http://cb-pd-decoder.cacheblend-workload.svc.cluster.local:8002 \ --host 0.0.0.0 \ --port 30000 \ --intra-node-data-parallel-size 1 ports: - name: http containerPort: 30000 readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 5 periodSeconds: 10 --- apiVersion: v1 kind: Service metadata: name: cb-pd-router namespace: cacheblend-workload spec: selector: app: cb-pd-router ports: - name: http port: 30000 targetPort: http ``` ```bash theme={null} kubectl apply -f vllm-cacheblend-pd.yaml ``` ### Verification **PD ConfigMap shape** With `spec.pd` set, the engine's connection ConfigMap carries three role keys. The decoder key must not contain `CBKVConnector` — the decoder never blends: ```bash theme={null} kubectl -n tensormesh-platform get configmap tensormesh-cacheblend-connection \ -o go-template='{{range $k, $_ := .data}}{{$k}}{{"\n"}}{{end}}' # kv-transfer-config.json kv-transfer-config-prefiller.json kv-transfer-config-decoder.json kubectl -n tensormesh-platform get configmap tensormesh-cacheblend-connection \ -o jsonpath='{.data.kv-transfer-config-decoder\.json}' | grep -c CBKVConnector # 0 ``` **Each role got its own injection** ```bash theme={null} for r in prefiller decoder; do P=$(kubectl -n cacheblend-workload get pod -l lmcache.ai/pd-role=$r -o name | head -1) echo "$r injected=$(kubectl -n cacheblend-workload get $P -o jsonpath='{.metadata.annotations.lmcache\.ai/cacheblend-injected}')" \ "init=$(kubectl -n cacheblend-workload get $P -o jsonpath='{.spec.initContainers[*].name}')" \ "nixl_port=$(kubectl -n cacheblend-workload get $P -o jsonpath='{.spec.containers[0].env[?(@.name=="VLLM_NIXL_SIDE_CHANNEL_PORT")].value}')" done ``` Expected: `injected` present on **both**; `init=cb-plugin-stage` on the prefiller **only** (empty on the decoder); `nixl_port=5557` on the prefiller (its pre-set value) and `5558` on the decoder (injected from `spec.pd.nixlSideChannelPort`). A `lmcache.ai/cacheblend-skip-reason` annotation on either pod means the webhook skipped it — see [Common mistakes](#common-mistakes). **Requests flow through both roles** ```bash theme={null} kubectl -n cacheblend-workload port-forward svc/cb-pd-router 30000:30000 & curl -s localhost:30000/v1/completions -H 'Content-Type: application/json' \ -d '{"model":"Qwen3-0.6B","prompt":"Hello from PD","max_tokens":16}' | jq -r '.choices[0].text' ``` A completion back means the router split the request, the prefiller produced KV, and the decoder received it over NIXL and generated from it. **Non-prefix blend engaged on the prefiller** Same check as aggregated mode — blending happens only on the prefill leg, so only the prefiller's traffic can produce `shifted` retrieves: ```bash theme={null} kubectl -n tensormesh-platform logs -l app.kubernetes.io/instance=tensormesh-cacheblend --tail=500 \ | grep -E "\[match_probe\] .* matches=[1-9]|Retrieved pre-computed for [1-9].* shifted=[1-9]" ``` ## L2 external storage offloading The blend engine's L1 is CPU DRAM, bounded by `l1.sizeGB`. Adding an L2 adapter lets chunks that fall out of L1 be served from external storage instead of recomputed; the CacheBlend lookup reads L2 through the engine's L2→L1 prefetch, so nothing changes on the vLLM side. For example for file system offloading, add the following l2.yaml to `cacheBlend.spec` in your helm values: ```yaml l2.yaml theme={null} l2Backend: raw: type: fs_native # C++ filesystem adapter config: base_path: /data/lmcache/l2 # = the volumeMount below max_capacity_gb: 500 # cap on the adapter's disk use storePolicy: default # write-through: L1 keeps the hot set, L2 extends it volumes: - name: lmcache-l2 hostPath: path: /mnt/nvme0/lmcache-l2 # per-node local NVMe (the engine is a DaemonSet) type: DirectoryOrCreate volumeMounts: - name: lmcache-l2 mountPath: /data/lmcache/l2 ``` ### Verification **Adapter engaged** ```bash theme={null} kubectl -n tensormesh-platform get cacheblendengine tensormesh-cacheblend -o jsonpath='{.spec.l2Backend.raw.type}{"\n"}' # fs_native E=$(kubectl -n tensormesh-platform get pod -l app.kubernetes.io/instance=tensormesh-cacheblend -o name | head -1) kubectl -n tensormesh-platform exec $E -- sh -c 'ls /data/lmcache/l2 | head' # populated after the first requests ``` **Chunks written to and read back from L2** The engine exposes its counters on the same port as `/status` (`8080` by default): ```bash theme={null} kubectl -n tensormesh-platform exec $E -- curl -s localhost:8080/metrics \ | grep -E '^lmcache_mp_l2_(store_completed_objects|prefetch_load_completed)' ``` `lmcache_mp_l2_store_completed_objects_chunks_total` > 0 means chunks were stored to L2; `lmcache_mp_l2_prefetch_load_completed_chunks_total` > 0 means chunks were read back from L2 into L1 for a lookup. With `storePolicy: default` the second only rises after L1 evictions — send more distinct long prompts than L1 holds, then repeat an early one. ## Helm values reference | Value | Type | Default | Description | | - | - | - | - | | `cacheBlend.enabled` | bool | `false` | Create a `CacheBlendEngine` CR. Requires `webhook.enabled: true` + cert-manager. | | `cacheBlend.name` | string | `""` | Engine CR name. Empty = `-cacheblend`. | | `cacheBlend.namespace` | string | `""` | Namespace for the engine + its connection ConfigMap. Empty (recommended) = release namespace; opt-in pods bind it cross-namespace. | | `cacheBlend.spec.l1.sizeGB` | int | `60` | L1 (host-DRAM) chunk cache size, in GiB. | | `cacheBlend.spec.image` | object | `lmcache/vllm-openai` | Blend-engine image (the lmcache server). | | `cacheBlend.spec.injection.payloadImage` | object | — | PRIVATE `cacheblend-plugin` image injected as the init container. Must match the engine's lmcache build. | | `cacheBlend.spec.injection.imagePullSecrets` | list | `[]` | Pull secret names appended to opt-in pods for the private payload image. The Secret(s) must already exist in the vLLM pod's namespace. | | `cacheBlend.spec.l2Backend` | object | — | L2 tier behind L1: `raw.type`/`raw.config` (adapter, e.g. `fs_native` with `base_path`), `storePolicy` (`default` \| `skip_l1`). See [L2 filesystem offloading](#l2-filesystem-offloading). | | `cacheBlend.spec.volumes` / `volumeMounts` | list | `[]` | Extra volumes for the engine pod — where the L2 `base_path` is mounted. | | `webhook.enabled` | bool | `true` | Enable the CacheBlend mutating webhook. Requires cert-manager. | ## Common mistakes * Running the opt-in vLLM pod in the **operator (release) namespace** — the webhook excludes it, so injection never happens * A **bare** engine ref (`tensormesh-cacheblend`) instead of `/tensormesh-cacheblend` — the webhook looks in the pod's own namespace and finds no engine * Setting a `command:` on the vLLM container — the webhook skips injection (appended args would never reach `vllm serve`) * Workload namespace **not PSS-privileged** — Pod Security rejects the injected `hostIPC` * The `cacheblend-plugin-pull` secret missing from the **workload** namespace — the injected init container `ImagePullBackOff`s (the secret must live where the pod runs, not in the operator namespace) * **Mismatched** `cacheblend-plugin` and `vllm-openai` images — vLLM crashes at startup with an `ImportError`; pin a matched pair * `webhook.enabled: false` or no cert-manager — the engine deploys but nothing injects the plugin # CPU Offloading Source: https://docs.tensormesh.ai/v1.0.0/configuration/cpu-offloading Offload KV cache from GPU memory into host DRAM — the default cache tier. **CPU offloading** moves KV cache off the GPU into the node's system memory (host DRAM). This is the engine's L1 tier — the first place cached KV lands once it no longer fits in GPU HBM — and it's on by default, configured with a single value: `engine.spec.l1.sizeGB`. That value reserves host DRAM (not GPU memory) on each node; GPU HBM stays owned by vLLM's own paged KV cache, so the operator never competes with it. It reuses prompt prefixes from host RAM, keeping GPU memory free for active batches, with no extra storage or dependencies. ## The cache tiers Hot data is promoted toward the GPU; cold data spills toward disk: | Tier | Lives in | Speed | Configured by | | - | - | - | - | | **GPU HBM (L0)** | vLLM paged KV cache | Fastest | vLLM | | **CPU offloading (L1)** | Host DRAM | Fast | `engine.spec.l1.sizeGB` | | **External storage (L2)** | Disk / NFS / remote | Slower | [`engine.spec.l2Backend`](/v1.0.0/configuration/filesystem-offloading) | CPU offloading alone gives the core benefit: a prefix computed once is reused from RAM instead of recomputed on the GPU. Add [external storage](/v1.0.0/configuration/filesystem-offloading) only when your working set outgrows DRAM or you need reuse to survive restarts. ## Configuration CPU offloading reserves 60 GB of host DRAM per node by default. To set it explicitly: ```yaml my-values.yaml theme={null} engine: enabled: true spec: l1: sizeGB: 120 # host DRAM reserved for KV cache, per node image: repository: lmcache/vllm-openai tag: v0.5.5 pullPolicy: IfNotPresent ``` ```bash theme={null} helm upgrade --install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ -n tensormesh-platform --create-namespace \ -f my-values.yaml --wait ``` `l1.sizeGB` **applies per node**: every engine pod reserves this much DRAM on the node it runs on. Size it to your workload (next) rather than relying on the default. ## Sizing Size the cache to hold your **hot working set** — the prefixes reused often. Too small and warm entries evict before reuse; sized right and repeat traffic stays in memory. A practical loop: size it to a comfortable share of each node's free RAM (leaving headroom for vLLM and the OS), then watch the hit rate and L1 read/write/eviction counters in [Metrics](/v1.0.0/observability/metrics). Sustained high writes and evictions with low reads mean the hot set isn't fitting; grow `l1.sizeGB`. To convert a token count to GB for a model, use the [KV Cache Size Calculator](https://docs.lmcache.ai/getting_started/kv_cache_calculator.html). ## Eviction When the cache fills, the engine evicts. Tune the watermark and drop ratio in the engine spec: ```yaml theme={null} engine: spec: eviction: triggerWatermark: 0.8 # start evicting at 80% full evictionRatio: 0.2 # drop 20% when triggered ``` With [external storage](/v1.0.0/configuration/filesystem-offloading) configured, evicted entries spill there instead of being dropped, so a warm prefix pushed out of DRAM can reload from disk rather than recompute. ## Verify ```bash theme={null} kubectl get pods -n tensormesh-platform -l app.kubernetes.io/component=cache-engine ``` Send repeated traffic, then confirm L1 reads climb while writes level off, via `lmcache_mp_l1_read_chunks_total`, `_write_chunks_total`, and `_evicted_chunks_total`. See **[Metrics](/v1.0.0/observability/metrics)** for the PromQL and healthy-vs-concerning patterns. ## When it helps CPU offloading pays off in proportion to how much your prompts **repeat**: shared system prompts, few-shot preambles, multi-turn chat, or RAG where the same context recurs. Mostly-unique, one-off prompts have nothing to reuse, so the cache fills and evicts without benefit. Two things also gate whether a prefix is cached at all: * Prompts shorter than the engine's `server.chunkSize` (default **256 tokens**) store nothing. * A prefix must be reused **before it evicts**; keep the hot set resident by sizing the cache well, and add [external storage](/v1.0.0/configuration/filesystem-offloading) when it outgrows DRAM. ## Next steps Add a disk or remote tier behind host DRAM for larger working sets and restart survival. Right-size the cache from the hit rate and read/write counters on real traffic. # External Storage Offloading Source: https://docs.tensormesh.ai/v1.0.0/configuration/filesystem-offloading Use a filesystem-backed L2 cache with Tensormesh Platform by mounting storage into the engine pod and configuring the raw L2 adapter. Filesystem offloading means using LMCache's **filesystem-backed L2 adapter** as a backing store behind the in-memory L1 cache. In operator terms, this is an `engine.spec.l2Backend.raw` configuration plus a mounted path inside the engine pod. Use this when you want: * a simple on-cluster L2 tier without Redis or object storage * warm reuse across engine restarts * a shared POSIX-style backing store, if your cluster can provide one ## What this is in the operator model The chart does **not** expose a first-class `filesystem.enabled=true` value. Filesystem offloading is configured through the LMCacheEngine CR passthrough: * `engine.spec.l2Backend.raw` * `engine.spec.volumes` * `engine.spec.volumeMounts` That is because filesystem L2 is currently treated as a **raw adapter** rather than a typed operator field like `l2Backend.resp`. ## Choose your storage The engine needs a writable directory at the L2 `base_path`. There are two ways to give it one. Pick with this: | Use | When | | - | - | | **A host directory** (`hostPath`) | Your nodes have local disk and you are happy for each node to keep its own cache. **This covers most clusters.** | | **A PersistentVolumeClaim** | Any of: nodes are ephemeral (managed/autoscaled clusters), you want one node to reuse what another cached, or your cluster policy blocks `hostPath`. | One command tells you whether a PVC is even possible here: ```bash theme={null} kubectl get storageclass ``` If that prints `No resources found`, the cluster has no dynamic provisioning and **a PVC can never bind** — use Option A. ## Option A — host directory (recommended) Mount a real directory from the node. Nothing to create first, nothing to provision. ```yaml my-values.yaml theme={null} engine: spec: volumes: - name: lmcache-l2 hostPath: path: /mnt/nvme0/lmcache-l2 # a path with free space, on every GPU node type: DirectoryOrCreate volumeMounts: - name: lmcache-l2 mountPath: /data/lmcache/l2 ``` Two things to get right: * **The path must exist, or be creatable, on every node that runs an engine pod.** `DirectoryOrCreate` makes it if missing. If one node lacks the disk, that pod fails to mount. * **Each node keeps its own cache.** Nothing is shared between nodes. That is the right shape for extending capacity beyond RAM; it is the wrong shape if you want cross-node reuse — use Option B with `ReadWriteMany` for that. That is the whole setup. Skip to [Minimal example](#minimal-example). ## Option B — PersistentVolumeClaim Use this when Option A does not fit. **The operator does not create the volume for you** — the claim must exist before the engine pods start, or they stay `Pending` forever. **1. Pick a storage class.** Names are cluster-specific; there is no universal default. ```bash theme={null} kubectl get storageclass ``` **2. Create the claim**, substituting a name from that list. ```yaml lmcache-l2-pvc.yaml theme={null} apiVersion: v1 kind: PersistentVolumeClaim metadata: name: lmcache-l2 namespace: tensormesh-platform spec: accessModes: - ReadWriteOnce # ReadWriteMany to share across nodes resources: requests: storage: 500Gi # size to your L2 working set storageClassName: REPLACE_ME # a name from `kubectl get storageclass` ``` If one class is marked `(default)`, you can omit `storageClassName` and get that one. **3. Apply it and wait for `Bound`.** ```bash theme={null} kubectl apply -f lmcache-l2-pvc.yaml kubectl get pvc -n tensormesh-platform # NAME STATUS VOLUME CAPACITY ACCESS MODES AGE # lmcache-l2 Bound pvc-... 500Gi RWO 5s ``` **4. Reference it from the engine.** ```yaml my-values.yaml theme={null} engine: spec: volumes: - name: lmcache-l2 persistentVolumeClaim: claimName: lmcache-l2 volumeMounts: - name: lmcache-l2 mountPath: /data/lmcache/l2 ``` **One `ReadWriteOnce` claim cannot serve more than one engine pod.** The engine runs as a DaemonSet, so a multi-node cluster gets one engine pod per GPU node. An RWO claim binds to a single node, and every other engine pod stays `Pending`. For more than one engine pod, use a `ReadWriteMany` storage class, or create one claim per node. ## Minimal example This uses Option A (a host directory). Swap the `volumes` block for the PVC version if you chose Option B — everything else is identical. ```yaml my-values.yaml theme={null} engine: enabled: true spec: l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v0.5.5 pullPolicy: IfNotPresent l2Backend: raw: type: fs config: base_path: /data/lmcache/l2 storePolicy: default prefetchPolicy: default prefetchMaxInFlight: 8 volumes: - name: lmcache-l2 hostPath: path: /mnt/nvme0/lmcache-l2 type: DirectoryOrCreate volumeMounts: - name: lmcache-l2 mountPath: /data/lmcache/l2 ``` Apply it with: ```bash theme={null} helm upgrade --install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ -n tensormesh-platform --create-namespace \ -f my-values.yaml --wait ``` ## What the fields mean * `l2Backend.raw.type: fs` * selects the filesystem-backed L2 adapter * `l2Backend.raw.config.base_path` * the directory inside the engine container where LMCache stores L2 files * `volumes` / `volumeMounts` * make that path real and writable inside the engine pod * `storePolicy: default` * keep normal L1 behavior and also store evicted/eligible keys into L2 * `prefetchPolicy: default` * allow misses to load data back from L2 into L1 * `prefetchMaxInFlight` * caps concurrent L2-to-L1 loads to avoid flooding L1 memory ## Recommended storage choices ### Shared cache across nodes Use a storage class or backing system that gives you: * `ReadWriteMany` * the same mounted path in every engine pod * enough throughput for concurrent reads during warm prefills Examples: * NFS * EFS / Filestore / Azure Files * CephFS This is the right shape when you want one node to benefit from data another node stored. ### Fast local spill on each node Use: * local SSD / NVMe * hostPath * `ReadWriteOnce` PVC pinned to the node This is still useful if your main goal is to extend capacity beyond RAM on a single node, but it is not a cross-node shared L2. ## Optional `fs` adapter knobs The `fs` adapter accepts a few useful extra fields: ```yaml theme={null} l2Backend: raw: type: fs config: base_path: /data/lmcache/l2 relative_tmp_dir: tmp read_ahead_size: 1048576 use_odirect: false ``` What they do: * `relative_tmp_dir` * subdirectory under `base_path` for temporary write files * `read_ahead_size` * issue a small initial read to encourage filesystem readahead * `use_odirect` * bypass the OS page cache; only use this if you understand the alignment/performance tradeoff If you need adapter-specific features beyond these, pass them through under `raw.config`. The operator forwards that JSON to LMCache as-is. ## Persistence and restart behavior Filesystem L2 is useful because it survives engine pod restarts as long as the mounted storage survives. That means: * `emptyDir` is **not** a good choice if you want persistence across pod restarts * a PVC or durable host-backed path is the right choice if restart survival matters ## Storage Sizing Sizing is the next question after “should I use filesystem offloading at all?” The practical approach: size each tier to its working set, deploy, then let the [Metrics](/v1.0.0/observability/metrics) confirm whether the hot set fits — the hit rate and L1/L2 activity counters tell you directly whether to grow or shrink each tier. To translate a token count into bytes of cache for a specific model, the upstream [LMCache KV Cache Size Calculator](https://docs.lmcache.ai/getting_started/kv_cache_calculator.html) gives the per-token KV size. ### L1 sizing Use L1 when you want the fastest possible warm-hit behavior. In the operator, L1 is: * `engine.spec.l1.sizeGB` General guidance: * size L1 for the hot working set you expect to be reused frequently * if L1 is too small, warm entries will evict before they are reused * if L1 comfortably holds the hot set, repeat traffic can stay in memory and avoid L2 reads Start L1 near your estimated hot set, then adjust from the L1 read/write/eviction counters in [Metrics](/v1.0.0/observability/metrics): sustained high writes and evictions with low reads mean the hot set isn't fitting. ### L2 sizing Use L2 when you want capacity beyond RAM or warm reuse across restarts and, if storage is shared, across nodes. General guidance: * size L2 for the larger working set that does not fit in L1 * if L2 is too small, you will churn older entries and lose warm benefit between bursts * durable L2 is most useful when prompt reuse is real and frequent enough to pay back the extra I/O Grow L2 while the L2 load counters and overall hit rate in [Metrics](/v1.0.0/observability/metrics) keep climbing; once more capacity stops buying additional hits, you've reached the useful size for your traffic. ### Expected performance by hit rate The performance you should expect depends heavily on where hits land: * mostly **L1 hits** * best TTFT improvement * lowest warm-request latency * mostly **L2 hits** * still useful, but slower than L1 because data must be loaded back into memory * mostly **misses** * little or no cache benefit To see the L1/L2/miss split for your own traffic, watch the lookup hit rate and per-tier counters in [Metrics](/v1.0.0/observability/metrics). ## Verification After rollout: ```bash theme={null} kubectl get pvc -n tensormesh-platform kubectl get lmcacheengine -n tensormesh-platform kubectl describe lmcacheengine -n tensormesh-platform kubectl get pods -n tensormesh-platform -l app.kubernetes.io/component=cache-engine kubectl logs -n tensormesh-platform -l app.kubernetes.io/component=cache-engine --tail=200 ``` What to check: * the PVC is `Bound` * the engine pod is `Running` * the mounted path exists and is writable * the engine does not fail while parsing the L2 adapter config * warm requests eventually produce L2-related store/load activity rather than only L1 behavior If you want a stronger functional check, combine this with a small L1 and repeated long prompts so you can force eviction and subsequent reload from L2. ## Engine pods stay `Pending` If engine pods never leave `Pending`, check the scheduler's reason. There are no container logs to read yet, so `kubectl logs` returns nothing — the explanation is in the pod's events: ```bash theme={null} kubectl describe pod -n tensormesh-platform | tail -20 ``` **`persistentvolumeclaim "lmcache-l2" not found`** ``` Warning FailedScheduling default-scheduler 0/2 nodes are available: persistentvolumeclaim "lmcache-l2" not found. ``` The claim referenced by `engine.spec.volumes` doesn't exist. Either create it ([Option B](#option-b--persistentvolumeclaim)) — the pending pods then schedule on their own once it reports `Bound`, with no restart or re-apply — or switch to a host directory ([Option A](#option-a--host-directory-recommended)), which needs no claim at all. **Some engine pods `Pending` while others run** A `ReadWriteOnce` claim binds to a single node, so exactly one engine pod can use it. Every other engine pod waits forever. Switch to a `ReadWriteMany` storage class, or give each node its own claim. **PVC stuck `Pending` with `storageclass ... not found`** ``` Warning ProvisioningFailed persistentvolume-controller storageclass.storage.k8s.io "standard" not found ``` The `storageClassName` in the PVC names a class this cluster does not have. Class names are cluster-specific — list yours with `kubectl get storageclass` and use one of those. Note that `storageClassName` is immutable, so the PVC has to be deleted and re-created rather than edited: ```bash theme={null} kubectl delete pvc lmcache-l2 -n tensormesh-platform # fix storageClassName, then kubectl apply -f lmcache-l2-pvc.yaml ``` **`pod has unbound immediate PersistentVolumeClaims`** The PVC exists but is still `Pending` rather than `Bound`. Check that its `storageClassName` matches a class that exists and has a working provisioner: ```bash theme={null} kubectl get pvc -n tensormesh-platform kubectl describe pvc lmcache-l2 -n tensormesh-platform | tail -10 kubectl get storageclass ``` ## Common mistakes * referencing `claimName: lmcache-l2` without creating the PVC * engine pods stay `Pending` with `persistentvolumeclaim "lmcache-l2" not found` * sharing one `ReadWriteOnce` claim across a multi-node engine DaemonSet * only one pod binds; the rest stay `Pending` * mounting a PVC but forgetting to set `l2Backend.raw` * setting `base_path` to a directory that is not actually mounted into the pod * assuming node-local storage is shared across nodes * using `emptyDir` and expecting data to survive a pod restart * mounting anything at `/dev/shm` * this is unrelated to filesystem L2 and can break CUDA IPC for MP mode ## When to use something else Choose a RESP backend instead when you want: * a clearly shared remote cache tier * centralized capacity management * auth-managed network storage with fewer filesystem semantics to think about Choose filesystem offloading when you want: * the simplest durable L2 on Kubernetes * no Redis dependency * direct use of an existing POSIX storage system # KVCache Encryption Source: https://docs.tensormesh.ai/v1.0.0/configuration/kvcache-encryption Encrypt KV cache bytes at rest in the L2 tier with aesgcm, using a master key you provide as a Kubernetes Secret. TMO uses **`aesgcm`** to encrypt KV cache bytes at rest in the **L2 external storage** — Redis, object storage, or a filesystem — where that storage sits outside your trust boundary. You enable it on the engine CR and supply the master key as a Kubernetes `Secret`; the default `hkdf` provider derives a per-tenant key from [`cache_salt`](/v1.0.0/management/multi-tenancy) ## Encryption Scope | Tier | Contents | Encrypted? | | - | - | - | | **L2** (Redis / object store / filesystem) | Durable KV cache bytes | **Yes** | | **L1** (host RAM) | Hot KV cache | No | | **L0** (GPU HBM) | Active KV cache | No | ## Prerequisites | Component | Version | | - | - | | Tensormesh Platform helm chart | `1.0.0` | | LMCache Operator (`lmcache/lmcache-operator`) | `v0.5.5` | | LMCache vLLM (`lmcache/vllm-openai`) | `v0.5.5` | ## Step 1: Create the master-key Secret Create the `Secret` yourself, in the **engine's namespace** — the reference is same-namespace only, so an engine CR cannot be used to read Secrets from other namespaces. The data key **must** be named `master`. Only that key is projected into the pod; the operator mounts it read-only at `/etc/lmcache/keys/master`. ```bash theme={null} # 16 bytes for AES-128 (default), 32 bytes for AES-256 head -c 16 /dev/urandom > master.key kubectl -n tensormesh-platform create secret generic lmcache-l2-master-key \ --from-file=master=master.key ``` ## Step 2: Enable L2 serde encryption on the engine Apply aesgcm KVCache encryption to whichever L2 adapter you have configured — `resp` (Redis) or `raw` (filesystem, object storage). Add a `serde` block alongside it: ```yaml my-values.yaml theme={null} engine: enabled: true spec: l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v0.5.5 l2Backend: # Whichever adapter you already use — the serde wraps either one. raw: type: fs config: base_path: /data/lmcache/l2 serde: aesgcm: masterKeySecretRef: name: lmcache-l2-master-key # must exist in the engine's namespace ``` ```bash theme={null} helm upgrade --install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ -n tensormesh-platform --create-namespace \ -f my-values.yaml --wait ``` ## Configuration reference All fields live under `engine.spec.l2Backend.serde`. | Field | Default | Description | | - | - | - | | `aesgcm.masterKeySecretRef.name` | — | **Required.** Secret in the engine's namespace holding the master key under the `master` data key. | | `aesgcm.keyProvider` | `hkdf` | How per-`cache_salt` keys are derived. Only `hkdf` (HKDF-SHA256) is implemented. | | `aesgcm.aesBits` | `128` | AES key size: `128` or `256`. | On key size: **AES-128 is the right default.** It is computationally unbreakable, and KV cache is short-lived and regenerable, so the "harvest now, decrypt later" argument for 256 barely applies. Choose `256` when a compliance mandate requires it — and remember the master key file must then be 32 bytes. Validation is enforced at `kubectl apply` / `helm upgrade` time: * exactly one serde type must be set (only `aesgcm` exists today) * `masterKeySecretRef.name` must be non-empty * the request is **rejected** if a `raw` adapter config already sets its own `serde` key — one would silently overwrite the other ## Cost and overhead * **Storage:** a fixed **29 bytes per chunk** (a 1-byte format version, a 12-byte IV, and a 16-byte GCM authentication tag). Ciphertext is the same length as plaintext — there is no padding. * **Compute:** AES-GCM on server AES-NI hardware runs at roughly 4–8 GB/s per core for AES-128, and it runs on the **L1↔L2 path — not the inference hot path**. It does not touch the L0↔L1 CUDA-IPC transfer, so in practice the cost hides behind L2 (e.g. S3) latency. Encryption composes *after* quantization, so if you need to reduce L2 size, quantization is the lever — compression is not a useful alternative here (KV bytes are high-entropy and barely shrink). ## Failure behavior A decryption failure — tampered bytes, or the wrong key — fails the **integrity check** on the GCM tag. That surfaces as a load failure, which LMCache treats as a **cache miss**: the entry is re-fetched or recomputed. There is no silent corruption, and no served-from-cache garbage. This makes key rotation destructive to existing cache. Changing the master key means every L2 entry written under the old key fails its integrity check and is treated as a miss, so the L2 tier effectively starts cold. Plan rotation for a window where a cold L2 is acceptable. ## Verification After the engine pods restart: ```bash theme={null} # 1. The Secret is mounted read-only at the expected path kubectl -n tensormesh-platform exec ds/tensormesh-platform-default-engine -- \ ls -l /etc/lmcache/keys/master # 2. The engine's --l2-adapter argument carries the serde config kubectl -n tensormesh-platform get ds tensormesh-platform-default-engine \ -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -A1 l2-adapter ``` Then confirm the bytes at rest are actually opaque — the real test of the feature. Drive some traffic, let entries spill to L2, and inspect the stored objects directly (for the `fs` adapter, read a file under `base_path`; for Redis, `GET` a cache key). You should see high-entropy bytes, not recognizable prompt content. If you see plaintext, the most likely cause is an operator image older than the version above: the field is accepted by the CRD schema but never reconciled. Check the operator image tag. ## Recommended practice * Enable it when **L2 storage is outside your trust boundary**; skip it when L2 is as trusted as the GPU nodes themselves (it buys nothing there, since L1/L0 are plaintext regardless). * Keep the default **`aesBits: 128`** unless a compliance mandate says otherwise. * Store the master key in a real secret manager and sync it into the cluster (External Secrets, Vault, SOPS) rather than committing generated key files. * Treat the master key as **fleet-wide** — it is not a per-tenant boundary. For tenant-vs-tenant isolation, give each tenant its own engine. * Do not rely on encryption to hide **who cached how much** — that is metadata, and it stays visible. # Peer-to-Peer KV Transfer Source: https://docs.tensormesh.ai/v1.0.0/configuration/p2p Enable direct KV cache transfer between engine pods so warm prefixes can be read from any peer in the fleet, not just the local L1 or a shared L2 backend. Peer-to-peer (P2P) KV transfer lets engine pods read cached KV data directly from each other's memory over a transfer channel. When a pod receives a request whose prefix was already computed by another pod, it can fetch the KV tensors from that peer instead of recomputing them. This supplements the L1 (local memory) and L2 (Redis, filesystem, etc.) tiers with a third path: **any peer in the fleet can serve as a warm cache source**. P2P is most useful when: * your fleet serves many models or tenants and prompt reuse spans nodes * you want warm-hit latency close to L1 without provisioning a large shared L2 * your cluster has high-bandwidth interconnect (InfiniBand, RoCE, or fast TCP) For background on P2P in LMCache, see the upstream [P2P documentation](https://docs.lmcache.ai/mp/p2p.html). ## Prerequisites P2P requires the following minimum versions: | Component | Minimum version | | - | - | | Tensormesh Operator helm chart | `0.5.0` | | LMCache Operator (`lmcache/lmcache-operator`) | `v0.5.0` | | LMCache vLLM (`lmcache/vllm-openai`) | `v0.5.0` | ## Configuration Enable P2P by setting `engine.p2p.enabled: true` in your values file. The chart handles the rest — no manual CLI flags needed. ```yaml theme={null} engine: p2p: enabled: true port: 48500 # transfer channel port peers connect to l1AlignBytesOverwrite: 65536 # 64 KiB alignment, recommended for P2P ``` P2P also requires that every engine pod can reach every other engine pod on the transfer channel port. How you achieve that depends on your cluster's network fabric — see the networking options below. ## Networking options ### Option 1: Host networking (easiest to get started) The simplest approach is `hostNetwork: true`. The engine pod binds directly to the host's network namespace, so peers can reach each other without any CNI overlay or NAT. Because the pod shares the host's port space, move the default server ports to high values to avoid conflicts with other services on the node: ```yaml my-values.yaml theme={null} engine: enabled: true name: tensormesh-engine p2p: enabled: true port: 48500 # P2P transfer channel l1AlignBytesOverwrite: 65536 # 64 KiB, recommended for P2P spec: hostNetwork: true server: port: 45555 # gRPC (default 5555) httpPort: 48080 # health/admin (default 8080) l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v ``` Apply it with: ```bash theme={null} helm upgrade --install tensormesh-operator \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator \ -n tensormesh-operator --create-namespace \ -f my-values.yaml --wait ``` When `hostNetwork` is `true`, the operator automatically sets `dnsPolicy: ClusterFirstWithHostNet` so cluster DNS still works inside the engine pod. ### Option 2: HCA passthrough (coming soon) **Coming soon** — recommended for InfiniBand environments. HCA (Host Channel Adapter) passthrough exposes the InfiniBand device directly to the engine pod. This gives the transfer channel native RDMA verbs without any overlay or kernel bypass emulation. The operator will support scheduling HCA resources via the [k8s-rdma-shared-dev-plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin) so that each engine pod gets a dedicated HCA device. ### Option 3: SR-IOV (coming soon) **Coming soon** — recommended for RoCE environments. SR-IOV (Single Root I/O Virtualization) creates lightweight virtual functions (VFs) from a physical NIC, each assigned to a pod. This gives near-hardware RDMA performance while allowing multiple pods to share the same physical NIC. The operator will integrate with the [SR-IOV Network Operator](https://github.com/k8snetworkplumbingwg/sriov-network-operator) to request VFs via a `NetworkAttachmentDefinition`. ### Option 4: Macvlan (coming soon) **Coming soon** — recommended for RoCE environments. Macvlan creates a virtual interface directly on the physical NIC's L2 segment. Each pod gets its own MAC address and IP on the fabric network, bypassing the CNI overlay. This is simpler to set up than SR-IOV (no VF provisioning) but still gives pods direct L2 reachability on the RDMA fabric. The operator will integrate with [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) to attach a secondary macvlan interface to engine pods. ### Option 5: TCP (non-RDMA environments) If your cluster does not have InfiniBand or RoCE hardware, P2P still works over TCP. Performance is lower than RDMA but still faster than recomputing prefills from scratch. Use the same host-networking configuration from Option 1 and set `UCX_TLS` to TCP-only transports: ```yaml theme={null} engine: spec: env: - name: UCX_TLS value: "tcp,sm,self" ``` No special NIC passthrough or CNI plugin is needed. This is the right choice for: * development and testing clusters * cloud environments without RDMA-capable instances * clusters where the CNI overlay provides sufficient bandwidth ## Helm values reference | Value | Type | Default | Description | | - | - | - | - | | `engine.p2p.enabled` | bool | `false` | Enable P2P KV transfer between engine pods. | | `engine.p2p.port` | int | `8500` | Port the transfer channel advertises to peers. | | `engine.p2p.l1AlignBytesOverwrite` | int | `65536` | L1 buffer alignment in bytes. 64 KiB recommended for P2P. | | `engine.spec.hostNetwork` | bool | `false` | Run the pod in the host's network namespace. | | `engine.spec.server.port` | int | `5555` | gRPC server port. Move to a high port when using `hostNetwork`. | | `engine.spec.server.httpPort` | int | `8080` | HTTP health/admin port. Move to a high port when using `hostNetwork`. | ## Verification After deploying with P2P enabled, verify that the engine pods started correctly and the P2P transfer channel is active. ### Check the engine pod is running ```bash theme={null} kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine ``` ### Query the engine's L2 backends Curl the engine's HTTP endpoint to list the active L2 backends. When P2P is enabled, a P2P backend should appear alongside any configured L2 (Redis, filesystem, etc.): ```bash theme={null} ENGINE_POD=$(kubectl get pods -n tensormesh-operator \ -l app.kubernetes.io/component=cache-engine -o name | head -1) # Port-forward the HTTP port (use your configured httpPort) kubectl port-forward -n tensormesh-operator "$ENGINE_POD" 48080:48080 & curl -s http://localhost:48080/l2/backends | jq . ``` The L2 backends endpoint and its response format may change in future releases. This section will be updated as the API stabilizes. ## Common mistakes * Forgetting to move `server.port` and `server.httpPort` to high ports when using `hostNetwork`, causing bind failures on ports already in use by the host * Not setting `engine.p2p.enabled: true` — `hostNetwork` alone does not turn on P2P * Using the default P2P port (`8500`) on `hostNetwork` when another service already binds to it * Setting `UCX_TLS` to an RDMA transport (e.g. `rc_verbs`) without the corresponding device passthrough — UCX will fail to open the transport and the engine will not start # PD Disaggregation Source: https://docs.tensormesh.ai/v1.0.0/configuration/pd-disaggregation Separate the prefill and decode phases onto dedicated vLLM instances to improve TTFT and GPU utilization. PD disaggregation runs the two phases of LLM inference on separate vLLM instances: **prefill** handles prompt processing (compute-bound) and **decode** handles token generation (memory-bandwidth-bound). A router dispatches each request to the right pool. The prefiller transfers its KV tensors to the decoder over NIXL (GPU-to-GPU via the NVIDIA Transfer Library), so the decoder starts generating tokens without rerunning the prompt. PD disaggregation is most useful when: * TTFT degrades under heavy load because prefill and decode compete for the same GPU * you want to scale prefill and decode replicas independently to match traffic patterns * your cluster has GPU-to-GPU interconnect (NVLink, InfiniBand) that NIXL can exploit ## How it works A **single** `LMCacheEngine` DaemonSet serves both prefiller and decoder vLLM pods on every node. When `spec.pd` is set, the engine creates a connection ConfigMap with three `kv-transfer-config` keys: | Key | Content | | - | - | | `kv-transfer-config.json` | Bare `LMCacheMPConnector` (fallback for non-PD pods) | | `kv-transfer-config-prefiller.json` | `MultiConnector(NixlConnector + LMCacheMPConnector)` with `kv_role=kv_producer` | | `kv-transfer-config-decoder.json` | `MultiConnector(NixlConnector + LMCacheMPConnector)` with `kv_role=kv_consumer` | Each vLLM pod mounts the engine's connection ConfigMap and reads the role-specific key directly via `--kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-.json)"`. No webhook is required for this approach. The router is a plain `vllm-router` Deployment that you deploy alongside your vLLM pods. The operator does not manage it — you configure the `--prefill` and `--decode` URLs directly in the Deployment args. ## Prerequisites | Component | Minimum version | | - | - | | Tensormesh Platform helm chart | `1.0.0` | | LMCache Operator (`lmcache/lmcache-operator`) | `v0.5.5` | | LMCache vLLM (`lmcache/vllm-openai`) | `v0.5.5` | | cert-manager | any recent release | Create a dedicated workload namespace for your vLLM pods and label it PSS-privileged (the pods use `hostIPC: true` and `hostNetwork: true`): ```bash theme={null} kubectl create ns pd-workload kubectl label ns pd-workload pod-security.kubernetes.io/enforce=privileged ``` ## Install the chart A single engine handles both prefiller and decoder vLLM pods. Set `namespace: pd-workload` so the engine lands in the same namespace as your vLLM pods. ```yaml my-values.yaml theme={null} engine: enabled: true name: lmcache-engine namespace: pd-workload # same namespace as your vLLM pods spec: l1: sizeGB: 100 image: repository: lmcache/vllm-openai tag: v0.5.5 pullPolicy: IfNotPresent server: port: 5555 httpPort: 8080 chunkSize: 256 pd: nixlSideChannelPort: 5558 nixlLoadFailurePolicy: fail ``` ```bash theme={null} helm upgrade --install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ -n tensormesh-platform --create-namespace \ -f my-values.yaml --wait ``` ## Deploy vLLM and the router Deploy your vLLM prefiller, decoder, and router together. Both vLLM Deployments opt in to the same engine; the `lmcache.ai/pd-role` annotation tells the webhook which `kv-transfer-config` key to inject. ```yaml vllm-pd.yaml theme={null} ############################################################################### # PREFILLER ############################################################################### apiVersion: apps/v1 kind: Deployment metadata: name: vllm-prefiller namespace: pd-workload spec: replicas: 1 selector: matchLabels: app: vllm-prefiller template: metadata: labels: app: vllm-prefiller spec: hostIPC: true hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: vllm image: lmcache/vllm-openai:v0.5.5 imagePullPolicy: IfNotPresent securityContext: capabilities: add: ["IPC_LOCK"] command: ["/bin/sh", "-c"] args: - | exec python3 -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --port 8001 \ --enforce-eager \ --gpu-memory-utilization 0.4 \ --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-prefiller.json)" env: - name: PYTHONHASHSEED value: "0" - name: HUGGING_FACE_HUB_TOKEN valueFrom: secretKeyRef: name: hf-token # kubectl create secret generic hf-token --from-literal=token= key: token - name: VLLM_NIXL_SIDE_CHANNEL_HOST valueFrom: fieldRef: fieldPath: status.podIP - name: VLLM_NIXL_SIDE_CHANNEL_PORT value: "5557" # distinct port from decoder to avoid conflict on the same host ports: - name: http containerPort: 8001 volumeMounts: - name: kv-transfer-config mountPath: /etc/lmcache readOnly: true resources: limits: nvidia.com/gpu: "1" volumes: - name: kv-transfer-config configMap: name: lmcache-engine-connection # created by the LMCacheEngine controller --- apiVersion: v1 kind: Service metadata: name: vllm-prefiller namespace: pd-workload spec: selector: app: vllm-prefiller ports: - name: http port: 8001 targetPort: http --- ############################################################################### # DECODER ############################################################################### apiVersion: apps/v1 kind: Deployment metadata: name: vllm-decoder namespace: pd-workload spec: replicas: 1 selector: matchLabels: app: vllm-decoder template: metadata: labels: app: vllm-decoder spec: hostIPC: true hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: vllm image: lmcache/vllm-openai:v0.5.5 imagePullPolicy: IfNotPresent securityContext: capabilities: add: ["IPC_LOCK"] command: ["/bin/sh", "-c"] args: - | exec python3 -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --port 8002 \ --enforce-eager \ --gpu-memory-utilization 0.4 \ --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-decoder.json)" env: - name: PYTHONHASHSEED value: "0" - name: HUGGING_FACE_HUB_TOKEN valueFrom: secretKeyRef: name: hf-token key: token - name: VLLM_NIXL_SIDE_CHANNEL_HOST valueFrom: fieldRef: fieldPath: status.podIP - name: VLLM_NIXL_SIDE_CHANNEL_PORT value: "5558" ports: - name: http containerPort: 8002 volumeMounts: - name: kv-transfer-config mountPath: /etc/lmcache readOnly: true resources: limits: nvidia.com/gpu: "1" volumes: - name: kv-transfer-config configMap: name: lmcache-engine-connection --- apiVersion: v1 kind: Service metadata: name: vllm-decoder namespace: pd-workload spec: selector: app: vllm-decoder ports: - name: http port: 8002 targetPort: http --- ############################################################################### # ROUTER — plain Deployment, not operator-managed ############################################################################### apiVersion: apps/v1 kind: Deployment metadata: name: vllm-router namespace: pd-workload spec: replicas: 1 selector: matchLabels: app: vllm-router template: metadata: labels: app: vllm-router spec: containers: - name: router image: vllm/vllm-router:nightly # update to a stable tag when one is available command: ["/bin/sh", "-c"] args: - | exec vllm-router \ --policy round_robin \ --vllm-pd-disaggregation \ --prefill http://vllm-prefiller.pd-workload.svc.cluster.local:8001 \ --decode http://vllm-decoder.pd-workload.svc.cluster.local:8002 \ --host 0.0.0.0 \ --port 30000 \ --intra-node-data-parallel-size 1 ports: - name: http containerPort: 30000 readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 5 periodSeconds: 10 --- apiVersion: v1 kind: Service metadata: name: vllm-router namespace: pd-workload spec: selector: app: vllm-router ports: - name: http port: 30000 targetPort: http ``` ```bash theme={null} kubectl apply -f vllm-pd.yaml ``` The ConfigMap `lmcache-engine-connection` is created automatically when the `LMCacheEngine` reaches `Running` state. Wait for the engine to be ready before applying `vllm-pd.yaml`: `kubectl get lmcacheengine lmcache-engine -n pd-workload` ## NIXL RDMA networking NIXL uses UCX for GPU-to-GPU KV transfer. UCX requires valid RDMA GIDs, which are derived from the host's network interfaces. Under standard overlay CNI (each pod has its own network namespace) the GID table inside the pod is empty and UCX backend initialization fails. **Workarounds:** | Approach | Complexity | Notes | | - | - | - | | `hostNetwork: true` + `dnsPolicy: ClusterFirstWithHostNet` | Low | Quick test. With hostNetwork both roles share the host IP — set distinct `VLLM_NIXL_SIDE_CHANNEL_PORT` values (e.g. 5557 / 5558) in the pod env; the webhook will not override a pre-set value. | | SR-IOV with Multus | High | Production-grade. Each pod gets a dedicated VF with its own GID. No `hostNetwork` required. | For the `hostNetwork` workaround, add these fields to both the prefiller and decoder pod specs and pre-set distinct NIXL ports: ```yaml theme={null} spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: vllm env: - name: VLLM_NIXL_SIDE_CHANNEL_PORT value: "5557" # 5558 for the decoder ``` ## Multiple models One `LMCacheEngine` is shared across all models. Add a new router Deployment for each additional model, changing `--prefill`, `--decode`, `--port`, and the resource names: ```yaml theme={null} # router-qwen.yaml — second model alongside the Llama router above apiVersion: apps/v1 kind: Deployment metadata: name: vllm-router-qwen namespace: pd-workload spec: replicas: 1 selector: matchLabels: app: vllm-router-qwen template: metadata: labels: app: vllm-router-qwen spec: containers: - name: router image: vllm/vllm-router:nightly command: ["/bin/sh", "-c"] args: - | exec vllm-router \ --policy round_robin \ --vllm-pd-disaggregation \ --prefill http://vllm-prefiller-qwen.pd-workload.svc.cluster.local:8003 \ --decode http://vllm-decoder-qwen.pd-workload.svc.cluster.local:8004 \ --host 0.0.0.0 \ --port 30001 \ --intra-node-data-parallel-size 1 ports: - name: http containerPort: 30001 readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 5 periodSeconds: 10 --- apiVersion: v1 kind: Service metadata: name: vllm-router-qwen namespace: pd-workload spec: selector: app: vllm-router-qwen ports: - name: http port: 30001 targetPort: http ``` ## CacheBlend PD PD disaggregation also works with [CacheBlend](/v1.0.0/configuration/cacheblend). Use a `CacheBlendEngine` for the prefiller role and a plain `LMCacheEngine` for the decoder. Because CacheBlend ships a modified vLLM build, set `pd.enforceHandshakeCompat: false` on the prefiller to disable strict NIXL version negotiation: ```yaml theme={null} cacheBlend: enabled: true spec: pd: nixlSideChannelPort: 5557 enforceHandshakeCompat: false # required when prefiller and decoder run different vLLM builds ``` The decoder side uses a standard `LMCacheEngine` with `spec.pd` set (no `enforceHandshakeCompat` needed on the decoder). ## Helm values reference ### Engine `pd` spec | Field | Type | Default | Description | | - | - | - | - | | `pd.nixlSideChannelPort` | int | `5558` | Port the NIXL agent advertises for side-channel negotiation. With `hostNetwork: true`, use distinct values per role (e.g. 5557 / 5558) by pre-setting `VLLM_NIXL_SIDE_CHANNEL_PORT` in the pod env — the webhook will not override a pre-set value. | | `pd.nixlLoadFailurePolicy` | string | `fail` | What to do when the remote NIXL peer is unreachable. `fail` aborts; `ignore` falls back to local prefill. | | `pd.enforceHandshakeCompat` | bool | (NIXL default) | Set to `false` to disable strict NIXL version negotiation — required when a CacheBlend prefiller and a standard decoder run different vLLM builds. Omit to let NIXL use its own default. | ## Verification ### Engine reconciled ```bash theme={null} kubectl -n pd-workload get lmcacheengine kubectl -n pd-workload get configmap | grep connection ``` The engine should be `Running` and the `lmcache-engine-connection` ConfigMap should contain both prefiller and decoder keys. If it is missing, pods start without the connector config and KV transfers will silently fail. ### Router running ```bash theme={null} kubectl -n pd-workload get deploy vllm-router kubectl -n pd-workload get svc vllm-router ``` ### Connector config reached the pods After the vLLM pods start: ```bash theme={null} # NIXL env vars kubectl -n pd-workload get pod -l app=vllm-prefiller \ -o jsonpath='{.items[0].spec.containers[0].env[*].name}{"\n"}' \ | tr ' ' '\n' | grep NIXL ``` Expected output includes `VLLM_NIXL_SIDE_CHANNEL_HOST` and `VLLM_NIXL_SIDE_CHANNEL_PORT`. The manifests above mount the ConfigMap and pass `--kv-transfer-config` directly, so these values come from your own pod spec. Only a webhook-injected pod carries an `lmcache.ai/lmcache-injected` annotation, and the Helm chart does not register the LMCache webhook — so don't expect that annotation here. ### Send a request ```bash theme={null} kubectl port-forward -n pd-workload svc/vllm-router 30000:30000 & curl http://localhost:30000/v1/completions \ -H "Content-Type: application/json" \ -d '{"model":"Llama-3.1-8B-Instruct","prompt":"The capital of France is","max_tokens":20}' ``` ## Common mistakes * Omitting `namespace: ` on the engine — the engine and its connection ConfigMap land in the operator namespace; the vLLM pods mount the ConfigMap by name, so make sure the engine is in the same namespace as the pods * Applying `vllm-pd.yaml` before the engine is `Running` — the `lmcache-engine-connection` ConfigMap does not exist yet and the pods fail to start; wait for `kubectl get lmcacheengine` to show `Running` * NIXL RDMA backend fails to initialize — standard overlay CNI does not populate RDMA GIDs inside pods; `hostNetwork: true` + `dnsPolicy: ClusterFirstWithHostNet` is required for RDMA to work * Using the same `VLLM_NIXL_SIDE_CHANNEL_PORT` for prefiller and decoder — with `hostNetwork: true` both pods share the host IP so the ports conflict; use `5557` for prefiller and `5558` for decoder * Missing `IPC_LOCK` capability — NIXL needs to pin memory for RDMA operations; add `capabilities.add: ["IPC_LOCK"]` to the container security context * Workload namespace not PSS-privileged — Pod Security rejects `hostIPC: true` and `hostNetwork: true`; label the namespace `pod-security.kubernetes.io/enforce=privileged` # Compatibility Matrix / Supported Models Source: https://docs.tensormesh.ai/v1.0.0/installation/compatibility Models, serving engines, cache features, and recovery scenarios supported on the latest Tensormesh release — and the exact versions to deploy. This page lists what the current Tensormesh release supports: the models it serves, the serving engines it runs in front of, the cache and storage features it provides, and the failures it recovers from — along with the exact versions to deploy. Tensormesh is released and supported as a single matched set, so the versions below are the ones to run together. ## Compatibility matrix Tensormesh is released as a matched set. Deploy a single row's versions together — the cache engine, operator, and Helm chart are supported as a unit. | Cache engine | Operator | Helm chart | Serving image | Status | As of | | - | - | - | - | :-: | - | | `v0.5.5` | `v0.5.5` | `1.0.0` | `lmcache/vllm-openai:v0.5.5` | ✅ Supported | 2026-09-17 | *Current as of **2026-09-17**.* Deploy these versions together and pin them explicitly — never `latest`. Mixing versions across the cache engine, operator, and chart is not supported. # Model Support In Tensormesh Platform we distinguish what is verified and supported. A **verified** model or feature means it is part and tested as part of Tensormesh CI. It is ready for production. A **supported** model or feature requires testing prior to go on production as it is supported on LMCache opensource version but it has not been verified in our CI. In other word, we know/assume it works but it is not part of our CI process. ## Supported models ### CPU/Storage offloading | Model family | Models | | - | - | | Kimi | Kimi K3, Kimi K2.7, Kimi K2.5 | | GLM | GLM-5.2, GLM-5.1, GLM-4.7, GLM-4.5, GLM-4 | | Qwen | Qwen3.6 , Qwen3.5, Qwen3 | | DeepSeek | DeepSeek V4 Flash, DeepSeek V4 Pro, DeepSeek R1 | | MiniMax | MiniMax M3, MiniMax M2.7, MiniMax M2.5 | | Gemma | Gemma4, Gemma3 | | gpt-oss | gpt-oss-20b, gpt-oss-120b | | llama | llama 3.1 | ### CacheBlend | Model family | Models | | - | - | | GLM | GLM-5.2 | | MiniMax | MiniMax M3, MiniMax M2.5 | | Gemma | Gemma3 | | gpt-oss | gpt-oss-20b, gpt-oss-120b | | llama | llama 3.1 | ## Tensormesh Platform Verified ### CPU offloading performance CPU offloading performance speed up verified in the current release. | Model | Status | | - | - | | Deepseek r1 1.5b | ✅ VERIFIED | | glm 4 9b | ✅ VERIFIED | | glm 5.2 | ✅ VERIFIED | | Minimax m3 | ✅ VERIFIED | | Qwen3 0.6B | ✅ VERIFIED | | Qwen3.6 27b | ✅ VERIFIED | | Deepseek v4 Flash | ✅ VERIFIED | ### Cache features & External storage Cache and storage capabilities available on the current release. | Capability | Type | Status | | - | - | - | | l2 Encryption | — | ✅ VERIFIED | | L2 filesystem offload (fs\_native) | Filesystem | ✅ VERIFIED | | Payload Injection | — | ✅ VERIFIED | On the roadmap: GPUDirect Storage (GDS), pd Disaggregation, Peer-to-peer KV transfer. ### CacheBlend Non-prefix KV cache reuse — cached chunks are reused at shifted positions (re-RoPE + partial recompute), beyond what prefix caching can do. | Model | Status | | - | - | | glm 5 2 | ✅ VERIFIED | | gpt oss 120b | ✅ VERIFIED | | llama-3.1-70B | ✅ VERIFIED | ### Serving engines Serving engines supported in front of the Tensormesh cache. | Serving engine | Notes | Status | | - | - | - | | vLLM | Bundled in the serving image | ✅ VERIFIED | | NVIDIA Dynamo | — | ✅ VERIFIED | On the roadmap: SGLang, TensorRT-LLM, vLLM via stack-operator. ### Resilience Failures Tensormesh detects and recovers from automatically. | Scenario | Status | | - | - | | Coordinator pod Kill | ✅ VERIFIED | | Operator pod failure recovery | ✅ VERIFIED | | Vllm pod Kill | ✅ VERIFIED | | Cache engine pod failure recovery | ✅ VERIFIED | # E2E Quickstart Source: https://docs.tensormesh.ai/v1.0.0/installation/example Install Tensormesh Platform, point vLLM at the engine, run an inference, and verify the KV cache is reused. ## Step 1: Install Tensormesh Platform Follow [Install with Helm](/v1.0.0/installation/helm) If you already run your own vLLM Deployment, do not replace it with this demo manifest. Use [Modify an Existing Deployment](/v1.0.0/installation/existing-deployment) for the minimum patch set instead. ## Step 2: Deploy vLLM connected to the engine The chart creates a ConfigMap named `-connection` holding the `kv-transfer-config.json` that tells vLLM how to reach the engine over the MP connector. ### Read the connector config from the engine's ConfigMap ```bash theme={null} kubectl get cm tensormesh-platform-default-engine-connection \ -n tensormesh-platform \ -o jsonpath='{.data.kv-transfer-config\.json}' | jq -c . ``` You should get one line like this: ```json theme={null} {"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-platform-default-engine.tensormesh-platform.svc.cluster.local","lmcache.mp.isolated_ipc":true,"lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"} ``` ### Create the Deployment Paste that line as the `--kv-transfer-config` value in `vllm-demo.yaml`: ```yaml vllm-demo.yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: vllm-demo namespace: tensormesh-platform labels: { app: vllm-demo } spec: selector: { matchLabels: { app: vllm-demo } } template: metadata: labels: { app: vllm-demo } spec: nodeSelector: nvidia.com/gpu.present: "true" hostIPC: true containers: - name: vllm image: lmcache/vllm-openai:v # use the same version as the lmcache engine env: - { name: PYTHONHASHSEED, value: "0" } args: - Qwen/Qwen3-0.6B - --gpu-memory-utilization - "0.6" - --no-enable-prefix-caching - --max-model-len - "32768" - --kv-transfer-config # the single line from the previous step - '{"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-platform-default-engine.tensormesh-platform.svc.cluster.local","lmcache.mp.isolated_ipc":true,"lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"}' ports: [{ name: http, containerPort: 8000 }] resources: limits: { nvidia.com/gpu: "1", memory: 32Gi } requests: { cpu: "2", memory: 6Gi } readinessProbe: httpGet: { path: /health, port: http } initialDelaySeconds: 60 periodSeconds: 15 failureThreshold: 40 --- apiVersion: v1 kind: Service metadata: name: inference namespace: tensormesh-platform spec: selector: { app: vllm-demo } ports: [{ name: http, port: 8000, targetPort: http }] ``` Apply and wait for the Pod to become Ready (5–10 min on first boot for image pull + model download): ```bash theme={null} kubectl apply -f vllm-demo.yaml kubectl wait --for=condition=Ready pod -l app=vllm-demo \ -n tensormesh-platform --timeout=15m ``` ## Step 3: Fire two identical requests, see the cache work The simplest possible "is LMCache working?" test is to send two identical long prompts. The first populates the cache; the second should reuse the stored KV blocks. Port-forward the service in one terminal: ```bash theme={null} kubectl port-forward -n tensormesh-platform svc/inference 8000:8000 --address 127.0.0.1 ``` If port `8000` is already in use locally, forward to a different local port (for example `... 8001:8000`) and use that port in the requests below. In a second terminal, build the request once into a variable, then fire it twice. Keeping the JSON in a `PAYLOAD` variable (rather than inlining `$(jq …)` inside a multi-line `curl`) makes it safe to copy-paste and guarantees both calls send the **identical** prompt: ```bash theme={null} KV_PROMPT=$(python3 -c "print('Tell me a long story about a brave knight. ' * 30)") PAYLOAD=$(jq -n --arg p "$KV_PROMPT" \ '{model:"Qwen/Qwen3-0.6B", prompt:$p, max_tokens:20, temperature:0}') # Call 1: cold, should STORE KV into LMCache time curl -sS http://localhost:8000/v1/completions \ -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text' # Call 2: identical, should RETRIEVE prefix KV from LMCache time curl -sS http://localhost:8000/v1/completions \ -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text' ``` Then check the engine logs for store and retrieve markers: ```bash theme={null} # select by the engine CR's name; the component label (cache-engine) also matches # CacheBlend engine pods when those are enabled kubectl logs -n tensormesh-platform --tail=200 \ -l app.kubernetes.io/instance=tensormesh-platform-default-engine \ | grep -E "Stored [0-9]+ tokens|Prefetch request completed.*retained keys" ``` A working install logs `Stored N tokens` after call 1 and `Prefetch request completed ... retained keys` during call 2. Sample output from a Qwen3-0.6B install on a single A100 40 GB, prompt repeated 30 times for \~514 tokens: ```text theme={null} ========================================== CALL 1 (cold — expect STORE) ========================================== elapsed: 1.466s usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534} ========================================== CALL 2 (warm — expect RETRIEVE) ========================================== elapsed: 0.548s usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534} ========================================== ENGINE LOG MARKERS ========================================== LMCache INFO: Stored 512 tokens in 0.029 seconds LMCache INFO: Prefetch request completed (L1+L2): 2/2 retained keys (2 L1, 0 L2) in 0.9 ms ``` Call 2 was \~2.7× faster than call 1 (1.466 s → 0.548 s). The engine reports `2/2 retained keys` from L1, meaning the second request reused the KV blocks stored during the first. vLLM's own metrics line confirms the same number from the inference side: ```text theme={null} External prefix cache hit rate: 49.8% ``` (The counter is cumulative across both calls: call 1 hit 0 of 514 tokens, call 2 hit 512 of 514, so 512/1028 = 49.8%. For a two-call test \~50% is the ceiling, not a partial hit.) Prompts must exceed the engine's `chunk_size` (256 tokens by default) for any KV blocks to be stored. Short prompts produce no markers. That's expected, not a failure. ## Step 4: Run the benchmark For a more realistic measurement than two curls, use `vllm bench serve` to compare a cold pass against a warm pass. Run **two passes with the same `--seed`** so the prompts repeat and the second pass hits the cache. Rather than installing the vLLM CLI locally, run it **inside the cluster as a `Job`**. The vLLM image already ships the CLI, talks to the `inference` Service directly (no port-forward), and a `Job` runs it once to completion and stops. The benchmark is a pure HTTP client, so it needs **no GPU**. Save as `benchmark-job.yaml`: ```yaml benchmark-job.yaml theme={null} apiVersion: batch/v1 kind: Job metadata: name: lmcache-benchmark namespace: tensormesh-platform spec: backoffLimit: 0 ttlSecondsAfterFinished: 3600 template: metadata: labels: { app: lmcache-benchmark } spec: restartPolicy: Never containers: - name: bench image: lmcache/vllm-openai:v # same tag as the Deployment, so the node's cached image is reused imagePullPolicy: IfNotPresent env: - { name: HF_HOME, value: /tmp/hf } # writable under OpenShift's random UID - { name: HF_HUB_DISABLE_TELEMETRY, value: "1" } command: ["/bin/sh", "-c"] args: - | set -e run() { vllm bench serve --backend openai --base-url http://inference:8000 \ --model Qwen/Qwen3-0.6B --dataset-name random \ --num-prompts 20 --random-input-len 20480 \ --random-output-len 1 --seed 32 } echo "### PASS 1 (cold) ###"; run sleep 5 echo "### PASS 2 (warm) ###"; run volumeMounts: - { name: hf, mountPath: /tmp/hf } resources: requests: { cpu: "1", memory: 2Gi } limits: { cpu: "2", memory: 4Gi } volumes: - { name: hf, emptyDir: {} } ``` Run it and stream the results: ```bash theme={null} kubectl apply -f benchmark-job.yaml kubectl wait --for=condition=complete job/lmcache-benchmark \ -n tensormesh-platform --timeout=10m kubectl logs -f job/lmcache-benchmark -n tensormesh-platform ``` The logs contain two `Serving Benchmark Result` blocks. A working cache shows **pass 2 with much lower TTFT** (mean/p50/p99). The speedup scales with `--random-input-len`: with a fixed `--seed`, both passes send the same 20 prompts, so each unique body is prefilled on the cold pass and served from cache on the warm pass. Bigger `--random-input-len` → bigger delta. Example result from a working run with `--num-prompts 20`, `--random-input-len 20480`, and `--seed 32`: ```text theme={null} ### PASS 1 (cold) ### Successful requests: 20 Benchmark duration (s): 11.00 Request throughput (req/s): 1.82 Mean TTFT (ms): 6399.80 Median TTFT (ms): 6456.31 P99 TTFT (ms): 10896.55 ### PASS 2 (warm) ### Successful requests: 20 Benchmark duration (s): 5.64 Request throughput (req/s): 3.55 Mean TTFT (ms): 3697.56 Median TTFT (ms): 3753.21 P99 TTFT (ms): 5584.58 ``` In the healthy case above, pass 2 cuts TTFT substantially relative to pass 1 and nearly doubles request throughput. Keep `num_prompts × random-input-len` of KV under `engine.spec.l1.sizeGB` (else pass 1's entries evict before pass 2 reads them, causing warm misses), and `random-input-len + random-output-len ≤ --max-model-len` (32768 in the Deployment above). Clean up the Job (it also self-deletes after 1h via `ttlSecondsAfterFinished`): ```bash theme={null} kubectl delete job lmcache-benchmark -n tensormesh-platform ``` ## Next steps Full chart reference, install modes, every tunable value. Every `values.yaml` key, with example overlays. Metrics, dashboards, and performance tuning. `Pending` pods, image pull, hung uninstall, ownership conflicts. # Modify an Existing vLLM Deployment Source: https://docs.tensormesh.ai/v1.0.0/installation/existing-deployment Patch an existing vLLM deployment to consume the LMCache engine created by the Tensormesh Operator. Most customers already run vLLM some other way before they install the Tensormesh Platform. This page covers the **minimum changes** needed to patch an existing vLLM Deployment so it uses the operator-managed LMCache engine in MP mode. There are two ways to do it, and they end at the same pod spec: | | [Webhook injection](#webhook-injection) | [Modify the Deployment directly](#modify-the-vllm-deployment-directly) | | - | - | - | | What you change | two lines of pod metadata | the container's args, env and volumes | | Who writes the wiring | the mutating webhook, at admission | you, in your own manifests | | Needs cert-manager + `webhook.enabled` | yes | no | | `lmcache` version matching | automatic, via [payload injection](#payload-injection-stop-matching-lmcache-versions-by-hand) | yours to manage (or stage the payload yourself) | | Engine in another namespace | no — the webhook resolves the engine in the pod's own namespace | yes — the connector config carries a fully-qualified Service name | | Works with `command:` on the container | no — the webhook skips such pods | yes | ## Webhook injection ### Enable injection in the chart Injection is configured once, on the operator release — nothing here goes into your vLLM Deployment. Two values matter: the webhook must be on (it is by default), and the engine must carry a payload image if you want [payload injection](#payload-injection-stop-matching-lmcache-versions-by-hand) as well as connection wiring. ```yaml my-values.yaml theme={null} webhook: enabled: true # default; requires cert-manager engine: enabled: true namespace: my-workload # NOT the release namespace — see the warning below spec: l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v0.5.5 injection: payloadImage: repository: lmcache/lmcache-payload tag: v0.5.5 # must match the engine's lmcache — same wire protocol pullPolicy: IfNotPresent ``` ```bash theme={null} helm upgrade --install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ --namespace tensormesh-platform --create-namespace \ -f my-values.yaml ``` ### Opt your Deployment in Add two lines of metadata on the vLLM pod template: ```yaml theme={null} spec: template: metadata: labels: lmcache.ai/lmcache-inject: "true" annotations: lmcache.ai/lmcache-engine: "tensormesh-platform-default-engine" ``` At admission the webhook adds, to the target container: | Injected | Purpose | | - | - | | `--kv-transfer-config ` | the engine's connection contract, read from `-connection` | | `/dev/shm` | MP-mode shared memory — pod-private in isolated-IPC mode (the default since operator `v0.5.5`); the host's `/dev/shm`, or `hostIPC`, only when the engine sets `isolatedIPC: false` | | `PYTHONHASHSEED=0` | deterministic token hashing, so vLLM and the engine derive the same cache keys | | `lmcache.ai/lmcache-injected: "true"` | idempotency stamp — a second admission is a no-op | **The engine must be in the same namespace as the pod.** The injector resolves `-connection` in the pod's own namespace (`req.Namespace`), so the annotation takes a bare engine name, not `/`. The chart's own release namespace is excluded from the webhook, so run the vLLM Deployment in a namespace alongside an engine — that is why the values above put the engine in `my-workload`. If you cannot move either one, [wire the Deployment directly](#modify-the-vllm-deployment-directly) instead. ### Payload injection: stop matching lmcache versions by hand Ordinarily your vLLM image must already carry a **compatible external `lmcache`**, matched to the engine's version. That is the awkward part of running an existing Deployment against a managed engine, and payload injection removes it. When the engine carries an `injection.payloadImage` ([set it in the chart](#enable-injection-in-the-chart)), the webhook injects these as well as the connection wiring: | Injected | Purpose | | - | - | | `lmcache-payload-stage` init container | copies the engine's `lmcache` into a shared volume | | `lmcache-payload` emptyDir + read-only mount | where the staged runtime lands | | `PYTHONPATH=/lmcache-payload` | makes vLLM import the **staged** lmcache ahead of its baked-in one | So an older vLLM image runs the engine's lmcache without being rebuilt. The payload image is published alongside every engine release and nightly, with matching tags (`v0.5.5`, `latest-nightly`, …) — pin it to the engine's tag, because client and server share a versioned protocol. ### Verify the injection Confirm the pod spec was rewritten: ```bash theme={null} kubectl get pod -o jsonpath='{.metadata.annotations}' | grep lmcache kubectl get pod -o jsonpath='{.spec.initContainers[*].name}' kubectl get pod -o jsonpath='{.spec.containers[0].env[?(@.name=="PYTHONPATH")].value}' ``` Then prove which `lmcache` is actually imported at runtime — the staged one, and the baked-in one for contrast: ```bash theme={null} # staged (what vLLM uses) kubectl exec -c vllm -- python3 -c 'import lmcache; print(lmcache.__version__, lmcache.__file__)' # 0.5.5 /lmcache-payload/lmcache/__init__.py # baked into the image kubectl exec -c vllm -- env -u PYTHONPATH python3 -c 'import lmcache; print(lmcache.__version__, lmcache.__file__)' # 0.5.2 /opt/venv/lib/python3.12/site-packages/lmcache/__init__.py ``` Different paths and versions means payload injection is doing its job. ### When the webhook skips your pod Injection **fails open**: the pod is always admitted, and the reason is stamped on `lmcache.ai/lmcache-skip-reason`. Read it before assuming the webhook is broken: ```bash theme={null} kubectl get pod -o jsonpath='{.metadata.annotations.lmcache\.ai/lmcache-skip-reason}' ``` | Skip reason | Meaning | What to do | | - | - | - | | `command-override` | the container sets `command:` | move the flags into `args:` so the image's entrypoint runs — see below | | `engine-not-found` | no `-connection` in the pod's namespace | fix the annotation, or move the pod to the engine's namespace | | `kv-transfer-config-present` | you already pass `--kv-transfer-config` | intentional; yours is never clobbered | | `target-container-not-found` | multi-container pod, wrong target | name it with `lmcache.ai/lmcache-container` | | `payload-image-unset` | engine has no `injection.payloadImage` | set it, or supply `lmcache` in your image | ## Modify the vLLM Deployment directly Use this when webhook injection is not an option — no cert-manager, `webhook.enabled: false`, a container that sets `command:`, an engine in a different namespace, or simply because you want every change visible in your own manifests. You write by hand exactly what the webhook would have written. ### What to add | What the webhook injects | What you add by hand | | - | - | | `--kv-transfer-config ` | the `kv-transfer-config.json` value from the `-connection` ConfigMap, passed as an arg | | `PYTHONHASHSEED=0` | the same env var on the vLLM container | | `/dev/shm` wiring | a volume at `/dev/shm` matching [the engine's IPC mode](#match-the-engines-ipc-mode) | | staged `lmcache` | an `lmcache` in your image matching the engine's version — or [stage the payload yourself](#stage-lmcache-yourself) | Start by reading the contract the engine published: ```bash theme={null} kubectl get configmap -connection -n \ -o jsonpath='{.data.kv-transfer-config\.json}' ``` ```json theme={null} { "kv_connector": "LMCacheMPConnector", "kv_connector_module_path": "lmcache.integration.vllm.lmcache_mp_connector", "kv_role": "kv_both", "kv_connector_extra_config": { "lmcache.mp.host": "tcp://..svc.cluster.local", "lmcache.mp.port": "5555" } } ``` The host is fully qualified, so — unlike webhook injection — your vLLM pod does not have to run in the engine's namespace. Mounting the ConfigMap still does; when the namespaces differ, paste the JSON into your manifest instead. ### Patch your Deployment Referencing the ConfigMap through an env var keeps the container **args-only**, so nothing about how your image starts has to change — Kubernetes expands `$(LMCACHE_KV_CONFIG)` in `args` from the env defined on the same container: ```yaml theme={null} spec: template: spec: containers: - name: vllm env: # Deterministic token hashing. Without it vLLM and the engine hash the # same prompt to different keys, and every lookup misses. - name: PYTHONHASHSEED value: "0" # The engine's connection contract. Read at pod start, so a rolling # restart picks up a changed engine port or name. - name: LMCACHE_KV_CONFIG valueFrom: configMapKeyRef: name: tensormesh-platform-default-engine-connection key: kv-transfer-config.json args: # … your existing vLLM args, unchanged … - --kv-transfer-config - $(LMCACHE_KV_CONFIG) volumeMounts: - name: lmcache-dev-shm mountPath: /dev/shm volumes: # isolatedIPC: true (the default) — see the next section. - name: lmcache-dev-shm emptyDir: medium: Memory ``` If your container already runs a shell through `command:`, mount the ConfigMap as a file and read it inline instead — this is the form the e2e suite uses: ```yaml theme={null} args: - | exec python3 -m vllm.entrypoints.openai.api_server \ --model \ --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config.json)" volumeMounts: # on the vLLM container - { name: kv-transfer-config, mountPath: /etc/lmcache, readOnly: true } volumes: # on the pod spec - name: kv-transfer-config configMap: { name: tensormesh-platform-default-engine-connection } ``` ### Match the engine's IPC mode The `/dev/shm` wiring is not cosmetic: it is how the engine and vLLM exchange CUDA IPC handles, and it has to agree with the engine's own setting. Check it with: ```bash theme={null} kubectl get lmcacheengine -n \ -o jsonpath='{.spec.isolatedIPC}{" hostIPC="}{.spec.hostIPC}{"\n"}' ``` | Engine | What the pod needs | | - | - | | `isolatedIPC: true` — unset defaults to this for `gpuVendor: nvidia` | a pod-private `/dev/shm`: `emptyDir` with `medium: Memory`, as above. Nothing is shared with the engine; the volume exists because Kubernetes' default 64Mi `/dev/shm` is too small for vLLM's own workers | | `isolatedIPC: false` | the **host's** tmpfs: `hostPath` with `path: /dev/shm`, `type: Directory`, mounted at `/dev/shm` | | `isolatedIPC: false` **and** `hostIPC: true` | `hostIPC: true` on the pod spec, and no `/dev/shm` volume at all | ### Stage lmcache yourself Without the webhook, your vLLM image must already carry an `lmcache` matching the engine's version — client and server share a versioned wire protocol. If it does not, replicate what payload injection does, using the same payload image: ```yaml theme={null} spec: template: spec: initContainers: # No command/args: the payload image's own entrypoint copies its tree # into $SHARED_DIR. - name: lmcache-payload-stage image: lmcache/lmcache-payload:v0.5.5 # pin to the engine's lmcache env: - name: SHARED_DIR value: /lmcache-payload volumeMounts: - name: lmcache-payload mountPath: /lmcache-payload containers: - name: vllm env: # Import the staged tree ahead of the image's baked-in lmcache. - name: PYTHONPATH value: /lmcache-payload volumeMounts: - name: lmcache-payload mountPath: /lmcache-payload readOnly: true volumes: - name: lmcache-payload emptyDir: {} ``` Verify it the same way as [the webhook path](#verify-the-injection) — compare `import lmcache` with and without `PYTHONPATH`. ## OpenShift-specific patch This applies to **both** approaches. Having the shared-memory wiring in the pod spec — however it got there — does not grant the pod permission to use it. On OpenShift the vLLM pod also needs an SCC that permits it: `hostIPC`, or the `/dev/shm` hostPath used when the engine runs with `isolatedIPC: false`. The chart handles only the engine pod's ServiceAccount; it does not patch your pre-existing vLLM Deployment. (An engine left on the `isolatedIPC: true` default needs none of this — its `/dev/shm` is a plain pod-private `emptyDir`.) Under webhook injection, a pod whose SCC forbids the injected wiring is **rejected after** a successful injection, so the skip-reason annotation will be absent — look at the pod's events instead. Your vLLM pod therefore needs: * `hostIPC: true` * a `serviceAccountName` bound to a suitable SCC If your vLLM Deployment is in the same namespace as the chart-managed engine, you can usually reuse the chart-created privileged ServiceAccount: ```yaml theme={null} spec: template: spec: hostIPC: true serviceAccountName: tensormesh-platform-engine-privileged ``` If the Deployment is in another namespace, that namespace needs its own ServiceAccount plus SCC binding. ## Verify the patched deployment This applies to both approaches — it checks that KV actually reaches the engine, not how the pod was wired. After rolling out the Deployment: ```bash theme={null} kubectl rollout status deploy/ --timeout=15m kubectl get pod -l app= -o wide kubectl exec -it -- \ python3 -c 'import lmcache.integration.vllm.lmcache_mp_connector as m; print(m.__file__)' ``` Then send two identical long prompts through the vLLM Service and check the engine logs. Port-forward the Service in one terminal: ```bash theme={null} kubectl port-forward -n svc/ 8000:8000 ``` In a second terminal, build the request once and send it twice: ```bash theme={null} PROMPT=$(python3 -c "print('Tell me a long story about a brave knight. ' * 30)") PAYLOAD=$(jq -n --arg p "$PROMPT" \ '{model:"Qwen/Qwen3-0.6B", prompt:$p, max_tokens:20, temperature:0}') # Call 1 — cold, should STORE KV into LMCache time curl -sS http://localhost:8000/v1/completions \ -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text' # Call 2 — identical, should RETRIEVE prefix KV from LMCache time curl -sS http://localhost:8000/v1/completions \ -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text' ``` Then inspect the engine logs: ```bash theme={null} # select by the engine CR's name — the component label (cache-engine) also # matches CacheBlend engine pods when those are enabled kubectl logs -n \ -l app.kubernetes.io/instance=tensormesh-platform-default-engine --tail=200 \ | grep -E "Stored [0-9]+ tokens|Prefetch request completed.*prefix hits" ``` Healthy behavior is: * call 1 prints a normal completion and the engine logs `Stored N tokens` * call 2 sends the exact same payload and the engine logs `Prefetch request completed ... prefix hits=N` * call 2 is usually noticeably faster than call 1 for a long enough prompt # Getting Started Source: https://docs.tensormesh.ai/v1.0.0/installation/getting-started Pick an install method for Tensormesh Platform and verify your cluster is ready. This page covers what you need before installing **Tensormesh Platform** and helps you pick an install method. The Helm method installs the operator and its CRDs; "Modify an Existing vLLM Deployment" wires an existing vLLM workload to an operator you've already installed. ## What gets installed A small controller-manager Deployment that watches the CRs below cluster-wide. It also serves the mutating webhook that wires vLLM pods to the engine. `LMCacheEngine`, `LMCacheCoordinator`, and `CacheBlendEngine` — the specs the operator reconciles. One LMCache cache server pod per GPU node, created by the operator from your CR. A fleet-wide Deployment the engines register with, for cross-node cache coordination. ## Prerequisites The chart's `kubeVersion` constraint is `>=1.28.0-0`. OpenShift 4.16+ and EKS / GKE / AKS on a supported control plane all qualify. The engine DaemonSet schedules onto nodes labeled `nvidia.com/gpu.present=true`. On clusters with the NVIDIA GPU Operator installed, this label is set automatically. On clusters without it, label your GPU nodes manually: ```bash theme={null} kubectl label node nvidia.com/gpu.present=true ``` On a cluster with no GPU nodes the install still succeeds; the DaemonSet stays at `0` desired replicas until a matching node appears. The NVIDIA GPU Operator exposes GPUs via the Container Device Interface (CDI). Before GPU Operator v25.10.0, CDI was off by default (`cdi.enabled: false`). With it off, the engine/vLLM pods get stuck in `ContainerCreating` with no clear error. On those versions, enable CDI in the GPU `ClusterPolicy`: ```bash theme={null} oc patch clusterpolicy gpu-cluster-policy --type=merge -p '{ "spec": { "cdi": { "enabled": true, "default": true } } }' ``` GPU Operator v25.10.0+ enables CDI by default; no action needed. See [Troubleshooting → Pod stuck in `ContainerCreating`](/v1.0.0/installation/troubleshooting#pod-stuck-in-containercreating-cdi) if you hit this. Installation creates cluster-scoped objects (a CRD, a `ClusterRole`, and a `ClusterRoleBinding`). You need permission to create those. `kubectl` on Kubernetes, `oc` on OpenShift. The two are interchangeable for everything in these docs. Examples are shown with `kubectl` unless an OpenShift-specific command is required. ## Pick an install method The primary path. Single `helm install` command, simple upgrades and rollbacks. Patch an existing vLLM Deployment to mount the engine connection ConfigMap and use LMCache MP mode safely. ## Verify your cluster is ready Run these three preflight checks before installing. ```bash theme={null} kubectl version ``` Look at the `Server Version` line: anything `v1.28.0` or newer qualifies. OpenShift users can also run `oc version`; the `Kubernetes Version` line is what matters. ```bash theme={null} kubectl get nodes -l nvidia.com/gpu.present=true ``` A non-empty list means the engine DaemonSet has somewhere to schedule. An empty list is fine for an operator-only install; see [Install with Helm](/v1.0.0/installation/helm#quick-install). ```bash theme={null} kubectl auth can-i create customresourcedefinitions.apiextensions.k8s.io kubectl auth can-i create clusterroles.rbac.authorization.k8s.io kubectl auth can-i create clusterrolebindings.rbac.authorization.k8s.io ``` All three must return `yes`. As a shortcut, `kubectl auth can-i '*' '*' --all-namespaces` returning `yes` means you're effectively cluster-admin. Each command also prints a `Warning: resource '...' is not namespace scoped` line; that's expected (these *are* cluster-scoped resources) and harmless. The `yes`/`no` answer is what matters. If any check fails: bump the cluster version, label the GPU nodes (see the prerequisites above), or ask a cluster admin for the missing RBAC. # Install with Helm Source: https://docs.tensormesh.ai/v1.0.0/installation/helm Install Tensormesh Platform from the official Helm chart. Helm is the primary install path for Tensormesh Platform. A single `helm install` command deploys the operator, registers its CRDs (`LMCacheEngine`, `LMCacheCoordinator`, `CacheBlendEngine`), and by default creates an lmcache engine CR that the operator reconciles into a DaemonSet on your GPU nodes, plus a fleet coordinator the engines register with. ## Prerequisites * Helm `3.8` or newer (`helm version`) * Cluster access with permission to create cluster-scoped resources * A reachable Kubernetes 1.28+ cluster, see [Getting Started](/v1.0.0/installation/getting-started) * [cert-manager](https://cert-manager.io/docs/installation/). The chart enables the mutating webhook by default and cert-manager issues its serving certificate: ```bash theme={null} kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml kubectl -n cert-manager wait --for=condition=Available deploy --all --timeout=180s ``` * **Access token from the Tensormesh team** The chart is distributed through a private registry. To install it, request an access token from [https://www.tensormesh.ai/contact](https://www.tensormesh.ai/contact). You'll use it to authenticate before installing. ## Quick Install Authenticate to the registry with the token Tensormesh provided, write your configuration into `my-values.yaml`, and install: ```yaml my-values.yaml theme={null} engine: enabled: true spec: privileged: true l1: sizeGB: 200 # per-node L1 cache size image: repository: lmcache/vllm-openai tag: v0.5.5 # pin the tag the Compatibility Matrix pairs with your chart pullPolicy: IfNotPresent coordinator: spec: image: repository: lmcache/vllm-openai tag: v0.5.5 # Same image as engine pullPolicy: IfNotPresent ``` ```bash theme={null} # One-time: log in with the token Tensormesh gave you. When authenticating with a token, # the username is not validated. Any non-empty value works (we use "tensormesh" here). echo '' | helm registry login artifacts.tensormesh.ai -u tensormesh --password-stdin helm install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ --namespace tensormesh-platform \ --create-namespace \ -f my-values.yaml \ --wait ``` **If install fails, jump to [Troubleshooting](/v1.0.0/installation/troubleshooting).** ## Verify ```bash theme={null} helm list -n tensormesh-platform ``` Example: ```text theme={null} NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION tensormesh-platform tensormesh-platform 1 2026-08-12 02:05:00 UTC deployed tensormesh-platform-1.0.0 v0.5.5 ``` `STATUS` should be `deployed`. ```bash theme={null} kubectl get pods -n tensormesh-platform ``` Example: ```text theme={null} NAME READY STATUS RESTARTS AGE tensormesh-platform-5f47f76bd6-bzmqf 1/1 Running 0 2m tensormesh-platform-coordinator-7c6446f7c8-zc7g7 1/1 Running 0 2m tensormesh-platform-default-engine-gfj2k 1/1 Running 0 2m ``` ```bash theme={null} kubectl get crd | grep lmcache.ai ``` Example: ```text theme={null} cacheblendengines.lmcache.lmcache.ai 2026-08-12T02:04:10Z lmcachecoordinators.lmcache.lmcache.ai 2026-08-12T02:04:10Z lmcacheengines.lmcache.lmcache.ai 2026-08-12T02:04:10Z ``` The chart ships all three CRDs (`crds.enabled=true`). ```bash theme={null} kubectl get lmcacheengines -n tensormesh-platform ``` Example: ```text theme={null} NAME PHASE READY DESIRED AGE tensormesh-platform-default-engine Running 1 1 2m ``` ```bash theme={null} # select by the engine CR's name (instance label), the component label # (cache-engine) also matches CacheBlend engine pods when those are enabled kubectl get pods -n tensormesh-platform -o wide \ -l app.kubernetes.io/instance=tensormesh-platform-default-engine ``` Example: ```text theme={null} NAME READY STATUS RESTARTS AGE IP NODE tensormesh-platform-default-engine-gfj2k 1/1 Running 0 2m 10.129.6.79 worker-gpu-a100-1 ``` Expect one pod per GPU node. On a cluster with no GPU nodes this list is empty, which is correct, not a failure. ```bash theme={null} helm test tensormesh-platform -n tensormesh-platform ``` Example: ```text theme={null} NAME: tensormesh-platform LAST DEPLOYED: Sat May 24 02:05:00 2026 NAMESPACE: tensormesh-platform STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: tensormesh-platform-test Last Started: 2026-05-24 02:08:00 +0000 UTC Last Completed: 2026-05-24 02:08:01 +0000 UTC Phase: Succeeded TEST SUITE: tensormesh-platform-test-basic Last Started: 2026-05-24 02:08:10 +0000 UTC Last Completed: 2026-05-24 02:08:14 +0000 UTC Phase: Succeeded TEST SUITE: tensormesh-platform-test-coordinator Last Started: 2026-05-24 02:08:20 +0000 UTC Last Completed: 2026-05-24 02:08:23 +0000 UTC Phase: Succeeded TEST SUITE: tensormesh-platform-test-lmcache-webhook Last Started: 2026-05-24 02:08:15 +0000 UTC Last Completed: 2026-05-24 02:08:19 +0000 UTC Phase: Succeeded ``` ## Upgrade ```bash theme={null} helm upgrade tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 -n tensormesh-platform \ -f my-values.yaml ``` Roll back if needed: ```bash theme={null} helm history tensormesh-platform -n tensormesh-platform helm rollback tensormesh-platform -n tensormesh-platform ``` ## Uninstall ```bash theme={null} helm uninstall tensormesh-platform -n tensormesh-platform ``` A pre-delete hook removes the chart-managed CRs (`LMCacheEngine`, `LMCacheCoordinator`, and `CacheBlendEngine` if enabled) first, so their finalizers run cleanly before the operator Deployment is torn down. **The CRDs survive `helm uninstall`**. Same convention cert-manager and kube-prometheus-stack use. To remove them (and cascade-delete any remaining CRs): ```bash theme={null} kubectl delete crd lmcacheengines.lmcache.lmcache.ai \ lmcachecoordinators.lmcache.lmcache.ai \ cacheblendengines.lmcache.lmcache.ai ``` ## Helm Values | Key | Default | What it creates | | - | - | - | | `crds.enabled` | `true` | The three CRD schemas | | `operator.enabled` | `true` | The controller-manager Deployment | | `engine.enabled` | `true` | An `LMCacheEngine` CR, which the operator reconciles into a DaemonSet (one cache server per GPU node) | | `coordinator.enabled` | `true` | An `LMCacheCoordinator` CR, one fleet-wide Deployment the engines register with | | `webhook.enabled` | `true` | Two pod-mutating webhooks: **CacheBlend** injection (`lmcache.ai/cacheblend-inject`) and **LMCache connection** injection (`lmcache.ai/lmcache-inject`). **Requires cert-manager**; rendering fails fast without it. Opt in your vLLM pods with the label, see [Modify an Existing vLLM Deployment](/v1.0.0/installation/existing-deployment) | | `cacheBlend.enabled` | `false` | A `CacheBlendEngine` CR, see [CacheBlend](/v1.0.0/configuration/cacheblend). Requires `webhook.enabled=true` | | `observability.enabled` | `false` | An OpenTelemetry Collector CR. Requires the OpenTelemetry Operator already installed | | `openshift.enabled` | `false` | The privileged SCC binding the engine's `hostIPC` needs | | `tests.enabled` | `true` | The `helm test` smoke-test pods | ### See every value The table above is the common subset. For the full list with defaults: ```bash theme={null} helm show values \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform --version 1.0.0 ``` See the [Configuration reference](/v1.0.0/reference/configuration) for per-section detail and example overlays. ## Next steps Install, deploy vLLM against the engine, see the cache hit, run the benchmark. Patch the workload you already run to consume the engine connection ConfigMap. Engine `Pending`, `ImagePullBackOff`, hung uninstalls, and more. # Troubleshooting Source: https://docs.tensormesh.ai/v1.0.0/installation/troubleshooting Diagnose common install and day-2 issues with Tensormesh Platform. This page covers the most common install and runtime issues. For everything else, gather the diagnostic bundle described at the bottom and open a support thread. ## Contents **Install & scheduling** * [Engine pod is `Pending`](#engine-pod-is-pending) — node-selector / GPU label mismatch * [Pod stuck in `ContainerCreating`](#pod-stuck-in-containercreating-cdi) — CDI disabled (older NVIDIA GPU Operator) * [Engine pod `CreateContainerError` on OpenShift](#engine-pod-createcontainererror-on-openshift) — missing privileged SCC * [Operator pod `ImagePullBackOff`](#operator-pod-imagepullbackoff) — registry auth / air-gap * [Install fails: cert-manager not installed](#install-fails-cert-manager-not-installed) — the mutating webhook needs cert-manager * [Install fails with `CRD ... exists and cannot be imported`](#install-fails-with-crd-exists-and-cannot-be-imported) — ownership conflict from a prior install **Uninstall & cleanup** * [`helm uninstall` hangs](#helm-uninstall-hangs) — stuck CR finalizer * [CRD survives uninstall (by design)](#crd-survives-uninstall-by-design) * [Cleaning up an inconsistent or orphaned install](#cleaning-up-an-inconsistent-or-orphaned-install) — no release record / orphaned resources **Metrics & observability** * [Metrics endpoint returns `401`](#metrics-endpoint-returns-401) * OTel Collector / traces not flowing → see [Observability → Troubleshooting](/v1.0.0/observability#troubleshooting) **Getting help** * [Gathering a diagnostic bundle](#gathering-a-diagnostic-bundle) ## Engine pod is `Pending` ```bash theme={null} kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine # NAME READY STATUS RESTARTS AGE # tmo-engine-xxxxx 0/1 Pending 0 2m ``` Almost always a node-selector mismatch. The engine DaemonSet selects nodes labeled `nvidia.com/gpu.present=true` by default — if no node carries that label, no pod is scheduled. ```bash theme={null} # What labels do my nodes carry? kubectl get nodes -L nvidia.com/gpu.present # Add the label to a GPU node manually kubectl label node nvidia.com/gpu.present=true ``` If you use a different label in your environment, override the selector instead: ```yaml my-values.yaml theme={null} engine: spec: nodeSelector: mycompany.io/gpu-tier: "premium" ``` If you use filesystem offloading with a PVC, a `Pending` engine can instead mean the claim isn't `Bound` — see [External Storage Offloading](/v1.0.0/configuration/filesystem-offloading#engine-pods-stay-pending). ## Pod stuck in `ContainerCreating` (CDI) ```bash theme={null} kubectl get pods -n tensormesh-operator -o wide # NAME READY STATUS RESTARTS AGE # tmo-engine-xxxxx 0/1 ContainerCreating 0 5m ``` The pod **scheduled onto a GPU node** (so it's past `Pending`) but the container can't be created. The usual cause on NVIDIA GPU clusters is that the **Container Device Interface (CDI)** is disabled. The NVIDIA GPU Operator injects GPUs via CDI, and **before GPU Operator v25.10.0 it defaulted to `cdi.enabled: false`** — with no obvious error, the pod just hangs in `ContainerCreating`. Enable CDI in the GPU `ClusterPolicy`: ```bash theme={null} oc patch clusterpolicy gpu-cluster-policy --type=merge -p '{ "spec": { "cdi": { "enabled": true, "default": true } } }' ``` GPU Operator **v25.10.0+** enables CDI by default — no patch needed. Confirm it's CDI and not something else: `kubectl describe pod ` — a CDI problem shows a `Failed to create … CDI device injection` / `unresolvable CDI devices` event. A missing-image event (`ImagePullBackOff`) or volume-mount error points elsewhere. ## Engine pod `CreateContainerError` on OpenShift ```bash theme={null} kubectl describe pod -n tensormesh-operator # Events: # Warning Failed ... unable to start container: hostIPC is not allowed ``` The pod is requesting `hostIPC` but its ServiceAccount is not bound to an SCC that permits it. Add `openshift.enabled: true` to your values file and upgrade: ```bash theme={null} helm upgrade tensormesh-operator \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator \ --version -n tensormesh-operator \ -f values-openshift.yaml ``` ## Operator pod `ImagePullBackOff` ```bash theme={null} kubectl describe pod -n tensormesh-operator # Events: # Warning Failed ... Failed to pull image: unauthorized ``` The image lives in a registry your cluster cannot authenticate to. Create an image pull secret and reference it from the chart: ```bash theme={null} kubectl create secret docker-registry tmo-pull \ -n tensormesh-operator \ --docker-server= \ --docker-username= \ --docker-password= ``` ```yaml my-values.yaml theme={null} operator: image: pullSecrets: - tmo-pull ``` For air-gapped clusters, mirror the operator image into your internal registry and override `operator.image.repository` and `operator.image.tag`. ## Install fails: cert-manager not installed ```bash theme={null} helm install tensormesh-operator oci://artifacts.tensormesh.ai/.../tensormesh-operator ... # Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: # resource mapping not found for name: "..." ... ensure CRDs are installed first: # no matches for kind "Issuer" in version "cert-manager.io/v1" ``` The mutating webhook is enabled by default (`webhook.enabled=true`), and the chart provisions its serving certificate with cert-manager — so it creates `Issuer` and `Certificate` resources (`cert-manager.io/v1`). Without cert-manager's CRDs on the cluster, those kinds don't resolve and the install fails immediately. Install cert-manager first (see [Install with Helm → Prerequisites](/v1.0.0/installation/helm#prerequisites)), then re-run: ```bash theme={null} kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml kubectl -n cert-manager wait --for=condition=Available deploy --all --timeout=180s ``` If you don't need the CacheBlend / LMCache injection webhooks, disable them instead — the operator then runs with `ENABLE_WEBHOOKS=false` and needs no cert-manager: ```yaml my-values.yaml theme={null} webhook: enabled: false ``` The webhook's `failurePolicy` is `Ignore`, so once installed, a webhook that is temporarily down (for example while its certificate is still being issued) does not block pod admission — injection is simply skipped until it recovers. ## Install fails with `CRD ... exists and cannot be imported` ```bash theme={null} helm install tensormesh-operator oci://artifacts.tensormesh.ai/.../tensormesh-operator ... # Error: INSTALLATION FAILED: unable to continue with install: # CustomResourceDefinition "lmcacheengines.lmcache.lmcache.ai" in namespace "" exists # and cannot be imported into the current release: invalid ownership metadata; # annotation validation error: key "meta.helm.sh/release-name" must equal "tensormesh-operator": # current value is "" ``` The `LMCacheEngine` CRD is cluster-scoped — there is only one of it. A previous install under a different release name (or namespace) annotated it as its own; Helm refuses to silently transfer ownership to your new release. This is a Helm-wide safety check, not specific to this chart — cert-manager, kube-prometheus-stack, and Argo CD all behave identically. Before doing either fix below, check whether the prior release is still active: `helm list -A`. If you see a real release that owns the CRD, leave it alone — running a second operator against the same CRD will fight reconciliation. Use the existing release (`helm upgrade ...`) instead. ### Option A — adopt the CRD into your new release (recommended) Use this when the prior release is gone (was uninstalled, or is on a cluster you don't own anymore) but the CRD survived because of `helm.sh/resource-policy: keep`. ```bash theme={null} kubectl annotate crd lmcacheengines.lmcache.lmcache.ai \ meta.helm.sh/release-name= \ meta.helm.sh/release-namespace= --overwrite kubectl label crd lmcacheengines.lmcache.lmcache.ai \ app.kubernetes.io/managed-by=Helm --overwrite # Re-run the install helm install oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator \ --version \ --namespace --create-namespace ``` Existing `LMCacheEngine` CRs survive — only the CRD's ownership annotations change. The `LMCacheEngine` CRD is cluster-scoped — one of it exists for the whole cluster. Helm tracks which release owns each resource via three stamps: | Field | Purpose | | - | - | | annotation `meta.helm.sh/release-name` | which release owns this | | annotation `meta.helm.sh/release-namespace` | in which namespace | | label `app.kubernetes.io/managed-by=Helm` | "some Helm release manages this" | The adoption commands rewrite those stamps so your new release becomes the recognized owner. The CRD's schema, any existing `LMCacheEngine` CRs, and everything in any namespace are unchanged — it's a pure metadata edit, fully reversible by running the same commands with the old values. ### Option B — delete the CRD and let the install recreate it Safe **only** when no `LMCacheEngine` CRs exist anywhere on the cluster. Confirm first: ```bash theme={null} kubectl get lmcacheengines -A # must return "No resources found" kubectl delete crd lmcacheengines.lmcache.lmcache.ai # Re-run the install — it will create the CRD fresh with the right ownership helm install oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator \ --version \ --namespace --create-namespace ``` Deleting the CRD cascade-deletes every `LMCacheEngine` CR on the cluster. Use Option A if you have any CRs you want to keep. ## `helm uninstall` hangs The pre-delete hook is waiting for the `LMCacheEngine` CR's finalizer to drain. Usually this means the operator was already gone when the CR was deleted, so nobody is running the finalizer. ```bash theme={null} # 1. Confirm the CR is stuck kubectl get lmcacheengines -A # 2. Inspect operator logs (if it's still around) kubectl logs -n tensormesh-operator \ deploy/tensormesh-operator # 3. Force-clear the finalizer kubectl patch lmcacheengine -n \ -p '{"metadata":{"finalizers":[]}}' --type=merge # 4. Re-run uninstall helm uninstall tensormesh-operator -n tensormesh-operator ``` Removing finalizers bypasses any cleanup the operator would have performed on backing resources. Use only when the operator is definitively gone. If `helm list -A` shows **no release at all** (the operator *and* the release record are already gone, not just a stuck uninstall), this quick fix doesn't apply — follow [Cleaning up an inconsistent or orphaned install](#cleaning-up-an-inconsistent-or-orphaned-install) for the full manual teardown. ## Metrics endpoint returns `401` The controller-runtime auth filter is on by default. Either disable it for scrapers that cannot authenticate: ```yaml my-values.yaml theme={null} operator: metrics: secure: false ``` Or grant the scraping ServiceAccount the `system:auth-delegator` ClusterRole so it can authenticate with the kube-apiserver TokenReview API: ```bash theme={null} kubectl create clusterrolebinding metrics-auth \ --clusterrole=system:auth-delegator \ --serviceaccount=: ``` ## CRD survives uninstall (by design) The `LMCacheEngine` CRD is annotated `helm.sh/resource-policy: keep`, so `helm uninstall` leaves it (and any remaining CRs) in place. This is the same convention used by cert-manager and kube-prometheus-stack — it prevents data loss when you upgrade in place. To remove the CRD and cascade-delete any remaining CRs: ```bash theme={null} kubectl delete crd lmcacheengines.lmcache.lmcache.ai ``` ## Cleaning up an inconsistent or orphaned install Use this when the normal `helm uninstall` path doesn't apply — for example: * `helm list -A` shows **no release**, but the engine pod, `LMCacheEngine` CR, services, and CRD are still present (the release record was removed but its resources weren't). * `helm uninstall` **hangs**, or `kubectl delete namespace` gets stuck in `Terminating`. The root cause is almost always the same: the `LMCacheEngine` CR carries a finalizer (`lmcache.ai/cleanup`) that **only the operator can drain**. If the operator is gone — or was removed before the CR — nothing runs the finalizer, so the CR can't be deleted, and anything waiting on it (a namespace delete, an uninstall hook) blocks indefinitely. Clean up manually, **in this order**: The operator isn't coming back to drain it, so remove the finalizer yourself: ```bash theme={null} kubectl patch lmcacheengine -n \ -p '{"metadata":{"finalizers":[]}}' --type=merge ``` (Find the name with `kubectl get lmcacheengine -A`.) This is safe **only** when the operator is definitively gone — it skips the cleanup the operator would have done, which is fine when you're tearing everything down anyway. With the finalizer cleared, this completes instead of hanging — it sweeps the DaemonSet, pods, services, ConfigMaps, ServiceAccounts, and RoleBindings. ```bash theme={null} kubectl delete namespace ``` Cluster-scoped, so the namespace delete didn't touch it. Removing it gives a clean slate (and avoids the ownership conflict on the next install). ```bash theme={null} kubectl delete crd lmcacheengines.lmcache.lmcache.ai ``` The operator's `ClusterRole`/`ClusterRoleBinding` are cluster-scoped and survive a namespace delete. Remove any leftovers: ```bash theme={null} kubectl get clusterrole,clusterrolebinding | grep -i tensormesh # delete anything it lists, e.g.: kubectl delete clusterrole,clusterrolebinding -l app.kubernetes.io/name=tensormesh-operator ``` **Order matters.** Clear the finalizer *before* deleting the namespace. If you delete the namespace first, it gets stuck in `Terminating` (blocked on the un-drainable CR) until you clear the finalizer anyway. After these four steps, `kubectl get ns ` and `kubectl get crd lmcacheengines.lmcache.lmcache.ai` should both return `NotFound` — a true clean slate. ## Gathering a diagnostic bundle When opening a support thread, include: ```bash theme={null} # Versions helm version kubectl version kubectl get nodes -o wide # Release state helm list -A helm get values tensormesh-operator -n tensormesh-operator helm get manifest tensormesh-operator -n tensormesh-operator # Live state kubectl get all -n tensormesh-operator kubectl get lmcacheengines -A -o yaml # Logs kubectl logs -n tensormesh-operator \ deploy/tensormesh-operator --tail=500 kubectl logs -n tensormesh-operator \ -l app.kubernetes.io/component=cache-engine --tail=500 # Events kubectl get events -n tensormesh-operator --sort-by=.lastTimestamp ``` Rare hashing fallback: if LMCache logs that it is using **builtin hash** instead of the default `blake3` path, set `PYTHONHASHSEED=0` in the vLLM pod and restart it. This is not the normal MP requirement and it is unrelated to MQ payload-shape mismatches. ## Next steps Re-check the install steps against the chart's documented values. Re-validate prerequisites if multiple issues are stacking up. # NVIDIA Dynamo Source: https://docs.tensormesh.ai/v1.0.0/integrations/dynamo Offload KV cache from Dynamo vLLM workers to the Tensormesh engine over cross-pod CUDA IPC. ## Support matrix | Dynamo version | TMO chart version | LMCache image | Status | | - | - | - | - | | `1.4.0` | `1.0.0` | `lmcache/vllm-openai:v0.5.2` | ✅ Validated | **Non-prefix caching ([CacheBlend](/v1.0.0/configuration/cacheblend)) is not supported** through this integration — only prefix-based KV reuse. ## Prerequisites * A Kubernetes cluster with at least one GPU node * Helm 3.8+ and `kubectl` * **Access token from the Tensormesh team** for the chart registry * A Hugging Face token for model download * [cert-manager](https://cert-manager.io/docs/installation/) — the chart enables its webhook by default and rendering requires the cert-manager APIs ## Step 1 — install Dynamo ```bash theme={null} export NAMESPACE=dynamo-system export DYNAMO_VERSION=1.4.0 helm upgrade --install dynamo-platform \ "https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${DYNAMO_VERSION}.tgz" \ --namespace "$NAMESPACE" \ --create-namespace \ --wait \ --timeout=10m kubectl get pods --namespace "$NAMESPACE" ``` ## Step 2 — create the model access secret Both the frontend and the worker reference `hf-token-secret`; the pods fail to start without it. ```bash theme={null} kubectl create secret generic hf-token-secret \ --namespace "$NAMESPACE" \ --from-literal=HF_TOKEN= \ --dry-run=client -o yaml | kubectl apply -f - ``` ## Step 3 — install the Tensormesh Platform Write the values file — the engine tag is pinned per the [support matrix](#support-matrix): ```yaml my-values.yaml theme={null} engine: enabled: true namespace: dynamo-system spec: l1: sizeGB: 20 # small for the integration test; bump for production image: repository: lmcache/vllm-openai tag: v0.5.2 # vllm-runtime:1.4.0 ships lmcache 0.5.2 pullPolicy: IfNotPresent isolatedIPC: false # operator >= v0.5.5 defaults to isolated IPC; the v0.5.2 engine and the hostIPC worker need the legacy wiring coordinator: enabled: false ``` ```bash theme={null} # One-time: log in with the token Tensormesh gave you. echo '' | helm registry login artifacts.tensormesh.ai -u tensormesh --password-stdin helm install tensormesh-platform \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-platform \ --version 1.0.0 \ --namespace dynamo-system \ --create-namespace \ -f my-values.yaml \ --wait ``` Wait for the engine to reconcile, and note the ConfigMap it publishes — the worker mounts it in the next step: ```bash theme={null} kubectl -n dynamo-system get lmcacheengine # NAME PHASE READY DESIRED # tensormesh-platform-default-engine Running 1 1 kubectl -n dynamo-system get cm tensormesh-platform-default-engine-connection ``` ## Step 4 — deploy vLLM and the frontend in Dynamo Save as `agg.yaml`. Compared to Dynamo's stock aggregated example, the **worker** carries the LMCache wiring; the frontend is unchanged. ```yaml agg.yaml theme={null} apiVersion: nvidia.com/v1beta1 kind: DynamoGraphDeployment metadata: name: vllm-agg spec: components: - name: Frontend podTemplate: spec: containers: - envFrom: - secretRef: name: hf-token-secret image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0 name: main replicas: 1 type: frontend - name: VllmDecodeWorker podTemplate: spec: # Required for CUDA IPC between vLLM and the LMCache server hostIPC: true containers: - args: - --model - Qwen/Qwen3-0.6B - --no-enable-prefix-caching command: - /bin/sh - -c - exec python3 -m dynamo.vllm "$@" --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config.json)" - -- env: - name: PYTHONHASHSEED # deterministic prefix hashing across processes value: "0" envFrom: - secretRef: name: hf-token-secret image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0 name: main resources: limits: nvidia.com/gpu: "1" requests: # Increase this value for larger models. ephemeral-storage: 2Gi # Root uid required for CUDA IPC between vLLM and the LMCache server: # the shared-CUDA-tensor handshake fails under a non-root uid. securityContext: runAsUser: 0 volumeMounts: - mountPath: /etc/lmcache name: kv-transfer-config readOnly: true workingDir: /workspace/examples/backends/vllm volumes: - configMap: name: tensormesh-platform-default-engine-connection name: kv-transfer-config replicas: 1 sharedMemorySize: "0" type: worker ``` ```bash theme={null} kubectl -n dynamo-system apply -f agg.yaml ``` ### Why each change exists | Change | Why | | - | - | | `hostIPC: true` | The MP connector moves KV tensors over CUDA IPC through the **host's** `/dev/shm`, shared with the node-local engine pod. | | `sharedMemorySize: "0"` | Disables the `/dev/shm` tmpfs the Dynamo controller adds by default — an emptyDir at `/dev/shm` would shadow the host's and break `cudaIpcOpenMemHandle`. | | `securityContext: runAsUser: 0` | The shared-CUDA-tensor handshake fails under a non-root uid. | | `PYTHONHASHSEED=0` | LMCache requires deterministic token hashing across processes for consistent cache keys. | | the `command` wrapper | Reads the connector JSON from the mounted ConfigMap at launch, so the config always matches what the operator generated; Dynamo's `args` still flow through via `"$@"`. | | the ConfigMap volume | `-connection`, published by the operator next to the engine. Same-namespace only — this is why everything lives in `dynamo-system`. | | `--no-enable-prefix-caching` | vLLM's own prefix cache would serve repeats before LMCache is exercised. Keep it while verifying; drop it if you want both cache layers in production. | ## Step 5 — send a request ```bash theme={null} kubectl port-forward \ --namespace "$NAMESPACE" \ "service/vllm-agg-frontend" 8000:8000 \ >/tmp/dynamo-port-forward.log 2>&1 & until curl --silent --fail http://localhost:8000/health >/dev/null; do sleep 2 done curl --silent --show-error http://localhost:8000/v1/chat/completions \ --header "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen3-0.6B", "messages": [ {"role": "user", "content": "What is NVIDIA Dynamo?"} ], "max_tokens": 200 }' ``` ## Verify the cache is working Send the **same long prompt twice** (it must exceed the engine's `chunk_size` — 256 tokens by default — or nothing is stored). The response's `usage` block reports the reuse from the vLLM side: ```text theme={null} call 1: "cached_tokens": 0 call 2: "cached_tokens": 256 ``` And the engine pod on the worker's node logs both sides of the transaction: ```bash theme={null} kubectl -n dynamo-system logs -l app.kubernetes.io/instance=tensormesh-platform-default-engine \ --tail=100 | grep -E "Stored|Retrieved" # Stored 256 tokens in 0.011 seconds # Retrieved 256 tokens in 0.003 seconds ``` A `Creating v1 connector with name: LMCacheMPConnector` line in the worker's log and a `Registered KV cache for GPU ID ... with N layers` line in the engine's log confirm the CUDA IPC handshake completed at startup. # Install on SUSE Rancher / RKE2 Source: https://docs.tensormesh.ai/v1.0.0/integrations/suse Run Tensormesh Platform on SUSE Rancher + RKE2 (SLES) for SUSE AI — supported versions and platform notes. Tensormesh Platform runs on SUSE — SUSE Rancher managing RKE2 on SUSE Linux Enterprise Server — the same way it does on vanilla Kubernetes. Unlike OpenShift, RKE2 permits the `hostIPC` the cache engine needs out of the box, so **no security overlay is required**. Read this **after** [Install with Helm](/v1.0.0/installation/helm). This page adds the SUSE-specific platform notes; it does not replace the Helm install steps. ## Supported SUSE platform Tensormesh supports **SUSE AI** on SUSE Rancher–managed RKE2. Validated on: | Component | Supported version | | - | - | | SUSE Linux Enterprise Server | 15 SP6 | | SUSE Rancher | 2.14.3 | | RKE2 | `v1.35.6+rke2r1` | | Storage | Longhorn (SUSE Storage) — ReadWriteMany | | GPU | NVIDIA (validated on L4) | Tensormesh tracks the latest supported RKE2 and SUSE Rancher versions per the [SUSE Rancher support matrix](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/latest/). The operator, cache engine, and Helm chart are supported as a matched set — deploy a single row from the [compatibility matrix](/v1.0.0/installation/compatibility) together. ## Why SUSE needs no extra config Engine pods request `hostIPC` so the LMCache cache server can share memory with the vLLM worker processes outside the pod. RKE2's default [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/) profile is `privileged`, which permits `hostIPC` — so the engine pod starts with no OpenShift-style SecurityContextConstraint and no `openshift.enabled` flag. You only need to act if you have **hardened the target namespace to the `baseline` or `restricted` PSA level**. In that case either label the namespace `pod-security.kubernetes.io/enforce=privileged`, or run the cache in colocated (sidecar) mode, which needs no `hostIPC`. ## Prerequisites * An **RKE2 cluster managed by SUSE Rancher** (the supported versions above). * **GPU-enabled node(s).** Set up the NVIDIA GPU stack on the cluster first — the driver, the containerd runtime integration, and the device plugin — following NVIDIA's guidance for RKE2 (for example, the NVIDIA GPU Operator). This is a platform prerequisite, independent of Tensormesh; on SLES, match the driver to the running kernel. The cache engine schedules only on nodes labeled `nvidia.com/gpu.present=true`. * A **ReadWriteMany StorageClass** for the shared model hub — **Longhorn (SUSE Storage)** works out of the box (install `open-iscsi` on every node first). ## Install Install with Helm exactly as on vanilla Kubernetes — pin **explicit image tags** (never `latest`): ```bash theme={null} helm install tensormesh-operator \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator \ --version \ --namespace tensormesh-operator --create-namespace \ --wait ``` The chart is distributed through a private registry. Request an access token from the Tensormesh team, then authenticate before installing — see [Install with Helm](/v1.0.0/installation/helm#quick-install). ## Verify ```bash theme={null} # Nodes are SUSE Linux Enterprise + RKE2 kubectl get nodes -o custom-columns=\ 'NODE:.metadata.name,OS:.status.nodeInfo.osImage,KUBERNETES:.status.nodeInfo.kubeletVersion' # The cache engine pod is Running (hostIPC is allowed by default on RKE2) kubectl get pods -n tensormesh-operator \ -l app.kubernetes.io/component=cache-engine ``` ## Commercial support Tensormesh provides commercial support for Tensormesh Platform on the SUSE Rancher and RKE2 versions listed above. Contact the Tensormesh team for support terms. ## Next steps Full install, upgrade, and uninstall flow. The matched operator / cache engine / chart version set. Engine scheduling, GPU, and permission diagnostics. # Welcome to Tensormesh Platform Source: https://docs.tensormesh.ai/v1.0.0/introduction Self-hosted KV caching for your vLLM fleet on Kubernetes. **Tensormesh Platform** runs a KV-cache engine on your own Kubernetes cluster to cut time-to-first-token (TTFT) and free GPU memory. It installs as a Kubernetes operator: a single controller manages a fleet of cache engines next to your vLLM pods, so repeated and long-context prompts are served from cache instead of recomputed on the GPU — and that cache survives across requests, pods, and optionally nodes. Preflight your cluster and pick an install method. Install → run inference → benchmark the cache win. ## Benefits Cached prefixes skip recomputation, so warm requests return their first token sooner. KV cache lives in CPU memory and on disk, freeing HBM for active batches. Cache outlives a single request — across pods, restarts, and optionally across nodes. One operator, standard CRs, Helm/GitOps install, and metrics through Prometheus. ## How caching tiers fit together Each tier is larger and cheaper than the one above it; the engine promotes hot data upward and spills cold data downward: | Tier | Where it lives | Configured by | Best for | | - | - | - | - | | **GPU HBM (L0)** | vLLM's own paged KV cache | vLLM | The active batch | | **CPU offloading (L1)** | Host DRAM (system RAM) | [`engine.spec.l1.sizeGB`](/v1.0.0/configuration/cpu-offloading) | Fast warm reuse beyond what fits on the GPU | | **External storage (L2)** | Local disk, NFS, or a remote store | [`engine.spec.l2Backend`](/v1.0.0/configuration/filesystem-offloading) | Large working sets and reuse across restarts/nodes | Start with **[CPU offloading](/v1.0.0/configuration/cpu-offloading)**; it's on by default. Add **[external storage](/v1.0.0/configuration/filesystem-offloading)** when your working set outgrows RAM or must survive restarts. ## What gets deployed A controller-manager Deployment that watches the Tensormesh CRs cluster-wide and reconciles them into running cache infrastructure. A DaemonSet (one cache-engine pod per GPU node) that stores and serves KV cache to the vLLM pods on that node. A fleet-wide service the engines register with, enabling cross-node cache awareness. An optional OpenTelemetry stack that ships cache metrics and traces to Prometheus and Tempo. ## Explore the docs The primary install path, plus upgrades and rollbacks. The default host-DRAM cache tier. The signals that show whether the cache is reducing recompute. A console for monitoring your cache fleet, thresholds, and alerts. # Multi-Tenancy Source: https://docs.tensormesh.ai/v1.0.0/management/multi-tenancy Isolate cache reuse and capacity across tenants, users, and environments with cache_salt and per-tenant quotas. When several tenants, users, or environments share one LMCache deployment, you usually want two kinds of isolation: * **KVCache isolation** — control *who is allowed to share cached prefixes*. A prompt cached for one tenant must not be reused by another. This is what `cache_salt` provides. * **Capacity isolation** — control *how much cache each tenant may consume*, so one noisy tenant cannot evict everyone else's entries. This is what per-tenant **quotas** and the **IsolatedLRU** eviction policy provide. This page covers both, and which layer owns each setting. ## Who configures what Multi-tenancy spans three layers. Putting a setting in the wrong layer is the most common mistake. | Concern | Owner | Where it is set | | - | - | - | | Reuse boundary per request (`cache_salt`) | **Inference API caller** (vLLM / your app) | The request body sent to inference engine | | Engine deployment, cache size, eviction config | **Operator (TMO)** | `LMCacheEngine` CRD / Helm values | | Per-tenant storage budgets (quotas) | **Runtime admin** | LMCache HTTP cache-admin API on the engine (or coordinator) | | Fleet-wide per-tenant usage tracking | **Operator (TMO)** | `LMCacheCoordinator` | ## KVCache isolation with `cache_salt` `cache_salt` is a **per-request isolation key**. It tells TMO which requests are allowed to share cached KV and which ones must stay separate. The behavior is simple: * same model + same `cache_salt` → kv cache access is allowed * same model + different `cache_salt` → kv cache access is blocked ### How to set it `cache_salt` is a first-class field on OpenAI-compatible requests (vLLM). Include it directly in the request body. ```bash theme={null} curl -s http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen/Qwen3-0.6B", "messages": [ {"role": "user", "content": "Summarize the quarterly report."} ], "cache_salt": "tenant-acme", "max_tokens": 128, "temperature": 0 }' ``` ### How to choose a value Pick a value that is: * stable across requests that should share cache * different across requests that must stay isolated * low-cardinality enough that it does not explode your metrics labels Good choices: `tenant-acme`, `customer-123`, `prod`, `workspace-42`. Bad choices: a random UUID per request, a timestamp, or any value that changes on every call. If the value changes on every request, you destroy reuse and effectively turn every request into a cache miss. ### Constraints `cache_salt` is validated by TMO. It: * must not contain `@` * must not contain `/` * must not contain `\` * must not contain a NUL byte * must be at most **128** characters A salt that violates these is rejected with a `ValueError`. Keep it simple: short ASCII identifiers such as `tenant-acme` or `prod` are the safest choice. The empty string (`""`) is the default, unsalted domain. In the cache-admin HTTP API described below, the empty salt cannot appear in a URL path, so it is represented by the reserved sentinel `_default` (for example `GET /quota/_default`). ## Capacity isolation: per-tenant quotas KVCache isolation keeps tenants from *reading* each other's data. It does not stop one tenant from *filling* the cache capacity. For that, TMO supports per-tenant quotas backed by the **IsolatedLRU** eviction policy. How it works: * **IsolatedLRU** keeps a separate LRU ordering per `cache_salt` instead of one global list, so eviction victims for one tenant are chosen only from that tenant's entries. * A **quota** assigns each `cache_salt` a storage budget in bytes. LMCache tracks usage per salt and evicts a salt's oldest entries once it crosses its budget. * Quotas use **allowlist semantics**: a `cache_salt` with no registered quota has an effective limit of **0 bytes**. Its stores are accepted on the hot path but evicted on the next eviction cycle (\~1s). Only salts with an explicit quota retain cached data. Plan to register a quota for every tenant you want cached. The eviction loop is governed by these engine-side fields (LMCache eviction config): | Field | Default | Meaning | | - | - | - | | `eviction_policy` | `LRU` | `LRU`, `IsolatedLRU`, or `noop` | | `trigger_watermark` | `0.8` | Per-salt usage fraction (of quota) that triggers eviction | | `eviction_ratio` | `0.2` | Fraction of over-quota bytes evicted per cycle | Through the operator, set these under the CRD `eviction` block: `eviction.policy`, `eviction.triggerWatermark`, and `eviction.evictionRatio`. The CRD `eviction.policy` currently accepts only `LRU` and `noop` — per-salt `IsolatedLRU` eviction is applied automatically by the **coordinator** for fleet-wide quotas (see below), not set per-engine through the CRD. ### Managing quotas at runtime The engine exposes an HTTP cache-admin frontend on `spec.httpPort` (default **8080**). Quotas are created, inspected, and removed there: | Method | Path | Body | Purpose | | - | - | - | - | | `PUT` | `/quota/{cache_salt}` | `{"limit_gb": }` | Create or update a tenant's budget | | `GET` | `/quota/{cache_salt}` | — | Read a tenant's limit and current usage | | `DELETE` | `/quota/{cache_salt}` | — | Remove a budget (entries evicted next cycle) | | `GET` | `/quota` | — | List every registered quota and its usage | ```bash theme={null} # Give tenant-acme a 5 GB budget (engine HTTP frontend on httpPort 8080) curl -s -X PUT http://:8080/quota/tenant-acme \ -H 'Content-Type: application/json' \ -d '{"limit_gb": 5}' # Inspect it curl -s http://:8080/quota/tenant-acme ``` Use the `_default` sentinel to manage the unsalted domain: `PUT /quota/_default`. ### Fleet-wide quotas via the coordinator The HTTP API above is per-engine. To track and bound a tenant's usage across an entire fleet of engines, register the engines with an `LMCacheCoordinator` and enable event reporting on the engine's coordinator block: ```yaml theme={null} apiVersion: lmcache.lmcache.ai/v1alpha1 kind: LMCacheEngine spec: coordinator: ref: name: my-coordinator # an LMCacheCoordinator in the same namespace l2EventReporting: true # report L2 store/lookup events for fleet-wide tracking l2EventFlushInterval: 1 # seconds between event flushes ``` With reporting on, each engine forwards per-salt usage events to the coordinator, which aggregates them and runs fleet-wide per-salt quota enforcement and eviction. The coordinator side exposes the same `/quota/*` admin endpoints on its own address — `PUT`/`GET`/`DELETE /quota/{cache_salt}`, `GET /quota`, plus fleet defaults via `PUT /quota/config` — and its own sweep controls (`evictionCheckInterval`, default 5s; `evictionRatio`, default 0.2). ## Deployment-level isolation `cache_salt` and quotas isolate tenants *within* one shared cache. For harder isolation, give each tenant its own engine: * **Per-namespace engines** — run a separate `LMCacheEngine` CR per tenant namespace. The chart's multi-tenant mode (`operator.enabled=true`, `engine.enabled=false`) installs only the operator; each tenant then ships their own engine CR. Caches are not shared across separate engines at all. * **Node placement** — `spec.nodeSelector`, `spec.affinity`, and `spec.tolerations` pin a tenant's engine to dedicated GPU nodes. * **RBAC / network** — `spec.serviceAccountName`, `spec.podLabels`, and `spec.podAnnotations` let you attach per-tenant RBAC and NetworkPolicy. Choose deployment-level isolation when tenants must not share GPUs or memory at all; choose `cache_salt` + quotas when you want the efficiency of one shared pool with logical separation. ## Observability LMCache tags several metrics with `cache_salt`, which is useful for per-tenant hit-rate analysis: * `lmcache_mp_lookup_requested` — tokens submitted for lookup (labels: `model_name`, `cache_salt`) * `lmcache_mp_lookup_hit` — tokens served from cache (labels: `model_name`, `cache_salt`) * `lmcache_mp_real_reuse_gap` — seconds between a chunk's last write and next read (label: `cache_salt`) * `lmcache_mp_real_reuse_gap_objects` — the same gap measured in chunks (label: `cache_salt`) Divide `lookup_hit` by `lookup_requested`, grouped by `cache_salt`, for a per-tenant hit rate. Treat `cache_salt` as a potentially high-cardinality dimension. Per-tenant or per-environment salts are fine; per-request salts will blow up your metrics cardinality (and destroy reuse). ## Verification `cache_salt` reuse behavior is straightforward to check by hand: 1. Send a long prompt with `cache_salt: "tenant-a"`. 2. Send the **same** prompt again with `cache_salt: "tenant-a"` → reuse is allowed. 3. Send the same prompt with `cache_salt: "tenant-b"` → it must **not** reuse `tenant-a`'s entries, even though the prompt is identical. To confirm a hit at the engine level, inspect the engine pod logs — the engine logs KV cache registration and serving activity that the end-to-end test harness keys on to assert cache behavior. Per-tenant hit rates are also visible through the metrics above. The current end-to-end test suite exercises basic cache hit/miss and L1/L2 storage, but does not yet include automated coverage for `cache_salt` isolation, per-tenant quotas, or `IsolatedLRU`. Validate these flows in a staging deployment before relying on them in production. ## Recommended practice * Use **no `cache_salt`** if all traffic is inside one trust boundary. * Use **one stable salt per tenant or environment** when traffic must be isolated. * Avoid **per-request salts** unless you explicitly want to disable sharing. * If you need capacity fairness, enable **fleet-wide quotas via the coordinator** (which applies per-salt IsolatedLRU eviction) and register a **quota for every tenant** you want cached (remember: unregistered salts get 0 bytes). * For strict isolation, give each tenant its own **engine per namespace** instead of a shared pool. # Observability Source: https://docs.tensormesh.ai/v1.0.0/observability Export LMCache engine metrics and traces via an OpenTelemetry Collector — to in-cluster Prometheus/Tempo or an external backend. The chart can deploy an **OpenTelemetry observability stack** that collects metrics and traces from the LMCache engine and ships them to the backends you choose: * An **OTel Collector** receives OTLP from the engine pods. * **Metrics** are exposed for Prometheus to scrape (`ServiceMonitor`) and/or pushed to an external Prometheus-compatible endpoint (`remoteWrite`). * **Traces** are sent to an in-cluster **Tempo**, an **external** backend (e.g. Grafana Cloud), or both. It's **off by default** — opt in with `observability.enabled=true`. The chart deploys the Collector and (optionally) Tempo. It does **not** ship Grafana or any dashboards — bring your own Grafana and point it at Prometheus (metrics) and Tempo (traces). Enabling observability also stops the engine serving `/metrics` directly. Its metrics move to the OTLP path, so Prometheus must scrape the Collector instead. ## Prerequisites The chart creates custom resources but does **not** install the operators that reconcile them. Install these first: | Operator | Provides | Needed for | | - | - | - | | **OpenTelemetry Operator** | `OpenTelemetryCollector` CRD | the Collector (always, when observability is enabled) | | **Prometheus Operator** | `ServiceMonitor` CRD | in-cluster metric scraping (`prometheus.serviceMonitor.enabled`) | | **Tempo Operator** | `TempoMonolithic` CRD | an in-cluster trace backend (`traces.tempoCR.enabled`) | ```bash theme={null} kubectl get crd | grep -E 'opentelemetrycollectors|servicemonitors.monitoring|tempomonolithics' ``` ## Quickstart — metrics only The smallest useful config: a Collector + a `ServiceMonitor`, no trace backend. Put it in a values file, and pass it alongside your existing values file so `helm upgrade` doesn't drop your current settings: ```yaml observability-values.yaml theme={null} observability: enabled: true prometheus: serviceMonitor: enabled: true additionalLabels: release: # must match your Prometheus' serviceMonitorSelector ``` ```bash theme={null} helm upgrade --install tensormesh-operator \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator \ --version -n tensormesh-operator --create-namespace \ -f my-values.yaml -f observability-values.yaml \ --wait ``` This creates the `OpenTelemetryCollector` CR (the OTel Operator reconciles it into a Collector Deployment) and a `ServiceMonitor`. That built-in `ServiceMonitor` covers only the Collector's own health (`otelcol_*` on `:8888`); to get the LMCache engine metrics (`lmcache_mp_*` on `:8889`) into Prometheus, add the second `ServiceMonitor` under [Exporting metrics](#exporting-metrics). The engine is **auto-wired** to send OTLP to the Collector — confirm: ```bash theme={null} kubectl get lmcacheengine -n tensormesh-operator -o yaml | grep -A3 extraArgs # extraArgs: # - --enable-tracing # - --otlp-endpoint # - http://-otel-collector-collector..svc:4317 ``` ### Collector ports | Port | Service | Purpose | | - | - | - | | `4317` / `4318` | `…-collector` | OTLP gRPC / HTTP **receiver** — the engine sends here | | `8889` | `…-collector` | **Prometheus exporter** — LMCache *application* metrics, for scraping (empty until the engine sends data) | | `8888` | `…-collector-monitoring` | the Collector's **own internal** metrics (`otelcol_*`) | ## Exporting metrics Two independent options — use either or both: **Scrape in-cluster (ServiceMonitor)** — your Prometheus discovers and scrapes the Collector: ```yaml theme={null} observability: prometheus: serviceMonitor: enabled: true interval: 30s ``` The chart's `ServiceMonitor` scrapes the Collector's internal port (`8888`), not the Prometheus exporter (`8889`). Add your own `ServiceMonitor` for the `…-collector` Service: select on `operator.opentelemetry.io/collector-service-type: base`, use `port: prometheus`, and label it for your Prometheus. **Push to an external Prometheus (`remoteWrite`)** — e.g. Grafana Cloud. Store the auth header in a Secret and reference it from the Collector's environment (see [Authenticating to external backends](#authenticating-to-external-backends)): ```yaml theme={null} observability: otelCollector: envFrom: - secretRef: name: grafana-cloud-credentials prometheus: remoteWrite: enabled: true endpoint: https:///api/prom/push headers: Authorization: ${env:GC_METRICS_AUTH} # resolved from the Secret externalLabels: cluster: my-cluster ``` ## Exporting traces The engine emits traces once observability is enabled; they need a backend to land in. **In-cluster Tempo** (`traces.tempoCR.enabled`) — the chart creates a `TempoMonolithic` CR (reconciled by the Tempo Operator) and wires the Collector to export to it: ```yaml theme={null} observability: enabled: true traces: tempoCR: enabled: true ``` **External backend** (BYOB — e.g. Grafana Cloud Tempo) — point the Collector at an external OTLP endpoint with auth from a Secret: ```yaml theme={null} observability: otelCollector: envFrom: - secretRef: name: grafana-cloud-credentials traces: externalEndpoint: enabled: true endpoint: :443 tls: insecure: false headers: Authorization: ${env:GC_TRACES_AUTH} ``` **Both** — enable `tempoCR` *and* `externalEndpoint` together and the Collector fans traces out to both backends simultaneously. With neither enabled there is no traces pipeline, so spans are discarded and `otelcol_receiver_accepted_spans` never appears. ## Authenticating to external backends Never put tokens in `values.yaml` or `--set`. Store them in a Secret as full auth headers, mount it into the Collector via `otelCollector.envFrom`, and reference the env vars with `${env:VAR}` in the endpoint `headers`: ```bash theme={null} # Build the auth header(s) — example for a Grafana Cloud-style Basic token. # Replace and with your own; do NOT commit these. TRACES_AUTH="Basic $(printf '%s:%s' '' '' | base64)" METRICS_AUTH="Basic $(printf '%s:%s' '' '' | base64)" kubectl create secret generic grafana-cloud-credentials \ -n tensormesh-operator \ --from-literal=GC_TRACES_AUTH="$TRACES_AUTH" \ --from-literal=GC_METRICS_AUTH="$METRICS_AUTH" ``` The Collector loads these as environment variables (via `envFrom`), and `${env:GC_TRACES_AUTH}` / `${env:GC_METRICS_AUTH}` in the values above resolve at runtime — so the token never appears in the chart values or the rendered CR. ## Debug exporter For development, `observability.otelCollector.debug=true` adds a debug exporter that logs every received metric to the Collector's stdout (verbosity `detailed`). Useful for confirming data shape; **turn it off in production** (it's noisy). ```bash theme={null} kubectl logs -n tensormesh-operator deploy/-otel-collector-collector -f ``` ## Verify telemetry is flowing Telemetry is only produced when the engine does work — send inference through vLLM (see the [end-to-end example](/v1.0.0/installation/example)) first. Metrics export on an interval (\~60 s), so allow a minute after traffic. The **source of truth** is the Collector's own counters, not log volume: ```bash theme={null} kubectl port-forward -n tensormesh-operator \ svc/-otel-collector-collector-monitoring 8888:8888 # other terminal: curl -s localhost:8888/metrics | grep -E 'otelcol_receiver_accepted_(metric_points|spans)' ``` A **non-zero** `otelcol_receiver_accepted_metric_points` / `_spans` confirms the engine's OTLP reached the Collector. You can also see the engine's **application** metrics on the Prometheus exporter (`8889` on the main `…-collector` service): ```bash theme={null} kubectl port-forward -n tensormesh-operator \ svc/-otel-collector-collector 8889:8889 # other terminal: curl -s http://localhost:8889/metrics | grep lmcache | head -20 ``` ```text theme={null} # HELP lmcache_mp_l1_memory_usage_bytes Bytes currently held in L1 cache # TYPE lmcache_mp_l1_memory_usage_bytes gauge lmcache_mp_l1_memory_usage_bytes{instance="",otel_scope_name="lmcache.l1_manager"} 0 # HELP lmcache_mp_l1_eviction_loop_ticks_total L1 eviction-loop iterations (every cycle) # TYPE lmcache_mp_l1_eviction_loop_ticks_total counter lmcache_mp_l1_eviction_loop_ticks_total{instance="",otel_scope_name="lmcache.l1"} 227 # HELP lmcache_mp_active_prefetch_jobs Number of active prefetch jobs # TYPE lmcache_mp_active_prefetch_jobs gauge lmcache_mp_active_prefetch_jobs{instance="",otel_scope_name="lmcache.mp_engine"} 0 ``` A batch of `lmcache_mp_*` series = the engine is exporting through the Collector. Empty means no traffic has hit the engine yet. Finally, run the bundled assertion: ```bash theme={null} helm test tensormesh-operator -n tensormesh-operator --logs # the …-test-otel-metrics pod scrapes :8888 and asserts otelcol_ metrics exist ``` Don't verify with a `kubectl run … --image=curlimages/curl -- curl …` pod — that image's entrypoint is already `curl`, so the args double up and it fails. Use the port-forward. ## Disabling and toggling `observability.enabled=false` (the default) produces **zero** observability resources, and CRD validation is skipped — so a default install needs none of the operators above. Toggling is clean: `helm upgrade` with `enabled=true` creates the Collector CR; setting it back to `false` removes the CR and its Collector pods. ## Troubleshooting **`otelcol_receiver_accepted_*` stays at 0 after traffic** The engine isn't reaching the Collector. Check, in order: 1. The engine CR has the tracing `extraArgs` (above). If not, confirm `observability.enabled=true` was applied: `helm get values tensormesh-operator -n tensormesh-operator`. 2. The OTLP endpoint resolves — the Collector's `…-collector` service on port `4317`. 3. The engine pod restarted after the flags were added (its age should post-date enabling observability). **Auth errors (401/403) in Collector logs (external backend)** The Secret's auth header is wrong or expired. Verify the Secret and the `${env:…}` references match. **Collector pod never becomes Ready / install fails on a missing CRD** The OpenTelemetry Operator (or Tempo Operator, if `tempoCR.enabled`) isn't installed — re-check the prerequisite CRDs. ## Next steps Full install → inference → benchmark, including generating the traffic that produces telemetry. All chart values, including the `observability.*` keys. # Metrics Source: https://docs.tensormesh.ai/v1.0.0/observability/metrics The cache-engine metrics the operator exposes, and how to read the ones that matter. Every cache-engine pod exports Prometheus metrics. This page covers how the operator exposes them and the signals that show whether the cache is reducing recompute. ## How metrics reach Prometheus Metrics reach Prometheus one of two ways. The **OpenTelemetry Collector** path is the recommended setup. With `observability.enabled`, each engine pushes its metrics (and traces) to an OTel Collector, and Prometheus scrapes the Collector. Enabling it also stops the engine serving `/metrics` directly. See [Observability](/v1.0.0/observability) for the full setup. The **engine-endpoint** path is an alternative when you don't run the collector. Each engine pod serves `/metrics` directly, and the operator creates a metrics `Service`, plus a `ServiceMonitor` (when `serviceMonitor.enabled` is set) so Prometheus auto-discovers it: ```yaml theme={null} engine: enabled: true spec: prometheus: enabled: true port: 9090 serviceMonitor: enabled: true labels: release: kube-prometheus-stack # must match your Prometheus' serviceMonitorSelector ``` ## Verify metrics are flowing First, find which path this engine uses. The check is whether the engine is pushing OTLP: ```bash theme={null} kubectl get lmcacheengine -n -o yaml | grep -q otlp-endpoint \ && echo "OTLP / collector path" || echo "engine-endpoint path" ``` The engine pushes metrics to an OpenTelemetry Collector, so it exposes no scrapeable `lmcache_mp_*` endpoint itself. Read them from the Collector's Prometheus exporter and confirm the pipeline in [Observability → Verify telemetry is flowing](/v1.0.0/observability#verify-telemetry-is-flowing), or just query Prometheus below. The engine serves `/metrics` on its `--prometheus-port` (default `9090`), behind the `-metrics` Service: ```bash theme={null} kubectl get endpoints -metrics -n # one IP per engine pod kubectl port-forward -n svc/-metrics 9090:9090 curl -s http://localhost:9090/metrics | grep lmcache_mp_ | head ``` `connection refused` means nothing is serving `/metrics` there; confirm `prometheus.enabled` is set and the engine isn't on the OTLP path. The fleet-wide source of truth is your Prometheus, the same data the [Operator UI](/v1.0.0/ui/introduction) reads: ```bash theme={null} kubectl port-forward -n svc/ 9091:9090 curl -s 'http://localhost:9091/api/v1/query?query=sum(rate(lmcache_mp_lookup_hit_tokens_total[5m]))/sum(rate(lmcache_mp_lookup_requested_tokens_total[5m]))' \ | jq -r '.data.result[0].value[1]' ``` A batch of `lmcache_mp_*` lines (or a number from the query) confirms metrics are flowing. Empty means no traffic has hit the engine yet; send some inference through vLLM first. ## Pull every metric See the **complete** set straight from Prometheus (works on either path): ```bash theme={null} # every cache-engine metric name Prometheus holds curl -s 'http:///api/v1/label/__name__/values' \ | jq -r '.data[] | select(startswith("lmcache_mp_"))' # the labels you can slice by (model, tenant, backend, node) curl -s 'http:///api/v1/series?match[]=lmcache_mp_lookup_hit_tokens_total' \ | jq '.data[0] | keys' ``` The **[complete metric reference](#complete-metric-reference)** below catalogs every one. ## What to watch first The signals to check first, in priority order. ### 1. Lookup hit rate How much of each requested prefix was served from cache instead of recomputed on the GPU. ```promql theme={null} sum(rate(lmcache_mp_lookup_hit_tokens_total[5m])) / sum(rate(lmcache_mp_lookup_requested_tokens_total[5m])) ``` | Metric | Meaning | | - | - | | `lmcache_mp_lookup_requested_tokens_total` | Tokens the engine was asked to look up | | `lmcache_mp_lookup_hit_tokens_total` | Tokens found in cache | ### 2. L1 activity L1 is the [host-DRAM tier](/v1.0.0/configuration/cpu-offloading). These counters show whether it's serving reads (good) or just churning writes and evictions (undersized). | Metric | Meaning | | - | - | | `lmcache_mp_l1_read_chunks_total` | Chunks served from DRAM | | `lmcache_mp_l1_write_chunks_total` | Chunks written into DRAM | | `lmcache_mp_l1_evicted_chunks_total` | Chunks evicted under pressure | * Reads rising on warm traffic → L1 is doing its job. * Writes high, reads low → poor reuse, or the hot set doesn't fit — see [CPU offloading sizing](/v1.0.0/configuration/cpu-offloading#sizing). * Evictions climbing fast → L1 is undersized for the working set. ### 3. L2 activity Only relevant if you've configured [external storage](/v1.0.0/configuration/filesystem-offloading). These confirm data is landing in L2 and warm traffic is loading it back. | Metric | Meaning | | - | - | | `lmcache_mp_l2_store_completed_requests_total` | Data written to L2 | | `lmcache_mp_l2_load_completed_requests_total` | Data loaded back from L2 | | `lmcache_mp_l2_prefetch_failure_chunks_total` | Prefetches that failed (miss or L1 pressure) | ### 4. Retrieved chunk volume ```promql theme={null} sum(rate(lmcache_mp_num_chunks_loaded_total[5m])) ``` How much data the engine loads back into service. If hit rate is rising but this stays flat, the warm path isn't actually loading anything. ## Slice by model and tenant Many engine metrics carry labels you can group by: * `model_name` — per-model hit rate and volume * `cache_salt` — per-tenant isolation * `l2_name` — which L2 backend ```promql theme={null} sum by (model_name) (rate(lmcache_mp_lookup_hit_tokens_total[5m])) / sum by (model_name) (rate(lmcache_mp_lookup_requested_tokens_total[5m])) ``` Keep `cache_salt` coarse: one salt per tenant is fine, but one per request explodes your time-series count and hurts cache reuse. ## Complete metric reference Most metrics carry `model_name`, `cache_salt`, and `l2_name` labels for [slicing](#slice-by-model-and-tenant). Counters (`_total`) need `rate(...[5m])`; gauges are instantaneous; histograms expose `_bucket` / `_sum` / `_count`. A metric only appears once it's been recorded, so an idle engine shows only the always-on gauges; the activity counters appear after traffic hits the cache. ### Lookup & hit rate | Metric | Type | Measures | | - | - | - | | `lmcache_mp_lookup_requested_tokens_total` | counter | Tokens the engine was asked to look up (hit-rate denominator) | | `lmcache_mp_lookup_hit_tokens_total` | counter | Tokens found in cache (hit-rate numerator) | ### L1 — CPU offloading (host DRAM) | Metric | Type | Measures | | - | - | - | | `lmcache_mp_l1_usage_ratio` | gauge | L1 fill level, `0..1` | | `lmcache_mp_l1_memory_usage_bytes` | gauge | Bytes currently held in L1 | | `lmcache_mp_l1_read_chunks_total` | counter | Chunks served from L1 (reuse activity) | | `lmcache_mp_l1_write_chunks_total` | counter | Chunks written into L1 (ingest) | | `lmcache_mp_l1_evicted_chunks_total` | counter | Chunks evicted from L1 under pressure | | `lmcache_mp_l1_read_failure_chunks_total` | counter | L1 reads that failed | | `lmcache_mp_l1_eviction_loop_ticks_total` | counter | Eviction-loop iterations (every cycle) | | `lmcache_mp_l1_eviction_loop_triggered_total` | counter | Eviction-loop iterations where the policy actually ran | ### L2 — external storage | Metric | Type | Measures | | - | - | - | | `lmcache_mp_l2_usage_bytes` | gauge | Bytes currently held in L2, per `l2_name` | | `lmcache_mp_l2_adapters` | gauge | L2 adapters on the store controller, by `state` (`active` / `draining`) | | `lmcache_mp_num_inflight_l2_stores` | gauge | Store operations in flight to L2 | | `lmcache_mp_num_inflight_l2_loads` | gauge | Load operations in flight from L2 | | `lmcache_mp_inflight_load_memory_usage_bytes` | gauge | Memory held by in-flight L2 loads | | `lmcache_mp_l2_store_submitted_requests_total` | counter | L2 store requests submitted | | `lmcache_mp_l2_store_submitted_objects_chunks_total` | counter | Chunks submitted for L2 store | | `lmcache_mp_l2_store_completed_requests_total` | counter | L2 store requests completed | | `lmcache_mp_l2_store_completed_objects_chunks_total` | counter | Chunks written to L2 | | `lmcache_mp_l2_load_completed_requests_total` | counter | L2 load requests completed | | `lmcache_mp_l2_prefetch_lookup_requests_total` | counter | L2 prefetch lookup requests submitted | | `lmcache_mp_l2_prefetch_lookup_objects_chunks_total` | counter | Chunks submitted for L2 prefetch lookup | | `lmcache_mp_l2_prefetch_hit_chunks_total` | counter | Prefix chunks found in an L2 lookup | | `lmcache_mp_l2_prefetch_load_submitted_requests_total` | counter | L2 prefetch load requests submitted | | `lmcache_mp_l2_prefetch_load_submitted_objects_chunks_total` | counter | Chunks submitted for L2 load | | `lmcache_mp_l2_prefetch_load_completed_chunks_total` | counter | Chunks successfully loaded from L2 | | `lmcache_mp_l2_prefetch_failure_chunks_total` | counter | Prefetches that failed (`reason` = `l1_oom` / `not_found`) | | `lmcache_mp_num_chunks_loaded_total` | counter | Chunks loaded back into service (the warm path) | ### Throughput Histograms in **GB/s**: compute the live rate as `rate(_sum[5m]) / rate(_count[5m])`. | Metric | Direction | | - | - | | `lmcache_mp_l0_l1_store_throughput_GB_per_second` | GPU → CPU (store into L1) | | `lmcache_mp_l0_l1_load_throughput_GB_per_second` | CPU → GPU (load from L1) | | `lmcache_mp_l2_store_throughput_GB_per_second` | L1 → L2 (store to external) | | `lmcache_mp_l2_load_throughput_GB_per_second` | L2 → L1 (load from external) | ### Prefetch & event bus (health) | Metric | Type | Measures | | - | - | - | | `lmcache_mp_active_prefetch_jobs` | gauge | Prefetch jobs currently running | | `lmcache_mp_active_p2p_lookup_jobs` | gauge | Active peer-to-peer lookup jobs (with [P2P](/v1.0.0/configuration/p2p) enabled) | | `lmcache_mp_event_bus_queue_depth` | gauge | Depth of the internal event queue | | `lmcache_mp_event_bus_drain_lag_seconds` | gauge | Seconds since the oldest queued event; rising = the drain thread is falling behind | | `lmcache_mp_event_bus_dropped_events_total` | counter | Events dropped (queue overflow, a saturation signal) | ### Reuse gap How long cached data lives before it's reused, the input to sizing decisions. | Metric | Type | Measures | | - | - | - | | `lmcache_mp_real_reuse_gap_seconds` | histogram | Time between storing a chunk and reusing it | | `lmcache_mp_real_reuse_gap_objects` | histogram | Reuse-gap distribution across cached objects | ### Lifecycle histograms Per-tier block/chunk timing: how long cached data lives, sits idle, and waits between reuses. Newer engine builds expose more of these. All are histograms (`_bucket` / `_sum` / `_count`). | Metric | Tier | Measures | | - | - | - | | `lmcache_mp_l0_block_lifetime_seconds` | GPU (L0) | Block lifetime, allocation → eviction | | `lmcache_mp_l0_block_idle_before_evict_seconds` | GPU (L0) | Idle time before a block is evicted | | `lmcache_mp_l0_block_reuse_gap_seconds` | GPU (L0) | Gap between consecutive block accesses | | `lmcache_mp_l1_chunk_lifetime_seconds` | L1 (DRAM) | Chunk lifetime, allocation → eviction | | `lmcache_mp_l1_chunk_idle_before_evict_seconds` | L1 (DRAM) | Idle time before a chunk is evicted | | `lmcache_mp_l1_chunk_reuse_gap_seconds` | L1 (DRAM) | Gap between consecutive touches of a chunk | | `lmcache_mp_l1_chunk_evict_reuse_gap_seconds` | L1 (DRAM) | Time from chunk eviction to next reuse | vLLM exports its own `vllm:*` metrics on a separate port (TTFT, running/waiting requests, HBM KV usage). Pair those with the cache-engine metrics above for the full request-to-cache picture. ## When to go deeper The catalog above is complete for operating the cache. For low-level internals (exact histogram-bucket boundaries, precise semantics, and cardinality notes), see the upstream multiprocess-engine reference: [MP observability](https://docs.lmcache.ai/mp/observability.html). ## Next steps An interface that visualizes these metrics: fleet, hit rate, capacity, health. Ship metrics and traces through an OpenTelemetry Collector to Prometheus and Tempo. Tune the L1 tier the L1 metrics above describe. # Architecture Source: https://docs.tensormesh.ai/v1.0.0/reference/architecture Tensormesh Platform installs from a single Helm chart. The chart runs a controller (the **operator**) that reconciles `LMCacheEngine` custom resources into a **DaemonSet of LMCache cache servers**, stands up a fleet **coordinator**, and (when observability is enabled) wires those engines to an **OpenTelemetry Collector** that fans metrics and traces out to Prometheus and Tempo. ## What gets created, and by whom Every piece follows the same pattern: **the chart writes a declaration (a CR); a controller reconciles it into a running thing.** The chart runs exactly **one** controller itself — the operator. The Collector and Tempo are reconciled by operators **you pre-install**; the chart only submits their CRs. This table is the precise "which thing belongs where": | Declaration (the chart creates this) | Reconciled by | Becomes | Scope | | - | - | - | - | | `LMCacheEngine` CR | **operator** (this chart) | engine DaemonSet (1 pod per GPU node) | Namespaced | | `LMCacheCoordinator` CR | **operator** (this chart) | coordinator Deployment + ClusterIP Service (`:9300`) | Namespaced | | `CacheBlendEngine` CR *(Preview)* | **operator** (this chart) | CacheBlend DaemonSet (1 pod per GPU node) | Namespaced | | `OpenTelemetryCollector` CR | **OpenTelemetry Operator** (prerequisite) | OTel Collector pod | Namespaced | | `TempoMonolithic` CR | **Tempo Operator** (prerequisite) | Tempo instance | Namespaced | | `ServiceMonitor` | **Prometheus Operator** (prerequisite) | a scrape target (no new pod) | Namespaced | | CRDs `lmcacheengines`, `lmcachecoordinators`, `cacheblendengines` (`.lmcache.lmcache.ai`) | None (registered with the API server) | define the `LMCacheEngine`, `LMCacheCoordinator`, and `CacheBlendEngine` kinds | **Cluster** | The operator itself runs as a plain `Deployment` (no CR needed) and owns the engine DaemonSet's lifecycle. ## Data flow ```text theme={null} vLLM pod ──hostIPC / MP connector──▶ LMCache engine LMCache engine ──OTLP gRPC───────────▶ OTel Collector OTel Collector ──Prometheus scrape───▶ Prometheus (via ServiceMonitor) OTel Collector ──OTLP traces─────────▶ Tempo / external backend ``` | Hop | Transport | Port | | - | - | - | | vLLM → engine | shared-memory IPC (`hostIPC`) | — (same node) | | engine → Collector | OTLP gRPC | `4317` | | Collector → Prometheus | scrape of the Collector's exporter | `8889` | | Collector → itself | `otelcol_*` internal metrics | `8888` | | Collector → Tempo / external | OTLP | per backend | See [Observability](/v1.0.0/observability) for enabling and verifying this path. ## Resources the chart creates Everything the chart can render, with its **scope**. This is the table to consult before a cleanup, because **cluster-scoped resources survive a namespace delete** and must be removed explicitly. | Resource | API group | Scope | Created when | Cleanup notes | | - | - | - | - | - | | **CustomResourceDefinitions** `lmcacheengines`, `lmcachecoordinators`, `cacheblendengines` (`.lmcache.lmcache.ai`) | `apiextensions.k8s.io` | **Cluster** | `crds.enabled` (default `true`) | ⚠️ All three **kept on `helm uninstall`** (`resource-policy: keep`). Survive namespace delete. Remove with `kubectl delete crd …`. | | **ClusterRole** (operator) | `rbac.authorization.k8s.io` | **Cluster** | `operator.rbac.create` | ⚠️ Survives namespace delete; sweep manually. | | **ClusterRoleBinding** (operator) | `rbac.authorization.k8s.io` | **Cluster** | `operator.rbac.create` | ⚠️ Survives namespace delete; sweep manually. | | **ClusterRole / ClusterRoleBinding** (pre-delete hook) | `rbac.authorization.k8s.io` | **Cluster** | `helm uninstall` hook | Created and removed by the hook; can orphan if uninstall fails mid-way. | | `LMCacheEngine` (CR) | `lmcache.lmcache.ai/v1alpha1` | Namespaced | `engine.enabled` (default `true`) | Has a `lmcache.ai/cleanup` **finalizer**: if the operator is gone, it blocks namespace deletion. See [troubleshooting](/v1.0.0/installation/troubleshooting#cleaning-up-an-inconsistent-or-orphaned-install). | | `LMCacheCoordinator` (CR) | `lmcache.lmcache.ai/v1alpha1` | Namespaced | `coordinator.enabled` (default `true`) | Goes with the namespace. | | `CacheBlendEngine` (CR) *(Preview)* | `lmcache.lmcache.ai/v1alpha1` | Namespaced | `cacheBlend.enabled` | Goes with the namespace. | | `OpenTelemetryCollector` (CR) | `opentelemetry.io` | Namespaced | `observability.enabled` | Goes with the namespace. | | `TempoMonolithic` (CR) | `tempo.grafana.com` | Namespaced | `observability.traces.tempoCR.enabled` | Goes with the namespace. | | `ServiceMonitor` (CR) | `monitoring.coreos.com` | Namespaced | `operator.serviceMonitor.enabled` and/or `observability.prometheus.serviceMonitor.enabled` | Goes with the namespace. | | Deployment (controller-manager) | `apps` | Namespaced | `operator.enabled` | Goes with the namespace. | | DaemonSet (engine) | `apps` | Namespaced | created by the operator from the `LMCacheEngine` CR | Removed when the CR/namespace is. | | Deployment + Service (coordinator) | `apps` / core | Namespaced | created by the operator from the `LMCacheCoordinator` CR | Removed when the CR/namespace is. | | DaemonSet (CacheBlend engine) *(Preview)* | `apps` | Namespaced | created by the operator from the `CacheBlendEngine` CR | Removed when the CR/namespace is. | | Service (metrics) | core | Namespaced | `operator.enabled` | Goes with the namespace. | | ServiceAccount (operator) | core | Namespaced | `operator.serviceAccount.create` | Goes with the namespace. | | Role / RoleBinding (operator) | `rbac.authorization.k8s.io` | Namespaced | `operator.rbac.create` | Goes with the namespace. | | ServiceAccount + RoleBinding (privileged SCC) | core / `rbac…` | Namespaced | `openshift.enabled` | Goes with the namespace. Binds to the built-in `system:openshift:scc:privileged` ClusterRole (**do not delete that**). | | NetworkPolicy | `networking.k8s.io` | Namespaced | `operator.networkPolicy.enabled` | Goes with the namespace. | | Job + ServiceAccount + Role + RoleBinding (pre-delete hook) | batch / core / `rbac…` | Namespaced | `helm uninstall` hook | Transient; removed after the hook runs. | ### What a namespace delete leaves behind If you `kubectl delete namespace` instead of `helm uninstall`, these **cluster-scoped** resources orphan and need explicit cleanup: 1. **CRDs** `lmcacheengines`, `lmcachecoordinators`, and `cacheblendengines` (all `.lmcache.lmcache.ai`) 2. **ClusterRole** for the operator 3. **ClusterRoleBinding** for the operator ```bash theme={null} kubectl delete crd lmcacheengines.lmcache.lmcache.ai lmcachecoordinators.lmcache.lmcache.ai cacheblendengines.lmcache.lmcache.ai kubectl get clusterrole,clusterrolebinding | grep -i tensormesh # delete what it lists ``` See [Troubleshooting → Cleaning up an inconsistent or orphaned install](/v1.0.0/installation/troubleshooting#cleaning-up-an-inconsistent-or-orphaned-install) for the full recipe (including clearing the `LMCacheEngine` finalizer first). ## Next steps Every `values.yaml` key and example overlays. The OTel Collector / Tempo / Prometheus wiring in detail. # CLI Source: https://docs.tensormesh.ai/v1.0.0/reference/cli The Tensormesh Platform CLI for observing and operating LMCacheEngine fleets on Kubernetes. The **Tensormesh Platform CLI** is the administrator command line for Tensormesh Platform. It is built for the platform and SRE engineers who run the cache: it reads LMCacheEngine resources and talks to the KV-cache server pod on each GPU node. The command itself is `tmo-cli`. It covers two surfaces: * the Kubernetes CRD (`lmcacheengines.lmcache.lmcache.ai`), for engine status, conditions, and endpoints * each server pod's HTTP control API, reached through the Kubernetes API-server proxy, for cache state, quotas, and diagnostics This is the cluster-side administrator CLI. It is not the **Tensormesh CLI** (`tm`), which manages accounts, billing, and serverless inference against the hosted API. ## Install The CLI ships with the gated Tensormesh Platform distribution rather than through public Homebrew or krew. It installs from the Tensormesh registry using the same access token as the Platform chart and images. That token is scoped to artifact pull and grants no access to Tensormesh source code; if you don't have one, request it from the Tensormesh team. It is the same token used in the [install guide](/v1.0.0/installation/helm). The install script needs `curl`, `tar`, and either `shasum` or `sha256sum`. ### One-line install ```bash theme={null} curl -sSfL https://operator.tensormesh.ai/install-cli.sh | TMO_TOKEN= sh ``` The script detects your OS and architecture, pulls the matching binary from the Tensormesh registry, checks the archive against the digest recorded in the OCI manifest, and installs to `/usr/local/bin`, falling back to `~/.local/bin` when that isn't writable so no `sudo` is required. To pin a version or change the target, pass flags after `-s --`: ```bash theme={null} curl -sSfL https://operator.tensormesh.ai/install-cli.sh | sh -s -- \ --token --version --install-dir ~/bin ``` `--version` takes a release tag including the leading `v`, and defaults to `latest`. Each flag also has an environment-variable form: `TMO_TOKEN`, `TMO_VERSION`, `TMO_INSTALL_DIR`. ### Manual install (air-gapped or CI) To pull the artifact yourself, authenticate to the Tensormesh registry and use [`oras`](https://oras.land): ```bash theme={null} echo '' | oras login artifacts.tensormesh.ai -u tensormesh --password-stdin oras pull artifacts.tensormesh.ai/tensormesh-production/cli/tmo-cli:latest # or a : tag tar -xzf tmo-cli_*_linux_amd64.tar.gz tmo-cli # match your OS/arch install -m 0755 tmo-cli ~/.local/bin/tmo-cli ``` Archives are named `tmo-cli___.tar.gz`, where `` drops the leading `v` of the release tag. Builds are published for linux and darwin, on amd64 and arm64. ### Verify ```bash theme={null} tmo-cli version tmo-cli diag preflight # checks your RBAC against what the admin commands need ``` ### Shell completion ```bash theme={null} source <(tmo-cli completion zsh) # this session; also bash, fish, powershell ``` To install permanently, run `tmo-cli completion --help` and follow the instructions for your shell and platform. ## How it connects `tmo-cli` uses your existing kubeconfig. If `kubectl` works, `tmo-cli` works. Resolution order is `--kubeconfig`, then `$KUBECONFIG`, then the `tmo-cli` config default, then `~/.kube/config`. ```bash theme={null} tmo-cli config use-context prod-us-east tmo-cli config set-default --output table tmo-cli config view ``` Commands that talk to engine pods (`node`, `cache`, `quota`, `diag`) go through the Kubernetes API-server proxy by default, so they need no network path beyond the one `kubectl` already uses and RBAC on `pods/proxy` gates them. Pass `--direct` to reach pod host IPs instead, which is useful when running inside the cluster. ## Flags Four flags are accepted by every command: | Flag | Meaning | | - | - | | `--kubeconfig` | path to kubeconfig, overriding `$KUBECONFIG` | | `--context` | kubeconfig context to use | | `-o, --output table\|json\|yaml` | output format; JSON and YAML carry a versioned envelope (`tmoOutputVersion: v1`) | | `--no-color` | disable ANSI color; also honors `NO_COLOR`, and auto-off when stdout is not a TTY | The rest are command-scoped, and the scoping matters when you script: | Flag | Where it applies | | - | - | | `-n, --namespace` | every command except `config` and `version` (default `default`) | | `-e, --engine` | commands that reach engine pods: `node list`, `cache *`, `quota *`, `diag health\|periodic\|threads\|bundle\|loglevel set`. `engine` commands take the name as an argument instead | | `-A, --all-namespaces` | `engine list`, plus the pod fan-out commands. Not available on `quota *` or `diag preflight` | | `--node` | `cache status`, `cache clear`, `quota *`, `diag threads`, `diag loglevel set`, `engine logs` | | `--direct` | all pod-reaching commands | | `--workers`, `--pod-timeout` | fan-out commands only (defaults 8 and 5s). Not on `quota *`, which queries a single server | ## Exit codes Stable and scriptable. Changing one is a breaking change. | Code | Meaning | | - | - | | `0` | success | | `1` | generic or unexpected error | | `2` | usage error: bad flag, unknown command, or a removed `engine` write verb | | `3` | not found (engine, server, cache\_salt) | | `4` | validation or guardrail rejection | | `5` | timed out, for example `engine wait` | | `6` | confirmation declined, or `--yes` required and absent | `diag health` overloads `1`: it exits `0` when every pod in scope is reachable and healthy, and `1` when any pod is unhealthy or unreachable. That makes it usable directly as a health gate in CI. ## Common tasks ### See the fleet ```bash theme={null} tmo-cli engine list -A # all engines, all namespaces tmo-cli engine list -A -w # watch for changes (table output only) tmo-cli engine describe my-cache # spec, status, conditions, endpoints, events ``` `engine describe` looks up recent events by default; pass `--no-events` to skip that call on a busy cluster. ### Deploy or change engines (owned by Helm) Fleet membership and engine spec belong to the Helm release. `tmo-cli` cannot create, apply, edit, or delete engines; those verbs exit `2` and point you at Helm. To add, change, or remove an engine, edit the chart values and run `helm upgrade`. The CLI's job in a rollout is the readiness gate afterwards: ```bash theme={null} helm upgrade tensormesh-operator oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator -f values.yaml tmo-cli engine wait my-cache --for=AllInstancesReady --timeout=5m tmo-cli engine describe my-cache ``` `--for` accepts `Available`, `AllInstancesReady`, or `ConfigValid`, and defaults to `AllInstancesReady`. `--timeout` defaults to `5m`. A timeout exits `5`. ### Inspect servers and cache state ```bash theme={null} tmo-cli node list -e my-cache # per-node servers with live health tmo-cli cache status -e my-cache # aggregated /status, per-node and TOTAL tmo-cli cache status -e my-cache --node gpu-node-07 # raw /status passthrough for one server tmo-cli engine logs my-cache --node gpu-node-07 -f # --all multiplexes every pod ``` `engine logs` shows the last 100 lines per pod; change that with `--tail` (`-1` for the whole log). ### Operate (destructive, gated) ```bash theme={null} tmo-cli cache clear -e my-cache --node gpu-node-07 # one server; prompts tmo-cli cache clear -e my-cache --all --yes # whole fleet, no prompt tmo-cli quota list -e my-cache # per-cache_salt L2 budgets tmo-cli quota set acme-prod --limit 48 -e my-cache # --limit is in GB tmo-cli quota rm bench-harness -e my-cache --yes # data evicts next cycle ``` `cache clear` bypasses read and write locks and drops L1 contents. It refuses while a server still has active sessions; add `--force` to clear anyway. It and `quota rm` print a blast-radius preamble and prompt for confirmation, which `-y/--yes` skips. In a non-TTY without `--yes` they exit `6` rather than guess. Quotas are enforced only on engines running the `IsolatedLRU` eviction policy. Against any other policy the CLI accepts the command and warns, but the budget has no effect. Quota commands query one server (the first ready endpoint, or `--node`), since every server of an engine shares the same quota config. ### Diagnose and file a ticket ```bash theme={null} tmo-cli diag health -e my-cache # healthcheck + periodic-thread sweep tmo-cli diag periodic -e my-cache # per-server periodic-thread detail tmo-cli diag threads --node gpu-node-07 -e my-cache tmo-cli diag loglevel set lmcache DEBUG --node gpu-node-07 -e my-cache tmo-cli diag bundle -e my-cache --output-file support.tgz ``` `diag loglevel set` takes a logger and one of `DEBUG`, `INFO`, `WARNING`, `ERROR`, and applies to one server with `--node` or the whole engine with `--all`. `diag bundle` collects the CR, recent events, and each server's `/conf`, `/status`, `/threads`, and `/periodic-threads-health` into a gzip tarball, writing to `tmo-bundle--.tgz` unless you pass `--output-file`. Secret-looking values are redacted, and `/env` is excluded entirely unless you pass `--include-env` (it is still redacted). Attach the tarball to a support ticket. ## Output and scripting Table output is for reading. `-o json` and `-o yaml` are the automation contract, wrapped in an envelope carrying `tmoOutputVersion: v1`: ```bash theme={null} tmo-cli node list -e my-cache -o json | jq '.items[] | select(.healthy != true)' tmo-cli engine get my-cache -o json | jq '.item.status.readyInstances' ``` Use `.healthy != true`, not `.healthy == false`. A pod the CLI could not reach has no `healthy` field at all, only `error`, so an equality test silently drops the unreachable pods you are looking for. Single-item commands render under `.item`, lists under `.items`. Color is disabled automatically in pipes and non-terminals, and never appears in JSON or YAML. ## Troubleshooting | Symptom | Cause and fix | | - | - | | `... forbidden ... ask for the 'tensormesh-admin' role` | your RBAC lacks a verb the command needs; run `tmo-cli diag preflight` to see which | | `node list` or `cache status` returns nothing | the engine has no ready endpoints; check that a GPU node is up and the engine is `Running` | | `unreachable` in a health table | that server's HTTP API did not answer; check `tmo-cli engine describe` and pod status | | `unknown shorthand flag: 'A'` | that command has no `-A`; `quota` and `diag preflight` are namespace-scoped | | `engine wait` exits `5` | the condition did not become true within `--timeout` | | `cache clear` refuses | the server has active sessions; re-run with `--force` once you are sure | | fan-out commands hang on a large fleet | raise `--workers` or `--pod-timeout`; partial results still render | `tmo-cli` does not phone home. There is no active telemetry in the shipped build. # Configuration Source: https://docs.tensormesh.ai/v1.0.0/reference/configuration Every values.yaml key for the Tensormesh Platform Helm chart, plus ready-to-use example overlays. Configure Tensormesh Platform with a **values file** (`-f my-values.yaml`) rather than `--set` flags. It's reproducible and reviewable, and the chart validates it against a bundled `values.schema.json` at parse time (typos and wrong types fail fast). To see every key with its live default: ```bash theme={null} helm show values oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator --version ``` **Pin explicit image tags** in your values file, never `latest`. With `imagePullPolicy: IfNotPresent`, a node that cached an image under a mutable tag keeps the stale copy. Pin the chart (`--version `) and the runtime image (`v` or an explicit nightly). By default the operator image tag follows the chart `appVersion`, but you can still override it with `operator.image.tag`. See the [Compatibility Matrix](/v1.0.0/installation/compatibility) for the validated version tuple. ## Top level | Key | Type | Default | Description | | - | - | - | - | | `crds.enabled` | bool | `true` | Install the three CRD schemas: `LMCacheEngine`, `LMCacheCoordinator`, `CacheBlendEngine`. | | `nameOverride` | string | `""` | Override the chart name used in resource names. | | `fullnameOverride` | string | `""` | Fully override the generated `-` prefix. | ## `operator` — controller-manager | Key | Type | Default | Description | | - | - | - | - | | `operator.enabled` | bool | `true` | Deploy the controller-manager. Disable for CR-only installs. | | `operator.replicas` | int | `1` | Replicas. Leader-elected; `>1` = HA standby. | | `operator.image.repository` | string | `lmcache/lmcache-operator` | Operator image repo. | | `operator.image.tag` | string | `""` | Empty = chart `appVersion`. Pinning the chart version is the explicit pin. | | `operator.image.pullPolicy` | string | `IfNotPresent` | `Always` / `IfNotPresent` / `Never`. | | `operator.image.pullSecrets` | list | `[]` | Image pull secret names. | | `operator.resources` | object | `requests 10m/256Mi, limits 500m/1Gi` | Operator container resources. Override these for unusually large clusters, but the chart now defaults to a more conservative memory floor than the original low-memory settings. | | `operator.leaderElection.enabled` | bool | `true` | Required when `replicas > 1`. | | `operator.healthProbe.port` | int | `8081` | Liveness/readiness probe port. | | `operator.metrics.enabled` | bool | `true` | Expose controller-manager metrics. | | `operator.metrics.port` | int | `8443` | Metrics port. | | `operator.metrics.secure` | bool | `true` | HTTPS + authn/authz filter on metrics. | | `operator.metrics.enableHTTP2` | bool | `false` | Off by default (CVE guidance). | | `operator.metrics.tls.*` | — | — | Mount a TLS cert (`certPath`, `certName`, `certKey`, `secretName`). Empty = self-signed. | | `operator.serviceAccount.create` | bool | `true` | Create the operator ServiceAccount. | | `operator.serviceAccount.name` | string | `""` | Override SA name. | | `operator.serviceAccount.annotations` | object | `{}` | SA annotations. | | `operator.rbac.create` | bool | `true` | Create the operator **ClusterRole + ClusterRoleBinding** (cluster-scoped). | | `operator.serviceMonitor.enabled` | bool | `false` | `ServiceMonitor` for the operator's own metrics (`:8443`). | | `operator.serviceMonitor.interval` / `.scrapeTimeout` | string | `30s` / `10s` | Scrape timing. | | `operator.serviceMonitor.labels` / `.tlsConfig` | object | `{}` | Selector labels / TLS override. | | `operator.networkPolicy.enabled` | bool | `false` | Restrict ingress to the operator pod. | | `operator.networkPolicy.metricsNamespaceSelector` | object | `{matchLabels: {metrics: enabled}}` | Who may scrape metrics. | | `operator.nodeSelector` / `.tolerations` / `.affinity` | — | `{}` / `[]` / `{}` | Operator pod scheduling. | | `operator.podAnnotations` / `.podLabels` / `.priorityClassName` | — | `{}` / `{}` / `""` | Operator pod metadata. | ## `webhook` — pod-mutating webhook | Key | Type | Default | Description | | - | - | - | - | | `webhook.enabled` | bool | `true` | Serve the **CacheBlend** pod-mutating webhook. **Requires cert-manager**: rendering fails fast without it. When `false`, the operator runs with `ENABLE_WEBHOOKS=false` and no cert-manager objects are created. | | `webhook.port` | int | `9443` | Container port the webhook server listens on. | The webhook patches opted-in vLLM **pods** at admission, and is a prerequisite for `cacheBlend.enabled`. The chart registers **both** pod injectors: CacheBlend (`lmcache.ai/cacheblend-inject`) and LMCache **connection** injection (`lmcache.ai/lmcache-inject`). Both take effect on a chart install. The release namespace is excluded from injection, and the engine is resolved in the opted-in pod's own namespace. See [Modify an Existing Deployment](/v1.0.0/installation/existing-deployment). ## `engine` — the LMCacheEngine CR | Key | Type | Default | Description | | - | - | - | - | | `engine.enabled` | bool | `true` | Create an `LMCacheEngine` CR in this release. | | `engine.name` | string | `""` | CR name. Empty = `-default-engine`. | | `engine.namespace` | string | `""` | CR namespace. Empty = release namespace. | | `engine.spec` | object | `{l1: {sizeGB: 60}, privileged: false}` | Rendered **verbatim** into the CR's `spec`. Required: `l1.sizeGB`. | | `engine.spec.l1.sizeGB` | int | `60` | Per-node L1 cache size in GB. | | `engine.spec.privileged` | bool | `false` | Run engine containers privileged. Set `true` when the node runtime cannot otherwise expose GPUs. Requires operator `v0.5.1`+. `gpuVendor: amd` forces it regardless. | | `engine.p2p.enabled` | bool | `false` | Peer-to-peer KV transfer between engine pods (sets `--p2p-advertise-url`, which is what turns P2P on). | | `engine.p2p.port` | int | `8500` | Port advertised to peers; the URL is `:`. | | `engine.p2p.l1AlignBytesOverwrite` | int | `65536` | L1 buffer alignment. Appended last, so it overrides the operator default (4 KiB). 64 KiB is recommended for P2P. | `engine.spec` is a passthrough to the `LMCacheEngine` CRD, so its full surface is defined by the CRD, not the chart. The fields below have **no chart default**: leave one out and the operator's own default applies. Inspect the live schema with `kubectl explain lmcacheengine.spec` once the CRD is installed. | Field | Example | What it does | | - | - | - | | `image.{repository,tag,pullPolicy}` | `lmcache/vllm-openai`, `v` | Engine container image. Pin the tag the [Compatibility Matrix](/v1.0.0/installation/compatibility) pairs with your chart. | | `server.{port,chunkSize,maxWorkers}` | `5555`, `256`, `1` | Cache-server listener and KV chunk size in tokens. Prompts shorter than `chunkSize` store nothing. | | `eviction.{triggerWatermark,evictionRatio}` | `0.8`, `0.2` | Start evicting at 80% full, drop 20% when triggered. | | `prometheus.{enabled,port}` | `true`, `9090` | Engine-side metrics endpoint. | | `l2Backend` | `{resp: {host: redis, port: 6379}}` | External L2 tier behind L1. | | `l2Backend.serde.aesgcm` | see below | At-rest encryption for L2. See [KVCache Encryption](/v1.0.0/configuration/kvcache-encryption). Requires operator `v0.5.3`+. | | `nodeSelector` | `{nvidia.com/gpu.present: "true"}` | Where engine pods run. This is the operator's default, and vLLM pods must land on the same nodes to reach the engine. | | `serviceAccountName` | `lmcache-engine-sa` | Must already exist in the engine namespace. | | `logLevel` | `INFO` | `DEBUG` / `INFO` / `WARNING` / `ERROR`. | | `extraArgs` | `["--foo", "bar"]` | Appended to the server command; skipped if the flag is already present. | ## `openshift` — privileged SCC for hostIPC | Key | Type | Default | Description | | - | - | - | - | | `openshift.enabled` | bool | `false` | Bind a privileged SCC to engine pods (required on OpenShift). | | `openshift.sccClusterRole` | string | `system:openshift:scc:privileged` | ClusterRole granting `use` on the SCC. | | `openshift.serviceAccount.create` | bool | `true` | Create the privileged SA in the engine namespace. | | `openshift.serviceAccount.name` | string | `""` | Override SA name. Empty = `-engine-privileged`. | | `openshift.serviceAccount.annotations` | object | `{}` | SA annotations. | ## `coordinator` — the LMCacheCoordinator CR | Key | Type | Default | Description | | - | - | - | - | | `coordinator.enabled` | bool | `true` | Create an `LMCacheCoordinator` CR — one fleet-wide Deployment the engines register with. | | `coordinator.spec` | object | `{}` | Rendered **verbatim** into the CR. Empty means every knob (image, port `9300`, replicas, timeouts, eviction) takes its server-side default. | Override only to tune a specific knob: `replicas`, `port`, `instanceTimeout`, `evictionRatio`, `triggerWatermark`, `logLevel`, `resourceOverrides`. ## `cacheBlend` — the CacheBlendEngine CR *(Preview)* | Key | Type | Default | Description | | - | - | - | - | | `cacheBlend.enabled` | bool | `false` | Create a `CacheBlendEngine` CR. Requires `webhook.enabled=true` **and** cert-manager. | | `cacheBlend.name` | string | `""` | CR name. Empty = `-cacheblend`. | | `cacheBlend.namespace` | string | `""` | CR namespace. Empty = release namespace. | | `cacheBlend.spec` | object | `{l1: {sizeGB: 60}}` | Rendered **verbatim** into the CR. Required: `l1.sizeGB` (host DRAM, same as the engine's L1). | `cacheBlend.spec` accepts the same passthrough fields as `engine.spec`, plus CacheBlend-only ones: `blend.{checkLayer,recompRatio}` (how much KV is recomputed rather than reused), `injection.{payloadImage,imagePullSecrets,cudagraph}` (what the webhook injects into opt-in vLLM pods), and `server.chunkSize`, which is **fixed at 256** for CacheBlend — it must equal vLLM's `--block-size 64` × 4. `cacheBlend.spec.injection.payloadImage` is the **private** plugin init-container image and is required for injection to happen at all. Set it together with a pull secret that exists in the vLLM pod's namespace. See [CacheBlend](/v1.0.0/configuration/cacheblend) and `examples/values-cacheblend.yaml`. ## `observability` — OTel Collector, metrics, traces | Key | Type | Default | Description | | - | - | - | - | | `observability.enabled` | bool | `false` | Create an `OpenTelemetryCollector` CR and auto-wire the engine. Requires the OpenTelemetry Operator CRD. | | `observability.otelCollector.resources` | object | `{}` | Collector pod resources. | | `observability.otelCollector.debug` | bool | `false` | Debug exporter — logs every metric/trace to stdout. Dev only. | | `observability.otelCollector.envFrom` | list | `[]` | `envFrom` entries (e.g. a `secretRef`) to mount auth tokens; reference with `${env:VAR}`. | | `observability.prometheus.serviceMonitor.enabled` | bool | `true` | `ServiceMonitor` for the Collector's prometheus exporter (`:8889`). | | `observability.prometheus.serviceMonitor.interval` | string | `30s` | Scrape interval. | | `observability.prometheus.serviceMonitor.additionalLabels` | object | `{}` | Selector labels (e.g. `release: kube-prometheus-stack`). | | `observability.prometheus.remoteWrite.enabled` | bool | `false` | Push metrics to a Prometheus remote-write endpoint (Grafana Cloud, Mimir, …). | | `observability.prometheus.remoteWrite.endpoint` | string | `""` | Remote-write URL. | | `observability.prometheus.remoteWrite.headers` | object | `{}` | Headers (use `${env:VAR}` for secrets). | | `observability.prometheus.remoteWrite.externalLabels` | object | `{}` | Labels added to all remote-written metrics. | | `observability.traces.externalEndpoint.enabled` | bool | `false` | Push traces to an external OTLP endpoint. | | `observability.traces.externalEndpoint.endpoint` | string | `""` | OTLP gRPC endpoint (e.g. `tempo.monitoring:4317`). | | `observability.traces.externalEndpoint.tls.insecure` | bool | `true` | Set `false` for TLS endpoints (e.g. Grafana Cloud :443). | | `observability.traces.externalEndpoint.headers` | object | `{}` | Headers (use `${env:VAR}`). | | `observability.traces.tempoCR.enabled` | bool | `false` | Create a `TempoMonolithic` CR. Requires the Tempo Operator. | ## `preDelete` and `tests` | Key | Type | Default | Description | | - | - | - | - | | `preDelete.image` | string | `bitnami/kubectl:latest` | Image for the pre-delete hook Job (removes the CR before the operator). | | `preDelete.timeoutSeconds` | int | `120` | Hook Job `activeDeadlineSeconds`. | | `tests.enabled` | bool | `true` | Render `helm test` smoke pods. | | `tests.image` | string | `bitnami/kubectl:latest` | kubectl-based test pod image. | | `tests.curlImage` | string | `curlimages/curl:8.10.1` | curl-based test pod image. | | `tests.integration.enabled` | bool | `false` | Run the engine serve/store/retrieve e2e. Needs a GPU node. | | `tests.integration.vllmImage` | string | `lmcache/vllm-openai:latest-nightly` | Image the integration test serves with. | | `tests.integration.model` | string | `Qwen/Qwen3-0.6B` | Model the integration test loads. Keep it small. | | `tests.cacheblend.enabled` | bool | `false` | Run the CacheBlend reconcile + webhook-wiring test. | | `tests.cacheblend.integration.enabled` | bool | `false` | Run the CacheBlend injection + serve e2e. Needs a GPU node. | | `tests.cacheblend.integration.model` | string | `Qwen/Qwen3-0.6B` | Model the CacheBlend e2e loads. | | `tests.cacheblend.integration.workloadNamespace` | string | `""` | Empty = `-cb-itest`. **Must differ from the release namespace** — the webhook excludes its own. | ## Example overlays The chart repo ships ready-to-use overlays under `examples/`: | File | Shape | | - | - | | `values-minimal.yaml` | Operator + engine + coordinator, native filesystem L2. Smallest viable install. | | `values-openshift.yaml` | Adds the privileged SCC binding for `hostIPC`. | | `values-p2p.yaml` | Adds peer-to-peer KV transfer between engine pods (with `hostNetwork`). | | `values-cacheblend.yaml` | `CacheBlendEngine` + webhook for non-prefix KV reuse. Requires cert-manager. | | `values-l2-encrypted.yaml` | At-rest encryption of the L2 tier. See [KVCache Encryption](/v1.0.0/configuration/kvcache-encryption). | | `values-pd.yaml` | Prefill/decode disaggregation: one engine DaemonSet serving both roles. | | `values-observability-e2e.yaml` | Engine + full OTel stack (Collector + ServiceMonitor + debug). | ### Minimal (Kubernetes) ```yaml my-values.yaml theme={null} engine: enabled: true spec: l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v pullPolicy: IfNotPresent ``` ### OpenShift ```yaml values-openshift.yaml theme={null} openshift: enabled: true engine: enabled: true spec: l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v pullPolicy: IfNotPresent ``` ### Observability (metrics + traces to a local Tempo) ```yaml values-observability.yaml theme={null} openshift: enabled: true engine: enabled: true spec: l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v pullPolicy: IfNotPresent observability: enabled: true otelCollector: debug: true prometheus: serviceMonitor: enabled: true traces: tempoCR: enabled: true ``` Apply any of these with: ```bash theme={null} helm upgrade --install tensormesh-operator \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator \ --version -n tensormesh-operator --create-namespace \ -f my-values.yaml --wait ``` ## Next steps What each value deploys, and which resources are cluster-scoped. The install/upgrade/uninstall flow. # Glossary / Vocabulary Source: https://docs.tensormesh.ai/v1.0.0/reference/glossary Plain-language definitions of the terms used throughout the Tensormesh Platform docs. Short definitions of the terms you'll meet in these docs, grouped by area. ## Tensormesh | Term | Meaning | | - | - | | **Tensormesh Platform** | The product: self-hosted KV caching for your vLLM fleet, deployed on Kubernetes as an operator that reconciles your cache resources (`LMCacheEngine`, `LMCacheCoordinator`, `CacheBlendEngine`) into a running engine fleet. | | **vLLM** | The open-source LLM inference server that serves your models. The cache engine runs alongside vLLM and offloads its KV cache, so shared prefixes are served from cache instead of recomputed on the GPU. | | **LMCache** | The open-source KV (key-value) cache for LLM inference that the engine runs. When requests share a prefix (system prompt, RAG context, prior turn), it lets later requests reuse the earlier request's attention KV tensors instead of recomputing them. | | **Engine** | A running LMCache cache server pod (one per GPU node, via a DaemonSet). Created by the operator from an `LMCacheEngine` CR. | | **Coordinator / `LMCacheCoordinator`** | A fleet-wide service the engines register with on startup (a Deployment plus a ClusterIP Service on port 9300), enabling cross-node cache awareness. Reconciled from an `LMCacheCoordinator` CR; enabled by default with server-side defaults, so no configuration is required. | | **`LMCacheEngine`** | The custom resource you write to declare an engine. Group/version `lmcache.lmcache.ai/v1alpha1`. | | **CacheBlend / `CacheBlendEngine`** | A Tensormesh technique (and its per-node CR) that reuses KV cache beyond exact prefix matches (any shared block of tokens, even after a per-request preamble), then recomputes a small subset to stitch the blocks together. Useful for RAG. See [Non-Prefix KV Caching](/v1.0.0/configuration/cacheblend). | | **MP mode / `LMCacheMPConnector`** | "Multi-process" mode: vLLM connects to the engine on the same node over shared-memory IPC (`hostIPC`). The connector ships KV blocks between vLLM and the engine. | | **L0 / L1 / L2** | Cache tiers. **L0** is GPU HBM (vLLM's own paged KV cache). **L1** is CPU offloading: host DRAM, sized by `engine.spec.l1.sizeGB`. **L2** is external storage: an optional backend (Redis, S3, filesystem) set via `engine.spec.l2Backend`. | | **Prefetch** | Loading KV blocks from L2 (external storage) back into L1 (host DRAM) on a miss, so the next reuse is served from memory. Tuned by `engine.spec.l2Backend.prefetchPolicy` and `prefetchMaxInFlight`. | | **KV cache** | The key/value attention tensors vLLM computes for the tokens in a prompt. Reusing them across requests instead of recomputing is what saves GPU work and lowers TTFT. | | **KV cache hit** | A later request reused stored KV blocks for a shared prefix. Visible as `prefix hits` in engine logs and lower TTFT. | | **TTFT** | Time to first token — how long a request waits before the model emits its first output token. Cache hits cut it by skipping prefix recomputation. | ## Kubernetes | Term | Meaning | | - | - | | **CRD** (CustomResourceDefinition) | Extends the Kubernetes API with a new resource *kind*. Cluster-scoped. Here: `lmcacheengines.lmcache.lmcache.ai`. | | **CR** (Custom Resource) | An *instance* of a CRD-defined kind, e.g. an `LMCacheEngine`. | | **Operator / controller** | A program that watches CRs and reconciles them into lower-level objects (Deployments, DaemonSets, …). | | **DaemonSet** | Runs one pod per matching node — the shape used for the engine (one per GPU node). | | **Deployment** | Maintains N identical pods with rolling updates — the shape used for the operator. | | **Finalizer** | A marker on a resource that blocks its deletion until a controller runs cleanup. The `LMCacheEngine`'s `lmcache.ai/cleanup` finalizer can block a namespace delete if the operator is gone. | | **Namespaced vs cluster-scoped** | Namespaced resources are removed when their namespace is deleted; **cluster-scoped** ones (CRD, ClusterRole, ClusterRoleBinding) survive and must be cleaned up explicitly. | | **SCC** (Security Context Constraint) | OpenShift's per-pod security policy. Engine pods need the `privileged` SCC because they use `hostIPC`. | | **ServiceMonitor** | A Prometheus Operator CR telling Prometheus what to scrape. | ## Helm | Term | Meaning | | - | - | | **Chart** | A versioned, packaged bundle of Kubernetes templates + default values. | | **Release** | A named installation of a chart in a namespace. Helm stores its history (as Secrets) to enable upgrade/rollback. | | **Values / values file** | User configuration that overrides chart defaults, supplied via `-f my-values.yaml` (preferred) or `--set`. | | **appVersion vs version** | `version` is the *chart's* version; `appVersion` is the version of the *app* it deploys. The operator image tag defaults to `appVersion`. | | **OCI artifact** | A chart stored in a container registry (here GHCR) using OCI media types, pulled with `oci://…` instead of a classic HTTP repo. | | **Hook** | A Job/pod Helm runs at a lifecycle point (pre-install, pre-delete, test). The chart's pre-delete hook removes the engine CR before the operator. | ## Observability | Term | Meaning | | - | - | | **OTLP** | OpenTelemetry Protocol — how the engine ships metrics/traces to the Collector (gRPC on `:4317`). | | **OTel Collector** | Receives OTLP from the engine and exports onward (Prometheus, Tempo, external backends). Managed by the OpenTelemetry Operator from a CR the chart creates. | | **Prometheus exporter (`:8889`)** | The Collector port that re-exposes LMCache *application* metrics (`lmcache_*`) for scraping. | | **Collector internals (`:8888`)** | The Collector's *own* metrics (`otelcol_*`), used to confirm data is flowing. | | **Tempo** | A trace storage backend. The chart can deploy a `TempoMonolithic`; you view traces in your own Grafana via a Tempo data source. | | **`remoteWrite`** | Pushing metrics to an external Prometheus-compatible endpoint (Grafana Cloud, Mimir, Thanos) instead of (or alongside) in-cluster scraping. | # Access & Security Source: https://docs.tensormesh.ai/v1.0.0/ui/access-and-security Who can reach the dashboard, how to put SSO in front of it, and why it's safe to run in production. The UI is safe to run in a production cluster: it is strictly read-only, holds no credentials of its own, and by default is reachable only from inside the cluster. *** ## Read-Only by Design The dashboard can look, but never touch: | It can | It cannot | | - | - | | See your engines and their status | Create, change, or delete anything | | See pods, nodes, and events | Change quotas, restart pods, or clear caches | | Read engine status and metrics through the Kubernetes API | Reach pods directly over the network | Because it can't act, the UI names the problem and the fix (Health → Issues & Fixes); you make the change with [`tmo-cli`](/v1.0.0/reference/cli), `kubectl`, or Helm. The UI gets its cluster access from Kubernetes at runtime: no credentials are baked into the image or the chart, and none ever reach the browser. *** ## How It's Reached By default the UI is private to the cluster. If you have cluster access, a port-forward is all you need: ```bash theme={null} kubectl -n tensormesh-platform port-forward svc/tmo-ui-tensormesh-operator-ui 8080:80 ``` For shared access, enable the ingress (`ingress.enabled=true`, with your own hosts and TLS) and put single sign-on in front of it (see below). *** ## Single Sign-On The UI has **no built-in login**. When you expose it to a team, sign-in happens at the ingress, the standard pattern for in-cluster dashboards. The ingress checks each request against your identity provider (Okta, Google, Entra, Keycloak, or any OIDC provider) and passes the signed-in identity to the UI, which shows **Signed in as** the authenticated user. * Works with any identity provider and any ingress controller. * Offboarding someone in your identity provider revokes their UI access immediately. There's no separate user list to maintain. *** ## Related * [Installation](/v1.0.0/ui/installation) * [Configuration](/v1.0.0/ui/configuration) * [Thresholds & Notifications](/v1.0.0/ui/thresholds-and-notifications) # Configuration Source: https://docs.tensormesh.ai/v1.0.0/ui/configuration Connect metric history, GPU telemetry, and additional clusters; the rest needs no setup. Most of the dashboard works the moment it's installed. Two things need connecting: your **Prometheus** (for the trend charts), and, on GPU clusters, **GPU telemetry** (for per-GPU detail): | What you see | Where it comes from | What you configure | | - | - | - | | Fleet, nodes, health, cache internals, quotas | The operator and engines | No setup needed | | Trend charts (hit rate, capacity, eviction, throughput over time) | Your Prometheus | `prometheus.url` + engine metrics scraping: [two steps](#trend-charts-connect-your-prometheus) | | Per-GPU cards (utilization, memory, hardware faults) | NVIDIA's GPU telemetry exporter | `dcgmService` (optional) | Everything is set through Helm values: | Value | What it does | Default | | - | - | - | | `engineNamespace` | The namespace where your engines run | *(the release namespace)* | | `prometheus.url` | Where the UI reads metric history from | *(unset)* | | `prometheus.service` | Alternative: reach Prometheus through Kubernetes instead of a direct URL (`service.namespace:port`) | *(unset)* | | `dcgmService` | GPU telemetry source (`service.namespace:port`) | *(unset)* | | `clusters` | Watch several clusters in one dashboard | *(unset, single cluster)* | | `ingress.enabled` | Expose the UI beyond port-forward | `false` | Cluster access itself needs no configuration: the UI picks it up automatically when running in-cluster. *** ## Trend Charts: Connect Your Prometheus Live numbers come straight from the engines; **history** comes from your Prometheus. Two things must both be true: the UI can reach a Prometheus, **and** the engine's metrics are arriving in it. **1. Point the UI at your Prometheus.** Find its service with `kubectl get svc -A | grep -i prometheus`. The name varies by install: ```yaml theme={null} prometheus: url: http://..svc:9090 # e.g. kube-prometheus-stack-prometheus.monitoring.svc (default kube-prometheus-stack) # e.g. kube-prom-stack-kube-prome-prometheus.monitoring.svc (name follows your Helm release) ``` **2. Make sure the engine's metrics reach that Prometheus.** How they get there depends on your setup: Prometheus scraping the engine directly, or the engine pushing through an OTel Collector. Both paths (and how to verify which one you're on) are covered in the operator's [Metrics](/v1.0.0/observability/metrics) page. The UI just reads what Prometheus already has. Install the community kube-prometheus-stack. It includes the Prometheus Operator that makes `ServiceMonitor`s work: ```bash theme={null} helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ -n monitoring --create-namespace ``` Then `prometheus.url` is `http://kube-prometheus-stack-prometheus.monitoring.svc:9090`, and the ServiceMonitor label above is `release: kube-prometheus-stack`. Charts fill once both halves are in place; if they stay empty while everything else works, see [Troubleshooting](/v1.0.0/ui/troubleshooting). *** ## Per-GPU Detail: Connect GPU Telemetry (GPU clusters) Per-GPU utilization, memory fill, and hardware-fault detail come from NVIDIA's DCGM exporter, standard on GPU clusters (it ships with the NVIDIA GPU Operator), but not part of Tensormesh. Find your exporter's service and point `dcgmService` at it (`service.namespace:port`): ```yaml theme={null} # find yours: kubectl get svc -A | grep -i dcgm dcgmService: dcgm-exporter.cw-exporters:9400 # CoreWeave clusters # dcgmService: dcgm-exporter.gpu-operator:9400 # e.g. an NVIDIA GPU Operator install ``` If it's unset or unreachable, the Fleet Map shows one row per node instead of per-GPU cards; everything else is unaffected. If your GPU cluster doesn't have the exporter at all, it ships with the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html): install that per NVIDIA's docs and the exporter service appears with it. ### GPU trend charts need Prometheus to scrape DCGM The live per-GPU cards fall back to `dcgmService` directly, but the over-time charts read history from Prometheus. Confirm DCGM is being scraped: query `DCGM_FI_DEV_GPU_UTIL` in Prometheus and expect series back. On some managed clusters the exporter ships a `ServiceMonitor` labelled for a different Prometheus (e.g. CoreWeave labels theirs `environment: grafana-monitoring`, while kube-prometheus-stack selects `release: `), so it returns zero series and the GPU trend charts stay empty. Fix it by adding a `ServiceMonitor` for the DCGM exporter's service with the label your Prometheus selects. *** ## Multiple Clusters One dashboard can watch several clusters: the fleet-wide pages merge every cluster's engines (labeled `· prod` / `· staging`), and per-engine pages scope to the right one. The cluster the UI runs **in** is automatic. Each *additional* cluster needs just two things: its `apiUrl` and a **read-only token**. There are two steps: **1. Put your clusters (with tokens) in a Secret:** ```bash theme={null} kubectl create secret generic tmo-ui-clusters -n tensormesh-platform \ --from-literal=TMO_CLUSTERS='[ {"name":"prod","apiUrl":"https://prod-api.example.com","token":""}, {"name":"staging","apiUrl":"https://staging-api.example.com","token":""} ]' ``` **2. Point your values file at it:** ```yaml theme={null} extraEnvFrom: - secretRef: name: tmo-ui-clusters ``` Reinstall, and every cluster shows up. Each entry can also set its own `namespace`, `prometheusUrl`, or `dcgmService`, and **`"insecure": true`** if that cluster's API uses a self-signed certificate. Run this **in that cluster**: it creates a read-only service account (the same access the UI uses) and prints a token: ```bash theme={null} kubectl apply -f - <<'EOF' apiVersion: v1 kind: ServiceAccount metadata: { name: tmo-ui-reader, namespace: tensormesh-platform } --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: { name: tmo-ui-reader } rules: - { apiGroups: ["lmcache.lmcache.ai"], resources: ["lmcacheengines", "lmcachecoordinators"], verbs: ["get", "list", "watch"] } - { apiGroups: [""], resources: ["pods", "namespaces", "nodes", "events"], verbs: ["get", "list"] } - { apiGroups: [""], resources: ["pods/proxy", "services/proxy"], verbs: ["get"] } --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: { name: tmo-ui-reader } roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: tmo-ui-reader } subjects: - { kind: ServiceAccount, name: tmo-ui-reader, namespace: tensormesh-platform } EOF kubectl -n tensormesh-platform create token tmo-ui-reader --duration=8760h ``` ### Sharing one token across clusters To skip per-cluster tokens, list the clusters in your values file and put a single `TMO_KUBE_TOKEN` in the Secret; every cluster inherits it. ```yaml theme={null} # values.yaml clusters: - { name: prod, apiUrl: https://prod-api.example.com } - { name: dev, apiUrl: https://dev-api.example.com } # add `insecure: true` to any entry whose API uses a self-signed certificate extraEnvFrom: - secretRef: { name: tmo-ui-clusters } ``` ```bash theme={null} kubectl create secret generic tmo-ui-clusters -n tensormesh-platform \ --from-literal=TMO_KUBE_TOKEN='' ``` *** ## Related * [Troubleshooting & FAQ](/v1.0.0/ui/troubleshooting) * [Monitoring Your Fleet](/v1.0.0/ui/monitoring-your-fleet) * [Operator Observability](/v1.0.0/observability) # Installation Source: https://docs.tensormesh.ai/v1.0.0/ui/installation Install the Operator UI into your cluster, next to the operator. The UI installs the same way as the operator, from the same private registry — the access token you already have works for both. If you don't have one, request it from the Tensormesh team. Install the operator and at least one `LMCacheEngine` first: the UI is a read-only dashboard over your engines, so it installs fine on an empty cluster but shows "No LMCacheEngines found" until engines exist. *** ## Prerequisites * Helm `3.8` or newer (`helm version`) * A Kubernetes 1.28+ cluster with [Tensormesh Platform](/v1.0.0/installation/helm) installed, plus at least one `LMCacheEngine` * **Prometheus running in the cluster** (with the Prometheus Operator), and the **engine's metrics scraped into it**. This powers every trend chart and takes two small steps, covered in [Configuration](/v1.0.0/ui/configuration#trend-charts-connect-your-prometheus). * **The NVIDIA DCGM exporter, if you run GPUs**: standard on GPU clusters (it ships with the NVIDIA GPU Operator, and isn't part of Tensormesh). It drives the per-GPU utilization, memory, and hardware-fault cards. * **An access token from the Tensormesh team** — the operator's token works *** ## Versions Pin an explicit chart version — don't use `latest`. Like the [platform compatibility matrix](/v1.0.0/installation/compatibility), each row is a matched set: run the UI chart against the platform versions listed with it. | UI version | Platform Helm chart | Status | As of | | - | - | :-: | - | | `1.0.0` | `1.0.0` | ✅ Supported | 2026-09-17 | *** ## Install ```bash theme={null} # username isn't validated for token auth — any non-empty value works echo '' | helm registry login artifacts.tensormesh.ai -u tensormesh --password-stdin ``` If you already logged in to install the operator, skip this step. ```yaml ui-values.yaml theme={null} engineNamespace: tensormesh-platform # where your engines run prometheus: url: http://kube-prom-stack-kube-prome-prometheus.monitoring.svc:9090 # yours: kubectl get svc -A | grep -i prometheus dcgmService: dcgm-exporter.cw-exporters:9400 # GPU cards — yours: kubectl get svc -A | grep -i dcgm imagePullSecrets: - name: tensormesh-registry # the operator's pull secret works here too ``` The UI picks up cluster access automatically once it's running inside the cluster. You never set a kube API URL or token. See [Configuration](/v1.0.0/ui/configuration) for how each value maps to what you see, plus multiple clusters. ```bash theme={null} helm upgrade --install tmo-ui \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator-ui \ --version 1.0.0 -n tensormesh-platform --create-namespace \ -f ui-values.yaml ``` The [table above](#versions) lists the current chart version and the operator versions it works with. Pin an explicit version — don't use `latest`. *** ## Verify ```bash theme={null} helm test tmo-ui -n tensormesh-platform ``` The bundled test confirms the UI is up and serving. If anything looks off once you open the dashboard, see [Troubleshooting & FAQ](/v1.0.0/ui/troubleshooting). *** ## Open the Dashboard The install doesn't expose the UI outside the cluster. To open it, port-forward the service: ```bash theme={null} kubectl -n tensormesh-platform port-forward svc/tmo-ui-tensormesh-operator-ui 8080:80 # open http://localhost:8080 ``` For shared team access, enable the ingress (`ingress.enabled=true` with your own hosts and TLS). The UI has no built-in login, so if you expose it beyond port-forward, put SSO in front at the ingress (see [Access & Security](/v1.0.0/ui/access-and-security)). *** ## What Gets Installed A single dashboard pod and a service, nothing on your GPU nodes. Its cluster access is **strictly read-only**: it can look at engines, pods, and metrics, and cannot modify anything. *** ## Upgrade Re-run the install with a new `--version` (same values file): ```bash theme={null} helm upgrade --install tmo-ui \ oci://artifacts.tensormesh.ai/tensormesh-production/charts/tensormesh-operator-ui \ --version -n tensormesh-platform -f ui-values.yaml ``` Roll back if needed: ```bash theme={null} helm history tmo-ui -n tensormesh-platform helm rollback tmo-ui -n tensormesh-platform ``` *** ## Uninstall ```bash theme={null} helm uninstall tmo-ui -n tensormesh-platform ``` The UI keeps no cluster state, so nothing is left behind. *** ## Next Steps Connect metric history, GPU telemetry, and multiple clusters. Triage what needs attention, drill into nodes and GPUs, and track cache and performance trends. # Overview Source: https://docs.tensormesh.ai/v1.0.0/ui/introduction The Tensormesh Operator UI — a dashboard for your cache fleets: health, capacity, and GPU visibility. The Tensormesh Operator UI is a live view of your cache fleets: every engine, every GPU node, and the storage tiers behind them. It installs into the same cluster as the operator. The UI is read-only: it shows what's wrong and which setting to adjust, but you make every change yourself with [`tmo-cli`](/v1.0.0/reference/cli), Helm, or the engine configuration. See [Access & Security](/v1.0.0/ui/access-and-security) for details. *** ## What You Can See Every engine and node, its readiness and health, and what needs attention. How full each cache tier is, whether storage backends are healthy, and early warning of problems. Hit rate, eviction, capacity, and throughput trends — confirm a fix worked or catch a problem building. Per-workload cache budgets and who's over them, plus per-GPU utilization and hardware health. *** ## Next Steps Install the UI next to your operator. Connect metric history, GPU telemetry, and additional clusters. Triage what needs attention, drill into nodes and GPUs, and track cache and performance trends. Decide what counts as a problem, and get notified when it happens. # Metrics Source: https://docs.tensormesh.ai/v1.0.0/ui/metrics Every chart on the Operator UI Metrics page. The **Metrics** screen shows trends over time for one engine. Pick an engine with the filter in the page header and a time range with the range selector. Charts are grouped into four tabs. Every chart reads a signal from the platform [Metrics reference](/v1.0.0/observability/metrics); this page maps each chart on the screen to the metric behind it. It needs your Prometheus connected (see [Configuration](/v1.0.0/ui/configuration)), and the GPU charts need the DCGM exporter. ## Serving GPU and host load, and how much of each request the cache serves. | Chart | What it shows | Metric | | - | - | - | | GPU Utilization | Per-node GPU busy %, from the DCGM exporter | `DCGM_FI_DEV_GPU_UTIL` | | Host CPU Utilization | Per-node CPU busy % | `node_cpu_utilisation` (rate) | | Cache Hit Rate | Share of requested tokens served from cache instead of recomputed | `lmcache_mp_lookup_hit_tokens_total` / `lmcache_mp_lookup_requested_tokens_total` | ## Cache Capacity How full each tier is, and how fast entries turn over. | Chart | What it shows | Metric | | - | - | - | | L1 Fill Per Node | Host-DRAM (L1) fill per node | `lmcache_mp_l1_usage_ratio` (bytes: `lmcache_mp_l1_memory_usage_bytes`) | | GPU HBM Fill | GPU memory used vs total, per node | `DCGM_FI_DEV_FB_USED` / (`DCGM_FI_DEV_FB_USED` + `DCGM_FI_DEV_FB_FREE`) | | L1 Eviction Rate | Chunks evicted from L1 under pressure | `lmcache_mp_l1_evicted_chunks_total` | | L2 Backend Fill / Stored | External-storage (L2) bytes held, or objects stored | `lmcache_mp_l2_usage_bytes`, `lmcache_mp_l2_store_submitted_objects_chunks_total` | ## Cache Activity Reads vs writes, and traffic between tiers. | Chart | What it shows | Metric | | - | - | - | | L1 Activity — Read vs Write | L1 reads (reuse) vs writes (ingest) | `lmcache_mp_l1_read_chunks_total` / `lmcache_mp_l1_write_chunks_total` | | L2 Activity — Store vs Load | L2 store vs load requests | `lmcache_mp_l2_store_completed_requests_total` / `lmcache_mp_l2_load_completed_requests_total` | | L0↔L1 Throughput — Store vs Load | GPU↔DRAM transfer rate (GB/s) | `lmcache_mp_l0_l1_store_throughput_GB_per_second` / `lmcache_mp_l0_l1_load_throughput_GB_per_second` | | L2 Throughput — Store vs Load | External-storage transfer rate (GB/s) | `lmcache_mp_l2_store_throughput_GB_per_second` / `lmcache_mp_l2_load_throughput_GB_per_second` | | Retrieved Chunk Volume | Chunks loaded back into service | `lmcache_mp_num_chunks_loaded_total` | | Prefetch Failure Rate | Prefetches that failed (miss or L1 pressure) | `lmcache_mp_l2_prefetch_failure_chunks_total` | ## Quotas Per-workload cache budget and usage — which workloads (keyed by tenant `cache_salt`) are within or over their configured budget. ## Related * [Monitoring Your Fleet](/v1.0.0/ui/monitoring-your-fleet) * [Thresholds & Notifications](/v1.0.0/ui/thresholds-and-notifications) * [Metrics reference](/v1.0.0/observability/metrics) # Monitoring Your Fleet Source: https://docs.tensormesh.ai/v1.0.0/ui/monitoring-your-fleet Overview, Fleet Map, Metrics, Health, and Thresholds — what each screen shows and when to use it. Here's what each screen is for: | Screen | What it's for | | - | - | | **Overview** | The landing page — what needs attention right now | | **Fleet Map** | Drill down from engines to nodes to individual GPUs | | **Metrics** | Trends over time — utilization, cache fill, activity, quotas | | **Health** | Inside the engine — the cache pipeline, events, and suggested fixes | | **Thresholds** | Decide what counts as a problem | Overview and Fleet Map cover the whole fleet (and every cluster, in a [multi-cluster](/v1.0.0/ui/configuration#multiple-clusters) setup); the other screens focus on one engine, chosen with the filter in the page header. *** ## Overview The landing page: overall fleet health, per-engine cards, and two lists that name what needs attention. Each entry links straight to the affected node. * **Alerts** — facts your cluster is reporting: an engine not fully ready, a failing health check (with the failing nodes named), a storage backend down. * **Issues** — *your* [thresholds](/v1.0.0/ui/thresholds-and-notifications) applied to live values, each naming the worst offender. *** ## Fleet Map Every engine, node, and GPU in one view. Pick a lens (Health, Utilization, or Capacity) and the whole map colors by it, so the affected node stands out. Selecting a node shows its key numbers, its GPUs (when GPU telemetry is connected), and its recent trends. Nodes can be nicknamed, for example by the model they serve. *** ## Metrics Per-engine trends over time across four tabs: Serving, Cache Capacity, Cache Activity, and Quotas. See [Metrics](/v1.0.0/ui/metrics) for every chart and the metric behind it. *** ## Health What's happening inside the engine: health tiles, the cache pipeline, and the event log. Issues & Fixes turns anything failing into a problem card with a suggested fix that matches the cause (a storage-backend failure points at the backend, not "restart the pod"). Health here is the engine's own self-assessment, not just pod status, so it catches problems standard Kubernetes tooling can't see, like a storage connection silently disabled after repeated failures while the pod still looks fine. *** ## Thresholds Where you decide what counts as degraded vs. critical: driving the colors, the Issues list, and notifications. Covered in [Thresholds & Notifications](/v1.0.0/ui/thresholds-and-notifications). *** ## Related * [Thresholds & Notifications](/v1.0.0/ui/thresholds-and-notifications) * [Multi-Tenancy](/v1.0.0/management/multi-tenancy) * [Troubleshooting & FAQ](/v1.0.0/ui/troubleshooting) * [CLI Reference](/v1.0.0/reference/cli) # Thresholds & Notifications Source: https://docs.tensormesh.ai/v1.0.0/ui/thresholds-and-notifications Decide what counts as a problem, and get notified the moment it happens. Thresholds turn live numbers into severity. Each metric on the Thresholds page has a *degraded* (warn) and a *critical* band: | Metric | Default warn / critical | | - | - | | **L1 (host DRAM) cache fill** | 78% / 90% | | **GPU memory fill** | 96% / 99% | | **L2 (external) storage fill** | 70% / 85% | | **Pod restarts** | 0 / 2 | | **Prefetch failures** | 1 / 10 per second | | **Evictions** | 100 / 300 per second | Reset to Defaults restores these baselines. High GPU *compute* utilization is intentionally never alarmed: busy GPUs are the goal, not a fault. Thresholds drive three things: the severity colors across the dashboard, the Issues list on Overview (each issue linking to the worst offender), and notifications. *** ## Scope Set global defaults, then override per engine where needed. An engine inherits the global band for anything it doesn't override, and each level has a one-click revert. Thresholds, node nicknames, and the notification toggle are saved in your browser, not on the cluster: the UI never writes to your cluster. They change what the dashboard flags, not how the engine behaves. *** ## Notifications The dashboard refreshes every 30 seconds. When an issue turns critical, you get a notification in the dashboard, and optionally on your desktop (enable System alerts in the sidebar). Clicking it takes you straight to the affected node. Three rules keep notifications worth reading: * Only new problems notify: opening the page doesn't re-announce ones that already existed. * The same problem won't notify again for 5 minutes, so a metric bouncing in and out of critical doesn't spam you. * Warning-level problems color the dashboard but never notify: only critical ones do. *** ## Related * [Monitoring Your Fleet](/v1.0.0/ui/monitoring-your-fleet) * [Troubleshooting & FAQ](/v1.0.0/ui/troubleshooting) # Troubleshooting & FAQ Source: https://docs.tensormesh.ai/v1.0.0/ui/troubleshooting Common setup problems and short answers to frequent questions. ## Troubleshooting The image can't be pulled: the registry pull secret is missing from the namespace, or its token can't read the package. Confirm the `imagePullSecrets` name in your values matches a secret that exists in the namespace, and that its token has `read:packages` (the operator's pull secret works). Recreate it with: ```bash theme={null} kubectl create secret docker-registry tensormesh-registry -n tensormesh-platform \ --docker-server=artifacts.tensormesh.ai --docker-username= --docker-password= ``` The install doesn't expose the UI outside the cluster: the pod is fine, it's just not reachable yet. Start a port-forward and **keep the terminal open**: ```bash theme={null} kubectl -n tensormesh-platform port-forward svc/tmo-ui-tensormesh-operator-ui 8080:80 ``` then open [http://localhost:8080](http://localhost:8080). For shared team access, enable the ingress behind SSO instead. See [Access & Security](/v1.0.0/ui/access-and-security). `engineNamespace` doesn't match the namespace where your engines actually run. The engines or GPUs are idle: utilization and hit rate only move under inference traffic. (GPU *memory* stays allocated, so it can be non-zero while utilization reads 0.) If there *is* traffic and charts are still flat, check that `prometheus.url` is right. The engine metrics aren't in the Prometheus the UI reads. The most common cause: the engine's metrics endpoint was never enabled (it's off by default). Set `engine.spec.prometheus.enabled: true` and add the `ServiceMonitor` so Prometheus scrapes it, both shown in [Configuration](/v1.0.0/ui/configuration#trend-charts-connect-your-prometheus). Also confirm `prometheus.url` points at the right Prometheus (`kubectl get svc -A | grep -i prometheus`). Fleet-wide totals are still correct; the UI just can't tell which node each metric came from, usually because your metrics collector runs as one shared service instead of per-node. Ask the Tensormesh team for the collector setting that restores per-node attribution. GPU telemetry isn't connected: set `dcgmService` to your exporter's service (find it with `kubectl get svc -A | grep -i dcgm`; see [Configuration](/v1.0.0/ui/configuration)). Without it, node cards show one row per node; everything else is unaffected. The over-time GPU charts read history from Prometheus, so Prometheus has to be **scraping DCGM**; the live cards only need `dcgmService`. Check it: query `DCGM_FI_DEV_GPU_UTIL` in Prometheus and expect series back. If it's empty, the exporter's `ServiceMonitor` is often labelled for a different Prometheus (some managed clusters, e.g. CoreWeave, use `environment: grafana-monitoring` instead of `release: `). Add a `ServiceMonitor` for the DCGM service with the label your Prometheus selects. See [Configuration](/v1.0.0/ui/configuration#per-gpu-detail-connect-gpu-telemetry-gpu-clusters). Cluster access is automatic only for the cluster the UI runs **in**. Each extra `clusters` entry needs its own `apiUrl` and an access token, supplied through a Kubernetes Secret wired in with the chart's `extraEnvFrom`, not in your values file. See [Configuration](/v1.0.0/ui/configuration#multiple-clusters). *** ## FAQ No. The UI observes only, by design. It names the failing component and the fix (Health → Issues & Fixes); you act via [`tmo-cli`](/v1.0.0/reference/cli), `kubectl`, or Helm. A handful of lightweight read requests per open page every 30 seconds, paused when the tab is in the background. Nothing runs on your GPU nodes. It depends on what's unreachable. If the **control plane** can't be reached, the monitoring pages (Overview, Fleet Map, Metrics, Health) show an explicit error (*"The control plane didn't respond"*) with a **Retry** button, never zeros or mock data. If the cluster is simply empty or idle, you get real zeros and a "No LMCacheEngines found" banner. If only metric **history** is missing (Prometheus not connected), the trend charts are empty while everything else works. *** ## Related * [Configuration](/v1.0.0/ui/configuration) * [Installation](/v1.0.0/ui/installation) * [Operator Installation Troubleshooting](/v1.0.0/installation/troubleshooting)