Skip to main content
This page covers the most common install and runtime issues. For everything else, gather the diagnostic bundle described at the bottom and open a support thread.

​
Contents

Install & scheduling Uninstall & cleanup Metrics & observability Getting help

​
Engine pod is Pending

kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine
# NAME              READY   STATUS    RESTARTS   AGE
# tmo-engine-xxxxx  0/1     Pending   0          2m
Almost always a node-selector mismatch. The engine DaemonSet selects nodes labeled nvidia.com/gpu.present=true by default — if no node carries that label, no pod is scheduled. 
# What labels do my nodes carry?
kubectl get nodes -L nvidia.com/gpu.present

# Add the label to a GPU node manually
kubectl label node <node-name> nvidia.com/gpu.present=true
If you use a different label in your environment, override the selector instead:
my-values.yaml
engine:
  spec:
    nodeSelector:
      mycompany.io/gpu-tier: "premium"

​
Pod stuck in ContainerCreating (CDI)

kubectl get pods -n tensormesh-operator -o wide
# NAME              READY   STATUS              RESTARTS   AGE
# tmo-engine-xxxxx  0/1     ContainerCreating   0          5m
The pod scheduled onto a GPU node (so it’s past Pending) but the container can’t be created. The usual cause on NVIDIA GPU clusters is that the Container Device Interface (CDI) is disabled. The NVIDIA GPU Operator injects GPUs via CDI, and before GPU Operator v25.10.0 it defaulted to cdi.enabled: false — with no obvious error, the pod just hangs in ContainerCreating. Enable CDI in the GPU ClusterPolicy: 
oc patch clusterpolicy gpu-cluster-policy --type=merge -p '{
  "spec": { "cdi": { "enabled": true, "default": true } }
}'
GPU Operator v25.10.0+ enables CDI by default — no patch needed.
Confirm it’s CDI and not something else: kubectl describe pod <pod> — a CDI problem shows a Failed to create … CDI device injection / unresolvable CDI devices event. A missing-image event (ImagePullBackOff) or volume-mount error points elsewhere.

​
Engine pod CreateContainerError on OpenShift

kubectl describe pod -n tensormesh-operator <engine-pod>
# Events:
#   Warning  Failed  ... unable to start container: hostIPC is not allowed
The pod is requesting hostIPC but its ServiceAccount is not bound to an SCC that permits it. Add openshift.enabled: true to your values file and upgrade: 
helm upgrade tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version 0.4.5 -n tensormesh-operator \
  -f values-openshift.yaml
See Install on OpenShift for the full overlay and for the “bring your own SA” pattern.

​
Operator pod ImagePullBackOff

kubectl describe pod -n tensormesh-operator <operator-pod>
# Events:
#   Warning  Failed  ... Failed to pull image: unauthorized
The image lives in a registry your cluster cannot authenticate to. Create an image pull secret and reference it from the chart: 
kubectl create secret docker-registry tmo-pull \
  -n tensormesh-operator \
  --docker-server=<registry> \
  --docker-username=<user> \
  --docker-password=<password>
my-values.yaml
operator:
  image:
    pullSecrets:
      - tmo-pull
For air-gapped clusters, mirror the operator image into your internal registry and override operator.image.repository and operator.image.tag.

​
Install fails with CRD ... exists and cannot be imported

helm install tensormesh-operator oci://ghcr.io/.../tensormesh-operator ...
# Error: INSTALLATION FAILED: unable to continue with install:
# CustomResourceDefinition "lmcacheengines.lmcache.lmcache.ai" in namespace "" exists
# and cannot be imported into the current release: invalid ownership metadata;
# annotation validation error: key "meta.helm.sh/release-name" must equal "tensormesh-operator":
# current value is "<other-release>"
The LMCacheEngine CRD is cluster-scoped — there is only one of it. A previous install under a different release name (or namespace) annotated it as its own; Helm refuses to silently transfer ownership to your new release. This is a Helm-wide safety check, not specific to this chart — cert-manager, kube-prometheus-stack, and Argo CD all behave identically.
Before doing either fix below, check whether the prior release is still active: helm list -A. If you see a real release that owns the CRD, leave it alone — running a second operator against the same CRD will fight reconciliation. Use the existing release (helm upgrade <its-name> ...) instead.
Use this when the prior release is gone (was uninstalled, or is on a cluster you don’t own anymore) but the CRD survived because of helm.sh/resource-policy: keep. 
kubectl annotate crd lmcacheengines.lmcache.lmcache.ai \
  meta.helm.sh/release-name=<new-release> \
  meta.helm.sh/release-namespace=<new-namespace> --overwrite
kubectl label crd lmcacheengines.lmcache.lmcache.ai \
  app.kubernetes.io/managed-by=Helm --overwrite

# Re-run the install
helm install <new-release> oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version <version> \
  --namespace <new-namespace> --create-namespace
Existing LMCacheEngine CRs survive — only the CRD’s ownership annotations change.
The LMCacheEngine CRD is cluster-scoped — one of it exists for the whole cluster. Helm tracks which release owns each resource via three stamps:
FieldPurpose
annotation meta.helm.sh/release-namewhich release owns this
annotation meta.helm.sh/release-namespacein which namespace
label app.kubernetes.io/managed-by=Helm”some Helm release manages this”
The adoption commands rewrite those stamps so your new release becomes the recognized owner. The CRD’s schema, any existing LMCacheEngine CRs, and everything in any namespace are unchanged — it’s a pure metadata edit, fully reversible by running the same commands with the old values.

​
Option B — delete the CRD and let the install recreate it

Safe only when no LMCacheEngine CRs exist anywhere on the cluster. Confirm first: 
kubectl get lmcacheengines -A         # must return "No resources found"
kubectl delete crd lmcacheengines.lmcache.lmcache.ai

# Re-run the install — it will create the CRD fresh with the right ownership
helm install <new-release> oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version <version> \
  --namespace <new-namespace> --create-namespace
Deleting the CRD cascade-deletes every LMCacheEngine CR on the cluster. Use Option A if you have any CRs you want to keep.

​
helm uninstall hangs

The pre-delete hook is waiting for the LMCacheEngine CR’s finalizer to drain. Usually this means the operator was already gone when the CR was deleted, so nobody is running the finalizer. 
# 1. Confirm the CR is stuck
kubectl get lmcacheengines -A

# 2. Inspect operator logs (if it's still around)
kubectl logs -n tensormesh-operator \
  deploy/tensormesh-operator

# 3. Force-clear the finalizer
kubectl patch lmcacheengine -n <ns> <name> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

# 4. Re-run uninstall
helm uninstall tensormesh-operator -n tensormesh-operator
Removing finalizers bypasses any cleanup the operator would have performed on backing resources. Use only when the operator is definitively gone.
If helm list -A shows no release at all (the operator and the release record are already gone, not just a stuck uninstall), this quick fix doesn’t apply — follow Cleaning up an inconsistent or orphaned install for the full manual teardown.

​
Metrics endpoint returns 401

The controller-runtime auth filter is on by default. Either disable it for scrapers that cannot authenticate:
my-values.yaml
operator:
  metrics:
    secure: false
Or grant the scraping ServiceAccount the system:auth-delegator ClusterRole so it can authenticate with the kube-apiserver TokenReview API: 
kubectl create clusterrolebinding metrics-auth \
  --clusterrole=system:auth-delegator \
  --serviceaccount=<scraper-namespace>:<scraper-sa>

​
CRD survives uninstall (by design)

The LMCacheEngine CRD is annotated helm.sh/resource-policy: keep, so helm uninstall leaves it (and any remaining CRs) in place. This is the same convention used by cert-manager and kube-prometheus-stack — it prevents data loss when you upgrade in place. To remove the CRD and cascade-delete any remaining CRs: 
kubectl delete crd lmcacheengines.lmcache.lmcache.ai

​
Cleaning up an inconsistent or orphaned install

Use this when the normal helm uninstall path doesn’t apply — for example:
  • helm list -A shows no release, but the engine pod, LMCacheEngine CR, services, and CRD are still present (the release record was removed but its resources weren’t).
  • helm uninstall hangs, or kubectl delete namespace gets stuck in Terminating.
The root cause is almost always the same: the LMCacheEngine CR carries a finalizer (lmcache.ai/cleanup) that only the operator can drain. If the operator is gone — or was removed before the CR — nothing runs the finalizer, so the CR can’t be deleted, and anything waiting on it (a namespace delete, an uninstall hook) blocks indefinitely. Clean up manually, in this order:
1

Clear the stuck CR's finalizer

The operator isn’t coming back to drain it, so remove the finalizer yourself:
kubectl patch lmcacheengine <name> -n <namespace> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge
(Find the name with kubectl get lmcacheengine -A.) This is safe only when the operator is definitively gone — it skips the cleanup the operator would have done, which is fine when you’re tearing everything down anyway.
2

Delete the namespace

With the finalizer cleared, this completes instead of hanging — it sweeps the DaemonSet, pods, services, ConfigMaps, ServiceAccounts, and RoleBindings.
kubectl delete namespace <namespace>
3

Delete the CRD

Cluster-scoped, so the namespace delete didn’t touch it. Removing it gives a clean slate (and avoids the ownership conflict on the next install).
kubectl delete crd lmcacheengines.lmcache.lmcache.ai
4

Sweep orphaned cluster-scoped RBAC

The operator’s ClusterRole/ClusterRoleBinding are cluster-scoped and survive a namespace delete. Remove any leftovers:
kubectl get clusterrole,clusterrolebinding | grep -i tensormesh
# delete anything it lists, e.g.:
kubectl delete clusterrole,clusterrolebinding -l app.kubernetes.io/name=tensormesh-operator
Order matters. Clear the finalizer before deleting the namespace. If you delete the namespace first, it gets stuck in Terminating (blocked on the un-drainable CR) until you clear the finalizer anyway.
After these four steps, kubectl get ns <namespace> and kubectl get crd lmcacheengines.lmcache.lmcache.ai should both return NotFound — a true clean slate.

​
Gathering a diagnostic bundle

When opening a support thread, include: 
# Versions
helm version
kubectl version
kubectl get nodes -o wide

# Release state
helm list -A
helm get values tensormesh-operator -n tensormesh-operator
helm get manifest tensormesh-operator -n tensormesh-operator

# Live state
kubectl get all -n tensormesh-operator
kubectl get lmcacheengines -A -o yaml

# Logs
kubectl logs -n tensormesh-operator \
  deploy/tensormesh-operator --tail=500
kubectl logs -n tensormesh-operator \
  -l app.kubernetes.io/component=cache-engine --tail=500

# Events
kubectl get events -n tensormesh-operator --sort-by=.lastTimestamp
Rare hashing fallback: if LMCache logs that it is using builtin hash instead of the default blake3 path, set PYTHONHASHSEED=0 in the vLLM pod and restart it. This is not the normal MP requirement and it is unrelated to MQ payload-shape mismatches.

​
Next steps

Helm install reference

Re-check the install steps against the chart’s documented values.

Getting Started

Re-validate prerequisites if multiple issues are stacking up.