Troubleshooting

This page covers the most common install and runtime issues. For everything else, gather the diagnostic bundle described at the bottom and open a support thread.

Install & scheduling

Engine pod is Pending — node-selector / GPU label mismatch
Pod stuck in ContainerCreating — CDI disabled (older NVIDIA GPU Operator)
Engine pod CreateContainerError on OpenShift — missing privileged SCC
Operator pod ImagePullBackOff — registry auth / air-gap
Install fails with CRD ... exists and cannot be imported — ownership conflict from a prior install

Uninstall & cleanup

helm uninstall hangs — stuck CR finalizer
CRD survives uninstall (by design)
Cleaning up an inconsistent or orphaned install — no release record / orphaned resources

Metrics & observability

Metrics endpoint returns 401
OTel Collector / traces not flowing → see Observability → Troubleshooting

Getting help

Gathering a diagnostic bundle

Engine pod is `Pending`

kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine
# NAME              READY   STATUS    RESTARTS   AGE
# tmo-engine-xxxxx  0/1     Pending   0          2m

Almost always a node-selector mismatch. The engine DaemonSet selects nodes labeled nvidia.com/gpu.present=true by default — if no node carries that label, no pod is scheduled.

# What labels do my nodes carry?
kubectl get nodes -L nvidia.com/gpu.present

# Add the label to a GPU node manually
kubectl label node <node-name> nvidia.com/gpu.present=true

If you use a different label in your environment, override the selector instead:

my-values.yaml

engine:
  spec:
    nodeSelector:
      mycompany.io/gpu-tier: "premium"

Pod stuck in `ContainerCreating` (CDI)

kubectl get pods -n tensormesh-operator -o wide
# NAME              READY   STATUS              RESTARTS   AGE
# tmo-engine-xxxxx  0/1     ContainerCreating   0          5m

The pod scheduled onto a GPU node (so it’s past Pending) but the container can’t be created. The usual cause on NVIDIA GPU clusters is that the Container Device Interface (CDI) is disabled. The NVIDIA GPU Operator injects GPUs via CDI, and before GPU Operator v25.10.0 it defaulted to cdi.enabled: false — with no obvious error, the pod just hangs in ContainerCreating. Enable CDI in the GPU ClusterPolicy:

oc patch clusterpolicy gpu-cluster-policy --type=merge -p '{
  "spec": { "cdi": { "enabled": true, "default": true } }
}'

GPU Operator v25.10.0+ enables CDI by default — no patch needed.

Confirm it’s CDI and not something else: kubectl describe pod <pod> — a CDI problem shows a Failed to create … CDI device injection / unresolvable CDI devices event. A missing-image event (ImagePullBackOff) or volume-mount error points elsewhere.

Engine pod `CreateContainerError` on OpenShift

kubectl describe pod -n tensormesh-operator <engine-pod>
# Events:
#   Warning  Failed  ... unable to start container: hostIPC is not allowed

The pod is requesting hostIPC but its ServiceAccount is not bound to an SCC that permits it. Add openshift.enabled: true to your values file and upgrade:

helm upgrade tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version 0.4.5 -n tensormesh-operator \
  -f values-openshift.yaml

See Install on OpenShift for the full overlay and for the “bring your own SA” pattern.

Operator pod `ImagePullBackOff`

kubectl describe pod -n tensormesh-operator <operator-pod>
# Events:
#   Warning  Failed  ... Failed to pull image: unauthorized

The image lives in a registry your cluster cannot authenticate to. Create an image pull secret and reference it from the chart:

kubectl create secret docker-registry tmo-pull \
  -n tensormesh-operator \
  --docker-server=<registry> \
  --docker-username=<user> \
  --docker-password=<password>

my-values.yaml

operator:
  image:
    pullSecrets:
      - tmo-pull

For air-gapped clusters, mirror the operator image into your internal registry and override operator.image.repository and operator.image.tag.

Install fails with `CRD ... exists and cannot be imported`

helm install tensormesh-operator oci://ghcr.io/.../tensormesh-operator ...
# Error: INSTALLATION FAILED: unable to continue with install:
# CustomResourceDefinition "lmcacheengines.lmcache.lmcache.ai" in namespace "" exists
# and cannot be imported into the current release: invalid ownership metadata;
# annotation validation error: key "meta.helm.sh/release-name" must equal "tensormesh-operator":
# current value is "<other-release>"

The LMCacheEngine CRD is cluster-scoped — there is only one of it. A previous install under a different release name (or namespace) annotated it as its own; Helm refuses to silently transfer ownership to your new release. This is a Helm-wide safety check, not specific to this chart — cert-manager, kube-prometheus-stack, and Argo CD all behave identically.

Before doing either fix below, check whether the prior release is still active: helm list -A. If you see a real release that owns the CRD, leave it alone — running a second operator against the same CRD will fight reconciliation. Use the existing release (helm upgrade <its-name> ...) instead.

Option A — adopt the CRD into your new release (recommended)

Use this when the prior release is gone (was uninstalled, or is on a cluster you don’t own anymore) but the CRD survived because of helm.sh/resource-policy: keep.

kubectl annotate crd lmcacheengines.lmcache.lmcache.ai \
  meta.helm.sh/release-name=<new-release> \
  meta.helm.sh/release-namespace=<new-namespace> --overwrite
kubectl label crd lmcacheengines.lmcache.lmcache.ai \
  app.kubernetes.io/managed-by=Helm --overwrite

# Re-run the install
helm install <new-release> oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version <version> \
  --namespace <new-namespace> --create-namespace

Existing LMCacheEngine CRs survive — only the CRD’s ownership annotations change.

What do these commands actually do?

The LMCacheEngine CRD is cluster-scoped — one of it exists for the whole cluster. Helm tracks which release owns each resource via three stamps:

Field	Purpose
annotation `meta.helm.sh/release-name`	which release owns this
annotation `meta.helm.sh/release-namespace`	in which namespace
label `app.kubernetes.io/managed-by=Helm`	”some Helm release manages this”

The adoption commands rewrite those stamps so your new release becomes the recognized owner. The CRD’s schema, any existing LMCacheEngine CRs, and everything in any namespace are unchanged — it’s a pure metadata edit, fully reversible by running the same commands with the old values.

Option B — delete the CRD and let the install recreate it

Safe only when no LMCacheEngine CRs exist anywhere on the cluster. Confirm first:

kubectl get lmcacheengines -A         # must return "No resources found"
kubectl delete crd lmcacheengines.lmcache.lmcache.ai

# Re-run the install — it will create the CRD fresh with the right ownership
helm install <new-release> oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version <version> \
  --namespace <new-namespace> --create-namespace

Deleting the CRD cascade-deletes every LMCacheEngine CR on the cluster. Use Option A if you have any CRs you want to keep.

`helm uninstall` hangs

The pre-delete hook is waiting for the LMCacheEngine CR’s finalizer to drain. Usually this means the operator was already gone when the CR was deleted, so nobody is running the finalizer.

# 1. Confirm the CR is stuck
kubectl get lmcacheengines -A

# 2. Inspect operator logs (if it's still around)
kubectl logs -n tensormesh-operator \
  deploy/tensormesh-operator

# 3. Force-clear the finalizer
kubectl patch lmcacheengine -n <ns> <name> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

# 4. Re-run uninstall
helm uninstall tensormesh-operator -n tensormesh-operator

Removing finalizers bypasses any cleanup the operator would have performed on backing resources. Use only when the operator is definitively gone.

If helm list -A shows no release at all (the operator and the release record are already gone, not just a stuck uninstall), this quick fix doesn’t apply — follow Cleaning up an inconsistent or orphaned install for the full manual teardown.

Metrics endpoint returns `401`

The controller-runtime auth filter is on by default. Either disable it for scrapers that cannot authenticate:

my-values.yaml

operator:
  metrics:
    secure: false

Or grant the scraping ServiceAccount the system:auth-delegator ClusterRole so it can authenticate with the kube-apiserver TokenReview API:

kubectl create clusterrolebinding metrics-auth \
  --clusterrole=system:auth-delegator \
  --serviceaccount=<scraper-namespace>:<scraper-sa>

CRD survives uninstall (by design)

The LMCacheEngine CRD is annotated helm.sh/resource-policy: keep, so helm uninstall leaves it (and any remaining CRs) in place. This is the same convention used by cert-manager and kube-prometheus-stack — it prevents data loss when you upgrade in place. To remove the CRD and cascade-delete any remaining CRs:

kubectl delete crd lmcacheengines.lmcache.lmcache.ai

Cleaning up an inconsistent or orphaned install

Use this when the normal helm uninstall path doesn’t apply — for example:

helm list -A shows no release, but the engine pod, LMCacheEngine CR, services, and CRD are still present (the release record was removed but its resources weren’t).
helm uninstall hangs, or kubectl delete namespace gets stuck in Terminating.

The root cause is almost always the same: the LMCacheEngine CR carries a finalizer (lmcache.ai/cleanup) that only the operator can drain. If the operator is gone — or was removed before the CR — nothing runs the finalizer, so the CR can’t be deleted, and anything waiting on it (a namespace delete, an uninstall hook) blocks indefinitely. Clean up manually, in this order:

Clear the stuck CR's finalizer

The operator isn’t coming back to drain it, so remove the finalizer yourself:

kubectl patch lmcacheengine <name> -n <namespace> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

(Find the name with kubectl get lmcacheengine -A.) This is safe only when the operator is definitively gone — it skips the cleanup the operator would have done, which is fine when you’re tearing everything down anyway.

Delete the namespace

With the finalizer cleared, this completes instead of hanging — it sweeps the DaemonSet, pods, services, ConfigMaps, ServiceAccounts, and RoleBindings.

kubectl delete namespace <namespace>

Delete the CRD

Cluster-scoped, so the namespace delete didn’t touch it. Removing it gives a clean slate (and avoids the ownership conflict on the next install).

kubectl delete crd lmcacheengines.lmcache.lmcache.ai

Sweep orphaned cluster-scoped RBAC

The operator’s ClusterRole/ClusterRoleBinding are cluster-scoped and survive a namespace delete. Remove any leftovers:

kubectl get clusterrole,clusterrolebinding | grep -i tensormesh
# delete anything it lists, e.g.:
kubectl delete clusterrole,clusterrolebinding -l app.kubernetes.io/name=tensormesh-operator

Order matters. Clear the finalizer before deleting the namespace. If you delete the namespace first, it gets stuck in Terminating (blocked on the un-drainable CR) until you clear the finalizer anyway.

After these four steps, kubectl get ns <namespace> and kubectl get crd lmcacheengines.lmcache.lmcache.ai should both return NotFound — a true clean slate.

Gathering a diagnostic bundle

When opening a support thread, include:

# Versions
helm version
kubectl version
kubectl get nodes -o wide

# Release state
helm list -A
helm get values tensormesh-operator -n tensormesh-operator
helm get manifest tensormesh-operator -n tensormesh-operator

# Live state
kubectl get all -n tensormesh-operator
kubectl get lmcacheengines -A -o yaml

# Logs
kubectl logs -n tensormesh-operator \
  deploy/tensormesh-operator --tail=500
kubectl logs -n tensormesh-operator \
  -l app.kubernetes.io/component=cache-engine --tail=500

# Events
kubectl get events -n tensormesh-operator --sort-by=.lastTimestamp

Rare hashing fallback: if LMCache logs that it is using builtin hash instead of the default blake3 path, set PYTHONHASHSEED=0 in the vLLM pod and restart it. This is not the normal MP requirement and it is unrelated to MQ payload-shape mismatches.

Installation

Feature List

Integrations

Management

Operator UI

Reference

Contents

Engine pod is `Pending`

Pod stuck in `ContainerCreating` (CDI)

Engine pod `CreateContainerError` on OpenShift

Operator pod `ImagePullBackOff`

Install fails with `CRD ... exists and cannot be imported`

Option A — adopt the CRD into your new release (recommended)

Option B — delete the CRD and let the install recreate it

`helm uninstall` hangs

Metrics endpoint returns `401`

CRD survives uninstall (by design)

Cleaning up an inconsistent or orphaned install

Gathering a diagnostic bundle

Next steps

Helm install reference

Getting Started

​Contents

​Engine pod is Pending

​Pod stuck in ContainerCreating (CDI)

​Engine pod CreateContainerError on OpenShift

​Operator pod ImagePullBackOff

​Install fails with CRD ... exists and cannot be imported

​Option A — adopt the CRD into your new release (recommended)

​Option B — delete the CRD and let the install recreate it

​helm uninstall hangs

​Metrics endpoint returns 401

​CRD survives uninstall (by design)

​Cleaning up an inconsistent or orphaned install

​Gathering a diagnostic bundle

​Next steps

Helm install reference

Getting Started

Contents

Engine pod is `Pending`

Pod stuck in `ContainerCreating` (CDI)

Engine pod `CreateContainerError` on OpenShift

Operator pod `ImagePullBackOff`

Install fails with `CRD ... exists and cannot be imported`

Option A — adopt the CRD into your new release (recommended)

Option B — delete the CRD and let the install recreate it

`helm uninstall` hangs

Metrics endpoint returns `401`

CRD survives uninstall (by design)

Cleaning up an inconsistent or orphaned install

Gathering a diagnostic bundle

Next steps