> ## Documentation Index > Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt > Use this file to discover all available pages before exploring further. # Troubleshooting > Diagnose common install and day-2 issues with the Tensormesh Operator. This page covers the most common install and runtime issues. For everything else, gather the diagnostic bundle described at the bottom and open a support thread. ## Contents **Install & scheduling** * [Engine pod is `Pending`](#engine-pod-is-pending) — node-selector / GPU label mismatch * [Pod stuck in `ContainerCreating`](#pod-stuck-in-containercreating-cdi) — CDI disabled (older NVIDIA GPU Operator) * [Engine pod `CreateContainerError` on OpenShift](#engine-pod-createcontainererror-on-openshift) — missing privileged SCC * [Operator pod `ImagePullBackOff`](#operator-pod-imagepullbackoff) — registry auth / air-gap * [Install fails with `CRD ... exists and cannot be imported`](#install-fails-with-crd-exists-and-cannot-be-imported) — ownership conflict from a prior install **Uninstall & cleanup** * [`helm uninstall` hangs](#helm-uninstall-hangs) — stuck CR finalizer * [CRD survives uninstall (by design)](#crd-survives-uninstall-by-design) * [Cleaning up an inconsistent or orphaned install](#cleaning-up-an-inconsistent-or-orphaned-install) — no release record / orphaned resources **Metrics & observability** * [Metrics endpoint returns `401`](#metrics-endpoint-returns-401) * OTel Collector / traces not flowing → see [Observability → Troubleshooting](/operator/observability#troubleshooting) **Getting help** * [Gathering a diagnostic bundle](#gathering-a-diagnostic-bundle) ## Engine pod is `Pending` ```bash theme={null} kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine # NAME READY STATUS RESTARTS AGE # tmo-engine-xxxxx 0/1 Pending 0 2m ``` Almost always a node-selector mismatch. The engine DaemonSet selects nodes labeled `nvidia.com/gpu.present=true` by default — if no node carries that label, no pod is scheduled. ```bash theme={null} # What labels do my nodes carry? kubectl get nodes -L nvidia.com/gpu.present # Add the label to a GPU node manually kubectl label node nvidia.com/gpu.present=true ``` If you use a different label in your environment, override the selector instead: ```yaml my-values.yaml theme={null} engine: spec: nodeSelector: mycompany.io/gpu-tier: "premium" ``` ## Pod stuck in `ContainerCreating` (CDI) ```bash theme={null} kubectl get pods -n tensormesh-operator -o wide # NAME READY STATUS RESTARTS AGE # tmo-engine-xxxxx 0/1 ContainerCreating 0 5m ``` The pod **scheduled onto a GPU node** (so it's past `Pending`) but the container can't be created. The usual cause on NVIDIA GPU clusters is that the **Container Device Interface (CDI)** is disabled. The NVIDIA GPU Operator injects GPUs via CDI, and **before GPU Operator v25.10.0 it defaulted to `cdi.enabled: false`** — with no obvious error, the pod just hangs in `ContainerCreating`. Enable CDI in the GPU `ClusterPolicy`: ```bash theme={null} oc patch clusterpolicy gpu-cluster-policy --type=merge -p '{ "spec": { "cdi": { "enabled": true, "default": true } } }' ``` GPU Operator **v25.10.0+** enables CDI by default — no patch needed. Confirm it's CDI and not something else: `kubectl describe pod ` — a CDI problem shows a `Failed to create … CDI device injection` / `unresolvable CDI devices` event. A missing-image event (`ImagePullBackOff`) or volume-mount error points elsewhere. ## Engine pod `CreateContainerError` on OpenShift ```bash theme={null} kubectl describe pod -n tensormesh-operator # Events: # Warning Failed ... unable to start container: hostIPC is not allowed ``` The pod is requesting `hostIPC` but its ServiceAccount is not bound to an SCC that permits it. Add `openshift.enabled: true` to your values file and upgrade: ```bash theme={null} helm upgrade tensormesh-operator \ oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \ --version 0.4.5 -n tensormesh-operator \ -f values-openshift.yaml ``` See [Install on OpenShift](/operator/installation/openshift) for the full overlay and for the "bring your own SA" pattern. ## Operator pod `ImagePullBackOff` ```bash theme={null} kubectl describe pod -n tensormesh-operator # Events: # Warning Failed ... Failed to pull image: unauthorized ``` The image lives in a registry your cluster cannot authenticate to. Create an image pull secret and reference it from the chart: ```bash theme={null} kubectl create secret docker-registry tmo-pull \ -n tensormesh-operator \ --docker-server= \ --docker-username= \ --docker-password= ``` ```yaml my-values.yaml theme={null} operator: image: pullSecrets: - tmo-pull ``` For air-gapped clusters, mirror the operator image into your internal registry and override `operator.image.repository` and `operator.image.tag`. ## Install fails with `CRD ... exists and cannot be imported` ```bash theme={null} helm install tensormesh-operator oci://ghcr.io/.../tensormesh-operator ... # Error: INSTALLATION FAILED: unable to continue with install: # CustomResourceDefinition "lmcacheengines.lmcache.lmcache.ai" in namespace "" exists # and cannot be imported into the current release: invalid ownership metadata; # annotation validation error: key "meta.helm.sh/release-name" must equal "tensormesh-operator": # current value is "" ``` The `LMCacheEngine` CRD is cluster-scoped — there is only one of it. A previous install under a different release name (or namespace) annotated it as its own; Helm refuses to silently transfer ownership to your new release. This is a Helm-wide safety check, not specific to this chart — cert-manager, kube-prometheus-stack, and Argo CD all behave identically. Before doing either fix below, check whether the prior release is still active: `helm list -A`. If you see a real release that owns the CRD, leave it alone — running a second operator against the same CRD will fight reconciliation. Use the existing release (`helm upgrade ...`) instead. ### Option A — adopt the CRD into your new release (recommended) Use this when the prior release is gone (was uninstalled, or is on a cluster you don't own anymore) but the CRD survived because of `helm.sh/resource-policy: keep`. ```bash theme={null} kubectl annotate crd lmcacheengines.lmcache.lmcache.ai \ meta.helm.sh/release-name= \ meta.helm.sh/release-namespace= --overwrite kubectl label crd lmcacheengines.lmcache.lmcache.ai \ app.kubernetes.io/managed-by=Helm --overwrite # Re-run the install helm install oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \ --version \ --namespace --create-namespace ``` Existing `LMCacheEngine` CRs survive — only the CRD's ownership annotations change. The `LMCacheEngine` CRD is cluster-scoped — one of it exists for the whole cluster. Helm tracks which release owns each resource via three stamps: | Field | Purpose | | ------------------------------------------- | -------------------------------- | | annotation `meta.helm.sh/release-name` | which release owns this | | annotation `meta.helm.sh/release-namespace` | in which namespace | | label `app.kubernetes.io/managed-by=Helm` | "some Helm release manages this" | The adoption commands rewrite those stamps so your new release becomes the recognized owner. The CRD's schema, any existing `LMCacheEngine` CRs, and everything in any namespace are unchanged — it's a pure metadata edit, fully reversible by running the same commands with the old values. ### Option B — delete the CRD and let the install recreate it Safe **only** when no `LMCacheEngine` CRs exist anywhere on the cluster. Confirm first: ```bash theme={null} kubectl get lmcacheengines -A # must return "No resources found" kubectl delete crd lmcacheengines.lmcache.lmcache.ai # Re-run the install — it will create the CRD fresh with the right ownership helm install oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \ --version \ --namespace --create-namespace ``` Deleting the CRD cascade-deletes every `LMCacheEngine` CR on the cluster. Use Option A if you have any CRs you want to keep. ## `helm uninstall` hangs The pre-delete hook is waiting for the `LMCacheEngine` CR's finalizer to drain. Usually this means the operator was already gone when the CR was deleted, so nobody is running the finalizer. ```bash theme={null} # 1. Confirm the CR is stuck kubectl get lmcacheengines -A # 2. Inspect operator logs (if it's still around) kubectl logs -n tensormesh-operator \ deploy/tensormesh-operator # 3. Force-clear the finalizer kubectl patch lmcacheengine -n \ -p '{"metadata":{"finalizers":[]}}' --type=merge # 4. Re-run uninstall helm uninstall tensormesh-operator -n tensormesh-operator ``` Removing finalizers bypasses any cleanup the operator would have performed on backing resources. Use only when the operator is definitively gone. If `helm list -A` shows **no release at all** (the operator *and* the release record are already gone, not just a stuck uninstall), this quick fix doesn't apply — follow [Cleaning up an inconsistent or orphaned install](#cleaning-up-an-inconsistent-or-orphaned-install) for the full manual teardown. ## Metrics endpoint returns `401` The controller-runtime auth filter is on by default. Either disable it for scrapers that cannot authenticate: ```yaml my-values.yaml theme={null} operator: metrics: secure: false ``` Or grant the scraping ServiceAccount the `system:auth-delegator` ClusterRole so it can authenticate with the kube-apiserver TokenReview API: ```bash theme={null} kubectl create clusterrolebinding metrics-auth \ --clusterrole=system:auth-delegator \ --serviceaccount=: ``` ## CRD survives uninstall (by design) The `LMCacheEngine` CRD is annotated `helm.sh/resource-policy: keep`, so `helm uninstall` leaves it (and any remaining CRs) in place. This is the same convention used by cert-manager and kube-prometheus-stack — it prevents data loss when you upgrade in place. To remove the CRD and cascade-delete any remaining CRs: ```bash theme={null} kubectl delete crd lmcacheengines.lmcache.lmcache.ai ``` ## Cleaning up an inconsistent or orphaned install Use this when the normal `helm uninstall` path doesn't apply — for example: * `helm list -A` shows **no release**, but the engine pod, `LMCacheEngine` CR, services, and CRD are still present (the release record was removed but its resources weren't). * `helm uninstall` **hangs**, or `kubectl delete namespace` gets stuck in `Terminating`. The root cause is almost always the same: the `LMCacheEngine` CR carries a finalizer (`lmcache.ai/cleanup`) that **only the operator can drain**. If the operator is gone — or was removed before the CR — nothing runs the finalizer, so the CR can't be deleted, and anything waiting on it (a namespace delete, an uninstall hook) blocks indefinitely. Clean up manually, **in this order**: The operator isn't coming back to drain it, so remove the finalizer yourself: ```bash theme={null} kubectl patch lmcacheengine -n \ -p '{"metadata":{"finalizers":[]}}' --type=merge ``` (Find the name with `kubectl get lmcacheengine -A`.) This is safe **only** when the operator is definitively gone — it skips the cleanup the operator would have done, which is fine when you're tearing everything down anyway. With the finalizer cleared, this completes instead of hanging — it sweeps the DaemonSet, pods, services, ConfigMaps, ServiceAccounts, and RoleBindings. ```bash theme={null} kubectl delete namespace ``` Cluster-scoped, so the namespace delete didn't touch it. Removing it gives a clean slate (and avoids the ownership conflict on the next install). ```bash theme={null} kubectl delete crd lmcacheengines.lmcache.lmcache.ai ``` The operator's `ClusterRole`/`ClusterRoleBinding` are cluster-scoped and survive a namespace delete. Remove any leftovers: ```bash theme={null} kubectl get clusterrole,clusterrolebinding | grep -i tensormesh # delete anything it lists, e.g.: kubectl delete clusterrole,clusterrolebinding -l app.kubernetes.io/name=tensormesh-operator ``` **Order matters.** Clear the finalizer *before* deleting the namespace. If you delete the namespace first, it gets stuck in `Terminating` (blocked on the un-drainable CR) until you clear the finalizer anyway. After these four steps, `kubectl get ns ` and `kubectl get crd lmcacheengines.lmcache.lmcache.ai` should both return `NotFound` — a true clean slate. ## Gathering a diagnostic bundle When opening a support thread, include: ```bash theme={null} # Versions helm version kubectl version kubectl get nodes -o wide # Release state helm list -A helm get values tensormesh-operator -n tensormesh-operator helm get manifest tensormesh-operator -n tensormesh-operator # Live state kubectl get all -n tensormesh-operator kubectl get lmcacheengines -A -o yaml # Logs kubectl logs -n tensormesh-operator \ deploy/tensormesh-operator --tail=500 kubectl logs -n tensormesh-operator \ -l app.kubernetes.io/component=cache-engine --tail=500 # Events kubectl get events -n tensormesh-operator --sort-by=.lastTimestamp ``` Rare hashing fallback: if LMCache logs that it is using **builtin hash** instead of the default `blake3` path, set `PYTHONHASHSEED=0` in the vLLM pod and restart it. This is not the normal MP requirement and it is unrelated to MQ payload-shape mismatches. ## Next steps Re-check the install steps against the chart's documented values. Re-validate prerequisites if multiple issues are stacking up.