Skip to main content

​
Troubleshooting

The image can’t be pulled — the GHCR pull secret is missing from the namespace, or its token can’t read the package. Confirm the imagePullSecrets name in your values matches a secret that exists in the namespace, and that its token has read:packages (the operator’s pull secret works). Recreate it with:
The install doesn’t expose the UI outside the cluster — the pod is fine, it’s just not reachable yet. Start a port-forward and keep the terminal open:
then open http://localhost:8080. For shared team access, enable the ingress behind SSO instead — see Access & Security.
engineNamespace doesn’t match the namespace where your engines actually run.
The engines or GPUs are idle — utilization and hit rate only move under inference traffic. (GPU memory stays allocated, so it can be non-zero while utilization reads 0.) If there is traffic and charts are still flat, check that prometheus.url is right.
The engine metrics aren’t reaching the Prometheus the UI reads. Confirm the operator’s observability is on (observability.enabled=true on the operator chart) and that prometheus.url points at the Prometheus those metrics land in. See Configuration.
Fleet-wide totals are still correct; the UI just can’t tell which node each metric came from — usually because your metrics collector runs as one shared service instead of per-node. Ask the Tensormesh team for the collector setting that restores per-node attribution.
GPU telemetry isn’t connected — set dcgmService (see Configuration). Without it, node cards show one row per node; everything else is unaffected.
Cluster access is automatic only for the cluster the UI runs in. Each extra clusters entry needs its own apiUrl and an access token — supplied through a Kubernetes Secret wired in with the chart’s extraEnvFrom, not in your values file. See Configuration.

​
FAQ

No — the UI observes only, by design. It names the failing component and the fix (Health → Issues & Fixes); you act via tmo-cli, kubectl, or Helm.
A handful of lightweight read requests per open page every 30 seconds, paused when the tab is in the background. Nothing runs on your GPU nodes.
It depends on what’s unreachable. If the control plane can’t be reached, the monitoring pages (Overview, Fleet Map, Metrics, Health) show an explicit error — “The control plane didn’t respond” — with a Retry button, never zeros or mock data. If the cluster is simply empty or idle, you get real zeros and a “No LMCacheEngines found” banner. If only metric history is missing (Prometheus not connected), the trend charts are empty while everything else works.