> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting & FAQ

> Common setup problems and short answers to frequent questions.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Pod stuck in ImagePullBackOff" icon="box">
    The image can't be pulled — the GHCR pull secret is missing from the namespace, or its token
    can't read the package. Confirm the `imagePullSecrets` name in your values matches a secret that
    exists in the namespace, and that its token has `read:packages` (the operator's pull secret
    works). Recreate it with:

    ```bash theme={null}
    kubectl create secret docker-registry tensormesh-ghcr -n tensormesh-operator \
      --docker-server=ghcr.io --docker-username=<you> --docker-password=<token>
    ```
  </Accordion>

  <Accordion title="Can't open the dashboard / connection refused" icon="plug">
    The install doesn't expose the UI outside the cluster — the pod is fine, it's just not reachable
    yet. Start a port-forward and **keep the terminal open**:

    ```bash theme={null}
    kubectl -n tensormesh-operator port-forward svc/tmo-ui-tensormesh-operator-ui 8080:80
    ```

    then open [http://localhost:8080](http://localhost:8080). For shared team access, enable the
    ingress behind SSO instead — see [Access & Security](/ui/access-and-security).
  </Accordion>

  <Accordion title="&#x22;No LMCacheEngines found&#x22;" icon="magnifying-glass">
    `engineNamespace` doesn't match the namespace where your engines actually run.
  </Accordion>

  <Accordion title="Numbers sit at 0 (charts or GPU utilization)" icon="gauge">
    The engines or GPUs are idle — utilization and hit rate only move under inference traffic.
    (GPU *memory* stays allocated, so it can be non-zero while utilization reads 0.) If there
    *is* traffic and charts are still flat, check that `prometheus.url` is right.
  </Accordion>

  <Accordion title="Trend charts empty, but Health and Quotas work" icon="chart-line">
    The engine metrics aren't in the Prometheus the UI reads. The most common cause: the
    engine's metrics endpoint was never enabled — it's off by default. Set
    `engine.spec.prometheus.enabled: true` and add the `ServiceMonitor` so Prometheus scrapes
    it — both shown in
    [Configuration](/ui/configuration#trend-charts-connect-your-prometheus). Also
    confirm `prometheus.url` points at the right Prometheus
    (`kubectl get svc -A | grep -i prometheus`).
  </Accordion>

  <Accordion title="Per-node charts collapse into one &#x22;Engine-wide&#x22; line" icon="layer-group">
    Fleet-wide totals are still correct; the UI just can't tell which node each metric came
    from — usually because your metrics collector runs as one shared service instead of
    per-node. Ask the Tensormesh team for the collector setting that restores per-node
    attribution.
  </Accordion>

  <Accordion title="No per-GPU cards on the Fleet Map" icon="microchip">
    GPU telemetry isn't connected — set `dcgmService` to your exporter's service (find it
    with `kubectl get svc -A | grep -i dcgm` — see
    [Configuration](/ui/configuration)). Without it, node cards show one row per
    node; everything else is unaffected.
  </Accordion>

  <Accordion title="GPU trend charts empty (but per-GPU cards work)" icon="chart-line">
    The over-time GPU charts read history from Prometheus, so Prometheus has to be **scraping
    DCGM** — the live cards only need `dcgmService`. Check it: query `DCGM_FI_DEV_GPU_UTIL` in
    Prometheus and expect series back. If it's empty, the exporter's `ServiceMonitor` is often
    labelled for a different Prometheus (some managed clusters, e.g. CoreWeave, use
    `environment: grafana-monitoring` instead of `release: <your-release>`). Add a
    `ServiceMonitor` for the DCGM service with the label your Prometheus selects. See
    [Configuration](/ui/configuration#per-gpu-detail-connect-gpu-telemetry-gpu-clusters).
  </Accordion>

  <Accordion title="A second cluster shows nothing (multiple clusters)" icon="sitemap">
    Cluster access is automatic only for the cluster the UI runs **in**. Each extra `clusters` entry
    needs its own `apiUrl` and an access token — supplied through a Kubernetes Secret wired in with
    the chart's `extraEnvFrom`, not in your values file. See
    [Configuration](/ui/configuration#multiple-clusters).
  </Accordion>
</AccordionGroup>

***

## FAQ

<AccordionGroup>
  <Accordion title="Can I change quotas, restart pods, or clear the cache from the UI?">
    No — the UI observes only, by design. It names the failing component and the fix
    (Health → Issues & Fixes); you act via [`tmo-cli`](/reference/cli), `kubectl`, or
    Helm.
  </Accordion>

  <Accordion title="What load does it put on my cluster?">
    A handful of lightweight read requests per open page every 30 seconds, paused when the tab
    is in the background. Nothing runs on your GPU nodes.
  </Accordion>

  <Accordion title="What happens when a data source is down?">
    It depends on what's unreachable. If the **control plane** can't be reached, the monitoring
    pages (Overview, Fleet Map, Metrics, Health) show an explicit error — *"The control plane
    didn't respond"* — with a **Retry** button, never zeros or mock data. If the cluster is
    simply empty or idle, you get real zeros and a "No LMCacheEngines found" banner. If only
    metric **history** is missing (Prometheus not connected), the trend charts are empty while
    everything else works.
  </Accordion>
</AccordionGroup>

***

## Related

* [Configuration](/ui/configuration)
* [Installation](/ui/installation)
* [Operator Installation Troubleshooting](/installation/troubleshooting)