> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuration

> Connect metric history, GPU telemetry, and additional clusters — everything else works out of the box.

Most of the dashboard works the moment it's installed. Two things need connecting: your
**Prometheus** (for the trend charts), and — on GPU clusters — **GPU telemetry** (for per-GPU detail):

| What you see                                                      | Where it comes from             | What you configure       |
| ----------------------------------------------------------------- | ------------------------------- | ------------------------ |
| Fleet, nodes, health, cache internals, quotas                     | The operator and engines        | No setup needed          |
| Trend charts (hit rate, capacity, eviction, throughput over time) | Your Prometheus                 | `prometheus.url`         |
| Per-GPU cards (utilization, memory, hardware faults)              | NVIDIA's GPU telemetry exporter | `dcgmService` — optional |

Everything is set through Helm values:

| Value                | What it does                                                                                        | Default                    |
| -------------------- | --------------------------------------------------------------------------------------------------- | -------------------------- |
| `engineNamespace`    | The namespace where your engines run                                                                | *(the release namespace)*  |
| `prometheus.url`     | Where the UI reads metric history from                                                              | *(unset)*                  |
| `prometheus.service` | Alternative: reach Prometheus through Kubernetes instead of a direct URL (`service.namespace:port`) | *(unset)*                  |
| `dcgmService`        | GPU telemetry source (`service.namespace:port`)                                                     | *(unset)*                  |
| `clusters`           | Watch several clusters in one dashboard                                                             | *(unset — single cluster)* |
| `ingress.enabled`    | Expose the UI beyond port-forward                                                                   | `false`                    |

Cluster access itself needs no configuration — the UI picks it up automatically when running
in-cluster.

***

## Trend Charts: Connect Your Prometheus

Live numbers come straight from the engines; **history** comes from your Prometheus. Point
`prometheus.url` at the Prometheus your engine metrics land in (or use `prometheus.service` if
you'd rather not open a direct path):

```yaml theme={null}
prometheus:
  url: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
```

Engine metrics get *into* Prometheus through the operator's observability stack — a
`ServiceMonitor` or OTel Collector, turned on with `observability.enabled=true` on the operator
chart (see [Operator Observability](/operator/observability)). The UI simply reads whichever
Prometheus those metrics reach. Charts fill once both halves are in place; if they stay empty
while everything else works, see [Troubleshooting](/operator/ui/troubleshooting).

***

## Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Per-GPU utilization, memory fill, and hardware-fault detail come from NVIDIA's DCGM exporter —
standard on GPU clusters (it ships with the NVIDIA GPU Operator), but not part of Tensormesh.
Point `dcgmService` at its service (`service.namespace:port`):

```yaml theme={null}
# the exact service name and namespace depend on your DCGM / GPU Operator install
dcgmService: dcgm-exporter.gpu-operator:9400
```

If it's unset or unreachable, the Fleet Map simply shows one row per node instead of per-GPU
cards; everything else is unaffected.

***

## Multiple Clusters

One dashboard can watch several clusters — the fleet-wide pages merge every cluster's engines
(labeled `· prod` / `· staging`), and per-engine pages scope to the right one.

The cluster the UI runs **in** is automatic. Each *additional* cluster needs just two things: its
`apiUrl` and a **read-only token**. It's two steps:

**1. Put your clusters (with tokens) in a Secret:**

```bash theme={null}
kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
  --from-literal=TMO_CLUSTERS='[
    {"name":"prod","apiUrl":"https://prod-api.example.com","token":"<prod-token>"},
    {"name":"staging","apiUrl":"https://staging-api.example.com","token":"<staging-token>"}
  ]'
```

**2. Point your values file at it:**

```yaml theme={null}
extraEnvFrom:
  - secretRef:
      name: tmo-ui-clusters
```

Reinstall, and every cluster shows up. Each entry can also set its own `namespace`,
`prometheusUrl`, or `dcgmService` — and **`"insecure": true`** if that cluster's API uses a
self-signed certificate.

<Accordion title="How do I get a read-only token for a cluster?">
  Run this **in that cluster** — it creates a read-only service account (the same access the UI
  uses) and prints a token:

  ```bash theme={null}
  kubectl apply -f - <<'EOF'
  apiVersion: v1
  kind: ServiceAccount
  metadata: { name: tmo-ui-reader, namespace: tensormesh-operator }
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata: { name: tmo-ui-reader }
  rules:
    - { apiGroups: ["lmcache.lmcache.ai"], resources: ["lmcacheengines", "lmcachecoordinators"], verbs: ["get", "list", "watch"] }
    - { apiGroups: [""], resources: ["pods", "namespaces", "nodes", "events"], verbs: ["get", "list"] }
    - { apiGroups: [""], resources: ["pods/proxy", "services/proxy"], verbs: ["get"] }
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata: { name: tmo-ui-reader }
  roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: tmo-ui-reader }
  subjects:
    - { kind: ServiceAccount, name: tmo-ui-reader, namespace: tensormesh-operator }
  EOF

  kubectl -n tensormesh-operator create token tmo-ui-reader --duration=8760h
  ```
</Accordion>

<Tip>
  **One token for every cluster?** Skip the per-cluster tokens — list the clusters in your **values
  file** and put a single `TMO_KUBE_TOKEN` in the Secret; every cluster inherits it.

  ```yaml theme={null}
  # values.yaml
  clusters:
    - { name: prod, apiUrl: https://prod-api.example.com }
    - { name: dev, apiUrl: https://dev-api.example.com }
    # add `insecure: true` to any entry whose API uses a self-signed certificate
  extraEnvFrom:
    - secretRef: { name: tmo-ui-clusters }
  ```

  ```bash theme={null}
  kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
    --from-literal=TMO_KUBE_TOKEN='<shared-token>'
  ```
</Tip>

***

## Related

* [Troubleshooting & FAQ](/operator/ui/troubleshooting)
* [Monitoring Your Fleet](/operator/ui/monitoring-your-fleet)
* [Operator Observability](/operator/observability)
