Configuration

Most of the dashboard works the moment it’s installed. Two things need connecting: your Prometheus (for the trend charts), and — on GPU clusters — GPU telemetry (for per-GPU detail):

What you see	Where it comes from	What you configure
Fleet, nodes, health, cache internals, quotas	The operator and engines	No setup needed
Trend charts (hit rate, capacity, eviction, throughput over time)	Your Prometheus	`prometheus.url`
Per-GPU cards (utilization, memory, hardware faults)	NVIDIA’s GPU telemetry exporter	`dcgmService` — optional

Everything is set through Helm values:

Value	What it does	Default
`engineNamespace`	The namespace where your engines run	(the release namespace)
`prometheus.url`	Where the UI reads metric history from	(unset)
`prometheus.service`	Alternative: reach Prometheus through Kubernetes instead of a direct URL (`service.namespace:port`)	(unset)
`dcgmService`	GPU telemetry source (`service.namespace:port`)	(unset)
`clusters`	Watch several clusters in one dashboard	(unset — single cluster)
`ingress.enabled`	Expose the UI beyond port-forward	`false`

Cluster access itself needs no configuration — the UI picks it up automatically when running in-cluster.

Trend Charts: Connect Your Prometheus

Live numbers come straight from the engines; history comes from your Prometheus. Point prometheus.url at the Prometheus your engine metrics land in (or use prometheus.service if you’d rather not open a direct path):

prometheus:
  url: http://kube-prometheus-stack-prometheus.monitoring.svc:9090

Engine metrics get into Prometheus through the operator’s observability stack — a ServiceMonitor or OTel Collector, turned on with observability.enabled=true on the operator chart (see Operator Observability). The UI simply reads whichever Prometheus those metrics reach. Charts fill once both halves are in place; if they stay empty while everything else works, see Troubleshooting.

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Per-GPU utilization, memory fill, and hardware-fault detail come from NVIDIA’s DCGM exporter — standard on GPU clusters (it ships with the NVIDIA GPU Operator), but not part of Tensormesh. Point dcgmService at its service (service.namespace:port):

# the exact service name and namespace depend on your DCGM / GPU Operator install
dcgmService: dcgm-exporter.gpu-operator:9400

If it’s unset or unreachable, the Fleet Map simply shows one row per node instead of per-GPU cards; everything else is unaffected.

Multiple Clusters

One dashboard can watch several clusters — the fleet-wide pages merge every cluster’s engines (labeled · prod / · staging), and per-engine pages scope to the right one. The cluster the UI runs in is automatic. Each additional cluster needs just two things: its apiUrl and a read-only token. It’s two steps: 1. Put your clusters (with tokens) in a Secret:

kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
  --from-literal=TMO_CLUSTERS='[
    {"name":"prod","apiUrl":"https://prod-api.example.com","token":"<prod-token>"},
    {"name":"staging","apiUrl":"https://staging-api.example.com","token":"<staging-token>"}
  ]'

2. Point your values file at it:

extraEnvFrom:
  - secretRef:
      name: tmo-ui-clusters

Reinstall, and every cluster shows up. Each entry can also set its own namespace, prometheusUrl, or dcgmService — and "insecure": true if that cluster’s API uses a self-signed certificate.

How do I get a read-only token for a cluster?

Run this in that cluster — it creates a read-only service account (the same access the UI uses) and prints a token:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: { name: tmo-ui-reader, namespace: tensormesh-operator }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: tmo-ui-reader }
rules:
  - { apiGroups: ["lmcache.lmcache.ai"], resources: ["lmcacheengines", "lmcachecoordinators"], verbs: ["get", "list", "watch"] }
  - { apiGroups: [""], resources: ["pods", "namespaces", "nodes", "events"], verbs: ["get", "list"] }
  - { apiGroups: [""], resources: ["pods/proxy", "services/proxy"], verbs: ["get"] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: tmo-ui-reader }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: tmo-ui-reader }
subjects:
  - { kind: ServiceAccount, name: tmo-ui-reader, namespace: tensormesh-operator }
EOF

kubectl -n tensormesh-operator create token tmo-ui-reader --duration=8760h

One token for every cluster? Skip the per-cluster tokens — list the clusters in your values file and put a single TMO_KUBE_TOKEN in the Secret; every cluster inherits it.

# values.yaml
clusters:
  - { name: prod, apiUrl: https://prod-api.example.com }
  - { name: dev, apiUrl: https://dev-api.example.com }
  # add `insecure: true` to any entry whose API uses a self-signed certificate
extraEnvFrom:
  - secretRef: { name: tmo-ui-clusters }

kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
  --from-literal=TMO_KUBE_TOKEN='<shared-token>'

Installation

Management

Operator UI

Reference

Configuration

Trend Charts: Connect Your Prometheus

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Multiple Clusters

​Trend Charts: Connect Your Prometheus

​Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

​Multiple Clusters

​Related

Trend Charts: Connect Your Prometheus

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Multiple Clusters

Related