Configuration

Most of the dashboard works the moment it’s installed. Two things need connecting: your Prometheus (for the trend charts), and, on GPU clusters, GPU telemetry (for per-GPU detail):

What you see	Where it comes from	What you configure
Fleet, nodes, health, cache internals, quotas	The operator and engines	No setup needed
Trend charts (hit rate, capacity, eviction, throughput over time)	Your Prometheus	`prometheus.url` + engine metrics scraping: two steps
Per-GPU cards (utilization, memory, hardware faults)	NVIDIA’s GPU telemetry exporter	`dcgmService` (optional)

Everything is set through Helm values:

Value	What it does	Default
`engineNamespace`	The namespace where your engines run	(the release namespace)
`prometheus.url`	Where the UI reads metric history from	(unset)
`prometheus.service`	Alternative: reach Prometheus through Kubernetes instead of a direct URL (`service.namespace:port`)	(unset)
`dcgmService`	GPU telemetry source (`service.namespace:port`)	(unset)
`clusters`	Watch several clusters in one dashboard	(unset, single cluster)
`ingress.enabled`	Expose the UI beyond port-forward	`false`

Cluster access itself needs no configuration: the UI picks it up automatically when running in-cluster.

Trend Charts: Connect Your Prometheus

Live numbers come straight from the engines; history comes from your Prometheus. Two things must both be true: the UI can reach a Prometheus, and the engine’s metrics are arriving in it. 1. Point the UI at your Prometheus. Find its service with kubectl get svc -A | grep -i prometheus. The name varies by install:

prometheus:
  url: http://<prometheus-service>.<namespace>.svc:9090
# e.g. kube-prometheus-stack-prometheus.monitoring.svc      (default kube-prometheus-stack)
# e.g. kube-prom-stack-kube-prome-prometheus.monitoring.svc (name follows your Helm release)

2. Make sure the engine’s metrics reach that Prometheus. How they get there depends on your setup: Prometheus scraping the engine directly, or the engine pushing through an OTel Collector. Both paths (and how to verify which one you’re on) are covered in the operator’s Metrics page. The UI just reads what Prometheus already has.

No Prometheus in the cluster at all?

Install the community kube-prometheus-stack. It includes the Prometheus Operator that makes ServiceMonitors work:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Then prometheus.url is http://kube-prometheus-stack-prometheus.monitoring.svc:9090, and the ServiceMonitor label above is release: kube-prometheus-stack.

Charts fill once both halves are in place; if they stay empty while everything else works, see Troubleshooting.

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Per-GPU utilization, memory fill, and hardware-fault detail come from NVIDIA’s DCGM exporter, standard on GPU clusters (it ships with the NVIDIA GPU Operator), but not part of Tensormesh. Find your exporter’s service and point dcgmService at it (service.namespace:port):

# find yours: kubectl get svc -A | grep -i dcgm
dcgmService: dcgm-exporter.cw-exporters:9400     # CoreWeave clusters
# dcgmService: dcgm-exporter.gpu-operator:9400   # e.g. an NVIDIA GPU Operator install

If it’s unset or unreachable, the Fleet Map shows one row per node instead of per-GPU cards; everything else is unaffected. If your GPU cluster doesn’t have the exporter at all, it ships with the NVIDIA GPU Operator: install that per NVIDIA’s docs and the exporter service appears with it.

GPU trend charts need Prometheus to scrape DCGM

The live per-GPU cards fall back to dcgmService directly, but the over-time charts read history from Prometheus. Confirm DCGM is being scraped: query DCGM_FI_DEV_GPU_UTIL in Prometheus and expect series back. On some managed clusters the exporter ships a ServiceMonitor labelled for a different Prometheus (e.g. CoreWeave labels theirs environment: grafana-monitoring, while kube-prometheus-stack selects release: <your-release>), so it returns zero series and the GPU trend charts stay empty. Fix it by adding a ServiceMonitor for the DCGM exporter’s service with the label your Prometheus selects.

Multiple Clusters

One dashboard can watch several clusters: the fleet-wide pages merge every cluster’s engines (labeled · prod / · staging), and per-engine pages scope to the right one. The cluster the UI runs in is automatic. Each additional cluster needs just two things: its apiUrl and a read-only token. There are two steps: 1. Put your clusters (with tokens) in a Secret:

kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
  --from-literal=TMO_CLUSTERS='[
    {"name":"prod","apiUrl":"https://prod-api.example.com","token":"<prod-token>"},
    {"name":"staging","apiUrl":"https://staging-api.example.com","token":"<staging-token>"}
  ]'

2. Point your values file at it:

extraEnvFrom:
  - secretRef:
      name: tmo-ui-clusters

Reinstall, and every cluster shows up. Each entry can also set its own namespace, prometheusUrl, or dcgmService, and "insecure": true if that cluster’s API uses a self-signed certificate.

How do I get a read-only token for a cluster?

Run this in that cluster: it creates a read-only service account (the same access the UI uses) and prints a token:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: { name: tmo-ui-reader, namespace: tensormesh-operator }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: tmo-ui-reader }
rules:
  - { apiGroups: ["lmcache.lmcache.ai"], resources: ["lmcacheengines", "lmcachecoordinators"], verbs: ["get", "list", "watch"] }
  - { apiGroups: [""], resources: ["pods", "namespaces", "nodes", "events"], verbs: ["get", "list"] }
  - { apiGroups: [""], resources: ["pods/proxy", "services/proxy"], verbs: ["get"] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: tmo-ui-reader }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: tmo-ui-reader }
subjects:
  - { kind: ServiceAccount, name: tmo-ui-reader, namespace: tensormesh-operator }
EOF

kubectl -n tensormesh-operator create token tmo-ui-reader --duration=8760h

To skip per-cluster tokens, list the clusters in your values file and put a single TMO_KUBE_TOKEN in the Secret; every cluster inherits it.

# values.yaml
clusters:
  - { name: prod, apiUrl: https://prod-api.example.com }
  - { name: dev, apiUrl: https://dev-api.example.com }
  # add `insecure: true` to any entry whose API uses a self-signed certificate
extraEnvFrom:
  - secretRef: { name: tmo-ui-clusters }

kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
  --from-literal=TMO_KUBE_TOKEN='<shared-token>'

Overview

Installation

Features

Operations

Integrations

Operator UI

Reference

Trend Charts: Connect Your Prometheus

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

GPU trend charts need Prometheus to scrape DCGM

Multiple Clusters

​Trend Charts: Connect Your Prometheus

​Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

​GPU trend charts need Prometheus to scrape DCGM

​Multiple Clusters

​Sharing one token across clusters

​Related

Trend Charts: Connect Your Prometheus

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

GPU trend charts need Prometheus to scrape DCGM

Multiple Clusters

Sharing one token across clusters

Related