Configuration

Most of the dashboard works the moment it’s installed. Two things need connecting: your Prometheus (for the trend charts), and — on GPU clusters — GPU telemetry (for per-GPU detail):

What you see	Where it comes from	What you configure
Fleet, nodes, health, cache internals, quotas	The operator and engines	No setup needed
Trend charts (hit rate, capacity, eviction, throughput over time)	Your Prometheus	`prometheus.url` + engine metrics scraping — two steps
Per-GPU cards (utilization, memory, hardware faults)	NVIDIA’s GPU telemetry exporter	`dcgmService` — optional

Everything is set through Helm values:

Value	What it does	Default
`engineNamespace`	The namespace where your engines run	(the release namespace)
`prometheus.url`	Where the UI reads metric history from	(unset)
`prometheus.service`	Alternative: reach Prometheus through Kubernetes instead of a direct URL (`service.namespace:port`)	(unset)
`dcgmService`	GPU telemetry source (`service.namespace:port`)	(unset)
`clusters`	Watch several clusters in one dashboard	(unset — single cluster)
`ingress.enabled`	Expose the UI beyond port-forward	`false`

Cluster access itself needs no configuration — the UI picks it up automatically when running in-cluster.

Trend Charts: Connect Your Prometheus

Live numbers come straight from the engines; history comes from your Prometheus. Two things must both be true: the UI can reach a Prometheus, and the engine’s metrics are arriving in it. 1. Point the UI at your Prometheus. Find its service with kubectl get svc -A | grep -i prometheus — the name varies by install:

prometheus:
  url: http://<prometheus-service>.<namespace>.svc:9090
# e.g. kube-prometheus-stack-prometheus.monitoring.svc      (default kube-prometheus-stack)
# e.g. kube-prom-stack-kube-prome-prometheus.monitoring.svc (name follows your Helm release)

2. Get the engine metrics into that Prometheus. The engine doesn’t expose metrics for scraping by default. Turn its metrics endpoint on — this goes in the operator chart’s values (it configures the LMCacheEngine), not the UI’s:

engine:
  spec:
    prometheus:
      enabled: true        # the engine serves /metrics on :8080 (the `http` port)

then give Prometheus a ServiceMonitor to scrape it — the operator already creates the <engine>-metrics Service it selects:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: lmcache-engine
  namespace: tensormesh-operator
  labels:
    release: <your-prometheus-release>   # must match your Prometheus' serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: cache-engine
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

observability.enabled=true on the operator chart is a different pipeline — it pushes metrics out through an OTel Collector (for example to a managed metrics service). That only feeds the UI if your collector happens to deliver into the same Prometheus the UI reads. The endpoint + ServiceMonitor wiring above is self-contained and works on any cluster.

No Prometheus in the cluster at all?

Install the community kube-prometheus-stack — it includes the Prometheus Operator that makes ServiceMonitors work:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Then prometheus.url is http://kube-prometheus-stack-prometheus.monitoring.svc:9090, and the ServiceMonitor label above is release: kube-prometheus-stack.

Charts fill once both halves are in place; if they stay empty while everything else works, see Troubleshooting.

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Per-GPU utilization, memory fill, and hardware-fault detail come from NVIDIA’s DCGM exporter — standard on GPU clusters (it ships with the NVIDIA GPU Operator), but not part of Tensormesh. Find your exporter’s service and point dcgmService at it (service.namespace:port):

# find yours: kubectl get svc -A | grep -i dcgm
dcgmService: dcgm-exporter.cw-exporters:9400     # CoreWeave clusters
# dcgmService: dcgm-exporter.gpu-operator:9400   # e.g. an NVIDIA GPU Operator install

If it’s unset or unreachable, the Fleet Map simply shows one row per node instead of per-GPU cards; everything else is unaffected. If your GPU cluster doesn’t have the exporter at all, it ships with the NVIDIA GPU Operator — install that per NVIDIA’s docs and the exporter service appears with it.

GPU trend charts need Prometheus to scrape DCGM (the live per-GPU cards fall back to dcgmService directly, but the over-time charts read history from Prometheus). Confirm it’s being scraped — query DCGM_FI_DEV_GPU_UTIL in Prometheus and expect series back. On some managed clusters the exporter ships a ServiceMonitor labelled for a different Prometheus (e.g. CoreWeave labels theirs environment: grafana-monitoring, while kube-prometheus-stack selects release: <your-release>), so it returns zero series and the GPU trend charts stay empty. Fix it by adding a ServiceMonitor for the DCGM exporter’s service with the label your Prometheus selects.

Multiple Clusters

One dashboard can watch several clusters — the fleet-wide pages merge every cluster’s engines (labeled · prod / · staging), and per-engine pages scope to the right one. The cluster the UI runs in is automatic. Each additional cluster needs just two things: its apiUrl and a read-only token. It’s two steps: 1. Put your clusters (with tokens) in a Secret:

kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
  --from-literal=TMO_CLUSTERS='[
    {"name":"prod","apiUrl":"https://prod-api.example.com","token":"<prod-token>"},
    {"name":"staging","apiUrl":"https://staging-api.example.com","token":"<staging-token>"}
  ]'

2. Point your values file at it:

extraEnvFrom:
  - secretRef:
      name: tmo-ui-clusters

Reinstall, and every cluster shows up. Each entry can also set its own namespace, prometheusUrl, or dcgmService — and "insecure": true if that cluster’s API uses a self-signed certificate.

How do I get a read-only token for a cluster?

Run this in that cluster — it creates a read-only service account (the same access the UI uses) and prints a token:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: { name: tmo-ui-reader, namespace: tensormesh-operator }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: tmo-ui-reader }
rules:
  - { apiGroups: ["lmcache.lmcache.ai"], resources: ["lmcacheengines", "lmcachecoordinators"], verbs: ["get", "list", "watch"] }
  - { apiGroups: [""], resources: ["pods", "namespaces", "nodes", "events"], verbs: ["get", "list"] }
  - { apiGroups: [""], resources: ["pods/proxy", "services/proxy"], verbs: ["get"] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: tmo-ui-reader }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: tmo-ui-reader }
subjects:
  - { kind: ServiceAccount, name: tmo-ui-reader, namespace: tensormesh-operator }
EOF

kubectl -n tensormesh-operator create token tmo-ui-reader --duration=8760h

One token for every cluster? Skip the per-cluster tokens — list the clusters in your values file and put a single TMO_KUBE_TOKEN in the Secret; every cluster inherits it.

# values.yaml
clusters:
  - { name: prod, apiUrl: https://prod-api.example.com }
  - { name: dev, apiUrl: https://dev-api.example.com }
  # add `insecure: true` to any entry whose API uses a self-signed certificate
extraEnvFrom:
  - secretRef: { name: tmo-ui-clusters }

kubectl create secret generic tmo-ui-clusters -n tensormesh-operator \
  --from-literal=TMO_KUBE_TOKEN='<shared-token>'

Installation

Feature List

Integrations

Management

Operator UI

Reference

Trend Charts: Connect Your Prometheus

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Multiple Clusters

​Trend Charts: Connect Your Prometheus

​Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

​Multiple Clusters

​Related

Trend Charts: Connect Your Prometheus

Per-GPU Detail: Connect GPU Telemetry (GPU clusters)

Multiple Clusters

Related