Observability - Tensormesh User Documentation

The chart can deploy an OpenTelemetry observability stack that collects metrics and traces from the LMCache engine and ships them to the backends you choose:

An OTel Collector receives OTLP from the engine pods.
Metrics are exposed for Prometheus to scrape (ServiceMonitor) and/or pushed to an external Prometheus-compatible endpoint (remoteWrite).
Traces are sent to an in-cluster Tempo, an external backend (e.g. Grafana Cloud), or both.

It’s off by default — opt in with observability.enabled=true.

The chart deploys the Collector and (optionally) Tempo. It does not ship Grafana or any dashboards — bring your own Grafana and point it at Prometheus (metrics) and Tempo (traces).

Prerequisites

The chart creates custom resources but does not install the operators that reconcile them. Install these first:

Operator	Provides	Needed for
OpenTelemetry Operator	`OpenTelemetryCollector` CRD	the Collector (always, when observability is enabled)
Prometheus Operator	`ServiceMonitor` CRD	in-cluster metric scraping (`prometheus.serviceMonitor.enabled`)
Tempo Operator	`TempoMonolithic` CRD	an in-cluster trace backend (`traces.tempoCR.enabled`)

kubectl get crd | grep -E 'opentelemetrycollectors|servicemonitors.monitoring|tempomonolithics'

When observability.enabled=true, the chart requires the OpenTelemetryCollector CRD — the install fails fast if the OpenTelemetry Operator isn’t present.

Quickstart — metrics only

The smallest useful config: a Collector + a ServiceMonitor, no trace backend. Put it in a values file:

observability-values.yaml

openshift:
  enabled: true                 # OpenShift only — remove on plain Kubernetes
# Operator image is supplied by the chart version (--version 0.4.5).
observability:
  enabled: true
  prometheus:
    serviceMonitor:
      enabled: true

helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version 0.4.5 -n tensormesh-operator --create-namespace \
  -f observability-values.yaml \
  --wait

This creates the OpenTelemetryCollector CR (the OTel Operator reconciles it into a Collector Deployment) and a ServiceMonitor. The engine is auto-wired to send OTLP to the Collector — confirm:

kubectl get lmcacheengine -n tensormesh-operator -o yaml | grep -A3 extraArgs
# extraArgs:
# - --enable-tracing
# - --otlp-endpoint
# - http://<release>-otel-collector-collector.<namespace>.svc:4317

Collector ports

Port	Service	Purpose
`4317` / `4318`	`…-collector`	OTLP gRPC / HTTP receiver — the engine sends here
`8889`	`…-collector`	Prometheus exporter — LMCache application metrics, for scraping (empty until the engine sends data)
`8888`	`…-collector-monitoring`	the Collector’s own internal metrics (`otelcol_*`)

Exporting metrics

Two independent options — use either or both: Scrape in-cluster (ServiceMonitor) — your Prometheus discovers and scrapes the Collector:

observability:
  prometheus:
    serviceMonitor:
      enabled: true
      interval: 30s

Push to an external Prometheus (remoteWrite) — e.g. Grafana Cloud. Store the auth header in a Secret and reference it from the Collector’s environment (see Authenticating to external backends):

observability:
  otelCollector:
    envFrom:
      - secretRef:
          name: grafana-cloud-credentials
  prometheus:
    remoteWrite:
      enabled: true
      endpoint: https://<your-prometheus-endpoint>/api/prom/push
      headers:
        Authorization: ${env:GC_METRICS_AUTH}    # resolved from the Secret
      externalLabels:
        cluster: my-cluster

Exporting traces

The engine emits traces once observability is enabled; they need a backend to land in. In-cluster Tempo (traces.tempoCR.enabled) — the chart creates a TempoMonolithic CR (reconciled by the Tempo Operator) and wires the Collector to export to it:

observability:
  enabled: true
  traces:
    tempoCR:
      enabled: true

External backend (BYOB — e.g. Grafana Cloud Tempo) — point the Collector at an external OTLP endpoint with auth from a Secret:

observability:
  otelCollector:
    envFrom:
      - secretRef:
          name: grafana-cloud-credentials
  traces:
    externalEndpoint:
      enabled: true
      endpoint: <your-tempo-endpoint>:443
      tls:
        insecure: false
      headers:
        Authorization: ${env:GC_TRACES_AUTH}

Both — enable tempoCR and externalEndpoint together and the Collector fans traces out to both backends simultaneously. Without any trace backend, traces are only visible in the Collector’s debug log (below).

Authenticating to external backends

Never put tokens in values.yaml or --set. Store them in a Secret as full auth headers, mount it into the Collector via otelCollector.envFrom, and reference the env vars with ${env:VAR} in the endpoint headers:

# Build the auth header(s) — example for a Grafana Cloud-style Basic token.
# Replace <INSTANCE_ID> and <API_TOKEN> with your own; do NOT commit these.
TRACES_AUTH="Basic $(printf '%s:%s' '<TRACES_INSTANCE_ID>' '<API_TOKEN>' | base64)"
METRICS_AUTH="Basic $(printf '%s:%s' '<METRICS_INSTANCE_ID>' '<API_TOKEN>' | base64)"

kubectl create secret generic grafana-cloud-credentials \
  -n tensormesh-operator \
  --from-literal=GC_TRACES_AUTH="$TRACES_AUTH" \
  --from-literal=GC_METRICS_AUTH="$METRICS_AUTH"

The Collector loads these as environment variables (via envFrom), and ${env:GC_TRACES_AUTH} / ${env:GC_METRICS_AUTH} in the values above resolve at runtime — so the token never appears in the chart values or the rendered CR.

Debug exporter

For development, observability.otelCollector.debug=true adds a debug exporter that logs every received metric and trace to the Collector’s stdout (verbosity detailed). Useful for confirming data shape; turn it off in production (it’s noisy).

kubectl logs -n tensormesh-operator deploy/<release>-otel-collector-collector -f

Verify telemetry is flowing

Telemetry is only produced when the engine does work — send inference through vLLM (see the end-to-end example) first. Metrics export on an interval (~60 s), so allow a minute after traffic. The source of truth is the Collector’s own counters, not log volume:

kubectl port-forward -n tensormesh-operator \
  svc/<release>-otel-collector-collector-monitoring 8888:8888
# other terminal:
curl -s localhost:8888/metrics | grep -E 'otelcol_receiver_accepted_(metric_points|spans)'

A non-zero otelcol_receiver_accepted_metric_points / _spans confirms the engine’s OTLP reached the Collector. You can also see the engine’s application metrics on the Prometheus exporter (8889 on the main …-collector service):

kubectl port-forward -n tensormesh-operator \
  svc/<release>-otel-collector-collector 8889:8889
# other terminal:
curl -s http://localhost:8889/metrics | grep lmcache | head -20

# HELP lmcache_mp_l1_memory_usage_bytes Bytes currently held in L1 cache
# TYPE lmcache_mp_l1_memory_usage_bytes gauge
lmcache_mp_l1_memory_usage_bytes{instance="<id>",otel_scope_name="lmcache.l1_manager"} 0
# HELP lmcache_mp_l1_eviction_loop_ticks_total L1 eviction-loop iterations (every cycle)
# TYPE lmcache_mp_l1_eviction_loop_ticks_total counter
lmcache_mp_l1_eviction_loop_ticks_total{instance="<id>",otel_scope_name="lmcache.l1"} 227
# HELP lmcache_mp_active_prefetch_jobs Number of active prefetch jobs
# TYPE lmcache_mp_active_prefetch_jobs gauge
lmcache_mp_active_prefetch_jobs{instance="<id>",otel_scope_name="lmcache.mp_engine"} 0

A batch of lmcache_mp_* series = the engine is exporting through the Collector. Empty means no traffic has hit the engine yet. Finally, run the bundled assertion:

helm test tensormesh-operator -n tensormesh-operator --logs
# the …-test-otel-metrics pod scrapes :8888 and asserts otelcol_ metrics exist

Don’t verify with a kubectl run … --image=curlimages/curl -- curl … pod — that image’s entrypoint is already curl, so the args double up and it fails. Use the port-forward.

Disabling and toggling

observability.enabled=false (the default) produces zero observability resources, and CRD validation is skipped — so a default install needs none of the operators above. Toggling is clean: helm upgrade with enabled=true creates the Collector CR; setting it back to false removes the CR and its Collector pods.

Troubleshooting

otelcol_receiver_accepted_* stays at 0 after traffic The engine isn’t reaching the Collector. Check, in order:

The engine CR has the tracing extraArgs (above). If not, confirm observability.enabled=true was applied: helm get values tensormesh-operator -n tensormesh-operator.
The OTLP endpoint resolves — the Collector’s …-collector service on port 4317.
The engine pod restarted after the flags were added (its age should post-date enabling observability).

Auth errors (401/403) in Collector logs (external backend) The Secret’s auth header is wrong or expired. Verify the Secret and the ${env:…} references match. Collector pod never becomes Ready / install fails on a missing CRD The OpenTelemetry Operator (or Tempo Operator, if tempoCR.enabled) isn’t installed — re-check the prerequisite CRDs.

Next steps

End-to-end example

Full install → inference → benchmark, including generating the traffic that produces telemetry.

Install with Helm

All chart values, including the observability.* keys.

​Prerequisites

​Quickstart — metrics only

​Collector ports

​Exporting metrics

​Exporting traces

​Authenticating to external backends

​Debug exporter

​Verify telemetry is flowing

​Disabling and toggling

​Troubleshooting

​Next steps

End-to-end example

Install with Helm

Prerequisites

Quickstart — metrics only

Collector ports

Exporting metrics

Exporting traces

Authenticating to external backends

Debug exporter

Verify telemetry is flowing

Disabling and toggling

Troubleshooting

Next steps