> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Export LMCache engine metrics and traces via an OpenTelemetry Collector — to in-cluster Prometheus/Tempo or an external backend.

The chart can deploy an **OpenTelemetry observability stack** that collects metrics and
traces from the LMCache engine and ships them to the backends you choose:

* An **OTel Collector** receives OTLP from the engine pods.
* **Metrics** are exposed for Prometheus to scrape (`ServiceMonitor`) and/or pushed to an
  external Prometheus-compatible endpoint (`remoteWrite`).
* **Traces** are sent to an in-cluster **Tempo**, an **external** backend (e.g. Grafana
  Cloud), or both.

It's **off by default** — opt in with `observability.enabled=true`.

<Note>
  The chart deploys the Collector and (optionally) Tempo. It does **not** ship Grafana or
  any dashboards — bring your own Grafana and point it at Prometheus (metrics) and Tempo
  (traces).
</Note>

## Prerequisites

The chart creates custom resources but does **not** install the operators that reconcile
them. Install these first:

| Operator                   | Provides                     | Needed for                                                       |
| -------------------------- | ---------------------------- | ---------------------------------------------------------------- |
| **OpenTelemetry Operator** | `OpenTelemetryCollector` CRD | the Collector (always, when observability is enabled)            |
| **Prometheus Operator**    | `ServiceMonitor` CRD         | in-cluster metric scraping (`prometheus.serviceMonitor.enabled`) |
| **Tempo Operator**         | `TempoMonolithic` CRD        | an in-cluster trace backend (`traces.tempoCR.enabled`)           |

```bash theme={null}
kubectl get crd | grep -E 'opentelemetrycollectors|servicemonitors.monitoring|tempomonolithics'
```

<Warning>
  When `observability.enabled=true`, the chart **requires** the `OpenTelemetryCollector`
  CRD — the install fails fast if the OpenTelemetry Operator isn't present.
</Warning>

## Quickstart — metrics only

The smallest useful config: a Collector + a `ServiceMonitor`, no trace backend. Put it in
a values file:

```yaml observability-values.yaml theme={null}
openshift:
  enabled: true                 # OpenShift only — remove on plain Kubernetes
# Operator image is supplied by the chart version (--version 0.4.5).
observability:
  enabled: true
  prometheus:
    serviceMonitor:
      enabled: true
```

```bash theme={null}
helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version 0.4.5 -n tensormesh-operator --create-namespace \
  -f observability-values.yaml \
  --wait
```

This creates the `OpenTelemetryCollector` CR (the OTel Operator reconciles it into a
Collector Deployment) and a `ServiceMonitor`. The engine is **auto-wired** to send OTLP to
the Collector — confirm:

```bash theme={null}
kubectl get lmcacheengine -n tensormesh-operator -o yaml | grep -A3 extraArgs
# extraArgs:
# - --enable-tracing
# - --otlp-endpoint
# - http://<release>-otel-collector-collector.<namespace>.svc:4317
```

### Collector ports

| Port            | Service                  | Purpose                                                                                                   |
| --------------- | ------------------------ | --------------------------------------------------------------------------------------------------------- |
| `4317` / `4318` | `…-collector`            | OTLP gRPC / HTTP **receiver** — the engine sends here                                                     |
| `8889`          | `…-collector`            | **Prometheus exporter** — LMCache *application* metrics, for scraping (empty until the engine sends data) |
| `8888`          | `…-collector-monitoring` | the Collector's **own internal** metrics (`otelcol_*`)                                                    |

## Exporting metrics

Two independent options — use either or both:

**Scrape in-cluster (ServiceMonitor)** — your Prometheus discovers and scrapes the Collector:

```yaml theme={null}
observability:
  prometheus:
    serviceMonitor:
      enabled: true
      interval: 30s
```

**Push to an external Prometheus (`remoteWrite`)** — e.g. Grafana Cloud. Store the auth
header in a Secret and reference it from the Collector's environment (see
[Authenticating to external backends](#authenticating-to-external-backends)):

```yaml theme={null}
observability:
  otelCollector:
    envFrom:
      - secretRef:
          name: grafana-cloud-credentials
  prometheus:
    remoteWrite:
      enabled: true
      endpoint: https://<your-prometheus-endpoint>/api/prom/push
      headers:
        Authorization: ${env:GC_METRICS_AUTH}    # resolved from the Secret
      externalLabels:
        cluster: my-cluster
```

## Exporting traces

The engine emits traces once observability is enabled; they need a backend to land in.

**In-cluster Tempo** (`traces.tempoCR.enabled`) — the chart creates a `TempoMonolithic` CR
(reconciled by the Tempo Operator) and wires the Collector to export to it:

```yaml theme={null}
observability:
  enabled: true
  traces:
    tempoCR:
      enabled: true
```

**External backend** (BYOB — e.g. Grafana Cloud Tempo) — point the Collector at an external
OTLP endpoint with auth from a Secret:

```yaml theme={null}
observability:
  otelCollector:
    envFrom:
      - secretRef:
          name: grafana-cloud-credentials
  traces:
    externalEndpoint:
      enabled: true
      endpoint: <your-tempo-endpoint>:443
      tls:
        insecure: false
      headers:
        Authorization: ${env:GC_TRACES_AUTH}
```

**Both** — enable `tempoCR` *and* `externalEndpoint` together and the Collector fans traces
out to both backends simultaneously. Without any trace backend, traces are only visible in
the Collector's debug log (below).

## Authenticating to external backends

Never put tokens in `values.yaml` or `--set`. Store them in a Secret as full auth headers,
mount it into the Collector via `otelCollector.envFrom`, and reference the env vars with
`${env:VAR}` in the endpoint `headers`:

```bash theme={null}
# Build the auth header(s) — example for a Grafana Cloud-style Basic token.
# Replace <INSTANCE_ID> and <API_TOKEN> with your own; do NOT commit these.
TRACES_AUTH="Basic $(printf '%s:%s' '<TRACES_INSTANCE_ID>' '<API_TOKEN>' | base64)"
METRICS_AUTH="Basic $(printf '%s:%s' '<METRICS_INSTANCE_ID>' '<API_TOKEN>' | base64)"

kubectl create secret generic grafana-cloud-credentials \
  -n tensormesh-operator \
  --from-literal=GC_TRACES_AUTH="$TRACES_AUTH" \
  --from-literal=GC_METRICS_AUTH="$METRICS_AUTH"
```

The Collector loads these as environment variables (via `envFrom`), and
`${env:GC_TRACES_AUTH}` / `${env:GC_METRICS_AUTH}` in the values above resolve at runtime —
so the token never appears in the chart values or the rendered CR.

## Debug exporter

For development, `observability.otelCollector.debug=true` adds a debug exporter that logs
every received metric and trace to the Collector's stdout (verbosity `detailed`). Useful
for confirming data shape; **turn it off in production** (it's noisy).

```bash theme={null}
kubectl logs -n tensormesh-operator deploy/<release>-otel-collector-collector -f
```

## Verify telemetry is flowing

Telemetry is only produced when the engine does work — send inference through vLLM (see the
[end-to-end example](/operator/installation/example)) first. Metrics export on an interval
(\~60 s), so allow a minute after traffic.

The **source of truth** is the Collector's own counters, not log volume:

```bash theme={null}
kubectl port-forward -n tensormesh-operator \
  svc/<release>-otel-collector-collector-monitoring 8888:8888
# other terminal:
curl -s localhost:8888/metrics | grep -E 'otelcol_receiver_accepted_(metric_points|spans)'
```

A **non-zero** `otelcol_receiver_accepted_metric_points` / `_spans` confirms the engine's
OTLP reached the Collector.

You can also see the engine's **application** metrics on the Prometheus exporter (`8889` on
the main `…-collector` service):

```bash theme={null}
kubectl port-forward -n tensormesh-operator \
  svc/<release>-otel-collector-collector 8889:8889
# other terminal:
curl -s http://localhost:8889/metrics | grep lmcache | head -20
```

```text theme={null}
# HELP lmcache_mp_l1_memory_usage_bytes Bytes currently held in L1 cache
# TYPE lmcache_mp_l1_memory_usage_bytes gauge
lmcache_mp_l1_memory_usage_bytes{instance="<id>",otel_scope_name="lmcache.l1_manager"} 0
# HELP lmcache_mp_l1_eviction_loop_ticks_total L1 eviction-loop iterations (every cycle)
# TYPE lmcache_mp_l1_eviction_loop_ticks_total counter
lmcache_mp_l1_eviction_loop_ticks_total{instance="<id>",otel_scope_name="lmcache.l1"} 227
# HELP lmcache_mp_active_prefetch_jobs Number of active prefetch jobs
# TYPE lmcache_mp_active_prefetch_jobs gauge
lmcache_mp_active_prefetch_jobs{instance="<id>",otel_scope_name="lmcache.mp_engine"} 0
```

A batch of `lmcache_mp_*` series = the engine is exporting through the Collector. Empty
means no traffic has hit the engine yet. Finally, run the bundled assertion:

```bash theme={null}
helm test tensormesh-operator -n tensormesh-operator --logs
# the …-test-otel-metrics pod scrapes :8888 and asserts otelcol_ metrics exist
```

<Warning>
  Don't verify with a `kubectl run … --image=curlimages/curl -- curl …` pod — that image's
  entrypoint is already `curl`, so the args double up and it fails. Use the port-forward.
</Warning>

## Disabling and toggling

`observability.enabled=false` (the default) produces **zero** observability resources, and
CRD validation is skipped — so a default install needs none of the operators above. Toggling
is clean: `helm upgrade` with `enabled=true` creates the Collector CR; setting it back to
`false` removes the CR and its Collector pods.

## Troubleshooting

**`otelcol_receiver_accepted_*` stays at 0 after traffic**
The engine isn't reaching the Collector. Check, in order:

1. The engine CR has the tracing `extraArgs` (above). If not, confirm `observability.enabled=true`
   was applied: `helm get values tensormesh-operator -n tensormesh-operator`.
2. The OTLP endpoint resolves — the Collector's `…-collector` service on port `4317`.
3. The engine pod restarted after the flags were added (its age should post-date enabling observability).

**Auth errors (401/403) in Collector logs (external backend)**
The Secret's auth header is wrong or expired. Verify the Secret and the `${env:…}` references match.

**Collector pod never becomes Ready / install fails on a missing CRD**
The OpenTelemetry Operator (or Tempo Operator, if `tempoCR.enabled`) isn't installed — re-check the prerequisite CRDs.

## Next steps

<CardGroup cols={2}>
  <Card title="End-to-end example" icon="rocket-launch" href="/operator/installation/example">
    Full install → inference → benchmark, including generating the traffic that produces telemetry.
  </Card>

  <Card title="Install with Helm" icon="ship-wheel" href="/operator/installation/helm">
    All chart values, including the `observability.*` keys.
  </Card>
</CardGroup>