Skip to main content
The chart installs a controller (the operator) that reconciles LMCacheEngine custom resources into a DaemonSet of LMCache cache servers, and — when observability is enabled — wires those engines to an OpenTelemetry Collector that fans metrics and traces out to Prometheus and Tempo.

​
What gets created, and by whom

Every piece follows the same pattern: the chart writes a declaration (a CR); a controller reconciles it into a running thing. The chart runs exactly one controller itself — the Tensormesh Operator. The Collector and Tempo are reconciled by operators you pre-install; the chart only submits their CRs. This table is the precise “which thing belongs where”:
Declaration (the chart creates this)Reconciled byBecomesScope
LMCacheEngine CRTensormesh Operator (this chart)engine DaemonSet — 1 pod per GPU nodeNamespaced
OpenTelemetryCollector CROpenTelemetry Operator (prerequisite)OTel Collector podNamespaced
TempoMonolithic CRTempo Operator (prerequisite)Tempo instanceNamespaced
ServiceMonitorPrometheus Operator (prerequisite)a scrape target (no new pod)Namespaced
CRD lmcacheengines.lmcache.lmcache.ai— (registered with the API server)defines the LMCacheEngine kindCluster
The Tensormesh Operator itself runs as a plain Deployment (no CR needed) and owns the engine DaemonSet’s lifecycle.

​
Data flow

vLLM pod  ──hostIPC / MP connector──▶  LMCache engine
LMCache engine  ──OTLP gRPC───────────▶  OTel Collector
OTel Collector  ──Prometheus scrape───▶  Prometheus   (via ServiceMonitor)
OTel Collector  ──OTLP traces─────────▶  Tempo / external backend
HopTransportPort
vLLM → engineshared-memory IPC (hostIPC)— (same node)
engine → CollectorOTLP gRPC4317
Collector → Prometheusscrape of the Collector’s exporter8889
Collector → itselfotelcol_* internal metrics8888
Collector → Tempo / externalOTLPper backend
See Observability for enabling and verifying this path.

​
Resources the chart creates

Everything the chart can render, with its scope — this is the table to consult before a cleanup, because cluster-scoped resources survive a namespace delete and must be removed explicitly.
ResourceAPI groupScopeCreated whenCleanup notes
CustomResourceDefinition lmcacheengines.lmcache.lmcache.aiapiextensions.k8s.ioClustercrds.enabled (default true)⚠️ Kept on helm uninstall (resource-policy: keep). Survives namespace delete. Remove with kubectl delete crd ….
ClusterRole (operator)rbac.authorization.k8s.ioClusteroperator.rbac.create⚠️ Survives namespace delete — sweep manually.
ClusterRoleBinding (operator)rbac.authorization.k8s.ioClusteroperator.rbac.create⚠️ Survives namespace delete — sweep manually.
ClusterRole / ClusterRoleBinding (pre-delete hook)rbac.authorization.k8s.ioClusterhelm uninstall hookCreated and removed by the hook; can orphan if uninstall fails mid-way.
LMCacheEngine (CR)lmcache.lmcache.ai/v1alpha1Namespacedengine.enabled (default true)Has a lmcache.ai/cleanup finalizer — if the operator is gone, it blocks namespace deletion. See troubleshooting.
OpenTelemetryCollector (CR)opentelemetry.ioNamespacedobservability.enabledGoes with the namespace.
TempoMonolithic (CR)tempo.grafana.comNamespacedobservability.traces.tempoCR.enabledGoes with the namespace.
ServiceMonitor (CR)monitoring.coreos.comNamespacedoperator.serviceMonitor.enabled and/or observability.prometheus.serviceMonitor.enabledGoes with the namespace.
Deployment (controller-manager)appsNamespacedoperator.enabledGoes with the namespace.
DaemonSet (engine)appsNamespacedcreated by the operator from the CRRemoved when the CR/namespace is.
Service (metrics)coreNamespacedoperator.enabledGoes with the namespace.
ServiceAccount (operator)coreNamespacedoperator.serviceAccount.createGoes with the namespace.
Role / RoleBinding (operator)rbac.authorization.k8s.ioNamespacedoperator.rbac.createGoes with the namespace.
ServiceAccount + RoleBinding (privileged SCC)core / rbac…Namespacedopenshift.enabledGoes with the namespace. Binds to the built-in system:openshift:scc:privileged ClusterRole — do not delete that.
NetworkPolicynetworking.k8s.ioNamespacedoperator.networkPolicy.enabledGoes with the namespace.
Job + ServiceAccount + Role + RoleBinding (pre-delete hook)batch / core / rbac…Namespacedhelm uninstall hookTransient; removed after the hook runs.

​
The three things a namespace delete leaves behind

If you kubectl delete namespace instead of helm uninstall, these cluster-scoped resources orphan and need explicit cleanup:
  1. CRD lmcacheengines.lmcache.lmcache.ai
  2. ClusterRole for the operator
  3. ClusterRoleBinding for the operator
kubectl delete crd lmcacheengines.lmcache.lmcache.ai
kubectl get clusterrole,clusterrolebinding | grep -i tensormesh   # delete what it lists
See Troubleshooting → Cleaning up an inconsistent or orphaned install for the full recipe (including clearing the LMCacheEngine finalizer first).

​
Next steps

Configuration

Every values.yaml key and example overlays.

Observability

The OTel Collector / Tempo / Prometheus wiring in detail.