> ## Documentation Index > Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt > Use this file to discover all available pages before exploring further. # Architecture > What the chart deploys, how the pieces connect, and which resources are cluster-scoped (and can orphan). The chart installs a controller (the **operator**) that reconciles `LMCacheEngine` custom resources into a **DaemonSet of LMCache cache servers**, and — when observability is enabled — wires those engines to an **OpenTelemetry Collector** that fans metrics and traces out to Prometheus and Tempo. ## What gets created, and by whom Every piece follows the same pattern: **the chart writes a declaration (a CR); a controller reconciles it into a running thing.** The chart runs exactly **one** controller itself — the Tensormesh Operator. The Collector and Tempo are reconciled by operators **you pre-install**; the chart only submits their CRs. This table is the precise "which thing belongs where": | Declaration (the chart creates this) | Reconciled by | Becomes | Scope | | --------------------------------------- | ----------------------------------------- | ------------------------------------- | ----------- | | `LMCacheEngine` CR | **Tensormesh Operator** (this chart) | engine DaemonSet — 1 pod per GPU node | Namespaced | | `OpenTelemetryCollector` CR | **OpenTelemetry Operator** (prerequisite) | OTel Collector pod | Namespaced | | `TempoMonolithic` CR | **Tempo Operator** (prerequisite) | Tempo instance | Namespaced | | `ServiceMonitor` | **Prometheus Operator** (prerequisite) | a scrape target (no new pod) | Namespaced | | CRD `lmcacheengines.lmcache.lmcache.ai` | — (registered with the API server) | defines the `LMCacheEngine` kind | **Cluster** | The Tensormesh Operator itself runs as a plain `Deployment` (no CR needed) and owns the engine DaemonSet's lifecycle. ## Data flow ```text theme={null} vLLM pod ──hostIPC / MP connector──▶ LMCache engine LMCache engine ──OTLP gRPC───────────▶ OTel Collector OTel Collector ──Prometheus scrape───▶ Prometheus (via ServiceMonitor) OTel Collector ──OTLP traces─────────▶ Tempo / external backend ``` | Hop | Transport | Port | | ---------------------------- | ---------------------------------- | ------------- | | vLLM → engine | shared-memory IPC (`hostIPC`) | — (same node) | | engine → Collector | OTLP gRPC | `4317` | | Collector → Prometheus | scrape of the Collector's exporter | `8889` | | Collector → itself | `otelcol_*` internal metrics | `8888` | | Collector → Tempo / external | OTLP | per backend | See [Observability](/operator/observability) for enabling and verifying this path. ## Resources the chart creates Everything the chart can render, with its **scope** — this is the table to consult before a cleanup, because **cluster-scoped resources survive a namespace delete** and must be removed explicitly. | Resource | API group | Scope | Created when | Cleanup notes | | ---------------------------------------------------------------- | ----------------------------- | ----------- | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **CustomResourceDefinition** `lmcacheengines.lmcache.lmcache.ai` | `apiextensions.k8s.io` | **Cluster** | `crds.enabled` (default `true`) | ⚠️ **Kept on `helm uninstall`** (`resource-policy: keep`). Survives namespace delete. Remove with `kubectl delete crd …`. | | **ClusterRole** (operator) | `rbac.authorization.k8s.io` | **Cluster** | `operator.rbac.create` | ⚠️ Survives namespace delete — sweep manually. | | **ClusterRoleBinding** (operator) | `rbac.authorization.k8s.io` | **Cluster** | `operator.rbac.create` | ⚠️ Survives namespace delete — sweep manually. | | **ClusterRole / ClusterRoleBinding** (pre-delete hook) | `rbac.authorization.k8s.io` | **Cluster** | `helm uninstall` hook | Created and removed by the hook; can orphan if uninstall fails mid-way. | | `LMCacheEngine` (CR) | `lmcache.lmcache.ai/v1alpha1` | Namespaced | `engine.enabled` (default `true`) | Has a `lmcache.ai/cleanup` **finalizer** — if the operator is gone, it blocks namespace deletion. See [troubleshooting](/operator/installation/troubleshooting#cleaning-up-an-inconsistent-or-orphaned-install). | | `OpenTelemetryCollector` (CR) | `opentelemetry.io` | Namespaced | `observability.enabled` | Goes with the namespace. | | `TempoMonolithic` (CR) | `tempo.grafana.com` | Namespaced | `observability.traces.tempoCR.enabled` | Goes with the namespace. | | `ServiceMonitor` (CR) | `monitoring.coreos.com` | Namespaced | `operator.serviceMonitor.enabled` and/or `observability.prometheus.serviceMonitor.enabled` | Goes with the namespace. | | Deployment (controller-manager) | `apps` | Namespaced | `operator.enabled` | Goes with the namespace. | | DaemonSet (engine) | `apps` | Namespaced | created by the operator from the CR | Removed when the CR/namespace is. | | Service (metrics) | core | Namespaced | `operator.enabled` | Goes with the namespace. | | ServiceAccount (operator) | core | Namespaced | `operator.serviceAccount.create` | Goes with the namespace. | | Role / RoleBinding (operator) | `rbac.authorization.k8s.io` | Namespaced | `operator.rbac.create` | Goes with the namespace. | | ServiceAccount + RoleBinding (privileged SCC) | core / `rbac…` | Namespaced | `openshift.enabled` | Goes with the namespace. Binds to the built-in `system:openshift:scc:privileged` ClusterRole — **do not delete that**. | | NetworkPolicy | `networking.k8s.io` | Namespaced | `operator.networkPolicy.enabled` | Goes with the namespace. | | Job + ServiceAccount + Role + RoleBinding (pre-delete hook) | batch / core / `rbac…` | Namespaced | `helm uninstall` hook | Transient; removed after the hook runs. | ### The three things a namespace delete leaves behind If you `kubectl delete namespace` instead of `helm uninstall`, these **cluster-scoped** resources orphan and need explicit cleanup: 1. **CRD** `lmcacheengines.lmcache.lmcache.ai` 2. **ClusterRole** for the operator 3. **ClusterRoleBinding** for the operator ```bash theme={null} kubectl delete crd lmcacheengines.lmcache.lmcache.ai kubectl get clusterrole,clusterrolebinding | grep -i tensormesh # delete what it lists ``` See [Troubleshooting → Cleaning up an inconsistent or orphaned install](/operator/installation/troubleshooting#cleaning-up-an-inconsistent-or-orphaned-install) for the full recipe (including clearing the `LMCacheEngine` finalizer first). ## Next steps Every `values.yaml` key and example overlays. The OTel Collector / Tempo / Prometheus wiring in detail.