> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Install from the OpenShift software catalog (certified operator)

> Install the Red Hat certified Tensormesh Platform operator from the OpenShift software catalog (OperatorHub), then complete the cluster steps an OLM install leaves to you.

Tensormesh Platform is a **Red Hat Certified Operator**: on OpenShift you install it from the
in-cluster software catalog, and Operator Lifecycle Manager (OLM) installs and upgrades the
operator. Its images are certified by Red Hat and distributed from the Tensormesh registry, so
you add your registry access token to the cluster **before** installing. New to Tensormesh
Platform? Start with the [Introduction](https://docs.tensormesh.ai/v1.0.1/introduction.md).

> [!NOTE]
>
> This is a different install path from [Helm](https://docs.tensormesh.ai/v1.0.1/installation/helm.md). OLM installs the
> operator, the CRDs, the mutating webhooks and their serving certificates, so
> **cert-manager is not needed**. It does **not** create the resources the Helm chart
> creates for you; this page covers them. Don't install from the catalog on a cluster that
> already runs the Helm chart: both own the same CRDs and webhooks. Uninstall the chart first,
> and note that deleting its CRDs also deletes every engine and coordinator it created.

| Step | Required | Why you do it yourself |
| - | - | - |
| [1–3: Registry access token, operator, namespace](#1-add-the-tensormesh-registry-access-token) | Yes | Every image is pulled from the Tensormesh registry. Without the token the operator pod cannot start. |
| [4: Coordinator and engine](#4-create-the-coordinator-and-the-engine) | Yes | The Helm chart creates a coordinator and an engine by default. On OLM you create them yourself. |
| [5: Connect vLLM](#5-connect-your-vllm-deployment) | Yes | You opt your vLLM pods in with a label and an annotation. |
| [6: CacheBlend](#6-cacheblend-optional) or [7: SGLang](#7-sglang-optional) | Alternative to steps 4–5 | Their engines share the host IPC namespace with the serving pods, so a cluster administrator grants a least-privilege SCC. |
| [8: Network access](#8-restrict-network-access-recommended) | Recommended | Limits who can reach the cache server port. |
| [9: Observability](#9-observability-optional) | Optional | The OpenTelemetry Collector needs an operator you install. |
| [Upgrade](#upgrade) and [uninstall](#uninstall) | Day 2 | Image tags live in your custom resources, and custom resources must be deleted before the operator. |

## Prerequisites

> **OpenShift 4.20 or later**
>
> The certified bundle supports OpenShift **4.20 and later**. On an older cluster the
> operator does not appear in the catalog.

> **NVIDIA GPU nodes and the GPU Operator, driver R580 or later**
>
> GPU nodes are **x86\_64** with NVIDIA data-center GPUs. Install the **NVIDIA GPU Operator**
> (with its Node Feature Discovery dependency) from the software catalog and create its
> default `ClusterPolicy`. The serving image is built on CUDA 13.0, so GPU nodes need an
> **NVIDIA driver R580 or later**. The GPU Operator installs the driver too, and which
> version depends on the GPU Operator release: older releases default to an earlier branch
> (v25.3 defaults to R570), so check the driver it deployed. The GPU Operator also labels GPU
> nodes `nvidia.com/gpu.present=true` and registers the `nvidia` RuntimeClass, which the
> cache engines use. On GPU Operator versions before v25.10.0, also
> [enable CDI](https://docs.tensormesh.ai/v1.0.1/installation/troubleshooting.md#pod-stuck-in-containercreating-cdi).
>
> Check that your GPU nodes are labeled and that the driver is R580 or later:
>
> ```bash theme={null}
> oc get nodes -l nvidia.com/gpu.present=true   # must list your GPU nodes
> oc exec -n nvidia-gpu-operator -c nvidia-driver-ctr \
>   "$(oc get pod -n nvidia-gpu-operator -l app.kubernetes.io/component=nvidia-driver -o name | head -1)" \
>   -- nvidia-smi --query-gpu=driver_version --format=csv,noheader   # 580 or later
> ```

> **Node resources**
>
> Each cache engine runs one pod on **every** GPU node. The pod requests 4 CPUs and
> `l1.sizeGB` + 5 GiB of memory (limit 1.5×), on top of your vLLM pods. The serving image is
> about 10 GB, so allow at least 30 GB of free container storage on each GPU node.

> **cluster-admin**
>
> Installing an operator, editing the cluster pull secret and granting an SCC are
> cluster-admin actions.

> **A Tensormesh registry access token**
>
> The access token from the Tensormesh team, the same one used for Helm installs and the
> CLI. If you don't have it, contact [Tensormesh](https://www.tensormesh.ai/contact). Every
> Tensormesh image (the operator, the serving image that also runs the cache engines and the
> coordinator, the LMCache payload and the CacheBlend plugin) is pulled from
> `artifacts.tensormesh.ai`.

> **Network, proxy and FIPS**
>
> The examples download model weights from Hugging Face (`huggingface.co`). The operator
> does not pass the cluster-wide proxy settings to the pods it creates or connects: if your
> cluster uses a proxy, set `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` on your vLLM
> Deployment and through `spec.env` on the engines, and include `.svc`, `.cluster.local` and
> your cluster's pod and service networks in `NO_PROXY`, so that traffic between vLLM, the
> engines and the coordinator stays inside the cluster. Disconnected (air-gapped) clusters
> are not supported in this release, and Tensormesh Platform is not supported on clusters
> installed in FIPS mode: its images are not built for FIPS 140 compliance.

## 1. Add the Tensormesh registry access token

Do this **first**. OLM pulls the operator image as soon as you install, so without the
token the operator pod stays in `ImagePullBackOff`.

Add the token to the cluster's global pull secret, which covers every node and every
namespace. The pull secret holds every registry credential of the cluster, so the commands
run in a private temporary directory, and the token is read from a prompt rather than typed
on the command line:

```bash theme={null}
read -rs -p "Tensormesh registry access token: " TOKEN; echo
(
  cd "$(mktemp -d)" && umask 077
  oc get secret/pull-secret -n openshift-config \
    --template='{{index .data ".dockerconfigjson" | base64decode}}' > pull-secret.json
  oc registry login --registry="artifacts.tensormesh.ai" \
    --auth-basic="x-access-token:$TOKEN" --to=pull-secret.json
  oc set data secret/pull-secret -n openshift-config \
    --from-file=.dockerconfigjson=pull-secret.json
  rm -f pull-secret.json
)
unset TOKEN
```

This is Red Hat's procedure for
[updating the global cluster pull secret](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html-single/images/index#images-update-global-pull-secret_using-image-pull-secrets),
with the Tensormesh registry filled in. The Machine Config Operator rolls the new secret out to
every node without draining or rebooting them. Check the entry, then wait until every machine
config pool has finished updating:

```bash theme={null}
oc get secret/pull-secret -n openshift-config \
  --template='{{index .data ".dockerconfigjson" | base64decode}}' | grep -c artifacts.tensormesh.ai   # 1
oc get mcp   # wait until every pool shows UPDATED True and UPDATING False
```

On hosted control planes (ROSA with HCP, HyperShift), follow Red Hat's procedure for adding a
pull secret to a hosted cluster instead.

## 2. Install the operator

1. In the OpenShift web console, go to **Ecosystem → Software Catalog**.
2. Search for **Tensormesh Platform**; the listing carries the **Certified** badge. Click
   **Install**.
3. Keep the `stable` channel and the default installation mode, **All namespaces on the
   cluster**, and confirm. For **Update approval**, see [Upgrade](#upgrade): with **Manual**,
   you decide when the operator upgrades, but OLM then applies Manual approval to every
   operator in the `openshift-operators` namespace and upgrades them together.

OLM installs the operator into the `openshift-operators` namespace, together with the three
CRDs and the mutating webhooks. Check that it is running:

```bash theme={null}
oc get csv -n openshift-operators | grep tensormesh-platform   # PHASE: Succeeded
oc get pods -n openshift-operators | grep tensormesh-platform  # STATUS: Running
oc get crd | grep lmcache.lmcache.ai
# cacheblendengines.lmcache.lmcache.ai
# lmcachecoordinators.lmcache.lmcache.ai
# lmcacheengines.lmcache.lmcache.ai
```

If the operator pod is in `ImagePullBackOff`, check step 1, then delete the pod once the
pull-secret rollout has finished; it restarts and pulls the image.

## 3. Prepare a workload namespace

The coordinator, the cache engine and the vLLM pods that use it must all run in **the same
namespace**: an engine finds its coordinator in its own namespace, the webhook looks up the
engine in the pod's namespace, and OpenShift gives each namespace its own UID range, so pods
in different namespaces can't share memory with an engine. This page uses `llm-serving`; if
you use another namespace, replace `llm-serving` everywhere below.

Create the namespace and a ServiceAccount for the Tensormesh workloads:

```bash theme={null}
oc create namespace llm-serving
oc create serviceaccount tensormesh -n llm-serving
```

The coordinator, the engines and the vLLM examples below all run as the `tensormesh`
ServiceAccount. The default engine needs no extra permissions; [CacheBlend](#6-cacheblend-optional)
and [SGLang](#7-sglang-optional) need host IPC, which you grant to this ServiceAccount there.

## 4. Create the coordinator and the engine

The **`LMCacheCoordinator`** tracks which cache engines are alive and drives cache eviction
across them; engines register with it at startup. Every setup needs one coordinator per
namespace.

Then choose **one engine** for the namespace. Each engine runs a pod on every GPU node and
reserves its `l1.sizeGB` of memory there:

* the default **`LMCacheEngine`** below, for vLLM (steps 4 and 5);
* a **`CacheBlendEngine`**, for RAG workloads ([step 6](#6-cacheblend-optional));
* an **`LMCacheEngine` in host-IPC mode**, for SGLang ([step 7](#7-sglang-optional)).

You can run two engines side by side only if both memory reservations fit on every GPU node.

Set `spec.image` on every resource as shown. If it is unset, the operator uses a `latest`
tag, which for engines is the public community image, not the certified build. The samples in
the console's **Create instance** form carry the certified images but no `serviceAccountName`:
add `serviceAccountName: tensormesh` when you start from them.

```yaml coordinator.yaml theme={null}
apiVersion: lmcache.lmcache.ai/v1alpha1
kind: LMCacheCoordinator
metadata:
  name: coordinator
  namespace: llm-serving
spec:
  serviceAccountName: tensormesh
  image:
    repository: artifacts.tensormesh.ai/tensormesh-production/images/coordinator-controllers
    tag: v0.5.6-ubi
```

```yaml engine.yaml theme={null}
apiVersion: lmcache.lmcache.ai/v1alpha1
kind: LMCacheEngine
metadata:
  name: engine
  namespace: llm-serving
spec:
  gpuVendor: nvidia
  serviceAccountName: tensormesh
  image:
    repository: artifacts.tensormesh.ai/tensormesh-production/images/vllm-openai
    tag: v0.5.6-ubi
  coordinator:
    ref:
      name: coordinator
  l1:
    sizeGB: 40           # first cache tier: CPU memory reserved on each GPU node
```

With this `l1`, the engine pod requests 45 GiB of memory on every GPU node (see
[Node resources](#prerequisites)); size `l1` so that it fits next to your vLLM pods.

On NVIDIA GPUs the engine runs in **isolated IPC** mode by default (`isolatedIPC` unset): the
engine and the connected vLLM pods exchange KV cache over CUDA IPC, and each vLLM pod gets a
private in-memory `/dev/shm`. Isolated IPC needs no host IPC and no extra SCC.

The coordinator and engine images are built on the serving image, so the first start on a
node can take 15 to 20 minutes while it pulls. Wait for both resources to report `Running`:

```bash theme={null}
oc apply -f coordinator.yaml -f engine.yaml
oc -n llm-serving wait lmcachecoordinator/coordinator lmcacheengine/engine \
  --for=jsonpath='{.status.phase}'=Running --timeout=30m
oc -n llm-serving get lmcachecoordinator,lmcacheengine
# NAME                                                PHASE     READY   REPLICAS   ENDPOINT
# lmcachecoordinator.lmcache.lmcache.ai/coordinator   Running   1       1          http://coordinator.llm-serving.svc:9300
# NAME                                      PHASE     READY   DESIRED
# lmcacheengine.lmcache.lmcache.ai/engine   Running   1       1
```

The engine's `READY` count should equal your number of GPU nodes. If no node carries
`nvidia.com/gpu.present=true`, the engine stays `Pending` and the wait times out.

For storage tiers and sizing, see [filesystem offloading](https://docs.tensormesh.ai/v1.0.1/configuration/filesystem-offloading.md),
[KV cache encryption](https://docs.tensormesh.ai/v1.0.1/configuration/kvcache-encryption.md) and
[cache sizing](https://docs.tensormesh.ai/v1.0.1/configuration/cache-sizing).

## 5. Connect your vLLM deployment

Add one label and one annotation to the vLLM pod template; the labels call this *injection*.
When the pod is created, the operator's webhook adds the LMCache connection settings and the
shared-memory access, so you don't write any LMCache configuration yourself.

Create the vLLM Deployment **after** the engine is `Running`: vLLM connects to the engine when
it starts. If the webhook can't connect a pod, it lets the pod start without LMCache rather
than block it, so check the result below.

```yaml vllm.yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: llm-serving
spec:
  replicas: 1
  progressDeadlineSeconds: 1800             # the first start pulls a large image and the model
  strategy:
    type: Recreate                          # free the GPU before starting the new pod
  selector:
    matchLabels: { app: vllm }
  template:
    metadata:
      labels:
        app: vllm
        lmcache.ai/lmcache-inject: "true"     # opt in to the webhook
      annotations:
        lmcache.ai/lmcache-engine: "engine"   # the LMCacheEngine in this namespace
    spec:
      serviceAccountName: tensormesh
      containers:
        - name: vllm
          image: artifacts.tensormesh.ai/tensormesh-production/images/vllm-openai:v0.5.6-ubi
          args: ["--model", "Qwen/Qwen3-8B", "--port", "8000"]
          ports: [{ containerPort: 8000 }]
          readinessProbe:                     # Ready only once the model is loaded
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
            failureThreshold: 90
          resources:
            limits: { nvidia.com/gpu: 1 }
          volumeMounts:
            - { name: model-cache, mountPath: /tmp/huggingface }   # the image's HF_HOME
      volumes:
        - name: model-cache
          emptyDir: {}                        # or a PersistentVolumeClaim to keep weights across restarts
```

Leave `command` unset on the vLLM container: the webhook skips containers that override it.
The image's entrypoint is `vllm serve`, so `args` starts with the model options.

```bash theme={null}
oc apply -f vllm.yaml
oc -n llm-serving rollout status deployment/vllm --timeout=30m   # downloads the model on first start
```

Check that the pod was connected:

```bash theme={null}
oc get pod -n llm-serving -l app=vllm \
  -o jsonpath='{.items[0].metadata.annotations}' | tr ',' '\n' | grep lmcache
# "lmcache.ai/lmcache-injected":"true"   -> connected
# "lmcache.ai/lmcache-skip-reason":"..."  -> not connected; see Troubleshooting
```

If the skip reason is `engine-not-found` because you created the Deployment before the engine,
run `oc -n llm-serving rollout restart deployment/vllm` once the engine is `Running`.

Then check that the cache works end to end. Send a long prompt, restart vLLM so that its GPU
memory is empty, and send the same prompt again: the engine stores the KV cache the first time
and serves it back to the new pod the second time. The new pod downloads the model again
unless `model-cache` is a PersistentVolumeClaim.

```bash theme={null}
ASK='import json, urllib.request
prompt = "Tell me a long story about a brave knight. " * 60
req = urllib.request.Request("http://localhost:8000/v1/completions",
    data=json.dumps({"model": "Qwen/Qwen3-8B", "prompt": prompt, "max_tokens": 20}).encode(),
    headers={"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req))["usage"])'

oc -n llm-serving exec deploy/vllm -- python3 -c "$ASK"
oc -n llm-serving rollout restart deployment/vllm
oc -n llm-serving rollout status deployment/vllm --timeout=30m
oc -n llm-serving exec deploy/vllm -- python3 -c "$ASK"

# read the log of the engine on the vLLM pod's node
NODE=$(oc -n llm-serving get pod -l app=vllm -o jsonpath='{.items[0].spec.nodeName}')
ENGINE_POD=$(oc -n llm-serving get pod -l app.kubernetes.io/instance=engine \
  --field-selector spec.nodeName="$NODE" -o name)
oc -n llm-serving logs "$ENGINE_POD" | grep -E "Stored|Retrieved"
# ... LMCache INFO: Stored 512 tokens in 0.004 seconds ...      <- first request
# ... LMCache INFO: Retrieved 512 tokens in 0.003 seconds ...   <- after the restart
```

By default each node's engine keeps its own cache, so on a cluster with several GPU nodes the
restarted pod may land on a node whose engine hasn't seen the prompt, and you see only `Stored`
there. A [shared filesystem cache](https://docs.tensormesh.ai/v1.0.1/configuration/filesystem-offloading.md#shared-cache-across-nodes)
or [P2P](https://docs.tensormesh.ai/v1.0.1/configuration/p2p.md) lets engines reuse each other's cache.

The example uses the certified serving image, which already contains LMCache. To keep your
own vLLM image, add the LMCache payload image to `engine.yaml`; the webhook then adds the
matching LMCache client to your pods (see
[payload injection](https://docs.tensormesh.ai/v1.0.1/installation/existing-deployment.md#payload-injection-stop-matching-lmcache-versions-by-hand)
for how it works; the Helm and cert-manager steps there don't apply to OLM):

```yaml theme={null}
spec:
  injection:
    payloadImage:
      repository: artifacts.tensormesh.ai/tensormesh-production/images/lmcache-payload
      tag: v0.5.6-ubi
```

## 6. CacheBlend (optional)

CacheBlend reuses KV cache for text that is not at the start of the prompt, such as
retrieved documents in RAG.

### Allow host IPC

A CacheBlend engine always shares `/dev/shm` with its vLLM pods. On OpenShift it does that
through the host IPC namespace (`hostIPC: true`), and the webhook gives the connected vLLM
pods host IPC too. OpenShift's default `restricted-v2` SCC forbids host IPC, so apply the
least-privilege **`lmcache-engine`** SCC. It allows host IPC and nothing more: non-root, all
capabilities dropped, no privileged containers, no host network, PID or ports, no `hostPath`
volumes, seccomp `RuntimeDefault`. GPUs reach the pods through the NVIDIA RuntimeClass and
CDI, not host paths.

```yaml lmcache-engine-scc.yaml theme={null}
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: lmcache-engine
  annotations:
    kubernetes.io/description: >-
      Least-privilege SCC for Tensormesh Platform cache engines and the vLLM pods connected
      to them: host IPC, and otherwise no more than restricted-v2 allows.
allowHostIPC: true
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowPrivilegeEscalation: false
allowHostDirVolumePlugin: false
defaultAddCapabilities: null
allowedCapabilities: null
requiredDropCapabilities: ["ALL"]
runAsUser: { type: MustRunAsRange }
seLinuxContext: { type: MustRunAs }
fsGroup: { type: MustRunAs }
supplementalGroups: { type: RunAsAny }
seccompProfiles: ["runtime/default"]
readOnlyRootFilesystem: false
volumes: [configMap, downwardAPI, emptyDir, persistentVolumeClaim, projected, secret]
users: []
groups: []
```

A cluster administrator applies it once per cluster, then lets the `tensormesh`
ServiceAccount use it in each namespace that runs CacheBlend or SGLang. Only pods that run as
that ServiceAccount get host IPC, but anyone who can create pods in the namespace can choose
that ServiceAccount, so grant it only in namespaces you trust:

```bash theme={null}
oc apply -f lmcache-engine-scc.yaml

oc create role lmcache-engine-scc --verb=use \
  --resource=scc --resource-name=lmcache-engine -n llm-serving
oc create rolebinding lmcache-engine-scc --role=lmcache-engine-scc \
  --serviceaccount=llm-serving:tensormesh -n llm-serving
```

> [!NOTE]
>
> OpenShift admits pods through SCCs, and the namespace-level Pod Security enforce mode is
> `privileged` by default, so no namespace label is needed. If your cluster policy sets a
> stricter `pod-security.kubernetes.io/enforce` label on namespaces, set it to `privileged` on
> this one; host IPC is outside the `baseline` and `restricted` profiles.

### Create the CacheBlend engine

If you followed steps 4 and 5 and are switching to CacheBlend, delete the step-5 vLLM
Deployment and the step-4 engine first, to free the GPU and the engine's memory:
`oc delete deployment vllm -n llm-serving` and `oc delete lmcacheengine engine -n llm-serving`.

```yaml cacheblend.yaml theme={null}
apiVersion: lmcache.lmcache.ai/v1alpha1
kind: CacheBlendEngine
metadata:
  name: cacheblend
  namespace: llm-serving
spec:
  hostIPC: true                               # share /dev/shm through host IPC; the SCC forbids the hostPath alternative
  gpuVendor: nvidia
  serviceAccountName: tensormesh
  image:
    repository: artifacts.tensormesh.ai/tensormesh-production/images/vllm-openai
    tag: v0.5.6-ubi
  coordinator:
    ref:
      name: coordinator
  l1:
    sizeGB: 40
  blend:
    checkLayer: 1                             # depends on the model: 1 for Qwen3-8B
  injection:
    payloadImage:                             # the CacheBlend plugin; required
      repository: artifacts.tensormesh.ai/tensormesh-production/images/cacheblend-plugin
      tag: v0.5.6-ubi
```

> [!WARNING]
>
> Set `blend.checkLayer` for your model from the
> [Model Configurations](https://docs.tensormesh.ai/v1.0.1/configuration/cacheblend.md#model-configurations) table. A wrong
> value causes no errors, but some responses come back incorrect.

```bash theme={null}
oc apply -f cacheblend.yaml
oc -n llm-serving wait cacheblendengine/cacheblend --for=jsonpath='{.status.phase}'=Running --timeout=30m
```

### Connect vLLM to CacheBlend

Use the CacheBlend label and annotation **instead of** the step-5 ones. This is the step-5
Deployment with the opt-in swapped; the webhook adds the CacheBlend plugin and vLLM flags:

```yaml vllm-cacheblend.yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-cacheblend
  namespace: llm-serving
spec:
  replicas: 1
  progressDeadlineSeconds: 1800             # the first start pulls a large image and the model
  strategy:
    type: Recreate                          # free the GPU before starting the new pod
  selector:
    matchLabels: { app: vllm-cacheblend }
  template:
    metadata:
      labels:
        app: vllm-cacheblend
        lmcache.ai/cacheblend-inject: "true"    # opt in to CacheBlend
      annotations:
        lmcache.ai/cacheblend-engine: "cacheblend"
    spec:
      serviceAccountName: tensormesh            # bound to the lmcache-engine SCC above
      containers:
        - name: vllm
          image: artifacts.tensormesh.ai/tensormesh-production/images/vllm-openai:v0.5.6-ubi
          args: ["--model", "Qwen/Qwen3-8B", "--port", "8000"]
          ports: [{ containerPort: 8000 }]
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
            failureThreshold: 90
          resources:
            limits: { nvidia.com/gpu: 1 }
          volumeMounts:
            - { name: model-cache, mountPath: /tmp/huggingface }
      volumes:
        - name: model-cache
          emptyDir: {}
```

```bash theme={null}
oc apply -f vllm-cacheblend.yaml
oc -n llm-serving rollout status deployment/vllm-cacheblend --timeout=30m
oc get pod -n llm-serving -l app=vllm-cacheblend \
  -o jsonpath='{.items[0].metadata.annotations}' | tr ',' '\n' | grep cacheblend
# "lmcache.ai/cacheblend-injected":"true"   -> connected
```

If no pod appears, run `oc -n llm-serving describe replicaset -l app=vllm-cacheblend`. The
message `unable to validate against any security context constraint` means the SCC binding
above is missing.

Once your RAG traffic repeats documents at different positions in the prompt, the counters of
the engine on the vLLM pod's node show how many prompt tokens CacheBlend reused at non-prefix
positions:

```bash theme={null}
NODE=$(oc -n llm-serving get pod -l app=vllm-cacheblend -o jsonpath='{.items[0].spec.nodeName}')
CB_POD=$(oc -n llm-serving get pod -l app.kubernetes.io/instance=cacheblend \
  --field-selector spec.nodeName="$NODE" -o name)
oc -n llm-serving exec "$CB_POD" -- curl -s localhost:8080/metrics \
  | grep -E '^lmcache_blend_lookup_(non_prefix_hit|requested)_tokens'
# lmcache_blend_lookup_requested_tokens_total 71680.0
# lmcache_blend_lookup_non_prefix_hit_tokens_total 33792.0   <- greater than 0: blending is active
```

The [CacheBlend guide](https://docs.tensormesh.ai/v1.0.1/configuration/cacheblend.md) covers the CacheBlend settings; its
Helm steps don't apply here.

## 7. SGLang (optional)

The webhook also connects SGLang servers. SGLang's LMCache connector supports only CUDA IPC
over a shared `/dev/shm`, not isolated IPC, so on OpenShift it needs an engine with
`isolatedIPC: false` and `hostIPC: true`, and the [host IPC grant](#allow-host-ipc) from
step 6. The webhook skips SGLang pods whose engine runs isolated IPC. Like CacheBlend, this
engine is usually created instead of the step-4 one: if you are switching from step 5 or 6,
delete that vLLM Deployment and engine first.

```yaml sglang-engine.yaml theme={null}
apiVersion: lmcache.lmcache.ai/v1alpha1
kind: LMCacheEngine
metadata:
  name: sglang-engine
  namespace: llm-serving
spec:
  isolatedIPC: false
  hostIPC: true                               # share /dev/shm through host IPC; the SCC forbids the hostPath alternative
  gpuVendor: nvidia
  serviceAccountName: tensormesh
  image:
    repository: artifacts.tensormesh.ai/tensormesh-production/images/vllm-openai
    tag: v0.5.6-ubi
  coordinator:
    ref:
      name: coordinator
  l1:
    sizeGB: 40
```

The SGLang image is yours to supply; it is not one of the certified images. It must contain
LMCache at the engine's version, such as the community image `lmcache/sglang:v0.5.6`. Opt
SGLang pods in with the `lmcache.ai/sglang-inject` label and the `lmcache.ai/lmcache-engine`
annotation; the webhook adds `--enable-lmcache` and the engine address. OpenShift runs the
pod as a non-root user, so point `HOME` at a writable directory:

```yaml sglang.yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang
  namespace: llm-serving
spec:
  replicas: 1
  progressDeadlineSeconds: 1800             # the first start pulls a large image and the model
  strategy:
    type: Recreate                          # free the GPU before starting the new pod
  selector:
    matchLabels: { app: sglang }
  template:
    metadata:
      labels:
        app: sglang
        lmcache.ai/sglang-inject: "true"
      annotations:
        lmcache.ai/lmcache-engine: "sglang-engine"
    spec:
      serviceAccountName: tensormesh            # bound to the lmcache-engine SCC
      containers:
        - name: sglang
          image: lmcache/sglang:v0.5.6
          command: ["python3", "-m", "sglang.launch_server"]
          args: ["--model-path", "Qwen/Qwen3-8B", "--host", "0.0.0.0", "--port", "30000"]
          env:
            - { name: HOME, value: /tmp }
            - { name: HF_HOME, value: /tmp/huggingface }
          ports: [{ containerPort: 30000 }]
          readinessProbe:
            httpGet: { path: /health, port: 30000 }
            periodSeconds: 10
            failureThreshold: 90
          resources:
            limits: { nvidia.com/gpu: 1 }
          volumeMounts:
            - { name: model-cache, mountPath: /tmp/huggingface }
      volumes:
        - name: model-cache
          emptyDir: {}
```

```bash theme={null}
oc apply -f sglang-engine.yaml
oc -n llm-serving wait lmcacheengine/sglang-engine --for=jsonpath='{.status.phase}'=Running --timeout=30m
oc apply -f sglang.yaml
oc -n llm-serving rollout status deployment/sglang --timeout=30m
oc get pod -n llm-serving -l app=sglang \
  -o jsonpath='{.items[0].metadata.annotations}' | tr ',' '\n' | grep lmcache
# "lmcache.ai/sglang-injected":"true"   -> connected
```

After the first long requests, the log of the `sglang-engine` pod on the SGLang pod's node
shows `Stored` lines, as in step 5.

## 8. Restrict network access (recommended)

The cache server port accepts connections from any pod in the cluster unless you restrict
it. Set `spec.networkPolicy.enabled: true` on each `LMCacheEngine` and `CacheBlendEngine`,
in the YAML or with a patch:

```bash theme={null}
oc -n llm-serving patch lmcacheengine engine --type=merge \
  -p '{"spec":{"networkPolicy":{"enabled":true}}}'
```

The operator then creates a NetworkPolicy named `<engine>-ingress` for the engine's pods:

* The cache server port (5555 by default) accepts only pods that carry one of the webhook's
  opt-in labels set to `"true"`, from any namespace: `lmcache.ai/lmcache-inject` or
  `lmcache.ai/sglang-inject` for an `LMCacheEngine`, `lmcache.ai/cacheblend-inject` for a
  `CacheBlendEngine`. This limits who can reach the cache, but it is not tenant isolation:
  any pod with that label can connect.
* The HTTP port (8080), which serves status, metrics and the cache operations the
  [CLI](https://docs.tensormesh.ai/v1.0.1/reference/cli.md) uses, the metrics port (9090) and, when enabled, the
  [P2P](https://docs.tensormesh.ai/v1.0.1/configuration/p2p.md) port stay open to all pods. Add your own NetworkPolicy to
  restrict them.
* All other incoming connections to the engine pods are blocked.

Setting the field back to `false` deletes the policy.

Check that it took effect:

```bash theme={null}
oc get lmcacheengine engine -n llm-serving \
  -o jsonpath='{.status.conditions[?(@.type=="NetworkPolicyEffective")].status}'   # True
```

The policy needs a network plugin that enforces NetworkPolicy; OVN-Kubernetes, the OpenShift
default, does. It has no effect on pods that use `hostNetwork: true`.

## 9. Observability (optional)

The OLM install does not create the OpenTelemetry Collector or the ServiceMonitors that
`helm install --set observability.enabled=true` does. Without a collector, each engine serves
Prometheus metrics at `/metrics` on its HTTP port (8080). To collect engine metrics and traces
centrally, install the **Red Hat build of OpenTelemetry** operator from the software catalog
and create a collector in the workload namespace:

```yaml collector.yaml theme={null}
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: tensormesh
  namespace: llm-serving
spec:
  mode: deployment
  config:
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: "0.0.0.0:4317" }
    exporters:
      prometheus: { endpoint: "0.0.0.0:8889" }
      debug: {}
    service:
      pipelines:
        metrics: { receivers: [otlp], exporters: [prometheus] }
        traces: { receivers: [otlp], exporters: [debug] }
```

Then point the engine at it: `--otlp-endpoint` sends the engine's metrics and traces to the
collector, and `--enable-tracing` turns on tracing. Scrape the collector (port 8889) instead
of the engine:

```bash theme={null}
oc apply -f collector.yaml
oc -n llm-serving patch lmcacheengine engine --type=merge \
  -p '{"spec":{"extraArgs":["--enable-tracing","--otlp-endpoint=http://tensormesh-collector.llm-serving.svc:4317"]}}'
```

To send traces to a tracing backend such as Tempo, replace the `debug` exporter with that
backend's OTLP exporter. To scrape the collector with the cluster's monitoring stack,
[enable monitoring for user-defined projects](https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.20/html-single/configuring_user_workload_monitoring/index#enabling-monitoring-for-user-defined-projects-uwm_preparing-to-configure-the-monitoring-stack-uwm)
and create a ServiceMonitor for the collector's `tensormesh-collector` Service. The metrics
themselves are described in [Observability](https://docs.tensormesh.ai/v1.0.1/observability.md).

## Upgrade

The operator, its CRDs and its webhooks upgrade through OLM, according to the **Update
approval** you chose in step 2: with **Automatic**, OLM upgrades the operator as soon as a new
version reaches the `stable` channel; with **Manual**, you approve the install plan in
**Ecosystem → Installed Operators → Tensormesh Platform**.

The engine, coordinator and serving images don't upgrade with the operator: they are the tags
set in your custom resources and Deployments. After each operator upgrade:

1. Check the [OpenShift certified bundle table](https://docs.tensormesh.ai/v1.0.1/installation/compatibility.md#openshift-certified-bundle)
   for the image tags of the new version.
2. Update `spec.image.tag`, and `spec.injection.payloadImage.tag` where set, on every
   `LMCacheCoordinator`, `LMCacheEngine` and `CacheBlendEngine`, the image of your vLLM
   Deployments, and your SGLang image (LMCache at the new engine version). Mixing versions
   across the operator and the engines is not supported.

OLM does not roll an operator back to an earlier version; if an upgrade causes problems,
contact [Tensormesh](https://www.tensormesh.ai/contact).

## Uninstall

> [!WARNING]
>
> Delete the custom resources **before** the operator. OLM does not delete them when you
> uninstall the operator, so their engine DaemonSets and coordinator would keep running,
> unmanaged, until you delete them.

First delete your vLLM and SGLang Deployments, then the custom resources in every namespace,
and wait until they are gone:

```bash theme={null}
oc delete cacheblendengine,lmcacheengine,lmcachecoordinator --all -A
oc get cacheblendengine,lmcacheengine,lmcachecoordinator -A   # expect: No resources found
```

Then uninstall the operator. In the console, open **Ecosystem → Installed Operators →
Tensormesh Platform** and choose **Actions → Uninstall Operator**. Or use the CLI, following
Red Hat's [procedure for deleting an Operator](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html-single/operators/index#olm-deleting-operator-from-a-cluster-using-cli_olm-deleting-operators-from-a-cluster):

```bash theme={null}
CSV=$(oc get subscription.operators.coreos.com tensormesh-platform -n openshift-operators \
  -o jsonpath='{.status.currentCSV}')
oc delete subscription.operators.coreos.com tensormesh-platform -n openshift-operators
oc delete csv "$CSV" -n openshift-operators
```

The CRDs, the `lmcache-engine` SCC, the namespace grants and your registry access token stay
in place, so a reinstall picks up where you left off. Remove them only if you are done with
Tensormesh Platform:

```bash theme={null}
oc delete crd lmcacheengines.lmcache.lmcache.ai \
  lmcachecoordinators.lmcache.lmcache.ai cacheblendengines.lmcache.lmcache.ai
oc delete rolebinding,role lmcache-engine-scc -n llm-serving
oc delete serviceaccount tensormesh -n llm-serving
oc delete scc lmcache-engine
```

To remove the registry access token, edit the global pull secret as in step 1 and delete the
`artifacts.tensormesh.ai` entry.

## Troubleshooting

| Symptom | Check |
| - | - |
| Operator or engine pod in `ImagePullBackOff` | The registry access token (step 1) and that the pull-secret rollout finished (`oc get mcp`). |
| An engine stays `Pending`, or its `READY` count is lower than your GPU nodes | `oc -n llm-serving describe daemonset <engine>`; a pod stuck `Pending` usually lacks CPU or memory for the engine's request (see [Node resources](#prerequisites)). With no GPU-labeled node, the engine stays `Pending`. |
| No CacheBlend or SGLang pod appears | `oc -n llm-serving describe replicaset -l app=<name>`; `unable to validate against any security context constraint` means the [host IPC grant](#allow-host-ipc) is missing. |
| A pod has `lmcache.ai/lmcache-skip-reason` (vLLM), `lmcache.ai/cacheblend-skip-reason` or `lmcache.ai/sglang-skip-reason` | `engine-not-found`: no engine with that name exists in the pod's namespace (check the annotation, the namespace, and that the engine was created first). `command-override`: the vLLM container sets `command`. `target-container-not-found`: the container named by the pod's annotation doesn't exist. `isolated-ipc-unsupported` (SGLang): the engine runs isolated IPC. `payload-image-unset` (CacheBlend): set `injection.payloadImage`. Fix it, then restart the Deployment. `kv-transfer-config-present` is informational: the pod is connected and your own `--kv-transfer-config` was kept. |

For more, see [Troubleshooting](https://docs.tensormesh.ai/v1.0.1/installation/troubleshooting.md); its Helm commands
don't apply to an OLM install.

## Next steps

> **[CacheBlend](https://docs.tensormesh.ai/v1.0.1/configuration/cacheblend.md)**
>
> Model settings and how to check that blending is active.

> **[CLI](https://docs.tensormesh.ai/v1.0.1/reference/cli.md)**
>
> `tmo-cli` checks engine status, clears the cache and runs diagnostics. It installs
> separately, with the same registry access token.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.