> ## Documentation Index > Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt > Use this file to discover all available pages before exploring further. # E2E Quickstart Example > Install the operator, point vLLM at the engine, run an inference, and verify the KV cache is reused. This walkthrough takes a fresh Kubernetes (or OpenShift) cluster with at least one GPU node, installs the Tensormesh Operator from the Helm chart, deploys a vLLM Pod connected to the LMCache engine in MP mode, runs a smoke test, and verifies the KV cache is being reused. Plan for about 15 minutes — most of that is the vLLM image pull and Qwen3-0.6B download on first boot. For deep references, see [Install with Helm](/operator/installation/helm) and [Install on OpenShift](/operator/installation/openshift). This page distills both into the shortest path to a working inference. ## Prerequisites * Kubernetes 1.28+ with at least one GPU node labeled `nvidia.com/gpu.present=true`. * Helm 3.8+ and `kubectl` (or `oc` on OpenShift) configured against your cluster. * **An access token from the Tensormesh team.** Authenticate with `helm registry login ghcr.io` before installing — see [Helm install](/operator/installation/helm#install). Run the three preflight checks from [Getting Started](/operator/installation/getting-started#verify-your-cluster-is-ready) to confirm cluster, GPU label, and RBAC before continuing. ## Step 1 — install the operator Put your configuration in a values file rather than `--set` flags. Create `tensormesh-values.yaml`: ```yaml tensormesh-values.yaml theme={null} # The operator image is supplied by the chart version (--version 0.4.5) — nothing to pin here. engine: enabled: true spec: l1: sizeGB: 60 image: repository: lmcache/vllm-openai tag: v0.4.5 # pin explicitly pullPolicy: IfNotPresent # OpenShift only — grants the privileged SCC the engine's hostIPC needs. # Delete this block on plain Kubernetes. openshift: enabled: true ``` Install with the values file: ```bash theme={null} helm upgrade --install tensormesh-operator \ oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \ --version 0.4.5 \ --namespace tensormesh-operator --create-namespace \ -f tensormesh-values.yaml \ --wait --timeout 5m ``` **Pin to immutable tags, never `latest`.** The chart version (`--version 0.4.5`) selects the matched operator image; pin the vLLM image to `v0.4.5`. With `imagePullPolicy: IfNotPresent`, a node that already cached an image under `latest` keeps the stale copy — so `latest` can silently run an old build. An explicit, immutable tag avoids this entirely. Confirm the operator and engine pods are both Running: ```bash theme={null} kubectl get pods -n tensormesh-operator ``` You should see exactly two pods, both `Running 1/1`: | Pod name | What it is | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `tensormesh-operator--` | The **operator** (controller-manager Deployment). The two-segment suffix (`-`) marks it as Deployment-owned. One per cluster. | | `tensormesh-operator-default-engine-` | The **engine** (LMCache cache server, DaemonSet). One per GPU node. The operator creates it from the `LMCacheEngine` CR, so it appears a few seconds after the operator. | (If you also see a `...-test-...` pod, that's a leftover `helm test` pod — harmless; delete it with `kubectl delete pod -n tensormesh-operator`.) If install fails with `CRD ... exists and cannot be imported`, jump to [Troubleshooting → CRD ownership conflict](/operator/installation/troubleshooting#install-fails-with-crd-exists-and-cannot-be-imported). ## Step 2 — deploy vLLM connected to the engine The chart creates a ConfigMap named `tensormesh-operator-default-engine-connection` containing the `kv-transfer-config.json` that tells vLLM how to reach the engine over the MP connector. The Deployment below mounts that ConfigMap and passes it as `--kv-transfer-config`. If you already run your own vLLM Deployment, do not replace it with this demo manifest. Use [Modify an Existing Deployment](/operator/installation/existing-deployment) for the minimum patch set instead. Save as `vllm-demo.yaml`: ```yaml vllm-demo.yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: vllm-demo namespace: tensormesh-operator labels: { app: vllm-demo } spec: replicas: 1 selector: { matchLabels: { app: vllm-demo } } template: metadata: { labels: { app: vllm-demo } } spec: hostIPC: true # required for MP mode serviceAccountName: tensormesh-operator-engine-privileged # reuse the chart's privileged SA nodeSelector: nvidia.com/gpu.present: "true" # colocate with the engine containers: - name: vllm image: lmcache/vllm-openai:v0.4.5 # pin explicitly, never :latest imagePullPolicy: IfNotPresent env: - { name: HF_HUB_DISABLE_TELEMETRY, value: "1" } command: ["/bin/sh", "-c"] args: - | exec python3 -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen3-0.6B \ --port 8000 \ --gpu-memory-utilization 0.6 \ --no-enable-prefix-caching \ --max-model-len 32768 \ --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config.json)" ports: [{ name: http, containerPort: 8000 }] volumeMounts: - { name: kv-transfer-config, mountPath: /etc/lmcache, readOnly: true } - { name: hf-cache, mountPath: /root/.cache/huggingface } resources: limits: { nvidia.com/gpu: "1", memory: 32Gi } # `memory` request is scheduler-accounting only; the limit is the real cap. # Lower this if the node is already heavily booked (e.g. the engine reserves # a large L1 cache in host memory). requests: { nvidia.com/gpu: "1", cpu: "2", memory: 6Gi } readinessProbe: httpGet: { path: /health, port: http } initialDelaySeconds: 60 periodSeconds: 15 failureThreshold: 40 volumes: - name: kv-transfer-config configMap: name: tensormesh-operator-default-engine-connection - name: hf-cache emptyDir: { sizeLimit: 20Gi } --- apiVersion: v1 kind: Service metadata: name: inference namespace: tensormesh-operator spec: selector: { app: vllm-demo } ports: [{ name: http, port: 8000, targetPort: http }] ``` Apply and wait for the Pod to become Ready (5–10 min on first boot for image pull + model download): ```bash theme={null} kubectl apply -f vllm-demo.yaml kubectl wait --for=condition=Ready pod -l app=vllm-demo \ -n tensormesh-operator --timeout=15m ``` ## Step 3 — fire two identical requests, see the cache work The simplest possible "is LMCache working?" test is to send two identical long prompts. The first populates the cache; the second should reuse the stored KV blocks. Port-forward the service in one terminal: ```bash theme={null} kubectl port-forward -n tensormesh-operator svc/inference 8000:8000 ``` If port `8000` is already in use locally, forward to a different local port — e.g. `... 8001:8000` — and use that port in the requests below. In a second terminal, build the request once into a variable, then fire it twice. Keeping the JSON in a `PAYLOAD` variable (rather than inlining `$(jq …)` inside a multi-line `curl`) makes it safe to copy-paste and guarantees both calls send the **identical** prompt: ```bash theme={null} PROMPT=$(python3 -c "print('Tell me a long story about a brave knight. ' * 30)") PAYLOAD=$(jq -n --arg p "$PROMPT" \ '{model:"Qwen/Qwen3-0.6B", prompt:$p, max_tokens:20, temperature:0}') # Call 1 — cold, should STORE KV into LMCache time curl -sS http://localhost:8000/v1/completions \ -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text' # Call 2 — identical, should RETRIEVE prefix KV from LMCache time curl -sS http://localhost:8000/v1/completions \ -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text' ``` Set `PROMPT` and `PAYLOAD` in the **same shell** you run the `curl`s in. If `$PAYLOAD` is empty the request carries no prompt, no KV is stored, and you'll see no cache markers — which looks like a failure but is just an unset variable. Then check the engine logs for store and retrieve markers: ```bash theme={null} kubectl logs -n tensormesh-operator -l app.kubernetes.io/component=cache-engine --tail=200 \ | grep -E "Stored [0-9]+ tokens|Prefetch request completed.*prefix hits" ``` A working install logs `Stored N tokens` after call 1 and `Prefetch request completed ... prefix hits=N` during call 2. Sample output from a Qwen3-8B install on a single A100 40 GB, prompt repeated 30 times for \~514 tokens: ```text theme={null} ========================================== CALL 1 (cold — expect STORE) ========================================== elapsed: 1.466s usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534} ========================================== CALL 2 (warm — expect RETRIEVE) ========================================== elapsed: 0.548s usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534} ========================================== ENGINE LOG MARKERS ========================================== LMCache INFO: Stored 512 tokens in 0.029 seconds LMCache INFO: Prefetch request completed (L1+L2): 2/2 prefix hits (2 L1, 0 L2) in 0.9 ms ``` Call 2 was \~2.7× faster than call 1 (1.466 s → 0.548 s). The engine reports `2/2 prefix hits` from L1, meaning the second request reused the KV blocks stored during the first. vLLM's own metrics line confirms the same number from the inference side: ```text theme={null} External prefix cache hit rate: 49.8% ``` (\~half the prompt was the shared prefix, so 49.8% is the expected hit ratio.) Prompts must exceed the engine's `chunk_size` (256 tokens by default) for any KV blocks to be stored. Short prompts produce no markers — that's expected, not a failure. ## Step 4 — run the benchmark For a more realistic measurement than two curls, use `vllm bench serve` to compare a cold pass against a warm pass. Run **two passes with the same `--seed`** so the prompts repeat and the second pass hits the cache. Rather than installing the vLLM CLI locally, run it **inside the cluster as a `Job`** — the vLLM image already ships the CLI, talks to the `inference` Service directly (no port-forward), and a `Job` runs it once to completion and stops. The benchmark is a pure HTTP client, so it needs **no GPU**. Save as `benchmark-job.yaml`: ```yaml benchmark-job.yaml theme={null} apiVersion: batch/v1 kind: Job metadata: name: lmcache-benchmark namespace: tensormesh-operator spec: backoffLimit: 0 ttlSecondsAfterFinished: 3600 template: metadata: labels: { app: lmcache-benchmark } spec: restartPolicy: Never nodeSelector: nvidia.com/gpu.present: "true" # reuse the cached vLLM image; the client needs no GPU containers: - name: bench image: lmcache/vllm-openai:v0.4.5 imagePullPolicy: IfNotPresent env: - { name: HF_HOME, value: /tmp/hf } # writable under OpenShift's random UID - { name: HF_HUB_DISABLE_TELEMETRY, value: "1" } command: ["/bin/sh", "-c"] args: - | set -e run() { vllm bench serve --backend openai --base-url http://inference:8000 \ --model Qwen/Qwen3-0.6B --dataset-name random \ --num-prompts 20 --random-input-len 20480 \ --random-output-len 1 --seed 32 } echo "### PASS 1 (cold) ###"; run sleep 5 echo "### PASS 2 (warm) ###"; run volumeMounts: - { name: hf, mountPath: /tmp/hf } resources: requests: { cpu: "1", memory: 2Gi } limits: { cpu: "2", memory: 4Gi } volumes: - { name: hf, emptyDir: {} } ``` Run it and stream the results: ```bash theme={null} kubectl apply -f benchmark-job.yaml kubectl wait --for=condition=complete job/lmcache-benchmark \ -n tensormesh-operator --timeout=10m kubectl logs -f job/lmcache-benchmark -n tensormesh-operator ``` The logs contain two `Serving Benchmark Result` blocks. A working cache shows **pass 2 with much lower TTFT** (mean/p50/p99). The speedup scales with `--random-input-len`: with a fixed `--seed`, both passes send the same 20 prompts, so each unique body is prefilled on the cold pass and served from cache on the warm pass — bigger `--random-input-len` → bigger delta. Example result from a working run with `--num-prompts 20`, `--random-input-len 20480`, and `--seed 32`: ```text theme={null} ### PASS 1 (cold) ### Successful requests: 20 Benchmark duration (s): 11.00 Request throughput (req/s): 1.82 Mean TTFT (ms): 6399.80 Median TTFT (ms): 6456.31 P99 TTFT (ms): 10896.55 ### PASS 2 (warm) ### Successful requests: 20 Benchmark duration (s): 5.64 Request throughput (req/s): 3.55 Mean TTFT (ms): 3697.56 Median TTFT (ms): 3753.21 P99 TTFT (ms): 5584.58 ``` In the healthy case above, pass 2 cuts TTFT substantially relative to pass 1 and nearly doubles request throughput. Keep `num_prompts × random-input-len` of KV under `engine.spec.l1.sizeGB` (else pass 1's entries evict before pass 2 reads them — warm misses), and `random-input-len ≤ --max-model-len` (32768 in the Deployment above). Clean up the Job (it also self-deletes after 1h via `ttlSecondsAfterFinished`): ```bash theme={null} kubectl delete job lmcache-benchmark -n tensormesh-operator ``` `--seed` **must match** across both passes — that's what makes the prompts identical and therefore cacheable. Different seeds = different prompts = no cache hits. Why a `Job` and not a `Deployment`? A `Job` runs its pod to **completion** and stops — the right shape for a one-shot benchmark. A `Deployment` would restart the benchmark forever. ## Step 5 — observability For the full OpenTelemetry stack — an OTel Collector that receives metrics and traces from the engine, a Prometheus `ServiceMonitor`, and optional Tempo tracing — see the dedicated **[Observability](/operator/observability)** page. It covers the prerequisites, the `observability.*` values, and how to verify telemetry is actually flowing. A quick liveness check without the full stack — the operator writes status back to the `LMCacheEngine` CR: ```bash theme={null} kubectl get lmcacheengine -n tensormesh-operator -o wide kubectl describe lmcacheengine -n tensormesh-operator ``` `PHASE` should be `Ready` and `READY` should equal `DESIRED` (one per GPU node). ### Query the engine's metrics If you enabled observability (`observability.enabled=true` — see the [Observability](/operator/observability) page), the OTel Collector re-exposes the engine's metrics on port `8889` for Prometheus. After some traffic (Steps 3–4), port-forward and grep for the `lmcache_` series: ```bash theme={null} kubectl port-forward -n tensormesh-operator \ svc/tensormesh-operator-otel-collector-collector 8889:8889 # in another terminal: curl -s http://localhost:8889/metrics | grep lmcache | head -20 ``` Expected output — a batch of `lmcache_mp_*` gauges and counters: ```text theme={null} # HELP lmcache_mp_l1_memory_usage_bytes Bytes currently held in L1 cache # TYPE lmcache_mp_l1_memory_usage_bytes gauge lmcache_mp_l1_memory_usage_bytes{instance="",otel_scope_name="lmcache.l1_manager"} 0 # HELP lmcache_mp_l1_eviction_loop_ticks_total L1 eviction-loop iterations (every cycle) # TYPE lmcache_mp_l1_eviction_loop_ticks_total counter lmcache_mp_l1_eviction_loop_ticks_total{instance="",otel_scope_name="lmcache.l1"} 227 # HELP lmcache_mp_active_prefetch_jobs Number of active prefetch jobs # TYPE lmcache_mp_active_prefetch_jobs gauge lmcache_mp_active_prefetch_jobs{instance="",otel_scope_name="lmcache.mp_engine"} 0 ``` A batch of `lmcache_mp_*` series confirms the engine is exporting metrics through the Collector. Empty output means observability isn't enabled yet, or no traffic has hit the engine. Port `8889` is the Collector's **Prometheus exporter** (LMCache *application* metrics). Don't confuse it with `8888` on the `…-monitoring` service, which serves the Collector's *own* `otelcol_*` internals. ## Cleanup The ordering principle: **remove the `LMCacheEngine` CR before the CRD or the namespace, while the operator is still alive to run its finalizer.** `helm uninstall` does this for you — its pre-delete hook deletes the CR first, then removes the operator. ```bash theme={null} # 1. Remove the vLLM workload you deployed. kubectl delete -f vllm-demo.yaml # 2. Uninstall the release. The pre-delete hook deletes the LMCacheEngine CR (the operator # drains its finalizer) before the operator itself is removed. helm uninstall tensormesh-operator -n tensormesh-operator # 3. Delete the now-empty namespace (helm uninstall already removed the workload + RBAC). kubectl delete namespace tensormesh-operator # 4. The CRD is kept by design (helm.sh/resource-policy: keep). Remove it for a clean slate. kubectl delete crd lmcacheengines.lmcache.lmcache.ai ``` Don't delete the namespace or CRD *before* the CR is gone. The `LMCacheEngine` CR has a finalizer the operator must run; remove the operator (or the CRD) first and the CR can't drain — the namespace then hangs in `Terminating`. If you've already hit that, or `helm list -A` shows no release but resources remain, see [Troubleshooting → Cleaning up an inconsistent or orphaned install](/operator/installation/troubleshooting#cleaning-up-an-inconsistent-or-orphaned-install). ## Next steps Full chart reference, install modes, every tunable value. Every `values.yaml` key, with example overlays. Metrics, dashboards, and performance tuning. `Pending` pods, image pull, hung uninstall, ownership conflicts.