E2E Quickstart

Step 1 — install Tensormesh Operator

If you already run your own vLLM Deployment, do not replace it with this demo manifest. Use Modify an Existing Deployment for the minimum patch set instead.

Step 2 — deploy vLLM connected to the engine

The chart creates a ConfigMap named <engine>-connection holding the kv-transfer-config.json that tells vLLM how to reach the engine over the MP connector.

Read the connector config from the engine’s ConfigMap

kubectl get cm tensormesh-operator-default-engine-connection \
  -n tensormesh-operator \
  -o jsonpath='{.data.kv-transfer-config\.json}' | jq -c .

You should get one line like this:

{"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-operator-default-engine.tensormesh-operator.svc.cluster.local","lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"}

Create the Deployment

Paste that line as the --kv-transfer-config value in vllm-demo.yaml:

vllm-demo.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-demo
  namespace: tensormesh-operator
  labels: { app: vllm-demo }
spec:
  selector: { matchLabels: { app: vllm-demo } }
  template:
    metadata:
      labels: { app: vllm-demo }
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      hostIPC: true                        
      containers:
        - name: vllm
          image: lmcache/vllm-openai:v0.5.3   # use the same version as the lmcache engine
          env:
            - { name: PYTHONHASHSEED, value: "0" }
          args:
            - Qwen/Qwen3-0.6B
            - --gpu-memory-utilization
            - "0.6"
            - --no-enable-prefix-caching  
            - --max-model-len
            - "32768"
            - --kv-transfer-config
            # the single line from the previous step
            - '{"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-operator-default-engine.tensormesh-operator.svc.cluster.local","lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"}'
          ports: [{ name: http, containerPort: 8000 }]
          resources:
            limits:   { nvidia.com/gpu: "1", memory: 32Gi }  
            requests: { cpu: "2", memory: 6Gi }
          readinessProbe:                  
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 40
---
apiVersion: v1
kind: Service
metadata:
  name: inference
  namespace: tensormesh-operator
spec:
  selector: { app: vllm-demo }
  ports: [{ name: http, port: 8000, targetPort: http }]

Apply and wait for the Pod to become Ready (5–10 min on first boot for image pull + model download):

kubectl apply -f vllm-demo.yaml
kubectl wait --for=condition=Ready pod -l app=vllm-demo \
  -n tensormesh-operator --timeout=15m

Step 3 — fire two identical requests, see the cache work

The simplest possible “is LMCache working?” test is to send two identical long prompts. The first populates the cache; the second should reuse the stored KV blocks. Port-forward the service in one terminal:

kubectl port-forward -n tensormesh-operator svc/inference 8000:8000 --address 127.0.0.1

If port 8000 is already in use locally, forward to a different local port — e.g. ... 8001:8000 — and use that port in the requests below.

In a second terminal, build the request once into a variable, then fire it twice. Keeping the JSON in a PAYLOAD variable (rather than inlining $(jq …) inside a multi-line curl) makes it safe to copy-paste and guarantees both calls send the identical prompt:

PROMPT=$(python3 -c "print('Tell me a long story about a brave knight. ' * 30)")
PAYLOAD=$(jq -n --arg p "$PROMPT" \
  '{model:"Qwen/Qwen3-0.6B", prompt:$p, max_tokens:20, temperature:0}')

# Call 1 — cold, should STORE KV into LMCache
time curl -sS http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text'

# Call 2 — identical, should RETRIEVE prefix KV from LMCache
time curl -sS http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text'

Then check the engine logs for store and retrieve markers:

# select by the engine CR's name; the component label (cache-engine) also matches
# CacheBlend engine pods when those are enabled
kubectl logs -n tensormesh-operator --tail=200 \
  -l app.kubernetes.io/instance=tensormesh-operator-default-engine \
  | grep -E "Stored [0-9]+ tokens|Prefetch request completed.*prefix hits"

A working install logs Stored N tokens after call 1 and Prefetch request completed ... prefix hits=N during call 2.

What a successful run looks like

Sample output from a Qwen3-0.6B install on a single A100 40 GB, prompt repeated 30 times for ~514 tokens:

==========================================
 CALL 1 (cold — expect STORE)
==========================================
elapsed: 1.466s
usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534}

==========================================
 CALL 2 (warm — expect RETRIEVE)
==========================================
elapsed: 0.548s
usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534}

==========================================
 ENGINE LOG MARKERS
==========================================
LMCache INFO: Stored 512 tokens in 0.029 seconds
LMCache INFO: Prefetch request completed (L1+L2): 2/2 prefix hits (2 L1, 0 L2) in 0.9 ms

Call 2 was ~2.7× faster than call 1 (1.466 s → 0.548 s). The engine reports 2/2 prefix hits from L1, meaning the second request reused the KV blocks stored during the first. vLLM’s own metrics line confirms the same number from the inference side:

External prefix cache hit rate: 49.8%

(The counter is cumulative across both calls: call 1 hit 0 of 514 tokens, call 2 hit 512 of 514, so 512/1028 = 49.8%. For a two-call test ~50% is the ceiling, not a partial hit.)

Prompts must exceed the engine’s chunk_size (256 tokens by default) for any KV blocks to be stored. Short prompts produce no markers — that’s expected, not a failure.

Step 4 — run the benchmark

For a more realistic measurement than two curls, use vllm bench serve to compare a cold pass against a warm pass. Run two passes with the same --seed so the prompts repeat and the second pass hits the cache. Rather than installing the vLLM CLI locally, run it inside the cluster as a Job — the vLLM image already ships the CLI, talks to the inference Service directly (no port-forward), and a Job runs it once to completion and stops. The benchmark is a pure HTTP client, so it needs no GPU. Save as benchmark-job.yaml:

benchmark-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: lmcache-benchmark
  namespace: tensormesh-operator
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels: { app: lmcache-benchmark }
    spec:
      restartPolicy: Never
      containers:
        - name: bench
          image: lmcache/vllm-openai:v0.5.3   # same tag as the Deployment, so the node's cached image is reused
          imagePullPolicy: IfNotPresent
          env:
            - { name: HF_HOME, value: /tmp/hf }   # writable under OpenShift's random UID
            - { name: HF_HUB_DISABLE_TELEMETRY, value: "1" }
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              run() {
                vllm bench serve --backend openai --base-url http://inference:8000 \
                  --model Qwen/Qwen3-0.6B --dataset-name random \
                  --num-prompts 20 --random-input-len 20480 \
                  --random-output-len 1 --seed 32
              }
              echo "### PASS 1 (cold) ###"; run
              sleep 5
              echo "### PASS 2 (warm) ###"; run
          volumeMounts:
            - { name: hf, mountPath: /tmp/hf }
          resources:
            requests: { cpu: "1", memory: 2Gi }
            limits:   { cpu: "2", memory: 4Gi }
      volumes:
        - { name: hf, emptyDir: {} }

Run it and stream the results:

kubectl apply -f benchmark-job.yaml
kubectl wait --for=condition=complete job/lmcache-benchmark \
  -n tensormesh-operator --timeout=10m
kubectl logs -f job/lmcache-benchmark -n tensormesh-operator

The logs contain two Serving Benchmark Result blocks. A working cache shows pass 2 with much lower TTFT (mean/p50/p99). The speedup scales with --random-input-len: with a fixed --seed, both passes send the same 20 prompts, so each unique body is prefilled on the cold pass and served from cache on the warm pass — bigger --random-input-len → bigger delta. Example result from a working run with --num-prompts 20, --random-input-len 20480, and --seed 32:

### PASS 1 (cold) ###
Successful requests:                     20
Benchmark duration (s):                  11.00
Request throughput (req/s):              1.82
Mean TTFT (ms):                          6399.80
Median TTFT (ms):                        6456.31
P99 TTFT (ms):                           10896.55

### PASS 2 (warm) ###
Successful requests:                     20
Benchmark duration (s):                  5.64
Request throughput (req/s):              3.55
Mean TTFT (ms):                          3697.56
Median TTFT (ms):                        3753.21
P99 TTFT (ms):                           5584.58

In the healthy case above, pass 2 cuts TTFT substantially relative to pass 1 and nearly doubles request throughput.

Keep num_prompts × random-input-len of KV under engine.spec.l1.sizeGB (else pass 1’s entries evict before pass 2 reads them — warm misses), and random-input-len + random-output-len ≤ --max-model-len (32768 in the Deployment above).

Clean up the Job (it also self-deletes after 1h via ttlSecondsAfterFinished):

kubectl delete job lmcache-benchmark -n tensormesh-operator

Next steps

Install with Helm

Full chart reference, install modes, every tunable value.

Configuration

Every values.yaml key, with example overlays.

Observability

Metrics, dashboards, and performance tuning.

Troubleshooting

Pending pods, image pull, hung uninstall, ownership conflicts.

Installation

Feature List

Integrations

Management

Operator UI

Reference

Step 1 — install Tensormesh Operator

Step 2 — deploy vLLM connected to the engine

Read the connector config from the engine’s ConfigMap

Create the Deployment

Step 3 — fire two identical requests, see the cache work

Step 4 — run the benchmark

Next steps

Install with Helm

Configuration

Observability

Troubleshooting

​Step 1 — install Tensormesh Operator

​Step 2 — deploy vLLM connected to the engine

​Read the connector config from the engine’s ConfigMap

​Create the Deployment

​Step 3 — fire two identical requests, see the cache work

​Step 4 — run the benchmark

​Next steps

Install with Helm

Configuration

Observability

Troubleshooting

Step 1 — install Tensormesh Operator

Step 2 — deploy vLLM connected to the engine

Read the connector config from the engine’s ConfigMap

Create the Deployment

Step 3 — fire two identical requests, see the cache work

Step 4 — run the benchmark

Next steps