> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# E2E Quickstart
> Install the operator, point vLLM at the engine, run an inference, and verify the KV cache is reused.
## Step 1 — install Tensormesh Operator
Follow [Install with Helm](/installation/helm)
If you already run your own vLLM Deployment, do not replace it with this demo manifest.
Use [Modify an Existing Deployment](/installation/existing-deployment) for the
minimum patch set instead.
## Step 2 — deploy vLLM connected to the engine
The chart creates a ConfigMap named `-connection` holding the
`kv-transfer-config.json` that tells vLLM how to reach the engine over the MP connector.
### Read the connector config from the engine's ConfigMap
```bash theme={null}
kubectl get cm tensormesh-operator-default-engine-connection \
-n tensormesh-operator \
-o jsonpath='{.data.kv-transfer-config\.json}' | jq -c .
```
You should get one line like this:
```json theme={null}
{"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-operator-default-engine.tensormesh-operator.svc.cluster.local","lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"}
```
### Create the Deployment
Paste that line as the `--kv-transfer-config` value in `vllm-demo.yaml`:
```yaml vllm-demo.yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-demo
namespace: tensormesh-operator
labels: { app: vllm-demo }
spec:
selector: { matchLabels: { app: vllm-demo } }
template:
metadata:
labels: { app: vllm-demo }
spec:
nodeSelector:
nvidia.com/gpu.present: "true"
hostIPC: true
containers:
- name: vllm
image: lmcache/vllm-openai:v0.5.3 # use the same version as the lmcache engine
env:
- { name: PYTHONHASHSEED, value: "0" }
args:
- Qwen/Qwen3-0.6B
- --gpu-memory-utilization
- "0.6"
- --no-enable-prefix-caching
- --max-model-len
- "32768"
- --kv-transfer-config
# the single line from the previous step
- '{"kv_connector":"LMCacheMPConnector","kv_connector_extra_config":{"lmcache.mp.host":"tcp://tensormesh-operator-default-engine.tensormesh-operator.svc.cluster.local","lmcache.mp.port":"5555"},"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both"}'
ports: [{ name: http, containerPort: 8000 }]
resources:
limits: { nvidia.com/gpu: "1", memory: 32Gi }
requests: { cpu: "2", memory: 6Gi }
readinessProbe:
httpGet: { path: /health, port: http }
initialDelaySeconds: 60
periodSeconds: 15
failureThreshold: 40
---
apiVersion: v1
kind: Service
metadata:
name: inference
namespace: tensormesh-operator
spec:
selector: { app: vllm-demo }
ports: [{ name: http, port: 8000, targetPort: http }]
```
Apply and wait for the Pod to become Ready (5–10 min on first boot for image pull + model download):
```bash theme={null}
kubectl apply -f vllm-demo.yaml
kubectl wait --for=condition=Ready pod -l app=vllm-demo \
-n tensormesh-operator --timeout=15m
```
## Step 3 — fire two identical requests, see the cache work
The simplest possible "is LMCache working?" test is to send two identical long prompts.
The first populates the cache; the second should reuse the stored KV blocks.
Port-forward the service in one terminal:
```bash theme={null}
kubectl port-forward -n tensormesh-operator svc/inference 8000:8000 --address 127.0.0.1
```
If port `8000` is already in use locally, forward to a different local port — e.g.
`... 8001:8000` — and use that port in the requests below.
In a second terminal, build the request once into a variable, then fire it twice. Keeping
the JSON in a `PAYLOAD` variable (rather than inlining `$(jq …)` inside a multi-line
`curl`) makes it safe to copy-paste and guarantees both calls send the **identical**
prompt:
```bash theme={null}
PROMPT=$(python3 -c "print('Tell me a long story about a brave knight. ' * 30)")
PAYLOAD=$(jq -n --arg p "$PROMPT" \
'{model:"Qwen/Qwen3-0.6B", prompt:$p, max_tokens:20, temperature:0}')
# Call 1 — cold, should STORE KV into LMCache
time curl -sS http://localhost:8000/v1/completions \
-H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text'
# Call 2 — identical, should RETRIEVE prefix KV from LMCache
time curl -sS http://localhost:8000/v1/completions \
-H 'Content-Type: application/json' -d "$PAYLOAD" | jq -r '.choices[0].text'
```
Then check the engine logs for store and retrieve markers:
```bash theme={null}
# select by the engine CR's name; the component label (cache-engine) also matches
# CacheBlend engine pods when those are enabled
kubectl logs -n tensormesh-operator --tail=200 \
-l app.kubernetes.io/instance=tensormesh-operator-default-engine \
| grep -E "Stored [0-9]+ tokens|Prefetch request completed.*prefix hits"
```
A working install logs `Stored N tokens` after call 1 and
`Prefetch request completed ... prefix hits=N` during call 2.
Sample output from a Qwen3-0.6B install on a single A100 40 GB, prompt repeated 30 times
for \~514 tokens:
```text theme={null}
==========================================
CALL 1 (cold — expect STORE)
==========================================
elapsed: 1.466s
usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534}
==========================================
CALL 2 (warm — expect RETRIEVE)
==========================================
elapsed: 0.548s
usage: {'prompt_tokens': 514, 'completion_tokens': 20, 'total_tokens': 534}
==========================================
ENGINE LOG MARKERS
==========================================
LMCache INFO: Stored 512 tokens in 0.029 seconds
LMCache INFO: Prefetch request completed (L1+L2): 2/2 prefix hits (2 L1, 0 L2) in 0.9 ms
```
Call 2 was \~2.7× faster than call 1 (1.466 s → 0.548 s). The engine reports `2/2 prefix
hits` from L1, meaning the second request reused the KV blocks stored during the first.
vLLM's own metrics line confirms the same number from the inference side:
```text theme={null}
External prefix cache hit rate: 49.8%
```
(The counter is cumulative across both calls: call 1 hit 0 of 514 tokens, call 2 hit 512
of 514, so 512/1028 = 49.8%. For a two-call test \~50% is the ceiling, not a partial hit.)
Prompts must exceed the engine's `chunk_size` (256 tokens by default) for any KV blocks
to be stored. Short prompts produce no markers — that's expected, not a failure.
## Step 4 — run the benchmark
For a more realistic measurement than two curls, use `vllm bench serve` to compare a cold
pass against a warm pass. Run **two passes with the same `--seed`** so the prompts repeat
and the second pass hits the cache.
Rather than installing the vLLM CLI locally, run it **inside the cluster as a `Job`** — the
vLLM image already ships the CLI, talks to the `inference` Service directly (no
port-forward), and a `Job` runs it once to completion and stops. The benchmark is a pure
HTTP client, so it needs **no GPU**.
Save as `benchmark-job.yaml`:
```yaml benchmark-job.yaml theme={null}
apiVersion: batch/v1
kind: Job
metadata:
name: lmcache-benchmark
namespace: tensormesh-operator
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 3600
template:
metadata:
labels: { app: lmcache-benchmark }
spec:
restartPolicy: Never
containers:
- name: bench
image: lmcache/vllm-openai:v0.5.3 # same tag as the Deployment, so the node's cached image is reused
imagePullPolicy: IfNotPresent
env:
- { name: HF_HOME, value: /tmp/hf } # writable under OpenShift's random UID
- { name: HF_HUB_DISABLE_TELEMETRY, value: "1" }
command: ["/bin/sh", "-c"]
args:
- |
set -e
run() {
vllm bench serve --backend openai --base-url http://inference:8000 \
--model Qwen/Qwen3-0.6B --dataset-name random \
--num-prompts 20 --random-input-len 20480 \
--random-output-len 1 --seed 32
}
echo "### PASS 1 (cold) ###"; run
sleep 5
echo "### PASS 2 (warm) ###"; run
volumeMounts:
- { name: hf, mountPath: /tmp/hf }
resources:
requests: { cpu: "1", memory: 2Gi }
limits: { cpu: "2", memory: 4Gi }
volumes:
- { name: hf, emptyDir: {} }
```
Run it and stream the results:
```bash theme={null}
kubectl apply -f benchmark-job.yaml
kubectl wait --for=condition=complete job/lmcache-benchmark \
-n tensormesh-operator --timeout=10m
kubectl logs -f job/lmcache-benchmark -n tensormesh-operator
```
The logs contain two `Serving Benchmark Result` blocks. A working cache shows **pass 2 with
much lower TTFT** (mean/p50/p99). The speedup scales with `--random-input-len`: with a fixed
`--seed`, both passes send the same 20 prompts, so each unique body is prefilled on the cold
pass and served from cache on the warm pass — bigger `--random-input-len` → bigger delta.
Example result from a working run with `--num-prompts 20`, `--random-input-len 20480`, and
`--seed 32`:
```text theme={null}
### PASS 1 (cold) ###
Successful requests: 20
Benchmark duration (s): 11.00
Request throughput (req/s): 1.82
Mean TTFT (ms): 6399.80
Median TTFT (ms): 6456.31
P99 TTFT (ms): 10896.55
### PASS 2 (warm) ###
Successful requests: 20
Benchmark duration (s): 5.64
Request throughput (req/s): 3.55
Mean TTFT (ms): 3697.56
Median TTFT (ms): 3753.21
P99 TTFT (ms): 5584.58
```
In the healthy case above, pass 2 cuts TTFT substantially relative to pass 1 and nearly
doubles request throughput.
Keep `num_prompts × random-input-len` of KV under `engine.spec.l1.sizeGB` (else pass 1's
entries evict before pass 2 reads them — warm misses), and `random-input-len + random-output-len ≤ --max-model-len` (32768 in the Deployment above).
Clean up the Job (it also self-deletes after 1h via `ttlSecondsAfterFinished`):
```bash theme={null}
kubectl delete job lmcache-benchmark -n tensormesh-operator
```
## Next steps
Full chart reference, install modes, every tunable value.
Every `values.yaml` key, with example overlays.
Metrics, dashboards, and performance tuning.
`Pending` pods, image pull, hung uninstall, ownership conflicts.