> ## Documentation Index > Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt > Use this file to discover all available pages before exploring further. # NVIDIA Dynamo > Offload KV cache from Dynamo vLLM workers to the Tensormesh engine over cross-pod CUDA IPC. ## Prerequisites * A Kubernetes cluster with at least one GPU node * Helm 3.8+ and `kubectl` * **Access token from the Tensormesh team** for the chart registry * A Hugging Face token for model download * [cert-manager](https://cert-manager.io/docs/installation/) — the chart enables its webhook by default and rendering requires the cert-manager APIs ## Step 1 — install Dynamo ```bash theme={null} export NAMESPACE=dynamo-system export DYNAMO_VERSION=1.4.0 helm upgrade --install dynamo-platform \ "https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${DYNAMO_VERSION}.tgz" \ --namespace "$NAMESPACE" \ --create-namespace \ --wait \ --timeout=10m kubectl get pods --namespace "$NAMESPACE" ``` ## Step 2 — create the model access secret Both the frontend and the worker reference `hf-token-secret`; the pods fail to start without it. ```bash theme={null} kubectl create secret generic hf-token-secret \ --namespace "$NAMESPACE" \ --from-literal=HF_TOKEN= \ --dry-run=client -o yaml | kubectl apply -f - ``` ## Step 3 — install the Tensormesh Operator Write the values file — the engine tag is pinned to the LMCache version bundled in the Dynamo image (see the version-matching note above): ```yaml my-values.yaml theme={null} engine: enabled: true namespace: dynamo-system spec: l1: sizeGB: 20 # small for the integration test; bump for production image: repository: lmcache/vllm-openai tag: v0.5.2 # vllm-runtime:1.4.0 ships lmcache 0.5.2 pullPolicy: IfNotPresent coordinator: enabled: false ``` ```bash theme={null} # One-time: log in with the token Tensormesh gave you. echo '' | helm registry login ghcr.io -u tensormesh --password-stdin helm install tensormesh-operator \ oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \ --version 0.5.2 \ --namespace dynamo-system \ --create-namespace \ -f my-values.yaml \ --wait ``` Wait for the engine to reconcile, and note the ConfigMap it publishes — the worker mounts it in the next step: ```bash theme={null} kubectl -n dynamo-system get lmcacheengine # NAME PHASE READY DESIRED # tensormesh-operator-default-engine Running 1 1 kubectl -n dynamo-system get cm tensormesh-operator-default-engine-connection ``` ## Step 4 — deploy vLLM and the frontend in Dynamo Save as `agg.yaml`. Compared to Dynamo's stock aggregated example, the **worker** carries the LMCache wiring; the frontend is unchanged. ```yaml agg.yaml theme={null} apiVersion: nvidia.com/v1beta1 kind: DynamoGraphDeployment metadata: name: vllm-agg spec: components: - name: Frontend podTemplate: spec: containers: - envFrom: - secretRef: name: hf-token-secret image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0 name: main replicas: 1 type: frontend - name: VllmDecodeWorker podTemplate: spec: # Required for CUDA IPC between vLLM and the LMCache server hostIPC: true containers: - args: - --model - Qwen/Qwen3-0.6B - --no-enable-prefix-caching command: - /bin/sh - -c - exec python3 -m dynamo.vllm "$@" --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config.json)" - -- env: - name: PYTHONHASHSEED # deterministic prefix hashing across processes value: "0" envFrom: - secretRef: name: hf-token-secret image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0 name: main resources: limits: nvidia.com/gpu: "1" requests: # Increase this value for larger models. ephemeral-storage: 2Gi # Root uid required for CUDA IPC between vLLM and the LMCache server: # the shared-CUDA-tensor handshake fails under a non-root uid. securityContext: runAsUser: 0 volumeMounts: - mountPath: /etc/lmcache name: kv-transfer-config readOnly: true workingDir: /workspace/examples/backends/vllm volumes: - configMap: name: tensormesh-operator-default-engine-connection name: kv-transfer-config replicas: 1 sharedMemorySize: "0" type: worker ``` ```bash theme={null} kubectl -n dynamo-system apply -f agg.yaml ``` ### Why each change exists | Change | Why | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `hostIPC: true` | The MP connector moves KV tensors over CUDA IPC through the **host's** `/dev/shm`, shared with the node-local engine pod. | | `sharedMemorySize: "0"` | Disables the `/dev/shm` tmpfs the Dynamo controller adds by default — an emptyDir at `/dev/shm` would shadow the host's and break `cudaIpcOpenMemHandle`. | | `securityContext: runAsUser: 0` | The shared-CUDA-tensor handshake fails under a non-root uid. | | `PYTHONHASHSEED=0` | LMCache requires deterministic token hashing across processes for consistent cache keys. | | the `command` wrapper | Reads the connector JSON from the mounted ConfigMap at launch, so the config always matches what the operator generated; Dynamo's `args` still flow through via `"$@"`. | | the ConfigMap volume | `-connection`, published by the operator next to the engine. Same-namespace only — this is why everything lives in `dynamo-system`. | | `--no-enable-prefix-caching` | vLLM's own prefix cache would serve repeats before LMCache is exercised. Keep it while verifying; drop it if you want both cache layers in production. | ## Step 5 — send a request ```bash theme={null} kubectl port-forward \ --namespace "$NAMESPACE" \ "service/vllm-agg-frontend" 8000:8000 \ >/tmp/dynamo-port-forward.log 2>&1 & until curl --silent --fail http://localhost:8000/health >/dev/null; do sleep 2 done curl --silent --show-error http://localhost:8000/v1/chat/completions \ --header "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen3-0.6B", "messages": [ {"role": "user", "content": "What is NVIDIA Dynamo?"} ], "max_tokens": 200 }' ``` ## Verify the cache is working Send the **same long prompt twice** (it must exceed the engine's `chunk_size` — 256 tokens by default — or nothing is stored). The response's `usage` block reports the reuse from the vLLM side: ```text theme={null} call 1: "cached_tokens": 0 call 2: "cached_tokens": 256 ``` And the engine pod on the worker's node logs both sides of the transaction: ```bash theme={null} kubectl -n dynamo-system logs -l app.kubernetes.io/instance=tensormesh-operator-default-engine \ --tail=100 | grep -E "Stored|Retrieved" # Stored 256 tokens in 0.011 seconds # Retrieved 256 tokens in 0.003 seconds ``` A `Creating v1 connector with name: LMCacheMPConnector` line in the worker's log and a `Registered KV cache for GPU ID ... with N layers` line in the engine's log confirm the CUDA IPC handshake completed at startup.