NVIDIA Dynamo

Prerequisites

A Kubernetes cluster with at least one GPU node
Helm 3.8+ and kubectl
Access token from the Tensormesh team for the chart registry
A Hugging Face token for model download
cert-manager — the chart enables its webhook by default and rendering requires the cert-manager APIs

Step 1 — install Dynamo

export NAMESPACE=dynamo-system
export DYNAMO_VERSION=1.4.0

helm upgrade --install dynamo-platform \
  "https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${DYNAMO_VERSION}.tgz" \
  --namespace "$NAMESPACE" \
  --create-namespace \
  --wait \
  --timeout=10m

kubectl get pods --namespace "$NAMESPACE"

Step 2 — create the model access secret

Both the frontend and the worker reference hf-token-secret; the pods fail to start without it.

kubectl create secret generic hf-token-secret \
  --namespace "$NAMESPACE" \
  --from-literal=HF_TOKEN=<YOUR_HF_TOKEN> \
  --dry-run=client -o yaml | kubectl apply -f -

Step 3 — install the Tensormesh Operator

Write the values file — the engine tag is pinned to the LMCache version bundled in the Dynamo image (see the version-matching note above):

my-values.yaml

engine:
  enabled: true
  namespace: dynamo-system
  spec:
    l1:
      sizeGB: 20                      # small for the integration test; bump for production
    image:
      repository: lmcache/vllm-openai
      tag: v0.5.2                     # vllm-runtime:1.4.0 ships lmcache 0.5.2
      pullPolicy: IfNotPresent

coordinator:
  enabled: false

# One-time: log in with the token Tensormesh gave you.
echo '<TOKEN_FROM_TENSORMESH>' | helm registry login ghcr.io -u tensormesh --password-stdin

helm install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  --version 0.5.2 \
  --namespace dynamo-system \
  --create-namespace \
  -f my-values.yaml \
  --wait

Wait for the engine to reconcile, and note the ConfigMap it publishes — the worker mounts it in the next step:

kubectl -n dynamo-system get lmcacheengine
# NAME                                 PHASE     READY   DESIRED
# tensormesh-operator-default-engine   Running   1       1

kubectl -n dynamo-system get cm tensormesh-operator-default-engine-connection

Step 4 — deploy vLLM and the frontend in Dynamo

Save as agg.yaml. Compared to Dynamo’s stock aggregated example, the worker carries the LMCache wiring; the frontend is unchanged.

agg.yaml

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: vllm-agg
spec:
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - envFrom:
          - secretRef:
              name: hf-token-secret
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0
          name: main
    replicas: 1
    type: frontend
  - name: VllmDecodeWorker
    podTemplate:
      spec:
        # Required for CUDA IPC between vLLM and the LMCache server
        hostIPC: true
        containers:
        - args:
          - --model
          - Qwen/Qwen3-0.6B
          - --no-enable-prefix-caching
          command:
          - /bin/sh
          - -c
          - exec python3 -m dynamo.vllm "$@" --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config.json)"
          - --
          env:
          - name: PYTHONHASHSEED   # deterministic prefix hashing across processes
            value: "0"
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              # Increase this value for larger models.
              ephemeral-storage: 2Gi
          # Root uid required for CUDA IPC between vLLM and the LMCache server:
          # the shared-CUDA-tensor handshake fails under a non-root uid.
          securityContext:
            runAsUser: 0
          volumeMounts:
          - mountPath: /etc/lmcache
            name: kv-transfer-config
            readOnly: true
          workingDir: /workspace/examples/backends/vllm
        volumes:
        - configMap:
            name: tensormesh-operator-default-engine-connection
          name: kv-transfer-config
    replicas: 1
    sharedMemorySize: "0"
    type: worker

kubectl -n dynamo-system apply -f agg.yaml

Why each change exists

Change	Why
`hostIPC: true`	The MP connector moves KV tensors over CUDA IPC through the host’s `/dev/shm`, shared with the node-local engine pod.
`sharedMemorySize: "0"`	Disables the `/dev/shm` tmpfs the Dynamo controller adds by default — an emptyDir at `/dev/shm` would shadow the host’s and break `cudaIpcOpenMemHandle`.
`securityContext: runAsUser: 0`	The shared-CUDA-tensor handshake fails under a non-root uid.
`PYTHONHASHSEED=0`	LMCache requires deterministic token hashing across processes for consistent cache keys.
the `command` wrapper	Reads the connector JSON from the mounted ConfigMap at launch, so the config always matches what the operator generated; Dynamo’s `args` still flow through via `"$@"`.
the ConfigMap volume	`<engine-name>-connection`, published by the operator next to the engine. Same-namespace only — this is why everything lives in `dynamo-system`.
`--no-enable-prefix-caching`	vLLM’s own prefix cache would serve repeats before LMCache is exercised. Keep it while verifying; drop it if you want both cache layers in production.

Step 5 — send a request

kubectl port-forward \
  --namespace "$NAMESPACE" \
  "service/vllm-agg-frontend" 8000:8000 \
  >/tmp/dynamo-port-forward.log 2>&1 &

until curl --silent --fail http://localhost:8000/health >/dev/null; do
  sleep 2
done

curl --silent --show-error http://localhost:8000/v1/chat/completions \
  --header "Content-Type: application/json" \
  --data '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "What is NVIDIA Dynamo?"}
    ],
    "max_tokens": 200
  }'

Verify the cache is working

Send the same long prompt twice (it must exceed the engine’s chunk_size — 256 tokens by default — or nothing is stored). The response’s usage block reports the reuse from the vLLM side:

call 1:  "cached_tokens": 0
call 2:  "cached_tokens": 256

And the engine pod on the worker’s node logs both sides of the transaction:

kubectl -n dynamo-system logs -l app.kubernetes.io/instance=tensormesh-operator-default-engine \
  --tail=100 | grep -E "Stored|Retrieved"
# Stored 256 tokens in 0.011 seconds
# Retrieved 256 tokens in 0.003 seconds

A Creating v1 connector with name: LMCacheMPConnector line in the worker’s log and a Registered KV cache for GPU ID ... with N layers line in the engine’s log confirm the CUDA IPC handshake completed at startup.

Installation

Configuration

Integrations

Management

Operator UI

Reference

Prerequisites

Step 1 — install Dynamo

Step 2 — create the model access secret

Step 3 — install the Tensormesh Operator

Step 4 — deploy vLLM and the frontend in Dynamo

Why each change exists

Step 5 — send a request

Verify the cache is working

​Prerequisites

​Step 1 — install Dynamo

​Step 2 — create the model access secret

​Step 3 — install the Tensormesh Operator

​Step 4 — deploy vLLM and the frontend in Dynamo

​Why each change exists

​Step 5 — send a request

​Verify the cache is working

Prerequisites

Step 1 — install Dynamo

Step 2 — create the model access secret

Step 3 — install the Tensormesh Operator

Step 4 — deploy vLLM and the frontend in Dynamo

Why each change exists

Step 5 — send a request

Verify the cache is working