PD Disaggregation

PD disaggregation splits the two phases of LLM inference — prefill (prompt processing, compute-bound) and decode (token generation, memory-bandwidth-bound) — onto separate vLLM instances. Each pool is sized and scheduled for its workload, and a router dispatches requests to the right pool. KV tensors produced by the prefiller are transferred directly to the decoder over NIXL (GPU-to-GPU via the NVIDIA Transfer Library), so the decoder never re-computes the prompt. PD disaggregation is most useful when:

TTFT degrades under heavy load because prefill and decode compete for the same GPU
you want to scale prefill and decode replicas independently to match traffic patterns
your cluster has GPU-to-GPU interconnect (NVLink, InfiniBand) that NIXL can exploit

How it works

A single LMCacheEngine DaemonSet serves both prefiller and decoder vLLM pods on every node. When spec.pd is set, the engine’s connection ConfigMap emits three kv-transfer-config keys:

Key	Content
`kv-transfer-config.json`	Bare `LMCacheMPConnector` (fallback for non-PD pods)
`kv-transfer-config-prefiller.json`	`MultiConnector(NixlConnector + LMCacheMPConnector)` with `kv_role=kv_producer`
`kv-transfer-config-decoder.json`	`MultiConnector(NixlConnector + LMCacheMPConnector)` with `kv_role=kv_consumer`

Each vLLM pod mounts the engine’s connection ConfigMap and reads the role-specific key directly via --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-<role>.json)". No webhook is required for this approach. The router is a plain vllm-router Deployment that you deploy alongside your vLLM pods. The operator does not manage it — you configure the --prefill and --decode URLs directly in the Deployment args.

Prerequisites

Component	Minimum version
Tensormesh Operator helm chart	`0.5.3`
LMCache Operator (`lmcache/lmcache-operator`)	`v0.5.3`
LMCache vLLM (`lmcache/vllm-openai`)	`v0.5.3`
cert-manager	any recent release

Create a dedicated workload namespace for your vLLM pods and label it PSS-privileged (the pods use hostIPC: true and hostNetwork: true):

kubectl create ns pd-workload
kubectl label ns pd-workload pod-security.kubernetes.io/enforce=privileged

Install the chart

A single engine handles both prefiller and decoder vLLM pods. Set namespace: pd-workload so the engine lands in the same namespace as your vLLM pods.

my-values.yaml

engine:
  enabled: true
  name: lmcache-engine
  namespace: pd-workload        # same namespace as your vLLM pods
  spec:
    l1:
      sizeGB: 100
    image:
      repository: lmcache/vllm-openai
      tag: v0.5.3
      pullPolicy: IfNotPresent
    server:
      port: 5555
      httpPort: 8080
      chunkSize: 256
    pd:
      nixlSideChannelPort: 5558
      nixlLoadFailurePolicy: fail

helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  -n tensormesh-operator --create-namespace \
  -f my-values.yaml --wait

Deploy vLLM and the router

Deploy your vLLM prefiller, decoder, and router together. Both vLLM Deployments opt in to the same engine; the lmcache.ai/pd-role annotation tells the webhook which kv-transfer-config key to inject.

vllm-pd.yaml

###############################################################################
# PREFILLER
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-prefiller
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-prefiller
  template:
    metadata:
      labels:
        app: vllm-prefiller
    spec:
      hostIPC: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: vllm
          image: lmcache/vllm-openai:v0.5.3
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec python3 -m vllm.entrypoints.openai.api_server \
                --model meta-llama/Llama-3.1-8B-Instruct \
                --port 8001 \
                --enforce-eager \
                --gpu-memory-utilization 0.4 \
                --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-prefiller.json)"
          env:
            - name: PYTHONHASHSEED
              value: "0"
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token   # kubectl create secret generic hf-token --from-literal=token=<your-token>
                  key: token
            - name: VLLM_NIXL_SIDE_CHANNEL_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: VLLM_NIXL_SIDE_CHANNEL_PORT
              value: "5557"        # distinct port from decoder to avoid conflict on the same host
          ports:
            - name: http
              containerPort: 8001
          volumeMounts:
            - name: kv-transfer-config
              mountPath: /etc/lmcache
              readOnly: true
          resources:
            limits:
              nvidia.com/gpu: "1"
      volumes:
        - name: kv-transfer-config
          configMap:
            name: lmcache-engine-connection   # created by the LMCacheEngine controller
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-prefiller
  namespace: pd-workload
spec:
  selector:
    app: vllm-prefiller
  ports:
    - name: http
      port: 8001
      targetPort: http
---
###############################################################################
# DECODER
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-decoder
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-decoder
  template:
    metadata:
      labels:
        app: vllm-decoder
    spec:
      hostIPC: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: vllm
          image: lmcache/vllm-openai:v0.5.3
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec python3 -m vllm.entrypoints.openai.api_server \
                --model meta-llama/Llama-3.1-8B-Instruct \
                --port 8002 \
                --enforce-eager \
                --gpu-memory-utilization 0.4 \
                --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-decoder.json)"
          env:
            - name: PYTHONHASHSEED
              value: "0"
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
            - name: VLLM_NIXL_SIDE_CHANNEL_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: VLLM_NIXL_SIDE_CHANNEL_PORT
              value: "5558"
          ports:
            - name: http
              containerPort: 8002
          volumeMounts:
            - name: kv-transfer-config
              mountPath: /etc/lmcache
              readOnly: true
          resources:
            limits:
              nvidia.com/gpu: "1"
      volumes:
        - name: kv-transfer-config
          configMap:
            name: lmcache-engine-connection
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-decoder
  namespace: pd-workload
spec:
  selector:
    app: vllm-decoder
  ports:
    - name: http
      port: 8002
      targetPort: http
---
###############################################################################
# ROUTER — plain Deployment, not operator-managed
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-router
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-router
  template:
    metadata:
      labels:
        app: vllm-router
    spec:
      containers:
        - name: router
          image: vllm/vllm-router:nightly   # update to a stable tag when one is available
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec vllm-router \
                --policy round_robin \
                --vllm-pd-disaggregation \
                --prefill http://vllm-prefiller.pd-workload.svc.cluster.local:8001 \
                --decode  http://vllm-decoder.pd-workload.svc.cluster.local:8002 \
                --host 0.0.0.0 \
                --port 30000 \
                --intra-node-data-parallel-size 1
          ports:
            - name: http
              containerPort: 30000
          readinessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-router
  namespace: pd-workload
spec:
  selector:
    app: vllm-router
  ports:
    - name: http
      port: 30000
      targetPort: http

kubectl apply -f vllm-pd.yaml

The ConfigMap lmcache-engine-connection is created automatically when the LMCacheEngine reaches Running state. Wait for the engine to be ready before applying vllm-pd.yaml: kubectl get lmcacheengine lmcache-engine -n pd-workload

NIXL RDMA networking

NIXL uses UCX for GPU-to-GPU KV transfer. UCX requires valid RDMA GIDs, which are derived from the host’s network interfaces. Under standard overlay CNI (each pod has its own network namespace) the GID table inside the pod is empty and UCX backend initialization fails. Workarounds:

Approach	Complexity	Notes
`hostNetwork: true` + `dnsPolicy: ClusterFirstWithHostNet`	Low	Quick test. With hostNetwork both roles share the host IP — set distinct `VLLM_NIXL_SIDE_CHANNEL_PORT` values (e.g. 5557 / 5558) in the pod env; the webhook will not override a pre-set value.
SR-IOV with Multus	High	Production-grade. Each pod gets a dedicated VF with its own GID. No `hostNetwork` required.

For the hostNetwork workaround, add these fields to both the prefiller and decoder pod specs and pre-set distinct NIXL ports:

spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  containers:
    - name: vllm
      env:
        - name: VLLM_NIXL_SIDE_CHANNEL_PORT
          value: "5557"    # 5558 for the decoder

Multiple models

One LMCacheEngine is shared across all models. Add a new router Deployment for each additional model, changing --prefill, --decode, --port, and the resource names:

# router-qwen.yaml — second model alongside the Llama router above
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-router-qwen
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-router-qwen
  template:
    metadata:
      labels:
        app: vllm-router-qwen
    spec:
      containers:
        - name: router
          image: vllm/vllm-router:nightly
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec vllm-router \
                --policy round_robin \
                --vllm-pd-disaggregation \
                --prefill http://vllm-prefiller-qwen.pd-workload.svc.cluster.local:8003 \
                --decode  http://vllm-decoder-qwen.pd-workload.svc.cluster.local:8004 \
                --host 0.0.0.0 \
                --port 30001 \
                --intra-node-data-parallel-size 1
          ports:
            - name: http
              containerPort: 30001
          readinessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-router-qwen
  namespace: pd-workload
spec:
  selector:
    app: vllm-router-qwen
  ports:
    - name: http
      port: 30001
      targetPort: http

CacheBlend PD

PD disaggregation also works with CacheBlend. Use a CacheBlendEngine for the prefiller role and a plain LMCacheEngine for the decoder. Because CacheBlend ships a modified vLLM build, set pd.enforceHandshakeCompat: false on the prefiller to disable strict NIXL version negotiation:

cacheBlend:
  enabled: true
  spec:
    pd:
      nixlSideChannelPort: 5557
      enforceHandshakeCompat: false   # required when prefiller and decoder run different vLLM builds

The decoder side uses a standard LMCacheEngine with spec.pd set (no enforceHandshakeCompat needed on the decoder).

Helm values reference

Engine `pd` spec

Field	Type	Default	Description
`pd.nixlSideChannelPort`	int	`5558`	Port the NIXL agent advertises for side-channel negotiation. With `hostNetwork: true`, use distinct values per role (e.g. 5557 / 5558) by pre-setting `VLLM_NIXL_SIDE_CHANNEL_PORT` in the pod env — the webhook will not override a pre-set value.
`pd.nixlLoadFailurePolicy`	string	`fail`	What to do when the remote NIXL peer is unreachable. `fail` aborts; `ignore` falls back to local prefill.
`pd.enforceHandshakeCompat`	bool	(NIXL default)	Set to `false` to disable strict NIXL version negotiation — required when a CacheBlend prefiller and a standard decoder run different vLLM builds. Omit to let NIXL use its own default.

Verification

Engine reconciled

kubectl -n pd-workload get lmcacheengine
kubectl -n pd-workload get configmap | grep connection

The engine should be Running and have a lmcache-engine-connection ConfigMap containing both prefiller and decoder keys. The ConfigMap is the gate the webhook reads — if it is missing, the pod starts without NIXL injection and PD transfers will not work.

Router running

kubectl -n pd-workload get deploy vllm-router
kubectl -n pd-workload get svc vllm-router

Injection happened

After the vLLM pods start:

# Check injection annotation
kubectl -n pd-workload get pod -l app=vllm-prefiller \
  -o jsonpath='{.items[0].metadata.annotations.lmcache\.ai/lmcache-injected}{"\n"}'

# Check NIXL env vars injected
kubectl -n pd-workload get pod -l app=vllm-prefiller \
  -o jsonpath='{.items[0].spec.containers[0].env[*].name}{"\n"}' \
  | tr ' ' '\n' | grep NIXL

Expected output includes VLLM_NIXL_SIDE_CHANNEL_HOST and VLLM_NIXL_SIDE_CHANNEL_PORT.

Send a request

kubectl port-forward -n pd-workload svc/vllm-router 30000:30000 &

curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Llama-3.1-8B-Instruct","prompt":"The capital of France is","max_tokens":20}'

Common mistakes

Omitting namespace: <workload-ns> on the engine — the engine and its connection ConfigMap land in the operator namespace; the vLLM pods mount the ConfigMap by name, so make sure the engine is in the same namespace as the pods
Applying vllm-pd.yaml before the engine is Running — the lmcache-engine-connection ConfigMap does not exist yet and the pods fail to start; wait for kubectl get lmcacheengine to show Running
NIXL RDMA backend fails to initialize — standard overlay CNI does not populate RDMA GIDs inside pods; hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet is required for RDMA to work
Using the same VLLM_NIXL_SIDE_CHANNEL_PORT for prefiller and decoder — with hostNetwork: true both pods share the host IP so the ports conflict; use 5557 for prefiller and 5558 for decoder
Missing IPC_LOCK capability — NIXL needs to pin memory for RDMA operations; add capabilities.add: ["IPC_LOCK"] to the container security context
Workload namespace not PSS-privileged — Pod Security rejects hostIPC: true and hostNetwork: true; label the namespace pod-security.kubernetes.io/enforce=privileged

Installation

Configuration

Management

Reference

How it works

Prerequisites

Install the chart

Deploy vLLM and the router

NIXL RDMA networking

Multiple models

CacheBlend PD

Helm values reference

Engine `pd` spec

Verification

Engine reconciled

Router running

Injection happened

Send a request

Common mistakes

​How it works

​Prerequisites

​Install the chart

​Deploy vLLM and the router

​NIXL RDMA networking

​Multiple models

​CacheBlend PD

​Helm values reference

​Engine pd spec

​Verification

​Engine reconciled

​Router running

​Injection happened

​Send a request

​Common mistakes

How it works

Prerequisites

Install the chart

Deploy vLLM and the router

NIXL RDMA networking

Multiple models

CacheBlend PD

Helm values reference

Engine `pd` spec

Verification

Engine reconciled

Router running

Injection happened

Send a request

Common mistakes