> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# PD Disaggregation

> Separate the prefill and decode phases onto dedicated vLLM instances to improve TTFT and GPU utilization at scale.

PD disaggregation splits the two phases of LLM inference — **prefill** (prompt processing, compute-bound)
and **decode** (token generation, memory-bandwidth-bound) — onto separate vLLM instances. Each pool is
sized and scheduled for its workload, and a router dispatches requests to the right pool. KV tensors
produced by the prefiller are transferred directly to the decoder over NIXL (GPU-to-GPU via the NVIDIA
Transfer Library), so the decoder never re-computes the prompt.

PD disaggregation is most useful when:

* TTFT degrades under heavy load because prefill and decode compete for the same GPU
* you want to scale prefill and decode replicas independently to match traffic patterns
* your cluster has GPU-to-GPU interconnect (NVLink, InfiniBand) that NIXL can exploit

## How it works

A **single** `LMCacheEngine` DaemonSet serves both prefiller and decoder vLLM pods on every node.
When `spec.pd` is set, the engine's connection ConfigMap emits three `kv-transfer-config` keys:

| Key                                 | Content                                                                         |
| ----------------------------------- | ------------------------------------------------------------------------------- |
| `kv-transfer-config.json`           | Bare `LMCacheMPConnector` (fallback for non-PD pods)                            |
| `kv-transfer-config-prefiller.json` | `MultiConnector(NixlConnector + LMCacheMPConnector)` with `kv_role=kv_producer` |
| `kv-transfer-config-decoder.json`   | `MultiConnector(NixlConnector + LMCacheMPConnector)` with `kv_role=kv_consumer` |

Each vLLM pod mounts the engine's connection ConfigMap and reads the role-specific key directly via
`--kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-<role>.json)"`. No webhook is required
for this approach.

The router is a plain `vllm-router` Deployment that you deploy alongside your vLLM pods. The operator
does not manage it — you configure the `--prefill` and `--decode` URLs directly in the Deployment args.

## Prerequisites

| Component                                     | Minimum version    |
| --------------------------------------------- | ------------------ |
| Tensormesh Operator helm chart                | `0.5.3`            |
| LMCache Operator (`lmcache/lmcache-operator`) | `v0.5.3`           |
| LMCache vLLM (`lmcache/vllm-openai`)          | `v0.5.3`           |
| cert-manager                                  | any recent release |

Create a dedicated workload namespace for your vLLM pods and label it PSS-privileged (the pods use
`hostIPC: true` and `hostNetwork: true`):

```bash theme={null}
kubectl create ns pd-workload
kubectl label ns pd-workload pod-security.kubernetes.io/enforce=privileged
```

## Install the chart

A single engine handles both prefiller and decoder vLLM pods. Set `namespace: pd-workload` so the
engine lands in the same namespace as your vLLM pods.

```yaml my-values.yaml theme={null}
engine:
  enabled: true
  name: lmcache-engine
  namespace: pd-workload        # same namespace as your vLLM pods
  spec:
    l1:
      sizeGB: 100
    image:
      repository: lmcache/vllm-openai
      tag: v0.5.3
      pullPolicy: IfNotPresent
    server:
      port: 5555
      httpPort: 8080
      chunkSize: 256
    pd:
      nixlSideChannelPort: 5558
      nixlLoadFailurePolicy: fail
```

```bash theme={null}
helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  -n tensormesh-operator --create-namespace \
  -f my-values.yaml --wait
```

## Deploy vLLM and the router

Deploy your vLLM prefiller, decoder, and router together. Both vLLM Deployments opt in to the same
engine; the `lmcache.ai/pd-role` annotation tells the webhook which `kv-transfer-config` key to inject.

```yaml vllm-pd.yaml theme={null}
###############################################################################
# PREFILLER
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-prefiller
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-prefiller
  template:
    metadata:
      labels:
        app: vllm-prefiller
    spec:
      hostIPC: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: vllm
          image: lmcache/vllm-openai:v0.5.3
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec python3 -m vllm.entrypoints.openai.api_server \
                --model meta-llama/Llama-3.1-8B-Instruct \
                --port 8001 \
                --enforce-eager \
                --gpu-memory-utilization 0.4 \
                --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-prefiller.json)"
          env:
            - name: PYTHONHASHSEED
              value: "0"
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token   # kubectl create secret generic hf-token --from-literal=token=<your-token>
                  key: token
            - name: VLLM_NIXL_SIDE_CHANNEL_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: VLLM_NIXL_SIDE_CHANNEL_PORT
              value: "5557"        # distinct port from decoder to avoid conflict on the same host
          ports:
            - name: http
              containerPort: 8001
          volumeMounts:
            - name: kv-transfer-config
              mountPath: /etc/lmcache
              readOnly: true
          resources:
            limits:
              nvidia.com/gpu: "1"
      volumes:
        - name: kv-transfer-config
          configMap:
            name: lmcache-engine-connection   # created by the LMCacheEngine controller
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-prefiller
  namespace: pd-workload
spec:
  selector:
    app: vllm-prefiller
  ports:
    - name: http
      port: 8001
      targetPort: http
---
###############################################################################
# DECODER
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-decoder
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-decoder
  template:
    metadata:
      labels:
        app: vllm-decoder
    spec:
      hostIPC: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: vllm
          image: lmcache/vllm-openai:v0.5.3
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec python3 -m vllm.entrypoints.openai.api_server \
                --model meta-llama/Llama-3.1-8B-Instruct \
                --port 8002 \
                --enforce-eager \
                --gpu-memory-utilization 0.4 \
                --kv-transfer-config "$(cat /etc/lmcache/kv-transfer-config-decoder.json)"
          env:
            - name: PYTHONHASHSEED
              value: "0"
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
            - name: VLLM_NIXL_SIDE_CHANNEL_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: VLLM_NIXL_SIDE_CHANNEL_PORT
              value: "5558"
          ports:
            - name: http
              containerPort: 8002
          volumeMounts:
            - name: kv-transfer-config
              mountPath: /etc/lmcache
              readOnly: true
          resources:
            limits:
              nvidia.com/gpu: "1"
      volumes:
        - name: kv-transfer-config
          configMap:
            name: lmcache-engine-connection
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-decoder
  namespace: pd-workload
spec:
  selector:
    app: vllm-decoder
  ports:
    - name: http
      port: 8002
      targetPort: http
---
###############################################################################
# ROUTER — plain Deployment, not operator-managed
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-router
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-router
  template:
    metadata:
      labels:
        app: vllm-router
    spec:
      containers:
        - name: router
          image: vllm/vllm-router:nightly   # update to a stable tag when one is available
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec vllm-router \
                --policy round_robin \
                --vllm-pd-disaggregation \
                --prefill http://vllm-prefiller.pd-workload.svc.cluster.local:8001 \
                --decode  http://vllm-decoder.pd-workload.svc.cluster.local:8002 \
                --host 0.0.0.0 \
                --port 30000 \
                --intra-node-data-parallel-size 1
          ports:
            - name: http
              containerPort: 30000
          readinessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-router
  namespace: pd-workload
spec:
  selector:
    app: vllm-router
  ports:
    - name: http
      port: 30000
      targetPort: http
```

```bash theme={null}
kubectl apply -f vllm-pd.yaml
```

<Note>
  The ConfigMap `lmcache-engine-connection` is created automatically when the `LMCacheEngine` reaches
  `Running` state. Wait for the engine to be ready before applying `vllm-pd.yaml`:
  `kubectl get lmcacheengine lmcache-engine -n pd-workload`
</Note>

## NIXL RDMA networking

NIXL uses UCX for GPU-to-GPU KV transfer. UCX requires valid RDMA GIDs, which are derived from the
host's network interfaces. Under standard overlay CNI (each pod has its own network namespace) the
GID table inside the pod is empty and UCX backend initialization fails.

**Workarounds:**

| Approach                                                   | Complexity | Notes                                                                                                                                                                                           |
| ---------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hostNetwork: true` + `dnsPolicy: ClusterFirstWithHostNet` | Low        | Quick test. With hostNetwork both roles share the host IP — set distinct `VLLM_NIXL_SIDE_CHANNEL_PORT` values (e.g. 5557 / 5558) in the pod env; the webhook will not override a pre-set value. |
| SR-IOV with Multus                                         | High       | Production-grade. Each pod gets a dedicated VF with its own GID. No `hostNetwork` required.                                                                                                     |

For the `hostNetwork` workaround, add these fields to both the prefiller and decoder pod specs and
pre-set distinct NIXL ports:

```yaml theme={null}
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  containers:
    - name: vllm
      env:
        - name: VLLM_NIXL_SIDE_CHANNEL_PORT
          value: "5557"    # 5558 for the decoder
```

## Multiple models

One `LMCacheEngine` is shared across all models. Add a new router Deployment for each additional
model, changing `--prefill`, `--decode`, `--port`, and the resource names:

```yaml theme={null}
# router-qwen.yaml — second model alongside the Llama router above
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-router-qwen
  namespace: pd-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-router-qwen
  template:
    metadata:
      labels:
        app: vllm-router-qwen
    spec:
      containers:
        - name: router
          image: vllm/vllm-router:nightly
          command: ["/bin/sh", "-c"]
          args:
            - |
              exec vllm-router \
                --policy round_robin \
                --vllm-pd-disaggregation \
                --prefill http://vllm-prefiller-qwen.pd-workload.svc.cluster.local:8003 \
                --decode  http://vllm-decoder-qwen.pd-workload.svc.cluster.local:8004 \
                --host 0.0.0.0 \
                --port 30001 \
                --intra-node-data-parallel-size 1
          ports:
            - name: http
              containerPort: 30001
          readinessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-router-qwen
  namespace: pd-workload
spec:
  selector:
    app: vllm-router-qwen
  ports:
    - name: http
      port: 30001
      targetPort: http
```

## CacheBlend PD

PD disaggregation also works with [CacheBlend](/operator/configuration/cacheblend). Use a
`CacheBlendEngine` for the prefiller role and a plain `LMCacheEngine` for the decoder. Because
CacheBlend ships a modified vLLM build, set `pd.enforceHandshakeCompat: false` on the prefiller to
disable strict NIXL version negotiation:

```yaml theme={null}
cacheBlend:
  enabled: true
  spec:
    pd:
      nixlSideChannelPort: 5557
      enforceHandshakeCompat: false   # required when prefiller and decoder run different vLLM builds
```

The decoder side uses a standard `LMCacheEngine` with `spec.pd` set (no `enforceHandshakeCompat`
needed on the decoder).

## Helm values reference

### Engine `pd` spec

| Field                       | Type   | Default        | Description                                                                                                                                                                                                                                         |
| --------------------------- | ------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pd.nixlSideChannelPort`    | int    | `5558`         | Port the NIXL agent advertises for side-channel negotiation. With `hostNetwork: true`, use distinct values per role (e.g. 5557 / 5558) by pre-setting `VLLM_NIXL_SIDE_CHANNEL_PORT` in the pod env — the webhook will not override a pre-set value. |
| `pd.nixlLoadFailurePolicy`  | string | `fail`         | What to do when the remote NIXL peer is unreachable. `fail` aborts; `ignore` falls back to local prefill.                                                                                                                                           |
| `pd.enforceHandshakeCompat` | bool   | (NIXL default) | Set to `false` to disable strict NIXL version negotiation — required when a CacheBlend prefiller and a standard decoder run different vLLM builds. Omit to let NIXL use its own default.                                                            |

## Verification

### Engine reconciled

```bash theme={null}
kubectl -n pd-workload get lmcacheengine
kubectl -n pd-workload get configmap | grep connection
```

The engine should be `Running` and have a `lmcache-engine-connection` ConfigMap containing both
prefiller and decoder keys. The ConfigMap is the gate the webhook reads — if it is missing, the pod
starts without NIXL injection and PD transfers will not work.

### Router running

```bash theme={null}
kubectl -n pd-workload get deploy vllm-router
kubectl -n pd-workload get svc vllm-router
```

### Injection happened

After the vLLM pods start:

```bash theme={null}
# Check injection annotation
kubectl -n pd-workload get pod -l app=vllm-prefiller \
  -o jsonpath='{.items[0].metadata.annotations.lmcache\.ai/lmcache-injected}{"\n"}'

# Check NIXL env vars injected
kubectl -n pd-workload get pod -l app=vllm-prefiller \
  -o jsonpath='{.items[0].spec.containers[0].env[*].name}{"\n"}' \
  | tr ' ' '\n' | grep NIXL
```

Expected output includes `VLLM_NIXL_SIDE_CHANNEL_HOST` and `VLLM_NIXL_SIDE_CHANNEL_PORT`.

### Send a request

```bash theme={null}
kubectl port-forward -n pd-workload svc/vllm-router 30000:30000 &

curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Llama-3.1-8B-Instruct","prompt":"The capital of France is","max_tokens":20}'
```

## Common mistakes

* Omitting `namespace: <workload-ns>` on the engine — the engine and its connection ConfigMap land in the operator namespace; the vLLM pods mount the ConfigMap by name, so make sure the engine is in the same namespace as the pods
* Applying `vllm-pd.yaml` before the engine is `Running` — the `lmcache-engine-connection` ConfigMap does not exist yet and the pods fail to start; wait for `kubectl get lmcacheengine` to show `Running`
* NIXL RDMA backend fails to initialize — standard overlay CNI does not populate RDMA GIDs inside pods; `hostNetwork: true` + `dnsPolicy: ClusterFirstWithHostNet` is required for RDMA to work
* Using the same `VLLM_NIXL_SIDE_CHANNEL_PORT` for prefiller and decoder — with `hostNetwork: true` both pods share the host IP so the ports conflict; use `5557` for prefiller and `5558` for decoder
* Missing `IPC_LOCK` capability — NIXL needs to pin memory for RDMA operations; add `capabilities.add: ["IPC_LOCK"]` to the container security context
* Workload namespace not PSS-privileged — Pod Security rejects `hostIPC: true` and `hostNetwork: true`; label the namespace `pod-security.kubernetes.io/enforce=privileged`