> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Peer-to-Peer KV Transfer

> Enable direct KV cache transfer between engine pods so warm prefixes can be read from any peer in the fleet, not just the local L1 or a shared L2 backend.

Peer-to-peer (P2P) KV transfer lets engine pods read cached KV data directly from each other's
memory over a transfer channel. When a pod receives a request whose prefix was already computed by
another pod, it can fetch the KV tensors from that peer instead of recomputing them. This
supplements the L1 (local memory) and L2 (Redis, filesystem, etc.) tiers with a third path:
**any peer in the fleet can serve as a warm cache source**.

P2P is most useful when:

* your fleet serves many models or tenants and prompt reuse spans nodes
* you want warm-hit latency close to L1 without provisioning a large shared L2
* your cluster has high-bandwidth interconnect (InfiniBand, RoCE, or fast TCP)

For background on P2P in LMCache, see the upstream
[P2P documentation](https://docs.lmcache.ai/mp/p2p.html).

## Prerequisites

P2P requires the following minimum versions:

| Component                                     | Minimum version |
| --------------------------------------------- | --------------- |
| Tensormesh Operator helm chart                | `0.5.0`         |
| LMCache Operator (`lmcache/lmcache-operator`) | `v0.5.0`        |
| LMCache vLLM (`lmcache/vllm-openai`)          | `v0.5.0`        |

## Configuration

Enable P2P by setting `engine.p2p.enabled: true` in your values file. The chart handles the rest
— no manual CLI flags needed.

```yaml theme={null}
engine:
  p2p:
    enabled: true
    port: 48500                  # transfer channel port peers connect to
    l1AlignBytesOverwrite: 65536 # 64 KiB alignment, recommended for P2P
```

P2P also requires that every engine pod can reach every other engine pod on the transfer channel
port. How you achieve that depends on your cluster's network fabric — see the networking options
below.

## Networking options

### Option 1: Host networking (easiest to get started)

The simplest approach is `hostNetwork: true`. The engine pod binds directly to the host's network
namespace, so peers can reach each other without any CNI overlay or NAT.

Because the pod shares the host's port space, move the default server ports to high values to
avoid conflicts with other services on the node:

```yaml my-values.yaml theme={null}
engine:
  enabled: true
  name: tensormesh-engine
  p2p:
    enabled: true
    port: 48500                  # P2P transfer channel
    l1AlignBytesOverwrite: 65536 # 64 KiB, recommended for P2P
  spec:
    hostNetwork: true
    server:
      port: 45555                # gRPC (default 5555)
      httpPort: 48080            # health/admin (default 8080)
    l1:
      sizeGB: 60
    image:
      repository: lmcache/vllm-openai
      tag: v0.5.0
```

Apply it with:

```bash theme={null}
helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  -n tensormesh-operator --create-namespace \
  -f my-values.yaml --wait
```

<Note>
  When `hostNetwork` is `true`, the operator automatically sets `dnsPolicy: ClusterFirstWithHostNet`
  so cluster DNS still works inside the engine pod.
</Note>

### Option 2: HCA passthrough (coming soon)

<Info>
  **Coming soon** — recommended for InfiniBand environments.
</Info>

HCA (Host Channel Adapter) passthrough exposes the InfiniBand device directly to the engine pod.
This gives the transfer channel native RDMA verbs without any overlay or kernel bypass
emulation. The operator will support scheduling HCA resources via the
[k8s-rdma-shared-dev-plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin) so that each
engine pod gets a dedicated HCA device.

### Option 3: SR-IOV (coming soon)

<Info>
  **Coming soon** — recommended for RoCE environments.
</Info>

SR-IOV (Single Root I/O Virtualization) creates lightweight virtual functions (VFs) from a physical
NIC, each assigned to a pod. This gives near-hardware RDMA performance while allowing multiple pods
to share the same physical NIC. The operator will integrate with the
[SR-IOV Network Operator](https://github.com/k8snetworkplumbingwg/sriov-network-operator) to
request VFs via a `NetworkAttachmentDefinition`.

### Option 4: Macvlan (coming soon)

<Info>
  **Coming soon** — recommended for RoCE environments.
</Info>

Macvlan creates a virtual interface directly on the physical NIC's L2 segment. Each pod gets its
own MAC address and IP on the fabric network, bypassing the CNI overlay. This is simpler to set
up than SR-IOV (no VF provisioning) but still gives pods direct L2 reachability on the RDMA
fabric. The operator will integrate with
[Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) to attach a secondary macvlan
interface to engine pods.

### Option 5: TCP (non-RDMA environments)

If your cluster does not have InfiniBand or RoCE hardware, P2P still works over TCP. Performance
is lower than RDMA but still faster than recomputing prefills from scratch.

Use the same host-networking configuration from Option 1 and set `UCX_TLS` to TCP-only transports:

```yaml theme={null}
engine:
  spec:
    env:
      - name: UCX_TLS
        value: "tcp,sm,self"
```

No special NIC passthrough or CNI plugin is needed. This is the right choice for:

* development and testing clusters
* cloud environments without RDMA-capable instances
* clusters where the CNI overlay provides sufficient bandwidth

## Helm values reference

| Value                              | Type | Default | Description                                                           |
| ---------------------------------- | ---- | ------- | --------------------------------------------------------------------- |
| `engine.p2p.enabled`               | bool | `false` | Enable P2P KV transfer between engine pods.                           |
| `engine.p2p.port`                  | int  | `8500`  | Port the transfer channel advertises to peers.                        |
| `engine.p2p.l1AlignBytesOverwrite` | int  | `65536` | L1 buffer alignment in bytes. 64 KiB recommended for P2P.             |
| `engine.spec.hostNetwork`          | bool | `false` | Run the pod in the host's network namespace.                          |
| `engine.spec.server.port`          | int  | `5555`  | gRPC server port. Move to a high port when using `hostNetwork`.       |
| `engine.spec.server.httpPort`      | int  | `8080`  | HTTP health/admin port. Move to a high port when using `hostNetwork`. |

## Verification

After deploying with P2P enabled, verify that the engine pods started correctly and the P2P
transfer channel is active.

### Check the engine pod is running

```bash theme={null}
kubectl get pods -n tensormesh-operator -l app.kubernetes.io/component=cache-engine
```

### Query the engine's L2 backends

Curl the engine's HTTP endpoint to list the active L2 backends. When P2P is enabled, a P2P
backend should appear alongside any configured L2 (Redis, filesystem, etc.):

```bash theme={null}
ENGINE_POD=$(kubectl get pods -n tensormesh-operator \
  -l app.kubernetes.io/component=cache-engine -o name | head -1)

# Port-forward the HTTP port (use your configured httpPort)
kubectl port-forward -n tensormesh-operator "$ENGINE_POD" 48080:48080 &

curl -s http://localhost:48080/l2/backends | jq .
```

<Note>
  The L2 backends endpoint and its response format may change in future releases. This section
  will be updated as the API stabilizes.
</Note>

## Common mistakes

* Forgetting to move `server.port` and `server.httpPort` to high ports when using `hostNetwork`,
  causing bind failures on ports already in use by the host
* Not setting `engine.p2p.enabled: true` — `hostNetwork` alone does not turn on P2P
* Using the default P2P port (`8500`) on `hostNetwork` when another service already binds to it
* Setting `UCX_TLS` to an RDMA transport (e.g. `rc_verbs`) without the corresponding device
  passthrough — UCX will fail to open the transport and the engine will not start