> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Non-Prefix KV Caching

> Reuse KV cache beyond prefix boundaries with CacheBlend

[CacheBlend](https://arxiv.org/abs/2405.16444) is a technique introduced by the Tensormesh team that
reuses precomputed KV caches beyond prefix boundaries. Rather than requiring an exact prefix match, it
reuses any contiguous block of previously computed tokens regardless of position, then selectively
recomputes a small subset of tokens to partially update each reused block.

CacheBlend is most useful when:

* prompt reuse is **non-prefix** — shared blocks sit after a per-request preamble, system prompt, or other documents
* you serve **RAG** workloads where the same documents are retrieved in varying order across requests

## Prerequisites

| Component                                     | Minimum version    |
| --------------------------------------------- | ------------------ |
| Tensormesh Operator helm chart                | `0.5.3`            |
| LMCache Operator (`lmcache/lmcache-operator`) | `v0.5.3`           |
| LMCache vLLM (`lmcache/vllm-openai`)          | `v0.5.3`           |
| cacheblend-plugin (`private image`)           | `v0.5.3`           |
| cert-manager                                  | any recent release |

## Install Chart

Install the chart with CacheBlend enabled. This creates the `CacheBlendEngine` — a per-node DaemonSet —
in the `tensormesh-operator` namespace:

```yaml my-values.yaml theme={null}
# REQUIRED — the mutating webhook injects the plugin (needs cert-manager).
webhook:
  enabled: true

cacheBlend:
  enabled: true
  name: tensormesh-cacheblend        # engine created in the release namespace
  spec:
    l1:
      sizeGB: 200
    image:
      repository: lmcache/vllm-openai
      tag: <matched-tag>             # same lmcache build as the plugin below
      pullPolicy: IfNotPresent
    injection:
      payloadImage:
        repository: tensormesh/cacheblend-plugin   # PRIVATE
        tag: <matched-tag>
        pullPolicy: IfNotPresent
      imagePullSecrets:
        - name: cacheblend-plugin-pull             # must exist in each workload ns
```

```bash theme={null}
helm upgrade --install tensormesh-operator \
  oci://ghcr.io/tensormesh-production/charts/tensormesh-operator \
  -n tensormesh-operator --create-namespace \
  -f my-values.yaml --wait
```

## Opting a vLLM pod in

CacheBlend plugin attaches to vLLM pods **you deploy** — the operator does not create them. First create the
workload namespace (PSS-privileged) and the plugin pull secret — the private init container is pulled in
the **pod's** namespace, so the secret must live here:

```bash theme={null}
kubectl create ns cacheblend-workload
kubectl label ns cacheblend-workload pod-security.kubernetes.io/enforce=privileged
kubectl -n cacheblend-workload create secret docker-registry cacheblend-plugin-pull \
  --docker-server=docker.io --docker-username=<user> --docker-password=<token>
```

Then add a label + annotation to your pod template and launch vLLM **args-only** (a `command:` override
makes the webhook skip injection):

```yaml theme={null}
metadata:
  labels:
    lmcache.ai/cacheblend-inject: "true"                                        # opt in
  annotations:
    lmcache.ai/cacheblend-engine: "tensormesh-operator/tensormesh-cacheblend"   # <release-ns>/<name>
```

Everything else (image, model, resources, replicas) stays your normal vLLM config. At admission the
webhook injects the plugin init container, `hostIPC`, the `CUSTOM` attention backend, and the
`--kv-transfer-config` (`CBKVConnector`) pointing at the node-local engine; blend tunables
(`cb.check_layer`, `cb.recomp_ratio`) come from the engine config.

## Helm values reference

| Value                                        | Type   | Default               | Description                                                                                                                            |
| -------------------------------------------- | ------ | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `cacheBlend.enabled`                         | bool   | `false`               | Create a `CacheBlendEngine` CR. Requires `webhook.enabled: true` + cert-manager.                                                       |
| `cacheBlend.name`                            | string | `""`                  | Engine CR name. Empty = `<fullname>-cacheblend`.                                                                                       |
| `cacheBlend.namespace`                       | string | `""`                  | Namespace for the engine + its connection ConfigMap. Empty (recommended) = release namespace; opt-in pods bind it cross-namespace.     |
| `cacheBlend.spec.l1.sizeGB`                  | int    | `60`                  | L1 (host-DRAM) chunk cache size, in GiB.                                                                                               |
| `cacheBlend.spec.image`                      | object | `lmcache/vllm-openai` | Blend-engine image (the lmcache server).                                                                                               |
| `cacheBlend.spec.injection.payloadImage`     | object | —                     | PRIVATE `cacheblend-plugin` image injected as the init container. Must match the engine's lmcache build.                               |
| `cacheBlend.spec.injection.imagePullSecrets` | list   | `[]`                  | Pull secret names appended to opt-in pods for the private payload image. The Secret(s) must already exist in the vLLM pod's namespace. |
| `webhook.enabled`                            | bool   | `true`                | Enable the CacheBlend mutating webhook. Requires cert-manager.                                                                         |

## Verification

### Engine reconciled

```bash theme={null}
kubectl -n tensormesh-operator get cacheblendengine tensormesh-cacheblend
kubectl -n tensormesh-operator get configmap tensormesh-cacheblend-connection
```

The `<engine>-connection` ConfigMap is the reconcile proof — and the gate the webhook reads. If it is
missing, injection silently fail-opens (the pod runs without CacheBlend).

### Webhook wired

```bash theme={null}
kubectl get mutatingwebhookconfiguration tensormesh-operator-mutating-webhook \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | head -c 20; echo
```

A non-empty `caBundle` means cert-manager issued the serving cert and the webhook can be called.

### Injection happened

After creating an opt-in vLLM pod in the workload namespace:

```bash theme={null}
P=$(kubectl -n cacheblend-workload get pod -l lmcache.ai/cacheblend-inject=true -o name | head -1)
kubectl -n cacheblend-workload get $P -o jsonpath='{.metadata.annotations.lmcache\.ai/cacheblend-injected}{"\n"}'  # present = injected
kubectl -n cacheblend-workload get $P -o jsonpath='{.spec.initContainers[*].name}{"\n"}'   # cb-plugin-stage
kubectl -n cacheblend-workload get $P -o jsonpath='{.spec.hostIPC}{"\n"}'                   # true
```

### Non-prefix blend actually engaged

Send two requests that share a block **after different-length preambles**, then check the engine logs.
A `shifted` retrieve means a chunk was reused at a different position than it was cached — i.e. a
non-prefix blend (select the engine by `instance` to skip any regular LMCacheEngine in the same ns):

```bash theme={null}
kubectl -n tensormesh-operator logs -l app.kubernetes.io/instance=tensormesh-cacheblend --tail=500 \
  | grep -E "\[match_probe\] .* matches=[1-9]|Retrieved pre-computed for [1-9].* shifted=[1-9]"
```

## Common mistakes

* Running the opt-in vLLM pod in the **operator (release) namespace** — the webhook excludes it, so
  injection never happens
* A **bare** engine ref (`tensormesh-cacheblend`) instead of `<release-ns>/tensormesh-cacheblend` —
  the webhook looks in the pod's own namespace and finds no engine
* Setting a `command:` on the vLLM container — the webhook skips injection (appended args would never
  reach `vllm serve`)
* Workload namespace **not PSS-privileged** — Pod Security rejects the injected `hostIPC`
* The `cacheblend-plugin-pull` secret missing from the **workload** namespace — the injected init
  container `ImagePullBackOff`s (the secret must live where the pod runs, not in the operator namespace)
* **Mismatched** `cacheblend-plugin` and `vllm-openai` images — vLLM crashes at startup with an
  `ImportError`; pin a matched pair
* `webhook.enabled: false` or no cert-manager — the engine deploys but nothing injects the plugin