> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# KVCache Encryption

> Encrypt KV cache bytes at rest in the L2 tier with aesgcm, using a master key you provide as a Kubernetes Secret.

TMO uses **`aesgcm`** to encrypt KV cache bytes at rest in the **L2 external storage** — Redis, object storage, or
a filesystem — where that storage sits outside your trust boundary. You enable it on
the engine CR and supply the master key as a Kubernetes `Secret`; the default `hkdf` provider derives
a per-tenant key from  [`cache_salt`](/operator/management/multi-tenancy)

## Encryption Scope

| Tier                                       | Contents               | Encrypted? |
| ------------------------------------------ | ---------------------- | ---------- |
| **L2** (Redis / object store / filesystem) | Durable KV cache bytes | **Yes**    |
| **L1** (host RAM)                          | Hot KV cache           | No         |
| **L0** (GPU HBM)                           | Active KV cache        | No         |

## Prerequisites

| Component                                     | Minimum version |
| --------------------------------------------- | --------------- |
| Tensormesh Operator helm chart                | `0.5.3`         |
| LMCache Operator (`lmcache/lmcache-operator`) | `v0.5.3`        |
| LMCache vLLM (`lmcache/vllm-openai`)          | `v0.5.3`        |

## Step 1: Create the master-key Secret

Create the `Secret` yourself, in the **engine's namespace** — the reference is same-namespace only, so an engine CR cannot be used to read Secrets
from other namespaces.

The data key **must** be named `master`. Only that key is projected into the pod; the operator
mounts it read-only at `/etc/lmcache/keys/master`.

```bash theme={null}
# 16 bytes for AES-128 (default), 32 bytes for AES-256
head -c 16 /dev/urandom > master.key

kubectl -n tensormesh-operator create secret generic lmcache-l2-master-key \
  --from-file=master=master.key
```

## Step 2: Enable L2 serde encryption on the engine

Apply aesgcm KVCache encryption to whichever L2 adapter you have configured — `resp` (Redis) or `raw` (filesystem,
object storage). Add a `serde` block alongside it:

```yaml my-values.yaml theme={null}
operator:
  image:
    repository: lmcache/lmcache-operator
    tag: v0.5.3

engine:
  enabled: true
  spec:
    l1:
      sizeGB: 60
    l2Backend:
      # Whichever adapter you already use — the serde wraps either one.
      raw:
        type: fs
        config:
          base_path: /data/lmcache/l2
      serde:
        aesgcm:
          masterKeySecretRef:
            name: lmcache-l2-master-key   # must exist in the engine's namespace
```

```bash theme={null}
helm upgrade --install tensormesh-operator tensormesh/tensormesh-operator \
  -n tensormesh-operator --create-namespace \
  -f my-values.yaml
```

## Configuration reference

All fields live under `engine.spec.l2Backend.serde`.

| Field                            | Default | Description                                                                                        |
| -------------------------------- | ------- | -------------------------------------------------------------------------------------------------- |
| `aesgcm.masterKeySecretRef.name` | —       | **Required.** Secret in the engine's namespace holding the master key under the `master` data key. |
| `aesgcm.keyProvider`             | `hkdf`  | How per-`cache_salt` keys are derived. Only `hkdf` (HKDF-SHA256) is implemented.                   |
| `aesgcm.aesBits`                 | `128`   | AES key size: `128` or `256`.                                                                      |

On key size: **AES-128 is the right default.** It is computationally unbreakable, and KV cache is
short-lived and regenerable, so the "harvest now, decrypt later" argument for 256 barely applies.
Choose `256` when a compliance mandate requires it — and remember the master key file must then be
32 bytes.

Validation is enforced at `kubectl apply` / `helm upgrade` time:

* exactly one serde type must be set (only `aesgcm` exists today)
* `masterKeySecretRef.name` must be non-empty
* the request is **rejected** if a `raw` adapter config already sets its own `serde` key — one would
  silently overwrite the other

## Cost and overhead

* **Storage:** a fixed **29 bytes per chunk** (a 1-byte format version, a 12-byte IV, and a 16-byte
  GCM authentication tag). Ciphertext is the same length as plaintext — there is no padding.
* **Compute:** AES-GCM on server AES-NI hardware runs at roughly 4–8 GB/s per core for AES-128, and
  it runs on the **L1↔L2 path — not the inference hot path**. It does not touch the L0↔L1 CUDA-IPC
  transfer, so in practice the cost hides behind L2 (e.g. S3) latency.

Encryption composes *after* quantization, so if you need to reduce L2 size, quantization is the
lever — compression is not a useful alternative here (KV bytes are high-entropy and barely shrink).

## Failure behavior

A decryption failure — tampered bytes, or the wrong key — fails the **integrity check** on the GCM
tag. That surfaces as a load failure, which LMCache treats as a **cache miss**: the entry is
re-fetched or recomputed. There is no silent corruption, and no served-from-cache garbage.

<Warning>
  This makes key rotation destructive to existing cache. Changing the master key means every L2 entry
  written under the old key fails its integrity check and is treated as a miss, so the L2 tier
  effectively starts cold. Plan rotation for a window where a cold L2 is acceptable.
</Warning>

## Verification

After the engine pods restart:

```bash theme={null}
# 1. The Secret is mounted read-only at the expected path
kubectl -n tensormesh-operator exec ds/tensormesh-engine -- \
  ls -l /etc/lmcache/keys/master

# 2. The engine's --l2-adapter argument carries the serde config
kubectl -n tensormesh-operator get ds tensormesh-engine \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -A1 l2-adapter
```

Then confirm the bytes at rest are actually opaque — the real test of the feature. Drive some
traffic, let entries spill to L2, and inspect the stored objects directly (for the `fs` adapter, read
a file under `base_path`; for Redis, `GET` a cache key). You should see high-entropy bytes, not
recognizable prompt content.

<Note>
  If you see plaintext, the most likely cause is an operator image at v0.5.2 or older: the field was
  accepted by the CRD schema but never reconciled. Check the operator image tag.
</Note>

## Recommended practice

* Enable it when **L2 storage is outside your trust boundary**; skip it when L2 is as trusted as the
  GPU nodes themselves (it buys nothing there, since L1/L0 are plaintext regardless).
* Keep the default **`aesBits: 128`** unless a compliance mandate says otherwise.
* Store the master key in a real secret manager and sync it into the cluster (External Secrets,
  Vault, SOPS) rather than committing generated key files.
* Treat the master key as **fleet-wide** — it is not a per-tenant boundary. For tenant-vs-tenant
  isolation, give each tenant its own engine.
* Do not rely on encryption to hide **who cached how much** — that is metadata, and it stays visible.