Skip to main content
TMO uses aesgcm to encrypt KV cache bytes at rest in the L2 external storage — Redis, object storage, or a filesystem — where that storage sits outside your trust boundary. You enable it on the engine CR and supply the master key as a Kubernetes Secret; the default hkdf provider derives a per-tenant key from cache_salt

​
Encryption Scope

​
Prerequisites

​
Step 1: Create the master-key Secret

Create the Secret yourself, in the engine’s namespace — the reference is same-namespace only, so an engine CR cannot be used to read Secrets from other namespaces. The data key must be named master. Only that key is projected into the pod; the operator mounts it read-only at /etc/lmcache/keys/master.

​
Step 2: Enable L2 serde encryption on the engine

Apply aesgcm KVCache encryption to whichever L2 adapter you have configured — resp (Redis) or raw (filesystem, object storage). Add a serde block alongside it:
my-values.yaml

​
Configuration reference

All fields live under engine.spec.l2Backend.serde. On key size: AES-128 is the right default. It is computationally unbreakable, and KV cache is short-lived and regenerable, so the “harvest now, decrypt later” argument for 256 barely applies. Choose 256 when a compliance mandate requires it — and remember the master key file must then be 32 bytes. Validation is enforced at kubectl apply / helm upgrade time:
  • exactly one serde type must be set (only aesgcm exists today)
  • masterKeySecretRef.name must be non-empty
  • the request is rejected if a raw adapter config already sets its own serde key — one would silently overwrite the other

​
Cost and overhead

  • Storage: a fixed 29 bytes per chunk (a 1-byte format version, a 12-byte IV, and a 16-byte GCM authentication tag). Ciphertext is the same length as plaintext — there is no padding.
  • Compute: AES-GCM on server AES-NI hardware runs at roughly 4–8 GB/s per core for AES-128, and it runs on the L1↔L2 path — not the inference hot path. It does not touch the L0↔L1 CUDA-IPC transfer, so in practice the cost hides behind L2 (e.g. S3) latency.
Encryption composes after quantization, so if you need to reduce L2 size, quantization is the lever — compression is not a useful alternative here (KV bytes are high-entropy and barely shrink).

​
Failure behavior

A decryption failure — tampered bytes, or the wrong key — fails the integrity check on the GCM tag. That surfaces as a load failure, which LMCache treats as a cache miss: the entry is re-fetched or recomputed. There is no silent corruption, and no served-from-cache garbage.
This makes key rotation destructive to existing cache. Changing the master key means every L2 entry written under the old key fails its integrity check and is treated as a miss, so the L2 tier effectively starts cold. Plan rotation for a window where a cold L2 is acceptable.

​
Verification

After the engine pods restart:
Then confirm the bytes at rest are actually opaque — the real test of the feature. Drive some traffic, let entries spill to L2, and inspect the stored objects directly (for the fs adapter, read a file under base_path; for Redis, GET a cache key). You should see high-entropy bytes, not recognizable prompt content.
If you see plaintext, the most likely cause is an operator image at v0.5.2 or older: the field was accepted by the CRD schema but never reconciled. Check the operator image tag.
  • Enable it when L2 storage is outside your trust boundary; skip it when L2 is as trusted as the GPU nodes themselves (it buys nothing there, since L1/L0 are plaintext regardless).
  • Keep the default aesBits: 128 unless a compliance mandate says otherwise.
  • Store the master key in a real secret manager and sync it into the cluster (External Secrets, Vault, SOPS) rather than committing generated key files.
  • Treat the master key as fleet-wide — it is not a per-tenant boundary. For tenant-vs-tenant isolation, give each tenant its own engine.
  • Do not rely on encryption to hide who cached how much — that is metadata, and it stays visible.