Multi-Tenancy

• KVCache isolation — control who is allowed to share cached prefixes. A prompt cached for one tenant must not be reused by another. This is what cache_salt provides.
• Capacity isolation — control how much cache each tenant may consume, so one noisy tenant cannot evict everyone else’s entries. This is what per-tenant quotas and the IsolatedLRU eviction policy provide.

​

Who configures what

​

KVCache isolation with cache_salt

• same model + same cache_salt → kv cache access is allowed
• same model + different cache_salt → kv cache access is blocked

​

How to set it

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "Summarize the quarterly report."}
    ],
    "cache_salt": "tenant-acme",
    "max_tokens": 128,
    "temperature": 0
  }'

​

How to choose a value

• stable across requests that should share cache
• different across requests that must stay isolated
• low-cardinality enough that it does not explode your metrics labels

​

Constraints

• must not contain @
• must not contain /
• must not contain \
• must not contain a NUL byte
• must be at most 128 characters

​

Capacity isolation: per-tenant quotas

• IsolatedLRU keeps a separate LRU ordering per cache_salt instead of one global list, so eviction victims for one tenant are chosen only from that tenant’s entries.
• A quota assigns each cache_salt a storage budget in bytes. LMCache tracks usage per salt and evicts a salt’s oldest entries once it crosses its budget.
• Quotas use allowlist semantics: a cache_salt with no registered quota has an effective limit of 0 bytes. Its stores are accepted on the hot path but evicted on the next eviction cycle (~1s). Only salts with an explicit quota retain cached data. Plan to register a quota for every tenant you want cached.

​

Managing quotas at runtime

# Give tenant-acme a 5 GB budget (engine HTTP frontend on httpPort 8080)
curl -s -X PUT http://<engine-host>:8080/quota/tenant-acme \
  -H 'Content-Type: application/json' \
  -d '{"limit_gb": 5}'

# Inspect it
curl -s http://<engine-host>:8080/quota/tenant-acme

​

Fleet-wide quotas via the coordinator

apiVersion: lmcache.ai/v1alpha1
kind: LMCacheEngine
spec:
  coordinator:
    ref:
      name: my-coordinator      # an LMCacheCoordinator in the same namespace
    l2EventReporting: true      # report L2 store/lookup events for fleet-wide tracking
    l2EventFlushInterval: 1     # seconds between event flushes

​

Deployment-level isolation

• Per-namespace engines — run a separate LMCacheEngine CR per tenant namespace. The chart’s multi-tenant mode (operator.enabled=true, engine.enabled=false) installs only the operator; each tenant then ships their own engine CR. Caches are not shared across separate engines at all.
• Node placement — spec.nodeSelector, spec.affinity, and spec.tolerations pin a tenant’s engine to dedicated GPU nodes.
• RBAC / network — spec.serviceAccountName, spec.podLabels, and spec.podAnnotations let you attach per-tenant RBAC and NetworkPolicy.

​

Observability

• lmcache_mp.lookup_requested — tokens submitted for lookup (labels: model_name, cache_salt)
• lmcache_mp.lookup_hit — tokens served from cache (labels: model_name, cache_salt)
• lmcache_mp.real_reuse_gap — seconds between a chunk’s last write and next read (label: cache_salt)
• lmcache_mp.real_reuse_gap_objects — the same gap measured in chunks (label: cache_salt)

​

Verification

1. Send a long prompt with cache_salt: "tenant-a".
2. Send the same prompt again with cache_salt: "tenant-a" → reuse is allowed.
3. Send the same prompt with cache_salt: "tenant-b" → it must not reuse tenant-a’s entries, even though the prompt is identical.

​

Recommended practice

• Use no cache_salt if all traffic is inside one trust boundary.
• Use one stable salt per tenant or environment when traffic must be isolated.
• Avoid per-request salts unless you explicitly want to disable sharing.
• If you need capacity fairness, enable IsolatedLRU and register a quota for every tenant you want cached (remember: unregistered salts get 0 bytes).
• For strict isolation, give each tenant its own engine per namespace instead of a shared pool.