> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Tenancy

> Isolate cache reuse and capacity across tenants, users, and environments with cache_salt and per-tenant quotas.

When several tenants, users, or environments share one LMCache deployment, you usually want two
kinds of isolation:

* **Reuse isolation** — control *who is allowed to share cached prefixes*. A prompt cached for one
  tenant must not be reused by another. This is what `cache_salt` provides.
* **Capacity isolation** — control *how much cache each tenant may consume*, so one noisy tenant
  cannot evict everyone else's entries. This is what per-tenant **quotas** and the **IsolatedLRU**
  eviction policy provide.

This page covers both, and which layer owns each setting.

## Who configures what

Multi-tenancy spans three layers. Putting a setting in the wrong layer is the most common mistake.

| Concern                                        | Owner                            | Where it is set                                             |
| ---------------------------------------------- | -------------------------------- | ----------------------------------------------------------- |
| Reuse boundary per request (`cache_salt`)      | **API caller** (vLLM / your app) | The request body sent to vLLM — *not* an operator field     |
| Engine deployment, cache size, eviction config | **Operator (TMO)**               | `LMCacheEngine` CRD / Helm values                           |
| Per-tenant storage budgets (quotas)            | **Runtime admin**                | LMCache HTTP cache-admin API on the engine (or coordinator) |
| Fleet-wide per-tenant usage tracking           | **Operator (TMO)**               | `LMCacheEngine.spec.coordinator` + an `LMCacheCoordinator`  |

Note in particular: the operator deploys the LMCache engine, but it does **not** set `cache_salt`.
There is no `cache_salt` field in any CRD or Helm value. The salt is attached to each inference
request by the caller.

## Reuse isolation with `cache_salt`

`cache_salt` is a **per-request isolation key**. It tells LMCache which requests are allowed to
share cached KV and which ones must stay separate.

The behavior is simple:

* same model + same prompt prefix + same `cache_salt` → cache reuse is allowed
* same model + same prompt prefix + different `cache_salt` → cache reuse is blocked
* no `cache_salt` → the request uses the default unsalted cache domain

Internally, `cache_salt` is part of the cache key identity (the `ObjectKey`), alongside the chunk
hash, model name, and worker rank. Two requests whose content hashes identically but whose salts
differ resolve to **distinct** keys, so they never collide in L1 or L2. Unsalted traffic uses the
same on-disk key format as before salts existed, so introducing a salt for some tenants does not
invalidate existing unsalted entries.

### How to set it

`cache_salt` is a first-class field on OpenAI-compatible requests to vLLM. Include it directly in
the request body.

```bash theme={null}
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "Summarize the quarterly report."}
    ],
    "cache_salt": "tenant-acme",
    "max_tokens": 128,
    "temperature": 0
  }'
```

For plain completions:

```bash theme={null}
curl -s http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "prompt": "Summarize the quarterly report.",
    "cache_salt": "tenant-acme",
    "max_tokens": 128,
    "temperature": 0
  }'
```

vLLM propagates the salt through its LMCache connector to both the scheduler (lookup) and worker
(store/retrieve) paths, so it consistently scopes reads and writes for that request.

### How to choose a value

Pick a value that is:

* stable across requests that should share cache
* different across requests that must stay isolated
* low-cardinality enough that it does not explode your metrics labels

Good choices: `tenant-acme`, `customer-123`, `prod`, `workspace-42`.

Bad choices: a random UUID per request, a timestamp, or any value that changes on every call. If
the value changes on every request, you destroy reuse and effectively turn every request into a
cache miss.

### Constraints

`cache_salt` is validated by LMCache. It:

* must not contain `@`
* must not contain `/`
* must not contain `\`
* must not contain a NUL byte
* must be at most **128** characters

A salt that violates these is rejected with a `ValueError`. Keep it simple: short ASCII
identifiers such as `tenant-acme` or `prod` are the safest choice.

<Note>
  The empty string (`""`) is the default, unsalted domain. In the cache-admin HTTP API described
  below, the empty salt cannot appear in a URL path, so it is represented by the reserved sentinel
  `_default` (for example `GET /quota/_default`).
</Note>

## Capacity isolation: per-tenant quotas

Reuse isolation keeps tenants from *reading* each other's data. It does not stop one tenant from
*filling* the cache. For that, LMCache supports per-tenant quotas backed by the **IsolatedLRU**
eviction policy.

How it works:

* **IsolatedLRU** keeps a separate LRU ordering per `cache_salt` instead of one global list, so
  eviction victims for one tenant are chosen only from that tenant's entries.
* A **quota** assigns each `cache_salt` a storage budget in bytes. LMCache tracks usage per salt
  and evicts a salt's oldest entries once it crosses its budget.
* Quotas use **allowlist semantics**: a `cache_salt` with no registered quota has an effective
  limit of **0 bytes**. Its stores are accepted on the hot path but evicted on the next eviction
  cycle (\~1s). Only salts with an explicit quota retain cached data. Plan to register a quota for
  every tenant you want cached.

The eviction loop is governed by these engine-side fields (LMCache eviction config):

| Field               | Default | Meaning                                                   |
| ------------------- | ------- | --------------------------------------------------------- |
| `eviction_policy`   | `LRU`   | `LRU`, `IsolatedLRU`, or `noop`                           |
| `trigger_watermark` | `0.8`   | Per-salt usage fraction (of quota) that triggers eviction |
| `eviction_ratio`    | `0.2`   | Fraction of over-quota bytes evicted per cycle            |

<Warning>
  **Operator support is currently partial.** The `LMCacheEngine` CRD's structured `eviction.policy`
  field only accepts `LRU` today (`eviction.triggerWatermark` and `eviction.evictionRatio` are
  exposed). To run `IsolatedLRU`, pass it to the engine through the CRD escape hatches —
  `spec.extraArgs` (extra CLI flags appended last) or `spec.env` — rather than `eviction.policy`.
  Per-tenant quotas themselves are always set at runtime through the HTTP API below, not through the
  CRD.
</Warning>

### Managing quotas at runtime

The engine exposes an HTTP cache-admin frontend on `spec.httpPort` (default **8080**). Quotas are
created, inspected, and removed there:

| Method   | Path                  | Body                    | Purpose                                      |
| -------- | --------------------- | ----------------------- | -------------------------------------------- |
| `PUT`    | `/quota/{cache_salt}` | `{"limit_gb": <float>}` | Create or update a tenant's budget           |
| `GET`    | `/quota/{cache_salt}` | —                       | Read a tenant's limit and current usage      |
| `DELETE` | `/quota/{cache_salt}` | —                       | Remove a budget (entries evicted next cycle) |
| `GET`    | `/quota`              | —                       | List every registered quota and its usage    |

```bash theme={null}
# Give tenant-acme a 5 GB budget (engine HTTP frontend on httpPort 8080)
curl -s -X PUT http://<engine-host>:8080/quota/tenant-acme \
  -H 'Content-Type: application/json' \
  -d '{"limit_gb": 5}'

# Inspect it
curl -s http://<engine-host>:8080/quota/tenant-acme
```

Use the `_default` sentinel to manage the unsalted domain: `PUT /quota/_default`.

### Fleet-wide quotas via the coordinator

The HTTP API above is per-engine. To track and bound a tenant's usage across an entire fleet of
engines, register the engines with an `LMCacheCoordinator` and enable event reporting on the
engine's coordinator block:

```yaml theme={null}
apiVersion: lmcache.ai/v1alpha1
kind: LMCacheEngine
spec:
  coordinator:
    ref:
      name: my-coordinator      # an LMCacheCoordinator in the same namespace
    l2EventReporting: true      # report L2 store/lookup events for fleet-wide tracking
    l2EventFlushInterval: 1     # seconds between event flushes
```

With reporting on, each engine forwards per-salt usage events to the coordinator, which aggregates
them and runs fleet-wide per-salt quota enforcement and eviction. The coordinator side exposes the
matching admin endpoints (`PUT`/`DELETE /l2/quota/{cache_salt}`, `GET /l2/status[/{cache_salt}]`)
and its own sweep controls (`evictionCheckInterval`, default 5s; `evictionRatio`, default 0.2).

## Deployment-level isolation

`cache_salt` and quotas isolate tenants *within* one shared cache. For harder isolation, give each
tenant its own engine:

* **Per-namespace engines** — run a separate `LMCacheEngine` CR per tenant namespace. The chart's
  multi-tenant mode (`operator.enabled=true`, `engine.enabled=false`) installs only the operator;
  each tenant then ships their own engine CR. Caches are not shared across separate engines at all.
* **Node placement** — `spec.nodeSelector`, `spec.affinity`, and `spec.tolerations` pin a tenant's
  engine to dedicated GPU nodes.
* **RBAC / network** — `spec.serviceAccountName`, `spec.podLabels`, and `spec.podAnnotations` let
  you attach per-tenant RBAC and NetworkPolicy.

Choose deployment-level isolation when tenants must not share GPUs or memory at all; choose
`cache_salt` + quotas when you want the efficiency of one shared pool with logical separation.

## Observability

LMCache tags several metrics with `cache_salt`, which is useful for per-tenant hit-rate analysis:

* `lmcache_mp.lookup_requested` — tokens submitted for lookup (labels: `model_name`, `cache_salt`)
* `lmcache_mp.lookup_hit` — tokens served from cache (labels: `model_name`, `cache_salt`)
* `lmcache_mp.real_reuse_gap` — seconds between a chunk's last write and next read (label:
  `cache_salt`)
* `lmcache_mp.real_reuse_gap_objects` — the same gap measured in chunks (label: `cache_salt`)

Divide `lookup_hit` by `lookup_requested`, grouped by `cache_salt`, for a per-tenant hit rate.

<Warning>
  Treat `cache_salt` as a potentially high-cardinality dimension. Per-tenant or per-environment salts
  are fine; per-request salts will blow up your metrics cardinality (and destroy reuse).
</Warning>

## Verification

`cache_salt` reuse behavior is straightforward to check by hand:

1. Send a long prompt with `cache_salt: "tenant-a"`.
2. Send the **same** prompt again with `cache_salt: "tenant-a"` → reuse is allowed.
3. Send the same prompt with `cache_salt: "tenant-b"` → it must **not** reuse `tenant-a`'s entries,
   even though the prompt is identical.

To confirm a hit at the engine level, inspect the engine pod logs — the engine logs KV cache
registration and serving activity that the end-to-end test harness keys on to assert cache
behavior. Per-tenant hit rates are also visible through the metrics above.

<Note>
  The current end-to-end test suite exercises basic cache hit/miss and L1/L2 storage, but does not
  yet include automated coverage for `cache_salt` isolation, per-tenant quotas, or `IsolatedLRU`.
  Validate these flows in a staging deployment before relying on them in production.
</Note>

## Recommended practice

* Use **no `cache_salt`** if all traffic is inside one trust boundary.
* Use **one stable salt per tenant or environment** when traffic must be isolated.
* Avoid **per-request salts** unless you explicitly want to disable sharing.
* If you need capacity fairness, enable **IsolatedLRU** and register a **quota for every tenant**
  you want cached (remember: unregistered salts get 0 bytes).
* For strict isolation, give each tenant its own **engine per namespace** instead of a shared pool.