> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics

> How LMCache engine metrics are exposed by the operator, which Prometheus metrics matter first, and where to go deeper.

The operator can expose LMCache multiprocess metrics from each engine pod. These are the same
LMCache server metrics documented upstream, but the operator is responsible for wiring the
Kubernetes pieces around them:

* the engine pod runs `lmcache server`
* the engine exposes a Prometheus `/metrics` endpoint
* the operator creates a metrics `Service`
* optionally, the operator also creates a `ServiceMonitor`

For customers, the right split is:

* use **this page** for operator-specific setup and the small set of metrics that matter first
* use the upstream [LMCache Observability docs](https://docs.lmcache.ai/production/observability/index.html) for the full metric catalog and deeper internals

## What the operator exposes

When Prometheus is enabled on the `LMCacheEngine`, the operator configures the engine with a
Prometheus port and creates a metrics `Service` for it.

At a high level:

* the engine exports LMCache MP metrics
* Prometheus can scrape them through the metrics `Service`
* if `serviceMonitor.enabled` is set and the Prometheus Operator CRD exists, the operator can also
  create a `ServiceMonitor`

Typical production shape:

```yaml theme={null}
engine:
  enabled: true
  spec:
    prometheus:
      enabled: true
      port: 9090
      serviceMonitor:
        enabled: true
        labels:
          release: kube-prometheus-stack
```

## Verify the metrics endpoint

Check that the metrics `Service` exists:

```bash theme={null}
kubectl get svc -n <operator-namespace>
```

Port-forward it locally:

```bash theme={null}
kubectl port-forward -n <operator-namespace> svc/<engine-name>-metrics 9090:9090
```

Then confirm the endpoint responds:

```bash theme={null}
curl -s http://localhost:9090/metrics | head
```

If you enabled `ServiceMonitor`, also confirm it exists:

```bash theme={null}
kubectl get servicemonitor -n <operator-namespace>
```

## Metrics to watch first

You do **not** need the full LMCache metric catalog to answer the first operational questions.
Start with these.

### 1. Lookup hit rate

These counters tell you how much of the requested prefix was served from LMCache.

* `lmcache_mp_lookup_requested_tokens_total`
* `lmcache_mp_lookup_hit_tokens_total`

PromQL:

```promql theme={null}
sum(rate(lmcache_mp_lookup_hit_tokens_total[5m]))
/ sum(rate(lmcache_mp_lookup_requested_tokens_total[5m]))
```

Use this to answer:

* is LMCache actually hitting?
* did a rollout or config change reduce reuse?
* is one model or tenant getting poor reuse?

### 2. L1 activity

These counters tell you whether the in-memory cache is being used and churned heavily.

* `lmcache_mp_l1_read_chunks_total`
* `lmcache_mp_l1_write_chunks_total`
* `lmcache_mp_l1_evicted_chunks_total`

Watch for:

* writes increasing but reads staying low -> poor reuse
* evictions rising too quickly -> L1 is likely undersized for the working set

### 3. L2 store and prefetch activity

If you use filesystem offloading or another L2 backend, these metrics tell you whether the engine is
actually using L2:

* `lmcache_mp_l2_store_completed_requests_total`
* `lmcache_mp_l2_load_completed_requests_total`
* `lmcache_mp_l2_prefetch_failure_chunks_total`

Use this to answer:

* is L2 receiving data?
* is warm traffic actually loading from L2?
* are prefetches failing because of misses or L1 pressure?

### 4. Retrieved chunk volume

This counter tells you how much data LMCache is actually loading back into the engine:

* `lmcache_mp_num_chunks_loaded_total`

If your hit rate claims to be improving but loaded chunk volume stays flat, your warm path may not be
working the way you expect.

### 5. Reuse-gap histograms

These histograms help you size storage and understand whether cached data is living long enough to be
reused:

* `lmcache_mp_real_reuse_gap_seconds`
* `lmcache_mp_real_reuse_gap_objects_chunks`

These are more advanced, but they are useful when you are deciding:

* whether L1 is too small
* whether L2 retention is worth the storage cost
* whether one tenant’s cache is staying hot enough to justify a larger quota

## Per-model and per-tenant slicing

Many LMCache MP metrics carry labels such as:

* `model_name`
* `cache_salt`
* `l2_name`

This is useful, but `cache_salt` can be high-cardinality. If you export these metrics into
Prometheus, be careful about exploding time-series counts by using one unique salt per request.

Operational rule:

* per-tenant `cache_salt` is reasonable
* per-request `cache_salt` is usually a bad idea for both reuse and observability

## Healthy patterns

Healthy LMCache behavior usually looks like this:

* lookup hit rate rises on repeated or warm traffic
* L1 reads increase on warm traffic
* L2 loads increase when warm traffic is served from offloaded cache
* eviction is present but not dominating all traffic

Concerning patterns:

* high L1 writes with almost no L1 reads
* constant eviction pressure
* no L2 loads even though you expect warm reuse from offloaded cache
* high prefetch failure counts

## When to go deeper

Use the upstream LMCache docs when you need:

* the full MP metric catalog
* event/log/tracing details
* exact metric semantics and cardinality notes
* advanced observability and health-monitor design

Start here:

* [LMCache Observability overview](https://docs.lmcache.ai/production/observability/index.html)
* [LMCache metrics reference](https://docs.lmcache.ai/production/observability/metrics.html)
* [LMCache MP observability details](https://docs.lmcache.ai/mp/observability.html)