Metrics

Every cache-engine pod exports Prometheus metrics. This page covers how the operator exposes them and the signals that show whether the cache is reducing recompute.

How metrics reach Prometheus

Metrics reach Prometheus one of two ways. The OpenTelemetry Collector path is the recommended setup. With observability.enabled, each engine pushes its metrics (and traces) to an OTel Collector, and Prometheus scrapes the Collector. Enabling it also stops the engine serving /metrics directly. See Observability for the full setup. The engine-endpoint path is an alternative when you don’t run the collector. Each engine pod serves /metrics directly, and the operator creates a metrics Service, plus a ServiceMonitor (when serviceMonitor.enabled is set) so Prometheus auto-discovers it:

engine:
  enabled: true
  spec:
    prometheus:
      enabled: true
      port: 9090
      serviceMonitor:
        enabled: true
        labels:
          release: kube-prometheus-stack   # must match your Prometheus' serviceMonitorSelector

Verify metrics are flowing

First, find which path this engine uses. The check is whether the engine is pushing OTLP:

kubectl get lmcacheengine <engine-name> -n <operator-namespace> -o yaml | grep -q otlp-endpoint \
  && echo "OTLP / collector path" || echo "engine-endpoint path"

OTLP / collector path
Engine-endpoint path

The engine pushes metrics to an OpenTelemetry Collector, so it exposes no scrapeable lmcache_mp_* endpoint itself. Read them from the Collector’s Prometheus exporter and confirm the pipeline in Observability → Verify telemetry is flowing, or just query Prometheus below.

The engine serves /metrics on its --prometheus-port (default 9090), behind the <engine-name>-metrics Service:

kubectl get endpoints <engine-name>-metrics -n <operator-namespace>   # one IP per engine pod
kubectl port-forward -n <operator-namespace> svc/<engine-name>-metrics 9090:9090
curl -s http://localhost:9090/metrics | grep lmcache_mp_ | head

connection refused means nothing is serving /metrics there; confirm prometheus.enabled is set and the engine isn’t on the OTLP path.

The fleet-wide source of truth is your Prometheus, the same data the Operator UI reads:

kubectl port-forward -n <prometheus-namespace> svc/<prometheus> 9091:9090
curl -s 'http://localhost:9091/api/v1/query?query=sum(rate(lmcache_mp_lookup_hit_tokens_total[5m]))/sum(rate(lmcache_mp_lookup_requested_tokens_total[5m]))' \
  | jq -r '.data.result[0].value[1]'

A batch of lmcache_mp_* lines (or a number from the query) confirms metrics are flowing. Empty means no traffic has hit the engine yet; send some inference through vLLM first.

Pull every metric

See the complete set straight from Prometheus (works on either path):

# every cache-engine metric name Prometheus holds
curl -s 'http://<prometheus>/api/v1/label/__name__/values' \
  | jq -r '.data[] | select(startswith("lmcache_mp_"))'

# the labels you can slice by (model, tenant, backend, node)
curl -s 'http://<prometheus>/api/v1/series?match[]=lmcache_mp_lookup_hit_tokens_total' \
  | jq '.data[0] | keys'

The complete metric reference below catalogs every one.

What to watch first

The signals to check first, in priority order.

1. Lookup hit rate

How much of each requested prefix was served from cache instead of recomputed on the GPU.

sum(rate(lmcache_mp_lookup_hit_tokens_total[5m]))
/ sum(rate(lmcache_mp_lookup_requested_tokens_total[5m]))

Metric	Meaning
`lmcache_mp_lookup_requested_tokens_total`	Tokens the engine was asked to look up
`lmcache_mp_lookup_hit_tokens_total`	Tokens found in cache

2. L1 activity

L1 is the host-DRAM tier. These counters show whether it’s serving reads (good) or just churning writes and evictions (undersized).

Metric	Meaning
`lmcache_mp_l1_read_chunks_total`	Chunks served from DRAM
`lmcache_mp_l1_write_chunks_total`	Chunks written into DRAM
`lmcache_mp_l1_evicted_chunks_total`	Chunks evicted under pressure

Reads rising on warm traffic → L1 is doing its job.
Writes high, reads low → poor reuse, or the hot set doesn’t fit — see CPU offloading sizing.
Evictions climbing fast → L1 is undersized for the working set.

3. L2 activity

Only relevant if you’ve configured external storage. These confirm data is landing in L2 and warm traffic is loading it back.

Metric	Meaning
`lmcache_mp_l2_store_completed_requests_total`	Data written to L2
`lmcache_mp_l2_load_completed_requests_total`	Data loaded back from L2
`lmcache_mp_l2_prefetch_failure_chunks_total`	Prefetches that failed (miss or L1 pressure)

4. Retrieved chunk volume

sum(rate(lmcache_mp_num_chunks_loaded_total[5m]))

How much data the engine loads back into service. If hit rate is rising but this stays flat, the warm path isn’t actually loading anything.

Slice by model and tenant

Many engine metrics carry labels you can group by:

model_name — per-model hit rate and volume
cache_salt — per-tenant isolation
l2_name — which L2 backend

sum by (model_name) (rate(lmcache_mp_lookup_hit_tokens_total[5m]))
/ sum by (model_name) (rate(lmcache_mp_lookup_requested_tokens_total[5m]))

Keep cache_salt coarse: one salt per tenant is fine, but one per request explodes your time-series count and hurts cache reuse.

Complete metric reference

Most metrics carry model_name, cache_salt, and l2_name labels for slicing. Counters (_total) need rate(...[5m]); gauges are instantaneous; histograms expose _bucket / _sum / _count. A metric only appears once it’s been recorded, so an idle engine shows only the always-on gauges; the activity counters appear after traffic hits the cache.

Lookup & hit rate

Metric	Type	Measures
`lmcache_mp_lookup_requested_tokens_total`	counter	Tokens the engine was asked to look up (hit-rate denominator)
`lmcache_mp_lookup_hit_tokens_total`	counter	Tokens found in cache (hit-rate numerator)

L1 — CPU offloading (host DRAM)

Metric	Type	Measures
`lmcache_mp_l1_usage_ratio`	gauge	L1 fill level, `0..1`
`lmcache_mp_l1_memory_usage_bytes`	gauge	Bytes currently held in L1
`lmcache_mp_l1_read_chunks_total`	counter	Chunks served from L1 (reuse activity)
`lmcache_mp_l1_write_chunks_total`	counter	Chunks written into L1 (ingest)
`lmcache_mp_l1_evicted_chunks_total`	counter	Chunks evicted from L1 under pressure
`lmcache_mp_l1_read_failure_chunks_total`	counter	L1 reads that failed
`lmcache_mp_l1_eviction_loop_ticks_total`	counter	Eviction-loop iterations (every cycle)
`lmcache_mp_l1_eviction_loop_triggered_total`	counter	Eviction-loop iterations where the policy actually ran

L2 — external storage

Metric	Type	Measures
`lmcache_mp_l2_usage_bytes`	gauge	Bytes currently held in L2, per `l2_name`
`lmcache_mp_l2_adapters`	gauge	L2 adapters on the store controller, by `state` (`active` / `draining`)
`lmcache_mp_num_inflight_l2_stores`	gauge	Store operations in flight to L2
`lmcache_mp_num_inflight_l2_loads`	gauge	Load operations in flight from L2
`lmcache_mp_inflight_load_memory_usage_bytes`	gauge	Memory held by in-flight L2 loads
`lmcache_mp_l2_store_submitted_requests_total`	counter	L2 store requests submitted
`lmcache_mp_l2_store_submitted_objects_chunks_total`	counter	Chunks submitted for L2 store
`lmcache_mp_l2_store_completed_requests_total`	counter	L2 store requests completed
`lmcache_mp_l2_store_completed_objects_chunks_total`	counter	Chunks written to L2
`lmcache_mp_l2_load_completed_requests_total`	counter	L2 load requests completed
`lmcache_mp_l2_prefetch_lookup_requests_total`	counter	L2 prefetch lookup requests submitted
`lmcache_mp_l2_prefetch_lookup_objects_chunks_total`	counter	Chunks submitted for L2 prefetch lookup
`lmcache_mp_l2_prefetch_hit_chunks_total`	counter	Prefix chunks found in an L2 lookup
`lmcache_mp_l2_prefetch_load_submitted_requests_total`	counter	L2 prefetch load requests submitted
`lmcache_mp_l2_prefetch_load_submitted_objects_chunks_total`	counter	Chunks submitted for L2 load
`lmcache_mp_l2_prefetch_load_completed_chunks_total`	counter	Chunks successfully loaded from L2
`lmcache_mp_l2_prefetch_failure_chunks_total`	counter	Prefetches that failed (`reason` = `l1_oom` / `not_found`)
`lmcache_mp_num_chunks_loaded_total`	counter	Chunks loaded back into service (the warm path)

Throughput

Histograms in GB/s: compute the live rate as rate(<name>_sum[5m]) / rate(<name>_count[5m]).

Metric	Direction
`lmcache_mp_l0_l1_store_throughput_GB_per_second`	GPU → CPU (store into L1)
`lmcache_mp_l0_l1_load_throughput_GB_per_second`	CPU → GPU (load from L1)
`lmcache_mp_l2_store_throughput_GB_per_second`	L1 → L2 (store to external)
`lmcache_mp_l2_load_throughput_GB_per_second`	L2 → L1 (load from external)

Prefetch & event bus (health)

Metric	Type	Measures
`lmcache_mp_active_prefetch_jobs`	gauge	Prefetch jobs currently running
`lmcache_mp_active_p2p_lookup_jobs`	gauge	Active peer-to-peer lookup jobs (with P2P enabled)
`lmcache_mp_event_bus_queue_depth`	gauge	Depth of the internal event queue
`lmcache_mp_event_bus_drain_lag_seconds`	gauge	Seconds since the oldest queued event; rising = the drain thread is falling behind
`lmcache_mp_event_bus_dropped_events_total`	counter	Events dropped (queue overflow, a saturation signal)

Reuse gap

How long cached data lives before it’s reused, the input to sizing decisions.

Metric	Type	Measures
`lmcache_mp_real_reuse_gap_seconds`	histogram	Time between storing a chunk and reusing it
`lmcache_mp_real_reuse_gap_objects`	histogram	Reuse-gap distribution across cached objects

Lifecycle histograms

Per-tier block/chunk timing: how long cached data lives, sits idle, and waits between reuses. Newer engine builds expose more of these. All are histograms (_bucket / _sum / _count).

Metric	Tier	Measures
`lmcache_mp_l0_block_lifetime_seconds`	GPU (L0)	Block lifetime, allocation → eviction
`lmcache_mp_l0_block_idle_before_evict_seconds`	GPU (L0)	Idle time before a block is evicted
`lmcache_mp_l0_block_reuse_gap_seconds`	GPU (L0)	Gap between consecutive block accesses
`lmcache_mp_l1_chunk_lifetime_seconds`	L1 (DRAM)	Chunk lifetime, allocation → eviction
`lmcache_mp_l1_chunk_idle_before_evict_seconds`	L1 (DRAM)	Idle time before a chunk is evicted
`lmcache_mp_l1_chunk_reuse_gap_seconds`	L1 (DRAM)	Gap between consecutive touches of a chunk
`lmcache_mp_l1_chunk_evict_reuse_gap_seconds`	L1 (DRAM)	Time from chunk eviction to next reuse

vLLM exports its own vllm:* metrics on a separate port (TTFT, running/waiting requests, HBM KV usage). Pair those with the cache-engine metrics above for the full request-to-cache picture.

When to go deeper

The catalog above is complete for operating the cache. For low-level internals (exact histogram-bucket boundaries, precise semantics, and cardinality notes), see the upstream multiprocess-engine reference: MP observability.

Next steps

Operator UI

An interface that visualizes these metrics: fleet, hit rate, capacity, health.

Observability stack

Ship metrics and traces through an OpenTelemetry Collector to Prometheus and Tempo.

CPU offloading

Tune the L1 tier the L1 metrics above describe.

Overview

Installation

Features

Operations

Integrations

Operator UI

Reference

How metrics reach Prometheus

Verify metrics are flowing

Pull every metric

What to watch first

1. Lookup hit rate

2. L1 activity

3. L2 activity

4. Retrieved chunk volume

Slice by model and tenant

Complete metric reference

Lookup & hit rate

L1 — CPU offloading (host DRAM)

L2 — external storage

Throughput

Prefetch & event bus (health)

Reuse gap

Lifecycle histograms

When to go deeper

Next steps

Operator UI

Observability stack

CPU offloading

​How metrics reach Prometheus

​Verify metrics are flowing

​Pull every metric

​What to watch first

​1. Lookup hit rate

​2. L1 activity

​3. L2 activity

​4. Retrieved chunk volume

​Slice by model and tenant

​Complete metric reference

​Lookup & hit rate

​L1 — CPU offloading (host DRAM)

​L2 — external storage

​Throughput

​Prefetch & event bus (health)

​Reuse gap

​Lifecycle histograms

​When to go deeper

​Next steps

Operator UI

Observability stack

CPU offloading

How metrics reach Prometheus

Verify metrics are flowing

Pull every metric

What to watch first

1. Lookup hit rate

2. L1 activity

3. L2 activity

4. Retrieved chunk volume

Slice by model and tenant

Complete metric reference

Lookup & hit rate

L1 — CPU offloading (host DRAM)

L2 — external storage

Throughput

Prefetch & event bus (health)

Reuse gap

Lifecycle histograms

When to go deeper

Next steps