Skip to main content
The operator can expose LMCache multiprocess metrics from each engine pod. These are the same LMCache server metrics documented upstream, but the operator is responsible for wiring the Kubernetes pieces around them:
  • the engine pod runs lmcache server
  • the engine exposes a Prometheus /metrics endpoint
  • the operator creates a metrics Service
  • optionally, the operator also creates a ServiceMonitor
For customers, the right split is:
  • use this page for operator-specific setup and the small set of metrics that matter first
  • use the upstream LMCache Observability docs for the full metric catalog and deeper internals

​
What the operator exposes

When Prometheus is enabled on the LMCacheEngine, the operator configures the engine with a Prometheus port and creates a metrics Service for it. At a high level:
  • the engine exports LMCache MP metrics
  • Prometheus can scrape them through the metrics Service
  • if serviceMonitor.enabled is set and the Prometheus Operator CRD exists, the operator can also create a ServiceMonitor
Typical production shape: 
engine:
  enabled: true
  spec:
    prometheus:
      enabled: true
      port: 9090
      serviceMonitor:
        enabled: true
        labels:
          release: kube-prometheus-stack

​
Verify the metrics endpoint

Check that the metrics Service exists: 
kubectl get svc -n <operator-namespace>
Port-forward it locally: 
kubectl port-forward -n <operator-namespace> svc/<engine-name>-metrics 9090:9090
Then confirm the endpoint responds: 
curl -s http://localhost:9090/metrics | head
If you enabled ServiceMonitor, also confirm it exists: 
kubectl get servicemonitor -n <operator-namespace>

​
Metrics to watch first

You do not need the full LMCache metric catalog to answer the first operational questions. Start with these.

​
1. Lookup hit rate

These counters tell you how much of the requested prefix was served from LMCache.
  • lmcache_mp_lookup_requested_tokens_total
  • lmcache_mp_lookup_hit_tokens_total
PromQL: 
sum(rate(lmcache_mp_lookup_hit_tokens_total[5m]))
/ sum(rate(lmcache_mp_lookup_requested_tokens_total[5m]))
Use this to answer:
  • is LMCache actually hitting?
  • did a rollout or config change reduce reuse?
  • is one model or tenant getting poor reuse?

​
2. L1 activity

These counters tell you whether the in-memory cache is being used and churned heavily.
  • lmcache_mp_l1_read_chunks_total
  • lmcache_mp_l1_write_chunks_total
  • lmcache_mp_l1_evicted_chunks_total
Watch for:
  • writes increasing but reads staying low -> poor reuse
  • evictions rising too quickly -> L1 is likely undersized for the working set

​
3. L2 store and prefetch activity

If you use filesystem offloading or another L2 backend, these metrics tell you whether the engine is actually using L2:
  • lmcache_mp_l2_store_completed_requests_total
  • lmcache_mp_l2_load_completed_requests_total
  • lmcache_mp_l2_prefetch_failure_chunks_total
Use this to answer:
  • is L2 receiving data?
  • is warm traffic actually loading from L2?
  • are prefetches failing because of misses or L1 pressure?

​
4. Retrieved chunk volume

This counter tells you how much data LMCache is actually loading back into the engine:
  • lmcache_mp_num_chunks_loaded_total
If your hit rate claims to be improving but loaded chunk volume stays flat, your warm path may not be working the way you expect.

​
5. Reuse-gap histograms

These histograms help you size storage and understand whether cached data is living long enough to be reused:
  • lmcache_mp_real_reuse_gap_seconds
  • lmcache_mp_real_reuse_gap_objects_chunks
These are more advanced, but they are useful when you are deciding:
  • whether L1 is too small
  • whether L2 retention is worth the storage cost
  • whether one tenant’s cache is staying hot enough to justify a larger quota

​
Per-model and per-tenant slicing

Many LMCache MP metrics carry labels such as:
  • model_name
  • cache_salt
  • l2_name
This is useful, but cache_salt can be high-cardinality. If you export these metrics into Prometheus, be careful about exploding time-series counts by using one unique salt per request. Operational rule:
  • per-tenant cache_salt is reasonable
  • per-request cache_salt is usually a bad idea for both reuse and observability

​
Healthy patterns

Healthy LMCache behavior usually looks like this:
  • lookup hit rate rises on repeated or warm traffic
  • L1 reads increase on warm traffic
  • L2 loads increase when warm traffic is served from offloaded cache
  • eviction is present but not dominating all traffic
Concerning patterns:
  • high L1 writes with almost no L1 reads
  • constant eviction pressure
  • no L2 loads even though you expect warm reuse from offloaded cache
  • high prefetch failure counts

​
When to go deeper

Use the upstream LMCache docs when you need:
  • the full MP metric catalog
  • event/log/tracing details
  • exact metric semantics and cardinality notes
  • advanced observability and health-monitor design
Start here: