Getting Started
Preflight your cluster and pick an install method.
End-to-End Example
Install → run inference → benchmark the cache win.
Benefits
Faster prefills
Cached prefixes skip recomputation, so warm requests return their first token sooner.
More GPU headroom
KV cache lives in CPU memory and on disk, freeing HBM for active batches.
Reuse that persists
Cache outlives a single request — across pods, restarts, and optionally across nodes.
Kubernetes-native
One operator, standard CRs, Helm/GitOps install, and metrics through Prometheus.
How caching tiers fit together
Each tier is larger and cheaper than the one above it; the engine promotes hot data upward and spills cold data downward:
Start with CPU offloading — it’s on by default. Add
external storage when your working set outgrows RAM or
must survive restarts.
What gets deployed
Operator
A controller-manager Deployment that watches the Tensormesh CRs cluster-wide and reconciles
them into running cache infrastructure.
Cache engine
A DaemonSet — one cache-engine pod per GPU node — that stores and serves KV cache to the
vLLM pods on that node.
Coordinator
A fleet-wide service the engines register with, enabling cross-node cache awareness.
Observability
An optional OpenTelemetry stack that ships cache metrics and traces to Prometheus and Tempo.
Explore the docs
Install with Helm
The primary install path, plus upgrades and rollbacks.
CPU offloading
The default host-DRAM cache tier.
Metrics
The signals that show whether the cache is reducing recompute.
Operator UI
A console for monitoring your cache fleet, thresholds, and alerts.