> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensormesh.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CLI

> tmo-cli — the Tensormesh operator/admin CLI for observing and operating LMCacheEngine fleets on Kubernetes.

`tmo-cli` is the Tensormesh **administrator CLI**: a `kubectl`-familiar tool for
platform/SRE engineers to observe and operate **LMCacheEngine** resources (the
KV-cache controllers) and their per-GPU-node server pods.

It unifies two surfaces — the Kubernetes CRD (`lmcacheengines.lmcache.lmcache.ai`)
for engine lifecycle and status, and each server pod's HTTP control API (reached
through the Kubernetes API-server proxy) for cache state, quotas, and diagnostics.

<Note>
  `tmo-cli` is the **admin/SRE** CLI talking directly to your Kubernetes cluster
  (the data plane).
</Note>

## Install

`tmo-cli` is distributed privately as part of the gated Tensormesh product — not through
public Homebrew or krew. It installs from the **Tensormesh container registry** (GHCR)
using the same **access token** as the operator chart and images: a token scoped to
artifact pull only, which grants **no access to Tensormesh source code**. If you don't
have one, request it from the Tensormesh team — it's the same token used in the
[operator install guide](/operator/installation/helm).

### One-line install (recommended)

```bash theme={null}
curl -sSfL https://operator.tensormesh.ai/install-cli.sh | TMO_TOKEN=<TOKEN_FROM_TENSORMESH> sh
```

The script detects your OS and architecture, pulls the matching binary from GHCR, verifies
its SHA-256 against the registry digest, and installs it to `/usr/local/bin` (falling back
to `~/.local/bin` if that isn't writable — no `sudo`). To pin a version or change the
target, pass flags after `-s --`:

```bash theme={null}
curl -sSfL https://operator.tensormesh.ai/install-cli.sh | sh -s -- \
  --token <TOKEN_FROM_TENSORMESH> --version <version> --install-dir ~/bin
```

### Manual install (air-gapped / CI)

Prefer to pull the artifact yourself? Authenticate to GHCR with your token and use
[`oras`](https://oras.land):

```bash theme={null}
echo '<TOKEN_FROM_TENSORMESH>' | oras login ghcr.io -u tensormesh --password-stdin

oras pull ghcr.io/tensormesh-production/cli/tmo-cli:<version>   # or :latest
tar -xzf tmo-cli_*_linux_amd64.tar.gz tmo-cli                   # match your OS/arch
install -m 0755 tmo-cli ~/.local/bin/tmo-cli
```

### Verify

```bash theme={null}
tmo-cli version
tmo-cli diag preflight       # confirm your kube RBAC is sufficient
```

## How it connects

`tmo-cli` uses your existing **kubeconfig** — if `kubectl` works, `tmo-cli` works.
Resolution precedence: `--kubeconfig` → `$KUBECONFIG` → the `tmo-cli` config
default → `~/.kube/config`.

```bash theme={null}
tmo-cli config use-context prod-us-east
tmo-cli config set-default --output table
tmo-cli config view
```

HTTP commands (`node`, `cache`, `diag`, `quota`) reach engine pods through the
Kubernetes **API-server proxy** by default — no extra network path, and RBAC on
`pods/proxy` gates them. Pass `--direct` to reach pod host IPs directly when
running inside the cluster.

## Global flags

| Flag                                     | Meaning                                                              |
| ---------------------------------------- | -------------------------------------------------------------------- |
| `-n, --namespace`                        | namespace (default `default`)                                        |
| `-A, --all-namespaces`                   | across all namespaces (list / HTTP commands)                         |
| `-e, --engine`                           | restrict HTTP commands to one engine                                 |
| `-o, --output table\|json\|yaml`         | output format; JSON is stable and versioned (`tmoOutputVersion: v1`) |
| `--context`, `--kubeconfig`              | kube target overrides                                                |
| `--no-color`                             | disable ANSI color (honors `NO_COLOR`; auto-off when not a TTY)      |
| `--direct`, `--workers`, `--pod-timeout` | HTTP fan-out controls                                                |

## Exit codes

Stable and scriptable:

| Code | Meaning                                                     |
| ---- | ----------------------------------------------------------- |
| `0`  | success                                                     |
| `1`  | generic / unexpected error                                  |
| `2`  | usage error (e.g. a removed `engine` write-verb → use Helm) |
| `3`  | not found (engine, server, salt)                            |
| `4`  | validation / guardrail rejection                            |
| `5`  | timed out (e.g. `engine wait`)                              |
| `6`  | confirmation declined / `--yes` required                    |

## Common tasks

### See the fleet

```bash theme={null}
tmo-cli engine list -A                # all engines, all namespaces
tmo-cli engine list -A -w             # watch for changes (table only)
tmo-cli engine describe my-engine     # spec + status + conditions + endpoints + events
```

### Deploying or changing engines (owned by Helm)

<Note>
  Engine fleet membership and spec are owned by the **Helm release** — `tmo-cli`
  does not create, apply, edit, or delete engines. To add, change, or remove an
  engine, edit the chart values and `helm upgrade`.
</Note>

The CLI's role in a rollout is the **readiness gate** after the upgrade:

```bash theme={null}
helm upgrade tensormesh-operator oci://ghcr.io/tensormesh-production/charts/tensormesh-operator -f values.yaml
tmo-cli engine wait my-cache --for=AllInstancesReady --timeout=5m   # exit 5 on timeout
tmo-cli engine describe my-cache
```

### Inspect servers and cache state

```bash theme={null}
tmo-cli node list -e my-cache                       # per-node servers + live health
tmo-cli cache status -e my-cache                    # aggregated /status + per-node + TOTAL
tmo-cli cache status -e my-cache --node gpu-node-07 # raw /status for one server
tmo-cli engine logs my-cache --node gpu-node-07 -f  # stream logs (--all to multiplex)
```

### Operate (destructive — gated)

```bash theme={null}
tmo-cli cache clear -e my-cache --node gpu-node-07  # one server; prompts
tmo-cli cache clear -e my-cache --all --yes         # whole fleet, no prompt

tmo-cli quota list -e my-cache                      # per-cache_salt budgets
tmo-cli quota set acme-prod --limit 48 -e my-cache
tmo-cli quota rm bench-harness -e my-cache --yes    # data evicts next cycle
```

<Warning>
  Destructive runtime operations (`cache clear`, `quota rm`) print a blast-radius
  preamble and require confirmation; pass `-y/--yes` to bypass. In a non-TTY
  without `--yes` they refuse (exit 6) rather than guess.
</Warning>

### Diagnose and file a ticket

```bash theme={null}
tmo-cli diag health -e my-cache       # healthcheck + periodic-thread sweep
tmo-cli diag periodic -e my-cache     # per-server periodic-thread detail
tmo-cli diag threads --node gpu-node-07 -e my-cache
tmo-cli diag loglevel set lmcache DEBUG --node gpu-node-07 -e my-cache
tmo-cli diag bundle -e my-cache       # redacted support tarball (.tgz)
```

`diag bundle` collects the CR, recent events, and each server's `/conf`,
`/status`, `/threads`, and `/periodic-threads-health` into a gzip tarball.
**Secrets are redacted by default**; `/env` is included only with `--include-env`
(and is still redacted). Attach the tarball to a support ticket.

## Output and scripting

Tables are for humans; `-o json` (and `-o yaml`) is the automation contract,
wrapped in an envelope with `tmoOutputVersion: v1`:

```bash theme={null}
tmo-cli node list -e my-cache -o json | jq '.items[] | select(.healthy==false)'
tmo-cli engine get my-cache -o json | jq '.item.status.readyInstances'
```

Color is off in pipes and non-terminals automatically, and can be forced off with
`--no-color` or `NO_COLOR=1`.

## Troubleshooting

| Symptom                                                 | Cause / fix                                                                        |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `... forbidden ... ask for the 'tensormesh-admin' role` | your RBAC lacks a needed verb; run `tmo-cli diag preflight` to see which           |
| `node list` / `cache status` empty                      | the engine has no ready endpoints — is a GPU node up and the engine `Running`?     |
| `unreachable` in a health table                         | that server's HTTP API didn't answer; check `tmo-cli engine describe` + pod status |
| `engine wait` exits `5`                                 | the condition didn't become true within `--timeout`                                |
| HTTP commands hang on a big fleet                       | tune `--workers` / `--pod-timeout`; partial results still render                   |

<Note>
  `tmo-cli` does not phone home — there is no telemetry in the default build.
</Note>