# vm-migration Runbook

Operator procedures for a tenant-history migration campaign: moving
VictoriaMetrics tenant data off the vmstorage nodes that are about to be
removed, then decommissioning those nodes. This runbook owns operator
sequence, safety gates, recovery, and maintenance. Exact HTTP request,
response, status, and default semantics live in [api.md](api.md); build and
CLI basics live in the top-level `README.md`.

## Campaign Overview

A campaign shrinks a vmcluster's vmstorage topology from
`sourceReplicaCount` to `targetReplicaCount` without losing the history
stored on the removed ordinals:

1. Stop new writes to (and later reads from) the drained ordinals — git
   patches applied by ArgoCD.
2. Copy each tenant's history, shard by shard, from a temporary vmselect
   that reads only the drained ordinals into the production vminsert — the
   serve loop.
3. Verify per shard that the retained-ordinals view matches the drained
   view, then remove the nodes and reclaim storage.

Three actors cooperate:

| Actor | Owns | Changed via |
|-------|------|-------------|
| Git repo + ArgoCD | VMCluster spec at `deployments/dataplane/vmcluster/envs/<region>/vmcluster.yaml` | yq patch → human commit → ArgoCD sync |
| vm-migration serve | manifests, vmctl copies, verification, temp vmselect lifecycle | HTTP API ([api.md](api.md)) |
| kubectl / Grafana | cluster state, query correctness | read-only checks |

Per tenant the serve loop runs `manifest` → `migrate` (vmctl vm-native per
`shardStepInterval` shard, newest window first) → `verify` (origin vs
candidate datapoint counts within a fixed 1% tolerance).
Tenants are processed strictly sequentially; a re-run resumes at the first
incomplete step.

Different migrations may run concurrently at the API level, but the
temporary vmselect names are fixed: two campaigns in the same namespace
would collide on `vmselect-migration` / `vmselect-candidate`. Run one
campaign per cluster at a time.

Phase map — every phase ends at a gate; do not start the next one until the
exit condition holds:

| Phase | Scope | Exit condition |
|-------|-------|----------------|
| 0 Preflight | cluster + serve sanity | VMCluster operational, STORAGE COUNT equals `sourceReplicaCount`, serve reachable |
| 1 Prepare | tenants, plan, git patches, temp vmselects | both temp vmselect deployments have all desired replicas Ready |
| 2 Migrate | probe, run, monitor | summary: `done == total`, `failed == 0`, every tenant fully verified |
| 3 Decommission | dashboard baseline, git patches, undeploy | STORAGE COUNT equals `targetReplicaCount`, dashboards flat, temp vmselects gone |
| 4 Cleanup | PVC/EBS reclamation | operator confirms resources reclaimed |

## Safety Classification

| Class | Actions | Rule |
|-------|----------|------|
| Read-only | serve GET endpoints, `plan` / `tenants` / `version` CLI, `kubectl get` / `describe`, Grafana | run freely |
| State-changing HTTP | `POST /migrations/{name}`, `/run`, `/probe`, `/deploy`, `/undeploy`, `/tune` | confirm with the operator before each call |
| Git-mediated | yq patches to `vmcluster.yaml` | produce the diff only — the operator reviews, commits, and waits for ArgoCD sync |

`POST /cancel` is the exception among state-changing endpoints: run it
promptly when a stop is needed. Use its optional `jobId` guard to avoid
cancelling a newer job; a cancelled run can resume with a re-POST `/run`.

serve has no built-in authentication and is expected behind an authenticating
reverse proxy; treat the base URL as trusted-internal.

## Prerequisites

| Requirement | Value |
|-------------|-------|
| Binary | `make build` (or `make install`) from this directory |
| serve base URL (`$BASE`) | regional gateway `https://www.ds.<region-suffix>.aws.observability.tidbcloud.com/internal/vm-migration`, or `http://localhost:8080` when port-forwarded |
| kubectl context | `controlplane.o11y-prod-regional-<region-suffix>` |
| Git checkout | observability monorepo (for the vmcluster.yaml patches) |
| Tools | `yq`, `jq`, `curl`, Grafana access for dashboard sampling |

Known example — region `prod-aws-us-west-2-staging` (region-suffix
`us-west-2-staging`): base
`https://www.ds.us-west-2-staging.aws.observability.tidbcloud.com/internal/vm-migration`,
context `controlplane.o11y-prod-regional-us-west-2-staging`.

Conventions below: `$BASE` is the serve base URL, `$NAME` the migration name
(a short campaign label, e.g. `prod-us-west-2-staging` — independent of the
region string), `$REGION` the plan's region, `$NAMESPACE` the plan's Kubernetes
namespace, and `$REPO` the monorepo checkout.

## Phase 0 — Preflight

```bash
kubectl -n $NAMESPACE get vmcluster vmcluster
kubectl -n $NAMESPACE describe vmcluster vmcluster
curl -s $BASE/migrations | jq .
```

Confirm before proceeding:

- VMCluster status `operational`; INSERT/STORAGE/SELECT COUNTs match
  expectations — STORAGE COUNT must equal `sourceReplicaCount`.
- vmagent split rules are frozen — no concurrent topology edits.
- serve is up and lists existing migrations; no campaign against this cluster
  is active. If `$NAME` already exists, inspect its status before proceeding.

## Phase 1 — Prepare

### 1.1 Collect the tenant inventory

Four sources, then a merge. All read-only:

```bash
# Active dedicated clusters (kubectl in the current context)
vm-migration tenants dedicated k8s --output tenants-dedicated-k8s.json

# Archived dedicated clusters (aws s3)
vm-migration tenants dedicated s3 --s3-bucket <bucket> --output tenants-dedicated-s3.json

# Essential tier, stage 1: mgmt gRPC listing (port-forwarded mgmt addr)
vm-migration tenants essential mgmt --addr <addr> --region <region> --provider <provider> --output essential-mgmt.json

# Stage 2: join regional RDS for vm account/project ids
vm-migration tenants essential rds --clusters-in essential-mgmt.json --secret-file <secret-file> --db-host <db-host> --output essential-rds.json

# Stage 3: resolve deleted essential clusters from the S3 archive
vm-migration tenants essential s3 --clusters-in essential-rds.json --s3-bucket <bucket> --output tenants-essential.json

# Merge into the flat tenant array; archived (s3) wins over k8s on collision
vm-migration tenants merge \
  --input tenants-dedicated-k8s.json \
  --input tenants-dedicated-s3.json \
  --input tenants-essential.json \
  --output tenants.json
```

Each intermediate file is `{"total": N, "tenants": [{"tenant": "786:0",
"bizType": "staging-tidbcloud", "source": "k8s"}]}`; the merged output is a
flat sorted tenant array. Data-integrity caveats — the manual part of this
step:

- Sources overlap; merge deduplicates by tenant, preferring the archived
  entry.
- Some clusters have undergone project migration — spot-check surprising
  `bizType` values.
- Review the final `tenants.json` (count, tenants, bizTypes) with the
  cluster owner before compiling: this list defines migration scope.

### 1.2 Author and compile the plan

```bash
vm-migration plan init --output plan.json
# ... fill in the fields below ...
vm-migration plan compile --plan-path plan.json --tenants tenants.json --output plan-compile.json
```

Fields to fill (defaults in `internal/cli/templates/plan-template.json`):

| Field | Meaning |
|-------|---------|
| `namespace` | Kubernetes namespace containing the VMCluster and temporary vmselects |
| `region` | e.g. `prod-aws-us-west-2-staging`; derives the repo env path |
| `sourceReplicaCount` / `targetReplicaCount` | must satisfy `0 < target < source` |
| `retentionWindow.start` / `.end` | RFC3339 UTC; the history window to migrate |
| `shardStepInterval` | vmctl copy granularity per shard, e.g. `6h` |
| `vmselects.image`, per-target `replicas` / `resources` | temp vmselect specs |
| `vmctl.concurrency` / `rateLimitBytes` / `stepInterval` | vmctl knobs; `rateLimitBytes` null = unlimited |

Endpoints are derived from `namespace` and fixed names — never hand-written:
origin `vmselect-vmcluster`, insert `vminsert-vmcluster`, temp
`vmselect-migration` (reads only the drained ordinals — the export source)
and `vmselect-candidate` (reads only the retained ordinals — the
verification path).

`plan compile` only validates and embeds the tenant list; it never writes
runtime files. Review `plan-compile.json` before submission.

### 1.3 Submit the compiled plan

```bash
curl -X POST $BASE/migrations/$NAME \
  -H 'Content-Type: application/json' \
  --data-binary @plan-compile.json
```

Confirm the returned migration name, then fetch the stored plan before
continuing. If the name is already registered or preparation was interrupted,
stop; see
[Migration registration and removal](#migration-registration-and-removal).
Do not replace runtime files while serve is running. See the
[prepare contract](api.md#post-migrationsname) for the exact response and
conflict behavior.

### 1.4 Review the expansion

```bash
curl -s $BASE/migrations/$NAME/plan | jq .    # runtime plan verbatim
curl -s $BASE/migrations/$NAME/steps | jq .   # pending steps + exact vmctl command lines
```

`/steps` is read-only and shows what the next run would execute — check the
vmctl command lines, shard windows, and tenant order before dispatching
anything.

### 1.5 enable-dedup (git patch)

Stabilizes origin/candidate count comparisons during verification. The patch
is idempotent:

```bash
yq -i 'with(select(.kind == "VMCluster"); .spec.vmselect.extraArgs."dedup.minScrapeInterval" = "1ms")' \
  $REPO/deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
git -C $REPO diff -- deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
```

Review the diff, commit, push, and wait for ArgoCD sync before continuing.
vm-migration never performs the commit.

### 1.6 drain-insert (git patch)

New writes hash only to the retained vmstorages; the drained ordinals keep
their history for export:

```bash
yq -i 'with(select(.kind == "VMCluster"); .spec.vmstorage.maintenanceInsertNodeIDs = [<drained ordinals>])' \
  $REPO/deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
git -C $REPO diff -- deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
```

Drained ordinals for source 4 → target 3 are `[3]`; for 8 → 6, `[6, 7]`.
Commit, push, wait for ArgoCD sync.

### 1.7 Deploy the temporary vmselects

```bash
curl -X POST $BASE/migrations/$NAME/deploy
curl -N $BASE/migrations/$NAME/logs      # rollout progress
kubectl -n $NAMESPACE get svc  | grep select
kubectl -n $NAMESPACE get pods | grep select
```

`vmselect-migration` and `vmselect-candidate` are always deployed together;
all desired replicas must be Ready before Phase 2. See the
[deploy contract](api.md#post-migrationsnamedeploy) for job behavior.

## Phase 2 — Migrate

### 2.1 Probe and estimate

```bash
curl -X POST $BASE/migrations/$NAME/probe
JOB_ID=$(curl -s $BASE/migrations/$NAME | jq -r '.jobId // .lastJobId')
curl -s "$BASE/migrations/$NAME/jobs/$JOB_ID/artifact?format=text"
```

Watch `/logs` for completion before fetching the probe artifact. The jq
fallback works while the probe is active and after it becomes the latest
finished job; see the [job model](api.md#job-model) and
[artifact contract](api.md#get-migrationsnamejobsidartifact) for exact field
and availability semantics.

The probe artifact reports per-shard bytes, duration, and rate plus totals:

- `16B` shards are effectively empty (export envelope only) — they migrate
  in milliseconds.
- total bytes / aggregate rate gives a wall-clock estimate for the run.
- a tenant that is empty across every shard deserves a second look — either
  it genuinely has no data in the window, or the retention window is wrong.

The probe is read-only with respect to shard progress; it may generate a
missing manifest as its only migration-state write. It does not import data or
mark shards migrated or verified.

### 2.2 Tune (optional)

```bash
curl $BASE/migrations/$NAME/tune | jq .    # effective knobs
curl -X POST $BASE/migrations/$NAME/tune \
  -H 'Content-Type: application/json' \
  -d '{"vmctl": {"concurrency": 8, "rateLimitBytes": 104857600}}'
```

`vmctl` overrides apply at the next shard boundary; `vmselect` overrides at
the next `/deploy`. Use the probe estimate to decide: raise concurrency for
long runs, set `rateLimitBytes` to keep export bandwidth bounded. See the
[tune contract](api.md#post-migrationsnametune) for replacement and reset
semantics.

### 2.3 Run

```bash
curl -X POST $BASE/migrations/$NAME/run
# optional body: {"retries": N}; omitted defaults to 5, explicit 0 disables
```

An omitted body runs with 5 retries, so transient tenant failures self-heal;
send `{"retries": 0}` for a single controlled pass. Retries only re-run work
that remains incomplete; hard errors and cancellation do not retry. See the
[run contract](api.md#post-migrationsnamerun) for retry and conflict
semantics.

### 2.4 Monitor

```bash
watch -n 2 'curl -s "$BASE/migrations/$NAME/summary?format=text"'
curl -N $BASE/migrations/$NAME/logs
```

Use the summary to identify the current tenant, step, shard progress, and
throughput. The exit condition is `complete == true` (`done == total`,
`failed == 0`, every tenant fully verified). The
[summary contract](api.md#get-migrationsnamesummary) defines the text and JSON
representations.

### 2.5 Failures and resume

Before retrying, fetch the migration status and summary. Interpret fields and
category values using the
[migration status](api.md#get-migrationsname) and
[error category](api.md#error-categories) contracts.

| Symptom | Check | Action |
|---------|-------|--------|
| Run stopped and no job is active | Find the first incomplete tenant and inspect `loopStatus.tenants[].steps[].lastError` | Re-POST `/run`; completed steps and shards are skipped |
| Category is `transient` | Look for network failures or upstream HTTP 5xx in `/logs` | Re-run, optionally with a bounded retry count |
| Category is `permanent` | Inspect the tenant action log for invalid data or usage | Correct the input, or replace the campaign through the registration procedure before re-running |
| Category is `infrastructure` | Check binaries, credentials, endpoints, temp vmselects, and cluster readiness | Restore the missing precondition, then re-run |
| Verification repeatedly fails for one tenant | Confirm enable-dedup is effective; compare the retention window with the drain-insert cutover; check for late origin samples | Correct the cluster or window issue before another retry |
| Tenant is running with no byte progress | Tail `/logs` and check whether vmctl is scanning an empty shard | Wait if output advances; investigate only after output and status both stall |
| Dispatch returns a conflict | Check `GET /migrations/$NAME` for the active job | Wait for it or cancel the matching job ID before dispatching another |

To stop an active job:

```bash
curl -X POST $BASE/migrations/$NAME/cancel \
  -H 'Content-Type: application/json' \
  -d '{"jobId":"<id>"}'
```

Keep the `jobId` guard when acting on a previously observed job. See the
[cancel contract](api.md#post-migrationsnamecancel) for guarded and unguarded
behavior.

## Phase 3 — Decommission

Start only after Phase 2's exit condition holds and the operator has chosen
a rollback window — the temp vmselects and the drained vmstorages are the
rollback path; do not remove them early.

### 3.1 Dashboard baseline

Sample the panels that matter before touching the cluster: dataplane
resource overview scoped to the cluster/namespace plus a few key tenant
dashboards covering the migrated window. Record values; after each step
below, re-check the same panels against this baseline.

### 3.2 drain-select (git patch)

Production queries stop touching the drained vmstorages:

```bash
yq -i 'with(select(.kind == "VMCluster"); .spec.vmstorage.maintenanceSelectNodeIDs = [<drained ordinals>])' \
  $REPO/deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
git -C $REPO diff -- deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
```

Commit, push, wait for ArgoCD sync; re-check dashboards.

### 3.3 scale-down (git patch)

The StatefulSet removes the highest ordinals in reverse order while PVCs
are retained:

```bash
yq -i 'with(select(.kind == "VMCluster"); .spec.vmstorage.replicaCount = <target>)' \
  $REPO/deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
git -C $REPO diff -- deployments/dataplane/vmcluster/envs/$REGION/vmcluster.yaml
```

After ArgoCD sync, watch until STORAGE COUNT drops and status stays
operational:

```bash
kubectl get vmcluster -n $NAMESPACE vmcluster
# NAME       INSERT COUNT   STORAGE COUNT   SELECT COUNT   AGE    STATUS
# vmcluster  2              3               2              233d   operational
```

Compare dashboards against the baseline — data must show no fluctuation.

### 3.4 Undeploy the temporary vmselects

```bash
curl -X POST $BASE/migrations/$NAME/undeploy
curl -N $BASE/migrations/$NAME/logs
kubectl -n $NAMESPACE get svc  | grep select    # only vmselect-vmcluster remains
kubectl -n $NAMESPACE get pods | grep select
```

Confirm both temporary services and pods are gone. See the
[undeploy contract](api.md#post-migrationsnameundeploy) for job behavior.

## Rollback

Everything before Phase 4 is reversible — the drained ordinals keep their
data until their PVCs are reclaimed. Each rollback is the reverse git patch
plus the same commit → ArgoCD sync wait, applied in the opposite order the
changes were made:

| Campaign state | Rollback |
|----------------|----------|
| After drain-insert, before running | Revert the `maintenanceInsertNodeIDs` patch; new writes hash to all ordinals again. Nothing has moved yet. |
| Mid-migrate, abandoning | `POST /cancel` and do not re-post `/run`; revert the drain-insert patch to restore normal writes. Migration never deletes or mutates source data — the origin copy stays intact on the drained ordinals, so abandoning loses nothing. `POST /undeploy` the temp vmselects once the decision is final. |
| After drain-select, dashboards regress | Revert the `maintenanceSelectNodeIDs` patch; queries touch all ordinals again. |
| After scale-down, problems found | The removed ordinals' PVCs are retained — patch `replicaCount` back to the source count and let the StatefulSet reattach them. |

The window closes at [Phase 4.1](#41-pvc-and-ebs-reclamation): once the
PVCs and EBS volumes of the removed ordinals are gone, the drained copy no
longer exists and only a fresh campaign can rebuild it.

## Phase 4 — Cleanup

### 4.1 PVC and EBS reclamation

Scale-down retained the PVCs of the removed ordinals; reclaiming them
destroys the last copy of that node's history — the point of no return for
rollback (see [Rollback](#rollback)). Per object:

```bash
kubectl -n $NAMESPACE get pvc | grep vmstorage          # candidates
kubectl -n $NAMESPACE get pvc <pvc> -o jsonpath='{.spec.volumeName}'
```

Identify the PVCs of the removed ordinals only (for a VMCluster named
`vmcluster`, e.g. `vmstorage-vmcluster-vmstorage-3` for ordinal 3). After
operator confirmation, delete the PVC — with a `Delete` reclaim policy the
PV and the underlying EBS volume go with it; otherwise remove the EBS
volume from the AWS side once the PV is gone. Verify in AWS that the
volumes are gone (or snapshot-then-delete if an extra safety net is
wanted).

## Service Operations

### Start and ownership

```bash
vm-migration serve --addr :8080 --migrations-dir ./migrations
```

Use one serve process per migrations directory. The process holds
`<migrations-dir>/serve.lock` for its lifetime and refuses to start if another
process owns the directory. At startup it loads subdirectories containing
`migration-loop.json`; malformed migrations or directories without that file
are skipped and reported on stderr.

After a start or restart, confirm the expected names appear before dispatching
work:

```bash
curl -s $BASE/migrations | jq .
```

### Stop safely

Send SIGINT or SIGTERM and wait for the process to exit. Shutdown cancels
in-flight work and gives HTTP requests five seconds to drain. A second signal
forces exit, so reserve it for a stuck shutdown.

Before changing anything under the migrations directory, confirm serve has
exited and no other process holds `serve.lock`. Never edit, move, or remove a
migration directory while serve is running.

### Recover interrupted jobs

Restart recovery depends on the interrupted job kind:

| Job | Startup behavior | Operator action |
|-----|------------------|-----------------|
| `migrate-run` | Automatically starts a new job from persisted progress, with the default retry budget | Monitor the new job ID and summary; do not dispatch a second run |
| `export-probe` | Records the attempt as interrupted and does not re-run it | Re-POST `/probe` only if a fresh estimate is still needed |
| `vmselect-deploy` / `vmselect-undeploy` | Does not auto-resume cluster mutation | Check Kubernetes state, then re-POST the same idempotent operation if needed |
| Manually cancelled or normally finished job | Never retried | Dispatch new work only when the campaign procedure calls for it |

If startup reports an unreadable in-flight marker or job history, preserve the
files for diagnosis. Do not delete the marker to force progress; repair or
restore the migration state first.

### Migration registration and removal

Always create a migration through `POST /migrations/{name}`. A directory copied
into the migrations volume after startup is not registered until restart.

For a prepare conflict, stop serve and inspect these exact paths:

- `<migrations-dir>/<name>.tmp` means preparation was interrupted. After
  confirming no prepare process remains, remove only this staging directory,
  restart serve, and submit the compiled plan again.
- `<migrations-dir>/<name>` is committed migration state. Restart serve to load
  it. To intentionally replace it, first archive the whole directory outside
  the migrations root; never remove individual status files and reuse the
  remainder.

After archiving committed state, restart serve, confirm the old name is absent,
and submit the corrected compiled plan through the API.

There is no migration DELETE endpoint. To retire a completed campaign, stop
serve, archive or remove its entire directory, then restart and confirm the
name no longer appears. Treat removal as destructive because it discards job
history, artifacts, progress, and tenant action logs.

### Logs and retained state

- Process startup, shutdown, registry, and recovery diagnostics go to stderr.
- `/logs` streams current progress and observed action output; it is not the
  durable tenant log.
- Full tenant action output is retained in `tenants-*.log` under the migration
  directory. Copy the relevant file out before removing a campaign.
- `jobs.jsonl` is append-only and job artifacts accumulate. Plan periodic
  retention cleanup, performed only while serve is stopped.
- `jobs-inflight.json` is recovery state, not a cleanup target. Leave it intact
  unless its corruption has been diagnosed and a state-repair procedure has
  been agreed.