# Observability Stack Upgrade Playbook — vm-operator / vmcluster / loki > Scope: staged, per-region upgrade of the three core dataplane components. > Pilot region: `us-west-2-staging` (cluster `o11y-prod-regional-us-west-2-staging`), completed 2026-09-03. > Status: **staging done, prod rollout pending** --- ## 1. Current State | Component | us-west-2-staging (pilot) | All other regions | Target | |-----------|---------------------------|-------------------|--------| | vm-operator | **v0.73.1** (chart 0.66.2) | v0.62.0 (chart 0.51.4) | v0.73.1 / chart 0.66.2 | | vmcluster (insert/select/storage) | **v1.148.2-cluster** | v1.122.2-cluster(-hotfix-20260305) | v1.148.2-cluster | | vmalert-shared | **v1.148.2-cluster** | v1.122.2-cluster | v1.148.2-cluster | | loki | **3.7.7** (chart 6.55.0) | 3.5.3 (chart 6.29.0) | 3.7.7 / chart 6.55.0 | Design docs (Feishu, Chinese): - VM operator: https://pingcap.feishu.cn/wiki/ZJUaw5vtqiN4Iak4WNwcQvBVnRd - Loki: https://pingcap.feishu.cn/wiki/T5PbwwisBiM8m8kEmHJcwJkBnha All upgrade work for the pilot region is on branch `jinwei/upgrade_usw2_stg`. --- ## 2. Upgrade Mechanisms (added during the pilot) ### 2.1 Per-region chart version override Chart versions for externally-sourced charts used to be single global constants in the deploy scripts, which made per-region staged upgrades impossible. Both deploy paths (direct helm and ops-server managed) now support an optional per-region override file. If the file is absent, the global default applies — existing regions are unaffected. | Component | Default (global) | Override file | |-----------|------------------|---------------| | vm-operator | `VM_OPERATOR_CHART_VERSION=0.51.4` in `deployments/infra-v2/deploy.sh` / `deploy-with-ops.sh` | `deployments/infra-v2/vm-operator/envs/{env}/regional-{vendor}-{logical_region}.chart-version` | | loki | `6.29.0` in `deployments/dataplane/deploy.sh` / `deploy-with-ops.sh` (`LOKI_CHART_VERSION` env also works) | `deployments/dataplane/loki/envs/{env}-{vendor}-{logical_region}/chart-version` | To upgrade one region: drop a `chart-version` file with the new version into that region's directory. Nothing else changes for other regions. `destroy-region.sh` cleans the override up together with the region (vm-operator: explicit removal; loki: whole env dir is removed). ### 2.2 Publishing external charts to the OCI registry Managed (ops-server) regions pull charts from `oci://.dkr.ecr.us-west-2.amazonaws.com/observability-charts/:`. External chart specs live in `.github/scripts/publish-changed-charts.sh` (`external_chart_specs`); new versions are appended there, old ones kept. Publish a single external chart version via the `Publish Observability Charts` workflow: ```bash gh workflow run publish-observability-charts.yaml \ --ref -f git-ref= \ -f publish-charts="victoria-metrics-operator:0.66.2" # or "loki:6.55.0" ``` The `publish-charts` input accepts local chart names and external `name` / `name:version` selectors. Both dev and prod registries are always published by the workflow. Side effect: the `update-byoc-chart-sync` job opens a PR in `tidbcloud/docker-image-controller` adding the new tag to the BYOC allowlist — expected, review and merge it. ### 2.3 Image sync Third-party images must exist in each vendor's private registry (`third-party/...` in ECR for AWS). Use the `Image Sync` workflow: ```bash gh workflow run image-sync.yaml \ -f source-image="grafana/loki:3.7.7" \ -f dest-image-name="third-party/grafana/loki" \ -f env="prod" -f cloud-providers="aws" ``` The destination tag is copied from the source image tag. For tidbcloud-fork images (`victoriametrics/*`), images are built and pushed directly by the fork repo's `Build And Upload Image` workflow — check ECR availability there first before syncing manually. ### 2.4 Deploy trigger Both workflows dispatch per logical region and route to the ops-server managed path when `managedByOpsServer=true` in `deployments/deploy-groups.txt`: ```bash # infra: vm-operator (+ kube-prometheus-stack, cert-manager, vector, ...) gh workflow run infra-cd-v2.yaml --ref -f git-ref= \ -f env="prod" -f region="us-west-2-staging" -f vendor="aws" # dataplane: vmcluster, loki, alertmanager, sli-reporter gh workflow run dataplane-cd.yaml --ref -f git-ref= \ -f env="prod" -f region="us-west-2-staging" -f vendor="aws" ``` Note: `us-west-2-staging` is a **prod-group** logical region (`envs/prod*/...`), so `env=prod` is correct despite the name. --- ## 3. Per-Component Runbook ### 3.1 vm-operator (v0.62.0 → v0.73.1) Purpose: gzip-packed rule ConfigMaps eliminate the vmalert pod churn (v0.62.0 repacked ConfigMap→volume indexes every reconcile, rolling the deployment every ~22 minutes). Steps for a new region: 1. Sync images (once per vendor, already done for AWS prod): `victoriametrics/operator:v0.73.1` and `victoriametrics/operator:config-reloader-v0.73.1` (new dependency) → `third-party/victoriametrics/operator:{v0.73.1,config-reloader-v0.73.1}`. 2. Add `.chart-version` (content `0.66.2`) to the region's vm-operator env dir. 3. Edit `regional-{vendor}-{logical_region}.yaml`: `image.tag` → `v0.73.1`; remove `VM_VMALERTDEFAULT_CONFIGRELOADIMAGE` (removed upstream); add `VM_CUSTOMCONFIGRELOADERIMAGE` pointing at the config-reloader image in that region's registry. 4. Backup CRs, then dispatch `infra-cd-v2` for the region: `kubectl get vmalert,vmrule,vmcluster,vmagent -A -o yaml > backup.yaml` 5. Verify (see 4.1). ### 3.2 vmcluster (v1.122.2-cluster → v1.148.2-cluster) - Fork images are built by `tidbcloud/VictoriaMetrics` CI into ECR; all `v1.148.2-cluster` images (insert/select/storage/alert/agent/backup) are already in the prod ECR. - The fork's tenant-search hotfixes (`v1.122.2-cluster-hotfix-20260305`) are included in `v1.148.2-cluster` (verified at code level, not just by tag). - Edit the region's `deployments/dataplane/vmcluster/envs/{env}-{vendor}-{logical_region}/values.yaml`: set `vminsert/vmselect/vmstorage/vmalert` image tags to `v1.148.2-cluster`. - Dispatch `dataplane-cd`. The operator rolls in order: vmstorage (one by one, ~5 min each with `minReadySeconds`), then vmselect, then vminsert. - Verify (see 4.2). ### 3.3 loki (3.5.3 → 3.7.7, chart 6.29.0 → 6.55.0) Purpose: fix the query-frontend goroutine leak that stalls read queries (grafana/loki#21042, fixed by #21062 in 3.7.0+); security batch (Go 1.26.5, BusyBox removal, multiple HIGH CVEs). Steps for a new region: 1. Sync image `grafana/loki:3.7.7` → `third-party/grafana/loki:3.7.7` (per vendor registry). 2. Publish chart `loki:6.55.0` (see 2.2) — required for managed regions. 3. Add `chart-version` (content `6.55.0`) to the region's loki env dir. 4. Edit the region's `values.yaml`: `loki.image.tag` → `3.7.7`, and pin `serviceAccount.name: loki-ha` (chart ≥ 6.46.0 derives the SA name from the release fullname; pinning protects the IRSA/pod-identity binding). 5. Pre-checks: `helm template` with 6.29.0 vs 6.55.0 and diff (expected-only changes, see 4.3); validate the rendered config with `docker run --rm -v :/etc/loki/config.yaml grafana/loki:3.7.7 -config.file=/etc/loki/config.yaml -verify-config=true`. 6. Dispatch `dataplane-cd` for the region. 7. Verify (see 4.3). Do NOT touch `volumeClaimTemplates` during the upgrade (triggers StatefulSet recreation, grafana/helm-charts#19217). --- ## 4. Verification Checklists ### 4.1 vm-operator ```bash kubectl -n self-monitoring get deploy victoria-metrics-operator -o jsonpath='{.spec.template.spec.containers[0].image}' # v0.73.1 kubectl get cm -n observability -o name | grep -c 'vmalert.*rulefiles' # = max group-name duplication (see 5.1) kubectl get cm vm-vmalert-shared-rulefiles-0 -n observability -o json | jq '.binaryData | keys' # ["rules.yaml"] kubectl get pod -l app.kubernetes.io/instance=vmalert-shared -n observability -o jsonpath='{.items[0].spec.initContainers[*].name}' # config-init # vmalert /api/v1/rules: group/rule count and 0 load errors (compare with pre-upgrade) ``` ### 4.2 vmcluster ```bash kubectl get pods -n observability | grep -E 'vminsert|vmselect|vmstorage' # all on v1.148.2-cluster, Ready kubectl get vmcluster vmcluster -n observability -o jsonpath='{.status.updateStatus}' # operational # smoke: query vmselect /select/multitenant/prometheus/api/v1/query?query=count(up) ``` ### 4.3 loki ```bash # loki_build_info shows version="3.7.7", go1.26.5 kubectl port-forward -n observability pod/ 3100:3100 curl -s localhost:3100/metrics | grep -E '^loki_build_info|^go_goroutines' # historical query smoke: /loki/api/v1/query_range over the last hour # ring: /ring on a write pod — all ingesters ACTIVE # logs: no continuous "failed mapping AST" + "context canceled"; no panics ``` Expected render diff (chart 6.29.0 → 6.55.0): `compactor_address`(http) → `compactor_grpc_address`(9095), `bucketnames: ""` (bucket names stay in the explicit `storage_config.aws.s3` URL), memberlist FQDN, `fsGroupChangePolicy: OnRootMismatch`, probe tuning, sidecar container order. --- ## 5. Known Gotchas (learned during the pilot) ### 5.1 vmalert ConfigMap count is driven by group-name duplication, not size v0.73.1 packs rule groups in two phases: (1) batches must have unique group names (vmalert rejects duplicate group names within one rules file), (2) size splitting at 512KB with 50% headroom. Since our rules are generated per tenant cluster from the same template, the ConfigMap count equals the **maximum duplication count of any group name** (≈ number of tenant clusters), not ~4-7 as the design doc estimated. In staging it converged to 22. Content changes now reload in place via the config-reloader sidecar, but **adding/removing a tenant cluster still changes the bucket count and rolls the vmalert deployment once** — expected, and far rarer than before. ### 5.2 Orphan plaintext ConfigMaps stuck in Terminating Old (v0.62.0) rulefiles ConfigMaps carry finalizer `apps.victoriametrics.com/finalizer`. The new operator only finalizes ConfigMaps in its current desired set, so orphaned ones hang in Terminating forever. Remove the finalizer manually after the upgrade: ```bash kubectl patch cm -n observability --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]' ``` ### 5.3 One-time vminsert/vmselect rollout from the default preStop hook Operator ≥ v0.72.0 injects `lifecycle.preStop.sleep: 15s` by default (`VM_ENABLE_DEFAULT_PRESTOP_HOOK=true`, K8s ≥ 1.29) to reduce traffic loss during rollouts. This changes pod templates and rolls vminsert/vmselect once at operator-upgrade time (vmstorage is explicitly excluded). Set the env to `false` before the operator upgrade if even that one rollout must be avoided. ### 5.4 Misc observations - v0.73.1 logs a huge `bin_data_diff` dump on every rule ConfigMap update — rotates operator logs quickly; historical debugging may need cluster log search instead of `kubectl logs`. - `cannot decrease PVC=vmstorage-db size from=... to=50Gi` warning every reconcile: the vmcluster values still declare `storage: 50Gi` while volume-autoscaler grew the PVCs. Align the values with reality when touching that file. - Loki ≥ 3.5.8 images have no BusyBox: `kubectl exec -- sh` fails; use `kubectl debug` ephemeral containers. --- ## 6. Remaining Rollout Plan 1. [ ] Merge `jinwei/upgrade_usw2_stg` (contains: per-region version override mechanism, chart publish selector support, us-west-2-staging values). 2. [ ] Observe staging for 24h: no recurring vmalert rollouts, loki read `go_goroutines` flat, TSDB head series stable, query/ingest SLO normal. 3. [ ] Sync images to gcp/azure/alicloud registries (Image Sync workflow, `cloud-providers=gcp,azure,alicloud`). 4. [ ] Update the variants templates (`vm-operator/variants/*`, `loki/variants/*-values-tmpl.yaml`) so new regions start on the new versions; then flip the global defaults in the deploy scripts. 5. [ ] Prod rollout per region: add the region's override file + values change, dispatch the workflow, verify, observe 15–30 min. Suggested order: non-premium first, `us-west-2-premium` (largest rule volume) separately in a low-traffic window. 6. [ ] After all regions are on the new versions, remove the per-region override files and the legacy external chart spec entries (loki 6.29.0, victoria-metrics-operator 0.51.4). ### Rollback - vm-operator / loki: remove the region's `chart-version` override, revert the region's values file, re-dispatch the same workflow (or `helm rollback` directly in the region). - vmcluster: revert the image tags in the region's values file and re-dispatch `dataplane-cd`. No storage-format changes between v1.122.2 and v1.148.2 were found in the changelog review, but take a vmbackup snapshot before upgrading premium regions.