---
name: o11y-release
description: |
  Run safe gray releases (phased rollouts) for TiDB Cloud O11Y components.
  Use this whenever users ask to deploy, roll out, canary, or rollback regional-server,
  global-server, drhouse, dataplane, cloud-addon, agent-controller, infra, CRD,
  or data-collector. This skill plans waves, triggers GitHub CD workflows, reuses
  built images when supported, verifies health in Grafana Prometheus per region,
  and applies rollback procedure on failed waves.
---

# O11Y Release Skill

Use this skill to execute production releases with **small blast radius first**,
then expand wave by wave only after verification passes.

## Region Metadata Source

- Region/group metadata lives in
  [`deployments/deploy-groups.txt`](../../deployments/deploy-groups.txt).
- Treat this file as the source of truth for `env`, `vendor`, `region`,
  `logical_region`, `clusters`, and `isByoc`.

## Prerequisites

### 1) gh-cli

```bash
gh auth status
gh api user --jq '.login'
```

If auth is missing: `gh auth login`.

### 2) grafana-mcp-server

Required env vars:

- `GRAFANA_URL`
- `GRAFANA_SERVICE_ACCOUNT_TOKEN`

Quick checks:

```bash
: "${GRAFANA_URL:?GRAFANA_URL is required}"
: "${GRAFANA_SERVICE_ACCOUNT_TOKEN:?GRAFANA_SERVICE_ACCOUNT_TOKEN is required}"
curl -sf "${GRAFANA_URL}/api/health"
```

Recommended Mcp server setup with minimum tools usage:
```json
{
  "mcpServers": {
    "grafana": {
      "command": "uvx",
      "args": ["mcp-grafana","--enabled-tools","loki,prometheus,datasource,dashboard,search","--disable-write"],
      "env": {
        "GRAFANA_URL": "https://self-monitoring-grafana.observability.tidbcloud.com",
        "GRAFANA_SERVICE_ACCOUNT_TOKEN": "<your_viewer_token>"
      }
    }
  }
}
```

Minimum token capability: datasource read/query for Prometheus checks.

## Release Workflow

1. **Classify component** and choose the CD workflow plus namespace scope.
2. **Plan waves** from low traffic to high traffic.
3. **Run wave 1** with workflow_dispatch.
4. **Capture built images** from workflow logs (if component supports overrides).
5. **Run next waves** with reused images when possible.
6. **Verify each wave** in Grafana before proceeding.
7. **Pause and ask the user what to do next** if wave checks confirm a real
   service/pod abnormality.
8. **Rollback immediately** only if the user chooses rollback.

Read exact mappings in:

- `references/components.md`
- `references/monitoring.md`

## 1) Select Workflow

Use `references/components.md` to map component -> workflow -> inputs ->
namespaces.

Canonical command pattern:

```bash
gh workflow run "{Workflow Name}" \
  --repo tidbcloud/observability \
  --ref {branch} \
  -f git-ref={branch} \
  -f env={env} \
  -f vendor={vendor} \
  -f "region={r1,r2,...}"
```

## 2) Wave Planning Rules

- Wave 1: 1-3 lowest traffic regions (canary)
- Wave 2: low traffic
- Wave 3: medium traffic
- Wave 4: high traffic / critical

Never mix first-time risky regions with highest-traffic regions in one wave.

## 3) Image Reuse Optimization

For workflows supporting image override (notably regional-server):

1. Run first wave normally.
2. Read "Final Image Resolution" logs.
3. Extract image refs and reuse for later waves.

Example follow-up run with fixed images:

```bash
gh workflow run "Regional Server CD" \
  --repo tidbcloud/observability \
  --ref {branch} \
  -f git-ref={branch} \
  -f env=prod \
  -f vendor=aws \
  -f "region={wave_regions}" \
  -f "regional-server-image={rs_image}" \
  -f "data-server-image={ds_image}"
```

## 4) Wave Verification Gate

For each region in a wave, query Prometheus datasource
`Prometheus-regional-{logical_region}`.

Before running the PromQL checks, load the component namespace scope from
`references/components.md`.

- For a single namespace, use `namespace="that-namespace"`
- For multi-namespace workflows, use a regex matcher such as
  `namespace=~"kube-system|self-monitoring|observability"`

Do **not** hardcode `observability` unless the component mapping says that is
the only namespace involved.

Run checks in sequence — details and PromQL in `references/monitoring.md`:

1. **Rollout convergence** — wait until generation mismatch clears (up to 5 min) before checking anything else
2. **Ready replicas** > 0
3. **Unavailable replicas** == 0 — if this is the *only* failure and there are no restarts/CrashLoops, wait 2 min and re-check once before stopping (transient scheduling blip vs. real failure)
4. **CrashLoopBackOff** — empty result (required, no grace period)
5. **Restart spike over 5 m** == 0 — use 5 m window, not 10 m, to avoid capturing pre-existing restarts

If any check remains unresolved after retries, or if a service/pod abnormality
is confirmed earlier (for example CrashLoopBackOff, repeated restarts, no ready
replicas, or a retry that still shows unavailable replicas), **stop the rollout
immediately and ask the user what to do next**. Do not silently continue to the
next region and do not make the rollback decision on the user's behalf.

Use `ask_user` and include:

1. The component, region, workflow run URL, and namespace scope
2. The abnormal signal and a short failure pattern summary from
   `references/monitoring.md`
3. Clear next-step choices such as:
   - rollback now
   - wait and re-check
   - inspect logs / investigate
   - continue anyway / override

Only proceed after the user answers.

Detailed queries, thresholds, and the interpretation table are in `references/monitoring.md`.

## 5) Rollback Policy

Preferred rollback after user approval: re-run workflow with last-known-good git ref.

Emergency rollback (when cluster access is available):

```bash
kubectl rollout undo deployment/{deployment} -n {namespace}
```

Resolve `{namespace}` from `references/components.md`. If a workflow touches
multiple namespaces, list the changed deployments first and roll back each
deployment in its own namespace rather than assuming a single shared namespace.

After rollback, rerun the same verification gate with the same namespace scope.
If the user chooses investigation instead of rollback, keep the rollout paused
and report findings before taking further release actions.

## Output Format

When executing this skill, return a concise release report with:

1. Component/workflow/ref
2. Namespace scope
3. Wave plan
4. Per-wave run IDs
5. Per-region verification table
6. User decision / next action when blocked by abnormality
7. Rollback action (if any)