# Agent CLI Execution

This section is for an agent, or for a human intentionally following the agent path in a terminal-only workflow.

## Dev Release Details

### 1. Dev Release Preparation

1. Sync local state and resolve the target `RELEASE_SHA` from `main`.
2. Confirm there are no unexpected local changes.
3. Watch or trigger the four dev image workflows:
   - `dev-ci-global.yaml`
   - `dev-ci-regional.yaml`
   - `dev-ci-pipeline.yaml`
   - `dev-ci-prometheus.yaml`
4. Update the matching dev manifests.
5. Merge the manifest PR.
6. Run the matching dev deploy workflows from `main`.

### 2. Global Control Plane Dev Release

Edit:

- `global/deployment/kustomize/envs/dev/kustomization.yaml`

Set:

- `images[].newTag` to `RELEASE_SHA`

Deploy from `main` with:

```bash
gh workflow run -R "$REPO" universal-deploy-global.yaml \
  --ref main \
  -f env=dev \
  -f use_apply=false

gh workflow run -R "$REPO" universal-deploy-global.yaml \
  --ref main \
  -f env=dev \
  -f use_apply=true
```

### 3. Regional Control Plane Dev Release

Edit:

- `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-aws/kustomization.yaml`
- `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-gcp/kustomization.yaml`
- `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-azure/kustomization.yaml`

Set:

- each `images[].newTag` to `RELEASE_SHA`

Deploy with:

```bash
gh workflow run -R "$REPO" dev-deploy-regional.yaml \
  --ref main \
  -f use_apply=false

gh workflow run -R "$REPO" dev-deploy-regional.yaml \
  --ref main \
  -f use_apply=true
```

Current limitation:

- this workflow only deploys AWS dev `us-west-2`

### 4. OTel Push Pipeline Dev Release

Edit:

- `global/deployment/kustomize/envs/dev/configmap.yaml`

Set:

- `helmChart[name=pipeline].imageTag` to `RELEASE_SHA`

Then apply the global dev config via `universal-deploy-global.yaml` with `env=dev`.

Important runtime note:

- this changes the image used for new or reconciled collector deployments
- existing dev collectors may still need reconcile or rollout if immediate cutover is required

### 5. Prometheus Pull Pipeline Dev Release

Edit:

- `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/aws/kustomization.yaml`
- `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/gcp/kustomization.yaml`
- `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/azure/kustomization.yaml`

Set:

- each `images[].newTag` to `RELEASE_SHA`

Deploy with:

```bash
gh workflow run -R "$REPO" dev-deploy-prometheus-regional.yaml \
  --ref main \
  -f use_apply=false

gh workflow run -R "$REPO" dev-deploy-prometheus-regional.yaml \
  --ref main \
  -f use_apply=true
```

Current limitation:

- this workflow only deploys AWS dev `us-west-2`

## Prod Release Details

### 1. Release Preparation

1. Sync local state and inspect the diff from the previous prod release.
2. Confirm there are no unexpected changes.
3. Create and push a `release/v*` tag.
4. Create the GitHub Release with `gh release create`.
5. Watch the four auto-triggered prod image builds:
   - global
   - regional
   - OTel pipeline
   - Prometheus

If Prometheus needs to be rebuilt from the same tag, or built from a hotfix branch, run:

```bash
gh workflow run -R "$REPO" prod-ci-prometheus.yaml --ref "$TAG"

PROM_RUN_ID="$(gh run list -R "$REPO" \
  --workflow prod-ci-prometheus.yaml \
  --commit "$RELEASE_SHA" \
  --event workflow_dispatch \
  --limit 1 \
  --json databaseId \
  --jq '.[0].databaseId')"

gh run watch -R "$REPO" "$PROM_RUN_ID" --compact --exit-status
```

### 2. Global Control Plane Prod Release

The built image tag is the Git commit SHA, which is the same value as `RELEASE_SHA` for a tagged release.

Edit:

- `global/deployment/kustomize/envs/prod/kustomization.yaml`

Set:

- `images[].newTag` to `RELEASE_SHA`

Then submit and merge the PR:

```bash
git checkout -b release/global-"${RELEASE_SHA:0:12}"
$EDITOR global/deployment/kustomize/envs/prod/kustomization.yaml
git add global/deployment/kustomize/envs/prod/kustomization.yaml
git commit -m "release: bump global prod image to ${RELEASE_SHA}"
git push -u origin HEAD

gh pr create -R "$REPO" \
  --base main \
  --title "release: bump global prod image to ${RELEASE_SHA}" \
  --body "Update the prod global control plane image to ${RELEASE_SHA}."

gh pr merge -R "$REPO" --squash --auto --delete-branch
```

Deploy from `main` in two steps: diff first, then apply.

```bash
gh workflow run -R "$REPO" universal-deploy-global.yaml \
  --ref main \
  -f env=prod \
  -f use_apply=false

gh workflow run -R "$REPO" universal-deploy-global.yaml \
  --ref main \
  -f env=prod \
  -f use_apply=true
```

Minimum production checks:

```bash
aws eks update-kubeconfig --region us-west-2 --name o11y-prod-global-us-west-2
kubectl -n prod-ms get deploy,sts,pods | rg 'metrics-integration'
kubectl -n prod-ms rollout status deploy/metrics-integration-service
kubectl -n prod-ms rollout status statefulset/metrics-integration-workflow
```

### 3. Regional Control Plane Prod Release

After the tag push, wait for `prod-ci-regional.yaml` to finish.

Sync the regional image to GCP and Azure:

```bash
export REGIONAL_SOURCE_IMAGE="557537366020.dkr.ecr.us-west-2.amazonaws.com/tidbcloud/metrics-integration-regional:${RELEASE_SHA}"

gh workflow run -R "$REPO" universal-sync-multicloud-image.yaml --ref main \
  -f source-image="$REGIONAL_SOURCE_IMAGE" \
  -f dest-image-name=metrics-integration/metrics-integration-regional \
  -f env=prod \
  -f source-cloud-providers=aws \
  -f dest-cloud-providers=gcp,azure
```

Edit:

- `regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-aws/kustomization.yaml`
- `regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-gcp/kustomization.yaml`
- `regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-azure/kustomization.yaml`

Set:

- each `images[].newTag` to `RELEASE_SHA`

Then open and merge the PR:

```bash
git checkout -b release/regional-"${RELEASE_SHA:0:12}"
$EDITOR regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-aws/kustomization.yaml
$EDITOR regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-gcp/kustomization.yaml
$EDITOR regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-azure/kustomization.yaml
git add regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-aws/kustomization.yaml
git add regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-gcp/kustomization.yaml
git add regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-azure/kustomization.yaml
git commit -m "release: bump regional prod images to ${RELEASE_SHA}"
git push -u origin HEAD

gh pr create -R "$REPO" \
  --base main \
  --title "release: bump regional prod images to ${RELEASE_SHA}" \
  --body "Update AWS, GCP, and Azure prod regional control plane images to ${RELEASE_SHA}."

gh pr merge -R "$REPO" --squash --auto --delete-branch
```

Deploy from `main`:

```bash
gh workflow run -R "$REPO" prod-deploy-regional.yaml \
  --ref main \
  -f use_apply=false

gh workflow run -R "$REPO" prod-deploy-regional.yaml \
  --ref main \
  -f use_apply=true
```

### 4. OTel Push Pipeline Prod Release

After the tag push, wait for `prod-ci-pipeline.yaml` to finish.

Sync the pipeline image to GCP and Azure:

```bash
export PIPELINE_SOURCE_IMAGE="557537366020.dkr.ecr.us-west-2.amazonaws.com/tidbcloud/metrics-integration-pipeline:${RELEASE_SHA}"

gh workflow run -R "$REPO" universal-sync-multicloud-image.yaml --ref main \
  -f source-image="$PIPELINE_SOURCE_IMAGE" \
  -f dest-image-name=metrics-integration/metrics-integration-pipeline \
  -f env=prod \
  -f source-cloud-providers=aws \
  -f dest-cloud-providers=gcp,azure
```

Edit:

- `global/deployment/kustomize/envs/prod/configmap.yaml`

Set:

- `helmChart[name=pipeline].imageTag` to `RELEASE_SHA`

Do not change `pipelineImageOverride` for a normal release unless the registry path itself changes.

Then open and merge the PR:

```bash
git checkout -b release/pipeline-"${RELEASE_SHA:0:12}"
$EDITOR global/deployment/kustomize/envs/prod/configmap.yaml
git add global/deployment/kustomize/envs/prod/configmap.yaml
git commit -m "release: bump prod pipeline image tag to ${RELEASE_SHA}"
git push -u origin HEAD

gh pr create -R "$REPO" \
  --base main \
  --title "release: bump prod pipeline image tag to ${RELEASE_SHA}" \
  --body "Update the prod global manager config to use pipeline image tag ${RELEASE_SHA}."

gh pr merge -R "$REPO" --squash --auto --delete-branch
```

Apply the global config through the global deploy workflow:

```bash
gh workflow run -R "$REPO" universal-deploy-global.yaml \
  --ref main \
  -f env=prod \
  -f use_apply=false

gh workflow run -R "$REPO" universal-deploy-global.yaml \
  --ref main \
  -f env=prod \
  -f use_apply=true
```

Important runtime note:

- The global config change updates what the manager uses for new or reconciled pipelines.
- Existing running collector deployments may still need a manual image rollout if immediate production cutover is required.

The current list of globally managed pipelines is in:

- `global/deployment/kustomize/envs/prod/configmap.yaml`

Use a command like this for existing prod collectors:

```bash
export NEW_IMAGE="557537366020.dkr.ecr.us-west-2.amazonaws.com/tidbcloud/metrics-integration-pipeline:${RELEASE_SHA}"
ids=(20 21 28 36)

for id in "${ids[@]}"; do
  ns="pipeline-${id}"
  deploy="pipeline-${id}-collector"
  kubectl -n "$ns" set image deployment/"$deploy" pipeline="$NEW_IMAGE"
  kubectl -n "$ns" rollout status deployment/"$deploy"
done
```

If a pipeline runs in GCP or Azure, use the matching registry path for `NEW_IMAGE`.

Do not use `global/deployment/scripts/replace_all_pipeline_image.sh` as-is for prod without checking it first.
It currently hardcodes a non-prod ECR registry.

### 5. Prometheus Pull Pipeline Prod Release

For a normal release, this image now auto-builds from the `release/v*` tag.
Use `workflow_dispatch` only when you need to rerun it or build from a hotfix branch:

```bash
gh workflow run -R "$REPO" prod-ci-prometheus.yaml --ref "$TAG"
```

Sync the Prometheus image to GCP and Azure:

```bash
export PROM_SOURCE_IMAGE="557537366020.dkr.ecr.us-west-2.amazonaws.com/tidbcloud/metrics-integration-prometheus:${RELEASE_SHA}"

gh workflow run -R "$REPO" universal-sync-multicloud-image.yaml --ref main \
  -f source-image="$PROM_SOURCE_IMAGE" \
  -f dest-image-name=metrics-integration/metrics-integration-prometheus \
  -f env=prod \
  -f source-cloud-providers=aws \
  -f dest-cloud-providers=gcp,azure
```

Edit:

- `pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/aws/kustomization.yaml`
- `pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/gcp/kustomization.yaml`
- `pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/azure/kustomization.yaml`

Set:

- each `images[].newTag` to `RELEASE_SHA`

Then open and merge the PR:

```bash
git checkout -b release/prometheus-"${RELEASE_SHA:0:12}"
$EDITOR pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/aws/kustomization.yaml
$EDITOR pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/gcp/kustomization.yaml
$EDITOR pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/azure/kustomization.yaml
git add pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/aws/kustomization.yaml
git add pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/gcp/kustomization.yaml
git add pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/azure/kustomization.yaml
git commit -m "release: bump prod prometheus images to ${RELEASE_SHA}"
git push -u origin HEAD

gh pr create -R "$REPO" \
  --base main \
  --title "release: bump prod prometheus images to ${RELEASE_SHA}" \
  --body "Update AWS, GCP, and Azure prod Prometheus integration images to ${RELEASE_SHA}."

gh pr merge -R "$REPO" --squash --auto --delete-branch
```

Deploy from `main`:

```bash
gh workflow run -R "$REPO" prod-deploy-prometheus-regional.yaml \
  --ref main \
  -f use_apply=false

gh workflow run -R "$REPO" prod-deploy-prometheus-regional.yaml \
  --ref main \
  -f use_apply=true
```

Minimum production checks:

```bash
kubectl -n prometheus-integration get deploy,pods,ingress
kubectl -n prometheus-integration rollout status deployment/prom-integration
```