# Existing Pipeline Collector Image Rollout Plan (Prod)

Date: 2026-05-22 (Friday Night, CST)
Owner: <release-owner>
Change Type: Existing pipeline collector image rollout only (no global config apply in this run)
Workflow: `Prod Rollout Existing Pipeline Image`
File: `.github/workflows/prod-rollout-existing-pipeline-image.yaml`

## 1) Goal and Scope

Goal:
- Roll out a new pipeline collector image to all existing prod pipeline deployments across all clouds/regions.

Scope:
- In scope: existing `pipeline-*` namespaces with collector deployment (`collector` or `pipeline-<id>-collector`).
- Out of scope: global config rollout, new pipeline creation, unrelated services.

Special requirement:
- In AWS rollout sequence, `ap-northeast-1` must be released last.

## 2) Required Inputs

- `image_tag` (required): `<TO_FILL>`
- `image_repo_override` (optional): leave empty unless emergency repo override is needed.
- `pipeline_id_allowlist` (optional): empty means all existing pipelines.
- `pipeline_id_blocklist` (optional): comma-separated IDs to exclude.
- `rollout_timeout`: default `10m`.

## 3) Safety Guards

- Always run dry-run first: `apply=false`.
- Use cloud + region selectors to control blast radius.
- Region-by-region apply for AWS (do not use `aws/all` apply in one run for this release).
- Keep rollback image tag ready before first apply.

Rollback image tag:
- `<TO_FILL_PREVIOUS_GOOD_TAG>`

## 4) Pre-checklist (T-30 min)

- [ ] Confirm target `image_tag` exists in production registry path.
- [ ] Confirm workflow on `main` is available and dispatchable.
- [ ] Confirm release owner has Actions permissions.
- [ ] Confirm no concurrent collector rollout job is running.
- [ ] Confirm rollback tag and command are prepared.

## 5) Execution Plan (Friday Night)

Time window (suggested):
- 20:00-20:20 CST: dry-run validation
- 20:20-22:30 CST: staged apply
- 22:30-23:00 CST: final verification and summary

### Phase A: Dry-run validation (all clouds)

Run workflow with:
- `env=prod`
- `cloud=all`
- `region=all`
- `image_tag=<TO_FILL>`
- `apply=false`
- `pipeline_id_allowlist=`
- `pipeline_id_blocklist=`

Expected:
- Reports generated per cloud/region.
- No cluster write.
- No unexpected authentication or target-discovery errors.

### Phase B: Apply by cloud/region

Run apply in controlled batches.

1) GCP (all regions)
- `cloud=gcp`, `region=all`, `apply=true`

2) Azure (all regions)
- `cloud=azure`, `region=all`, `apply=true`

3) Alicloud (all regions)
- `cloud=alicloud`, `region=all`, `apply=true`

4) AWS (region-by-region, `ap-northeast-1` last)

Recommended AWS order:
1. `ap-southeast-3`
2. `ap-northeast-2`
3. `ap-south-1`
4. `ap-southeast-1`
5. `eu-central-1`
6. `us-east-1`
7. `us-west-2`
8. `ap-southeast-5`
9. `ap-east-1`
10. `eu-west-1`
11. `af-south-1`
12. `ap-northeast-3`
13. `ap-southeast-2`
14. `ap-southeast-7`
15. `sa-east-1`
16. `us-east-2`
17. `ap-northeast-1`  <-- LAST

For each AWS region run:
- `cloud=aws`
- `region=<region>`
- `apply=true`
- same `image_tag` and list controls

Gate between regions:
- Continue only if current region reports no critical rollout failures.

## 6) Verification Checklist (each run)

- [ ] Workflow run succeeded.
- [ ] `rollout-report-<cloud>-<region>` artifact exists.
- [ ] Failures are triaged (not ignored).
- [ ] Sample namespaces show new image tag in deployment.
- [ ] No sustained collector error spike after rollout.

## 7) Rollback Plan

Trigger rollback if:
- Widespread rollout failures
- Sustained ingestion break
- Critical regression confirmed

Rollback action:
- Re-run same workflow/selector set with:
  - `image_tag=<TO_FILL_PREVIOUS_GOOD_TAG>`
  - `apply=true`
- Use same staged order (for AWS keep controlled region-by-region).

## 8) Run Log Template

Fill during execution:

- Dry-run run URL: <TO_FILL>
- GCP apply run URL: <TO_FILL>
- Azure apply run URL: <TO_FILL>
- Alicloud apply run URL: <TO_FILL>
- AWS runs:
  - ap-southeast-3: <TO_FILL>
  - ap-northeast-2: <TO_FILL>
  - ap-south-1: <TO_FILL>
  - ap-southeast-1: <TO_FILL>
  - eu-central-1: <TO_FILL>
  - us-east-1: <TO_FILL>
  - us-west-2: <TO_FILL>
  - ap-southeast-5: <TO_FILL>
  - ap-east-1: <TO_FILL>
  - eu-west-1: <TO_FILL>
  - af-south-1: <TO_FILL>
  - ap-northeast-3: <TO_FILL>
  - ap-southeast-2: <TO_FILL>
  - ap-southeast-7: <TO_FILL>
  - sa-east-1: <TO_FILL>
  - us-east-2: <TO_FILL>
  - ap-northeast-1 (last): <TO_FILL>

Post-release summary:
- Total targets: <TO_FILL>
- Success: <TO_FILL>
- Failed: <TO_FILL>
- Skipped: <TO_FILL>
- Follow-ups: <TO_FILL>