# Prod Hotfix And Rollback

## Prod Hotfix Notes

For a hotfix, the CLI-first path is:

```bash
git checkout -b hotfix/<topic> origin/main
# apply the code fix
git push -u origin HEAD

gh workflow run -R "$REPO" prod-ci-global.yaml --ref hotfix/<topic>
gh workflow run -R "$REPO" prod-ci-regional.yaml --ref hotfix/<topic>
gh workflow run -R "$REPO" prod-ci-pipeline.yaml --ref hotfix/<topic>
gh workflow run -R "$REPO" prod-ci-prometheus.yaml --ref hotfix/<topic>
```

After the production fix is stable:

1. merge the hotfix back to `main`
2. create a release tag from the merged commit
3. create a GitHub Release so production history remains traceable

## Rollback Notes

Rollback is also terminal-first:

1. identify the previous good SHA
2. revert the manifest files to that SHA
3. open and merge a rollback PR
4. rerun the same deploy workflow with diff first, then apply
5. if existing OTel collectors were manually updated, set their image back to the previous good registry and tag

Useful checks for `prod`:

```bash
gh release view -R "$REPO" "$PREV_TAG"
git show "$PREV_TAG":global/deployment/kustomize/envs/prod/kustomization.yaml
git show "$PREV_TAG":global/deployment/kustomize/envs/prod/configmap.yaml
```

For `dev`, replace the prod tag lookup, manifest paths, and deploy workflows with their dev equivalents.

## Practical Recommendation

For a full `dev` release, use this order:

1. resolve the target SHA on `main`
2. wait for or trigger the four dev image builds
3. merge the dev manifest PRs
4. deploy global dev
5. deploy regional dev
6. deploy Prometheus dev
7. reconcile live OTel collectors only if immediate cutover is needed

For a full `prod` release, use this order:

1. create the release tag and GitHub Release
2. wait for global, regional, and OTel pipeline image builds
3. run the Prometheus image build
4. sync regional, pipeline, and Prometheus images to GCP and Azure
5. merge manifest PRs
6. deploy global
7. deploy regional
8. deploy Prometheus regional service
9. perform live pipeline collector updates only if the release requires immediate cutover

This order keeps the repo state, image state, and production state aligned, while remaining fully operable from `git`, `gh`, and terminal-based cluster tooling.