# Environment Overview This guide covers both `dev` and `prod` release work in this repository. It keeps environment-specific commands in one place, while sharing the same component model and CLI-first workflow. It is written for two operating modes: - `Manual human flow`: a human engineer can use GitHub web pages, Actions pages, and PR pages. - `Agent CLI flow`: an agent such as Codex uses `git`, `gh`, local file edits, and cluster CLI tools to complete the release end to end without relying on the GitHub web UI. The scope covers: - global control plane - regional control plane - OTel push pipeline for Datadog / New Relic - Prometheus pull pipeline - release-related production operations and rollback notes ## Build workflows - Global control plane: - dev: `.github/workflows/dev-ci-global.yaml` - prod: `.github/workflows/prod-ci-global.yaml` - Regional control plane: - dev: `.github/workflows/dev-ci-regional.yaml` - prod: `.github/workflows/prod-ci-regional.yaml` - OTel push pipeline: - dev: `.github/workflows/dev-ci-pipeline.yaml` - prod: `.github/workflows/prod-ci-pipeline.yaml` - Prometheus pull pipeline: - dev: `.github/workflows/dev-ci-prometheus.yaml` - prod: `.github/workflows/prod-ci-prometheus.yaml` ## Deploy workflows - Global control plane: - dev: `.github/workflows/universal-deploy-global.yaml` with `env=dev` - prod: `.github/workflows/universal-deploy-global.yaml` with `env=prod` - Regional control plane: - dev: `.github/workflows/dev-deploy-regional.yaml` - prod: `.github/workflows/prod-deploy-regional.yaml` - Prometheus regional service: - dev: `.github/workflows/dev-deploy-prometheus-regional.yaml` - prod: `.github/workflows/prod-deploy-prometheus-regional.yaml` ## Files That Carry Image Versions - Global control plane image: - dev: `global/deployment/kustomize/envs/dev/kustomization.yaml` - prod: `global/deployment/kustomize/envs/prod/kustomization.yaml` - Regional control plane images: - dev: - `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-aws/kustomization.yaml` - `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-gcp/kustomization.yaml` - `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-azure/kustomization.yaml` - prod: - `regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-aws/kustomization.yaml` - `regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-gcp/kustomization.yaml` - `regional/deployment/kustomize/prod/metrics-integration-regional-server/envs/prod-azure/kustomization.yaml` - OTel push pipeline image used by the global manager: - dev: `global/deployment/kustomize/envs/dev/configmap.yaml` - prod: `global/deployment/kustomize/envs/prod/configmap.yaml` - Prometheus pull pipeline images: - dev: - `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/aws/kustomization.yaml` - `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/gcp/kustomization.yaml` - `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/azure/kustomization.yaml` - prod: - `pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/aws/kustomization.yaml` - `pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/gcp/kustomization.yaml` - `pipeline/deployment/prom-app-delivery/prod/prom-regional-service/cloud-variants/azure/kustomization.yaml` ## Coverage Notes - `dev` regional deploy workflow currently covers AWS `us-west-2` only. - `dev` Prometheus deploy workflow currently covers AWS `us-west-2` only. - `dev` GCP and Azure manifests exist and render, but this repo does not currently wire them into a dev deploy workflow. - OTel pipeline versioning is environment-specific and is maintained in the global configmap for each environment. ## Recent Dev Rollout Record Target SHA: - `8a156e2ef775a6982e4bfb45ed46ce5cd09efe9d` Dev-only version files updated in this change: - `global/deployment/kustomize/envs/dev/kustomization.yaml` - `global/deployment/kustomize/envs/dev/configmap.yaml` - `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-aws/kustomization.yaml` - `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-gcp/kustomization.yaml` - `regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-azure/kustomization.yaml` - `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/aws/kustomization.yaml` - `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/gcp/kustomization.yaml` - `pipeline/deployment/prom-app-delivery/dev/prom-regional-service/cloud-variants/azure/kustomization.yaml` What changed: - global control plane dev image tag was bumped to `8a156e2ef775a6982e4bfb45ed46ce5cd09efe9d` - regional control plane dev image tags were aligned to the same SHA for AWS, GCP, and Azure kustomizations - OTel pipeline dev image tag was bumped in `global/deployment/kustomize/envs/dev/configmap.yaml` - Prometheus regional dev image tags were aligned to the same SHA in the cloud-variant kustomizations Validated locally: - `kubectl kustomize global/deployment/kustomize/envs/dev` - `kubectl kustomize regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-aws` - `kubectl kustomize regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-gcp` - `kubectl kustomize regional/deployment/kustomize/dev/metrics-integration-regional-server/envs/dev-azure` - `kubectl kustomize pipeline/deployment/prom-app-delivery/dev/prom-regional-service/envs/dev-aws-us-west-2` - `kubectl kustomize pipeline/deployment/prom-app-delivery/dev/prom-regional-service/envs/dev-aws-us-east-1` - `kubectl kustomize pipeline/deployment/prom-app-delivery/dev/prom-regional-service/envs/dev-gcp-us-west1` - `kubectl kustomize pipeline/deployment/prom-app-delivery/dev/prom-regional-service/envs/dev-gcp-us-west2` - `kubectl kustomize pipeline/deployment/prom-app-delivery/dev/prom-regional-service/envs/dev-azure-eastus` - `kubectl kustomize pipeline/deployment/prom-app-delivery/dev/prom-regional-service/envs/dev-azure-eastus2` - `kubectl kustomize pipeline/deployment/prom-app-delivery/dev/prom-regional-service/envs/dev-azure-southeastasia` Runnable dev release chain confirmed in this repo: 1. Build images for the target SHA with: - `dev-ci-global.yaml` - `dev-ci-regional.yaml` - `dev-ci-pipeline.yaml` - `dev-ci-prometheus.yaml` 2. Merge the dev manifest bump for the four components above. 3. Deploy global dev with `universal-deploy-global.yaml`: - first `env=dev,use_apply=false` - then `env=dev,use_apply=true` 4. Deploy regional dev with `dev-deploy-regional.yaml`. - current workflow coverage in this repo is AWS dev `us-west-2` only - GCP and Azure image tags are aligned in kustomization, but there is no matching dev deploy workflow here today 5. Deploy Prometheus dev with `dev-deploy-prometheus-regional.yaml`. - current workflow coverage in this repo is AWS dev `us-west-2` only - other dev env kustomizations render correctly, but are not wired into a dev deploy workflow here today Important note for OTel pipeline: - the OTel pipeline image version is maintained in `global/deployment/kustomize/envs/dev/configmap.yaml` - applying `universal-deploy-global.yaml` updates the global manager config used for new or reconciled pipeline deployments - if an existing dev collector must switch immediately, reconcile or roll that collector after the global config change Guardrail: - no prod manifest path was changed in this dev update ## Rules of Engagement - `dev` releases normally use a target commit SHA from `main`, not a release tag. - Use `release/v*` tags for normal `prod` releases. - Manual release work may use the GitHub web UI. - Agent-driven release work should use `git`, `gh`, and cluster CLI tools instead of depending on the web UI. - For normal `prod` releases, build images from a release tag. - For `prod` hotfixes, `workflow_dispatch` on a hotfix branch is allowed, but merge the hotfix back to `main` after production is stable. - For deploy workflows, run the workflow from `main` after the manifest PR is merged, so the cluster deploy uses the merged repo state.