# Region Lifecycle Analysis

Date: 2026-03-28

## Scope

This document focuses on the region lifecycle driven by `tidbcloud/auto-deploy`, especially:

- how a new region is opened
- how application delivery is delegated to `app-delivery`
- how infra follow-up changes can be delegated to `infra-cd`
- how a region is destroyed

Primary evidence is from the AWS dedicated region path because it has the clearest complete chain in local source:

- `auto-deploy/.github/workflows/deploy-aws-region.yml`
- `auto-deploy/etc/deploy-jobs/aws-region-control/`
- `auto-deploy/internal/job/runner.go`
- `auto-deploy/internal/argocd/argocd.go`
- `auto-deploy/.github/workflows/destroy-aws-region-k8s.yml`
- `auto-deploy/internal/destroy-aws-region-k8s.go`
- `app-delivery/.github/workflows/deploy-new-region-*.yaml`
- `app-delivery/.github/workflows/destroy-region-application.yaml`

Where this document extrapolates from the AWS path to other clouds or next-gen flows, it marks that as inference.

## Executive Summary

`auto-deploy` is not just a collection of workflows. It is a generic region orchestration engine.

Confirmed facts:

- workflows such as `deploy-aws-region.yml` call a compiled `deploy-cli`
- the CLI receives a `jobs-path`
- that path points to a directory of job definitions
- the job runner loads both jobs and `finalizers/`
- each job can depend on other jobs and can attach a finalizer
- job outputs are persisted to object storage and then fed into later jobs
- some jobs or finalizers trigger GitHub workflows in `app-delivery`
- some finalizers patch and release `infra-cd`
- destroy flow runs finalizers first, then unregisters the cluster from ArgoCD, then destroys the cluster

This means the region lifecycle is a staged control loop:

`metadata -> infrastructure jobs -> application registration and deployment -> cross-repo follow-up -> steady state`

and the reverse on destroy:

`application removal -> cluster unregister -> infrastructure teardown -> region registry cleanup`

## Orchestration Model

### Confirmed Facts

The CLI commands describe the model directly:

- jobs live in a `jobs-path` folder
- job options are passed separately
- the runner reads normal jobs and `finalizers/`
- the runner computes dependencies between jobs
- finalizers are attached to the associated jobs and also ordered based on job dependencies
- the runner renders a common job input template and injects `jobOutput` and `jobOptions`

The generated common job input includes:

- `env`
- `globalEnv`
- `name`
- `cloudProvider`
- `region`
- `jobId`
- `terraformStateBucket`
- `imageRegistry`
- `k8s.*`
- `aws.controlplaneAccount`
- `aws.dataplaneAccounts`

This is why later scripts can read a previous job output such as `jobOutput.iam` and continue the pipeline.

### Key Files

- `~/coding/auto-deploy/cmd/deploy-cli/commands/deploy-aws-region.go`
- `~/coding/auto-deploy/internal/job/runner.go`
- `~/coding/auto-deploy/internal/job/job_input.yml.tpl`

## Region Open Flow

### Phase 0: Workflow Entry

Confirmed facts:

- `deploy-aws-region.yml` is a `workflow_dispatch` entrypoint
- inputs include `tidb-cloud-instance`, `region-name`, `aws-region`, `meta-uri`, and `advance-options`
- the workflow builds `deploy-cli`
- the workflow parses advanced options and calls:

`./deploy-cli deploy-aws-region --jobs-path=etc/deploy-jobs/aws-region-control ...`

Implication:

- the workflow itself is only the outer shell
- the real lifecycle logic lives in the deploy jobs directory and the runner

### Phase 1: Region Metadata and Runtime Context

Confirmed facts:

- `auto-deploy/meta/dev.yaml`, `staging.yaml`, and `prod.yaml` store per-region metadata
- this metadata contains cluster names, cloud region, OIDC issuer, public IPs, output bucket, tf-state bucket, dataplane accounts, and job options for downstream services
- examples include regional DNS settings, auth0 settings, WAF settings, cluster-controller settings, and observability settings

Implication:

- `auto-deploy` owns the operational metadata for a region bring-up
- it is not just invoking Terraform blindly

### Phase 2: Infrastructure and Platform Job Graph

Representative AWS dedicated region jobs are defined under:

- `auto-deploy/etc/deploy-jobs/aws-region-control/`

Confirmed facts from job annotations:

- `iam` has no dependencies
- `external-secrets-operator` depends on `iam`
- `cert-manager` depends on `iam` and `gateway-api-crds`
- `crossplane-aws-provider` depends on `iam` and `victoria-metrics-operator`
- `kong-gateway` depends on `iam`, `aws-load-balancer-controller`, and `cert-manager`
- `o11y-agent` depends on `iam` and `victoria-metrics-operator`
- `regional-server` depends on `iam`, `external-secrets-operator`, `kong-gateway`, `victoria-metrics-operator`, `crossplane-aws-provider`, and `o11y-agent`
- `tcidm` depends on `iam` and `cluster-service`
- `dp-provider` depends on `iam`, `cluster-service`, and `dp-network`

This gives a concrete stage model:

1. bootstrap: `iam`
2. platform prerequisites: secret operator, cert-manager, gateway CRDs, prometheus, crossplane, o11y
3. regional control-plane services: `regional-server`, `cluster-service`, `tcidm`, `dp-provider`
4. post-provisioning finalizers

### Phase 3: Application Delivery Handoff

Confirmed facts:

- the job script `aws-region-control/crossplane-aws-provider` invokes `gh workflow run` against `tidbcloud/app-delivery`
- it triggers:
  - `deploy-new-region-upbound-crossplane.yaml`
  - `deploy-new-region-crossplane-aws-crds.yaml`
  - `deploy-new-region-crossplane-aws-provider.yaml`
- the job script `aws-region-control/regional-server` reads IAM outputs and other region options needed to build application inputs
- the corresponding `app-delivery` workflows create pull requests, trigger checks, and enable automerge
- `deploy-new-region-regional-server.yaml` edits manifests and `app-registry/regional-server.yaml`, then creates a PR and enables squash automerge

Implication:

- `auto-deploy` does not directly edit `app-delivery` files in-place
- it delegates workload registration to `app-delivery` via workflow-triggered PR automation

This is an important boundary:

- `auto-deploy` owns orchestration and runtime parameters
- `app-delivery` remains the authoritative manifest repo

### Phase 4: Cross-Repo Infra Follow-Up

Confirmed facts:

- `aws-region-control/finalizers/dp-network` can clone `infra-cd`
- it edits `projects/apiserver/Pulumi.prod.yaml`
- it creates and merges PRs to both `main` and a release branch
- it creates a new tag to trigger the production GitHub Action
- it then waits for `gardener-arch-prod-pipeline` to succeed

Implication:

- region open is not complete at the application layer alone
- some regional actions require control-plane infra allowlist or network adjustments in `infra-cd`

This is the strongest concrete evidence that the region lifecycle spans multiple repositories.

## Region Destroy Flow

### Phase 0: Destroy Entry

Confirmed facts:

- `destroy-aws-region-k8s.yml` is a `workflow_dispatch` entrypoint
- it calls:

`./deploy-cli destroy-aws-region-k8s ...`

### Phase 1: Finalizer Jobs Run First

Confirmed facts from `destroy-aws-region-k8s.go`:

- it loads region metadata
- it verifies the region type and cloud
- it runs `job.RunFinalizerJobs(region)` before unregistering the cluster

This ordering matters.

The system intentionally removes higher-level application and IAM attachments before tearing down the cluster itself.

### Phase 2: Application Removal via `app-delivery`

Confirmed facts:

- `finalizers/regional-server` triggers `destroy-region-application.yaml` in `app-delivery` for `regional-server`
- after waiting for relevant services and providerconfigs to disappear, it triggers another destroy for `regional-server-crds`
- `finalizers/crossplane-aws-provider` triggers destroys for:
  - `crossplane-aws-provider`
  - `crossplane-aws-crds`
  - `upbound-crossplane`
- `app-delivery/.github/workflows/destroy-region-application.yaml` removes region manifests, creates a PR, runs checks, and only auto-approves/automerge for non-prod

Implication:

- destroy flow reverses the same ownership boundary used on create
- `auto-deploy` orchestrates
- `app-delivery` still owns manifest deletion

### Phase 3: ArgoCD Unregister

Confirmed facts:

- after finalizer jobs complete, `destroy-aws-region-k8s.go` calls `argocd.UnregisterK8s(&region.K8s)`
- `argocd.UnregisterK8s` first waits until all ArgoCD applications targeting that cluster are deleted
- only then does it delete the ArgoCD cluster secret

Implication:

- cluster unregistration is intentionally blocked on application cleanup
- this is a safety barrier that prevents orphaned ArgoCD application records from pointing to a disappearing cluster

### Phase 4: Cluster Destruction and Registry Cleanup

Confirmed facts:

- after ArgoCD unregister, the cloud provider destroys the Kubernetes cluster
- region job outputs are cleared
- metadata is saved back
- the global client deletes region records from the control plane
- next-gen flow additionally deletes regional config center before deleting the region record

Implication:

- region destroy is both an infrastructure teardown and a control-plane deregistration flow

## Finalizer Semantics

Finalizers are a key design choice in this system.

Confirmed facts:

- jobs may declare `tidbcloud.com/finalizer`
- finalizer jobs are separate Kubernetes jobs loaded from `finalizers/`
- a finalizer may not itself declare another finalizer or explicit `depends-on`
- the runner derives finalizer ordering from the dependencies of the associated normal jobs

Why this matters:

- main jobs handle provisioning / deployment
- finalizers handle cleanup / reverse operations
- ordering is kept consistent with the forward dependency graph

This is one of the clearest signals that `auto-deploy` is designed as a lifecycle orchestrator rather than a one-shot installer.

## Cross-Repo Control Boundaries

### `auto-deploy`

Owns:

- lifecycle orchestration
- region metadata
- cluster registration/unregistration in ArgoCD
- handoff timing to downstream repos

Does not own:

- canonical application manifests
- canonical runtime config files
- canonical shared infra code in other repos

### `app-delivery`

Owns:

- region workload manifests
- application registration into target clusters
- manifest removal during destroy

Does not own:

- the initial decision to create/destroy a region
- cluster registration in ArgoCD

### `infra-cd`

Owns:

- infra API and related control-plane infra definitions
- account/project stack changes
- some network/control-plane follow-up changes required by region rollout

Does not own:

- the orchestration of per-region workload bring-up

## Failure Points

The main failure points are structurally predictable:

1. metadata and workflow input mismatch
   Example: logical region names, account roles, or advanced options do not match actual region state.

2. job graph partial success
   Since jobs and finalizers are staged and stateful, a failure can leave valid outputs for some jobs and missing outputs for later jobs.

3. cross-repo PR automation failure
   `auto-deploy` depends on `gh workflow run`, PR creation, checks, and automerge in `app-delivery` and sometimes `infra-cd`.

4. ArgoCD cleanup lag
   Destroy flow explicitly waits for applications to disappear before unregistering the cluster.

5. production-specific release friction
   Some destroy/create automations are looser in non-prod and stricter in prod.

## Rollback Characteristics

### Region Open Rollback

There is no single rollback button.

Rollback is layered:

1. stop or avoid further `app-delivery` PR merges
2. run destroy-region-application for already registered apps if needed
3. run region destroy flow through `auto-deploy`
4. revert any follow-up `infra-cd` changes if they were already applied

Relative difficulty: High to Very High

### Region Destroy Rollback

Destroy is partially reversible only before infrastructure is fully gone.

Once these are removed, recovery requires a fresh open flow:

- ArgoCD cluster registration
- application manifests
- IAM roles / cluster resources
- region registry entries

Relative difficulty: Very High

## Inference: Other Clouds and Other Region Modes

This section is inference from structure, not directly from one exact end-to-end path.

Evidence:

- there are parallel job directories for:
  - `aws-byoc-region-control`
  - `aws-nextgen-region-control`
  - `azure-region-control`
  - `o11y-aws-region-control`
  - `o11y-azure-region-control`
  - `alicloud-nextgen-region-control`
- deploy CLI commands for AWS, Azure, AliCloud, and o11y variants all accept `jobs-path` and `job-options`

Reasonable inference:

- the same orchestration pattern is reused across clouds and deployment types
- differences likely live in the job set, metadata schema, and downstream workflows, not in the fundamental orchestration model

Confidence: medium

## Key Conclusions

1. `auto-deploy` is the lifecycle controller for regions, not merely a utility repo.
2. The lifecycle is intentionally multi-repo: `auto-deploy` orchestrates, `app-delivery` owns manifest state, and `infra-cd` may be patched as part of control-plane completion.
3. ArgoCD registration is a first-class lifecycle step and a strict destroy barrier.
4. Finalizers are the core reverse-order safety mechanism.
5. Region rollout is the highest-coupling deployment path in this system.

## Evidence Index

- `~/coding/auto-deploy/.github/workflows/deploy-aws-region.yml`
- `~/coding/auto-deploy/.github/workflows/destroy-aws-region-k8s.yml`
- `~/coding/auto-deploy/.github/workflows/register-new-region.yml`
- `~/coding/auto-deploy/cmd/deploy-cli/commands/deploy-aws-region.go`
- `~/coding/auto-deploy/internal/job/runner.go`
- `~/coding/auto-deploy/internal/job/job_input.yml.tpl`
- `~/coding/auto-deploy/internal/argocd/argocd.go`
- `~/coding/auto-deploy/internal/destroy-aws-region-k8s.go`
- `~/coding/auto-deploy/etc/deploy-jobs/aws-region-control/*.yaml`
- `~/coding/auto-deploy/etc/deploy-jobs/aws-region-control/finalizers/*.yaml`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/regional-server`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/crossplane-aws-provider`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/finalizers/regional-server`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/finalizers/crossplane-aws-provider`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/finalizers/dp-network`
- `~/coding/app-delivery/.github/workflows/deploy-new-region-regional-server.yaml`
- `~/coding/app-delivery/.github/workflows/destroy-region-application.yaml`