# Deployment Structure Analysis

Date: 2026-03-28

## Scope

This document analyzes the deployment structure represented in the knowledge base and the local source trees under `~/coding/`.

Primary scope:

- `tidbcloud/aws-shared-cd`
- `tidbcloud/app-delivery`
- `tidbcloud/auto-deploy`
- `tidbcloud/infra-cd`
- `tidbcloud/oneconfig`

Supplementary scope:

- `tidbcloud/byoc-account-setup`

Excluded by repository filter:

- `tidbcloud/db9-cd`

Evidence for the exclusion is `./.local/review-history/repos-filtered.txt`.

## Executive Summary

The deployment structure is split across four main control layers plus one supplementary customer-side setup layer:

1. `infra-cd` and `aws-shared-cd` define cloud accounts, shared clusters, infra APIs, and core platform components.
2. `auto-deploy` orchestrates region bring-up and teardown, registers clusters into ArgoCD, and triggers follow-up changes in other deployment repositories.
3. `app-delivery` is the GitOps application delivery layer. It stores ArgoCD-managed manifests, overlays, and app registrations for services and region components.
4. `oneconfig` is the GitOps runtime configuration layer. It applies per-service config files into Kubernetes configmaps and supports hot reload through the config service SDK.
5. `byoc-account-setup` is a supplementary customer-side bootstrap repository for BYOC accounts. It is not part of the main internal control loop, but it is part of the overall deployment surface.

This means there is no single "deployment repo". The real deployment path is multi-repo:

`infra definition repo -> orchestration repo -> app manifest repo / config repo -> ArgoCD or Jenkins -> target clusters / configmaps -> running services`

## Repository Roles

| Repo | Primary role | Control layer | Main trigger model | Main target |
| --- | --- | --- | --- | --- |
| `infra-cd` | Infra API and multi-account/project management | Cloud account + infra platform | GitHub Actions and Jenkins | AWS accounts, GCP projects, infra API stacks |
| `aws-shared-cd` | Shared free-tier cluster and regional platform definition | Shared cluster/platform | GitHub Actions on PR/push and release workflows | EKS/ACK shared environments |
| `auto-deploy` | Region orchestration and cross-repo automation | Orchestration/control plane | GitHub Actions `workflow_dispatch` | New/destroyed regions, ArgoCD registration, follow-up repo changes |
| `app-delivery` | Service manifests and cluster application registration | Workload delivery | PR merge plus ArgoCD continuous sync | Kubernetes workloads in dev/staging/prod clusters |
| `oneconfig` | Runtime config source of truth | Runtime configuration | PR merge plus Jenkins apply | Kubernetes configmaps and hot-reloaded services |
| `byoc-account-setup` | Customer-side BYOC bootstrap | Customer bootstrap | Manual execution by customer | Bastion, IAM roles, optional security checks |

## Layered Architecture

### Layer 1: Cloud Account and Project Foundation

Owned mainly by `infra-cd`.

Confirmed facts:

- `infra-cd` manages `projects/aws-accounts`, `projects/gcp-projects`, `projects/apiserver`, and `projects/infra-api`.
- `infra-cd` uses Pulumi stacks `dev`, `staging`, and `prod`.
- `infra-cd` applies dev and staging from `main`, while at least the multi-account production path is separated through tags matching `multi-account-prod/*`.

Implication:

- `infra-cd` is the authoritative repo for shared account/project state and core infra API deployment.
- Other repos depend on its outputs or on infrastructure it makes reachable.

Evidence:

- `~/coding/infra-cd/README.md`
- `~/coding/infra-cd/projects/*`
- `~/coding/infra-cd/.github/workflows/multi-account.yml`
- `~/coding/infra-cd/.github/workflows/multi-account-prod-cd.yaml`
- `~/coding/infra-cd/doc/gha.md`

### Layer 2: Shared Cluster and Platform Definition

Owned mainly by `aws-shared-cd`, with `infra-cd` also covering platform-adjacent pieces.

Confirmed facts:

- `aws-shared-cd` stores `pool/eks-freetier` and `pool/ack-freetier`.
- It uses Pulumi stack files such as `Pulumi.dev-...yaml`, `Pulumi.staging-...yaml`, and `Pulumi.prod-...yaml`.
- PRs and pushes to `main` and `release-*` branches trigger the freetier workflow.
- The branch mapping is explicit for dev and staging.
- Production release is split out through release workflows that update `aws-shared-cd-prod`, not directly through the normal dev/staging path.

Implication:

- `aws-shared-cd` is the authoritative definition for shared free-tier cluster/platform topologies in non-prod.
- Production for this domain has an extra indirection and should be treated as a separate release surface.

Evidence:

- `~/coding/aws-shared-cd/readme.md`
- `~/coding/aws-shared-cd/.github/workflows/freetier.yml`
- `~/coding/aws-shared-cd/.github/workflows/release.yml`
- `~/coding/aws-shared-cd/.github/workflows/release-alicloud.yml`
- `~/coding/aws-shared-cd/pool/`

### Layer 3: Region Orchestration

Owned by `auto-deploy`.

Confirmed facts:

- `auto-deploy` is not a passive config repo. It contains a Go CLI, GitHub Actions workflows, Terraform modules, environment metadata, and deploy job definitions.
- Region operations are manual or bot-triggered `workflow_dispatch` jobs such as `deploy-aws-region.yml`, `register-new-region.yml`, destroy flows, BYOC flows, and next-gen flows.
- The repo keeps environment metadata in `meta/dev.yaml`, `meta/staging.yaml`, `meta/prod.yaml`, and o11y variants.
- The repo directly registers clusters into ArgoCD by creating/updating ArgoCD cluster secrets.
- The repo directly triggers or edits other deployment repos during multi-step region workflows.

Confirmed cross-repo couplings:

- `auto-deploy` references `tidbcloud/app-delivery` in deploy-job scripts for region application deployment and destruction.
- `auto-deploy` updates `infra-cd/projects/apiserver/Pulumi.prod.yaml` in one finalizer path and then creates and merges PRs plus tags to trigger downstream infra deployment.

Implication:

- `auto-deploy` is the orchestration brain for dedicated / next-gen / BYOC regional rollout, not the source of workload manifests themselves.
- It sits between infrastructure primitives and application GitOps delivery.

Evidence:

- `~/coding/auto-deploy/README.md`
- `~/coding/auto-deploy/.github/workflows/deploy-aws-region.yml`
- `~/coding/auto-deploy/.github/workflows/register-new-region.yml`
- `~/coding/auto-deploy/meta/*.yaml`
- `~/coding/auto-deploy/internal/argocd/argocd.go`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/regional-server`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/finalizers/dp-network`

### Layer 4: Application Delivery

Owned by `app-delivery`.

Confirmed facts:

- `app-delivery` is an ArgoCD manifest repository for TiDB Cloud services.
- It stores service manifests, `base` and `envs/*` overlays, and application registration under `app-registry`.
- ArgoCD continuously synchronizes these manifests to dev, staging, and prod environments.
- `ApplicationSet` files define mappings from environment names to concrete cluster destinations.
- The repository has extensive automation for version bumps, new-region deployment helpers, merge checks, and protection checks.

Environment model:

- Root-level service trees and `app-registry/` act as the default delivery surface.
- There are dedicated `dev/` and `staging/` trees with their own app-registry and service overlays.
- The merge-check workflow explicitly treats prod differently by filtering out `dev/` and `staging/` paths before evaluating production-impacting changes.

Implication:

- `app-delivery` is the authoritative manifest source for what runs in Kubernetes.
- It does not own the cluster itself; it assumes the cluster has already been registered into ArgoCD.

Evidence:

- `~/coding/app-delivery/README.md`
- `~/coding/app-delivery/app-registry/config-center.yaml`
- `~/coding/app-delivery/.github/workflows/merge-check.yaml`
- `~/coding/app-delivery/.github/workflows/deploy-new-region-regional-server.yaml`
- `~/coding/app-delivery/.github/file-filters.yaml`
- `~/coding/app-delivery/dev/`

### Layer 5: Runtime Configuration

Owned by `oneconfig`.

Confirmed facts:

- `oneconfig` maps branches to environments:
  - `dev` -> dev env, `nightly` namespace
  - `staging` -> staging env, `staging` namespace
  - `prod` -> prod env, `prod` namespace
  - `personal/*` -> dev env, personal namespace
- Its Jenkins pipeline resolves namespace from branch name and runs `apply.sh`.
- `apply.sh` converts each top-level service directory into a Kubernetes configmap named `oneconfig-<service>`.
- The config service SDK and Viper remote provider support hot reload from these configmaps.

Implication:

- `oneconfig` is the authoritative source of runtime configuration values, not of workload shape.
- It affects service behavior after deployment without requiring application image changes.

Evidence:

- `~/coding/oneconfig/README.md`
- `~/coding/oneconfig/Jenkinsfile`
- `~/coding/oneconfig/apply.sh`
- `~/coding/oneconfig/*/config.yaml`

## Environment Mapping

### Cross-Repo Mapping Summary

| Repo | Dev | Staging | Prod | Personal / Ephemeral |
| --- | --- | --- | --- | --- |
| `oneconfig` | `dev` branch -> `nightly` namespace | `staging` branch -> `staging` namespace | `prod` branch -> `prod` namespace | `personal/*` -> dev env, custom namespace |
| `aws-shared-cd` | `main` branch + `Pulumi.dev.*.yaml` | `release-*` branch + `Pulumi.staging.*.yaml` | `Pulumi.prod.*.yaml` exists, but release path is mediated by `aws-shared-cd-prod` | create-region PR path exists |
| `infra-cd` | Pulumi `dev` stack | Pulumi `staging` stack | Pulumi `prod` stack, with special production workflow/tag path for at least multi-account | manual workflow dispatch supports selected env |
| `app-delivery` | `dev/` tree and dev ArgoCD | `staging/` tree and staging ArgoCD | root/default delivery surface and prod ArgoCD | new-region workflows create per-region overlays |
| `auto-deploy` | `meta/dev.yaml` and workflow environment `dev` | `meta/staging.yaml` and workflow environment `staging` | `meta/prod.yaml` and workflow environment `prod` | logical region names and specialized workflow inputs create ephemeral-like regions |

### Important Observation

There is no universal environment encoding shared by all repos.

Different repos use different primary keys:

- branch names in `oneconfig`
- branch plus Pulumi stack files in `aws-shared-cd`
- Pulumi stack name and sometimes tags in `infra-cd`
- path and ArgoCD destination name in `app-delivery`
- workflow environment plus metadata YAML in `auto-deploy`

This is a structural characteristic of the current deployment system, not just a documentation gap.

## End-to-End Change Flows

### Flow A: Runtime Configuration Change

1. Change config in `oneconfig/<service>/...`.
2. Open PR against `dev`, `staging`, `prod`, or `personal/*`.
3. Merge.
4. Jenkins resolves the target namespace from the branch.
5. `apply.sh` applies configmaps into the target namespace.
6. Services subscribed through the oneconfig SDK hot reload.

Primary source of truth:

- `oneconfig`

Operational property:

- Fast rollback by revert PR and re-apply.

### Flow B: Application Manifest Change

1. Change service manifests or app registration in `app-delivery`.
2. PR merge-check validates protected files, app name conflicts, and prod-impact conditions.
3. Merge.
4. ArgoCD continuously syncs manifests to the target cluster.
5. Kubernetes resources converge to the desired state.

Primary source of truth:

- `app-delivery`

Operational property:

- Base and app-registry changes can have broad blast radius.

### Flow C: Shared Cluster / Platform Change

1. Change Pulumi code or stack config in `aws-shared-cd`.
2. Open PR to `main` or `release-*`.
3. PR triggers preview.
4. Merge or push triggers apply for matching stack files.
5. For prod release, the release workflow updates `aws-shared-cd-prod`.

Primary source of truth:

- `aws-shared-cd` for dev/staging shared stack definitions
- `aws-shared-cd-prod` for released production state in this domain

Operational property:

- Production path is split and therefore harder to reason about without both repos.

### Flow D: Infra API / Multi-Account Change

1. Change Pulumi code or config in `infra-cd`.
2. PR triggers preview for relevant projects/stacks.
3. Merge to `main` updates dev/staging for supported flows.
4. Production is released through a specialized tag-based or version-bump path depending on the component.

Primary source of truth:

- `infra-cd`

Operational property:

- More controlled than app-delivery because stack updates are explicit and environment-specific.

### Flow E: Open a New Region

1. Trigger `auto-deploy` workflow such as `deploy-aws-region`.
2. The workflow builds the deploy CLI and loads environment-specific metadata and job definitions.
3. `auto-deploy` provisions or configures region infrastructure and cluster metadata.
4. `auto-deploy` registers the cluster into ArgoCD.
5. `auto-deploy` triggers or prepares `app-delivery` changes for region applications.
6. Some finalizers update `infra-cd` or other dependent systems if required.
7. `register-new-region.yml` updates the central region registry through the internal API.

Primary source of truth:

- `auto-deploy` for orchestration state and region metadata
- `app-delivery` for the workloads that will run in the region
- `infra-cd` for infra APIs and some control-plane reachability

Operational property:

- Highest operational complexity and widest failure surface.

## Cross-Repo Dependencies

### Confirmed Dependencies

| Upstream | Downstream | Relationship |
| --- | --- | --- |
| `auto-deploy` | `app-delivery` | region workflows deploy or destroy region applications by invoking app-delivery automation |
| `auto-deploy` | `infra-cd` | some finalizers patch `infra-cd` config and trigger downstream production release steps |
| `auto-deploy` | ArgoCD state | registers and unregisters cluster secrets directly |
| `app-delivery` | target clusters | ArgoCD sync applies manifests to registered clusters |
| `app-delivery` | `oneconfig` consumers | deploys `config-center`, which is part of runtime config delivery |
| `oneconfig` | running services | configmaps are hot reloaded through SDK subscriptions |
| `infra-cd` | region services | provides infra API and account/project foundations used by regional control planes |
| `aws-shared-cd` | application layers | prepares shared cluster/platform surfaces later consumed by app delivery |
| `byoc-account-setup` | BYOC onboarding | prepares customer-side IAM and bastion prerequisites before PingCAP-managed flows can operate |

### Important Structural Consequence

`auto-deploy` is the strongest coupling point in the system.

It bridges:

- cloud provisioning
- ArgoCD cluster registration
- application rollout initiation
- region registry updates
- infra-cd follow-up changes

Any analysis of deployment ordering that ignores `auto-deploy` will miss real rollout dependencies.

## Source of Truth Matrix

| Concern | Primary source of truth | Secondary / consumer repos |
| --- | --- | --- |
| Shared free-tier cluster definition | `aws-shared-cd` | `app-delivery`, runtime services |
| Infra API and account/project state | `infra-cd` | `auto-deploy`, control-plane services |
| Regional rollout plan and region metadata | `auto-deploy` | `app-delivery`, ArgoCD, internal region registry |
| Kubernetes workload manifests | `app-delivery` | ArgoCD |
| Runtime service configuration | `oneconfig` | config-center and subscribed services |
| Customer-side BYOC bootstrap | `byoc-account-setup` | customer account, later BYOC operations |

## Rollback Boundaries

| Repo | Rollback shape | Relative difficulty | Notes |
| --- | --- | --- | --- |
| `oneconfig` | revert config PR and re-apply | Low | fastest path, namespace-scoped, hot reload capable |
| `app-delivery` | revert manifest / app-registry change and let ArgoCD sync | Medium | safe for isolated overlays, risky for `base` or shared app-registry changes |
| `infra-cd` | revert Pulumi config/code and run stack update | Medium to High | resource drift and irreversible infra changes remain possible |
| `aws-shared-cd` | revert Pulumi config/code or adjust release repo state | High | branch and release split makes prod rollback less direct |
| `auto-deploy` | workflow-specific destroy / cleanup plus possible downstream repo rollback | Very High | multi-step, cross-repo, may require manual cleanup |
| `byoc-account-setup` | customer-side Terraform/manual rollback | High | outside the main internal automation loop |

## Risks and Gaps

### Confirmed Structural Risks

1. Environment modeling is inconsistent across repos.
   Different repos use branches, paths, stacks, tags, namespaces, logical region names, and GitHub environments as environment keys.

2. Production release paths are split and non-uniform.
   `aws-shared-cd` production release depends on `aws-shared-cd-prod`; `infra-cd` production also has specialized release handling.

3. `auto-deploy` has high hidden coupling.
   It reaches into ArgoCD state, `app-delivery`, `infra-cd`, and internal APIs. Failures can leave partially converged state across repositories and systems.

4. `app-delivery` has broad blast-radius paths.
   The README explicitly warns against changing `base` directly because it can trigger all environments including prod.

5. `oneconfig` still contains migration-era compatibility behavior.
   The Jenkinsfile duplicates `nightly` into `nightly-ms`, which indicates legacy environment coupling that should be treated carefully.

### Evidence Gaps

1. `aws-shared-cd-prod` is referenced but not represented in this knowledge base.
   Production analysis for the shared free-tier domain is therefore incomplete.

2. The current KB does not yet connect this deployment analysis to a full service dependency graph for all deployed services.
   The next useful step is to connect major app-delivery services with their `service-mapping` outputs.

3. The exact production delivery path for every `infra-cd` subproject is not fully unified in one doc.
   Current evidence is enough to establish that production is special, but not enough to compress every subproject into one identical release rule.

## Recommended Follow-Up

1. Add `aws-shared-cd-prod` into deployment metadata if it is in scope.
2. Build a repo-to-service map between `app-delivery` applications and `knowledge/repos/*/service-mapping/`.
3. Record a separate region-lifecycle document centered on `auto-deploy`, because it is the highest-complexity control repo.
4. Record a separate production-promotion document because prod conventions differ materially across repos.

## Evidence Index

- `./.local/review-history/repos-filtered.txt`
- `knowledge/deployment/deployment-repos.json`
- `knowledge/deployment/deployment-repos.md`
- `knowledge/deployment/deployment-repos-non-tidbcloud.json`
- `~/coding/infra-cd/README.md`
- `~/coding/infra-cd/Jenkinsfile`
- `~/coding/infra-cd/doc/gha.md`
- `~/coding/infra-cd/.github/workflows/multi-account.yml`
- `~/coding/infra-cd/.github/workflows/multi-account-prod-cd.yaml`
- `~/coding/aws-shared-cd/readme.md`
- `~/coding/aws-shared-cd/.github/workflows/freetier.yml`
- `~/coding/aws-shared-cd/.github/workflows/release.yml`
- `~/coding/aws-shared-cd/.github/workflows/release-alicloud.yml`
- `~/coding/app-delivery/README.md`
- `~/coding/app-delivery/.github/workflows/merge-check.yaml`
- `~/coding/app-delivery/.github/workflows/deploy-new-region-regional-server.yaml`
- `~/coding/app-delivery/.github/file-filters.yaml`
- `~/coding/app-delivery/app-registry/config-center.yaml`
- `~/coding/oneconfig/README.md`
- `~/coding/oneconfig/Jenkinsfile`
- `~/coding/oneconfig/apply.sh`
- `~/coding/auto-deploy/README.md`
- `~/coding/auto-deploy/.github/workflows/deploy-aws-region.yml`
- `~/coding/auto-deploy/.github/workflows/register-new-region.yml`
- `~/coding/auto-deploy/internal/argocd/argocd.go`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/regional-server`
- `~/coding/auto-deploy/etc/deploy-job-scripts/aws-region-control/finalizers/dp-network`
- `~/coding/auto-deploy/meta/dev.yaml`
- `~/coding/auto-deploy/meta/staging.yaml`
- `~/coding/auto-deploy/meta/prod.yaml`
- `~/coding/byoc-account-setup/README.md`