# terraform/aws/o11y-region

Root Terraform module that provisions a full AWS **o11y (observability) region** — the Terraform-native replacement for the legacy eksctl + bash + CloudFormation flow driven by `etc/deploy-job-scripts/o11y-aws-region-control/`. Modeled directly on [../../alicloud/o11y-region](../../alicloud/o11y-region).

## What this module owns

- EKS cluster `o11y-{env}-regional-{logical_region_name}` in the o11y control-plane AWS account (Bottlerocket worker node group in a single AZ, `max-pods = 110`, vpc-cni prefix delegation)
- VPC `10.1.0.0/16` (default) across exactly 2 AZs, public + private subnets, single NAT gateway
- 4 customer IAM policies (documents kept verbatim under `policies/`) — **ensured, not managed**: they are account-level shared resources (no cluster suffix). `null_resource.shared_iam_policies` creates them only if missing (same semantics as the old `basic/iam` script) and never deletes them, so re-deploys and sibling regions in the same account are safe. 8 IRSA roles are fully managed (ported from `basic/aws/cloudformation/iam.yaml`)
- ACM certificates for the rs/ds domains (public ACM + Route53 DNS validation, or account Private CA when `use_pca = true`) and WAFv2 IPSet/WebACL pairs (office CIDRs copied from `ingress.yaml`)
- DMS: `{cluster}-dms` IRSA role + 2 CloudWatch alarms in the SNS topic's region (topic ARN read from `aws/o11y-global/{env}.json` state)
- 4 S3 buckets: `o11y-{env}-{logical}-backups`, `o11y-{env}-shared-{logical}`, `loki-{env}-{logical}-o11y` (91-day lifecycle on prefix `fake/`), `{cluster}-ds-logs` (public access block + ELB delivery policy)
- Optional RDS MySQL 8.4 (`enable_rds`): MultiAZ gp3 20→1000 GB, db `kubestore`, final snapshot `{cluster}-final`, credentials in Secrets Manager (`recovery_window_in_days = 0`)
- VPC endpoint service on the `ngx-w-{logical}` NLB (acceptance not required, private DNS `prometheus-endpoint-{az_suffix}.{hosted_zone}` + Route53 verification TXT + verification wait), allowed principals, S3 gateway endpoint on all VPC route tables
- ArgoCD cluster registration (`modules/argocd_cluster`, same config shape as deploy-cli `RegisterK8s`)
- All in-cluster o11y services via Argo Workflows (`../../modules/app`, repo `tidbcloud/observability` unless noted), in this order:
  1. `register-region-meta` (new-region-meta.yaml)
  2. `o11y-infra` → 3. `o11y-infra-cd` (waits eventrouter/prometheus/alertmanager/volume-autoscaler/self-vector/vm-operator/cert-manager)
  4. `cloud-addon` → 5. `cloud-addon-cd` (waits ingress-nginx r/w, external-dns, cluster-autoscaler, aws-lbc)
  6. `o11y-dataplane` → 7. `o11y-dataplane-cd` (waits vmcluster, loki-ha, alertmanager)
  8. `static-principal-access` (agent-writer roles; **no finalizer by design**)
  9. `o11y-regional-server` → 10. `o11y-regional-server-cd`
  11. `o11y-drhouse` → `o11y-drhouse-cd`
  12. `register-region-global` → `global-server-cd` → `o11y-infra-cd-global`
  13. `loki-datasource` (`tidbcloud/app-delivery`)
  14. `runbooks-region-meta` (`tidbcloud/runbooks`, ref `master`)
  15. `jumpserver` (`tidbcloud/app-delivery`, new-jump-server-dst.yaml; finalizer destroy-region-application.yaml with `application=devops-jump-server-dst`)

## State and account resolution

- **Backend**: S3 bucket `autonomous-deployment-terraform-state` (us-west-2), native S3 lock. State key is NOT in code — supply it at init:
  ```sh
  terraform init -backend-config="key=aws/o11y-region/<env>-<logical_region_name>.json"
  ```
- **Two AWS provider aliases**:
  - _(default)_: profile `local.o11y_account`, region `var.aws_region`
  - `aws.tf_state`: profile `default`, region `us-west-2` — reads state objects from the bucket
- `local.o11y_account` is resolved at plan time from `aws/o11y-global/{env}.json` (also provides `sns_topic_arn`). `aws/base/{env}.json` provides `pca_arn` and is only read when `use_pca = true`. The `default` AWS profile must be able to read the state bucket before `terraform plan`.
- **Dev has no `aws/o11y-global/dev.json`** (the o11y-global module only supports staging/prod). For dev, set `o11y_account` + `sns_topic_arn` explicitly — this skips the o11y-global state read entirely. The workflow exposes this via the optional `sns-topic-arn` input (account then comes from the `aws-account` input).

## Required variables (no defaults)

| Variable | Example |
|---|---|
| `aws_region` | `ap-east-1` |
| `logical_region_name` | `ap-east-1` |
| `availability_zones` | `["ap-east-1a", "ap-east-1b"]` — exactly 2 AZ **names**; the worker node group goes to index `[0]` |

## Key variable defaults

`env=dev`, `vpc_cidr=10.1.0.0/16`, `eks_version="1.35"`, worker `r6a.xlarge` desired 3 / min 1 / max 10 / maxPods 110, `enable_rds=false`, `rds_instance_class=db.t4g.large`, `rds_deletion_protection=false`, `nlb_allowed_principals` = the 4 existing account roots, `admin_role_names=[cicd-role, autonomous-oncall]` (non-prod envs additionally get `autonomous-cent-stor-dev`, which only exists in the dev/staging account — matching the legacy `meta/o11y-{dev,staging}.yaml` admin-roles), `jump_service_account_role_arn=arn:aws:iam::557537366020:role/jump-service-role`, `jump_server_role_arn=arn:aws:iam::<o11y_account>:role/jump-service-role`.

## Destroy semantics (everything unused is destroyed)

Destroy order is enforced by the dependency graph:

1. App-module **finalizer workflows** run first (each `modules/app` finalizer removes its gitops manifests). `static-principal-access` has none (agent-writer roles are additive and reused across re-provisions — explicit exemption).
2. `null_resource.k8s_cleanup` (every workload-deploying app module `depends_on` it, so it is destroyed after them, before EKS): deletes `ingress/regionalingress` (-n observability) and the LB-type `svc nginx-write` / `svc ingress-nginx-write-controller` / `ingress nginx-read` / `ingress nginx-ingress` (-n kube-system) with `kubectl wait --for=delete`; polls `elbv2` until no LB tagged `kubernetes.io/cluster/{cluster}` remains (30-min cap, skips on API errors); deletes all PVCs in `observability`/`self-monitoring`/`default` (`--wait --timeout=600s`, tolerates timeout) so the CSI driver removes the EBS volumes.
3. `null_resource.bucket_flush` empties the 4 buckets (`aws s3 rm --recursive`, skips missing buckets) before the `aws_s3_bucket` deletions — versioning is off and `force_destroy = false`, so this ordering is what makes bucket deletion succeed.
4. EKS, IAM, ACM/WAF, endpoint service, VPC are deleted by Terraform normally. RDS takes a final snapshot (`{cluster}-final`); the secret is deleted immediately.

Known leftovers (by design): the RDS final snapshot, the agent-writer roles, and the 4 shared account-level IAM policies (ensured, never destroyed — see above). The legacy dataplane `{env}-{logical}-dp4o11yrs` roles / `auto-deploy-{cluster}-iam-dataplane` stacks are **not** managed or cleaned by this module or the workflow: that dataplane path is deprecated, and stacks belonging to legacy regions remain the responsibility of the legacy destroy flow.

## Non-obvious constraints

- **`modules/app` runs `kubectl wait` with suppressed output and `kubectl wait --for=create` fails immediately on missing CRDs** — every cross-step dependency must be an explicit `depends_on`; do not rely on Terraform's implicit ordering for the app chain.
- **Kubeconfig is ephemeral**: `/tmp/{cluster_name}-kubeconfig`, exec-based (`aws eks get-token --profile <o11y_account>`). Local AWS CLI profiles must be named by account ID.
- **NLB lookup**: `data.aws_lb.ngx_w` only resolves after `cloud-addon-cd` has deployed the `ngx-w-{logical}` NLB; `null_resource.wait_ngx_w_active` blocks on `aws elbv2 wait load-balancer-available` first.
- **Private DNS verification TXT**: record FQDN is `<private_dns_name_configuration.name>.<private_dns_name>` (same shape as `terraform/aws/byoc/shoot`); `aws_vpc_endpoint_service_private_dns_verification` then waits up to 30 min.
- **`hostedZoneId` arg** passed to cloud-addon keeps the legacy `/hostedzone/<id>` prefix (it is used as external-dns `txtOwnerId`).
- **`elb_account_id`** (per-region ELB account for the ds-logs bucket policy) defaults to `null` → statement omitted. The old flow set it per region via `elbAccountIdForS3BucketPolicy` in `meta/o11y-*.yaml` (required for pre-2022-08 regions like ap-east-1) — pass it via the workflow's `elb-account-id` input.
- **`nlb_allowed_principals`** defaults to the prod set. dev/staging use different principals (see `nlbAllowedPrincipals` in `meta/o11y-*.yaml`) — pass them via the workflow's `nlb-allowed-principals` input.
- `null_resource.k8s_cleanup` also `depends_on` the ACM certificates and WAF WebACLs, so on destroy they are deleted only after the load balancers that reference them are drained.
- Deleting the VPC endpoint service requires consumers to have disconnected first — an operational constraint, not a Terraform one.
- Worker `max-pods` is injected via Bottlerocket TOML user data (`bootstrap_extra_args`); EKS merges its bootstrap settings with it.

## Related documentation

- [alicloud/o11y-region](../../alicloud/o11y-region) — direct pattern template for this module
- [aws/premium AGENTS.md](../premium/AGENTS.md) — AWS EKS/VPC conventions
- [aws/byoc/shoot AGENTS.md](../byoc/shoot/AGENTS.md) — the BYOC variant of the same o11y stack (destroy ordering quirks)