# terraform/aws/byoc/shoot

Child Terraform module — **not a root module**. Called by the parent `../` via `module "shoot"`. Do not run `terraform init` or apply here directly.

Deploys the customer-side O11Y (observability) EKS cluster and supporting stack in the **dataplane (customer) AWS account**. This is not the TiDB data plane; TiDB clusters are managed by the TiDB Cloud control plane, not Terraform.

## Provider aliases

| Alias | Purpose |
|---|---|
| _(default)_ | Dataplane account — all primary resources |
| `aws.tf_state` | Reads remote S3 state objects |
| `aws.seed` | Cross-account reads from seed account (e.g. `clinic-auto-deploy-token` in Secrets Manager) |

There is no `aws.cloudformation` alias here — CloudFormation fallbacks for `o11y_hosted_zone_id` etc. are resolved at the parent level.

## Key locals and naming

- Cluster name: `o11y-{env}-{logical_region_name}`
- VPC name: `{cluster_name}-vpc`
- Default VPC CIDR: `10.1.0.0/16`

## EKS node instance type

Defaults to `r6a.xlarge` if available in the region; falls back to `r6i.xlarge`. This is different from what the parent docs describe for the seed cluster.

## IAM roles that must pre-exist

These roles are **not created by this module** — they must exist in the dataplane account before apply. The customer is responsible for creating them using the `iam_roles/` scripts in [tidbcloud/byoc-account-setup](https://github.com/tidbcloud/byoc-account-setup).

- `tidbcloud-o11y-eks-cluster-role`
- `tidbcloud-o11y-eks-node-role`
- `tidbcloud-o11y-control-plane-role` — assumed by vmcluster, Loki, alertmanager, regional server, drhouse
- `tidbcloud-sli-firehose-role`
- `tidbcloud-o11y-agent-role` (pre-set agent role for IRSA)
- `tidbcloud-o11y-apigw-role` (used as API Gateway access principal)
- `auto-deploy-cli` (used by `aws-role-arn` in CD workflows)

## `lib.py` and `uv` dependency

`basic-iam.tf` uses `local-exec` provisioners that invoke `lib.py` via:

```
interpreter = ["uv", "run", "-w", "boto3", "python", "-c"]
```

`uv` must be on PATH during apply. The script mutates the `tidbcloud-o11y-control-plane-role` trust policy to add/remove the EKS OIDC provider entry.

## AWS CLI profile convention

The kubeconfig exec command and several `local-exec` blocks use `--profile ${account_id}` (the numeric AWS account ID). Local AWS CLI profiles must be named by account ID.

## CloudFormation templates

Two CloudFormation stacks are managed inline using templates from `cloudformation/`:

- `cloudformation/ingress.yaml` — provisions ACM cert, WAF ACL, and outputs `RsCertificateArn`, `RsWebAclArn`, `DsCertificateArn`, `DsWebAclArn`
- `cloudformation/vpc-endpoint-service.yaml` — creates a VPC endpoint service backed by the nginx-write NLB, outputs `VpcEndpointServiceId`

## Apply dependency chain (sequential)

Most O11Y stack modules dispatch GitHub Actions workflows via `module "app"` from `tidbcloud/observability`. Each step must succeed before the next:

1. `module.vpc` + `module.eks` + `module.kubeconfig`
2. `module.register-region-meta` (observability `new-region-meta.yaml`)
3. `module.o11y-infra` → `module.o11y-infra-cd`
4. `module.cloud-addon` → `module.cloud-addon-cd` → `null_resource.wait-ngx-w-active` (polls for `ngx-w-{logical_region_name}` NLB to become active)
5. `module.o11y-dataplane` → `module.o11y-dataplane-cd`
6. `module.o11y-regional-server` → `module.o11y-regional-server-cd`
7. `module.o11y-drhouse` → `module.o11y-drhouse-cd`
8. `module.register-region-global` → `module.global-server-cd` → `module.o11y-infra-cd-global`
9. `module.loki-datasource` (depends on `ingress` CF stack + `o11y-regional-server-cd`)

`module.runbooks-region-meta` dispatches to `tidbcloud/runbooks` (not `tidbcloud/observability`) and has no explicit dependency on the chain above.

## Destroy ordering quirk

`null_resource.cloud-addon-svc-finalizer` runs `kubectl delete svc/ingress` commands on destroy to remove AWS NLBs (and their ENIs) **before** EKS and VPC are destroyed. If this provisioner fails or is skipped, VPC deletion will fail due to dangling ENIs.

> **Real-world blocker:** In practice, the security team applies deletion protection to the AWS LB resources and the relevant Kubernetes Service/Ingress objects. This prevents Terraform from completing the destroy automatically. **Before initiating a region teardown**, manually remove the deletion protection from both the AWS Load Balancer(s) and the corresponding Kubernetes resources (e.g. remove finalizers / deletion-protection annotations), otherwise `terraform destroy` will stall or fail.

## Key output consumed by parent

- `lb_nginx_w` — DNS name of the `ngx-w-{logical_region_name}` NLB. The parent module passes this to `seed` as `o11y_write_nlb`.

## ECR PrivateLink (optional)

Set `vpc.enable_ecr_privatelink = true` to create Interface VPC endpoints for ECR. When enabled, `byoc_ecr` must be a valid ECR registry hostname (e.g. `123456789012.dkr.ecr.eu-west-1.amazonaws.com`); a `precondition` will fail with a clear message if it is not.

> **Cost consideration:** Without PrivateLink, container image pulls from ECR traverse the public internet, which incurs significant AWS data transfer charges on the customer's bill. Enable this option for production deployments to keep ECR traffic within the AWS network and avoid those egress costs.

## `regional_server_img_tag`

Pinned to a specific image tag in `vars.tf:104`. Check this value before upgrading the regional server or data server images.

> **Note:** "regional server" here refers to the **O11Y regional server** (deployed by `module.o11y-regional-server` in this shoot module), not the TiDB Cloud control-plane regional server that lives in `module.seed`.

## S3 buckets created

> **Before destroy:** These buckets must be emptied manually before running `terraform destroy`. AWS will refuse to delete a non-empty S3 bucket, causing the destroy to fail. Clear all objects (including versioned objects and delete markers if versioning is enabled) from each bucket first.

| Bucket name pattern | Purpose |
|---|---|
| `o11y-{env}-{logical_region_name}-backups` | O11Y backups |
| `o11y-{env}-shared-{logical_region_name}` | vmcluster (VictoriaMetrics) storage |
| `loki-{env}-{logical_region_name}-o11y` | Loki log storage |
| `{cluster_name}-ds-logs` | ALB/NLB access logs for data server |
| `tidbcloud-{env}-{logical_region_name}-byoc-audit` | BYOC audit logs |
| `tidbcloud-sli-data-{env}-{logical_region_name}` | SLI metrics (Parquet via Kinesis Firehose + Glue) |

## Related Documentation

- [../AGENTS.md](../AGENTS.md) — parent module: workflow commands, backend config, provider setup, CloudFormation fallbacks