# terraform/aws/byoc

Root Terraform module that provisions AWS BYOC (Bring Your Own Cloud) infrastructure for TiDB Cloud. Deploys two child modules in parallel: `seed/` (control plane, runs in the seed AWS account) and `shoot/` (customer-side O11Y infrastructure, runs in the customer/dataplane AWS account). Note: `shoot` is **not** the TiDB data plane — TiDB clusters are deployed by the TiDB Cloud control plane, not by Terraform.

## Workflow

> **Prerequisites — customer account initialisation:** Before running `terraform plan`, confirm that the customer has completed the account setup steps described in [tidbcloud/byoc-account-setup](https://github.com/tidbcloud/byoc-account-setup). That repository contains three components the customer must run inside their own AWS account:
> 1. **`iam_roles/`** — creates the minimal IAM roles and policies required by the BYOC data plane and O11Y stack.
> 2. **`bastion/`** — deploys the bastion EC2 instance that gives PingCAP a controlled, auditable access path into the customer account.
> 3. **`aws_sec_conf_check/`** _(optional)_ — validates that the AWS environment meets security requirements.
>
> If the customer has not used the official CloudFormation-backed setup and instead provisioned resources via their own scripts, the CloudFormation stacks (`tidbcloud-byoc-setup-dataplane`, `tidbcloud-byoc-setup-o11y`) will not exist. In that case pass `tidb_hosted_zone_id`, `o11y_hosted_zone_id`, and `pca_arn` explicitly as vars to skip the CloudFormation lookups (see [CloudFormation fallbacks](#cloudformation-fallbacks)).

```sh
# State key must be supplied at init time — it is NOT hardcoded in terraform.tf
terraform init -backend-config="key=aws/byoc/<env>-<logical_region_name>.json"
terraform plan  -var-file=test.tfvars
terraform apply -var-file=test.tfvars
```

`test.tfvars` targets `dev` / `us-west-2` / tenant `2130533` and is safe to use for local planning.

## Backend

- Bucket: `autonomous-deployment-terraform-state`, region `us-west-2`, native S3 locking (`use_lockfile = true`).
- State key **not** hardcoded — always pass `-backend-config="key=..."` at `terraform init`.

## Provider aliases

| Alias | Profile | Region | Purpose |
|---|---|---|---|
| _(default)_ | `local.seed_account` (resolved from S3 state) | `var.aws_region` | Seed account resources |
| `aws.shoot` | `var.dataplane_account` | `var.aws_region` | Shoot account resources |
| `aws.cloudformation` | `var.dataplane_account` | `var.cloudformation_region` (default `us-west-2`) | Reads CloudFormation stacks in customer account |
| `aws.tf_state` | `default` | `us-west-2` | Reads remote state objects from S3 |

The default provider profile is `local.seed_account`, which is resolved at plan time from S3 state — not a fixed string. Local AWS CLI profiles must match the account IDs.

## Remote state dependencies

Read-only S3 state objects consumed at plan time:

| Key | Variable |
|---|---|
| `aws/base/{env}.json` | VPC/account base outputs |
| `aws/account/seed/{env}.json` | `seed_account`, `base_account`, hosted zone name/ID |
| `aws/clinic/{env}.json` (or `staging.json` for `dev`) | `clinic_endpoint_service_name` |
| `aws/o11y-global/prod.json` | Always reads `prod` regardless of `env` |

## CloudFormation fallbacks

> **Background:** This fallback mechanism was introduced after a customer POC incident where the customer refused to use CloudFormation to initialize their account. Instead, they wrote their own scripts based on the CloudFormation templates we provided. As a result, no CloudFormation stacks existed in their account, and we could not read the required values (hosted zone IDs, PCA ARN) from stack outputs. The explicit var overrides (`tidb_hosted_zone_id`, `o11y_hosted_zone_id`, `pca_arn`) were added as escape hatches to support such non-standard account setups.

`tidb_hosted_zone_id`, `o11y_hosted_zone_id`, and `pca_arn` default to `null`. When null, they are read from CloudFormation stacks in the **shoot account**:

- `tidbcloud-byoc-setup-dataplane` → `HostedZoneId`, `PCAArn`
- `tidbcloud-byoc-setup-o11y` → `O11yHostedZoneId`

These stacks must exist in `var.cloudformation_region` (default `us-west-2`) inside the dataplane account before apply. Pass the vars explicitly to skip the CloudFormation lookups.

## Cross-module dependency

`shoot` exports `lb_nginx_w` (nginx NLB DNS name), which is consumed by `seed` as `o11y_write_nlb`. Terraform resolves this automatically; do not apply `seed` in isolation if the shoot NLB does not yet exist.

## Kubeconfig

Written to `/tmp/{cluster_name}-kubeconfig` during apply by the `kubeconfig` module. All `helm`, `kubernetes`, and `kubectl` providers consume it via `config_path`. Not persisted after the apply process exits.

## Key variable defaults

- `env`: `"dev"`
- `cloudformation_region`: `"us-west-2"`
- EKS K8s version: `"1.34"` (both seed and shoot)
- Seed node AMI type: `BOTTLEROCKET_x86_64`, instance type: `m6i.large`
- Shoot node instance type: `r6a.xlarge` (falls back to `r6i.xlarge`) — different from seed
- `shoot` `regional_server_img_tag` has a pinned default — check `shoot/vars.tf` before upgrading

## `apply_commit` output

`terraform.tf` records the git HEAD SHA at apply time via `data.local_command.git_rev_parse`. This requires `git` on PATH during apply.

## Child modules

- `seed/` — EKS cluster + in-cluster services in seed account; also calls `module.region_registration` to register the BYOC region globally. Uses `alekc/kubectl` provider (>= 2.0.0) in addition to AWS.
- `shoot/` — EKS cluster + **customer-side O11Y (observability) infrastructure** in the customer account; `aws.seed` alias maps back to the root default provider for cross-account calls from shoot to seed. Note: `shoot` does **not** deploy the TiDB data plane (i.e. TiDB clusters, TiKV, PD, etc.) — those are managed by the TiDB Cloud control plane, not by Terraform. What `shoot` deploys is the supporting O11Y stack (e.g. metrics/log agents, NLBs for write paths) that enables observability for workloads running in the customer account.

## Related Documentation

- [seed/AGENTS.md](seed/AGENTS.md) — seed child module: in-cluster services, required env vars, EKS admin roles, deployment ordering
- [shoot/AGENTS.md](shoot/AGENTS.md) — shoot child module: IAM prerequisites, `uv`/`lib.py` dependency, S3 buckets, destroy ordering quirk
- [../../alicloud/AGENTS.md](../../alicloud/AGENTS.md) — parallel structure reference for Alicloud modules
- [../../../.github/AGENTS.md](../../../.github/AGENTS.md)