# AWS production Terraform state adoption

This runbook is the fail-closed recovery contract for the AWS production
Terraform root. It covers adoption of existing resources when the canonical
remote state is absent. It does not authorize a production change by itself.

## Fixed backend identities

Use exactly these keys. Do not shorten or improvise them:

| Terraform root | S3 key |
| --- | --- |
| `infra/aws/environments/prod` | `infra/aws/environments/prod/terraform.tfstate` |
| `infra/aws/bootstrap` | `infra/aws/bootstrap/terraform.tfstate` |

The bootstrap state is separate because it owns the state bucket and lock
table. Never import bootstrap resources into the production application state.
The GCP DAX root also remains separate at its existing key.

## Mandatory read-only preflight

Before any normal production `terraform plan` or `terraform apply`, run from
the repository root:

```bash
python3 scripts/terraform_state_backend_preflight.py \
  --bucket "$TF_STATE_BUCKET" \
  --lock-table "$TF_LOCK_TABLE"
```

The preflight uses only these metadata operations:

- S3 `GetBucketVersioning`, `GetBucketEncryption`, and
  `GetPublicAccessBlock`;
- DynamoDB `DescribeTable`;
- S3 `HeadObject` for the exact canonical production key.

It never calls S3 `GetObject`, reads the state body, pulls Terraform state, or
writes to AWS. It requires bucket versioning, KMS default encryption, all four
public-access blocks, an active `LockID` lock table, the exact state object, and
KMS encryption on that current object version itself.

This command is an operator gate, not a hook built into Terraform. The bucket,
region, lock table, and canonical key subsequently passed to `terraform init`
must be the same reviewed identities. A successful check of one backend does
not authorize initializing another backend; automation must pass the same
values to both steps before normal plan/apply is enabled.

Exit `3` means the canonical state object is absent. Exit `2` means a backend
protection is unsafe or could not be verified. Either result blocks normal
plan/apply. Do not bypass the result with a different key, local state, an
empty state upload, `-lock=false`, or `terraform apply -refresh-only`.

## Adoption preparation

When state is absent:

1. Freeze normal production Terraform plan/apply jobs and announce the
   adoption window. Existing application deployment workflows are a separate
   concern and should not receive Terraform permissions as a workaround.
2. Confirm the intended repository revision and generate/review its dependency
   lock file without the backend. Do not let the adoption window resolve a
   floating provider version. Build the import manifest from that exact
   revision; addresses from another branch are not interchangeable.
3. Reconcile configuration to the live resource identities before import.
   Any planned create, delete, or replace is a configuration/adoption blocker,
   not an approved infrastructure change.
4. Use a dedicated least-privilege adoption identity. It needs only the reads
   required for refresh plus narrowly scoped state/lock access and the provider
   import reads. Do not reuse a deployment role merely because it is available.
5. Record read-only recovery evidence: bucket versioning and protection status,
   lock-table status, and the state object version identifier after each future
   state write. Do not copy, print, attach, or commit the state body.

## Approved sequencing contract

Adoption must be delivered in separately reviewed batches:

1. Adopt the backend bucket controls and lock table into the dedicated
   bootstrap key.
2. Land configuration-only reconciliation for live names, CIDRs, external DNS
   ownership, IAM policy continuity, and other replacement-causing drift.
3. Import production resources in dependency order: network; security groups
   and existing rules; OIDC and EC2 IAM; EC2; ECR; ACM/ALB; ElastiCache; then
   SSM parameters last.
4. After each batch, run a refresh-only plan, inspect action metadata, and stop
   on every create/delete/replace. Capture the new S3 version identifier before
   starting the next batch.
5. Run a normal plan only after every reviewed import batch is complete. A
   normal apply remains prohibited until the plan contains no unreviewed
   production change and the operator has a tested state-version recovery path.

Import itself writes Terraform state. It must therefore happen only in an
explicitly approved adoption operation, never in this preflight or CI
validation job.

## Secrets and output handling

AWS provider v5 can store decrypted `SecureString` parameter values in raw
Terraform state during import or refresh. For the SSM batch:

- require the encrypted remote backend and the narrowest state access first;
- do not use local state;
- do not run or publish `terraform show -json`, `terraform state pull`, a saved
  plan artifact, or verbose provider logs;
- never paste state, parameter values, or plan JSON into an issue or PR;
- stop if Terraform cannot persist to the remote backend, because recovery can
  leave a local state file containing sensitive values.

The preflight deliberately checks only object metadata and never validates by
reading state contents.

## Recovery boundary

If an import is mapped to the wrong address, stop immediately. Do not repair it
with a normal apply. Preserve the current S3 version identifier, review the
last known-good version metadata, and use a separately approved state-only
recovery procedure. State rollback and live-resource rollback are different
operations; restoring a state version must not be combined with cloud mutation.

The adoption is complete only when the canonical object exists, the preflight
passes, all declared live resources have reviewed ownership, a normal plan is
free of unreviewed changes, and recovery evidence is retained without exposing
the state body.