# tidbcloud extract — setup

Provisions the AWS landing zone for this extract unit. Config is read from the
source's `source.yaml` (bucket names, prefixes, retention) so infra and the engine
share one source of truth.

## Given (not provisioned here)

- A TiDB Cloud cluster + the SQL connection string (`TIDB_SOURCE_DSN`).
- A TiDB Cloud API key pair (`TIDB_CLOUD_PUBLIC_KEY` / `TIDB_CLOUD_PRIVATE_KEY`) for the
  cluster's org, in the source's `secrets.env`. The engine bootstraps the named
  `tidb_cloud_cli_profile` from this pair at runtime (`ensure-profile`, also run
  automatically before `start-export` / `write-manifest`), so auth is portable to a cloud
  runner rather than bound to a hand-created `~/.ticloud/config.toml`. If the named profile
  already exists locally it is left as-is (local-dev convenience).

## Provisioned by terragrunt (`grunt/`)

Two buckets, split by sensitivity + lifecycle, plus least-privilege IAM:

| Bucket | Holds | Lifecycle |
|--------|-------|-----------|
| `s3.data.bucket` | Parquet snapshots (sensitive, transient) | `retention.s3_snapshots_days` |
| `s3.metadata.bucket` | manifests + runtime logs (durable provenance) | `retention.manifests_days` |

- `*-tidb-export-writer` IAM user — write-only to the data bucket; its keys are handed
  to TiDB Cloud as the export S3 credentials.
- `*-s3-loader-reader` IAM user — read-only on the data bucket; its keys are used by the
  `s3_bulk_load` `COPY INTO`.

## Apply

```bash
brew install terragrunt                      # one-time
aws sso login --profile pingcap-datalake     # operator credentials

cd ingestion/extract/tidbcloud/setup/grunt/nexuscrm
terragrunt apply
```

State is local (`terraform.tfstate` here, gitignored). Switch to an S3 backend when
setup is shared across machines.

## Wire the issued keys into the ignored secrets files

```bash
# extract — export writer
terragrunt output -raw export_access_key_id      # -> TIDB_EXPORT_AWS_ACCESS_KEY_ID
terragrunt output -raw export_secret_access_key  # -> TIDB_EXPORT_AWS_SECRET_ACCESS_KEY
# load — loader reader
terragrunt output -raw loader_access_key_id      # -> NEXUSCRM_S3_ACCESS_KEY_ID
terragrunt output -raw loader_secret_access_key  # -> NEXUSCRM_S3_SECRET_ACCESS_KEY
```

Paste into `ingestion/extract/tidbcloud/sources/nexuscrm/secrets.env` and
`ingestion/load/s3_bulk_load/sources/nexuscrm/secrets.env` respectively. These files are
gitignored; access-key material is never tracked.