# TiDB X AWS Configuration Reference

Load this reference while planning AWS, checking artifact provenance, rendering
configuration, or reviewing network and storage choices.

## Contents

- [Approval Block](#approval-block)
- [Provisioning Variables](#provisioning-variables)
- [Deployment Context](#deployment-context)
- [S3 And IAM Contract](#s3-and-iam-contract)
- [Artifact Gate](#artifact-gate)
- [Topology Contract](#topology-contract)
- [Network Ports](#network-ports)
- [Existing EC2 Reuse Gate](#existing-ec2-reuse-gate)

## Approval Block

Before an AWS mutation, print this block and obtain approval:

```text
AWS account: <account-id> (<expected-account-id check: pass/fail>)
Region/AZ: <region>/<az>
Cluster: <cluster-name>
Owner: <owning-team>
TTL/ExpiresAt: <timestamp>
Approved cost ceiling: <usd>
Estimated burn: <usd-per-hour> hourly, <usd-per-day> daily
Estimated total through expiry: <usd>
Resources:
  control=<existing-control-host>, pd=<count>x<type>, tidb=<count>x<type>,
  tikv=<count>x<type>, monitor=<count>x<type>, pressure=<count>x<type>
Storage:
  root EBS=<per-role sizes>; TiKV root/local-data capacity=<size>
S3/IAM:
  bucket=<globally-unique-bucket>, prefix=<fresh-prefix>,
  TiKV role/profile=<name>, credentials=EC2 instance profile
Network:
  VPC/subnet=<ids>, public IP=<yes/no>, operator CIDR=<cidr>,
  S3 path=<gateway-endpoint|nat|public-ip>/<resource-id>,
  route table=<verified-route-table-id>, other extras=<approved values>
Cleanup:
  owner=<cleanup-owning-team>, pause command=<path>, destroy command=<path>,
  resources preserved or deleted=<exact inventory>
Approval SHA256: <hash-over-all-fields-above>
```

The approval gate passes only when the AWS account matches,
`ESTIMATED_TOTAL_COST_USD` fits the ceiling, and every instance, volume, public
endpoint, S3 resource, IAM resource, and cleanup action is explicitly covered.
The provisioner recomputes the hash and refuses any changed input.

## Provisioning Variables

The bundled plan and provision scripts consume these variables:

```bash
export PROFILE="<aws-profile>"
export EXPECTED_AWS_ACCOUNT_ID="<account-id>"
export REGION="<region>"
export AZ="<availability-zone>"
export VPC_ID="<vpc-id>"
export SUBNET_ID="<subnet-id>"
export AMI="<linux-amd64-ami>"
export KEY_NAME="<ec2-key-pair>"
export RUN_DIR="<absolute-run-directory>"
export STATE_FILE="$RUN_DIR/state.env"
export RUN_LOG_DIR="$RUN_DIR/logs"

export CLUSTER_NAME="<cluster-name>"
export PROVISION_TOKEN="<unique-idempotency-token>"
export OWNER="<owning-team>"
export CREATED_BY="<stable-automation-label>"
export CLEANUP_OWNER="<cleanup-owning-team>"
export EXPIRES_AT="<iso-8601>"
export COST_CEILING_USD="<approved-limit>"
export ESTIMATED_BURN_RATE_USD_PER_HOUR="<estimated-hourly-usd>"
export ESTIMATED_BURN_RATE_USD_PER_DAY="<estimated-daily-usd>"
export ESTIMATED_TOTAL_COST_USD="<estimated-total-usd-through-expiry>"
export CONTROL_HOST_ID="<existing-control-instance-id>"
export CONTROL_SG="<control-host-security-group-id>"
export OPERATOR_CIDR="<approved-cidr>"
export ASSOCIATE_PUBLIC_IP="<0-or-1>"
export S3_NETWORK_MODE="<gateway-endpoint|nat|public-ip>"
export S3_NETWORK_RESOURCE_ID="<vpce-id|nat-id|igw-id>"
export S3_NETWORK_POLICY_APPROVED=1
export SSH_PORT="<topology-ssh-port>"
export TIDB_PORT="<tidb-port>"
export GRAFANA_PORT="<grafana-port>"
export PROMETHEUS_PORT="<prometheus-port>"
export NETWORK_EXTRAS="<explicit-approved-network-extras-or-none>"

export S3_BUCKET="<globally-unique-bucket>"
export S3_PREFIX="<fresh-cluster-prefix>"
export TIKV_IAM_ROLE="<cluster-name>-tikv-role"
export TIKV_IAM_POLICY="<cluster-name>-s3-prefix"

export CONTROL_TYPE="<existing-control-host-type>"
export PD_COUNT=3
export PD_TYPE="<pd-instance-type>"
export PD_ROOT_GB="<pd-root-gb>"
export TIDB_COUNT=2
export TIDB_TYPE="<tidb-instance-type>"
export TIDB_ROOT_GB="<tidb-root-gb>"
export TIKV_COUNT=3
export TIKV_TYPE="<tikv-instance-type>"
export TIKV_ROOT_GB="<tikv-root-and-local-data-gb>"
export MONITOR_COUNT=1
export MONITOR_TYPE="<monitor-instance-type>"
export MONITOR_ROOT_GB="<monitor-root-gb>"
export PRESSURE_COUNT=0
# When PRESSURE_COUNT is greater than zero, also set PRESSURE_TYPE and
# PRESSURE_ROOT_GB explicitly.
```

The control/build host is an existing prerequisite and is not created by the
provisioner. TiKV local storage holds `db`, raft, WAL, cache, and working state.
Select `TIKV_ROOT_GB` from measured local-state growth, workload size, and
headroom. Record a topology caveat when reusing a root volume that also contains
an older cluster. Both planning and apply verify that `AMI` is an available
x86_64 EBS-backed image and use its reported `RootDeviceName`; no device name is
hard-coded.

Use a team or service value for `OWNER` and a stable tool/workflow value for
`CREATED_BY`. Do not derive either tag or any resource name from a personal
username, workstation account, or value copied from a previous run.
Choose a new `PROVISION_TOKEN` for each state file, keep it unchanged when
reconciling that run, limit it to 48 letters, digits, dots, underscores, or
hyphens, and do not encode an account ID or personal username in it.

No public-address or S3-route default is assumed. The accepted combinations are:

| `S3_NETWORK_MODE` | `ASSOCIATE_PUBLIC_IP` | Resource |
|---|---:|---|
| `gateway-endpoint` | `0` | Regional S3 gateway endpoint associated with the subnet route table |
| `nat` | `0` | Available NAT gateway selected by the subnet default route |
| `public-ip` | `1` | Attached internet gateway selected by the subnet default route |

`aws_plan.sh` and `aws_provision.sh` resolve the subnet route table and verify
the selected route objects. For NAT mode they also verify the NAT subnet's
internet-gateway path. Set `S3_NETWORK_POLICY_APPROVED=1` only after reviewing
the gateway-endpoint or surrounding network policy for the selected bucket and
prefix. The final data-plane proof is at least one cluster-written object under
the fresh run-specific prefix. Validation accepts an object already written by
the current cluster; if the prefix is still empty, it runs SQL smoke and waits
up to five minutes for the first object. Public ingress remains limited to
`OPERATOR_CIDR`.

## Deployment Context

Set one explicit context and use it to render the topology and every command:

```bash
export CLUSTER_NAME="<cluster-name>"
export TIUP_METADATA_VERSION="<tiup-metadata-version>"
export DEPLOY_SSH_USER="<ami-bootstrap-user>"
export DEPLOY_SSH_KEY="<path-to-bootstrap-private-key>"
export CLUSTER_KNOWN_HOSTS="$RUN_DIR/cluster_known_hosts"

export TOPOLOGY_FILE="$RUN_DIR/topology.yaml"
export WORKER_CONFIG="$RUN_DIR/cse-worker.toml"
export PD_TARBALL="$RUN_DIR/artifacts/pd.tar.gz"
export TIDB_TARBALL="$RUN_DIR/artifacts/tidb.tar.gz"
export TIKV_TARBALL="$RUN_DIR/artifacts/tikv.tar.gz"
export WORKER_TARBALL="$RUN_DIR/artifacts/tikv-worker.tar.gz"

export TIDB_RUNTIME_USER="<tidb-runtime-user>"
export TIDB_RUNTIME_GROUP="<tidb-runtime-group>"
export TIDB_RUNTIME_HOME="<absolute-runtime-home>"
export TIUP_DEPLOY_DIR="<absolute-tiup-deploy-dir>"
export TIUP_DATA_DIR="<absolute-tiup-data-dir>"
export KEYSPACE_NAME="<keyspace-name>"

export PD_CLIENT_PORT="<pd-client-port>"
export PD_PEER_PORT="<pd-peer-port>"
export TIDB_PORT="<tidb-port>"
export TIDB_STATUS_PORT="<tidb-status-port>"
export TIKV_PORT="<tikv-port>"
export TIKV_STATUS_PORT="<tikv-status-port>"
export GRAFANA_PORT="<grafana-port>"
export PROMETHEUS_PORT="<prometheus-port>"
```

`DEPLOY_SSH_USER` is the AMI/bootstrap login accepted by `tiup cluster deploy`.
`TIDB_RUNTIME_USER` is `global.user` in the rendered topology and is also the
worker service user. `TIDB_RUNTIME_GROUP` is that user's actual runtime group;
do not infer it from the username. After deployment, TiUP's internal cluster key
connects as `TIDB_RUNTIME_USER`. `KEY_NAME` must identify the public key
corresponding to `DEPLOY_SSH_KEY`. The deployment and runtime users may be the
same, but the skill never assumes that they are or derives either one from a
workstation account.

Run all bundled scripts from the existing Linux control/build host. It owns the
state file, rendered topology, worker configuration, artifacts, TiUP metadata,
and AWS CLI profile. When operating remotely, first connect to that host and
then run the commands there; the scripts do not embed a control-host login.
Populate `CLUSTER_KNOWN_HOSTS` with the cluster hosts' independently verified
SSH host keys; do not accept keys opportunistically during deployment.
The state file is an allowlisted base64 key/value record, not a shell script.
Update it only through `write_state`; readers reject unknown or duplicate keys.

## S3 And IAM Contract

The provisioner creates a private bucket with SSE-S3, full public-access block,
and resource tags. Its IAM policy grants:

- `s3:GetBucketLocation` on the bucket;
- prefix-restricted `s3:ListBucket` on the bucket;
- get, put, delete, and multipart object operations on `<prefix>/*`.

Only TiKV instances receive the profile. The co-located worker inherits the same
metadata credentials. `topology-tidbx.yaml` and `cse-worker.toml` omit
`s3-key-id` and `s3-secret-key`.

The S3 control-plane gate is a put/head/delete round trip using a regular empty
file at `<prefix>/.preflight`. After cluster start, the data-plane gate requires
at least one object written below the same prefix.

Canonical TiDB X validation also requires
`tidb_enable_resource_control=ON`. Upstream nextgen TiDB defaults it to ON, and
an OFF value means the intended resource-control runtime contract is not active;
do not silently downgrade this gate.

## Artifact Gate

Required control-host files:

| Archive | Root binary | Runtime identity |
|---|---|---|
| `$RUN_DIR/artifacts/pd.tar.gz` | `pd-server` | Next Generation PD |
| `$RUN_DIR/artifacts/tidb.tar.gz` | `tidb-server` | Next Generation TiDB |
| `$RUN_DIR/artifacts/tikv.tar.gz` | `tikv-server` | Cloud Storage Engine TiKV |
| `$RUN_DIR/artifacts/tikv-worker.tar.gz` | `tikv-worker` | Cloud Storage Engine worker |

Run this gate on the Linux control/build host:

```bash
set -euo pipefail
verify_artifact() {
  (
  set -euo pipefail
  local archive="$1"
  local binary="$2"
  local expected_identity="$3"
  local work_dir
  local file_output
  local version_output
  work_dir="$(mktemp -d)"
  trap 'rm -rf "$work_dir"' EXIT

  test -f "$archive"
  test "$(tar -tzf "$archive" | wc -l | tr -d ' ')" = "1"
  test "$(tar -tzf "$archive")" = "$binary"
  tar -xzf "$archive" -C "$work_dir" "$binary"
  file_output="$(file "$work_dir/$binary")"
  printf '%s\n' "$file_output"
  printf '%s\n' "$file_output" | grep -Eq 'ELF 64-bit LSB.*x86-64'
  version_output="$("$work_dir/$binary" -V 2>&1)"
  printf '%s\n' "$version_output"
  printf '%s\n' "$version_output" | grep -F "$expected_identity"
  sha256sum "$archive"
  )
}

artifact_commit() (
  set -euo pipefail
  local archive="$1"
  local binary="$2"
  local work_dir
  local commit
  work_dir="$(mktemp -d)"
  trap 'rm -rf "$work_dir"' EXIT
  tar -xzf "$archive" -C "$work_dir" "$binary"
  commit="$("$work_dir/$binary" -V 2>&1 | sed -n 's/^Git Commit Hash:[[:space:]]*//p' | head -n 1)"
  test -n "$commit"
  printf '%s\n' "$commit"
)

verify_artifact "$PD_TARBALL" pd-server "Kernel Type: Next Generation"
verify_artifact "$TIDB_TARBALL" tidb-server "Kernel Type: Next Generation"
verify_artifact "$TIKV_TARBALL" tikv-server "Cloud Storage Engine"
verify_artifact "$WORKER_TARBALL" tikv-worker "Cloud Storage Engine"
TIKV_COMMIT="$(artifact_commit "$TIKV_TARBALL" tikv-server)"
WORKER_COMMIT="$(artifact_commit "$WORKER_TARBALL" tikv-worker)"
test "$TIKV_COMMIT" = "$WORKER_COMMIT"
printf 'TiKV/worker shared commit: %s\n' "$TIKV_COMMIT"
```

Record the output in an artifact manifest. Every binary must be ELF Linux amd64.
PD and TiDB must report `Kernel Type: Next Generation`; TiKV and worker must
report Cloud Storage Engine and the same exact commit. macOS Mach-O artifacts
are local build evidence only.

## Topology Contract

Render `topology-tidbx.yaml` and `cse-worker.toml` from one set of values:

- runtime user/home, deploy/data directories, keyspace, and component ports;
- S3 endpoint `https://s3.<region>.amazonaws.com`;
- bucket, region, and fresh prefix;
- every PD private endpoint;
- `remote-worker-addr` and `remote-coprocessor-addr` both set to the worker HTTP
  endpoint `http://127.0.0.1:9500/coprocessor`.

The worker gRPC `cop-addr` remains `19000`; it is not a TiKV remote HTTP URL.
Use at least four vCPUs per TiKV for normal validation. A cost-capped two-vCPU
smoke node needs `readpool.unified.max-thread-count: 2` and is not a capacity
test.

## Network Ports

Prefer a self-referencing cluster security group plus full ingress from the
control-host security group. Expose only approved ports to `OPERATOR_CIDR`.

| Port | Component | Source |
|---:|---|---|
| topology SSH port | SSH | control SG; optional operator CIDR |
| `PD_CLIENT_PORT` / `PD_PEER_PORT` | PD | cluster SG and control SG |
| `TIDB_PORT` | TiDB SQL | control SG; optional operator CIDR |
| `TIDB_STATUS_PORT` | TiDB status | cluster/monitor SG |
| `TIKV_PORT` / `TIKV_STATUS_PORT` | TiKV | cluster SG and control SG |
| 9500/19000 | worker HTTP/gRPC | local host or cluster SG |
| `GRAFANA_PORT` | Grafana | approved operator CIDR |
| `PROMETHEUS_PORT` | Prometheus | control SG or approved operator CIDR |

## Existing EC2 Reuse Gate

Before reusing hosts, inventory `tiup cluster list`, running services, bound
ports, deploy/data directories, attached volumes, instance profiles, and S3
prefixes. Stop the old cluster while preserving its metadata. Give the new
cluster distinct ports, deploy/data directories, and prefix, and record both
clusters in the final inventory. A fresh EC2 deployment stays on standard ports
and directories.