# TiDB X AWS Multinode Runbook

This is the main path for an empty canonical TiDB X cluster on AWS EC2. It ends
at a validated cluster handoff; application data and workloads remain in their
own test plans.

Run this entire workflow from the designated Linux control/build host. It must
hold the AWS CLI profile, this skill folder, the state file, rendered configs,
artifacts, and TiUP metadata. If the host is remote, connect to it before Step 1.

Create one isolated directory for the run and set the paths from
`tidbx-aws-config.md`:

```bash
export RUN_DIR="<absolute-run-directory>"
export STATE_FILE="$RUN_DIR/state.env"
export RUN_LOG_DIR="$RUN_DIR/logs"
install -d "$RUN_DIR/artifacts" "$RUN_LOG_DIR"
```

## Contents

- [1. Plan And Approve AWS](#1-plan-and-approve-aws)
- [2. Provision AWS, S3, And IAM](#2-provision-aws-s3-and-iam)
- [3. Stage And Verify Artifacts](#3-stage-and-verify-artifacts)
- [4. Render Topology And Worker Configuration](#4-render-topology-and-worker-configuration)
- [5. Deploy, Patch, And Start](#5-deploy-patch-and-start)
- [6. Install The Co-located Workers](#6-install-the-co-located-workers)
- [7. Validate The Empty Cluster](#7-validate-the-empty-cluster)
- [8. Handoff, Pause, Or Destroy](#8-handoff-pause-or-destroy)

## 1. Plan And Approve AWS

Fill the variables in `tidbx-aws-config.md`, then run:

```bash
scripts/aws_plan.sh
```

Save the output in the running record. Review the account, cost, TTL, network
exposure, root EBS sizes, S3 bucket/prefix, role/profile, and exact pause and
destroy inventories. Confirm that the subnet has the selected, verified S3
route. Continue only after approval, then export the exact printed hash:

```bash
export APPROVED_PLAN_SHA256="<approved-plan-sha256>"
```

## 2. Provision AWS, S3, And IAM

The provisioner is dry-run by default:

```bash
scripts/aws_provision.sh
scripts/aws_provision.sh --apply
```

It creates and records:

- a private encrypted S3 bucket and fresh prefix;
- a prefix-scoped TiKV role and instance profile;
- a cluster security group;
- PD, TiDB, TiKV, monitor, and optional pressure EC2 nodes;
- the profile association on TiKV nodes only.

Before mutation it verifies the approved S3 endpoint, NAT, or internet-gateway
path. It records deterministic names before non-EC2 creates and writes returned
IDs immediately. Before each EC2 request it records a unique client token,
allowing cleanup to recover a successful request whose response was interrupted.
It then waits for EC2 health and performs the S3 put/head/delete preflight. Keep
the state file with the run record, including after a failed provision.

## 3. Stage And Verify Artifacts

Place these prebuilt tarballs on the control host:

```bash
export PD_TARBALL="$RUN_DIR/artifacts/pd.tar.gz"
export TIDB_TARBALL="$RUN_DIR/artifacts/tidb.tar.gz"
export TIKV_TARBALL="$RUN_DIR/artifacts/tikv.tar.gz"
export WORKER_TARBALL="$RUN_DIR/artifacts/tikv-worker.tar.gz"
```

Run the artifact gate in `tidbx-aws-config.md` and save its complete output.

## 4. Render Topology And Worker Configuration

Copy and fill:

- `references/topology-tidbx.yaml` as `$RUN_DIR/topology.yaml`;
- `references/cse-worker.toml` as `$RUN_DIR/cse-worker.toml`.

Replace every placeholder. Use the same S3 endpoint, bucket, region, prefix,
PD endpoints, runtime identity/home, directories, keyspace, and ports in both
files and the deployment commands. Add or remove server entries so the topology
matches the exact provisioned node inventory. Runtime S3 credentials come from
the TiKV instance profile, so both rendered files omit static keys.

```bash
export TOPOLOGY_FILE="$RUN_DIR/topology.yaml"
export WORKER_CONFIG="$RUN_DIR/cse-worker.toml"
if grep -En '<[^>]+>' "$TOPOLOGY_FILE" "$WORKER_CONFIG"; then
  echo "unresolved placeholders" >&2
  exit 1
fi

. scripts/common.sh
for context_name in \
  TOPOLOGY_FILE WORKER_CONFIG PD_TARBALL TIDB_TARBALL TIKV_TARBALL WORKER_TARBALL TIUP_METADATA_VERSION \
  DEPLOY_SSH_USER DEPLOY_SSH_KEY CLUSTER_KNOWN_HOSTS TIDB_RUNTIME_USER \
  TIDB_RUNTIME_GROUP TIDB_RUNTIME_HOME TIUP_DEPLOY_DIR TIUP_DATA_DIR KEYSPACE_NAME \
  PD_CLIENT_PORT PD_PEER_PORT TIDB_PORT TIDB_STATUS_PORT TIKV_PORT TIKV_STATUS_PORT \
  GRAFANA_PORT PROMETHEUS_PORT; do
  write_state "$STATE_FILE" "$context_name" "${!context_name}"
done
```

## 5. Deploy, Patch, And Start

On the control host:

```bash
export PATH="$HOME/.tiup/bin:$PATH"
. scripts/common.sh

write_state "$STATE_FILE" TIUP_CLUSTER_DEPLOY_INTENT "1"
run_step "tiup deploy" 1800 \
  tiup cluster deploy "$CLUSTER_NAME" "$TIUP_METADATA_VERSION" "$TOPOLOGY_FILE" \
  --user "$DEPLOY_SSH_USER" -i "$DEPLOY_SSH_KEY" \
  --ignore-config-check -y
write_state "$STATE_FILE" TIUP_CLUSTER_DEPLOYED "1"

run_step "patch pd" 1200 \
  tiup cluster patch "$CLUSTER_NAME" "$PD_TARBALL" \
  -R pd --offline -y --overwrite
run_step "patch tidb" 1200 \
  tiup cluster patch "$CLUSTER_NAME" "$TIDB_TARBALL" \
  -R tidb --offline -y --overwrite
run_step "patch tikv" 1200 \
  tiup cluster patch "$CLUSTER_NAME" "$TIKV_TARBALL" \
  -R tikv --offline -y --overwrite

run_step "tiup first start" 1800 \
  tiup cluster start "$CLUSTER_NAME" --init -y
```

`TIUP_METADATA_VERSION` is TiUP metadata only. The offline patches define the
runtime. All three patches precede the first start. `DEPLOY_SSH_USER` is only
the initial host login; the rendered `global.user` is `TIDB_RUNTIME_USER`.

If first start reports `SYSTEM keyspace is not bootstrapped`, follow only that
recovery branch in `tidbx-aws-troubleshooting.md`, then recheck the cluster.

## 6. Install The Co-located Workers

```bash
export STATE_FILE="$RUN_DIR/state.env"
export CLUSTER_SSH_KEY="$HOME/.tiup/storage/cluster/clusters/$CLUSTER_NAME/ssh/id_rsa"
write_state "$STATE_FILE" CLUSTER_SSH_KEY "$CLUSTER_SSH_KEY"
scripts/install_cse_worker.sh
scripts/install_cse_worker.sh --apply
```

The installer stops an existing worker, installs the binary and rendered config
under `TIDB_RUNTIME_HOME`, starts the service as
`TIDB_RUNTIME_USER:TIDB_RUNTIME_GROUP`, and checks
`/healthz` on every TiKV host. The SSH user and key are the TiUP runtime identity
and generated cluster key, not an AMI or workstation default.

## 7. Validate The Empty Cluster

```bash
export TIDB_PORT="<tidb-port>"
export MYSQL_USER="<validation-sql-user>"
export TIKV_PORT="<tikv-port>"
export TIUP_DEPLOY_DIR="<absolute-tiup-deploy-dir>"
export TIDB_RUNTIME_HOME="<absolute-runtime-home>"
export STATE_FILE="$RUN_DIR/state.env"
scripts/validate_cluster.sh
scripts/validate_cluster.sh --apply
```

The validator checks every TiKV's worker, IAM profile, remote configuration,
and relevant logs; checks the shared real-S3 contract; runs SQL smoke and
`ANALYZE TABLE`; confirms that the S3 prefix contains cluster-written data; and
persists `tidb_pessimistic_txn_fair_locking=OFF` for cloud-engine OLTP
compatibility. It also requires `tidb_enable_resource_control=ON`, which is part
of the canonical TiDB X runtime rather than an optional smoke-test setting.

## 8. Handoff, Pause, Or Destroy

At handoff, record cluster endpoints, Grafana endpoint, artifact manifest,
TiUP display, worker health, S3 prefix, IAM profile, EC2 IDs, EBS sizes,
security-group rules, and the exact pause/destroy commands.

For a recoverable stop:

```bash
scripts/pause_cluster.sh
scripts/pause_cluster.sh --apply
```

This stops workers, TiUP, then EC2 while preserving S3, IAM, EBS, and TiUP
metadata.

For final destruction, review the printed inventory before applying:

```bash
scripts/cleanup_aws.sh
export APPROVED_CLEANUP_SHA256="<sha256-printed-by-the-plan>"
scripts/cleanup_aws.sh --apply
```

The cleanup script accepts a partial state file from an interrupted provision.
It deletes only resources whose identifiers were recorded or recovered through
recorded EC2 client tokens, and skips a TiUP cluster that was never deployed.