# TiDB X AWS Multinode Troubleshooting

Load this reference when a canonical deployment step fails, times out, resumes
on reused hosts, pauses, or enters final cleanup.

## Contents

- [Watchdog And Evidence](#watchdog-and-evidence)
- [SYSTEM Keyspace Bootstrap](#system-keyspace-bootstrap)
- [S3 And IAM Failures](#s3-and-iam-failures)
- [Worker And DFS Failures](#worker-and-dfs-failures)
- [Existing Host Conflicts](#existing-host-conflicts)
- [Local Disk Exhaustion](#local-disk-exhaustion)
- [Pause And Resume](#pause-and-resume)
- [Final Cleanup](#final-cleanup)
- [Common Failures](#common-failures)

## Watchdog And Evidence

Run long AWS, SSH, and TiUP commands through `run_step` from `scripts/common.sh`.
Recommended deadlines:

| Step | Timeout |
|---|---|
| EC2 provisioning | 20 minutes |
| Artifact upload | 30 minutes |
| TiUP deploy | 30 minutes |
| One TiUP role patch | 20 minutes |
| First start | 30 minutes |
| SYSTEM bootstrap | 3 minutes |
| Worker install and health | 5 minutes per host |
| Post-deploy validation | 10 minutes |

On failure, save the step log, `tiup cluster display`, affected systemd status,
the last 80 relevant component-log lines, EC2 status, and S3/IAM status before
choosing a recovery. SSH calls should use a connection timeout and keepalives.

## SYSTEM Keyspace Bootstrap

Use this branch only when the first start log contains
`SYSTEM keyspace is not bootstrapped`.

From the control host, copy the helper to one TiDB node using the TiUP cluster
key, then run it with the rendered PD endpoints:

```bash
export CLUSTER_SSH_KEY="$HOME/.tiup/storage/cluster/clusters/$CLUSTER_NAME/ssh/id_rsa"
export REMOTE_HELPER_DIR="$TIDB_RUNTIME_HOME/bootstrap-system-keyspace"
ssh -i "$CLUSTER_SSH_KEY" -p "$SSH_PORT" -o StrictHostKeyChecking=yes \
  -o "UserKnownHostsFile=$CLUSTER_KNOWN_HOSTS" "$TIDB_RUNTIME_USER@<tidb-private-ip>" \
  "install -d '$REMOTE_HELPER_DIR'"
scp -i "$CLUSTER_SSH_KEY" -P "$SSH_PORT" -o StrictHostKeyChecking=yes \
  -o "UserKnownHostsFile=$CLUSTER_KNOWN_HOSTS" \
  scripts/common.sh scripts/bootstrap_system_keyspace.sh \
  "$TIDB_RUNTIME_USER@<tidb-private-ip>:$REMOTE_HELPER_DIR/"

ssh -i "$CLUSTER_SSH_KEY" -p "$SSH_PORT" -o StrictHostKeyChecking=yes \
  -o "UserKnownHostsFile=$CLUSTER_KNOWN_HOSTS" "$TIDB_RUNTIME_USER@<tidb-private-ip>" \
  "chmod +x '$REMOTE_HELPER_DIR/bootstrap_system_keyspace.sh' && \
   PD_ENDPOINTS='<pd-1>:<pd-client-port>,<pd-2>:<pd-client-port>,<pd-3>:<pd-client-port>' \
   TIDB_DEPLOY_DIR='$TIUP_DEPLOY_DIR/tidb-$TIDB_PORT' \
   SYSTEM_PORT='<unused-tidb-port>' \
   SYSTEM_STATUS_PORT='<unused-tidb-status-port>' \
   '$REMOTE_HELPER_DIR/bootstrap_system_keyspace.sh' --apply"
```

Wait 30-60 seconds for the normal TiDB service to retry, then run
`tiup cluster display`. Bootstrap is complete only when the temporary SYSTEM
TiDB listened successfully and all normal TiDB instances are Up.

Select two unused ports and pass them explicitly; the helper has no user, home,
directory, or port defaults.

## S3 And IAM Failures

Check these layers in order:

1. `aws sts get-caller-identity` matches the approved account.
2. The bucket has full public-access block and SSE-S3.
3. The role policy covers the exact bucket and prefix.
4. Every TiKV instance reports the expected instance-profile ARN.
5. The subnet route table still selects the recorded endpoint, NAT gateway, or
   internet gateway, and every TiKV can reach the regional S3 endpoint.
6. A regular-file put/head/delete preflight passes at `<prefix>/.preflight`.
7. TiKV and worker configs contain the same endpoint, bucket, region, and prefix.
8. TiKV and worker logs are clear of `AccessDenied`, `NoCredential`,
   `SignatureDoesNotMatch`, invalid URL, and remote compaction errors.

An instance-profile association can remain `associating` while an EC2 instance
is stopped. Start the instance, wait for status checks, then verify the
association again.

## Worker And DFS Failures

The worker contract is:

- HTTP `addr` is `0.0.0.0:9500`;
- gRPC `cop-addr` is `0.0.0.0:19000`;
- both TiKV `kvengine.remote-*` URLs use the HTTP `/coprocessor` endpoint;
- the worker and TiKV share one S3 configuration and one cloud-engine commit;
- `register = true` and every PD private endpoint is present.

After changing worker configuration, rerun `install_cse_worker.sh --apply` and
the full validation. `ANALYZE TABLE ... other error: empty string` usually means
`remote-worker-addr` is absent. Connection-refused or missing-SST errors usually
mean the DFS values differ or the worker is unhealthy.

## Existing Host Conflicts

When reused EC2 nodes contain another TiUP cluster:

1. Record `tiup cluster list`, both cluster names, ports, deploy/data dirs, and
   prefixes.
2. Stop the older cluster while preserving its metadata and data.
3. Render distinct ports, directories, and prefix for the new cluster.
4. Limit later pause and cleanup actions to the new inventory.

This preserves the older test state while giving the new deployment a fresh
nextgen bootstrap boundary.

## Local Disk Exhaustion

S3-backed TiKV still grows local `db`, raft, WAL, cache, and working state.
Symptoms include disks near 100%, TiKV becoming Disconnected, and SQL returning
`Region is unavailable`.

Stop workload progress, inspect `df` and `du`, then expand the exact TiKV EBS
volumes and filesystems or rebuild with the approved size. Preserve older data
unless its deletion is explicitly approved. Revalidate TiUP and every worker
before continuing.

## Pause And Resume

`pause_cluster.sh --apply` stops `cse-worker`, the TiUP cluster, and then EC2.
It preserves S3, IAM, EBS, security-group rules, and TiUP metadata.

To resume:

```bash
aws --profile "$PROFILE" --region "$REGION" ec2 start-instances \
  --instance-ids $ALL_INSTANCE_IDS
aws --profile "$PROFILE" --region "$REGION" ec2 wait instance-status-ok \
  --instance-ids $ALL_INSTANCE_IDS
```

Refresh public IPs if used, start the TiUP cluster, start `cse-worker` on every
TiKV, then rerun `validate_cluster.sh --apply`. After validation succeeds, reset
the resumable pause phases:

```bash
write_state "$STATE_FILE" PAUSE_WORKERS_STOPPED "0"
write_state "$STATE_FILE" PAUSE_TIUP_STOPPED "0"
write_state "$STATE_FILE" CLUSTER_PAUSED "0"
```

## Final Cleanup

Final cleanup has a separate destructive gate. Review the state file and the
plan output from `cleanup_aws.sh`; confirm exact TiUP cluster, EC2 IDs, S3
bucket/prefix, role/profile/policy, and security group. The cleanup sequence is:

1. stop every worker and destroy only the named TiUP cluster;
2. terminate the recorded EC2 instances;
3. delete only the recorded S3 prefix, then its cluster-owned empty bucket;
4. remove the role from the profile and delete the recorded IAM objects;
5. delete the recorded cluster security group;
6. record the post-delete verification.

An interrupted provision may leave a mix of created identifiers and pre-recorded
creation intents. Run cleanup from the same control/build host. It recovers EC2
by client token and the security group by the provision-token tag, verifies live
provenance before deletion, and skips a TiUP cluster never marked deployed. An
S3 intent without successful creation plus tags is preserved for manual audit;
it is never promoted to deletion authority. Do not copy identifiers from a
previous run or another account.

The cleanup plan also retries `DescribeInstances` for every recorded EC2 client
token to cover API-response loss and EC2 eventual consistency. If it reports no
visible instance for a token from an interrupted request, wait and rerun the
plan before approving deletion.

The cleanup plan prints an approval SHA256 over the reconciled destructive
inventory. Export it as `APPROVED_CLEANUP_SHA256` only after review. Any state or
inventory change requires a fresh plan and approval.

## Common Failures

| Symptom | Cause | Recovery |
|---|---|---|
| `kernel type mismatch` | PD or TiDB is not a nextgen build | Replace artifact, offline-patch, restart |
| PD rejects its version | Build used a non-semver release version | Produce a PD artifact with legal `RELEASE_VERSION` |
| `SYSTEM keyspace is not bootstrapped` | Fresh SYSTEM tables need initialization | Run the scoped SYSTEM bootstrap branch |
| `GetAllKeyspacesGCStates ... UNIMPLEMENTED` | Vanilla PD is running | Offline-patch nextgen PD and restart |
| S3 403 or `NoCredential` | Missing/wrong TiKV instance profile or policy | Repair the exact association/prefix policy and rerun preflight |
| Remote SST read or compaction fails | Worker DFS differs from TiKV DFS | Render both from one S3 value set and reinstall workers |
| `ANALYZE TABLE` returns empty-string error | `remote-worker-addr` is absent or wrong | Set both remote URLs to port 9500 `/coprocessor` |
| TiKV unified-readpool panic | Instance has too few vCPUs | Use at least four vCPUs or set a smoke-only thread cap |
| Region unavailable during seed | TiKV local storage is full | Expand the exact volumes/filesystems and revalidate |
| New deploy collides with old ports | Reused host still owns old TiUP metadata | Stop old cluster and allocate isolated ports, dirs, and prefix |