## Module 3: Isolate The Subnet 1 in Availability Zone 1
### Step 1: Simulate a complete Availability Zone failure in Availability Zone 1
+ Return to [Terminal 1](ssh://monitor1-ssh#1) and execute the script to simulate a complete Availability Zone failure by blocking subnet `demo-subnet-1` (subnet 10.90.1.0/24 in Availability Zone 1).
```bash
VPC_ID=`aws ec2 describe-vpcs \
--filters "Name=tag:Name,Values=${LAB:USER_UNIQUE_TAG}" \
--query "Vpcs[0].VpcId" \
--output text \
--region ${LAB:REGION_CODE}`
SUBNET_ID=`aws ec2 describe-subnets \
--filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-1" \
--query "Subnets[0].SubnetId" \
--output text \
--region ${LAB:REGION_CODE}`
CAGE_NACL_ID=`aws ec2 describe-network-acls \
--filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" "Name=tag:Name,Values=cage" \
--query "NetworkAcls[0].NetworkAclId" \
--output text \
--region ${LAB:REGION_CODE}`
ASSOC_ID=`aws ec2 describe-network-acls \
--filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \
--query "NetworkAcls[0].Associations" \
--output text \
--region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'`
aws ec2 replace-network-acl-association \
--association-id ${ASSOC_ID} \
--network-acl-id ${CAGE_NACL_ID} \
--region ${LAB:REGION_CODE}
```
```output
{
"NewAssociationId": "aclassoc-0402a6103febf799f"
}
```
### Step 2: Verify that several components are disconnected
+ In [Terminal 1](ssh://monitor1-ssh#1), check the cluster status again to verify that several components, including PD, TiDB and TiKV servers are disconnected/isolated because of the Availability Zone failure.
+ In the sample output, the impacted servers are `${LAB:HOST_PD1_PRIVATE_IP}:2379 pd`, `${LAB:HOST_DB1_PRIVATE_IP}:4000 tidb`, and `${LAB:HOST_KV1_PRIVATE_IP}:20160 tikv`. Therefore, one-third of the cluster components have lost connection.
```bash
tiup cluster display tidb-demo
```
```output
Cluster type: tidb
Cluster name: tidb-demo
Cluster version: v${LAB:TIDB_VERSION}
Deploy user: ec2-user
SSH type: builtin
Dashboard URL: http://${LAB:HOST_PD1_PUBLIC_IP}:2379/dashboard
Grafana URL: http://${LAB:HOST_MONITOR1_PRIVATE_IP}:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
${LAB:HOST_MONITOR1_PRIVATE_IP}:9093 alertmanager ${LAB:HOST_MONITOR1_PRIVATE_IP} 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
${LAB:HOST_MONITOR1_PRIVATE_IP}:3000 grafana ${LAB:HOST_MONITOR1_PRIVATE_IP} 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
${LAB:HOST_PD1_PRIVATE_IP}:2379 pd ${LAB:HOST_PD1_PRIVATE_IP} 2379/2380 linux/x86_64 Down /tidb-data/pd-2379 /tidb-deploy/pd-2379
${LAB:HOST_PD2_PRIVATE_IP}:2379 pd ${LAB:HOST_PD2_PRIVATE_IP} 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379
${LAB:HOST_PD3_PRIVATE_IP}:2379 pd ${LAB:HOST_PD3_PRIVATE_IP} 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
${LAB:HOST_MONITOR1_PRIVATE_IP}:9090 prometheus ${LAB:HOST_MONITOR1_PRIVATE_IP} 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
${LAB:HOST_DB1_PRIVATE_IP}:4000 tidb ${LAB:HOST_DB1_PRIVATE_IP} 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000
${LAB:HOST_DB2_PRIVATE_IP}:4000 tidb ${LAB:HOST_DB2_PRIVATE_IP} 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
${LAB:HOST_KV1_PRIVATE_IP}:20160 tikv ${LAB:HOST_KV1_PRIVATE_IP} 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
${LAB:HOST_KV2_PRIVATE_IP}:20160 tikv ${LAB:HOST_KV2_PRIVATE_IP} 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
${LAB:HOST_KV3_PRIVATE_IP}:20160 tikv ${LAB:HOST_KV3_PRIVATE_IP} 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 11
```
### Step 3: Verify that your INSERT and QUERY workloads are still running
+ In [Terminal 2](ssh://monitor1-ssh#2) and [Terminal 3](ssh://monitor1-ssh#3), verify that your sample INSERT/QUERY workloads are still running. Note that the INSERT and QUERY workloads are working with only one TiDB server instance now.
```output
Worker 1 -> TiDB:Mon May 13 05:51:52 UTC 2024 at ${LAB:HOST_DB2_PRIVATE_IP}
Worker 2 -> TiDB:Mon May 13 05:51:52 UTC 2024 at ${LAB:HOST_DB2_PRIVATE_IP}
Worker 1 -> TiDB:Mon May 13 05:51:53 UTC 2024 at ${LAB:HOST_DB2_PRIVATE_IP}
Worker 2 -> TiDB:Mon May 13 05:51:53 UTC 2024 at ${LAB:HOST_DB2_PRIVATE_IP}
Worker 1 -> TiDB:Mon May 13 05:51:54 UTC 2024 at ${LAB:HOST_DB2_PRIVATE_IP}
...
```
```output
Mon May 13 05:51:51 UTC 2024
COUNT(event)
5481
Mon May 13 05:51:54 UTC 2024
COUNT(event)
5485
...
```
### Step 4: Restore the connectivity for Availability Zone 1
+ In [Terminal 1](ssh://monitor1-ssh#1), run the scripts to restore connectivity for `demo-subnet-1`. In a real-world scenario on TiDB Cloud, you would not run these scripts yourself—TiDB Cloud manages recovery. If an availability zone issue is brief, disconnected components rejoin automatically once the network returns. For prolonged data center outages, TiDB Cloud can scale capacity in healthy zones while evicting unreachable nodes. Either way, applications keep using healthy TiDB endpoints—the same outcome you observed in this Self-Managed exercise.
For more advanced TiDB skills, visit [https://tidb.io/education/](https://tidb.io/education/).
```bash
VPC_ID=`aws ec2 describe-vpcs \
--filters "Name=tag:Name,Values=${LAB:USER_UNIQUE_TAG}" \
--query "Vpcs[0].VpcId" \
--output text \
--region ${LAB:REGION_CODE}`
SUBNET_ID=`aws ec2 describe-subnets \
--filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-1" \
--query "Subnets[0].SubnetId" \
--output text \
--region ${LAB:REGION_CODE}`
DEFAULT_NACL_ID=`aws ec2 describe-network-acls \
--filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \
--query "NetworkAcls[0].NetworkAclId" \
--output text \
--region ${LAB:REGION_CODE}`
ASSOC_ID=`aws ec2 describe-network-acls \
--filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" \
--query "NetworkAcls[0].Associations" \
--output text \
--region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'`
aws ec2 replace-network-acl-association \
--association-id ${ASSOC_ID} \
--network-acl-id ${DEFAULT_NACL_ID} \
--region ${LAB:REGION_CODE}
```
```output
{
"NewAssociationId": "aclassoc-0df1e05cfc70356a8"
}
```
### Step 5: Verify that all cluster components are synchronized
+ Verify the cluster status to ensure that the previously disconnected components are now synchronized.
```bash
tiup cluster display tidb-demo
```
```output
Cluster type: tidb
Cluster name: tidb-demo
Cluster version: v${LAB:TIDB_VERSION}
Deploy user: ec2-user
SSH type: builtin
Dashboard URL: http://${LAB:HOST_PD1_PUBLIC_IP}:2379/dashboard
Grafana URL: http://${LAB:HOST_MONITOR1_PRIVATE_IP}:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
${LAB:HOST_MONITOR1_PRIVATE_IP}:9093 alertmanager ${LAB:HOST_MONITOR1_PRIVATE_IP} 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
${LAB:HOST_MONITOR1_PRIVATE_IP}:3000 grafana ${LAB:HOST_MONITOR1_PRIVATE_IP} 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
${LAB:HOST_PD1_PRIVATE_IP}:2379 pd ${LAB:HOST_PD1_PRIVATE_IP} 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
${LAB:HOST_PD2_PRIVATE_IP}:2379 pd ${LAB:HOST_PD2_PRIVATE_IP} 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379
${LAB:HOST_PD3_PRIVATE_IP}:2379 pd ${LAB:HOST_PD3_PRIVATE_IP} 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
${LAB:HOST_MONITOR1_PRIVATE_IP}:9090 prometheus ${LAB:HOST_MONITOR1_PRIVATE_IP} 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
${LAB:HOST_DB1_PRIVATE_IP}:4000 tidb ${LAB:HOST_DB1_PRIVATE_IP} 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
${LAB:HOST_DB2_PRIVATE_IP}:4000 tidb ${LAB:HOST_DB2_PRIVATE_IP} 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
${LAB:HOST_KV1_PRIVATE_IP}:20160 tikv ${LAB:HOST_KV1_PRIVATE_IP} 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
${LAB:HOST_KV2_PRIVATE_IP}:20160 tikv ${LAB:HOST_KV2_PRIVATE_IP} 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
${LAB:HOST_KV3_PRIVATE_IP}:20160 tikv ${LAB:HOST_KV3_PRIVATE_IP} 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 11
```
+ Press Ctrl+C in [Terminal 2](ssh://monitor1-ssh#2) and [Terminal 3](ssh://monitor1-ssh#3) to stop both workloads.