## Module 2: Simulate The Secondary DC Failure By Isolating Network Access to DC 2

The objective of this practice is to simulate the complete unavailability of Data Center 2, observe the TiDB cluster's status, and restore it.

  ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-5-en.png)

The method to simulate a Data Center 2 failure is to sever the network communication between Data Center 1 and Data Center 2. This will prevent Data Center 2's servers from receiving or responding to external requests, making it appear to Data Center 1 that Data Center 2 has experienced a complete failure.

### Step 1. Connect to the monitor server in data center 1

+ Open [Terminal 1](ssh://az1monitor1-ssh#1), which will connect to monitor1 in data center 1.

### Step 2. Start the workload

+ Start `sysbench` to act as the application running. Once started, proceed to the next step.

  ```bash
  sysbench \
      --config-file=sysbench_config oltp_read_write \
      --tables=6 \
      --mysql-ignore-errors=all run
  ```

  ```output
  sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2)

  Running the test with following options:
  Number of threads: 1
  Report intermediate results every 1 second(s)
  Initializing random number generator from current time

  Initializing worker threads...

  Threads started!

  [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00
  [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00
  ...
  ```

### Step 3. Verify the replication status is in-sync

+ Click to open [Terminal 2](ssh://az1monitor1-ssh#2), which will connect to monitor 1 in DC1 to verify the data replication status is in `sync` mode.

+ Verify the data replication status.

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync",
     "state_id": 1006,
     "acid_consistent": true
   }
  }
  ```

### Step 4. Block the network access to DC2 (Availability Zone 2)

+ Execute the script to simulate a complete Availability Zone failure by blocking subnet `demo-subnet-2` (subnet 10.90.2.0/24 in AZ2).

  ```bash
  VPC_ID=`aws ec2 describe-vpcs \
    --filters "Name=tag:Name,Values=${LAB:USER_UNIQUE_TAG}" \
    --query "Vpcs[0].VpcId" \
    --output text --region ${LAB:REGION_CODE}`
  SUBNET_ID=`aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \
    --query "Subnets[0].SubnetId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  CAGE_NACL_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" "Name=tag:Name,Values=cage" \
    --query "NetworkAcls[0].NetworkAclId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  ASSOC_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \
    --query "NetworkAcls[0].Associations" \
    --output text \
    --region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'`
  aws ec2 replace-network-acl-association \
    --association-id ${ASSOC_ID} \
    --network-acl-id ${CAGE_NACL_ID} \
    --region ${LAB:REGION_CODE}
  ```

  ```output
  {
  "NewAssociationId": "aclassoc-0402a6103febf799f"
  }
  ```

### Step 5. Observe the status of the application and TiDB

+ Return to [Terminal 1](ssh://az1monitor1-ssh#1), sysbench is blocked to access to TiDB cluster for about 70 seconds which is a little bit longer than the block window setting (default setting is 60 seconds). The reason is that the TiDB cluster needs time to gather network status, process the TiDB cluster status switch, and synchronize the status to each TiKV nodes.

  ```output
  ...
  [ 493s ] thds: 6 tps: 278.00 qps: 5525.08 (r/w/o: 3866.05/1082.02/577.01) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
  [ 494s ] thds: 6 tps: 274.99 qps: 5505.84 (r/w/o: 3853.89/1079.97/571.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
  [ 495s ] thds: 6 tps: 271.99 qps: 5447.84 (r/w/o: 3814.89/1065.97/566.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
  [ 496s ] thds: 6 tps: 70.00 qps: 1406.04 (r/w/o: 985.03/278.01/143.00) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00
  [ 497s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 498s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ...
  [ 544s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 545s ] thds: 6 tps: 0.00 qps: 29.00 (r/w/o: 26.00/0.00/3.00) lat (ms,95%): 0.00 err/s: 3.00 reconn/s: 0.00
  [ 546s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ...
  [ 566s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 567s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 568s ] thds: 6 tps: 98.00 qps: 1968.01 (r/w/o: 1376.00/388.00/204.00) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00
  ...
  [ 569s ] thds: 6 tps: 244.00 qps: 4883.91 (r/w/o: 3418.93/950.98/513.99) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
  [ 570s ] thds: 6 tps: 237.00 qps: 4734.95 (r/w/o: 3313.96/924.99/495.99) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
  [ 571s ] thds: 6 tps: 264.02 qps: 5279.31 (r/w/o: 3693.21/1039.06/547.03) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
  ...
  ```

+ Return to [Terminal 2](ssh://az1monitor1-ssh#2), observe the nodes status in TiDB cluster. The nodes which are located in dc2 are all down.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://18.236.254.103:2379/dashboard
  Grafana URL:        http://10.90.1.90:3000,http://10.90.2.17:3000
  ID                 Role        Host         Ports        OS/Arch       Status        Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------        --------                    ----------
  10.90.1.90:3000    grafana     10.90.1.90   3000         linux/x86_64  Up            -                           /tidb-deploy/grafana-3000
  10.90.2.17:3000    grafana     10.90.2.17   3000         linux/x86_64  Down          -                           /tidb-deploy/grafana-3000
  10.90.1.176:2379   pd          10.90.1.176  2379/2380    linux/x86_64  Up|L          /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.73:2379    pd          10.90.1.73   2379/2380    linux/x86_64  Up            /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.198:2379   pd          10.90.2.198  2379/2380    linux/x86_64  Down          /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.90:10090   prometheus  10.90.1.90   10090/12020  linux/x86_64  Up            /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.17:10090   prometheus  10.90.2.17   10090/12020  linux/x86_64  Down          /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.106:4000   tidb        10.90.1.106  4000/10080   linux/x86_64  Up            -                           /tidb-deploy/tidb-4000
  10.90.2.61:4000    tidb        10.90.2.61   4000/10080   linux/x86_64  Down          -                           /tidb-deploy/tidb-4000
  10.90.1.245:20160  tikv        10.90.1.245  20160/20180  linux/x86_64  Up            /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.45:20160   tikv        10.90.1.45   20160/20180  linux/x86_64  Up            /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.125:20160  tikv        10.90.2.125  20160/20180  linux/x86_64  Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.96:20160   tikv        10.90.2.96   20160/20180  linux/x86_64  Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 13
  ```

+ Observe the TiDB sync mode changes, from `sync` to `async_wait`, and `async` subsequential. During the period of `async_wait`, pd sends mode switch instructions to all tikv nodes. Once this process is completed, the sync mode transitions to `async`.

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync",
     "state_id": 1006,
     "acid_consistent": true
   }
  }
  ```

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "async_wait",
     "state_id": 1001,
     "acid_consistent": true
   }
  }
  ```

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "async",
     "state_id": 483,
     "acid_consistent": true
   }
  }
  ```

### Step 6. Recover the network access of data center 2

+ Recover the network. The previously lost nodes will come back to join the TiDB cluster again.

  ```bash
  VPC_ID=`aws ec2 describe-vpcs \
    --filters "Name=tag:Name,Values=${LAB:USER_UNIQUE_TAG}" \
    --query "Vpcs[0].VpcId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  SUBNET_ID=`aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \
    --query "Subnets[0].SubnetId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  DEFAULT_NACL_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \
    --query "NetworkAcls[0].NetworkAclId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  ASSOC_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" \
    --query "NetworkAcls[0].Associations" \
    --output text --region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'`
  aws ec2 replace-network-acl-association \
    --association-id ${ASSOC_ID} \
    --network-acl-id ${DEFAULT_NACL_ID} \
    --region ${LAB:REGION_CODE}
  ```

  ```output
  {
  "NewAssociationId": "aclassoc-0df1e05cfc70356a8"
  }
  ```

+ Switch to [Terminal 1](ssh://az1monitor1-ssh#1), observe the status of the application and TiDB cluster. `sysbench` is running stably.

  ```output
  [ 1039s ] thds: 6 tps: 268.99 qps: 5409.79 (r/w/o: 3787.85/1067.96/553.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
  [ 1040s ] thds: 6 tps: 273.01 qps: 5455.15 (r/w/o: 3814.11/1083.03/558.02) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00
  [ 1041s ] thds: 6 tps: 265.00 qps: 5282.95 (r/w/o: 3702.96/1043.99/535.99) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
  ...
  [ 1105s ] thds: 6 tps: 260.00 qps: 5244.02 (r/w/o: 3669.02/1043.00/532.00) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00
  [ 1106s ] thds: 6 tps: 267.99 qps: 5339.86 (r/w/o: 3740.90/1052.97/545.99) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
  ```

+ Check the TiDB cluster status. All nodes in DC2 should be back online. If you're following the lab steps swiftly, you might notice the status of some TiKV nodes changing from `Disconnected` to `Up` gradually.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.92.147.161:2379/dashboard
  Grafana URL:        http://10.90.1.212:3000,http://10.90.2.99:3000
  ID                 Role        Host         Ports        OS/Arch       Status        Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------        --------                    ----------
  10.90.1.212:3000   grafana     10.90.1.212  3000         linux/x86_64  Up            -                           /tidb-deploy/grafana-3000
  10.90.2.99:3000    grafana     10.90.2.99   3000         linux/x86_64  Up            -                           /tidb-deploy/grafana-3000
  10.90.1.156:2379   pd          10.90.1.156  2379/2380    linux/x86_64  Up            /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.232:2379   pd          10.90.1.232  2379/2380    linux/x86_64  Up|L          /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.104:2379   pd          10.90.2.104  2379/2380    linux/x86_64  Up            /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.212:10090  prometheus  10.90.1.212  10090/12020  linux/x86_64  Up            /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.99:10090   prometheus  10.90.2.99   10090/12020  linux/x86_64  Up            /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.23:4000    tidb        10.90.1.23   4000/10080   linux/x86_64  Up            -                           /tidb-deploy/tidb-4000
  10.90.2.122:4000   tidb        10.90.2.122  4000/10080   linux/x86_64  Up            -                           /tidb-deploy/tidb-4000
  10.90.1.178:20160  tikv        10.90.1.178  20160/20180  linux/x86_64  Up            /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.6:20160    tikv        10.90.1.6    20160/20180  linux/x86_64  Up            /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.250:20160  tikv        10.90.2.250  20160/20180  linux/x86_64  Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.90:20160   tikv        10.90.2.90   20160/20180  linux/x86_64  Up            /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 13
  ```

  ```output
  ...
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.2.250:20160  tikv        10.90.2.250  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  ...
  ```

+ Check the current replication status of the TiDB cluster. Within only several seconds, the status is switched to `sync` from `sync_recover`.

![60%](https://lab-static.pingcap.com/quick-demo/demo-303-6-en.png)

+ `sync_recover` mode.

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync_recover",
     "state_id": 1009,
     "acid_consistent": false,
     "total_regions": 78,
     "recover_progress": 0.08974359
   }
  }
  ```

+ `sync` mode.

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync",
     "state_id": 1010,
     "acid_consistent": true
   }
  }
  ```

![60%](https://lab-static.pingcap.com/quick-demo/demo-303-61-en.png)

+ At this point, TiDB restores data synchronization between the two data centers to the SYNC state, achieving the goal of RPO=0 resilience against data center failures.

### Step 7. Stop the sysbench workload

+ Go to [Terminal 1](ssh://az1monitor1-ssh#1), terminate the sysbench thread, and close Terminal 1.

+ Close Terminal 2.

### Summary for Module 2

+ Through the above steps, we simulated the entire process of TiDB handling a complete DC2 failure and its recovery to availability.
  + When DC2 becomes unavailable, TiDB enters a blocking window to confirm the failure. During this blocking window, application read and write operations to the database are affected.
  + After exceeding the blocking window, TiDB operates using only the nodes in DC1 and transitions to the `async` state. At this point, application read and write operations return to normal.
  + When DC2 recovers from the failure, TiDB enters the `sync_recover` state, synchronizing data from the nodes in DC1 to the nodes in DC2. This process does not affect the availability of the database for application access.
  + Once the data synchronization is complete, the TiDB cluster transitions back to the `sync` mode.