## Module 3: Simulate The Primary DC Failure By Terminating All Nodes in DC 1

The objective of this exercise is to simulate the complete unavailability of the primary data center and how to restore its availability.

When DC1 experiences a complete failure, DC2 still retains the latest data in the follower TiKV instance. Together with a backup learner TiKV instance, a two-replica cluster can be quickly restored to resume service.

  ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-7-en.png)

We need to start a new TiDB cluster through the PD in the disaster recovery data center (retaining the data, but with a new cluster ID). Additionally, we need to scale out a PD node (a pre-reserved standby PD node) and convert the learner TiKV node in the disaster recovery data center into a voting TiKV to form a functional Raft group with another TiKV node.

### Step 1. Connect to the monitor server in data center 1

+ Open [Terminal 1](ssh://az1monitor1-ssh#1), which will connect to monitor1 in data center 1.

### Step 2. Start the workload

+ Start `sysbench` in both DC1 and DC2 to act as the application running, the task will be runing for 60 minutes.

+ Start the `sysbench` workload in DC1.

  ```bash
  sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run
  ```

  ```output
  sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2)

  Running the test with following options:
  Number of threads: 1
  Report intermediate results every 1 second(s)
  Initializing random number generator from current time

  Initializing worker threads...

  Threads started!

  [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00
  [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00
  ...
  ```

+ Open [Terminal 2](ssh://az2monitor1-ssh#2), which will connect to monitor1 in data center 2. Start the `sysbench` workload in DC2.

+ Set up the `sysbench` configuration for DC2.

  ```bash
  cat << EOF > sysbench_config
  mysql-host=${HOST_AZ2DB1_PRIVATE_IP}
  mysql-port=4000
  mysql-user=root
  mysql-password=
  mysql-db=sbtest
  time=3600
  threads=6
  report-interval=1
  EOF
  cat sysbench_config
  ```

  ```output
  mysql-host=10.90.2.125
  mysql-port=4000
  mysql-user=root
  mysql-password=
  mysql-db=sbtest
  time=3600
  threads=16
  report-interval=1
  ```

+ Start the `sysbench` workload in DC2.

  ```bash
  sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run
  ```

  ```output
  Threads started!

  [ 1s ] thds: 6 tps: 316.47 qps: 6401.24 (r/w/o: 4486.46/1158.05/756.73) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
  [ 2s ] thds: 6 tps: 330.09 qps: 6611.89 (r/w/o: 4631.33/1171.34/809.23) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
  ...
  ```

### Step 3. Verify the replication status is in-sync

+ Open [Terminal 3](ssh://az2monitor1-ssh#3), which will connect to monitor1 in data center 2 to start the `sysbench` workload in DC2. to verify the data replication status is in the `sync` mode.

+ Verify the data replication status.

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync",
     "state_id": 1006,
     "acid_consistent": true
   }
  }
  ```

### Step 4. Simulate a complete failure of the primary data center

+ Simulate a complete failure of the primary data center by terminating all TiDB processes on the nodes of the TiDB cluster. Note: We keep the application alive to observe the accessibility of the TiDB cluster.

+ Terminate all the processes and remove data dir of each tidb, tikv and pd nodes that located in DC1. If prompted, answer `yes`.

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1DB1_PRIVATE_IP} "sudo systemctl stop tidb-4000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1KV1_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1KV2_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1PD1_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1PD2_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

+ Terminate the processes and remove data dir of monitoring.

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1MONITOR1_PRIVATE_IP} "sudo systemctl stop prometheus-10090.service; sudo systemctl stop grafana-3000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

### Step 5. Observe the status of the application and TiDB

+ Observe the nodes status in TiDB cluster. The status of all the pd servers is `Down`, all the tikv is `N/A` and one of the tidb servers is `Down`. Only the tidb server located in DC2 is alive.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

  To update this component:   tiup update cluster
  To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Grafana URL:        http://10.90.1.182:3000,http://10.90.2.252:3000
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.1.182:3000   grafana     10.90.1.182  3000         linux/x86_64  Down    -                           /tidb-deploy/grafana-3000
  10.90.2.252:3000   grafana     10.90.2.252  3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.1.213:2379   pd          10.90.1.213  2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.90:2379    pd          10.90.1.90   2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.65:2379    pd          10.90.2.65   2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.182:10090  prometheus  10.90.1.182  10090/12020  linux/x86_64  Down    /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.252:10090  prometheus  10.90.2.252  10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.196:4000   tidb        10.90.1.196  4000/10080   linux/x86_64  Down    -                           /tidb-deploy/tidb-4000
  10.90.2.226:4000   tidb        10.90.2.226  4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.1.228:20160  tikv        10.90.1.228  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.91:20160   tikv        10.90.1.91   20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.202:20160  tikv        10.90.2.202  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.206:20160  tikv        10.90.2.206  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 13
  ```

+ Observe the TiDB sync mode. Because we can't access the pd leader which is located in DC1, we need to access the follower pd which is located in DC2. The operation is timeout because the request can't be redirected to the leader.

  ```bash
  curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  [PD:apiutil:ErrRedirectNoLeader]redirect finds no leader
  ```

+ Switch to [Terminal 1](ssh://az1monitor1-ssh#1). We can see that the workload goes to 0 QPS because the TiDB cluster is unavailability.

  ```output
  ...
  [ 57s ] thds: 6 tps: 193.99 qps: 3834.86 (r/w/o: 2679.90/665.98/488.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
  [ 58s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 59s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ...
  ```

+ Switch to [Terminal 2](ssh://az2monitor1-ssh#2). We can find the workload goes to 0 QPS because of TiDB cluster unavailability.

  ```output
  ...
  [ 34s ] thds: 6 tps: 116.00 qps: 2313.99 (r/w/o: 1624.99/385.00/304.00) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
  [ 35s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 36s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ...
  ```

### Step 6. Reshape the TiDB cluster with surviving nodes only

+ Switch to [Terminal 3](ssh://az2monitor1-ssh#3), overwrite the meta file of `tiup` by the prepared meta file which only contains the nodes that located in data center 2 (AZ2).

+ Backup the meta file before overwriting it.

  ```bash
  cp /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml.backup
  ```

+ Check the yaml file `majority-DC2.yaml`.

  ```bash
  cat majority-DC2.yaml
  ```

  ```output
  user: ec2-user
  tidb_version: v${LAB:TIDB_VERSION}
  topology:
   global:
     user: "ec2-user"
     ssh_port: 22
     deploy_dir: "/tidb-deploy"
     data_dir: "/tidb-data"
     arch: "amd64"
   monitored:
     node_exporter_port: 9100
     blackbox_exporter_port: 9115
     deploy_dir: "/tidb-deploy/monitored-9100"
     data_dir: "/tidb-data/monitor_9115"
     log_dir: "/tidb-deploy/monitor-9100/log"
   server_configs:
     pd:
       replication.location-labels: ["dc","logic","rack","host"]
   tidb_servers:
     - host: 10.90.2.210
       port: 4000
       status_port: 10080
       deploy_dir: "/tidb-deploy/tidb-4000"
       log_dir: "/tidb-deploy/tidb-4000/log"
   tikv_servers:
     - host: 10.90.2.111
       port: 20160
       status_port: 20180
       deploy_dir: "/tidb-deploy/tikv-20160"
       data_dir: "/tidb-data/tikv_data"
       config:
         server.labels: { dc: "dc2",logic: "logic3",rack: "r1",host: "10.90.2.111" }
     - host: 10.90.2.91
       port: 20160
       status_port: 20180
       deploy_dir: "/tidb-deploy/tikv-20160"
       data_dir: "/tidb-data/tikv_data"
       config:
         server.labels: { dc: "dc2",logic: "logic4",rack: "r1",host: "10.90.2.91" }
   pd_servers:
     - host: 10.90.2.83
       client_port: 2379
       peer_port: 2380
       deploy_dir: "/tidb-deploy/pd-2379"
       data_dir: "/tidb-data/pd_data"
       advertise_client_addr: "http://54.212.25.205:2379"
   monitoring_servers:
     - host: 10.90.2.107
       port: 10090
       deploy_dir: "/tidb-deploy/prometheus-10090"
       data_dir: "/tidb-data/prometheus_data"
   grafana_servers:
     - host: 10.90.2.107
       port: 3000
       deploy_dir: "/tidb-deploy/grafana-3000"
  ```

+ Replace the meta file by using the provisioned file.

  ```bash
  cp -rf majority-DC2.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml
  ```

### Step 7. Stop the TiDB cluster

+ Check TiDB cluster status.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ...
  ID                 Role        Host         Ports        OS/Arch  Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------  ------  --------                    ----------
  10.90.2.14:3000    grafana     10.90.2.14   3000         /        Up      -                           /tidb-deploy/grafana-3000
  10.90.2.53:2379    pd          10.90.2.53   2379/2380    /        Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.14:10090   prometheus  10.90.2.14   10090        /        Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.97:4000    tidb        10.90.2.97   4000/10080   /        Up      -                           /tidb-deploy/tidb-4000
  10.90.2.170:20160  tikv        10.90.2.170  20160/20180  /        N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.78:20160   tikv        10.90.2.78   20160/20180  /        N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 6
  ```

+ Stop the nodes in TiDB cluster. Be patient, this operation takes several minutes in our hands-on configuration (it's not for production).

  ```bash
  tiup cluster stop tidb-demo -y
  ```

  ```output
  ...
  Stopping component grafana
   Stopping instance 10.90.2.14
   Stop grafana 10.90.2.14:3000 success
  Stopping component prometheus
   Stopping instance 10.90.2.14
   Stop prometheus 10.90.2.14:10090 success
  Stopping component tidb
   Stopping instance 10.90.2.97
   Stop tidb 10.90.2.97:4000 success
  Stopping component tikv
   Stopping instance 10.90.2.170
   Stopping instance 10.90.2.78
   Stop tikv 10.90.2.170:20160 success
   Stop tikv 10.90.2.78:20160 success
  Stopping component pd
   Stopping instance 10.90.2.53
   Stop pd 10.90.2.53:2379 success
  Stopping component node_exporter
   Stopping instance 10.90.2.170
   Stopping instance 10.90.2.14
   Stopping instance 10.90.2.53
   Stopping instance 10.90.2.97
   Stopping instance 10.90.2.78
   Stop 10.90.2.170 success
   Stop 10.90.2.78 success
   Stop 10.90.2.53 success
   Stop 10.90.2.97 success
   Stop 10.90.2.14 success
  Stopping component blackbox_exporter
   Stopping instance 10.90.2.170
   Stopping instance 10.90.2.14
   Stopping instance 10.90.2.97
   Stopping instance 10.90.2.78
   Stopping instance 10.90.2.53
   Stop 10.90.2.78 success
   Stop 10.90.2.170 success
   Stop 10.90.2.97 success
   Stop 10.90.2.53 success
   Stop 10.90.2.14 success
  Stopped cluster `tidb-demo` successfully
  ```

+ Check TiDB nodes status, all pd, tidb, tikv and prometheus are down.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ...
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.2.82:3000    grafana     10.90.2.82   3000         linux/x86_64  Down    -                           /tidb-deploy/grafana-3000
  10.90.2.125:2379   pd          10.90.2.125  2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.82:10090   prometheus  10.90.2.82   10090/12020  linux/x86_64  Down    /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.173:4000   tidb        10.90.2.173  4000/10080   linux/x86_64  Down    -                           /tidb-deploy/tidb-4000
  10.90.2.129:20160  tikv        10.90.2.129  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.216:20160  tikv        10.90.2.216  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 6
  ```

### Step 8. Recover the Placement Driver

+ Install the pd-recover tool.

  ```bash
  tiup install pd-recover:v${LAB:TIDB_VERSION}
  ```

  ```output
  download https://tiup-mirrors.pingcap.com/pd-recover-v${LAB:TIDB_VERSION}-darwin-arm64.tar.gz 13.88 MiB / 13.88 MiB 100.00% 6.99 MiB/s
  ```

+ SSH to placement driver server which is located in DC2 to force start a new placement driver cluster.

+ Log in to pd. If prompted, answer `yes`.

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ2PD1_PRIVATE_IP}
  ```

+ Check the status of pd which is the synchronous mode of TiDB cluster before DC1 is lost, and it presents the current data consistency of TiDB.

  ```bash
  cat /tidb-data/pd_data/DR_STATE
  ```

  ```output
  {"state":"sync","state_id":1947}
  ```

+ Modify the pd's initial script by adding the `--force-new-cluster` parameter. Note: this parameter is used to start a new etcd cluster of pd.

  ```bash
  sed -i '/pd-server/ s#pd-server#pd-server --force-new-cluster#' /tidb-deploy/pd-2379/scripts/run_pd.sh
  ```

+ Exit to monitor OS.

  ```bash
  exit
  ```

  ```output
  logout
  Connection to 10.90.2.26 closed.
  ```

+ Start the cluster by starting the pd node.

  ```bash
  tiup cluster start tidb-demo -N ${HOST_AZ2PD1_PRIVATE_IP}:2379 -y
  ```

  ```output
  ...
  Starting cluster tidb-demo...
  + [ Serial ] - SSHKeySet: privateKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa, publicKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa.pub
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.26
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.254
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.182
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.49
  + [ Serial ] - StartCluster
  Starting component pd
   Starting instance 10.90.2.26:2379
   Start instance 10.90.2.26:2379 success
  Starting component node_exporter
   Starting instance 10.90.2.26
   Start 10.90.2.26 success
  Starting component blackbox_exporter
   Starting instance 10.90.2.26
   Start 10.90.2.26 success
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Started cluster `tidb-demo` successfully
  ```

+ Verify that pd is up and running.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

     To update this component:   tiup update cluster
     To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.94.179.113:2379/dashboard
  Grafana URL:        http://10.90.2.82:3000
  ID                 Role        Host         Ports        OS/Arch  Status        Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------  ------        --------                    ----------
  10.90.2.107:3000   grafana     10.90.2.107  3000         /        Down          -                           /tidb-deploy/grafana-3000
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /        Up|L          /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /        Down          /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /        Down          -                           /tidb-deploy/tidb-4000
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /        Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /        Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 6
  ```

+ Repair pd's metadata. After this step, the `alloc_id` in the storage automatically increases by a safe value of `100000000`.

  ```bash
  tiup pd-recover:v${LAB:TIDB_VERSION} --from-old-member --endpoints=http://${HOST_AZ2PD1_PRIVATE_IP}:2379
  ```

  ```output
  download https://tiup-mirrors.pingcap.com/pd-recover-v${LAB:TIDB_VERSION}-linux-amd64.tar.gz 15.87 MiB / 15.87 MiB 100.00% 3.78 MiB/s
  Checking updates for component pd-recover... Starting component pd-recover: /home/ec2-user/.tiup/components/pd-recover/v{LAB:TIDB_VERSION}/pd-recover --from-old-member --endpoints=http://10.90.2.83:2379
  recover success! please restart the PD cluster
  ```

+ Restart the pd instance.

  ```bash
  tiup cluster restart tidb-demo -N ${HOST_AZ2PD1_PRIVATE_IP}:2379 -y
  ```

  ```output
  ...
  + [ Serial ] - RestartCluster
  Stopping component pd
    Stopping instance 10.90.2.83
    Stop pd 10.90.2.83:2379 success
  Stopping component node_exporter
    Stopping instance 10.90.2.83
    Stop 10.90.2.83 success
  Stopping component blackbox_exporter
    Stopping instance 10.90.2.83
    Stop 10.90.2.83 success
  Starting component pd
    Starting instance 10.90.2.83:2379
    Start instance 10.90.2.83:2379 success
  Starting component node_exporter
    Starting instance 10.90.2.83
    Start 10.90.2.83 success
  Starting component blackbox_exporter
    Starting instance 10.90.2.83
    Start 10.90.2.83 success
  Restarted cluster `tidb-demo` successfully
  ```

### Step 9. Scale out one new Placement Driver server

+ Check the yaml file.

  ```bash
  cat scale-out-pd.yaml
  ```

  ```output
  pd_servers:
   - host: 10.90.2.102
     client_port: 2379
     peer_port: 2380
     deploy_dir: "/tidb-deploy/pd-2379"
     data_dir: "/tidb-data/pd_data"
     advertise_client_addr: "http://35.161.129.28:2379"
  ```

+ Scale out PD node.

  ```bash
  tiup cluster scale-out tidb-demo scale-out-pd.yaml --user ec2-user -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem -y
  ```

  ```output
  ...
  Starting component pd
   Starting instance 10.90.2.102:2379
   Start instance 10.90.2.102:2379 success
  Starting component node_exporter
   Starting instance 10.90.2.102
   Start 10.90.2.102 success
  Starting component blackbox_exporter
   Starting instance 10.90.2.102
   Start 10.90.2.102 success
  + Refresh components conifgs
   - Generate config pd -> 10.90.2.26:2379 ... Done
   - Generate config pd -> 10.90.2.102:2379 ... Done
   - Generate config tikv -> 10.90.2.254:20160 ... Done
   - Generate config tikv -> 10.90.2.182:20160 ... Done
   - Generate config tidb -> 10.90.2.49:4000 ... Done
   - Generate config prometheus -> 10.90.2.38:10090 ... Done
   - Generate config grafana -> 10.90.2.38:3000 ... Done
  + Reload prometheus and grafana
   - Reload prometheus -> 10.90.2.38:10090 ... Done
   - Reload grafana -> 10.90.2.38:3000 ... Done
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Scaled cluster `tidb-demo` out successfully
  ```

+ Verify that 2 PD nodes in DC2 are all up and running.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

     To update this component:   tiup update cluster
     To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.94.179.113:2379/dashboard
  Grafana URL:        http://10.90.2.82:3000
  ID                 Role        Host         Ports        OS/Arch       Status        Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------        --------                    ----------
  10.90.2.107:3000   grafana     10.90.2.107  3000         /             Up            -                           /tidb-deploy/grafana-3000
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /             Up|L          /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.96:2379    pd          10.90.2.96   2379/2380    linux/x86_64  Up            /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /             Down          /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /             Down          -                           /tidb-deploy/tidb-4000
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /             Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /             Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 7
  ```

### Step 10. Setup the cluster as `majority` mode

+ Verify cluster still in `dr-auto-sync` replication mode.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode
  ```

  ```output
  The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository.
  download https://tiup-mirrors.pingcap.com/ctl-v${LAB:TIDB_VERSION}-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 97.75 MiB/s
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.83:2379 config show replication-mode
  {
   "replication-mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label-key": "dc",
     "primary": "dc1",
     "dr": "dc2",
     "primary-replicas": 2,
     "dr-replicas": 1,
     "wait-store-timeout": "1m0s",
     "wait-recover-timeout": "0s",
     "pause-region-split": "false"
   }
  }
  ```

+ Check the `majority` rules file.

  ```bash
  cat majority-rules.json
  ```

  ```output
  [
   {
     "group_id": "pd",
     "group_index": 0,
     "group_override": false,
     "rules": [
       {
         "group_id": "pd",
         "id": "voters",
         "start_key": "",
         "end_key": "",
         "role": "voter",
         "count": 3,
         "location_labels": ["dc","logic","rack", "host"]
       }
     ]
   }
  ]
  ```

+ Load the `majority` placement rules configuration.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/majority-rules.json"
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/majority-rules.json
  "Update rules and groups successfully."
  ```

+ Check placement rules is loaded.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules show -u ${HOST_AZ2PD1_PRIVATE_IP}:2379
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules show -u 10.90.2.125:2379
  [
   {
     "group_id": "pd",
     "id": "voters",
     "start_key": "",
     "end_key": "",
     "role": "voter",
     "is_witness": false,
     "count": 3,
     "location_labels": [
       "dc",
       "logic",
       "rack",
       "host"
     ],
     "create_timestamp": 1734071450
   }
  ]
  ```

+ Enable the `majority` configuration.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode majority
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set max-replicas 3
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 config set replication-mode majority
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 config set max-replicas 3
  Success!
  ```

+ Verify cluster `majority` replication mode is effective.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 config show replication-mode
  {
   "replication-mode": "majority",
   "dr-auto-sync": {
     "label-key": "dc",
     "primary": "dc1",
     "dr": "dc2",
     "primary-replicas": 2,
     "dr-replicas": 1,
     "wait-store-timeout": "1m0s",
     "wait-recover-timeout": "0s",
     "pause-region-split": "false"
   }
  }
  ```

### Step 11. Remove the unavailable TiKV nodes

+ Start TiKV nodes.

  ```bash
  tiup cluster start tidb-demo -R tikv -y
  ```

  ```output
  ...
  Starting component tikv
   Starting instance 10.90.2.129:20160
   Starting instance 10.90.2.216:20160
   Start instance 10.90.2.216:20160 success
   Start instance 10.90.2.129:20160 success
  Starting component node_exporter
   Starting instance 10.90.2.129
   Starting instance 10.90.2.216
   Start 10.90.2.216 success
   Start 10.90.2.129 success
  Starting component blackbox_exporter
   Starting instance 10.90.2.129
   Starting instance 10.90.2.216
   Start 10.90.2.216 success
   Start 10.90.2.129 success
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Started cluster `tidb-demo` successfully
  ...
  ```

+ Wait for the 2 TiKV nodes are up.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ...
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.2.82:3000    grafana     10.90.2.82   3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.125:2379   pd          10.90.2.125  2379/2380    linux/x86_64  Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.135:2379   pd          10.90.2.135  2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.82:10090   prometheus  10.90.2.82   10090/12020  linux/x86_64  Down    /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.173:4000   tidb        10.90.2.173  4000/10080   linux/x86_64  Down    -                           /tidb-deploy/tidb-4000
  10.90.2.129:20160  tikv        10.90.2.129  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.216:20160  tikv        10.90.2.216  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 7
  ```

+ Remove the unavailable TiKV nodes.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores --auto-detect
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.110:2379 unsafe remove-failed-stores --auto-detect
  Success! Request has been accepted.
  ```

+ Show the status of the removal operation. After a while, you can confirm the process is completed by identifying the following description: `"info": "Unsafe recovery Finished"` at the end of the command result.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores show
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 unsafe remove-failed-stores show
  [
   {
     "info": "Unsafe recovery enters collect report stage",
     "time": "2024-12-13 07:01:51.714",
     "details": [
       "auto detect mode with no specified Failed stores"
     ]
   },
   {
     "info": "Unsafe recovery enters force leader stage",
     "time": "2024-12-13 07:01:54.000",
     "actions": {
       "store 1": [
         "force leader on regions: 7, 9, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 101, 104, 107, 110, 113, 116, 119, 122, 125, 128, 131, 134, 137, 140, 143, 146, 149, 152, 155, 158, 209, 213, 217, 221, 225, 229, 233, 237, 241, 245, 249, 253, 1978, 2013, 2018, 1993, 2043, 2048, 1983, 2023, 2028, 1988, 2033, 2038, 1998, 2063, 2068, 2008, 2053, 2058, 35, 31, 27, 23, 19, 17, 11, 13, 3"
       ],
       "store 16": [
         "force leader on regions: "
       ]
     }
   },
   {
     "info": "Unsafe recovery enters demote Failed voter stage",
     "time": "2024-12-13 07:02:04.586",
     "actions": {
       "store 1": [
         "region 7 demotes peers { id:75 store_id:2 }, { id:170 store_id:15 }",
         ...
         "region 3 demotes peers { id:77 store_id:2 }, { id:208 store_id:15 }"
       ]
     }
   },
   {
     "info": "Unsafe recovery Finished",
     "time": "2024-12-13 07:02:14.002",
     "details": [
       "affected table ids: 74, 94, 110, 24, 58, 78, 92, 107, 281474976710649, 56, 82, 84, 106, 108, 109, 281474976710652, 281474976710654, 52, 111, 50, 66, 281474976710653, 48, 98, 54, 38, 42, 46, 62, 72, 76, 6, 281474976710651, 86, 28, 30, 40, 68, 96, 20, 60, 4, 18, 34, 80, 12, 10, 32, 70, 8, 64, 22, 36, 88, 90, 102, 14, 26, 44, 281474976710650, 16",
       "no newly created empty regions"
     ]
   }
  ]
  ```

### Step 12. Start the recovered TiDB cluster

+ Start the tidb and prometheus instances.

  ```bash
  tiup cluster start tidb-demo -R tidb,prometheus -y
  ```

  ```output
  Starting component tidb
   Starting instance 10.90.2.49:4000
   Start instance 10.90.2.49:4000 success
  Starting component prometheus
   Starting instance 10.90.2.38:10090
   Start instance 10.90.2.38:10090 success
  Starting component node_exporter
   Starting instance 10.90.2.38
   Starting instance 10.90.2.49
   Start 10.90.2.49 success
   Start 10.90.2.38 success
  Starting component blackbox_exporter
   Starting instance 10.90.2.38
   Starting instance 10.90.2.49
   Start 10.90.2.49 success
   Start 10.90.2.38 success
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Started cluster `tidb-demo` successfully
  ```

+ Check the status of the TiDB cluster. The state of all the nodes in DC2 is up.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

     To update this component:   tiup update cluster
     To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.94.179.113:2379/dashboard
  Grafana URL:        http://10.90.2.82:3000
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.2.82:3000    grafana     10.90.2.82   3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.125:2379   pd          10.90.2.125  2379/2380    linux/x86_64  Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.135:2379   pd          10.90.2.135  2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.82:10090   prometheus  10.90.2.82   10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.173:4000   tidb        10.90.2.173  4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.2.129:20160  tikv        10.90.2.129  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.216:20160  tikv        10.90.2.216  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 7
  ```

+ Verify the replication mode of the TiDB cluster.

  ```bash
  curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "majority",
   "dr-auto-sync": {
     "label_key": "",
     "state": "",
     "acid_consistent": false
   }
  }
  ```

### Step 13. Verify that the sysbench workload is recovered

+ Switch to [Terminal 1](ssh://az1monitor1-ssh#1), the QPS of the sysbench workload is still 0. Because tidb server located in DC1 is unavailable.

  ```output
  ...
  [ 1697s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 1698s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 1699s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ```

+ Switch to [Terminal 2](ssh://az2monitor1-ssh#2), the QPS of the sysbench workload back to normal.

  ```output
  ...
  [ 1618s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 1619s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 6.00
  [ 1620s ] thds: 6 tps: 321.06 qps: 6505.17 (r/w/o: 4564.82/1103.20/837.15) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
  [ 1621s ] thds: 6 tps: 313.95 qps: 6261.93 (r/w/o: 4378.25/1098.81/784.87) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
  ```

### Step 14. Enable DR Auto-Sync again

+ After DC1 recovers, we need to recover the TiDB cluster back to DR Auto-Sync deployment.

+ Switch to [Terminal 3](ssh://az2monitor1-ssh#3), rejoin the lost nodes back into the TiDB cluster. Check the yaml file.

  ```bash
  cat scale-out-DC1.yaml
  ```

  ```output
  tidb_servers:
   - host: 10.90.1.58
     port: 4000
     status_port: 10080
     deploy_dir: "/tidb-deploy/tidb-4000"
     log_dir: "/tidb-deploy/tidb-4000/log"
  tikv_servers:
   - host: 10.90.1.219
     port: 20160
     status_port: 20180
     deploy_dir: "/tidb-deploy/tikv-20160"
     data_dir: "/tidb-data/tikv_data"
     config:
       server.labels: { dc: "dc1",logic: "logic1",rack: "r1",host: "10.90.1.219" }
   - host: 10.90.1.71
     port: 20160
     status_port: 20180
     deploy_dir: "/tidb-deploy/tikv-20160"
     data_dir: "/tidb-data/tikv_data"
     config:
       server.labels: { dc: "dc1",logic: "logic2",rack: "r1",host: "10.90.1.71" }
  pd_servers:
   - host: 10.90.1.95
     client_port: 2379
     peer_port: 2380
     deploy_dir: "/tidb-deploy/pd-2379"
     data_dir: "/tidb-data/pd_data"
     advertise_client_addr: "http://34.214.79.226:2379"
   - host: 10.90.1.55
     client_port: 2379
     peer_port: 2380
     deploy_dir: "/tidb-deploy/pd-2379"
     data_dir: "/tidb-data/pd_data"
     advertise_client_addr: "http://54.202.186.157:2379"
  monitoring_servers:
   - host: 10.90.1.221
     port: 10090
     deploy_dir: "/tidb-deploy/prometheus-10090"
     data_dir: "/tidb-data/prometheus_data"
  grafana_servers:
   - host: 10.90.1.221
     port: 3000
     deploy_dir: "/tidb-deploy/grafana-3000"
  ```

+ Scale out nodes in DC1.

  ```bash
  tiup cluster scale-out tidb-demo scale-out-DC1.yaml --user ec2-user -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem -y
  ```

  ```output
  ...
  Starting component pd
   Starting instance 10.90.1.95:2379
   Start instance 10.90.1.95:2379 success
   Starting instance 10.90.1.55:2379
   Start instance 10.90.1.55:2379 success
  Starting component tikv
   Starting instance 10.90.1.71:20160
   Starting instance 10.90.1.219:20160
   Start instance 10.90.1.71:20160 success
   Start instance 10.90.1.219:20160 success
  Starting component tidb
   Starting instance 10.90.1.58:4000
   Start instance 10.90.1.58:4000 success
  Starting component prometheus
   Starting instance 10.90.1.221:10090
   Start instance 10.90.1.221:10090 success
  Starting component grafana
   Starting instance 10.90.1.221:3000
   Start instance 10.90.1.221:3000 success
  Starting component node_exporter
   Starting instance 10.90.1.55
   Starting instance 10.90.1.58
   Starting instance 10.90.1.221
   Starting instance 10.90.1.95
   Starting instance 10.90.1.219
   Starting instance 10.90.1.71
   Start 10.90.1.71 success
   Start 10.90.1.95 success
   Start 10.90.1.219 success
   Start 10.90.1.55 success
   Start 10.90.1.221 success
   Start 10.90.1.58 success
  Starting component blackbox_exporter
   Starting instance 10.90.1.55
   Starting instance 10.90.1.58
   Starting instance 10.90.1.221
   Starting instance 10.90.1.95
   Starting instance 10.90.1.219
   Starting instance 10.90.1.71
   Start 10.90.1.219 success
   Start 10.90.1.95 success
   Start 10.90.1.71 success
   Start 10.90.1.55 success
   Start 10.90.1.221 success
   Start 10.90.1.58 success
  + Refresh components conifgs
   - Generate config pd -> 10.90.2.83:2379 ... Done
   - Generate config pd -> 10.90.2.96:2379 ... Done
   - Generate config pd -> 10.90.1.95:2379 ... Done
   - Generate config pd -> 10.90.1.55:2379 ... Done
   - Generate config tikv -> 10.90.2.111:20160 ... Done
   - Generate config tikv -> 10.90.2.91:20160 ... Done
   - Generate config tikv -> 10.90.1.219:20160 ... Done
   - Generate config tikv -> 10.90.1.71:20160 ... Done
   - Generate config tidb -> 10.90.2.210:4000 ... Done
   - Generate config tidb -> 10.90.1.58:4000 ... Done
   - Generate config prometheus -> 10.90.2.107:10090 ... Done
   - Generate config prometheus -> 10.90.1.221:10090 ... Done
   - Generate config grafana -> 10.90.2.107:3000 ... Done
   - Generate config grafana -> 10.90.1.221:3000 ... Done
  + Reload prometheus and grafana
   - Reload prometheus -> 10.90.2.107:10090 ... Done
   - Reload grafana -> 10.90.2.107:3000 ... Done
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Scaled cluster `tidb-demo` out successfully
  ```

+ Check the TiDB cluster status, all the nodes are recovered.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.1.221:3000   grafana     10.90.1.221  3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.107:3000   grafana     10.90.2.107  3000         /             Up      -                           /tidb-deploy/grafana-3000
  10.90.1.55:2379    pd          10.90.1.55   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.95:2379    pd          10.90.1.95   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /             Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.96:2379    pd          10.90.2.96   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.221:10090  prometheus  10.90.1.221  10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /             Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.58:4000    tidb        10.90.1.58   4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /             Up      -                           /tidb-deploy/tidb-4000
  10.90.1.219:20160  tikv        10.90.1.219  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.71:20160   tikv        10.90.1.71   20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 14
  ```

+ Load the DR Auto-Sync placement rules configuration.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/dr-anto-sync-rules.json"
  ```

  ```output
  The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository.
  download https://tiup-mirrors.pingcap.com/ctl-v${LAB:TIDB_VERSION}-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 103.24 MiB/s
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/dr-anto-sync-rules.json
  "Update rules and groups successfully."
  ```

+ Install `jq` tool.

  ```bash
  sudo yum install jq -y
  ```

  ```output
  ...
  Installed:
   jq.x86_64 0:1.5-1.amzn2.0. 2

  Dependency Installed:
   oniguruma.x86_64 0:5.9.6-1.amzn2.0. 7

  Complete!
  ```

+ Check for TiKV regions without learner peers. Verify that there are no regions without learner peers and ensure data synchronization between voter and learner roles. This step ensures that the cluster's data is consistently replicated across all roles, which is essential for maintaining high availability and failover readiness.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 region --jq='.regions[] | select(.peers | any(.role_name=="Learner") | not) |  {id: .id, peers: [.peers]}'
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://34.215.73.186:2379 region --jq=.regions[] | select(.peers | any(.role_name=="Learner") | not) |  {id: .id, peers: [.peers]}
  ```

+ Enable the DR Auto-Sync configuration.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync label-key dc
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary dc1
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr dc2
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary-replicas 2
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr-replicas 1
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync label-key dc
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary dc1
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr dc2
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary-replicas 2
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr-replicas 1
  Success! The replication mode config is updated."
  ```

+ Verify that the configuration is effective.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379:2379 config show replication-mode
  {
   "replication-mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label-key": "dc",
     "primary": "dc1",
     "dr": "dc2",
     "primary-replicas": 2,
     "dr-replicas": 1,
     "wait-store-timeout": "1m0s",
     "wait-recover-timeout": "0s",
     "pause-region-split": "false"
   }
  }
  ```

### Step 15. Verify that the replication status is back to `sync`

+ Check the current replication status of the cluster. You might see `sync_recover` mode or `recover_progress` before the status becomes to `sync` mode temporarily.

  ```bash
  curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync_recover",
     "state_id": 2003,
     "acid_consistent": false,
     "total_regions": 60,
     "synced_regions": 37,
     "recover_progress": 0.98333335
   }
  }
  ```

  + Finally, you will get the `sync` state.

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync",
     "state_id": 378,
     "acid_consistent": true
   }
  }
  ```

+ Switch to [Terminal 1](ssh://az1monitor1-ssh#1), the QPS of the sysbench workload is recovered.

  ```output
  ...
  [ 2309s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 2310s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 2311s ] thds: 6 tps: 105.01 qps: 2170.15 (r/w/o: 1530.11/416.03/224.02) lat (ms,95%): 100000.00 err/s: 0.00 reconn/s: 6.00
  [ 2312s ] thds: 6 tps: 193.97 qps: 3869.44 (r/w/o: 2705.61/763.89/399.94) lat (ms,95%): 34.33 err/s: 0.00 reconn/s: 0.00
  ```

### Step 16. Transfer the leaders of PD and TiKV to data center 1

+ Switch to [Terminal 3](ssh://az2monitor1-ssh#3), scale in the 2nd placement driver node in data center 2.

  ```bash
  tiup cluster scale-in tidb-demo -N "${HOST_AZ2PD2_PRIVATE_IP}:2379" -y
  ```

  ```output
  ...
  Scaled cluster `tidb-demo` in successfully
  ```

+ You can execute this command to try to transfer pd leader to DC1 whatever the previous step's result is.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 member leader transfer "pd-${HOST_AZ1PD1_PRIVATE_IP}-2379"
  ```

  ```output
  The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository.
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379
  Success!
  ```

+ Check the TiDB cluster status. Ensure the leader of pd is located in DC1 and the pd nodes count goes to 3.

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.1.221:3000   grafana     10.90.1.221  3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.107:3000   grafana     10.90.2.107  3000         /             Up      -                           /tidb-deploy/grafana-3000
  10.90.1.55:2379    pd          10.90.1.55   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.95:2379    pd          10.90.1.95   2379/2380    linux/x86_64  Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /             Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.221:10090  prometheus  10.90.1.221  10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /             Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.58:4000    tidb        10.90.1.58   4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /             Up      -                           /tidb-deploy/tidb-4000
  10.90.1.219:20160  tikv        10.90.1.219  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.71:20160   tikv        10.90.1.71   20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 13
  ```

+ By examining the region distribution of each role, we can confirm that the DR Auto-Sync configuration has successfully taken effect on TiKV nodes.

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 store
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.83:2379 store
  {
   "count": 4,
   "stores": [
     {
       "store": {
         "id": 1,
         "address": "10.90.2.111:20160",
         ...
       },
       "status": {
         "leader_count": 0,
         "region_count": 79,
         ...
       }
     },
     {
       "store": {
         "id": 14,
         "address": "10.90.2.91:20160",
         ...
       },
       "status": {
         "leader_count": 0,
         "region_count": 79,
         "learner_count": 79,
         ...
       }
     },
     {
       "store": {
         "id": 2001,
         "address": "10.90.1.219:20160",
         ...
       },
       "status": {
         "leader_count": 42,
         "region_count": 79,
         ...
       }
     },
     {
       "store": {
         "id": 2002,
         "address": "10.90.1.71:20160",
         ...
       },
       "status": {
         "leader_count": 37,
         "region_count": 79,
         ...
       }
     }
   ]
  }
  ```

+ From the information above, we can identify the role of each TiKV node from the region distribution (Your store id might differ from the example shown below):
  + Store 1 is `follower`: "leader_count": 0, "region_count": 79
  + Store 14 is `learner`: "leader_count": 0, "region_count": 79, "learner_count": 79
  + Store 2001 is `voter`: "leader_count": 42, "region_count": 79
  + Store 2002 is `voter`: "leader_count": 37, "region_count": 79

+ Go to [Terminal 1](ssh://az1monitor1-ssh#1), terminate the sysbench thread, and close Terminal 1.

+ Go to [Terminal 2](ssh://az2monitor1-ssh#2), terminate the sysbench thread, and close Terminal 2.

+ Close Terminal 3.

### Summary for Module 3

+ Through this module, we observed that even if the primary data center experiences a complete failure, we can still successfully restore the TiDB cluster and ensure RPO=0. Roughly speaking, the recovery operations include the following steps:
  + Recover the only remaining PD in DC2 and use it to create a new cluster.
  + Scale out a new PD in DC2 to form a new Raft group.
  + Modify the cluster mode to majority mode.
  + Remove the unavailable TiKV nodes located in DC1.
  + Restart all cluster nodes and run with two surviving TiKV nodes. Observe the recovery of application read and write operations.
  + Fix the failure.
  + Initialize the nodes in DC1 and reintegrate them into the TiDB cluster.
  + Modify the cluster configuration to revert back to DR Auto-Sync mode.