## 模块 3: 通过终止 DC1 中的所有节点模拟主数据中心故障

本练习的目标是模拟主数据中心完全不可用，并恢复其可用性。

当 DC1 完全故障时，DC2 仍然保留着最新的数据在跟随 TiKV 实例中。结合备份的 learner TiKV 实例，可以迅速恢复一个两副本集群，继续提供服务。

![60%](https://lab-static.pingcap.com/quick-demo/demo-303-7-en.png)

我们需要通过灾备数据中心中的 PD 启动一个新的 TiDB 集群（保留数据，但有一个新的集群 ID）。另外，我们需要扩展一个 PD 节点（预留的备用 PD 节点），并将灾备数据中心中的 learner TiKV 节点转换为投票 TiKV，以便与另一个 TiKV 节点组成一个功能齐全的 Raft group。

### 步骤 1. 连接到 DC1 中的监视服务器

+ 打开 [终端1](ssh://az1monitor1-ssh#1)，连接到数据中心1的 monitor1。

### 步骤 2. 启动工作负载

+ 在 DC1 中启动 sysbench 工作负载。

  ```bash
  sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run
  ```

  ```output
  sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2)

  Running the test with following options:
  Number of threads: 1
  Report intermediate results every 1 second(s)
  Initializing random number generator from current time

  Initializing worker threads...

  Threads started!

  [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00
  [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00
  ...
  ```

+ 打开 [终端2](ssh://az2monitor1-ssh#2)，连接到数据中心2的 monitor1。在 DC2 启动 `sysbench` 工作负载。

+ 为 DC2 设置 sysbench 配置。

  ```bash
  cat << EOF > sysbench_config
  mysql-host=${HOST_AZ2DB1_PRIVATE_IP}
  mysql-port=4000
  mysql-user=root
  mysql-password=
  mysql-db=sbtest
  time=3600
  threads=6
  report-interval=1
  EOF
  cat sysbench_config
  ```

  ```output
  mysql-host=10.90.2.125
  mysql-port=4000
  mysql-user=root
  mysql-password=
  mysql-db=sbtest
  time=3600
  threads=16
  report-interval=1
  ```

+ 在 DC2 中启动 sysbench 工作负载。

  ```bash
  sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run
  ```

  ```output
  Threads started!

  [ 1s ] thds: 6 tps: 316.47 qps: 6401.24 (r/w/o: 4486.46/1158.05/756.73) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
  [ 2s ] thds: 6 tps: 330.09 qps: 6611.89 (r/w/o: 4631.33/1171.34/809.23) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
  ...
  ```

### 步骤 3. 验证复制状态是否同步

+ 打开 [终端3](ssh://az2monitor1-ssh#3)，连接到数据中心2的 monitor1，用于在 DC2 启动 `sysbench` 工作负载，并验证数据复制状态是否为 `sync` 模式。

+ 验证数据复制状态。

  ```bash
  curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync",
     "state_id": 1006,
     "acid_consistent": true
   }
  }
  ```

### 步骤 4. 模拟主数据中心完全故障

+ 模拟主数据中心的完全故障，通过终止 TiDB 集群中 DC1 节点上所有 TiDB 进程来模拟。

+ 终止所有进程并移除 DC1 中每个 TiDB、TiKV 和 PD 节点的数据目录。如果有提示，回答 `yes`。

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1DB1_PRIVATE_IP} "sudo systemctl stop tidb-4000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1KV1_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1KV2_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1PD1_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1PD2_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

+ 终止监控的进程并移除数据目录。

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ1MONITOR1_PRIVATE_IP} "sudo systemctl stop prometheus-10090.service; sudo systemctl stop grafana-3000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;"
  ```

### 步骤 5. 观察应用程序和 TiDB 的状态

+ 观察 TiDB 集群中的节点状态。所有的 PD 服务器状态为 `Down`，所有的 TiKV 状态为 `N/A`，且其中一台 TiDB 服务器为 `Down`。只有位于 DC2 的 TiDB 服务器仍然存活。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

  To update this component:   tiup update cluster
  To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Grafana URL:        http://10.90.1.182:3000,http://10.90.2.252:3000
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.1.182:3000   grafana     10.90.1.182  3000         linux/x86_64  Down    -                           /tidb-deploy/grafana-3000
  10.90.2.252:3000   grafana     10.90.2.252  3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.1.213:2379   pd          10.90.1.213  2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.90:2379    pd          10.90.1.90   2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.65:2379    pd          10.90.2.65   2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.182:10090  prometheus  10.90.1.182  10090/12020  linux/x86_64  Down    /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.252:10090  prometheus  10.90.2.252  10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.196:4000   tidb        10.90.1.196  4000/10080   linux/x86_64  Down    -                           /tidb-deploy/tidb-4000
  10.90.2.226:4000   tidb        10.90.2.226  4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.1.228:20160  tikv        10.90.1.228  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.91:20160   tikv        10.90.1.91   20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.202:20160  tikv        10.90.2.202  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.206:20160  tikv        10.90.2.206  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 13
  ```

+ 观察 TiDB 同步模式。由于无法访问位于 DC1 的 PD leader，我们需要访问位于 DC2 的 PD follower。此时操作超时，因为请求无法重定向到 leader 节点。

  ```bash
  curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  [PD:apiutil:ErrRedirectNoLeader]redirect finds no leader
  ```

+ 切换到[终端 1](ssh://az1monitor1-ssh#1)。可以看到工作负载的 QPS 为 0，因为 TiDB 集群不可用。

  ```output
  ...
  [ 57s ] thds: 6 tps: 193.99 qps: 3834.86 (r/w/o: 2679.90/665.98/488.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
  [ 58s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 59s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ...
  ```

+ 切换到[终端 2](ssh://az2monitor1-ssh#2)。可以发现工作负载的 QPS 为 0，因为 TiDB 集群不可用。

  ```output
  ...
  [ 34s ] thds: 6 tps: 116.00 qps: 2313.99 (r/w/o: 1624.99/385.00/304.00) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
  [ 35s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 36s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ...
  ```

### 步骤 6. 仅使用幸存节点重塑 TiDB 集群

+ 切换到[终端 3](ssh://az2monitor1-ssh#3)，使用仅包含位于 DC2（AZ2）节点的准备好的 meta 文件覆盖 TiUP 的 meta 文件。

  ```bash
  cp /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml.backup
  ```

+ 检查 yaml 文件 `majority-DC2.yaml`。

  ```bash
  cat majority-DC2.yaml
  ```

  ```output
  user: ec2-user
  tidb_version: v${LAB:TIDB_VERSION}
  topology:
   global:
     user: "ec2-user"
     ssh_port: 22
     deploy_dir: "/tidb-deploy"
     data_dir: "/tidb-data"
     arch: "amd64"
   monitored:
     node_exporter_port: 9100
     blackbox_exporter_port: 9115
     deploy_dir: "/tidb-deploy/monitored-9100"
     data_dir: "/tidb-data/monitor_9115"
     log_dir: "/tidb-deploy/monitor-9100/log"
   server_configs:
     pd:
       replication.location-labels: ["dc","logic","rack","host"]
   tidb_servers:
     - host: 10.90.2.210
       port: 4000
       status_port: 10080
       deploy_dir: "/tidb-deploy/tidb-4000"
       log_dir: "/tidb-deploy/tidb-4000/log"
   tikv_servers:
     - host: 10.90.2.111
       port: 20160
       status_port: 20180
       deploy_dir: "/tidb-deploy/tikv-20160"
       data_dir: "/tidb-data/tikv_data"
       config:
         server.labels: { dc: "dc2",logic: "logic3",rack: "r1",host: "10.90.2.111" }
     - host: 10.90.2.91
       port: 20160
       status_port: 20180
       deploy_dir: "/tidb-deploy/tikv-20160"
       data_dir: "/tidb-data/tikv_data"
       config:
         server.labels: { dc: "dc2",logic: "logic4",rack: "r1",host: "10.90.2.91" }
   pd_servers:
     - host: 10.90.2.83
       client_port: 2379
       peer_port: 2380
       deploy_dir: "/tidb-deploy/pd-2379"
       data_dir: "/tidb-data/pd_data"
       advertise_client_addr: "http://54.212.25.205:2379"
   monitoring_servers:
     - host: 10.90.2.107
       port: 10090
       deploy_dir: "/tidb-deploy/prometheus-10090"
       data_dir: "/tidb-data/prometheus_data"
   grafana_servers:
     - host: 10.90.2.107
       port: 3000
       deploy_dir: "/tidb-deploy/grafana-3000"
  ```

+ 使用准备好的文件替换 `meta` 文件。

  ```bash
  cp -rf majority-DC2.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml
  ```

### 步骤 7. 停止 TiDB 集群

+ 检查 TiDB 集群状态。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ...
  ID                 Role        Host         Ports        OS/Arch  Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------  ------  --------                    ----------
  10.90.2.14:3000    grafana     10.90.2.14   3000         /        Up      -                           /tidb-deploy/grafana-3000
  10.90.2.53:2379    pd          10.90.2.53   2379/2380    /        Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.14:10090   prometheus  10.90.2.14   10090        /        Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.97:4000    tidb        10.90.2.97   4000/10080   /        Up      -                           /tidb-deploy/tidb-4000
  10.90.2.170:20160  tikv        10.90.2.170  20160/20180  /        N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.78:20160   tikv        10.90.2.78   20160/20180  /        N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 6
  ```

+ 停止 TiDB 集群中的节点。请耐心等待，此操作在我们的动手配置中需要几分钟时间（这不是生产环境配置）。

  ```bash
  tiup cluster stop tidb-demo -y
  ```

  ```output
  ...
  Stopping component grafana
  Stopping instance 10.90.2.14
  Stop grafana 10.90.2.14:3000 success
  Stopping component prometheus
  Stopping instance 10.90.2.14
  Stop prometheus 10.90.2.14:10090 success
  Stopping component tidb
  Stopping instance 10.90.2.97
  Stop tidb 10.90.2.97:4000 success
  Stopping component tikv
  Stopping instance 10.90.2.170
  Stopping instance 10.90.2.78
  Stop tikv 10.90.2.170:20160 success
  Stop tikv 10.90.2.78:20160 success
  Stopping component pd
  Stopping instance 10.90.2.53
  Stop pd 10.90.2.53:2379 success
  Stopping component node_exporter
  Stopping instance 10.90.2.170
  Stopping instance 10.90.2.14
  Stopping instance 10.90.2.53
  Stopping instance 10.90.2.97
  Stopping instance 10.90.2.78
  Stop 10.90.2.170 success
  Stop 10.90.2.78 success
  Stop 10.90.2.53 success
  Stop 10.90.2.97 success
  Stop 10.90.2.14 success
  Stopping component blackbox_exporter
  Stopping instance 10.90.2.170
  Stopping instance 10.90.2.14
  Stopping instance 10.90.2.97
  Stopping instance 10.90.2.78
  Stopping instance 10.90.2.53
  Stop 10.90.2.78 success
  Stop 10.90.2.170 success
  Stop 10.90.2.97 success
  Stop 10.90.2.53 success
  Stop 10.90.2.14 success
  Stopped cluster `tidb-demo` successfully
  ```

+ 检查 TiDB 节点状态，所有 PD、TiDB、TiKV 和 Prometheus 都为 `Down`。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ...
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.2.82:3000    grafana     10.90.2.82   3000         linux/x86_64  Down    -                           /tidb-deploy/grafana-3000
  10.90.2.125:2379   pd          10.90.2.125  2379/2380    linux/x86_64  Down    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.82:10090   prometheus  10.90.2.82   10090/12020  linux/x86_64  Down    /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.173:4000   tidb        10.90.2.173  4000/10080   linux/x86_64  Down    -                           /tidb-deploy/tidb-4000
  10.90.2.129:20160  tikv        10.90.2.129  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.216:20160  tikv        10.90.2.216  20160/20180  linux/x86_64  N/A     /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 6
  ```

### 步骤 8. 恢复 Placement Driver

+ 安装 `pd-recover` 工具。

  ```bash
  tiup install pd-recover:v${LAB:TIDB_VERSION}
  ```

  ```output
  download https://tiup-mirrors.tidb.io/pd-recover-v${LAB:TIDB_VERSION}-darwin-arm64.tar.gz 13.88 MiB / 13.88 MiB 100.00% 6.99 MiB/s
  ```

+ 通过 SSH 登录位于 DC2 的 PD 服务器，强制启动一个新集群。登录到 PD。如果出现提示，请回答 `yes`.

  ```bash
  ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${HOST_AZ2PD1_PRIVATE_IP}
  ```

+ 检查 PD 的状态，其显示 TiDB 集群在 DC1 丢失前的同步模式，以及当前 TiDB 的数据一致性。

  ```bash
  cat /tidb-data/pd_data/DR_STATE
  ```

  ```output
  {"state":"sync","state_id":1947}
  ```

+ 通过添加 `--fource-new-cluster` 参数修改 PD 的启动脚本。

  ```bash
  sed -i '/pd-server/ s#pd-server#pd-server --force-new-cluster#' /tidb-deploy/pd-2379/scripts/run_pd.sh
  ```

+ 从 PD 退出到 TiUP 中控机。

  ```bash
  exit
  ```

  ```output
  logout
  Connection to 10.90.2.26 closed.
  ```

+ 通过启动 PD 节点创建一个新集群。

  ```bash
  tiup cluster start tidb-demo -N ${HOST_AZ2PD1_PRIVATE_IP}:2379 -y
  ```

  ```output
  ...
  Starting cluster tidb-demo...
  + [ Serial ] - SSHKeySet: privateKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa, publicKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa.pub
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.26
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.254
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.182
  + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.49
  + [ Serial ] - StartCluster
  Starting component pd
  Starting instance 10.90.2.26:2379
  Start instance 10.90.2.26:2379 success
  Starting component node_exporter
  Starting instance 10.90.2.26
  Start 10.90.2.26 success
  Starting component blackbox_exporter
  Starting instance 10.90.2.26
  Start 10.90.2.26 success
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Started cluster `tidb-demo` successfully
  ```

+ 确认 PD 已启动并运行正常。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

     To update this component:   tiup update cluster
     To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.94.179.113:2379/dashboard
  Grafana URL:        http://10.90.2.82:3000
  ID                 Role        Host         Ports        OS/Arch  Status        Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------  ------        --------                    ----------
  10.90.2.107:3000   grafana     10.90.2.107  3000         /        Down          -                           /tidb-deploy/grafana-3000
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /        Up|L          /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /        Down          /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /        Down          -                           /tidb-deploy/tidb-4000
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /        Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /        Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 6
  ```

+ 修复 pd 的元数据。此操作完成后，存储中的 `alloc_id` 会自动安全地增加 `100000000`。

  ```bash
  tiup pd-recover:v${LAB:TIDB_VERSION} --from-old-member --endpoints=http://${HOST_AZ2PD1_PRIVATE_IP}:2379
  ```

  ```output
  download https://tiup-mirrors.pingcap.com/pd-recover-v${LAB:TIDB_VERSION}-linux-amd64.tar.gz 15.87 MiB / 15.87 MiB 100.00% 3.78 MiB/s
  Checking updates for component pd-recover... Starting component pd-recover: /home/ec2-user/.tiup/components/pd-recover/v{LAB:TIDB_VERSION}/pd-recover --from-old-member --endpoints=http://10.90.2.83:2379
  recover success! please restart the PD cluster
  ```

+ 重启 pd 实例.

  ```bash
  tiup cluster restart tidb-demo -N ${HOST_AZ2PD1_PRIVATE_IP}:2379 -y
  ```

  ```output
  ...
  + [ Serial ] - RestartCluster
  Stopping component pd
    Stopping instance 10.90.2.83
    Stop pd 10.90.2.83:2379 success
  Stopping component node_exporter
    Stopping instance 10.90.2.83
    Stop 10.90.2.83 success
  Stopping component blackbox_exporter
    Stopping instance 10.90.2.83
    Stop 10.90.2.83 success
  Starting component pd
    Starting instance 10.90.2.83:2379
    Start instance 10.90.2.83:2379 success
  Starting component node_exporter
    Starting instance 10.90.2.83
    Start 10.90.2.83 success
  Starting component blackbox_exporter
    Starting instance 10.90.2.83
    Start 10.90.2.83 success
  Restarted cluster `tidb-demo` successfully
  ```

### 步骤 9. 扩展一台新的 Placement Driver 服务器

+ 检查 YAML 文件。

  ```bash
  cat scale-out-pd.yaml
  ```

  ```output
  pd_servers:
   - host: 10.90.2.102
     client_port: 2379
     peer_port: 2380
     deploy_dir: "/tidb-deploy/pd-2379"
     data_dir: "/tidb-data/pd_data"
     advertise_client_addr: "http://35.161.129.28:2379"
  ```

+ 扩容 PD 节点。

  ```bash
  tiup cluster scale-out tidb-demo scale-out-pd.yaml --user ec2-user -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem -y
  ```

  ```output
  ...
  Starting component pd
  Starting instance 10.90.2.102:2379
  Start instance 10.90.2.102:2379 success
  Starting component node_exporter
  Starting instance 10.90.2.102
  Start 10.90.2.102 success
  Starting component blackbox_exporter
  Starting instance 10.90.2.102
  Start 10.90.2.102 success
  + Refresh components conifgs
   - Generate config pd -> 10.90.2.26:2379 ... Done
   - Generate config pd -> 10.90.2.102:2379 ... Done
   - Generate config tikv -> 10.90.2.254:20160 ... Done
   - Generate config tikv -> 10.90.2.182:20160 ... Done
   - Generate config tidb -> 10.90.2.49:4000 ... Done
   - Generate config prometheus -> 10.90.2.38:10090 ... Done
   - Generate config grafana -> 10.90.2.38:3000 ... Done
  + Reload prometheus and grafana
   - Reload prometheus -> 10.90.2.38:10090 ... Done
   - Reload grafana -> 10.90.2.38:3000 ... Done
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Scaled cluster `tidb-demo` out successfully
  ```

+ 检查 DC2 中的两个 PD 节点是否均已启动并正常运行。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

     To update this component:   tiup update cluster
     To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.94.179.113:2379/dashboard
  Grafana URL:        http://10.90.2.82:3000
  ID                 Role        Host         Ports        OS/Arch       Status        Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------        --------                    ----------
  10.90.2.107:3000   grafana     10.90.2.107  3000         /             Up            -                           /tidb-deploy/grafana-3000
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /             Up|L          /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.96:2379    pd          10.90.2.96   2379/2380    linux/x86_64  Up            /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /             Down          /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /             Down          -                           /tidb-deploy/tidb-4000
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /             Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /             Disconnected  /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 7
  ```

### 步骤 10. 将集群设置为 `majority` 模式

+ 验证集群仍处于 `dr-auto-sync` 复制模式。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode
  ```

  ```output
  The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository.
  download https://tiup-mirrors.tidb.io/ctl-v${LAB:TIDB_VERSION}-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 97.75 MiB/s
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.83:2379 config show replication-mode
  {
   "replication-mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label-key": "dc",
     "primary": "dc1",
     "dr": "dc2",
     "primary-replicas": 2,
     "dr-replicas": 1,
     "wait-store-timeout": "1m0s",
     "wait-recover-timeout": "0s",
     "pause-region-split": "false"
   }
  }
  ```

+ 检查 `majority` 规则文件。

  ```bash
  cat majority-rules.json
  ```

  ```output
  [
   {
     "group_id": "pd",
     "group_index": 0,
     "group_override": false,
     "rules": [
       {
         "group_id": "pd",
         "id": "voters",
         "start_key": "",
         "end_key": "",
         "role": "voter",
         "count": 3,
         "location_labels": ["dc","logic","rack", "host"]
       }
     ]
   }
  ]
  ```

+ 加载 `majority` Placement Rules 配置。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/majority-rules.json"
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/majority-rules.json
  "Update rules and groups successfully."
  ```

+ 检查 Placement Rules 是否已加载。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules show -u ${HOST_AZ2PD1_PRIVATE_IP}:2379
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules show -u 10.90.2.125:2379
  [
   {
     "group_id": "pd",
     "id": "voters",
     "start_key": "",
     "end_key": "",
     "role": "voter",
     "is_witness": false,
     "count": 3,
     "location_labels": [
       "dc",
       "logic",
       "rack",
       "host"
     ],
     "create_timestamp": 1734071450
   }
  ]
  ```

+ 启用 `majority` 配置。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode majority
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set max-replicas 3
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 config set replication-mode majority
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 config set max-replicas 3
  Success!
  ```

+ 验证集群 `majority` 复制模式已生效。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 config show replication-mode
  {
   "replication-mode": "majority",
   "dr-auto-sync": {
     "label-key": "dc",
     "primary": "dc1",
     "dr": "dc2",
     "primary-replicas": 2,
     "dr-replicas": 1,
     "wait-store-timeout": "1m0s",
     "wait-recover-timeout": "0s",
     "pause-region-split": "false"
   }
  }
  ```

### 步骤 11. 删除不可用的 TiKV 节点

+ 启动 TiKV 节点。

  ```bash
  tiup cluster start tidb-demo -R tikv -y
  ```

  ```output
  ...
  Starting component tikv
   Starting instance 10.90.2.129:20160
   Starting instance 10.90.2.216:20160
   Start instance 10.90.2.216:20160 success
   Start instance 10.90.2.129:20160 success
  Starting component node_exporter
   Starting instance 10.90.2.129
   Starting instance 10.90.2.216
   Start 10.90.2.216 success
   Start 10.90.2.129 success
  Starting component blackbox_exporter
   Starting instance 10.90.2.129
   Starting instance 10.90.2.216
   Start 10.90.2.216 success
   Start 10.90.2.129 success
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Started cluster `tidb-demo` successfully
  ...
  ```

+ 等待两个 TiKV 节点启动并正常运行。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ...
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.2.82:3000    grafana     10.90.2.82   3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.125:2379   pd          10.90.2.125  2379/2380    linux/x86_64  Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.135:2379   pd          10.90.2.135  2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.82:10090   prometheus  10.90.2.82   10090/12020  linux/x86_64  Down    /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.173:4000   tidb        10.90.2.173  4000/10080   linux/x86_64  Down    -                           /tidb-deploy/tidb-4000
  10.90.2.129:20160  tikv        10.90.2.129  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.216:20160  tikv        10.90.2.216  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 7
  ```

+ 移除不可用的 TiKV 节点。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores --auto-detect
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.110:2379 unsafe remove-failed-stores --auto-detect
  Success! Request has been accepted.
  ```

+ 显示移除操作的状态。一段时间后，你可以通过以下描述确认该过程已完成：命令结果末尾显示 `"info": "Unsafe recovery Finished"`。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores show
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.125:2379 unsafe remove-failed-stores show
  [
   {
     "info": "Unsafe recovery enters collect report stage",
     "time": "2024-12-13 07:01:51.714",
     "details": [
       "auto detect mode with no specified Failed stores"
     ]
   },
   {
     "info": "Unsafe recovery enters force leader stage",
     "time": "2024-12-13 07:01:54.000",
     "actions": {
       "store 1": [
         "force leader on regions: 7, 9, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 101, 104, 107, 110, 113, 116, 119, 122, 125, 128, 131, 134, 137, 140, 143, 146, 149, 152, 155, 158, 209, 213, 217, 221, 225, 229, 233, 237, 241, 245, 249, 253, 1978, 2013, 2018, 1993, 2043, 2048, 1983, 2023, 2028, 1988, 2033, 2038, 1998, 2063, 2068, 2008, 2053, 2058, 35, 31, 27, 23, 19, 17, 11, 13, 3"
       ],
       "store 16": [
         "force leader on regions: "
       ]
     }
   },
   {
     "info": "Unsafe recovery enters demote Failed voter stage",
     "time": "2024-12-13 07:02:04.586",
     "actions": {
       "store 1": [
         "region 7 demotes peers { id:75 store_id:2 }, { id:170 store_id:15 }",
         ...
         "region 3 demotes peers { id:77 store_id:2 }, { id:208 store_id:15 }"
       ]
     }
   },
   {
     "info": "Unsafe recovery Finished",
     "time": "2024-12-13 07:02:14.002",
     "details": [
       "affected table ids: 74, 94, 110, 24, 58, 78, 92, 107, 281474976710649, 56, 82, 84, 106, 108, 109, 281474976710652, 281474976710654, 52, 111, 50, 66, 281474976710653, 48, 98, 54, 38, 42, 46, 62, 72, 76, 6, 281474976710651, 86, 28, 30, 40, 68, 96, 20, 60, 4, 18, 34, 80, 12, 10, 32, 70, 8, 64, 22, 36, 88, 90, 102, 14, 26, 44, 281474976710650, 16",
       "no newly created empty regions"
     ]
   }
  ]
  ```

### 步骤 12. 启动恢复的 TiDB 集群

+ 启动 tidb 和 Prometheus 实例。

  ```bash
  tiup cluster start tidb-demo -R tidb,prometheus -y
  ```

  ```output
  Starting component tidb
  Starting instance 10.90.2.49:4000
  Start instance 10.90.2.49:4000 success
  Starting component prometheus
  Starting instance 10.90.2.38:10090
  Start instance 10.90.2.38:10090 success
  Starting component node_exporter
  Starting instance 10.90.2.38
  Starting instance 10.90.2.49
  Start 10.90.2.49 success
  Start 10.90.2.38 success
  Starting component blackbox_exporter
  Starting instance 10.90.2.38
  Starting instance 10.90.2.49
  Start 10.90.2.49 success
  Start 10.90.2.38 success
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Started cluster `tidb-demo` successfully
  ```

+ 检查 TiDB 集群的状态。确认 DC2 中所有节点的状态均为 `Up`。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  A new version of cluster is available: v1.15.1 -> v1.16.1

     To update this component:   tiup update cluster
     To update all components:   tiup update --all

  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.94.179.113:2379/dashboard
  Grafana URL:        http://10.90.2.82:3000
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.2.82:3000    grafana     10.90.2.82   3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.125:2379   pd          10.90.2.125  2379/2380    linux/x86_64  Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.135:2379   pd          10.90.2.135  2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.82:10090   prometheus  10.90.2.82   10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.173:4000   tidb        10.90.2.173  4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.2.129:20160  tikv        10.90.2.129  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.216:20160  tikv        10.90.2.216  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 7
  ```

+ 验证 TiDB 集群的复制模式。

  ```bash
  curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "majority",
   "dr-auto-sync": {
     "label_key": "",
     "state": "",
     "acid_consistent": false
   }
  }
  ```

### 步骤 13. 验证 sysbench 工作负载是否已恢复

+ 切换到[终端 1](ssh://az1monitor1-ssh#1)，Sysbench 工作负载的 QPS 仍为 0，因为位于 DC1 的 TiDB 服务器不可用。

  ```output
  ...
  [ 1697s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 1698s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 1699s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  ```

+ 切换到[终端 2](ssh://az2monitor1-ssh#2)，Sysbench 工作负载的 QPS 恢复正常。

  ```output
  ...
  [ 1618s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 1619s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 6.00
  [ 1620s ] thds: 6 tps: 321.06 qps: 6505.17 (r/w/o: 4564.82/1103.20/837.15) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
  [ 1621s ] thds: 6 tps: 313.95 qps: 6261.93 (r/w/o: 4378.25/1098.81/784.87) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
  ```

### 步骤 14. 再次启用 DR Auto-Sync

+ 当 DC1 恢复后，需要将 TiDB 集群恢复为 DR Auto-Sync 部署。

+ 切换到[终端 3](ssh://az2monitor1-ssh#3)，将丢失的节点重新整合回 TiDB 集群。检查 YAML 文件配置。

  ```bash
  cat scale-out-DC1.yaml
  ```

  ```output
  tidb_servers:
   - host: 10.90.1.58
     port: 4000
     status_port: 10080
     deploy_dir: "/tidb-deploy/tidb-4000"
     log_dir: "/tidb-deploy/tidb-4000/log"
  tikv_servers:
   - host: 10.90.1.219
     port: 20160
     status_port: 20180
     deploy_dir: "/tidb-deploy/tikv-20160"
     data_dir: "/tidb-data/tikv_data"
     config:
       server.labels: { dc: "dc1",logic: "logic1",rack: "r1",host: "10.90.1.219" }
   - host: 10.90.1.71
     port: 20160
     status_port: 20180
     deploy_dir: "/tidb-deploy/tikv-20160"
     data_dir: "/tidb-data/tikv_data"
     config:
       server.labels: { dc: "dc1",logic: "logic2",rack: "r1",host: "10.90.1.71" }
  pd_servers:
   - host: 10.90.1.95
     client_port: 2379
     peer_port: 2380
     deploy_dir: "/tidb-deploy/pd-2379"
     data_dir: "/tidb-data/pd_data"
     advertise_client_addr: "http://34.214.79.226:2379"
   - host: 10.90.1.55
     client_port: 2379
     peer_port: 2380
     deploy_dir: "/tidb-deploy/pd-2379"
     data_dir: "/tidb-data/pd_data"
     advertise_client_addr: "http://54.202.186.157:2379"
  monitoring_servers:
   - host: 10.90.1.221
     port: 10090
     deploy_dir: "/tidb-deploy/prometheus-10090"
     data_dir: "/tidb-data/prometheus_data"
  grafana_servers:
   - host: 10.90.1.221
     port: 3000
     deploy_dir: "/tidb-deploy/grafana-3000"
  ```

+ 扩容 DC1 中的节点。

  ```bash
  tiup cluster scale-out tidb-demo scale-out-DC1.yaml --user ec2-user -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem -y
  ```

  ```output
  ...
  Starting component pd
  Starting instance 10.90.1.95:2379
  Start instance 10.90.1.95:2379 success
  Starting instance 10.90.1.55:2379
  Start instance 10.90.1.55:2379 success
  Starting component tikv
  Starting instance 10.90.1.71:20160
  Starting instance 10.90.1.219:20160
  Start instance 10.90.1.71:20160 success
  Start instance 10.90.1.219:20160 success
  Starting component tidb
  Starting instance 10.90.1.58:4000
  Start instance 10.90.1.58:4000 success
  Starting component prometheus
  Starting instance 10.90.1.221:10090
  Start instance 10.90.1.221:10090 success
  Starting component grafana
  Starting instance 10.90.1.221:3000
  Start instance 10.90.1.221:3000 success
  Starting component node_exporter
  Starting instance 10.90.1.55
  Starting instance 10.90.1.58
  Starting instance 10.90.1.221
  Starting instance 10.90.1.95
  Starting instance 10.90.1.219
  Starting instance 10.90.1.71
  Start 10.90.1.71 success
  Start 10.90.1.95 success
  Start 10.90.1.219 success
  Start 10.90.1.55 success
  Start 10.90.1.221 success
  Start 10.90.1.58 success
  Starting component blackbox_exporter
  Starting instance 10.90.1.55
  Starting instance 10.90.1.58
  Starting instance 10.90.1.221
  Starting instance 10.90.1.95
  Starting instance 10.90.1.219
  Starting instance 10.90.1.71
  Start 10.90.1.219 success
  Start 10.90.1.95 success
  Start 10.90.1.71 success
  Start 10.90.1.55 success
  Start 10.90.1.221 success
  Start 10.90.1.58 success
  + Refresh components conifgs
   - Generate config pd -> 10.90.2.83:2379 ... Done
   - Generate config pd -> 10.90.2.96:2379 ... Done
   - Generate config pd -> 10.90.1.95:2379 ... Done
   - Generate config pd -> 10.90.1.55:2379 ... Done
   - Generate config tikv -> 10.90.2.111:20160 ... Done
   - Generate config tikv -> 10.90.2.91:20160 ... Done
   - Generate config tikv -> 10.90.1.219:20160 ... Done
   - Generate config tikv -> 10.90.1.71:20160 ... Done
   - Generate config tidb -> 10.90.2.210:4000 ... Done
   - Generate config tidb -> 10.90.1.58:4000 ... Done
   - Generate config prometheus -> 10.90.2.107:10090 ... Done
   - Generate config prometheus -> 10.90.1.221:10090 ... Done
   - Generate config grafana -> 10.90.2.107:3000 ... Done
   - Generate config grafana -> 10.90.1.221:3000 ... Done
  + Reload prometheus and grafana
   - Reload prometheus -> 10.90.2.107:10090 ... Done
   - Reload grafana -> 10.90.2.107:3000 ... Done
  + [ Serial ] - UpdateTopology: cluster=tidb-demo
  Scaled cluster `tidb-demo` out successfully
  ```

+ 检查 TiDB 集群状态，确认所有节点已恢复。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.1.221:3000   grafana     10.90.1.221  3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.107:3000   grafana     10.90.2.107  3000         /             Up      -                           /tidb-deploy/grafana-3000
  10.90.1.55:2379    pd          10.90.1.55   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.95:2379    pd          10.90.1.95   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /             Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.96:2379    pd          10.90.2.96   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.221:10090  prometheus  10.90.1.221  10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /             Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.58:4000    tidb        10.90.1.58   4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /             Up      -                           /tidb-deploy/tidb-4000
  10.90.1.219:20160  tikv        10.90.1.219  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.71:20160   tikv        10.90.1.71   20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 14
  ```

+ 加载 DR Auto-Sync Placement Rules 配置。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/dr-anto-sync-rules.json"
  ```

  ```output
  The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository.
  download https://tiup-mirrors.tidb.io/ctl-v${LAB:TIDB_VERSION}-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 103.24 MiB/s
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/dr-anto-sync-rules.json
  "Update rules and groups successfully."
  ```

+ 安装 `jq` 工具。

  ```bash
  sudo yum install jq -y
  ```

  ```output
  ...
  Installed:
   jq.x86_64 0:1.5-1.amzn2.0. 2

  Dependency Installed:
   oniguruma.x86_64 0:5.9.6-1.amzn2.0. 7

  Complete!
  ```

+ 检查是否存在没有 Learner Peers 的 Region。验证是否没有 Region 缺少 Learner Peers，并确保投票角色和 Learner 角色之间的数据同步。此步骤保证集群的数据在所有角色中一致复制，以维持高可用性和故障转移准备。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 region --jq='.regions[] | select(.peers | any(.role_name=="Learner") | not) |  {id: .id, peers: [.peers]}'
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://34.215.73.186:2379 region --jq=.regions[] | select(.peers | any(.role_name=="Learner") | not) |  {id: .id, peers: [.peers]}
  ```

+ 启用 DR Auto-Sync 配置。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync label-key dc
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary dc1
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr dc2
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary-replicas 2
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr-replicas 1
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync label-key dc
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary dc1
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr dc2
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary-replicas 2
  Success! The replication mode config is updated."

  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr-replicas 1
  Success! The replication mode config is updated."
  ```

### 步骤 15. 验证复制状态是否恢复为 `sync`

+ 验证配置是否生效。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379:2379 config show replication-mode
  {
   "replication-mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label-key": "dc",
     "primary": "dc1",
     "dr": "dc2",
     "primary-replicas": 2,
     "dr-replicas": 1,
     "wait-store-timeout": "1m0s",
     "wait-recover-timeout": "0s",
     "pause-region-split": "false"
   }
  }
  ```

+ 检查集群的当前复制状态。在状态变为 `sync` 模式之前，如果操作较快，可以看到 `sync_recover` 模式和 `recover_progress` 状态。

  ```bash
  curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status
  ```

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync_recover",
     "state_id": 2003,
     "acid_consistent": false,
     "total_regions": 60,
     "synced_regions": 37,
     "recover_progress": 0.98333335
   }
  }
  ```

+ 最终，TiDB 将恢复到 `sync` 状态。

  ```output
  {
   "mode": "dr-auto-sync",
   "dr-auto-sync": {
     "label_key": "dc",
     "state": "sync",
     "state_id": 378,
     "acid_consistent": true
   }
  }
  ```

+ 检查 sysbench 工作负载是否已恢复。切换到[终端 1](ssh://az1monitor1-ssh#1)，sysbench 工作负载的 QPS 恢复正常。

  ```output
  ...
  [ 2309s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 2310s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
  [ 2311s ] thds: 6 tps: 105.01 qps: 2170.15 (r/w/o: 1530.11/416.03/224.02) lat (ms,95%): 100000.00 err/s: 0.00 reconn/s: 6.00
  [ 2312s ] thds: 6 tps: 193.97 qps: 3869.44 (r/w/o: 2705.61/763.89/399.94) lat (ms,95%): 34.33 err/s: 0.00 reconn/s: 0.00
  ```

### 步骤 16. 将 PD 和 TiKV 的领导者转移到 DC1

+ 切换到[终端 3](ssh://az2monitor1-ssh#3)，缩容 DC2 中的第二个 PD 节点。

  ```bash
  tiup cluster scale-in tidb-demo -N "${HOST_AZ2PD2_PRIVATE_IP}:2379" -y
  ```

  ```output
  ...
  Scaled cluster `tidb-demo` in successfully
  ```

+ 无论上一步结果如何，您可以执行此命令尝试将 pd leader 转移至 DC1。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 member leader transfer "pd-${HOST_AZ1PD1_PRIVATE_IP}-2379"
  ```

  ```output
  The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository.
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379
  Success!
  ```

+ 检查 TiDB 集群状态，确保 PD 的 Leader 位于 DC1 且 PD 节点数变为 3。

  ```bash
  tiup cluster display tidb-demo
  ```

  ```output
  ID                 Role        Host         Ports        OS/Arch       Status  Data Dir                    Deploy Dir
  --                 ----        ----         -----        -------       ------  --------                    ----------
  10.90.1.221:3000   grafana     10.90.1.221  3000         linux/x86_64  Up      -                           /tidb-deploy/grafana-3000
  10.90.2.107:3000   grafana     10.90.2.107  3000         /             Up      -                           /tidb-deploy/grafana-3000
  10.90.1.55:2379    pd          10.90.1.55   2379/2380    linux/x86_64  Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.95:2379    pd          10.90.1.95   2379/2380    linux/x86_64  Up|L    /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.2.83:2379    pd          10.90.2.83   2379/2380    /             Up      /tidb-data/pd_data          /tidb-deploy/pd-2379
  10.90.1.221:10090  prometheus  10.90.1.221  10090/12020  linux/x86_64  Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.2.107:10090  prometheus  10.90.2.107  10090        /             Up      /tidb-data/prometheus_data  /tidb-deploy/prometheus-10090
  10.90.1.58:4000    tidb        10.90.1.58   4000/10080   linux/x86_64  Up      -                           /tidb-deploy/tidb-4000
  10.90.2.210:4000   tidb        10.90.2.210  4000/10080   /             Up      -                           /tidb-deploy/tidb-4000
  10.90.1.219:20160  tikv        10.90.1.219  20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.1.71:20160   tikv        10.90.1.71   20160/20180  linux/x86_64  Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.111:20160  tikv        10.90.2.111  20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  10.90.2.91:20160   tikv        10.90.2.91   20160/20180  /             Up      /tidb-data/tikv_data        /tidb-deploy/tikv-20160
  Total nodes: 13
  ```

+ 通过检查每个角色的 Region 分布，我们可以确认 DR Auto-Sync 配置已成功应用于 TiKV 节点。

  ```bash
  tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 store
  ```

  ```output
  Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.2.83:2379 store
  {
   "count": 4,
   "stores": [
     {
       "store": {
         "id": 1,
         "address": "10.90.2.111:20160",
         ...
       },
       "status": {
         "leader_count": 0,
         "region_count": 79,
         ...
       }
     },
     {
       "store": {
         "id": 14,
         "address": "10.90.2.91:20160",
         ...
       },
       "status": {
         "leader_count": 0,
         "region_count": 79,
         "learner_count": 79,
         ...
       }
     },
     {
       "store": {
         "id": 2001,
         "address": "10.90.1.219:20160",
         ...
       },
       "status": {
         "leader_count": 42,
         "region_count": 79,
         ...
       }
     },
     {
       "store": {
         "id": 2002,
         "address": "10.90.1.71:20160",
         ...
       },
       "status": {
         "leader_count": 37,
         "region_count": 79,
         ...
       }
     }
   ]
  }
  ```

+ 从以上信息中，我们可以通过 Region 分布确定每个 TiKV 节点的角色（您的 store id 可能与以下示例不同）：
  + Store 1 是 `follower`：`"leader_count": 0, "region_count": 79`
  + Store 14 是 `learner`：`"leader_count": 0, "region_count": 79, "learner_count": 79`
  + Store 2001 是 `voter`：`"leader_count": 42, "region_count": 79`
  + Store 2002 是 `voter`：`"leader_count": 37, "region_count": 79`

+ 切换到 [终端 1](ssh://az1monitor1-ssh#1)，终止 sysbench 线程并关闭 终端 1。

+ 切换到 [终端 2](ssh://az2monitor1-ssh#2)，终止 sysbench 线程并关闭 终端 2。

+ 关闭 终端 3。

### 模块 3 小结

+ 通过本次实践，我们观察到即使主数据中心发生完全故障，我们仍然可以成功恢复 TiDB 集群并确保 RPO=0。大致来说，恢复操作包括以下步骤：
  + 恢复位于 DC2 的唯一 PD，并使用其创建新集群。
  + 在 DC2 中扩容一个新的 PD，组成一个新的 Raft 组。
  + 修改集群模式为多数模式（majority mode）。
  + 移除位于 DC1 的不可用 TiKV 节点。
  + 重启所有集群节点并运行剩余的两个 TiKV 节点，观察应用读写操作的恢复情况。
  + 修复故障。
  + 初始化 DC1 中的节点并将它们重新加入 TiDB 集群。
  + 修改集群配置，恢复至 DR Auto-Sync 模式。