# 双数据中心部署方案 - TiDB 容灾自动同步 ## 概览 在金融行业中，双数据中心架构是常见的部署模式，尤其在核心业务系统中，银行和金融机构对数据高可用性和灾备能力有极高要求。TiDB 的 DR Auto-Sync 方案正是为这种环境设计的，它能够在双数据中心基础设施下提供强大的数据保护能力，确保 RPO（恢复点目标）为 0 和分钟级的 RTO（恢复时间目标）。具体而言，该方案具备以下特点： + 支持 RPO = 0：确保任何数据中心故障情况下不会丢失数据。 + 支持分钟级 RTO：故障发生后能够在几分钟内恢复服务。 + 支持双活服务：允许两个数据中心同时提供高可用数据服务。 本实验模拟了在双数据中心场景下部署 DR Auto-Sync 的过程，展示如何在任一数据中心不可用或网络断开时，实现数据一致性和高可用性。 ### 实验时长 + 完成此练习大约需要 100 分钟。 ### DR Auto-Sync 方案介绍 默认部署的挑战 + 默认情况下，所有数据库副本（3 个副本）形成一个 Raft 组，其中任意两个副本即可确保数据库的可用性和数据持久性。对于每次数据写入，只要在 Raft 组内的任意两个 TiKV 中提交成功，客户端就会收到成功响应。然而，如果 DC1 持有最新的两个数据副本并发生完全宕机，这两个副本就会丢失，而 Raft 组无法保证最新数据已复制到 DC2。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-1-en.png) DR Auto-Sync 方案 + 该方案在 Raft 算法中引入了 commit group（提交组）的概念。写入数据提交时，不仅要满足多数节点持久化的原则，还必须在至少两个 commit group 中提交完成。通过设置两个数据中心分别对应一个 commit group，保证了两个数据中心都持有最新的数据副本，确保这任一数据中心故障情况下的数据安全。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-2-en.png) 此外，该方案在 DC2 中包含一个额外的 `learner` 节点，它不参与投票，仅在 DC2 是唯一存活的数据中心时用于恢复 TiDB 集群。 DR Auto-Sync 的高可用场景 + DR Auto-Sync 方案使用三种不同的状态来标识 TiKV 数据同步模式。集群的复制模式可以在这三种状态之间自适应切换。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-3-en.png) + 同步模式：TiDB 默认在同步复制模式下运行。在此模式下，灾备数据中心中至少有一个副本与主数据中心同步。Raft 算法确保写入的最新数据都在到灾备数据中心得到持久化。在这种状态下，任一数据中心的整体故障都能确保 TiDB 集群达到 RPO=0。 + 异步模式：当某个数据中心发生整体故障时，TiDB 切换到异步复制模式。在此模式下，这两个数据中心的数据未完全同步。Raft 算法退化为仅遵循多数派协议，在存活的数据中心内完成复制数据。由于已经发生故障，如果发生第二次故障，TiDB 集群无法保证 RPO=0。只有在初始故障解决后，集群才能重新获得 RPO=0 的高可用能力。 + 同步恢复模式：当故障的数据中心恢复后，两个数据中心之间开启数据同步。此时，TiDB 切换到同步恢复模式，此模式意味着这两个数据中心同步在进行中。数据同步完成后，TiDB 重新切换回同步模式。然后，TiDB 集群重新获得 RPO=0 高可用能力。 ### 实验任务 + 模块 1：搭建 TiDB 集群并配置 DR Auto-Sync 部署，验证高可用模式的正确设置。 + 模块 2：模拟灾备数据中心发生故障或网络隔离，观察集群的自动切换和恢复机制。 + 模块 3：模拟主数据中心发生故障，验证集群的故障切换行为及快速恢复能力。 ### 实验目标 本实验通过模拟双数据中心场景，展示 TiDB DR Auto-Sync 方案如何实现跨数据中心的高可用性和强一致性。这种方案不仅解决了传统双数据中心架构下可能出现的故障和数据丢失问题，还通过动态切换同步模式，使 TiDB 在双数据中心环境下具备极高的灵活性和灾备能力。 DR Auto-Sync 特别适合金融行业需要高可靠性和零数据丢失的核心业务场景，例如支付系统、清算系统和账户管理系统。通过这次实验，您可以更直观地了解 TiDB 在双数据中心架构下的独特优势及实际应用能力。 + **macOS** 用户： + 点击下载私钥文件 [pe-class-key-oregon.pem](/api/labs/demo_001/vm_key?region_name=oregon)。此文件在本动手实验中用于登录远程 Linux。 + 将私钥文件保存在 `~/.ssh/` 文件夹中。 + 使用以下命令设置权限并检查文件是否到位。 ```bash chmod 400 ~/.ssh/pe-class-key-oregon.pem ls -lF ~/.ssh/pe-class-key-oregon.pem ``` ```output ... -r--------@ 1 username staff 1674 Jun 21 2023 /Users/username/.ssh/pe-class-key-oregon.pem ``` + **Windows 10** 用户： + 点击下载私钥文件 [pe-class-key-oregon.pem](/api/labs/demo_001/vm_key?region_name=oregon)。此文件在本动手实验中用于登录远程 Linux。 + 将私钥文件保存在 `%userprofile%\.ssh` 文件夹中。 + 使用以下命令检查文件是否到位。请使用 CMD，而非 Powershell。 ```cmd dir %userprofile%\.ssh\pe-class-key-oregon.pem ``` ```output ... 2024/11/14 10:22 1,674 pe-class-key-oregon.pem 1 Files(s) 1,674 bytes ... ``` + **Windows 8** 或更早版本用户: + 点击下载私钥文件 [pe-class-key-oregon.ppk](/api/labs/demo_001/vm_key?region_name=oregon&key_type=ppk)。此文件在本动手实验中用于登录远程 Linux。 + 下载并安装 [PuTTY](https://www.putty.org/)。 > [!TIP] > 实验室环境部署完成后，点击底部栏上的 **前进** 进入 **模块 1**。 ## 模块 1: 设置具有 DR Auto-Sync 配置的 TiDB 集群 在本实践中，我们将设置一个启用 DR Auto-Sync 配置的集群。以下节点分布在两个数据中心： + DC1 作为主数据中心，位于 aws AZ1：pd*2，tidb*1，tikv*2，tiup(monitor)*1 + DC2 作为灾难恢复数据中心，位于 aws AZ2：pd*1(额外1个备用节点)，tidb*1，tikv*2，tiup(monitor)*1 + Sysbench 作为应用程序部署在每个数据中心（对应下图中的 sysbench）。注意：测试简化了负载均衡服务，DC1 中 sysbench 直接连接 DC1 中的 tidb server，DC2 同理。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-4-en.png) ### 步骤 1: 连接到第一个数据中心的监视节点 + 如果您在实验中使用 Web SSH，请单击 Web SSH 按钮或图标打开终端会话，然后跳过本步骤直接从 **步骤 2** 开始。 + 打开本地终端(终端 1)，连接到 monitor 1 服务器以部署 TiDB 集群。连接到远程堡垒机 Linux。如果没有连接凭证，请查看假设部分。如果提示，请回答 `yes`。 + **macOS** 用户: ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 10** 用户: ```cmd ssh -i %userprofile%\.ssh\pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 8** 或更早版本用户: + 运行 PuTTY，在 `Session` -> `Host Name`, 输入 `ec2-user@monitor1_address`。 + 在 `Connection` -> `SSH` -> `Auth` -> `Credentials` -> `Private key file for authentication`, 浏览并输入私钥位置。 + 现在，您应当可以通过 PuTTY 登陆远程 Linux 了。 + 登录后显示的信息如下： ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### 步骤 2. 查看 TiDB 集群拓扑配置。它跨 2 个可用区 + 查看已配置的集群拓扑 YAML。 ```bash cat dr-auto-sync.yaml ``` ```output global: user: "ec2-user" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" arch: "amd64" monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 deploy_dir: "/tidb-deploy/monitored-9100" data_dir: "/tidb-data/monitor_9115" log_dir: "/tidb-deploy/monitor-9100/log" server_configs: pd: replication.location-labels: ["dc","logic","rack","host"] tidb_servers: - host: 10.90.1.163 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" - host: 10.90.2.111 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" tikv_servers: - host: 10.90.1.80 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic1",rack: "r1",host: "10.90.1.80" } - host: 10.90.1.45 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic2",rack: "r1",host: "10.90.1.45" } - host: 10.90.2.33 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic3",rack: "r1",host: "10.90.2.33" } - host: 10.90.2.113 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic4",rack: "r1",host: "10.90.2.113" } pd_servers: - host: 10.90.1.19 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://35.88.184.77:2379" - host: 10.90.1.44 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://34.219.233.52:2379" - host: 10.90.2.108 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://18.236.155.129:2379" monitoring_servers: - host: 10.90.1.147 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" - host: 10.90.2.82 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" grafana_servers: - host: 10.90.1.147 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" - host: 10.90.2.82 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" ``` ### 步骤 3. 设置 TiDB 镜像并执行部署前预检查 ```bash tiup mirror set tidb-community-server-tidb_version-linux-amd64 tiup cluster check dr-auto-sync.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-oregon.pem \ --apply ``` ```output ... + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks - Applying changes on 10.90.2.33 ... Done - Applying changes on 10.90.2.113 ... Done - Applying changes on 10.90.1.163 ... Done - Applying changes on 10.90.2.111 ... Done - Applying changes on 10.90.1.147 ... Done - Applying changes on 10.90.1.44 ... Done - Applying changes on 10.90.2.108 ... Done - Applying changes on 10.90.1.45 ... Done - Applying changes on 10.90.2.82 ... Done - Applying changes on 10.90.1.19 ... Done - Applying changes on 10.90.1.80 ... Done ``` ### 步骤 4. 部署 TiDB 集群 ```bash tiup cluster deploy tidb-demo vtidb_version ./dr-auto-sync.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-oregon.pem \ --yes ``` ```output ... Cluster `tidb-demo` deployed successfully, you can start it with command: `tiup cluster start tidb-demo --init` ``` ### 步骤 5. 启动 TiDB 集群 ```bash tiup cluster start tidb-demo ``` ```output ... Started cluster `tidb-demo` successfully ``` ### 步骤 6. 获取 TiDB 集群状态 + 关键集群组件分布在 2 个子网中( DC1 的 10.90.1.0/24，DC2 的 10.90.2.0/24)，分别位于 2 个可用区。每个 DC 都有 2 个 PD 服务器(DC2 包含一个备用节点)、2 个 TiKV 服务器和 1 个 TiDB 服务器。因此，我们可以使用这种部署来模拟双数据中心环境。 ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.188.60.25:2379/dashboard Grafana URL: http://10.90.1.114:3000,http://10.90.2.122:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.114:3000 grafana 10.90.1.114 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.122:3000 grafana 10.90.2.122 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.114:10090 prometheus 10.90.1.114 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.122:10090 prometheus 10.90.2.122 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.81:4000 tidb 10.90.1.81 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.207:4000 tidb 10.90.2.207 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.163:20160 tikv 10.90.1.163 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.83:20160 tikv 10.90.1.83 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.17:20160 tikv 10.90.2.17 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.19:20160 tikv 10.90.2.19 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ### 步骤 7. 确保 Placement Driver Leader 位于 DC1 + 如果 PD Leader 不在 DC1 中（IP 属于 10.90.1.0/24 子网），将 Leader 转移到 DC1。 ```output ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 ``` + 您可以执行此命令，无论上一步的结果如何，都尝试将 PD Leader 转移到 DC1。 ```bash tiup ctl:vtidb_version pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 member leader transfer "pd-${HOST_AZ1PD1_PRIVATE_IP}-2379" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` + 检查 PD Leader 的位置，确认其已切换到 DC1。 ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- ... 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 ... ``` ### 步骤 8. 在 DC2 中创建对等的 TiUP 中控机 + 将 tiup 的集群 meta 文件复制到位于灾难恢复数据中心 (DC2) 的 monitor 2 服务器的 tiup 目录。如果提示，请回答 `yes`。 ```bash scp -i ~/.ssh/pe-class-key-oregon.pem -r .tiup/storage/ ec2-user@az2monitor1_address:/home/ec2-user/.tiup/ ``` ```output ... run_blackbox_exporter_10.90.1.163.sh 100% 519 356.8KB/s 00:00 run_blackbox_exporter_10.90.2.82.sh 100% 519 414.9KB/s 00:00 meta.yaml 100% 4481 3.2MB/s 00:00 ``` + 打开新终端(终端 2 )。登录到 monitor 2 服务器以验证 tiup 元文件是否正常工作。 ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az2monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + 确认 monitor 2 中的 tiup 可以正常工作。 ```bash tiup cluster display tidb-demo ``` ```output Checking updates for component cluster... A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.88.184.77:2379/dashboard Grafana URL: http://10.90.1.147:3000,http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.147:3000 grafana 10.90.1.147 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.44:2379 pd 10.90.1.44 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.108:2379 pd 10.90.2.108 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.147:10090 prometheus 10.90.1.147 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.163:4000 tidb 10.90.1.163 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.111:4000 tidb 10.90.2.111 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.45:20160 tikv 10.90.1.45 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.80:20160 tikv 10.90.1.80 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.113:20160 tikv 10.90.2.113 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.33:20160 tikv 10.90.2.33 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ### 步骤 9. 为您的 TiDB 集群启用 DR Auto-Sync + 回到终端 1。 ```bash exit ``` + 检查当前集群的数据复制模式，应为默认的 `majority` 模式。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "majority", "dr-auto-sync": { "label_key": "", "state": "", "acid_consistent": false } } ``` + 查看为 DR Auto-Sync 配置准备的 Placement Rules 文件。集群包含两个数据中心，voter 角色被限制在 DC1。在 DC2 中，有 follower 和 learner 角色。 ```bash cat dr-anto-sync-rules.json ``` ```output [ { "group_id": "pd", "group_index": 0, "group_override": false, "rules": [ { "group_id": "pd", "id": "dc1", "start_key": "", "end_key": "", "role": "voter", "count": 2, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "dc", "op": "in", "values": ["dc1"]}] }, { "group_id": "pd", "id": "logic3", "start_key": "", "end_key": "", "role": "follower", "count": 1, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "logic", "op": "in", "values": ["logic3"]}] }, { "group_id": "pd", "id": "logic4", "start_key": "", "end_key": "", "role": "learner", "count": 1, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "logic", "op": "in", "values": ["logic4"]}] } ] } ] ``` ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-41-en.png) + 加载 Placement Rules 配置，期间 `ctl` 工具会被自动下载并安装。 ```bash tiup ctl:vtidb_version pd config placement-rules rule-bundle \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 save \ --in="/home/ec2-user/dr-anto-sync-rules.json" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. download https://tiup-mirrors.tidb.io/ctl-vtidb_version-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 103.24 MiB/s Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/dr-anto-sync-rules.json "Update rules and groups successfully." ``` + 检查 Placement Rules 配置是否已加载。 ```bash tiup ctl:vtidb_version pd config placement-rules show -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules show -u http://10.90.2.162:2379 [ { "group_id": "pd", "id": "dc1", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 2, "label_constraints": [ { "key": "dc", "op": "in", "values": [ "dc1" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 }, { "group_id": "pd", "id": "logic3", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 1, "label_constraints": [ { "key": "logic", "op": "in", "values": [ "logic3" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 }, { "group_id": "pd", "id": "logic4", "start_key": "", "end_key": "", "role": "learner", "is_witness": false, "count": 1, "label_constraints": [ { "key": "logic", "op": "in", "values": [ "logic4" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 } ] ``` + 安装 `jq` 工具。 ```bash sudo yum install jq -y ``` ```output ... Installed: jq.x86_64 0:1.5-1.amzn2.0. 2 Dependency Installed: oniguruma.x86_64 0:5.9.6-1.amzn2.0. 7 Complete! ``` + 检查没有 learner peer 的 regions。验证所有 regions 都包含 learner peer，并确保 voter 和 learner 角色之间的数据同步。此步骤确保集群的数据在所有角色中一致复制，这对维护高可用性和故障切换准备至关重要。 ```bash tiup ctl:vtidb_version pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 region \ --jq='.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]}' ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://34.215.73.186:2379 region --jq=.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]} ``` + 启用 DR Auto-Sync 配置。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync label-key dc tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary dc1 tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr dc2 tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary-replicas 2 tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr-replicas 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync label-key dc Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary dc1 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr dc2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary-replicas 2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr-replicas 1 Success! The replication mode config is updated." ``` + 通过这些参数的配置，集群实现了数据中心位置感知和数据分布的控制。 + set replication-mode dr-auto-sync：将 TiDB 的复制模式设置为 dr-auto-sync。 + set replication-mode dr-auto-sync label-key dc：使 PD 能够根据节点的 dc 标签识别它们所属的数据中心，从而实现数据分布的控制。 + set replication-mode dr-auto-sync primary dc1：将 DC1 设置为主数据中心，使其成为 TiDB 集群运行时的主要写入节点位置，所有数据更新优先写入主数据中心。 + set replication-mode dr-auto-sync dr dc2：将 DC2 设置为灾备数据中心，其中 TiKV 节点处于 learner 角色。 + set replication-mode dr-auto-sync primary-replicas 2：设置主数据中心（DC1）的副本数量为 2。 + set replication-mode dr-auto-sync dr-replicas 1：设置灾备数据中心（DC2）的副本数量为 1（不包含 learner 角色）。 + 验证配置是否生效。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379:2379 config show replication-mode { "replication-mode": "dr-auto-sync", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` + 检查集群当前的复制状态。如果操作速度较快，在状态变为 `sync` 模式之前，您可能会看到 `sync_recover` 模式和 `recover_progress` 进度。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 2003, "acid_consistent": false, "total_regions": 60, "synced_regions": 37, "recover_progress": 0.98333335 } } ``` + 最终，您将看到 TiDB 处于 `sync` 状态。 ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 378, "acid_consistent": true } } ``` + 作为初始化状态，集群处于同步复制模式。PD 将状态信息发送到 TiKV，两个数据中心（两个 commit group）严格按照同步复制模式工作. ### 步骤 10. 创建 sysbench 数据库 + 登录到 TiDB。 ```bash mysql -h ${HOST_AZ1DB1_PRIVATE_IP} -P 4000 -u root --connect-expired-password ``` ```output mysql> ``` + 为 sysbench 创建数据库。 ```bash CREATE DATABASE sbtest; ``` ```output Query OK, 0 rows affected (0.13 sec) ``` + 退出到 monitor 的操作系统命令行。 ```bash exit ``` ```output Bye ``` + 设置 DC1 的 sysbench 配置。 ```bash cat << EOF > sysbench_config mysql-host=${HOST_AZ1DB1_PRIVATE_IP} mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=6 report-interval=1 EOF cat sysbench_config ``` ```output mysql-host=10.90.1.125 mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=16 report-interval=1 ``` + 准备 sysbench 数据。 ```bash sysbench --config-file=sysbench_config oltp_point_select --tables=6 --table-size=100 prepare ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest3'... Creating table 'sbtest1'... Creating table 'sbtest5'... Creating table 'sbtest6'... Creating table 'sbtest4'... Creating table 'sbtest2'... Inserting 100 records into 'sbtest3' Creating a secondary index on 'sbtest3'... Inserting 100 records into 'sbtest1' Creating a secondary index on 'sbtest1'... Inserting 100 records into 'sbtest4' Creating a secondary index on 'sbtest4'... Inserting 100 records into 'sbtest6' Creating a secondary index on 'sbtest6'... Inserting 100 records into 'sbtest2' Creating a secondary index on 'sbtest2'... Inserting 100 records into 'sbtest5' Creating a secondary index on 'sbtest5'... ``` ### 模块 1 小结 + 至此，我们已经完成： + TiDB 集群的创建。 + 启用 TiDB 集群 DR Auto-Sync 部署方案并验证其状态。 + 准备了用于模拟应用行为的 sysbench 配置。 ## 模块 2: 通过隔离 DC2 网络访问模拟灾备数据中心故障 该实践的目标是模拟 DC2 的完全不可用，观察 TiDB 集群的状态并恢复它。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-5-en.png) 模拟 DC2 故障的方法是切断 DC1 和 DC2 之间的网络通信。这将阻止 DC2 的服务器接收或响应外部请求，从而让 DC1 认为 DC2 发生了完全故障。 ### 步骤 1. 连接到 DC1 中的 TiUP 中控机 + 打开本地终端（终端1），连接到 monitor 1。 + **macOS** 用户: ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 10** 用户: ```cmd ssh -i %userprofile%\.ssh\pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### 步骤 2. 启动模拟应用负载 + 启动 sysbench 模拟应用负载。 ```bash sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 1 Report intermediate results every 1 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 ... ``` ### 步骤 3. 验证数据复制状态 + 打开一个新的终端（终端 2），登录到 monitor 1 服务器，验证数据复制状态是否处于 `sync` 模式。 ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + 验证数据复制状态。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ### 步骤 4. 阻塞整个 DC2（AZ2）的网络访问 + 执行脚本通过阻塞子网 demo-subnet-2（位于 AZ2 的子网 10.90.2.0/24）来模拟完整的可用区故障。 ```bash VPC_ID=`aws ec2 describe-vpcs \ --filters "Name=tag:Name,Values=trainer_name" \ --query "Vpcs[0].VpcId" \ --output text \ --region us-west-2` SUBNET_ID=`aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \ --query "Subnets[0].SubnetId" \ --output text \ --region us-west-2` CAGE_NACL_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" "Name=tag:Name,Values=cage" \ --query "NetworkAcls[0].NetworkAclId" \ --output text \ --region us-west-2` ASSOC_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \ --query "NetworkAcls[0].Associations" \ --output text \ --region us-west-2 | grep ${SUBNET_ID} | awk -F" " '{print $1}'` aws ec2 replace-network-acl-association \ --association-id ${ASSOC_ID} \ --network-acl-id ${CAGE_NACL_ID} \ --region us-west-2 ``` ```output { "NewAssociationId": "aclassoc-0402a6103febf799f" } ``` ### 步骤 5. 观察应用程序和 TiDB 的状态 + Sysbench 被阻塞访问 TiDB 集群约 70 秒，稍长于阻塞窗口设置（默认设置为 60 秒）。原因是 TiDB 集群需要时间收集网络状态，处理 TiDB 集群状态切换，并将状态同步至每个 TiKV。 ```output ... [ 493s ] thds: 6 tps: 278.00 qps: 5525.08 (r/w/o: 3866.05/1082.02/577.01) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 [ 494s ] thds: 6 tps: 274.99 qps: 5505.84 (r/w/o: 3853.89/1079.97/571.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 [ 495s ] thds: 6 tps: 271.99 qps: 5447.84 (r/w/o: 3814.89/1065.97/566.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00 [ 496s ] thds: 6 tps: 70.00 qps: 1406.04 (r/w/o: 985.03/278.01/143.00) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00 [ 497s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 498s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... [ 544s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 545s ] thds: 6 tps: 0.00 qps: 29.00 (r/w/o: 26.00/0.00/3.00) lat (ms,95%): 0.00 err/s: 3.00 reconn/s: 0.00 [ 546s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... [ 566s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 567s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 568s ] thds: 6 tps: 98.00 qps: 1968.01 (r/w/o: 1376.00/388.00/204.00) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00 ... [ 569s ] thds: 6 tps: 244.00 qps: 4883.91 (r/w/o: 3418.93/950.98/513.99) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00 [ 570s ] thds: 6 tps: 237.00 qps: 4734.95 (r/w/o: 3313.96/924.99/495.99) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 [ 571s ] thds: 6 tps: 264.02 qps: 5279.31 (r/w/o: 3693.21/1039.06/547.03) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00 ... ``` + 观察 TiDB 集群中节点的状态。位于 DC2 的节点全部处于 `Down` 状态。 ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://18.236.254.103:2379/dashboard Grafana URL: http://10.90.1.90:3000,http://10.90.2.17:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.90:3000 grafana 10.90.1.90 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.17:3000 grafana 10.90.2.17 3000 linux/x86_64 Down - /tidb-deploy/grafana-3000 10.90.1.176:2379 pd 10.90.1.176 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.73:2379 pd 10.90.1.73 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.198:2379 pd 10.90.2.198 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.90:10090 prometheus 10.90.1.90 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.17:10090 prometheus 10.90.2.17 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.106:4000 tidb 10.90.1.106 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.61:4000 tidb 10.90.2.61 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.1.245:20160 tikv 10.90.1.245 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.45:20160 tikv 10.90.1.45 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.125:20160 tikv 10.90.2.125 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.96:20160 tikv 10.90.2.96 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` + 观察 TiDB 同步模式的变化，从 `sync` 切换到 `sync_wait`，然后是 `async`。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "async_wait", "state_id": 1001, "acid_consistent": true } } ``` ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "async", "state_id": 483, "acid_consistent": true } } ``` ### 步骤 6. 恢复 DC2 的网络访问 + 恢复 DC2 的网络访问。之前丢失的节点将重新加入 TiDB 集群。 ```bash VPC_ID=`aws ec2 describe-vpcs \ --filters "Name=tag:Name,Values=trainer_name" \ --query "Vpcs[0].VpcId" \ --output text \ --region us-west-2` SUBNET_ID=`aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \ --query "Subnets[0].SubnetId" \ --output text \ --region us-west-2` DEFAULT_NACL_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \ --query "NetworkAcls[0].NetworkAclId" \ --output text \ --region us-west-2` ASSOC_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" \ --query "NetworkAcls[0].Associations" \ --output text \ --region us-west-2 | grep ${SUBNET_ID} | awk -F" " '{print $1}'` aws ec2 replace-network-acl-association \ --association-id ${ASSOC_ID} \ --network-acl-id ${DEFAULT_NACL_ID} \ --region us-west-2 ``` ```output { "NewAssociationId": "aclassoc-0df1e05cfc70356a8" } ``` + 观察 Sysbench 和 TiDB 集群的状态。 ```output [ 1039s ] thds: 6 tps: 268.99 qps: 5409.79 (r/w/o: 3787.85/1067.96/553.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 [ 1040s ] thds: 6 tps: 273.01 qps: 5455.15 (r/w/o: 3814.11/1083.03/558.02) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00 [ 1041s ] thds: 6 tps: 265.00 qps: 5282.95 (r/w/o: 3702.96/1043.99/535.99) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00 ... [ 1105s ] thds: 6 tps: 260.00 qps: 5244.02 (r/w/o: 3669.02/1043.00/532.00) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00 [ 1106s ] thds: 6 tps: 267.99 qps: 5339.86 (r/w/o: 3740.90/1052.97/545.99) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 ``` + 查看 TiDB 集群状态。DC2 中的节点都已经恢复。如果您的操作足够快，可能会看到一些 TiKV 节点的状态从 `Disconnected` 逐渐切换到 `Up`。 ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.92.147.161:2379/dashboard Grafana URL: http://10.90.1.212:3000,http://10.90.2.99:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.212:3000 grafana 10.90.1.212 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.99:3000 grafana 10.90.2.99 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.156:2379 pd 10.90.1.156 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.232:2379 pd 10.90.1.232 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.104:2379 pd 10.90.2.104 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.212:10090 prometheus 10.90.1.212 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.99:10090 prometheus 10.90.2.99 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.23:4000 tidb 10.90.1.23 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.122:4000 tidb 10.90.2.122 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.178:20160 tikv 10.90.1.178 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.6:20160 tikv 10.90.1.6 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.250:20160 tikv 10.90.2.250 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.90:20160 tikv 10.90.2.90 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.250:20160 tikv 10.90.2.250 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 ... ``` + 检查 TiDB 集群当前的复制状态。在几秒钟内，状态从 `sync_recover` 切换到 `sync`。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-6-en.png) + `sync_recover` 模式。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 1009, "acid_consistent": false, "total_regions": 78, "recover_progress": 0.08974359 } } ``` + `sync` 模式。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1010, "acid_consistent": true } } ``` ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-61-en.png) + 此时，TiDB 将两个数据中心之间的数据同步恢复到 `sync` 状态，实现了 RPO=0 的数据中心故障恢复能力。 ### 步骤 7. 停止 Sysbench 工作负载 + 切换到终端 1 和 终端 2，终止 Sysbench 线程并关闭终端。 ### 模块 2 小结 + 通过上述步骤，我们模拟了 TiDB 处理整个 DC2 故障并恢复到可用状态的整个过程。 + 当 DC2 不可用时，TiDB 进入阻塞窗口以确认故障。在此阻塞窗口期间，应用程序对数据库的读写操作会受到影响。 + 超过阻塞窗口后，TiDB 仅使用 DC1 中的节点并转换为 `async` 状态。此时，应用程序的读写操作恢复正常。 + 当 DC2 从故障中恢复时，TiDB 进入 `sync_recover` 状态，将 DC1 中的节点的最新数据同步到 DC2 中的节点。此过程不会影响数据库的应用访问可用性。 + 数据同步完成后，TiDB 集群重新切换回 `sync` 模式。 ## 模块 3: 通过终止 DC1 中的所有节点模拟主数据中心故障 本练习的目标是模拟主数据中心完全不可用，并恢复其可用性。 当 DC1 完全故障时，DC2 仍然保留着最新的数据在跟随 TiKV 实例中。结合备份的 learner TiKV 实例，可以迅速恢复一个两副本集群，继续提供服务。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-7-en.png) 我们需要通过灾备数据中心中的 PD 启动一个新的 TiDB 集群（保留数据，但有一个新的集群 ID）。另外，我们需要扩展一个 PD 节点（预留的备用 PD 节点），并将灾备数据中心中的 learner TiKV 节点转换为投票 TiKV，以便与另一个 TiKV 节点组成一个功能齐全的 Raft group。 ### 步骤 1. 连接到 DC1 中的监视服务器 + 打开一个本地终端（终端 1），并连接 monitor 1 服务器。 + **macOS** 用户: ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 10** 用户: ```cmd ssh -i %userprofile%\.ssh\pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### 步骤 2. 启动工作负载 + 在 DC1 中启动 sysbench 工作负载。 ```bash sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 1 Report intermediate results every 1 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 ... ``` + 打开一个新的本地终端（终端 2），连接到 monitor 2 服务器。启动 DC2 中的 sysbench 工作负载。 ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az2monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + 为 DC2 设置 sysbench 配置。 ```bash cat << EOF > sysbench_config mysql-host=${HOST_AZ2DB1_PRIVATE_IP} mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=6 report-interval=1 EOF cat sysbench_config ``` ```output mysql-host=10.90.2.125 mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=16 report-interval=1 ``` + 在 DC2 中启动 sysbench 工作负载。 ```bash sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run ``` ```output Threads started! [ 1s ] thds: 6 tps: 316.47 qps: 6401.24 (r/w/o: 4486.46/1158.05/756.73) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00 [ 2s ] thds: 6 tps: 330.09 qps: 6611.89 (r/w/o: 4631.33/1171.34/809.23) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00 ... ``` ### 步骤 3. 验证复制状态是否同步 + 打开一个新的终端（终端 3），登录到 monitor 2 服务器，验证数据复制状态是否处于 `sync` 模式。 ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az2monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + 验证数据复制状态。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ### 步骤 4. 模拟主数据中心完全故障 + 模拟主数据中心的完全故障，通过终止 TiDB 集群中 DC1 节点上所有 TiDB 进程来模拟。 + 终止所有进程并移除 DC1 中每个 TiDB、TiKV 和 PD 节点的数据目录。如果有提示，回答 `yes`。 ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1DB1_PRIVATE_IP} "sudo systemctl stop tidb-4000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1KV1_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1KV2_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1PD1_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1PD2_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` + 终止监控的进程并移除数据目录。 ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1MONITOR1_PRIVATE_IP} "sudo systemctl stop prometheus-10090.service; sudo systemctl stop grafana-3000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ### 步骤 5. 观察应用程序和 TiDB 的状态 + 观察 TiDB 集群中的节点状态。所有的 PD 服务器状态为 `Down`，所有的 TiKV 状态为 `N/A`，且其中一台 TiDB 服务器为 `Down`。只有位于 DC2 的 TiDB 服务器仍然存活。 ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Grafana URL: http://10.90.1.182:3000,http://10.90.2.252:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.182:3000 grafana 10.90.1.182 3000 linux/x86_64 Down - /tidb-deploy/grafana-3000 10.90.2.252:3000 grafana 10.90.2.252 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.213:2379 pd 10.90.1.213 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.90:2379 pd 10.90.1.90 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.65:2379 pd 10.90.2.65 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.182:10090 prometheus 10.90.1.182 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.252:10090 prometheus 10.90.2.252 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.196:4000 tidb 10.90.1.196 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.2.226:4000 tidb 10.90.2.226 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.228:20160 tikv 10.90.1.228 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.91:20160 tikv 10.90.1.91 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.202:20160 tikv 10.90.2.202 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.206:20160 tikv 10.90.2.206 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` + 观察 TiDB 同步模式。由于无法访问位于 DC1 的 PD leader，我们需要访问位于 DC2 的 PD follower。此时操作超时，因为请求无法重定向到 leader 节点。 ```bash curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output [PD:apiutil:ErrRedirectNoLeader]redirect finds no leader ``` + 切换到终端 1。可以看到工作负载的 QPS 为 0，因为 TiDB 集群不可用。 ```output ... [ 57s ] thds: 6 tps: 193.99 qps: 3834.86 (r/w/o: 2679.90/665.98/488.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00 [ 58s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 59s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... ``` + 切换到终端 2。可以发现工作负载的 QPS 为 0，因为 TiDB 集群不可用。 ```output ... [ 34s ] thds: 6 tps: 116.00 qps: 2313.99 (r/w/o: 1624.99/385.00/304.00) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00 [ 35s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 36s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... ``` ### 步骤 6. 仅使用幸存节点重塑 TiDB 集群 + 切换到终端 3，使用仅包含位于 DC2（AZ2）节点的准备好的 meta 文件覆盖 TiUP 的 meta 文件。 ```bash cp /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml.backup ``` + 检查 yaml 文件 `majority-DC2.yaml`。 ```bash cat majority-DC2.yaml ``` ```output user: ec2-user tidb_version: vtidb_version topology: global: user: "ec2-user" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" arch: "amd64" monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 deploy_dir: "/tidb-deploy/monitored-9100" data_dir: "/tidb-data/monitor_9115" log_dir: "/tidb-deploy/monitor-9100/log" server_configs: pd: replication.location-labels: ["dc","logic","rack","host"] tidb_servers: - host: 10.90.2.210 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" tikv_servers: - host: 10.90.2.111 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic3",rack: "r1",host: "10.90.2.111" } - host: 10.90.2.91 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic4",rack: "r1",host: "10.90.2.91" } pd_servers: - host: 10.90.2.83 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://54.212.25.205:2379" monitoring_servers: - host: 10.90.2.107 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" grafana_servers: - host: 10.90.2.107 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" ``` + 使用准备好的文件替换 `meta` 文件。 ```bash cp -rf majority-DC2.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml ``` ### 步骤 7. 停止 TiDB 集群 + 检查 TiDB 集群状态。 ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.14:3000 grafana 10.90.2.14 3000 / Up - /tidb-deploy/grafana-3000 10.90.2.53:2379 pd 10.90.2.53 2379/2380 / Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.14:10090 prometheus 10.90.2.14 10090 / Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.97:4000 tidb 10.90.2.97 4000/10080 / Up - /tidb-deploy/tidb-4000 10.90.2.170:20160 tikv 10.90.2.170 20160/20180 / N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.78:20160 tikv 10.90.2.78 20160/20180 / N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 6 ``` + 停止 TiDB 集群中的节点。 ```bash tiup cluster stop tidb-demo -y ``` ```output ... Stopping component grafana Stopping instance 10.90.2.14 Stop grafana 10.90.2.14:3000 success Stopping component prometheus Stopping instance 10.90.2.14 Stop prometheus 10.90.2.14:10090 success Stopping component tidb Stopping instance 10.90.2.97 Stop tidb 10.90.2.97:4000 success Stopping component tikv Stopping instance 10.90.2.170 Stopping instance 10.90.2.78 Stop tikv 10.90.2.170:20160 success Stop tikv 10.90.2.78:20160 success Stopping component pd Stopping instance 10.90.2.53 Stop pd 10.90.2.53:2379 success Stopping component node_exporter Stopping instance 10.90.2.170 Stopping instance 10.90.2.14 Stopping instance 10.90.2.53 Stopping instance 10.90.2.97 Stopping instance 10.90.2.78 Stop 10.90.2.170 success Stop 10.90.2.78 success Stop 10.90.2.53 success Stop 10.90.2.97 success Stop 10.90.2.14 success Stopping component blackbox_exporter Stopping instance 10.90.2.170 Stopping instance 10.90.2.14 Stopping instance 10.90.2.97 Stopping instance 10.90.2.78 Stopping instance 10.90.2.53 Stop 10.90.2.78 success Stop 10.90.2.170 success Stop 10.90.2.97 success Stop 10.90.2.53 success Stop 10.90.2.14 success Stopped cluster `tidb-demo` successfully ``` + 检查 TiDB 节点状态，所有 PD、TiDB、TiKV 和 Prometheus 都为 `Down`。 ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Down - /tidb-deploy/grafana-3000 10.90.2.125:2379 pd 10.90.2.125 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.173:4000 tidb 10.90.2.173 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.2.129:20160 tikv 10.90.2.129 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.216:20160 tikv 10.90.2.216 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 6 ``` ### 步骤 8. 恢复 Placement Driver + 安装 `pd-recover` 工具。 ```bash tiup install pd-recover:vtidb_version ``` ```output download https://tiup-mirrors.tidb.io/pd-recover-vtidb_version-darwin-arm64.tar.gz 13.88 MiB / 13.88 MiB 100.00% 6.99 MiB/s ``` + 通过 SSH 登录位于 DC2 的 PD 服务器，强制启动一个新集群。登录到 PD。如果出现提示，请回答 `yes`. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ2PD1_PRIVATE_IP} ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + 检查 PD 的状态，其显示 TiDB 集群在 DC1 丢失前的同步模式，以及当前 TiDB 的数据一致性。 ```bash cat /tidb-data/pd_data/DR_STATE ``` ```output {"state":"sync","state_id":1947} ``` + 通过添加 `--fource-new-cluster` 参数修改 PD 的启动脚本。 ```bash sed -i '/pd-server/ s#pd-server#pd-server --force-new-cluster#' /tidb-deploy/pd-2379/scripts/run_pd.sh ``` + 从 PD 退出到 TiUP 中控机。 ```bash exit ``` ```output logout Connection to 10.90.2.26 closed. ``` + 通过启动 PD 节点创建一个新集群。 ```bash tiup cluster start tidb-demo -N ${HOST_AZ2PD1_PRIVATE_IP}:2379 -y ``` ```output ... Starting cluster tidb-demo... + [ Serial ] - SSHKeySet: privateKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa, publicKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa.pub + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.26 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.254 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.182 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.49 + [ Serial ] - StartCluster Starting component pd Starting instance 10.90.2.26:2379 Start instance 10.90.2.26:2379 success Starting component node_exporter Starting instance 10.90.2.26 Start 10.90.2.26 success Starting component blackbox_exporter Starting instance 10.90.2.26 Start 10.90.2.26 success + [ Serial ] - UpdateTopology: cluster=tidb-demo Started cluster `tidb-demo` successfully ``` + 确认 PD 已启动并运行正常。 ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.94.179.113:2379/dashboard Grafana URL: http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.107:3000 grafana 10.90.2.107 3000 / Down - /tidb-deploy/grafana-3000 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Down - /tidb-deploy/tidb-4000 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 6 ``` + 修复 pd 的元数据。此操作完成后，存储中的 `alloc_id` 会自动安全地增加 `100000000`。 ```bash tiup pd-recover:vtidb_version --from-old-member --endpoints=http://${HOST_AZ2PD1_PRIVATE_IP}:2379 ``` ```output download https://tiup-mirrors.pingcap.com/pd-recover-vtidb_version-linux-amd64.tar.gz 15.87 MiB / 15.87 MiB 100.00% 3.78 MiB/s Checking updates for component pd-recover... Starting component pd-recover: /home/ec2-user/.tiup/components/pd-recover/v8.5.0/pd-recover --from-old-member --endpoints=http://10.90.2.31:2379 recover success! please restart the PD cluster ``` ### 步骤 9. 扩展一台新的 Placement Driver 服务器 + 检查 YAML 文件。 ```bash cat scale-out-pd.yaml ``` ```output pd_servers: - host: 10.90.2.102 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://35.161.129.28:2379" ``` + 扩容 PD 节点。 ```bash tiup cluster scale-out tidb-demo scale-out-pd.yaml --user ec2-user -i ~/.ssh/pe-class-key-oregon.pem -y ``` ```output ... Starting component pd Starting instance 10.90.2.102:2379 Start instance 10.90.2.102:2379 success Starting component node_exporter Starting instance 10.90.2.102 Start 10.90.2.102 success Starting component blackbox_exporter Starting instance 10.90.2.102 Start 10.90.2.102 success + Refresh components conifgs - Generate config pd -> 10.90.2.26:2379 ... Done - Generate config pd -> 10.90.2.102:2379 ... Done - Generate config tikv -> 10.90.2.254:20160 ... Done - Generate config tikv -> 10.90.2.182:20160 ... Done - Generate config tidb -> 10.90.2.49:4000 ... Done - Generate config prometheus -> 10.90.2.38:10090 ... Done - Generate config grafana -> 10.90.2.38:3000 ... Done + Reload prometheus and grafana - Reload prometheus -> 10.90.2.38:10090 ... Done - Reload grafana -> 10.90.2.38:3000 ... Done + [ Serial ] - UpdateTopology: cluster=tidb-demo Scaled cluster `tidb-demo` out successfully ``` + 检查 DC2 中的两个 PD 节点是否均已启动并正常运行。 ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.94.179.113:2379/dashboard Grafana URL: http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.107:3000 grafana 10.90.2.107 3000 / Up - /tidb-deploy/grafana-3000 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.96:2379 pd 10.90.2.96 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Down - /tidb-deploy/tidb-4000 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 7 ``` ### 步骤 10. 将集群设置为 `majority` 模式 + 验证集群仍处于 `dr-auto-sync` 复制模式。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. download https://tiup-mirrors.tidb.io/ctl-vtidb_version-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 97.75 MiB/s Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.83:2379 config show replication-mode { "replication-mode": "dr-auto-sync", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` + 检查 `majority` 规则文件。 ```bash cat majority-rules.json ``` ```output [ { "group_id": "pd", "group_index": 0, "group_override": false, "rules": [ { "group_id": "pd", "id": "voters", "start_key": "", "end_key": "", "role": "voter", "count": 3, "location_labels": ["dc","logic","rack", "host"] } ] } ] ``` + 加载 `majority` Placement Rules 配置。 ```bash tiup ctl:vtidb_version pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/majority-rules.json" ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/majority-rules.json "Update rules and groups successfully." ``` + 检查 Placement Rules 是否已加载。 ```bash tiup ctl:vtidb_version pd config placement-rules show -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules show -u 10.90.2.125:2379 [ { "group_id": "pd", "id": "voters", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 3, "location_labels": [ "dc", "logic", "rack", "host" ], "create_timestamp": 1734071450 } ] ``` + 启用 `majority` 配置。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode majority tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set max-replicas 3 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 config set replication-mode majority Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 config set max-replicas 3 Success! ``` + 验证集群 `majority` 复制模式已生效。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 config show replication-mode { "replication-mode": "majority", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` ### 步骤 11. 删除不可用的 TiKV 节点 + 启动 TiKV 节点。 ```bash tiup cluster start tidb-demo -R tikv -y ``` ```output ... Starting component tikv Starting instance 10.90.2.129:20160 Starting instance 10.90.2.216:20160 Start instance 10.90.2.216:20160 success Start instance 10.90.2.129:20160 success Starting component node_exporter Starting instance 10.90.2.129 Starting instance 10.90.2.216 Start 10.90.2.216 success Start 10.90.2.129 success Starting component blackbox_exporter Starting instance 10.90.2.129 Starting instance 10.90.2.216 Start 10.90.2.216 success Start 10.90.2.129 success + [ Serial ] - UpdateTopology: cluster=tidb-demo Started cluster `tidb-demo` successfully ... ``` + 等待两个 TiKV 节点启动并正常运行。 ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.125:2379 pd 10.90.2.125 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.135:2379 pd 10.90.2.135 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.173:4000 tidb 10.90.2.173 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.2.129:20160 tikv 10.90.2.129 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.216:20160 tikv 10.90.2.216 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 7 ``` + 获取位于 DC1（AZ1）的 store ID。可以发现有两个存储节点的状态为 `Down` 或 `Disconnected`。记录这些 ID，以便在下一步使用。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 store ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 store { "count": 4, "stores": [ { "store": { "id": 16, ... "state_name": "Up" }, "status": { ... } }, { "store": { "id": 1, ... "state_name": "Up" }, "status": { ... } }, { "store": { "id": 2, ... "state_name": "Down" }, "status": { ... } }, { "store": { "id": 15, ... "state_name": "Down" }, "status": { ... } } ] } ``` + 如果 `state_name` 为 `Down`，请移除不可用的 TiKV 节点。使用你在上一步中记录的 store ID 替换命令中的 n1 和 n2。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores n1,n2 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 unsafe remove-failed-stores 2,15 Success! Request has been accepted." ``` + 显示移除操作的状态。一段时间后，你可以通过以下描述确认该过程已完成：命令结果末尾显示 `"info": "Unsafe recovery Finished"`。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores show ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 unsafe remove-failed-stores show [ { "info": "Unsafe recovery enters collect report stage", "time": "2024-12-13 07:01:51.714", "details": [ "Failed stores 2, 15" ] }, { "info": "Unsafe recovery enters force leader stage", "time": "2024-12-13 07:01:54.000", "actions": { "store 1": [ "force leader on regions: 7, 9, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 101, 104, 107, 110, 113, 116, 119, 122, 125, 128, 131, 134, 137, 140, 143, 146, 149, 152, 155, 158, 209, 213, 217, 221, 225, 229, 233, 237, 241, 245, 249, 253, 1978, 2013, 2018, 1993, 2043, 2048, 1983, 2023, 2028, 1988, 2033, 2038, 1998, 2063, 2068, 2008, 2053, 2058, 35, 31, 27, 23, 19, 17, 11, 13, 3" ], "store 16": [ "force leader on regions: " ] } }, { "info": "Unsafe recovery enters demote Failed voter stage", "time": "2024-12-13 07:02:04.586", "actions": { "store 1": [ "region 7 demotes peers { id:75 store_id:2 }, { id:170 store_id:15 }", ... "region 3 demotes peers { id:77 store_id:2 }, { id:208 store_id:15 }" ] } }, { "info": "Unsafe recovery Finished", "time": "2024-12-13 07:02:14.002", "details": [ "affected table ids: 74, 94, 110, 24, 58, 78, 92, 107, 281474976710649, 56, 82, 84, 106, 108, 109, 281474976710652, 281474976710654, 52, 111, 50, 66, 281474976710653, 48, 98, 54, 38, 42, 46, 62, 72, 76, 6, 281474976710651, 86, 28, 30, 40, 68, 96, 20, 60, 4, 18, 34, 80, 12, 10, 32, 70, 8, 64, 22, 36, 88, 90, 102, 14, 26, 44, 281474976710650, 16", "no newly created empty regions" ] } ] ``` ### 步骤 12. 启动恢复的 TiDB 集群 + 启动 tidb 和 Prometheus 实例。 ```bash tiup cluster start tidb-demo -R tidb,prometheus -y ``` ```output Starting component tidb Starting instance 10.90.2.49:4000 Start instance 10.90.2.49:4000 success Starting component prometheus Starting instance 10.90.2.38:10090 Start instance 10.90.2.38:10090 success Starting component node_exporter Starting instance 10.90.2.38 Starting instance 10.90.2.49 Start 10.90.2.49 success Start 10.90.2.38 success Starting component blackbox_exporter Starting instance 10.90.2.38 Starting instance 10.90.2.49 Start 10.90.2.49 success Start 10.90.2.38 success + [ Serial ] - UpdateTopology: cluster=tidb-demo Started cluster `tidb-demo` successfully ``` + 检查 TiDB 集群的状态。确认 DC2 中所有节点的状态均为 `Up`。 ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.94.179.113:2379/dashboard Grafana URL: http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.125:2379 pd 10.90.2.125 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.135:2379 pd 10.90.2.135 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.173:4000 tidb 10.90.2.173 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.129:20160 tikv 10.90.2.129 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.216:20160 tikv 10.90.2.216 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 7 ``` + 验证 TiDB 集群的复制模式。 ```bash curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "majority", "dr-auto-sync": { "label_key": "", "state": "", "acid_consistent": false } } ``` ### 步骤 13. 验证 sysbench 工作负载是否已恢复 + 切换到终端 1，Sysbench 工作负载的 QPS 仍为 0，因为位于 DC1 的 TiDB 服务器不可用。 ```output ... [ 1697s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 1698s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 1699s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ``` + 切换到终端 2，Sysbench 工作负载的 QPS 恢复正常。 ```output ... [ 1618s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 1619s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 6.00 [ 1620s ] thds: 6 tps: 321.06 qps: 6505.17 (r/w/o: 4564.82/1103.20/837.15) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00 [ 1621s ] thds: 6 tps: 313.95 qps: 6261.93 (r/w/o: 4378.25/1098.81/784.87) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 ``` ### 步骤 14. 再次启用 DR Auto-Sync + 当 DC1 恢复后，需要将 TiDB 集群恢复为 DR Auto-Sync 部署。 + 将丢失的节点重新整合回 TiDB 集群。检查 YAML 文件配置。 ```bash cat scale-out-DC1.yaml ``` ```output tidb_servers: - host: 10.90.1.58 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" tikv_servers: - host: 10.90.1.219 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic1",rack: "r1",host: "10.90.1.219" } - host: 10.90.1.71 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic2",rack: "r1",host: "10.90.1.71" } pd_servers: - host: 10.90.1.95 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://34.214.79.226:2379" - host: 10.90.1.55 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://54.202.186.157:2379" monitoring_servers: - host: 10.90.1.221 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" grafana_servers: - host: 10.90.1.221 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" ``` + 扩容 DC1 中的节点。 ```bash tiup cluster scale-out tidb-demo scale-out-DC1.yaml --user ec2-user -i ~/.ssh/pe-class-key-oregon.pem -y ``` ```output ... Starting component pd Starting instance 10.90.1.95:2379 Start instance 10.90.1.95:2379 success Starting instance 10.90.1.55:2379 Start instance 10.90.1.55:2379 success Starting component tikv Starting instance 10.90.1.71:20160 Starting instance 10.90.1.219:20160 Start instance 10.90.1.71:20160 success Start instance 10.90.1.219:20160 success Starting component tidb Starting instance 10.90.1.58:4000 Start instance 10.90.1.58:4000 success Starting component prometheus Starting instance 10.90.1.221:10090 Start instance 10.90.1.221:10090 success Starting component grafana Starting instance 10.90.1.221:3000 Start instance 10.90.1.221:3000 success Starting component node_exporter Starting instance 10.90.1.55 Starting instance 10.90.1.58 Starting instance 10.90.1.221 Starting instance 10.90.1.95 Starting instance 10.90.1.219 Starting instance 10.90.1.71 Start 10.90.1.71 success Start 10.90.1.95 success Start 10.90.1.219 success Start 10.90.1.55 success Start 10.90.1.221 success Start 10.90.1.58 success Starting component blackbox_exporter Starting instance 10.90.1.55 Starting instance 10.90.1.58 Starting instance 10.90.1.221 Starting instance 10.90.1.95 Starting instance 10.90.1.219 Starting instance 10.90.1.71 Start 10.90.1.219 success Start 10.90.1.95 success Start 10.90.1.71 success Start 10.90.1.55 success Start 10.90.1.221 success Start 10.90.1.58 success + Refresh components conifgs - Generate config pd -> 10.90.2.83:2379 ... Done - Generate config pd -> 10.90.2.96:2379 ... Done - Generate config pd -> 10.90.1.95:2379 ... Done - Generate config pd -> 10.90.1.55:2379 ... Done - Generate config tikv -> 10.90.2.111:20160 ... Done - Generate config tikv -> 10.90.2.91:20160 ... Done - Generate config tikv -> 10.90.1.219:20160 ... Done - Generate config tikv -> 10.90.1.71:20160 ... Done - Generate config tidb -> 10.90.2.210:4000 ... Done - Generate config tidb -> 10.90.1.58:4000 ... Done - Generate config prometheus -> 10.90.2.107:10090 ... Done - Generate config prometheus -> 10.90.1.221:10090 ... Done - Generate config grafana -> 10.90.2.107:3000 ... Done - Generate config grafana -> 10.90.1.221:3000 ... Done + Reload prometheus and grafana - Reload prometheus -> 10.90.2.107:10090 ... Done - Reload grafana -> 10.90.2.107:3000 ... Done + [ Serial ] - UpdateTopology: cluster=tidb-demo Scaled cluster `tidb-demo` out successfully ``` + 检查 TiDB 集群状态，确认所有节点已恢复。 ```bash tiup cluster display tidb-demo ``` ```output ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.221:3000 grafana 10.90.1.221 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.107:3000 grafana 10.90.2.107 3000 / Up - /tidb-deploy/grafana-3000 10.90.1.55:2379 pd 10.90.1.55 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.95:2379 pd 10.90.1.95 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.96:2379 pd 10.90.2.96 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.221:10090 prometheus 10.90.1.221 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.58:4000 tidb 10.90.1.58 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Up - /tidb-deploy/tidb-4000 10.90.1.219:20160 tikv 10.90.1.219 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.71:20160 tikv 10.90.1.71 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 14 ``` + 加载 DR Auto-Sync Placement Rules 配置。 ```bash tiup ctl:vtidb_version pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/dr-anto-sync-rules.json" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. download https://tiup-mirrors.tidb.io/ctl-vtidb_version-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 103.24 MiB/s Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/dr-anto-sync-rules.json "Update rules and groups successfully." ``` + 安装 `jq` 工具。 ```bash sudo yum install epel-release sudo yum install jq -y ``` ```output ... Installed: jq.x86_64 0:1.5-1.amzn2.0. 2 Dependency Installed: oniguruma.x86_64 0:5.9.6-1.amzn2.0. 7 Complete! ``` + 检查是否存在没有 Learner Peers 的 Region。验证是否没有 Region 缺少 Learner Peers，并确保投票角色和 Learner 角色之间的数据同步。此步骤保证集群的数据在所有角色中一致复制，以维持高可用性和故障转移准备。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 region --jq='.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]}' ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://34.215.73.186:2379 region --jq=.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]} ``` + 提升 Learner Peers 的角色转换（可选操作）。如果仍有 Region 未转换为 Learner Peers，可使用以下命令提升 Region 转换为 Learner Region。第一个数字为 Region ID，第二个数字为 Store ID。如果上一步没有结果，则请`跳过此步骤`。 ```bash operator add remove-peer 6066 6 ``` + 启用 DR Auto-Sync 配置。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync label-key dc tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary dc1 tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr dc2 tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary-replicas 2 tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr-replicas 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync label-key dc Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary dc1 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr dc2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary-replicas 2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr-replicas 1 Success! The replication mode config is updated." ``` ### 步骤 15. 验证复制状态是否恢复为 `sync` + 验证配置是否生效。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379:2379 config show replication-mode { "replication-mode": "dr-auto-sync", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` + 检查集群的当前复制状态。在状态变为 `sync` 模式之前，如果操作较快，可以看到 `sync_recover` 模式和 `recover_progress` 状态。 ```bash curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 2003, "acid_consistent": false, "total_regions": 60, "synced_regions": 37, "recover_progress": 0.98333335 } } ``` + 最终，TiDB 将恢复到 `sync` 状态。 ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 378, "acid_consistent": true } } ``` + 检查 sysbench 工作负载是否已恢复。切换到终端 1，sysbench 工作负载的 QPS 恢复正常。 ```output ... [ 2309s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 2310s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 2311s ] thds: 6 tps: 105.01 qps: 2170.15 (r/w/o: 1530.11/416.03/224.02) lat (ms,95%): 100000.00 err/s: 0.00 reconn/s: 6.00 [ 2312s ] thds: 6 tps: 193.97 qps: 3869.44 (r/w/o: 2705.61/763.89/399.94) lat (ms,95%): 34.33 err/s: 0.00 reconn/s: 0.00 ``` ### 步骤 16. 将 PD 和 TiKV 的领导者转移到 DC1 + 缩容 DC2 中的第二个 PD 节点。 ```bash tiup cluster scale-in tidb-demo -N "${HOST_AZ2PD2_PRIVATE_IP}:2379" -y ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` + 无论上一步结果如何，您可以执行此命令尝试将 pd leader 转移至 DC1。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 member leader transfer "pd-${HOST_AZ1PD1_PRIVATE_IP}-2379" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` + 检查 TiDB 集群状态，确保 PD 的 Leader 位于 DC1 且 PD 节点数变为 3。 ```bash tiup cluster display tidb-demo ``` ```output ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.221:3000 grafana 10.90.1.221 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.107:3000 grafana 10.90.2.107 3000 / Up - /tidb-deploy/grafana-3000 10.90.1.55:2379 pd 10.90.1.55 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.95:2379 pd 10.90.1.95 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.221:10090 prometheus 10.90.1.221 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.58:4000 tidb 10.90.1.58 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Up - /tidb-deploy/tidb-4000 10.90.1.219:20160 tikv 10.90.1.219 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.71:20160 tikv 10.90.1.71 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` + 通过检查每个角色的 Region 分布，我们可以确认 DR Auto-Sync 配置已成功应用于 TiKV 节点。 ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 store ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.83:2379 store { "count": 4, "stores": [ { "store": { "id": 1, "address": "10.90.2.111:20160", ... }, "status": { "leader_count": 0, "region_count": 79, ... } }, { "store": { "id": 14, "address": "10.90.2.91:20160", ... }, "status": { "leader_count": 0, "region_count": 79, "learner_count": 79, ... } }, { "store": { "id": 2001, "address": "10.90.1.219:20160", ... }, "status": { "leader_count": 42, "region_count": 79, ... } }, { "store": { "id": 2002, "address": "10.90.1.71:20160", ... }, "status": { "leader_count": 37, "region_count": 79, ... } } ] } ``` + 从以上信息中，我们可以通过 Region 分布确定每个 TiKV 节点的角色（您的 store id 可能与以下示例不同）： + Store 1 是 `follower`：`"leader_count": 0, "region_count": 79` + Store 14 是 `learner`：`"leader_count": 0, "region_count": 79, "learner_count": 79` + Store 2001 是 `voter`：`"leader_count": 42, "region_count": 79` + Store 2002 是 `voter`：`"leader_count": 37, "region_count": 79` ### 模块 3 小结 + 通过本次实践，我们观察到即使主数据中心发生完全故障，我们仍然可以成功恢复 TiDB 集群并确保 RPO=0。大致来说，恢复操作包括以下步骤： + 恢复位于 DC2 的唯一 PD，并使用其创建新集群。 + 在 DC2 中扩容一个新的 PD，组成一个新的 Raft 组。 + 修改集群模式为多数模式（majority mode）。 + 移除位于 DC1 的不可用 TiKV 节点。 + 重启所有集群节点并运行剩余的两个 TiKV 节点，观察应用读写操作的恢复情况。 + 修复故障。 + 初始化 DC1 中的节点并将它们重新加入 TiDB 集群。 + 修改集群配置，恢复至 DR Auto-Sync 模式。 ## 结论 通过以上三次实践，我们验证了 TiDB DR Auto-Sync 部署解决方案的高可用性，确保即使在数据中心级别的故障中也不会发生数据丢失。 尤其是在金融服务行业（FSI），许多客户在双数据中心环境中运营以实现高可用性。TiDB 引入了 DR Auto-Sync 方案，以解决双数据中心部署中的挑战，在保证数据一致性的同时提高了系统可用性。 如需深入了解如何使用 TiDB 数据迁移迁移到 TiDB，我们鼓励您探索 [TiDB 培训与认证](https://pingcap.com/education/) 并参加 TiDB 管理讲师指导的培训课程。这个全面的课程将为您提供成功管理和迁移到 TiDB 所需的知识和技能。