## 模块 2: 通过隔离 DC2 网络访问模拟灾备数据中心故障 该实践的目标是模拟 DC2 的完全不可用，观察 TiDB 集群的状态并恢复它。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-5-en.png) 模拟 DC2 故障的方法是切断 DC1 和 DC2 之间的网络通信。这将阻止 DC2 的服务器接收或响应外部请求，从而让 DC1 认为 DC2 发生了完全故障。 ### 步骤 1. 连接到 DC1 中的 TiUP 中控机 + 打开 [终端1](ssh://az1monitor1-ssh#1)，该终端已连接到数据中心1的 monitor1。 ### 步骤 2. 启动模拟应用负载 + 启动 sysbench 模拟应用负载。 ```bash sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 1 Report intermediate results every 1 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 ... ``` ### 步骤 3. 验证数据复制状态 + 点击打开 [终端2](ssh://az1monitor1-ssh#2)，该终端已连接到 DC1 的 monitor 1，验证数据复制状态为 `sync` 模式。 + 验证数据复制状态。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ### 步骤 4. 阻塞整个 DC2（AZ2）的网络访问 + 执行脚本通过阻塞子网 demo-subnet-2（位于 AZ2 的子网 10.90.2.0/24）来模拟完整的可用区故障。 ```bash VPC_ID=`aws ec2 describe-vpcs \ --filters "Name=tag:Name,Values=${LAB:USER_UNIQUE_TAG}" \ --query "Vpcs[0].VpcId" \ --output text \ --region ${LAB:REGION_CODE}` SUBNET_ID=`aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \ --query "Subnets[0].SubnetId" \ --output text \ --region ${LAB:REGION_CODE}` CAGE_NACL_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" "Name=tag:Name,Values=cage" \ --query "NetworkAcls[0].NetworkAclId" \ --output text \ --region ${LAB:REGION_CODE}` ASSOC_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \ --query "NetworkAcls[0].Associations" \ --output text \ --region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'` aws ec2 replace-network-acl-association \ --association-id ${ASSOC_ID} \ --network-acl-id ${CAGE_NACL_ID} \ --region ${LAB:REGION_CODE} ``` ```output { "NewAssociationId": "aclassoc-0402a6103febf799f" } ``` ### 步骤 5. 观察应用程序和 TiDB 的状态 + 返回 [终端 1](ssh://az1monitor1-ssh#1), sysbench 被阻塞访问 TiDB 集群约 70 秒，稍长于阻塞窗口设置（默认设置为 60 秒）。原因是 TiDB 集群需要时间收集网络状态，处理 TiDB 集群状态切换，并将状态同步至每个 TiKV。 ```output ... [ 493s ] thds: 6 tps: 278.00 qps: 5525.08 (r/w/o: 3866.05/1082.02/577.01) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 [ 494s ] thds: 6 tps: 274.99 qps: 5505.84 (r/w/o: 3853.89/1079.97/571.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 [ 495s ] thds: 6 tps: 271.99 qps: 5447.84 (r/w/o: 3814.89/1065.97/566.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00 [ 496s ] thds: 6 tps: 70.00 qps: 1406.04 (r/w/o: 985.03/278.01/143.00) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00 [ 497s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 498s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... [ 544s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 545s ] thds: 6 tps: 0.00 qps: 29.00 (r/w/o: 26.00/0.00/3.00) lat (ms,95%): 0.00 err/s: 3.00 reconn/s: 0.00 [ 546s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... [ 566s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 567s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 568s ] thds: 6 tps: 98.00 qps: 1968.01 (r/w/o: 1376.00/388.00/204.00) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00 ... [ 569s ] thds: 6 tps: 244.00 qps: 4883.91 (r/w/o: 3418.93/950.98/513.99) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00 [ 570s ] thds: 6 tps: 237.00 qps: 4734.95 (r/w/o: 3313.96/924.99/495.99) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 [ 571s ] thds: 6 tps: 264.02 qps: 5279.31 (r/w/o: 3693.21/1039.06/547.03) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00 ... ``` + 返回 [终端2](ssh://az1monitor1-ssh#2), 观察 TiDB 集群中节点的状态。位于 DC2 的节点全部处于 `Down` 状态。 ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://18.236.254.103:2379/dashboard Grafana URL: http://10.90.1.90:3000,http://10.90.2.17:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.90:3000 grafana 10.90.1.90 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.17:3000 grafana 10.90.2.17 3000 linux/x86_64 Down - /tidb-deploy/grafana-3000 10.90.1.176:2379 pd 10.90.1.176 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.73:2379 pd 10.90.1.73 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.198:2379 pd 10.90.2.198 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.90:10090 prometheus 10.90.1.90 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.17:10090 prometheus 10.90.2.17 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.106:4000 tidb 10.90.1.106 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.61:4000 tidb 10.90.2.61 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.1.245:20160 tikv 10.90.1.245 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.45:20160 tikv 10.90.1.45 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.125:20160 tikv 10.90.2.125 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.96:20160 tikv 10.90.2.96 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` + 观察 TiDB 同步模式的变化，从 `sync` 切换到 `sync_wait`，然后是 `async`。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "async_wait", "state_id": 1001, "acid_consistent": true } } ``` ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "async", "state_id": 483, "acid_consistent": true } } ``` ### 步骤 6. 恢复 DC2 的网络访问 + 恢复 DC2 的网络访问。之前丢失的节点将重新加入 TiDB 集群。 ```bash VPC_ID=`aws ec2 describe-vpcs \ --filters "Name=tag:Name,Values=${LAB:USER_UNIQUE_TAG}" \ --query "Vpcs[0].VpcId" \ --output text \ --region ${LAB:REGION_CODE}` SUBNET_ID=`aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \ --query "Subnets[0].SubnetId" \ --output text \ --region ${LAB:REGION_CODE}` DEFAULT_NACL_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \ --query "NetworkAcls[0].NetworkAclId" \ --output text \ --region ${LAB:REGION_CODE}` ASSOC_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" \ --query "NetworkAcls[0].Associations" \ --output text \ --region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'` aws ec2 replace-network-acl-association \ --association-id ${ASSOC_ID} \ --network-acl-id ${DEFAULT_NACL_ID} \ --region ${LAB:REGION_CODE} ``` ```output { "NewAssociationId": "aclassoc-0df1e05cfc70356a8" } ``` + 返回 [终端 1](ssh://az1monitor1-ssh#1)，观察 Sysbench 和 TiDB 集群的状态。 ```output [ 1039s ] thds: 6 tps: 268.99 qps: 5409.79 (r/w/o: 3787.85/1067.96/553.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 [ 1040s ] thds: 6 tps: 273.01 qps: 5455.15 (r/w/o: 3814.11/1083.03/558.02) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00 [ 1041s ] thds: 6 tps: 265.00 qps: 5282.95 (r/w/o: 3702.96/1043.99/535.99) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00 ... [ 1105s ] thds: 6 tps: 260.00 qps: 5244.02 (r/w/o: 3669.02/1043.00/532.00) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00 [ 1106s ] thds: 6 tps: 267.99 qps: 5339.86 (r/w/o: 3740.90/1052.97/545.99) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 ``` + 查看 TiDB 集群状态。DC2 中的节点都已经恢复。如果您的操作足够快，可能会看到一些 TiKV 节点的状态从 `Disconnected` 逐渐切换到 `Up`。 ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.92.147.161:2379/dashboard Grafana URL: http://10.90.1.212:3000,http://10.90.2.99:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.212:3000 grafana 10.90.1.212 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.99:3000 grafana 10.90.2.99 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.156:2379 pd 10.90.1.156 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.232:2379 pd 10.90.1.232 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.104:2379 pd 10.90.2.104 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.212:10090 prometheus 10.90.1.212 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.99:10090 prometheus 10.90.2.99 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.23:4000 tidb 10.90.1.23 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.122:4000 tidb 10.90.2.122 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.178:20160 tikv 10.90.1.178 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.6:20160 tikv 10.90.1.6 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.250:20160 tikv 10.90.2.250 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.90:20160 tikv 10.90.2.90 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.250:20160 tikv 10.90.2.250 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 ... ``` + 检查 TiDB 集群当前的复制状态。在几秒钟内，状态从 `sync_recover` 切换到 `sync`。 ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-6-en.png) + `sync_recover` 模式。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 1009, "acid_consistent": false, "total_regions": 78, "recover_progress": 0.08974359 } } ``` + `sync` 模式。 ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1010, "acid_consistent": true } } ``` ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-61-en.png) + 此时，TiDB 将两个数据中心之间的数据同步恢复到 `sync` 状态，实现了 RPO=0 的数据中心故障恢复能力。 ### 步骤 7. 停止 Sysbench 工作负载 + 回到 [终端1](ssh://az1monitor1-ssh#1)，终止 sysbench 线程，并关闭终端1。 + 关闭终端2。 ### 模块 2 小结 + 通过上述步骤，我们模拟了 TiDB 处理整个 DC2 故障并恢复到可用状态的整个过程。 + 当 DC2 不可用时，TiDB 进入阻塞窗口以确认故障。在此阻塞窗口期间，应用程序对数据库的读写操作会受到影响。 + 超过阻塞窗口后，TiDB 仅使用 DC1 中的节点并转换为 `async` 状态。此时，应用程序的读写操作恢复正常。 + 当 DC2 从故障中恢复时，TiDB 进入 `sync_recover` 状态，将 DC1 中的节点的最新数据同步到 DC2 中的节点。此过程不会影响数据库的应用访问可用性。 + 数据同步完成后，TiDB 集群重新切换回 `sync` 模式。