--- title: "Dual Data Center Deployment Solution - DR Auto-Sync" version: "v2" status: "GA" description: "In the financial industry, a dual data center architecture is a common deployment model, especially for core business systems. Banks and financial institutions place a high premium on data availability and disaster recovery capabilities. TiDB's DR Auto-sync solution is specifically designed for such environments, providing robust data protection on dual data center infrastructures while ensuring an RPO (Recovery Point Objective) of 0 and an RTO (Recovery Time Objective) measured in minutes. This lab simulates the deployment of DR Auto-Sync in a dual data center scenario, demonstrating how to achieve data consistency and high availability when one of the data centers is unavailable or experiences network disruptions." --- # Dual Data Center Deployment Solution - DR Auto-Sync ## Overview In the financial industry, a dual data center architecture is a common deployment model, especially for core business systems. Banks and financial institutions place a high premium on data availability and disaster recovery capabilities. TiDB's DR Auto-Sync solution is specifically designed for such environments, providing robust data protection on dual data center infrastructures while ensuring an RPO (Recovery Point Objective) of 0 and an RTO (Recovery Time Objective) measured in minutes. Specifically, this solution offers the following features: + RPO = 0: Ensures no data loss even in the event of a data center failure. + Minute-level RTO: Enables service recovery within minutes after a failure. + Active-active services: Allows both data centers to simultaneously provide highly available data services. This lab simulates the deployment of DR Auto-Sync in a dual data center scenario, demonstrating how to achieve data consistency and high availability when one of the data centers is unavailable or experiences network disruptions. ### DR Auto-Sync Solution Introduction Challenges with Default Deployment + By default, all database replicas (3 replicas) form a Raft group where any two replicas can ensure database availability and data persistence. For each data write, as long as it is committed in any two TiKVs within the Raft group, the client will receive a success response. However, if Data Center 1 holds the two latest data replicas and it experiences a complete outage, these two replicas are lost, and the Raft group can not guarantee the latest data is replicated to Data Center 2. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-1-en.png) DR Auto-Sync Solution + This solution introduces the concept of a commit group to the Raft algorithm. For data to be committed, it must not only meet the majority rule but also be present in at least two commit groups. This guarantees that both data centers hold the latest data copies, ensuring data safety. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-2-en.png) Additionally, this solution includes an extra learner node in DC2, which doesn't participate in voting and is used to restore the TiDB cluster only when DC2 is the sole surviving data center. High Availability Scenarios with DR Auto-Sync + The DR auto-sync solution uses three distinct states to identify TiKV data synchronization modes. The replication mode of the cluster can automatically and adaptively switch between the three statuses. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-3-en.png) + Sync Mode: TiDB operates in synchronous replication mode by default. In this mode, at least one replica in the disaster recovery data center synchronizes with the primary data center. The Raft algorithm ensures that each data is replicated to the disaster recovery data center. In this state, the overall failure of any data center can ensure that the TiDB cluster achieves an RPO=0. + Async Mode: When a failure occurs in one entire data center, TiDB switches to asynchronous replication mode. In this mode, the data in these two data centers are not fully synchronized. The Raft algorithm degrades to follow only the majority quorum protocol, completing data replication within the surviving data center. Since a failure has already occurred, the TiDB cluster cannot guarantee an RPO=0 if a second failure happens. The cluster can only regain its RPO=0 high availability capability after the initial failure is resolved. + Sync-recover Mode: The data center failure has been resolved, and data synchronization between the two data centers is in progress. TiDB switches to synchronous recovery mode. In this mode, these two data centers are not fully synchronized. After data synchronization is completed, TiDB switches back to Sync Mode. Then, the TiDB cluster regains RPO=0 HA capability. ### Tasks + Module 1: Demonstration 1: Set up the TiDB cluster and configure DR Auto-Sync deployment, verifying the correct setup of the high-availability mode. + Module 2: Simulate a disaster recovery data center failure or network isolation, observing the cluster's automatic switching and recovery mechanisms. + Module 3: Simulate a primary data center failure, verifying the cluster's failover behavior and rapid recovery capabilities. ### Objectives Through simulating a dual data center scenario, this lab demonstrates how TiDB's DR Auto-Sync solution achieves high availability and strong consistency across data centers. This solution not only resolves potential failures and data loss issues in traditional dual data center architectures but also enhances TiDB's flexibility and disaster recovery capabilities through dynamic synchronization mode switching. DR Auto-Sync is particularly suitable for core business scenarios in the financial industry requiring high reliability and zero data loss, such as payment systems, clearing systems, and account management systems. This lab offers an intuitive understanding of TiDB's unique advantages and practical capabilities in a dual data center architecture. ### Prerequisites + **macOS** users: + Click the private key file [pe-class-key-oregon.pem](/api/labs/demo_001/vm_key?region_name=oregon) to download it. This file is required to log in to the remote Linux for this hands-on lab. + Save the private key file from the download folder to `~/.ssh/` folder. + Use the following commands to set the permission and check whether the file is in place. ```bash chmod 400 ~/.ssh/pe-class-key-oregon.pem ls -lF ~/.ssh/pe-class-key-oregon.pem ``` ```output ... -r--------@ 1 username staff 1674 Jun 21 2023 /Users/username/.ssh/pe-class-key-oregon.pem ``` + **Windows 10** users: + Click the private key file [pe-class-key-oregon.pem](/api/labs/demo_001/vm_key?region_name=oregon) to download it. This file is required to log in to the remote Linux for this hands-on lab. + Save the private key file from the download folder in `%userprofile%\.ssh` folder. + Use the following command to check if the file is in place. Note that you should use CMD, not PowerShell. ```cmd dir %userprofile%\.ssh\pe-class-key-oregon.pem ``` ```output ... 2024/11/14 10:22 1,674 pe-class-key-oregon.pem 1 Files(s) 1,674 bytes ... ``` + **Windows 8** or earlier users: + Click the private key file [pe-class-key-oregon.ppk](/api/labs/demo_001/vm_key?region_name=oregon&key_type=ppk) to download it. This file is required to log in to the remote Linux for this hands-on lab. + Download and install [PuTTY](https://www.putty.org/). ### Duration + This lab should take you approximately 100 minutes to complete. > [!TIP] > Once the lab environment is deployed, click **Next** on the bottom bar to begin **Module 1**. ## Module 1: Set Up the TiDB Cluster With DR Auto-Sync Configuration In this practice, we will set up a cluster with DR Auto-Sync configuration enabled. The following nodes are distributed across two data centers: + DC1 as primary data center, located in aws AZ1: `pd*2, tidb*1, tikv*2, tiup(monitor)*1` + DC2 as disaster recovery data center, located in aws AZ2: `pd*1(with extra 1 node standby), tidb*1, tikv*2, tiup(monitor)*1` + Sysbench acts as the application and is deployed in each data center (Corresponding to the `client` in the diagram below). Note: The test simplifies the load balancing service, with sysbench in DC1 directly connecting to the TiDB server in DC1, and sysbench in DC2 doing the same. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-4-en.png) ### Step 1. Connect to a monitor node in the first data center + Open a local terminal (Terminal 1), connect to a monitor 1 server to deploy a TiDB cluster. Connect to remote bastion Linux. If you do not possess the credentials to connect, kindly review the `Prerequisites` section. If prompted, answer `yes`. + **macOS** users: ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 10** users: ```cmd ssh -i %userprofile%\.ssh\pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 8** or earlier users: + Run PuTTY, in `Session` -> `Host Name`, type in `ec2-user@monitor1_address`. + In `Connection` -> `SSH` -> `Auth` -> `Credentials` -> `Private key file for authentication`, browse to find the location of your key file. + Now, you should be able to connect to the remote Linux. + The banner after you logged in: ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### Step 2. Review the multi availability zone TiDB cluster topology configuration + Review the provisioned cluster topology YAML. ```bash cat dr-auto-sync.yaml ``` ```output global: user: "ec2-user" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" arch: "amd64" monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 deploy_dir: "/tidb-deploy/monitored-9100" data_dir: "/tidb-data/monitor_9115" log_dir: "/tidb-deploy/monitor-9100/log" server_configs: pd: replication.location-labels: ["dc","logic","rack","host"] tidb_servers: - host: 10.90.1.163 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" - host: 10.90.2.111 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" tikv_servers: - host: 10.90.1.80 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic1",rack: "r1",host: "10.90.1.80" } - host: 10.90.1.45 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic2",rack: "r1",host: "10.90.1.45" } - host: 10.90.2.33 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic3",rack: "r1",host: "10.90.2.33" } - host: 10.90.2.113 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic4",rack: "r1",host: "10.90.2.113" } pd_servers: - host: 10.90.1.19 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://35.88.184.77:2379" - host: 10.90.1.44 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://34.219.233.52:2379" - host: 10.90.2.108 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://18.236.155.129:2379" monitoring_servers: - host: 10.90.1.147 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" - host: 10.90.2.82 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" grafana_servers: - host: 10.90.1.147 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" - host: 10.90.2.82 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" ``` ### Step 3. Set TiUP mirror and execute `precheck` command ```bash tiup mirror set tidb-community-server-tidb_version-linux-amd64 tiup cluster check dr-auto-sync.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-oregon.pem \ --apply ``` ```output ... + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks - Applying changes on 10.90.2.33 ... Done - Applying changes on 10.90.2.113 ... Done - Applying changes on 10.90.1.163 ... Done - Applying changes on 10.90.2.111 ... Done - Applying changes on 10.90.1.147 ... Done - Applying changes on 10.90.1.44 ... Done - Applying changes on 10.90.2.108 ... Done - Applying changes on 10.90.1.45 ... Done - Applying changes on 10.90.2.82 ... Done - Applying changes on 10.90.1.19 ... Done - Applying changes on 10.90.1.80 ... Done ``` ### Step 4. Deploy the TiDB cluster ```bash tiup cluster deploy tidb-demo vtidb_version ./dr-auto-sync.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-oregon.pem \ --yes ``` ```output ... Cluster `tidb-demo` deployed successfully, you can start it with command: `tiup cluster start tidb-demo --init` ``` ### Step 5. Start the TiDB cluster ```bash tiup cluster start tidb-demo ``` ```output ... Started cluster `tidb-demo` successfully ``` ### Step 6. Verity the status of the TiDB cluster + Get the cluster status. Key cluster components are spread across 2 subnets (10.90.1.0/24 for DC1, 10.90.2.0/24 for DC2), which are located in 2 Availability Zones. For each data center, there are 2 PD servers(DC2 contains one standby pd node), 2 TiKV servers, and 1 TiDB servers. Therefore, we can use this deployment to simulate a two-data-center environment. ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.188.60.25:2379/dashboard Grafana URL: http://10.90.1.114:3000,http://10.90.2.122:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.114:3000 grafana 10.90.1.114 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.122:3000 grafana 10.90.2.122 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.114:10090 prometheus 10.90.1.114 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.122:10090 prometheus 10.90.2.122 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.81:4000 tidb 10.90.1.81 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.207:4000 tidb 10.90.2.207 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.163:20160 tikv 10.90.1.163 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.83:20160 tikv 10.90.1.83 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.17:20160 tikv 10.90.2.17 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.19:20160 tikv 10.90.2.19 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ### Step 7. Make sure the Placement Driver leader is located in data center 1 + If pd leader is not located in DC1 (IP belongs to 10.90.1.0/24 subnet), transfer the leader to DC1. ```output ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 ``` + You can execute this command to try to transfer pd leader to DC1 whatever the previous step's result is. ```bash tiup ctl:vtidb_version pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 member leader transfer "pd-${HOST_AZ1PD1_PRIVATE_IP}-2379" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` + Check the location of pd leader, which is switched to DC1. ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- ... 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 ... ``` ### Step 8. Create a peering TiUP monitor in data center 2 + Copy the cluster meta file of tiup to Monitor 2 (tiup directory) which is located at disaster recovery data center (DC2). If prompted, answer `yes`. ```bash scp -i ~/.ssh/pe-class-key-oregon.pem -r .tiup/storage/ ec2-user@az2monitor1_address:/home/ec2-user/.tiup/ ``` ```output ... run_blackbox_exporter_10.90.1.163.sh 100% 519 356.8KB/s 00:00 run_blackbox_exporter_10.90.2.82.sh 100% 519 414.9KB/s 00:00 meta.yaml 100% 4481 3.2MB/s 00:00 ``` + Open a new terminal (Terminal 2). Log in to Monitor 2 to verify that the tiup meta file is functioning properly. + Log in to monitor2. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az2monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + Confirm that `tiup` in monitor 2 can work well. ```bash tiup cluster display tidb-demo ``` ```output Checking updates for component cluster... A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.88.184.77:2379/dashboard Grafana URL: http://10.90.1.147:3000,http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.147:3000 grafana 10.90.1.147 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.44:2379 pd 10.90.1.44 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.108:2379 pd 10.90.2.108 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.147:10090 prometheus 10.90.1.147 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.163:4000 tidb 10.90.1.163 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.111:4000 tidb 10.90.2.111 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.45:20160 tikv 10.90.1.45 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.80:20160 tikv 10.90.1.80 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.113:20160 tikv 10.90.2.113 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.33:20160 tikv 10.90.2.33 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ### Step 9. Enable DR Auto-Sync for your TiDB cluster + Go back to Terminal 1. ```bash exit ``` + Check the current cluster data replication mode, which should be `majority` as default. Note: The result `"acid_consistent": false` means the data replication between primary data center and disaster recovery data center is not in `Sync` mode. So, the `ACID` properties of transactions cannot be absolutely guaranteed when primary data center experiences a complete failure. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "majority", "dr-auto-sync": { "label_key": "", "state": "", "acid_consistent": false } } ``` + View the placement rules file that prepared for DR Auto-Sync configuration. There are two data centers, and voter role is limited in DC1. In DC2, there is a follower and a learner role. ```bash cat dr-anto-sync-rules.json ``` ```output [ { "group_id": "pd", "group_index": 0, "group_override": false, "rules": [ { "group_id": "pd", "id": "dc1", "start_key": "", "end_key": "", "role": "voter", "count": 2, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "dc", "op": "in", "values": ["dc1"]}] }, { "group_id": "pd", "id": "logic3", "start_key": "", "end_key": "", "role": "follower", "count": 1, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "logic", "op": "in", "values": ["logic3"]}] }, { "group_id": "pd", "id": "logic4", "start_key": "", "end_key": "", "role": "learner", "count": 1, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "logic", "op": "in", "values": ["logic4"]}] } ] } ] ``` ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-41-en.png) + By importing the above placement rules, PD can perceive the distribution design of different nodes (Learner more: [PD Scheduling Best Practices - Cluster topology awareness](https://docs.pingcap.com/tidb/stable/pd-scheduling-best-practices#cluster-topology-awareness)). This enables the replicas of Regions within the cluster to be dispersed as much as possible. PD schedules based on the location-labels configuration. The configuration ["dc", "logic", "rack", "host"] defines a 4-level topology structure: + The cluster is divided into multiple dc (data centers). + Each dc contains multiple logic (logical zones). + Each logic contains multiple rack units. + Each rack contains multiple host units. + When scheduling, PD will: + Prioritize placing the Region's Peer in different dc. + If this is not feasible (e.g., 3 replicas are configured but there are only 2 data centers available), PD ensures placement in different logic zones. + If the number of logic zones is also insufficient to ensure isolation, it will then attempt to isolate at the rack level. + If the number of rack units is still inadequate, it will attempt isolation at the host level. + Load the placement rules configuration, the `ctl` tool will be downloaded and installed automatically. ```bash tiup ctl:vtidb_version pd config placement-rules rule-bundle \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 save \ --in="/home/ec2-user/dr-anto-sync-rules.json" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. download https://tiup-mirrors.pingcap.com/ctl-vtidb_version-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 103.24 MiB/s Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/dr-anto-sync-rules.json "Update rules and groups successfully." ``` + Check if placement rules configuration is loaded. ```bash tiup ctl:vtidb_version pd config placement-rules show -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules show -u http://10.90.2.162:2379 [ { "group_id": "pd", "id": "dc1", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 2, "label_constraints": [ { "key": "dc", "op": "in", "values": [ "dc1" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 }, { "group_id": "pd", "id": "logic3", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 1, "label_constraints": [ { "key": "logic", "op": "in", "values": [ "logic3" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 }, { "group_id": "pd", "id": "logic4", "start_key": "", "end_key": "", "role": "learner", "is_witness": false, "count": 1, "label_constraints": [ { "key": "logic", "op": "in", "values": [ "logic4" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 } ] ``` + Install `jq` tool. ```bash sudo yum install jq -y ``` ```output ... Installed: jq.x86_64 0:1.5-1.amzn2.0. 2 Dependency Installed: oniguruma.x86_64 0:5.9.6-1.amzn2.0. 7 Complete! ``` + Check for TiKV regions without learner peers. Verify that there are no regions without learner peers and ensure data synchronization between voter and learner roles. This step ensures that the cluster's data is consistently replicated across all roles, which is essential for maintaining high availability and failover readiness. ```bash tiup ctl:vtidb_version pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 region \ --jq='.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]}' ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://34.215.73.186:2379 region --jq=.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]} ``` + Enable the DR Auto-Sync configuration. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync label-key dc tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary dc1 tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr dc2 tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary-replicas 2 tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr-replicas 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync label-key dc Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary dc1 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr dc2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary-replicas 2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr-replicas 1 Success! The replication mode config is updated." ``` + Through the configuration of these parameters, the cluster achieves location awareness of data centers and control over data distribution. + set replication-mode dr-auto-sync: Set the replication mode of TiDB to dr-auto-sync. + set replication-mode dr-auto-sync label-key dc: Enable PD to identify the data centers to which nodes belong based on their dc labels, thereby achieving data distribution control. + set replication-mode dr-auto-sync primary dc1: Set DC1 as the primary data center, making it the main write node location for the TiDB cluster during runtime. All data updates are prioritized to be written to the primary data center. + set replication-mode dr-auto-sync dr dc2: Set DC2 as the disaster recovery data center, where TiKV nodes in the learner role reside. + set replication-mode dr-auto-sync primary-replicas 2: Set the number of replicas in the primary data center (DC1) to 2. + set replication-mode dr-auto-sync dr-replicas 1: Set the number of replicas in the disaster recovery data center (DC2) to 1 (learner role not contained). + Verify whether the configuration is effective. ```bash tiup ctl:vtidb_version pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379:2379 config show replication-mode { "replication-mode": "dr-auto-sync", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` + Check the current replication status of the cluster. You can see `sync_recover` mode and `recover_progress` before the status becomes to `sync` mode if your operation is fast. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 2003, "acid_consistent": false, "total_regions": 60, "synced_regions": 37, "recover_progress": 0.98333335 } } ``` + Finally, you will get the `sync` state of TiDB. ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 378, "acid_consistent": true } } ``` + As an initialization status, the cluster is in the synchronous replication mode. PD sends the status information to TiKV, and the two data centers (two commit groups) strictly follow the synchronous replication mode to work. ### Step 10. Create a sysbench database + Log in to TiDB. ```bash mysql -h ${HOST_AZ1DB1_PRIVATE_IP} -P 4000 -u root --connect-expired-password ``` ```output mysql> ``` + Create a database for `sysbench`. ```sql CREATE DATABASE sbtest; EXIT; ``` ```output Query OK, 0 rows affected (0.13 sec) Bye ``` + Set up the `sysbench` configuration for DC1. ```bash cat << EOF > sysbench_config mysql-host=${HOST_AZ1DB1_PRIVATE_IP} mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=6 report-interval=1 EOF cat sysbench_config ``` ```output mysql-host=10.90.1.125 mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=16 report-interval=1 ``` + Prepare `sysbench` data for the next module. ```bash sysbench \ --config-file=sysbench_config oltp_point_select \ --tables=6 \ --table-size=100 prepare ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest3'... Creating table 'sbtest1'... Creating table 'sbtest5'... Creating table 'sbtest6'... Creating table 'sbtest4'... Creating table 'sbtest2'... Inserting 100 records into 'sbtest3' Creating a secondary index on 'sbtest3'... Inserting 100 records into 'sbtest1' Creating a secondary index on 'sbtest1'... Inserting 100 records into 'sbtest4' Creating a secondary index on 'sbtest4'... Inserting 100 records into 'sbtest6' Creating a secondary index on 'sbtest6'... Inserting 100 records into 'sbtest2' Creating a secondary index on 'sbtest2'... Inserting 100 records into 'sbtest5' Creating a secondary index on 'sbtest5'... ``` ### Summary for Module 1 + At this point, we have completed: + The creation of the TiDB cluster. + Enabled the TiDB cluster DR Auto-Sync deployment plan and verified its status. + Prepared the sysbench configuration for simulating application behavior. ## Module 2: Simulate The Secondary DC Failure By Isolating Network Access to DC 2 The objective of this practice is to simulate the complete unavailability of Data Center 2, observe the TiDB cluster's status, and restore it. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-5-en.png) The method to simulate a Data Center 2 failure is to sever the network communication between Data Center 1 and Data Center 2. This will prevent Data Center 2's servers from receiving or responding to external requests, making it appear to Data Center 1 that Data Center 2 has experienced a complete failure. ### Step 1. Connect to the monitor server in data center 1 + Open a local terminal (Terminal 1), connect to monitor server 1. + **macOS** users: ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 10** users: ```cmd ssh -i %userprofile%\.ssh\pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### Step 2. Start the workload + Start `sysbench` to act as the application running. Once started, proceed to the next step. ```bash sysbench \ --config-file=sysbench_config oltp_read_write \ --tables=6 \ --mysql-ignore-errors=all run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 1 Report intermediate results every 1 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 ... ``` ### Step 3. Verify the replication status is in-sync + Open a new terminal (Terminal 2). Log in to Monitor 1 in DC1 to verify the data replication status is in `sync` mode. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + Verify the data replication status. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ### Step 4. Block the network access to DC2 (Availability Zone 2) + Execute the script to simulate a complete Availability Zone failure by blocking subnet `demo-subnet-2` (subnet 10.90.2.0/24 in AZ2). ```bash VPC_ID=`aws ec2 describe-vpcs \ --filters "Name=tag:Name,Values=trainer_name" \ --query "Vpcs[0].VpcId" \ --output text --region us-west-2` SUBNET_ID=`aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \ --query "Subnets[0].SubnetId" \ --output text \ --region us-west-2` CAGE_NACL_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" "Name=tag:Name,Values=cage" \ --query "NetworkAcls[0].NetworkAclId" \ --output text \ --region us-west-2` ASSOC_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \ --query "NetworkAcls[0].Associations" \ --output text \ --region us-west-2 | grep ${SUBNET_ID} | awk -F" " '{print $1}'` aws ec2 replace-network-acl-association \ --association-id ${ASSOC_ID} \ --network-acl-id ${CAGE_NACL_ID} \ --region us-west-2 ``` ```output { "NewAssociationId": "aclassoc-0402a6103febf799f" } ``` ### Step 5. Observe the status of the application and TiDB + Sysbench is blocked to access to TiDB cluster for about 70 seconds which is a little bit longer than the block window setting (default setting is 60 seconds). The reason is that the TiDB cluster needs time to gather network status, process the TiDB cluster status switch, and synchronize the status to each TiKV nodes. ```output ... [ 493s ] thds: 6 tps: 278.00 qps: 5525.08 (r/w/o: 3866.05/1082.02/577.01) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 [ 494s ] thds: 6 tps: 274.99 qps: 5505.84 (r/w/o: 3853.89/1079.97/571.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 [ 495s ] thds: 6 tps: 271.99 qps: 5447.84 (r/w/o: 3814.89/1065.97/566.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00 [ 496s ] thds: 6 tps: 70.00 qps: 1406.04 (r/w/o: 985.03/278.01/143.00) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00 [ 497s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 498s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... [ 544s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 545s ] thds: 6 tps: 0.00 qps: 29.00 (r/w/o: 26.00/0.00/3.00) lat (ms,95%): 0.00 err/s: 3.00 reconn/s: 0.00 [ 546s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... [ 566s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 567s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 568s ] thds: 6 tps: 98.00 qps: 1968.01 (r/w/o: 1376.00/388.00/204.00) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00 ... [ 569s ] thds: 6 tps: 244.00 qps: 4883.91 (r/w/o: 3418.93/950.98/513.99) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00 [ 570s ] thds: 6 tps: 237.00 qps: 4734.95 (r/w/o: 3313.96/924.99/495.99) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 [ 571s ] thds: 6 tps: 264.02 qps: 5279.31 (r/w/o: 3693.21/1039.06/547.03) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00 ... ``` + Observe the nodes status in TiDB cluster. The nodes which are located in dc2 are all down. ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://18.236.254.103:2379/dashboard Grafana URL: http://10.90.1.90:3000,http://10.90.2.17:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.90:3000 grafana 10.90.1.90 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.17:3000 grafana 10.90.2.17 3000 linux/x86_64 Down - /tidb-deploy/grafana-3000 10.90.1.176:2379 pd 10.90.1.176 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.73:2379 pd 10.90.1.73 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.198:2379 pd 10.90.2.198 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.90:10090 prometheus 10.90.1.90 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.17:10090 prometheus 10.90.2.17 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.106:4000 tidb 10.90.1.106 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.61:4000 tidb 10.90.2.61 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.1.245:20160 tikv 10.90.1.245 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.45:20160 tikv 10.90.1.45 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.125:20160 tikv 10.90.2.125 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.96:20160 tikv 10.90.2.96 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` + Observe the TiDB sync mode changes, from `sync` to `async_wait`, and `async` subsequential. During the period of `async_wait`, pd sends mode switch instructions to all tikv nodes. Once this process is completed, the sync mode transitions to `async`. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "async_wait", "state_id": 1001, "acid_consistent": true } } ``` ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "async", "state_id": 483, "acid_consistent": true } } ``` ### Step 6. Recover the network access of data center 2 + Recover the network. The previously lost nodes will come back to join the TiDB cluster again. ```bash VPC_ID=`aws ec2 describe-vpcs \ --filters "Name=tag:Name,Values=trainer_name" \ --query "Vpcs[0].VpcId" \ --output text \ --region us-west-2` SUBNET_ID=`aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-2" \ --query "Subnets[0].SubnetId" \ --output text \ --region us-west-2` DEFAULT_NACL_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \ --query "NetworkAcls[0].NetworkAclId" \ --output text \ --region us-west-2` ASSOC_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" \ --query "NetworkAcls[0].Associations" \ --output text --region us-west-2 | grep ${SUBNET_ID} | awk -F" " '{print $1}'` aws ec2 replace-network-acl-association \ --association-id ${ASSOC_ID} \ --network-acl-id ${DEFAULT_NACL_ID} \ --region us-west-2 ``` ```output { "NewAssociationId": "aclassoc-0df1e05cfc70356a8" } ``` + Observe the status of the application and TiDB cluster. `sysbench` is running stably. ```output [ 1039s ] thds: 6 tps: 268.99 qps: 5409.79 (r/w/o: 3787.85/1067.96/553.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 [ 1040s ] thds: 6 tps: 273.01 qps: 5455.15 (r/w/o: 3814.11/1083.03/558.02) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00 [ 1041s ] thds: 6 tps: 265.00 qps: 5282.95 (r/w/o: 3702.96/1043.99/535.99) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00 ... [ 1105s ] thds: 6 tps: 260.00 qps: 5244.02 (r/w/o: 3669.02/1043.00/532.00) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00 [ 1106s ] thds: 6 tps: 267.99 qps: 5339.86 (r/w/o: 3740.90/1052.97/545.99) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00 ``` + Check the TiDB cluster status. All nodes in DC2 should be back online. If you're following the lab steps swiftly, you might notice the status of some TiKV nodes changing from `Disconnected` to `Up` gradually. ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.92.147.161:2379/dashboard Grafana URL: http://10.90.1.212:3000,http://10.90.2.99:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.212:3000 grafana 10.90.1.212 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.99:3000 grafana 10.90.2.99 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.156:2379 pd 10.90.1.156 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.232:2379 pd 10.90.1.232 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.104:2379 pd 10.90.2.104 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.212:10090 prometheus 10.90.1.212 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.99:10090 prometheus 10.90.2.99 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.23:4000 tidb 10.90.1.23 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.122:4000 tidb 10.90.2.122 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.178:20160 tikv 10.90.1.178 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.6:20160 tikv 10.90.1.6 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.250:20160 tikv 10.90.2.250 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.90:20160 tikv 10.90.2.90 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.250:20160 tikv 10.90.2.250 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 ... ``` + Check the current replication status of the TiDB cluster. Within only several seconds, the status is switched to `sync` from `sync_recover`. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-6-en.png) + `sync_recover` mode. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 1009, "acid_consistent": false, "total_regions": 78, "recover_progress": 0.08974359 } } ``` + `sync` mode. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1010, "acid_consistent": true } } ``` ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-61-en.png) + At this point, TiDB restores data synchronization between the two data centers to the SYNC state, achieving the goal of RPO=0 resilience against data center failures. ### Step 7. Stop the sysbench workload + Switch to Terminal 1 and 2, terminate the sysbench thread and exit the terminals. ### Summary for Module 2 + Through the above steps, we simulated the entire process of TiDB handling a complete DC2 failure and its recovery to availability. + When DC2 becomes unavailable, TiDB enters a blocking window to confirm the failure. During this blocking window, application read and write operations to the database are affected. + After exceeding the blocking window, TiDB operates using only the nodes in DC1 and transitions to the `async` state. At this point, application read and write operations return to normal. + When DC2 recovers from the failure, TiDB enters the `sync_recover` state, synchronizing data from the nodes in DC1 to the nodes in DC2. This process does not affect the availability of the database for application access. + Once the data synchronization is complete, the TiDB cluster transitions back to the `sync` mode. ## Module 3: Simulate The Primary DC Failure By Terminating All Nodes in DC 1 The objective of this exercise is to simulate the complete unavailability of the primary data center and how to restore its availability. When DC1 experiences a complete failure, DC2 still retains the latest data in the follower TiKV instance. Together with a backup learner TiKV instance, a two-replica cluster can be quickly restored to resume service. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-7-en.png) We need to start a new TiDB cluster through the PD in the disaster recovery data center (retaining the data, but with a new cluster ID). Additionally, we need to scale out a PD node (a pre-reserved standby PD node) and convert the learner TiKV node in the disaster recovery data center into a voting TiKV to form a functional Raft group with another TiKV node. ### Step 1. Connect to the monitor server in data center 1 + Open a local terminal (Terminal 1), and connect to monitor server 1. + **macOS** users: ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` + **Windows 10** users: ```cmd ssh -i %userprofile%\.ssh\pe-class-key-oregon.pem ec2-user@az1monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### Step 2. Start the workload + Start `sysbench` in both DC1 and DC2 to act as the application running, the task will be runing for 60 minutes. + Start the `sysbench` workload in DC1. ```bash sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 1 Report intermediate results every 1 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 1s ] thds: 1 tps: 61.90 qps: 1240.90 (r/w/o: 868.53/247.58/124.79) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 [ 2s ] thds: 1 tps: 58.02 qps: 1176.46 (r/w/o: 824.32/236.09/116.04) lat (ms,95%): 18.28 err/s: 0.00 reconn/s: 0.00 ... ``` + Open a new local terminal (terminal 2), connect to monitor server 2. Start the `sysbench` workload in DC2. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az2monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + Set up the `sysbench` configuration for DC2. ```bash cat << EOF > sysbench_config mysql-host=${HOST_AZ2DB1_PRIVATE_IP} mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=6 report-interval=1 EOF cat sysbench_config ``` ```output mysql-host=10.90.2.125 mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=16 report-interval=1 ``` + Start the `sysbench` workload in DC2. ```bash sysbench --config-file=sysbench_config oltp_read_write --tables=6 --mysql-ignore-errors=all run ``` ```output Threads started! [ 1s ] thds: 6 tps: 316.47 qps: 6401.24 (r/w/o: 4486.46/1158.05/756.73) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00 [ 2s ] thds: 6 tps: 330.09 qps: 6611.89 (r/w/o: 4631.33/1171.34/809.23) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00 ... ``` ### Step 3. Verify the replication status is in-sync Open a new terminal (Terminal 3). Log in to monitor server 2 in DC2 to verify the data replication status is in the `sync` mode. + Log in to monitor2. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@az2monitor1_address ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + Verify the data replication status. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 1006, "acid_consistent": true } } ``` ### Step 4. Simulate a complete failure of the primary data center + Simulate a complete failure of the primary data center by terminating all TiDB processes on the nodes of the TiDB cluster. Note: We keep the application alive to observe the accessibility of the TiDB cluster. + Terminate all the processes and remove data dir of each tidb, tikv and pd nodes that located in DC1. If prompted, answer `yes`. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1DB1_PRIVATE_IP} "sudo systemctl stop tidb-4000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1KV1_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1KV2_PRIVATE_IP} "sudo systemctl stop tikv-20160.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1PD1_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1PD2_PRIVATE_IP} "sudo systemctl stop pd-2379.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` + Terminate the processes and remove data dir of monitoring. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ1MONITOR1_PRIVATE_IP} "sudo systemctl stop prometheus-10090.service; sudo systemctl stop grafana-3000.service; sudo systemctl stop node_exporter-9100.service; sudo systemctl stop blackbox_exporter-9115.service; sudo rm -rf /tidb-deploy/; sudo rm -rf /tidb-data;" ``` ### Step 5. Observe the status of the application and TiDB + Observe the nodes status in TiDB cluster. The status of all the pd servers is `Down`, all the tikv is `N/A` and one of the tidb servers is `Down`. Only the tidb server located in DC2 is alive. ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Grafana URL: http://10.90.1.182:3000,http://10.90.2.252:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.182:3000 grafana 10.90.1.182 3000 linux/x86_64 Down - /tidb-deploy/grafana-3000 10.90.2.252:3000 grafana 10.90.2.252 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.213:2379 pd 10.90.1.213 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.90:2379 pd 10.90.1.90 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.65:2379 pd 10.90.2.65 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.182:10090 prometheus 10.90.1.182 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.252:10090 prometheus 10.90.2.252 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.196:4000 tidb 10.90.1.196 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.2.226:4000 tidb 10.90.2.226 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.228:20160 tikv 10.90.1.228 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.91:20160 tikv 10.90.1.91 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.202:20160 tikv 10.90.2.202 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.206:20160 tikv 10.90.2.206 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` + Observe the TiDB sync mode. Because we can't access the pd leader which is located in DC1, we need to access the follower pd which is located in DC2. The operation is timeout because the request can't be redirected to the leader. ```bash curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output [PD:apiutil:ErrRedirectNoLeader]redirect finds no leader ``` + Switch to Terminal 1. We can see that the workload goes to 0 QPS because the TiDB cluster is unavailability. ```output ... [ 57s ] thds: 6 tps: 193.99 qps: 3834.86 (r/w/o: 2679.90/665.98/488.98) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00 [ 58s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 59s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... ``` + Switch to Terminal 2. We can find the workload goes to 0 QPS because of TiDB cluster unavailability. ```output ... [ 34s ] thds: 6 tps: 116.00 qps: 2313.99 (r/w/o: 1624.99/385.00/304.00) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00 [ 35s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 36s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ... ``` ### Step 6. Reshape the TiDB cluster with surviving nodes only + Switch to terminal 3, overwrite the meta file of `tiup` by the prepared meta file which only contains the nodes that located in data center 2 (AZ2). + Backup the meta file before overwriting it. ```bash cp /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml.backup ``` + Check the yaml file `majority-DC2.yaml`. ```bash cat majority-DC2.yaml ``` ```output user: ec2-user tidb_version: vtidb_version topology: global: user: "ec2-user" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" arch: "amd64" monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 deploy_dir: "/tidb-deploy/monitored-9100" data_dir: "/tidb-data/monitor_9115" log_dir: "/tidb-deploy/monitor-9100/log" server_configs: pd: replication.location-labels: ["dc","logic","rack","host"] tidb_servers: - host: 10.90.2.210 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" tikv_servers: - host: 10.90.2.111 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic3",rack: "r1",host: "10.90.2.111" } - host: 10.90.2.91 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic4",rack: "r1",host: "10.90.2.91" } pd_servers: - host: 10.90.2.83 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://54.212.25.205:2379" monitoring_servers: - host: 10.90.2.107 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" grafana_servers: - host: 10.90.2.107 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" ``` + Replace the meta file by using the provisioned file. ```bash cp -rf majority-DC2.yaml /home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/meta.yaml ``` ### Step 7. Stop the TiDB cluster + Check TiDB cluster status. ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.14:3000 grafana 10.90.2.14 3000 / Up - /tidb-deploy/grafana-3000 10.90.2.53:2379 pd 10.90.2.53 2379/2380 / Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.14:10090 prometheus 10.90.2.14 10090 / Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.97:4000 tidb 10.90.2.97 4000/10080 / Up - /tidb-deploy/tidb-4000 10.90.2.170:20160 tikv 10.90.2.170 20160/20180 / N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.78:20160 tikv 10.90.2.78 20160/20180 / N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 6 ``` + Stop the nodes in TiDB cluster. ```bash tiup cluster stop tidb-demo -y ``` ```output ... Stopping component grafana Stopping instance 10.90.2.14 Stop grafana 10.90.2.14:3000 success Stopping component prometheus Stopping instance 10.90.2.14 Stop prometheus 10.90.2.14:10090 success Stopping component tidb Stopping instance 10.90.2.97 Stop tidb 10.90.2.97:4000 success Stopping component tikv Stopping instance 10.90.2.170 Stopping instance 10.90.2.78 Stop tikv 10.90.2.170:20160 success Stop tikv 10.90.2.78:20160 success Stopping component pd Stopping instance 10.90.2.53 Stop pd 10.90.2.53:2379 success Stopping component node_exporter Stopping instance 10.90.2.170 Stopping instance 10.90.2.14 Stopping instance 10.90.2.53 Stopping instance 10.90.2.97 Stopping instance 10.90.2.78 Stop 10.90.2.170 success Stop 10.90.2.78 success Stop 10.90.2.53 success Stop 10.90.2.97 success Stop 10.90.2.14 success Stopping component blackbox_exporter Stopping instance 10.90.2.170 Stopping instance 10.90.2.14 Stopping instance 10.90.2.97 Stopping instance 10.90.2.78 Stopping instance 10.90.2.53 Stop 10.90.2.78 success Stop 10.90.2.170 success Stop 10.90.2.97 success Stop 10.90.2.53 success Stop 10.90.2.14 success Stopped cluster `tidb-demo` successfully ``` + Check TiDB nodes status, all pd, tidb, tikv and prometheus are down. ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Down - /tidb-deploy/grafana-3000 10.90.2.125:2379 pd 10.90.2.125 2379/2380 linux/x86_64 Down /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.173:4000 tidb 10.90.2.173 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.2.129:20160 tikv 10.90.2.129 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.216:20160 tikv 10.90.2.216 20160/20180 linux/x86_64 N/A /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 6 ``` ### Step 8. Recover the Placement Driver + Install the pd-recover tool. ```bash tiup install pd-recover:vtidb_version ``` ```output download https://tiup-mirrors.pingcap.com/pd-recover-vtidb_version-darwin-arm64.tar.gz 13.88 MiB / 13.88 MiB 100.00% 6.99 MiB/s ``` + SSH to placement driver server which is located in DC2 to force start a new placement driver cluster. + Log in to pd. If prompted, answer `yes`. ```bash ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@${HOST_AZ2PD1_PRIVATE_IP} ``` ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` + Check the status of pd which is the synchronous mode of TiDB cluster before DC1 is lost, and it presents the current data consistency of TiDB. ```bash cat /tidb-data/pd_data/DR_STATE ``` ```output {"state":"sync","state_id":1947} ``` + Modify the pd's initial script by adding the `--force-new-cluster` parameter. Note: this parameter is used to start a new etcd cluster of pd. ```bash sed -i '/pd-server/ s#pd-server#pd-server --force-new-cluster#' /tidb-deploy/pd-2379/scripts/run_pd.sh ``` + Exit to monitor OS. ```bash exit ``` ```output logout Connection to 10.90.2.26 closed. ``` + Start the cluster by starting the pd node. ```bash tiup cluster start tidb-demo -N ${HOST_AZ2PD1_PRIVATE_IP}:2379 -y ``` ```output ... Starting cluster tidb-demo... + [ Serial ] - SSHKeySet: privateKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa, publicKey=/home/ec2-user/.tiup/storage/cluster/clusters/tidb-demo/ssh/id_rsa.pub + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.26 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.38 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.254 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.182 + [Parallel] - UserSSH: user=ec2-user, host=10.90.2.49 + [ Serial ] - StartCluster Starting component pd Starting instance 10.90.2.26:2379 Start instance 10.90.2.26:2379 success Starting component node_exporter Starting instance 10.90.2.26 Start 10.90.2.26 success Starting component blackbox_exporter Starting instance 10.90.2.26 Start 10.90.2.26 success + [ Serial ] - UpdateTopology: cluster=tidb-demo Started cluster `tidb-demo` successfully ``` + Verify that pd is up and running. ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.94.179.113:2379/dashboard Grafana URL: http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.107:3000 grafana 10.90.2.107 3000 / Down - /tidb-deploy/grafana-3000 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Down - /tidb-deploy/tidb-4000 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 6 ``` + Repair pd's metadata. After this step, the `alloc_id` in the storage automatically increases by a safe value of `100000000`. ```bash tiup pd-recover:vtidb_version --from-old-member --endpoints=http://${HOST_AZ2PD1_PRIVATE_IP}:2379 ``` ```output download https://tiup-mirrors.pingcap.com/pd-recover-vtidb_version-linux-amd64.tar.gz 15.87 MiB / 15.87 MiB 100.00% 3.78 MiB/s Checking updates for component pd-recover... Starting component pd-recover: /home/ec2-user/.tiup/components/pd-recover/v8.5.0/pd-recover --from-old-member --endpoints=http://10.90.2.31:2379 recover success! please restart the PD cluster ``` ### Step 9. Scale out one new Placement Driver server + Check the yaml file. ```bash cat scale-out-pd.yaml ``` ```output pd_servers: - host: 10.90.2.102 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://35.161.129.28:2379" ``` + Scale out PD node. ```bash tiup cluster scale-out tidb-demo scale-out-pd.yaml --user ec2-user -i ~/.ssh/pe-class-key-oregon.pem -y ``` ```output ... Starting component pd Starting instance 10.90.2.102:2379 Start instance 10.90.2.102:2379 success Starting component node_exporter Starting instance 10.90.2.102 Start 10.90.2.102 success Starting component blackbox_exporter Starting instance 10.90.2.102 Start 10.90.2.102 success + Refresh components conifgs - Generate config pd -> 10.90.2.26:2379 ... Done - Generate config pd -> 10.90.2.102:2379 ... Done - Generate config tikv -> 10.90.2.254:20160 ... Done - Generate config tikv -> 10.90.2.182:20160 ... Done - Generate config tidb -> 10.90.2.49:4000 ... Done - Generate config prometheus -> 10.90.2.38:10090 ... Done - Generate config grafana -> 10.90.2.38:3000 ... Done + Reload prometheus and grafana - Reload prometheus -> 10.90.2.38:10090 ... Done - Reload grafana -> 10.90.2.38:3000 ... Done + [ Serial ] - UpdateTopology: cluster=tidb-demo Scaled cluster `tidb-demo` out successfully ``` + Verify that 2 PD nodes in DC2 are all up and running. ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.94.179.113:2379/dashboard Grafana URL: http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.107:3000 grafana 10.90.2.107 3000 / Up - /tidb-deploy/grafana-3000 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.96:2379 pd 10.90.2.96 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Down - /tidb-deploy/tidb-4000 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 7 ``` ### Step 10. Setup the cluster as `majority` mode + Verify cluster still in `dr-auto-sync` replication mode. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. download https://tiup-mirrors.pingcap.com/ctl-vtidb_version-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 97.75 MiB/s Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.83:2379 config show replication-mode { "replication-mode": "dr-auto-sync", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` + Check the `majority` rules file. ```bash cat majority-rules.json ``` ```output [ { "group_id": "pd", "group_index": 0, "group_override": false, "rules": [ { "group_id": "pd", "id": "voters", "start_key": "", "end_key": "", "role": "voter", "count": 3, "location_labels": ["dc","logic","rack", "host"] } ] } ] ``` + Load the `majority` placement rules configuration. ```bash tiup ctl:vtidb_version pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/majority-rules.json" ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/majority-rules.json "Update rules and groups successfully." ``` + Check placement rules is loaded. ```bash tiup ctl:vtidb_version pd config placement-rules show -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules show -u 10.90.2.125:2379 [ { "group_id": "pd", "id": "voters", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 3, "location_labels": [ "dc", "logic", "rack", "host" ], "create_timestamp": 1734071450 } ] ``` + Enable the `majority` configuration. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode majority tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set max-replicas 3 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 config set replication-mode majority Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 config set max-replicas 3 Success! ``` + Verify cluster `majority` replication mode is effective. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 config show replication-mode { "replication-mode": "majority", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` ### Step 11. Remove the unavailable TiKV nodes + Start TiKV nodes. ```bash tiup cluster start tidb-demo -R tikv -y ``` ```output ... Starting component tikv Starting instance 10.90.2.129:20160 Starting instance 10.90.2.216:20160 Start instance 10.90.2.216:20160 success Start instance 10.90.2.129:20160 success Starting component node_exporter Starting instance 10.90.2.129 Starting instance 10.90.2.216 Start 10.90.2.216 success Start 10.90.2.129 success Starting component blackbox_exporter Starting instance 10.90.2.129 Starting instance 10.90.2.216 Start 10.90.2.216 success Start 10.90.2.129 success + [ Serial ] - UpdateTopology: cluster=tidb-demo Started cluster `tidb-demo` successfully ... ``` + Wait for the 2 TiKV nodes are up. ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.125:2379 pd 10.90.2.125 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.135:2379 pd 10.90.2.135 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.173:4000 tidb 10.90.2.173 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.2.129:20160 tikv 10.90.2.129 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.216:20160 tikv 10.90.2.216 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 7 ``` + Get the id of the TiKV stores that are located in DC1 (AZ1). We can find 2 stores in a state of `Down` or `Disconnected`. Record the id for the next step. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 store ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 store { "count": 4, "stores": [ { "store": { "id": 16, ... "state_name": "Up" }, "status": { ... } }, { "store": { "id": 1, ... "state_name": "Up" }, "status": { ... } }, { "store": { "id": 2, ... "state_name": "Down" }, "status": { ... } }, { "store": { "id": 15, ... "state_name": "Down" }, "status": { ... } } ] } ``` + If the `state_name` is `Down`, remove unavailable TiKV nodes. Replace the value of n1 & n2 in the command by using the store id you recorded in the previous step. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores n1,n2 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 unsafe remove-failed-stores 2,15 Success! Request has been accepted." ``` + Show the status of the removal operation. After a while, you can confirm the process is completed by identifying the following description: `"info": "Unsafe recovery Finished"` at the end of the command result. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 unsafe remove-failed-stores show ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.125:2379 unsafe remove-failed-stores show [ { "info": "Unsafe recovery enters collect report stage", "time": "2024-12-13 07:01:51.714", "details": [ "Failed stores 2, 15" ] }, { "info": "Unsafe recovery enters force leader stage", "time": "2024-12-13 07:01:54.000", "actions": { "store 1": [ "force leader on regions: 7, 9, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 101, 104, 107, 110, 113, 116, 119, 122, 125, 128, 131, 134, 137, 140, 143, 146, 149, 152, 155, 158, 209, 213, 217, 221, 225, 229, 233, 237, 241, 245, 249, 253, 1978, 2013, 2018, 1993, 2043, 2048, 1983, 2023, 2028, 1988, 2033, 2038, 1998, 2063, 2068, 2008, 2053, 2058, 35, 31, 27, 23, 19, 17, 11, 13, 3" ], "store 16": [ "force leader on regions: " ] } }, { "info": "Unsafe recovery enters demote Failed voter stage", "time": "2024-12-13 07:02:04.586", "actions": { "store 1": [ "region 7 demotes peers { id:75 store_id:2 }, { id:170 store_id:15 }", ... "region 3 demotes peers { id:77 store_id:2 }, { id:208 store_id:15 }" ] } }, { "info": "Unsafe recovery Finished", "time": "2024-12-13 07:02:14.002", "details": [ "affected table ids: 74, 94, 110, 24, 58, 78, 92, 107, 281474976710649, 56, 82, 84, 106, 108, 109, 281474976710652, 281474976710654, 52, 111, 50, 66, 281474976710653, 48, 98, 54, 38, 42, 46, 62, 72, 76, 6, 281474976710651, 86, 28, 30, 40, 68, 96, 20, 60, 4, 18, 34, 80, 12, 10, 32, 70, 8, 64, 22, 36, 88, 90, 102, 14, 26, 44, 281474976710650, 16", "no newly created empty regions" ] } ] ``` ### Step 12. Start the recovered TiDB cluster + Start the tidb and prometheus instances. ```bash tiup cluster start tidb-demo -R tidb,prometheus -y ``` ```output Starting component tidb Starting instance 10.90.2.49:4000 Start instance 10.90.2.49:4000 success Starting component prometheus Starting instance 10.90.2.38:10090 Start instance 10.90.2.38:10090 success Starting component node_exporter Starting instance 10.90.2.38 Starting instance 10.90.2.49 Start 10.90.2.49 success Start 10.90.2.38 success Starting component blackbox_exporter Starting instance 10.90.2.38 Starting instance 10.90.2.49 Start 10.90.2.49 success Start 10.90.2.38 success + [ Serial ] - UpdateTopology: cluster=tidb-demo Started cluster `tidb-demo` successfully ``` + Check the status of the TiDB cluster. The state of all the nodes in DC2 is up. ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: vtidb_version Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.94.179.113:2379/dashboard Grafana URL: http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.125:2379 pd 10.90.2.125 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.135:2379 pd 10.90.2.135 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.173:4000 tidb 10.90.2.173 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.129:20160 tikv 10.90.2.129 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.216:20160 tikv 10.90.2.216 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 7 ``` + Verify the replication mode of the TiDB cluster. ```bash curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "majority", "dr-auto-sync": { "label_key": "", "state": "", "acid_consistent": false } } ``` ### Step 13. Verify that the sysbench workload is recovered + Switch to Terminal 1, the QPS of the sysbench workload is still 0. Because tidb server located in DC1 is unavailable. ```output ... [ 1697s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 1698s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 1699s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 ``` + Switch to Terminal 2, the QPS of the sysbench workload back to normal. ```output ... [ 1618s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 1619s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 6.00 [ 1620s ] thds: 6 tps: 321.06 qps: 6505.17 (r/w/o: 4564.82/1103.20/837.15) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00 [ 1621s ] thds: 6 tps: 313.95 qps: 6261.93 (r/w/o: 4378.25/1098.81/784.87) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00 ``` ### Step 14. Enable DR Auto-Sync again + After DC1 recovers, we need to recover the TiDB cluster back to DR Auto-Sync deployment. + Rejoin the lost nodes back into the TiDB cluster. Check the yaml file. ```bash cat scale-out-DC1.yaml ``` ```output tidb_servers: - host: 10.90.1.58 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" tikv_servers: - host: 10.90.1.219 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic1",rack: "r1",host: "10.90.1.219" } - host: 10.90.1.71 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic2",rack: "r1",host: "10.90.1.71" } pd_servers: - host: 10.90.1.95 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://34.214.79.226:2379" - host: 10.90.1.55 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://54.202.186.157:2379" monitoring_servers: - host: 10.90.1.221 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" grafana_servers: - host: 10.90.1.221 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" ``` + Scale out nodes in DC1. ```bash tiup cluster scale-out tidb-demo scale-out-DC1.yaml --user ec2-user -i ~/.ssh/pe-class-key-oregon.pem -y ``` ```output ... Starting component pd Starting instance 10.90.1.95:2379 Start instance 10.90.1.95:2379 success Starting instance 10.90.1.55:2379 Start instance 10.90.1.55:2379 success Starting component tikv Starting instance 10.90.1.71:20160 Starting instance 10.90.1.219:20160 Start instance 10.90.1.71:20160 success Start instance 10.90.1.219:20160 success Starting component tidb Starting instance 10.90.1.58:4000 Start instance 10.90.1.58:4000 success Starting component prometheus Starting instance 10.90.1.221:10090 Start instance 10.90.1.221:10090 success Starting component grafana Starting instance 10.90.1.221:3000 Start instance 10.90.1.221:3000 success Starting component node_exporter Starting instance 10.90.1.55 Starting instance 10.90.1.58 Starting instance 10.90.1.221 Starting instance 10.90.1.95 Starting instance 10.90.1.219 Starting instance 10.90.1.71 Start 10.90.1.71 success Start 10.90.1.95 success Start 10.90.1.219 success Start 10.90.1.55 success Start 10.90.1.221 success Start 10.90.1.58 success Starting component blackbox_exporter Starting instance 10.90.1.55 Starting instance 10.90.1.58 Starting instance 10.90.1.221 Starting instance 10.90.1.95 Starting instance 10.90.1.219 Starting instance 10.90.1.71 Start 10.90.1.219 success Start 10.90.1.95 success Start 10.90.1.71 success Start 10.90.1.55 success Start 10.90.1.221 success Start 10.90.1.58 success + Refresh components conifgs - Generate config pd -> 10.90.2.83:2379 ... Done - Generate config pd -> 10.90.2.96:2379 ... Done - Generate config pd -> 10.90.1.95:2379 ... Done - Generate config pd -> 10.90.1.55:2379 ... Done - Generate config tikv -> 10.90.2.111:20160 ... Done - Generate config tikv -> 10.90.2.91:20160 ... Done - Generate config tikv -> 10.90.1.219:20160 ... Done - Generate config tikv -> 10.90.1.71:20160 ... Done - Generate config tidb -> 10.90.2.210:4000 ... Done - Generate config tidb -> 10.90.1.58:4000 ... Done - Generate config prometheus -> 10.90.2.107:10090 ... Done - Generate config prometheus -> 10.90.1.221:10090 ... Done - Generate config grafana -> 10.90.2.107:3000 ... Done - Generate config grafana -> 10.90.1.221:3000 ... Done + Reload prometheus and grafana - Reload prometheus -> 10.90.2.107:10090 ... Done - Reload grafana -> 10.90.2.107:3000 ... Done + [ Serial ] - UpdateTopology: cluster=tidb-demo Scaled cluster `tidb-demo` out successfully ``` + Check the TiDB cluster status, all the nodes are recovered. ```bash tiup cluster display tidb-demo ``` ```output ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.221:3000 grafana 10.90.1.221 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.107:3000 grafana 10.90.2.107 3000 / Up - /tidb-deploy/grafana-3000 10.90.1.55:2379 pd 10.90.1.55 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.95:2379 pd 10.90.1.95 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.96:2379 pd 10.90.2.96 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.221:10090 prometheus 10.90.1.221 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.58:4000 tidb 10.90.1.58 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Up - /tidb-deploy/tidb-4000 10.90.1.219:20160 tikv 10.90.1.219 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.71:20160 tikv 10.90.1.71 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 14 ``` + Load the DR Auto-Sync placement rules configuration. ```bash tiup ctl:vtidb_version pd config placement-rules rule-bundle -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 save --in="/home/ec2-user/dr-anto-sync-rules.json" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. download https://tiup-mirrors.pingcap.com/ctl-vtidb_version-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 103.24 MiB/s Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/dr-anto-sync-rules.json "Update rules and groups successfully." ``` + Install `jq` tool. ```bash sudo yum install epel-release sudo yum install jq -y ``` ```output ... Installed: jq.x86_64 0:1.5-1.amzn2.0. 2 Dependency Installed: oniguruma.x86_64 0:5.9.6-1.amzn2.0. 7 Complete! ``` + Check for TiKV regions without learner peers. Verify that there are no regions without learner peers and ensure data synchronization between voter and learner roles. This step ensures that the cluster's data is consistently replicated across all roles, which is essential for maintaining high availability and failover readiness. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 region --jq='.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]}' ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://34.215.73.186:2379 region --jq=.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]} ``` + Promote the role conversion of Learner Peers (optional operation). If there are Regions that have not been converted to Learner Peers, use the following command to promote the Region to a Learner Region. The first number is the Region ID, and the second number is the Store ID. Note: If no result returned in the previous step, please `Skip this step`. ```bash operator add remove-peer 6066 6 ``` + Enable the DR Auto-Sync configuration. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync label-key dc tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary dc1 tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr dc2 tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary-replicas 2 tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr-replicas 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync label-key dc Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary dc1 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr dc2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary-replicas 2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr-replicas 1 Success! The replication mode config is updated." ``` + Verify that the configuration is effective. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ2PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u http://10.90.2.162:2379:2379 config show replication-mode { "replication-mode": "dr-auto-sync", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` ### Step 15. Verify that the replication status is back to `sync` + Check the current replication status of the cluster. You might see `sync_recover` mode or `recover_progress` before the status becomes to `sync` mode temporarily. ```bash curl ${HOST_AZ2PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 2003, "acid_consistent": false, "total_regions": 60, "synced_regions": 37, "recover_progress": 0.98333335 } } ``` + Finally, you will get the `sync` state. ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 378, "acid_consistent": true } } ``` + Switch to Terminal 1, the QPS of the sysbench workload is recovered. ```output ... [ 2309s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 2310s ] thds: 6 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00 [ 2311s ] thds: 6 tps: 105.01 qps: 2170.15 (r/w/o: 1530.11/416.03/224.02) lat (ms,95%): 100000.00 err/s: 0.00 reconn/s: 6.00 [ 2312s ] thds: 6 tps: 193.97 qps: 3869.44 (r/w/o: 2705.61/763.89/399.94) lat (ms,95%): 34.33 err/s: 0.00 reconn/s: 0.00 ``` ### Step 16. Transfer the leaders of PD and TiKV to data center 1 + Scale in the 2nd placement driver node in data center 2. ```bash tiup cluster scale-in tidb-demo -N "${HOST_AZ2PD2_PRIVATE_IP}:2379" -y ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` + You can execute this command to try to transfer pd leader to DC1 whatever the previous step's result is. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 member leader transfer "pd-${HOST_AZ1PD1_PRIVATE_IP}-2379" ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` + Check the TiDB cluster status. Ensure the leader of pd is located in DC1 and the pd nodes count goes to 3. ```bash tiup cluster display tidb-demo ``` ```output ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.221:3000 grafana 10.90.1.221 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.107:3000 grafana 10.90.2.107 3000 / Up - /tidb-deploy/grafana-3000 10.90.1.55:2379 pd 10.90.1.55 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.95:2379 pd 10.90.1.95 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.221:10090 prometheus 10.90.1.221 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.58:4000 tidb 10.90.1.58 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Up - /tidb-deploy/tidb-4000 10.90.1.219:20160 tikv 10.90.1.219 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.71:20160 tikv 10.90.1.71 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` + By examining the region distribution of each role, we can confirm that the DR Auto-Sync configuration has successfully taken effect on TiKV nodes. ```bash tiup ctl:vtidb_version pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 store ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.2.83:2379 store { "count": 4, "stores": [ { "store": { "id": 1, "address": "10.90.2.111:20160", ... }, "status": { "leader_count": 0, "region_count": 79, ... } }, { "store": { "id": 14, "address": "10.90.2.91:20160", ... }, "status": { "leader_count": 0, "region_count": 79, "learner_count": 79, ... } }, { "store": { "id": 2001, "address": "10.90.1.219:20160", ... }, "status": { "leader_count": 42, "region_count": 79, ... } }, { "store": { "id": 2002, "address": "10.90.1.71:20160", ... }, "status": { "leader_count": 37, "region_count": 79, ... } } ] } ``` + From the information above, we can identify the role of each TiKV node from the region distribution (Your store id might differ from the example shown below): + Store 1 is `follower`: "leader_count": 0, "region_count": 79 + Store 14 is `learner`: "leader_count": 0, "region_count": 79, "learner_count": 79 + Store 2001 is `voter`: "leader_count": 42, "region_count": 79 + Store 2002 is `voter`: "leader_count": 37, "region_count": 79 ### Summary for Module 3 + Through this module, we observed that even if the primary data center experiences a complete failure, we can still successfully restore the TiDB cluster and ensure RPO=0. Roughly speaking, the recovery operations include the following steps: + Recover the only remaining PD in DC2 and use it to create a new cluster. + Scale out a new PD in DC2 to form a new Raft group. + Modify the cluster mode to majority mode. + Remove the unavailable TiKV nodes located in DC1. + Restart all cluster nodes and run with two surviving TiKV nodes. Observe the recovery of application read and write operations. + Fix the failure. + Initialize the nodes in DC1 and reintegrate them into the TiDB cluster. + Modify the cluster configuration to revert back to DR Auto-Sync mode. ## Conclusion Through the three practices above, we have validated the high availability of the TiDB DR Auto-Sync deployment solution, ensuring no data loss even in the event of a data center-level failure. Especially in FSI, many clients operate in two data center environments for high availability. TiDB introduces the DR Auto-Sync (Data Replication Auto Synchronous) mechanism to address the two data center deployment challenges, ensuring data consistency while enhancing system availability. ```hbsptform { "ui": { "titleLevel": 3 }, "portalId": "4466002", "formId": "89fa8368-eba1-4b84-b91e-eb9eb6613ddf", "region": "na1", "sfdcCampaignId": "701RC00000VqCODYA3" } ```