## Module 1: Set Up the TiDB Cluster With DR Auto-Sync Configuration In this practice, we will set up a cluster with DR Auto-Sync configuration enabled. The following nodes are distributed across two data centers: + DC1 as primary data center, located in aws AZ1: `pd x2, tidb x1, tikv x2, tiup(monitor) x1` + DC2 as disaster recovery data center, located in aws AZ2: `pd x1(with extra 1 node standby), tidb x1, tikv x2, tiup(monitor) x1` + Sysbench acts as the application and is deployed in each data center (Corresponding to the `client` in the diagram below). Note: The test simplifies the load balancing service, with sysbench in DC1 directly connecting to the TiDB server in DC1, and sysbench in DC2 doing the same. ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-4-en.png) ### Step 1. Connect to a monitor node in the first data center + Open [Terminal 1](ssh://az1monitor1-ssh#1) to connect to remote bastion Linux in DC1. ### Step 2. Review the multi availability zone TiDB cluster topology configuration + Review the provisioned cluster topology YAML. ```bash cat dr-auto-sync.yaml ``` ```output global: user: "ec2-user" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" arch: "amd64" monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115 deploy_dir: "/tidb-deploy/monitored-9100" data_dir: "/tidb-data/monitor_9115" log_dir: "/tidb-deploy/monitor-9100/log" server_configs: pd: replication.location-labels: ["dc","logic","rack","host"] tidb_servers: - host: 10.90.1.163 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" - host: 10.90.2.111 port: 4000 status_port: 10080 deploy_dir: "/tidb-deploy/tidb-4000" log_dir: "/tidb-deploy/tidb-4000/log" tikv_servers: - host: 10.90.1.80 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic1",rack: "r1",host: "10.90.1.80" } - host: 10.90.1.45 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc1",logic: "logic2",rack: "r1",host: "10.90.1.45" } - host: 10.90.2.33 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic3",rack: "r1",host: "10.90.2.33" } - host: 10.90.2.113 port: 20160 status_port: 20180 deploy_dir: "/tidb-deploy/tikv-20160" data_dir: "/tidb-data/tikv_data" config: server.labels: { dc: "dc2",logic: "logic4",rack: "r1",host: "10.90.2.113" } pd_servers: - host: 10.90.1.19 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://35.88.184.77:2379" - host: 10.90.1.44 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://34.219.233.52:2379" - host: 10.90.2.108 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://18.236.155.129:2379" monitoring_servers: - host: 10.90.1.147 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" - host: 10.90.2.82 port: 10090 deploy_dir: "/tidb-deploy/prometheus-10090" data_dir: "/tidb-data/prometheus_data" grafana_servers: - host: 10.90.1.147 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" - host: 10.90.2.82 port: 3000 deploy_dir: "/tidb-deploy/grafana-3000" ``` ### Step 3. Set TiUP mirror and execute `precheck` command ```bash tiup mirror set tidb-community-server-${LAB:TIDB_VERSION}-linux-amd64 tiup cluster check dr-auto-sync.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-${LAB:REGION_NAME}.pem \ --apply ``` ```output ... + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks - Applying changes on 10.90.2.33 ... Done - Applying changes on 10.90.2.113 ... Done - Applying changes on 10.90.1.163 ... Done - Applying changes on 10.90.2.111 ... Done - Applying changes on 10.90.1.147 ... Done - Applying changes on 10.90.1.44 ... Done - Applying changes on 10.90.2.108 ... Done - Applying changes on 10.90.1.45 ... Done - Applying changes on 10.90.2.82 ... Done - Applying changes on 10.90.1.19 ... Done - Applying changes on 10.90.1.80 ... Done ``` ### Step 4. Deploy the TiDB cluster ```bash tiup cluster deploy tidb-demo v${LAB:TIDB_VERSION} ./dr-auto-sync.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-${LAB:REGION_NAME}.pem \ --yes ``` ```output ... Cluster `tidb-demo` deployed successfully, you can start it with command: `tiup cluster start tidb-demo --init` ``` ### Step 5. Start the TiDB cluster ```bash tiup cluster start tidb-demo ``` ```output ... Started cluster `tidb-demo` successfully ``` ### Step 6. Verity the status of the TiDB cluster + Get the cluster status. Key cluster components are spread across 2 subnets (10.90.1.0/24 for DC1, 10.90.2.0/24 for DC2), which are located in 2 Availability Zones. For each data center, there are 2 PD servers(DC2 contains one standby pd node), 2 TiKV servers, and 1 TiDB servers. Therefore, we can use this deployment to simulate a two-data-center environment. ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.188.60.25:2379/dashboard Grafana URL: http://10.90.1.114:3000,http://10.90.2.122:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.114:3000 grafana 10.90.1.114 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.122:3000 grafana 10.90.2.122 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.114:10090 prometheus 10.90.1.114 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.122:10090 prometheus 10.90.2.122 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.81:4000 tidb 10.90.1.81 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.207:4000 tidb 10.90.2.207 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.163:20160 tikv 10.90.1.163 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.83:20160 tikv 10.90.1.83 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.17:20160 tikv 10.90.2.17 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.19:20160 tikv 10.90.2.19 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ### Step 7. Make sure the Placement Driver leader is located in data center 1 + If pd leader is not located in DC1 (IP belongs to 10.90.1.0/24 subnet), transfer the leader to DC1. ```output ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 ``` + You can execute this command to try to transfer pd leader to DC1 whatever the previous step's result is. ```bash tiup ctl:v${LAB:TIDB_VERSION} pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 member leader transfer "pd-${HOST_AZ1PD1_PRIVATE_IP}-2379" ``` ```output The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` + Check the location of pd leader, which is switched to DC1. ```bash tiup cluster display tidb-demo ``` ```output ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- ... 10.90.1.106:2379 pd 10.90.1.106 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.162:2379 pd 10.90.2.162 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 ... ``` ### Step 8. Create a peering TiUP monitor in data center 2 + Copy the cluster meta file of tiup to Monitor 2 (tiup directory) which is located at disaster recovery data center (DC2). If prompted, answer `yes`. ```bash scp -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem -r .tiup/storage/ ec2-user@${LAB:HOST_AZ2MONITOR1_PRIVATE_IP}:/home/ec2-user/.tiup/ ``` ```output ... run_blackbox_exporter_10.90.1.163.sh 100% 519 356.8KB/s 00:00 run_blackbox_exporter_10.90.2.82.sh 100% 519 414.9KB/s 00:00 meta.yaml 100% 4481 3.2MB/s 00:00 ``` + Open a new terminal - [Terminal 2](ssh://az2monitor1-ssh#2) connect to monitor 2 to verify that the tiup meta file is functioning properly. + On [Terminal 2](ssh://az2monitor1-ssh#2), confirm that `tiup` in monitor 2 can work well. ```bash tiup cluster display tidb-demo ``` ```output Checking updates for component cluster... A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.88.184.77:2379/dashboard Grafana URL: http://10.90.1.147:3000,http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.1.147:3000 grafana 10.90.1.147 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.82:3000 grafana 10.90.2.82 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.19:2379 pd 10.90.1.19 2379/2380 linux/x86_64 Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.44:2379 pd 10.90.1.44 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.108:2379 pd 10.90.2.108 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.1.147:10090 prometheus 10.90.1.147 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.82:10090 prometheus 10.90.2.82 10090/12020 linux/x86_64 Up /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.1.163:4000 tidb 10.90.1.163 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.111:4000 tidb 10.90.2.111 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.45:20160 tikv 10.90.1.45 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.1.80:20160 tikv 10.90.1.80 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.113:20160 tikv 10.90.2.113 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.33:20160 tikv 10.90.2.33 20160/20180 linux/x86_64 Up /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 13 ``` ### Step 9. Enable DR Auto-Sync for your TiDB cluster + Go back to [Terminal 1](ssh://az1monitor1-ssh#1), check the current cluster data replication mode, which should be `majority` as default. Note: The result `"acid_consistent": false` means the data replication between primary data center and disaster recovery data center is not in `Sync` mode. So, the `ACID` properties of transactions cannot be absolutely guaranteed when primary data center experiences a complete failure. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "majority", "dr-auto-sync": { "label_key": "", "state": "", "acid_consistent": false } } ``` + View the placement rules file that prepared for DR Auto-Sync configuration. There are two data centers, and voter role is limited in DC1. In DC2, there is a follower and a learner role. ```bash cat dr-anto-sync-rules.json ``` ```output [ { "group_id": "pd", "group_index": 0, "group_override": false, "rules": [ { "group_id": "pd", "id": "dc1", "start_key": "", "end_key": "", "role": "voter", "count": 2, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "dc", "op": "in", "values": ["dc1"]}] }, { "group_id": "pd", "id": "logic3", "start_key": "", "end_key": "", "role": "follower", "count": 1, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "logic", "op": "in", "values": ["logic3"]}] }, { "group_id": "pd", "id": "logic4", "start_key": "", "end_key": "", "role": "learner", "count": 1, "location_labels": ["dc", "logic", "rack", "host"], "label_constraints": [{"key": "logic", "op": "in", "values": ["logic4"]}] } ] } ] ``` ![60%](https://lab-static.pingcap.com/quick-demo/demo-303-41-en.png) + By importing the above placement rules, PD can perceive the distribution design of different nodes (Learner more: [PD Scheduling Best Practices - Cluster topology awareness](https://docs.pingcap.com/tidb/stable/pd-scheduling-best-practices#cluster-topology-awareness)). This enables the replicas of Regions within the cluster to be dispersed as much as possible. PD schedules based on the location-labels configuration. The configuration ["dc", "logic", "rack", "host"] defines a 4-level topology structure: + The cluster is divided into multiple dc (data centers). + Each dc contains multiple logic (logical zones). + Each logic contains multiple rack units. + Each rack contains multiple host units. + When scheduling, PD will: + Prioritize placing the Region's Peer in different dc. + If this is not feasible (e.g., 3 replicas are configured but there are only 2 data centers available), PD ensures placement in different logic zones. + If the number of logic zones is also insufficient to ensure isolation, it will then attempt to isolate at the rack level. + If the number of rack units is still inadequate, it will attempt isolation at the host level. + Load the placement rules configuration, the `ctl` tool will be downloaded and installed automatically. ```bash tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules rule-bundle \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 save \ --in="/home/ec2-user/dr-anto-sync-rules.json" ``` ```output The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository. download https://tiup-mirrors.pingcap.com/ctl-v${LAB:TIDB_VERSION}-linux-amd64.tar.gz 408.65 MiB / 408.65 MiB 100.00% 103.24 MiB/s Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules rule-bundle -u http://10.90.2.162:2379 save --in=/home/ec2-user/dr-anto-sync-rules.json "Update rules and groups successfully." ``` + Check if placement rules configuration is loaded. ```bash tiup ctl:v${LAB:TIDB_VERSION} pd config placement-rules show -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd config placement-rules show -u http://10.90.2.162:2379 [ { "group_id": "pd", "id": "dc1", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 2, "label_constraints": [ { "key": "dc", "op": "in", "values": [ "dc1" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 }, { "group_id": "pd", "id": "logic3", "start_key": "", "end_key": "", "role": "voter", "is_witness": false, "count": 1, "label_constraints": [ { "key": "logic", "op": "in", "values": [ "logic3" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 }, { "group_id": "pd", "id": "logic4", "start_key": "", "end_key": "", "role": "learner", "is_witness": false, "count": 1, "label_constraints": [ { "key": "logic", "op": "in", "values": [ "logic4" ] } ], "location_labels": [ "dc", "logic", "rack", "host" ], "version": 2, "create_timestamp": 1733292613 } ] ``` + Install `jq` tool. ```bash sudo yum install jq -y ``` ```output ... Installed: jq.x86_64 0:1.5-1.amzn2.0. 2 Dependency Installed: oniguruma.x86_64 0:5.9.6-1.amzn2.0. 7 Complete! ``` + Check for TiKV regions without learner peers. Verify that there are no regions without learner peers and ensure data synchronization between voter and learner roles. This step ensures that the cluster's data is consistently replicated across all roles, which is essential for maintaining high availability and failover readiness. ```bash tiup ctl:v${LAB:TIDB_VERSION} pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 region \ --jq='.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]}' ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://34.215.73.186:2379 region --jq=.regions[] | select(.peers | any(.role_name=="Learner") | not) | {id: .id, peers: [.peers]} ``` + Enable the DR Auto-Sync configuration. ```bash tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync label-key dc tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary dc1 tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr dc2 tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync primary-replicas 2 tiup ctl:v${LAB:TIDB_VERSION} pd -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config set replication-mode dr-auto-sync dr-replicas 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync label-key dc Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary dc1 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr dc2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync primary-replicas 2 Success! The replication mode config is updated." Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379 config set replication-mode dr-auto-sync dr-replicas 1 Success! The replication mode config is updated." ``` + Through the configuration of these parameters, the cluster achieves location awareness of data centers and control over data distribution. + set replication-mode dr-auto-sync: Set the replication mode of TiDB to dr-auto-sync. + set replication-mode dr-auto-sync label-key dc: Enable PD to identify the data centers to which nodes belong based on their dc labels, thereby achieving data distribution control. + set replication-mode dr-auto-sync primary dc1: Set DC1 as the primary data center, making it the main write node location for the TiDB cluster during runtime. All data updates are prioritized to be written to the primary data center. + set replication-mode dr-auto-sync dr dc2: Set DC2 as the disaster recovery data center, where TiKV nodes in the learner role reside. + set replication-mode dr-auto-sync primary-replicas 2: Set the number of replicas in the primary data center (DC1) to 2. + set replication-mode dr-auto-sync dr-replicas 1: Set the number of replicas in the disaster recovery data center (DC2) to 1 (learner role not contained). + Verify whether the configuration is effective. ```bash tiup ctl:v${LAB:TIDB_VERSION} pd \ -u ${HOST_AZ1PD1_PRIVATE_IP}:2379 config show replication-mode ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.162:2379:2379 config show replication-mode { "replication-mode": "dr-auto-sync", "dr-auto-sync": { "label-key": "dc", "primary": "dc1", "dr": "dc2", "primary-replicas": 2, "dr-replicas": 1, "wait-store-timeout": "1m0s", "wait-recover-timeout": "0s", "pause-region-split": "false" } } ``` + Check the current replication status of the cluster. You can see `sync_recover` mode and `recover_progress` before the status becomes to `sync` mode if your operation is fast. ```bash curl ${HOST_AZ1PD1_PRIVATE_IP}:2379/pd/api/v1/replication_mode/status ``` ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync_recover", "state_id": 2003, "acid_consistent": false, "total_regions": 60, "synced_regions": 37, "recover_progress": 0.98333335 } } ``` + Finally, you will get the `sync` state of TiDB. ```output { "mode": "dr-auto-sync", "dr-auto-sync": { "label_key": "dc", "state": "sync", "state_id": 378, "acid_consistent": true } } ``` + As an initialization status, the cluster is in the synchronous replication mode. PD sends the status information to TiKV, and the two data centers (two commit groups) strictly follow the synchronous replication mode to work. ### Step 10. Create a sysbench database + Log in to TiDB. ```bash mysql -h ${HOST_AZ1DB1_PRIVATE_IP} -P 4000 -u root --connect-expired-password ``` ```output mysql> ``` + Create a database for `sysbench`. ```sql CREATE DATABASE sbtest; EXIT; ``` ```output Query OK, 0 rows affected (0.13 sec) Bye ``` + Set up the `sysbench` configuration for DC1. ```bash cat << EOF > sysbench_config mysql-host=${HOST_AZ1DB1_PRIVATE_IP} mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=6 report-interval=1 EOF cat sysbench_config ``` ```output mysql-host=10.90.1.125 mysql-port=4000 mysql-user=root mysql-password= mysql-db=sbtest time=3600 threads=16 report-interval=1 ``` + Prepare `sysbench` data for the next module. ```bash sysbench \ --config-file=sysbench_config oltp_point_select \ --tables=6 \ --table-size=100 prepare ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest3'... Creating table 'sbtest1'... Creating table 'sbtest5'... Creating table 'sbtest6'... Creating table 'sbtest4'... Creating table 'sbtest2'... Inserting 100 records into 'sbtest3' Creating a secondary index on 'sbtest3'... Inserting 100 records into 'sbtest1' Creating a secondary index on 'sbtest1'... Inserting 100 records into 'sbtest4' Creating a secondary index on 'sbtest4'... Inserting 100 records into 'sbtest6' Creating a secondary index on 'sbtest6'... Inserting 100 records into 'sbtest2' Creating a secondary index on 'sbtest2'... Inserting 100 records into 'sbtest5' Creating a secondary index on 'sbtest5'... ``` + Close Terminal 1 and Terminal 2. ### Summary for Module 1 + At this point, we have completed: + The creation of the TiDB cluster. + Enabled the TiDB cluster DR Auto-Sync deployment plan and verified its status. + Prepared the sysbench configuration for simulating application behavior.