--- title: "(WIP) Three Data Center Deployment Solution" version: "v2" status: "WIP" description: "In this lab, we will introduce a three-data-center deployment model for TiDB, it provides banks and financial institutions with extreme data availability and disaster recovery capabilities." --- # (WIP) Three Data Center Deployment Solution ## Overview In this lab, the TiDB cluster is deployed across three Availability Zones, each corresponding to a subnet. You will simulate a network blackout in Subnet 1. This will disconnect one TiDB server, one TiKV server, and one PD server from the cluster. Infrastructure failures do not require server-side failover as database clients automatically connect to the remaining TiDB server instances. ![80%](https://lab-static.pingcap.com/quick-demo/demo_001_en.png) ### Duration + This lab should take you approximately 60 minutes to complete. ### Tasks + Review the TiDB cluster status and topology. + Blockade the network communication for an entire availability zone. + Observe and verify that the TiDB cluster is able to maintain functionality even in the event of an entire availability zone being inaccessible. ### Prerequisites > [!NOTE] > If you plan to use Web SSH in your lab, you can skip this section. Once the lab environment is ready, the Web SSH button will appear. And, click **Next** on the bottom bar to proceed to **Module 1**. + **macOS** users: + Click the private key file [pe-class-key-${LAB:REGION_NAME}.pem](/api/labs/demo_001/vm_key?region_name=${LAB:REGION_NAME}) to download it. This file is required to log in to the remote Linux for this hands-on lab. + Save the private key file from the download folder to your `~/.ssh/` folder. + Use the following commands to set the permission and check whether the file is in place. ```bash chmod 400 ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ls -lF ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ``` ```output ... -r--------@ 1 username staff 1674 Jun 21 2023 /Users/username/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ``` + **Windows 10** users: + Click the private key file [pe-class-key-${LAB:REGION_NAME}.pem](/api/labs/demo_001/vm_key?region_name=${LAB:REGION_NAME}) to download it. This file is required to log in to the remote Linux for this hands-on lab. + Save the private key file from the download folder in `%userprofile%\.ssh` folder. + Use the following command to check if the file is in place. Note that you should use CMD, not PowerShell. ```cmd dir %userprofile%\.ssh\pe-class-key-${LAB:REGION_NAME}.pem ``` ```output ... 2024/11/14 10:22 1,674 pe-class-key-${LAB:REGION_NAME}.pem 1 Files(s) 1,674 bytes ... ``` + **Windows 8** or earlier users: + Click the private key file [pe-class-key-${LAB:REGION_NAME}.ppk](/api/labs/demo_001/vm_key?region_name=${LAB:REGION_NAME}&key_type=ppk) to download it. This file is required to log in to the remote Linux for this hands-on lab. + Download and install [PuTTY](https://www.putty.org/). ## Module 1: Deploy A TiDB Cluster Across Three Availability Zones ### Step 1. SSH login to node ${LAB:HOST_MONITOR1_PUBLIC_IP} + If you use Web SSH for the lab, click `Web SSH` button or icon to open the terminal session, and proceed to **Step 2**. + For connection details see Prerequisites section. If prompted, answer `yes`. + **macOS** users: Open **two** terminals, in **each** terminal execute the following command. ```bash ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${LAB:HOST_MONITOR1_PUBLIC_IP} ``` + **Windows 10** users: Open **two** CMD windows, in **each** CMD window execute the following command. ```cmd ssh -i %userprofile%\.ssh\pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${LAB:HOST_MONITOR1_PUBLIC_IP} ``` + **Windows 8** or earlier users: + Start PuTTY by double clicking the PuTTY icon, in `Session` -> `Host Name`, type in `ec2-user@${LAB:HOST_MONITOR1_PUBLIC_IP}`. + In `Connection` -> `SSH` -> `Auth` -> `Credentials` -> `Private key file for authentication`, browse to find the location of your key file. + Now, you should be able to connect to the remote Linux, open **two** sessions by double clicking the new session config you just created. + The banner displayed after logging in appears as follows. Please note that you will need **two** separate terminals, which means you must log into the remote Linux system **two** times. ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### Step 2. Start the TiDB cluster across multiple Availability Zones + In **Terminal 1**, review the provisioned cluster topology YAML. ```bash cat eleven-nodes.yaml ``` ```output global: user: "ec2-user" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" arch: "amd64" server_configs: pd: replication.max-replicas: 3 tidb: security.auto-tls: true graceful-wait-before-shutdown: 10 pd_servers: - host: 10.90.3.8 client_port: 2379 peer_port: 2380 advertise_client_addr: "http://13.114.31.174:2379" - host: 10.90.3.52 client_port: 2379 peer_port: 2380 advertise_client_addr: "http://18.182.10.60:2379" - host: 10.90.3.89 client_port: 2379 peer_port: 2380 advertise_client_addr: "http://3.112.235.40:2379" tidb_servers: - host: 10.90.2.254 port: 4000 status_port: 10080 deploy_dir: /tidb-deploy/tidb-4000 log_dir: /tidb-deploy/tidb-4000/log - host: 10.90.2.228 port: 4000 status_port: 10080 deploy_dir: /tidb-deploy/tidb-4000 log_dir: /tidb-deploy/tidb-4000/log tikv_servers: - host: 10.90.3.80 port: 20160 status_port: 20180 - host: 10.90.3.254 port: 20160 status_port: 20180 - host: 10.90.3.182 port: 20160 status_port: 20180 monitoring_servers: - host: 10.90.4.52 grafana_servers: - host: 10.90.4.52 alertmanager_servers: - host: 10.90.4.52 tiproxy_servers: - host: 10.90.4.30 ``` + Fix system configurations in order to deploy a TiDB cluster. ```bash tiup mirror set tidb-community-server-${LAB:TIDB_VERSION}-linux-amd64 tiup cluster check eleven-nodes.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-${LAB:REGION_NAME}.pem \ --apply ``` ```output . . . + Try to apply changes to fix failed checks - Applying changes on 10.90.4.30 ... ⠦ Sysctl: host=10.90.4.30 vm.swappiness = 0 + Try to apply changes to fix failed checks - Applying changes on 10.90.4.30 ... ⠧ Sysctl: host=10.90.4.30 vm.swappiness = 0 + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks - Applying changes on 10.90.4.30 ... Done - Applying changes on 10.90.3.80 ... Done - Applying changes on 10.90.3.254 ... Done + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks - Applying changes on 10.90.4.30 ... Done - Applying changes on 10.90.3.80 ... Done - Applying changes on 10.90.3.254 ... Done - Applying changes on 10.90.3.8 ... Done - Applying changes on 10.90.3.52 ... Done - Applying changes on 10.90.3.89 ... Done - Applying changes on 10.90.3.182 ... Done - Applying changes on 10.90.2.254 ... Done - Applying changes on 10.90.2.228 ... Done - Applying changes on 10.90.4.52 ... Done ``` + Deploy a TiDB cluster. ```bash tiup cluster deploy tidb-demo ${LAB:TIDB_VERSION} ./eleven-nodes.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-${LAB:REGION_NAME}.pem \ --yes ``` ```output . . . Cluster `tidb-demo` deployed successfully, you can start it with command: `tiup cluster start tidb-demo --init` ``` + Start the deployed TiDB cluster. ```bash tiup cluster start tidb-demo ``` ```output . . . Started cluster `tidb-demo` successfully ``` ### Step 3. Verify the status of the TiDB cluster + The key cluster components are distributed across three subnets: 10.90.1.0/24, 10.90.2.0/24, and 10.90.3.0/24, each corresponding to an Availability Zone. + The cluster includes 3 PD servers, 3 TiKV servers, and 2 TiDB servers. + Pay particular attention to subnet 10.90.1.0/24, which hosts a PD server, a TiDB server, and a TiKV server. + While the IP addresses in your environment and sample output may vary, the subnet configuration remains consistent. ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://18.182.10.60:2379/dashboard Grafana URL: http://10.90.4.52:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.52:9093 alertmanager 10.90.4.52 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.52:3000 grafana 10.90.4.52 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.52:2379 pd 10.90.3.52 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.8:2379 pd 10.90.3.8 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.89:2379 pd 10.90.3.89 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.52:9090 prometheus 10.90.4.52 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.2.228:4000 tidb 10.90.2.228 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.254:4000 tidb 10.90.2.254 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.182:20160 tikv 10.90.3.182 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.254:20160 tikv 10.90.3.254 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.80:20160 tikv 10.90.3.80 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.30:6000 tiproxy 10.90.4.30 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 12 ``` ## Module 2: Start The Fund Transfer Workloads ### Step 1. Install dependencies of BankTransferTransaction tool. + Install python dependencies. ```bash pip3 install -r ./BankTransferTransaction/requirements.txt ``` ```output ... Successfully installed argparse-1.4.0 black-23.3.0 click-8.1.8 coverage-7.2.7 exceptiongroup-1.2.2 flake8-5.0.4 importlib-metadata-4.2.0 iniconfig-2.0.0 isort-5.11.5 mccabe-0.7.0 mypy-extensions-1.0.0 mysql-connector-2.2.9 packaging-24.0 pathspec-0.11.2 platformdirs-4.0.0 pluggy-1.2.0 pycodestyle-2.9.1 pyflakes-2.5.0 pytest-7.4.4 pytest-cov-4.1.0 python-dotenv-0.21.1 pytz-2024.2 tomli-2.0.1 typed-ast-1.5.5 typing-extensions-4.7.1 zipp-3.15.0 ``` ### Step 2. Initiate the data for BankTransferTransaction tool. + Initialize the database structure. ```bash python3 ./BankTransferTransaction/database/init_db.py --host ${LAB:HOST_TIPROXY1_PRIVATE_IP} ``` ```output Connecting to TiDB server at 10.90.4.30... create database/tables success! ``` + Generate test data. ```bash python3 ./BankTransferTransaction/database/seed_data.py --host ${LAB:HOST_TIPROXY1_PRIVATE_IP} ``` ```output Connecting to TiDB server at 10.90.4.30... Starting account initialization... Successfully created 100 accounts Data initialization completed! Statistics: Total accounts: 100 Total transaction records: 0 ``` ### Step 3. Start BankTransferTransaction tool to execute transfer transaction, the task will be runing for 60 minutes. + Start BankTransferTransaction ```bash python3 ./BankTransferTransaction/run_transfers.py 60 --host ${LAB:HOST_TIPROXY1_PRIVATE_IP} ``` ```output Connecting to TiDB server at 10.90.4.30... Starting transfer transactions... ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Time Status Sender Receiver Amount Sender Balance Receiver Balance Note ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 1s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 2s SUCCESS user_26 user_27 100.00 USD 2900.00->2800.00 3100.00->3200.00 3s SUCCESS user_45 user_55 200.00 USD 5100.00->4900.00 8500.00->8700.00 4s SUCCESS user_94 user_10 400.00 USD 6400.00->6000.00 4900.00->5300.00 5s SUCCESS user_18 user_11 600.00 USD 5700.00->5100.00 9400.00->10000.00 ``` ## Module 3: Isolate The Subnet 1 In Availability Zone 1 ### Step 1. Simulate a complete Availability Zone failure in Availability Zone 1 + Return to **Terminal 1** and execute the script to simulate a complete Availability Zone failure by blocking subnet `demo-subnet-1` (subnet 10.90.1.0/24 in Availability Zone 1). ```bash VPC_ID=`aws ec2 describe-vpcs \ --filters "Name=tag:Name,Values=user_unique_tag" \ --query "Vpcs[0].VpcId" \ --output text \ --region ${LAB:REGION_CODE}` SUBNET_ID=`aws ec2 describe-subnets \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-1" \ --query "Subnets[0].SubnetId" \ --output text \ --region ${LAB:REGION_CODE}` CAGE_NACL_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" "Name=tag:Name,Values=cage" \ --query "NetworkAcls[0].NetworkAclId" \ --output text \ --region ${LAB:REGION_CODE}` ASSOC_ID=`aws ec2 describe-network-acls \ --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \ --query "NetworkAcls[0].Associations" \ --output text \ --region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'` aws ec2 replace-network-acl-association \ --association-id ${ASSOC_ID} \ --network-acl-id ${CAGE_NACL_ID} \ --region ${LAB:REGION_CODE} ``` ```output { "NewAssociationId": "aclassoc-0402a6103febf799f" } ``` ### Step 2. Verify that several components are disconnected + In **Terminal 1**, check the cluster status again to verify that several components, including PD, TiDB and TiKV servers are disconnected/isolated because of the Availability Zone failure. + In the sample output, the impacted servers are `10.90.1.208:2379 pd`, `10.90.1.247:4000 tidb`, and `10.90.1.122:20160 tikv`. Therefore, one-third of the cluster components have lost connection. ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://52.13.86.218:2379/dashboard Grafana URL: http://10.90.4.74:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.74:9093 alertmanager 10.90.4.74 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.74:3000 grafana 10.90.4.74 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.208:2379 pd 10.90.1.208 2379/2380 linux/x86_64 Down /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.39:2379 pd 10.90.2.39 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.68:2379 pd 10.90.3.68 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.74:9090 prometheus 10.90.4.74 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.247:4000 tidb 10.90.1.247 4000/10080 linux/x86_64 Down - /tidb-deploy/tidb-4000 10.90.2.33:4000 tidb 10.90.2.33 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.122:20160 tikv 10.90.1.122 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.5:20160 tikv 10.90.2.5 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.14:20160 tikv 10.90.3.14 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 Total nodes: 11 ``` ### Step 3. Observe the application status. + BankTransferTransaction tool is blocked to access to TiDB cluster for about 10 seconds which is a little bit longer than the block window setting (default setting is 60 seconds). The reason is that the TiDB cluster needs time to gather network status, process the TiDB cluster status switch, and synchronize the status to each TiKV nodes. ```output ... ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Time Status Sender Receiver Amount Sender Balance Receiver Balance Note ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 108s SUCCESS user_98 user_29 100.00 USD 9000.00->8900.00 2600.00->2700.00 109s SUCCESS user_12 user_32 200.00 USD 4200.00->4000.00 6200.00->6400.00 110s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 111s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 112s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 113s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 114s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 115s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 116s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 117s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 118s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 119s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 120s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 121s WAIT N/A N/A 0.00 USD 0.00->0.00 0.00->0.00 Waiting for transaction... 122s SUCCESS user_34 user_67 900.00 USD 2000.00->1100.00 7500.00->8400.00 ... ``` ### Step 4. Scale-in the offline nodes + The immediate operations after confirming the AZ1 failure which can't be recovered soon should be keeping the cluster with the same capacity. The following steps are restoring the cluster size same with before. + Scale in the failed nodes which are located in AZ1. ```bash tiup cluster scale-in tidb-demo -N "${LAB:HOST_PD1_PRIVATE_IP}:2379,${LAB:HOST_DB1_PRIVATE_IP}:4000,${LAB:HOST_KV1_PRIVATE_IP}:20160" -y ``` ```output The component `ctl` version vtidb_version is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/vtidb_version/ctl pd -u 10.90.1.19:2379 member leader transfer pd-10.90.1.106-2379 Success! ``` ### Step 5. Scale-out new nodes into the cluster + Check the yaml file. ```bash cat scale-out-new-nodes.yaml ``` ```output pd_servers: - host: 10.90.2.102 client_port: 2379 peer_port: 2380 deploy_dir: "/tidb-deploy/pd-2379" data_dir: "/tidb-data/pd_data" advertise_client_addr: "http://35.161.129.28:2379" ``` + Scale the new nodes(pd4, db3 and kv4) into TiDB cluster. ```bash tiup cluster scale-out tidb-demo scale-out-new-nodes.yaml --user ec2-user -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem -y ``` ```output ... Starting component pd Starting instance 10.90.2.102:2379 Start instance 10.90.2.102:2379 success Starting component node_exporter Starting instance 10.90.2.102 Start 10.90.2.102 success Starting component blackbox_exporter Starting instance 10.90.2.102 Start 10.90.2.102 success + Refresh components conifgs - Generate config pd -> 10.90.2.26:2379 ... Done - Generate config pd -> 10.90.2.102:2379 ... Done - Generate config tikv -> 10.90.2.254:20160 ... Done - Generate config tikv -> 10.90.2.182:20160 ... Done - Generate config tidb -> 10.90.2.49:4000 ... Done - Generate config prometheus -> 10.90.2.38:10090 ... Done - Generate config grafana -> 10.90.2.38:3000 ... Done + Reload prometheus and grafana - Reload prometheus -> 10.90.2.38:10090 ... Done - Reload grafana -> 10.90.2.38:3000 ... Done + [ Serial ] - UpdateTopology: cluster=tidb-demo Scaled cluster `tidb-demo` out successfully ``` + Verify that PD/db/kv nodes in DC2 are all up and running. ```bash tiup cluster display tidb-demo ``` ```output A new version of cluster is available: v1.15.1 -> v1.16.1 To update this component: tiup update cluster To update all components: tiup update --all Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.94.179.113:2379/dashboard Grafana URL: http://10.90.2.82:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.2.107:3000 grafana 10.90.2.107 3000 / Up - /tidb-deploy/grafana-3000 10.90.2.83:2379 pd 10.90.2.83 2379/2380 / Up|L /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.96:2379 pd 10.90.2.96 2379/2380 linux/x86_64 Up /tidb-data/pd_data /tidb-deploy/pd-2379 10.90.2.107:10090 prometheus 10.90.2.107 10090 / Down /tidb-data/prometheus_data /tidb-deploy/prometheus-10090 10.90.2.210:4000 tidb 10.90.2.210 4000/10080 / Down - /tidb-deploy/tidb-4000 10.90.2.111:20160 tikv 10.90.2.111 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 10.90.2.91:20160 tikv 10.90.2.91 20160/20180 / Disconnected /tidb-data/tikv_data /tidb-deploy/tikv-20160 Total nodes: 7 ``` + Press Ctrl+C in **Terminal 2** and **Terminal 3** to stop both workloads. ## Conclusion Thanks to TiDB's distributed architecture, failures at the data center or Availability Zone level don't need to trigger a server-side failover and let the database clients wait on the failover to complete. Instead, database clients can seamlessly connect to the available TiDB server instances, ensuring continuous business operations. Should such disasters occur, the application can set an operation timeout by their business requirements. Unlike traditional systems, TiDB doesn't require server-side failover to complete before the database client or server-side proxy can re-connect. ```hbsptform { "ui": { "titleLevel": 3 }, "portalId": "4466002", "formId": "89fa8368-eba1-4b84-b91e-eb9eb6613ddf", "region": "na1", "sfdcCampaignId": "701RC00000VqCODYA3" } ```