---
title: "Self-Assessment: Deploying and Managing TiDB Clusters by using TiUP"
version: "v2"
status: "PLACEHOLDER"
description: "In this hands-on lab, you will learn how to deploy a TiDB cluster using TiUP. Specifically, we will cover how to deploy a TiDB cluster using TiUP, configure TiDB, and interact with the TiDB cluster. This lab is a practical workshop that allows you to learn how to deploy a TiDB cluster using TiUP."
---

# Self-Assessment: Deploying and Managing TiDB Clusters by using TiUP

## Overview

In this lab, the TiDB cluster is deployed across three Availability Zones, each corresponding to a subnet.

You will simulate a network blackout in Subnet 1. This will disconnect one TiDB server, one TiKV server, and one PD server from the cluster.

Infrastructure failures do not require server-side failover as database clients automatically connect to the remaining TiDB server instances.

![80%](https://lab-static.pingcap.com/quick-demo/demo_001_en.png)

### Duration

+ This lab should take you approximately 60 minutes to complete.

### Tasks

+ Review the TiDB cluster status and topology.
+ Blockade the network communication for an entire availability zone.
+ Observe and verify that the TiDB cluster is able to maintain functionality even in the event of an entire availability zone being inaccessible.

### Prerequisites

> [!NOTE]
> If you plan to use Web SSH in your lab, you can skip this section. Once the lab environment is ready, the Web SSH button will appear. And, click **Next** on the bottom bar to proceed to **Module 1**.

+ **macOS** users:
  + Click the private key file [pe-class-key-${LAB:REGION_NAME}.pem](/api/labs/demo_001/vm_key?region_name=${LAB:REGION_NAME}) to download it. This file is required to log in to the remote Linux for this hands-on lab.
  + Save the private key file from the download folder to your `~/.ssh/` folder.
  + Use the following commands to set the permission and check whether the file is in place.

    ```bash
    chmod 400 ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem
    ls -lF ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem
    ```

    ```output
    ...
    -r--------@ 1 username staff 1674 Jun 21 2023 /Users/username/.ssh/pe-class-key-${LAB:REGION_NAME}.pem
    ```

+ **Windows 10** users:
  + Click the private key file [pe-class-key-${LAB:REGION_NAME}.pem](/api/labs/demo_001/vm_key?region_name=${LAB:REGION_NAME}) to download it. This file is required to log in to the remote Linux for this hands-on lab.
  + Save the private key file from the download folder in `%userprofile%\.ssh` folder.
  + Use the following command to check if the file is in place. Note that you should use CMD, not PowerShell.
  
    ```cmd
    dir %userprofile%\.ssh\pe-class-key-${LAB:REGION_NAME}.pem
    ```

    ```output
    ...
    2024/11/14 10:22 1,674 pe-class-key-${LAB:REGION_NAME}.pem
    1 Files(s) 1,674 bytes
    ...
    ```

+ **Windows 8** or earlier users:
  + Click the private key file [pe-class-key-${LAB:REGION_NAME}.ppk](/api/labs/demo_001/vm_key?region_name=${LAB:REGION_NAME}&key_type=ppk) to download it. This file is required to log in to the remote Linux for this hands-on lab.
  + Download and install [PuTTY](https://www.putty.org/).

## Module 1: Deploy A TiDB Cluster Across Three Availability Zones

### Step 1. SSH login to node ${LAB:HOST_MONITOR1_PUBLIC_IP}

+ If you use Web SSH for the lab, click `Web SSH` button or icon to open the terminal session, and proceed to **Step 2**.

+ For connection details see <kbd>Prerequisites</kdb> section. If prompted, answer `yes`.

  + **macOS** users: Open **three** terminals, in **each** terminal execute the following command.

    ```bash
    ssh -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${LAB:HOST_MONITOR1_PUBLIC_IP}
    ```

  + **Windows 10** users: Open **three** CMD windows, in **each** CMD window execute the following command. 

    ```cmd
    ssh -i %userprofile%\.ssh\pe-class-key-${LAB:REGION_NAME}.pem ec2-user@${LAB:HOST_MONITOR1_PUBLIC_IP}
    ```

  + **Windows 8** or earlier users: 
    + Start PuTTY by double clicking the PuTTY icon, in `Session` -> `Host Name`, type in `ec2-user@${LAB:HOST_MONITOR1_PUBLIC_IP}`.
    + In `Connection` -> `SSH` -> `Auth` -> `Credentials` -> `Private key file for authentication`, browse to find the location of your key file.
    + Now, you should be able to connect to the remote Linux, open **three** sessions by double clicking the new session config you just created. 

  + The banner displayed after logging in appears as follows. Please note that you will need **three** separate terminals, which means you must log into the remote Linux system **three** times.

    ```output
     ,     #_
    ~\_  ####_        Amazon Linux 2
    ~~  \_#####\
    ~~     \###|      
    ~~       \#/ ___
    ~~       V~' '->
    ~~~         /    A newer version of Amazon Linux is available!
     ~~._.   _/
        _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.
      _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/
    ```

### Step 2. Start the TiDB cluster across multiple Availability Zones

+ In **Terminal 1**, review the provisioned cluster topology YAML.

  ```bash
  cat nine-nodes.yaml
  ```
  ```output
  global:
      user: "ec2-user"
      ssh_port: 22
      deploy_dir: "/tidb-deploy"
      data_dir: "/tidb-data"
      arch: "amd64"
  server_configs:
      pd:
          replication.max-replicas: 3
  pd_servers:
    - host: 10.90.3.145
      client_port: 2379
      peer_port: 2380
      advertise_client_addr: "http://54.203.147.80:2379"
    - host: 10.90.1.184
      client_port: 2379
      peer_port: 2380
      advertise_client_addr: "http://34.221.154.223:2379"
    - host: 10.90.2.106
      client_port: 2379
      peer_port: 2380
      advertise_client_addr: "http://35.86.165.210:2379"
  tidb_servers:
    - host: 10.90.1.177
      port: 4000
      status_port: 10080
      deploy_dir: /tidb-deploy/tidb-4000
      log_dir: /tidb-deploy/tidb-4000/log
    - host: 10.90.2.16
      port: 4000
      status_port: 10080
      deploy_dir: /tidb-deploy/tidb-4000
      log_dir: /tidb-deploy/tidb-4000/log
  tikv_servers:
    - host: 10.90.3.115
      port: 20160
      status_port: 20180
    - host: 10.90.2.143
      port: 20160
      status_port: 20180
    - host: 10.90.1.247
      port: 20160
      status_port: 20180
  monitoring_servers:
    - host: 10.90.4.92
  grafana_servers:
    - host: 10.90.4.92
  alertmanager_servers:
    - host: 10.90.4.92
  ```

+ Fix system configurations in order to deploy a TiDB cluster.

  ```bash
  tiup mirror set tidb-community-server-${LAB:TIDB_VERSION}-linux-amd64
  tiup cluster check nine-nodes.yaml \
    --user ec2-user \
    -i /home/ec2-user/.ssh/pe-class-key-${LAB:REGION_NAME}.pem \
    --apply
  ```
  ```output
  .
  .
  .
  + Try to apply changes to fix failed checks
   - Applying changes on 10.90.3.145 ... ⠧ Sysctl: host=10.90.3.145 vm.swappiness = 0
  + Try to apply changes to fix failed checks
   - Applying changes on 10.90.3.145 ... ⠇ Sysctl: host=10.90.3.145 vm.swappiness = 0
  + Try to apply changes to fix failed checks
  + Try to apply changes to fix failed checks
   - Applying changes on 10.90.3.145 ... Done
  + Try to apply changes to fix failed checks
   - Applying changes on 10.90.3.145 ... Done
  + Try to apply changes to fix failed checks
   - Applying changes on 10.90.3.145 ... Done
   - Applying changes on 10.90.2.106 ... Done
   - Applying changes on 10.90.2.143 ... Done
   - Applying changes on 10.90.1.177 ... Done
   - Applying changes on 10.90.1.184 ... Done
   - Applying changes on 10.90.3.115 ... Done
   - Applying changes on 10.90.1.247 ... Done
   - Applying changes on 10.90.2.16 ... Done
   - Applying changes on 10.90.4.92 ... Done
  ``` 

+ Deploy a TiDB cluster.

  ```bash
  tiup cluster deploy tidb-demo ${LAB:TIDB_VERSION} ./nine-nodes.yaml \
    --user ec2-user \
    -i /home/ec2-user/.ssh/pe-class-key-${LAB:REGION_NAME}.pem \
    --yes
  ```
  ```output
  .
  .
  .
  Cluster `tidb-demo` deployed successfully, you can start it with command: `tiup cluster start tidb-demo --init`
  ```

+ Start the deployed TiDB cluster. 

  ```bash
  tiup cluster start tidb-demo
  ```
  ```output
  .
  .
  .
  Started cluster `tidb-demo` successfully
  ```

### Step 3. Verify the status of the TiDB cluster

+ The key cluster components are distributed across three subnets: 10.90.1.0/24, 10.90.2.0/24, and 10.90.3.0/24, each corresponding to an Availability Zone.

+ The cluster includes 3 PD servers, 3 TiKV servers, and 2 TiDB servers.

+ Pay particular attention to subnet 10.90.1.0/24, which hosts a PD server, a TiDB server, and a TiKV server.

+ While the IP addresses in your environment and sample output may vary, the subnet configuration remains consistent.

  ```bash
  tiup cluster display tidb-demo
  ```
  ```output
  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://35.86.135.34:2379/dashboard
  Grafana URL:        http://10.90.4.7:3000
  ID                 Role          Host         Ports        OS/Arch       Status  Data Dir                      Deploy Dir
  --                 ----          ----         -----        -------       ------  --------                      ----------
  10.90.4.7:9093     alertmanager  10.90.4.7    9093/9094    linux/x86_64  Up      /tidb-data/alertmanager-9093  /tidb-deploy/alertmanager-9093
  10.90.4.7:3000     grafana       10.90.4.7    3000         linux/x86_64  Up      -                             /tidb-deploy/grafana-3000
  10.90.1.110:2379   pd            10.90.1.110  2379/2380    linux/x86_64  Up      /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.2.63:2379    pd            10.90.2.63   2379/2380    linux/x86_64  Up      /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.3.162:2379   pd            10.90.3.162  2379/2380    linux/x86_64  Up|L    /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.4.7:9090     prometheus    10.90.4.7    9090/12020   linux/x86_64  Up      /tidb-data/prometheus-9090    /tidb-deploy/prometheus-9090
  10.90.1.116:4000   tidb          10.90.1.116  4000/10080   linux/x86_64  Up      -                             /tidb-deploy/tidb-4000
  10.90.2.15:4000    tidb          10.90.2.15   4000/10080   linux/x86_64  Up      -                             /tidb-deploy/tidb-4000
  10.90.1.35:20160   tikv          10.90.1.35   20160/20180  linux/x86_64  Up      /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  10.90.2.138:20160  tikv          10.90.2.138  20160/20180  linux/x86_64  Up      /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  10.90.3.220:20160  tikv          10.90.3.220  20160/20180  linux/x86_64  Up      /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  Total nodes: 11
  ```

## Module 2: Start The Read And Write Workloads

### Step 1. Create a table for test purposes

+ In **Terminal 2**, connect to one of the TiDB server instances, create a testing table, and exit to the terminal.

  ```bash
  mysql -h ${HOST_DB1_PRIVATE_IP} -P 4000 -uroot
  ```
  ```sql
  DROP TABLE IF EXISTS test.dummy; 
  CREATE TABLE test.dummy (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(80), event JSON);
  EXIT;
  ```

### Step 2. Apply the INSERT workload

+ In **Terminal 2**, run the sample Java program to initiate the INSERT workload. 

+ The program connects to a TiDB server instance using two workers to execute INSERT operations. If the connected TiDB server loses connectivity, the program will automatically switch to another available TiDB server. The output displays the private IP addresses of the TiDB servers handling the INSERT requests — there should be two. Leave the workload running and proceed to the next step.

  ```bash
  cd scripts
  javac -cp .:misc/mysql-connector-java-5.1.36-bin.jar DemoJdbcEndlessInsertDummyCSP.java
  java -cp .:misc/mysql-connector-java-5.1.36-bin.jar DemoJdbcEndlessInsertDummyCSP
  ```
  ```output
  Worker 2 -> TiDB:Thu May 09 07:05:33 UTC 2024 at 10.90.1.116
  Worker 1 -> TiDB:Thu May 09 07:05:33 UTC 2024 at 10.90.2.15
  Worker 1 -> TiDB:Thu May 09 07:05:34 UTC 2024 at 10.90.1.116
  Worker 2 -> TiDB:Thu May 09 07:05:34 UTC 2024 at 10.90.2.15
  Worker 1 -> TiDB:Thu May 09 07:05:35 UTC 2024 at 10.90.2.15
  Worker 2 -> TiDB:Thu May 09 07:05:35 UTC 2024 at 10.90.1.116
  Worker 1 -> TiDB:Thu May 09 07:05:36 UTC 2024 at 10.90.1.116
  ...
  ```

### Step 3. Apply the sample QUERY workload

+ In **Terminal 3**, run the script below. The script attempts to connect to two TiDB server instances to execute queries. If one TiDB server fails or is unable to respond, the script automatically switches to another available instance. Note that the TiDB server instances are given only 1 second to respond to connection requests, a setting similar to the Java program in Step 2. The output will show the row count in the test.dummy table steadily increasing. Leave the workload running and proceed to the next step.

  ```bash
  qdb1(){
  mysql -h ${HOST_DB1_PRIVATE_IP} -P 4000 -uroot --connect-timeout 1 2>/dev/null << EOF
    SELECT COUNT(event) FROM test.dummy;
  EOF
  }

  qdb2(){
  mysql -h ${HOST_DB2_PRIVATE_IP} -P 4000 -uroot --connect-timeout 1 2>/dev/null << EOF
    SELECT COUNT(event) FROM test.dummy;
  EOF
  }

  query1(){
    echo;
    date;
    qdb1 || qdb2
    sleep 2;
  }

  query2(){
    echo;
    date;
    qdb2 || qdb1
    sleep 2;
  }

  while true; do
    query1;
    query2;
  done;
  ```
  ```output
  Mon May 13 05:27:33 UTC 2024
  COUNT(event)
  2608

  Mon May 13 05:27:35 UTC 2024
  COUNT(event)
  2612

  Mon May 13 05:27:37 UTC 2024
  COUNT(event)
  2616

  Mon May 13 05:27:39 UTC 2024
  COUNT(event)
  2620
  ...
  ```

### Step 4. Verify that the TiKV Region replicas setting is configured to three

+ Return to **Terminal 1**, and verify that the TiKV Region replicas setting is configured to three. This ensures that if one of the TiKV servers is lost due to a data center or Availability Zone failure, the remaining two TiKV servers can continue to operate normally.

  ```bash
  mysql -h ${HOST_DB1_PRIVATE_IP} -P 4000 -uroot
  ```
  ```sql
  SHOW CONFIG WHERE type='pd' AND name='replication.max-replicas';
  EXIT
  ```
  ```output
  +------+---------------------+--------------------------+-------+
  | Type | Instance            | Name                     | Value |
  +------+---------------------+--------------------------+-------+
  | pd   | 52.13.86.218:2379   | replication.max-replicas | 3     |
  | pd   | 34.212.25.193:2379  | replication.max-replicas | 3     |
  | pd   | 34.222.233.112:2379 | replication.max-replicas | 3     |
  +------+---------------------+--------------------------+-------+
  3 rows in set (0.02 sec)
  Bye
  ```

## Module 3: Isolate The Subnet 1 In Availability Zone 1 

### Step 1. Simulate a complete Availability Zone failure in Availability Zone 1

+ Return to **Terminal 1** and execute the script to simulate a complete Availability Zone failure by blocking subnet `demo-subnet-1` (subnet 10.90.1.0/24 in Availability Zone 1).

  ```bash
  VPC_ID=`aws ec2 describe-vpcs \
    --filters "Name=tag:Name,Values=user_unique_tag" \
    --query "Vpcs[0].VpcId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  SUBNET_ID=`aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-1" \
    --query "Subnets[0].SubnetId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  CAGE_NACL_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" "Name=tag:Name,Values=cage" \
    --query "NetworkAcls[0].NetworkAclId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  ASSOC_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \
    --query "NetworkAcls[0].Associations" \
    --output text \
    --region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'`
  aws ec2 replace-network-acl-association \
    --association-id ${ASSOC_ID} \
    --network-acl-id ${CAGE_NACL_ID} \
    --region ${LAB:REGION_CODE}
  ```
  ```output
  {
   "NewAssociationId": "aclassoc-0402a6103febf799f"
  }
  ```

### Step 2. Verify that several components are disconnected

+ In **Terminal 1**, check the cluster status again to verify that several components, including PD, TiDB and TiKV servers are disconnected/isolated because of the Availability Zone failure. 

+ In the sample output, the impacted servers are `10.90.1.208:2379   pd`, `10.90.1.247:4000   tidb`, and `10.90.1.122:20160  tikv`. Therefore, one-third of the cluster components have lost connection.

  ```bash
  tiup cluster display tidb-demo
  ```
  ```output
  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://52.13.86.218:2379/dashboard
  Grafana URL:        http://10.90.4.74:3000
  ID                 Role          Host         Ports        OS/Arch       Status        Data Dir                      Deploy Dir
  --                 ----          ----         -----        -------       ------        --------                      ----------
  10.90.4.74:9093    alertmanager  10.90.4.74   9093/9094    linux/x86_64  Up            /tidb-data/alertmanager-9093  /tidb-deploy/alertmanager-9093
  10.90.4.74:3000    grafana       10.90.4.74   3000         linux/x86_64  Up            -                             /tidb-deploy/grafana-3000
  10.90.1.208:2379   pd            10.90.1.208  2379/2380    linux/x86_64  Down          /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.2.39:2379    pd            10.90.2.39   2379/2380    linux/x86_64  Up|L          /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.3.68:2379    pd            10.90.3.68   2379/2380    linux/x86_64  Up            /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.4.74:9090    prometheus    10.90.4.74   9090/12020   linux/x86_64  Up            /tidb-data/prometheus-9090    /tidb-deploy/prometheus-9090
  10.90.1.247:4000   tidb          10.90.1.247  4000/10080   linux/x86_64  Down          -                             /tidb-deploy/tidb-4000
  10.90.2.33:4000    tidb          10.90.2.33   4000/10080   linux/x86_64  Up            -                             /tidb-deploy/tidb-4000
  10.90.1.122:20160  tikv          10.90.1.122  20160/20180  linux/x86_64  Disconnected  /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  10.90.2.5:20160    tikv          10.90.2.5    20160/20180  linux/x86_64  Up            /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  10.90.3.14:20160   tikv          10.90.3.14   20160/20180  linux/x86_64  Up            /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  Total nodes: 11
  ```

### Step 3. Verify that your INSERT and QUERY workloads are still running

+ In **Terminal 2** and **Terminal 3**, verify that your sample INSERT/QUERY workloads are still running. Note that the INSERT and QUERY workloads are working with only one TiDB server instance now.

  ```output
  Worker 1 -> TiDB:Mon May 13 05:51:52 UTC 2024 at 10.90.2.33
  Worker 2 -> TiDB:Mon May 13 05:51:52 UTC 2024 at 10.90.2.33
  Worker 1 -> TiDB:Mon May 13 05:51:53 UTC 2024 at 10.90.2.33
  Worker 2 -> TiDB:Mon May 13 05:51:53 UTC 2024 at 10.90.2.33
  Worker 1 -> TiDB:Mon May 13 05:51:54 UTC 2024 at 10.90.2.33
  ...
  ```
  ```output
  Mon May 13 05:51:51 UTC 2024
  COUNT(event)
  5481

  Mon May 13 05:51:54 UTC 2024
  COUNT(event)
  5485
  ...
  ```

### Step 4. Restore the connectivity for Availability Zone 1

+ In **Terminal 1**, run the scripts to restore connectivity for `demo-subnet-1`. In a real-world scenario, this step is typically unnecessary. If the issue in the Availability Zone is quickly resolved, the disconnected components automatically rejoin the cluster once the network is restored. If the problem persists due to prolonged Availability Zone or data center issues, you can mitigate the impact by scaling out new components in the remaining Availability Zones and evicting the disconnected ones. Regardless of the approach, TiDB ensures uninterrupted business operations.
For more advanced TiDB skills, visit [https://tidb.io/education/](https://tidb.io/education/).

  ```bash
  VPC_ID=`aws ec2 describe-vpcs \
    --filters "Name=tag:Name,Values=user_unique_tag" \
    --query "Vpcs[0].VpcId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  SUBNET_ID=`aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=demo-subnet-1" \
    --query "Subnets[0].SubnetId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  DEFAULT_NACL_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=true" \
    --query "NetworkAcls[0].NetworkAclId" \
    --output text \
    --region ${LAB:REGION_CODE}`
  ASSOC_ID=`aws ec2 describe-network-acls \
    --filters "Name=vpc-id,Values=${VPC_ID}" "Name=default,Values=false" \
    --query "NetworkAcls[0].Associations" \
    --output text \
    --region ${LAB:REGION_CODE} | grep ${SUBNET_ID} | awk -F" " '{print $1}'`
  aws ec2 replace-network-acl-association \
    --association-id ${ASSOC_ID} \
    --network-acl-id ${DEFAULT_NACL_ID} \
    --region ${LAB:REGION_CODE}
  ```
  ```output
  {
   "NewAssociationId": "aclassoc-0df1e05cfc70356a8"
  }
  ```

### Step 5. Verify that all cluster components are synchronized

+ Verify the cluster status to ensure that the previously disconnected components are now synchronized.

  ```bash
  tiup cluster display tidb-demo
  ```
  ```output
  Cluster type:       tidb
  Cluster name:       tidb-demo
  Cluster version:    v${LAB:TIDB_VERSION}
  Deploy user:        ec2-user
  SSH type:           builtin
  Dashboard URL:      http://52.13.86.218:2379/dashboard
  Grafana URL:        http://10.90.4.74:3000
  ID                 Role          Host         Ports        OS/Arch       Status  Data Dir                      Deploy Dir
  --                 ----          ----         -----        -------       ------  --------                      ----------
  10.90.4.74:9093    alertmanager  10.90.4.74   9093/9094    linux/x86_64  Up      /tidb-data/alertmanager-9093  /tidb-deploy/alertmanager-9093
  10.90.4.74:3000    grafana       10.90.4.74   3000         linux/x86_64  Up      -                             /tidb-deploy/grafana-3000
  10.90.1.208:2379   pd            10.90.1.208  2379/2380    linux/x86_64  Up      /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.2.39:2379    pd            10.90.2.39   2379/2380    linux/x86_64  Up|L    /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.3.68:2379    pd            10.90.3.68   2379/2380    linux/x86_64  Up      /tidb-data/pd-2379            /tidb-deploy/pd-2379
  10.90.4.74:9090    prometheus    10.90.4.74   9090/12020   linux/x86_64  Up      /tidb-data/prometheus-9090    /tidb-deploy/prometheus-9090
  10.90.1.247:4000   tidb          10.90.1.247  4000/10080   linux/x86_64  Up      -                             /tidb-deploy/tidb-4000
  10.90.2.33:4000    tidb          10.90.2.33   4000/10080   linux/x86_64  Up      -                             /tidb-deploy/tidb-4000
  10.90.1.122:20160  tikv          10.90.1.122  20160/20180  linux/x86_64  Up      /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  10.90.2.5:20160    tikv          10.90.2.5    20160/20180  linux/x86_64  Up      /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  10.90.3.14:20160   tikv          10.90.3.14   20160/20180  linux/x86_64  Up      /tidb-data/tikv-20160         /tidb-deploy/tikv-20160
  Total nodes: 11
  ```

+ Press <kbd>Ctrl</kbd>+<kbd>C</kbd> in **Terminal 2** and **Terminal 3** to stop both workloads.

## Conclusion

Thanks to TiDB's distributed architecture, failures at the data center or Availability Zone level don't need to trigger a server-side failover and let the database clients wait on the failover to complete. Instead, database clients can seamlessly connect to the available TiDB server instances, ensuring continuous business operations. Should such disasters occur, the application can set an operation timeout by their business requirements. Unlike traditional systems, TiDB doesn't require server-side failover to complete before the database client can connect.

For further information about TiDB training and certification, please visit [https://tidb.io/education/](https://tidb.io/education/).