# 体验 TiDB 的零停机时间
### 实验概述
+ 在本实验中，您将体验在计划内的缩容和扩容操作，以及计划外的系统故障情况中 TiDB 的零停机能力。
+ 在练习 1 中，您将体验到，在数据库 SQL 层（TiDB server）实例发生故障的情况下，您的业务依然能够保持运行。
+ 在练习 2 中，您将体验到，在数据库存储层（TiKV server）实例故障的情况下，您的业务依然能够保持运行。
+ 在练习 3 中，您将体验到，即使数据库元数据层（Placement Driver）实例发生故障，您的业务也不会受到影响。
### 注意事项
+ 有些实验步骤需要您在两个不同的终端窗口中输入命令。您可能会发现，启动两个终端并在实验练习期间保持开启是很方便的。
+ 您的练习环境可能与本实验不同，包括但不限于 `IP地址`、`端口号`、`用户名`、`密码`、`目录`和练习脚本的估计运行时间。
+ 在您的终端窗口中，`$` 是操作系统命令提示符。
+ 在您终端窗口中的 `tidb>` 或 `mysql>` 是 SQL 命令提示符。
+ 这些练习可以在一台单独的机器上运行，比如您的笔记本电脑。
### 前提条件
+ 您的操作系统是 macOS 或 Linux。
+ 你已经克隆了[实验存储库](https://github.com/pingcap/tidb-course-201-lab.git)
+ 您已经完成练习 - 创建一个 TiDB Playground 集群
+ 你已安装了[JDK 8 或更高版本](https://docs.oracle.com/javase/8/docs/technotes/guides/install/install_overview.html)
## 练习 1：SQL 层实例故障转移演示
### 预计完成时间
+ 完成这个练习大约需要您花费 15 分钟。
### 实验任务
1. 在 `test` 库中创建一个表。
2. 初始化 Java 程序，在创建的表中插入行。
3. 在 SQL 层上进行扩容和缩容。
4. 在 TiDB Playground 集群中，故意终止一个 SQL 层（TiDB server）实例。
5. 验证 Java 程序不受影响。
6. 将健康的 SQL Layer 实例返回(向外扩展)到 TiDB Playground 集群。
### 实验步骤
1. 以用户 `root` 登录，并确保 Region 的最大副本数设置为3。
```
$ cd tidb-course-201-lab/scripts/
$ ./connect-4000.sh
```
```sql
SET CONFIG PD `replication.max-replicas`=3;
SELECT SLEEP(1);
SHOW CONFIG WHERE type='pd' AND name='replication.max-replicas';
```
```
tidb:4000> SET CONFIG PD `replication.max-replicas`=3;
Query OK, 0 rows affected (0.47 sec)
tidb:4000> SELECT SLEEP(1);
+----------+
| SLEEP(1) |
+----------+
| 0 |
+----------+
1 row in set (1.01 sec)
tidb:4000> SHOW CONFIG WHERE type='pd' AND name='replication.max-replicas';
+------+----------------+--------------------------+-------+
| Type | Instance | Name | Value |
+------+----------------+--------------------------+-------+
| pd | 127.0.0.1:5380 | replication.max-replicas | 3 |
| pd | 127.0.0.1:5379 | replication.max-replicas | 3 |
| pd | 127.0.0.1:5381 | replication.max-replicas | 3 |
+------+----------------+--------------------------+-------+
3 rows in set (0.02 sec)
```
2. ，打开一个新终端窗口并执行脚本 `playground-check.sh` 来查看 TiDB Playground 集群的状态。请务必关注 TiDB server 实例的状态。
```
$ ./playground-check.sh
...
Pid Role Uptime
--- ---- ------
...
2760 tidb 33m21.006866672s
2761 tidb 33m20.948299789s
...
```
3. 通过执行 `02-demo-jdbc-endless-insert-dummy.sh` 来初始化工作负载。该脚本调用一个虚拟 Java 程序，该程序模拟假设的单击流事件。该程序由两个并发线程组成，即 'worker1' 和 'worker2'，它们都可以循环连接到 TiDB Playground 集群中的两个 TiDB server 实例。需要注意的是，没有为这个 Java 程序创建任何表，因此，它不能在 TiDB server (端口 4000 和端口 4001 )上成功执行任何任务。
请查看脚本。
```
$ cat 02-demo-jdbc-endless-insert-dummy.sh
#!/bin/bash
# Run ./02-demo-jdbc-endless-insert-dummy.sh
rm -f DemoJdbcEndlessInsertDummyV3.class
javac -cp .:misc/mysql-connector-java-5.1.36-bin.jar DemoJdbcEndlessInsertDummyV3.java
sleep 1
java -cp .:misc/mysql-connector-java-5.1.36-bin.jar DemoJdbcEndlessInsertDummyV3 $*
```
执行工作负载脚本，并确保此终端窗口在后续练习中始终保持开启。
```
$ ./02-demo-jdbc-endless-insert-dummy.sh
Error - Worker 2 - TiDB Server (port:4001) Table 'test.dummy' doesn't exist at Sat Apr 15 20:43:27 CST 2023
Error - Worker 1 - TiDB Server (port:4000) Table 'test.dummy' doesn't exist at Sat Apr 15 20:43:27 CST 2023
Error - Worker 1 - TiDB Server (port:4001) Table 'test.dummy' doesn't exist at Sat Apr 15 20:43:27 CST 2023
Error - Worker 2 - TiDB Server (port:4000) Table 'test.dummy' doesn't exist at Sat Apr 15 20:43:27 CST 2023
Error - Worker 1 - TiDB Server (port:4000) Table 'test.dummy' doesn't exist at Sat Apr 15 20:43:27 CST 2023
Error - Worker 2 - TiDB Server (port:4001) Table 'test.dummy' doesn't exist at Sat Apr 15 20:43:28 CST 2023
Error - Worker 2 - TiDB Server (port:4000) Table 'test.dummy' doesn't exist at Sat Apr 15 20:43:28 CST 2023
...
```
4. 返回 MySQL CLI，在 `test` 库中，创建一个名为 `dummy` 的表。（TiDB 也支持 `JSON` 数据类型。）
```sql
DROP TABLE IF EXISTS test.dummy;
CREATE TABLE test.dummy (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(80), event JSON);
```
```
tidb:4000> DROP TABLE IF EXISTS test.dummy;
Query OK, 0 rows affected, 1 warning (0.00 sec)
tidb:4000> CREATE TABLE test.dummy (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(80), event JSON);
Query OK, 0 rows affected (0.59 sec)
```
5. 验证该程序是否在 Java 程序终端窗口上正常运行。请注意，在示例 Java 程序中，短暂的连接用于演示目的，并不推荐对于基于 JDBC 的应用程序使用这种方法。
```
Error - Worker 2 - TiDB Server (port:4000) Table 'test.dummy' doesn't exist at Sat Apr 15 20:52:17 CST 2023
Error - Worker 2 - TiDB Server (port:4001) Table 'test.dummy' doesn't exist at Sat Apr 15 20:52:17 CST 2023
Error - Worker 1 - TiDB Server (port:4000) Table 'test.dummy' doesn't exist at Sat Apr 15 20:52:17 CST 2023
Error - Worker 1 - TiDB Server (port:4001) Table 'test.dummy' doesn't exist at Sat Apr 15 20:52:17 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 20:52:18 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 20:52:18 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 20:52:18 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 20:52:19 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 20:52:19 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 20:52:19 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 20:52:19 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 20:52:20 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 20:52:20 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 20:52:20 CST 2023
```
6. 回到 MySQL CLI，检验 `test.dummy` 表中的行数是否增加。
```sql
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
SELECT SLEEP(1);
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
```
```
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 30676 | 683 |
| worker2 | 30677 | 672 |
+---------+---------+--------------+
2 rows in set (0.01 sec)
tidb:4000> SELECT SLEEP(1);
+----------+
| SLEEP(1) |
+----------+
| 0 |
+----------+
1 row in set (1.00 sec)
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 30679 | 687 |
| worker2 | 30680 | 674 |
+---------+---------+--------------+
2 rows in set (0.01 sec)
```
7. 打开一个新的终端窗口，并运行脚本 `02-1-planned-scale-in-tidb-server.sh`，通过移除一个 TiDB server 实例来缩容 SQL 层。假设您已经确定没有必要维护两台 server，那么单个 TiDB server 可以有效地管理所有工作负载。
仔细查看缩容脚本。
```
$ cd tidb-courses-201-lab/scripts/
$ cat 02-1-planned-scale-in-tidb-server.sh
#!/bin/bash
# Scale-in the first TiDB server (its port should be 4000)
TIDB_SERVER_1_PID=`~/.tiup/bin/tiup playground display 2>/dev/null | grep -i tidb | sort -g | head -n 1 | awk -F" " '{print $1}'`
~/.tiup/bin/tiup playground scale-in --pid ${TIDB_SERVER_1_PID}
~/.tiup/bin/tiup playground display
```
执行缩容脚本。
```
$ ./02-1-planned-scale-in-tidb-server.sh
...
Pid Role Uptime
--- ---- ------
...
2761 tidb 1h29m16.989845496s
...
```
8. 观察运行 Java 程序的终端窗口，可以注意到 `worker1` 和 `worker2` 现在都运行在 `TiDB Server (port:4001)` 上。
```
...
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:49 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:50 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:51 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:06:52 CST 2023
...
```
9. 通过端口 `4001` 连接到剩余的 TiDB server 实例，检验 `test.dummy` 表中的行数是否仍在增加，工作负载是否仍在运行。 一个有趣结果是 `MAX(id)` 列的返回值可能在两个查询之间相同，而 `COUNT(event)` 的值则在增加。然而，无论结果是怎样，不会受影响到后续的练习。关于 TiDB 在分布式环境中的 `AUTO_INCREMENT` 的更多信息，请参加免费的 Workshop- [TiDB Workshop for MySQL Users](https://pingcap.com/education)以了解更多。
```
$ ./connect-4001.sh
```
```sql
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
SELECT SLEEP(1);
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
```
```
tidb:4001> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 33009 | 2359 |
| worker2 | 33010 | 2352 |
+---------+---------+--------------+
2 rows in set (0.05 sec)
tidb:4001> SELECT SLEEP(1);
+----------+
| SLEEP(1) |
+----------+
| 0 |
+----------+
1 row in set (1.00 sec)
tidb:4001> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 33014 | 2362 |
| worker2 | 33013 | 2353 |
+---------+---------+--------------+
2 rows in set (0.01 sec)
```
10. 目前集群的状态是怎样的? 有潜在的风险吗? 如果工作负载增加或单个 TiDB server 无法满足高可用性需求，那么仅依靠单个 server 是无法处理工作负载并满足所需的高可用性 SLA 的。为了解决这个问题，应该向集群中添加一个额外的 TiDB server。在实际操作中，应该根据系统的SQL工作负载确定所需的 TiDB server 实例数量。
查看扩容脚本。
```
$ cat ./02-2-planned-scale-out-tidb-server.sh
#!/bin/bash
# Scale-out one TiDB server
~/.tiup/bin/tiup playground scale-out --db 1
~/.tiup/bin/tiup playground display
```
执行此脚本，通过添加一个 TiDB server 来扩展 SQL 层。
```
$ ./02-2-planned-scale-out-tidb-server.sh
...
Pid Role Uptime
--- ---- ------
...
2761 tidb 1h43m49.828035701s
7192 tidb 6.022653957s
...
```
11. 查看运行 Java 程序的终端窗口，可以注意到 `worker1` 和 `worker2` 再次运行在 `TiDB Server (port:4000)` 和 `TiDB Server (port:4001)` 上。
```
...
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 21:46:25 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:46:26 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:46:26 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 21:46:27 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 21:46:27 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:46:27 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 21:46:27 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:46:27 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:46:28 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 21:46:28 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 21:46:28 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:46:29 CST 2023
```
12. 到目前为止，我们已经练习了 SQL 层的计划内和计划外的缩容和扩容。现在，执行脚本 `02-3-unplanned-destroy-tidb-server.sh` 来模拟 TiDB server 系统的意外故障。
查看故障制造脚本。
```
$ cat 02-3-unplanned-destroy-tidb-server.sh
#!/bin/bash
# Kill one TiDB server instance.
TIDB_SERVER_1_PID=`~/.tiup/bin/tiup playground display 2>/dev/null | grep -i tidb | sort -g -r | head -n 1 | awk -F" " '{print $1}'`
kill -9 ${TIDB_SERVER_1_PID}
~/.tiup/bin/tiup playground display
```
执行该脚本来强制终止 TiDB server 的单个实例。
```
$ ./02-3-unplanned-destroy-tidb-server.sh
Pid Role Uptime
--- ---- ------
...
2761 tidb 2h13m30.567709922s
7192 tidb signal: killed
```
13. 在验证过 `worker1` 和 `worker2` 仅在单个 TiDB server 上仍可以工作后，请继续确认 `test.dummy` 表的行数仍在增加，程序仍在运行。
查看 Java 程序的终端窗口。
```
...
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:53:57 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:53:58 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:53:58 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:53:58 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 21:53:58 CST 2023
```
```sql
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
SELECT SLEEP(1);
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
```
```
tidb:4001> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 63431 | 7351 |
| worker2 | 63432 | 7338 |
+---------+---------+--------------+
2 rows in set (0.05 sec)
tidb:4001> SELECT SLEEP(1);
+----------+
| SLEEP(1) |
+----------+
| 0 |
+----------+
1 row in set (1.00 sec)
tidb:4001> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 63431 | 7355 |
| worker2 | 63432 | 7341 |
+---------+---------+--------------+
2 rows in set (0.03 sec)
```
14. 执行 `02-4-recover-capacity-tidb-server.sh` 脚本清理失效的 TiDB server，并将一个 TiDB server 添加回集群，将 SQL 层 TiDB server 的数量恢复为 2。
查看脚本
```
$ cat 02-4-recover-capacity-tidb-server.sh
#!/bin/bash
# Clean killed TiDB servers, should be only 1 TiDB server need to be cleaned as per the exercise workflow. However, this routine will clean all killed TiDB servers.
for PID in `~/.tiup/bin/tiup playground display 2>/dev/null | egrep -i '(killed|exit)' | grep -i tidb | sort -g -r | awk -F" " '{print $1}'`; do
~/.tiup/bin/tiup playground scale-in --pid ${PID}
done;
sleep 3;
# Scale-out 1 TiDB server
~/.tiup/bin/tiup playground scale-out --db 1
~/.tiup/bin/tiup playground display
```
执行脚本。
```
$ ./02-4-recover-capacity-tidb-server.sh
...
To connect new added TiDB: mysql --comments --host 127.0.0.1 --port 4000 -u root -p (no password)
...
Pid Role Uptime
--- ---- ------
...
2761 tidb 2h29m58.446768313s
9234 tidb 6.138662531s
...
```
15. 检验 `worker1` 和 `worker2` 是否在两个 TiDB server 上正常运行。而且 `test.dummy` 表中的行数仍在增加，程序仍在运行。
```
...
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:30:11 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 22:30:11 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 22:30:11 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:30:11 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 22:30:12 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:30:12 CST 2023
```
```sql
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
SELECT SLEEP(1);
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
```
```
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 90482 | 9145 |
| worker2 | 90481 | 9133 |
+---------+---------+--------------+
2 rows in set (0.02 sec)
tidb:4000> SELECT SLEEP(1);
+----------+
| SLEEP(1) |
+----------+
| 0 |
+----------+
1 row in set (1.00 sec)
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 90484 | 9146 |
| worker2 | 90483 | 9135 |
+---------+---------+--------------+
2 rows in set (0.02 sec)
```
16. 同样，由于它实现了重新连接逻辑，因此对 Java 程序没有明显的负面影响。到目前为止，我们体验了 “计划中的” SQL 层缩容后在扩容；“计划外” SQL 层实例故障（强制缩容）后在扩容。在实际生产环境中，TiDB servers 通常部署在负载均衡器后面，以避免客户端需要了解每个 TiDB server 的具体连接细节。更进一步，PingCAP 提供了[TiProxy](https://github.com/pingcap/TiProxy)，它确保连接和服务发现的持久性，允许无缝处理计划外的中断和 TiDB server 的计划扩展，而无需断开客户端。此外，负载均衡器和 TiProxy 在[TiDB Cloud Dedicated and Serverless](https://tidbcloud.com/free-trial/?utm_source=edu&utm_medium=referral&utm_campaign=201.1.E2) 中预配置。通过提供的[链接](https://www.pingcap.com/demo/?utm_source=edu&utm_medium=referral&utm_campaign=201.1.E2)，您可以从 PingCAP 专业人员那里获得更详细的信息。
## 练习 2：存储层实例故障切换演示
### 预计完成时间
+ 完成这个练习大约需要花费您 10 分钟。
### 实验任务
1. 强制终止一个 TiKV server。
2. 验证 Java 程序不受影响。
### 实验步骤
1. 执行脚本 `playground-check.sh` 来查看 TiDB Playground 集群状态。正常情况下应该有 3 个 TiKV server。
```
$ ./playground-check.sh
...
Pid Role Uptime
--- ---- ------
2754 pd 2h47m33.055713498s
2755 pd 2h47m32.998182468s
2756 pd 2h47m32.932384756s
2757 tikv 2h47m32.863143492s
2758 tikv 2h47m32.79121259s
2759 tikv 2h47m32.731007068s
2761 tidb 2h47m32.607392002s
9234 tidb 17m40.299286264s
2794 tiflash 2h46m35.83098919s
```
2. 在 TiDB 中，存储层（TiKV server）其对应的 SQL 层一样，都支持扩容和缩容操作。此外，SQL 层使数据库客户端应用程序的存储层对水平扩展和系统故障转移恢复无感知成为可能。执行 `02-6-unplanned-destroy-tikv-server.sh` 脚本来模拟 TiKV 节点失效情况。
运行以下脚本来强制终止一个 TiKV servers。
```
$ ./02-6-unplanned-destroy-tikv-server.sh
...
Pid Role Uptime
--- ---- ------
...
2757 tikv signal: killed
2758 tikv 2h54m57.742433044s
2759 tikv 2h54m57.682216051s
...
```
3. 观察到程序没有影响。
`worker1` 和 `worker2` 正在 TiDB server 上运行。如果您刚刚终止的 TiKV server 恰好是程序当前插入数据 Region 的 Raft Leader，那么您可能会遇到到一个几秒的暂停。
```
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:51:38 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:51:38 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 22:51:38 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 22:51:39 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:51:39 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:51:39 CST 2023
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 22:51:40 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 22:51:40 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 22:51:40 CST 2023
```
检验表 `test.dummy` 中的行是否可以被填充。您可以使用 `connect-4000.sh` 或 `connect-4001.sh` 其中一个来建立连接（两者都是有效选项）。
```sql
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
tidb:4000> SELECT SLEEP(1);
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
```
```
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 93341 | 12047 |
| worker2 | 93342 | 11950 |
+---------+---------+--------------+
2 rows in set (0.12 sec)
tidb:4000> SELECT SLEEP(1);
+----------+
| SLEEP(1) |
+----------+
| 0 |
+----------+
1 row in set (1.00 sec)
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 93345 | 12050 |
| worker2 | 93344 | 11952 |
+---------+---------+--------------+
2 rows in set (0.03 sec)
```
4. 评估当前状态。有 3 个副本分布在 2 个TiKV server 上。是否存在潜在风险？尽管业务尚未中断，但由于 `replication.max-replicas` 的值为 3，TiDB Playground 集群无法承受再丢失一个 TiKV server，否则将导致数据丢失，存储层需要添加一个新 server 以达到至少 3 个server。然而，需要一个真正的 TiDB 集群来全面展示存储层的管理，这超出了本课程的范围，也需要额外的技能。因此，建议参加由讲师指导的培训 - [TiDB 管理](https://www.pingcap.com/education/)，以获得在管理 TiDB 集群方面的高级技能和知识。此外，还有另一种推荐的方法，即使用[TiDB Cloud Serverless or Dedicated](https://tidbcloud.com/free-trial/?utm_source=edu&utm_medium=referral&utm_campaign=201.1.E2)，这是一种在云端完全托管的 TiDB 服务。您可以访问[play.tidbcloud.com](https://play.tidbcloud.com/?utm_source=edu&utm_medium=referral&utm_campaign=201.1.E2)了解 TiDB Cloud Serverless 提供的所有优势。
5. 执行 `02-6-discard-failed-tikv-server.sh` 来清理失效的 TiKV 存储。
```
$ ./02-6-discard-failed-tikv-server.sh
...
tikv will be stop when tombstone
```
6. 目前为止，在 Java 程序上没有任何影响和数据丢失。我们经历了一个“计划外”的 TiKV 实例失败（强制缩容）。再次强调，在生产环境中不应使用 Playground 集群。
## 练习 3：元数据层实例故障转移展示
### 预计完成时间
+ 完成这个练习大约需要花费您 10 分钟。
### 实验任务
1. 强制终止一个 Placement Drivers，来模拟系统故障。
2. 验证系统故障对 Java 程序是无感知的。
3. 通过添加一个新实例来替换失效的实例，将 Placement Drivers 实例的数量恢复为 3。
### 实验步骤
1. 您可以执行 `02-7-unplanned-destroy-pd-server.sh` 脚本来模拟一个 Placement Driver 实例的故障，该脚本将活跃的 Placement Driver 实例从 3 个减少到 2 个。需要注意的是，在之前的练习 2-2 中，一个 TiKV server 已经宕机了，可以通过再故障一个 Placement Driver server，来模拟同时丢失一个 Placement Driver 服务器和一个 TiKV 服务器导致的可用区(数据中心)级别的故障。幸运的是，即使在这种情况下，只要每个部分的大多数组件保持可操作，TiDB 就能够继续无缝地为业务提供服务。可能会有最多几秒钟的轻微停顿(如果灾难影响到 Leader)，而云提供商可能需要几分钟或几小时来解决问题。
```
$ ./02-7-unplanned-destroy-pd-server.sh
...
Pid Role Uptime
--- ---- ------
2754 pd signal: killed
2755 pd 3h25m38.538802911s
2756 pd 3h25m38.472993959s
2757 tikv signal: killed
2758 tikv 3h25m38.331398029s
2759 tikv 3h25m38.271188632s
2761 tidb 3h25m38.147573414s
9234 tidb 55m45.839467817s
2794 tiflash 3h24m41.37116303s
```
2. 检查并确认 Java 程序不受影响。重要的是要注意，如果您恰好在前一步中终止了 Leader Placement Driver，程序可能会经历几秒钟的轻微停顿，这表示故障转移的开始。示例程序涉及许多小事务的快速执行，这需要向 PD leader 发出大量 TSO 请求。参加免费课程[TiDB 数据库核心原理与架构](https://learn.pingcap.com/learner/course/960001)该课程提供了深入的内容可以帮助您全面了解主要组件，如 Placement Driver、TiKV server 和 TiDB server 以及 TSO。
检验表 `test.dummy` 中的行是否正在填充。您可以使用 `connect-4000.sh` 或 `connect-4001.sh` 建立连接 （两者都是有效选项）。
```sql
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
SELECT SLEEP(1);
SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
```
```
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 97242 | 15948 |
| worker2 | 97241 | 15849 |
+---------+---------+--------------+
2 rows in set (0.11 sec)
tidb:4000> SELECT SLEEP(1);
+----------+
| SLEEP(1) |
+----------+
| 0 |
+----------+
1 row in set (1.00 sec)
tidb:4000> SELECT name, MAX(id), COUNT(event) FROM test.dummy GROUP BY name ORDER BY name;
+---------+---------+--------------+
| name | MAX(id) | COUNT(event) |
+---------+---------+--------------+
| worker1 | 97243 | 15951 |
| worker2 | 97244 | 15850 |
+---------+---------+--------------+
2 rows in set (0.03 sec)
```
3. 目前有一个 Leader 和两个 Follower 分布在两个 Placement Driver 实例中。如果另一个 Placement Driver 实例失效，Raft 共识算法将被破坏。只需执行 `02-8-recover-capacity-pd-server.sh` 脚本，将活跃的 Placement Driver 实例数量恢复到 3 个。
```
$ ./02-8-recover-capacity-pd-server.sh
...
Pid Role Uptime
--- ---- ------
2755 pd 3h40m11.836368008s
2756 pd 3h40m11.770562606s
12310 pd 2.227006522s
...
```
4. 按下 `Ctrl-C` 来终止 Java 程序。
```
Worker 2 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 23:36:43 CST 2023
Worker 2 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 23:36:43 CST 2023
Worker 1 - TiDB Server (port:4000) - INSERTING INTO test.dummy at Sat Apr 15 23:36:44 CST 2023
Worker 1 - TiDB Server (port:4001) - INSERTING INTO test.dummy at Sat Apr 15 23:36:44 CST 2023
^C
```
4. 终止 TiDB Playground 集群并清除所有数据。
在之前运行了 `playground-start-demo.sh` 的终端窗口中，按下`Ctrl-C`。
```
^CPlayground receive signal: interrupt
...
```
清理 Playground 数据。
```
$ ~/.tiup/bin/tiup clean demo
Clean instance of `playground`, directory: /Users/username/.tiup/data/demo
```
## 总结
TiDB 是经过精心设计，旨在消除计划内和计划外的停机时间。它支持组件层级的在线扩容和缩容，包括在线滚动升级和 online DDL。它内置的自我修复功能应用在计算、存储和元数据层面，确保即使面临软件、硬件故障或可用区(数据中心)宕机，业务也可以持续运行。此外，用户可以在[TiDB Cloud](https://tidbcloud.com/free-trial/?utm_source=edu&utm_medium=referral&utm_campaign=201.1.E2)上体验这些优点而无需额外设置。参加免费课程[TiDB Workshop for MySQL Users](https://www.pingcap.com/education/)和由讲师带领的培训[TiDB Administration](https://www.pingcap.com/education/)，学习在线滚动升级、online DDL、TiDB 集群上的灾难恢复部署等内容。
## 结束练习