# TiDB 零宕机在线升级与 TiProxy ## 概述 本实验展示了 TiDB 的无缝在线滚动升级能力，避免了对客户端应用程序的中断——这是传统数据库升级中常见的问题。TiDB 的云原生、松耦合架构允许逐个组件、逐个实例进行升级。TiProxy 增强了这一能力，通过将客户端会话和无状态 TiDB 服务器解耦。在 TiDB 服务器重启进行升级时，TiProxy 无缝地将会话迁移到活跃实例，确保在整个升级过程中客户端操作不中断。 ![80%](https://lab-static.pingcap.com/quick-demo/demo_005_en.png) ### 实验时长 + 本实验大约需 60 分钟完成。 ### 任务 + 查看 TiDB 集群状态和拓扑。 + 将 TiDB 集群从版本 7.5.4 升级到 8.1.1。 + 观察并验证 TiDB 集群在升级过程中能够保持功能并接受 DML 和查询操作。 > [!TIP] > 点击 **开始** 启动实验环境。 > 环境准备就绪后，点击底部工具栏中的 **前进** 进入 **模块 1**。 > 然后，按照实验指南的步骤连接到实践环境并开始你的任务。 ## 模块 1: 部署 TiDB 集群及 TiProxy ### 步骤 1. 登录到远程 Linux + 点击这里 - [终端 1](ssh://monitor1-ssh#1) 打开一个新的终端，连接到为你的实践实验分配的虚拟机。 + 登录后显示的横幅： ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### 步骤 2. 检查集群拓扑 YAML 文件 ```bash cat eleven-nodes.yaml ``` ```output global: user: "ec2-user" ssh_port: 22 deploy_dir: "/tidb-deploy" data_dir: "/tidb-data" arch: "amd64" server_configs: pd: replication.max-replicas: 3 tidb: security.auto-tls: true graceful-wait-before-shutdown: 10 pd_servers: - host: 10.90.3.160 client_port: 2379 peer_port: 2380 advertise_client_addr: "http://35.92.30.186:2379" - host: 10.90.1.162 client_port: 2379 peer_port: 2380 advertise_client_addr: "http://52.34.128.150:2379" - host: 10.90.2.223 client_port: 2379 peer_port: 2380 advertise_client_addr: "http://54.245.183.8:2379" tidb_servers: - host: 10.90.1.157 port: 4000 status_port: 10080 deploy_dir: /tidb-deploy/tidb-4000 log_dir: /tidb-deploy/tidb-4000/log - host: 10.90.2.6 port: 4000 status_port: 10080 deploy_dir: /tidb-deploy/tidb-4000 log_dir: /tidb-deploy/tidb-4000/log tikv_servers: - host: 10.90.3.132 port: 20160 status_port: 20180 - host: 10.90.2.171 port: 20160 status_port: 20180 - host: 10.90.1.222 port: 20160 status_port: 20180 monitoring_servers: - host: 10.90.4.33 grafana_servers: - host: 10.90.4.33 alertmanager_servers: - host: 10.90.4.33 tiproxy_servers: - host: 10.90.1.153 ``` ### 步骤 3. 修改系统设置以部署 TiDB 集群 ```bash tiup mirror set tidb-community-server-7.5.4-linux-amd64 tiup cluster check eleven-nodes.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-oregon.pem \ --apply ``` ```output ... + Try to apply changes to fix failed checks - Applying changes on 10.90.1.162 ... ⠧ Sysctl: host=10.90.1.162 vm.swappiness = 0 - Applying changes on 10.90.2.223 ... ⠧ Sysctl: host=10.90.2.223 vm.swappiness = 0 - Applying changes on 10.90.3.132 ... ⠧ Sysctl: host=10.90.3.132 vm.swappiness = 0 + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks - Applying changes on 10.90.1.162 ... Done - Applying changes on 10.90.2.223 ... Done + Try to apply changes to fix failed checks - Applying changes on 10.90.1.162 ... Done + Try to apply changes to fix failed checks + Try to apply changes to fix failed checks - Applying changes on 10.90.1.162 ... Done - Applying changes on 10.90.2.223 ... Done - Applying changes on 10.90.3.132 ... Done - Applying changes on 10.90.2.171 ... Done - Applying changes on 10.90.1.222 ... Done - Applying changes on 10.90.4.33 ... Done - Applying changes on 10.90.3.160 ... Done - Applying changes on 10.90.1.153 ... Done - Applying changes on 10.90.2.210 ... Done - Applying changes on 10.90.1.157 ... Done - Applying changes on 10.90.2.6 ... Done ``` ### 步骤 4. 部署 TiDB 集群 ```bash tiup cluster deploy tidb-demo 7.5.4 ./eleven-nodes.yaml \ --user ec2-user \ -i /home/ec2-user/.ssh/pe-class-key-oregon.pem \ --yes ``` ```output ... Cluster `tidb-demo` deployed successfully, you can start it with command: `tiup cluster start tidb-demo --init` ``` ### 步骤 5. 启动已部署的 TiDB 集群 ```bash tiup cluster start tidb-demo ``` ```output ... Started cluster `tidb-demo` successfully ``` ## 模块 2: 对 TiDB 集群应用读写工作负载 ### 步骤 1. 查看插入工作负载脚本 + 在 [终端 2](ssh://monitor1-ssh#2) 中，查看插入工作负载脚本。它包含一个 Java 程序，可以在出现任何 JDBC 异常时将消息打印到控制台。在最终输出中，您可能会注意到诸如 `Work ? discards connection and try to get a new connection` 的消息。需要注意的是，在演示期间，您不应遇到这些消息。TiProxy 可以在升级过程关闭 TiDB 服务器实例之前迁移数据库客户端会话，以确保不中断的操作。 ```bash cat scripts/quick-demo-005-jdbc-endless-insert-dummy.sh ``` ```output #!/bin/bash # Run ./quick-demo-005-jdbc-endless-insert-dummy.sh rm -f DemoJdbcEndlessInsertDummyCSPWithLBLongConn.class javac -cp .:misc/mysql-connector-java-5.1.36-bin.jar DemoJdbcEndlessInsertDummyCSPWithLBLongConn.java sleep 1 java -cp .:misc/mysql-connector-java-5.1.36-bin.jar DemoJdbcEndlessInsertDummyCSPWithLBLongConn ${1} ``` ```bash cat scripts/DemoJdbcEndlessInsertDummyCSPWithLBLongConn.java | grep -A10 catch | grep Work ``` ```output System.out.println("Work 1 discards connection and try to get a new connection"); System.out.println("Work 2 discards connection and try to get a new connection"); System.out.println("Work 3 discards connection and try to get a new connection"); System.out.println("Work 4 discards connection and try to get a new connection"); ``` ### 步骤 2. 运行脚本以应用插入负载 + 保持在 [终端 2](ssh://monitor1-ssh#2) 中，运行脚本以应用样本插入负载，使用 4 个工人。保持负载运行并进入下一步。 ```bash mysql -h ${HOST_TIPROXY1_PRIVATE_IP} -P 6000 -uroot << EOF DROP TABLE IF EXISTS test.dummy; CREATE TABLE test.dummy (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(80), event JSON, tidb_instance VARCHAR(30)); EOF cd scripts ./quick-demo-005-jdbc-endless-insert-dummy.sh ${HOST_TIPROXY1_PRIVATE_IP} ``` ```output Worker 3 -> TiDB:Thu May 16 09:38:55 UTC 2024 at 10.90.1.153 Worker 4 -> TiDB:Thu May 16 09:38:56 UTC 2024 at 10.90.1.153 Worker 2 -> TiDB:Thu May 16 09:38:56 UTC 2024 at 10.90.1.153 Worker 1 -> TiDB:Thu May 16 09:38:56 UTC 2024 at 10.90.1.153 Worker 3 -> TiDB:Thu May 16 09:38:56 UTC 2024 at 10.90.1.153 Worker 4 -> TiDB:Thu May 16 09:38:57 UTC 2024 at 10.90.1.153 Worker 2 -> TiDB:Thu May 16 09:38:57 UTC 2024 at 10.90.1.153 Worker 1 -> TiDB:Thu May 16 09:38:57 UTC 2024 at 10.90.1.153 Worker 3 -> TiDB:Thu May 16 09:38:57 UTC 2024 at 10.90.1.153 Worker 4 -> TiDB:Thu May 16 09:38:58 UTC 2024 at 10.90.1.153 Worker 2 -> TiDB:Thu May 16 09:38:58 UTC 2024 at 10.90.1.153 ... ``` ### 步骤 3. 运行脚本以应用查询负载 + 在 [终端 3](ssh://monitor1-ssh#3)，运行负责查询负载的脚本，它会检查每个 TiDB 服务器实例处理的请求数量。保持负载运行并进入下一步。 ```bash qdb(){ mysql -h ${HOST_TIPROXY1_PRIVATE_IP} -P 6000 -uroot --connect-timeout 1 2>/dev/null << EOF SELECT tidb_instance, COUNT(event) FROM test.dummy GROUP BY tidb_instance ORDER BY tidb_instance; EOF } query(){ echo; date; qdb sleep 2; } while true; do query; done; ``` ```output Thu May 16 09:42:20 UTC 2024 tidb_instance COUNT(event) 10.90.1.124:10080 466 10.90.2.122:10080 466 Thu May 16 09:42:22 UTC 2024 tidb_instance COUNT(event) 10.90.1.124:10080 470 10.90.2.122:10080 470 Thu May 16 09:42:24 UTC 2024 tidb_instance COUNT(event) 10.90.1.124:10080 474 10.90.2.122:10080 474 ... ``` ## 模块 3: 在负载运行时升级 TiDB 集群 ### 步骤 1. 设置 TiUP 镜像位置 + 返回 [终端 1](ssh://monitor1-ssh#1)，运行 tiup 命令更改 TiUP 本地镜像至 v8.1.1。 ```bash tiup mirror set tidb-community-server-8.1.1-linux-amd64 ``` ```output Successfully set mirror to /home/ec2-user/tidb-community-server-8.1.1-linux-amd64 ``` ### 步骤 2. 升级你的 TiDB 集群 + 运行 `tiup` 命令将 TiDB 集群从 v7.5.4 升级至 v8.1.1。在升级进行时，切换到 [终端 2](ssh://monitor1-ssh#2) 和 [终端 3](ssh://monitor1-ssh#3)。 ```bash tiup cluster upgrade tidb-demo 8.1.1 --yes ``` ```output Before the upgrade, it is recommended to read the upgrade guide at https://docs.tidb.io/tidb/stable/upgrade-tidb-using-tiup and finish the preparation steps. This operation will upgrade tidb v7.5.4 cluster tidb-demo to v8.1.1: will upgrade and restart component " pd" to "v8.1.1", will upgrade and restart component " tikv" to "v8.1.1", will upgrade and restart component " tidb" to "v8.1.1", ... + [ Serial ] - Download: component=grafana, version=v8.1.1, os=linux, arch=amd64 + [ Serial ] - Download: component=tikv, version=v8.1.1, os=linux, arch=amd64 + [ Serial ] - Download: component=tidb, version=v8.1.1, os=linux, arch=amd64 + [ Serial ] - Download: component=prometheus, version=v8.1.1, os=linux, arch=amd64 + [ Serial ] - Download: component=pd, version=v8.1.1, os=linux, arch=amd64 ... Upgrading component pd Restarting instance 10.90.1.135:2379 Restart instance 10.90.1.135:2379 success Restarting instance 10.90.2.182:2379 Restart instance 10.90.2.182:2379 success Restarting instance 10.90.3.231:2379 Restart instance 10.90.3.231:2379 success Upgrading component tikv Evicting 11 leaders from store 10.90.1.55:20160... Still waiting for 11 store leaders to transfer... Still waiting for 11 store leaders to transfer... ... Upgrading component tidb Restarting instance 10.90.1.124:4000 Restart instance 10.90.1.124:4000 success Restarting instance 10.90.2.122:4000 Restart instance 10.90.2.122:4000 success ... ``` ### 步骤 3. 验证插入与查询负载未中断 + 观察 [终端 2](ssh://monitor1-ssh#2) 和 [终端 3](ssh://monitor1-ssh#3)，确保示例应用程序连接和事务在升级期间从未中断。 + 等待 [终端 1](ssh://monitor1-ssh#1) 中的升级命令完成，再进行下一步。 ```output ... Upgraded cluster `tidb-demo` successfully ``` ### 步骤 4. 验证升级结果 + 运行命令获取集群状态并确认集群版本是 8.1.1。 ```bash tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v8.1.1 Deploy user: ec2-user SSH type: builtin Dashboard URL: http://18.138.239.33:2379/dashboard Grafana URL: http://10.90.4.8:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.8:9093 alertmanager 10.90.4.8 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.8:3000 grafana 10.90.4.8 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.152:2379 pd 10.90.1.152 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.73:2379 pd 10.90.2.73 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.163:2379 pd 10.90.3.163 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.8:9090 prometheus 10.90.4.8 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.2.180:4000 tidb 10.90.2.180 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.89:4000 tidb 10.90.3.89 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.88:20160 tikv 10.90.1.88 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.20:20160 tikv 10.90.2.20 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.69:20160 tikv 10.90.3.69 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.60:6000 tiproxy 10.90.1.60 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 12 ``` + 按 Ctrl+C 停止 [终端 2](ssh://monitor1-ssh#2) 和 [终端 3](ssh://monitor1-ssh#3) 中的工作负载。 ## 结论 由于 TiDB 服务器处理 SQL 层的连接，在线升级 TiDB 服务器比升级 TiKV 和 PD 更具挑战性。此次练习证明了在 TiProxy 的帮助下，此类计划内停机可以被规避。在升级 TiDB 服务器之前，服务器会从 Placement Driver 获取信息以识别要升级的服务器。在此之前，TiProxy 会将所有会话从受影响的服务器迁移到其他 TiDB 服务器。因此逐一升级每个节点，直到所有 TiDB 服务器都完成升级。TiDB 集群的升级在不中断客户端会话的情况下完成。数据库停机时间和应用程序停机时间均为零。如果您使用 TiDB Cloud，这些功能，包括 TiProxy 的会话管理，开箱即用。在 TiDB Cloud 中，您可以请求 PingCAP 为您的 TiDB 集群启用或禁用 TiProxy。 在生产环境中，您甚至可以在多个 TiProxy 实例前面放置一个负载均衡器，以平衡 TiProxy 层的工作负载。 欲了解更多 TiDB 培训与认证信息，请访问 [https://tidb.io/education/](https://tidb.io/education/)。