## 模块 2：使用网络负载均衡器时，TiDB 缩容的应用最佳实践

本模块将介绍在使用网络负载均衡器（NLB）时，TiDB Server 缩容场景下的应用最佳实践。

下图展示了当前 TiDB 集群的拓扑结构。带红色虚线的 TiDB Server 节点即将被缩容。

![demo016-m7-tapo-nlb_aa_002_20250821.png 85%](https://lab-static.pingcap.com/images/2025/9/5/3fac225ddd21a2a170c5a725da8590ffb3a465d3.png)

### 步骤 1. 将 TiDB 节点注册到网络负载均衡器

  ```bash
  source ./hosts-env.sh
  source ./cloud-env.sh

  LB_ARN=`aws elbv2 describe-load-balancers --names ${LAB:USER_UNIQUE_TAG} --query "LoadBalancers[0].LoadBalancerArn" --output text --region ${LAB:REGION_CODE}`

  DEMO_TG_ARN=`aws elbv2 describe-target-groups --load-balancer-arn ${LB_ARN} --query "TargetGroups[0].TargetGroupArn" --output text --region ${LAB:REGION_CODE}`

  aws elbv2 register-targets --target-group-arn ${DEMO_TG_ARN} --targets "Id=${HOST_DB1_PRIVATE_IP},Port=4000" "Id=${HOST_DB2_PRIVATE_IP},Port=4000" --region ${LAB:REGION_CODE}

  sleep 60;
  echo TiDB instances registered to NLB target group ${DEMO_TG_ARN}}
  ```

  ```output
  TiDB instances registered to NLB target group arn:aws:elasticloadbalancing:us-west-2:xxxxxxxxxxxx:targetgroup/xxxxxx-TG-Z2TREFSV0YSJ/8efa9967b824f524
  ```

### 步骤 2. 检查目标组中 TiDB 节点的健康状态

  ```bash
  aws elbv2 describe-target-health --region ${LAB:REGION_CODE} --target-group-arn ${DEMO_TG_ARN} --output table
  ```

  ```output
  -------------------------------------------------
  |             DescribeTargetHealth              |
  +-----------------------------------------------+
  ||          TargetHealthDescriptions           ||
  |+---------------------------------------------+|
  ||               HealthCheckPort               ||
  |+---------------------------------------------+|
  ||  10080                                      ||
  |+---------------------------------------------+|
  |||                  Target                   |||
  ||+-------------------+---------------+-------+||
  ||| AvailabilityZone  |      Id       | Port  |||
  ||+-------------------+---------------+-------+||
  |||  ap-northeast-1a  |  10.90.2.178  |  4000 |||
  ||+-------------------+---------------+-------+||
  |||               TargetHealth                |||
  ||+------------------+------------------------+||
  |||  State           |  healthy               |||
  ||+------------------+------------------------+||
  ||          TargetHealthDescriptions           ||
  |+---------------------------------------------+|
  ||               HealthCheckPort               ||
  |+---------------------------------------------+|
  ||  10080                                      ||
  |+---------------------------------------------+|
  |||                  Target                   |||
  ||+-------------------+--------------+--------+||
  ||| AvailabilityZone  |     Id       | Port   |||
  ||+-------------------+--------------+--------+||
  |||  ap-northeast-1a  |  10.90.2.70  |  4000  |||
  ||+-------------------+--------------+--------+||
  |||               TargetHealth                |||
  ||+------------------+------------------------+||
  |||  State           |  healthy               |||
  ||+------------------+------------------------+|
  ```

### 步骤 3. 获取负载均衡器的 DNS 名称

  ```bash
  LB_DNS=`aws elbv2 describe-load-balancers --region ${LAB:REGION_CODE} --names ${LAB:USER_UNIQUE_TAG} --query 'LoadBalancers[0].DNSName' --output text`
  echo ${LB_DNS}
  ```

  ```output
  g-xxxx-xxxx-xx-xx-xxxxx-xxxx.elb.ap-northeast-1.amazonaws.com
  ```

### 步骤 4. 运行无重连逻辑的应用代码

+ 在 [终端 1](ssh://monitor1-ssh#1) 中，阅读并运行如下代码。
+ 代码会启动 4 个线程，每个线程会无限循环地连接到 NLB，并更新一条记录。`sleep(4)` 用于模拟长查询。
+ 如果线程被路由到即将被缩容的 TiDB 节点，可能会遇到第 1 步中描述的 A、B 两种情况。

  ```bash
  cat demo_no_retry.py
  ```

  ```output
  while True:
      try:
          conn = connect(
              database="test",
              host=lb_dns,
              port=4000,
              user="root",
              password="",
          )
          cursor = conn.cursor(prepared=True)

          random_integer = random.randint(0, 10)
          random_id=random.randint(1, 10)
          query = "UPDATE test.users SET balance = " + str(random_integer) + " WHERE id = " + str(random_id) + ";"

          cursor.execute("BEGIN")
          cursor.execute(query)
          time.sleep(4)

          try:
              cursor.execute("COMMIT")
              print(f"Thread {thread_id}: Committed. user {random_id} balance updated to: {random_integer}")
          except Error as e:
              print(f"Thread {thread_id}: Error during query execution: {e}")
          try:
              cursor.close()
              conn.close()
          except Error as e:
              pass

      except (Error, MySQLInterfaceError) as connect_err:
          print(f"Thread {thread_id}: Connection or execution error: {connect_err}")
  ```

  ```bash
  python3 demo_no_retry.py ${LB_DNS}
  ```

  ```output
  ...
  Thread 3: Committed. user 3 balance updated to: 3
  Thread 4: Committed. user 9 balance updated to: 4
  ...
  ```

### 步骤 5. 运行带重试逻辑的应用代码

+ 在 [终端 2](ssh://monitor1-ssh#2) 中，阅读并运行如下代码。
+ 本次代码增加了重试逻辑，防止 TiDB 缩容期间连接中断。
+ 重试间隔为 5 秒。

  ```bash
  cat demo_retry.py
  ```

  ```output
  ...
  def retry(max_attempts=5, delay=5):
      def decorator(func):
          @wraps(func)
          def wrapper(*args, **kwargs):
              for attempt in range(max_attempts):
                  try:
                      return func(*args, **kwargs)
                  except (Error, MySQLInterfaceError) as e:
                      if attempt == max_attempts - 1:  # Last attempt
                          print(f"All {max_attempts} attempts failed. Last error: {e}")
                          raise
                      print(f"Attempt {attempt + 1} failed: {e}. Retrying...")
                      time.sleep(delay)
              return None
          return wrapper
      return decorator

  @retry(max_attempts=5, delay=5)
  def execute_database_operation(thread_id):
  ...
  ```

  ```bash
  LB_DNS=`aws elbv2 describe-load-balancers --region ${LAB:REGION_CODE} --names ${LAB:USER_UNIQUE_TAG} --query 'LoadBalancers[0].DNSName' --output text`
  echo ${LB_DNS}
  ```

  ```output
  g-xxxx-xxxx-xx-xx-xxxxx-xxxx.elb.ap-northeast-1.amazonaws.com
  ```

  ```bash
  python3 demo_retry.py ${LB_DNS}
  ```

  ```output
  ...
  Thread 2: Committed. user 4 balance updated to: 2
  Thread 1: Committed. user 1 balance updated to: 7
  ...
  ```

### 步骤 6. 缩容 TiDB 节点

+ 在 [终端 3](ssh://monitor1-ssh#3) 中，执行以下命令：

  ```bash
  ~/.tiup/bin/tiup cluster scale-in tidb-demo --node ${HOST_DB2_PRIVATE_IP}:4000 -y
  ```

  ```output
  ...
  Scaled cluster `tidb-demo` in successfully
  ```

### 步骤 7. 观察 [终端 1](ssh://monitor1-ssh#1) 输出

+ 您可能会看到如下报错信息：

  ```output
  Thread X: Connection or execution error: Lost connection to MySQL server during query
  ```

  ```output
  Thread X: Connection or execution error: 2003 (HY000): Can't connect to MySQL server on 'g-xxx-xxx-xxx-xxx-77fd842711201a37.elb.ap-northeast-1.amazonaws.com:4000' (111)
  ```

### 步骤 8. 观察 [终端 2](ssh://monitor1-ssh#2) 输出

+ 在[终端 2](ssh://monitor1-ssh#2)中，您可能会看到如下信息。应用会最多重试 5 次，每次间隔 5 秒。如模块 1 所述，NLB 最坏情况下需要 15 秒检测到 TiDB 节点不健康，并将查询引导到健康的 TiDB。我们应用的重试窗口大于发现 TiDB 不健康状态的最长时间，因此可实现零中断。

  ```output
  Attempt 1 failed: Lost connection to MySQL server during query. Retrying...
  ```

  ```output
  Attempt 1 failed: 2003 (HY000): Can't connect to MySQL server on 'g-pingcap-com-ch-migrate-6540307-d8eaff67a1fb8f9e.elb.ap-northeast-1.amazonaws.com:4000' (111). Retrying...
  ```

+ [终端 2](ssh://monitor1-ssh#2) 的结果表明，`demo_connection_reconnect.py` 通过简单的重试逻辑，平滑应对了 TiDB 节点缩容，应用可充分利用 TiDB 的零中断特性。

### 步骤 9. 停止压力测试

+ 在 [终端 1](ssh://monitor1-ssh#1) 和 [终端 2](ssh://monitor1-ssh#2) 中，使用 `ctrl` + `c` 停止压力测试。

### 步骤 10. 从目标组中注销 TiDB 节点

+ 在 [终端 1](ssh://monitor1-ssh#1) 中，执行以下命令（**必须在终端 1 执行**）：

  ```bash
  TIDB_TARGETS=$(aws elbv2 describe-target-health --region ${LAB:REGION_CODE} --target-group-arn ${DEMO_TG_ARN} --query 'TargetHealthDescriptions[?Target.Port==`4000`].Target' --output json)
  ```

  ```bash
  if [ "$TIDB_TARGETS" != "[]" ]; then
    aws elbv2 deregister-targets --region ${LAB:REGION_CODE} --target-group-arn ${DEMO_TG_ARN} --targets "$TIDB_TARGETS"
    echo "Deregistered TiDB servers"
  else
    echo "No TiDB servers found to deregister"
  fi
  ```

  ```output
  Deregistered TiDB servers
  ```

### 步骤 11. 检查注销结果

+ 通过以下命令检查注销结果，可能看到如下输出：

  ```bash
  aws elbv2 describe-target-health --region ${LAB:REGION_CODE} --target-group-arn ${DEMO_TG_ARN} --output table
  ```

  ```output
  ----------------------
  |DescribeTargetHealth|
  +--------------------+
  ```

  ```output
  -------------------------------------------------------------
  |                   DescribeTargetHealth                    |
  +-----------------------------------------------------------+
  ||                TargetHealthDescriptions                 ||
  |+--------------------------------------+------------------+|
  ||  HealthCheckPort                     |  10080           ||
  |+--------------------------------------+------------------+|
  |||                        Target                         |||
  ||+---------------------------+---------------------------+||
  |||  AvailabilityZone         |  ap-northeast-1a          |||
  |||  Id                       |  10.90.2.226              |||
  |||  Port                     |  4000                     |||
  ||+---------------------------+---------------------------+||
  |||                     TargetHealth                      |||
  ||+--------------+----------------------------------------+||
  |||  Description |  Target deregistration is in progress  |||
  |||  Reason      |  Target.DeregistrationInProgress       |||
  |||  State       |  draining                              |||
  ||+--------------+----------------------------------------+||
  ||                TargetHealthDescriptions                 ||
  |+--------------------------------------+------------------+|
  ||  HealthCheckPort                     |  10080           ||
  |+--------------------------------------+------------------+|
  |||                        Target                         |||
  ||+---------------------------+---------------------------+||
  |||  AvailabilityZone         |  ap-northeast-1a          |||
  |||  Id                       |  10.90.2.172              |||
  |||  Port                     |  4000                     |||
  ||+---------------------------+---------------------------+||
  |||                     TargetHealth                      |||
  ||+--------------+----------------------------------------+||
  |||  Description |  Target deregistration is in progress  |||
  |||  Reason      |  Target.DeregistrationInProgress       |||
  |||  State       |  draining                              |||
  ||+--------------+----------------------------------------+||
  ```

### 总结

+ 本模块介绍了在使用网络负载均衡器时，TiDB Server 缩容场景下的应用最佳实践。
+ 实验表明，只要应用有足够的重试时间窗口，便可在 TiDB Server 缩容时实现零中断。