# 2021-04-25-14:04:20+0800 vertical scaling tidb-server to c5.9xlarge

- Cause: CloudChat's cluster tidb pods got OOM killed again and again

- Operation:

```sh
kubectl get shoot 4bc76ea -n garden-prod -oyaml > shoot.yaml
kubectl edit shoot 4bc76ea -n garden-prod
# add the following to shoot
    - labels:
        gardener.node.az.info: ap-northeast-2a
        gardener.node.machine.type: c5.9xlarge
        gardener.wg.name: tidb1311285575352848384-tidb-2a
        gardener.wg.prefix.name: tidb1311285575352848384-tidb
        tenant: "1311265642040528896"
      machine:
        image:
          name: centos
          version: 7.8.2003
        type: c5.9xlarge
      maxSurge: 1
      maxUnavailable: 0
      maximum: 3
      minimum: 3
      name: "3490631a"
      taints:
      - effect: NoSchedule
        key: dedicated
        value: tidb-tidb
      volume:
        size: 50Gi
        type: gp2
      zones:
      - ap-northeast-2a
    - labels:
        gardener.node.az.info: ap-northeast-2b
        gardener.node.machine.type: c5.9xlarge
        gardener.wg.name: tidb1311285575352848384-tidb-2b
        gardener.wg.prefix.name: tidb1311285575352848384-tidb
        tenant: "1311265642040528896"
      machine:
        image:
          name: centos
          version: 7.8.2003
        type: c5.9xlarge
      maxSurge: 1
      maxUnavailable: 0
      maximum: 2
      minimum: 2
      name: "3490631b"
      taints:
      - effect: NoSchedule
        key: dedicated
        value: tidb-tidb
      volume:
        size: 50Gi
        type: gp2
      zones:
      - ap-northeast-2b
    - labels:
        gardener.node.az.info: ap-northeast-2c
        gardener.node.machine.type: c5.9xlarge
        gardener.wg.name: tidb1311285575352848384-tidb-2c
        gardener.wg.prefix.name: tidb1311285575352848384-tidb
        tenant: "1311265642040528896"
      machine:
        image:
          name: centos
          version: 7.8.2003
        type: c5.9xlarge
      maxSurge: 1
      maxUnavailable: 0
      maximum: 2
      minimum: 2
      name: "3490631c"
      taints:
      - effect: NoSchedule
        key: dedicated
        value: tidb-tidb
      volume:
        size: 50Gi
        type: gp2
      zones:
      - ap-northeast-2c
# Taint all c5.2xlarge nodes
kubectl taint nodes -l gardener.node.machine.type=c5.2xlarge memory=small:NoSchedule
# Untaint privileged-tidb node
kubectl taint nodes ip-10-250-31-109.ap-northeast-2.compute.internal memory=small:NoSchedule-
```

- Impact:

The TiDB pods will be scheduled to c5.9xlarge nodes which has enough memory.

No new pods will be scheduled to c5.2xlarge nodes.

- Recovery operation:

1. Remove the newly added worker groups
2. Untaint the c5.2xlarge nodes

- Time window: 2021-04-24 18:30 ~ 19:00