# 2022-08-25-13:13:10+0800 Migrate ossinsight cluster tiflash to single az

- Cause:

https://jira.tidbcloud.com/browse/DM-6489
To reduce internal cluster tiflash cross AZ traffic cost

- Operation:

The project id for ossinsight cluster is 1372813089188931282 and cluster id is 1379661944623214066. The cluster has 3 Tiflash pods, distributed in 3 AZs (tiflash-0:us-west-2a, tiflash-1:us-west-2b, tiflash-2:us-west-2c)

Follows this migration guide https://pingcap.feishu.cn/wiki/wikcnbMDCas1dibmPARbtoSaLte#doxcn28SsWS4yy8G40SPu0YXpff

1. Enable `tiflash_single_az` featuregate for ossinsight project
https://github.com/tidbcloud/infra-cd/pull/1241/files

2. Add `node.scaler.cloud.pingcap.com/tiflash-az` annotation to `Cluster`

``` shell
kubectl --kubeconfig ~/infra-api.kubeconfig annotate cluster 1379661944623214066 -n prod-1372813089188931282 node.scaler.cloud.pingcap.com/tiflash-az=us-west-2c
```

Here we choose `us-west-2c` AZ for tiflash

3. Increase tiflash replicas

``` shell
kubectl --kubeconfig ~/infra-api.kubeconfig edit cluster -n prod-1372813089188931282 1379661944623214066 # set tiflash replicas from 3 to 5
```

4. Check infra-provider logs for "Tiflash single AZ deploy" to ensure the cluster has been enabled `tiflash_single_az` feature and newly created tiflash nodes are in `us-west-2c`

5. After the newly created tiflash pods are ready, add `node.scaler.cloud.pingcap.com/ignore` annotation to `TidbCluster` CR

``` shell
kubectl annotate tc -n tidb1379661944623214066 db node.scaler.cloud.pingcap.com/ignore=true
```

6. Offline the us-west-2a and us-west-2c tiflash

``` shell
kubectl edit tc -n tidb1379661944623214066 db # add tiflash.tidb.pingcap.com/delete-slots: '[0,1]' annotation and decrease tiflash replicas from 5 to 3
```

7. Waiting the old tiflash pods offline

``` shell
kubectl get po -n tidb1379661944623214066 -w
```

8. After the pods are offline, remove `node.scaler.cloud.pingcap.com/ignore` annotation for `TidbCluster` CR and delete the old PVC. Ensure the PV and nodes are deleted

``` shell
kubectl edit tc -n tidb1379661944623214066 db # remove node.scaler.cloud.pingcap.com/ignore annotation
kubectl delete pvc -n tidb1379661944623214066 data0-db-tiflash-0 data0-db-tiflash-1
```

- Impact:

The tiflash pods will be migrated to single AZ.

- Recovery operation:

Disable `tiflash_single_az` feature for the project, and then migrate the tiflash to 3 AZs

- Time window:

2022-08-25 14:00 ~ 18:00