# TiDB Failover with TiDB Operator

> - **Objective:** Learn TiDB Failover with TiDB Operator
> - **Prerequisites:** 
>     - background knowledge of TiDB
>     - background knowledge of Kubernetes and TiDB Operator
>     - AWS account
>     - TiDB cluster on AWS
> - **Optionality:** Required
> - **Estimated time:** 30min

This document focuses on demonstrating the failover of TiDB with TiDB Operator. One design goal of TiDB Operator is to automatically handle failures. 

## Prepare

> **Note** 
> 
> We must have enough avaiable nodes for the cluster to have a spare; for example, if we have 3 TiKV pods, we must prepare at least 4 Kubernetes nodes.
>
> If you've deployed using Terraform, you should adjust the `default_cluster_tikv_count` and `default_cluster_pd_count` variables accordingly.

To demonstrate failover, you should put some load on your cluster. For instructions on how to run sysbench, refer to [Run Sysbench](/hc/en-us/articles/360049229634).

## TiKV Failover

In this section, we will explore the failover scenarios for TiKV. TiKV uses the Raft consensus algorithm to achieve high availability. Data stored in TikV has multiple replicas. 

By default, each data has 3 replicas and each replica is placed in 3 TiKV pods. As long as 2 replicas are available, there should be no impact to the sysbench workload.

On AWS, EC2 instances can be provisioned in different Availability Zones (AZ). TiDB Operator schedules TiKV pods in different Availability Zones for high availability. For example, if there are 3 TiKV pods, the 3 TiKV pods will be scheduled on nodes in different AZs.

### List Pods 

You can use the following command to list all pods for the TiDB cluster.

```{.bash .copyable}
kubectl get pod -n ${namespace}
```

```{.output}
NAME                                    READY   STATUS    RESTARTS   AGE
my-cluster-discovery-85cbf5454b-x7gjl   1/1     Running   0          10h
my-cluster-pd-0                         1/1     Running   0          3d17h
my-cluster-pd-1                         1/1     Running   0          3d17h
my-cluster-pd-2                         1/1     Running   0          3d17h
my-cluster-tidb-0                       2/2     Running   0          3d18h
my-cluster-tidb-1                       2/2     Running   0          3d18h
my-cluster-tikv-0                       1/1     Running   0          3d12h
my-cluster-tikv-1                       1/1     Running   0          3d12h
my-cluster-tikv-2                       1/1     Running   0          3d12h
```

### Kill One TiKV Pod

You can use the following command to delete one TiKV pod:

```{.bash .copyable}
kubectl delete pod my-cluster-tikv-2 -n ${namespace}
```

```{.output}
pod "my-cluster-tikv-2" deleted
``` 

After TiKV pod deletion, we can check the status again and we can see that a new TiKV pod with the same name is created by TiDB Operator. Note that the age of `my-cluster-tikv-2` should be a small number, usually a couple of seconds. 

```{.bash .copyable}
kubectl get pod my-cluster-tikv-2 -n ${namespace}
```

```{.output}
NAME                READY    STATUS    RESTARTS   AGE
my-cluster-tikv-2   1/1      Running   0          3s
``` 

In addition, the sysbench workload is not affected.

### Kill Two TiKV Pods 

You can use the following command to delete two TiKV pods:

```{.bash .copyable}
kubectl delete pod my-cluster-tikv-1 my-cluster-tikv-2 -n ${namespace}
```

```{.output}
pod "my-cluster-tikv-1" deleted
pod "my-cluster-tikv-2" deleted
``` 

After TiKV pod deletion, we can check the status again and we can found that the pod deleted will be recreated by TiDB Operator. Note that the age of `my-cluster-tikv-1` and `my-cluster-tikv-2` should be a small number, usually a couple of seconds. 

In this case, however, the QPS of sysbench workload will drop to `0` and return to normal after a few seconds.

### Delete One TiKV Node

#### Configure TiKV

> - **Optionality:** Optional

PD marks a TiKV store as down after 30 minutes by default. To reduce the wait time, you can configure `max-store-down-time` in `TiDBCluster` CR:

```{.bash .copyable}
kubectl edit tc my-cluster -n ${namespace}
```

You should set `pd.spec.config.schedule.max-store-down-time` to `5m`.

#### Node Deletion

You can delete TiKV nodes in AWS console: 

![delete node in AWS console](https://i.imgur.com/77ymQCc.png)

After deleting one TiKV node, the sysbench's QPS will decrease and the status of one TiKV pod will change to `pending` as there are no available machines to schedule the pod on. 

![sysbench output](https://i.imgur.com/p9FKUMP.png) 

To allow TiDB operator to schedule the `pending` pod, we need to add new TiKV nodes. You can add new TiKV nodes in AWS console: 

![add new pod using "Launch More Like This" in AWS console](https://i.imgur.com/h3Q3TFw.png)

> **Note**
> 
> As TiDB Operator schedules TiKV pods in different Availability Zones, ensure that the AZ of the new node matches the one that has been deleted.
 
After waiting for `max-store-down-time5` + `tikvFailoverPeriod` (`30m` + `5m` by default), TiDB Operator creates a new TiKV pod and data will be moved to the new pod.

#### Cleanup

If the failed TiKV pod comes back online, TiDB Operator does not automatically delete the newly created node, and you need to manually drop it and restore the original number of nodes. 

To do this, you can delete the TiKV node from the `status.tikv.failureStores` field of the `TidbCluster` CR. 

### Make One TiKV Pod Continuously Unavailable 

We can use `kubectl patch` command to update the image of a TiKV pod and point to a fake image: 

```{.bash .copyable}
kubectl patch pod my-cluster-tikv-2 -n ${namespace} -p '{"spec":{"containers":[{"name":"tikv","image":"hub.pingcap.net/pingcap/fake"}]}}'
```

After executing the above command, the status of the TiKV pod will change to `ErrImagePull`: 

```{.bash .copyable}
kubectl get pod my-cluster-tikv-2 -n ${namespace}
```

```{.output}
NAME                READY   STATUS         RESTARTS   AGE
my-cluster-tikv-2   0/1     ErrImagePull   0          57m
```

The above operation makes one TiKV pod stays in a failure state. After waiting for `max-store-down-time5` + `tikvFailoverPeriod` (`30m` + `5m` by default), TiDB Operator creates a new TiKV pod and data will be moved to the new pod.

#### Cleanup

If the failed TiKV pod comes back online, TiDB Operator does not automatically delete the newly created node, and you need to manually drop it and restore the original number of nodes. 

To do this, you can delete the TiKV node from the `status.tikv.failureStores` field of the `TidbCluster` CR. 

## PD Failover

In this section, we demonstrate how TiDB operator handles PD failures.

### Make one PD pod continuously unavailable

We can use `kubectl patch` command to update the image of PD pod: 

```{.bash .copyable}
kubectl patch pod my-cluster-pd-2 -n ${namespace} -p '{"spec":{"containers":[{"name":"pd","image":"hub.pingcap.net/pingcap/fake"}]}}'
```

After executing the above command, the status of the pd pod will change to `ErrImagePull`: 

```{.bash .copyable}
kubectl get pod my-cluster-pd-2 -n ${namespace}
```

```{.output}
NAME        READY   STATUS         RESTARTS   AGE
my-cluster-pd-2   0/1     ErrImagePull   0          57m
```  

After `pdFailoverPeriod` (`5m` by default), a new PD Pod with the same name will be created and data will be moved to the new Pod.

One important difference between PD and TiKV failover is that if the failed PD pod comes back online, TiDB Operator automatically deletes the newly created node.