# Restoring a Cluster After an Unsafe Recovery
>💡 The audience for this article is front-line DBAs, covering operations when there is a cluster majority copy loss and non-secure recovery
## test environment
For our test, there were three copies of each region, distributed over 5 TiKV instances. The host IPs were setup as follows:
|
|
|---|
Component | IP |
TiDB | 172.16.7.200 |
PD | 172.16.7.200 |
TiKV | 172.16.7.176 |
| 172.16.7.182 |
| 172.16.7.200 |
| 172.16.7.222 |
| 172.16.7.229 |
## 1\. Background
------------------------
TiDB defaults to 3 replicas, and between the replicas, the leader is elected through raft and the majority is to ensure strong consistency and high availability. If the majority is lost, the region may be unavailable, or even permanent data loss may occur. If similar unrecoverable failures occur , The business needs to be restored urgently. At this time, the TiDB cluster needs to be restored quickly and the data is restored to the greatest extent. The specific situation is divided into the following categories:
* Only one copy is lost, no need to deal with;
* If the majority copy is lost, data needs to be restored based on the remaining copy. In extreme cases, the latest transaction-level data will be lost. At this time, the business side needs to redo the latest data at the time of the disaster;
* All copies are lost. At this time, the data loss of the region cannot be recovered. Business basis is required: redo data related to all lost tables in the copy.
## 2\. Simulate the occurrence of a disaster, that is, the loss of the majority
------------------------
1\. Get all store ids, find the store id corresponding to a certain region of a table
Execute the command to get the mapping relationship between store id and ip
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 store
```
store id | ip |
|---|
1 | 172.16.7.182 |
4 | 172.16.7.222 |
5 | 172.16.7.176 |
104 | 172.16.7.200 |
105 | 172.16.7.229 |
## Viewing sbtest.sbtest1 table region distribution
```sql
show table sbtest.sbtest1 regions;
```
Here we select the two copies 151 and 403 of region id 149 of the sbtest.sbtest1 table as the object of the fault simulation, and find the addresses corresponding to these two copies
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 region 149
```
From the above figure, we can see that the 151 copy corresponds to store\_id 104 and the 403 copy corresponds to store\_id 105, which correspond to the 172.16.7.200 and 172.16.7.229 machines respectively.
## 2\. Manually simulate a downtime tikv node
------------------------
### Turn off automatic restart
```bash
vim /etc/systemd/system/tikv-30161.service
```
Make the configuration effective
```bash
systemctl daemon-reload
```
Kill the TiKV process of the 172.16.7.200 and 172.16.7.229 machines, and confirm that the TiKV node is down
```bash
kill -9 \[tikv server pid\]
```
Delete the tikv deployment directory and data of the failed node
```bash
rm -rf /home/tidb-deploy\_xq/tikv-30161
rm -rf /home/tidb-data\_xq/tikv-30161
```
Table loses majority copy verification
## 3\. Unsafe recover steps and instructions
------------------------
### 1\. Query the downtime TiKV store id, here are 104 and 105
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 store | grep -C 1 -E "172.16.7.200|172.16.7.229"
```
### 2\. Obtain the default parameter values of region-schedule-limit, replica-schedule-limit, leader-schedule-limit, merge-schedule-limit, record them and set these four parameters to 0
View parameter values:
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config show | grep -E'"region-schedule-limit"|replica-schedule-limit|leader-schedule-limit|merge-schedule-limit'
```
Modify parameter value
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set region-schedule-limit 0
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set replica-schedule-limit 0
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set leader-schedule-limit 0
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set merge-schedule-limit 0
```
double check
### 3\. Check the region where "more than half of the replicas are on the failed node" and record the region id
Failure node store id 104 and 105
Enter pd interactive command terminal
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 -i
```
View the region id of the lost majority copy
```bash
region --jq=".regions\[\] | {id: .id, peer\_stores: \[.peers\[\].store\_id\] | select(length as $total | map(if .==(104,105) then. else empty end) | length>=$total-length) }"
```
### 4\. Stop all TiKV nodes
```bash
tiup cluster stop tidb-xq -R=tikv
```
### 5\. Remove the store id 104 and 105 information of the failed TiKV node on all normal TiKV nodes
Send the ./.tiup/components/ctl/v5.2.1/tikv-ctl command under the tiup machine to the healthy tikv node
```bash
scp ./.tiup/components/ctl/v5.2.1/tikv-ctl 172.16.7.176:/root/
```
Remove fault store id information
```bash
./tikv-ctl --data-dir /home/tidb-data\_xq/tikv-21160 unsafe-recover remove-fail-stores -s 104,105 --all-regions
```
### 6\. Stop the PD node
```bash
tiup cluster stop tidb-xq -R=pd
```
### 7\. Start PD, TiKV node
```bash
tiup cluster start tidb-xq -R=pd,tikv
```
### 8\. View the region without a leader
Enter pd interactive command terminal
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 -i
```
View the region without a leader
```bash
region --jq'.regions\[\]|select(has("leader")|not)|{id: .id,peer\_stores: \[.peers\[\].store\_id\]}'
```
### 9\. Check the table data at this time, and the region distribution
### 10\. Restore the scheduling parameters, continue to check the table region again, so far, the restoration is over
```bash
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set region-schedule-limit 2048
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set replica-schedule-limit 64
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set leader-schedule-limit 4
tiup ctl:v5.2.1 pd -u 172.16.7.200:9233 config set merge-schedule-limit 8
```
------------------------
> **Product**: TiDB
> **Component**: TiKV
> **Affected Versions**: 5.2.1
> **Tags**: kb TiDB Supportability Troubleshoot