# SOP: TiKV Unsafe Recovery From Data Loss

## Scope

This SOP describes manual unsafe-recovery handling for TiKV data-loss scenarios,
especially for versions earlier than `v6.1.0`. It also explains how to assess
data-loss risk and when to prefer online unsafe recovery in newer versions.

## Background

### Raft basics

- TiKV uses 3 replicas by default. When a cluster has at least 3 TiKV instances,
  each Region normally has 3 replicas and forms one Raft group.
- Raft allows the cluster to keep serving requests without data loss as long as
  more than half of the replicas in a Raft group remain healthy.
- In a 3-replica setup:
  - losing 1 replica should not affect correctness, and availability impact is
    usually limited to transient jitter
  - losing 2 replicas can make some Regions unavailable, and permanent data
    loss becomes possible if the failed replicas cannot be recovered
- This document assumes the default 3-replica deployment. For clusters with
  more replicas, apply the same majority principle carefully.

### Typical failure scenarios

- Data-center power loss followed by recovery, where two TiKV disks slow down
  at the same time and Raft state machines stop making progress
- Multiple TiKV instances deployed without correct host labels, so one host
  failure removes multiple replicas of the same Region
- Non-standard operations, for example scaling in multiple TiKV instances with
  `--force` at the same time
- Similar cases where the failure boundary is clear and majority is lost

## Recovery Strategy by Failure Scope

### Fewer failed TiKV instances than half of the replica count

If the number of failed TiKV instances is less than half of the replica count,
data loss is not expected.

Example: in a 3-replica cluster without `sync-log`, one TiKV keeps panicking
after power recovery and cannot start.

Recommended handling:

1. Rolling-restart all TiKV instances that can still start, so common
   `commit index is out of range` issues are cleared first.
2. Try to start the failed TiKV again and identify the corruption type from logs.
3. Apply targeted repair based on the corruption type.

#### Corruption-type identification

- If panic logs mention `last index` or `commit index`, the Raft state machine
  is likely corrupted.
- If logs show `SST file size mismatch`, RocksDB was likely corrupted during
  snapshot apply.

#### Repair actions

- Raft state machine corruption:
  - find the affected Region ID in TiKV logs
  - tombstone the damaged peer

    ```bash
    pd-ctl operator add remove-peer <region_id> <store_id>
    tikv-ctl --data-dir /path/to/tikv tombstone -p 127.0.0.1:2379 -r <region_id>
    ```

- RocksDB corruption:
  - TiDB components do not provide an in-place repair tool for this case
  - the data on the failed instance must usually be discarded
  - this does not imply data loss if the number of failed TiKV instances is
    still less than half of the replica count, because the surviving majority
    still preserves the committed data

### Failed TiKV instances greater than or equal to half of the replica count

If the number of failed TiKV instances is greater than or equal to half of the
replica count, data loss is possible.

Possible situations include:

- all replicas of some Regions were lost during the failure
- more than half of the replicas of some Regions were lost, and the surviving
  replicas did not contain the latest writes because the Raft group never
  reached a majority and could not serve as leader

During recovery, keep recording affected `region_id` values so later data-loss
assessment stays traceable.

Recommended handling:

1. Use `pd-ctl` to identify Regions with at least half of their replicas on the
   failed TiKV stores, and record their Region IDs.
2. On all healthy instances, remove peers that belong to failed stores.
3. Restart TiKV and PD so surviving Regions can elect leaders again.
4. Handle Regions whose replicas were fully lost.

#### Regions with all replicas lost

- Check Regions without leaders.
- Determine which tables those Regions belong to.
- Inform the user that data in those Regions is lost and cannot be recovered
  from the cluster itself.
- Create empty Regions to clear `Region is unavailable`.

In practice, the best recovery path is often to re-import or re-consume data.
Ask the user:

- which tables were being written shortly before the failure
- whether DDL happened around the same time
- whether upstream data can be replayed

## Manual Unsafe-Recover Procedure

### 1. Disable scheduling

Before recovery, disable PD scheduling to minimize interference. Save the
current values first so they can be restored later.

```bash
config set region-schedule-limit 0
config set replica-schedule-limit 0
config set leader-schedule-limit 0
config set merge-schedule-limit 0
```

### 2. Identify Regions affected by failed stores

- Requirement: PD must be running.
- Goal: find Regions where at least half of the replicas are on failed stores.
- Note: if the Region API returns a very large payload, `jq` may consume too
  much memory and fail with OOM.

Example failed stores:

```text
15,22,43,45,46
```

Example command:

```bash
pd-ctl -u <endpoint> -d region --jq='.regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length as $total | map(if . == (15,22,43,45,46) then . else empty end) | length >= $total - length)}'
```

### 3. Shut down TiKV and remove failed-store peers

- Run this on healthy machines only.
- TiKV instances must be shut down before running the local unsafe-recover
  command.
- Before shutdown, ensure PD has been idle for at least 10 minutes and is no
  longer generating new operators.

```bash
tikv-ctl --db /path/to/tikv-data/db unsafe-recover remove-fail-stores -s <s1,s2,...> --all-regions
```

Example:

```bash
tikv-ctl --db /path/to/tikv-data/db unsafe-recover remove-fail-stores -s 30725906 --all-regions
```

### 4. Start TiKV and restart PD

Start TiKV, restart PD nodes if needed, and confirm that TiKV can report the
latest Region metadata back to PD.

### 5. Check Regions without leaders

- Requirement: PD must be running.

```bash
pd-ctl -u <endpoint> -d region --jq '.regions[] | select(has("leader") | not) | {id: .id, peer_stores: [.peers[].store_id]}'
```

### 6. Map failed Regions to tables

- Requirement: PD, TiKV, and TiDB must all be running.
- Goal: identify which table owns the unavailable Region.

```bash
curl "http://{TiDBIP}:10080/regions/{regionID}"
```

### 7. Resolve `Region is unavailable`

- Requirement: PD and TiKV must be running.
- Action: create an empty Region for the lost Region ID.

```bash
tikv-ctl --db /path/to/tikv-data/db recreate-region --pd <endpoint> -r <region_id>
```

### 8. Check data consistency

- Requirement: PD, TiKV, and TiDB must all be running.
- Check index consistency and table consistency after recovery.

Example:

```sql
SELECT COUNT(*) AS c1 FROM table_name;
SELECT COUNT(*) AS c2 FROM table_name FORCE INDEX (`idx_name`);
SELECT c1 = c2;
ADMIN CHECK TABLE table_name;
```

### 9. Resume scheduling

Restore PD scheduler limits to their original values.

```bash
config set region-schedule-limit {default-value}
config set replica-schedule-limit {default-value}
config set leader-schedule-limit {default-value}
config set merge-schedule-limit {default-value}
```

## Versions Above v6.1.0

Online unsafe recovery was introduced in `v6.1.0`. In versions below `v6.1`,
it should still be treated as experimental and used cautiously.

For modern clusters, prefer the official online unsafe-recovery workflow unless:

- the version is known to have unsafe-recovery bugs
- TiFlash or learner-related issues block the workflow
- manual unsafe recovery is required as a fallback