---
name: tikv-recovery
description: |
  Reference handbook for TiKV recovery and data-loss scenarios.
  Use when handling unsafe recovery, replica majority loss, orphan peers,
  TiFlash-related recovery blocking, tombstone operations, or SST recovery.
  Includes recovery paths, limitations, and operational cautions.
---

# TiKV Recovery Handbook

Known recovery workflows and failure patterns for TiKV on-call diagnosis.

## When To Use This Skill

Use this skill when:

- regions lose majority and become unavailable
- unsafe recovery is required or gets stuck
- orphan peers persist after recovery
- SST-level recovery or corruption-repair references are needed

## First Checks

- Confirm whether the issue is data-loss, replica-loss, or metadata inconsistency.
- Record the affected `region_id`, `store_id`, and component versions.
- Verify whether TiFlash replicas are involved.
- Confirm whether the cluster is already in unsafe-recovery flow.
- Decide whether the immediate goal is service restoration, metadata cleanup, or post-recovery consistency repair.

## Routing Guide

- `Unsafe Recovery`: majority loss, unsafe-recovery workflow selection, or unsafe-recovery failure patterns
- `references/sop-unsafe-recover-from-data-loss.md`: manual unsafe-recovery background, detailed procedure, and version-specific cautions

## Recovery Principles

- Treat unsafe recovery as a last-resort operation when quorum is already lost.
- Assume some recent data loss may be unavoidable once majority replicas are gone.
- Record affected regions and stores before manual cleanup operations such as `tombstone`.
- Check whether TiFlash or learner replicas change the recovery path before taking action.

## Unsafe Recovery

### ✅ Majority of replicas lost and unsafe recovery is required

- Detailed reference: [`references/sop-unsafe-recover-from-data-loss.md`](references/sop-unsafe-recover-from-data-loss.md)
- Description: when a region loses its majority of replicas, it can no longer elect a leader and the cluster reports `Region is unavailable`. In this situation, unsafe recovery is required. Because the majority is lost, the remaining replica data may not be the latest version, so some recent data loss may be unavoidable.
- Workaround:
  - `>= v6.1.0`: use online unsafe recovery. Because `v6.1.0` itself has known issues, `v6.5.0` is the recommended stable baseline.
    - Upgrade both PD and TiKV to `v6.5.0` at the same time.
  - `< v6.1.0`: use manual unsafe recovery. See [`SOP: TiKV Unsafe Recovery From Data Loss`](references/sop-unsafe-recover-from-data-loss.md).
- Related oncall:
  - v5.0.2 multiple replicas lost `APID-11444` / `GTOC-7724`
  - v5.0.6 region unavailable due to two TiKV instances going down `ONCALL-7059` / `TICKET-4139`
  - v6.1.7 GC failed after unsafe recovery `ONCALL-7086`
    - The user upgraded only PD to `v6.5.0` for online unsafe recovery, which is not recommended.
    - During recovery, some regions were found where the leader data version was older than the follower version. One repair path was:
      - shut down the TiKV node hosting the outdated leader
      - use `tikv-ctl tombstone` to remove the outdated leader peer
      - delete the stale region cache on PD, then restart the TiKV instance so GC can resume
      - if regions without leaders remain, delete their cache entries from PD:

        ```bash
        curl -X DELETE "http://{pd}:2379/pd/api/v1/admin/cache/region/<region_id>"
        ```

      - after some time, check whether holes still exist and recover those regions further if needed

### ✅ Unsafe recovery gets stuck when learner nodes exist on a failed store ([pd#6690](https://github.com/tikv/pd/issues/6690))

- Symptom: during unsafe recovery, PD does not treat learner replicas on failed stores as down peers. These learners may then be force-elected as leaders during recovery, but because they are actually down and cannot receive snapshots, the whole unsafe recovery eventually times out.
- Sample logs:

```text
[2024/09/06 15:58:28.789 +08:00] [INFO] [unsafe_recovery_controller.go:451] ["Unsafe recovery receives invalid store report"] [store-id=5] [expected-step=2] [obtained-step=1]
...
[2024/09/06 16:09:09.336 +08:00] [INFO] [unsafe_recovery_controller.go:547] ["{\"info\":\"Unsafe recovery failed: Exceeds timeout 2024-09-06 16:07:47.001187948 +0800 CST m=+729.316738498\",\"time\":\"2024-09-06 16:09:09.334\""]
```

- Workaround: upgrade to a fixed version, or perform manual unsafe recovery instead.
- Fixed version: `v6.5.4` / `v7.1.1` / `v7.3.0`
- Related oncall:
  - v6.5.2 unsafe recovery got stuck `ONCALL-7385` / `TICKET-4938`

### ✅ Online unsafe recovery gets stuck when TiFlash replicas exist ([tikv#18197](https://github.com/tikv/tikv/issues/18197))

- Symptom: online unsafe recovery times out when the affected regions include TiFlash replicas.
- Workaround: use offline unsafe recovery instead.
- Fixed version: `v8.5.2` / `v8.1.3` / `v7.5.7` / `v7.1.7` / `v6.5.13`
- Related oncall:
  - v7.5.4 recover data after forced scale-in `TICKET-6331` / `ONCALL-7905`

### ✅ Orphan peer persists after unsafe recovery and keeps reporting errors ([tikv#18121](https://github.com/tikv/tikv/issues/18121))

- Description: after multiple TiKV nodes suffer physical corruption and unsafe recovery is executed, an orphan peer may remain on some TiKV instances. That orphan peer keeps trying to contact its remembered leader, which no longer exists, so the node keeps printing persistent errors.
- Typical log:

```text
[2024/11/14 01:26:36.709 +08:00] [ERROR] [raft_client.rs:796] ["resolve store address failed"] [err_code=KV:Unknown] [err="Other("[src/server/resolve.rs:100]: unknown error
"[components/pd_client/src/util.rs:878]: invalid store ID 76169447, not found "")"] [store_id=76169447]
```

- Workaround:
  - Use online unsafe recovery to remove the invalid store:

    ```bash
    pd-ctl -u <pd_addr> unsafe remove-failed-stores <store-id>
    ```

  - Or clean up the orphan peer manually:
    - Step 1: determine whether the region still exists by checking historical TiKV logs and region metadata
    - Identify affected regions:

      ```bash
      pd-ctl region scan | jq '.regions[] | select(.peers[].store_id == <store-id>) | .id'
      ```

    - Step 2: handle by region status
      - Case A: the region no longer exists
        - reconstruct the region replica through unsafe recovery
      - Case B: the region still exists
        - mark the orphan peer as tombstone after shutting down the TiKV instance:

          ```bash
          tikv-ctl --data-dir /path/to/tikv tombstone -p 127.0.0.1:2379 -r <region_id>
          ```

- Related oncall:
  - v6.5.2 TiKV kept reporting `resolve store address failed` after restart `TICKET-5442`
  - v6.5.6 / v7.1.3 TiKV logs repeatedly reported `resolve store address failed` `TICKET-5723` / `TICKET-5718` / `ONCALL-7674`

## Related Skills

- For TiKV storage corruption, SST, or RocksDB-side failure patterns, read `skills/diagnosis/tikv-storage/SKILL.md`.
- For TiKV panic and crash scenarios around recovery operations, read `skills/diagnosis/tikv-panic/SKILL.md`.
- For raftstore and peer-lifecycle issues outside explicit recovery flows, read `skills/diagnosis/tikv-raftstore/SKILL.md`.