# [Known Issue] Replica inconsistency after unexpected TiKV restart may cause `default not found` ([tikv#16706](https://github.com/tikv/tikv/issues/16706))

Source note: adapted from GitHub issue [tikv#16706](https://github.com/tikv/tikv/issues/16706).

## Issue

- Link: [tikv#16706](https://github.com/tikv/tikv/issues/16706)
- Summary: an abnormal TiKV restart can trigger a RocksDB bug and lose local data on one replica. This creates replica inconsistency inside the same region. If the corrupted replica is the leader, DML may fail with errors such as `default not found`, while follower read may still succeed.

## Phenomenon

- A specific SQL fails consistently on one key or row.
- The error may look like:

    ```text
    default not found: key:7480000000000002FF535F7280000000BBFFA34E480000000000FAF9B0F3831A1FFFD5, maybe read truncated/dropped table data?
    ```

- TiKV logs or monitoring show that one or more TiKV nodes restarted unexpectedly before the problem started.
- Reads may behave differently depending on which replica is used:
  - leader read fails with `default not found`
  - follower read may succeed if the follower replica is healthy

## Root Cause

During an abnormal TiKV restart, the RocksDB bug in [tikv#16706](https://github.com/tikv/tikv/issues/16706) may cause local data loss on one replica. The three replicas of the same region then become inconsistent.

If the lost data is on the current leader replica, requests served by that leader may fail with `default not found`. If leadership is transferred to a healthy replica, the same SQL may succeed again, which is a strong signal that this is a replica-corruption issue instead of a logical SQL problem.

## Diagnostic Steps

-  Reproduce the failure with a specific SQL statement and capture the exact key from the error message.

    ```text
    default not found: key:7480000000000002FF535F7280000000BBFFA34E480000000000FAF9B0F3831A1FFFD5, maybe read truncated/dropped table data?
    ```

-  Check TiKV logs or monitoring and confirm whether any TiKV node restarted abnormally around the time the issue began.

- Locate the region containing the problematic key by querying `INFORMATION_SCHEMA.TIKV_REGION_STATUS`. The key in this error is already in memory-comparable encoded form, so it can be searched directly:

    ```sql
    SELECT *
    FROM INFORMATION_SCHEMA.TIKV_REGION_STATUS
    WHERE start_key <= '7480000000000002FF535F7280000000BBFFA34E480000000000FAF9B0F3831A1FFFD5'
    AND end_key > '7480000000000002FF535F7280000000BBFFA34E480000000000FAF9B0F3831A1FFFD5'\G
    ```

Example result:

    ```text
    *************************** 1. row ***************************
    REGION_ID: 539795756
    START_KEY: 7480000000000002FF4F00000000000000F8
    END_KEY: 7480000000000002FF5B00000000000000F8
    APPROXIMATE_SIZE: 1
    APPROXIMATE_KEYS: 10
    ```

- Query region peer information with `pd-ctl` and identify the current leader and all replicas:

    ```bash
    tiup ctl:v7.5.2 pd region 539795756
    ```

Example result:

    ```json
    {
    "id": 539795756,
    "peers": [
        {"role_name": "Voter", "id": 539795757, "store_id": 1},
        {"role_name": "Voter", "id": 539795758, "store_id": 5},
        {"role_name": "Voter", "id": 539795759, "store_id": 4}
    ],
    "leader": {"role_name": "Voter", "id": 539795758, "store_id": 5}
    }
    ```

- Transfer the leader to another replica and re-run the failing SQL. If the SQL succeeds after leadership moves, the original leader replica is likely corrupted:

    ```bash
    tiup ctl:v7.5.2 pd operator add transfer-leader 539795756 4
    tiup ctl:v7.5.2 pd region 539795756
    ```

-  As a quick additional check, try follower read:

```sql
SET [SESSION | GLOBAL] tidb_replica_read = 'follower';
```

If follower read succeeds while leader read fails, that also strongly suggests replica inconsistency. Note that this method is probabilistic because follower selection is not guaranteed to hit a healthy replica every time.

## Workaround

The remediation depends on whether at least one correct replica still exists.

### Case 1: At least one healthy replica exists

No data loss is required.

-  Transfer the leader to a healthy replica:

```bash
pd-ctl operator add transfer-leader <region-id> <store-id>
```

-  Remove the damaged replicas one by one:

```bash
pd-ctl operator add remove-peer <region-id> <store-id>
```

-  Let PD add new replicas automatically, or add them manually:

```bash
pd-ctl operator add add-peer <region-id> <store-id>
```

- Verify replica replacement progress:

```bash
pd-ctl region <region-id>
```

**Important:** remove damaged peers one at a time. Removing multiple majority peers at once can make the region unavailable.

### Case 2: All replicas of the region are corrupted

Data loss is unavoidable for that key range.

- Identify the affected row(s) and delete the corrupted data directly.
- Avoid read-based repair SQL if reads themselves trigger `default not found`.
- Prefer direct row-id or key-based cleanup operations that minimize additional reads.

## Affected Version

- Versions before the following fixed releases on each maintained branch may be affected.

## Fixed Version

- v7.5.2
- v7.1.5
- v6.5.9
- v6.1.8
- v5.4.4

## Related Oncall

- v6.5.3 leader reports default not found after TiKV restarts, while follower read works normally ONCALL-7554