# [Known Issue] GC is stuck by corrupt transaction ([tikv#11187](https://github.com/tikv/tikv/issues/11187)) Source note: adapted from GitHub issue [tikv#11187](https://github.com/tikv/tikv/issues/11187). ## Issue - Link: [tikv#11187](https://github.com/tikv/tikv/issues/11187) - Symptom: GC gets stuck at the `resolve-locks` step because a corrupt transaction leaves a lock whose `min_commit_ts` is greater than the committed `commit_ts` ## Phenomenon TiDB's GC has been stuck for a long time. From the keyword `gc_worker` in TiDB's log, it turns out that the GC process became stuck at the resolve-locks step: ```text [2024/07/26 14:48:16.909 +08:00] [ERROR] [gc_worker.go:226] ["[gc worker] runGCJob"] [error="unexpected resolve err: commit_ts_expired: "] ``` ```text [2025/01/03 18:42:54.492 +08:00] [Info] [range_task.go:310] ["canceling range task because of error"] [name=resolve-locks-runner] [startKey=] [endKey=7480000000000000195f728000000006967b0f] [error=" unexpected resolve err: commit_ts_expired: "] ``` From the stuck information, we can see that in the current resolve-lock phase, the transaction corresponding to the current lock has already been committed. When we tried to commit with its corresponding `commit_ts` (`attempted_commit_ts`), we found that it was greater than the `min_commit_ts` written in the lock and was refused. ## Root Cause Pessimistic transactions may contain keys that don't need pessimistic locks. These locks are prewritten without any constraint checks. We do not add pessimistic locks for non-unique indexes, and TiKV's pessimistic transaction prewrite does not check all lock states, resulting in [tikv#11187](https://github.com/tikv/tikv/issues/11187). When async_commit is enabled, TiKV considers the transaction submission successful after prewriting all keys. At this time, if another transaction reads the lock, because the transaction corresponding to the lock has already prewritten the key, it will calculate `commit_ts` and commit the transaction itself. During this process, if network latency increases and the first prewrite retries, the corresponding TiKV may receive the first prewrite (the one the client assumed timed out) after the current transaction commits. Example event sequence: 1. txn1 prewrite key1, key2, key3; key1 meets network timeout, retries and succeeds. key1, key2, key3 prewrite finished. 2. txn2 meets lock key1 (txn1), finds all locks (key1, key2, key3) for txn1 exist, so it commits txn1. 3. TiKV receives txn1's first prewrite and locks key1 successfully. The reason for the failure of resolve lock is that the prewrite request (step 3) that arrived too late used async commit, which calculated a larger `min_commit_ts` field through the `max_ts` on the TiKV node. The original transaction (txn1) had already committed with a smaller `commit_ts`, so resolving the lock with the smaller `commit_ts` failed and could not be processed. ## Trigger Condition - The user's network environment is poor (slow). - async commit or 1PC is enabled. - Only non-unique indexes are affected. ## Diagnostic Process Search for `gc_worker` in TiDB logs to find resolve lock failure logs: ```text tidb_10.50.101.70_4000.log:[2025/01/03 18:43:54.525 +08:00] [Error] [gc_worker.go:630] ["[gc worker] resolve locks returns an error"] [uuid=64ac074aaf40016] [error="scanRegion from PD failed, startKey: ..."] ``` Search for `range_task` to see the reason for the failure: ```text [2025/01/03 18:42:54.492 +08:00] [Info] [range_task.go:310] ["canceling range task because of error"] [name=resolve-locks-runner] [startKey=] [endKey=7480000000000000195f728000000006967b0f] [error=" unexpected resolve err: commit_ts_expired: "] ``` After identifying the problem key, use `tikv-ctl` to convert it to hex format (note: the key here is unencoded, i.e., not memory comparable): ```bash tiup ctl:v7.5.2 tikv --to-hex "t\200\000\000\000\000\000\000\025_i\200\000\000\000\000\000\000\001\004\0060\277\024Y\`\000\006\003\200\000\000\000\000\000\004\362" # Output: # 7480000000000000155F698000000000000001040630BF14596000060380000000000004F2 ``` Then access the TiDB HTTP API to get its MVCC info: ```bash curl http://10.50.101.74:10080/mvcc/hex/7480000000000000155F698000000000000001040630BF14596000060380000000000004F2 ``` ```json { "key": "7480000000000000155F698000000000000001040630BF14596000060380000000000004F2", "region_id": 1938600, "value": { "info": { "lock": { "type": 1, "start_ts": 446066598787153941, "primary": "dIAAAAAAAAAVX2mAAAAAAAAAAgOAAAAAAAANDQ==" }, "writes": [ { "type": 1, "start_ts": 446066598787153941, "commit_ts": 446066599141310467 }, { "start_ts": 446066457229393926, "commit_ts": 446066457242501124, "short_value": "MA==" } ] } } } ``` We can see that in the MVCC for this key, comparing `start_ts` (446066598787153941), the transaction corresponding to the lock already has a write record (has been committed). This confirms that the key received a prewrite request from the same transaction after the commit request, which matches this bug. In this case, we can safely clean up the lock without data loss. ## Workaround ### Restore GC by Cleaning Up Problematic Locks Although there is no risk of data loss, the operation requires stopping the TiKV instance. #### Step 1: Locate the region where the lock is located The key entered via PD's region API must be encoded, while the key seen in TiDB logs is unencoded. Use `tikv-ctl` to encode it. There are two ways: #### Option A: Locate by encoded hex key ```bash # Encode the key tiup ctl:v7.5.2 tikv --encode "t\200\000\000\000\000\000\000\025_i\200\000\000\000\000\000\000\001\004\0060\277\024Y\`\000\006\003\200\000\000\000\000\000\004\362" # Output: # 7480000000000000FF155F698000000000FF000001040630BF14FF5960000603800000FF00000004F2000000FC # Locate the region with pd-ctl tiup ctl:v7.5.2 pd region key "7480000000000000FF155F698000000000FF000001040630BF14FF5960000603800000FF00000004F2000000FC" ``` #### Option B: Locate by encoded escaped key ```bash tiup ctl:v7.5.2 tikv --to-escaped '7480000000000000FF155F698000000000FF000001040630BF14FF5960000603800000FF00000004F2000000FC' # Output: # t\200\000\000\000\000\000\000\377\025_i\200\000\000\000\000\377\000\000\001\004\0060\277\024\377Y`\000\006\003\200\000\000\377\000\000\000\004\362\000\000\000\374 tiup ctl:v7.5.2 pd region key --format=encode 't\200\000\000\000\000\000\000\377\025_i\200\000\000\000\000\377\000\000\001\004\0060\277\024\377Y`\000\006\003\200\000\000\377\000\000\000\004\362\000\000\000\374' ``` Both commands return region info with `peers` listing the store IDs of all three replicas. #### Step 2: Stop TiKV and fix the problem region Process the three replica TiKV instances **one by one**: stop → fix → start → next. **Note:** Each TiKV downtime should not exceed 30 minutes. First, use read-only mode to confirm the abnormal locks: ```bash # Read-only check tikv-ctl --data-dir /data/tikv --log-level=info recover-mvcc --read-only -r {region-id} -p {pd-address} ``` If more than one key has a broken lock, confirm each one's MVCC status. Encode the key (it is already hex-encoded, so only escape it): ```bash tikv-ctl --to-escaped "key-in-hex-from-lock" ``` Then add the `z` prefix to get its MVCC: ```bash tikv-ctl --data-dir /data/tikv mvcc \ -k "zt\200\000\000\000\000\000\000\377\025_i\200\000\000\000\000\377\000\000\001\004\0060\277\024\377Y`\000\006\003\200\000\000\377\000\000\000\004\362\000\000\000\374" \ --show-cf=lock,write,default ``` If the lock's `start_ts` appears in the write CF, it confirms the same bug and the lock can be safely removed. Then run the actual repair: ```bash tikv-ctl --data-dir /data/tikv --log-level=info recover-mvcc -r {region-id} -p {pd-address} ``` After repair, re-run read-only mode to confirm recovery before starting this TiKV and moving to the next one. #### Step 3: Check GC status In most cases, GC will automatically recover after repairing all replicas for the current key. In rare cases, multiple transactions may be corrupt. It is recommended to fix TiKV instance by instance: ```bash # Get all exception locks on one TiKV (read-only) tiup ctl:v7.5.2 tikv --data-dir /DATA/disk2/shirly/tiup/tidb-data/tikv-20165 \ --log-level=info recover-mvcc --read-only -a -p 127.0.0.1:2379 --threads=10 ``` Confirm the MVCC status of each exception lock key, then repair the entire TiKV: ```bash tiup ctl:v7.5.2 tikv --data-dir /DATA/disk2/shirly/tiup/tidb-data/tikv-20165 \ --log-level=info recover-mvcc -a -p 127.0.0.1:2379 --threads=10 ``` ### Avoid Encountering This Issue Disable async commit and 1PC (if the performance impact is acceptable): ```sql set @@global.tidb_enable_async_commit = 0; set @@global.tidb_enable_1pc = 0; ``` ## Related Oncall - GC resolve lock stuck after upgrading from v4.0.14 to v6.5.8 ONCALL-7312 / TICKET-4713 - v5.4.3 GC stuck TICKET-2789 - v6.1.0 GC stuck for more than a year; multiple transactions found corrupt during the process TICKET-5537