--- name: transaction-oncall-handbook description: | Reference handbook for TiDB Cloud transaction on-call diagnosis. Use when investigating transaction-related issues: pessimistic lock bugs, async-commit correctness, stale/follower read anomalies, GC stalls, write hotspots, pipelined DML deadlocks, or client-go region-cache issues. Each entry documents phenomenon, root cause, workaround, fixed version, and related oncall tickets, covering TiDB/TiKV versions from v5.x to v8.5+. --- # Transaction Oncall Handbook Known issues and on-call cases for TiDB/TiKV transaction subsystems, organized by category. Source note: adapted from the Feishu `Transaction oncall handbook` . ## When To Use This Skill Use this skill when: - the issue is clearly transaction-related rather than a pure storage or compute problem - symptoms include deadlock, lock wait timeout, write conflict, `PessimisticLockNotFound`, stale-read anomalies, follower-read anomalies, GC stall, or transaction correctness risk - the incident may involve TiDB transaction behavior, TiKV MVCC behavior, or `client-go` region-cache behavior - you need known-version impact, workaround, and related oncall references for transaction cases ## First Checks - Identify whether the issue is correctness-related, latency-related, or availability-related. - Confirm whether the workload uses pessimistic transaction, async commit, 1PC, stale read, follower read, or pipelined DML. - Record the exact TiDB, TiKV, PD, and `client-go` versions. - Check whether the issue is scoped to one SQL pattern, one table, one region, or the whole cluster. - Decide whether the immediate goal is mitigating user impact or proving root cause. ## Routing Guide - `Unconfirmed Issues`: suspected correctness or behavior anomalies that still need confirmation - `Pessimistic Transaction`: deadlock, fair-lock, lock wait, or pessimistic-lock lifecycle issues - `Async Commit`: async-commit lock resolution and correctness-related issues - `Follower Read` / `Stale Read`: replica-read path, TSO scope, or linearizability issues - `System Variables and Configuration`: behavior changes caused by TiDB settings or compatibility gaps - `Client-go (Region Cache)`: region-cache, store-cache, request batching, or client memory issues - `Transaction Issues (TiDB Side)`: TiDB transaction-manager and DDL-adjacent transaction issues - `TiKV Storage`: transaction-visible corruption or replica inconsistency exposed by TiKV storage bugs - `Hotspots (TiDB Backoff)`: lock contention, write hotspots, or backoff-driven transaction latency - `Tool-Induced Transaction Anomalies (BR, PITR)`: restore or recovery workflows that introduce transaction anomalies ## Status Labels - `Fixed`: issue is fixed in one or more release lines - `In progress`: a fix is in progress or under active investigation - `Unconfirmed`: the issue is suspected but not yet fully confirmed - `Performance`: performance-related issue without a documented fixed PR - `Correctness`: correctness-sensitive issue ## Unconfirmed Issues ### Unconfirmed Correctness: Secondary Keys Stuck Because `lock.min_commit_ts` > `pk.commit_ts` - Phenomenon: one key failed to commit since the min_commit_ts > pk.commit_ts, we could see: - The commit information for primary key (write cf) - Lock information, but no write cf, which is different with [tikv#11187](https://github.com/tikv/tikv/issues/11187) - Root Cause: maybe there is a bug in compute min_commit_ts? - Related oncall: - v8.5.2 cdc Fail to resolve ts TICKET-6926 ONCALL-8061 - v8.5.1 cdc Fail to resolve ts ONCALL-8000 - Workaround: ONCALL-8000 / Hotfix 19020 - Hotfix [tikv#19020](https://github.com/tikv/tikv/pull/19020) ### Unconfirmed Correctness: Suspected correctness issue with async commit - Phenomenon: under network instability, data written with async commit may potentially have correctness issues. The root cause or reproducibility has not yet been confirmed. The following log shows that the primary key is committed while we could not found the secondaries keys. ``` [2025/03/26 20:13:41.523 +00:00] [WARN] [prewrite.go:415] ["async commit cannot proceed since the returned minCommitTS is zero, fallback to normal path"] [conn=730212792] [session_alias=] [startTS=456922239593087375] [2025/03/26 20:13:41.531 +00:00] [ERROR] [commit.go:200] ["2PC failed commit key after primary key committed"] [error="Error(Txn(Error(Mvcc(Error(TxnLockNotFound { start_ts: TimeStamp(456922239593087375), commit_ts: TimeStamp(456922240510592670), key: [?] })))))"] [txnStartTS=456922239593087375] [commitTS=456922240510592670] [keys="[?]"] ``` - Related oncall: - v7.5.3 index entries inconsistent with data rows TCOC-3394 202504 TCOC-3394 Investigation ### Unconfirmed: Follower Read latency spikes (10s+) caused by leader change ([tikv#18417](https://github.com/tikv/tikv/issues/18417)) - Description: During a follower read, the follower sends a read index request to the leader to confirm whether its replica is up to date. If a leader change occurs at this moment, the follower must wait for a full election timeout (default 10 seconds) before retrying. As a result, snapshot latency spikes of 10 seconds or more may be observed. - Workaround: disable follower read - Related oncall: - v8.1.2 TiKV async_snapshot long tail latency(10s+) investigation TCOC-3407 ### Unconfirmed: Loadbase split does not work for follower read - Phenomenon: Currently, loadbase split only applies to leader regions. Hotspots caused by follower read cannot be detected or split, which may lead to TiKV hotspots and increased latency. - Workaround: N/A. Consider disabling follower read - Related oncall: - v7.1.1 High TiDB and TiKV CPU usage TCID-11207/TCOC-3067 ### Unconfirmed Correctness: Data loss: Default value not found - Phenomenon: The tikv metrics panel reports a critical error: `default value not found`. TiKV logs also show this error, and the missing data key can be identified from the log details. - Root Cause: Data corruption or a program bug caused the write CF record to relate to a missing default CF value. In such cases, TiKV may panic. - Workaround: - Locate and delete the row corresponding to the missing key - By default, the configuration `panic_when_unexpected_key_or_data` is set to false, meaning TiKV will return an error without panic. If you prefer TiKV to panic immediately when such an error occurs, set this configuration to true. - Related oncall: - v6.5.5 After performing PITR restore and skipping part of the log, TiKV reported "Default value not found" TICKET-4903 ## Pessimistic Transaction ### Fixed: Locking logic changes when upgrading from low version to v7.X (cause more deadlocks) ([tidb#45533](https://github.com/pingcap/tidb/issues/45533)) - Phenomenon: After upgrading from low version to v7.X, clusters may report a significant increase in deadlocks. The root causes are changes in locking behavior: - In insert ignore and replace statements, duplicate keys were not locked in versions earlier than v6.1.6, this was fixed in [tidb#42121](https://github.com/pingcap/tidb/issues/42121) - In update statements, unchanged unique index keys were not locked in versions earlier than v6.5.2, fixed in [tidb#36438](https://github.com/pingcap/tidb/issues/36438) - Workaround: disable [tidb_lock_unchanged_keys](https://docs.pingcap.com/zh/tidb/stable/system-variables/#tidb_lock_unchanged_keys-%E4%BB%8E-v711-%E5%92%8C-v730-%E7%89%88%E6%9C%AC%E5%BC%80%E5%A7%8B%E5%BC%95%E5%85%A5) - Related oncall: - v8.5.5 Two session update one row parallelly hit deadlock error TICKET-8179 - upgrading from v6.1.3 to v7.5.6, there were a lot of deadlocks TCOC-4007 TCOCP-290 ### Fixed: Pessimistic Lock remain after session is killed ([tidb#61454](https://github.com/pingcap/tidb/issues/61454)) - Description: When a long-running SQL statement is manually terminated by Kill command, it triggers a close conn operation. If the session is still in an active transaction, TiDB will attempt a rollback. However, during rollback, the kill flag prevents pessimistic locks from being properly released, resulting in residual pessimistic locks. When SQL retries afterward, it must wait for the previous transaction's locks to expire (around 20 seconds) before proceeding, leading to noticeable latency for users. - Workaround: No immediate workaround; wait for the pessimistic locks to expire automatically (~20 seconds) - Fixed version: v8.5.4/v8.1.3/v7.5.8/v7.1.7/v9.0.0-beta.2 - Related oncall: - v7.5.1 Pessimistic locks remain after session killed GTOC-7634 ### `FOR UPDATE` lock does not block: other session acquires lock while the first still holds it (Fair Locking enabled) ([tidb#66571](https://github.com/pingcap/tidb/issues/66571)) - Detailed reference: [`references/tidb#66571.md`](references/tidb#66571.md) - KB: [Known-issue] FOR UPDATE lock does not block: other session acquires lock while the first still holds it (Fair Locking enabled) [tidb#66571](https://github.com/pingcap/tidb/issues/66571) - Description: Unexpected lock behavior for point get `FOR UPDATE` read on unique key. Two sessions try to lock the same key; the first should hold the key, while the second transaction locks the key successfully instead of timing out. - Affected version: v8.5+, master - Related oncall: - v8.5.4 Locking behavior change in v8.5.4 GTOC-8245 ### Fixed: TiDB reports PessimisticLockNotFound error after enabling fair_locking ([tidb#58279](https://github.com/pingcap/tidb/issues/58279)) - Detailed reference: [`references/tidb#58279.md`](references/tidb#58279.md) - Related KB: [Known issue] Report PessimisticLockNotFound when enable fair-lock [tidb#58279](https://github.com/pingcap/tidb/issues/58279) - Description: when fair lock mode is enabled, TiDB may encounter a PessimisticLockNotFound error under specific lock/unlock timing conditions: - When the current key's lock is released, TiKV wakes up the first transaction (txn1) in the waiting queue and grants the lock to it. - After receiving the wake-up notification, txn1 calls retryAggressiveLocking to retry acquiring the lock and resets its ttlManager, assuming the primary key might change. - However, if the primary key has not changed, TiDB skips the actual locking operation. - Since the ttlManager only runs when a pessimistic lock request is sent, skipping the lock acquisition prevents it from starting the ttl heartbeat. - As a result, the lock may expire after a period of time and gets cleaned up. When this happens, the transaction commit fails with an error similar to: ``` E, [2024-12-09T08:30:16.383161 #3664] ERROR -- : Mysql2::Error: tikv aborts txn: Error(Txn(Error(Mvcc(Error(PessimisticLockNotFound { start_ts: TimeStamp(454479204809965680), key: [116, 128, 0, 0, 0, 0, 7, 6, 178, 95, 114, 128, 0, 0, 0, 0, 0, 0, 3], reason: LockMissingAmendFail }))))): COMMIT ``` - Reproduction Example: ```sql /* init */ drop table if exists t; /* init */ create table t (id int primary key, v int); /* init */ insert into t values (1, 1); /* t2 */ begin; /* t2 */ update t set v = v + 1 where id = 1; /* t1 */ begin; /* t1 */ update t set v = v + 1 where id = 1; /* t2 */ commit; /* t1 */ select sleep(30); /* t1 */ rollback; ``` - Fixed version: v8.5.4/v8.1.3/v7.5.8/v7.1.7 - Workaround: ```sql set global tidb_pessimistic_txn_fair_locking=0; ``` - Related oncall: - v8.1.0 how to fix the PessimisticLockNotFound error TCOC-2917/TCID-10991 - v7.1.5 PessimisticLockNotFound APID-11148 ### In Progress: PessimisticLockNotFound caused by calculated commit_ts conflict with for_update_ts under fair lock mode ([tidb#41792](https://github.com/pingcap/tidb/issues/41792) [tidb#60679](https://github.com/pingcap/tidb/issues/60679)) - Description: when `tidb_pessimistic_txn_fair_locking` (enabled by default since v7.0.0) is enabled, in scenarios when commit_ts is calculated (e.g., async-commit or 1PC), lock conflicts may lead to PessimisticLockNotFound errors during transaction commit. Here is an event sequence example (see [tidb#60679](https://github.com/pingcap/tidb/issues/60679) for more details and reproduce steps): - The transaction attempts to lock k1. When awakened by fair_locking, it uses the current k1's last_commit_ts(=10) as for_update_ts and wakes the transaction. - TiDB fetches a new tso2 from PD as for_update_ts. Since the previous last_commit_ts(=10) is calculated, the new for_update_ts ts2 may be the same as last_commit_ts. - When retrying statements, if k1 no longer needs locking but k2 does, TiDB calls exitAggressiveLockingIfInapplicable and asynchronously: - Rolls back k1's lock, leaving a for_update_ts=10 rollback record. - Subsequent attempts to lock k1 still use for_update_ts=10 and was rolled back by the previous asynchronous rollback. - When committing the transaction, TiKV meets the rollback with for_update_ts=10 and reports PessimisticLockNotFound. - Root Cause: Calculated commit_ts may conflict with subsequently generated forUpdateTS. - Trigger condition: - Pessimistic lock enabled - tidb_pessimistic_txn_fair_locking enabled - 1PC or async_commit enabled (the commit tso is calculated) - Workaround: avoid trigger condition, choose one of the following: - Disable async-commit & 1PC: avoid using calculated commit_ts - Disable tidb_pessimistic_txn_fair_locking - Affected version: >7.0 - Related oncall: - v8.5.1 LockNotFound reason: LockMissingAmendFail TICKET-6827 / ONCALL-8034 ### Fixed: Pessimistic transaction reports 9007 write conflict error ([tikv#11612](https://github.com/tikv/tikv/issues/11612)) - Description: under normal circumstances, pessimistic transactions should not encounter a write conflict error. In the affected versions, when TiKV reads a committed-version newer than the current written version, it incorrectly returns a write conflict without checking whether the transaction is pessimistic. - In the fixed versions, TiKV adds an additional check for pessimistic transactions: - If the `for-update-ts` has not been updated, it returns a PessimisticLockNotFound error - If the `for-update-ts` has been updated, TiKV proceeds with the normal prewrite locking process. - Workaround: upgrade to the fixed version - Fixed version: v6.1.1/v5.4.2/v5.3.2/v5.1.5 - Related oncall: - v5.1.4 pessimistic transaction returned 9007 error TICKET-5466 ## Async Commit ### Fixed: TiDB resolve async commit lock stuck in infinite loop due to region merge ([tidb#59494](https://github.com/pingcap/tidb/issues/59494)) - Description: During pessimistic lock acquisition, if TiDB encounters locks left by async commit transactions, and within the same region some keys of T1 have locks while other keys of the same T1 in the same region do not have locks, TiDB may repeatedly try to resolve the async commit locks due to a logical flaw. This causes an infinite resolve-lock loop and eventually results in a timeout error. One common reason for this mixed lock state is region-merge, where previously separate regions containing different keys are merged together. - Symptoms: - A large number of query_check_secondary_locks requests appear in the metrics - Logs repeatedly show lock timeout for the same region (transaction) - Workaround: - Manually delete the remaining locks (TODO: How) - Upgrade to a fixed version - Mitigation: Disable the async-commit feature - Fixed version: v8.5.4/v8.1.3/v7.5.8/v7.1.7/v6.5.13 - Related oncall: - v7.1.5 update statement stuck TICKET-5641/ONCALL-7643 ## Pipelined DML ### Fixed: TiKV deadlock caused by pipelined DML leading to write stalls ([tikv#18384](https://github.com/tikv/tikv/issues/18384) [tikv#18385](https://github.com/tikv/tikv/issues/18385)) - Detailed reference: [`references/tikv#18384.md`](references/tikv#18384.md) - Related KB: [Known issue] deadlock in Pipelined DML in TIKV [tikv#18384](https://github.com/tikv/tikv/issues/18384) [tikv#18385](https://github.com/tikv/tikv/issues/18385) - Description: A deadlock bug in TiKV's handling of Pipelined DML can cause the entire write process to stall. When this issue occurs, you may observe: - scheduler pending commands continuously increasing - raftstore CPU usage dropping close to 0 - Workaround: - Avoid using pipeline DML together with resolved-ts feature; disable one of them. - If TiKV is already stuck, the only effective recovery is to restart the affected TiKV instance. - Affected version: v8.5.1 - Fixed version: v8.5.2 - Related oncall: - v8.5.1 TiKV stuck when using pipelined DML TCOC-3373 ## Follower Read ### Fixed: zone label misinterpreted as txn-scope in TiDB causes TSO retrieval error ([tidb#59402](https://github.com/pingcap/tidb/issues/59402)) - Phenomenon: When using stale read or setting tidb_replica_read to closest-replicas, this bug incorrectly sets the txn-scope to the read replica scope, causing TSO retrieval errors. Example logs (reference code: [snapshot.go#L443C7-L444](https://github.com/tikv/client-go/blob/d66e460ff577797cf2ba500542738c07936dcebc/txnkv/txnsnapshot/snapshot.go#L443C7-L444)): ``` [ERROR] [tso_client.go:662] ["[tso] dispatch tso request error"] [dc-location=us-east-1a] [error="[PD:client:ErrClientGetTSO]get TSO failed, unknown dc-location us-east-1a to the client"] [2025/02/11 04:08:22.920 +00:00] [ERROR] [region_request.go:773] ["validate read ts failed for request"] ``` - Fixed versions: v6.5.12 / v7.1.7 / v7.5.6 / v8.1.3 / v8.5.2 - Related oncall: - v8.5.1 validate read ts failed for request: unknown dc-location us-east-1a to the client GTOC-7393 NAID-1112 - v8.5.1 reported "validate read ts failed for request" TCOC-3538 ### Fixed: TiDB panic when `tidb_replica_read` is `closest-adaptive` ([tidb#60175](https://github.com/pingcap/tidb/issues/60175)) - Detailed reference: [`references/tidb#60175.md`](references/tidb#60175.md) - Phenomenon: When `tidb_replica_read` is set to `closest-adaptive` (default setting for newly created v7.5.6 cluster on cloud), there is a panic bug during selecting the target replica for reading. Example panic logs: ```text github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTask.func1 /workspace/source/tidb/pkg/store/copr/coprocessor.go:1116 runtime.gopanic /usr/local/go/src/runtime/panic.go:914 runtime.panicmem /usr/local/go/src/runtime/panic.go:261 runtime.sigpanic /usr/local/go/src/runtime/signal_unix.go:861 github.com/tikv/client-go/v2/internal/locate.(*Store).IsLabelsMatch /root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-.../internal/locate/region_cache.go:2780 github.com/tikv/client-go/v2/internal/locate.(*accessFollower).isCandidate /root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-.../internal/locate/region_request.go:1020 ``` - Workaround: change tidb_replica_read to `closest-replica` or `leader` to bypass this issue. - Fixed version: v8.5.3/v8.1.3/v7.5.7/v7.1.7 - Related oncall: - v7.5.6 invalid memory address causing SQL panic TCID-11399/TCOC-3308 ### Large follower resolved-ts gap caused by stale peers - Detailed reference: [`references/follower-resolved-ts-gap-stale-peer.md`](references/follower-resolved-ts-gap-stale-peer.md) - Phenomenon: follower resolved-ts gap becomes unusually large, often together with `leader missing` logs on stale peers. - Workaround: - Keep the stale peer as-is if it has no practical impact. - Otherwise remove the stale replica with `tikv-ctl tombstone`. - Related oncall: - v6.5.9 the max gap of follower resolved-ts became large `ONCALL-7019` ## Stale Read ### Fixed: Stale Read may cause data inconsistency when PD physical TSO update is delayed ([tidb#56809](https://github.com/pingcap/tidb/issues/56809)) - Detailed reference: [`references/tidb#56809.md`](references/tidb#56809.md) - Phenomenon: When executing a stale read query with a specific timestamp (super new timestamp), e.g., `SELECT ... AS OF TIMESTAMP`, TiDB may generate a TSO (e.g., tso-10) that is newer than PD's current physical TSO (e.g., tso-1). This can happen when PD experiences write latency or physical clock update delay. As a result, a series of events may occur in the following order (tso-1 < tso-5 < tso-10), leading to data consistency issues: - The stale read uses tso-10 to read data from TiKV and encounters an error `cannot set read timestamp to a future time` - The stale read retries and falls back to reading from the leader; TiKV leader then advances its max-tso to tso-10 - TiDB later reads using tso-5 (get from PD), resulting in inconsistent data due to timestamp reordering, with the following error: ```text Error 1105 (HY000): Txn 453423637808807957 Retrying aggressive locking with ForUpdateTS (453423637808807963) less than previous LockedWithConflictTS (453423637831352321) system: 2024-10-23 01:58:31.405 -0700 PDT logic: 1 ``` - Note: the above error may not always appear, but whenever LockedWithConflictTS appears, its logical part will always be 1. - Workaround: Fix the PD issue when physical time updates get stuck or delayed - Related KB: [Critical Bug] Stale read may break linearizability of Async Commit / 1PC transactions [tidb#56809](https://github.com/pingcap/tidb/issues/56809) - Fixed version: v8.5.0/v8.1.2/v7.5.5/v7.1.6/v6.5.12 - Related oncall: - v7.5.1 PD physical time update stuck for 10s, causing data inconsistency GTOC-7200 ### Fixed: TiDB misreads zone label causing dataNotReady during stale read ([tidb#55388](https://github.com/pingcap/tidb/issues/55388)) - Description: when a user configures a zone label in TiDB, executing the stale read may report errors like the following, with backoff reaching over 500ms: ```text [INFO] [snapshot.go:656] ["[for debug] snapshot.get"] [TxnScope=us-west-1] [replicaReadScope=us-west-1] [isGlobalStaleRead=false] labels = { zone = "us-west-1", host = "tidb1" } ``` - Root Cause: with a configured zone label, TiDB incorrectly uses the zone value as TxnScope and ReplicaReadScope during stale read. In fact, TxnScope functionality is largely deprecated, and these parameters should always be global. This issue has been fixed in [tidb#55388](https://github.com/pingcap/tidb/issues/55388). - Fixed version: v8.1.3/v7.5.4/v7.1.6/v6.5.11 - Workaround: upgrade to a fixed version or avoid using zone labels temporarily - Related oncall: - v7.5.2 stale read latency increase GTOC-7075 ### Stale-read latency spikes due to heavy schema loading - Detailed reference: [`references/stale-read-latency-heavy-schema-loading.md`](references/stale-read-latency-heavy-schema-loading.md) - Phenomenon: stale-read latency spikes appear when concurrent DDL forces repeated schema loading and schema-cache misses. - Workaround: upgrade to a version that includes the fix associated with `tidb#53428`. - Related oncall: - v6.5.4 `internal_others_leader` and schema load showed latency spikes `GTOC-6921` ## System Variables and Configuration ### Fixed: Pessimistic lock execution exceeds [innodb_lock_wait_timeout](https://docs.pingcap.com/zh/tidb/stable/system-variables/#innodb_lock_wait_timeout) without returning `Lock wait timeout exceeded` error ([tidb#56688](https://github.com/pingcap/tidb/issues/56688)) - Description: In TiDB, when a transaction encounters a write conflict, the system does not check `innodb_lock_wait_timeout` before retrying. As a result, even after exceeding the timeout threshold, the transaction continues running instead of returning a `Lock wait timeout exceeded` error. - Workaround: upgrade to the fixed version or set `max_execution_time` to limit query duration - Fixed version: v6.5.12/v7.1.6/v7.5.5/v8.1.3/v8.4.0 - Related oncall: - v6.5.11 One delete lasted for more than 1 hour TICKET-8121 - v6.5.3 update SQL kept running long after innodb_lock_wait_timeout and was killed by memory-quota ONCALL-7448 - v8.1.1 innodb_lock_wait_timeout not taking effect GTOC-7421 ### Performance: High transaction conflicts and tidb_lock_unchanged_keys causing increased latency - Description: when transaction conflicts are severe, the latch wait duration increases significantly, while async write and snapshot latency remain normal. In such cases, pessimistic locks tend to wait longer. Starting from v6.5.3, to ensure transaction consistency, TiDB began locking even unchanged keys in insert ignore and replace statements ([tidb#42210](https://github.com/pingcap/tidb/issues/42210)). This change may lead to performance regressions in high-conflict workloads. In v7.1.1 and v7.3.0+, you may mitigate this by setting the variable `tidb_lock_unchanged_keys` to OFF, but note that doing so may cause transaction inconsistency. - Metrics: - `tidb->backoff->lock resolved ops` shows increased lock fast counts, indicating frequent read/write conflicts. - The slow log shows higher retried execution time, meaning more tries due to pessimistic lock waits or deadlocks. - Workaround: - Optimize application logic to reduce transaction conflicts. - In v7.1.1 and v7.3.0+, disable unchanged key locking (note: this reduces lock contention from insert ignore and replace, but introduces a risk of inconsistent transactions): ```sql set global tidb_lock_unchanged_keys = OFF; ``` - Related oncall: - v7.5.6 write latency > 1s when inserting 1000 rows TCID-11623 - v6.5.3 increased GRPC heartbeat/resolve lock latency ONCALL-6980 - v7.5.4 TIKV read/write slowdown TICKET-6048 ONCALL-7823 ### Fixed: `max_execution_time` does not take effect on SELECT FOR UPDATE ([tidb#62960](https://github.com/pingcap/tidb/issues/62960)) - Phenomenon: according to TiDB's official doc, the session variable `max_execution_time` only applies to read-only SQL statements. However, in practice with the latest TiDB version, SELECT FOR UPDATE statements are not affected by this setting. In contrast, MySQL behaves differently: - max_execution_time applies to the query phase for SELECT FOR UPDATE - When lock contention occurs, the timeout is controlled by innodb_lock_wait_timeout instead - Workaround: Use `innodb_lock_wait_timeout` as a workaround to limit lock wait time. However, this cannot restrict queries that inherently take a long time due to scanning or processing large amounts of data. - Affected versions: <= v6.5.12/v7.5.6/v8.5.3 - Related oncall: - v7.5.6 max_execution_time incompatible behavior with MySQL NAID-11563 GTOC-7837 ### Fixed: `SELECT FOR UPDATE` semantic lost when plan cache is reused in `auto-commit` mode ([tidb#54652](https://github.com/pingcap/tidb/issues/54652)) - Phenomenon: In TiDB, when executing SELECT FOR UPDATE with plan cache enabled under auto-commit mode, although the transaction appears to be started and the SELECT FOR UPDATE is executed, the plan may reuse an autocommit plan. This causes the lock semantic to fail, and the data is not actually locked. - Workaround: disable [plan cache](https://docs.pingcap.com/zh/tidb/stable/system-variables/#tidb_enable_prepared_plan_cache-%E4%BB%8E-v610-%E7%89%88%E6%9C%AC%E5%BC%80%E5%A7%8B%E5%BC%95%E5%85%A5) ```sql set global tidb_enable_prepared_plan_cache=OFF; ``` - Fixed version: v8.1.1/v7.5.3/v7.1.6/v6.5.11/v6.1.7/v5.4.3 - Related oncall: - v6.5.9 TiKV select for update lock semantic not effective ONCALL-7284 ### Performance: tidb_table_cache_lease causing increased write latency - Phenomenon: when using Cache table, writes to the cached table may experience second-level latency. The duration of this latency is controlled by the global variable `tidb_table_cache_lease`. - Observation: TiKV GRPC duration remains normal, and TiDB runtime is normal, making this latency hard to detect due to lack of metrics. - Workaround: reduce `tidb_table_cache_lease` - Related oncall: - v7.5.4 insert latency jitter ST-1611 GTOC-7480 ### Performance: Unrestricted [`tidb_store_limit`](https://docs.pingcap.com/tidb/stable/system-variables/#tidb_store_limit-new-in-v304-and-v40) causes TiKV CPU saturation under high concurrent writes - Description: `tidb_store_limit` is unlimited by default, meaning there is no cap on the number of concurrent writes sent to TiKV. Under high concurrency, TiKV can easily hit resource bottlenecks, leading to: - Certain TiKV resources (CPU, I/O, etc.) reaching limits - Significant increase in TiKV processing latency - Workaround: - If TiKV resources cannot be scaled, it is recommended to reduce `tidb_store_limit` to limit concurrency and prevent TiKV from being overloaded. - Related oncall: - v7.1.5 high lock wait timeout TCOC-2439/TCID-10595 ### Performance: Setting txn-total-size-limit may cause TiDB OOM - Description: When `txn-total-size-limit` is set, the transaction size is no longer limited by `tidb_mem_quota_query`. As a result, large transactions can consume excessive memory and cause TiDB to run out of memory. - Workaround: - Avoid setting `txn-total-size-limit` when system memory of TiDB is tight, and ensure the value is properly tuned to prevent oversized transactions. - Related oncall: - v6.5.3 delete SQL caused TiDB OOM ONCALL-7457/TICKET-5125 ## Client-go (Region Cache) ### TiDB memory retention risk after client-go PriorityQueue change ([client-go#1834](https://github.com/tikv/client-go/issues/1834)) - Detailed reference: [`references/client-go#1834.md`](references/client-go#1834.md) - KB: [Known-issue] TiDB Memory Retention Risk After client-go PriorityQueue Change [client-go#1834](https://github.com/tikv/client-go/issues/1834) [client-go#1095](https://github.com/tikv/client-go/issues/1095) - Fixed version: v8.5.5 - Affected version: v8.0.0+ - Related oncall: - v8.5.3 After insert finished, the memory usage of tidb didn't released TICKET-8201 ### Fixed: Inconsistent store liveness between region-cache and store-cache when follower-read enabled ([client-go#1823](https://github.com/tikv/client-go/issues/1823) [client-go#1826](https://github.com/tikv/client-go/issues/1826)) - Detailed reference: [`references/client-go-store-pointer-inconsistency.md`](references/client-go-store-pointer-inconsistency.md) - Description: When a TiDB cluster is using follower-read (for example, using prefer-leader), and a TiKV or TiFlash store updates its address or labels, an inconsistency may occur between the region-cache and store-cache. A new store will be created and replace the old one in store-cache with the same store id, while the old store pointer may still be held in region-cache. This may cause: - For regions whose leader (replica) is already cached in region-cache: the status will never change and the related leader keeps unavailable - For newly accessed regions (not previously cached), the new store pointer is used and the leader (replica) will be available - Key logs: ```text [INFO] [store_cache.go:487] ["store address or labels changed, add new store and mark old store deleted"] [store=100][old-addr=tikv-old-address...] [INFO] [store_cache.go:648] ["[health check] store became reachable"] [storeID=100] ``` - Key metrics: - Most requests still went through follower read despite prefer-leader being enabled. - The related TiKV still has some QPS. - Workaround: restart the affected TiDB instance - Related oncall: - v8.5.5 inconsistent store liveness between region-cache and store-cache when follower-read enabled GTOC-8095 ### TiKV client panic in client_batch ([client-go#1839](https://github.com/tikv/client-go/issues/1839) [client-go#1841](https://github.com/tikv/client-go/issues/1841)) - Description: tikv client panic in client_batch ```text [ERROR] [client_batch.go:307] [batchSendLoop] [r="runtime error: slice bounds out of range [-33:]"] github.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop.func1 /tidb/tikv-client-go/internal/client/client_batch.go:309 runtime.gopanic /usr/local/go/src/runtime/panic.go:914 runtime.goPanicSliceB /usr/local/go/src/runtime/panic.go:154 ``` - Workaround: N/A - Related oncall: - v8.5.2 Reopen TiDB panic in client_batch GTOC-8147/GTOC-8146 NAID-11778 ### In Progress: TiKV with empty-addr added into cache causes region unavailable ([client-go#1746](https://github.com/tikv/client-go/issues/1746)) - Phenomenon: Due to the bug, when a TiKV store encounters an exception, an empty addr tikv may be written into the cache and cannot be updated when the tikv recovers. As a result: - regions on this store become unavailable, and logs contain a large number of errors such as `throwing pseudo region error due to no replica available` - PD gets lots of scan-region requests - Trigger condition: the TiKV node experienced an exception - Workaround: restart the TiDB instance - Related oncall: - v8.1.2 Problematic TiKV node and high query latency GTOC-7864 NAID-11583 ### In Progress: Update unique key check requests (BatchGet) within the same region are not batched together ([tidb#55427](https://github.com/pingcap/tidb/issues/55427)) - Detailed reference: [`references/tidb#55427.md`](references/tidb#55427.md) - Description: When updating rows that contain unique keys, TiDB performs Get requests to check uniqueness. However, if the update involves multiple unique key checks on the same region or multiple rows that reside within the same region, TiDB fails to group those Get requests into a single BatchGet. This causes higher transaction latency. - Workaround: use `INSERT ON DUPLICATE KEY UPDATE` instead of UPDATE - Fixed version: not v8.5.3, maybe fixed by [client-go#1411](https://github.com/tikv/client-go/pull/1411) - Related oncall: - v7.5.2 slow update performance TCOC-2474 - v7.5 insert statement triggered over 2W batchGet requests, causing overall slowness TCID-10713 ### Fixed: Transaction retry does not release membuf memory, causing memory quota exceed and failure ([client-go#1451](https://github.com/tikv/client-go/issues/1451) [tidb#56837](https://github.com/pingcap/tidb/issues/56837)) - Description: When a transaction fails due to a write conflict error and retries, TiDB does not release the memory allocated in the previous membuf. This can cause the retry transaction to quickly exceed `tidb_mem_quota_query` and fail. Example warning logs: ```text [WARN] [expensivequery.go:118] ["memory exceeds quota"] [cost_time=267.783085908s] [mem_max="3223271582 Bytes (3.00 GB)"] [sql="update ACT_PLATFORM_CHANNEL_EXEC_CONTACT set ACT_NO_RESPOND_TM = ? where CURR_NODE_BATCH_ID = ?"] ``` - Workaround: upgrade to fixed version. v8.4.0 [client-go#1451](https://github.com/tikv/client-go/issues/1451) is not specifically for this issue, but it fixes the memory issue. - Fixed version: v8.1.2/v7.5.5/v7.1.7/v6.5.12 - Related oncall: - v6.5.3 update SQL retry after write-conflict quickly exceeded tidb_mem_quota_query and was killed ONCALL-7448 ## Transaction Issues (TiDB Side) ### Performance: Insert latency increases when setting specified IDs on auto_increment columns - Description: When a user frequently inserts records with manually specified IDs into an auto_increment column, TiDB performs a rebase of the auto_increment meta information for each insert. This frequent rebasing leads to noticeable write latency increases. - Key metrics: - In the tidb->meta panel: - autoID QPS: reflects the frequency of rebase operations - autoID duration: shows the latency of each rebase operation - In the slow log, latency during the prepare phase increases since rebasing occurs in this phase. - Workaround: - If frequent manual specified ID insertion is required, consider removing the auto_increment property to avoid repeated rebase overhead. - Related oncall: - v5.4.0 auto_increment auto rebase ONCALL-6989 ### In Progress: TiDB goroutine leak when using Cached table with TiFlash replica ([tidb#63329](https://github.com/pingcap/tidb/issues/63329)) - Description: A goroutine and memory leak occurs when cached table is used together with a TiFlash replica. Detailed reproduction steps are described in the issue. - Workaround: - Restart the TiDB instance - Avoid using cached table with tiflash replicas simultaneously (this is a non-standard operation and not recommended) - Related oncall: - v8.5.2 TiDB goroutine leak when using cached table + tiflash replica at the same time GTOC-7885 ### Fixed: TiKV frequent panic due to committed version with start_ts=0 ([tidb#57980](https://github.com/pingcap/tidb/issues/57980)) - KB: Critical+Bug+TiKV+panic+after+adding+index+in+ingest+mode - Phenomenon: When `tidb_ddl_enable_fast_reorg` is enabled but `tidb_enable_dist_task` is not. If TiDB generates a large number of SST files and uploads them to TiKV (especially for very large tables), the batch generation may not finish all SSTs at once. During this process, start_ts may be left uninitialized, causing some SST files to have start_ts=0. - In version v8.1.1, tidb_enable_dist_task is enabled by default, but clusters upgraded from older versions may have it set to false. - Panics mainly occur in: - get_keys in TiKV GC worker - Prewrite checks for the most recent committed versions ```text // panic during gc gc_keys(XX) [FATAL] [lib.rs:478] ["assertion failed: self.0 > 0"] // backtrace: txn_types::timestamp::TimeStamp::prev ... // panic during prewrite step [FATAL] [lib.rs:478] ["assertion failed: !last_change_ts.is_zero()"] // backtrace: prewrite::PrewriteMutation::check_for_newer_version ... ``` - Trigger Conditions: This may occur when all of the following conditions are met: - tidb_ddl_enable_fast_reorg is enabled - tidb_enable_dist_task is disabled - Adding an index on a large table - Workaround: - STOP the related DDL job if needed: ```sql ADMIN CANCEL DDL JOBS job_id [, job_id] ...; ``` - Identify the problematic indexes, drop them - Wait for next GC; if GC stalls, try to recover it. After the GC lifetime, no panic should happen. - Disable DDL reorg: ```sql set global tidb_ddl_enable_fast_reorg=OFF; ``` - If tidb_ddl_enable_fast_reorg is enabled, ensure tidb_enable_dist_task is also enabled. - Fixed version: v8.1.2 - Related oncall: - v8.1.1 tikv panic frequently NAID-11667/GTOC-7948 NAID-11667 Incident Report - v8.1.1 TiKV frequent panic due to commit record with start_ts=0 GTOC-7280/NAID-10977 ### Fixed: TiDB transaction manager bug causes Invalid Transaction error ([tidb#35686](https://github.com/pingcap/tidb/issues/35686) [tidb#35679](https://github.com/pingcap/tidb/issues/35679)) - Phenomenon: When executing certain special transactions in TiDB, due to a bug in the transaction manager (TxnManager), committing the transaction may fail with error: ```sql mysql> select * from information_schema.DDL_JOBS as of timestamp now(); ERROR 8024 (HY000): invalid transaction ``` - Workaround: - Explicitly start the transaction with `BEGIN...COMMIT` - Upgrade TiDB to v6.2.0 or above - Related oncall: - v6.1.6 TiDB cannot commit after setting read-only TICKET-4531 ONCALL-7246 ### In Progress: GC stalls due to transaction write corruption during network instability ([tikv#11187](https://github.com/tikv/tikv/issues/11187)) - Detailed reference: [`references/tikv#11187.md`](references/tikv#11187.md) - Symptom: TiKV GC stalls with resolved-ts frozen, showing critical errors in gc_worker logs: ```text [2024/07/26 14:48:16.909 +08:00] [ERROR] [gc_worker.go:226] ["[gc worker] runGCJob"] [error="unexpected resolve err: commit_ts_expired: "] ``` - Root Cause and workaround: [Known Issue] GC is stuck by corrupt transaction [tikv#11187](https://github.com/tikv/tikv/issues/11187) - Related oncall: - v6.5.5 GC is stuck TICKET-6337 ONCALL-7911 - v6.5.2 Resolve lock failed TICKET-5916 ONCALL-7745 - v4.0.14->v6.5.8 GC got stuck ONCALL-7312/TICKET-4713 - v5.4.3 GC is stuck TICKET-2789 - v6.1.0 GC is stuck TICKET-5537 - v6.5.2 backup failed TICKET-5423 - ONCALL-8000 - TICKET-6926 ### In Progress: Duplicate records appear in the DATA_LOCK_WAITS ([tidb#60227](https://github.com/pingcap/tidb/issues/60227) [tidb#37771](https://github.com/pingcap/tidb/issues/37771)) - Description: the `information_schema.data_lock_waits` table may contain duplicate entries, causing the same lock wait information to appear multiple times. - Related oncall: - v6.5.6 multiple duplicate entries found in DATA_LOCK_WAITS TICKET-5853/ONCALL-7726 ### In Progress: TiDB SELECT FOR UPDATE behaves inconsistently with MySQL ([tidb#60281](https://github.com/pingcap/tidb/issues/60281) [tidb#4619](https://github.com/pingcap/tidb/issues/4619)) - Description: In TiDB, when executing a SELECT FOR UPDATE statement that includes a subquery, the target rows may not be properly locked, resulting in behavior inconsistent with MySQL. - This may lead to potential data races or update conflicts in concurrent workloads, especially for applications relying on row-level locking semantics. - Related oncall: - v8.5.0 Inconsistencies between TiDB and MySQL for SELECT FOR UPDATE GTOC-7488 ## TiKV Storage ### Fixed: Some replicas report default not found after TiKV restart ([tikv#16706](https://github.com/tikv/tikv/issues/16706)) - Detailed reference: [`references/tikv#16706.md`](references/tikv#16706.md) - Phenomenon: After TiKV restarts due to certain or uncertain reasons, a RocksDB bug ([tikv#16706](https://github.com/tikv/tikv/issues/16706)) is triggered, causing data loss on some TiKV instances. Key information: - Not all replicas were corrupted. In the current case, only some replicas are corrupted. Reading from the leader may report default not found, but when we try to read from followers it may work normally. This indicates data inconsistency between replicas. - Workaround: - Transfer the leader to a healthy replica: ```bash pd-ctl operator add transfer-leader ``` - Remove the corrupted replica: ```bash pd-ctl operator add remove-peer ``` - Wait for PD to automatically add new replicas for the region, or add replicas manually. - Fixed version: v7.5.2/v7.1.5/v6.5.9/v6.1.8/v5.4.4 - Related oncall: - v6.5.3 leader reports default not found after tikv restarts, while follower read works normally ONCALL-7554 ## Hotspots (TiDB Backoff) TiDB backoff increase is usually related to write hotspots, and generally arises from two main causes: - Key-level hotspot: Severe transaction lock conflicts - Transactions contending for the same keys result in frequent retries, causing increased backoff. - Region-level hotspot: Region related retries caused by scheduling - Requests may be retried due to region scheduling, migration, or PD/scheduler operations, which increases backoff duration. ### In Progress: Continuous writes causing region hotspots and increased backoff ([tidb#16920](https://github.com/pingcap/tidb/issues/16920)) - Description: When large transactions write to hotspot regions, TiKV tries to pre-split these regions. However, for continuous data writes, even the split regions may still become hotspots. TiKV frequently detects these hotspots and performs splits, causing frequent changes to region metadata. As a result, TiDB experiences increased backoff. Backoff errors are generally region-related. - Workaround: - Use randomization or other strategies to avoid creating hotspot regions during writes. - [tidb#16920](https://github.com/pingcap/tidb/issues/16920) do pre-split during 2PC to avoid hotspot - Related oncall: - v6.5.4 large transaction execution triggered significant backoff ONCALL-7334/TICKET-4786 ### Performance: Concentrated hotspots (hot small table lock contention) causing thundering herd - Description: when a cluster experiences highly concentrated hotspots, after the current transaction holding the lock finishes, it first wakes up the first transaction in the wait queue. After `pessimistic-txn.wake-up-delay-duration`, all transactions in the queue are woken up simultaneously. If the wait queue is long, these transactions retry concurrently, further aggravating the hotspot issue. - Workaround: - TiDB is not well-optimized for highly concentrated hotspot scenarios; try to mitigate this issue from the application side. - Increase `pessimistic-txn.wake-up-delay-duration` appropriately to reduce unnecessary retries. - Related oncall: - v8.1.0 lock contention on a small table GTOC-719 ### Performance: Hot insert conflicts causing high write latency - Phenomenon: When transactions encounter severe write-write conflicts, pessimistic lock on TiKV will wait for `pessimistic-txn.wait-for-lock-timeout` (default 1s) and then return KeyIsLocked. TiDB will retry, and in high conflict scenarios, repeated retries quickly increase write latency. Example flow: - First Execution: - TiDB attempts to acquire a pessimistic lock on relevant key via PessimisticLock RPC to TiKV - TiKV detects an existing lock on the key, waits for pessimistic-txn.wait-for-lock-timeout (~1s), then returns KeyIsLocked error - TiDB receives KeyIsLocked. If the lock is recent (newer than current lock), it will not resolve it and re-sends the PessimisticLock RPC. If we see KV_total: 7.037876296, suppose 7 RPCs are sent, each taking ~1s, all returning KeyIsLocked. - On the 8th RPC, the lock is released, but another transaction has written new data, resulting in WriteConflict error, and then TiDB retries using a new for_update_ts. - Second Execution: - Quick errors with `[kv:1062]Duplicate entry` - Workaround: - No direct workaround; reduce conflict probability from the application side. - Related oncall: - v7.5.4 insert retries delayed up to 7s ST-1404/TCOC-3100 ### In Progress: AutoCommit transaction hotspot conflict optimization ([tidb#58675](https://github.com/pingcap/tidb/issues/58675) [tidb#58676](https://github.com/pingcap/tidb/issues/58676)) - Phenomenon: When Auto-commit transactions experience heavy conflicts, overall processing latency increases significantly. By default, for auto-commit transactions: - The first prewrite attempt uses optimistic locking - If lock conflict occurs, it falls back to pessimistic locking - In scenarios with extremely high contention, the probability of lock conflicts under optimistic mode is very high. In such cases, performing a resolve lock almost always fails (since the lock is already held by another running transaction), making the operation meaningless and adding extra overhead. In [tidb#58676](https://github.com/pingcap/tidb/issues/58676), this was optimized: when the first optimistic prewrite fails, TiDB skips the resolve lock step and switches directly to pessimistic mode, reducing unnecessary latency. - Workaround: - Modify SQL to use explicit transactions (pessimistic transaction): `BEGIN...COMMIT` - Enable the TiDB configuration `pessimistic-auto-commit` - Upgrade to the version that includes this optimization. - Related oncall: - v7.5.3 TiDBP999 LatencySurgeForBingoMessage O11Y-1698640 ### Performance: TiKV CPU hotspot caused by hot write keys - Description: when a small number of keys become highly contended or frequently accessed, TiKV may experience CPU hotspots. In such cases, the QPS of batch-get, pessimistic_lock, pessimistic_rollback usually increases significantly. We can identify these hot keys by querying the `information_schema.data_lock_waits` table. When multiple transactions concurrently update the same key, they frequently block each other. For example, if three transactions are waiting for the same key, once the first transaction is woken up, the other two are also awakened. All three will then attempt to acquire the same lock simultaneously. If each transaction involves many such keys, this "lock wake-up storm" can significantly worsen CPU usage. - Why does PD not scatter such hotspots? This type of hotspot mainly results from lock contention. While it heavily consumes CPU resources, it does not generate significant read/write traffic in raftstore, therefore: - load-base-split cannot split regions based on batch-get requests. - PD's hotspot scheduler cannot redistribute regions based on CPU usage alone. - Workaround: - Immediate mitigation: Use evict-leader to evict region leaders containing the hot keys, attempting to redistribute the load. - Best practice: Optimize the application logic to avoid small-scale hot write patterns, which are not well-suited for TiDB's distributed architecture. - Related oncall: - v7.1.1 TiKV CPU hotspot persisted after OOM TCOC-2007 ## Tool-Induced Transaction Anomalies (BR, PITR) ### Fixed: BR restore fails to rebase auto-random ID, causing duplicated committed key ([tidb#52255](https://github.com/pingcap/tidb/issues/52255)) - Description: when using BR for data restore, if a table contains an auto_random ID column, there is a critical bug that BR does not rebase the auto_random ID after restore. This may lead to duplicate ID conflicts. If the application uses `INSERT ON DUPLICATE KEY UPDATE` statements without explicitly specifying an ID, the system will generate a new ID before updating. Once ID duplication occurs, the update may be applied to the wrong row (based on the duplicated ID) instead of the intended unique key, leading to data inconsistency and writing errors. - Workaround: manually rebase the auto_random column for affected tables using `ALTER TABLE ... AUTO_RANDOM_BASE`: - Query the current maximum auto-increment bit ID: ```sql select (id & (1 << (64-1-5)) - 1) as inc_id from BB order by inc_id desc limit 1; ``` - Query the current NEXT_GLOBAL_ROW_ID: ```sql show table BB next_row_id; ``` - Rebase the table using the maximum value: ```sql alter table BB auto_random_base= max(inc_id) + 1; ``` - Affected versions: - v6.1.5-v6.1.7 - v6.5.0-v6.5.8 - v7.1.0-v7.1.4 - v7.5.0-v7.5.1 - Fixed version: v7.5.1/v7.1.5/v6.5.9 - Related oncall: - v6.5.7 Transaction correctness issue TCOC-2010