# [Critical Bug] Stale read may break linearizability of Async Commit / 1PC transactions ([tidb#56809](https://github.com/pingcap/tidb/issues/56809))

Source note: adapted from GitHub issue [tidb#56809](https://github.com/pingcap/tidb/issues/56809).

## Issue

- Link: [tidb#56809](https://github.com/pingcap/tidb/issues/56809)
- Summary: stale read may use a timestamp that is newer than the maximum TSO already allocated by PD. If the request then falls back to leader read, TiKV may incorrectly advance its `max_ts`, causing Async Commit / 1PC transactions to commit with an excessively large `commit_ts`. As a result, writes from a transaction may be invisible to reads that start strictly after that transaction commits, which breaks linearizability and can lead to follow-up data corruption at the application layer.

## Phenomenon

- The cluster uses stale read, for example `AS OF TIMESTAMP` or other stale-read syntax.
- Data written by Async Commit / 1PC transactions is sometimes not immediately visible after commit.
- The issue can be rare and difficult to reproduce consistently.
- In some cases, TiDB logs show errors like:

```text
Error 1105 (HY000): Txn 453423637808807957 Retrying aggressive locking with ForUpdateTS (453423637808807963) less than previous LockedWithConflictTS (453423637831352321)
```

- When this issue happens and `LockedWithConflictTS` is observed, its logical part is always `1`.

## Root Cause

When stale read is used, TiDB should guarantee that the chosen read timestamp does not exceed the maximum timestamp already allocated by PD:

```text
read_ts <= pd_allocated_max_ts
```

In affected versions, this guarantee is missing. Stale read may therefore use a future timestamp in scenarios such as:

- the user manually specifies a very new read time
- machine clocks drift
- PD write latency is high, so timestamp allocation lags behind real time

If the stale read request to TiKV fails, TiDB may retry it in leader-read mode with the `stale_read` flag unset. For a normal non-stale read, TiKV uses the request's `read_ts` to advance its local `max_ts`.

That `max_ts` participates in the calculation of `commit_ts` for Async Commit / 1PC transactions. If a future `read_ts` incorrectly pushes `max_ts` upward, a later Async Commit / 1PC transaction can commit with an excessively large `commit_ts`. Once that happens, the ordering between writes and later reads may become inconsistent, breaking linearizability.

## Diagnostic Steps

1. Confirm that stale read is actually used in the cluster. If the workload never uses stale read, this issue does not apply.

1. Check whether the application has observed lost linearizability for Async Commit / 1PC transactions, for example:
   - a transaction commits successfully
   - a read that starts strictly after the commit still cannot see the written data

1. If you can identify suspicious transactions, inspect their timestamps. Abnormally committed Async Commit / 1PC transactions related to this issue have a `commit_ts` whose logical part is always `1`.

1. Search TiDB logs for this error pattern:

```text
Retrying aggressive locking with ForUpdateTS (...) less than previous LockedWithConflictTS (...)
```

If `LockedWithConflictTS` appears and its logical part is `1`, confidence in this diagnosis increases.

1. Do not exclude this issue only because the above error is missing. The issue can still occur without printing that log.

1. Check whether the cluster also has one or more contributing conditions:
   - clock drift on cluster machines
   - poor PD write performance
   - stale read with a timestamp that is too close to "now" or effectively in the future from PD's perspective

1. Use `pd-ctl tso` if you need to inspect the physical/logical parts of a TSO directly.

## Workaround

The only strict workaround is to stop using stale read.

If stale read cannot be removed immediately, reduce the probability of hitting the bug by doing all of the following:

- Avoid using stale read with timestamps that are too new.
- Make sure machine clocks across the cluster are precise and stable.
- Make sure PD has healthy write performance and does not lag when updating physical time.

## Affected Version

- v6.5.0-v6.5.11
- v7.1.0-v7.1.5
- v7.5.0-v7.5.4
- v8.1.0-v8.1.1

## Fixed Version

- v8.5.0
- v8.1.2
- v7.5.5
- v7.1.6
- v6.5.12

## Related Oncall

- v7.5.1 PD physical time update stuck for 10s, causing data inconsistency GTOC-7200