# [Known Issue] Deadlock in Pipelined DML in TiKV ([tikv#18384](https://github.com/tikv/tikv/issues/18384), [tikv#18385](https://github.com/tikv/tikv/issues/18385))

Source note: adapted from GitHub issues [tikv#18384](https://github.com/tikv/tikv/issues/18384) and [tikv#18385](https://github.com/tikv/tikv/issues/18385).

## Issue

- Link: [tikv#18384](https://github.com/tikv/tikv/issues/18384), [tikv#18385](https://github.com/tikv/tikv/issues/18385)
- Symptom: a deadlock bug in TiKV's Pipelined DML path can cause writes to stall completely

## Phenomenon

When this issue occurs, the whole write path may freeze:

- `scheduler pending commands` keeps increasing
- `raftstore` CPU usage drops close to `0`
- writes remain stuck until the affected TiKV instance is restarted

## Root Cause

This issue is caused by a deadlock in TiKV's implementation of Pipelined DML.
Two different lock paths use inconsistent lock ordering:

- the `get` path used by `resolved-ts`
- the write path used by Pipelined DML

Because these two modules acquire locks in different orders, TiKV can enter a deadlock state and stop making progress on writes.

## Diagnostic Steps

1. When writes are stuck, identify the problematic TiKV instances from `TiKV-details -> Cluster -> QPS`.

2. Check `TiKV-details -> Scheduler` and verify that the following metrics keep increasing until the TiKV is restarted:

   - `scheduler pending commands`
   - `scheduler running commands`
   - memory quota related usage

3. Check `TiKV-details -> Duration` and confirm that `raftstore` writing is not slow and that there is almost no write flow.

## Workaround
Because the deadlock occurs between the Pipelined DML path and the `resolved-ts` module, avoid using them together. Disable either one of the following:
- Pipelined DML
- `resolved-ts`
If the deadlock has already occurred, the only effective recovery is to restart the affected TiKV instance.

## Affected Version
- v8.5.1

## Fixed Version
- v8.5.2

## Related Oncall
- v8.5.1 TiKV stuck when using pipelined DML TCOC-3373