# [Known Issue] gRPC thread hotspot causes long-tail latency in TiKV ([tikv#11009](https://github.com/tikv/tikv/issues/11009))

Source note: adapted from GitHub issue [tikv#11009](https://github.com/tikv/tikv/issues/11009).

## Issue

- Link: [tikv#11009](https://github.com/tikv/tikv/issues/11009)
- Summary: TiKV stream RPC connections are bound to fixed gRPC worker threads after establishment. Under heavy traffic, this can create thread-level load imbalance and long-tail latency.

## Phenomenon

- CPU usage in `TiKV-details -> CPU -> gRPC pool CPU` is visibly uneven when grouped by thread name.
- The TiDB -> TiKV RPC layer shows long-tail latency.
- `replica raft log duration` also shows long-tail latency.
- In some cases, symptoms are accompanied by `LockNotFound` or `TxnLockNotFound` bursts because the hottest streams are delayed disproportionately.

## Root Cause

TiKV keeps long-lived stream RPCs pinned to the gRPC threads that accepted them. The most important streams include:

- client read/write requests through `BatchCommands`
- inter-TiKV raft communication through `Raft` and `BatchRaft`

Because the binding is static, a small number of hot streams can overload only some gRPC threads while other threads remain underutilized. This imbalance creates long-tail latency in critical request paths.

## Diagnostic Steps

1. Check `TiKV-details -> CPU -> gRPC pool CPU` and confirm whether thread-level utilization is skewed.
2. Check TiDB -> TiKV RPC latency and confirm that only the tail grows significantly.
3. Check `replica raft log duration` and confirm that raft-side latency also shows a long tail.
4. Correlate the timing with write surge, raft traffic increase, or hot read/write paths.

## Workaround

- Increase `server.grpc-concurrency`.
- Increase `server.grpc-raft-conn-num`.
- Restart TiKV after changing these parameters.

## Related References

- Internal tracking: [FRM-2527](https://internal.pingcap.net/jira/browse/FRM-2527)
- Client-go enhancement: [client-go#1098](https://github.com/tikv/client-go/pull/1098) (not GA)

## Related Oncall

- v7.5.6 gRPC CPU hotspot during write surge `TCOC-3877`
- v7.5.2 `LockNotFound` errors `TICKET-6135`
- v7.5.3 TiKV RPC layer latency spike `TCID-11431` / `TCID-11441` / `TCOC-3397`
- v6.5.10 replica raft log duration spike and raft log gap increase
- v6.5.10 `TxnLockNotFound` burst `TICKET-5912` / `ONCALL-7746`