# Network Issue Diagnosis

Read this file when transport latency or connectivity failure looks more important than TiKV local processing time.

This guide is derived from the internal KB page:
- `[Troubleshooting Guide] Handling Latency Spikes in TiDB Due to Network Issues`

## When To Read

Read this file when the active branch is `network-issues` and the main question
is whether dominant latency is on transport rather than inside TiKV local
processing.

## Goal

The goal is to determine:
- which hop is affected
- whether the issue is interruption, jitter, or bandwidth saturation
- which metric proves transport time is dominant
- what evidence rules out gRPC-thread or host-stall false positives

## Key Questions

Keep these questions aligned with the main skill:
- which hop is affected: TiDB <-> TiKV, TiKV <-> TiKV, TiDB <-> PD, or PD <-> PD
- is the issue interruption, latency jitter, or bandwidth saturation
- which metric proves transport time is dominant over TiKV local processing
- what evidence rules out gRPC thread exhaustion or host stalls as false positives
- is the issue really on TiKV or upstream of TiKV

## Guardrails

- if the dominant time is on TiDB <-> PD or PD <-> PD, say explicitly that it is not a TiKV local root cause
- do not conclude network failure from `unreachable` alone
- keep TiDB-side and TiKV-side transport hops separate
- avoid using a transport label when TiKV local stages are already clearly dominant

## 1. What Network Diagnosis Must Answer

Every network-focused analysis must answer these questions in order:

1. Which hop is affected: TiDB <-> TiKV, TiKV <-> TiKV, TiDB <-> PD, or PD <-> PD?
2. Is the issue interruption, latency jitter, or bandwidth saturation?
3. Which metric proves transport time is dominant over TiKV local processing?
4. What evidence rules out gRPC thread exhaustion or host stalls as a false network signal?
5. Is the issue really on TiKV, or is it upstream of TiKV?

Important wording rule:
- if the dominant time is on TiDB <-> PD or PD <-> PD, say explicitly that this is not a TiKV local root cause

## 2. First Split: Which Hop

Use this split before deep read-path or write-path diagnosis:

- TiDB <-> TiKV:
  - `KV Request duration` and `RPC Layer Latency` rise together
  - TiDB slow logs show `tikvRPC` backoff or suspicious RPC time
  - batch-client reconnect or receive-delay metrics rise
- TiKV <-> TiKV:
  - `unreachable` failures appear
  - `Commit log duration` rises above `Append log duration` while append / apply stay relatively normal
- TiDB <-> PD:
  - `PD TSO RPC Duration` rises while PD server-side TSO handle time stays low
- PD <-> PD:
  - etcd peer round-trip time rises and PD transaction duration exceeds WAL fsync by an unusual margin

Do not mix these hops together. The mitigation and ownership differ.

## 3. TiDB <-> TiKV

There are three common network issue types here:
- interruption
- latency jitter
- bandwidth saturation

### A. Network Interruption

Main checks:
- `TiDB -> Batch Client -> Batch Receive Average Duration`
- `TiDB -> Batch Client -> Batch Client Unavailable Duration`
- `TiDB -> Batch Client -> Wait Connection Establish Duration`
- TiDB logs for `batchRecvLoop re-create` or equivalent gRPC reconnection logs
- TiDB slow logs with `tikvRPC` backoff or large `lock_rpc`

Interpretation:
- receive duration spikes together with unavailable duration or connect-establish duration:
  - network interruption is plausible
- slow logs show `tikvRPC` backoff:
  - connection loss or reconnect delay is plausible

### B. Network Latency Jitter

Main checks:
- TiDB slow logs for `tikvRPC` backoff or suspicious RPC duration
- `TiDB -> KV Request -> RPC Layer Latency`
- `TiDB -> Batch Client -> Batch Receive Average Duration`

Interpretation:
- `RPC Layer Latency` rises unexpectedly while TiKV local stages stay relatively clean:
  - transport latency is the main suspect
- `Batch Receive Average Duration` long tails rise:
  - TiDB <-> TiKV network jitter is more plausible

Version note:
- before `v8.5.1`, the long tail of `Batch Receive Average Duration` is often a useful signal for this issue

### C. Network Bandwidth Saturation

Main checks:
- `Node exporter -> Network`
- `bw_in_allowance_exceeded`
- `bw_out_allowance_exceeded`

Interpretation:
- allowance-exceeded metrics rise:
  - the node exceeded its network bandwidth allowance and throttling is plausible

## 4. TiKV <-> TiKV

Main checks:
- `TiKV -> Error -> Server report failures`
- `TiKV -> Raft IO -> Commit log duration`
- `TiKV -> Raft IO -> Append log duration`
- `TiKV -> Raft IO -> Apply log duration`

Interpretation:
- `unreachable` failures appear:
  - transport failure is plausible, but not yet proven
- `Commit log duration` rises much more than `Append log duration` while append / apply remain stable:
  - replication transport or follower response delay is a strong suspect

This pattern is especially useful when:
- append log stays near normal
- apply log stays near normal
- only commit log inflates

That means the leader local persistence is not the main bottleneck; the time is more likely spent in replication transport, follower persistence, or the response path.

## 5. False Positives: Not Every `unreachable` Means Network

Two common non-network explanations can mimic TiKV <-> TiKV network failure:

1. gRPC thread exhaustion
2. gRPC threads stuck by host stalls or memory-allocation issues

Main checks:
- `TiKV -> Thread CPU -> gRPC`
- long KV requests or oversized payload clues
- kernel / host stall evidence such as hung tasks or allocstall

Guardrail:
- do not conclude `network issue` from `unreachable` alone
- if gRPC threads are exhausted or blocked, that can also break TiKV-to-TiKV heartbeats and message handling

## 6. TiDB <-> PD And PD <-> PD

These are important upstream splits even though they are not TiKV local root causes.

### A. TiDB <-> PD

Main checks:
- `TiDB -> PD Client -> Client CMD Fail OPS`
- `TiDB -> PD Client -> PD TSO RPC Duration`
- `PD -> TiDB -> PD client requests handle duration`
- `PD -> TiDB -> PD Server TSO Handle Time`

Interpretation:
- `PD TSO RPC Duration` high while `PD Server TSO Handle Time` stays low:
  - the main time is on network transmission, not PD server execution

### B. PD <-> PD

Main checks:
- `PD -> ETCD -> Handle transactions duration`
- `PD -> ETCD -> WAL fsync`
- `PD -> ETCD -> 99% Peer round trip time in seconds`

Interpretation:
- transaction duration high and clearly above WAL fsync:
  - network transmission between PD peers is plausible
- peer RTT above about `10ms`:
  - PD-peer network delay is plausible

## 7. Workaround And Mitigation Direction

Match the mitigation to the proven hop:

- TiDB <-> TiKV interruption or jitter:
  - investigate network path stability first
  - for versions before `v8.5.1`, consider lowering `tikv-client.grpc-keepalive-timeout` from `3` to `1`
  - from `v8.5.1`, values as low as `0.05` become possible after the related client-go improvements
- bandwidth saturation:
  - reduce traffic burst or move to higher network bandwidth capacity
- TiKV <-> TiKV:
  - check transport quality, retransmit, drop, and any gRPC-thread false positives
- TiDB <-> PD or PD <-> PD:
  - route mitigation to the PD path instead of calling it a TiKV local bottleneck

## 8. Minimal Network Checklist

Always check these before writing the conclusion:
- whether `RPC Layer Latency` rises with KV latency
- whether TiDB slow logs show `tikvRPC` backoff
- batch-client receive, unavailable, and connect-establish metrics
- `unreachable` failures
- `commit log` versus `append log`
- node network allowance, retransmit, or drop signals
- gRPC-thread false positives
- whether the issue is upstream of TiKV

## 9. Output Template

State the conclusion in this order:

1. which hop is affected
2. whether it is interruption, jitter, or bandwidth saturation
3. which metric proves transport time is dominant
4. what evidence rules out gRPC-thread or host-stall false positives
5. whether the issue is truly on TiKV or upstream of TiKV

Do not compress everything into `network slow`.