# Snapshot Backlog And Scatter-Slow Diagnosis

Read this file when `scatter`, rebalance, or `hot leader` does not progress and snapshot work may be blocked.

## When To Read

Read this file when the active branch is `snapshot-backlog` and the real
question is why balancing or migration progress stalled, not merely whether a
store stayed hot.

## Goal

The goal is to determine:
- whether balancing was blocked by snapshot progress rather than PD scheduling itself
- which snapshot stage was slow first
- whether the best evidence points to worker-pool backlog, delete-range backlog, transport, apply pressure, or store-local IO

## Key Questions

Keep these questions aligned with the main skill:
- is the problem really leader imbalance, or is migration blocked by snapshot progress
- which stage is slow first: generate, local read IO, send, or apply
- what evidence points to worker-pool backlog, delete-range backlog, or store-local IO
- what evidence is still missing before naming the lower-layer cause

## Guardrails

- do not stop at `hot leader`, `leader imbalance`, or `PD is still scheduling`
- do not recommend `evict leader` or more scatter as root-cause mitigation unless snapshot progress is healthy
- if the lower layer is still not proven, say `snapshot progress blocked` instead of overclaiming

## 1. What This Path Is For

Use this path when the visible symptom is:
- one store stays hot because region movement is slow
- scatter runs for a long time and is stopped before completion
- snapshot generate wait, snapshot pending tasks, or snapshot duration is high on one store
- `hot leader` is only the surface symptom and the real question is why balancing did not finish

Do not stop at:
- `hot leader`
- `leader imbalance`
- `PD is still scheduling`

The real diagnosis target is:
- why snapshot progress is slow
- which stage is slow first
- which lower layer blocked that stage

## 2. First Split

Separate these possibilities:

1. PD did not create or keep the operator
2. PD scheduled correctly, but snapshot generation on TiKV is slow
3. snapshot generation is ready, but send / transport is slow
4. snapshot arrived, but apply side is slow

If PD operator evidence is present and the region movement stays stuck on one step for a long time:
- do not blame PD first
- continue into TiKV snapshot stages

## 3. Main Metrics To Check

Check these first:
- PD operator / scheduler evidence for the affected region
- snapshot generate wait or generate duration panels
- `Storage -> Storage async snapshot duration`
- per-store snapshot send / apply panels if available
- `Thread CPU -> Raftstore`
- `Thread CPU -> Region worker` or equivalent worker pool
- per-store disk read throughput and IO wait for the outlier store

When available, also check:
- pending delete-range tasks
- region-worker backlog
- snapshot task queue length

## 4. Diagnosis Order

### A. Confirm balancing is really blocked

Need:
- a store that stays hot
- operator or scatter evidence showing rebalance was attempted
- one snapshot-related stage that is clearly slower than healthy stores

### B. Find the earliest slow stage

Use this order:
1. snapshot generate wait
2. snapshot local read IO
3. snapshot send / transport
4. snapshot apply

Interpretation:
- generate wait high first:
  - suspect worker-pool backlog or shared-thread contention
- local read IO high first:
  - suspect RocksDB / disk wait on that store
- send high first:
  - suspect transport or receiver-side pressure
- apply high first:
  - suspect receiver-side apply pressure

## 5. High-Frequency TiKV-Side Causes

### Shared worker-pool contention

Typical symptom:
- snapshot generate wait is high
- thread CPU is not fully saturated
- tasks are pending for a long time

One important pattern:
- delete-range cleanup and snapshot generation may effectively compete through the same worker resources
- then `hot leader` is not the cause; it is fallout from slow movement

### Pending delete-range backlog

Typical symptom:
- rebalance or scatter is slow
- pending delete-range work accumulates
- generate snapshot wait rises because cleanup work occupies the worker pool

This is exactly the kind of case where:
- asking PD to move more leaders does not solve the real bottleneck
- the correct output should explain why snapshot generation could not progress

### Store-local IO slowdown

Typical symptom:
- one store is an outlier
- snapshot read IO is capped by disk throughput or IO wait
- other stores are comparatively healthy

Then:
- compare against healthy stores instead of cluster average
- avoid claiming general cluster imbalance as the root cause

## 6. Output Rules

The conclusion should say:
- whether balancing was blocked by snapshot progress
- which exact snapshot stage was slow first
- whether the best evidence points to worker-pool backlog, delete-range backlog, or store-local IO
- what evidence is still missing before a deeper root cause claim

Good wording:
- `scatter was slow because snapshot generation on store X was blocked`
- `hot leader persisted because migration progress was limited by snapshot backlog`
- `snapshot progress was blocked; lower-layer cause is still suspected IO / worker-pool contention`

Bad wording:
- `PD scheduling issue`
- `just evict leader`
- `hot leader caused the problem`