---
name: pd-scheduling
description: |
  Reference handbook for PD scheduler and region-health incidents.
  Use when investigating region checker behavior, patrol-region drift,
  scatter conflicts, balance-region storms, low-space rebalancing,
  evict-leader anomalies, or scheduler interactions after topology changes.
---

# PD Scheduling

Known PD region-checker and scheduler patterns for on-call diagnosis.

Source note: adapted from the Feishu `PD oncall handbook` <https://pingcap.feishu.cn/wiki/E5KkwI6VJiFA4nkcaVEcDhWonqu>.

## When To Use This Skill

Use this skill when:

- region health metrics do not match the real cluster state
- `scatter`, `balance-region`, `balance-leader`, or `evict-leader` behaves unexpectedly
- disk expansion or low-space transitions trigger large waves of scheduling
- region distribution looks wrong but the root cause appears to be PD scheduling policy

## First Checks

- Confirm which scheduler or checker is acting: region checker, balance-region, balance-leader, scatter-range, or evict-leader.
- Check whether the cluster recently crossed `low-space-ratio`.
- Record whether `scatter`, `DISTRIBUTE TABLE`, or large split or merge operations were issued.
- Confirm whether TiKV-side config is consistent for region size and key thresholds.

## Known Cases

- Detailed reference: [`references/scheduling-known-cases.md`](references/scheduling-known-cases.md)

### Patrol region time stays high after abnormal regions are repaired

- Phenomenon: after TiKV down and replica repair complete, `patrol region time` remains high and does not fall back.
- Related oncall:
  - v6.5.4 patrol time stayed high after TiKV down recovery `GTOC-7610`

### Scatter region can fail during Lightning import because regions are not healthy

- Phenomenon: Lightning import shows skew and `scatter` logs report `region not replicated during scatter`.
- Root cause: `schedule=deny` prevents region self-repair, so PD refuses scatter requests for unhealthy regions.
- Workaround:
  - upgrade to a release that allows placement-rule checking and repair under `schedule-deny`
  - or disable `schedule-deny` or repair abnormal regions manually first
- Related oncall:
  - v6.5.8 Lightning import had severe skew and rising oversized regions `GTOC-7098` / `JPID-10136`

### Inconsistent TiKV region-size or key config makes PD metrics wrong

- Phenomenon: PD suddenly shows many oversized or undersized regions because TiKV instances do not agree on `region-max-size` or `region-max-keys`.
- Workaround: keep region-size and key-threshold configuration consistent across all TiKV instances.
- Related oncall:
  - v8.1.1 PD suddenly showed many oversized-region-count values `NAID-11015`

### Disk expansion after low-space period can trigger a balance-region storm

- Phenomenon: when several TiKV nodes had already crossed `low-space-ratio`, later disk expansion changes PD's balancing mode and can trigger large-scale rebalance.
- Workaround:
  - expand earlier, before stores cross `low-space-ratio`
  - if it already happened, reduce `store-limit` and `region-schedule-limit`
  - consider lowering `low-space-ratio` on very large disks
- Related oncall:
  - v7.1.1 disk expansion triggered many balance-region operators `TCID-11613` / `TCOC-3622`

### Scatter-region can violate placement rule or isolation level temporarily

- Phenomenon: PD validates each replacement target independently during scatter, but can still choose multiple targets that conflict with each other as a final set.
- Workaround: no direct workaround; region checker will repair the layout later.
- Related oncall:
  - v7.5.2 region unavailable after one host went down under host isolation config `TICKET-6907` / `ONCALL-8065`

### Scatter-range-scheduler can conflict with global balancing

- Phenomenon: range-level balancing can fight with global balance-region or balance-leader scheduling and produce uneven results.
- Workaround:
  - pre-split so scatter happens naturally
  - avoid relying on scatter-range as a strong production guarantee
- Related oncall:
  - v6.5.6 scatter-range-schedule was inconsistent `TICKET-4439`
  - v7.5.6 post-write duration was very high after some time `TCID-11555` / `TCOC-3509`

### Evict-leader-scheduler cannot be added or removed

- Phenomenon: PD may report `store is paused for leader transfer` and refuse add or remove operations for `evict-leader-scheduler`.
- Workaround: transfer PD leader.
- Fixed version:
  - `v7.5.4` / `v8.1.2` / `v6.5.12` / `v6.1.8` / `v5.4.4`
- Related oncall:
  - v6.5.7 regional recovery could not restore leader scheduling `ST-992`
  - v6.5.7 evict-leader-scheduler add failed `ST-960` / `GTOC-7126` / `GTOC-7127`