# PD Scheduling Known Cases

Source note: adapted from the Feishu `PD oncall handbook` <https://pingcap.feishu.cn/wiki/E5KkwI6VJiFA4nkcaVEcDhWonqu>.

## Patrol Region Time Stays High After Abnormal Regions Are Repaired

- Phenomenon:
  - `patrol region time` remains high even after TiKV-down repair completes.
- Related oncall:
  - v6.5.4 patrol time stayed high after TiKV down recovery `GTOC-7610`

## Scatter Region Can Fail During Lightning Import Because Regions Are Not Healthy

- Phenomenon:
  - Lightning import shows skew and PD logs report `region not replicated during scatter`.
- Root cause:
  - `schedule=deny` blocks region self-repair, so PD refuses scatter for unhealthy regions.
- Workaround:
  - Upgrade to a release that allows placement-rule checking and repair under `schedule-deny`.
  - Or disable `schedule-deny` or repair abnormal regions first.
- Related oncall:
  - v6.5.8 Lightning import had severe skew and rising oversized regions `GTOC-7098` / `JPID-10136`

## Inconsistent TiKV Region-Size or Key Config Makes PD Metrics Wrong

- Phenomenon:
  - PD suddenly shows many oversized or undersized regions because TiKV instances do not agree on `region-max-size` or `region-max-keys`.
- Workaround:
  - Keep region-size and key-threshold configuration consistent across all TiKV instances.
- Related oncall:
  - v8.1.1 PD suddenly showed many oversized-region-count values `NAID-11015`

## Disk Expansion After Low-Space Period Can Trigger a Balance-Region Storm

- Phenomenon:
  - After several TiKV nodes crossed `low-space-ratio`, later disk expansion changes PD's balancing mode and can trigger many balance-region operators.
- Workaround:
  - Expand earlier, before stores cross `low-space-ratio`.
  - If it already happened, reduce `store-limit` and `region-schedule-limit`.
  - Consider lowering `low-space-ratio` on very large disks.
- Related oncall:
  - v7.1.1 disk expansion triggered many balance-region operators `TCID-11613` / `TCOC-3622`

## Scatter-Region Can Violate Placement Rule or Isolation Level Temporarily

- Phenomenon:
  - PD validates each replacement target independently during scatter, but the final set can still violate placement or isolation constraints.
- Workaround:
  - No direct workaround; region checker repairs the layout later.
- Related oncall:
  - v7.5.2 region unavailable after one host went down under host isolation config `TICKET-6907` / `ONCALL-8065`

## Scatter-Range-Scheduler Can Conflict With Global Balancing

- Phenomenon:
  - Range-level balancing can fight with global balance-region or balance-leader scheduling and produce uneven results.
- Workaround:
  - Pre-split so scatter happens naturally.
  - Do not rely on scatter-range as a strong production guarantee.
- Related oncall:
  - v6.5.6 scatter-range-schedule was inconsistent `TICKET-4439`
  - v7.5.6 post-write duration was very high after some time `TCID-11555` / `TCOC-3509`

## `DISTRIBUTE TABLE` Can Crash PD Through Balance-Range-Scheduler

- Phenomenon:
  - After executing `DISTRIBUTE TABLE`, PD can crash with a nil-pointer panic.
- Root cause:
  - The affected path passes a nil region into range-scheduler option handling without a nil guard.
- Workaround:
  - Disable merge or avoid using balance-range-scheduler in affected versions.
- Related oncall:
  - v8.5.1 PD panic during range balancing `GTOC-8261`

## Region Score Can Change Suddenly After Adding or Removing Evict-Leader

- Phenomenon:
  - After leader transfer, a follower becoming leader can report stale approximate size or key information briefly, and balance-region scheduling may start unexpectedly.
- Workaround:
  - Adjust store limit before adding or removing `evict-leader-scheduler`.
- Related oncall:
  - v7.5.6 region score changed suddenly after removing evict leader `TCOC-4499`

## Small Region Count Per Store Can Create Redundant Balance-Region Scheduling

- Phenomenon:
  - When the average region count per TiKV is very small, balance-region error tolerance can create large numbers of redundant scheduling actions.
- Workaround:
  - Increase `tolerant-size-ratio`.
- Related oncall:
  - v7.1.6 a 6-TiKV cluster with only 5 regions kept reporting region errors `TICKET-5558` / `ONCALL-7604`

## Evict-Leader-Scheduler Cannot Be Added or Removed

- Phenomenon:
  - PD may report `store is paused for leader transfer` and refuse add or remove operations for `evict-leader-scheduler`.
- Workaround:
  - Transfer PD leader.
- Fixed version:
  - `v7.5.4` / `v8.1.2` / `v6.5.12` / `v6.1.8` / `v5.4.4`
- Related oncall:
  - v6.5.7 regional recovery could not restore leader scheduling `ST-992`
  - v6.5.7 evict-leader-scheduler add failed `ST-960` / `GTOC-7126` / `GTOC-7127`