---
name: pd-operations
description: |
  Reference handbook for PD operational incidents.
  Use when investigating ETCD capacity exhaustion, TLS rollout failures,
  Dashboard issues, Prometheus scrape gaps, PD leader CPU spikes,
  or TSO-related operational bottlenecks.
---

# PD Operations

Known ETCD, TLS, Dashboard, observability, and TSO pressure patterns for PD on-call diagnosis.

Source note: adapted from the Feishu `PD oncall handbook` <https://pingcap.feishu.cn/wiki/E5KkwI6VJiFA4nkcaVEcDhWonqu>.

## When To Use This Skill

Use this skill when:

- PD loses service because ETCD is full or unhealthy
- enabling TLS breaks PD expansion or cluster startup
- Dashboard becomes unavailable or fills local disk
- Prometheus cannot scrape PD leader metrics reliably
- PD leader CPU is high because of TSO pressure

## First Checks

- Confirm whether the incident is ETCD, TLS, Dashboard, scraping, or TSO related.
- Identify whether only PD leader is affected or all PD nodes are degraded.
- Capture current ETCD alarm state, PD health, and Dashboard address.
- Record whether TLS or expansion happened shortly before the incident.

## Known Cases

- Detailed reference: [`references/operations-known-cases.md`](references/operations-known-cases.md)

### ETCD capacity exhaustion makes PD unable to serve

- Phenomenon: once ETCD backend usage exceeds `quota-backend-bytes`, ETCD becomes read-only and PD can lose leader service.
- Workaround:
  - manually compact and defragment ETCD
  - clear ETCD alarm state after recovery
- Related oncall:
  - v5.1.1 CDC filled ETCD space and PD could not serve `TICKET-5757` / `ONCALL-7691`

### Dashboard does not support high availability

- Phenomenon: if the PD node running Dashboard crashes, Dashboard becomes unavailable.
- Workaround:

```bash
pd-ctl config set dashboard-address <new_address>
```

- Related oncall:
  - v7.5.6 PD Dashboard HA question `GTOC-7901` / `APID-11724`

### Dashboard log search can lock PD's internal database

- Phenomenon: broad log searches through Dashboard can cause `database is locked` and exhaust PD resources.
- Workaround: no strong workaround beyond reducing search scope.
- Related oncall:
  - v6.5.5 multiple alerts around query duration and node disk utilization `APID-11562`

### Dashboard can fill up `/tmp`

- Phenomenon: Dashboard log-search temporary files are stored under `/tmp` and not cleaned automatically.
- Workaround: clean `/tmp` regularly.
- Related oncall:
  - v6.5.10 too much Dashboard log exhausted system disk `TICKET-6620`

### PD leader CPU surges because of TSO pressure

- Phenomenon: PD leader CPU becomes high during peak traffic, often with TSO-heavy workloads and many TiDB instances.
- Workaround:
  - increase TiDB-side TSO batching wait time when appropriate
  - enable TSO follower proxy when the workload pattern fits

```sql
SET GLOBAL tidb_tso_client_batch_max_wait_time = 1;
SET GLOBAL tidb_enable_tso_follower_proxy = ON;
```

- Related oncall:
  - v8.5.2 high PD CPU due to TSO request pressure `APID-11722` / `GTOC-7900`
  - v5.4.1 PD leader CPU intermittent spikes `TICKET-5717` / `ONCALL-7672`

### Large clusters can fail to expose PD leader metrics to Prometheus

- Phenomenon: `curl` can fetch `/metrics` slowly, but Prometheus times out before completion and Grafana shows no PD leader metrics.
- Workaround:
  - increase Prometheus scrape timeout and intervals for PD metrics
- Related oncall:
  - v5.4.1 Prometheus could not fetch PD leader metrics `ONCALL-7610` / `TICKET-5546`

### Enabling TLS can break PD expansion

- Phenomenon: enabling TLS can fail during PD expansion if some nodes do not restart within TiUP timeout, leaving PD peer information inconsistent.
- Workaround:
  - repair abnormal PD member state first
  - disable and re-enable TLS if needed
  - rerun with a larger TiUP `--wait-timeout`
- Related oncall:
  - v7.1.5 PD expansion failed after enabling TLS `TICKET-4708` / `ONCALL-7310`