# PD Operations Known Cases

Source note: adapted from the Feishu `PD oncall handbook` <https://pingcap.feishu.cn/wiki/E5KkwI6VJiFA4nkcaVEcDhWonqu>.

## ETCD Capacity Exhaustion Makes PD Unable to Serve

- Phenomenon:
  - Once ETCD backend usage exceeds `quota-backend-bytes`, ETCD becomes read-only and PD can lose leader service.
- Workaround:
  - Manually compact and defragment ETCD.
  - Clear ETCD alarm state after recovery.
- Related oncall:
  - v5.1.1 CDC filled ETCD space and PD could not serve `TICKET-5757` / `ONCALL-7691`

## Dashboard Does Not Support High Availability

- Phenomenon:
  - If the PD node running Dashboard crashes, Dashboard becomes unavailable.
- Workaround:

```bash
pd-ctl config set dashboard-address <new_address>
```

- Related oncall:
  - v7.5.6 PD Dashboard HA question `GTOC-7901` / `APID-11724`

## Dashboard Log Search Can Lock PD's Internal Database

- Phenomenon:
  - Broad log searches through Dashboard can cause `database is locked` and exhaust PD resources.
- Workaround:
  - No strong workaround beyond reducing search scope.
- Related oncall:
  - v6.5.5 multiple alerts around query duration and node disk utilization `APID-11562`

## Dashboard Can Fill Up `/tmp`

- Phenomenon:
  - Dashboard log-search temporary files are stored under `/tmp` and not cleaned automatically.
- Workaround:
  - Clean `/tmp` regularly.
- Related oncall:
  - v6.5.10 too much Dashboard log exhausted system disk `TICKET-6620`

## PD Leader CPU Surges Because of TSO Pressure

- Phenomenon:
  - PD leader CPU becomes high during peak traffic, often with TSO-heavy workloads and many TiDB instances.
- Workaround:
  - Increase TiDB-side TSO batching wait time when appropriate.
  - Enable TSO follower proxy when the workload pattern fits.

```sql
SET GLOBAL tidb_tso_client_batch_max_wait_time = 1;
SET GLOBAL tidb_enable_tso_follower_proxy = ON;
```

- Related oncall:
  - v8.5.2 high PD CPU due to TSO request pressure `APID-11722` / `GTOC-7900`
  - v5.4.1 PD leader CPU intermittent spikes `TICKET-5717` / `ONCALL-7672`

## Large Clusters Can Fail to Expose PD Leader Metrics to Prometheus

- Phenomenon:
  - `curl` can fetch `/metrics` slowly, but Prometheus times out before completion and Grafana shows no PD leader metrics.
- Workaround:
  - Increase Prometheus scrape timeout and intervals for PD metrics.
- Related oncall:
  - v5.4.1 Prometheus could not fetch PD leader metrics `ONCALL-7610` / `TICKET-5546`

## Enabling TLS Can Break PD Expansion

- Phenomenon:
  - Enabling TLS can fail during PD expansion if some nodes do not restart within TiUP timeout, leaving PD peer information inconsistent.
- Workaround:
  - Repair abnormal PD member state first.
  - Disable and re-enable TLS if needed.
  - Rerun with a larger TiUP `--wait-timeout`.
- Related oncall:
  - v7.1.5 PD expansion failed after enabling TLS `TICKET-4708` / `ONCALL-7310`

## Dashboard Login Can Fail Because Certificates Lack `127.0.0.1` SAN

- Phenomenon:
  - After enabling TiDB-to-client TLS, Dashboard login can fail with:

```text
tls: failed to verify certificate: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
```

- Root cause:
  - The certificate SANs do not include `127.0.0.1`, while Dashboard validation requires it in the affected access path.
- Workaround:
  - Regenerate or update the certificate so SAN includes `127.0.0.1`.
- Related oncall:
  - v7.1.0 Dashboard login failed after TLS enable `GTOC-6832`

## PD Common-Name Configuration Has Limits Under TLS

- Phenomenon:
  - PD can reject common-name configuration with `only supports one CN`.
- Root cause:
  - PD supports only one configured CN value, so all authenticated peers must match that single CN.
- Workaround:
  - Standardize all authenticated objects on one CN value.

## Prometheus May Need Larger Scrape Windows on Very Large Clusters

- Phenomenon:
  - PD leader metrics are reachable by direct `curl`, but Prometheus still fails because default timeout is too short.
- Workaround:
  - Increase:
    - `scrape_timeout`
    - `scrape_interval`
    - `evaluation_interval`
- Related oncall:
  - v5.4.1 Prometheus could not fetch PD leader metrics `ONCALL-7610` / `TICKET-5546`

## TiUP Can Silently Fail to Apply PD Config Changes

- Phenomenon:
  - TiUP updates PD configuration without returning an obvious error, but the change does not actually take effect.
- Workaround:
  - Verify the effective PD config after every TiUP change instead of trusting the command result alone.
- Related oncall:
  - v8.5.1 TiUP update of PD config did not take effect `APID-11212`

## High `flow-round-by-digit` Makes Key Visualizer Unreliable

- Phenomenon:
  - If `flow-round-by-digit` is set too high, traffic statistics become too coarse and Key Visualizer no longer matches real hotspot behavior well.
- Workaround:
  - Set `flow-round-by-digit` to `3`.
- Related oncall:
  - v7.5.4 Key Visualizer hotspots did not match `TIDB_HOT_REGIONS_HISTORY` `EMID-10035` / `GTOC-7314`

## Go Runtime GC Can Cause Brief PD CPU Spikes Without User Impact

- Phenomenon:
  - PD CPU can spike briefly during Go GC, especially when many idle CPU cores are available, but the spike may not affect workload performance.
- Workaround:
  - Usually do nothing.
  - If needed for confirmation, test with `GOMAXPROCS` or GC-related tuning in a controlled environment.
- Related oncall:
  - v5.4.2 PD CPU intermittently spiked during off-peak hours `ONCALL-6881`

## Container GOMAXPROCS Mismatch Can Cause TSO Long Tail

- Phenomenon:
  - In containerized deployments, PD may use host CPU count as `GOMAXPROCS`, causing runtime overhead and TSO long-tail latency.
- Workaround:
  - Set `GOMAXPROCS` to the actual container CPU limit.
- Related oncall:
  - v6.5.11 containerized PD had TSO long-tail latency `TICKET-8173`