# Dedicated — Document Summaries

> Generated: 2026-02-15
> Source: Top 10 Dedicated TiDB performance and diagnostics documents from Feishu wiki

---

## Document 1: Key metrics for troubleshooting
**Source**: https://pingcap.feishu.cn/wiki/W7WIwnc94iB3gJkG2vVcBHOmnig
**Type**: Reference Guide / Metrics Catalog
**Reading Status**: Full Read

### Key Points
- Comprehensive catalog of Grafana monitoring panels organized by TiDB/TiKV/PD components
- Defines the "must-see metrics" hierarchy for troubleshooting any performance issue
- Covers the full monitoring stack: Prometheus + Grafana + AlertManager + TiDB Dashboard
- Organized by: Overall Cluster Status → Duration → Workload → TiDB Resources → TiKV Resources → TiKV Duration → Read-related → Write-related (Raft) → RocksDB → PD

### Diagnostic Procedures
- **Overall assessment flow**: Check Duration panels → Compare with baseline → Check QPS/workload → Identify scope (all instances or specific)
- **TiDB-side flow**: Check TSO wait/RPC → KV Request Duration (by store/type) → KV Backoff → compare TiDB KV duration with TiKV gRPC duration to isolate network vs processing
- **TiKV-side flow**: Check gRPC duration → Thread CPU → Read path (Unified Read Pool → Coprocessor) or Write path (Raft Propose → Raft IO → Apply)
- **PD-side flow**: Check Region health → Balance statistics → Hot read/write regions

### Metrics / Thresholds
- **gRPC poll CPU**: Should be < 80% × `server.grpc-concurrency` (default 5)
- **Raft store CPU**: Should be < 80% × `raftstore.store-pool-size` (default 2); keep below 120% when pool size = 2
- **Scheduler worker CPU**: Should be < 90% × `storage.scheduler-worker-pool-size` (default 8 for >=16 cores, 4 otherwise)
- **Unified read pool CPU**: Keep between 60%-90% of pool size; default pool = 80% of CPU cores
- **Slow Score**: When reaches 100 (v7.5+), PD auto-evicts all leaders from that TiKV node
- **Region health**: Pending peers should be < 100; missing peers should not be persistently > 0
- **Server is busy**: Should be 0 in normal case (Write Stall, Channel Full events)
- **Server report failures**: Should be 0 in normal case

### Tools / Commands
- TiDB Performance Map: https://asktug.com/_/tidb-performance-map/
- Performance Overview: https://docs.pingcap.com/tidb/stable/performance-tuning-methods/
- Key Grafana dashboards referenced: Overview, TiDB, TiKV-details, PD

### Must-See Metrics List (Quick Reference)
```
TiDB --> Query Summary --> Duration / QPS / CPS by type
TiDB --> KV Request / KV Errors --> KV Request Duration
TiDB --> Server / PD Client
TiKV-details --> Cluster / Errors / PD
TiKV-details --> gRPC --> gRPC Duration / Message Count
TiKV-details --> Threads CPU
TiKV-details --> Duration / Storage
Read: TiKV Details --> Unified Read pool / Coprocessor Overview / Coprocessor Details
Write: TiKV Details --> Raft Propose / Raft IO / Scheduler xxx
TiKV-details --> RocksDB-kv
PD --> Region health
PD --> Cluster / Statistics-hot read/hot write
```

### Raw Content Excerpt
> Duration: execution time. The duration between the time that the client's network request is sent to TiDB and the time that the request is returned to the client after TiDB has executed it. In general, client requests are sent in the form of SQL statements, but can also include the execution time of commands such as COM_PING, COM_SLEEP, COM_STMT_FETCH, and COM_SEND_LONG_DATA. Because TiDB supports Multi-Query, it supports sending multiple SQL statements at one time, such as select 1; select 1; select 1;. In this case, the total execution time of this query includes the execution time of all SQL statements. By comparing these metrics with baseline metrics (the latency under normal conditions), you can determine whether the current cluster latency shows any significant issues.

---

## Document 2: TiDB write slow Troubleshooting Guide
**Source**: https://pingcap.feishu.cn/wiki/GgYhwMVLLi157EkmbqccnhStnyh
**Type**: Troubleshooting Guide / Deep-dive Technical Reference
**Reading Status**: Full Read (67K chars, extracted via agent)

### Key Points
- **Definitive guide** for diagnosing TiDB write slowness, covering the full write path end-to-end
- Covers: TiDB Server write process → TiKV gRPC → Scheduler (flow control + latch + snapshot) → Raftstore (propose → append → commit → apply) → RocksDB (WAL → memtable → flush → compaction)
- Includes pessimistic locking, 2PC (prewrite + commit), and lock resolution mechanisms
- Covers backoff/retry mechanisms with hardcoded timeout values
- Provides RocksDB Write Stall root causes and solutions
- Covers GC impact on write performance

### Diagnostic Procedures

#### Master Flow (TiDB Side):
1. Check Overview monitoring → server load, CPU, disk IO, network
2. Check TiDB → KV Errors and TiKV-Detail → Errors for abnormalities
3. Parse Duration high? → CPU issue or too many insert...values
4. Compile Duration high? → Non-correlated subqueries
5. Get Token Duration high? → TokenLimiter reached
6. TSO acquisition slow? → Check PD TSO RPC Duration / TSO Async Duration / TSO Wait Duration
7. Execution Duration high? → Drill into KV request details

#### KV Duration Flow:
- TiKV gRPC duration short but TiDB KV duration long → **NETWORK ISSUE** (check Ping latency, Network Traffic)
- Both long → **TiKV-side issue** → Drill into Scheduler Duration

#### Scheduler Flow:
- Formula: `Scheduler duration = latch-wait + async-snapshot + load lock + async-write`
- CPU uneven → lock conflicts or write hotspot
- Latch wait high → multiple txns on same key
- async-write high → Drill into Raftstore

#### Raftstore Flow:
- Formula: `async_write = Propose wait + Append log + Commit log + Apply wait + Apply log`
- Propose wait high → raftstore CPU / append log slow / batch too small
- Commit log high → Raft flow control (increase `raft-max-inflight-msgs` from 256 to 1024)
- Apply wait high → apply thread saturated / disk latency
- Apply log high → KV RocksDB write performance

#### RocksDB Write Stall Flow:
- Too many memtables → increase `max_write_buffer_number` (default 5)
- Too many L0 SST files → increase `max-sub-compactions` (default 3)
- Compaction pending bytes → increase `soft-pending-compaction-bytes-limit` (64G) / `hard-pending-compaction-bytes-limit` (256G)

### Metrics / Thresholds
| Metric | Threshold |
|--------|-----------|
| Get Token Duration | < 2us |
| Parse Duration | < 10ms |
| Compile Duration | < 30ms |
| gRPC poll CPU | < grpc-concurrency x 80% |
| Scheduler worker CPU | < scheduler-worker-pool-size x 80% |
| Raft store CPU | < 80% |
| Async apply CPU | < 90% |
| KV single request timeout | 60s |
| Pessimistic lock wait (innodb_lock_wait_timeout) | 50s |
| L0 slowdown trigger | 20 SST files |
| L0 stop trigger | 36 SST files |
| Soft pending compaction limit | 64GB |
| Hard pending compaction limit | 256GB |
| txn-total-size-limit | 100MB default (max 10GB) |

### Backoff Max Durations (hardcoded, ms):
| Backoff Type | Max ms |
|-------------|--------|
| tsoMaxBackoff | 15,000 |
| prewriteMaxBackoff | 20,000 |
| pessimisticLockMaxBackoff | 10,000 |
| GcResolveLockMaxBackoff | 100,000 |
| maxSplitRegionsBackoff | 120,000 |
| batchGetMaxBackoff | 20,000 |
| copNextMaxBackoff | 20,000 |
| getMaxBackoff | 20,000 |
| cleanupMaxBackoff | 20,000 |
| GcOneRegionMaxBackoff | 20,000 |
| deleteRangeOneRegionMaxBackoff | 100,000 |
| rawkvMaxBackoff | 20,000 |
| splitRegionBackoff | 20,000 |
| waitScatterRegionFinishBackoff | 120,000 |
| locateRegionMaxBackoff | 20,000 |
| pessimisticRollbackMaxBackoff | 10,000 |

### Key Grafana Panels (Write Path)
```
TiDB --> Server --> Get Token Duration
TiDB --> Executor --> Parse Duration / Compile Duration / Execution Duration
TiDB --> PD Client --> PD TSO RPC Duration / PD TSO Wait Duration
TiDB --> KV Request --> KV Request Duration 999 by store/type
TiDB --> Transaction --> Acquire Pessimistic Locks Duration / Statement Lock Keys
TiDB --> KV Errors --> KV Backoff OPS / Lock Resolve OPS
TiKV-Detail --> Thread CPU --> gRPC poll CPU / Scheduler worker CPU / Raft store CPU / Async apply CPU
TiKV-Detail --> gRPC --> gRPC message duration / count / failed
TiKV-Detail --> Storage --> storage async snapshot duration / Storage async write duration
TiKV-Detail --> Scheduler-prewrite --> Scheduler command duration / latch wait duration
TiKV-Detail --> Raft Propose --> Propose wait duration / Apply wait duration per server
TiKV-Detail --> Raft IO --> Append log / Commit log / Apply log duration per server
TiKV-Detail --> RocksDB KV --> Write operations / Write WAL duration / Write Duration / Write Stall Reason
TiKV-Detail --> RocksDB KV --> Compaction operations / pending bytes / Compression ratio
```

### Tools / Commands
- Compression tuning: Default `["no","no","lz4","lz4","lz4","zstd","zstd"]` → High-compression `["no","no","zstd","zstd","zstd","zstd","zstd"]`
- Write Stall diagnosis: Check `RocksDB KV/raft → Write Stall Reason` panel, or search RocksDB log for "Stalling"
- GC write throttle: `gc.max-write-bytes-per-sec` (v3.0.6+, dynamically modifiable)
- Key configs: `[storage] scheduler-concurrency` (default 2048000 latches), `raftstore.raft-max-inflight-msgs` (default 256)

### Raw Content Excerpt
> Storage async write duration = Propose wait + Append log + Commit log + Apply wait + Apply log. The Raftstore thread pool is the most complex thread pool in TiKV. The default size (configured by raftstore.store-pool-size) of this thread pool is 2. When the size of the StoreWriter thread pool is 0, all write requests are written into RocksDB in the way of fsync by the Raftstore thread. Keep the overall CPU usage of the Raftstore thread below 60%. When the number of Raftstore threads is 2, keep the Raft store CPU on Grafana below 120%. Do not increase the size of the Raftstore thread pool to improve write performance without careful consideration, because this might increase the disk burden and degrade performance.

---

## Document 3: TiDB OOM SOP
**Source**: https://pingcap.feishu.cn/wiki/LHhxw2e7Pif5mnk99PEcjwBUndb
**Type**: SOP (Standard Operating Procedure)
**Reading Status**: Full Read

### Key Points
- Two-path OOM diagnosis: TiDB stays online vs TiDB crashes (killed by system OOM killer)
- Key diagnostic tool: `tidb-runtime > memory usage` metric to pinpoint when OOM happens and when memory starts growing
- Uses `expensive_query` logs and Top SQL on New O11Y/Clinic to identify candidate SQLs
- Memory configuration check via `INFORMATION_SCHEMA.MEMORY_USAGE`
- Critical: if `MEMORY_LIMIT > total-vm`, set `tidb_server_memory_limit < total-vm * 0.8`
- NUMA-awareness issue: TiDB may fail to recognize actual available memory on a NUMA node

### Diagnostic Procedures
1. View `tidb-runtime > memory usage` metric → identify when OOM happens and when memory starts growing
2. **If TiDB stays online (not killed)**:
   - Near OOM moment, check `tidb.log` with keyword `expensive_query` OR Top SQL on New O11Y/Clinic
   - Check: any SQL starting at time memory began growing? Many `expensive_query` logs with similar pattern?
   - Compare with "Typical OOM problems" to confirm root cause
3. **If TiDB crashes (killed by OOM killer)**:
   - Gather Linux dmesg: `dmesg -T | grep oom`
   - Check memory settings: `select * from INFORMATION_SCHEMA.MEMORY_USAGE;`
   - If MEMORY_LIMIT > total-vm → set `tidb_server_memory_limit < total-vm * 0.8`
   - If still crashing after setting limit → possible bug in global memory controller
   - Gather `oom_record` from tidb log directory
   - Search for `global memory controller` in TiDB logs
   - Use heap profiling to identify top memory-consuming functions
   - Analyze source code or escalate to TiDB Query Executor team

### Metrics / Thresholds
- **Memory limit rule**: `tidb_server_memory_limit < total-vm * 0.8`
- NUMA case study: TICKET-6047 — TiDB bound to NUMA node without `resource_control`, couldn't recognize actual available memory

### Tools / Commands
```sql
SELECT * FROM INFORMATION_SCHEMA.MEMORY_USAGE;
```
```bash
dmesg -T | grep oom
```
- Search TiDB logs for: `expensive_query`, `global memory controller`
- Check `oom_record` file in tidb log directory
- Use heap profiling for memory function analysis
- Case study reference: EasyPost TiDB OOM Case Study [issues/56440]

### Raw Content Excerpt
> When you troubleshoot OOM issues, follow this process: View the tidb-runtime > memory usage metric to check: when OOM happens, when the memory starts growing before OOM. If TiDB keeps online and isn't killed by the system OOM killer -> Troubleshooting OOM in 10min. Near the OOM moment, check the tidb.log with the keyword expensive_query OR Top SQL on New O11Y on Clinic. Check the candidate SQLs: whether there is any SQL statement that starts executing at the moment when the memory starts growing, whether there are many expensive_query logs with a similar SQL pattern.

---

## Document 4: Enhance Network/IO Hotspot Diagnosis Design Doc
**Source**: https://pingcap.feishu.cn/wiki/DI0mw3YVNiCfmukGPohctrGnnde
**Type**: Design Document
**Reading Status**: Full Read

### Key Points
- Extends TopSQL framework beyond CPU to cover **Network** and **IO** dimensions
- Motivated by TCOC/GTOC issue reviews: locating query SQL by resource dimension is high-frequency scenario
- Four resource dimensions: CPU (existing TopSQL), Memory (not difficult), **Network** (new), **IO** (new)
- Network hotspot examples: WeTech case — region response size too large + gRPC compression overhead → gRPC thread blocking; ONCALL7830 — large gRPC requests blocking
- **Dedicated-first** development, then migration to NextGen and OP

### Diagnostic Procedures (New Capabilities Proposed)
- **TiDB**: For a specific node, display top N SQL statements by cumulative user communication volume (sum of request + response data)
- **TiKV**:
  - Top N SQL/Table/Region by cumulative communication volume with TiDB
  - Top N SQL/Table/Region by cumulative logical read/write IO
  - Top N Regions by cumulative CPU usage (extends existing SQL/Table-level TopSQL)
- **gRPC layer**: New monitoring and alerting for `grpc_wait_time`

### Architecture
- **Existing framework**: TiDB/TiKV collect data per second → upload to O11Y via protocol per minute → O11Y persists → Clinic queries and presents via UI
- **TiDB changes**: New `StmtCollector` collects client network traffic (from `tidb_server_packet_io_bytes`), stored in `SessionVars`. Multi-dimensional PickTopN (CPU + Network), merge results. Data volume ~2x original.
- **TiKV changes**: New aggregation by `sql+plan+table_id` and `region_id` (instead of `sql+plan+region_id+keyranges`). Multi-dimensional PickTopN for CPU, network, IO independently. Data volume ~3x original.
- **New upload**: New `ResourceUsageRecord` type for region-aggregated data via gRPC `Report` stream
- **O11Y**: New schemas for Network/IO fields + Region-aggregated table
- **Clinic**: Adapts to new O11Y query functions

### Metrics / Thresholds
- `grpc_wait_time` as monitoring and alerting indicator for high gRPC load
- TopSQL currently: Top 100 per second (TiKV/TiDB), uploaded per minute
- Concern: uploaded data volume may be 2-3x larger; N may be reduced for performance

### Tools / Commands
```protobuf
service ResourceUsageAgent {
    rpc Report(stream ResourceUsageRecord) returns (EmptyResponse) {}
}
message ResourceUsageRecord {
    oneof record_oneof {
        GroupTagRecord record = 1;
    }
}
```

### Raw Content Excerpt
> We hope to leverage the existing TopSQL framework to provide users with more dimensions of "TopSQL" views and gRPC layer monitoring and alerting. The plan is to develop for Dedicated first, and then migrate it to NextGen and OP. For a specific TiKV node, display the top N SQL/Table/Region with the highest cumulative communication volume with TiDB within a certain period of time. For a specific TiKV node, display the top N SQL statements/Tables/Regions with the highest cumulative logical read and write IOs within a certain time range.

---

## Document 5: Auto-Performance Diagnosis Proposal
**Source**: https://pingcap.feishu.cn/wiki/wikcnsjBuLOA6HZg5wui5n3r7Jb
**Type**: Design Proposal
**Reading Status**: Inaccessible (old "doc" format, API returns 1770002 not found)

### Key Points
- Document uses legacy Feishu "doc" format (obj_type: "doc") which is not supported by the docx rawContent API
- Title suggests an automated performance diagnosis system proposal
- Created: 2022-05-23, last edited: 2022-07-04
- Cannot extract content; would need manual browser access

---

## Document 6: Diagnose Pain Points
**Source**: https://pingcap.feishu.cn/wiki/YzhBwkqwCihUFxkMm9GcBynWnXg
**Type**: Analysis / Problem Statement
**Reading Status**: Full Read

### Key Points
- Categorizes diagnostic pain points into: **Resource usage**, **Conflict**, **Performance**, **Cluster Bottleneck**
- Resource usage issues subdivided by: CPU, Memory, IO, Network
- Each pain point linked to real TCOC/GTOC/Wetech oncall cases
- Key recurring question: "What is the root cause SQL for resource spikes?"
- Priority ranking from team discussion: (1) Resource → SQL reverse lookup, (2) Transaction conflict system table persistence, (3) Execution time breakdown + AI-assisted diagnosis

### Diagnostic Pain Points Catalog

#### Resource Usage — Service degradation from insufficient resources:
| Resource | Pain Point | Real Case |
|----------|-----------|-----------|
| IO | Duration spikes from IOPS limit exceeded by large union all query | TCOC-4372 |
| CPU | Duration spikes from thread pool contention / hot region consuming raft thread resources | APID-11454/GTOC-7734 (Wetech) |
| CPU | Hot region consuming read pool CPU resources | TCOC-2629 (Wetech) |
| CPU/IO | Duration spikes from background jobs (analyze, DDL add index/SST inject) | TCOC-4145, UE-O11Y-1902175 |
| Network | Alert from slow PD TSO duration caused by network bandwidth limit | O11Y-2004978 (Wetech) |

#### Conflict — Read/Write Conflict Diagnosis:
| Pain Point | Real Cases |
|-----------|-----------|
| Write QPS drops to 0 due to write contention | 0105-lock-conflict, saas02-replay-cps, bt-wallet-txn-conflict |
| What are the contention SQL statements and transactions? | TCOC-4683, TCOC-4625 |

#### Performance — Query Latency:
| Pain Point | Real Cases |
|-----------|-----------|
| Wetech replay/upgrade/scale-in/restart performance comparison | Wallet/Saas Performance Analysis |
| Per-query latency breakdown (e.g., 4.0.13 vs 7.5.5 oltp_point_select) | N/A |
| Cluster bottleneck snowball analysis | TCOC-4683 f4samurai PoC |

### Enhancement Proposals
1. Break down system CPU, I/O, memory resource consumption → reverse lookup SQL
2. Enhance persisted lock view (data_lock_waits, deadlocks system tables)
3. Break down enough information for critical path (latency-breakdown, performance-tuning-methods)
4. Integrate AI for basic triage and analysis (slow log + grafana metrics)

### Meeting Action Items
1. Summarize current TopSQL metric collection methods for review
2. Kernel diagnostics team to schedule system table persistence work
3. Execution breakdown + AI diagnosis needs both sides to collaborate

### Raw Content Excerpt
> [IO] Duration spikes caused by iops limit exceed which is caused by large union all query. Q: What is the root cause of IO usage? [CPU] Duration spikes because of thread pool contention, like hot spot. Q: What are the queries that triggered the hot region which consumes a lot of raft thread resources? [Network] Alert caused by slow PD TSO duration which is caused by network bandwidth limit. Enhancement: Break down the system CPU, I/O, and memory resource consumption. Integrate AI for the basic triage and analysis, for slow log and grafana metrics.

---

## Document 7: Clinic User Guide
**Source**: https://pingcap.feishu.cn/wiki/wikcnMLhidTmDEwhBAFAeUPVedq
**Type**: User Guide
**Reading Status**: Inaccessible (old "doc" format, API returns 1770002 not found)

### Key Points
- Document uses legacy Feishu "doc" format (obj_type: "doc") which is not supported by the docx rawContent API
- Title indicates a user guide for PingCAP Clinic diagnostic tool
- Created: 2021-12-01, last edited: 2022-08-17
- Cannot extract content; would need manual browser access

---

## Document 8: Top SQL Internals (Top SQL Principles)
**Source**: https://pingcap.feishu.cn/wiki/EhrlwHUWJiB9hjkYVcGcRlGJnUh
**Type**: Technical Deep-dive / Architecture Document
**Reading Status**: Full Read

### Key Points
- Three-layer architecture: **Sampling (Kernel)** → **Aggregation (Kernel)** → **O11Y layer**
- Uses Go runtime pprof + goroutine label mechanism for SQL-level CPU profiling
- Per-second profiling with Top 100 SQL retention, uploaded per minute
- O11Y downsamples: Top 100 → Top 10, 1s → 15s granularity
- UI queries Top 5 SQL by cumulative cpu_time within selected time range

### Architecture Details

#### Kernel Side (TiDB):
1. **Data Collection**: pprof CPU profiling every second on all goroutines
   - At SQL execution start: `pprof.WithLabels()` binds SQL Digest + Plan Digest to `context.Context`
   - `pprof.SetGoroutineLabels()` records labels to current goroutine
   - Child goroutines inherit labels (coloring)
   - Filter samples without SQL/Plan labels → get per-SQL CPU usage
2. **Data Upload**: Keep only Top 100 SQL per second; excess merged into "others" (cpu time aggregated)
   - Accumulate 1 minute in memory (no aggregation), push via gRPC to subscriber (ng-monitoring / o11y agent)

#### O11Y Side:
1. **Downsampling**: Top 100 → Top 10, 1s → 15s window
   - Within 15s window, accumulate: cpu_time, exec_count, exec_duration
2. **Others merging**: 90 records beyond Top 10 merged into "others"
   - cpu_time: fully accumulated
   - exec_count / exec_duration: sampled only, not fully accumulated
3. **Storage**: Converted to Metrics format → pushed to VictoriaMetrics

#### UI Query Side:
- PromQL query: accumulate all SQL cpu_time in range → select Top 5
- Top 5: show all Plans with per-plan total cpu_time
- Non-Top 5: merged into "others"

### Feature Comparison
| Feature | TopSQL | SQL Statements | Slow Query |
|---------|--------|---------------|------------|
| Data Type | Time-series metrics | Aggregated statistics | Event logs |
| Granularity | Per time point | Per time window | Per execution |
| Running SQL | Yes (included) | No (only completed) | No (only completed) |
| Use Case | "What's causing CPU spike NOW?" | "Post-hoc analysis" | "Individual slow query" |
| SQL Parameters | No (digest only) | No (digest only) | Yes (normalized on TiDB Cloud) |
| Threshold | N/A | N/A | 300ms default |

### Metrics / Thresholds
- Top N per second (kernel): 100
- Top N per 15s (O11Y): 10
- Top N displayed (UI): 5
- Slow query threshold: 300ms default
- Downsampling window: 15 seconds

### Raw Content Excerpt
> Sampling layer (kernel): uses Go runtime `pprof` plus goroutine labels to measure SQL-level CPU usage. Aggregation layer (kernel): keeps only Top 100 SQL every second, packages by minute, and uploads. O11Y layer: further trims to Top 10 with 15s downsampling; the rest are merged into `others`. Top SQL is time-series data similar to metrics, showing the Top N SQL with highest CPU at each time point; it includes still-running SQL and is therefore useful for diagnosing which live queries are causing a current CPU spike.

---

## Document 9: PD oncall handbook
**Source**: https://pingcap.feishu.cn/wiki/E5KkwI6VJiFA4nkcaVEcDhWonqu
**Type**: Oncall Handbook / Troubleshooting Reference
**Reading Status**: Full Read (115K chars, extracted via agent)

### Key Points
- **Massive** PD oncall reference covering 24+ topic areas with 150+ oncall ticket references
- Topics: Region health, Region merge, Placement rules, Schedulers (balance-region, balance-leader, hot-region, evict-leader, evict-slow-store), Hot region scheduling, Scale in/out, Store limit, PD node/member management, PD panic, gRPC/HTTP interface, BatchScanRegions, TSO, Region holes, Store heartbeat, Resource Control, TLS, ETCD, Dashboard
- Status legend: fixed, in progress, unconfirmed, won't fix, correctness-related

### Diagnostic Procedures

#### TSO Issues:
- Go runtime stalls under high PD load → `ErrLogicOverflow`, clock offset > 2s
- Fix: Set `GOGC=300`, `GOMAXPROCS=32`, use Active PD Follower (v8.5+), microservice refactoring
- TSO batch tuning: `SET GLOBAL tidb_tso_client_batch_max_wait_time = 1;`, `SET GLOBAL tidb_enable_tso_follower_proxy = ON;`
- Container GOMAXPROCS: Must match actual container core count, not physical machine
- Related tickets: TCOC-4751, GTOC-8098, GTOC-8103, TICKET-6158, TCOC-4353, TICKET-8173

#### Region Health:
- Patrol region: Scans 128 regions per `patrol-region-interval` (10ms); ~78s for 1M regions
- If patrol slow: Reduce `patrol-region-interval` (tried 1ms); v8.3+ auto-adjusts scan count
- Region holes: If cache mismatch → `curl -X DELETE http://{pd_addr}/pd/api/v1/admin/cache/regions`; if data loss → online unsafe recover (v6.1+)

#### Scheduling Issues:
- Disk expansion triggers massive balance-region: Expand before reaching `low-space-ratio` (20%); adjust to 0.1 for large disks
- Hot region but no hotspot detected: Reduce `src-tolerance-ratio` and `dst-tolerance-ratio` to 1.1
- Evict-leader-scheduler: Only transfers existing leaders, doesn't prevent new ones; use placement-rule for permanent exclusion
- Evict-slow-store: v7.5.4+ adds CPU to slow detection (previously hot stores misjudged as slow)
- Leader imbalance after placement-rule changes: PD selects smallest store ID as target; balance-leader-scheduler eventually fixes

#### ETCD Issues:
- ETCD space full (quota-backend-bytes default 8GB): Manual compaction + defrag
- PD parallel scaling: Must add one PD at a time; use `pd-recover` if hit
- PD disk latency → leader switch: Log keyword `"kv gets too slow"`, `"save to etcd meet error"`
- Deleting PD follower causes leader re-election (etcd#20337)
- PD rolling restart without leader for 1+ minutes in large clusters

#### Scale in/out Best Practices:
- Two bottlenecks: PD scheduler generation speed + TiKV operator consumption (snap-io-max-bytes-per-sec)
- Scaling >5 TiKV: Set `max-store-preparing-time` to 10s
- Store limit for decommission: Don't use `store limit all` (overwrites remove-peer limit)
- Decommissioned TiKV restart causes latency jitter if PD already cleaned store info

#### Resource Control:
- `LTBTokenRPCMaxDelay` default 1s; decrease to 200ms for better latency
- RU estimation: `avg_ru_per_sql * expected_qps = estimated_total_ru`

### Metrics / Thresholds
| Parameter | Default | Notes |
|-----------|---------|-------|
| `low-space-ratio` | 0.2 (20%) | Below this, PD avoids migrating data to node |
| `patrol-region-interval` | 10ms | |
| Patrol scan batch | 128 regions | v8.3+ auto-adjusted |
| `quota-backend-bytes` (etcd) | 8GB | ETCD read-only when exceeded |
| `tso-update-physical-interval` | 50ms | |
| `MaxRequestBytes` (gRPC) | 150MB | BatchScanRegion body limit |
| `LTBTokenRPCMaxDelay` | 1s | Decrease to 200ms for better latency |
| Prometheus `scrape_timeout` | 10s | Increase to 30s for large clusters |
| Store tombstone cleanup | 1 month | After tombstone state |
| L0 slowdown/stop trigger | 20/36 SST files | Referenced in Write Stall |

### Tools / Commands
```bash
# Force delete store
curl -X DELETE http://pd-address/pd/api/v1/store/{id}?force=true
# Delete region cache
curl -X DELETE http://{pd_addr}/pd/api/v1/admin/cache/regions
# PD profile
wget http://127.0.0.1:2379/pd/api/v1/debug/pprof/zip\?seconds\=10 -O pprof.zip
# ETCD defrag
ETCDCTL_API=3 etcdctl defrag --data-dir /data2/tiup/tidb-data/pd-2379
ETCDCTL_API=3 etcdctl alarm disarm
# Table scatter
curl -X POST http://{TiDBIP}:10080/tables/{db}/{table}/scatter
# Scheduler tuning
scheduler config balance-hot-region-scheduler set src-tolerance-ratio 1.1
pd-ctl config set max-store-preparing-time 10s
pd-ctl config set tolerant-size-ratio 1
# Rate limiting
config set service-middleware grpc-rate-limit GetRegion qps 100
```

```sql
-- TSO tuning
SET GLOBAL tidb_tso_client_batch_max_wait_time = 1;
SET GLOBAL tidb_enable_tso_follower_proxy = ON;
-- Placement policies for permanent leader exclusion
CREATE PLACEMENT POLICY leader_not_in_bj LEADER_CONSTRAINTS="[-zone=bj3]";
ALTER RANGE global PLACEMENT POLICY leader_not_in_bj;
-- Split table
SPLIT TABLE t INDEX idx3 BETWEEN ("2010-01-01 00:00:00", "a") AND ("2010-01-01 00:00:00", "z") REGIONS 10;
```

```bash
# Go runtime tuning for PD
Environment="GOGC=300"      # in systemd service override
Environment="GOMAXPROCS=32" # in systemd service override
```

### Raw Content Excerpt
> PD oncall reference answer. For region health checks, PD has a process that evaluates health from the region perspective and performs corresponding adjustments, including: generating operators when region replicas (peers) are abnormal to accelerate recovery; triggering split when a region is too large; generating scheduling operations when a region does not match the current replica/placement rules; and attempting merge when regions are too small. Leader imbalance after placement-rule adjustment or large-scale TiKV failures: when a large proportion of TiKV nodes need `evict-leader`, PD selects the store with the smallest store ID among all eligible target stores.

---

## Document 10: DBaaS OnCall Manual
**Source**: https://pingcap.feishu.cn/wiki/wikcnetRFrv2USNO9usmUxxfkff
**Type**: Operational Manual / OnCall Runbook
**Reading Status**: Full Read

### Key Points
- Operational manual for TSE/Kernel Oncall supporting TiDB Cloud customers
- Covers: Debug portal access, Clinic access, TiDB Dashboard, Metrics, Logs, Configuration changes, Upgrades, Scale operations
- SOC2 compliance required for all access
- Internal users only
- Describes the full oncall workflow: Get cluster info → Know runbook → Investigate → Take action

### Diagnostic Procedures

#### Getting Started:
1. Get Cluster & Tenant Information from the Board (tenant id, plan, cluster id, status, type)
2. Know runbooks for alerts (DBaaS TiDB Alert Runbook)

#### Investigation Tools:
- **TiDB Dashboard**: SQL statement / Slow Queries / Key Visualizer / Top SQL / Manual Profiling / Continuous Profiling
- **Clinic** (primary tool):
  - Production: https://clinic.pingcap.com/
  - Staging: http://clinic-staging-1072990385.us-west-2.elb.amazonaws.com:8085
  - Dev: https://nightly.debug.tidbcloud.com/
  - Search by tenant/cluster id → Click "Clinic" → TiDB Dashboard / Metrics / Instance Logs
- **Metrics**: Access via Clinic → cluster detail → "Metrics" link
- **Logs**: Access via Clinic → cluster detail → "Instance Logs" link (Kibana)

#### Viewing tidb-operator logs (Dedicated):
1. Get cluster_id from user/TSE
2. Get shoot id and region from dashboard: `https://grafana.oauth.prod.aws.tidbcloud.com/d/3px9i92nz/cluster-map`
3. Access Loki: `{cluster="shoot/aws/us-west-2/shoot--prod--{shoot-id}",namespace="tidb-admin", container="tidb-operator"}`

#### Viewing kubelet logs:
- Same shoot id lookup, then: `{cluster="shoot/aws/us-west-2/shoot--prod--{shoot-id}",syslog="kubelet"}`

### Actions Available:
- **Configuration changes**: TSE can change tidb/tikv/pd/tiflash configs via Clinic assistant tools → GitOPS
- **Upgrades**: Via GitOPS
- **Scale-in**: Via Management Portal (customers only have scale-out in console); TSE must ensure sufficient space
- **Variables**: Workaround available but requires modifying PD config + rolling restart

### Tools / Commands
- Debug Portal: `https://debug.tidbcloud.com/` (deprecated, use Clinic)
- Clinic Production: `https://clinic.pingcap.com/`
- Grafana cluster map: `https://grafana.oauth.prod.aws.tidbcloud.com/d/3px9i92nz/cluster-map`
- Loki for operator logs: `{cluster="shoot/{cloud}/{region}/shoot--prod--{shoot-id}",namespace="tidb-admin", container="tidb-operator"}`
- View current configs on TiDB Dashboard
- Management Portal for scale-in operations

### Raw Content Excerpt
> Clinic: For prod and staging debugging and monitoring (Grafana, Logs, TiDB Dashboard) customers' clusters. Only PingCAP employees who pass SOC2 can log in. Please use FeiLian for faster access. For Advanced diagnostic features, such as log cluster, realtime diagnosis, etc. The customer cannot change the configurations of tidb, tikv, pd, and tiflash. But TSE can change the configuration of tidb, tikv, pd, and tiflash on behalf of a customer if a support ticket comes or they need help customers do performance tuning. You can use the assistant tools on Clinic to create a configuration change request.

---

## Cross-Document Insights

### Common Themes Across All Documents:
1. **Resource → SQL reverse lookup** is the #1 pain point (Docs 4, 6, 8): Need to map CPU/IO/Network spikes back to specific SQL statements
2. **TopSQL is foundational** but limited to CPU (Docs 4, 8): Network/IO extensions are in design/development (Dedicated-first)
3. **Write path is the most complex diagnostic path** (Docs 1, 2): Requires tracing through gRPC → Scheduler → Raft → RocksDB with specific metrics at each stage
4. **PD is a critical single point** for TSO, scheduling, and cluster coordination (Docs 1, 9): Go runtime GC/scheduling under load is a recurring issue
5. **Clinic is the primary diagnostic tool** for TiDB Cloud (Doc 10): Integrates Grafana, logs, TiDB Dashboard, TopSQL, Continuous Profiling

### Key Diagnostic Workflows Referenced:
1. **Duration spike**: Doc 1 (metrics catalog) → Doc 2 (write-specific) or Doc 1 (read path via Coprocessor)
2. **OOM**: Doc 3 (SOP) → memory usage metric → expensive_query → INFORMATION_SCHEMA.MEMORY_USAGE
3. **Hot region/IO**: Doc 4 (new Network/IO TopSQL) → Doc 1 (existing metrics)
4. **PD issues**: Doc 9 (comprehensive PD handbook) → Doc 1 (PD metrics)
5. **General oncall**: Doc 10 (tools/access) → specific docs by problem type

### Dedicated-Specific Details:
- Network/IO TopSQL: Developed for Dedicated first (Doc 4)
- Clinic is the primary access tool for Dedicated clusters (Doc 10)
- Configuration changes go through GitOPS workflow (Doc 10)
- Debug portal deprecated in favor of Clinic (Doc 10)
- tidb-operator logs accessed via Loki with shoot id pattern (Doc 10)
- Dedicated tier uses Gardener shoot clusters (Doc 10)

### Reading Status Summary:
| # | Document | Status | Size |
|---|----------|--------|------|
| 1 | Key metrics for troubleshooting | Full Read | ~8K chars |
| 2 | TiDB write slow Troubleshooting Guide | Full Read | ~67K chars |
| 3 | TiDB OOM SOP | Full Read | ~2K chars |
| 4 | Enhance Network/IO Hotspot Diagnosis | Full Read | ~5K chars |
| 5 | Auto-Performance Diagnosis Proposal | Inaccessible | Old doc format |
| 6 | Diagnose Pain Points | Full Read | ~3K chars |
| 7 | Clinic User Guide | Inaccessible | Old doc format |
| 8 | Top SQL Internals | Full Read | ~3K chars |
| 9 | PD oncall handbook | Full Read | ~115K chars |
| 10 | DBaaS OnCall Manual | Full Read | ~5K chars |