# Premium / NextGen (TiDB-X) -- Document Summaries

> Generated: 2026-02-15 (v2 - full re-read with enriched content)
> Source: Feishu Wiki knowledge base mining - 10 Premium/NextGen priority documents

---

## Document 1: Performance Insight Design and Implementation

**Source**: https://pingcap.feishu.cn/wiki/Xv33w2ZxSi4cJ0k3BlvcAoS4n1d
**Type**: Architecture Design / AI-Powered Diagnostics
**Reading Status**: Full Read
**Product**: TiDB Cloud (Premium + On-Premise)

### Key Points

- AI-powered performance diagnostics system that collects metrics and top SQL from Clinic/O11Y infrastructure, produces `report.json`, feeds it to AI for automatic analysis
- Defines 10 concrete diagnostic rules with specific thresholds for identifying TiDB performance bottlenecks
- Privacy-preserving: removes `digest_text` from topStatements before AI analysis, only provides metrics (avg_processed_keys, avg_total_keys, sum_cop_task_num). No SQL text, schema, or execution plan exposed to AI
- Validated with real customer data (Evergent PoC) with multiple report iterations
- Reports are available at Clinic portal with JSON-structured AI output
- Prompt has V1 and V2 versions; V2 is the current production version

### Architecture / Design Decisions

- **Pipeline**: Agent collects metrics + top SQL -> consolidated into `report.json` -> AI analysis -> performance insight report
- **AI Output Schema**: JSON with `summary`, `workload_characteristics`, `performance_bottlenecks[]` (each with `description/name`, `analysis/fact`, `suggestion`)
- Integration with Clinic API for data collection
- AI model dependency: waiting for internal model (Q1 timeline noted for OpenAPI integration)
- Structured JSON output schema for AI responses

### 10 Diagnostic Rules (Prompt V2)

| # | Rule | Condition | Threshold | Recommendation |
|---|------|-----------|-----------|----------------|
| 1 | High Parse Time | Parse phase in `Database_time:by_phase` | >5% of DB time | Check prepared statement caching; verify `CPS:by_type#mean` |
| 2 | High Compile Time | Compile phase | >10% of DB time | Check plan cache miss ratio: `Plan_cache_miss_OPS:total / Plan_cache_OPS:total` |
| 3 | High TSO Wait | `Wait_start_TSO_time:total` vs `Database_time:total` | >10% | Investigate TSO service |
| 4 | Bottleneck Outside TiDB | Top `TPS:by_orthogonality#mean` is `commit-pessimistic` AND `Avg_idle_connection_duration_in_txn:total > Avg_query_duration:total` | N/A | Application/client bottleneck |
| 5 | High KV Reads/Query | `sum_cop_task_num / exec_count` in topStatements | >20 | Review schema/index (composite/covering indexes) |
| 6 | Too Many MVCC Versions | `avg_total_keys / avg_processed_keys` in topStatements | >50 | Enable in-memory engine |
| 7 | High Disk Usage | `TiKV_disk_usage_max:total#mean` | >60% | Increase disk capacity |
| 8 | High Scheduling Latency | `P99_PD_wait_TSO_duration - Avg_PD_grpc_TSO_duration` | >2ms | Add more TiDB nodes |
| 9 | High Raft Log Write Latency | `Avg_append_log_duration:total#mean` | >3ms | Faster disks or separate Raft log disk |
| 10 | Slow Raft Log Replication | `Avg_commit_log_duration - Avg_append_log_duration` | >5ms | Network bottleneck or gRPC thread CPU; add TiKV nodes |

### Roadmap

- EBS IO jitter detection, Network jitter detection, Statistics sync load issues
- OOM/CPU profiling analysis, Index Advisor integration
- H1: SQL statement top 10, Database time chart, AI processing with internal model, new scenario rules
- Support non-default system variables by initsql

### Real Customer Validation Examples

**Evergent PoC findings:**
- High TSO wait (20.7% of DB time, 147.58s mean vs 710.96s total)
- High scheduling latency (P99 16.70ms vs avg 1.12ms = 15.58ms delta, well above 2ms threshold)
- Pessimistic lock wait time: 232.65
- CPU usage over 80% (max 1572.47%)

### Raw Content Excerpt

```
**Role:**
You are an expert in diagnosing and analyzing performance issues in **TiDB** databases.

8. **TiDB is under load and TiDB scheduling latency is high**
    - If `(P99_PD_wait_TSO_duration:total#mean - Avg_PD_grpc_TSO_duration:total#mean)` > 2ms,
      it indicates that the TiDB instance is under load, leading to slow scheduling latency.
    - Recommend adding **More TiDB Nodes**.

9. **High Raft Log Write Latency (>3ms)**
   - If `Avg_append_log_duration:total#mean` > 3ms, **disk I/O latency** for Raft log persistence
     is too high.
   - Suggest using **faster disks** or separating **Raft logs to a dedicated disk**.
```

---

## Document 2: DataBase Performance Core Metrics and Formulas

**Source**: https://pingcap.feishu.cn/wiki/wikcnKQcRvY58Gtgpph6A9UUD4b
**Type**: Reference / Metrics Documentation
**Reading Status**: Inaccessible (old "doc" format, API returned error 1770002)
**Product**: TiDB Cloud

### Key Points

- Document is in legacy Feishu "doc" format (not "docx"), which cannot be read via the docx rawContent API
- Based on the title, contains core database performance metrics definitions and calculation formulas
- Created 2022-04-28, last edited 2025-05-07 -- long-lived reference document
- Likely the foundational reference doc used by Performance Insight (Doc 1)

### Architecture / Design Decisions

- N/A (content inaccessible)

### Metrics / Thresholds

- N/A (content inaccessible)

### Troubleshooting Procedures

- N/A (content inaccessible)

### Raw Content Excerpt

Content unavailable - legacy doc format. Recommend accessing via Feishu web UI directly.

---

## Document 3: Autoscaler Framework Design

**Source**: https://pingcap.feishu.cn/wiki/KnIYwVXu9i8ytmk53fVcJkuTnpd
**Type**: Architecture Design
**Reading Status**: Full Read
**Product**: TiDB Cloud NextGen (Premium)

### Key Points

- 3-layer autoscaling framework designed for multi-year evolution of TiDB Cloud autoscaling
- Milestone-1 deploys Autoscaler in **global control plane** (with ability to move to regional)
- All dependent modules are **builtin Go interfaces** for flexibility
- HPA (Horizontal Pod Autoscaler) chosen as the scaling executor -- Kubernetes community standard
- One cluster gets exactly one executable scaler (can have multiple mockup scalers)
- Cluster-scalers run in parallel, each in a single thread/routine
- One-direction communication pattern (no callbacks) for simplicity and cross-process compatibility

### Architecture / Design Decisions

**3-Layer Structure:**

- **Layer 1 (Top-Layer Framework)**: Entrance point, integrates algorithms/input/output, supports multiple code types (Go builtin first, Python/Java/HTTP later). One all-in-one scaler per cluster, cluster-scalers run in parallel.
- **Layer 2 (Narrow-Scaler Framework)**: "Phased Component-Scaler" -- accelerates development. "Phased" = one scaling action at a time (current control plane limitation). Component-level = scales one component at a time. Scaling in/out only (no up/down -- "not a very useful feature in practice").
- **Layer 3 (Algorithm Implementation)**: Implements the scaler interface. Easily replaced/adjusted via config or ops API at runtime.

**Go Interface Specs:**

```go
type InputDataProviderV1 interface {
    GetRangeMetrics(clusterID string, name MetricNameV1, from Time, to Time) ([]Float64, []Time, error)
    GetClusterTopology(clusterID string) (ClusterTopologyV1, error)
}

type ClusterScalingExecutorV1 interface {
    SetNewClusterTopology(current ClusterTopologyV1, expected ClusterTopologyV1) error
}

type ClusterScalerV1 interface {
    PullNewClusterTopology(current ClusterTopology) (ClusterTopology, error)
}

type ComponentScalerV1 interface {
    PullNewComponentModification(componentName string) (ComponentModificationV1, error)
}

type ComponentModificationV1 struct {
    componentName    string
    currentReplicas  uint
    expectedReplicas uint
}
```

**HPA-based Executor:**
- CRUD APIs for fine-grained control, triggering scaling actions, adjusting config at runtime
- Custom Metric Provider extends scaling abilities:
  - TiDB/TiKV worker scaling based on DXF instructions
  - Coprocessor Worker: aggressive scaling step for bursty workloads

**Target components**: TiDB, Copr worker, TiKV worker, TBD TiFlash compute

**Config Model:**
- `ScalerMeta` (Name+Version=ID, Type enum for BuiltinGoInterface/Python/Java/HttpRest)
- `ScalingInput` (interface type/version, data source for metrics)
- `ScalingExecutor` (interface type/version)
- `ScalingDefaultParams` (overridable per cluster)
- `ClusterScaler` (ClusterID + ScalerName/Version + ScalingParams)

### Metrics / Thresholds

- Input metrics include: TiDBQPS, TiDBLatencyAvg, TiDBLatencyP99, TiDBCPUUsage, TiDBMemUsage, TiKVCPUUsage, TiKVMemUsage

### Troubleshooting Procedures

- N/A (design document)

### Raw Content Excerpt

```
Phased component-scaler is a sub-framework to gain rapid development capabilities:
- The scaler definition of top-layer is powerful and flexible: Can calculate decisions
  and perform scaling at anytime in any way, no restraint.
- Currently the scalers' capacity are limited based on the current cluster control plane:
  The current cluster control plane only support one scaling action at a time, can't overlapped.
- So we use a sub-framework to speedup development yet keep the framework flexible:
  Design a narrow interface "phased component-scaler", much easier to implement.
  When the capacity of TiDB control plane expended, we can turn to the top-layer interface.
```

---

## Document 4: Find the Correct Threshold to Scale TiDB with HPA

**Source**: https://pingcap.feishu.cn/wiki/BB8YwRa7vivrVekcYscckE3Mn5d
**Type**: Benchmarking / Threshold Analysis
**Reading Status**: Full Read
**Product**: TiDB Cloud NextGen (Premium)

### Key Points

- Systematic benchmark to find the CPU utilization threshold for HPA-based TiDB scaling
- Test cluster: 1x TiDB c8g.2xlarge (8C16G), 6x TiKV c8g.2xlarge, AWS us-west-2
- Workload: TPC-C with 500 warehouses, varying thread counts (T4 through T72)
- **Inflection point identified at wh500-T56 workload** -- performance stops scaling linearly
- At inflection point: TiDB Avg CPU utilization = 89-90%, CPU usage = 635%
- Cross-AZ scaling has negative/neutral impact at moderate loads; same-AZ scaling always positive

### Architecture / Design Decisions

- Testing methodology: fix TiDB count (1 instance), ensure other components are NOT bottlenecks, gradually increase concurrency to find inflection point
- Verified that adding TiDB CPU resources after inflection point yields positive QPS gains
- Cross-AZ vs same-AZ scaling yields different benefits (cross-AZ adds network latency)

### Metrics / Thresholds

**CPU Utilization Progression (Single TiDB 8C16G):**

| Threads | QPS | Avg CPU Util | CPU Usage | tpmTotal |
|---------|-----|-------------|-----------|----------|
| T4 | 2,090 | 11% | 78.7% | - |
| T8 | 4,040 | 19% | 141% | 16,681 |
| T16 | 8,080 | 35% | 255% | 33,418 |
| T24 | 11,660 | 51% | 367% | 47,660 |
| T32 | 15,430 | 64% | 460% | 63,511 |
| T40 | 18,620 | 76% | 545% | 76,245 |
| T48 | 21,700 | 85% | 612% | 89,001 |
| **T56 (inflection)** | **23,490** | **89%** | **635%** | **96,407** |
| T64 | 21,480-22,550 | 87-92% | 615-652% | 88,640-92,201 |
| T72 | 23,100 | 94% | 673% | 94,170 |

**Scaling Verification (2 TiDB):**

| Threads | 1 TiDB | 2 TiDB Same-AZ | 2 TiDB Diff-AZ |
|---------|--------|-----------------|-----------------|
| T48 | 89,001 | 90,341 (+1.5%) | 83,896 (-5.7%) |
| T56 | 96,407 | 104,261 (+8.1%) | 96,518 (+0.1%) |
| T64 | 92,201 | 118,898 (+28.9%) | 106,030 (+15.0%) |
| T72 | 94,170 | 129,198 (+37.2%) | 117,016 (+24.3%) |

**Three Threshold Strategies (all divided by 1.1 for HPA 10% tolerance):**

| Approach | Raw CPU % | HPA Target | Notes |
|----------|-----------|------------|-------|
| **Conservative** | >78% | >71% | Don't scale at T48; only scale after inflection |
| **Performance-Oriented (preferred)** | 81% | ~74% | Scale at inflection point T56 |
| **Cost-Oriented** | 85% | ~77% | Scale at T72, delay scaling |
| **Max utilization** | 80-90% | 73-82% | When CPU doesn't help perf, maximize usage |

**Critical Finding:** Do NOT scale at T48/78% CPU -- scaling to different AZ at this point causes NEGATIVE QPS impact.

### Troubleshooting Procedures

- N/A (benchmark analysis)

### Raw Content Excerpt

```
Threshold Determination Strategy:
- The inflection point exists and appears at wh500-T56
- The scaling benefit is not obvious at the wh500-T56 point but gradually becomes
  apparent for workloads after the inflection point

Conservative: CPU utilization threshold > 78% (divided by 1.1 for HPA tolerance)
Performance-Oriented (preferred): CPU utilization = 81% (divided by 1.1)
Cost-Oriented: CPU utilization = 85% (divided by 1.1)

With RCU set to 5000-20000, TiDB CPU usage can only hit 18%-21% on 8C16G node.
```

---

## Document 5: AutoScaler -- Metric Analysis & Tech Validation

**Source**: https://pingcap.feishu.cn/wiki/ZqlKww4PwiqFmUky7ZKcMwijnYc
**Type**: Technical Analysis / Algorithm Validation
**Reading Status**: Full Read
**Product**: TiDB Cloud NextGen / Dedicated

### Key Points

- Comprehensive analysis of autoscaling metrics and forecasting algorithms
- Metrics classified into 3 categories: business, performance, component topology
- Objective: adjust component topology to offset business metric changes on performance, maintaining stability and reducing costs
- Tested Holt-Winters forecasting for 6-hour ahead prediction
- Built polynomial model to fit QPS-TiDB_num-duration relationship
- REC (Resource Estimation Curve) validation with TPC-C benchmarks
- L2 Normalization used for metric normalization
- Extrapolation beyond 3 TiDB instances performs poorly -- needs more training data

### Architecture / Design Decisions

- **Metric Categories**:
  1. **Business metrics**: reflect user behavior, follow patterns, predictable (write_rpc_qps, get_rpc_qps, cop_rpc_qps)
  2. **Performance metrics**: affected by business metrics AND topology changes (latency/duration)
  3. **Component topology**: tikv_num, tidb_num (what we adjust)
- **O11Y Metric Access**: Auth0 bearer token -> cluster metadata API -> Prometheus query_range API
- **Metric Normalization**: L2 (unit vector) normalization to eliminate magnitude influence
- **Forecasting**: Holt-Winters (exponential smoothing) for 6-hour predictions at 15-min intervals
- **Performance Modeling**: Polynomial regression fitting QPS x TiDB_num -> P99 duration
  - Works for 1-3 TiDB instances; extrapolation >3 performs poorly
- **System Performance Modeling problem formulation**:
  - Establish quantitative relationship between QPS, TiDB node count, and request latency
  - Given predicted QPS + P99 duration constraint, determine minimum TiDB nodes for SLA

### Metrics / Thresholds

**TiKV metrics monitored:**
- Write: `tikv_grpc_msg_duration_seconds_count` types: `kv_prewrite|kv_commit|kv_pessimistic_lock`
- Read: types: `kv_batch_get|kv_batch_get_command|kv_get|kv_scan`
- Coprocessor: type: `coprocessor`
- Instance count: `tikv_engine_size_bytes` (count by instance)
- TiDB QPS: `tidb_executor_statement_total`

**Query patterns:**
```bash
# 15-minute downsampling
curl -G ".../api/v1/query_range" \
  --data-urlencode 'query=avg_over_time(sum(rate(tikv_grpc_msg_duration_seconds_count{
    cluster_id=~".*ID",type=~"kv_prewrite|kv_commit|kv_pessimistic_lock"}[1m])))[15m:1m]' \
  --data-urlencode 'step=900s'
```

**Holt-Winters forecast example (6-hour ahead):**
- Predictions at 15-min intervals from 2025-05-12 00:00 to 05:45
- RU range: ~10,670 to ~11,890 (relatively stable prediction)

### Troubleshooting Procedures

- N/A (research/validation document)

### Raw Content Excerpt

```
Classify metrics into three categories: business metrics, performance metrics,
and Component topology.
- Business metrics reflect user behavior, follow certain patterns, and can be
  predicted using historical data and algorithms.
- Changes in business metrics will immediately affect performance metrics.
- Adjusting the resource specifications (Component topology, eg: scale in/out)
  will also immediately affect performance metrics.
- Our objective is to adjust Component topology to offset the impact of business
  metrics changes on performance metrics, maintaining system stability and reducing costs.

System Performance Modeling:
Establish a quantitative relationship between QPS, the number of TiDB nodes,
and request latency (duration).

Resource Optimization Decision-Making:
Given a predicted QPS and a constraint on P99 duration, determine the minimum
number of TiDB nodes required to meet the SLA requirements.
```

---

## Document 6: TiDB NextGen Premium OnCall SOP

**Source**: https://pingcap.feishu.cn/wiki/OMIOwxGcUiXs2rkrdUec4U0gnTc
**Type**: Standard Operating Procedures (SOP)
**Reading Status**: Full Read
**Product**: TiDB Cloud NextGen Premium

### Key Points

- Comprehensive OnCall operations runbook for NextGen Premium clusters
- Covers profiling, debugging, and configuration operations across cloud providers (AWS + Alicloud)
- References multiple operational scripts stored at `/storage/shenjun/hack/`
- Distinguishes between AWS and Alicloud operational procedures
- Many TODOs indicate the SOP is still under active development

### Architecture / Design Decisions

- Operations are cloud-provider specific (AWS vs Alicloud have different paths)
- Files exported via S3 bucket: `s3://tidbcloud-devops-jumppod-diagnose-prod/`
- TiDB variable modification requires privileged TiDB pod with keyspace awareness
- Node debugging uses `debug-node-as-admin.sh` with `chroot /host` (note: doesn't work on AWS Bottlerocket)

### Troubleshooting Procedures

**TiKV Profiling:**
1. `Run /storage/shenjun/hack/copy-tikv-tls.sh <tikv-pod>` -- download TLS certificates
2. `kubectl port-forward <tikv-pod> 20180:20180`
3. `Run /storage/shenjun/hack/profile-tikv.sh <tls-dir>` -- perform profiling
4. `aws s3 cp ./<file> s3://tidbcloud-devops-jumppod-diagnose-prod/<dir>/<file>`

**TiFlash Profiling:**
1. `/storage/shenjun/hack/patch-tiflashgroup-malloc-conf.sh <tiflashgroup>` -- patch malloc config
2. Heap files stored in `/data0` of TiFlash container
3. Find mount path: `cat /proc/<pid>/mountinfo | grep data0`, then `findmnt /dev/nvme1n1`
4. Export varies by cloud:
   - Alicloud: Download directly via Workbench UI
   - AWS: Upload to S3 from jumpserver pod (`k cp` or `tar` for symlinks)

**Modify TiDB Variables:**
1. `/storage/shenjun/hack/neat-privileged-tidb.sh <privileged-tidb> > /tmp/my-privileged-tidb.yaml`
2. `kubectl apply -f /tmp/my-privileged-tidb.yaml`
3. Find keyspace_name, TLS secret
4. `kubectl port-forward po/my-privileged-tidb 10080:10080`
5. `/storage/shenjun/hack/activate-tidb-local.sh <secret-name> <keyspace-name>`
6. `/storage/shenjun/hack/connect-tidb-as-madmin.sh <my-privileged-tidb>`
7. `SET GLOBAL <key>=<value>`

**View Kubelet Logs:**
1. `/storage/shenjun/hack/debug-node-as-admin.sh <node>` + `chroot /host`
2. Config: `/var/lib/kubelet/ack-managed-config.yaml` (Alicloud) or `ps -eaf | grep kubelet`
3. Logs: `journalctl -u kubelet --no-pager -f --since "-5min"`
4. Note: kubelet NOT managed by systemd on AWS

**Export Files from Pod:**
- Regular files: `k cp <pod>:<path> -c <container> <local-path>`
- Symlinks: `k exec <pod> -c <container> -- tar cf - </path/to/file> | tar xf - -C <./local_dir/>`

### Metrics / Thresholds

- N/A (operational procedures)

### Raw Content Excerpt

```
TiKV Profiling and Download to Local
Run /storage/shenjun/hack/copy-tikv-tls.sh <tikv-pod> to download TLS certificates
kubectl port-forward <tikv-pod> 20180:20180
Run /storage/shenjun/hack/profile-tikv.sh <tls-dir> to perform profiling

Modify TiDB Variables
/storage/shenjun/hack/neat-privileged-tidb.sh <privileged-tidb> > /tmp/my-privileged-tidb.yaml
kubectl apply -f /tmp/my-privileged-tidb.yaml
Find the keyspace_name for the target TiDB
/storage/shenjun/hack/activate-tidb-local.sh <secret-name> <keyspace-name>
/storage/shenjun/hack/connect-tidb-as-madmin.sh <my-privileged-tidb>
SET GLOBAL <key>=<value>
```

---

## Document 7: Troubleshooting Scenario (Premium)

**Source**: https://pingcap.feishu.cn/wiki/V5grwygtUi7ZjZk8oYSc50ktn6d
**Type**: Troubleshooting Framework / Data Classification
**Reading Status**: Full Read
**Product**: TiDB Cloud Premium

### Key Points

- Categorizes troubleshooting data into layers for systematic diagnosis
- Provides a mental model for TSE/RD troubleshooting workflow: data gathering -> root cause -> operation
- Categorizes remediation operations by target object

### Architecture / Design Decisions

- Layered troubleshooting model: Cloud -> IaaS -> TiDB -> Database

### Data Categories for Root Cause Analysis

1. **Cloud Resources Related**:
   - K8s resource status (kubectl API server)
   - Cloud resources: NAT, LB, Route53, SecurityGroup, EBS
   - Status: CSP web console; Logs: CloudWatch

2. **IaaS Resources Related**:
   - Metrics: Node CPU/memory/disk (K8s generated, O11Y collected); TiDB node labels/CPU/memory/EBS/network (node_exporter)
   - Logs: Sys logs (K8s generated, O11Y collected), admin node logs

3. **TiDB Related**:
   - Cluster info: Topology, region, tenant, TiDB version, SLA, CPU/Memory/QPS trends, database size
   - Metrics: TiDB, TiKV/TiFlash, PD, TiCDC (component-generated, vmagent collected)
   - Logs: Component logs (TiDB generated, vector daemonset collected)

4. **Database Related**:
   - Slow query, Top SQL, SQL Statement, Key Visualizer, Profile

### Operations to Solve Problems

| Target | Tool |
|--------|------|
| Cloud Resources | CSP web console |
| K8s | kubectl |
| TiDB | Change component configuration, change system variables |

### Troubleshooting Procedures

1. **Read data to locate root cause** -- systematically check each layer from cloud down to database
2. **Operations to solve** -- modify cloud resources via CSP console, K8s via kubectl, TiDB via config/variable changes

### Raw Content Excerpt

```
When TSE or RD handles troubleshooting, he needs to read some data to locate the root cause.
The data could be divided into the following categories:
- Cloud Resources Related: K8s resource status, Cloud resource (NAT, LB, route53, securitygroup, ebs)
- IaaS Resources Related: Node cpu/memory/system disk, Pod Daemonset/deployment, image version
- TiDB Related: Topology, region, tenant, tidb version, SLA, CPU, Memory, QPS Trend, database size
- Database Related: Slow query, Top SQL, SQL Statement, Key visualizer, Profile
```

---

## Document 8: Observability & Diagnosibility

**Source**: https://pingcap.feishu.cn/wiki/SXDaw7VylirlAFk2ci7cAiJanDf
**Type**: Container / Index Page
**Reading Status**: Empty Document (title only, no content)
**Product**: TiDB Cloud

### Key Points

- Document exists as a wiki node with child nodes (has_child: true) -- it's a parent container page
- No actual content in the document body
- Created 2024-10-30
- Wiki node token: `SXDaw7VylirlAFk2ci7cAiJanDf`, space_id: `7431754965686730753`

### Architecture / Design Decisions

- N/A (empty container page -- real content in child nodes)

### Metrics / Thresholds

- N/A

### Troubleshooting Procedures

- N/A

### Raw Content Excerpt

(empty -- title-only container page)

---

## Document 9: Slow Query Schema

**Source**: https://pingcap.feishu.cn/wiki/WwElwrBEDiz3IskGC7uckid8neg
**Type**: Data Schema / Technical Reference
**Reading Status**: Full Read
**Product**: TiDB Cloud (Premium/NextGen)

### Key Points

- Defines three PyArrow schemas for slow query data processing
- Uses Apache Arrow (PyArrow) for schema definition -- suggests Parquet/columnar storage for data pipeline
- Comprehensive field coverage: 80+ fields in slowlogs_schema
- Separate schemas for queries and plans enable privacy-preserving analysis (metrics without SQL text)

### Architecture / Design Decisions

- Three-schema design:
  1. **slowlogs_schema** (80+ fields): Complete slow query log record
  2. **querys_schema**: `digest` -> `query` text mapping (separated for privacy)
  3. **plans_schema**: `plan_digest` + `digest` -> `binary_plan` mapping (separated for privacy)
- NextGen-specific fields: `worker_role`, `resource_group`, `request_unit_read/write`, `time_queued_by_rc`, `is_tiflash`

### Metrics / Thresholds

**Key timing fields:**
- `query_time`, `parse_time`, `compile_time`, `rewrite_time`, `optimize_time`
- `wait_ts`, `prewrite_time`, `commit_time`, `get_commit_ts_time`, `commit_backoff_time`
- `cop_time`, `process_time`, `wait_time`, `backoff_time`, `lockkeys_time`
- `kv_total`, `pd_total`, `backoff_total`, `write_sql_response_total`

**Key count/size fields:**
- `write_keys`, `write_size`, `prewrite_region`, `request_count`
- `total_keys`, `process_keys`, `result_rows`, `mem_max`, `disk_max`

**RocksDB-specific:**
- `rocksdb_delete_skipped_count`, `rocksdb_key_skipped_count`
- `rocksdb_block_cache_hit_count`, `rocksdb_block_read_count`, `rocksdb_block_read_byte`

**Coprocessor:**
- `cop_proc_avg/p90/max/addr`, `cop_wait_avg/p90/max/addr`

**Resource Control:**
- `resource_group`, `request_unit_read`, `request_unit_write`, `time_queued_by_rc`

### Troubleshooting Procedures

- N/A (schema reference)

### Raw Content Excerpt

```python
slowlogs_schema = pa.schema([
    ("timestamp", pa.int64()),
    ("digest", pa.string()),
    ("plan_digest", pa.string()),
    ("query_time", pa.float64()),
    ("parse_time", pa.float64()),
    ("compile_time", pa.float64()),
    ("total_keys", pa.int64()),
    ("process_keys", pa.int64()),
    ("resource_group", pa.string()),
    ("request_unit_read", pa.float64()),
    ("request_unit_write", pa.float64()),
    ("worker_role", pa.string()),
    ("is_tiflash", pa.bool_()),
    # ... 60+ more fields
])

querys_schema = pa.schema([
    ("digest", pa.string()),
    ("query", pa.string())
])

plans_schema = pa.schema([
    ("plan_digest", pa.string()),
    ("digest", pa.string()),
    ("binary_plan", pa.string())
])
```

---

## Document 10: OOM Detection

**Source**: https://pingcap.feishu.cn/wiki/YwVywpw4oiuAymkFeg8cFW8Gn0f
**Type**: Container / Index Page
**Reading Status**: Empty Document (title only, no content)
**Product**: TiDB Cloud NextGen

### Key Points

- Document exists as a wiki node with child nodes (has_child: true) -- parent container page
- No content in document body
- Created 2024-10-16, part of the NextGen diagnostics space
- OOM detection is referenced in Doc 1 (Performance Insight) as a roadmap item: "OOM or CPU usage high -> Profiling analysis"
- Wiki node token: `YwVywpw4oiuAymkFeg8cFW8Gn0f`, space_id: `7426314386213289986`

### Architecture / Design Decisions

- N/A (empty container page)

### Metrics / Thresholds

- N/A

### Troubleshooting Procedures

- N/A

### Raw Content Excerpt

(empty -- title-only container page)

---

# Cross-Document Analysis

## Document Relationships

```
Performance Insight (Doc 1)
  |-- uses --> Core Metrics & Formulas (Doc 2) [reference data]
  |-- uses --> Slow Query Schema (Doc 9) [data contract]
  |-- roadmap --> OOM Detection (Doc 10) [planned feature]

Autoscaler Framework (Doc 3)
  |-- validated by --> HPA Threshold Benchmark (Doc 4) [threshold data]
  |-- research --> Metric Analysis & Tech Validation (Doc 5) [algorithms]

OnCall SOP (Doc 6)
  |-- follows --> Troubleshooting Scenario (Doc 7) [framework]
  |-- needs --> Observability & Diagnosibility (Doc 8) [data sources]
```

## Key Thresholds Summary (Across All Docs)

| Metric | Threshold | Action | Source |
|--------|-----------|--------|--------|
| Parse time % of DB time | >5% | Check prepared statement caching | Doc 1 |
| Compile time % of DB time | >10% | Check plan cache effectiveness | Doc 1 |
| TSO wait % of DB time | >10% | Investigate TSO service | Doc 1 |
| KV reads per query | >20 | Review index design | Doc 1 |
| MVCC versions ratio (total/processed keys) | >50 | Enable in-memory engine | Doc 1 |
| TiKV disk usage | >60% | Increase disk capacity | Doc 1 |
| PD wait TSO - gRPC TSO delta | >2ms | Add TiDB nodes | Doc 1 |
| Raft log append latency | >3ms | Faster disks / separate Raft logs | Doc 1 |
| Raft log commit-append delta | >5ms | Add TiKV nodes / check network | Doc 1 |
| TiDB CPU utilization (HPA, performance) | 81% raw / ~74% HPA target | Scale out TiDB | Doc 4 |
| TiDB CPU utilization (HPA, conservative) | 78% raw / ~71% HPA target | Scale out TiDB | Doc 4 |
| TiDB CPU utilization (HPA, cost) | 85% raw / ~77% HPA target | Scale out TiDB | Doc 4 |
| HPA tolerance factor | 1.1 (10%) | Applied to all HPA thresholds | Doc 4 |
| TiDB CPU inflection point (8C16G) | 89% avg utilization at T56 | Performance degrades beyond this | Doc 4 |

## Key Prometheus Queries (from Doc 5)

```promql
# TiKV write RPC QPS (15-min downsampled)
avg_over_time(sum(rate(tikv_grpc_msg_duration_seconds_count{
  cluster_id=~".*ID",type=~"kv_prewrite|kv_commit|kv_pessimistic_lock"}[1m])))[15m:1m]

# TiKV read RPC QPS
avg_over_time(sum(rate(tikv_grpc_msg_duration_seconds_count{
  cluster_id=~".*ID",type=~"kv_batch_get|kv_batch_get_command|kv_get|kv_scan"}[1m])))[15m:1m]

# TiKV coprocessor RPC QPS
avg_over_time(sum(rate(tikv_grpc_msg_duration_seconds_count{
  cluster_id=~".*ID",type=~"coprocessor"}[1m])))[15m:1m]

# TiKV instance count
count(count(tikv_engine_size_bytes{cluster_id=~".*ID"}) by (instance))

# TiDB QPS (1-hour downsampled)
avg_over_time(sum(rate(tidb_executor_statement_total{
  cluster_id=~".*ID"}[1m]))[60m:1m])
```

## Unreadable Documents (Need Alternative Access)

1. **Doc 2 (Core Metrics)**: Old `doc` format, needs web UI or format conversion
2. **Doc 8 (Observability & Diagnosibility)**: Container page, need to read child nodes
3. **Doc 10 (OOM Detection)**: Container page, need to read child nodes

## Metric Collection Stack (from Docs 5, 7)

| Layer | Collector | What |
|-------|-----------|------|
| TiDB components | vmagent | TiDB/TiKV/PD/TiCDC metrics |
| IaaS nodes | node_exporter | CPU/memory/disk/network |
| K8s | O11Y | Pod/DaemonSet/Deployment metrics |
| Logs | vector (DaemonSet) | Component logs |
| Cloud | CloudWatch | NAT/LB/EBS logs |

## Document 11 (B1): WeTech Cluster Latency Evaluation

- **Source**: [WeTech Cluster Latency Evaluation](https://pingcap.feishu.cn/wiki/W2uqwAZSZilIk0kJfP0cQUOfne0)
- **Quality**: 5 | **Last Edited**: 2025-09-12 | **Freshness**: Aging (155d)
- **Purpose**: Defines a universal methodology for evaluating cluster latency health and performance headroom using efficiency curves (QPS vs latency).

### Key Points

1. **Queuing theory foundation**: All TiDB performance follows queuing system behavior
   - Simple queue (single worker): latency has a clear inflection point at max processing capacity
   - Multi-consumer queue (thread pool): similar inflection behavior
   - Complex system (TiDB/TiKV pipeline): multiple queuing stages in series/parallel, but dominant bottleneck makes it behave like simple queue

2. **Efficiency curve (QPS vs Latency)**:
   - Left of inflection: latency increases near-linearly with QPS (healthy zone)
   - At inflection: maximum QPS achievable before degradation
   - Right of inflection: latency spikes dramatically, system unusable

3. **Measurement methodology**:
   - **With load testing**: Apply increasing pressure, record QPS + latency pairs, plot curve, find inflection point
   - **Without load testing** (production clusters): Use historical data from stable periods as sampling points, treat like pressure test results
   - Select stable time windows (internal state stable, input pattern stable, QPS may vary)
   - More sampling points = better accuracy

4. **Performance headroom calculation**:
   - System capacity = QPS at inflection point
   - Performance headroom = System capacity - Current QPS
   - If business latency tolerance > inflection latency: use max QPS as capacity
   - If business latency tolerance < inflection latency: find QPS at tolerance threshold

### WeTech Production Cluster Benchmarks

| Cluster | Metric | Inflection Zone | Target | Notes |
|---------|--------|-----------------|--------|-------|
| Order (AP SQL) | P999 | 500ms-1s | 125-250ms | Keep well left of inflection |
| Order (Write SQL) | kv_prewrite P99 | 100-200ms | <50ms, max 100ms | Use storage-layer metric, not entry latency |
| SaaS | P999 | 500ms-1s | 125-250ms | Mixed workload, same target as Order AP |
| Wallet | P999 | Far from inflection | 60ms (business req) | Already excellent, scale with business growth |

### Diagnostic Application

- **Is latency reasonable?** -> Plot current position on efficiency curve; left of inflection = healthy
- **How much headroom?** -> Calculate (inflection QPS - current QPS)
- **Which metric for writes?** -> Use kv_prewrite gRPC latency (storage layer), NOT entry latency
- **Mixed workload cluster?** -> Use P999 as primary indicator; if need precision, use manual sampling workflow
- **Precise evaluation needed?** -> Human selects stable windows -> script collects data -> plot curve -> calculate

### Related Algorithms

- TiKV slow node eviction algorithm v2 (Slow Node Hunting) uses efficiency curves
- New autoscale algorithm proposal uses efficiency curves