# Root Cause Classification Standard

## 1. Introduction

This document defines the standardized, three-level classification system for identifying the technical root cause of an issue. This system is intended for use by engineering and technical support teams after a thorough investigation is complete.

Its primary goal is to move beyond symptom-level analysis and pinpoint the specific technical component, system behavior, or external factor that was the ultimate source of the problem. The data gathered from this system is designed to be highly actionable for driving engineering roadmaps, prioritizing technical debt, and improving product stability.

## 2. Design Philosophy

The design of this system is based on two core principles:

* **Specificity and Depth**: The system uses a three-level hierarchy (`L1` -> `L2` -> `L3`) to provide the necessary granularity to be actionable for engineers. The goal is to answer **"Which specific component failed and why?"** rather than just "What was the problem?".
* **Separation of Concerns**: This classification system is distinct from the user-facing "Problem Pattern" classification. It should be applied at the **end** of a ticket's lifecycle, once the definitive technical cause has been determined.

## 3. Usage Guidelines

1. **Classify at Resolution**: This classification should be filled out by the engineer or technical expert when the ticket is being resolved or closed, not upon creation.
2. **Select the Deepest Path**: Always choose the most specific and accurate classification path available. If an issue was caused by a stale stats, `kernel` -> `optimizer_executor` -> `stale_stats` is the correct choice.
3. **Use Orthogonal Dimensions**: For a richer analysis, this system should be used in conjunction with a separate, orthogonal "Root Cause Nature" dimension (e.g., `Software Defect`, `Configuration Issue`, `Workload-Related`) to capture the "why" as well as the "what/where".

## 4. Classification Table

The following is the complete definition table for the Root Cause Classification system.

| Level 1 (L1)            | Level 2 (L2)                   | Level 3 (L3)                  | Description                                                                                     |
| :---------------------- | :----------------------------- | :---------------------------- | :---------------------------------------------------------------------------------------------- |
| **Kernel**        | **Optimizer & Executor** | `incorrect_plan`            | The optimizer chose a suboptimal execution plan.                                                |
|                         |                                | `stale_stats`               | Outdated or missing statistics led to a poor plan choice.                                       |
|                         |                                | `executor_inefficiency`     | A specific execution operator (e.g., Join, Sort) was inefficient.                               |
|                         |                                | `expression_bottleneck`     | The calculation of a complex expression or function was the bottleneck.                         |
|                         | **Hotspot Issue**        | `tikv_cpu_hotspot`          | Abnormally high CPU usage on a specific TiKV node.                                              |
|                         |                                | `single_region_hotspot`     | Load is concentrated on a single data Region.                                                   |
|                         |                                | `tikv_io_hotspot`           | Abnormally high IO pressure on a specific TiKV node.                                            |
|                         |                                | `memory_leak`               | Sustained memory growth suggesting a software memory leak.                                      |
|                         | **Storage Engine**       | `compaction_issue`          | Problems related to the data compaction process.                                                |
|                         |                                | `write_stall`               | The                   initiated a stall to slow down incoming writes.                           |
|                         |                                | `corruption`                | Data corruption was detected at the storage level.                                              |
|                         | **CDC Fault**            | `checkpoint_delay`          | TiCDC Checkpoint latency, causing accumulated sync delays.                                      |
|                         |                                | `changefeed_failure`        | A TiCDC Changefeed task has failed, interrupting data sync.                                     |
|                         | **GC**                   | `tikv_gc`                   | Abnormal or slow TiKV garbage collection process.                                               |
|                         | **Timestamp**            | `tso`                       | Abnormal PD timestamp service, causing transaction delays or failures.                          |
|                         | **Cluster Stability**    | `node_restart`              | A cluster node has unexpectedly quit or was restarted.                                          |
|                         |                                | `pd_leader_switch`          | The PD leader has switched, potentially causing minor service jitter.                           |
|                         | **Correctness**          | `(N/A)`                     | Issues related to data consistency, transaction isolation, or logical correctness.              |
| **Control Plane** | **Bug**                  | `frontend`                  | A defect in the user interface or interaction logic.                                            |
|                         |                                | `backend_api`               | A defect in a backend API service.                                                              |
|                         |                                | `scheduler`                 | A defect in the task or cluster scheduling logic.                                               |
|                         | **Config Issue**         | `platform_misconfig`        | A misconfiguration in the control plane itself.                                                 |
| **Cloud Vendor**  | **Resource Outage**      | `single_instance_failure`   | Failure of a single compute instance                                                            |
|                         |                                | `single_volume_failure`     | Failure or unavailability of a single storage volume                                            |
|                         |                                | `zonal_outage`              | A network or service outage at the Availability Zone (AZ) level.                                |
|                         |                                | `regional_outage`           | A core service outage at the entire Region level.                                               |
|                         | **Resource Degradation** | `network_performance_issue` | The underlying network is experiencing high latency or packet loss, but is not completely down. |
|                         |                                | `storage_performance_issue` | Storage volume (disk) IOPS/throughput is significantly below expected levels.                   |
|                         |                                | `compute_performance_issue` | Degraded performance of a compute instance (e.g., 'noisy neighbor' issue).                      |
|                         | **Service Outage**       | `dependent_service_outage`  | A dependent cloud service (e.g., object storage, DNS) is unavailable.                           |