--- name: tikv-fast-tune description: | TiKV performance diagnosis using the Fast Tune methodology. Provides a deterministic decision tree that an agent executes node-by-node to identify performance bottlenecks across TiKV's internal layers. Use when troubleshooting TiKV write/read latency, throughput drops, or jitter issues on any TiDB Cloud cluster. --- # TiKV Fast Tune Performance Diagnosis This skill encodes the Fast Tune methodology (Liu Cong, PingCAP 2020) as a **machine-executable decision tree**. The agent starts at ROOT and follows edges mechanically -- each node specifies exactly what metric to fetch, what condition to evaluate, and which node to visit next. Two capabilities: 1. **Decision Tree** -- Deterministic node-by-node diagnostic flowchart 2. **Metric Catalog** -- Exact PromQL for every metric, adapted for O11Y API **TiKV version compatibility:** The VERSION_DETECT node (run after PRE_INFRA_CHECK) automatically probes for raft-engine metrics. On TiKV 7.x+ (raft-engine replaces RaftDB), metric names in W_ROCKSDB_WRITE, W_WAL_SYNC, and W_BATCH_SIZE are swapped via the TIKV_VERSION variable. No manual verification needed. ## Terminology | Term | Meaning | |------|---------| | **CLUSTER_ID** | The SharedPool ID (e.g., `2013516099403907072`). Used as `tidb_cluster_id` in PromQL and as `applicationID` for O11Y UUID resolution. | | **Spark** | Confirmed correlation between Result and Cause metric (\|r\| > 0.7 or threshold exceeded) | | **Result metric** | The symptom (typically Write-RPC QPS or latency) | | **Cause metric** | The hypothesized root cause | ## Core Concept: Clashing Spark Detection Spark detection is a **routing heuristic** (branch selector), not a causal scoring mechanism. It determines which branch of the decision tree to follow next. Actual diagnosis is confirmed by threshold checks at terminal nodes, not by correlation alone. At nodes marked with `spark_check`, the agent overlays a Result metric with a Cause metric and computes correlation. Use the `detect_spark()` function defined in the Spark Detection section below. - `|r| > 0.7` = **SPARK** (hypothesis supported -- strong co-movement detected, follow YES branch) - `|r| < 0.3` = **NO_SPARK** (move to NO branch) - `0.3 <= |r| <= 0.7` = **WEAK** (investigate further, note in report) - Constant series or <200 points = **INCONCLUSIVE** (note, continue to next node) - Both series CV < 5% = **STEADY_STATE** (skip correlation, use absolute threshold on cause metric instead) **STEADY_STATE handling at spark_check nodes:** When `detect_spark()` returns STEADY_STATE, correlation is meaningless (both series are near-constant). Instead, compare the `cause_mean` value from the return dict against the node's **steady_state_threshold** (defined per node below). If `cause_mean` exceeds the threshold, treat as SPARK (the cause metric is persistently elevated even though it's not fluctuating). If `cause_mean` is below the threshold, treat as WEAK (note the steady state in report but continue to next node). If no steady_state_threshold is defined for a node, treat as WEAK. **Confidence labels on terminal nodes:** - **HIGH** -- Threshold-based diagnosis (e.g., write stall > threshold, cache hit < 95%). No orthogonal confirmation needed. - **MEDIUM** -- Spark-based diagnosis. Specify one orthogonal confirming metric per node in report. - **LOW** -- Fallback/elimination path. Flag as tentative, recommend manual investigation. **Inverted metrics:** high latency = low QPS. `r < -0.7` also counts as a spark. ## TiKV Architecture (Layer Depth) ``` Layer 0: gRPC Server (shallowest) Layer 1: TXN Scheduler / Latch Layer 2: Coprocessor / MVCC Storage Layer 3: RaftStore (StoreLoop + ApplyLoop) Layer 4: RocksDB (KVDB + RaftDB) Layer 5: Disk (deepest) ``` **Rule:** When multiple layers show problems, the deepest is the root cause. Write path: L0 -> L1 (latch) -> L2 (MVCC) -> L3 (StoreLoop: RaftDB, ApplyLoop: KVDB) -> L5 Read path: L0 -> L2 (Coprocessor/Storage -> Snapshot) -> L4 (memtable -> cache -> SST) -> L5 ## O11Y Label Adaptation Remove `k8s_cluster` filter (O11Y UUID already scopes data). Keep `tidb_cluster_id="${CLUSTER_ID}"`. Test with instant query first. --- ## Decision Tree Node Schema Every node follows this structure: ``` id: Unique node identifier action: What to do at this node metrics: PromQL queries to execute (keyed by name) condition: How to evaluate the fetched data branches: Where to go based on condition outcome YES/SPARK: -> next node or terminal diagnosis NO: -> next node or terminal diagnosis diagnosis: (terminal nodes only) Root cause description config: (terminal nodes only) Recommended config changes ``` The agent MUST execute every node reached by following tree edges. Never skip a reachable node. **parallel_checks semantics:** Execute ALL listed sub-nodes. Collect ALL findings (not just the first hit). Report the **deepest-layer finding** as the primary diagnosis. If multiple sub-nodes produce diagnoses, report all of them ranked by layer depth (deepest = most likely root cause). If ALL sub-nodes return negative (no diagnosis), follow the `all_negative` branch if defined, or report "UNDIAGNOSED at this layer." **Confidence-priority exception:** If a shallower finding has strictly higher confidence than ALL deeper findings (e.g., L1 HIGH vs L5 MEDIUM), report the shallower finding as primary and the deeper findings as secondary. The deepest-layer rule applies only when findings have equal confidence. Rationale: a HIGH-confidence shallow finding has stronger evidence than a MEDIUM-confidence deep finding, which may be coincidental correlation rather than true root cause. **Caveat:** The deepest-layer-wins rule assumes a linear causal chain (deeper layer causes shallower symptoms). Under feedback loops (e.g., GC stalls -> MVCC accumulation -> more GC work), the rule may not identify the initiating cause. When multiple layers show findings, report ALL with layer depths and let the operator decide which to address first. --- ## Decision Tree: Entry Point The agent starts at PRE_INFRA_CHECK, then VERSION_DETECT, then proceeds to ROOT. ``` NODE: PRE_INFRA_CHECK [MANDATORY GATE - run before ROOT] action: Check infrastructure health to avoid misdiagnosis. Uses K8S_UUID (resolved via o11y-metrics-api auto-discovery). metrics: cpu_throttle_rate: > rate(container_cpu_cfs_throttled_periods_total{ namespace="tidb${CLUSTER_ID}", pod=~".*tikv.*"}[5m]) / rate(container_cpu_cfs_periods_total{ namespace="tidb${CLUSTER_ID}", pod=~".*tikv.*"}[5m]) memory_usage_ratio: > container_memory_working_set_bytes{ namespace="tidb${CLUSTER_ID}", pod=~".*tikv.*", container="tikv"} / container_spec_memory_limit_bytes{ namespace="tidb${CLUSTER_ID}", pod=~".*tikv.*", container="tikv"} pd_request_p99: > histogram_quantile(0.99, sum(rate( tikv_pd_request_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) by (le)) sentinel_check: > tikv_grpc_msg_duration_seconds_count{ tidb_cluster_id="${CLUSTER_ID}"} region_count: > sum(tikv_raftstore_region_count{ tidb_cluster_id="${CLUSTER_ID}", type="region"}) resolved_ts_lag: > max(tikv_resolved_ts_min_resolved_ts_gap_seconds{ tidb_cluster_id="${CLUSTER_ID}"}) titan_gc_duration: > sum(rate(tikv_engine_blob_gc_duration_seconds_sum{ tidb_cluster_id="${CLUSTER_ID}"}[5m])) titan_blob_file_count: > sum(tikv_engine_titandb_num_blob_files_at_level{ tidb_cluster_id="${CLUSTER_ID}"}) note: cpu_throttle_rate and memory_usage_ratio use K8S_UUID endpoint. All other metrics use APP_UUID endpoint. Checks 5-7 are fail-open: skip silently if metric not available (feature not enabled or metric not exposed). policy: fail-open. If a metric is unavailable, log warning and continue. checks: 1. CPU throttling: condition: avg(cpu_throttle_rate) > 0.25? YES: DIAGNOSIS "CPU throttling detected (>25% periods throttled)" ACTION "Raise CPU limits or reduce workload. Do NOT diagnose as disk issue." NOTE "This would be misdiagnosed as 'disk slow' at W_FRONTEND_FLOW without this check." END 2. Memory pressure: condition: avg(memory_usage_ratio) > 0.90? YES: DIAGNOSIS "Memory pressure detected (RSS >90% of limit)" ACTION "Increase memory limits. Check block-cache-size. Risk of OOM kill." NOTE "OOM restart would be misdiagnosed as 'single-instance bad disk' at S1_BALANCE." END 3. PD health: condition: avg(pd_request_p99) > 1s? YES: DIAGNOSIS "PD communication slow (P99 >1s)" ACTION "Check PD cluster health. Network partition or PD overload." NOTE "Would be misdiagnosed as 'RaftStore pool too small' at W_RAFTSTORE." END 4. Metric availability: condition: sentinel_check returns empty result? YES: DIAGNOSIS "INCONCLUSIVE: TiKV metrics not available in O11Y" ACTION "Verify CLUSTER_ID and O11Y registration." END 5. Region split storms: metrics: region_count: > sum(tikv_raftstore_region_count{ tidb_cluster_id="${CLUSTER_ID}", type="region"}) condition: Query at start and end of window. If delta(region_count) > 20% of initial value, region churn is likely disrupting performance. YES: DIAGNOSIS "Region split/merge storm detected (>20% region count change)" confidence: HIGH (threshold-based) ACTION "Check PD scheduling. Pause balance-region/split with pd-ctl. Wait for region count to stabilize before diagnosing further." NOTE "Would be misdiagnosed as latch contention (W_LATCH) or PD scheduling (W_PD_SCHEDULING) without this check." END 6. TiCDC resolved-TS lag: metrics: resolved_ts_lag: > max(tikv_resolved_ts_min_resolved_ts_gap_seconds{ tidb_cluster_id="${CLUSTER_ID}"}) condition: avg(resolved_ts_lag) > 300s (5 minutes)? policy: fail-open (metric absent if CDC not enabled -- skip silently) YES: FLAG_AND_CONTINUE DIAGNOSIS "TiCDC resolved-TS lag >5min, blocking MVCC GC" confidence: HIGH (threshold-based) ACTION "Check TiCDC changefeed health. Stalled changefeed blocks GC, causing MVCC version accumulation that slows both reads and writes." NOTE "FLAG_AND_CONTINUE: TiCDC lag is a root cause but other independent issues (bad disk, CPU, etc.) can coexist. MVCC-related findings in the tree below (R6_GC, R2B_SCAN_COUNT) may be secondary to this. Continue tree execution; include this flag in final report." 7. Titan GC pressure: metrics: titan_gc_duration: > sum(rate(tikv_engine_blob_gc_duration_seconds_sum{ tidb_cluster_id="${CLUSTER_ID}"}[5m])) titan_blob_file_count: > sum(tikv_engine_titandb_num_blob_files_at_level{ tidb_cluster_id="${CLUSTER_ID}"}) condition: titan_gc_duration > 0 AND titan_blob_file_count > 100000? policy: fail-open (metrics absent if Titan not enabled -- skip silently) YES: FLAG_AND_CONTINUE DIAGNOSIS "Titan GC pressure (high blob file count with active GC)" confidence: MEDIUM (threshold-based; confirm with disk IO during Titan GC) ACTION "Check Titan GC settings. Consider disabling Titan if blob-file-size is too small. Adjust min-gc-batch-size, max-gc-batch-size." NOTE "FLAG_AND_CONTINUE: Titan GC competes for disk IO with normal compaction. Disk IO findings below (S3_COMPACTION_FLOW, W_WAL_SYNC, W_FRONTEND_FLOW) may be amplified by Titan GC. Continue tree execution; include this flag in final report." ALL pass: -> VERSION_DETECT ``` ``` NODE: VERSION_DETECT [MANDATORY - run after PRE_INFRA_CHECK, before ROOT] action: Detect TiKV version (5.x-6.x vs 7.x+) by probing for raft-engine metrics. TiKV 7.x replaces RaftDB with raft-engine, changing metric names used in W_ROCKSDB_WRITE, W_WAL_SYNC, and W_BATCH_SIZE. metrics: raft_engine_probe: > tikv_raft_engine_write_duration_seconds_count{ tidb_cluster_id="${CLUSTER_ID}"} condition: Does raft_engine_probe return any data? error_handling: If the raft_engine_probe query itself errors (network timeout, auth failure, 5xx), treat as probe failure. Default to TIKV_VERSION = "5.x" (original metric names) as a safe fallback. Log "VERSION_DETECT: probe query failed, defaulting to 5.x metrics." branches: YES (raft-engine present -> TiKV 7.x+): Set TIKV_VERSION = "7.x" Metric adapter (swap these metric names in affected nodes): W_ROCKSDB_WRITE: raftdb_write_p99 -> avg(tikv_raft_engine_write_duration_seconds{ tidb_cluster_id="${CLUSTER_ID}", type="write_percentile99"}) W_WAL_SYNC: wal_p99 -> avg(tikv_raft_engine_write_duration_seconds{ tidb_cluster_id="${CLUSTER_ID}", type="wal_percentile99"}) wal_max -> avg(tikv_raft_engine_write_duration_seconds{ tidb_cluster_id="${CLUSTER_ID}", type="wal_max"}) W_BATCH_SIZE: Note: raft-engine does not expose per-write WAL size as a gauge. Skip W_BATCH_SIZE trend analysis; log "W_BATCH_SIZE skipped (raft-engine)". S2_WRITE_STALL: raftdb_stall: Skip on 7.x. Raft-engine has no RocksDB write stall mechanism (no L0/memtable/pending-bytes stalls). Log "S2_WRITE_STALL: raftdb_stall skipped (raft-engine has no RocksDB write stall). Checking kvdb_stall only." -> ROOT NO (no raft-engine -> TiKV 5.x-6.x): Set TIKV_VERSION = "5.x" Use original metric names in all nodes (no swaps). -> ROOT ``` ``` NODE: ROOT action: Fetch Write-RPC P99 and Read-RPC P99 over investigation window metrics: write_p99: > histogram_quantile(0.99, sum(rate( tikv_grpc_msg_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type=~"kv_prewrite|kv_commit|kv_pessimistic_lock"}[2m])) by (le)) read_p99: > histogram_quantile(0.99, sum(rate( tikv_grpc_msg_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type=~"kv_get|kv_batch_get|kv_batch_get_command|kv_scan|coprocessor" }[2m])) by (le)) condition: Compare avg(write_p99) vs avg(read_p99) over window. "Elevated" means: P99 > 50ms sustained for >50% of the window, OR P99 > 200ms at any point (spike detection). If user reports a specific symptom, use their threshold instead. branches: write elevated (write_p99 > 50ms): -> S1_BALANCE read elevated (read_p99 > 50ms): -> R0_READ_BALANCE both elevated: -> S1_BALANCE first, then R0_READ_BALANCE neither: END "No performance issue detected" ``` --- ## Decision Tree: Write Performance (S1 -> W nodes) ``` NODE: S1_BALANCE action: Fetch per-instance Write-RPC QPS and P99 latency metrics: write_qps_by_inst: > sum(rate(tikv_grpc_msg_duration_seconds_count{ tidb_cluster_id="${CLUSTER_ID}", type=~"kv_prewrite|kv_commit|kv_pessimistic_lock"}[2m])) by (instance) write_p99_by_inst: > histogram_quantile(0.99, sum(rate( tikv_grpc_msg_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type=~"kv_prewrite|kv_commit|kv_pessimistic_lock" }[2m])) by (instance, le)) condition: Is one instance latency >2x the median of all instances? branches: YES (outlier): DIAGNOSIS "Single-instance problem (bad disk, hot region)" ACTION "Investigate that instance only. Check disk, CPU, regions." Set OUTLIER_INSTANCE = "" NOTE "All subsequent write subtree queries (S2 through W nodes) MUST add instance='${OUTLIER_INSTANCE}' filter to avoid signal dilution across healthy instances. Clear OUTLIER_INSTANCE when entering read tree (R0_READ_BALANCE has its own per-instance check)." Also continue -> S2_WRITE_STALL for that instance NO (uniform): Set OUTLIER_INSTANCE = "" (no filter) -> S2_WRITE_STALL ``` ``` NODE: S2_WRITE_STALL action: Fetch Write Stall metrics from KVDB and RaftDB instance_filter: If OUTLIER_INSTANCE is set, add instance="${OUTLIER_INSTANCE}" to ALL metrics in this node and downstream write subtree nodes. metrics: kvdb_stall: > avg(tikv_engine_write_stall{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="write_stall_percentile99"}) raftdb_stall: > avg(tikv_engine_write_stall{ tidb_cluster_id="${CLUSTER_ID}", db="raft", type="write_stall_percentile99"}) condition: Write Stall sustained > 0 for >= 3 consecutive samples at step interval? (Single non-zero samples are noise on cloud disks. Require persistence.) branches: YES (sustained stall): -> S2A_STALL_CAUSE NO: -> S3_COMPACTION_FLOW ``` ``` NODE: S2A_STALL_CAUSE action: Check pending compaction bytes to identify stall cause metrics: pending_bytes: > sum(tikv_engine_pending_compaction_bytes{ tidb_cluster_id="${CLUSTER_ID}", db="kv"}) pending_by_inst: > sum(tikv_engine_pending_compaction_bytes{ tidb_cluster_id="${CLUSTER_ID}", db="kv"}) by (instance) condition: Is pending_bytes > 16GB (1/4 of soft limit 64GB)? branches: YES (high pending): DIAGNOSIS "Compaction cannot keep up" confidence: HIGH (threshold-based: pending bytes > 16GB) CAUSES "Write pressure too high, ingest SST from PD, IO limiter too low" CONFIG tikv.rocksdb.max-background-jobs=8, max-sub-compactions=3, remove IO limiter if too low, write-buffer-size=128MB END NO (low pending): DIAGNOSIS "Flush bottleneck (memtable->SST too slow) or CPU overloaded" confidence: MEDIUM (elimination-based: stall present but low pending) CONFIG tikv.rocksdb.max-background-flushes (increase beyond default 2), reduce client write pressure if CPU>80% END ``` ``` NODE: S3_COMPACTION_FLOW action: Fetch compaction IO flow, correlate with Write-RPC QPS metrics: write_rpc_qps: > sum(rate(tikv_grpc_msg_duration_seconds_count{ tidb_cluster_id="${CLUSTER_ID}", type=~"kv_prewrite|kv_commit|kv_pessimistic_lock"}[2m])) compaction_flow: > sum(rate(tikv_engine_compaction_flow_bytes{ tidb_cluster_id="${CLUSTER_ID}", db="kv"}[2m])) spark_check: write_rpc_qps vs compaction_flow (inverse: high flow = low QPS) steady_state_threshold: compaction_flow > 200MB/s (sustained high compaction) branches: SPARK: DIAGNOSIS "Compaction IO is causing write performance fluctuation" confidence: MEDIUM (spark-based; confirm with disk IOPS spike during compaction) CONFIG tikv.rocksdb.rate-bytes-per-sec (set IO limiter to smooth flow, but not too low -> pending bytes -> Write Stall), write-buffer-size=128MB (bigger reduces write amplification) Also check -> S4_ASYNC_WRITE for deeper causes INCONCLUSIVE or WEAK: Note compaction as possible contributor. Continue -> S4_ASYNC_WRITE. Include compaction_flow mean value in report for operator review. NO_SPARK: -> S4_ASYNC_WRITE ``` ``` NODE: S4_ASYNC_WRITE [KEY BRANCH POINT] action: Fetch Storage async-write P99, correlate with Write-RPC QPS metrics: async_write_p99: > histogram_quantile(0.99, sum(rate( tikv_storage_engine_async_request_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type="write" }[2m])) by (le)) spark_check: write_rpc_qps (from S3) vs async_write_p99 steady_state_threshold: async_write_p99 > 50ms (sustained high async write latency) note: Async-write sits between Scheduler(L1) and RaftStore(L3). This determines whether the problem is ABOVE or BELOW storage. branches: NO_SPARK (problem ABOVE storage): -> W_ABOVE SPARK (problem BELOW storage): -> W_BELOW INCONCLUSIVE or WEAK: Check BOTH W_ABOVE and W_BELOW. Report findings from both branches. ``` ``` NODE: W_ABOVE [Problem in Scheduler / Latch / MVCC] action: Run parallel checks on latch, scheduler threads, PD scheduling parallel_checks: -> W_LATCH -> W_SCHEDULER -> W_PD_SCHEDULING all_negative: If ALL sub-nodes return NO: DIAGNOSIS "UNDIAGNOSED at Scheduler/Latch/MVCC layer (L1-L2)." confidence: LOW (elimination-based fallback) ACTION "Check client-side TXN patterns, MVCC version accumulation, or cross-link to read tree (R1_READ_TYPE) for MVCC-read-during-write." END ``` ``` NODE: W_LATCH action: Fetch latch wait duration and lock-manager metrics metrics: latch_p99: > histogram_quantile(0.99, sum(rate( tikv_scheduler_latch_wait_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) by (le)) lock_wait_count: > sum(rate(tikv_lock_manager_waiter_lifetime_duration_count{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) conflict_count: > sum(rate(tikv_scheduler_too_busy_total{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) pessimistic_lock_wait: > histogram_quantile(0.99, sum(rate( tikv_storage_engine_async_request_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type="acquire_pessimistic_lock" }[2m])) by (le)) caveat: This metric incorrectly includes scheduler wait-for-thread time in some TiKV versions. note: lock_wait_count and conflict_count may be absent on older TiKV versions. See sub_condition below for absent-metric handling. condition: Is avg(latch_p99) > 10ms? branches: YES (high latch): sub_condition: Determine TXN mode and conflict pattern: - If lock_wait_count > 0 AND pessimistic_lock_wait > 5ms -> pessimistic TXN with conflicts - If conflict_count > 0 AND lock_wait_count == 0 -> optimistic TXN with conflicts - If both == 0 AND metrics are present -> no conflicts detected (confirmed) - If either metric is absent (query returns empty) -> conflict status indeterminate branches: optimistic TXN with conflicts: DIAGNOSIS "Client write conflicts cause latch waiting" confidence: HIGH (threshold-based: latch P99 > 10ms + optimistic conflicts) ACTION "Switch optimistic->pessimistic TXN to reduce rollbacks" END pessimistic TXN with conflicts: DIAGNOSIS "Client write conflicts on pessimistic TXN" confidence: HIGH (threshold-based: latch P99 > 10ms + pessimistic conflicts) ACTION "Conflicts are expected in pessimistic mode. Check if lock wait timeout is too high, or reduce hotspot writes." NOTE "Do NOT switch to optimistic -- would make it worse." END no conflicts (confirmed): DIAGNOSIS "Insufficient scheduler threads causing artificial latch" confidence: MEDIUM (elimination-based: high latch without conflicts) CONFIG tikv.storage.scheduler-worker-pool-size (increase from 4) END conflict status indeterminate: DIAGNOSIS "Latch cause indeterminate (conflict metrics unavailable)" confidence: LOW (INCONCLUSIVE: cannot distinguish conflict-based vs thread-starvation latch without lock_wait_count/conflict_count) ACTION "Check TiKV version. On 5.x+, these metrics should exist. If absent, verify metric scrape config. Manual inspection of TiKV logs for 'txn_lock' or 'scheduler_too_busy' recommended." END NO: "Latch is not the cause" ``` ``` NODE: W_SCHEDULER action: Fetch scheduler command duration, correlate with write QPS metrics: sched_cmd_p99: > histogram_quantile(0.99, sum(rate( tikv_scheduler_command_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) by (le)) spark_check: write_rpc_qps vs sched_cmd_p99 steady_state_threshold: sched_cmd_p99 > 20ms (sustained high scheduler latency) branches: SPARK: DIAGNOSIS "Scheduler is the bottleneck, threads are starved" confidence: MEDIUM (spark-based; confirm with scheduler CPU utilization) CONFIG tikv.storage.scheduler-worker-pool-size (increase, but too big wastes CPU) END NO_SPARK: DIAGNOSIS "Problem may be MVCC read during write" next: -> R1_READ_TYPE (cross-link to Read tree) ``` ``` NODE: W_PD_SCHEDULING action: Fetch PD scheduling events, correlate with write QPS metrics: pd_schedule: > sum(rate(pd_schedule_operators_count{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) by (type) note: This is a PD-originated metric (registered in pd/pkg/schedule/operator/metrics.go). It may NOT be available via TiKV's APP_UUID endpoint in O11Y. If unavailable, fall through to NO_SPARK (fail-open). The original Fast Tune dashboard (tikv_fast_tune.json) uses this metric assuming PD scrape is co-located. policy: fail-open. If metric returns empty, treat as NO_SPARK. spark_check: write_rpc_qps vs pd_schedule steady_state_threshold: none (no meaningful absolute threshold for scheduling rate) branches: SPARK: DIAGNOSIS "PD region scheduling is disrupting write performance" confidence: MEDIUM (spark-based; confirm with pd-ctl operator show) ACTION "Slow down balancing with pd-ctl" NOTE "Common after adding/removing TiKV instances" END NO_SPARK: "PD scheduling is not the cause" ``` ``` NODE: W_BELOW [Problem in RaftStore / RocksDB / Disk] action: Check RocksDB write latency next: -> W_ROCKSDB_WRITE ``` ``` NODE: W_ROCKSDB_WRITE action: Fetch RocksDB write latency (KVDB + RaftDB), correlate with write QPS metrics: kvdb_write_p99: > avg(tikv_engine_write_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="write_percentile99"}) raftdb_write_p99: > avg(tikv_engine_write_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="raft", type="write_percentile99"}) kvdb_write_max: > avg(tikv_engine_write_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="write_max"}) note: These are GaugeVec with pre-computed percentiles (updated on RocksDB stats dump interval). Check BOTH p99 and max -- cloud disks (AWS gp2) may show stable p99 but erratic max. spark_check: write_rpc_qps vs kvdb_write_p99 (also check kvdb_write_max) steady_state_threshold: kvdb_write_p99 > 100000us (100ms sustained high RocksDB write) branches: SPARK (RocksDB write slow): -> W_DISK_GROUP (drill into disk causes) NO_SPARK (RocksDB OK): -> W_RAFTSTORE (check thread pool) INCONCLUSIVE or WEAK: Check BOTH W_DISK_GROUP and W_RAFTSTORE. Report findings from both branches. ``` ``` NODE: W_DISK_GROUP action: Run parallel checks on disk-level causes parallel_checks: -> W_WAL_SYNC -> W_BATCH_SIZE -> W_MUTEX -> W_FRONTEND_FLOW all_negative: If ALL sub-nodes return NO: DIAGNOSIS "UNDIAGNOSED at disk/RocksDB layer (L4-L5). RocksDB write is slow but no specific disk-level cause identified." confidence: LOW (elimination-based fallback) ACTION "Check device-level metrics (node_disk_* via K8S_UUID), cloud disk burst credit exhaustion, or IO scheduler contention." END ``` ``` NODE: W_WAL_SYNC action: Fetch RaftDB WAL fsync latency cross_link: "Write-side disk check (W_WAL_SYNC) and read-side disk check (R5C) measure different I/O paths. W_WAL_SYNC checks RaftDB sequential WAL fsync; R5C checks KVDB random read latency. Different verdicts are expected and not contradictory." metrics: wal_p99: > avg(tikv_engine_wal_file_sync_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="raft", type="wal_file_sync_percentile99"}) wal_max: > avg(tikv_engine_wal_file_sync_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="raft", type="wal_file_sync_max"}) spark_check: write_rpc_qps vs wal_p99 steady_state_threshold: wal_p99 > 10000us (10ms sustained high WAL fsync) branches: SPARK: If TIKV_VERSION == "5.x": DIAGNOSIS "Disk fsync is slow (common on non-enterprise disks)" confidence: MEDIUM (spark-based; confirm with node_disk_write_time_seconds) ACTION "Use better disk. Or use fsync-ctrl, increase batch size." If TIKV_VERSION == "7.x": DIAGNOSIS "Raft-engine WAL write+sync is slow (metric includes serialization + fsync)" confidence: MEDIUM (spark-based; confirm with node_disk_write_time_seconds) ACTION "Check disk IOPS. If disk is fast, investigate raft-engine batch size or entry size. The raft-engine WAL metric is composite -- high values may reflect batch assembly overhead rather than disk latency alone." END NO_SPARK: "WAL sync is not the bottleneck" ``` ``` NODE: W_BATCH_SIZE action: Fetch RocksDB WAL write size (proxy for batch size) metrics: wal_size_p99: > avg(tikv_engine_write_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="write_wal_size_percentile99"}) note: This is WAL write size P99, used as proxy for batch size. condition: Compute linear slope of wal_size_p99 over 3 consecutive 5-minute windows (20 samples each at step=15s). If slope > 0 in all 3 windows (monotonically increasing across 15 minutes), the batch size is growing. branches: YES (growing): DIAGNOSIS "Loop speed degradation cascade" confidence: MEDIUM (trend-based; confirm with disk IOPS increasing concurrently) DETAIL "Disk slows -> loop slows -> messages queue -> batch grows -> loop slows more" ACTION "Find what slowed disk initially (compaction, IO contention)" CONFIG tikv.raftstore.store-max-batch-size=256 (max limit) END NO: "Batch size is stable, not the cause" ``` ``` NODE: W_MUTEX action: Fetch RocksDB perf context mutex wait metrics: mutex_nanos: > avg(tikv_engine_write_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="write_db_mutex_wait_nanos"}) prerequisite: Only investigate if RocksDB write slow BUT disk latency normal condition: Is avg(mutex_nanos) > 100000 (100us)? If yes, correlate with write latency using detect_spark(). 100us of mutex wait per write operation is significant for SSD-backed storage. branches: YES: DIAGNOSIS "Something holds RocksDB mutex too long" confidence: MEDIUM (correlation-based; confirm with specific mutex holder) sub_checks: - "PD scheduling (ingest SST holds mutex)" -> "Disable/reduce scheduling with pd-ctl" - "GC worker (compaction filter holds mutex)" -> "Use compaction filter, check GC" END NO: "Mutex is not the cause" ``` ``` NODE: W_FRONTEND_FLOW action: Check if workload changes are causing the performance shift metrics: frontend_write: > sum(rate(tikv_engine_flow_bytes{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="bytes_written"}[2m])) frontend_read: > sum(rate(tikv_engine_flow_bytes{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="bytes_read"}[2m])) spark_check: write_rpc_qps vs frontend_write (or frontend_read) steady_state_threshold: frontend_write > 100MB/s (sustained high write flow) branches: SPARK: DIAGNOSIS "Client-side workload change affecting performance" confidence: MEDIUM (spark-based; confirm with application-side QPS metrics) branches: high write flow: "Client write pressure increased. Reduce client-side load." high read flow: "Read workload affecting write. Check GC, client queries." END NO_SPARK: DIAGNOSIS "Possible disk hardware limit (not confirmed by frontend flow)" confidence: LOW (fallback/elimination; requires manual device-level verification) ACTION "Verify with IOPS and throughput BW metrics. Check cloud disk quotas." NOTE "This is a weak conclusion. If PRE_INFRA_CHECK passed, disk is the most likely remaining cause, but confirm with device-level metrics (node_disk_* via K8S_UUID) before recommending disk upgrade." END ``` ``` NODE: W_RAFTSTORE action: Check RaftStore thread pool sizing metrics: propose_wait_p99: > histogram_quantile(0.99, sum(rate( tikv_raftstore_request_wait_time_duration_secs_bucket{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) by (le)) condition: Is propose_wait >> RocksDB write latency (from W_ROCKSDB_WRITE)? caveat_storewriter: > TiKV 6.x+ separates StoreWriter from RaftStore. If store-io-pool-size > 0, the write path goes through StoreWriter threads. In that case, propose_wait may appear normal even if StoreWriter is the bottleneck. Check tikv_raftstore_store_write_task_wait_duration_seconds if available. caveat_apply_loop: > This node checks propose_wait (Store loop). The Apply loop (KVDB write) is a DISTINCT bottleneck. If propose_wait is fine but async_write is slow, check tikv_raftstore_apply_wait_time_duration_secs if available. Apply loop bottleneck manifests as high KVDB write latency with normal RaftDB write. branches: YES (long wait, short loop): DIAGNOSIS "RaftStore thread pool too small, messages queue up" confidence: MEDIUM (threshold-based; confirm with store-pool CPU saturation) CONFIG tikv.raftstore.store-pool-size=2 (increase carefully), tikv.raftstore.apply-pool-size=2 (increase carefully) WARNING "Too big = small IO batches = LOWER performance" END NO (proportional): DIAGNOSIS "RaftStore pool OK. Check read path for MVCC read during write." next: -> R1_READ_TYPE ``` --- ## Decision Tree: Read Performance (R nodes) ``` NODE: R0_READ_BALANCE action: Fetch per-instance Read-RPC P99 latency (mirrors S1_BALANCE for writes) metrics: read_p99_by_inst: > histogram_quantile(0.99, sum(rate( tikv_grpc_msg_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type=~"kv_get|kv_batch_get|kv_batch_get_command|kv_scan|coprocessor" }[2m])) by (instance, le)) condition: Is one instance read latency >2x the median of all instances? branches: YES (outlier): DIAGNOSIS "Single-instance read problem (bad disk, hot region, cold cache)" confidence: HIGH (threshold-based: >2x median) ACTION "Investigate that instance only. Check disk latency, block cache, region distribution. May indicate asymmetric compaction or disk degradation on one node." Also continue -> R1_READ_TYPE for that instance NO (uniform): -> R1_READ_TYPE ``` ``` NODE: R1_READ_TYPE action: Fetch Coprocessor P99 and Get-RPC P99 separately metrics: copr_p99: > histogram_quantile(0.99, sum(rate( tikv_grpc_msg_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type=~"coprocessor"}[2m])) by (le)) get_p99: > histogram_quantile(0.99, sum(rate( tikv_grpc_msg_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type=~"kv_batch_get|kv_batch_get_command|kv_get|kv_scan" }[2m])) by (le)) condition: Which P99 exceeds 50ms (same threshold as ROOT)? branches: Coprocessor elevated (copr_p99 > 50ms): -> R2_COPR Get elevated (get_p99 > 50ms): -> R3_GET Both elevated: -> R2_COPR first, then R3_GET Neither elevated: END "No read performance issue detected at RPC level. If called from write tree (W_SCHEDULER/W_RAFTSTORE), the MVCC-read-during-write hypothesis is not confirmed." ``` ``` NODE: R2_COPR action: Check coprocessor thread pool, scan volume, and KVDB read parallel_checks: -> R2A_COPR_THREADS -> R2B_SCAN_COUNT -> R5_KVDB_READ all_negative: If ALL sub-nodes return NO: DIAGNOSIS "UNDIAGNOSED at coprocessor layer (L2). Coprocessor is slow but no thread pool, scan volume, or KVDB issue found." confidence: LOW (elimination-based fallback) ACTION "Check for expression evaluation bottleneck (complex WHERE clauses, UDFs) or uncommon coprocessor request types." END ``` ``` NODE: R2A_COPR_THREADS action: Fetch coprocessor queue wait, handle duration, and CPU usage metrics: copr_wait_p99: > histogram_quantile(0.99, sum(rate( tikv_coprocessor_request_wait_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type="all"}[2m])) by (le)) copr_handle_p99: > histogram_quantile(0.99, sum(rate( tikv_coprocessor_request_handle_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type="all"}[2m])) by (le)) tikv_cpu_usage: > avg(rate(process_cpu_seconds_total{ tidb_cluster_id="${CLUSTER_ID}", job=~".*tikv.*"}[2m])) tikv_container_cpu: > avg(rate(container_cpu_usage_seconds_total{ namespace="tidb${CLUSTER_ID}", pod=~".*tikv.*", container="tikv"}[2m])) note: tikv_cpu_usage uses APP_UUID endpoint. tikv_container_cpu uses K8S_UUID endpoint. Use whichever is available; prefer tikv_container_cpu for accuracy. condition: Is copr_wait_p99 > 2x copr_handle_p99? (concrete: wait/handle ratio > 2) branches: YES (long queue): sub_condition: Is tikv_cpu_usage (or tikv_container_cpu) > 0.8 (80%)? branches: YES: DIAGNOSIS "CPU overloaded. Cannot add threads. Reduce workload or scale." confidence: HIGH (threshold-based: CPU > 80% + queue wait ratio > 2x) NO: DIAGNOSIS "Coprocessor pool undersized. Increase thread count." confidence: HIGH (threshold-based: queue wait > 2x handle time + CPU available) END NO: "Threads sufficient. Execution itself is slow." -> check R2B_SCAN_COUNT and R5_KVDB_READ ``` ``` NODE: R2B_SCAN_COUNT action: Fetch scan volume metrics: scan_keys: > sum(rate(tikv_coprocessor_scan_keys_sum{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) copr_req_count: > sum(rate(tikv_coprocessor_request_duration_seconds_count{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) copr_by_type: > sum(rate(tikv_coprocessor_request_duration_seconds_count{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) by (req) condition: Compute keys_per_request = scan_keys / copr_req_count. - keys_per_request > 10000: excessive scan volume (MVCC bloat or full-table scan) - Determine cause by checking copr_by_type breakdown: If copr_by_type{req="select"} / copr_req_count > 0.8 AND keys_per_request > 10000: -> high request count with broad scans (client-side issue) If copr_by_type{req="select"} / copr_req_count < 0.8 AND keys_per_request > 10000: -> low request count but high keys per request (MVCC versions/tombstones) branches: YES (keys_per_request > 10000): branches: high req ratio (select > 80%): DIAGNOSIS "Client-side full table scans or heavy queries" confidence: HIGH (threshold-based: >10k keys/req + select-dominated) ACTION "Optimize queries. Add indexes." low req ratio (select < 80%): DIAGNOSIS "Too many MVCC versions or tombstones" confidence: HIGH (threshold-based: >10k keys/req + non-select) next -> R6_GC END NO (keys_per_request <= 10000): "Scanning is not the cause" ``` ``` NODE: R3_GET action: Check lease-read rate, async-snapshot, and KVDB read parallel_checks: -> R3A_LEASE_READ -> R3B_ASYNC_SNAP -> R5_KVDB_READ all_negative: If ALL sub-nodes return NO: DIAGNOSIS "UNDIAGNOSED at Get/Scan layer. Lease-read, snapshot, and KVDB all appear healthy." confidence: LOW (elimination-based fallback) ACTION "Check for client-side timeout misconfiguration, network latency between TiDB and TiKV, or uncommon read patterns (stale read, follower read)." END ``` ``` NODE: R3A_LEASE_READ action: Fetch local read count and read-index propose count metrics: local_read: > sum(rate(tikv_raftstore_local_read_executed_requests{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) read_index: > sum(rate(tikv_raftstore_read_index_propose_total{ tidb_cluster_id="${CLUSTER_ID}"}[2m])) condition: Is local_read / (local_read + read_index) < 0.9? branches: YES (low lease-read): DIAGNOSIS "Many reads fall out-of-lease (needs Raft write first = slow)" confidence: HIGH (threshold-based: lease-read ratio < 90%) CAUSES "Leader transfers, region balancing" ACTION "Adjust PD scheduling via pd-ctl. Reduce leader transfer rate." END NO: "Lease-read rate is fine" ``` ``` NODE: R3B_ASYNC_SNAP action: Fetch async-snapshot duration metrics: snap_p99: > histogram_quantile(0.99, sum(rate( tikv_storage_engine_async_request_duration_seconds_bucket{ tidb_cluster_id="${CLUSTER_ID}", type="snapshot" }[2m])) by (le)) condition: Is snap_p99 > 10ms with jitters? branches: YES: DIAGNOSIS "Snapshot acquisition slow (out-of-lease regions)" confidence: HIGH (threshold-based: snap P99 > 10ms with jitter) ACTION "Check PD scheduling. Adjust with pd-ctl." END NO: "Snapshot is fast. Problem is in KVDB." -> R5_KVDB_READ ``` ``` NODE: R5_KVDB_READ action: Check block cache, SST files, and disk read parallel_checks: -> R5A_CACHE -> R5B_SST -> R5C_DISK_READ all_negative: If ALL sub-nodes return NO: DIAGNOSIS "UNDIAGNOSED at KVDB layer. No block cache, SST, or disk issue found." confidence: LOW (elimination-based fallback; requires manual investigation) ACTION "Check OS-level metrics (memory pages, context switching, CPU load). Consider client-side workload changes or uncommon failure modes." END ``` ``` NODE: R5A_CACHE action: Fetch block cache hit rate metrics: hit_rate: > sum(rate(tikv_engine_cache_efficiency{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="block_cache_hit"}[2m])) / (sum(rate(tikv_engine_cache_efficiency{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="block_cache_hit"}[2m])) + sum(rate(tikv_engine_cache_efficiency{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="block_cache_miss"}[2m]))) note: Cache hit rate changes often follow compaction cycles. Even with a spark, cache miss is often a RESULT not the root cause. condition: Is hit_rate < 0.95? branches: YES: DIAGNOSIS "Block cache misses -> SST reads -> disk IO -> slow reads" confidence: HIGH (threshold-based: cache hit rate < 95%) CONFIG "Increase block-cache-size if memory allows. Also check compaction (R5B)." END NO: "Cache is not the cause" ``` ``` NODE: R5B_SST action: Fetch SST file count per level (compaction health indicator) metrics: sst_files: > tikv_engine_num_files_at_level{ tidb_cluster_id="${CLUSTER_ID}", db="kv"} IMPORTANT: This is a GAUGE. Do NOT use rate(). Query raw values. condition: Check SST file counts by level. Thresholds (RocksDB defaults): - L0 > 20 files: compaction slowdown trigger (level0-slowdown-writes-trigger=20) - L0 > 36 files: compaction stop trigger (level0-stop-writes-trigger=36) - Total files across L5+L6 > 500: high read amplification Evaluate: max over window of any per-instance L0 file count. branches: YES (L0 > 20 or total L5+L6 > 500): DIAGNOSIS "Compaction behind. Too many SSTs = high read amplification." confidence: HIGH (threshold-based: L0 > 20 is RocksDB's own slowdown trigger) CONFIG tikv.rocksdb.max-background-jobs=8, max-sub-compactions=3 END NO: "Compaction is healthy" ``` ``` NODE: R5C_DISK_READ action: Check disk read latency cross_link: "Read-side disk check (R5C) and write-side disk check (W_WAL_SYNC) measure different I/O paths and different RocksDB instances. R5C checks KVDB random read latency; W_WAL_SYNC checks RaftDB sequential WAL fsync. They can legitimately produce different verdicts (e.g., disk read OK but fsync slow, or vice versa)." metrics: read_p99: > avg(tikv_engine_read_micro_seconds{ tidb_cluster_id="${CLUSTER_ID}", db="kv", type="read_percentile99"}) note: GaugeVec with pre-computed percentiles. For device-level metrics, use node_disk_read_time_seconds_total via K8S_UUID if available. condition: Is avg(read_p99) > 500us (0.5ms)? Correlate with read RPC latency using detect_spark(). 500us is a conservative threshold for SSD. branches: YES: DIAGNOSIS "Disk read is slow" confidence: MEDIUM (correlation-based; confirm with node_disk_read_time_seconds) ACTION "Check IOPS, throughput BW. Upgrade disk." END NO: "Disk is not the bottleneck" ``` ``` NODE: R6_GC action: Check MVCC GC status metrics: gc_keys: > sum(tikv_gcworker_gc_keys{ tidb_cluster_id="${CLUSTER_ID}"}) by (type) mvcc_versions: > sum(tikv_engine_estimate_num_keys{ tidb_cluster_id="${CLUSTER_ID}", db="kv"}) gc_safe_point_lag: > time() - max(tidb_gc_safe_point{ tidb_cluster_id="${CLUSTER_ID}"}) typical_pattern: 1. QPS drops as MVCC versions accumulate 2. GC starts: versions become tombstones, STILL dragging QPS 3. Compaction clears tombstones, QPS recovers condition: MVCC version accumulation detected if ANY of: - gc_safe_point_lag > 600s (GC safe point >10min behind) - mvcc_versions slope > 0 over investigation window (keys growing) - gc_keys{type="delete"} / gc_keys{type="total"} > 0.3 (high tombstone ratio) Note: gc_safe_point_lag metric may not exist; use fail-open. branches: YES: DIAGNOSIS "MVCC GC not keeping up. Excessive versions slow reads AND writes." confidence: HIGH (threshold-based: MVCC version accumulation) ACTION "Lower GC life time. Use compaction filter. Increase memtable count." END NO: DIAGNOSIS "GC is healthy. Problem may be client-side workload." confidence: LOW (elimination fallback; all TiKV-side checks exhausted) END ``` --- ## Spark Detection Function Used at nodes marked with `spark_check`. Uses **first-differenced Pearson** to remove shared trends and reduce autocorrelation inflation from overlapping rate windows. Aligns series by timestamp (not array index). Handles multi-series, constant series, and minimum sample size. **Why first-differencing:** `rate(...[2m])` at `step=15s` creates 87.5% sample overlap between adjacent points, inflating raw Pearson |r|. The same codebase abandoned raw correlation in copr-worker-scaling (line 428: "old approach used correlation which was confounding, not causal"). First-differencing (`diff`) removes shared trends and substantially reduces autocorrelation. ```python import json, math def load_series(path): """Load time series from O11Y query_range response. Returns {timestamp: value} dict for alignment. Handles multi-series by summing all series at each timestamp. """ with open(path) as f: data = json.load(f) results = data.get('data', {}).get('result', []) if not results: return {} ts_map = {} for series in results: for ts, val in series['values']: ts_map[ts] = ts_map.get(ts, 0.0) + float(val) return ts_map def detect_spark(result_path, cause_path): """First-differenced Pearson correlation with timestamp alignment. Returns: {r_raw, r_diff, reliable, verdict, reason} r_raw: Pearson on raw series (for reference only) r_diff: Pearson on first-differenced series (primary signal) verdict: SPARK, NO_SPARK, WEAK, or INCONCLUSIVE """ result_map = load_series(result_path) cause_map = load_series(cause_path) # Align by timestamp (intersection of both series) common_ts = sorted(set(result_map) & set(cause_map)) n = len(common_ts) if n < 200: return {"r_raw": 0, "r_diff": 0, "reliable": False, "verdict": "INCONCLUSIVE", "reason": f"Only {n} aligned points (need >=200)"} result = [result_map[t] for t in common_ts] cause = [cause_map[t] for t in common_ts] def var(x): m = sum(x) / len(x) return sum((xi - m) ** 2 for xi in x) / len(x) def cv(x): """Coefficient of variation. Returns 0 if mean is ~0.""" m = sum(x) / len(x) if abs(m) < 1e-10: return 0.0 return math.sqrt(var(x)) / abs(m) # Steady-state detection: if both series are nearly constant (CV < 5%), # correlation is meaningless. Fall back to absolute threshold comparison. # This catches the false-negative case where constant high compaction # suppresses QPS at a constant low level (r_diff ~ 0). result_cv = cv(result) cause_cv = cv(cause) if result_cv < 0.05 and cause_cv < 0.05: return {"r_raw": 0, "r_diff": 0, "reliable": True, "verdict": "STEADY_STATE", "reason": f"Both series near-constant (result CV={result_cv:.3f}, " f"cause CV={cause_cv:.3f}). Correlation not applicable. " f"Use absolute threshold comparison instead.", "result_mean": sum(result) / len(result), "cause_mean": sum(cause) / len(cause)} def pearson(x, y): n = len(x) if n < 3 or var(x) < 1e-10 or var(y) < 1e-10: return 0.0 mx, my = sum(x) / n, sum(y) / n num = sum((a - mx) * (b - my) for a, b in zip(x, y)) dx = math.sqrt(sum((a - mx) ** 2 for a in x)) dy = math.sqrt(sum((b - my) ** 2 for b in y)) return num / (dx * dy) if dx * dy > 0 else 0.0 # Raw Pearson (for reference, NOT used for verdict) r_raw = pearson(result, cause) # First-differenced Pearson (primary signal) d_result = [result[i] - result[i-1] for i in range(1, len(result))] d_cause = [cause[i] - cause[i-1] for i in range(1, len(cause))] r_diff = pearson(d_result, d_cause) if var(d_result) < 1e-10 or var(d_cause) < 1e-10: return {"r_raw": r_raw, "r_diff": 0, "reliable": False, "verdict": "INCONCLUSIVE", "reason": "Constant series after differencing (no variation)"} # Verdict based on first-differenced correlation if abs(r_diff) > 0.7: return {"r_raw": r_raw, "r_diff": r_diff, "reliable": True, "verdict": "SPARK", "reason": f"Strong diff-correlation r_diff={r_diff:.3f} (raw r={r_raw:.3f})"} elif abs(r_diff) > 0.3: return {"r_raw": r_raw, "r_diff": r_diff, "reliable": True, "verdict": "WEAK", "reason": f"Weak r_diff={r_diff:.3f} (raw r={r_raw:.3f}), investigate"} else: return {"r_raw": r_raw, "r_diff": r_diff, "reliable": True, "verdict": "NO_SPARK", "reason": f"No diff-correlation r_diff={r_diff:.3f} (raw r={r_raw:.3f})"} ``` --- ## Config Knobs Reference | Config | Default | Tune When (Node) | |--------|---------|------------------| | `tikv.storage.scheduler-worker-pool-size` | 4 | High latch without conflicts (W_LATCH, W_SCHEDULER) | | `tikv.raftstore.store-pool-size` | 2 | Store loop wait >> loop duration (W_RAFTSTORE) | | `tikv.raftstore.apply-pool-size` | 2 | Apply loop wait >> loop duration (W_RAFTSTORE) | | `tikv.rocksdb.max-background-jobs` | 8 | Write Stall or high pending bytes (S2A, R5B) | | `tikv.rocksdb.max-sub-compactions` | 3 | Write Stall or high pending bytes (S2A, R5B) | | `tikv.rocksdb.max-background-flushes` | 2 | Flush too slow (S2A) | | `tikv.rocksdb.rate-bytes-per-sec` | "" | Compaction IO spikes (S3) | | `tikv.rocksdb.defaultcf.write-buffer-size` | 128MB | Reduce write amplification (S2A, S3) | | `tikv.rocksdb.defaultcf.soft-pending-compaction-bytes-limit` | 64G | Write Stall threshold (S2A) | | `tikv.raftstore.store-max-batch-size` | 256 | Loop degradation cascade (W_BATCH_SIZE) | | `tikv.raftstore.apply-max-batch-size` | 256 | Loop degradation cascade (W_BATCH_SIZE) | | `tikv.raftdb.max-background-jobs` | 4 | RaftDB compaction | | `tikv.raftdb.max-sub-compactions` | 2 | RaftDB compaction | ## Execution Guide ### Step 1: Setup ```bash source .claude/skills/o11y-metrics-api/o11y-token.sh AUTH0_TOKEN=$(get_o11y_token) # Resolve APP_UUID, DS_ENDPOINT via o11y-metrics-api skill CLUSTER_ID="" ``` ### Step 2: Define Time Range ```bash START_TIME=1707696000 # epoch seconds, incident window start END_TIME=1707703200 # 2-4 hours recommended STEP="15s" # max fidelity for jitter detection ``` **O11Y point cap:** `query_range` returns at most 11,000 points per series. At `step=15s`, this limits the window to ~46 hours. At `step=60s`, ~7.6 days. If your window exceeds the cap, O11Y silently truncates the response from the end. For investigation windows >24h, use `step=60s` or `step=300s`. ### Step 3: Execute Decision Tree Start at `PRE_INFRA_CHECK`. At each node: 1. Fetch listed metrics via `query_range` 2. Evaluate condition (threshold or spark detection) 3. Follow matching branch 4. At `parallel_checks`, execute ALL sub-nodes 5. Stop at terminal nodes (DIAGNOSIS + END) ### Step 4: Report ``` 1. Symptom: 2. Tree path: PRE_INFRA_CHECK -> VERSION_DETECT -> ROOT -> S1 -> S2 -> S4 -> W_BELOW -> W_ROCKSDB -> W_DISK_GROUP -> W_WAL 3. Flags: 4. Root cause: 5. Evidence: 6. Recommendation: 7. Caveats: ``` ## Known Caveats | Issue | Nodes | Description | |-------|-------|-------------| | TiKV 7.x metric adapter | W_ROCKSDB_WRITE, W_WAL_SYNC, W_BATCH_SIZE, S2_WRITE_STALL | VERSION_DETECT auto-swaps metric names for raft-engine. W_BATCH_SIZE is skipped on 7.x (no equivalent gauge). S2_WRITE_STALL raftdb_stall is skipped on 7.x (raft-engine has no RocksDB write stall mechanism). | | Raft-engine WAL semantic gap | W_WAL_SYNC | On 7.x, `tikv_raft_engine_write_duration_seconds{type="wal"}` includes serialization + fsync, not just fsync. Spark correlation remains directionally valid but diagnosis text is version-conditional to avoid misattributing batch assembly overhead to disk latency. | | O11Y point cap | All nodes | `query_range` returns max 11,000 points per series. At step=15s, max window ~46h. Silent truncation if exceeded. | | Latch metric bug | W_LATCH | Includes scheduler wait time in some versions. Conflict metrics (lock_wait_count, conflict_count) may be absent on older TiKV; handled via "conflict status indeterminate" branch (LOW, INCONCLUSIVE). | | Pre-computed gauges | W_ROCKSDB_WRITE, R5C | `tikv_engine_*_micro_seconds` updates on dump interval | | Cloud disk behavior | W_ROCKSDB_WRITE | AWS gp2: stable P99, erratic max at quota | | SST gauge metric | R5B_SST | `tikv_engine_num_files_at_level` is a gauge, NOT counter | | Rate window overlap | All spark nodes | `rate[2m]` at `step=15s` = 87.5% overlap. Mitigated by first-differencing in detect_spark(). | | GC metric delay | R6_GC | GC metrics may lag behind actual activity | | StoreWriter gap | W_RAFTSTORE | TiKV 6.x+ separates StoreWriter; propose_wait may miss this bottleneck | | Apply loop gap | W_RAFTSTORE | Tree checks Store loop but not Apply loop separately | | CDC metric absence | PRE_INFRA_CHECK #6 | `resolved_ts_lag` absent if TiCDC not enabled; fail-open by design. If present and elevated, uses FLAG_AND_CONTINUE (does not halt tree execution). | | Titan metric absence | PRE_INFRA_CHECK #7 | Titan metrics absent if Titan not enabled; fail-open by design. If present and elevated, uses FLAG_AND_CONTINUE (does not halt tree execution). | | Region count method | PRE_INFRA_CHECK #5 | Uses start/end comparison, not continuous monitoring; may miss brief storms | | Multi-tenant interference | All nodes | CLUSTER_ID scoping may not cleanly separate tenant impacts on shared TiKV | | Raft append log gap | W_RAFTSTORE | Tree checks WAL sync and RocksDB write but not raft log append duration as a separate node. Coverage is implicit (raft append = RaftDB write + WAL sync). | | PD metric endpoint | W_PD_SCHEDULING | `pd_schedule_operators_count` is PD-originated; may not be available via TiKV APP_UUID endpoint. Fail-open. | | Feedback loop limitation | All parallel_checks | Deepest-layer-wins rule assumes linear causality; feedback loops may obscure initiating cause. Mitigated by confidence-priority exception (HIGH shallow overrides MEDIUM deep). | | STEADY_STATE at spark nodes | All spark_check nodes | When both series are near-constant (CV < 5%), correlation is inapplicable. Uses steady_state_threshold per node to determine SPARK vs WEAK. Nodes without defined thresholds default to WEAK. | | Instance signal dilution | S1_BALANCE -> S2+ | When S1_BALANCE detects an outlier instance, OUTLIER_INSTANCE propagates through write subtree. Without this, cluster-wide avg() dilutes per-instance signals in 5+ node clusters. | | VERSION_DETECT probe failure | VERSION_DETECT | If raft-engine probe query errors (not empty, but actual failure), defaults to 5.x metrics as safe fallback. | ## Relationship to Other Skills | Skill | Relationship | |-------|-------------| | `o11y-metrics-api` | **Dependency** -- authentication and endpoint resolution | | `copr-worker-scaling` | **Complementary** -- HPA evaluation; this diagnoses underlying TiKV perf |