# Postmortem: TCOC-4964

> Workspace: `ws-1775119091093-df9c035dc`
> Date: 2026-04-15
> Reviewer: Claude Agent

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCOC-4964 |
| Alert Type | TiDB CPU高 |
| Cluster ID | 10518506906906971664 |
| Cluster Type | dedicated |
| TiDB Version | v8.5.4 |
| Time Window | 2026-03-17 18:00 BJ - 2026-03-18 09:30 BJ |

**Issue Description**: 北美saas02集群4个TiDB节点CPU持续打高，从正常16核升至54核峰值。

## Actual Root Cause

> Human-confirmed root cause from Jira comments.

**Root cause**: The **reuse chunk capability** triggered OOM when processing wide-column queries during NTDML-Delete operations on `dh_finance_delayed_task` table. This led to:
1. Increased memory allocation by Golang runtime (4.17 → 8.93 GB/s alloc rate, 2x increase)
2. More frequent GC cycles
3. High-frequency NTDML batch execution (74x increase: 2/s → 147/s statement count)

The NTDML-Delete operations, though deleting few actual rows (~10.9 rows/s), forced full `SELECT scan → transaction → execute → commit` cycles at 967 statements/s peak. Combined with the wide-column (56 columns) internal scans that prevented chunk reuse, this created sustained CPU pressure.

**Mitigation**: Implemented workaround to skip reuse chunk when total output column size exceeds threshold. CPU recovered after TiDB scale-out at BJ 09:30 on Mar 18.

## Agent Diagnosis Summary

> Agent's final diagnosis after interactive guidance.

**Agent correctly identified**:
1. **Primary cause**: NTDML-Delete on `dh_finance_delayed_task` table (74x increase, 2/s → 147/s)
2. **Memory pressure**: Alloc rate doubled from 4.17 to 8.93 GB/s
3. **CPU spike evidence**: 28 occurrences of CPU >30 cores on Mar 18 vs 0 on Mar 14-17
4. **Execution overhead**: High-frequency small transactions (967 statements/s peak) with low affected_rows
5. **Scale-out timing**: Correctly noted that CPU recovery at BJ 09:30 was due to node expansion, not issue resolution

**Agent initially confused**:
- Initially suspected the 56-column SELECT query on `dh_active_receive_logs` as the root cause
- Required multiple user interventions to check query timing and realize this query **decreased** during fault period (52 executions during fault vs 227 before fault)
- Eventually confirmed via Loki logs that NTDML operations targeted `dh_finance_delayed_task`, not `dh_active_receive_logs`

**Agent missed**:
- Did not discover the reuse chunk mechanism as the underlying cause (requires domain knowledge not present in metrics/logs)
- Did not explain *why* NTDML with low affected_rows could cause high CPU (chunk allocation issue)

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 4 | Correctly identified NTDML-Delete as primary cause, but missed the reuse chunk mechanism detail |
| Evidence Completeness | 5 | Queried metrics, TopSQL, SlowQuery, Loki logs; cross-referenced multiple sources |
| Causal Chain | 4 | Established NTDML → memory allocation → CPU spike chain, but mechanism incomplete |
| Actionability | 3 | Identified the problematic table/operation but no specific recommendation (workaround came from Jira) |
| Efficiency | 3 | 100+ tool calls with multiple false leads (56-col SELECT investigation took many rounds) |
| **Overall** | **3.8** | |

## Human Guidance Log

### Intervention 1: Scale-out timing clarification

- **Round**: #8 (approximately)
- **User said**: "正常时候是因为 tidb 节点做扩容了，缓解了这个问题。在北京时间3.18上早上9点半左右做的扩容。"
- **Agent's mistake**: Agent was treating CPU recovery at 09:30 as evidence that the issue self-resolved
- **Correct behavior**: Recognize that scale-out is a mitigation, not a resolution; the root cause still exists
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: REASONING

### Intervention 2: Query timing analysis

- **Round**: #9
- **User said**: "你再看下故障前后这个query是否有变化，如果有变化比如增加字段，是否是在故障开始的时间做的"
- **Agent's mistake**: Agent suspected the 56-column SELECT but didn't check whether it actually increased during fault period
- **Correct behavior**: Always verify timing correlation before claiming causation; query frequency should increase during fault, not decrease
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: REASONING / CROSS-REF

### Intervention 3: Confirmation of NTDML target table

- **Round**: #11
- **User said**: "是对这个表的删除么dh_active_receive_logs？有证据么"
- **Agent's mistake**: Agent conflated the wide-column SELECT target (`dh_active_receive_logs`) with the NTDML-Delete target
- **Correct behavior**: Query Loki logs to determine which table NTDML operations actually targeted
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: CROSS-REF

## Skill Deficiencies

### D1: Timing correlation validation required

- **Type**: REASONING / CROSS-REF
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent identifies a suspicious query/operation but doesn't verify timing correlation before declaring it as root cause
- **Expected behavior**: When a query or operation is suspected, immediately check:
  1. Does its frequency/duration **increase** during the fault period?
  2. Does it **decrease** when fault resolves (excluding external mitigations like scale-out)?
  3. Is the correlation consistent across multiple time windows?
- **Suggested fix**: Add a reasoning checkpoint: "Before concluding X is the root cause, verify X's timing pattern matches the fault timeline"
- **Priority**: P1-high
- **Status**: open

### D2: Separate query targets from operation targets

- **Type**: KNOWLEDGE / CROSS-REF
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent conflates the table being queried (e.g., in SlowQuery) with the table being modified by framework operations (e.g., NTDML-Delete target)
- **Expected behavior**: When NTDML or similar framework operations are involved:
  1. Check Loki logs for the actual target table (from log messages)
  2. SlowQuery/TopSQL shows *execution queries*, not necessarily the *target table*
  3. Cross-reference both to understand the full picture
- **Suggested fix**: Add knowledge: "NTDML-Delete logs show target table explicitly; TopSQL/SlowQuery show queries executed during the process, which may scan different tables"
- **Priority**: P1-high
- **Status**: open

### D3: Scale-out is mitigation, not resolution

- **Type**: REASONING
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent treats CPU drop after scale-out as evidence that the issue self-resolved
- **Expected behavior**: Recognize that:
  - Scale-out distributes load but doesn't fix root cause
  - If problematic operations continue at same rate after scale-out, the issue persists (just with more capacity)
  - Distinguish between "symptom alleviated" and "root cause resolved"
- **Suggested fix**: Add reasoning rule: "If CPU drops due to scale-out/scale-up, check whether the problematic operation rate also dropped; if not, the root cause is still active"
- **Priority**: P2-medium
- **Status**: open

## Key Queries

### NTDML-Delete statement count cross-day comparison

Verified via metrics query (comparing Mar 14-17 vs Mar 18) that NTDML-Delete statement count increased 74x.

### NTDML-Delete target table (Loki logs)

```bash
# Query Loki logs during fault period to find NTDML target tables
# Pattern: "batch-delete" or "NTDML" in log messages
# Result: dh_finance_delayed_task had 1000+ log entries (limit hit), confirming it as the primary target
```

### 56-column SELECT timing analysis

```bash
# SlowQuery timing breakdown (BJ timezone)
# Pre-fault (Mar 17 08:00-18:00, 10h): 227 executions
# Fault period (Mar 17 18:00 - Mar 18 09:30, 15.5h): 52 executions
# Post-scale-out (Mar 18 09:41+): 539 executions
# Conclusion: Query frequency DECREASED during fault, ruling it out as root cause
```

## Lessons Learned

1. **Timing correlation is critical**: Always verify that suspected causes show increased activity during fault periods and decreased activity when faults resolve (excluding external mitigations).

2. **Framework operations != query operations**: NTDML and similar frameworks may execute queries on different tables than their target tables. Cross-reference Loki logs (target tables) with TopSQL/SlowQuery (execution queries).

3. **Low affected_rows != low overhead**: NTDML-Delete with low affected_rows can still cause high CPU due to framework overhead (scanning, transaction setup, GC from chunk allocation failures). Look at statement *count*, not just affected_rows.

4. **Scale-out is not resolution**: When CPU drops after scale-out, check whether the problematic operation rate also dropped. If not, the root cause is still active and will return if load increases.

5. **Agent requires domain knowledge for mechanism-level diagnosis**: The agent correctly identified the observable cause (NTDML + memory allocation) but missed the reuse chunk mechanism, which requires TiDB internal knowledge not present in metrics/logs. This is acceptable for operational diagnosis but limits deeper architectural recommendations.