# Expected Diagnosis: TCOC-4964

## Root Cause

NTDML-Delete operations on `dh_finance_delayed_task` table increased 74x (from 2/s to 147/s) during the fault period. These high-frequency batch operations triggered the reuse chunk OOM issue: the 56-column internal scans prevented chunk reuse, causing memory allocation rate to double (4.17 → 8.93 GB/s) and GC overhead to spike. This resulted in sustained CPU pressure (16 → 54 cores peak) despite low actual affected_rows (~10.9 rows/s).

## Evidence Chain

1. **Metrics**: NTDML-Delete statement count increased 74x on Mar 18 vs Mar 14-17 baseline; no other operation showed significant change
2. **Metrics**: Alloc rate doubled from 4.17 to 8.93 GB/s during fault period
3. **Metrics**: CPU >30 cores occurred 28 times on Mar 18 vs 0 times on Mar 14-17
4. **Loki logs**: NTDML-Delete target table confirmed as `dh_finance_delayed_task` (1000+ log entries during fault period)
5. **SlowQuery**: 56-column SELECT on `dh_active_receive_logs` **decreased** during fault (52 executions) vs pre-fault (227 executions), ruling it out
6. **Jira comments**: Reuse chunk OOM issue confirmed as mechanism; workaround implemented to skip reuse chunk for wide-column queries

## Causality

```
NTDML-Delete on dh_finance_delayed_task (74x increase)
  → High-frequency batch execution (967 statements/s peak)
  → Internal SELECT scans with 56 columns
  → Reuse chunk fails due to wide column size
  → Memory allocation doubles (4.17 → 8.93 GB/s)
  → GC frequency increases
  → CPU sustained at 30-54 cores
```

## Severity

High — sustained CPU elevation from 16 to 54 cores for 15.5 hours, affecting 4 out of N TiDB nodes. Mitigated by scale-out but root cause required code workaround.

## Recommendations

1. **Immediate**: Implement workaround to skip reuse chunk when total output column size exceeds threshold (already applied per Jira)
2. **Short-term**: Review NTDML batch sizing and frequency for `dh_finance_delayed_task` to reduce statement count
3. **Long-term**: Optimize reuse chunk mechanism to handle wide-column queries without triggering OOM

## Scoring Rubric

> Checklist for regression testing. The agent must satisfy ALL items to pass.

- [ ] Identify NTDML-Delete as the primary operation that changed (74x increase)
- [ ] Query and report the target table for NTDML-Delete operations (`dh_finance_delayed_task` from Loki logs)
- [ ] Report memory allocation rate increase (approximately 2x: 4.17 → 8.93 GB/s)
- [ ] Note the timing mismatch: 56-column SELECT on `dh_active_receive_logs` **decreased** during fault period, ruling it out as root cause
- [ ] Explain that low affected_rows does not mean low overhead; NTDML framework execution itself causes CPU load
- [ ] Distinguish scale-out (mitigation) from root cause resolution
- [ ] Cross-reference at least 3 data sources (metrics, SlowQuery, Loki logs) to establish causality
- [ ] Provide specific evidence that NTDML statement count is the only metric with 10x+ change comparing fault day to baseline days