Now let me complete KB Step A (tier architecture) and Step B (pitfall catalog) before evidence collection.**KB Step A complete:** This is a **Premium** tier cluster. Metric routing:
- TiDB metrics → use `cluster_id = 10748116073341772391`
- TiKV/PD/TiFlash/node metrics → use `parent_id` (need to fetch from cluster metadata)

**KB Step B complete:** Loaded pitfall catalog. Key rules:
- ic-001: Do NOT attribute ANALYZE as primary root cause without ruling out application SQL first
- ic-002: If decoded_plan shows index issue, put CREATE INDEX as P0, not just ANALYZE

Now let me get the cluster metadata to find `parent_id`, then run `collect_evidence.py`.**KB Step A confirmed:** Premium cluster, `parent_id = 2026948324023603200`. TiKV/PD/TiFlash metrics use `parent_id`.

Now convert the time window and run evidence collection:
- `2026-03-31T10:00:00Z` → `1743415200`
- `2026-03-31T12:11:00Z` → `1743422260`Collection succeeded (2 EBS errors are non-critical — EBS metrics unavailable for this Premium cluster, which is expected). Now reading the evidence summary:All evidence collected. Now let me analyze carefully, starting with Node C → D → E.

**Key observations from evidence:**

1. **TiDB CPU 严重饱和（Gate HOTSPOT DETECTED）**: 5个TiDB实例CPU均超过81%，峰值87%（krx4fy: 27.0/31 cores）
2. **TiDB P99延迟**: `krx4fy` 实例P99高达**3.51秒**，显著高于其他实例（0.10-0.12s），是明显的单实例异常
3. **TopSQL无数据**: 无法直接通过TopSQL做CPU归因
4. **Coprocessor-worker大量重启**: 29个实例全部发生了1次重启（33次count changes，3个pod稳定）
5. **Coprocessor-worker内存峰值**: 所有实例内存峰值 ~16-18GB
6. **慢查询主力**: DELETE语句（4978次，avg 1.26s，max 19.39s），大量SELECT和UPDATE
7. **decoded_plan**: DELETE做了`TableRangeScan` → `Selection` (cop filter on tikv) → 实际扫了**5000行**但est=18,772,224（估算严重偏高），process_keys=**8,829,715**（扫描了大量key）
8. **GC Pause P99**: warning级别（0.117s），gc_duration_rate krx4fy峰值0.083s/s

让我进一步检查coprocessor-worker重启的时间分布与TiDB高CPU的关系，以及DELETE语句的更多细节：No inline time-series datapoints available (only peaks/means collected), but peaks are sufficient for analysis. Now let me query TiKV-related metrics via the parent_id to complete the picture, and also query the coprocessor-worker count timeline to understand when the restarts happened:Now I have enough evidence to synthesize the root cause. Let me apply the KB Node E cross-checks:

**KB pitfall gate check:**
- ic-001: No ANALYZE in TopSQL (TopSQL is empty). Slow queries are all application SQL. ✅ Safe.
- ic-002: The DELETE plan shows huge est/actRows divergence (est=18,772,224 vs act=5,000) and does a TableRangeScan with `process_keys=8,829,715`. This is a structural access-path issue. ✅ Will check index.

**Query-root-cause assessment:**
- TopSQL has NO data → cannot directly attribute CPU to a specific SQL digest via TopSQL
- TiDB CPU saturation is at the TiDB layer (not TiKV). The gate says TiDB hotspot, not TiKV hotspot.
- The hotspot is TiDB CPU, not TiKV. This is different from TiKV query hotspot.
- 5 TiDB instances all saturated (81-87%) → workload-wide CPU pressure on TiDB layer
- coprocessor-worker restarts (29 instances all restarted once) — this is a critical signal for Premium tier

**Node D — TiDB CPU analysis:**
- 5 TiDB instances all saturated: this is cluster-wide, not a single-instance outlier for TiDB
- `krx4fy` has P99=3.51s (vs peers ~0.1s) — this instance is worse, likely hottest at 87% CPU
- GC pause P99 = 0.117s (warning level) on 4 instances including krx4fy; gc_duration_rate for krx4fy = 0.083s/s — elevated but not severe enough to be the primary driver
- Without TopSQL data, cannot confirm which specific SQL dominates TiDB CPU

**Coprocessor-worker analysis (Premium-specific):**
- 29 out of ~30 coprocessor-worker pods restarted once during the window
- Memory peaks: 16.2–18.2 GB per pod — **very high memory usage**
- Component count changes = 33 for 3 final pods (meaning repeated scale-up/down during the window)
- This strongly suggests **coprocessor-worker OOM** events — high memory leading to pod crashes and restarts

**Causality chain:**
The workload (SELECT/UPDATE/DELETE at ~66k/18k/4.9k stmt/s respectively) is driving heavy TiKV coprocessor work. The coprocessor-worker pods in the Premium pool were running near their memory limits (~16-18 GB). The massive TableRangeScan in the DELETE (est=18M, process_keys=8.8M per execution, 4978 execs) and complex SELECT queries were pushing coprocessor-worker memory high. Coprocessor-worker OOM/restarts caused coprocessor request failures/retries, which backed up to TiDB, causing TiDB CPU saturation (all 5 instances at 81-87% CPU quota), leading to P999 SQL duration spikes.

**TTL tasks also in play:** 45+ TTL tasks active vs limit=11, suggesting TTL DELETE operations are adding significant additional read/write load on TiKV, contributing to coprocessor pressure.

---

**Root Cause**: Premium 集群的 coprocessor-worker pod 在事故窗口内因内存耗尽（OOM）发生大规模重启（29/30个实例各重启一次），导致 coprocessor 请求积压与重试，进而引发 TiDB 层多实例 CPU 全面饱和（81-87% 配额），造成 SQL P999 尾延迟大幅上升。

---

以下是完整诊断结论：

---

## 根因分析报告 — 集群 10748116073341772391

---

**Root Cause**: Premium 资源池的 **coprocessor-worker 大规模 OOM 崩溃重启**，触发 coprocessor 请求失败与重试积压，导致所有 TiDB 实例 CPU 持续饱和（81–87%），引发 P999 SQL 尾延迟严重抬升。

---

**Key Evidence**:

1. **Coprocessor-worker 全面 OOM 重启**：事故窗口内资源池 29 个 coprocessor-worker pod 均发生 1 次重启（`component_count.changes=33`），内存峰值 16.2–18.2 GB/pod（平均 ~16.7 GB），接近或触发容器内存上限；另有 2 个未列入重启列表的 pod（`zzlsmu`/`zzpoqz`）峰值 18.1–18.2 GB，为本窗口最高值。
2. **TiDB CPU 全面饱和（Gate: HOTSPOT DETECTED）**：5 个 TiDB 实例 CPU 均超 80% 配额，峰值 krx4fy 87%（27.0/31 cores）、7sixuh 84%、n3jm51 83%、79yuza 82%、7qc8pe 81%；单实例 krx4fy 同时出现 P99 延迟 3.51s（其余实例 ~0.1s），是 coprocessor 重试积压最严重的节点。
3. **DELETE 语句执行计划严重低效（est/act 严重失衡 + 超大范围扫描）**：`t_seamless_log_tiptop` DELETE（4978 次/窗口，avg 1.26s，max 19.39s）执行计划为 `TableRangeScan` → TiKV `Selection`，est=18,772,224 而 act=5,000，`process_keys=8,829,715`（单次）；该语句将大量 coprocessor 计算推送至 TiKV/coprocessor-worker，显著放大内存压力。
4. **TiDB GC Pause P99 警告**：krx4fy 实例 GC pause P99=0.117s，gc_duration_rate 峰值 0.083s/s（告警级），说明高 CPU + 高内存对象分配已导致 Go GC 负担加重，加剧了该实例的 P99 延迟。
5. **TTL 任务严重积压**：`current TTL tasks count exceeds limit`（count=45, limit=11），事故窗口内 `t_seamless_log_tiptop` 等多张表的 TTL DELETE 任务大量堆积，进一步加剧 TiKV coprocessor 读写压力。

---

**Causality**:

高频 DELETE+SELECT 大范围扫描（`t_seamless_log_tiptop` 每次 process_keys ~880 万）+ TTL 任务积压（45 tasks vs limit 11）→ coprocessor-worker 内存持续高位（~16–18 GB/pod）→ **OOM 触发 29/30 pod 崩溃重启** → coprocessor 请求失败/超时/重试积压回 TiDB → **TiDB 所有 5 实例 CPU 饱和（81–87%）** → SQL 排队等待上升 → **P999 延迟大幅抬升，krx4fy P99=3.51s**

---

**Ruled Out / Not Relevant**:

EBS 无节流（INFRA CLEAR，EBS 磁盘 IO util=0）；TiKV/PD 组件无重启无异常；TopSQL 数据缺失无法直接归因 CPU 到具体 SQL，但 coprocessor-worker OOM 证据链独立于 TopSQL；ANALYZE 未出现在慢查询列表中，不适用 ic-001。

---

**Severity**: **Critical** — 5个TiDB实例CPU全面饱和，coprocessor-worker大规模OOM，SQL P999延迟超3.5s，业务 SELECT/UPDATE/DELETE 全类型受影响。

---

**Recommendations**:

### P0 — 立即缓解

1. **优化 DELETE `t_seamless_log_tiptop` 的执行计划（access path 问题）**：
   - 当前计划：`TableRangeScan(id BETWEEN ? AND ?) + cop Selection(created_time >= ? AND created_time < ?)`，est=18M vs act=5K，process_keys=8.8M/次
   - 问题：`id` 范围扫出大量行后再在 TiKV cop 层按 `created_time` 过滤，产生巨量 coprocessor 计算
   - **优先操作**：在 `t_seamless_log_tiptop` 上为 `(created_time, id)` 或 `(created_time)` 创建复合索引，使过滤下推至 IndexRangeScan 级别，消除大范围 TableScan
   - 同时用 `ANALYZE TABLE c66_seamless_others.t_seamless_log_tiptop` 更新统计信息（当前 stats 严重过期，`est=18,772,224 >> act=5,000`）
   - 在修复索引前可先用 SQL Binding 强制该 DELETE 使用现有较优路径或小批量分批删除

2. **评估 TTL 任务配置**：`count=45` 远超 `limit=11`，说明 TTL 大幅超出默认并发上限。检查哪些表配置了 TTL（`INFORMATION_SCHEMA.TABLE_TTL_STATUS`），考虑错峰调度或拆分业务 DELETE 任务，避免在业务高峰期与 TTL 并发竞争 coprocessor 资源。

### P1 — 短期稳定性

3. **调整 Premium 资源池 coprocessor-worker 内存配额或扩容**：所有 coprocessor-worker pod 峰值内存均在 16–18 GB，已触发 OOM。联系 PingCAP/TiDB Cloud 运营团队评估上调 coprocessor-worker 内存规格或增加 pod 数量，以应对当前工作负载规模。

4. **采集连续性 Profiling（Continuous Profiling）**：在下一次高 CPU 复发时，通过 Clinic 采集 krx4fy 等高负载 TiDB 实例的 CPU profile，以精确定位 TiDB CPU 消耗函数（compile/execute/coprocessor-request 重试），验证 GC 是否也是次要贡献因素。

### P2 — 长期优化

5. **Review SELECT 慢查询**（ed185a3, a028483d, e08ee53）：avg 2.33–3.84s，max 17–20s，预览为 `t_seamless_wallet` 相关查询，需获取 decoded_plan 确认是否也有大范围扫描，按 ic-002 模式检查索引覆盖情况。
6. **评估业务高峰期的分批 DELETE 策略**：将 `BETWEEN ? AND ?` 范围 DELETE 改为小批量循环（如每次 DELETE 1000 行），降低单次 coprocessor 压力。