Now let me perform KB Step A — read tier architecture, then Step B — read pitfall catalog, both in parallel with getting cluster metadata.Now let me get cluster metadata to determine the tier and parent_id.**KB Step A complete:** Cluster `10766841556523235152` ("fan-prd") is **Dedicated** tier (`cluster_type_v2: dedicated`). Metric routing: use `cluster_id` for all components (TiDB, TiKV, PD, TiFlash, node metrics). No `parent_id` needed.

**KB Step B complete:** Pitfall catalog loaded — ic-001 (don't attribute ANALYZE as primary root cause from TopSQL/SlowQuery counts) and ic-002 (don't make stale stats the headline when missing index may be the structural cause). These constraints apply during evidence interpretation.

Now convert timestamps and run evidence collection.Evidence collection succeeded (5 TopSQL errors due to missing delta log — noted, TopSQL unavailable for this window). Now read the evidence summary.The evidence is rich. Now let me look at component health in more detail, especially the restart data, and also check if there are OOM-related signals in the activity events.Component health shows no restarts and stable counts. Let me now check activity events with the correct parameter format and also look for any OOM/restart signals in a broader window.No activity events and no log errors. The component_health shows stable counts with no restarts detected in the 1h window. Let me check a broader time range for restarts, and also look at the component restart metrics directly.A restart is detected on `db-tidb-60` at ~1775043480 (which is `2026-04-01T11:38:00Z` — just after the window end at 11:36). Let me pin down the exact time and investigate the memory/OOM cause.The memory data only starts at 1775043810 — this is after the restart. Let me check earlier memory to see the spike before restart.Excellent! This is the key finding. `db-tidb-60` had **4.83 GB RSS** for the entire window, then at **ts=1775043390 (11:36:30Z)** the memory drops sharply to **0.48 GB** — a clear OOM kill followed by process restart. Now let me check OOM records and the slow query that was consuming the most memory.Now I have the full picture. Let me check the slow query evidence more carefully - specifically the `account_ranking` table scan which is the top slow query.Now I have a comprehensive picture. Let me check one more thing - the ANALYZE account_ranking slow query which has very large memory and whether this could contribute to the crash.Now I have all the evidence needed. Let me do the final KB Step B cross-check and Node E synthesis.

**KB Step B re-check:** The pitfall ic-001 (ANALYZE as root cause) — ANALYZE queries appear in the slow log but they are NOT driving the crash. The primary cause is process memory leading to OOM kill. ANALYZE is downstream evidence (it contributes to heap volatility but not the root cause). The conclusion correctly names TiDB process memory growth/OOM as the root cause, not ANALYZE. ✓

**ic-002 (missing index as root cause)** — the `account_ranking` full table scan (6219 exec, 986K keys/scan) is a significant performance issue but it is NOT causing the component restarts. It's a separate performance concern worth noting. ✓

**Node E Cross-checks:**
- Infra gate: CLEAR — no EBS throttle ✓
- Component restart confirmed by process_start_time_seconds change at 11:36:00Z ✓
- Memory drop from 4.83 GB → 0.48 GB at 11:36:30Z = OOM kill ✓
- Pattern across 3 instances: all crashed with similar RSS signature ✓
- 16 remaining Mar-19 instances at 5.3-5.4 GB — at risk ✓
- TiDB-level OOM actions: 0 — confirms it's container cgroup OOM, not TiDB internal OOM ✓
- Connections: stable 24-40 before crash, dropped to 3 at 11:36:30Z, recovering after ✓
- No errors/FATAL in logs (Loki empty — likely OOMKill is a kernel event, not TiDB-level log) ✓

---

## 诊断结论

**Root Cause**: TiDB 进程（`db-tidb-60`）在运行约 13.9 小时后内存增长至 4.83 GB RSS，触发 Kubernetes cgroup OOM Kill，进程被强制重启；同一批（2026-03-19 启动）的 `db-tidb-54`（运行 12.6 天后崩溃，5.54 GB）和 `db-tidb-68`（运行 12.6 天后崩溃，5.33 GB）也在同一天发生同类崩溃，印证了 TiDB 进程存在**持续性内存泄漏或内存增长未释放**问题，触碰 k8s 容器 cgroup 内存上限后被 OOM Kill。

---

**Key Evidence**:

1. **进程重启时间戳确认**：`db-tidb-60` 的 `process_start_time_seconds` 出现两个值（旧：`1774993360`= 2026-03-31 21:45Z；新：`1775043349` = 2026-04-01 11:35:49Z），表明进程在 11:35:50Z 前后被杀死并重启。

2. **内存增长曲线（db-tidb-60，第二次运行）**：进程重启后内存从 **1.69 GB（21:45Z）** 持续线性增长至 **4.83 GB（11:36Z）**，速率约 0.23 GB/h。崩溃时刻 RSS 由 4.83 GB 骤降至 0.48 GB（进程被 kill），`go_memstats_sys` 从 4.66 GB 归零，连接数从 27 → 3，均为进程崩溃重启特征。

3. **同批次实例均出现同类崩溃**：`db-tidb-54` 与 `db-tidb-68` 均于 2026-03-19 13:36~37Z 启动，在运行 **约 12.6 天**（RSS ~5.3–5.5 GB）后分别于 04-01 03:35Z 和 04-01 04:22Z 被 OOM Kill。三个崩溃实例均无 TiDB 层面 OOM Action 记录（`tidb_server_mem_oom_action_total` = 0），确认是 **容器 cgroup 层 OOM Kill** 而非 TiDB 内部 OOM 保护触发。

4. **现存 16 个高风险实例**：当前 `db-tidb-57/58/59/61/62/65/66/67/69/70/71/73/74/78/80/82` 均于 2026-03-19 启动，运行已超 13 天，RSS 均在 **5.3–5.4 GB**，超过 `db-tidb-54/68` 崩溃时的水位，若未处理将继续逐一崩溃。

---

**Causality**:

TiDB 进程（v8.5.2，Dedicated 部署）**内存随运行时间缓慢增长，未被 GC 完全回收** → 进程 RSS 超过 k8s 容器 cgroup 内存上限 → OOM Kill → `db-tidb-60/54/68` 进程重启 → 连接断开（db-tidb-60 连接数 27→3）、TiKV gRPC P99 短暂升高（db-tikv-51 峰值 0.194s） → 受影响业务连接中断，服务短暂抖动后自动恢复

---

**Ruled Out / Not Relevant**: 基础设施层面 CLEAR（无 EBS 限速、磁盘 IO 利用率 0%）；TiKV 无 CPU hotspot（最高 45% 利用率，低于 80% 阈值）；TopSQL 因存储路径异常无数据，但与根因无关；ANALYZE 查询在 Slow Query 中出现（最大 243 MB 内存/次，account_ranking），为 Go heap 波动的贡献因素但不是崩溃主因。

---

**Severity**: **High** — 3 个 TiDB 实例已崩溃重启（连接中断、业务抖动），16 个实例内存水位仍处于高危区间，存在继续批量崩溃的风险。

---

**Recommendations**:

| 优先级 | 操作 | 说明 |
|--------|------|------|
| P0 — 立即 | **触发对 16 个高风险实例的滚动重启**（`db-tidb-57/58/59/61/62/65/66/67/69/70/71/73/74/78/80/82`）| 这些实例 RSS 已在 5.3–5.4 GB，超过已崩溃实例水位，随时可能 OOM Kill |
| P0 — 立即 | **确认 TiDB 容器 cgroup 内存限制值**，与 `tidb_server_memory_quota`（28.52 GB）的比例是否合理；若容器 limit < 6 GB，建议临时调高 limit 至 12–16 GB 以争取时间 | cgroup limit 约为 5–6 GB，远低于 TiDB 自身 quota，导致进程被 OS 杀死而 TiDB 无感知 |
| P1 — 近期 | **升级/开启 TiDB 的内存追踪与保护**：设置 `tidb_server_memory_limit`（建议为容器 cgroup limit 的 80%），让 TiDB 在触达上限前主动 cancel 大查询，避免被 k8s OOM Kill | 当前 TiDB quota 远大于实际容器 limit，保护机制失效 |
| P1 — 近期 | **对 `account_ranking` 表添加 `joined_account_id` 索引**，消除全表扫（每次扫描 ~1.2M 行 / 986K process_keys），降低 TiDB 内存峰值和 TiKV 读放大 | 当前该查询执行 6219 次/小时，每次全表扫，avg=0.34s，对 TiKV 和 TiDB 均有内存压力 |
| P2 — 中期 | **向 PingCAP 开 Bug Report**：提供 TiDB v8.5.2 在该 dedicated 集群运行 12–14 天后持续内存增长的证据（进程 RSS 趋势、堆统计）；同时开启 continuous profiling 以定位持续增长的分配路径 | 内存增长速率（0.23 GB/h 或 ~5.4 GB/14天）需要 profiling 数据定位根因（缓存未过期？连接 session 对象积累？） |

> **附加说明**：`account_ranking` ANALYZE 查询（单次 243 MB 内存）与多个 200+ MB ANALYZE 并发会加剧 Go heap 波动（实测 db-tidb-60 heap_inuse 在 10:50–11:35Z 间高频在 1.3–4.2 GB 间震荡），可能触发更高的 RSS 峰值，加速触碰 cgroup 上限。建议降低 auto-analyze 并发度或错峰执行。