Now I'll execute KB Step A and Step B in parallel before collecting evidence.KB Step A and Step B complete. Now I need cluster metadata to determine the tier. Let me get cluster metadata first.**KB Step A Result:** This is a **Premium** cluster. Metric routing:
- TiDB metrics: use `cluster_id = 10109909546444159555`
- TiKV/PD/TiFlash/node metrics: use `parent_id = 2013516099403907072`

**KB Step B Result:** Pitfall rules applied:
- ic-001: Do NOT name ANALYZE as primary root cause — deprioritize ANALYZE in TopSQL/SlowQuery
- ic-002: Do NOT make "stale statistics" the P0 headline if a missing index is the structural root cause

Now convert the time window and collect evidence:Evidence collected successfully (2 EBS TiFlash/TiKV errors are non-critical 422s). Now read the summary:Now I have complete evidence. Let me perform Node C → D → E analysis.

**Node C: App-layer Triage**

Key observations:
- **TiDB P99 latency**: extremely low, 13ms peak across all instances — no latency anomaly at TiDB layer
- **TiDB CPU**: 34–37% utilization (10.5–11.5 cores vs 31 quota) — normal, no saturation
- **TiDB GC**: GC pause P99 up to 117ms on 2 instances (warning severity), but GC duration rate is negligible (0.002 s/s), GC frequency 0.2/s — GC is not the bottleneck
- **Gate: INFRA CLEAR** — no EBS throttle, disk IO normal
- **Gate: HOTSPOT not detected** — all TiKV instances within normal variance
- **RCU**: 500K RU/s peak on `default` group, request_p90 = 0.5ms (normal), available_ru = 0.0 → **RU floor at 0 is potentially significant**
- **TopSQL**: No data returned
- **Workload**: Select ~17K/s, Set ~14K/s, Insert ~7K/s, Commit ~6.8K/s — heavy OLTP read+write

**Key anomaly — TiDB ERROR logs: 156,880 errors in 49 minutes (54 min window)**. This is extremely high — equivalent to ~53 errors/second sustained throughout the window.

**Node D: Component Deep-dive**

The massive TiDB ERROR count (156,880) is the dominant signal. This must be investigated further. Let me query the actual error content from Loki logs.The log query returned INFO-level connection timeout messages, but the error count was 156,880. Let me query specifically for ERROR-level entries:Now let me perform the final Node E cross-check and gate validation.

**Node E: Cross-check and Synthesize**

**Pre-check gate:**
- KB pitfall ic-001: No ANALYZE in TopSQL (TopSQL returned no data); slow query list does not include ANALYZE. ✓
- KB pitfall ic-002: No ANALYZE as headline. The findings point to stale statistics + possible missing index. Need to check ic-002: for orders_tpg the index EXISTS (`I_ORDERS_TPG_BT_B_P_P_RT`) but has catastrophic stats divergence (est=6.8 vs act=380K). This is a stale statistics case where the index is present — ic-002 does NOT override here since the index already exists and the structural problem is the statistic accuracy. ✓

**Query-root-cause gate check:**
- The leading cause is query-driven: slow queries do exist, but DB-side P99 = 13ms and QPS is increasing (not dropping). The business symptoms (service unavailable, timeout, QPS drop) are NOT confirmed by DB metrics.
- TiKV CPU hotspot: NOT detected by gate checks. TiKV CPU max 4.82 cores (well under saturation).
- TopSQL: No data returned.
- Conditions NOT all met → query-root-cause status NOT confirmed → do NOT load `tidb-query-tuning`.
- Slow queries are treated as **secondary contributing factors** (amplifying connection hold time).

**Cross-reference validation:**
- ✅ Metrics evidence: Connection count surges 24K+; error disconnections 27-135/s persistent; DB QPS actually increasing during incident window
- ✅ Query evidence: orders_tpg estRows=6.8 vs actRows=380K (stale stats); t_game_legal avg 5.95s (likely full scan)
- ✅ Log/event evidence: TSO warnings (pre-existing, not incident-specific); JWKS errors (benign); no critical errors; no component restarts

**Timeline alignment:**
- The "incident window" 02:35-03:24 shows no discrete inflection point in DB metrics. The connection surges and error disconnections are present before, during, and after the window. This means the **application perceived the outage** during a period when connection pool exhaustion reached a threshold, but the DB itself was stable throughout.

---

## 根因分析报告

**根因（Root Cause）**

集群数据库层自身运行正常；业务侧感知到的"服务不可用、查询超时、QPS 下跌"，**根本原因是应用层连接池周期性打满（Connection Pool Exhaustion）**，叠加两条高频/高延迟慢查询持续占用连接，导致应用侧新请求无法获取连接而超时报错。

---

**关键证据链（Key Evidence）**

| # | 证据 | 具体数值 | 含义 |
|---|------|----------|------|
| 1 | TiDB 总连接数周期性突刺 | 单实例峰值 6,605（4 实例合计 ~25,000+）；以约 20 分钟为周期反复出现 | 连接池爆满触发应用层请求积压 |
| 2 | TiDB 错误断连率 | 全周期 27–135 次/秒（事故窗口前已存在，非突发新增） | 大量连接因错误被关闭，和上游连接打满互为因果 |
| 3 | 数据库侧 QPS 实际**上升**而非下降 | 02:35 约 43K/s → 03:24 约 46K/s，持续增加 | DB 层未见 QPS 下跌，说明"QPS 下跌"仅在应用层可见（被池限流） |
| 4 | TiDB P99 延迟 / TiKV gRPC P99 正常 | TiDB P99=13ms，P999=49ms；TiKV gRPC P99=22ms；Coprocessor P99=2ms | 数据库计算/存储层无瓶颈 |
| 5 | `orders_tpg` 慢查询：统计信息严重过时 | estRows=**6.8** vs actRows=**380,026**（误差 55,000×）；exec_count=2617（~53次/分），avg 0.41s，process_keys=776K/次 | 索引已存在（`I_ORDERS_TPG_BT_B_P_P_RT`）但统计信息极度陈旧，每次扫描 39 万行，总 KV 读压高 |
| 6 | `t_game_legal` 慢查询延迟极高 | avg=**5.95s**，max=**9.82s**，exec_count=201 | 单次执行持续占用连接 5–10s，加重连接积压 |
| 7 | 基础设施及组件健康正常 | EBS throttle=0，Disk IO util=0，coprocessor-worker/tikv/pd 无重启，RCU request p90=0.5ms | 排除 EBS/磁盘/组件故障 |

---

**因果链（Causality）**

```
orders_tpg 统计信息过时（est=6.8 vs act=380K）
  + t_game_legal 全表扫描（avg 5.95s）
  ↓ 大量慢查询长期占用连接（53次/分 × 0.41s + 4次/分 × 5.95s）
  ↓ TiDB 单实例连接数周期性突刺至 6,600+（总计 ~25,000）
  ↓ 应用连接池耗尽 → 新请求无法获取连接
  ↓ 应用侧：查询超时 + 服务不可用 + 感知 QPS 下跌
```

---

**排除项（Ruled Out / Not Relevant）**

- 无 TiKV CPU 热点（gate 检测 pass，TiKV CPU 最高 4.8 cores，24 实例均衡）
- 无 EBS/磁盘 IO 瓶颈
- 无 PD Leader 切换或 TSO 服务失败（"get timestamp too slow" 为每分钟 PD metering write 触发的周期性小抖动，30–160ms，事故前后均存在）
- 无组件 OOM 或重启
- TiDB 错误日志中的 156,880 条"error"实为 INFO 级 `waitTimeout=300` 空闲连接清理日志（含 "error" 关键字被计入），非真实业务错误
- TopSQL 无数据，无法做 SQL 级 CPU 归因；慢查询归类为**加剧因素**，而非独立根因

---

**严重性（Severity）**

**High** — 业务侧感知服务不可用，但数据库层处于健康状态；影响范围取决于应用连接池大小与超时配置，具有周期性复发风险。

---

**处置建议（Recommendations）**

### 紧急（立即）

1. **检查应用侧连接池配置**：池最大连接数是否与 DB 侧承受能力匹配（当前 4 实例 × 6,600 = 26,400 个连接，应用是否设置了合理的池上限和 `acquire_timeout`）。排查是否存在连接泄漏（请求完成后未归还连接）。

2. **运行 `ANALYZE TABLE orders_tpg`**（针对 `billtime`、`flag` 列的统计信息刷新）：

   ```sql
   ANALYZE TABLE c66_dc.orders_tpg COLUMNS billtime, flag;
   ```

   **优先级说明**：索引 `I_ORDERS_TPG_BT_B_P_P_RT` 已存在且被使用，问题在于统计信息极度陈旧（est=6.8 vs act=380K，误差 55,000 倍）。刷新统计信息后优化器能正确评估行数，从而改善执行计划选择和 KV 扫描量。

3. **排查 `t_game_legal` 慢查询**：确认 `game_biz_id` 列是否有索引。若无，添加：

   ```sql
   ALTER TABLE t_game_legal ADD INDEX idx_game_biz_id (game_biz_id);
   ```

   当前 avg=5.95s / max=9.82s 严重影响连接占用时长。

### 短期（本周内）

4. **配置 TiDB `max_connections` 限流**（Premium 集群）：为每个 TiDB 实例设置合理的最大连接数上限，避免瞬间 6,600+ 连接对内存和调度造成冲击。建议参考当前 quota（31 CPU）和业务并发量设定，例如 `--max-connections=2000`（需要与应用团队确认）。

5. **开启自动统计信息收集调优**：调整 `tidb_auto_analyze_ratio`（当前建议 0.1）和时间窗口，确保 `orders_tpg` 等高写入表的统计信息保持新鲜：

   ```sql
   SET GLOBAL tidb_auto_analyze_ratio = 0.1;
   SET GLOBAL tidb_auto_analyze_start_time = '02:00 +0000';
   SET GLOBAL tidb_auto_analyze_end_time = '06:00 +0000';
   ```

6. **添加应用层重试与熔断机制**：在连接获取失败时快速失败（fast fail）并上报，而非无限重试（重试风暴会加剧连接堆积）。

### 中期（排查连接周期性突刺根因）

7. **追踪约 20 分钟一次的连接突刺触发源**：日志中连接浪涌以约 20 分钟为周期，疑似某定时任务或批量操作触发（如定时报表、定时同步、定时结算）。建议在应用侧配合数据库 connection_id 追踪周期性连接来源，并为该类任务使用独立连接池或限流策略。

---

详细证据数据可在工作区报告中查看：`evidences/workspace_evidence_report.json`