# Expected Diagnosis: O11Y-2392287

## Root Cause

TiDBExpensiveQueryCntGt100For1m alert fired on cluster prod-us-saas-gosaas-region-tidb-v85-03 (dedicated, v8.5.4). Over 100 expensive queries per minute sustained for at least 1 hour (two consecutive alerts at 19:26 and 20:16 UTC). The expensive queries caused TiKV gRPC P99 latency spikes on db-tikv-28 (9.2s) and db-tikv-6 (2.9s), triggering correlated NetworkLagIncrease alerts. Root cause is unoptimized SQL workload generating excessive expensive queries.

## Evidence Chain

1. **Metrics**: TiDB expensive query count > 100/min sustained over ~1 hour window
2. **TopSQL TiDB**: Identify top CPU-consuming SQL digests during the alert window
3. **TopSQL TiKV**: Confirm TiKV-side hotspot on db-tikv-28 and db-tikv-6 with high gRPC latency
4. **SlowQuery**: Find slow query digests with high execution time / process_keys
5. **Execution Plan**: Decoded plan shows full table scan or missing index on the expensive queries
6. **Events/Alerts**: NetworkLagIncreaseForWetech at 20:46 UTC correlates with TiKV latency spike; TiFlash jitter is background noise
7. **Loki Logs**: Check for expensive query warnings or TiKV backpressure logs

## Causality

Unoptimized SQL (likely missing index or full table scan) -> 100+ expensive queries/min -> TiKV coprocessor overload on db-tikv-28/db-tikv-6 -> gRPC P99 latency spike (9.2s) -> NetworkLagIncrease alert -> TiDBExpensiveQueryCntGt100For1m alert

## Severity

Medium — alert auto-recovered, no outage, but sustained expensive query load indicates optimization needed.

## Recommendations

1. Identify the specific SQL digest(s) causing expensive queries via TopSQL and add appropriate indexes
2. Review execution plans for full table scans or large range scans
3. Consider query-level resource control (tidb_mem_quota_query) to limit impact of individual expensive queries
4. Monitor for recurrence of the pattern (two alerts within 1 hour)

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy these to pass.
- [ ] Identifies alert type as TiDBExpensiveQueryCntGt100For1m (expensive query count)
- [ ] Establishes correct incident time window around 2026-04-13 19:26-20:16 UTC
- [ ] Queries TopSQL TiDB component to identify top CPU-consuming SQL digests
- [ ] Queries TopSQL TiKV component to identify hotspot TiKV instances
- [ ] Queries SlowQuery logs to find slow query digests in the window
- [ ] Fetches decoded execution plan (decoded_plan field) for the top expensive query
- [ ] Identifies specific table/SQL causing the expensive queries
- [ ] Correlates NetworkLagIncrease alert with TiKV gRPC latency spike
- [ ] Distinguishes background noise (TiFlash jitter, TiKV memory) from root cause
- [ ] Provides actionable recommendation (add index, optimize SQL, or resource control)