# Expected Diagnosis: TCOC-5011

## Root Cause

3 out of 4 coprocessor TiKV workers went offline simultaneously at 02:35 UTC (initial trigger cause TBD), leaving a single worker to handle the entire cluster's coprocessor requests. The remaining worker overloaded — memory spiked from normal 0.2GB to 10-12GB — and entered a repeated **OOM crash-restart loop** (worker `zqrjxu` restarted 3 times, `zz2l2h` restarted 2 times between 02:49-03:22). During each OOM restart, the cluster had zero available coprocessor workers, causing coprocessor P99 latency to spike to 100+ seconds, TiKV gRPC blocking, and full cluster QPS collapse from 40K to 334. Recovery occurred at 03:24 UTC when 2 new workers (`rsgx68`, `swedq3`) launched successfully.

**Key insight**: The OOM of remaining workers is the critical amplifier — without OOM, a single worker could have degraded gracefully. The OOM crash-restart loop turned a capacity reduction into a full 50-minute outage.

## Evidence Chain

1. **Metrics (P99 latency)**: TiDB query P99 latency spiked from <0.02s to 54-129s between 02:40-03:06 UTC
2. **Metrics (QPS)**: QPS collapsed from 40K to 334 during the incident
3. **Metrics (coprocessor)**: Coprocessor Index P99 latency reached 82-104 seconds across all 24 TiKV instances
4. **Metrics (gRPC)**: TiKV gRPC P99 latency reached 100-103 seconds on all instances
5. **Component health (process_start_time)**: 3 coprocessor workers disappeared at 02:35, only 1 remained (75% capacity loss)
6. **Worker memory**: Remaining workers hit 10-12GB memory (vs normal 0.2GB), indicating overload → OOM cycle
7. **SlowQuery**: 8,473 slow queries with max latency 2,022 seconds during incident
8. **Loki logs**: vmagent unable to scrape 3 worker pods from 02:54; coprocessor wait times of 2m+ visible in TiDB logs
9. **Events**: Worker pod restarts visible — `zqrjxu` restarted 3 times, `zz2l2h` restarted 2 times

## Causality

3 coprocessor workers offline (cause TBD) → single worker handles full load → memory 0.2GB → 12GB → **OOM crash** → 0 workers available → coprocessor P99 100s+ → TiKV gRPC blocked → TiDB queries timeout → QPS collapse 40K→334 → worker restarts but immediately OOM again (loop x5) → 50min outage → 2 new workers launch at 03:24 → recovery

## Severity

Critical — full cluster unavailable for ~50 minutes

## Recommendations

1. **Immediate**: Set coprocessor worker `minReplicas=3` to prevent capacity collapse
2. **Short-term**: Tune HPA metrics to avoid false-positive scale-down under load; add worker count < 3 alert
3. **Medium-term**: Investigate root cause of initial 3-worker simultaneous offline (K8s events, HPA logs)
4. **Long-term**: Add circuit breaker / backpressure in coprocessor request path to prevent cascade

## Scoring Rubric

To pass this case, the Agent must:
- [ ] Identify coprocessor worker failure as root cause (not query-level issue)
- [ ] Identify **OOM as the critical amplifier** — worker memory 10-12GB, repeated crash-restart loop
- [ ] Check component health / uptime for ALL components including coprocessor workers
- [ ] Use `--parent-id` when running `collect_evidence.py` on Premium clusters (worker metrics require Resource Pool scope)
- [ ] Determine correct incident time window from metrics (not Jira creation time)
- [ ] Reconstruct the failure timeline: worker offline → overload → OOM loop → cascade → recovery
- [ ] Query Loki logs to confirm worker unavailability
- [ ] Provide capacity-protection recommendation (minReplicas or similar)