# Expected Diagnosis: TCOC-JP-20260522-0528 (JP Batch 2026-05-22–2026-05-28)

## Root Cause

Nine JP customer tickets processed in batch triage mode. Three tickets have confirmed root causes: TCOC-5289 (v8.5.4 optimizer skyline pruning regression, fixed in v8.5.5 PR #65054), TCOC-5285/5294 (v8.5.3 StatementContext mutex race tidb#58600 causing fatal crash), TCOC-5311 (Mercari Go client Digest Auth Nonce not cleared after timeout). TCOC-5309 is a product UX bug (K8s 63-char label limit not enforced in UI).

## Evidence Chain

1. TCOC-5289: PR #63591 (v8.5.4) changed `&&` to aggressive pruning in `compareCandidates` → optimal index eliminated → estRows explosion (55 → 551K) → near-full scan performance
2. TCOC-5285/5294: Concurrent KILL CONNECTION + CLUSTER_PROCESSLIST → StatementContext.Reset() overwrites mutex in-place → second goroutine unlocks freed mutex → Go fatal: sync: unlock of unlocked mutex → 19,301 connections dropped
3. TCOC-5311: API latency + client 5s timeout → disconnect without Nonce invalidation → stale Nonce in retry → perpetual 401 loop
4. TCOC-5309: User backup name (50 chars) + system prefix/suffix = 65 chars → K8s label validation: `must be no more than 63 characters` → Job creation fails silently

## Causality

- TCOC-5289: Optimizer enhancement (v8.5.4 PR #63591) → over-aggressive skyline pruning → wrong index selected → row estimate explosion → full scan performance
- TCOC-5285/5294: Go fatal crash (tidb#58600) → 19,301 instant connection drops → GCP ILB backend change → 9-minute degraded recovery
- TCOC-5295: New large changefeed (32 RCU) → suspected resource starvation of existing changefeed → sync rate drops to 0 (unconfirmed)
- TCOC-5309: Missing UI input validation → user provides valid-looking name → backend adds prefix+suffix → exceeds K8s limit → silent hang

## Severity

- TCOC-5288: Low (latency from client side, TiDB unaffected)
- TCOC-5289: High (production query performance regression)
- TCOC-5294: Critical (P2 recurring connection drop)
- TCOC-5295: Critical (P1 changefeed stopped)
- TCOC-5306: Medium (TLS failure, resolved)
- TCOC-5307: High (P2 latency, conflicting analysis)
- TCOC-5309: Low (P3, workaround available)
- TCOC-5311: Medium (P1 API 401 loop from client bug)
- TCOC-5317: Low (P3 Datadog metrics gap)

## Recommendations

1. TCOC-5289/5294: Proactively contact Mercari about upgrading from v8.5.3/v8.5.4 to v8.5.5+
2. TCOC-5295: Assign engineer to investigate Changefeed resource isolation (P1 with no analysis is unacceptable)
3. TCOC-5309: File product bug for backup name validation (UI should enforce ≤49 chars)
4. TCOC-5307: Resolve conflicting analysis between Alexandru and Anthony; need Clinic data to confirm whether coprocessor thread pool or TiFlash write IOPS was the bottleneck
5. TCOC-5288: Follow up with Mercari ecforce on client-side connection pool monitoring

## Scoring Rubric

> Checklist for regression testing. The agent must satisfy these to pass.

- [ ] For TCOC-5289: Identifies v8.5.4 optimizer skyline pruning regression (PR #63591) as root cause
- [ ] For TCOC-5289: Mentions v8.5.5 fix (PR #65054) and SQL Binding as workaround
- [ ] For TCOC-5285/5294: Identifies tidb#58600 (StatementContext mutex race) as root cause
- [ ] For TCOC-5285/5294: Links the two incidents as the same recurring bug
- [ ] For TCOC-5285: Explains two-phase nature (fatal crash + GCP ILB connection reset)
- [ ] For TCOC-5311: Correctly attributes 401 loop to client-side Go Digest Auth bug, not TiDB Cloud
- [ ] For TCOC-5309: Identifies K8s 63-char label limit as root cause of backup hang
- [ ] For TCOC-5288: Notes that TiDB internal metrics show no anomaly (client-side issue)
- [ ] For TCOC-5295: Flags as P1 with no engineering analysis (gap)
- [ ] For TCOC-5317: Flags as Todo with no engineering analysis (gap)
- [ ] Completes all 9 tickets with no human intervention