# Postmortem: TCOC-JP-20260522-0528 (JP Batch 2026-05-22–2026-05-28)

> Workspace: `ws-1779938829498-cdddd9d38`
> Date: 2026-05-30
> Reviewer: claude-code

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCOC-JP-20260522-0528 |
| Alert Type | Batch Jira triage (JP customer tickets, week of 2026-05-22–2026-05-28) |
| Tickets | TCOC-5288, TCOC-5289, TCOC-5294, TCOC-5295, TCOC-5306, TCOC-5307, TCOC-5309, TCOC-5311, TCOC-5317 |
| Workspace Name | JP 0522-0528 |
| Environment | prod |

**Issue Description**: Batch triage of 9 JP customer Jira tickets using tidb-perf-diagnosis skill. Agent read Jira content and produced concise summaries, timelines, and root cause analysis for each ticket.

## Per-Ticket Summary

### TCOC-5288 — Mercari ecforce: DB latency elevated (2026-05-19, ~28h)

- **Status**: PENDING FOR SUPPORT (open)
- **Agent conclusion**: Latency observed in customer Datadog (30–60ms vs 10–20ms baseline) is external (client-side: network RTT, connection pool, driver overhead). TiDB internal metrics (QPS, KV OPS, Clinic) show no corresponding anomaly. Only noteworthy signal: Debezium connection count growing since 5/18, but timing doesn't fully align with latency window.
- **Actual root cause**: Not yet confirmed. Agent correctly identified this as likely client-side rather than TiDB-internal.
- **Assessment**: Agent correctly surfaced the key discrepancy (Datadog vs Clinic metrics divergence) and recommended client-side investigation. Good analysis given limited data.

### TCOC-5289 — ecforce (v8.5.4): Optimizer skyline pruning regression (NOT EXISTS query)

- **Status**: Resolved
- **Agent conclusion**: v8.5.4 introduced a skyline pruning regression via PR #63591. The change added Eq/IN coverage into the compareCandidates comparison dimension, causing over-aggressive pruning that incorrectly eliminated the optimal index for correlated subquery (Apply + NOT EXISTS). Result: estimated rows exploded from ~55–237 to 551K–394M. Fixed in v8.5.5 PR #65054 (`&&` → `||` in compareCandidates). SQL Binding as immediate workaround.
- **Actual root cause**: Confirmed TiDB v8.5.4 product bug. Accurate.
- **Assessment**: Agent accurately explained the skyline pruning mechanism, the regression source, the fix PR, and the workaround. Complete and correct.

### TCOC-5294 — Mercari main: Connection drop (2026-05-24, recurrence of TCOC-5285)

- **Status**: UNDER INVESTIGATION
- **Agent conclusion**: Second recurrence of same v8.5.3 fatal race condition (tidb#58600). TiDB panic at 21:18 JST (concurrent KILL CONNECTION + CLUSTER_PROCESSLIST → StatementContext mutex race → fatal crash). TiKV-15 CPU anomaly and RaftStore Errors noted but timing doesn't fully align; cluster has not been upgraded yet.
- **Actual root cause**: Same as TCOC-5285/TCID-12718 (tidb#58600). Confirmed second recurrence.
- **Assessment**: Agent correctly linked to TCOC-5285 and identified v8.5.3 bug as root cause. Noted ongoing investigation.

### TCOC-5285 — Mercari main: Connection drop (2026-05-20, first occurrence / TCID-12718)

- **Status**: (loaded via Jira fetch during session)
- **Agent conclusion**: Two-phase incident: (1) db-tidb-961 fatal crash from Go mutex race in StatementContext.Reset() during concurrent KILL + CLUSTER_PROCESSLIST (19,301 connections dropped instantly). (2) GCP ILB/PSC reset of existing connections over 9 minutes following backend member change (TCOC-4772 pattern). Root cause: v8.5.3 bug tidb#58600.
- **Actual root cause**: Confirmed. Accurate two-phase analysis.
- **Assessment**: Excellent — detailed causal chain, identified known bug, linked to GCP ILB behavior.

### TCOC-5295 — Mercari eagle-insight: Changefeed starvation (P1)

- **Status**: Todo (no engineering analysis)
- **Agent conclusion**: New 32-RCU changefeed `eagle-insight-mysql` created on 5/25 may be starving existing 16-RCU `changefeed-kafka-20260421`. Suspect resource competition in Changefeed scheduler. No technical analysis in Jira yet.
- **Actual root cause**: Unknown. Agent's initial hypothesis (resource starvation) is plausible.
- **Assessment**: Correctly identified gap (P1 with no analysis), formed reasonable initial hypothesis.

### TCOC-5306 — Mercari main: TLS handshake failure (SSLV3_ALERT_UNEXPECTED_MESSAGE)

- **Status**: Resolved
- **Agent conclusion**: `ssl.SSLError: SSLV3_ALERT_UNEXPECTED_MESSAGE` in Python 3.13 pymysql. Hypothesis: Python 3.13 tightened TLS handshake behavior (upgraded OpenSSL interaction), creating pymysql compatibility issue rather than server-side change. No engineering conclusion documented despite Resolved status.
- **Actual root cause**: Unknown (resolved without stated reason). Agent's Python 3.13 hypothesis is plausible.
- **Assessment**: Correctly noted the unresolved mystery (closed without explanation), offered reasonable hypothesis.

### TCOC-5307 — mcweb-prod (v7.1.6): Latency spikes after memory quota increase + scale-out

- **Status**: PENDING FOR SUPPORT
- **Agent conclusion**: Conflicting analyses between Alexandru Nicu (TiKV coprocessor thread pool flood from heavy queries unblocked by memory quota increase) and Anthony Nie (TiFlash write IOPS bottleneck from concurrent write surge). Two hypotheses unresolved. Agent correctly presented both without picking a side.
- **Actual root cause**: Not yet determined. Two competing hypotheses in Jira.
- **Assessment**: Accurately captured both conflicting hypotheses. Correctly noted no resolution.

### TCOC-5309 — Infcurion (dev-tokyo-shared): Backup job name too long (K8s 63-char limit)

- **Status**: Resolved
- **Agent conclusion**: Backup name `dev-tokyo-shared-01-stg-satudora-20260526-before-2` (50 chars) + system prefix `backup-` + suffix `-2988830` = 65 chars, exceeding K8s label 63-char limit. Job creation fails silently with K8s validation error. Workaround: use ≤48-char name. Product fix: add UI validation for ≤49 chars (48 safe).
- **Actual root cause**: Confirmed product bug (K8s job name length validation missing).
- **Assessment**: Perfect — clear root cause, workaround, and product fix recommendation.

### TCOC-5311 — Mercari side-3: API 401 loop from Go Digest Auth Nonce caching bug

- **Status**: UNDER INVESTIGATION
- **Agent conclusion**: Two-phase: (1) API latency >3s (server 171ms + network) → client 5s timeout → connection abort. (2) Go client's custom Digest Auth implementation didn't clear Nonce cache after timeout abort → subsequent requests use stale Nonce → perpetual 401 loop. Root cause: client-side Digest Auth bug in Mercari's Go application (not TiDB Cloud API issue).
- **Actual root cause**: Confirmed client-side Go HTTP Digest Auth bug.
- **Assessment**: Excellent — identified two-phase cascade, correctly attributed to client code not TiDB Cloud.

### TCOC-5317 — applibot: Datadog metrics gap (2026-05-28 03:47–04:03 JST, 16 min)

- **Status**: Todo
- **Agent conclusion**: No technical analysis in Jira. Likely TiDB Cloud → Datadog push pipeline interruption (restart, network blip, rate limiting) but no evidence.
- **Actual root cause**: Unknown.
- **Assessment**: Correctly noted lack of analysis and the likely investigation path.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 4 | Correctly identified confirmed root causes for TCOC-5289/5285/5309/5311; good hypotheses for open tickets |
| Evidence Completeness | 4 | Read all available Jira data; fetched TCOC-5285 on demand; used devops skill for tenant lookup |
| Causal Chain | 4 | TCOC-5285/5311/5289 have clear A→B→C chains; open tickets correctly noted as incomplete |
| Actionability | 4 | Clear upgrade/workaround recommendations for resolved cases; appropriate caveats for open ones |
| Efficiency | 5 | 39 tool calls, 0 human interventions, no stuck calls, no wasted effort |
| **Overall** | **4.2** | Strong batch triage with no false positives |

## Human Guidance Log

No human interventions required. Agent completed all 9 tickets autonomously.

## Skill Deficiencies

No new skill deficiencies identified in this session. The agent correctly:
- Identified all confirmed root causes
- Noted where analysis was missing without fabricating conclusions
- Used devops skill appropriately for tenant lookups
- Linked related cases (TCOC-5285 ↔ TCOC-5294, TCOC-4772 ↔ TCOC-5306)

## Lessons Learned

- The tidb-perf-diagnosis skill works well for Jira triage mode: reading multiple tickets, extracting timelines and root causes, identifying gaps.
- Recurring incidents (TCOC-5285 → TCOC-5294) indicate customers need proactive upgrade prompting after known bugs are found — the agent correctly identified this pattern.
- TCOC-5295 (Changefeed starvation, P1, no analysis) and TCOC-5317 (Datadog gap, Todo, no analysis) represent gaps in JP team response capacity.
- TCOC-5309's K8s job name truncation is a recurring class of product UX bug — should be tracked separately.