# ISS-0111: collect_evidence.py 不采集 TiProxy 路由指标

## Background

**Case**: TCOC-5387 (Cygames wiz2-prd-tidb11, 2026-06-11)
**Workspace**: ws-1782112035290-90355e8ee

TCOC-5387 diagnosed a TiProxy bug (tiproxy/issues/1172) causing CPU imbalance: 14/15 TiDB nodes at 67–75% CPU while Zone A TiDB (`db-tidb-0-ac`) received near-zero traffic. The root cause was TiProxy DNS confusion after simultaneous pod restart — pods cached stale long connections to the wrong zonal owner, causing empty CPU metrics to be returned for Zone A TiDB, which were interpreted as "fully saturated" and excluded from routing.

The agent correctly identified the root cause in this case by cross-referencing TiDB CPU metric asymmetry with TopSQL distribution and engineer Jira comments. However, no direct routing evidence was available from the data collected — the agent had to rely on the engineer's post-incident comment to confirm the "zero connections to Zone A" state.

## Observed Gap

`collect_evidence.py` does not collect TiProxy-specific metrics. In the TCOC-5387 workspace, the agent collected:

- Cluster metrics (TiDB/TiKV/PD CPU, memory, disk IO)
- TopSQL (TiDB and TiKV)
- SlowQuery logs with decoded_plan
- Loki error/warn logs
- Component health (restarts, uptime)
- Activity Hub events

But not:
- `tiproxy_session_total` — per-node connection count, would directly show Zone A = 0 connections
- `tiproxy_balance_events_total` — routing rebalancing events, would show frequency of routing decisions
- `process_resident_memory_bytes{component="tiproxy"}` — TiProxy memory usage over time (for the coexisting memory leak)

Without these metrics, the agent cannot independently confirm:
1. Which TiProxy nodes are not routing to Zone A (zero connections to Zone A TiDB)
2. Whether routing events are occurring (balance_events at 0 would confirm routing is stuck)
3. The rate of memory leak in TiProxy nodes (needed to assess OOM risk)

## Root Cause Analysis

`collect_evidence.py` was written before TiProxy was commonly deployed in Dedicated clusters. The script's metric collection sections (A1 through A6) focus on TiDB/TiKV/PD/coprocessor components. No section covers TiProxy-specific routing metrics.

The script does not detect TiProxy topology at all — there is no check for `component="tiproxy"` label in Prometheus before attempting collection.

## Fix Plan

### Step 1: Detect TiProxy presence

In the topology detection phase of `collect_evidence.py`, add a check for TiProxy instances:

```python
# After existing component detection
tiproxy_instances = query_prometheus(
    f'count(process_resident_memory_bytes{{component="tiproxy",cluster_id="{cluster_id}"}}) > 0',
    start, end
)
has_tiproxy = bool(tiproxy_instances and tiproxy_instances[0]['value'][1] != '0')
```

### Step 2: Collect TiProxy routing metrics (conditional on has_tiproxy)

Add a new section A7 (or append to A1 metrics section):

```python
if has_tiproxy:
    # Connection distribution per TiProxy node
    tiproxy_sessions = query_prometheus(
        f'sum by (instance)(tiproxy_session_total{{cluster_id="{cluster_id}"}})',
        start, end, step='1m'
    )
    save_to_file('tiproxy_session_distribution.json', tiproxy_sessions)

    # Routing balance events (rate)
    tiproxy_balance = query_prometheus(
        f'rate(tiproxy_balance_events_total{{cluster_id="{cluster_id}"}}[5m])',
        start, end, step='1m'
    )
    save_to_file('tiproxy_balance_events.json', tiproxy_balance)

    # Memory usage for leak detection
    tiproxy_memory = query_prometheus(
        f'process_resident_memory_bytes{{component="tiproxy",cluster_id="{cluster_id}"}}',
        start, end, step='5m'
    )
    save_to_file('tiproxy_memory.json', tiproxy_memory)
```

### Step 3: Include in evidence summary

Update `evidence_summary_for_agent.txt` generation to include a TiProxy section when collected:

```
## TiProxy Routing (if applicable)
- Session distribution by node: {min}/{max} connections per node
- Balance event rate: {avg_rate} events/s
- Memory trend: {start_mb}→{end_mb} MB (delta: {delta_mb} MB)
```

### Step 4: Update SKILL.md routing guidance

In the TiProxy section of tidb-clinic-diagnosis/SKILL.md (or add one), note:
- When `tiproxy_session_total` shows zero connections to a zone's TiDB nodes → routing exclusion
- When `tiproxy_balance_events_total` rate = 0 → routing is stuck, not rebalancing
- Memory leak diagnosis: track `process_resident_memory_bytes{component="tiproxy"}` trend over hours

## Verification

Test with TCOC-5387 workspace (ws-1782112035290-90355e8ee):
1. Run updated collect_evidence.py on cluster 10947714666900611238 during 2026-06-11T08:00–12:00+08:00
2. Confirm `tiproxy_session_distribution.json` shows Zone A TiDB node receiving ~0 connections
3. Confirm `tiproxy_balance_events.json` shows low/zero balance event rate during incident
4. Confirm these metrics appear in `evidence_summary_for_agent.txt`