# ISS-0106: Agent 缺少 TSO 服务故障的直接证据采集路径

## Background

Two independent production cases surfaced the same gap: when the agent concludes that TSO is
the root cause of latency spikes, users immediately ask for the direct evidence and can't get it.

**O11Y-2581453** (ws-1780545125287-e5d2aefe0, prod):
> "Show me the evidence for 'gRPC TSO method P99 latency spiked to 10s'. Which grafana panel to look at?"

**O11Y-2581988** (ws-1780578204483-f13426462, prod):
> "Find direct evidence. Did the TSO fail to allocate timestamps? Was the write to etcd successful?"

In both cases the agent gave TSO-related conclusions but:
1. Could not point to specific Clinic metric queries or Grafana panels for TSO gRPC latency
2. Could not verify whether TSO actually failed to *allocate* timestamps (vs merely being slow)
3. Could not check etcd write success/failure as a TSO health signal

## Root Cause Analysis

`collect_evidence.py` does not query PD/TSO-specific metrics as part of the standard evidence
collection. The SKILL.md Node C logic treats "TSO high latency" as a derived observation from
TiDB-side metrics (e.g., `tidb_session_execute_duration_seconds` breakdown), but does not
include a dedicated TSO verification step.

The TSO diagnostic path requires two classes of evidence:

**Class A — TSO service health (PD-side):**
- `pd_server_tso_handle_duration_seconds` — P99 latency of TSO handle loop
- `pd_client_cmd_handle_cmds_duration_seconds{type="tso"}` — client-observed TSO wait
- `pd_server_tso_handle_cmds_total{type="success"} / total` — allocation success rate
- `pd_server_tso_handle_cmds_failed_total` — failed allocations (direct "did TSO fail" evidence)
- PD leader election events — TSO service disrupts when PD leader changes

**Class B — etcd write health (for "was etcd write successful?" questions):**
- `etcd_disk_wal_fsync_duration_seconds` — WAL fsync P99 (should be <10ms on gp3)
- `etcd_disk_backend_commit_duration_seconds` — backend commit latency
- `pd_raftstore_apply_duration_seconds` — raft apply latency (TiKV, for context)
- PD Loki logs for `"slow" OR "fsync" OR "etcd took too long"` patterns

## Fix Plan

### Step 1: Add TSO evidence segment to `collect_evidence.py`

Add new segment **A5b** (after Activity Hub, before summary) for TSO/PD health:

```python
# A5b: TSO and PD health (query when PD or TSO anomaly suspected)
def collect_tso_evidence(cluster_id, start_ts, end_ts, step, output_dir):
    tso_metrics = {
        "tso_handle_duration_p99": 'histogram_quantile(0.99, sum(rate(pd_server_tso_handle_duration_seconds_bucket{cluster_id="%s"}[5m])) by (le))' % cluster_id,
        "tso_client_wait_p99": 'histogram_quantile(0.99, sum(rate(pd_client_cmd_handle_cmds_duration_seconds_bucket{type="tso",cluster_id="%s"}[5m])) by (le))' % cluster_id,
        "tso_failed_total": 'increase(pd_server_tso_handle_cmds_failed_total{cluster_id="%s"}[5m])' % cluster_id,
        "etcd_wal_fsync_p99": 'histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{cluster_id="%s"}[5m])) by (le))' % cluster_id,
        "etcd_backend_commit_p99": 'histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{cluster_id="%s"}[5m])) by (le))' % cluster_id,
    }
    # Query each via Clinic metrics API and write to tso_evidence.json
```

### Step 2: Update `SKILL.md` Node C routing for TSO-related alerts

Add a TSO diagnosis branch in Node C:

```
When PD TSO latency is mentioned or agent suspects TSO as root cause:
  REQUIRED evidence before asserting TSO failure:
  1. Query pd_server_tso_handle_cmds_failed_total — non-zero = TSO actually failed allocations
  2. Query pd_server_tso_handle_duration_seconds P99 — should be <1ms normally
  3. Query etcd_disk_wal_fsync_duration_seconds P99 — >10ms suggests disk pressure on etcd
  4. Check PD leader election events in Activity Hub (leader change = TSO disruption)

When user asks "which Grafana panel to look at" for TSO metrics:
  Direct them to: TiDB Cloud Clinic → Metrics → PD → TSO panel group
  Key panels: "TSO Handle Duration", "TSO Wait Duration (Client)", "etcd WAL fsync"
```

### Step 3: Add evidence citation requirement

In SKILL.md Node D (reasoning rules), add:

> If TSO is asserted as root cause: MUST cite at least one of:
> - `pd_server_tso_handle_cmds_failed_total > 0` during incident window
> - `pd_server_tso_handle_duration_seconds` P99 spike aligned with user-reported latency
> - PD log entry containing "tso" + "slow" or "timeout"
>
> Without such evidence, output: "TSO high wait duration observed on the client side; direct
> evidence of TSO allocation failures not yet verified. Query pd_server_tso_handle_cmds_failed_total
> to confirm."

## Verification

After fix, for a TSO-related case:
1. `collect_evidence.py` should produce `tso_evidence.json` with the above metrics
2. `evidence_summary_for_agent.txt` should include `tso_failed_count` and `etcd_fsync_p99`
3. Agent diagnosis should cite specific metric values (e.g., "pd_server_tso_handle_cmds_failed_total
   increased by 147 during 14:30-15:00 UTC") rather than general statements
4. When asked "which Grafana panel", agent should name "PD → TSO → TSO Handle Duration"

## Related Issues

- ISS-0086: TSO Wait Duration not considered in latency attribution (upstream cause)
- ISS-0081: Making claims without etcd log/metric evidence
- ISS-0059: Cannot provide Grafana internal dashboard links