# ISS-0105 Plan: Agent lacks knowledge of TiDB Cloud Essential cluster control plane issues

## Background

GitHub issue #636: workspace ws-1780481866811-7d85039da (prod, TCID-12761, zhipeng.su@pingcap.com)

User feedback: "应该是 essential 控制面的问题" (Should be an Essential control plane issue)

The user diagnosed the root cause as an Essential cluster control plane problem, but the agent
apparently did not identify this. This represents a KNOWLEDGE gap in the agent's understanding
of TiDB Cloud Essential cluster architecture and the types of problems that originate from its
control plane.

## What is the TiDB Cloud Essential Control Plane

TiDB Cloud Essential is the serverless/entry-level cluster tier. Its "control plane" refers to
the infrastructure layer that manages:
- Cluster lifecycle operations (start/stop/scale)
- Resource allocation for serverless clusters (RCU assignment)
- Connection routing and load balancing for shared infrastructure
- Internal scheduling decisions (e.g., cold start, resource reclamation)

Control plane issues manifest as:
- Unexpected cluster pauses or restarts not triggered by user
- Connection failures during control plane operations
- Latency spikes caused by resource reallocation
- Errors in cluster status transitions

## What Went Wrong

The agent lacks:
1. Knowledge of Essential cluster control plane architecture and failure modes
2. Diagnostic data sources to detect control plane issues (e.g., specific metrics or events
   that signal control plane operations vs. user workload issues)
3. Decision rules to route "Essential cluster behaving oddly" toward control plane investigation

Current `collect_evidence.py` and `SKILL.md` focus on Dedicated cluster diagnostics
(TiKV/TiDB/PD nodes, Raft, IOPS) which are inapplicable to Essential's shared-infrastructure model.

## Relevant Cases

- TCID-12761 (ws-1780481866811-7d85039da): user identifies control plane as root cause

## Root Cause Analysis

The `tidb-clinic-diagnosis` SKILL.md does not include:
- Essential cluster tier identification at intake
- Control plane issue detection patterns
- Guidance on what metrics/events signal control plane vs. workload issues

## Fix Plan

### Step 1: Add Essential cluster tier detection in SKILL.md

In the intake (A0) section, after identifying the cluster:

```
If cluster tier is "Essential" (Serverless):
  - Diagnostic data sources differ from Dedicated clusters
  - TopSQL/SlowQuery may be unavailable (if data proxy not supported)
  - Component-level metrics (TiKV/TiDB/PD instance counts) may not apply
  - Focus on: Activity Hub events, connection metrics, RCU consumption, latency percentiles
```

### Step 2: Add Essential control plane failure patterns to SKILL.md

Document known Essential control plane failure patterns:

1. **Cluster cold start latency**: After a pause/resume cycle, first connection may be slow
2. **Resource reclamation**: Control plane may reclaim resources during low-activity periods,
   causing brief unavailability
3. **RCU throttling**: If RCU limit is reached, cluster may throttle or reject connections
4. **Shared infrastructure events**: May affect multiple clusters in the same pool

Diagnostic signals:
- Activity Hub: look for `PauseCluster`, `ResumeCluster`, `ScaleRCU`, `ModifyClusterConfig` events
- Connection metrics: sudden drop to 0 followed by recovery
- Latency: sudden P99 spike with no corresponding CPU/IOPS increase (control plane overhead)
- Error logs: "cluster not ready", "resource limit exceeded"

### Step 3: Add routing rule for Essential cluster complaints

If user reports Essential cluster issue with no clear performance root cause in metrics:
1. Check Activity Hub for recent control plane operations
2. Check if cluster was paused/resumed recently
3. Check RCU consumption trend (was RCU limit hit?)
4. If none of the above, escalate to "possible control plane bug" and recommend filing
   a support ticket with the specific time window

### Step 4: Add to collect_evidence.py

For Essential clusters, A5 (Activity Hub events) should use different event type filters:
- Include: `PauseCluster`, `ResumeCluster`, `ScaleRCU`, `ModifyRCUConfig`
- Add dedicated "essential_events" section to evidence_summary

## Verification

1. Run diagnosis on an Essential cluster with a control plane complaint
2. Agent should identify the cluster tier as Essential in the intake step
3. Agent should check Activity Hub for control plane events
4. Agent should correctly attribute unexplained latency/connection issues to control plane
   operations when Activity Hub shows relevant events