---
name: grafana-panel-finder
description: |
  Find Grafana dashboard panels, extract PromQL queries, and generate URLs for TiDB cluster metrics.
  Outputs: (1) PromQL queries for metric APIs, (2) Grafana URLs for visualization/screenshots.
  Use when: user needs metric queries, Grafana panel URLs, or dashboard screenshots.
  Trigger phrases: "grafana panel", "PromQL", "metric query", "TiDB CPU", "TiKV metrics", "dashboard URL".
---

# Grafana Panel Finder

Find Grafana dashboard panels, extract PromQL queries, and generate URLs for TiDB cluster metrics.

**Primary outputs:**
1. **PromQL queries** - For programmatic metric retrieval via Clinic/O11Y APIs
2. **Grafana URLs** - For visualization and screenshots
3. **Panel metadata** - Component, dashboard, semantic usage description

## Core Task

Given a user's metric query (e.g., "TiKV CPU", "QPS"), find the corresponding Grafana panel and return:
- Panel ID, Title, Dashboard name
- PromQL query (for API calls)
- Grafana URL (for visualization)
- Semantic description (to verify it matches user intent)

## Data Sources

**PSC Knowledge Base** - Panel metadata with PromQL queries:
- Location: `/opt/agentkit/data/psc_knowledge_base_*.json`
- Common paths: `../../agentkit/data/` or find via `find / -name "psc_knowledge_base_*.json" 2>/dev/null`
- **CRITICAL Version Selection Rule (by product line):**
  - If `cluster_type == "shared"` or `cluster_type_v2` in (`starter`, `essential`) → **use `psc_knowledge_base_serverless.json`**
  - If `cluster_type` in (`nextgen-host`, `premium`) or `cluster_type_v2 == "premium"` → **use `psc_knowledge_base_nextgen.json`**
    - `premium` clusters have a `parent_id` pointing to the `nextgen-host` Resource Pool — use `parent_id` for TiKV/PD metrics
  - If `cluster_type == "dedicated"` → match tidb_version: `psc_knowledge_base_dedicated_v8.5.x.json`, `dedicated_v7.5.x.json`, `dedicated_v7.5.3.json`
  - Check `cluster_type` and `cluster_type_v2` from `cluster_meta.json` (written at source preparation time)

**Dashboard Mapping** - UID and slug for URL generation:
- Location: `/opt/agentkit/data/dashboard_mapping.yaml`
- Contains version-specific dashboard UIDs and slugs

**Panel Metadata Fields:**
```json
{
  "panel_id": 8,
  "title": "CPU",
  "dashboard": "[SharedScope]TiKV-Details-Cluster-Clinic",
  "component": "TIKV",
  "group": "Cluster",
  "used_for": "monitoring CPU utilization of TiKV instances",
  "promql": ["sum(rate(process_cpu_seconds_total{...}[$__rate_interval])) by (instance)"],
  "variables": ["k8s_cluster", "tidb_cluster", "instance"]
}
```

## Search Strategies

**Choose the strategy that best fits the user's question:**

### Strategy A: Exact Match (Known Metric)
**When to use:** User asks for specific, well-known metrics like "TiKV CPU", "TiDB Memory", "QPS"

**Approach:** Match component + exact title
```bash
# Recommended tool: jq (fast and precise)
jq '[.[] | select(.component == "TIKV" and .title == "CPU")]' psc_file.json

# Alternative: Python (if jq not available)
python3 -c "
import json
data = json.load(open('psc_file.json'))
result = [p for p in data if p['component'] == 'TIKV' and p['title'] == 'CPU']
print(json.dumps(result, indent=2))
"

# Fallback: grep (slower, less precise)
grep -A10 '"component": "TIKV"' psc_file.json | grep -A10 '"title": "CPU"'
```

### Strategy B: Semantic Search (Fuzzy/Complex Query)
**When to use:** User asks vague questions like "IO issues", "performance bottleneck", "connection problems"

**Approach:** Search `used_for` field for keywords
```bash
# Example: "IO problem" → search for "io" or "disk" in used_for
jq '[.[] | select(.used_for | test("io|disk|read|write"; "i"))]' psc_file.json

# Example: "connection issues" → search for "connection" or "concurrent"
jq '[.[] | select(.used_for | test("connection|concurrent|session"; "i"))]' psc_file.json
```

### Strategy C: Group Exploration (Browsing)
**When to use:** User isn't sure what they want, or wants to see related metrics

**Approach:** List panels by component + group
```bash
# List all TiKV Cluster group panels
jq '[.[] | select(.component == "TIKV" and .group == "Cluster") | {panel_id, title, used_for}]' psc_file.json

# List all TiDB Server overview panels
jq '[.[] | select(.component == "TIDB" and (.group | test("Server|Cluster"; "i"))) | {panel_id, title}]' psc_file.json
```

## Validation & Selection

**When multiple candidates are found, prioritize:**

1. **Semantic Match** (MOST IMPORTANT) - Read `used_for` field carefully:
   - Does it match user's intent? (e.g., "overall CPU" vs "TTL job CPU")
   - Avoid overly specific panels unless user explicitly asks
   - Example: "CPU Usage" with used_for="monitoring CPU utilization" ✅ vs used_for="TTL job performance" ❌

2. **Simplicity** - Prefer simple, direct titles:
   - "CPU" > "Process Stat Cpu Usage"
   - "Memory Usage" > "Plan Cache Memory Usage"

3. **Group Importance** - Overview groups > specialized groups:
   - `group="Cluster"` or `"Server"` > `"TTL"`, `"Resource Manager"`, `"Executor"`

4. **Dashboard Relevance** - Basic dashboards > specialized ones:
   - TiDB-Cluster > TiDB-Summary (for basic metrics)
   - TiKV-Details > TiKV-Troubleshooting

## Tool Selection Priority

1. **jq** (Recommended) - Fast, precise, widely available
2. **Python json module** - More verbose but reliable
3. **grep + manual parsing** - Slow, error-prone, last resort

**Choose based on availability, not rigid rules.**

## URL Generation

### Determine Dashboard UID and Slug

From `dashboard_mapping.yaml`:
- Nextgen (Premium/BYOC, `cluster_type` in `nextgen-host`/`premium`): UIDs end with `nextgenv9_0_x`
- Dedicated (e.g., v8.5.x): UIDs end with `tidbcloudv8_5_x`
- Serverless (`cluster_type == "shared"`): Unique UIDs per dashboard (see Serverless section)
- Cross-version dashboards: `clinic-dataplane-node`, `clinic-resource-overview`

### URL Templates

**Standard Dashboards (TiDB, TiKV, PD):**
```
https://clinic.pingcap.com/grafana/d/{DASHBOARD_UID}/{SLUG}?orgId=1&var-tidb_cluster={CLUSTER_ID}&var-org={ORG_ID}&var-org_type=cloud&from={START_TIME}&to={END_TIME}&viewPanel={PANEL_ID}
```

**NodeInfo Dashboard:**
```
?orgId=1&var-k8s_cluster=All&var-tidb_cluster={CLUSTER_ID}&var-namespace=tidb{CLUSTER_ID}&from={START_TIME}&to={END_TIME}&viewPanel={PANEL_ID}
```

**Resource Overview:**
```
?orgId=1&var-cluster=All&var-tidb_cluster={CLUSTER_ID}&var-namespace=tidb{CLUSTER_ID}&from={START_TIME}&to={END_TIME}&viewPanel={PANEL_ID}
```

## Output Format

**Present results in a clear, actionable format:**

```
Found TiKV CPU panel:

Panel Information:
- Panel ID: 8
- Title: CPU
- Dashboard: TiKV-Details-Cluster-Clinic (Nextgen)
- Component: TIKV, Group: Cluster
- Used For: Monitoring CPU utilization of TiKV instances

PromQL Queries (for clinic_query_metrics):
1. sum(rate(process_cpu_seconds_total{k8s_cluster="$k8s_cluster",tidb_cluster_id="$tidb_cluster",...}[$__rate_interval])) by (instance)
2. tikv_server_cpu_cores_quota{...}

Required Variables: k8s_cluster, tidb_cluster, instance

Grafana URL:
https://clinic.pingcap.com/grafana/d/e53523ea9ee7fcf4nextgenv9_0_x/sharedscope-tikv-details-cluster-clinic?orgId=1&var-tidb_cluster=2026601847170531328&...&viewPanel=8

Next Steps:
- Use PromQL query #1 with clinic_query_metrics to fetch metric data
- Or use Grafana URL with grafana-screenshot skill to capture visualization
```

## Common Metrics Reference (Quick Hints Only)

⚠️ **Warning: These are hints, not absolute references**
- Panel IDs may vary by version
- Always verify with actual search
- Use this to understand what to search for, not as final answer

| User Query | Search Strategy | Expected Result | Notes |
|------------|-----------------|-----------------|-------|
| "TiKV CPU" | component=TIKV + title="CPU" | Panel ~8, TiKV-Details, Cluster group | Exact title match |
| "TiDB Memory" | component=TIDB + title="Memory Usage" | Panel ~3, TiDB-Cluster | General memory, not cache |
| "QPS" | component=TIDB + title="QPS" | Panel ~12, TiDB-Summary | Query per second |
| "IO 问题" | used_for contains "io" or "disk" | Multiple candidates | Semantic search required |
| "连接数" | used_for contains "connection" | Multiple candidates | Pick based on context |

## Version-Specific Notes

### Serverless (Starter/Essential clusters)

**IMPORTANT:** When `clusterDeployTypeV2` is `starter` or `essential`, ALWAYS use the serverless PSC and dashboard mapping. These dashboards use `control_plane_info` and `cluster_id`/`keyspace_id` as primary template variables (not `tidb_cluster`).

**PSC File:** `psc_knowledge_base_serverless.json`

**Dashboards:**
- TiDB Cluster: `kFzFvOTSz` / `tidb-serverless-cluster-metrics-clinic`
- TiKV Details: `_eCx0ZDvz` / `starter-essential-tikv-details-for-clinic`
- TiFlash Summary: `EHufAWDDz` / `starter-essential-tiflash-summary-for-clinic`
- ETCD Group: `IFisAZvvz` / `starter-essential-etcd-group-for-clinic`
- PD TSO: `NdbU0WDDz` / `starter-essential-pd-tso-for-clinic`
- PD MS: `hIP_AZDDk` / `starter-essential-pd-ms-for-clinic`
- Node Info: `rYdddlPWa` / `node-exporter-full`

**How to get `control_plane_info` (CRITICAL):**

The `control_plane_info` value is built from three fields in the Clinic API cluster response:

```
{shootName}/{cloudProvider}-{region}/devtier-infra
```

- `shootName` — from Clinic API `/dashboard/clusters` response (e.g., `eu-central-1-f01`)
- `cloudProvider` — e.g., `aws`, `gcp`
- `region` — e.g., `eu-central-1`, `us-west-2`

Example: `eu-central-1-f01/aws-eu-central-1/devtier-infra`

These fields are available in `cluster_meta.json`:
- `shoot_name` → shootName
- `cloud_provider` → cloudProvider
- `region` → region

**URL Template (Serverless TiDB Cluster):**
```
https://clinic.pingcap.com/grafana/d/kFzFvOTSz/tidb-serverless-cluster-metrics-clinic?orgId=1&var-cluster_id={CLUSTER_ID}&var-cluster_name={CLUSTER_NAME}&var-tenant_id={TENANT_ID}&var-tenant_name={TENANT_NAME}&var-control_plane_info={CONTROL_PLANE_INFO}&from={START_TIME}&to={END_TIME}&viewPanel={PANEL_ID}
```

**URL Template (Serverless TiKV/PD/ETCD/TiFlash/Node):**
```
https://clinic.pingcap.com/grafana/d/{DASHBOARD_UID}/{SLUG}?orgId=1&var-control_plane_info={CONTROL_PLANE_INFO}&from={START_TIME}&to={END_TIME}&viewPanel={PANEL_ID}
```

**URL Variables for Serverless:**

| Variable | Source | Example |
|----------|--------|---------|
| `var-cluster_id` | `cluster_meta.json` → `cluster_id` | `10203210933334851193` |
| `var-cluster_name` | Clinic API → `clusterName` | `plaud-file-service-eu-prod` |
| `var-tenant_id` | `cluster_meta.json` → `tenant_id` | `1372813089209291473` |
| `var-tenant_name` | Clinic API → `tenantName` | `Plaud` |
| `var-control_plane_info` | Built from `shoot_name/provider-region/devtier-infra` | `eu-central-1-f01/aws-eu-central-1/devtier-infra` |

**Note:** `var-control_plane_info` must be URL-encoded (slashes → `%2F`).

**Key differences from Dedicated/Premium dashboards:**
- Uses `control_plane_info` instead of `tidb_cluster`/`k8s_cluster`
- TiDB cluster dashboard additionally uses `cluster_id` and `keyspace_id` for per-cluster filtering
- TiKV/PD/TiFlash dashboards are shared across the control plane (no per-cluster filtering)

**Metric routing for Starter/Essential (CRITICAL):**
- **TiDB metrics only**: Use `cluster_id` directly. Replace `$keyspace_id` in PSC PromQL with the `keyspace_id` from `cluster_meta.json`.
- **All other components** (TiKV, PD, TiFlash, Node, etc.): These are shared across the control plane. Use PromQL from PSC files **as-is** — they already contain the correct filters (e.g., `job=~".*tikv-serverless.*"`). Do NOT add `control_plane_info` to PromQL — it's only used as a Grafana dashboard URL variable (`var-control_plane_info`), not as a PromQL label filter.

### Nextgen (Premium/BYOC clusters)

**IMPORTANT:** When `cluster_type` is `nextgen-host` OR `premium`, ALWAYS use nextgen dashboards and PSC (`psc_knowledge_base_nextgen.json`), even if the cluster's `tidb_version` field shows a different version (e.g., v8.5.4). Premium clusters run on a nextgen-host Resource Pool — the `parent_id` in `cluster_meta.json` points to the Resource Pool cluster ID (needed for TiKV/PD shared-scope metrics via `tidb_cluster_id`).

**PSC File:** `psc_knowledge_base_nextgen.json`

**Dashboards:**
- TiDB: `35c74b1577d3d8ednextgenv9_0_x` / `userscope-tidb-cluster-clinic` (UserScope)
- TiKV: `e53523ea9ee7fcf4nextgenv9_0_x` / `sharedscope-tikv-details-cluster-clinic`
- PD: `8325807b4eb1e39nextgenv9_0_x` / `sharedscope-pd-cluster-clinic`
- TiCDC: `89aaa7d17cd84099nextgenv9_0_x` / `sharedscope-prometheus-ticdc-new-arch-clinic`
- TiDB-Worker: `9a8b756ecab66237nextgenv9_0_x` / `sharedscope-tidb-worker-cluster-clinic`
- TiFlash-Proxy-Details: `ba0ee4f01d857d38nextgenv9_0_x` / `sharedscope-tiflash-proxy-details-cluster-clinic`
- TiFlash-Summary: `7e2ee264c3347a4bnextgenv9_0_x` / `sharedscope-tiflash-summary-cluster-clinic`
- TiProxy-Summary: `6536753b46911b41nextgenv9_0_x` / `sharedscope-tiproxy-summary-cluster-clinic`

**Missing in nextgen:** TiDB-Summary, Backup-and-Import

**Premium cluster `var-tidb_cluster` rules (CRITICAL):**

Premium clusters (`cluster_type == "premium"`) share TiKV/PD/TiFlash infrastructure with their Resource Pool (`parent_id`). The `var-tidb_cluster` URL parameter differs by dashboard scope:

| Dashboard | Scope | `var-tidb_cluster` value |
|-----------|-------|--------------------------|
| TiDB (UserScope) | Per-cluster | `cluster_id` (the premium cluster itself) |
| TiKV, PD, TiFlash, TiCDC, TiProxy (SharedScope) | Resource Pool | `parent_id` (the nextgen-host Resource Pool) |

Example: Premium cluster `10530333593343689954` with `parent_id = 2026601847170531328`:
- TiDB dashboard URL: `var-tidb_cluster=10530333593343689954`
- TiKV dashboard URL: `var-tidb_cluster=2026601847170531328`

For `nextgen-host` clusters (Resource Pool itself), always use their own `cluster_id` for all dashboards.

### Dedicated (dedicated clusters)

**PSC Files:** `psc_knowledge_base_dedicated_v8.5.x.json`, `psc_knowledge_base_dedicated_v7.5.x.json`, `psc_knowledge_base_dedicated_v7.5.3.json`

Match by `tidb_version`:
- dedicated_v8.5.x: UIDs end with `tidbcloudv8_5_x`
- dedicated_v7.5.x: UIDs end with `tidbcloudv7_5_x`
- dedicated_v7.5.3: UIDs end with `tidbcloudv7_5_3`

Slugs: `tidb-cluster-clinic`, `tikv-details-cluster-clinic`, etc.

### Cross-Version (NodeInfo, Resource Overview)

- No version suffix
- Same UID across all versions
- Different URL parameters (see URL Templates)

## Fallback Strategies

**If no exact match found:**

1. **Fuzzy title search**
```bash
jq '[.[] | select(.title | test("cpu|memory|qps"; "i"))]' psc_file.json
```

2. **Semantic search in used_for**
```bash
jq '[.[] | select(.used_for | test("cpu|memory|performance"; "i"))]' psc_file.json
```

3. **List related panels by group**
```bash
jq '[.[] | select(.component == "TIKV" and .group == "Cluster") | {panel_id, title, used_for}]' psc_file.json
```

4. **Explain and ask for clarification**
```
"没有找到精确匹配 '磁盘' 的面板。搜索了 component=TIKV 和 title 包含 '磁盘'。
找到相关面板：
- Threads IO (Panel 2258) - 用于监控 IO 吞吐
- Write duration (Panel 2301) - 用于监控写入延迟
请问您想查看哪个？或者可以换个关键词描述一下需求。"
```

## Common Issues & Solutions

**Problem: PSC file not found**
→ Use `find` to locate: `find /opt -name "psc_knowledge_base_*.json" 2>/dev/null`

**Problem: Multiple candidates, unsure which to pick**
→ Read `used_for` field, choose the most general/simple one
→ If still uncertain, list all candidates and let user choose

**Problem: No exact match found**
→ Follow Fallback Strategies above
→ Don't fail silently - explain what was searched

**Problem: Dashboard UID not in mapping.yaml**
→ Use closest version (dedicated_v8.5.x for v8.x dedicated clusters)
→ Or check dashboard JSON files directly

**Problem: jq not available**
→ Use Python: `python3 -c "import json; ..."`
→ Or grep (slower): `grep -A10 '"component": "TIKV"' psc_file.json`

## Anti-Pattern Examples

**❌ Don't do this:**
```bash
# Blindly using Quick Reference Panel ID without verification
PANEL_ID=168  # From table, assume it's always TiDB CPU
# Generate URL directly without checking PSC file
```

**❌ Don't do this:**
```bash
# Forcing jq even when it's not available
jq '.[] | ...' psc_file.json  # Fails if jq not installed
# Better: Check availability first, or use Python fallback
```

**❌ Don't do this:**
```bash
# Ignoring used_for field
# Pick first result without reading semantic description
# Result: May get "TTL CPU" instead of "General CPU"
```

**✅ Do this instead:**
```bash
# 1. Search by semantic meaning
jq '.[] | select(.component == "TIDB" and (.title | test("CPU"; "i"))) |
    select(.used_for | test("monitor.*CPU.*utilization"; "i"))'

# 2. Verify result matches user intent by reading used_for
# 3. If jq fails, gracefully fall back to Python
# 4. If multiple candidates, explain differences and ask user
```

**Rationale:** Panel IDs change across versions, but semantic meaning is stable. Tools may vary by environment, but logic should be consistent.

## Utility: Time Range Calculation

**For metric queries and Grafana URLs, calculate Unix timestamps:**

```bash
# Current time and 2 hours ago (Grafana uses milliseconds)
END_TIME=$(date +%s)000
START_TIME=$((END_TIME - 7200000))  # 2 hours = 7200 seconds = 7200000 ms

# From specific time (alert time)
ALERT_TIME="2026-03-11 04:16:06"
END_TIME=$(date -d "$ALERT_TIME" +%s)000  # Linux
# OR
END_TIME=$(date -j -f "%Y-%m-%d %H:%M:%S" "$ALERT_TIME" +%s)000  # macOS

# 1 hour before alert
START_TIME=$((END_TIME - 3600000))

# Python alternative (cross-platform)
python3 -c "
from datetime import datetime
dt = datetime.strptime('2026-03-11 04:16:06', '%Y-%m-%d %H:%M:%S')
print(int(dt.timestamp()) * 1000)
"
```

**Common scenarios:**
- "故障前1小时到故障点" → `start = fault_time - 3600000, end = fault_time`
- "最近2小时" → `end = now, start = now - 7200000`
- "告警时间前后30分钟" → `start = alert_time - 1800000, end = alert_time + 1800000`

## Example Workflow

**User asks:** "查询 TiKV CPU 使用率"

```bash
# 1. Determine version
CLUSTER_TYPE="nextgen-host"  # From cluster metadata
PSC_FILE="psc_knowledge_base_nextgen.json"

# 2. Search panel (Strategy A: Exact Match)
jq '.[] | select(.component == "TIKV" and .title == "CPU")' "$PSC_FILE"

# 3. Extract metadata
# Panel ID: 8
# Dashboard: [SharedScope]TiKV-Details-Cluster-Clinic
# PromQL: sum(rate(process_cpu_seconds_total{...}))

# 4. Get dashboard UID from mapping.yaml
# TiKV-Details (nextgen): e53523ea9ee7fcf4nextgenv9_0_x / sharedscope-tikv-details-cluster-clinic

# 5. Generate URL
URL="https://clinic.pingcap.com/grafana/d/e53523ea9ee7fcf4nextgenv9_0_x/sharedscope-tikv-details-cluster-clinic?orgId=1&var-tidb_cluster=2026601847170531328&var-org=1372813089197032230&var-org_type=cloud&from=1773158972000&to=1773166172000&viewPanel=8"

# 6. Return results (see Output Format above)
```

## Key Principles

1. **Goal over process** - Focus on finding the right panel, not following exact steps
2. **Flexibility over rigidity** - Adapt tools and strategies based on availability
3. **Validation over assumption** - Always check `used_for` to confirm semantic match
4. **Clarity over brevity** - Provide complete information for downstream use

**This is a knowledge guide, not a script. Use judgment and adapt as needed.**