---
name: alert-log-analyzer
description: Analyze TiDB Cloud alerts (Import/DM/CDC/BR) and locate root causes by querying logs. Use when user wants to troubleshoot alert issues or find root causes of failed tasks.
---

# Alert Log Analyzer Skill

Analyze TiDB Cloud alerts (Import/DM/CDC/BR) and locate root causes by querying logs.

## Prerequisites

- **clinic-api skill** - Must be installed and have `CLINIC_API_KEY` configured
  - Configuration: Add `export CLINIC_API_KEY="your-key"` to `~/.bashrc`

## Usage

When a user provides an alert (TCOC or O11Y) to troubleshoot:

### Step 1: Parse Alert Information

Extract key information from Jira or alert:
- **cluster_id**: TiDB cluster ID
- **import_id / task_id**: Task ID (Import/DM/CDC/BR)
- **alert_time**: Alert occurrence time
- **alert_name**: Alert name (e.g., ImportTiKVImportTooSlow)
- **namespace**: K8s namespace (if available)

### Step 1.5 (Optional): Search Historical Alerts

Search for similar past alerts in Feishu knowledge base:

**Knowledge Base**: https://pingcap.feishu.cn/wiki/M6UVwB2WKiZhzJkA9NZcxlpknWc

Search for:
- Same alert_name (e.g., `ImportTiKVImportTooSlow`)
- Same cluster or tenant historical alerts
- Previous handling methods and solutions

### Step 2: Query Cluster Logs

```bash
# 1. Get available labels for a cluster
curl -s "https://clinic.pingcap.com/data-proxy/loki/{cluster_id}/api/v1/labels" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}"

# 2. Query logs based on component type
# Import: app=import-{import_id}-xxx (query around alert time!)
# DM: app=dm-worker-{task_id}-xxx  
# CDC: app=ticdc-{cluster_id}-xxx
# BR: app=backup-ab-{cluster_id}-{backup_id}-xxxxx (use app label)
```

### Step 3: Analyze Logs

Common error patterns:

| Component | Common Errors | Keywords |
|-----------|---------------|----------|
| Import | Region split failure, EpochNotMatch | `EpochNotMatch`, `split region failed` |
| Import | TLS connection interruption | `TLS handshake error`, `EOF` |
| DM | Task failure, network timeout | `error`, `timeout`, `failed` |
| CDC | Changefeed lag, sync failure | `error`, `sync`, `lag` |
| BR | Backup/Restore failure | `error`, `failed`, `timeout` |

## Query Examples

### Import Alert Troubleshooting

**✅ Import logs ARE available in Loki!**

The key is to query **around the alert time** - import tasks don't run all the time.

**Import log query:**

```bash
# CRITICAL: Query the time window when import was running (not all time!)
# Get import_id from alert, then find the app name

CLUSTER_ID="10922378279555364285"
IMPORT_ID="1203950"
# Alert time: 2026-03-05 20:20:00 UTC

# Step 1: Find the exact app name (contains import_id)
START=$(date -d '2026-03-05T19:00:00Z' +%s%N)
END=$(date -d '2026-03-05T21:00:00Z' +%s%N)

curl -s -G "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/label/app/values" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" | grep "import-${IMPORT_ID}"

# Step 2: Query import logs (around alert time!)
curl -s -G "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/query_range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  --data-urlencode 'query={app="import-'"${IMPORT_ID}"'-xxxxx"}' \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode 'limit=30'

# Step 3: Search for errors
curl -s -G "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/query_range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  --data-urlencode 'query={app="import-'"${IMPORT_ID}"'-xxxxx"} |= "error"' \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode 'limit=20'
```

**Common Import Error Keywords:**

| Keyword | Meaning | Possible Cause |
|---------|---------|----------------|
| `TLS handshake error` | TLS handshake failure | Network instability, connection drops |
| `EOF` | Connection closed | Client/server disconnect |
| `error` | General error | Check message details |
| `context canceled` | Task cancelled | Import cancelled or timeout |

**Important Notes:**
- Import logs only exist when import task is running
- MUST query around the alert time, not arbitrary time ranges
- App name format: `import-{import_id}-{random_suffix}`

### BR (Backup/Restore) Alert Troubleshooting

**CRITICAL: Use `app` label (NOT `service_name`) to query BR logs!**

The BR logs are under the `app` label with format: `backup-ab-{cluster_id}-{backup_id}-xxxxx`

**Step 1: Query BR logs using `app` label**

```bash
CLUSTER_ID="10083921845422295624"
BACKUP_ID="2971751"

# 1. Find the exact app name for this backup
START=$(date -d '2026-03-11T18:00:00Z' +%s%N)
END=$(date -d '2026-03-11T19:00:00Z' +%s%N)

# Query all app values containing "backup" and your cluster/backup ID
curl -s -G "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/label/app/values" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" | grep "backup-ab-${CLUSTER_ID}-${BACKUP_ID}"

# 2. Query BR task logs using app label
# Example: app=backup-ab-10083921845422295624-2971751-w9dch
curl -s -G "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/query_range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  --data-urlencode 'query={app="backup-ab-'"${CLUSTER_ID}"'-'"${BACKUP_ID}"'-xxxxx"}' \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode 'limit=30'

# 3. Search for errors in BR logs
curl -s -G "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/query_range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  --data-urlencode 'query={app="backup-ab-'"${CLUSTER_ID}"'-'"${BACKUP_ID}"'-xxxxx"} |= "error"' \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode 'limit=20'
```

**Key Points:**
- **ALWAYS use `app` label** - not `service_name`
- The app name format is: `backup-ab-{cluster_id}-{backup_id}-{random_suffix}`
- Get the exact app name from step 1, then use in step 2

**Common BR Error Keywords:**

| Keyword | Meaning | Possible Cause |
|---------|---------|----------------|
| `receive a backup response timeout` | Backup response timeout | Network latency, high TiKV load, large number of ranges |
| `get timestamp too slow` | PD timestamp get slow | PD pressure or network issue |
| `error` | Error | Specific error message |
| `failed` | Failure | Backup failure |
| `context canceled` | Context canceled | Task cancelled or timed out |

**Common BR Alert Types:**

| Alert Name | Meaning | Troubleshooting Focus |
|------------|---------|---------------------|
| TiDBClusterBackupDurationTooLong | Backup duration too long | Check for timeout logs, backup progress |
| TiDBClusterBackupFailed | Backup failed | Check error/failed logs |
| TiDBClusterRestoreDurationTooLong | Restore duration too long | Check restore progress logs |
| TiDBClusterRestoreFailed | Restore failed | Check error logs |

**BR Optimization Recommendations:**

1. Add rate limit: `--ratelimit 128MB`
2. Increase concurrency: `--concurrency 32`
3. For large clusters (30+ TiKV nodes), consider 24h alert threshold

### Generic Log Query Template

```bash
# Replace these variables before use
CLUSTER_ID="cluster_id"
APP_PATTERN="app name (can use partial match)"
START_TIME="2026-03-05T20:00:00Z"
END_TIME="2026-03-05T20:30:00Z"
LIMIT=30

# Query logs (URL encode the query)
curl -s "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/query_range?query=%7Bapp%3D%22${APP_PATTERN}%22%7D&start=${START_TIME}&end=${END_TIME}&limit=${LIMIT}" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}"

# Search with keyword
KEYWORD="error"
curl -s "https://clinic.pingcap.com/data-proxy/loki/${CLUSTER_ID}/api/v1/query_range?query=%7Bapp%3D%22${APP_PATTERN}%22%7D%20%7C%3D%20%22${KEYWORD}%22&start=${START_TIME}&end=${END_TIME}&limit=${LIMIT}" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}"
```

## Output Format

After troubleshooting, output the following structure:

```
## Alert Analysis

| Field | Value |
|-------|-------|
| Alert | {alert_name} |
| Cluster | {cluster_name} ({cluster_id}) |
| Task ID | {task_id} |
| Time | {alert_time} |

### Error Logs

```
{error log content}
```

### Root Cause Analysis

{problem description}

### Recommendations

{actionable suggestions}
```

## Notes

1. **3-layer Clusters**: Some 3-layer architecture logs may be in control plane, need to distinguish
2. **Time Range**: Query time range should extend 30 minutes before and after alert time
3. **API Rate Limit**: Avoid frequent calls, control query frequency appropriately
4. **Historical Lookup**: Always check Feishu Bitable for similar past alerts before deep-diving into logs
5. **Query Around Alert Time**: Import/BR logs only exist when task is running - MUST query around alert time, not arbitrary time ranges