# DataService Infra Grafana Skill

Access DataService Infra Grafana dashboards for Import/DM/BR monitoring.

## Prerequisites

**Environment Variable**: `DATASERVICE_GRAFANA_TOKEN`

Set in `~/.bashrc`:
```bash
export DATASERVICE_GRAFANA_TOKEN="*"
```

**Usage**: Always source the environment variable before running queries:
```bash
source ~/.bashrc
```

## Dashboard Information

- **Dashboard URL**: https://grafana.oauth.prod.aws.tidbcloud.com/d/9sVoewCIz/global-control-user-metrics
- **Dashboard UID**: 9sVoewCIz

## Standard Query Workflow

When querying Grafana metrics, always follow this workflow to ensure accuracy:

### Step 1: Get Dashboard from Grafana API

```bash
source ~/.bashrc
curl -s -H "Authorization: Bearer \${DATASERVICE_GRAFANA_TOKEN}" \
  "https://grafana.oauth.prod.aws.tidbcloud.com/api/dashboards/uid/9sVoewCIz" \
  > /tmp/dashboard.json
```

### Step 2: Get Dashboard Variables (from templating)

```bash
source ~/.bashrc
curl -s -H "Authorization: Bearer \${DATASERVICE_GRAFANA_TOKEN}" \
  "https://grafana.oauth.prod.aws.tidbcloud.com/api/dashboards/uid/9sVoewCIz" | \
  python3 -c "
import json, sys
d = json.load(sys.stdin)
templating = d.get('dashboard', {}).get('templating', {}).get('list', [])
for t in templating:
    name = t.get('name')
    query = t.get('query', '')
    print(f'{name}={query}')
"
```

This will output:
```
precheck_errors=(INSTR(message, '[Lightning:PreCheck:ErrCheckDataSource') > 0 OR INSTR(message, '[Lightning:PreCheck:ErrPreCheckFailed') > 0)
userdata_errors=(INSTR(message, 'ErrChecksumMismatc') > 0 OR INSTR(message, 'ErrRestoreTable') > 0 OR ...)
non_internal_tenant=(tenant_id NOT IN (1372813089194211283, 1372813089187791289, 1372813089189381281, 1372813089208161328, 1372813089188691290))
alias_non_internal_tenant=(a.tenant_id NOT IN (1372813089194211283, 1372813089187791289, 1372813089189381281, 1372813089208161328, 1372813089188691290))
import_speed_min_size=10737418240
```

### Step 3: Get SQL Queries from Dashboard

Extract all SQL queries from the dashboard:

```bash
cat /tmp/dashboard.json | python3 -c "
import json, sys
d = json.load(sys.stdin)
panels = d.get('dashboard', {}).get('panels', [])
for panel in panels:
    title = panel.get('title', 'Untitled')
    targets = panel.get('targets', [])
    for t in targets:
        sql = t.get('rawSql', '')
        if sql:
            print(f'# {title}')
            print(sql)
            print()
"
```

### Step 4: Replace Variables in SQL

Replace these patterns in the SQL:
- `$__timeFilter(created_at)` → `created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY) AND created_at <= NOW()`
- `${non_internal_tenant:sql_condition}` → Use value from Step 2
- `${alias_non_internal_tenant:sql_condition}` → Use value from Step 2
- `${precheck_errors:sql_condition}` → Use value from Step 2
- `${userdata_errors:sql_condition}` → Use value from Step 2

### Step 5: Execute Query

```bash
source ~/.bashrc
curl -s -H "Authorization: Bearer \${DATASERVICE_GRAFANA_TOKEN}" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "queries": [{
      "refId": "A",
      "datasourceId": 37,
      "rawSql": "YOUR_SQL_WITH_VARIABLES_REPLACED",
      "format": "table"
    }],
    "from": "now-30d",
    "to": "now"
  }' \
  "https://grafana.oauth.prod.aws.tidbcloud.com/api/ds/query"
```

## Quick Query Examples

### Import Summary (Last 30 Days)

```bash
source ~/.bashrc
curl -s -H "Authorization: Bearer \${DATASERVICE_GRAFANA_TOKEN}" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "queries": [
      {
        "refId": "A",
        "datasourceId": 37,
        "rawSql": "SELECT count(1) as TotalJobs FROM imports WHERE created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY) AND server_tier=\"dedicated\" AND status!=\"canceled\" AND (tenant_id NOT IN (1372813089194211283, 1372813089187791289, 1372813089189381281, 1372813089208161328, 1372813089188691290))",
        "format": "table"
      },
      {
        "refId": "B",
        "datasourceId": 37,
        "rawSql": "SELECT count(1) as Completed FROM imports WHERE created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY) AND server_tier=\"dedicated\" AND status=\"completed\" AND (tenant_id NOT IN (1372813089194211283, 1372813089187791289, 1372813089189381281, 1372813089208161328, 1372813089188691290))",
        "format": "table"
      },
      {
        "refId": "C",
        "datasourceId": 37,
        "rawSql": "SELECT count(1) as Failed FROM imports WHERE created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY) AND server_tier=\"dedicated\" AND status=\"failed\" AND (tenant_id NOT IN (1372813089194211283, 1372813089187791289, 1372813089189381281, 1372813089208161328, 1372813089188691290))",
        "format": "table"
      }
    ],
    "from": "now-30d",
    "to": "now"
  }' \
  "https://grafana.oauth.prod.aws.tidbcloud.com/api/ds/query"
```

## Common Error Patterns

| Error Pattern | Category | Description |
|---------------|----------|-------------|
| `%PreCheckFailed%` | PreCheckFailed | Table not empty or schema mismatch |
| `%checksum%` | ChecksumMismatch | Data checksum mismatch |
| `%DiskSpace%` | DiskSpaceNotEnough | TiKV storage insufficient |
| `%cluster is deleted%` | ClusterDeleted | Target cluster was deleted |
| `%unknown columns%` | SchemaMismatch | Column definition mismatch |
| `%TLS handshake%` | NetworkError | Network connectivity issues |

## Notes

1. **Always use the workflow**: Get dashboard → get variables → replace → query
2. This workflow ensures queries stay aligned with the dashboard
3. All times are in UTC
4. Filter by `server_tier="dedicated"` for dedicated tier clusters
5. Datasource ID is 37 (Dataflow-Service-MySQL)