---
name: o11y-data-export
description: |
  Create, manage O11Y monitoring pipelines (integration tasks) for TiDB Cloud Clusters.
  Use when: creating monitoring pipelines, exporting slow queries, exporting raw logs, exporting SQL statements,
  setting up monitoring pipelines, accessing exported S3 data, data export to S3, pipeline management,
  or anything involving the monitoringPipelines API.
  Trigger phrases: "monitoring pipeline", "export slow query", "export raw log",
  "export sql statement", "data export", "s3 export", "create pipeline", "monitoringPipelines".
---

# O11Y Data Export (Monitoring Pipelines)

Export TiDB Cloud cluster data (slow queries, raw logs, SQL statements) to S3.

**Prerequisites**: Ask the user for these values before starting:

| Variable | Required | Description | Example |
|----------|----------|-------------|--------|
| `CLUSTER_ID` | Yes | TiDB Cloud cluster ID | `10413684027166477248` |
| `ENV` | Yes | Target environment for bizType resolution | `prod`, `dev`, `staging` |
| `DEPLOY_TYPE` | No | Cluster type (if unknown, use Strategy B to infer via clinic-api) | `dedicated`, `premium`, `resource_pool`, `starter`, `essential` |

Also ask for the export time range (`START_TIME`, `END_TIME`) if the user is diagnosing an incident.

## Setup

### Step 1: Authenticate

```bash
source ~/{SKILL_PATH}/o11y-auth/o11y-auth.sh
get_o11y_auth
```

### Step 2: Resolve cluster metadata

Choose one strategy based on what the user provided.

**Strategy A** -- User provided both `CLUSTER_ID` and `DEPLOY_TYPE`:

```bash
export CLUSTER_ID="10413684027166477248"
export ENV="prod"
export DEPLOY_TYPE="premium"
export BIZ_TYPE="$(resolve_biz_type "$DEPLOY_TYPE" "$ENV")"

source ~/{SKILL_PATH}/o11y-data-export/resolve-cluster.sh
resolve_cluster_meta
```

**Strategy B** -- User provided `CLUSTER_ID` + `ENV` but not `DEPLOY_TYPE` (use clinic-api to infer it, clinic token is needed for this):

```bash
source ~/{SKILL_PATH}/clinic-api/scripts/clinic-shell.sh
get_clinic_auth

export CLUSTER_ID="10413684027166477248"
export ENV="prod"
CLINIC_RESULT=$(clinic_search_clusters "$CLUSTER_ID")
export DEPLOY_TYPE=$(echo "$CLINIC_RESULT" | jq -r '.items[0].clusterDeployType')

if [ -z "$DEPLOY_TYPE" ] || [ "$DEPLOY_TYPE" = "null" ]; then
    echo "ERROR: Could not find cluster $CLUSTER_ID in Clinic" >&2
    exit 1
fi

export BIZ_TYPE="$(resolve_biz_type "$DEPLOY_TYPE" "$ENV")"

source ~/{SKILL_PATH}/o11y-data-export/resolve-cluster.sh
resolve_cluster_meta
```

### Step 3: Source the pipeline API

```bash
source ~/{SKILL_PATH}/o11y-data-export/pipeline-api.sh
```

After setup, these env vars are ready: `O11Y_TOKEN`, `O11Y_AUTH0_TOKEN`, `BIZ_TYPE`, `VENDOR`, `REGION`, `ORG_ID`, `GS_URL`, `CLUSTER_ID`, plus AWS credentials for S3 access.

If `resolve_cluster_meta` fails, check stderr -- common causes are an expired `O11Y_AUTH0_TOKEN` (re-run `get_o11y_auth`) or invalid `BIZ_TYPE` for the given cluster.

## Creating Exports

```bash
pipeline_create slowquery "$START_TIME" "$END_TIME"
pipeline_create rawlog "$START_TIME" "$END_TIME" --components tidb,tikv,pd
pipeline_create sqlstmt "$START_TIME" "$END_TIME"
```

`--components` is **required** for rawlog exports. Ask the user which components to include (any combination of `tidb`, `tikv`, `pd`). Defaults to all three if omitted.

Other optional arguments:

```bash
pipeline_create rawlog "$START_TIME" --components tidb,tikv
pipeline_create slowquery "$START_TIME" "$END_TIME" --name my-export --prefix custom/path
```

| Option | Default | Description |
|--------|---------|-------------|
| `--name` | `<dataType>-export` | Display name for the task |
| `--prefix` | `exports/<CLUSTER_ID>/<dataType>` | S3 key prefix |
| `--components` | `tidb,tikv,pd` | **Required for rawlog.** Components to export |

Only one task per `dataType` per cluster is allowed. Delete the existing one first if you need to recreate.

Response includes a task `id` and the S3 `destinationInfo.accessPoint` (bucket name, unique per GitHub user).

## Managing Exports

```bash
pipeline_list                      # all tasks for the cluster
pipeline_list slowquery            # filter by dataType

pipeline_get "$TASK_ID"            # get task details and phase

pipeline_wait "$TASK_ID"           # poll until ACTIVE (default 300s timeout)
pipeline_wait "$TASK_ID" 600       # custom timeout

pipeline_delete "$TASK_ID"         # remove a task after task completed
```

The `phase` field in task responses indicates readiness. Wait for `ACTIVE` before expecting data in S3.

## Accessing Exported Data

Data lands in the S3 bucket from the create response. Use the AWS credentials from `o11y-auth`.

```bash
aws s3 ls "s3://<bucket_name>/exports/${CLUSTER_ID}/slowquery/"

aws s3 cp "s3://<bucket_name>/exports/${CLUSTER_ID}/slowquery/" ./local-dir/ --recursive
```

For rawlog, files are partitioned by component and hour: `<prefix>/<component>/<hour_partition>/...`.

## Troubleshooting

| Issue | Fix |
|-------|-----|
| 401 Unauthorized | Re-authenticate via `o11y-auth` skill |
| 404 Not Found | Check `BIZ_TYPE` and `CLUSTER_ID` are correct, **never** try to list all clusters |
| Duplicate task error | Delete existing task for that dataType first |
| Empty S3 bucket | Wait for task to become ACTIVE first |
| S3 access denied | STS credentials expired -- re-authenticate via `o11y-auth` |
| Clinic cluster not found | Verify CLUSTER_ID is correct; cluster may not be indexed in Clinic yet |