---
name: alertstore-api
description: |
  Query and manage alerts in alertstore — the single source of truth for TiDB Cloud
  operational alerts. Use this skill whenever the user asks about alerts, alert
  lifecycle (ack, resolve, assign), alert subscriptions, mute rules, or alertstore
  events. Also use when the user mentions alertstore, alert state,
  or wants to look up an alert by ID (ALT-* format). Covers all three environments
  (prod, staging, dev).
---

Alertstore is the SSOT for TiDB Cloud operational alerts. It receives from
Alertmanager, classifies, routes, and tracks the full lifecycle of each alert.

This skill talks to alertstore through the shared devops gateway, which means
authentication and environment selection work exactly like the devops-api skill.

# Ownership and Jira

Alertstore stores **all** alerts, but not all alerts are managed here. Each
alert has an `attrs.owned` field (boolean, frozen at creation):

- **`owned: true`** — alertstore runs the full lifecycle. Ack, resolve, assign,
  and comment through this API. Alertstore mirrors the issue to Jira for
  compatibility, but alertstore is the source of truth.
- **`owned: false`** — a shadow copy of a Jira-managed alert. The alert is
  stored and queryable, but lifecycle actions (ack, resolve, assign) should be
  done in **Jira**, not here. Alertstore records the data for visibility but
  does not drive notifications or ownership for these alerts.

When investigating an alert, check `attrs.owned` first to know which system to
act in.

# Authentication

Alertstore shares the devops gateway. The same `DEVOPS_API_TOKEN` is used.
Set it in `.env` or as an environment variable before calling the script.

# Environments

| Environment | `--env` value | Gateway |
|---|---|---|
| Production | `prod` (default) | `https://prod.devops.pingcap.com` |
| Staging | `staging` | `https://staging.devops.pingcap.com` |
| Dev | `dev` | `https://dev.devops.pingcap.com` |

If the user does not specify an environment, default to **prod**.

# API Access

Send requests via the script:

```bash
scripts/alertstore_http.sh --path <path> [--env prod|staging|dev] \
  [--method METHOD] [--query "k=v"] [--body '{}']
```

The `--path` is relative to `/api/v1/alertstore`. For example `--path /alerts`
hits `/api/v1/alertstore/alerts`. See the script's `--help` for all options.

# API Reference

## Alerts

### List alerts

```
GET /alerts
```

Query parameters:

| Param | Description |
|---|---|
| `state` | `open`, `acked`, `resolved` |
| `classification` | `page`, `ticket`, `logging` |
| `region` | Cloud region, e.g. `aws-us-west-2` |
| `biz_region` | Sales region: NA, EMEA, APAC |
| `tenant_kind` | `internal` or `external` |
| `customer_type` | SKA, KA, SMB, UNKNOWN |
| `assigned_to` | Assignee email |
| `cluster_id` | Cluster ID |
| `tenant_id` | Tenant ID |
| `limit` | Page size (default 50) |
| `offset` | Pagination offset (default 0) |

Results are ordered newest-first by `created_at`. No total count is returned;
page with `offset` until the response is empty.

Each item is a flat object with top-level fields (`id`, `state`, `alertname`,
`classification`, `region`, `tenant_id`, `cluster_id`, `assigned_to`,
`created_at`, `updated_at`, …) plus an `attrs` object containing the full
label set and enriched fields.

Example:

```bash
./scripts/alertstore_http.sh --path /alerts \
  --query "state=open&classification=page&limit=10"
```

### Get alert detail

```
GET /alerts/:id
```

Returns a single `AlertDetail` object:

| Field | Description |
|---|---|
| `id`, `state`, `alertname`, … | Same top-level fields as list |
| `attrs` | Full label set + enriched fields |
| `am_status` | Current Alertmanager status: `firing`, `resolved`, or `unknown` |
| `instance_counts` | `{firing, resolved, unknown}` — per-instance breakdown |
| `instances[]` | Each instance with its observations (events) |
| `audit[]` | System audit log (state changes, IM topic creation, AM status updates) |
| `common_facts` | Label-derived header fields resolved across all instances |
| `common_labels` | Labels every instance agrees on |
| `external_issue_url` | Jira issue link (if mirrored) |
| `receipts[]` | Delivery receipts |

Example:

```bash
./scripts/alertstore_http.sh --path /alerts/ALT-2608-1742
```

### Get alert timeline

```
GET /alerts/:id/timeline
```

Returns timeline entries (comments, state changes, IM messages) for an alert.
Returns `null` when the alert has no human-initiated timeline entries — this is
normal for alerts nobody has interacted with yet. System events (AM status
changes, IM topic creation) appear in the `audit` field of the alert detail
instead.

```bash
./scripts/alertstore_http.sh --path /alerts/ALT-2608-1742/timeline
```

### Acknowledge an alert

```
POST /alerts/:id/ack
```

Body: `{"user": "email@pingcap.com"}`

Returns 409 Conflict if the alert is not in a state that allows acking.

```bash
./scripts/alertstore_http.sh --method POST --path /alerts/ALT-2608-1742/ack \
  --body '{"user":"someone@pingcap.com"}'
```

### Resolve an alert

```
POST /alerts/:id/resolve
```

Body:

| Field | Required | Description |
|---|---|---|
| `user` | yes | Who is resolving |
| `resolution` | yes | `handled`, `self_recovered`, or `noise` |
| `note` | no | Free-text explanation |

Resolution meanings:
- **handled** — a human intervened (fixed, mitigated, or notified the customer)
- **self_recovered** — the problem went away on its own
- **noise** — false positive, expected activity, or duplicate

```bash
./scripts/alertstore_http.sh --method POST --path /alerts/ALT-2608-1742/resolve \
  --body '{"user":"someone@pingcap.com","resolution":"handled","note":"scaled up the node"}'
```

### Assign an alert

```
POST /alerts/:id/assign
```

Body:

| Field | Required | Description |
|---|---|---|
| `user` | yes | Who is assigning |
| `assignee` | yes | New owner email |
| `reason` | no | Why the reassignment |

```bash
./scripts/alertstore_http.sh --method POST --path /alerts/ALT-2608-1742/assign \
  --body '{"user":"me@pingcap.com","assignee":"oncall@pingcap.com","reason":"escalating to oncall"}'
```

### Comment on an alert

```
POST /alerts/:id/comment
```

Body: `{"user": "email@pingcap.com", "text": "comment content"}`

The comment is written to the timeline (SSOT) and mirrored to the IM topic.

```bash
./scripts/alertstore_http.sh --method POST --path /alerts/ALT-2608-1742/comment \
  --body '{"user":"me@pingcap.com","text":"checked the cluster, CPU is back to normal"}'
```

## Events

Events are the raw Alertmanager webhook payloads — every firing and resolved
notification, regardless of classification. Alerts (the lifecycle records) are
created from events; one alert may have many events.

### List events

```
GET /events
```

Query parameters:

| Param | Description |
|---|---|
| `alertname` | Alert rule name |
| `classification` | `page`, `ticket`, `logging` |
| `cluster_id` | Cluster ID |
| `tenant_id` | Tenant ID |
| `region` | Cloud region |
| `biz_region` | Sales region |
| `tenant_kind` | `internal` or `external` |
| `customer_type` | SKA, KA, SMB, UNKNOWN |
| `status` | Alertmanager status: `firing` or `resolved` |
| `limit` | Page size (default 50) |
| `offset` | Pagination offset (default 0) |

Results are ordered newest-first. Each event contains `id`, `status` (`firing`
or `resolved`), `fingerprint`, and an `attrs` object with the full label set.
The `observation_type` field distinguishes the Alertmanager signal (`firing` /
`resolved`); `instance_id` ties the event back to the alert's instance.

```bash
./scripts/alertstore_http.sh --path /events \
  --query "alertname=TiKV_disk_almost_full&status=firing&limit=20"
```

### Get event

```
GET /events/:id
```

```bash
./scripts/alertstore_http.sh --path /events/12345
```

## Subscriptions

> These are configuration operations best done in the alertstore Web UI where you
> can preview channels and filters interactively. The API is documented here for
> scripting and automation use cases.

Subscriptions route alert notifications to IM channels. Each subscription has a
mode and a filter that selects which alerts it receives.

### List subscriptions

```
GET /subscriptions
```

```bash
./scripts/alertstore_http.sh --path /subscriptions
```

### List available channels

```
GET /channels
```

Returns IM groups the alertstore bot belongs to — the set of valid targets.

```bash
./scripts/alertstore_http.sh --path /channels
```

### Create subscription

```
POST /subscriptions
```

Body:

| Field | Required | Description |
|---|---|---|
| `channel_id` | yes | Lark group ID |
| `channel_name` | no | Display name (resolved server-side) |
| `mode` | yes | Delivery mode |
| `filter` | yes | JSON filter object |
| `created_by` | no | Creator email |

```bash
./scripts/alertstore_http.sh --method POST --path /subscriptions \
  --body '{"channel_id":"oc_xxx","mode":"topic_forward","filter":{"region":"aws-us-west-2"}}'
```

### Update subscription

```
PUT /subscriptions/:id
```

Body: `{"mode": "...", "filter": {...}}`

### Delete subscription

```
DELETE /subscriptions/:id
```

```bash
./scripts/alertstore_http.sh --method DELETE --path /subscriptions/42
```

## Mute Rules

> Like subscriptions, mute rules are best managed in the alertstore Web UI where
> you can see active rules and their remaining duration at a glance. Use the API
> for scripting or automated maintenance windows.

Mute rules suppress delivery for alerts matching a filter, for a specified
duration. The alert is still recorded — only notifications are suppressed.

### List active mute rules

```
GET /mute-rules
```

```bash
./scripts/alertstore_http.sh --path /mute-rules
```

### Create mute rule

```
POST /mute-rules
```

Body:

| Field | Required | Description |
|---|---|---|
| `filter` | yes | JSON filter (same schema as subscription filters) |
| `duration` | yes | Go duration string, e.g. `2h`, `30m` |
| `reason` | no | Why the mute |
| `created_by` | no | Creator email |

```bash
./scripts/alertstore_http.sh --method POST --path /mute-rules \
  --body '{"filter":{"alertname":"TiKV_disk_almost_full","region":"aws-us-west-2"},"duration":"2h","reason":"planned maintenance"}'
```

### Delete mute rule

```
DELETE /mute-rules/:id
```

```bash
./scripts/alertstore_http.sh --method DELETE --path /mute-rules/7
```

# Quick Reference

## Alert states

`open` → `acked` → `resolved`

## Classifications

| Value | Meaning |
|---|---|
| `page` | Requires immediate human response |
| `ticket` | Requires response within SLA |
| `logging` | Recorded only, no notification |

## Resolutions

| Value | Meaning |
|---|---|
| `handled` | Human intervened |
| `self_recovered` | Resolved on its own |
| `noise` | Should not have fired |

## Alert ID format

`ALT-<weeknum>-<seq>`, e.g. `ALT-2608-1742`.