---
name: incidents
description: |
  Query and inspect TiDB Cloud incidents, safely prepare incident creation payloads,
  read timelines and postmortem actions, and locate the canonical severity, SLA,
  SOP, War Room, and postmortem references.
---

# TiDB Cloud Incidents

Use this sub-skill for Ops Portal incident records and incident-governance questions.
Read `references/incident-sources.md` before classifying severity, interpreting SLA,
or preparing a postmortem.

## Critical Safety Rules

1. Incident creation and updates are production mutations. **Never send a POST, PUT,
   or DELETE request until the full environment, endpoint, and JSON payload have been
   shown to the user and the user explicitly confirms execution.**
2. Do not use Production to test incident creation. Use `dev` or `staging`, and only
   after explicit confirmation.
3. `need_generate_feishu=true` creates an external side effect after the incident row
   is written. If message generation fails, the incident may already exist. Re-query
   before retrying to avoid a duplicate incident.
4. The DELETE route is not a reliable rollback: the backend handler currently does not
   implement deletion. Treat incident creation as non-reversible through this API.
5. Do not put customer secrets, credentials, raw logs containing sensitive data, or
   unredacted personal data in incident fields, Feishu messages, Jira links, or reports.

## Read Incidents

The shared wrapper adds `/api/v1/devops` and authentication automatically.

### List all incidents

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --env prod \
  --path /incidents
```

`GET /incidents` returns all incident records ordered by `created_at desc`. The current
backend does not expose server-side pagination or query filters, so filter locally with
`jq`.

### Get one incident

```bash
INCIDENT_ID=30030
skills/platform/devops-api/scripts/devops_http.sh \
  --env prod \
  --path "/incidents/${INCIDENT_ID}"
```

The normal Production UI link is:

```text
https://ops.tidbcloud.com/incidents/{incident_id}
```

### Useful filters

By ID:

```bash
skills/platform/devops-api/scripts/devops_http.sh --path /incidents |
jq '.items[] | select(.id == 30030)'
```

By tier and lifecycle status:

```bash
skills/platform/devops-api/scripts/devops_http.sh --path /incidents |
jq '[.items[] | select(.tier == "t1" and .status != "archived")]'
```

By component substring:

```bash
skills/platform/devops-api/scripts/devops_http.sh --path /incidents |
jq '[.items[] | select((.components // "") | test("Monitoring"; "i"))]'
```

By DevOps creation time:

```bash
START='2026-06-01T00:00:00Z'
END='2026-07-01T00:00:00Z'
skills/platform/devops-api/scripts/devops_http.sh --path /incidents |
jq --arg start "${START}" --arg end "${END}" \
  '[.items[] | select(.created_at >= $start and .created_at < $end)]'
```

For monthly operational reporting, default to `created_at` as the inclusion field and
state the timezone/window explicitly. Use `incident_start_time` only when the user asks
for incidents that began during the period. Keep `created_at`, `incident_start_time`,
and `incident_resolve_time` separate; they answer different questions.

## Field Interpretation

Common fields:

| Field | Meaning |
|---|---|
| `id` | Numeric incident ID, usually presented as `INC-{id}` |
| `name` | Incident title |
| `created_at` | DevOps record creation timestamp |
| `status` | Incident lifecycle status |
| `tier` | Internal incident severity (`t1`, `t2`, `t3`, or `x`) |
| `components` | Space-delimited affected component labels |
| `biz_regions` | Space-delimited business-region labels |
| `oncallers` | On-call participants |
| `leaders` | Incident leadership participants |
| `fbowners` | API owner list; do not infer a role name beyond the source field without confirmation |
| `feishu_url` | Linked Feishu incident or postmortem document |
| `jira_issues` / `jira_issue_links` | Linked Jira work |
| `description` | Incident description |
| `incident_start_time` | When impact began |
| `incident_resolve_time` | When impact ended or service recovered |
| `log_items` | Incident timeline/log entries returned by detail API |
| `postmortem_available` | Whether an imported DevOps postmortem is available |

Production lifecycle values observed at the time this sub-skill was authored include
`ongoing`, `post-mortem`, `under-improvement`, and `archived`. Before a mutation, read
current incidents and reuse a valid current value rather than inventing one.

Severity and SLA are different:

- Severity determines incident response and internal prioritization.
- SLA defines contractual service availability and possible service credits.
- An incident can be operationally important without an SLA breach, and an SLA review
  must use the current legal SLA rather than the severity label alone.

## Create an Incident

Endpoint:

```text
POST /api/v1/devops/incidents
```

### Required workflow

1. Confirm environment. Do not default a write to Production silently.
2. Read the current severity document and determine `tier` from observed impact.
3. Search recent incidents for duplicates by title, start time, components, and scope.
4. Confirm Incident Owner, Fire Master, affected components, regions, start time, and
   linked Feishu/Jira records.
5. Construct the full payload with `need_generate_feishu=false` by default.
6. Show the exact payload and side effects to the user.
7. Execute only after explicit user confirmation.
8. Read the returned `incident_id`, then GET the incident and verify every field.
9. If the POST returns an error after Feishu generation was requested, list/search
   incidents before retrying because the database row may already exist.

### Payload template

```json
{
  "name": "YYYYMMDD - concise customer-impact title",
  "status": "ongoing",
  "tier": "t3",
  "components": "TiDB Cluster-Manager",
  "biz_regions": "apac",
  "oncallers": ["oncaller@pingcap.com"],
  "leaders": ["incident.owner@pingcap.com"],
  "fbowners": [],
  "feishu_url": "https://pingcap.feishu.cn/wiki/REPLACE_ME",
  "jira_issues": "https://tidb.atlassian.net/browse/REPLACE_ME",
  "description": "Customer-visible impact, confirmed scope, and current facts only.",
  "incident_start_time": "2026-07-19T01:00:00Z",
  "incident_resolve_time": null,
  "need_generate_feishu": false
}
```

Write the payload to a file so quoting is reviewable:

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --env staging \
  --method POST \
  --path /incidents \
  --body-file /tmp/incident-create.json
```

The backend performs very limited creation validation. Invalid status, tier, owner,
component, or region values may be persisted, so validation and human confirmation are
mandatory.

Creating an incident record does not itself create a War Room. Follow the Incident SOP
and War Room workflow separately. `need_generate_feishu=true` only requests the incident
Feishu message path implemented by this API.

## Other Mutation Endpoints

All endpoints below require payload preview, environment confirmation, explicit user
approval, and a read-after-write check:

| Capability | Method and path | Important note |
|---|---|---|
| Update incident fields/lifecycle | `PUT /incidents/{incident_id}` | Read the current object first and show the complete intended state |
| Add/update incident log | `POST /incidents/{incident_id}/logs`; `PUT /incidents/{incident_id}/logs/{log_item_id}` | Changes the incident timeline |
| Generate/update Feishu message | `POST /incidents/{incident_id}/feishu_message` | Sends or changes communication; external side effect |
| Create Feishu task | `POST /incidents/{incident_id}/feishu_task` | Creates an assigned task in Feishu |
| Create/update postmortem | `POST /incidents/{incident_id}/postmortem`; `POST /incidents/{incident_id}/postmortem/{postmortem_id}` | Distinct from editing a linked Feishu postmortem document |
| Create/update postmortem timeline | `POST /incidents/{incident_id}/postmortem/timeline`; `POST /incidents/{incident_id}/postmortem/timeline/{timeline_item_id}` | Alters imported review evidence |
| Create/update/sync action | `POST /incidents/{incident_id}/postmortems/{postmortem_id}/todoactions[/...]` | May create or synchronize Feishu tasks |

Do not use the DELETE endpoints as a normal workflow. Some delete handlers are stubs or
may not provide a safe rollback.

## Incident Logs and Timeline

Read timeline entries:

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --path "/incidents/${INCIDENT_ID}/logs"
```

Read one entry:

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --path "/incidents/${INCIDENT_ID}/logs/${LOG_ITEM_ID}"
```

Log types in the backend include `text`, `postmortem`, `general`, and `milestone`.
Creating or updating logs is a mutation and requires the same payload-preview and
explicit-confirmation gate as incident creation.

## Postmortems

Read imported postmortem records:

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --path "/incidents/${INCIDENT_ID}/postmortem"
```

Read one imported postmortem:

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --path "/incidents/${INCIDENT_ID}/postmortem/${POSTMORTEM_ID}"
```

Imported postmortem fields cover summary, impact, detection time, resolution, root
causes, background, bug mechanism, lessons learned, timeline, and action items.

Many real incidents keep the postmortem in the incident's `feishu_url` and may return an
empty DevOps postmortem list. Always check both the API and linked Feishu document. Do not
conclude that no postmortem exists from an empty API response alone.

## Postmortem Actions

Read actions through the canonical action route when the postmortem ID is known:

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --path "/incidents/${INCIDENT_ID}/postmortems/${POSTMORTEM_ID}/todoactions"
```

A legacy read alias also exists:

```bash
skills/platform/devops-api/scripts/devops_http.sh \
  --path "/incidents/${INCIDENT_ID}/postmortem/todoaction"
```

Action statuses are `InProgress`, `Done`, or `Canceled`. An empty DevOps action list does
not prove that there are no follow-up actions: check the Feishu postmortem action table
and Jira links/labels as well.

When measuring completion:

- Use Jira `statusCategory`/resolution for Jira-tracked actions.
- Use DevOps action status for imported DevOps actions.
- Do not count a Feishu checkbox as completion without source-system evidence.
- Keep postmortem action-row count, Jira-tracked count, untracked count, and completion
  rate as separate metrics.

Creating, updating, deleting, or syncing actions can create Feishu tasks or alter task
state. Preview the full payload and require explicit confirmation.

## Incident Lifecycle Checklist

### Activation

- Confirm observed impact against the severity source.
- Identify Incident Owner and Fire Master.
- Create/activate the War Room according to the SOP.
- Record affected regions, components, tenants, start time, and customer impact.
- Acknowledge and respond within the SOP target.

### Facilitation

- Maintain factual timeline/milestones.
- Link alerts, cases, clusters, Jira work, Feishu group/doc, and decisions.
- Keep customer-facing updates consistent and redact sensitive data.
- Distinguish Incident Owner responsibility for internal resolution from Fire Master
  responsibility for customer/case communication and escalation flow.

### Closure

- Record resolved time and verified recovery evidence.
- Complete the postmortem within the SOP target.
- Create only validated follow-up actions; operational War Room tasks do not
  automatically become postmortem actions.
- Give each action a concrete owner, deadline, reason, and verification/test evidence.
- Review actions weekly until closure.

## Related Repository Skills

Use the correct adjacent skill rather than duplicating its workflow:

| Need | Skill | Boundary |
|---|---|---|
| Query/update Jira issues and postmortem actions | `skills/platform/jira-api/SKILL.md` | Jira is the execution/status source for Jira-tracked actions |
| Audit TCOC Jira issues misclassified as Incident | `skills/daily_work/tcoc-incident-review/SKILL.md` | This reviews Jira issue types; it does not create Ops Portal incident records |
| Draft a TCOCP-style support RCA | `skills/daily_work/tcocp-rca-writer/SKILL.md` | Use only when the user explicitly requests that RCA format; it is distinct from the canonical TiDB Cloud Incident Review Template |

## Source of Truth

Read `references/incident-sources.md` for canonical links, freshness guidance, and which
source to use for severity, SLA, process, War Room, postmortem structure, and backend API
behavior.