# Incident Sources of Truth Use these sources by purpose. Do not replace one source with another: incident severity, contractual SLA, operational process, War Room mechanics, and postmortem structure answer different questions. Links and document contents were verified on 2026-07-19. Re-read the current source at the time of use because legal commitments and internal process can change. ## Canonical Sources | Purpose | Source | Usage | |---|---|---| | Incident severity and classification | [TiDB Cloud Incident Severity Levels](https://pingcap.feishu.cn/wiki/OTu2wDVUBiHscpkT2k3cabgCnOf) | Internal T1/T2/T3/X definitions, category thresholds, and severity-adjustment rule | | Incident operating process | [TiDBCloud Incident SOP](https://pingcap.feishu.cn/wiki/AymbwF9odi2OWGksK9qcLdD8ncd) | Roles, activation, War Room, facilitation, closure, postmortem, and action follow-up | | War Room operations | [Incident War Room](https://pingcap.feishu.cn/wiki/OwCDwUWE3iMXMVk1rDjcqrQfnAd) | Workspace creation, check-in, role assignment, collaboration, links, and task handling | | Postmortem structure | [Incident Review Template](https://pingcap.feishu.cn/wiki/YkFrwgQuQisosFkYPv9cA3fjnYe) | Required review fields and sections for impact, timeline, RCA, lessons, and actions | | Customer-facing contractual SLA | [Service Level Agreement for TiDB Cloud Services](https://www.pingcap.com/legal/service-level-agreement-for-tidb-cloud-services/) | Current legal uptime commitment, service credits, claim requirements, exclusions, and effective date | | Historical SLA versions | [SLA archive](https://www.pingcap.com/legal/service-level-agreement-for-tidb-cloud-services/archive/) | Determine which legal SLA version applied at the incident/customer contract date | | Backend API behavior | `tidbcloud/devops`: `api/incident/incident_api.go`, `pkg/controller/incident/incident_controller.go`, `pkg/service/incident/incident_service.go`, `entity/incident.go` | Endpoint registration, request/response fields, runtime behavior, and side effects | ## How to Choose a Source ### Severity Use the severity document to answer: - Does observed impact exceed the internal incident threshold? - Is the incident T1, T2, T3, or X? - Does the PingCAP-side human-error/responsibility adjustment apply? - Which category applies: availability, modification, performance, billing, WebConsole, API, DM/import, ChangeFeed, metrics, or another documented category? Do not derive severity only from the customer's support plan or SLA percentage. ### SLA Use the current legal page and, when necessary, its archive to answer: - Which service types and production workloads are covered? - What monthly uptime commitment applied? - Does the event qualify as Downtime under the contract? - Is a service credit potentially available? - What claim timing, evidence, and redaction requirements apply? - Does an exclusion such as scheduled maintenance, third-party failure, customer action, overload, or beta functionality apply? The legal page is authoritative. Do not quote cached percentages without checking the page's effective date and the contract period. As of the page verified on 2026-07-19, the displayed version was last updated 2026-06-30. The SLA requires sensitive/confidential information in claim logs to be removed or replaced with asterisks. Apply the same redaction principle to incident summaries and assistant output. ### SOP and Roles Use the SOP for response process and accountability. Key principles in the current SOP: - Acknowledge/respond within 15 minutes. - Complete postmortems within three business days. - Review follow-up actions weekly until closure. - Follow-up actions take priority over regular feature work. - Actions require solid test evidence. Role boundary: - Incident Owner drives internal incident resolution and the whole incident lifecycle. - Fire Master drives customer/case communication, frontline alignment, and escalation flow while synchronizing with the Incident Owner. - Global Support handles customer tickets and escalates cross-customer/incident signals. An urgent case is not automatically an incident. If no Incident Owner/incident has been activated, the Fire Master still owns resolution push, case/customer communication, and R&D escalation. Once an incident is active, the Incident Owner leads internal multi-team resolution while the Fire Master covers the customer/case side. ### War Room Use the War Room guide for tool-level collaboration. Creating an Ops Portal incident via `POST /incidents` is not equivalent to creating a War Room. Follow the SOP and War Room workflow explicitly. War Room operational tasks are used during live response. They do not automatically become postmortem actions; validate long-term actions during the review. ### Postmortem Use the template for structure: - Header and incident timing - Summary - Customer/technical impact - Timeline and time to detect - Resolution - Root cause, background, and bug mechanism - What worked / did not work / where the team was lucky - Top issues - Action items with priority, type, deadline, status, Jira, item, and assignee - Appendix and recording Do not reduce the review to an action dump. Assess recurrence control, long-term target state, residual risk, verification evidence, and management help required. ## Supplementary Sources Requiring Confirmation The Wiki also contains older or explicitly work-in-progress pages: | Source | Caveat | |---|---| | [TiDBCloud Incident Notification WIP](https://pingcap.feishu.cn/wiki/JxUowWW1ziIlCtkjQ0Nc9547noe) | Marked WIP and not recently maintained; do not treat as current policy without owner confirmation | | [Status Page - Incident External Communication](https://pingcap.feishu.cn/wiki/ALg4wgIqTiE2cgkYxl9cE7CankJ) | Older operational guidance; verify against the current SOP and communication owner before use | ## Runtime API Caveats - `GET /incidents` currently returns the full list without server-side pagination/filter parameters; use local filtering. - Runtime Production status values can differ from old source-code constants. Read current records before writing. - Incident creation validation is weak; payload quality is the caller's responsibility. - `need_generate_feishu=true` writes the incident first and then attempts Feishu message generation. A failure response can therefore represent partial success. - Incident DELETE is not implemented as a reliable rollback. - DevOps postmortem/action endpoints may be empty while a Feishu postmortem and Jira actions exist. Check all linked systems before reporting that evidence is absent.