---
name: tcocp-rca-writer
description: |
  Use ONLY when a TCOCP-styled RCA (root cause analysis) report is explicitly requested. Trigger phrase: "Write an RCA for TCOCP".
---

# TCOCP-styled RCA Writer

Use this skill to draft a RCA (root cause analysis) report for the TCOCP (TCOC-Postmortem) process.

## Workflow

### Step 1 - Verify tickets are loaded

Ensure the user has mentioned a Jira TCOC/GTOC ticket. We call this the "TCOC ticket number".

If no TCOC ticket number is found, halt and ask the user for a TCOC ticket number:

> No TCOC ticket found. Please add a TCOC ticket as source.

The user may have also mentioned a Jira TCID/ST/O11Y/NAID/APID/JPID/EMID ticket. We call this the "ST ticket number". If the user has not mentioned an ST ticket number, use the `customfield_10152` field of the TCOC ticket. If it is still missing, leave the ST ticket number empty.

The user may have also added a Lark/Feishu chat group ID, in the form "oc_xxx". If the "Lark Im" skill exists, use it to load the chat messages to as INTERNAL references to polish the event timeline. Ignore this if there is no "Lark Im" skill.

### Step 2 - Fill in the template

Open [`assets/template.md`](assets/template.md) and fill in the placeholders (items like `{this}`) with information from all added tickets.

#### General guideline

You MUST NOT change any text outside the placeholders `{***}`.

DO NOT include any cluster ID, tenant ID, changefeed ID etc. If multiple clusters are involved, distinguish them textually if possible (e.g. "upstream cluster" and "downstream cluster").

RCA documents are written for technical staffs, and technical terms such as "resolvedTS lag" should be preserved. No need to rephrase them into layman language.

#### "Summary"

All "`{*** summary}`" placeholders should be concise, preferrably within one sentence.

Remember to use the SAME `trigger timestamp` inside all `{approximated duration(trigger timestamp → *** timestamp)}` calculation.

Do not add the "~" symbol before the approximated duration.

The approximated duration should be rounded to:
* nearest 0.5 days, for >24 hours
* nearest 0.5 hours, for 1-24 hours
* nearest 5 minutes, for <1 hour

| Placeholder | Guideline |
|-------------|-----------|
| `{severity}` | S1/S2/S3/S4. |
| `{impact scope}` | Affected clusters / projects / tenants / number of users. Remember to exclude cluster ID / project ID etc. |
| `{user impact summary}` | Should paraphrase how the customer describes the problem. |
| `{trigger timestamp}` | The time that the problem first appeared, e.g. execution returns error continuously, replication progress stopped, CPU usage rised above threshold, etc. |
| `{detection timestamp}` | The time which the problem is first detected by the customer or our O11Y system. |
| `{diagnosis timestamp}` | The time which we have finished analysis, and found either the root cause or workable mitigation. |
| `{recovery timestamp}`  | The time which the problem is subdued, e.g. execution reported normal result, replication resumed and caught up, CPU drops below threshold, etc. |


#### "Timeline"

All "`{ ... *** timeline ... }`" placeholders should be paragraphs written in this form:

~~~~
*{timestamp}*: {description of event}

*{timestamp}*: {description of event}

...
~~~~

Timeline events should be precise and detailed, except:

* Ignore events irrelevant to the problem. e.g. ❌ DO NOT WRITE "Lark war-room chat created".
* No need to emphasize in which channel a person made a comment.
* Jira comments made by "Marian Zhang", "Charles Zhu" or "Peter Zhu" are usually bots, especially those in the form "notified (someone) by lark". Treat these as bot comments and ignore them.

Timestamps should be written in the ISO-8601 form like `2006-01-02 15:04:05 MST`. Use the customer's time zone if possible, otherwise default to UTC+8.

Multiple events separated less than 1 minute should be coalesced.

Avoid including future conclusions into the timeline event. For example, even if analysis determined a cluster becomes unresponsive because of insufficient resource, ❌ DO NOT WRITE *"...: The cluster became unresponsive due to insufficient resource"*. Instead, ✅ JUST WRITE *"...: The cluster became unresponsive."*.

| Placeholder | Guideline |
|-------------|-----------|
| `{ ... trigger timeline ... }` | Events leading to the problem before detection. This section should contain the "trigger timestamp". |
| `{ ... detection timeline ... }` | Events after the problem has been detected until support team starts diagnosis. The *first* event must be at "detection timestamp". |
| `{ ... diagnosis timeline ... }` | Events after diagnosis begins until the root cause is identified or a working mitigation is planned. The *last* event must be at "diagnosis timestamp". |
| `{ ... mitigation timeline ... }` | Events about how the mitigation is implemented. This section is intended to be distilled into an SOP in the future. |
| `{ ... recovery timeline ... }` | Events about how the system has recovered from the problem, until the support team declared the problem resolved. The *first* event must be at "recovery timestamp". |
| `{ ... postmortem timeline ... }` | Any additional events after recovery, e.g. bug fix PR subsmissions. |

### "Root Cause Analysis"

All paragraphs "`{ ... *** ... }`" SHOULD NOT contain further subheaders.

For `{... analysis steps ...}` and `{... conclusion ...}`, write in plain paragraphs, and DO NOT use bullet points or numbered lists. This makes the RCA section more natural to readers.

❌ DO NOT WRITE:

```md
#### Layer 1: Query is Unresponsive

Key findings:
1. The server only has 2 CPU cores.
2. At 12:34, the customer started the heavy workload.
3. This saturated the CPU and stalled other operations.
```

✅ WRITE:

```md
The root cause of the query being unresponsive was that the server only has 2 CPU cores.

At 12:34, the customer started the heavy workload. This saturated the CPU and stalled other operations.
```

| Placeholder | Guideline |
|-------------|-----------|
| `{... reproduction steps ...}` | Step-by-step reproduction of the problem. Skip if no reproduction steps are written in the tickets. |
| `{... analysis steps ...}` | Details *how* the root cause is found. Include how the support team used information from metrics, logs, profiling, and monitoring. Any "wrong turns" during the analysis should also be documented. |
| `{... conclusion ...}` | Describes the final root cause. No need to include the mitigation or solution here. |

#### "Solution"

| Placeholder | Guideline |
|-------------|-----------|
| `{mitigation summary}` | The immediate workaround. |
| `{optional: alternative mitigation summary}` | Any other workaround that is applicable, which may be inferior to the main workaround in terms of speed or robustness, but should still be considered if the main workaround is not feasible. Leave this blank if such alternative does not exist, or is equivalent to the "mitigation summary" above. |
| `{solution summary}` | The long-term fix which prevents the problem entirely, e.g. code changes, product improvements, or design fixes. |

### Step 3 - Generate the report

After filling in the template, write the resulting document into the file `output/RCA_{TCOC ticket number}.md`, and then say:

```
RCA is generated at `output/RCA_{TCOC ticket number}.md`.
```

No need to summarize the content in the chat. No need to report how you have succesfully used the skill.

Only if you cannot write the file into the filesystem (e.g. sandbox restriction), copy the content directly in the chat.