# Operation

*Note*: The following only applies to production environment

## Accessing k8s

Refer to https://pingcap.feishu.cn/wiki/XxiTwrtLAipB45kqcepc1RpOnsf

## Alert rules runbook

The alert rules runbook is managed via markdown style text under [docs folder](./docs), it will be synced to the [wiki repo](https://github.com/tidbcloud/runbooks/wiki) via github workflow.

## Grafana dashboards & Alert rules

For adding new alert rules, please first check out the how-to docs: https://pingcap.feishu.cn/wiki/wikcnPd8Xw2uyvCf0LwvU8wu24d

For Dev: you can exec `make all` format and lint all `grafana json`, `prometheus rules` and `pulumi code`.

> **Note**: the grafana dashboards and alert rules are applied to dev/stg/prod at the same time. So please make sure the alert rules are tested manually before merge to master

## Develop Pulumi Code

Setup local development environment

```bash
unset KUBECONFIG
export TEST_MODE=true
export PULUMI_CONFIG_PASSPHRASE=wearepingcap
pulumi login --local
pulumi stack select --create --secrets-provider=passphrase runbooks-dev
pulumi up
```

`make format`: will format all pulumi typescript code

`make check-pulumi`: will lint all pulumi typescript code

## Add New Alert Rules

`make base`: generate all yaml(PrometheusRules) file in base cluster

`make seed`: generate all yaml(PrometheusRules) file in seed cluster

`make shoot`: generate all yaml(ConfigMap) file in shoot cluster

All alert rules are in the `rules` directory. You can run `make check-rules` for linting.

**Note**: When adding new alert rules, please make sure adding a corresponding runbook in the [Wiki](https://github.com/tidbcloud/runbooks/wiki).

**Approval**: Every new alert rule must be approved by both the on-call team that owns it and the Support Enablement Engineering Team before it can be merged.

Best practices:
- make sure every alert rule has a `component` label to indicate which physical component it belongs to
- [write useful prometheus rules (Chinese)](https://docs.google.com/document/d/1hNJkkubU81aXhLKdBisgjBU5Tv-8Qll0lLPCQXnYTMQ/edit)
- [Awesome Prometheus alerts](https://awesome-prometheus-alerts.grep.to/)
- [rate-then-sum-never-sum-then-rate](https://www.robustperception.io/rate-then-sum-never-sum-then-rate/)

## Alert Routing Labels

Pick a label based on who should handle the alert:

- **A specific person** — whether a permanent owner or a temporary override → set `assign` to that person's email.
- **An on-call rotation** → set `oncall_team` to one of the teams in the [Team List](#team-list) below.

### `oncall_team`

Every alert rule handled by a rotation should carry an `oncall_team` label identifying the team that owns it.

```yaml
labels:
  severity: critical
  component: tikv
  oncall_team: resilience   # routes the alert to this team's on-call
```

**How routing works**: jiraport routes each alert to the owning team's on-call based on its `oncall_team` label. This has replaced the legacy `assign.yml` routing (component + tier + stability_governance matching), so `oncall_team` is now the primary mechanism — make sure every rotation-handled alert carries it.

> **Need a team that isn't in the [Team List](#team-list)?** Contact the Support Enablement Engineering Team to have it added. Adding a team or changing the routing setup requires their review.

### `assign`

Use the `assign` label to assign an alert directly to a specific person, bypassing team routing. The value must be a company email address.

```yaml
labels:
  severity: major
  component: aipower
  assign: letian.wang@pingcap.com   # bypasses team routing, assigns directly
```

When `assign` contains an `@` sign, jiraport assigns the alert to that person without consulting the assign rules. Use this for alerts that a specific individual always handles regardless of on-call rotation, or as a temporary override to redirect an alert to a particular person.

### Team List

| `oncall_team` | Description |
|---|---|
| [`resilience`](https://oncall.pingcap.net/oncall/team/resilience) | Dedicated tier data-plane alerts (tikv/tiflash/tidb/pd without kernel governance) |
| [`dedicated_service`](https://oncall.pingcap.net/oncall/team/dedicated_service) | Infra and management components (k8s, etcd, cert-manager, infra-provider, etc.) |
| [`global_control`](https://oncall.pingcap.net/oncall/team/global_control) | API gateway, dataservice, tidb-management-service |
| [`premium`](https://oncall.pingcap.net/oncall/team/premium) | Next-gen cluster service, dataplane manager, next-gen infra |
| [`serverless`](https://oncall.pingcap.net/oncall/team/serverless) | Serverless tier general alerts |
| [`tidb_x_tools`](https://oncall.pingcap.net/oncall/team/tidb_x_tools) | TiCDC, BR, DM/import/export alerts for Starter/Essential/Premium tiers |
| [`o11y`](https://oncall.pingcap.net/oncall/team/o11y) | Monitoring infrastructure (alertmanager, vmagent, metering, loki) |
| [`tikv_unified`](https://oncall.pingcap.net/oncall/team/tikv_unified) | TiKV kernel governance alerts |
| [`tiflash`](https://oncall.pingcap.net/oncall/team/tiflash) | TiFlash kernel alerts |
| [`pd`](https://oncall.pingcap.net/oncall/team/pd) | PD kernel alerts |
| [`tidb_executor`](https://oncall.pingcap.net/oncall/team/tidb_executor) | TiDB executor / compute kernel alerts |
| [`tidb_optimizer`](https://oncall.pingcap.net/oncall/team/tidb_optimizer) | TiDB optimizer kernel alerts |
| [`tidb_sql`](https://oncall.pingcap.net/oncall/team/tidb_sql) | TiDB SQL / DDL kernel alerts |
| [`ingress_service`](https://oncall.pingcap.net/oncall/team/ingress_service) | DM, import/export, Dumpling, TiDB Binlog, sync-diff-inspector, Lightning |
| [`tidb_txn`](https://oncall.pingcap.net/oncall/team/tidb_txn) | TiDB transaction kernel alerts |
| [`tiproxy`](https://oncall.pingcap.net/oncall/team/tiproxy) | TiProxy alerts across all tiers |
| [`ticdc`](https://oncall.pingcap.net/oncall/team/ticdc) | TiCDC kernel alerts |
| [`br`](https://oncall.pingcap.net/oncall/team/br) | BR kernel alerts |
| [`operator`](https://oncall.pingcap.net/oncall/team/operator) | TiDB operator, and premium dataplane controller |
| [`billing`](https://oncall.pingcap.net/oncall/team/billing) | Billing service |
| [`account`](https://oncall.pingcap.net/oncall/team/account) | Account service |
| [`clinic`](https://oncall.pingcap.net/oncall/team/clinic) | Clinic and ng-monitoring |
| [`security`](https://oncall.pingcap.net/oncall/team/security) | Security alerts |

### Nighttime and Special Routing

Nighttime on-call, summary-based routing, and customer/tenant-specific rules are managed separately in jiraport. These special rules take priority over `oncall_team` routing. To add or modify nighttime or tenant-specific/special routing, contact the Support Enablement Engineering Team.

## Add New Grafana Dashboard

Contact Clinic Team