# Alertstore Design Document

## Background

Current alert pipeline spans four repos with tangled responsibilities:

- **jiralert**: Alertmanager webhook → Jira issue creation, dedup, precheck/validate, clinic enrichment
- **jiraport**: Jira webhook → notification state machine, 12 action commands, Lark/assign/escalate
- **feishu_messager_v2**: Lark card delivery + phone calls (Ali/Twilio), read receipt → ack callback
- External AI services: triggered by Jira webhook, reads Jira API

Problems:
- Jira is both data source and notification trigger — rate-limited, expensive, tightly coupled
- Severity inflation — too many Critical/Major alerts that need no action
- Alert context scattered across Jira, multiple Lark groups, on-caller memory
- Mute/dedup logic spread across jiralert, jiraport, and alertmanager
- On-call handoff loses context between groups

## Principles

- A. Every Page/Ticket must genuinely require human response
- B. Every alert must have a clear, executable SOP
- C. Alert actions prioritize avoiding user impact, then preventing system degradation
- D. High-frequency alerts should be automated
- E. Alerts with >80% false positive rate must be reworked or removed
- F. Alertmanager output is not all "alerts" — classify before routing
- G. Every alert must have a clear responder and rule owner
- H. POC/test clusters differ only in delivery, not in storage or generation

## Architecture

```
Alertmanager ──webhook──→ alertstore (SSOT)
                            │
                            ├── event store (raw, all alerts)
                            │
                            ├── classifier
                            │     classification label from alert rule (page/ticket/logging)
                            │     + override: non-KA customer → logging
                            │     + override: recent clinic activity → downgrade
                            │
                            ├── alert lifecycle (state machine)
                            │     open → acked → resolved
                            │
                            ├── delivery layer
                            │     ├── im_topic     post topic to main group
                            │     ├── im_forward   forward to subscribed groups (topic or card)
                            │     ├── phone_call   via feishu_messager (phase 1)
                            │     └── jira_*       create/comment/transition the mirrored issue
                            │                      (also triggers existing Jira webhook → external AI services)
                            │
                            ├── ack ingress (3 channels)
                            │     ├── alertstore web UI
                            │     ├── Jira webhook (transitional)
                            │     └── Lark bot /ack
                            │
                            ├── subscription management
                            │     groups self-select: topic_forward / card_only / event / customer
                            │     filter by: region / alertname / classification / ...
                            │
                            └── API layer
                                  read/write for external services
                                  wraps DB + Lark topic content
```

### What Gets Retired

| Repo | Status |
|------|--------|
| jiralert | Fully replaced by alertstore |
| jiraport alert logic | Replaced; non-alert Jira event handling TBD |
| feishu_messager_v2 | Phase 1: alertstore calls it for phone; Phase 2: internalize |

### What Stays (Downstream)

| System | Role |
|--------|------|
| Jira | Passive consumer — alertstore creates issues for compatibility |
| External AI services | Triggered by alertstore webhook (or Jira webhook during transition) |
| IM groups | Primary interaction surface for discussion — NOT the record. Every message in an alert's bound topic is captured into `alert_timeline` (the progress SSOT); alertstore never reads IM history as a data source. See "Progress SSOT" below. |

## Data Model

Naming:
- **am_webhook_receipt**: one Alertmanager webhook delivery; raw payload and
  delivery diagnostics only, never the source of alert business context
- **event**: one immutable observation of one Alertmanager alert instance
  (`first_firing`, `repeat_firing`, or `resolved`)
- **alert_instance**: one occurrence identified by `(fingerprint, starts_at)`;
  stores the current occurrence snapshot and its first/last/resolved observation
  times
- **alert_instance_binding**: many-to-many binding between occurrences and human
  alerts; an occurrence can be paged into another alert after a stale aggregate
  is detached
- **alert**: canonical public context and human workflow record (only
  classification=page/ticket; logging only has events/instances)
- **alert_audit**: every structured operation on an alert (state changes, phone calls, deliveries, mute creation)
- **alert_timeline**: progress SSOT — discussion messages captured from the bound IM topic plus comments written via web/API; readers merge audit+timeline by time
- **subscription**: IM channel delivery preferences

### Progress SSOT: capture, not proxy

IM is the interaction surface; the record lives in alertstore. Rationale:
IM content is not durable (recall / edit / group dissolution / bot removal),
reading IM history on demand couples availability to the IM vendor
(exactly when it's down you need the timeline most), and reports /
postmortems / AI consumption need structured queryable data.

- Bound topic messages are captured in real time via IM event subscription
  into `alert_timeline`; recalls only annotate (`recalled_at`), never delete.
- Structured ops (`/ack`, `/resolve <resolution>`, `/assign`, `/mute`) are
  explicit bot commands driving the state machine; free discussion is
  stored verbatim and never parsed for state.
- Comments written via web/API land in the timeline first, then mirror to
  the topic through the delivery outbox — IM being down degrades the echo,
  not the record.
- The alert card carries a note that topic replies are recorded.
- Phase 2: media persistence to object storage (phase 1 stores a
  placeholder + message ref; recalled media is lost — accepted), message
  edit events, attach-group (bind extra groups/threads to one alert via a
  binding table `alert_im_binding(alert_id, channel_id, thread_id)` —
  today an alert binds exactly one topic via `alert.im_thread_id`), and
  card action buttons.
- Explicitly out of scope: reading IM history as a data source (capture
  starts at binding time, no backfill), parsing free text for state or
  intent, capturing messages outside bound topics or DMs.

Design decisions:
- a webhook receipt is operational evidence, not public context; the `alert`
  record owns cluster, tenant, region, namespace, business type, component,
  message, labels, and annotations used by human workflows
- event stores **one row per observation** (not per webhook group)
- alert_instance stores **one row per occurrence**; resolve does not erase it,
  so the complete per-instance lifecycle remains queryable after Alertmanager
  stops returning the alert
- alert is always **new** on firing (no reopen of resolved alerts)
- Business fields (cluster_id, tenant_id, region, etc.) stored inside JSON `attrs` column with generated column indexes — avoids wide tables, easy to add fields
- IM and issue tracker references use generic names (im_thread_id, not lark_topic_id)
- Subscription allows **multiple rows per channel** (different filters, different modes)

### Alertmanager ingestion model

```sql
CREATE TABLE am_webhook_receipt (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    receiver VARCHAR(255) NOT NULL,
    group_key VARCHAR(255) NOT NULL,
    truncated_alerts INT NOT NULL,
    raw_payload JSON NOT NULL,
    received_at TIMESTAMP(3) NOT NULL
);

CREATE TABLE alert_instance (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    fingerprint VARCHAR(255) NOT NULL,
    starts_at TIMESTAMP(3) NOT NULL,
    state VARCHAR(32) NOT NULL,
    labels JSON NOT NULL,
    annotations JSON NOT NULL,
    first_observed_at TIMESTAMP(3) NOT NULL,
    last_observed_at TIMESTAMP(3) NOT NULL,
    resolved_observed_at TIMESTAMP(3) NULL,
    UNIQUE KEY uk_occurrence (fingerprint, starts_at)
);

CREATE TABLE event (
    id            BIGINT AUTO_INCREMENT PRIMARY KEY,
    receipt_id    BIGINT NULL,
    instance_id   BIGINT NULL,
    group_key     VARCHAR(255)  NOT NULL,        -- alertmanager group key (shared by alerts in same group)
    fingerprint   VARCHAR(255)  NOT NULL,        -- individual alert fingerprint
    status        VARCHAR(32)   NOT NULL,        -- firing / resolved
    observation_type VARCHAR(32) NOT NULL,        -- first_firing / repeat_firing / resolved
    attrs         JSON          NOT NULL,        -- all business fields:
                                                 --   alertname, classification, original_classification,
                                                 --   cluster_id, tenant_id, region, component,
                                                 --   biz_region, tenant_kind, customer_type,
                                                 --   labels, annotations, generator_url
    starts_at     TIMESTAMP(3)  NOT NULL,        -- alert's own starts_at
    ends_at       TIMESTAMP(3)  NULL,            -- alert's own ends_at (null if still firing)
    received_at   TIMESTAMP(3)  NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    -- generated columns for indexing
    alertname         VARCHAR(255) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.alertname'))) STORED,
    classification    VARCHAR(32)  GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.classification'))) STORED,
    cluster_id        VARCHAR(255) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.cluster_id'))) STORED,
    tenant_id         VARCHAR(255) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.tenant_id'))) STORED,
    region            VARCHAR(64)  GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.region'))) STORED,
    biz_region        VARCHAR(64)  GENERATED ALWAYS AS (COALESCE(JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.biz_region')), '')) VIRTUAL,
    tenant_kind       VARCHAR(32)  GENERATED ALWAYS AS (COALESCE(JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.tenant_kind')), '')) VIRTUAL,
    customer_type     VARCHAR(32)  GENERATED ALWAYS AS (COALESCE(JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.customer_type')), '')) VIRTUAL,

    INDEX idx_group_key (group_key),
    INDEX idx_fingerprint (fingerprint),
    INDEX idx_alertname (alertname),
    INDEX idx_classification (classification),
    INDEX idx_cluster (cluster_id),
    INDEX idx_tenant (tenant_id),
    INDEX idx_region (region),
    INDEX idx_event_biz_region (biz_region),
    INDEX idx_event_tenant_kind (tenant_kind),
    INDEX idx_event_customer_type (customer_type),
    INDEX idx_received (received_at)
);

CREATE TABLE alert_instance_binding (
    alert_id VARCHAR(64) NOT NULL,
    instance_id BIGINT NOT NULL,
    PRIMARY KEY (alert_id, instance_id)
);
```

The exact executable schema is migrations `000004_add_instance_lifecycle` and
`000005_add_tenant_profile`. The latter's two columns are VIRTUAL rather than
STORED because TiDB cannot add a STORED generated column via `ALTER TABLE`;
they are indexed, which is what keeps a filter on either from recomputing the
JSON extraction per row. They also coalesce to `''`, because unlike every
column beside them they extract an attrs key that is absent whenever enrichment
found nothing and from every row predating the migration.
`event.attrs` remains an immutable observation snapshot. `alert.attrs` is the
canonical public context, while `alert_instance.labels/annotations` preserve
the differing context of each occurrence.

`alert.attrs` is written once, from the occurrence that opened the alert, and
the aggregation key (`AggLabels`) does not cover component, region, tenant, biz
type or environment — occurrences that disagree on those still fold into one
alert. So a rendered header (Lark card, Jira description, detail page) resolves
those fields across the whole occurrence set through `alert.CommonFact`, which
prints the shared value when every occurrence agrees and names the distinct
values when they do not. This is the line alertmanager draws between
`commonLabels` and per-alert `labels`; alertstore has to draw it itself, because
its aggregate is not alertmanager's group — one alert can span several
notifications, and one notification can split across classifications. Fields
that come from enrichment — clinic's cluster metadata, the ops portal's
`biz_region` and `tenant_kind` — have no occurrence-level counterpart and stay
as `alert.attrs` recorded them.

The same split applies to the raw label sets. `alert.CommonLabels` is the
intersection over the alert's occurrences and is printed once, at alert level;
each occurrence then shows only `alert.DistinctLabels` — what it carries beyond
that, which is the part that says *which* pod, disk or store. A key every
occurrence carries at a different value is not common and stays on the
occurrence: it is precisely what tells them apart. Occurrences keep their full
label set in the database and in the API; this is a view over them.

An alert holding a single occurrence is the case the intersection cannot serve:
it agrees with itself on everything, so the whole label set would land at alert
level and the occurrence would show nothing at all — and single-occurrence
alerts are the ones read most. There `CommonLabels` falls back to a fixed list
of alert-level keys (`alertLevelLabels`: alertgroup, alertname, component,
o11y_biz_type, o11y_region, o11y_tenant_id, oncall_team, severity, tenant,
tenant_id, tidb_cluster_id, tier), naming the fleet-wide context outright and
leaving the rest on the occurrence.

### alert_id_seq (ID sequence counter)

```sql
CREATE TABLE alert_id_seq (
    prefix      VARCHAR(16)  NOT NULL,             -- "ALT-2607"
    next_val    BIGINT       NOT NULL DEFAULT 1,
    PRIMARY KEY (prefix)
);
```

ID format: `ALT-YYMM-{seq}`, e.g. `ALT-2607-42`. The ID is deliberately
classification-neutral; page/ticket semantics live in the `classification`
field. Sequence resets each month and is generated atomically via
`INSERT ... ON DUPLICATE KEY UPDATE next_val = next_val + 1`.

### alert (Page and Ticket only)

```sql
CREATE TABLE alert (
    id              VARCHAR(32)   PRIMARY KEY,     -- ALT-2607-42
    state           VARCHAR(32)   NOT NULL,        -- open/acked/resolved
    attrs           JSON          NOT NULL,        -- all business fields:
                                                   --   fingerprint, alertname, classification,
                                                   --   cluster_id, tenant_id, region, component,
                                                   --   biz_region, tenant_kind, customer_type,
                                                   --   summary, description,
                                                   --   labels (original alert labels)
    -- assignment
    assigned_to     VARCHAR(255)  NOT NULL DEFAULT '',
    acked_by        VARCHAR(255)  NOT NULL DEFAULT '',
    ack_source      VARCHAR(32)   NOT NULL DEFAULT '',  -- web / im / issue_tracker
    resolution      VARCHAR(64)   NOT NULL DEFAULT '',   -- how it was resolved (see below)
    resolution_note VARCHAR(1024) NOT NULL DEFAULT '',   -- free-text detail behind the resolution

    -- timers
    call_delay_until  TIMESTAMP(3) NULL,           -- when to start phone call (Page only)
    response_deadline TIMESTAMP(3) NULL,
    recovery_deadline TIMESTAMP(3) NULL,

    -- external references (generic, not provider-specific)
    im_thread_id      VARCHAR(255) NOT NULL DEFAULT '',  -- topic/thread ID in IM
    im_channel_id     VARCHAR(255) NOT NULL DEFAULT '',  -- main group channel ID
    external_issue_key VARCHAR(255) NOT NULL DEFAULT '', -- Jira key (transitional, removable)

    -- timestamps
    created_at      TIMESTAMP(3)  NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
    updated_at      TIMESTAMP(3)  NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
    acked_at        TIMESTAMP(3)  NULL,
    resolved_at     TIMESTAMP(3)  NULL,

    -- generated columns for indexing
    fingerprint     VARCHAR(255)  GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.fingerprint'))) STORED,
    alertname       VARCHAR(255)  GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.alertname'))) STORED,
    classification  VARCHAR(32)   GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.classification'))) STORED,
    cluster_id      VARCHAR(255)  GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.cluster_id'))) STORED,
    tenant_id       VARCHAR(255)  GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.tenant_id'))) STORED,
    region          VARCHAR(64)   GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.region'))) STORED,
    biz_region      VARCHAR(64)   GENERATED ALWAYS AS (COALESCE(JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.biz_region')), '')) VIRTUAL,
    tenant_kind     VARCHAR(32)   GENERATED ALWAYS AS (COALESCE(JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.tenant_kind')), '')) VIRTUAL,
    customer_type   VARCHAR(32)   GENERATED ALWAYS AS (COALESCE(JSON_UNQUOTE(JSON_EXTRACT(attrs, '$.customer_type')), '')) VIRTUAL,

    INDEX idx_state (state),
    INDEX idx_fingerprint (fingerprint),
    INDEX idx_alertname (alertname),
    INDEX idx_classification (classification),
    INDEX idx_assigned (assigned_to),
    INDEX idx_cluster (cluster_id),
    INDEX idx_tenant (tenant_id),
    INDEX idx_region (region),
    INDEX idx_biz_region (biz_region),
    INDEX idx_tenant_kind (tenant_kind),
    INDEX idx_customer_type (customer_type),
    INDEX idx_created (created_at)
);
```

### alert_audit (audit log)

Records **every operation** on an alert, including external side effects. External actions (phone calls, IM messages, Jira syncs) just store a log entry — no state tracking or retry logic at the audit level.

```sql
CREATE TABLE alert_audit (
    id          BIGINT AUTO_INCREMENT PRIMARY KEY,
    alert_id    VARCHAR(36)  NOT NULL,
    event       VARCHAR(64)  NOT NULL,
    old_state   VARCHAR(32)  NOT NULL DEFAULT '',
    new_state   VARCHAR(32)  NOT NULL DEFAULT '',
    actor       VARCHAR(255) NOT NULL DEFAULT '',  -- user email or "system"
    source      VARCHAR(32)  NOT NULL DEFAULT '',  -- web/im/issue_tracker/system/agent
    detail      JSON,                              -- event-specific payload
    created_at  TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    INDEX idx_alert (alert_id),
    INDEX idx_created (created_at)
);
```

Event types:

| event | actor | detail example |
|-------|-------|----------------|
| created | system | `{"classification":"page","fingerprint":"..."}` |
| classified | system | `{"original":"page","final":"ticket","reason":"recent_activity"}` |
| assigned | system / user | `{"assignee":"wang@..","from_schedule":true}` |
| reassigned | user | `{"old":"wang@..","new":"li@.."}` |
| ack | user | `{"source":"im"}` |
| resolved | user / system | `{"source":"web","comment":"confirmed recovered"}` |
| am_resolved | system | `{"fingerprint":"..."}` |
| am_firing | system | `{"fingerprint":"...","note":"re-firing after previous resolve"}` |
| mute_rule_created | user | `{"filter":{"alertname":"X","cluster_id":"Y"},"expires":"2h"}` |
| mute_rule_expired | system | `{"rule_id":"..."}` |
| muted_by_rule | system | `{"rule_id":"...","reason":"maintenance window"}` |
| im_urgent | system | `{"oncaller":"wang@..","method":"im_urgent"}` |
| phone_called | system | `{"oncaller":"wang@..","number":"+86...","provider":"ali","success":true}` (phase 2) |
| phone_failed | system | `{"oncaller":"wang@..","error":"no answer","will_retry":true}` |
| im_topic_created | system | `{"thread_id":"...","channel_id":"..."}` |
| im_forwarded | system | `{"target_channel":"oc_apac","mode":"topic_forward"}` |
| im_card_sent | system | `{"target_channel":"oc_dedicated","mode":"card_only"}` |
| jira_created | system | `{"issue_key":"O11Y-12345"}` |
| jira_synced | system | `{"issue_key":"O11Y-12345","state":"resolved","transition":"Done","moved":true}` |
| webhook_sent | system | `{"url":"https://ai-svc/...","status":200}` |
| comment | user / agent | `{"text":"AI analysis: predict_linear 7d..."}` |
| call_delay_started | system | `{"until":"2026-05-13T10:04:00Z"}` |
| call_delay_expired | system | `{}` |
| backup_called | system | `{"backup":"li@..","reason":"primary_no_answer"}` |

### subscription

```sql
CREATE TABLE subscription (
    id          VARCHAR(36)  PRIMARY KEY,          -- UUID
    channel_id  VARCHAR(255) NOT NULL,             -- IM channel ID (Lark chat_id, Slack channel_id, ...)
    channel_name VARCHAR(255) NOT NULL DEFAULT '',
    mode        VARCHAR(32)  NOT NULL,             -- topic_forward / card_only / event / customer
    filter      JSON         NOT NULL,             -- {"region":["apac"],"classification":["page","ticket"],...}
    created_by  VARCHAR(255) NOT NULL DEFAULT '',
    created_at  TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
    updated_at  TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),

    INDEX idx_channel (channel_id)
);
```

Multiple subscriptions per channel allowed — e.g., one for `page → topic_forward`, another for `ticket → card_only`.

### mute_rule

```sql
CREATE TABLE mute_rule (
    id          VARCHAR(36)  PRIMARY KEY,          -- UUID
    filter      JSON         NOT NULL,             -- {"alertname":"TiKVInstanceLowSpace","cluster_id":"abc-123",...}
    reason      VARCHAR(1024) NOT NULL DEFAULT '',
    created_by  VARCHAR(255) NOT NULL DEFAULT '',
    expires_at  TIMESTAMP(3) NOT NULL,             -- must have expiry, no permanent mutes
    created_at  TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    INDEX idx_expires (expires_at)
);
```

Mute rules are checked **before** alert creation. If an incoming event matches an active mute rule (not expired), it is stored as event only — no alert is created, no notification is sent. Filter fields match against event attrs (same fields as subscription filter: alertname, cluster_id, tenant_id, region, labels, etc.), and are validated on write for the same reason a subscription filter is: the matcher skips an unreadable filter and an empty one matches everything, so a typo would otherwise be stored as a rule that quietly mutes the fleet.

"Stored as event only" is literal, and it is the whole scope of a mute: only a
page or ticket ever reaches the check (logging returns before it, having no
alert to suppress), and the raw observation stream is never suppressed. See
"The raw observation stream" for why a rule scoped to an alertname must not be
able to quiet another group's subscription.

## Alert State Machine

### Phase 1

```
open ──ack──→ acked ──resolve(human)──→ resolved
```

Page and Ticket both start as `open`. The only difference is delivery — Page also triggers a phone call via feishu_messager.

### Full design (phase 2+)

```
open ──ack──→ acked ──resolve(human)──→ resolved
  │
  ├─ call_delay expires → phone call oncaller     (delivery layer)
  └─ no answer          → phone call backup       (delivery layer)
```

All states: `open`, `acked`, `resolved`

Call escalation does not add lifecycle states. An earlier draft had `pending`
and `paging` in the state machine, but they answer a different question —
"how far has the paging machinery got" rather than "has a human taken this" —
and every consumer of state (Jira mapping, cards, filters) would have had to
treat all three unclaimed states alike. Paging progress is delivery state:
`call_delay_until`, delivery tasks and the audit log carry it.

Only **human** resolve is a real state transition. `resolved` is the only terminal state.

Alertmanager resolved observations update the bound `alert_instance` and are
shown in the alert timeline, but do NOT change human alert state. The alert
stays open until a human resolves it with a resolution reason.

Dedup: occurrences are deduplicated by `(fingerprint, starts_at)`. Human alerts
are aggregated by the versioned aggregation key; each occurrence is explicitly
bound to the selected alert.

`GET /api/alerts/{id}` returns all bound occurrences and their observations,
plus computed firing/resolved/unknown counts and the latest Alertmanager
observation time.

Notes:
- Mute is handled at the event level via mute_rules — muted events never create alerts
- Escalation is just an assign to a different person — no separate state, audit log records the reason
- Resolve requires a `resolution` verdict, plus an optional free-text note.
  Three values, kept deliberately few so different resolvers pick the same one
  (an earlier seven-value set had boundaries nobody drew consistently, which
  made the metrics built on it random):
  - `handled` — a human intervened: fixed, mitigated, or notified the customer
  - `self_recovered` — the resolver's verdict that the problem went away on its
    own. Not "Alertmanager sent resolved": on a rate-of-change alert (e.g. a
    p999-increase rule that always recovers once the value plateaus) the signal
    recovering says nothing about whether the problem is gone
  - `noise` — should not have fired: false positive, expected activity, duplicate
  Together they answer the two questions the quality metrics ask (Principle E):
  should this alert have fired, and did it need a human. Finer distinctions
  (fixed vs mitigated, which kind of noise) go in the note until a report
  actually consumes them

### State Transitions

### Phase 1 transitions

| From | Event | To | Side Effects |
|------|-------|----|-------------|
| (new) | alert firing, class=page | open | create alert, assign oncaller, post IM topic, phone call via feishu_messager |
| (new) | alert firing, class=ticket | open | create alert, assign oncaller, post IM topic |
| open | ack | acked | — |
| open | resolve | resolved | update IM topic, transition Jira |
| open | assign | open | update assigned_to, notify new assignee |
| open | instance appended (from AM) | open | log audit entry; notify which instance started firing, its own text, and how many now fire; refresh the card's occurrence list |
| open | instance resolved (from AM) | open | log audit entry; notify which instance recovered and how many remain firing; refresh the card's occurrence list |
| acked | resolve | resolved | update IM topic, transition Jira |
| acked | assign | acked | update assigned_to, notify new assignee |
| acked | instance resolved (from AM) | acked | log audit entry; only prompt human resolution when no instance remains firing |
| any | approaching deadline | — | notify oncaller in IM topic |

### Phase 2+ delivery-layer additions

Escalation events keep the alert in `open`; only ack moves it. Each step is
audit-logged, which is how the paging trail stays visible without a state.

| Alert state | Event | Side Effects |
|-------------|-------|-------------|
| (new) | alert firing, class=page | create alert (open), assign oncaller, post IM topic, start call_delay timer |
| open | call_delay expires | phone call oncaller |
| open | ack | cancel call_delay / backup timer |
| open | no answer, backup timeout | phone call backup oncaller |

## Classification

Classification comes from alertmanager rule labels. Alertstore applies two override layers:

```go
func (c *Classifier) Classify(ctx context.Context, alert *AlertEvent) Classification {
    class := alert.Labels["classification"] // from alert rule

    // Override 1: non-KA customer filter
    if !c.customerSvc.IsKA(ctx, alert.TenantID) {
        if shouldMuteForNonKA(alert.Name) {
            return Logging
        }
    }

    // Override 2: preserve jiralert's rule-scoped ActivityHub precheck.
    if class == Page && c.clinic.HasRelatedActivity(
        ctx, alert.Name, alert.ClusterID, alert.StartsAt,
    ) {
        return Ticket
    }

    return class
}
```

The Clinic client carries only the legacy EVENT prechecks for frequent-restart
alerts and `PDDiscoverDownStore`; unrelated activity never changes
classification. Clinic and ActivityHub failures are fail-open. Cluster metadata
(tenant fallback, name, version, product tier) is stored in event/alert attrs,
and a Clinic portal link derived from the configured Clinic base URL is shown
in the Web UI and IM cards.

### Tenant enrichment (`biz_region`, `tenant_kind`, `customer_type`)

Two things an alert's labels never say and Clinic does not know: which sales
region the account belongs to, and whether the account is one of ours or a
customer's. Labels carry the *cloud* region (`ap-northeast-1`) — where the
cluster runs, not who owns it — and Clinic knows clusters, not accounts.

Ingestion asks the DevOps ops portal (`pkg/devops`,
`GET /api/v1/devops-internal/tenants/{id}`) once per tenant, cached for 30
minutes, and records `biz_region` (NA/EMEA/APAC/JAPAN/LATAM/RDG),
`tenant_kind` (`internal`/`external`) and `customer_type` (SKA/KA/SMB, which the
portal calls segmentation) in event and alert attrs. It runs after
the Clinic lookup, because Clinic is what recovers the tenant ID for an alert
whose labels carry only a cluster. Failures are fail-open, as everywhere else on
this path: a header row is not worth an alert.

The call carries no credential. The portal's `/api/v1/devops-internal` group has
no auth middleware and is reachable on the private network only — alertstore
runs in the same cluster, the same arrangement as the oncall and core-server
listeners above, and `services/jsm-syncer` reads tenants over the same route.
The authenticated `/api/v1/devops` route renders the identical payload through
the same converter, so a deployment that has to go through the shared gateway
sets `devops.api_token` and changes nothing else.

jiralert and jiraport read the same two columns straight out of
`devops_prod.tenants` with raw SQL (`kind`, and `sales_force_region` falling
back to `country_region`). Alertstore asks the service that owns the table
instead, which is also where the Salesforce override on each field is already
applied — the precedence lives in one place rather than being re-derived per
caller. `tenant_kind` is also written to the mirrored Jira issue's
`customfield_10159`, which is what jiraport's assignment rules key on to tell an
internal alert from a customer's; leaving it unset would silently reclassify
every mirrored issue as external.

No YAML rules or template engine. Change classification logic by changing code
and deploying.

## Customer-facing alerts (`visibility: external`)

Some alert rules are written for the customer, not for us. cloud_server pushes
those to the tenant's own subscribers — email, Slack, PagerDuty, Lark and the
rest — and alertstore receives a copy through the same Alertmanager feed as
everything else. They are the one class of alert whose notification has already
reached its audience before it reaches us.

The rule says so, with `visibility: external`. That label is the whole contract:
it lives where the audience is actually decided, and everything below follows
from it. Absent means internal, which is the direction that fails safe —
mistaking an internal alert for a customer-facing one would silently stop paging
for it.

**They are never a response.** `Classifier` forces them to `logging` before any
other layer runs, and returns outright so nothing can raise them back. Two
things would otherwise go wrong: these rules carry no `response_class`, so the
severity fallback would page an on-caller for every critical one; and the page
would be the *second* notification for a condition the customer is already
looking at. Being classified `logging` means no alert record, no owner, no Jira
issue, and no 24-hour aggregate — `StalenessWindow` never applies to them.

**They are still recorded in full.** One event row per notification, one
`alert_instance` per occurrence. "How many times was this customer told, and
when" is an event query, and a more precise answer than an alert could give.

### Notices

A channel subscribes to them by choosing the `customer` mode. Delivery is one
card per stored event — firing and resolved alike — because that is the
granularity at which the customer was told: cloud_server sends the moment
Alertmanager delivers, with no aggregation of its own, so collapsing repeats
here would hide exactly what the customer keeps being told. The card names the
alert, the cluster, the tenant and the time, carries the prose the customer
received, and says plainly that nobody was paged. It has no Ack button, because
there is nothing here to pick up.

That equivalence between the two streams holds only if both come from the same
notification. The internal copy currently arrives via the `pingcap-*` receivers
while the customer's goes out through `customer`, so their `group_wait` /
`repeat_interval` settings have to agree by hand. Routing alertstore as a second
webhook on the `customer` receiver would make them the same delivery and remove
the need to keep two configurations in step. Worth doing; not done yet.

```json
{
    "channel_id": "oc_customer_watch",
    "mode": "customer",
    "filter": {}
}
```

**The mode is the opt-in**, which is why the audience is not a filter field.
One of these re-fires for as long as the condition holds, so a channel that
subscribed to alerts must not start receiving a per-notification stream because
some filter it left empty happened to match. Nothing crosses between modes: an
external alert reaches `customer` subscriptions and nothing else.

Mute rules apply. Muting one means what muting always means here, "stop telling
us"; it cannot reach the customer's own notification, which was sent before this
copy arrived.

### The raw observation stream (`event` mode)

`event` mode is the same machinery pointed at internal alerts: one card per
stored observation, for every classification, delivered to whoever wants to
watch the stream Alertmanager actually produces.

**It is not restricted to `logging`,** and the reason is the whole point of
having it. An alert is an aggregate, and it collapses an occurrence's repeat
firings into *nothing at all* — a re-firing of an already-bound occurrence
produces no reply and no card refresh, deliberately, because someone being
paged does not need to be paged again. The consequence is that "this page has
been re-firing every hour since Tuesday" has nowhere to appear. Nothing in the
alert view can show it; the events are all in the database. That is the
question this mode exists to answer, and restricting it to logging — the
classification nobody responds to by definition — would have left it unaskable
while making the mode nearly useless.

So the card leads with the observation's own relationship to its occurrence's
history: `first firing`, `repeat firing — the condition has not cleared since
the last notification`, or `resolved`. That field comes free: `ApplyObservation`
already computes it to stamp `event.observation_type`.

Because a page-classified observation *is* somebody's problem, the card has to
say so. When the observation landed on an alert it names it, links to it, and
says the handling is happening there; only when it opened none does it fall
back to "Recorded only. This observation opened no alert." Both readings matter
— one stops a reader double-handling something already owned, the other stops
them ignoring it because event cards are usually nobody's problem.

Everything else follows from being internal rather than customer-facing: it
reads the internal annotation convention (`summary` titles, `description`
elaborates), is blue rather than orange, and never claims anyone outside was
told. A channel subscribed to both can tell them apart at a glance, which is
the point of them being separate modes.

**Mute does not reach this stream, deliberately.** A mute rule suppresses the
response — the alert and everything downstream of one — and has always promised
the record survives it ("stored as event only"). Now that the record is
deliverable, that promise has teeth: a muted observation still produces a card,
and the card names the rule and its stated reason.

Silencing the stream instead would be wrong three ways. A rule is written from
inside one alert's topic and scoped to an alertname and cluster, never to a
channel, so an on-caller quieting their own page would blind whichever other
group subscribed to those observations, without either side knowing. For
customer-facing alerts it would be worse than blind: muting cannot stop
cloud_server telling the tenant, so the customer would keep being notified while
our copy went quiet — the exact divergence the notice stream exists to prevent.
And it would make mute unobservable by the one surface that spares it: this is
where you can see what an active rule is currently hiding.

A stream that is too noisy is a routing problem, and the subscription is the
knob for it — per channel, owned by whoever receives the cards, with no expiry
to forget about.

**Delivery mechanics.** One `im_notice` outbox task per (event, channel), so a
channel whose send fails is retried alone rather than re-delivering to everyone
who already got it. The task carries its rendered content rather than a
reference to the occurrence: a notice reports an event, not a state, and
re-reading the occurrence when the task runs would let a retry announce a
condition that has since changed — describing a message the customer never
received. It is the only task kind with no `alert_id`, and therefore the only
one that cannot be delayed by anything else in alertstore being down.

The prose needs one substitution. Customer-facing rules put the subject in
`description` and the body in `message` (the reverse of the internal
convention), and leave `TiDBClusterName` / `TiDBCloudDomainName` for
cloud_server to fill in on its way out. alertstore resolves those two from the
alert's own labels so the copy reads as what was sent rather than as the
template behind it; `TiDBProjectName`, which has no counterpart in the labels,
is left standing rather than guessed at. This duplicates a contract owned by
another repository, deliberately and narrowly — the alternative is an internal
card showing a literal `TiDBClusterName`, which reads as a broken alert.

**What this does not claim.** The card says the customer was *notified*, not
that they *received* it. cloud_server's delivery is best-effort — a goroutine
per channel with up to three in-memory retries and no outbox — so a send can
fail after Alertmanager delivered. Reconciling against its `alert_notification`
table would be a different and much larger piece of work.

`alertstore_notices_delivered_total{status}` counts what went out. Compared
against events ingested for the same feed, a gap means channels asked for
notices and are not getting them — and unlike an alert, nobody is waiting on
one, so nothing else would say so.

## IM Abstraction

Alertstore core code never references Lark/Slack directly. All IM interaction goes through:

```go
type IMProvider interface {
    CreateTopic(ctx context.Context, channelID, title, body string) (threadID string, err error)
    ReplyToTopic(ctx context.Context, threadID, text string) error
    UpdateTopicTitle(ctx context.Context, threadID, title string) error
    ForwardTopic(ctx context.Context, threadID, targetChannelID string) error
    SendCard(ctx context.Context, channelID string, card *Card) (messageID string, err error)
    GetReadUsers(ctx context.Context, messageID string) ([]string, error)
    MakeUrgent(ctx context.Context, messageID, userID string) error
    GetUserPhone(ctx context.Context, userID string) (string, error)
}
```

Phase 1 implementation: `pkg/im/lark/`. Switching to Slack = add `pkg/im/slack/`, no core code changes.

## On-Call Owner Resolution

Alerts are owned by the engineer on duty, resolved from the oncall v2 service
(`services/oncall`). Implemented in `pkg/oncall`, the client for that service —
named after the upstream system, not after the schedule lookup, because phone
calling is expected to move behind the same service (see "Phone Call" below)
and will be added to this client rather than a second one.

**Resolution runs in the delivery worker** (`delivery.Worker.resolveOwner`), not
at ingest. Three reasons, in order:

1. The outbox's retry (30s→15m, ~45min) *is* the repair mechanism. If the oncall
   service is down, the topic still goes out immediately, the task retries, and
   the owner plus the buzz that depends on it are filled in once the service
   recovers. Resolving at ingest would leave the alert permanently unowned,
   because a repeat firing dedups into the existing alert without revisiting
   assignment.
2. It keeps the on-call lookup off the webhook request path.
3. Delivering to the owner — announce, forward, buzz — happens there anyway,
   and all of it needs the owner's identity. Resolving it in the same task
   keeps one directory lookup and no state between the steps.

Routing:

0. An alert on an **internal tenant** is handed to nobody. That account is one
   of ours — a test cluster, an internal tier — so nobody outside is living
   through the alert, and putting an on-caller on the hook for it is a cost with
   no one on the other side. The alert is still recorded, still posted to the
   topic and still mirrored to Jira; it arrives owned by nobody. Only the
   automatic hand-off is skipped: `/assign` and the web UI still work, because
   "nobody is paged for it" is not "nobody may hold it".

   The kind comes from the ops portal (see "Tenant enrichment"), and anything it
   could not answer about — no tenant label, an unnumbered one, a failed lookup
   — counts as a customer's. An unenriched alert still reaches an on-caller,
   which is the assumption that fails safe.
1. Team comes from the alert's `oncall_team` label — the same label jiraport
   routes on, so alert rules need no change.
2. No label falls back to `oncall.default_team`. With neither, the alert stays
   unowned (`no_team`).
3. `GET /api/v1/teams/{ref}/current-oncall` returns the team's primary and
   backup lanes plus its managers. The first primary engineer becomes the
   owner; an empty primary lane falls through to backup, and an empty backup
   lane escalates to the first manager rather than leaving the alert unowned.
   Only a team with all three empty yields `schedule_gap`. Managers are the
   last resort, not a lane: a sustained `manager` outcome means that team's
   rotation has a hole someone needs to fix.

Properties:

- **Fail-open, then repair.** An oncall outage never delays the topic. The card
  goes out without the owner line, and the retry backfills the owner, the web
  UI and the buzz. Never trade a delivered alert for a correctly-attributed one.
- **A terminal answer is not retried.** `no_team` and `schedule_gap` are real
  answers — the task completes and the alert stays unowned until someone runs
  `/assign`. Only `lookup_error` retries.
- **Cached per team** (`cache_ttl`, default 1m), so alertmanager
  `repeat_interval` re-fires don't hammer the oncall service. Failures are not
  cached.
- **Emails are the owner identity** across schedule, web API, and bot `/assign`;
  provider user IDs never reach the database. `runIMTopic` resolves email →
  provider ID once and passes it to both the card mention and the buzz payload,
  so the directory is queried once per alert. A resigned address is recorded as
  the owner but gets no mention and no buzz — retrying cannot conjure a
  directory entry.
- **A terminal failure does not retry either.** A team the service does not
  have (`unknown_team`, usually a typo in an alert rule's label or in
  `default_team`) or a refused request needs a human to fix configuration;
  spending the outbox's eight attempts on it only buries the error. Rate
  limiting and 5xx stay retryable.
- Every resolution increments `alertstore_oncall_lookups_total{outcome}`.
  Outcomes: `primary`, `backup`, `manager`, `no_team`, `schedule_gap`,
  `unknown_team`, `lookup_error`, `disabled`. Sustained `manager` means a
  rotation has a hole; the other non-`primary` outcomes mean alerts are landing
  unowned.
- Deployments talk to the oncall service's **internal listener**
  (`oncall-internal.<ns>.svc.cluster.local`), which is ClusterIP-only and
  authenticates nothing, so alertstore carries no credential for it at all.
  Reads there are anonymous and leave no audit record — that service only
  records history on writes — so there is nothing to attribute. Pointing at the
  authenticated listener instead (as prod will, having no internal one) only
  requires setting `api_key` to a service key with the `oncall:read` scope.

Owner resolution is what arms the buzz for `page` alerts: with no owner there
is nobody to notify.

### Who the reassign picker offers

Reassignment in the web UI searches **core-server's user directory**
(`services/core-server`, a mirror of the Lark org chart) through `pkg/directory`
— deliberately not the on-call roster. Whoever picks an alert up next is
routinely someone who is not on duty for it: an escalation, or a handoff to the
person who wrote the code. Offering only the rotation would make the common case
the one you have to type around.

`GET /bff/users?q=` proxies the search rather than letting the browser call
core-server, so the gateway credential stays server-side and the UI keeps
talking to one origin. core-server matches the keyword against email, Chinese
name, English name, nickname and the name's pinyin, so one field serves `wang`,
`王` and `wk@` alike, and it excludes people the directory marks unavailable.

Like the oncall service, core-server sits in the same cluster and its own
listener authenticates nothing — the shared devops gateway does that in front of
it. Alertstore talks to that listener
(`core-server.<ns>.svc.cluster.local:8080`) and so carries no credential for it
at all.

Fail-open, like every other lookup here: a directory that is unconfigured or
unreachable returns an empty list, not an error. The assignee field is free text
either way — the list is a shortcut past typing an address in full, never a
constraint on who may hold an alert — so an operator who knows the address is
never blocked by a search that cannot answer. Answers are cached per keyword
(`cache_ttl`, default 5m) so a burst of typing costs core-server one request.

### Where the owner shows up

The card cannot be edited after it is posted, so it carries only what cannot
change: `Oncall team`, taken from the alert's own label. It never names a
person. Who holds the alert right now is state, and state that outlives its
surface goes stale — a card naming the original on-caller keeps naming them
after three handovers.

The current owner lives in the topic's replies (`im_owner`) and in the web UI:

| Source | Announcement |
|---|---|
| on-call resolution at creation | `im_owner` reply: `Assigned to @person` |
| web UI reassign | `im_owner` reply: `Reassigned to @person by actor` |
| Jira assignee change | `im_owner` reply: `Reassigned to @person by actor` |
| bot `/assign` in the topic | the command's own `✅` acknowledgement |

### Recording an owner vs reaching them

A handover is two things with different stakes, so it is two tasks.

```
im_owner         resolve the owner (schedule, or the assignee the caller
                 committed) and persist it
                 -> enqueue im_owner_notify
                 -> announce in the topic

im_owner_notify  forward the topic into the owner's own chat
                 -> buzz that message, if the alert is a page
```

Acking claims the alert: whoever acks becomes the assignee, through every
entry point. Without that an alert reads as assigned to one person and acked by
another, and nobody can tell which of them is working it. The reverse holds
too — reassigning an acked alert returns it to open, because the
acknowledgement belonged to whoever gave it and the new owner has not seen it
yet.

Who may ack is deliberately not restricted. The case a restriction would block
— the on-caller is asleep and a colleague covers — is the one that matters
most, and refusing there is worse than a surprising handover. Having to own
what you ack is its own deterrent against an idle click. A restriction would
also be theatre: `/ack` and the web UI reach the same transition.

**The Ack button lives on the alert's card, not on this reply.** The card is
singular and rewritable; replies are immutable and accumulate. An action is an
affordance for current state, so it belongs on the surface that tracks state —
otherwise a handover chain leaves a trail of live buttons, and pressing an old
one takes the alert back from whoever holds it now. The button is therefore
present exactly while the alert is unacked, and every transition refreshes the
card through a hook on the alert service, so no entry point can forget to.

`im_owner` is the record. The reply is the only place outside the database that
says who holds this alert, and for a reassignment it is the only evidence the
handover happened at all — without it the database says one thing and the topic
says another. It enqueues before replying so the retry stays honest: the
enqueue is idempotent, so a failed reply retries the reply and nothing else.

`im_owner_notify` is the delivery, and it must never be able to hold the record
hostage. A bot that cannot message someone, or a scope nobody granted, costs
the delivery only. This was learned the hard way: with all four steps in one
task, a missing `im:message.urgent` scope meant the group was never told who
owned the alert, and eight retries put eight copies of the topic in the owner's
chat.

The notify task inherits its parent's dedup key, so each handover gets its own
delivery — handing an alert back to someone who held it before has to reach
them again, not collapse into the notification they already have.

**The buzz gets no retry budget.** It is a nudge to look, not the page itself;
a phone call will be that. Retrying it through the outbox would mean
re-forwarding, since the message it buzzes is created in the same call, and
retrying in place would be worse — the worker runs tasks one at a time, so
sleeping there stalls every alert behind it. A failure is recorded as
`im_urgent_failed` and the task completes.

The forward is the same pointer subscribed groups receive, not a second
rendering: one alert has exactly one topic, and this is a way into it. Which is
also its limit — **the owner must be a member of the main group**, or the
pointer opens onto a topic they cannot read. That is an operational
requirement, not something the forward can paper over.

The buzz targets the forwarded copy rather than the group card. A direct
message is unambiguously the recipient's to be notified about, whereas whether
Lark permits an urgent notification on a group message is unverified — jiraport
explicitly disabled it for group receivers.

Consequences worth accepting:

- Scanning the group's topic list shows the team, not the person; identifying
  the owner costs opening the topic. Absent information beats stale
  information, and the forward reaches the owner directly.
- The card renders identically whether or not the oncall service is up, because
  the team needs no network call. An outage costs a delayed reply, not a hole
  in the card.
- Forwarded topics (`topic_forward`) inherit the replies. Standalone cards
  (`card_only`) are a lightweight ping by design and never show an owner.

## The Cutover (transitional)

While jiralert is still running, alertstore receives **every** alert — jiralert
forwards them all — but is the primary responder for only some. That distinction
is not a Jira concern, and putting it there was a mistake: it governs everything
alertstore does that reaches a person or an external system.

An alert is **owned** when its rule carries `response_class` — the same label
that names the tier (page/ticket/logging). jiralert reads it and stops
responding to that rule. One signal, living in the rule, so there is nothing to
keep in step across two repositories — the rule change that hands a rule over is
the same change that switches jiralert off for it, and the granularity is the
rule rather than a whole receiver. The alternative (a receiver allow-list here,
`disable_warning` there) is two halves of one switch in two places, where drift
one way means everything happens twice and the other way means nothing happens.

Ownership and tier are one label because they turned out to be one question
asked twice: a rule written to say what response it wants is a rule whose
response this service is running. They were two labels — `as_response` for
ownership, `response_class` for the tier — and a rule carrying one but not the
other was answered twice or not at all, which is the failure the single signal
was chosen to avoid in the first place.

The consequence is that handing a rule back means dropping the label, and a rule
with no label declares no tier: the severity fallback classifies it, exactly as
it always has for the rules jiralert still owns. There is no way to say "not
mine, but page-class", and nothing needs one — a shadow alert's tier is
jiralert's business, not this service's.

Everything else is a **shadow** alert: a record of jiralert's copy.

| | owned | shadow |
|---|---|---|
| event / instance / alert stored | ✅ | ✅ |
| topic in alertstore's main group | ✅ | ✅, marked "Recorded only" |
| owner resolved, assigned, buzzed | ✅ | ❌ |
| forwarded to subscribed groups | ✅ | ❌ |
| Jira issue created and synced | ✅ | ❌ |
| Ack button on the card | ✅ | ❌ |

A shadow alert does none of the three because jiralert has already ticketed it
and jiraport has already paged for it; doing any of it here would be the second
time — a second page to the same on-caller, a second card in someone else's
group, a second ticket. The topic stays, because it is alertstore's own record in
alertstore's own group, and because watching real alerts land is how the pipeline
gets validated before it owns anything. It has to say what it is, though: a
shadow topic is indistinguishable from an owned one otherwise, and someone
reading it would assume they had been paged and that acking there handles the
alert.

Properties:

- **The label's presence is the signal**, because that is what jiralert keys on.
  A value reading `false`/`0`/`no`/`off` hands one rule back without deleting the
  label, which is what a rollback needs.
- **The label name is a constant in both services**, not configuration. A setting
  would put the contract back in two places, where a typo decides whether an
  alert is handled twice or not at all. Renaming it is a code change in both,
  which is honest about what it costs.
- **Ownership is frozen at creation**, in `alert.attrs.owned`. Re-evaluating it
  per event would let an alert recorded silently start paging people hours later
  because its rule was handed over in the meantime, and would orphan an alert
  someone is already working when a rule is handed back.
- **`cutover.own_everything: true`** makes alertstore the primary responder for
  everything it receives, ignoring the label. Correct once jiralert no longer
  responds to this deployment's feed at all — and the setting a non-prod
  deployment uses to run the full pipeline against real traffic.
- **Unlabelled means not ours.** Acting twice is the failure mode with no undo,
  so the default is to assume jiralert still owns it.
- **jiralert only stops responding to an alert it is actually forwarding here.**
  Otherwise a misconfigured forward would mean *neither* system handled it;
  failing towards a duplicate is the recoverable direction.

This whole section — the label, the config block, the gates — is deleted with
jiralert.

## Delivery Layer

### IM Topic (main group)

Every Page/Ticket creates a topic in the main alert group:
- Topic title: `[Page] TiKVInstanceLowSpace - cluster-abc-123`
- Topic body: alert details, clinic links, assigned oncaller
- All subsequent updates (ack, resolve, comments, AI analysis) posted as replies in the topic
- IM topic is the discussion SSOT

### IM Forward (subscribed groups)

Based on subscription config:
- **topic_forward**: forward the topic to the group (recipients can reply, synced)
- **card_only**: send a summary card (no thread, lightweight notification)

The other two modes (`event`, `customer`) do not deliver an alert at all — they
deliver one card per stored event and never reach this path. See "Subscription
Model" and "Customer-facing alerts".

### Phone Call

Phase 1: call feishu_messager_v2 API (`POST /messageForward` with `is_urgent=true`).

Phase 2: internalize Ali/Twilio calling logic. The core flow:
1. Query schedule API for oncaller
2. Look up phone number (internal system → IM Contact API fallback)
3. Route: domestic → Ali DYVMS, international → Twilio
4. Track call status, retry/escalate on failure

### Jira Mirror (transitional, removable)

alertstore creates the ticket jiralert used to, so the automation and team
workflows that read Jira keep working while alertstore becomes the primary
alert path. No abstraction — `pkg/jira` is a direct client written to be
deleted, along with the three `jira_*` delivery task kinds and the dispatcher's
gate, in one commit once nothing reads Jira.

Configured under `jira:`, which holds only what genuinely differs between
deployments: where Jira is, who we are, and which project this environment files
into. Leaving `base_url`/`user`/`token`/`project` empty disables it entirely,
which is the end state.

Everything else about the issue — issue type, the three custom field IDs, the
ack/resolve transition names — is fixed in `pkg/jira`. All of it is set once in
jiralert's `defaults:` block and overridden by no receiver, and most of it cannot
vary: there is one Jira instance, so there is one set of field IDs. A knob for a
value with one possible setting is not flexibility, it is a second place for the
truth to live.

Whether an alert gets an issue at all is not decided here; it follows from
whether alertstore owns the alert. See "The cutover".

**Ownership decides it.** Only an alert alertstore owns gets an issue — see
"The cutover" below, which is a property of the alert rather than of this
package. `alertstore_jira_issues_total` is where a mislabelled rule shows up.

**Which project.** jiralert writes to three — `O11Y` `[PROD]`, `O11YSTAG`
`[STAGING]`, `O11YDEV` `[DEV]` — and keys them on the Alertmanager receiver,
because one jiralert serves every environment. Each alertstore serves exactly
one, so `jira.project` and `jira.summary_prefix` follow the deployment, the same
way nothing else here is labelled with its environment (see "Deployment: one
alertstore per environment").

That holds only while no jiralert forwards across environments. jiralert prod
used to shadow-write into the **dev** alertstore, which would have forced this
one deployment to decide per alert whether an issue belonged in `O11Y` or
`O11YDEV` — with a duplicate in prod's project as the cost of getting it wrong.
Prod's forward is off (`helm/jiralert/values-prod.yaml`), and the rule is: a
jiralert forwards to its own environment's alertstore or to none.

The consequence for the dev deployment is that it now sees dev's alert traffic
rather than prod's. Its Clinic/ActivityHub configuration was chosen when it
mirrored prod and is worth revisiting.

**What it writes.** Close enough to jiralert that downstream matching keeps
working: same project, the same `[ENV]alertname` summary shape, the same
component whitelist (unknown components land in `other` rather than failing the
create), and the same tenant / cluster / raw-alert custom fields. Priority comes
from the `severity` label those per-severity receivers were routed by, falling
back to classification.

What deliberately differs is deduplication. jiralert searched Jira for a
reusable issue because Jira was its only memory; alertstore has already decided
what belongs together, so **one alert is one issue** and aggregation never
round-trips through Jira. Occurrences joining or recovering become comments, the
way jiralert's changelog comments did.

**Delivery.** Three outbox task kinds, so the webhook path never blocks on Jira
and a failed create is retried rather than lost:

| kind | when | dedup key |
|---|---|---|
| `jira_create` | alert created | empty — one per alert |
| `jira_comment` | occurrence fired / recovered | the event ID, same as the IM reply |
| `jira_transition` | alertstore ack / resolve | the target state |

`jira_create` and `im_topic` are siblings, not a chain: a Jira outage must not
delay the page, and an IM outage must not cost the issue automation reads. The
issue key is stored on the alert immediately after creation and is also the
idempotency guard — a retry that finds a key stops, which is why losing that
write is logged loudly rather than retried.

A transition Jira does not offer (the issue is already closed) is a real answer,
not a failure: the task records it and still leaves its comment.

**The loop guard.** Jira changes flow back in through `/webhooks/jira`, so a
transition whose source is `issue_tracker` is never written back. Without it,
closing an issue would walk through alertstore's `open → acked → resolved`
lifecycle and drag the issue back to "In Progress" on the way past.

**Where the link shows up.** `alert.external_issue_key`, the alert's card (added
by the refresh that follows creation — the card is posted before the issue
exists) and the web UI, which reads `external_issue_url` derived server-side so
the browser holds no second copy of the Jira base URL.

### Webhook Out (phase 2, not in phase 1)

HTTP POST to registered endpoints on alert state changes. Replaces Jira webhook as trigger for external AI services. Phase 1 relies on Jira sync to trigger existing Jira webhooks — external AI services don't need to change.

## Ack Ingress

Three channels, all call the same internal method:

```go
func (s *AlertService) Ack(ctx context.Context, alertID, user, source string) error
```

### 1. Web UI

```
POST /api/alerts/{id}/ack
Body: {"user": "wang@pingcap.com"}
```

### 2. Jira Webhook (transitional, removable)

```
POST /webhooks/jira
```

Listens for:
- Status → In Progress → ack
- Status → Done/Resolved → resolve
- Assignee change → assign

Maps external_issue_key back to alert_id, calls internal method. Thin adapter. Remove when Jira sync is disabled.

### 3. IM Bot

```
POST /webhooks/im
```

Oncaller sends `/ack` in topic. Bot parses command, resolves alert from im_thread_id, calls internal method, replies with confirmation card.

Other bot commands: `/resolve`, `/assign @xxx`, `/mute 2h`

## Subscription Model

Replaces centralized routing rules in jiralert (receiver config), jiraport (policy-matcher), and feishu_messager (chatTag→chatId mapping).

Groups self-register via API (phase 1) or bot command (phase 2):

```json
{
    "channel_id": "oc_apac_alert",
    "mode": "topic_forward",
    "filter": {
        "region": ["apac", "japan"],
        "classification": ["page", "ticket"]
    }
}
```

Modes — a mode says *what* a subscription is about, not merely how it arrives,
and each has its own unit of delivery:

| mode | unit | delivers |
|---|---|---|
| `topic_forward` | alert | a pointer into the alert's topic; replies sync back |
| `card_only` | alert | a standalone card |
| `event` | event | the raw internal observation stream, every classification |
| `customer` | event | notifications the customer already received |

The first two are alerts somebody is expected to act on. The last two are things
nobody was paged for, arriving at Alertmanager's notification rate rather than
once per alert — so they are separate modes rather than filter values, which is
what makes them opt-in. A filter narrows *within* a mode and never crosses one.

Filter fields (all optional, empty = match all within the mode):
- `region`
- `classification` — restricted to what the mode can carry: `page`/`ticket` for
  the alert modes, all three for `event`, `logging` for `customer` (a
  customer-facing alert is forced to it). Naming one a mode cannot deliver is
  refused at write time rather than left to be discovered by its silence.
- `alertname`
- `cluster_id`
- `tenant_id`
- `biz_region` — the tenant's sales region, as opposed to `region` above, which
  is the cloud region: a team covering EMEA customers is not the same set of
  alerts as one covering `eu-central-1`. Not checked against a list, because
  that enum belongs to the ops portal and can grow without telling alertstore.
- `tenant_kind` — `internal` or `external`. A third word is refused at write
  time, like a classification a mode cannot deliver.

  These three are the only fields matched case-insensitively. They come from an
  enum whose spelling differs by field (`NA`, `internal`, `KA`), and a filter written
  `Internal` that silently matched nothing would be indistinguishable from one
  whose alerts had simply not fired. Everything else is matched exactly: a
  cluster ID or an alertname means what it says.
- `customer_type` — the sales tier (SKA/KA/SMB). The name is alertstore's own
  and predates its source: the field was designed with the service and had
  nothing to populate it until the portal lookup arrived, so a subscription
  naming it used to match nothing at all.
- `component`
- `labels` — a map of label name to accepted values, matched against the raw
  alertmanager labels. Every name listed must match (naming a second label
  narrows); values within one name are alternatives. A name the alert does not
  carry matches nothing; there is no exists-check and no negation, because a
  filter says which alerts it wants and every other field reads that way.

  The fields above are promoted attrs — normalized, and sometimes assembled
  from labels not named after them — so they cannot stand in for whatever an
  alert rule happens to carry. `oncall_team` is the case that motivated this
  and has no promoted attr at all: a team subscribing (or muting) by ownership
  rather than by enumerating the clusters and alertnames it owns this quarter
  is filtering on a label. The web UI gives it its own input for that reason,
  writing `labels.oncall_team` like any other.

  Labels are matched raw, with nothing defaulted. `oncall_team` in particular
  does not fall back to `default_team`: that default lives in the oncall
  config, which neither the mute check nor the event fan-out carries, and a
  fallback only some paths could apply would make one rule match in one place
  and not the other. An alert with no team label therefore matches no named
  team, even when assignment handed it to the default team.

Who an alert was written for is deliberately *not* a filter field: it is the
mode. See "Customer-facing alerts".

Adding a new region group no longer requires code change or restart.

## API

### Alert Events

```
GET /alertstore/api/v1/events                          list raw events (filterable)
GET /alertstore/api/v1/events/{id}                     single event detail
```

### Alerts

```
GET  /alertstore/api/v1/alerts                         list alerts (filterable, paginated)
GET  /alertstore/api/v1/alerts/{id}                    alert detail + audit log + am_status
GET  /alertstore/api/v1/alerts/{id}/context            IM topic content (wrapped)
POST /alertstore/api/v1/alerts/{id}/ack               {"user": "..."}
POST /alertstore/api/v1/alerts/{id}/resolve           {"user": "...", "resolution": "self_recovered", "note": "..."}
POST /alertstore/api/v1/alerts/{id}/assign            {"user": "...", "assignee": "..."}
POST /alertstore/api/v1/alerts/{id}/mute              {"user": "...", "duration": "2h", "reason": "..."}
POST /alertstore/api/v1/alerts/{id}/comment           {"user": "...", "text": "..."}
```

### Subscriptions

```
GET    /alertstore/api/v1/channels                     list IM groups the bot has joined (picker source)
GET    /alertstore/api/v1/subscriptions                list all
POST   /alertstore/api/v1/subscriptions                create
PUT    /alertstore/api/v1/subscriptions/{id}           update
DELETE /alertstore/api/v1/subscriptions/{id}           delete
```

### Mute Rules

```
GET    /alertstore/api/v1/mute-rules                   list active rules
POST   /alertstore/api/v1/mute-rules                   create
DELETE /alertstore/api/v1/mute-rules/{id}              delete
```

### Filter Preview

```
POST /alertstore/api/v1/filter-preview   {"mode": "...", "filter": {...}, "windows": ["168h","24h"]}
```

What a filter would have matched, for the two forms above. A filter is silent
about what it fails to match, so a subscription that will never fire and one
that was never going to fire look identical in the form, and a mute rule that
silences more than its author meant announces nothing at all. The only way to
tell them apart before saving is to run the filter over history.

Takes a bare filter rather than the id of a saved one, so it answers for a rule
nobody has stored yet — the only moment the answer changes anything. `mode` is
required and has no default, because it decides which history the filter is run
over:

| mode | counted over |
|---|---|
| `topic_forward`, `card_only` | alerts, excluding shadow alerts (they never fan out) |
| `event` | observations whose rule is internal |
| `customer` | observations whose rule is customer-facing |
| `mute` | alerts, including shadow ones |

Mute counts alerts rather than observations because that is all a rule can
suppress: the check sits after the logging short-circuit and before the alert is
created, so counting observations would report resolved notifications and
logging traffic a rule never had the chance to touch — a number several times
the number of pages actually silenced. It includes shadow alerts because the
check runs before ownership is decided.

The response carries counts per window, a seven-day daily breakdown, the top
alertnames, and the three most recent matches. The breakdown is the part that
catches what counts cannot: a rule matching 300 things where 290 are the
intended alertname and 10 are something else has a total that looks perfectly
reasonable.

Matching is the production matcher. SQL narrows only on indexed columns whose
comparison is exact (region, classification, alertname, cluster_id, tenant_id);
every candidate row is then put through `SubscriptionFilter.Matches` with a
target from `subscription.TargetFromAlert`/`TargetFromEvent` — the same call the
fan-out and the mute check make. Case-folded fields and the JSON-only ones
(component, labels) are deliberately not pushed down: a second implementation of
"does this filter match" is the one bug this endpoint cannot afford.

One scan reads at most 1000 rows, newest first. The bound matters because the
filter most in need of a preview is the empty one — a new form starts empty —
so the worst case is the first render of every dialog. Ordering newest-first is
what makes it explicable: a truncated scan is exact about recent history and a
lower bound on older history, and each window reports which of the two it is in
`exact`.

### BFF (web UI only)

Endpoints shaped for one screen rather than for the resource model, published
under the same prefix so the shared gateway needs no second forwarding rule.

```
GET /alertstore/api/v1/bff/alerts/counts               sidebar badge counts
GET /alertstore/api/v1/bff/users?q=                    assignee autocomplete (core-server directory)
```

### Webhooks (ingress)

```
POST /webhooks/alertmanager                            alertmanager webhook (Bearer token required)
POST /public/v1/alertstore/webhooks/alertmanager       public gateway route (Bearer token required)
POST /alertstore/webhooks/jira                         Jira status/assignee changes (transitional)
POST /alertstore/webhooks/im                           IM bot commands (Lark phase 1)
```

## Web UI

Two pages, no auth (phase 1, internal network only, user self-identifies via localStorage):

### Event List

Table with columns: alertname, classification, status (firing/resolved), cluster, tenant, received_at.
Filters: alertname, classification, cluster_id, tenant_id, region, time range.
Shows all raw events including logging — useful for investigating alert history and verifying mute rules.

### Alert List

Table with columns: ID, state, classification, alertname, cluster, assignee, am_status, age.
Filters: classification, state, region, assignee, time range.

### Alert Detail

- Header: alertname, action buttons (Ack, Resolve, Assign, Mute)
- Two cards side by side, split by what changes. The wide one is what the alert
  **is** — id, classification, cluster, tenant, region, component, labels,
  summary — frozen when it was created. The narrow column beside it is what the
  alert is **doing**: state, monitor status, assignee, created, last monitor
  event, ack, resolution. The two status rows are labelled for whose verdict
  they carry — State is ours, Monitor status is the monitoring side's — rather
  than for the wire fields (`state`, `am_status`) behind them.
  One field per row there, so the answer to "where is this now" is read in one
  glance rather than found among fifteen labels that did not move. The column
  stacks under the facts on narrow screens.
- Assign: the assignee field searches the user directory as you type (see "Who
  the reassign picker offers") and stays free text if you would rather not
- Audit log: all state changes, phone calls, comments, AI analysis results
- Links: Clinic, IM topic, Jira issue (if exists)

## External Dependencies

| Dependency | Purpose | Required | Abstracted |
|-----------|---------|----------|------------|
| MySQL | event + alert + audit + subscription storage | Yes | No |
| IM (Lark phase 1) | topic creation, forwarding, bot, read receipts | Yes | Yes (IMProvider) |
| Internal customer system | KA lookup for classification override | Yes | No |
| Clinic API + ActivityHub | rule-scoped classification override, cluster info enrichment, portal deep link | No (fail-open) | Yes (Clinic client) |
| DevOps ops portal | tenant biz region + internal/external kind | No (fail-open) | Yes (`pkg/devops`) |
| Oncall v2 service | duty schedule lookup (phone calling later) | No (fail-open) | Yes (`pkg/oncall`) |
| Core-server user directory | name/email search behind the reassign picker | No (fail-open) | Yes (`pkg/directory`) |
| Jira API | create/update issue, status transition (compatibility) | Transitional | No (designed to delete) |
| feishu_messager_v2 API | phone calls (Ali/Twilio) | Transitional | No (will internalize) |
| Alertmanager | webhook source | Yes |

Clinic authentication is injected through `CLINIC_TOKEN`; ActivityHub uses
`EVENT_KEY`. Neither value belongs in Git. The ops portal needs no credential
in-cluster (see "Tenant enrichment"); `DEVOPS_API_TOKEN` exists only for a
deployment reaching it through the shared gateway. The dev deployment points at jiralert
prod's ActivityHub endpoint, chosen while it mirrored the jiralert prod feed.
That forward is off now (see "Jira Mirror"), so the endpoint is worth revisiting.

## Deployment: one alertstore per environment

One instance per env (dev / staging / prod), same as jiralert today —
env is implicit in deployment, never a label or code concept.

- Blast-radius isolation: a dev alert flood can't slow prod paging;
  prod's instance runs on prod-grade infra and DB.
- The dev instance doubles as the test bed for alertstore itself:
  releases land there first, exercised by real dev alert traffic against
  a sandbox IM group, then promote to prod.
- dev/staging need no "record-only mode" — they run the full pipeline
  into a sandbox group nobody is paged in.
- Cross-env aggregate metrics come from Prometheus scraping each
  instance's `/metrics`, not SQL.
- Revisit only if a cross-env unified view is ever needed (e.g. non-oncall
  subscriptions spanning envs); the change would be an aggregation layer,
  not a merge.

## Phase 1 Scope

### Build

- Event store (write + query API)
- Ticket state machine (full lifecycle)
- Classifier (label + KA override + clinic override)
- Delivery: im_topic + im_forward + the Jira mirror (webhook_out via Jira webhook naturally)
- Delivery: phone via feishu_messager API
- Ack ingress: web API + Jira webhook + Lark bot
- Subscription: CRUD API
- Oncall service integration (read oncaller)
- Web UI: event list + alert list + alert detail
- Response/recovery deadline reminder: notify oncaller in IM topic when approaching deadline

### Don't Build

- Auth / SSO (user self-identifies)
- Dashboard / charts
- Phone internalization (keep feishu_messager)
- Bot command for subscription management (API only)
- Historical data migration
- SLA calculation
- Complex escalation chains
- Alertmanager silence sync
- Webhook out (rely on Jira webhook in phase 1)
- Auto resolve
- call_delay + phone (phase 2, along with phone internalization)