# AI-Native Alert Platform Date: 2026-08-19 Revised: 2026-08-20 Status: proposal Language: This document uses ASD-STE100 Simplified Technical English. ## 1 Purpose This document tells how to change alertstore. Today alertstore is an alert platform for persons. It must become an alert platform for persons and agents. ## 2 Technical names | Name | Meaning | |---|---| | alertstore | The service that keeps all operational alerts. It is the single source of truth. | | agent | A program that uses a large language model to do work on an alert. | | oncaller | The person who has responsibility for an alert. | | verdict | The answer that tells why an alert closed. The three values are `handled`, `self_recovered` and `noise`. | | auto-close | A change of an alert to `resolved` that alertstore makes without a person. | | quiet period | The time for which a signal must stay clear before an auto-close. | | guard | A control that acts by itself. It does not need a person to look. | | raise | The action that increases the urgency of an alert. | | outcome data | The facts that come later and show if an answer was correct. | | amendment | A change of the verdict on an alert that is already `resolved`. | ## 3 Background Two facts control this design. **Fact 1. All of the present alertstore is from the time before agents.** The classification `page` / `ticket` / `logging` tells how much attention a person must give. The lifecycle `open` / `acked` / `resolved` tells if a person holds the alert and if a person closed it. The verdict tells what a person did. **Fact 2. The `ticket` queue is larger than the capacity of the persons.** More `ticket` alerts come in each day than the oncallers can examine. Many are never examined. For the `page` class the oncallers already use agents of their own, outside alertstore. Fact 2 is the more important one. It shows that the two classes have different problems, and that they need different designs. ## 4 Principle An AI-native alert platform is different from an alert platform with an agent at the side. The difference is this: **safety comes from guards that act, not from processes that need a person to look.** A process that needs regular attention decays, or it becomes a ceremony. The platform operates in a system where attention is the most rare resource. Any safety mechanism that spends attention will fail. ## 5 Rules ### Rule 1 — Two regimes, one for each class | | `page` | `ticket` | |---|---|---| | Attention today | Sufficient | Insufficient | | Comparison | A careful person | Nobody | | Risk | An agent answer changes what the person thinks | Nothing is examined and nothing is measured | | First goal | Give evidence early | Process the queue | | Human deadline | For each alert | For the queue only | Do not design one mechanism for both. The `ticket` class needs the agent to be the primary processor. The `page` class keeps the person as the primary processor until measurements show more is safe. ### Rule 2 — Keep the classification axis, but change its meaning Do not change the values `page`, `ticket` and `logging`. Do not change the 402 Prometheus rules that carry the `response_class` label. The axis gives a time limit for responsibility. It tells this: if no processor closes the alert in the time limit, alertstore must interrupt a person. - `page` — the limit is minutes. - `ticket` — the limit is the queue, not the single alert. See Rule 8. - `logging` — there is no limit. With this meaning the person is the last defence, not the only processor. ### Rule 3 — An agent has authority over the record, never over communication An agent has no verb that sends a message and no verb that makes a telephone call. An agent changes facts about an alert. The delivery layer then makes the notifications, with the rules that control it today. A direct-message verb would be a second notification channel. That channel would have no rate limit, no duplicate control and no state. Divide the record verbs into two groups: - **Add-only verbs** — give evidence, write a note, `raise` the urgency, and set the assignee while the alert is `open`. These verbs cannot lose an alert. **They need no measurement and no grant.** - **Remove verbs** — auto-close, and a decrease of urgency. These verbs can lose an alert. They need the guards in Rule 6. An agent must never write the verdict `noise`, by a close or by an amendment. The `noise` verdict is the input to the rule-quality reports. A machine that marks its own input as noise makes a loop with no person in it. ### Rule 4 — The assignee is always a person Write `assigned_to` at creation, from the oncall schedule. Never write an agent into that field. The field answers one question: who has responsibility if this goes wrong. An agent cannot hold responsibility, and an agent that stops leaves the alert with no owner. Assignment is not notification. On a certified alertname the person is the named last defence. If the agent finishes first, that person is never disturbed. Two conditions come with this rule: 1. At a shift change, reassign all `open` alerts that no person has acked. Without this the tiers send an alert to a person who is off shift. 2. Send each person a summary of the agent activity in their name. A person with fifty closes they never saw does not have responsibility. That person has blame. ### Rule 5 — The signal closes the alert, not the agent An agent may close an alert only when all of these are true: 1. Alertmanager shows all bound instances as resolved. 2. The signal stays clear for the quiet period. 3. No person has acked the alert. 4. The recurrence guard does not stop the close (Rule 6). "The signal still fires, but I judge it harmless" is a judgment. Never automate it. These conditions are deterministic. **An auto-close needs no judgment from a large language model.** What alertstore approves for an alertname is a simple behaviour: close when the signal stays clear. The agent supplies the verdict label and the narrative. Therefore an auto-close is easy to reason about, easy to record and easy to stop. Get the quiet period from the history: take a high percentile of the times between a clear signal and the next fire of the same alert. ### Rule 6 — Guards, not review Do not make the first release wait for a review by a person. Persons who have no time to examine `ticket` alerts also have no time to review them. A design that waits for that review never starts, and it collects no history. Use these guards. Each one acts by itself: 1. **Recurrence guard.** If the same aggregation key has N auto-closes in M days, the next occurrence goes to a person instead. 2. **Re-open as escalation.** If an alert fires again after an auto-close, alertstore does not make a new alert. It sets the alert to `open`, it sends the alert to a person, and it counts one failure for that alertname. 3. **Rate guard.** If the auto-close rate for an alertname increases sharply, stop the auto-close for that alertname. 4. **Amendment guard.** If persons amend many closes of one alertname to `noise`, stop the auto-close for that alertname. See Rule 7. 5. **Raise limit.** An agent has a maximum number of raises for each day, and it removes duplicates by aggregation key. 6. **One switch.** One control stops all auto-closes. Approve an alertname for auto-close with one offline query over the history. A named person reads the result and makes the decision one time. Do not build scorecards, a review board or a grant that expires on a date. Remove an approval when a guard trips, not when a calendar day arrives. ### Rule 7 — A person must be able to amend a verdict, in bulk An agent closes only as `self_recovered`. Without an amendment path the `noise` signal disappears, and bad rules stay hidden behind green reports. - An amendment changes `alert.resolution`. The state stays `resolved`. The amendment is an audit record with the old value, the new value, the person and the reason. - **The amendment must work on many alerts at one time**, selected by the same filter format as the subscriptions. One alert at a time is of no use in a queue that nobody reads. - Put the control in the daily summary of the auto-closes, next to each alertname. Two clicks must be sufficient. - An amendment must not overwrite `agent_run.verdict`. The later reports compare the agent answer with the outcome, so the agent answer must stay. A bulk amendment is also the cheapest source of human labels for the `ticket` class. One person gives forty labels in two minutes. ### Rule 8 — The `ticket` class has a queue deadline, not an alert deadline A per-alert deadline on a `ticket` alert is not true. The queue is larger than the capacity, so the breach rate would be near 100 percent. Measure the queue: how long a dangerous item waits before a person sees it, and how old the backlog is. The agent is the part that puts the queue in order. The ladder of Rule 9 is `page` machinery. The `ticket` class does not use it, and the first release does not change it. ### Rule 9 — A rule may ask for a window before the first call The `page` ladder is built. Eight steps go out through the delivery outbox, and each step gets its target from the oncall roster: | Step | Offset from step 0 | Target | |---|---|---| | 0 | 0 | oncaller | | 1 | +8 min | oncaller | | 2 | +9 min | oncaller (a 1-minute gap defeats Do-Not-Disturb) | | 3 | +17 min | backup, or the oncaller | | 4, 5 | +25, +33 min | leader | | 6 | +34 min | leader (the same 1-minute gap) | | 7 | +42 min | fire_master | Each step reads the alert state before it calls, so an `ack` stops the rest without a separate cancel. A reassign by a person cancels the pending steps and starts a new generation at step 0. The sequence was sized against a one-minute wait before the first call. Step 0 never implemented that wait, so every later step ran one minute later than it was meant to. The offsets above are the intended ones. Some conditions clear on their own in the first minutes. The author of the rule knows which ones, so a rule may ask for a window before step 0. With `page_delay: 1m` the ladder reaches the absolute times it was designed for: ```yaml annotations: page_delay: 5m ``` Four decisions come with it: 1. **An annotation, not a label.** A label joins the identity of an alert. To add one to a rule changes the fingerprint of every alert that rule produces, and the fingerprint is what occurrence dedup and the aggregation key are built on. This value never reaches the routing or the grouping in Alertmanager, so it stays out of the identity. 2. **The window is measured from the alert**, not from the moment the delivery worker starts. A slow pipeline must not extend a window that someone sized against the condition. Time that ingest and the lookups spent counts against the window. Note that the `for:` clause of the rule is separate and comes before all of this. 3. **Absent means zero**, which is what every rule does today. Do not annotate the 402 rules. Annotate the few where the author knows the condition clears. An unusable value also means zero: a mistake in an annotation may lose a delay, but it must not stop a call. 4. **A cap in the code bounds it.** Two reasonable numbers compose into an unreasonable one, when a later agent shift adds to what the rule asked for. This rule is the pattern for Rule 2 in the small: the urgency of the condition is what the rule author knows, so it goes in the repository with the rule. The agent shift is a property of agent quality, it changes each week, and it stays in alertstore as data. ### Rule 10 — Show the agent answer An earlier version of this document held back the agent verdict, to keep the human answers independent. That rule is now removed. The oncallers already use their own agents for `page` alerts, so the answers are already not independent, and alertstore does not even record it. To hold back the verdict pushes the person to a channel outside the platform. Show the evidence and the verdict on the card, for both classes. Measure with outcome data instead (Section 6). Keep only the blind order in the review sample: hold the agent answer until the person answers. Do not put the agent verdict into the resolve dialog as a default value. ### Rule 11 — Approvals are the certification for an action An action that changes the world — a restart, a scale, a failover — is not a record verb and not a communication verb. It is a third group. Statistics are necessary only for an action with no person present. **For an action with a person present, the approval of the person is the certification, one time for each instance.** Therefore an approval-gated remediation needs no measurement program. It needs three objects: - an **approval request**, which the directed tier carries as its content; - an **action record**: what ran, on which target, under which approval, with what result; - a **target budget**: two agents must not restart the same store at the same time, and no target accepts more than a set number of actions in an hour. Key the budget on the target, not on the alert. Rules for an approval: - An expiry means "not approved". Never approve by a timeout. - To approve implies `ack`. The person took responsibility for a change to production. - To reject has two forms: **reject and take** (this acks and stops the ladder) and **reject only** (the ladder continues and carries the reason). - An approval is an authority, not an execution. Check the conditions again at the time of the execution. ### Rule 12 — Do not add states. Add tracks Keep `open` / `acked` / `resolved`. `acked` keeps one meaning: a person took responsibility. There is no agent ack. Four tracks answer four questions: | Question | Where | |---|---| | Who has responsibility, and did a person take it | `alert.state`, `assigned_to`, `acked_by` | | Does a processor run now | `agent_run.status` | | Does a decision wait for a person | `agent_approval.state` | | Where is the ladder (`page` only) | `escalation_step` | Join the tracks in the view layer, not in a state value. "Waits for approval" in a list is a mark that a query calculates. It is not a stored state. ## 6 Measurement Do not measure with the agreement between an agent answer and a person answer. On the `page` class the persons already see agent answers from other tools. On the `ticket` class most alerts get no answer from a person at all. Measure with data that does not come through a click: 1. **Outcome data.** For each closed alert, the world gives the answer in a few days. Did the alert fire again? Did the same aggregation key become a `page`? Was the alert re-opened? Did an SLO fall, or did a support ticket arrive in the same window? An agent that calls a condition harmless, and that condition pages three days later, is wrong. No human label is necessary. 2. **The raise loop.** A raised alert always gets attention, so a raise makes its own label. The agent supplies ground truth for the group it calls dangerous, and the outcome data covers the group it passes. 3. **Bulk amendments** (Rule 7). Few, cheap, and per alertname. 4. **The review sample.** About five agent-closed alerts each day go into the daily summary, with the agent answer held until the person answers. This finds large errors. It is not a statistic for each alertname. 5. **The silent-miss rate.** Count the incidents and the support tickets that have no alert. Join the support records to the alert records by cluster and time window. A system that lowers its own sensitivity must measure what it stops sending. **This measurement keeps all the others honest.** Write the outcome report as a batch job that runs each week. It is a query, not a platform. The data collects by itself, so this job can come later — but the records it reads must start on the first day. ## 7 Data model ```sql -- One row for each run of an agent on an alert. CREATE TABLE agent_run ( id BIGINT AUTO_INCREMENT PRIMARY KEY, alert_id VARCHAR(32) NOT NULL, agent VARCHAR(64) NOT NULL, trigger_kind VARCHAR(32) NOT NULL, -- auto / manual / scheduled session_ref VARCHAR(255) NOT NULL DEFAULT '', -- deep link to the agent session status VARCHAR(16) NOT NULL DEFAULT 'dispatched', verdict VARCHAR(32) NOT NULL DEFAULT '', -- same values as the person verdict summary VARCHAR(2048) NOT NULL DEFAULT '', detail JSON, -- evidence, quiet period, rule used started_at TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), finished_at TIMESTAMP(3) NULL, INDEX idx_agent_run_alert (alert_id, started_at) ); ALTER TABLE alert ADD COLUMN agent_verdict VARCHAR(32) NOT NULL DEFAULT ''; ALTER TABLE alert ADD COLUMN agent_summary VARCHAR(2048) NOT NULL DEFAULT ''; ALTER TABLE alert ADD COLUMN resolved_by VARCHAR(255) NOT NULL DEFAULT ''; -- user:x / agent:y ALTER TABLE alert ADD COLUMN reopened_count INT NOT NULL DEFAULT 0; ``` Do not add a subscription table for agents. Add a channel kind to the subscription table that exists, so a team connects its own agent through the API that exists: subscription.channel_kind = im | webhook Add a delivery task kind `agent_dispatch`. It sends a webhook and it uses the retry logic of the delivery outbox. Later releases add `agent_approval` and the action record of Rule 11. Do not build them before the first remediation is real. Do not put an autonomy value in any table before the guard for that value exists. ## 8 Interface ``` POST /api/v1/alertstore/alerts/{id}/agent-runs -- start a run PATCH /api/v1/alertstore/alerts/{id}/agent-runs/{run_id} -- write the result POST /api/v1/alertstore/alerts/{id}/timeline -- kind=agent_note POST /api/v1/alertstore/alerts/{id}/raise -- increase the urgency POST /api/v1/alertstore/alerts/amend-resolution -- bulk amendment, by filter ``` Give each agent its own token. The token supplies the identity for the audit records. Add `agent` to the list of audit sources. ## 9 The first release Build these six items: 1. **The rails.** The webhook channel kind, the `agent_dispatch` task kind, the `agent_run` table, the `agent_note` timeline kind, and the agent tokens. All of these are small changes that use the machinery that exists. 2. **One triage agent** on the resilience-agents-platform. It reads every `ticket` alert and every `page` alert. It gets the runbook from `runbook_url`. It does a read-only diagnosis. It writes the evidence, the summary and the verdict to the card. It calls `raise` when the condition is dangerous, with a rate limit. 3. **Auto-close for the `ticket` class**, with the conditions of Rule 5 and the guards of Rule 6. Start with the alertnames where the history query shows the largest margin. Watch them for two weeks. Then add more. 4. **A number and a switch.** A daily summary of the auto-closes, by alertname, in the group topic. One control that stops all auto-closes. 5. **Bulk amendment**, with the control inside that daily summary. 6. **A complete record of each auto-close**: the evidence, the quiet period, the rule that ran. This cannot be collected later. Exclude these from the first release: - All further ladder work: the recover phase (Section 11), and the shift of step 0 that an agent grant would supply. The `page_delay` of Rule 9 has landed, but it is a value the rule author sets. No agent moves it. - The authority to decrease urgency. - All remediation, including the approval-gated form. It is next, not first. - Grants that expire, scorecards, and a review queue. - Structured runbook data. The first agent reads the free text, and it will show which runbooks are bad. The first release pays for itself alone. If nothing else ever ships, the `ticket` queue gets processed, the dangerous few reach a person, and the backlog falls in an honest way. ## 10 The end state - **`ticket` class: the agent is the primary processor.** Persons work from a summary that the agent puts in order, against a queue SLO. Auto-close keeps the queue honest. The outcome report runs each week. - **`page` class: the person is the primary processor**, until the measurements show that more is safe. The agent runs from creation and gives evidence in the first minutes. One tap approves the common runbook actions. - **Rule health closes the loop.** The shape of the data proposes a rule change; a join against the impact data decides it; the silent-miss rate stops the loop from a trade of recall for quiet. - **Safety is guards, outcomes and a small amount of directed attention.** There is no review board and no calendar. A switch has an owner and a link to its evidence. A guard removes the switch when the evidence changes. ## 11 Limits of this document - Rule 6 re-opens an alert. The design today makes a new alert for each fire and never re-opens a closed one. The interaction with the aggregation key and the stale window needs more work. - The recurrence guard counts by aggregation key, and that key has a version. A change of the version will reset the count without a warning. Map the versions, or lower the approvals when the version changes. - The ladder of Rule 9 has no recover phase. An `ack` stops every remaining step, and nothing arms after it. A person who acks at 03:00 to stop the telephone, and then does nothing, leaves an alert that no mechanism touches again. The ladder measures the time to a person and has no measure of the time to a resolution. The repair is one mechanism with two phases: an `ack` retires the first phase and arms a second one, which only a `resolve` stops. This is not agent work, and it is the largest hole in what exists today. - The ladder is one hardcoded sequence for all `page` alerts. Per-team or per-alertname sequences are not needed yet. Do not make them configurable before a team asks. - The silent-miss measurement needs support data joined to alert data. That join is not built.