# Active Service Impact Escalation - Optimization ID: `active-service-impact-escalation-2026-07-24` - Owner: auto-oncall-services - Source reference: `APID-29230` `/ao wrong` - Implementation status: merged through `tidbcloud/aisupport#498` and available in the staging runtime by the `2026-07-27` Focus baseline - Effective status: `optimized_unverified`; the runtime replay preserved impact and human escalation, but the stable Jira comment refresh is still pending ## Bounded problem The `APID-29230` analysis treated generic host/network reachability as evidence that the incident was a limited network fluctuation. It did not first reconcile the affected TiDB instance's zero connections and lost request traffic with the node hang and the cluster/business QPS decline. That understated both customer impact and the need for immediate human production handling. This optimization adds one shared incident contract. Active host, network, and service incidents must first align per-instance TiDB connections, QPS/request traffic, and process/service state with blackbox/probe, load-balancer, and cluster/business outcomes against a nearby pre-incident baseline. Probe or ICMP success cannot override contradictory TiDB service outcomes, and non-zero aggregate traffic cannot hide a material relative decline. Disappearance of an instance's previously present service metrics plus sustained scrape loss is a service-layer contradiction rather than a benign monitoring gap unless there is independent monitoring-only evidence. When it aligns with a material cluster/business QPS decline, the incident must be labeled at least High and `human_required`, with immediate human escalation and no automated production operation. ## Verification evidence - Contract tests cover the bundled OP diagnosis skill and the shared dry-run, live initial, continuation, and Jira draft prompt surfaces. - The replay used source job `90004`, source run `90001`, and the original historical cutoff `2026-07-23T14:11:33.221Z`. Only reply-safe customer comment `23276383` was visible; later comments, including the `/ao wrong` feedback, were excluded. - The first valid local candidate replay, `2a732e77-d9fa-4081-b6c6-36e3edb3852c`, failed the criterion. It returned Medium and `internal_only`: although it observed continuous scrape loss and missing service metrics for `10.107.40.177:10080`, it treated them as a monitoring gap and used remaining non-zero aggregate QPS to understate the impact. - One minimal reinforcement added the pre-incident baseline and relative-QPS rules. The only rerun, `e4a203fe-2be2-47dd-9416-298dc85688b0`, passed at `2026-07-24T11:31:27+08:00`: it returned High and `human_required`, requested immediate human escalation, and kept all automated activity read-only. - The passing replay aligned the affected instance's sustained `up=0`, missing connection series (previously 76–90), and missing per-instance query series with cluster QPS. Cluster QPS averaged about 79,855 before the incident and 43,530 afterward (a 45.5% decline), reaching about 20,511 at the cutoff (74.6% below the value immediately before onset). - Generic probe success remained almost entirely 1. The replay correctly identified that as a contradiction, not proof of TiDB service health. Direct load-balancer/backend state was unavailable through the bounded read-only transport and was explicitly left for immediate human validation. - The candidate OP skill loaded by the passing replay exactly matched the branch (`sha256` `ba04e4b588bfef4b683eb975f02b9233c0031385cb2bc5dea330741116594e0a`). The Wetech provider skill likewise matched its source (`sha256` `f21d95371c22ea4f3253702612a56ee98299a82f105cc6c9dc5f2854f60f955d`). - Passing result and evidence are retained locally under `/tmp/auto-oncall-impact-replay-20260724.isrDiu/runs/e4a203fe-2be2-47dd-9416-298dc85688b0/`. No staging deployment, Jira write, or production operation was performed. ### NLB/backend evidence capability audit At `2026-07-24T12:05:53+08:00`, a follow-up audit queried the exact `Order01` Prometheus datasource (`uid=GKCZSzGvz`) over the original bounded window `1784813400`–`1784815893`. The read-only `browser-query-range` checks and results were: - `count by(__name__,job) ({__name__=~".*(nlb|elb|load.?balanc|backend|target.?health|healthy.?host|unhealthy.?host).*"})` returned only 19 PD/etcd disk-backend metric series. None represented a load balancer or target health. - `count by(__name__,job) ({__name__=~".*(networkelb|applicationelb|aws_elb|target.?group|haproxy|envoy|nginx_ingress|traefik|upstream).*"})` returned only three TiFlash raft-upstream latency metric series. - The Alibaba/cloud LB family check covering `aliyun`, `alicloud`, `slb`, `alb`, `clb`, `nlb`, load-balancer, server-group, backend-server, and healthy/unhealthy-server names returned zero series. - Raw `probe_success{instance="10.107.40.177:10080"}` had only `group=tidb`, `job=tidb_port_probe`, and the instance address. It is a direct component-port probe, not NLB/backend health, and has no load-balancer, listener, target-group, or backend identity label. - A broad exporter-job count timed out with HTTP 504 and was not retried. The bounded replacement, `count by(job) (up)`, succeeded and exposed only component, monitor/node, and per-host blackbox/port-probe scrape jobs; it did not expose an AWS/Alibaba cloud or load-balancer exporter. Audit evidence is retained under `/tmp/wetech-nlb-audit-20260724.nmJl8Z/`. The current datasource therefore cannot provide an auditable NLB/backend-health observation that can be joined to `10.107.40.177:10080` at the historical cutoff. This is an external provider/monitoring-coverage blocker, not an auto-oncall evidence-selection rule defect. No additional replay was run, preserving the already-passing High, `human_required`, immediate-escalation, and read-only result. ## Unresolved gaps - The change is effective in staging but not yet fully verified through the required Jira write/readback step. - The Wetech Prometheus datasource does not expose NLB/backend-health metrics with labels that can be joined to the affected TiDB instance at the original cutoff. The provider/monitoring owner must first export target health and a stable load-balancer/listener/target-group-to-instance mapping before auto-oncall can consume this dimension. - The local replay could not directly verify the deeper host/network/process trigger. - The stable Jira internal comment has not been refreshed or verified by comment ID or idempotency key. - Runtime behavior against a future active incident remains unverified. ## Next validation action Rerun `APID-29230` at its original historical cutoff after the current fault-domain change is deployed. Confirm the same High or Critical, `human_required`, immediate-escalation, and read-only behavior. Then update the existing `auto-live:v1:APID-29230:initial` internal comment in place and verify it by comment ID before promoting this problem to `optimized_verified`. Track the NLB dimension separately with the Wetech monitoring/provider owner: expose historical target-health state plus stable listener/target-group/backend identity labels, then validate one bounded query that maps the target to `10.107.40.177:10080`. Do not block the service-impact escalation on that external handoff.