# Wetech Cloud Alert Rule Retrieval

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "wetech-cloud-alert-rule-retrieval-2026-08-20",
  "problems": [
    {
      "problemId": "wetech-cloud-memory-alert-semantics-not-retrieved",
      "summary": "A Wetech Cloud TiKV memory-warning diagnosis can compare process RSS with advertised node size because the Scenario Router is not given the applicable customer alert calculation, even when the ticket facts distinguish a node maximum from the cluster average.",
      "status": "candidate",
      "applicability": "Wetech customer TiDB Cloud tickets with a non-empty orgId that ask why a maximum or total TiKV node memory alert fired.",
      "issueKeys": ["ST-2865"],
      "sourceRefs": ["https://tidb.atlassian.net/browse/ST-2865#problem:wetech-cloud-alert-rule-semantics-not-retrieved"],
      "signalTextIncludesAny": ["TiKV memory warning", "MaxTiKVMemoryUsageTooHigh", "TotalTiKVMemoryUsageTooHigh", "RSS divided by node memory"],
      "observedThrough": "2026-08-20",
      "effectiveAt": null,
      "verifiedAt": null,
      "attemptedAction": "Use the exact Wetech-managed alert name or unambiguous alert text plus the typed Cloud boundary as Router applicability when Jira Account Alias is absent; require reconciliation to compare only equivalent quantities and treat an unproven below-threshold historical reconstruction as a Support-owned gap rather than a reason for human_required.",
      "evidenceUrl": "https://github.com/tidbcloud/runbooks/tree/master/rules/customer"
    }
  ]
}
-->

- Optimization ID: `wetech-cloud-alert-rule-retrieval-2026-08-20`
- Owner: repository product knowledge selected and applied by Scenario Router
- Source reference: `ST-2865` and `tidbcloud/runbooks/rules/customer`
- Implementation status: follow-up implemented and tested; deployment pending
- Effective status: `candidate`

## Bounded problem

The ticket asks why a Wetech Cloud TiKV memory warning fired. The incident
facts can distinguish a per-node maximum from the cluster average, but the
Router does not currently receive the matching customer-alert calculation and
can compare an unrelated process RSS value with the advertised node size.

## Candidate change

- Recall compact candidates for the maximum and total TiKV memory alerts.
- Require an explicit alert or warning signal; ordinary TiKV memory analysis
  does not recall alert-rule knowledge.
- Keep Wetech Cloud applicability and Max-versus-Total selection with the
  Scenario Router.
- Load only selected rule content; do not send every customer alert rule to an
  Agent.
- Keep alert-trigger semantics separate from incident root-cause attribution.

## Verification status

Focused contract tests and the full 926-test suite pass. In an offline
ST-2865-style real-model routing simulation, bounded recall supplied both Max
and Total candidates; the Scenario Router selected only
`wetech-cloud-max-tikv-memory-alert` from the per-node maximum 82.94% and
cluster-average 76% facts. The selected-topic loader then materializes the Max
topic without the Total formula. A separate APID/Wetech OP simulation received
the same recalled candidates but selected no Cloud alert topic.

The deployed `ST-2865` replay completed on 2026-08-20 as run
`2f6afbc7-7c7e-434e-a687-ba5256df26a5`. Both alert-rule topics were materialized
in the workspace knowledge index, but an exact selector reproduction with that
ticket text returned no bounded candidate, and the Scenario Router returned
`productKnowledgeRefs=[]`. The final answer was materially better because the
Provider evidence established the aligned cgroup utilization and headroom, not
because this rule-retrieval action was applied. This attempted action therefore
must not receive optimization credit or suppress the unresolved knowledge
selection signal.

The same replay's historical `ticket.json` retained `orgId` and `clusterId`
but had no Account Alias. A direct Jira field read confirmed that
`customfield_10095` is absent on the source issue; the replay sanitizer already
preserves Account Alias when it exists. The missing value is therefore not a
workspace handoff loss and must not be fabricated. The follow-up action uses
the exact managed alert identity already present in the customer question as
the bounded Wetech-topic applicability signal.

An exact selector simulation then exposed a separate literal recall mismatch:
the Jira alert says `Max memory utilization across TiKV nodes exceeded...`
without the word `alert`, while the old keyword required `nodes alert`; the
Chinese summary also omits the space in `TiKV内存告警`. The bounded keywords now
include these exact alert forms. A generic TiKV memory value without an alert
name or alert-text form still recalls neither rule.

After the keyword and applicability correction, a real-model Router simulation
used the original ST-2865 alert text, the Cloud `orgId` and `clusterId`, and no
Account Alias. Bounded recall returned the Max and Total candidates, and the
Router selected only `wetech-cloud-max-tikv-memory-alert`. The Router thread was
`01a01eb8-9e36-70e2-b11a-19328790095a`.

The correction was deployed in staging image
`1ab7a9b3cff70fc99d79965bc2dacf8d0407f4ee`. Read replay
`309e120e-df3c-494d-a60b-6a49a5c59a24` selected only the Max topic, returned
`internal_only`, kept process RSS separate from the rule's host-memory
quantity, and was judged `ai_better` with no AI gaps. Write replay
`143249dc-5a07-4618-8324-34e66c5ada90` also selected only the Max topic, but a
different three-scenario route treated a 78.58% Provider reconstruction as a
contradiction to the frozen `>80%` firing event and returned `human_required`.
That decision was `not_compared`, so the bounded optimization remains a
candidate until the reconstruction-gap decision is made stable.

The write replay updated Jira internal comment `23432370`; the already judged
`ai_better` read draft was then restored directly to that exact comment. Jira
readback confirmed `public=false` and the expected host-metric boundary. No
public comment or issue transition was performed.

A branch-local real-model reconciliation simulation then reused the write
replay's exact three Scenario Diagnostic Agent decisions and compact round-two
handoffs. With the historical firing-event boundary added, the Router returned
`internal_only`, kept `requestedContext=[]`, treated the 78.58% reconstruction
as a Support-owned semantics check, and did not recommend expansion. The
Router thread was `01a01ed9-4a73-7392-bd53-e4bb7dcad8dd`.

On the follow-up branch, a real-model simulation reused the frozen historical
cutoff and the already materialized Clinic facts. The Router selected only
`wetech-cloud-max-tikv-memory-alert`, ignored the RSS-ranked db-tikv-43 as an
alert-node substitute, and returned `internal_only` even though the frozen Jira
context contained `Escalated`. It kept one correct remaining boundary: the
db-tikv-9 cgroup working-set series is not the rule's host-memory calculation,
so it cannot prove the exact firing node or duration. The Router and Final
Response thread ids were `01a01e8d-78c4-7aa2-82d6-348f87313030` and
`01a01e8e-864f-7a03-a60c-04be903bf57b`.

## Unresolved gaps

- The actual configured threshold and duration remain owned by the alert event.
- The corrected deployed Router selects the Max rule, but the decision is not
  yet stable when a later incomplete reconstruction falls below the historical
  alert threshold.
- The alert rule does not identify the workload or allocator that caused memory
  growth.

## Next validation action

Run the full contract suite, deploy the historical reconstruction-gap decision,
then repeat the case-specific historical ST-2865 replay more than once. The
criterion is that every replay selects the Max topic, keeps cgroup/process and
host-memory quantities distinct, does not use RSS ranking to identify the
alerting node, and remains `internal_only` when an unproven reconstruction is
below the frozen firing threshold. Refresh the mapped Jira internal comment
with the passing draft before promoting this problem beyond `candidate`.
