# Wetech Metric Cardinality Bound

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "wetech-metric-cardinality-bound-2026-08-12",
  "problems": [
    {
      "problemId": "wetech-high-cardinality-prometheus-response-memory-amplification",
      "summary": "A broad per-instance and per-thread Wetech Prometheus query returned about 90 MB and 3.4 million points, causing multiple decoded and summarized copies to drive APID-29259 replay memory to about 1.73 GiB.",
      "status": "optimized_verified",
      "issueKeys": ["APID-29259"],
      "sourceRefs": [],
      "signalTextIncludesAny": ["high cardinality", "grafana_query_result_too_large", "Prometheus response", "APID-29259"],
      "observedThrough": "2026-08-12",
      "effectiveAt": "2026-08-12T06:49:36Z",
      "verifiedAt": "2026-08-12T07:45:00Z",
      "attemptedAction": "Require bounded metric label cardinality and reject Wetech Prometheus responses above 8 MiB before JSON decoding, preserving the exact gap for one narrower refinement.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/tree/main/auto-oncall-services/docs/opts/2026-08-12-wetech-metric-cardinality-bound.md"
    }
  ]
}
-->

- Owner: auto-oncall-services
- Source: APID-29259 replay `0e67e865-4e0d-40cb-8881-cacbe2acf451`
- Status: optimized_verified

The peak is explainable but unnecessary: the helper held raw bytes, decoded
JSON text, nested Python values, numeric summaries, and serialization buffers
for a result whose final handoff needed only bounded leading facts.

The Evidence Agent now aggregates irrelevant labels or uses top-k, while the
Provider Executor rejects an oversized response before decoding. Deployment
APID-29259 replay `669ba399-75a4-4818-bb54-739e5df24efc` executed bounded
Prometheus queries whose raw metric artifacts were generally tens to hundreds
of KiB instead of the prior 90 MB response. The isolated 2 GiB Pod peaked at
1,141,694,464 bytes (about 1.06 GiB), down from about 1.73 GiB, with no OOM.