{
  "version": 7,
  "last_updated": "2026-07-08",
  "purpose": "Catalog of known bad inference patterns from automated or rushed analysis. After forming a conclusion, match tags and reasoning; if the conclusion fits a pitfall without new disambiguating evidence, re-analyze.",
  "entries": [
    {
      "id": "ic-001",
      "title": "Treating ANALYZE as the primary root cause from TopSQL / Slow Query",
      "example_cases": [
        "O11Y-2328599"
      ],
      "wrong_pattern": "ANALYZE appears in TopSQL or Slow Query (or ranks high by count or latency), so you conclude ANALYZE is the main bottleneck or root cause.",
      "why_wrong": "ANALYZE is common during stats collection and a single statement can be slow, but it is usually not the main application workload driver. If you do not deprioritize ANALYZE when reading TopSQL/Slow Query and instead treat ANALYZE as the primary suspect, you are likely to mis-attribute the issue.",
      "correct_procedure": "When analyzing TopSQL and Slow Query, ignore ANALYZE rows first: prioritize hotspot and slow-query attribution for non-ANALYZE statements. Only after ruling out other SQL, separately assess whether stats work is materially consuming resources. Separately, in the written root-cause paragraph, do not lead with ANALYZE as the primary cause unless disambiguating evidence shows it dominates the hotspot.",
      "conclusion_rule": "The final root-cause statement must not name ANALYZE (or auto-analyze / stats-collection workload) as the primary root cause by default. Primary cause should be attributed to application SQL, execution plan, or schema (e.g. missing index, bad plan) unless you prove ANALYZE dominates TiKV/TiDB hotspot CPU or latency versus non-ANALYZE digests.",
      "corrective_checks": [
        "Before writing the summary: confirm the primary cause is not ANALYZE unless evidence shows it dominates hotspot metrics vs application queries",
        "In TopSQL / Slow Query results, deprioritize ANALYZE rows when scanning; examine CPU and latency for other digests / statements",
        "Use TopSQL CPU share on hot instances to attribute non-ANALYZE statements, then build the causal chain",
        "Do not name ANALYZE as the primary cause from SlowQuery row counts alone or merely because ANALYZE appears in the list",
        "If large est/act divergence appears on a join/filter column, also run ic-002 checks before listing ANALYZE as P0 remediation"
      ],
      "tags": [
        "analyze",
        "topsql",
        "slow-query",
        "cpu",
        "o11y-2328599"
      ]
    },
    {
      "id": "ic-002",
      "title": "Stale statistics / ANALYZE as P0 while missing index is the structural root cause",
      "example_cases": [
        "O11Y-2328599"
      ],
      "wrong_pattern": "decoded_plan shows huge estRows vs actRows divergence, IndexFullScan/IndexLookUp, or you infer stale statistics — so you make \"stale statistics\" the headline root cause and put ANALYZE TABLE as the only or first (P0) remediation, while treating missing index on the bottleneck table/column as optional or P2.",
      "why_wrong": "Large divergence often coexists with bad access paths. If the selective predicate or join key is not backed by a suitable index, the dominant cost may be structural (full scan / huge lookup on table c) rather than \"estimates only\". Refreshing stats can change cardinality estimates but may not remove the need for a composite or prefix index on the hot column. Real incidents may mitigate with SQL binding that **forces an index** — that signals index/access-path was central to the fix, not merely ANALYZE order.",
      "correct_procedure": "On the bottleneck operator from decoded_plan: (1) Name the table and columns driving the scan. (2) Check whether an index exists that supports the join/filter (e.g. tag_id, message_id) with good selectivity. (3) If evidence points to missing or wrong index for that predicate, state **missing index (or inadequate access path)** as the primary root cause in the RCA headline. (4) List **CREATE INDEX** (or equivalent DDL) as P0 when that gap is established. (5) Treat ANALYZE / auto-analyze tuning as **supporting** or parallel, not the sole headline fix, unless you prove after (1)-(2) that only estimates are wrong and indexes are already optimal. (6) If oncall used index-hint SQL binding, treat that as evidence weighting index/access-path over stats-only narrative.",
      "conclusion_rule": "Do not write the root-cause headline as only \"stale statistics\" or make ANALYZE TABLE the exclusive P0 remediation when the bottleneck operator and column semantics indicate a **missing or wrong index** for the hot JOIN/filter. In that case, the headline must lead with **schema/access path** (missing index on the identified table/column); statistics may appear in the evidence chain as a contributing factor, not as the single primary cause. Recommendation priority in the report: **CREATE INDEX (or binding that mirrors the needed index path) before or alongside ANALYZE**, unless disambiguating evidence shows index coverage is already adequate and only histograms are wrong.",
      "corrective_checks": [
        "From decoded_plan: identify the operator with largest time/rows; map to table alias and columns (e.g. table c, tag_id)",
        "Ask: does an index exist whose prefix supports the WHERE/JOIN predicate? If no → missing index is a primary-class root cause candidate",
        "If est/act divergence is large on IndexFullScan/IndexLookUp, do not auto-rank ANALYZE above CREATE INDEX without answering the index-exists question",
        "If incident notes mention SQL binding with USE_INDEX / index name, weight index/access-path in the RCA over stats-only headline",
        "In the Recommendations table, avoid P0 = only ANALYZE when index gap is established; P0 should be index DDL or equivalent unless proven otherwise"
      ],
      "tags": [
        "analyze",
        "stale-statistics",
        "missing-index",
        "decoded-plan",
        "est-rows",
        "indexfullscan",
        "sql-binding",
        "o11y-2328599"
      ]
    },
    {
      "id": "ic-003",
      "title": "Verifying a user's claim in a proxy dimension instead of the claimed dimension",
      "example_cases": [
        "ws-eval-1783401504440-53741dd8"
      ],
      "wrong_pattern": "The user claims an entity is slow (a latency claim); the agent verifies the premise with CPU or another prominent metric and treats the premise as confirmed or refuted from that proxy.",
      "why_wrong": "Dimensions are not interchangeable. A TiKV store can be the slowest by gRPC duration with unremarkable CPU (compaction, IO stall, scheduling), and can be CPU-hot while serving reads at normal latency. Verifying a proxy dimension produces wrong confirmations and refutations and steers the whole investigation plan.",
      "correct_procedure": "Map the claim to its direct measure first: 'slower' on a TiKV store -> per-store gRPC message duration p99 / slow score / slow-query duration attributed to that store; 'slower' on TiDB -> handle_query duration p99 by instance; 'high CPU' -> process CPU vs quota; 'OOM' -> memory metrics and OOM logs; 'writes slow' -> apply/commit duration and raft propose wait. Compare the named entity against peer entities and a baseline window in that dimension. Use proxy dimensions only as candidate mechanisms after the premise is settled.",
      "conclusion_rule": "A premise is verified only by the direct measure of its own dimension; a proxy metric can corroborate a verified premise but can never confirm or refute it.",
      "corrective_checks": [
        "Does the verification metric measure the same quantity the user claimed?",
        "Was the named entity compared against peers and a baseline window in that metric?",
        "If the claimed dimension is absent from compacted summaries, was it fetched from raw matrices or a targeted collection before judging the premise?"
      ],
      "tags": [
        "premise-verification",
        "dimension-mismatch",
        "latency",
        "tikv"
      ]
    },
    {
      "id": "ic-004",
      "title": "Treating collector hints or top-N summaries as established anomalies",
      "example_cases": [
        "ws-eval-1783399913895-40a9231a"
      ],
      "wrong_pattern": "critical_evidence_hints flags or the compacted evidence summary's emphasis are treated as confirmed anomalies; entities absent from the top-N view are assumed healthy; the investigation follows the handoff's emphasis instead of the user's question.",
      "why_wrong": "Hints are keyword-triggered flags generated by text matching, and the summary keeps only top-N rows per metric group — false positives and false negatives are structural. Absence from a compacted view means the row was cut, not that the entity is healthy.",
      "correct_procedure": "Confirm each hint in the artifact it cites before planning around it. When the user names an entity that is missing from the summary, fetch its data explicitly from the full raw matrices or a targeted collection. Anchor the plan on the user's question; hints pointing elsewhere mean the question needs harder checking, not dropping.",
      "conclusion_rule": "A hint or summary row is a lead; it becomes evidence only after confirmation in the underlying artifact.",
      "corrective_checks": [
        "Was every hint used in the plan confirmed in the evidence file it cites?",
        "Was any user-named entity absent from the summary fetched explicitly before being judged normal?",
        "Does the plan address the user's question, not just the handoff's emphasis?"
      ],
      "tags": [
        "handoff",
        "hints",
        "top-n",
        "premise-verification"
      ]
    },
    {
      "id": "ic-005",
      "title": "Treating pending compaction as the final root cause instead of the flow-control path it triggers",
      "example_cases": [
        "tikv-node-slowdown-during-incident-window"
      ],
      "wrong_pattern": "Elevated pending compaction bytes or a compaction backlog on a slow store is presented as the final root cause of query slowdown.",
      "why_wrong": "Pending compaction is background debt; by itself it does not slow foreground queries. It hurts latency only through the throttle path it triggers on that store: RocksDB write stalls, TiKV flow control / admission control, scheduler backpressure ('server is busy'). Naming compaction as the final root cause skips the mechanism that actually produces the user-visible slowdown and misdirects remediation toward compaction tuning.",
      "correct_procedure": "When one store is slow and pending compaction is elevated, verify the throttle path on that store: write stall metrics (tikv_engine_write_stall, RocksDB stall reasons / level0 file counts), flow control and admission control activity (scheduler too-busy / throttle counters, flow control gauges), and 'server is busy' / write stall log lines in the window. If present, state the primary path as TiKV-side flow control / backpressure / write stall on the named store, with pending compaction as the surface signal or intermediate phenomenon. Optionally identify the upstream driver (write burst or ingest, IO throughput limits or cloud-volume throttling, stall configuration, region scheduling). Aim remediation at the flow-control path: admission control / throttling settings, stall thresholds, and TiKV flow-control follow-up - not at compaction alone.",
      "conclusion_rule": "Pending compaction may appear only as a surface signal or intermediate mechanism; when a single store's queries are slow with compaction backlog, the primary path to verify and report is TiKV flow control / backpressure / write stall on that store.",
      "corrective_checks": [
        "Were write stall / flow control / admission control signals checked on the slow store (stall metrics, scheduler too-busy or throttle counters, 'server is busy' or write-stall log lines)?",
        "Does the conclusion scope the problem to the named store and state the flow-control/backpressure path as primary, with compaction as a surface signal?",
        "Do the recommendations target admission control / throttling / stall follow-up rather than compaction tuning alone?"
      ],
      "tags": [
        "tikv",
        "compaction",
        "flow-control",
        "backpressure",
        "write-stall",
        "root-cause-depth"
      ]
    },
    {
      "id": "ic-006",
      "title": "Clearing a component via control-plane probes, an empty slice, or a failed-to-collect dimension",
      "example_cases": [
        "ws-eval-1783483163845-a46691df"
      ],
      "wrong_pattern": "A root-cause candidate is rejected because a control-plane or metadata operation looks healthy (fast describe/list/ping, e.g. Kafka describe-topics in microseconds), because its error signatures are absent from a filtered or time-sampled log slice fetched for a different hypothesis, or because the metric/panel dimension that would show it failed to collect at all (mass HTTP 422/5xx, tenant-token resolution failure, empty result) — and the investigation then commits high confidence to whichever class it could collect.",
      "why_wrong": "Control-plane calls exercise a different code path than the data path that actually fails: a sink can answer metadata queries instantly while every produce/write is dying with EOF, broken pipe, or TLS errors. Absence from a slice means the fetch did not cover it. And a dimension whose collection failed wholesale was never observed — 'metrics all returned 422' is not 'metrics are normal'. In every case the competing class was not measured, so it cannot be ruled out, and the class that happened to be collectable must not win by default.",
      "correct_procedure": "Before rejecting a candidate or promoting a primary, check the candidate's own data path in its own dimension across the full incident window, fetched without filters tuned to a different hypothesis. When that dimension is unobservable — control-plane-only, empty slice, or a failed collection — mark the class unverified, cap classification_confidence, record the unobserved dimension in missing_evidence, and prefer one more targeted collection (retry the failed queries, widen the window) over concluding around the hole.",
      "conclusion_rule": "Rejection and high-confidence promotion both need direct data-path evidence in the candidate's own dimension over the full window; control-plane health, empty slices, and failed collections observe nothing.",
      "corrective_checks": [
        "Fetch the candidate's data-path error logs/latency (write/produce/apply, client-library errors, the metric that would show the competing class) over the full incident window without hypothesis-specific filters before writing evidence_against or setting a high classification_confidence.",
        "If the only exoneration is a metadata/describe/ping probe, an empty filtered slice, or a dimension that failed to collect, mark the candidate unverified and cap confidence instead of rejecting it or concluding around it."
      ],
      "tags": [
        "rejection",
        "control-plane-proxy",
        "filtered-slice",
        "failed-collection",
        "confidence-cap",
        "verification-dimension"
      ]
    }
  ],
  "maintenance": "Append new entries with id ic-NNN, tags, and corrective_checks when postmortems or reviews find repeated bad automation outcomes."
}