# Workload Hotspot ## Question Is the incident explained by a concentrated CPU, RU, QPS, scan, or execution workload, and which digest or plan contributes materially to that hotspot? ## Enter When Use for an explicit CPU/RU/QPS/scan hotspot. Also run it as an independent secondary scenario for an overall latency increase. A single-query diagnosis does not automatically select this branch: it needs an independent resource signal or explicit workload-attribution question. ## First Batch 1. Align an instance-labelled CPU, RU, QPS, duration, or scan signal to the incident window and select the affected instance at the event sample. 2. Use TopSQL on that exact instance and a bounded hotspot window as the primary workload surface. Compare digest/plan contribution, execution rate, CPU, duration per execution, and scan rates that the provider actually returns. 3. Use Slow Query only as supporting evidence for extreme executions or a candidate digest; it does not replace the TopSQL concentration check. Broad lists omit SQL text and plans. Retain complete Digests, component, instance, time windows, contribution and supported execution-rate metrics. The supplied target remains comparison context; do not filter out other major contributors. Retrieve one candidate's SQL or execution detail by Digest only when a load-bearing mechanism question needs it. A directly supported CPU attribution does not require a plan audit or SQL text. ## Decision TopSQL rank alone is only a workload candidate. Confirm a hotspot only when its contribution aligns with the affected entity, event window, and resource or latency result. An empty, retained-out, top-N-limited, or failed TopSQL response is a bounded evidence limitation, not proof that no hotspot exists. For a CPU hotspot question, TopSQL is direct workload-to-CPU attribution, not merely correlation: when its same-component, same-instance, incident-aligned CPU contribution explains the hotspot and an aligned process-CPU change or recovery corroborates it, return `confirmed` and `causal_confirmed` for that bounded workload cause. Lower-level operator and storage evidence is auxiliary; do not require it, transaction shape, or a request-to-latency chain to confirm CPU attribution. Preserve unknown internal mechanisms separately and do not extend that conclusion to latency causality. Refine a backend alternative only when a material residual or conflicting evidence could change the diagnosis or action, not simply because an operator detail is missing. Partial contribution remains partial, and dominant resident load alone does not explain an increase. For a node-specific question, use exact `component` and `instance` scope before attributing CPU to a SQL. Omit instance only when the question is explicitly which workload is largest across the cluster. A global candidate does not prove node attribution; if it remains load-bearing, localize that Digest to the affected instance. With a Digest but no instance, compare its distribution across instances. With instance plus Digest, compare its plans on that node. Treat TopSQL CPU as accumulated CPU time, not wall time: `cpu_ms` already integrates parallel execution across cores. Never multiply it by core count. For a bounded window, `cpu_ms / (window_seconds * 1000)` is the SQL's average CPU cores over that window. Compare it only with process CPU normalized to average cores on the same instance and interval. When the deterministic handoff establishes that a supplied SQL is absent from the complete SQL-level Top20 for the selected scope, use the returned cutoff share as a conservative upper bound. That is enough to call the SQL non-dominant in the captured node or global window; do not request rank 21 or deeper tail rows merely to locate it. This aggregate bound does not reject a rare wait-heavy Slow Query. Full-window share answers whether a SQL is a dominant resident workload; it does not answer how much that SQL explains a reported CPU increase. When the ticket asks what caused CPU to rise on one selected instance, phase-delta attribution remains load-bearing even if the full-window share already proves that the supplied SQL is only a partial contributor. Do not preemptively split the initial batch into opening, middle, or closing TopSQL phases. After the same-instance process series establishes two non-overlapping comparable phases inside the canonical diagnostic range, use the next adaptive refinement for equal-duration or otherwise normalized baseline and hot phases. Align them to the reported onset and observed rise or plateau. The baseline must precede the reported increase; never relabel an after-onset low point as the baseline merely because it is lower than a later point, and do not call arbitrary first and second halves comparable. If the canonical range begins at or after the reported onset, preserve the exact missing-pre-increase-baseline limitation instead of calculating CPU-increase attribution. For both valid phases acquire the same-instance process CPU and node-scoped TopSQL distribution, retaining the supplied SQL, the leading node-local alternative, and the selected-scope CPU total. Normalize every quantity to average cores, then compare `delta_target_sql_cores`, `delta_leading_sql_cores`, and `delta_scope_topsql_cores` with `delta_process_cores`. Treat each SQL delta divided by the process delta as an attribution estimate, not exact accounting: TopSQL sampling can omit tail work and process CPU includes non-query work. A positive process delta with a non-positive target delta means the target is background load but does not explain the increase. A positive target delta smaller than the process delta is a partial contribution. A ratio above one is not capped at 100%; it indicates sampling, phase, or offsetting-work effects and cannot be presented as exact share. If the process delta is not distinguishable from within-phase variation, preserve an unstable-denominator limitation. Stop after the bounded comparison or the exact Provider limitation. For an explicit SQL-to-node-CPU-increase attribution question, perform this phase comparison before spending a later refinement on leader, scheduler, Raft, or storage alternatives. Test those backend alternatives afterward only when the phase comparison leaves a material node-CPU residual or cannot be completed. An operation name is not a mechanism. When an initial batch finds Commit, Prewrite, ResolveLock, or another TiKV request latency large enough to rival the reported tail, do not convert that request-type aggregate into lock contention or a workload root cause. If the result has aggregated away the TiKV store or instance and the backend alternative was not tested, the newly observed alternative is load-bearing: use the single adaptive refinement to localize the same request result to the affected store and compare it with one direct apply/snapshot, RocksDB/storage, scheduler, or leader-distribution boundary on that same entity and interval. Accept either an aligned mechanism or an exact provider limitation as the stop condition. The same backend alternative is load-bearing for a cluster-wide tail-latency incident when the candidate SQL mechanism and actual capacity exhaustion both remain unproven. Aggregate request-type latency that is lower than the reported tail can bound that one request path, but it cannot by itself reject a store-local apply/snapshot, RocksDB/storage, scheduler, or leader-distribution boundary. If no direct per-store backend discriminator has been tested, use the single adaptive refinement for one such discriminator instead of stopping on a correlated SQL type or aggregate CPU utilization. Accept either a bounded backend result or the exact Provider limitation as the stop condition. A long Slow Query that is absent from TopSQL may be wait-heavy or outside the returned top-N. Do not use that absence to reject the separate `cluster-query-latency` identity or mechanism evidence. ## Impact And Mitigation When direct per-store evidence establishes a TiKV hotspot or backend boundary and impact is active, preserve a conditional, human-owned mitigation without claiming that it was executed. Support may evaluate a temporary leader evacuation from the currently observed hot store or a bounded reduction in scheduling pressure only after reading back and recording the current leader distribution, scheduler configuration, and store limits. Change one boundary at a time, observe the same latency and distribution signals, and restore the recorded values when the stop condition is met or impact does not improve. Never copy a historical store id or scheduling value into a new recommendation, and never infer a hot store from request type, TopSQL rank, or a window-wide maximum. Production mitigation remains privileged and requires a human owner. For a recovered historical incident, state this only as a recurrence playbook; do not claim continuing impact or that the mitigation is currently required. ## Strongest Alternative And Stop Compare the leading workload with the next candidate and the strongest non-workload resource or wait explanation. Stop when one digest/plan materially explains the aligned hotspot, the resource signal is not concentrated, or the exact TopSQL limitation is recorded. Do not stop on a material aggregate TiKV request-latency alternative while its store identity and one direct backend discriminator remain untested and the one permitted refinement is still available. For an otherwise unexplained cluster-wide tail, do not treat a lower aggregate request-type latency as proof that every store-local backend boundary is absent.