---
name: tiflash-health-inspection
description: |
  Inspect TiFlash health for TiDB Cloud clusters from Clinic/O11Y metrics and generate a Markdown report.
  Use for TiFlash health checks, bottleneck reports, or issues in query/MPP/resource/IO/Raft signals.
---

# TiFlash Health Inspection

## Overview

Run a Cloud-oriented TiFlash health inspection from monitoring metrics and
produce a Markdown report. Prefer evidence from Clinic Data Proxy PromQL and
classify findings as `Healthy`, `Warning`, `Critical`, or `Unknown`.

This skill is metrics-first. It can identify likely TiFlash service risks and
likely bottleneck direction, but single-SQL attribution still needs TiDB slow
logs, execution summary, or the query plan.

## Load References First

Read both reference files before evaluating:

- `references/promql-reference.md`
- `references/inspection-rules.md`

Use `promql-reference.md` to select PromQL and labels. Use
`inspection-rules.md` for thresholds, persistence windows, and report judgment.
Do not fetch TiFlash Grafana JSON files during automatic inspection; the PromQL
reference already captures the required metric semantics.

## Preferred Automation

When running a TiFlash health inspection through Clinic Data Proxy, prefer using
`tiflash_health_inspection.py` from this skill directory to produce the Markdown
report. This helper implements the default Clinic Data Proxy path for cluster
discovery, TiFlash metric selector discovery, rule evaluation, and compact or
full report rendering.

Use the manual workflow in this document instead when:

- Clinic Data Proxy is unavailable.
- O11Y metrics or direct Prometheus is explicitly required.
- The user requests checks not implemented by the script.
- The script marks important domains `Unknown` and additional label discovery or
  fallback data sources are needed.

Default single-cluster execution:

```bash
python3 tiflash_health_inspection.py \
  --cluster-id "<cluster_id>" \
  --window-minutes 30 \
  --output "tiflash-health-inspection-<cluster_id>.md"
```

Use `--format full` for expanded per-cluster sections and `--include-queries`
when concrete PromQL evidence is needed for handoff or reproduction.

Repeat `--cluster-id` to inspect a selected set of clusters and render one
aggregate report:

```bash
python3 tiflash_health_inspection.py \
  --cluster-id "<cluster_id_1>" \
  --cluster-id "<cluster_id_2>" \
  --cluster-id "<cluster_id_3>" \
  --window-minutes 30 \
  --jobs 4 \
  --output "tiflash-health-inspection-selected.md"
```

Default batch execution uses concurrent per-cluster inspection. Start with a
small concurrency value to avoid overloading Clinic Data Proxy:

```bash
python3 tiflash_health_inspection.py \
  --window-minutes 30 \
  --jobs 4 \
  --output "tiflash-health-inspection-batch.md"
```

The helper pre-skips clusters whose metadata or topology clearly reports zero
TiFlash nodes before running PromQL. Within each cluster, metric series
discovery is cached so coverage checks, selector resolution, and Unknown
classification do not repeat the same `query_series` calls.

## Inputs

Collect:

- Cluster identifier or Clinic cluster link, if the user wants one specific cluster.
- Optional TiFlash instance filter.
- Inspection time range. Default to the most recent `30m`.
- Optional long trend window. Default to `24h` for capacity and recurring issue checks.
- Query step. Default to `60`; use `30` for incident windows when data volume is acceptable.
- Data source used: Clinic Data Proxy or O11Y metrics.

Normalize runtime values before running PromQL:

- `START` and `END` are absolute Unix timestamps.
- `step` is query resolution in seconds, such as `60`.
- Use the same bare-second step format for Clinic Data Proxy,
  O11Y-compatible Prometheus APIs, and direct Prometheus.
- Rate windows are explicit durations. Use `1m` for TiFlash service counters
  unless a rule states otherwise.
- Do not pass Grafana variables such as `$__rate_interval`, `$namespace`,
  `$instance`, `$k8s_cluster`, or `$tidb_cluster` to Clinic Data Proxy. Resolve
  them first from metadata or metric series.

Default batch scope:

- Use `1372813089209061633` as the fixed default organization search id only
  when the user explicitly asks for batch inspection or does not provide a
  cluster id and accepts the default batch scope.
- When batch inspection is confirmed, use `clinic-api/cluster-metadata` to search
  clusters under this id, then inspect every eligible cluster.
- Treat `1372813089209061633` as the user-facing search id for cluster discovery.
  Do not blindly reuse it as the Clinic REST internal `orgID` for
  `/clinic/api/v1/orgs/{orgID}/clusters/{clusterID}` paths. For each discovered
  cluster, read the returned `orgID` and use that internal value for REST calls.

If the user provides a cluster id, inspect only that cluster. Resolve Cloud
metadata first through `clinic-api` where available. For Premium/resource-pool
environments, preserve the Cloud label set discovered from metric series.

## Cluster Discovery

Use `clinic-api/cluster-metadata` for discovery.

Locate the installed `clinic-api` skill directory before sourcing helpers. Try
these paths in order and use the first one that exists:

- `skills/platform/clinic-api` when running from this repository root.
- `${CODEX_HOME:-$HOME/.codex}/skills/clinic-api` for Codex.
- `$HOME/.claude/skills/clinic-api` for Claude.
- `$HOME/.config/opencode/skills/clinic-api` for OpenCode.

Default batch discovery:

```bash
for candidate in \
  "skills/platform/clinic-api" \
  "${CODEX_HOME:-$HOME/.codex}/skills/clinic-api" \
  "$HOME/.claude/skills/clinic-api" \
  "$HOME/.config/opencode/skills/clinic-api"; do
  if [ -f "${candidate}/scripts/clinic-shell.sh" ]; then
    CLINIC_API_SKILL_DIR="${candidate}"
    break
  fi
done
[ -n "${CLINIC_API_SKILL_DIR:-}" ] || {
  echo "clinic-api skill not found" >&2
  exit 1
}
source "${CLINIC_API_SKILL_DIR}/scripts/clinic-shell.sh"
get_clinic_auth
clinic_search_clusters "1372813089209061633" "true"
```

From the returned `items`, keep clusters that are active and have TiFlash, or
whose topology/metadata indicates TiFlash is configured. If topology is
available, use `clinic_get_topology "<clusterID>"` and check for
`component_type="tiflash"` with a positive node count.

Pre-detect clusters without TiFlash before running PromQL:

- If search metadata contains `topology.tiflash` and its value is `0`, classify
  the cluster as `Skipped: no TiFlash`.
- If topology API returns a TiFlash component only as a configuration object but
  its `node_count`, `nodeCount`, `replicas`, or `count` is `0`, classify the
  cluster as `Skipped: no TiFlash`.
- Do not treat a TiFlash storage configuration or component row with zero nodes
  as an active TiFlash deployment.
- Only run TiFlash PromQL when metadata/topology shows active TiFlash nodes, or
  when shared/Starter/Essential topology is unavailable and TiFlash metric
  series prove TiFlash presence.

Exact single-cluster discovery:

```bash
clinic_search_cluster_by_id "<clusterID>" "true"
```

Always filter search results by exact `clusterID` before reading fields. Branch
clusters can appear in fuzzy search results and must not be mistaken for the
target cluster.

For every discovered cluster, collect:

- `clusterID`
- cluster name
- `orgID` from the search result, for Clinic REST paths
- `tenantID`
- provider, region, deploy type, project id, parent id if present
- TiFlash topology when available

Classify clusters without TiFlash evidence consistently:

- `Skipped: no TiFlash`: metadata or topology clearly shows the cluster has no
  TiFlash component, or TiFlash node count is `0`.
- `Unknown`: TiFlash is expected or cannot be ruled out, but topology is
  unavailable, core TiFlash metric series are missing, or labels cannot be
  resolved.

Shared, Starter, and Essential tiers may not expose topology. For those tiers,
check TiFlash metric series before deciding whether to skip the cluster.

## Data Sources

Use data sources in this order:

1. `clinic-api/metrics`: query Clinic Data Proxy Prometheus APIs.
2. `o11y-metrics-api`: query TiDB Cloud O11Y Prometheus-compatible APIs.
3. Direct Prometheus/Grafana only when the user provides an explicit endpoint or link.

For Clinic Data Proxy, prefer `ClinicAPI.query_range()` / `query_instant()` and
`query_series()` for missing metrics. If a query returns no data, first discover
the available labels for that metric before changing the rule.

Automatic inspection must not depend on direct TiFlash Grafana file access. At
runtime, execute the PromQL from `references/promql-reference.md` through
the installed `clinic-api/metrics/SUBSKILL.md`. Resolve `CLINIC_API_SKILL_DIR`
with the same path search used for shell discovery:

```python
import os
import sys
from pathlib import Path

candidate_roots = [
    Path("skills/platform/clinic-api"),
    Path(os.environ.get("CODEX_HOME", str(Path.home() / ".codex"))) / "skills/clinic-api",
    Path.home() / ".claude/skills/clinic-api",
    Path.home() / ".config/opencode/skills/clinic-api",
]
clinic_api_skill_dir = next(
    root for root in candidate_roots if (root / "scripts/clinic_api.py").exists()
)
sys.path.insert(0, str(clinic_api_skill_dir / "scripts"))
from clinic_api import ClinicAPI

api = ClinicAPI()
result = api.query_range(
    "CLUSTER_ID",
    'sum(rate(tiflash_coprocessor_request_error{tidb_cluster="CLUSTER_ID"}[1m])) by (reason)',
    START,
    END,
    step="60",
)
```

For missing TiFlash data, use `api.query_series("CLUSTER_ID",
"tiflash_coprocessor_request_error", START, END)` or the specific metric name to
discover labels before marking the check `Unknown`.

When a reference query uses `<selector>`, build it from series discovered for
that same metric family. Start with a bare metric `query_series()` over the
inspected window, choose labels from returned TiFlash series, and then construct
the narrowest selector that still returns data.

For rules that combine multiple metric families, discover and build one selector
per metric family, then join only on stable labels such as `instance`. Do not
copy selector labels from the numerator metric to the denominator metric. For
example, CPU ratio should use a selector discovered from
`tiflash_proxy_process_cpu_seconds_total` for CPU usage and a selector
discovered from `tiflash_system_current_metric_LogicalCPUCores` for CPU quota.

Common label patterns include:

```promql
k8s_cluster="<k8s_cluster>", tidb_cluster="<tidb_cluster>", instance=~"<instance_regex>"
tidb_cluster_id="<cluster_id>", instance=~"<instance_regex>"
cluster_id="tidb<cluster_id>", instance=~"<instance_regex>"
```

Do not blindly reuse every cluster label discovered from one metric family on
another family. For example, if `tiflash_proxy_process_cpu_seconds_total` has no
`job` label, do not add a `job` selector to CPU queries. TiFlash-prefixed metric
families such as `tiflash_*` and `tiflash_proxy_*` are already TiFlash-scoped;
rewrite only selector labels that are needed to isolate the target cluster or
instances. Keep the metric name, aggregation, and rule semantics unchanged.

Apply the same rule to TiFlash Proxy checks: do not fetch Grafana files during
automatic inspection. Use the proxy-related PromQL already recorded in
`references/promql-reference.md`, especially:

- `tiflash_proxy_tikv_server_report_failure_msg_total` for server report failures.
- `tiflash_proxy_thread_cpu_seconds_total{name=~"GRPCComp.*"}` for gRPC async
  client CPU and thread saturation checks.
- `tiflash_proxy_threads_io_bytes_total` for TiFlash proxy IO throughput.

When these proxy queries return empty through Clinic Data Proxy, run
`query_series()` on the metric name to discover the actual labels, then rewrite
only selector labels such as `tidb_cluster`, `tidb_cluster_id`, `cluster_id`,
`k8s_cluster`, and `instance`. Do not change the metric family, aggregation, or
rule threshold unless the PromQL reference is explicitly updated.

### O11Y Metrics Fallback

Use O11Y only after Clinic Data Proxy is unavailable, incomplete, or explicitly
requested by the user. Follow `o11y-metrics-api/SKILL.md`; do not invent an O11Y
endpoint or skip target resolution.

Locate the installed `o11y-metrics-api` skill before sourcing its token helper.
Try these paths in order:

- `skills/platform/o11y-metrics-api` when running from this repository root.
- `${CODEX_HOME:-$HOME/.codex}/skills/o11y-metrics-api` for Codex.
- `$HOME/.claude/skills/o11y-metrics-api` for Claude.
- `$HOME/.config/opencode/skills/o11y-metrics-api` for OpenCode.

Required metadata:

- `CLUSTER_ID`
- `VENDOR`, from Clinic `clusterProviderName`
- `REGION`, from Clinic `clusterRegionName`
- `DEPLOY_TYPE`, from Clinic `deployType`, `clusterDeployType`, or
  `clusterDeployTypeV2`
- `PARENT_ID`, for Premium/resource-pool clusters when present
- `TENANT_ID`, for K8s infrastructure target discovery
- `START`, `END`, and `step` as absolute Unix timestamps and a bare-second step
  such as `60`

Resolve the O11Y application id:

- Dedicated/BYOC: use `CLUSTER_ID`.
- Resource pool: use `CLUSTER_ID`.
- Premium: use `PARENT_ID` when present; otherwise use `CLUSTER_ID` and verify
  the returned O11Y application record before querying.
- Starter/essential/shared clusters may not have usable O11Y TiFlash metrics;
  keep the status `Unknown` or `Skipped: no TiFlash` when resolution fails.

Minimal O11Y setup:

```bash
for candidate in \
  "skills/platform/o11y-metrics-api" \
  "${CODEX_HOME:-$HOME/.codex}/skills/o11y-metrics-api" \
  "$HOME/.claude/skills/o11y-metrics-api" \
  "$HOME/.config/opencode/skills/o11y-metrics-api"; do
  if [ -f "${candidate}/o11y-token.sh" ]; then
    O11Y_METRICS_SKILL_DIR="${candidate}"
    break
  fi
done
[ -n "${O11Y_METRICS_SKILL_DIR:-}" ] || {
  echo "o11y-metrics-api skill not found" >&2
  exit 1
}
source "${O11Y_METRICS_SKILL_DIR}/o11y-token.sh"
AUTH0_TOKEN=$(get_o11y_token)

APP_ID="${PARENT_ID:-$CLUSTER_ID}"
if [ "${DEPLOY_TYPE}" = "dedicated" ] || [ "${DEPLOY_TYPE}" = "byoc" ]; then
  APP_ID="${CLUSTER_ID}"
fi

GS_ENDPOINT="http://www.gs.us-west-2.aws.observability.tidbcloud.com/api/v1/clusters/"
APP_INFO=$(curl -s "${GS_ENDPOINT}" \
  -H "Authorization: Bearer ${AUTH0_TOKEN}" \
  -G \
  --data-urlencode "vendor=${VENDOR}" \
  --data-urlencode "region=${REGION}" \
  --data-urlencode "applicationID=${APP_ID}")

APP_UUID=$(echo "${APP_INFO}" | jq -r '.data.clusters[0].id // empty')
TENANT_ID=$(echo "${APP_INFO}" | jq -r ".data.clusters[0].tenantID // \"${TENANT_ID:-}\"")
DS_ENDPOINT="https://www.ds.${REGION}.${VENDOR}.observability.tidbcloud.com"
[ -n "${APP_UUID}" ] || {
  echo "failed to resolve O11Y application UUID" >&2
  exit 1
}
```

Resolve the K8s infrastructure UUID before running node, kube, or Cloud disk
metrics:

```bash
NAMESPACE="${NAMESPACE:-tidb${APP_ID}}"
DP_ENTRIES=$(curl -s "${GS_ENDPOINT}" \
  -H "Authorization: Bearer ${AUTH0_TOKEN}" \
  -G \
  --data-urlencode "vendor=${VENDOR}" \
  --data-urlencode "region=${REGION}" \
  --data-urlencode "tenantID=${TENANT_ID}" \
  | jq '[.data.clusters[]? | select((.applicationID // "") | startswith("dp-"))]')

K8S_UUID=""
DP_COUNT=$(echo "${DP_ENTRIES}" | jq 'length')
if [ "${DP_COUNT}" -eq 1 ]; then
  K8S_UUID=$(echo "${DP_ENTRIES}" | jq -r '.[0].id')
elif [ "${DP_COUNT}" -gt 1 ]; then
  for i in $(seq 0 $((DP_COUNT - 1))); do
    UUID=$(echo "${DP_ENTRIES}" | jq -r ".[$i].id")
    COUNT=$(curl -s "${DS_ENDPOINT}/internal/v2/metrics/${UUID}/api/v1/query" \
      -H "Authorization: Bearer ${AUTH0_TOKEN}" \
      -G \
      --data-urlencode "query=count(kube_pod_info{namespace=\"${NAMESPACE}\"})" \
      | jq -r 'try (.data.result[0].value[1] | tonumber | floor) catch 0')
    if [ "${COUNT}" -gt 0 ]; then
      K8S_UUID="${UUID}"
      break
    fi
  done
fi
```

Choose the O11Y metrics target by metric family:

| Metric family | O11Y target |
|---|---|
| `tiflash_*` | `APP_UUID` |
| `node_*` | `K8S_UUID` |
| `kube_*` | `K8S_UUID` |
| `aws_ebs_*`, `aws_ebs_csi_*` | `K8S_UUID` |

O11Y range query pattern:

```bash
TARGET_UUID="${APP_UUID}"
QUERY='sum(rate(tiflash_coprocessor_request_error{tidb_cluster="CLUSTER_ID"}[1m])) by (reason)'
curl -s "${DS_ENDPOINT}/internal/v2/metrics/${TARGET_UUID}/api/v1/query_range" \
  -H "Authorization: Bearer ${AUTH0_TOKEN}" \
  -G \
  --data-urlencode "query=${QUERY}" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode "step=${step:-60}"
```

For O11Y label discovery, first try the same selector used for Clinic Data
Proxy. If the result is empty, run a short range query for the bare metric name,
for example `tiflash_coprocessor_request_error`, and inspect returned metric
labels. If the O11Y endpoint supports `/api/v1/series`, use it with the same
metric name, `START`, and `END`. Rewrite only selector labels and keep the
metric family, aggregation, and rule semantics unchanged.

Do not query infrastructure metrics through `APP_UUID`. If a `node_*`, `kube_*`,
`aws_ebs_*`, or `aws_ebs_csi_*` query is empty and `K8S_UUID` is unresolved, mark
the infrastructure domain `Unknown` instead of concluding that the metric is
healthy or absent.

### Direct Prometheus Fallback

Use direct Prometheus only when the user provides a reachable Prometheus base URL
or an explicit Grafana datasource/proxy URL plus any required token. Direct
Prometheus access is a runtime data source; it does not require downloading
Grafana JSON.

Required inputs:

- `PROMETHEUS_BASE_URL`, for example `https://prom.example.com`.
- Optional auth header or bearer token.
- Absolute `START` and `END` Unix timestamps.
- `step`, default `60`.
- Cluster selector labels discovered from series metadata.

Prometheus API calls:

```bash
QUERY='sum(rate(tiflash_proxy_tikv_server_report_failure_msg_total{tidb_cluster="CLUSTER_ID"}[1m])) by (type, instance, store_id)'
curl -sG "${PROMETHEUS_BASE_URL}/api/v1/query_range" \
  -H "Authorization: Bearer ${PROMETHEUS_TOKEN}" \
  --data-urlencode "query=${QUERY}" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode "step=60"
```

Discover labels before changing selectors:

```bash
curl -sG "${PROMETHEUS_BASE_URL}/api/v1/series" \
  -H "Authorization: Bearer ${PROMETHEUS_TOKEN}" \
  --data-urlencode 'match[]=tiflash_proxy_tikv_server_report_failure_msg_total' \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}"
```

Minimal Python extraction pattern:

```python
import json
import urllib.parse
import urllib.request

def prom_query_range(base_url, query, start, end, step="60", token=None):
    params = urllib.parse.urlencode({
        "query": query,
        "start": start,
        "end": end,
        "step": step,
    })
    req = urllib.request.Request(f"{base_url}/api/v1/query_range?{params}")
    if token:
        req.add_header("Authorization", f"Bearer {token}")
    with urllib.request.urlopen(req, timeout=120) as resp:
        return json.loads(resp.read())

def series_max(result):
    rows = []
    for item in result.get("data", {}).get("result", []):
        values = [float(v[1]) for v in item.get("values", []) if v[1] not in ("NaN", "+Inf", "-Inf")]
        if values:
            rows.append({"metric": item.get("metric", {}), "max": max(values), "avg": sum(values) / len(values)})
    return rows
```

For every direct Prometheus result, keep the same judgment contract as Clinic
Data Proxy: summarize max, average, duration above threshold, affected labels,
and query expression. Do not report raw time series in the final Markdown.

## Cloud Label Handling

TiFlash metric series commonly use:

```promql
{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance"}
```

Some Cloud data sources or generated metric catalogs use `tidb_cluster_id` or
`cluster_id` instead of `tidb_cluster`. When a query is empty:

1. Run `query_series` for the metric over the inspected window.
2. Check whether the labels are `tidb_cluster`, `tidb_cluster_id`, `cluster_id`,
   or a parent resource-pool id.
3. Rewrite only the selector labels; keep the metric name, aggregation, and rule
   semantics unchanged.

For infrastructure metrics that join node data to TiFlash pods, use the concrete
cluster namespace, usually `tidb<cluster_id>`, and preserve `label_replace(...,
"instance", "$1", "node", "(.*)")`. The node selector must be derived from the
TiFlash pod-to-node mapping. If that mapping cannot be resolved, keep the
infrastructure domain `Unknown` instead of widening the query to every node in
the cluster.

For PromQL copied from Grafana dashboards, replace variables before execution:

| Grafana variable | Runtime replacement |
|---|---|
| `$__rate_interval` | `1m` unless a rule states otherwise |
| `$namespace` | concrete namespace, usually `tidb<cluster_id>` |
| `$instance` | discovered node or TiFlash instance regex |
| `$k8s_cluster` | discovered Kubernetes cluster label |
| `$tidb_cluster` | discovered TiDB cluster label, usually the cluster id |

## Minimal Inspection Set

Run this set first for every eligible cluster. It is the default inspection
scope when the user asks for a general TiFlash health check and does not request
deep bottleneck analysis.

| Tier | Domain | Required checks | Purpose |
|---|---|---|---|
| 1 | Coverage | D1 minimal core metrics, TiFlash topology or TiFlash metric presence | Decide whether the cluster can be evaluated |
| 1 | Query | Request QPS, request duration p99, error QPS, handling requests | Detect user-facing query risk |
| 1 | Resource | CPU ratio, memory ratio, store capacity | Detect common resource saturation |
| 1 | MPP | Active MPP query count, longest live MPP task, exchange queueing data | Detect blocked or backed-up MPP execution |
| 2 | Storage IO | fsync duration, write stall, IO limiter pending | Attribute latency to storage pressure |
| 2 | Network IO | node drop/error counters, server report failures, gRPC async client saturation | Attribute latency to transport or proxy pressure |
| 2 | Raft | wait index p99, wait index timeout | Detect replica wait and transport/storage side effects |
| 2 | Pipeline | task duration by type, task status change OPS, pool saturation | Attribute execution wait to CPU, IO, await, or stalled progress |

Default execution policy:

1. Always run Tier 1 after metadata and label discovery.
2. Run Tier 2 when any Tier 1 check is `Warning`, `Critical`, or `Unknown`, or
   when the user asks for bottleneck attribution.
3. Run the `24h` trend window only for baseline-dependent duration checks,
   capacity growth, recurring issue checks, and active-count baseline checks.
4. In batch mode, stop after Tier 1 for clusters that are clearly `Healthy`
   unless the user requests full evidence for every cluster.
5. If a Tier 1 required metric is missing after label discovery, mark the
   cluster `Unknown` and do not infer health from Tier 2 signals alone.

Minimal output contract:

- Health Summary includes total checked and skipped counts.
- One Actionable Findings row for every `Critical` or actionable `Warning`
  finding, capped to the most important findings in compact reports.
  Weak Q1 retry/data-movement error signals should be listed as supporting
  evidence unless they coincide with request latency, MPP backlog, transport,
  Raft, or another user-facing abnormal signal.
- One Domain Summary row per evaluated TiFlash cluster. In compact mode, omit
  clusters classified as `Skipped: no TiFlash`; keep only the skipped count in
  Health Summary. Include skipped cluster details only in full mode.
- One compact Unknowns section for evaluated TiFlash clusters only. If there are
  no unknowns, write `None`.
- One Skipped Clusters section for clusters classified as `Skipped: no TiFlash`.
  Do not put skipped clusters in the Unknowns section.
- Supporting signals for weak Q1 findings and no-activity histogram rules.
- Evidence Queries are omitted by default. Include concrete PromQL selectors
  only in full mode or explicit query appendix mode.

## Inspection Workflow

For batch mode, run this workflow per cluster and then produce one aggregate
Markdown report with per-cluster sections.

Execution checklist for each cluster:

1. Resolve cluster metadata and TiFlash topology.
2. Discover metric selector labels from one minimal core TiFlash metric, usually
   `tiflash_proxy_process_cpu_seconds_total` or
   `tiflash_coprocessor_request_duration_seconds_bucket`.
3. Build `<selector>`, `<namespace>`, and `<node_instance_regex>`.
4. Run minimal core coverage checks. Stop at cluster-level `Unknown` if minimal
   core coverage is missing after label discovery.
5. Run query, MPP, resource, storage, network, Raft, and Pipeline rules.
6. For empty domain-specific metrics, run `query_series()` before marking that
   domain `Unknown`.
7. Extract the rule result fields defined in `inspection-rules.md`.
8. Correlate abnormal rule windows before assigning likely bottleneck.
9. Produce compact Markdown summary and actions. Produce expanded sections and
   evidence queries only when full evidence is requested.

Run checks in this order for each eligible cluster:

1. Data coverage and TiFlash liveness.
2. Query health: request QPS, request duration, error QPS, handling requests.
3. MPP task health: active query count, longest live MPP task, exchange queue,
   MPP establish connection, run MPP task duration.
4. Resource health: CPU, memory, store capacity and short-term capacity risk.
5. Storage IO: TiFlash IO throughput, fsync duration, write stall, IO limiter
   pending.
6. Network IO: node network drops/errors/traffic, MPP exchange throughput and
   queueing, TiFlash Proxy server report failures, gRPC thread saturation.
7. Raft and replica health: wait index duration, wait index timeout, raft
   throughput.
8. Pipeline Model: CPU queue, IO queue, await, task status changes, task pool
   saturation.
9. Bottleneck attribution: classify as `Compute`, `Storage`, `Network/Exchange`,
   `Mixed`, or `Unknown` with confidence.

Do not mark the report `Healthy` when minimal core TiFlash metrics are missing.
Use `Unknown` for missing data or unresolved architecture/labels. If minimal
core metrics are present but a workload-specific or domain-specific metric family
is missing, keep the overall status driven by covered checks and mark that
domain `Unknown`.

## Judgment Rules

Use four states:

- `Critical`: user-facing impact or high-risk internal failure is likely.
- `Warning`: abnormal signal exists, but direct impact is not proven.
- `Healthy`: all required signals are present and no warning/critical rules fire.
- `Unknown`: required metrics are missing, labels are unresolved, or the inspected
  window has insufficient data.

Overall status:

- Any `Critical` finding -> `Critical`.
- Otherwise any actionable `Warning` finding -> `Warning`.
- Weak Q1 retry/data-movement warnings alone do not raise the overall status
  above `Healthy`; report them as supporting evidence.
- Otherwise minimal core checks are covered and no domain has a high-risk
  unknown -> `Healthy`.
- Missing minimal core coverage -> `Unknown`.

Coverage status:

- `Covered`: required metric families for that section returned usable samples.
- `Partial`: at least one optional or domain-specific metric is missing.
- `Missing`: required metric families for that section are absent after label
  discovery.

For batch mode, keep the aggregate report compact. Always include a Domain
Summary table for evaluated TiFlash clusters. Omit clusters without TiFlash from
the compact body and represent them only through the skipped count in Health
Summary. Do not expand per-cluster rule tables by default. Expand full
per-cluster sections, including skipped clusters, only when the user asks for
full evidence or query reproduction. For `Healthy` clusters, list the cluster in
the summary and omit detailed sections unless the user asks for full evidence.

For long-running MPP tasks, do not classify solely from lifetime. Treat
`tiflash_mpp_task_monitor` as suspicious only when combined with lack of
progress, queue growth, request duration growth, or resource/transport errors.
Ignore `tiflash_mpp_task_monitor` samples whose value is older than two days
(`> 172800s`) so stale task artifacts do not trigger M1 in recent-window
inspections.

## Markdown Report Format

Produce Markdown only. Do not generate HTML for this skill.

Default to a compact human-readable report. Keep the body focused on actionable
findings, domain status, supporting signals, and next actions. Do not print full
PromQL in the default report; put PromQL in an appendix only when the user asks
for full evidence.

Use this compact structure by default:

````markdown
# TiFlash Health Inspection Report

## Health Summary
Status: <Healthy|Warning|Critical|Unknown>
Scope: <default org search id 1372813089209061633 | single cluster>
Time Range: <start> ~ <end>
Data Source: <Clinic Data Proxy|O11Y|Direct Prometheus>
Clusters Checked: <N>
Conclusion: <one sentence>

## Actionable Findings
| Severity | Rule | Evidence | Impact |
|---|---|---|---|

## Domain Summary
| Cluster | Query | MPP | Resource | Storage IO | Network/Raft | Pipeline |
|---|---|---|---|---|---|---|

## Unknowns
None.

## Skipped Clusters
| Cluster | Reason |
|---|---|
| <cluster> | <no TiFlash topology or TiFlash node count is 0> |

When unknowns exist in compact mode:

| Scope | Missing Data | Effect |
|---|---|---|
| <cluster>/<rule> | <metric missing|selector unresolved|metric samples|query failed|series discovery failed> | <short effect on verdict> |

## Supporting Signals
- <weak Q1 retry/data-movement or no-activity histogram signals>

## Recommended Actions
1. <next action>
````

Use the full structure only when the user asks for full evidence:

````markdown
## Cluster Summary
| Cluster | Status | Coverage | TiFlash | Tier Run | Likely Bottleneck | Top Finding |
|---|---|---|---|---|---|---|

## Cluster: <cluster_name> (<cluster_id>)

### Metadata
| Field | Value |
|---|---|
| Provider / Region | <provider>/<region> |
| Deploy Type | <deploy type> |
| TiFlash Instances | <count or instance list> |
| Selector | `<concrete selector>` |
| Namespace | `<namespace>` |

### Coverage
| Domain | Status | Metric Families |
|---|---|---|

### Query Health
| Rule | Status | Observed | Threshold | Evidence |
|---|---|---|---|---|

### MPP Task Health
| Rule | Status | Observed | Threshold | Evidence |
|---|---|---|---|---|

### Resource Health
| Rule | Status | Observed | Threshold | Evidence |
|---|---|---|---|---|

### Storage IO
| Rule | Status | Observed | Threshold | Evidence |
|---|---|---|---|---|

### Network IO
| Rule | Status | Observed | Threshold | Evidence |
|---|---|---|---|---|

### Raft And Replica Health
| Rule | Status | Observed | Threshold | Evidence |
|---|---|---|---|---|

### Pipeline Model
| Rule | Status | Observed | Threshold | Evidence |
|---|---|---|---|---|

### Bottleneck Attribution
Likely Bottleneck: <Compute|Storage|Network/Exchange|Mixed|Unknown>
Confidence: <High|Medium|Low>

Evidence:
- <evidence 1>
- <evidence 2>

Unknowns:
- <classified missing data reason, with effect on verdict>

### Evidence Queries
Data Source: <Clinic Data Proxy|O11Y|Direct Prometheus>

```promql
<query>
```

### Recommended Actions
- <action>
````

Use `## Cluster: ...` for each full per-cluster section. Keep all sections
inside a cluster at `###` so batch reports remain readable and do not interleave
cluster details with top-level summary sections.

For every abnormal finding, include:

- Rule id and title.
- Metric values and affected instances.
- Threshold and persistence window.
- PromQL evidence, only in full mode or explicit query appendix mode.
- Whether the value is a hard signal or an empirical heuristic.

For every `Unknown` section in compact mode, include only the classified missing
data reason and effect on verdict. Use `metric missing`, `selector unresolved`,
`metric samples`, `query failed`, or `series discovery failed`; do not use a
generic `rule cannot be evaluated` effect. Include the label discovery query only
in full mode or explicit query appendix mode.

## Bottleneck Attribution

Compute-like:

- R1 is `Warning` or worse.
- P1 or CPU-side P5 is `Warning` or worse.
- Storage IO, network, exchange queue, and Raft signals are normal.

Storage-like:

- S2, S3, S4, P2, or IO-side P5 is `Warning` or worse.
- CPU is not saturated, or latency increases faster than CPU usage.

Network/exchange-like:

- Node network drops/errors are nonzero.
- `tiflash_proxy_tikv_server_report_failure_msg_total` is nonzero, especially
  `type="unreachable"`.
- Q3 is `Warning` or worse.
- `tiflash_exchange_queueing_data_bytes` grows while exchange/response throughput
  is near zero.

Mixed:

- Multiple layers are abnormal in the same time window and no single layer
  clearly dominates.

Unknown:

- Required metrics are missing or labels cannot be resolved.

## Evidence Discipline

- Prefer sustained conditions over isolated spikes.
- Compare affected instances against peers.
- Correlate timing across request duration, resource saturation, queue growth,
  and transport errors.
- Keep raw time series out of the final report; summarize max, p99, average,
  duration above threshold, and affected instances.