--- name: ticdc-health-inspection description: Use when auditing TiCDC health from metrics, especially for clinic.pingcap.com clusters. First distinguish TiCDC new arch from old arch by probing the `ticdc_maintainer_handle_event_duration` metric family; for new arch, use the full maintainer/dispatcher/scheduler rule set, and for old arch, use owner lag plus sink output and downstream latency rules. --- # Ticdc Health Inspection ## Overview Run a production health inspection for TiCDC with architecture-aware labels and rules. TiCDC has two runtime architectures: - `new arch`: identified by the existence of the `ticdc_maintainer_handle_event_duration` metric family. - `old arch`: `ticdc_maintainer_handle_event_duration` is absent, while `ticdc_owner_resolved_ts_lag` and `ticdc_owner_checkpoint_ts_lag` are present. If Clinic cluster metadata shows that TiCDC is not deployed in the cluster, mark the cluster status as normal and treat the TiCDC inspection as healthy instead of unknown or failed. Generate per-changefeed and cluster-level verdicts with evidence and follow-up actions. ## Load References First Read: - `references/dashboard-mapping.md` - `references/inspection-rules.md` Do not start rule evaluation before loading both files. ## Inputs Collect: - Cluster identifier in clinic. - Organization id (`org_id`) from clinic metadata. Use this value for Clinic portal links. - Organization name from clinic metadata. - Cluster name from clinic metadata. - Clinic base URL. Reuse the host from a user-provided Clinic link when present; otherwise use the active Clinic environment host. - Optional existing Clinic portal link. Reuse its `/portal/#/orgs/.../clusters/...` path directly when available. - Optional changefeed filter. - Time windows: - `short_window`: `30m` - `long_window`: `24h` - `step`: `30s` Use `short_window` for incident detection and `long_window` for trend confirmation. Keep the absolute inspected time range as `from_sec` and `to_sec` when available so the final Clinic link opens the same window. ## Architecture Selection Before classifying the TiCDC runtime architecture, first check TiCDC deployment presence: 1. Use Clinic metadata and node inventory to determine whether any TiCDC component, pod, node, service, or job is deployed. 2. If metadata conclusively shows that no TiCDC component is deployed, stop TiCDC rule evaluation and mark the result as `Healthy`. - Set `TiCDC deployment` to `not deployed`. - Set `TiCDC runtime architecture` to `N/A`. - Set cluster status to `normal`. - Set `healthy_changefeed=0` and `total_changefeed=0`. - Do not run metric coverage guards or liveness rules for TiCDC. - Do not report missing TiCDC metrics as `Unknown` or `Critical`. 3. If deployment metadata is unavailable or inconclusive, continue with runtime architecture probes. Do not infer `not deployed` from missing metrics alone. After confirming TiCDC is deployed or deployment presence is inconclusive, classify the TiCDC runtime architecture: 1. Probe the `ticdc_maintainer_handle_event_duration` metric family over `short_window`. - In Prometheus / Clinic Data Proxy this may appear as histogram series such as `ticdc_maintainer_handle_event_duration_bucket`, `_sum`, or `_count`. - Prefer a series probe with `{__name__=~"ticdc_maintainer_handle_event_duration.*", }`. 2. If series exist, classify as `new arch`. 3. If no `ticdc_maintainer_handle_event_duration.*` series exist, probe both: - `ticdc_owner_resolved_ts_lag{}` - `ticdc_owner_checkpoint_ts_lag{}` 4. If both owner lag metrics exist, classify as `old arch`. 5. If neither probe is conclusive, mark result as `Unknown` and stop. For `new arch`, select one dashboard profile before querying: - Premium architecture: - Dashboard file: `metrics/nextgengrafana/ticdc_new_arch_next_gen.json` - Cluster label: `sharedpool_id` - Keyspace label: `keyspace_name` - Dedicated architecture: - Dashboard file: `metrics/grafana/ticdc_new_arch.json` - Cluster label: `tidb_cluster` - Keyspace label: `namespace` Resolve the `new arch` dashboard profile in this order: 1. Use `clinic-api` cluster metadata. 2. Fallback: probe label presence described in `references/dashboard-mapping.md`. 3. If still unknown, mark result as `Unknown` and stop. For `old arch`, do not run the new-arch maintainer, dispatcher, scheduler, dynamic stream, event service, or event store rules. Only evaluate the old-arch rules in `references/inspection-rules.md`. ## Data Collection Use data sources in this order: 1. `clinic-api`: cluster metadata, deployment type, node inventory. 2. Clinic Data Proxy Prometheus metrics, or `o11y-metrics-api` if Clinic Data Proxy is unavailable: Prometheus `query_range` for all inspection rules. 3. `o11y-data-export`: export evidence for ticket or incident attachment. If these skills are unavailable, run equivalent PromQL directly and keep the same rule semantics. ## Cluster Clinic Link Build one report-level Clinic cluster link: - Preferred when a user already provided a Clinic link: ```text Reuse the same /portal/#/orgs//clusters/ path from that link. ``` - Preferred when building a fresh link: ```text /portal/#/orgs//clusters/?from=&to= ``` - Fallback when the exact inspected range is unavailable: ```text /portal/#/orgs//clusters/ ``` Rules: - Use `org_id` for the Clinic portal `orgs/<...>` path segment. - The same `org_id` is also used by Clinic REST paths such as `GET /clinic/api/v1/orgs/{orgID}/clusters/{clusterID}`. - Do not use `tenant_id` in the Clinic portal URL. `tenant_id` can still be relevant for O11Y / NGM headers, but it is not the value used in the Clinic portal path. - If a valid Clinic portal link is already available, prefer reusing its host and `orgs/<...>/clusters/<...>` path instead of rebuilding it from scratch. - Reuse the same Clinic host that matches the queried environment. Do not switch between `.com`, `.com.cn`, `dev`, or `staging`. - Prefer the exact inspected time range in epoch seconds so clicking the link lands on the same evidence window. ## Inspection Workflow Run checks in fixed order: 1. Resolve Clinic metadata, node inventory, and metric selectors. 2. Check whether TiCDC is deployed. 3. If TiCDC is not deployed, return `Healthy` immediately with no TiCDC findings. 4. Classify TiCDC runtime architecture as `new arch`, `old arch`, or `Unknown`. 5. If `new arch`, run: - Liveness and ownership. - Changefeed state gate. - Control-plane uniqueness (maintainer and table trigger dispatcher). - End-to-end lag decomposition. - Throughput continuity. - Backpressure and memory control. - Resource trend and capacity risks (memory, disk, CPU). - Sink and messaging errors. - DDL and scheduler health, including catch-up invariants. - Optional upstream TiKV sanity. 6. If `old arch`, run only: - Owner resolved/checkpoint lag. - Per-changefeed sink output rows per second. - Kafka sink request latency when Kafka sink metrics exist. - MySQL/TiDB sink flush latency when MySQL sink metrics exist. Apply exact metrics and thresholds from `references/inspection-rules.md`. ## Judgment Rules Use four result states: - `Critical`: any critical rule is true. - `Warning`: no critical rule, at least one warning rule is true. - `Healthy`: no critical or warning rules and data coverage is sufficient. - `Unknown`: required metrics are missing or architecture cannot be resolved. Health light mapping: - `Critical` -> `🔴` - `Warning` -> `🟡` - `Healthy` -> `🟢` - `Unknown` -> `⚫` (only when data is missing or architecture unresolved) Do not classify as `Healthy` when any changefeed is in `failed`, `warning`, or long `pending` state. Treat a changefeed as idle only when both upstream and sink throughput stay near zero in the short window. In idle mode, skip throughput mismatch rules but keep state and error checks. Do not mark a changefeed abnormal solely because `ticdc_sink_write_bytes_total` or derived `sink_write_bytes` is `0`. Zero sink write bytes is evidence only; use lag growth, state, errors, or other bottleneck rules to decide severity. ## Output Format Report with this structure: - `Health summary (🔴🟡🟢⚫)` - `Organization name` - `Cluster name` - `Cluster id` - `Clinic cluster link` - `TiCDC deployment` - `Architecture profile` - `TiCDC runtime architecture` - `Data sources used` - `Time windows` - `Critical findings` - `Warning findings` - `Per-changefeed lag decomposition` - `Per-changefeed sink output` - `Evidence queries` - `Evidence links` - `Overall verdict` - `Recommended next actions` Include concrete metric values, timestamps, and query expressions for every finding. For each abnormal metric, include at least one clickable URL to the same query and time range. Link requirements for each finding: - `explore_url` (required) - `panel_url` (optional, include when available from data source) - `dashboard_url` (optional, include when available from data source) Link requirements for the report: - `cluster_clinic_url` (required once per report) If `panel_url`/`dashboard_url` cannot be resolved, still provide `explore_url`. Finding block format: ```text - Rule: Severity: Metric: Value: Query: Time: ~ Links: - Explore: - Panel: - Dashboard: ``` Use this fixed summary template at the top: ```text Health Summary: <🔴|🟡|🟢|⚫> () Organization: Cluster: (id=) Clinic: Conclusion: Counts: critical=, warning=, healthy_changefeed=, total_changefeed= Top Risks: ``` ## Notes Use windows that are longer than metric refresh intervals in code: - Coordinator metrics refresh every `5s`. - Event service collector refreshes every `5s`. - Subscription client metrics refresh every `10s`. - Some maintainer gauges refresh every `20s`. Prefer `5m+` persistence checks to avoid false positives from sampling jitter.