# Inspection Rules

Use the selectors defined in `dashboard-mapping.md`.

## Rule Conventions

- Evaluate each rule on `short_window=30m` with `step=30s` unless noted.
- Treat a rule as true only when persistence condition is satisfied.
- Emit findings at changefeed granularity first, then aggregate to cluster verdict.

## Runtime Architecture Detection

Classify the TiCDC runtime architecture before evaluating health rules.

### P0 Deployment presence gate

Before probing TiCDC metric families, inspect Clinic metadata and node inventory.

If metadata conclusively shows that no TiCDC component, pod, node, service, or job is deployed in the cluster:

- Set `TiCDC deployment` to `not deployed`.
- Set `TiCDC runtime architecture` to `N/A`.
- Set cluster status to `normal`.
- Mark the TiCDC inspection result as `Healthy`.
- Set `critical_count=0`, `warning_count=0`, `healthy_changefeed_count=0`, and `total_changefeed_count=0`.
- Skip all TiCDC metric probes, liveness rules, data coverage guards, and per-changefeed rules.
- Do not report missing TiCDC metrics as `Unknown`, `Warning`, or `Critical`.

If metadata is unavailable or inconclusive, continue with A1/A2 probes. Do not infer `not deployed` from missing TiCDC metrics alone.

### A1 New arch probe

```promql
count({__name__=~"ticdc_maintainer_handle_event_duration.*", <cluster_selector>})
```

If the result is positive or series exist in the short window, classify the cluster as `new arch`.
This is a metric family probe: in Prometheus / Clinic Data Proxy it may appear as histogram series such as `ticdc_maintainer_handle_event_duration_bucket`, `_sum`, or `_count`.

### A2 Old arch probe

If `ticdc_maintainer_handle_event_duration.*` has no series, probe both owner lag metrics:

```promql
max(ticdc_owner_resolved_ts_lag{<cf_selector>}) by (<keyspace_label>, changefeed)
```

```promql
max(ticdc_owner_checkpoint_ts_lag{<cf_selector>}) by (<keyspace_label>, changefeed)
```

If both owner lag metrics have series in the short window, classify the cluster as `old arch`.

If neither A1 nor A2 is conclusive, mark the inspection as `Unknown` because runtime architecture cannot be resolved.

## New-Arch Rules

Use this section only after A1 classifies the cluster as `new arch`.

## State Mapping

`ticdc_owner_status` comes from `config.FeedState.ToInt()`:

- `0`: normal
- `1`: pending
- `2`: failed
- `3`: stopped
- `4`: finished
- `5`: removed/other
- `6`: warning
- `7`: uninitialized
- `-1`: unknown

Only `normal` and `warning` are considered running by code, but `warning` is degraded.

## Liveness And Ownership

### L1 Critical: TiCDC instances unavailable

```promql
count(up{job=~".*ticdc.*", <cluster_selector>} == 1)
```

Apply this rule only after P0 confirms TiCDC is deployed or deployment presence is inconclusive.
Critical when result is `0` for `>= 1m`.

### L2 Critical: owner heartbeat missing

```promql
sum(rate(ticdc_owner_ownership_counter{<cluster_selector>}[4m]))
```

Critical when `<= 0` for `>= 5m`.

### L3 Warning: frequent process restarts

```promql
changes(process_start_time_seconds{job=~".*ticdc.*", <cluster_selector>}[30m])
```

Warning when any instance value `> 2`.

### L4 Critical: owner uniqueness violated

Owner candidates query (same pattern as dashboard `Ownership History`):

```promql
sum(rate(ticdc_owner_ownership_counter{<cluster_selector>}[240s])) by (instance) > bool 0.5
```

Owner count query:

```promql
sum(sum(rate(ticdc_owner_ownership_counter{<cluster_selector>}[240s])) by (instance) > bool 0.5)
```

- Critical: owner count `> 1` for `>= 2m`.
- Warning: owner count `== 0` for `>= 2m`.

## Changefeed State Gate

### S1 Critical: failed changefeed

```promql
max(ticdc_owner_status{<cf_selector>}) by (<keyspace_label>, changefeed)
```

Critical when status `== 2`.

### S2 Warning: warning changefeed persisted

Use same query as S1.

Warning when status `== 6` for `>= 10m`.

### S3 Warning: pending too long

Use same query as S1.

Warning when status `== 1` for `>= 10m`.

## End-To-End Lag Decomposition

Define:

- `owner_ckp_lag`:

```promql
max(ticdc_owner_checkpoint_ts_lag{<cf_selector>}) by (<keyspace_label>, changefeed)
```

- `maintainer_ckp_lag`:

```promql
max(ticdc_maintainer_checkpoint_ts_lag{<cf_selector>}) by (<keyspace_label>, changefeed)
```

- `dm_ckp_lag`:

```promql
max(ticdc_dispatchermanager_checkpoint_ts_lag{<cf_selector>}) by (<keyspace_label>, changefeed)
```

- `es_sent_lag`:

```promql
max(ticdc_event_service_resolved_ts_lag{<cluster_selector>, type="sent"}) by (instance)
```

- `es_recv_lag`:

```promql
max(ticdc_event_service_resolved_ts_lag{<cluster_selector>, type="received"}) by (instance)
```

- `store_lag`:

```promql
max(ticdc_event_store_resolved_ts_lag{<cluster_selector>}) by (instance)
```

- `puller_lag`:

```promql
max(ticdc_log_puller_resolved_ts_lag{<cluster_selector>})
```

Rules:

- G1 Critical: `owner_ckp_lag > 600` for `>= 10m`.
- G2 Warning: `owner_ckp_lag > 180` for `>= 5m`.
- G3 Warning: `(owner_ckp_lag - maintainer_ckp_lag) > 60` for `>= 5m`.
- G4 Warning: `(maintainer_ckp_lag - dm_ckp_lag) > 60` for `>= 5m`.
- G5 Warning: `(es_sent_lag - es_recv_lag) > 60` for `>= 5m`.
- G6 Warning: `(store_lag - es_recv_lag) > 60` for `>= 5m`.

## Control-Plane Uniqueness

### C1 Critical: changefeed has multiple maintainers

```promql
sum(ticdc_changefeed_maintainer_counter{<cf_selector>}) by (<keyspace_label>, changefeed)
```

Critical when value `> 1`.

### C2 Warning: running changefeed maintainer count is not exactly one

Use C1 query with S1 state query together.

Warning when state is running (`0`, `1`, `6`) and maintainer count `!= 1` for `>= 5m`.

### C3 Critical: table trigger dispatcher multiplicity per event type

```promql
sum(ticdc_dispatchermanager_table_trigger_dispatcher_count{<cf_selector>}) by (<keyspace_label>, changefeed, event_type)
```

Critical when any `(<keyspace_label>, changefeed, event_type)` value `> 1`.

### C4 Warning: missing table trigger event dispatcher for running changefeed

Use C3 query.

Warning when running changefeed has `event_type="eventDispatcher"` count `!= 1` for `>= 5m`.

## Throughput Continuity

Define:

- `upstream_write_bytes`:

```promql
sum(rate(tidb_tikvclient_txn_write_size_bytes_sum{<cluster_selector>, scope=~"general"}[1m]))
```

- `sink_rows`:

```promql
sum(rate(ticdc_sink_dml_event_count{<cf_selector>}[1m])) by (<keyspace_label>, changefeed)
```

- `sink_write_bytes`:

```promql
sum(rate(ticdc_sink_write_bytes_total{<cf_selector>}[1m])) by (<keyspace_label>, changefeed)
```

Idle condition:
- `upstream_write_bytes < 1024` and `sink_rows < 1` for the whole short window.

Rules:
- T1 Critical: not idle, `upstream_write_bytes` is high (`> 1 MiB/s`), `sink_rows < 1`, and `owner_ckp_lag` is increasing for `>= 5m`.
- T2 Evidence: report `sink_write_bytes` for context only.

Do not emit a warning or critical finding solely because `sink_write_bytes` is `0` or near zero.
Some healthy changefeeds and sink types can have zero `ticdc_sink_write_bytes_total` during the inspected window.
Use lag growth, changefeed state, sink errors, sink rows, or sink-specific latency rules to determine abnormality.

## Backpressure And Memory

Memory ratio by module:

```promql
sum(ticdc_dynamic_stream_memory_usage{<cluster_selector>, module="<module>", type="used"}) /
sum(ticdc_dynamic_stream_memory_usage{<cluster_selector>, module="<module>", type="max"})
```

Use modules:
- `event-collector`
- `event-collector-redo`
- `log-puller`

Rules:
- B1 Warning: any module ratio `> 0.80` for `>= 3m`.
- B2 Critical: any module ratio `> 0.95` for `>= 1m` and pending queue length keeps growing.
- B3 Warning: `increase(ticdc_event_service_skip_scan_count_sum{<cluster_selector>}[5m]) > 100`.
- B4 Critical: `increase(ticdc_event_service_skip_scan_count_sum{<cluster_selector>}[5m]) > 500`.
- B5 Warning: `ticdc_event_service_pending_scan_task_count` has positive slope for `>= 10m`.

Pending queue references:

```promql
sum(ticdc_dynamic_stream_pending_queue_len{<cluster_selector>, module=~"event-collector|event-collector-redo|log-puller"}) by (instance, module)
```

## Resource Trend And Capacity Risks

### R1 Memory keeps rising (OOM risk)

Estimate total memory from CPU cores:

```text
total_memory_bytes = (cpu_cores + 1) * 4 GiB
cpu_cores uses ticdc_server_go_max_procs
```

Example:
- `cpu_cores=31` -> `total_memory=(31+1)*4=128 GiB`.

Synthetic total memory per instance:

```promql
on(instance) (ticdc_server_go_max_procs{job=~".*ticdc.*", <cluster_selector>} + 1) * 4 * 1024 * 1024 * 1024
```

RSS growth ratio in 30 minutes:

```promql
100 *
increase(process_resident_memory_bytes{job=~".*ticdc.*", <cluster_selector>}[30m]) /
on(instance) ((ticdc_server_go_max_procs{job=~".*ticdc.*", <cluster_selector>} + 1) * 4 * 1024 * 1024 * 1024)
```

RSS usage ratio:

```promql
100 *
process_resident_memory_bytes{job=~".*ticdc.*", <cluster_selector>} /
on(instance) ((ticdc_server_go_max_procs{job=~".*ticdc.*", <cluster_selector>} + 1) * 4 * 1024 * 1024 * 1024)
```

RSS slope:

```promql
deriv(process_resident_memory_bytes{job=~".*ticdc.*", <cluster_selector>}[30m])
```

Guard condition:
- Apply this rule only when RSS usage ratio is already `> 50%` to reduce false positives.

- Warning: growth ratio `> 10%` in `30m`, slope `> 0` for `>= 15m`, and usage ratio `> 50%`.
- Critical: growth ratio `> 30%` in `30m`, slope `> 0` for `>= 15m`, and usage ratio `> 50%`.
- Escalate one level when usage ratio is already `> 70%`.

### R2 Data size on disk keeps rising (capacity risk)

Event-store on-disk size total:

```promql
sum(ticdc_event_store_on_disk_data_size{<cluster_selector>})
```

On-disk growth ratio in 30 minutes:

```promql
100 *
delta(sum(ticdc_event_store_on_disk_data_size{<cluster_selector>})[30m:1m]) /
clamp_min(avg_over_time(sum(ticdc_event_store_on_disk_data_size{<cluster_selector>})[30m:1m]), 1)
```

Guard condition:
- Apply this rule only when on-disk total is already `> 1 GiB` to avoid tiny-base noise.

- Warning: growth ratio `> 10%` in `30m` and positive slope for `>= 10m`.
- Critical: growth ratio `> 30%` in `30m` and positive slope for `>= 10m`.

### R3 CPU risk grading

CPU percent per TiCDC instance:

```promql
100 *
rate(process_cpu_seconds_total{job=~".*ticdc.*", <cluster_selector>}[1m]) /
on(instance) ticdc_server_go_max_procs{job=~".*ticdc.*", <cluster_selector>}
```

- Warning (medium risk): `> 80` for `>= 5m`.
- Critical (high risk): `> 90` for `>= 5m`.

## Sink And Messaging Errors

### E1 Sink execution errors

```promql
sum(increase(ticdc_sink_execution_error{<cf_selector>}[5m])) by (<keyspace_label>, changefeed, event_type)
```

- Warning: any increase `> 0`.
- Critical: total increase `> 50` in `5m`.

### E2 Message drop and congestion

```promql
sum(rate(ticdc_messaging_drop_message_counter{<cluster_selector>}[1m])) by (instance, type)
```

```promql
sum(rate(ticdc_messaging_error_counter{<cluster_selector>}[1m])) by (instance, type, message)
```

- Warning: drop rate `> 0`.
- Critical: drop rate `> 10/s` for `>= 1m`.

### E3 Resolve lock tasks dropped

```promql
increase(ticdc_subscription_client_resolve_lock_task_drop_count{<cluster_selector>}[5m])
```

- Warning: `> 0`.
- Critical: `> 100`.

## DDL And Scheduler

### D1 Blocking DDL persisted

```promql
max(ticdc_ddl_exec_blocking{<cf_selector>}) by (<keyspace_label>, changefeed, mode)
```

- Warning: `> 0` for `>= 5m`.
- Critical: `> 0` for `>= 30m`.

### D2 Maintainer event channel backlog

```promql
max(ticdc_maintainer_event_ch_len{<cf_selector>}) by (<keyspace_label>, changefeed)
```

- Warning: `> 1000` for `>= 5m`.
- Critical: `> 5000` for `>= 5m`.

### D3 Table replication absent count

```promql
sum(ticdc_scheduler_table_replication_state{<cf_selector>, state="Absent"}) by (<keyspace_label>, changefeed)
```

Warning when sustained `> 0` with lag growth.

### D4 Catch-up invariant: schedule tasks should be zero

Catch-up condition:
- `owner_ckp_lag < 10` for `>= 10m`.

Schedule task query:

```promql
sum(ticdc_scheduler_task{<cf_selector>}) by (<keyspace_label>, changefeed, mode)
```

Warning when catch-up condition is true but schedule tasks stay `> 0` for `>= 10m`.

### D5 Catch-up invariant: absent table replication state should be zero

Use D3 query.

Warning when catch-up condition is true and absent count stays `> 0` for `>= 10m`.

## Optional Sink-Specific Bottleneck Hints

Use only when corresponding sink metrics exist:

- MySQL/TiDB sink:
  - Worker busy ratio:

```promql
sum(rate(ticdc_sink_txn_worker_batch_flush_duration_sum{<cf_selector>}[1m])) /
sum(rate(ticdc_sink_txn_worker_total_duration_sum{<cf_selector>}[1m])) * 100
```

Warning when `> 95` for `>= 10m` with lag growth.

- MQ sink:
  - `ticdc_sink_mq_worker_send_message_duration` p99 high (`> 5s`) with lag growth.
  - Kafka in-flight requests keeps increasing.

- Cloud storage sink:
  - `ticdc_sink_cloud_storage_worker_busy_ratio` near saturation and write/flush duration high.

- Redo:
  - `ticdc_redo_worker_busy_ratio` near saturation with lag growth.

## Old-Arch Rules

Use this section only after A2 classifies the cluster as `old arch`.
Do not evaluate new-arch-only maintainer, dispatcher, scheduler, dynamic stream, event service, or event-store rules for old-arch clusters.

Old-arch rules use these metrics as the core health signals:

- `ticdc_owner_resolved_ts_lag`
- `ticdc_owner_checkpoint_ts_lag`
- `ticdc_sink_batch_row_count_sum`
- `ticdc_sink_kafka_producer_request_latency`
- `ticdc_sink_txn_worker_flush_duration_sum`
- `ticdc_sink_txn_worker_flush_duration_count`

### O1 Critical: owner checkpoint lag critical

```promql
max(ticdc_owner_checkpoint_ts_lag{<cf_selector>}) by (<keyspace_label>, changefeed)
```

Critical when value `> 600` seconds for `>= 10m`.

### O2 Warning: owner checkpoint lag warning

Use the O1 query.

Warning when value `> 180` seconds for `>= 5m`.

### O3 Critical: owner resolved lag critical

```promql
max(ticdc_owner_resolved_ts_lag{<cf_selector>}) by (<keyspace_label>, changefeed)
```

Critical when value `> 600` seconds for `>= 10m`.

### O4 Warning: owner resolved lag warning

Use the O3 query.

Warning when value `> 180` seconds for `>= 5m`.

### O5 Evidence: old-arch sink output rows per second

```promql
sum(rate(ticdc_sink_batch_row_count_sum{<cf_selector>, instance=~"<ticdc_instance_regex>"}[2m]))
by (<keyspace_label>, changefeed, instance)
```

Use this value to report per-changefeed sink output throughput.
This metric is evidence by default, not an automatic warning.

Warning when checkpoint lag or resolved lag is already warning-level and sink output stays `< 1 row/s` for `>= 5m`.

### O6 Warning: Kafka sink producer request latency high

```promql
sum(ticdc_sink_kafka_producer_request_latency{<cf_selector>, instance=~"<ticdc_instance_regex>"})
by (<keyspace_label>, changefeed, instance, broker)
```

If this metric exists for a changefeed, treat that changefeed as Kafka sink.

Warning when any value `> 1` second for `>= 5m`.

### O7 Warning: MySQL/TiDB sink flush latency high

```promql
sum(rate(ticdc_sink_txn_worker_flush_duration_sum{<cf_selector>, instance=~"<ticdc_instance_regex>"}[2m]))
by (<keyspace_label>, changefeed, instance)
/
sum(rate(ticdc_sink_txn_worker_flush_duration_count{<cf_selector>, instance=~"<ticdc_instance_regex>"}[2m]))
by (<keyspace_label>, changefeed, instance)
```

If this metric exists for a changefeed, treat that changefeed as MySQL/TiDB sink.

Warning when any value `> 1` second for `>= 5m`.

## Data Coverage Guard

Bypass data coverage guards when P0 conclusively shows that TiCDC is not deployed; that case is `Healthy`, not `Unknown`.

Mark a `new arch` result as `Unknown` when either condition holds:
- New-arch dashboard profile cannot be resolved.
- Any new-arch core lag signal is missing in the whole short window:
  - `owner_ckp_lag`
  - `maintainer_ckp_lag`
  - `dm_ckp_lag`
  - `es_sent_lag`

Mark an `old arch` result as `Unknown` when any old-arch core lag signal is missing in the whole short window:
- `ticdc_owner_resolved_ts_lag`
- `ticdc_owner_checkpoint_ts_lag`

Do not mark an old-arch result as `Unknown` only because Kafka or MySQL/TiDB sink latency metrics are absent. Their absence means that sink type was not detected from those metric families.

## Final Verdict Logic

- `Critical`: at least one critical rule is true.
- `Warning`: no critical rule and at least one warning rule is true.
- `Healthy`: no critical or warning rule and data coverage is complete, or P0 conclusively shows TiCDC is not deployed.
- `Unknown`: fails data coverage guard.

## Health Light Summary

Map verdict to health light:
- `Critical` -> `🔴`
- `Warning` -> `🟡`
- `Healthy` -> `🟢`
- `Unknown` -> `⚫`

Cluster-level summary rule:
- If TiCDC is conclusively not deployed, cluster overall verdict is `Healthy` for TiCDC health inspection.
- Calculate per-changefeed verdict first.
- Cluster overall verdict takes the maximum severity:
  - `Critical` > `Warning` > `Healthy`.
- If data coverage guard fails for core metrics, use `Unknown/⚫`.

Required summary output fields:
- `org_name`
- `cluster_name`
- `cluster_id`
- `health_light`
- `overall_verdict`
- `critical_count`
- `warning_count`
- `healthy_changefeed_count`
- `total_changefeed_count`
- `top_risks` (max 3, ordered by severity)