# Pipeline Metric Failure Policy Reference

Scope:
- `pipeline/internal/otel_collector/processor/thanos/thanos.go`
- `pipeline/internal/otel_collector/receiver/metadb/fetcher_by_taskqueue.go`
- `pipeline/internal/otel_collector/exporter/otlphttpexporter/factory.go`
- `pipeline/internal/otel_collector/extension/taskqueue/runners.go`

This document describes how a metric query failure affects one task window in the current pipeline implementation.

## 1) High-level rules

For one task window, the current code applies the following rules:

1. Most core metric query failures immediately `nack` the task and return an error.
2. A small set of DM auxiliary metrics only log the failure and continue.
3. Cluster-local preconditions can fail for one cluster without failing the whole task if other clusters still produce datapoints.
4. If the task finishes with no datapoints at all, the task is `nack`ed.
5. The exporter is the normal place where a successful task is finally `ack`ed.
6. `taskqueue` owns the final retry/bypass policy: any requeued task retries until `now - firstFailedAt >= 5m`, then `ack and continue`.

## 2) Decision points by layer

### Receiver layer

File: `pipeline/internal/otel_collector/receiver/metadb/fetcher_by_taskqueue.go`

- `fetchData(...)` returns error:
  - log error
  - call `taskqueue.NackWithRequeue(...)`
- `ConsumeMetrics(...)` returns error:
  - only log at receiver layer
  - task outcome then depends on whether downstream explicitly `ack`ed or `nack`ed, or whether taskqueue timeout requeues it

### Processor layer

File: `pipeline/internal/otel_collector/processor/thanos/thanos.go`

- Most core metrics:
  - log error
  - `q.nackTask(task)`
  - return error immediately
- DM auxiliary metrics:
  - log error only
  - continue querying the rest of the task
- No datapoints in the whole task window:
  - log warning
  - `q.nackTask(task)`
  - return error
- Partial datapoints:
  - update `task.AckTimestamp` to the last existing datapoint timestamp truncated to `task.Step`
  - return success

### Exporter layer

File: `pipeline/internal/otel_collector/exporter/otlphttpexporter/factory.go`

- exporter send succeeds:
  - call `taskQueue.Ack(task)`
- exporter send fails:
  - call `taskqueue.NackWithRequeue(...)`

### Taskqueue layer

File: `pipeline/internal/otel_collector/extension/taskqueue/runners.go`

- first `nack` or timeout for one task:
  - set `firstFailedAt` if not set yet
- before `5m` since `firstFailedAt`:
  - requeue the same task window
- after `5m` since `firstFailedAt`:
  - set `AckTimestamp = EndTimestamp.Truncate(Step)`
  - persist progress
  - generate the next task window

## 3) Metric groups and failure policy

| Group | Metric | Failure behavior | Notes |
| --- | --- | --- | --- |
| Dedicated core | `tidb_cloud.node_memory_capacity_byte` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.node_memory_used_bytes` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.node_memory_available_bytes` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.node_cpu_seconds_total` | Immediate `nack` and return error | Dedicated-only, Datadog-oriented |
| Dedicated core | `tidb_cloud.node_cpu_seconds_total_rate` | Immediate `nack` and return error | Dedicated-only, New Relic-oriented |
| Dedicated core | `tidb_cloud.node_storage_capacity_bytes` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.node_storage_used_bytes` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.node_storage_available_bytes` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.tidb_cloud.db_component_uptime` | Immediate `nack` and return error | Dedicated-only |
| DM core | `tidb_cloud.dm_task_state` | Immediate `nack` and return error | Dedicated-only |
| DM auxiliary | `tidb_cloud.dm_syncer_replication_lag_bucket` | Log only, continue | Dedicated-only |
| DM auxiliary | `tidb_cloud.dm_syncer_replication_lag_gauge` | Log only, continue | Dedicated-only |
| DM auxiliary | `tidb_cloud.dm_relay_read_error_count` | Log only, continue | Dedicated-only |
| Dedicated core | `tidb_cloud.db_kv_request_duration` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.node_disk_read_latency` | Immediate `nack` and return error | Dedicated-only |
| Dedicated core | `tidb_cloud.node_disk_write_latency` | Immediate `nack` and return error | Dedicated-only |
| Shared DB core | `tidb_cloud.db_total_connection` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_active_connections` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_disconnections` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_database_time` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_query_per_second` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_failed_queries` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_command_per_second` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_queries_using_plan_cache_ops` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_average_query_duration` | Immediate `nack` and return error | All plans |
| Shared DB core | `tidb_cloud.db_transaction_per_second` | Immediate `nack` and return error | All plans |
| RU core | `tidb_cloud.resource_manager_resource_unit_read_request_unit` | Immediate `nack` and return error | All plans |
| RU core | `tidb_cloud.resource_manager_resource_unit_write_request_unit` | Immediate `nack` and return error | All plans |
| CDC core | `tidb_cloud.cdc_changefeed_latency` | Immediate `nack` and return error | CDC-enabled plans |
| CDC core | `tidb_cloud.cdc_changefeed_resolved_ts_lag` | Immediate `nack` and return error | CDC-enabled plans |
| CDC core | `tidb_cloud.cdc_changefeed_status` | Immediate `nack` and return error | CDC-enabled plans |
| CDC core | `tidb_cloud.cdc_changefeed_replica_rows` | Immediate `nack` and return error | CDC-enabled plans |
| NextGen core | `tidb_cloud.db_row_storage_used_bytes` | Immediate `nack` and return error | NextGen only |
| NextGen core | `tidb_cloud.db_columnar_storage_used_bytes` | Immediate `nack` and return error | NextGen only |

## 4) Cluster-level preconditions

Some failures are handled at cluster granularity rather than metric granularity.

Examples:
- cluster metadata parsing fails
- metric data address injection fails
- cluster is not eligible for a plan-specific metric group

In those cases the current cluster can be skipped while the task continues with other clusters. The task only becomes a hard failure if the whole task window ends with no datapoints.

## 5) Worked examples

### Example A: one soft-fail metric does not block the task

Task window:
- `Timestamp = 10:00:00`
- `EndTimestamp = 10:01:00`

Results:
- `tidb_cloud.dm_task_state` succeeds with last datapoint at `10:00:45`
- `tidb_cloud.dm_syncer_replication_lag_bucket` fails
- `tidb_cloud.dm_syncer_replication_lag_gauge` fails
- `tidb_cloud.dm_relay_read_error_count` fails

Outcome:
- task is not nack'ed by those auxiliary failures
- `AckTimestamp` becomes `10:00:45`
- if exporter succeeds, the task is acked and progress advances

### Example B: one core metric failure retries the whole window

Task window:
- `Timestamp = 10:00:00`
- `EndTimestamp = 10:01:00`

Results:
- `tidb_cloud.db_total_connection` succeeds with last datapoint at `10:00:30`
- `tidb_cloud.db_active_connections` succeeds with last datapoint at `10:00:45`
- `tidb_cloud.db_kv_request_duration` fails

Outcome:
- processor immediately `nack`s the task
- the whole task window is retried by `taskqueue`
- if retries keep failing for 5 minutes since `firstFailedAt`, taskqueue bypasses the window and continues

## 6) Useful source references

- Metric failure branches: `pipeline/internal/otel_collector/processor/thanos/thanos.go`
- Receiver nack path: `pipeline/internal/otel_collector/receiver/metadb/fetcher_by_taskqueue.go`
- Exporter ack/nack path: `pipeline/internal/otel_collector/exporter/otlphttpexporter/factory.go`
- Retry/bypass policy: `pipeline/internal/otel_collector/extension/taskqueue/runners.go`