# Essential and Starter RU PromQL Alignment

## Summary

This change updates the `tidbcloud_resource_manager_resource_request_unit_total` PromQL query
used by pipeline for Essential and Starter clusters to use the serverless RU formula while still
scoping every term to the target keyspace.

## Background

Before this fix, `QueryRUTotal` in pipeline used the same matcher-based formula for all NextGen
service plans:

```
rate(resource_manager_resource_unit_read_request_unit_sum{<matcher>, type=~"|tp"}[2m])
+ rate(resource_manager_resource_unit_write_request_unit_sum{<matcher>, type=~"|tp"}[2m])
```

For Dedicated and Premium clusters this formula works correctly, because those clusters carry
per-cluster label matchers (`tidb_cluster_id`, `resource_group`, etc.) that scope the query to
a single cluster's data.

Essential and Starter clusters run on shared infrastructure. The label-scoped matcher formula
returns `0` on their datasource because the raw component-level series (`type=~"|tp"`) used by
the non-serverless formula are not exposed there.

The serverless datasource is also not single-cluster isolated. A production cn-hongkong Essential
validation showed multiple `resource_group` values in the same datasource, including non-zero
`default` RU. The RU series did not expose `cluster_id`, `tidb_cluster_id`, or `keyspace_id`
selectors; the available per-keyspace selector was `resource_group`.

For the validated Essential cluster:

| Field | Value |
|-------|-------|
| `cluster_id` | `10821642679432589811` |
| resolved `keyspace_id` | `4` |
| RU selector | `resource_group="4"` |

The RU dependency series supported these relevant labels:

| Metric | Useful selector | Notes |
|--------|-----------------|-------|
| `resource_manager_resource_unit_read_request_unit_sum` | `resource_group` | Also has `name`, `tidb_cluster`, infra labels; no `cluster_id`/`keyspace_id`. |
| `resource_manager_resource_unit_write_request_unit_sum` | `resource_group` | Same label shape as read RU. |
| `resource_manager_resource_unit_sql_layer_request_unit_sum` | `resource_group` | Same keyspace selector, no `type` label. |

Because `resource_group=default` and other groups can have non-zero values, the serverless
aggregate formula must not be used without a keyspace/resource-group matcher.

## Change

`QueryRUTotal` now branches on service plan before building the PromQL expression:

- **Essential / Starter** — resolves the target cluster keyspace ID, converts it to
  `resource_group="<keyspace_id>"`, then applies that matcher to every term in the serverless
  RU formula.
- **All other plans** (Premium, Dedicated, etc.) — keeps the existing matcher-scoped formula
  unchanged.

The serverless PromQL used for Essential and Starter is:

```promql
(sum(rate(resource_manager_resource_unit_read_request_unit_sum{resource_group="<keyspace_id>"}[1m])) or vector(0))
+ (sum(rate(resource_manager_resource_unit_write_request_unit_sum{resource_group="<keyspace_id>"}[1m])) or vector(0))
+ (sum(rate(resource_manager_resource_unit_sql_layer_request_unit{resource_group="<keyspace_id>"}[1m])) or vector(0))
+ (sum(rate(resource_manager_resource_unit_sql_layer_request_unit_sum{resource_group="<keyspace_id>"}[1m])) or vector(0))
+ (sum(rate(gateway_connection_flow_bytes_sum{namespace="tidb-admin",direction="out",resource_group="<keyspace_id>"}[1m])/(1024)) or vector(0))
+ (sum(rate(sum(tikv_worker_load_data_wru_cost_counter{resource_group="<keyspace_id>"})[1m])) or vector(0))
+ (sum(increase(sum(export_ru_cost_counter{resource_group="<keyspace_id>"})[120s])/120) or vector(0))
```

Each term is guarded by `or vector(0)` so that missing series produce a `0` contribution
rather than making the whole expression return an empty vector. The matcher is intentionally
also applied to optional terms that were absent in the validated datasource; if those series
appear later without a keyspace/resource-group label, they should not be mixed into a single
instance's RU total.

`resource_group` is only an internal datasource selector for Essential and Starter. The exported
Prometheus metric remains an instance-level total and does not expose a `resource_group` label:

```prometheus
tidbcloud_resource_manager_resource_request_unit_total{instance_name="<instance_name>",instance_id="<instance_id>"} <value>
```

## Validation Notes

Direct datasource checks through mirrord showed that unscoped RU queries mix data from multiple
resource groups. Example values from the validated cn-hongkong datasource:

| Query | Approximate value |
|-------|-------------------|
| Full serverless formula without selector | `35386` |
| Same formula with `resource_group="4"` | `31058` |
| `resource_group="default"` contribution | `4329` |

Selector checks for `resource_manager_resource_unit_read_request_unit_sum`:

| Selector | Result |
|----------|--------|
| `{cluster_id="10821642679432589811"}` | `0` |
| `{tidb_cluster_id=~".*10821642679432589811"}` | `0` |
| `{keyspace_id="4"}` | `0` |
| `{resource_group="4"}` | non-zero |

## Files Changed

| File | Change |
|------|--------|
| `pipeline/internal/datasource/central_thanos/central_thanos.go` | Added serverless RU total helper with matcher support; updated `QueryRUTotal` to resolve keyspace and use `resource_group="<keyspace_id>"` for Essential/Starter. |
| `pipeline/internal/datasource/central_thanos/prometheus_builder.go` | Keeps `resource_group` out of the exported Essential/Starter total RU metric labels. |
| `pipeline/internal/datasource/central_thanos/cluster_matcher_test.go` | Verifies Essential/Starter RU total PromQL includes the `resource_group` matcher. |
| `pipeline/tools/prom-e2e/dashboard/essential-grafana.json` | Points the Essential RU panel at the exported `tidbcloud_resource_manager_resource_request_unit_total` metric without grouping by `resource_group`. |
| `pipeline/tools/prom-e2e/dashboard/starter-grafana.json` | Points the Starter RU panel at the exported `tidbcloud_resource_manager_resource_request_unit_total` metric without grouping by `resource_group`. |
| `artifacts/dashboard/grafana/{essential,starter}-grafana.json` | Keeps released Grafana dashboards aligned with the exported total RU metric and no `resource_group` grouping. |
| `artifacts/dashboard/datadog/{essential,starter}-datadog.json` | Keeps released Datadog dashboards aligned with the exported total RU metric and no `resource_group` grouping. |
| `artifacts/dashboard/newrelic/{essential,starter}-newrelic.json` | Keeps released New Relic dashboards aligned with the exported total RU metric and no `resource_group` facet. |