# Starter TiDB monitoring contract

Date: 2026-07-11
Status: implementation input for issue #788 / task #179
CP9 baseline: `controlplane9@4e9a28f783b4b3a7ac9ad1da32568942d49e71ce`

## Decision

Starter observability should integrate a normal Prometheus/Grafana stack using the current TiDB Operator `PodMonitor` discovery model as its upstream reference. It should not make the legacy `TidbMonitor` CR the production boundary: current TiDB Operator v2 removed that CR specifically because it is difficult to integrate into production-grade monitoring systems ([source](https://github.com/pingcap/docs-tidb-operator/blob/ffd87feda942bad491ce49240c56d72891124346/en/v2-vs-v1.md#L83-L85)).

The required Starter scrape surface is:

1. the root PD;
2. both custom TSO instances and the custom Resource Manager instance;
3. the root TiDB plus every live TiDB pod in the runtime-selected serving lane;
4. all three heterogeneous TiKV instances;
5. one node-exporter, kubelet, and cAdvisor target per Ready Linux node, plus kube-state-metrics;
6. the currently annotated CSE workers and DM master as conditional completeness targets, without making the core TiDB dashboards depend on them.

`tidb-gateway-inner` is not TiProxy and currently has no Prometheus endpoint. Its expected TiProxy target count is therefore **zero**, and the TiProxy dashboard is explicitly unsupported rather than silently “healthy.”

All database targets must carry one canonical logical cluster identity:

```text
k8s_cluster = <resolved Starter EKS/Kubernetes cluster name>
tidb_cluster = tidb-serverless-serverless-cluster
```

These labels must be attached at scrape time. Prometheus documents `external_labels` as labels added when communicating with external systems such as federation, remote storage, and Alertmanager ([source](https://github.com/prometheus/prometheus/blob/f282b5c83b40a9fccb58825662326203a4c07457/docs/configuration/configuration.md#L85-L92)); therefore relying on `external_labels` alone for Grafana queries against the local TSDB is not a valid contract.

The pinned dashboard bundle requires **two datasource names**, not one. Provision one physical Prometheus through both aliases:

| Grafana datasource | Required value |
| --- | --- |
| Primary | name `tidb-cluster`, UID `PB0C37C6A9EC17FDC`, default `true` |
| Kubernetes alias | name `k8s-cluster`, a separate stable UID, default `false` |
| URL/type | the same in-cluster Prometheus URL, type `prometheus` |

Using the upstream hard-coded UID on the primary datasource also satisfies the few object-style datasource references in `ticdc_new_arch.json` ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/ticdc_new_arch.json#L5644-L5651)). This avoids rewriting any dashboard datasource query.

## Source ledger

Only primary repositories and the current CP9 tree were used.

| Source | Fixed revision | Purpose |
| --- | --- | --- |
| [`pingcap/monitoring`](https://github.com/pingcap/monitoring/tree/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator) | `89115a1ae6e7cb6d9f522330bd174fab329d8fa8` | 25 dashboard JSON files, provisioning transform, datasource names |
| [`pingcap/tidb-operator`](https://github.com/pingcap/tidb-operator/tree/7bcbb0ef23ba72c9aae0bffd67389915ffbfdb32) | `7bcbb0ef23ba72c9aae0bffd67389915ffbfdb32` | current component annotations, ports, and paths |
| [`pingcap/tidb-operator` release-1.6](https://github.com/pingcap/tidb-operator/tree/83884b9589e6dae2ebefafde6b1e69199b1c2c84) | `83884b9589e6dae2ebefafde6b1e69199b1c2c84` | v1 discovery/relabel behavior matching CP9's current CRs |
| [`docs-tidb-operator`](https://github.com/pingcap/docs-tidb-operator/tree/ffd87feda942bad491ce49240c56d72891124346) | `ffd87feda942bad491ce49240c56d72891124346` | current supported Prometheus integration |
| [`kube-prometheus`](https://github.com/prometheus-operator/kube-prometheus/tree/a44e949d79c6a608fe2261edcff9adaef7c00923) | `a44e949d79c6a608fe2261edcff9adaef7c00923` | node-exporter, KSM, kubelet, and cAdvisor baseline |
| [`prometheus`](https://github.com/prometheus/prometheus/tree/f282b5c83b40a9fccb58825662326203a4c07457) | `f282b5c83b40a9fccb58825662326203a4c07457` | Kubernetes SD and external-label semantics |
| [`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics/tree/v2.19.1) | tag `v2.19.1`, commit `45e1513e95beaaa2f2df92af9958cd9837bb66ab` | current metric names and label allowlist |
| [`tidbcloud/controlplane9`](https://github.com/tidbcloud/controlplane9/tree/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce) | `4e9a28f783b4b3a7ac9ad1da32568942d49e71ce` | current Starter workload and certificate inventory |

## Upstream discovery contract

Current TiDB Operator documentation uses a `PodMonitor` that:

- selects pods managed by `tidb-operator`;
- keeps only `prometheus.io/scrape=true` targets;
- uses HTTPS and a CA/client certificate/key when cluster TLS is enabled;
- rewrites the address to the component peer DNS name;
- maps namespace, cluster, pod name, and component to `kubernetes_namespace`, `cluster`, `instance`, and `component`;
- constructs `tidb_cluster` as `${namespace}-${instance}`.

The exact example is in [`monitor-a-tidb-cluster.md`](https://github.com/pingcap/docs-tidb-operator/blob/ffd87feda942bad491ce49240c56d72891124346/en/monitor-a-tidb-cluster.md#L24-L111). TiDB Operator v1 generated the same essential target-label vocabulary and peer-DNS rewrite ([jobs](https://github.com/pingcap/tidb-operator/blob/83884b9589e6dae2ebefafde6b1e69199b1c2c84/pkg/monitor/monitor/template.go#L94-L117), [relabeling](https://github.com/pingcap/tidb-operator/blob/83884b9589e6dae2ebefafde6b1e69199b1c2c84/pkg/monitor/monitor/template.go#L296-L435)).

CP9 cannot copy that example unchanged:

- its TSO and Resource Manager are custom StatefulSets rather than standard operator-generated pods;
- its three TiKV `TidbCluster` objects have distinct metadata names and only reference the root cluster through `spec.cluster` ([p0](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/serverless-tikv-cluster.yaml#L1-L34), [p1](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/serverless-tikv-cluster.yaml#L171-L204), [p2](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/serverless-tikv-cluster.yaml#L341-L374)); v1 derives `app.kubernetes.io/instance` from each CR name when no instance label is present ([source](https://github.com/pingcap/tidb-operator/blob/83884b9589e6dae2ebefafde6b1e69199b1c2c84/pkg/apis/pingcap/v1alpha1/tidbcluster.go#L1248-L1257));
- the selected pool's TiDB pods live in a different namespace and do not have the operator's peer-service layout;
- Kubernetes pod SD creates a target for every declared container port ([source](https://github.com/prometheus/prometheus/blob/f282b5c83b40a9fccb58825662326203a4c07457/docs/configuration/configuration.md#L2480-L2505)), so a raw scrape job must keep the annotated metrics port before rewriting the address or it can scrape the same endpoint more than once.

The CP9 implementation therefore needs explicit, tested scrape jobs/relabel rules around the upstream vocabulary.

## Required target matrix

`N` below is the resolved number of Ready Linux nodes, not a hard-coded constant. With the planned six-role defaults, `N` is 17 (`3+3+3+4+3+1`), but status must compare against the resolved topology.

| Target | Discovery and expected count | Endpoint | Required `job` and `component` | Notes |
| --- | --- | --- | --- | --- |
| Root PD | root `TidbCluster`, 1 | HTTPS `/metrics`, `2379`, peer DNS | `pd` | Current root CR enables TLS and has one PD ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/serverless-cluster.yaml#L1-L35)). |
| TSO PDMs | `pd-tso-server`, 2 | HTTPS `/metrics`, `2379`, `<pod>.tso-service.tidb-serverless.svc` | `tso` | The annotations, replicas, and advertised DNS are explicit in the current manifest ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/pd-tso-server.yaml#L30-L84)). |
| Resource Manager PDM | `pd-resource-manager-server`, 1 | HTTPS `/metrics`, `2379`, `<pod>.resource-manager-service.tidb-serverless.svc` | `resource-manager` | The v1 generated monitor did not have an RM job, so this is a CP9-owned explicit job ([manifest](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/pd-resource-manager-server.yaml#L37-L91)). |
| Root TiDB | root `TidbCluster`, 1 | HTTPS `/metrics`, `10080`, peer DNS | `tidb` | The CR has one TLS-enabled TiDB ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/serverless-cluster.yaml#L224-L267)); v1 annotates TiDB status port 10080 ([source](https://github.com/pingcap/tidb-operator/blob/83884b9589e6dae2ebefafde6b1e69199b1c2c84/pkg/manager/member/tidb_member_manager.go#L1096-L1099)). |
| Product-owned TiDB lanes | all live TiDB pods observed across `tidb-pool` and `super-vip-tidb-pool`; runtime pod facts determine which lane is active | HTTPS `/metrics`, `10080`, Pod IP with an explicit TLS server name | `tidb` | The regular template has scrape annotations ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/envs/dev-full/starter-regional/tidb-pool/tidb-server-pod-template.yaml#L7-L57)); the reviewed super-vip template has the port/TLS probes but no scrape annotations ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/envs/dev-full/starter-regional/super-vip-tidb-pool/tidb-server-pod-template.yaml#L7-L68)). Phase 3 must make both possible runtime lanes discoverable. |
| TiKV p0/p1/p2 | all three heterogeneous CRs, 1 each | HTTPS `/metrics`, `20180`, each CR's peer DNS | `tikv` | v1's official default status port is 20180 and it adds the annotation ([source](https://github.com/pingcap/tidb-operator/blob/83884b9589e6dae2ebefafde6b1e69199b1c2c84/pkg/manager/member/tikv_member_manager.go#L497-L501)). Override all three targets to the one canonical `tidb_cluster`. |
| node-exporter | one DaemonSet target per Ready Linux node, `N` | upstream secure service endpoint | `node-exporter` | Upstream sets `instance` from the pod's node name ([source](https://github.com/prometheus-operator/kube-prometheus/blob/a44e949d79c6a608fe2261edcff9adaef7c00923/manifests/nodeExporter-serviceMonitor.yaml#L12-L31)) and tolerates all taints ([source](https://github.com/prometheus-operator/kube-prometheus/blob/a44e949d79c6a608fe2261edcff9adaef7c00923/manifests/nodeExporter-daemonset.yaml#L100-L118)). |
| kube-state-metrics | one ready deployment target unless CP9 deliberately chooses HA | metrics endpoint | `kube-state-metrics` | Use cluster-wide read-only list/watch RBAC ([source](https://github.com/prometheus-operator/kube-prometheus/blob/a44e949d79c6a608fe2261edcff9adaef7c00923/manifests/kubeStateMetrics-clusterRole.yaml#L11-L47)). Add `cluster=<k8s cluster>` at scrape time. |
| kubelet | one per Ready node, `N` | HTTPS `/metrics` | `kubelet` | The upstream ServiceMonitor uses bearer auth, HTTPS, and preserves `metrics_path` ([source](https://github.com/prometheus-operator/kube-prometheus/blob/a44e949d79c6a608fe2261edcff9adaef7c00923/manifests/kubernetesControlPlane-serviceMonitorKubelet.yaml#L11-L56)). |
| cAdvisor | one per Ready node, `N` | HTTPS `/metrics/cadvisor` | `kubelet` | Keep the same job and distinguish it with `metrics_path=/metrics/cadvisor`, following upstream ([source](https://github.com/prometheus-operator/kube-prometheus/blob/a44e949d79c6a608fe2261edcff9adaef7c00923/manifests/kubernetesControlPlane-serviceMonitorKubelet.yaml#L57-L86)). |
| TiProxy/gateway | expected 0 | none | none | `tidb-gateway-inner` exposes only MySQL 3306 and probe 8081 with health/readiness paths ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-gateway-inner/workload.yaml#L15-L34)). Do not scrape the health endpoint as TiProxy. |

Current upstream TiDB Operator v2 uses `/metrics` on PD 2379, TSO/RM 3379, TiDB 10080, TiKV 20180, and TiProxy 3080 ([PD](https://github.com/pingcap/tidb-operator/blob/7bcbb0ef23ba72c9aae0bffd67389915ffbfdb32/api/core/v1alpha1/pd_types.go#L30-L38), [TSO/RM](https://github.com/pingcap/tidb-operator/blob/7bcbb0ef23ba72c9aae0bffd67389915ffbfdb32/api/core/v1alpha1/tso_types.go#L23-L29), [TiDB/TiKV](https://github.com/pingcap/tidb-operator/blob/7bcbb0ef23ba72c9aae0bffd67389915ffbfdb32/api/core/v1alpha1/tidb_types.go#L32-L40), [TiProxy](https://github.com/pingcap/tidb-operator/blob/7bcbb0ef23ba72c9aae0bffd67389915ffbfdb32/api/core/v1alpha1/tiproxy_types.go#L23-L33)). CP9 must nevertheless use its rendered workload endpoints: its custom TSO/RM both listen on 2379. Legacy v1 TiProxy used `3080 /api/metrics` ([source](https://github.com/pingcap/tidb-operator/blob/83884b9589e6dae2ebefafde6b1e69199b1c2c84/pkg/manager/member/tiproxy_member_manager.go#L384-L396)), which is another reason not to infer a TiProxy contract from the gateway's probe port.

### Conditional completeness targets

These targets should be collected when their endpoint exists, but they must not gate the 11 core/partial dashboards:

| Target | Current state | Contract |
| --- | --- | --- |
| `tikv-worker` | 1 annotated HTTPS target on `19000 /metrics` | scrape as `job=component=tikv-worker` |
| `cop-worker` | 1 annotated HTTPS target on `19000 /metrics` | scrape as `job=component=cop-worker`; retain the source label `cop-worker-sts` separately for forensics |
| DM master | 1 operator-generated target; DM worker replicas are 0 | scrape the v1 operator annotation on `8261 /metrics`, but classify the two DM dashboards conditional/unsupported until a real DM workload and worker are accepted |
| TiDB Operator controller | current Deployment has no declared metrics port, Service, or scrape annotation | no inferred target; add a separate self-monitoring contract only if CP9 exposes and tests a real controller endpoint |

The CSE worker endpoints are in the current manifests ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/personas/app-delivery/base/tidb-serverless/serverless-cse-workers.yaml#L18-L113)), and TiDB Operator v1 officially annotates DM master on port 8261 ([source](https://github.com/pingcap/tidb-operator/blob/83884b9589e6dae2ebefafde6b1e69199b1c2c84/pkg/manager/member/dm_master_member_manager.go#L623-L628)).

## Exact sample-label contract

Every database-family scrape must produce:

| Label | Value |
| --- | --- |
| `k8s_cluster` | resolved Starter Kubernetes/EKS cluster name; static per session |
| `tidb_cluster` | `tidb-serverless-serverless-cluster`; static across PD, PDMs, root/pool TiDB, TiKV, CSE, and DM |
| `kubernetes_namespace` | actual target namespace |
| `namespace` | actual target namespace when the exporter does not already emit it |
| `pod` | Kubernetes pod name |
| `instance` | Kubernetes pod name, matching TiDB Operator behavior |
| `component` | canonical component from the target matrix |
| `job` | the same canonical component value as `component` |
| `cluster` | source workload/CR instance for audit; use `serverless-cluster` for custom PDM/pool targets without an operator instance label |

Both `job` and `component` are intentional. The pinned raw dashboards query `job`; upstream's initializer rewrites non-node/pod dashboards from `job` to `component` and widens the `tidb_cluster` matcher ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/init.sh#L121-L135)). Emitting both labels with the same canonical value makes either deterministic provisioning form work and prevents a mixed-mode failure.

For Kubernetes metrics:

- attach `cluster=<resolved Kubernetes cluster name>` to kube-state-metrics, kubelet/cAdvisor, and node-exporter samples for `pods.json`;
- also attach `k8s_cluster=<same value>` for consistent cross-surface diagnostics;
- use node name as node-exporter `instance`;
- retain native KSM/cAdvisor `namespace`, `pod`, `container`, and `node` labels;
- preserve `metrics_path` on kubelet/cAdvisor.

## TLS and target addressing

All TiDB-family endpoints are TLS-enabled in current Starter. Prometheus must mount a namespaced monitoring client secret containing `ca.crt`, `tls.crt`, and `tls.key`; do not use blanket `insecure_skip_verify` for database targets.

Current CP9 generates one shared certificate with SANs for the fixed PD, TSO, RM, root TiDB, and all three TiKV peer DNS names ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/internal/cmd/starter_cmd.go#L1991-L2034)). It copies the material into `tidb-serverless` and workload pool namespaces, but not into an observability namespace ([source](https://github.com/tidbcloud/controlplane9/blob/4e9a28f783b4b3a7ac9ad1da32568942d49e71ce/internal/cmd/starter_cmd.go#L2052-L2125)). Phase 4 must create the observability-namespace copy through the existing certificate owner, not generate an unrelated CA.

Addressing rules:

- use the fixed peer DNS names for root PD/TiDB and each TiKV;
- use the explicit headless-service pod DNS names for TSO/RM;
- discover TiDB pods dynamically in both product-owned serving lanes by pod identity and annotated port; if scraping Pod IPs, set TLS `server_name` to a SAN on the shared certificate, such as `serverless-cluster-tidb.tidb-serverless.svc`;
- keep the annotated metrics port before the address rewrite to avoid duplicate targets.

The kube-prometheus kubelet/node-exporter examples use service-account bearer auth and `insecureSkipVerify` around Kubernetes/node-serving certificates. CP9 may reuse that upstream choice or route through an authenticated API-server proxy, but that exception must not leak into database target TLS.

## Datasource and dashboard contract

The upstream provisioning files define `tidb-cluster` and `k8s-cluster` separately ([component datasource](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/datasources/tidb-cluster-datasource.yaml#L1-L9), [Kubernetes datasource](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/datasources/k8s-datasource.yaml#L1-L9)), and the initializer provisions both ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/init.sh#L161-L184)). `nodes.json` explicitly queries `k8s-cluster` and expects `job="node-exporter"` ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/nodes.json#L1899-L1912)); `pods.json` also uses `k8s-cluster` ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/pods.json#L942-L981)).

Therefore the zero-query-rewrite implementation is two datasource aliases to the same Prometheus. Rewriting every dashboard to one datasource is possible, but is a CP9 dashboard fork and is not needed.

Static audit of the 25 pinned files found:

- 22 component dashboards define both `k8s_cluster` and `tidb_cluster` variables;
- `tikv-cdc.json` defines `tidb_cluster` but not `k8s_cluster`;
- `nodes.json` and `pods.json` use `k8s-cluster` instead;
- 5 files have no top-level UID: `tici_meta.json`, `tici_reader.json`, `tici_worker.json`, `tidb_resource_control.json`, and `tiproxy_summary.json`;
- all 25 top-level titles are unique.

Consequently, the bundle gate must validate 25 parseable JSON files and 25 unique titles. It may validate uniqueness of UIDs that are present, but it cannot require all upstream UIDs to be non-empty unless CP9 explicitly injects and records five deterministic UIDs. Grafana readiness should enumerate the 25 expected titles, not assume 25 upstream UIDs.

### The Pods dashboard needs an explicit compatibility decision

`pods.json` references `$namespace` but does not declare a `namespace` variable ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/pods.json#L942-L995)). The upstream initializer replaces that token with exactly one namespace ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/init.sh#L174-L184)). Starter needs at least `tidb-serverless` and the runtime-selected pool namespace.

The recommended CP9 compatibility transform is to inject one multi-select Grafana `namespace` query variable whose allowed values come from `kube_pod_info{cluster="$cluster",namespace=~"<core>|tidb-pool|super-vip-tidb-pool"}` and rewrite the dashboard's exact `namespace="$namespace"` PromQL matchers to `namespace=~"$namespace"`. Grafana formats multi-value and All selections as regex values, so leaving equality matchers in place would not cover all namespaces. The transform must be deterministic, tested, and included in bundle provenance/hash. Without this transform, provision the upstream file but report the dashboard as single-namespace/partial; do not claim it covers the entire Starter cluster.

Current KSM creates `kube_pod_container_resource_requests` and `kube_pod_container_resource_limits` with a `resource` label ([source](https://github.com/kubernetes/kube-state-metrics/blob/v2.19.1/docs/metrics/workload/pod-metrics.md#L31-L33)) and represents endpoint readiness through `kube_endpoint_address{ready=...}` ([source](https://github.com/kubernetes/kube-state-metrics/blob/v2.19.1/docs/metrics/service/endpoint-metrics.md#L9-L10)). The pinned Pods dashboard still queries `kube_pod_container_resource_*_memory_bytes` and `kube_endpoint_address_available` ([memory](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/pods.json#L575-L600), [endpoint](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/pods.json#L308-L315)). Those panels are known-empty with current KSM unless a separately approved PromQL compatibility patch is added. This does not invalidate the rest of the dashboard.

`nodes.json` joins `kube_node_labels`, so KSM must enable at least the node and pod labels needed by the dashboards. KSM says label metrics are not exposed by default and are controlled by `--metric-labels-allowlist` ([source](https://github.com/kubernetes/kube-state-metrics/blob/v2.19.1/docs/developer/cli-arguments.md#L63-L67)). The simplest compatible setting is `--metric-labels-allowlist=nodes=[*],pods=[*]`, with the documented performance cost accepted for this bounded Starter environment.

## Dashboard support matrix

All 25 files may be provisioned, but status must distinguish supported, partial, and absent components.

| Classification | Dashboards | Expected behavior |
| --- | --- | --- |
| Core supported | `overview.json`, `performance_overview.json`, `pd.json`, `tidb.json`, `tidb_runtime.json`, `tidb_resource_control.json`, `tikv_details.json`, `tikv_summary.json`, `tikv_trouble_shooting.json`, `nodes.json` | representative variables and series must be non-empty; resource-control panels remain feature-dependent |
| Core partial | `pods.json` | namespace-variable transform required for both namespaces; three obsolete KSM query families remain known-empty unless separately patched |
| Unsupported: no component | `lightning.json`, `TiCDC-Monitor-Summary.json`, `ticdc.json`, `ticdc_new_arch.json`, `tiflash_summary.json`, `tiflash_proxy_summary.json`, `tiflash_proxy_details.json`, `tici_meta.json`, `tici_reader.json`, `tici_worker.json`, `tikv-cdc.json` | provisionable, but label `unsupported/not_present`; never count empty panels as a scrape failure |
| Unsupported: no TiProxy endpoint | `tiproxy_summary.json` | `tidb-gateway-inner` is not a substitute; additionally the upstream `tidb_cluster` variable contains `$kuberentes` instead of `$k8s_cluster` ([source](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/tiproxy_summary.json#L2697-L2737)) |
| Conditional/incomplete DM | `DM-Monitor-Standard.json`, `DM-Monitor-Professional.json` | current DM master can be scraped, but worker replicas are zero and no DM workload is in task #179 acceptance; report conditional/unsupported |

This is 10 core supported + 1 core partial + 14 unsupported/conditional = 25.

## Readiness and acceptance

Pod rollout success is necessary but insufficient. `cp9 starter status` should query Prometheus and distinguish discovery, scrape, label, and series failures.

### Required target-health checks

For the resolved labels, require:

- `count(up{job="pd"} == 1) == 1`;
- `count(up{job="tso"} == 1) == 2`;
- `count(up{job="resource-manager"} == 1) == 1`;
- `count(up{job="tikv"} == 1) == 3`;
- `count(up{job="tidb"} == 1) == 1 + <live product-owned TiDB pods across both possible lanes>`;
- one healthy `node-exporter`, kubelet `/metrics`, and cAdvisor `/metrics/cadvisor` target per resolved Ready Linux node;
- at least one healthy kube-state-metrics target.

Every database target counted above must have the exact canonical `k8s_cluster` and `tidb_cluster` values. Expected-zero Gateway/TiProxy is an `unsupported_component` status, not `target_missing`.

### Representative series/variable checks

Require non-empty results for:

```promql
pd_cluster_status{k8s_cluster="<k8s>",tidb_cluster="tidb-serverless-serverless-cluster"}
tidb_server_connections{k8s_cluster="<k8s>",tidb_cluster="tidb-serverless-serverless-cluster"}
tikv_engine_block_cache_size_bytes{k8s_cluster="<k8s>",tidb_cluster="tidb-serverless-serverless-cluster"}
node_boot_time_seconds{job="node-exporter"}
kube_pod_info{cluster="<k8s>"}
container_cpu_usage_seconds_total{job="kubelet",metrics_path="/metrics/cadvisor"}
```

These are the real variable anchors in the pinned PD, TiDB, TiKV, Nodes, and Pods dashboards ([PD](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/pd.json#L17358-L17395), [TiDB](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/tidb.json#L27955-L28027), [TiKV](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/tikv_summary.json#L4688-L4729), [Nodes](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/nodes.json#L1899-L1912), [Pods](https://github.com/pingcap/monitoring/blob/89115a1ae6e7cb6d9f522330bd174fab329d8fa8/monitor-snapshot/master/operator/dashboards/pods.json#L942-L981)).

Recommended named failures are `target_missing`, `target_down`, `label_missing`, `series_missing`, `datasource_missing`, `dashboard_missing`, `dashboard_partial`, and `unsupported_component`.

## Reuse versus CP9 rendering boundary

### Reuse and pin

- all 25 upstream dashboard JSON files at `pingcap/monitoring@89115a1...` and their per-file/archive provenance;
- upstream initializer transformations only as a deterministic build-time reference, never as a mutable runtime download;
- TiDB Operator's annotation, TLS, peer-DNS, and target-label vocabulary;
- kube-prometheus's node-exporter, KSM, kubelet, cAdvisor, RBAC, and ServiceMonitor semantics.

### CP9 must render or adapt

- observability namespace, Prometheus/Grafana workloads and Services, retention/storage, scheduling, RBAC, and digest-pinned/mirrored images;
- a monitoring-namespace TLS client secret owned by the existing Starter certificate flow;
- explicit scrape discovery for custom TSO/RM, heterogeneous TiKV, and the runtime-selected TiDB lane;
- annotated-port de-duplication for raw pod SD;
- the canonical static `k8s_cluster`/`tidb_cluster` labels and the full label table above;
- the `tidb-cluster` primary datasource and `k8s-cluster` alias pointing to the same Prometheus;
- the deterministic Pods namespace-variable plus regex-matcher compatibility transform, or an honest single-namespace/partial status;
- target/series/dashboard readiness and explicit unsupported-component reporting;
- all-role tolerations for node-exporter and topology-derived expected target counts.

If CP9 deploys Prometheus Operator, it can reuse the official `PodMonitor`/`ServiceMonitor` objects directly after the CP9-specific relabel additions. If it deploys raw Prometheus, CP9 owns a faithful translation of those objects into `scrape_configs`, including Kubernetes RBAC, TLS, target de-duplication, and every readiness invariant above.

## Plan changes required by this research

1. Replace the proposed single-datasource normalization with two datasource names backed by one Prometheus; give primary `tidb-cluster` the upstream hard-coded UID.
2. Change “25 unique UIDs” to “25 unique titles, unique non-empty upstream UIDs,” or explicitly authorize deterministic UID injection for the five missing values.
3. Add a Pods namespace compatibility decision; one upstream substitution cannot cover both the core and selected-pool namespaces.
4. Put `k8s_cluster` on samples at scrape time rather than relying only on `external_labels`.
5. Add explicit TSO, Resource Manager, heterogeneous TiKV, and both product-owned TiDB lane jobs; annotation-only generic pod discovery is not enough.
6. Make both possible runtime lane templates emit stable discovery metadata; the reviewed super-vip template did not.
7. Record Gateway/TiProxy, TiFlash, TiCDC, TiCI, Lightning, TiKV-CDC, and incomplete DM dashboards as unsupported/conditional rather than failing full-stack readiness.
8. Keep the known-obsolete Pods KSM panels visible as `dashboard_partial` unless a separate PromQL compatibility patch is approved.

With these changes, Phase 4 can implement and test a complete, production-shaped Starter monitoring surface without inventing a false gateway endpoint or forking the upstream component queries.