# Grafana Metrics Datasource Route Convergence

## Current Design

Clinic Grafana currently has two metrics datasource entrypoints with different routing contracts.

`Prometheus` is configured in `clinic-cd/src/grafana/provisioning/datasources/index.ts` and points to `/metric/query`. Its backend route is registered in `pkg/api/proxy.go` and handled by `pkg/proxy/metric_proxy.go`. The current input is mostly `clinic-tidbcluster` cookie, which is populated by `/grafana/*` from `var-tidb_cluster` in the dashboard URL or referer. For Starter and Essential clusters, this path treats the cluster as `shared/serverless`, calls `GetServerlessTidbCluster`, and proxies to the selected child O11y cluster `InternalReadUri`.

`o11y-global-data` is configured in the same provisioning file and points to `/metric/o11y/query`. Its backend route is handled by `pkg/proxy/o11y_proxy.go`. The current input is `o11y.cluster_info` or `control_plane_info`, discovered from cookie, header, referer, or PromQL labels. The `control_plane_info` value is derived from dashboard variables; cluster detail already exposes a realtime serverless `shootName`, and dashboards/use sites can use that as the control plane identity. The handler then translates `control_plane_info` to `o11y_cluster_info` through global O11y cluster cache/listing and proxies by O11y cluster ID. This is the parent/global metrics path, but its input contract is not the TiDB Cloud cluster ID, and the translation key includes O11y `projectID`, which can become stale when project ownership changes.

The O11y client also has two cluster APIs in use: `GET /api/v1/clusters` via `ListCluster`, and `GET /api/v1/global/clusters` via `ListGlobalClusters`.

`clusters.parent_id` is not a valid source for this work. The parent-child relationship can change, and for Starter/Essential it must be resolved at request time. In the current Grafana path, `control_plane_info` already carries the parent identity in the form `projectID/vendor-region/applicationID`, for example `us-east-1-f03/aws-us-east-1/devtier-infra`. The implementation should parse that value and call O11y `GET /api/v1/clusters` instead of reading or caching the relationship from Clinic DB.

## Risks

- Cookie and referer are page-state based. They can be stale or wrong when users switch dashboards, use multiple tabs, or when Grafana sends async datasource requests after variables change.
- `Prometheus` and `o11y-global-data` currently resolve different identities: TiDB Cloud cluster ID vs O11y global cluster info. This makes route behavior hard to reason about and hard to test.
- Replacing `o11y-global-data` routing in one step can break existing dashboards that still send only `control_plane_info` or `o11y_cluster_info`.
- Starter/Essential child and parent metrics are intentionally different. A unified code path must not accidentally make both datasources resolve to the same O11y cluster.
- Essential V2 is not the same as Starter/Essential. Existing `deploy_type_v2=essential v2` belongs to the nextgen class and must not be routed through the Starter/Essential child/parent split unless product requirements change.
- Resolving parent identity from Clinic DB would be stale by design because the mapping may change independently of Clinic sync.
- If a future path needs to derive parent identity from only a TiDB Cloud child `cluster_id`, that lookup should be isolated behind a mgmt-facing placeholder. The current Grafana migration does not need to block on it because `control_plane_info` already carries parent identity.

## Target Contract

Grafana metrics routing should use one internal route resolution API:

```text
ResolveMetricsTarget(cluster_id, scope)
```

`cluster_id` is the TiDB Cloud cluster ID.

`scope=child` is used by the `Prometheus` datasource. For Starter/Essential, it resolves the O11y child cluster by calling `GET /api/v1/clusters` with the serverless/devtier request for the TiDB Cloud cluster ID, then proxies to the child cluster `InternalReadUri`.

`scope=parent` is used by the `o11y-global-data` datasource. For Starter/Essential, it parses `control_plane_info` into `projectID`, `vendor`, `region`, and `applicationID`. It then resolves the parent O11y cluster via `GET /api/v1/clusters`, not `/api/v1/global/clusters`, and proxies to the parent/global metrics target.

Both Grafana metrics datasources should use the same public backend route:

```text
/metric/query?metrics_scope=child  -> child metrics target
/metric/query?metrics_scope=parent -> parent metrics target
```

Their backend should share the same resolver and proxy target builder. `metrics_scope` defaults to `child` only for backward compatibility with the existing `Prometheus` datasource. `parent` must be explicit and must not fall back to child metrics if parent lookup fails.

For Grafana Prometheus provisioning, keep the datasource `url` as `/metric/query` and set `jsonData.customQueryParameters: "metrics_scope=parent"` for `o11y-global-data`. This makes Grafana append the internal scope parameter to Prometheus API requests without putting a query string directly in the datasource base URL.

## Canonical Inputs

The canonical input is the TiDB Cloud cluster ID carried by each datasource request.

Preferred extraction order:

1. Explicit request parameter or header, for example `cluster_id` / `x-clinic-cluster-id`, if available.
2. PromQL matcher in the current Prometheus request: `tidb_cluster_id`, `tidb_cluster`, or `cluster_id`.
3. Legacy `clinic-tidbcluster` cookie.
4. Legacy `var-tidb_cluster` from referer or request query.

Cookie and referer remain compatibility fallback only. Fallback use should be logged with enough labels to identify dashboards that still need query updates.

`control_plane_info` is the parent identity input for this slice. It should be parsed and sent to O11y `GET /api/v1/clusters`. `o11y_cluster_info`, `o11y.cluster_info`, and `shootName` are legacy parent-routing inputs and should not be used by the new `o11y-global-data` canonical route. The migration target is to stop using the old `control_plane_info -> o11y_cluster_info` cache/table lookup entirely for parent metrics and rely on O11y-provided APIs instead.

## Implementation Steps

1. Add a small request extractor in `pkg/proxy` that reads the current Prometheus request and returns a TiDB Cloud cluster ID.
   - Reuse the existing PromQL request parsing shape from `GetO11yPromQL`.
   - Extend matcher extraction to support `tidb_cluster_id`, `tidb_cluster`, and `cluster_id`.
   - Keep cookie/referer fallback outside the parser so the canonical and legacy paths are visible.

2. Add `metrics_scope` parsing to the existing `/metric/query` route.
   - Supported values are `child` and `parent`.
   - Missing value defaults to `child` for existing `Prometheus` behavior.
   - Invalid values should return a clear client error.

3. Introduce a shared metrics target model in `pkg/o11y/urls` or a nearby package.
   - It should represent target scheme, host, path mode, base path, O11y cluster ID, and whether v2 data-service path is required.
   - It should support internal-read targets, data-service ID targets, multitenant targets, and historical targets.
   - `ClusterDataURLsService` should continue to call child URL resolution only; it should not become control-plane/parent aware.

4. Add a unified resolver API.
   - `ResolveMetricsTarget(ctx, cluster, scope, controlPlaneInfo)` where `scope` is `child` or `parent`.
   - Implement it as a Go resolver under `pkg/o11y/urls`, not as an HTTP call to `/internal/clusters/{cluster_id}/cluster-dataurls`.
   - `child` should preserve current `Prometheus` behavior for dedicated, premium/nextgen, BYOC, and Starter/Essential.
   - `parent` should replace the `o11y-global-data` `o11y.cluster_info` cache-based route with request-time O11y API resolution. The TiDB Cloud `cluster_id` remains the permission and dashboard context key; `control_plane_info` carries the parent identity for this slice.
   - `parent` must not read `clusters.parent_id`.
   - `parent` should parse `control_plane_info` as `projectID/vendor-region/applicationID` and call O11y `GET /api/v1/clusters` with the parsed parent identity.
   - `parent` should not call `GetO11yClusterInfoFromCache`, `InitClusterInfoCache`, or `ListGlobalClusterInfo`.

5. Change `Prometheus` route to call the unified resolver with `scope=child`.
   - Keep existing permission checks.
   - Share child metrics target resolution with `ClusterURLsResolver.Resolve()` so Grafana and cluster-dataurls do not carry separate O11y URL lookup logic.
   - Keep existing Starter/Essential multiple O11y child cluster disambiguation until a better O11Y-side contract exists.
   - Preserve existing historical behavior for supported deploy types.

6. Change `o11y-global-data` provisioning to use the unified route.
   - Update its datasource URL from `/metric/o11y/query` to `/metric/query`.
   - Set `jsonData.customQueryParameters` to `metrics_scope=parent`.
   - Keep datasource name `o11y-global-data`.
   - Ensure its `keepCookies` includes the same cluster context as `Prometheus` during the transition, especially `clinic-tidbcluster`, until explicit query/header cluster ID is available everywhere.

7. Change parent-scope handling to call the unified resolver.
   - Add cluster read permission based on the TiDB Cloud cluster ID.
   - Remove `o11y.cluster_info` lookup from the parent metrics path. Keep `control_plane_info` only as parsed parent identity input, not as a cache lookup key.
   - Remove the parent/global canonical dependency on `GET /api/v1/global/clusters`.

8. Unify proxy dispatch.
   - Reuse one proxy builder for both datasources after target resolution.
   - Avoid keeping separate host/path mutation logic in `metric_proxy.go` and `o11y_proxy.go`.

9. Update Grafana/dashboard usage.
   - Ensure `o11y-global-data` keeps enough request context to send or infer TiDB Cloud cluster ID.
   - For dashboards that do not include any cluster label in PromQL, add an explicit matcher or query parameter where possible.
   - Leave datasource names unchanged for the first slice to reduce dashboard migration risk.

## Verification Checklist

- Unit test request extraction from GET `query`, GET `match[]`, POST form body, cookie fallback, and referer fallback.
- Unit test `/metric/query` scope parsing: missing means `child`, `child` calls child resolver, `parent` calls parent resolver, invalid scope errors.
- Unit test `Prometheus` Starter/Essential `scope=child` resolves child `InternalReadUri`.
- Unit test `o11y-global-data` Starter/Essential `scope=parent` resolves parent target through parsed `control_plane_info` and O11y `GET /api/v1/clusters`, and does not use `ListGlobalClusters`, `GetO11yClusterInfoFromCache`, or `clusters.parent_id`.
- Unit test `child` and `parent` for the same Starter/Essential cluster return different O11y targets.
- Unit test Essential V2 still follows nextgen behavior, not Starter/Essential child/parent routing.
- Unit test permission denial for `o11y-global-data` when the user cannot read the input cluster ID.
- Manual check legacy dashboards have been migrated away from `control_plane_info` before removing `/metric/o11y/query`.
- Run focused package tests for `pkg/proxy`, `pkg/o11y`, and `pkg/api` routes touched by this work.

## Non-Goals

- Do not delete the `Prometheus` datasource in this change.
- Do not rename public datasource names in Grafana provisioning in the first slice.
- Do not make `o11y-global-data` fallback to child metrics when parent resolution fails.
- Do not keep `/api/v1/global/clusters` as the canonical route for Grafana metrics.
- Do not read or persist Starter/Essential parent mapping through Clinic DB.
- Do not keep the old `control_plane_info -> o11y_cluster_info` cache/table lookup as a fallback for the new parent metrics route.
- Do not change data-proxy metrics behavior in this slice, except for sharing helper code if it is already behavior-preserving.