# Grafana Loki Datasource O11y Route Convergence

## Current Design

Clinic Grafana log links currently choose a physical Loki datasource before the request reaches the backend.

`clinic-cd/src/grafana/provisioning/datasources/index.ts` provisions many region/provider datasources such as `Loki-aws-us-east-1`, `Loki-alicloud-ap-southeast-1`, and premium/BYOC variants. Each datasource points directly to a physical O11y data-service Loki URL like:

```text
https://www.ds.<region>.<provider>.observability.tidbcloud.com/loki/self-monitoring
```

`clinic-ui/packages/clinic-web/src/utils/links.ts` constructs Grafana URLs and injects `var-datasource` with `Loki-{provider}-{region}`. Starter/Essential has extra dev/staging hardcoding:

```text
Loki-{provider}-{region}-{suffix}
```

It also builds Starter/Essential `control_plane_info` in UI as:

```text
<region>-f01/<provider>-<region>/devtier-infra
```

and builds LogQL around control-plane labels such as `cluster`, `namespace`, and `container`. These LogQL selectors usually do not contain the TiDB Cloud cluster ID, so backend routing cannot rely only on parsing LogQL.

There are two UI log entry paths for Starter/Essential:

1. The primary path opens Grafana Explore directly from `CloudInstanceLogsModal`, using `getServerlessComponentLogPath*()` and the `REACT_APP_LOKI_INSTANCE_LOGS_URL_TEMPLATE` template. It bypasses the serverless logs dashboard.
2. The legacy path opens `/grafana/d/3_lrvOoIz/tidb-serverless-cluster-logs-clinic` only for the `TiDB(legacy)` link and passes `var-datasource`, `var-cluster_id`, and `var-control_plane_info`.

Therefore, migrating only the dashboard datasource variable is not sufficient. The Explore URL generation must also switch to the virtual datasource.

Backend O11y URL resolution already has a central place for cluster URLs: `pkg/o11y/urls.ClusterURLsResolver`. For metrics, it now has `ResolveMetricsTarget(ctx, cluster, scope, controlPlaneInfo)`. For logs, `Resolve()` still derives `ClusterURLs.LokiURL` from `ClusterMetricsURL` by taking the metrics host and appending `/loki/self-monitoring`.

That derivation encodes the old assumption that one provider/region maps to one Loki. It is no longer valid when one region can have multiple Loki instances.

The O11y document defines the new source of truth: `GET /api/v1/clusters` returns the cluster routing object, and that object should include `logQueryUri`. Clinic should route Loki requests to this O11y-provided `logQueryUri` instead of deriving Loki from physical region or metrics host.

## Risks

- Region/provider datasource names are not unique enough once a region has multiple Loki instances.
- UI-generated datasource names can become stale when O11y moves cluster-to-Loki placement.
- Starter/Essential log queries use control-plane labels; the TiDB Cloud cluster ID is dashboard context, not necessarily part of LogQL.
- Cookie/referer cluster context can be stale when users switch dashboards or use multiple tabs. It can be a compatibility path, but it should not become the intended source of truth.
- Changing all Loki paths at once can break Dedicated, Premium, BYOC, data-proxy cluster logs, and system logs. The first slice should only migrate Starter/Essential Grafana log entrypoints.
- `ClusterURLs.LokiURL` is currently consumed by data-proxy. Replacing it must preserve non-serverless behavior or be scoped behind a new resolver.
- Grafana Loki datasource behavior is not identical to Prometheus. The backend route must support the path shape Grafana sends, for example `/log/query/loki/api/v1/query_range`.
- `control_plane_info` is not generated consistently today. Starter/Essential metrics uses `shootName/<provider>-<region>/devtier-infra`, while the legacy serverless logs path currently synthesizes `<region>-f01/<provider>-<region>/devtier-infra`.
- Local `clinic-grafana` sources do not currently contain the serverless logs dashboard JSON identified by UID/title. The first implementation must verify the live/generated dashboard source before assuming dashboard-only changes are enough.

## Target Contract

Clinic should introduce a virtual Loki datasource for TiDB Cloud cluster logs:

```yaml
- name: o11y-loki-data
  uid: o11y-loki-data
  type: loki
  access: proxy
  url: ${clinicAddr}/log/query
  jsonData:
    keepCookies:
      - clinic.session_id
      - clinic.current_org
      - clinic.current_orgType
      - clinic-tidbcluster
      - clinic.control_plane_info
```

The datasource should not encode provider, region, physical Loki name, or data-service host.

Backend should expose a Loki-compatible proxy route:

```text
GET/POST /log/query/loki/api/v1/query
GET/POST /log/query/loki/api/v1/query_range
GET      /log/query/loki/api/v1/labels
GET      /log/query/loki/api/v1/label/:name/values
GET      /log/query/loki/api/v1/series
```

The canonical routing input is the TiDB Cloud cluster ID. Preferred extraction order:

1. Explicit request parameter or header, for example `cluster_id` or `x-clinic-cluster-id`.
2. Dashboard variable query parameters, for example `var-cluster_id` / `var-tidb_cluster`, if Grafana preserves them.
3. `clinic-tidbcluster` cookie populated by the existing Grafana proxy route.
4. Referer fallback from the Grafana dashboard URL.

For Starter/Essential in the first slice, backend resolves logs through the parent/control-plane O11y cluster:

```go
ResolveLogsTarget(ctx, cluster, LogsScopeParent, controlPlaneInfo)
```

The resolver parses `controlPlaneInfo` as:

```text
projectID/vendor-region/applicationID
```

Then it calls O11y `GET /api/v1/clusters` through the existing O11y client path and uses `ClusterInfo.LogQueryUri` as the Loki proxy target.

For future non-serverless migration, the same resolver can support:

```go
ResolveLogsTarget(ctx, cluster, LogsScopeCluster, "")
```

but Dedicated, Premium, BYOC, data-proxy cluster logs, and system logs should remain unchanged in the first PR.

If Grafana Loki datasource cannot reliably append custom query parameters for route scope, use route aliases instead of relying on datasource query parameters:

```text
/log/query/parent/loki/api/v1/query_range  -> LogsScopeParent
/log/query/cluster/loki/api/v1/query_range -> LogsScopeCluster
```

Both aliases should call the same internal resolver and proxy implementation.

## Source Of Truth

- O11y `GET /api/v1/clusters` is the source of truth for the target Loki URL.
- `ClusterInfo.LogQueryUri` is the source of truth for Loki routing.
- TiDB Cloud cluster ID is the permission and dashboard-context key.
- `control_plane_info` is only the Starter/Essential parent identity input. It is not a datasource name, cache key, or physical Loki selector.
- Clinic DB must not persist or infer parent-to-Loki mapping.
- UI must not construct physical `Loki-{provider}-{region}` for Starter/Essential after migration.

## Implementation Steps

1. Add O11y response schema support.
   - Add `LogQueryUri string json:"logQueryUri"` to `pkg/o11y.ClusterInfo`.
   - Keep `InternalReadUri` for metrics.
   - Add tests for decoding `logQueryUri`.

2. Add a logs target resolver in `pkg/o11y/urls`.
   - Define `LogsScope` with at least `parent` and `cluster`.
   - Implement `ResolveLogsTarget(ctx, cluster, scope, controlPlaneInfo)`.
   - For `parent`, reuse the same control-plane parsing and `GetServerlessControlPlaneCluster` contract as metrics parent scope.
   - Return `logQueryUri`; do not call `lokiURLFromMetricsURL`.
   - Error clearly when O11y returns no cluster or a cluster with empty `logQueryUri`.

3. Keep `ClusterURLsResolver.Resolve()` behavior stable for existing data-proxy consumers.
   - Do not remove `ClusterURLs.LokiURL` in the first PR.
   - Do not switch data-proxy `/data-proxy/loki/{cluster_id}` to the new route in the same PR.
   - Optionally add a TODO or separate plan item to replace `lokiURLFromMetricsURL` once data-proxy migration is ready.
   - Do not change `/data-proxy/system-logs`; its canonical routing key is datasource, not TiDB Cloud cluster ID.

4. Add a Clinic backend Loki proxy handler.
   - Register `/log/query` and optional scoped aliases `/log/query/:logs_scope`.
   - Accept Grafana Loki API paths and forward them to `LogQueryUri + /loki/api/v1/...`.
   - Reuse existing request/response proxy helpers where possible, but keep Loki path joining separate from Prometheus path joining.
   - Preserve GET and POST request bodies and query parameters.
   - Strip only Clinic routing parameters such as `cluster_id`, `logs_scope`, and dashboard variables before forwarding.
   - Add O11y request headers consistently with metrics proxy where the target cluster requires them.

5. Add cluster context extraction for Loki requests.
   - Reuse the existing Grafana cookie/referer conventions from `pkg/api/proxy.go`.
   - Prefer explicit `cluster_id` / `var-cluster_id` over cookies.
   - Log fallback source, but do not expose internal O11y IDs in user-facing errors.

6. Add RBAC/permission checks.
   - Resolve TiDB Cloud cluster by input cluster ID.
   - Check the user can read the cluster before calling O11y.
   - Parent log routing must not bypass cluster permission just because `control_plane_info` exists.

7. Add `o11y-loki-data` in `clinic-cd`.
   - Provision it as a Loki datasource pointing at Clinic backend.
   - Keep old physical Loki datasources in place for Dedicated, Premium, BYOC, ad hoc system logs, and rollback.

8. Update Starter/Essential UI links in `clinic-ui`.
   - `getServerlessInstanceLogPath` and serverless component log helpers should set `var-datasource=o11y-loki-data`.
   - Stop using `getServerlessLokiDataSource()` for Starter/Essential paths.
   - Update the direct Explore path from `CloudInstanceLogsModal`; it is the main current Starter/Essential log entrypoint.
   - Keep the legacy `TiDB(legacy)` dashboard path compatible, but do not treat it as the only migration target.
   - Stop hardcoding `region-f01` when `shootName` / control-plane info is available from cluster detail.
   - Preserve `var-cluster_id` and `var-control_plane_info` in Grafana URLs.
   - Check OOM detail links separately if Starter/Essential OOM logs are in scope; they currently also synthesize `Loki-{provider}-{region}`.

9. Update the serverless logs dashboard if needed.
   - First locate the real source of `tidb-serverless-cluster-logs-clinic`; it is not present in the current local `clinic-grafana` checkout.
   - Ensure its datasource variable can select `o11y-loki-data`.
   - Ensure queries do not depend on physical datasource names.
   - Keep LogQL selectors unchanged unless the O11y log label contract changes.

## Verification Checklist

- Unit test `ClusterInfo` decodes `logQueryUri`.
- Unit test `ResolveLogsTarget(parent)` for Starter/Essential calls O11y `GET /api/v1/clusters` and returns `LogQueryUri`.
- Unit test `ResolveLogsTarget(parent)` fails when `control_plane_info` is missing or invalid.
- Unit test no-cluster and empty-`logQueryUri` errors produce clear user-facing errors without O11y internal IDs.
- Unit test Loki proxy path joining:
  - `/log/query/loki/api/v1/query_range`
  - `/log/query/loki/api/v1/labels`
  - `/log/query/loki/api/v1/label/container/values`
  - `/log/query/loki/api/v1/series`
- Unit test GET and POST preserve Loki request semantics.
- Unit test route parameters used for Clinic routing are not forwarded to Loki.
- Unit test permission denial happens before O11y proxying.
- Unit test Starter/Essential UI helpers output `var-datasource=o11y-loki-data`.
- Manual dev test with a Starter/Essential dashboard confirms Grafana datasource request goes through Clinic backend and reaches the O11y `logQueryUri`.
- Manual rollback test confirms old physical Loki datasources still exist.

## Non-Goals

- Do not remove physical `Loki-*` datasources in the first PR.
- Do not change Dedicated, Premium, BYOC, system logs, or data-proxy Loki behavior in the first PR.
- Do not parse physical Loki target from region/provider in new Starter/Essential code.
- Do not infer Loki URL from metrics URL in the new logs resolver.
- Do not persist parent-child or cluster-Loki mapping in Clinic DB.
- Do not rewrite LogQL unless O11y changes the log label contract.
- Do not expose O11y internal cluster IDs or URLs in frontend error responses.

## Subagent Work Split

Parallel read-only tasks:

1. UI/Grafana audit
   - Scope: `clinic-ui`, `clinic-cd`, `clinic-grafana`.
   - Output: exact files that generate or consume `var-datasource`, `var-cluster_id`, and `var-control_plane_info`; list which Starter/Essential links must switch to `o11y-loki-data`.
   - Completed finding: the main current Starter/Essential log entrypoint is the Explore URL path from `CloudInstanceLogsModal`, not only the legacy serverless logs dashboard.

2. Backend routing audit
   - Scope: `pkg/o11y/urls`, `pkg/o11y/proto.go`, `pkg/api/proxy.go`, `pkg/proxy`.
   - Output: exact insertion points for `ResolveLogsTarget`, route registration, Loki proxy handler, and tests.
   - Completed finding: `ClusterURLs.LokiURL` is still consumed by data-proxy, and `/data-proxy/system-logs` is a separate datasource-routed API.

Worker tasks:

1. Backend worker
   - Owns: `pkg/o11y`, `pkg/o11y/urls`, `pkg/proxy`, `pkg/api` tests.
   - Must not edit: `clinic-ui`, `clinic-cd`, `clinic-grafana`.
   - Deliverable: `logQueryUri` schema, `ResolveLogsTarget`, `/log/query` Loki proxy route, focused tests.

2. CD worker
   - Owns: `clinic-cd/src/grafana/provisioning/datasources/index.ts`.
   - Must not edit: backend or UI.
   - Deliverable: dev-safe `o11y-loki-data` datasource inserted near `#PLACEHOLDER_FOR_O11Y_LOKI`; old physical Loki datasources retained.

3. UI worker
   - Owns: `clinic-ui/packages/clinic-web/src/utils/links.ts`, `CloudInstanceLogsModal` callers, and related focused tests if present.
   - Must not edit: backend, `clinic-cd`.
   - Deliverable: Starter/Essential Explore and legacy log links use `o11y-loki-data`, preserve `cluster_id` and `control_plane_info`, and stop physical Loki datasource synthesis for those paths.

4. Dashboard discovery worker
   - Owns: read-only discovery in `clinic-grafana` and deployment/generated dashboard sources.
   - Deliverable: identify the real source for `tidb-serverless-cluster-logs-clinic` before any dashboard JSON edit.

Implementation PR split:

1. `clinic` backend PR: schema + resolver + Loki proxy + tests.
2. `clinic-cd` PR: provision `o11y-loki-data` for dev first, keep old datasources.
3. `clinic-ui` PR: switch Starter/Essential log links to the virtual datasource.
4. Optional `clinic-grafana` PR: update serverless log dashboard datasource variable if dashboard JSON requires it.