# Essential V2 Datadog and Prometheus Integration Support

Status: Draft

Date: 2026-07-14

Owners: Metrics Integration

## Summary

Add `Essential_V2` as a first-class service plan for Datadog and Prometheus integrations.

Although the name is similar to the existing `Essential` plan, `Essential_V2` uses the
Premium/NextGen observability model. It must therefore use the Premium O11y route and
Premium metric matchers instead of the legacy shared/serverless Essential path.

The implementation should preserve `Essential_V2` as the service-plan value throughout
Global and Pipeline. It should reuse Premium behavior through centralized capability
classification rather than rewriting the plan to `Premium` or duplicating Premium cases
across metric builders.

## Goals

- Allow Global to create Datadog and Prometheus integrations for clusters whose management
  service plan is `Essential_V2`.
- Route `Essential_V2` queries to the NextGen O11y datasource.
- Use Premium-compatible PromQL matchers and metric coverage.
- Preserve current Dedicated, Premium, Essential, and Starter behavior.
- Cover new integrations and pipeline recreation/update paths with unit and staging tests.

## Non-goals

- Changing credential validation or lifecycle semantics for Datadog integrations.
- Changing the existing Essential or Starter shared/keyspace routing.
- Adding new metrics that are not already exposed for Premium.
- Changing customer-facing metric names or output label schemas.
- Formally enabling New Relic in this change. The shared OTel path must not regress, but
  New Relic product support should be handled separately unless explicitly requested.

## Evidence and Assumptions

Clinic `origin/master` at commit `c629cced61b2349e7daf74bb5fc4a46ac795d34c`
(`update metrics (#1567)`, 2026-07-13) provides the current reference behavior:

- management service `Essential_V2` maps to `deploy_type=premium` and
  `deploy_type_v2=essential v2`;
- it queries O11y Global using the NextGen bizType and omits `projectID`;
- it uses the returned `internalReadUri` as the Prometheus API base URL;
- it injects `tidb_cluster_id=<cluster ID>` into PromQL;
- it does not perform the old Essential `cluster_id -> keyspace_id` lookup;
- it classifies Essential V2 with Premium/Premium Reserved for metrics query behavior.

Based on that implementation, this design treats Essential V2 metrics as Premium-compatible.
Before production rollout, a real Essential V2 cluster must confirm that O11y registration,
CDC metrics, RU labels, and metric availability match this assumption.

## Current State and Gaps

### Management API dependency

`global/go.mod` currently pins management API version
`v0.0.0-20250724063032-e025f4e9d8cf`. Its `ServicePlan` enum contains Starter,
Essential, Premium, and BYOC, but not `Essential_V2`.

Global consequently cannot recognize the new enum without upgrading this dependency.

### Global service-plan handling

`global/internal/infrastructure/grpc/grpcli/mgmt_svc.go` extracts only the existing
service plans. `global/pkg/helm/pipeline/types.go` also rejects any non-dedicated service
plan other than Premium, Essential, and Starter.

This means pipeline config generation currently fails before an Essential V2 integration
can be deployed.

### Pipeline service-plan classification

`pipeline/central/entity/constant.go` recognizes only Dedicated, Premium, Essential, and
Starter. An `Essential_V2` target currently fails configuration normalization.

Several runtime decisions depend on these classifications:

- NextGen datasource routing;
- project-less O11y lookup;
- metric registry and CDC inclusion;
- `tidb_cluster_id`, `cluster_id`, `keyspace_id`, and RU matcher selection;
- instance-level output labels;
- resource-group handling for RU totals.

Adding the plan only to config validation is therefore insufficient.

### O11y bizType configuration

Global and Prometheus deployment configs have plan-to-bizType maps for `premium`,
`essential`, and `starter`, but no `essential_v2` key. Unknown plans fall back to the
Dedicated entry, which would silently route Essential V2 to the wrong datasource.

## Proposed Behavior

| Capability | Essential | Premium | Essential V2 |
| --- | --- | --- | --- |
| O11y family | devtier/shared | NextGen | NextGen |
| Prod bizType | `prod-devtier` | `prod-tidbcloud-nextgen` | `prod-tidbcloud-nextgen` |
| O11y `projectID` | omitted | omitted | omitted |
| DB cluster matcher | `cluster_id` | `tidb_cluster_id` | `tidb_cluster_id` |
| Shared-series isolation | `keyspace_id` | `tidb_cluster_id` | `tidb_cluster_id` |
| RU isolation | keyspace/resource group | Premium matcher | Premium matcher |
| Serverless behavior | yes | no | no |
| Output hierarchy | instance/tenant | instance/tenant | instance/tenant |
| CDC metrics | no | yes | yes, subject to staging verification |

The canonical config value is `Essential_V2`. Matching is case-insensitive after trimming,
and the `bizTypeConfigs` lookup key is `essential_v2`.

## Design

### 1. Upgrade and consume the management API enum

Upgrade `github.com/tidbcloud/tidb-management-service/api` to a version containing
`commonv1.ServicePlan_Essential_V2`.

Update Global service-plan extraction so a NextGen cluster with that enum produces exactly:

```yaml
servicePlan: Essential_V2
```

Update Global validation to accept `Essential_V2` for the appropriate non-dedicated deploy
type. Do not accept it for a Dedicated cluster.

Clinic currently pins management API version
`v0.0.0-20260509113450-50cd135e1533`, which contains the required enum and is the initial
upgrade candidate. Confirm it is compatible with Global's other management API consumers
before pinning it; using the same verified API line reduces enum and cluster-shape drift.

### 2. Add a first-class Pipeline plan and capabilities

Add `ServicePlanEssentialV2 = "Essential_V2"` and accept it in
`NormalizeServicePlan`.

The capability methods should return:

| Method | Result |
| --- | --- |
| `IsNextGen()` | `true` |
| `HasProjectConcept()` | `false` |
| `IsInstanceLevel()` | `true` |
| `IsServerless()` | `false` |
| `SupportsCDC()` | `true` |

Introduce a centralized Premium-metrics classification, for example:

```go
func (p ServicePlan) UsesPremiumMetrics() bool {
    switch normalizedPlan(p) {
    case ServicePlanPremium, ServicePlanEssentialV2:
        return true
    default:
        return false
    }
}
```

The final implementation may use a different name, but metric builders and matcher helpers
should depend on one capability instead of repeating string switches. Existing Premium checks
should be migrated only where necessary for Essential V2 support, keeping the patch scoped.

### 3. Route Essential V2 through NextGen O11y

Add the following plan mapping in Global, Pipeline deployment configs, and local/example
configs:

| Environment | Key | bizType |
| --- | --- | --- |
| Dev | `essential_v2` | `dev-tidbcloud-nextgen` |
| Staging | `essential_v2` | `staging-tidbcloud-nextgen` |
| Prod | `essential_v2` | `prod-tidbcloud-nextgen` |

The O11y Global lookup should be equivalent to:

```text
GET /api/v1/clusters/
  ?vendor=<provider>
  &region=<region>
  &tenantID=<tenant-or-organization-id>
  &applicationID=<cluster-id>
  &bizType=<env>-tidbcloud-nextgen
```

`projectID` must not be included. The pipeline continues to use the returned
`internalReadUri` as the Prometheus-compatible query endpoint.

The current Global target config uses management `OrganizationId` as `tenantId`. Before
rollout, compare this value with a real Essential V2 O11y registration. Clinic falls back to
the cluster tenant ID when organization ID is absent for a Premium-class cluster. If real
Essential V2 metadata can have a zero organization ID, Metrics Integration needs the same
fallback rather than sending `tenantID=0`.

### 4. Use Premium metric query semantics

All metric matcher helpers must classify Essential V2 with Premium:

- exact cluster matcher: `tidb_cluster_id="<cluster-id>"`;
- aggregate DB matcher: `tidb_cluster_id="<cluster-id>"`;
- shared TiKV/TiFlash series matcher: `tidb_cluster_id="<cluster-id>"`;
- RU matcher and aggregation: same labels and grouping as Premium;
- TiDB cluster regex cases: same behavior as Premium where a regex remains required.

Essential V2 must never enter the old Essential/Starter keyspace discovery path. In
particular, it must not query `tidb_server_connections` by `cluster_id` to discover a
`keyspace_id`.

The Prometheus registry should expose the same set of metrics as Premium, including CDC,
DB performance, RU total, row storage, and columnar storage metrics. Datadog uses the same
query builders through the OTel pipeline and therefore receives the same source metrics and
matchers.

### 5. Preserve output compatibility

Essential V2 remains an instance-level plan. Its exported resource labels should match the
Premium shape:

- keep instance and tenant identity;
- rename cluster-facing labels to the established instance-facing names;
- remove Dedicated-only project and node/component labels according to existing transforms;
- remove the internal `service_plan` attribute before export where the current pipeline does
  so for other plans.

No metric names should change as part of this feature.

### 6. Configuration failure behavior

Missing `essential_v2` bizType configuration must fail validation or be detected by tests.
It must not silently fall back to Dedicated routing for a supported `Essential_V2` plan.

The existing fallback for genuinely old configs and empty service plans should remain
unchanged for backward compatibility.

## Expected Code Changes

### Global

- `global/go.mod` and `global/go.sum`: upgrade the management API dependency.
- `global/internal/infrastructure/grpc/grpcli/mgmt_svc.go`: extract `Essential_V2`.
- `global/pkg/helm/pipeline/types.go`: accept and render the plan.
- `global/pkg/helm/pipeline/types_test.go`: cover extraction, validation, and YAML output.
- Global dev/staging/prod/local config maps: add `essential_v2` bizType mappings.

### Pipeline

- `pipeline/central/entity/constant.go`: add the plan and Premium-metrics capability.
- `pipeline/central/entity/service_plan_test.go`: cover all capability results.
- `pipeline/central/config/config.go`: document and validate the new bizType key.
- `pipeline/internal/datasource/metric_data_addr.go`: recognize Essential V2 as NextGen and
  omit `projectID`.
- `pipeline/internal/datasource/central_thanos/central_thanos.go`: apply Premium matchers,
  RU behavior, and avoid keyspace lookup.
- `pipeline/internal/service/prometheus/registry.go`: register Premium-equivalent metrics.
- OTel metadata, transform, and metric-model tests: verify Premium-equivalent output labels.
- Pipeline example and Prometheus app-delivery configs: add environment mappings.

### Regional

No protocol or persistence change is currently expected. Regional should be covered by a
pipeline deployment smoke test because it transports and deploys the generated config.

## Test Plan

### Unit tests

Global:

- management `Essential_V2` becomes `servicePlan: Essential_V2`;
- NextGen validation accepts it;
- Dedicated validation rejects it;
- rendered target config retains empty/zero project fields without attempting a project lookup.

Pipeline entity/config:

- normalization accepts exact, lowercase, mixed-case, and whitespace-padded values;
- capability results match the table above;
- `GetBizTypeByServicePlan("Essential_V2")` selects the NextGen mapping;
- a supported plan with a missing mapping does not silently select Dedicated.

Datasource and metrics:

- O11y query contains NextGen bizType and excludes `projectID`;
- DB, shared-series, RU, and CDC builders use Premium-equivalent matchers;
- no keyspace lookup runs for Essential V2;
- registry metric names match Premium exactly;
- Prometheus and OTel/Datadog output labels match Premium exactly.

Run module-local tests:

```bash
cd global && go test ./...
cd pipeline && go test ./...
cd regional && go test ./...
```

### Local E2E

Add or extend a case under `pipeline/tools/local-e2e/` with a mocked O11y Global response:

1. target cluster has `servicePlan: Essential_V2`;
2. O11y lookup expects the NextGen bizType and no project ID;
3. response contains an `internalReadUri`;
4. query endpoint asserts `tidb_cluster_id` and rejects `cluster_id`/`keyspace_id` isolation;
5. Prometheus scrape and Datadog OTel export both produce non-empty metrics.

### Staging E2E

Using a real Essential V2 cluster:

1. verify management metadata and the tenant/organization identifier;
2. create one Prometheus integration and one Datadog integration;
3. verify O11y lookup returns exactly one active cluster;
4. compare the registered metric set with a Premium cluster;
5. verify DB, RU, storage, and CDC metrics are non-empty;
6. verify exported labels contain the customer cluster ID and no cross-cluster series;
7. update/recreate both pipelines and repeat the checks;
8. delete both integrations and verify cleanup.

## Rollout Plan

1. Confirm the management API version and Essential V2 metadata shape.
2. Land dependency, classification, routing, matcher, config, and test changes together.
3. Build and publish Pipeline images containing Essential V2 normalization.
4. Deploy environment bizType configuration and Global support in dev, then staging.
5. Complete the real-cluster staging E2E and compare results with Premium.
6. Roll out to prod and initially limit creation to known Essential V2 clusters if a feature
   gate is available.
7. Monitor O11y lookup failures, empty query results, pipeline restarts, and exporter errors.

Global must not start creating Essential V2 pipelines that reference an older Pipeline image
which rejects the new service-plan value.

## Rollback

- Stop new Essential V2 integration creation at Global.
- Roll Global configuration and image back to the previous release.
- Delete or recreate only Essential V2 pipelines created by the new release if their Pipeline
  image cannot be retained safely.
- Existing Dedicated, Premium, Essential, and Starter integrations require no migration and
  should continue using their prior images and config.

Do not rewrite persisted Essential V2 records to Premium during rollback; that would lose the
actual product plan and make later reconciliation ambiguous.

## Risks and Mitigations

| Risk | Impact | Mitigation |
| --- | --- | --- |
| Management API version lacks or changes the enum | Global build or extraction failure | Pin a verified version and add enum-level tests |
| Essential V2 O11y tenant ID differs from organization ID | No datasource found | Validate a real cluster and add Clinic-compatible fallback if needed |
| Missing bizType map silently falls back to Dedicated | Queries wrong O11y backend | Validate supported-plan mappings and test every environment config |
| A matcher remains hard-coded to Premium only | Partial/empty metrics | Centralize `UsesPremiumMetrics()` and compare registry/query output |
| Essential V2 CDC differs from Premium | Empty CDC metrics or noisy retries | Make staging CDC verification a release gate; disable only that capability if disproved |
| Old Pipeline image receives the new plan | Startup/config failure | Enforce rollout ordering and test generated image/config pairing |
| Shared OTel changes alter New Relic behavior | Unintended regression | Run existing OTel tests and avoid declaring New Relic support in this scope |

## Open Questions

1. Is Clinic's management API version
   `v0.0.0-20260509113450-50cd135e1533` approved and compatible with Global's other API use?
2. Does every Essential V2 cluster have a non-zero management `OrganizationId`, or is the
   Clinic tenant-ID fallback required?
3. Are CDC metrics officially part of Essential V2, or only structurally available in O11y?
4. Is Essential V2 RU labeling exactly the Premium resource-group model in every region?
5. Are existing Essential V2 integrations or failed pipeline records present and in need of
   backfill/recreation after rollout?

## Estimate

Assuming the management API and O11y registration are ready:

| Work | Estimate |
| --- | --- |
| Dependency and Global metadata/config support | 1-2 engineer-days |
| Pipeline classification, routing, and matchers | 2-3 engineer-days |
| Unit and local E2E coverage | 1-2 engineer-days |
| Staging verification and fixes | 1 engineer-day |
| Total | 5-8 engineer-days |

Cross-team dependency or O11y metadata differences can extend the elapsed time to roughly two
weeks without materially increasing the core implementation size.