# TiDB X Local-to-Real Observability Validation Guide

## Goal

This guide extends the local validation loop into a real-platform acceptance flow:

- first validate local `Prometheus`, `OTel -> Datadog`, and `OTel -> New Relic` outputs
- then create real integrations against a live TiDB Cloud cluster
- finally verify data directly in:
  - Grafana
  - Datadog
  - New Relic

The focus here is not `main vs feature` diffing. The focus is end-to-end visibility on the real external platforms.

## Relationship to the local workspace guide

Recommended sequence:

1. Complete local validation with `2026-03-15-local-validate-workspace-guide.en.md`
2. Confirm `test-results/` has expected outputs
3. Create real integrations
4. Verify metrics, labels, and dashboards on external platforms

## Recommended minimum scope

Do not start with all cluster types at once.

Recommended first milestone:

- choose one real cluster
- choose one target platform
- complete one full end-to-end validation path

Suggested platform priorities:

- Prometheus + Grafana
- Datadog
- New Relic

After one cluster is verified, expand to:

- Dedicated
- Premium
- Essential
- Starter

## Prerequisites

In addition to the local validation prerequisites, you need:

- a real TiDB Cloud cluster id
- TiDB Cloud API credentials:
  - `TIDB_CLOUD_PUBLIC_KEY`
  - `TIDB_CLOUD_PRIVATE_KEY`
- a valid Datadog API key
- a valid New Relic ingest key or API key
- an available Grafana instance

Useful references:

- `docs/spec/2026-02-22-tidb-x-support/04-metric-output-and-coverage.md`

## Step 1: Complete local output validation first

Before creating real integrations, confirm these local artifacts exist:

- `prometheus/*.txt`
- `otel-dd/*.ndjson`
- `otel-nr/*.ndjson`

Typical commands:

```bash
cd validate-workspace/metrics-integration
cp test-scripts/test.env.template test-scripts/test.env
bash test-scripts/run-all.sh
```

Recommended checks:

- Prometheus output is non-empty and includes `tidbcloud_` metrics
- Datadog OTel output is non-empty and includes `tidb_cloud.` metrics
- New Relic OTel output is non-empty and includes `tidb_cloud.` metrics

## Step 2: Prepare environment variables for OpenAPI calls

```bash
export TIDB_CLOUD_PUBLIC_KEY="<your-public-key>"
export TIDB_CLOUD_PRIVATE_KEY="<your-private-key>"

export CLUSTER_ID="<your-cluster-id>"

export DATADOG_API_KEY="<your-datadog-api-key>"
export DATADOG_SITE="US1"

export NEWRELIC_API_KEY="<your-newrelic-api-key>"
export NEWRELIC_SITE="US_OTLP"
```

## Step 3: Create real integrations

Recommended flow:

1. Create a Prometheus integration.
2. Verify endpoint visibility through Prometheus or Grafana.
3. Create a Datadog integration and verify metrics arrival.
4. Create a New Relic integration and verify metrics arrival.

## Step 4: Validate platform-side visibility

### Prometheus / Grafana

Confirm:

- scrape endpoint is reachable
- expected `tidbcloud_` metrics are queryable
- dashboards show actual time series

### Datadog

Confirm:

- metrics appear in Metrics Explorer
- expected tags are present
- dashboard widgets can render the data

### New Relic

Confirm:

- metrics are visible through Query Builder or NRQL
- expected attributes are preserved
- dashboards can render the metric series

## Minimum acceptance criteria

For a single cluster, the validation is considered successful when:

- Prometheus path shows valid metrics
- Datadog path shows valid metrics
- New Relic path shows valid metrics
- core metric semantics match across the three paths

## Recommended core metrics to compare

- `db_query_per_second`
- `db_total_connection`
- `db_active_connections`
- `node_cpu_seconds_total`
- `db_database_time`

## Final Goal

The final closed loop is:

- local outputs are correct
- integration creation succeeds
- real external platforms display the expected data

That is the confidence threshold needed before broader rollout.