# Prometheus E2E Toolkit

`pipeline/tools/prom-e2e/` provides a local closed-loop stack for TiDB-X Prometheus validation:

- start four local `cmd/prometheus` exporters for `Dedicated`, `Premium`, `Essential`, and `Starter`
- scrape them with a local Prometheus container
- provision Grafana with the checked-in dashboards
- verify plan-aware representative metrics through the local Prometheus API

The verification logic follows the current code behavior:

- `Dedicated`: verify scrape health only. Dedicated does not expose the NextGen `tidbcloud_db_*` / storage metric family in `pipeline/internal/service/prometheus/registry.go`.
- `Premium` / `Essential` / `Starter`: verify `tidbcloud_db_total_connection` and `tidbcloud_db_row_storage_used_bytes`.

## Quick Start

The toolkit reuses the local routing and cluster settings from
`pipeline/tools/local-e2e/env/test.env` by default.
It runs the exporters, Prometheus, and Grafana with Docker host networking so
they can reach the local mirrord tunnels bound on `127.0.0.1`.

Start the stack:

```bash
bash pipeline/tools/prom-e2e/scripts/up.sh
```

Verify metrics and dashboard provisioning:

```bash
bash pipeline/tools/prom-e2e/scripts/verify.sh
```

Stop everything:

```bash
bash pipeline/tools/prom-e2e/scripts/down.sh
```

## Endpoints

- Prometheus: `http://127.0.0.1:19090`
- Grafana: `http://127.0.0.1:13000`
  - default user/password: `admin` / `admin`

## Layout

```text
pipeline/tools/prom-e2e/
  dashboard/
  docker-compose.yaml
  grafana/
    provisioning/
  scripts/
  generated/
```

Generated runtime files are written under `pipeline/tools/prom-e2e/generated/`.