# Dev Prometheus E2E Toolkit

`pipeline/tools/dev-prom-e2e/` starts a minimal local `Prometheus + Grafana`
stack for one remote dev scrape target.

This toolkit does not start local exporters and does not run automated checks.
It only loads the provided dev `scrape_config`, then lets you inspect the
results in Prometheus or Grafana.

The default scrape target is:

- job: `dev-integration-premium-keep-24h`
- target: `https://us-west-2.aws.dev.prom.integration.tidbcloud.com/public_api/v1/clusters/10925905225882017226/metrics?format=Prometheus`

It reuses the checked-in dashboards under `pipeline/tools/prom-e2e/dashboard/`.

## Quick Start

Start the stack:

```bash
bash pipeline/tools/dev-prom-e2e/scripts/up.sh
```

Stop the stack:

```bash
bash pipeline/tools/dev-prom-e2e/scripts/down.sh
```

## Public Grafana Demo via Tailscale Funnel

If you want to temporarily expose Grafana on the public internet for a demo,
use the userspace `tailscaled` wrapper. It runs against an isolated socket and
state directory under `pipeline/tools/dev-prom-e2e/generated/tailscale/`, so it
does not modify a system-level Tailscale daemon.

Required:

- `tailscaled` and `tailscale` installed locally
- a Tailscale auth key on first login
- a non-default Grafana admin password

Example:

```bash
cd pipeline
DEV_PROM_E2E_GRAFANA_ADMIN_PASSWORD='replace-me-now' \
DEV_PROM_E2E_TAILSCALE_AUTHKEY='tskey-...' \
make dev-prom-e2e-public-up
```

Check status:

```bash
cd pipeline
make dev-prom-e2e-public-status
```

Stop the public endpoint and the local stack:

```bash
cd pipeline
make dev-prom-e2e-public-down
```

Optional environment variables:

- `DEV_PROM_E2E_TAILSCALE_AUTHKEY_FILE`: read the auth key from a file instead
  of an inline env var
- `DEV_PROM_E2E_TAILSCALE_HOSTNAME`: override the temporary Funnel node name
- `DEV_PROM_E2E_TAILSCALE_FUNNEL_PORT`: public HTTPS port, one of `443`,
  `8443`, or `10000`; default `443`
- `DEV_PROM_E2E_GRAFANA_ADMIN_USER`: override the Grafana admin user; default
  `admin`
- `DEV_PROM_E2E_GRAFANA_ADMIN_PASSWORD`: required for public exposure; local
  default remains `admin`

## Endpoints

- Prometheus: `http://127.0.0.1:19190`
- Grafana: `http://127.0.0.1:13100`
  - default user/password: `admin` / `admin`
  - when using the Funnel target, set a temporary non-default password first

## Layout

```text
pipeline/tools/dev-prom-e2e/
  docker-compose.yaml
  grafana/
    provisioning/
  scripts/
  generated/
```

Generated runtime files are written under
`pipeline/tools/dev-prom-e2e/generated/`.