# TiDB X Metrics Local E2E Plan

## Overview

This document describes the local closed-loop verification toolkit under `pipeline/tools/local-e2e/`.

Commands below assume you run them from the repository root.

The toolkit is used to:

- compare `main` and feature-branch metric outputs on Dedicated clusters
- verify Premium / Essential / Starter support on the feature branch
- collect Prometheus and OTel-path outputs into local result files

Prometheus expectations are plan-aware:

- `Dedicated`: compare baseline vs feature and inspect dedicated-only metrics. Do not use NextGen DB/storage metrics as pass criteria.
- `Premium` / `Essential` / `Starter`: verify the NextGen DB/storage metric family on the feature branch.

## Directory Layout

```text
pipeline/tools/local-e2e/
  README.md
  bench/
  configs/
    otel/
      otel-config-dd.yaml
      otel-config-nr.yaml
    vector/
      vector-dd.yaml
      vector-nr.yaml
  docs/
    tidb-x-testplan.md
  env/
    test.env.template
  generated/
    bin/
    configs/
    results/
  scripts/
    test-common.sh
    gen-configs.sh
    run-prometheus-test.sh
    run-otel-test.sh
    run-all.sh
    diff-results.sh
```

## Safety Boundaries

- The toolkit is local-only and opt-in.
- It does not change any production runtime path.
- It does not modify online deployment manifests or service startup behavior.
- Generated binaries, configs, and results are written only under `pipeline/tools/local-e2e/generated/`.
- Secrets are stored only in `pipeline/tools/local-e2e/env/test.env`, which is gitignored.
- The optional `bench/` load scripts are not executed by the default verification flow.

## Prerequisites

- Go
- Docker
- `git`
- `curl`
- `bash`
- optional: `jq`
- optional: `nc`

Pull Vector once in advance:

```bash
docker pull timberio/vector:0.47.0-debian
```

## Environment Setup

Create the local environment file:

```bash
cp pipeline/tools/local-e2e/env/test.env.template pipeline/tools/local-e2e/env/test.env
```

Then fill in:

- cluster IDs
- tenant and project IDs
- cloud / region
- `AUTH0_CLIENT_SECRET`
- optional local debug URLs

## Main Workflow

Run the full local verification workflow:

```bash
bash pipeline/tools/local-e2e/scripts/run-all.sh
```

This flow will:

1. Ensure a `main` worktree exists.
2. Build feature and `main` binaries into `pipeline/tools/local-e2e/generated/bin/`.
3. Generate per-cluster configs into `pipeline/tools/local-e2e/generated/configs/`.
4. Run Prometheus verification.
5. Run OTel verification.
6. Save results under `pipeline/tools/local-e2e/generated/results/<TEST_ROUND_NAME>/`.

## Result Comparison

Compare Dedicated `main` vs feature outputs:

```bash
bash pipeline/tools/local-e2e/scripts/diff-results.sh
```

The diff script summarizes:

- Prometheus metric-name differences
- OTel Datadog-path differences
- OTel NewRelic-path differences
- feature-only cluster output summaries for Premium / Essential / Starter

## Generated Output

All runtime artifacts are stored under:

```text
pipeline/tools/local-e2e/generated/
  bin/
  configs/
  results/
```

This prevents the local toolkit from polluting:

- repository root
- unrelated `pipeline/` source directories
- top-level `test-configs/`
- top-level `test-results/`

## Optional Load Tools

The `bench/` directory contains optional TPC-C preparation and load helpers:

- `prepare.sh`
- `run.sh`
- `tiflash.sh`

These are separate from the default closed-loop verification process and should only be used intentionally.

## Config Inputs

Static tool-owned configs live under:

- `pipeline/tools/local-e2e/configs/otel/`
- `pipeline/tools/local-e2e/configs/vector/`

Generated per-run configs are written to:

- `pipeline/tools/local-e2e/generated/configs/`

## Notes on Online Safety

To avoid impacting online services:

- the toolkit does not alter online deployment code
- no production command path imports anything from `pipeline/tools/local-e2e/`
- all execution must be triggered manually by engineers
- local verification state stays in tool-local generated directories

If bench traffic is used, it should be treated as an explicit manual load action rather than part of the default verification flow.