# Local E2E Toolkit

`pipeline/tools/local-e2e/` contains the local-only closed-loop verification utilities for TiDB X metrics integration.

Commands below assume you run them from the repository root.

## Goals

- Keep all local verification assets under one isolated directory.
- Keep generated binaries, configs, and results out of the main source tree.
- Avoid any impact on online services or production code paths.

## Metric Scope

Prometheus payload expectations are service-plan specific and follow the current code:

- `Dedicated`: compare Dedicated outputs against the baseline branch and inspect dedicated-only metric families. Do not expect NextGen-only `tidbcloud_db_total_connection` or storage metrics.
- `Premium` / `Essential` / `Starter`: expect the NextGen DB/storage metric family such as `tidbcloud_db_total_connection` and `tidbcloud_db_row_storage_used_bytes`.

## Layout

```text
pipeline/tools/local-e2e/
  README.md
  bench/
  configs/
    otel/
    vector/
  docs/
    tidb-x-testplan.md
  env/
    test.env.template
  generated/
  scripts/
    run-all.sh
    gen-configs.sh
    run-prometheus-test.sh
    run-otel-test.sh
    diff-results.sh
    test-common.sh
```

## Safety

- This toolkit is opt-in only.
- It does not change any production runtime code path.
- All generated files are written under `pipeline/tools/local-e2e/generated/`.
- Sensitive local environment values live in `pipeline/tools/local-e2e/env/test.env`, which is gitignored.
- The load-generation scripts in `bench/` are separate from the default verification flow and are not run by `run-all.sh`.

## Quick Start

1. Copy the environment template:

```bash
cp pipeline/tools/local-e2e/env/test.env.template pipeline/tools/local-e2e/env/test.env
```

2. Fill in the required cluster IDs, tenant/project info, and `AUTH0_CLIENT_SECRET`.

3. Start mirrord port-forward for local testing:

```bash
cd global
MIRRORD_CHECK_VERSION=false mirrord port-forward \
  --config-file .mirrord/port-forward.yaml \
  -L 29080:www.gs.us-west-2.aws.observability.tidbcloud.com:80 \
  -L 29484:www.ds.us-west-2-staging.aws.observability.tidbcloud.com:443 \
  -L 29482:www.ds.us-east-1-serverless.aws.observability.tidbcloud.com:443 \
  -L 29483:www.ds.us-east-1-premium.aws.observability.tidbcloud.com:443
```

Set `LOCAL_TEST_GLOBAL_SERVER_URL=http://127.0.0.1:29080` and use per-plan
`LOCAL_TEST_REGIONAL_SERVER_URL_*` values such as
`https://127.0.0.1:29484` / `https://127.0.0.1:29482` / `https://127.0.0.1:29483`.

4. Run the full local verification flow:

```bash
bash pipeline/tools/local-e2e/scripts/run-all.sh
```

5. Compare dedicated `main` vs feature outputs:

```bash
bash pipeline/tools/local-e2e/scripts/diff-results.sh
```

6. Optional: chaos replay with mocked upstream + real Datadog sink

This flow mocks o11y global/regional query endpoints locally, while keeping downstream as real Datadog via Vector `datadog_metrics` sink.

Required env additions in `pipeline/tools/local-e2e/env/test.env` (do not commit secrets):

```bash
DD_API_KEY="<datadog-test-api-key>"
DD_SITE="datadoghq.com"
DD_NAMESPACE="chaos.test"
```

Run:

```bash
bash pipeline/tools/local-e2e/scripts/run-dd-chaos-replay.sh
```

Optional tuning:

```bash
REPLAY_PHASES="empty:45,error:45,data:90" \
REPLAY_WAIT_SECS=210 \
RUN_ID="chaos_manual_001" \
DD_NAMESPACE="chaos.test" \
bash pipeline/tools/local-e2e/scripts/run-dd-chaos-replay.sh
```

Artifacts are written under:

- `pipeline/tools/local-e2e/generated/results/<TEST_ROUND_NAME>/chaos-dd/<RUN_ID>/`
  - `summary.txt`
  - `pipeline.log`
  - `mock-o11y.log`
  - `vector.log`

Datadog verification hints:

- Query metrics under namespace `chaos.test`.
- Filter by tag `chaos_run:<RUN_ID>`.
- Expect empty/error window then recovery in the same run.

## Generated Files

The toolkit writes all generated artifacts to:

- `pipeline/tools/local-e2e/generated/bin/`
- `pipeline/tools/local-e2e/generated/configs/`
- `pipeline/tools/local-e2e/generated/results/`

This keeps temporary outputs scoped under the toolkit directory instead of scattering them across other pipeline source directories or the repository root.

## Optional Load Tools

The `bench/` directory contains optional TPC-C preparation and load scripts for manual investigation.
These are not part of the default closed-loop verification path.

## Detailed Plan

See [docs/tidb-x-testplan.md](docs/tidb-x-testplan.md).