# metrics-integration

`metrics-integration` is the control-plane and runtime codebase for TiDB Cloud metrics integrations.
It provisions and operates exporter pipelines for TiDB clusters, including TiDB-X service plans, and contains the control-plane services, per-region executor, and collector runtime used to deliver metrics to downstream systems.

The repository is split into multiple Go modules. There is no repository-root `go.mod`, so development commands should be run from the relevant module directory.

## What is in this repository

- `global/`: global control plane, internal API server, workflow workers, and management CLI
- `regional/`: per-region execution service used by the global control plane for region-scoped operations
- `pipeline/`: metrics collector runtime and Prometheus-facing binary
- `helm-charts/pipeline/`: Helm chart used to deploy pipeline workloads
- `helm-charts/prometheus/`: Helm chart for Prometheus-related deployment assets
- `docs/`: design docs, test plans, release notes, operations guides, and reference material
- `pipeline/tools/local-e2e/`: local validation helpers and scripts

## Architecture at a glance

### Global

The global service is the main control plane. It:

- loads configuration through Viper
- initializes dependency injection, database access, encryptor, and dataplane kube configs
- runs the internal gRPC server and grpc-gateway
- starts background FastFlow workers for pipeline lifecycle workflows
- exposes an internal Cobra-based CLI for operational jobs

Main entrypoints:

- `global/cmd/metrics-integration/main.go`
- `global/internal/domain/cmd/root.go`
- `global/internal/domain/container/container.go`

### Regional

The regional service mirrors the server-side wiring for region-local execution. It is used when the global control plane delegates operations to a regional executor instead of talking to the dataplane directly.

### Pipeline

The pipeline module contains the metrics collector runtime and Prometheus-facing service. It renders and consumes pipeline configuration, discovers cluster metadata, queries observability data sources, and exports metrics to downstream systems.

The pipeline process supports runtime config reload for config-file based updates, which is important for Helm- or ConfigMap-driven deployments.

## Supported workflows

This repository is used for:

- provisioning and updating metrics integrations
- running background workflows that manage pipeline lifecycle actions
- regional execution for dataplane-facing tasks
- collecting and exporting metrics for TiDB Dedicated and TiDB-X service plans
- local validation of Prometheus, Datadog, and New Relic export paths

## Prerequisites

Recommended local tooling:

- Go 1.24.x
- `make`
- `docker`
- `helm`
- `kubectl`
- `buf` for proto generation
- `mirrord` for cluster-connected local development
- `jq`, `curl`, and `nc` for local validation flows

## Repository layout

```text
.
├── docs/
├── global/
├── helm-charts/
│   ├── pipeline/
│   └── prometheus/
├── pipeline/
│   └── tools/
│       └── local-e2e/
├── regional/
└── ...
```

## Getting started

### 1. Choose the module you want to work on

Most work falls into one of these paths:

- `global/` for API, workflows, and CLI changes
- `regional/` for region-local service changes
- `pipeline/` for collector, Prometheus, exporter, and runtime config changes

### 2. Use local config samples as shape references

Configuration is module-local. Useful starting points:

- `global/deployment/config/local/config.yaml`
- `regional/deployment/config/local/config.yaml`
- `pipeline/example/local-run-config.yaml`
- `pipeline/example/metrics-config.yaml`
- `pipeline/example/otel-config.yaml`
- `pipeline/example/prometheus-config.yaml`

Do not commit real credentials or environment-specific secrets. Treat checked-in local configs as development references only, and create sanitized local copies when needed.

### 3. Run the services locally

#### Global service

```bash
cd global
make run-server
```

Starts the global API stack, including the internal gRPC server and grpc-gateway.

#### Global workflow workers

```bash
cd global
make run-workflow
```

Starts the FastFlow background workers used for pipeline lifecycle workflows.

#### Regional service

```bash
cd regional
make run
```

#### Pipeline collector

```bash
cd pipeline
make run
```

This runs the collector with the checked-in example local configs under `pipeline/example/`.

#### Pipeline Prometheus binary

```bash
cd pipeline
make prometheus-run
```

### 4. Build binaries

```bash
cd global && make build
cd regional && make build
cd pipeline && make build-pipeline
cd pipeline && make build-prometheus
```

### 5. Generate protobuf stubs

Each module owns its own API definitions:

```bash
cd global && make proto
cd regional && make proto
```

## Global management CLI

The global module ships an internal management CLI built with Cobra.

Command groups include:

- `server`
- `workflow`
- `job workflow`
- `job integration`

Typical local form:

```bash
cd global
go run ./cmd/metrics-integration/main.go -c ./deployment/config/local/config.yaml <command>
```

Examples:

```bash
cd global
go run ./cmd/metrics-integration/main.go -c ./deployment/config/local/config.yaml server
go run ./cmd/metrics-integration/main.go -c ./deployment/config/local/config.yaml workflow
go run ./cmd/metrics-integration/main.go -c ./deployment/config/local/config.yaml job integration list --cluster-id <CLUSTER_ID>
```

See `docs/operations/2026-03-20-cli-usage-guide.md` for the confirmed command tree, required flags, and operational cautions.

## Helm and deployment assets

- `helm-charts/pipeline/` contains the pipeline Helm chart used by the control plane
- `helm-charts/prometheus/` contains Prometheus deployment-related Helm assets
- `global/deployment/` and `regional/deployment/` contain local config, docker, kustomize, terraform, and workflow assets

Global and regional Docker images can be built with module-local `make docker-build` targets. The pipeline module provides `make docker-build-pipeline` and `make docker-build-prometheus`.

## Testing and validation

Automated test coverage is limited, so module-local `go test ./...` is a good baseline check when making code changes:

```bash
cd global && go test ./...
cd regional && go test ./...
cd pipeline && go test ./...
```

For higher-confidence validation, use the documentation under `docs/test/`, especially:

- `docs/test/2026-03-15-local-validate-workspace-guide.md`
- `docs/test/2026-03-15-local-validate-real-observability-guide.md`
- `docs/test/2026-02-23-tidb-x-acceptance-test-plan.md`

## Documentation map

- `docs/README.md`: top-level index for repository documentation
- `docs/design/`: architecture notes, implementation plans, and design proposals
- `docs/operations/`: CLI and operational workflows
- `docs/reference/`: metrics and export reference material
- `docs/release/`: rollout, risk, and release checklists
- `docs/test/`: acceptance plans, validation guides, and test artifacts

Start with `docs/README.md` if you need a documentation index.

## Common development notes

- There is no monorepo root build target; use module-local `Makefile`s.
- The global service performs DB auto-migrations for core integration tables on startup.
- Workflow logic is defined in `global/internal/domain/workflow/` and wired through DI in `global/internal/domain/container/container.go`.
- Helm rendering and pipeline deployment logic live under `global/pkg/helm/` and workflow actions under `global/internal/domain/workflow/action/`.
- The pipeline module is the main place to work on metric definitions, label transforms, datasource routing, and exporter behavior.

## Related docs worth reading first

- `docs/design/2026-02-23-tidb-x-metrics-integration-design.md`
- `docs/design/2026-03-20-core-svc-to-tidb-mgmt-migration-design.md`
- `docs/design/2026-03-20-pipeline-backfill-batching-design.md`
- `docs/design/2026-03-20-pipeline-topology-hot-update-design-en.md`
- `docs/operations/2026-03-20-operations-index.md`