# TiDB NextGen Local Docker Compose

This directory starts a local non-TLS TiDB NextGen cluster with one PD, three TiKV stores, one default `tikv-worker`, one coprocessor worker, one `SYSTEM` TiDB, one `local_normal` TiDB, and MinIO-backed DFS/S3.

## Start

```bash
cd scripts/docker-compose
docker compose up -d
```

## Resource Limits

PD, all three TiKV stores, both TiKV worker containers, and both TiDB containers are limited to `4.0` CPUs and `8G` memory in `docker-compose.yml`.

If the cluster is already running, recreate it so the runtime applies the new limits. This preserves named volumes and works with Docker Compose and Podman Compose:

```bash
docker compose down
docker compose up -d
```

Docker Engine and cgroups-v2/rootful Podman apply these limits. Rootless Podman on cgroups-v1 accepts the compose file but logs that resource limits are ignored.

## Test A Local TiKV Build

From the repository root, build debug `tikv-server` and `tikv-worker` binaries and place them into a local image based on the release NextGen TiKV image:

```bash
make build-compose-tikv-image
```

By default this builds:

```text
tikv:compose-debug
```

Run the compose cluster with that image:

```bash
cd scripts/docker-compose
export TIKV_COMPOSE_IMAGE=tikv:compose-debug
docker compose down
docker compose up -d
```

The local image replaces `/tikv-server` and `/tikv-worker`. It inherits the rest of the filesystem from `hub-zot.pingcap.net/mirrors/tidbx/tikv/tikv/image:release-nextgen-202603`.

Useful overrides:

```bash
TIKV_COMPOSE_IMAGE=my-tikv:test make build-compose-tikv-image
TIKV_COMPOSE_BASE_IMAGE=hub-zot.pingcap.net/mirrors/tidbx/tikv/tikv/image:release-nextgen-202603 make build-compose-tikv-image
```

## Stop And Clean

```bash
cd scripts/docker-compose
docker compose down
docker compose down -v
```

The first command stops containers but keeps volumes. The second command removes the persistent PD, TiKV, worker, and MinIO volumes.

## Connect

System keyspace TiDB:

```bash
mysql -h 127.0.0.1 -P 4000 -u root
```

Normal keyspace TiDB:

```bash
mysql -h 127.0.0.1 -P 4001 -u root
```

Smoke SQL for both:

```sql
SELECT VERSION();
SHOW DATABASES;
CREATE DATABASE IF NOT EXISTS compose_smoke_test;
USE compose_smoke_test;
CREATE TABLE IF NOT EXISTS t (id BIGINT PRIMARY KEY, v VARCHAR(64));
INSERT INTO t VALUES (1, 'ok') ON DUPLICATE KEY UPDATE v = VALUES(v);
SELECT * FROM t;
```

## Validation

```bash
docker compose ps
docker compose logs -f pd
docker compose logs -f tikv-1
docker compose logs -f tikv-worker
docker compose logs -f copr-worker
docker compose logs -f tidb-system
docker compose logs -f tidb-normal
```

## Log Files

The compose startup commands create log directories before launching each daemon. Main component logs are written to:

```text
pd:           /var/log/pd/pd.log
tikv-1:       /var/log/tikv/tikv.log
tikv-2:       /var/log/tikv/tikv.log
tikv-3:       /var/log/tikv/tikv.log
tikv-worker:  /var/log/tikv-worker/tikv-worker.log
copr-worker:  /var/log/tikv-worker/copr-worker.log
tidb-system:  /var/log/tidb/tidb.log
tidb-normal:  /var/log/auditlog/tidb.log
```

TiDB slow logs remain separate:

```text
tidb-system:  /var/log/tidb/slowlog
tidb-normal:  /var/log/auditlog/slowlog
```

Tail them from the running containers:

```bash
docker compose exec pd sh -lc 'tail -n 100 /var/log/pd/pd.log'
docker compose exec tikv-1 sh -lc 'tail -n 100 /var/log/tikv/tikv.log'
docker compose exec tikv-worker sh -lc 'tail -n 100 /var/log/tikv-worker/tikv-worker.log'
docker compose exec copr-worker sh -lc 'tail -n 100 /var/log/tikv-worker/copr-worker.log'
docker compose exec tidb-system sh -lc 'tail -n 100 /var/log/tidb/tidb.log'
docker compose exec tidb-normal sh -lc 'tail -n 100 /var/log/auditlog/tidb.log'
```

PD checks:

```bash
docker compose exec pd /pd-ctl -u http://pd:2379 member
docker compose exec pd /pd-ctl -u http://pd:2379 store
docker compose exec pd /pd-ctl -u http://pd:2379 keyspace list
```

MinIO bucket checks:

```bash
docker compose exec minio sh -lc 'ls -1 /data | sort'
docker compose exec minio sh -lc 'curl -fsS http://127.0.0.1:9000/minio/health/live'
```

Expected buckets:

```text
tidbcloud-local-dfs
tidbcloud-local-global-sort
tidbcloud-local-metering
```

## Keyspace Bootstrap

`create-keyspace` runs after PD is healthy and before `tidb-normal` starts. `tidb-normal` also waits for `tidb-system` to become healthy, because the normal-keyspace TiDB startup checks that the `SYSTEM` keyspace has completed TiDB bootstrap. The keyspace bootstrap service uses the branch-verified PD CLI syntax:

```bash
/pd-ctl -u http://pd:2379 keyspace create local_normal --config gc_management_type=keyspace_level
```

The script is idempotent. It first waits for `SYSTEM`, skips creation when `local_normal` already exists, then verifies both keyspaces are visible and that `gc_management_type=keyspace_level` appears in the keyspace list output.

## DFS And S3 Variables

The cloud-storage-engine branch reads DFS settings from `components/kvengine/src/dfs/config.rs`. The compose file sets these for TiKV, `tikv-worker`, `copr-worker`, and TiDB containers:

```text
DFS_BACKEND=s3
DFS_PREFIX=cse
DFS_S3_BUCKET=tidbcloud-local-dfs
DFS_S3_ENDPOINT=http://minio:9000
DFS_S3_KEY_ID=minioadmin
DFS_S3_SECRET_KEY=minioadmin
DFS_S3_REGION=us-west-2
DFS_REMOTE_COMPACTOR_ADDR=http://tikv-worker:19000/compact
DFS_ALLOW_FALLBACK_LOCAL=false
```

`DFS_S3_ENDPOINT`, `DFS_S3_KEY_ID`, and `DFS_S3_SECRET_KEY` are required for local MinIO because there is no AWS IAM role. The endpoint includes a port, so the TiKV S3 DFS client uses path-style-compatible addressing for MinIO. TiDB also gets AWS-compatible variables and its bootstrap SQL sets:

```sql
SET GLOBAL tidb_cloud_storage_uri = 's3://tidbcloud-local-global-sort/global-sort?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3A%2F%2Fminio%3A9000&force-path-style=true&region=us-west-2';
```

TiDB source/tests on this branch accept `endpoint`, `access-key`, `secret-access-key`, `force-path-style`, and `region` URI parameters for S3-compatible storage.

MinIO is explicitly configured with `MINIO_REGION=us-west-2` and `MINIO_REGION_NAME=us-west-2`. Without that, MinIO reports the default bucket region as `us-east-1`, and TiDB rejects the `tidb_cloud_storage_uri` region check.

The compose file also clears inherited proxy variables and sets `NO_PROXY`/`no_proxy` for the local service names. This is required in proxy-enabled shells because PD and gRPC clients must not route `pd:2379`, `minio:9000`, or TiKV worker URLs through a host proxy.