# TiFlash Manager

`tiflash-manager` is a Go service that does two jobs for multi-pool TiFlash deployments in Kubernetes:

1. Resolve the TiFlash endpoints a client should use for a keyspace or tenant token.
2. Reconcile pool-specific Kubernetes resources by cloning them from a template pool when needed.

It is designed for deployments where a default TiFlash pool always exists and some keyspaces are routed to specialized pools.

## What It Does

- Watches a ConfigMap and keeps an in-memory mapping from `keyspace_id` or `tenant_token` to a TiFlash pool.
- Watches Pods for the default pool and every desired specialized pool.
- Returns only running Pods with a valid `apps.kubernetes.io/pod-index` label and a resolvable service DNS name.
- Falls back to the default pool if the preferred pool is missing or currently has no valid stores.
- Optionally creates missing pool `Service`, `StatefulSet`, and `HorizontalPodAutoscaler` objects from a template pool.
- Prunes previously managed pool resources when a pool is removed from the desired state.

## How Pool Resolution Works

For `GET /v1/cluster` and `GET /resume-and-get-topology`:

1. The service resolves a preferred pool from `tenant_token` or `keyspace_id`.
2. If that pool exists and has at least one valid running Pod, it is returned.
3. Otherwise the request falls back to the default pool.
4. If neither pool has usable stores, the service returns `503`.

This makes the routing path tolerant to partial pool failures while keeping the default pool as a safe fallback.

## Requirements

- Kubernetes access, either:
  - in cluster via `rest.InClusterConfig()`, or
  - outside the cluster via `$KUBECONFIG` or `~/.kube/config`
- An existing default TiFlash pool in the target namespace
- If you want the manager to create new pools, the template pool must already have:
  - a peer `Service`
  - a `StatefulSet`
  - a `HorizontalPodAutoscaler`
- TiFlash Pods should expose the label `apps.kubernetes.io/pod-index`, because DNS addresses are generated from `{POOL}` and `{INDEX}`

## Configuration

### Runtime Flags

| Flag | Default | Description |
| --- | --- | --- |
| `-http-port` | `8080` | HTTP server port |
| `-tiflash-port` | `3930` | TiFlash service port returned to clients |
| `-tiflash-namespace` | `tidb-serverless` | Namespace containing TiFlash resources |
| `-tiflash-config-map` | `tiflash-manager-config` | ConfigMap name watched by the manager |
| `-tiflash-default-pool` | `main` | Default fallback pool |
| `-tiflash-template-pool` | `main` | Default template pool used when creating new pools |
| `-tiflash-statefulset-name-format` | `tiflash-{POOL}` | StatefulSet name template |
| `-tiflash-peer-service-name-format` | `tiflash-manager-{POOL}-peer` | Peer Service name template |
| `-tiflash-hpa-name-format` | `tiflash-{POOL}` | HPA name template |
| `-tiflash-service-dns` | `tiflash-{POOL}-{INDEX}.tiflash-manager-{POOL}-peer.tidb-serverless.svc.cluster.local` | Per-Pod service DNS template |

### Environment Variables

| Variable | Default | Description |
| --- | --- | --- |
| `LOG_LEVEL` | `info` | One of `debug`, `info`, `warn`, `error`, `dpanic`, `panic`, `fatal` |

### ConfigMap Schema

The manager watches the ConfigMap named by `-tiflash-config-map` and expects a key named `tiflash-manager-config.yaml`.

Example:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tiflash-manager-config
  namespace: tidb-serverless
data:
  tiflash-manager-config.yaml: |
    keyspaces:
      "10001": analytics
      "10002": main
      "10003": batch

    pools:
      analytics:
        create: true
        templatePool: main
      batch:
        create: true
        templatePool: main
```

Semantics:

- `keyspaces` is the preferred keyspace-to-pool mapping.
- `specialize` is still accepted as a deprecated alias and merged behind `keyspaces`.
- `pools.<name>.create: true` tells the reconciler to clone missing resources for that pool from the template pool.
- `pools.<name>.templatePool` overrides the global `-tiflash-template-pool` for one pool.
- Any pool referenced by `keyspaces` becomes part of the desired pool set even if it is not explicitly listed under `pools`.
- The default pool is always present in the desired pool set.

Operationally, this means:

- If a pool already exists, mapping a keyspace to it is enough for routing and Pod watching.
- If a pool does not exist yet, add a `pools` entry with `create: true` so the manager can create the supporting Kubernetes resources.
- If the ConfigMap is missing, the service keeps serving the default pool only.

## API

### `GET /healthz`

Simple liveness endpoint.

Response:

```text
OK
```

### `GET /metrics`

Prometheus metrics endpoint.

Application middleware currently exposes:

- `http_requests_total`
- `http_request_duration_seconds`

Requests to `/metrics` and `/healthz` are excluded from these middleware metrics.

### `GET /v1/cluster`

Returns the resolved TiFlash cluster for a `keyspace_id` or `tenant_token`.

Query parameters:

- `keyspace_id`: optional if `tenant_token` is provided
- `tenant_token`: optional if `keyspace_id` is provided

At least one of them is required.

Example:

```bash
curl "http://127.0.0.1:8080/v1/cluster?keyspace_id=10001"
```

Example response:

```json
{
  "cluster": {
    "pool": "analytics",
    "port": 3930,
    "pods": [
      {
        "name": "tiflash-analytics-0",
        "address": "tiflash-analytics-0.tiflash-manager-analytics-peer.tidb-serverless.svc.cluster.local"
      }
    ]
  }
}
```

Error response example:

```json
{
  "error": {
    "error-code": 400,
    "error-msg": "keyspace_id or tenant_token are required"
  }
}
```

### `GET /v1/clusters`

Returns all currently watched pools and their resolved Pods.

Example:

```bash
curl "http://127.0.0.1:8080/v1/clusters"
```

### `GET /resume-and-get-topology`

Returns a flattened list of `host:port` entries for the resolved pool.

Query parameters:

- `keyspace_id`: required
- `resource_group`: optional
- `trace_id`: optional

`resource_group` is accepted and logged. `trace_id` is attached to request context for logging and tracing.

Example:

```bash
curl "http://127.0.0.1:8080/resume-and-get-topology?keyspace_id=10001&trace_id=req-1"
```

Example response:

```json
{
  "hasError": 0,
  "errorInfo": "",
  "state": "resumed",
  "topology": [
    "tiflash-analytics-0.tiflash-manager-analytics-peer.tidb-serverless.svc.cluster.local:3930"
  ],
  "timestamp": "1744199160000000000"
}
```

## Reconciliation Behavior

When `create: true` is set for a desired pool, the reconciler attempts to ensure:

- peer `Service`
- `StatefulSet`
- `HorizontalPodAutoscaler`

Creation is template-based:

- the source objects are read from the pool named by `templatePool`
- pool-scoped names and selectors are rewritten
- unrelated strings are left unchanged

Managed resources are labeled and later pruned only if they belong to the manager's desired set. This limits deletion to resources previously created and marked as managed by `tiflash-manager`.

## Build

Build the server binary:

```bash
make build
```

Or directly:

```bash
go build -o bin/server ./cmd/server
```

## Run Locally

Use your current kubeconfig:

```bash
go run ./cmd/server
```

Or run the built binary with explicit flags:

```bash
LOG_LEVEL=debug ./bin/server \
  -http-port=8080 \
  -tiflash-namespace=tidb-serverless \
  -tiflash-config-map=tiflash-manager-config
```

## Docker

The repository includes a `Dockerfile`, but it uses internal base images from `hub.pingcap.net`. You need access to that registry for `docker build` to succeed.

```bash
docker build -t tiflash-manager .
```

## Development

Run tests:

```bash
go test ./...
```

Current local result:

```text
ok  	github.com/tidbcloud/tiflash-manager/pkg/core
ok  	github.com/tidbcloud/tiflash-manager/pkg/handler
```

## Repository Layout

- `cmd/server`: application entrypoint
- `pkg/config`: flag-based runtime configuration
- `pkg/core`: pool cache, ConfigMap-driven desired state, and reconciler logic
- `pkg/handler`: HTTP handlers
- `pkg/k8s`: Kubernetes client bootstrap and utilities
- `.github/workflows/cd.yml`: image build workflow