# capacity-planner

Dataplane Capacity Planner.

Design docs:
- Background: Dataplane Capacity Planner detailed design (Feishu CWGJdP0baoJr6VxaRybcYwsin4g, revision 539).
- Current implementation: when mode=AUTO, the Capacity Planner scrapes resource metrics for target pods into an embedded Prometheus TSDB, evaluates restricted PromQL selectors, chooses a cost-optimized tier and replica count, and patches the target object with the tier patch plus `spec.replicas`.

## Current status

Stack (bottom-up): PR1 skeleton -> PR2 API/CRD -> PR4 manager wiring -> PR6 minimal validator -> PR7 Helm chart -> PR8 image packaging -> PR9 install smoke -> active reconciler.

- A compilable, testable controller-runtime manager: metrics / health / ready endpoints, optional leader election.
- The `Autoscaler` Reconciler is registered as the only controller (single worker, `MaxConcurrentReconciles=1`, primary watch = Autoscaler only). `DISABLED` mode is read-only. `AUTO` and `DRY_RUN` use the same tier, resource metrics, and cost-optimized policy path; `AUTO` reads the latest target, merges the desired capacity according to its installed CRD schema, and submits one optimistic-lock merge patch, while `DRY_RUN` logs the recommendation without calling the executor. Explicit merge rules are reserved for application contracts that intentionally differ from CRD ownership semantics: `nodeSelector` is atomic in Group CRDs, but capacity changes merge its `node-spec` key so an orthogonal node-pool Kubernetes-version selector is preserved. Unspecified paths continue to follow CRD `x-kubernetes-map-type` and `x-kubernetes-list-type` markers, including name-keyed `containers` and `env`. Neither mode writes Autoscaler status.
- Every `AUTO` and `DRY_RUN` evaluation — successful or not — appends one `decision` record (distinguished by `mode`) to an hourly UTC JSONL file under `records.dir`. The record carries: decision ID; occurrence time; planner version and commit; mode; outcome (`ok`, `waiting_metrics`, `read_error`, `computation_error`, or `execution_error`) plus error text; Autoscaler and target identity; generation; effective policy; replica bounds; per-metric PromQL, spec, and timestamped samples; the full tier set; the sorted metrics-target Pod name set; best-effort current capacity; and the recommendation. `DRY_RUN` records without executing; `AUTO` records only after execution succeeds or fails. The reconciler always logs a compact summary without the full record. Raw Pod metrics and the rendered patch are never logged.
- `decision_time` is the record occurrence time. Capturing the provider's exact query evaluation time remains follow-up work for historical input replay.
- Tier templates can be loaded and validated at startup with `--tier-templates-config=<path>`. The sample [`config/tier-templates.yaml`](config/tier-templates.yaml) defines the production TiDBGroup ladder `4C16G -> 8C16G -> 16C32G` and its complete node-spec, CPU/memory, and `GOMEMLIMIT` mutation contract. These resource quantities intentionally match the exact values emitted by CSNG provisioning because resource matchers use exact quantity equality; changing either side independently would make the affected tier unavailable.
- Runtime settings are loaded from one strict [`config/capacity-planner.yaml`](config/capacity-planner.yaml). `metrics` is reserved for the capacity planner's own Prometheus endpoint; embedded TSDB, query API, scraper, and recording-rule settings use the top-level `storage`, `query`, `scrapers`, and `rules` sections. Tier templates remain in their independent [`config/tier-templates.yaml`](config/tier-templates.yaml).
- Metrics packages are split by meaning: `pkg/runtime/metrics` owns the capacity planner's own Prometheus collectors, `pkg/metrics` contains only the decision Provider contract and its data model, `pkg/target` defines namespaced Pod selectors, and `pkg/prom` contains target discovery, the embedded TSDB, scraper manager, scraper implementations, query implementations, and the independent recording-rule evaluator. `pkg/prom/discovery` watches Autoscalers and caches targets by exact `targetRef.kind`; requesting an empty kind returns every unique cached target. Each scraper implements one `Scrape` operation and independently asks discovery for its configured target kind before resolving matching Pods through the manager cache. The scraper manager only schedules calls: calls to the same scraper are serialized while different scrapers cannot block one another.
- The metrics-server scraper writes configured resources such as `cpu` and `memory`; the Pod watcher converts configured JSONPath matches into gauges. The direct Pod metrics scraper only pulls annotated Pod endpoints and writes raw samples. Its exact metric-name allowlist is `scrapers.podMetrics.metrics`; an empty list retains every exposed metric. The independent `pkg/prom/rules` evaluator runs on `rules.interval`, queries any data already present in the embedded TSDB, and writes additional recorded series without waiting for or coupling to a scrape cycle. The TiDB example records `pod:tidb_server_query:rate1m` and aggregates away `type`, `result`, and `resource_group` while preserving every raw `tidb_server_query_total` series.
- Scrapers write stale markers when a Pod or series disappears; a failed direct endpoint retains its last series while other Pods still commit successfully. Metrics access requires `get/list/watch` on core Pods and `list` on metrics.k8s.io Pods. Direct Pod scraping uses the Pod network and needs no Kubernetes proxy permission. The decision provider in `pkg/prom/query` accepts only a selector or matrix selector per expression. Independently, `pkg/prom/httpquery` exposes an optional read-only Prometheus HTTP API with standard PromQL support; it is disabled by default and does not register remote-write or TSDB admin routes.
- `pkg/webhook`: minimal `AutoscalerValidator` typed `admission.Validator[*v1alpha1.Autoscaler]`. **Compiled and unit-tested but NOT served/deployed**: no webhook path registration, no Service/ValidatingWebhookConfiguration/TLS/cert material.
- [`charts/capacity-planner`](charts/capacity-planner) is the canonical chart location. The chart mounts tier templates, starts this binary with production JSON logging, and grants the active reconciler's required API surface: observe Autoscalers, list target Pods and PodMetrics, get and patch supported `core.pingcap.com` Group targets, and get only those target CRDs for schema-aware merging. The read-only Prometheus query API remains disabled by default. **No Service, no ValidatingWebhookConfiguration, no TLS/cert, no webhook values** - an HTTP surface must not be exposed without matching authentication and TLS. The chart is linted/rendered/verified (`make verify/chart`, Helm `v3.18.5` pinned in service-local `bin/`). For local development only, `make deploy` explicitly enables the query API, creates a ClusterIP Service, and provisions it as Grafana's default Prometheus datasource. Install command (documented only):
  `helm upgrade --install capacity-planner charts/capacity-planner --namespace <ns> --create-namespace --set image.repository=<repo> --set image.tag=<tag>`
- `Dockerfile` (two-stage: `golang:1.26.5-alpine` builder, `gcr.io/pingcap-public/pingcap/alpine:3.22.1` runtime - same base as dataplane-manager): builds `/capacity-planner` only, `USER 65532:65532`, exact `ENTRYPOINT ["/capacity-planner"]`, version via `--build-arg GIT_VERSION/GIT_COMMIT` from `make VERSION=...`. `make dockerbuild` remains build-only; `make test/container` builds and removes a temporary Docker image. For persistent local Kind, `make deploy/kube` installs rootless BuildKit in the cluster, while `make push` sends the source through the Kubernetes exec transport and has BuildKit push the image directly to the TLS registry. No OCI archive or image layer is returned to the host. `make deploy` creates the local environment, builds and pushes the image, then installs or upgrades the chart. This is local-development wiring only: there is still no registry login, CD wiring, production push, or production deployment.
- `make test/install` (hack/verify-install.sh): one-shot install smoke in an ephemeral single-node Kind cluster (`kind v0.32.0`, node `kindest/node:v1.36.1@sha256:3489c767...`) - unique per-PID cluster/release/namespace/image tag, build + `kind load docker-image`, `helm install --wait` (finite timeout, Deployment Ready) + `helm status` asserts `deployed`, then idempotent cleanup (uninstall -> delete cluster -> rmi) with preflight collision checks and leftover assertions. Never touches user clusters/images; no kubectl; no Autoscaler/Group creation; no real metrics; no capacity writes. **Not an install path**: no AppBox wiring, no shared/long-lived cluster, no push/login.
- Not wired into any existing CSNG startup or production deployment path. Production rollout still needs webhook serving decisions and rollout ownership checks.

## Version pins (2026-08-12)

| Component | Version | Purpose |
|---|---|---|
| controller-runtime | v0.24.1 | runtime dependency (go.mod) |
| k8s.io/{api,apimachinery,client-go} | v0.36.0 | matching controller-runtime |
| controller-gen | v0.21.0 | generator (`make generate` pins via go install) |
| setup-envtest | v0.24.2-0.20260713111223-0f529e22d5c0 | envtest asset manager (own module on release-0.24; exact pseudo-version, `make test/envtest`) |
| envtest K8s assets | 1.36.2 | integration-test API server/etcd |
| kubebuilder | v4.15.0 | generation layout reference only; not a runtime dependency |
| Kind | v0.32.0 | persistent local cluster and isolated install smoke |
| kubectl | v1.36.1 | local cluster client, aligned with the persistent Kind node image |
| MetalLB | v0.16.1 | in-cluster L2 LoadBalancer for the local registry |
| BuildKit | v0.31.2 | rootless in-cluster image build and direct registry push |
| Grafana chart | 12.10.0 (Grafana 13.1.1) | local query-engine visualization |
| Go | 1.26.5 (go.mod directive) | aligned with the repo's internal services |
| Prometheus | v0.314.0 | embedded TSDB and PromQL engine |

## Development

```bash
make build                    # aggregate all build/* targets
make build/capacity-planner   # compile ./cmd -> build/bin/capacity-planner
make dockerbuild              # build all local container images; never push
make deploy                   # aggregate deploy/kube, deploy/crd, planner, and Grafana
make push                     # aggregate all push/* targets
make testdeploy               # install E2E dependencies and apply the E2E planner config
make test/e2e                 # deploy the project and test dependencies, then run Ginkgo
make test/unit                # unit tests only
make test/evaluation          # local offline evaluation framework and catalog tests
make test/envtest             # envtest integration tests
make test                     # aggregate every test/* target, including cluster tests
make fmt                      # aggregate all fmt/* targets
make lint                     # aggregate Go and chart lint
make generate                 # generate DeepCopy code and CRD manifests
make sync/crd                 # sync the generated CRD into the chart
make verify                   # currently verify the chart only
make test/container           # image smoke: build + user/entrypoint assertions + --help run
make test/install             # ephemeral Kind install smoke (kind v0.32.0)
```

## End-to-end test

`make deploy` creates or reuses the local cluster and deploys the project. The
cluster contains one control-plane
and three worker nodes. Its API server is published at
`https://127.0.0.1:36443` on macOS and `https://172.17.0.1:36443` elsewhere by
default, with that address included in the API server certificate. The generated
kubeconfig is rewritten to use this endpoint; on Linux, this lets a development
container on the default Docker bridge reach a Kind cluster running on its host.
The command also installs a registry as a Kubernetes
Deployment and Service in the `local-registry` namespace; it does not start an
external registry container. `make deploy/kube` prints the LoadBalancer address and
generated CA path. It also installs a rootless
`buildkitd` Deployment in the `buildkit` namespace, configured to trust that
registry CA. Make installs the service-local `buildctl` and `kubectl` clients
on demand; a host Docker daemon is not used for image builds or deploys:

```bash
make deploy
```

`make push` sends the local Dockerfile and source tree to rootless BuildKit over
Kubernetes Pod exec, then BuildKit pushes the result directly to the registry
with its image exporter.
There is intentionally no host-side OCI archive and no separate ORAS upload.
`make deploy` aggregates `deploy/kube`, `deploy/crd`,
`deploy/capacity-planner`, and `deploy/grafana`, so it is sufficient on its own.
The deployed release is
`capacity-planner` in `tidb-admin`, and its rendered Deployment name is
`capacity-planner-capacity-planner`. It uses
`172.18.x.x:5000/capacity-planner:local` (with the actual allocated IP) and
`imagePullPolicy: Always`, so rebuilding the mutable `local` tag is picked up.
The deploy step also applies the current Autoscaler CRD before the atomic Helm
upgrade and waits for the rendered Deployment to become available. The chart
renders all runtime settings into `/etc/capacity-planner/capacity-planner.yaml`
and keeps tier templates in the independent `tier-templates.yaml`. The runtime
config enables direct Pod scraping with a `10s` per-Pod timeout, a 32 MiB
response limit, and up to 16 concurrent requests. It also enables
the planner's Prometheus-compatible query engine on port `9090`, creates the
cluster-only `capacity-planner-capacity-planner-query` Service, and installs
Grafana in the `monitoring` namespace. Grafana's default `Capacity Planner`
Prometheus datasource points to that Service. The query engine remains
cluster-only, while Grafana is exposed at a dedicated MetalLB LoadBalancer
address without a NodePort. Grafana also provisions a `Capacity Planner Metrics`
dashboard for `cpu`, `memory`, `pod_status_running`, `pod_status_ready`, and
TiDB query QPS from the recorded `pod:tidb_server_query:rate1m` series, with
namespace, group, and Pod filters. `make deploy` prints its direct HTTP URL.

Access Grafana at the printed LoadBalancer URL and retrieve its generated admin
password with:

```bash
bin/kubectl --kubeconfig kubeconfig -n monitoring get secret capacity-planner-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo
```

If the host does not have a route to the Kind container network, use
`make forward/grafana` as a fallback and open `http://127.0.0.1:3000`.
The provisioned dashboard path is
`/d/capacity-planner-metrics/capacity-planner-metrics`.

The registry Service is exposed by an in-cluster MetalLB L2 LoadBalancer with
`allocateLoadBalancerNodePorts: false`; no NodePort is allocated. Each Kind
node installs the generated CA, verifies the registry's HTTPS endpoint, and
configures containerd for the exact `<LoadBalancer-IP>:5000` registry address.
The Kind configuration has no registry `extraPortMappings`, and the registry
Pod has no `hostPort`. BuildKit pushes and Kind nodes pull through the
LoadBalancer; the host does not need a route to the registry. By default,
`make deploy/kube` reserves two addresses near the top of the Kind container network's
IPv4 subnet: one for the registry and one for Grafana. Explicit
`KIND_REGISTRY_LOAD_BALANCER_IP` and `KIND_GRAFANA_LOAD_BALANCER_IP` values must
be different and belong to that same reachable network.
Registry data is stored on the control-plane node and persists until the Kind
cluster is deleted; it is intended only for local development.
Override the defaults with `KIND_CLUSTER_NAME`,
`KIND_API_SERVER_ADDRESS`, `KIND_API_SERVER_PORT`, `KIND_REGISTRY_IMAGE`,
`KIND_NETWORK`, `KIND_REGISTRY_LOAD_BALANCER_IP`,
`KIND_GRAFANA_LOAD_BALANCER_IP`, or
`KIND_REGISTRY_CERT_DIR`. BuildKit overrides are `BUILDKIT_VERSION`,
`BUILDKIT_IMAGE`, `BUILDKIT_NAMESPACE`, `BUILDKIT_NAME`, and `BUILDCTL`.
When selecting a BuildKit version other than the pinned release, also set
`BUILDCTL_DOWNLOAD_SHA256` to the official release archive checksum.
kubectl overrides are `KUBECTL_VERSION`, `KUBECTL`, and
`KUBECTL_DOWNLOAD_SHA256`.
Image and deployment overrides are `IMAGE_NAME`, `IMAGE_TAG`,
`DEPLOY_RELEASE`, `DEPLOY_NAMESPACE`, `DEPLOY_TIMEOUT`,
`DEPLOY_QUERY_BIND_ADDRESS`, `DEPLOY_QUERY_SERVICE`, and `DEPLOY_QUERY_PORT`.
Grafana overrides are `GRAFANA_CHART`, `GRAFANA_CHART_VERSION`,
`GRAFANA_RELEASE`, `GRAFANA_NAMESPACE`, `GRAFANA_LOAD_BALANCER_POOL`,
`GRAFANA_DASHBOARD`, and `GRAFANA_LOCAL_PORT`. The kubeconfig
identity running `make push` or `make deploy` must be allowed to
get Pods and use `pods/exec` in the BuildKit namespace.

`make test/e2e` runs the real-cluster workflow in one command. It first runs
`make deploy`, then `make testdeploy`. The regular deployment creates the Kind
environment, applies the Autoscaler CRD, pushes the current image, and deploys
capacity-planner and Grafana. The test deployment installs metrics-server and
TiDB Operator into `tidb-admin`, then `testdeploy/capacity-planner` upgrades the
same planner release with the E2E tier templates. Ginkgo starts only after both
aggregates complete. `SynchronizedBeforeSuite` verifies the shared namespace,
required CRDs, metrics APIService, TiDB Operator Deployment, and the in-cluster
capacity-planner Deployment.

The specs still create isolated test resources, wait for observable API state,
and remove those resources afterward. An ordered serial `Describe` owns the
shared TiDB cluster and test cases. Ginkgo's verbose runner emits environment,
case, metrics, and scaling progress while the suite is running. `make deploy`,
`make testdeploy`, and `make test/e2e` default to the module-local
`./kubeconfig`; override it with
`E2E_KUBECONFIG=/path/to/kubeconfig`. The Helm releases and Autoscaler CRD
remain in the cluster after the test exits.

The e2e root contains only `e2e.go` (suite registration and environment
lifecycle) and `e2e_test.go` (the Go test entrypoint). Component specs live
under `test/e2e/suite`; the current cases form the `suite/tidb` package. Custom
resource builders are centralized in `test/e2e/data`, and every polling,
readiness, metrics, and deletion wait is centralized in `test/e2e/waiter`.

`make testdeploy` installs the vendored, Kind-compatible metrics-server chart
and the official TiDB Operator `v2.2.0-alpha.5` OCI chart. TiDB Operator itself
installs its CRDs. `testdeploy/capacity-planner` reuses the project Helm release
and replaces only its tier-template value with
`test/e2e/testdata/tier-templates.yaml`; no configuration is passed explicitly
on the Make command line. The suite
creates exactly one real one-replica PD, TiKV, and TiDB cluster and waits for
every component to become Ready and the Cluster to become Available. Workload
Pods from the dedicated `test/e2e/workload` package connect to TiDB's MySQL port
and continuously execute SQL `BENCHMARK()` queries.

One array in `test/e2e/suite/tidb/tidb.go` is the complete scaling contract. Each
entry maps one fixed-rate SQL workload directly to an expected CPU tier and
replica count. The single test loop applies the workload, waits for the exact
tier and replica count, holds that recommendation while TiDB Operator completes
the Ready rollout, then advances to the next entry. Its stages climb and descend
through `100m`, `300m`, and `600m` tiers, with the peak stage scaling the `600m`
tier from one replica to two.
Specs run sequentially in an ordered Ginkgo container with
`ContinueOnFailure`, but each starts by restoring the shared TiDBGroup to `100m`
with one replica and removing prior workloads, so a failed case neither skips
nor supplies state to the next case.

This workflow creates cluster-scoped RBAC, admission, APIService, and CRD
resources, and must only be used with an isolated disposable test cluster.
Repeated `make deploy` and `make testdeploy` calls upgrade the fixed Helm
releases instead of creating duplicate installations.

The dependency versions, release names, chart locations, namespace, and SQL
workload image can be overridden with
`E2E_TIDB_OPERATOR_VERSION`, `E2E_TIDB_OPERATOR_RELEASE`,
`E2E_TIDB_OPERATOR_CHART`, `E2E_METRICS_SERVER_RELEASE`,
`E2E_METRICS_SERVER_CHART`, `E2E_METRICS_SERVER_VALUES`,
`E2E_ADMIN_NAMESPACE`, `E2E_TIER_TEMPLATES_CONFIG`, and
`E2E_SQL_WORKLOAD_IMAGE`. `make deploy` always rebuilds and pushes the current
capacity-planner image, so the E2E run exercises the current working tree.

Filter specs by matching their Ginkgo names against `FOCUS`:

```bash
make test/e2e FOCUS='SQLWorkloadFollowsTierAndReplicaStages$'
```

Default binary behavior starts the Autoscaler Reconciler, scrapes metrics-server every 15 seconds, retains samples for one hour in `/tmp/capacity-planner-tsdb`, writes decision records under `/tmp/capacity-planner-records`, and serves `/healthz` and `/readyz`. Supplying `--config` loads all runtime settings from one strict YAML file; without it, standalone defaults are used. The only other configuration flag is `--tier-templates-config`, which intentionally keeps tier templates in their existing independent YAML file. Both metrics HTTP surfaces remain disabled by default and require TLS/authn/authz when enabled. `DISABLED` Autoscalers are read-only; `AUTO` and `DRY_RUN` Autoscalers require tier templates and metrics-server access.

The complete runtime config and independent tier-template config can be checked
during startup:

```bash
build/bin/capacity-planner \
  --config=config/capacity-planner.yaml \
  --tier-templates-config=config/tier-templates.yaml
```

The sample keeps the direct Pod metric allowlist under `scrapers` and the
independently scheduled recording rule under `rules`:

```yaml
scrapers:
  podMetrics:
    metrics:
    - tidb_server_query_total
rules:
  records:
  - record: pod:tidb_server_query:rate1m
    expr: |
      sum by (namespace, group, component, pod) (
        rate(tidb_server_query_total[1m])
      )
```