# TiDB HPA Scale-In Overview

## Purpose

This document explains the **TiDB replica auto scale-in** path in Cluster Service.

It focuses on:

- what the feature does
- who is responsible for each step
- when scale-in is enabled or blocked
- what default values and operational side effects matter

It intentionally avoids code-level walkthroughs.

## Scope

This feature applies to the **TiDB HPA path** managed by Cluster Service.

It is **not**:

- RU-based scale-in or RU-driven spec downgrade
- manual scale-in validation logic
- the autoscaling path for TiDB worker / TiKV worker / Coprocessor worker
- the TiProxy path

Those paths have their own logic and should not be confused with TiDB HPA scale-in.

## Core Idea

TiDB already has HPA-based auto scale-out. The new capability adds a **guarded auto scale-in** path for TiDB replicas.

The key idea is:

- HPA remains the component that actually decides replica changes.
- TiDB scale-in is protected by an extra **CPU gate metric**.
- TiDB can scale in only when that gate explicitly allows it.

So this is not "plain HPA scale-in". It is **HPA scale-in with an extra TiDB-specific admission check**.

## Default State

TiDB scale-in is **disabled by default**.

For a newly created TiDB HPA:

- `scaleDown.selectPolicy` is `Disabled`
- there is no TiDB scale-in threshold annotation
- there is no TiDB scale-in gate metric

This means a new TiDB HPA can auto scale out, but it will **not auto scale in** unless someone explicitly enables that behavior.

The same principle also applies to existing HPAs during normal reconcile:

- reconcile can keep HPA objects aligned
- but it does not treat TiDB scale-in as a "big bang" default rollout

## Who Does What

### 1. Workflow / HPA provisioning layer

The workflow layer is responsible for creating and reconciling the TiDB HPA object.

Its responsibility is:

- ensure the TiDB HPA exists
- set min/max replicas
- keep the HPA object structurally healthy
- preserve the default "scale-out on, scale-in off" posture unless scale-in was explicitly enabled

### 2. Scale configuration API

`SetScaleOutConfig` is the main path for configuring TiDB scale-in policy.

Its responsibility is:

- accept the TiDB scale-in CPU threshold
- write that threshold onto the HPA
- ensure the TiDB gate metric is attached
- make sure TiDB `scaleDown` is no longer `Disabled`, otherwise the gate would never have any effect

In short, this path configures **how TiDB is allowed to scale in**.

### 3. HPA behavior API

`SetHPAScaleBehavior` is the explicit on/off switch for HPA behavior.

Its responsibility is:

- enable or disable `scaleUp`
- enable or disable `scaleDown`
- for TiDB, when scaleDown is explicitly enabled, also ensure the TiDB gate infrastructure is present

In short, this path controls **whether TiDB scale-in is enabled at all**.

### 4. Metrics provider

The metrics provider is the runtime judge for TiDB scale-in.

Its responsibility is:

- read the TiDB HPA threshold
- inspect current pod CPU utilization
- return a gate result that either allows or blocks scale-in

This provider is intentionally conservative:

- if pod metrics are incomplete, it blocks scale-in
- if pods are not ready, it blocks scale-in
- only when data is complete and CPU is low enough does it allow scale-in

### 5. Kubernetes HPA controller

HPA remains the final autoscaling engine.

Its responsibility is:

- evaluate the configured metrics
- combine the ordinary CPU / memory metrics with the TiDB gate metric
- decide whether replica count should go up, stay the same, or go down

The gate metric does **not** replace normal HPA metrics. It acts as a **one-way guard** for the scale-in direction.

### 6. Underlying workload controller

After HPA decides to reduce replicas, the underlying workload controller applies the new replica count to the TiDB group.

So the execution chain is still:

HPA decision -> target replicas change -> TiDB workload reconciles to that replica count

## Runtime Path

The normal TiDB scale-in path works like this:

1. Cluster Service creates or reconciles a TiDB HPA for the TiDB group.
2. By default, that HPA can scale out but cannot scale in.
3. An operator or control-plane request explicitly enables TiDB scale-in.
4. The HPA now has both:
   - a TiDB scale-in CPU threshold
   - a TiDB gate metric
5. HPA continues to evaluate regular autoscaling metrics.
6. The metrics provider evaluates whether TiDB CPU is low enough for scale-in.
7. If the gate blocks, HPA must not scale TiDB in.
8. If the gate allows, HPA may scale TiDB in, subject to normal HPA scale-down behavior.
9. The TiDB workload controller applies the reduced replica count.

## How the Gate Behaves

The TiDB gate is intentionally simple:

- if average CPU utilization is **below** the threshold, scale-in is allowed
- if average CPU utilization is **equal to or above** the threshold, scale-in is blocked
- if metrics are incomplete, scale-in is blocked

This means the gate is **single-directional**:

- it is there to control **scale-in**
- it is not the main driver for **scale-out**

That distinction is important. The gate is not a third general-purpose autoscaling metric. It is a **TiDB scale-in permission check**.

## Default Values and Numbers

### Scale-in CPU threshold

The default TiDB scale-in CPU threshold is **30%**.

That means TiDB scale-in is only allowed when average CPU utilization is **strictly below 30%**.

### TiDB HPA scale-down behavior

Once TiDB scale-in is enabled, the HPA scale-down behavior is still intentionally cautious.

Current TiDB scale-down values are:

- **stabilization window:** `3600s` (1 hour)
- **percent policy:** up to `20%` every `1800s` (30 minutes)
- **pods policy:** up to `3 pods` every `1800s` (30 minutes)
- **selectPolicy:** `Min`

Because the HPA uses `selectPolicy = Min` for TiDB scale-in, the smaller reduction wins.

Examples:

- with 20 replicas, `20% = 4`, so the effective cap is `3 replicas` per 30 minutes
- with 10 replicas, `20% = 2`, so the effective cap is `2 replicas` per 30 minutes
- with 5 replicas, `20% = 1`, so the effective cap is about `1 replica` per 30 minutes

So TiDB scale-in is not just "CPU low, then shrink". It is:

- CPU low enough to pass the gate
- plus HPA stabilization
- plus HPA step limits

## Operational Side Effects

### 1. Manual scale-out is gated when HPA scaleDown is enabled

Once HPA scaleDown is enabled for a component, manual scale-out is no longer treated as fully independent.

For TiDB, this means:

- if HPA scaleDown is enabled
- manual scale-out requests may be rejected by the service-side HPA gate
- operators must disable HPA scaleDown first if they want manual scale-out to proceed directly

This is an important operational consequence and should be expected.

### 2. Manual scale-in is a different path

This TiDB HPA gate does not redefine the separate manual scale-in safety checks.

So "HPA auto scale-in" and "manual scale-in" are related operationally, but they are not the same control path.

### 3. Existing clusters are not auto-converted by default

This feature is designed as **explicit opt-in**, not silent global enablement.

That means:

- existing TiDB HPAs do not automatically become scale-in capable
- someone must explicitly enable scaleDown or configure the TiDB scale-in threshold

## Component Boundaries

### TiDB

TiDB is the only component in this feature that uses the **CPU gate + HPA scaleDown** path described here.

### TiProxy

TiProxy does not use this TiDB scale-in path and keeps scaleDown disabled by default.

### TiDB worker / TiKV worker / Coprocessor worker

These components do not use the TiDB CPU gate.

They may still use HPA or autoscaling hints, but they are controlled by different runtime signals and should not be documented as part of the TiDB gate design.

## Summary

TiDB scale-in in Cluster Service is a **guarded HPA replica scale-in path**.

The responsibilities are split clearly:

- workflow layer creates and reconciles the HPA
- configuration APIs enable and configure TiDB scale-in
- the metrics provider decides whether scale-in is currently allowed
- HPA enforces stabilization and step limits
- the workload controller applies the final replica change

Operationally, the most important things to remember are:

- TiDB auto scale-in is **off by default**
- enabling it is an **explicit action**
- the gate only controls **scale-in**
- the default threshold is **30% CPU**
- once enabled, TiDB still scales in slowly because of the **1 hour stabilization window** and the **20% / 3 pods per 30 minutes** limits