# TiDBPodRescheduler Algorithm

## Purpose

`TiDBPodRescheduler` is a proactive placement optimizer for TiDB free-cache pods.
It reacts to `NodeScore` updates and moves idle TiDB pods from lower-score nodes
to higher-score nodes so the free-cache pool follows the current node score
distribution.

The score is used only to rank nodes relative to each other. The controller does
not use absolute score thresholds or average scores as a pre-check before
running the original reschedule algorithm.

## Text Diagram

```text
NodeScore update
  |
  v
List TiDB nodes for the pool
  |
  v
Sort by SchedulerNodeScore, low -> high
  |
  +-----------------------+------------------------+
  |                       |                        |
  v                       v                        v
Source nodes          Middle nodes             Target nodes
bottom 20%            remaining nodes          within 20 of highest score
or NodeNeedEvict
  |                       |
  |                       +-- required gap; does not donate pods
  |
  v
          collect free-cache pods from source nodes only
  |
  v
          Evict selected free-cache pod
  |
  v
          Create replacement pod with node affinity
  |
  v
                    Lands on target node
```

## Trigger

The controller watches the `NodeScore` object named `node-score-map` in the
controller namespace. Reconcile runs only when `SchedulerNodeScore` is non-empty.

For each TiDB manager known to the reconciler, it calls:

```go
r.reschedule(ctx, tManager, nodeScoreMap.Status)
```

Before each reschedule pass it also deletes previously created reschedule pods
that are stuck in `Pending`.

## Node Selection

The controller lists nodes with:

- `serverless/component=tidb`
- optional node-group filter from the TiDB pool

The nodes are sorted by `SchedulerNodeScore` in ascending order. A missing score
uses Go's map zero value, so it behaves like score `0`.

After sorting, nodes are split into three groups.

## Target Nodes

`rescheduleToNodes` are nodes that should receive free-cache pods.

A node is a target when:

- it is not being deleted
- it is not marked by `NodeNeedEvict`
- it is Ready
- it is schedulable
- its score is within 20 points of the highest score

The comparison is:

```go
nodeScore.SchedulerNodeScore[nodeName] > highestScore - 20
```

The target list is reversed after collection so the highest-score nodes are
processed first.

## Source Nodes

`rescheduleFromNodes` are nodes that should give up free-cache pods.

A node is a source when either condition is true:

- it is marked by `NodeNeedEvict` for CPU or memory
- it is in the bottom 20 percent of the sorted node list

`NodeNeedEvict` is checked before target selection. A node marked by
`NodeNeedEvict` becomes a source even if its score is within the target window.
It does not bypass the rest of the algorithm as a global mode.

## Middle Nodes

`rescheduleMiddleNodes` are all remaining nodes.

Middle nodes are a separation guard between source and target groups. At least
one middle node must exist; otherwise the controller treats the score bands as
too close and skips rescheduling. Middle nodes do not donate free-cache pods.

## Free-Cache Reserve

The controller calculates a target free-cache count per node:

```go
freeCachePerNode = ceil(totalFreeCache / nodeCount)
```

For source nodes it keeps 80 percent of that reserve:

```go
freeCachePerFromNode = ceil(freeCachePerNode * 0.8)
```

Free pods are collected only from source nodes, after the
`freeCachePerFromNode` reserve. Middle nodes are not collected from.

The candidate list is capped at `types.MaxAbnormalTidbPods`.

There is no pod startup grace-period filter. A pod returned by
`GetTidbFromFreeCache` is eligible once it is beyond the per-node reserve.

## Target Demand

For each target node, the controller skips it if `canNodeReschedule` returns
false. A failed target is cooled down for `20m` using
`types.AnnoNodeRescheduleFailTimestamp`.

For a usable target node, demand is:

```go
needPods = freeCachePerNode - currentFreeCacheOnTarget - incomingPodsOnTarget
```

`incomingPodsOnTarget` comes from abnormal pods whose node name or node affinity
already points to the target node.

If `needPods <= 0`, the target already has enough free-cache pods and is skipped.

## Eviction and Replacement

For every needed pod:

1. Pick the next collected free pod.
2. Evict it through the Kubernetes eviction subresource.
3. Create a new TiDB pod from the pool template.
4. Add `types.AnnoTidbReschedule=true`.
5. Add required node affinity to force the new pod onto the target node.

If eviction returns `TooManyRequests`, processing stops immediately and the next
reconcile can retry. Other eviction failures skip that free pod and continue.

## Pending Pod Cleanup

Pods created by this controller have `types.AnnoTidbReschedule=true`. The
controller indexes pending pods with that annotation.

At the beginning of each reschedule pass, pending reschedule pods are deleted.
If a pending pod had node affinity to a target node, that node is annotated with
`types.AnnoNodeRescheduleFailTimestamp`. That annotation makes
`canNodeReschedule` reject the node for 20 minutes.

## Skip Reasons

The controller records `RescheduleSkipCounter` with these reasons:

- `NoToNodes`: no eligible target node
- `NoMiddleNodes`: no middle node exists, so source and target bands are too close
- `NoFromNodes`: no source node exists
- `NoFreePods`: no free-cache pods are available after reserves
- `NodeNotReschedulable`: target node is in failure cooldown
- `EnoughFreePods`: target already has enough free-cache pods
- `NoMoreFreePods`: demand remains but collected free pods are exhausted

It records `RescheduleCounter` for:

- `EvictFreePodFail`
- `EvictFreePodSuccess`
- `CreateNewPodFail`
- `CreateNewPodSuccess`

## Important Non-Goals

This controller is not a Cluster Autoscaler coordinator. It does not:

- decide whether the cluster should scale up
- reject rescheduling based on average node score
- require a minimum number of healthy nodes
- require the best node to exceed an absolute score threshold
- treat `NodeNeedEvict` as a global forced-eviction mode

Those decisions would change the original behavior. The original behavior is
purely relative: sort nodes by score, pick better nodes as targets, pick worse
or explicitly evicted nodes as sources, require a middle-node gap, then move
source free-cache pods subject to the existing reserve and pending-pod
safeguards.