# NodeRebalance Cross-AZ Algorithm

## Purpose

`NodeRebalance` handles AWS Spot rebalance signals for TiDB pool nodes. Its goal is not to pack evicted pods into the same Availability Zone (AZ). Its goal is to move workload away from the AZ that received the Spot rebalance signal by blocking Cluster Autoscaler (CA) scale-up in that AZ and letting pending pods trigger scale-up in other AZs.

## Key Ideas

- `RebalanceSignalTaintKey`: added to a Spot node after AWS sends a rebalance recommendation.
- `PreventCAScaleUpTaintKey`: added by this controller to all non-deleting nodes in the same node group and AZ.
- `allNodes`: all non-deleting nodes in the current node's node group and AZ.
- `evictNode`: evicts pods from the current rebalance-signal node being reconciled. It skips deleting pods and DaemonSet pods.

## Cross-AZ Intent

When a Spot node in an AZ receives a rebalance signal, allowing CA to scale up in the same AZ can keep the workload in the place we are trying to move away from. The desired behavior is:

1. Block CA scale-up in the current AZ.
2. Evict pods from the signaled node.
3. Let unscheduled pods become pending.
4. Let CA choose another AZ that can accept scale-up.

This is different from local pod packing. The controller does not ask whether the
current AZ looks good or bad by score. Once the workflow chooses to move away
from the signaled AZ, `PreventCAScaleUpTaintKey` is applied to that AZ and the
evicted pods are allowed to drive scheduling somewhere else.

## Text Diagram

```text
Before
------
AZ-a / node-group-x
  node-a1: Spot rebalance signal
  node-a2: normal
  node-a3: normal

AZ-b / node-group-x
  node-b1: normal
  node-b2: normal

NodeRebalance action
--------------------
1. List all non-deleting nodes in AZ-a / node-group-x
2. Taint every listed node with PreventCAScaleUpTaintKey
3. Evict pods only from the reconciled signal node

After
-----
AZ-a / node-group-x
  node-a1: RebalanceSignalTaintKey + PreventCAScaleUpTaintKey
  node-a2: PreventCAScaleUpTaintKey
  node-a3: PreventCAScaleUpTaintKey

Pending pods
  |
  v
Cluster Autoscaler does not add nodes in AZ-a because AZ-a is blocked
  |
  v
Scale-up can happen in another eligible AZ
```

## Reconcile Flow

```text
Node update triggers NodeRebalance
  |
  v
Load current node
  |
  v
List all non-deleting nodes in the same node group and AZ
  |
  v
Find nodes with RebalanceSignalTaintKey
  |
  +-- 0 signal nodes
  |     |
  |     v
  |   Remove PreventCAScaleUpTaintKey from nodes in this node group/AZ
  |
  +-- More than 1 signal node
  |     |
  |     v
  |   Add PreventCAScaleUpTaintKey to all nodes in this node group/AZ
  |     |
  |     v
  |   Evict pods from the current node if it has RebalanceSignalTaintKey
  |
  +-- Exactly 1 signal node
        |
        v
      Read rebalance signal timestamp
        |
        +-- Signal is older than maxRebalanceSignalTime
        |     |
        |     v
        |   Add PreventCAScaleUpTaintKey to all nodes in this node group/AZ
        |     |
        |     v
        |   Evict pods from the current node if it has RebalanceSignalTaintKey
        |
        +-- Signal is recent
              |
              v
            If any node in this node group/AZ already has PreventCAScaleUpTaintKey,
            complete the taint onto remaining nodes.
            Otherwise leave nodes unchanged and wait for another reconcile.
```

## Behavior By Case

| Case | Action |
| ---- | ------ |
| No rebalance signal nodes | Remove `PreventCAScaleUpTaintKey` and permit normal CA scale-up. |
| Multiple rebalance signal nodes | Add `PreventCAScaleUpTaintKey` to all nodes in the current node group/AZ, then evict pods from the current signal node. |
| One recent rebalance signal node | If the current AZ already has a prevent taint, complete the prevent taint onto the rest of the AZ; otherwise wait. |
| One old rebalance signal node | Add `PreventCAScaleUpTaintKey` to all nodes in the current node group/AZ, then evict pods from the current signal node. |
| Eviction hits PDB rate limit | Requeue after `retryTooManyRequestsTime`. |

## Expected CA Interaction

After `PreventCAScaleUpTaintKey` is present on the current AZ, pods evicted from the signaled node may become pending. Those pending pods should not cause CA to add nodes in the blocked AZ. Instead, CA can scale another eligible AZ, allowing the workload to rebalance away from the AZ that received the Spot signal.

## Metrics

```prometheus
# Whether this node group/AZ is currently blocking CA scale-up.
noderebalance_prevent_ca_scale_up{node_group, node_zone}

# Number of nodes whose pods were evicted by NodeRebalance.
noderebalance_evict_node_total{node, node_group, node_zone}
```

## Non-Goals

- Do not use NodeScorer as a gate for this workflow.
- Do not choose target nodes for the evicted pods.
- Do not force pods to stay in the same AZ.
- Do not evict pods from every rebalance-signal node in a single reconcile; eviction is scoped to the current reconciled node.

## Related Code

- Controller flow: `pkg/controllers/noderebalance/controller.go`
- Rebalance helpers and eviction: `pkg/controllers/noderebalance/rebalance.go`
- Metrics: `pkg/metrics/noderebalance_metrics.go`