# NodeInstanceGroup Management Design

## Overview

This document describes **NodeInstanceGroup management** in Cluster Service NG. The system manages **NodeInstanceGroups** at **region scope** (global within the service deployment), not per SharedPool. It supports:

- creating a node spec with a primary instance type attribute (initial status: `PENDING`),
- activating / deprecating a node spec via `SetNodeInstanceGroupStatus`,
- configuring ordered fallback instances (instance type + real spec + priority),
- updating fallback instances via `UpdateNodeInstanceGroup`,
- per-dataplane enable/disable overlays,
- deprecating a node spec (NoSchedule taint),
- marking instance types as **disabled** (NoSchedule taint),
- automatic shortage handling with fallback nodepools.

All writes to Dataplane Manager (DPM) are executed via workflow actions (DAG), not directly in the server layer.

## Concepts

### NodeInstanceGroup (region-level)

NodeInstanceGroup is the region-level specification for a node spec (e.g. `8c16g`) and its primary/fallback instance types. Instance type is a NodeInstanceGroup attribute.

### DataplaneNodeInstanceGroup (per-dataplane overlay)

DataplaneNodeInstanceGroup is an **overlay config** that controls whether a NodeInstanceGroup is enabled for a specific dataplane. It is optional and only applies when present.

Overlay resolution semantics: **`DataplaneNodeInstanceGroup.Status` simply overrides `NodeInstanceGroup.Status`** for that specific dataplane. The effective status for a given dataplane is determined as follows:

```
effectiveStatus(dataplaneName, nodeSpec, primaryInstance):
  override = LoadDataplaneNodeInstanceGroup(dataplaneName, nodeSpec, primaryInstance)
  group = LoadNodeInstanceGroup(nodeSpec, primaryInstance)
  
  if override != nil:
    return override.Status   // Dataplane-level override takes precedence
  else:
    return group.Status      // Fall back to region-level status
```

Overlay rules (derived from the resolution above):

- **NodeInstanceGroup status = ACTIVE**: all dataplanes can use it unless the overlay marks it DEPRECATED.
- **status = PENDING**: only dataplanes with overlay status = ACTIVE can use it (override takes precedence).
- **status = DEPRECATED**: no dataplanes can use it unless the overlay marks it ACTIVE (override takes precedence).
- **status = RETIRING**: retirement in progress; the retiring processor cron is cleaning up nodepools. Cannot be overridden to ACTIVE.
- **status = RETIRED**: fully retired; all nodepools have been cleaned up and can be safely deleted.

### NodePool (dataplane-level, in DPM)

NodePool is the actual K8s node pool created in a dataplane. For every dataplane that is allowed to use a NodeInstanceGroup, the system materializes a NodePool from the region-level NodeInstanceGroup.

### Fallback NodePool

A fallback NodePool is created when shortage is detected. It uses a larger (or different) instance type attribute but is labeled with the **primary** `node_spec` to allow pods to schedule during shortage. It also carries a label that records the **real spec** of the fallback instance type.

## Data Models

### NodeInstanceGroup (MySQL schema)

```go
type NodeInstanceGroup struct {
    ID                    string         // varchar(128), primary key
    NodeSpec              string         // varchar(64), e.g. "8c16g" (uniqueIndex with PrimaryInstance)
    PrimaryInstance       string         // varchar(64), e.g. "m5.2xlarge" (uniqueIndex with NodeSpec)
    CPU                   uint32         // cpu cores of NodeSpec
    MemoryGi              uint32         // memory Gi of NodeSpec
    AvailableCPUCore      uint32         // available cpu after reserve
    AvailableMemoryMi     uint32         // available memory (MiB) after reserve
    WithLocalDisk         bool
    LocalDiskSizeGi       uint32
    Arch                  string         // CPU architecture, e.g. "arm64" or "amd64"
    Priority              uint32         // scheduling priority for primary instance (currently fixed to 1000)
    FallbackInstances     string         // protojson of fallback instances
    DisabledInstanceTypes string         // protojson of disabled instance types
    Phase                 string         // "active" | "pending" | "deprecated" | "retiring" | "retired"
    CreatedAt             time.Time
    UpdatedAt             time.Time
    DeletedAt             gorm.DeletedAt // soft delete
}
```

Proto representation (domain entity layer uses proto type):

```protobuf
message NodeInstanceGroup {
  string name = 1;                       // display name in NcMg format (e.g. 8C16G), OUTPUT_ONLY
  NodeSpec node_spec = 2;                // detailed spec information
  string primary_instance_type = 3;      // primary cloud instance type (e.g. m5.2xlarge)
  uint32 priority = 7;                   // scheduling priority for primary instance (currently fixed to 1000)
  repeated FallbackInstance fallback_instances = 4;  // ordered by priority
  repeated string disabled_instance_types = 5;       // OUTPUT_ONLY
  Status status = 6;                     // OUTPUT_ONLY

  message FallbackInstance {
    string instance_type = 1;            // cloud instance type (e.g. m5.4xlarge)
    NodeSpec node_spec = 2;              // real spec of the fallback instance
    uint32 priority = 3;                 // required scheduling priority (< primary priority 1000)
  }

  enum Status {
    STATUS_UNSPECIFIED = 0;
    ACTIVE = 1;
    PENDING = 2;
    DEPRECATED = 3;
    RETIRING = 4;
    RETIRED = 5;
  }
}
```

Rules:

- At steady state, **one active NodeInstanceGroup per NodeSpec**. During instance type change workflows, two ACTIVE groups for the same NodeSpec may coexist temporarily.
- Pending / deprecated NodeInstanceGroups can co-exist with the active one (for history/rollback).
- `WithLocalDisk == true` **disallows** fallback instances.
- `DisabledInstanceTypes` is a global list: any instance type in the list is treated as **disabled** for selection.

### DataplaneNodeInstanceGroup (MySQL schema)

```go
type DataplaneNodeInstanceGroup struct {
    ID              string         // varchar(128), primary key
    DataplaneName   string         // varchar(128) (uniqueIndex with NodeSpec + PrimaryInstance)
    NodeSpec        string         // varchar(64)
    PrimaryInstance string         // varchar(64)
    Status          string         // "active" | "deprecated" | "retiring" | "retired"
    CreatedAt       time.Time
    UpdatedAt       time.Time
    DeletedAt       gorm.DeletedAt // soft delete
}
```

Proto representation:

```protobuf
message DataplaneNodeInstanceGroup {
  string dataplane_name = 1;
  NodeSpec node_spec = 2;
  string primary_instance_type = 3;
  Status status = 4;                    // OUTPUT_ONLY

  enum Status {
    STATUS_UNSPECIFIED = 0;
    ACTIVE = 1;
    DEPRECATED = 2;
    RETIRING = 3;
    RETIRED = 4;
  }
}
```

### NodeInstanceGroupState

State machine for shortage handling on a per-(dataplane, nodeSpec, primaryInstance) basis.

```go
type NodeInstanceGroupState struct {
    ID                            string     // varchar(128), primary key
    DataplaneName                 string     // varchar(128) (uniqueIndex with NodeSpec + PrimaryInstance)
    NodeSpec                      string     // varchar(64) (uniqueIndex)
    PrimaryInstance               string     // varchar(64) (uniqueIndex)
    State                         string     // "normal" | "shortage_detected" | "fallback_creating" | "fallback_ready" | "recovering" | "draining_fallback" | "deleting_fallback" | "deprecated"

    StateEnteredAt                time.Time  // when the current state was entered
    FencingToken                  int64      // monotonic counter for distributed lock validation
    LastShortageReason            string     // reason for the last shortage detection
    CooldownUntil                 *time.Time // if set, don't process until this time
    CreatedAt                     time.Time
    UpdatedAt                     time.Time
    DeletedAt                     gorm.DeletedAt
}
```

**States:**

| State | Description |
|-------|-------------|
| `normal` | No shortage, primary nodepool healthy |
| `shortage_detected` | Shortage confirmed, preparing to create fallback |
| `fallback_creating` | Fallback nodepool creation in progress |
| `fallback_ready` | Fallback active and serving traffic |
| `recovering` | Primary healthy, observing stability before draining |
| `draining_fallback` | Passive drain: NoSchedule taint prevents new scheduling; existing pods run until natural restart (days/weeks/months) |
| `deleting_fallback` | Deleting one fallback nodepool (lowest priority first) |
| `deprecated` | Node instance group deprecated, no shortage handling |

### NodeInstanceGroupLock

Distributed lock per (DataplaneName, NodeSpec, PrimaryInstance) for idempotent DPM operations.
This table does **NOT use soft delete** — lock rows are transient and should be fully removed on release.

```go
type NodeInstanceGroupLock struct {
    DataplaneName   string    // varchar(128), composite primary key
    NodeSpec        string    // varchar(64), composite primary key
    PrimaryInstance string    // varchar(64), composite primary key
    OwnerID         string    // varchar(128), instance ID holding the lock
    FencingToken    int64     // monotonic counter, incremented on each acquisition
    ExpiresAt       time.Time // lease expiration time
    CreatedAt       time.Time
    UpdatedAt       time.Time
}
```

### Table Names (GORM default)

- NodeInstanceGroup: `node_instance_groups`
- NodeInstanceGroupState: `dataplane_node_instance_group_states`
- NodeInstanceGroupLock: `dataplane_node_instance_group_locks`
- DataplaneNodeInstanceGroup: `dataplane_node_instance_groups`

## Naming Conventions

NodePool ResourcePrefix uses short, deterministic prefixes. The full NodePool CR name is built by DPM (Dataplane Manager) by appending instance/spec/dataplane/hash/k8s-version to this prefix.

- **Primary**: `primary`
- **Fallback**: `fallback-{normalizedPrimaryInstance}` (e.g., `fallback-m5-2xlarge`)

Instance type dots/underscores are normalized to `-` in names only (labels keep original value):
- AWS: `.` → `-` (e.g., `m5.2xlarge` → `m5-2xlarge`)
- GCP: `-` → `-` (no change, e.g., `n2-standard-8`)
- Azure: `_` → `-` (e.g., `Standard_D8s_v3` → `standard-d8s-v3`)

Examples:

- Primary pool: `primary` (DPM builds: `primary-8c16g-dp-us-east-1-{hash}-{k8s-version}`)
- Fallback pool: `fallback-m5-2xlarge` (DPM builds: `fallback-m5-2xlarge-m5a-2xlarge-8c16g-dp-us-east-1-{hash}-{k8s-version}`)

The short prefix approach avoids the 63-character limit for K8s resource names while maintaining readability.

## Labels and Taints

### Labels

#### InstanceTypeManager Managed NodePools

Primary nodepool labels:

- `tidbcloud.com/node-spec`: primary node spec (e.g. `8c16g`)
- `core.pingcap.com/priority`: primary priority (currently fixed to `1000`)
- `tidbcloud.com/managed-by`: `node-instance-group-manager`

**Priority Rules**:
- Primary instance priority is currently fixed to 1000
- Fallback instance priority must be < 1000 and must be specified explicitly
- Higher priority means preferred for scheduling
- During instance type replacement (e.g., g3 -> g4), use the temporary dual-ACTIVE workflow and status transitions instead of changing the primary priority from 1000

**Fallback Instance Constraints**:
- Fallback `node_spec` must provide **at least as much capacity** as the primary spec (CPU cores and Memory GiB must be >= primary)
- This is enforced because fallback nodepools advertise the **primary spec** for resource scheduling; a smaller fallback would overcommit the underlying machine
- Example: if primary is `8C16G`, a fallback of `4C8G` is rejected, but `16C32G` is allowed

Fallback nodepool labels:

- `tidbcloud.com/node-spec`: **primary** spec (critical for scheduling — pods selecting the primary spec can schedule to fallback nodes)
- `tidbcloud.com/real-node-spec`: **real** spec of the fallback instance
- `tidbcloud.com/shortage-detected-at`: unix timestamp
- `core.pingcap.com/priority`: custom fallback priority (< 1000, required)
- `tidbcloud.com/managed-by`: `node-instance-group-manager`

#### Dataplane Default NodePools

Dataplane default pools (created during dataplane initialization) **do not have** `tidbcloud.com/managed-by` or `tidbcloud.com/node-spec` labels. They are matched by:

- CPU cores and Memory GiB (parsed from node spec like "4C16G")
- Default priority: 1000 (treated as primary)

These pools can only be primary pools, never fallbacks.

#### Deprecation Labels

Added when a nodepool is deprecated:

- `tidbcloud.com/deprecation-phase`: `"soft"` or `"hard"`
- `tidbcloud.com/deprecated-at`: unix timestamp

Note: capacity/instance-type fields are not duplicated in labels; they are kept in NodePool fields.

### Taints

All taints use key `tidbcloud.com/nodegroup-state` with effect `NoSchedule`.

- `deprecated` – node instance group deprecation
- `shortage-fallback` – block new scheduling on fallback during recovery (passive drain; existing pods are NOT evicted)
- `disabled` – instance type marked as disabled

## API Interface List

All NodeInstanceGroup APIs are region-scoped (no region/cloud parameter).

### CreateNodeInstanceGroup

- gRPC: `NodeInstanceGroupService.CreateNodeInstanceGroup`
- HTTP: `POST /nodeInstanceGroups`
- Request: `CreateNodeInstanceGroupRequest`
  - `node_instance_group.node_spec` (required)
  - `node_instance_group.primary_instance_type` (required)
  - `node_instance_group.fallback_instances` (optional, but each fallback's `priority` is required)
- Response: `NodeInstanceGroup` (status = `PENDING`)

### ListNodeInstanceGroups

- gRPC: `NodeInstanceGroupService.ListNodeInstanceGroups`
- HTTP: `GET /nodeInstanceGroups`
- Request: `ListNodeInstanceGroupsRequest`
  - `node_spec` (optional, equals node spec in NcMg format, e.g. `8C16G`)
- Response: `ListNodeInstanceGroupsResponse`

### SetNodeInstanceGroupStatus

- gRPC: `NodeInstanceGroupService.SetNodeInstanceGroupStatus`
- HTTP: `POST /nodeInstanceGroups/{name}:setStatus`
- Request: `SetNodeInstanceGroupStatusRequest`
  - `name` (required, equals node spec in NcMg format)
  - `primary_instance_type` (required)
  - `status` (required: `ACTIVE` | `DEPRECATED` | `RETIRING`)
- Response: `NodeInstanceGroup`
- Constraints:
  - Only `ACTIVE`, `DEPRECATED`, and `RETIRING` are allowed (not `PENDING` or `RETIRED`).
  - `RETIRING` / `RETIRED` → any other status transition is **forbidden** (terminal lifecycle).
  - `RETIRING` requires the current region-level status to be `DEPRECATED` **and** all dataplane overrides to be `DEPRECATED`.
  - When transitioning to `RETIRING`, all matching dataplane overrides are automatically set to `RETIRING`.

### DeprecateNodeInstanceGroup (legacy shortcut)

- gRPC: `NodeInstanceGroupService.DeprecateNodeInstanceGroup`
- HTTP: `POST /nodeInstanceGroups/{name}:deprecate`
- Request: `DeprecateNodeInstanceGroupRequest`
  - `name` (required, equals node spec in NcMg format)
  - `primary_instance_type` (required)
- Response: `NodeInstanceGroup`
- Note: This is a convenience alias that calls `SetNodeInstanceGroupStatus` with `DEPRECATED`.

### ListDataplaneNodeInstanceGroups

- gRPC: `NodeInstanceGroupService.ListDataplaneNodeInstanceGroups`
- HTTP: `GET /nodeInstanceGroups/dataplanes/{dataplane_name}`
- Request: `ListDataplaneNodeInstanceGroupsRequest`
  - `dataplane_name` (required)
- Response: `ListDataplaneNodeInstanceGroupsResponse`

### SetDataplaneNodeInstanceGroupStatus

- gRPC: `NodeInstanceGroupService.SetDataplaneNodeInstanceGroupStatus`
- HTTP: `POST /nodeInstanceGroups/dataplanes/{dataplane_name}:setStatus`
- Request: `SetDataplaneNodeInstanceGroupStatusRequest`
  - `dataplane_name` (required)
  - `node_spec` (required)
  - `primary_instance_type` (required)
  - `status` (required: `ACTIVE` | `DEPRECATED`)
- Response: `DataplaneNodeInstanceGroup`

### SetNodeInstanceGroupInstanceTypeStatus

- gRPC: `NodeInstanceGroupService.SetNodeInstanceGroupInstanceTypeStatus`
- HTTP: `POST /nodeInstanceGroups/{name}:setInstanceTypeStatus`
- Request: `SetNodeInstanceGroupInstanceTypeStatusRequest`
  - `name` (required, equals node spec in NcMg format)
  - `primary_instance_type` (required)
  - `instance_type` (required, must be the primary or a configured fallback)
  - `status` (required: `NORMAL` | `DISABLED`)
- Response: `NodeInstanceGroup`

### UpdateNodeInstanceGroup

- gRPC: `NodeInstanceGroupService.UpdateNodeInstanceGroup`
- HTTP: `PUT /nodeInstanceGroups/{name}`
- Request: `UpdateNodeInstanceGroupRequest`
  - `name` (required, equals node spec in NcMg format)
  - `primary_instance_type` (required)
  - `fallback_instances` (required)
- Response: `NodeInstanceGroup`
- Constraints:
  - Only `fallback_instances` can be modified; primary instance type and node spec are immutable.
  - Cannot remove existing fallback instances (would leave orphaned nodepools).
  - Cannot add fallback instances when `with_local_disk` is true.
  - Region-level metadata update only; no dedicated update DAG is submitted.

### DeleteNodeInstanceGroup

- gRPC: `NodeInstanceGroupService.DeleteNodeInstanceGroup`
- HTTP: `DELETE /nodeInstanceGroups/{name}`
- Request: `DeleteNodeInstanceGroupRequest`
  - `name` (required, equals node spec in NcMg format)
  - `primary_instance_type` (required)
- Response: `NodeInstanceGroup`
- Constraints:
  - Only `RETIRED` groups can be deleted — RETIRED guarantees all nodepools have been cleaned up by the retiring processor.
## DAG Workflow Architecture

### Plan and Action Naming

Each operation has a **plan** (root node) and an **action** (dependent node). Action names use an `-action` suffix to avoid DAG self-dependency conflicts.

| Plan Name | Action Name |
|-----------|-------------|
| `create-node-instance-group` | `create-node-instance-group-action` |
| `deprecate-node-instance-group` | `deprecate-node-instance-group-action` |
| `delete-node-instance-group` | `delete-node-instance-group-action` |
| `set-node-instance-group-instance-type-status` | `set-node-instance-group-instance-type-status-action` |

### ExecID Format

ExecIDs include the instance type (normalized, dots → dashes) for uniqueness:

```
{operation}-{dataplaneName}-{nodeSpec}-{instanceType}-{traceIdHash}
```

Examples:
- `set-node-instance-group-active-dp-ff48602a-4c32g-r8g-xlarge-abc123`
- `delete-node-instance-group-dp-ff48602a-4c32g-r8g-xlarge-def456`

### DAG Args

Common args passed to all NodeInstanceGroup DAGs via `buildNodeInstanceGroupDagArgs`:

| Key | Description |
|-----|-------------|
| `dataplane_id` | Target dataplane name |
| `node_spec` | Normalized node spec (e.g. `4c32g`) |
| `primary_instance` | Primary instance type (e.g. `r8g.xlarge`) |
| `trace_id` | Request trace ID |
| `fallback_instances` | (optional) protojson-encoded fallback instances |
| `avail_cpu` | (optional) Available CPU cores |
| `avail_memory_mi` | (optional) Available memory in MiB |
| `with_local_disk` | (optional) `"true"` if local disk |
| `local_disk_size_gi` | (optional) Local disk size in GiB |

SetInstanceTypeStatus additionally passes:

| Key | Description |
|-----|-------------|
| `instance_type` | Target instance type |
| `instance_type_status` | `"normal"` or `"disabled"` |

## System Flow (Server → DAG → DPM)

### Create NodeInstanceGroup

Purpose: create a region-level NodeInstanceGroup with initial status `PENDING`. **No DAGs are submitted** — the group must be explicitly activated via `SetNodeInstanceGroupStatus`.

Flow:

1. Validate NodeSpec format and parse into CPU/MemoryGi.
2. Validate primary instance type.
3. Validate fallback instances (unique, no self-reference, priority < 1000 and required, node_spec >= primary node_spec).
4. Reject fallback instances if `with_local_disk` is true.
5. Compute defaults for available CPU (`cpu - 1`) and available memory (via `tioperator.ComputeAvailableMemory`).
6. Set status = `PENDING`, name = normalized nodeSpec.
7. Check for existing group with same (nodeSpec, primaryInstance) — reject if exists.
8. Persist NodeInstanceGroup to DB.
9. Return the created group. **No DAGs are submitted.**

### Set NodeInstanceGroup Status

Purpose: change region-level availability and submit DAGs to materialize/deprecate NodePools.

Flow:

1. Only `ACTIVE`, `DEPRECATED`, and `RETIRING` are valid target statuses.
2. `RETIRING` / `RETIRED` → any other status transition is **forbidden** — these are terminal lifecycle states.
3. Persist the new status.
4. List all dataplanes via DPM `ListDataPlanes`.
5. For each dataplane:
   - If status = `ACTIVE`: skip dataplanes with override = `DEPRECATED`. Submit `create-node-instance-group` DAG for others.
   - If status = `DEPRECATED`: skip dataplanes with override = `ACTIVE`. Submit `deprecate-node-instance-group` DAG for others.
   - If status = `RETIRING`: validate current status is `DEPRECATED` and all dataplane overrides are `DEPRECATED`. Set all matching dataplane overrides to `RETIRING`. No DAGs are submitted — the retiring processor cron handles cleanup.

### Set DataplaneNodeInstanceGroup Status

Purpose: override the region-level status for a specific dataplane.

Flow:

1. Load the region-level NodeInstanceGroup (must exist).
2. Load or create the dataplane override record.
3. Upsert `dataplane_node_instance_groups` record with new status.
4. If status = `ACTIVE`, submit `create-node-instance-group` DAG for the dataplane.
5. If status = `DEPRECATED`, submit `deprecate-node-instance-group` DAG for the dataplane.

### Set Instance Type Status (Disable/Enable)

Purpose: mark a primary or fallback instance type as disabled or re-enable it.

Flow:

1. Validate the instance type belongs to the group (primary or fallback list).
2. Update `disabled_instance_types` list on NodeInstanceGroup (add or remove).
3. List all dataplanes.
4. Submit `set-node-instance-group-instance-type-status` DAG for each dataplane, with extra args for `instance_type` and `instance_type_status`.

### Update NodeInstanceGroup (Fallback Instances)

Purpose: modify the fallback instances list for an existing NodeInstanceGroup.

Flow:

1. Validate new fallback instances (unique, valid, priority < 1000 and required, no self-reference with primary, node_spec >= primary node_spec).
2. Ensure no existing fallback is removed (prevents orphaned nodepools).
3. Reject if `with_local_disk` is true.
4. Persist updated fallback list.
5. Return updated region-level metadata (no dedicated update DAG).

### Delete NodeInstanceGroup

Purpose: delete a fully retired NodeInstanceGroup.

Flow:

1. Load the target group (must exist).
2. Ensure the group status is `RETIRED` — only RETIRED groups can be deleted.
   - RETIRED guarantees all nodepools have already been cleaned up by the retiring processor cron.
3. Delete NodeInstanceGroup record and matching dataplane overrides from DB.

**Note on Fallback Tracking:**
- Failed fallback attempts are tracked via nodepool status (FAILED), not a separate database field
- During shortage handling, failed fallbacks are automatically skipped when selecting next fallback

## Shortage Detector (Cron)

The shortage detector runs per dataplane (DPM `ListDataPlanes`) and drives the shortage state machine.

Key behaviors:

- Skips processing if the NodeInstanceGroup is not enabled for the dataplane (overlay rules).
- Selects fallback instance by **priority order** (highest first), skipping disabled/failed/existing types.
- Supports **cascading fallbacks**: multiple fallback nodepools can run simultaneously.
- Creates fallback nodepools with real spec labels and lower priority.
- Applies `shortage-fallback` taint during recovery to drain fallback nodes.
- **Graceful deletion**: Deletes fallbacks one at a time (lowest priority first) with scale-up grace period.
- **Disabled instance type awareness**: Uses `NodeInstanceGroup.DisabledInstanceTypes` as the authoritative source. A disabled primary triggers fallback creation; disabled fallbacks are skipped in favor of lower-priority alternatives.

### State Machine

```
┌─────────┐     pending pods      ┌─────────────────┐
│  normal │ ────────────────────▶│ shortage_detected│
└─────────┘                      └─────────────────┘
     ▲                                    │
     │                                    │ cooldown: 2min
     │                                    ▼
     │                           ┌─────────────────┐
     │                           │ fallback_creating│
     │                           └─────────────────┘
     │                                    │
     │              fallbackReady         │ fallbackFailed/Missing/Timeout
     │                     │              │ (keep failed, retry next)
     │                     ▼              │
     │              ┌─────────────────┐   │
     └──────────────│  fallback_ready │◄──┘
                    └─────────────────┘
                           │
              no pending pods │ shortage persists
                           │    (create additional fallback)
                           ▼
                    ┌─────────────────┐     pending pods      ┌─────────────────┐
                    │   recovering    │ ─────────────────────▶│  fallback_ready │
                    │ (cooldown 15min)│                         │   (continue)    │
                    └─────────────────┘                         └─────────────────┘
                           │
                           │ stable
                           ▼
                    ┌─────────────────┐
                    │draining_fallback│◄──────────────────────────────────────┐
                    │ (grace period   │                                       │
                    │  5min for scale)│                                       │
                    └─────────────────┘                                       │
                           │                                                  │
              has shortage │    no shortage, drained                         │
                           │                                                  │
                           ▼                                                  │
                    ┌─────────────────┐    delete lowest priority            │
                    │  fallback_ready │◄─────────────────────────────────────┘
                    │ (stop deleting) │    │
                    └─────────────────┘    │
                                           ▼
                                    ┌─────────────────┐
                                    │deleting_fallback│
                                    │ (delete one,    │
                                    │  back to drain) │
                                    └─────────────────┘
                                           │
                                           │ no more fallbacks
                                           ▼
                                      ┌─────────┐
                                      │  normal │
                                      └─────────┘
```

### State Transitions

| From | To | Trigger | Action |
|------|-----|---------|--------|
| `normal` | `shortage_detected` | pending pods detected OR primary instance disabled | Cooldown 2min for confirmation |
| `shortage_detected` | `normal` | no pending pods AND primary not disabled | False alarm |
| `shortage_detected` | `fallback_creating` | create fallback | Start creating highest priority fallback |
| `fallback_creating` | `fallback_ready` | fallback ACTIVE | Transition to ready (no taint applied) |
| `fallback_creating` | `shortage_detected` | fallback FAILED/Missing/Timeout | Keep failed fallback (skipped on retry), retry with next priority |
| `fallback_ready` | `recovering` | primary healthy, no pending pods | Start 15min stability observation |
| `fallback_ready` | `fallback_ready` | shortage persists | Create additional fallback (cascading) |
| `recovering` | `draining_fallback` | 15min stable, no shortage | Apply shortage-fallback taint, start drain |
| `recovering` | `fallback_ready` | shortage reappears | Revert to fallback (cancel recovery) |
| `draining_fallback` | `fallback_ready` | shortage detected after grace period | Stop draining, revert |
| `draining_fallback` | `deleting_fallback` | drained, no shortage | Delete lowest priority fallback |
| `deleting_fallback` | `draining_fallback` | fallback deleted, more remain | Continue draining next fallback |
| `deleting_fallback` | `normal` | last fallback deleted | Recovery complete |

### Timing Constants

```go
const (
    ShortageConfirmationDuration = 2 * time.Minute   // Confirm shortage before acting
    RecoveryStabilityDuration    = 15 * time.Minute  // Must be stable before draining (drain is passive, hard to reverse)
    DrainGracePeriod             = 30 * time.Minute  // Initial cooldown; actual drain is passive (NoSchedule, not NoExecute — days/weeks/months)
    FallbackCreatingTimeout      = 15 * time.Minute  // Fallback creation deadline
    PrimaryScaleUpGracePeriod    = 5 * time.Minute   // Wait for primary to scale up
)
```

### Health Checks

**Primary NodePool Health:**
- Status `ACTIVE` or `UPDATING` is considered healthy (regardless of replica count)
- During fallback usage, primary replicas may be 0 because primary is **overflow-only** (lower priority than fallback), and new pods are preferentially scheduled to the fallback
- Even if the nodepool status is healthy, a **disabled primary instance type** (per `DisabledInstanceTypes`) is treated as **not healthy** — recovery to primary is blocked until the instance type is re-enabled
- Even if status is `ACTIVE`/`UPDATING`, `CheckNodePoolScaleActivities` API is checked — if `insufficient_capacity=true`, the primary is treated as **not healthy** (cloud provider cannot fulfill scale-up requests)

**Shortage Detection (hasPendingPods):**
- **First**: Checks `NodeInstanceGroup.DisabledInstanceTypes` — if primary instance type is disabled, immediately returns shortage (`true`) regardless of pod state or nodepool status
- **Second**: Checks actual pending pods targeting the nodeSpec via K8s client
- **Third**: Confirms shortage via `CheckNodePoolScaleActivities` API — if `insufficient_capacity=true`, shortage is confirmed
- **Fallback**: Only if the API fails, falls back to checking nodepool status (`NODEPOOL_FAILED` → shortage)

**NodePool Matching:**
- **Dataplane default pools**: Matched by CPU/Memory (no `managed-by` label)
- **InstanceTypeManager pools**: Matched by `tidbcloud.com/node-spec` label and priority

### Cascading Fallbacks

When shortage persists despite having an active fallback:

1. Check if existing fallbacks are healthy (non-FAILED **and** non-disabled)
2. Disabled fallback instance types (per `DisabledInstanceTypes`) are skipped and not counted as healthy — the system immediately tries the next lower-priority fallback
3. Find next available fallback instance type (by priority), skipping disabled/failed/existing types
4. Create additional fallback nodepool
5. Multiple fallbacks can serve traffic simultaneously
6. During recovery, delete fallbacks one by one (lowest priority first)

### Passive Draining Semantics

**The `draining_fallback` state uses passive draining, NOT active eviction.**

All shortage-related taints use `NoSchedule` effect (not `NoExecute`). This means:

- **New pods** are prevented from being scheduled on tainted fallback nodes.
- **Existing pods** continue running on fallback nodes until they naturally restart (rolling update, OOM, crash, manual restart, etc.).
- For long-running stateful workloads like TiDB/TiKV, pods may run for **days, weeks, or even months** without restarting.
- The `draining_fallback` state can therefore persist for an extended period — this is by design.

**Drain check loop:**

1. Enter `draining_fallback` → 30min initial cooldown (`DrainGracePeriod`).
2. Each cron tick (~30s) after the 30min grace period:
   - Check `hasPendingPods` → if shortage reappears, revert to `fallback_ready`.
   - Check `isFallbackDrained` (all fallback replicas == 0, **sustained for 10 minutes**) → if not drained, return early and wait for next tick.
3. Only when all fallback pods have been confirmed terminated for the full confirmation period does the state transition to `deleting_fallback`.

> **Note on the 10-minute confirmation window:** Due to DPM reporting delays, `CurrentReplicas` from the dataplane manager may not reflect the true state immediately. The `isFallbackDrained` check requires replicas to be 0 for at least `ReplicasZeroConfirmDuration` (10 minutes) before confirming drain completion. This prevents premature fallback deletion when DPM's cached replica count hasn't caught up to reality. The zero-since timestamp is tracked in-memory; on controller restart the worst case is re-waiting the full 10 minutes.

**For N fallback nodepools**, total drain time ≈ N × (30min cooldown + actual drain time per fallback), where actual drain time depends entirely on business-level pod lifecycle, not on any timeout.

### Disabled Instance Type Handling

The `NodeInstanceGroup.DisabledInstanceTypes` field is the **authoritative source** for whether an instance type has been manually disabled by an operator. This is distinct from nodepool taints (which are a K8s-level scheduling mechanism).

**Behavior when primary instance type is disabled:**

1. `hasPendingPods` returns `true` — even if the nodepool is `ACTIVE`, a disabled primary means pods cannot be effectively scheduled, so it's treated as a shortage.
2. `isPrimaryNodePoolHealthy` returns `false` — recovery to a disabled primary is blocked. The system remains in `fallback_ready` state until the primary is re-enabled.
3. The state machine transitions `normal → shortage_detected → fallback_creating` as if a real capacity shortage occurred.

**Behavior when fallback instance types are disabled:**

1. `selectFallbackInstanceType` skips disabled instance types and tries the next priority.
2. `checkFallbackReadiness` ignores disabled fallback pools — they are treated as if they don't exist. If all fallback pools are disabled, readiness returns `fallbackMissing` so the system retries with the next available fallback.
3. `handleFallbackShortage` does not count disabled fallbacks as "healthy" when deciding whether to create additional fallbacks.

**Authoritative source:** `nodeGroupRepo.LoadBySpecAndPrimary(nodeSpec, primaryInstance).GetDisabledInstanceTypes()` — NOT nodepool taints.

## Code References

| Concern | File |
|---------|------|
| Proto definitions | `api/cluster-service-ng/v1beta/services.proto`, `resources.proto` |
| Server layer | `domain/server/node_instance_group.go` |
| Service interface | `domain/server/server.go` |
| gRPC handlers | `domain/app/app.go` |
| Entity types | `domain/entity/nodegroup.go` |
| MySQL schema | `pkg/storage/mysql/schema/node_instance_group.go` |
| State repository | `pkg/storage/mysql/node_instance_group/state.go` |
| DAG plans | `domain/workflow/dag/node.go` |
| DAG actions | `domain/workflow/action/create_node_instance_group.go`, `deprecate_...`, `delete_...`, `set_node_instance_group_instance_type_status.go` |
| Naming conventions | `pkg/nodegroup/naming.go` |
| Labels | `pkg/nodegroup/labels.go` |
| Taints | `pkg/nodegroup/taints.go` |
| Constants | `pkg/nodegroup/consts.go` |
| Shortage detector | `domain/app/cron/shortage_detector.go` |
| Retiring processor | `domain/app/cron/retiring_processor.go` |

## Implementation Notes

### UpdateNodeInstanceGroup Behavior

When updating fallback instances via `UpdateNodeInstanceGroup`:

- **Cannot remove existing fallback instances** — would leave orphaned nodepools
- **Adding new fallback instances** is allowed and takes effect immediately
- This is a region-level metadata update; no dedicated update DAG is submitted
- Does not affect currently active fallbacks during shortage handling

### Shortage Detector Concurrency

The shortage detector uses per-(dataplane, nodeSpec, primaryInstance) distributed locks:

- Lock duration: 5 minutes (`DefaultLockDuration`)
- Fencing token prevents stale lock holders from making changes
- Each dataplane is processed independently
- Multiple instance types for the same nodeSpec are processed in parallel

### Recovery Safety

During recovery (deleting fallbacks):

1. **Primary scale-up grace period**: 5 minutes after deleting a fallback before checking shortage
2. **Gradual deletion**: Only one fallback deleted at a time
3. **Automatic rollback**: If shortage reappears during recovery, immediately stop deletion and revert to `fallback_ready`
4. **Priority order**: Lower priority fallbacks (lower priority number) deleted first

## Retiring Processor (Cron)

The retiring processor is a cron job (`RetiringProcessorJob`) that handles the asynchronous cleanup of RETIRING node instance groups. It runs every 3 hours by default.

### Purpose

When a NodeInstanceGroup transitions to `RETIRING`, its nodepools may still be serving traffic. The retiring processor periodically checks each dataplane to determine when nodepools can be safely removed.

### Flow

1. List all NodeInstanceGroups with status = `RETIRING`.
2. List all dataplanes via DPM `ListDataPlanes`.
3. For each RETIRING group, process each dataplane:
   a. Skip dataplanes whose override is already `RETIRED`.
   b. List all nodepools on the dataplane.
   c. Match nodepools belonging to this group (by `tidbcloud.com/node-spec` label or CPU/Memory + instance type).
   d. If any matching nodepool has `currentReplicas > 0` → skip (still in use).
   e. If all matching nodepools have `currentReplicas == 0` → delete them via DPM and set the dataplane override to `RETIRED`.
4. After processing all dataplanes: if all dataplane overrides are `RETIRED`, set the region-level group status to `RETIRED`.

### Lifecycle

```
DEPRECATED ──(SetStatus RETIRING)──▶ RETIRING ──(cron: all dataplanes retired)──▶ RETIRED ──(Delete)──▶ removed
```

### Configuration

| Config Key | Default | Description |
|------------|---------|-------------|
| `enableRetiringProcessor` | `false` | Enable/disable the cron job |
| `retiringProcessorCron` | `0 */10 * * * *` | Cron expression (every 10 minutes) |

### Concurrency Safety

- **Local in-process guard**: Uses a `sync.Mutex` with `TryLock` to prevent overlapping executions within a single process/replica. If the mutex is already held by a previous run, the new execution is skipped.
- **Distributed cross-replica guard**: Uses a MySQL advisory lock obtained via `LockRepo` (backed by `GET_LOCK`) to ensure at most one replica in the deployment runs the processor at a time. If the advisory lock cannot be acquired (e.g. another replica holds it or the database is unavailable), the new execution is skipped.
- Together, these two layers provide at-most-one active run across all replicas, without queueing or delaying skipped runs until the next scheduled trigger.

### NodePool Matching

Nodepools are matched to a group by:
- **InstanceTypeManager-managed pools**: `tidbcloud.com/node-spec` label matches the group's nodeSpec AND instance type is in the group's instance type set (primary + all fallbacks).
- **Dataplane default pools**: CPU/Memory match AND instance type is in the group's instance type set.

## TODO

- **Auto-sync NodePools for newly created Dataplanes**: When a NodeInstanceGroup is set to `ACTIVE`, all current and future dataplanes in the region should have the corresponding NodePool created. However, Dataplane Manager currently only creates NodePools during dataplane initialization based on `NodePoolConfig` — it is not aware of NodeInstanceGroups added after the dataplane was created. Similarly, if a new dataplane is created after a NodeInstanceGroup is already `ACTIVE`, that dataplane will not automatically have the corresponding NodePool. A reconciliation mechanism is needed (e.g., a post-dataplane-creation hook, or periodic scan) to ensure newly created dataplanes automatically get NodePools for all ACTIVE NodeInstanceGroups.