# TiDB Multi-AZ NodeGroup / ASG Design

Date: 2026-04-17
Scope: `aws-shared-cd/pool/eks-freetier`

## 1. Background

Today every EKS ManagedNodeGroup / underlying ASG created by this repo is **single-AZ**: each private subnet drives one ASG, and the per-AZ loop in `buildServerlessEKS` (`pool/eks-freetier/index.ts:559-568`) calls `asg.createFleet(...)` once per AZ. A fleet decides which AZs to land in via `ManagedOptions.zones: ZoneType[]` (`pool/eks-freetier/asg.ts:86, 674`). The `tidb-fleet` (where TiDB SQL pods run) defaults to `["primary", "standby"]`, so only 2 of the 3 configured AZs carry TiDB capacity.

Goal: let the TiDB SQL compute layer run across all 3 configured AZs, using one ASG that spans 3 subnets. Keep all existing nodegroups untouched.

## 2. Non-Goals

- Changing the existing single-AZ path for `tidb-fleet` or any other fleet.
- Multi-AZ-ing TiKV / PD / TiFlash nodegroups (stateful, replica topology concerns).
- Provider-side (`aws-shared-provider`) TiDB pod scheduling changes. That is a separate PR in the provider repo, coordinated via the rollout order in §9.
- Multi-AZ support for `eks-cn-freetier` / `ack-freetier`. Can follow the same pattern later.
- Placement group / local-SSD / cross-AZ EBS compatibility. The design explicitly rejects those combinations via validation.
- Automatic deletion / migration of existing single-AZ TiDB ASGs. Done manually by ops following §9.

## 3. Constraints (from brainstorming)

- Existing nodegroups must not be affected; multi-AZ is additive and opt-in per Pulumi stack.
- New multi-AZ TiDB nodegroup becomes the default for TiDB SQL going forward.
- Workloads on the new TiDB nodegroup are **not** managed by `tidb-operator`; the caller controls pod scheduling. We are free to design a new node-label contract instead of preserving the legacy `serverless.tidbcloud.com/zone` contract.
- Scope of "TiDB": `tidb-fleet` (and optionally `tidb-on-demand-fleet`). Not the `export-*-tidb-fleet` family.

## 4. High-Level Design

Introduce **new, separate fleet config keys** (`tidb-maz-fleet` / `tidb-maz-nodegroup` — and optionally `tidb-on-demand-maz-fleet` / `tidb-on-demand-maz-nodegroup`) that carry `multiAZ: true` in their `ManagedOptions`. These fleets are created as **one ASG spanning 3 subnets** (primary + standby + witness AZ), exactly once per cluster, outside the per-AZ loop. The existing `tidb-fleet` / `tidb-nodegroup` blocks are untouched.

Two fleet names (old single-AZ and new multi-AZ) **coexist during migration**, which is what lets §9 roll out without Pending pods: the old ASG persists while pods are still selecting it, and is deprovisioned only after pods have moved. A shared name with just a boolean flip would force `pulumi up` to delete the old ASGs in the same deploy that creates the new ones, evicting pods whose selectors don't yet match the new nodes.

`multiAZ: true` is still a general `ManagedOptions` flag (not hard-coded to "tidb-maz"), so future fleets can opt in the same way without another refactor.

### 4.1 Invocation flow

`pool/eks-freetier/index.ts` (the block currently at `index.ts:559-568`):

```ts
awsvpc.privateSubnetIds.then(async (subNets) => {
  const azSubnets: AZSubnet[] = await Promise.all(
    subNets.map(s => s.apply(id =>
      aws.ec2.getSubnet({ id }).then(info => ({ az: info.availabilityZone, subnetId: info.id }))
    ))
  );

  // New: multi-AZ fleets created once, using all 3 subnets.
  createServerlessMultiAZASG(c, prefix, azSubnets, managedASGRole);

  // Existing: per-AZ loop, unchanged. Old tidb-fleet still flows through here.
  azSubnets.forEach(({ az, subnetId }) => {
    asg.createFleet(c, prefix, az, subnetId, managedASGRole, "default");
    createServerlessASG(c, prefix, az, subnetId, managedASGRole);
  });
});
```

`pool/eks-freetier/serverless.ts` new helper:

```ts
export function createServerlessMultiAZASG(c, prefix, azSubnets, role) {
  if (config.getObject(`tidb-maz-fleet`) != null) {
    asg.createMultiAZFleet(c, prefix, azSubnets, role, "tidb-maz");
  }
  if (config.getObject(`tidb-on-demand-maz-fleet`) != null) {
    asg.createMultiAZFleet(c, prefix, azSubnets, role, "tidb-on-demand-maz");
  }
}
```

Both new fleet blocks are optional: a stack that has not opted in simply omits them. No change to existing stacks' yaml is required to deploy this PR.

### 4.2 Why `multiAZ` lives at the nodegroup (default options) layer of the *new* fleet

A fleet's size variants (`tidb-maz-x2xl`, `tidb-maz-vip-r4xl`, …) must all share the same AZ topology. Mixing single-AZ and multi-AZ variants inside one fleet has no real use case and confuses any caller selecting TiDB nodes. So `multiAZ: true` is set once on `eks-freetier:tidb-maz-nodegroup` (the new fleet's defaults) and applies to every override inside `tidb-maz-fleet`.

## 5. Config Schema

Add one field to `ManagedOptions` / `Override` in `asg.ts`:

```ts
multiAZ?: boolean;   // default false; only valid on tidb-maz-fleet and future *-maz-fleet keys
```

**Existing stacks stay as-is**, e.g. `Pulumi.prod-us-east-1-f01.yaml` keeps its current `tidb-fleet` / `tidb-nodegroup` blocks **untouched**.

**Stacks opting into multi-AZ TiDB** add two *new* config blocks (parallel to the existing ones, not replacing them):

```yaml
eks-freetier:tidb-maz-nodegroup:
  category: "compute"
  component: "tidb"
  multiAZ: true        # new flag: this fleet uses the multi-AZ code path
  min: 0

eks-freetier:tidb-maz-fleet:
  - tier: "standard"
    name: "tidb-maz-x2xl"
    releaseTag: "32v251029"
    capacityType: "MIXED"
    mixedAsgOptions:
      alternativeAutoReplace: true
      alternativeParallelScaleUp: true
      alternativeScaleUpWait: "60s"
    spot:
      name: "tidb-maz-tx2xl"
      instanceTypes: ["m6g.2xlarge", "m7g.2xlarge", "m6a.2xlarge"]
      max: 200
    onDemand:
      name: "tidb-maz-od-x2xl"
      instanceTypes: ["m6g.2xlarge", "m7g.2xlarge", "m6a.2xlarge", "m7a.2xlarge", "m6i.2xlarge", "m7i.2xlarge"]
      max: 100
```

During the migration window (§9 step 2 through step 5), **both** `tidb-fleet` and `tidb-maz-fleet` exist in the yaml and in Pulumi state. After step 5 removes `tidb-fleet`, only the multi-AZ fleet remains.

When `multiAZ: true` the following existing fields are **ignored** (validation rejects them, §8):

- `zones`
- `primary`, `standby`, `witness` (`ZoneTypeOverride`)

## 6. asg.ts Code Changes

### 6.1 New entry point

```ts
export interface AZSubnet { az: string; subnetId: string; }

export function createMultiAZFleet(
  c: eks.Cluster,
  prefix: string,
  azSubnets: AZSubnet[],     // must have length 3
  role: aws.iam.Role,
  name: string,
) {
  for (const options of loadFleetOptions(name)) {
    if (!options.multiAZ) continue;   // single-AZ fleets handled by the existing path
    createMultiAZASGWithOptions(c, prefix, azSubnets, role, options);
  }
}
```

### 6.2 Skip multi-AZ fleets in the old path

At the top of `createASGWithOptions`:

```ts
if (options.multiAZ) return null;
```

This ensures the per-AZ loop does not duplicate work for multi-AZ fleets.

### 6.3 New `createMultiAZASGWithOptions` / `createWithMultiAZ`

Parallel to `createASGWithOptions` → `createManaged` → `createWithPlacementGroup`. The new path is a separate function (not a refactor of the existing `Context` interface) so the single-AZ code path is byte-for-byte unchanged.

Differences versus the single-AZ path:

| Item | Single-AZ path | Multi-AZ path |
|---|---|---|
| `subnetIds` passed to `eks.ManagedNodeGroupOptions` | `[ctx.subnetId]` | all 3 subnet IDs from `azSubnets` |
| Node group name | `{prefix}-{azSuffix}-{name}-{releaseTag}` | `{prefix}-{name}-{releaseTag}` (no AZ suffix) |
| Placement group | Built per `placement.strategy` | None; `placement.strategy` **must** be `""` (validated in §8) |
| LaunchTemplate `placement` | Binds single AZ + optional PG | **Field omitted entirely** |
| `serverlessZone` variable | Resolved to `primary`/`standby` | Not computed; passes `undefined` through |
| Zone override merge (`options.primary`/`.standby`/`.witness`) | Applied based on current AZ | Skipped |
| `MIXED` capacity → `mixedNodeGroup()` | Called with `placementGroup` | Still called, always `placementGroup = null` |

Functions reused verbatim (no changes):

- `amiType()`, `eksReleaseVersion()`, `nodeTags()`, `translateEksTaintEffect()`
- `nodeLabels()` / `asgTags()` — they already handle `serverlessZone === undefined` (asg.ts:190-192) and simply omit the zone label, which is exactly what we want
- `mixedNodeGroup()` — accepts a null `placementGroup` and skips placement logic
- `setupASGTags()` — AZ-independent

### 6.4 Rejected alternative

Parameterising `Context` to hold `availabilityZones: string[]` / `subnetIds: string[]` was considered and rejected: it diffuses the change across every call site, couples the two paths, and makes the single-AZ path riskier to touch. The parallel-function approach keeps the blast radius strictly inside the new multi-AZ code.

## 7. Node Label Contract

Labels on nodes in a multi-AZ TiDB nodegroup:

| Label | Value | Source | Present on old single-AZ node? |
|---|---|---|---|
| `serverless.tidbcloud.com/node` | `tidb` | `options.component` | yes |
| `serverless.tidbcloud.com/nodegroup` | e.g. `tidb-x2xl-spot` | fleet override `name` | yes |
| `serverless.tidbcloud.com/tier` | e.g. `standard` | `tier` | yes |
| `serverless.tidbcloud.com/kubelet-version` | e.g. `1.32.9-20250627` | release tag | yes |
| `serverless.tidbcloud.com/capacity-type` | `ON_DEMAND` / `SPOT` | MIXED branch | yes |
| `aws.amazon.com/eks-local-ssd` | `true`/`false` | `ssd` | yes |
| `topology.kubernetes.io/zone` | e.g. `us-east-1a/b/c` | kubelet auto-fill, actual AZ of the node | yes (all EKS nodes) |
| `serverless.tidbcloud.com/zone` | — | — | **absent** (vs. `primary`/`standby` on old) |

Cluster-autoscaler ASG tags (`k8s.io/cluster-autoscaler/node-template/label/...`) follow the same set — i.e. no `serverless.tidbcloud.com/zone` tag on the new ASG.

### 7.1 Caller-side contract (outside this infra PR, documented for the provider team)

Node selection (use the new multi-AZ fleet names, e.g. `tidb-maz-tx2xl` / `tidb-maz-od-x2xl`, so the selector unambiguously picks multi-AZ nodes and can coexist with the old `tidb-*` selector during migration):

```yaml
nodeSelector:
  serverless.tidbcloud.com/node: tidb
  serverless.tidbcloud.com/nodegroup: <multi-AZ fleet override name, e.g. tidb-maz-tx2xl>
```

Cross-AZ spread:

```yaml
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway   # or DoNotSchedule, per availability policy
    labelSelector:
      matchLabels: { app: tidb, ... }
```

Rationale: old TiDB pods select by `serverless.tidbcloud.com/zone=primary|standby`. Because new nodes do not carry that label, the old selector would leave pods Pending — this is deliberate and drives the rollout ordering in §9.

## 8. Validation

In `validateOptions`, when `options.multiAZ === true`:

- `placement.strategy` must be `""`. AWS placement groups bind to a single AZ; any other strategy is incompatible with a multi-AZ ASG → reject with explicit error.
- `scaleInProtected` must be `false`. Multi-AZ ASGs rely on AWS rebalancing across AZs; scale-in protection defeats that.
- `category` must be `compute`. Prevents accidentally flagging storage / control-plane fleets as multi-AZ (out of scope for this change).
- `zones`, `primary`, `standby`, `witness` must be absent / empty. These fields are meaningless under multi-AZ and should fail loudly rather than silently ignored.

## 9. Rollout Order (per stack)

Each stack migrates TiDB compute to multi-AZ in ordered steps, so no pod goes Pending. The key invariant: the **old** `tidb-fleet` block stays in the yaml (and its ASGs stay in Pulumi state) until all TiDB pods have moved off it.

1. **PR**: land multi-AZ support in `asg.ts` + `serverless.ts` + `index.ts`. No stack opts in yet. Existing stacks are byte-for-byte unaffected (no `tidb-maz-fleet` block → `createServerlessMultiAZASG` is a no-op for that stack).
2. **PR**: for the target stack, add new `tidb-maz-nodegroup` + `tidb-maz-fleet` config blocks (see §5) **without touching** the existing `tidb-fleet` block. Set the new fleet's `min > 0` to pre-warm capacity roughly matching the current steady-state TiDB node count. Deploy. Observe the new multi-AZ ASG coming up with nodes Ready across 3 AZs; old single-AZ TiDB ASGs remain untouched and keep running their pods.
3. **PR (provider repo)**: update TiDB pod scheduling to target `serverless.tidbcloud.com/nodegroup=tidb-maz-*` (+ topology spread on `topology.kubernetes.io/zone`). Deploy. TiDB pods roll onto the new multi-AZ nodegroup; pods drain off the old single-AZ nodes.
4. Old `tidb-fleet` ASGs naturally scale to 0 via cluster-autoscaler (their `min: 0`); the ASG resources themselves stay in Pulumi state.
5. **PR (after a stabilisation period)**: delete the old `tidb-fleet` + `tidb-nodegroup` config blocks from the stack yaml to fully deprovision the ASGs. Verify zero residual pods on old nodes first.
6. Post-rollout: lower the multi-AZ fleet's `min` back to the long-term steady-state value (often 0 with CA).

Steps 1 and 2 can be combined in a single PR per stack if desired. Steps 2 and 3 are deliberately separate deploys — that gap is where the old ASG keeps serving live pods while the new ASG warms up.

## 10. Validation After Deploy (what this infra design must prove)

1. One EKS ManagedNodeGroup exists per multi-AZ fleet override (one per size variant), each with 3 subnet IDs.
2. `kubectl get node -l serverless.tidbcloud.com/nodegroup=<name>` shows nodes across 3 `topology.kubernetes.io/zone` values.
3. Cluster-autoscaler scales the multi-AZ ASG up and down for pending / idle TiDB pods without log errors.
4. **Out of scope for this design**: TiDB pod scheduling and distribution, which is verified on the provider side.

## 11. Risks

1. **Cross-AZ data-transfer cost.** TiDB SQL pods talk to TiKV across AZs regardless of this change (TiKV region leaders are already distributed). Net incremental cost is small, but worth watching post-rollout.
2. **Cluster-autoscaler with multi-AZ ASGs.** Upstream guidance prefers one ASG per AZ. In our case nodes carry no AZ affinity and pods use topology spread (not strict anti-affinity), which is the supported condition for multi-AZ ASGs + CA. Scale-down picks don't consider AZ balance; AWS's own AZ rebalancer compensates. Observe CA logs post-rollout.
3. **Spot interruptions across AZs.** With `capacity-rebalance` already enabled (asg.ts:367) and multi-AZ pool breadth, availability should be no worse than today's per-AZ spot ASGs.
4. **Launch template without `placement.availabilityZone`.** Confirmed safe: the field is only required when a placement group is set. No downstream consumer in this repo depends on it.
5. **Name collisions.** Old node group names include an AZ suffix; new multi-AZ names do not. No collision.
6. **CA ASG tags.** Must drop the `…/node-template/label/serverless.tidbcloud.com/zone` tag on the multi-AZ ASG, otherwise CA mis-models the nodes it will create.