# Etcd Rolling Upgrade Orchestration

Date: 2026-04-23
Scope: `aws-shared-cd/pool/{eks,ack}-freetier`, shared across `etcd` and `keyspace-etcd-group-*`

## 1. Background

The two etcd-based StatefulSets in this repo use the default Kubernetes rolling update:

- `pool/manifests/etcd.yaml` — the control-plane etcd (`etcd` sts in `tidb-admin` ns, 3 replicas, HTTP, no auth).
- `pool/manifests/serverless/keyspace-etcd-group/sts.yaml` — per-group keyspace etcd (`keyspace-etcd-group-*` sts in `tidb-serverless` ns, 3 replicas, HTTPS with mTLS via secret `keyspace-etcd-group-<n>-cluster-secret`).

During EKS version upgrades the node selector (`serverless.tidbcloud.com/kubelet-version`) changes, which rolls the StatefulSet. Kubernetes' default behavior goes high-ordinal → low-ordinal and waits for each pod's `Ready` condition before moving to the next. This is safe for quorum but **not smooth**:

- Raft leader is not considered. If the leader happens to be on `etcd-2`, killing it forces a leader election. The new leader may be `etcd-1`, which is then killed → another election. Worst case is 2–3 elections per upgrade.
- Neither sts has a `readinessProbe`, so `Ready=True` only means "container started", not "etcd has joined quorum and caught up the raft log".
- There is no cool-down between pod restarts; the next pod can be killed the instant the previous one reports Ready, before it has fully caught up.

Each leader election causes a brief write stall (hundreds of ms to a few seconds) that propagates to PD/keyspace control-plane callers. During EKS upgrades this is visible as elevated error rates on operations that hit etcd.

Goal: take over the upgrade flow for these StatefulSets so that EKS version upgrades (and any other spec change that forces a roll) produce **zero leader elections** and a deliberate, health-checked cadence.

## 2. Non-Goals

- Changing etcd image, configuration, or cluster membership semantics. Same 3-node HTTP etcd / 3-node TLS keyspace-etcd; only the **upgrade choreography** changes.
- Managing *non*-etcd StatefulSets (TiKV / PD / TiDB are handled elsewhere).
- Auto-remediation of failed upgrades. A Job that fails stops and waits for human attention; it does not roll back.
- Changing the existing ownership layering. The main `etcd` is applied by `pool/{eks,ack}-freetier/etcd.ts`; `keyspace-etcd-group-*` is applied by `pool/lib/serverless/keyspace-etcd-group.ts` via `InstallKeyspaceEtcdGroup`, called from `pool/eks-freetier/serverless.ts:1175`. Both stay where they are; the rollout component is added alongside.
- Protecting against concurrent third-party deletion of etcd pods (node drain, manual `kubectl delete`). Existing protections (`safe-to-evict: false`, custom evicter ignore label, critical-services node isolation) are assumed to hold.

## 3. Constraints (from brainstorming)

- One reusable component must serve both `etcd` (HTTP, `tidb-admin`) and `keyspace-etcd-group-*` (TLS, `tidb-serverless`). Per-sts differences (namespace, service name, TLS cert paths) are expressed via a per-instance `ConfigMap`.
- Pulumi stays the single declarative entry point. Users run `pulumi up`; the rollout happens as a side-effect of that apply and Pulumi **waits for success** before returning.
- Rolling upgrade must be idempotent and resumable: re-running `pulumi up` on the same target version is a no-op; re-running after a mid-flight failure picks up where it left off.
- `updateStrategy: RollingUpdate` with `partition` as the control surface; the script decrements `partition` to drive paced rolls. `OnDelete` was the original choice and was abandoned after CI revealed an unrecoverable K8s validation conflict against existing live state (see §10.1).
- First-time cluster creation (greenfield) must succeed without the Job failing or blocking.

## 4. High-Level Design

Introduce a Pulumi `ComponentResource` `EtcdRollingUpgrade` in `pool/lib/rolling-upgrade/`. Each instance manages the upgrade flow for a single StatefulSet. For each managed sts:

1. Pulumi updates `sts.spec.template` to the target version (image, nodeSelector, etc.) as today, **but** the sts has `updateStrategy: { type: RollingUpdate, rollingUpdate: { partition: replicas } }`. Because every pod ordinal is `< partition`, the sts controller will **not** auto-roll any pod even though `updateRevision` has advanced.
2. Pulumi creates / ensures a per-instance `ConfigMap` carrying the rollout parameters (sts name, namespace, service, TLS config, timings).
3. Pulumi creates a `Job` whose name embeds a `versionHash` computed from all Pulumi inputs that influence `sts.spec.template`. Same hash → same Job name → Pulumi sees it already exists → no new rollout. Different hash → new Job.
4. The Job runs a shared `rollout.sh` script (mounted from a second, immutable `ConfigMap`), which orchestrates: move leader to `etcd-0`, decrement `partition` from `replicas-1` down to `1` to roll high → low (each step waits for the corresponding pod to reach the target revision + etcd health), move leader to `etcd-(N-1)`, set `partition: 0` to roll `etcd-0`, then re-assert `partition: replicas` to re-lock the sts.
5. Pulumi waits for the Job to reach `Complete` condition. A failed Job fails the `pulumi up`.

### 4.1 Component shape

```ts
// pool/lib/rolling-upgrade/index.ts
export interface EtcdTLSConfig {
  caFile: string;                              // path inside the etcd pod
  certFile: string;
  keyFile: string;
}

export interface EtcdRollingUpgradeArgs {
  stsName: string;                             // e.g. "etcd", "keyspace-etcd-group-0"
  namespace: pulumi.Input<string>;
  serviceName: string;                         // headless service for per-pod DNS
  clientPort?: number;                         // default 2379
  tls?: EtcdTLSConfig;                         // omit = HTTP
  versionHash: pulumi.Input<string>;           // sha1 of pulumi inputs affecting template
  cooldownSeconds?: number;                    // default 30
  readyTimeoutSeconds?: number;                // default 300
  healthTimeoutSeconds?: number;               // default 60
  dependsOnSts: pulumi.Input<pulumi.Resource>; // sts ConfigFile, ensures sts applied first
}

export class EtcdRollingUpgrade extends pulumi.ComponentResource {
  public readonly job: k8s.batch.v1.Job;
  constructor(name: string, args: EtcdRollingUpgradeArgs, opts?: pulumi.ComponentResourceOptions);
}
```

The component produces four child resources:

1. `ServiceAccount` + `Role` + `RoleBinding` scoped to the target namespace, with verbs:
   - `get`, `list`, `watch`, `patch` on `statefulsets` (read `status.updateRevision`, write `spec.updateStrategy.rollingUpdate.partition`)
   - `get`, `list`, `watch` on `pods` (read revision label + ready condition for `pod_up_to_date`; no `delete` — the sts controller is the only actor that deletes pods)
   - `create`, `get` on `pods/exec` (run `etcdctl` inside etcd containers)
2. A per-instance `ConfigMap` named `${stsName}-rollout-config-${versionHash:0:8}` (one per target version; previous versions' ConfigMaps can be GC'd by `ownerReferences` or `ttlSecondsAfterFinished` via the Job).
3. A shared `ConfigMap` `etcd-rollout-script` (namespace-scoped, one per namespace) holding `rollout.sh`. Created lazily: if another instance already created it in the same namespace, reuse.
4. The `Job`, name `${stsName}-rollout-${versionHash:0:8}`, mounting both ConfigMaps.

The Job uses `restartPolicy: OnFailure`, `backoffLimit: 0` (a failure means human attention, not silent retry), and `ttlSecondsAfterFinished: 604800` (keep one week for postmortem). Pulumi's default await behavior on `batch/v1.Job` is to wait for `Complete` — no special annotation needed.

### 4.2 Per-instance ConfigMap schema

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: keyspace-etcd-group-0-rollout-config-<hash8>
  namespace: tidb-serverless
data:
  config.yaml: |
    stsName: keyspace-etcd-group-0
    namespace: tidb-serverless
    serviceName: keyspace-etcd-group-0-service
    clientPort: 2379
    tls:
      enabled: true
      caFile: /var/lib/etcd-tls/ca.crt
      certFile: /var/lib/etcd-tls/tls.crt
      keyFile: /var/lib/etcd-tls/tls.key
    cooldownSeconds: 30
    readyTimeoutSeconds: 300
    healthTimeoutSeconds: 60
```

The HTTP etcd variant omits the `tls` block (or sets `tls.enabled: false`).

### 4.3 Rollout algorithm (`rollout.sh`)

The script runs under POSIX `sh` (alpine/k8s image uses BusyBox ash). The canonical source is `pool/lib/rolling-upgrade/rollout.sh`; the full content is in implementation plan Task 3. The algorithm in prose:

1. **Inputs**: read `stsName`, `namespace`, `serviceName`, `clientPort`, `tls.*`, `cooldownSeconds`, `readyTimeoutSeconds`, `healthTimeoutSeconds` from a mounted ConfigMap at `/config/config.yaml`.
2. **Constants**: read `REPLICAS` from `sts.spec.replicas` and `TARGET_REV` from `sts.status.updateRevision`. Build `ENDPOINTS` = comma-joined `${scheme}://${stsName}-{0..N-1}.${svc}.${ns}:${port}`.
3. **Helpers**:
   - `set_partition <n>` — `kubectl patch sts ${stsName} --type=merge -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":<n>}}}}'`. Idempotent at the API server. The sts controller picks up the change and rolls pods with ordinal ≥ `<n>` that are not at `TARGET_REV`, one at a time under `OrderedReady`.
   - `etcdctl_on <pod> …` — exec inside `<pod>` against its own endpoint (per-pod queries like `endpoint health`, `move-leader`).
   - `etcdctl_cluster_on <pod> …` — exec inside `<pod>` against all cluster endpoints (cluster-view queries like `endpoint status`). Unreachable endpoints produce stderr warnings + non-zero exit; reachable rows still appear in stdout, so callers swallow the exit code and parse stdout.
   - `pick_live_pod` — walk `-0, -1, …, -(N-1)`, return the first one we can `kubectl exec` into.
   - `get_leader_pod <probe>` — run `etcdctl_cluster_on <probe> endpoint status -w simple`; `-w simple` columns are `endpoint,memberID,version,dbSize,isLeader,isLearner,…`; awk `$5=="true"` to get the leader endpoint, strip to pod name.
   - `member_id_of <probe> <pod>` — `etcdctl member list -w simple` columns are `ID,status,name,peerURLs,clientURLs,isLearner`; awk `$3==<pod>` yields the hex ID. We use hex strings end-to-end because etcd member IDs are random uint64 often exceeding 2⁵³ — JSON+jq parses lose precision.
   - `move_leader_to <target>` — probe for current leader, early-exit if already `<target>`, otherwise `etcdctl_on <leader> move-leader <id>`. `move-leader` RPCs must be issued at the current leader (followers return "not leader"); the helper's first duty is locating that leader. Retries up to 5 times with small sleeps to cover the brief window where the target may still be catching up raft log after a restart.
   - `pod_up_to_date <pod>` — true iff `pod.metadata.labels.controller-revision-hash == TARGET_REV` AND `Ready=True`.
   - `wait_pod_healthy <pod>` — poll `pod_up_to_date` then `etcdctl_on <pod> endpoint health`, both bounded by their respective timeouts. Timeouts are computed inline (`deadline=$(date +%s)+TIMEOUT` + `until ... sleep 2`) rather than via `timeout sh -c "…"`, because POSIX `sh` does not export shell functions into subshells.
4. **Main flow**:
   1. Fast path: if every `${stsName}-${i}` is already `pod_up_to_date`, log and exit 0.
   2. Wait for cluster to have an elected leader (bounded by `readyTimeoutSeconds`).
   3. `move_leader_to ${stsName}-0` — idempotent; no-op if leader already there.
   4. For `i` from `N-1` down to `1`: skip if `pod_up_to_date`, else `set_partition $i` (the sts controller deletes the pod with ordinal `i` and recreates it at the target revision), `wait_pod_healthy`, sleep `cooldownSeconds`.
   5. `move_leader_to ${stsName}-$((N-1))` — now that the new-revision `-(N-1)` is the target.
   6. If `-0` not `pod_up_to_date`: `set_partition 0`, `wait_pod_healthy`.
   7. `set_partition $REPLICAS` — re-lock the sts so any further template change waits for the next Job.

**Correctness argument (zero uncontrolled elections):**

- Step 4.3 makes `-0` the leader (via a deliberate, single leader transfer — counts as one `leader_changes_seen_total` increment if the leader was not already on `-0`). Step 4.4 rolls `-(N-1) … -1`; none is the leader, so no election.
- Step 4.5 transfers leader to `-(N-1)` (already on target revision since it just rolled). Rolling `-0` in 4.6 does not affect the leader.
- Quorum is maintained throughout because `OrderedReady` + partition-by-one ensures only one pod is down at any moment, and N=3 tolerates 1 failure.
- Step 4.7 re-asserts `partition: replicas`. Without this, a stray `pulumi up` that bumps `sts.spec.template` again would cause the sts controller to begin auto-rolling immediately (partition was left at 0), defeating the entire orchestration.

**Idempotency under crash-resume:**

- `pod_up_to_date` is evaluated against live cluster state (`sts.status.updateRevision` + pod labels), not script-local state. Any pod already rolled is skipped on resume regardless of where the previous attempt crashed.
- `move_leader_to` re-discovers the current leader on every call. Whether leader is still on `-0` (good case) or has migrated during the crash window (e.g. transient network issue), the helper issues `move-leader` at the real current leader, not a hard-coded one. This fixes the bug where exec-ing the command at `-0` when `-0` is not leader would silently fail.
- The "wait for leader" step at main 4.2 prevents proceeding past a temporarily leaderless cluster (e.g. mid-election after a third-party restart).

### 4.4 Why `controller-revision-hash` is the source of truth

StatefulSet controller generates a new `ControllerRevision` every time `sts.spec.template` changes and sets `sts.status.updateRevision` to its hash. When the controller creates a pod (whether at first startup or re-creating a deleted pod), it stamps `metadata.labels.controller-revision-hash` with the revision of the template it used. Thus:

- `pod.label.controller-revision-hash == sts.status.updateRevision` ⟺ this pod runs the target template.
- The Job **never writes revisions** and **never deletes pods**. It only patches `partition`; the sts controller is what deletes and re-creates pods with the correct label.
- The Pulumi-computed `versionHash` is **only** used for Job/ConfigMap naming (so the same version's Pulumi apply doesn't create a new Job). Rollout logic reads Kubernetes' own `updateRevision`. The two hashes don't need to agree.

This is also why the rollout's `pod_up_to_date` check uses both `controller-revision-hash` (version match) **and** `Ready=True` (container running): a pod can have the target revision but not yet be serving (just re-created), and without both checks we'd advance prematurely.

### 4.5 Trigger semantics

For each managed sts we need:

1. A Pulumi input that, when changed, produces a new `versionHash`.
2. A `dependsOnSts` reference so Pulumi applies the sts object before creating the Job.

**Main etcd** (`pool/{eks,ack}-freetier/etcd.ts`): the existing `k8s.yaml.ConfigFile` already computes `releaseTag`-derived `nodeSelector`. `versionHash = sha1(JSON.stringify({ image, kubeletVersion }))`. Add a new call site:

```ts
const etcdSts = new k8s.yaml.ConfigFile("etcd", { file: "../manifests/etcd.yaml", transformations: [...] });
new EtcdRollingUpgrade("etcd", {
  stsName: "etcd",
  namespace: ns.metadata.name,
  serviceName: "etcd-headless",
  versionHash: pulumi.output({ image, kubeletVersion }).apply(o => sha1(JSON.stringify(o))),
  dependsOnSts: etcdSts,
}, { provider: c.provider, parent: etcdSts });
```

**keyspace-etcd-group**: applied by this repo in `pool/lib/serverless/keyspace-etcd-group.ts`. `InstallKeyspaceEtcdGroup` currently returns nothing; we change it to return the `sts` handle (plus whatever else callers need), and extend it to instantiate `EtcdRollingUpgrade` once per group. Each group gets its own Job, so K groups produce K Jobs that run in parallel under Pulumi's default resource concurrency.

`versionHash` for keyspace-etcd-group is derived from the inputs that actually flow into the sts template: currently `resources` (through `withStatefulSetResourcesRequirements`), and — once `releaseTag` is also passed through from `keyspaceEtcdGroupSpec.releaseTag` into `InstallKeyspaceEtcdGroup` — the kubelet-version nodeSelector. Today `InstallKeyspaceEtcdGroup` does **not** receive `releaseTag` from `pool/eks-freetier/serverless.ts:1175–1186` even though `KeyspaceEtcdGroupOptions.releaseTag` exists, so keyspace-etcd does not currently pin kubelet-version and does not roll on EKS upgrades. Wiring `releaseTag` through is a small adjacent change required for the rollout component to actually be exercised on EKS version bumps; it is included in the rollout plan (§9) as part of this work.

```ts
// pool/lib/serverless/keyspace-etcd-group.ts
const sts = new k8s.yaml.ConfigFile(`${clusterName}-${name}`, { ... }, { ... });
new EtcdRollingUpgrade(`${clusterName}-${name}`, {
  stsName: name,                                     // e.g. "keyspace-etcd-group-0"
  namespace,
  serviceName: `${name}-peer`,
  tls: {
    caFile: "/var/lib/etcd-tls/ca.crt",
    certFile: "/var/lib/etcd-tls/tls.crt",
    keyFile: "/var/lib/etcd-tls/tls.key",
  },
  versionHash: pulumi.output({ resources, releaseTag }).apply(o => sha1(JSON.stringify(o))),
  dependsOnSts: sts,
}, { provider, parent: sts });
```

### 4.6 StatefulSet manifest changes

Lock both StatefulSets at `partition = replicas` so the sts controller never auto-rolls:

```yaml
# pool/manifests/etcd.yaml
# pool/manifests/serverless/keyspace-etcd-group/sts.yaml
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 3       # equals replicas; no ordinal qualifies for auto-roll
  podManagementPolicy: OrderedReady
  replicas: 3
```

Steady-state invariant: `partition == replicas`. The rollout Job drives upgrades by temporarily decrementing `partition` and then resetting it to `replicas` on exit (success or fast-path); see §4.3 step 4.7.

`partition` and `replicas` are hardcoded together. If `replicas` is ever changed (capacity expansion / contraction), both numbers must change in lockstep; the rollout script reads `REPLICAS` from `sts.spec.replicas` on each run and uses it for the final re-lock, so the script self-corrects, but the manifest itself needs the manual edit.

Two Pulumi-side knobs make this work end-to-end:

1. **`pulumi.com/skipAwait: "true"` annotation on the StatefulSet.** Each `ConfigFile` transformation injects:

   ```ts
   if (obj.kind === "StatefulSet") {
     obj.metadata.annotations = obj.metadata.annotations ?? {};
     obj.metadata.annotations["pulumi.com/skipAwait"] = "true";
   }
   ```

   The K8s provider's default sts await waits for `status.currentRevision == status.updateRevision` + ready replicas. Under `partition = replicas`, the sts controller intentionally refuses to advance `currentRevision` — that's the whole lock — so the default await would never complete, and the rollout Job (which `dependsOn` the sts) would never be created. `skipAwait` declares "the K8s state convergence I care about isn't on this object; it's on the Job that follows." Pulumi returns from the sts patch immediately, creates the Job, then awaits the Job's `Complete` condition (default for `batch/v1.Job`), which is the real "rollout finished" signal.

2. **No `ignoreChanges` on `partition`.** The manifest's `partition: 3` is the canonical desired state, and Pulumi reconciles it on every apply. The Job's transient writes (3 → 2 → 1 → 0 → 3) sit between two Pulumi reconciles and the Job's final `set_partition $REPLICAS` step (plus its EXIT/INT/TERM trap) ensures the lock is re-asserted before the Job exits — so a healthy Pulumi run sees `partition = 3` both before patching and at the end of the Job.

   We initially tried `ignoreChanges: ["spec.updateStrategy.rollingUpdate.partition"]` to bulletproof against concurrent writes during a mid-flight Job; that design was abandoned during PR #7593's first dev apply when we discovered `ignoreChanges` does *not* exempt the *initial* write — Pulumi sees the manifest input differs from live (`3 != 0` on first apply, because the field defaults to 0), classifies it as a diff, and `ignoreChanges` skips the diff entirely. The net effect was that `partition` stayed at the K8s-default `0` after Pulumi's patch, so the sts controller started auto-rolling pods the instant template changed — exactly the uncontrolled rolling we were trying to prevent. The fix: rely on the Job's own re-lock on every exit path (see §4.3 step 4.7 and the EXIT/INT/TERM trap) rather than `ignoreChanges`. The remaining race window — operator runs `pulumi up` *while* a Job is mid-roll — is narrow, recoverable (Pulumi's patch competes with the Job's `set_partition`; whoever writes last wins, and the Job's final re-lock pulls the SS back to the invariant), and not part of the supported workflow.

`podManagementPolicy: OrderedReady` is preserved. Under partition-driven RollingUpdate, the policy guarantees the sts controller waits for each rolled pod to become `Ready` before considering the next ordinal — exactly what we need between `set_partition $i` and `set_partition $((i-1))`.

No readinessProbe is added to the sts. The rollout script's `etcdctl endpoint health` check substitutes for it in upgrade ordering decisions. Adding one would be a good hygiene change but is out of scope and doesn't affect correctness of this design.

## 5. Greenfield Behavior (First-Time Cluster Creation)

When a cluster is stood up for the first time, Pulumi applies the sts and the rollout Job in the same `pulumi up`. Timeline:

1. `t0` — Pulumi creates sts. K8s sts controller generates the first `ControllerRevision`; `updateRevision == currentRevision == <hash-A>`.
2. `t1` — Pulumi creates Job (ServiceAccount + ConfigMaps first, then Job). `dependsOnSts` orders this after sts, but "after" only means the sts object exists, not that its pods are Ready.
3. `t2` — Job pod starts, runs `rollout.sh`:
   - Reads `TARGET_REV = hash-A`.
   - Runs the fast-path loop. If all pods already have label `controller-revision-hash=hash-A` and are Ready, exits 0 immediately. (Possible if sts pods came up fast enough.)
   - Otherwise, waits up to `READY_TIMEOUT=300s` for `etcdctl_on ${STS}-0 endpoint health` to succeed. Since etcd bootstrap needs all 3 members to form initial quorum, this waits until `etcd-0`, `etcd-1`, `etcd-2` all join. 3-replica bootstrap on warm images typically completes in 20–60s.
   - Once healthy, `move-leader` to `etcd-0` (already the leader usually, since the lowest-ordinal member often wins first election; `|| true` on idempotence).
   - Walks the roll loop: every `pod_up_to_date` check succeeds (all at `hash-A`), every iteration hits `continue`. Moves leader to `etcd-(N-1)` (real move, harmless). Checks `etcd-0`, skips.
   - Exits 0.
4. Pulumi sees Job `Complete`, finishes `pulumi up` successfully.

Total greenfield overhead is one extra Job run with essentially zero work done (plus two leader moves on a healthy cluster). Acceptable.

## 6. Idempotency & Resumability

- **Same-version re-apply**: `versionHash` unchanged → Job name unchanged → Pulumi sees the Job resource already exists → no-op. Even if Pulumi *did* re-trigger it, `rollout.sh` fast-path exits immediately.
- **Mid-flight failure**: Job fails at pod `etcd-1` after successfully rolling `etcd-2`. Next `pulumi up` keeps the same `versionHash`; Job already exists → no new Job. Human intervention: either manually delete the failed Job (triggering Pulumi to re-create it) or inspect and continue manually. When Job re-runs: `pod_up_to_date(etcd-2)` = true (skip), `pod_up_to_date(etcd-1)` = false → continue.
- **Interrupted during cool-down**: sleep is interrupted, Job fails. Resumption: previous pod is already at target revision, loop skips it, continues with the next.
- **Leader moved away from `etcd-0` between step 1 and the loop**: `move_leader_to` re-discovers the current leader on every call (see §4.3 — helper issues `move-leader` at the real leader, not hard-coded `etcd-0`). If leader drifted to some pod during the crash window, the next run still transfers it back to `-0` before proceeding. No silent election caused by wrong-target `move-leader`.

## 7. Error Handling & Observability

- **Job failure**: `backoffLimit: 0`, so a single script non-zero exit fails the Job permanently. `ttlSecondsAfterFinished: 604800` keeps the Job + its pod logs for a week. Pulumi surfaces this as a failed `pulumi up`.
- **Logs**: `rollout.sh` uses `set -eu` and echoes each step. `kubectl logs job/<job-name>` shows full timeline. No structured logs / metrics in v1.
- **Timeouts**:
  - `READY_TIMEOUT=300s` per pod (sts controller to re-create, pod to report Ready, revision label to propagate).
  - `HEALTH_TIMEOUT=60s` per pod (etcd to answer `endpoint health`).
  - Cluster bootstrap wait also uses `READY_TIMEOUT`.
- **No automatic rollback**. If a new etcd image is broken, the rollout halts mid-flight, leaving the cluster with mixed versions. Human operator decides whether to revert the Pulumi input (triggers a fresh rollout to the previous version) or patch forward.

### 7.1 Runbook: Manual re-run of a failed rollout Job

A failed Job lingers in the cluster for 7 days (`ttlSecondsAfterFinished: 604800`). Because `backoffLimit: 0`, k8s does not retry it automatically, and because the Job's name embeds the target `versionHash`, Pulumi treats "Job exists with right name" as a no-op and will not re-run it on subsequent `pulumi up` either. The script is idempotent, so the recovery procedure is: dump the Job yaml, delete the failed object, re-apply. Steps:

```sh
NS=tidb-admin              # or tidb-serverless for keyspace-etcd-group-*
JOB=etcd-rollout-<hash8>   # fill in the real hash, e.g. from `kubectl -n $NS get jobs`

# 1) Diagnose.
kubectl -n $NS logs job/$JOB
kubectl -n $NS describe job $JOB

# 2) Export a clean yaml. The preferred tool is kubectl-neat (strips server-injected
#    fields and controller-uid labels). Fall back to yq if neat is unavailable.
kubectl -n $NS get job $JOB -o yaml | kubectl neat > /tmp/$JOB.yaml
#   -- or --
kubectl -n $NS get job $JOB -o yaml \
  | yq 'del(
      .metadata.resourceVersion, .metadata.uid, .metadata.generation,
      .metadata.creationTimestamp, .metadata.managedFields, .status,
      .spec.selector,
      .spec.template.metadata.labels["batch.kubernetes.io/controller-uid"],
      .spec.template.metadata.labels["controller-uid"],
      .spec.template.metadata.labels["batch.kubernetes.io/job-name"],
      .spec.template.metadata.labels["job-name"]
    )' > /tmp/$JOB.yaml

# 3) Delete the failed Job (cascade=background removes its failed Pod).
kubectl -n $NS delete job $JOB

# 4) Re-apply. The script's fast-path + controller-revision-hash checks skip
#    whichever pods already rolled; work continues from the interrupted pod.
kubectl -n $NS apply -f /tmp/$JOB.yaml
kubectl -n $NS logs -f job/$JOB
```

After the Job completes, consider running `pulumi refresh` in the owning stack to re-align Pulumi state with the manually re-applied object. Without `refresh`, Pulumi may surface a drift on the next `preview`.

Failure modes that this procedure does *not* fix:

- **Broken image / config**: pods crash-loop at the new revision, `wait_pod_healthy` times out, Job fails again. Resolution: revert the Pulumi input (e.g. `releaseTag`) to the last-good value and `pulumi up`; a new Job targets the rollback and the script pulls any stuck pod back to the old revision.
- **etcd quorum lost**: the cluster has <2/3 members healthy. The Job's "wait for leader" step fails. Resolution: restore quorum manually before re-running (out of scope for this component).

## 8. Security

- **RBAC scope**: `Role` + `RoleBinding`, **namespace-scoped only**. No ClusterRole. Verbs listed in §4.1 are the minimum needed.
- **`pods/exec`**: grants full shell inside etcd containers. This is equivalent to what a cluster admin already has; the Role is pinned to the one namespace, and the ServiceAccount is used only by this Job. No long-lived token is issued to users.
- **etcdctl auth for TLS etcd**: `kubectl exec` runs inside the pod, reusing the certs already mounted at `/var/lib/etcd-tls/`. The rollout Job itself never handles cert material.

## 9. Rollout Plan

1. **Land the `EtcdRollingUpgrade` component** (`pool/lib/rolling-upgrade/{index.ts, rollout.sh, rbac.ts}`) with full tests — unit tests for `versionHash` computation, integration test that runs the Job against a kind cluster with the real etcd manifest.
2. **Add `updateStrategy: RollingUpdate` with `partition: replicas` + `pulumi.com/skipAwait` annotation + invoke component** for the main `etcd` sts in one `eks-freetier` dev stack (e.g. `dev-us-east-1-f01`). Manually bump an etcd-affecting input (e.g. `releaseTag`) to exercise the Job on a dev cluster. Verify: zero leader elections in etcd metrics, Job logs clean, partition returns to `replicas` after the Job exits, cluster stays healthy.
3. **Repeat for ack-freetier dev** stack.
4. **Roll to staging / prod stacks** in the usual EKS-upgrade order, each gated by §9 step 2's success.
5. **Wire `releaseTag` into `InstallKeyspaceEtcdGroup`** (small adjacent change — `pool/eks-freetier/serverless.ts:1175` currently omits `releaseTag` from the options object; add it from `keyspaceEtcdGroupSpec.releaseTag`). Without this, keyspace-etcd doesn't pin kubelet-version and doesn't roll on EKS upgrades, so the rollout component would never fire on that sts.
6. **Instantiate `EtcdRollingUpgrade` inside `InstallKeyspaceEtcdGroup`**, rolling behind each group's own `versionHash`. K groups → K independent Jobs running in parallel, each managing its own 3-pod cluster.

Deprecation of the old "implicit k8s rolling update" path happens by virtue of `partition: replicas` — the sts controller has nothing to roll until the Job lowers the partition.

## 10. Alternatives Considered

### 10.1 `updateStrategy: OnDelete` (the original v1 design, abandoned)

Approach: switch the sts to `updateStrategy: { type: OnDelete }`, have the Job orchestrate by `kubectl delete pod` directly. Initially chosen for being declaratively simple — "Pulumi writes the strategy once, the Job is the only actor that deletes pods."

Abandoned during PR #7593 CI when the very first `pulumi preview` against existing clusters revealed an unrecoverable K8s API validation error:

```
StatefulSet.apps "etcd" is invalid: spec.updateStrategy.rollingUpdate:
Invalid value: {Partition:0, MaxUnavailable:nil}:
only allowed for updateStrategy 'RollingUpdate'
```

Root cause: every existing etcd StatefulSet in the fleet was created under default `RollingUpdate` strategy. The K8s API server auto-populates `spec.updateStrategy.rollingUpdate: {partition: 0}` when the strategy is RollingUpdate. With `PULUMI_K8S_ENABLE_SERVER_SIDE_APPLY: false` (the workflow default), Pulumi's client-side three-way merge does not delete fields absent from the new manifest if they exist in live state. So Pulumi sends a patch that *adds* `type: OnDelete` but leaves the existing `rollingUpdate: {partition: 0}` block intact. K8s validates the merged result and rejects: rollingUpdate is only allowed alongside `type: RollingUpdate`.

Attempted in-band fixes that did not work:

- **`rollingUpdate: null` in the YAML manifest.** Parsed by Pulumi's YAML loader and dropped before constructing the resource diff — the resulting patch is unchanged.
- **`command:local:Command` that does `kubectl patch ... rollingUpdate: null` as a pre-step.** Local Command resources only run during `pulumi up`, not `pulumi preview`. CI preview still hits the validation error and fails red.
- **Server-side apply per-resource annotation.** Pulumi's k8s provider supports SSA only at the provider level (`PULUMI_K8S_ENABLE_SERVER_SIDE_APPLY=true` env or `enableServerSideApply: true` on a constructed provider), not via a per-resource annotation. Switching the whole stack to SSA or building a dedicated SSA provider for just the etcd resources is a much larger refactor with its own ownership-migration risk.

The only deployable fix was operational: per-cluster manual `kubectl patch sts ... --type=merge -p '{"spec":{"updateStrategy":{"type":"OnDelete","rollingUpdate":null}}}'` before every cluster's first apply of this change. With ~dozens of clusters across dev/staging/prod (each ack/eks stack has 1× `tidb-admin/etcd` + 3× `keyspace-etcd-group`), the operational toil and the inability to make preview green made OnDelete infeasible.

Partition-based RollingUpdate avoids all of this: the strategy stays at its original `RollingUpdate` (no K8s validation conflict), and the Job drives rolls via a field K8s already accepts. The four original objections to partition (contested field, "confusing permanent partition: 3", mid-flight partition value, "no capability over OnDelete") all have mechanical answers:

- **Contested field**: handled by serializing the only two writers. Pulumi writes `partition = replicas` exactly once per `pulumi up`, at the start of the SS patch. The Job then drives `replicas → ... → 0 → replicas` and is the only writer until it exits. Pulumi's await on the sts is disabled (`pulumi.com/skipAwait: "true"`), so Pulumi proceeds to await the *Job's* `Complete` condition instead — `pulumi up` does not return until the Job has reset `partition` to `replicas`. We initially tried `ignoreChanges` to make this race-free; it backfired (see §4.6) because `ignoreChanges` also skips the *initial* write from K8s-default `0` to manifest's `replicas`.
- **"Confusing permanent partition: 3"**: re-framed as an *invariant*. `partition == replicas` in steady state means "the sts is locked from auto-rolling unless the Job acts." Reading the manifest literally describes the intended behavior.
- **Mid-flight partition value**: handled by `set_partition $REPLICAS` on every script exit path (fast-path early return, normal end-of-run, and an EXIT/INT/TERM trap that catches the failure paths the explicit calls miss). A failed Job mid-flight leaves partition at some `i < replicas`; the trap fires before the container exits and re-asserts `partition = replicas`. The next Job re-asserts again on its own exit, so the post-recovery state is invariant.
- **"No capability over OnDelete"**: correct, and that is exactly why partition is acceptable now — given OnDelete is blocked by the validation issue above, partition gives the same capability *and* is deployable.

### 10.2 Dedicated etcd operator

Use an existing operator (e.g. etcd-operator, etcd-druid). Rejected because:

- Heavy: introduces a new controller, its RBAC footprint, its own CRDs, an ongoing upgrade/maintenance burden.
- Not aligned with how the rest of this repo works (Pulumi + raw manifests; no operators for other components like TiDB's own tooling is via tidb-operator but that's a separate layer).
- The orchestration we need is ~100 lines of shell; operator is disproportionate.

### 10.3 Imperative runbook / human operator

Script lives in a wiki; human runs it before each EKS upgrade. Rejected because:

- Exactly the status quo of "not smooth upgrades" — humans forget leader moves, cool-downs, or trigger the rollout concurrent with other pulumi changes.
- Zero audit trail compared to Job logs retained for a week.

### 10.4 Standalone Go controller

A custom controller that watches the sts and does the rollout. Rejected for v1 because:

- Controller is a long-lived resource with its own deployment, HA, image-build pipeline.
- A Job fires only when needed, carries its own version identity, logs to one place, and dies.
- If future requirements demand event-driven rollout (e.g. auto-retry, multi-cluster coordination), we can graduate the shell script to a controller without touching the ConfigMap schema.

## 11. Decisions Locked In

- **Orchestrator image**: v1 pulls `alpine/k8s:1.34.x` directly from Docker Hub. It is a low-frequency pull (once per rollout Job, per sts, per version) so the public-registry dependency is acceptable. If pull rate limiting or airgapped stacks force a change later, mirroring to an internal registry is a one-line image-reference swap.
- **keyspace-etcd-group integration**: `InstallKeyspaceEtcdGroup` owns the sts today (see §4.5); the rollout component is instantiated inside it, one per group. No ownership change.
- **Multi-group concurrency**: Jobs run in parallel. `K` groups → `K` independent Jobs, each managing its own 3-pod cluster, no cross-group serialization. Each group's raft is independent, so parallel upgrades cannot cause cross-group quorum loss. Node-capacity risk during concurrent EKS drain is accepted: critical-services nodegroup already has `safe-to-evict: false` on etcd pods, so drain blocks until each group's rollout individually completes.
- **Cool-down tunability**: `cooldownSeconds` is already a `EtcdRollingUpgradeArgs` field (default 30). v1 hardcodes the default at both call sites; a future Pulumi config passthrough is a trivial plumb-through when needed.