# Etcd Rolling Upgrade Orchestration Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

> **⚠ Plan is superseded for the StatefulSet manifest + script tasks.** During PR #7593 CI, the chosen `updateStrategy: OnDelete` strategy was blocked by a K8s API validation error on every existing cluster (live state's auto-populated `rollingUpdate` block conflicts with `OnDelete`, and Pulumi's client-side merge cannot remove it). The design pivoted to `updateStrategy: RollingUpdate` with `partition: replicas` as the lock, decremented by the rollout Job. See the **updated spec** at `docs/superpowers/specs/2026-04-23-etcd-rolling-upgrade-design.md` (§4.6 manifest changes, §10.1 alternatives) for the current approach. Task 1, Task 2, and the `kubectl delete pod` snippets in Task 3 below are out of date and should be interpreted via the spec.

**Goal:** Ship a reusable Pulumi component `EtcdRollingUpgrade` that takes over rolling-upgrade choreography for `etcd` (HTTP, `tidb-admin`) and `keyspace-etcd-group-*` (TLS, `tidb-serverless`) StatefulSets, producing zero raft leader elections during EKS version upgrades.

**Architecture (current, post-pivot):** `updateStrategy: RollingUpdate` with `rollingUpdate.partition: replicas` on both StatefulSets, plus `ignoreChanges` on the partition field so Pulumi does not fight the Job. Per-sts Pulumi `ComponentResource` that creates a `ServiceAccount`/`Role`/`RoleBinding`, per-version `ConfigMap` with rollout parameters, shared `ConfigMap` with a `rollout.sh` orchestrator script, and a `Job` that executes the script. The Job pattern runs leader-move → decrement partition high→low → leader-move → set partition to 0 → reset partition to replicas, with `controller-revision-hash`-based idempotency.

**Tech Stack:** Pulumi TypeScript (Kubernetes provider), `k8s.yaml.ConfigFile` manifests, POSIX shell + `kubectl` + `jq` + `yq` running in `alpine/k8s:1.34.x` on the cluster.

---

## Reference Documents

- Spec: `docs/superpowers/specs/2026-04-23-etcd-rolling-upgrade-design.md`
- Main etcd manifest: `pool/manifests/etcd.yaml`
- Keyspace etcd manifest: `pool/manifests/serverless/keyspace-etcd-group/sts.yaml`
- Main etcd installers: `pool/eks-freetier/etcd.ts`, `pool/ack-freetier/etcd.ts`
- Keyspace etcd installer: `pool/lib/serverless/keyspace-etcd-group.ts`
- Keyspace etcd caller: `pool/eks-freetier/serverless.ts:1167–1186`

## File Map

| Path | Action | Responsibility |
| --- | --- | --- |
| `pool/manifests/etcd.yaml` | Modify | Add `updateStrategy: { type: OnDelete }` to `StatefulSet.spec` |
| `pool/manifests/serverless/keyspace-etcd-group/sts.yaml` | Modify | Same as above |
| `pool/lib/rolling-upgrade/rollout.sh` | Create | Generic orchestrator script; reads `/config/config.yaml`, drives pod deletion order, leader moves, health checks |
| `pool/lib/rolling-upgrade/rbac.ts` | Create | Helper to build `ServiceAccount` + `Role` + `RoleBinding` scoped to a namespace with the minimum verbs |
| `pool/lib/rolling-upgrade/index.ts` | Create | `EtcdRollingUpgrade` `ComponentResource`; builds per-instance ConfigMap + Job + RBAC; re-exports `EtcdTLSConfig` + `EtcdRollingUpgradeArgs` |
| `pool/eks-freetier/etcd.ts` | Modify | Compute `versionHash`, instantiate `EtcdRollingUpgrade` for main etcd |
| `pool/ack-freetier/etcd.ts` | Modify | Same wiring as EKS, ACK-flavored provider |
| `pool/lib/serverless/keyspace-etcd-group.ts` | Modify | Pass `releaseTag` through the sts transformation (pre-existing bug) and instantiate `EtcdRollingUpgrade` with TLS config |
| `pool/eks-freetier/serverless.ts:1175` | Modify | Plumb `keyspaceEtcdGroupSpec.releaseTag` into `InstallKeyspaceEtcdGroup` call |

## Testing Note

This repo has no unit test harness — no `*.test.ts`/`*.spec.ts` files and no test runner in either stack's `package.json`. Per-task verification uses tools already available to a Pulumi TS project:

- `tsc --noEmit` (typecheck) inside the stack directory
- `shellcheck` on `rollout.sh`
- `pulumi preview` on a sandbox stack (see Task 11) for cross-cutting validation before deploying

The final acceptance check (Task 12) runs an actual upgrade against a dev cluster and reads etcd metrics for leader-election counters.

---

## Task 1: Add `updateStrategy: OnDelete` to main etcd manifest

**Files:**
- Modify: `pool/manifests/etcd.yaml:65-67`

- [ ] **Step 1: Inspect the existing sts spec block**

Run: `sed -n '58,72p' pool/manifests/etcd.yaml`

Expected: lines show `apiVersion: apps/v1`, `kind: StatefulSet`, `spec:` with `podManagementPolicy: OrderedReady`, `replicas: 3`, `selector:` …

- [ ] **Step 2: Insert `updateStrategy: { type: OnDelete }` above `podManagementPolicy`**

Change `pool/manifests/etcd.yaml` so the sts spec reads:

```yaml
spec:
  updateStrategy:
    type: OnDelete
  podManagementPolicy: OrderedReady
  replicas: 3
```

- [ ] **Step 3: Verify the YAML parses**

Run: `python3 -c "import yaml,sys; list(yaml.safe_load_all(open('pool/manifests/etcd.yaml')))" && echo OK`

Expected: `OK`

- [ ] **Step 4: Commit**

```bash
git add pool/manifests/etcd.yaml
git commit -m "etcd: switch main etcd sts to updateStrategy: OnDelete"
```

---

## Task 2: Add `updateStrategy: OnDelete` to keyspace-etcd-group manifest

**Files:**
- Modify: `pool/manifests/serverless/keyspace-etcd-group/sts.yaml:8-11`

- [ ] **Step 1: Inspect the spec header**

Run: `sed -n '1,15p' pool/manifests/serverless/keyspace-etcd-group/sts.yaml`

Expected: shows `spec:` with `podManagementPolicy: OrderedReady`, `replicas: 3`, `selector:`.

- [ ] **Step 2: Insert `updateStrategy: { type: OnDelete }`**

Change so spec reads:

```yaml
spec:
  updateStrategy:
    type: OnDelete
  podManagementPolicy: OrderedReady
  replicas: 3
```

- [ ] **Step 3: Verify parses**

Run: `python3 -c "import yaml,sys; list(yaml.safe_load_all(open('pool/manifests/serverless/keyspace-etcd-group/sts.yaml')))" && echo OK`

Expected: `OK`

- [ ] **Step 4: Commit**

```bash
git add pool/manifests/serverless/keyspace-etcd-group/sts.yaml
git commit -m "keyspace-etcd-group: switch sts to updateStrategy: OnDelete"
```

---

## Task 3: Write the orchestrator shell script

**Files:**
- Create: `pool/lib/rolling-upgrade/rollout.sh`

- [ ] **Step 1: Create the directory**

Run: `mkdir -p pool/lib/rolling-upgrade`

- [ ] **Step 2: Write `rollout.sh`**

Create `pool/lib/rolling-upgrade/rollout.sh` with these exact contents:

```sh
#!/bin/sh
# Drives an etcd StatefulSet rolling upgrade with zero uncontrolled elections.
# Reads config from /config/config.yaml (mounted ConfigMap).
# Orchestration: move leader to -0, roll (N-1)..1, move leader to -(N-1), roll -0.
# Idempotent + crash-resumable: pods at sts.status.updateRevision are skipped.
# Written to run under POSIX ash (alpine/k8s image uses busybox sh).

set -eu

CFG=/config/config.yaml
DIAL_TO="3s"

STS=$(yq         '.stsName'                   "$CFG")
NS=$(yq          '.namespace'                 "$CFG")
SVC=$(yq         '.serviceName'               "$CFG")
PORT=$(yq        '.clientPort // 2379'        "$CFG")
COOLDOWN=$(yq    '.cooldownSeconds // 30'     "$CFG")
READY_TO=$(yq    '.readyTimeoutSeconds // 300' "$CFG")
HEALTH_TO=$(yq   '.healthTimeoutSeconds // 60' "$CFG")

TLS_ON=$(yq '.tls.enabled // false' "$CFG")
SCHEME=http
ETCDCTL_TLS=""
if [ "$TLS_ON" = "true" ]; then
  SCHEME=https
  CA=$(yq   '.tls.caFile'   "$CFG")
  CERT=$(yq '.tls.certFile' "$CFG")
  KEY=$(yq  '.tls.keyFile'  "$CFG")
  ETCDCTL_TLS="--cacert=$CA --cert=$CERT --key=$KEY"
fi

REPLICAS=$(kubectl -n "$NS" get sts "$STS" -o jsonpath='{.spec.replicas}')
TARGET_REV=$(kubectl -n "$NS" get sts "$STS" -o jsonpath='{.status.updateRevision}')
if [ -z "$REPLICAS" ] || [ -z "$TARGET_REV" ]; then
  echo "FATAL: could not read replicas or updateRevision from sts $STS" >&2
  exit 1
fi

# Build cluster-wide endpoints list once.
ENDPOINTS=""
i=0
while [ "$i" -lt "$REPLICAS" ]; do
  ep="${SCHEME}://${STS}-$i.${SVC}.${NS}:${PORT}"
  if [ -z "$ENDPOINTS" ]; then ENDPOINTS="$ep"; else ENDPOINTS="$ENDPOINTS,$ep"; fi
  i=$((i + 1))
done

ep_of() { echo "${SCHEME}://$1.${SVC}.${NS}:${PORT}"; }

# Run etcdctl inside <pod>, targeting that pod's own local endpoint (per-pod commands).
etcdctl_on() {
  pod=$1; shift
  kubectl -n "$NS" exec "$pod" -c etcd -- \
    sh -c "etcdctl --endpoints=$(ep_of "$pod") --dial-timeout=$DIAL_TO $ETCDCTL_TLS $*"
}

# Run etcdctl inside <pod>, targeting all cluster endpoints (cluster-view queries).
etcdctl_cluster_on() {
  pod=$1; shift
  kubectl -n "$NS" exec "$pod" -c etcd -- \
    sh -c "etcdctl --endpoints=$ENDPOINTS --dial-timeout=$DIAL_TO $ETCDCTL_TLS $*"
}

# Find a pod we can exec into. Tries -0, -1, ..., -(N-1). Prints name or fails.
pick_live_pod() {
  j=0
  while [ "$j" -lt "$REPLICAS" ]; do
    p="${STS}-$j"
    if kubectl -n "$NS" exec "$p" -c etcd -- true >/dev/null 2>&1; then
      echo "$p"
      return 0
    fi
    j=$((j + 1))
  done
  return 1
}

# Return the pod name of the current raft leader, or empty string if undetermined.
# endpoint status -w simple columns: endpoint,memberID,version,dbSize,isLeader,isLearner,...
# Unreachable endpoints produce stderr warnings + non-zero exit, but reachable rows
# still appear on stdout — so we swallow the exit code and parse stdout.
get_leader_pod() {
  probe=$1
  out=$(etcdctl_cluster_on "$probe" endpoint status -w simple 2>/dev/null || true)
  echo "$out" | awk -F', ' '$5=="true" { print $1; exit }' \
    | sed -E 's|^https?://([^.]+)\..*$|\1|'
}

# Hex member ID for a named pod, or empty on failure.
# member list -w simple columns: ID,status,name,peerURLs,clientURLs,isLearner
member_id_of() {
  probe=$1; target=$2
  kubectl -n "$NS" exec "$probe" -c etcd -- \
    sh -c "etcdctl --dial-timeout=$DIAL_TO $ETCDCTL_TLS member list -w simple" 2>/dev/null \
    | awk -F', ' -v n="$target" '$3==n { print $1 }'
}

# Move raft leader to $1. No-op if already there.
# Retries a few times to cover the short window after target restart where log is still catching up.
move_leader_to() {
  target=$1
  attempt=1
  while [ "$attempt" -le 5 ]; do
    probe=$(pick_live_pod) || { echo "[rollout] ERROR: no live pod to probe" >&2; return 1; }
    leader=$(get_leader_pod "$probe")
    if [ -z "$leader" ]; then
      echo "[rollout] attempt $attempt: could not locate leader via $probe; retrying" >&2
      sleep 3; attempt=$((attempt + 1)); continue
    fi
    if [ "$leader" = "$target" ]; then
      echo "[rollout] leader already on $target"
      return 0
    fi
    id=$(member_id_of "$probe" "$target")
    if [ -z "$id" ]; then
      echo "[rollout] attempt $attempt: no member id for $target; retrying" >&2
      sleep 3; attempt=$((attempt + 1)); continue
    fi
    echo "[rollout] attempt $attempt: transfer leader $leader -> $target (id=$id)"
    if etcdctl_on "$leader" move-leader "$id"; then return 0; fi
    echo "[rollout] attempt $attempt: move-leader failed; retrying" >&2
    sleep 5; attempt=$((attempt + 1))
  done
  echo "[rollout] ERROR: gave up moving leader to $target" >&2
  return 1
}

pod_up_to_date() {
  pod=$1
  kubectl -n "$NS" get pod "$pod" >/dev/null 2>&1 || return 1
  rev=$(kubectl -n "$NS" get pod "$pod" \
    -o jsonpath='{.metadata.labels.controller-revision-hash}' 2>/dev/null || echo "")
  ready=$(kubectl -n "$NS" get pod "$pod" \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "")
  [ "$rev" = "$TARGET_REV" ] && [ "$ready" = "True" ]
}

# Wait for <pod> to reach target revision + Ready, then pass etcdctl endpoint health.
# Timeouts are inline (not via `timeout` + `sh -c`) because POSIX sh does not export functions.
wait_pod_healthy() {
  pod=$1
  deadline=$(( $(date +%s) + READY_TO ))
  until pod_up_to_date "$pod"; do
    if [ "$(date +%s)" -ge "$deadline" ]; then
      echo "[rollout] ERROR: timed out waiting for $pod to reach $TARGET_REV + Ready" >&2
      return 1
    fi
    sleep 2
  done
  deadline=$(( $(date +%s) + HEALTH_TO ))
  until etcdctl_on "$pod" endpoint health >/dev/null 2>&1; do
    if [ "$(date +%s)" -ge "$deadline" ]; then
      echo "[rollout] ERROR: timed out waiting for $pod etcd health" >&2
      return 1
    fi
    sleep 2
  done
}

echo "[rollout] sts=$STS ns=$NS replicas=$REPLICAS target_rev=$TARGET_REV tls=$TLS_ON"

# ---- fast path: everything already at target revision ----
all_done=true
i=0
while [ "$i" -lt "$REPLICAS" ]; do
  if ! pod_up_to_date "${STS}-$i"; then all_done=false; break; fi
  i=$((i + 1))
done
if $all_done; then
  echo "[rollout] all pods already at $TARGET_REV; nothing to do"
  exit 0
fi

# ---- wait until cluster has a leader (greenfield / post-restart safety) ----
echo "[rollout] waiting for cluster to elect a leader"
probe=$(pick_live_pod) || { echo "[rollout] FATAL: no live pod to probe" >&2; exit 1; }
deadline=$(( $(date +%s) + READY_TO ))
while [ -z "$(get_leader_pod "$probe")" ]; do
  if [ "$(date +%s)" -ge "$deadline" ]; then
    echo "[rollout] FATAL: cluster has no leader after ${READY_TO}s" >&2
    exit 1
  fi
  sleep 2
  probe=$(pick_live_pod) || { echo "[rollout] FATAL: no live pod to probe" >&2; exit 1; }
done

# ---- 1) move leader to -0 ----
move_leader_to "${STS}-0"

# ---- 2) roll (N-1) .. 1 ----
i=$((REPLICAS - 1))
while [ "$i" -ge 1 ]; do
  pod="${STS}-$i"
  if pod_up_to_date "$pod"; then
    echo "[rollout] skip $pod (already $TARGET_REV)"
    i=$((i - 1))
    continue
  fi
  echo "[rollout] deleting $pod"
  kubectl -n "$NS" delete pod "$pod" --wait=false
  wait_pod_healthy "$pod"
  echo "[rollout] $pod healthy; cooldown ${COOLDOWN}s"
  sleep "$COOLDOWN"
  i=$((i - 1))
done

# ---- 3) move leader to -(N-1), then roll -0 ----
move_leader_to "${STS}-$((REPLICAS - 1))"

pod="${STS}-0"
if ! pod_up_to_date "$pod"; then
  echo "[rollout] deleting $pod"
  kubectl -n "$NS" delete pod "$pod" --wait=false
  wait_pod_healthy "$pod"
fi

echo "[rollout] done"
```

- [ ] **Step 3: Run `shellcheck`**

Run: `shellcheck pool/lib/rolling-upgrade/rollout.sh`

Expected: no output (exit 0). If `shellcheck` is not installed locally, run `docker run --rm -v "$PWD":/mnt koalaman/shellcheck:stable /mnt/pool/lib/rolling-upgrade/rollout.sh`.

- [ ] **Step 4: Commit**

```bash
git add pool/lib/rolling-upgrade/rollout.sh
git commit -m "rolling-upgrade: add orchestrator shell script"
```

---

## Task 4: Write RBAC helper

**Files:**
- Create: `pool/lib/rolling-upgrade/rbac.ts`

- [ ] **Step 1: Write `rbac.ts`**

Create `pool/lib/rolling-upgrade/rbac.ts`:

```ts
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

export interface RolloutRBACArgs {
  namespace: pulumi.Input<string>;
  stsName: string;
}

export interface RolloutRBAC {
  serviceAccount: k8s.core.v1.ServiceAccount;
  role: k8s.rbac.v1.Role;
  roleBinding: k8s.rbac.v1.RoleBinding;
}

export function createRolloutRBAC(
  name: string,
  args: RolloutRBACArgs,
  opts: pulumi.ComponentResourceOptions
): RolloutRBAC {
  const saName = `${args.stsName}-rollout`;

  const serviceAccount = new k8s.core.v1.ServiceAccount(
    `${name}-sa`,
    {
      metadata: { name: saName, namespace: args.namespace },
    },
    opts
  );

  const role = new k8s.rbac.v1.Role(
    `${name}-role`,
    {
      metadata: { name: saName, namespace: args.namespace },
      rules: [
        {
          apiGroups: ["apps"],
          resources: ["statefulsets"],
          verbs: ["get", "list", "watch", "patch"],
        },
        {
          apiGroups: [""],
          resources: ["pods"],
          verbs: ["get", "list", "watch", "delete"],
        },
        {
          apiGroups: [""],
          resources: ["pods/exec"],
          verbs: ["create", "get"],
        },
      ],
    },
    opts
  );

  const roleBinding = new k8s.rbac.v1.RoleBinding(
    `${name}-rb`,
    {
      metadata: { name: saName, namespace: args.namespace },
      subjects: [{ kind: "ServiceAccount", name: saName, namespace: args.namespace }],
      roleRef: { kind: "Role", name: saName, apiGroup: "rbac.authorization.k8s.io" },
    },
    { ...opts, dependsOn: [role, serviceAccount] }
  );

  return { serviceAccount, role, roleBinding };
}
```

- [ ] **Step 2: Typecheck inside eks-freetier stack**

The lib directory is shared across stacks; typecheck by running inside the stack that imports it (we'll add the import in Task 6, so for now just check the file itself parses by running the stack's typecheck over the whole tree).

Run:
```bash
cd pool/eks-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors. If the stack's existing code has pre-existing errors (unlikely — CI runs tsc), note them but don't fix unrelated ones.

- [ ] **Step 3: Commit**

```bash
git add pool/lib/rolling-upgrade/rbac.ts
git commit -m "rolling-upgrade: add namespace-scoped RBAC helper"
```

---

## Task 5: Write `EtcdRollingUpgrade` ComponentResource

**Files:**
- Create: `pool/lib/rolling-upgrade/index.ts`

- [ ] **Step 1: Write `index.ts`**

Create `pool/lib/rolling-upgrade/index.ts`:

```ts
import * as crypto from "crypto";
import * as fs from "fs";
import * as path from "path";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import { createRolloutRBAC } from "./rbac";

export interface EtcdTLSConfig {
  caFile: string;
  certFile: string;
  keyFile: string;
}

export interface EtcdRollingUpgradeArgs {
  stsName: string;
  namespace: pulumi.Input<string>;
  serviceName: string;
  clientPort?: number;
  tls?: EtcdTLSConfig;
  versionHash: pulumi.Input<string>;
  cooldownSeconds?: number;
  readyTimeoutSeconds?: number;
  healthTimeoutSeconds?: number;
  dependsOnSts: pulumi.Input<pulumi.Resource>;
}

export function computeVersionHash(inputs: Record<string, unknown>): string {
  const canonical = JSON.stringify(inputs, Object.keys(inputs).sort());
  return crypto.createHash("sha1").update(canonical).digest("hex").slice(0, 8);
}

const ROLLOUT_SCRIPT = fs.readFileSync(path.join(__dirname, "rollout.sh"), "utf-8");
const ORCHESTRATOR_IMAGE = "alpine/k8s:1.34.0";

export class EtcdRollingUpgrade extends pulumi.ComponentResource {
  public readonly job: k8s.batch.v1.Job;

  constructor(name: string, args: EtcdRollingUpgradeArgs, opts: pulumi.ComponentResourceOptions = {}) {
    super("tidbcloud:rolling-upgrade:Etcd", name, {}, opts);

    const childOpts: pulumi.ComponentResourceOptions = { ...opts, parent: this };

    const rbac = createRolloutRBAC(name, { namespace: args.namespace, stsName: args.stsName }, childOpts);

    const scriptCm = new k8s.core.v1.ConfigMap(
      `${name}-script`,
      {
        metadata: {
          name: `${args.stsName}-rollout-script`,
          namespace: args.namespace,
        },
        data: { "rollout.sh": ROLLOUT_SCRIPT },
      },
      childOpts
    );

    const configYaml = pulumi.output(args.versionHash).apply(() => {
      const cfg: Record<string, unknown> = {
        stsName: args.stsName,
        namespace: args.namespace,
        serviceName: args.serviceName,
        clientPort: args.clientPort ?? 2379,
        cooldownSeconds: args.cooldownSeconds ?? 30,
        readyTimeoutSeconds: args.readyTimeoutSeconds ?? 300,
        healthTimeoutSeconds: args.healthTimeoutSeconds ?? 60,
      };
      if (args.tls) {
        cfg.tls = { enabled: true, ...args.tls };
      } else {
        cfg.tls = { enabled: false };
      }
      return JSON.stringify(cfg);
    });

    const configCm = new k8s.core.v1.ConfigMap(
      `${name}-config`,
      {
        metadata: {
          name: pulumi.interpolate`${args.stsName}-rollout-config-${args.versionHash}`,
          namespace: args.namespace,
        },
        data: { "config.yaml": configYaml },
      },
      childOpts
    );

    this.job = new k8s.batch.v1.Job(
      `${name}-job`,
      {
        metadata: {
          name: pulumi.interpolate`${args.stsName}-rollout-${args.versionHash}`,
          namespace: args.namespace,
        },
        spec: {
          backoffLimit: 0,
          ttlSecondsAfterFinished: 604800,
          template: {
            metadata: { name: pulumi.interpolate`${args.stsName}-rollout-${args.versionHash}` },
            spec: {
              serviceAccountName: rbac.serviceAccount.metadata.name,
              restartPolicy: "OnFailure",
              containers: [
                {
                  name: "orchestrator",
                  image: ORCHESTRATOR_IMAGE,
                  command: ["/bin/sh", "/scripts/rollout.sh"],
                  volumeMounts: [
                    { name: "script", mountPath: "/scripts" },
                    { name: "config", mountPath: "/config" },
                  ],
                },
              ],
              volumes: [
                { name: "script", configMap: { name: scriptCm.metadata.name, defaultMode: 0o755 } },
                { name: "config", configMap: { name: configCm.metadata.name } },
              ],
            },
          },
        },
      },
      {
        ...childOpts,
        dependsOn: [args.dependsOnSts, rbac.roleBinding, scriptCm, configCm],
      }
    );

    this.registerOutputs({ job: this.job });
  }
}
```

Key points to preserve when editing:
- `ROLLOUT_SCRIPT` is inlined at Pulumi-compile time from the shell script committed in Task 3, so there is only one source of truth.
- `computeVersionHash(inputs)` sorts keys for canonical JSON so the same inputs always produce the same hash; exported so `etcd.ts` / `keyspace-etcd-group.ts` can call it.
- `scriptCm` is named per-sts, **not** per-version (so multiple versions of the same sts share the script ConfigMap, and multiple stses in the same namespace get distinct ConfigMaps without collision).
- `configCm` / `job` names embed `versionHash` so Pulumi treats a version bump as a resource replacement.
- `backoffLimit: 0` → a single failure fails the Job permanently; Pulumi's default Job-await waits for `Complete` condition, so Pulumi fails the `pulumi up`.

- [ ] **Step 2: Typecheck**

```bash
cd pool/eks-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors.

- [ ] **Step 3: Spot-check the compiled behavior of `computeVersionHash`**

This is a pure function; verify via `node`:

```bash
cd pool/eks-freetier
node -e '
  const { computeVersionHash } = require("../lib/rolling-upgrade");
  const a = computeVersionHash({ image: "foo", kubelet: "1.34" });
  const b = computeVersionHash({ kubelet: "1.34", image: "foo" });
  const c = computeVersionHash({ image: "foo", kubelet: "1.35" });
  console.log(a, b, c);
  if (a !== b) { console.error("NOT CANONICAL"); process.exit(1); }
  if (a === c) { console.error("COLLIDES"); process.exit(1); }
  console.log("OK");
'
cd ../..
```

Expected: prints three 8-char hex strings; the first two are equal, the third differs; ends with `OK`. If the stack does not directly compile TS to JS on `require`, run `npx tsc --module commonjs --outDir /tmp/rolling-upgrade-build pool/lib/rolling-upgrade/*.ts && node -e '…'` pointed at the compiled output.

- [ ] **Step 4: Commit**

```bash
git add pool/lib/rolling-upgrade/index.ts
git commit -m "rolling-upgrade: add EtcdRollingUpgrade ComponentResource"
```

---

## Task 6: Wire main etcd into `pool/eks-freetier/etcd.ts`

**Files:**
- Modify: `pool/eks-freetier/etcd.ts`

- [ ] **Step 1: Import the component**

Add at the top of `pool/eks-freetier/etcd.ts`, after existing `../lib/*` imports:

```ts
import { EtcdRollingUpgrade, computeVersionHash } from "../lib/rolling-upgrade";
```

- [ ] **Step 2: Capture the `ConfigFile` as a variable and invoke `EtcdRollingUpgrade`**

Replace the trailing `return new k8s.yaml.ConfigFile(...)` block (currently `pool/eks-freetier/etcd.ts:125–152`) with:

```ts
  const cf = new k8s.yaml.ConfigFile(
    "etcd",
    {
      file: "../manifests/etcd.yaml",
      transformations: [
        withNamespace(ns.metadata.name),
        withStatefulSetPVCStorageClass("datadir", "ebs-sc"),
        tryAddMetaDataAnnotations("v1", "ServiceAccount", { "eks.amazonaws.com/role-arn": roleArn }),
        (obj, opts) => {
          if (obj.kind === "StatefulSet" && releaseTag) {
            statefulSetAddNodeSelector({
              "serverless.tidbcloud.com/kubelet-version": eksReleaseVersion(releaseTag!),
            })(obj, opts);
          }
          if (obj.kind === "CronJob") {
            obj.spec.jobTemplate.spec.template.spec.initContainers.forEach((c: any) => {
              if (c.name === "rclone-config-init") {
                c.image = "ghcr.io/rclone/rclone:1.71.2";
                c.command = ["/bin/sh", "-ec", buildRcloneInitScript(bucketName)];
              }
            });
          }
        },
      ],
    },
    { provider: c.provider, dependsOn: deps }
  );

  new EtcdRollingUpgrade(
    "etcd",
    {
      stsName: "etcd",
      namespace: ns.metadata.name,
      serviceName: "etcd-headless",
      versionHash: pulumi
        .output({ releaseTag: releaseTag ?? "" })
        .apply((o) => computeVersionHash(o)),
      dependsOnSts: cf,
    },
    { provider: c.provider, parent: cf }
  );

  return cf;
```

Why `releaseTag` is the only hash input here: it's the sole Pulumi variable that affects `sts.spec.template` (via `nodeSelector`). Image version is hardcoded in the manifest. If the hardcoded image in `etcd.yaml` changes, k8s will generate a new `updateRevision` regardless of Pulumi's `versionHash`; the Job would then need to be forced re-run (see "Manual re-trigger" in the spec §6 if this happens — out of scope for this plan).

- [ ] **Step 3: Typecheck**

```bash
cd pool/eks-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors.

- [ ] **Step 4: Commit**

```bash
git add pool/eks-freetier/etcd.ts
git commit -m "eks-freetier: wire EtcdRollingUpgrade into main etcd install"
```

---

## Task 7: Wire main etcd into `pool/ack-freetier/etcd.ts`

**Files:**
- Modify: `pool/ack-freetier/etcd.ts`

- [ ] **Step 1: Check whether ACK has `releaseTag` plumbed**

Run: `grep -n "releaseTag\|eksReleaseVersion\|kubelet-version" pool/ack-freetier/etcd.ts`

Expected: either no matches (ACK doesn't pin kubelet-version on main etcd today) or shows an existing `releaseTag` variable. Action depends on result:

  - If no matches: in the Pulumi config schema ACK does not pin kubelet-version for main etcd, so `versionHash` should hash `{}` (constant). The `EtcdRollingUpgrade` still runs — first apply creates the Job (fast-path, no-op because pods are new); subsequent applies reuse the same hash → Pulumi no-op. This is intentional: rollout logic is still available for future use, and adding it is free.
  - If matches: mirror the EKS pattern.

- [ ] **Step 2: Add import**

At the top of `pool/ack-freetier/etcd.ts`:

```ts
import { EtcdRollingUpgrade, computeVersionHash } from "../lib/rolling-upgrade";
```

- [ ] **Step 3: Capture `ConfigFile` and instantiate `EtcdRollingUpgrade`**

Replace the trailing `return new k8s.yaml.ConfigFile(...)` block with:

```ts
  const cf = new k8s.yaml.ConfigFile(
    "etcd",
    {
      file: "../manifests/etcd.yaml",
      transformations: [
        withNamespace(ns.metadata.name),
        withStatefulSetPVCStorageClass("datadir", "ebs-sc"),
        tryAddMetaDataAnnotations("v1", "ServiceAccount", { "pod-identity.alibabacloud.com/role-name": roleName }),
        (obj) => {
          if (obj.kind === "CronJob") {
            obj.spec.jobTemplate.spec.template.spec.initContainers.forEach((c: any) => {
              if (c.name === "rclone-config-init") {
                c.image = "registry.cn-hangzhou.aliyuncs.com/acs/ack-ram-tool:1.2.0";
                c.command = ["/bin/sh", "-ec", buildRcloneInitScript(bucketName)];
              }
            });
          }
        },
      ],
    },
    { provider: p, dependsOn: deps }
  );

  new EtcdRollingUpgrade(
    "etcd",
    {
      stsName: "etcd",
      namespace: ns.metadata.name,
      serviceName: "etcd-headless",
      versionHash: pulumi.output({}).apply((o) => computeVersionHash(o)),
      dependsOnSts: cf,
    },
    { provider: p, parent: cf }
  );

  return cf;
```

Note: the `computeVersionHash({})` produces a constant hash — the Job runs once per new cluster (as a no-op on greenfield) and never re-runs on subsequent `pulumi up`. If ACK ever adds `releaseTag`, feed it into the hash the same way EKS does.

- [ ] **Step 4: Typecheck**

```bash
cd pool/ack-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors.

- [ ] **Step 5: Commit**

```bash
git add pool/ack-freetier/etcd.ts
git commit -m "ack-freetier: wire EtcdRollingUpgrade into main etcd install"
```

---

## Task 8: Fix the keyspace-etcd `releaseTag` bug (plumb through from serverless.ts)

**Files:**
- Modify: `pool/eks-freetier/serverless.ts:1167–1186`

- [ ] **Step 1: Show the current call site**

Run: `sed -n '1167,1190p' pool/eks-freetier/serverless.ts`

Expected: the `for` loop currently calls `InstallKeyspaceEtcdGroup` with `{ clusterName, namespace: nsName, name: ..., resources: keyspaceEtcdGroupResources }` — **no `releaseTag`**.

- [ ] **Step 2: Add `releaseTag` to the options**

Modify the call so it reads:

```ts
  // Keyspace Etcd Groups
  for (let i = 0; i < keyspaceEtcdGroupCount; i++) {
    InstallKeyspaceEtcdGroup(
      freeTierStackProvider,
      {
        clusterName,
        namespace: nsName,
        name: `keyspace-etcd-group-${i}`,
        resources: keyspaceEtcdGroupResources,
        releaseTag: keyspaceEtcdGroupSpec?.releaseTag,
      },
      clusterIssuer,
      ns
    );
  }
```

- [ ] **Step 3: Typecheck**

```bash
cd pool/eks-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors. `KeyspaceEtcdGroupSpec.releaseTag` is already `string` (not optional) in the interface at `serverless.ts:112`, so `keyspaceEtcdGroupSpec?.releaseTag` is `string | undefined`, which matches `KeyspaceEtcdGroupOptions.releaseTag?: string`.

- [ ] **Step 4: Commit**

```bash
git add pool/eks-freetier/serverless.ts
git commit -m "eks-freetier: plumb keyspaceEtcdGroupSpec.releaseTag into InstallKeyspaceEtcdGroup"
```

---

## Task 9: Wire `EtcdRollingUpgrade` into `InstallKeyspaceEtcdGroup`

**Files:**
- Modify: `pool/lib/serverless/keyspace-etcd-group.ts`

- [ ] **Step 1: Add imports**

At the top of `pool/lib/serverless/keyspace-etcd-group.ts`, add:

```ts
import { EtcdRollingUpgrade, computeVersionHash } from "../rolling-upgrade";
```

- [ ] **Step 2: Instantiate `EtcdRollingUpgrade` and return the sts handle**

Replace the trailing `const sts = new k8s.yaml.ConfigFile(…)` block (currently ending the function body at line 131) with:

```ts
  const sts = new k8s.yaml.ConfigFile(
    `${clusterName}-${name}`,
    {
      file: "../manifests/serverless/keyspace-etcd-group/sts.yaml",
      transformations: [
        withNamespace(namespace),
        withName(name),
        withStatefulSetResourcesRequirements(resources),
        (obj: any, _opts: pulumi.CustomResourceOptions) => {
          obj.metadata.labels[appLabel] = name;
          obj.spec.serviceName = `${name}-peer`;
          obj.spec.selector.matchLabels[appLabel] = name;
          obj.spec.template.metadata.labels[appLabel] = name;
          const podSpec = obj.spec.template.spec;
          podSpec.volumes[0].secret.secretName = serverCertName(name);
          podSpec.topologySpreadConstraints[0].labelSelector.matchLabels[appLabel] = name;
        },
        statefulSetAddEnvs(
          "etcd",
          { name: "SERVICE_NAME", value: `${name}-peer` },
          { name: "GROUP_NAME", value: name }
        ),
      ].concat(
        optionalList(
          !!releaseTag,
          statefulSetAddNodeSelector({
            "serverless.tidbcloud.com/kubelet-version": eksReleaseVersion(releaseTag!),
          })
        )
      ),
    },
    { provider, dependsOn: [...deps, cert, peerService, clientService, monitor] }
  );

  new EtcdRollingUpgrade(
    `${clusterName}-${name}`,
    {
      stsName: name,
      namespace,
      serviceName: `${name}-peer`,
      tls: {
        caFile: "/var/lib/etcd-tls/ca.crt",
        certFile: "/var/lib/etcd-tls/tls.crt",
        keyFile: "/var/lib/etcd-tls/tls.key",
      },
      versionHash: pulumi
        .output({ releaseTag: releaseTag ?? "", resources })
        .apply((o) => computeVersionHash(o)),
      dependsOnSts: sts,
    },
    { provider, parent: sts }
  );

  return sts;
```

Two small changes versus the current function:
1. The last line changes from no return to `return sts;` — matches the updated type signature (see step 3).
2. `EtcdRollingUpgrade` is instantiated after `sts`, parented under it, depending on it.

- [ ] **Step 3: Update the function return type to expose the sts**

Change the function signature to:

```ts
export function InstallKeyspaceEtcdGroup(
  provider: k8s.Provider,
  { clusterName, namespace, name, resources, releaseTag }: KeyspaceEtcdGroupOptions,
  ...deps: pulumi.Input<pulumi.Resource>[]
): k8s.yaml.ConfigFile {
```

(Explicit return type helps consumers that may want to depend on the sts externally.)

- [ ] **Step 4: Typecheck**

```bash
cd pool/eks-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors.

- [ ] **Step 5: Commit**

```bash
git add pool/lib/serverless/keyspace-etcd-group.ts
git commit -m "keyspace-etcd-group: wire EtcdRollingUpgrade per group"
```

---

## Task 10: Full-tree typecheck for both stacks

- [ ] **Step 1: EKS**

```bash
cd pool/eks-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors.

- [ ] **Step 2: ACK**

```bash
cd pool/ack-freetier
npx tsc --noEmit -p tsconfig.json
cd ../..
```

Expected: no errors.

- [ ] **Step 3: `npm run lint` in each stack**

```bash
(cd pool/eks-freetier && npm run lint)
(cd pool/ack-freetier && npm run lint 2>/dev/null || true)
```

Expected: EKS lint passes. ACK may not have a `lint` script; skip if missing.

- [ ] **Step 4: Commit if any fixes needed**

No commit expected here if prior tasks were clean.

---

## Task 11: `pulumi preview` on the dev stack

**Files (read-only):**
- `pool/eks-freetier/Pulumi.dev-us-east-1-f01.yaml`

This is the smoke test that Pulumi constructs the expected resources.

- [ ] **Step 1: Choose the dev stack**

Run: `cd pool/eks-freetier && pulumi stack select dev-us-east-1-f01 && cd ../..`

Expected: stack selected. If the stack is not logged in, follow `pulumi login` instructions from existing docs (out of scope for this task).

- [ ] **Step 2: Preview**

Run:
```bash
cd pool/eks-freetier
pulumi preview --diff 2>&1 | tee /tmp/preview-etcd-rolling-upgrade.txt
cd ../..
```

Expected in `/tmp/preview-etcd-rolling-upgrade.txt`:

- StatefulSet resources for `etcd` and each `keyspace-etcd-group-*`: **update in place** with `+ spec.updateStrategy: { type: OnDelete }` (depending on how the Kubernetes provider represents absent fields — may show as a no-op if k8s defaults were already OnDelete; unlikely; flag if so).
- New resources:
  - `kubernetes:core/v1:ServiceAccount etcd-etcd-sa` in `tidb-admin` (similar for keyspace groups in `tidb-serverless`)
  - `kubernetes:rbac.authorization.k8s.io/v1:Role etcd-etcd-role`
  - `kubernetes:rbac.authorization.k8s.io/v1:RoleBinding etcd-etcd-rb`
  - `kubernetes:core/v1:ConfigMap etcd-etcd-script` (data: `rollout.sh`)
  - `kubernetes:core/v1:ConfigMap etcd-etcd-config` (data: `config.yaml`)
  - `kubernetes:batch/v1:Job etcd-etcd-job`
  - Analogous set per `keyspace-etcd-group-<i>`
- keyspace-etcd-group sts now includes the `serverless.tidbcloud.com/kubelet-version` nodeSelector (was missing pre-fix — Task 8). This is a **pod-template change** → k8s will generate a new revision when applied, which is exactly what the Job is there to handle.

- [ ] **Step 3: Sanity-check Job names embed a short hash**

Run: `grep -o '"name": "etcd-rollout-[a-f0-9]\{8\}"' /tmp/preview-etcd-rolling-upgrade.txt | head -3` (or similar depending on Pulumi's preview format)

Expected: names like `etcd-rollout-3f7a1c9b`, `keyspace-etcd-group-0-rollout-...` etc. If Pulumi preview doesn't render the hash (because `versionHash` is an `Output`), accept that and verify on apply.

- [ ] **Step 4: Stop here without applying**

Do **not** run `pulumi up` in this task. Task 12 covers the real deploy.

---

## Task 12: Apply + verify on a dev EKS cluster

This is the acceptance task. It requires a running dev EKS cluster and cluster credentials.

- [ ] **Step 1: Note baseline etcd metrics**

Run (from a pod or host with kubectl access to the dev cluster):
```bash
kubectl -n tidb-admin port-forward pod/etcd-0 2379:2379 &
PF=$!
sleep 2
curl -s http://localhost:2379/metrics | grep -E '^etcd_server_leader_changes_seen_total'
kill $PF
```

Record the value of `etcd_server_leader_changes_seen_total` — this is the before value.

- [ ] **Step 2: `pulumi up` (first apply — wires everything in)**

```bash
cd pool/eks-freetier
pulumi up --yes
cd ../..
```

Expected:

- Resources created without errors.
- Each Job reaches `Complete`. With brand-new rollout plumbing on an already-rolled cluster, the fast-path in `rollout.sh` should short-circuit (all pods already at target revision), so Jobs complete in <10s each. Pulumi waits for each Job and then returns success.
- No etcd pods restart (nothing to roll on first apply, unless `updateStrategy` change alone caused a restart; it should not — `updateStrategy` is a metadata-level field for sts controller, not a pod-template change).

- [ ] **Step 3: Verify baseline metrics unchanged**

Re-run the metrics capture from Step 1. Expected: `etcd_server_leader_changes_seen_total` **unchanged** (no elections).

- [ ] **Step 4: Force a rollout by bumping `releaseTag`**

Edit `pool/eks-freetier/Pulumi.dev-us-east-1-f01.yaml`:
- Locate `eks-freetier:etcd:` (if present with `releaseTag`) and bump to the new kubelet version we're targeting for the EKS upgrade (coordinate with whichever release is the actual upgrade target).
- Locate `eks-freetier:keyspace-etcd-group-spec:` → `releaseTag:` (line ~690 in `Pulumi.prod-us-east-1-f01.yaml` as a reference pattern) and bump similarly.

- [ ] **Step 5: `pulumi up` — this time should trigger real rollouts**

```bash
cd pool/eks-freetier
pulumi up --yes
cd ../..
```

Expected:

- Each sts's `versionHash` changes → a new Job is created.
- Old Jobs are GC'd via `ttlSecondsAfterFinished` (not immediately, but eventually).
- Each new Job runs the full choreography:
  1. wait for cluster health
  2. move leader to `-0` (or no-op if already leader)
  3. delete `-2`, wait Ready+healthy, sleep 30s
  4. delete `-1`, wait Ready+healthy, sleep 30s
  5. move leader to `-2`
  6. delete `-0`, wait Ready+healthy
- Total time per sts: ~3 pods × (pod restart ~20s + 30s cooldown) + leader moves ≈ 2–3 min.
- All Jobs complete; `pulumi up` succeeds.

- [ ] **Step 6: Re-check `etcd_server_leader_changes_seen_total`**

Expected: increased by **at most 2** — one deliberate leader transfer at the start (to `-0`, unless already there) and one before rolling `-0` (to `-(N-1)`). The counter tracks *all* leader changes including deliberate transfers, so the acceptance criterion is "no more than 2 increments". To distinguish deliberate transfers from uncontrolled elections, cross-check the rollout Job log from Step 7 — each deliberate transfer logs `[rollout] attempt N: transfer leader <from> -> <to>`. Any increment beyond those logged transfers is an uncontrolled election and should be treated as a regression.

- [ ] **Step 7: Inspect Job logs for one sts**

Run:
```bash
kubectl -n tidb-admin logs job/etcd-rollout-<hash>
```

Expected: the `[rollout]` echo lines in sequence: cluster health → leader move → delete etcd-2 → healthy → cooldown → delete etcd-1 → healthy → cooldown → leader move to etcd-2 → delete etcd-0 → healthy → done.

- [ ] **Step 8: Record results in spec §9 progress and commit stack config**

```bash
git add pool/eks-freetier/Pulumi.dev-us-east-1-f01.yaml
git commit -m "eks-freetier: bump releaseTag on dev stack to exercise rolling-upgrade"
```

---

## Self-Review

Covered spec requirements:

| Spec requirement | Task |
| --- | --- |
| §4 OnDelete on both sts | Task 1, 2 |
| §4.1 ComponentResource signature | Task 5 |
| §4.1 RBAC | Task 4 |
| §4.2 per-instance ConfigMap | Task 5 |
| §4.3 rollout.sh algorithm | Task 3 |
| §4.4 revision-hash-based idempotency | Task 3 (script logic), Task 5 (ConfigMap naming) |
| §4.5 trigger for main etcd | Task 6, 7 |
| §4.5 trigger for keyspace-etcd-group | Task 9 |
| §4.5 `releaseTag` plumbing fix | Task 8 |
| §5 greenfield fast-path | Task 3 (fast-path block) |
| §6 idempotency | Task 3 (skip logic), Task 5 (Job name per hash) |
| §8 RBAC scope | Task 4 |
| §9.1 component + tests | Task 3 (shellcheck), Task 5 (typecheck, hash canonicality) |
| §9.2–9.4 roll to dev | Task 11, 12 |
| §9.5 `releaseTag` wiring | Task 8 |
| §9.6 `EtcdRollingUpgrade` in `InstallKeyspaceEtcdGroup` | Task 9 |
| §11 orchestrator image `alpine/k8s` | Task 5 (`ORCHESTRATOR_IMAGE`) |
| §11 parallel per-group Jobs | Task 9 (per-group instantiation — Pulumi parallelism default) |

No placeholders, no "TODO"s, no handwavy "add error handling" steps.

Type consistency: `EtcdRollingUpgrade`, `EtcdTLSConfig`, `EtcdRollingUpgradeArgs`, `computeVersionHash` — names used consistently across Task 5 (definition) and Tasks 6, 7, 9 (consumers). RBAC helper signature `createRolloutRBAC(name, args, opts)` defined in Task 4, consumed in Task 5.