# Critical-Services Exclusive Tiers Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Implement the config-gated `criticalTier` opt-in (spec pin + identity label + exclusion anti-affinity) for the five in-scope critical-services consumers, plus the first canary (ai-gateway on staging-us-east-1-f01, via `release-0.62`).

**Architecture:** One shared helper block in `pool/lib/transformation/index.ts` (constants, preview-time tier validation against `general-guaranteed-fleet`, sts/deployment transformations). Each consumer's existing per-stack config object gains an optional tier field; unset means bit-for-bit legacy behavior. The TidbCluster PD path gets a CR-specific transformation in `serverless.ts`. Spec: `docs/superpowers/specs/2026-07-13-critical-services-exclusive-tiers-design.md`.

**Tech Stack:** Pulumi TypeScript (`pool/eks-freetier`, shared `pool/lib`), k8s.yaml.ConfigFile transformations, per-stack `Pulumi.<stack>.yaml` config.

**Verification model (no unit-test infra in this repo):** CI is prettier check-fmt + eslint + Pulumi preview; there is no jest/mocha and we do not add one. Each code task verifies with `npx tsc --noEmit` + lint + fmt; Task 2 also behaviorally exercises the pure transformation with a scratch ts-node script (not committed). The end-to-end gate is the PR CI Pulumi preview: staging-us-east-1-f01 shows exactly the expected diff, every other stack shows no changes. Never run `pulumi up`/`preview` locally.

**Branches (revised 2026-07-14, after dev-us-east-1-f01 was deleted):** the mechanism (Tasks 2-7) lands on `iosmanthus/critical-services-exclusive-tiers-spec` → PR #8284 → `main`, code + docs only, no stack config. The canary + preview run on a second branch cut from `release-0.62` (Task 8), because the only live non-prod stack is staging-us-east-1-f01 and release-branch PRs preview exactly the staging stacks. A merge to either branch auto-applies, so neither PR gets the `pulumi:up` label and neither merges before the preview is read.

---

## File Map

- Modify: `pool/lib/transformation/index.ts` — append critical-exclusive constants + helpers (one cohesive block; this module is the established home for all transformations)
- Modify: `pool/eks-freetier/etcd.ts` — `EtcdConfig.criticalTier`, sts transformation, `versionHash`
- Modify: `pool/lib/serverless/keyspace-etcd-group.ts` — `KeyspaceEtcdGroupOptions.criticalTier`, sts transformation, `versionHash`
- Modify: `pool/eks-freetier/serverless.ts` — `KeyspaceEtcdGroupSpec.criticalTier` pass-through; `PDClusterConfig.pdCriticalTier` + `withPDCriticalExclusive` (TidbCluster CR); `PDClusterConfig.tsoServerCriticalTier` + pd-tso sts wiring
- Modify: `pool/eks-freetier/ai-gateway.ts` — `AIGatewayConfig.criticalTier`, deployment wiring
- Modify (on `release-0.62`): `pool/eks-freetier/Pulumi.staging-us-east-1-f01.yaml` — add `guaranteed-xl` fleet entry, opt ai-gateway into it

Config field naming: standalone config blocks use `criticalTier` (etcd, ai-gateway, keyspace-etcd-group-spec). The `pd-cluster` block hosts several workloads, so its fields follow the block's existing prefix convention: `pdCriticalTier`, `tsoServerCriticalTier`.

Identity label values (`serverless.tidbcloud.com/critical-app`): `etcd`, `pd`, `pd-tso`, `keyspace-etcd` (shared by all keyspace-etcd-group-N — groups may co-locate with each other, consistent with the spec's single identity for this consumer), `ai-gateway`.

---

### Task 1: Workspace setup and baseline

**Files:** none (environment only)

- [ ] **Step 1.1: Install workspace deps at pool root** (deps are hoisted; installing under eks-freetier alone breaks with TS2307)

```bash
cd pool && npm install
```

Expected: completes without errors (warnings OK).

- [ ] **Step 1.2: Record the type-check baseline**

```bash
cd pool/eks-freetier && npx tsc --noEmit; echo "exit=$?"
```

Expected: `exit=0`. If non-zero, capture the pre-existing error list to a scratch file — later tasks must not add NEW errors relative to it, and do not fix pre-existing ones.

- [ ] **Step 1.3: Confirm fmt/lint baseline**

```bash
cd pool/eks-freetier && npm run check-fmt && npm run lint
```

Expected: both pass (they gate CI on main today).

---

### Task 2: Shared helper in `pool/lib/transformation/index.ts`

**Files:**
- Modify: `pool/lib/transformation/index.ts` (append at end of file, after the last export ~line 855)

- [ ] **Step 2.1: Append the critical-exclusive block**

```ts
// ---------------------------------------------------------------------------
// critical-services spec tiers + workload-exclusive scheduling.
// See docs/superpowers/specs/2026-07-13-critical-services-exclusive-tiers-design.md.
// ---------------------------------------------------------------------------

export const criticalAppLabelKey = "serverless.tidbcloud.com/critical-app";
export const nodegroupLabelKey = "serverless.tidbcloud.com/nodegroup";

export function criticalTierNodegroup(tier: string): string {
  return `guaranteed-${tier}`;
}

// Fails the pulumi program at preview time when a workload opts into a tier the
// stack's general-guaranteed-fleet doesn't carry (entry names drift per stack),
// instead of leaving the pod Pending after apply.
export function validateCriticalTier(app: string, tier: string): string {
  const nodegroup = criticalTierNodegroup(tier);
  const fleet = new pulumi.Config().requireObject<{ name?: string }[]>("general-guaranteed-fleet");
  const names = fleet.map((entry) => entry.name ?? "");
  if (!names.includes(nodegroup)) {
    throw new Error(
      `criticalTier "${tier}" of "${app}" needs fleet entry "${nodegroup}" in general-guaranteed-fleet; this stack only has: ${names.join(
        ", "
      )}`
    );
  }
  return nodegroup;
}

// Repels every pod that declares a different critical-app identity. Pods
// without the label (legacy consumers, DaemonSets) are unaffected — the
// converging "C1" semantics from the spec. Kubernetes required anti-affinity
// is symmetric, so a migrated pod already on a node also blocks later pods of
// other migrated apps.
export function criticalExclusiveAntiAffinityTerm(app: string) {
  return {
    topologyKey: "kubernetes.io/hostname",
    namespaceSelector: {},
    labelSelector: {
      matchExpressions: [
        { key: criticalAppLabelKey, operator: "Exists" },
        { key: criticalAppLabelKey, operator: "NotIn", values: [app] },
      ],
    },
  };
}

export const addCriticalExclusive =
  (gv: string, kind: string, app: string, tier: string) => (obj: any, _opts: pulumi.CustomResourceOptions) => {
    // Validate before the kind guard so a bad tier fails the preview even when
    // this transformation never matches an object in the file.
    const nodegroup = validateCriticalTier(app, tier);
    if (!isMatchGVK(obj, gv, kind)) {
      return;
    }
    const template = obj.spec.template;
    template.metadata = template.metadata ?? {};
    template.metadata.labels = {
      ...template.metadata.labels,
      [criticalAppLabelKey]: app,
    };
    template.spec.nodeSelector = {
      ...template.spec.nodeSelector,
      [nodegroupLabelKey]: nodegroup,
    };
    const affinity = (template.spec.affinity = template.spec.affinity ?? {});
    const podAntiAffinity = (affinity.podAntiAffinity = affinity.podAntiAffinity ?? {});
    if (!podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution) {
      podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution = [];
    }
    podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution.push(criticalExclusiveAntiAffinityTerm(app));
  };

export const statefulSetAddCriticalExclusive = (app: string, tier: string) =>
  addCriticalExclusive("apps/v1", "StatefulSet", app, tier);

export const deploymentAddCriticalExclusive = (app: string, tier: string) =>
  addCriticalExclusive("apps/v1", "Deployment", app, tier);
```

Notes for the implementer:
- `pulumi` and `isMatchGVK` are already in this module's scope (top-of-file import / earlier export). Add nothing else.
- Validation is inside the returned closure (not at wrapper-construction time). Callers use `optionalList(!!cfg.criticalTier, statefulSetAddCriticalExclusive(app, cfg.criticalTier!))`, and `optionalList` evaluates its element argument eagerly — construction must therefore be side-effect-free even when the flag is unset.
- Appending (never replacing) the anti-affinity term preserves each workload's own required/preferred terms (PD's self-spread, pd-tso's preferred spread).

- [ ] **Step 2.2: Type-check**

```bash
cd pool/eks-freetier && npx tsc --noEmit; echo "exit=$?"
```

Expected: same result as the Task 1 baseline (no new errors).

- [ ] **Step 2.3: Behavioral scratch test (positive + negative), not committed**

Write `<scratchpad>/critical-exclusive-check.ts` (use the session scratchpad dir, NOT the repo):

```ts
import { deploymentAddCriticalExclusive, criticalAppLabelKey } from "./lib/transformation";

function freshObj(): any {
  return {
    apiVersion: "apps/v1",
    kind: "Deployment",
    spec: {
      template: {
        metadata: { labels: { app: "ai-gateway" } },
        spec: {
          affinity: {
            nodeAffinity: { requiredDuringSchedulingIgnoredDuringExecution: { nodeSelectorTerms: [] } },
            podAntiAffinity: { preferredDuringSchedulingIgnoredDuringExecution: [{ weight: 100 }] },
          },
        },
      },
    },
  };
}

// positive: tier exists in fleet (injected via PULUMI_CONFIG below)
const obj = freshObj();
deploymentAddCriticalExclusive("ai-gateway", "xl")(obj, {} as any);
const tpl = obj.spec.template;
if (tpl.metadata.labels[criticalAppLabelKey] !== "ai-gateway") throw new Error("identity label missing");
if (tpl.metadata.labels["app"] !== "ai-gateway") throw new Error("existing labels clobbered");
if (tpl.spec.nodeSelector["serverless.tidbcloud.com/nodegroup"] !== "guaranteed-xl") throw new Error("pin missing");
const required = tpl.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution;
if (required.length !== 1 || required[0].topologyKey !== "kubernetes.io/hostname") throw new Error("term missing");
if (tpl.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution.length !== 1)
  throw new Error("preferred terms clobbered");
if (tpl.spec.affinity.nodeAffinity === undefined) throw new Error("nodeAffinity clobbered");
const exprs = required[0].labelSelector.matchExpressions;
if (exprs[0].operator !== "Exists" || exprs[1].operator !== "NotIn" || exprs[1].values[0] !== "ai-gateway")
  throw new Error("selector wrong");

// kind guard: StatefulSet-only wrapper must ignore a Deployment and vice versa
const other = freshObj();
other.kind = "StatefulSet";
deploymentAddCriticalExclusive("ai-gateway", "xl")(other, {} as any);
if (other.spec.template.metadata.labels[criticalAppLabelKey]) throw new Error("kind guard failed");

// negative: unknown tier must throw naming app, tier, and available entries
let threw = false;
try {
  deploymentAddCriticalExclusive("ai-gateway", "4xl")(freshObj(), {} as any);
} catch (e: any) {
  threw = true;
  if (!/4xl.*ai-gateway.*guaranteed-4xl/s.test(e.message)) throw new Error(`bad message: ${e.message}`);
}
if (!threw) throw new Error("missing-tier validation did not throw");

console.log("OK");
```

Run it with pulumi config injected via env (pulumi reads `PULUMI_NODEJS_PROJECT`/`PULUMI_CONFIG` from the environment; config object values are JSON-encoded strings):

```bash
cd pool/eks-freetier
cp <scratchpad>/critical-exclusive-check.ts ./critical-exclusive-check.tmp.ts
PULUMI_NODEJS_PROJECT=eks-freetier \
PULUMI_CONFIG='{"eks-freetier:general-guaranteed-fleet":"[{\"name\":\"guaranteed\"},{\"name\":\"guaranteed-xl\"}]"}' \
npx ts-node --transpile-only ./critical-exclusive-check.tmp.ts
rm ./critical-exclusive-check.tmp.ts
```

Expected: `OK`. (The copy into the package dir is so ts-node resolves `./lib/transformation` and `@pulumi/pulumi` from the workspace; delete it afterwards — it must not be committed. If `npx ts-node` is unavailable, `npm exec --yes -- ts-node@10 --transpile-only ...` from the same directory.)

- [ ] **Step 2.4: fmt + lint + commit**

```bash
cd pool/eks-freetier && npm run fmt && npm run check-fmt && npm run lint
git status --short   # verify ONLY pool/lib/transformation/index.ts is modified
git add pool/lib/transformation/index.ts
git commit -m "pool: add critical-exclusive scheduling transformation helpers

statefulSetAddCriticalExclusive/deploymentAddCriticalExclusive apply the
three-piece contract (nodegroup pin, critical-app identity label, appended
required podAntiAffinity term) with preview-time validation of the tier
against the stack's general-guaranteed-fleet entries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
```

---

### Task 3: etcd opt-in (`pool/eks-freetier/etcd.ts`)

**Files:**
- Modify: `pool/eks-freetier/etcd.ts` (interface ~line 15-23, transformation block ~line 141-161, versionHash ~line 188-193)

- [ ] **Step 3.1: Extend `EtcdConfig`**

```ts
export interface EtcdConfig {
  releaseTag?: string;
  // Per-region override for the admin etcd container resources. When unset the
  // manifest defaults apply (requests cpu1/8Gi, limits cpu2/8Gi). Set smaller on
  // low-traffic regions so etcd stops reserving a whole guaranteed-xl node by
  // request. Note: the etcd --quota-backend-bytes (8Gi) is independent — only
  // shrink memory well below it, which is safe for a small admin etcd.
  resources?: k8s.types.input.core.v1.ResourceRequirements;
  // Opt-in to critical-services workload-exclusive scheduling: the spec-tier
  // name ("large"/"xl"/...), resolved to fleet entry guaranteed-<tier>.
  // Unset keeps legacy free scheduling on the critical-services pool.
  criticalTier?: string;
}
```

- [ ] **Step 3.2: Import the helper**

In the existing `../lib/transformation` import block add `statefulSetAddCriticalExclusive`:

```ts
import {
  tryAddMetaDataAnnotations,
  withNamespace,
  withStatefulSetPVCStorageClass,
  statefulSetAddNodeSelector,
  statefulSetAddCriticalExclusive,
} from "../lib/transformation";
```

- [ ] **Step 3.3: Apply in the inline StatefulSet transformation** (same style as the existing `releaseTag`/`etcdResources` branches; insert after the `etcdResources` branch)

```ts
          if (obj.kind === "StatefulSet" && etcdConfig.criticalTier) {
            statefulSetAddCriticalExclusive("etcd", etcdConfig.criticalTier)(obj, opts);
          }
```

- [ ] **Step 3.4: Fold into `versionHash`** — REQUIRED: without this the partition-gated `EtcdRollingUpgrade` Job never fires for the template change (the `etcdResources` lesson, commit 2394bf960). Conditional so untouched stacks keep their existing hash:

```ts
      versionHash: pulumi
        .output({
          releaseTag: releaseTag ?? "",
          ...(etcdResources ? { resources: etcdResources } : {}),
          ...(etcdConfig.criticalTier ? { criticalTier: etcdConfig.criticalTier } : {}),
        })
        .apply((o) => computeVersionHash(o)),
```

- [ ] **Step 3.5: Verify and commit**

```bash
cd pool/eks-freetier && npx tsc --noEmit && npm run fmt && npm run check-fmt && npm run lint
git add pool/eks-freetier/etcd.ts
git commit -m "eks-freetier: wire criticalTier opt-in into admin etcd

Config-gated on eks-freetier:etcd.criticalTier; folded into the rolling-
upgrade versionHash (only when set) so the template change actually rolls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
```

---

### Task 4: keyspace-etcd opt-in (`pool/lib/serverless/keyspace-etcd-group.ts` + pass-through)

**Files:**
- Modify: `pool/lib/serverless/keyspace-etcd-group.ts` (options ~line 17-26, destructure ~line 30, sts transformations ~line 136-143, versionHash ~line 159)
- Modify: `pool/eks-freetier/serverless.ts` (`KeyspaceEtcdGroupSpec` ~line 120-124, call site ~line 1205-1214)

- [ ] **Step 4.1: Extend `KeyspaceEtcdGroupOptions` and destructure**

```ts
export interface KeyspaceEtcdGroupOptions {
  clusterName: string;
  namespace: string;
  name: string;
  resources: ResourcesRequirements;
  releaseTag?: string;
  // Image used by the fix-permissions initContainer. Defaults to the
  // manifest's "busybox" when unset.
  busyboxImage?: string;
  // Opt-in to critical-services workload-exclusive scheduling (spec-tier name,
  // e.g. "xl"). All keyspace-etcd groups share the "keyspace-etcd" identity.
  criticalTier?: string;
}
```

```ts
export function InstallKeyspaceEtcdGroup(
  provider: k8s.Provider,
  { clusterName, namespace, name, resources, releaseTag, busyboxImage, criticalTier }: KeyspaceEtcdGroupOptions,
  ...deps: pulumi.Input<pulumi.Resource>[]
): k8s.yaml.ConfigFile {
```

- [ ] **Step 4.2: Import and apply the transformation**

Add `statefulSetAddCriticalExclusive` to the existing `../transformation` import list, then extend the sts transformations (after the existing `optionalList(!!releaseTag, ...)` concat):

```ts
      ].concat(
        optionalList(
          !!releaseTag,
          statefulSetAddNodeSelector({
            "serverless.tidbcloud.com/kubelet-version": eksReleaseVersion(releaseTag!),
          })
        ),
        optionalList(!!criticalTier, statefulSetAddCriticalExclusive("keyspace-etcd", criticalTier!))
      ),
```

(`optionalList` returns `T[]`, and `Array.prototype.concat` accepts multiple arrays — keep it one `.concat` call with two arguments.)

- [ ] **Step 4.3: Fold into `versionHash`** (same trap as etcd — conditional key):

```ts
      versionHash: pulumi
        .output({ releaseTag: releaseTag ?? "", resources, ...(criticalTier ? { criticalTier } : {}) })
        .apply((o) => computeVersionHash(o)),
```

- [ ] **Step 4.4: Pass through from `serverless.ts`**

```ts
interface KeyspaceEtcdGroupSpec {
  resources: ResourcesRequirements;
  releaseTag: string;
  busyboxImage?: string;
  // Opt-in to critical-services workload-exclusive scheduling (spec-tier name).
  criticalTier?: string;
}
```

At the `InstallKeyspaceEtcdGroup` call site add one field:

```ts
        releaseTag: keyspaceEtcdGroupSpec?.releaseTag,
        busyboxImage: keyspaceEtcdGroupSpec?.busyboxImage,
        criticalTier: keyspaceEtcdGroupSpec?.criticalTier,
```

- [ ] **Step 4.5: Verify and commit**

```bash
cd pool/eks-freetier && npx tsc --noEmit && npm run fmt && npm run check-fmt && npm run lint
git add pool/lib/serverless/keyspace-etcd-group.ts pool/eks-freetier/serverless.ts
git commit -m "eks-freetier: wire criticalTier opt-in into keyspace-etcd groups

Gated on keyspace-etcd-group-spec.criticalTier; all groups share the
keyspace-etcd identity; hashed into the rolling-upgrade versionHash only
when set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
```

---

### Task 5: serverless-cluster PD opt-in (TidbCluster CR, `pool/eks-freetier/serverless.ts`)

**Files:**
- Modify: `pool/eks-freetier/serverless.ts` (`PDClusterConfig` ~line 189-223, new function next to `withAdditionalNodeSelector` ~line 2599, wiring in the `serverlessCluster` ConfigFile ~line 1092-1114, import block ~line 28-54)

- [ ] **Step 5.1: Extend `PDClusterConfig`**

Add one field (keep it adjacent to `pdResourcesRequirements`):

```ts
  pdResourcesRequirements: PDResourcesRequirements;
  // Opt-in to critical-services workload-exclusive scheduling for PD pods
  // (spec-tier name, e.g. "xl"). Applied to the TidbCluster CR's pd component.
  pdCriticalTier?: string;
```

- [ ] **Step 5.2: Import the shared pieces**

Extend the existing `./lib/transformation` import list with:

```ts
  criticalAppLabelKey,
  nodegroupLabelKey,
  validateCriticalTier,
  criticalExclusiveAntiAffinityTerm,
  statefulSetAddCriticalExclusive,
```

(`statefulSetAddCriticalExclusive` is used by Task 6 in this same file; importing it here keeps this file's diff self-consistent if tasks land separately — remove from this step if Task 6 already merged it.)

- [ ] **Step 5.3: Add the CR-specific transformation** (place directly after `withAdditionalNodeSelector`, ~line 2619)

```ts
function withPDCriticalExclusive(tier: string) {
  return (obj: any, _opts: pulumi.CustomResourceOptions) => {
    const nodegroup = validateCriticalTier("pd", tier);
    assertGVK(obj, "pingcap.com/v1alpha1", "TidbCluster");
    obj.spec.pd.nodeSelector = {
      ...obj.spec.pd.nodeSelector,
      [nodegroupLabelKey]: nodegroup,
    };
    // ComponentSpec.labels are merged into the PD pod labels by tidb-operator
    // (the CR already relies on this for the zr-evicter ignore label).
    obj.spec.pd.labels = {
      ...obj.spec.pd.labels,
      [criticalAppLabelKey]: "pd",
    };
    const affinity = (obj.spec.pd.affinity = obj.spec.pd.affinity ?? {});
    const podAntiAffinity = (affinity.podAntiAffinity = affinity.podAntiAffinity ?? {});
    if (!podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution) {
      podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution = [];
    }
    podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution.push(criticalExclusiveAntiAffinityTerm("pd"));
  };
}
```

- [ ] **Step 5.4: Wire into the `serverlessCluster` transformations**

Insert after the existing `withAdditionalNodeSelector("pd", {...})` entry (order matters only in that it must come BEFORE `withConfigForCSEOperator`, which the file comments require to stay last):

```ts
        withAdditionalNodeSelector("pd", {
          "topology.kubernetes.io/zone": primaryZone,
        }),
        ...optionalList(!!pdClusterConfig.pdCriticalTier, withPDCriticalExclusive(pdClusterConfig.pdCriticalTier!)),
```

- [ ] **Step 5.5: Verify and commit**

```bash
cd pool/eks-freetier && npx tsc --noEmit && npm run fmt && npm run check-fmt && npm run lint
git add pool/eks-freetier/serverless.ts
git commit -m "eks-freetier: wire pdCriticalTier opt-in into serverless-cluster PD

Patches the TidbCluster CR pd component: nodegroup pin via nodeSelector,
critical-app pod label via ComponentSpec.labels, exclusion term appended to
the CR's existing required podAntiAffinity (self-spread term preserved).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
```

Note: the CR's existing required self-spread term (`app.kubernetes.io/component: pd`, hostname) is an array entry; we append, never replace. tidb-operator propagating `pd.labels`/`pd.affinity` to pods must be confirmed live on dev before staging (spec §6) — it is NOT verifiable at preview.

---

### Task 6: pd-tso opt-in (`pool/eks-freetier/serverless.ts`)

**Files:**
- Modify: `pool/eks-freetier/serverless.ts` (`PDClusterConfig`, pd-tso transformations ~line 1150-1163)

- [ ] **Step 6.1: Extend `PDClusterConfig`**

Add next to the other `tsoServer*` fields:

```ts
  tsoServerResourcesRequirements?: ResourcesRequirements;
  // Opt-in to critical-services workload-exclusive scheduling for pd-tso-server
  // (spec-tier name, e.g. "xl").
  tsoServerCriticalTier?: string;
```

- [ ] **Step 6.2: Wire into `pdTsoServerTransformations`** (same style as the existing `tsoServerResourcesRequirements` conditional push, directly after it)

```ts
    if (pdClusterConfig.tsoServerCriticalTier) {
      pdTsoServerTransformations.push(
        statefulSetAddCriticalExclusive("pd-tso", pdClusterConfig.tsoServerCriticalTier)
      );
    }
```

(`statefulSetAddCriticalExclusive` import added in Task 5 Step 5.2.)

- [ ] **Step 6.3: Verify and commit**

```bash
cd pool/eks-freetier && npx tsc --noEmit && npm run fmt && npm run check-fmt && npm run lint
git add pool/eks-freetier/serverless.ts
git commit -m "eks-freetier: wire tsoServerCriticalTier opt-in into pd-tso-server

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
```

---

### Task 7: ai-gateway opt-in (`pool/eks-freetier/ai-gateway.ts`)

**Files:**
- Modify: `pool/eks-freetier/ai-gateway.ts` (interface ~line 24-29, import blocks ~line 5-18, deployment transformations ~line 77-94)

- [ ] **Step 7.1: Extend `AIGatewayConfig`**

```ts
interface AIGatewayConfig {
  image: string;
  qdrantImage: string;
  replicas: number;
  logLevel: string;
  // Opt-in to critical-services workload-exclusive scheduling (spec-tier name,
  // e.g. "xl"). Unset keeps legacy free scheduling.
  criticalTier?: string;
}
```

- [ ] **Step 7.2: Imports**

Add `deploymentAddCriticalExclusive` to the `./lib/transformation` import list, and add:

```ts
import { optionalList } from "./lib/utils";
```

- [ ] **Step 7.3: Wire into the deployment transformations**

```ts
      transformations: [
        deploymentWithImage("qdrant-api", aiGatewayConfig.image),
        deploymentWithReplicas(aiGatewayConfig.replicas),
        deploymentAddArgs(
          // ... unchanged ...
        ),
        deploymentAddEnvs(
          // ... unchanged ...
        ),
      ].concat(
        optionalList(
          !!aiGatewayConfig.criticalTier,
          deploymentAddCriticalExclusive("ai-gateway", aiGatewayConfig.criticalTier!)
        )
      ),
```

(Only the `.concat(...)` is new; the array contents stay untouched.)

- [ ] **Step 7.4: Verify and commit**

```bash
cd pool/eks-freetier && npx tsc --noEmit && npm run fmt && npm run check-fmt && npm run lint
git add pool/eks-freetier/ai-gateway.ts
git commit -m "eks-freetier: wire criticalTier opt-in into ai-gateway

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
```

---

### Task 8: staging-us-east-1-f01 canary config (on `release-0.62`)

**Files:**
- Modify: `pool/eks-freetier/Pulumi.staging-us-east-1-f01.yaml` (fleet ~line 207-219, ai-gateway block ~line 586-590) — **on the `release-0.62` branch**, not main.

Context (revised 2026-07-14): the original canary target dev-us-east-1-f01 was **deleted**, and dev-us-east-1-f02 is torn down — its preview fails for everyone (`getRouteTable/getLoadBalancer/getLaunchTemplate: query returned no results`, helm falling back to k8s v1.20.0), unrelated to this work. The first canary moves to **staging-us-east-1-f01**, which lives on `release-0.62`. On a release branch, `.github/build_matrix.py` takes the `--staging-fallback-dir` path, so a PR's preview matrix resolves to the staging stacks only — the dead dev stacks never enter it.

staging-f01's fleet has exactly ONE entry, `guaranteed` (xlarge mix, primary-zone override to 2xlarge, releaseTag `31v250627`) — no `guaranteed-xl`, which is exactly the drift `validateCriticalTier` exists for. ai-gateway there is `replicas: 1` and requests 1 vCPU / 8Gi (manifest), so tier `large` (~6.9Gi allocatable) does not fit — the canary tier is `xl`. `guaranteed-large` is NOT added (no consumer needs it yet).

- [ ] **Step 8.1: Branch off `release-0.62` and carry the mechanism over**

```bash
git fetch origin release-0.62
git checkout -b iosmanthus/critical-services-exclusive-tiers-r062 origin/release-0.62
git cherry-pick 175ed5d7d 317d36779 333cf4413 9658d0f48 2c9e4a892 a02233df1   # Tasks 2-7, mechanism only
```

Resolve any conflicts against release-0.62's shape of `serverless.ts` / `etcd.ts` (the branch may lag main). Re-verify after cherry-picking: `cd pool && npm install`, then from `pool/eks-freetier`: `npx tsc --noEmit` (baseline = 2 pre-existing `../node_modules` errors), `npm run check-fmt`, `npm run lint`.

- [ ] **Step 8.2: Add the `guaranteed-xl` fleet entry**

Append as a second entry of `eks-freetier:general-guaranteed-fleet` (after `guaranteed`), mirroring the stack's existing family mix and releaseTag (`31v250627` — NOT dev's tag):

```yaml
    - tier: "standard"
      name: "guaranteed-xl" # critical-exclusive tier menu entry (spec 4.1)
      releaseTag: "31v250627"
      instanceTypes: ["m6g.xlarge", "m7g.xlarge", "m6a.xlarge", "m8g.xlarge"]
      max: 5
      primary:
        min: 0
      standby:
        min: 0
      witness:
        min: 0
```

(No primary-zone `instanceTypes` override — unlike the `guaranteed` entry, this tier must be xlarge in every zone or the pin lies about the node size.)

- [ ] **Step 8.3: Opt ai-gateway in**

Add ONE line to the existing `eks-freetier:ai-gateway` block (leave image tags/replicas as they are):

```yaml
    criticalTier: "xl" # workload-exclusive canary; requests 1c/8Gi need xl (large allocatable ~6.9Gi)
```

- [ ] **Step 8.4: Commit and push**

```bash
git add pool/eks-freetier/Pulumi.staging-us-east-1-f01.yaml
git commit -m "staging-us-east-1-f01: add guaranteed-xl tier, opt ai-gateway into exclusive scheduling

First canary of the critical-exclusive contract: new min-0 guaranteed-xl
fleet entry + ai-gateway pinned to it with workload-level anti-affinity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
git push -u origin iosmanthus/critical-services-exclusive-tiers-r062
```

---

### Task 9: PR preview verification and merge gate

**Files:** none (CI verification)

- [ ] **Step 9.1: Open the release-0.62 PR (preview-only) and let CI run**

```bash
gh pr create --base release-0.62 --title "[release-0.62] eks-freetier: critical-services spec tiers + workload-exclusive scheduling" --body "..."
gh pr checks <N>
```

**Do NOT add the `pulumi:up` label** — on this workflow a labeled PR applies on every push. Preview-only until the diff is reviewed. Never run pulumi locally; read previews from CI (repo convention).

- [ ] **Step 9.2: Verify the staging-us-east-1-f01 preview diff**

In the `pulumi (staging-us-east-1-f01)` job log, the Pulumi `Resources:` summary must show ONLY:
- creates for the `guaranteed-xl` nodegroup/ASG/launch-template resources (one set per zone: primary/standby/witness);
- an update to the ai-gateway `Deployment` (pod template: nodeSelector + `critical-app` label + required podAntiAffinity term).

Anything else — especially any etcd / PD / pd-tso / keyspace-etcd resource — means a config gate leaks. Stop and investigate before merging. (Unrelated drift already present on staging, e.g. image bumps merged to release-0.62 but not yet applied, may also appear; classify each line before accepting.)

- [ ] **Step 9.3: Verify the mechanism is inert everywhere else**

`grep -rn "criticalTier\|CriticalTier" pool/*/Pulumi.*.yaml` must match staging-us-east-1-f01 only. Any other stack in the preview matrix must show `unchanged`.

- [ ] **Step 9.4: Merge gate (user decision)**

Report preview results and hand off. Merging to `release-0.62` **auto-applies** (push-to-release-branch triggers `update`), which executes the canary: CA creates one `guaranteed-xl` node and moves ai-gateway onto it. Do not merge without the user's go-ahead.

- [ ] **Step 9.5: Post-apply canary verification (after merge, on staging-us-east-1-f01)**

```bash
aws eks update-kubeconfig --name staging-us-east-1-f01 --profile dbaas-dev --region us-east-1   # CONFIG_MAP auth: use dbaas-dev, not sso
kubectl -n tidb-admin get pods -l app.kubernetes.io/name=ai-gateway \
  -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,APP:.metadata.labels.serverless\.tidbcloud\.com/critical-app'
kubectl get node -l serverless.tidbcloud.com/nodegroup=guaranteed-xl
kubectl get pods -A --field-selector spec.nodeName=<ai-gateway-node> -o wide
```

Expected: ai-gateway pod carries `critical-app=ai-gateway`, runs on a `guaranteed-xl` node, and that node hosts no other non-DaemonSet workload pods.

- [ ] **Step 9.6: Backport the mechanism to `main`**

PR #8284 (branch `iosmanthus/critical-services-exclusive-tiers-spec`, base `main`) carries the same mechanism commits + the spec/plan docs, with NO stack config. Merge it after the staging canary validates, so main and release-0.62 don't diverge.

---

## Explicitly Out of This Plan (follow-up rollout work)

- Opting in the remaining consumers (etcd, PD, pd-tso, keyspace-etcd) on staging, then f04/prod — each is a per-stack YAML change riding release trains, in the spec §5 order (quorum workloads one per apply).
- Adding `guaranteed-large`/other tier entries to stacks as right-sized consumers need them.
- Backports of the mechanism code to `release-0.6x` branches when the first prod stack opts in.
- ACK parity.