# Critical-Services Spec Tiers + Workload-Exclusive Scheduling

Date: 2026-07-13
Scope: `aws-shared-cd/pool/eks-freetier` (mechanism lives in shared `pool/lib`, so ACK can adopt later; this spec only rolls out EKS)

## 1. Background

The `critical-services` pool is built from the `general-guaranteed` fleet:

- `eks-freetier:general-guaranteed-fleet` is a list of entries (per stack: `guaranteed-xl`, `guaranteed-4xl`, `guaranteed-8xl`, …), each becoming its own set of managed nodegroups per zone.
- All entries share the single `eks-freetier:general-guaranteed-nodegroup` config with `component: critical-services`, so every node in every entry carries the same label **and** taint `serverless.tidbcloud.com/node=critical-services:NoSchedule` (`pool/eks-freetier/asg.ts` `nodeLabels()` / taint emission).
- Nodes additionally already carry `serverless.tidbcloud.com/nodegroup: <entry name>` (e.g. `guaranteed-xl`), and `asgTags()` already mirrors all node labels + taints into `k8s.io/cluster-autoscaler/node-template/*` ASG tags, so scale-from-zero works for any label-pinned pod.

Seven workloads consume the pool today, all selecting **only** `serverless.tidbcloud.com/node: critical-services` (+ matching toleration):

| # | Workload | Kind / apply site |
|---|----------|-------------------|
| 1 | `etcd` (tidb-admin) | StatefulSet, `pool/manifests/etcd.yaml` via `pool/eks-freetier/etcd.ts` |
| 2 | `serverless-cluster` PD | `TidbCluster` CR, `pool/manifests/serverless/cluster.yaml` via `pool/eks-freetier/serverless.ts` |
| 3 | `pd-tso-server` | StatefulSet, `pool/manifests/serverless/pd-tso-server.yaml` |
| 4 | `keyspace-etcd-group-*` | StatefulSet, `pool/manifests/serverless/keyspace-etcd-group/sts.yaml` via `pool/lib/serverless/keyspace-etcd-group.ts` |
| 5 | `ai-gateway` | Deployment, `pool/manifests/ai-gateway/deployment.yaml` via `pool/eks-freetier/ai-gateway.ts` |
| 6 | `cassandra-proxy` | Deployment, `pool/manifests/serverless/cassandra-proxy/deployment.yaml` via `pool/eks-freetier/cassandra-proxy.ts`. Config-gated; deployed on prod-ap-northeast-1-f01/f02 only. **Out of scope** (see 3). |
| 7 | `tiflash-manager` | Deployment, `pool/manifests/serverless/tiflash-manager/deployment-tiflash-manager.yaml` via `pool/eks-freetier/tiflash-manager.ts`. Config-gated; only ever enabled on the (now deleted) dev-us-east-1-f01 — no live stack runs it. **Out of scope** (see 3). |

Consumers 1–5 are the migration scope; 6–7 stay on legacy scheduling (decision recorded in 7).

Because nothing pins a node size, two problems recur:

1. **No size targeting.** A pod that fits several fleet entries can be scaled up onto any of them; with the priority expander all entries tie at the same priority (the 4xl/8xl wrong-size scale-up incident). Right-sizing a workload (e.g. f04's PD-on-xlarge downsize) currently requires deleting every other entry from the stack.
2. **Free co-location.** Workloads share nodes arbitrarily. The only "exclusivity" today is ad hoc: etcd's default `requests == limits == 8Gi` happens to fill a `guaranteed-xl` node by request. Noisy-neighbor and blast-radius characteristics are accidental, not designed.

## 2. Goals

- Offer a menu of **spec tiers** inside the critical-services pool; an opted-in workload pins an explicit tier.
- **Workload-level exclusivity** among opted-in workloads: two different workloads never share a node. Replicas of the *same* workload remain governed by that workload's own (pre-existing) anti-affinity.
- **Per-stack, per-workload opt-in.** A workload without the new config knob behaves exactly as today, on the same nodes. Migration is gradual; nothing forces a flag day.
- **Converging semantics** ("C1"): exclusivity between any two *migrated* workloads holds immediately; exclusivity against *un-migrated* workloads is established as those workloads migrate. Once the five in-scope consumers are migrated, exclusivity is complete on every stack except where an out-of-scope consumer runs (see 3).

## 3. Non-Goals

- **No new node pool / component / taint.** A dedicated `critical-exclusive` pool (taint-hard isolation) was proposed and rejected during brainstorming; we stay inside the existing `critical-services` label/taint.
- **Not pod-level exclusivity** (1 pod = 1 node). A workload's replicas may share a node if its own anti-affinity allows it.
- **No protection against un-migrated legacy workloads** co-locating with migrated ones during the migration window. This is the accepted cost of C1; it converges to zero.
- **Not reusing `serverless.tidbcloud.com/tier`** (hot/standard/warm/cold) to express size — that label keeps its existing storage-temperature semantics.
- **`cassandra-proxy` and `tiflash-manager` are out of scope** (user decision, 2026-07-13). They keep legacy scheduling indefinitely. Consequence: on the stacks where they run (cassandra-proxy on prod-ap-northeast-1-f01/f02; tiflash-manager nowhere live today), they remain able to co-locate onto migrated workloads' nodes — the exclusivity guarantee carries this known, stack-local exception. Bringing them in later is a one-field opt-in like any other consumer.
- **No admission/policy enforcement** (Kyverno/OPA). The contract is enforced by shared helper code + review; all consumer manifests live in this repo.
- **No ACK rollout.** The helper lands in shared `pool/lib` and is designed to be ACK-compatible, but verifying ACK node labels and migrating ACK stacks is future work.

## 4. Design

### 4.1 Pool layer: tiers are fleet entries (YAML-only)

The tier menu is the existing fleet mechanism; no infra code changes. Standard entry names (kept identical across stacks, since shared code refers to them by name):

| Tier | Fleet entry name | Shape | Instance types |
|------|------------------|-------|----------------|
| `large` | `guaranteed-large` | 2 vCPU / 8 Gi | `m*.large` |
| `xl` | `guaranteed-xl` | 4 vCPU / 16 Gi | `m*.xlarge` |
| `2xl` | `guaranteed-2xl` | 8 vCPU / 32 Gi | `m*.2xlarge` |
| `4xl` | `guaranteed-4xl` | 16 vCPU / 64 Gi | `m*.4xlarge` |
| `8xl` | `guaranteed-8xl` | 32 vCPU / 128 Gi | `m*.8xlarge` |

Each stack carries only the subset it needs. Adding a tier to a stack = adding one entry to `eks-freetier:general-guaranteed-fleet` in that stack's `Pulumi.<stack>.yaml` (min 0 everywhere; CA scales from zero on demand).

`guaranteed-large` is the floor of the menu, intended for minimal deployments (right-sized etcd, small gateways). Its bandwidth is burst-based (EC2 `DescribeInstanceTypes`, 2026-07-13): sustained network baseline 0.75–0.94 Gbps and EBS baseline ~79 MB/s / 3600 IOPS (below the gp3 default of 125 MB/s — the instance, not the volume, is the bottleneck), bursting to 12.5 Gbps / 1250 MB/s on credits. Fine for steady control-plane traffic; etcd snapshot/defrag and the S3 backup CronJob ride burst credits (infrequent, acceptable). Note PD does not fit: even the f04-downsized PD (2 vCPU / 8 Gi requests) exceeds a large node's ~1.9 vCPU / ~6.9 Gi allocatable — PD's floor stays `xl`.

A `medium` tier was evaluated and rejected: x86 m-family has no `.medium` (the tier would be Graviton-only, losing multi-family capacity fallback), EBS baseline is ~39 MB/s on 1 vCPU, ENI-bound max-pods is 8 (DaemonSets alone nearly exhaust it under the current CNI setup), and ~3.3 Gi allocatable barely fits even a right-sized etcd.

### 4.2 Scheduling contract (the three-piece set)

An opted-in workload gets three additions to its pod template. Existing `serverless.tidbcloud.com/node: critical-services` selection and toleration stay untouched.

```yaml
# 1. Spec pin — nodeSelector ANDs with any existing nodeAffinity
nodeSelector:
  serverless.tidbcloud.com/nodegroup: guaranteed-<size>

# 2. Identity label on the pod
metadata:
  labels:
    serverless.tidbcloud.com/critical-app: <app>   # etcd | pd | pd-tso | keyspace-etcd | ai-gateway

# 3. Workload-level mutual exclusion (appended to existing anti-affinity terms)
podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: kubernetes.io/hostname
      namespaceSelector: {}          # all namespaces (k8s >= 1.22; fleet is on 1.31+)
      labelSelector:
        matchExpressions:
          - { key: serverless.tidbcloud.com/critical-app, operator: Exists }
          - { key: serverless.tidbcloud.com/critical-app, operator: NotIn, values: ["<app>"] }
```

Why this shape:

- `Exists` **and** `NotIn` (ANDed) match exactly "a pod that has declared a critical-app identity, and it is not mine". DaemonSets and system pods carry no such label → unaffected. Legacy un-migrated critical workloads carry no such label → unaffected (C1 convergence).
- Kubernetes required anti-affinity is **symmetric**: a migrated pod already on a node also repels later pods of other migrated apps, even before those pods' own terms are evaluated.
- Same-app replicas don't match `NotIn [<app>]` → replica spreading remains the workload's own business (all five in-scope consumers already have hostname anti-affinity or spread constraints where they need them).

### 4.3 Delivery: config-gated transformation in shared lib

One helper in `pool/lib/transformation.ts`, mirroring the existing `statefulSetAddNodeSelector` pattern:

```ts
// Applies the three-piece contract to a Deployment/StatefulSet pod template.
// - adds nodeSelector {nodegroup: guaranteed-<tier>}
// - adds pod label critical-app=<app>
// - APPENDS the exclusion term to podAntiAffinity.required (never replaces
//   existing terms — several consumers already carry their own anti-affinity)
withCriticalExclusive(app: string, tier: string): Transformation
```

Each workload's existing per-stack config block gains one optional field that acts as the whole opt-in switch (unset ⇒ bit-for-bit legacy behavior, same pattern as `etcd.resources`):

```yaml
eks-freetier:etcd:
  criticalTier: "xl"        # maps to nodegroup guaranteed-xl
```

Wiring per consumer:

| Workload | Config key | Mechanism |
|----------|-----------|-----------|
| etcd | `eks-freetier:etcd.criticalTier` | `etcd.ts` ConfigFile transformation. **Must be folded into `EtcdRollingUpgrade.versionHash`** (only when set), or the partition-gated rollout never fires — same trap as `etcd.resources` (fixed in 2394bf960). |
| keyspace-etcd-group | field on the existing keyspace-etcd-group spec config | `pool/lib/serverless/keyspace-etcd-group.ts`; same `versionHash` requirement as etcd. |
| serverless-cluster PD | field on the serverless-cluster config block | `serverless.ts` patches the `TidbCluster` CR: `pd.nodeSelector` += nodegroup pin, `pd.labels` += critical-app, `pd.affinity.podAntiAffinity.required` += exclusion term (PD already carries a self-spread required term — append). Operator propagates `pd.labels` to pods (ComponentSpec.Labels); verify on dev before trusting. |
| pd-tso-server | field on its config block | transformation at its apply site in `serverless.ts`. |
| ai-gateway | field on its config block | `ai-gateway.ts` + helper. |

Exact config-key names for consumers 2–5 are confirmed at implementation time; the pattern is identical for all.

**Fail-fast validation:** the helper (or a small shared check) reads `general-guaranteed-fleet` for the current stack and errors at preview time if `guaranteed-<tier>` is not among the entries. This turns "opted into a tier the stack doesn't have" from Pending-pods-at-2am into a red preview.

### 4.4 Sizing guidance

Exclusivity is guaranteed by anti-affinity, not by request math — but an opted-in workload should still set `requests` close to its tier's allocatable (minus DaemonSet/kube-reserved overhead, roughly 0.5 vCPU / 1 Gi on these sizes). Otherwise the workload exclusively holds a node it barely uses; picking a smaller tier is the fix. That is the point of having a menu.

Bandwidth is a sizing dimension too: `large` (and anything below `4xl`) has burst-type network/EBS baselines (see 4.1) — a workload with sustained throughput needs belongs on a bigger tier even if its CPU/memory fit a smaller one.

### 4.5 Autoscaling behavior

- **Scale-from-zero** works: node-template tags already carry the `nodegroup` label and the taint; an empty template node trivially satisfies the anti-affinity simulation.
- **Expander ties disappear for migrated workloads:** a pinned pod matches exactly one fleet entry's ASGs, so the priority expander has nothing to tie-break (root cause of the 4xl/8xl wrong-size scale-up).
- **Scale-in:** CA's placement simulation respects required anti-affinity symmetry, so it will not consolidate two exclusive workloads onto one node. `safe-to-evict: false` pods (etcd/PD/pd-tso) keep pinning their nodes exactly as today.
- **Cost:** exclusivity raises node count relative to today's densest packing; per-workload tier right-sizing (4.4) is the mitigation. Expect a temporary bump during migration while both scheduling regimes coexist.

## 5. Migration & Rollout

- **Stack order:** staging-us-east-1-f01 → prod canary us-west-2-f04 (already right-sized to xl) → remaining prod stacks riding their release branches. (The dev EKS stacks are gone — dev-us-east-1-f01 was deleted and f02 torn down — so staging is the first canary. Work lands on `release-0.62`, whose PR preview matrix resolves to the staging stacks only; the change is backported to `main` separately.)
- **Workload order per stack:** stateless first (`ai-gateway`), then `pd-tso`, `keyspace-etcd`, `etcd`, PD last.
- Each opt-in is a pod-template change ⇒ one rolling restart of that workload. etcd/keyspace-etcd roll one-at-a-time via `EtcdRollingUpgrade`; PD via the operator's `canaryUpgrade`. None of these have PDBs (known state) — do not migrate two quorum workloads in the same apply.
- A freshly opted-in workload may trigger CA scale-up (it now refuses nodes hosting other migrated apps). Expected and self-limiting.
- **Rollback** = unset `criticalTier` for that workload on that stack; next apply rolls it back to legacy scheduling.
- **Validation:** Pulumi previews via PR CI only (never local `pulumi up`). Post-apply per stack: pods Running on the pinned nodegroup (`kubectl get pods -o wide` + node labels), no two distinct `critical-app` values on one node, CA scales the pinned tier from zero.

## 6. Risks & Edge Cases

| Risk | Handling |
|------|----------|
| `versionHash` omission for etcd/keyspace-etcd ⇒ template change silently never rolls | Called out in 4.3; implementation must hash `criticalTier` (only when set, to avoid spurious rollout Jobs on untouched stacks). |
| Tier missing from a stack's fleet ⇒ Pending pods | Preview-time fail-fast validation (4.3). |
| Migration-window co-location with legacy workloads | Accepted C1 semantics; converges as migration completes. |
| Operator not propagating PD pod labels/affinity as expected | Verify live on staging before any prod stack; PD is last in the workload order anyway. |
| Two exclusive pods pending simultaneously ⇒ serialized CA scale-ups | Acceptable on pools this small; no action. |
| A future workload tolerates the critical-services taint but skips the contract | It behaves like a legacy consumer (can co-locate). Convention + review; helper makes compliance a one-liner. Admission enforcement deliberately out of scope. |

## 7. Decisions Recorded from Brainstorming

- Exclusivity granularity: **workload-level** (not pod-level, not tier-level).
- Migration: **opt-in, gradual** (not flag-day, not new-workloads-only).
- Mechanism: **in-pool scheduling layer** (“option C”) chosen over a new taint-isolated pool (A) and per-workload nodegroups (B).
- Migration-window semantics: **C1 converging** (`Exists` + `NotIn`) over C2 immediate (namespace-scoped bare `NotIn`), trading window-period completeness for zero DaemonSet risk and no namespace allowlist.
- Naming: reuse `serverless.tidbcloud.com/nodegroup` for spec pinning (no new node label); identity label key `serverless.tidbcloud.com/critical-app`; config knob `criticalTier`.
- Scope cut (2026-07-13): `cassandra-proxy` and `tiflash-manager` excluded from migration; they keep legacy scheduling, accepting the stack-local co-location exception documented in 3.
- Tier menu floor (2026-07-13): `guaranteed-large` added for minimal deployments; `medium` evaluated and rejected (Graviton-only, 39 MB/s EBS baseline, max-pods 8, allocatable too small — see 4.1).