--- title: Single-AZ HA Improvement via EC2 Placement Groups updated: 2026-08-04 rev: 4 (round 4 PR review fixes: F1 reframed as "gate computed before topology normalization", N3 history reframed via #2609 → #2791 → #2805, built-in NodePool protection deferred to follow-up; annotation gate and one-time reset in DPM rolled back) watches: - internal/dataplane-manager/api/v1alpha1/nodepool.go - internal/dataplane-manager/pkg/namer/namer.go - internal/dataplane-manager/pkg/controllers/nodepool/task/eks/placementgroup.go - internal/dataplane-manager/pkg/controllers/nodepool/task/eks/nodegroup.go - internal/dataplane-manager/pkg/controllers/nodepool/taskbuilder.go - internal/dataplane-manager/pkg/service/dataplane/dataplaneservice.go - api/tidbcloud/dataplane-manager/v1/resources.proto - api/cluster-service-ng/v1beta/resources.proto - domain/workflow/utils.go - domain/workflow/action/apply_component_group.go - domain/workflow/action/apply_pd_group.go - domain/workflow/consts.go - domain/workflow/action/create_node_instance_group.go - domain/app/cron/shortage_detector.go - pkg/runtime/actions/action.go - domain/server/node_instance_group.go - deployments/templates/pd_group.templ - deployments/templates/tikv_group.templ --- ## Summary In single-AZ deployments, AWS may co-locate all EC2 instances for a component on the same physical rack. A single rack-level failure (power, top-of-rack switch) can then take down quorum for PD or TiKV simultaneously — turning a survivable failure into a cluster outage. This design introduces EC2 Partition Placement Groups managed via Crossplane (`ec2v1beta1.PlacementGroup`) to spread instances across distinct physical hardware failure domains within the AZ. Full HA requires three layers configured together: 1. **Physical** — EC2 partition PG ensures different partitions land on different racks. 2. **Scheduling** — per-partition NodeGroups and `topologySpreadConstraints` spread pods evenly; cluster-autoscaler scales the correct partition NodeGroup. 3. **Data** — PD `location-labels` and TiKVGroup `schedulePolicies` ensure Raft replicas for the same range do not share a partition. ## Problem Without placement constraints, the EKS node provisioner places all instances for a single-AZ cluster wherever AWS finds capacity. This can put all PD or TiKV nodes on a single rack. A rack failure then eliminates Raft quorum before any automatic recovery can act. Multi-AZ clusters are protected because EKS NodeGroups span availability zones. Single-AZ clusters have no equivalent protection today. EC2 Partition Placement Groups solve single-AZ rack isolation, but two structural constraints shape the implementation: - Partition PGs span exactly one AZ. A multi-AZ cluster **cannot** host a PG, by definition. - The feature is EKS-only: Azure / GCP / ACK have their own single-AZ primitives and do not need a DPM-managed PG. So the only configuration that could possibly opt in is **AWS + single AZ**. Everything else is automatically out of scope. The runtime derives eligibility from that check; the caller-supplied `placement_group_enabled` flag is accepted but ignored (kept in proto / CRD for back-compat, marked Deprecated). See "Source of truth" below for the durable-record contract. ## Why Partition Placement Groups AWS offers three placement group strategies: | Strategy | Isolation unit | Instance cap per AZ per group | |----------|---------------|-------------------------------| | `cluster` | None — tightly packs instances | No cap | | `spread` | Each instance on a separate rack | **7** | | `partition` | Each partition on separate racks | No practical cap | `spread` is rejected: its hard cap of 7 instances per AZ per group blocks horizontal scaling of TiDB nodes. `partition` with 3 partitions provides equivalent rack isolation without an instance-count limit. 3 partitions matches the Raft replication factor for PD and TiKV — no single rack can take down quorum. ## Source of truth `NodePoolSpec.PlacementGroupEnabled` is the durable record read by the EKS controller on every reconcile (Create / PostCreate / Delete / PostDelete / sibling counts / NodeGroup partition branch). It is **authoritative for the NodePool side** — set at `CreateNodePool` time and never re-derived by the controller. On the CSNG side, the gate is resolved through a **two-step inheritance chain** before it reaches the wire: 1. **Sibling inheritance** (CSNG `create_node_instance_group.go`'s `resolvePlacementGroupEnabled`): if any non-builtin workload NodePool exists in the dataplane, inherit `Spec.PlacementGroupEnabled` from it. This is the durable-record preservation step — a legacy single-AZ, PG-disabled primary keeps its fallback on the legacy per-zone path even though the dataplane topology itself is eligible for partition PGs. 2. **DAG-arg fallback** (CSNG `node_instance_group.go` → DAG arg → `resolvePlacementGroupEnabled`): if no workload sibling exists (first NodePool in a fresh dataplane), fall back to `SharedPool.Provider.PlacementGroupEnabled`, which is set at `CreateSharedPool` time from `initConfig.NodeZones` + provider type. For new SharedPools this matches the topology derivation; for legacy SharedPools it is the legacy false record. 3. **Default false**: extreme edge case where the DAG arg is missing. DPM (`dataplaneservice.go` `CreateNodePool`) **honours the wire value** and persists it on `NodePoolSpec.PlacementGroupEnabled`. A topology sanity check rejects `placement_group_enabled=true` on multi-AZ or non-EKS dataplanes so a stale or buggy caller cannot pin an unsupported flag onto the spec; `false` is always accepted (legacy per-zone behaviour is safe on every topology, and `false` is also the explicit durable record for a legacy single-AZ, PG-disabled pool). Concretely: - New AWS single-AZ pool → first NodePool: `Spec.PlacementGroupEnabled=true` (from SharedPool gate or sibling chain); subsequent siblings inherit `true`. - Legacy single-AZ EKS pool created before this design → already on `Spec.PlacementGroupEnabled=false`; siblings inherit `false` and stay on the legacy per-zone path (the regression fgksgf called out in PR review). - New multi-AZ pool, single-AZ non-AWS pool, or any ACK/AKS pool → `Spec.PlacementGroupEnabled=false` at first NodePool; topology sanity check rejects a sibling attempting to claim `true`. - Legacy pool → keeps its original `Spec.PlacementGroupEnabled` value across reconciles; topology changes after creation do not flip eligibility. Re-creation is the only way to migrate. Built-in NodePools (admin / o11y / ...) are created by the dataplane task in `pkg/controllers/dataplane/task/nodepool.go` with annotation `tidbcloud.com/builtin-nodepool=true`. CSNG's sibling lookup filters builtins out via the `nodegroup.LabelNodeSpec` discriminator on `Spec.Labels` (workload pools carry the label via `BuildPrimaryLabels`; builtins carry no `Spec.Labels`). The original `#2609` design rule ("built-in NodePools do not use PG") is still authoritative — built-ins are out of the PG scope by design. **However, explicit enforcement of that rule (annotation gate in the EKS PG controller + a one-time spec reset for legacy `Spec=true` built-ins) is deferred to a follow-up**; see "Built-in NodePool protection (deferred)" below for the historical chain and rationale. This PR treats built-in NodePools as ordinary workload pools on the PG side, so legacy built-ins with `Spec.PlacementGroupEnabled=true` (left over from `#2791`/`#2805`) keep their current PG behaviour until the follow-up lands. The CSNG-side `SharedPool.Provider.PlacementGroupEnabled` (set at `CreateSharedPool` time from `initConfig.NodeZones` + provider type) is the **per-dataplane durable record** for the scheduling side: every consumer that drives `location-labels`, PD/TiKV config topology, and the component `BuildTopologies` chain reads that single source. The DAG arg carries this value as the first-pool fallback in the sibling-inheritance chain. ## Scope In scope: - `internal/dataplane-manager`: new `placementGroup` task, per-partition NodeGroup loop in `nodeGroup` task, `PlacementGroupName` namer helper, `NodePoolSpec.PlacementGroupEnabled` field (already present, now treated as the durable record). - `api/tidbcloud/dataplane-manager/v1/resources.proto`: keep `placement_group_enabled` on `NodePool`; the comment now describes the source-of-truth chain (CSNG sibling inheritance → SharedPool gate → DPM topology sanity check), not Deprecated. - `internal/dataplane-manager/pkg/service/dataplane/dataplaneservice.go`: `CreateNodePool` honours `req.NodePool.PlacementGroupEnabled` and persists it on `NodePoolSpec.PlacementGroupEnabled`. When the caller requests `true`, the dataplane topology must be single-AZ EKS — otherwise the request is rejected with `ParameterInvalid`. `false` is always accepted. `resolvePlacementGroupEnabled` becomes a topology-sanity-check helper used only when the wire value is `true`. - `api/cluster-service-ng/v1beta/resources.proto`: keep `placement_group_enabled` on `Provider` with Deprecated comment. - `pkg/runtime/actions/action.go`: add `ActionArgBool` type and `GetBool` method. - `domain/workflow/consts.go`: one new workflow arg key (`PlacementGroupEnabledKey`) for passing PG enabled flag into `CreateNodeInstanceGroupAction`. - `domain/server/node_instance_group.go`: extend `buildNodeInstanceGroupDagArgs` to accept `placementGroupEnabled bool`; resolve the per-dataplane gate from `SharedPool.Provider` at the `PlanCreateNodeInstanceGroup` call sites; pass `false` at `PlanDeprecateNodeInstanceGroup` and retire sites (these plans do not consume the value). - `domain/server/create_sharedpool.go`: derive `Provider.PlacementGroupEnabled` from `initConfig.NodeZones` after provider/HA resolution; drop the AWS-only validation that used to reject user input; the caller-supplied field is parsed but never persisted. - `domain/workflow/action/create_node_instance_group.go`: implement `resolvePlacementGroupEnabled` (sibling inheritance via `ListNodePools` + DAG-arg fallback); send the result on the `CreateNodePoolRequest` wire. - `domain/workflow/utils.go`: extend `BuildTopologies` to emit `tidbcloud.com/pg-partition` topologies when PG enabled. - `domain/workflow/action/apply_component_group.go`: read the gate from `SharedPool.Provider.PlacementGroupEnabled` (not the zone count) and pass to `BuildTopologies`; inject `location-labels` for PD and PD_SCHEDULING components. - `domain/app/cron/shortage_detector.go`: inherit `PlacementGroupEnabled` from the primary NodePool's spec, never from the shared-pool gate or zone count. The receiving-end change in DPM (honouring wire value) is what makes the inheritance chain deliver the durable record. Out of scope: - Automatic migration of existing NodeGroups into placement groups (requires NodePool recreation). - ACK (Alibaba Cloud) and AKS (Azure) providers — all EKS changes are scoped to the `eks/` task tree, and the gRPC `CreateNodePool` path returns `PlacementGroupEnabled=false` for non-EKS providers without consulting topology. - CA configuration (`--balancing-ignore-labels`) — required operational tuning but not a code change in this repo. - **Built-in NodePool protection** — annotation gate in the EKS PG controller and one-time spec reset for legacy `Spec=true` built-ins. See the dedicated subsection below for the historical chain (`#2609` → `#2791` → `#2805`) and why restoring the original rule requires more than flipping a spec field. ## Design ### API Changes #### `NodePoolSpec` — durable record `internal/dataplane-manager/api/v1alpha1/nodepool.go` ```go PlacementGroupEnabled bool `json:"placementGroupEnabled,omitempty"` ``` `PlacementGroupEnabled` is the durable record of "AWS + single-AZ?" at NodePool creation. The EKS controller reads it on every reconcile (Create / PostCreate / Delete / PostDelete / sibling counts / NodeGroup partition branch); the gRPC `CreateNodePool` path writes it; subsequent reconciles never overwrite it. Defaults to `false`. Strategy (`partition`) and partition count (`3`) are hardcoded in the task. #### Public `NodePool` proto — deprecated comment removed in PR #3040 `api/tidbcloud/dataplane-manager/v1/resources.proto` ```protobuf // PlacementGroupEnabled is the durable per-NodePool gate set at // NodePool creation and read by the EKS controllers on every reconcile. // The source-of-truth chain lives on the CSNG side // (see docs/design/single-az-ha-placement-groups.md). bool placement_group_enabled = ; ``` The wire field **is** honoured by `dataplaneservice.go` `CreateNodePool` with a topology sanity check (`true` rejected on multi-AZ / non-EKS with `ParameterInvalid`; `false` always accepted). `ListNodePools` returns it on the CR. The earlier "Deprecated: ignored" comment was removed because the field is no longer inert — it carries the producer's intent through the chain. #### `Provider` proto — durable per-pool gate `api/cluster-service-ng/v1beta/resources.proto` ```protobuf // Durable per-pool gate. CSNG server writes this at CreateSharedPool // time from initConfig.NodeZones + provider type; every scheduling // consumer reads it. bool placement_group_enabled = ; ``` The CSNG server derives `Provider.PlacementGroupEnabled` from the workload's `initConfig.NodeZones` + provider type ("AWS + single-AZ?") at CreateSharedPool time and persists it on the shared pool. Every scheduling consumer (NodePool creation, NIG creation, shortage-detector fallback, component topology, PD location labels) reads it via `placementGroupEnabledForDataplanes`. ### EKS Infrastructure Layer (dataplane-manager) #### New task: `placementGroup` `internal/dataplane-manager/pkg/controllers/nodepool/task/eks/placementgroup.go` The eligibility check is `placementGroupEnabled(np *v1alpha1.NodePool) bool`: ```go func placementGroupEnabled(np *v1alpha1.NodePool) bool { if np == nil { return false } return np.Spec.PlacementGroupEnabled } ``` The annotation-gate variant of this helper (which would short-circuit on `tidbcloud.com/builtin-nodepool=true`) is the explicit built-in protection described in the deferred subsection below; it is not added in this PR. Reading from spec (not from dataplane or VPC) makes the value the source of truth itself: it survives across deletion reconciles (the NodePool CR persists while the parent dataplane and VPC may not), preserves the legacy per-pool intent for pools that were created before this design, and never silently flips eligibility on a topology change after creation. Re-deriving from live topology on every reconcile was the original design and broke three production guarantees (orphan PG on ZONAL teardown, silent migration of legacy single-AZ/PG-disabled pools, producer/consumer rollout skew) — see the PR description for the regression cases. - **Create**: returns `Done` immediately if `!placementGroupEnabled(np)`. Otherwise applies one `ec2v1beta1.PlacementGroup` CR (strategy=`partition`, partitionCount=`3`) using the stable name. If a CR with that name already exists (shared by another NodePool for the same instance type), verifies it is Ready and returns Done without re-applying. - **PostCreate**: polls by name, requeues until `xpv1.TypeReady`. - **Delete**: lists all NodePool CRs in the dataplane namespace whose `placementGroupEnabled` returns true and that share the same instance type. If more than one still exists (i.e., a fallback pool also references this PG), skips deletion and returns Done. Otherwise proceeds to patch and delete the PG CR. The reconciler skips `loadDependencies` on Delete reconciles, so VPC is nil and `t.GetNodeZones()` is empty — but the eligibility check reads from spec, not VPC, so Delete still runs correctly. This is the orphan-PG fix. - **PostDelete**: verifies the CR is gone. Naming: `{instanceType}-{namespacedHash}-pg` via a new `PlacementGroupName(instanceType, namespacedHash string) string` helper in `pkg/namer/namer.go`. The resource prefix (`primary` / `fallback-*`) is intentionally excluded so that primary and same-instance-type fallback NodePools produce the same name and share one PG in AWS. Example CR: ```yaml apiVersion: ec2.aws.upbound.io/v1beta1 kind: PlacementGroup metadata: name: tidbcloud-m8g-4xlarge-dataplane-metada-8b3eb320-pg spec: forProvider: region: us-east-1 strategy: partition partitionCount: 3 managementPolicies: [Create, Update, Observe, LateInitialize] ``` #### Modified task: `nodeGroup` — partition loop `internal/dataplane-manager/pkg/controllers/nodepool/task/eks/nodegroup.go` When `placementGroupEnabled(t.NodePool)` returns true, the existing zone loop is replaced by a partition loop over 3 hardcoded partitions. All partitions share the same subnet (single AZ). ```go if placementGroupEnabled(t.NodePool) { const partitionCount = 3 zone, subnetID, err := t.singleZoneSubnetID() // errors if multiple zones configured for i := 0; i < partitionCount; i++ { // LaunchTemplate: placement[0].groupName = pgName, placement[0].partitionNumber = i // NodeGroup Labels: "tidbcloud.com/pg-partition" = strconv.Itoa(i) } } else { // existing zone loop — unchanged } ``` New helpers: - `perPartitionNodegroupName(zone string, partitionIdx int, ...) string` — pattern: `{prefix}-{instanceType}-{zone}-p{i}-{k8sVersion}`. - `singleZoneSubnetID() (zone, subnetID string, err error)` — returns the first zone and subnet; errors if multiple zones are present. The `tidbcloud.com/pg-partition` label is static in the NodeGroup spec. EKS propagates it to every node in the group; the cluster-autoscaler reads it from the node template for topology simulation — no runtime labeling needed. Example resources for partition p0: ```yaml # LaunchTemplate spec: forProvider: placement: - groupName: tidbcloud-m8g-4xlarge-dataplane-metada-8b3eb320-pg partitionNumber: 0 # float64 in Go type; unquoted in YAML ``` ```yaml # NodeGroup spec: forProvider: subnetIds: [subnet-0aaa111bbb222ccc3] # same for all 3 partition NodeGroups labels: tidbcloud.com/pg-partition: "0" ``` #### PG fields in `CreateNodeInstanceGroupAction` `domain/workflow/action/create_node_instance_group.go` currently receives only `DataplaneIdKey` and `NodeInstanceGroupKey`. `NodeInstanceGroup` is a region-level entity and must not carry cluster-level PG settings. **`ActionArgBool` and `GetBool`** — `pkg/runtime/actions/action.go` currently only has `ActionArgString`/`GetString` and `ActionArgInt`. Add `ActionArgBool` following the same pattern: ```go type actionArgBool bool func ActionArgBool(b bool) *actionArgBool { arg := actionArgBool(b) return &arg } func (arg actionArgBool) String() string { return strconv.FormatBool(bool(arg)) } func (arg *actionArgBool) FromString(s string) error { v, err := strconv.ParseBool(s) if err != nil { return err } *arg = actionArgBool(v) return nil } func (args ActionArgs) GetBool(name string) (bool, error) { val, ok := args[name] if !ok { return false, fmt.Errorf("ActionArgs.GetBool: arg `%v` not found", name) } return strconv.ParseBool(val.String()) } ``` **`buildNodeInstanceGroupDagArgs`** — extend `domain/server/node_instance_group.go` to accept `placementGroupEnabled bool`: ```go func buildNodeInstanceGroupDagArgs(ctx context.Context, group *entity.NodeInstanceGroup, dataplaneName string, placementGroupEnabled bool) (actions.ActionArgs, error) { groupData, err := protojson.Marshal(group) if err != nil { return nil, fmt.Errorf("failed to marshal node instance group: %w", err) } return actions.ActionArgs{ workflow.DataplaneIdKey: actions.ActionArgString(dataplaneName), workflow.NodeInstanceGroupKey: actions.ActionArgString(string(groupData)), workflow.PlacementGroupEnabledKey: actions.ActionArgBool(placementGroupEnabled), utils.TraceIdKey: utils.GetTraceId(ctx), }, nil } ``` The function has 5 call sites. Only the two `PlanCreateNodeInstanceGroup` sites pass the real value from `SharedPool.Provider`; the `PlanDeprecateNodeInstanceGroup` and retire sites pass `false`: ```go // PlanCreateNodeInstanceGroup call sites — load sharedPool first: args, err := buildNodeInstanceGroupDagArgs(ctx, group, dataplaneName, sharedPool.GetProvider().GetPlacementGroupEnabled()) // PlanDeprecateNodeInstanceGroup / retire call sites: args, err := buildNodeInstanceGroupDagArgs(ctx, group, dataplaneName, false) ``` One new constant is added to `domain/workflow/consts.go`: ```go PlacementGroupEnabledKey = "placement_group_enabled" ``` In `create_node_instance_group.go`, the action reads the arg: ```go pgEnabled, _ := args.GetBool(workflow.PlacementGroupEnabledKey) // false if key absent ``` And forwards it in `CreateNodePoolRequest`: ```go NodePool: &dplanemgrv1.NodePool{ // ... existing fields ... PlacementGroupEnabled: pgEnabled, }, ``` The shortage detector path is unchanged: it reads `PlacementGroupEnabled` directly from `ListNodePools` on the primary pool. #### Task pipeline wiring `internal/dataplane-manager/pkg/controllers/nodepool/taskbuilder.go` ```go case consts.ProviderTypeEKS: tasks = append(tasks, ekstask.NewPlacementGroup(c), // must precede NodeGroup ekstask.NewNodeGroup(c), ) ``` `PlacementGroup.PostCreate` polls until Ready, so `NodeGroup.Create` runs only after the placement group exists in AWS. #### Built-in NodePools (APPBox-created) — explicit protection deferred to follow-up Built-in NodePools are created from `APPBoxConfig.NodePools` in `pkg/controllers/dataplane/task/nodepool.go` with annotation `tidbcloud.com/builtin-nodepool: "true"`. They are worker pool templates — admin/o11y are created by the appbox controller as EKS NodeGroups (Upbound resources), not regional NodePool CRs. The annotation is consumed only by CSNG's sibling-inheritance filter (workload chain); the EKS PG controller does not gate on it. **Producer-side responsibility**: the CSNG gRPC `CreateNodePool` path is the producer for non-AppBox workload NodePools. For AppBox clusters the gRPC path does NOT fire on bootstrap (the 16 worker pools are pre-created from `APPBoxConfig.NodePools`), so the static controller in `pkg/controllers/dataplane/task/nodepool.go` IS the producer for these pools and must stamp `Spec.PlacementGroupEnabled = isSingleAzEKS(Config.Provider, DataPlane.Spec.NodeZones)` at create time. The EKS PG controller then reads the durable spec and creates the partition placement group on its next reconcile. The reconcile path does NOT rewrite `Spec.PlacementGroupEnabled` on existing built-in pools. Legacy `Spec=true` pools (created under `#2791` from a per-pool yaml knob) are preserved across reconciles even if topology later changes — a topology flip must not silently migrate a legacy pool off the partition path (which would orphan its PG and 3 partition NodeGroups). Clusters deployed under the post-`#3040` / pre-fix window with `Spec=false` are migrated by deleting and recreating the cluster; there is no automated reset in the controller. Restoring the original `#2609` annotation gate (skip PGs entirely for any pool with the annotation) is still deferred to the follow-up described at the end of this section — the annotation is only a CSNG filter, not an EKS PG controller gate, today. **Historical chain** (why the deferred work is needed, not just a code cleanup): 1. `#2609` (initial PG design) — the design rule was: built-in NodePools do not set `Spec.PlacementGroupEnabled`, so the proto default `false` applies. Implementation honoured this by simply not assigning the field on the built-in CR. CSNG sibling lookup filters built-ins out via the `nodegroup.LabelNodeSpec` discriminator on `Spec.Labels`. No explicit gate in the EKS PG controller — the constraint was implicit in "default false on built-in". 2. `#2791` — added `placementGroupEnabled` to `APPBoxConfig.NodePools` and explicitly wrote it onto `Spec.PlacementGroupEnabled` on built-in CRs; also actively updated existing built-in NodePools. Tests asserted that the built-in field can become `true`. The design doc was not updated to reflect the change, so the rule became implicit in the OPPOSITE direction. 3. `#2805` — backport of `#2791` into `release-0.28`. After this point, built-in NodePools on single-AZ EKS envs where the config knob was set were running on PGs and had partition NodeGroups in production. 4. `#3040` — honours the wire value on `CreateNodePool` and resolves the gate through CSNG sibling inheritance. Removed the static controller's per-pool yaml write (the `#2791` knob) on the assumption that the gRPC `CreateNodePool` path would be the sole producer. **This assumption is wrong for AppBox clusters** where the gRPC path does not fire on bootstrap, so built-in NodePool CRs created on or after #3040 land with `Spec.PlacementGroupEnabled=false` (Go zero value) and never receive a partition PG. 5. **This PR (post-#3040 fix)** — closes the AppBox producer gap by having the static controller stamp the topology-derived value at create time. The annotation gate in the EKS PG controller remains deferred to the follow-up. **Why this PR does not flip legacy built-in `Spec=true` back to `false`**: doing so would record the durable spec as `false` while leaving the actual PG and partition NodeGroups in AWS unchanged. A future reconcile that reads from spec would skip Delete cleanup and orphan the PG; a follow-up that adds the annotation gate would also need to drive a destructive migration of every PG and 3 partition NodeGroups per env (5 prod single-AZ EKS envs known to be affected: `ap-northeast-1-ess`, `ap-east-1-ess`, `ap-south-1-ess`, `us-west-2-ess`, `us-east-1-byoc-poc`). That is the follow-up's scope, not this PR's. #### Fallback NodePools (shortage detector) The shortage detector in `domain/app/cron/shortage_detector.go` creates fallback pools via `CreateNodePool`. Fallback NodePools must carry the same PG settings as the primary pool. The detector reads the primary pool from `ListNodePools` (which returns `PlacementGroupEnabled` via `nodePoolCR2API`) and forwards the primary's spec value to the fallback `CreateNodePoolRequest`: ```go primary, err := selectPrimaryNodePool(nodePools, primarySpec) if err != nil { return fmt.Errorf("failed to select primary nodepool for fallback placement group setting: %w", err) } placementGroupEnabled := primary.GetPlacementGroupEnabled() req := &dplanemgrv1.CreateNodePoolRequest{ DataPlaneId: dataPlaneName, ResourcePrefix: resourcePrefix, NodePool: &dplanemgrv1.NodePool{ // ... existing fields ... PlacementGroupEnabled: placementGroupEnabled, }, } ``` `selectPrimaryNodePool` failure is propagated as a retryable error rather than silently defaulting to `false` (which would pin the wrong durable record on the fallback and never converge, since a later retry hits `AlreadyExists`). When the fallback uses the **same instance type** as the primary, `PlacementGroupName` resolves to the same value (instance type + dataplane hash), and the `placementGroup` task's idempotent Create path detects the existing CR and returns Done — no second PG is created. The fallback's NodeGroups reference the same placement group in AWS, so all nodes (primary and fallback) are isolated across the same set of physical racks. When the fallback uses a **different instance type** (capacity substitution), `PlacementGroupName` resolves to a different value and a new PG is created for the fallback. Nodes in the fallback PG are independently isolated across 3 racks. Because `topologySpreadConstraints` on `tidbcloud.com/pg-partition` distributes pods by label value (0/1/2), pods on fallback nodes still land in 3 distinct partitions — each PG's partitions are independent of the other, so no single rack failure takes down all nodes of any one label value. ### Operator CR Layer (cluster-service-ng) All component templates (`pd_group`, `tikv_group`, `tidb_group`, `tiproxy_group`, `tiflash_group`, `tso_group`, `scheduling_group`, `resource_manager_group`, `tidb_worker_group`, `tikv_worker_group`) use the same `{{!Topologies}}` injection point inside `schedulePolicies.evenlySpread.topologies`. Single-AZ PG mode follows the same pattern as multi-AZ: only the topology key changes from `topology.kubernetes.io/zone` to `tidbcloud.com/pg-partition`. No template files need to change. `BuildTopologies` is already called for every component type at a single call site in `apply_component_group.go` (line 387). The call site reads the durable per-pool gate from `SharedPool.Provider.PlacementGroupEnabled` rather than re-deriving from the zone list, so the same value governs both scheduling labels and partition placement. #### `BuildTopologies` — emit partition topologies `domain/workflow/utils.go` When `pgEnabled=true`, emit 3 partition topologies (hardcoded); when `pgEnabled=false`, emit zone topologies as today: ```go func BuildTopologies(compType apiv1.ComponentType, zones []string, templArgs map[string]string, pgEnabled bool) error { if pgEnabled { const partitionCount = 3 topologies := make([]string, 0, partitionCount) for i := 0; i < partitionCount; i++ { topologies = append(topologies, fmt.Sprintf("- topology:\n tidbcloud.com/pg-partition: \"%d\"", i)) } templArgs["Topologies"] = strings.Join(topologies, "\n") return nil } // existing zone-based path unchanged ... } ``` `pgEnabled=false` always takes the zone-based path. The resulting `schedulePolicies` in both PDGroup and TiKVGroup: ```yaml schedulePolicies: - evenlySpread: topologies: - topology: tidbcloud.com/pg-partition: "0" - topology: tidbcloud.com/pg-partition: "1" - topology: tidbcloud.com/pg-partition: "2" type: EvenlySpread ``` Call-site update — the change in `apply_component_group.go` covers all component types at once: ```go placementGroupEnabled := sharedPool.GetProvider().GetPlacementGroupEnabled() err = workflow.BuildTopologies( comp.GetComponentType(), sharedPool.GetProvider().GetNodeZones(), templArgs, placementGroupEnabled, ) ``` Reading from the shared-pool gate (not the zone count) keeps scheduling and infrastructure in agreement. Re-deriving per apply used to flip only the scheduling side, pinning workloads on labels that the preserved legacy nodes did not carry. #### PD `location-labels` injection `domain/workflow/action/apply_component_group.go` (inside `buildCompTempArgs`) `location-labels = ["tidbcloud.com/pg-partition"]` must be present in the PD configuration for Raft replica placement to be rack-aware. Without it, replicas of the same key range may land on TiKV stores in the same partition despite correct pod spread. This must be injected for both `ComponentType_PD` and `ComponentType_PD_SCHEDULING`. In standard PD mode, the main PD process owns replica placement. In micro-service mode (`mode: ms`), the SchedulingGroup owns it. Injecting into both is correct and safe. `ComponentType_PD_TSO` and `ComponentType_RESOURCE_MANAGER` do not handle Raft replica placement and must not receive this injection. The injection must handle the case where the config already contains a `[replication]` section without `location-labels`. Appending a second `[replication]` block is invalid TOML; insert into the existing section instead: ```go if sharedPool.GetProvider().GetPlacementGroupEnabled() && (comp.GetComponentType() == apiv1.ComponentType_PD || comp.GetComponentType() == apiv1.ComponentType_PD_SCHEDULING) { const locationLabelsLine = ` location-labels = ["tidbcloud.com/pg-partition"]` config := templArgs["Config"] if !strings.Contains(config, "location-labels") { if strings.Contains(config, "[replication]") { config = strings.Replace(config, "[replication]", "[replication]\n"+locationLabelsLine, 1) } else { config += "\n[replication]\n" + locationLabelsLine } } templArgs["Config"] = config } ``` The same gate (`sharedPool.GetProvider().GetPlacementGroupEnabled()`) drives both the `BuildTopologies` call and the PD `location-labels` injection. Both consumers see the same value, so a legacy single-AZ/PG-disabled pool keeps the legacy per-zone NodeGroups AND the legacy scheduling, and a new single-AZ pool gets partition nodes AND `pg-partition` constraints together. ### Caller Responsibility The NodeGroup task and `BuildTopologies` are workload-agnostic. `Provider.PlacementGroupEnabled` is set once per `SharedPool` at `CreateSharedPool` time (from `initConfig.NodeZones` + provider type) and applies uniformly to all components (`pd`, `tikv`, `tidb`, `tiproxy`). Strategy and partition count are hardcoded in the task layer. The caller-supplied `SharedPool.placement_group_enabled` field is parsed but ignored, so the caller does not need to set it correctly for PG to work — and a caller that does set it (incorrectly or otherwise) does not break the contract. ### IAM Prerequisites The Crossplane provider role must have the following EC2 permissions before any `PlacementGroup` CR is applied: ```json { "Effect": "Allow", "Action": [ "ec2:CreatePlacementGroup", "ec2:DescribePlacementGroups", "ec2:DeletePlacementGroup" ], "Resource": "*" } ``` Without these permissions, `PlacementGroup.Create` fails with `403 UnauthorizedOperation` and NodeGroups that reference the PG fail with `InvalidRequestException: The placement group '...' is unknown`. ## Implementation Outline 1. `NodePoolSpec.PlacementGroupEnabled` already exists; `make generate` and `make manifests` keep the CRD aligned. 2. `api/tidbcloud/dataplane-manager/v1/resources.proto` keeps `placement_group_enabled` on `NodePool`; the comment now documents the source-of-truth chain (CSNG sibling inheritance → SharedPool gate → DPM topology sanity check), not Deprecated. 3. `dataplaneservice.go` `CreateNodePool`: honour `req.NodePool.PlacementGroupEnabled` and persist it on `NodePoolSpec.PlacementGroupEnabled`. When the wire value is `true`, call `resolvePlacementGroupEnabled` as a topology sanity check — `true` on multi-AZ or non-EKS is rejected with `ParameterInvalid`. `false` is always accepted. `resolvePlacementGroupEnabled` becomes a topology-eligibility helper used only when wire value is `true`; topology unresolved returns an error so a later retry can still converge. 4. `PlacementGroupName(instanceType, namespacedHash string) string` lives in `pkg/namer/namer.go`. 5. `placementgroup.go` task: `placementGroupEnabled(np)` reads `np.Spec.PlacementGroupEnabled` only (no annotation gate in this PR — built-in protection is deferred). Create idempotent / PostCreate polls / Delete with NodePool-count guard / PostDelete. 6. `nodegroup.go`: partition loop driven by `placementGroupEnabled(t.NodePool)`. 7. `NewPlacementGroup` precedes `NewNodeGroup` in `taskbuilder.go`. 8. `pkg/controllers/dataplane/task/nodepool.go`: built-in NodePools continue to carry the `tidbcloud.com/builtin-nodepool=true` annotation. For AppBox clusters the static controller IS the producer that stamps `Spec.PlacementGroupEnabled = isSingleAzEKS(Config.Provider, DataPlane.Spec.NodeZones)` at create time. The reconcile path does not rewrite the spec on existing pools — legacy `Spec=true` pools survive reconciles, clusters deployed under the post-`#3040` / pre-fix window with `Spec=false` are migrated by deleting and recreating the cluster. The annotation is consumed only by the CSNG sibling-inheritance filter; the EKS PG controller does not gate on it (deferred to follow-up). 9. `api/cluster-service-ng/v1beta/resources.proto`: `Provider.placement_group_enabled` kept with Deprecated comment. 10. `pkg/runtime/actions/action.go`: `ActionArgBool` and `GetBool`. 11. `domain/workflow/consts.go`: `PlacementGroupEnabledKey` for DAG arg channel. 12. `domain/server/create_sharedpool.go`: derive `Provider.PlacementGroupEnabled` from `initConfig.NodeZones` + provider type after provider/HA resolution; drop the AWS-only validation that used to reject user input; the caller-supplied field is parsed but never persisted. 13. `domain/server/node_instance_group.go`: `placementGroupEnabledForDataplanes(ctx, names)` bulk helper runs once before opening the transaction (per PR review Finding 1); ACTIVE branch indexes into the result map; DEPRECATED and retire branches pass a constant `false` (the consuming plans do not read the value). `buildNodeInstanceGroupDagArgs` extended to accept `placementGroupEnabled bool`. 14. `domain/workflow/action/create_node_instance_group.go`: implement `resolvePlacementGroupEnabled(ctx, dataplaneName, args)` that - lists existing NodePools in the dataplane via `dataplaneClient.ListNodePools`, - filters built-in admin/o11y pools out via the `nodegroup.LabelNodeSpec` discriminator on `Spec.Labels` (workload pools carry the label via `BuildPrimaryLabels`; builtins carry no `Spec.Labels`), - returns `Spec.PlacementGroupEnabled` of the first workload sibling found (the durable record), - falls back to the DAG arg (= SharedPool gate) when no workload sibling exists (first NodePool in a fresh dataplane), - logs and falls back to the DAG arg on `ListNodePools` RPC failure (transient DPM errors must not block NodePool creation). Send the result on the `CreateNodePoolRequest` wire; DPM honours it. 15. `domain/workflow/utils.go`: extend `BuildTopologies` to emit `tidbcloud.com/pg-partition` topologies when PG enabled. 16. `domain/workflow/action/apply_component_group.go`: read the gate from `SharedPool.Provider.PlacementGroupEnabled` (not the zone count) and pass to `BuildTopologies`; inject `location-labels` for PD and PD_SCHEDULING components. 17. `domain/app/cron/shortage_detector.go`: inherit `PlacementGroupEnabled` from the primary NodePool's spec (`selectPrimaryNodePool`), never from the shared-pool gate or zone count. Propagate `selectPrimaryNodePool` failure as a retryable error. ## Test Plan **dataplane-manager unit tests** - `placementgroup_test.go`: `Spec.PlacementGroupEnabled=false` returns `Done` immediately; when enabled, PG CR is created with strategy=partition and partitionCount=3; PG CR name matches namer output; Create is idempotent when CR already exists. `TestPlacementGroupDeleteRunsWithNilVPC` covers the orphan-PG fix (VPC nil on Delete reconcile, spec=true → PG still deleted). `TestPlacementGroupSkippedForLegacyPGDisabledPool` covers the no-silent-migration invariant (spec=false → PG skipped regardless of dataplane). Built-in scope guard tests are deferred to the follow-up. - `nodegroup_test.go`: disabled path runs existing zone loop unchanged; enabled path creates exactly 3 NodeGroups with correct `Placement.GroupName`, `Placement.PartitionNumber` (0/1/2), and `tidbcloud.com/pg-partition` labels; delete removes all 3 partition NodeGroups and LaunchTemplates. - `dataplaneservice_test.go`: `CreateNodePool` honours the wire value and persists it on `Spec.PlacementGroupEnabled` (single-AZ + wire=true → true; wire=false → false). The topology sanity check rejects wire=true on multi-AZ (`ParameterInvalid`) and on non-EKS providers (`ParameterInvalid`). Unresolved topology surfaces as a retryable error regardless of wire value. `ListNodePools` returns the field from CR. - `dataplane/task/nodepool_test.go`: the static controller stamps `Spec.PlacementGroupEnabled` from `isSingleAzEKS(Config.Provider, DataPlane.Spec.NodeZones)`. Tests cover single-AZ EKS → true, multi-AZ EKS → false, non-EKS (ACK) → false even when single-AZ-shaped, and idempotent reconcile (resourceVersion preserved when all mutable fields match). - `pkg/runtime/actions/action_test.go`: `ActionArgBool(true).String()` round-trips correctly; `GetBool` returns error when key absent; `GetBool` returns correct value for both true and false. **cluster-service-ng unit tests** - `domain/server/create_sharedpool_test.go`: provider is nil going in; `initConfig.NodeZones` drives the expected gate. - `domain/app/app_test.go`: `TestCreateSharedPoolDerivesProviderPlacementGroupEnabledFromInitConfigNodeZones` covers the public ingress — the mock service asserts the serverImpl writes `Provider.PlacementGroupEnabled` from `initConfig.NodeZones` even when `SharedPool.Provider == nil` going in (the wire field is OUTPUT_ONLY). - `domain/workflow/utils_test.go`: `pgEnabled=false` emits zone topologies unchanged (applies to all component types); `pgEnabled=true` emits exactly 3 `pg-partition` entries (0/1/2) regardless of component type. - `domain/workflow/action/apply_component_group_test.go`: - PD with PG enabled: `templArgs["Config"]` contains `location-labels`. - PD_SCHEDULING with PG enabled: `templArgs["Config"]` contains `location-labels`. - PD_TSO with PG enabled: `templArgs["Config"]` does NOT contain `location-labels`. - RESOURCE_MANAGER with PG enabled: `templArgs["Config"]` does NOT contain `location-labels`. - PD with PG disabled: config unchanged. - Config already has `[replication]` section without `location-labels`: key inserted into existing section, no second `[replication]` block. - Config already has `location-labels`: not duplicated. - `domain/server/node_instance_group_test.go`: `TestSetNodeInstanceGroupStatusUsesPerDataplanePlacementGroupEnabled` covers the per-dataplane ACTIVE-branch resolution; DEPRECATED transitions do not round-trip the repo. - `domain/app/cron/shortage_detector_test.go`: `TestCreateFallbackNodeInstanceGroup_PropagatesMissingPrimary` covers the retryable-error path when `selectPrimaryNodePool` fails; fallback `CreateNodePoolRequest` carries `PlacementGroupEnabled` from the primary's spec. `TestCreateFallbackNodeInstanceGroup_InheritsPrimaryPlacementGroupEnabled` plus the new legacy-`false`-primary regression test pin the durable-record preservation: a legacy single-AZ, PG-disabled primary keeps its fallback on the legacy per-zone path on the wire even though the dataplane topology itself is eligible for partition PGs. - `domain/workflow/action/create_node_instance_group_test.go`: the new `TestCreateNodeInstanceGroupAction_Invoke` sub-tests cover the CSNG sibling-inheritance chain — `inherits false from legacy single-AZ sibling when DAG arg is true` (fgksgf's regression), `inherits true from sibling when DAG arg is also true`, `first nodepool on fresh dataplane uses DAG arg after builtin filter` (builtin discriminator via `nodegroup.LabelNodeSpec`), and `listnodepools failure falls back to DAG arg` (transient DPM RPC error resilience). **E2E verification** - One placement group per distinct instance type among PG-enabled components (e.g., TiKV and PD use different instance types → 2 PGs; if TiDB and TiProxy share an instance type → 1 PG for both). Name matches `{instanceType}-{namespacedHash}-pg` with no resource prefix. - 3 EKS NodeGroups per NodePool (p0/p1/p2) referencing the correct PG name and partition number. - Each node carries `tidbcloud.com/pg-partition` label. - CA ASG tags include `k8s.io/cluster-autoscaler/node-template/label/tidbcloud.com/pg-partition`. - All component groups (`schedulePolicies.evenlySpread.topologies`) use `tidbcloud.com/pg-partition` instead of `topology.kubernetes.io/zone`. - PDGroup config contains `location-labels = ["tidbcloud.com/pg-partition"]` in `[replication]` section. - `pd-ctl config show replication` shows `"location-labels": ["tidbcloud.com/pg-partition"]`. - Same-instance-type shortage pool creation: no second PG CR is created; existing PG is shared. - Default deployment (`PlacementGroupEnabled=false`) is unchanged end-to-end. ## Risks | Risk | Mitigation | |------|-----------| | Partition PG applied to multi-AZ cluster | The CSNG source-of-truth chain (sibling inheritance → SharedPool gate → topology check at `CreateSharedPool` time) never derives `Spec.PlacementGroupEnabled=true` for a multi-AZ dataplane. The DPM topology sanity check rejects wire=true on multi-AZ as a defence in depth so a stale or buggy caller cannot pin `true` onto the spec. | | Partition PG applied to single-AZ non-AWS cluster | Same check: provider must be AWS for `Spec.PlacementGroupEnabled=true`. Single-AZ Alicloud / ACK / AKS pools derive `false` from the provider check; their existing zone loop is unchanged. The DPM topology sanity check rejects wire=true on non-EKS. | | Orphan PlacementGroup on ZONAL teardown | The EKS controller's eligibility check reads `np.Spec.PlacementGroupEnabled` (not VPC), so the Delete reconcile still drives `placementGroup.Delete` even when `loadDependencies` is skipped and `t.VPC` is nil. | | Silent migration of legacy single-AZ, PG-disabled pools | `Spec.PlacementGroupEnabled` is written **once** at NodePool creation; subsequent reconciles never overwrite it. Legacy pools retain their original spec value across topology changes after creation. The CSNG sibling-inheritance chain pins a legacy false primary onto its fallback so siblings do not silently migrate onto the partition path while the primary stays on the legacy per-zone path. Re-creation is the only way to migrate. | | Producer / consumer rollout skew | The DPM gRPC `CreateNodePool` honours `req.NodePool.PlacementGroupEnabled` and persists it on `Spec.PlacementGroupEnabled` (with topology sanity check). The CSNG NIG action resolves the value through sibling inheritance before forwarding, so the wire value matches what the EKS controller reads on every reconcile. | | EKS NodeGroup quota exhausted | Partition mode creates 3 NodeGroups per NodePool instead of 1. Default quota is 30 per cluster. Request an increase before enabling for clusters with many components. | | NodePool recreation required for existing clusters | EC2 placement group membership is set at instance launch. Existing nodes cannot be moved in. Rolling NodePool recreation is required when enabling on an existing cluster. | | CA scales unevenly under simultaneous pod pressure | Add `--balancing-ignore-labels=tidbcloud.com/pg-partition` to the cluster-autoscaler. Without it, CA treats the 3 partition NodeGroups as different and skips group-balancing, causing uneven scale-out (p0=3, p1=1, p2=2 observed in testing). | | `location-labels` missing in micro-service PD mode | Injecting into `ComponentType_PD_SCHEDULING` covers the micro-service scheduling owner. Both PD and PD_SCHEDULING receive the injection so either mode is covered. | | Duplicate `[replication]` TOML block | The injection checks for an existing `[replication]` section and inserts into it rather than appending a second block. | | Same-instance-type shortage pools on wrong racks | PG name derives from instance type + dataplane hash (no resource prefix). Same-instance-type shortage pools produce the same PG name; `placementGroup.Create` is idempotent and reuses the existing PG — all nodes share the same rack isolation. Different-instance-type shortage pools create an independent PG; each PG isolates its own nodes across 3 racks, so `topologySpreadConstraints` still prevents quorum loss. | | Crossplane PG reconcile delay | `PostCreate` requeues with `deploy.DefaultRequeueAfter` until Ready — same pattern as the existing NodeGroup `PostCreate`. |