# Manual TiBR node placement rollout

## Purpose

After the TiBR node placement change is deployed, newly created TiBR workloads select a fitting node pool by both `node-spec` and `tidbcloud.com/node-pool-k8s-version`.

Existing resource pools are not migrated automatically. Roll them out one resource pool at a time by entering the regional control plane, port-forwarding the Cluster Service Kubernetes Service, and calling the existing component config API directly:

```text
POST /sharedPools/{shared_pool_id}:setConfigVersion
```

The rollout affects existing resources as follows:

- TiBR auto backup is validated, deleted, and recreated with the selectors.
- TiBRGC is updated in place so future GC Jobs use the selectors.
- Future manual backup, manual restore, and lightweight restore Jobs use the selectors without a migration.
- Already-running one-shot backup, restore, and GC Jobs are not migrated.
- Classic restore is excluded from this placement change.

## Prerequisites

Before submitting a rollout:

1. Confirm the Cluster Service version containing the placement change is deployed in the region.
2. Confirm the target resource pool is `AVAILABLE` and not paused.
3. Confirm no TiBR auto-backup or TiBRGC upgrade workflow is already running for the resource pool.
4. Start with one canary resource pool. Do not submit a regional bulk rollout.
5. Enter the regional control-plane jump and confirm its Kubernetes context points to the intended regional control-plane cluster.

Cluster Service is deployed in the regional control plane with this layout:

```text
Namespace:    cluster-ng-service
Service:      cluster-ng-server
Service port: 80
Pod port:     8080
API prefix:   /api/v1beta
```

Verify the Service and its selected pods before continuing:

```bash
kubectl config current-context
kubectl -n cluster-ng-service get service cluster-ng-server
kubectl -n cluster-ng-service get pods -l app=cluster-ng-server -o wide
```

In one terminal on the regional control-plane jump, keep the port-forward running:

```bash
kubectl -n cluster-ng-service port-forward service/cluster-ng-server 18080:80
```

In a second terminal on the same jump, configure the local endpoint and target resource pool:

```bash
CSNG_LOCAL_URL='http://127.0.0.1:18080/api/v1beta'
RESOURCE_POOL_ID='634362268536038285'
```

The following requests must be sent from that second jump terminal while the port-forward is active. They call the Cluster Service pod through its Kubernetes Service and do not use the externally routed Cluster Service endpoint.

## Roll out TiBR auto backup

Submit the existing component config update API for `TIBR_AUTOBACKUP`:

```bash
curl --fail-with-body --request POST \
  --noproxy '*' \
  "${CSNG_LOCAL_URL}/sharedPools/${RESOURCE_POOL_ID}:setConfigVersion" \
  --header 'Content-Type: application/json' \
  --data "{
    \"sharedPoolId\": \"${RESOURCE_POOL_ID}\",
    \"components\": [
      {
        \"componentType\": \"TIBR_AUTOBACKUP\"
      }
    ],
    \"scope\": \"CONFIG\"
  }"
```

A successful response means the upgrade workflow was submitted; it does not mean the rollout has finished. The workflow performs these steps in order:

1. Resolve and validate a fitting node pool.
2. Delete the current TiBR custom resource.
3. Recreate TiBR with both placement selectors.
4. Wait for TiBR to become ready and synced.

Placement validation happens before deletion. If no fitting node pool can be resolved, the workflow fails without deleting the current TiBR custom resource.

### Verify auto backup

Wait for the `upgrade-tibr-auto-backup` workflow to finish, then verify:

- The workflow completed without a failed action.
- The TiBR pod is ready.
- Its pod template contains the expected `node-spec` and `tidbcloud.com/node-pool-k8s-version` selectors.
- The pod runs on a node matching both selectors.
- A scheduled auto backup can run successfully.

The physical-node vCPU metric may continue to include the old node until Cluster Autoscaler removes it. Treat workflow and pod readiness as the rollout completion signal; verify the metric separately after scale-down.

Do not blindly retry the API call. Every accepted auto-backup update intentionally deletes and recreates TiBR, even when its existing placement already matches. Inspect the previous workflow result before resubmitting.

## Roll out TiBRGC

After the auto-backup canary is healthy, update the existing TiBRGC custom resource:

```bash
curl --fail-with-body --request POST \
  --noproxy '*' \
  "${CSNG_LOCAL_URL}/sharedPools/${RESOURCE_POOL_ID}:setConfigVersion" \
  --header 'Content-Type: application/json' \
  --data "{
    \"sharedPoolId\": \"${RESOURCE_POOL_ID}\",
    \"components\": [
      {
        \"componentType\": \"TIBRGC_T2_DFSGC\"
      }
    ],
    \"scope\": \"CONFIG\"
  }"
```

Either `TIBRGC_T2_DFSGC` or `TIBRGC_T3_ARCHIVE` triggers the same full TiBRGC apply workflow; one component type is sufficient. If the resource pool has no applicable backup GC setting, the workflow skips the apply.

### Verify TiBRGC

Wait for the `upgrade-tibr-gc` workflow to finish, then verify:

- The TiBRGC custom resource is running.
- Its T2 and T3 Job templates contain both placement selectors when those strategies are configured.
- Newly created TiBRGC Jobs run on matching nodes.

Existing TiBRGC Jobs are not moved or recreated by this operation.

## Continue or stop

Proceed to the next resource pool only after both workflows and their verification checks succeed for the canary. Stop the rollout on any placement resolution, workflow, readiness, or backup/GC functional failure. Automatic discovery, throttling, progress tracking, and regional migration are intentionally deferred to a separate implementation.