# Dedicated AKS API Server VNet Integration Migration Runbook

This document describes how to migrate **existing** Azure AKS AppBoxes in the **Dedicated** product to API Server VNet Integration. After deploying a `regional-server` version that includes this change, **new** AKS clusters enable VNet Integration by default; existing clusters require **opt-in**.

## Background

| Item | Description |
|---|---|
| Why existing clusters opt-in | Enabling API Server VNet Integration on an existing AKS cluster is a **one-way operation**, and Azure requires an **immediate restart** after the change; the controller **does not** perform `az aks stop/start` |
| New clusters | VNet Integration is enabled by default when there is no Crossplane `KubernetesCluster`, or when the CR exists but Azure has not assigned `status.atProvider.id` yet |
| Control plane identity | When VNet Integration is enabled, the AKS control plane switches from **SystemAssigned** to a controller-managed **UserAssigned** identity (`{aks-name}-cp-mi`) |
| RBAC responsibilities | `AKSManagedIdentity` creates `{aks-name}-cp-mi-vnet-ra` (VNet-level `Network Contributor` for the control-plane identity); under a UserAssigned control plane, `NodeRoleAssignment` creates `{aks-name}-cp-mi-des-ra` (DES Operator on the node resource group for cp-mi) and no longer creates or refreshes legacy `{aks-name}-vnet-ra` / `{aks-name}-des-ra` |

### Controller task order (AppBox reconcile)

```
AKSManagedIdentity → AKS → NodeRoleAssignment → ...
```

After migration opt-in, the controller runs `AKSManagedIdentity` before `AKS`. If the identity or its VNet role assignment is not Ready, reconciliation stops and requeues before the AKS spec is changed.

### Related Annotations

| Key | Attached to | Meaning |
|-----|-------------|---------|
| `cloud.pingcap.com/enable-aks-apiserver-vnet-integration=true` | APPBox | **Opt-in** switch for existing clusters |
| `cloud.pingcap.com/aks-apiserver-vnet-integration-requested=true` | Crossplane `KubernetesCluster` | Controller has written the desired VNet Integration state |

### CIDR and Subnet

The AKS CIDR dispatcher allocates pod, node, service, and PLS NAT ranges first, then carves the first **`/28`** from the remaining reserved range for the API Server delegated subnet. For the common `/19` VPC CIDR layout, that remaining range is a `/23`. Crossplane Subnet name:

```text
{vnet-name}-apiserver
```

Delegation: `Microsoft.ContainerService/managedClusters`.

VPC Status field: `status.controlPlaneSubnets`.

### Reference Documentation

- [Microsoft AKS API Server VNet Integration](https://learn.microsoft.com/en-us/azure/aks/api-server-vnet-integration)
- [Convert an existing AKS cluster](https://learn.microsoft.com/en-us/azure/aks/api-server-vnet-integration#convert-an-existing-aks-cluster-to-api-server-vnet-integration)
- [AKS stop/start](https://learn.microsoft.com/en-us/azure/aks/start-stop-cluster)

---

## Prerequisites

- The target environment has deployed a **regional-server** version that includes API Server subnet pre-creation logic.
- The target APPBox is healthy; `kubectl get nodes` (using the existing kubeconfig) succeeds.
- The **`/28` API Server segment** allocated by the AKS dispatcher in the target VPC CIDR is **not in use** and does not overlap with existing Azure subnets.
- A maintenance window has been approved: AKS stop/start and brief workload interruption.
- Azure CLI or equivalent permissions for the target subscription are available.
- The regional-server ClusterRole includes `managedidentity.azure.upbound.io` permissions. `make manifests` updates generated RBAC in this repository; the deployed app-delivery RBAC must also include the same API group before rollout.

---

## Migration Steps

### 1. Confirm APPBox and AKS readiness

```bash
kubectl get appbox <appbox-name> -n <namespace> -o yaml
kubectl get vpc <vpc-name> -n <namespace> -o yaml
kubectl --kubeconfig <target-kubeconfig> get nodes
```

Record:

- The VPC name associated with the APPBox
- `status.kubernetesClusterName` (Crossplane `KubernetesCluster` name)
- Azure resource group and AKS name (from Crossplane status or Azure portal)

### 2. Ensure the API Server subnet is created

After the new controller is deployed, existing **Active VPCs** need one reconcile trigger to create the `{vnet-name}-apiserver` subnet and populate `status.controlPlaneSubnets`. The VPC controller ignores status-only updates, but it does watch **spec generation**, **label**, and **annotation** changes.

Check whether the subnet appears in status:

```bash
VNET_NAME="<vnet-name>"   # Usually VPC status.vpc.name
VPC_NAME="<vpc-name>"
NS="<namespace>"

kubectl get vpc "${VPC_NAME}" -n "${NS}" \
  -o jsonpath="{.status.controlPlaneSubnets[?(@.name=='${VNET_NAME}-apiserver')].subnetId}"
echo
```

If empty, trigger VPC reconcile via an **annotation change**. This annotation is only used to create an object update event; it is not a VPC feature switch.

```bash
kubectl annotate vpc "${VPC_NAME}" -n "${NS}" \
  cloud.pingcap.com/trigger-time="$(date -u +%Y%m%dT%H%M%SZ)" \
  --overwrite
```

Wait for the delegated subnet to become Ready:

```bash
kubectl get subnet "${VNET_NAME}-apiserver" -o yaml
kubectl get vpc "${VPC_NAME}" -n "${NS}" \
  -o jsonpath="{.status.controlPlaneSubnets[?(@.name=='${VNET_NAME}-apiserver')].subnetId}"
echo
```

**Stop condition**: The `/28` subnet cannot be created, or conflicts with existing address space - resolve CIDR/subnet issues before proceeding.

### 3. Opt-in: annotate existing APPBox

```bash
kubectl annotate appbox <appbox-name> -n <namespace> \
  cloud.pingcap.com/enable-aks-apiserver-vnet-integration=true --overwrite
```

The controller will then:

1. `AKSManagedIdentity`: create `{aks-name}-cp-mi` and `{aks-name}-cp-mi-vnet-ra`
2. `AKS`: apply the Crossplane CR with UserAssigned control-plane identity, `APIServerAccessProfile`, and the requested marker
3. `NodeRoleAssignment`: create `{aks-name}-cp-mi-des-ra` (principal from cp-mi); skip creating or refreshing legacy `{aks-name}-vnet-ra` and `{aks-name}-des-ra` because the control plane is now UserAssigned

If legacy `{aks-name}-vnet-ra` or `{aks-name}-des-ra` Crossplane `RoleAssignment` objects already exist from the SystemAssigned era, they are not removed by the normal create/update path. They bind permissions to the old SystemAssigned MI and do not block migration; optional cleanup is described in the troubleshooting section below.

### 4. Wait for Crossplane / Azure sync

**KubernetesCluster requested marker:**

```bash
kubectl get kubernetescluster <aks-crossplane-name> \
  -o jsonpath='{.metadata.annotations.cloud\.pingcap\.com/aks-apiserver-vnet-integration-requested}'
echo
# Expected output: true
```

**UserAssigned Identity:**

```bash
kubectl get userassignedidentity <aks-name>-cp-mi -o yaml
kubectl get roleassignment <aks-name>-cp-mi-vnet-ra -o yaml
kubectl get roleassignment <aks-name>-cp-mi-des-ra -o yaml
```

**AKS status on Azure side:**

```bash
az aks show --resource-group <resource-group> --name <aks-name> \
  --query '{identity:identity, apiServerAccessProfile:apiServerAccessProfile, provisioningState:provisioningState}'
```

Expected:

- `identity.type` is `UserAssigned`
- `apiServerAccessProfile.enableVnetIntegration` is `true`
- `apiServerAccessProfile.subnetId` points to the `{vnet-name}-apiserver` subnet

> **Crossplane does not automatically restart AKS.** Azure documentation requires stop/start as soon as possible after enabling VNet Integration; otherwise the API Server may still use the old network path, causing intermittent connectivity issues.

### 5. Restart AKS immediately within the maintenance window

```bash
az aks stop --resource-group <resource-group> --name <aks-name>

az aks show --resource-group <resource-group> --name <aks-name> \
  --query '{powerState:powerState.code, provisioningState:provisioningState}'
# Expected powerState: Stopped, provisioningState: Succeeded

az aks start --resource-group <resource-group> --name <aks-name>

az aks show --resource-group <resource-group> --name <aks-name> \
  --query '{powerState:powerState.code, provisioningState:provisioningState}'
# Expected powerState: Running, provisioningState: Succeeded
```

**How to check status after stop:**

| Field | Normal value |
|------|--------------|
| `powerState.code` | `Stopped` → `Running` after restart |
| `provisioningState` | `Succeeded` (investigate if stuck in `Stopping`/`Starting` for a long time) |

Dataplane workloads are unavailable while stopped; complete start and verification within the maintenance window.

### 6. Verification

```bash
kubectl --kubeconfig <target-kubeconfig> get nodes
kubectl get appbox <appbox-name> -n <namespace> -o yaml

az aks show --resource-group <resource-group> --name <aks-name> \
  --query 'apiServerAccessProfile.enableVnetIntegration'
```

Confirm the API Server still meets existing public access policies. If `ApiserverPublicAccessCidrs` is configured with a restrictive list, the controller appends the active Internet/NAT egress IPs to `authorizedIPRanges`. If the list is empty, no `authorizedIPRanges` are written; if it contains `0.0.0.0/0`, NAT IPs are not appended.

---

## Success Criteria

- Azure: `powerState=Running`, `provisioningState=Succeeded`
- `apiServerAccessProfile.enableVnetIntegration=true`
- `kubectl get nodes` succeeds
- APPBox returns to Ready/Active
- Opt-in annotation on APPBox is retained (as a record of desired state)

---

## Stop and Rollback Boundaries

| Stage | Rollback possible? | Action |
|------|--------------------|--------|
| APPBox opt-in applied, but `KubernetesCluster` has **not yet** written `aks-apiserver-vnet-integration-requested` | Yes | Remove the opt-in annotation on APPBox and wait for reconcile |
| `KubernetesCluster` **already has** `aks-apiserver-vnet-integration-requested=true` | Treat as not reversible | The controller treats the request marker as sticky desired state. VNet Integration is also one-way once Azure accepts it; proceed with stop/start and verification, or contact Azure support |

Do not attempt to "undo" by deleting Crossplane desired state after the request marker is written - Azure may have already accepted the change, and the controller will continue to request VNet Integration while the marker remains.

---

## Troubleshooting

### APPBox reconcile stuck on AKSManagedIdentity

- Check whether `UserAssignedIdentity` / `RoleAssignment` CRs are Ready
- Check whether regional-server RBAC includes `managedidentity.azure.upbound.io`
- Check whether VPC `status.vpc.name` is populated (VNet scope RBAC depends on this field)

### AKS patch fails

- Confirm the API Server subnet is delegated and Ready
- Confirm `{aks-name}-cp-mi`, `{aks-name}-cp-mi-vnet-ra`, and `{aks-name}-cp-mi-des-ra` are Ready
- Review Crossplane `KubernetesCluster` events / Azure activity log

### Still abnormal after restart

- Reconfirm `enableVnetIntegration` is true on the Azure side
- Check whether authorized IP ranges include the current NAT Gateway egress IP
- Check whether AKS `provisioningState` remains non-`Succeeded` for an extended period

### Legacy `{aks-name}-vnet-ra` remains

After migration to UserAssigned, `NodeRoleAssignment` no longer creates or refreshes the legacy `{aks-name}-vnet-ra` (bound to the old SystemAssigned principal). Existing Crossplane `RoleAssignment` objects are not automatically pruned on the create/update path, and the delete path also skips this name once the current AKS spec is UserAssigned.

If cleanup is required, first confirm AKS has switched to UserAssigned identity and the new `{aks-name}-cp-mi-vnet-ra` is Ready. Then delete the Crossplane `RoleAssignment` object:

```bash
kubectl delete roleassignment <aks-name>-vnet-ra
```

Only delete the Azure RBAC assignment directly if the Crossplane object is already gone and Azure still shows an orphaned assignment.

### Legacy `{aks-name}-des-ra` remains

After migration to UserAssigned, `NodeRoleAssignment` no longer creates or refreshes the legacy `{aks-name}-des-ra` (bound to the old SystemAssigned principal). CMEK DES access is provided by `{aks-name}-cp-mi-des-ra` instead. The legacy object is not automatically pruned; it grants DES Operator to the unused SystemAssigned MI and does not block the cluster.

If cleanup is required, first confirm `{aks-name}-cp-mi-des-ra` is Ready. Then delete the Crossplane `RoleAssignment` object:

```bash
kubectl delete roleassignment <aks-name>-des-ra
```

---

## Differences from New Cluster Behavior

| Scenario | VNet Integration | Identity |
|------|-------------------|----------|
| New AKS (no existing Crossplane CR / not yet provisioned) | Enabled by default | UserAssigned (`{aks-name}-cp-mi`) |
| Existing provisioned AKS, no opt-in annotation | **Not enabled** | Remains SystemAssigned |
| Existing provisioned AKS, with opt-in annotation | Enabled | Switches to UserAssigned + **manual restart** |
| Existing AKS with requested marker or observed VNet Integration enabled | Kept enabled even if APPBox opt-in annotation is later removed | Remains UserAssigned |