# AKS API Server VNet Integration Migration

This runbook is for migrating existing non-BYOC Azure AKS AppBoxes to API server VNet integration. New AKS clusters are created with VNet integration by default after the controller change is deployed.

## Background

Existing clusters are opt-in because enabling API server VNet integration on an existing AKS cluster is one-way and requires an immediate cluster restart. The controller does not stop or start AKS clusters.

Non-BYOC AKS clusters use a controller-managed user-assigned control plane identity when API server VNet integration is requested. The controller creates the identity and grants it `Network Contributor` on every node subnet and the API server subnet before the AKS spec is patched. Existing non-BYOC clusters switch from their system-assigned control plane identity to the controller-managed user-assigned identity during opt-in migration.

References:

- Microsoft AKS API Server VNet Integration: https://learn.microsoft.com/en-us/azure/aks/api-server-vnet-integration
- Microsoft AKS stop/start: https://learn.microsoft.com/en-us/azure/aks/start-stop-cluster

## Preconditions

- The dataplane-manager build containing API server subnet precreation is deployed.
- The target AppBox is healthy and its AKS kubeconfig works.
- The target VPC CIDR has the AKS CIDR dispatcher's delegated `/28` API server subnet range unused.
- The controller-managed AKS control plane identity exists and its subnet role assignments are ready.
- A maintenance window is approved for AKS stop/start and workload disruption.
- Azure CLI or an equivalent approved Azure operation path is available for the target subscription.

## Procedure

1. Confirm AppBox and AKS readiness.

   ```bash
   kubectl get appbox <appbox-name> -n <namespace> -o yaml
   kubectl get vpc <vpc-name> -n <namespace> -o yaml
   kubectl --kubeconfig <target-kubeconfig> get nodes
   ```

2. Ensure the API server subnet is ready.

   Existing active VPCs may need a metadata change to drive VPC reconciliation after the new controller build is deployed. If the API server subnet is missing from VPC status, annotate the VPC and wait for the delegated subnet to become ready.

   ```bash
   kubectl get vpc <vpc-name> -n <namespace> \
     -o jsonpath='{.status.controlPlaneSubnets[?(@.name=="<vnet-name>-apiserver")].subnetId}'

   kubectl annotate vpc <vpc-name> -n <namespace> \
     cloud.pingcap.com/aks-apiserver-vnet-integration-reconcile-at="$(date -u +%Y%m%dT%H%M%SZ)" --overwrite

   kubectl get subnet <vnet-name>-apiserver -o yaml
   kubectl get vpc <vpc-name> -n <namespace> \
     -o jsonpath='{.status.controlPlaneSubnets[?(@.name=="<vnet-name>-apiserver")].subnetId}'
   ```

   Stop the migration if the delegated `/28` subnet cannot be created or if the selected range overlaps an existing subnet.

3. Opt the existing AppBox into the AKS spec patch.

   ```bash
   kubectl annotate appbox <appbox-name> -n <namespace> \
     cloud.pingcap.com/enable-aks-apiserver-vnet-integration=true --overwrite
   ```

4. Wait for Crossplane and Azure to report the user-assigned control plane identity and API server VNet integration.

   ```bash
   kubectl get kubernetescluster <aks-crossplane-name> \
     -o jsonpath='{.metadata.annotations.cloud\.pingcap\.com/aks-apiserver-vnet-integration-requested}'

   az aks show --resource-group <resource-group> --name <aks-name> \
     --query '{identity:identity, apiServerAccessProfile:apiServerAccessProfile}'
   ```

5. Immediately restart AKS in the approved window.

   ```bash
   az aks stop --resource-group <resource-group> --name <aks-name>
   az aks show --resource-group <resource-group> --name <aks-name> \
     --query '{powerState:powerState.code, provisioningState:provisioningState}'

   az aks start --resource-group <resource-group> --name <aks-name>
   az aks show --resource-group <resource-group> --name <aks-name> \
     --query '{powerState:powerState.code, provisioningState:provisioningState}'
   ```

6. Validate the cluster.

   ```bash
   kubectl --kubeconfig <target-kubeconfig> get nodes
   kubectl get appbox <appbox-name> -n <namespace> -o yaml
   ```

## Success Criteria

- Azure reports `powerState=Running` and `provisioningState=Succeeded`.
- The AKS API server remains reachable through the existing public endpoint policy.
- `kubectl get nodes` succeeds.
- AppBox returns to ready/active status.
- The annotation remains on the AppBox as durable desired state.

## Abort Boundary

API server VNet integration cannot be disabled after it is enabled. Before the controller writes the AKS request marker, removing the AppBox opt-in annotation aborts the request. After the `KubernetesCluster` has `cloud.pingcap.com/aks-apiserver-vnet-integration-requested=true`, do not attempt rollback by removing desired state; the Azure enable operation may already be accepted. Continue with restart and recovery validation, or escalate to Azure support if the operation is stuck.