# Azure BYOC Controlplane to Dataplane Design for Dataplane-Manager

## Context

`dataplane-manager` runs in our Azure tenant and needs to manage dataplane AKS clusters in customer tenants for BYOC. This design covers three areas:
1. **Network connectivity**: Private AKS with VNet integration + dual private link (API server + Kong)
2. **Cloud resource & AKS API auth**: Multi-tenant Azure AD app with workload identity federation
3. **Agent auth**: Kong auth-server plugin reusing K8s RBAC (same as AWS BYOC)

## Overall Architecture

```
Our Tenant (Control Plane)                          Customer Tenant (Dataplane)
┌────────────────────────────────┐                  ┌──────────────────────────────────────────────┐
│                                │                  │                                              │
│  Control Plane AKS             │                  │  Dataplane AKS (private + VNet integration)  │
│  ┌──────────────────────────┐  │                  │  ┌──────────────────────────────────────────┐│
│  │ dataplane-manager pod    │  │                  │  │                                          ││
│  │  SA: dpm-sa              │  │                  │  │  API server (in apiserver subnet /28)    ││
│  │  Projected JWT ──┐       │  │                  │  │    ← kube-apiserver LB (auto-created)   ││
│  │                  │       │  │                  │  │    ← apiserver-pls → PE in our VNet     ││
│  │                  ▼       │  │                  │  │                                          ││
│  │  kubelogin exchanges     │  │                  │  │  Kong (clustergateway)                   ││
│  │  JWT for Azure AD token  │  │                  │  │    /cluster-agent → cluster-agent svc   ││
│  │  (cross-tenant via       │  │                  │  │    /proxy        → cluster-proxy svc    ││
│  │   multi-tenant app)      │  │                  │  │    /dataflow-agent → dataflow-agent svc ││
│  │                  │       │  │                  │  │    auth-server plugin: K8s RBAC          ││
│  └──────────────────┼───────┘  │                  │  └──────────┬──────────────┬───────────────┘│
│                     │          │                  │             │              │                 │
│  Control Plane VNet │          │                  │  Customer VNet             │                 │
│  ┌──────────────────┼───────┐  │                  │  ┌─────────┼──────────────┼───────────────┐│
│  │                  │       │  │                  │  │  pls NAT subnet                        ││
│  │  PE (API server) │       │  │  Private Link 1  │  │    apiserver-pls                       ││
│  │  ───────────────────────────┼──────────────────┼──│      backed by kube-apiserver LB       ││
│  │  PE (Kong)       │       │  │  Private Link 2  │  │    kong-pls                            ││
│  │  ───────────────────────────┼──────────────────┼──│      backed by Kong internal LB        ││
│  │                  │       │  │                  │  │                                          ││
│  │  Shared Private DNS Zones│  │                  │  └──────────────────────────────────────────┘│
│  │  → per-dataplane A records│ │                  │                                              │
│  └──────────────────────────┘  │                  │  Azure AD validates tokens via              │
│                                │                  │  managed AAD RBAC integration               │
│  Crossplane provider-azure    │                  │                                              │
│  ProviderConfig:              │                  │  ARM API (resource management)              │
│    clientID: <multi-tenant>   │  Token exchange   │  ┌──────────────────────────────────────────┐│
│    tenantID: <customer>       │──────────────────┼──│ VNet, Subnets, AKS, Storage, PLS, etc.  ││
│    source: OIDCTokenFile      │                  │  │ RBAC: SP has Contributor role             ││
│                                │                  │  └──────────────────────────────────────────┘│
└────────────────────────────────┘                  └──────────────────────────────────────────────┘
```

**Two private links**: API server access (kubectl) is direct via its own PLS/PE, independent of Kong. Agent traffic goes through Kong's PLS/PE. This means kubectl always works even during Kong failures.

---

## Part 1: Cross-Tenant Authentication (Multi-Tenant App)

### Problem

`DefaultAzureCredential` and workload identity work within one tenant. For BYOC, our pods need to authenticate to customer's Azure AD tenant for both ARM API (resource management) and AKS API server access.

### Design

Use a **multi-tenant Azure AD app registration** with **federated identity credentials** — no secrets needed.

#### One-Time Setup (Our Side)

```
App Registration: "TiDB Cloud Dataplane Manager"
├── Type: Multi-tenant
├── Application (client) ID: <our-app-client-id>
└── Federated Identity Credentials:
     ├── Issuer: <our-AKS-OIDC-issuer-URL>
     ├── Subject: system:serviceaccount:dpm-ns:dpm-sa
     └── Audience: api://AzureADTokenExchange
```

Also add federated credential for Crossplane provider-azure service account.

#### Per-Customer Onboarding

1. Customer consents to our multi-tenant app → Service Principal created in their tenant
2. Customer assigns RBAC roles to the SP on their subscription:
   - `Contributor` (or custom role) for resource management
   - `Network Contributor` for VNet/PLS/PE
   - `User Access Administrator` for role assignments
3. Customer configures AKS authorization for our SP (one of):
   - Create or choose an Azure AD group, add the SP as a member, and provide that group's object ID to us for `AdminGroupObjectIDs`
   - Or assign "Azure Kubernetes Service RBAC Cluster Admin" role on the AKS resource

#### Runtime Token Exchange (Secretless)

```
Pod projected SA token (K8s JWT)
  → POST https://login.microsoftonline.com/{customer-tenant}/oauth2/v2.0/token
    client_id = <our-multi-tenant-app>
    client_assertion = <K8s JWT>
    client_assertion_type = urn:ietf:params:oauth:client-assertion-type:jwt-bearer
    scope = https://management.azure.com/.default  (for ARM)
          or <aks-server-id>/.default               (for AKS API)
  → Azure AD validates federated credential on our app registration
  → Returns access token scoped to customer's tenant
```

### Code Changes

#### `pkg/providerconfig/initializer.go` — Crossplane ProviderConfig for BYOC

```go
// For BYOC: use multi-tenant app's clientID + customer's tenantID
// Credential source stays OIDCTokenFile (same projected SA token)
func setAzureProviderConfigSpec(pc *azurev1beta1.ProviderConfig, subscriptionID, tenantID, clientID string) {
    pc.Spec.Credentials = azurev1beta1.ProviderCredentials{
        Source: oidcTokenFile,  // unchanged
    }
    pc.Spec.SubscriptionID = ptr.To(subscriptionID)
    pc.Spec.TenantID = ptr.To(tenantID)   // customer's tenant for BYOC
    pc.Spec.ClientID = ptr.To(clientID)    // multi-tenant app for BYOC
}
```

The `AzureAccountEntry` already has `TenantID` and `ClientID` fields — just populate with customer values.

#### `pkg/cloud/azure/utils.go` — Cross-tenant kubelogin conversion

```go
// New function: ConvertToWorkloadIdentityLoginCrossTenant
// For BYOC, kubelogin needs explicit --client-id and --tenant-id
// (same-tenant infers from env; cross-tenant cannot)
func ConvertToWorkloadIdentityLoginCrossTenant(original []byte, clientID, tenantID string) ([]byte, error)
```

Current `ConvertToWorkloadIdentityLogin()` strips `--client-id` and `--tenant-id`. The new function preserves/injects them:
```
kubelogin get-token --login workloadidentity --server-id <aks-server-id> --client-id <our-app> --tenant-id <customer-tenant>
```

#### `pkg/controllers/appbox/task/aks/aks.go` — AKS creation for BYOC

- For BYOC, `AdminGroupObjectIDs` should contain customer-provided Azure AD group object IDs whose members include our multi-tenant app's service principal
- Alternatively, customer can rely on Azure RBAC role assignment on the AKS resource instead of AAD admin groups
- `createKubeconfigSecret`: use `ConvertToWorkloadIdentityLoginCrossTenant` for BYOC

#### `pkg/config/config.go` — BYOC config

Add to `BYOCConfig`:
```go
type BYOCConfig struct {
    // existing fields...
    MultiTenantAppClientID string `yaml:"multiTenantAppClientID"`
}
```

### Files to Modify

| File | Change |
|------|--------|
| `pkg/providerconfig/initializer.go` | Use customer tenantID + app clientID for BYOC ProviderConfig |
| `pkg/cloud/azure/utils.go` | Add `ConvertToWorkloadIdentityLoginCrossTenant()` |
| `pkg/controllers/appbox/task/aks/aks.go` | BYOC kubeconfig with cross-tenant args |
| `pkg/config/config.go` | Add `MultiTenantAppClientID` to BYOC config |

---

## Part 2: Private Network Connectivity (Dual Private Link)

### Problem

All control-plane → dataplane traffic must go through private link, not public internet. This includes AKS API server access (kubectl) and agent communication.

### Design

Use a **private AKS cluster with API Server VNet Integration** (BYO VNet mode) and **two separate private links**:

1. **API server private link**: PLS backed by the `kube-apiserver` LB (auto-created by AKS) → PE in our VNet → direct kubectl access
2. **Kong private link**: PLS backed by Kong's internal LB → PE in our VNet → agent traffic (cluster-agent, cluster-proxy, dataflow-agent)

```
kubectl traffic:
  dataplane-manager → PE (API server) → apiserver-pls → kube-apiserver LB → API server

Agent traffic:
  dataplane-manager → PE (Kong) → kong-pls → Kong LB → {agents}
```

AKS is created **private from the start** with VNet integration — no public endpoint, no public→private transition. The `kube-apiserver` LB is automatically created by AKS when VNet integration is enabled; we just create a PLS backed by it.

Private DNS zone ownership is **shared regional infrastructure**, not per-dataplane infrastructure:
- Gateway DNS zone is pre-created and configured via `dnsProviderConfig.azureDNSZone`
- API server DNS zone is pre-created and configured via `dnsProviderConfig.azureAPIServerDNSZone`
- `dataplane-manager` creates only per-dataplane `PrivateDNSARecord` resources in those zones
- `dataplane-manager` does not create or own `PrivateDNSZone` or `PrivateDNSZoneVirtualNetworkLink`

### AKS Configuration

```go
// Private AKS with VNet integration (BYO mode)
if c.IsBYOC() {
    aks.Spec.ForProvider.PrivateClusterEnabled = ptr.To(true)
    aks.Spec.ForProvider.PrivateDNSZone = ptr.To("None")
    aks.Spec.ForProvider.PrivateClusterPublicFqdnEnabled = ptr.To(false)
    aks.Spec.ForProvider.APIServerAccessProfile = []v1beta1.APIServerAccessProfileParameters{
        {
            VnetIntegrationEnabled: ptr.To(true),
            SubnetID:               ptr.To(apiServerSubnetID),  // /28 dedicated subnet
        },
    }
}
```

The API server subnet (`/28`, delegated to `Microsoft.ContainerService/managedClusters`) is created alongside the node and PLS NAT subnets during VPC setup.

### Bootstrap Sequence

1. AKS created as private cluster with VNet integration
2. `kube-apiserver` LB auto-created by AKS in MC_ resource group
3. `az aks command invoke` used to create apiserver-pls (PLS backed by kube-apiserver LB)
4. apiserver-pe created in our VNet → PE approved
5. apiserver DNS A record created in the shared API server private DNS zone
6. Verify kubectl works via `PE → apiserver-pls → kube-apiserver LB → API server`
7. All subsequent kubectl access via API server PE (direct, no Kong dependency)
8. Deploy remaining components (Kong, agents, etc.) via kubectl over private link
9. Kong LB ready → kong-pls created → kong-pe created → gateway DNS A record created
10. Agent traffic flows via Kong PE

### Deletion Sequence (Reverse)

1. Delete Kong DNS A record, Kong PE, kong-pls
2. Delete components via kubectl (still works via API server PE)
3. Delete API server DNS A record, apiserver-pe, apiserver-pls
4. Delete AKS cluster

No need to re-enable public access during deletion — API server PE remains available until the last step.

### New Task Files

#### `task/aks/apiserver_pls.go` — Create PLS for API server

Creates a PLS backed by the **existing `kube-apiserver` LB** that AKS auto-creates with VNet integration. This is the same PLS creation pattern as Kong PLS, just referencing a different LB.

- Discover `kube-apiserver` LB in MC_ resource group: `az network lb show --name "kube-apiserver" --resource-group "MC_<rg>_<aks>_<location>"`
- Get LB frontend IP config ID
- Create `azurenetworkv1beta1.PrivateLinkService`:
  - `LoadBalancerFrontendIPConfigurationIds` → kube-apiserver LB frontend
  - `NATIPConfiguration` → from `VPC.Status.PLSNATSubnets`
  - `ProviderConfigReference` → customer's subscription
- Wait for PLS ready, store alias in APPBox status
- On delete: remove PLS

During initial bootstrap (before PE is ready), this task uses `az aks command invoke` to run within the cluster context if needed. However, the PLS creation itself is an ARM resource operation, not a kubectl command, so it can be done via Crossplane/Azure SDK directly.

**Reuses pattern from**: `privatelinkservice/task/aks/endpoint_service.go`

```bash
# Discovery of existing kube-apiserver LB
export MC_RG="MC_${DP_RESOURCE_GROUP}_${DP_AKS_NAME}_${DP_LOCATION}"
export APISERVER_LB_FRONTEND_ID=$(az network lb frontend-ip list \
  --resource-group $MC_RG \
  --lb-name "kube-apiserver" \
  --query "[0].id" -o tsv)

# Create PLS backed by kube-apiserver LB (same pattern as Kong PLS)
az network private-link-service create \
  --resource-group $DP_RESOURCE_GROUP \
  --name "apiserver-pls" \
  --vnet-name $DP_VNET_NAME \
  --subnet $DP_PLS_NAT_SUBNET \
  --lb-frontend-ip-configs $APISERVER_LB_FRONTEND_ID \
  --location $DP_LOCATION
```

#### `task/aks/apiserver_endpoint.go` — Create PE for API server in our VNet

Creates the Private Endpoint for the API server PLS in our control plane VNet.

- Create `azurenetworkv1beta1.PrivateEndpoint` with:
  - `ProviderConfigReference` → **our** subscription (`Config.ControlPlaneCrossplaneProviderConfig`)
  - `SubnetID` → our control plane subnet
  - `PrivateConnectionResourceAlias` → apiserver-pls alias from previous task
  - `IsManualConnection: true` (cross-tenant)
- Self-approve PE connection via Azure SDK (cross-tenant auth)
- Wait for PE ready
- On delete: remove PE

**Reuses pattern from**: `privatelinkendpoint/task/aks/endpoint.go`

#### `task/aks/apiserver_dns.go` — Create API server DNS A record + kubeconfig readiness

Creates the `PrivateDNSARecord` for the API server PE in the shared regional API server private DNS zone.

- Read AKS `PrivateFQDN`
- Derive record name from the configured shared root domain (`dnsProviderConfig.azureAPIServerDNSZone.domain`)
- Create `azurenetworkv1beta1.PrivateDNSARecord`
  - `ProviderConfigReference` → DNS provider config for the shared API server zone
  - `ZoneName` / `ResourceGroupName` → shared zone config
  - `Records` → API server PE private IP
- Wait for record ready
- Create kubeconfig secret using the API server private FQDN
- Mark `K8SReady=true` and trigger one extra reconcile so remote client registration can happen
- On delete: remove only the A record

#### `task/aks/cluster_gateway_service.go` — Create PLS for Kong

Same as before — creates PLS backed by Kong's internal LB. No changes from existing design.

- Get Kong LB frontend IP config ID (from `APPBox.Status.GatewayBaseUrl` → Azure SDK to find LB resource)
- Create PLS with `LoadBalancerFrontendIPConfigurationIds` → Kong's LB frontend
- `NATIPConfiguration` → from `VPC.Status.PLSNATSubnets`
- Wait for PLS ready, store alias in APPBox status
- On delete: remove PLS

**Reuses pattern from**: `privatelinkservice/task/aks/endpoint_service.go`

#### `task/aks/cluster_gateway_endpoint.go` — Create PE for Kong in our VNet

Same as before for the PE itself — creates the PE for Kong PLS in our VNet.

- Create `azurenetworkv1beta1.PrivateEndpoint` with:
  - `ProviderConfigReference` → **our** subscription
  - `SubnetID` → our control plane subnet
  - `PrivateConnectionResourceAlias` → kong-pls alias
  - `IsManualConnection: true` (cross-tenant)
- Self-approve PE connection, wait for PE ready
- On delete: remove PE

**Reuses pattern from**: `privatelinkendpoint/task/aks/endpoint.go`

#### `task/aks/cluster_gateway_dns.go` — Create gateway DNS A record

Creates the `PrivateDNSARecord` for the Kong PE in the shared regional gateway private DNS zone.

- Use record name `privatelink-<aksClusterName>`
- Create `azurenetworkv1beta1.PrivateDNSARecord`
  - `ProviderConfigReference` → DNS provider config for `dnsProviderConfig.azureDNSZone`
  - `ZoneName` / `ResourceGroupName` → shared gateway zone config
  - `Records` → Kong PE private IP
- Wait for record ready
- Update `APPBox.Status.GatewayBaseUrl`
- On delete: remove only the A record

### Modified Files

#### `task/aks/aks.go` — Private AKS with VNet integration + kubeconfig

For BYOC:
- Set `PrivateClusterEnabled`, `PrivateDNSZone: "None"`, `VnetIntegrationEnabled`, `SubnetID` on AKS spec
- Replace kubeconfig `server` URL with `https://<api-pe-dns>` (direct API server, standard kubeconfig — no path prefix needed)

#### `task/aks/subnet.go` — API server subnet

Add a `/28` subnet delegated to `Microsoft.ContainerService/managedClusters` for the API server. Created alongside existing node and PLS NAT subnets.

#### `taskbuilder.go` — Insert BYOC tasks in AKS pipeline

```
Current AKS pipeline:
  ... → AKS → AdminNodePool → ... → ClusterGateway → ApiGatewayService → ClusterRole → ...

BYOC AKS pipeline:
  ... → AKS (private, VNet integrated)
      → ApiServerPLS               ← new (PLS backed by kube-apiserver LB)
      → ApiServerEndpoint          ← new (PE in our VNet)
      → ApiServerDNS               ← new (shared-zone A record + kubeconfig readiness)
      → [kubectl now works via API server PE]
      → AdminNodePool → ... → ClusterGateway → ApiGatewayService
      → ClusterGatewayService      ← new (PLS backed by Kong LB)
      → ClusterGatewayEndpoint     ← new (PE in our VNet)
      → ClusterGatewayDNS          ← new (shared-zone A record)
      → ClusterRole → ...
```

Key difference from AWS BYOC: API server PLS/PE are established **before** deploying components, because AKS is private from the start. Kong PLS/PE come later in the pipeline, after Kong is deployed.

#### `config/config.go` — Azure BYOC private link config

Add Azure-specific fields to `BYOCPrivateLink`:
```go
type BYOCPrivateLink struct {
    // existing AWS fields...
    ControlPlaneVPCID            string   // existing (AWS)
    ControlPlaneSubnetIDs        []string // existing (AWS)
    ControlPlaneSecurityGroupIDs []string // existing (AWS)
    // Azure-specific:
    ControlPlaneVNetID           string   // our VNet resource ID
    ControlPlaneSubnetID         string   // our subnet ID for PE placement
    ControlPlaneResourceGroup    string   // our RG for PE resources
}
```

Also add a dedicated shared-zone config for the API server private DNS zone:
```go
type DNSProviderConfig struct {
    AzureDNSZone          AzureDNSZoneConfig // existing shared gateway zone
    AzureAPIServerDNSZone AzureDNSZoneConfig // new shared API server zone
}
```

### Files Summary

| File | Action | Purpose |
|------|--------|---------|
| `task/aks/apiserver_pls.go` | **New** | Create PLS backed by kube-apiserver LB |
| `task/aks/apiserver_endpoint.go` | **New** | Create PE for API server in our VNet |
| `task/aks/apiserver_dns.go` | **New** | Create API server DNS A record in shared zone and finalize kubeconfig readiness |
| `task/aks/cluster_gateway_service.go` | **New** | Create PLS backed by Kong LB in customer VNet |
| `task/aks/cluster_gateway_endpoint.go` | **New** | Create PE for Kong in our VNet |
| `task/aks/cluster_gateway_dns.go` | **New** | Create gateway DNS A record in shared zone |
| `task/aks/aks.go` | Modify | Private AKS + VNet integration + kubeconfig |
| `task/aks/subnet.go` | Modify | Add API server subnet (/28) |
| `taskbuilder.go` | Modify | Insert 6 BYOC tasks in AKS pipeline |
| `config/config.go` | Modify | Azure BYOC private link config fields + shared API server DNS zone config |

### Code Patterns to Reuse

| Existing Code | File | Reuse For |
|---------------|------|-----------|
| PLS creation + NAT config | `privatelinkservice/task/aks/endpoint_service.go` | `apiserver_pls.go` and `cluster_gateway_service.go` |
| PE creation with manual connection | `privatelinkendpoint/task/aks/endpoint.go` | `apiserver_endpoint.go` and `cluster_gateway_endpoint.go` |
| DNS A record creation | `privatelinkendpoint/task/aks/dns.go` | `apiserver_dns.go` and `cluster_gateway_dns.go` |
| PE connection approval | `privatelinkservice/task/aks/endpoint.go` | Both endpoint files |
| Kong LB service creation | `task/aks/api_gateway_service.go` | Already provides `GatewayBaseUrl` |
| VPCEndpoint in CP VPC | `task/eks/cluster_gateway_endpoint.go` | Reference for Azure PE equivalent |
| VPCEndpointService | `task/eks/cluster_gateway_service.go` | Reference for Azure PLS equivalent |

### Why Dual Private Link (Not Single Kong Forward)

We evaluated routing all traffic (including kubectl) through Kong via a single private link. The dual private link approach was chosen because:

1. **No public→private transition** — AKS is private from creation. The single-link approach requires starting public (to deploy Kong), then restricting public access after PE is ready, introducing state machine complexity and risk.
2. **kubectl independent of Kong** — If Kong fails, kubectl still works via the API server PE. This is critical for debugging and recovery.
3. **Simpler code** — `apiserver_pls.go` (~150 lines) reuses the existing PLS pattern. The single-link approach requires a state machine for public→private transition (~250 lines, higher complexity).
4. **kube-apiserver LB is free** — AKS auto-creates it with VNet integration. No manual LB creation needed.

See `docs/azure-byoc-apiserver-access-comparison.md` for the full comparison.

---

## Part 3: Agent Authentication (Kong auth-server Plugin)

### Problem

Control-plane components access dataplane agents (cluster-proxy, cluster-agent, dataflow-agent) through Kong. These requests need authentication and authorization.

### Design

Reuse the existing Kong `auth-server` plugin — it is **cloud-agnostic**. The plugin validates bearer tokens by forwarding them to the K8s API server for RBAC checks.

```
Control-plane                         Kong                            AKS API Server
    │                                  │                                   │
    │ 1. Get Azure AD token            │                                   │
    │    (via GetDataPlaneAuthToken)    │                                   │
    │                                  │                                   │
    │ 2. Authorization: Bearer <token> │                                   │
    │ ────────────────────────────────►│                                   │
    │                                  │ 3. GET kubernetes.default/apis/   │
    │                                  │    core.pingcap.com/v1alpha1/     │
    │                                  │    namespaces/default/clusters    │
    │                                  │    Authorization: Bearer <token>  │
    │                                  │ ─────────────────────────────────►│
    │                                  │                                   │
    │                                  │ 4. AKS validates via Azure AD    │
    │                                  │    K8s RBAC checks permission    │
    │                                  │    200=ok, 401/403=denied        │
    │                                  │ ◄─────────────────────────────────│
    │                                  │                                   │
    │ 5. Forward to agent (or 401/403) │                                   │
    │ ◄────────────────────────────────│                                   │
```

The plugin (`helm/kong/plugin-auth-server/handler.lua`) is entirely cloud-agnostic:
- Extracts Bearer token from `Authorization` header
- Sends it to `https://kubernetes.default/apis/core.pingcap.com/v1alpha1/namespaces/default/clusters`
- K8s API server handles cloud-specific token validation (Azure AD for AKS, IAM Authenticator for EKS)
- Returns pass/fail based on K8s RBAC

### The Only Gap: Azure Token Generation

`cloud_provider_extractor_azure.go:97-102` currently returns an empty token:
```go
func (m *azureCloudProviderExtractor) GetDataPlaneAuthToken(_ string, _ *v1alpha1.APPBox) (string, error) {
    return "", nil  // ← needs implementation
}
```

For Azure, this needs to generate an Azure AD bearer token that the AKS API server accepts:

```go
func (m *azureCloudProviderExtractor) GetDataPlaneAuthToken(accountID string, appBox *v1alpha1.APPBox) (string, error) {
    // 1. Use azidentity to get token for AKS server-id audience
    //    For same-tenant: DefaultAzureCredential
    //    For BYOC cross-tenant: ClientAssertionCredential with multi-tenant app
    //
    // 2. The token must be accepted by AKS's Azure AD integration
    //    Audience: the AKS server application ID (e.g., "6dae42f8-4368-4678-94ff-3960e28e3630")
    //
    // 3. Cache with expiration (similar to AWS implementation with sync.Map)
    //
    // Reference: AWS implementation at cloud_provider_extractor_aws.go:321-348
    //   - Uses token.NewGenerator().GetWithOptions() for EKS
    //   - Caches tokens in sync.Map with 5-second expiration buffer
}
```

### Enable auth-server Plugin on Agent Ingresses

The plugin and `KongClusterPlugin` CR are already deployed via Kong Helm chart. Just enable on agent ingresses:

In Helm values / template configs:
```yaml
# cluster-proxy
ingress:
  enabled: true
  plugins: auth-server

# cluster-gateway (Kong itself)
plugins:
  configMaps:
    - pluginName: auth-server
      name: kong-plugin-auth-server

# dataflow-agent
ingress:
  enabled: true
  plugins: auth-server

# cluster-agent (via cluster-ng-service config)
kongDataplanePlugins: auth-server
```

### Files to Modify

| File | Change |
|------|--------|
| `pkg/service/dataplane/cloud_provider_extractor_azure.go` | Implement `GetDataPlaneAuthToken()` using Azure AD token |
| `examples/k8s-dedicated-templates.yaml` | Enable `auth-server` plugin on agent ingresses |
| `examples/k8s-dp-templates.yaml` | Enable `auth-server` plugin on dataflow-agent ingress |

### No Changes Needed

| Component | Why |
|-----------|-----|
| `helm/kong/plugin-auth-server/handler.lua` | Cloud-agnostic — works with any K8s bearer token |
| `helm/kong/plugin-auth-server/schema.lua` | Default `authorization_endpoint` points to `kubernetes.default` |
| `KongClusterPlugin` CR | Already created by Kong Helm chart |

---

## Implementation Order

### Phase 1: Cross-Tenant Auth Foundation

1. Register multi-tenant app in our Azure AD tenant
2. Add federated identity credentials for dpm-sa and crossplane-sa
3. Implement `ConvertToWorkloadIdentityLoginCrossTenant()` in `azure/utils.go`
4. Update `providerconfig/initializer.go` for BYOC ProviderConfig
5. Update `config.go` with BYOC config fields

### Phase 2: Private AKS + Dual Private Link

6. Update `subnet.go` to create API server subnet (/28, delegated)
7. Update `aks.go` for private AKS with VNet integration
8. Implement `apiserver_pls.go` (PLS backed by kube-apiserver LB)
9. Implement `apiserver_endpoint.go` (PE for API server in our VNet)
10. Implement `apiserver_dns.go` (shared-zone A record for API server + kubeconfig readiness)
11. Implement `cluster_gateway_service.go` (PLS backed by Kong LB)
12. Implement `cluster_gateway_endpoint.go` (PE for Kong in our VNet)
13. Implement `cluster_gateway_dns.go` (shared-zone A record for Kong)
14. Update `aks.go` for BYOC kubeconfig with API server PE URL
15. Update `taskbuilder.go` to wire 6 BYOC tasks

### Phase 3: Agent Auth

16. Implement `GetDataPlaneAuthToken()` in `cloud_provider_extractor_azure.go`
17. Enable `auth-server` plugin on agent ingresses in template configs

### Phase 4: AKS Creation Adjustments

18. Update `aks.go` for BYOC admin group / Azure RBAC configuration

## Verification

### Cross-Tenant Auth
- Create ProviderConfig with customer's tenantID → verify Crossplane can list resources in customer's subscription
- Retrieve kubeconfig with cross-tenant args → verify `kubectl get nodes` works against customer's AKS

### Private Link — API Server
- Verify `kube-apiserver` LB exists in MC_ resource group after AKS creation
- Verify apiserver-pls created with kube-apiserver LB frontend
- Verify apiserver-pe created in our VNet with allocated private IP
- Verify the shared API server DNS zone has an A record that resolves to the PE private IP
- `kubectl --server=https://<api-pe-dns>:443 get ns` works via private link
- Verify AKS API not reachable via public FQDN (private from creation)

### Private Link — Kong
- Verify kong-pls created in customer VNet with Kong LB frontend
- Verify kong-pe created in our VNet with allocated private IP
- Verify the shared gateway DNS zone has an A record that resolves to the PE private IP

### Agent Auth
- `curl -k https://<PE-DNS>/cluster-agent/...` without token → 401
- Generate Azure AD token via `GetDataPlaneAuthToken()` → `curl` with `Authorization: Bearer <token>` → 200
- Verify token with insufficient RBAC → 403