# Nextgen Metering Auto Region Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add automatic new-region generation for nextgen metering across AWS, Alicloud, and Azure.

**Architecture:** Add a `deploy-new-region.sh` script to each nextgen metering tree, with provider-specific templates under `variants/{env}-{cloud}`. A GitHub workflow selects the correct work directory, invokes the script with explicit JSON arguments, registers the generated overlay in app-registry, and opens an automated PR.

**Tech Stack:** Bash, jq, Ruby YAML checks in tests, Kustomize via `kustomize build` or `kubectl kustomize`, GitHub Actions, Kustomize YAML manifests.

---

## File Structure

- Create `.github/scripts/test-nextgen-metering-deploy-new-region.sh`
  - Shell regression test for generated overlays.
  - Runs prod AWS, staging Alicloud, and dev Azure examples in temporary copies.
- Create `nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh`
  - Main generation script.
- Copy the same deploy script to:
  - `staging/nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh`
  - `dev/nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh`
- Create provider templates under:
  - `nextgen-regional-control-plane/nextgen-metering-server/variants/prod-aws`
  - `nextgen-regional-control-plane/nextgen-metering-server/variants/prod-alicloud`
  - `nextgen-regional-control-plane/nextgen-metering-server/variants/prod-azure`
  - `staging/nextgen-regional-control-plane/nextgen-metering-server/variants/staging-aws`
  - `staging/nextgen-regional-control-plane/nextgen-metering-server/variants/staging-alicloud`
  - `staging/nextgen-regional-control-plane/nextgen-metering-server/variants/staging-azure`
  - `dev/nextgen-regional-control-plane/nextgen-metering-server/variants/dev-aws`
  - `dev/nextgen-regional-control-plane/nextgen-metering-server/variants/dev-alicloud`
  - `dev/nextgen-regional-control-plane/nextgen-metering-server/variants/dev-azure`
- Create `.github/workflows/deploy-new-region-next-gen-metering-server.yaml`
  - Workflow dispatcher and automated PR creation.

---

### Task 1: Add Failing Generation Tests

**Files:**
- Create: `.github/scripts/test-nextgen-metering-deploy-new-region.sh`

- [ ] **Step 1: Write the failing test script**

Create `.github/scripts/test-nextgen-metering-deploy-new-region.sh` with:

```bash
#!/usr/bin/env bash
set -euo pipefail

ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
TMP_DIR=$(mktemp -d)
trap 'rm -rf "${TMP_DIR}"' EXIT

fail() {
  echo "FAIL: $*" >&2
  exit 1
}

assert_file() {
  [ -f "$1" ] || fail "expected file $1"
}

assert_no_file() {
  [ ! -f "$1" ] || fail "unexpected file $1"
}

assert_no_placeholders() {
  local dir="$1"
  if grep -R "TMPL_" "$dir" >/dev/null; then
    grep -R "TMPL_" "$dir" >&2
    fail "unreplaced placeholders under $dir"
  fi
}

assert_yaml() {
  local file="$1"
  local ruby_expr="$2"
  local expected="$3"
  local actual
  actual=$(ruby -ryaml -e "doc = YAML.load_file(ARGV[0]); value = ${ruby_expr}; print value" "$file")
  [ "$actual" = "$expected" ] || fail "$file expected $ruby_expr == $expected, got $actual"
}

copy_component_tree() {
  local rel="$1"
  mkdir -p "${TMP_DIR}/$(dirname "$rel")"
  cp -R "${ROOT}/${rel}" "${TMP_DIR}/${rel}"
}

kustomize_build() {
  local dir="$1"
  if command -v kustomize >/dev/null 2>&1; then
    kustomize build "$dir" >/dev/null
  elif command -v kubectl >/dev/null 2>&1; then
    kubectl kustomize "$dir" >/dev/null
  else
    echo "SKIP: kustomize and kubectl are unavailable; skipped kustomize build for $dir"
  fi
}

write_json() {
  local path="$1"
  shift
  printf '%s\n' "$*" > "$path"
}

test_prod_aws() {
  local rel="nextgen-regional-control-plane/nextgen-metering-server"
  copy_component_tree "$rel"
  local component="${TMP_DIR}/${rel}"
  local args="${TMP_DIR}/prod-aws-args.json"
  write_json "$args" '{
    "imageRegistry": "example.com/aws",
    "imageTag": "test-aws",
    "bucket": "tidbcloud-prod-us-east-2-ng-nextgen-metering",
    "startTime": 1700000000,
    "irsaRoleArn": "arn:aws:iam::111111111111:role/prod-us-east-2-ng-dataplane-manager",
    "meteringRoleArn": "arn:aws:iam::111111111111:role/prod-us-east-2-ng-metering-read-write",
    "featureGateFilter": {"gate-nextgen-test": {"fullSwitchTime": 1700000010}}
  }'

  bash "${component}/deploy-new-region.sh" us-east-2 prod aws us-east-2-ng "$args"

  local out="${component}/env/prod-aws-us-east-2-ng"
  assert_file "${out}/kustomization.yaml"
  assert_file "${out}/service-account.yaml"
  assert_file "${out}/service.yaml"
  assert_file "${out}/serviceconfig.yaml"
  assert_file "${out}/statefulset.yaml"
  assert_no_file "${out}/externalsecret.yaml"
  assert_no_placeholders "$out"
  assert_yaml "${out}/service-account.yaml" 'doc["metadata"]["annotations"]["eks.amazonaws.com/role-arn"]' 'arn:aws:iam::111111111111:role/prod-us-east-2-ng-dataplane-manager'
  assert_yaml "${out}/serviceconfig.yaml" 'doc["env"]' 'prod'
  assert_yaml "${out}/serviceconfig.yaml" 'doc["vendor"]' 'aws'
  assert_yaml "${out}/serviceconfig.yaml" 'doc["nextGenMetering"]["storageOptions"]["AWS_IAM_ROLE_ARN"]' 'arn:aws:iam::111111111111:role/prod-us-east-2-ng-metering-read-write'
  assert_yaml "${out}/serviceconfig.yaml" 'doc["nextGenMetering"]["featureGateFilter"]["gate-nextgen-test"]["fullSwitchTime"]' '1700000010'
  assert_yaml "${out}/statefulset.yaml" 'doc["spec"]["template"]["spec"]["containers"][0]["image"]' 'example.com/aws/tidbcloud/nextgen-metering-server:test-aws'
  kustomize_build "$out"
}

test_staging_alicloud() {
  local rel="staging/nextgen-regional-control-plane/nextgen-metering-server"
  copy_component_tree "$rel"
  local component="${TMP_DIR}/${rel}"
  local args="${TMP_DIR}/staging-alicloud-args.json"
  write_json "$args" '{
    "imageRegistry": "example.com/ali",
    "imageTag": "test-ali",
    "bucket": "nextgen-metering-staging-ap-southeast-2-ng",
    "startTime": 1700000100,
    "awsReporterRoleArn": "arn:aws:iam::222222222222:role/ali-ap-southeast-2-billing-staging-reporter"
  }'

  bash "${component}/deploy-new-region.sh" ap-southeast-2 staging alicloud ap-southeast-2-ng "$args"

  local out="${component}/env/staging-ali-ap-southeast-2-ng"
  assert_file "${out}/externalsecret.yaml"
  assert_no_placeholders "$out"
  assert_yaml "${out}/serviceconfig.yaml" 'doc["vendor"]' 'alicloud'
  assert_yaml "${out}/serviceconfig.yaml" 'doc["nextGenMetering"]["bucketEndpoint"]' 'https://oss-ap-southeast-2.aliyuncs.com'
  assert_yaml "${out}/statefulset.yaml" 'doc["spec"]["template"]["spec"]["containers"][0]["env"].find { |env| env["name"] == "AWS_ROLE_ARN" }["value"]' 'arn:aws:iam::222222222222:role/ali-ap-southeast-2-billing-staging-reporter'
  assert_yaml "${out}/statefulset.yaml" 'doc["spec"]["template"]["spec"]["volumeClaimTemplates"][0]["spec"]["storageClassName"]' 'alicloud-disk-essd'
  kustomize_build "$out"
}

test_dev_azure() {
  local rel="dev/nextgen-regional-control-plane/nextgen-metering-server"
  copy_component_tree "$rel"
  local component="${TMP_DIR}/${rel}"
  local args="${TMP_DIR}/dev-azure-args.json"
  write_json "$args" '{
    "imageRegistry": "example.com/azure",
    "imageTag": "test-azure",
    "bucket": "nextgen-metering-dev-eastus2-ng",
    "startTime": 1700000200,
    "azureWorkloadIdentityClientID": "00000000-1111-2222-3333-444444444444",
    "azureStorageAccountName": "meteringaccount",
    "bucketEndpoint": "https://meteringaccount.blob.core.windows.net",
    "awsReporterRoleArn": "arn:aws:iam::333333333333:role/azure-eastus2-ng-billing-dev-reporter"
  }'

  bash "${component}/deploy-new-region.sh" eastus2 dev azure eastus2-ng "$args"

  local out="${component}/env/dev-azure-eastus2-ng"
  assert_no_placeholders "$out"
  assert_yaml "${out}/service-account.yaml" 'doc["metadata"]["annotations"]["azure.workload.identity/client-id"]' '00000000-1111-2222-3333-444444444444'
  assert_yaml "${out}/serviceconfig.yaml" 'doc["vendor"]' 'azure'
  assert_yaml "${out}/serviceconfig.yaml" 'doc["nextGenMetering"]["storageOptions"]["AZURE_STORAGE_ACCOUNT_NAME"]' 'meteringaccount'
  assert_yaml "${out}/statefulset.yaml" 'doc["spec"]["template"]["metadata"]["labels"]["azure.workload.identity/use"]' 'true'
  assert_yaml "${out}/statefulset.yaml" 'doc["spec"]["template"]["spec"]["containers"][0]["env"].find { |env| env["name"] == "AWS_ROLE_ARN" }["value"]' 'arn:aws:iam::333333333333:role/azure-eastus2-ng-billing-dev-reporter'
  kustomize_build "$out"
}

test_prod_aws
test_staging_alicloud
test_dev_azure

echo "nextgen metering deploy-new-region tests passed"
```

- [ ] **Step 2: Run the test to verify it fails**

Run:

```bash
bash .github/scripts/test-nextgen-metering-deploy-new-region.sh
```

Expected: FAIL because `deploy-new-region.sh` does not exist in the copied nextgen metering trees.

- [ ] **Step 3: Commit the failing test**

Run:

```bash
git add .github/scripts/test-nextgen-metering-deploy-new-region.sh
git commit -m "test: cover nextgen metering region generation"
```

---

### Task 2: Add Deploy Script and Templates

**Files:**
- Create: `nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh`
- Create: `nextgen-regional-control-plane/nextgen-metering-server/variants/prod-aws/*`
- Create: `nextgen-regional-control-plane/nextgen-metering-server/variants/prod-alicloud/*`
- Create: `nextgen-regional-control-plane/nextgen-metering-server/variants/prod-azure/*`
- Copy to matching `staging/.../variants/staging-*`
- Copy to matching `dev/.../variants/dev-*`

- [ ] **Step 1: Run the failing test again**

Run:

```bash
bash .github/scripts/test-nextgen-metering-deploy-new-region.sh
```

Expected: FAIL because production code and templates are still missing.

- [ ] **Step 2: Implement the generation script**

Create `nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh` with a Bash script that:

- Validates five positional arguments.
- Normalizes `additionalArgs` whether the JSON file contains an object or a JSON-encoded string.
- Computes deploy env names with the approved nextgen metering conventions.
- Copies `variants/${env}-${cloud}` into `env/${deploy_env}`.
- Replaces placeholders with `sed -i.bak` and deletes only the created backup file.
- Requires `imageRegistry`, `imageTag`, `bucket`, and `startTime`.
- Requires `irsaRoleArn` for AWS.
- Requires `awsReporterRoleArn` for Alicloud.
- Requires `azureWorkloadIdentityClientID`, `azureStorageAccountName`, `bucketEndpoint`, and `awsReporterRoleArn` for Azure.
- Writes optional `plan`, `featureGateFilter`, `enablePprof`, `meteringRoleArn`, `ramSecretKey`, and timing defaults.
- Fails if generated files still contain `TMPL_`.

- [ ] **Step 3: Add provider templates**

Add template files for `prod-aws`, `prod-alicloud`, and `prod-azure`.

The AWS templates must include:

- EKS ServiceAccount role annotation.
- S3 Delta path.
- Optional `roleArn` and `AWS_IAM_ROLE_ARN` placeholder lines.
- `gp2` storage class.

The Alicloud templates must include:

- Plain ServiceAccount.
- `externalsecret.yaml`.
- OSS endpoint.
- `DUCKDB_PATCH_DELTA_SO`.
- AWS web identity token projection for reporter access.
- RAM secret env refs.
- `alicloud-disk-essd` storage class.

The Azure templates must include:

- Azure workload identity ServiceAccount annotation.
- Azure workload identity pod label.
- Azure Blob endpoint and storage account config.
- AWS web identity token projection for reporter access.
- `default` storage class.

- [ ] **Step 4: Copy script and templates to staging and dev trees**

Run:

```bash
cp nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh staging/nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh
cp nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh dev/nextgen-regional-control-plane/nextgen-metering-server/deploy-new-region.sh
mkdir -p staging/nextgen-regional-control-plane/nextgen-metering-server/variants
mkdir -p dev/nextgen-regional-control-plane/nextgen-metering-server/variants
cp -R nextgen-regional-control-plane/nextgen-metering-server/variants/prod-aws staging/nextgen-regional-control-plane/nextgen-metering-server/variants/staging-aws
cp -R nextgen-regional-control-plane/nextgen-metering-server/variants/prod-alicloud staging/nextgen-regional-control-plane/nextgen-metering-server/variants/staging-alicloud
cp -R nextgen-regional-control-plane/nextgen-metering-server/variants/prod-azure staging/nextgen-regional-control-plane/nextgen-metering-server/variants/staging-azure
cp -R nextgen-regional-control-plane/nextgen-metering-server/variants/prod-aws dev/nextgen-regional-control-plane/nextgen-metering-server/variants/dev-aws
cp -R nextgen-regional-control-plane/nextgen-metering-server/variants/prod-alicloud dev/nextgen-regional-control-plane/nextgen-metering-server/variants/dev-alicloud
cp -R nextgen-regional-control-plane/nextgen-metering-server/variants/prod-azure dev/nextgen-regional-control-plane/nextgen-metering-server/variants/dev-azure
```

- [ ] **Step 5: Run tests to verify the implementation passes**

Run:

```bash
bash .github/scripts/test-nextgen-metering-deploy-new-region.sh
```

Expected: PASS with `nextgen metering deploy-new-region tests passed`.

- [ ] **Step 6: Commit script and templates**

Run:

```bash
git add nextgen-regional-control-plane/nextgen-metering-server staging/nextgen-regional-control-plane/nextgen-metering-server dev/nextgen-regional-control-plane/nextgen-metering-server
git commit -m "feat: add nextgen metering region generator"
```

---

### Task 3: Add GitHub Workflow

**Files:**
- Create: `.github/workflows/deploy-new-region-next-gen-metering-server.yaml`

- [ ] **Step 1: Add workflow file**

Create `.github/workflows/deploy-new-region-next-gen-metering-server.yaml` with:

```yaml
name: Deploy new region next-gen metering-server

on:
  workflow_dispatch:
    inputs:
      cloud:
        description: "which cloud provider: aws, alicloud or azure"
        required: true
      env:
        description: "which env: dev, staging or prod"
        required: true
      region:
        description: "deploy to which region"
        required: true
      logicalRegionName:
        description: "deploy to which logical region"
        required: true
      cluster:
        description: "k8s cluster name"
        required: true
      jobId:
        description: "deploy job id"
        required: false
      additionalArgs:
        description: "JSON string of deployment arguments for nextgen-metering-server. Common fields: imageRegistry, imageTag, bucket, startTime. AWS requires irsaRoleArn. Alicloud requires awsReporterRoleArn. Azure requires azureWorkloadIdentityClientID, azureStorageAccountName, bucketEndpoint, and awsReporterRoleArn."
        type: string
        required: true
      author:
        description: "the person who invokes this action"
        required: false
        default: tidbcloud-bot

jobs:
  deployRegion:
    runs-on: tidbcloud-org-runners-v3
    steps:
      - uses: actions/checkout@v4
      - name: Set up yq
        uses: vegardit/gha-setup-yq@v1
        with:
          use-cache: true
          version: 4.44.1
      - name: Setup jq
        uses: vegardit/gha-setup-jq@v1
      - name: Set work directory and deploy env
        run: |
          work_dir="nextgen-regional-control-plane"
          if [ "${{ github.event.inputs.env }}" = "dev" ]; then
            work_dir="dev/nextgen-regional-control-plane"
          elif [ "${{ github.event.inputs.env }}" = "staging" ]; then
            work_dir="staging/nextgen-regional-control-plane"
          fi

          provider_alias="${{ github.event.inputs.cloud }}"
          if [ "$provider_alias" = "alicloud" ]; then
            provider_alias="ali"
          fi

          deploy_env="${{ github.event.inputs.env }}-${provider_alias}-${{ github.event.inputs.logicalRegionName }}"
          if [ "${{ github.event.inputs.env }}" = "dev" ]; then
            if [ "${{ github.event.inputs.cloud }}" = "aws" ]; then
              deploy_env="dev-${{ github.event.inputs.logicalRegionName }}"
            elif [ "${{ github.event.inputs.cloud }}" = "azure" ]; then
              deploy_env="dev-azure-${{ github.event.inputs.logicalRegionName }}"
            elif [ "${{ github.event.inputs.cloud }}" = "alicloud" ]; then
              deploy_env="alicloud-dev-${{ github.event.inputs.logicalRegionName }}"
            fi
          fi

          echo "work_dir=${work_dir}" >> "$GITHUB_ENV"
          echo "DEPLOY_ENV=${deploy_env}" >> "$GITHUB_ENV"
      - name: Check if deployed
        run: |
          bash .github/scripts/check-deployed.sh \
            "$work_dir/app-registry/nextgen-metering-server.yaml" \
            "${{ github.event.inputs.cluster }}"
      - name: Add new region manifests
        run: |
          cat > args.json <<'JSON_ARGS'
          ${{ github.event.inputs.additionalArgs }}
          JSON_ARGS
          bash "$work_dir/nextgen-metering-server/deploy-new-region.sh" \
            "${{ github.event.inputs.region }}" \
            "${{ github.event.inputs.env }}" \
            "${{ github.event.inputs.cloud }}" \
            "${{ github.event.inputs.logicalRegionName }}" \
            args.json
          rm args.json
      - name: Register nextgen-metering-server deployment to new region
        run: |
          yq -i '.spec.generators.0.list.elements += {"env": strenv(DEPLOY_ENV), "cluster": "${{ github.event.inputs.cluster }}"}' \
            "$work_dir/app-registry/nextgen-metering-server.yaml"
      - name: Create Pull Request
        id: cpr
        uses: peter-evans/create-pull-request@v8
        with:
          token: ${{ github.token }}
          commit-message: Deploy nextgen-metering-server in ${{ github.event.inputs.cloud }} ${{ github.event.inputs.env }} ${{ github.event.inputs.logicalRegionName }}
          committer: GitHub <noreply@github.com>
          author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
          signoff: false
          base: main
          branch: "create-pull-request/deploy-nextgen-metering-server-${{ github.event.inputs.cloud }}-${{ github.event.inputs.env }}-${{ github.event.inputs.logicalRegionName }}"
          branch-suffix: timestamp
          delete-branch: true
          assignees: ${{ github.event.inputs.author }}
          title: "Deploy nextgen-metering-server in next-gen ${{ github.event.inputs.cloud }} ${{ github.event.inputs.env }} ${{ github.event.inputs.logicalRegionName }}"
          body: |
            This PR is triggered by:
            - deploy job ${{ github.event.inputs.jobId }}
          labels: |
            automated pr
      - name: Mark PR as Approved in FAD
        uses: ./.github/actions/release-pr
        if: steps.cpr.outputs.pull-request-operation == 'created'
        with:
          pr-link: ${{ steps.cpr.outputs.pull-request-url }}
          ac-public-key: ${{ secrets.AC_PUBLIC_KEY }}
          ac-private-key: ${{ secrets.AC_PRIVATE_KEY }}
      - name: Trigger Check Run
        uses: ./.github/actions/run-check
        if: steps.cpr.outputs.pull-request-operation == 'created'
        with:
          pr-link: ${{ steps.cpr.outputs.pull-request-url }}
          token: ${{ secrets.GIT_ACTION_BOT }}
      - name: Check outputs
        run: |
          echo "Pull Request Number - ${{ steps.cpr.outputs.pull-request-number }}"
          echo "Pull Request URL - ${{ steps.cpr.outputs.pull-request-url }}"
      - name: Enable Pull Request Automerge
        if: steps.cpr.outputs.pull-request-operation == 'created'
        uses: peter-evans/enable-pull-request-automerge@v1
        with:
          token: ${{ secrets.GIT_ACTION_BOT }}
          pull-request-number: ${{ steps.cpr.outputs.pull-request-number }}
          merge-method: squash
      - name: Approve the PR
        if: steps.cpr.outputs.pull-request-operation == 'created'
        uses: hmarr/auto-approve-action@v2
        with:
          github-token: ${{ secrets.GIT_ACTION_BOT }}
          pull-request-number: ${{ steps.cpr.outputs.pull-request-number }}
```

- [ ] **Step 2: Run YAML and generation tests**

Run:

```bash
ruby -ryaml -e 'YAML.load_file(".github/workflows/deploy-new-region-next-gen-metering-server.yaml"); puts "workflow yaml ok"'
bash .github/scripts/test-nextgen-metering-deploy-new-region.sh
```

Expected: first command prints `workflow yaml ok`; second command passes.

- [ ] **Step 3: Commit workflow**

Run:

```bash
git add .github/workflows/deploy-new-region-next-gen-metering-server.yaml
git commit -m "ci: deploy nextgen metering to new regions"
```

---

### Task 4: Final Verification

**Files:**
- Read-only verification across changed files.

- [ ] **Step 1: Inspect git status**

Run:

```bash
git status --short
```

Expected: no uncommitted changes except the plan file if it has not been committed.

- [ ] **Step 2: Run full local verification**

Run:

```bash
bash .github/scripts/test-nextgen-metering-deploy-new-region.sh
ruby -ryaml -e 'YAML.load_file(".github/workflows/deploy-new-region-next-gen-metering-server.yaml"); puts "workflow yaml ok"'
git diff --check HEAD~3..HEAD
```

Expected:

- Generation test passes.
- Workflow YAML parses.
- `git diff --check` has no whitespace errors.

- [ ] **Step 3: Commit the implementation plan if still uncommitted**

Run:

```bash
git add docs/superpowers/plans/2026-05-18-nextgen-metering-auto-region.md
git commit -m "docs: plan nextgen metering auto region"
```

- [ ] **Step 4: Summarize changed commits**

Run:

```bash
git log --oneline --decorate -n 6
```

Expected: recent commits include the spec, test, generator, workflow, and plan commits.