# Add/Delete an EKS Region for Shared-Tier

Author: Wang Hongliang, Yiwen Chen

Freshness: 2021/11/12

Docs:
- https://docs.google.com/document/d/1GQsis4CAzAEsHkbRn9D6jdyVRqWPvMfWbTsF0JGDg1c/edit

## Add new EKS Region
Generally, the main procedure of adding region action includes these steps:

1. automotically submit 4 related PR by github action
2. deploy a EKS region
3. get the NAT Gateway Public IP of the new EKS region, add these IPs to thanos whitelist and Infra API server whitelist
4. merge PR in tidbcloud/shared-provider to deploy the shared-provider in new region, merge PR in tidbcloud/infra to deploy monitoring system, merge PR in tidbcloud/runbooks to deploy alarming system

Here's the details:

1. run action at [https://github.com/tidbcloud/shared-cd/actions/workflows/add-new-eks-freetier-region.yml](https://github.com/tidbcloud/shared-cd/actions/workflows/add-new-eks-freetier-region.yml)

    ![add new region action](./media/add-new-region-action.png)

    There are some issues you need to consider(which is mentioned in the action tips):

    1. `f01` is the identifier for the EKS clusters in a same region, it should be unique.
    2. EKS in different AWS region use different AMI, you need to use the corresponding AMI. You can refer to these AMI-id at [Amazon EKS optimized Amazon Linux AMIs - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) (You should use the AMI annotated with EKS version 1.21，x86).

    Once the action finished, there will be 4 PRs at these locations:
    1. [https://github.com/tidbcloud/shared-cd/pulls](https://github.com/tidbcloud/shared-cd/pulls)
    2. [https://github.com/tidbcloud/shared-provider/pulls](https://github.com/tidbcloud/shared-provider/pulls)
    3. [https://github.com/tidbcloud/infra/pulls](https://github.com/tidbcloud/infra/pulls)
    4. [https://github.com/tidbcloud/runbooks/pulls](https://github.com/tidbcloud/runbooks/pulls)

2. add whitelist(Prod)

    1. merge PR in [tidbcloud/shared-cd](https://github.com/tidbcloud/shared-cd/pulls), deploy new EKS cluster. 
    2. When the deploying action finished, get the public ip of the NAT gateway by action [https://github.com/tidbcloud/shared-cd-prod/actions/workflows/query_nat_gateway.yml](https://github.com/tidbcloud/shared-cd-prod/actions/workflows/query_nat_gateway.yml). The IP shows in the last output of action.
    3. Refer to these PR, add NAT gateway IP to these repo:
        1. add the whitelist of thanos: [https://github.com/tidbcloud/infra-cd/pull/329](https://github.com/tidbcloud/infra-cd/pull/329)
        2. add the whitelist of Infra API server: [https://github.com/tidbcloud/infra-cd/pull/329](https://github.com/tidbcloud/infra-cd/pull/329) (You need to cherry-pick this PR to the release branch)

3. check these quotas before deploying:
    1. Rules per VPC Security Group
    2. Network Load Balancer ENIs per VPC
    3. Network Load Balancers per Region
    4. modify AWS CNI [https://github.com/aws/amazon-vpc-cni-k8s#max_eni](https://github.com/aws/amazon-vpc-cni-k8s#max_eni)

4. merge these PR (if cd pipeline failed, just rerun, you can also check if there's something you miss in PR)
    1. [https://github.com/tidbcloud/shared-cd/pulls](https://github.com/tidbcloud/shared-cd/pulls), if you're going to deploy staging/prod region, you need to cherrypick the PR to the release-* branch.
    2. [https://github.com/tidbcloud/infra/pulls](https://github.com/tidbcloud/infra/pulls), merge the PR related to monitoring system. Check if you need to cherrypick.
    3. [https://github.com/tidbcloud/runbooks/pulls](https://github.com/tidbcloud/runbooks/pulls), merge the PR related to alarming system. 
    4. [https://github.com/tidbcloud/shared-provider/pulls](https://github.com/tidbcloud/shared-provider/pulls), merge the PR in shared-provider. If the PR in shared-provider merged before the eks/monitoring/alarming system PR, the cd pipeline in shared-provider will fail. You can rerun the cd pipeline after the PR above merged.
5. verify the operation success, refer to prod checklist.
6. add region in DBaaS backend.

## Destroy EKS sharedtier Region

1. ensure that there's an node working, otherwise the destroy pipeline will fail.
2. destroy the resources that will block the pipeline first.
    1. delete all TiDBCluster and TiDB Monitor in the EKS region you want to destroy. Ensure that the svc and pvc deleted.
    2. clean up the s3 bucket to store backup datas. s3 bucket can only be delete when it's empty.
    3. Delete Monitor
3. run action [https://github.com/tidbcloud/shared-cd/actions/workflows/destroy-eks-freetier-region.yml](https://github.com/tidbcloud/shared-cd/actions/workflows/destroy-eks-freetier-region.yml) to destroy region 
4. if the action failed, and aws-auth has been removed, you can directly delete EKS on AWS console, and clean up VPC, EIP, IAM role manually.
5. submit PR to remove the unused stack config.