# 2022-04-07-19:43:06+0800 Remove leaked elb

- Cause: https://jira.tidbcloud.com/browse/DM-4731
- Operation:

The following ELB are leaked:
* ap-northeast-1: ade0f104dd1bd4b16bbb300e515b2994
* ap-northeast-1: a76e95a09e04744558e655adccef38b2
* ap-northeast-1: ae036a79a813541ad88445d665774e4e
* us-west-2: a1f2d9ffa9b1a4cad9ff385b5d53994b
* us-west-2: a1212dba4e6f34140a76059e7c8f9cf0
* us-west-2: a0208d892cc4448ca846ea9acea1a4e9

It can be confirmed as follows:
1. Describe the tags for these ELB
2. Using the tenant, shoot and tidb cluster tags to confirm there is no alive cluster for the ELB

``` shell
⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ aws elbv2 describe-tags --resource-arns arn:aws:elasticloadbalancing:ap-northeast-1:380838443567:loadbalancer/net/ade0f104dd1bd4b16bbb300e515b2994/5e50d7d5abcc8842 arn:aws:elasticloadbalancing:ap-northeast-1:380838443567:loadbalancer/net/a76e95a09e04744558e655adccef38b2/d0783d86f94921e9 arn:aws:elasticloadbalancing:ap-northeast-1:380838443567:loadbalancer/net/ae036a79a813541ad88445d665774e4e/6e3f857388eac7c0 --region ap-northeast-1 --output yaml
TagDescriptions:
- ResourceArn: arn:aws:elasticloadbalancing:ap-northeast-1:380838443567:loadbalancer/net/ae036a79a813541ad88445d665774e4e/6e3f857388eac7c0
  Tags:
  - Key: billing-tenant-id
    Value: '1372813089191661282'
  - Key: kubernetes.io/service-name
    Value: tidb1379661944598884065/db-tidb
  - Key: cluster
    Value: '1379661944598884065'
  - Key: billing-cluster-id
    Value: '1379661944598884065'
  - Key: component
    Value: nlb-private
  - Key: billing-cluster-status-from-central
    Value: CREATING
  - Key: billing-project-id
    Value: '1372813089190401289'
  - Key: project
    Value: '1372813089190401289'
  - Key: kubernetes.io/cluster/shoot--prod--8e7539dc
    Value: owned
  - Key: KubernetesCluster
    Value: shoot--prod--8e7539dc
  - Key: tenant
    Value: '1372813089191661282'
  - Key: usedby
    Value: customer-tidb
- ResourceArn: arn:aws:elasticloadbalancing:ap-northeast-1:380838443567:loadbalancer/net/a76e95a09e04744558e655adccef38b2/d0783d86f94921e9
  Tags:
  - Key: billing-tenant-id
    Value: '1372813089188241297'
  - Key: kubernetes.io/service-name
    Value: tidb1379661944582354070/db-tidb
  - Key: cluster
    Value: '1379661944582354070'
  - Key: billing-cluster-id
    Value: '1379661944582354070'
  - Key: kubernetes.io/cluster/shoot--prod--491a4c63
    Value: owned
  - Key: billing-cluster-status-from-central
    Value: CREATING
  - Key: billing-project-id
    Value: '1372813089188181288'
  - Key: project
    Value: '1372813089188181288'
  - Key: KubernetesCluster
    Value: shoot--prod--491a4c63
  - Key: component
    Value: NLB
  - Key: environment
    Value: prod
  - Key: servicetype
    Value: dedicated
  - Key: tenant
    Value: '1372813089188241297'
  - Key: usedby
    Value: customer-tidb
- ResourceArn: arn:aws:elasticloadbalancing:ap-northeast-1:380838443567:loadbalancer/net/ade0f104dd1bd4b16bbb300e515b2994/5e50d7d5abcc8842
  Tags:
  - Key: billing-tenant-id
    Value: '1'
  - Key: kubernetes.io/service-name
    Value: tidb1379661944602934065/db-tidb
  - Key: cluster
    Value: '1379661944602934065'
  - Key: billing-cluster-id
    Value: '1379661944602934065'
  - Key: billing-cluster-status-from-central
    Value: CREATING
  - Key: billing-project-id
    Value: '1372813089190911286'
  - Key: project
    Value: '1372813089190911286'
  - Key: KubernetesCluster
    Value: shoot--prod--ddeefae3
  - Key: component
    Value: nlb-private
  - Key: servicetype
    Value: dedicated
  - Key: kubernetes.io/cluster/shoot--prod--ddeefae3
    Value: owned
  - Key: tenant
    Value: '1'
  - Key: usedby
    Value: customer-tidb
⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ aws elbv2 describe-tags --resource-arns arn:aws:elasticloadbalancing:us-west-2:380838443567:loadbalancer/net/a1f2d9ffa9b1a4cad9ff385b5d53994b/ec7718e8716d1802 arn:aws:elasticloadbalancing:us-west-2:380838443567:loadbalancer/net/a1212dba4e6f34140a76059e7c8f9cf0/24569f7da80deaa3 arn:aws:elasticloadbalancing:us-west-2:380838443567:loadbalancer/net/a0208d892cc4448ca846ea9acea1a4e9/f7f7388da088b9c3 --region us-west-2 --output yaml
TagDescriptions:
- ResourceArn: arn:aws:elasticloadbalancing:us-west-2:380838443567:loadbalancer/net/a1212dba4e6f34140a76059e7c8f9cf0/24569f7da80deaa3
  Tags:
  - Key: billing-tenant-id
    Value: '1372813089187791289'
  - Key: kubernetes.io/service-name
    Value: tidb1379661944600774066/db-tidb-private
  - Key: cluster
    Value: '1379661944600774066'
  - Key: billing-cluster-id
    Value: '1379661944600774066'
  - Key: component
    Value: nlb
  - Key: billing-cluster-status-from-central
    Value: CREATING
  - Key: billing-project-id
    Value: '1372813089189141300'
  - Key: project
    Value: '1372813089189141300'
  - Key: KubernetesCluster
    Value: shoot--prod--bc674294
  - Key: kubernetes.io/cluster/shoot--prod--bc674294
    Value: owned
  - Key: tenant
    Value: '1372813089187791289'
  - Key: usedby
    Value: customer-tidb
- ResourceArn: arn:aws:elasticloadbalancing:us-west-2:380838443567:loadbalancer/net/a0208d892cc4448ca846ea9acea1a4e9/f7f7388da088b9c3
  Tags:
  - Key: billing-tenant-id
    Value: '1372813089188361295'
  - Key: kubernetes.io/service-name
    Value: tidb1379661944582864065/db-tidb
  - Key: cluster
    Value: '1379661944582864065'
  - Key: billing-cluster-id
    Value: '1379661944582864065'
  - Key: billing-cluster-status-from-central
    Value: CREATING
  - Key: billing-project-id
    Value: '1372813089188451287'
  - Key: project
    Value: '1372813089188451287'
  - Key: KubernetesCluster
    Value: shoot--prod--a30a77d
  - Key: component
    Value: NLB
  - Key: environment
    Value: prod
  - Key: servicetype
    Value: dedicated
  - Key: kubernetes.io/cluster/shoot--prod--a30a77d
    Value: owned
  - Key: tenant
    Value: '1372813089188361295'
  - Key: usedby
    Value: customer-tidb
- ResourceArn: arn:aws:elasticloadbalancing:us-west-2:380838443567:loadbalancer/net/a1f2d9ffa9b1a4cad9ff385b5d53994b/ec7718e8716d1802
  Tags:
  - Key: billing-tenant-id
    Value: '1372813089191661282'
  - Key: kubernetes.io/service-name
    Value: tidb1379661944598584065/db-tidb
  - Key: cluster
    Value: '1379661944598584065'
  - Key: billing-cluster-id
    Value: '1379661944598584065'
  - Key: billing-cluster-status-from-central
    Value: CREATING
  - Key: billing-project-id
    Value: '1372813089190401289'
  - Key: project
    Value: '1372813089190401289'
  - Key: KubernetesCluster
    Value: shoot--prod--fa5db089
  - Key: kubernetes.io/cluster/shoot--prod--fa5db089
    Value: owned
  - Key: component
    Value: NLB
  - Key: environment
    Value: prod
  - Key: servicetype
    Value: dedicated
  - Key: tenant
    Value: '1372813089191661282'
  - Key: usedby
    Value: customer-tidb

❯ kubectl --kubeconfig ~/infra.kubeconfig get network -A  | grep fa5db089
prod-1372813089190401289   aws-us-west-2                  Normal          fa5db089   dbaas-prod-us-west-2-1372813089190401289         37d

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get cluster -n prod-1372813089190401289
No resources found in prod-1372813089190401289 namespace.

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get network -A  | grep bc674294
prod-1372813089189141300   aws-us-west-2                  Normal          bc674294   dbaas-prod-us-west-2-1372813089189141300         114d

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get cluster -n prod-1372813089189141300
No resources found in prod-1372813089189141300 namespace.

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get network -A  | grep a30a77d
prod-1372813089188451287   aws-us-west-2                  Normal          a30a77d    dbaas-prod-us-west-2-1372813089188451287         160d

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get cluster -n prod-1372813089188451287
No resources found in prod-1372813089188451287 namespace.

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get network -A  | grep 8e7539dc
prod-1372813089190401289   aws-ap-northeast-1             Hibernated      8e7539dc   dbaas-prod-ap-northeast-1-1372813089190401289    35d

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get cluster -n prod-1372813089190401289
No resources found in prod-1372813089190401289 namespace.

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get network -A  | grep 491a4c63
prod-1372813089188181288   aws-ap-northeast-1             PendingDelete   491a4c63   dbaas-prod-ap-northeast-1-1372813089188181288    170d

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get cluster -n prod-1372813089188181288
No resources found in prod-1372813089188181288 namespace.

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get network -A  | grep ddeefae3
prod-1372813089190911286   aws-ap-northeast-1             Normal          ddeefae3   dbaas-prod-ap-northeast-1-1372813089190911286    10d

⛵ prod-base-us-west-2 in infra on  master [!?] on ☁️  prod
❯ kubectl --kubeconfig ~/infra.kubeconfig get cluster -n prod-1372813089190911286
NAME                  NETWORK              PHASE      VERSION   AGE
1379661944604434068   aws-ap-northeast-1   Creating             6h1m
```

These ELB does not belong to any alive tidb clusters which means they are leaked.

So we can safely delete these ELBs from AWS console to save our cost.

- Impact: The leaked resources will be deleted
- Recovery operation: N/A

- Time window:
2022-04-07 23:00 ~ 2022-04-08 01:00