# 2021-04-12-14:44:54+0800 Delete retirement aws instance

- Cause: https://jira.tidbcloud.com/browse/DM-578
- Operation:
1. login aws ec2 web console, search instance-id `i-03da639c497a675c2`
2. private IPv4 DNS is kubernetes node name, node name is `ip-10-0-111-200.us-west-2.compute.internal`
3. ensuse node `ip-10-0-111-200.us-west-2.compute.internal` instance is `i-03da639c497a675c2`
```bash
sh-4.2$ kubectl get node ip-10-0-111-200.us-west-2.compute.internal -o yaml | grep i-03da639c497a675c2
  providerID: aws:///us-west-2a/i-03da639c497a675c2
```
4. Mark node as unschedulable
```bash
kubectl cordon ip-10-0-111-200.us-west-2.compute.internal
```
5. graceful termination pod in node `ip-10-0-111-200.us-west-2.compute.internal`
```
kubectl drain ip-10-0-111-200.us-west-2.compute.internal --force --ignore-daemonsets --delete-local-data
```
6. wait pod schedule into other node
7. terminate instance `i-03da639c497a675c2` in aws web console

- Impact: all pod in `ip-10-0-111-200.us-west-2.compute.internal` will schedule into other node. but not affect customer
```yaml
  Namespace                     Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                     ----                                  ------------  ----------  ---------------  -------------  ---
  extension-provider-aws-v5jc7  mtu-customizer-9mc8x                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         4d17h
  falco                         falco-nkbfh                           100m (1%)     100m (1%)   256Mi (0%)       256Mi (0%)     4d17h
  garden                        gardener-apiserver-684f9d76f-5kgrt    200m (2%)     200m (2%)   1024M (3%)       1024M (3%)     4d17h
  kube-system                   aws-node-j7fcz                        10m (0%)      0 (0%)      0 (0%)           0 (0%)         4d17h
  kube-system                   kube-proxy-w9f6f                      100m (1%)     0 (0%)      0 (0%)           0 (0%)         4d17h
  logging                       loki-2                                4 (50%)       4 (50%)     4Gi (13%)        4Gi (13%)      4d18h
  logging                       loki-fluent-bit-loki-tbcnq            20m (0%)      1 (12%)     100Mi (0%)       200Mi (0%)     4d17h
  monitoring                    dead-mans-switch-6df869d88c-m9ffz     5m (0%)       5m (0%)     30Mi (0%)        30Mi (0%)      4d17h
  monitoring                    node-exporter-9zmhr                   112m (1%)     270m (3%)   200Mi (0%)       220Mi (0%)     4d17h
```
- Recovery operation: can not recovery
- Time window: 2021-04-12-15:30:00 ~ 2021-04-12-15:40:00