# 2021-05-27-12:18:43+0800 restart unready node in prod-base-us-west-2

- Cause: https://pingcap.slack.com/archives/C01U9F6S1CL/p1622086727007400
The machine where the alertmanager is located is down，cause WatchDog Down alert.

- Operation:
1. found unready node
```sh
kubectl get node | grep NotReady
```
2. get instance-id from node spec.providerID information, the instance-id is i-037a8746fc385b6da
```sh
$ kubectl get node ip-10-0-178-227.us-west-2.compute.internal --template={{.spec.providerID}}
aws:///us-west-2c/i-037a8746fc385b6da
```

3. restart instance
```sh
aws ec2 reboot-instances --instance-ids i-037a8746fc385b6da
```

4. watch node until ready
```sh
kubectl get node -w
```

5. force delete alertmanager pod to recreate it
```sh
kubectl delete po alertmanager-main-0 -n monitoring --grace-period=0 --force
```

- Impact: node will ready
- Recovery operation:N/A
- Time window: 11:55~12:00 CST, May 27