## What Happening
发生 WatchDog Down 报警报警时，表明 alert manager 挂了。alert manager 会发送心跳给 dead mans switch 服务，如果超过五分钟没有接受到心跳 dead mans switch 服务会直接往 jiralert 发送报警.

## Related Issues or Alerts
https://pingcap.pagerduty.com/incidents/PDLHJ7D?utm_source=slack&utm_campaign=channel

## Diagnosis
### 由于 NodeNotReady 导致
1. 查看当前 alertmanager 的状态, 发现是
```sh
$ kubectl get po -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Terminating   0          88m
```

## Solution
先通过重启机器快速恢复服务，具体如何排查 NodeNotReady 的原因见 [EKS-Node-Not-Ready](https://github.com/tidbcloud/runbooks/wiki/EKS-Node-Not-Ready)
1. found unready node
```sh
kubectl get node | grep NotReady
```
2. get instance-id from node spec.providerID information, the instance-id is i-037a8746fc385b6da
```sh
$ kubectl get node ip-10-0-178-227.us-west-2.compute.internal --template={{.spec.providerID}}
aws:///us-west-2c/i-037a8746fc385b6da
```

3. restart instance
```sh
aws ec2 reboot-instances --instance-ids i-037a8746fc385b6da
```

## Post-Check
watch node until ready
```sh
kubectl get node -w
```