## What Happening
Node in NotReady status

## Related Issues or Alerts
<!-- Parse issue or alert link-->

## Diagnosis

### use of closed network connection

1. Find node NotReady
```
🕙[ 18:03:55 ] ❯ kg node | grep NotReady
ip-10-0-178-242.us-west-2.compute.internal   NotReady   <none>   16d     v1.17.12-eks-7684af
ip-10-0-182-225.us-west-2.compute.internal   NotReady   <none>   16d     v1.17.12-eks-7684af
```
2. Use kubectl describe node ip-10-0-178-242.us-west-2.compute.internal, find kubelet stop post node status
```
Conditions:
  Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----             ------    -----------------                 ------------------                ------              -------
  MemoryPressure   Unknown   Wed, 28 Apr 2021 16:43:12 +0800   Wed, 28 Apr 2021 16:44:42 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure     Unknown   Wed, 28 Apr 2021 16:43:12 +0800   Wed, 28 Apr 2021 16:44:42 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure      Unknown   Wed, 28 Apr 2021 16:43:12 +0800   Wed, 28 Apr 2021 16:44:42 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready            Unknown   Wed, 28 Apr 2021 16:43:12 +0800   Wed, 28 Apr 2021 16:44:42 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
```

3. use aws ssm agent ssh to node
```bash
❯ kg node ip-10-0-178-242.us-west-2.compute.internal -o yaml | grep providerID
providerID: aws:///us-west-2c/i-05241190bec1013e2
❯ aws ssm start-session --target i-05241190bec1013e2
```

4. use journalctl view kubelet logs, it is a bug caused by golang network package

```bash
$ journalctl -u kubelet -r | grep 'node status'
Apr 28 08:49:00 ip-10-0-178-242.us-west-2.compute.internal kubelet[4141]: E0428 08:49:00.829083    4141 kubelet_node_status.go:402] Error updating node status, will retry: error getting node "ip-10-0-178-242.us-west-2.compute.internal": Get https://08A3C9212150518ECA1B44480567F22C.gr7.us-west-2.eks.amazonaws.com/api/v1/nodes/ip-10-0-178-242.us-west-2.compute.internal?timeout=10s: write tcp 10.0.178.242:33614->10.0.110.119:443: use of closed network connection
```

## Solution

1) restart node will fixes it
```bash
aws ec2 reboot-instances --instance-ids i-05241190bec1013e2 --output text
```

## Post-Check
<!-- Check issue fixes -->
watch node until it ready~
```bash
kubectl get node -w 
```