# 2021-09-17-14:51:27+0800 Fix fluent-bit unready

- Cause: `fluent bit pod unready alert`
- Operation:
pod unready cause by master node do not have enough CPU resources, another node have enough CPU resources. we can found have 2 coredns schedule to the node, we can delete the pod make it can schedule to another node. 

```yaml
Non-terminated Pods:          (22 in total)
  Namespace                   Name                                                   CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                   ------------  ----------  ---------------  -------------  ---
  cert-manager                aws-pca-issuer-6bd84ccfc5-5752b                        20m (1%)      0 (0%)      40Mi (1%)        0 (0%)         7h17m
  cert-manager                cert-manager-67bc8c79d9-hvp5b                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         7h22m
  cert-manager                cert-manager-cainjector-759496659c-889sd               0 (0%)        0 (0%)      0 (0%)           0 (0%)         7h22m
  cert-manager                cert-manager-webhook-7c75b89bf6-wkdxw                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         7h22m
  kube-system                 blackbox-exporter-ddb667d9-7x8kk                       20m (1%)      20m (1%)    15Mi (0%)        35Mi (1%)      7h22m
  kube-system                 calico-kube-controllers-7469f8b5c5-qxfdf               10m (0%)      50m (2%)    50Mi (1%)        100Mi (3%)     7h22m
  kube-system                 calico-node-th9mb                                      250m (13%)    800m (41%)  100Mi (3%)       700Mi (27%)    7h17m
  kube-system                 calico-node-vertical-autoscaler-5ddb459945-hv6pt       10m (0%)      10m (0%)    50Mi (1%)        50Mi (1%)      7h22m
  kube-system                 calico-typha-deploy-65cd9c79dc-jphl7                   200m (10%)    500m (26%)  100Mi (3%)       700Mi (27%)    6h40m
  kube-system                 calico-typha-horizontal-autoscaler-8487bc78b9-6b2s8    10m (0%)      10m (0%)    50Mi (1%)        50Mi (1%)      7h22m
  kube-system                 calico-typha-vertical-autoscaler-7cc9d6954b-jszcj      10m (0%)      10m (0%)    50Mi (1%)        50Mi (1%)      7h22m
  kube-system                 coredns-67ff54b854-dwvh9                               400m (20%)    400m (20%)  100Mi (3%)       100Mi (3%)     7h22m
  kube-system                 coredns-67ff54b854-qlpk7                               400m (20%)    400m (20%)  100Mi (3%)       100Mi (3%)     7h21m
  kube-system                 csi-driver-node-2jpqz                                  20m (1%)      50m (2%)    50Mi (1%)        80Mi (3%)      7h19m
  kube-system                 kube-proxy-cfsm2                                       20m (1%)      100m (5%)   64Mi (2%)        200Mi (7%)     7h19m
  kube-system                 metrics-server-f9b5676bb-xsjsh                         20m (1%)      80m (4%)    100Mi (3%)       100Mi (3%)     7h22m
  kube-system                 node-exporter-99g4j                                    25m (1%)      25m (1%)    40Mi (1%)        100Mi (3%)     7h19m
  kube-system                 node-problem-detector-d44dz                            20m (1%)      200m (10%)  20Mi (0%)        100Mi (3%)     7h19m
  kube-system                 vpn-shoot-7cc767798c-lm8pb                             100m (5%)     100m (5%)   100Mi (3%)       100Mi (3%)     7h22m
  tidb-admin                  advanced-statefulset-controller-7786b96bcc-27jrk       200m (10%)    500m (26%)  50Mi (1%)        300Mi (11%)    7h22m
  tidb-admin                  dbaas-proxy-674dd589b8-lpxnk                           20m (1%)      100m (5%)   80Mi (3%)        256Mi (10%)    7h17m
  tidb-admin                  tidb-scheduler-5dc7b464f7-nvckz                        160m (8%)     500m (26%)  100Mi (3%)       300Mi (11%)    7h22m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1915m (99%)   3855m (200%)
  memory             1259Mi (49%)  3421Mi (135%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-1Gi      0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
```

1. schedule coredns pod to another node
```bash
kubectl delete po coredns-67ff54b854-dwvh9 -n kube-system
```

2. watch filebeat and fluent-bit pod until it ready
```bash
kubectl get po -n logging -w
```

- Impact: `NONE`
- Recovery operation: `NONE`
- Time window: `2021-09-17-14:51:27+0800~2021-09-17-14:55:27+0800`