# 2021-11-20-06:53:35-0800 prometheus stuck in pending state

- Cause:
  - The root cause is the prometheus is auto scaled by vpa with very high memory request. The auto scaling is due to a bug in tikv metrics explained in this [document](https://docs.google.com/document/d/1VJzwxhqZ8cXf3jUhxrF8v8J097s3FH9jhfaK4DnEQiA/edit?n=Prometheus_Data_Migration_Checklist#heading=h.12pnvk1hlcbl)
    The root cause is the prometheus is auto scaled by vpa with very high memory request. The auto scaling is due to a bug in tikv metrics explained in this document. And the corresponding [fix](https://github.com/tidbcloud/infra-api/pull/504) but it's not released to prod env yet. And the corresponding fix is here [PR](https://github.com/tidbcloud/infra-api/pull/504) but it's not released to prod env yet.
  - ```
      Pod prometheus-0 can't be scheduled on prod-seed-ap-southeast-1-ap-southeast-1a-a01cc8bc-NodeGroup-15VVSW096LPR9, predicate failed: PodFitsResources predicate mismatch, reason: Insufficient memory,
      FailedScheduling 99s (x174 over 46m) default-scheduler 0/9 nodes are available: 2 Insufficient cpu, 9 Insufficient memory.
    ```
- Operation:

  - Raised PR to add new worker node [PR](https://github.com/tidbcloud/infra/pull/1706/files)
  - `kubectl delete pods prometheus-0 -n shoot--prod--b37987b1`
  - `kubectl delete pvc prometheus-db-prometheus-0 -n shoot--prod--c7e9f09`
  - `kubectl describe pods prometheus-0 -n shoot--prod--b37987b1 `
  - `kubectl describe nodes`

- Impact:
  loss of metrics during the downtime period

- Recovery operation:

  - Added new worker node (Did not fix the issue EBS can not cross AZ, so even there is enough compute resource in one AZ, the stateful pod may still unable to schedule due to the PV is already in another AZ )
  - Restart pod and pvc (Did not fix the issue)
  - Tried to Re-assign stateless pods to different nodes (Did not fix the issue)
  - Manually fixed the corresponding prometheus configurations in shoot cluster.

- Time window:
  - 09:47 AM to Nov 19, 2021 at 10:07 PM (for 12 hours)