## What Happening
The "VMAgentJobMissing" alert indicates that the vmagent in the seed/base cluster is offline or unable to write to o11y backend.

## Related Issues or Alerts

## Diagnosis
> How to investigate the problem

1. Check the alert details for the specific instance of the alert to gather additional information about the affected VMAgent.
2. Access [Grafana](https://clinic.pingcap.com/grafana/d/G7Z9GzMGzxx/vmagent-dashboard?orgId=1&refresh=10s) to check the status associated with the vmagent instance.
3. If the pod status is abnormal, go to solution 1
4. If the log errors count is increasing, check the log to determine if it's due to a failure in writing to the backend (excluding "reset by peer" and "no such host"). If this is the case, go to solution 2

## Solution

1. Pod status abnormal:

    1. check if it's caused by configuration error, if so, fix the configuration and restart the pod.
    2. delete the pod to trigger a restart to see if it is recovered.

2. VM Cluster abnormal:
    1. verify the status of the corresponding region's VM cluster. Typically, investigate both VM insert and storage pod logs and status.
    2. if the vminsert's `storage connection saturation` is high and vminsert pod is under high pressure, consider resize or add more replicas for vminsert.
    3. if the concurrent insert is reaching the limit and the vmstorage's cpu usage or lsm parts is high and exists assited merge, consider resize the vmstorage instance.

## Post-Check

1. Verify the VMAgent's logs for any new error messages or warnings.
2. Verify the VMCluster's status, ensure the ingestion rate is steady.
3. Validate that the "VMAgentJobMissing" alert is no longer being triggered.