# 2021-06-30-16:24:01+0800 fix cloudchat monitor federation down

- Cause: https://pingcap.pagerduty.com/incidents/P83XWAK
- Operation: manually separate the scrape job of TiKV and other components

```
$ kubectl edit configmap -n garden prod-seed-ap-northeast-2-tidb-monitor

# the diff should be
>     - job_name: tikv-monitor
>       honor_labels: true
>       honor_timestamps: true
>       params:
>         match[]:
>         - '{job=~".*tikv"}'
>       scrape_interval: 30s
>       scrape_timeout: 30s
>       metrics_path: /federate
>       scheme: http
>       tls_config:
>         cert_file: /srv/kubernetes/prometheus-kubelet/prometheus-kubelet.crt
>         key_file: /srv/kubernetes/prometheus-kubelet/prometheus-kubelet.key
>         insecure_skip_verify: true
>       relabel_configs:
>       - source_labels: [__meta_kubernetes_endpoints_name, __meta_kubernetes_endpoint_port_name]
>         separator: ;
>         regex: db-prometheus;http-prometheus
>         replacement: $1
>         action: keep
>       kubernetes_sd_configs:
>       - api_server: https://kube-apiserver:443
>         role: endpoints
>         tls_config:
>           ca_file: /etc/prometheus/seed/ca.crt
>           cert_file: /etc/prometheus/seed/prometheus.crt
>           key_file: /etc/prometheus/seed/prometheus.key
>           insecure_skip_verify: false
25d52
<         - '{job=~".*tikv"}'
```

- Impact: the configmap will be automatically reloaded by proemtheus and the alert would be fixed
- Recovery operation: N/A
- Time window: 16:00~16:10 CST, 2021-06-30