<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*

- [Troubleshooting](#troubleshooting)
  - [Introduction](#introduction)
  - [grafana crash by duplicate name](#grafana-crash-by-duplicate-name)
    - [Detail](#detail)
    - [Solution](#solution)
    - [Related](#related)
  - [loki entry out of order' for stream](#loki-entry-out-of-order-for-stream)
    - [Detail](#detail-1)
    - [Solution](#solution-1)
    - [Related](#related-1)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

# Troubleshooting

## Introduction

This is some experiences accumulated by DBaaS SRE in troubleshooting. The template is

```markdown
## Problem
> Add a short description of the problem
#### Solution
> Describe how to solve this problem step by step
### Detail
> Detailed information of the problem
### Related
> The linked issue pr or article
```

## grafana crash by duplicate name
grafana folder and dashboards can not have duplicate name. usually we think they are different resources. so it's easy to make mistakes

### Detail
this is grafana crash logs:
```bash
t=2020-09-09T06:09:05+0000 lvl=eror msg="A service failed" logger=server err="Failed to provision dashboards: Failed to provision config 6: got invalid response. expected folder, found dashboard"
t=2020-09-09T06:09:05+0000 lvl=eror msg="Server shutdown" logger=server reason="Failed to provision dashboards: Failed to provision config 6: got invalid response. expected folder, found dashboard"
```

### Solution
Avoid it if possible, you can name your dashboards as `Folder/...`

### Related
- https://github.com/tidbcloud/infra/issues/41


## loki entry out of order' for stream
fluent-bit failed send logs because of `entry out of order`

### Detail
this is loki logs
```bash
level=warn ts=2020-09-08T10:28:14.615816806Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=159.345µs err="rpc error: code = Code(400) desc = entry with timestamp 2020-09-08 10:28:13.595066592 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.628692366 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.636294619 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.645937502 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.662209662 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.695825909 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.696834359 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.708720878 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.724498316 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\nentry with timestamp 2020-09-08 10:28:13.72936418 +0000 UTC ignored, reason: 'entry out of order' for stream: {cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"},\ntotal ignored: 25 out of 25" msg="gRPC\n"
```

by the logs, we can find logs only have `{cluster=\"base/eks/us-west-2\", job=\"fluent-bit\", stream=\"stdout\"}` labels. Normally, there will be `instance` label to identify logs.

View the startup log of fluent-bit, loki-fluent-bit-plugin can not fetch metadate info from apiserver, it cause log does not have `instance` label. finally, we find loki clusterrole is missing.
```bash
[2020/09/09 04:10:17] [ warn] [filter:kubernetes:kubernetes.0] could not get meta for POD loki-fluent-bit-loki-krlfg
```

### Solution
I think it's a problem in `loki-fluent-bit-plugin`. it should `exit 1` or `keep reporting errors` instead of just reporting warning info in the startup log

### Related
- https://github.com/tidbcloud/infra/issues/38