# Logging Stack This document introduces the current logging stack. We use [fluent-bit](https://github.com/fluent/fluent-bit) for log collection, [loki](https://github.com/grafana/loki) and AWS s3 to store logs, and grafana to query logs. See [design doc](https://docs.google.com/document/d/1susRpRD2uZ_cRrVoUa36ivcQI2wOHft6MchtO2M7DzU/edit#heading=h.efxhx09hxyyo) for more details. ## Log Collection We will deploy fluent-bit in the cluster through daemonset. The fluent-bit will collect all container logs and specify systemd service (docker, kubelet, containerd) logs. The container log must be printed to stdout/stderr, docker will automatically rotate. Log collection config: ``` [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /run/fluent-bit/flb_kube.db Mem_Buf_Limit 5MB [INPUT] Name systemd Tag host.* Path /var/log/journal Systemd_Filter _SYSTEMD_UNIT=docker.service Systemd_Filter _SYSTEMD_UNIT=kubelet.service Systemd_Filter _SYSTEMD_UNIT=containerd.service Systemd_Filter _SYSTEMD_UNIT=cloud-init.service DB /run/fluent-bit/flb_kube.db Mem_Buf_Limit 5MB ``` Logs will be processed by [fluent-bit-loki-plugin](https://github.com/grafana/loki/tree/master/cmd/fluent-bit) when they are sent to loki. `fluent-bit-loki-plugin` will add some labels for logs. ![log-labels](../images/log-labels.png) Log output config: - mTLS for loki and fluent-bit - Each fluent-bit has a different cluster label ``` [Output] Name loki Match * ca_file /fluent-bit/ssl/ca.crt cert_file /fluent-bit/ssl/tls.crt key_file /fluent-bit/ssl/tls.key extract_parsed_field SYSLOG_IDENTIFIER:syslog,_HOSTNAME:hostname,_TRANSPORT:stream Url https://ab38fa67a29b444b2923a6713bde1db1-595817497.us-west-2.elb.amazonaws.com:443/loki/api/v1/push TenantID "" BatchWait 1 BatchSize 10240 Labels {job="fluent-bit",cluster="base/eks/us-west-2"} RemoveKeys kubernetes,stream AutoKubernetesLabels false LabelMapPath /fluent-bit/etc/labelmap.json LineFormat json LogLevel info ``` See [fluent-bit](https://github.com/tidbcloud/infra/blob/51fcfd015c1d5a8ad9c0639a88dd856c3dc3cc19/deploy/monitoring/manifest/loki-stack/charts/fluent-bit/templates/configmap.yaml#L21-L37) for more detail. ## Deploy We use pulumi to deploy logging stack, you can find the code in [logging pulumi code](https://github.com/tidbcloud/infra/blob/51fcfd015c1d5a8ad9c0639a88dd856c3dc3cc19/deploy/monitoring/index.ts#L259-L290) Logs will be sent to S3 and use S3 to store long term logs. See [almost-zero-dependencies-setup](https://grafana.com/docs/loki/latest/configuration/examples/#almost-zero-dependencies-setup) for loki deploy. ## Monitor Loki and Fluent-bit We have monitored loki and fluent-bit, you can find the dashboard in grafana. ![log-labels](../images/fluent-bit-loki-dashboards.png) We encountered fluent-bit failed to send logs due to certificate expiration, so we have some alerts when fluent-bit has not pushed the log for a long time. [fluent-bit alerts](https://github.com/tidbcloud/infra/blob/51fcfd015c1d5a8ad9c0639a88dd856c3dc3cc19/deploy/monitoring/manifest/thanos/thanos-rule-configmap.yaml#L671-L684): ```yaml - name: fluent-bit rules: - alert: NoOutputBytesProcessed expr: rate(fluentbit_output_proc_bytes_total[5m]) == 0 annotations: message: | Fluent Bit instance {{ $labels.pod }}'s output plugin {{ $labels.name }} has not processed any bytes for at least 10 minutes. summary: No Output Bytes Processed for: 10m labels: severity: critical ``` ## Examples ### How TiDB General Logs and Slow Logs are Collected When TiDB (aka tidb-server here) runs, it prints general logs and slow logs both, which are two different kinds of logs. If we mix them in the stdout/stderr, it's not easy to separate them later, so we need to separate them when collecting. When tidb-server starts in the k8s container, we don't pass the `--log-file` option to it, which makes its general logs will be outputed to stdout/stderr according to tidb-server logic, while we pass the `--log-slow-query` option, which makes the slow logs will be stored in the file. Verification: ```sh > kubectl exec -it db-tidb-0 -c tidb -n nightly -- /bin/sh / # ps aux | grep tidb-server 1 root 7d11 /tidb-server --store=tikv --advertise-address=db-tidb-0.db-tidb-peer.nightly.svc --host=0.0.0.0 --path=db-pd:2379 --config=/etc/tidb/tidb.toml --log-slow-query=/var/log/tidb/slowlog 54 root 0:00 grep tidb-server / # cat /etc/tidb/tidb.toml [log] [log.file] max-backups = 3 ``` The shell script `/usr/local/bin/tidb_start_script.sh` in the container controls how to start the tidb-server. You can view its content by the following commands. ```sh > kubectl exec -it db-tidb-0 -c tidb -n nightly -- /bin/sh / # cat /usr/local/bin/tidb_start_script.sh #!/bin/sh # This script is used to start tidb containers in kubernetes cluster ... ARGS="--store=tikv \ --advertise-address=${POD_NAME}.${HEADLESS_SERVICE_NAME}.${NAMESPACE}.svc \ --host=0.0.0.0 \ --path=${CLUSTER_NAME}-pd:2379 \ --config=/etc/tidb/tidb.toml " ... if [[ ! -z "${SLOW_LOG_FILE}" ]] then ARGS="${ARGS} --log-slow-query=${SLOW_LOG_FILE:-}" fi echo "start tidb-server ..." echo "/tidb-server ${ARGS}" exec /tidb-server ${ARGS} ``` The general logs are printed to the stdout/stderr, they will be stored in the `/var/log/containers/*.log` by the docker, then be sent to the loki by the fulent-bit as explained in above. The slow logs are stored in the `/var/log/tidb/slowlog` file. We run another simple container to mount the slow logs file, and use `tail -n0 -F` command to print the slow logs to the stdout of this container. They will be stored in the `/var/log/containers/*.log` by the docker, then be sent to the loki by the fulent-bit as same as the general logs. ```yaml > kubectl describe pod/db-tidb-0 -n nightly Name: db-tidb-0 Namespace: nightly ... Containers: slowlog: ... Image: busybox:1.26.2 Command: sh -c touch /var/log/tidb/slowlog; tail -n0 -F /var/log/tidb/slowlog; ... Mounts: /var/log/tidb from slowlog (rw) ... tidb: ... Image: pingcap/tidb:v4.0.0-rc.2 ... Command: /bin/sh /usr/local/bin/tidb_start_script.sh Environment: CLUSTER_NAME: db TZ: BINLOG_ENABLED: false SLOW_LOG_FILE: /var/log/tidb/slowlog POD_NAME: db-tidb-0 (v1:metadata.name) NAMESPACE: nightly (v1:metadata.namespace) HEADLESS_SERVICE_NAME: db-tidb-peer Mounts: ... /var/log/tidb from slowlog (rw) ... ``` By doing this, tidb-server general logs and slow logs are collected in the different containers, so they have different `container` label when they are stored in the loki. The former has `{container='tidb'}` label while the latter has `{container='slowlog'}` label. We can use this label to differ them in the grafana.