{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      },
      {
        "datasource": "Infinity",
        "enable": true,
        "iconColor": "blue",
        "name": "anomaly",
        "target": {
          "columns": [],
          "filters": [],
          "format": "dataframe",
          "global_query_id": "",
          "limit": 100,
          "matchAny": false,
          "refId": "",
          "root_selector": "",
          "source": "url",
          "tags": [],
          "type": "json",
          "url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/annotations",
          "url_options": {
            "data": "",
            "method": "GET",
            "params": [
              {
                "key": "start_ts",
                "value": "${__from:date:seconds}"
              },
              {
                "key": "end_ts",
                "value": "${__to:date:seconds}"
              },
              {
                "key": "tidb_cluster",
                "value": "${tidb_cluster}"
              },
              {
                "key": "measurement",
                "value": "avg_latency_analysis_anno"
              }
            ]
          }
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": 14,
  "iteration": 1647503404820,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 82,
      "panels": [],
      "title": "Read/Write scenario and imbalance",
      "type": "row"
    },
    {
      "datasource": "Infinity",
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 1
      },
      "id": 98,
      "options": {
        "content": "在延迟性能排查过程中，除上图主线以外，读写场景的区分、不同场景下实例间的不平衡性等也是应该充分考虑和排查的，在此给出一些线索，供排查人员参考。如 TiKV 侧延迟正常，则以下为空。\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n{{#if (eq total_grpc_good_time_1 0)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n在延迟异常及前 10 分钟内，写请求占 {{grpc_proportion_write_time_1}}% ，Coprocessor 读请求占 {{grpc_proportion_copr_time_1}}%, 点读请求占 {{grpc_proportion_small_read_time_1}}%。\n\n这段时间，写请求的不平衡度为 {{imbalance_write_ratio_time_1}}， 基线 1 的不平衡度为 {{imbalance_write_baseline_1d_time_1}}， 基线 2 的不平衡度为 {{imbalance_write_baseline_1h_time_1}}。较基线相比，{{#if (eq imbalance_write_trend_time_1 0)}}无明显变化。{{/if}}{{#if (eq imbalance_write_trend_time_1 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_write_trend_time_1 2)}}有下降趋势。{{/if}}\n\n这段时间，Coprocessor 读请求的不平衡度为 {{imbalance_copr_ratio_time_1}}， 基线 1 的不平衡度为 {{imbalance_copr_baseline_1d_time_1}}， 基线 2 的不平衡度为 {{imbalance_copr_baseline_1h_time_1}}。较基线相比，{{#if (eq imbalance_copr_trend_time_1 0)}}无明显变化。{{/if}}{{#if (eq imbalance_copr_trend_time_1 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_copr_trend_time_1 2)}}有下降趋势。{{/if}}\n\n这段时间，点读请求的不平衡度为 {{imbalance_small_read_ratio_time_1}}， 基线 1 的不平衡度为 {{imbalance_small_read_baseline_1d_time_1}}， 基线 2 的不平衡度为 {{imbalance_small_read_baseline_1h_time_1}}。较基线相比，{{#if (eq imbalance_small_read_trend_time_1 0)}}无明显变化。{{/if}}{{#if (eq imbalance_small_read_trend_time_1 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_small_read_trend_time_1 2)}}有下降趋势。{{/if}}\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_2 1)}}\n{{#if (eq total_grpc_good_time_2 0)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n在延迟异常及前 10 分钟内，写请求占 {{grpc_proportion_write_time_2}}% ，coprocessor 读请求占 {{grpc_proportion_copr_time_2}}%, 点读请求占 {{grpc_proportion_small_read_time_2}}%。\n\n这段时间，写请求的不平衡度为 {{imbalance_write_ratio_time_2}}， 基线 1 的不平衡度为 {{imbalance_write_baseline_1d_time_2}}， 基线 2 的不平衡度为 {{imbalance_write_baseline_1h_time_2}}。较基线相比，{{#if (eq imbalance_write_trend_time_2 0)}}无明显变化。{{/if}}{{#if (eq imbalance_write_trend_time_2 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_write_trend_time_2 2)}}有下降趋势。{{/if}}\n\n这段时间，coprocessor 读请求的不平衡度为 {{imbalance_copr_ratio_time_2}}， 基线 1 的不平衡度为 {{imbalance_copr_baseline_1d_time_2}}， 基线 2 的不平衡度为 {{imbalance_copr_baseline_1h_time_2}}。较基线相比，{{#if (eq imbalance_copr_trend_time_2 0)}}无明显变化。{{/if}}{{#if (eq imbalance_copr_trend_time_2 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_copr_trend_time_2 2)}}有下降趋势。{{/if}}\n\n这段时间，点读请求的不平衡度为 {{imbalance_small_read_ratio_time_2}}， 基线 1 的不平衡度为 {{imbalance_small_read_baseline_1d_time_2}}， 基线 2 的不平衡度为 {{imbalance_small_read_baseline_1h_time_2}}。较基线相比，{{#if (eq imbalance_small_read_trend_time_2 0)}}无明显变化。{{/if}}{{#if (eq imbalance_small_read_trend_time_2 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_small_read_trend_time_2 2)}}有下降趋势。{{/if}}\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_3 1)}}\n{{#if (eq total_grpc_good_time_3 0)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n在延迟异常及前 10 分钟内，写请求占 {{grpc_proportion_write_time_3}}% ，coprocessor 读请求占 {{grpc_proportion_copr_time_3}}%, 点读请求占 {{grpc_proportion_small_read_time_3}}%。\n\n这段时间，写请求的不平衡度为 {{imbalance_write_ratio_time_3}}， 基线 1 的不平衡度为 {{imbalance_write_baseline_1d_time_3}}， 基线 2 的不平衡度为 {{imbalance_write_baseline_1h_time_3}}。较基线相比，{{#if (eq imbalance_write_trend_time_3 0)}}无明显变化。{{/if}}{{#if (eq imbalance_write_trend_time_3 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_write_trend_time_3 2)}}有下降趋势。{{/if}}\n\n这段时间，coprocessor 读请求的不平衡度为 {{imbalance_copr_ratio_time_3}}， 基线 1 的不平衡度为 {{imbalance_copr_baseline_1d_time_3}}， 基线 2 的不平衡度为 {{imbalance_copr_baseline_1h_time_3}}。较基线相比，{{#if (eq imbalance_copr_trend_time_3 0)}}无明显变化。{{/if}}{{#if (eq imbalance_copr_trend_time_3 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_copr_trend_time_3 2)}}有下降趋势。{{/if}}\n\n这段时间，点读请求的不平衡度为 {{imbalance_small_read_ratio_time_3}}， 基线 1 的不平衡度为 {{imbalance_small_read_baseline_1d_time_3}}， 基线 2 的不平衡度为 {{imbalance_small_read_baseline_1h_time_3}}。较基线相比，{{#if (eq imbalance_small_read_trend_time_3 0)}}无明显变化。{{/if}}{{#if (eq imbalance_small_read_trend_time_3 1)}}有上升趋势。{{/if}}{{#if (eq imbalance_small_read_trend_time_3 2)}}有下降趋势。{{/if}}\n\n{{/if}}\n{{/if}}\n{{/if}}",
        "defaultContent": "The query didn't return any results.",
        "everyRow": true
      },
      "targets": [
        {
          "columns": [],
          "data": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
          "datasource": "Infinity",
          "filters": [],
          "format": "dataframe",
          "global_query_id": "",
          "refId": "A",
          "root_selector": "",
          "source": "url",
          "type": "json",
          "url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
          "url_options": {
            "data": "",
            "method": "GET",
            "params": [
              {
                "key": "measurement",
                "value": "avg_latency_analysis_dt"
              },
              {
                "key": "start_ts",
                "value": "${__from:date:seconds}"
              },
              {
                "key": "end_ts",
                "value": "${__to:date:seconds}"
              },
              {
                "key": "tidb_cluster",
                "value": "${tidb_cluster}"
              },
              {
                "key": "default_1",
                "value": "total_latency_good"
              }
            ]
          }
        }
      ],
      "title": "基于平均延迟性能的自动化诊断报告",
      "type": "marcusolsson-dynamictext-panel"
    },
    {
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 9
      },
      "id": 12,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single"
        }
      },
      "targets": [
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_count[1m]))",
          "interval": "",
          "legendFormat": "Total gRPC count",
          "refId": "A"
        },
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_count{type=~\"kv_prewrite|kv_commit|kv_pessimistic_lock\"}[1m]))",
          "hide": false,
          "interval": "",
          "legendFormat": "Write gRPC count",
          "refId": "B"
        },
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_count{type=~\"kv_get|kv_scan|kv_batch_get|kv_batch_get_command\"}[1m]))",
          "hide": false,
          "interval": "",
          "legendFormat": "Get & batch get gRPC count",
          "refId": "C"
        },
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_count{type=\"coprocessor\"}[1m]))",
          "hide": false,
          "interval": "",
          "legendFormat": "Coprocessor gRPC count",
          "refId": "D"
        }
      ],
      "title": "gRPC scenario",
      "type": "timeseries"
    },
    {
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 9
      },
      "id": 60,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single"
        }
      },
      "targets": [
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"kv_prewrite|kv_commit|kv_pessimistic_lock\"}[1m])) by (instance)",
          "interval": "",
          "legendFormat": "Write gRPC count: {{instance}}",
          "refId": "A"
        }
      ],
      "title": "Write imbalance by instance",
      "type": "timeseries"
    },
    {
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 17
      },
      "id": 64,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single"
        }
      },
      "targets": [
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"coprocessor\"}[1m])) by (instance)",
          "interval": "",
          "legendFormat": "Coprocessor gRPC count: {{instance}}",
          "refId": "A"
        }
      ],
      "title": "Coprocessor imbalance by instance",
      "type": "timeseries"
    },
    {
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 17
      },
      "id": 62,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single"
        }
      },
      "targets": [
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"kv_get|kv_scan|kv_batch_get|kv_batch_get_command\"}[1m])) by (instance)",
          "interval": "",
          "legendFormat": "Get & batch get gRPC count: {{instance}}",
          "refId": "A"
        }
      ],
      "title": "Get & batch get imbalance by instance",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 25
      },
      "id": 92,
      "panels": [],
      "title": "Latency by TiKV scenario",
      "type": "row"
    },
    {
      "datasource": "Infinity",
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 26
      },
      "id": 94,
      "options": {
        "content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Coprocessor gRPC duration: 1s\n>\n> Get & batch get duration: 0.06s\n>\n> Write gRPC duration: 0.46455s\n\n在 TiKV gRPC message duration 有问题的时间段，我们对三大类读写操作进行分析。\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_1 0)}}\n\n在这段时间，有 {{instance_write_latency_time_1}} 个 TiKV 实例写延迟高， 有 {{instance_copr_latency_time_1}} 个 TiKV 节点 Coprocessor 读延迟高，有 {{instance_smallread_latency_time_1}} 个 TiKV 节点点读延迟高。\n\n### 写延迟高的实例\n\n{{#if (eq write_latency_instance_time_1_1 1)}}\n- {{write_latency_instance_time_1_1_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_2 1)}}\n- {{write_latency_instance_time_1_2_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_3 1)}}\n- {{write_latency_instance_time_1_3_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_4 1)}}\n- {{write_latency_instance_time_1_4_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_5 1)}}\n- {{write_latency_instance_time_1_5_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_6 1)}}\n- {{write_latency_instance_time_1_6_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_7 1)}}\n- {{write_latency_instance_time_1_7_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_8 1)}}\n- {{write_latency_instance_time_1_8_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_9 1)}}\n- {{write_latency_instance_time_1_9_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_1_10 1)}}\n- {{write_latency_instance_time_1_10_addr}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n### Coprocessor 读延迟高的实例\n\n{{#if (eq copr_latency_instance_time_1_1 1)}}\n- {{copr_latency_instance_time_1_1_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_2 1)}}\n- {{copr_latency_instance_time_1_2_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_3 1)}}\n- {{copr_latency_instance_time_1_3_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_4 1)}}\n- {{copr_latency_instance_time_1_4_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_5 1)}}\n- {{copr_latency_instance_time_1_5_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_6 1)}}\n- {{copr_latency_instance_time_1_6_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_7 1)}}\n- {{copr_latency_instance_time_1_7_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_8 1)}}\n- {{copr_latency_instance_time_1_8_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_9 1)}}\n- {{copr_latency_instance_time_1_9_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_1_10 1)}}\n- {{copr_latency_instance_time_1_10_addr}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n### 点读延迟高的实例\n\n{{#if (eq smallread_latency_instance_time_1_1 1)}}\n- {{smallread_latency_instance_time_1_1_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_2 1)}}\n- {{smallread_latency_instance_time_1_2_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_3 1)}}\n- {{smallread_latency_instance_time_1_3_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_4 1)}}\n- {{smallread_latency_instance_time_1_4_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_5 1)}}\n- {{smallread_latency_instance_time_1_5_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_6 1)}}\n- {{smallread_latency_instance_time_1_6_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_7 1)}}\n- {{smallread_latency_instance_time_1_7_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_8 1)}}\n- {{smallread_latency_instance_time_1_8_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_9 1)}}\n- {{smallread_latency_instance_time_1_9_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_1_10 1)}}\n- {{smallread_latency_instance_time_1_10_addr}}\n\n{{/if}}\n\n{{/if}}\n(以上为异常项，如上述无异常则空白)\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_2 0)}}\n\n在这段时间，有 {{instance_write_latency_time_2}} 个 tikv 实例写延迟高， 有 {{instance_copr_latency_time_2}} 个 tikv 节点 coprocessor 读延迟高，有 {{instance_smallread_latency_time_2}} 个 tikv 节点点读延迟高。\n\n### 写延迟高的实例\n\n{{#if (eq write_latency_instance_time_2_1 1)}}\n- {{write_latency_instance_time_2_1_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_2 1)}}\n- {{write_latency_instance_time_2_2_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_2 1)}}\n- {{write_latency_instance_time_2_2_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_3 1)}}\n- {{write_latency_instance_time_2_3_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_4 1)}}\n- {{write_latency_instance_time_2_4_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_5 1)}}\n- {{write_latency_instance_time_2_5_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_6 1)}}\n- {{write_latency_instance_time_2_6_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_7 1)}}\n- {{write_latency_instance_time_2_7_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_8 1)}}\n- {{write_latency_instance_time_2_8_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_9 1)}}\n- {{write_latency_instance_time_2_9_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_2_10 1)}}\n- {{write_latency_instance_time_2_10_addr}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n### Coprocessor 读延迟高的实例\n\n{{#if (eq copr_latency_instance_time_2_1 1)}}\n- {{copr_latency_instance_time_2_1_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_2 1)}}\n- {{copr_latency_instance_time_2_2_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_3 1)}}\n- {{copr_latency_instance_time_2_3_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_4 1)}}\n- {{copr_latency_instance_time_2_4_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_5 1)}}\n- {{copr_latency_instance_time_2_5_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_6 1)}}\n- {{copr_latency_instance_time_2_6_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_7 1)}}\n- {{copr_latency_instance_time_2_7_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_8 1)}}\n- {{copr_latency_instance_time_2_8_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_9 1)}}\n- {{copr_latency_instance_time_2_9_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_2_10 1)}}\n- {{copr_latency_instance_time_2_10_addr}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n### 点读延迟高的实例\n\n{{#if (eq smallread_latency_instance_time_2_1 1)}}\n- {{smallread_latency_instance_time_2_1_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_2 1)}}\n- {{smallread_latency_instance_time_2_2_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_3 1)}}\n- {{smallread_latency_instance_time_2_3_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_4 1)}}\n- {{smallread_latency_instance_time_2_4_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_5 1)}}\n- {{smallread_latency_instance_time_2_5_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_6 1)}}\n- {{smallread_latency_instance_time_2_6_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_7 1)}}\n- {{smallread_latency_instance_time_2_7_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_8 1)}}\n- {{smallread_latency_instance_time_2_8_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_9 1)}}\n- {{smallread_latency_instance_time_2_9_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_2_10 1)}}\n- {{smallread_latency_instance_time_2_10_addr}}\n\n{{/if}}\n\n{{/if}}\n(以上为异常项，如上述无异常则空白)\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_3 0)}}\n\n在这段时间，有 {{instance_write_latency_time_3}} 个 tikv 实例写延迟高， 有 {{instance_copr_latency_time_3}} 个 tikv 节点 coprocessor 读延迟高，有 {{instance_smallread_latency_time_3}} 个 tikv 节点点读延迟高。\n\n### 写延迟高的实例\n\n{{#if (eq write_latency_instance_time_3_1 1)}}\n- {{write_latency_instance_time_3_1_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_2 1)}}\n- {{write_latency_instance_time_3_2_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_2 1)}}\n- {{write_latency_instance_time_3_2_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_3 1)}}\n- {{write_latency_instance_time_3_3_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_4 1)}}\n- {{write_latency_instance_time_3_4_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_5 1)}}\n- {{write_latency_instance_time_3_5_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_6 1)}}\n- {{write_latency_instance_time_3_6_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_7 1)}}\n- {{write_latency_instance_time_3_7_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_8 1)}}\n- {{write_latency_instance_time_3_8_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_9 1)}}\n- {{write_latency_instance_time_3_9_addr}}\n\n{{/if}}\n{{#if (eq write_latency_instance_time_3_10 1)}}\n- {{write_latency_instance_time_3_10_addr}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n### Coprocessor 读延迟高的实例\n\n{{#if (eq copr_latency_instance_time_3_1 1)}}\n- {{copr_latency_instance_time_3_1_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_2 1)}}\n- {{copr_latency_instance_time_3_2_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_3 1)}}\n- {{copr_latency_instance_time_3_3_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_4 1)}}\n- {{copr_latency_instance_time_3_4_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_5 1)}}\n- {{copr_latency_instance_time_3_5_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_6 1)}}\n- {{copr_latency_instance_time_3_6_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_7 1)}}\n- {{copr_latency_instance_time_3_7_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_8 1)}}\n- {{copr_latency_instance_time_3_8_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_9 1)}}\n- {{copr_latency_instance_time_3_9_addr}}\n\n{{/if}}\n{{#if (eq copr_latency_instance_time_3_10 1)}}\n- {{copr_latency_instance_time_3_10_addr}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n### 点读延迟高的实例\n\n{{#if (eq smallread_latency_instance_time_3_1 1)}}\n- {{smallread_latency_instance_time_3_1_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_2 1)}}\n- {{smallread_latency_instance_time_3_2_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_3 1)}}\n- {{smallread_latency_instance_time_3_3_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_4 1)}}\n- {{smallread_latency_instance_time_3_4_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_5 1)}}\n- {{smallread_latency_instance_time_3_5_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_6 1)}}\n- {{smallread_latency_instance_time_3_6_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_7 1)}}\n- {{smallread_latency_instance_time_3_7_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_8 1)}}\n- {{smallread_latency_instance_time_3_8_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_9 1)}}\n- {{smallread_latency_instance_time_3_9_addr}}\n\n{{/if}}\n{{#if (eq smallread_latency_instance_time_3_10 1)}}\n- {{smallread_latency_instance_time_3_10_addr}}\n\n{{/if}}\n\n{{/if}}\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n{{/if}}",
        "defaultContent": "The query didn't return any results.",
        "everyRow": true
      },
      "targets": [
        {
          "columns": [],
          "data": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
          "datasource": "Infinity",
          "filters": [],
          "format": "dataframe",
          "global_query_id": "",
          "refId": "A",
          "root_selector": "",
          "source": "url",
          "type": "json",
          "url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
          "url_options": {
            "data": "",
            "method": "GET",
            "params": [
              {
                "key": "measurement",
                "value": "avg_latency_analysis_dt"
              },
              {
                "key": "start_ts",
                "value": "${__from:date:seconds}"
              },
              {
                "key": "end_ts",
                "value": "${__to:date:seconds}"
              },
              {
                "key": "tidb_cluster",
                "value": "${tidb_cluster}"
              },
              {
                "key": "default_1",
                "value": "total_latency_good"
              }
            ]
          }
        }
      ],
      "title": "基于平均延迟性能的自动化诊断报告",
      "type": "marcusolsson-dynamictext-panel"
    },
    {
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 34
      },
      "id": 42,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single"
        }
      },
      "targets": [
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"coprocessor\"}[1m])) / sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"coprocessor\"}[1m]))",
          "interval": "",
          "legendFormat": "avg-total",
          "refId": "A"
        },
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "(sum(rate(tikv_grpc_msg_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"coprocessor\"}[1m])) by (instance)) / (sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"coprocessor\"}[1m])) by (instance))",
          "hide": true,
          "interval": "",
          "legendFormat": "avg-{{instance}}",
          "refId": "B"
        }
      ],
      "title": "Coprocessor gRPC duration",
      "type": "timeseries"
    },
    {
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 34
      },
      "id": 44,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single"
        }
      },
      "targets": [
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"kv_batch_get|kv_batch_get_command|kv_get|kv_scan\"}[1m])) / sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"kv_batch_get|kv_batch_get_command|kv_get|kv_scan\"}[1m]))",
          "interval": "",
          "legendFormat": "avg-total",
          "refId": "A"
        },
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "(sum(rate(tikv_grpc_msg_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"kv_batch_get|kv_batch_get_command|kv_get|kv_scan\"}[1m])) by (instance)) / (sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=~\"kv_batch_get|kv_batch_get_command|kv_get|kv_scan\"}[1m])) by (instance))",
          "hide": true,
          "interval": "",
          "legendFormat": "avg-{{instance}}",
          "refId": "B"
        }
      ],
      "title": "Get & batch get duration",
      "type": "timeseries"
    },
    {
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 42
      },
      "id": 24,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single"
        }
      },
      "targets": [
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "sum(rate(tikv_grpc_msg_duration_seconds_sum{type=~\"kv_prewrite|kv_commit|kv_pessimistic_lock\"}[1m])) / sum(rate(tikv_grpc_msg_duration_seconds_count{type=~\"kv_prewrite|kv_commit|kv_pessimistic_lock\"}[1m]))",
          "hide": false,
          "interval": "",
          "legendFormat": "avg-total",
          "refId": "A"
        },
        {
          "datasource": "Prometheus",
          "exemplar": false,
          "expr": "(sum(rate(tikv_grpc_msg_duration_seconds_sum{type=~\"kv_prewrite|kv_commit|kv_pessimistic_lock\"}[1m])) by (instance)) / (sum(rate(tikv_grpc_msg_duration_seconds_count{type=~\"kv_prewrite|kv_commit|kv_pessimistic_lock\"}[1m])) by (instance))",
          "hide": true,
          "interval": "",
          "legendFormat": "avg-{{instance}}",
          "refId": "B"
        }
      ],
      "title": "Write gRPC duration",
      "type": "timeseries"
    }
  ],
  "refresh": false,
  "schemaVersion": 34,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "datasource": "Prometheus-fake",
        "definition": "",
        "hide": 0,
        "includeAll": false,
        "label": "tidb_cluster",
        "multi": false,
        "name": "tidb_cluster",
        "options": [],
        "query": {
          "query": "label_values({job=\"pd\"},tidb_cluster)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": true,
          "text": [
            "All"
          ],
          "value": [
            "$__all"
          ]
        },
        "datasource": "Prometheus",
        "definition": "",
        "hide": 0,
        "includeAll": true,
        "label": "instance",
        "multi": true,
        "name": "instance",
        "options": [],
        "query": {
          "query": "label_values(pd_cluster_status{tidb_cluster=\"$tidb_cluster\"}, instance)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "TiKV scenario imbalance and avg latency of instances",
  "uid": "Al5q79L7z",
  "version": 6,
  "weekStart": ""
}