{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
},
{
"datasource": "Infinity",
"enable": true,
"iconColor": "blue",
"name": "anomaly",
"target": {
"columns": [],
"filters": [],
"format": "dataframe",
"global_query_id": "",
"limit": 100,
"matchAny": false,
"refId": "",
"root_selector": "",
"source": "url",
"tags": [],
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/annotations",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "measurement",
"value": "avg_latency_analysis_anno"
}
]
}
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": 11,
"iteration": 1647507131453,
"links": [],
"liveNow": false,
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 68,
"panels": [],
"title": "Summary based on average latency analysis",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 11,
"w": 24,
"x": 0,
"y": 1
},
"id": 28,
"links": [],
"options": {
"content": "> 限于 Grafana dynamic text 插件不能支持 nested 结构，目前展示的问题时间段数量以及异常实例数量均为硬编码，问题时间段上限为 3 个，每个时间段每类 gRPC 异常实例数量上限为 10 个。\n>\n> 报告组织结构外层为 TiDB 平均延迟的异常时间段，每一个异常时间段内，具有相同结构的内容，如“TiDB 侧”、“TiKV 侧”及“TiKV 读写场景、实例不平衡性及异常延迟”。\n\n本报告对 Query Latency 进行拆解，探索在 Query Latency 发现异常时，是哪些组件导致的问题。\n\n注意：如果 Total Latency 显示 No Data，则以下报告没有任何参考意义！\n\n整体分析思路基于 TiDB 的 Performance Map，见下图。\n\n ![\"perf\"](\"https://download.pingcap.com/images/docs-cn/dashboard/dashboard-diagnostics-time-relation.png\")
",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "{\n \"job_start_time\": 1640762139654,\n \"job_end_time\": 1640769467920\n}",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "inline",
"type": "json",
"url": "",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "avg_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good,conn_idle_good"
}
]
}
}
],
"title": "基于平均延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 12
},
"id": 66,
"panels": [],
"title": "Total latency - from client request to response",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 13
},
"id": 84,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Connection idle duration: 0.228s\n>\n> Total latency: 0.4s\n\n{{#if (eq total_latency_good 1)}}\n{{#if (eq conn_idle_good 1)}}\n集群平均延迟没有明显异常，且用户应用侧连接空闲时间也无明显异常，其它类型问题，建议结合“基于 P99 延迟性能的自动化诊断报告”、其它监控面板或 TiDB Dashboard 相关工具进行人工排查。\n\n{{/if}}\n{{#if (eq conn_idle_good 0)}}\n\n集群平均延迟没有明显异常，但应用连接空闲时间在“Connection Idle Duration”图表标记段内存在异常，考虑优先排查用户应用是否存在优化空间，然后再结合其它工具进行进一步分析。\n{{/if}}\n{{/if}}\n\n{{#if (eq total_latency_good 0)}}\n\n从指标 Total latency 看，以下时间段存在异常（图表中多于 3 个异常时间段时，限于插件原因暂时显示 3 个）。\n\n{{#if (eq time_1 1)}}\n\n集群从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内，系统延迟存在异常。\n\n在本段时间内，活跃连接占总体的比例为 {{conn_prop_time_1}}%，失败的 QPS 占总体的比例为 {{qps_prop_time_1}}%。\n\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n集群从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内，系统延迟存在异常。\n\n在本段时间内，活跃连接占总体的比例为 {{conn_prop_time_2}}%，失败的 QPS 占总体的比例为 {{qps_prop_time_2}}%。\n\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n集群从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内，系统延迟存在异常。\n\n在本段时间内，活跃连接占总体的比例为 {{conn_prop_time_3}}%，失败的 QPS 占总体的比例为 {{qps_prop_time_3}}%。\n\n{{/if}}\n\n{{#if (eq conn_idle_good 0)}}\n\n同时值得注意的是，应用连接空闲时间在“Connection Idle Duration”图表标记段内存在异常，考虑同时确认用户应用是否存在优化空间。\n{{/if}}\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "avg_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good,conn_idle_good"
}
]
}
}
],
"title": "基于平均延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"avg"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 21
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"pluginVersion": "8.3.4",
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tidb_server_handle_query_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", sql_type!=\"internal\"}[1m])) / sum(rate(tidb_server_handle_query_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", sql_type!=\"internal\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Total latency",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 21
},
"id": 102,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"pluginVersion": "8.3.4",
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tidb_executor_statement_total{tidb_cluster=\"$tidb_cluster\"}[1m]))",
"interval": "",
"legendFormat": "Total",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tidb_server_execute_error_total{tidb_cluster=\"$tidb_cluster\"}[1m]))",
"hide": false,
"interval": "",
"legendFormat": "Failed",
"refId": "B"
}
],
"title": "QPS",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 29
},
"id": 98,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"pluginVersion": "8.3.4",
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "(sum(rate(tidb_server_conn_idle_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", in_txn=\"1\"}[1m])) / sum(rate(tidb_server_conn_idle_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", in_txn=\"1\"}[1m])))",
"interval": "",
"legendFormat": "avg-in-txn",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "(sum(rate(tidb_server_conn_idle_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", in_txn=\"0\"}[1m])) / sum(rate(tidb_server_conn_idle_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", in_txn=\"0\"}[1m])))",
"hide": true,
"interval": "",
"legendFormat": "avt-not-in-txn",
"refId": "B"
}
],
"title": "Connection idle duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 29
},
"id": 100,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"pluginVersion": "8.3.4",
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(tidb_server_connections{tidb_cluster=\"$tidb_cluster\"})",
"interval": "",
"legendFormat": "Connection total",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tidb_server_handle_query_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", sql_type!=\"internal\"}[1m]))",
"hide": false,
"interval": "",
"legendFormat": "Active connections",
"refId": "B"
}
],
"title": "Connection count",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 37
},
"id": 72,
"panels": [],
"title": "Latency - TiDB side layer 1",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 38
},
"id": 86,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Get token duration: 2us\n>\n> Parse duration: 0.01s\n>\n> Compile duration: 0.038s\n>\n> Execute duration: 0.26s\n\n在指标 Total Latency 有问题的时间段，我们基于 Performance Map 寻找对其有影响的指标。\n\n![\"perf\"](\"https://s2.loli.net/2022/03/01/arqgj4KJEAXyv5f.png\")
\n\n在下拆的 4 个指标中，在 Total latency 异常的时间段附近，我们发现以下指标也存在异常。（如无异常则为空白）\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq get_token_latency_good_time_1 0)}}- Get token duration{{/if}}\n{{#if (eq parse_good_time_1 0)}}- Parse duration{{/if}}\n{{#if (eq compile_good_time_1 0)}}- Compile duration{{/if}}\n{{#if (eq execute_latency_good_time_1 0)}}- Execute duration{{/if}}\n\n{{#if (eq compile_good_time_1 0)}}\n\n这段时间内 Compile Duration 存在异常，建议点击“Compile Duration”面板左上角的 Panel link，查看进一步信息。\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq get_token_latency_good_time_2 0)}}- Get token duration{{/if}}\n{{#if (eq parse_good_time_2 0)}}- Parse duration{{/if}}\n{{#if (eq compile_good_time_2 0)}}- Compile duration{{/if}}\n{{#if (eq execute_latency_good_time_2 0)}}- Execute duration{{/if}}\n\n{{#if (eq compile_good_time_2 0)}}\n\n这段时间内 Compile Duration 存在异常，建议点击“Compile Duration”面板左上角的 Panel link，查看进一步信息。\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq get_token_latency_good_time_3 0)}}- Get token duration{{/if}}\n{{#if (eq parse_good_time_3 0)}}- Parse duration{{/if}}\n{{#if (eq compile_good_time_3 0)}}- Compile duration{{/if}}\n{{#if (eq execute_latency_good_time_3 0)}}- Execute duration{{/if}}\n\n{{#if (eq compile_good_time_3 0)}}\n\n这段时间内 Compile Duration 存在异常，建议点击“Compile Duration”面板左上角的 Panel link，查看进一步信息。\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "avg_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good,conn_idle_good"
}
]
}
}
],
"title": "基于平均延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "µs"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 46
},
"id": 30,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tidb_server_get_token_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\"}[1m])) / sum(rate(tidb_server_get_token_duration_seconds_count{tidb_cluster=\"$tidb_cluster\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Get token duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"avg"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 46
},
"id": 32,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "(sum(rate(tidb_session_parse_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])) / sum(rate(tidb_session_parse_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Parse duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 54
},
"id": 34,
"links": [
{
"targetBlank": true,
"title": "Sub - Compile Duration",
"url": "/d/rXMpi9Y7k/sub-compile-duration?${__all_variables}&${__url_time_range}"
}
],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "(sum(rate(tidb_session_compile_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])) / sum(rate(tidb_session_compile_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Compile duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 54
},
"id": 36,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "(sum(rate(tidb_session_execute_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])) / sum(rate(tidb_session_execute_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Execute duration",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 62
},
"id": 74,
"panels": [],
"title": "Execute latency - TiDB side layer 2 ",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 63
},
"id": 88,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Wait TSO duration: 0.03195s\n>\n> KV request duration: 0.4s\n>\n> KV backoff duration: 0.5s\n\n\n在指标 Execute Duration 有问题的时间段，我们基于 Performance Map 寻找对其有影响的指标。\n\n![\"perf\"](\"https://s2.loli.net/2022/03/01/w1VZ9RnHGdIxcfQ.png\")
\n\n（图中灰色指标 TiDB_execute 暂时尚未被 Prometheus 追踪）\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq wait_start_tso_good_time_1 0)}}- Wait_start_TSO duration{{/if}}\n{{#if (eq kv_backoff_good_time_1 0)}}- KV backoff duration{{/if}}\n{{#if (eq kv_request_good_time_1 0)}}- KV request duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq total_grpc_good_time_1 1)}}\n{{#if (eq kv_request_good_time_1 1)}}\n\nTiKV 侧以及 TiDB 请求 KV 的延迟均正常，建议优先排查执行计划，或 TiFlash(如有)，或结合其它工具（如 TiDB Dashboard）进行排查。\n{{/if}}\n{{/if}}\n\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq wait_start_tso_good_time_2 0)}}- Wait_start_TSO duration{{/if}}\n{{#if (eq kv_backoff_good_time_2 0)}}- KV backoff duration{{/if}}\n{{#if (eq kv_request_good_time_2 0)}}- KV request duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq total_grpc_good_time_2 1)}}\n{{#if (eq kv_request_good_time_2 1)}}\n\nTiKV 侧以及 TiDB 请求 KV 的延迟均正常，建议优先排查执行计划，或 TiFlash(如有)，或结合其它工具（如 TiDB Dashboard）进行排查。\n{{/if}}\n{{/if}}\n\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq wait_start_tso_good_time_3 0)}}- Wait_start_TSO duration{{/if}}\n{{#if (eq kv_backoff_good_time_3 0)}}- KV backoff duration{{/if}}\n{{#if (eq kv_request_good_time_3 0)}}- KV request duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq total_grpc_good_time_3 1)}}\n{{#if (eq kv_request_good_time_3 1)}}\n\nTiKV 侧以及 TiDB 请求 KV 的延迟均正常，建议优先排查执行计划，或 TiFlash(如有)，或结合其它工具（如 TiDB Dashboard）进行排查。\n{{/if}}\n{{/if}}\n\n{{/if}}\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "avg_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good,conn_idle_good"
}
]
}
}
],
"title": "基于平均延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 71
},
"id": 40,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "(sum(rate(tidb_session_execute_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])) / sum(rate(tidb_session_execute_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])))",
"hide": true,
"interval": "",
"legendFormat": "ave-execute-duration",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "(sum(rate(pd_client_cmd_handle_cmds_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", type=\"wait\"}[1m])) / sum(rate(pd_client_cmd_handle_cmds_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", type=\"wait\"}[1m])))",
"hide": false,
"interval": "",
"legendFormat": "avg",
"refId": "B"
}
],
"title": "Wait TSO duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 71
},
"id": 38,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_tikvclient_request_seconds_bucket{tidb_cluster=\"$tidb_cluster\", store!=\"0\"}[1m])) by (le))",
"hide": true,
"interval": "",
"legendFormat": "Average kv request duration",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tidb_tikvclient_request_seconds_sum{tidb_cluster=\"$tidb_cluster\", store!=\"0\"}[1m])) / sum(rate(tidb_tikvclient_request_seconds_count{tidb_cluster=\"$tidb_cluster\", store!=\"0\"}[1m]))",
"hide": false,
"interval": "",
"legendFormat": "avg",
"refId": "B"
}
],
"title": "KV request duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 79
},
"id": 56,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tidb_tikvclient_backoff_seconds_sum{tidb_cluster=\"$tidb_cluster\"}[1m])) / sum(rate(tidb_tikvclient_backoff_seconds_count{tidb_cluster=\"$tidb_cluster\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "KV backoff duration",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 87
},
"id": 70,
"panels": [],
"title": "Latency - TiKV side layer 1",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 88
},
"id": 90,
"links": [
{
"targetBlank": true,
"title": "TiKV scenario imbalance and avg latency of instances",
"url": "d/Al5q79L7z/tikv-scenario-imbalance-and-avg-latency-of-instances?${__all_variables}&${__url_time_range}"
}
],
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> TiKV gRPC duration: 0.4s\n\n在指标 kv_request Duration 有问题的时间段，我们基于 Performance Map 寻找对其有影响的指标。\n\n![\"perf\"](\"https://s2.loli.net/2022/03/01/iwZP84mOMf3C9Kh.png\")
\n\n（图中灰色指标 network_send_receive 暂时尚未被 Prometheus 追踪）\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_1 1)}}\n{{#if (eq kv_request_good_time_1 0)}}\n\nTiKV 侧无异常，建议先排除 TiDB 与 TiKV 网络连接问题，然后结合 TiDB Dashboard 的 Slow query 和 Top SQL 等功能进一步排查。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq total_grpc_good_time_1 0)}}\n\nTiKV 侧有异常，可继续对指标进行下探。\n\n额外的，如果想从读写场景的角度去分析 TiKV 侧各类 gRPC 请求的不平衡性以及平均延迟较高的 TiKV 实例，可以访问当前 Panel 左上角的 Panel link 前往查看。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_2 1)}}\n{{#if (eq kv_request_good_time_2 0)}}\n\nTiKV 侧无异常，建议先排除 TiDB 与 TiKV 网络连接问题，然后结合 TiDB Dashboard 的 Slow query 和 Top SQL 等功能进一步排查。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq total_grpc_good_time_2 0)}}\n\nTiKV 侧有异常，可继续对指标进行下探。\n\n额外的，如果想从读写场景的角度去分析 TiKV 侧各类 gRPC 请求的不平衡性以及平均延迟较高的 TiKV 实例，可以访问当前 Panel 左上角的 Panel link 前往查看。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_3 1)}}\n{{#if (eq kv_request_good_time_3 0)}}\n\nTiKV 侧无异常，建议先排除 TiDB 与 TiKV 网络连接问题，然后结合 TiDB Dashboard 的 Slow query 和 Top SQL 等功能进一步排查。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq total_grpc_good_time_3 0)}}\n\nTiKV 侧有异常，可继续对指标进行下探。\n\n额外的，如果想从读写场景的角度去分析 TiKV 侧各类 gRPC 请求的不平衡性以及平均延迟较高的 TiKV 实例，可以访问当前 Panel 左上角的 Panel link 前往查看。\n\n{{/if}}\n{{/if}}\n\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "avg_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good,conn_idle_good"
}
]
}
}
],
"title": "基于平均延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"avg"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 88
},
"id": 4,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tikv_grpc_msg_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\"}[1m])) / sum(rate(tikv_grpc_msg_duration_seconds_count{tidb_cluster=\"$tidb_cluster\"}[1m]))",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "avg",
"refId": "A"
}
],
"title": "TiKV gRPC duration",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 96
},
"id": 78,
"panels": [],
"title": "Latency - TiKV side layer 2",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 97
},
"id": 96,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Latch wait duration: 0.02s\n>\n> Async write duration: 0.19s\n>\n> Coprocessor wait duration: 0.05s\n>\n> Coprocessor handle duration: 0.25s\n>\n> Async snapshot duration: 0.2s\n\n在 TiKV 各类读写 gRPC 中继续下探时，async_snapshot 作为读请求操作的一部分，又会参与到 Coprocessor 和 写操作过程中，而 async_snapshot 本身无法区分用于哪类操作，所以在这一版块中，仅对以下指标进行单独的异常检测。\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_1 0)}}\n\n{{#if (eq copr_wait_good_time_1 0)}}- Coprocessor wait duration{{/if}}\n{{#if (eq copr_handle_good_time_1 0)}}- Coprocessor handle duration{{/if}}\n{{#if (eq latch_wait_good_time_1 0)}}- Latch wait duration{{/if}}\n{{#if (eq async_write_good_time_1 0)}}- Async write duration{{/if}}\n{{#if (eq async_snapshot_good_time_1 0)}}- Async snapshot duration{{/if}}\n{{#if (eq rocksdb_read_good_time_1 0)}}- RocksDB read duration{{/if}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_2 0)}}\n\n{{#if (eq copr_wait_good_time_2 0)}}- coprocessor wait duration{{/if}}\n{{#if (eq copr_handle_good_time_2 0)}}- coprocessor handle duration{{/if}}\n{{#if (eq latch_wait_good_time_2 0)}}- latch wait duration{{/if}}\n{{#if (eq async_write_good_time_2 0)}}- async write duration{{/if}}\n{{#if (eq async_snapshot_good_time_2 0)}}- async snapshot duration{{/if}}\n{{#if (eq rocksdb_read_good_time_2 0)}}- rocksdb read duration{{/if}}\n\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_3 0)}}\n\n{{#if (eq copr_wait_good_time_3 0)}}- coprocessor wait duration{{/if}}\n{{#if (eq copr_handle_good_time_3 0)}}- coprocessor handle duration{{/if}}\n{{#if (eq latch_wait_good_time_3 0)}}- latch wait duration{{/if}}\n{{#if (eq async_write_good_time_3 0)}}- async write duration{{/if}}\n{{#if (eq async_snapshot_good_time_3 0)}}- async snapshot duration{{/if}}\n{{#if (eq rocksdb_read_good_time_3 0)}}- rocksdb read duration{{/if}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n{{/if}}\n\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/diagnosis/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "avg_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good,conn_idle_good"
}
]
}
}
],
"title": "基于平均延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"avg"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 105
},
"id": 48,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tikv_scheduler_latch_wait_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\"}[1m])) / sum(rate(tikv_scheduler_latch_wait_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_scheduler_latch_wait_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\"}[1m])) by (le,instance))",
"hide": true,
"interval": "",
"legendFormat": "",
"refId": "B"
}
],
"title": "Latch wait duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 105
},
"id": 112,
"links": [
{
"targetBlank": true,
"title": "Sub - Async Write Duration",
"url": "/d/i0BOn9L7k/sub-async-write-duration?${__all_variables}&${__url_time_range}"
}
],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tikv_storage_engine_async_request_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"write\"}[1m])) / sum(rate(tikv_storage_engine_async_request_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"write\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"write\"}[1m])) by (le,instance))",
"hide": true,
"interval": "",
"legendFormat": "",
"refId": "B"
}
],
"title": "Async write duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 113
},
"id": 54,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tikv_coprocessor_request_wait_seconds_sum{tidb_cluster=\"$tidb_cluster\"}[1m])) / sum(rate(tikv_coprocessor_request_wait_seconds_count{tidb_cluster=\"$tidb_cluster\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Coprocessor wait duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 113
},
"id": 58,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tikv_coprocessor_request_handle_seconds_sum{tidb_cluster=\"$tidb_cluster\"}[1m])) / sum(rate(tikv_coprocessor_request_handle_seconds_count{tidb_cluster=\"$tidb_cluster\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Coprocessor handle duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 121
},
"id": 52,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "sum(rate(tikv_storage_engine_async_request_duration_seconds_sum{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"snapshot\"}[1m])) / sum(rate(tikv_storage_engine_async_request_duration_seconds_count{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"snapshot\"}[1m]))",
"interval": "",
"legendFormat": "avg",
"refId": "A"
}
],
"title": "Async snapshot duration (ATTENTION: write and coprocessor may use it too!)",
"type": "timeseries"
}
],
"refresh": false,
"schemaVersion": 34,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"datasource": "Prometheus-fake",
"definition": "",
"hide": 0,
"includeAll": false,
"label": "tidb_cluster",
"multi": false,
"name": "tidb_cluster",
"options": [],
"query": {
"query": "label_values({job=\"pd\"},tidb_cluster)",
"refId": "StandardVariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
},
{
"allValue": ".*",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"datasource": "Prometheus",
"definition": "",
"hide": 0,
"includeAll": true,
"label": "instance",
"multi": true,
"name": "instance",
"options": [],
"query": {
"query": "label_values(pd_cluster_status{tidb_cluster=\"$tidb_cluster\"}, instance)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
},
{
"hide": 2,
"name": "org",
"query": "",
"skipUrlSync": false,
"type": "constant"
}
]
},
"time": {
"from": "now",
"to": "now-6h"
},
"timepicker": {},
"timezone": "",
"title": "Average Latency Analysis",
"uid": "nPzC1XYnz",
"version": 14,
"weekStart": ""
}