{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
},
{
"datasource": "Infinity",
"enable": true,
"iconColor": "blue",
"name": "anomaly",
"target": {
"columns": [],
"filters": [],
"format": "dataframe",
"global_query_id": "",
"limit": 100,
"matchAny": false,
"refId": "",
"root_selector": "",
"source": "url",
"tags": [],
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/annotations",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "measurement",
"value": "p99_latency_analysis_anno"
}
]
}
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": 9,
"iteration": 1647954263497,
"links": [],
"liveNow": false,
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 68,
"panels": [],
"title": "Summary based on P99 latency analysis",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 11,
"w": 24,
"x": 0,
"y": 1
},
"id": 28,
"links": [],
"options": {
"content": "> 限于 Grafana dynamic text 插件不能支持 nested 结构，目前展示的问题时间段数量以及异常实例数量均为硬编码，问题时间段上限为 3 个，每个时间段每类 gRPC 异常实例数量上限为 10 个。\n>\n> 报告组织结构外层为 P99 总延迟的异常时间段，每一个异常时间段内，具有相同的结构，即“基本结论”、“TiDB 侧”、“TiKV 侧”及“TiKV 读写场景、实例不平衡性及异常延迟”。\n\n本报告对 Query Latency 进行拆解，探索在 Query Latency 发现异常时，是哪些组件导致的问题。\n\n整体分析思路基于 TiDB 的 Performance Map，见下图。\n\n ![\"perf\"](\"https://download.pingcap.com/images/docs-cn/dashboard/dashboard-diagnostics-time-relation.png\")
",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "{\n \"job_start_time\": 1640762139654,\n \"job_end_time\": 1640769467920\n}",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "inline",
"type": "json",
"url": "",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "p99_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good"
}
]
}
}
],
"title": "基于 P99 延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 12
},
"id": 66,
"panels": [],
"title": "Total latency - from client request to response",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 13
},
"id": 84,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Total latency: {{ tidb_p99_rt:total }} s\n\n{{#if (eq total_latency_good 1)}}\n\n集群 P99 延迟没有明显异常，其它类型问题，建议结合其它监控指标或 TiDB Dashboard 相关工具进行人工排查。\n{{/if}}\n\n{{#if (eq total_latency_good 0)}}\n\n从指标 Total latency 看，以下时间段存在异常（图表中多于 3 个异常时间段时，限于插件原因暂时显示 3 个）。\n\n{{#if (eq time_1 1)}}\n\n集群从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内，系统延迟存在异常。\n\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n集群从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内，系统延迟存在异常。\n\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n集群从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内，系统延迟存在异常。\n\n{{/if}}\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "p99_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good"
}
]
}
}
],
"title": "基于 P99 延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"TiDB P99 duration"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 13
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"pluginVersion": "8.3.4",
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_server_handle_query_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\"}[1m])) by (le))",
"interval": "",
"legendFormat": "TiDB P99 duration",
"refId": "A"
}
],
"title": "Total latency",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 21
},
"id": 72,
"panels": [],
"title": "Latency - TiDB side layer 1",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 22
},
"id": 86,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Get token duration: {{ get_token_duration }} us\n>\n> Parse duration: {{ parse_duration:general }} s\n>\n> Compile duration: {{ compile_duration:general }} s\n>\n> Execute duration: {{ execute_duration:general }} s\n\n在指标 Total Latency 有问题的时间段，我们基于 Performance Map 寻找对其有影响的指标。\n\n![\"perf\"](\"https://s2.loli.net/2022/03/01/arqgj4KJEAXyv5f.png\")
\n\n在下拆的 4 个指标中，在 Total latency 异常的时间段附近，我们发现以下指标也存在异常。（如无异常则为空白）\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq get_token_latency_good_time_1 0)}}- Get token duration{{/if}}\n{{#if (eq parse_good_time_1 0)}}- Parse duration{{/if}}\n{{#if (eq compile_good_time_1 0)}}- Compile duration{{/if}}\n{{#if (eq execute_latency_good_time_1 0)}}- Execute duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq get_token_latency_good_time_2 0)}}- Get token duration{{/if}}\n{{#if (eq parse_good_time_2 0)}}- Parse duration{{/if}}\n{{#if (eq compile_good_time_2 0)}}- Compile duration{{/if}}\n{{#if (eq execute_latency_good_time_2 0)}}- Execute duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n## 基本结论\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq get_token_latency_good_time_3 0)}}- Get token duration{{/if}}\n{{#if (eq parse_good_time_3 0)}}- Parse duration{{/if}}\n{{#if (eq compile_good_time_3 0)}}- Compile duration{{/if}}\n{{#if (eq execute_latency_good_time_3 0)}}- Execute duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "p99_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good"
}
]
}
}
],
"title": "基于 P99 延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "µs"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 30
},
"id": 30,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_server_get_token_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 get token duration",
"refId": "A"
}
],
"title": "Get token duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 30
},
"id": 34,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_session_compile_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 compile duration",
"refId": "A"
}
],
"title": "Compile duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 38
},
"id": 32,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_session_parse_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 parse duration",
"refId": "A"
}
],
"title": "Parse duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 38
},
"id": 36,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_session_execute_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", sql_type=\"general\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 execute duration",
"refId": "A"
}
],
"title": "Execute duration",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 46
},
"id": 74,
"panels": [],
"title": "Execute latency - TiDB side layer 2 ",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 47
},
"id": 88,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Wait TSO duration: {{ wait_start_tso_duration }} s\n>\n> KV request duration: {{ kv_request_duration }} s\n>\n> KV backoff duration: {{ kv_backoff_duration }} s\n\n在指标 Execute Duration 有问题的时间段，我们基于 Performance Map 寻找对其有影响的指标。\n\n![\"perf\"](\"https://s2.loli.net/2022/03/01/w1VZ9RnHGdIxcfQ.png\")
\n\n（图中灰色指标 TiDB_execute 暂时尚未被 Prometheus 追踪）\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq wait_start_tso_good_time_1 0)}}- Wait_start_TSO duration{{/if}}\n{{#if (eq kv_backoff_good_time_1 0)}}- KV backoff duration{{/if}}\n{{#if (eq kv_request_good_time_1 0)}}- KV request duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq total_grpc_good_time_1 1)}}\n{{#if (eq kv_request_good_time_1 1)}}\n\nTiKV 侧以及 TiDB 请求 KV 的延迟均正常，建议优先排查执行计划，或 TiFlash(如有)，或结合其它工具（如 TiDB Dashboard）进行排查。\n{{/if}}\n{{/if}}\n\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq wait_start_tso_good_time_2 0)}}- Wait_start_TSO duration{{/if}}\n{{#if (eq kv_backoff_good_time_2 0)}}- KV backoff duration{{/if}}\n{{#if (eq kv_request_good_time_2 0)}}- KV request duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq total_grpc_good_time_2 1)}}\n{{#if (eq kv_request_good_time_2 1)}}\n\nTiKV 侧以及 TiDB 请求 KV 的延迟均正常，建议优先排查执行计划，或 TiFlash(如有)，或结合其它工具（如 TiDB Dashboard）进行排查。\n{{/if}}\n{{/if}}\n\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n## 基本结论\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq wait_start_tso_good_time_3 0)}}- Wait_start_TSO duration{{/if}}\n{{#if (eq kv_backoff_good_time_3 0)}}- KV backoff duration{{/if}}\n{{#if (eq kv_request_good_time_3 0)}}- KV request duration{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq total_grpc_good_time_3 1)}}\n{{#if (eq kv_request_good_time_3 1)}}\n\nTiKV 侧以及 TiDB 请求 KV 的延迟均正常，建议优先排查执行计划，或 TiFlash(如有)，或结合其它工具（如 TiDB Dashboard）进行排查。\n{{/if}}\n{{/if}}\n\n{{/if}}\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "p99_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good"
}
]
}
}
],
"title": "基于 P99 延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 55
},
"id": 40,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_pdclient_ts_future_wait_seconds_bucket{tidb_cluster=\"$tidb_cluster\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 wait start tso duration",
"refId": "A"
}
],
"title": "Wait TSO duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 55
},
"id": 56,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_tikvclient_backoff_seconds_bucket{tidb_cluster=\"$tidb_cluster\"}[5m])) by (le))",
"interval": "",
"legendFormat": "P99 kv backoff",
"refId": "A"
}
],
"title": "KV backoff duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 63
},
"id": 38,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_tikvclient_request_seconds_bucket{tidb_cluster=\"$tidb_cluster\", store!=\"0\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 kv request duration",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tidb_tikvclient_request_seconds_bucket{tidb_cluster=\"$tidb_cluster\", store!=\"0\"}[1m])) by (le,type))",
"hide": true,
"interval": "",
"legendFormat": "kv-request-duration-{{type}}",
"refId": "B"
}
],
"title": "KV request duration",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 71
},
"id": 70,
"panels": [],
"title": "Latency - TiKV side layer 1",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 72
},
"id": 90,
"links": [
{
"targetBlank": true,
"title": "TiKV scenario imbalance and latency of instances",
"url": "/d/dVkSaIY7k/tikv-scenario-imbalance-and-latency-of-instances?${__all_variables}"
}
],
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> TiKV gRPC duration: {{ tikv_p99_rt:total }} s\n\n在指标 kv_request Duration 有问题的时间段，我们基于 Performance Map 寻找对其有影响的指标。\n\n![\"perf\"](\"https://s2.loli.net/2022/03/01/iwZP84mOMf3C9Kh.png\")
\n\n（图中灰色指标 network_send_receive 暂时尚未被 Prometheus 追踪）\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_1 1)}}\n{{#if (eq kv_request_good_time_1 0)}}\n\nTiKV 侧无异常，建议先排除 TiDB 与 TiKV 网络连接问题，然后结合 TiDB Dashboard 的 Slow query 和 Top SQL 等功能进一步排查。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq total_grpc_good_time_1 0)}}\n\nTiKV 侧有异常，可继续对指标进行下探。\n\n额外的，如果想从读写场景的角度去分析 TiKV 侧各类 gRPC 请求的不平衡性以及 P99 延迟较高的 TiKV 实例，可以访问当前 Panel 左上角的 Panel link 前往查看。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_2 1)}}\n{{#if (eq kv_request_good_time_2 0)}}\n\nTiKV 侧无异常，建议先排除 TiDB 与 TiKV 网络连接问题，然后结合 TiDB Dashboard 的 Slow query 和 Top SQL 等功能进一步排查。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq total_grpc_good_time_2 0)}}\n\nTiKV 侧有异常，可继续对指标进行下探。\n\n额外的，如果想从读写场景的角度去分析 TiKV 侧各类 gRPC 请求的不平衡性以及 P99 延迟较高的 TiKV 实例，可以访问当前 Panel 左上角的 Panel link 前往查看。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_3 1)}}\n{{#if (eq kv_request_good_time_3 0)}}\n\nTiKV 侧无异常，建议先排除 TiDB 与 TiKV 网络连接问题，然后结合 TiDB Dashboard 的 Slow query 和 Top SQL 等功能进一步排查。\n\n{{/if}}\n{{/if}}\n\n{{#if (eq total_grpc_good_time_3 0)}}\n\nTiKV 侧有异常，可继续对指标进行下探。\n\n额外的，如果想从读写场景的角度去分析 TiKV 侧各类 gRPC 请求的不平衡性以及 P99 延迟较高的 TiKV 实例，可以访问当前 Panel 左上角的 Panel link 前往查看。\n\n{{/if}}\n{{/if}}\n\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "p99_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good"
}
]
}
}
],
"title": "基于 P99 延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"TiKV P99 gRPC duration"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 72
},
"id": 4,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_grpc_msg_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\"}[1m]))by(le))",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "TiKV P99 gRPC duration",
"refId": "A"
}
],
"title": "TiKV gRPC duration",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 80
},
"id": 78,
"panels": [],
"title": "Latency - TiKV side layer 2",
"type": "row"
},
{
"datasource": "Infinity",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 81
},
"id": 96,
"options": {
"content": "> Baseline (注：算法并非直接使用下列值作为阈值来检测异常)\n>\n> Latch wait duration: {{ latch_wait_duration }} s\n>\n> Async write duration: {{ async_write_duration }} s\n>\n> Coprocessor wait duration: {{ copr_wait_duration }} s\n>\n> Coprocessor handle duration: {{ copr_handle_duration }} s\n>\n> Async snapshot duration: {{ async_snapshot_duration }} s\n\n在 TiKV 各类读写 gRPC 中继续下探时，async_snapshot 作为读请求操作的一部分，又会参与到 Coprocessor 和 写操作过程中，而 async_snapshot 本身无法区分用于哪类操作，所以在这一版块中，仅对以下指标进行单独的异常检测。\n\n{{#if (eq total_latency_good 0)}}\n{{#if (eq time_1 1)}}\n\n从 {{time_1_start_unix_rfc3339}} 到 {{time_1_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_1 0)}}\n\n{{#if (eq copr_wait_good_time_1 0)}}- Coprocessor wait duration{{/if}}\n{{#if (eq copr_handle_good_time_1 0)}}- Coprocessor handle duration{{/if}}\n{{#if (eq latch_wait_good_time_1 0)}}- Latch wait duration{{/if}}\n{{#if (eq async_write_good_time_1 0)}}- Async write duration{{/if}}\n{{#if (eq async_snapshot_good_time_1 0)}}- Async snapshot duration{{/if}}\n{{#if (eq rocksdb_read_good_time_1 0)}}- RocksDB read duration{{/if}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{#if (eq time_2 1)}}\n\n从 {{time_2_start_unix_rfc3339}} 到 {{time_2_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_2 0)}}\n\n{{#if (eq copr_wait_good_time_2 0)}}- coprocessor wait duration{{/if}}\n{{#if (eq copr_handle_good_time_2 0)}}- coprocessor handle duration{{/if}}\n{{#if (eq latch_wait_good_time_2 0)}}- latch wait duration{{/if}}\n{{#if (eq async_write_good_time_2 0)}}- async write duration{{/if}}\n{{#if (eq async_snapshot_good_time_2 0)}}- async snapshot duration{{/if}}\n{{#if (eq rocksdb_read_good_time_2 0)}}- rocksdb read duration{{/if}}\n\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n\n{{/if}}\n{{/if}}\n\n{{#if (eq time_3 1)}}\n\n从 {{time_3_start_unix_rfc3339}} 到 {{time_3_end_unix_rfc3339}} 时间范围内：\n\n{{#if (eq total_grpc_good_time_3 0)}}\n\n{{#if (eq copr_wait_good_time_3 0)}}- coprocessor wait duration{{/if}}\n{{#if (eq copr_handle_good_time_3 0)}}- coprocessor handle duration{{/if}}\n{{#if (eq latch_wait_good_time_3 0)}}- latch wait duration{{/if}}\n{{#if (eq async_write_good_time_3 0)}}- async write duration{{/if}}\n{{#if (eq async_snapshot_good_time_3 0)}}- async snapshot duration{{/if}}\n{{#if (eq rocksdb_read_good_time_3 0)}}- rocksdb read duration{{/if}}\n\n{{/if}}\n\n(以上为异常项，如上述无异常则空白)\n{{/if}}\n\n{{/if}}",
"defaultContent": "The query didn't return any results.",
"everyRow": true
},
"targets": [
{
"columns": [],
"data": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"datasource": "Infinity",
"filters": [],
"format": "dataframe",
"global_query_id": "",
"refId": "A",
"root_selector": "",
"source": "url",
"type": "json",
"url": "__CLINIC_ADDR__/clinic/api/v1/dynamic_text_value",
"url_options": {
"data": "",
"method": "GET",
"params": [
{
"key": "measurement",
"value": "p99_latency_analysis_dt"
},
{
"key": "start_ts",
"value": "${__from:date:seconds}"
},
{
"key": "end_ts",
"value": "${__to:date:seconds}"
},
{
"key": "tidb_cluster",
"value": "${tidb_cluster}"
},
{
"key": "default_1",
"value": "total_latency_good"
}
]
}
}
],
"title": "基于 P99 延迟性能的自动化诊断报告",
"type": "marcusolsson-dynamictext-panel"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"P99 latch wait duration"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 89
},
"id": 48,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_scheduler_latch_wait_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 latch wait duration",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_scheduler_latch_wait_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\"}[1m])) by (le,instance))",
"hide": true,
"interval": "",
"legendFormat": "",
"refId": "B"
}
],
"title": "Latch wait duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 89
},
"id": 50,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"write\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 async write duration",
"refId": "A"
},
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"write\"}[1m])) by (le,instance))",
"hide": true,
"interval": "",
"legendFormat": "",
"refId": "B"
}
],
"title": "Async write duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 97
},
"id": 54,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_coprocessor_request_wait_seconds_bucket{tidb_cluster=\"$tidb_cluster\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 coprocessor wait duration",
"refId": "A"
}
],
"title": "Coprocessor wait duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 97
},
"id": 58,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_coprocessor_request_handle_seconds_bucket{tidb_cluster=\"$tidb_cluster\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 coprocessor handle duration",
"refId": "A"
}
],
"title": "Coprocessor handle duration",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 105
},
"id": 52,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single"
}
},
"targets": [
{
"datasource": "Prometheus",
"exemplar": false,
"expr": "histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", type=\"snapshot\"}[1m])) by (le))",
"interval": "",
"legendFormat": "P99 async snapshot duration",
"refId": "A"
}
],
"title": "Async snapshot duration (ATTENTION: write and coprocessor will use it too!)",
"type": "timeseries"
}
],
"refresh": false,
"schemaVersion": 34,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"datasource": "Prometheus-fake",
"definition": "",
"hide": 0,
"includeAll": false,
"label": "tidb_cluster",
"multi": false,
"name": "tidb_cluster",
"options": [],
"query": {
"query": "label_values({job=\"pd\"}, tidb_cluster)",
"refId": "StandardVariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
},
{
"allValue": ".*",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"datasource": "Prometheus",
"definition": "",
"hide": 0,
"includeAll": true,
"label": "instance",
"multi": true,
"name": "instance",
"options": [],
"query": {
"query": "label_values(pd_cluster_status{tidb_cluster=\"$tidb_cluster\"}, instance)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Latency Analysis",
"uid": "JWQcYVY7k",
"version": 13,
"weekStart": ""
}