# [Known Issues] statements_summary 内存表读写锁冲突导致 TiDB panic 异常重启

## Key Words

statements_summary, TiDB panic, TiDB 异常重启

## Description

集群中多个 TIDB 节点在不同时间点均存在重启现象，TiDB 日志未发现异常报错，系统也没有触发 OOM，搜索 tidb_stderr.log 中的 panic，发现如下报错信息，从中可以看出问题跟 statement_summary 有关：

```golang
fatal error: concurrent map read and map write

goroutine 474139308 [running]:
runtime.throw(0x31c743b, 0x21)
    /usr/lib/golang/src/runtime/panic.go:1116 +0x72 fp=0xc0057d12d8 sp=0xc0057d12a8 pc=0x10d0e12
runtime.mapaccess2_faststr(0x2e08b60, 0xc00d54ced0, 0xc01d112760, 0x8, 0x1, 0xc000780000)
    /usr/lib/golang/src/runtime/map_faststr.go:116 +0x4a5 fp=0xc0057d1348 sp=0xc0057d12d8 pc=0x10ad6e5
github.com/pingcap/tidb/util/stmtsummary.(*stmtSummaryByDigest).toHistoryDatum(0xc016d10300, 0xf0, 0xc00ff8e9b0, 0xc048e73400, 0x0, 0x0, 0x1)
    /app/dev/project/jenkins-slave/workspace/tidb/util/stmtsummary/statement_summary.go:583 +0x21b fp=0xc0057d13e0 sp=0xc0057d1348 pc=0x1e40ddb
github.com/pingcap/tidb/util/stmtsummary.(*stmtSummaryByDigestMap).ToHistoryDatum(0xc00039d9e0, 0xc00ff8e9b0, 0x0, 0x1, 0x0, 0x0)
    /app/dev/project/jenkins-slave/workspace/tidb/util/stmtsummary/statement_summary.go:340 +0x185 fp=0xc0057d1488 sp=0xc0057d13e0 pc=0x1e3fc05
github.com/pingcap/tidb/executor.(*memtableRetriever).setDataForStatementsSummary(0xc00ff8eb40, 0x358c320, 0xc00c5dca20, 0x31c93dc, 0x22, 0x0, 0x0)
    /app/dev/project/jenkins-slave/workspace/tidb/executor/infoschema_reader.go:1636 +0x245 fp=0xc0057d14f8 sp=0xc0057d1488 pc=0x29a32a5
github.com/pingcap/tidb/executor.(*memtableRetriever).retrieve(0xc00ff8eb40, 0x3553940, 0xc005d8fec0, 0x358c320, 0xc00c5dca20, 0x0, 0x0, 0xc0057d15f8, 0x113b8d8, 0xc005d8fdd0)
    /app/dev/project/jenkins-slave/workspace/tidb/executor/infoschema_reader.go:133 +0xcf5 fp=0xc0057d1588 sp=0xc0057d14f8 pc=0x2993a15
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0xc025c72480, 0x3553940, 0xc005d8fec0, 0xc00ff8eb90, 0x49, 0x4d091f8)
    /app/dev/project/jenkins-slave/workspace/tidb/executor/memtable_reader.go:112 +0x49b fp=0xc0057d1680 sp=0xc0057d1588 pc=0x29e311b
github.com/pingcap/tidb/executor.Next(0x3553940, 0xc005d8fec0, 0x355b100, 0xc025c72480, 0xc00ff8eb90, 0x0, 0x0)
    /app/dev/project/jenkins-slave/workspace/tidb/executor/executor.go:262 +0x2e4 fp=0xc0057d17a0 sp=0xc0057d1680 pc=0x29670a4
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0xc001ff4f70, 0x3553940, 0xc005d8fec0, 0xc00ff8ec30, 0xc01c8ba500, 0xc024890480)
    /app/dev/project/jenkins-slave/workspace/tidb/executor/executor.go:1235 +0x22c fp=0xc0057d1870 sp=0xc0057d17a0 pc=0x296cfec
github.com/pingcap/tidb/executor.Next(0x3553940, 0xc005d8fec0, 0x355b480, 0xc001ff4f70, 0xc00ff8ec30, 0x0, 0x0)
    /app/dev/project/jenkins-slave/workspace/tidb/executor/executor.go:262 +0x2e4 fp=0xc0057d1990 sp=0xc0057d1870 pc=0x29670a4
github.com/pingcap/tidb/executor.(*CoprocessorDAGHandler).HandleRequest(0xc0057d1a90, 0x3553940, 0xc005d8fec0, 0xc00e931f80, 0xc00c5dca20)
    /app/dev/project/jenkins-slave/workspace/tidb/executor/coprocessor.go:66 +0x2e7 fp=0xc0057d1a58 sp=0xc0057d1990 pc=0x2954547
github.com/pingcap/tidb/server.(*rpcServer).handleCopRequest(0xc0033ae2d0, 0x3553940, 0xc005d8fec0, 0xc00e931f80, 0x0)
    /app/dev/project/jenkins-slave/workspace/tidb/server/rpc_server.go:184 +0x152 fp=0xc0057d1ac8 sp=0xc0057d1a58 pc=0x2bef2f2
github.com/pingcap/tidb/server.(*rpcServer).Coprocessor(0xc0033ae2d0, 0x3553940, 0xc005d8fec0, 0xc00e931f80, 0xc01091ec00, 0x0, 0x0)
    /app/dev/project/jenkins-slave/workspace/tidb/server/rpc_server.go:83 +0xa5 fp=0xc0057d1b18 sp=0xc0057d1ac8 pc=0x2bee0a5
github.com/pingcap/kvproto/pkg/tikvpb._Tikv_Coprocessor_Handler(0x314c560, 0xc0033ae2d0, 0x3553940, 0xc005d8fec0, 0xc00f0f7860, 0x0, 0x3553940, 0xc005d8fec0, 0xc005652480, 0x175)
    /root/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20201126113434-70db5fb4b0dc/pkg/tikvpb/tikvpb.pb.go:3360 +0x214 fp=0xc0057d1b88 sp=0xc0057d1b18 pc=0x1f8fa14
google.golang.org/grpc.(*Server).processUnaryRPC(0xc003366480, 0x3576dc0, 0xc0087f5c80, 0xc006c98100, 0xc0033ae3f0, 0x47cb0d0, 0x0, 0x0, 0x0)
    /root/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024 +0x522 fp=0xc0057d1e38 sp=0xc0057d1b88 pc=0x1630242
google.golang.org/grpc.(*Server).handleStream(0xc003366480, 0x3576dc0, 0xc0087f5c80, 0xc006c98100, 0x0)
    /root/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313 +0xd34 fp=0xc0057d1f68 sp=0xc0057d1e38 pc=0x16342b4
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0012f6f00, 0xc003366480, 0x3576dc0, 0xc0087f5c80, 0xc006c98100)
    /root/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722 +0xa5 fp=0xc0057d1fb8 sp=0xc0057d1f68 pc=0x16423c5
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0057d1fc0 sp=0xc0057d1fb8 pc=0x110a9a1
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /root/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:720 +0xa5
```

## Root Cause

业务频繁访问内存表 information_schema.cluster_statements_summary_history，由于代码缺陷读写时并发访问 map 但是读的时候没有加锁，触发并发读写冲突，导致 TiDB 出现 panic。

相关 issue [https://github.com/pingcap/tidb/issues/35340](https://github.com/pingcap/tidb/issues/35340)

## Diagnostic Steps

1. 业务访问 statements_summary 内存表，并引起 TiDB 异常重启

2. tidb_stderr.log 打印 panic 日志提示 fatal error: concurrent map read and map write，对比错误栈有 statement_summary 等关键字

## Resolution

修复 PR merge 后，升级到对应的 LTS 版本，包括目前已发布或即将发布的版本：v5.0.7、v5.1.5、v5.2.5、v5.3.2、v5.4.2、v6.1.1。

## Workaround

对访问 statements_summary 表的用户增加 super 权限或单独创建一个拥有 super 权限的用户，可以绕过并发读写冲突的问题。

## Affected Versions

v4.0.x、v5.0.0-v5.0.6、v5.1.0-v5.1.4、v5.2.0-v5.2.4、v5.3.0-v5.3.1、v5.4.0-v5.4.1、v6.0.0、v6.1.0