# [Known Issues] 在开启 binlog 的情况下，TiDB 执行 Delete 大事务跑批失败

## Key Words

binlog, tidb 重启, "write binlog failed", "grpc: message too large", 大事务, 跑批, 失败

## Description

在开启 binlog 的情况下，TiDB 集群在执行批量 DML 操作过程中，TiDB Server 实例自动重启。DML 操作涉及的事务较大，主要是 Update、Delete 操作等，在 tidb.log 中找到如下日志

```golang
[2021/09/01 21:28:48.963 +08:00] [INFO] [2pc.go:405] ["[BIG_TXN]"] [con=1570052] ["table ID"=4476] [size=372527123] [keys=7607567] [puts=0] [dels=7607567] [locks=0] [checks=0] [txnStartTS=427426537222701063]
[2021/09/01 21:29:00.280 +08:00] [INFO] [2pc.go:886] ["send TxnHeartBeat"] [startTS=427426537222701063] [newTTL=117450]  
[2021/09/01 21:29:01.545 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=9.23.3.15:8249] ["binlog type"=Prewrite] ["start ts"=427426537222701063] ["commit ts"=0] [length=8687044496] [error="rpc error: code = ResourceExhausted desc = grpc: message too large (8687044512 bytes)"]   
[2021/09/01 21:29:01.546 +08:00] [ERROR] [binloginfo.go:253] ["write binlog failed"] [binlog_type=Prewrite] [binlog_start_ts=427426537222701063] [binlog_commit_ts=0] [error="rpc error: code = ResourceExhausted desc = grpc: message too large (8687044512 bytes)"]
[2021/09/01 21:29:10.139 +08:00] [INFO] [2pc.go:886] ["send TxnHeartBeat"] [startTS=427426537222701063] [newTTL=127300]
[2021/09/01 21:29:20.140 +08:00] [INFO] [2pc.go:886] ["send TxnHeartBeat"] [startTS=427426537222701063] [newTTL=137300]
[2021/09/01 21:29:29.012 +08:00] [WARN] [session.go:472] ["can not retry txn"] [conn=1570052] [label=general] [error="[global:3]critical error rpc error: code = ResourceExhausted desc = grpc: message too large (8687044512 bytes)"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2021/09/01 21:29:29.012 +08:00] [WARN][session.go:487] ["commit failed"] [conn=1570052] ["finished txn"="Txn{state=invalid}"] [error="[global:3]critical error rpc error: code = ResourceExhausted desc = grpc: message too large (8687044512 bytes)"]
[2021/09/01 21:29:29.017 +08:00] [WARN] [session.go:1066] ["run statement failed"] [conn=1570052] [schemaVersion=8760] [error="[global:3]critical error rpc error: code = ResourceExhausted desc = grpc: message too large (8687044512 bytes)"] [session="{\n "currDBName": "ncisp",\n "id": 1570052,\n "status": 2,\n "strictMode": false,\n "user": {\n "Username": "root",\n "Hostname": "9.31.255.185",\n "CurrentUser": false,\n "AuthUsername": "root",\n "AuthHostname": "%"\n }\n}"]
[2021/09/01 21:29:29.017 +08:00] [FATAL] [conn.go:781] ["critical error, stop the server"] [conn=1570052] [error="[global:3]critical error rpc error: code = ResourceExhausted desc = grpc: message too large (8687044512 bytes)"[stack="github.com/pingcap/tidb/server(\*clientConn).Run\n\t/home/jenkins/agent/workspace/tidb_v4.0.8/go/src/github.com/pingcap/tidb/server/conn.go:781\ngithub.com/pingcap/tidb/server.   (\*Server).onConn\n\t/home/jenkins/agent/workspace/tidb_v4.0.8/go/src/github.com/pingcap/tidb/server/server.go:421"]  
```

## Root Cause

`delete` 事务存储在 TiDB Server 内存中 key-value 中的 value 为 null，TiDB 事务大小限制是按照 buffer 计算，所以没有触发事务大小的限制。在写 TiDB-binlog 时 key-value 中的 value 则为全行所有列的值，因此会存在非常高的放大问题。

TiDB Server 默认配置 go gRPC message size 的限制是 max uint64 (4294967295)，报错 `grpc: message too large (8687044512 bytes)` size 超过限制的 2 倍，导致 TiDB Server 写入 TiDB-binlog 时 gRPC 消息过大，因而触发 TiDB Server 重启保护。

## Diagnostic Steps

1. 检查 TiDB Server log 是否存在 `Error` 和 `WARN` 报错，报错标识 `grpc: message too large`

2. 确认 TiDB 集群拓扑是否包含 TiDB-binlog 配置，进而确认是否是 gRPC message tool large 导致

3. 了解执行失败的 DML 事务大小和业务逻辑，是否存在大事务，影响到 TiDB-binlog 的 message 传输过大。

## Resolution

分批执行，控制批次事务大小，如果是全量 delete 也可以考虑使用 truncate 类似操作进行替换。

## Workaround

N/A

## Affected Versions

All