# [KB]关于由于超大-Region-导致-BR-备份失败 ## Key Words BR,timeout,tikv request error,tikvPRC ## Issue BR 备份失败，日志中有报错显示超时，关键字 "tikvRPC backoffer.maxSleep 40000ms is exceeded" , "tikv request error: wait recvLoop: context deadline exceeded" , "epoch_not_match" ## Root Cause BR 在 scan region 设置了默认 60s 超时时间，由于 TiKV 中存在超大 region，导致默认时间无法扫描完成。 ```sql mysql> select * from tikv_region_status where approximate_size > 100\G; *************************** 1. row *************************** REGION_ID: 299695 START_KEY: 7480000000000007FF265F7280000024A0FFBDE1B80000000000FA END_KEY: 7480000000000007FF2700000000000000F8 TABLE_ID: 1782 DB_NAME: scribe TABLE_NAME: document IS_INDEX: 0 INDEX_ID: NULL INDEX_NAME: NULL EPOCH_CONF_VER: 325 EPOCH_VERSION: 2430 WRITTEN_BYTES: 0 READ_BYTES: 0 APPROXIMATE_SIZE: 39399 APPROXIMATE_KEYS: 9768284 REPLICATIONSTATUS_STATE: NULL REPLICATIONSTATUS_STATEID: NULL 1 row in set (3.03 sec) ``` TiKV 中存在超大 region 的原因是，Region 在没有新写入时有可能一直不做 split check Github Issue：[https://github.com/tikv/tikv/issues/10111](https://github.com/tikv/tikv/issues/10111)s ## Diagnostic Steps BR 日志中有报错，关键字 "tikvRPC backoffer.maxSleep 40000ms is exceeded" , "tikv request error: wait recvLoop: context deadline exceeded" , "epoch_not_match" ```golang I0829 21:21:00.280550 12 backup.go:93] [2021/08/29 21:21:00.278 +00:00] [WARN] [backoff.go:331] ["tikvRPC backoffer.maxSleep 40000ms is exceeded, errors:\nsend tikv request error: wait recvLoop: context deadline exceeded, ctx: region ID: 299695, meta: id:299695 start_key:\"t\\200\\000\\000\\000\\000\\000\\007&_r\\200\\000\\000$\\240\\275\\341\\270\" end_key:\"t\\200\\000\\000\\000\\000\\000\\007'\" region_epoch: peers: peers: peers: , peer: id:397404 store_id:12 , addr: scribe-hyd-tikv-10.scribe-hyd-tikv-peer.rigel-clusters-prod.svc:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv, try next peer later at 2021-08-29T21:19:58.70633192Z\nregion_id:299695, region_ver:2430, store_type:tikv, peer_addr:scribe-hyd-tikv-10.scribe-hyd-tikv-peer.rigel-clusters-prod.svc:20160, error:epoch_not_match:<> at 2021-08-29T21:19:59.776704726Z\nsend tikv request error: wait recvLoop: context deadline exceeded, ctx: region ID: 299695, meta: id:299695 start_key:\"t\\200\\000\\000\\000\\000\\000\\007&_r\\200\\000\\000$\\240\\275\\341\\270\" end_key:\"t\\200\\000\\000\\000\\000\\000\\007'\" region_epoch: peers: peers: peers: , peer: id:397404 store_id:12 , addr: scribe-hyd-tikv-10.scribe-hyd-tikv-peer.rigel-clusters-prod.svc:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv, try next peer later at 2021-08-29T21:21:00.278129169Z"] ``` 在对应 TiKV 日志中有该 region 的慢日志， 关键字 `slow-query`, `"`perf_stats.block_read_byte` ```golang [2021/08/31 16:23:52.282 +05:30] [INFO] [tracker.rs:248] [slow-query] [perf_stats.internal_delete_skipped_count=0] [perf_stats.internal_key_skipped_count=12482559] [perf_stats.block_read_byte=4377772672] [perf_stats.block_read_count=403939] [perf_stats.block_cache_hit_count=215] [scan.range.first="Some(start: 7480000000000007265F7280000024A0BDE1B8 end: 7480000000000007265F72FFFFFFFFFFFFFFFF00)"] [scan.ranges=1] [scan.total=6241281] [scan.processed=6241280] [scan.is_desc=false] [tag=select] [table_id=1830] [txn_start_ts=427400901371887617] [total_suspend_time=12ms] [total_process_time=59.986s] [handler_build_time=0ns] [wait_time.snapshot=4ms] [wait_time.schedule=0ns] [wait_time=4ms] [total_lifetime=60.002s] [remote_host=ipv4:10.49.197.28:48210] [region_id=299695] ``` ## Resolution 升级到 v4.0.14 或者 v5.0.2 及以上版本修复 ## Workaround 使用 [PD Contorl](https://docs.pingcap.com/zh/tidb/stable/pd-control##operator-check--show--add--remove) 对大 region 执行手动分裂 ```shell [operator add split-region --policy=approximate --将 Region 对半拆分成两个 Region，基于粗略估计值 operator add split-region --policy=scan --将 Region 对半拆分成两个 Region，基于精确扫描值 ```