# [Know Issue] TiFlash 异常重启，且重启过程中报 Reading with entries meet checksum not match 或者 invalid flag\u0000 in lock value 错误 ## Key Words Reading with entries meet checksum not match,DB::Exception: Too large string size,invalid flag \u0000 in lock value,RegionPersister::doPersist ## Description 集群中存在大事务写入等情况导致超过 256 MB 的 Region 产生（tiflash.log 日志中出现类似 `Created a new BlobStat [blob_id=339] [capacity=330405764]`， Tiflash 异常重启并且堆栈打印类似如下内容： ```log [2023/04/19 09:04:45.250 +08:00] [ERROR] [BaseDaemon.cpp:376] [########################################] [source=BaseDaemon] [thread_id=6009] [2023/04/19 09:04:45.251 +08:00] [ERROR] [BaseDaemon.cpp:377] ["(from thread 101) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=6009] [2023/04/19 09:04:45.251 +08:00] [ERROR] [BaseDaemon.cpp:407] ["Address: 0x7f6d9a751000"] [source=BaseDaemon] [thread_id=6009] [2023/04/19 09:04:45.251 +08:00] [ERROR] [BaseDaemon.cpp:413] ["Access: read."] [source=BaseDaemon] [thread_id=6009] [2023/04/19 09:04:45.251 +08:00] [ERROR] [BaseDaemon.cpp:419] ["Attempted access has violated the permissions assigned to the memory area."] [source=BaseDaemon] [thread_id=6009] [2023/04/19 09:04:45.292 +08:00] [ERROR] [BaseDaemon.cpp:569] [" 0x753aae1 faultSignalHandler(int, siginfo_t*, void*) [tiflash+122923745] libs/libdaemon/src/BaseDaemon.cpp:220 0x7f707f10ed90 [libc.so.6+347536] 0x8653b9c crc64::_detail::update_simd(unsigned long, void const*, unsigned long) [tiflash+140852124] libs/libcommon/src/crc64_sse2_asimd.cpp:63 0x865378d crc64::Digest::Digest(crc64::Mode)::$_2::__invoke(unsigned long, void const*, unsigned long) [tiflash+140851085] libs/libcommon/src/crc64.cpp:52 0x1b0ce43 DB::PS::V3::BlobStore::write(DB::WriteBatch&&, std::__1::shared_ptr const&) [tiflash+28364355] dbms/src/Storages/Page/V3/BlobStore.cpp:361 0x75a8c8a DB::PS::V3::PageStorageImpl::writeImpl(DB::WriteBatch&&, std::__1::shared_ptr const&) [tiflash+123374730] dbms/src/Storages/Page/V3/PageStorageImpl.cpp:138 0x7556597 DB::PageWriter::write(DB::WriteBatchWrapper&&, std::__1::shared_ptr) const [tiflash+123037079] dbms/src/Storages/Page/PageStorage.cpp:580 0x7f944a0 DB::RegionPersister::doPersist(std::__1::tuple&, DB::RegionTaskLock const&, DB::Region const&) [tiflash+133776544] dbms/src/Storages/Transaction/RegionPersister.cpp:111 0x7f93ca6 DB::RegionPersister::persist(DB::Region const&, DB::RegionTaskLock const&) [tiflash+133774502] dbms/src/Storages/Transaction/RegionPersister.cpp:72 0x7f3638d DB::KVStore::persistRegion(DB::Region const&, DB::RegionTaskLock const&, char const*) [tiflash+133391245] dbms/src/Storages/Transaction/KVStore.cpp:340 0x7f37752 DB::KVStore::forceFlushRegionDataImpl(DB::Region&, bool, DB::TMTContext&, DB::RegionTaskLock const&, unsigned long, unsigned long) [tiflash+133396306] dbms/src/Storages/Transaction/KVStore.cpp:425 0x7f36e38 DB::KVStore::canFlushRegionDataImpl(std::__1::shared_ptr const&, unsigned char, bool, DB::TMTContext&, DB::RegionTaskLock const&, unsigned long, unsigned long) [tiflash+133393976] dbms/src/Storages/Transaction/KVStore.cpp:410 0x7f371ee DB::KVStore::tryFlushRegionData(unsigned long, bool, bool, DB::TMTContext&, unsigned long, unsigned long) [tiflash+133394926] dbms/src/Storages/Transaction/KVStore.cpp:377 0x7f55ae8 TryFlushData [tiflash+133520104] dbms/src/Storages/Transaction/ProxyFFI.cpp:153 0x7f7080b4a1b7 _$LT$engine_store_ffi..observer..TiFlashObserver$LT$T$C$ER$GT$$u20$as$u20$raftstore..coprocessor..AdminObserver$GT$::pre_exec_admin::h8a2321ec830b8ac0 [libtiflash_proxy.so+23855543] 0x7f7081a2328d raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::apply_raft_cmd::h0d6480cc9a77a25f [libtiflash_proxy.so+39424653] 0x7f7081a3ec28 raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::hc5c1bcdc35a04965 [libtiflash_proxy.so+39537704] 0x7f7081a444ba raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h6af5feec71e8f970 [libtiflash_proxy.so+39560378] 0x7f7081a15bdc raftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::h0e908bd981bf034b [libtiflash_proxy.so+39369692] 0x7f7081a1a131 raftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::hceef58dc853a2734 [libtiflash_proxy.so+39387441] 0x7f7080c30f0e _$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::hcef411993d0cec81 [libtiflash_proxy.so+24801038] 0x7f7080ba8d03 batch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h735b90a4bed179e6 [libtiflash_proxy.so+24243459] 0x7f7080c98da8 std::sys_common::backtrace::__rust_begin_short_backtrace::h1cb2e050923d86be [libtiflash_proxy.so+25226664] 0x7f7080cde33e core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h3a3308035458aa5d [libtiflash_proxy.so+25510718] 0x7f70821d5915 std::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+47495445] /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108 0x7f707f159802 start_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=6009] ``` ## Root Cause 当 Tiflash 中 Region data 超过 256MB，而且还没有 decode 成列数据前就发生了 Region data 持久化，则会触发代码中内存越界访问。这时可能会有两种情况发生：1.如果没有发生 crash，会导致写入脏数据。2.如果发生 crash，会导致 tiflash 重启。重启后 restore 数据时，检查数据 checksum 等错误，从而导致反复重启 [[Issue]](https://github.com/pingcap/tiflash/issues/7316)。 ## Diagnostic Steps 1.集群中存在大事务写入等情况导致会有超过 256 MB 的 Region 产生（tiflash.log 日志中出现类似 `Created a new BlobStat [blob_id=339] [capacity=330405764]`，其中 capacity 大小超过 256MB 才会触发）。 2.tiflash 发生 crash 重启，crash 的堆栈中包含 "RegionPersister::doPersist"。 3.再次重启时打印出类似如下错误信息： `["DB::Exception: Reading with entries meet checksum not match ...[file=/path/to/kvstore/blobfile_xxx]"] ...` 或者 `[DB::Exception: invalid flag \u0000 in lock value 50267480000000000000505F728000000000C013FB8180B0E5CF84A28F06DBC805760A80000100000000000000000000000000000000000000000000000000]` ## Resolution 升级到未受影响版本 ## Workaround N/A ## Affected Versions v6.2.0 v6.3.0 v6.4.0 v6.5.0~v6.5.1 v6.6.0 v7.0.0