# [KB]TiFlash 实例 crash，不断重启失败 parameter out of bound in IcolumnString

## Key Words

tiflash,crash,panic,out of bound

## Description

 TiFlash 实例反复自动重启并且重启前有 ERROR 日志ERROR parameter out of bound in IcolumnString，日志内容：

```log
[2022/09/13 15:30:02.786 +08:00] [ERROR] [<unknown>] ["bool DB::GCManager::work():Code: 12, e.displayText() = DB::Exception: Parameter out of bound in IColumnString::insertRangeFrom method, start=0, length=8, src_size=4:  while reading DTFile /data2/jaysonhuang/tiup-deploy/data/tiflash-5018/data/t_77/stable/dmf_2: while apply gc Segment [1] [range=[-9223372036854775808,9223372036854775807)] [table=t_77], e.what() = DB::Exception, Stack trace:
0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x36d42e6]
1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x36c8116]
2. bin/tiflash/tiflash(DB::ColumnString::insertRangeFrom(DB::IColumn const&, unsigned long, unsigned long)+0x2ff) [0x759712f]
3. bin/tiflash/tiflash(DB::DM::copyColumnsData(std::vector<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> > > const&, COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> const&, std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0x7fe) [0x76f690e]
4. bin/tiflash/tiflash(DB::DM::DPFileReader::readRowsOnce(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0x135) [0x77059c5]
5. bin/tiflash/tiflash(DB::DM::DPFileReader::readRows(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0x55) [0x7706d85]
6. bin/tiflash/tiflash(DB::DM::DeltaValueReader::readRows(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*)+0xcf) [0x771c34f]
7. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, false>::writeInsertFromDelta(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long&)+0x94) [0x7698344]
8. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, false>::read()+0x863) [0x769b593]
9. bin/tiflash/tiflash(DB::DM::DMRowKeyFilterBlockInputStream<true>::read()+0x5f) [0x7692d2f]
10. bin/tiflash/tiflash(DB::DM::PKSquashingBlockInputStream<false>::read()+0x9f1) [0x7698d61]
11. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock()+0x2f) [0x7a3dcbf]
12. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x1dd0) [0x7a411a0]
13. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::read()+0x18) [0x7681968]
14. bin/tiflash/tiflash(DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DM::DMFileWriter::Flags)+0x20b) [0x7670cdb]
15. bin/tiflash/tiflash(DB::DM::createNewStable(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::IBlockInputStream> const&, unsigned long, DB::DM::WriteBatches&)+0xbd) [0x767143d]
16. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0x17d) [0x767ab7d]
17. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3e1) [0x76496b1]
18. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::onSyncGc(long)+0x5c1) [0x7650601]
19. bin/tiflash/tiflash(DB::GCManager::work()+0x7b1) [0x7a09911]
20. bin/tiflash/tiflash(DB::BackgroundProcessingPool::threadFunction()+0x947) [0x75dba17]
21. bin/tiflash/tiflash() [0x8b14f6f]
22. /lib64/libpthread.so.0(+0x6e63) [0x7fe4a07f6e63]
23. /lib64/libc.so.6(clone+0x6d) [0x7fe4a022b55d]
"] [thread_id=10]
```

v5.4.2/v6.1.1/v6.2.0 版本日志有细微差异：

```golang
[2022/09/13 15:59:30.623 +08:00] [ERROR] [Exception.cpp:85] ["DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t):Code: 9, e.displayText() = DB::Exception: Sizes of columns doesn't match: _INTERNAL_DELMARK(id=-1025): 8, source(id=2): 4, e.what() = DB::Exception, Stack trace:
       0x1cdb0ce    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30257358]
                    dbms/src/Common/Exception.h:41
       0x65b93c4    DB::Block::checkNumberOfRows() const [tiflash+106664900]
                    dbms/src/Core/Block.cpp:243
       0x735076e    bool DB::RegionBlockReader::readImpl<(DB::TMTPKType)0>(DB::Block&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > > const&, bool) [tiflash+120915822]
                    dbms/src/Storages/Transaction/RegionBlockReader.cpp:211
       0x732cc13    DB::GenRegionBlockDataWithSchema(std::__1::shared_ptr<DB::Region> const&, std::__1::shared_ptr<DB::DecodingStorageSchemaSnapshot const> const&, unsigned long, bool, DB::TMTContext&) [tiflash+120769555]
                    dbms/src/Storages/Transaction/PartitionStreams.cpp:606
       0x72ee5a6    DB::DM::SSTFilesToBlockInputStream::readCommitedBlock() [tiflash+120513958]
                    dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:238
       0x72edcca    DB::DM::SSTFilesToBlockInputStream::read() [tiflash+120511690]
                    dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:144
       0x71d25c5    DB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+119350725]
                    dbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:253
       0x72eeec2    DB::DM::PKSquashingBlockInputStream<true>::read() [tiflash+120516290]
                    dbms/src/Storages/DeltaMerge/PKSquashingBlockInputStream.h:68
       0x71d25c5    DB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+119350725]
                    dbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:253
       0x1c9f065    DB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock() [tiflash+30011493]
                    dbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.h:134
       0x1c9d16c    DB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+30003564]
                    dbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.cpp:51
       0x72e954c    DB::DM::SSTFilesToDTFilesOutputStream::write() [tiflash+120493388]
                    dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp:162
       0x72e2dc3    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+120466883]
                    dbms/src/Storages/Transaction/ApplySnapshot.cpp:340
       0x733183a    PreHandleSnapshot [tiflash+120789050]
                    dbms/src/Storages/Transaction/ProxyFFI.cpp:387
  0x7f059e8b3e2a    raftstore::engine_store_ffi::_$LT$impl$u20$raftstore..engine_store_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h718eff40e0ff22c6 [libtiflash_proxy.so+33197610]
  0x7f059e86be89    raftstore::store::snap::Snapshot::pre_handle_snapshot::h81c04c547dfcb1a8 [libtiflash_proxy.so+32902793]
  0x7f059df01536    yatp::task::future::RawTask$LT$F$GT$::poll::h68cf0c4ec7b4731e [libtiflash_proxy.so+23029046]
  0x7f059f51ec0d    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h6502e9a2b4ccd66e [libtiflash_proxy.so+46218253]
  0x7f059f52143d    std::sys_common::backtrace::__rust_begin_short_backtrace::hc5d6c9c6e9339eea [libtiflash_proxy.so+46228541]
  0x7f059f51413d    core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h8a6505e26346d93a [libtiflash_proxy.so+46174525]
  0x7f059ec5ca0a    std::sys::unix::thread::Thread::new::thread_start::hd39c5f08bdcda277 [libtiflash_proxy.so+37034506]
  0x7f059bfece63    start_thread [libpthread.so.0+28259]
                    /root/yuzhao/glibc-2.17/nptl/pthread_create.c:308
  0x7f059bb0655d    __clone [libc.so.6+968029]
                    /root/yuzhao/glibc-2.17/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113"] [thread_id=48]
```

## Root Cause

TiFlash 默认假设了带有 `primary key` 标记的列，它们的值都是非 null。但是实际使用过程中存在带有 `primary key` 标记的列的值是 null，这导致 Tiflash 在实际处理这些 `primary key`  列的 null 值时发生非预期 panic。

## Diagnostic Steps

N/A

## Resolution

升级到未受影响版本（ v5.4.3, v6.1.2, v6.3, and later version）

## Workaround

下线出问题的 TiFlash 实例并且设置问题表的 Tiflash 副本为 0，然后重新扩容 TiFlash 实例。

## Affected Versions

v4.0.x

v5.0.x

v5.1.x

v5.2.x

v5.3.x

v5.4.0~v5.4.2

v6.1.0~v6.1.1

v6.2.0