# Handling Write Stall (Continue...)


![`smart`](./student-guide/diagram/TiKVHandlingWriteStallContinue_aa_001_20221014.png)

------------------------------------------------------------
Frequently Asked Questions - Write Stall
1. Write Stall trigger reason and related parameters

- Too many memtables

When the size of the data written to the memtable reaches the value set by the write_buffer_size parameter, the memtable becomes immutable waiting to be flushed to the L0 level. If the memtable has reached the write_buffer_size size and the amount of memtable data in memory reaches the max_write_buffer_number limit, then RocksDB's write stall will be triggered, waiting for the immutable to be flushed to the L0 floor.

This case of stall due to too many memtables is generally caused by a large amount of instantaneous writes and slow memtable flushing to disk. If disk write speed does not improve, and only service peaks occur, this can be mitigated by increasing the max_write_buffer_number parameter for the corresponding cf.

write_buffer_size controls the size of a write memory memtable. When the memtable is full, the data will be solidified to disk. The higher the value, the better the performance of large writes. max_write_buffer_number controls the number of memtables in write memory. However, these two values are not as large as possible, which can delay the data load time when a DB is reopened.

There may be too many memtable files at this step that cause write stall, affecting the write speed.
This usually occurs when instantaneous writes are high and the memtable flush to disk is slow. If disk write speed does not improve, and only business spikes occur, this can be mitigated by increasing the max-write-buffer-number corresponding to cf (default is 5).

- Too many level0 files

L0 Compaction is performed in a universal compaction (size tiered) manner. Trigger Compation condition: The number of SST files at the L0 layer reaches the level0-file-num-compaction-trigger setting value, and the default in TiKV is 4.
When the number of SST files at the L0 layer reaches the level0-slowdown-writes-trigger setting, the default in TiKV is 20. RocksDB will slow down the write speed and allow L0 to continue compaction as soon as possible.
When the number of SST files at the L0 layer reaches the level0-stop-writes-trigger setting value, the default in TiKV is 36. RocksDB stops writing the file and compacts L0 as soon as possible
There may be too many level0 sst files at this step that cause a write stall, affecting the write speed.
If too many level0 sst files cause stalls, you can modify the TiKV parameter rocksdb.max-sub-compactions. The default value is 3. Speed up level0 sst down to compact. This parameter means that the compaction tasks from level0 to level1 are cut up to max-sub-compactions subtasks to be executed concurrently by multiple threads.

- pending bytes too large

Immutable flushing to L0 is flush, L0~L6 flushing to the next level is compaction, and Compaction pending bytes indicates the size of the waiting compaction. Too many compaction pending bytes will cause a write stall when the compaction pending bytes reaches the rocksdb.defaultcf.soft-pending-compaction-bytes-limit parameter value (default 64G), RocksDB will slow down the write speed; if the Compaction pending bytes reach the rocksdb.defaultcf.hard-pending-compaction-bytes-limit parameter value (the default is 256G), RocksDB will stop writing.

Immutable flushing to L0 is flush, L0~L6 flushing to the next level is compaction, and Compaction pending bytes indicates the size of the waiting compaction. Too many compaction pending bytes will cause a write stall when the compaction pending bytes reaches the rocksdb.defaultcf.soft-pending-compaction-bytes-limit parameter value (default 64G), RocksDB will slow down the write speed; if the Compaction pending bytes reach the rocksdb.defaultcf.hard-pending-compaction-bytes-limit parameter value (the default is 256G), RocksDB will stop writing.

If too many compaction pending bytes result in write stall, there are several solutions:
    1. Turn on the soft-pending-compaction-bytes-limit and hard-pending-compaction-bytes-limit parameters to prevent the triggering of write stall, but this is only a symptom and not a cause. The root cause should be slow compaction. Check if the user has done a compaction limit. If so, you need to let go of the restriction a little
    2. If disk IO capabilities continue to fail to keep up, scale out is recommended.
    3. If the disk's throughput reaches its maximum, causing a write stall, but the CPU resources are sufficient, you can try a compression algorithm with a higher compression ratio to relieve disk stress and use CPU resources for disk resources. For example, the default cf compaction pressure is high, and the adjustment parameter is `[rocksdb.defaultcf] compression-per-level = [“no”, “no”, “lz4”, “lz4”, “lz4”, “zstd”, “zstd”]`  changed to `compression-per-level = [“no”, “no”, “zstd”, “zstd”, “zstd”, “zstd”, “zstd”]`