# Draft: Common Handling of Issues with Compaction Pending Bytes

## Background: Summary of several OnCalls involving compaction pending bytes encountered in a row in August 2021

> [ONCALL-3385](https://internal.pingcap.net/jira/browse/ONCALL-3385&sa=D&source=editors&ust=1661903531583088&usg=AOvVaw3NjD7DF52LKiQqtsep1u8n) [​​ONCALL-3450](https://internal.pingcap.net/jira/browse/ONCALL-3450&sa=D&source=editors&ust=1661903531583303&usg=AOvVaw0_a_G2zCFwAw23XvFJs5Ug) [ONCALL-3465](https://internal.pingcap.net/jira/browse/ONCALL-3465&sa=D&source=editors&ust=1661903531583531&usg=AOvVaw0q_Q97fZA_1D6D5jmPnJ8x) [ONCALL-3425](https://internal.pingcap.net/jira/browse/ONCALL-3425&sa=D&source=editors&ust=1661903531583741&usg=AOvVaw3TGts4AzzFHUK_rcAWFckt) [ONCALL-3512](https://internal.pingcap.net/jira/browse/ONCALL-3512&sa=D&source=editors&ust=1661903531583913&usg=AOvVaw1eKXCWmo2Piv9cgKQjnyFj)

## Problem Symptoms and analysis

The main symptom is that too many compaction pending bytes may lead to write stalls, and too much compaction traffic may lead to increased IO latency. This is turn leads to a poor customer experience.

The direction of investigation can be summarized into three general categories:

1. Changes upstream of the storage layer lead to changes in read and write pressure, increasing the number of compaction pending bytes. Typical factors include: business traffic changes, cluster topology changes (scheduling, capacity expansion and contraction), etc.
2. The code itself is buggy and requires R&D intervention to investigate.
3. Problems with the hard drive (downstream): insufficient capacity, faulty performance, etc.

Points 1 and 3 above rely on the investigator's direct Analysis.

Methods of mitigation:

1. Adjust soft/hard pending-compaction-bytes-limit
2. Adjusts compaction rate limiter to limit compaction flow spikes. 5.0 auto-tuned compaction rate limiter.
3. Version 5.2 supports traffic control at the scheduler level instead of RocksDB's write stall mechanism, which can avoid the problem of write stall mechanism jamming Raftstore or Apply threads when the amount of writes is large, resulting in a drop in QPS.

## Analysis that can be made from the perspective of support engineers

1\. Collect information. How did the compaction pending bytes change? Were there any adjustments made to the business around this time? What has been done to the cluster? Has the configuration been modified?

For business traffic changes, besides confirming with users, you can also make some Analysis through monitoring. You can observe the total QPS, QPS and duration of each type of request, CPU resource consumption changes, etc.

For the read and write traffic of RocksDB, you can observe the read flow / write flow of RocksDB - KV panel in TiKV Details. You can also observe the write stall reason from it.

For data migration due to internal cluster scheduling, you can observe the leader and region changes in TiKV Summary, etc.

For hard disk performance, you can see the panel under Disk Performance. If the disk performance is poor, it may also cause pending compaction to pile up.

2\. Determine if the failure is related to an increase in compaction pending bytes.

The rise in compaction pending bytes is most of the time a side-effect, not necessarily the root cause. The first thing you can do is to see if there is an increase in read/write pressure on RocksDB. If there is, you can look for the source of the increase in read/write pressure, which is usually caused by business changes or scheduling. See if the failure time corresponds to the time of business changes and data migration..

If a write stall occurs, see write stall reason.

## Summary of specific isues

[ONCALL-3385](https://internal.pingcap.net/jira/browse/ONCALL-3385&sa=D&source=editors&ust=1661903531586788&usg=AOvVaw2kROND0Aah2uZOz69s6yap) Streak cluster instability

- Symptom: unstable service, QPS drops to zero, etc.
- Cause: The user modified the low-space-ratio, causing data migration, resulting in a write stall. User can wait for data rebalancing.
- Mitigation plan: increase pending-compaction-bytes-limit.

[ONCALL-3450](https://internal.pingcap.net/jira/browse/ONCALL-3450&sa=D&source=editors&ust=1661903531587334&usg=AOvVaw0CH7fLTjrMWC062vF7uchi)  Streak compaction causes cluster instability

- Symptom: Lost leader, unstable cluster
- Reason: The store limit is large, the scheduling is fast, and a large compaction is generated
- Analysis method: From the monitoring point of view, the scheduling of the fault period is frequent. From the configuration point of view, the store limit configuration is unreasonable (160).
- Workaround: Restore the default value of store limit (15).

[ONCALL-3465](https://internal.pingcap.net/jira/browse/ONCALL-3465&sa=D&source=editors&ust=1661903531587978&usg=AOvVaw2Z32rkvE6XxreF3_RrAo-H)  Streak write stall, TiKV is unavailable

- Symptom: L0 file compaction speed can not keep up with the generated speed
- Reason: code bug, unexpected del_range ingestion
- Analysis method: Read the log to find unexpected behavior
- Fix PR: [https://github.com/tikv/tikv/pull/9200](https://github.com/tikv/tikv/pull/9200&sa=D&source=editors&ust=1661903531588608&usg=AOvVaw35_XpWC7uq9682v58nDnw4)

[ONCALL-3425](https://internal.pingcap.net/jira/browse/ONCALL-3425&sa=D&source=editors&ust=1661903531588874&usg=AOvVaw3qgYossmjwv5KCf8aaI21o)  RazorPay TiDB read and write slow

- Symptom: writing slows down (10s level), compaction pending bytes nearly 100 G
- Reasons: (1) It may be a business change. The user's new SQL has caused a large increase in TiKV's resource consumption, which is mainly concentrated on CPU and disk bandwidth.
- Analysis method: From monitoring and comparing before and after the fault, a large amount of read traffic is found, and the trend is related to the write delay.

[ONCALL-3512](https://internal.pingcap.net/jira/browse/ONCALL-3512&sa=D&source=editors&ust=1661903531589438&usg=AOvVaw38K_n7Fpq6G6imtuniMigw)  CloudChat compaction pending bytes accumulation causes "significant" increase in read and write latency

- Symptom: increased read and write latency
- Reason: shrinking and relocating data results in an increase in compaction traffic, which affects the IO delay of reading and writing in the foreground
- Analysis method: From the monitoring point of view, the period of IO delay rise corresponds to the period corresponding to the change period of the number of regions.
- Solution: compaction rate limiter, which limits compaction traffic peaks. But this cluster is already turned on (rate-bytes-per-sec = "110MB"). Pressing lower worries that the compaction rate is not fast enough and some cases will generate write stall. 5.0 has auto-tuned compaction rate limiter.

## Original article

[Draft: Compaction Pending Bytes 的常见处理](https://docs.google.com/document/d/1_4G73Q7cm4v6ns3iZ7l1ZfZ9NiaZj8ur-ZRKZqmPJ9U/edit?n=Compaction_Pending_Bytes#heading=h.fw9q865n4rnl).

### Keywords

Compaction, TiKV, Write Stall, Read Stall, Performance