# Handling OOM Failures on a Cluster with a Massive Number of Queries

## Summary

-------

In some situations, an Out of Memory \[OOM\] failure may arise from a single, “boil the ocean” query. Think ‘SELECT \* FROM \*’ or something of that ilk. Another way to run into the OOM Killer is when an application generates a massive number of small queries that each ask for a small range of values from a table, but due to the sheer number of them, the system resources end up being overwhelmed, resulting in an OOM failure.

## Diagnosis

---------

Before a typical OOM event, you will see a spike in memory usage approaching 100% usage. The picture below shows the memory usage graph for a group of servers:

![Memory Usage Prior to OOM Incident](/hc/article_attachments/4408159521943/image-0.png)

In the case of a very massive number of small queries, you will also see an elevated number of Worker Pending Tasks:

![A screenshot of a computer
Description automatically generated with medium confidence](/hc/article_attachments/4408165001879/image-1.png)

In addition, you will see a corresponding spike in Garbage Collector tasks, all of which end up taking up memory resources:

![A screenshot of a computer
Description automatically generated with medium confidence](/hc/article_attachments/4408159521687/image-2.png)

## Solution

--------

Because every cluster (even an exceptionally large cluster) only has a finite number of resources, it is important to setup the root configuration to handle situations where (for instance) a nested query creates a process for every branch in the data set. A similar situation may arise in graph-query scenario, where each data item returned from one query fans-out into separate subqueries as the client-application algorithm reaches deeper and deeper into a tree of nodes.

To prevent this situation, it is recommended that you adjust the root configuration to use less resources per host, thus decreasing the pressure on an individual host’s resources.

For instance, if your configuration is set as follows:

```yaml
  tikv:

    raftstore.apply-pool-size: 8

    raftstore.store-pool-size: 8

    readpool.unified.max-thread-count: 32
```

the following configuration change would sharply reduce the pressure on CPU cores as well as memory:

```yaml
tikv:

    raftstore.apply-pool-size: 3

    raftstore.store-pool-size: 3
```

Be sure to remove the old readpool configuration:

~~readpool.unified.max-thread-count: 32~~

Deleting the readpool.unified.max-thread-count setting would tell the system to only use the default, which is 80% of CPU capacity.

By configuring the server in this manner, it becomes possible to stabilize a cluster that is experiencing OOM incidents. Once the OOM situation is managed, you should also examine and tune the query (or set of queries) that are invoking a large number of subqueries.

## Related Articles

----------------

[Tune TiKV Thread Pool Performance](https://docs.pingcap.com/tidb/stable/tune-tikv-thread-performance).

[OnCall-2751](https://internal.pingcap.net/jira/browse/ONCALL-2751?focusedCommentId=85861&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-85861): Streak tikv sudden high cpu and oom.

[OnCall-2769](https://internal.pingcap.net/jira/browse/ONCALL-2769): Streak TiKV OOM (very high Worker pending tasks?)

## Applies To

----------

All TiDB releases

## Meta

**#OOM #Performance #Architecture**