# Data Node runs out of memory (3.8 million IndexRequest)

**URL:** <https://discuss.elastic.co/t/data-node-runs-out-of-memory-3-8-million-indexrequest/161226>\
**Category:** Elasticsearch\
**Created:** [December 18, 2018, 12:26am UTC](https://discuss.elastic.co/t/data-node-runs-out-of-memory-3-8-million-indexrequest/161226 "2018-12-18T00:26:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![robert-blankenship](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert-blankenship/32/11287_2.png) [@robert-blankenship](https://discuss.elastic.co/u/robert-blankenship)\
**Post date:** [December 18, 2018, 12:26am UTC](https://discuss.elastic.co/t/data-node-runs-out-of-memory-3-8-million-indexrequest/161226/1 "2018-12-18T00:26:31Z")

</div>

The "Objects" column (2nd column) shows the total number of instances for that class/package, and the retained size shows the size of that class and all of its members.

There are thus 70,000 `TransportReplicationAction$ReroutePhase` , and their total size is ~15.6 GB.

 ![48%20PM](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e6da558531102e2a97663e1664a43d4fa4e6d02.png)

There are also 22,000 bulk requests:

 ![53%20PM](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a0b0ccba2d3fe1abb240b74088278b7b107fb78d.png)

3.8 million index requests:

 ![08%20PM](https://us1.discourse-cdn.com/elastic/original/3X/e/b/eb5ef45b995337dbb9d2f89d1b5a5352317208f0.png)

Here is an example document:

 ![16%20PM](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26926c38c23e2dfa53b421a7bb58421788d9b3b9.png)

However, my `thread_pool.bulk.queue_size` is only 50 for that node.

Additional info:  
46 ingest/hot nodes  
46 Logstash instances  
`thread_pool.bulk.queue_size: 50`  
`thread_pool.bulk.size: 17`  
batch size for ES publish is 200 documents per request.  
Each Logstash has 72 threads available for publishing.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 15, 2019, 12:55am UTC](https://discuss.elastic.co/t/data-node-runs-out-of-memory-3-8-million-indexrequest/161226/4 "2019-01-15T00:55:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
