# Elasticsearch cluster overloaded

**URL:** <https://discuss.elastic.co/t/elasticsearch-cluster-overloaded/118868>\
**Category:** Elasticsearch\
**Created:** [February 7, 2018, 3:21pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-overloaded/118868 "2018-02-07T15:21:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![fsperling](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fsperling](https://discuss.elastic.co/u/fsperling)\
**Post date:** [February 7, 2018, 3:21pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-overloaded/118868/1 "2018-02-07T15:21:43Z")

</div>

Hi,

Our elasticsearch cluster get overloaded from time to time.  
We see thread\_pool rejections on elasticsearch and logstash has error messages like the below

```
[logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@57757508 on EsThreadPoolExecutor[bulk, queue capacity = 500, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@127e2b61[Running, pool size = 32, active threads = 32, queued tasks = 507, completed tasks = 575009786]]"}) 

```

(Increasing the thread\_pool.bulk.queue\_size to 500 had helped a bit)

Our cluster has 26TB of data, 8 hot, 5 warm and 10 cold nodes. We have ~2000 indices across 6800 primary shards, replicated once. We are running elasticsearch 5.5. Each elasticsearch instance has 30GB of memory.  
The hot nodes have a max of 80 shards each.

Looking at our servers the cpu load and IO are very low. CPU is around 15%, IO 30% with peaks of 50%. File/ulimits are also fine.

We are wondering why elasticsearch isn't using more of the resources if it's under load / overloaded. And if there are settings in elasticsearch to improve the performance and get rid of those errors.

It seems the most load is from indexing so we increased indices.memory.index\_buffer\_size to 30%

Any tips would be appreciated.  
Cheers,  
Felix

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [February 7, 2018, 4:36pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-overloaded/118868/2 "2018-02-07T16:36:02Z")

</div>

Lots of indices and shards, indeed... This might be worth reading: [https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 7, 2018, 4:36pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-overloaded/118868/3 "2018-03-07T16:36:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.