# Elasticsearch bulk rejection error in logstash logs **URL:** \ **Category:** Elasticsearch\ **Created:** [September 21, 2020, 4:44pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397 "2020-09-21T16:44:27Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Kartikey\_Bhatore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kartikey_bhatore/32/70145_2.png) [@Kartikey\_Bhatore](https://discuss.elastic.co/u/Kartikey_Bhatore)\ **Post date:** [September 21, 2020, 4:44pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397/1 "2020-09-21T16:44:27Z")
Hello, I am using ELK stack v6.5.3. I have a 7 node cluster (5 datanodes [elasticsearch + logstash] and 2 coordinator nodes [elasticsearch + kibana]) from past some days I am getting below error on one of my logstash nodes **[2020-09-21T16:05:50,889][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=\>"es\_rejected\_execution\_exception", "reason"=\>"rejected execution of processing of [142901911][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[indexName-2020.39][0]] containing [5] requests, target allocation id: F50tII90RQuy4HdjO\_G6Kw, primary term: 1 on EsThreadPoolExecutor[name = node1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@638276c2[Running, pool size = 8, active threads = 8, queued tasks = 200, completed tasks = 99902900]]"})** There is a lot of gc as well. I read the below document but not able to figure out the problem. [https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster) CPU utilization is also going high for the same node around 98% as an 95 percentile aggregation with a @timestamp bucket of every second. Also there is a huge lag in the logs , I hope the reason is this only please suggest on this as well. Please help. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 21, 2020, 8:52pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397/2 "2020-09-21T20:52:48Z")
What is the output from `GET /_cluster/stats`? ---
**Author:** ![Kartikey\_Bhatore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kartikey_bhatore/32/70145_2.png) [@Kartikey\_Bhatore](https://discuss.elastic.co/u/Kartikey_Bhatore)\ **Post date:** [September 22, 2020, 6:18am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397/3 "2020-09-22T06:18:35Z")
> [@warkolm](#): > > `GET /_cluster/stats` ? { "\_nodes" : { "total" : 7, "successful" : 7, "failed" : 0 }, "cluster\_name" : "clusterName", "cluster\_uuid" : "WmJ66kYmSaa6osmtVFnkrQ", "timestamp" : 1600794514198, "status" : "green", "indices" : { "count" : 652, "shards" : { "total" : 1336, "primaries" : 682, "replication" : 0.9589442815249267, "index" : { "shards" : { "min" : 1, "max" : 6, "avg" : 2.049079754601227 }, "primaries" : { "min" : 1, "max" : 5, "avg" : 1.0460122699386503 }, "replication" : { "min" : 0.0, "max" : 1.0, "avg" : 0.9877300613496932 } } }, "docs" : { "count" : 9137129079, "deleted" : 2309 }, "store" : { "size\_in\_bytes" : 5226509834484 }, "fielddata" : { "memory\_size\_in\_bytes" : 326647976, "evictions" : 0 }, "query\_cache" : { "memory\_size\_in\_bytes" : 8315906, "total\_count" : 2175939, "hit\_count" : 44890, "miss\_count" : 2131049, "cache\_size" : 1548, "cache\_count" : 3290, "evictions" : 1742 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 19411, "memory\_in\_bytes" : 10035861353, "terms\_memory\_in\_bytes" : 7088193492, "stored\_fields\_memory\_in\_bytes" : 1451567376, "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory\_in\_bytes" : 48147968, "points\_memory\_in\_bytes" : 991423913, "doc\_values\_memory\_in\_bytes" : 456528604, "index\_writer\_memory\_in\_bytes" : 399487824, "version\_map\_memory\_in\_bytes" : 247484, "fixed\_bit\_set\_memory\_in\_bytes" : 0, "max\_unsafe\_auto\_id\_timestamp" : 1600761340377, "file\_sizes" : { } } }, "nodes" : { "count" : { "total" : 7, "data" : 5, "coordinating\_only" : 2, "master" : 5, "ingest" : 5 }, "versions" : [ "6.5.3" ], "os" : { "available\_processors" : 48, "allocated\_processors" : 48, "names" : [ { "name" : "Linux", "count" : 7 } ], "mem" : { "total\_in\_bytes" : 405139738624, "free\_in\_bytes" : 16613928960, "used\_in\_bytes" : 388525809664, "free\_percent" : 4, "used\_percent" : 96 } }, "process" : { "cpu" : { "percent" : 174 }, "open\_file\_descriptors" : { "min" : 413, "max" : 2374, "avg" : 1681 } }, "jvm" : { "max\_uptime\_in\_millis" : 1884453538, "versions" : [ { "version" : "1.8.0\_131", "vm\_name" : "Java HotSpot(TM) 64-Bit Server VM", "vm\_version" : "25.131-b11", "vm\_vendor" : "Oracle Corporation", "count" : 7 } ], "mem" : { "heap\_used\_in\_bytes" : 76398398920, "heap\_max\_in\_bytes" : 182117728256 }, "threads" : 924 }, "fs" : { "total\_in\_bytes" : 15218212196352, "free\_in\_bytes" : 9864063373312, "available\_in\_bytes" : 9167827349504 } } } hope this helps 🙂 I followed below doc and reduced the heap from 32 to 26 (I think I have given a huge amount for less number of shards following the basic formula of 20 shards per GB of heap , I have around 266 shards per node ) [https://www.elastic.co/blog/a-heap-of-trouble#fn4](https://www.elastic.co/blog/a-heap-of-trouble#fn4) ---
**Author:** ![Kartikey\_Bhatore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kartikey_bhatore/32/70145_2.png) [@Kartikey\_Bhatore](https://discuss.elastic.co/u/Kartikey_Bhatore)\ **Post date:** [October 6, 2020, 5:52pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397/4 "2020-10-06T17:52:02Z")
@warkolm Gentle reminder please......... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 6, 2020, 6:39pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397/5 "2020-10-06T18:39:36Z")
How many indices and shards are you actively indexing into? How are you indexing into Elasticsearch (what does you Elasticsearch output config in Logstash look like)? ---
**Author:** ![Kartikey\_Bhatore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kartikey_bhatore/32/70145_2.png) [@Kartikey\_Bhatore](https://discuss.elastic.co/u/Kartikey_Bhatore)\ **Post date:** [October 22, 2020, 11:58am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397/6 "2020-10-22T11:58:43Z")
I have around 50 weekly indices and 1 daily index. for the weekly index I have 1 shard and for daily I have 3 shards. I am simply giving my outputs in output.elasticsearch. Sorry I can not share the file here. Do you want ay specific info from that file? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 19, 2020, 11:58am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-rejection-error-in-logstash-logs/249397/7 "2020-11-19T11:58:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.