# ES v5.4.0 Bulk Requests Rejection

**URL:** https://discuss.elastic.co/t/es-v5-4-0-bulk-requests-rejection/152853
**Category:** Elasticsearch
**Created:** [October 17, 2018, 1:51pm UTC](https://discuss.elastic.co/t/es-v5-4-0-bulk-requests-rejection/152853 "2018-10-17T13:51:04Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Neha\_Sharma1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neha_sharma1/32/36504_2.png) [@Neha\_Sharma1](https://discuss.elastic.co/u/Neha_Sharma1)
#### Post date: [October 17, 2018, 1:51pm UTC](https://discuss.elastic.co/t/es-v5-4-0-bulk-requests-rejection/152853/1 "2018-10-17T13:51:04Z")

</div>

I have a cluster with data node 7 which has 100+ indexes, with shards number varying from 7-14, with 1 replica. As the indexes are time-series, all of them get created at 12 AM UTC. The problem is that all the primary shards of most of the indexes are allocated to one node(say N1) and few primary shards and replicas are assigned to other nodes.

Values set to:  
cluster.routing.rebalance.enable: all  
cluster.routing.allocation.allow\_rebalance: indices\_all\_active  
thread\_pool.bulk.queue\_size: 1000

During bulk indexing requests, all of the requests go to that node (N1) and CPU utilization increases for this node. A lot of requests are also rejected as the queue size is exceeded on that node. Whereas other nodes are pretty chilled out.

Doubts:

1. Is the above issue, is it because all the primary shards are on one node only?
2. If yes, Can I rebalance my primary shards by setting "cluster.routing.rebalance.enable" to "primaries". Would this configuration first rebalance my primary shards and then balance the replicas? Are there any repercussions.
3. Is there any other cause of the issue and is there a way to mitigate it?

---

<div class="post-metadata">

### Author: ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)
#### Post date: [October 18, 2018, 6:16pm UTC](https://discuss.elastic.co/t/es-v5-4-0-bulk-requests-rejection/152853/2 "2018-10-18T18:16:08Z")

</div>

Some reasons I could imagine lots of new shards getting assigned to one node would be

1. At the time of allocation this node has the fewest number of shards. Perhaps there's a race around that decision when 100+ indices are simultaneously created?
2. Disk watermarks or some other allocation limits are being hit in other nodes, preventing them from getting assigned. One way to gain insight into how ES is making these allocation decisions is to [manually change the allocation](https://www.elastic.co/guide/en/elasticsearch/reference/master/cluster-reroute.html) of one of these primary shards and see if ES responds with some other rebalancing.

If nothing jumps out at you along these lines, perhaps consider using [shard allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/master/shard-allocation-filtering.html) to ensure a more widespread initial (and eventual) distribution of the shards.

Good luck!

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 18, 2018, 6:19pm UTC](https://discuss.elastic.co/t/es-v5-4-0-bulk-requests-rejection/152853/3 "2018-10-18T18:19:31Z")

</div>

This seems to be a duplicate of [this issue](https://discuss.elastic.co/t/storage-performance-optimization/152393/3). Please do not open multiple threads for the same issue.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 15, 2018, 6:19pm UTC](https://discuss.elastic.co/t/es-v5-4-0-bulk-requests-rejection/152853/4 "2018-11-15T18:19:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
