# Unbalanced Elasticsearch Cluster Performance

**URL:** <https://discuss.elastic.co/t/unbalanced-elasticsearch-cluster-performance/92377>\
**Category:** Elasticsearch\
**Created:** [July 8, 2017, 9:53pm UTC](https://discuss.elastic.co/t/unbalanced-elasticsearch-cluster-performance/92377 "2017-07-08T21:53:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![MohammadMazraeh](https://avatars.discourse-cdn.com/v4/letter/m/8dc957/32.png) [@MohammadMazraeh](https://discuss.elastic.co/u/MohammadMazraeh)\
**Post date:** [July 8, 2017, 9:53pm UTC](https://discuss.elastic.co/t/unbalanced-elasticsearch-cluster-performance/92377/1 "2017-07-08T21:53:22Z")

</div>

We have an Elasticsearch cluster with 9 nodes with the following settings:

- Elasticsearch Version 5.1.2
- One Index in Cluster
- Primary Shard Storage Size: 3GB
- Number of Shards: 5
- Number of Replica: 3
- Node-1, Node-2 and Node-3 Master Only Nodes
- Node-4 through Node-9 Data Only Nodes
- No Parent Child Relationtip in Mappings
- Each node 24 GB of Ram, 18 Cores of CPU
- Disabled Swaped, Increased Open File Descriptor, 12 GB JVM Heap Memory
- [Elasticsearch Nest Client](https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/introduction.html) 'Static' Adaptor and List of all Nodes IPs

As you see we have an over allocation of resources on our nodes but under stress test only one node uses all it's available search threads. As I mentioned we have 18 cores and according to default search thread limit we have (3\*18/2)+1 = 28 search threads in each node.  
**Problems:**

- **Http Requests Are Not Balanced**
- **Other nodes don't use all their search threads. One nodes uses It's all threads and It's search queue gets large**

**What we have tested:**

- Use one coordinator node to balance requests (no change)  
**How we send requests:**
- We use Elasticsearch as a Search Engine and a `Jmeter` is used to put stress test on search services. Test services are web services which call Some SearchTemplates using [Elasticsearch Nest Client](https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/introduction.html)

[![Open HTTP Requests and Search Thread Pool][1]][1]

[![Cpu Usages][2]][2]

[![Queru Count and Fetch Count][3]][3]

Any idea is appreciated.  
[1]: [https://i.stack.imgur.com/H9qq6.png](https://i.stack.imgur.com/H9qq6.png)  
[2]: [https://i.stack.imgur.com/KYRAx.png](https://i.stack.imgur.com/KYRAx.png)  
[3]: [https://i.stack.imgur.com/n4I9m.png](https://i.stack.imgur.com/n4I9m.png)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 8, 2017, 10:07pm UTC](https://discuss.elastic.co/t/unbalanced-elasticsearch-cluster-performance/92377/2 "2017-07-08T22:07:27Z")

</div>

What version.  
How are you making requests - what client.

---

<div class="post-metadata">

**Author:** ![MohammadMazraeh](https://avatars.discourse-cdn.com/v4/letter/m/8dc957/32.png) [@MohammadMazraeh](https://discuss.elastic.co/u/MohammadMazraeh)\
**Post date:** [July 9, 2017, 4:21am UTC](https://discuss.elastic.co/t/unbalanced-elasticsearch-cluster-performance/92377/3 "2017-07-09T04:21:30Z")

</div>

We use Elasticsearch V 5.1.2 as a Search Engine and a `Jmeter` is used to put stress test on search services. Test services are web services which call Some SearchTemplates using [Elasticsearch Nest Clien](https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/introduction.html)t.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 6, 2017, 4:21am UTC](https://discuss.elastic.co/t/unbalanced-elasticsearch-cluster-performance/92377/4 "2017-08-06T04:21:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.