# 2 Data nodes being used unevenly

**URL:** <https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521>\
**Category:** Elasticsearch\
**Created:** [September 21, 2019, 9:24am UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521 "2019-09-21T09:24:47Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\
**Post date:** [September 21, 2019, 9:24am UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/1 "2019-09-21T09:24:47Z")

</div>

ES - v6.4.1  
3Master, 2Data, 2Coord nodes  
One replica setup. I use \_routing.

Can somebody please explain me the below behavior ??

When i send continuous requests to my both coordinating nodes, i have observed that only one data node is utilized at a time.  
When search threads in one data node function, at same time the ones in other data node are in park state and vice-versa.

Its evident from below image:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e41022dd5649edc54e11f7dcdf2535b72f482cb.png)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2019, 9:40am UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/2 "2019-09-21T09:40:57Z")

</div>

What is the configuration of your nodes? Do you use preference in addition to routing?

---

<div class="post-metadata">

**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\
**Post date:** [September 21, 2019, 9:43am UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/3 "2019-09-21T09:43:18Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> What is the configuration of your nodes?

Master - 4Core 8GbRam  
Data - 16Core 64GbRam (Heap allocated is 10Gb)  
Coord - 8Core 32Gb Ram

> [@Christian\_Dahlqvist](#):
>
> Do you use preference in addition to routing?

No, we do not use preference. Only routing is used.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2019, 10:00am UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/4 "2019-09-21T10:00:52Z")

</div>

What does your elasticsearch.yml look like? Any non-defaut settings?

---

<div class="post-metadata">

**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\
**Post date:** [September 21, 2019, 10:05am UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/5 "2019-09-21T10:05:33Z")

</div>

My expectation is both data nodes should serve equally at the same time.  
From the thread utilization of search thread it seems like traffic is routed to either of the Primary shard or the replica.

Data node yml is simple as below, literally nothing extra:

> cluster.name: prodcluster  
> search.remote.connect: false  
> node.name: data-01  
> node.master: false  
> node.data: true  
> node.ingest: false  
> node.ml: false  
> xpack.ml.enabled: false  
> network.host: _site_  
> network.publish\_host: _site_  
> discovery.zen.ping.unicast.hosts:  
> path.data:  
> path.logs:  
> path.repo:

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2019, 12:12pm UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/6 "2019-09-21T12:12:58Z")

</div>

Have you set `discovery.zen.minimum_master_nodes` to `2`? If not it is possible you could suffer from network partitions which can have strange and unexpected effects.

---

<div class="post-metadata">

**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\
**Post date:** [September 21, 2019, 12:16pm UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/7 "2019-09-21T12:16:00Z")

</div>

discovery.zen.minimum\_master\_nodes=2 has been set in elasticsearch.yml of master nodes only.

Does it need to be there in yml of Data nodes also ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2019, 12:25pm UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/8 "2019-09-21T12:25:49Z")

</div>

I don’t think so but am not sure.

---

<div class="post-metadata">

**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\
**Post date:** [September 21, 2019, 12:33pm UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/9 "2019-09-21T12:33:47Z")

</div>

Keeping that setting in master node configs has helped us since about 2year in production.

I have never noticed this behavior until today when i coincidentally saw that both data nodes are serving the requests in an alternate manner. Strange one.

Preface to this is performance issue. We are getting degraded performance when we hit requests with different routing values per request, in a mixed bag way.

Request 1: routing=a  
Request 2: routing=b  
Request 3: routing=c

N number of requests having these values in a mixed way. The performance suddenly decreases.

**But** is i hit thousands of requests say only for routing=a, the performance increases. Or for that matter any routing separately. Any lead as to why this issue might occur ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2019, 1:10pm UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/10 "2019-09-21T13:10:00Z")

</div>

sounds like a caching issue. You may want to try using the routing value as preference parameter as well.

---

<div class="post-metadata">

**Author:** ![Yashwantbokadia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yashwantbokadia/32/54649_2.png) [@Yashwantbokadia](https://discuss.elastic.co/u/Yashwantbokadia)\
**Post date:** [September 22, 2019, 5:46am UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/11 "2019-09-22T05:46:51Z")

</div>

In Data Nodes yml file not required to enter "discovery.zen.minimum\_master\_nodes=2"

---

<div class="post-metadata">

**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\
**Post date:** [September 22, 2019, 2:30pm UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/12 "2019-09-22T14:30:18Z")

</div>

Preference, i doubt will be of any help as i need proper load balancing.

I was more of relying on the 2 Coordinating nodes to do the smart load balancing for me.  
I am connecting to both my coordinating nodes from my java client.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 20, 2019, 2:30pm UTC](https://discuss.elastic.co/t/2-data-nodes-being-used-unevenly/200521/13 "2019-10-20T14:30:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
