ES - v6.4.1
3Master, 2Data, 2Coord nodes
One replica setup. I use _routing.
Can somebody please explain me the below behavior ??
When i send continuous requests to my both coordinating nodes, i have observed that only one data node is utilized at a time.
When search threads in one data node function, at same time the ones in other data node are in park state and vice-versa.
Its evident from below image:
What is the configuration of your nodes? Do you use preference in addition to routing?
Master - 4Core 8GbRam
Data - 16Core 64GbRam (Heap allocated is 10Gb)
Coord - 8Core 32Gb Ram
No, we do not use preference. Only routing is used.
What does your elasticsearch.yml look like? Any non-defaut settings?
My expectation is both data nodes should serve equally at the same time.
From the thread utilization of search thread it seems like traffic is routed to either of the Primary shard or the replica.
Data node yml is simple as below, literally nothing extra:
cluster.name: prodcluster
search.remote.connect: false
node.name: data-01
node.master: false
node.data: true
node.ingest: false
node.ml: false
xpack.ml.enabled: false
network.host: site
network.publish_host: site
discovery.zen.ping.unicast.hosts:
path.data:
path.logs:
path.repo:
Have you set discovery.zen.minimum_master_nodes to 2? If not it is possible you could suffer from network partitions which can have strange and unexpected effects.
discovery.zen.minimum_master_nodes=2 has been set in elasticsearch.yml of master nodes only.
Does it need to be there in yml of Data nodes also ?
I don’t think so but am not sure.
Keeping that setting in master node configs has helped us since about 2year in production.
I have never noticed this behavior until today when i coincidentally saw that both data nodes are serving the requests in an alternate manner. Strange one.
Preface to this is performance issue. We are getting degraded performance when we hit requests with different routing values per request, in a mixed bag way.
Request 1: routing=a
Request 2: routing=b
Request 3: routing=c
N number of requests having these values in a mixed way. The performance suddenly decreases.
But is i hit thousands of requests say only for routing=a, the performance increases. Or for that matter any routing separately. Any lead as to why this issue might occur ?
sounds like a caching issue. You may want to try using the routing value as preference parameter as well.
In Data Nodes yml file not required to enter "discovery.zen.minimum_master_nodes=2"
Preference, i doubt will be of any help as i need proper load balancing.
I was more of relying on the 2 Coordinating nodes to do the smart load balancing for me.
I am connecting to both my coordinating nodes from my java client.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.