I have a problem with elastic search, all the load is concentrated on just one node, if I add new nodes they just sit there doing nothing.
How can I make elastic search distribute the load around all the nodes on the cluster?
I'm running a 3 node cluster
The cluster is just serving search requests its not injesting or indexing data.
the biggest index is 100 million records
I'm searching product titles for keywords. nothing complex.
here's the output from:
curl localhost:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
xxx.xxx.xxx.48 9 99 21 39.54 37.08 31.97 mdi - ES2
xxx.xxx.xxx.77 11 99 40 9.82 11.21 11.24 mdi * ES1
xxx.xxx.xxx.223 11 99 16 2.75 4.05 4.04 di - ES3
as you can see most of the load is on ES2 while ES3 is doing nothing.
I'm sending all the search requests to ES3 in the hope that it would take over some of the work.
All these servers have: 24 cpus, 65gb ram
running: ES 6.4.1
I basically installed Elastic Search from scratch,
change the /etc/elasticsearch/jvm.options to -Xms24g -Xmx24g
set discovery.zen.minimum_master_nodes: 2
all nodes are data nodes, with two masters
I hooked up the nodes to the cluster as normal.
I've tried changing the number of primary shards from 5 to 10 with 1 replica but it doesn't make any difference.
I've tried restarting the servers / Elastic Search multiple times.
The cluster status is green.
Are you querying using preference, which would cause the same shards to be queried? Are the shards for most frequently queried indices evenly distributed?
What is the output of the hot threads API on the busy node?
Although it is not related, I would also recommend making the third node master-eligible as well. You always want a minimum of 3 master eligible nodes in a cluster.
I can not see any reason for this unless one of the nodes is misconfigured, there is some hardware issue or you are using one of the features I mentioned. What does disk I/O look like on the different nodes? Is there anything in the Elasticsearch logs?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.