I have a problem with elastic search, all the load is concentrated on just one node, if I add new nodes they just sit there doing nothing.
How can I make elastic search distribute the load around all the nodes on the cluster?
I'm running a 3 node cluster
The cluster is just serving search requests its not injesting or indexing data.
the biggest index is 100 million records
I'm searching product titles for keywords. nothing complex.
here's the output from:
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name xxx.xxx.xxx.48 9 99 21 39.54 37.08 31.97 mdi - ES2 xxx.xxx.xxx.77 11 99 40 9.82 11.21 11.24 mdi * ES1 xxx.xxx.xxx.223 11 99 16 2.75 4.05 4.04 di - ES3
as you can see most of the load is on ES2 while ES3 is doing nothing.
I'm sending all the search requests to ES3 in the hope that it would take over some of the work.
All these servers have: 24 cpus, 65gb ram
running: ES 6.4.1
I basically installed Elastic Search from scratch,
change the /etc/elasticsearch/jvm.options to -Xms24g -Xmx24g
set discovery.zen.minimum_master_nodes: 2
all nodes are data nodes, with two masters
I hooked up the nodes to the cluster as normal.
I've tried changing the number of primary shards from 5 to 10 with 1 replica but it doesn't make any difference.
I've tried restarting the servers / Elastic Search multiple times.
The cluster status is green.
i'm lost as what to try next!