I am using ES v2.4.0 on 5 nodes with a very search query (bool queries) workload, in AWS, using compute optimized servers.
I am querying the cluster via a HAProxy Load Balancer which uses roundrobin policy for each data node.
The cluster has one dedicated master node.
When I see the output of cat api / threads
All nodes are equal in all aspects, and have equal data. Everything is perfectly uniform across the 5 servers.
Then why does one node have so many active search threads?
Problem is, this makes the load avg on the 2nd server more than 100!