Unbalanced search performance

I am using ES v2.4.0 on 5 nodes with a very search query (bool queries) workload, in AWS, using compute optimized servers.
I am querying the cluster via a HAProxy Load Balancer which uses roundrobin policy for each data node.
The cluster has one dedicated master node.
When I see the output of cat api / threads

All nodes are equal in all aspects, and have equal data. Everything is perfectly uniform across the 5 servers.

Then why does one node have so many active search threads?
Problem is, this makes the load avg on the 2nd server more than 100!

Any chance the HAProxy policy is not correct? Could you post that here?

Okay so there has been a new development in this case.
I removed the HAProxy and instead directly query one of the data nodes. Still the same result. The load averages appear to be unbalanced.
I've brought a change in my cluster and since then, I've seen some balance being restored: I marked the data nodes as specifically "data nodes" in the configs and added 3 separate master nodes. Could this be related to the problem?
Should I add dedicated client nodes? Would that theoretically solve the issue?


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.