Unbalanced search performance

[avaneeshr (Avaneesh)]

Hi!
I am using ES v2.4.0 on 5 nodes with a very search query (bool queries) workload, in AWS, using compute optimized servers.
I am querying the cluster via a HAProxy Load Balancer which uses roundrobin policy for each data node.
The cluster has one dedicated master node.
When I see the output of cat api / threads

All nodes are equal in all aspects, and have equal data. Everything is perfectly uniform across the 5 servers.

Then why does one node have so many active search threads?
Problem is, this makes the load avg on the 2nd server more than 100!

[mujtabahussain]

Any chance the HAProxy policy is not correct? Could you post that here?

[avaneeshr (Avaneesh)]

Okay so there has been a new development in this case.
I removed the HAProxy and instead directly query one of the data nodes. Still the same result. The load averages appear to be unbalanced.
I've brought a change in my cluster and since then, I've seen some balance being restored: I marked the data nodes as specifically "data nodes" in the configs and added 3 separate master nodes. Could this be related to the problem?
Should I add dedicated client nodes? Would that theoretically solve the issue?

Thanks!