We are having problems with slow searching and we are suspecting it has to do with our data node disproportionately handling the indexing load. What we see in our monitoring is this flip flopping of indexing rate between our data nodes.
I have a suspicion this is related to the following cluster setting.
cluster.routing.allocation.awareness.attributes: aws_availability_zone
There are 4 data nodes in the cluster 2 per availability zone. The graphs above are from 2 nodes in the same AZ and the 2 other data nodes in the other AZ have the same pattern.