We have a 19 node hot/warm/master cluster. Logstash is pointed at only the hot nodes.
Recently we are seeing that no matter which hot node gets the actual bulk request, only a single node is handling the request, usually not the node that received the request. If we restart the single node handling the requests, this behavior just moves to a different node, but still only that node handles the bulk requests.
We have verified that shards (primary and replicas) are distributed across all nodes properly, with no hot spots. We have verified that the load balancing of logstash to multiple hot nodes is occurring by reviewing the network traffic as well.
Our indices have settings for routing to ensure they only go to nodes marked as hot, and we've verified that all the hot nodes do in fact still have their hot tagging.
But we cannot determine why all the requests are still being handled by only a single node.