We are seeing slow query performance on elastic-search. This coincide with increase in search queue size but at the same time active search threads as a percentage of search thread pool size is pretty low. This indicate that even though we have many threads that could be assigned to the search request queuing on the node, they are left idle.
One explanation to this could be that the default max concurrent search size on the multi-search request is the reason why we are seeing low thread utilization in search thread pool. Searches in these multi search request are the ones that are queued for thread bottlenecked by the max concurrent search size setting on the multi-search request. Does this make sense? Is there any other other reason why this should happen?