_msearch is an Elasticsearch API. What Kibana is doing when it calls this is form a bunch of requests, pass them to Elasticsearch, and render the results into visualizations. There is almost no work that the Kibana server does when this happens, so giving more CPU or RAM to Kibana won't help.
There are a few things that could potentially lead to this:
- Network slowness between ES and Kibana, or the Kibana server and the client browser. This seems most likely to me because of the huge gap between query duration and request duration.
- I suppose if you have saved searches in your dashboard, and the documents have a large number of fields, you might be running into this issue: [Discover] Discover times out when there's a lot of fields (b/c of highlight) · Issue #12131 · elastic/kibana · GitHub (related Discuss issue: Kibana very slow)
When you're looking at "query duration," what this should mean is looking at the "took" times for each result in the _msearch. Then I assume that "request duration" is the round-trip time for the _msearch, which you can find in the browser's dev tools. Here is an example measurement:
The took times total up to 19479ms.
When we look at the timing for this request, it turns out to be 11.57 seconds.
How can the request time be less than the total took times? The answer is because ES processes queries in the _msearch in parallel, so what we're primarily worried about is the timing for the one that took the longest, which for me is 11.37 seconds. That isn't too far away from the 11.57 seconds of the total request. Even though average took time is 2.1 seconds, the request waits on the full payload, so average took time isn't very useful.
If you could share some information like this in this thread, it would be really helpful to understand better what kind of problem we're facing.