We've recently noticed a significant degradation in the search performance of our ES 5.4.1 cluster - when using Kibana, going to "Discover" and asking to display the last 15 minutes of data from a specific index, half the time we are getting a Request Timeout error after 30 seconds.
We've narrowed down the issue to a specific index that seems to be the slowest, and when running Kibana's query manually it is evident that the slow part is the highlight query.
To give you some numbers, running the Kibana Discover query for the last 15m as-is (incl. highlight) takes usually over 20 seconds (when it doesn't timeout) to return ~2,500 documents.
Removing the "highlight" section brings the query execution time to under 1 second.
Does anyone have insight into what is causing the highlight query to be so slow?