I think I have experienced this as well. If I call /_msearch
with the same query Kibana uses, it takes 5000ms. If I remove the "highlight":...
part of the query, it returns in 100ms or less.
I tested on Elasticsearch 6.2.3, auditbeat 6.2.4 (provides the template).
Original query:
{"index":["infosec-auditbeat*"],"ignore_unavailable":true,"preference":1524525012740}
{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"5m","time_zone":"America/Los_Angeles","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp"],"query":{"bool":{"must":[{"query_string":{"query":"connect","analyze_wildcard":true,"default_field":"*"}},{"match_phrase":{"beat.hostname":{"query":"auditbeat-8mm7k"}}},{"range":{"@timestamp":{"gte":1524524346418,"lte":1524538746418,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}
{"responses":[{"took":5267,...}
And removing the highlight part:
{"index":["infosec-auditbeat*"],"ignore_unavailable":true,"preference":1524525012740}
{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"5m","time_zone":"America/Los_Angeles","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp"],"query":{"bool":{"must":[{"query_string":{"query":"connect","analyze_wildcard":true,"default_field":"*"}},{"match_phrase":{"beat.hostname":{"query":"auditbeat-8mm7k"}}},{"range":{"@timestamp":{"gte":1524524346418,"lte":1524538746418,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}}}
{"responses":[{"took":25,...}
As for my specific data:
GET /_cat/indices/infosec-auditbeat*
green open infosec-auditbeat-6.2.4-2018.04.24 7rGmI3A7T9anmVCphrflFw 5 1 713229 0 638.5mb 321mb
green open infosec-auditbeat-6.2.4-2018.04.23 YmY2OHc3RIOlkR1Xh1d0eA 5 1 329153 0 372.1mb 186.5mb
My mapping is moderate in size, GET /infosec-auditbeat*/_mapping
(two indices) returns a JSON object which, when pretty-printed, is 2660 lines. This is the default auditbeat index template except for the index name changed.