Recently we migrated some of our test environments from ELK 5.1.1 to Elasticsearch 6.3.0, Kibana 6.3.0 and keep Logstash v 5.1.1.
Now we have significant search performance degradation in comparing to ELK5.1.1. Especially it's very noticeable for search queries with many filters. Elasticsearch settings and instance types are the same for 5.1.1 and 6.3.0. Amount of data is the same.
Query can have up to 40 different filters. And for 5.1.1 search for last 4 hours takes 3 seconds, for 6.3.0 it takes 15 seconds. With increasing time range search time for 6.3.0 growths exponentially. For last 8, 12 or 24 hours query for 5.1.1 still takes 3-5 seconds. For 6.3.0 it's ~130 seconds for 8 hours and more than 5 minutes for 12 and 24 hours. We increased all timeout to 5 minutes, but it's not enough to get query result.
Example of saved search:
{
"index": "logstash-",
"filter": [],
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"": {}
},
"require_field_match": false,
"fragment_size": 2147483647
},
"query": {
"query_string": {
"query": "!"Error sending SMS" AND !"pid:" AND !"app.js" AND !"neighbours:" AND !"Wrong kind of auth" AND !"reductions:" AND !"messages_2018" AND !"get_org_and_acc" AND !"\"content_type" AND !"=SUPERVISOR REPORT====" AND !"=ERROR REPORT====" AND !"heap_size:" AND !"stack_size:" AND !"=CRASH REPORT====" AND !"ancestors:" AND !"Enabling test services"",
"analyze_wildcard": true
}
}
}