Performance issue while picking filter from Top 5 vs. by hand

#1

Hi all,

I have a question about how kibana creates it's filters and performs searches on elastic.

If I pick a filter criteria from the Top5 selection in the discovery view (last 15 minutes), the search kibana (and elastic) performs is really quick and without any performance impact on my cluster.

If I create the same filter by hand (Add filter -> user.name (ECS base schema) is "xyz") the search doesn't finish within 30 seconds, times out in Kibana and sends my elastics nodes to 100% CPU load for a couple of minutes. I already checked what's happening on ES side via hot_threads and all nodes are eating up the resources for search-threads.

So, what is the difference between selecting a filter from the Top5 selection vs. creating it by hand? It doesn't matter whether I select only five minutes timeframe for this search or a full day. If there shouldn't be a difference, can someone point me in the right direction to analyse this further?

Some background info for the data in my index: Currently I ingest logs from our global company wide proxy soultion with roughly 150mio documents per day in a daily index. The data is normalized to the Elastic Common Schema. The number of different terms for user.name should be somewhere between 40k-50k.

Furthermore, although probably more of an elastic related question, any suggestions how to optimize here? Would it potentially make sense to sort the data in my index by timestamp. Since everything in there will be logs from different sources, it is very likely almost every query will be focused on data from the last x minutes to days.

Best regards,
Sebastian

#2

Forgot to mention: The whole ELK stack is currently on version 7.2.

#3

The simple part of this question is how Kibana handles Discover. Discover is loading individual documents by recency, by default 500 per page. So when you see the Top 5 values in Discover, it is based on the sample you have loaded so far. The default Discover query sorts by the document creation time which is fast by default. When you add a filter, it is changing from a "match all documents" query to a filtered search query, and because that requires looking at more documents before returning, it might be slower or hitting resource limits.

I'm not an expert on Elasticsearch tuning, but I can point to our general recommendations:

https://www.elastic.co/guide/en/elasticsearch/reference/current/general-recommendations.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html

Also for time series data, you may want to consider using some kind of lifecycle policies for your indices to free up resources for newer data: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html

I'm sorry this is so general, if you want more specific performance advice you may need to identify what kind of bottlenecks you are seeing.

#4

Thanks for your help. Didn't think about the Top5 being based on the last 500 and therefore easy to calculate.

In the meantime I found the culprit for sending my cluster into high load. The suggested values in the filter editor. Those are build from all values in all indices (regardles how old) and therefore obviously take a lot of time to generate. There was already a similar topic here in the forum:

I disabled the suggested values feature in Kibana and now I can enter search terms without creating the high load situation.

I'm already using a lifecycle policy but since I want to have 90 days of data queriable that's already quite a lot of data.

Best regards,
Sebastian