Hi all,
I have a question about how kibana creates it's filters and performs searches on elastic.
If I pick a filter criteria from the Top5 selection in the discovery view (last 15 minutes), the search kibana (and elastic) performs is really quick and without any performance impact on my cluster.
If I create the same filter by hand (Add filter -> user.name (ECS base schema) is "xyz") the search doesn't finish within 30 seconds, times out in Kibana and sends my elastics nodes to 100% CPU load for a couple of minutes. I already checked what's happening on ES side via hot_threads and all nodes are eating up the resources for search-threads.
So, what is the difference between selecting a filter from the Top5 selection vs. creating it by hand? It doesn't matter whether I select only five minutes timeframe for this search or a full day. If there shouldn't be a difference, can someone point me in the right direction to analyse this further?
Some background info for the data in my index: Currently I ingest logs from our global company wide proxy soultion with roughly 150mio documents per day in a daily index. The data is normalized to the Elastic Common Schema. The number of different terms for user.name should be somewhere between 40k-50k.
Furthermore, although probably more of an elastic related question, any suggestions how to optimize here? Would it potentially make sense to sort the data in my index by timestamp. Since everything in there will be logs from different sources, it is very likely almost every query will be focused on data from the last x minutes to days.
Best regards,
Sebastian