Discover view is very slow

Elastic Stack Kibana
1

Hi,
we are using Kibana to browse our logs generated by filebeat and logstash.

Currently we have ~120.000.000 log records, indexed with daily indices. Each index has replica=2 and shards=1. An index is about 300-500 mb in size and contains ~1.000.000 records.

When opening the discover view of kibana it takes about 20 seconds to complete the _msearch HTTP request.

When executing this query manually I can verify that the time is spent at waiting for elasticsearch to return the result of the request.

Sample request:
{"index":["logstash-2017.07.04"],"ignore_unavailable":true,"preference":1499194753566}
{"size":5000,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647},"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30m","time_zone":"Europe/Berlin","min_doc_count":1}}},"query":{"bool":{"must":[{"match_all":{}},{"range":{"@timestamp":{"gte":1499108623749,"lte":1499195023750,"format":"epoch_millis"}}}],"must_not":[]}},"stored_fields":[""],"script_fields":{},"docvalue_fields":["@timestamp"]}

When removing the following part of the query the result is returned much faster.
"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30m","time_zone":"Europe/Berlin","min_doc_count":1}}}

I suspect that this part generates the histogram above the results. Is there any possibility to speed this up?

What also improves the performance is to reduce the "size" part in the query to e.g. "500" (option discover:sampleSize). But this limits the discover view to only show the last x records. Kibana is already loading more records on demand (when scrolling to the end), but only up to discover:sampleSize records. So I would like to set this value to a high one (e.g. 5000). Is there any possibility to increase the value without reducing the query performance?

Thanks for your help!
Jakob

2

Hi @foxylion,

there are several aspects to this.

We are aware that the discover view performance is far from acceptable and improving that is on my personal todo list. Aside from the rendering performance your cluster might also lack the resources to efficiently aggregate across that many documents. The query profiler included in the free tier of x-pack can help with diagnosing such performance bottlenecks. In addition, the single shard setting might prevent your cluster from taking advantage of the parallelism afforded by having multiple nodes with multiple CPUs.

As for hiding the histogram in discover, it has been requested before and is tracked on Github. Please feel free to add a comment describing your situation to the issue to improve our ability to assess its priority.

3

Hi Felix,
thanks for your reply. Is there an issue which tracks the performance improvements of the discover view? I would like to follow the progress, because that seems like the best solution.

Unless then I will try to increase the shards per index, maybe this will improve the overall query performance.

4

Here you go: https://github.com/elastic/kibana/issues/12665

Please don't hesitate to add comments describing your current situation and your requirements to that issue.

5

Thanks Felix,
I commented on the issue. :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.