@timestamp term tables make dashboard very slow

[Sjaak01 (Sjaak)]

Hi,

I have a bunch of table visualizations that rely on a @timestamp term to show a list of events. When selecting longer time ranges this absolutely kills dashboard performance.

Because of this some of my dashboards can take 2.5 minutes to load. Removing the tables makes loading much faster.

Below are the stats from a single visualization that takes 32 seconds to load but on my server (hosting both ES and Kibana) I see only a short CPU spike matching the query time. The complete query doesn't appear to take that long to run either so why does the browser need so long to render the visualization?

|Query Duration|25ms|
|Request Duration|2599ms|
|Hits|1622|

Another strange thing I noticed is that it appears loading time increases a lot when there is no data/little data after a certain time?

Loading times with a data set full of data over the past ~30 days and very little data after that. Performance is measured over several times.

Last 30 days
Loading time in browser: 6 ~ 8 seconds

|Query Duration|26ms|
|Request Duration|1508ms|
|Hits|719|

Last 60 days
Loading time in browser: ~30 seconds

|Query Duration|46ms|
|Request Duration|2007ms|
|Hits|1367|

Last 90 days
Loading time in browser: ~35 seconds

|Query Duration|24ms|
|Request Duration|2731ms|
|Hits|1622|

Despite the request duration being less than 3 seconds, it takes over 30 for a browser to render this single visualization. For some reason performance doesn't scale very well between 30 and 60 days either while performance between 60 and 90 days does seem to scale better.

No other queries were sent to the server while performing these tests.

Is there any way I can speed these visualizations up? I've changed some visualizations to data histrogram which speeds things up though its not an ideal situation. I've also tried limiting the amount of items that can get loaded when using a @timestamp term high enough so only 30 days is loaded (data is collected per hour) but because the descending option doesn't appear to be working in such a case it doesn't actually load the last 30 days worth of data so the returned data is unusable.

[tsullivan (Tim Sullivan)]

@Sjaak01

Despite the request duration being less than 3 seconds, it takes over 30 for a browser to render this single visualization. For some reason performance doesn't scale very well between 30 and 60 days either while performance between 60 and 90 days does seem to scale better.

You'll have to think about the number of aggregation buckets coming over the network and getting parsed in the browser. The Elasticsearch APIs return JSON data, which is parsed as a single string into Javascript objects in the browser. The Javascript runtime environment is a single thread (this is true as of the time writing this, because Kibana doesn't implement web workers), and parsing JSON is a CPU-intensive blocking operation.

So the more buckets in the Elasticsearch result, the larger the data string, and the longer it takes to parse. If you're using timestamp as the term to aggregate on, and you a document for every, say hour, a 90-day search would come up with about 129600 buckets. If you have sub-aggregations (split rows) the amount of data coming back is some multiple of that.

Along with that, Kibana proxies the entire response from Elasticsearch straight into the browser, to help with inspection. It doesn't filter or map the response to reduce the payload size.

It could be that you are trying to represent something like the actual raw documents in a table visualization. Have you thought of instead adding Saved Searches to your dashboard?

If Kibana has no built-in solutions that are appropriate for what you're trying to do, you could also think about building a custom visualization plugin that executes custom searches that using ES tools like source filtering, field collapsing, and composite aggregation to reduce the result data down to a more manageable size. If you can get your data with a query as opposed to an aggregation, or get it from a composite aggregation, the JSON data string will be much smaller as the values in the data will have way less depth.

[Sjaak01 (Sjaak)]

Thanks for the detailed explanation.

Yes I want to display the raw data. The tables I have are mostly meant to show log data so I want them ordered by timestamp. I'll have a look at the saved searched.

The Javascript runtime environment is a single thread (this is true as of the time writing this, because Kibana doesn't implement web workers), and parsing JSON is a CPU-intensive blocking operation.

So in theory a cpu with good single thread performance should load the data faster?

[tsullivan (Tim Sullivan)]

@Sjaak01

So in theory a cpu with good single thread performance should load the data faster?

In theory, yes. First check the response coming back from Elasticsearch is within reasonable time parameters, as well as the network response. If you find those don't seem to be a factor, then in theory a web browser with higher a faster CPU would be able to load the data faster.