Performance impact on latest git version

Elastic Stack Kibana
1

Hi all,
i have here a large Desktop to analyze flow traffic in our network.
On k3 i get this data (refresh) in 5-sec. and max. 180%CPU load, but with k4 in 73-sec. an 600 - 700% CPU load.

CPU load with k4:

this is an 8 core Server with 48GB RAM (24GB of RAM for ES 1.6), openJDK 1.8
How can i solve the issues ?
If someone both dashboards needed, please notify.

2

More detail would be useful... How similar are the dashboards / queries between Kibana 3 and Kibana 4?

3

Hi Tanya,
thanks for your help here.
The queries are the same. This is almost a one to one copy of a K3 Dashboard.
I think i have found the cause, but not why. In the K4 Dashboard there is a "tile map" as soon this is involved i get the high CPU load. However the interesting thing, if i test this "tile map" alone i get the data within seconds.
In addition, i see from time to time (not often and only since K4) this error in ES.

EsRejectedExecutionException: rejected execution (queue capacity 4000)

Why is the difference in the ES queue management between K3 and K4 so different?

4

It is best even if you find the problem itself :smile:
After i changed the index pattern from "logstash-*" to "[logstash-]YYYY.MM.DD ", all is well again.

5

The increase is due to seeing a more "complete" view of your geo data. Kibana 3 supported Elasticsearch 0.90 which did not have geo aggregations, so Kibana simply sucked 1000 values into the browser and made the best of it.

In Kibana 4 we use Elasticsearch's support for aggregating geo data, so the results represent much more than 1000 events Kibana 3 considered. The reason changing the pattern improves the situation is that Kibana can optimize which indices it looks at, thus reducing the need for Elasticsearch to load data that would just be filtered out later.