Suggestions for Kibana slowness with tens of millions of records

Hello Elastic Community. I have created an ELK cluster with 5 boxes, and on box 5 one instance of Kibana is running. All 5 boxes are ingesting with one of the nodes being a master node. We have filebeat, auditbeat and metricbeat running. Through Logstash we are parsing 6-7 million records per day, and at times 1 million records every 30 minutes.

It is necessary for us to be able to move quickly in Kibana with Discovery and looking at data, retrieving data and hydrating visualizations, but I am noticing more and more sluggishness, beach-ball while fetching and timeout errors.

What are some solutions for making Kibana lightning fast under this very large stack? I heard that you can make a node a controller only node and not ingest, and I also read that you can create multiple Kibana instances or something called load-balancing requests?

Could some advanced Kibana users out there give us some possibly strategy tips to either add instances or configure instances in a way that makes the GUI far more faster? We are still about to stand up packetbeat on a bare metal box with mirror porting, so our data ingest is only going to get larger. Thanks everyone!

Hi Rachel,

For the most part Kibana uses Elasticsearch for all the heavy lifting. Since you mention that you noticed the sluggishness as you started adding more data I'm assuming that it's not Kibana that's becoming slower but Elasticsearch. If that's the case adding more instances of Kibana behind a load-balancer are unlikely to help.

There is general advice for improving query performance here https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-search-speed.html the first point "Give memory to the filesystem cache" is probably the best place to start.

You can also experiment with adding more replicas to your data indices. This could improve the query performance although it will reduce the indexing performance. Since you don't mention any performance problems ingesting data adding a small performance penalty at ingest time is probably worth better performance when querying but you will have to balance these two.

Thank you Rudolf. Much appreciated. We will begin researching your suggestions.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.