Kibana is going slow

[jdbcplusnet]

Hi all

We have some performance issues with Kibana and Elasticsearch.

Both Elasticsearch and Kibana are 6.8.7 version. We have Elasticsearch Cluster with 3 nodes, one master and two slaves, 8GB for each node.

We have two Kibanas running with 1.4GB (default) and multiple Dashboards for showing real-time data every 30 seconds.

The first one is going slow and Dashboard are not refreshing ok, but the second one is working fine.

How we could know the numbers of concurrent users or improve Kibana responses from Elasticsearch?

[Wave (Andrew Sullivan)]

Hi @jdbcplusnet,
Take a look in Stack Monitoring | Overview. That should show you your search rate, and your search latency. That should give you an idea if your searches are the bottle neck. Then jump over to the Nodes tab and look at the cpu, load average, and jvm heap usage for all the nodes. See if anything looks off with those. You can click into the node for more detailed information. Finally, hopefully your kibana instances are in Stack Monitoring as well. If they are click into them and see if you see any performance problems. You might need to increase resources at any of the points mentioned above. Two three thoughts:

1. If your shards are too large (>100GB) they can slow down query performance. Ideally shot for around 50GB/shard.
2. Sometimes the client that is being used can be the bottleneck. We had a raspberry pi reloading a dashboard a long time ago and eventually had to stop because it just couldn't handle the rendering load.
3. Also, as you know 6.8.7 is EOL (end of life) so upgrading to the latest supported version you can is recommended.

[jdbcplusnet]

Thanks Andrew

Could you explain further the point 2 "Sometimes the client that is being used can be the bottleneck".

We have Kibana Dashboard / Visualizations refreshing Elasticsearch data each 30 seconds and several Spring-Boot microservices writing / reading to Elasticsearch.

Best regards.

[warkolm (Mark Walkom)]

What is the output from the _cluster/stats?pretty&human API?

@jdbcplusnet

Both Elasticsearch and Kibana are 6.8.7 version

Please note that version is EOL and no longer supported, you should be looking to upgrade as a matter of urgency.

@jdbcplusnet

one master and two slaves

Elasticsearch doesn't use that terminology. You can have a master node, but all others are nodes in the cluster.

[jdbcplusnet]

This is the ouput of _cluster/health?pretty=true&human

{
  "cluster_name" : "my_cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 151,
  "active_shards" : 302,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue" : "0s",
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent" : "100.0%",
  "active_shards_percent_as_number" : 100.0
}

[Wave (Andrew Sullivan)]

@jdbcplusnet

Sometimes the client that is being used can be the bottleneck

Sure, this is more antidotal, but about 4 years ago we had a raspberry-pi (a) loading a pretty big dashboard every 30 seconds or so as well. As you can imagine the little computer running wasn't really powerful enough to handle that. Client in this case means web browser. It had nothing really to do with the dashboard (which returned results quickly when viewing on a more traditional computer). Just trying to say if you have a crazy big dashboard and are using an older machine like a raspberry pi (version a) you might see degraded performance, but that has to do with the computer more than anything else.

[jdbcplusnet]

Thanks Andrew

It seems our Cluster Elasticsearch is working ok, but Kibana is not responding in time.

So we have increase Thread Pool from 4 to 16 as described here:

Increase Node JS Performance With Libuv Thread Pool - DEV Community

Is another way to increase Kibana (NodeJS) thread pool or Kibana performance?

Best regards

[Wave (Andrew Sullivan)]

I can't really speak to the node js idea, but am curious what the numbers look like in Stack Monitoring when these slow queries are running. Could you provide that information? Final thought would be look at what elastic says about running kibana in production environments.

[jdbcplusnet]

Hi Andrew

This is snapshot of Kibana monitoring:

We observed that only 300Mb are really used by Kibana (1.4GB available)

[Wave (Andrew Sullivan)]

Nice, that's good information. Thanks! Seems like there is a good relationship between HTTP connections and client response time. Doesn't look like kibana memory is a factor. Next step I think would be to see what the Cluster Overview looks like during the same time period. Like to see what the four big metrics look like: Search Rate, Search Latency, Indexing Rate, Indexing Latency. If there are no smoking guns there. We'll then want to look at each node in the cluster for that time period (from the Nodes tab in Stack Monitoring). Basically want to make sure all the nodes in the cluster have all the resources they need. Maybe there is one node that has a constraint and it is slowing down when the load goes up.

[jdbcplusnet]

Hi Andrew

We have moved some JVM processes running in the same machine with Elasticsearch master node and got better performance metrics.

[Wave (Andrew Sullivan)]

Ok, so don't see anything too off about those statistics, but a few thoughts:

1. It looks like you are running 3 nodes, so your master and data nodes are running on the same hosts. If so you should consider separating those out and let your master nodes be fully dedicated to that purpose.
2. Besides the previous advice above I'd take a look at slow logs as well to see if that turns anything up.