Elasticsearch cluster performance tuning

[AClerk]

Hi,
Following an error I got in Kibana and the following posts

Kibana looks like it cancels requests if they take more than 30 seconds?

I was running the newest version of Kibana, 7.6. I was noticing that during ingestion process, some of my visualizations are taking over 30 seconds. When they take that long it will expose the error: [esaggs] > Unexpected token u in JSON at position 0 When looking at the network request, it says cancelled. I think that it auto-cancels requests but it seems that the timeout will occur around the 30 second mark. I have some visualizations for the last 2 days, pretty much monitoring the count…

Issue after upgrading to 7.5.2 - SyntaxError: Unexpected token u in JSON at position 0 at JSON.parse (<anonymous>) at server.route.handler

Hello, Today we have performed an upgrate to Kiana 7.5.2 (ES at 7.5) and we have noticed an issue with one of our indices which was working fine. It seems like this particular index pattern is returning an error. Other seems to be working fine. { "type":"error", "@timestamp":"2020-01-27T08:58:44Z", "tags":[ ], "pid":23965, "level":"error", "error":{ "message":"Unexpected token u in JSON at position 0", "name":"SyntaxError", "stack":"SyntaxError: Unexp…

How can I optimise and tune the elastic cluster?

• Elasticsearch/Kibana version are 7.6.2
• I have 5 nodes in the cluster, all run on a CentOS 7 VMs
• 28G per node for JVM
• File swap is disabled on all nodes
• Using shards calculation according to this article.

Thanks!

[Christian_Dahlqvist]

I have not seen the article you linked to before. How many indices and shards do you have? What is the average size?

[AClerk]

Hi,
Currently I have
~ 700 indices
~ 2240 shards
~ 53GB Average size (primary+replica)

Thanks!

[Christian_Dahlqvist]

What issues are you having? What is it you want to tune for?

[AClerk]

Currently Kibana is hitting a timeout exception/error.
In the first link I gave, the user named @flash1293, advises to

Maybe it is possible to tune your server a bit without adding hardware. For guidance around this you can post in the ES forum and I'm sure you will find help.

So I am trying to understand if there is any further optimisation I can do on the cluster?!

[Christian_Dahlqvist]

The first thing to do is identify what is causing the timeouts.

If you look in the Elasticsearch logs, do you see indications of long and/or frequent GC? You have a lot of data on each node so it is possible that you are suffering from heap pressure.

What does CPU usage look like while you have queries timing out? Is any node saturated?

The last thing that commonly causes timeouts is issues around storage performance. Are you monitoring disk utilisation and iowait? What does disk usage look like during query timeouts?

[AClerk]

Hi @Christian_Dahlqvist
You have good questions. Unfortunately, I cannot answer them right now as I am not monitoring all those metrics. It will require some more time/effort.

In the short run, I am planning to monitor query times.
In the longer run, I would sure want to monitor more metrics and understand what is affecting the performance.

Cheers

[AClerk]

Hi @Christian_Dahlqvist
Any recommendations regarding Monitor queries running time?
What is the best way to do it?
Cheers!

[Christian_Dahlqvist]

Do you have Elasticsearch monitoring enabled?

[AClerk]

What do you mean?
Not sure how to answer.

Thanks!

[Christian_Dahlqvist]

Have a look at the docs around monitoring.

[AClerk]

currently, monitoring with filebeat and metricbeat.
trying to configure packetbeat for query runtime.