Kibana intermittently losing connectivity to Elasticsearch

[aidofitz (Adrian F)]

Hi,

Am seeing an issue in my v7.8 cluster with 16 x Kibana nodes, 16 x Elastic nodes. Kibana nodes keep reporting status as red, with error as in screenshot.

However, my Elastic cluster seems to be fine and I can test connections to it no problem. Other Kibana nodes are working okay at the same time. The affected Kibana nodes recover by themselves eventually but this can take hours.

Logs (below) don't really give any clue about what the problem is. I'm wondering if perhaps the affected node(s) are getting some temporary intermittent issue with connection to Elastic, but then not retrying or failing over to another node like they should.

Any suggestions what might be the problem welcome! Any settings I should look at to improve the situation? Thanks!

• Adrian

{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["status","plugin:xpack_main@7.8.0","error"],"pid":6,"state":"red","message":"Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes.","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["status","plugin:reporting@7.8.0","error"],"pid":6,"state":"red","message":"Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes.","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["status","plugin:beats_management@7.8.0","error"],"pid":6,"state":"red","message":"Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes.","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["status","plugin:elasticsearch@7.8.0","error"],"pid":6,"state":"red","message":"Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes.","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["status","plugin:xpack_main@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["status","plugin:reporting@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2022-01-06T18:25:09Z","tags":["status","plugin:beats_management@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to retrieve version information from Elasticsearch nodes."}

[warkolm (Mark Walkom)]

7.8 is EOL, so you should really upgrade.

However, why do you have so many Kibana instances?

[aidofitz (Adrian F)]

Hi,

Thanks, yes, we are planning a upgrade in the next month or so; however im trying to stabilise the existing system in the interim.

As to why 16 kibana nodes, primarily just for convenience in configuration. We have a server template we use that installs both an ES instance and a Kib instance on each server - simpler than having two separate install processes. I.e. we have 16 phyical machines underlying. Somewhat a lazy approach admittedly, but not sure it would be contributing to this issue?

thx!