Kibana 5.1 keeps crashing

(Shantanu Deshmukh) #1

We have a single node of Kibana 5.1 and two nodes of Elasticsearch 5.1. It has been deployed in production and just one admin uses the console. So there's hadly any load. Just a few queries are performed in a day as of now.

Still, even in this breezy situation Kibana keeps on crashing. I have enabled all logging options
logging.verbose: true
logging,silent: false
logging.quiet: false

Before crashing Kibana logged a warning that it's "Unable to revive connection" to elastic search node. And another warning saying "No living connections". That was the last log after which it had crashed.

As for configuration: , Kibana is running on SSL. Other than this no other configurations have been added.

Please help.

(Brandon Kobel) #2

@Shantanu_Deshmukh which OS are you running Kibana on? Do you see the memory increasing over time? Could you attach the full log here?

(Shantanu Deshmukh) #3

Hey Brandon,

We are running on RHEL 7.2. Please refer to screenshot of the log attached here

Thanks in advance.

(Brandon Kobel) #4

@Shantanu_Deshmukh I assume you're running Kibana as a service in RHEL 7.2? Do the syslogs have any more information regarding the crash? Are you running Elasticsearch on the same server as Kibana?

(Shantanu Deshmukh) #5

No, we are running Kibana directly using sh bin/ I do not know so much about system level so can you tell me about advantages of running Kibana as service rather than this way? About syslogs, I will have to check with our OS admin and let you know soon.
And no, we are not running ES on same server. We have 2 separate servers for ES

(Brandon Kobel) #6

@Shantanu_Deshmukh The syslogs are only relevant if you're running kibana as a service, if you're launching the process manually using /bin/ then don't worry about looking into that.

Are you certain that the last thing that Kibana prints to the terminal is about "No living connections"? If Kibana is crashing, normally there would be an error or an exit code printed to the terminal.

The "No living connections" log message is displayed when the network connection between Kibana and Elasticsearch is lost, and that alone shouldn't be causing Kibana to exit/crash.

(Shantanu Deshmukh) #7

Yes that is the last message. And I am also wondering about why Kibana isn't printing fatal error information. I have enabled all logging options in config file. Do I have to add some other setting also? I also tried redirecting console output to a log file by doing `sh bin/kibana >> /opt/kibana_logs/console.log. This also didn't show any logs.

(Brandon Kobel) #8

@Shantanu_Deshmukh using sh bin/kibana >> /opt/kibana_logs/console.log will only redirect stdout to the file, if you use sh bin/kibana >> /opt/kibana_logs/console.log 2>&1 you'll get both the stdout and the stderr written to the file.

If you set the following in your kibana.yml, we should get additional information that could help diagnose the issue logging.verbose: true

(Shantanu Deshmukh) #9

Hey Brandon, thanks for that piece of info. Will definitely try it out and post progress here.:+1:

(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.