We have a single node of Kibana 5.1 and two nodes of Elasticsearch 5.1. It has been deployed in production and just one admin uses the console. So there's hadly any load. Just a few queries are performed in a day as of now.
Still, even in this breezy situation Kibana keeps on crashing. I have enabled all logging options
logging.verbose: true
logging,silent: false
logging.quiet: false
Before crashing Kibana logged a warning that it's "Unable to revive connection" to elastic search node. And another warning saying "No living connections". That was the last log after which it had crashed.
As for configuration: , Kibana is running on SSL. Other than this no other configurations have been added.
@Shantanu_Deshmukh I assume you're running Kibana as a service in RHEL 7.2? Do the syslogs have any more information regarding the crash? Are you running Elasticsearch on the same server as Kibana?
No, we are running Kibana directly using sh bin/kibana.sh. I do not know so much about system level so can you tell me about advantages of running Kibana as service rather than this way? About syslogs, I will have to check with our OS admin and let you know soon.
And no, we are not running ES on same server. We have 2 separate servers for ES
@Shantanu_Deshmukh The syslogs are only relevant if you're running kibana as a service, if you're launching the process manually using /bin/kibana.sh then don't worry about looking into that.
Are you certain that the last thing that Kibana prints to the terminal is about "No living connections"? If Kibana is crashing, normally there would be an error or an exit code printed to the terminal.
The "No living connections" log message is displayed when the network connection between Kibana and Elasticsearch is lost, and that alone shouldn't be causing Kibana to exit/crash.
Yes that is the last message. And I am also wondering about why Kibana isn't printing fatal error information. I have enabled all logging options in config file. Do I have to add some other setting also? I also tried redirecting console output to a log file by doing `sh bin/kibana >> /opt/kibana_logs/console.log. This also didn't show any logs.
@Shantanu_Deshmukh using sh bin/kibana >> /opt/kibana_logs/console.log will only redirect stdout to the file, if you use sh bin/kibana >> /opt/kibana_logs/console.log 2>&1 you'll get both the stdout and the stderr written to the file.
If you set the following in your kibana.yml, we should get additional information that could help diagnose the issue logging.verbose: true
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
