I have an AWS instance with Kibana and ES on it set up as a client node, and is part of a 3-node ES cluster. Kibana's connection bounces, that is, goes from green->red->gren.... and I see these kibana logs
{"type":"log","@timestamp":"2016-11-07T14:23:36+00:00","tags":["status","plugin:elasticsearch","error"],"pid":30956,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2016-11-07T14:24:00+00:00","tags":["status","plugin:elasticsearch","info"],"pid":30956,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
Yes, I'm connecting to the ES node configured in kibana.yml. In my case, I had to regenerate my certs and restart the services, and things seem to be stable now.
{"type":"log","@timestamp":"2016-12-05T16:27:29+00:00","tags":["status","plugin:elasticsearch","error"],"pid":570,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2016-12-05T16:27:37+00:00","tags":["status","plugin:elasticsearch","info"],"pid":570,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
How can I debug this further? Note that I can see my logs in Kibana if I reload the Kibana page enough, so my Kibana/Elasticsearch connection (sometimes) works.
My instance has 16G memory, and ES is setup to use 8G. This instance has ES running in client mode, and also has Logstash and Kibana installed on it.
# free -m
total used free shared buff/cache available
Mem: 15629 13767 884 24 977 1422
Swap: 0 0 0
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.