Error loading kibana on a web browser - 503 service unavailable

[prajwal93]

I have an ELK cluster running and I have trouble loading Kibana in my web browser. It was working fine until 3 days before I made a change to logstash configuration. I installed Logstash-amazon_es plugin, used to load data to Amazon Elasticsearch cluster. All the services are up and running fine without errors. I have checked the logs and could not find any errors. I have traced back my steps and uninstalled all the changes made, but yet I am not able to resolve this Kibana errors.

About the ELK stack configuration:

1. A cluster of 4 nodes
2. Kibana is running behind a Nginx reverse proxy
3. The entire cluster is accessible inside a VPN and have an another Nginx proxy running
4. Kibana version - 6.8.13
5. Logstash version - 6.8.13
6. Elasticsearch version - 6.8.13

Errors I see in browser

Any help troubleshooting this is much appreciated!

[warkolm (Mark Walkom)]

Welcome to our community!

What do your Kibana logs show?

[prajwal93]

Hi Mark,

Thanks for the response.
I was able to resolve the issue.

The problem was, Kibana in one of nodes was down - it had problem connecting to elasticsearch.
I had to change the following in /etc/kibana/kibana.yml
elasticsearch.ssl.verificationMode: certificate to elasticsearch.ssl.verificationMode: none
However, I have the same configuration across all nodes and there was no problem with other nodes.

I was not able to figure out why. Can you please help?

Thanks

[warkolm (Mark Walkom)]

We'd need to see the Kibana logs.

[prajwal93]

{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:elasticsearch@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["license","debug","xpack"],"pid":16326,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["error","elasticsearch","data"],"pid":16326,"message":"Request error, retrying\nGET https://localhost:9200/_xpack => self signed certificate"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["license","warning","xpack"],"pid":16326,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:xpack_main@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:graph@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:spaces@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:security@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:searchprofiler@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:ml@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:tilemap@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:watcher@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:grokdebugger@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:logstash@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:beats_management@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:maps@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:index_management@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:index_lifecycle_management@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:rollup@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:remote_clusters@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:cross_cluster_replication@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","browser-driver","warning"],"pid":16326,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","browser-driver","debug"],"pid":16326,"message":"Browser installed at /var/lib/kibana/headless_shell-linux/headless_shell"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","debug"],"pid":16326,"message":"Browser type: chromium"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","debug"],"pid":16326,"message":"Chromium sandbox disabled: true"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["status","plugin:reporting@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from uninitialized to red - No Living connections","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","worker","debug"],"pid":16326,"message":"CSV: Registering CSV worker"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","esqueue","worker","debug"],"pid":16326,"message":"kjs2cbmq0clid669e36hp36w - Created worker for job type csv"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","worker","debug"],"pid":16326,"message":"PNG: Registering PNG worker"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","esqueue","worker","debug"],"pid":16326,"message":"kjs2cbms0clid669e36iylvi - Created worker for job type PNG"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","worker","debug"],"pid":16326,"message":"PDF: Registering PDF worker"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","esqueue","worker","debug"],"pid":16326,"message":"kjs2cbmt0clid669e36jtor8 - Created worker for job type printable_pdf"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","debug"],"pid":16326,"message":"Running on os \"linux\", distribution \"Centos\", release \"7.9.2009\""}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","task_manager"],"pid":16326,"message":"PollError No Living connections"}

[prajwal93]

I forgot to point out that the changes I made( elasticsearch.ssl.verificationMode: certificate to elasticsearch.ssl.verificationMode: none) was only on one node, the other three nodes have elasticsearch.ssl.verificationMode: certificate configuration