Error loading kibana on a web browser - 503 service unavailable

prajwal93 · January 8, 2021, 9:23am

I have an ELK cluster running and I have trouble loading Kibana in my web browser. It was working fine until 3 days before I made a change to logstash configuration. I installed Logstash-amazon_es plugin, used to load data to Amazon Elasticsearch cluster. All the services are up and running fine without errors. I have checked the logs and could not find any errors. I have traced back my steps and uninstalled all the changes made, but yet I am not able to resolve this Kibana errors.

About the ELK stack configuration:

A cluster of 4 nodes
Kibana is running behind a Nginx reverse proxy
The entire cluster is accessible inside a VPN and have an another Nginx proxy running
Kibana version - 6.8.13
Logstash version - 6.8.13
Elasticsearch version - 6.8.13

Errors I see in browser

Any help troubleshooting this is much appreciated!

warkolm · January 11, 2021, 12:17am

Welcome to our community!

What do your Kibana logs show?

prajwal93 · January 11, 2021, 4:38am

Hi Mark,

Thanks for the response.
I was able to resolve the issue.

The problem was, Kibana in one of nodes was down - it had problem connecting to elasticsearch.
I had to change the following in /etc/kibana/kibana.yml
elasticsearch.ssl.verificationMode: certificate to elasticsearch.ssl.verificationMode: none
However, I have the same configuration across all nodes and there was no problem with other nodes.

I was not able to figure out why. Can you please help?

Thanks

warkolm · January 11, 2021, 4:40am

We'd need to see the Kibana logs.

prajwal93 · January 11, 2021, 4:53am

{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:elasticsearch@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["license","debug","xpack"],"pid":16326,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["error","elasticsearch","data"],"pid":16326,"message":"Request error, retrying\nGET https://localhost:9200/_xpack => self signed certificate"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["license","warning","xpack"],"pid":16326,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:xpack_main@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:graph@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:spaces@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:security@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:searchprofiler@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:ml@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:tilemap@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:watcher@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:grokdebugger@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:logstash@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:beats_management@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:maps@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:index_management@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:index_lifecycle_management@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:rollup@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:remote_clusters@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:32Z","tags":["status","plugin:cross_cluster_replication@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","browser-driver","warning"],"pid":16326,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","browser-driver","debug"],"pid":16326,"message":"Browser installed at /var/lib/kibana/headless_shell-linux/headless_shell"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","debug"],"pid":16326,"message":"Browser type: chromium"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","debug"],"pid":16326,"message":"Chromium sandbox disabled: true"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["status","plugin:reporting@6.8.13","error"],"pid":16326,"state":"red","message":"Status changed from uninitialized to red - No Living connections","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","worker","debug"],"pid":16326,"message":"CSV: Registering CSV worker"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","esqueue","worker","debug"],"pid":16326,"message":"kjs2cbmq0clid669e36hp36w - Created worker for job type csv"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","worker","debug"],"pid":16326,"message":"PNG: Registering PNG worker"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","esqueue","worker","debug"],"pid":16326,"message":"kjs2cbms0clid669e36iylvi - Created worker for job type PNG"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","worker","debug"],"pid":16326,"message":"PDF: Registering PDF worker"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","esqueue","worker","debug"],"pid":16326,"message":"kjs2cbmt0clid669e36jtor8 - Created worker for job type printable_pdf"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["reporting","debug"],"pid":16326,"message":"Running on os \"linux\", distribution \"Centos\", release \"7.9.2009\""}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","data"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","elasticsearch","admin"],"pid":16326,"message":"No living connections"}
{"type":"log","@timestamp":"2021-01-11T04:25:33Z","tags":["warning","task_manager"],"pid":16326,"message":"PollError No Living connections"}

prajwal93 · January 12, 2021, 10:03am

I forgot to point out that the changes I made( elasticsearch.ssl.verificationMode: certificate to elasticsearch.ssl.verificationMode: none) was only on one node, the other three nodes have elasticsearch.ssl.verificationMode: certificate configuration

system · February 9, 2021, 10:03am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Connection Failed : Your elasticsearch server is down or unreachable Elasticsearch	2	2775	July 5, 2017
Cannot connect to the Elasticsearch cluster Kibana	3	3566	July 10, 2019
Kibana will not connect to elasticsearch Kibana	2	381	February 8, 2024
Failed to load indices and failed to load cluster Kibana	5	611	May 4, 2021
Kibana is throwing 503 error Kibana	2	2941	May 7, 2019

Error loading kibana on a web browser - 503 service unavailable

Related topics