Kibana Red timeout on healthy cluster

Elastic Stack Kibana
1

Hi running Kibana 6.2.4

Kibana was working fine and now I'm getting.

plugin:elasticsearch@6.2.4 Request Timeout after 30000ms

The cluster is 100% healthy. I also have another window open of the same Kibana UI but I have not refreshed the browser yet and I can see my dashboard executing just fine.

2

Everything seems to go green except the elasticsearch plugin?

{"type":"log","@timestamp":"2018-05-28T21:43:55Z","tags":["status","plugin:elasticsearch@6.2.4","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
...
{"type":"log","@timestamp":"2018-05-28T21:44:07Z","tags":["reporting","warning"],"pid":1,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
...
{"type":"log","@timestamp":"2018-05-28T21:44:37Z","tags":["status","plugin:elasticsearch@6.2.4","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["license","info","xpack"],"pid":1,"message":"Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active | expiry date: 2019-05-22T23:59:59+00:00"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["status","plugin:xpack_main@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["status","plugin:searchprofiler@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["status","plugin:ml@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["status","plugin:tilemap@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["status","plugin:watcher@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["status","plugin:graph@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["status","plugin:reporting@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-05-28T21:44:38Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":1,"message":"Stopping all Kibana monitoring collectors"}
{"type":"log","@timestamp":"2018-05-28T21:44:40Z","tags":["license","info","xpack"],"pid":1,"message":"Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active | expiry date: 2019-05-22T23:59:59+00:00"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["status","plugin:security@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["security","warning"],"pid":1,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
...
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["status","plugin:grokdebugger@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["status","plugin:dashboard_mode@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["status","plugin:logstash@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["status","plugin:apm@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["status","plugin:console@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["status","plugin:metrics@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-05-28T21:45:04Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
3

It's timing out when reaching out to Elasticsearch for it's health check. This is usually do to network issues or a cluster which can not respond in a timely manner (30 seconds)/

4

Cluster seemed healthy and _cat/health was instant

5

this is ok?

6

What check does Kibana run? If _cat/health was 100% and green what else would cause this?

7

Is Elasticsearch and Kibana on separate machines? This might be a networking issue.

Here is the health check which is ran: https://github.com/elastic/kibana/blob/master/src/core_plugins/elasticsearch/lib/health_check.js

Mostly it's similar to the following requests:

curl -X HEAD http://localhost:9200/

curl http://localhost:9200/_nodes/info

8

Yes, they are. Ok I'll keep that in mind next time it happens.

Thanks

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.