Kibana shuts down in elasticsearch clustering with master not discovered exception

Elastic Stack Kibana
1

Kibana stopped with below logs. I dont see any error in OS logs

{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["server","uuid","uuid"],"pid":14524,"message":"Resuming persistent Kibana instance UUID: 9087ad45-53b9-48d0-9f0f-f107d12f5494"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:elasticsearch@7.4.2-SNAPSHOT","info"],"pid":14524,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["license","debug","xpack"],"pid":14524,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":14524,"message":"Starting worker..."}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":14524,"message":"Polling for reindex operations"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:xpack_main@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:tilemap@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:watcher@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:logstash@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:beats_management@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:index_management@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:index_lifecycle_management@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:rollup@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:remote_clusters@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:cross_cluster_replication@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:file_upload@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:snapshot_restore@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:57Z","tags":["status","plugin:security@7.4.2","error"],"pid":14524,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-10-16T08:56:59Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:01Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:01Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:02Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:02Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:02Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:05Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:06Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:06Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:07Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:07Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:07Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:10Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:11Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:11Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:12Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:12Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:12Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:15Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:16Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:16Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:17Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:17Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:17Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:20Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:21Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:21Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:22Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:22Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:22Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:25Z","tags":["plugin","debug"],"pid":14524,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-10-16T08:57:26Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:26Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:27Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:27Z","tags":["debug","monitoring","kibana-monitoring"],"pid":14524,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2020-10-16T08:57:27Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":14524,"message":"Stopping worker..."}
{"type":"log","@timestamp":"2020-10-16T08:57:27Z","tags":["debug","root"],"pid":14524,"message":"shutting root down"}

I have another error in kibana error logs

FATAL [master_not_discovered_exception] null :: {"path":"/.kibana_task_manager","query":{},"statusCode":503,"response":"{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}"}

Will KIbana shuts down for above error?

2

Was it running, then suddenly shut down, or didn't even start? Usually the cause of this error is that the ES nodes can't ping each other on the expected port in one direction, the other or both. If the ES server isn't operational, this would be more of an Elasticsearch question in Discuss.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.