"Unable to connect to Elasticsearch. Error: [search_phase_execution_exception] all shards failed"

Hi Team,

I have been running ELK Cluster of version 7.8 with 3 master nodes and 5 data nodes. I am running this on RHEL 7.5. I have installed httpd to proxy the requests to Kibana. I installed httpd, Kibana, Elasticsearch and Logstash (with groks) on each of the 3 master nodes. We have been running from past 2 months in production for pushing postfix maillog using filebeat.

I have recently encountered the error "Kibana is not yet ready" on the browser. I drilled and checked for the kibana logs (as I see all the elasticsearch services are working fine and able to retrieve data through CURL). I observed the error message as "Unable to connect to Elasticsearch. Error: [search_phase_execution_exception] all shards failed".

I am not sure where exactly the issue might be and how it could be resolved. I could notice that the index ".kibana_task_manager_1" is showing RED and the shards of that index are unassigned.

I am not sure if the above error is related to our issue. How to resolve this without data loss of other indices.

Please share the full log you are seeing.

Also check your Elasticsearch logs.