Hi Team,
I have been running ELK Cluster of version 7.8 with 3 master nodes and 5 data nodes. I am running this on RHEL 7.5. I have installed httpd to proxy the requests to Kibana. I installed httpd, Kibana, Elasticsearch and Logstash (with groks) on each of the 3 master nodes. We have been running from past 2 months in production for pushing postfix maillog using filebeat.
I have recently encountered the error "Kibana is not yet ready" on the browser. I drilled and checked for the kibana logs (as I see all the elasticsearch services are working fine and able to retrieve data through CURL). I observed the error message as "Unable to connect to Elasticsearch. Error: [search_phase_execution_exception] all shards failed".
I am not sure where exactly the issue might be and how it could be resolved. I could notice that the index ".kibana_task_manager_1" is showing RED and the shards of that index are unassigned.
I am not sure if the above error is related to our issue. How to resolve this without data loss of other indices.