I have a 3-node Elasticsearch cluster. When I perform a rolling restart on the cluster, once the restart is completed, the Elasticsearch cluster status is coming as red. Even with multiple restarts I am unable to bring the cluster state to even yellow.
On checking with the Marvel plugin installed on all 3 of my Elasticsearch nodes, hovering over the i symbol next to the cluster status its showing this error:
.marvel-kibana has an unassigned primary
This problem is coming only for the .marvel-kibana index, the other indices there is no issue.
I am doing the following when doing the rolling restart:
- Disabling shard allocation on all nodes.
- Stopping Elasticsearch on all nodes by killing the running Elasticsearch process (I'm not running ES as a service, I'm starting it via a script).
- Starting Elasticsearch on the primary master node first via a script.
- Starting Elasticsearch on the other nodes.
- Enabling shard allocation on all nodes.
I checked the logs also, all nodes are joining the cluster after starting Elasticsearch on them. I can't forcibly reroute the shards, it would delete the data from the index.
I am unable to find a permanent solution to the problem even after 1 week of debugging. How to fix this issue?