I have 12nodes (include 3 master nodes and 9 data nodes).
I use Elasticsearch 6.2.3 , ubuntu desktop 512G Memory, about 1000shards in cluster.
Yesterday, the cluster status got red , because one of my data nodes crashed .
While I restarted my crashed node , the cluster still red.
when I use elasticsearch-head check my cluster , the node log shows below:
Caused by : org.elasticsearch.discovery.MasterNotDiscoveredException: ClusterBlockException[blocked by:[SERVICE_UNAVAILABLE/1/state not recovered /initialized];]
at....
....
....
[WARN] [r.suppressed] path: /_aliases, params:{}
org.elasticsearch.transport.RemoteTransportException: [master1][192.168.1.103:9300][indices:admin/get]
Caused by: org.elasticsearch.discovery.MasterNotDiscoveredException:
ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]]
at ...
...
...