Hello there,
I'm migrating my live data from our live elasticsearch server to our archive elasticsearch server. Both are single nodes. While doing this I missed to close indices on the archive server. Using curator I got the error "Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;" somewhere while executing the task. I tried to close indices on the archive server, but I always get "master_not_discovered_exception", regardless of what I'm trying to do, e. g. quering some statistics from the archive:
root@elk-archive-01 ~ # curl -X GET "10.0.0.210:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty" { "error" : { "root_cause" : [ { "type" : "master_not_discovered_exception", "reason" : null } ], "type" : "master_not_discovered_exception", "reason" : null }, "status" : 503 }
In the logs the following lines are suspicous:
failing [elected-as-master ([1] nodes joined)[{elk-archive-01-n1}{QGFmDMAITRu4JNov0SwH1Q}{RbaWF3t7QKyXKWDa3fEbYw}{10.0.0.210}{10.0.0.210:9300}{dilm}{ml.machine_memory=33449811968, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [19845] org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed [...] Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: timed out after 30s
Unfortunately I don't know where to upload the whole logfile 
How can I close the indices to get the server back up again? Thanks a lot for your help!