"master not discovered exception" after having 999 of 1000 max. shards open

Elastic Stack Elasticsearch
1

Hello there,
I'm migrating my live data from our live elasticsearch server to our archive elasticsearch server. Both are single nodes. While doing this I missed to close indices on the archive server. Using curator I got the error "Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;" somewhere while executing the task. I tried to close indices on the archive server, but I always get "master_not_discovered_exception", regardless of what I'm trying to do, e. g. quering some statistics from the archive:

root@elk-archive-01 ~ # curl -X GET "10.0.0.210:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

In the logs the following lines are suspicous:

failing [elected-as-master ([1] nodes joined)[{elk-archive-01-n1}{QGFmDMAITRu4JNov0SwH1Q}{RbaWF3t7QKyXKWDa3fEbYw}{10.0.0.210}{10.0.0.210:9300}{dilm}{ml.machine_memory=33449811968, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [19845]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
[...]
Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: timed out after 30s

Unfortunately I don't know where to upload the whole logfile :frowning:

How can I close the indices to get the server back up again? Thanks a lot for your help!

2

Ok, following "Can not elect master when restarting cluster from 7.3 upgrade" fixed the problem.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.