When I shutdown one of my ES Data Nodes shards don't go to the other DN

Elastic Stack Elasticsearch
1

Hi.
Got a strange one, When I shutdown one of my data nodes (my topology is 2 MN's, 2 DN's and 2 CN/IN's) the shards don't go to the other DN .. they just stay unassigned :frowning:

The other day I upgraded my system from 2.3.4 to 5.2.2 (6 x ES, 2 x KIB, 2 x LS) everything went real smoothly , I followed the procedure of first stop automatic allocation flush and then reenabled it again as per this URL:

https://www.elastic.co/guide/en/elasticsearch/reference/5.2/restart-upgrade.html

Just checked my template and don't have the 'allocate disabled' in place so why aren't my shards moving around?

curl -XGET els01:9200/_cluster/settings?pretty
{
  "persistent" : {
"cluster" : {
  "routing" : {
    "allocation" : {
      "disk" : {
        "watermark" : {
          "low" : "90%",
          "high" : "95%"
        }
      },
      "enable" : "all"
    }
  },
  "info" : {
    "update" : {
      "interval" : "30s"
    }
  }
}
  },
  "transient" : { }
}

Thanks.

2

Elasticsearch will not allocate the same shard twice on a node, which its why the replica will stay unassigned. You should also ideally have 3 master eligible nodes rather than 2. If you only have 2 master eligible nodes and one of them goes down, the cluster will not be able to elect a master as a strict majority (2 out of 2) is required.

3

Yes I understand that, my problem is one of my data nodes goes down and then shards go to the unassigned instead of moving to the other data node. I thought that they should move across - at least I thought they use to with 2.3.4.

4

Elasticsearch has as far as I know never allocated primary and replica on the same node. In order to maintain 2 copies of your data when a data node fails, you will need at least 3 data nodes.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.