Multiple datacenters with elasticsearch

Elastic Stack Elasticsearch
1

We have ES cluster running in one of the datacenters (say DC1). But we are planning to upgrade to a newer and bigger datacenter (say DC2). As part of migration (outageless), we thought we can spin up new nodes with same cluster settings as the current DC1 in DC2.

DC1 is running ES 1.5.2 and DC2 is running 1.7. While testing I notice that shards get allocated from DC1 to DC2. But the reverse does not happen when DC2 nodes are shutdown in a controlled manner. Meaning the shards do not get allocated back to DC1.

Is this because of version difference between in ES installs?

In my yaml files I made sure that the property discovery.zen.ping.unicast.hosts is set to the same master nodes in both DC1 and DC2. Also the cluster.name is the same. I can see the DC2 nodes in Marvel as well and DC1 to DC2 works great but not the reverse.

Is there any other better way to migrate from one DC to another without causing any outage and any staleness in data?

2

Yep. Once segments are merged on a newer version, ES will not allow them to be allocated to nodes running older versions.

Given you've already started down this path, you should see it through.

3

Thank you.