Details of ES Index Shard allocation behaviour

[Kumaravel]

I am using ES 2.3.2 without any awareness attribute and plan to use it for datacenter awareness. For testing, I have 1 master node and 4 data nodes out which 2 nodes are in DC1 and 2 nodes in DC2.
When the data nodes are updated with the settings for awareness, the shards are still distributed, but as soon as the master node is updated, the shards start reallocating.

Is there a way to update the awareness setting in a cluster, but initialize the movement of shards manually? Because I have around 5000 shards in my live cluster with 3 master nodes and 11 nodes in each datacenter (i.e. total 22 data nodes) and if this datacenter awareness starts movement of the shards at same time, it could impact the system.

Please help and also let me know if more details are required.

[warkolm (Mark Walkom)]

@Kumaravel

I am using ES 2.3.2

Why are you running 2.X still? You should really upgrade.

@Kumaravel

I have 1 master node

That's bad, see Important Configuration Changes | Elasticsearch: The Definitive Guide [2.x] | Elastic

@Kumaravel

2 nodes are in DC1 and 2 nodes in DC2

That may not be ideal. What is a datacenter for you, is it different cities?

[Kumaravel]

I am aware that I am not on the latest version on ES and we have planned for the upgrade to 5.x in the coming days.
The 1 master setup is only for the test local setup through docker for testing the awareness feature.

In my case the data center refers to different locations with a city.