No shards migrating to 2 newly added data nodes

[mwbogucki (Michael Bogucki)]

Hello,
We recently lost two data-nodes, from our cluster, within the same aws-region.
The cluster eventually rebalanced and is 100%/GREEN; however after re-adding the 2 missing datanodes,
I am not seeing any shards migrating to those two nodes.

_cat/health:
1527810438 23:47:18 nva-p-elasticsearch green 21 15 29904 14952 0 0 0 0 - 100.0%

curl -s 'localhost:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
2309 26.9gb 30.2gb 28.7gb 58.9gb 51 xxxxxxxxxxxx xxxxxxxxxxxx multi-data07.yello.ylo
2289 25.3gb 28.5gb 30.4gb 58.9gb 48 xxxxxxxxxxxx xxxxxxxxxxxx multi-data08.yello.ylo
2294 31.2gb 34.4gb 24.4gb 58.9gb 58 xxxxxxxxxxxx xxxxxxxxxxxx multi-data15.yello.ylo
2287 27.8gb 31.1gb 27.8gb 58.9gb 52 xxxxxxxxxxxx xxxxxxxxxxxx multi-data13.yello.ylo
2308 18.6gb 21.6gb 37.2gb 58.9gb 36 xxxxxxxxxxxx xxxxxxxxxxxx multi-data05.yello.ylo
2309 18.3gb 21.3gb 37.5gb 58.9gb 36 xxxxxxxxxxxx xxxxxxxxxxxx multi-data03.yello.ylo
0 0b 3gb 55.8gb 58.9gb 5 xxxxxxxxxxxx xxxxxxxxxxxx multi-data11.yello.ylo
0 0b 3gb 55.8gb 58.9gb 5 xxxxxxxxxxxx xxxxxxxxxxxx multi-data12.yello.ylo
2303 21.9gb 24.9gb 33.9gb 58.9gb 42 xxxxxxxxxxxx xxxxxxxxxxxx multi-data02.yello.ylo
2319 19.7gb 23gb 35.9gb 58.9gb 39 xxxxxxxxxxxx xxxxxxxxxxxx multi-data09.yello.ylo
2293 22.9gb 26.1gb 32.8gb 58.9gb 44 xxxxxxxxxxxx xxxxxxxxxxxx multi-data06.yello.ylo
2311 26.4gb 29.4gb 29.4gb 58.9gb 49 xxxxxxxxxxxx xxxxxxxxxxxx multi-data01.yello.ylo
2295 23.9gb 27.1gb 31.8gb 58.9gb 46 xxxxxxxxxxxx xxxxxxxxxxxx multi-data10.yello.ylo
2293 21.3gb 24.3gb 34.5gb 58.9gb 41 xxxxxxxxxxxx xxxxxxxxxxxx multi-data04.yello.ylo
2294 19.2gb 22.4gb 36.5gb 58.9gb 38 xxxxxxxxxxxx xxxxxxxxxxxx multi-data14.yello.ylo

curl -GET localhost:9200/_cluster/settings?pretty
{
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"rebalance" : {
"enable" : "none"
},
"allocation" : {
"cluster_concurrent_rebalance" : "10",
"enable" : "all"
}
}
}
}
}

Any suggestions would be highly helpful

Thank you.

--Mike

[Christian_Dahlqvist]

Do the two new nodes run exactly the same version of Elasticsearch as the other nodes?

[mwbogucki (Michael Bogucki)]

Hi Christian, Thank you for the reply. Yes presently 2.4.1

[mwbogucki (Michael Bogucki)]

Hi Guys,
I figured out the issue... which was staring at me in the face.... (tired eyes I guess)

Posting the solution... hopefully it'll help someone with a similar issue. <Mind you it's a silly oversight on my part>

curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : {"cluster.routing.rebalance.enable" : "all"} }'

As soon as that was invoked, I saw the cluster started rebalancing.

Looks like this was disabled due to an issue a few weeks ago.

Thanks.

--MIke