No shards migrating to 2 newly added data nodes

We recently lost two data-nodes, from our cluster, within the same aws-region.
The cluster eventually rebalanced and is 100%/GREEN; however after re-adding the 2 missing datanodes,
I am not seeing any shards migrating to those two nodes.

1527810438 23:47:18 nva-p-elasticsearch green 21 15 29904 14952 0 0 0 0 - 100.0%

curl -s 'localhost:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.percent host ip node
2309 26.9gb 30.2gb 28.7gb 58.9gb 51 xxxxxxxxxxxx xxxxxxxxxxxx multi-data07.yello.ylo
2289 25.3gb 28.5gb 30.4gb 58.9gb 48 xxxxxxxxxxxx xxxxxxxxxxxx multi-data08.yello.ylo
2294 31.2gb 34.4gb 24.4gb 58.9gb 58 xxxxxxxxxxxx xxxxxxxxxxxx multi-data15.yello.ylo
2287 27.8gb 31.1gb 27.8gb 58.9gb 52 xxxxxxxxxxxx xxxxxxxxxxxx multi-data13.yello.ylo
2308 18.6gb 21.6gb 37.2gb 58.9gb 36 xxxxxxxxxxxx xxxxxxxxxxxx multi-data05.yello.ylo
2309 18.3gb 21.3gb 37.5gb 58.9gb 36 xxxxxxxxxxxx xxxxxxxxxxxx multi-data03.yello.ylo
0 0b 3gb 55.8gb 58.9gb 5 xxxxxxxxxxxx xxxxxxxxxxxx multi-data11.yello.ylo
0 0b 3gb 55.8gb 58.9gb 5 xxxxxxxxxxxx xxxxxxxxxxxx multi-data12.yello.ylo
2303 21.9gb 24.9gb 33.9gb 58.9gb 42 xxxxxxxxxxxx xxxxxxxxxxxx multi-data02.yello.ylo
2319 19.7gb 23gb 35.9gb 58.9gb 39 xxxxxxxxxxxx xxxxxxxxxxxx multi-data09.yello.ylo
2293 22.9gb 26.1gb 32.8gb 58.9gb 44 xxxxxxxxxxxx xxxxxxxxxxxx multi-data06.yello.ylo
2311 26.4gb 29.4gb 29.4gb 58.9gb 49 xxxxxxxxxxxx xxxxxxxxxxxx multi-data01.yello.ylo
2295 23.9gb 27.1gb 31.8gb 58.9gb 46 xxxxxxxxxxxx xxxxxxxxxxxx multi-data10.yello.ylo
2293 21.3gb 24.3gb 34.5gb 58.9gb 41 xxxxxxxxxxxx xxxxxxxxxxxx multi-data04.yello.ylo
2294 19.2gb 22.4gb 36.5gb 58.9gb 38 xxxxxxxxxxxx xxxxxxxxxxxx multi-data14.yello.ylo

curl -GET localhost:9200/_cluster/settings?pretty
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"rebalance" : {
"enable" : "none"
"allocation" : {
"cluster_concurrent_rebalance" : "10",
"enable" : "all"

Any suggestions would be highly helpful

Thank you.


Do the two new nodes run exactly the same version of Elasticsearch as the other nodes?

Hi Christian, Thank you for the reply. Yes presently 2.4.1

Hi Guys,
I figured out the issue... which was staring at me in the face.... (tired eyes I guess)

Posting the solution... hopefully it'll help someone with a similar issue. <Mind you it's a silly oversight on my part>

curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : {"cluster.routing.rebalance.enable" : "all"} }'

As soon as that was invoked, I saw the cluster started rebalancing.

Looks like this was disabled due to an issue a few weeks ago.



This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.