Warm Nodes Rebalance not occurring

[FatalGlitch (Tom Callahan)]

We run a Master, Hot, Warm cluster setup, and we've noticed that when adding new warm nodes to increase overall warm storage, there's no rebalance that occurs.

We have allocation enabled for all, no exclusions for allocation, and we've even tried tweaking the balance settings with no luck.

Running a _reroute with zero parameters has been attempted as well, same deal no activity results.

below is the output of _cluster/settings (very basic):

{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"enable": "all"
}
}
},
"discovery": {
"zen": {
"minimum_master_nodes": "4"
}
},
"indices": {
"store": {
"throttle": {
"max_bytes_per_sec": "250mb"
}
}
}
},
"transient": {}
}

Our ES version is 5.6.5. Any ideas what could be hung up preventing the rebalance? Cluster state is green, nothing appears to be blocking in any way.

[FatalGlitch (Tom Callahan)]

For anyone who ends up with a similar issue, here's what we finally found:

Some background...
We run our Cluster on AWS, across a single region. That region has 6 availability zones, and those zones are used in the shard awareness settings. Because there are 6 zones, ES will not place more than 3 replicas in the same zone.

We run Searchguard on our cluster, and had auto-replicas turned on. When we added 2 more warm nodes, we ran into a situation where we had 4 nodes of the cluster in the same availability zone, and as such the auto-replicas could not assign the new replicas.

To fix this, we had to turn off auto-replicas on Searchguard, and manually set the number of replicas. Once this was resolved, the rebalance operations started again.

ES will prioritize index replicas over rebalance, so it was refusing to do any rebalancing until all the indices were "green"