Hi All-
I'm trying to investigate how ES should be rebalancing our active nodes. Last week I added 10 nodes to our cluster but come Monday it seems we're inserting new data into the new nodes but it isn't rebalancing. We have a lot of nodes over the default 85% used watermark and I was hoping it would allocate some of the shards there to the new nodes.
I've checked that cluster.routing.rebalance.enable is set to ALL but I can't find anything else special to explain why they aren't moving. Looking at the other similar issues I've seen excluded IP's and such but nothing that I've seen in our setup.
Below you'll see the allocation for our nodes. the disk usage ranges from 3% to 89%.
Any quick ideas anyone has seen with this?
Thanks!
name v m dup
es_chpm-21 5.6.3 - 85.54
es_chpm-36 5.6.3 - 83.37
es_chpm-18 5.6.3 - 71.78
es_chpm-31 5.6.3 - 48.36
es_chpm-11 5.6.3 - 86.11
es_chpm-client-01 5.6.3 - 16.96
es_chpm-26 5.6.3 - 86.97
es_chpm-19 5.6.3 - 75.31
es_chpm-06 5.6.3 - 7.47
es_chpm-master-01 5.6.3 - 9.64
es_chpm-12 5.6.3 - 85.08
es_chpm-14 5.6.3 - 85.37
es_chpm-08 5.6.3 - 10.10
es_chpm-03 5.6.3 - 4.31
es_chpm-37 5.6.3 - 70.07
es_chpm-02 5.6.3 - 5.05
es_chpm-01 5.6.3 - 3.44
es_chpm-30 5.6.3 - 86.23
es_chpm-25 5.6.3 - 86.50
es_chpm-13 5.6.3 - 67.58
es_chpm-27 5.6.3 - 76.37
es_chpm-15 5.6.3 - 60.54
es_chpm-07 5.6.3 - 87.31
es_chpm-24 5.6.3 - 88.67
es_chpm-22 5.6.3 - 86.74
es_chpm-29 5.6.3 - 45.78
es_chpm-34 5.6.3 - 74.29
es_chpm-32 5.6.3 - 72.05
es_chpm-client-02 5.6.3 - 9.64
es_chpm-16 5.6.3 - 52.85
es_chpm-38 5.6.3 - 83.83
es_chpm-09 5.6.3 - 13.37
es_chpm-04 5.6.3 - 8.28
es_chpm-35 5.6.3 - 63.22
es_chpm-33 5.6.3 - 88.17
es_chpm-40 5.6.3 - 72.07
es_chpm-05 5.6.3 - 14.25
es_chpm-28 5.6.3 - 73.67
es_chpm-master-02 5.6.3 * 15.15
es_chpm-39 5.6.3 - 77.35
es_chpm-17 5.6.3 - 64.20
es_chpm-20 5.6.3 - 57.46
es_chpm-10 5.6.3 - 15.15
es_chpm-23 5.6.3 - 88.01