I started out with a single Elasticsearch node (Node1).
5 primary shards, 1 replica
I have 30+ days of logs in here, the total size of my data on node1 is 340+G.
I now intend to add two more nodes to this cluster as I need to quickly start consuming a whole lot more of data into Elasticsearch.
I added node2, with the same configuration as above. I changed the following in the yml file:
cluster.name - same on Node 1, Node 2
discovery.zen.ping.unicast.hosts: Node 1 refers to Node 2 and vice versa
I start up Node 1 first, followed by Node 2.
I also set the following cluster properties:
Node 1 and Node 2 discovered each other and joined the same cluster. However, no shards were rebalanced and moved to Node 2. All the data is still in Node 1.
I then tweaked the disk high watermark and low watermark to help trigger the relocation. The master node reports the following in the logs:
[2016-04-17 13:44:32,499][WARN ][cluster.routing.allocation.decider] [production-debugging-node-1] high disk watermark [35%] exceeded on [M8cyrMtDRUm00syJbBhB8A][production-debugging-node-1][/opt/elasticsearch/data/production-debugging/nodes/0] free: 871.7gb[49.2%], shards will be relocated away from this node
However, no relocation begins, all the shards are still in Node 1. Am I missing something here?