New to Elastic Search, I set up 5 boxes/vms on azure with Ubuntu and setup ES. Its was running good, all default settings, except ES_HEAP_SIZE which I set to 50%, which is 14GB, MAX_LOCKED_MEMORY=unlimited and bootstrap.mlockall=true. Indexed 2 indices, one has 5 types (all 5 types combined around 170M documents) and the other only 1. Now I ran a 2 level hierarchical aggregation (bucket in bucket, no filters), thats what I needed. JVM ran out of memory and 2 nodes went down. The request timed out and the remaining 3 rebalanced the data. I restarted ES back on the 2 nodes that went down, one came back normal and rebalanced again but the other didn't initialize any data nor replicated any data. Its joined the cluster but is just sitting idle with no data. No idea why. There are no separate master, all nodes are default, both master and data. I don't know how this happened. Any ideas anyone.