Node Connected, But Not Being Utilized

Elastic Stack Elasticsearch
1

New to Elastic Search, I set up 5 boxes/vms on azure with Ubuntu and setup ES. Its was running good, all default settings, except ES_HEAP_SIZE which I set to 50%, which is 14GB, MAX_LOCKED_MEMORY=unlimited and bootstrap.mlockall=true. Indexed 2 indices, one has 5 types (all 5 types combined around 170M documents) and the other only 1. Now I ran a 2 level hierarchical aggregation (bucket in bucket, no filters), thats what I needed. JVM ran out of memory and 2 nodes went down. The request timed out and the remaining 3 rebalanced the data. I restarted ES back on the 2 nodes that went down, one came back normal and rebalanced again but the other didn't initialize any data nor replicated any data. Its joined the cluster but is just sitting idle with no data. No idea why. There are no separate master, all nodes are default, both master and data. I don't know how this happened. Any ideas anyone.

Thanks,
Kalyana.

2

What do the logs show on that node and also the current master?

3

The node just shows that its connected and master is recognized. Thats it. Doesn't initialize existing data. I waited for 1 day to see if it takes time to rebalance data but it didn't. I increased replica copies to 2 from the default 1 on the 2 indices present to see it gets used, but no, in fact it went to a unassigned replica state (monitoring from Marvel). I decreased it back to 1 replica. Today I see new marvel indices on it, but no actual data shards still.

4

What version are you on?

5

"number" : "1.7.0",
"lucene_version" : "4.10.4"

6

Any ideas anyone.

7

Have you set any allocation filtering at all?