I'm running cluster with 3 master nodes (master:true, data:false), 3 query(master/data:false) and 6 data nodes(master:false, data:true).
Today I got "Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space" on 6 data nodes at this same time, cluster swiched to red state, and only master and query instances were available.
I restarted all instances, but cluster booted with 0 indexes available and in green state:
"11:58:20,087][INFO ][gateway ] [node83-es-master] recovered  indices into cluster_state"
I still have all data on the data nodes, how can I recover from such failure?
I'm using elasticsearch 2.4.2 and filesystem on each instances are working fine, without any permission issue