# Cluster can't recovery indicates from disc **URL:** \ **Category:** Elasticsearch\ **Created:** [January 27, 2017, 11:04am UTC](https://discuss.elastic.co/t/cluster-cant-recovery-indicates-from-disc/73002 "2017-01-27T11:04:54Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![awro](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@awro](https://discuss.elastic.co/u/awro)\ **Post date:** [January 27, 2017, 11:04am UTC](https://discuss.elastic.co/t/cluster-cant-recovery-indicates-from-disc/73002/1 "2017-01-27T11:04:54Z")
Hi, I'm running cluster with 3 master nodes (master:true, data:false), 3 query(master/data:false) and 6 data nodes(master:false, data:true). Today I got "Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space" on 6 data nodes at this same time, cluster swiched to red state, and only master and query instances were available. I restarted all instances, but cluster booted with 0 indexes available and in green state: "11:58:20,087][INFO][gateway] [node83-es-master] recovered [0] indices into cluster\_state" I still have all data on the data nodes, how can I recover from such failure? I'm using elasticsearch 2.4.2 and filesystem on each instances are working fine, without any permission issue ---
**Author:** ![awro](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@awro](https://discuss.elastic.co/u/awro)\ **Post date:** [January 27, 2017, 1:50pm UTC](https://discuss.elastic.co/t/cluster-cant-recovery-indicates-from-disc/73002/2 "2017-01-27T13:50:41Z")
I also tried to switch one data instance to master:true, data:true and boot only that one, but I still getting "recovered [0] indices into cluster\_state". All nodes have copy of global-{numer}.st it's possibile to recovery master metadata from it ? ---
**Author:** ![awro](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@awro](https://discuss.elastic.co/u/awro)\ **Post date:** [January 27, 2017, 3:18pm UTC](https://discuss.elastic.co/t/cluster-cant-recovery-indicates-from-disc/73002/3 "2017-01-27T15:18:41Z")
I also tested this: Creating index with this same name as before, on disc nothing change, all files timestamps are old, nothing was actually created on disc, but elastic returned: {"acknowledged":true} on creation request. So right now I have 1 index, 10 allocated shards but 0 documents inside. ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [January 27, 2017, 8:55pm UTC](https://discuss.elastic.co/t/cluster-cant-recovery-indicates-from-disc/73002/4 "2017-01-27T20:55:52Z")
> [@awro](#): > > java.lang.OutOfMemoryError: Java heap space Looks as though your data nodes JVM are utilising high amounts of java heap memory that is why you see the OOM exception. You need to review if these machines are on a shared host (memory balloning?) how much memory is allocated to each machine and how much of the single machines memory is allocated to the heap. ---
**Author:** ![awro](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@awro](https://discuss.elastic.co/u/awro)\ **Post date:** [January 27, 2017, 9:10pm UTC](https://discuss.elastic.co/t/cluster-cant-recovery-indicates-from-disc/73002/5 "2017-01-27T21:10:38Z")
Each instances have 31gb heap size mlocked. Memory was only initial spark of this fire, real problem right now is forsing master to read data from data nodes and recreate cluster state ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 24, 2017, 9:11pm UTC](https://discuss.elastic.co/t/cluster-cant-recovery-indicates-from-disc/73002/6 "2017-02-24T21:11:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.