# ElasticSearch Recovery after data loss due to an Out of memory error **URL:** \ **Category:** Elasticsearch\ **Created:** [October 12, 2011, 5:59pm UTC](https://discuss.elastic.co/t/elasticsearch-recovery-after-data-loss-due-to-an-out-of-memory-error/5581 "2011-10-12T17:59:03Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Mike\_Peters](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_peters/32/3022_2.png) [@Mike\_Peters](https://discuss.elastic.co/u/Mike_Peters)\ **Post date:** [October 12, 2011, 5:59pm UTC](https://discuss.elastic.co/t/elasticsearch-recovery-after-data-loss-due-to-an-out-of-memory-error/5581/1 "2011-10-12T17:59:03Z")
Hi, Running ElasticSearch 0.16.4 in a 2 node setup (shards=3, replication=1), last night one of the nodes displayed an "Out of memory" error in its log. This morning it became apparent something is terribly wrong. Shortly after the Out of memory error, ElasticSearch wiped out about 2/3 of our index. We tried a rolling restart, complete cluster restart, flush, refresh, but nothing helped. The cluster comes back up with everything showing up healthy, but 2/3 of the data is GONE. Not sure if this is a known bug? To recover we rsync'ed an old index backup and manually pushed all updates from the last 24 hours. Wondering if there is a more natural way of recovery? Right now we are using the local-gateway. Would using s3 have helped us in such a situation? In our experience ElasticSearch built-in recovery does the trick in most cases, but we did experience a handful of cases such as this recent one, where something becomes corrupted beyond ElasticSearch ability to automatically recover it. We're looking to establish a best-practices procedure on how to recover in these cases. Would appreciate any feedback from the community as well as thoughts about whether gateway-s3 helps in providing better recovery. Our index size is 100gb and it changes frequently. Thanks, Mike Peters ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 14, 2011, 4:10pm UTC](https://discuss.elastic.co/t/elasticsearch-recovery-after-data-loss-due-to-an-out-of-memory-error/5581/2 "2011-10-14T16:10:28Z")
The problem with data possibly being lost on an OOM has been fixed in 0.17. Local gateway aim is to work its way up to a state where you never loose data, and it provides substantial benefit over a shared gateway because it considerably more lightweight. On Wed, Oct 12, 2011 at 7:59 PM, Mike Peters [mike@softwareprojects.com](mailto:mike@softwareprojects.com)wrote: > Hi, > > Running Elasticsearch 0.16.4 in a 2 node setup (shards=3, > replication=1), last night one of the nodes displayed an "Out of > memory" error in its log. > > This morning it became apparent something is terribly wrong. > > Shortly after the Out of memory error, Elasticsearch wiped out about > 2/3 of our index. > > We tried a rolling restart, complete cluster restart, flush, refresh, > but nothing helped. The cluster comes back up with everything showing > up healthy, but 2/3 of the data is GONE. > > Not sure if this is a known bug? > > To recover we rsync'ed an old index backup and manually pushed all > updates from the last 24 hours. > > Wondering if there is a more natural way of recovery? Right now we > are using the local-gateway. Would using s3 have helped us in such a > situation? > > In our experience Elasticsearch built-in recovery does the trick in > most cases, but we did experience a handful of cases such as this > recent one, where something becomes corrupted beyond Elasticsearch > ability to automatically recover it. > > We're looking to establish a best-practices procedure on how to > recover in these cases. > > Would appreciate any feedback from the community as well as thoughts > about whether gateway-s3 helps in providing better recovery. > > Our index size is 100gb and it changes frequently. > > Thanks, > Mike Peters ---
**Author:** ![AEvar\_Arnfjord\_Bjarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aevar_arnfjord_bjarm/32/2746_2.png) [@AEvar\_Arnfjord\_Bjarm](https://discuss.elastic.co/u/AEvar_Arnfjord_Bjarm)\ **Post date:** [October 15, 2011, 10:30am UTC](https://discuss.elastic.co/t/elasticsearch-recovery-after-data-loss-due-to-an-out-of-memory-error/5581/3 "2011-10-15T10:30:09Z")
On Oct 14, 2011 6:10 PM, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > The problem with data possibly being lost on an OOM has been fixed in > 0.17. Local gateway aim is to work its way up to a state where you never > loose data, and it provides substantial benefit over a shared gateway > because it considerably more lightweight. While stress-testing 0.17.6 I had some data corruption like this, the issue was fixed in 0.17.7 > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:51am UTC](https://discuss.elastic.co/t/elasticsearch-recovery-after-data-loss-due-to-an-out-of-memory-error/5581/4 "2017-07-06T03:51:51Z")