# OOM errors, shard destroyed, index destroyed, etc **URL:** \ **Category:** Elasticsearch\ **Created:** [June 13, 2012, 10:57pm UTC](https://discuss.elastic.co/t/oom-errors-shard-destroyed-index-destroyed-etc/8095 "2012-06-13T22:57:38Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![courtenay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/courtenay/32/1933_2.png) [@courtenay](https://discuss.elastic.co/u/courtenay)\ **Post date:** [June 13, 2012, 10:57pm UTC](https://discuss.elastic.co/t/oom-errors-shard-destroyed-index-destroyed-etc/8095/1 "2012-06-13T22:57:38Z")
Hey all, I've had a less than wonderful time with ES over the past few days after a year of good service. We have about 60gb of data in a troublesome index. 3 nodes, 2gb ram each, 1 replica (2 copies of data). It's been running fine up until the weekend. Here's what happens: - One of the nodes goes OOM. - Master node locks up - Other nodes remove master for timeout - Cluster stops responding - Other nodes either go OOM or lock up or start spewing errors - Those crash too So I have to generally restart the master and the OOM nodes, and sometimes the whole cluster. When it comes back up - Cluster reports unassigned shards - Cluster rebuilds, eventually - Cluster crashes again with OOM. Amazingly, it's always consistently the same shard that is unassigned. One of the crashes actually deleted all copies of that shard! So we had to rebuild the whole index from scratch (it's going to take about a week. On production systems.). Grr. So we rebuilt the cluster with way more shards (32 instead of 5), upgraded the hardware, (now running on 5 beefy machines with 17 or 58gb of ram each). And the while it's a different shard now that goes bad and has to be recovered, it's still the same consistent one, and we still get OOMs. So it appears to my uneducated self that there's a bad document there which is causing a leak -\> OOM errors which cascades failures to the whole cluster and causes split-brain issues which cause the deletions and other weird behavior. - Is there a way of inspecting the contents of that shard? - Is there a way of preventing the OOM error? The OOM crash log and our config is here: [https://gist.github.com/21c34e632adff490c08c](https://gist.github.com/21c34e632adff490c08c) Appreciate the help in advance! -- Courtenay ---
**Author:** ![courtenay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/courtenay/32/1933_2.png) [@courtenay](https://discuss.elastic.co/u/courtenay)\ **Post date:** [June 14, 2012, 12:22am UTC](https://discuss.elastic.co/t/oom-errors-shard-destroyed-index-destroyed-etc/8095/2 "2012-06-14T00:22:48Z")
More information inline. On Wednesday, June 13, 2012 3:57:38 PM UTC-7, courtenay wrote: > Hey all, > > I've had a less than wonderful time with ES over the past few days after a > year of good service. > > We have about 60gb of data in a troublesome index. 3 nodes, 2gb ram each, > 1 replica (2 copies of data). It's been running fine up until the weekend. > > Here's what happens: > > - One of the nodes goes OOM. This just happened again, on the same shard. (15) [16:31:37,949][WARN][index.engine.robin] [Adam II] [discussions\_production\_barf][15] failed engine java.lang.OutOfMemoryError: Java heap space And again 20 minutes later. I managed to capture some graphs! (attached) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 14, 2012, 9:44pm UTC](https://discuss.elastic.co/t/oom-errors-shard-destroyed-index-destroyed-etc/8095/3 "2012-06-14T21:44:29Z")
First, which version are you using? Second, are your nodes not even in terms of memory allocated to ES (heap size env var) or machine memory? On Thu, Jun 14, 2012 at 2:22 AM, courtenay [court3nay@gmail.com](mailto:court3nay@gmail.com) wrote: > More information inline. > > On Wednesday, June 13, 2012 3:57:38 PM UTC-7, courtenay wrote: > > > Hey all, > > > > I've had a less than wonderful time with ES over the past few days after > > a year of good service. > > > > We have about 60gb of data in a troublesome index. 3 nodes, 2gb ram each, > > 1 replica (2 copies of data). It's been running fine up until the weekend. > > > > Here's what happens: > > > > - One of the nodes goes OOM. > > This just happened again, on the same shard. (15) > > [16:31:37,949][WARN][index.engine.robin] [Adam II] > [discussions\_production\_barf][15] failed engine > java.lang.OutOfMemoryError: Java heap space > > And again 20 minutes later. I managed to capture some graphs! (attached) ---
**Author:** ![courtenay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/courtenay/32/1933_2.png) [@courtenay](https://discuss.elastic.co/u/courtenay)\ **Post date:** [June 14, 2012, 11:14pm UTC](https://discuss.elastic.co/t/oom-errors-shard-destroyed-index-destroyed-etc/8095/4 "2012-06-14T23:14:48Z")
0.19.4 The machines have different amounts of memory, yes. Either 6, 15, or 25 gb allocated to ES. So I can stop the "leak" by changing the thread pool model, but then what happens is that it indexes about 30 documents then freezes up. So .. maybe something is preventing indexing and the cache thread model would just keep spawning and run out of memory. And all my servers are sitting on 100-200% CPU despite not actually indexing or having much heavy load. Where can I start looking? Extra logs? On Jun 14, 2012, at 2:44 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > First, which version are you using? > Second, are your nodes not even in terms of memory allocated to ES (heap size env var) or machine memory? > > On Thu, Jun 14, 2012 at 2:22 AM, courtenay [court3nay@gmail.com](mailto:court3nay@gmail.com) wrote: > More information inline. > > On Wednesday, June 13, 2012 3:57:38 PM UTC-7, courtenay wrote: > Hey all, > > I've had a less than wonderful time with ES over the past few days after a year of good service. > > We have about 60gb of data in a troublesome index. 3 nodes, 2gb ram each, 1 replica (2 copies of data). It's been running fine up until the weekend. > > Here's what happens: > > - One of the nodes goes OOM. > > This just happened again, on the same shard. (15) > > [16:31:37,949][WARN][index.engine.robin] [Adam II] [discussions\_production\_barf][15] failed engine > java.lang.OutOfMemoryError: Java heap space > > And again 20 minutes later. I managed to capture some graphs! (attached) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:24am UTC](https://discuss.elastic.co/t/oom-errors-shard-destroyed-index-destroyed-etc/8095/5 "2017-07-06T03:24:02Z")