# OOM on Cold Cluster Start

**URL:** <https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420>\
**Category:** Elasticsearch\
**Created:** [November 14, 2013, 9:04pm UTC](https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420 "2013-11-14T21:04:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![David\_Kleiner](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@David\_Kleiner](https://discuss.elastic.co/u/David_Kleiner)\
**Post date:** [November 14, 2013, 9:04pm UTC](https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420/1 "2013-11-14T21:04:23Z")

</div>

Hi,

With the latest stable ES, I'm getting OOM on cold cluster start with heap  
size under 25-30G. I had to find a really beefy box to get the cluster up  
and running and then bind two more 10G heap ES nodes to it.

Once the cluster is operational, heap pressure stays under 10G. I have  
2-way cluster, with a single data-less gateway, 40 indices (mostly user  
logs fed by logstash, split by month), 392 total shards (both nodes),  
about 220G total space, 110G/node. I kept the default 5-shards / index.

Recovery on cold start was really painful and took hours of downtime until  
I found a big temporary node.

Any recommendations to avoid this situation on the next cold start  
appreciated!

Thank you,

David

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_Kleiner](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@David\_Kleiner](https://discuss.elastic.co/u/David_Kleiner)\
**Post date:** [November 14, 2013, 9:15pm UTC](https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420/2 "2013-11-14T21:15:14Z")

</div>

stack trace on cold start (single node, unbound to the cluster):

[2013-11-14 15:12:47,626][WARN][index.engine.robin] [Typeface]  
[eventlog-2013.11][0] failed to prepare/warm  
java.lang.OutOfMemoryError: Java heap space  
at  
org.elasticsearch.search.SearchService$IndexReaderWarmer.warm(SearchService.java:649)  
at  
org.elasticsearch.indices.warmer.InternalIndicesWarmer.warm(InternalIndicesWarmer.java:90)  
at  
org.elasticsearch.index.engine.robin.RobinEngine$RobinSearchFactory.newSearcher(RobinEngine.java:1622)  
at  
org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:155)  
at  
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1505)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:280)  
at  
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:660)  
at  
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)  
at  
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:744)  
[2013-11-14 15:12:50,555][WARN][cluster.action.shard] [Typeface]  
[nginx-2013.11][1] sending failed shard for [nginx-2013.11][1],  
node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
reason [Failed to create shard, message  
[IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
nested: OutOfMemoryError[Java heap space]; ]]  
[2013-11-14 15:12:50,556][WARN][cluster.action.shard] [Typeface]  
[nginx-2013.11][1] received shard failed for [nginx-2013.11][1],  
node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
reason [Failed to create shard, message  
[IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
nested: OutOfMemoryError[Java heap space]; ]]

On Thursday, November 14, 2013 1:04:23 PM UTC-8, David Kleiner wrote:

> Hi,
> 
> With the latest stable ES, I'm getting OOM on cold cluster start with heap  
> size under 25-30G. I had to find a really beefy box to get the cluster up  
> and running and then bind two more 10G heap ES nodes to it.
> 
> Once the cluster is operational, heap pressure stays under 10G. I have  
> 2-way cluster, with a single data-less gateway, 40 indices (mostly user  
> logs fed by logstash, split by month), 392 total shards (both nodes),  
> about 220G total space, 110G/node. I kept the default 5-shards / index.
> 
> Recovery on cold start was really painful and took hours of downtime until  
> I found a big temporary node.
> 
> Any recommendations to avoid this situation on the next cold start  
> appreciated!
> 
> Thank you,
> 
> David

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 14, 2013, 9:26pm UTC](https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420/3 "2013-11-14T21:26:59Z")

</div>

Hey,

are you using elasticsearch 0.90.6 or 0.90.7?  
Elasticsearch 0.90.7 was released yesterday to fix an issue, which might  
cause OOMs in that particular setup, see the release blog post at

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

--Alex

On Thu, Nov 14, 2013 at 10:15 PM, David Kleiner [david.kleiner@gmail.com](mailto:david.kleiner@gmail.com)wrote:

> stack trace on cold start (single node, unbound to the cluster):
> 
> [2013-11-14 15:12:47,626][WARN][index.engine.robin] [Typeface]  
> [eventlog-2013.11][0] failed to prepare/warm  
> java.lang.OutOfMemoryError: Java heap space  
> at  
> org.elasticsearch.search.SearchService$IndexReaderWarmer.warm(SearchService.java:649)  
> at  
> org.elasticsearch.indices.warmer.InternalIndicesWarmer.warm(InternalIndicesWarmer.java:90)  
> at  
> org.elasticsearch.index.engine.robin.RobinEngine$RobinSearchFactory.newSearcher(RobinEngine.java:1622)  
> at  
> org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:155)  
> at  
> org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
> at  
> org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1505)  
> at  
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:280)  
> at  
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:660)  
> at  
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)  
> at  
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> at java.lang.Thread.run(Thread.java:744)  
> [2013-11-14 15:12:50,555][WARN][cluster.action.shard] [Typeface]  
> [nginx-2013.11][1] sending failed shard for [nginx-2013.11][1],  
> node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
> reason [Failed to create shard, message  
> [IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
> nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
> nested: OutOfMemoryError[Java heap space]; ]]  
> [2013-11-14 15:12:50,556][WARN][cluster.action.shard] [Typeface]  
> [nginx-2013.11][1] received shard failed for [nginx-2013.11][1],  
> node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
> reason [Failed to create shard, message  
> [IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
> nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
> nested: OutOfMemoryError[Java heap space]; ]]
> 
> On Thursday, November 14, 2013 1:04:23 PM UTC-8, David Kleiner wrote:
> 
> > Hi,
> > 
> > With the latest stable ES, I'm getting OOM on cold cluster start with  
> > heap size under 25-30G. I had to find a really beefy box to get the  
> > cluster up and running and then bind two more 10G heap ES nodes to it.
> > 
> > Once the cluster is operational, heap pressure stays under 10G. I have  
> > 2-way cluster, with a single data-less gateway, 40 indices (mostly user  
> > logs fed by logstash, split by month), 392 total shards (both nodes),  
> > about 220G total space, 110G/node. I kept the default 5-shards / index.
> > 
> > Recovery on cold start was really painful and took hours of downtime  
> > until I found a big temporary node.
> > 
> > Any recommendations to avoid this situation on the next cold start  
> > appreciated!
> > 
> > Thank you,
> > 
> > David
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_Kleiner](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@David\_Kleiner](https://discuss.elastic.co/u/David_Kleiner)\
**Post date:** [November 14, 2013, 10:01pm UTC](https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420/4 "2013-11-14T22:01:56Z")

</div>

That was 0.90.6, I'll give 0.90.7 a try, thank you Alex!

On Thursday, November 14, 2013 1:26:59 PM UTC-8, Alexander Reelsen wrote:

> Hey,
> 
> are you using elasticsearch 0.90.6 or 0.90.7?  
> Elasticsearch 0.90.7 was released yesterday to fix an issue, which might  
> cause OOMs in that particular setup, see the release blog post at  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/0-90-7-released/)
> 
> --Alex
> 
> On Thu, Nov 14, 2013 at 10:15 PM, David Kleiner \<[david....@gmail.com](mailto:david....@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > stack trace on cold start (single node, unbound to the cluster):
> > 
> > [2013-11-14 15:12:47,626][WARN][index.engine.robin] [Typeface]  
> > [eventlog-2013.11][0] failed to prepare/warm  
> > java.lang.OutOfMemoryError: Java heap space  
> > at  
> > org.elasticsearch.search.SearchService$IndexReaderWarmer.warm(SearchService.java:649)  
> > at  
> > org.elasticsearch.indices.warmer.InternalIndicesWarmer.warm(InternalIndicesWarmer.java:90)  
> > at  
> > org.elasticsearch.index.engine.robin.RobinEngine$RobinSearchFactory.newSearcher(RobinEngine.java:1622)  
> > at  
> > org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:155)  
> > at  
> > org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
> > at  
> > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1505)  
> > at  
> > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:280)  
> > at  
> > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:660)  
> > at  
> > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)  
> > at  
> > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> > at java.lang.Thread.run(Thread.java:744)  
> > [2013-11-14 15:12:50,555][WARN][cluster.action.shard] [Typeface]  
> > [nginx-2013.11][1] sending failed shard for [nginx-2013.11][1],  
> > node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
> > reason [Failed to create shard, message  
> > [IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
> > nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
> > nested: OutOfMemoryError[Java heap space]; ]]  
> > [2013-11-14 15:12:50,556][WARN][cluster.action.shard] [Typeface]  
> > [nginx-2013.11][1] received shard failed for [nginx-2013.11][1],  
> > node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
> > reason [Failed to create shard, message  
> > [IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
> > nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
> > nested: OutOfMemoryError[Java heap space]; ]]
> > 
> > On Thursday, November 14, 2013 1:04:23 PM UTC-8, David Kleiner wrote:
> > 
> > > Hi,
> > > 
> > > With the latest stable ES, I'm getting OOM on cold cluster start with  
> > > heap size under 25-30G. I had to find a really beefy box to get the  
> > > cluster up and running and then bind two more 10G heap ES nodes to it.
> > > 
> > > Once the cluster is operational, heap pressure stays under 10G. I have  
> > > 2-way cluster, with a single data-less gateway, 40 indices (mostly user  
> > > logs fed by logstash, split by month), 392 total shards (both nodes),  
> > > about 220G total space, 110G/node. I kept the default 5-shards / index.
> > > 
> > > Recovery on cold start was really painful and took hours of downtime  
> > > until I found a big temporary node.
> > > 
> > > Any recommendations to avoid this situation on the next cold start  
> > > appreciated!
> > > 
> > > Thank you,
> > > 
> > > David
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_Kleiner](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@David\_Kleiner](https://discuss.elastic.co/u/David_Kleiner)\
**Post date:** [November 14, 2013, 10:24pm UTC](https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420/5 "2013-11-14T22:24:45Z")

</div>

Happy to report, with 0.90.7 the nodes came back up fast and with no heap  
issues. Guess I can bring the heap size down from 12G to 8G to give more  
room the the logstash instances.

Cheers!

David

On Thursday, November 14, 2013 2:01:56 PM UTC-8, David Kleiner wrote:

> That was 0.90.6, I'll give 0.90.7 a try, thank you Alex!
> 
> On Thursday, November 14, 2013 1:26:59 PM UTC-8, Alexander Reelsen wrote:
> 
> > Hey,
> > 
> > are you using elasticsearch 0.90.6 or 0.90.7?  
> > Elasticsearch 0.90.7 was released yesterday to fix an issue, which might  
> > cause OOMs in that particular setup, see the release blog post at  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/0-90-7-released/)
> > 
> > --Alex
> > 
> > On Thu, Nov 14, 2013 at 10:15 PM, David Kleiner [david....@gmail.com](mailto:david....@gmail.com)wrote:
> > 
> > > stack trace on cold start (single node, unbound to the cluster):
> > > 
> > > [2013-11-14 15:12:47,626][WARN][index.engine.robin] [Typeface]  
> > > [eventlog-2013.11][0] failed to prepare/warm  
> > > java.lang.OutOfMemoryError: Java heap space  
> > > at  
> > > org.elasticsearch.search.SearchService$IndexReaderWarmer.warm(SearchService.java:649)  
> > > at  
> > > org.elasticsearch.indices.warmer.InternalIndicesWarmer.warm(InternalIndicesWarmer.java:90)  
> > > at  
> > > org.elasticsearch.index.engine.robin.RobinEngine$RobinSearchFactory.newSearcher(RobinEngine.java:1622)  
> > > at  
> > > org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:155)  
> > > at  
> > > org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)  
> > > at  
> > > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1505)  
> > > at  
> > > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:280)  
> > > at  
> > > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:660)  
> > > at  
> > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)  
> > > at  
> > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> > > at java.lang.Thread.run(Thread.java:744)  
> > > [2013-11-14 15:12:50,555][WARN][cluster.action.shard] [Typeface]  
> > > [nginx-2013.11][1] sending failed shard for [nginx-2013.11][1],  
> > > node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
> > > reason [Failed to create shard, message  
> > > [IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
> > > nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
> > > nested: OutOfMemoryError[Java heap space]; ]]  
> > > [2013-11-14 15:12:50,556][WARN][cluster.action.shard] [Typeface]  
> > > [nginx-2013.11][1] received shard failed for [nginx-2013.11][1],  
> > > node[nc9mQX0vQzyWWINV0l8t9Q], [P], s[INITIALIZING], indexUUID [_na_],  
> > > reason [Failed to create shard, message  
> > > [IndexShardCreationException[[nginx-2013.11][1] failed to create shard];  
> > > nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];  
> > > nested: OutOfMemoryError[Java heap space]; ]]
> > > 
> > > On Thursday, November 14, 2013 1:04:23 PM UTC-8, David Kleiner wrote:
> > > 
> > > > Hi,
> > > > 
> > > > With the latest stable ES, I'm getting OOM on cold cluster start with  
> > > > heap size under 25-30G. I had to find a really beefy box to get the  
> > > > cluster up and running and then bind two more 10G heap ES nodes to it.
> > > > 
> > > > Once the cluster is operational, heap pressure stays under 10G. I have  
> > > > 2-way cluster, with a single data-less gateway, 40 indices (mostly user  
> > > > logs fed by logstash, split by month), 392 total shards (both nodes),  
> > > > about 220G total space, 110G/node. I kept the default 5-shards / index.
> > > > 
> > > > Recovery on cold start was really painful and took hours of downtime  
> > > > until I found a big temporary node.
> > > > 
> > > > Any recommendations to avoid this situation on the next cold start  
> > > > appreciated!
> > > > 
> > > > Thank you,
> > > > 
> > > > David
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:06am UTC](https://discuss.elastic.co/t/oom-on-cold-cluster-start/14420/6 "2017-07-06T02:06:52Z")

</div>