I tried several OS memory, ES heap combinations, the biggest being
48GiB for the operating system and 32GiB for ES heap and it still
fails with that.
Any idea or link to an open issue I could follow ?
Regards,
Thomas.
1. debug output:
[2015-01-14 12:01:55,740][DEBUG][indices.cluster] [Saint Elmo]
[mailspool][0] creating shard
[2015-01-14 12:01:55,741][DEBUG][index.service] [Saint Elmo]
[mailspool] creating shard\_id [0]
[2015-01-14 12:01:56,041][DEBUG][index.deletionpolicy] [Saint Elmo]
[mailspool][0] Using [keep\_only\_last] deletion policy
[2015-01-14 12:01:56,041][DEBUG][index.merge.policy] [Saint Elmo]
[mailspool][0] using [tiered] merge mergePolicy with
expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_on
ce[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb],
segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
[2015-01-14 12:01:56,041][DEBUG][index.merge.scheduler] [Saint Elmo]
[mailspool][0] using [concurrent] merge scheduler with max\_thread\_count[2],
max\_merge\_count[4]
[2015-01-14 12:01:56,042][DEBUG][index.shard.service] [Saint Elmo]
[mailspool][0] state: [CREATED]
[2015-01-14 12:01:56,043][DEBUG][index.translog] [Saint Elmo]
[mailspool][0] interval [5s], flush\_threshold\_ops [2147483647],
flush\_threshold\_size [200mb], flush\_threshold\_period [3
0m]
[2015-01-14 12:01:56,044][DEBUG][index.shard.service] [Saint Elmo]
[mailspool][0] state: [CREATED]-\>[RECOVERING], reason [from gateway]
[2015-01-14 12:01:56,044][DEBUG][index.gateway] [Saint Elmo]
[mailspool][0] starting recovery from local ...
[2015-01-14 12:01:56,048][DEBUG][river.cluster] [Saint Elmo]
processing [reroute\_rivers\_node\_changed]: execute
[2015-01-14 12:01:56,048][DEBUG][river.cluster] [Saint Elmo]
processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
[2015-01-14 12:01:56,048][DEBUG][cluster.service] [Saint Elmo]
processing [shard-failed ([mailspool][3], node[gOgAuHo4SXyfyuPpws0Usw],
[P], s[INITIALIZING]), reason [engine failure,
message [refresh failed][OutOfMemoryError[Java heap space]]]]: done
applying updated cluster\_state (version: 4)
[2015-01-14 12:01:56,062][DEBUG][index.engine.internal] [Saint Elmo]
[mailspool][0] starting engine
[2015-01-14 12:02:19,701][WARN][index.engine.internal] [Saint Elmo]
[mailspool][0] failed engine [refresh failed]
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.FixedBitSet.(FixedBitSet.java:187)
at
org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:104)
at
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
at
org.elasticsearch.common.lucene.search.OrFilter.getDocIdSet(OrFilter.java:55)
at
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
at
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:130)
at
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
at
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:554)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:287)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3271)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3262)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:421)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:292)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:267)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:257)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
at
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
at
org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:796)
at
org.elasticsearch.index.engine.internal.InternalEngine.delete(InternalEngine.java:692)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:798)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:268)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-01-14 12:02:19,704][DEBUG][index.service] [Saint Elmo]
[mailspool] [0] closing... (reason: [engine failure, message [refresh
failed][OutOfMemoryError[Java heap space]]])
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7fcc12b7-9024-466c-8a78-7d5678b0d605%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7fcc12b7-9024-466c-8a78-7d5678b0d605%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Thomas\_Cataldo](https://discuss.elastic.co/u/Thomas_Cataldo)
#### Post date: [January 14, 2015, 2:24pm UTC](https://discuss.elastic.co/t/out-of-memory-at-startup-with-large-index-and-parent-child-relation/21637/2 "2015-01-14T14:24:59Z")
Hi,
By removing all my translog files, ES can start without error.
On Wednesday, January 14, 2015 at 2:56:48 PM UTC+1, Thomas Cataldo wrote:
> Hi,
>
> I encounter a problem with a large index (38GB) that prevents ES 1.4.2
> from starting.
> The problem looks pretty similar to the one in
> [Large index no longer initialises under 1.4.0 and 1.4.0 Beta 1 due to OutOfMemoryException · Issue #8394 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/8394)
>
> I tried some of the recommandations from this post (and linked ones) :
>
> index.load\_fixed\_bitset\_filters\_eagerly: false
> index.warmer.enabled: false
> indices.breaker.total.limit: 30%
>
> And event with that, my server does not start [1].
>
> I uploaded to gist the mapping for the index :
> [mailspool index template, with mapping · GitHub](https://gist.github.com/tcataldo/c0b6b3dfec9823bf6523)
>
> I tried several OS memory, ES heap combinations, the biggest being
> 48GiB for the operating system and 32GiB for ES heap and it still
> fails with that.
>
> Any idea or link to an open issue I could follow ?
>
> Regards,
> Thomas.
>
> 1. debug output:
>
> [2015-01-14 12:01:55,740][DEBUG][indices.cluster] [Saint Elmo]
> [mailspool][0] creating shard
> [2015-01-14 12:01:55,741][DEBUG][index.service] [Saint Elmo]
> [mailspool] creating shard\_id [0]
> [2015-01-14 12:01:56,041][DEBUG][index.deletionpolicy] [Saint Elmo]
> [mailspool][0] Using [keep\_only\_last] deletion policy
> [2015-01-14 12:01:56,041][DEBUG][index.merge.policy] [Saint Elmo]
> [mailspool][0] using [tiered] merge mergePolicy with
> expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_on
> ce[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb],
> segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-01-14 12:01:56,041][DEBUG][index.merge.scheduler] [Saint Elmo]
> [mailspool][0] using [concurrent] merge scheduler with max\_thread\_count[2],
> max\_merge\_count[4]
> [2015-01-14 12:01:56,042][DEBUG][index.shard.service] [Saint Elmo]
> [mailspool][0] state: [CREATED]
> [2015-01-14 12:01:56,043][DEBUG][index.translog] [Saint Elmo]
> [mailspool][0] interval [5s], flush\_threshold\_ops [2147483647],
> flush\_threshold\_size [200mb], flush\_threshold\_period [3
> 0m]
> [2015-01-14 12:01:56,044][DEBUG][index.shard.service] [Saint Elmo]
> [mailspool][0] state: [CREATED]-\>[RECOVERING], reason [from gateway]
> [2015-01-14 12:01:56,044][DEBUG][index.gateway] [Saint Elmo]
> [mailspool][0] starting recovery from local ...
> [2015-01-14 12:01:56,048][DEBUG][river.cluster] [Saint Elmo]
> processing [reroute\_rivers\_node\_changed]: execute
> [2015-01-14 12:01:56,048][DEBUG][river.cluster] [Saint Elmo]
> processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
> [2015-01-14 12:01:56,048][DEBUG][cluster.service] [Saint Elmo]
> processing [shard-failed ([mailspool][3], node[gOgAuHo4SXyfyuPpws0Usw],
> [P], s[INITIALIZING]), reason [engine failure,
> message [refresh failed][OutOfMemoryError[Java heap space]]]]: done
> applying updated cluster\_state (version: 4)
> [2015-01-14 12:01:56,062][DEBUG][index.engine.internal] [Saint Elmo]
> [mailspool][0] starting engine
> [2015-01-14 12:02:19,701][WARN][index.engine.internal] [Saint Elmo]
> [mailspool][0] failed engine [refresh failed]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.util.FixedBitSet.(FixedBitSet.java:187)
> at
> org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:104)
> at
> org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
> at
> org.elasticsearch.common.lucene.search.OrFilter.getDocIdSet(OrFilter.java:55)
> at
> org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
> at
> org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:130)
> at
> org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
> at
> org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
> at
> org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
> at
> org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:554)
> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:287)
> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3271)
> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3262)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:421)
> at
> org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:292)
> at
> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:267)
> at
> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:257)
> at
> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171)
> at
> org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
> at
> org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
> at
> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
> at
> org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:796)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.delete(InternalEngine.java:692)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:798)
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:268)
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-14 12:02:19,704][DEBUG][index.service] [Saint Elmo]
> [mailspool] [0] closing... (reason: [engine failure, message [refresh
> failed][OutOfMemoryError[Java heap space]]])
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/91792194-7511-4d16-992b-67b23f6c8bb0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/91792194-7511-4d16-992b-67b23f6c8bb0%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/out-of-memory-at-startup-with-large-index-and-parent-child-relation/21637/3 "2017-07-06T00:39:01Z")