# ElasticSearch crashes in single node cluster- Issue #1

**URL:** <https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781>\
**Category:** Elasticsearch\
**Created:** [May 13, 2019, 10:07am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781 "2019-05-13T10:07:55Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 10:07am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/1 "2019-05-13T10:07:55Z")

</div>

Hi,

Environment:  
OS : Windows 64-bit  
ElasticSearch : 6.6.2  
Java: 1.8.151  
Cluster: Single Node cluster  
Index strategy: Indices are created daily and programmatically  
Shard: Single shard per index and no replication

We have been using elasticsearch for more than a month.  
The machine contains approximately 30 indices (data for 30 days).  
Total size of all the indices is around 100s of GBs.

Recently, We have been facing elasticsearch crashes in most of the machines.

Few queries from logs:

=============================================================================================================================================  
#1: Why GC is running frequently? Why does it consume time in seconds for Garbage collection?  
There is enough space remaining in the machine. And data rate is almost same in most of the days.  
Why is JVM crash because of out of memory exception?

[2019-05-08T02:18:58,275][WARN][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][12982] overhead, spent [513ms] collecting in the last [1s]  
[2019-05-08T02:19:55,482][INFO][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13039] overhead, spent [376ms] collecting in the last [1.1s]  
[2019-05-08T02:21:14,601][INFO][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13118] overhead, spent [441ms] collecting in the last [1s]  
[2019-05-08T02:22:05,676][INFO][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13169] overhead, spent [491ms] collecting in the last [1s]  
[2019-05-08T02:22:08,679][INFO][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13172] overhead, spent [475ms] collecting in the last [1s]  
[2019-05-08T02:22:10,097][WARN][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13173] overhead, spent [844ms] collecting in the last [1.4s]  
[2019-05-08T02:22:14,684][WARN][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13174] overhead, spent [3.4s] collecting in the last [3.4s]  
[2019-05-08T02:22:15,829][WARN][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13175] overhead, spent [2.2s] collecting in the last [2.2s]  
[2019-05-08T02:22:15,828][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [EIcXa59] fatal error in thread [elasticsearch[EIcXa59][search][T#78]], exiting  
java.lang.OutOfMemoryError: Java heap space  
at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:111) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:108) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:53) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.recycler.AbstractRecycler.obtain(AbstractRecycler.java:33) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:28) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.recycler.Recyclers$3.obtain(Recyclers.java:119) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.PageCacheRecycler.bytePage(PageCacheRecycler.java:159) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.AbstractBigArray.newBytePage(AbstractBigArray.java:117) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.BigByteArray.resize(BigByteArray.java:143) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.BigArrays.resizeInPlace(BigArrays.java:440) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.BigArrays.resize(BigArrays.java:487) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.BigArrays.grow(BigArrays.java:505) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.io.stream.BytesStreamOutput.ensureCapacity(BytesStreamOutput.java:158) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.ensureCapacity(ReleasableBytesStreamOutput.java:70) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:90) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.io.Streams$FlushOnCloseOutputStream.writeBytes(Streams.java:266) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:460) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.copyStream(JsonXContentGenerator.java:502) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeRawField(JsonXContentGenerator.java:352) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeRawField(JsonXContentGenerator.java:333) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.xcontent.XContentBuilder.rawField(XContentBuilder.java:967) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.xcontent.XContentHelper.writeRawField(XContentHelper.java:308) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.search.SearchHit.toInnerXContent(SearchHit.java:466) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.search.SearchHit.toXContent(SearchHit.java:408) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.search.SearchHits.toXContent(SearchHits.java:158) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.SearchResponseSections.toXContent(SearchResponseSections.java:106) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.SearchResponse.innerToXContent(SearchResponse.java:241) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.SearchResponse.toXContent(SearchResponse.java:221) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:57) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:33) ~[elasticsearch-6.6.2.jar:6.6.2]

Thanks,  
HGVP

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [May 13, 2019, 10:51am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/2 "2019-05-13T10:51:07Z")

</div>

HI @HPVRNP

It looks like you are suffering from heap pressure.

I would recommend trying to increase the size of the heap and check.

Regards,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 11:22am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/3 "2019-05-13T11:22:43Z")

</div>

#2 We have been running same queries programmatically through our custom application daily.  
Why are these errors coming all of a sudden? When will these issues arise?

[2019-05-08T02:23:03,516][WARN][r.suppressed] [EIcXa59] path: /dailylogs-08-05-2019/logentry/\_search, params: {typed\_keys=true, ignore\_unavailable=true, allow\_no\_indices=true, index=dailylogs-08-05-2019, type=logentry, track\_total\_hits=false}  
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_151]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_151]  
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_151]

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 11:24am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/4 "2019-05-13T11:24:08Z")

</div>

[2019-05-08T02:23:04,029][DEBUG][o.e.a.s.TransportSearchAction] [EIcXa59] All shards failed for phase: [query]  
[2019-05-08T02:23:04,029][WARN][r.suppressed] [EIcXa59] path: /dailylogs-08-05-2019/logentry/\_search, params: {typed\_keys=true, ignore\_unavailable=true, allow\_no\_indices=true, index=dailylogs-08-05-2019, type=logentry, track\_total\_hits=false}  
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:221) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:175) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:221) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:462) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.2.jar:6.6.2]

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 11:24am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/5 "2019-05-13T11:24:14Z")

</div>

```
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:368) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:362) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

```

[2019-05-08T02:23:04,536][DEBUG][o.e.a.s.TransportSearchAction] [EIcXa59] All shards failed for phase: [query]  
[2019-05-08T02:23:04,536][WARN][r.suppressed] [EIcXa59] path: /dailylogs-08-05-2019/logentry/\_search, params: {typed\_keys=true, ignore\_unavailable=true, allow\_no\_indices=true, index=dailylogs-08-05-2019, type=logentry, track\_total\_hits=false}  
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:221) ~[elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:175) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:221) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:462) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:368) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:362) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_151]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_151]  
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_151]

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 11:27am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/6 "2019-05-13T11:27:34Z")

</div>

# Continuation of issue #3:

#3: Below Out of memory exception happened in our ElasticSearch instance and crashed. Why?

[2019-05-08T22:30:08,757][WARN][o.e.c.InternalClusterInfoService] [EIcXa59] Failed to update node information for ClusterInfoUpdateJob within 15s timeout  
[2019-05-08T22:30:41,784][ERROR][o.e.ExceptionsHelper] [EIcXa59] fatal error  
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:264)  
at java.util.Optional.ifPresent(Optional.java:159)  
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:254)  
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)  
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)  
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)  
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)  
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)  
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)  
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)  
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)  
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)  
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)  
at java.lang.Thread.run(Thread.java:748)

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 11:28am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/7 "2019-05-13T11:28:15Z")

</div>

[2019-05-08T22:30:47,538][ERROR][o.e.ExceptionsHelper] [EIcXa59] fatal error  
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:264)  
at java.util.Optional.ifPresent(Optional.java:159)  
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:254)  
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)  
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)  
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 11:29am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/8 "2019-05-13T11:29:19Z")

</div>

[2019-05-08T22:32:00,256][ERROR][o.e.ExceptionsHelper] [EIcXa59] fatal error  
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:264)  
at java.util.Optional.ifPresent(Optional.java:159)  
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:254)  
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)  
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)  
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)  
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)  
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)  
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)  
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)  
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)  
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)  
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)  
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)  
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)  
at java.lang.Thread.run(Thread.java:748)

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 13, 2019, 11:31am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/9 "2019-05-13T11:31:05Z")

</div>

@harshbajaj16 How much size could be increased? Could you please help to determine the heap size calculation if 20000 data rate per second to elasticsearch?

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [May 15, 2019, 6:35am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/10 "2019-05-15T06:35:12Z")

</div>

Hi @HPVRNP,

It is generally recommended to keep it at 50% of available RAM on the host or 30GB.

Regards,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 15, 2019, 7:10am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/11 "2019-05-15T07:10:13Z")

</div>

Heap size is not only related to size ngest rate, so how you map you data, how much data you have in the cluster and the number of indices and shards also affect this. You therefore need to provide more information about your use case.

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 15, 2019, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/12 "2019-05-15T08:44:03Z")

</div>

Node stats:  
{  
"\_nodes" : {  
"total" : 1,  
"successful" : 1,  
"failed" : 0  
},  
"cluster\_name" : "elasticsearch",  
"nodes" : {  
"O3eKzIqcQiKuRPvLtuZz\_A" : {  
"timestamp" : 1557902953445,  
"name" : "O3eKzIq",  
"transport\_address" : "192.10.10.1:9300",  
"host" : "MY-MCHN",  
"ip" : "x.x.x.x",  
"roles" : ["master", "data", "ingest"],  
"attributes" : {  
"ml.machine\_memory" : "274761269248",  
"xpack.installed" : "true",  
"ml.max\_open\_jobs" : "20",  
"ml.enabled" : "true"  
},  
"indices" : {  
"docs" : {  
"count" : 1002974077,  
"deleted" : 0  
},  
"store" : {  
"size" : "220.2gb",  
"size\_in\_bytes" : 236496959084  
},  
"indexing" : {  
"index\_total" : 104083683,  
"index\_time" : "5.3h",  
"index\_time\_in\_millis" : 19127525,  
"index\_current" : 0,  
"index\_failed" : 0,  
"delete\_total" : 0,  
"delete\_time" : "0s",  
"delete\_time\_in\_millis" : 0,  
"delete\_current" : 0,  
"noop\_update\_total" : 0,  
"is\_throttled" : false,  
"throttle\_time" : "1m",  
"throttle\_time\_in\_millis" : 60416  
},  
"get" : {  
"total" : 0,  
"getTime" : "0s",  
"time\_in\_millis" : 0,  
"exists\_total" : 0,  
"exists\_time" : "0s",  
"exists\_time\_in\_millis" : 0,  
"missing\_total" : 0,  
"missing\_time" : "0s",  
"missing\_time\_in\_millis" : 0,  
"current" : 0  
},  
"search" : {  
"open\_contexts" : 0,  
"query\_total" : 469173,  
"query\_time" : "21.6m",  
"query\_time\_in\_millis" : 1301400,  
"query\_current" : 0,  
"fetch\_total" : 468907,  
"fetch\_time" : "36.6m",  
"fetch\_time\_in\_millis" : 2201700,  
"fetch\_current" : 0,  
"scroll\_total" : 0,  
"scroll\_time" : "0s",  
"scroll\_time\_in\_millis" : 0,  
"scroll\_current" : 0,  
"suggest\_total" : 0,  
"suggest\_time" : "0s",  
"suggest\_time\_in\_millis" : 0,  
"suggest\_current" : 0  
},  
"merges" : {  
"current" : 0,  
"current\_docs" : 0,  
"current\_size" : "0b",  
"current\_size\_in\_bytes" : 0,  
"total" : 19579,  
"total\_time" : "7.1h",  
"total\_time\_in\_millis" : 25904621,  
"total\_docs" : 609194118,  
"total\_size" : "117.6gb",  
"total\_size\_in\_bytes" : 126360521178,  
"total\_stopped\_time" : "0s",  
"total\_stopped\_time\_in\_millis" : 0,  
"total\_throttled\_time" : "3.6h",  
"total\_throttled\_time\_in\_millis" : 13195889,  
"total\_auto\_throttle" : "575.2mb",  
"total\_auto\_throttle\_in\_bytes" : 603234718  
},  
"refresh" : {  
"total" : 167817,  
"total\_time" : "3.2h",  
"total\_time\_in\_millis" : 11866891,  
"listeners" : 0  
},  
"flush" : {  
"total" : 170,  
"periodic" : 134,  
"total\_time" : "1.9m",  
"total\_time\_in\_millis" : 114253  
},  
"warmer" : {  
"current" : 0,  
"total" : 163254,  
"total\_time" : "5.7s",  
"total\_time\_in\_millis" : 5751  
},  
"query\_cache" : {  
"memory\_size" : "12.4mb",  
"memory\_size\_in\_bytes" : 13063752,  
"total\_count" : 6771258,  
"hit\_count" : 2081890,  
"miss\_count" : 4689368,  
"cache\_size" : 51,  
"cache\_count" : 972,  
"evictions" : 921  
},  
"fielddata" : {  
"memory\_size" : "16.9kb",  
"memory\_size\_in\_bytes" : 17336,  
"evictions" : 0  
},  
"completion" : {  
"size" : "0b",  
"size\_in\_bytes" : 0  
},  
"segments" : {  
"count" : 675,  
"memory" : "513mb",  
"memory\_in\_bytes" : 537967429,  
"terms\_memory" : "347.4mb",  
"terms\_memory\_in\_bytes" : 364351224,  
"stored\_fields\_memory" : "141.7mb",  
"stored\_fields\_memory\_in\_bytes" : 148593144,  
"term\_vectors\_memory" : "0b",  
"term\_vectors\_memory\_in\_bytes" : 0,  
"norms\_memory" : "0b",  
"norms\_memory\_in\_bytes" : 0,  
"points\_memory" : "23.6mb",  
"points\_memory\_in\_bytes" : 24833953,  
"doc\_values\_memory" : "184.6kb",  
"doc\_values\_memory\_in\_bytes" : 189108,  
"index\_writer\_memory" : "0b",  
"index\_writer\_memory\_in\_bytes" : 0,  
"version\_map\_memory" : "0b",  
"version\_map\_memory\_in\_bytes" : 0,  
"fixed\_bit\_set" : "0b",  
"fixed\_bit\_set\_memory\_in\_bytes" : 0,  
"max\_unsafe\_auto\_id\_timestamp" : 1557628526839,  
"file\_sizes" : { }  
},  
"translog" : {  
"operations" : 952201,  
"size" : "419.7mb",  
"size\_in\_bytes" : 440175659,  
"uncommitted\_operations" : 318689,  
"uncommitted\_size" : "150.1mb",  
"uncommitted\_size\_in\_bytes" : 157474537,  
"earliest\_last\_modified\_age" : 0  
},  
"request\_cache" : {  
"memory\_size" : "37.8kb",  
"memory\_size\_in\_bytes" : 38771,  
"evictions" : 0,  
"hit\_count" : 250,  
"miss\_count" : 52  
},  
"recovery" : {  
"current\_as\_source" : 0,  
"current\_as\_target" : 0,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
}  
},

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 15, 2019, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/13 "2019-05-15T08:44:48Z")

</div>

```
  "os" : {
    "timestamp" : 1557902953621,
    "cpu" : {
      "percent" : 13
    },
    "mem" : {
      "total" : "255.8gb",
      "total_in_bytes" : 274761269248,
      "free" : "204.6gb",
      "free_in_bytes" : 219749654528,
      "used" : "51.2gb",
      "used_in_bytes" : 55011614720,
      "free_percent" : 80,
      "used_percent" : 20
    },
    "swap" : {
      "total" : "291.3gb",
      "total_in_bytes" : 312864423936,
      "free" : "238gb",
      "free_in_bytes" : 255654916096,
      "used" : "53.2gb",
      "used_in_bytes" : 57209507840
    }
  },
  "process" : {
    "timestamp" : 1557902953631,
    "open_file_descriptors" : -1,
    "max_file_descriptors" : -1,
    "cpu" : {
      "percent" : 1,
      "total" : "2.3d",
      "total_in_millis" : 199910796
    },
    "mem" : {
      "total_virtual" : "2gb",
      "total_virtual_in_bytes" : 2212380672
    }
  },
  "jvm" : {
    "timestamp" : 1557902953632,
    "uptime" : "3.1d",
    "uptime_in_millis" : 274440608,
    "mem" : {
      "heap_used" : "920.3mb",
      "heap_used_in_bytes" : 965051480,
      "heap_used_percent" : 92,
      "heap_committed" : "989.8mb",
      "heap_committed_in_bytes" : 1037959168,
      "heap_max" : "989.8mb",
      "heap_max_in_bytes" : 1037959168,
      "non_heap_used" : "143.8mb",
      "non_heap_used_in_bytes" : 150858472,
      "non_heap_committed" : "154.6mb",
      "non_heap_committed_in_bytes" : 162185216,
      "pools" : {
        "young" : {
          "used" : "238.7mb",
          "used_in_bytes" : 250304992,
          "max" : "273mb",
          "max_in_bytes" : 286326784,
          "peak_used" : "273mb",
          "peak_used_in_bytes" : 286326784,
          "peak_max" : "273mb",
          "peak_max_in_bytes" : 286326784
        },
```

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 15, 2019, 8:45am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/14 "2019-05-15T08:45:27Z")

</div>

```
        "survivor" : {
          "used" : "0b",
          "used_in_bytes" : 0,
          "max" : "34.1mb",
          "max_in_bytes" : 35782656,
          "peak_used" : "34.1mb",
          "peak_used_in_bytes" : 35782656,
          "peak_max" : "34.1mb",
          "peak_max_in_bytes" : 35782656
        },
        "old" : {
          "used" : "681.6mb",
          "used_in_bytes" : 714746488,
          "max" : "682.6mb",
          "max_in_bytes" : 715849728,
          "peak_used" : "682.6mb",
          "peak_used_in_bytes" : 715849728,
          "peak_max" : "682.6mb",
          "peak_max_in_bytes" : 715849728
        }
      }
    },
    "threads" : {
      "count" : 353,
      "peak_count" : 372
    },
  "gc" : {
      "collectors" : {
        "young" : {
          "collection_count" : 34856,
          "collection_time" : "913ms",
          "collection_time_in_millis" : 913
        },
        "old" : {
          "collection_count" : 171882,
          "collection_time" : "6.8h",
          "collection_time_in_millis" : 24509028
        }
      }
    },
    "buffer_pools" : {
      "direct" : {
        "count" : 215,
        "used" : "167.9mb",
        "used_in_bytes" : 176142278,
        "total_capacity" : "167.9mb",
        "total_capacity_in_bytes" : 176142277
      },
      "mapped" : {
        "count" : 1335,
        "used" : "219gb",
        "used_in_bytes" : 235229473374,
        "total_capacity" : "219gb",
        "total_capacity_in_bytes" : 235229473374
      }
    },
    "classes" : {
      "current_loaded_count" : 16529,
      "total_loaded_count" : 17158,
      "total_unloaded_count" : 629
    }
  },
  "thread_pool" : {
    "analyze" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "ccr" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "fetch_shard_started" : {
      "threads" : 1,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 28,
      "completed" : 28
    },
    "fetch_shard_store" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "flush" : {
      "threads" : 1,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 5,
      "completed" : 1133
    },
    "force_merge" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "generic" : {
      "threads" : 29,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 29,
      "completed" : 856771
    },
    "get" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "index" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "listener" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "management" : {
      "threads" : 3,
      "queue" : 0,
      "active" : 1,
      "rejected" : 0,
      "largest" : 3,
      "completed" : 36552
    },
```

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 15, 2019, 8:45am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/15 "2019-05-15T08:45:59Z")

</div>

```
   "ml_autodetect" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "ml_datafeed" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "ml_utility" : {
      "threads" : 1,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 1,
      "completed" : 1
    },
    "refresh" : {
      "threads" : 6,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 10,
      "completed" : 167428
    },
    "rollup_indexing" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "search" : {
      "threads" : 85,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 85,
      "completed" : 938104
    },
    "search_throttled" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "security-token-key" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "snapshot" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "warmer" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "watcher" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "write" : {
      "threads" : 56,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 56,
      "completed" : 184270
    }
  },
```

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 15, 2019, 8:46am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/16 "2019-05-15T08:46:06Z")

</div>

"fs" : {  
"timestamp" : 1557902953633,  
"total" : {  
"total" : "291tb",  
"total\_in\_bytes" : 320040494694400,  
"free" : "186.7tb",  
"free\_in\_bytes" : 205350520750080,  
"available" : "186.7tb",  
"available\_in\_bytes" : 205350520750080  
},  
"least\_usage\_estimate" : {  
"path" : "F:\data\nodes\0",  
"total" : "291tb",  
"total\_in\_bytes" : 320040494694400,  
"available" : "186.7tb",  
"available\_in\_bytes" : 205350540345344,  
"used\_disk\_percent" : 35.83607582489556  
},  
"most\_usage\_estimate" : {  
"path" : "F:\\data\nodes\0",  
"total" : "291tb",  
"total\_in\_bytes" : 320040494694400,  
"available" : "186.7tb",  
"available\_in\_bytes" : 205350540345344,  
"used\_disk\_percent" : 35.83607582489556  
},  
"data" : [  
{  
"path" : "F:\data\nodes\0",  
"mount" : "User\_Data (E:)",  
"type" : "ReFS",  
"total" : "291tb",  
"total\_in\_bytes" : 320040494694400,  
"free" : "186.7tb",  
"free\_in\_bytes" : 205350520750080,  
"available" : "186.7tb",  
"available\_in\_bytes" : 205350520750080  
}  
]  
},  
"transport" : {  
"server\_open" : 0,  
"rx\_count" : 0,  
"rx\_size" : "0b",  
"rx\_size\_in\_bytes" : 0,  
"tx\_count" : 0,  
"tx\_size" : "0b",  
"tx\_size\_in\_bytes" : 0  
},  
"http" : {  
"current\_open" : 9,  
"total\_opened" : 1081  
},  
"breakers" : {  
"request" : {  
"limit\_size\_in\_bytes" : 622775500,  
"limit\_size" : "593.9mb",  
"estimated\_size\_in\_bytes" : 0,  
"estimated\_size" : "0b",  
"overhead" : 1.0,  
"tripped" : 0  
},  
"fielddata" : {  
"limit\_size\_in\_bytes" : 622775500,  
"limit\_size" : "593.9mb",  
"estimated\_size\_in\_bytes" : 17336,  
"estimated\_size" : "16.9kb",  
"overhead" : 1.03,  
"tripped" : 0  
},  
"in\_flight\_requests" : {  
"limit\_size\_in\_bytes" : 1037959168,  
"limit\_size" : "989.8mb",  
"estimated\_size\_in\_bytes" : 0,  
"estimated\_size" : "0b",  
"overhead" : 1.0,  
"tripped" : 0  
},  
"accounting" : {  
"limit\_size\_in\_bytes" : 1037959168,  
"limit\_size" : "989.8mb",  
"estimated\_size\_in\_bytes" : 537967429,  
"estimated\_size" : "513mb",  
"overhead" : 1.0,  
"tripped" : 0  
},  
"parent" : {  
"limit\_size\_in\_bytes" : 726571417,  
"limit\_size" : "692.9mb",  
"estimated\_size\_in\_bytes" : 537984765,  
"estimated\_size" : "513mb",  
"overhead" : 1.0,  
"tripped" : 0  
}  
},  
"script" : {  
"compilations" : 4,  
"cache\_evictions" : 0  
},  
"discovery" : {  
"cluster\_state\_queue" : {  
"total" : 0,  
"pending" : 0,  
"committed" : 0  
},  
"published\_cluster\_states" : {  
"full\_states" : 0,  
"incompatible\_diffs" : 0,  
"compatible\_diffs" : 0  
}  
},  
"ingest" : {  
"total" : {  
"count" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0,  
"current" : 0,  
"failed" : 0  
},  
"pipelines" : {  
"xpack\_monitoring\_2" : {  
"count" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0,  
"current" : 0,  
"failed" : 0,  
"processors" : [  
{  
"script" : {  
"count" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0,  
"current" : 0,  
"failed" : 0  
}  
},  
{  
"rename" : {  
"count" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0,  
"current" : 0,  
"failed" : 0  
}  
},  
{  
"set" : {  
"count" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0,  
"current" : 0,  
"failed" : 0  
}  
},  
{  
"gsub" : {  
"count" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0,  
"current" : 0,  
"failed" : 0  
}  
}  
] },  
"xpack\_monitoring\_6" : {  
"count" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0,  
"current" : 0,  
"failed" : 0,  
"processors" :   
}  
}  
},  
"adaptive\_selection" : {  
"O3eKzIqcQiKuRPvLtuZz\_A" : {  
"outgoing\_searches" : 0,  
"avg\_queue\_size" : 0,  
"avg\_service\_time" : "361.8micros",  
"avg\_service\_time\_ns" : 361862,  
"avg\_response\_time" : "851micros",  
"avg\_response\_time\_ns" : 851097,  
"rank" : "0.9"  
}  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 15, 2019, 9:07am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/17 "2019-05-15T09:07:25Z")

</div>

You have a very small heap configured so I would recommend you increase this significant ficantly.

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 15, 2019, 9:23am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/18 "2019-05-15T09:23:49Z")

</div>

Thanks Christian. Could you please give suggestions on heap size based on the stats?

---

<div class="post-metadata">

**Author:** ![HPVRNP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hpvrnp/32/37756_2.png) [@HPVRNP](https://discuss.elastic.co/u/HPVRNP)\
**Post date:** [May 15, 2019, 9:28am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/19 "2019-05-15T09:28:09Z")

</div>

We have day wise indexing and single type mapping. On an average, there would be 20GB data stored in an index per day (single shard indexing). At the max, we keep 45 indices which means we keep last 45 days of data in ElasticSearch. We delete all the old indices periodically.

What could be the optimal heap size for our use case?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 15, 2019, 9:36am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781/20 "2019-05-15T09:36:36Z")

</div>

It is hard to tell. What is the specification of the host it is running on? I would recommend increasing it gradually until you stop having issues. If you have swap enabled you should also disable this.

[Next page](https://discuss.elastic.co/t/elasticsearch-crashes-in-single-node-cluster-issue-1/180781.md?page=2)