ElasticSearch crashes in single node cluster- Issue #1

Elastic Stack Elasticsearch
1

Hi,

Environment:
OS : Windows 64-bit
ElasticSearch : 6.6.2
Java: 1.8.151
Cluster: Single Node cluster
Index strategy: Indices are created daily and programmatically
Shard: Single shard per index and no replication

We have been using elasticsearch for more than a month.
The machine contains approximately 30 indices (data for 30 days).
Total size of all the indices is around 100s of GBs.

Recently, We have been facing elasticsearch crashes in most of the machines.

Few queries from logs:

=============================================================================================================================================
#1: Why GC is running frequently? Why does it consume time in seconds for Garbage collection?
There is enough space remaining in the machine. And data rate is almost same in most of the days.
Why is JVM crash because of out of memory exception?

[2019-05-08T02:18:58,275][WARN ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][12982] overhead, spent [513ms] collecting in the last [1s]
[2019-05-08T02:19:55,482][INFO ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13039] overhead, spent [376ms] collecting in the last [1.1s]
[2019-05-08T02:21:14,601][INFO ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13118] overhead, spent [441ms] collecting in the last [1s]
[2019-05-08T02:22:05,676][INFO ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13169] overhead, spent [491ms] collecting in the last [1s]
[2019-05-08T02:22:08,679][INFO ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13172] overhead, spent [475ms] collecting in the last [1s]
[2019-05-08T02:22:10,097][WARN ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13173] overhead, spent [844ms] collecting in the last [1.4s]
[2019-05-08T02:22:14,684][WARN ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13174] overhead, spent [3.4s] collecting in the last [3.4s]
[2019-05-08T02:22:15,829][WARN ][o.e.m.j.JvmGcMonitorService] [EIcXa59] [gc][13175] overhead, spent [2.2s] collecting in the last [2.2s]
[2019-05-08T02:22:15,828][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [EIcXa59] fatal error in thread [elasticsearch[EIcXa59][search][T#78]], exiting
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:111) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:108) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:53) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.recycler.AbstractRecycler.obtain(AbstractRecycler.java:33) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:28) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.recycler.Recyclers$3.obtain(Recyclers.java:119) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.PageCacheRecycler.bytePage(PageCacheRecycler.java:159) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.AbstractBigArray.newBytePage(AbstractBigArray.java:117) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.BigByteArray.resize(BigByteArray.java:143) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.BigArrays.resizeInPlace(BigArrays.java:440) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.BigArrays.resize(BigArrays.java:487) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.BigArrays.grow(BigArrays.java:505) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.io.stream.BytesStreamOutput.ensureCapacity(BytesStreamOutput.java:158) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.ensureCapacity(ReleasableBytesStreamOutput.java:70) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:90) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.io.Streams$FlushOnCloseOutputStream.writeBytes(Streams.java:266) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:460) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.copyStream(JsonXContentGenerator.java:502) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeRawField(JsonXContentGenerator.java:352) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeRawField(JsonXContentGenerator.java:333) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.rawField(XContentBuilder.java:967) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentHelper.writeRawField(XContentHelper.java:308) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchHit.toInnerXContent(SearchHit.java:466) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchHit.toXContent(SearchHit.java:408) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchHits.toXContent(SearchHits.java:158) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.SearchResponseSections.toXContent(SearchResponseSections.java:106) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.SearchResponse.innerToXContent(SearchResponse.java:241) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.SearchResponse.toXContent(SearchResponse.java:221) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:57) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:33) ~[elasticsearch-6.6.2.jar:6.6.2]

Thanks,
HGVP

2

HI @HPVRNP

It looks like you are suffering from heap pressure.

I would recommend trying to increase the size of the heap and check.

Regards,
Harsh Bajaj

3

#2 We have been running same queries programmatically through our custom application daily.
Why are these errors coming all of a sudden? When will these issues arise?

[2019-05-08T02:23:03,516][WARN ][r.suppressed ] [EIcXa59] path: /dailylogs-08-05-2019/logentry/_search, params: {typed_keys=true, ignore_unavailable=true, allow_no_indices=true, index=dailylogs-08-05-2019, type=logentry, track_total_hits=false}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

4

[2019-05-08T02:23:04,029][DEBUG][o.e.a.s.TransportSearchAction] [EIcXa59] All shards failed for phase: [query]
[2019-05-08T02:23:04,029][WARN ][r.suppressed ] [EIcXa59] path: /dailylogs-08-05-2019/logentry/_search, params: {typed_keys=true, ignore_unavailable=true, allow_no_indices=true, index=dailylogs-08-05-2019, type=logentry, track_total_hits=false}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:221) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:175) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:221) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:462) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.2.jar:6.6.2]

5 
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:368) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:362) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

[2019-05-08T02:23:04,536][DEBUG][o.e.a.s.TransportSearchAction] [EIcXa59] All shards failed for phase: [query]
[2019-05-08T02:23:04,536][WARN ][r.suppressed ] [EIcXa59] path: /dailylogs-08-05-2019/logentry/_search, params: {typed_keys=true, ignore_unavailable=true, allow_no_indices=true, index=dailylogs-08-05-2019, type=logentry, track_total_hits=false}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:221) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:175) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:221) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:462) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:368) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:362) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

6

Continuation of issue #3:

#3: Below Out of memory exception happened in our ElasticSearch instance and crashed. Why?

[2019-05-08T22:30:08,757][WARN ][o.e.c.InternalClusterInfoService] [EIcXa59] Failed to update node information for ClusterInfoUpdateJob within 15s timeout
[2019-05-08T22:30:41,784][ERROR][o.e.ExceptionsHelper ] [EIcXa59] fatal error
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:264)
at java.util.Optional.ifPresent(Optional.java:159)
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:254)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.lang.Thread.run(Thread.java:748)

7

[2019-05-08T22:30:47,538][ERROR][o.e.ExceptionsHelper ] [EIcXa59] fatal error
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:264)
at java.util.Optional.ifPresent(Optional.java:159)
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:254)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at

8

[2019-05-08T22:32:00,256][ERROR][o.e.ExceptionsHelper ] [EIcXa59] fatal error
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:264)
at java.util.Optional.ifPresent(Optional.java:159)
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:254)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.lang.Thread.run(Thread.java:748)

9

@harshbajaj16 How much size could be increased? Could you please help to determine the heap size calculation if 20000 data rate per second to elasticsearch?

10

Hi @HPVRNP,

It is generally recommended to keep it at 50% of available RAM on the host or 30GB.

Regards,
Harsh Bajaj

1 Like
11

Heap size is not only related to size ngest rate, so how you map you data, how much data you have in the cluster and the number of indices and shards also affect this. You therefore need to provide more information about your use case.

12

Node stats:
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "elasticsearch",
"nodes" : {
"O3eKzIqcQiKuRPvLtuZz_A" : {
"timestamp" : 1557902953445,
"name" : "O3eKzIq",
"transport_address" : "192.10.10.1:9300",
"host" : "MY-MCHN",
"ip" : "x.x.x.x",
"roles" : [ "master", "data", "ingest" ],
"attributes" : {
"ml.machine_memory" : "274761269248",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"indices" : {
"docs" : {
"count" : 1002974077,
"deleted" : 0
},
"store" : {
"size" : "220.2gb",
"size_in_bytes" : 236496959084
},
"indexing" : {
"index_total" : 104083683,
"index_time" : "5.3h",
"index_time_in_millis" : 19127525,
"index_current" : 0,
"index_failed" : 0,
"delete_total" : 0,
"delete_time" : "0s",
"delete_time_in_millis" : 0,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time" : "1m",
"throttle_time_in_millis" : 60416
},
"get" : {
"total" : 0,
"getTime" : "0s",
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time" : "0s",
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 469173,
"query_time" : "21.6m",
"query_time_in_millis" : 1301400,
"query_current" : 0,
"fetch_total" : 468907,
"fetch_time" : "36.6m",
"fetch_time_in_millis" : 2201700,
"fetch_current" : 0,
"scroll_total" : 0,
"scroll_time" : "0s",
"scroll_time_in_millis" : 0,
"scroll_current" : 0,
"suggest_total" : 0,
"suggest_time" : "0s",
"suggest_time_in_millis" : 0,
"suggest_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size" : "0b",
"current_size_in_bytes" : 0,
"total" : 19579,
"total_time" : "7.1h",
"total_time_in_millis" : 25904621,
"total_docs" : 609194118,
"total_size" : "117.6gb",
"total_size_in_bytes" : 126360521178,
"total_stopped_time" : "0s",
"total_stopped_time_in_millis" : 0,
"total_throttled_time" : "3.6h",
"total_throttled_time_in_millis" : 13195889,
"total_auto_throttle" : "575.2mb",
"total_auto_throttle_in_bytes" : 603234718
},
"refresh" : {
"total" : 167817,
"total_time" : "3.2h",
"total_time_in_millis" : 11866891,
"listeners" : 0
},
"flush" : {
"total" : 170,
"periodic" : 134,
"total_time" : "1.9m",
"total_time_in_millis" : 114253
},
"warmer" : {
"current" : 0,
"total" : 163254,
"total_time" : "5.7s",
"total_time_in_millis" : 5751
},
"query_cache" : {
"memory_size" : "12.4mb",
"memory_size_in_bytes" : 13063752,
"total_count" : 6771258,
"hit_count" : 2081890,
"miss_count" : 4689368,
"cache_size" : 51,
"cache_count" : 972,
"evictions" : 921
},
"fielddata" : {
"memory_size" : "16.9kb",
"memory_size_in_bytes" : 17336,
"evictions" : 0
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 675,
"memory" : "513mb",
"memory_in_bytes" : 537967429,
"terms_memory" : "347.4mb",
"terms_memory_in_bytes" : 364351224,
"stored_fields_memory" : "141.7mb",
"stored_fields_memory_in_bytes" : 148593144,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "0b",
"norms_memory_in_bytes" : 0,
"points_memory" : "23.6mb",
"points_memory_in_bytes" : 24833953,
"doc_values_memory" : "184.6kb",
"doc_values_memory_in_bytes" : 189108,
"index_writer_memory" : "0b",
"index_writer_memory_in_bytes" : 0,
"version_map_memory" : "0b",
"version_map_memory_in_bytes" : 0,
"fixed_bit_set" : "0b",
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : 1557628526839,
"file_sizes" : { }
},
"translog" : {
"operations" : 952201,
"size" : "419.7mb",
"size_in_bytes" : 440175659,
"uncommitted_operations" : 318689,
"uncommitted_size" : "150.1mb",
"uncommitted_size_in_bytes" : 157474537,
"earliest_last_modified_age" : 0
},
"request_cache" : {
"memory_size" : "37.8kb",
"memory_size_in_bytes" : 38771,
"evictions" : 0,
"hit_count" : 250,
"miss_count" : 52
},
"recovery" : {
"current_as_source" : 0,
"current_as_target" : 0,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
}
},

13 
  "os" : {
    "timestamp" : 1557902953621,
    "cpu" : {
      "percent" : 13
    },
    "mem" : {
      "total" : "255.8gb",
      "total_in_bytes" : 274761269248,
      "free" : "204.6gb",
      "free_in_bytes" : 219749654528,
      "used" : "51.2gb",
      "used_in_bytes" : 55011614720,
      "free_percent" : 80,
      "used_percent" : 20
    },
    "swap" : {
      "total" : "291.3gb",
      "total_in_bytes" : 312864423936,
      "free" : "238gb",
      "free_in_bytes" : 255654916096,
      "used" : "53.2gb",
      "used_in_bytes" : 57209507840
    }
  },
  "process" : {
    "timestamp" : 1557902953631,
    "open_file_descriptors" : -1,
    "max_file_descriptors" : -1,
    "cpu" : {
      "percent" : 1,
      "total" : "2.3d",
      "total_in_millis" : 199910796
    },
    "mem" : {
      "total_virtual" : "2gb",
      "total_virtual_in_bytes" : 2212380672
    }
  },
  "jvm" : {
    "timestamp" : 1557902953632,
    "uptime" : "3.1d",
    "uptime_in_millis" : 274440608,
    "mem" : {
      "heap_used" : "920.3mb",
      "heap_used_in_bytes" : 965051480,
      "heap_used_percent" : 92,
      "heap_committed" : "989.8mb",
      "heap_committed_in_bytes" : 1037959168,
      "heap_max" : "989.8mb",
      "heap_max_in_bytes" : 1037959168,
      "non_heap_used" : "143.8mb",
      "non_heap_used_in_bytes" : 150858472,
      "non_heap_committed" : "154.6mb",
      "non_heap_committed_in_bytes" : 162185216,
      "pools" : {
        "young" : {
          "used" : "238.7mb",
          "used_in_bytes" : 250304992,
          "max" : "273mb",
          "max_in_bytes" : 286326784,
          "peak_used" : "273mb",
          "peak_used_in_bytes" : 286326784,
          "peak_max" : "273mb",
          "peak_max_in_bytes" : 286326784
        },
14 
        "survivor" : {
          "used" : "0b",
          "used_in_bytes" : 0,
          "max" : "34.1mb",
          "max_in_bytes" : 35782656,
          "peak_used" : "34.1mb",
          "peak_used_in_bytes" : 35782656,
          "peak_max" : "34.1mb",
          "peak_max_in_bytes" : 35782656
        },
        "old" : {
          "used" : "681.6mb",
          "used_in_bytes" : 714746488,
          "max" : "682.6mb",
          "max_in_bytes" : 715849728,
          "peak_used" : "682.6mb",
          "peak_used_in_bytes" : 715849728,
          "peak_max" : "682.6mb",
          "peak_max_in_bytes" : 715849728
        }
      }
    },
    "threads" : {
      "count" : 353,
      "peak_count" : 372
    },
  "gc" : {
      "collectors" : {
        "young" : {
          "collection_count" : 34856,
          "collection_time" : "913ms",
          "collection_time_in_millis" : 913
        },
        "old" : {
          "collection_count" : 171882,
          "collection_time" : "6.8h",
          "collection_time_in_millis" : 24509028
        }
      }
    },
    "buffer_pools" : {
      "direct" : {
        "count" : 215,
        "used" : "167.9mb",
        "used_in_bytes" : 176142278,
        "total_capacity" : "167.9mb",
        "total_capacity_in_bytes" : 176142277
      },
      "mapped" : {
        "count" : 1335,
        "used" : "219gb",
        "used_in_bytes" : 235229473374,
        "total_capacity" : "219gb",
        "total_capacity_in_bytes" : 235229473374
      }
    },
    "classes" : {
      "current_loaded_count" : 16529,
      "total_loaded_count" : 17158,
      "total_unloaded_count" : 629
    }
  },
  "thread_pool" : {
    "analyze" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "ccr" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "fetch_shard_started" : {
      "threads" : 1,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 28,
      "completed" : 28
    },
    "fetch_shard_store" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "flush" : {
      "threads" : 1,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 5,
      "completed" : 1133
    },
    "force_merge" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "generic" : {
      "threads" : 29,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 29,
      "completed" : 856771
    },
    "get" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "index" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "listener" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "management" : {
      "threads" : 3,
      "queue" : 0,
      "active" : 1,
      "rejected" : 0,
      "largest" : 3,
      "completed" : 36552
    },
15 
   "ml_autodetect" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "ml_datafeed" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "ml_utility" : {
      "threads" : 1,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 1,
      "completed" : 1
    },
    "refresh" : {
      "threads" : 6,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 10,
      "completed" : 167428
    },
    "rollup_indexing" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "search" : {
      "threads" : 85,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 85,
      "completed" : 938104
    },
    "search_throttled" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "security-token-key" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "snapshot" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "warmer" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "watcher" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "write" : {
      "threads" : 56,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 56,
      "completed" : 184270
    }
  },
16

"fs" : {
"timestamp" : 1557902953633,
"total" : {
"total" : "291tb",
"total_in_bytes" : 320040494694400,
"free" : "186.7tb",
"free_in_bytes" : 205350520750080,
"available" : "186.7tb",
"available_in_bytes" : 205350520750080
},
"least_usage_estimate" : {
"path" : "F:\data\nodes\0",
"total" : "291tb",
"total_in_bytes" : 320040494694400,
"available" : "186.7tb",
"available_in_bytes" : 205350540345344,
"used_disk_percent" : 35.83607582489556
},
"most_usage_estimate" : {
"path" : "F:\\data\nodes\0",
"total" : "291tb",
"total_in_bytes" : 320040494694400,
"available" : "186.7tb",
"available_in_bytes" : 205350540345344,
"used_disk_percent" : 35.83607582489556
},
"data" : [
{
"path" : "F:\data\nodes\0",
"mount" : "User_Data (E:)",
"type" : "ReFS",
"total" : "291tb",
"total_in_bytes" : 320040494694400,
"free" : "186.7tb",
"free_in_bytes" : 205350520750080,
"available" : "186.7tb",
"available_in_bytes" : 205350520750080
}
]
},
"transport" : {
"server_open" : 0,
"rx_count" : 0,
"rx_size" : "0b",
"rx_size_in_bytes" : 0,
"tx_count" : 0,
"tx_size" : "0b",
"tx_size_in_bytes" : 0
},
"http" : {
"current_open" : 9,
"total_opened" : 1081
},
"breakers" : {
"request" : {
"limit_size_in_bytes" : 622775500,
"limit_size" : "593.9mb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.0,
"tripped" : 0
},
"fielddata" : {
"limit_size_in_bytes" : 622775500,
"limit_size" : "593.9mb",
"estimated_size_in_bytes" : 17336,
"estimated_size" : "16.9kb",
"overhead" : 1.03,
"tripped" : 0
},
"in_flight_requests" : {
"limit_size_in_bytes" : 1037959168,
"limit_size" : "989.8mb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.0,
"tripped" : 0
},
"accounting" : {
"limit_size_in_bytes" : 1037959168,
"limit_size" : "989.8mb",
"estimated_size_in_bytes" : 537967429,
"estimated_size" : "513mb",
"overhead" : 1.0,
"tripped" : 0
},
"parent" : {
"limit_size_in_bytes" : 726571417,
"limit_size" : "692.9mb",
"estimated_size_in_bytes" : 537984765,
"estimated_size" : "513mb",
"overhead" : 1.0,
"tripped" : 0
}
},
"script" : {
"compilations" : 4,
"cache_evictions" : 0
},
"discovery" : {
"cluster_state_queue" : {
"total" : 0,
"pending" : 0,
"committed" : 0
},
"published_cluster_states" : {
"full_states" : 0,
"incompatible_diffs" : 0,
"compatible_diffs" : 0
}
},
"ingest" : {
"total" : {
"count" : 0,
"time" : "0s",
"time_in_millis" : 0,
"current" : 0,
"failed" : 0
},
"pipelines" : {
"xpack_monitoring_2" : {
"count" : 0,
"time" : "0s",
"time_in_millis" : 0,
"current" : 0,
"failed" : 0,
"processors" : [
{
"script" : {
"count" : 0,
"time" : "0s",
"time_in_millis" : 0,
"current" : 0,
"failed" : 0
}
},
{
"rename" : {
"count" : 0,
"time" : "0s",
"time_in_millis" : 0,
"current" : 0,
"failed" : 0
}
},
{
"set" : {
"count" : 0,
"time" : "0s",
"time_in_millis" : 0,
"current" : 0,
"failed" : 0
}
},
{
"gsub" : {
"count" : 0,
"time" : "0s",
"time_in_millis" : 0,
"current" : 0,
"failed" : 0
}
}
] },
"xpack_monitoring_6" : {
"count" : 0,
"time" : "0s",
"time_in_millis" : 0,
"current" : 0,
"failed" : 0,
"processors" :
}
}
},
"adaptive_selection" : {
"O3eKzIqcQiKuRPvLtuZz_A" : {
"outgoing_searches" : 0,
"avg_queue_size" : 0,
"avg_service_time" : "361.8micros",
"avg_service_time_ns" : 361862,
"avg_response_time" : "851micros",
"avg_response_time_ns" : 851097,
"rank" : "0.9"
}
}
}
}
}

17

You have a very small heap configured so I would recommend you increase this significant ficantly.

18

Thanks Christian. Could you please give suggestions on heap size based on the stats?

19

We have day wise indexing and single type mapping. On an average, there would be 20GB data stored in an index per day (single shard indexing). At the max, we keep 45 indices which means we keep last 45 days of data in ElasticSearch. We delete all the old indices periodically.

What could be the optimal heap size for our use case?

20

It is hard to tell. What is the specification of the host it is running on? I would recommend increasing it gradually until you stop having issues. If you have swap enabled you should also disable this.