# ES 2.3.3 - Cannot understand why OOM

**URL:** <https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818>\
**Category:** Elasticsearch\
**Created:** [March 8, 2017, 1:11pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818 "2017-03-08T13:11:50Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rzrucher](https://avatars.discourse-cdn.com/v4/letter/r/76d3ee/32.png) [@rzrucher](https://discuss.elastic.co/u/rzrucher)\
**Post date:** [March 8, 2017, 1:11pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/1 "2017-03-08T13:11:50Z")

</div>

I've a single node installation of ES with version 2.3.3 and I get an OOM. But I cannot understand why this OOM occurs. For now the server is only doing indexing (no search in progress).

Can someone give my a clue to understand ?

As we can see in the logfile, we have a lot of GC. I do bulk load by batch of 5Mb why the heap is using 29Gb !!!

---

<div class="post-metadata">

**Author:** ![rzrucher](https://avatars.discourse-cdn.com/v4/letter/r/76d3ee/32.png) [@rzrucher](https://discuss.elastic.co/u/rzrucher)\
**Post date:** [March 8, 2017, 1:14pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/2 "2017-03-08T13:14:38Z")

</div>

```
Here is the log (I've many other CG before these) : 

    [2017-03-08 12:02:24,974][WARN][monitor.jvm] [NGELK1] [gc][old][84676][159] duration [38.4s], collections [3]/[38.5s], total [38.4s]/[30.4m], memory [29.1gb]->[29.1gb]/[29.1gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [106.6mb]->[106.1mb]/[108.1mb]}{[old] [28.2gb]->[28.2gb]/[28.2gb]}
    [2017-03-08 12:03:12,946][WARN][monitor.jvm] [NGELK1] [gc][old][84677][162] duration [34.1s], collections [3]/[34.3s], total [34.1s]/[31m], memory [29.1gb]->[29.1gb]/[29.1gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [106.1mb]->[108mb]/[108.1mb]}{[old] [28.2gb]->[28.2gb]/[28.2gb]}
    [2017-03-08 12:48:40,642][WARN][transport.netty] [NGELK1] exception caught on transport layer [[id: 0x175ecf80, /127.0.0.1:60584 :> /127.0.0.1:9300]], closing connection
    java.lang.OutOfMemoryError: Java heap space
    	at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:33)
    	at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
    	at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
    	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:445)
    	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:128)
    	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    	at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
    	at org.jboss.netty.channel.Channels.write(Channels.java:704)
    	at org.jboss.netty.channel.Channels.write(Channels.java:671)
    	at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
    	at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:103)
    	at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:75)
    	at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58)
    	at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:134)
    	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:65)
    	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:61)
    	at org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:89)
    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    [2017-03-08 12:48:40,774][WARN][index.engine] [NGELK1] [ngt-liveserver-20161214][0] failed engine [lucene commit failed]
    java.lang.OutOfMemoryError: Java heap space
    [2017-03-08 12:48:40,778][WARN][index.shard] [NGELK1] [ngt-liveserver-20161214][0] failed to refresh after decreasing index buffer
    [ngt-liveserver-20161214][[ngt-liveserver-20161214][0]] EngineClosedException[CurrentState[CLOSED] Closed]
    	at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:329)
    	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:674)
    	at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:661)
    	at org.elasticsearch.index.shard.IndexShard.updateBufferSize(IndexShard.java:1155)
    	at org.elasticsearch.index.shard.IndexShard.checkIdle(IndexShard.java:1183)
    	at org.elasticsearch.indices.memory.IndexingMemoryController.checkIdle(IndexingMemoryController.java:302)
    	at org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:254)
    	at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:640)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    [2017-03-08 12:48:40,849][WARN][indices.cluster] [NGELK1] [[ngt-liveserver-20161214][0]] marking and sending shard failed due to [engine failure, reason [lucene commit failed]]
    java.lang.OutOfMemoryError: Java heap space
    [2017-03-08 12:48:40,825][WARN][index.translog] [NGELK1] [ngt-liveserver-20161214][0] failed to flush shard on translog threshold
    [ngt-liveserver-20161214][[ngt-liveserver-20161214][0]] FlushFailedEngineException[Flush failed]; nested: OutOfMemoryError[Java heap space];
    	at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:765)
    	at org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:782)
    	at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.doRun(TranslogService.java:222)
    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    Caused by: java.lang.OutOfMemoryError: Java heap space
    [2017-03-08 12:48:40,937][WARN][cluster.action.shard] [NGELK1] [ngt-liveserver-20161214][0] received shard failed for target shard [[ngt-liveserver-20161214][0], node[3AdkqZSIRsODeBvxSOJXfQ], [P], v[2], s[STARTED], a[id=RgbNorZTT32rz3fiVIsASg]], indexUUID [xCcQt5q5Qficvhz-FDGz_A], message [engine failure, reason [lucene commit failed]], failure [OutOfMemoryError[Java heap space]]
    java.lang.OutOfMemoryError: Java heap space
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 8, 2017, 1:32pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/3 "2017-03-08T13:32:50Z")

</div>

What is your use case? Do you have a lot of fields with analysed text that uses up heap or are you primarily using doc\_values? How many indices and shards do you have?

---

<div class="post-metadata">

**Author:** ![rzrucher](https://avatars.discourse-cdn.com/v4/letter/r/76d3ee/32.png) [@rzrucher](https://discuss.elastic.co/u/rzrucher)\
**Post date:** [March 8, 2017, 2:32pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/4 "2017-03-08T14:32:44Z")

</div>

My use case is to index many events with limited number of fields (~80) to be able to analyze them later in kibana.

All string fields are not\_analyzed.

For now, I've about 2billions of documents in around 200 indices (1 shard per index).

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 8, 2017, 2:35pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/5 "2017-03-08T14:35:31Z")

</div>

Do you have any custom config settings in your elasticsearch.yml file? What is your heap size?

---

<div class="post-metadata">

**Author:** ![rzrucher](https://avatars.discourse-cdn.com/v4/letter/r/76d3ee/32.png) [@rzrucher](https://discuss.elastic.co/u/rzrucher)\
**Post date:** [March 8, 2017, 2:38pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/6 "2017-03-08T14:38:36Z")

</div>

I've set ES\_HEAP\_SIZE="30000m"

And the machine has 8 cores, 64gb RAM and 3TB storage

All other settings are the default.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 8, 2017, 3:07pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/8 "2017-03-08T15:07:22Z")

</div>

Do you have any custom config settings in your elasticsearch.yml file?

---

<div class="post-metadata">

**Author:** ![rzrucher](https://avatars.discourse-cdn.com/v4/letter/r/76d3ee/32.png) [@rzrucher](https://discuss.elastic.co/u/rzrucher)\
**Post date:** [March 8, 2017, 3:09pm UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/9 "2017-03-08T15:09:21Z")

</div>

No, only:  
bootstrap.mlockall: true and cluster and node name.

But, in my indexing process I do the following :

- Start to read a logfile
- Create index for that logfile
- Set refresh to -1
- Index all event for the logfile
- Set refresh to 30s
- Do a forceMerge with 1 segment on this index
- Process next logfile

---

<div class="post-metadata">

**Author:** ![rzrucher](https://avatars.discourse-cdn.com/v4/letter/r/76d3ee/32.png) [@rzrucher](https://discuss.elastic.co/u/rzrucher)\
**Post date:** [March 9, 2017, 9:26am UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/10 "2017-03-09T09:26:34Z")

</div>

The problem was due to a dashboard display in kibana that needs aggregations over a large amount of data.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2017, 9:26am UTC](https://discuss.elastic.co/t/es-2-3-3-cannot-understand-why-oom/77818/11 "2017-04-06T09:26:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.