Java heap space once in a while

Elastic Stack Elasticsearch
1

I have been setting up a ELK server, 16gb ram and a 8core cpu. I have done a basic installation, no extra config on elasticsearch side. Every once in a while elasticsearch seems to break but a fter a restart of the service it function again properly, the messages logs that start tot trigger at the time of break is bellow.
What to do to correct this issue?

org.elasticsearch.index.translog.TranslogException: [logstash-2015.06.12][3] Failed to write operation [org.elasticsearch.index.translog.Translog$Create@781db269]
	at org.elasticsearch.index.translog.fs.FsTranslog.add(FsTranslog.java:393)
	at org.elasticsearch.index.engine.internal.InternalEngine.innerCreateNoLock(InternalEngine.java:505)
	at org.elasticsearch.index.engine.internal.InternalEngine.innerCreate(InternalEngine.java:444)
	at org.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:413)
	at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:428)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:437)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:149)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:512)
	at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:56)
	at org.elasticsearch.common.recycler.AbstractRecycler.obtain(AbstractRecycler.java:32)
	at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:29)
	at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39)
	at org.elasticsearch.common.recycler.Recyclers$5.obtain(Recyclers.java:161)
	at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39)
	at org.elasticsearch.cache.recycler.PageCacheRecycler.bytePage(PageCacheRecycler.java:177)
	at org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:453)
	at org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:466)
	at org.elasticsearch.common.io.stream.BytesStreamOutput.<init>(BytesStreamOutput.java:60)
	at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.<init>(ReleasableBytesStreamOutput.java:37)
	at org.elasticsearch.index.translog.fs.FsTranslog.add(FsTranslog.java:371)
	... 11 more
2

Add more memory, add more nodes, or reduce the amount of data in your cluster.
There's not too many ways around this one.

3

To ask the simple question, Even though your system has 16GB of ram this does not matter to a JVM. Elasticsearch by default runs the JVM at 1GB.

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
export ES_HEAP_SIZE=10g

Also, you may want to look at the type of searches your doing and the Number of them. If you are hitting OOM like Warkolm mentioned you might have to add more nodes.

Though you could look at improving your searches to and indexes so that they are not retrieving too much data, and having to search to many indexes.

Just some more idea's

1 Like
4

Thanks, i have added es_heap_size to 8gb for now and i will see what happens in the near future.

5

It has been a while since the original issue was reported.

In my case, I used to have memory issue on REDIS. As it has been controlled by setting up the maxmemory, I start to get OOM failure on ELAS side.

If a user does not want to increase nodes, can he have a "LRU" way to control it ?

Thanks,

Chun