Out of Memory Error on ElasticSearch running master only with 8 GB with version 5.2.2

Elastic Stack Elasticsearch
1 
[2018-04-10T07:38:28,846][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [_ras_master_9200] fatal error in thread [Thread-22], exiting
java.lang.OutOfMemoryError: Java heap space
        at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:99) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:96) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:53) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.recycler.AbstractRecycler.obtain(AbstractRecycler.java:33) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:28) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.recycler.Recyclers$3.obtain(Recyclers.java:119) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.PageCacheRecycler.bytePage(PageCacheRecycler.java:147) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.AbstractBigArray.newBytePage(AbstractBigArray.java:112) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.BigByteArray.resize(BigByteArray.java:141) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.BigArrays.resizeInPlace(BigArrays.java:438) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.BigArrays.resize(BigArrays.java:485) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.BigArrays.grow(BigArrays.java:502) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.io.stream.BytesStreamOutput.ensureCapacity(BytesStreamOutput.java:158) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:90) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.io.stream.StreamOutput.writeBytes(StreamOutput.java:119) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:346) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:495) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.ElasticsearchException.writeTo(ElasticsearchException.java:221) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.EsRejectedExecutionException.writeTo(EsRejectedExecutionException.java:65) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:838) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.ElasticsearchException.writeTo(ElasticsearchException.java:220) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.ActionTransportException.writeTo(ActionTransportException.java:64) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:838) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.ElasticsearchException.writeTo(ElasticsearchException.java:220) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.ActionTransportException.writeTo(ActionTransportException.java:64) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:838) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.BulkItemResponse$Failure.writeTo(BulkItemResponse.java:118) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.BulkItemResponse.writeTo(BulkItemResponse.java:341) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.BulkResponse.writeTo(BulkResponse.java:139) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.TcpTransport.buildMessage(TcpTransport.java:1179) ~[elasticsearch-5.2.2.jar:5.2.2]
2

I guess you have a question?

Sounds like here your bulk request can not be loaded in memory.
May be a too big bulk request but hard to know without any idea of what you have, what you are doing...

3

Thanks for the reply, yes we indexing data on elastic search with 3 master nodes and 4 data nodes.
With 8 GB of master node and 32 GB of data node.
It would be very help if you can tell me way to find out how many documents are getting processed on master node during at the time of indexing.

4

It would be very help if you can tell me way to find out how many documents are getting processed on master node during at the time of indexing.

None. Unless you are sending your bulk requests to the master node.

5

Oh and I forgot that you should update to latest 5.6 version.

6

Is there any specific performance improvements or bug fix related to the issue have posted?

We are sending all the request via master to data nodes. Can we directly send it to data nodes? If yes how could data node decide request is for that specific node.

7

Given the stack trace looks to be an OOM triggered by failing to add stack trace details to a bulk response my guess is that you have many many errors in your indexing requests that cause a lot of stack traces.

Can you try your indexing requests individually to get sight of what the errors are in your bulk request

8

Yes.

If yes how could data node decide request is for that specific node.

data nodes know about the cluster state.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.