Error "array index out of bounds java.lang.OutOfMemoryError: Java heap space"


(Prashant Agrawal) #1

Hi ES users,

I am getting the following exception while indexing huge amount of data(say ~5GB) to ES node.

Exception:

  1. array index out of bounds
    java.lang.OutOfMemoryError: Java heap space

    Dumping heap to java_pid6721.hprof ...

java.lang.OutOfMemoryError: PermGen space
Dumping heap to java_pid2686.hprof .
..
Heap dump file created [84028931 bytes in 1.444 secs]

These exception comes after indexing about ~3GB of data. And the exception are inconsistent that is any of two exception comes while indexing the same set of data.

Some more info regarding setup:
ES version : 1.0.1
ES_MIN_MEM: 256m
ES_MAX_MEM: 1g
JAVA_OPTS: -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError

bootstrap.mlockall true
ulimit -l unlimited
index.number_of_shards: 5
index.number_of_replicas: 0

Let me know if anything else is required as well.


(gkwelding) #2

The message is pretty obvious. Your node is running out of heap memory...
Increase it.

On Tuesday, March 4, 2014 1:36:51 PM UTC, Prashy wrote:

Hi ES users,

I am getting the following exception while indexing huge amount of
data(say
~5GB) to ES node.

Exception:

  1. /array index out of bounds
    java.lang.OutOfMemoryError: Java heap space
    /
    Dumping heap to java_pid6721.hprof ...

/java.lang.OutOfMemoryError: PermGen space
Dumping heap to java_pid2686.hprof .
/..
Heap dump file created [84028931 bytes in 1.444 secs]

These exception comes after indexing about ~3GB of data. And the exception
are inconsistent that is any of two exception comes while indexing the
same
set of data.

*Some more info regarding setup:
ES version : 1.0.1
ES_MIN_MEM: 256m
ES_MAX_MEM: 1g
JAVA_OPTS: -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError

bootstrap.mlockall true
ulimit -l unlimited
index.number_of_shards: 5
index.number_of_replicas: 0*

Let me know if anything else is required as well.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Error-array-index-out-of-bounds-java-lang-OutOfMemoryError-Java-heap-space-tp4050914.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/43aaeeae-820d-4b70-88d6-28442a1ebb64%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Prashant Agrawal) #3

I tried increasing the heap value by 2GB as well by ES_MAX_MEM: 2g but it gave the same error.


(gkwelding) #4

Adding a bit more to my rather short answer. Both exceptions essentially
mean the same thing. I would follow the basic heap allocation advice.
Allocate 50% of your system RAM to ES as catastrophic things happen when ES
runs out of RAM. Leave the other 50% to the system. So if you have a server
with 4gb RAM you need to assign ES 2gb leaving the system with the other
2gb (ass gin both min and max heap size to 2gb).

If you are still having problems after increasing your heap size, or if 1gb
is 50% of your server's RAM then you probably need to get a bigger box...

On Tuesday, March 4, 2014 1:36:51 PM UTC, Prashy wrote:

Hi ES users,

I am getting the following exception while indexing huge amount of
data(say
~5GB) to ES node.

Exception:

  1. /array index out of bounds
    java.lang.OutOfMemoryError: Java heap space
    /
    Dumping heap to java_pid6721.hprof ...

/java.lang.OutOfMemoryError: PermGen space
Dumping heap to java_pid2686.hprof .
/..
Heap dump file created [84028931 bytes in 1.444 secs]

These exception comes after indexing about ~3GB of data. And the exception
are inconsistent that is any of two exception comes while indexing the
same
set of data.

*Some more info regarding setup:
ES version : 1.0.1
ES_MIN_MEM: 256m
ES_MAX_MEM: 1g
JAVA_OPTS: -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError

bootstrap.mlockall true
ulimit -l unlimited
index.number_of_shards: 5
index.number_of_replicas: 0*

Let me know if anything else is required as well.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Error-array-index-out-of-bounds-java-lang-OutOfMemoryError-Java-heap-space-tp4050914.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cf5b04dd-a9b2-428c-937c-36570a3c4fc6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(gkwelding) #5

you also don't give us much information on how you're trying to index this
3gb of information. Are you using the bulk API? Are you refreshing after
every index action? etc...

On Tuesday, March 4, 2014 1:40:58 PM UTC, Prashy wrote:

I tried increasing the heap value by 2GB as well by ES_MAX_MEM: 2g but it
gave the same error.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Error-array-index-out-of-bounds-java-lang-OutOfMemoryError-Java-heap-space-tp4050914p4050916.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e83e7d88-9af5-418d-870f-b029ff8d3175%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Prashant Agrawal) #6

Just wanted to know that as I was using 1GB as Heap Size I was getting an error. So I increased it to 2GB (Heap , system has 4GB) so in that scenario also I got the error at same point.

So if I increased the memory from 1 GB to 2 GB, at least it should process one more record in compare to previous scenario.


(Prashant Agrawal) #7

Hi gkwelding,

I am using the bulk API for indexing the data. And also refresh parameter is not set.
So what could be the issue for that exception.

Let me know if you require any other input for the same.


(Clinton Gormley) #8

On 5 March 2014 06:35, prashy prashant.agrawal@paladion.net wrote:

I am using the bulk API for indexing the data.

How much data are you indexing in a single request? If you send all the
docs at once, you will run out of memory. The idea is to have bulk
requests of eg 1-5,000 docs at a time (but it depends how big your docs are)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKS3W%3DdhHrvsZUdjO6Lf6-MN_9dpDgim4bk8HyiNNxLGXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(gkwelding) #9

What he said! Which is why I asked about how you were indexing! Sorry I
didn't get chance to reply sooner but what Clinton said is true. Split down
your bulk indexing requests into chunks.

On Wednesday, March 5, 2014 11:53:36 AM UTC, Clinton Gormley wrote:

On 5 March 2014 06:35, prashy <prashant...@paladion.net <javascript:>>wrote:

I am using the bulk API for indexing the data.

How much data are you indexing in a single request? If you send all the
docs at once, you will run out of memory. The idea is to have bulk
requests of eg 1-5,000 docs at a time (but it depends how big your docs are)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/442521b9-5896-4547-ac8a-41b0727b7a98%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Prashant Agrawal) #10

Hi Clinton/gkwelding,

I tried sending the document one by one as well for indexing.

So suppose my document is of 5 MB so it gets indexed and not giving any exception. But in other hand for some documents which is in KB or about 1 MB is giving the exception of out of memory.

Though mapping are same for both the scenario. And the memory is 2 GB.

So what could be the issue for that?


(Prashant Agrawal) #11

Hi Clinton/gkwelding,

Any suggestion to overcome this issue as I tried by increasing the heap size and still getting the same error.

Below are the call logs for the issue:

error: array index out of bounds
[2014-03-05 21:03:25,212][WARN ][monitor.jvm
] [Node225] [gc][young][5079][15545] duration [10.1s], col
lections [2]/[18.7s], total [10.1s]/[2m], memory [1.9gb]->
[1.7gb]/[1.9gb], all_pools {[young] [20.7mb]->[7.7mb]/[33.
3mb]}{[survivor] [4.1mb]->[4.1mb]/[4.1mb]}{[old] [1.9gb]->
[1.7gb]/[1.9gb]}
error: array index out of bounds
[2014-03-05 21:07:58,490][WARN ][http.netty ] [Node225] Caught exception while handling client http traffic, closing connection [id: 0x1da386d5, /192.168.0.251:36240 => /192.168.0.225:9200]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3065)
at org.elasticsearch.common.bytes.BytesArray.toBytes(BytesArray.java:114)
at org.elasticsearch.common.io.Streams.copyToByteArray(Streams.java:165)
at org.elasticsearch.common.io.Streams.copyToByteArray(Streams.java:84)
at org.elasticsearch.http.HttpServer.handlePluginSite(HttpServer.java:197)
at org.elasticsearch.http.HttpServer$PluginSiteFilter.process(HttpServer.java:129)
at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:216)
at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:118)
at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:291)
at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:43)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)


(gkwelding) #12

My next guess, and it really is a guess now, is that you might be running
out of file descriptors. As
per http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-service.html
make sure MAX_OPEN_FILES and MAX_MAP_COUNT aren't too low. This may also
help, http://www.elasticsearch.org/tutorials/too-many-open-files/.

On Friday, March 7, 2014 10:51:15 AM UTC, Prashy wrote:

Hi Clinton/gkwelding,

Any suggestion to overcome this issue as I tried by increasing the heap
size
and still getting the same error.

Below are the call logs for the issue:

error: array index out of bounds
[2014-03-05 21:03:25,212][WARN ][monitor.jvm
] [Node225] [gc][young][5079][15545] duration [10.1s], col
lections [2]/[18.7s], total [10.1s]/[2m], memory [1.9gb]->
[1.7gb]/[1.9gb], all_pools {[young] [20.7mb]->[7.7mb]/[33.
3mb]}{[survivor] [4.1mb]->[4.1mb]/[4.1mb]}{[old] [1.9gb]->
[1.7gb]/[1.9gb]}
error: array index out of bounds
[2014-03-05 21:07:58,490][WARN ][http.netty ] [Node225]
Caught
exception while handling client http traffic, closing connection [id:
0x1da386d5, /192.168.0.251:36240 => /192.168.0.225:9200]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3065)
at
org.elasticsearch.common.bytes.BytesArray.toBytes(BytesArray.java:114)
at
org.elasticsearch.common.io.Streams.copyToByteArray(Streams.java:165)
at
org.elasticsearch.common.io.Streams.copyToByteArray(Streams.java:84)
at
org.elasticsearch.http.HttpServer.handlePluginSite(HttpServer.java:197)
at
org.elasticsearch.http.HttpServer$PluginSiteFilter.process(HttpServer.java:129)

    at 

org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:216)

    at 

org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:118)

    at 

org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)

    at 

org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:291)

    at 

org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:43)

    at 

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

    at 

org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)

    at 

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

    at 

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)

    at 

org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)

    at 

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)

    at 

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)

    at 

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

    at 

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

    at 

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

    at 

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)

    at 

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)

    at 

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

    at 

org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Error-array-index-out-of-bounds-java-lang-OutOfMemoryError-Java-heap-space-tp4050914p4051291.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ad5ff704-e942-4564-b789-079a0e118f1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Prashant Agrawal) #13

MAX_OPEN_FILES and MAX_MAP_COUNT are not being set explicitly, so having default value only i.e. 65535.

And still its crashing with same error :frowning:


(gkwelding) #14

Have you been able to explicitly check what the max file descriptors limit
is set to on your box?

Other than that I'll have to step out of this conversation and hand over to
somebody who actually knows what they're talking about...

On Friday, March 7, 2014 12:48:25 PM UTC, Prashy wrote:

MAX_OPEN_FILES and MAX_MAP_COUNT are not being set explicitly, so having
default value only i.e. 65535.

And still its crashing with same error :frowning:

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Error-array-index-out-of-bounds-java-lang-OutOfMemoryError-Java-heap-space-tp4050914p4051302.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/58898bad-e846-40ef-b4be-4e15abf25f66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Prashant Agrawal) #15

Hi gkwelding,

I have checked explicitly on my box and the value for MAX_OPEN_FILES and MAX_MAP_COUNT has been set to 65535.


(Clinton Gormley) #16

Also, are you using Ubuntu 10.04? I see you have slow young generation GC,
and that version of Ubuntu had a bug in that area.

On 10 March 2014 10:24, prashy prashant.agrawal@paladion.net wrote:

Hi gkwelding,

I have checked explicitly on my box and the value for MAX_OPEN_FILES and
MAX_MAP_COUNT has been set to 65535.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Error-array-index-out-of-bounds-java-lang-OutOfMemoryError-Java-heap-space-tp4050914p4051415.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1394443468998-4051415.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKTs3wz84fG4sV1f170J%3DUZAiJULdhUc8RXGVPGBKfqQoQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Prashant Agrawal) #17

I am using Red Hat.

[root@localhost /]# lsb_release
LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
[root@localhost /]# lsb_release -a
LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Release: 5.5
Codename: Tikanga


(system) #18