Lots of ArrayIndexOutOfBoundsException in my logs

Elastic Stack Elasticsearch
1

I see my logs growing very fast with the following message:

[2016-08-25 06:21:57,714][DEBUG][action.search.type       ] [**node_name**] [**indice_name_would_be_here**][2], node[oHyyob8ySNOtfwjKrFkgUQ], [R], v[9], s[STARTED], a[id=kvU5aUu5RPmGAkwvS8VgrQ]: Failed to execute [org.elasticsearch.action.search.SearchRequest@280a36cb] lastShard [true]
RemoteTransportException[[**node_name**][**IP**:9300][indices:data/read/search[phase/query]]]; nested: ArrayIndexOutOfBoundsException;
Caused by: java.lang.ArrayIndexOutOfBoundsException

This messages repeats for several indices under the same alias and many nodes.

Sometimes I also see the following stacktrace:

Failed to execute phase [query], all shards failed; shardFailures {[Z07URHO4QX65Hb8KAbMirg][**IP**:9300][indices:data/read/search[phase/query]]]; nested: ArrayIndexOutOfBoundsException; }
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:228)
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:174)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
	at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:211)
	at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:201)
	at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
...
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: ; nested: ArrayIndexOutOfBoundsException;
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:386)
	at org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:152)
	at org.elasticsearch.action.search.SearchPhaseExecutionException.getCause(SearchPhaseExecutionException.java:99)
	at java.lang.Throwable.printStackTrace(Unknown Source)
	at java.lang.Throwable.printStackTrace(Unknown Source)
	at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
...
	at org.apache.log4j.Category.log(Category.java:856)
	at org.elasticsearch.common.logging.log4j.Log4jESLogger.internalInfo(Log4jESLogger.java:125)
	at org.elasticsearch.common.logging.support.AbstractESLogger.info(AbstractESLogger.java:90)
	at org.elasticsearch.rest.BytesRestResponse.convert(BytesRestResponse.java:131)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:87)
	at org.elasticsearch.rest.action.support.RestActionListener.onFailure(RestActionListener.java:60)
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.raiseEarlyFailure(TransportSearchTypeAction.java:316)
	... 28 more
Caused by: java.lang.ArrayIndexOutOfBoundsException

I have no visibility to what type of query/aggregations users are running.

That could be causing that?
Using ES 2.2.0

Thanks.

1 Like
2

A similar thing was reported here: https://github.com/elastic/elasticsearch/issues/16391
which has been fixed in ES v2.3.1 https://github.com/elastic/elasticsearch/pull/17379