How can I identify the document which causes java.lang.ClassCastException

[MartinG]

Hello,

we have a problem with a "java.lang.ClassCastException". It seems that somebody has written data into several ES indexes which now causes queries to fail.

The error message in the elasticsearch.log is:
[2016-12-12 10:37:51,519][WARN ][transport ] [mcc5-kibana0-prod-mcc5-prod] Received response for a request that has timed out, sent [43176ms] ago, timed out [13176ms] ago, action [internal:discovery/zen/fd/master_ping], node [[mcc5-elastic31-prod-mcc5-prod][A49ZLeNuRmelWKc0_ntGaw][mcc5-elastic31-prod][inet[/10.31.232.83:9300]]{master=true}], id [1]
[2016-12-12 10:38:42,219][DEBUG][action.search.type ] [mcc5-kibana0-prod-mcc5-prod] [json_713][1]: Failed to execute [org.elasticsearch.action.search.SearchRequest@281f60b8] while moving to second phase
java.lang.ClassCastException: java.lang.Long cannot be cast to org.apache.lucene.util.BytesRef
at org.apache.lucene.search.FieldComparator$TermOrdValComparator.compareValues(FieldComparator.java:902)
at org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:172)
at org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:120)
at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:225)
at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:133)
at org.apache.lucene.search.TopDocs.merge(TopDocs.java:234)
at org.elasticsearch.search.controller.SearchPhaseController.sortDocs(SearchPhaseController.java:239)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryThenFetchAction.java:89)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.innerMoveToSecondPhase(TransportSearchTypeAction.java:403)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:202)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:178)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:175)
at org.elasticsearch.search.action.SearchServiceTransportAction$6.handleResponse(SearchServiceTransportAction.java:244)
at org.elasticsearch.search.action.SearchServiceTransportAction$6.handleResponse(SearchServiceTransportAction.java:235)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:163)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:132)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

I searched for a solution and found out, that there's maybe a field value bigger than allowed:
ReduceSearchPhaseException[Failed to execute phase [query], [reduce] ].

The problem occured on ES version 1.7.5.

My question now is: how can I identify the document and the field which causes that exception?

It would be great if somebody can help!

Greetings,

Martin

[dadoonet (David Pilato)]

Please format your code using </> icon as explained in this guide. It will make your post more readable.

It seems that the mapping has not been processed on every shard in the same manner. IIRC it can happen on old version when 2 requests goes to 2 shards in parallel and updates the mapping at the same time.

This has been fixed in recent version.

I don't know how you can identify this though.

A way to solve the issue in the future, may be send the mapping in advance when you create your index so you are sure it's propagated to all nodes/shards.
Or upgrade