ReduceSearchPhaseException[Failed to execute phase [query], [reduce] ]

snowtigersoft · August 5, 2013, 7:09am

I have some indices, all of them have a same mapping struct.
Today I found one of the indices goes wrong, it throw exception when sort
on some fields SOMETIMES, for example:

curl -XGET
"http://192.168.2.105:9200/weibo_status_201331/status/_search?pretty=true&from=0&size=1"
-d '{"sort": [{"created_at": "desc"}], "query": {"match_all":{}}}'

Sometimes it return a right result, but most of the time, it throw a
exception:
{
"error" : "ReduceSearchPhaseException[Failed to execute phase [query],
[reduce] ]; nested: ClassCastException[java.lang.Long cannot be cast to
java.lang.Integer]; ",
"status" : 500
}

When it throw exception, the detail error message from the log is:
[2013-08-05 15:08:00,793][DEBUG][action.search.type ] [pgy-data2]
[weibo_status_201331][0]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@60fa8a2f] while moving to
second phase
java.lang.ClassCastException: java.lang.Long cannot be cast to
java.lang.Integer
at java.lang.Integer.compareTo(Integer.java:52)
at
org.apache.lucene.search.FieldComparator.compareValues(FieldComparator.java:182)
at
org.apache.lucene.search.ShardFieldDocSortedHitQueue.lessThan(ShardFieldDocSortedHitQueue.java:137)
at
org.apache.lucene.search.ShardFieldDocSortedHitQueue.lessThan(ShardFieldDocSortedHitQueue.java:34)
at
org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:150)
at
org.elasticsearch.search.controller.SearchPhaseController.sortDocs(SearchPhaseController.java:200)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryThenFetchAction.java:90)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:229)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:208)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:205)
at
org.elasticsearch.search.action.SearchServiceTransportAction$2.handleResponse(SearchServiceTransportAction.java:156)
at
org.elasticsearch.search.action.SearchServiceTransportAction$2.handleResponse(SearchServiceTransportAction.java:147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:156)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:127)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

I'm confused how this happen? Why sometimes it throw exception but sometime
not?
I had try to fetch some of the data(about 10000 records) from the server
and rebuild the index in my local machine, everything works well.

BTW, the mapping is:
{
"status" : {
"properties" : {
"cid" : {
"type" : "integer"
},
"created_at" : {
"type" : "integer"
},
"emotions" : {
"properties" : {
"id" : {
"type" : "short"
}
}
},
"origin_text" : {
"type" : "string",
"index" : "no",
"store" : "yes"
},
"original_pic" : {
"type" : "string",
"index" : "no",
"store" : "yes"
},
"place" : {
"properties" : {
"coordinates" : {
"type" : "geo_point",
"lat_lon" : true
},
"poiid" : {
"type" : "string",
"index" : "not_analyzed",
"omit_norms" : true,
"index_options" : "docs"
},
"title" : {
"type" : "string",
"index" : "not_analyzed",
"omit_norms" : true,
"index_options" : "docs"
}
}
},
"ref_count" : {
"type" : "integer"
},
"ref_level" : {
"type" : "short"
},
"rt_created" : {
"type" : "integer"
}
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

brian_yoder · August 5, 2013, 4:35pm

Perhaps either your cid or created_at field was given a value that
exceeds the size of a 32-bit signed integer?

On Monday, August 5, 2013 3:09:06 AM UTC-4, 赵春友 wrote:

I have some indices, all of them have a same mapping struct.
Today I found one of the indices goes wrong, it throw exception when sort
on some fields SOMETIMES, for example:

curl -XGET "
http://192.168.2.105:9200/weibo_status_201331/status/_search?pretty=true&from=0&size=1
" -d '{"sort": [{"created_at": "desc"}], "query": {"match_all":{}}}'

Sometimes it return a right result, but most of the time, it throw a
exception:
{
"error" : "ReduceSearchPhaseException[Failed to execute phase [query],
[reduce] ]; nested: ClassCastException[java.lang.Long cannot be cast to
java.lang.Integer]; ",
"status" : 500
}

When it throw exception, the detail error message from the log is:
[2013-08-05 15:08:00,793][DEBUG][action.search.type ] [pgy-data2]
[weibo_status_201331][0]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@60fa8a2f] while moving to
second phase

I'm confused how this happen? Why sometimes it throw exception but
sometime not?
I had try to fetch some of the data(about 10000 records) from the server
and rebuild the index in my local machine, everything works well.

BTW, the mapping is:
{
"status" : {
"properties" : {
"cid" : {
"type" : "integer"
},
"created_at" : {
"type" : "integer"
},

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

snowtigersoft · August 26, 2013, 2:58am

Thanks InquiringMind, But it's not that reason. My current solution is dump
all data from the index, then rebuild it, but I don't think it is a perfect
solution.

I will provide more info about my ES server:

There are 3 ES servers in the cluster, the es version is 0.20.6.
The server running more than 20 processes to insert records into ES
server at the same time, when it found the index not exist, it will create
one, so maybe more than one processes will run the create index command at
the same time, I'm not sure whether this will cause the problem.
When I kill all running processes and rebuild the index, it works well
again.

Anyone can give me some suggestion?

On Tuesday, August 6, 2013 12:35:05 AM UTC+8, InquiringMind wrote:

Perhaps either your cid or created_at field was given a value that
exceeds the size of a 32-bit signed integer?

On Monday, August 5, 2013 3:09:06 AM UTC-4, 赵春友 wrote:

I have some indices, all of them have a same mapping struct.
Today I found one of the indices goes wrong, it throw exception when sort
on some fields SOMETIMES, for example:

curl -XGET "
http://192.168.2.105:9200/weibo_status_201331/status/_search?pretty=true&from=0&size=1
" -d '{"sort": [{"created_at": "desc"}], "query": {"match_all":{}}}'

Sometimes it return a right result, but most of the time, it throw a
exception:
{
"error" : "ReduceSearchPhaseException[Failed to execute phase [query],
[reduce] ]; nested: ClassCastException[java.lang.Long cannot be cast to
java.lang.Integer]; ",
"status" : 500
}

When it throw exception, the detail error message from the log is:
[2013-08-05 15:08:00,793][DEBUG][action.search.type ] [pgy-data2]
[weibo_status_201331][0]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@60fa8a2f] while moving to
second phase

I'm confused how this happen? Why sometimes it throw exception but
sometime not?
I had try to fetch some of the data(about 10000 records) from the server
and rebuild the index in my local machine, everything works well.

BTW, the mapping is:
{
"status" : {
"properties" : {
"cid" : {
"type" : "integer"
},
"created_at" : {
"type" : "integer"
},

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.