SearchPhaseController - NullPointerException

Sebastian_Gavarini · October 13, 2011, 7:28pm

Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the index
while I did the search. Anyway, the line in question is:

                    for (Facet facet1 :

queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

Sebastian_Gavarini · October 14, 2011, 12:30am

Update: I still can't replicate with a "curl" command, but under controlled
conditions I do see it all the time with the java API.

The query runs without problems as REST, the only thing I don't know how to
specify in the REST API compared to java, is the Operation Threading, witch
is THREAD_PER_SHARD, I don't know if this could be part of the problem.

What I see related to the facets iteration that SearchPhaseController is
doing in line 292 is that I am querying 5 indices, and in one of them the
value of the sort field is not mapped, so that index fails with:

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [ {
"index" : "index-main-e1",
"shard" : 0,
"status" : 400,
"reason" :
"RemoteTransportException[[Tugun][inet[/127.0.0.1:5000]][search/phase/query+fetch]];
nested: SearchParseException[[index-main-e1][0]: ;
nested: SearchParseException[[index-main-e1][0]: query[(title:something^5.0
| description:something| attrs:something^3.0)~0.1],from[0],size[10]: Parse
Failure [No mapping found for [normalized_price]]]; "
} ]
},
"hits" : {

but all the other 4 are ok and the results are returned when in REST API and
not in Java API. Could there be some problem when the facets are checked for
merging and that index didn't run? any clue?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 4:28 PM, Sebastian Gavarini sgavarini@gmail.comwrote:

Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the index
while I did the search. Anyway, the line in question is:
                    for (Facet facet1 :
queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

Sebastian_Gavarini · October 14, 2011, 12:36am

Sorry for the spam, forgot to add that I can't see the exact Java query in
my logs, because I am using:

String output = new
String(builder.internalBuilder().buildAsBytes(XContentType.JSON));

And all the extra parameters like indices, types, threading, search type,
timeout, (and I don't know if I am forgetting any other one) are not shown
by that line, any ideas on how to get a REST reproducible log from the Java
query?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 9:30 PM, Sebastian Gavarini sgavarini@gmail.comwrote:

Update: I still can't replicate with a "curl" command, but under controlled
conditions I do see it all the time with the java API.

The query runs without problems as REST, the only thing I don't know how to
specify in the REST API compared to java, is the Operation Threading, witch
is THREAD_PER_SHARD, I don't know if this could be part of the problem.

What I see related to the facets iteration that SearchPhaseController is
doing in line 292 is that I am querying 5 indices, and in one of them the
value of the sort field is not mapped, so that index fails with:

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [ {
"index" : "index-main-e1",
"shard" : 0,
"status" : 400,
"reason" : "RemoteTransportException[[Tugun][inet[/127.0.0.1:5000]][search/phase/query+fetch]];
nested: SearchParseException[[index-main-e1][0]: ;
nested: SearchParseException[[index-main-e1][0]: query[(title:something^5.0
| description:something| attrs:something^3.0)~0.1],from[0],size[10]: Parse
Failure [No mapping found for [normalized_price]]]; "
} ]
},
"hits" : {

but all the other 4 are ok and the results are returned when in REST API
and not in Java API. Could there be some problem when the facets are checked
for merging and that index didn't run? any clue?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 4:28 PM, Sebastian Gavarini sgavarini@gmail.comwrote:
Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the index
while I did the search. Anyway, the line in question is:
                    for (Facet facet1 :
queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

kimchy · October 14, 2011, 12:33pm

Which version are you using? If you can create it using hte Java API, can
you create a simple main program that starts a node, index sample data, and
executes a search that recreates it? Thats the simplest / fastest way to fix
it.

On Fri, Oct 14, 2011 at 2:36 AM, Sebastian Gavarini sgavarini@gmail.comwrote:

Sorry for the spam, forgot to add that I can't see the exact Java query in
my logs, because I am using:

String output = new
String(builder.internalBuilder().buildAsBytes(XContentType.JSON));

And all the extra parameters like indices, types, threading, search type,
timeout, (and I don't know if I am forgetting any other one) are not shown
by that line, any ideas on how to get a REST reproducible log from the Java
query?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 9:30 PM, Sebastian Gavarini sgavarini@gmail.comwrote:
Update: I still can't replicate with a "curl" command, but under
controlled conditions I do see it all the time with the java API.

The query runs without problems as REST, the only thing I don't know how
to specify in the REST API compared to java, is the Operation Threading,
witch is THREAD_PER_SHARD, I don't know if this could be part of the
problem.

What I see related to the facets iteration that SearchPhaseController is
doing in line 292 is that I am querying 5 indices, and in one of them the
value of the sort field is not mapped, so that index fails with:

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [ {
"index" : "index-main-e1",
"shard" : 0,
"status" : 400,
"reason" : "RemoteTransportException[[Tugun][inet[/127.0.0.1:5000]][search/phase/query+fetch]];
nested: SearchParseException[[index-main-e1][0]: ;
nested: SearchParseException[[index-main-e1][0]: query[(title:something^5.0
| description:something| attrs:something^3.0)~0.1],from[0],size[10]: Parse
Failure [No mapping found for [normalized_price]]]; "
} ]
},
"hits" : {

but all the other 4 are ok and the results are returned when in REST API
and not in Java API. Could there be some problem when the facets are checked
for merging and that index didn't run? any clue?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 4:28 PM, Sebastian Gavarini sgavarini@gmail.comwrote:
Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the index
while I did the search. Anyway, the line in question is:
                    for (Facet facet1 :
queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

Sebastian_Gavarini · October 17, 2011, 6:48pm

Hi Shay,

I took a couple of days to try to replicate the problem in an isolated
manner, previously it was possible to reproduce in our full java
application.

The problem appears sporadically in my 2 cores PC (once in a couple of runs
so pretty frequently anyway), so I would advice to run it a couple of times
to make it work. It seems to be related to a couple of simultaneous
conditions, two searches done in parallel, one search that is ok, and
another one that of the two indices, only one contains the sort field
requested.

In my example I create 3 indices, and run 2 searches, look at method
"launch".

gist.github.com

https://gist.github.com/sebaes/1293410

Java exception example



import java.util.HashMap;
import java.util.Map;

import org.elasticsearch.action.ActionFuture;
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.admin.indices.create.CreateIndexRequest;
import org.elasticsearch.action.admin.indices.create.CreateIndexResponse;
import org.elasticsearch.action.admin.indices.flush.FlushRequest;

This file has been truncated. show original

log

2011-10-17 15:35:07,703 INFO - Log4jESLogger.internalInfo[lin. 79]- [Lucas Brand] {elasticsearch/0.17.8}[4949]: initializing ...
2011-10-17 15:35:07,735 INFO - Log4jESLogger.internalInfo[lin. 79]- [Lucas Brand] loaded [], sites []
2011-10-17 15:35:09,187 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] using bytebuffer cache with small_buffer_size [1kb], large_buffer_size [1mb], small_cache_size [10mb], large_cache_size [500mb], direct [true]
2011-10-17 15:35:09,231 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] creating thread_pool [cached], type [cached], keep_alive [30s]
2011-10-17 15:35:09,233 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] creating thread_pool [index], type [cached], keep_alive [5m]
2011-10-17 15:35:09,234 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] creating thread_pool [search], type [cached], keep_alive [5m]
2011-10-17 15:35:09,234 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] creating thread_pool [percolate], type [cached], keep_alive [5m]
2011-10-17 15:35:09,235 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] creating thread_pool [management], type [scaling], min [1], size [20], keep_alive [5m]
2011-10-17 15:35:09,239 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] creating thread_pool [merge], type [scaling], min [1], size [20], keep_alive [5m]
2011-10-17 15:35:09,239 DEBUG - Log4jESLogger.internalDebug[lin. 71]- [Lucas Brand] creating thread_pool [snapshot], type [scaling], min [1], size [10], keep_alive [5m]

This file has been truncated. show original

IMPORTANT: The idea was to create in memory indices, but it isn't working, I
don't know what am I messing in the settings part, but the solution I found
was to delete everything in "/tmp/es/server" after each run.

Let me know if it doesn't work for you.

Thanks,
Sebastian.

On Fri, Oct 14, 2011 at 9:33 AM, Shay Banon kimchy@gmail.com wrote:

Which version are you using? If you can create it using hte Java API, can
you create a simple main program that starts a node, index sample data, and
executes a search that recreates it? Thats the simplest / fastest way to fix
it.

On Fri, Oct 14, 2011 at 2:36 AM, Sebastian Gavarini sgavarini@gmail.comwrote:
Sorry for the spam, forgot to add that I can't see the exact Java query in
my logs, because I am using:

String output = new
String(builder.internalBuilder().buildAsBytes(XContentType.JSON));

And all the extra parameters like indices, types, threading, search type,
timeout, (and I don't know if I am forgetting any other one) are not shown
by that line, any ideas on how to get a REST reproducible log from the Java
query?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 9:30 PM, Sebastian Gavarini sgavarini@gmail.comwrote:
Update: I still can't replicate with a "curl" command, but under
controlled conditions I do see it all the time with the java API.

The query runs without problems as REST, the only thing I don't know how
to specify in the REST API compared to java, is the Operation Threading,
witch is THREAD_PER_SHARD, I don't know if this could be part of the
problem.

What I see related to the facets iteration that SearchPhaseController is
doing in line 292 is that I am querying 5 indices, and in one of them the
value of the sort field is not mapped, so that index fails with:

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [ {
"index" : "index-main-e1",
"shard" : 0,
"status" : 400,
"reason" : "RemoteTransportException[[Tugun][inet[/127.0.0.1:5000]][search/phase/query+fetch]];
nested: SearchParseException[[index-main-e1][0]: ;
nested: SearchParseException[[index-main-e1][0]: query[(title:something^5.0
| description:something| attrs:something^3.0)~0.1],from[0],size[10]: Parse
Failure [No mapping found for [normalized_price]]]; "
} ]
},
"hits" : {

but all the other 4 are ok and the results are returned when in REST API
and not in Java API. Could there be some problem when the facets are checked
for merging and that index didn't run? any clue?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 4:28 PM, Sebastian Gavarini <sgavarini@gmail.com

wrote:
Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the
index while I did the search. Anyway, the line in question is:
                    for (Facet facet1 :
queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

Sebastian_Gavarini · October 17, 2011, 6:55pm

Sorry, forgot to add, the version was stated on my first email, it's 0.17.8

On Mon, Oct 17, 2011 at 3:48 PM, Sebastian Gavarini sgavarini@gmail.comwrote:

Hi Shay,

I took a couple of days to try to replicate the problem in an isolated
manner, previously it was possible to reproduce in our full java
application.

The problem appears sporadically in my 2 cores PC (once in a couple of runs
so pretty frequently anyway), so I would advice to run it a couple of times
to make it work. It seems to be related to a couple of simultaneous
conditions, two searches done in parallel, one search that is ok, and
another one that of the two indices, only one contains the sort field
requested.

In my example I create 3 indices, and run 2 searches, look at method
"launch".

Exception case · GitHub

IMPORTANT: The idea was to create in memory indices, but it isn't working,
I don't know what am I messing in the settings part, but the solution I
found was to delete everything in "/tmp/es/server" after each run.

Let me know if it doesn't work for you.

Thanks,
Sebastian.

On Fri, Oct 14, 2011 at 9:33 AM, Shay Banon kimchy@gmail.com wrote:
Which version are you using? If you can create it using hte Java API, can
you create a simple main program that starts a node, index sample data, and
executes a search that recreates it? Thats the simplest / fastest way to fix
it.

On Fri, Oct 14, 2011 at 2:36 AM, Sebastian Gavarini sgavarini@gmail.comwrote:
Sorry for the spam, forgot to add that I can't see the exact Java query
in my logs, because I am using:

String output = new
String(builder.internalBuilder().buildAsBytes(XContentType.JSON));

And all the extra parameters like indices, types, threading, search type,
timeout, (and I don't know if I am forgetting any other one) are not shown
by that line, any ideas on how to get a REST reproducible log from the Java
query?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 9:30 PM, Sebastian Gavarini <sgavarini@gmail.com

wrote:
Update: I still can't replicate with a "curl" command, but under
controlled conditions I do see it all the time with the java API.

The query runs without problems as REST, the only thing I don't know how
to specify in the REST API compared to java, is the Operation Threading,
witch is THREAD_PER_SHARD, I don't know if this could be part of the
problem.

What I see related to the facets iteration that SearchPhaseController is
doing in line 292 is that I am querying 5 indices, and in one of them the
value of the sort field is not mapped, so that index fails with:

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [ {
"index" : "index-main-e1",
"shard" : 0,
"status" : 400,
"reason" : "RemoteTransportException[[Tugun][inet[/127.0.0.1:5000]][search/phase/query+fetch]];
nested: SearchParseException[[index-main-e1][0]: ;
nested: SearchParseException[[index-main-e1][0]: query[(title:something^5.0
| description:something| attrs:something^3.0)~0.1],from[0],size[10]: Parse
Failure [No mapping found for [normalized_price]]]; "
} ]
},
"hits" : {

but all the other 4 are ok and the results are returned when in REST API
and not in Java API. Could there be some problem when the facets are checked
for merging and that index didn't run? any clue?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 4:28 PM, Sebastian Gavarini <
sgavarini@gmail.com> wrote:
Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the
index while I did the search. Anyway, the line in question is:
                    for (Facet facet1 :
queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

kimchy · October 18, 2011, 4:00am

Thanks for the recreation, managed to track down the problem and fix it:
Search: A failed search request might get overrun when trying another shard · Issue #1403 · elastic/elasticsearch · GitHub.

On Mon, Oct 17, 2011 at 8:55 PM, Sebastian Gavarini sgavarini@gmail.comwrote:

Sorry, forgot to add, the version was stated on my first email, it's 0.17.8

On Mon, Oct 17, 2011 at 3:48 PM, Sebastian Gavarini sgavarini@gmail.comwrote:
Hi Shay,

I took a couple of days to try to replicate the problem in an isolated
manner, previously it was possible to reproduce in our full java
application.

The problem appears sporadically in my 2 cores PC (once in a couple of
runs so pretty frequently anyway), so I would advice to run it a couple of
times to make it work. It seems to be related to a couple of simultaneous
conditions, two searches done in parallel, one search that is ok, and
another one that of the two indices, only one contains the sort field
requested.

In my example I create 3 indices, and run 2 searches, look at method
"launch".

Exception case · GitHub

IMPORTANT: The idea was to create in memory indices, but it isn't working,
I don't know what am I messing in the settings part, but the solution I
found was to delete everything in "/tmp/es/server" after each run.

Let me know if it doesn't work for you.

Thanks,
Sebastian.

On Fri, Oct 14, 2011 at 9:33 AM, Shay Banon kimchy@gmail.com wrote:
Which version are you using? If you can create it using hte Java API, can
you create a simple main program that starts a node, index sample data, and
executes a search that recreates it? Thats the simplest / fastest way to fix
it.

On Fri, Oct 14, 2011 at 2:36 AM, Sebastian Gavarini <sgavarini@gmail.com

wrote:
Sorry for the spam, forgot to add that I can't see the exact Java query
in my logs, because I am using:

String output = new
String(builder.internalBuilder().buildAsBytes(XContentType.JSON));

And all the extra parameters like indices, types, threading, search
type, timeout, (and I don't know if I am forgetting any other one) are not
shown by that line, any ideas on how to get a REST reproducible log from the
Java query?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 9:30 PM, Sebastian Gavarini <
sgavarini@gmail.com> wrote:
Update: I still can't replicate with a "curl" command, but under
controlled conditions I do see it all the time with the java API.

The query runs without problems as REST, the only thing I don't know
how to specify in the REST API compared to java, is the Operation Threading,
witch is THREAD_PER_SHARD, I don't know if this could be part of the
problem.

What I see related to the facets iteration that SearchPhaseController
is doing in line 292 is that I am querying 5 indices, and in one of them the
value of the sort field is not mapped, so that index fails with:

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [ {
"index" : "index-main-e1",
"shard" : 0,
"status" : 400,
"reason" : "RemoteTransportException[[Tugun][inet[/127.0.0.1:5000]][search/phase/query+fetch]];
nested: SearchParseException[[index-main-e1][0]: ;
nested: SearchParseException[[index-main-e1][0]: query[(title:something^5.0
| description:something| attrs:something^3.0)~0.1],from[0],size[10]: Parse
Failure [No mapping found for [normalized_price]]]; "
} ]
},
"hits" : {

but all the other 4 are ok and the results are returned when in REST
API and not in Java API. Could there be some problem when the facets are
checked for merging and that index didn't run? any clue?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 4:28 PM, Sebastian Gavarini <
sgavarini@gmail.com> wrote:
Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the
index while I did the search. Anyway, the line in question is:
                    for (Facet facet1 :
queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

Sebastian_Gavarini · October 18, 2011, 4:06am

Thanks for the quick fix!

On Tue, Oct 18, 2011 at 1:00 AM, Shay Banon kimchy@gmail.com wrote:

Thanks for the recreation, managed to track down the problem and fix it:
Search: A failed search request might get overrun when trying another shard · Issue #1403 · elastic/elasticsearch · GitHub.

On Mon, Oct 17, 2011 at 8:55 PM, Sebastian Gavarini sgavarini@gmail.comwrote:
Sorry, forgot to add, the version was stated on my first email, it's
0.17.8

On Mon, Oct 17, 2011 at 3:48 PM, Sebastian Gavarini sgavarini@gmail.comwrote:
Hi Shay,

I took a couple of days to try to replicate the problem in an isolated
manner, previously it was possible to reproduce in our full java
application.

The problem appears sporadically in my 2 cores PC (once in a couple of
runs so pretty frequently anyway), so I would advice to run it a couple of
times to make it work. It seems to be related to a couple of simultaneous
conditions, two searches done in parallel, one search that is ok, and
another one that of the two indices, only one contains the sort field
requested.

In my example I create 3 indices, and run 2 searches, look at method
"launch".

Exception case · GitHub

IMPORTANT: The idea was to create in memory indices, but it isn't
working, I don't know what am I messing in the settings part, but the
solution I found was to delete everything in "/tmp/es/server" after each
run.

Let me know if it doesn't work for you.

Thanks,
Sebastian.

On Fri, Oct 14, 2011 at 9:33 AM, Shay Banon kimchy@gmail.com wrote:
Which version are you using? If you can create it using hte Java API,
can you create a simple main program that starts a node, index sample data,
and executes a search that recreates it? Thats the simplest / fastest way to
fix it.

On Fri, Oct 14, 2011 at 2:36 AM, Sebastian Gavarini <
sgavarini@gmail.com> wrote:
Sorry for the spam, forgot to add that I can't see the exact Java query
in my logs, because I am using:

String output = new
String(builder.internalBuilder().buildAsBytes(XContentType.JSON));

And all the extra parameters like indices, types, threading, search
type, timeout, (and I don't know if I am forgetting any other one) are not
shown by that line, any ideas on how to get a REST reproducible log from the
Java query?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 9:30 PM, Sebastian Gavarini <
sgavarini@gmail.com> wrote:
Update: I still can't replicate with a "curl" command, but under
controlled conditions I do see it all the time with the java API.

The query runs without problems as REST, the only thing I don't know
how to specify in the REST API compared to java, is the Operation Threading,
witch is THREAD_PER_SHARD, I don't know if this could be part of the
problem.

What I see related to the facets iteration that SearchPhaseController
is doing in line 292 is that I am querying 5 indices, and in one of them the
value of the sort field is not mapped, so that index fails with:

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [ {
"index" : "index-main-e1",
"shard" : 0,
"status" : 400,
"reason" :
"RemoteTransportException[[Tugun][inet[/127.0.0.1:5000]][search/phase/query+fetch]];
nested: SearchParseException[[index-main-e1][0]: ;
nested: SearchParseException[[index-main-e1][0]: query[(title:something^5.0
| description:something| attrs:something^3.0)~0.1],from[0],size[10]: Parse
Failure [No mapping found for [normalized_price]]]; "
} ]
},
"hits" : {

but all the other 4 are ok and the results are returned when in REST
API and not in Java API. Could there be some problem when the facets are
checked for merging and that index didn't run? any clue?

Thanks,
Sebastian.

On Thu, Oct 13, 2011 at 4:28 PM, Sebastian Gavarini <
sgavarini@gmail.com> wrote:
Hi, I am seeing this Exception in ES 0.17.8:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [query_fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:231)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:206)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:237)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.handleResponse(SearchServiceTransportAction.java:230)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:292)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:226)
... 26 more

I am trying to reproduce it without luck so far, I was updating the
index while I did the search. Anyway, the line in question is:
                    for (Facet facet1 :
queryResultProvider.queryResult().facets()) {

and I think it shouldn't throw NPE whatever I am doing wrong.

Any ideas?

Thanks,
Sebastian.

Topic		Replies	Views
Query fails because of NullPointerException in QueryPhase Elasticsearch	8	1058	July 6, 2017
onQueryPhase listener org.elasticsearch.index.SearchSlowLog fails with NullpointerException Elasticsearch	4	545	November 12, 2021
Error in Elasticsearch NullPointerException at org.elasticsearch.plugins.PluginsService.loadSitePlugins Elasticsearch	9	917	July 6, 2017
TransportBulkAction - NullPointerException Elasticsearch	1	340	July 6, 2017
[ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats Elasticsearch	2	617	March 20, 2017

SearchPhaseController - NullPointerException

Related Topics