JSON ordering!? / solved SearchPhaseExecutionException[Failed to execute phase [query_fetch] issue

So it appears that ES wants its json hashes (for search, at least) to be in
a certain order. I was getting certain weird failures where it would
sometimes work and sometimes get the SearchPhaseExecutionException error.

This fails:

$ curl 

http://localhost:9200/discussions_test_68/discussion/_search?routing=68 -d
'{"query":{"filtered":{
>
"filter":{"and":{"must":[{"term":{"site_id":68}}]},"or":[{"term":{"public":"1"}},{"term":{"watchers":171}}]},
>
"query":{"bool":{"must":[{"text":{"_all":{"operator":"and","query":"testing
2"}}}]}}
> }}}'

{"error":"SearchPhaseExecutionException[Failed to execute phase 

[query_fetch], total failure; shardFailures
{[ogrJqB3SRM67cXPUZxx8rA][discussions_test][0]:
SearchParseException[[discussions_test][0]: from[-1],size[-1]: Parse
Failure [Failed to parse source [{"query":{"filtered":{
\n"filter":{"and":{"must":[{"term":{"site_id":68}}]},"or":[{"term":{"public":"1"}},{"term":{"watchers":171}}]},\n"query":{"bool":{"must":[{"text":{"_all":{"operator":"and","query":"testing
2"}}}]}}\n}}}]]]; nested: QueryParsingException[[discussions_test]
[filtered] requires 'query' element]; }]","status":500}

But then if we reverse the order of "filter" and "query" in
query[filtered], so that 'query' comes first --->

$ curl 

http://localhost:9200/discussions_test_68/discussion/_search?routing=68 -d
'{"query":{"filtered":{
>
"query":{"bool":{"must":[{"text":{"_all":{"operator":"and","query":"testing
2"}}}]}},
>
"filter":{"and":{"must":[{"term":{"site_id":68}}]},"or":[{"term":{"public":"1"}},{"term":{"watchers":171}}]}
> }}}'

{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Um, what? Am I crazy/stupid, or do you actually have to send the hash
values in a certain order for ES to accept them? FWIW the ruby library I am
using (ruby in general..) doesn't guarantee any particular ordering of
hash keys when serializing into JSON.

--Courtenay

--

On Tue, 2012-11-27 at 12:23 -0800, courtenay wrote:

So it appears that ES wants its json hashes (for search, at least) to
be in a certain order. I was getting certain weird failures where it
would sometimes work and sometimes get the
SearchPhaseExecutionException error.

It doesn't require a particular order, but it does require a valid
query, which yours is not. It uses a streaming JSON parser which may
fail differently based upon the order in which the bad query is
presented.

You are confusing and/or queries with bool

clint

--

On Tuesday, November 27, 2012 12:29:36 PM UTC-8, Clinton Gormley wrote:

On Tue, 2012-11-27 at 12:23 -0800, courtenay wrote:

So it appears that ES wants its json hashes (for search, at least) to
be in a certain order. I was getting certain weird failures where it
would sometimes work and sometimes get the
SearchPhaseExecutionException error.

It doesn't require a particular order, but it does require a valid
query, which yours is not. It uses a streaming JSON parser which may
fail differently based upon the order in which the bad query is
presented.

You are confusing and/or queries with bool

clint

Interesting -- but it doesn't always fail. So how is that a bad query, if
it sometimes works?

--

On Tuesday, November 27, 2012 12:37:25 PM UTC-8, courtenay wrote:

On Tuesday, November 27, 2012 12:29:36 PM UTC-8, Clinton Gormley wrote:

On Tue, 2012-11-27 at 12:23 -0800, courtenay wrote:

So it appears that ES wants its json hashes (for search, at least) to
be in a certain order. I was getting certain weird failures where it
would sometimes work and sometimes get the
SearchPhaseExecutionException error.

It doesn't require a particular order, but it does require a valid
query, which yours is not. It uses a streaming JSON parser which may
fail differently based upon the order in which the bad query is
presented.

You are confusing and/or queries with bool

clint

Interesting -- but it doesn't always fail. So how is that a bad query, if
it sometimes works?

(this is a simplified version of the actual queries we use -- I cut out a
bunch of other stuff like more conditions in "bool" )

--

Interesting -- but it doesn't always fail. So how is that a bad query,
if it sometimes works?

It may not throw an error (which I consider to be a bug) but it isn't
working, where "working" = "doing what you intend"

clint

--