Filtered And Query

Elastic Stack Elasticsearch
1

Hi,

The documentation is very clear and comprehensive. But I run into an
error when trying to do a query with an AND filter for values on
different keys. I do a POST on
http://localhost:9200/twitter/tweet/_search I base myself on the
following documentation :
http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/filtered_query/

The JSON of the query I make is :

{
"filtered" : {
"query" : {
"term" : { "user" : "kimchy" }
},
"filter" : {
"and" : {
"term" : { "message" : "search" }
}
}
}
}

The error message I get starts with :
{"error":"SearchPhaseExecutionException[Failed to execute phase
[query], total failure; shardFailures
{[6eaf3f5a-f1da-4c5b-9af6-cb1e146ed655][twitter][3]:
SearchParseException[[twitter][3]: query[null],from[-1],size[-1]:
Parse Failure [Failed to parse

Any suggestions to overcome my stupidity?

2

The below json just realtes to the query part of the search request. You
need to wrap it in a "query" element, for example:

{
"query" : {
"filtered" " {...}
},
"from" : ...
}

(I put the from there to show that there are other things a search request
can have, such as from/size, facets and so on, and thats why there is the
"query" element).

-shay.banon

On Tue, Aug 31, 2010 at 2:30 AM, Ted Karmel ted.karmel@gmail.com wrote:

Hi,

The documentation is very clear and comprehensive. But I run into an
error when trying to do a query with an AND filter for values on
different keys. I do a POST on
http://localhost:9200/twitter/tweet/_search I base myself on the
following documentation :

http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/filtered_query/

The JSON of the query I make is :

{
"filtered" : {
"query" : {
"term" : { "user" : "kimchy" }
},
"filter" : {
"and" : {
"term" : { "message" : "search" }
}
}
}
}

The error message I get starts with :
{"error":"SearchPhaseExecutionException[Failed to execute phase
[query], total failure; shardFailures
{[6eaf3f5a-f1da-4c5b-9af6-cb1e146ed655][twitter][3]:
SearchParseException[[twitter][3]: query[null],from[-1],size[-1]:
Parse Failure [Failed to parse

Any suggestions to overcome my stupidity?

3

Thank you Shay for the prompt reply. Wrapping in a query element
makes design sense when considering this allows other useful
parameters to be incorporated (like from/size makes pagination a
cinch).

I no longer get an error. But I do not get the result I expect. I am
searching for all the tweets by the user kimchy that contain the term
"search" in the message. There is such a tweet. But, when I make the
search request with the following JSON, I get no hits as a return:

{
"query": {
"filtered" : {
"query" : {
"term" : { "user" : "kimchy" }
},
"filter" : {
"and" : {
"query": {
"term" : { "message" : "whatever" }
} }
}
}
}
}

On Tue, Aug 31, 2010 at 10:17 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

The below json just realtes to the query part of the search request. You
need to wrap it in a "query" element, for example:
{
"query" : {
"filtered" " {...}
},
"from" : ...
}

(I put the from there to show that there are other things a search request
can have, such as from/size, facets and so on, and thats why there is the
"query" element).
-shay.banon
On Tue, Aug 31, 2010 at 2:30 AM, Ted Karmel ted.karmel@gmail.com wrote:

Hi,

The documentation is very clear and comprehensive. But I run into an
error when trying to do a query with an AND filter for values on
different keys. I do a POST on
http://localhost:9200/twitter/tweet/_search I base myself on the
following documentation :

http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/filtered_query/

The JSON of the query I make is :

{
"filtered" : {
"query" : {
"term" : { "user" : "kimchy" }
},
"filter" : {
"and" : {
"term" : { "message" : "search" }
}
}
}
}

The error message I get starts with :
{"error":"SearchPhaseExecutionException[Failed to execute phase
[query], total failure; shardFailures
{[6eaf3f5a-f1da-4c5b-9af6-cb1e146ed655][twitter][3]:
SearchParseException[[twitter][3]: query[null],from[-1],size[-1]:
Parse Failure [Failed to parse

Any suggestions to overcome my stupidity?

4

On Tue, 2010-08-31 at 12:47 +0200, Ted Karmel wrote:

I no longer get an error. But I do not get the result I expect. I am
searching for all the tweets by the user kimchy that contain the term
"search" in the message. There is such a tweet. But, when I make the
search request with the following JSON, I get no hits as a return:

Try this:

{
"query": {
"filtered" : {
"query" : {
"field": {
"message": "whatever"
}
},
"filter" : {
"term" : { "user" : "kimchy" }
}
}
}
}

clint

5

is that because the query on the "message" field is for "whatever" rather
than "search"?

On Tue, Aug 31, 2010 at 12:07 PM, Clinton Gormley
clinton@iannounce.co.ukwrote:

On Tue, 2010-08-31 at 12:47 +0200, Ted Karmel wrote:

I no longer get an error. But I do not get the result I expect. I am
searching for all the tweets by the user kimchy that contain the term
"search" in the message. There is such a tweet. But, when I make the
search request with the following JSON, I get no hits as a return:

Try this:

{
"query": {
"filtered" : {
"query" : {
"field": {
* "message": "whatever"*
}
},
"filter" : {
"term" : { "user" : "kimchy" }
}
}
}
}

clint

--

Paul Loy
paul@keteracel.com
http://www.keteracel.com/paul

6

Hi Clint,

Thank you very much for that. It works! But not sure how... field
replaces the AND filter? Would it work for more than two fields? In
any case, thanks...

Paul,
The term whatever was just included afterwards to troubleshoot (no
such tweet includes that term). I just copied and pasted the JSON.
My bad... :slight_smile:

On Tue, Aug 31, 2010 at 1:12 PM, Paul Loy keteracel@gmail.com wrote:

is that because the query on the "message" field is for "whatever" rather
than "search"?

On Tue, Aug 31, 2010 at 12:07 PM, Clinton Gormley clinton@iannounce.co.uk
wrote:

On Tue, 2010-08-31 at 12:47 +0200, Ted Karmel wrote:

I no longer get an error. But I do not get the result I expect. I am
searching for all the tweets by the user kimchy that contain the term
"search" in the message. There is such a tweet. But, when I make the
search request with the following JSON, I get no hits as a return:

Try this:

{
"query": {
"filtered" : {
"query" : {
"field": {
"message": "whatever"
}
},
"filter" : {
"term" : { "user" : "kimchy" }
}
}
}
}

clint

--

Paul Loy
paul@keteracel.com
http://www.keteracel.com/paul

7

On Tue, 2010-08-31 at 13:35 +0200, Ted Karmel wrote:

Hi Clint,

Thank you very much for that. It works! But not sure how... field
replaces the AND filter? Would it work for more than two fields? In
any case, thanks...

No, the query in this case is "show me all docs which contain the word
'whatever' in the 'message' field"

You could equally have used:

"query: {
"query_string": {
"query": "whatever",
"default_field": "message"
}
}

It's important to put the part that affects the relevance (or score) of
the query into the 'query' bit. So if you searched for "foo bar" then
docs with "foo" and "bar" would be more relevant than docs with just
"foo" or just "bar".

Then, you only want posts by the user 'kimchy'. So that is a filter. It
doesn't need to be scored. Either, this post is by kimchy, or it isn't.

So you put that into the filter. Filters are more efficient, because
they don't need to be scored, and they can be cached.

You don't need an 'and' filter in this case, because you only have one
filter.

If you wanted to say:

filter by:

  • user == kimchy
  • AND created > '2010-01-01'

then you would use an 'and' filter.

clint

8

Thanks Clint. That info is quite useful - especially for how I will
set up schemas later on.

Cheers

On Tue, Aug 31, 2010 at 1:46 PM, Clinton Gormley
clinton@iannounce.co.uk wrote:

On Tue, 2010-08-31 at 13:35 +0200, Ted Karmel wrote:

Hi Clint,

Thank you very much for that. It works! But not sure how... field
replaces the AND filter? Would it work for more than two fields? In
any case, thanks...

No, the query in this case is "show me all docs which contain the word
'whatever' in the 'message' field"

You could equally have used:

"query: {
"query_string": {
"query": "whatever",
"default_field": "message"
}
}

It's important to put the part that affects the relevance (or score) of
the query into the 'query' bit. So if you searched for "foo bar" then
docs with "foo" and "bar" would be more relevant than docs with just
"foo" or just "bar".

Then, you only want posts by the user 'kimchy'. So that is a filter. It
doesn't need to be scored. Either, this post is by kimchy, or it isn't.

So you put that into the filter. Filters are more efficient, because
they don't need to be scored, and they can be cached.

You don't need an 'and' filter in this case, because you only have one
filter.

If you wanted to say:

filter by:

  • user == kimchy
  • AND created > '2010-01-01'

then you would use an 'and' filter.

clint