Issue with query using named filter

Andrei · October 2, 2010, 3:37am

I'm trying to use the new named filters feature in 0.11 and getting a
QueryParsingException on this query:

{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
}
},
"_name": "foo"
},
{

                        "term": { "user_id": 11 },
                        "_name": "bar"
                }
            ]
        }
    }
}

}'

The trace is:

[20:36:30,713][DEBUG][action.search.type ] [Big Bertha]
[places_2010100101][0], node[cbecda40-810d-4f65-a163-4da95d9fe2dc],
[P], s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@107525ad]
org.elasticsearch.search.SearchParseException: [places_2010100101][0]:
from[-1],size[10]: Parse Failure [Failed to parse [
{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
}
},
"_name": "foo"
},
{

                        "term": { "user_id": 11 },
                        "_name": "bar"
                }
            ]
        }
    }
}

}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:
398)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:
311)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:
162)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:
131)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:
77)
at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:195)
at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction.access$000(TransportSearchTypeAction.java:81)
at org.elasticsearch.action.search.type.TransportSearchTypeAction
$BaseAsyncAction$1.run(TransportSearchTypeAction.java:154)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: org.elasticsearch.index.query.QueryParsingException:
[places_2010100101] No query parser registered for [null]
at
org.elasticsearch.index.query.xcontent.QueryParseContext.parseInnerFilter(QueryParseContext.java:
168)
at
org.elasticsearch.index.query.xcontent.OrFilterParser.parse(OrFilterParser.java:
64)
at
org.elasticsearch.index.query.xcontent.QueryParseContext.parseInnerFilter(QueryParseContext.java:
170)
at
org.elasticsearch.index.query.xcontent.FilteredQueryParser.parse(FilteredQueryParser.java:
67)
at
org.elasticsearch.index.query.xcontent.QueryParseContext.parseInnerQuery(QueryParseContext.java:
144)
at
org.elasticsearch.index.query.xcontent.XContentIndexQueryParser.parse(XContentIndexQueryParser.java:
234)
at
org.elasticsearch.index.query.xcontent.XContentIndexQueryParser.parse(XContentIndexQueryParser.java:
214)
at
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:
34)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:
385)
... 10 more

Clinton_Gormley · October 2, 2010, 6:46am

On Fri, 2010-10-01 at 20:37 -0700, Andrei wrote:

I'm trying to use the new named filters feature in 0.11 and getting a
QueryParsingException on this query:

I haven't tried these yet, but looking at the issue

github.com/elastic/elasticsearch

Search API: Allow to name filters and return per hit the filters it matched on

opened 09:23AM - 11 Sep 10 UTC

closed 09:38AM - 11 Sep 10 UTC

kimchy

>feature v0.11.0

This really make sense for `or` / `bool` filters. Now, each filter can accept a …`_name` in its top level definition, for example: ``` { "filtered" : { "query" : { "term" : { "name.first" : "shay" } }, "filter" : { "terms" : { "name.last" : ["banon", "kimchy"], "_name" : "test" } } } } ``` The search response will include for each hit the `matched_filters` it matched on. Note, the query filter had to be enhanced in order to support this. In order to set a name, the `fquery` filter should be used, which wraps a `query` (just so there will be a place to set a name for it), for example: ``` { "filtered" : { "query" : { "term" : { "name.first" : "shay" } }, "filter" : { "fquery" : { "query" : { "term" : { "name.last" : "banon" } }, "_name" : "test" } } } } ```

it seems that you should put the "_name" inside the filter:

{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
},
"_name": "foo"
}
},
{
"term": {
"user_id": 11,
"_name": "bar"
}
}
]
}
}
}
}'

clint

Andrei · October 2, 2010, 7:21am

That worked, thanks. Wish the error message was more helpful, though.

On Oct 1, 11:46 pm, Clinton Gormley clin...@iannounce.co.uk wrote:

On Fri, 2010-10-01 at 20:37 -0700, Andrei wrote:

I'm trying to use the new named filters feature in 0.11 and getting a
QueryParsingException on this query:

I haven't tried these yet, but looking at the issuehttp://github.com/elasticsearch/elasticsearch/issues/issue/364
it seems that you should put the "_name" inside the filter:

{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
},
"_name": "foo"
}
},
{
"term": {
"user_id": 11,
"_name": "bar"
}
}
]
}
}
}

}'

clint

Andrei · October 2, 2010, 7:25am

It seems that without the _name fields in the filters, the query runs
in < 0.01s, while with them it runs in about 2s. A drastic difference.
I only have about 120 documents in the index. Is there an explanation
for this?

time curl -XPOST "http://localhost:9200/places/place/_search?
pretty=true" -d'
{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
},
"_name": "geo_filter"
}
},
{
"term": {
"user_id": 11,
"_name": "user_filter"
}
}
]
}
}
}
}'

On Oct 2, 12:21 am, Andrei and...@zmievski.org wrote:

That worked, thanks. Wish the error message was more helpful, though.

kimchy · October 2, 2010, 12:45pm

There should not be that big different, can you post a test that fills
elasticsearch with data and recreates it? Note that the first time executing
it might take time, second executions should be faster.

-shay.banon

On Sat, Oct 2, 2010 at 9:25 AM, Andrei andrei@zmievski.org wrote:

It seems that without the _name fields in the filters, the query runs
in < 0.01s, while with them it runs in about 2s. A drastic difference.
I only have about 120 documents in the index. Is there an explanation
for this?

time curl -XPOST "http://localhost:9200/places/place/_search?
pretty=true" -d'
{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
},
"_name": "geo_filter"
}
},
{
"term": {
"user_id": 11,
"_name": "user_filter"
}
}
]
}
}
}
}'

On Oct 2, 12:21 am, Andrei and...@zmievski.org wrote:

That worked, thanks. Wish the error message was more helpful, though.

kimchy · October 2, 2010, 1:05pm

By first time, I meant first time something "touches" the lat/lon values, it
will cause them to be loaded, and then executions on them will be fast. Post
that, same filter execution will also be faster because filter results are
further cached.

On Sat, Oct 2, 2010 at 2:45 PM, Shay Banon shay.banon@elasticsearch.comwrote:

There should not be that big different, can you post a test that fills
elasticsearch with data and recreates it? Note that the first time executing
it might take time, second executions should be faster.

-shay.banon

On Sat, Oct 2, 2010 at 9:25 AM, Andrei andrei@zmievski.org wrote:

It seems that without the _name fields in the filters, the query runs
in < 0.01s, while with them it runs in about 2s. A drastic difference.
I only have about 120 documents in the index. Is there an explanation
for this?

time curl -XPOST "http://localhost:9200/places/place/_search?
pretty=true" -d'
{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
},
"_name": "geo_filter"
}
},
{
"term": {
"user_id": 11,
"_name": "user_filter"
}
}
]
}
}
}
}'

On Oct 2, 12:21 am, Andrei and...@zmievski.org wrote:

That worked, thanks. Wish the error message was more helpful, though.

Andrei · October 3, 2010, 12:24am

I posted the test file that contains a sequence of curl commands that
creates and populates the index and does the search with and without
named filters.

http://zmievski.org/dump/es11-test.zip

–Andrei

On Oct 2, 5:45 am, Shay Banon shay.ba...@elasticsearch.com wrote:

There should not be that big different, can you post a test that fills
elasticsearch with data and recreates it? Note that the first time executing
it might take time, second executions should be faster.

-shay.banon

kimchy · October 3, 2010, 8:20am

Hey,

That little extra data seems to have hit the curl problem discussed in
another thread:
http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/c99db77c9f92af99#.
If you add to the curl command: -H 'Expect:', then it will run as fast as it
should be.

I am trying to see if its possible to support this expected behavior from
curl, but the http lib I use does not have a nice extension point for
handling it, so working on a patch for them. In any case, that extra
chitchat is better disabled even if there is support for this curl
behavior.

-shay.banon

On Sun, Oct 3, 2010 at 2:24 AM, Andrei andrei@zmievski.org wrote:

I posted the test file that contains a sequence of curl commands that
creates and populates the index and does the search with and without
named filters.

http://zmievski.org/dump/es11-test.zip

–Andrei

On Oct 2, 5:45 am, Shay Banon shay.ba...@elasticsearch.com wrote:

There should not be that big different, can you post a test that fills
elasticsearch with data and recreates it? Note that the first time
executing
it might take time, second executions should be faster.

-shay.banon

Andrei · October 4, 2010, 1:07am

Ran into another snag, where I can't seem to attach a _name field to
an "and" filter inside an "or" filter:

{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"term": { "user_id": 11, "_name": "user" }
},
{
"and": [
{ "terms": { "user_id":
[19,23,100,101] } },
{ "range": { "privacy": { "gt": 0 } } }
],
"_name": "contacts"
}
]
}
}
}
}

This gives me:

...
Caused by: org.elasticsearch.index.query.QueryParsingException:
[places_2010100101] [or] filter requires 'filters' to be set on it'
...

Where should the _name field go in that "and" filter?

–Andrei

On Oct 1, 11:46 pm, Clinton Gormley clin...@iannounce.co.uk wrote:

On Fri, 2010-10-01 at 20:37 -0700, Andrei wrote:

I'm trying to use the new named filters feature in 0.11 and getting a
QueryParsingException on this query:

I haven't tried these yet, but looking at the issuehttp://github.com/elasticsearch/elasticsearch/issues/issue/364
it seems that you should put the "_name" inside the filter:

{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
},
"_name": "foo"
}
},
{
"term": {
"user_id": 11,
"_name": "bar"
}
}
]
}
}
}

}'

clint

kimchy · October 4, 2010, 10:18am

With an and filter, you need to put the filters under a filter field (as
an array), and then provide the _name there, for example:

"and" : {
"filters" : [
{ "term" : .. },
{ "term" ...}
],
"_name" : "the filter name"
}

On Mon, Oct 4, 2010 at 3:07 AM, Andrei andrei@zmievski.org wrote:

Ran into another snag, where I can't seem to attach a _name field to
an "and" filter inside an "or" filter:

{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"term": { "user_id": 11, "_name": "user" }
},
{
"and": [
{ "terms": { "user_id":
[19,23,100,101] } },
{ "range": { "privacy": { "gt": 0 } } }
],
"_name": "contacts"
}
]
}
}
}
}

This gives me:

...
Caused by: org.elasticsearch.index.query.QueryParsingException:
[places_2010100101] [or] filter requires 'filters' to be set on it'
...

Where should the _name field go in that "and" filter?

–Andrei

On Oct 1, 11:46 pm, Clinton Gormley clin...@iannounce.co.uk wrote:

On Fri, 2010-10-01 at 20:37 -0700, Andrei wrote:

I'm trying to use the new named filters feature in 0.11 and getting a
QueryParsingException on this query:

I haven't tried these yet, but looking at the issuehttp://
Search API: Allow to name filters and return per hit the filters it matched on · Issue #364 · elastic/elasticsearch · GitHub
it seems that you should put the "_name" inside the filter:

{
"size": 10,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"or": [
{
"geo_bounding_box": {
"geo": {
"top_left" : {
"lat" : 47.55,
"lon" : -124.10
},
"bottom_right" : {
"lat" : 31.06,
"lon" : -77
}
},
"_name": "foo"
}
},
{
"term": {
"user_id": 11,
"_name": "bar"
}
}
]
}
}
}

}'

clint