Elasticsearch Filter And Query

Elastic Stack Elasticsearch
1

Hi All

Good Morning to All

In elastic search query have Filter and Query..

Which first execute filter or query..

--
Regards

Shanmugam.M

--

2

AFAIK Filter first reduces the number of docs to query on.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 janv. 2013 à 07:48, Shanmugam Muthanandham shanmuthu83@gmail.com a écrit :

Hi All

Good Morning to All

In elastic search query have Filter and Query..

Which first execute filter or query..

--
Regards

Shanmugam.M

--

--

3

Okie Thanks David

On Fri, Jan 4, 2013 at 12:32 PM, David Pilato david@pilato.fr wrote:

AFAIK

--
Regards

Shanmugam.M

--

4

I would say it depends. If you are asking about top level filterhttp://www.elasticsearch.org/guide/reference/api/search/filter.htmland top
level queryhttp://www.elasticsearch.org/guide/reference/api/search/query.htmlthen query is executed first and then results are filtered.

On Friday, January 4, 2013 2:25:06 AM UTC-5, shanmugam wrote:

Okie Thanks David

On Fri, Jan 4, 2013 at 12:32 PM, David Pilato <da...@pilato.fr<javascript:>

wrote:

AFAIK

--
Regards

Shanmugam.M

--

5

Thanks Igor for correcting my mistake.
So, in which case filter is applied before the query?

Le 4 janvier 2013 à 14:24, Igor Motov imotov@gmail.com a écrit :

I would say it depends. If you are asking about top level filter
http://www.elasticsearch.org/guide/reference/api/search/filter.html and top
level query
http://www.elasticsearch.org/guide/reference/api/search/query.html then
query is executed first and then results are filtered.

On Friday, January 4, 2013 2:25:06 AM UTC-5, shanmugam wrote:

Okie Thanks David

On Fri, Jan 4, 2013 at 12:32 PM, David Pilato da...@pilato.fr wrote:
> > > AFAIK

--

Regards

Shanmugam.M

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

6

In 0.20.x and below, it's probably valid to say that filter is applied
before the query in the filtered query. Technically, both query and filter
documents are advanced together, but since filter is consulted first, it
can short-circuit the query execution. In 0.21.x it's more complicated and
depends on types of queries and filters and can be specifiedhttps://github.com/elasticsearch/elasticsearch/commit/2786e29a10ba2b92bfbff6d396dd2c401aa94193 on
the query.

On Friday, January 4, 2013 8:46:33 AM UTC-5, David Pilato wrote:

Thanks Igor for correcting my mistake.
So, in which case filter is applied before the query?

Le 4 janvier 2013 à 14:24, Igor Motov <imo...@gmail.com <javascript:>> a
écrit :

I would say it depends. If you are asking about top level filterhttp://www.elasticsearch.org/guide/reference/api/search/filter.htmland top
level queryhttp://www.elasticsearch.org/guide/reference/api/search/query.htmlthen
query is executed first and then results are filtered.

On Friday, January 4, 2013 2:25:06 AM UTC-5, shanmugam wrote:

Okie Thanks David

On Fri, Jan 4, 2013 at 12:32 PM, David Pilato da...@pilato.fr wrote:

AFAIK

--

Regards

Shanmugam.M

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

7

Thanks Igor!

Le 4 janvier 2013 à 16:32, Igor Motov imotov@gmail.com a écrit :

In 0.20.x and below, it's probably valid to say that filter is applied before
the query in the filtered query. Technically, both query and filter documents
are advanced together, but since filter is consulted first, it can
short-circuit the query execution. In 0.21.x it's more complicated and depends
on types of queries and filters and can be specified
https://github.com/elasticsearch/elasticsearch/commit/2786e29a10ba2b92bfbff6d396dd2c401aa94193
on the query.

--

8

I assume by "short circuit" you mean that if a filter results in NO
matches then the query is skipped.
Otherwise, how does running the filter first help in a scenerio where a
filter might reduce the total documents to 1% of all documents? My user
view was that two "bitsets"of matches: one from the query & one from the
filter where "and"ed together, so what matter would it make to do one
or the other 1st?
Apparently, such a simplistic view is incorrect. I'm interested to know
how to make filters (which might be cached) feed into a query, so the
query doesn't bother with as many docs and only processes that with has
been pre-filtered.

-Paul

On 1/4/2013 7:32 AM, Igor Motov wrote:

In 0.20.x and below, it's probably valid to say that filter is applied
before the query in the filtered query. Technically, both query and
filter documents are advanced together, but since filter is consulted
first, it can short-circuit the query execution. In 0.21.x it's more
complicated and depends on types of queries and filters and can be
specified
https://github.com/elasticsearch/elasticsearch/commit/2786e29a10ba2b92bfbff6d396dd2c401aa94193 on
the query.

--

9

Paul,

I think your "simplistic" view is correct in some cases. But there are also
cases where "bitsets" don't exist. Consider this somewhat absurd query for
example: filter_or_query.sh · GitHub

On Saturday, January 5, 2013 4:06:39 AM UTC-5, P Hill wrote:

I assume by "short circuit" you mean that if a filter results in NO
matches then the query is skipped.
Otherwise, how does running the filter first help in a scenerio where a
filter might reduce the total documents to 1% of all documents? My user
view was that two "bitsets"of matches: one from the query & one from the
filter where "and"ed together, so what matter would it make to do one or
the other 1st?
Apparently, such a simplistic view is incorrect. I'm interested to know
how to make filters (which might be cached) feed into a query, so the query
doesn't bother with as many docs and only processes that with has been
pre-filtered.

-Paul

On 1/4/2013 7:32 AM, Igor Motov wrote:

In 0.20.x and below, it's probably valid to say that filter is applied
before the query in the filtered query. Technically, both query and filter
documents are advanced together, but since filter is consulted first, it
can short-circuit the query execution. In 0.21.x it's more complicated and
depends on types of queries and filters and can be specifiedhttps://github.com/elasticsearch/elasticsearch/commit/2786e29a10ba2b92bfbff6d396dd2c401aa94193 on
the query.

--

10

My mental model does help me to see why this is a special case.

you inserted 10 docs each with a doc field 1 .. 10.
The filter finds 5, 6 AND 7; The query only matches 5 OR 7.
So why would each NOT be represented as a bitset (a compact list of docs
that match each represented in order by a yes or no)? Is it because of
the constant score? I'm not seeing it.

-Paul

On 1/7/2013 7:11 PM, Igor Motov wrote:

Paul,

I think your "simplistic" view is correct in some cases. But there are
also cases where "bitsets" don't exist. Consider this somewhat absurd
query for example: filter_or_query.sh · GitHub

On Saturday, January 5, 2013 4:06:39 AM UTC-5, P Hill wrote:

I assume by "short circuit" you mean that if a filter results in
NO matches then the query is skipped.
Otherwise, how does running the filter first help in a scenerio
where a filter might reduce the total documents to 1% of all
documents?  My user view was that two "bitsets"of matches: one
from the query & one from the filter where "and"ed together, so
what matter would  it make to do one or the other 1st?
Apparently, such a simplistic view is incorrect.  I'm interested
to know how to make filters (which might be cached) feed into a
query, so the query doesn't bother with as many docs and only
processes that with has been pre-filtered.

-Paul

On 1/4/2013 7:32 AM, Igor Motov wrote:

In 0.20.x and below, it's probably valid to say that filter is
applied before the query in the filtered query. Technically, both
query and filter documents are advanced together, but since
filter is consulted first, it can short-circuit the query
execution. In 0.21.x it's more complicated and depends on types
of queries and filters and can be specified
<https://github.com/elasticsearch/elasticsearch/commit/2786e29a10ba2b92bfbff6d396dd2c401aa94193> on
the query.

--

--