# No noticeable change in "filtered" response time - only when using simple query

**URL:** <https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748>\
**Category:** Elasticsearch\
**Created:** [July 10, 2013, 6:56pm UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748 "2013-07-10T18:56:41Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Martin\_Konecny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_konecny/32/2239_2.png) [@Martin\_Konecny](https://discuss.elastic.co/u/Martin_Konecny)\
**Post date:** [July 10, 2013, 6:56pm UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748/1 "2013-07-10T18:56:41Z")

</div>

Consider the following simple query

{  
"query":{  
"match\_all":{

```
  }

```

},  
"filter":{  
"bool":{  
"must":[  
{  
"terms":{  
"gender":[  
"m"  
]  
}  
}  
]  
}  
},  
"sort":[  
{  
"sub":{  
"order":"desc"  
}  
}  
],  
"from":10,  
"size":10  
}

This was taking 800ms when run on 50 million records. I tried speeding  
things up using "filtered", but the response time remains the same:

{  
"query":{  
"filtered":{  
"query":{  
"match\_all":{

```
        }
     },
     "filter":{
        "bool":{
           "must":[
              {
                 "terms":{
                    "gender":[
                       "m"
                    ]
                 }
              }
           ]
        }
     }
  }

```

},  
"sort":[  
{  
"sub":{  
"order":"desc"  
}  
}  
],  
"from":10,  
"size":10  
}

Note that in these two queries, the "must" parameter is:

[{"terms":{"gender":["m","f",""]}}]

If I increase the "must" parameters to

[  
{"term":{"sr\_loc":"1"}},  
{"range":{"birth\_es\_date":{"from":"19770101","to":"19970527"}}},  
{"term":{"loc":"SA"}},  
{"terms":{"gender":["f"]}}  
]

Then there is a huge difference between the before and after "filtered"  
optimization (drops from 800ms to 30).

Is it because the simpler "must" parameter returns a much larger result set  
which cannot be cached?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 10, 2013, 7:39pm UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748/2 "2013-07-10T19:39:09Z")

</div>

Hi

So first, the top level "filter" parameter should only be used when you are  
wanting to facet on unfiltered results, but filter the search results. At  
all other times, you should use a "filtered" query instead.

That said, your filter on gender probably matches about 25 million  
documents, which you are then sorting. I'm guessing that the sort is  
taking a disproportional amount of time.

Normally, a filtered query will try to apply the filters before running the  
query. In your example where your filter matches lots of documents, if you  
were to combine that with a simple query (eg { match: { name: "ezekiel"}})  
then this query may actually be faster than the filter, as "ezekiel" is  
likely to appear in far fewer documents than gender "m". The filtered  
query does try to detect these anomalies, but this can also be controlled  
by the undocumented "strategy" parameter.

However, queries usually come from users, and it is difficult to know in  
advance if they are going to be simple (and fast) or complex (and slower)  
queries. Using the default strategy for filtered at least gives you some  
consistency in response times, by reducing the total number of docs that  
the query has to examine.

Your example where you include several must clauses is doing just that -  
reducing the total number of docs that the query needs to examine by a much  
larger percentage than your first query.

Note: all cached filters are the same size. it doesn't depend on how many  
docs match or not. It uses a bitset to represent every doc in the index,  
with each bit set to either 1 or 0

Clint

On 10 July 2013 20:56, Martin Konecny [martin.konecny@gmail.com](mailto:martin.konecny@gmail.com) wrote:

> Consider the following simple query
> 
> {  
> "query":{  
> "match\_all":{
> 
> ```
> }
> 
> ```
> 
> },  
> "filter":{  
> "bool":{  
> "must":[  
> {  
> "terms":{  
> "gender":[  
> "m"  
> ]  
> }  
> }  
> ]  
> }  
> },  
> "sort":[  
> {  
> "sub":{  
> "order":"desc"  
> }  
> }  
> ],  
> "from":10,  
> "size":10  
> }
> 
> This was taking 800ms when run on 50 million records. I tried speeding  
> things up using "filtered", but the response time remains the same:
> 
> {  
> "query":{  
> "filtered":{  
> "query":{  
> "match\_all":{
> 
> ```
> }
> },
> "filter":{
> "bool":{
> "must":[
> {
> "terms":{
> "gender":[
> "m"
> ]
> }
> }
> ]
> }
> }
> }
> 
> ```
> 
> },  
> "sort":[  
> {  
> "sub":{  
> "order":"desc"  
> }  
> }  
> ],  
> "from":10,  
> "size":10  
> }
> 
> Note that in these two queries, the "must" parameter is:
> 
> [{"terms":{"gender":["m","f",""]}}]
> 
> If I increase the "must" parameters to
> 
> [  
> {"term":{"sr\_loc":"1"}},  
> {"range":{"birth\_es\_date":{"from":"19770101","to":"19970527"}}},  
> {"term":{"loc":"SA"}},  
> {"terms":{"gender":["f"]}}  
> ]
> 
> Then there is a huge difference between the before and after "filtered"  
> optimization (drops from 800ms to 30).
> 
> Is it because the simpler "must" parameter returns a much larger result  
> set which cannot be cached?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Martin\_Konecny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_konecny/32/2239_2.png) [@Martin\_Konecny](https://discuss.elastic.co/u/Martin_Konecny)\
**Post date:** [July 10, 2013, 8:07pm UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748/3 "2013-07-10T20:07:04Z")

</div>

There actually isn't any "text searches" being done by the user. Hence the  
match\_all:{} parameter - we are simply providing a UI for the user to  
filter all users based on some criteria - no text input for searching.

Is there some optimization I could do in this case? I understand what you  
said that the filtered result set is very large, and therefore the query  
takes a long time on it, but in this case, elasticsearch should not be  
apply a query to the result set (there is no query!).

M

On Wednesday, 10 July 2013 15:39:09 UTC-4, Clinton Gormley wrote:

> Hi
> 
> So first, the top level "filter" parameter should only be used when you  
> are wanting to facet on unfiltered results, but filter the search results.  
> At all other times, you should use a "filtered" query instead.
> 
> That said, your filter on gender probably matches about 25 million  
> documents, which you are then sorting. I'm guessing that the sort is  
> taking a disproportional amount of time.
> 
> Normally, a filtered query will try to apply the filters before running  
> the query. In your example where your filter matches lots of documents, if  
> you were to combine that with a simple query (eg { match: { name:  
> "ezekiel"}}) then this query may actually be faster than the filter, as  
> "ezekiel" is likely to appear in far fewer documents than gender "m". The  
> filtered query does try to detect these anomalies, but this can also be  
> controlled by the undocumented "strategy" parameter.
> 
> However, queries usually come from users, and it is difficult to know in  
> advance if they are going to be simple (and fast) or complex (and slower)  
> queries. Using the default strategy for filtered at least gives you some  
> consistency in response times, by reducing the total number of docs that  
> the query has to examine.
> 
> Your example where you include several must clauses is doing just that -  
> reducing the total number of docs that the query needs to examine by a much  
> larger percentage than your first query.
> 
> Note: all cached filters are the same size. it doesn't depend on how many  
> docs match or not. It uses a bitset to represent every doc in the index,  
> with each bit set to either 1 or 0
> 
> Clint
> 
> On 10 July 2013 20:56, Martin Konecny \<[martin....@gmail.com](mailto:martin....@gmail.com) \<javascript:\>\>wrote:
> 
> > Consider the following simple query
> > 
> > {  
> > "query":{  
> > "match\_all":{
> > 
> > ```
> > }
> > 
> > ```
> > 
> > },  
> > "filter":{  
> > "bool":{  
> > "must":[  
> > {  
> > "terms":{  
> > "gender":[  
> > "m"  
> > ]  
> > }  
> > }  
> > ]  
> > }  
> > },  
> > "sort":[  
> > {  
> > "sub":{  
> > "order":"desc"  
> > }  
> > }  
> > ],  
> > "from":10,  
> > "size":10  
> > }
> > 
> > This was taking 800ms when run on 50 million records. I tried speeding  
> > things up using "filtered", but the response time remains the same:
> > 
> > {  
> > "query":{  
> > "filtered":{  
> > "query":{  
> > "match\_all":{
> > 
> > ```
> > }
> > },
> > "filter":{
> > "bool":{
> > "must":[
> > {
> > "terms":{
> > "gender":[
> > "m"
> > ]
> > }
> > }
> > ]
> > }
> > }
> > }
> > 
> > ```
> > 
> > },  
> > "sort":[  
> > {  
> > "sub":{  
> > "order":"desc"  
> > }  
> > }  
> > ],  
> > "from":10,  
> > "size":10  
> > }
> > 
> > Note that in these two queries, the "must" parameter is:
> > 
> > [{"terms":{"gender":["m","f",""]}}]
> > 
> > If I increase the "must" parameters to
> > 
> > [  
> > {"term":{"sr\_loc":"1"}},  
> > {"range":{"birth\_es\_date":{"from":"19770101","to":"19970527"}}},  
> > {"term":{"loc":"SA"}},  
> > {"terms":{"gender":["f"]}}  
> > ]
> > 
> > Then there is a huge difference between the before and after "filtered"  
> > optimization (drops from 800ms to 30).
> > 
> > Is it because the simpler "must" parameter returns a much larger result  
> > set which cannot be cached?
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 11, 2013, 1:08pm UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748/4 "2013-07-11T13:08:27Z")

</div>

Hi Martin

On 10 July 2013 22:07, Martin Konecny [martin.konecny@gmail.com](mailto:martin.konecny@gmail.com) wrote:

> There actually isn't any "text searches" being done by the user. Hence the  
> match\_all:{} parameter - we are simply providing a UI for the user to  
> filter all users based on some criteria - no text input for searching.
> 
> Is there some optimization I could do in this case? I understand what you  
> said that the filtered result set is very large, and therefore the query  
> takes a long time on it, but in this case, elasticsearch should not be  
> apply a query to the result set (there is no query!).

Sure - the match\_all query is optimized for such cases already. So  
essentially just the filter is being applied. As I said, I reckon that  
most of the time is being consumed by sorting 25 million documents.

In order to optimize this, you want to reduce the number of documents that  
match. You're sorting on the "sub" field. I have no idea what this field  
contains but I'll pretend that it has values 1..100. If you know that you  
have eg more than 1000 results where gender=m and sub \> 90, then just add  
in another filter on sub, eg:

{  
"query": {  
"filtered": {  
"filter": {  
"bool": {  
"must": [  
{ "range": { "sub": { "gte": 90 }}},  
{ "term": { "gender": "m" }}  
]  
}  
}  
}  
},  
"sort": { "sub": { "order": "desc"}}  
}

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [July 11, 2013, 7:03pm UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748/5 "2013-07-11T19:03:09Z")

</div>

Hi Clinton,

On Wed, Jul 10, 2013 at 12:39 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:

> So first, the top level "filter" parameter should only be used when you  
> are wanting to facet on unfiltered results, but filter the search results.  
> At all other times, you should use a "filtered" query instead.

I am curious about your statement regarding facets on unfiltered results.  
In the past, I have seen no result differences with facets using  
query+filter versus filtered queries. What differences should occur?  
Ultimately I use facet filters (using the same filter) so the type of query  
shouldn't matter, but I haven't done much testing. Wondering if there is  
something that I might have missed.

Ivan

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [July 11, 2013, 7:15pm UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748/6 "2013-07-11T19:15:28Z")

</div>

I just ran some tests, and there is a difference between queries. It has  
been a while since I have compared different queries. Might have to change  
my logic around, thanks to Martin for starting this conversation.

--  
Ivan

On Thu, Jul 11, 2013 at 12:03 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:

> Hi Clinton,
> 
> On Wed, Jul 10, 2013 at 12:39 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
> 
> > So first, the top level "filter" parameter should only be used when you  
> > are wanting to facet on unfiltered results, but filter the search results.  
> > At all other times, you should use a "filtered" query instead.
> 
> I am curious about your statement regarding facets on unfiltered results.  
> In the past, I have seen no result differences with facets using  
> query+filter versus filtered queries. What differences should occur?  
> Ultimately I use facet filters (using the same filter) so the type of query  
> shouldn't matter, but I haven't done much testing. Wondering if there is  
> something that I might have missed.
> 
> Ivan

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:26am UTC](https://discuss.elastic.co/t/no-noticeable-change-in-filtered-response-time-only-when-using-simple-query/12748/7 "2017-07-06T02:26:55Z")

</div>