# Elasticsearch Sorting before applying filters? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 17, 2015, 12:04am UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199 "2015-02-17T00:04:33Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jamesma](https://avatars.discourse-cdn.com/v4/letter/j/c4cdca/32.png) [@jamesma](https://discuss.elastic.co/u/jamesma)\ **Post date:** [February 17, 2015, 12:04am UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/1 "2015-02-17T00:04:33Z")
Hello, I have noticed that when I'm applying sorting in my queries, the return times are remaining relatively constant at ~1.5s no matter what filters I apply (Even if I get 0 hits). It seems like ES might be sorting the entire index, then applying the filters after. As a test I checked how long it took ES to sort the entire index and it was about 2s. Here are some results for different queries comparing how many hits they got to how much time they took(ms): Took: 1563 Hits: 7000 Took: 1660 Hits: 239360 Took: 1657 Hits: 0 As you can see the time the queries are taking remains relatively constant when I have sorting on no matter how many hits I get. What is happening here? As a baseline if I remove sorting I get the following: Took: 150 Hits: 7000 Took: 134 Hits: 239360 Took: 0 Hits: 0 \*All these tests were done with a return size of 500, so I would expect queries with more hits to return slightly faster -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 17, 2015, 6:57am UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/2 "2015-02-17T06:57:08Z")
It probably depends on what your query looks like, it'd help if you could gist/pastebin/etc a link to a sample. On 17 February 2015 at 11:04, [jamesma@opendns.com](mailto:jamesma@opendns.com) wrote: > Hello, > > I have noticed that when I'm applying sorting in my queries, the return > times are remaining relatively constant at ~1.5s no matter what filters I > apply (Even if I get 0 hits). It seems like ES might be sorting the > entire index, then applying the filters after. > > As a test I checked how long it took ES to sort the entire index and it > was about 2s. > > Here are some results for different queries comparing how many hits they > got to how much time they took(ms): > > Took: 1563 > Hits: 7000 > > Took: 1660 > Hits: 239360 > > Took: 1657 > Hits: 0 > > As you can see the time the queries are taking remains relatively constant > when I have sorting on no matter how many hits I get. What is happening > here? > > As a baseline if I remove sorting I get the following: > > Took: 150 > Hits: 7000 > > Took: 134 > Hits: 239360 > > Took: 0 > Hits: 0 > > \*All these tests were done with a return size of 500, so I would expect > queries with more hits to return slightly faster > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_wH%3DTb%3DwL1GTJLkROWOvkoww395WaG-B5NKz0y2Hc%2BMw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_wH%3DTb%3DwL1GTJLkROWOvkoww395WaG-B5NKz0y2Hc%2BMw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jamesma](https://avatars.discourse-cdn.com/v4/letter/j/c4cdca/32.png) [@jamesma](https://discuss.elastic.co/u/jamesma)\ **Post date:** [February 17, 2015, 4:59pm UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/3 "2015-02-17T16:59:10Z")
We are working in PyES, our queries currently look like this, but we have tried putting the sort directly into the body too. It made no difference def termQuery(org, field, term, write, size=500): q = { "query": { "filtered": { "filter": { "and": [ {"term": {"Org ID": org}}, {"term": {field: term}} ] } } } } r = es.search(index='\_all', body=q, size=size, sort='Timestamp:desc') On Monday, February 16, 2015 at 4:04:33 PM UTC-8, Jay wrote: > Hello, > > I have noticed that when I'm applying sorting in my queries, the return > times are remaining relatively constant at ~1.5s no matter what filters I > apply (Even if I get 0 hits). It seems like ES might be sorting the > entire index, then applying the filters after. > > As a test I checked how long it took ES to sort the entire index and it > was about 2s. > > Here are some results for different queries comparing how many hits they > got to how much time they took(ms): > > Took: 1563 > Hits: 7000 > > Took: 1660 > Hits: 239360 > > Took: 1657 > Hits: 0 > > As you can see the time the queries are taking remains relatively constant > when I have sorting on no matter how many hits I get. What is happening > here? > > As a baseline if I remove sorting I get the following: > > Took: 150 > Hits: 7000 > > Took: 134 > Hits: 239360 > > Took: 0 > Hits: 0 > > \*All these tests were done with a return size of 500, so I would expect > queries with more hits to return slightly faster -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9001a0ca-6d58-4d38-839e-d30840df9dbc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9001a0ca-6d58-4d38-839e-d30840df9dbc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jamesma](https://avatars.discourse-cdn.com/v4/letter/j/c4cdca/32.png) [@jamesma](https://discuss.elastic.co/u/jamesma)\ **Post date:** [February 17, 2015, 5:59pm UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/4 "2015-02-17T17:59:22Z")
We are working in PyES, our queries currently look like this, but we have tried putting the sort directly into the body too. It made no difference def termQuery(org, field, term, write, size=500): q = { "query": { "filtered": { "filter": { "and": [ {"term": {"Org ID": org}}, {"term": {field: term}} ] } } } } r = es.search(index='\_all', body=q, size=size, sort='Timestamp:desc') On Monday, February 16, 2015 at 10:57:36 PM UTC-8, Mark Walkom wrote: > It probably depends on what your query looks like, it'd help if you could > gist/pastebin/etc a link to a sample. > > On 17 February 2015 at 11:04, \<[jam...@opendns.com](mailto:jam...@opendns.com) \\> wrote: > > > Hello, > > > > I have noticed that when I'm applying sorting in my queries, the return > > times are remaining relatively constant at ~1.5s no matter what filters I > > apply (Even if I get 0 hits). It seems like ES might be sorting the > > entire index, then applying the filters after. > > > > As a test I checked how long it took ES to sort the entire index and it > > was about 2s. > > > > Here are some results for different queries comparing how many hits they > > got to how much time they took(ms): > > > > Took: 1563 > > Hits: 7000 > > > > Took: 1660 > > Hits: 239360 > > > > Took: 1657 > > Hits: 0 > > > > As you can see the time the queries are taking remains relatively > > constant when I have sorting on no matter how many hits I get. What is > > happening here? > > > > As a baseline if I remove sorting I get the following: > > > > Took: 150 > > Hits: 7000 > > > > Took: 134 > > Hits: 239360 > > > > Took: 0 > > Hits: 0 > > > > \*All these tests were done with a return size of 500, so I would expect > > queries with more hits to return slightly faster > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Bipul\_Dutta](https://avatars.discourse-cdn.com/v4/letter/b/db5fbb/32.png) [@Bipul\_Dutta](https://discuss.elastic.co/u/Bipul_Dutta)\ **Post date:** [February 17, 2015, 6:32pm UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/5 "2015-02-17T18:32:32Z")
Wondering if adding search\_type=dfs\_query\_then\_fetch will change anything. On Tue, Feb 17, 2015 at 9:59 AM, Jay [jamesma@opendns.com](mailto:jamesma@opendns.com) wrote: > We are working in PyES, our queries currently look like this, but we have > tried putting the sort directly into the body too. It made no difference > > def termQuery(org, field, term, write, size=500): > q = { > "query": { > "filtered": { > "filter": { > "and": [ > {"term": {"Org ID": org}}, > {"term": {field: term}} > ] > } > } > } > } > r = es.search(index='\_all', body=q, size=size, sort='Timestamp:desc') > > On Monday, February 16, 2015 at 10:57:36 PM UTC-8, Mark Walkom wrote: > > > It probably depends on what your query looks like, it'd help if you could > > gist/pastebin/etc a link to a sample. > > > > On 17 February 2015 at 11:04, [jam...@opendns.com](mailto:jam...@opendns.com) wrote: > > > > > Hello, > > > > > > I have noticed that when I'm applying sorting in my queries, the return > > > times are remaining relatively constant at ~1.5s no matter what filters I > > > apply (Even if I get 0 hits). It seems like ES might be sorting the > > > entire index, then applying the filters after. > > > > > > As a test I checked how long it took ES to sort the entire index and it > > > was about 2s. > > > > > > Here are some results for different queries comparing how many hits they > > > got to how much time they took(ms): > > > > > > Took: 1563 > > > Hits: 7000 > > > > > > Took: 1660 > > > Hits: 239360 > > > > > > Took: 1657 > > > Hits: 0 > > > > > > As you can see the time the queries are taking remains relatively > > > constant when I have sorting on no matter how many hits I get. What is > > > happening here? > > > > > > As a baseline if I remove sorting I get the following: > > > > > > Took: 150 > > > Hits: 7000 > > > > > > Took: 134 > > > Hits: 239360 > > > > > > Took: 0 > > > Hits: 0 > > > > > > \*All these tests were done with a return size of 500, so I would expect > > > queries with more hits to return slightly faster > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- This message contains confidential information and is intended only for the individual named. If you are not the named addressee, any delivery, disclosure, dissemination or distribution of this e-mail is unlawful and strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error, as information could be intercepted, corrupted, lost, destroyed, arrive late, be incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which errors or omissions arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. (Res.Sys-dct1) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jamesma](https://avatars.discourse-cdn.com/v4/letter/j/c4cdca/32.png) [@jamesma](https://discuss.elastic.co/u/jamesma)\ **Post date:** [February 17, 2015, 6:58pm UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/6 "2015-02-17T18:58:54Z")
Hey so we figured out that the constant slow query speed when sorting is because we were clearing the field data before every query which made ES have to load the field data back to cache each time. Note ES doesn't just load the field data for the hits of the specific query when sorting, it will load the field data for every doc in the entire index, which was causing our performance issue. On Tue, Feb 17, 2015 at 10:32 AM, Bipul Dutta [bipul.dutta@resolvesys.com](mailto:bipul.dutta@resolvesys.com) wrote: > Wondering if adding search\_type=dfs\_query\_then\_fetch will change anything. > > On Tue, Feb 17, 2015 at 9:59 AM, Jay [jamesma@opendns.com](mailto:jamesma@opendns.com) wrote: > > > We are working in PyES, our queries currently look like this, but we have > > tried putting the sort directly into the body too. It made no difference > > > > def termQuery(org, field, term, write, size=500): > > q = { > > "query": { > > "filtered": { > > "filter": { > > "and": [ > > {"term": {"Org ID": org}}, > > {"term": {field: term}} > > ] > > } > > } > > } > > } > > r = es.search(index='\_all', body=q, size=size, sort='Timestamp:desc') > > > > On Monday, February 16, 2015 at 10:57:36 PM UTC-8, Mark Walkom wrote: > > > > > It probably depends on what your query looks like, it'd help if you > > > could gist/pastebin/etc a link to a sample. > > > > > > On 17 February 2015 at 11:04, [jam...@opendns.com](mailto:jam...@opendns.com) wrote: > > > > > > > Hello, > > > > > > > > I have noticed that when I'm applying sorting in my queries, the > > > > return times are remaining relatively constant at ~1.5s no matter what > > > > filters I apply (Even if I get 0 hits). It seems like ES might be > > > > sorting the entire index, then applying the filters after. > > > > > > > > As a test I checked how long it took ES to sort the entire index and > > > > it was about 2s. > > > > > > > > Here are some results for different queries comparing how many hits > > > > they got to how much time they took(ms): > > > > > > > > Took: 1563 > > > > Hits: 7000 > > > > > > > > Took: 1660 > > > > Hits: 239360 > > > > > > > > Took: 1657 > > > > Hits: 0 > > > > > > > > As you can see the time the queries are taking remains relatively > > > > constant when I have sorting on no matter how many hits I get. What is > > > > happening here? > > > > > > > > As a baseline if I remove sorting I get the following: > > > > > > > > Took: 150 > > > > Hits: 7000 > > > > > > > > Took: 134 > > > > Hits: 239360 > > > > > > > > Took: 0 > > > > Hits: 0 > > > > > > > > \*All these tests were done with a return size of 500, so I would expect > > > > queries with more hits to return slightly faster > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > This message contains confidential information and is intended only for > the individual named. If you are not the named addressee, any delivery, > disclosure, dissemination or distribution of this e-mail is unlawful and > strictly prohibited. Please notify the sender immediately by e-mail if > you have received this e-mail by mistake and delete this e-mail from your > system. E-mail transmissions cannot be guaranteed to be secure or without > error, as information could be intercepted, corrupted, lost, destroyed, > arrive late, be incomplete, or contain viruses. The sender therefore does > not accept liability for any errors or omissions in the contents of this > message, which errors or omissions arise as a result of e-mail > transmission. If verification is required, please request a hard-copy > version. (Res.Sys-dct1) > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/xPewWFV-LKo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/xPewWFV-LKo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL\_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [February 17, 2015, 7:44pm UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/7 "2015-02-17T19:44:15Z")
James, your analysis is correct. The way sorting works with elasticsearch is that the query returns an iterator over the documents that are matched and then elasticsearch consumes this iterator and puts documents into a priority queue in order to select the top ones. On Tue, Feb 17, 2015 at 7:58 PM, James Ma [jamesma@opendns.com](mailto:jamesma@opendns.com) wrote: > Hey so we figured out that the constant slow query speed when sorting is > because we were clearing the field data before every query which made ES > have to load the field data back to cache each time. > > Note ES doesn't just load the field data for the hits of the specific > query when sorting, it will load the field data for every doc in the entire > index, which was causing our performance issue. > > On Tue, Feb 17, 2015 at 10:32 AM, Bipul Dutta [bipul.dutta@resolvesys.com](mailto:bipul.dutta@resolvesys.com) > wrote: > > > Wondering if adding search\_type=dfs\_query\_then\_fetch will change anything. > > > > On Tue, Feb 17, 2015 at 9:59 AM, Jay [jamesma@opendns.com](mailto:jamesma@opendns.com) wrote: > > > > > We are working in PyES, our queries currently look like this, but we > > > have tried putting the sort directly into the body too. It made no > > > difference > > > > > > def termQuery(org, field, term, write, size=500): > > > q = { > > > "query": { > > > "filtered": { > > > "filter": { > > > "and": [ > > > {"term": {"Org ID": org}}, > > > {"term": {field: term}} > > > ] > > > } > > > } > > > } > > > } > > > r = es.search(index='\_all', body=q, size=size, sort='Timestamp:desc') > > > > > > On Monday, February 16, 2015 at 10:57:36 PM UTC-8, Mark Walkom wrote: > > > > > > > It probably depends on what your query looks like, it'd help if you > > > > could gist/pastebin/etc a link to a sample. > > > > > > > > On 17 February 2015 at 11:04, [jam...@opendns.com](mailto:jam...@opendns.com) wrote: > > > > > > > > > Hello, > > > > > > > > > > I have noticed that when I'm applying sorting in my queries, the > > > > > return times are remaining relatively constant at ~1.5s no matter what > > > > > filters I apply (Even if I get 0 hits). It seems like ES might be > > > > > sorting the entire index, then applying the filters after. > > > > > > > > > > As a test I checked how long it took ES to sort the entire index and > > > > > it was about 2s. > > > > > > > > > > Here are some results for different queries comparing how many hits > > > > > they got to how much time they took(ms): > > > > > > > > > > Took: 1563 > > > > > Hits: 7000 > > > > > > > > > > Took: 1660 > > > > > Hits: 239360 > > > > > > > > > > Took: 1657 > > > > > Hits: 0 > > > > > > > > > > As you can see the time the queries are taking remains relatively > > > > > constant when I have sorting on no matter how many hits I get. What > > > > > is happening here? > > > > > > > > > > As a baseline if I remove sorting I get the following: > > > > > > > > > > Took: 150 > > > > > Hits: 7000 > > > > > > > > > > Took: 134 > > > > > Hits: 239360 > > > > > > > > > > Took: 0 > > > > > Hits: 0 > > > > > > > > > > \*All these tests were done with a return size of 500, so I would > > > > > expect queries with more hits to return slightly faster > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > This message contains confidential information and is intended only for > > the individual named. If you are not the named addressee, any delivery, > > disclosure, dissemination or distribution of this e-mail is unlawful and > > strictly prohibited. Please notify the sender immediately by e-mail if > > you have received this e-mail by mistake and delete this e-mail from your > > system. E-mail transmissions cannot be guaranteed to be secure or without > > error, as information could be intercepted, corrupted, lost, destroyed, > > arrive late, be incomplete, or contain viruses. The sender therefore does > > not accept liability for any errors or omissions in the contents of this > > message, which errors or omissions arise as a result of e-mail > > transmission. If verification is required, please request a hard-copy > > version. (Res.Sys-dct1) > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/xPewWFV-LKo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/xPewWFV-LKo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAL\_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAL\_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5kdwONm\_LTb58vYCHFSEaiKun819sUnb5hOo96Fw1WuQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5kdwONm_LTb58vYCHFSEaiKun819sUnb5hOo96Fw1WuQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jamesma](https://avatars.discourse-cdn.com/v4/letter/j/c4cdca/32.png) [@jamesma](https://discuss.elastic.co/u/jamesma)\ **Post date:** [February 17, 2015, 8:30pm UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/8 "2015-02-17T20:30:49Z")
So we notice that our field data only gets cached AFTER two queries are made. For some reason after the first query the field data isn't cached but after we run the query a second time we see the field data get cached on Marvel. How and when does ES store the field data into cache? On Tuesday, February 17, 2015 at 11:44:21 AM UTC-8, Adrien Grand wrote: > James, your analysis is correct. The way sorting works with elasticsearch > is that the query returns an iterator over the documents that are matched > and then elasticsearch consumes this iterator and puts documents into a > priority queue in order to select the top ones. > > On Tue, Feb 17, 2015 at 7:58 PM, James Ma \<[jam...@opendns.com](mailto:jam...@opendns.com) > \\> wrote: > > > Hey so we figured out that the constant slow query speed when sorting is > > because we were clearing the field data before every query which made ES > > have to load the field data back to cache each time. > > > > Note ES doesn't just load the field data for the hits of the specific > > query when sorting, it will load the field data for every doc in the entire > > index, which was causing our performance issue. > > > > On Tue, Feb 17, 2015 at 10:32 AM, Bipul Dutta \<[bipul...@resolvesys.com](mailto:bipul...@resolvesys.com) > > \\> wrote: > > > > > Wondering if adding search\_type=dfs\_query\_then\_fetch will change > > > anything. > > > > > > On Tue, Feb 17, 2015 at 9:59 AM, Jay \<[jam...@opendns.com](mailto:jam...@opendns.com) \\> > > > wrote: > > > > > > > We are working in PyES, our queries currently look like this, but we > > > > have tried putting the sort directly into the body too. It made no > > > > difference > > > > > > > > def termQuery(org, field, term, write, size=500): > > > > q = { > > > > "query": { > > > > "filtered": { > > > > "filter": { > > > > "and": [ > > > > {"term": {"Org ID": org}}, > > > > {"term": {field: term}} > > > > ] > > > > } > > > > } > > > > } > > > > } > > > > r = es.search(index='\_all', body=q, size=size, sort='Timestamp:desc') > > > > > > > > On Monday, February 16, 2015 at 10:57:36 PM UTC-8, Mark Walkom wrote: > > > > > > > > > It probably depends on what your query looks like, it'd help if you > > > > > could gist/pastebin/etc a link to a sample. > > > > > > > > > > On 17 February 2015 at 11:04, [jam...@opendns.com](mailto:jam...@opendns.com) wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I have noticed that when I'm applying sorting in my queries, the > > > > > > return times are remaining relatively constant at ~1.5s no matter what > > > > > > filters I apply (Even if I get 0 hits). It seems like ES might be > > > > > > sorting the entire index, then applying the filters after. > > > > > > > > > > > > As a test I checked how long it took ES to sort the entire index and > > > > > > it was about 2s. > > > > > > > > > > > > Here are some results for different queries comparing how many hits > > > > > > they got to how much time they took(ms): > > > > > > > > > > > > Took: 1563 > > > > > > Hits: 7000 > > > > > > > > > > > > Took: 1660 > > > > > > Hits: 239360 > > > > > > > > > > > > Took: 1657 > > > > > > Hits: 0 > > > > > > > > > > > > As you can see the time the queries are taking remains relatively > > > > > > constant when I have sorting on no matter how many hits I get. What > > > > > > is happening here? > > > > > > > > > > > > As a baseline if I remove sorting I get the following: > > > > > > > > > > > > Took: 150 > > > > > > Hits: 7000 > > > > > > > > > > > > Took: 134 > > > > > > Hits: 239360 > > > > > > > > > > > > Took: 0 > > > > > > Hits: 0 > > > > > > > > > > > > \*All these tests were done with a return size of 500, so I would > > > > > > expect queries with more hits to return slightly faster > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a% > > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e0b97088-6454-4fe2-8707-c0f629aecf0a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f96a81dc-0551-452e-80ef-9e128e08d8e0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > This message contains confidential information and is intended only for > > > the individual named. If you are not the named addressee, any delivery, > > > disclosure, dissemination or distribution of this e-mail is unlawful and > > > strictly prohibited. Please notify the sender immediately by e-mail if > > > you have received this e-mail by mistake and delete this e-mail from your > > > system. E-mail transmissions cannot be guaranteed to be secure or without > > > error, as information could be intercepted, corrupted, lost, destroyed, > > > arrive late, be incomplete, or contain viruses. The sender therefore does > > > not accept liability for any errors or omissions in the contents of this > > > message, which errors or omissions arise as a result of e-mail > > > transmission. If verification is required, please request a hard-copy > > > version. (Res.Sys-dct1) > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/xPewWFV-LKo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/xPewWFV-LKo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAAKpE%2BVmnPqnoe-wuxdDbwzjBL41MxGZ3vwuPeYzpxr-eL2fTw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAL\_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAL\_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL_yQra%2By8JhdVfJMF%3DKzP60tkj93D7SST4WVo3yxZQ-SwXr9A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1ca2b900-aca1-4989-88b0-24e481d33309%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1ca2b900-aca1-4989-88b0-24e481d33309%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:32am UTC](https://discuss.elastic.co/t/elasticsearch-sorting-before-applying-filters/22199/9 "2017-07-06T00:32:06Z")