# Each first different query is slow **URL:** \ **Category:** Elasticsearch\ **Created:** [October 21, 2013, 11:10am UTC](https://discuss.elastic.co/t/each-first-different-query-is-slow/14035 "2013-10-21T11:10:59Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Lucian\_Precup](https://avatars.discourse-cdn.com/v4/letter/l/50afbb/32.png) [@Lucian\_Precup](https://discuss.elastic.co/u/Lucian_Precup)\ **Post date:** [October 21, 2013, 11:10am UTC](https://discuss.elastic.co/t/each-first-different-query-is-slow/14035/1 "2013-10-21T11:10:59Z")
Hello Elasticsearch and Elasticsearch users, I encounter a strange issue with query performance. I am able to reproduce this behavior with simple match queries, without using filters, facets or sorts. In summary, the first query is always slow (like seconds). Then, launching the very same query brings the expected performance (like under 300 milliseconds). Adding an additional parameter to the query (parameter that narrow the number of results) leads to the same behavior: first query slow, next query ok. Here are some details: I am using elasticsearch 0.90.1. The index contains about 8 million documents and it has 3 gb. One shard, zero replicas. For the queries reproducing the issue, I am using two fields: firstname and lastname. I first launch a match\_all query with a terms\_facet on lastname in order to obtain the most commonly used names: { "from": 0, "size": 0, "query": { "match\_all": {} }, "facets": { "lastname": { "terms": { "field": "lastname", "size": 1000 } } } } The I launch a match query with each of the 1000 names using JMeter: { "from": 0, "size": 200, "query": { "bool": { "must": [ { "multi\_match": { "query": "${LASTNAME}", "fields": [ "lastname" ] } } ] } } } For the first launch of each of the 1000 queries I have rather slow times (the max is at about 11 seconds and every query takes at least a couple of hundreds of milliseconds). Queries with names in the beginning of the list are generally slower (because they bring more results). For the second launch the maximum "took time" drops to 500 ms (which is O.K. for our case) and every query is faster than it was on its first launch. For the third launch, I add a first name to the query. I just chose a common name at random (say Jean - John) { "from": 0, "size": 200, "query": { "bool": { "must": [ { "multi\_match": { "query": "${LASTNAME}", "fields": [ "lastname" ] } }, { "multi\_match": { "query": "jean", "fields": [ "firstname" ] } } ] } } } Queries are again slow. Not as slow as the first launch but I feel like the acceleration is just due to less results (because of the additional parameter). Relaunching the queries brings correct performance again (fourth launch). To add one last thing. Relaunching the fourth launch after some server inactivity (like over the weekend) brings up slow performance again (500ms to 8 seconds). Can this be related to some caching done by the file system ? Have you encountered something similar ? Thank you for your hints. Best regards, Lucian Precup -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 21, 2013, 11:51am UTC](https://discuss.elastic.co/t/each-first-different-query-is-slow/14035/2 "2013-10-21T11:51:20Z")
Hi Lucian, It does sound like you are looking at file cache issues. After the first query, the terms list and posting lists for the top 1000 lastnames are loaded into the file cache of the OS. The subsequent queries are then run in memory (by the OS , as it doesn't need to go to file) and are therefore faster. When you add the first name, you already have the lastname posting list in memory but the first name ones need to be read from disk. After the weekend the file cache has probably been cleared out and things need to be read from disk again. Do you leave enough memory for the file system cache of the OS? How much memory does the machine have? how much is allocated to ES? Cheers, Boaz On Monday, October 21, 2013 1:10:59 PM UTC+2, Lucian Precup wrote: > Hello Elasticsearch and Elasticsearch users, > > I encounter a strange issue with query performance. I am able to reproduce > this behavior with simple match queries, without using filters, facets or > sorts. In summary, the first query is always slow (like seconds). Then, > launching the very same query brings the expected performance (like under > 300 milliseconds). Adding an additional parameter to the query (parameter > that narrow the number of results) leads to the same behavior: first query > slow, next query ok. > > Here are some details: > I am using elasticsearch 0.90.1. > The index contains about 8 million documents and it has 3 gb. One shard, > zero replicas. For the queries reproducing the issue, I am using two > fields: firstname and lastname. > > I first launch a match\_all query with a terms\_facet on lastname in order > to obtain the most commonly used names: > { > "from": 0, > "size": 0, > "query": { > "match\_all": {} > }, > "facets": { > "lastname": { > "terms": { > "field": "lastname", > "size": 1000 > } > } > } > } > > The I launch a match query with each of the 1000 names using JMeter: > { > "from": 0, > "size": 200, > "query": { > "bool": { > "must": [ > { > "multi\_match": { > "query": "${LASTNAME}", > "fields": [ > "lastname" > ] > } > } > ] > } > } > } > > For the first launch of each of the 1000 queries I have rather slow times > (the max is at about 11 seconds and every query takes at least a couple of > hundreds of milliseconds). Queries with names in the beginning of the list > are generally slower (because they bring more results). > For the second launch the maximum "took time" drops to 500 ms (which is > O.K. for our case) and every query is faster than it was on its first > launch. > > For the third launch, I add a first name to the query. I just chose a > common name at random (say Jean - John) > { > "from": 0, > "size": 200, > "query": { > "bool": { > "must": [ > { > "multi\_match": { > "query": "${LASTNAME}", > "fields": [ > "lastname" > ] > } > }, > { > "multi\_match": { > "query": "jean", > "fields": [ > "firstname" > ] > } > } > ] > } > } > } > Queries are again slow. Not as slow as the first launch but I feel like > the acceleration is just due to less results (because of the additional > parameter). > Relaunching the queries brings correct performance again (fourth launch). > > To add one last thing. Relaunching the fourth launch after some server > inactivity (like over the weekend) brings up slow performance again (500ms > to 8 seconds). > > Can this be related to some caching done by the file system ? Have you > encountered something similar ? > > Thank you for your hints. > > Best regards, > Lucian Precup -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [October 21, 2013, 2:18pm UTC](https://discuss.elastic.co/t/each-first-different-query-is-slow/14035/3 "2013-10-21T14:18:54Z")
Hey, it seems you have some processes running that invalidate the filesystem cache. A very possible candidate for this is the updatedb command, which is run daily by cron to create a small database for the locate command - but there might be others as well. --Alex On Mon, Oct 21, 2013 at 1:51 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > Hi Lucian, > > It does sound like you are looking at file cache issues. After the first > query, the terms list and posting lists for the top 1000 lastnames are > loaded into the file cache of the OS. The subsequent queries are then run > in memory (by the OS , as it doesn't need to go to file) and are therefore > faster. When you add the first name, you already have the lastname posting > list in memory but the first name ones need to be read from disk. > > After the weekend the file cache has probably been cleared out and things > need to be read from disk again. > > Do you leave enough memory for the file system cache of the OS? How much > memory does the machine have? how much is allocated to ES? > > Cheers, > Boaz > > On Monday, October 21, 2013 1:10:59 PM UTC+2, Lucian Precup wrote: > > > Hello Elasticsearch and Elasticsearch users, > > > > I encounter a strange issue with query performance. I am able to > > reproduce this behavior with simple match queries, without using filters, > > facets or sorts. In summary, the first query is always slow (like seconds). > > Then, launching the very same query brings the expected performance (like > > under 300 milliseconds). Adding an additional parameter to the query > > (parameter that narrow the number of results) leads to the same behavior: > > first query slow, next query ok. > > > > Here are some details: > > I am using elasticsearch 0.90.1. > > The index contains about 8 million documents and it has 3 gb. One shard, > > zero replicas. For the queries reproducing the issue, I am using two > > fields: firstname and lastname. > > > > I first launch a match\_all query with a terms\_facet on lastname in order > > to obtain the most commonly used names: > > { > > "from": 0, > > "size": 0, > > "query": { > > "match\_all": {} > > }, > > "facets": { > > "lastname": { > > "terms": { > > "field": "lastname", > > "size": 1000 > > } > > } > > } > > } > > > > The I launch a match query with each of the 1000 names using JMeter: > > { > > "from": 0, > > "size": 200, > > "query": { > > "bool": { > > "must": [ > > { > > "multi\_match": { > > "query": "${LASTNAME}", > > "fields": [ > > "lastname" > > ] > > } > > } > > ] > > } > > } > > } > > > > For the first launch of each of the 1000 queries I have rather slow times > > (the max is at about 11 seconds and every query takes at least a couple of > > hundreds of milliseconds). Queries with names in the beginning of the list > > are generally slower (because they bring more results). > > For the second launch the maximum "took time" drops to 500 ms (which is > > O.K. for our case) and every query is faster than it was on its first > > launch. > > > > For the third launch, I add a first name to the query. I just chose a > > common name at random (say Jean - John) > > { > > "from": 0, > > "size": 200, > > "query": { > > "bool": { > > "must": [ > > { > > "multi\_match": { > > "query": "${LASTNAME}", > > "fields": [ > > "lastname" > > ] > > } > > }, > > { > > "multi\_match": { > > "query": "jean", > > "fields": [ > > "firstname" > > ] > > } > > } > > ] > > } > > } > > } > > Queries are again slow. Not as slow as the first launch but I feel like > > the acceleration is just due to less results (because of the additional > > parameter). > > Relaunching the queries brings correct performance again (fourth launch). > > > > To add one last thing. Relaunching the fourth launch after some server > > inactivity (like over the weekend) brings up slow performance again (500ms > > to 8 seconds). > > > > Can this be related to some caching done by the file system ? Have you > > encountered something similar ? > > > > Thank you for your hints. > > > > Best regards, > > Lucian Precup > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Lucian\_Precup](https://avatars.discourse-cdn.com/v4/letter/l/50afbb/32.png) [@Lucian\_Precup](https://discuss.elastic.co/u/Lucian_Precup)\ **Post date:** [October 22, 2013, 1:35pm UTC](https://discuss.elastic.co/t/each-first-different-query-is-slow/14035/4 "2013-10-22T13:35:49Z")
Hello Boaz, Hello Alexander, Thank you for your replies. Your hints make sense considering our environments. We will launch some more tests to validate these hypothesis. Thanks 🙂 Best regards, Lucian On Monday, October 21, 2013 4:18:54 PM UTC+2, Alexander Reelsen wrote: > Hey, > > it seems you have some processes running that invalidate the filesystem > cache. A very possible candidate for this is the updatedb command, which is > run daily by cron to create a small database for the locate command - but > there might be others as well. > > --Alex > > On Mon, Oct 21, 2013 at 1:51 PM, Boaz Leskes \<[b.le...@gmail.com](mailto:b.le...@gmail.com)\ > > > wrote: > > > Hi Lucian, > > > > It does sound like you are looking at file cache issues. After the first > > query, the terms list and posting lists for the top 1000 lastnames are > > loaded into the file cache of the OS. The subsequent queries are then run > > in memory (by the OS , as it doesn't need to go to file) and are therefore > > faster. When you add the first name, you already have the lastname posting > > list in memory but the first name ones need to be read from disk. > > > > After the weekend the file cache has probably been cleared out and things > > need to be read from disk again. > > > > Do you leave enough memory for the file system cache of the OS? How much > > memory does the machine have? how much is allocated to ES? > > > > Cheers, > > Boaz > > > > On Monday, October 21, 2013 1:10:59 PM UTC+2, Lucian Precup wrote: > > > > > Hello Elasticsearch and Elasticsearch users, > > > > > > I encounter a strange issue with query performance. I am able to > > > reproduce this behavior with simple match queries, without using filters, > > > facets or sorts. In summary, the first query is always slow (like seconds). > > > Then, launching the very same query brings the expected performance (like > > > under 300 milliseconds). Adding an additional parameter to the query > > > (parameter that narrow the number of results) leads to the same behavior: > > > first query slow, next query ok. > > > > > > Here are some details: > > > I am using elasticsearch 0.90.1. > > > The index contains about 8 million documents and it has 3 gb. One shard, > > > zero replicas. For the queries reproducing the issue, I am using two > > > fields: firstname and lastname. > > > > > > I first launch a match\_all query with a terms\_facet on lastname in order > > > to obtain the most commonly used names: > > > { > > > "from": 0, > > > "size": 0, > > > "query": { > > > "match\_all": {} > > > }, > > > "facets": { > > > "lastname": { > > > "terms": { > > > "field": "lastname", > > > "size": 1000 > > > } > > > } > > > } > > > } > > > > > > The I launch a match query with each of the 1000 names using JMeter: > > > { > > > "from": 0, > > > "size": 200, > > > "query": { > > > "bool": { > > > "must": [ > > > { > > > "multi\_match": { > > > "query": "${LASTNAME}", > > > "fields": [ > > > "lastname" > > > ] > > > } > > > } > > > ] > > > } > > > } > > > } > > > > > > For the first launch of each of the 1000 queries I have rather slow > > > times (the max is at about 11 seconds and every query takes at least a > > > couple of hundreds of milliseconds). Queries with names in the beginning of > > > the list are generally slower (because they bring more results). > > > For the second launch the maximum "took time" drops to 500 ms (which is > > > O.K. for our case) and every query is faster than it was on its first > > > launch. > > > > > > For the third launch, I add a first name to the query. I just chose a > > > common name at random (say Jean - John) > > > { > > > "from": 0, > > > "size": 200, > > > "query": { > > > "bool": { > > > "must": [ > > > { > > > "multi\_match": { > > > "query": "${LASTNAME}", > > > "fields": [ > > > "lastname" > > > ] > > > } > > > }, > > > { > > > "multi\_match": { > > > "query": "jean", > > > "fields": [ > > > "firstname" > > > ] > > > } > > > } > > > ] > > > } > > > } > > > } > > > Queries are again slow. Not as slow as the first launch but I feel like > > > the acceleration is just due to less results (because of the additional > > > parameter). > > > Relaunching the queries brings correct performance again (fourth launch). > > > > > > To add one last thing. Relaunching the fourth launch after some server > > > inactivity (like over the weekend) brings up slow performance again (500ms > > > to 8 seconds). > > > > > > Can this be related to some caching done by the file system ? Have you > > > encountered something similar ? > > > > > > Thank you for your hints. > > > > > > Best regards, > > > Lucian Precup > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:11am UTC](https://discuss.elastic.co/t/each-first-different-query-is-slow/14035/5 "2017-07-06T02:11:10Z")