# What is causing random query time fluctuations **URL:** \ **Category:** Elasticsearch\ **Created:** [April 11, 2014, 1:24pm UTC](https://discuss.elastic.co/t/what-is-causing-random-query-time-fluctuations/16956 "2014-04-11T13:24:43Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 11, 2014, 1:24pm UTC](https://discuss.elastic.co/t/what-is-causing-random-query-time-fluctuations/16956/1 "2014-04-11T13:24:43Z")
When I repeat the same search over and over again I get somewhat wild fluctuations in run time: 4ms, 4ms, 87ms, 17ms, 157ms, 4ms, 4ms, 4ms Any ideas what might cause this? This only seems to happen on my production cluster so I'm not likely to be able to make full reproduction steps. I causing it with a simple match query without any filters or anything. Normally I trace performance stuff by causing lots of traffic and looking at hot\_threads but that these fluctuations don't feel like they'd be easy to catch that way. I have 16 nodes on real hardware with somewhat slow disks and 30GB heaps with 96GB total ram. We write to the indexes constantly but slowly. Maybe 2-3 updates (not new docs, changes to old docs) per second per shard. Any ideas would be great, Thanks Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [April 11, 2014, 1:47pm UTC](https://discuss.elastic.co/t/what-is-causing-random-query-time-fluctuations/16956/2 "2014-04-11T13:47:50Z")
What is this query like? if it does sorting or faceting for example this could be the field data being invalidated and loaded again, and this will point at low levels of free RAM. Phrase queries require loading term position data that is also being cached but can get invalidated. Etc etc Also, this new feature may be worth having a look at [https://github.com/elasticsearch/elasticsearch/blob/master/docs/reference/indices/benchmark.asciidoc](https://github.com/elasticsearch/elasticsearch/blob/master/docs/reference/indices/benchmark.asciidoc) -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Fri, Apr 11, 2014 at 2:24 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > When I repeat the same search over and over again I get somewhat wild > fluctuations in run time: 4ms, 4ms, 87ms, 17ms, 157ms, 4ms, 4ms, 4ms > > Any ideas what might cause this? This only seems to happen on my > production cluster so I'm not likely to be able to make full reproduction > steps. I causing it with a simple match query without any filters or > anything. > > Normally I trace performance stuff by causing lots of traffic and looking > at hot\_threads but that these fluctuations don't feel like they'd be easy > to catch that way. > > I have 16 nodes on real hardware with somewhat slow disks and 30GB heaps > with 96GB total ram. We write to the indexes constantly but slowly. Maybe > 2-3 updates (not new docs, changes to old docs) per second per shard. > > Any ideas would be great, > > Thanks > > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zvw1Nnuvh\_B%3DkGm\_4K7Rs97CgSOqzB3DFX1j1gz%2Bajg%3Dw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zvw1Nnuvh_B%3DkGm_4K7Rs97CgSOqzB3DFX1j1gz%2Bajg%3Dw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 11, 2014, 2:02pm UTC](https://discuss.elastic.co/t/what-is-causing-random-query-time-fluctuations/16956/3 "2014-04-11T14:02:07Z")
Could this be caused by Lucene merges at some point? I guess also that when a segment is commited, it could take some time to compute cache for new segments? Are you using warmers? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 11 avr. 2014 à 15:24, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) a écrit : > > When I repeat the same search over and over again I get somewhat wild fluctuations in run time: 4ms, 4ms, 87ms, 17ms, 157ms, 4ms, 4ms, 4ms > > Any ideas what might cause this? This only seems to happen on my production cluster so I'm not likely to be able to make full reproduction steps. I causing it with a simple match query without any filters or anything. > > Normally I trace performance stuff by causing lots of traffic and looking at hot\_threads but that these fluctuations don't feel like they'd be easy to catch that way. > > I have 16 nodes on real hardware with somewhat slow disks and 30GB heaps with 96GB total ram. We write to the indexes constantly but slowly. Maybe 2-3 updates (not new docs, changes to old docs) per second per shard. > > Any ideas would be great, > > Thanks > > ## Nik > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 11, 2014, 2:31pm UTC](https://discuss.elastic.co/t/what-is-causing-random-query-time-fluctuations/16956/4 "2014-04-11T14:31:39Z")
Itamar: Its really just a match query. No sorting or anything: POST /enwiki\_content/\_search { "\_source": false, "query": { "multi\_match": { "query": "main page", "fields": [ "title.plain" ] } } } I mean, I do more complicated things most of the time, but I'm still able to reproduce the fluctuations with the match query. David: I never turned on warmers or any eager loading. I'll give that a shot and report back. Nik On Fri, Apr 11, 2014 at 10:02 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > Could this be caused by Lucene merges at some point? I guess also that > when a segment is commited, it could take some time to compute cache for > new segments? > > Are you using warmers? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 11 avr. 2014 à 15:24, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) a écrit : > > When I repeat the same search over and over again I get somewhat wild > fluctuations in run time: 4ms, 4ms, 87ms, 17ms, 157ms, 4ms, 4ms, 4ms > > Any ideas what might cause this? This only seems to happen on my > production cluster so I'm not likely to be able to make full reproduction > steps. I causing it with a simple match query without any filters or > anything. > > Normally I trace performance stuff by causing lots of traffic and looking > at hot\_threads but that these fluctuations don't feel like they'd be easy > to catch that way. > > I have 16 nodes on real hardware with somewhat slow disks and 30GB heaps > with 96GB total ram. We write to the indexes constantly but slowly. Maybe > 2-3 updates (not new docs, changes to old docs) per second per shard. > > Any ideas would be great, > > Thanks > > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr)[https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd15A1L2y39Tm%2Bidp9DHv-g8d-k74Qk\_maVGsj--6o9kjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd15A1L2y39Tm%2Bidp9DHv-g8d-k74Qk_maVGsj--6o9kjQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 11, 2014, 3:07pm UTC](https://discuss.elastic.co/t/what-is-causing-random-query-time-fluctuations/16956/5 "2014-04-11T15:07:08Z")
Even simpler: POST /enwiki\_content/\_search { "\_source": false, "query": { "match": { "title.plain": "main page" } } } I tried adding that query as a warmer and I still get fluctuations. I tried raising the refresh interval to 30s and I still get it too. I'm reading the times from the "took" field. I think this is ok because it should eliminate stuff like network latency from the measure. Do you think eager norms loading would help? I wonder if I should look at eager norms loading.... I wonder if this is just the kind of thing that'll average out over all the queries or will it get magnified? As in could these bunch up and turn into some kind of stampede? On closer examination, ganglia is telling me that young GCs average about 45ms on the production cluster and occur about every two seconds. That might fit. Also, I see the same delays when I manually hammer smaller indexes, just less frequently. Nik On Fri, Apr 11, 2014 at 10:31 AM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > Itamar: > Its really just a match query. No sorting or anything: > POST /enwiki\_content/\_search > { > "\_source": false, > "query": { > "multi\_match": { > "query": "main page", > "fields": [ > "title.plain" > ] > } > } > } > > I mean, I do more complicated things most of the time, but I'm still able > to reproduce the fluctuations with the match query. > David: > I never turned on warmers or any eager loading. I'll give that a shot and > report back. > > Nik > > On Fri, Apr 11, 2014 at 10:02 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > Could this be caused by Lucene merges at some point? I guess also that > > when a segment is commited, it could take some time to compute cache for > > new segments? > > > > Are you using warmers? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 11 avr. 2014 à 15:24, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) a écrit : > > > > When I repeat the same search over and over again I get somewhat wild > > fluctuations in run time: 4ms, 4ms, 87ms, 17ms, 157ms, 4ms, 4ms, 4ms > > > > Any ideas what might cause this? This only seems to happen on my > > production cluster so I'm not likely to be able to make full reproduction > > steps. I causing it with a simple match query without any filters or > > anything. > > > > Normally I trace performance stuff by causing lots of traffic and looking > > at hot\_threads but that these fluctuations don't feel like they'd be easy > > to catch that way. > > > > I have 16 nodes on real hardware with somewhat slow disks and 30GB heaps > > with 96GB total ram. We write to the indexes constantly but slowly. Maybe > > 2-3 updates (not new docs, changes to old docs) per second per shard. > > > > Any ideas would be great, > > > > Thanks > > > > Nik > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z\_qRxAOsi2yrc\_5RGBmQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2hR-CDOsOy-o1Opqc8V6GWJ5Z_qRxAOsi2yrc_5RGBmQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr)[https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/D9F22255-CBFE-4DE5-BBF2-BEF7F76624FC%40pilato.fr?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd16tqDRcDQQykm0UAAWeygUbuCTYi%2B%3D6UocSkbf7k7nOw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd16tqDRcDQQykm0UAAWeygUbuCTYi%2B%3D6UocSkbf7k7nOw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:36am UTC](https://discuss.elastic.co/t/what-is-causing-random-query-time-fluctuations/16956/6 "2017-07-06T01:36:28Z")