# Guaranteed upper bound for near real time search **URL:** \ **Category:** Elasticsearch\ **Created:** [January 2, 2015, 8:17am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454 "2015-01-02T08:17:44Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![vishrut\_goyal1](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vishrut\_goyal1](https://discuss.elastic.co/u/vishrut_goyal1)\ **Post date:** [January 2, 2015, 8:17am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/1 "2015-01-02T08:17:44Z")
Hello, Although real time searches are not possible in Elasticsearch, but "near real time" are possible by setting "refresh\_interval" to "1s" (1 second). The problem is that even after setting refresh interval to 1 second, it's not "guaranteed" that an indexed document will be available for search after 1 second. My load tests indicate that sometimes even after 3 seconds of indexing a document, it is not available for search. As per the elasticsearch documentation, "refresh\_interval" controls "how often the refresh operation will be executed". It does not provide an upper bound on the delay between indexing a document and that document being available for search. Is there any other setting in Elasticsearch that can guarantee such bound? Thanks, Vishrut -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 2, 2015, 8:45am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/2 "2015-01-02T08:45:07Z")
Do you compute this 3s delay between when you send the document and the search request? Or between the index response from elasticsearch and the search request? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 2 janv. 2015 à 09:17, [vishrut.goyal@gmail.com](mailto:vishrut.goyal@gmail.com) a écrit : > > Hello, > > Although real time searches are not possible in Elasticsearch, but "near real time" are possible by setting "refresh\_interval" to "1s" (1 second). > The problem is that even after setting refresh interval to 1 second, it's not "guaranteed" that an indexed document will be available for search after 1 second. My load tests indicate that sometimes even after 3 seconds of indexing a document, it is not available for search. As per the elasticsearch documentation, "refresh\_interval" controls "how often the refresh operation will be executed". It does not provide an upper bound on the delay between indexing a document and that document being available for search. > > Is there any other setting in Elasticsearch that can guarantee such bound? > > ## Thanks, Vishrut > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8BF8E59E-2233-4BD4-B89E-C5457185832A%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/8BF8E59E-2233-4BD4-B89E-C5457185832A%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 2, 2015, 8:51am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/3 "2015-01-02T08:51:55Z")
First, it seems you confuse "being available for search" with "near real time". These are two different things: 1. search after indexing is expected to take a long time because of the unpredictable overhead in worst case scenarios (e.g. creating index, creating mapping, creating document, creating segments, creating replica on other nodes, segment merge etc.) 2. near real time: getting a doc ID after indexing is very fast because the document is immediately available in a special Lucene segment kept in RAM (in the millisecond range) You can experiment and reduce refresh interval to 50ms and exercise Elasticsearch (term) query operation. On RAM-only clusters, you will get best results, but that has nothing to do with the (near) real time feature of Elasticsearch get operation. Also note the complexity of distributed systems. As long as there is no information about the workload distribution and no priority index queues are used, no upper time bound (deadlines) can be set in distributed indexing. If you want to find out about a faster real time switch between index write and read, you may have interest in using [http://lucene.apache.org/core/4\_10\_3/core/org/apache/lucene/search/ControlledRealTimeReopenThread.html](http://lucene.apache.org/core/4_10_3/core/org/apache/lucene/search/ControlledRealTimeReopenThread.html) but you have to use your own custom code, because Elasticsearch does not make use of ControlledRealTimeReopenThread. Jörg On Fri, Jan 2, 2015 at 9:17 AM, [vishrut.goyal@gmail.com](mailto:vishrut.goyal@gmail.com) wrote: > Hello, > > Although real time searches are not possible in Elasticsearch, but "near > real time" are possible by setting "refresh\_interval" to "1s" (1 second). > The problem is that even after setting refresh interval to 1 second, it's > not "guaranteed" that an indexed document will be available for search > after 1 second. My load tests indicate that sometimes even after 3 seconds > of indexing a document, it is not available for search. As per the > elasticsearch documentation, "refresh\_interval" controls "how often the > refresh operation will be executed". It does not provide an upper bound on > the delay between indexing a document and that document being available for > search. > > Is there any other setting in Elasticsearch that can guarantee such bound? > > Thanks, > Vishrut > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGj98SJgMt9NHVVKNKp0\_2NdOt\_QSsTHLvdN6ZcrkZ%2BaQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGj98SJgMt9NHVVKNKp0_2NdOt_QSsTHLvdN6ZcrkZ%2BaQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vishrut\_goyal1](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vishrut\_goyal1](https://discuss.elastic.co/u/vishrut_goyal1)\ **Post date:** [January 2, 2015, 9:01am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/4 "2015-01-02T09:01:43Z")
The 3 second delay is between the index response and the search request. I am blocking on indexing operation till I get the response, and then scheduling the search request 3 seconds after I get the response. Thanks, Vishrut On Friday, January 2, 2015 2:15:28 PM UTC+5:30, David Pilato wrote: > Do you compute this 3s delay between when you send the document and the > search request? Or between the index response from elasticsearch and the > search request? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 2 janv. 2015 à 09:17, [vishru...@gmail.com](mailto:vishru...@gmail.com) \ a écrit : > > Hello, > > Although real time searches are not possible in Elasticsearch, but "near > real time" are possible by setting "refresh\_interval" to "1s" (1 second). > The problem is that even after setting refresh interval to 1 second, it's > not "guaranteed" that an indexed document will be available for search > after 1 second. My load tests indicate that sometimes even after 3 seconds > of indexing a document, it is not available for search. As per the > elasticsearch documentation, "refresh\_interval" controls "how often the > refresh operation will be executed". It does not provide an upper bound on > the delay between indexing a document and that document being available for > search. > > Is there any other setting in Elasticsearch that can guarantee such bound? > > Thanks, > Vishrut > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5d8c2142-46f4-441e-b4f9-2af22f40721d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d8c2142-46f4-441e-b4f9-2af22f40721d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vishrut\_goyal1](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vishrut\_goyal1](https://discuss.elastic.co/u/vishrut_goyal1)\ **Post date:** [January 2, 2015, 9:09am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/5 "2015-01-02T09:09:36Z")
As per Elasticsearch documentation, "Get" operation is completely real time (unless explicitly disabled). We can immediately get the document using the doc ID immediately after indexing the document. I am talking of Search operation here, which can be made "near realtime" by controlling the value of "refresh\_interval". Thanks, Vishrut On Friday, January 2, 2015 2:22:03 PM UTC+5:30, Jörg Prante wrote: > First, it seems you confuse "being available for search" with "near real > time". These are two different things: > > 1. search after indexing is expected to take a long time because of the > unpredictable overhead in worst case scenarios (e.g. creating index, > creating mapping, creating document, creating segments, creating replica on > other nodes, segment merge etc.) > > 2. near real time: getting a doc ID after indexing is very fast because > the document is immediately available in a special Lucene segment kept in > RAM (in the millisecond range) > > You can experiment and reduce refresh interval to 50ms and exercise > Elasticsearch (term) query operation. On RAM-only clusters, you will get > best results, but that has nothing to do with the (near) real time feature > of Elasticsearch get operation. > > Also note the complexity of distributed systems. As long as there is no > information about the workload distribution and no priority index queues > are used, no upper time bound (deadlines) can be set in distributed > indexing. > > If you want to find out about a faster real time switch between index > write and read, you may have interest in using > > [ControlledRealTimeReopenThread (Lucene 4.10.3 API)](http://lucene.apache.org/core/4_10_3/core/org/apache/lucene/search/ControlledRealTimeReopenThread.html) > > but you have to use your own custom code, because Elasticsearch does not > make use of ControlledRealTimeReopenThread. > > Jörg > > On Fri, Jan 2, 2015 at 9:17 AM, \<[vishru...@gmail.com](mailto:vishru...@gmail.com) \\> wrote: > > > Hello, > > > > Although real time searches are not possible in Elasticsearch, but "near > > real time" are possible by setting "refresh\_interval" to "1s" (1 second). > > The problem is that even after setting refresh interval to 1 second, it's > > not "guaranteed" that an indexed document will be available for search > > after 1 second. My load tests indicate that sometimes even after 3 seconds > > of indexing a document, it is not available for search. As per the > > elasticsearch documentation, "refresh\_interval" controls "how often the > > refresh operation will be executed". It does not provide an upper bound on > > the delay between indexing a document and that document being available for > > search. > > > > Is there any other setting in Elasticsearch that can guarantee such bound? > > > > Thanks, > > Vishrut > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/550c4629-1a82-4940-9acb-f00f094f22ab%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/550c4629-1a82-4940-9acb-f00f094f22ab%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [January 2, 2015, 9:55am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/6 "2015-01-02T09:55:38Z")
The 1s refresh\_interval means that ES will open (takes some time) and warm (takes some more time) a new NRT reader, and after that reader is done opening, 1s later it will open again. So it's possible in your case it takes 2s to open + warm a new NRT reader (check the node's logs). But 2s is quite a long time for the reopen unless the index has changed a lot (which is unlikely with 1s refresh\_interval). Mike McCandless [http://blog.mikemccandless.com](http://blog.mikemccandless.com) On Fri, Jan 2, 2015 at 4:09 AM, [vishrut.goyal@gmail.com](mailto:vishrut.goyal@gmail.com) wrote: > As per Elasticsearch documentation, "Get" operation is completely real > time (unless explicitly disabled). We can immediately get the document > using the doc ID immediately after indexing the document. > I am talking of Search operation here, which can be made "near realtime" > by controlling the value of "refresh\_interval". > > Thanks, > Vishrut > > On Friday, January 2, 2015 2:22:03 PM UTC+5:30, Jörg Prante wrote: > > > First, it seems you confuse "being available for search" with "near real > > time". These are two different things: > > > > 1. search after indexing is expected to take a long time because of the > > unpredictable overhead in worst case scenarios (e.g. creating index, > > creating mapping, creating document, creating segments, creating replica on > > other nodes, segment merge etc.) > > > > 2. near real time: getting a doc ID after indexing is very fast because > > the document is immediately available in a special Lucene segment kept in > > RAM (in the millisecond range) > > > > You can experiment and reduce refresh interval to 50ms and exercise > > Elasticsearch (term) query operation. On RAM-only clusters, you will get > > best results, but that has nothing to do with the (near) real time feature > > of Elasticsearch get operation. > > > > Also note the complexity of distributed systems. As long as there is no > > information about the workload distribution and no priority index queues > > are used, no upper time bound (deadlines) can be set in distributed > > indexing. > > > > If you want to find out about a faster real time switch between index > > write and read, you may have interest in using > > > > [Index of /\_\_root/docs.lucene.apache.org/core/4\_10\_3/core/org/apache/lucene/search](http://lucene.apache.org/core/4_10_3/core/org/apache/lucene/search/) > > ControlledRealTimeReopenThread.html > > > > but you have to use your own custom code, because Elasticsearch does not > > make use of ControlledRealTimeReopenThread. > > > > Jörg > > > > On Fri, Jan 2, 2015 at 9:17 AM, [vishru...@gmail.com](mailto:vishru...@gmail.com) wrote: > > > > > Hello, > > > > > > Although real time searches are not possible in Elasticsearch, but "near > > > real time" are possible by setting "refresh\_interval" to "1s" (1 second). > > > The problem is that even after setting refresh interval to 1 second, > > > it's not "guaranteed" that an indexed document will be available for search > > > after 1 second. My load tests indicate that sometimes even after 3 seconds > > > of indexing a document, it is not available for search. As per the > > > elasticsearch documentation, "refresh\_interval" controls "how often the > > > refresh operation will be executed". It does not provide an upper bound on > > > the delay between indexing a document and that document being available for > > > search. > > > > > > Is there any other setting in Elasticsearch that can guarantee such > > > bound? > > > > > > Thanks, > > > Vishrut > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/347d1f45-c243-4c87-b4ae-e02eee039b13%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/550c4629-1a82-4940-9acb-f00f094f22ab%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/550c4629-1a82-4940-9acb-f00f094f22ab%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/550c4629-1a82-4940-9acb-f00f094f22ab%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/550c4629-1a82-4940-9acb-f00f094f22ab%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smReu8CeymcXEDpY80QLdKat2Rg-k6%3DtKAOtvFwasEYye2Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smReu8CeymcXEDpY80QLdKat2Rg-k6%3DtKAOtvFwasEYye2Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Diego\_de\_Freitas](https://avatars.discourse-cdn.com/v4/letter/d/8edcca/32.png) [@Diego\_de\_Freitas](https://discuss.elastic.co/u/Diego_de_Freitas)\ **Post date:** [November 21, 2016, 10:40am UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/7 "2016-11-21T10:40:12Z")
Do you know if it is possible to monitor this overhead over the second ? Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:05pm UTC](https://discuss.elastic.co/t/guaranteed-upper-bound-for-near-real-time-search/21454/8 "2017-07-05T22:05:33Z")