# Potential for realtime search? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 20, 2013, 3:59pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220 "2013-03-20T15:59:23Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![MarcusLongmuir](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@MarcusLongmuir](https://discuss.elastic.co/u/MarcusLongmuir)\ **Post date:** [March 20, 2013, 3:59pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/1 "2013-03-20T15:59:23Z")
I understand that ES is "near" realtime (updating every second by default). I think ES is awesome and I'm using it to great effect, but this minor issue keeps popping up and workarounds that make sure the user reads their own writes immediately aren't nice. Truly realtime search is obviously desirable and I assume having "near" realtime isn't the end goal for the project. Are there any plans as to how to achieve it? How about a mini-shard alongside each normal shard that holds the documents that are not yet inserted into the index? It could expose the same functionality as a normal shard, but it could be updated with each insertion, effectively acting as a sort of cache. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 20, 2013, 4:06pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/2 "2013-03-20T16:06:39Z")
If you need to read, then you should know that read is realtime. Search is not. ## Are you looking for realtime get ? David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 20 mars 2013 à 16:59, marcuslongmuir [marcuslongmuir@me.com](mailto:marcuslongmuir@me.com) a écrit : > I understand that ES is "near" realtime (updating every second by default). I think ES is awesome and I'm using it to great effect, but this minor issue keeps popping up and workarounds that make sure the user reads their own writes immediately aren't nice. > > Truly realtime search is obviously desirable and I assume having "near" realtime isn't the end goal for the project. Are there any plans as to how to achieve it? > > ## How about a mini-shard alongside each normal shard that holds the documents that are not yet inserted into the index? It could expose the same functionality as a normal shard, but it could be updated with each insertion, effectively acting as a sort of cache. > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![MarcusLongmuir](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@MarcusLongmuir](https://discuss.elastic.co/u/MarcusLongmuir)\ **Post date:** [March 20, 2013, 4:10pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/3 "2013-03-20T16:10:51Z")
I'm using ES in combination with Couchbase so I'm only using ES for searching and then Couchbase for everything else. On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > If you need to read, then you should know that read is realtime. > Search is not. > > ## Are you looking for realtime get ? > > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 20 mars 2013 à 16:59, marcuslongmuir \<[marcusl...@me.com](mailto:marcusl...@me.com) \\> > a écrit : > > I understand that ES is "near" realtime (updating every second by > default). I think ES is awesome and I'm using it to great effect, but this > minor issue keeps popping up and workarounds that make sure the user reads > their own writes immediately aren't nice. > > Truly realtime search is obviously desirable and I assume having "near" > realtime isn't the end goal for the project. Are there any plans as to how > to achieve it? > > How about a mini-shard alongside each normal shard that holds the > documents that are not yet inserted into the index? It could expose the > same functionality as a normal shard, but it could be updated with each > insertion, effectively acting as a sort of cache. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 20, 2013, 4:30pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/4 "2013-03-20T16:30:24Z")
But you talked about "the user reads their own writes". Just wondering if realtime search is really what you need. That said, I understand your concern here, because you have Couchbase in the middle and you have a first latency with Couchbase transport and another one with the ES refresh. Can you describe a bit more your use case and the reason you feel that you need realtime search? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 20 mars 2013 à 17:10, marcuslongmuir [marcuslongmuir@me.com](mailto:marcuslongmuir@me.com) a écrit : I'm using ES in combination with Couchbase so I'm only using ES for searching and then Couchbase for everything else. On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > If you need to read, then you should know that read is realtime. > Search is not. > > ## Are you looking for realtime get ? > > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > I understand that ES is "near" realtime (updating every second by default). I think ES is awesome and I'm using it to great effect, but this minor issue keeps popping up and workarounds that make sure the user reads their own writes immediately aren't nice. > > > > Truly realtime search is obviously desirable and I assume having "near" realtime isn't the end goal for the project. Are there any plans as to how to achieve it? > > > > ## How about a mini-shard alongside each normal shard that holds the documents that are not yet inserted into the index? It could expose the same functionality as a normal shard, but it could be updated with each insertion, effectively acting as a sort of cache. > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![MarcusLongmuir](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@MarcusLongmuir](https://discuss.elastic.co/u/MarcusLongmuir)\ **Post date:** [March 20, 2013, 4:39pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/5 "2013-03-20T16:39:44Z")
Sorry for the confusion. The most common use case is usually similar to a comment being posted on an article or post (generic discussion board). Assuming we're not using AJAX and the comment is posted via a form, the page is loaded, the comment is inserted and then the latest comments are retrieved for display via a search. Our user's latest comment is omitted because even though it was inserted before the search was performed it had yet to be inserted into the index. The workaround is simple - just add the comment to bottom as long as it wasn't included in the search results, but in more complex manifestations of this problem it isn't that easy. Its a rather generic issue. On Wednesday, March 20, 2013 4:30:24 PM UTC, David Pilato wrote: > But you talked about "the user reads their own writes". > Just wondering if realtime search is really what you need. > > That said, I understand your concern here, because you have Couchbase in > the middle and you have a first latency with Couchbase transport and > another one with the ES refresh. > > Can you describe a bit more your use case and the reason you feel that you > need realtime search? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 20 mars 2013 à 17:10, marcuslongmuir \<[marcusl...@me.com](mailto:marcusl...@me.com) \\> > a écrit : > > I'm using ES in combination with Couchbase so I'm only using ES for > searching and then Couchbase for everything else. > > On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > > > If you need to read, then you should know that read is realtime. > > Search is not. > > > > ## Are you looking for realtime get ? > > > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > I understand that ES is "near" realtime (updating every second by > > default). I think ES is awesome and I'm using it to great effect, but this > > minor issue keeps popping up and workarounds that make sure the user reads > > their own writes immediately aren't nice. > > > > Truly realtime search is obviously desirable and I assume having "near" > > realtime isn't the end goal for the project. Are there any plans as to how > > to achieve it? > > > > How about a mini-shard alongside each normal shard that holds the > > documents that are not yet inserted into the index? It could expose the > > same functionality as a normal shard, but it could be updated with each > > insertion, effectively acting as a sort of cache. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [March 20, 2013, 9:53pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/6 "2013-03-20T21:53:39Z")
Can you just add a refresh=true to your add comment call or lower the refresh interval for the comment index? That is assuming your add comment rate is lower then your search rate. [http://www.elasticsearch.org/guide/reference/api/index\_.html](http://www.elasticsearch.org/guide/reference/api/index_.html) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Karussell\_2](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@Karussell\_2](https://discuss.elastic.co/u/Karussell_2)\ **Post date:** [March 21, 2013, 9:58am UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/7 "2013-03-21T09:58:49Z")
on the lucene layer one could handle it via: > **[SearcherLifetimeManager prevents a broken search user experience](https://blog.mikemccandless.com/2011/11/searcherlifetimemanager-prevents-broken.html)** > > In the past, search indices were usually very static: you built them once, called optimize at the end and shipped them off, and didn't ch... you could ask the devs to include this somehow to elasticsearch, or probably there is already an open issue although I did find one. Peter. On Wednesday, March 20, 2013 5:39:44 PM UTC+1, marcuslongmuir wrote: > Sorry for the confusion. The most common use case is usually similar to a > comment being posted on an article or post (generic discussion board). > > Assuming we're not using AJAX and the comment is posted via a form, the > page is loaded, the comment is inserted and then the latest comments are > retrieved for display via a search. Our user's latest comment is omitted > because even though it was inserted before the search was performed it had > yet to be inserted into the index. The workaround is simple - just add the > comment to bottom as long as it wasn't included in the search results, but > in more complex manifestations of this problem it isn't that easy. > > Its a rather generic issue. > > On Wednesday, March 20, 2013 4:30:24 PM UTC, David Pilato wrote: > > > But you talked about "the user reads their own writes". > > Just wondering if realtime search is really what you need. > > > > That said, I understand your concern here, because you have Couchbase in > > the middle and you have a first latency with Couchbase transport and > > another one with the ES refresh. > > > > Can you describe a bit more your use case and the reason you feel that > > you need realtime search? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 20 mars 2013 à 17:10, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > I'm using ES in combination with Couchbase so I'm only using ES for > > searching and then Couchbase for everything else. > > > > On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > > > > > If you need to read, then you should know that read is realtime. > > > Search is not. > > > > > > ## Are you looking for realtime get ? > > > > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > I understand that ES is "near" realtime (updating every second by > > > default). I think ES is awesome and I'm using it to great effect, but this > > > minor issue keeps popping up and workarounds that make sure the user reads > > > their own writes immediately aren't nice. > > > > > > Truly realtime search is obviously desirable and I assume having "near" > > > realtime isn't the end goal for the project. Are there any plans as to how > > > to achieve it? > > > > > > How about a mini-shard alongside each normal shard that holds the > > > documents that are not yet inserted into the index? It could expose the > > > same functionality as a normal shard, but it could be updated with each > > > insertion, effectively acting as a sort of cache. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![MarcusLongmuir](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@MarcusLongmuir](https://discuss.elastic.co/u/MarcusLongmuir)\ **Post date:** [March 21, 2013, 11:37am UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/8 "2013-03-21T11:37:41Z")
That blog post appears to solve almost the opposite problem - that the user will see changes when they shouldn't. I've thought about setting refresh to true with every save, but it seems like it would be detrimental to performance. I'm planning a large cluster and the refresh must occur on each server. To do that with each save doesn't sound healthy and would be increasingly problematic as the number of servers grows. On Thursday, March 21, 2013 9:58:49 AM UTC, Karussell wrote: > on the lucene layer one could handle it via: > > [Changing Bits: SearcherLifetimeManager prevents a broken search user experience](http://blog.mikemccandless.com/2011/11/searcherlifetimemanager-prevents-broken.html) > > you could ask the devs to include this somehow to elasticsearch, or > probably there is already an open issue although I did find one. > > Peter. > > On Wednesday, March 20, 2013 5:39:44 PM UTC+1, marcuslongmuir wrote: > > > Sorry for the confusion. The most common use case is usually similar to a > > comment being posted on an article or post (generic discussion board). > > > > Assuming we're not using AJAX and the comment is posted via a form, the > > page is loaded, the comment is inserted and then the latest comments are > > retrieved for display via a search. Our user's latest comment is omitted > > because even though it was inserted before the search was performed it had > > yet to be inserted into the index. The workaround is simple - just add the > > comment to bottom as long as it wasn't included in the search results, but > > in more complex manifestations of this problem it isn't that easy. > > > > Its a rather generic issue. > > > > On Wednesday, March 20, 2013 4:30:24 PM UTC, David Pilato wrote: > > > > > But you talked about "the user reads their own writes". > > > Just wondering if realtime search is really what you need. > > > > > > That said, I understand your concern here, because you have Couchbase in > > > the middle and you have a first latency with Couchbase transport and > > > another one with the ES refresh. > > > > > > Can you describe a bit more your use case and the reason you feel that > > > you need realtime search? > > > > > > -- > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 20 mars 2013 à 17:10, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > I'm using ES in combination with Couchbase so I'm only using ES for > > > searching and then Couchbase for everything else. > > > > > > On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > > > > > > > If you need to read, then you should know that read is realtime. > > > > Search is not. > > > > > > > > ## Are you looking for realtime get ? > > > > > > > > David 😉 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > > > I understand that ES is "near" realtime (updating every second by > > > > default). I think ES is awesome and I'm using it to great effect, but this > > > > minor issue keeps popping up and workarounds that make sure the user reads > > > > their own writes immediately aren't nice. > > > > > > > > Truly realtime search is obviously desirable and I assume having "near" > > > > realtime isn't the end goal for the project. Are there any plans as to how > > > > to achieve it? > > > > > > > > How about a mini-shard alongside each normal shard that holds the > > > > documents that are not yet inserted into the index? It could expose the > > > > same functionality as a normal shard, but it could be updated with each > > > > insertion, effectively acting as a sort of cache. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 21, 2013, 11:54am UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/9 "2013-03-21T11:54:56Z")
On Thu, 2013-03-21 at 04:37 -0700, marcuslongmuir wrote: > That blog post appears to solve almost the opposite problem - that the > user will see changes when they shouldn't. Agreed > I've thought about setting refresh to true with every save, but it > seems like it would be detrimental to performance. I'm planning a > large cluster and the refresh must occur on each server. To do that > with each save doesn't sound healthy and would be increasingly > problematic as the number of servers grows. Agreed, again 🙂 I've also wished for some in-memory only index that is able to search for docs that have just been indexed but are not yet written to segments. However, (not knowing much about the internals) I think that would require creating an inverted index every time a new doc is indexed, and would probably have quite an impact on performance. I think the way you're currently managing it is the best solution: - user adds comment - index comment into ES, and keep comment data & metadata around - do search for comments, and filter out user's new comment by ID - return search results plus user's comment clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![q42jaap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/q42jaap/32/2241_2.png) [@q42jaap](https://discuss.elastic.co/u/q42jaap)\ **Post date:** [March 21, 2013, 1:04pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/10 "2013-03-21T13:04:25Z")
Andy Wick's suggestion is the best one for your usecase. Just make the user wait a litle bit before redirecting them to the page where the comments are shown. Lower the index refresh time if you really want to see a quick response. Doing the update to couchbase and elasticsearch in parallel is also an option maybe. On Wednesday, March 20, 2013 5:39:44 PM UTC+1, marcuslongmuir wrote: > Sorry for the confusion. The most common use case is usually similar to a > comment being posted on an article or post (generic discussion board). > > Assuming we're not using AJAX and the comment is posted via a form, the > page is loaded, the comment is inserted and then the latest comments are > retrieved for display via a search. Our user's latest comment is omitted > because even though it was inserted before the search was performed it had > yet to be inserted into the index. The workaround is simple - just add the > comment to bottom as long as it wasn't included in the search results, but > in more complex manifestations of this problem it isn't that easy. > > Its a rather generic issue. > > On Wednesday, March 20, 2013 4:30:24 PM UTC, David Pilato wrote: > > > But you talked about "the user reads their own writes". > > Just wondering if realtime search is really what you need. > > > > That said, I understand your concern here, because you have Couchbase in > > the middle and you have a first latency with Couchbase transport and > > another one with the ES refresh. > > > > Can you describe a bit more your use case and the reason you feel that > > you need realtime search? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 20 mars 2013 à 17:10, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > I'm using ES in combination with Couchbase so I'm only using ES for > > searching and then Couchbase for everything else. > > > > On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > > > > > If you need to read, then you should know that read is realtime. > > > Search is not. > > > > > > ## Are you looking for realtime get ? > > > > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > I understand that ES is "near" realtime (updating every second by > > > default). I think ES is awesome and I'm using it to great effect, but this > > > minor issue keeps popping up and workarounds that make sure the user reads > > > their own writes immediately aren't nice. > > > > > > Truly realtime search is obviously desirable and I assume having "near" > > > realtime isn't the end goal for the project. Are there any plans as to how > > > to achieve it? > > > > > > How about a mini-shard alongside each normal shard that holds the > > > documents that are not yet inserted into the index? It could expose the > > > same functionality as a normal shard, but it could be updated with each > > > insertion, effectively acting as a sort of cache. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![MarcusLongmuir](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@MarcusLongmuir](https://discuss.elastic.co/u/MarcusLongmuir)\ **Post date:** [March 21, 2013, 1:46pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/11 "2013-03-21T13:46:27Z")
As much of an impact on performance as a refresh with every change? On Thursday, March 21, 2013 11:54:56 AM UTC, Clinton Gormley wrote: > On Thu, 2013-03-21 at 04:37 -0700, marcuslongmuir wrote: > > > That blog post appears to solve almost the opposite problem - that the > > user will see changes when they shouldn't. > > Agreed > > > I've thought about setting refresh to true with every save, but it > > seems like it would be detrimental to performance. I'm planning a > > large cluster and the refresh must occur on each server. To do that > > with each save doesn't sound healthy and would be increasingly > > problematic as the number of servers grows. > > Agreed, again 🙂 > > I've also wished for some in-memory only index that is able to search > for docs that have just been indexed but are not yet written to > segments. However, (not knowing much about the internals) I think that > would require creating an inverted index every time a new doc is > indexed, and would probably have quite an impact on performance. > > I think the way you're currently managing it is the best solution: > > - user adds comment > - index comment into ES, and keep comment data & metadata around > - do search for comments, and filter out user's new comment by ID > - return search results plus user's comment > > clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![MarcusLongmuir](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@MarcusLongmuir](https://discuss.elastic.co/u/MarcusLongmuir)\ **Post date:** [March 21, 2013, 2:03pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/12 "2013-03-21T14:03:15Z")
It's a good solution for that use case, but its not a solution for all of the problems that near realtime causes. Say I have a flag that I can set to prevent new insertions. Example: \*Set the flag to prevent insertions. \*Perform a search of current documents. \*Delete retrieved documents. I might assume that I had deleted all documents, but if the search results are reflecting the state up to 1 second ago I might miss documents inserted just before the flag was set. The workarounds suggested include waiting for the refresh (delay for a second) or refreshing on every change - both of which are directly or indirectly detrimental to the user experience in this case. Elasticsearch is awesome, but it seems that a remnant of the days before dynamic indexing is holding it back. If you were building a search product from the ground up today I doubt near realtime would be the end goal. Can any of the contributors shed some light on whether realtime search is desired or is in the pipeline? On Thursday, March 21, 2013 1:04:25 PM UTC, Jaap Taal wrote: > Andy Wick's suggestion is the best one for your usecase. Just make the > user wait a litle bit before redirecting them to the page where the > comments are shown. Lower the index refresh time if you really want to see > a quick response. > Doing the update to couchbase and elasticsearch in parallel is also an > option maybe. > > On Wednesday, March 20, 2013 5:39:44 PM UTC+1, marcuslongmuir wrote: > > > Sorry for the confusion. The most common use case is usually similar to a > > comment being posted on an article or post (generic discussion board). > > > > Assuming we're not using AJAX and the comment is posted via a form, the > > page is loaded, the comment is inserted and then the latest comments are > > retrieved for display via a search. Our user's latest comment is omitted > > because even though it was inserted before the search was performed it had > > yet to be inserted into the index. The workaround is simple - just add the > > comment to bottom as long as it wasn't included in the search results, but > > in more complex manifestations of this problem it isn't that easy. > > > > Its a rather generic issue. > > > > On Wednesday, March 20, 2013 4:30:24 PM UTC, David Pilato wrote: > > > > > But you talked about "the user reads their own writes". > > > Just wondering if realtime search is really what you need. > > > > > > That said, I understand your concern here, because you have Couchbase in > > > the middle and you have a first latency with Couchbase transport and > > > another one with the ES refresh. > > > > > > Can you describe a bit more your use case and the reason you feel that > > > you need realtime search? > > > > > > -- > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 20 mars 2013 à 17:10, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > I'm using ES in combination with Couchbase so I'm only using ES for > > > searching and then Couchbase for everything else. > > > > > > On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > > > > > > > If you need to read, then you should know that read is realtime. > > > > Search is not. > > > > > > > > ## Are you looking for realtime get ? > > > > > > > > David 😉 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > > > I understand that ES is "near" realtime (updating every second by > > > > default). I think ES is awesome and I'm using it to great effect, but this > > > > minor issue keeps popping up and workarounds that make sure the user reads > > > > their own writes immediately aren't nice. > > > > > > > > Truly realtime search is obviously desirable and I assume having "near" > > > > realtime isn't the end goal for the project. Are there any plans as to how > > > > to achieve it? > > > > > > > > How about a mini-shard alongside each normal shard that holds the > > > > documents that are not yet inserted into the index? It could expose the > > > > same functionality as a normal shard, but it could be updated with each > > > > insertion, effectively acting as a sort of cache. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Karussell\_2](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@Karussell\_2](https://discuss.elastic.co/u/Karussell_2)\ **Post date:** [March 21, 2013, 3:10pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/13 "2013-03-21T15:10:18Z")
sorry, that was indeed exactly the opposite topic although I saved this article in my brain to solve this 'comment' problem...but I thought mike blogged about exactly this problem ... However .. one option which you could explore is a small 'feeding' index with a _very_ small refresh\_interval. This index will be copied from time to time to your normal 'static' index. Via aliases you can easily search over several indices. Avoiding duplicates should be easy if you filter both indices by time (explore the filter alias thing) Peter. On Thursday, March 21, 2013 3:03:15 PM UTC+1, marcuslongmuir wrote: > It's a good solution for that use case, but its not a solution for all of > the problems that near realtime causes. Say I have a flag that I can set to > prevent new insertions. > > Example: > \*Set the flag to prevent insertions. > \*Perform a search of current documents. > \*Delete retrieved documents. > > I might assume that I had deleted all documents, but if the search results > are reflecting the state up to 1 second ago I might miss documents inserted > just before the flag was set. > > The workarounds suggested include waiting for the refresh (delay for a > second) or refreshing on every change - both of which are directly or > indirectly detrimental to the user experience in this case. > > Elasticsearch is awesome, but it seems that a remnant of the days before > dynamic indexing is holding it back. If you were building a search product > from the ground up today I doubt near realtime would be the end goal. > > Can any of the contributors shed some light on whether realtime search is > desired or is in the pipeline? > > On Thursday, March 21, 2013 1:04:25 PM UTC, Jaap Taal wrote: > > > Andy Wick's suggestion is the best one for your usecase. Just make the > > user wait a litle bit before redirecting them to the page where the > > comments are shown. Lower the index refresh time if you really want to see > > a quick response. > > Doing the update to couchbase and elasticsearch in parallel is also an > > option maybe. > > > > On Wednesday, March 20, 2013 5:39:44 PM UTC+1, marcuslongmuir wrote: > > > > > Sorry for the confusion. The most common use case is usually similar to > > > a comment being posted on an article or post (generic discussion board). > > > > > > Assuming we're not using AJAX and the comment is posted via a form, the > > > page is loaded, the comment is inserted and then the latest comments are > > > retrieved for display via a search. Our user's latest comment is omitted > > > because even though it was inserted before the search was performed it had > > > yet to be inserted into the index. The workaround is simple - just add the > > > comment to bottom as long as it wasn't included in the search results, but > > > in more complex manifestations of this problem it isn't that easy. > > > > > > Its a rather generic issue. > > > > > > On Wednesday, March 20, 2013 4:30:24 PM UTC, David Pilato wrote: > > > > > > > But you talked about "the user reads their own writes". > > > > Just wondering if realtime search is really what you need. > > > > > > > > That said, I understand your concern here, because you have Couchbase > > > > in the middle and you have a first latency with Couchbase transport and > > > > another one with the ES refresh. > > > > > > > > Can you describe a bit more your use case and the reason you feel that > > > > you need realtime search? > > > > > > > > -- > > > > David 😉 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 20 mars 2013 à 17:10, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > > > I'm using ES in combination with Couchbase so I'm only using ES for > > > > searching and then Couchbase for everything else. > > > > > > > > On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > > > > > > > > > If you need to read, then you should know that read is realtime. > > > > > Search is not. > > > > > > > > > > ## Are you looking for realtime get ? > > > > > > > > > > David 😉 > > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > > > > > I understand that ES is "near" realtime (updating every second by > > > > > default). I think ES is awesome and I'm using it to great effect, but this > > > > > minor issue keeps popping up and workarounds that make sure the user reads > > > > > their own writes immediately aren't nice. > > > > > > > > > > Truly realtime search is obviously desirable and I assume having > > > > > "near" realtime isn't the end goal for the project. Are there any plans as > > > > > to how to achieve it? > > > > > > > > > > How about a mini-shard alongside each normal shard that holds the > > > > > documents that are not yet inserted into the index? It could expose the > > > > > same functionality as a normal shard, but it could be updated with each > > > > > insertion, effectively acting as a sort of cache. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![q42jaap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/q42jaap/32/2241_2.png) [@q42jaap](https://discuss.elastic.co/u/q42jaap)\ **Post date:** [March 21, 2013, 4:02pm UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/14 "2013-03-21T16:02:24Z")
For user initiated actions, where you only want 1 user to see the change realtime, the refresh=true option is almost always a good solution. Your user just added a comment and hit submit the comment, the user probably doesn't care whether it takes a second. If you're indexing a large amount of data in high speed then the referesh option would slow your indexation speed down by a huge factor. I think this is meant by performance. For Elasticsearch it doesn't really matter, although having lot's of clients waiting for refresh adds a little overhead. Jaap Taal [Q42 BV | tel 070 44523 42 | direct 070 44523 65 | [http://q42.nl](http://q42.nl) | Waldorpstraat 17F, Den Haag | Vijzelstraat 72 unit 4.23, Amsterdam | KvK 30164662 ] On Thu, Mar 21, 2013 at 4:10 PM, Karussell [tableyourtime@gmail.com](mailto:tableyourtime@gmail.com) wrote: > sorry, that was indeed exactly the opposite topic although I saved this > article in my brain to solve this 'comment' problem...but I thought mike > blogged about exactly this problem ... > > However .. one option which you could explore is a small 'feeding' index > with a _very_ small refresh\_interval. This index will be copied from time to > time to your normal 'static' index. Via aliases you can easily search over > several indices. Avoiding duplicates should be easy if you filter both > indices by time (explore the filter alias thing) > > Peter. > > On Thursday, March 21, 2013 3:03:15 PM UTC+1, marcuslongmuir wrote: > > > It's a good solution for that use case, but its not a solution for all of > > the problems that near realtime causes. Say I have a flag that I can set to > > prevent new insertions. > > > > Example: > > \*Set the flag to prevent insertions. > > \*Perform a search of current documents. > > \*Delete retrieved documents. > > > > I might assume that I had deleted all documents, but if the search results > > are reflecting the state up to 1 second ago I might miss documents inserted > > just before the flag was set. > > > > The workarounds suggested include waiting for the refresh (delay for a > > second) or refreshing on every change - both of which are directly or > > indirectly detrimental to the user experience in this case. > > > > Elasticsearch is awesome, but it seems that a remnant of the days before > > dynamic indexing is holding it back. If you were building a search product > > from the ground up today I doubt near realtime would be the end goal. > > > > Can any of the contributors shed some light on whether realtime search is > > desired or is in the pipeline? > > > > On Thursday, March 21, 2013 1:04:25 PM UTC, Jaap Taal wrote: > > > > > Andy Wick's suggestion is the best one for your usecase. Just make the > > > user wait a litle bit before redirecting them to the page where the comments > > > are shown. Lower the index refresh time if you really want to see a quick > > > response. > > > Doing the update to couchbase and elasticsearch in parallel is also an > > > option maybe. > > > > > > On Wednesday, March 20, 2013 5:39:44 PM UTC+1, marcuslongmuir wrote: > > > > > > > Sorry for the confusion. The most common use case is usually similar to > > > > a comment being posted on an article or post (generic discussion board). > > > > > > > > Assuming we're not using AJAX and the comment is posted via a form, the > > > > page is loaded, the comment is inserted and then the latest comments are > > > > retrieved for display via a search. Our user's latest comment is omitted > > > > because even though it was inserted before the search was performed it had > > > > yet to be inserted into the index. The workaround is simple - just add the > > > > comment to bottom as long as it wasn't included in the search results, but > > > > in more complex manifestations of this problem it isn't that easy. > > > > > > > > Its a rather generic issue. > > > > > > > > On Wednesday, March 20, 2013 4:30:24 PM UTC, David Pilato wrote: > > > > > > > > > But you talked about "the user reads their own writes". > > > > > Just wondering if realtime search is really what you need. > > > > > > > > > > That said, I understand your concern here, because you have Couchbase > > > > > in the middle and you have a first latency with Couchbase transport and > > > > > another one with the ES refresh. > > > > > > > > > > Can you describe a bit more your use case and the reason you feel that > > > > > you need realtime search? > > > > > > > > > > -- > > > > > David 😉 > > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > > > Le 20 mars 2013 à 17:10, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > > > > > I'm using ES in combination with Couchbase so I'm only using ES for > > > > > searching and then Couchbase for everything else. > > > > > > > > > > On Wednesday, March 20, 2013 4:06:39 PM UTC, David Pilato wrote: > > > > > > > > > > > If you need to read, then you should know that read is realtime. > > > > > > Search is not. > > > > > > > > > > > > ## Are you looking for realtime get ? > > > > > > > > > > > > David 😉 > > > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > > > > > Le 20 mars 2013 à 16:59, marcuslongmuir [marcusl...@me.com](mailto:marcusl...@me.com) a écrit : > > > > > > > > > > > > I understand that ES is "near" realtime (updating every second by > > > > > > default). I think ES is awesome and I'm using it to great effect, but this > > > > > > minor issue keeps popping up and workarounds that make sure the user reads > > > > > > their own writes immediately aren't nice. > > > > > > > > > > > > Truly realtime search is obviously desirable and I assume having > > > > > > "near" realtime isn't the end goal for the project. Are there any plans as > > > > > > to how to achieve it? > > > > > > > > > > > > How about a mini-shard alongside each normal shard that holds the > > > > > > documents that are not yet inserted into the index? It could expose the same > > > > > > functionality as a normal shard, but it could be updated with each > > > > > > insertion, effectively acting as a sort of cache. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:45am UTC](https://discuss.elastic.co/t/potential-for-realtime-search/11220/15 "2017-07-06T02:45:19Z")