# Querying a document immediately after indexing it **URL:** \ **Category:** Elasticsearch\ **Created:** [September 15, 2012, 4:46pm UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035 "2012-09-15T16:46:25Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![noor](https://avatars.discourse-cdn.com/v4/letter/n/c67d28/32.png) [@noor](https://discuss.elastic.co/u/noor)\ **Post date:** [September 15, 2012, 4:46pm UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035/1 "2012-09-15T16:46:25Z")
Hi guys, I know the subject seems little bit clumsy. Let me try to explain it with the exact usecase. Me and my team members are evaluating elasticsearch for an eCommerce project. We want our customers to search for products they want to buy from the search portal. For this we want to show our customer products in some form of grouping (where each group will represent multiple similar kind of products e.g. one search result record can be IPhone representing all IPhone series) in the search result. We know this cannot be achieved with elasticsearch ( [https://github.com/elasticsearch/elasticsearch/issues/256](https://github.com/elasticsearch/elasticsearch/issues/256)). So we decided to store the grouped representation of the products in elastic search instead of actual products with a simple criteria. At time of adding a product search for the representative product group based on product attributes and if we find a group then associate the product with the group otherwise create a new group from this newly added product. The problem we are facing here is that we cannot search a product (consider its as a group) which is just added into elasticsearch before adding a new product resulting in creation of another similar group. _Note:_ we are aware of elasticsearch realtime get and faceted search features and both seems irrelevant to our problem space. My question is that can we achieve realtime indexing without using refresh API (as it results in complete refresh of index) or is it the only way and if it is than what will be the performance impact of using refresh API. Thanks, Noor -- ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [September 15, 2012, 5:29pm UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035/2 "2012-09-15T17:29:44Z")
Hi, Elasticsearch is near-real time system. Refresh is needed for new documents to be searchable. To my knowledge, there is no other way around this now. On the other hand refresh happens every 1sec by default and you can change it (also on the fly > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. ). May be it would help if you describe in more details why you need real-time search in your case. To me it is surprising you need a true real-time product search in eCommerce site (what is the rate of index/update operations and can your users really learn they see results that are few seconds behind?). If facets do not apply to your case (btw why?) you might look at nested type or may be better to parent/child documents. After reading your email several times it sounds to me as if you might be already using parent/child where you create parent documents on the fly while indexing children and then the problem could be that you might not be able to check if particular parent already exists because of the refresh issue. Is that the case? Regards, Lukas On Sat, Sep 15, 2012 at 6:46 PM, Noor [nooremailbox@gmail.com](mailto:nooremailbox@gmail.com) wrote: > Hi guys, > > I know the subject seems little bit clumsy. Let me try to explain it with > the exact usecase. Me and my team members are evaluating elasticsearch for > an eCommerce project. We want our customers to search for products they > want to buy from the search portal. For this we want to show our customer > products in some form of grouping (where each group will represent multiple > similar kind of products e.g. one search result record can be IPhone > representing all IPhone series) in the search result. > > We know this cannot be achieved with elasticsearch ( > [Field Collapsing/Combining · Issue #256 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/256)). So we decided > to store the grouped representation of the products in Elasticsearch > instead of actual products with a simple criteria. At time of adding a > product search for the representative product group based on product > attributes and if we find a group then associate the product with the group > otherwise create a new group from this newly added product. > > The problem we are facing here is that we cannot search a product > (consider its as a group) which is just added into elasticsearch before > adding a new product resulting in creation of another similar group. > > _Note:_ we are aware of elasticsearch realtime get and faceted search > features and both seems irrelevant to our problem space. > > My question is that can we achieve realtime indexing without using refresh > API (as it results in complete refresh of index) or is it the only way and > if it is than what will be the performance impact of using refresh API. > > Thanks, > > Noor > > -- -- ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 15, 2012, 5:33pm UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035/3 "2012-09-15T17:33:44Z")
Hi We ran into exact the same issue and use the excellent patch for supporting result grouping by Martijn van Groningen (it runs in production since over a year). The only limitation is, that you need to use one shard (so unless you are not having millions of products this should not pose a problem). We did not group our products by series but rather by same images, this means that products like clothes with the same color and size are grouped into one search results (blue shirt in s/l/xl is grouped, the same shirt in red is not put into that group, but all red shirts in all sizes are grouped). You can see the patch in action at [http://www.lusini.de](http://www.lusini.de) We have implemented a streaming json river, which updates the products every 30 seconds (and we are not in any need of realtime indexing features). In case you want to try out the patch, clone this repo [https://github.com/lusini/elasticsearch-grouping-patches](https://github.com/lusini/elasticsearch-grouping-patches) and run build-patched-elasticsearch.sh If you have any more questions, feel free to ask. Regards, Alexander -- ---
**Author:** ![noor](https://avatars.discourse-cdn.com/v4/letter/n/c67d28/32.png) [@noor](https://discuss.elastic.co/u/noor)\ **Post date:** [September 15, 2012, 8:55pm UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035/4 "2012-09-15T20:55:16Z")
Thanks guys for giving your insight into our problem. @Lukas Yes, you understood our problem correct. The reason why faceted search is not helpful is because it gives you count of the tags in which you are interested in, like count of different demographics or occupations against different user profiles as described in the elasticsearch example for faceted search. It is not meant for grouping the products like Alexander mentioned in his reply. The reason why we need real time indexing is that we want to create the parent on the fly if it does not exist and then want to associate the child\*right after - the creation of parent. @Alexander I had also seen the elasticsearch grouping patch but like you said it is limited to one shard. We are actually building a cross store solution. In which each individual store will post its products to our solution and then customers will search different products from our cross store solution which will then (some how) take them to a specific store. Here different stores can submit similar kind of product to our solution due to this reason we want our search result to show similar products represented by a single representative group e.g one IPhone 4s will represent all the IPhone 4s submitted from different stores. We had plan to use scheduler based approach to avoid realtime indexing problem. But we again caught into the same loop because we want our representative product to be created at time of adding a new product if it does not find the group of similar nature. Thanks, Noor -- ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [September 16, 2012, 6:00am UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035/5 "2012-09-16T06:00:03Z")
Hi, if your product category ID would be predictable/deterministic (for example you know that iPhone5 would go to "iphone" category) then I think you can still do it because then you could use realtime get to check if this product group exists and I think then you should be fine creating them on the fly. Have you tried this approach? Regards, Lukas On Sat, Sep 15, 2012 at 10:55 PM, Noor [nooremailbox@gmail.com](mailto:nooremailbox@gmail.com) wrote: > Thanks guys for giving your insight into our problem. > > @Lukas > Yes, you understood our problem correct. The reason why faceted search is > not helpful is because it gives you count of the tags in which you are > interested in, like count of different demographics or occupations against > different user profiles as described in the elasticsearch example for > faceted search. It is not meant for grouping the products > like Alexander mentioned in his reply. > > The reason why we need real time indexing is that we want to create the > parent on the fly if it does not exist and then want to associate the child > > - right after\* the creation of parent. > > @Alexander > I had also seen the elasticsearch grouping patch but like you said it is > limited to one shard. > > We are actually building a cross store solution. In which each individual > store will post its products to our solution and then customers will search > different products from our cross store solution which will then (some how) > take them to a specific store. Here different stores can submit similar > kind of product to our solution due to this reason we want our search > result to show similar products represented by a single representative > group e.g one IPhone 4s will represent all the IPhone 4s submitted from > different stores. > > We had plan to use scheduler based approach to avoid realtime indexing > problem. But we again caught into the same loop because we want our > representative product to be created at time of adding a new product if it > does not find the group of similar nature. > > Thanks, > > Noor > > -- -- ---
**Author:** ![phill](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@phill](https://discuss.elastic.co/u/phill)\ **Post date:** [October 16, 2012, 8:49am UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035/6 "2012-10-16T08:49:19Z")
But, if you need to only find out if a document exists given it's ID, that is different than searching that is using the get interface. "By default, the get API is realtime, and is not affected by the refresh rate of the index (when data will become visible for search)." > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. To complete the thread called "writing parents and children docs" from 2 weeks ago, I implemented this and it seems to work fine. In fact, it was Lukáš who pointed out to me that get works different than search and claims to be real time. -Paul On 9/15/2012 11:00 PM, Lukáš Vlček wrote: > On Sat, Sep 15, 2012 at 10:55 PM, Noor \<[nooremailbox@gmail.com](mailto:nooremailbox@gmail.com) > [mailto:nooremailbox@gmail.com](mailto:nooremailbox@gmail.com)\> wrote: > > ``` > The reason why we need real time indexing is that we want to > create the parent on the fly if it does not exist and then want to > associate the child/*right after*/ the creation of parent. > > ``` -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:08am UTC](https://discuss.elastic.co/t/querying-a-document-immediately-after-indexing-it/9035/7 "2017-07-06T03:08:37Z")