# Refresh + child documents **URL:** \ **Category:** Elasticsearch\ **Created:** [September 12, 2013, 2:50pm UTC](https://discuss.elastic.co/t/refresh-child-documents/13571 "2013-09-12T14:50:49Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Simon\_Gaeremynck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simon_gaeremynck/32/2142_2.png) [@Simon\_Gaeremynck](https://discuss.elastic.co/u/Simon_Gaeremynck)\ **Post date:** [September 12, 2013, 2:50pm UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/1 "2013-09-12T14:50:49Z")
Hi, In our unit tests for our app, we're seeing a couple of intermittent search failures when doing the following: 1. Create / Update a bunch of resources 2. Trigger an index refresh (with consistency == 'all') 3. Search -\> Failure because of missing expected results The search in step 3 is a query that searches through child documents. Is it possible that when the refresh from step 2 returns, the child documents haven't been fully re-indexed yet? We're only seeing the failure intermittently on a low-spec box under heavy load. Kind regards, Simon -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 12, 2013, 3:28pm UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/2 "2013-09-12T15:28:06Z")
Hey, can you share the test maybe? --Alex On Thu, Sep 12, 2013 at 4:50 PM, Simon Gaeremynck [gaeremyncks@gmail.com](mailto:gaeremyncks@gmail.com)wrote: > Hi, > > In our unit tests for our app, we're seeing a couple of intermittent > search failures when doing the following: > > 1. Create / Update a bunch of resources > 2. Trigger an index refresh (with consistency == 'all') > 3. Search > -\> Failure because of missing expected results > > The search in step 3 is a query that searches through child documents. > Is it possible that when the refresh from step 2 returns, the child > documents haven't been > fully re-indexed yet? > > We're only seeing the failure intermittently on a low-spec box under heavy > load. > > Kind regards, > > Simon > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Simon\_Gaeremynck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simon_gaeremynck/32/2142_2.png) [@Simon\_Gaeremynck](https://discuss.elastic.co/u/Simon_Gaeremynck)\ **Post date:** [September 13, 2013, 3:10pm UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/3 "2013-09-13T15:10:59Z")
Hi Alexander, The test can be found at [1] but it's not that straightforward to set up as the app requires quite a bit of dependencies. First some background info: Our app has lots of content that can be shared with lots of users. Content can be marked as private to a set of users. We've modelled that in ES by having top-level content documents which have a child document per user that has access to the content item. When we do a general search we add the user id of the current user and run a has\_child filter as well. I'll try my best to explain what the test is doing and what happens in the background. 1. The first couple of lines creates a couple of users 2. A piece of content gets created by user A 3. That content item gets shared with user B 4. The content items gets put in the index 5. All the users who have access to that piece of content are added as child documents 6. Refresh the search index (consistency = all) 7. Search for the content item 8. Assert we get the content item Occasionally (25% of the time maybe) the assertion in step 8 fails. Is it possible that when our application gets a response from the refresh request (6) ES hasn't actually fully re-indexed everything? FWIW, we've set the number of shards to 1 and the number of replica's to 0 as per the elasticsearch.yml recommendation for dev environments and that seems to help somewhat. (presumably because there are less shards and replicas to process thus causing less IO) The full query can be found at [2] Kind regards, Simon [1] [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-content/tests/test-library-search.js#L247](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-content/tests/test-library-search.js#L247) [2] { "query": { "filtered": { "query": { "query\_string": { "fields": [ "q\_high^2.0", "q\_low^0.75" ], "query": "\*" } }, "filter": { "and": [ { "term": { "\_type": "resource" } }, { "term": { "resourceType": "content" } }, { "has\_child": { "type": "resource\_members", "query": { "terms": { "direct\_members": [ "u:camtest:gJ-kkTf-2W" ] } } } } ] } } }, "from": 1, "size": 25, "sort": [ { "\_score": { "order": "desc" } }, { "sort": "asc" } ], "min\_score": 0.2 } On Thursday, September 12, 2013 4:28:06 PM UTC+1, Alexander Reelsen wrote: > Hey, > > can you share the test maybe? > > --Alex > > On Thu, Sep 12, 2013 at 4:50 PM, Simon Gaeremynck \<[gaere...@gmail.com](mailto:gaere...@gmail.com)\ > > > wrote: > > > Hi, > > > > In our unit tests for our app, we're seeing a couple of intermittent > > search failures when doing the following: > > > > 1. Create / Update a bunch of resources > > 2. Trigger an index refresh (with consistency == 'all') > > 3. Search > > -\> Failure because of missing expected results > > > > The search in step 3 is a query that searches through child documents. > > Is it possible that when the refresh from step 2 returns, the child > > documents haven't been > > fully re-indexed yet? > > > > We're only seeing the failure intermittently on a low-spec box under > > heavy load. > > > > Kind regards, > > > > Simon > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [September 16, 2013, 8:19am UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/4 "2013-09-16T08:19:06Z")
Hi, Where do you execute the explicit refresh operation? (I can't see it in the script you have shared) Martijn On 13 September 2013 17:10, Simon Gaeremynck [gaeremyncks@gmail.com](mailto:gaeremyncks@gmail.com) wrote: > Hi Alexander, > > The test can be found at [1] but it's not that straightforward to set up > as the app requires quite a bit of dependencies. > > First some background info: > > Our app has lots of content that can be shared with lots of users. Content > can be marked as private to a set of users. > We've modelled that in ES by having top-level content documents which have > a child document per user that has access to the content item. > > When we do a general search we add the user id of the current user and run > a has\_child filter as well. > > I'll try my best to explain what the test is doing and what happens in the > background. > > 1. The first couple of lines creates a couple of users > 2. A piece of content gets created by user A > 3. That content item gets shared with user B > 4. The content items gets put in the index > 5. All the users who have access to that piece of content are added as > child documents > 6. Refresh the search index (consistency = all) > 7. Search for the content item > 8. Assert we get the content item > > Occasionally (25% of the time maybe) the assertion in step 8 fails. > > Is it possible that when our application gets a response from the refresh > request (6) ES hasn't > actually fully re-indexed everything? > > FWIW, we've set the number of shards to 1 and the number of replica's to 0 > as per the elasticsearch.yml recommendation > for dev environments and that seems to help somewhat. (presumably because > there are less shards and replicas to process > thus causing less IO) > > The full query can be found at [2] > > Kind regards, > > Simon > > [1] > [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-content/tests/test-library-search.js#L247](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-content/tests/test-library-search.js#L247) > [2] > { > "query": { > "filtered": { > "query": { > "query\_string": { > "fields": [ > "q\_high^2.0", > "q\_low^0.75" > ], > "query": "\*" > } > }, > "filter": { > "and": [ > { > "term": { > "\_type": "resource" > } > }, > { > "term": { > "resourceType": "content" > } > }, > { > "has\_child": { > "type": "resource\_members", > "query": { > "terms": { > "direct\_members": [ > "u:camtest:gJ-kkTf-2W" > ] > } > } > } > } > ] > } > } > }, > "from": 1, > "size": 25, > "sort": [ > { > "\_score": { > "order": "desc" > } > }, > { > "sort": "asc" > } > ], > "min\_score": 0.2 > } > > On Thursday, September 12, 2013 4:28:06 PM UTC+1, Alexander Reelsen wrote: > > > Hey, > > > > can you share the test maybe? > > > > --Alex > > > > On Thu, Sep 12, 2013 at 4:50 PM, Simon Gaeremynck [gaere...@gmail.com](mailto:gaere...@gmail.com)wrote: > > > > > Hi, > > > > > > In our unit tests for our app, we're seeing a couple of intermittent > > > search failures when doing the following: > > > > > > 1. Create / Update a bunch of resources > > > 2. Trigger an index refresh (with consistency == 'all') > > > 3. Search > > > -\> Failure because of missing expected results > > > > > > The search in step 3 is a query that searches through child documents. > > > Is it possible that when the refresh from step 2 returns, the child > > > documents haven't been > > > fully re-indexed yet? > > > > > > We're only seeing the failure intermittently on a low-spec box under > > > heavy load. > > > > > > Kind regards, > > > > > > Simon > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Simon\_Gaeremynck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simon_gaeremynck/32/2142_2.png) [@Simon\_Gaeremynck](https://discuss.elastic.co/u/Simon_Gaeremynck)\ **Post date:** [September 16, 2013, 8:25am UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/5 "2013-09-16T08:25:26Z")
Hi, That happens in SearchTestsUtil.searchAll [1]. We wait till: - all items from the "index doc" queue have been picked off - all items from the "delete doc" queue have been picked off - the index has been refreshed (with searchRefreshed). - Get all the data Kind regards, Simon [1] [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-search/lib/test/util.js#L30](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-search/lib/test/util.js#L30) On 16 Sep 2013, at 09:19, Martijn v Groningen [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com) wrote: > Hi, > > Where do you execute the explicit refresh operation? (I can't see it in the script you have shared) > > Martijn > > On 13 September 2013 17:10, Simon Gaeremynck [gaeremyncks@gmail.com](mailto:gaeremyncks@gmail.com) wrote: > Hi Alexander, > > The test can be found at [1] but it's not that straightforward to set up as the app requires quite a bit of dependencies. > > First some background info: > > Our app has lots of content that can be shared with lots of users. Content can be marked as private to a set of users. > We've modelled that in ES by having top-level content documents which have a child document per user that has access to the content item. > > When we do a general search we add the user id of the current user and run a has\_child filter as well. > > I'll try my best to explain what the test is doing and what happens in the background. > > 1. The first couple of lines creates a couple of users > 2. A piece of content gets created by user A > 3. That content item gets shared with user B > 4. The content items gets put in the index > 5. All the users who have access to that piece of content are added as child documents > 6. Refresh the search index (consistency = all) > 7. Search for the content item > 8. Assert we get the content item > > Occasionally (25% of the time maybe) the assertion in step 8 fails. > > Is it possible that when our application gets a response from the refresh request (6) ES hasn't > actually fully re-indexed everything? > > FWIW, we've set the number of shards to 1 and the number of replica's to 0 as per the elasticsearch.yml recommendation > for dev environments and that seems to help somewhat. (presumably because there are less shards and replicas to process > thus causing less IO) > > The full query can be found at [2] > > Kind regards, > > Simon > > [1] [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-content/tests/test-library-search.js#L247](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-content/tests/test-library-search.js#L247) > [2] > { > "query": { > "filtered": { > "query": { > "query\_string": { > "fields": [ > "q\_high^2.0", > "q\_low^0.75" > ], > "query": "\*" > } > }, > "filter": { > "and": [ > { > "term": { > "\_type": "resource" > } > }, > { > "term": { > "resourceType": "content" > } > }, > { > "has\_child": { > "type": "resource\_members", > "query": { > "terms": { > "direct\_members": [ > "u:camtest:gJ-kkTf-2W" > ] > } > } > } > } > ] > } > } > }, > "from": 1, > "size": 25, > "sort": [ > { > "\_score": { > "order": "desc" > } > }, > { > "sort": "asc" > } > ], > "min\_score": 0.2 > } > > On Thursday, September 12, 2013 4:28:06 PM UTC+1, Alexander Reelsen wrote: > Hey, > > can you share the test maybe? > > --Alex > > On Thu, Sep 12, 2013 at 4:50 PM, Simon Gaeremynck [gaere...@gmail.com](mailto:gaere...@gmail.com) wrote: > Hi, > > In our unit tests for our app, we're seeing a couple of intermittent search failures when doing the following: > > 1. Create / Update a bunch of resources > 2. Trigger an index refresh (with consistency == 'all') > 3. Search > -\> Failure because of missing expected results > > The search in step 3 is a query that searches through child documents. > Is it possible that when the refresh from step 2 returns, the child documents haven't been > fully re-indexed yet? > > We're only seeing the failure intermittently on a low-spec box under heavy load. > > Kind regards, > > Simon > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Met vriendelijke groet, > > Martijn van Groningen > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [September 16, 2013, 7:39pm UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/6 "2013-09-16T19:39:37Z")
So this executes the ES refresh? ```auto ElasticSearch.refresh(function(err) { ... }); ``` I don't know this library, so I can't really tell if the actual refresh is performed. The refresh response header should indicate on how many shards it successfully succeeded. Can you verify if this is equal to the number of shards of the index that you're refreshing? (primary and replica shards) On 16 September 2013 10:25, Simon Gaeremynck [gaeremyncks@gmail.com](mailto:gaeremyncks@gmail.com) wrote: > Hi, > > That happens in SearchTestsUtil.searchAll [1]. > We wait till: > > - all items from the "index doc" queue have been picked off > - all items from the "delete doc" queue have been picked off > - the index has been refreshed (with searchRefreshed). > - Get all the data > > Kind regards, > > Simon > > [1] > [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-search/lib/test/util.js#L30](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-search/lib/test/util.js#L30) > > On 16 Sep 2013, at 09:19, Martijn v Groningen \< > [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com)\> wrote: > > > Hi, > > > > Where do you execute the explicit refresh operation? (I can't see it in > > the script you have shared) > > > > Martijn > > > > On 13 September 2013 17:10, Simon Gaeremynck [gaeremyncks@gmail.com](mailto:gaeremyncks@gmail.com) > > wrote: > > Hi Alexander, > > > > The test can be found at [1] but it's not that straightforward to set up > > as the app requires quite a bit of dependencies. > > > > First some background info: > > > > Our app has lots of content that can be shared with lots of users. > > Content can be marked as private to a set of users. > > We've modelled that in ES by having top-level content documents which > > have a child document per user that has access to the content item. > > > > When we do a general search we add the user id of the current user and > > run a has\_child filter as well. > > > > I'll try my best to explain what the test is doing and what happens in > > the background. > > > > 1. The first couple of lines creates a couple of users > > 2. A piece of content gets created by user A > > 3. That content item gets shared with user B > > 4. The content items gets put in the index > > 5. All the users who have access to that piece of content are added as > > child documents > > 6. Refresh the search index (consistency = all) > > 7. Search for the content item > > 8. Assert we get the content item > > > > Occasionally (25% of the time maybe) the assertion in step 8 fails. > > > > Is it possible that when our application gets a response from the > > refresh request (6) ES hasn't > > actually fully re-indexed everything? > > > > FWIW, we've set the number of shards to 1 and the number of replica's to > > 0 as per the elasticsearch.yml recommendation > > for dev environments and that seems to help somewhat. (presumably > > because there are less shards and replicas to process > > thus causing less IO) > > > > The full query can be found at [2] > > > > Kind regards, > > > > Simon > > > > [1] > > [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-content/tests/test-library-search.js#L247](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-content/tests/test-library-search.js#L247) > > [2] > > { > > "query": { > > "filtered": { > > "query": { > > "query\_string": { > > "fields": [ > > "q\_high^2.0", > > "q\_low^0.75" > > ], > > "query": "\*" > > } > > }, > > "filter": { > > "and": [ > > { > > "term": { > > "\_type": "resource" > > } > > }, > > { > > "term": { > > "resourceType": "content" > > } > > }, > > { > > "has\_child": { > > "type": "resource\_members", > > "query": { > > "terms": { > > "direct\_members": [ > > "u:camtest:gJ-kkTf-2W" > > ] > > } > > } > > } > > } > > ] > > } > > } > > }, > > "from": 1, > > "size": 25, > > "sort": [ > > { > > "\_score": { > > "order": "desc" > > } > > }, > > { > > "sort": "asc" > > } > > ], > > "min\_score": 0.2 > > } > > > > On Thursday, September 12, 2013 4:28:06 PM UTC+1, Alexander Reelsen > > wrote: > > Hey, > > > > can you share the test maybe? > > > > --Alex > > > > On Thu, Sep 12, 2013 at 4:50 PM, Simon Gaeremynck [gaere...@gmail.com](mailto:gaere...@gmail.com) > > wrote: > > Hi, > > > > In our unit tests for our app, we're seeing a couple of intermittent > > search failures when doing the following: > > > > 1. Create / Update a bunch of resources > > 2. Trigger an index refresh (with consistency == 'all') > > 3. Search > > -\> Failure because of missing expected results > > > > The search in step 3 is a query that searches through child documents. > > Is it possible that when the refresh from step 2 returns, the child > > documents haven't been > > fully re-indexed yet? > > > > We're only seeing the failure intermittently on a low-spec box under > > heavy load. > > > > Kind regards, > > > > Simon > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Simon\_Gaeremynck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simon_gaeremynck/32/2142_2.png) [@Simon\_Gaeremynck](https://discuss.elastic.co/u/Simon_Gaeremynck)\ **Post date:** [September 23, 2013, 9:25am UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/7 "2013-09-23T09:25:43Z")
That is correct. I can confirm the refresh is performed. I get the following response: { ok: true, \_shards: { total: 2, successful: 1, failed: 0 } } That's on a single node instance with index.number\_of\_shards: 1 index.number\_of\_replicas: 0 I don't fully understand why the response has a total of 2 when there is only 1 shard though. Kind regards, Simon On 16 Sep 2013, at 20:39, Martijn v Groningen [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com) wrote: > So this executes the ES refresh? > > ```auto > ElasticSearch.refresh(function(err) { > ... > }); > > ``` > > I don't know this library, so I can't really tell if the actual refresh is performed. The refresh response header should indicate on how many shards it successfully succeeded. Can you verify if this is equal to the number of shards of the index that you're refreshing? (primary and replica shards) > > On 16 September 2013 10:25, Simon Gaeremynck [gaeremyncks@gmail.com](mailto:gaeremyncks@gmail.com) wrote: > Hi, > > That happens in SearchTestsUtil.searchAll [1]. > We wait till: > > - all items from the "index doc" queue have been picked off > - all items from the "delete doc" queue have been picked off > - the index has been refreshed (with searchRefreshed). > - Get all the data > > Kind regards, > > Simon > > [1] [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-search/lib/test/util.js#L30](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-search/lib/test/util.js#L30) > > On 16 Sep 2013, at 09:19, Martijn v Groningen [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com) wrote: > > > Hi, > > > > Where do you execute the explicit refresh operation? (I can't see it in the script you have shared) > > > > Martijn > > > > On 13 September 2013 17:10, Simon Gaeremynck [gaeremyncks@gmail.com](mailto:gaeremyncks@gmail.com) wrote: > > Hi Alexander, > > > > The test can be found at [1] but it's not that straightforward to set up as the app requires quite a bit of dependencies. > > > > First some background info: > > > > Our app has lots of content that can be shared with lots of users. Content can be marked as private to a set of users. > > We've modelled that in ES by having top-level content documents which have a child document per user that has access to the content item. > > > > When we do a general search we add the user id of the current user and run a has\_child filter as well. > > > > I'll try my best to explain what the test is doing and what happens in the background. > > > > 1. The first couple of lines creates a couple of users > > 2. A piece of content gets created by user A > > 3. That content item gets shared with user B > > 4. The content items gets put in the index > > 5. All the users who have access to that piece of content are added as child documents > > 6. Refresh the search index (consistency = all) > > 7. Search for the content item > > 8. Assert we get the content item > > > > Occasionally (25% of the time maybe) the assertion in step 8 fails. > > > > Is it possible that when our application gets a response from the refresh request (6) ES hasn't > > actually fully re-indexed everything? > > > > FWIW, we've set the number of shards to 1 and the number of replica's to 0 as per the elasticsearch.yml recommendation > > for dev environments and that seems to help somewhat. (presumably because there are less shards and replicas to process > > thus causing less IO) > > > > The full query can be found at [2] > > > > Kind regards, > > > > Simon > > > > [1] [https://github.com/oaeproject/Hilary/blob/master/node\_modules/oae-content/tests/test-library-search.js#L247](https://github.com/oaeproject/Hilary/blob/master/node_modules/oae-content/tests/test-library-search.js#L247) > > [2] > > { > > "query": { > > "filtered": { > > "query": { > > "query\_string": { > > "fields": [ > > "q\_high^2.0", > > "q\_low^0.75" > > ], > > "query": "\*" > > } > > }, > > "filter": { > > "and": [ > > { > > "term": { > > "\_type": "resource" > > } > > }, > > { > > "term": { > > "resourceType": "content" > > } > > }, > > { > > "has\_child": { > > "type": "resource\_members", > > "query": { > > "terms": { > > "direct\_members": [ > > "u:camtest:gJ-kkTf-2W" > > ] > > } > > } > > } > > } > > ] > > } > > } > > }, > > "from": 1, > > "size": 25, > > "sort": [ > > { > > "\_score": { > > "order": "desc" > > } > > }, > > { > > "sort": "asc" > > } > > ], > > "min\_score": 0.2 > > } > > > > On Thursday, September 12, 2013 4:28:06 PM UTC+1, Alexander Reelsen wrote: > > Hey, > > > > can you share the test maybe? > > > > --Alex > > > > On Thu, Sep 12, 2013 at 4:50 PM, Simon Gaeremynck [gaere...@gmail.com](mailto:gaere...@gmail.com) wrote: > > Hi, > > > > In our unit tests for our app, we're seeing a couple of intermittent search failures when doing the following: > > > > 1. Create / Update a bunch of resources > > 2. Trigger an index refresh (with consistency == 'all') > > 3. Search > > -\> Failure because of missing expected results > > > > The search in step 3 is a query that searches through child documents. > > Is it possible that when the refresh from step 2 returns, the child documents haven't been > > fully re-indexed yet? > > > > We're only seeing the failure intermittently on a low-spec box under heavy load. > > > > Kind regards, > > > > Simon > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen > > > > -- > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Met vriendelijke groet, > > Martijn van Groningen > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/M-ByUcpEDSM/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:15am UTC](https://discuss.elastic.co/t/refresh-child-documents/13571/8 "2017-07-06T02:15:09Z")