# Inconsistent search results from local/primary **URL:** \ **Category:** Elasticsearch\ **Created:** [August 25, 2011, 9:50pm UTC](https://discuss.elastic.co/t/inconsistent-search-results-from-local-primary/5253 "2011-08-25T21:50:10Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Paul\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_brown/32/1449_2.png) [@Paul\_Brown](https://discuss.elastic.co/u/Paul_Brown)\ **Post date:** [August 25, 2011, 9:50pm UTC](https://discuss.elastic.co/t/inconsistent-search-results-from-local-primary/5253/1 "2011-08-25T21:50:10Z")
Hi, ES-folk -- Ran into something odd today. For one index with a small number of documents, I see 2 results if I query with preference=local and 3 results if I query with preference=primary. (Without a preference set, the result bounces back and forth between 2 and 3 items.) In case it was just an issue with data settling, I waited a bit, but the mismatch is still there. The cluster seems happy. Is this a settings/configuration issue or a potential bug? Additional context is Elasticsearch 0.17.6, four-node cluster, S3 gateway, 5 shards with 1 replica. Thanks in advance! -- Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 26, 2011, 2:07pm UTC](https://discuss.elastic.co/t/inconsistent-search-results-from-local-primary/5253/2 "2011-08-26T14:07:35Z")
It sounds like a potential problem, they should be sync'ed up. You did not change anything in the refresh interval, right? Can you do the following and see if it still happens: 1. Issue a refresh and check. 2. Issue a flush and check. 3. Issue a refresh again and check. 4. Issue a full flush (curl -XPOST localhost:9200/\_flush?full=true) and check. 5. Issue a refresh and check. On Fri, Aug 26, 2011 at 12:50 AM, Paul Brown [prb@mult.ifario.us](mailto:prb@mult.ifario.us) wrote: > Hi, ES-folk -- > > Ran into something odd today. For one index with a small number of > documents, I see 2 results if I query with preference=local and 3 results if > I query with preference=primary. (Without a preference set, the result > bounces back and forth between 2 and 3 items.) In case it was just an issue > with data settling, I waited a bit, but the mismatch is still there. The > cluster seems happy. Is this a settings/configuration issue or a potential > bug? > > Additional context is Elasticsearch 0.17.6, four-node cluster, S3 gateway, > 5 shards with 1 replica. > > Thanks in advance! > > -- Paul ---
**Author:** ![Paul\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_brown/32/1449_2.png) [@Paul\_Brown](https://discuss.elastic.co/u/Paul_Brown)\ **Post date:** [August 26, 2011, 7:08pm UTC](https://discuss.elastic.co/t/inconsistent-search-results-from-local-primary/5253/3 "2011-08-26T19:08:09Z")
Hi, Shay -- The flush/refresh process was not effective. To add some additional context, it looks like the missing document was ingested during some HA verification testing where hosts were successively killed and restarted, so I'll propose that the missing document (index size 2 versus index size 3) was somehow lost in a race between switching masters and handoff. Is there a way to have a single index rebuild its replicas from a current master? -- Paul On Fri, Aug 26, 2011 at 7:07 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > It sounds like a potential problem, they should be sync'ed up. You did not > change anything in the refresh interval, right? > Can you do the following and see if it still happens: > > 1. Issue a refresh and check. > 2. Issue a flush and check. > 3. Issue a refresh again and check. > 4. Issue a full flush (curl -XPOST localhost:9200/\_flush?full=true) and > check. > 5. Issue a refresh and check. > On Fri, Aug 26, 2011 at 12:50 AM, Paul Brown [prb@mult.ifario.us](mailto:prb@mult.ifario.us) wrote: > > > Hi, ES-folk -- > > > > Ran into something odd today. For one index with a small number of > > documents, I see 2 results if I query with preference=local and 3 results if > > I query with preference=primary. (Without a preference set, the result > > bounces back and forth between 2 and 3 items.) In case it was just an issue > > with data settling, I waited a bit, but the mismatch is still there. The > > cluster seems happy. Is this a settings/configuration issue or a potential > > bug? > > > > Additional context is Elasticsearch 0.17.6, four-node cluster, S3 gateway, > > 5 shards with 1 replica. > > > > Thanks in advance! > > > > -- Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 26, 2011, 9:09pm UTC](https://discuss.elastic.co/t/inconsistent-search-results-from-local-primary/5253/4 "2011-08-26T21:09:08Z")
If you restart the node with the offending shard, it will resync itself against the other shard. But, this should not happen.. . I have several tests, both long running and integration ones that simulate nodes coming and going while indexing to make sure it does not happen, and they all pass. Can you maybe try and create a testcase for this? On Fri, Aug 26, 2011 at 10:08 PM, Paul Brown [prb@mult.ifario.us](mailto:prb@mult.ifario.us) wrote: > Hi, Shay -- > > The flush/refresh process was not effective. To add some additional > context, it looks like the missing document was ingested during some > HA verification testing where hosts were successively killed and > restarted, so I'll propose that the missing document (index size 2 > versus index size 3) was somehow lost in a race between switching > masters and handoff. > > Is there a way to have a single index rebuild its replicas from a > current master? > > -- Paul > > On Fri, Aug 26, 2011 at 7:07 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > It sounds like a potential problem, they should be sync'ed up. You did > > not > > change anything in the refresh interval, right? > > Can you do the following and see if it still happens: > > > > 1. Issue a refresh and check. > > 2. Issue a flush and check. > > 3. Issue a refresh again and check. > > 4. Issue a full flush (curl -XPOST localhost:9200/\_flush?full=true) and > > check. > > 5. Issue a refresh and check. > > On Fri, Aug 26, 2011 at 12:50 AM, Paul Brown [prb@mult.ifario.us](mailto:prb@mult.ifario.us) wrote: > > > > > Hi, ES-folk -- > > > > > > Ran into something odd today. For one index with a small number of > > > documents, I see 2 results if I query with preference=local and 3 > > > results if > > > I query with preference=primary. (Without a preference set, the result > > > bounces back and forth between 2 and 3 items.) In case it was just an > > > issue > > > with data settling, I waited a bit, but the mismatch is still there. > > > The > > > cluster seems happy. Is this a settings/configuration issue or a > > > potential > > > bug? > > > > > > Additional context is Elasticsearch 0.17.6, four-node cluster, S3 > > > gateway, > > > 5 shards with 1 replica. > > > > > > Thanks in advance! > > > > > > -- Paul ---
**Author:** ![Paul\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_brown/32/1449_2.png) [@Paul\_Brown](https://discuss.elastic.co/u/Paul_Brown)\ **Post date:** [August 26, 2011, 9:45pm UTC](https://discuss.elastic.co/t/inconsistent-search-results-from-local-primary/5253/5 "2011-08-26T21:45:32Z")
I'll give it a go. It's some combination of network partitions, cluster membership, etc., with the partition being the more difficult thing to simulate without multiple hosts. On Fri, Aug 26, 2011 at 2:09 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > If you restart the node with the offending shard, it will resync itself > against the other shard. But, this should not happen.. . I have several > tests, both long running and integration ones that simulate nodes coming and > going while indexing to make sure it does not happen, and they all pass. Can > you maybe try and create a testcase for this? > > On Fri, Aug 26, 2011 at 10:08 PM, Paul Brown [prb@mult.ifario.us](mailto:prb@mult.ifario.us) wrote: > > > Hi, Shay -- > > > > The flush/refresh process was not effective. To add some additional > > context, it looks like the missing document was ingested during some > > HA verification testing where hosts were successively killed and > > restarted, so I'll propose that the missing document (index size 2 > > versus index size 3) was somehow lost in a race between switching > > masters and handoff. > > > > Is there a way to have a single index rebuild its replicas from a > > current master? > > > > -- Paul > > > > On Fri, Aug 26, 2011 at 7:07 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > It sounds like a potential problem, they should be sync'ed up. You did > > > not > > > change anything in the refresh interval, right? > > > Can you do the following and see if it still happens: > > > > > > 1. Issue a refresh and check. > > > 2. Issue a flush and check. > > > 3. Issue a refresh again and check. > > > 4. Issue a full flush (curl -XPOST localhost:9200/\_flush?full=true) and > > > check. > > > 5. Issue a refresh and check. > > > On Fri, Aug 26, 2011 at 12:50 AM, Paul Brown [prb@mult.ifario.us](mailto:prb@mult.ifario.us) wrote: > > > > > > > Hi, ES-folk -- > > > > > > > > Ran into something odd today. For one index with a small number of > > > > documents, I see 2 results if I query with preference=local and 3 > > > > results if > > > > I query with preference=primary. (Without a preference set, the result > > > > bounces back and forth between 2 and 3 items.) In case it was just an > > > > issue > > > > with data settling, I waited a bit, but the mismatch is still there. > > > > The > > > > cluster seems happy. Is this a settings/configuration issue or a > > > > potential > > > > bug? > > > > > > > > Additional context is Elasticsearch 0.17.6, four-node cluster, S3 > > > > gateway, > > > > 5 shards with 1 replica. > > > > > > > > Thanks in advance! > > > > > > > > -- Paul ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:55am UTC](https://discuss.elastic.co/t/inconsistent-search-results-from-local-primary/5253/6 "2017-07-06T03:55:51Z")