# 3 Node Cluster With Nodes Out of Sync **URL:** \ **Category:** Elasticsearch\ **Created:** [November 25, 2014, 4:37pm UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941 "2014-11-25T16:37:23Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Yosi\_Haran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yosi_haran/32/44914_2.png) [@Yosi\_Haran](https://discuss.elastic.co/u/Yosi_Haran)\ **Post date:** [November 25, 2014, 4:37pm UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/1 "2014-11-25T16:37:23Z")
Hi Guys, We are running a 3 node cluster, and each node returns a different number of documents when issued a direct HTTP "\_count" call. The cluster holds about 150K documents and the differences range from 30~50 documents, but are still troubling. This shouldn't be a "split brain" problem, since we have set: discovery.zen.minimum\_master\_nodes: 2 We also have a "client" node, but since client nodes are eligible to be master, I understand that they shouldn't affect the master election process. Any Ideas about why and how this is happening? Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [November 25, 2014, 4:41pm UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/2 "2014-11-25T16:41:30Z")
minimum\_master\_nodes still doesn't protect you from all possible failure scenarios, see [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) What version are you running? -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran [yosi@my6sense.com](mailto:yosi@my6sense.com) wrote: > Hi Guys, > > We are running a 3 node cluster, and each node returns a different number > of documents when issued a direct HTTP "\_count" call. > > The cluster holds about 150K documents and the differences range from > 30~50 documents, but are still troubling. > > This shouldn't be a "split brain" problem, since we have set: > discovery.zen.minimum\_master\_nodes: 2 > We also have a "client" node, but since client nodes are eligible to be > master, I understand that they shouldn't affect the master election process. > > Any Ideas about why and how this is happening? > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu6r2z39iT%2B%3Dy7b6Brez%2BhLz8davSRaq0UmMviCCqV\_sQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu6r2z39iT%2B%3Dy7b6Brez%2BhLz8davSRaq0UmMviCCqV_sQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yosi\_Haran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yosi_haran/32/44914_2.png) [@Yosi\_Haran](https://discuss.elastic.co/u/Yosi_Haran)\ **Post date:** [November 25, 2014, 4:52pm UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/3 "2014-11-25T16:52:19Z")
Thanks! I'll check out the link you sent and see if it'll help... We're are running version 1.0.0: "version": { "number": "1.0.0", "build\_hash": "a46900e9c72c0a623d71b54016357d5f94c8ea32", "build\_timestamp": "2014-02-12T16:18:34Z", "build\_snapshot": false, "lucene\_version": "4.6" } On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote: > minimum\_master\_nodes still doesn't protect you from all possible failure > scenarios, see [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) > > What version are you running? > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran \<[yo...@my6sense.com](mailto:yo...@my6sense.com) > \\> wrote: > > > Hi Guys, > > > > We are running a 3 node cluster, and each node returns a different number > > of documents when issued a direct HTTP "\_count" call. > > > > The cluster holds about 150K documents and the differences range from > > 30~50 documents, but are still troubling. > > > > This shouldn't be a "split brain" problem, since we have set: > > discovery.zen.minimum\_master\_nodes: 2 > > We also have a "client" node, but since client nodes are eligible to be > > master, I understand that they shouldn't affect the master election process. > > > > Any Ideas about why and how this is happening? > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec97ea83-bfef-4831-8af0-9a7a86170124%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec97ea83-bfef-4831-8af0-9a7a86170124%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yosi\_Haran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yosi_haran/32/44914_2.png) [@Yosi\_Haran](https://discuss.elastic.co/u/Yosi_Haran)\ **Post date:** [November 25, 2014, 4:53pm UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/4 "2014-11-25T16:53:10Z")
Thanks! I'll check out the link and let you know if it helped. We are running version 1.0.0 On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote: > minimum\_master\_nodes still doesn't protect you from all possible failure > scenarios, see [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) > > What version are you running? > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran \<[yo...@my6sense.com](mailto:yo...@my6sense.com) > \\> wrote: > > > Hi Guys, > > > > We are running a 3 node cluster, and each node returns a different number > > of documents when issued a direct HTTP "\_count" call. > > > > The cluster holds about 150K documents and the differences range from > > 30~50 documents, but are still troubling. > > > > This shouldn't be a "split brain" problem, since we have set: > > discovery.zen.minimum\_master\_nodes: 2 > > We also have a "client" node, but since client nodes are eligible to be > > master, I understand that they shouldn't affect the master election process. > > > > Any Ideas about why and how this is happening? > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5addc70b-b76c-46d5-b122-cf730827d677%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5addc70b-b76c-46d5-b122-cf730827d677%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yosi\_Haran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yosi_haran/32/44914_2.png) [@Yosi\_Haran](https://discuss.elastic.co/u/Yosi_Haran)\ **Post date:** [November 25, 2014, 5:29pm UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/5 "2014-11-25T17:29:41Z")
1.0.0 On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote: > minimum\_master\_nodes still doesn't protect you from all possible failure > scenarios, see [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) > > What version are you running? > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran \<[yo...@my6sense.com](mailto:yo...@my6sense.com) > \\> wrote: > > > Hi Guys, > > > > We are running a 3 node cluster, and each node returns a different number > > of documents when issued a direct HTTP "\_count" call. > > > > The cluster holds about 150K documents and the differences range from > > 30~50 documents, but are still troubling. > > > > This shouldn't be a "split brain" problem, since we have set: > > discovery.zen.minimum\_master\_nodes: 2 > > We also have a "client" node, but since client nodes are eligible to be > > master, I understand that they shouldn't affect the master election process. > > > > Any Ideas about why and how this is happening? > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [November 25, 2014, 6:44pm UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/6 "2014-11-25T18:44:31Z")
I suggest you upgrade to 1.4 and try again - see > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran [yosi@my6sense.com](mailto:yosi@my6sense.com) wrote: > 1.0.0 > > On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote: > > > minimum\_master\_nodes still doesn't protect you from all possible failure > > scenarios, see [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) > > > > What version are you running? > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran [yo...@my6sense.com](mailto:yo...@my6sense.com) wrote: > > > > > Hi Guys, > > > > > > We are running a 3 node cluster, and each node returns a different > > > number of documents when issued a direct HTTP "\_count" call. > > > > > > The cluster holds about 150K documents and the differences range from > > > 30~50 documents, but are still troubling. > > > > > > This shouldn't be a "split brain" problem, since we have set: > > > discovery.zen.minimum\_master\_nodes: 2 > > > We also have a "client" node, but since client nodes are eligible to be > > > master, I understand that they shouldn't affect the master election process. > > > > > > Any Ideas about why and how this is happening? > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Ztx1Ze2dUNJno7MkD76PoDbuMp64o11sHsqKFMU1VeXaA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Ztx1Ze2dUNJno7MkD76PoDbuMp64o11sHsqKFMU1VeXaA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yosi\_Haran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yosi_haran/32/44914_2.png) [@Yosi\_Haran](https://discuss.elastic.co/u/Yosi_Haran)\ **Post date:** [November 26, 2014, 8:26am UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/7 "2014-11-26T08:26:44Z")
Alright, we'll try upgrading. Thanks 🙂 Meanwhile, any advice on how to fix an inconsistency once it is found? Is there an API to forcefully "sync" nodes, or at-least reindex from a specific node? On Tuesday, November 25, 2014 8:44:44 PM UTC+2, Itamar Syn-Hershko wrote: > I suggest you upgrade to 1.4 and try again - see > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/resiliency/current/index.html) > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran \<[yo...@my6sense.com](mailto:yo...@my6sense.com) > \\> wrote: > > > 1.0.0 > > > > On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > minimum\_master\_nodes still doesn't protect you from all possible failure > > > scenarios, see [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) > > > > > > What version are you running? > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > > > On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran [yo...@my6sense.com](mailto:yo...@my6sense.com) wrote: > > > > > > > Hi Guys, > > > > > > > > We are running a 3 node cluster, and each node returns a different > > > > number of documents when issued a direct HTTP "\_count" call. > > > > > > > > The cluster holds about 150K documents and the differences range from > > > > 30~50 documents, but are still troubling. > > > > > > > > This shouldn't be a "split brain" problem, since we have set: > > > > discovery.zen.minimum\_master\_nodes: 2 > > > > We also have a "client" node, but since client nodes are eligible to be > > > > master, I understand that they shouldn't affect the master election process. > > > > > > > > Any Ideas about why and how this is happening? > > > > > > > > Thanks! > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [November 26, 2014, 9:42am UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/8 "2014-11-26T09:42:58Z")
If this is replicas only, you should be able to set replica count to 0 and then after a while back to 2 again If this is sharded, then no, you'll have to reindex from scratch. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Wed, Nov 26, 2014 at 10:26 AM, Yosi Haran [yosi@my6sense.com](mailto:yosi@my6sense.com) wrote: > Alright, we'll try upgrading. Thanks 🙂 > > Meanwhile, any advice on how to fix an inconsistency once it is found? Is > there an API to forcefully "sync" nodes, or at-least reindex from a > specific node? > > On Tuesday, November 25, 2014 8:44:44 PM UTC+2, Itamar Syn-Hershko wrote: > > > I suggest you upgrade to 1.4 and try again - see > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > resiliency/current/index.html > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran [yo...@my6sense.com](mailto:yo...@my6sense.com) wrote: > > > > > 1.0.0 > > > > > > On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > minimum\_master\_nodes still doesn't protect you from all possible > > > > failure scenarios, see [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-) > > > > elasticsearch > > > > > > > > What version are you running? > > > > > > > > -- > > > > > > > > Itamar Syn-Hershko > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > Freelance Developer & Consultant > > > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > > > > > On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran [yo...@my6sense.com](mailto:yo...@my6sense.com) wrote: > > > > > > > > > Hi Guys, > > > > > > > > > > We are running a 3 node cluster, and each node returns a different > > > > > number of documents when issued a direct HTTP "\_count" call. > > > > > > > > > > The cluster holds about 150K documents and the differences range from > > > > > 30~50 documents, but are still troubling. > > > > > > > > > > This shouldn't be a "split brain" problem, since we have set: > > > > > discovery.zen.minimum\_master\_nodes: 2 > > > > > We also have a "client" node, but since client nodes are eligible to > > > > > be master, I understand that they shouldn't affect the master election > > > > > process. > > > > > > > > > > Any Ideas about why and how this is happening? > > > > > > > > > > Thanks! > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40goo > > > > > [glegroups.com](http://glegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuJuoVgzzc3bgguk5a9WmFkEaEPEv7DnbET-iQVREfriw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuJuoVgzzc3bgguk5a9WmFkEaEPEv7DnbET-iQVREfriw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:47am UTC](https://discuss.elastic.co/t/3-node-cluster-with-nodes-out-of-sync/20941/9 "2017-07-06T00:47:32Z")