3 Node Cluster With Nodes Out of Sync

Elastic Stack Elasticsearch
1

Hi Guys,

We are running a 3 node cluster, and each node returns a different number
of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from 30~50
documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to be
master, I understand that they shouldn't affect the master election process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

minimum_master_nodes still doesn't protect you from all possible failure
scenarios, see Jepsen: Elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran yosi@my6sense.com wrote:

Hi Guys,

We are running a 3 node cluster, and each node returns a different number
of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from
30~50 documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to be
master, I understand that they shouldn't affect the master election process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu6r2z39iT%2B%3Dy7b6Brez%2BhLz8davSRaq0UmMviCCqV_sQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks! I'll check out the link you sent and see if it'll help...

We're are running version 1.0.0:

"version": {
"number": "1.0.0",
"build_hash": "a46900e9c72c0a623d71b54016357d5f94c8ea32",
"build_timestamp": "2014-02-12T16:18:34Z",
"build_snapshot": false,
"lucene_version": "4.6"
}

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:

minimum_master_nodes still doesn't protect you from all possible failure
scenarios, see Jepsen: Elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran <yo...@my6sense.com
<javascript:>> wrote:

Hi Guys,

We are running a 3 node cluster, and each node returns a different number
of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from
30~50 documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to be
master, I understand that they shouldn't affect the master election process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ec97ea83-bfef-4831-8af0-9a7a86170124%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Thanks! I'll check out the link and let you know if it helped.

We are running version 1.0.0

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:

minimum_master_nodes still doesn't protect you from all possible failure
scenarios, see Jepsen: Elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran <yo...@my6sense.com
<javascript:>> wrote:

Hi Guys,

We are running a 3 node cluster, and each node returns a different number
of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from
30~50 documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to be
master, I understand that they shouldn't affect the master election process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5addc70b-b76c-46d5-b122-cf730827d677%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

1.0.0

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:

minimum_master_nodes still doesn't protect you from all possible failure
scenarios, see Jepsen: Elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran <yo...@my6sense.com
<javascript:>> wrote:

Hi Guys,

We are running a 3 node cluster, and each node returns a different number
of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from
30~50 documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to be
master, I understand that they shouldn't affect the master election process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

I suggest you upgrade to 1.4 and try again - see

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran yosi@my6sense.com wrote:

1.0.0

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:

minimum_master_nodes still doesn't protect you from all possible failure
scenarios, see Jepsen: Elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran yo...@my6sense.com wrote:

Hi Guys,

We are running a 3 node cluster, and each node returns a different
number of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from
30~50 documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to be
master, I understand that they shouldn't affect the master election process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Ztx1Ze2dUNJno7MkD76PoDbuMp64o11sHsqKFMU1VeXaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

Alright, we'll try upgrading. Thanks :slight_smile:

Meanwhile, any advice on how to fix an inconsistency once it is found? Is
there an API to forcefully "sync" nodes, or at-least reindex from a
specific node?

On Tuesday, November 25, 2014 8:44:44 PM UTC+2, Itamar Syn-Hershko wrote:

I suggest you upgrade to 1.4 and try again - see
Elasticsearch Platform — Find real-time answers at scale | Elastic

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran <yo...@my6sense.com
<javascript:>> wrote:

1.0.0

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:

minimum_master_nodes still doesn't protect you from all possible failure
scenarios, see Jepsen: Elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran yo...@my6sense.com wrote:

Hi Guys,

We are running a 3 node cluster, and each node returns a different
number of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from
30~50 documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to be
master, I understand that they shouldn't affect the master election process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

If this is replicas only, you should be able to set replica count to 0 and
then after a while back to 2 again

If this is sharded, then no, you'll have to reindex from scratch.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Nov 26, 2014 at 10:26 AM, Yosi Haran yosi@my6sense.com wrote:

Alright, we'll try upgrading. Thanks :slight_smile:

Meanwhile, any advice on how to fix an inconsistency once it is found? Is
there an API to forcefully "sync" nodes, or at-least reindex from a
specific node?

On Tuesday, November 25, 2014 8:44:44 PM UTC+2, Itamar Syn-Hershko wrote:

I suggest you upgrade to 1.4 and try again - see
Elasticsearch Platform — Find real-time answers at scale | Elastic
resiliency/current/index.html

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran yo...@my6sense.com wrote:

1.0.0

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:

minimum_master_nodes still doesn't protect you from all possible
failure scenarios, see Jepsen: Elasticsearch
elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran yo...@my6sense.com wrote:

Hi Guys,

We are running a 3 node cluster, and each node returns a different
number of documents when issued a direct HTTP "_count" call.

The cluster holds about 150K documents and the differences range from
30~50 documents, but are still troubling.

This shouldn't be a "split brain" problem, since we have set:
discovery.zen.minimum_master_nodes: 2
We also have a "client" node, but since client nodes are eligible to
be master, I understand that they shouldn't affect the master election
process.

Any Ideas about why and how this is happening?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuJuoVgzzc3bgguk5a9WmFkEaEPEv7DnbET-iQVREfriw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.