# Nodes Disagree About Cluster Membership, Stalled Unassigned Shards
**URL:** https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584
**Category:** Elasticsearch
**Created:** [November 6, 2012, 5:12pm UTC](https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584 "2012-11-06T17:12:23Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)
#### Post date: [November 6, 2012, 5:12pm UTC](https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584/1 "2012-11-06T17:12:23Z")
Hey all,
A few days ago, our cluster started reporting that the vast majority of our
shards were unassigned. I found this odd, and even over the weekend there
were 0 relocating or initializing shards. This morning, there are a few
initializing shards, but unassigned has been sitting around 260 for the
last 5 days. I tried restarting the cluster (which has usually been the
quickest way to shock it back into initializing shards quickly), but no
matter what I try, I can't get them to load. As of yet, I've found nothing
in the logs.
To add to the strangeness, some nodes seem to disagree about who's in the
cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
The master (and most nodes) report that there are 9 machines in the
cluster, though the missing nodes report that all 12 are present. I've
tried stopping and starting elasticsearch on each of the missing machines,
but that hasn't helped the situation. Like before, I haven't found anything
useful in the logs yet.
This cluster has been fairly reliable in general, so I'm hoping that this
looks very symptomatic of a particular issue to someone else.
Here's the gist of what the instances are
reporting: [https://gist.github.com/4026084](https://gist.github.com/4026084)
--
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [November 6, 2012, 10:44pm UTC](https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584/2 "2012-11-06T22:44:20Z")
> To add to the strangeness, some nodes seem to disagree about who's in the
> cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
> The master (and most nodes) report that there are 9 machines in the
> cluster, though the missing nodes report that all 12 are present. I've
> tried stopping and starting elasticsearch on each of the missing machines,
> but that hasn't helped the situation. Like before, I haven't found anything
> useful in the logs yet.
This smells a lot like an artifact of a split brain condition. Do you use
the 'minimum\_master\_nodes' property? (I forget now which ES 0.19 version
this came out with, 0.19.3 may be missing it).
I would iterate over each of the machines and grab the output of the
master\_node property, something like:
for host in index1.qa.acx index2.qa.acx index3.qa.acx; do curl -XGET
http://:9200$host/\_cluster/state?pretty=1
| fgrep master; done
Replace the 3 hosts in that example for loop with your host list and the
output SHOULD consistently report a single master node Identifier string.
there should be only one unique value, if there's 2 or more, you've had a
split brain and the cluster is disagreeing about who is the master.
More recent ES versions have better 'rejoining' behaviours in this case,
and if you're not using the miminum\_master\_nodes, I would really recommend
it.
cheers,
Paul Smith
--
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [November 7, 2012, 12:32pm UTC](https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584/3 "2012-11-07T12:32:04Z")
I agree with Paul. Setting minimum\_master\_nodes to 7 (it has to be more
than a half of the cluster) and upgrading elasticsearch to a more recent
version of elasticsearch should help. There were
significant improvements in cluster resiliency to this type of issues,
especially in 0.19.5.
On Tuesday, November 6, 2012 5:44:23 PM UTC-5, tallpsmith wrote:
> To add to the strangeness, some nodes seem to disagree about who's in the
>
> > cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
> > The master (and most nodes) report that there are 9 machines in the
> > cluster, though the missing nodes report that all 12 are present. I've
> > tried stopping and starting elasticsearch on each of the missing machines,
> > but that hasn't helped the situation. Like before, I haven't found anything
> > useful in the logs yet.
>
> This smells a lot like an artifact of a split brain condition. Do you use
> the 'minimum\_master\_nodes' property? (I forget now which ES 0.19 version
> this came out with, 0.19.3 may be missing it).
>
> I would iterate over each of the machines and grab the output of the
> master\_node property, something like:
>
> for host in index1.qa.acx index2.qa.acx index3.qa.acx; do curl -XGET
> http://:9200$host/\_cluster/state?pretty=1 | fgrep master; done
>
> Replace the 3 hosts in that example for loop with your host list and the
> output SHOULD consistently report a single master node Identifier string.
> there should be only one unique value, if there's 2 or more, you've had a
> split brain and the cluster is disagreeing about who is the master.
>
> More recent ES versions have better 'rejoining' behaviours in this case,
> and if you're not using the miminum\_master\_nodes, I would really recommend
> it.
>
> cheers,
>
> Paul Smith
--
---
### Author:  [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)
#### Post date: [November 7, 2012, 4:20pm UTC](https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584/4 "2012-11-07T16:20:13Z")
Thanks for the replies!
I got not only two distinct values there, but four; I'm going to see about
upgrading elasticsearch and using minimum\_master\_nodes as you'd suggested.
Thanks!
I'm still concerned about the number of unassigned shards, but I'll see
what happens once the upgrade's complete.
On Wednesday, November 7, 2012 4:32:04 AM UTC-8, Igor Motov wrote:
> I agree with Paul. Setting minimum\_master\_nodes to 7 (it has to be more
> than a half of the cluster) and upgrading elasticsearch to a more recent
> version of elasticsearch should help. There were
> significant improvements in cluster resiliency to this type of issues,
> especially in 0.19.5.
>
> On Tuesday, November 6, 2012 5:44:23 PM UTC-5, tallpsmith wrote:
>
> > To add to the strangeness, some nodes seem to disagree about who's in the
> >
> > > cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
> > > The master (and most nodes) report that there are 9 machines in the
> > > cluster, though the missing nodes report that all 12 are present. I've
> > > tried stopping and starting elasticsearch on each of the missing machines,
> > > but that hasn't helped the situation. Like before, I haven't found anything
> > > useful in the logs yet.
> >
> > This smells a lot like an artifact of a split brain condition. Do you
> > use the 'minimum\_master\_nodes' property? (I forget now which ES 0.19
> > version this came out with, 0.19.3 may be missing it).
> >
> > I would iterate over each of the machines and grab the output of the
> > master\_node property, something like:
> >
> > for host in index1.qa.acx index2.qa.acx index3.qa.acx; do curl -XGET
> > http://:9200$host/\_cluster/state?pretty=1 | fgrep master; done
> >
> > Replace the 3 hosts in that example for loop with your host list and the
> > output SHOULD consistently report a single master node Identifier string.
> > there should be only one unique value, if there's 2 or more, you've had a
> > split brain and the cluster is disagreeing about who is the master.
> >
> > More recent ES versions have better 'rejoining' behaviours in this case,
> > and if you're not using the miminum\_master\_nodes, I would really recommend
> > it.
> >
> > cheers,
> >
> > Paul Smith
--
---
### Author:  [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)
#### Post date: [November 7, 2012, 5:51pm UTC](https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584/5 "2012-11-07T17:51:28Z")
As far as the unassigned shards go, it turned out that I had some indexes
that came back after being deleted. Having re-deleted those, things are
much better now.
Thanks again!
On Wednesday, November 7, 2012 8:20:14 AM UTC-8, Dan Lecocq wrote:
> Thanks for the replies!
>
> I got not only two distinct values there, but four; I'm going to see about
> upgrading elasticsearch and using minimum\_master\_nodes as you'd suggested.
> Thanks!
>
> I'm still concerned about the number of unassigned shards, but I'll see
> what happens once the upgrade's complete.
>
> On Wednesday, November 7, 2012 4:32:04 AM UTC-8, Igor Motov wrote:
>
> > I agree with Paul. Setting minimum\_master\_nodes to 7 (it has to be more
> > than a half of the cluster) and upgrading elasticsearch to a more recent
> > version of elasticsearch should help. There were
> > significant improvements in cluster resiliency to this type of issues,
> > especially in 0.19.5.
> >
> > On Tuesday, November 6, 2012 5:44:23 PM UTC-5, tallpsmith wrote:
> >
> > > To add to the strangeness, some nodes seem to disagree about who's in
> > >
> > > > the cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
> > > > The master (and most nodes) report that there are 9 machines in the
> > > > cluster, though the missing nodes report that all 12 are present. I've
> > > > tried stopping and starting elasticsearch on each of the missing machines,
> > > > but that hasn't helped the situation. Like before, I haven't found anything
> > > > useful in the logs yet.
> > >
> > > This smells a lot like an artifact of a split brain condition. Do you
> > > use the 'minimum\_master\_nodes' property? (I forget now which ES 0.19
> > > version this came out with, 0.19.3 may be missing it).
> > >
> > > I would iterate over each of the machines and grab the output of the
> > > master\_node property, something like:
> > >
> > > for host in index1.qa.acx index2.qa.acx index3.qa.acx; do curl -XGET
> > > http://:9200$host/\_cluster/state?pretty=1 | fgrep master; done
> > >
> > > Replace the 3 hosts in that example for loop with your host list and the
> > > output SHOULD consistently report a single master node Identifier string.
> > > there should be only one unique value, if there's 2 or more, you've had a
> > > split brain and the cluster is disagreeing about who is the master.
> > >
> > > More recent ES versions have better 'rejoining' behaviours in this case,
> > > and if you're not using the miminum\_master\_nodes, I would really recommend
> > > it.
> > >
> > > cheers,
> > >
> > > Paul Smith
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:05am UTC](https://discuss.elastic.co/t/nodes-disagree-about-cluster-membership-stalled-unassigned-shards/9584/6 "2017-07-06T03:05:40Z")