A few days ago, our cluster started reporting that the vast majority of our
shards were unassigned. I found this odd, and even over the weekend there
were 0 relocating or initializing shards. This morning, there are a few
initializing shards, but unassigned has been sitting around 260 for the
last 5 days. I tried restarting the cluster (which has usually been the
quickest way to shock it back into initializing shards quickly), but no
matter what I try, I can't get them to load. As of yet, I've found nothing
in the logs.
To add to the strangeness, some nodes seem to disagree about who's in the
cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
The master (and most nodes) report that there are 9 machines in the
cluster, though the missing nodes report that all 12 are present. I've
tried stopping and starting elasticsearch on each of the missing machines,
but that hasn't helped the situation. Like before, I haven't found anything
useful in the logs yet.
This cluster has been fairly reliable in general, so I'm hoping that this
looks very symptomatic of a particular issue to someone else.
To add to the strangeness, some nodes seem to disagree about who's in the
cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
The master (and most nodes) report that there are 9 machines in the
cluster, though the missing nodes report that all 12 are present. I've
tried stopping and starting elasticsearch on each of the missing machines,
but that hasn't helped the situation. Like before, I haven't found anything
useful in the logs yet.
This smells a lot like an artifact of a split brain condition. Do you use
the 'minimum_master_nodes' property? (I forget now which ES 0.19 version
this came out with, 0.19.3 may be missing it).
I would iterate over each of the machines and grab the output of the
master_node property, something like:
Replace the 3 hosts in that example for loop with your host list and the
output SHOULD consistently report a single master node Identifier string.
there should be only one unique value, if there's 2 or more, you've had a
split brain and the cluster is disagreeing about who is the master.
More recent ES versions have better 'rejoining' behaviours in this case,
and if you're not using the miminum_master_nodes, I would really recommend
it.
I agree with Paul. Setting minimum_master_nodes to 7 (it has to be more
than a half of the cluster) and upgrading elasticsearch to a more recent
version of elasticsearch should help. There were
significant improvements in cluster resiliency to this type of issues,
especially in 0.19.5.
On Tuesday, November 6, 2012 5:44:23 PM UTC-5, tallpsmith wrote:
To add to the strangeness, some nodes seem to disagree about who's in the
cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
The master (and most nodes) report that there are 9 machines in the
cluster, though the missing nodes report that all 12 are present. I've
tried stopping and starting elasticsearch on each of the missing machines,
but that hasn't helped the situation. Like before, I haven't found anything
useful in the logs yet.
This smells a lot like an artifact of a split brain condition. Do you use
the 'minimum_master_nodes' property? (I forget now which ES 0.19 version
this came out with, 0.19.3 may be missing it).
I would iterate over each of the machines and grab the output of the
master_node property, something like:
Replace the 3 hosts in that example for loop with your host list and the
output SHOULD consistently report a single master node Identifier string.
there should be only one unique value, if there's 2 or more, you've had a
split brain and the cluster is disagreeing about who is the master.
More recent ES versions have better 'rejoining' behaviours in this case,
and if you're not using the miminum_master_nodes, I would really recommend
it.
I got not only two distinct values there, but four; I'm going to see about
upgrading elasticsearch and using minimum_master_nodes as you'd suggested.
Thanks!
I'm still concerned about the number of unassigned shards, but I'll see
what happens once the upgrade's complete.
On Wednesday, November 7, 2012 4:32:04 AM UTC-8, Igor Motov wrote:
I agree with Paul. Setting minimum_master_nodes to 7 (it has to be more
than a half of the cluster) and upgrading elasticsearch to a more recent
version of elasticsearch should help. There were
significant improvements in cluster resiliency to this type of issues,
especially in 0.19.5.
On Tuesday, November 6, 2012 5:44:23 PM UTC-5, tallpsmith wrote:
To add to the strangeness, some nodes seem to disagree about who's in the
cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
The master (and most nodes) report that there are 9 machines in the
cluster, though the missing nodes report that all 12 are present. I've
tried stopping and starting elasticsearch on each of the missing machines,
but that hasn't helped the situation. Like before, I haven't found anything
useful in the logs yet.
This smells a lot like an artifact of a split brain condition. Do you
use the 'minimum_master_nodes' property? (I forget now which ES 0.19
version this came out with, 0.19.3 may be missing it).
I would iterate over each of the machines and grab the output of the
master_node property, something like:
Replace the 3 hosts in that example for loop with your host list and the
output SHOULD consistently report a single master node Identifier string.
there should be only one unique value, if there's 2 or more, you've had a
split brain and the cluster is disagreeing about who is the master.
More recent ES versions have better 'rejoining' behaviours in this case,
and if you're not using the miminum_master_nodes, I would really recommend
it.
As far as the unassigned shards go, it turned out that I had some indexes
that came back after being deleted. Having re-deleted those, things are
much better now.
Thanks again!
On Wednesday, November 7, 2012 8:20:14 AM UTC-8, Dan Lecocq wrote:
Thanks for the replies!
I got not only two distinct values there, but four; I'm going to see about
upgrading elasticsearch and using minimum_master_nodes as you'd suggested.
Thanks!
I'm still concerned about the number of unassigned shards, but I'll see
what happens once the upgrade's complete.
On Wednesday, November 7, 2012 4:32:04 AM UTC-8, Igor Motov wrote:
I agree with Paul. Setting minimum_master_nodes to 7 (it has to be more
than a half of the cluster) and upgrading elasticsearch to a more recent
version of elasticsearch should help. There were
significant improvements in cluster resiliency to this type of issues,
especially in 0.19.5.
On Tuesday, November 6, 2012 5:44:23 PM UTC-5, tallpsmith wrote:
To add to the strangeness, some nodes seem to disagree about who's in
the cluster. We've got a 12-node cluster running 0.19.3 on m1.xlarge instances.
The master (and most nodes) report that there are 9 machines in the
cluster, though the missing nodes report that all 12 are present. I've
tried stopping and starting elasticsearch on each of the missing machines,
but that hasn't helped the situation. Like before, I haven't found anything
useful in the logs yet.
This smells a lot like an artifact of a split brain condition. Do you
use the 'minimum_master_nodes' property? (I forget now which ES 0.19
version this came out with, 0.19.3 may be missing it).
I would iterate over each of the machines and grab the output of the
master_node property, something like:
Replace the 3 hosts in that example for loop with your host list and the
output SHOULD consistently report a single master node Identifier string.
there should be only one unique value, if there's 2 or more, you've had a
split brain and the cluster is disagreeing about who is the master.
More recent ES versions have better 'rejoining' behaviours in this case,
and if you're not using the miminum_master_nodes, I would really recommend
it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.