# Full cluster restart consistently fails to assign all shards
**URL:** https://discuss.elastic.co/t/full-cluster-restart-consistently-fails-to-assign-all-shards/16723
**Category:** Elasticsearch
**Created:** [March 31, 2014, 7:38pm UTC](https://discuss.elastic.co/t/full-cluster-restart-consistently-fails-to-assign-all-shards/16723 "2014-03-31T19:38:29Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Brent\_Reed](https://discuss.elastic.co/u/Brent_Reed)
#### Post date: [March 31, 2014, 7:38pm UTC](https://discuss.elastic.co/t/full-cluster-restart-consistently-fails-to-assign-all-shards/16723/1 "2014-03-31T19:38:29Z")
I am running ES 1.0.1 (and have also verified the same problem with 1.1.0).
I have a cluster of 9 nodes - 8 are http/data nodes and 1 is http/master
(this is a dev/test cluster so running with only one master). I create a
new index with 8 shards no replicas, and populate the index. Everything is
running great. Then I do a full cluster restart. \*When everything comes
back up, however, all looks perfect EXCEPT that every time (this is very
consistent) I have a single shard that doesn't get assigned... \*Yes,
gateway is set to local - I am using a completely stock config file with
the exception of pathing (data, logs, plugins) and cluster name.
I can't find any information on this to determine if it is expected
behavior (i hope not) or how to resolve it. I have systematically been
changing elasticsearch.yaml configs to see if anything helps fix, but
nothing seems to resolve the issue. I should note that when simulating a
production environment with rolling restarts, there is no issue. Still,
this just feels like incorrect behavior...
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 31, 2014, 8:03pm UTC](https://discuss.elastic.co/t/full-cluster-restart-consistently-fails-to-assign-all-shards/16723/2 "2014-03-31T20:03:03Z")
Is it a primary, replica? Is it in an initialising or relocating state?
Do the logs show anything?
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 1 April 2014 06:38, Brent Reed [brent.j.reed@gmail.com](mailto:brent.j.reed@gmail.com) wrote:
> I am running ES 1.0.1 (and have also verified the same problem with 1.1.0).
>
> I have a cluster of 9 nodes - 8 are http/data nodes and 1 is http/master
> (this is a dev/test cluster so running with only one master). I create a
> new index with 8 shards no replicas, and populate the index. Everything is
> running great. Then I do a full cluster restart. \*When everything comes
> back up, however, all looks perfect EXCEPT that every time (this is very
> consistent) I have a single shard that doesn't get assigned... \*Yes,
> gateway is set to local - I am using a completely stock config file with
> the exception of pathing (data, logs, plugins) and cluster name.
>
> I can't find any information on this to determine if it is expected
> behavior (i hope not) or how to resolve it. I have systematically been
> changing elasticsearch.yaml configs to see if anything helps fix, but
> nothing seems to resolve the issue. I should note that when simulating a
> production environment with rolling restarts, there is no issue. Still,
> this just feels like incorrect behavior...
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aUWcqSgonGa280QxEqMqhM9MMU7NThHvkNbMPLSkV9eQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aUWcqSgonGa280QxEqMqhM9MMU7NThHvkNbMPLSkV9eQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Brent\_Reed](https://discuss.elastic.co/u/Brent_Reed)
#### Post date: [March 31, 2014, 8:52pm UTC](https://discuss.elastic.co/t/full-cluster-restart-consistently-fails-to-assign-all-shards/16723/3 "2014-03-31T20:52:02Z")
It is a primary shard (I don't have any replicas on this particular test
cluster). I am seeing the following log entry that correlates with the
failure, but it doesn't tell me much...
_[2014-03-31 13:47:20,691][DEBUG][gateway.local] [http1]
[bjr02][1]: not allocating, number\_of\_allocated\_shards\_found [0],
required\_number [1]_
Here is a SS (head plugin) after index creation:
[https://lh3.googleusercontent.com/-kVkJAa9rdY8/UznTcd0sF\_I/AAAAAAABRxg/qmk6eUE7LGI/s1600/before\_cluster\_restart.gif](https://lh3.googleusercontent.com/-kVkJAa9rdY8/UznTcd0sF_I/AAAAAAABRxg/qmk6eUE7LGI/s1600/before_cluster_restart.gif)
and after restart:
[https://lh4.googleusercontent.com/-HnY3qHswsJM/UznVFpmIsAI/AAAAAAABRxs/XJY2WyvFWgs/s1600/after\_cluster\_restart.gif](https://lh4.googleusercontent.com/-HnY3qHswsJM/UznVFpmIsAI/AAAAAAABRxs/XJY2WyvFWgs/s1600/after_cluster_restart.gif)
A manual curl call to allocate shard 1 will successfully add it back into
the cluster fully intact and working (no data in this particular index but
have verified with actual data so this isn't an index corruption type
scenario).
On Monday, March 31, 2014 2:03:03 PM UTC-6, Mark Walkom wrote:
> Is it a primary, replica? Is it in an initialising or relocating state?
> Do the logs show anything?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
>
> On 1 April 2014 06:38, Brent Reed \<[brent....@gmail.com](mailto:brent....@gmail.com) \\>wrote:
>
> > I am running ES 1.0.1 (and have also verified the same problem with
> > 1.1.0).
> >
> > I have a cluster of 9 nodes - 8 are http/data nodes and 1 is http/master
> > (this is a dev/test cluster so running with only one master). I create a
> > new index with 8 shards no replicas, and populate the index. Everything is
> > running great. Then I do a full cluster restart. \*When everything
> > comes back up, however, all looks perfect EXCEPT that every time (this is
> > very consistent) I have a single shard that doesn't get assigned... \*Yes,
> > gateway is set to local - I am using a completely stock config file with
> > the exception of pathing (data, logs, plugins) and cluster name.
> >
> > I can't find any information on this to determine if it is expected
> > behavior (i hope not) or how to resolve it. I have systematically been
> > changing elasticsearch.yaml configs to see if anything helps fix, but
> > nothing seems to resolve the issue. I should note that when simulating a
> > production environment with rolling restarts, there is no issue. Still,
> > this just feels like incorrect behavior...
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1398e083-f978-4c33-9054-fa8ded0d754d%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bed70da6-a658-416a-8cd5-d47d2fef24b6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bed70da6-a658-416a-8cd5-d47d2fef24b6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Brent\_Reed](https://discuss.elastic.co/u/Brent_Reed)
#### Post date: [March 31, 2014, 9:05pm UTC](https://discuss.elastic.co/t/full-cluster-restart-consistently-fails-to-assign-all-shards/16723/4 "2014-03-31T21:05:39Z")
A little more digging into the error, it sure looks like elasticsearch is
getting confused/broken when trying to recover. My 'http' (master only)
nodes appear to be getting included in attempts to recover, resulting in
the error posted above...
Perhaps I am jumping to conclusions here, but if so, sure smells like a bug
to me - a master only node should not even be considered for recovery
efforts.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d60e6fc0-5bff-4ea0-b7d4-f2cea8dcac08%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d60e6fc0-5bff-4ea0-b7d4-f2cea8dcac08%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:39am UTC](https://discuss.elastic.co/t/full-cluster-restart-consistently-fails-to-assign-all-shards/16723/5 "2017-07-06T01:39:12Z")