# UnavailableShardsException when adding document if number of nodes \< number of replicas **URL:** \ **Category:** Elasticsearch\ **Created:** [December 3, 2013, 7:23pm UTC](https://discuss.elastic.co/t/unavailableshardsexception-when-adding-document-if-number-of-nodes-number-of-replicas/14685 "2013-12-03T19:23:20Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Creg\_Bradley](https://avatars.discourse-cdn.com/v4/letter/c/d6d6ee/32.png) [@Creg\_Bradley](https://discuss.elastic.co/u/Creg_Bradley)\ **Post date:** [December 3, 2013, 7:23pm UTC](https://discuss.elastic.co/t/unavailableshardsexception-when-adding-document-if-number-of-nodes-number-of-replicas/14685/1 "2013-12-03T19:23:20Z")
Ordinarily, we would always have more nodes than replicas. However, in failure scenarios, if a number of nodes go down such that there are more replicas than nodes then we start getting the following exception: { "error":"UnavailableShardsException[[toomanyreplicas][4] [3] shardIt, [1] active : Timeout waiting for [1m], request: index {[toomanyreplicas][testdocument][QegOmpSTQZePZ1IaFeEmag], source[\n{\n "fileAttachment" : "ZmlnaHRpbmc="\n}\n]}]", "status":503 } This is running the following gist: > Is this a failure scenario that should be supported or do we need to try and disable document add if the cluster gets into this state? Detecting when we've reached this state seems like it will be problematic. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4aa45093-4251-42d4-a79a-2723e258e839%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4aa45093-4251-42d4-a79a-2723e258e839%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 4, 2013, 8:04am UTC](https://discuss.elastic.co/t/unavailableshardsexception-when-adding-document-if-number-of-nodes-number-of-replicas/14685/2 "2013-12-04T08:04:43Z")
Hey, can you describe the failure in more detail? Do you have failing all the nodes which hold a copy of specific shard, so you have data loss (you can check with the cluster state API, if the cluster state is red, as opposed to yellow or green, where you can index d). In case of a red cluster state those index requests are rejected anyway... but it might make sense to reflect that state in your application. --Alex On Tue, Dec 3, 2013 at 8:23 PM, Creg Bradley [creg.bradley@gmail.com](mailto:creg.bradley@gmail.com) wrote: > Ordinarily, we would always have more nodes than replicas. However, in > failure scenarios, if a number of nodes go down such that there are more > replicas than nodes then we start getting the following exception: > > { > "error":"UnavailableShardsException[[toomanyreplicas][4] [3] shardIt, > [1] active : Timeout waiting for [1m], request: index > {[toomanyreplicas][testdocument][QegOmpSTQZePZ1IaFeEmag], source[\n{\n > "fileAttachment" : "ZmlnaHRpbmc="\n}\n]}]", > "status":503 > } > > This is running the following gist: > [Repro UnavailableShardsException when number of replicas greater than number of nodes · GitHub](https://gist.github.com/creg/7775457) > > Is this a failure scenario that should be supported or do we need to try > and disable document add if the cluster gets into this state? Detecting > when we've reached this state seems like it will be problematic. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4aa45093-4251-42d4-a79a-2723e258e839%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4aa45093-4251-42d4-a79a-2723e258e839%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9UChY-8RV%3DQb4gaPLCzLHqSKc-nnFNTsDXgen37zb7sQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9UChY-8RV%3DQb4gaPLCzLHqSKc-nnFNTsDXgen37zb7sQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Creg\_Bradley](https://avatars.discourse-cdn.com/v4/letter/c/d6d6ee/32.png) [@Creg\_Bradley](https://discuss.elastic.co/u/Creg_Bradley)\ **Post date:** [December 5, 2013, 2:01am UTC](https://discuss.elastic.co/t/unavailableshardsexception-when-adding-document-if-number-of-nodes-number-of-replicas/14685/3 "2013-12-05T02:01:15Z")
Well, the Gist repros the problem. The result is that you have one node that has 5 master shards and 10 backups, 2 per master shard. No failure case is necessary. I only brought up failure modes because having multiple replicas of the same shard on the same node is stupid, but could happen if the right number of nodes fail. The cluster is green in kopf when in this state. On Wednesday, December 4, 2013 12:04:43 AM UTC-8, Alexander Reelsen wrote: > Hey, > > can you describe the failure in more detail? Do you have failing all the > nodes which hold a copy of specific shard, so you have data loss (you can > check with the cluster state API, if the cluster state is red, as opposed > to yellow or green, where you can index d). > > In case of a red cluster state those index requests are rejected anyway... > but it might make sense to reflect that state in your application. > > --Alex > > On Tue, Dec 3, 2013 at 8:23 PM, Creg Bradley \<[creg.b...@gmail.com](mailto:creg.b...@gmail.com)\ > > > wrote: > > > Ordinarily, we would always have more nodes than replicas. However, in > > failure scenarios, if a number of nodes go down such that there are more > > replicas than nodes then we start getting the following exception: > > > > { > > "error":"UnavailableShardsException[[toomanyreplicas][4] [3] shardIt, > > [1] active : Timeout waiting for [1m], request: index > > {[toomanyreplicas][testdocument][QegOmpSTQZePZ1IaFeEmag], source[\n{\n > > "fileAttachment" : "ZmlnaHRpbmc="\n}\n]}]", > > "status":503 > > } > > > > This is running the following gist: > > [Repro UnavailableShardsException when number of replicas greater than number of nodes · GitHub](https://gist.github.com/creg/7775457) > > > > Is this a failure scenario that should be supported or do we need to try > > and disable document add if the cluster gets into this state? Detecting > > when we've reached this state seems like it will be problematic. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4aa45093-4251-42d4-a79a-2723e258e839%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4aa45093-4251-42d4-a79a-2723e258e839%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5013cc0e-ad50-4848-a5de-37e4211a8d76%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5013cc0e-ad50-4848-a5de-37e4211a8d76%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:03am UTC](https://discuss.elastic.co/t/unavailableshardsexception-when-adding-document-if-number-of-nodes-number-of-replicas/14685/4 "2017-07-06T02:03:11Z")
Closed on July 6, 2017, 2:03am UTC