# How to handle system failures in Elasticsearch cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [January 13, 2015, 9:22pm UTC](https://discuss.elastic.co/t/how-to-handle-system-failures-in-elasticsearch-cluster/21616 "2015-01-13T21:22:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ddp](https://avatars.discourse-cdn.com/v4/letter/d/f6c823/32.png) [@ddp](https://discuss.elastic.co/u/ddp)\ **Post date:** [January 13, 2015, 9:22pm UTC](https://discuss.elastic.co/t/how-to-handle-system-failures-in-elasticsearch-cluster/21616/1 "2015-01-13T21:22:51Z")
Hi, Yesterday, hard disks on one of our node went bad and we have to bring down the physical machine which were running another 2 nodes of elasticsearch. We have hourly index with replication 2 and 50 shards per index. Each shards is currently 5 - 6 GB in size. It is more than 24 hrs and cluster is still trying to assign unassign shards. During this RED status our search is broken. Any recommandation how to handle such situations ? Darsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 13, 2015, 10:12pm UTC](https://discuss.elastic.co/t/how-to-handle-system-failures-in-elasticsearch-cluster/21616/2 "2015-01-13T22:12:23Z")
How many nodes did/do you have? What do your logs show? You should look at using > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. if you are running multiple nodes per physical machine. On 14 January 2015 at 10:22, Darsh [darsh.patil@gmail.com](mailto:darsh.patil@gmail.com) wrote: > Hi, > > Yesterday, hard disks on one of our node went bad and we have to bring > down the physical machine which were running another 2 nodes of > elasticsearch. We have hourly index with replication 2 and 50 shards per > index. Each shards is currently 5 - 6 GB in size. It is more than 24 hrs > and cluster is still trying to assign unassign shards. During this RED > status our search is broken. Any recommandation how to handle such > situations ? > > Darsh > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP\_Oe7Kxn7tDXtiQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP_Oe7Kxn7tDXtiQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![ddp](https://avatars.discourse-cdn.com/v4/letter/d/f6c823/32.png) [@ddp](https://discuss.elastic.co/u/ddp)\ **Post date:** [January 13, 2015, 11:51pm UTC](https://discuss.elastic.co/t/how-to-handle-system-failures-in-elasticsearch-cluster/21616/3 "2015-01-13T23:51:52Z")
Hi Mark, Thank you for your reply. Here is our cluster info 40 Physical machines with 200 GB RAM Each machine has 3 data nodes of ES with 30 GB RAM so total 120 data nodes. 5 dedicated master nodes. We are using 3_2 RAID and 2_2 RAID on each physical machine, I didn't find much in logs other than logs related to initializing shards We do have cluster.routing.allocation.same\_shard.host: true but nothing related to rack aweareness.Something we will look into it. On Tue, Jan 13, 2015 at 2:12 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > How many nodes did/do you have? What do your logs show? > > You should look at using > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-cluster.html#allocation-awareness) > if you are running multiple nodes per physical machine. > > On 14 January 2015 at 10:22, Darsh [darsh.patil@gmail.com](mailto:darsh.patil@gmail.com) wrote: > > > Hi, > > > > Yesterday, hard disks on one of our node went bad and we have to bring > > down the physical machine which were running another 2 nodes of > > elasticsearch. We have hourly index with replication 2 and 50 shards per > > index. Each shards is currently 5 - 6 GB in size. It is more than 24 hrs > > and cluster is still trying to assign unassign shards. During this RED > > status our search is broken. Any recommandation how to handle such > > situations ? > > > > Darsh > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP\_Oe7Kxn7tDXtiQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP_Oe7Kxn7tDXtiQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP\_Oe7Kxn7tDXtiQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP_Oe7Kxn7tDXtiQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Thanks Darsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJpS\_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJpS_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 14, 2015, 12:58am UTC](https://discuss.elastic.co/t/how-to-handle-system-failures-in-elasticsearch-cluster/21616/4 "2015-01-14T00:58:46Z")
What version of ES and java are you on? Is your cluster still red? Check the \_cat/allocation, \_cat/indices and \_cat/recovery endpoints for info on the status of things. On 14 January 2015 at 12:51, Darsh [darsh.patil@gmail.com](mailto:darsh.patil@gmail.com) wrote: > Hi Mark, > > Thank you for your reply. Here is our cluster info > > 40 Physical machines with 200 GB RAM > Each machine has 3 data nodes of ES with 30 GB RAM so total 120 data nodes. > 5 dedicated master nodes. > We are using 3_2 RAID and 2_2 RAID on each physical machine, > I didn't find much in logs other than logs related to initializing shards > We do have cluster.routing.allocation.same\_shard.host: true but nothing > related to rack aweareness.Something we will look into it. > > On Tue, Jan 13, 2015 at 2:12 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > > > How many nodes did/do you have? What do your logs show? > > > > You should look at using > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-cluster.html#allocation-awareness) > > if you are running multiple nodes per physical machine. > > > > On 14 January 2015 at 10:22, Darsh [darsh.patil@gmail.com](mailto:darsh.patil@gmail.com) wrote: > > > > > Hi, > > > > > > Yesterday, hard disks on one of our node went bad and we have to bring > > > down the physical machine which were running another 2 nodes of > > > elasticsearch. We have hourly index with replication 2 and 50 shards per > > > index. Each shards is currently 5 - 6 GB in size. It is more than 24 hrs > > > and cluster is still trying to assign unassign shards. During this RED > > > status our search is broken. Any recommandation how to handle such > > > situations ? > > > > > > Darsh > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP\_Oe7Kxn7tDXtiQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP_Oe7Kxn7tDXtiQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP\_Oe7Kxn7tDXtiQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP_Oe7Kxn7tDXtiQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Thanks > > Darsh > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAJpS\_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJpS_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAJpS\_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJpS_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Pq034fh7wa0U\_uECOGQ5UN-%3DWJ-XdP7RPSCxqAZYrRQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Pq034fh7wa0U_uECOGQ5UN-%3DWJ-XdP7RPSCxqAZYrRQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![ddp](https://avatars.discourse-cdn.com/v4/letter/d/f6c823/32.png) [@ddp](https://discuss.elastic.co/u/ddp)\ **Post date:** [January 14, 2015, 1:31am UTC](https://discuss.elastic.co/t/how-to-handle-system-failures-in-elasticsearch-cluster/21616/5 "2015-01-14T01:31:06Z")
We are using 1.4.1 version of ES. Cluster is in green status now. I think these settings as you pointed out will help. Since our cluster size is huge i think default 2 will be very slow. cluster.routing.allocation.cluster\_concurrent\_rebalance cluster.routing.allocation.node\_concurrent\_recoveries On Tue, Jan 13, 2015 at 4:58 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > What version of ES and java are you on? > Is your cluster still red? Check the \_cat/allocation, \_cat/indices and > \_cat/recovery endpoints for info on the status of things. > > On 14 January 2015 at 12:51, Darsh [darsh.patil@gmail.com](mailto:darsh.patil@gmail.com) wrote: > > > Hi Mark, > > > > Thank you for your reply. Here is our cluster info > > > > 40 Physical machines with 200 GB RAM > > Each machine has 3 data nodes of ES with 30 GB RAM so total 120 data > > nodes. > > 5 dedicated master nodes. > > We are using 3_2 RAID and 2_2 RAID on each physical machine, > > I didn't find much in logs other than logs related to initializing > > shards > > We do have cluster.routing.allocation.same\_shard.host: true but nothing > > related to rack aweareness.Something we will look into it. > > > > On Tue, Jan 13, 2015 at 2:12 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) > > wrote: > > > > > How many nodes did/do you have? What do your logs show? > > > > > > You should look at using > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-cluster.html#allocation-awareness) > > > if you are running multiple nodes per physical machine. > > > > > > On 14 January 2015 at 10:22, Darsh [darsh.patil@gmail.com](mailto:darsh.patil@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > Yesterday, hard disks on one of our node went bad and we have to bring > > > > down the physical machine which were running another 2 nodes of > > > > elasticsearch. We have hourly index with replication 2 and 50 shards per > > > > index. Each shards is currently 5 - 6 GB in size. It is more than 24 hrs > > > > and cluster is still trying to assign unassign shards. During this RED > > > > status our search is broken. Any recommandation how to handle such > > > > situations ? > > > > > > > > Darsh > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85a4986e-e5bb-403f-95f0-80cd4be8287e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP\_Oe7Kxn7tDXtiQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP_Oe7Kxn7tDXtiQ%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP\_Oe7Kxn7tDXtiQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_bgzLRvs2SL3oFCS%3DXeRPcpH3Tb6aP_Oe7Kxn7tDXtiQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Thanks > > > > Darsh > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAJpS\_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJpS_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAJpS\_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJpS_Sok2n7pZNR%3DKM0ur1YrhN9YeYCbyiXv1%2BG2Ywdh%3DEYtVg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Yv5kSX2baa8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Pq034fh7wa0U\_uECOGQ5UN-%3DWJ-XdP7RPSCxqAZYrRQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Pq034fh7wa0U_uECOGQ5UN-%3DWJ-XdP7RPSCxqAZYrRQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Pq034fh7wa0U\_uECOGQ5UN-%3DWJ-XdP7RPSCxqAZYrRQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Pq034fh7wa0U_uECOGQ5UN-%3DWJ-XdP7RPSCxqAZYrRQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Thanks Darsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJpS\_So-1Fuv5-k2hPb\_%3Dc0-41Rv3UiOi3Zk4qX9Ly4apCGKww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJpS_So-1Fuv5-k2hPb_%3Dc0-41Rv3UiOi3Zk4qX9Ly4apCGKww%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/how-to-handle-system-failures-in-elasticsearch-cluster/21616/6 "2017-07-06T00:39:09Z")