# Network outage broke shards

**URL:** <https://discuss.elastic.co/t/network-outage-broke-shards/129446>\
**Category:** Elasticsearch\
**Created:** [April 25, 2018, 9:41am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446 "2018-04-25T09:41:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![secana](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@secana](https://discuss.elastic.co/u/secana)\
**Post date:** [April 25, 2018, 9:41am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446/1 "2018-04-25T09:41:46Z")

</div>

Hi, we've currently had a network outage and two of our ES clusters showed a strange behavior after the network was restored.

In the last three indices, 1 of the 4 shards were broken and we had to delete the indices to be able to write again into the ES clusters. We lost data of the last three days, which is very painful.

Because two clusters have shown this behavior, I don't think it's an accident. Has someone of you any idea why that happend and how we can prevent it from happening again?

Disclaimer: I'm not a ES pro, I just use it 😃

Regards,  
secana

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 25, 2018, 9:55am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446/2 "2018-04-25T09:55:54Z")

</div>

Which version?  
How many nodes?  
Is minimum master nodes set?

---

<div class="post-metadata">

**Author:** ![secana](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@secana](https://discuss.elastic.co/u/secana)\
**Post date:** [April 25, 2018, 10:01am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446/3 "2018-04-25T10:01:46Z")

</div>

Hi,

ES Version 5.6.2  
Nodes: 3  
Minimal master nodes: Not set in elasticsearch.yml

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 25, 2018, 10:14am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446/4 "2018-04-25T10:14:26Z")

</div>

> Minimal master nodes: Not set in elasticsearch.yml

That's most likely your problem here.

---

<div class="post-metadata">

**Author:** ![secana](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@secana](https://discuss.elastic.co/u/secana)\
**Post date:** [April 25, 2018, 10:17am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446/5 "2018-04-25T10:17:12Z")

</div>

Thx for the very fast response. I'll set the minimal master nodes and hopefully the error doesn't occur the next time we have a network outage

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 25, 2018, 10:39am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446/6 "2018-04-25T10:39:58Z")

</div>

Make sure that you follow [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-node.html#split-brain) when you set it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 23, 2018, 10:40am UTC](https://discuss.elastic.co/t/network-outage-broke-shards/129446/7 "2018-05-23T10:40:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.