Network outage broke shards

secana · April 25, 2018, 9:41am

Hi, we've currently had a network outage and two of our ES clusters showed a strange behavior after the network was restored.

In the last three indices, 1 of the 4 shards were broken and we had to delete the indices to be able to write again into the ES clusters. We lost data of the last three days, which is very painful.

Because two clusters have shown this behavior, I don't think it's an accident. Has someone of you any idea why that happend and how we can prevent it from happening again?

Disclaimer: I'm not a ES pro, I just use it

Regards,
secana

dadoonet · April 25, 2018, 9:55am

Which version?
How many nodes?
Is minimum master nodes set?

secana · April 25, 2018, 10:01am

Hi,

ES Version 5.6.2
Nodes: 3
Minimal master nodes: Not set in elasticsearch.yml

dadoonet · April 25, 2018, 10:14am

Minimal master nodes: Not set in elasticsearch.yml

That's most likely your problem here.

secana · April 25, 2018, 10:17am

Thx for the very fast response. I'll set the minimal master nodes and hopefully the error doesn't occur the next time we have a network outage

Christian_Dahlqvist · April 25, 2018, 10:39am

Make sure that you follow these guidelines when you set it.

system · May 23, 2018, 10:40am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.