Network outage broke shards

Elastic Stack Elasticsearch
1

Hi, we've currently had a network outage and two of our ES clusters showed a strange behavior after the network was restored.

In the last three indices, 1 of the 4 shards were broken and we had to delete the indices to be able to write again into the ES clusters. We lost data of the last three days, which is very painful.

Because two clusters have shown this behavior, I don't think it's an accident. Has someone of you any idea why that happend and how we can prevent it from happening again?

Disclaimer: I'm not a ES pro, I just use it :smiley:

Regards,
secana

2

Which version?
How many nodes?
Is minimum master nodes set?

3

Hi,

ES Version 5.6.2
Nodes: 3
Minimal master nodes: Not set in elasticsearch.yml

4

Minimal master nodes: Not set in elasticsearch.yml

That's most likely your problem here.

5

Thx for the very fast response. I'll set the minimal master nodes and hopefully the error doesn't occur the next time we have a network outage

6

Make sure that you follow these guidelines when you set it.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.