# AWS network issue and data loss **URL:** \ **Category:** Elasticsearch\ **Created:** [December 7, 2017, 8:18pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741 "2017-12-07T20:18:58Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![mario12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario12/32/29324_2.png) [@mario12](https://discuss.elastic.co/u/mario12)\ **Post date:** [December 7, 2017, 8:18pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/1 "2017-12-07T20:18:58Z")
We are using es 2.4 version. All our nodes uses AWS ephemeral disk. During the network connectivity issue there is a chance that nodes may drop out of the cluster. This may cause data loss. Is there a settings that can be used to prevent this situation? Our cluster is setup with zone awareness. So, the shards are spread evenly across 3 az's. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 7, 2017, 8:21pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/2 "2017-12-07T20:21:59Z")
Why would this cause data loss? ---
**Author:** ![mario12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario12/32/29324_2.png) [@mario12](https://discuss.elastic.co/u/mario12)\ **Post date:** [December 7, 2017, 9:40pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/3 "2017-12-07T21:40:00Z")
@warkolm When two nodes leaves at the same time and if it hosts primary and replica of the same shard. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 7, 2017, 9:44pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/4 "2017-12-07T21:44:28Z")
Then the best way to prevent that is to have 2 replicas. ---
**Author:** ![mario12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario12/32/29324_2.png) [@mario12](https://discuss.elastic.co/u/mario12)\ **Post date:** [December 7, 2017, 9:48pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/5 "2017-12-07T21:48:17Z")
Instead is there any settings like wait for x minutes before reallocating shards to another node? Will this prevent data loss? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 7, 2017, 9:49pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/6 "2017-12-07T21:49:05Z")
If the nodes that contain both primary and replica are gone, you cannot reallocate the data. ---
**Author:** ![mario12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario12/32/29324_2.png) [@mario12](https://discuss.elastic.co/u/mario12)\ **Post date:** [December 7, 2017, 9:49pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/7 "2017-12-07T21:49:54Z")
In that scenario what happens when the nodes rejoin? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 7, 2017, 9:50pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/8 "2017-12-07T21:50:25Z")
Then things should return to green. ---
**Author:** ![mario12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario12/32/29324_2.png) [@mario12](https://discuss.elastic.co/u/mario12)\ **Post date:** [December 7, 2017, 9:52pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/9 "2017-12-07T21:52:10Z")
Will there be any data consistency issue after the nodes rejoins? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 7, 2017, 9:53pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/10 "2017-12-07T21:53:20Z")
There will be no state changes to these during that point at they are not part of the cluster. > [@mario12](#): > > We are using es 2.4 version You should really upgrade 🙂 ---
**Author:** ![mario12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario12/32/29324_2.png) [@mario12](https://discuss.elastic.co/u/mario12)\ **Post date:** [December 7, 2017, 9:54pm UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/11 "2017-12-07T21:54:58Z")
I agree we have to upgrade our es version. Thanks for your help on this. ---
**Author:** ![mujtabahussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mujtabahussain/32/17514_2.png) [@mujtabahussain](https://discuss.elastic.co/u/mujtabahussain)\ **Post date:** [December 8, 2017, 1:10am UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/12 "2017-12-08T01:10:04Z")
I would also recommend consistent and frequent backing up by using the [snapshot utility](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-snapshots.html). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 5, 2018, 1:10am UTC](https://discuss.elastic.co/t/aws-network-issue-and-data-loss/110741/13 "2018-01-05T01:10:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.