# Data loss after network disconnect **URL:** \ **Category:** Elasticsearch\ **Created:** [September 10, 2014, 10:25am UTC](https://discuss.elastic.co/t/data-loss-after-network-disconnect/19709 "2014-09-10T10:25:51Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Israel\_Tsadok](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/israel_tsadok/32/1283_2.png) [@Israel\_Tsadok](https://discuss.elastic.co/u/Israel_Tsadok)\ **Post date:** [September 10, 2014, 10:25am UTC](https://discuss.elastic.co/t/data-loss-after-network-disconnect/19709/1 "2014-09-10T10:25:51Z")
A temporary network disconnect of the master node caused a torrent of RELOCATING shards, and then one shard remained UNASSIGNED and the cluster state was left red. looking inside the index directory for the shard on the disk, I found that it was empty (i.e., the \_state and translog dirs were there, but the index dir had no files). Looking at the log files, I see that the disconnect happened around 11:42:05, and a few minutes later I start seeing these error messages: _[2014-09-10 11:45:33,341]_[WARN][indices.cluster] [buzzilla\_data008] [el-2011-10-31-0000][0] failed to start shard _[2014-09-10 11:45:33,342]_[WARN][cluster.action.shard] [buzzilla\_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments\* file found in store(least\_used[rate\_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]] The relevant log files are at > data009 is the original master, data017 is the new master, and data008 is where I found the empty index directory. I had to delete the unassigned index from the cluster to return to green state. I am running Elasticsearch 1.2.1 in a 20 node cluster. How does this happen? What can I do to prevent this from happening again? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADdQPqz%3DXZDMbHC7zpWgEdaqW4Xy\_VkX7EgRwfXsrJjuoQ50SA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADdQPqz%3DXZDMbHC7zpWgEdaqW4Xy_VkX7EgRwfXsrJjuoQ50SA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [September 12, 2014, 5:22pm UTC](https://discuss.elastic.co/t/data-loss-after-network-disconnect/19709/2 "2014-09-12T17:22:34Z")
How were these nodes doing in terms of available heap space before the disconnects occurred? On Wednesday, September 10, 2014 6:26:19 AM UTC-4, Israel Tsadok wrote: > A temporary network disconnect of the master node caused a torrent of > RELOCATING shards, and then one shard remained UNASSIGNED and the cluster > state was left red. > > looking inside the index directory for the shard on the disk, I found that > it was empty (i.e., the \_state and translog dirs were there, but the index > dir had no files). > > Looking at the log files, I see that the disconnect happened around > 11:42:05, and a few minutes later I start seeing these error messages: > > _[2014-09-10 11:45:33,341]_[WARN][indices.cluster] > [buzzilla\_data008] [el-2011-10-31-0000][0] failed to start shard > _[2014-09-10 11:45:33,342]_[WARN][cluster.action.shard] > [buzzilla\_data008] [el-2011-10-31-0000][0] sending failed shard for > [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], > s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch > index version after copying it over]; nested: > IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated > for local recovery (post api), should exist, but doesn't, current files: > ]; nested: IndexNotFoundException[no segments\* file found in > store(least\_used[rate\_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), > type=MERGE, rate=20.0)]): files: ]; ]] > > The relevant log files are at > [Data loss on network disconnect ยท GitHub](https://gist.github.com/itsadok/97453743d6b211681aca) > data009 is the original master, data017 is the new master, and data008 is > where I found the empty index directory. > > I had to delete the unassigned index from the cluster to return to green > state. > I am running Elasticsearch 1.2.1 in a 20 node cluster. > > How does this happen? What can I do to prevent this from happening again? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/749729f6-daa1-470c-a835-d8f5dd85ad87%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/749729f6-daa1-470c-a835-d8f5dd85ad87%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:02am UTC](https://discuss.elastic.co/t/data-loss-after-network-disconnect/19709/3 "2017-07-06T01:02:42Z")