# Detecting Data Loss **URL:** \ **Category:** Elasticsearch\ **Created:** [December 22, 2017, 1:43pm UTC](https://discuss.elastic.co/t/detecting-data-loss/113010 "2017-12-22T13:43:30Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Jeff\_Bolle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeff_bolle/32/52060_2.png) [@Jeff\_Bolle](https://discuss.elastic.co/u/Jeff_Bolle)\ **Post date:** [December 22, 2017, 1:43pm UTC](https://discuss.elastic.co/t/detecting-data-loss/113010/1 "2017-12-22T13:43:30Z")
Last night we lost 3 nodes in our cluster. From the logging I have it looks like they died sequentially over the course of about 2 hours. The cause of why they became unresponsive is not yet known. I was able to SSH into the machines this morning, but unable to restart the elasticsearch service (they were no longer seen as part of the cluster) or even run ps -aux. The nodes all have ephemeral disks (Google Cloud). When I rebooted the nodes the amount of free space on each of the disks was substantially higher than before, but the disks were not blank, and I didn't stop the node, so I would have expected the disks to remain intact. What I'd like help with is understanding how I can see if we lost data due to the conditions. I could restore the indicies from our backups, but I'd like to know if there is a more straightforward way to tell if shards / segments were deleted and data was lost (without having to remember how many docs I should have in each of my indicies). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 23, 2017, 7:25am UTC](https://discuss.elastic.co/t/detecting-data-loss/113010/2 "2017-12-23T07:25:49Z")
> [@Jeff\_Bolle](#): > > without having to remember how many docs I should have in each of my indicies The easiest way would be to have a X-Pack basic license with Monitoring to a secondary cluster, then you could just look at the stats. Without that, you are flying blind. ---
**Author:** ![Jeff\_Bolle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeff_bolle/32/52060_2.png) [@Jeff\_Bolle](https://discuss.elastic.co/u/Jeff_Bolle)\ **Post date:** [December 24, 2017, 2:45am UTC](https://discuss.elastic.co/t/detecting-data-loss/113010/3 "2017-12-24T02:45:34Z")
I'm saving logs off the box and aggregating them. What log line am I looking for? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 24, 2017, 5:14am UTC](https://discuss.elastic.co/t/detecting-data-loss/113010/4 "2017-12-24T05:14:55Z")
We don't log this sort of thing. ---
**Author:** ![Jeff\_Bolle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeff_bolle/32/52060_2.png) [@Jeff\_Bolle](https://discuss.elastic.co/u/Jeff_Bolle)\ **Post date:** [December 24, 2017, 1:40pm UTC](https://discuss.elastic.co/t/detecting-data-loss/113010/5 "2017-12-24T13:40:43Z")
What about if, during startup, a shard that was previously on the node is missing or corrupted? Any sort of indication that there were shenanigans at the FS level beneath elastic? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 24, 2017, 8:30pm UTC](https://discuss.elastic.co/t/detecting-data-loss/113010/6 "2017-12-24T20:30:59Z")
Then it will log that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 21, 2018, 8:31pm UTC](https://discuss.elastic.co/t/detecting-data-loss/113010/7 "2018-01-21T20:31:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.