# Another missing data after restart topic **URL:** \ **Category:** Elasticsearch\ **Created:** [January 29, 2015, 12:29am UTC](https://discuss.elastic.co/t/another-missing-data-after-restart-topic/21889 "2015-01-29T00:29:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [January 29, 2015, 12:29am UTC](https://discuss.elastic.co/t/another-missing-data-after-restart-topic/21889/1 "2015-01-29T00:29:47Z")
I read through the past couple months worth of "missing data" topics on this list, and didn't really find anything to help. So, apologies if I missed something in them that is obvious. Yesterday evening I ran OS updates to fix the glibc vulnerability. That meant I restarted my 4 ES nodes. Today, I checked ElasticHQ, and a bunch of this months logstash indices are missing tons of data. One day went from over a million logs to 53. When restarting, I did not bother to shut ES down cleanly. As far as I know, though, that should not cause ES to delete data... My data is stored on nfs mounts, so I double checked that they had mounted correctly. And I went to the hassle of cleanly shutting them down, one at a time, and making sure the mount points themselves were empty. A couple weren't, but the data was from months ago. Not this month. In my logs for the past couple days I don't see anything that really pops out at me. Just that my curator generated snapshots are all failing for some reason or another. I don't think that has anything to do with the missing data, but here's an entry from one of my log files anyway. [2015-01-26 03:00:10,145][WARN][snapshots] > [log-elasticsearch-04] [[logstash-2014.12.31][3]] > [laneprodelk:curator-20150126110002] failed to create snapshot > org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: > [logstash-2014.12.31][3] Failed to perform snapshot (index files) > at > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:503) > at > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:139) > at > org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:86) > at > org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:818) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.FileNotFoundException: > /mounts/prod\_backup/laneprodelk/indices/logstash-2014.12.31/3/\_\_0 (No such > file or directory) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.(FileOutputStream.java:221) > at java.io.FileOutputStream.(FileOutputStream.java:171) > at > org.elasticsearch.common.blobstore.fs.FsBlobContainer.createOutput(FsBlobContainer.java:85) > at > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshotFile(BlobStoreIndexShardRepository.java:555) > at > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:501) > ... 6 more ES version info: { "status" : 200, "name" : "log-elasticsearch-02", "cluster\_name" : "logstash-webservices", "version" : { "number" : "1.4.2", "build\_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c", "build\_timestamp" : "2014-12-16T14:11:12Z", "build\_snapshot" : false, "lucene\_version" : "4.10.2" }, "tagline" : "You Know, for Search" } I'm running the nodes on Ubuntu 12.04 64bit. I do have curator deleting data older than 365 days every night. It also closes indices older than 90 days, and optimizes them if they're older than 2 days. curator is version 2.0.1. So, can you help me figure this out? What could cause data loss? What should I look for in the logs? Which logs should I look at? Just elasticsearch.log, and indexname.log.date? Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/21f6a633-cf3b-4861-95d5-827116a82c81%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/21f6a633-cf3b-4861-95d5-827116a82c81%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 3, 2015, 8:06pm UTC](https://discuss.elastic.co/t/another-missing-data-after-restart-topic/21889/2 "2015-03-03T20:06:26Z")
Any suggestions? At all? It's time to run upgrades again, but since last time I lost data, I'm rather nervous about it. And I'd really like to have everything up to date while I'm at Elasticon next week... On Wednesday, January 28, 2015 at 4:29:47 PM UTC-8, David Reagan wrote: > I read through the past couple months worth of "missing data" topics on > this list, and didn't really find anything to help. So, apologies if I > missed something in them that is obvious. > > Yesterday evening I ran OS updates to fix the glibc vulnerability. That > meant I restarted my 4 ES nodes. Today, I checked ElasticHQ, and a bunch of > this months logstash indices are missing tons of data. One day went from > over a million logs to 53. > > When restarting, I did not bother to shut ES down cleanly. As far as I > know, though, that should not cause ES to delete data... > > My data is stored on nfs mounts, so I double checked that they had mounted > correctly. And I went to the hassle of cleanly shutting them down, one at a > time, and making sure the mount points themselves were empty. A couple > weren't, but the data was from months ago. Not this month. > > In my logs for the past couple days I don't see anything that really pops > out at me. Just that my curator generated snapshots are all failing for > some reason or another. I don't think that has anything to do with the > missing data, but here's an entry from one of my log files anyway. > > [2015-01-26 03:00:10,145][WARN][snapshots] > > > [log-elasticsearch-04] [[logstash-2014.12.31][3]] > > [laneprodelk:curator-20150126110002] failed to create snapshot > > org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: > > [logstash-2014.12.31][3] Failed to perform snapshot (index files) > > at > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:503) > > at > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:139) > > at > > org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:86) > > at > > org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:818) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.io.FileNotFoundException: > > /mounts/prod\_backup/laneprodelk/indices/logstash-2014.12.31/3/\_\_0 (No such > > file or directory) > > at java.io.FileOutputStream.open(Native Method) > > at java.io.FileOutputStream.(FileOutputStream.java:221) > > at java.io.FileOutputStream.(FileOutputStream.java:171) > > at > > org.elasticsearch.common.blobstore.fs.FsBlobContainer.createOutput(FsBlobContainer.java:85) > > at > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshotFile(BlobStoreIndexShardRepository.java:555) > > at > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:501) > > ... 6 more > > ES version info: > > { > "status" : 200, > "name" : "log-elasticsearch-02", > "cluster\_name" : "logstash-webservices", > "version" : { > "number" : "1.4.2", > "build\_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c", > "build\_timestamp" : "2014-12-16T14:11:12Z", > "build\_snapshot" : false, > "lucene\_version" : "4.10.2" > }, > "tagline" : "You Know, for Search" > } > > I'm running the nodes on Ubuntu 12.04 64bit. > > I do have curator deleting data older than 365 days every night. It also > closes indices older than 90 days, and optimizes them if they're older than > 2 days. > > curator is version 2.0.1. > > So, can you help me figure this out? > > What could cause data loss? > > What should I look for in the logs? > > Which logs should I look at? Just elasticsearch.log, and > indexname.log.date? > > Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f7846534-f700-452a-8532-dfa43532401b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f7846534-f700-452a-8532-dfa43532401b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Dennis](https://avatars.discourse-cdn.com/v4/letter/d/e56c9b/32.png) [@Dennis](https://discuss.elastic.co/u/Dennis)\ **Post date:** [March 4, 2015, 5:22am UTC](https://discuss.elastic.co/t/another-missing-data-after-restart-topic/21889/3 "2015-03-04T05:22:37Z")
Somewhre I've read that NFS is not recommended for ElasticSEarch: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. (see 'storage devices) section. I know, it does't help you know, sorry. On Tuesday, March 3, 2015 at 12:06:27 PM UTC-8, David Reagan wrote: > Any suggestions? At all? > > It's time to run upgrades again, but since last time I lost data, I'm > rather nervous about it. > > And I'd really like to have everything up to date while I'm at Elasticon > next week... > > On Wednesday, January 28, 2015 at 4:29:47 PM UTC-8, David Reagan wrote: > > > I read through the past couple months worth of "missing data" topics on > > this list, and didn't really find anything to help. So, apologies if I > > missed something in them that is obvious. > > > > Yesterday evening I ran OS updates to fix the glibc vulnerability. That > > meant I restarted my 4 ES nodes. Today, I checked ElasticHQ, and a bunch of > > this months logstash indices are missing tons of data. One day went from > > over a million logs to 53. > > > > When restarting, I did not bother to shut ES down cleanly. As far as I > > know, though, that should not cause ES to delete data... > > > > My data is stored on nfs mounts, so I double checked that they had > > mounted correctly. And I went to the hassle of cleanly shutting them down, > > one at a time, and making sure the mount points themselves were empty. A > > couple weren't, but the data was from months ago. Not this month. > > > > In my logs for the past couple days I don't see anything that really pops > > out at me. Just that my curator generated snapshots are all failing for > > some reason or another. I don't think that has anything to do with the > > missing data, but here's an entry from one of my log files anyway. > > > > [2015-01-26 03:00:10,145][WARN][snapshots] > > > > > [log-elasticsearch-04] [[logstash-2014.12.31][3]] > > > [laneprodelk:curator-20150126110002] failed to create snapshot > > > org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: > > > [logstash-2014.12.31][3] Failed to perform snapshot (index files) > > > at > > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:503) > > > at > > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:139) > > > at > > > org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:86) > > > at > > > org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:818) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.io.FileNotFoundException: > > > /mounts/prod\_backup/laneprodelk/indices/logstash-2014.12.31/3/\_\_0 (No such > > > file or directory) > > > at java.io.FileOutputStream.open(Native Method) > > > at java.io.FileOutputStream.(FileOutputStream.java:221) > > > at java.io.FileOutputStream.(FileOutputStream.java:171) > > > at > > > org.elasticsearch.common.blobstore.fs.FsBlobContainer.createOutput(FsBlobContainer.java:85) > > > at > > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshotFile(BlobStoreIndexShardRepository.java:555) > > > at > > > org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:501) > > > ... 6 more > > > > ES version info: > > > > { > > "status" : 200, > > "name" : "log-elasticsearch-02", > > "cluster\_name" : "logstash-webservices", > > "version" : { > > "number" : "1.4.2", > > "build\_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c", > > "build\_timestamp" : "2014-12-16T14:11:12Z", > > "build\_snapshot" : false, > > "lucene\_version" : "4.10.2" > > }, > > "tagline" : "You Know, for Search" > > } > > > > I'm running the nodes on Ubuntu 12.04 64bit. > > > > I do have curator deleting data older than 365 days every night. It also > > closes indices older than 90 days, and optimizes them if they're older than > > 2 days. > > > > curator is version 2.0.1. > > > > So, can you help me figure this out? > > > > What could cause data loss? > > > > What should I look for in the logs? > > > > Which logs should I look at? Just elasticsearch.log, and > > indexname.log.date? > > > > Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d31d1295-8aa8-443c-9847-f863ec32cdc2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d31d1295-8aa8-443c-9847-f863ec32cdc2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:28am UTC](https://discuss.elastic.co/t/another-missing-data-after-restart-topic/21889/4 "2017-07-06T00:28:41Z")