# Error creating snapshot with HDFS plugin, FileSystemClosed **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [June 9, 2015, 5:33pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243 "2015-06-09T17:33:09Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![tgreischel](https://avatars.discourse-cdn.com/v4/letter/t/ad7895/32.png) [@tgreischel](https://discuss.elastic.co/u/tgreischel)\ **Post date:** [June 9, 2015, 5:33pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/1 "2015-06-09T17:33:09Z")
ES - 1.5 ES Hadoop plugin - 2.1.0 b4 {"error":"RepositoryVerificationException[[esprod-usagetracking-2015-06] path is not accessible on master node]; nested: IOException[Filesystem closed]; ","status":500} The above is the error received when running a cronjob **_every other_** morning. That's the strange part, it works once and then the next run it fails out. So I only have snapshots for every other day. The snapshots do work, but again, only every other run. When I run the script over and over it returns the error every other run. Has anyone seen this before? ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [June 9, 2015, 6:09pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/2 "2015-06-09T18:09:38Z")
Weird. It looks like the `FileSystem` object used underneath by the plugin is affected somehow. What distro are you using? The plugin never closes the `FileSystem` not does it keep on creating a new one; however _other_ Hadoop clients running on the same machine might interfere with it as the `FileSystem` relies on an internal cache that can be affected. Do you have a bigger stacktrace (potentially from Elasticsearch itself)? Can you double check whether there are other jobs interfering with Hadoop every other day? Anything that sticks out from the Hadoop logs? Do you restart Elasticsearch by any chance? ---
**Author:** ![tgreischel](https://avatars.discourse-cdn.com/v4/letter/t/ad7895/32.png) [@tgreischel](https://discuss.elastic.co/u/tgreischel)\ **Post date:** [June 9, 2015, 6:20pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/3 "2015-06-09T18:20:17Z")
Running on CentOS 6.6. ES is the only thing running on this machine and is the only cronjob. ES is not restarted in between each snapshot creation, but it has been restarted before. Below is from the logs on 6/4: ``` [2015-06-04 08:45:12,011][INFO][repositories] [esprod00] update repository [esprod-usagetracking-2015-06] [2015-06-04 08:45:12,068][WARN][snapshots] [esprod00] failed to create snapshot [esprod-usagetracking-2015-06:snapshot-2015-06-04] org.elasticsearch.snapshots.SnapshotCreationException: [esprod-usagetracking-2015-06:snapshot-2015-06-04] failed to create snapshot at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:260) at org.elasticsearch.snapshots.SnapshotsService.beginSnapshot(SnapshotsService.java:278) at org.elasticsearch.snapshots.SnapshotsService.access$600(SnapshotsService.java:88) at org.elasticsearch.snapshots.SnapshotsService$1$1.run(SnapshotsService.java:204) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:707) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1448) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1390) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:394) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:390) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:390) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:334) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:849) at org.elasticsearch.hadoop.hdfs.blobstore.HdfsBlobContainer.createOutput(HdfsBlobContainer.java:71) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:235) ``` ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [June 9, 2015, 8:35pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/4 "2015-06-09T20:35:56Z")
This is helpful. What Hadoop version/distro are you using? P.S. Can you please use formatting - really simple but improves readability a lot. Thanks ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [June 9, 2015, 9:23pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/5 "2015-06-09T21:23:25Z")
@tgreischel Hi I've pushed a couple of updates which hopefully should fix your problem. 1. the `FileSystem` instance is checked to see whether it's alive or not, so in case it is closed, a new one will be created. 2. instead of using the typical API which relies on some Hadoop client caching (which can cause the `FileSystem` to be closed by other clients), a dedicated, private instance is now created instead which should be managed just by the plugin itself (though there is a shutdown hook that might close it, however see #1). I have pushed a new [dev build](https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs#development-snapshot) already in the repository - can you please try it out and let me know how it works for you. You shouldn't get the exception any more even for subsequent builds. Cheers, ---
**Author:** ![tgreischel](https://avatars.discourse-cdn.com/v4/letter/t/ad7895/32.png) [@tgreischel](https://discuss.elastic.co/u/tgreischel)\ **Post date:** [June 10, 2015, 1:14pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/6 "2015-06-10T13:14:31Z")
That worked perfect! The backup ran great without error. Thanks again for the help!! ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [June 10, 2015, 5:37pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/7 "2015-06-10T17:37:46Z")
Glad to hear it. Cheers! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:28pm UTC](https://discuss.elastic.co/t/error-creating-snapshot-with-hdfs-plugin-filesystemclosed/2243/8 "2017-07-06T13:28:17Z")