# No space - disaster **URL:** \ **Category:** Elasticsearch\ **Created:** [November 9, 2015, 3:01pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158 "2015-11-09T15:01:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![scaarup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scaarup/32/4615_2.png) [@scaarup](https://discuss.elastic.co/u/scaarup)\ **Post date:** [November 9, 2015, 3:01pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/1 "2015-11-09T15:01:27Z")
Running out of space on the elasticsearch-server is a total disaster. After freeing up some space, I am trying to start ES again, but the following error continues to appear in the log: ``` [2015-11-09 14:56:38,021][WARN][cluster.action.shard] [Terrax the Tamer] [fe-2015.11.09][2] received shard failed for [fe-2015.11.09][2], node[-Xyzs56LQIqGOqbYDns1eQ], [P], v[7], s[INITIALIZING], a[id=wD4WjCX0TyiLpgV-PUzdnQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2015-11-09T13:56:38.003Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileAlreadyExistsException[/var/lib/elasticsearch/elasticsearch/nodes/0/indices/fe-2015.11.09/2/translog/translog-13.ckp]; ]], indexUUID [2cReJ5KXR2yaywjvM7SO0g], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileAlreadyExistsException[/var/lib/elasticsearch/elasticsearch/nodes/0/indices/fe-2015.11.09/2/translog/translog-13.ckp]; ] [fe-2015.11.09][[fe-2015.11.09][2]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileAlreadyExistsException[/var/lib/elasticsearch/elasticsearch/nodes/0/indices/fe-2015.11.09/2/translog/translog-13.ckp]; at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:258) at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:60) at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:133) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: [fe-2015.11.09][[fe-2015.11.09][2]] EngineCreationFailureException[failed to create engine]; nested: FileAlreadyExistsException[/var/lib/elasticsearch/elasticsearch/nodes/0/indices/fe-2015.11.09/2/translog/translog-13.ckp]; at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:135) at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1349) at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1344) at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:889) at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:866) at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:249) ... 5 more Caused by: java.nio.file.FileAlreadyExistsException: /var/lib/elasticsearch/elasticsearch/nodes/0/indices/fe-2015.11.09/2/translog/translog-13.ckp at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:551) at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253) at java.nio.file.Files.copy(Files.java:1274) at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:304) at org.elasticsearch.index.translog.Translog.(Translog.java:166) at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:188) at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:131) ... 11 more ``` It actually appear so many times, that the ES logfile is more than 10GB, causing the server to be out of space again... And something strange happens in todays index while ES is stacktracing. 35G d2-2015.11.09 Currently 35GB and normally a day is about 8GB. Seems like some sort of an insane loop situation... How do I recover from this? ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [November 9, 2015, 6:44pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/2 "2015-11-09T18:44:02Z")
hey this seems like a bug to me and I think I see what the reason is. Can I ask you to: - open an issue for this here: [https://github.com/elastic/elasticsearch/issues/](https://github.com/elastic/elasticsearch/issues/) - link this entry here in the issue - list all files in `/var/lib/elasticsearch/elasticsearch/nodes/0/indices/fe-2015.11.09/2/translog` - try to start that node again with logging for `index.translog: DEBUG` and `index.engine: DEBUG` and paste the output on the issue thanks so much for reporting this. ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [November 11, 2015, 9:14am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/3 "2015-11-11T09:14:38Z")
@scaarup did you open that issue or should I do it? I can but I'd still appreciate the requested information ---
**Author:** ![scaarup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scaarup/32/4615_2.png) [@scaarup](https://discuss.elastic.co/u/scaarup)\ **Post date:** [November 11, 2015, 10:18am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/4 "2015-11-11T10:18:24Z")
Hi @s1monw. No I didn't because I had to recover and I therefore deleted all my indices. Meaning I cannot provide you with what you requested. Sorry about that. ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [November 11, 2015, 8:04pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/5 "2015-11-11T20:04:24Z")
bummer! would always be good to safe the data so we can reproduce this. I think I found the issue but I can't be 100% sure now [https://github.com/elastic/elasticsearch/pull/14695](https://github.com/elastic/elasticsearch/pull/14695) ---
**Author:** ![alphan](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@alphan](https://discuss.elastic.co/u/alphan)\ **Post date:** [November 16, 2015, 4:28pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/6 "2015-11-16T16:28:54Z")
I am not sure if this is still helpful. I was trying out ES and encountered the same issue. You can find the part of the zipped log and the directory tree here : [https://www.dropbox.com/s/gze0sssiv67fh1e/issue\_14695\_logs.tar.gz?dl=0](https://www.dropbox.com/s/gze0sssiv67fh1e/issue_14695_logs.tar.gz?dl=0) ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [December 9, 2015, 10:23am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/7 "2015-12-09T10:23:54Z")
Hello, I am currently experiencing a similar problem is there a solution to this? ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [December 9, 2015, 10:42am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/8 "2015-12-09T10:42:58Z")
can you provide all information from logs what kind of problem you are having as well as which version you are running? ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [December 9, 2015, 10:54am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/9 "2015-12-09T10:54:24Z")
another question is did you have any issues with running out of disk space? ---
**Author:** ![bharath](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@bharath](https://discuss.elastic.co/u/bharath)\ **Post date:** [December 9, 2015, 11:14am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/10 "2015-12-09T11:14:51Z")
i am also having the same issue more details regarding mem i still have 80% JVM another 60% "os":{"timestamp":1449649876869,"load\_average":0.04,"mem":{"total\_in\_bytes":33535873024,"free\_in\_bytes":26795671552,"used\_in\_bytes":6740201472,"free\_percent":80,"used\_percent":20},"swap":{"total\_in\_bytes":0,"free\_in\_bytes":0,"used\_in\_bytes":0}},"process":{"timestamp":1449649876869,"open\_file\_descriptors":218,"max\_file\_descriptors":4096,"cpu":{"percent":2,"total\_in\_millis":16846170},"mem":{"total\_virtual\_in\_bytes":4826849280}},"jvm":{"timestamp":1449649876869,"uptime\_in\_millis":170503757,"mem":{"heap\_used\_in\_bytes":434470248,"heap\_used\_percent":41,"heap\_committed\_in\_bytes":649949184,"heap\_max\_in\_bytes":1037959168,"non\_heap\_used\_in\_bytes":62325864,"non\_heap\_committed\_in\_bytes":92753920,"pools":{"young":{"used\_in\_bytes":40399312,"max\_in\_bytes":286326784,"peak\_used\_in\_bytes":71696384,"peak\_max\_in\_bytes":286326784},"survivor":{"used\_in\_bytes":8288368,"max\_in\_bytes":35782656,"peak\_used\_in\_bytes":8912896,"peak\_max\_in\_bytes":35782656},"old":{"used\_in\_bytes":385782568,"max\_in\_bytes":715849728,"peak\_used\_in\_bytes":431855616,"peak\_max\_in\_bytes":715849728}}}, ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [December 9, 2015, 12:13pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/11 "2015-12-09T12:13:08Z")
yes I did ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [December 9, 2015, 12:15pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/12 "2015-12-09T12:15:17Z")
I couldn't provide the log files because I deleted them, they were quite large : upto 35G. I deleted some translog files in some nodes and rebooted, it seems to have solved the issue. I hope no problem arises. ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [December 9, 2015, 12:36pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/13 "2015-12-09T12:36:34Z")
this issue has been fixed in 2.1 and 2.0.1 so please upgrade. ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [December 9, 2015, 12:45pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/14 "2015-12-09T12:45:03Z")
Ok, we are Currently using 2.0, I would like to know what makes the log files become so large. it led to running out of disk space. ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [December 9, 2015, 12:46pm UTC](https://discuss.elastic.co/t/no-space-disaster/34158/15 "2015-12-09T12:46:43Z")
I can't tell - would need to look at it's content which is tricky. Can you maybe look if you see something? I suspect some guice errors ☹ ---
**Author:** ![raghvendra](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@raghvendra](https://discuss.elastic.co/u/raghvendra)\ **Post date:** [June 29, 2017, 6:33am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/16 "2017-06-29T06:33:58Z")
Hey @s1monw , I have got the same issue in es 5.4.0. When i'm trying to start the elasticsearch via crontab.. it's giving these logs . > 2017-06-29T11:58:02,106][INFO][o.e.n.Node] [master] initializing ... > [2017-06-29T11:58:02,218][INFO][o.e.e.NodeEnvironment] [master] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable\_space [725.7gb], net total\_space [908.4gb], spins? [possibly], types [ext4] > [2017-06-29T11:58:02,219][INFO][o.e.e.NodeEnvironment] [master] heap size [3.9gb], compressed ordinary object pointers [true] > [2017-06-29T11:58:02,254][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [master] uncaught exception in thread [main] > org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to created node environment > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.4.0.jar:5.4.0] > Caused by: java.lang.IllegalStateException: Failed to created node environment > at org.elasticsearch.node.Node.(Node.java:265) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.node.Node.(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0] > ... 6 more > Caused by: java.io.IOException: failed to write in data directory [/home/mywavia/Downloads/elasticsearch-5.4.0/data/nodes/0/indices/A1JkmYVoSpGRNrJmATzwRg/\_state] write permission is required > at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1075) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1047) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:277) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.node.Node.(Node.java:262) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.node.Node.(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0] > ... 6 more > Caused by: java.nio.file.FileAlreadyExistsException: /home/mywavia/Downloads/elasticsearch-5.4.0/data/nodes/0/indices/A1JkmYVoSpGRNrJmATzwRg/\_state/.es\_temp\_file > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) ~[?:?] > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?] > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?] > at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?] > at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0\_131] > at java.nio.file.Files.createFile(Files.java:632) ~[?:1.8.0\_131] > at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1072) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1047) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:277) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.node.Node.(Node.java:262) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.node.Node.(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0] > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0] Can you help me with this ? ---
**Author:** ![raghvendra](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@raghvendra](https://discuss.elastic.co/u/raghvendra)\ **Post date:** [June 29, 2017, 7:06am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/17 "2017-06-29T07:06:59Z")
Hey .. when I went to that directory , and opened that file , it was empty . So , without having a second thought , i deleted that file and restarted the node , now every thing is working fine . But what I have done , was something , I was supposed to do , or there is any other better options that could have been tried ?? ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [June 29, 2017, 7:29am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/18 "2017-06-29T07:29:07Z")
> Hey .. when I went to that directory , and opened that file , it was empty . So , without having a second thought , i deleted that file and restarted the node , now every thing is working fine . > But what I have done , was something , I was supposed to do , or there is any other better options that could have been tried ?? we try to write a file in the data directory to ensure we have the correct permission. Yet if something fails during the node startup we might leave a file behind. There is a pull request open where I have concerns. [Node should start up despite of a lingering `.es_temp_file` by bleskes · Pull Request #21210 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/21210) deleting the file was the right thing to resolve the condition, do you know how you got into this? did you kill the node while it was starting? ---
**Author:** ![raghvendra](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@raghvendra](https://discuss.elastic.co/u/raghvendra)\ **Post date:** [June 29, 2017, 7:59am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/19 "2017-06-29T07:59:13Z")
Yes .. actually , this get started on startup via crontab scheduler so every time when I start my node , ES also starts , but yesterday , I didn't want to start ES , so I had killed all java process by > sudo killall -2 java . May be that's how I got into this error ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 4:09am UTC](https://discuss.elastic.co/t/no-space-disaster/34158/20 "2022-11-04T04:09:03Z")