# Out of space issue **URL:** \ **Category:** Elasticsearch\ **Created:** [April 25, 2016, 7:21am UTC](https://discuss.elastic.co/t/out-of-space-issue/48278 "2016-04-25T07:21:39Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![scaarup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scaarup/32/4615_2.png) [@scaarup](https://discuss.elastic.co/u/scaarup)\ **Post date:** [April 25, 2016, 7:21am UTC](https://discuss.elastic.co/t/out-of-space-issue/48278/1 "2016-04-25T07:21:39Z")
Hi all. I have the following in my elasticsearch conf: path.data: /var/lib/elasticsearch/DATA2,/var/lib/elasticsearch/DATA3,/var/lib/elasticsearch/DATA1 Now I have run out of discspace on /var/lib/elasticsearch/DATA2 but I have plenty of space on /var/lib/elasticsearch/DATA3. However elasticsearch is in a state where it refuses to receive any more data and it writes the following stacktrace several times per second: ``` [2016-04-25 09:11:01,599][WARN][cluster.action.shard] [mgp-es103] [audit-2016.16][2] received shard failed for target shard [[audit-2016.16][2], node[_8mzWslDT8yONVh5jO7-mw], [P], v[128564], s[INITIALIZING], a[id=Et5k8n6SRQO8r4ESbMAAPQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-04-25T07:11:00.733Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog.ckp -> /var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog-8124360722841973525.tlog: No space left on device]; ]]], indexUUID [UJk0HgzJQXeaUgQGrqUuyA], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog.ckp -> /var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog-7467623683839595706.tlog: No space left on device]; ] [audit-2016.16][[audit-2016.16][2]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog.ckp -> /var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog-7467623683839595706.tlog: No space left on device]; at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250) at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56) at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: [audit-2016.16][[audit-2016.16][2]] EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog.ckp -> /var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog-7467623683839595706.tlog: No space left on device]; at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:155) at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515) at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499) at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972) at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944) at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241) ... 5 more Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog.ckp -> /var/lib/elasticsearch/DATA2/dibs/nodes/0/indices/audit-2016.16/2/translog/translog-7467623683839595706.tlog: No space left on device at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:253) at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:581) at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253) at java.nio.file.Files.copy(Files.java:1274) at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:344) at org.elasticsearch.index.translog.Translog.(Translog.java:179) at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208) at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:151) ... 11 more ``` Why can't ES recover from this situation? Running version 2.3.1. ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [April 25, 2016, 2:12pm UTC](https://discuss.elastic.co/t/out-of-space-issue/48278/2 "2016-04-25T14:12:03Z")
I can't think of how to have ES fix itself except to move the data from DATA2 to a drive with more disk space (or increase the disks allocated to DATA2). I'm surprised that it got into this situation though - I would have thought that the disk allocator would have handled this ( [https://github.com/elastic/elasticsearch/issues/11271](https://github.com/elastic/elasticsearch/issues/11271) and [https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html)). Did you disable that by chance? ---
**Author:** ![scaarup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scaarup/32/4615_2.png) [@scaarup](https://discuss.elastic.co/u/scaarup)\ **Post date:** [April 26, 2016, 6:20am UTC](https://discuss.elastic.co/t/out-of-space-issue/48278/3 "2016-04-26T06:20:44Z")
No I havn't disabled that, and I can see it's enabled by default. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:56pm UTC](https://discuss.elastic.co/t/out-of-space-issue/48278/4 "2017-07-05T22:56:21Z")