# After server disk full, indexes unavailable **URL:** \ **Category:** Elasticsearch\ **Created:** [October 7, 2011, 10:01pm UTC](https://discuss.elastic.co/t/after-server-disk-full-indexes-unavailable/5542 "2011-10-07T22:01:01Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![drahmel7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drahmel7/32/2967_2.png) [@drahmel7](https://discuss.elastic.co/u/drahmel7)\ **Post date:** [October 7, 2011, 10:01pm UTC](https://discuss.elastic.co/t/after-server-disk-full-indexes-unavailable/5542/1 "2011-10-07T22:01:01Z")
My CentOS server box that was hosting Elastic Server ran out of disk space. After clearing disk space and bringing up ES, all indexes are unavailable. If I do this: curl -XGET [http://localhost:9200/\_status](http://localhost:9200/_status) I get: {"ok":true,"\_shards":{"total":50,"successful":0,"failed":0},"indices":{}} If I query my index: curl -XGET [http://localhost:9200/cast/\_status](http://localhost:9200/cast/_status) I get: {"ok":true,"\_shards":{"total":10,"successful":0,"failed":0},"indices":{}} If I do: curl -XGET '[http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)' I get: {"cluster\_name":"elasticsearch","status":"red","timed\_out":false,"number\_of\_nodes":1,"number\_of\_data\_nodes":1,"active\_primary\_shards":0,"active\_shards":0,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":50} When I look at the log I see this: [2011-10-07 10:18:30,188][WARN][indices.cluster] [Mountjoy] [cast][2] failed to start shard org.elasticsearch.index.shard.recovery.RecoveryFailedException: Index Shard [cast][2]: Recovery failed from [Mogul of the Mystic Mountain][p-w0IjpeTYqR4EbPXezzOQ][inet[/10.7.88.73:9300]] into [Mountjoy][cJoOqTimSiqAzfsyJMQlLg][inet[/10.7.88.73:9301]] at org.elasticsearch.index.shard.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:253) at org.elasticsearch.index.shard.recovery.RecoveryTarget.access$100(RecoveryTarget.java:71) at org.elasticsearch.index.shard.recovery.RecoveryTarget$2.run(RecoveryTarget.java:156) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: org.elasticsearch.transport.RemoteTransportException: [Mogul of the Mystic Mountain][inet[/10.7.88.73:9300]][index/shard/recovery/startRecovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [cast][2] Phase[1] Execution failed at org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1006) at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:420) at org.elasticsearch.index.shard.recovery.RecoverySource.recover(RecoverySource.java:110) at org.elasticsearch.index.shard.recovery.RecoverySource.access$1600(RecoverySource.java:60) at org.elasticsearch.index.shard.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:296) at org.elasticsearch.index.shard.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:285) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: org.elasticsearch.index.shard.recovery.RecoverFilesRecoveryException: [cast][2] Failed to transfer [201] files with total size of [16.1gb] at org.elasticsearch.index.shard.recovery.RecoverySource$1.phase1(RecoverySource.java:207) at org.[2011-10-07 11:07:34,844][WARN][indices.cluster] [Mountjoy] [cast][3] failed to start shard Is this corrupted? Is there a way to recover my index? Dan ---
**Author:** ![johno](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@johno](https://discuss.elastic.co/u/johno)\ **Post date:** [July 19, 2012, 4:52pm UTC](https://discuss.elastic.co/t/after-server-disk-full-indexes-unavailable/5542/2 "2012-07-19T16:52:56Z")
I have almost precisely the same problem. If there is a known fix, please post it! ElasticSearch appears to be especially sensitive to this kind of thing. I had logging going to a different filesystem, and after running out of space on the index filesystem, it proceeded to fill up a couple hundred gigs of logs on the other filesystem, dumping entire documents to the log. Since bulk indexing took twelve hours, I'd prefer to recover what I can. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/after-server-disk-full-indexes-unavailable/5542/3 "2017-07-06T03:19:47Z")