# ES FileSystemException (out of disk space)

**URL:** <https://discuss.elastic.co/t/es-filesystemexception-out-of-disk-space/104935>\
**Category:** Elasticsearch\
**Created:** [October 23, 2017, 4:51pm UTC](https://discuss.elastic.co/t/es-filesystemexception-out-of-disk-space/104935 "2017-10-23T16:51:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![shines2](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shines2](https://discuss.elastic.co/u/shines2)\
**Post date:** [October 23, 2017, 4:51pm UTC](https://discuss.elastic.co/t/es-filesystemexception-out-of-disk-space/104935/1 "2017-10-23T16:51:48Z")

</div>

My entire cluster that back-ends my graylog deployment stop working over the weekend.

It complaining that /proj/graylog (The directory where I store ES data) is full. A df shows that it's only 50% full:

Type Size Used Avail Use% Mounted on  
server:/graylog/graylog/ nfs 238G 117G 121G 50% /proj/graylog

```
curl -XGET 'http://localhost:9200/_cluster/allocation/explain?pretty'
{
  "index" : "in_graylog_18",
  "shard" : 1,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2017-10-23T09:52:46.654Z",
    "failed_allocation_attempts" : 5,
    "details" : "failed to create shard, failure FileSystemException[/proj/graylog/grayloges3/elasticsearch/graylog/nodes/0/indices/tkISaqIcQW21ZnwZrEj1kA/1/_state/state-1.st.tmp: No space left on device]",
    "last_allocation_status" : "no"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy",
  "node_allocation_decisions" : [
    {
      "node_id" : "kZnka3m4Tl-wwzG-MPBHOw",
      "node_name" : "grayloges3",
      "transport_address" : "10.66.8.202:9300",
      "node_decision" : "no",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "TkTj0jxBRD68v8jQdGmB5A"
      },
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2017-10-23T09:52:46.654Z], failed_attempts[5], delayed=false, details[failed to create shard, failure FileSystemException[/proj/graylog/grayloges3/elasticsearch/graylog/nodes/0/indices/tkISaqIcQW21ZnwZrEj1kA/1/_state/state-1.st.tmp: No space left on device]], allocation_status[deciders_no]]]"
        }
      ]
    }
  ]
}

```

I tried restarting es on each node but I have been unable to get it working again.

What process do I follow to get the cluster working again without losing any data?

It's a three node cluster running elasticsearch-5.5.2

---

<div class="post-metadata">

**Author:** ![shines2](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shines2](https://discuss.elastic.co/u/shines2)\
**Post date:** [October 23, 2017, 4:56pm UTC](https://discuss.elastic.co/t/es-filesystemexception-out-of-disk-space/104935/2 "2017-10-23T16:56:53Z")

</div>

I just found out from my storage admin that the disk was full (.snapshots was the culprit) this morning.

The disk is no longer full so how do I get the cluster to recover?

---

<div class="post-metadata">

**Author:** ![shines2](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shines2](https://discuss.elastic.co/u/shines2)\
**Post date:** [October 23, 2017, 5:05pm UTC](https://discuss.elastic.co/t/es-filesystemexception-out-of-disk-space/104935/3 "2017-10-23T17:05:20Z")

</div>

Ok I figured out how to recover the cluster...the magical command was: curl -XPOST 'localhost:9200/\_cluster/reroute?retry\_failed'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 20, 2017, 5:05pm UTC](https://discuss.elastic.co/t/es-filesystemexception-out-of-disk-space/104935/4 "2017-11-20T17:05:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.