No space left on device and then "FileAlreadyExistsException"

Elastic Stack Elasticsearch
1

We are using Elasticsearch:

{
"name": "Guardsman",
"cluster_name": "elasticsearch",
"version": {
"number": "2.0.0",
"build_hash": "de54438d6af8f9340d50c5c786151783ce7d6be5",
"build_timestamp": "2015-10-22T08:09:48Z",
"build_snapshot": false,
"lucene_version": "5.2.1"
},
"tagline": "You Know, for Search"
}

ES hosted on linux server (ubuntu).
The server had 50 GiB volume of storage.

Suddenly ES stopped working at all.

Search queries retrieve the following result:
"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed;

ES cluster health status was Red, all the shard status were red as well.

After digging in ES logs I figured out that there are no space on disk,
in order to resolve the issue, I create a new expended storage volume for ES server.

Unfartently, this did not resolve the problem, We still getting errors from ES.
This time FileAlreadyExistsException (while recovering shards according to ES's logs).

Attached logs file of ES from both, before expending the volume and after.

For being operative we deleted all of our indices at ES at then restart ES service,
and this of-curse solved the issue, BUT we lost all of our data.

  1. How can we avoid such of incidents in the future?
  2. Is there any resolution to "FileAlreadyExistsException"?

Thank in advance.

2
  1. Monitor the disk space use.
  2. You can try moving the file it refers to aside, that should help but it cannot guarantee against data loss.