Elastic search Cluster health due to disk space

Elastic Stack Elasticsearch
1

Hi,

I am currently experiencing an issue on my elastic cluster due to low disk space.

"explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [9.737415872042689%]"

I have located the files in question and it looks like the following is taking up the space.

/home/????/elasticsearch-6.5.1/data/nodes/0/indices

Forgive my naivety but could any one recommend a safe way of clearing out these indices or freeing up some space?

Many thanks,

2

It looks like you've got less than 10% free disk space, which means you cannot do a force merge to remove deleted documents (because the merge operation will write new segments before removing the old).

With insufficient space for a force merge you have three options left, that I can think of:

  1. Increase the disk space on the almost full server.
  2. Add more data nodes to your cluster to spread out the index shards on more nodes.
  3. Delete one large index (or more) to free up disk space immediately.

Good luck!