How to Cleanup Full Disk on Elasticsearch Node while Elasticsearch is not running

Elastic Stack Elasticsearch
1

I have Elasticsearch 7.4 with 1 master and 5 data nodes (Amazon ec2 instances). I have cluster.routing.allocation.disk.threshold_enabled setting = false Now one of the disk in a node became full and elasticsearch stopped due to disk full error.

ElasticsearchException[failed to load metadata]; nested: IOException[No space left on device];

/dev/nvme1n1p1  493G  468G     0 100% /disk1

After this, cluster shows only 4 data nodes, 9 shards are still unallocated and search requests are being denied, while other nodes still have enough space left on their disks.

I want to make some space by deleting old data through POST API but that won't work in this case.

Now my question is, how can I cleanup the disk on this node while it only contains elasticsearch data and is not available in the cluster.

Thank you for your help.

2

Welcome to our community! :smiley:

You will need to expand the disk and then delete things via the API. Do not delete things directly from the disk.

3

Thank you Mark, so that means we are left with only this option. There is no other option to move/remove data from an offline node, or an option to start elasticsearch on read-only / delete-only mode on a node with full disk?

4

This is not recommended and can cause problems. The default setting is there for a reason as bad things happen when you run out of disk space.

If none of your indices are red you might be able to delete all data on the node and restart so that Elasticsearch can reallocate unassigned shards.

I would recommend reenabling the watermarks though.

1 Like
5

Usually all you can do it purge other non-Elasticsearch data like old logs or other installed stuff if it's on a shared disk or filesystem.

Also check if you have root reserved space on that filesystem that may distributions include. Usually 5% for ext3/ext4.

See article on that.

You can run this to see if you have any:
tune2fs -l /dev/partition | grep 'Reserved'

Setting it to 0 can free up space enough to start and then migrate shards away.

1 Like
6

I did another thing. Went to the directory of unassigned shard, moved some files to my home directory, elasticsearch was up again with red status (1 unassigned shard). After that, I moved another shard to another node, it created space on disk, then I moved the files back to shard from home, and ran API to reallocate unassigned shard. Worked for me

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.