I have Elasticsearch 7.4 with 1 master and 5 data nodes (Amazon ec2 instances). I have cluster.routing.allocation.disk.threshold_enabled setting = false Now one of the disk in a node became full and elasticsearch stopped due to disk full error.
ElasticsearchException[failed to load metadata]; nested: IOException[No space left on device]; /dev/nvme1n1p1 493G 468G 0 100% /disk1
After this, cluster shows only 4 data nodes, 9 shards are still unallocated and search requests are being denied, while other nodes still have enough space left on their disks.
I want to make some space by deleting old data through POST API but that won't work in this case.
Now my question is, how can I cleanup the disk on this node while it only contains elasticsearch data and is not available in the cluster.
Thank you for your help.