ILM not reclaiming back space in ELK Host server

We have a ELK cluster with single node having 50gb disk space. We are sending metricbeat data from 200 servers to Elasticsearch.
Created an ILM policy for metricbeat index to do a rollover and delete the index after it crosses 5GB.
ILM is doing the index rollover and deletion but still we are not able to get back the disk space back.
Eventually disk percent used is increasing each day and after few days, disk is completely filled up.

Why ILM policy is not deleting the data and reclaiming space?

Could you please help us with this?

Hi @Anagha_nambiar Welcome to the community.

Please show your exact policy.

HI @stephenb Please find the below ILM policies





@stephenb Do we need to update something in metricbeat.yml file regarding this ILM policy?

Please do not share screenshots / images. They are very hard to read, can not even be read by others can not be searched or debugged...

Also I can not tell from all the screenshots which ILM you are having trouble with.

Please post the code in formatted text of the exact ILM policy you are having trouble with.

@stephenb I am having issue with metricbeat and heartbeat indexes.

Those two indexes are rolled over to new index but still the disk space is not reclaimed back.

Please excuse me as I am not getting to copy paste the code directly so I am sending the snap.

Please find the policy details below:

So you can see the indices are actually getting deleted?

How did you confirm that?

When an index is deleted the disk space is almost immediately recovered.

If so then the disk continually rising is probably not from the index data.

Perhaps the disk is being consumed by something else... like the Elasticsearch logs or something else.

@stephenb yes, I can see the indexes getting deleted from Index management in Kibana UI.

When you say, it might be because of Elasticsearch logs, how can we automate that deletion? And how can we find and delete the logs/others?

Could you please suggest?

I do not know what is actually taking the space, it could be any number of things.

I can say that when an index is deleted the disk space is reclaimed almost immediately so the source of your rising disk usage is probably something else.

Logging configuration, rotation, deletion etc..etc..

Your Linux admin should able help you find the sources of the directories taking up space.

Thank you @stephenb
I will try to check what you mentioned and clear the space.