Configure ILM : Delete index = delete data?

Elastic Stack Elasticsearch
1

Hello everyone,

I send data every day on a single index, and I would like to delete all the data that are older than one year in order to optimize the available space on my disk.

Will choosing the "Delete after 2 days" option in ILM delete all the data in my index that are older than 2 days and keep the ones that are newer than 2 days?

image
image1622×434 34.1 KB

Because in this printscreen it is write "Delete Data you no longer need". So I presume that my configuration will delete all data older than 2 days, and keep the rest, but i'm not sure.

Thanks in advance for your help.

2

The delete phase will delete the entire index based on the index creation date, it won't delete specific documents.

For example, if you have an index created two days ago that you are still writing to this index, when the delete phase trigger, it will delete the entire index, it doesn't matter if you have documents in the index that aren't older than two days, what matter is the date of the index creation.

3 Likes
3

Thank's for your reply @leandrojmp

I understand this strange behavior better now.

a) It means that if for example every me I create a new index, I can create an ILM that deletes the indexes after 400 days. So I will be sure to keep my data for at least 1 year?

b) I understand your explanation, you say that only the index is deleted. Does this mean that the data is still stored on the server? Because when you configure an ILM, it is noted "Delete data you no longer need." in the last phase of deletion.

Thanks in advance.

4

You can create a rollover in your ILM to create a new index based on scheduled interval, for example, it would rollover your data into a new index every month. The documentation explains better how to do this.

If you want to delete an index after 400 days, for example, and an index was created more then 400 days ago, it will be deleted even if the last documment ingested in this index is not 400 days old. The delete phase uses the index creation date to trigger a deletion.

Your data resides in your index, if you delete an index, the data is also deleted and space will be freed in the disk

1 Like
5

Ok I see ! Thank's !

Finally, creating an index each month is doing manually what a Rollover can do.

Last little question: when we create a Rollover, do we have the possibility to choose the syntax of the new indexes? For example with the number of each month.Or is the only possibility to use the increment which is set by default 0001, 0002...

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.