Monthly indices but ILM on daily basis

Elastic Stack Elasticsearch
1

Hello.

I had daily indices and a fixed retention time of 365 days via ILM. Now I changed to monthly indices to increase the index size. The problem now is that when the oldest index will be deleted, all data from now-(365-30) will be deleted. What is a good way to get around this? Summarized: I want an exact retention time of 365 days. Not more and not less. A naive idea is to reindex indices that become 11 months old into "daily" indices but I hope there are better ways.

2

ILM deletes complete indices so if you have monthly indices that is the deletion granularity. ILM therefore never allows exact retention. By changing to daily indices you can get more accurate but data will still only be deleted once per day.

3

I know. That's why I am seeking for a workaround. I just stumbled upon Delete API and Delete by query API. Until now I thought it is not possible to delete or alter documents from an index without deleting the whole index. I am wondering if they really simply delete documents or if they cause recreation of an index. I think I will do some performance tests with delete operations.

4

Delete by query deletes documents. Deleting documents this way is however much less efficient than simply deleting full indices as each delete in reality is an update with a tombstone record. It therefore results in much higher load and resource usage on the cluster.

5

Ah ok. So delete is not really delete. Since the main goal behind this is to satisfy data protection regulations, this would not fit my needs.

6

It is a delete, but space is only freed up through merging, which does not happen right away.

7

I stick with deleting indices. Thanks for your help. I don't see a close button here so I at least mark a post as solution.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.