Clean index when change IAM phase

Elastic Stack Elasticsearch
1

Hello,

I am doing some tests with the lifecycle policies on an index and I would like to know if it is possible that at the moment in which it passes from the WARM stage to the COLD stage, those documents that have a property with a specific value are filtered, so that the size of the index is reduced by removing them from the index.

2

ILM only deletes complete indices, never documents from within an index, so that is not possible.

3

And is there any other solution for this problem?

A possible solution that has occurred to me is to consult the indexes of the alias being processed, to obtain those that are in the COLD phase (for now the only possibility for this is to consult ALL and check one by one in which phase this using the GET /index/_ilm/explain endpoint) and dump the documents that meet the condition to a new index.

Furthermore, the first would have a policy of HOT of one day, WARM of 29 and COLD of 1, and the second a HOT and WARM of 0, COLD of 151 and FROZEN of 214.

I know that it is not a very orthodox solution, and that it depends on an external process, but I can't think of any other way.

4

The easiest way to handle this is probably to have 2 data streams/sets of indices, with different ILM policies, where data is written based on the retention requirement. These can all be linked to a single alias for reading, so the fact that there are two sets of indices would not necessarily be apparent to the user.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.