I have two types of index in my system:
Index_A, where I store all documents, with '@timestamp' representing the log injection time to Elasticsearch, and '_datetime' indicating the log's own timestamp. In order to minimus the number of shards in the system.
I also use Index_B. Here, I create a new index every day, so documents from the same day are stored in an index named Index_B_yyyy.mm.dd.
Now, I am looking to carry out housekeeping tasks.
Specifically, I want to move logs older than 2 months to a cold tier
and delete all logs older than 6 months.
Do you have any recommendations on how to achieve this effectively? Can Index Lifecycle Policies be used for this purpose?
Hi @Frances_Chu ,
Based on your scenario, you can likely use Index Lifecycle Management (ILM) for both types of indices, but you'll need different approaches.
Here are some supporting links:
It would also be good, for colleagues' suggestions, if you could mention which version you're using.
Alex
Thank you for the valuable information. It has been very fruitful. My Elasticsearch version is 8.12.0
Frances
ILM can be used to manage retention for time based indices like Index_B in your use case. ILM can however only delete complete indices, so will not be able to manage retention from a single index where you periodically need to delete documents using delete-by-query, which seems to be the case for your Index_A.
1 Like
It's very clear, thank you very much. I am also considering moving in this direction.