Index origination date based on logs age

Elastic Stack Logstash
1

Hello everyone,

I am trying to set a dynamic lifecycle "origination_date" in order in inject old logs into Elastic, does anybody know how to do this ?

Here is my situation. I need monthly indexes so I can't use "parse_origination_date" in order to set the logs' age (my index format is not yyyy.mm.dd), and, if I understood correctly, the default creation date will be today's date. As I have 3 years of data I wanna keep, I can't use daily indexes (more than 1 thousand indexes is too much for just one type of logs), and I want them to be automatically deleted 3 years after the real day of the log (not 3 years after the injection, that would make me up to 6 years of logs in approximately 3 years).
Actually, any solution that would make Elastic parse the real date of the log and not the injection one would be perfect for me.
Does anybody know if it is possible to do it without changing manually the origination_date ? If so, do you know how ?

Thank you very much

2

Welcome to our community! :smiley:

Unfortunately there's not a great way to do this sort of historic ingestion with ILM, as it will only ever use the index creation date and not the date in your data.

The best option I could suggest would be to not ingest old data unless you really need it, and assign a custom policy that will delete them based on yearly expiration dates.

3

Thank you for your quick reply, as we really need to ingest old data I'm gonna use a python script and the API in order to delete old indexes.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.