ILM Policy Clarification

[gbandasha (Gurjot Kaur Bandasha)]

Hello Team,

Firstly I would like to define my flow, my setup uses all three components ELK, and the data is sent from a Big Data Platform to Kafka from which logstash reads and sends it to elastic. So I planned to setup the ILM Policy at the logstash level.

I have two questions

1. I am processing historical data and when I send the past 5 years' worth of data all is kept in HOT index because the @timstamp is today. How can I make it cold based on the date inside the file ?

2. Below is what I have setup at the logstash config level but I have noticed that when I send all historical months the index doesn't get rollover all stays in the HOT phase.

   ilm_rollover_alias => "as-client-trans" ilm_pattern => "{now/M{yyyy.MM}}-000001" ilm_policy => "test" document_id => "%{unique_id}"

Regards,
Guru

[warkolm (Mark Walkom)]

1. Yes. The best way to do that would be to ingest the data by age, then manually migrate the indices with the older data indexing is done
2. That's probably related to point 1 above, and will depend on your policy

[gbandasha (Gurjot Kaur Bandasha)]

@warkolm

Yes. The best way to do that would be to ingest the data by age, then manually migrate the indices with the older data indexing is done

So I need more clarity on when you say manually migrate the indices, do you mean first send the data into seperate indices by month and year , like transaction_2022_01, transaction_2022_02 , transaction_2022_03 ... and then how to manually set it to cold phase ?

[warkolm (Mark Walkom)]

@gbandasha

and then how to manually set it to cold phase

Move to lifecycle step API | Elasticsearch Guide [8.5] | Elastic

[gbandasha (Gurjot Kaur Bandasha)]

I will try that, thanks @warkolm