ILM Policy Clarification

Elastic Stack Kibana
1

Hello Team,

Firstly I would like to define my flow, my setup uses all three components ELK, and the data is sent from a Big Data Platform to Kafka from which logstash reads and sends it to elastic. So I planned to setup the ILM Policy at the logstash level.

I have two questions

  1. I am processing historical data and when I send the past 5 years' worth of data all is kept in HOT index because the @timstamp is today. How can I make it cold based on the date inside the file ?

  2. Below is what I have setup at the logstash config level but I have noticed that when I send all historical months the index doesn't get rollover all stays in the HOT phase.

    ilm_rollover_alias => "as-client-trans" ilm_pattern => "{now/M{yyyy.MM}}-000001" ilm_policy => "test" document_id => "%{unique_id}"

Regards,
Guru

2
  1. Yes. The best way to do that would be to ingest the data by age, then manually migrate the indices with the older data indexing is done
  2. That's probably related to point 1 above, and will depend on your policy
4

So I need more clarity on when you say manually migrate the indices, do you mean first send the data into seperate indices by month and year , like transaction_2022_01, transaction_2022_02 , transaction_2022_03 ... and then how to manually set it to cold phase ?

5
6

I will try that, thanks @warkolm

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.