ILM Policy Clarification

Hello Team,

Firstly I would like to define my flow, my setup uses all three components ELK, and the data is sent from a Big Data Platform to Kafka from which logstash reads and sends it to elastic. So I planned to setup the ILM Policy at the logstash level.

I have two questions

  1. I am processing historical data and when I send the past 5 years' worth of data all is kept in HOT index because the @timstamp is today. How can I make it cold based on the date inside the file ?

  2. Below is what I have setup at the logstash config level but I have noticed that when I send all historical months the index doesn't get rollover all stays in the HOT phase.

    ilm_rollover_alias => "as-client-trans" ilm_pattern => "{now/M{yyyy.MM}}-000001" ilm_policy => "test" document_id => "%{unique_id}"

Regards,
Guru

  1. Yes. The best way to do that would be to ingest the data by age, then manually migrate the indices with the older data indexing is done
  2. That's probably related to point 1 above, and will depend on your policy