ILM Daily indices with date retention

Elastic Stack Elasticsearch
1

Hi, i wanna ask about ILM creating daily indices. Or more likely i am confused about ILM concept with indices based on size. For example: We are logging infrastructure logs from syslog with filebeat. When i use ILM like:

  "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb"
            }
          }
        },
        "delete" : {
          "min_age" : "5d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }

Which means that index is rolled in 50GB to new one and index is deleted after 5days. My daily increase on that index is about 5GB, which means that first roll will be in 10 days, but i need only 5 days retention. How ILM will work when i reach that 5 days? It will delete current index and create new one? That will delete all data and i will have no data for past 5 days. Or elastic will wait until that index will have 50G, than rolled him and delete it? Or it will wait another 5 days before deleting? I am little bit confused how can i achive that. I just want to have 5 days of data available no more no less.

2

Hi,
Instead of the max_size you can also use the max_age option and set it to 1d.
So 1 day after the index is created (I will call it logs-day01) elasticsearch will create a new index, for example logs-day02.
This would happen every day so the day after creating the logs-day02 it will create logs-day03.
So now you have a index for each day and you can delete a index after 5 days using ilm.
That's the way how you can archive what you want.

You will find many informations for this here

3

Thanks for the link on webinar. I tryed that max-age with 1 day, and i set settings.index.lifecycle.parse_origination_date on index template, but it's not rolled by days:

"filebeat-7.7.0-system-2020.05.27-000002" : {
      "index" : "filebeat-7.7.0-system-2020.05.27-000002",
      "managed" : true,
      "policy" : "filebeat-7.7.0-system",
      "lifecycle_date_millis" : 1590537600000,
      "age" : "1.3d",
      "phase" : "hot",
      "phase_time_millis" : 1590578472730,
      "action" : "rollover",
      "action_time_millis" : 1590579071927,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1590579071927,
      "phase_execution" : {
        "policy" : "filebeat-7.7.0-system",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "version" : 2,
        "modified_date_in_millis" : 1590576262914
      }
    },

The age is there 1.3d and it's index from yesterday, but todays data still going to that index instead of new one.

4

Can you find any errors in the elasticsearch log?

5

No, there are no errors in logs. But i think that instead of "age" of index it depends on "phase_time_millis" which is 20 hours now.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.