# ILM Daily indices with date retention

**URL:** <https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [May 27, 2020, 10:24am UTC](https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520 "2020-05-27T10:24:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jiri\_Petak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jiri_petak/32/69174_2.png) [@Jiri\_Petak](https://discuss.elastic.co/u/Jiri_Petak)\
**Post date:** [May 27, 2020, 10:24am UTC](https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520/1 "2020-05-27T10:24:07Z")

</div>

Hi, i wanna ask about ILM creating daily indices. Or more likely i am confused about ILM concept with indices based on size. For example: We are logging infrastructure logs from syslog with filebeat. When i use ILM like:

```auto
  "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb"
            }
          }
        },
        "delete" : {
          "min_age" : "5d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }

```

Which means that index is rolled in 50GB to new one and index is deleted after 5days. My daily increase on that index is about 5GB, which means that first roll will be in 10 days, but i need only 5 days retention. How ILM will work when i reach that 5 days? It will delete current index and create new one? That will delete all data and i will have no data for past 5 days. Or elastic will wait until that index will have 50G, than rolled him and delete it? Or it will wait another 5 days before deleting? I am little bit confused how can i achive that. I just want to have 5 days of data available no more no less.

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 27, 2020, 10:40am UTC](https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520/2 "2020-05-27T10:40:28Z")

</div>

Hi,  
Instead of the max\_size you can also use the max\_age option and set it to 1d.  
So 1 day after the index is created (I will call it logs-day01) elasticsearch will create a new index, for example logs-day02.  
This would happen every day so the day after creating the logs-day02 it will create logs-day03.  
So now you have a index for each day and you can delete a index after 5 days using ilm.  
That's the way how you can archive what you want.

You will find many informations for this [here](https://www.elastic.co/webinars/index-lifecycle-management-for-time-seres-data-in-elasticsearch)

---

<div class="post-metadata">

**Author:** ![Jiri\_Petak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jiri_petak/32/69174_2.png) [@Jiri\_Petak](https://discuss.elastic.co/u/Jiri_Petak)\
**Post date:** [May 28, 2020, 7:32am UTC](https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520/3 "2020-05-28T07:32:50Z")

</div>

Thanks for the link on webinar. I tryed that max-age with 1 day, and i set settings.index.lifecycle.parse\_origination\_date on index template, but it's not rolled by days:

```auto
"filebeat-7.7.0-system-2020.05.27-000002" : {
      "index" : "filebeat-7.7.0-system-2020.05.27-000002",
      "managed" : true,
      "policy" : "filebeat-7.7.0-system",
      "lifecycle_date_millis" : 1590537600000,
      "age" : "1.3d",
      "phase" : "hot",
      "phase_time_millis" : 1590578472730,
      "action" : "rollover",
      "action_time_millis" : 1590579071927,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1590579071927,
      "phase_execution" : {
        "policy" : "filebeat-7.7.0-system",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "version" : 2,
        "modified_date_in_millis" : 1590576262914
      }
    },

```

The age is there 1.3d and it's index from yesterday, but todays data still going to that index instead of new one.

---

<div class="post-metadata">

**Author:** ![KoettingSimon](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@KoettingSimon](https://discuss.elastic.co/u/KoettingSimon)\
**Post date:** [May 28, 2020, 7:55am UTC](https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520/4 "2020-05-28T07:55:16Z")

</div>

Can you find any errors in the elasticsearch log?

---

<div class="post-metadata">

**Author:** ![Jiri\_Petak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jiri_petak/32/69174_2.png) [@Jiri\_Petak](https://discuss.elastic.co/u/Jiri_Petak)\
**Post date:** [May 28, 2020, 8:02am UTC](https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520/5 "2020-05-28T08:02:39Z")

</div>

No, there are no errors in logs. But i think that instead of "age" of index it depends on "phase\_time\_millis" which is 20 hours now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 25, 2020, 8:02am UTC](https://discuss.elastic.co/t/ilm-daily-indices-with-date-retention/234520/6 "2020-06-25T08:02:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.