Monthly rolling period with Curator

Elastic Stack Elasticsearch
1

Hello,

Is it possible to define a monthly rolling period policy with Curator ?
I'm using daily indices with the following pattern metrics_%{YYYY.MM.dd}. Today index is metrics_2018.08.06.

What I want is to keep 1 month of metrics (no matter how many days there's in the previous month).

As far as I know, Curator is truncating the date to the most recent whole unit.
Consider the following actions.yml:

  0:
    action: delete_indices
    description : DELETE INDICES metrics daily
    options:
      ignore_empty_list: True
    filters:
      - kind: regex
        filtertype: pattern
        value: ^metrics_\d{4}\.\d{2}\.\d{2}$
      - filtertype: period
        source: name
        timestring: '%Y.%m.%d'
        range_from: -1
        range_to: 0
        unit: months
        exclude: True

If I'm running the command today, Curator will subtract a month (2018/05/08) then truncate/round the date (2018/05/01). Since I'm using exclude: True, every index older than 2018/05/01 will be removed.

In my case, I want to remove indices that are older than 2018/05/08.

Of course I can use unit: days but in this case I will have to use either 29, 30 or 31 days depending on how many days there's in the previous month.

Maybe we should add an option to control if we want to truncate/round the date or not ?

Does it make sense ?

Thanks,
Guillaume

I'm using Curator 5.4.1:

$ curator --version
curator, version 5.4.1
2

What you're asking is not currently possible in Curator. You can keep entire periods with the period filter, but there's no way to keep 28, 30, or 31 day periods deterministically. How accurate would that be anyway? What would that actually show? The 8th of June to the 8th of May is more a month than the 8th of March back to the 8th of February?

It seems quite arbitrary. I think that the "bank standard" 30 days as month is an acceptable window with days, which was the original Curator use case.

However, if you wanted to do something to still make this happen, you could:

  1. Use some clever shell scripting to set the unit_count with a shell variable.
  2. Just set the time picker in Kibana to only show month day to month day periods and set unit_count to 32 days to always have at least 31 days.
  3. Something else clever.

This just seems much more of a display-side problem than one to handle with Curator.

3

Interesting, I think you're right :smile:
Some of our users were confused that sometimes they could browser back to the 7th or the 6th (because the previous month was only 30 or 29 days) and sometimes not.

From a cluster point of view (shards, indices, storage) it's indeed a complete nonsense to keep either 31, 30 or 29 indices depending on the month.

Thanks for your quick reply :+1:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.