Curator action for deletion based on space and oldest data

Elastic Stack Elasticsearch
1

Hi,

I am trying to delete the oldest data based on space. I have two kind of data.
The name of the one is <some_name>-2021-05-04, for example,
and the name of the second type is like <some_name>-2021.05.04.

My first attempt, which included only <some_name>-2021-05-04 kind of data, was something like the below and indeed deleted the oldest with '%Y-%m-%d' format:
> 13:

            action: delete_indices
            description: "Delete indices based on space"
            options:
              timeout_override:
              continue_if_exception: True
              disable_action: False
              ignore_empty_list: True
            filters:
            - filtertype: pattern
              kind: regex
              value: ^(pmdata|fmdata)-.*
              exclude:
            - filtertype: space
              disk_space: 1
              use_age: True
              source: name
              timestring: '%Y-%m-%d'
              exclude:
            - filtertype: pattern
              kind: prefix
              value: .kibana
              exclude: True

I tried something like the below and didn't worked (I guess filters are "connected" with AND...):
> 13:

        action: delete_indices
        description: "Delete indices based on space"
        options:
          timeout_override:
          continue_if_exception: True
          disable_action: False
          ignore_empty_list: True
        filters:
        - filtertype: pattern
          kind: regex
          value: ^(pmdata|fmdata)-.*
          exclude:
        - filtertype: space
          disk_space: 1
          use_age: True
          source: name
          timestring: '%Y.%m.%d'
          exclude:
        - filtertype: space
          disk_space: 1
          use_age: True
          source: name
          timestring: '%Y-%m-%d'
          exclude:
        - filtertype: pattern
          kind: prefix
          value: .kibana
          exclude: True

How could be an action which would include both type of timestrings?
Thank you in advance!

2

Great question. The truth is, you can't do both in one action. But it should be pretty straightforward to duplicate the action, change the action number, and change the timestring value for the duplicated action.

3

@theuntergeek thank you for your prompt answer!
One more question regarding the duplicated action...
I will try to describe with an example.

In case in the action.yaml we have something like the following:

> 13:
>             action: delete_indices 1
>             description: "Delete indices based on space 1"
>             options:
>               timeout_override:
>               continue_if_exception: True
>               disable_action: False
>               ignore_empty_list: True
>             filters:
>             - filtertype: pattern
>               kind: regex
>               value: ^(pmdata|fmdata)-.*
>               exclude:
>             - filtertype: space
>               disk_space: 1
>               use_age: True
>               source: name
>               timestring: '%Y-%m-%d'
>               exclude:
>             - filtertype: pattern
>               kind: prefix
>               value: .kibana
>               exclude: True
> 14:
>             action: delete_indices 2
>             description: "Delete indices based on space 2"
>             options:
>               timeout_override:
>               continue_if_exception: True
>               disable_action: False
>               ignore_empty_list: True
>             filters:
>             - filtertype: pattern
>               kind: regex
>               value: debug-
>               exclude:
>             - filtertype: space
>               disk_space: 1
>               use_age: True
>               source: name
>               timestring: '%Y.%m.%d'
>               exclude:
>             - filtertype: pattern
>               kind: prefix
>               value: .kibana
>               exclude: True

and when the curator job is about to run, our elasticsearch is at 1.2 GB full, the curator will check first the action 13. If it finds enough data, will delete pmdata & fmdata , so after the action 13 elasticsearch will be something lesser than 1 GB. Then curator will run action 14 and since elasticsearch is less than 1 GB will not delete debug data.

Next time , when elasticsearch is full more than 1GB , curator will delete again pmdata and fmdata (if exist) etc.

In that way curator somehow there is possibility for curator to delete more data from the first (13) action.

Could be this a possible scenario?
If yes, is there any suggested possible solution?

Thank you again!!!

4

If your indices timestrings match the index creation dates, you could use a single action but use source: creation_date for the space filter instead of source: name. Then it won't matter what the date signature in the index name looks like.

5

Yes this could work :slight_smile:
But what would be a solution in case the creation of the index in elastic doesn't match exactly the time which is part of the name of the index?
For example the log is created at a pod at 10/05/2021 at 23:59 ( I have a field "measurement time" from where it takes the date for the name of the index ), so its name is pmdata-2021-05-10 , but due to a delay comes to elastic at 11/05/2021 at 00:05.
In that case will be deleted as it was from 11/05 and not from 10/05 ).

Thanks again! :slight_smile:

6

The only other means you may have to accomplish this is if both indices use the same field name for timestamp, and it is accurate. In such a case you could use field_stats based age determination.

7

I used the source: creation_date finally. Thanks for the support!!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.