# Curator action for deletion based on space and oldest data

**URL:** <https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362>\
**Category:** Elasticsearch\
**Tags:** curator\
**Created:** [May 7, 2021, 4:39am UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362 "2021-05-07T04:39:09Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [May 7, 2021, 4:39am UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/1 "2021-05-07T04:39:09Z")

</div>

Hi,

I am trying to delete the oldest data based on space. I have two kind of data.  
The name of the one is **\<some\_name\>-2021-05-04** , for example,  
and the name of the second type is like **\<some\_name\>-2021.05.04**.

My first attempt, which included only **\<some\_name\>-2021-05-04** kind of data, was something like the below and indeed deleted the oldest with '%Y-%m-%d' format:  
\> 13:

> ```
> action: delete_indices
> description: "Delete indices based on space"
> options:
> timeout_override:
> continue_if_exception: True
> disable_action: False
> ignore_empty_list: True
> filters:
> - filtertype: pattern
> kind: regex
> value: ^(pmdata|fmdata)-.*
> exclude:
> - filtertype: space
> disk_space: 1
> use_age: True
> source: name
> timestring: '%Y-%m-%d'
> exclude:
> - filtertype: pattern
> kind: prefix
> value: .kibana
> exclude: True
> 
> ```

I tried something like the below and didn't worked (I guess filters are "connected" with AND...):  
\> 13:

> ```
> action: delete_indices
> description: "Delete indices based on space"
> options:
> timeout_override:
> continue_if_exception: True
> disable_action: False
> ignore_empty_list: True
> filters:
> - filtertype: pattern
> kind: regex
> value: ^(pmdata|fmdata)-.*
> exclude:
> - filtertype: space
> disk_space: 1
> use_age: True
> source: name
> timestring: '%Y.%m.%d'
> exclude:
> - filtertype: space
> disk_space: 1
> use_age: True
> source: name
> timestring: '%Y-%m-%d'
> exclude:
> - filtertype: pattern
> kind: prefix
> value: .kibana
> exclude: True
> 
> ```

How could be an action which would include both type of timestrings?  
Thank you in advance!

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [May 7, 2021, 3:43pm UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/2 "2021-05-07T15:43:30Z")

</div>

Great question. The truth is, you can't do both in one action. But it should be pretty straightforward to duplicate the action, change the action number, and change the `timestring` value for the duplicated action.

---

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [May 10, 2021, 2:39pm UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/3 "2021-05-10T14:39:20Z")

</div>

@theuntergeek thank you for your prompt answer!  
One more question regarding the duplicated action...  
I will try to describe with an example.

In case in the action.yaml we have something like the following:

```
> 13:
> action: delete_indices 1
> description: "Delete indices based on space 1"
> options:
> timeout_override:
> continue_if_exception: True
> disable_action: False
> ignore_empty_list: True
> filters:
> - filtertype: pattern
> kind: regex
> value: ^(pmdata|fmdata)-.*
> exclude:
> - filtertype: space
> disk_space: 1
> use_age: True
> source: name
> timestring: '%Y-%m-%d'
> exclude:
> - filtertype: pattern
> kind: prefix
> value: .kibana
> exclude: True
> 14:
> action: delete_indices 2
> description: "Delete indices based on space 2"
> options:
> timeout_override:
> continue_if_exception: True
> disable_action: False
> ignore_empty_list: True
> filters:
> - filtertype: pattern
> kind: regex
> value: debug-
> exclude:
> - filtertype: space
> disk_space: 1
> use_age: True
> source: name
> timestring: '%Y.%m.%d'
> exclude:
> - filtertype: pattern
> kind: prefix
> value: .kibana
> exclude: True

```

and when the curator job is about to run, our elasticsearch is at 1.2 GB full, the curator will check first the **action 13**. If it finds enough data, will delete **pmdata & fmdata** , so after the action 13 elasticsearch will be something lesser than 1 GB. Then curator will run action 14 and since elasticsearch is less than 1 GB will not delete **debug** data.

Next time , when elasticsearch is full more than 1GB , curator will delete again pmdata and fmdata (if exist) etc.

In that way curator somehow there is possibility for curator to delete more data from the first (13) action.

Could be this a possible scenario?  
If yes, is there any suggested possible solution?

Thank you again!!!

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [May 10, 2021, 3:15pm UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/4 "2021-05-10T15:15:35Z")

</div>

If your indices `timestring`s match the index creation dates, you could use a single action but use [`source: creation_date`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/filtertype_space.html#_age_based_sorting_2) for the `space` filter instead of `source: name`. Then it won't matter what the date signature in the index name looks like.

---

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [May 11, 2021, 6:11am UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/5 "2021-05-11T06:11:26Z")

</div>

Yes this could work 🙂  
But what would be a solution in case the creation of the index in elastic doesn't match exactly the time which is part of the name of the index?  
For example the log is created at a pod at 10/05/2021 at 23:59 ( I have a field "measurement time" from where it takes the date for the name of the index ), so its name is **pmdata-2021-05-10** , but due to a delay comes to elastic at 11/05/2021 at 00:05.  
In that case will be deleted as it was from 11/05 and not from 10/05 ).

Thanks again! 🙂

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [May 11, 2021, 3:15pm UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/6 "2021-05-11T15:15:14Z")

</div>

The only other means you may have to accomplish this is if both indices use the same field name for timestamp, and it is accurate. In such a case you could use [`field_stats` based age determination](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/fe_source.html#_field_stats_based_ages_2).

---

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [May 27, 2021, 12:16pm UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/7 "2021-05-27T12:16:14Z")

</div>

I used the [`source: creation_date`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/filtertype_space.html#_age_based_sorting_2) finally. Thanks for the support!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 24, 2021, 12:17pm UTC](https://discuss.elastic.co/t/curator-action-for-deletion-based-on-space-and-oldest-data/272362/8 "2021-06-24T12:17:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.