# Curator's delete by disk space doesn't remove logs

**URL:** <https://discuss.elastic.co/t/curators-delete-by-disk-space-doesnt-remove-logs/79778>\
**Category:** Elasticsearch\
**Created:** [March 23, 2017, 4:32pm UTC](https://discuss.elastic.co/t/curators-delete-by-disk-space-doesnt-remove-logs/79778 "2017-03-23T16:32:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\
**Post date:** [March 23, 2017, 4:32pm UTC](https://discuss.elastic.co/t/curators-delete-by-disk-space-doesnt-remove-logs/79778/1 "2017-03-23T16:32:23Z")

</div>

Hi there!

There is one thing that worries me about Curator tool. I'd want to create a cap of disk space for elasticsearch. There exists really great filter for it which is space. Now I don't know if I use it properly, because it seems it doesn't remove any logs. The age filter (to remove logs older than i.e. 1 day) works flawlessly, but I can't configure space filter properly. My config goes as below:

```
action: delete_indices
description: >-
  Delete older than day or when index size exceeds 1 GB
options:
  ignore_empty_list: True
  continue_if_exception: False
  disable_action: False
filters:
- filtertype: age
  source: name
  direction: older
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 1
  exclude:
- filtertype: space
  disk_space: 1
  reverse: True
  use_age: False
  source: creation_date
  timestring:
  field:
  stats_result:
  exclude:

```

Maybe I'm doing something wrong, maybe mixing the filters or something, but I cannot get it to work. I invoke curator in cron every hour. I have installed curator version 4.2. It works - I mean the replicas are deleted, segments are merged, but this actions above doesn't work (ES disk usage grows and grows). Could anybody help me figure the problem out?

Cheers!

---

<div class="post-metadata">

**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\
**Post date:** [March 27, 2017, 8:52am UTC](https://discuss.elastic.co/t/curators-delete-by-disk-space-doesnt-remove-logs/79778/2 "2017-03-27T08:52:03Z")

</div>

Can somebody help with this?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [March 27, 2017, 3:51pm UTC](https://discuss.elastic.co/t/curators-delete-by-disk-space-doesnt-remove-logs/79778/3 "2017-03-27T15:51:58Z")

</div>

> [@Wojter](#):
>
> action: delete\_indices  
> description: \>-  
> Delete older than day or when index size exceeds 1 GB  
> options:  
> ignore\_empty\_list: True  
> continue\_if\_exception: False  
> disable\_action: False  
> filters:
> 
> - filtertype: age  
> source: name  
> direction: older  
> timestring: '%Y.%m.%d'  
> unit: days  
> unit\_count: 1  
> exclude:
> - filtertype: space  
> disk\_space: 1  
> reverse: True  
> use\_age: False  
> source: creation\_date  
> timestring:  
> field:  
> stats\_result:  
> exclude:

I used your _exact_ configuration file with `--dry-run` mode, and logging at DEBUG level. It works just fine for me:

```auto
2017-03-27 09:47:49,889 DEBUG curator.indexlist working_list:193 Generating working list of indices
2017-03-27 09:47:49,889 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.26 is actionable and remains in the list.
2017-03-27 09:47:49,889 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.26, summed disk usage is 3.2GB and disk limit is 1.0GB.
2017-03-27 09:47:49,889 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.25 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.25, summed disk usage is 6.3GB and disk limit is 1.0GB.
2017-03-27 09:47:49,890 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.24 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.24, summed disk usage is 9.5GB and disk limit is 1.0GB.
2017-03-27 09:47:49,890 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.23 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.23, summed disk usage is 12.7GB and disk limit is 1.0GB.
2017-03-27 09:47:49,890 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.22 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.22, summed disk usage is 15.9GB and disk limit is 1.0GB.
2017-03-27 09:47:49,891 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.21 is actionable and remains in the list.
2017-03-27 09:47:49,891 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.21, summed disk usage is 19.0GB and disk limit is 1.0GB.
2017-03-27 09:47:49,891 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.20 is actionable and remains in the list.
2017-03-27 09:47:49,891 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.20, summed disk usage is 22.2GB and disk limit is 1.0GB.
2017-03-27 09:47:49,891 DEBUG curator.indexlist __actionable:35 Index metricbeat-2017.03.19 is actionable and remains in the list.
2017-03-27 09:47:49,891 DEBUG curator.indexlist __excludify:58 Remains in actionable list: metricbeat-2017.03.19, summed disk usage is 25.3GB and disk limit is 1.0GB.

```

I can see that every index above 1.0GB of usage is being selected for deletion.

```auto
2017-03-27 09:47:49,907 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.19 with arguments: {}
2017-03-27 09:47:49,907 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.20 with arguments: {}
2017-03-27 09:47:49,907 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.21 with arguments: {}
2017-03-27 09:47:49,907 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.22 with arguments: {}
2017-03-27 09:47:49,907 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.23 with arguments: {}
2017-03-27 09:47:49,907 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.24 with arguments: {}
2017-03-27 09:47:49,908 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.25 with arguments: {}
2017-03-27 09:47:49,908 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: metricbeat-2017.03.26 with arguments: {}

```

Please note that the actual output includes _many_ more indices of different names. I am forced to limit the output here to not spam the post.

Have you tried running with DEBUG level output? What do you see?

---

<div class="post-metadata">

**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\
**Post date:** [March 27, 2017, 4:45pm UTC](https://discuss.elastic.co/t/curators-delete-by-disk-space-doesnt-remove-logs/79778/4 "2017-03-27T16:45:53Z")

</div>

Hi, thanks for reply. Right now I'm moving curator onto docker container, but when I'll do it, I'll test this and report what happens.

EDIT:  
Yeah, seems all is set and actions are done. Hovewer I have a question. You mentioned that every index above 1GB is being selected for deletion. I'd like the disk space to be a sum of disk usage of all indices and then remove the oldest. From what I've read in the docs this should work as I mentioned. So how it really works?

EDIT:  
I ran in debug mode and everything seems to work as I suspected - curator calculates total disk space used by index and compares with the limit, then removes the oldest indices. So everything works well, thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 24, 2017, 4:46pm UTC](https://discuss.elastic.co/t/curators-delete-by-disk-space-doesnt-remove-logs/79778/5 "2017-04-24T16:46:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.