Curator's delete by disk space doesn't remove logs

[Wojter (Wojciech)]

Hi there!

There is one thing that worries me about Curator tool. I'd want to create a cap of disk space for elasticsearch. There exists really great filter for it which is space. Now I don't know if I use it properly, because it seems it doesn't remove any logs. The age filter (to remove logs older than i.e. 1 day) works flawlessly, but I can't configure space filter properly. My config goes as below:

action: delete_indices
description: >-
  Delete older than day or when index size exceeds 1 GB
options:
  ignore_empty_list: True
  continue_if_exception: False
  disable_action: False
filters:
- filtertype: age
  source: name
  direction: older
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 1
  exclude:
- filtertype: space
  disk_space: 1
  reverse: True
  use_age: False
  source: creation_date
  timestring:
  field:
  stats_result:
  exclude:

Maybe I'm doing something wrong, maybe mixing the filters or something, but I cannot get it to work. I invoke curator in cron every hour. I have installed curator version 4.2. It works - I mean the replicas are deleted, segments are merged, but this actions above doesn't work (ES disk usage grows and grows). Could anybody help me figure the problem out?

Cheers!

[Wojter (Wojciech)]

Can somebody help with this?

[theuntergeek (Aaron Mildenstein)]

@Wojter

action: delete_indices
description: >-
Delete older than day or when index size exceeds 1 GB
options:
ignore_empty_list: True
continue_if_exception: False
disable_action: False
filters:

• filtertype: age
  source: name
  direction: older
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 1
  exclude:
• filtertype: space
  disk_space: 1
  reverse: True
  use_age: False
  source: creation_date
  timestring:
  field:
  stats_result:
  exclude:

I used your exact configuration file with --dry-run mode, and logging at DEBUG level. It works just fine for me:

2017-03-27 09:47:49,889 DEBUG          curator.indexlist           working_list:193  Generating working list of indices
2017-03-27 09:47:49,889 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.26 is actionable and remains in the list.
2017-03-27 09:47:49,889 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.26, summed disk usage is 3.2GB and disk limit is 1.0GB.
2017-03-27 09:47:49,889 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.25 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.25, summed disk usage is 6.3GB and disk limit is 1.0GB.
2017-03-27 09:47:49,890 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.24 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.24, summed disk usage is 9.5GB and disk limit is 1.0GB.
2017-03-27 09:47:49,890 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.23 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.23, summed disk usage is 12.7GB and disk limit is 1.0GB.
2017-03-27 09:47:49,890 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.22 is actionable and remains in the list.
2017-03-27 09:47:49,890 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.22, summed disk usage is 15.9GB and disk limit is 1.0GB.
2017-03-27 09:47:49,891 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.21 is actionable and remains in the list.
2017-03-27 09:47:49,891 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.21, summed disk usage is 19.0GB and disk limit is 1.0GB.
2017-03-27 09:47:49,891 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.20 is actionable and remains in the list.
2017-03-27 09:47:49,891 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.20, summed disk usage is 22.2GB and disk limit is 1.0GB.
2017-03-27 09:47:49,891 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-2017.03.19 is actionable and remains in the list.
2017-03-27 09:47:49,891 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: metricbeat-2017.03.19, summed disk usage is 25.3GB and disk limit is 1.0GB.

I can see that every index above 1.0GB of usage is being selected for deletion.

2017-03-27 09:47:49,907 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.19 with arguments: {}
2017-03-27 09:47:49,907 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.20 with arguments: {}
2017-03-27 09:47:49,907 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.21 with arguments: {}
2017-03-27 09:47:49,907 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.22 with arguments: {}
2017-03-27 09:47:49,907 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.23 with arguments: {}
2017-03-27 09:47:49,907 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.24 with arguments: {}
2017-03-27 09:47:49,908 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.25 with arguments: {}
2017-03-27 09:47:49,908 INFO               curator.utils           show_dry_run:646  DRY-RUN: delete_indices: metricbeat-2017.03.26 with arguments: {}

Please note that the actual output includes many more indices of different names. I am forced to limit the output here to not spam the post.

Have you tried running with DEBUG level output? What do you see?

[Wojter (Wojciech)]

Hi, thanks for reply. Right now I'm moving curator onto docker container, but when I'll do it, I'll test this and report what happens.

EDIT:
Yeah, seems all is set and actions are done. Hovewer I have a question. You mentioned that every index above 1GB is being selected for deletion. I'd like the disk space to be a sum of disk usage of all indices and then remove the oldest. From what I've read in the docs this should work as I mentioned. So how it really works?

EDIT:
I ran in debug mode and everything seems to work as I suspected - curator calculates total disk space used by index and compares with the limit, then removes the oldest indices. So everything works well, thanks!