Elasticsearch curator delete disk space

Elastic Stack Elasticsearch
1

Hi.

I'm trying to delete an index automatically using an elastic curator.
I think the index is deleted when setting as below, but I do not understand exactly how it works.

actions:
One:
action: delete_indices
options:
ignore_empty_list: True
timeout_override: 300
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: wazuh-
- filtertype: space
disk_space: 0.3
use_age: True
source: creation_date

When the individual index reaches the capacity set in disk_space, the old index is deleted from the oldest index. If the index list is deleted, it also includes logs that do not meet the capacity set in disk_space.
And I do not know how much of the entire index is deleted.

What I would like to do is to delete 10% of the total log from the old log when the index total capacity reaches the capacity set in disk_space.

I would appreciate your help.
thank you.

2

What these filters do:

  1. Select only indices beginning with wazuh-
  2. Sum the space of all indices selected by any and all previous filters, in this case, the filter in step 1. Delete all indices in excess of the specified amount of disk_space in gigabytes, starting with the oldest (determined by use_age: true).

What you're doing is deleting all wazuh- indices using in excess of 300 Mbytes. You can see the math used by running Curator with --dry-run and setting loglevel: DEBUG in the client settings yaml file.

3

At run time, the following result is displayed. If you check the store.size of the delete file, there is no index of 300mb.

curator --configcurator-config.yml delete_indices_space.yml --dry-run

2019-06-26 11:12:58,982 INFO Preparing Action ID: 1, "delete_indices"
2019-06-26 11:12:58,991 INFO Trying Action ID: 1, "delete_indices": No description given
2019-06-26 11:12:59,081 INFO DRY-RUN MODE. No changes will be made.
2019-06-26 11:12:59,081 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.03 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.04 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.05 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.06 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.07 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.08 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.09 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.10 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.11 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.12 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.13 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.14 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-alerts-3.x-2019.05.15 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.03 with arguments: {}
2019-06-26 11:12:59,082 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.05 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.06 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.07 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.08 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.09 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.10 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.11 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.12 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.13 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.14 with arguments: {}
2019-06-26 11:12:59,083 INFO DRY-RUN: delete_indices: wazuh-monitoring-3.x-2019.05.15 with arguments: {}
2019-06-26 11:12:59,084 INFO Action ID: 1, "delete_indices" completed.
2019-06-26 11:12:59,084 INFO Job completed.

If you run disk_space with 200mb, more indexes will be deleted than 300mb. I do not understand this part.

The store.size in each individual index file is up to 18mb, which is less than 200 or 300mb. I wonder if the index is selected in some criteria.
index Is not the capacity reference store.size?

I would appreciate your help.
thank you.

4

You're showing which indices were set to be deleted, but not the log lines which show how the space math is calculated.

As stated, Curator selects indices when the total sum of all indices is in excess of disk_space gigabytes, beginning with the newest (most recent) indices, as it will delete older indices in excess of the amount specified by disk_space. Curator does not look for a single index of that size. It's a sum.

5

Thanks to the friendly explanation, I understood.

But there is one thing that I do not understand.
If disk_size is set to 300mb, it will be deleted from the oldest index except 300mb index.

However, curator's index size calculation is strange.
This is indices store size as confirmed by GET _stats below.
"store": {
"size_in_bytes": 4447439230
},

I'm wondering if the size of the indices is above,
If disk_size of curator is set to 3G, deletion list should be generated, but there is no deletion list.

I do not know if the calculation method is wrong or there is something wrong with me.

I would appreciate your help.

thank you.

action: delete_indices
options:
  ignore_empty_list: True
  timeout_override: 300
  continue_if_exception: False
  disable_action: False
filters:
- filtertype: pattern
  kind: prefix
  value: wazuh-
- filtertype: space
  disk_space: 3
  use_age: True
  source: creation_date
6

Please paste the section of the DEBUG output which shows how the sizes are summed up and we can go over it together. It should be pretty clear how it sums the space consumed by the indices, starting from the most recent, so that all older indices above your 3G threshold are selected for the associated action (in this case, delete_indices).

7

I confirmed my mistake while confirming what you told me.
The size search method is invalid.
Thanks to me, I solved a problem that was difficult for me.
It has helped a lot.
thank you.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.