Elasticsearch Curator: Empty list

AJCandfield · June 18, 2020, 8:53am

Hi all,

We decided to install the Curator to delete indices with respect to age and size.

On our first run the tool was able to delete the Logstash indices that matched the criteria but when we specified any other index it would output this message:

    2020-06-18 08:43:09,446 INFO      Preparing Action ID: 1, "delete_indices"
    2020-06-18 08:43:09,447 INFO      Creating client object and testing connection
    2020-06-18 08:43:09,462 INFO      Instantiating client object
    2020-06-18 08:43:09,462 INFO      Testing client connectivity
    2020-06-18 08:43:09,473 INFO      Successfully created Elasticsearch client object with provided settings
    2020-06-18 08:43:09,477 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 30 days.
    2020-06-18 08:43:10,140 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
    2020-06-18 08:43:10,140 INFO      Action ID: 1, "delete_indices" completed.
    2020-06-18 08:43:10,140 INFO      Job completed.

In this example, we tried to select all "cloudtrail" indices:

---    
    actions:
          1:
            action: delete_indices
            description: >-
              Delete indices older than 30 days.
            options:
              ignore_empty_list: true
              disable_action: false
            filters:
            - filtertype: pattern
              kind: prefix
              value: cloudtrail
            - filtertype: age
              source: name
              direction: older
              timestring: '%Y.%m.%d'
              unit: days
              unit_count: 15

Can anyone help us explain why it doesn't work?

Thanks for your help,

Alex

theuntergeek · June 18, 2020, 1:35pm

Without debug logging enabled, I can't see what actual indices exist in your system, which reduces me to a best guess answer. In this case, that best-guess is that you do not have any cloudtrail-%Y.%m.%d indices older than 15 days (your description says 30, but the unit_count says 15). Another possibility is that your index timestring is connected by hyphens rather than periods, e.g. %Y-%m-%d.

Regardless, the reason you see an empty list error is that after both filters have passed over your index list, there are none remaining to act on. If you set loglevel: DEBUG, Curator will be much more verbose and show what it does see, and why it did or did not filter your indices.

AJCandfield · June 18, 2020, 2:56pm

Hi @theuntergeek,

Thanks for getting back to me.

I will try a few more times after enabling the Debugging mode. I will keep you posted.

Thanks,

Alex

AJCandfield · June 19, 2020, 5:20pm

Hi again @theuntergeek,

I followed your advice and I definitely got more info by setting the log level to DEBUG.

I think we're pretty close to a solution. This is what I see from the logs.

The curator selects all the indices:

...
Generating working list of indices
2020-06-19 17:06:56,898 DEBUG          curator.indexlist     iterate_over_stats:126  Index: cloudtrail-2020.05.13  Size: 7.4GB  Docs: 37510
2020-06-19 17:06:56,898 DEBUG          curator.indexlist     iterate_over_stats:126  Index: cloudtrail-2020.05.12  Size: 15.8GB  Docs: 79355
2020-06-19 17:06:56,898 DEBUG          curator.indexlist     iterate_over_stats:126  Index: cloudtrail-2020.05.11  Size: 4.1GB  Docs: 25720
2020-06-19 17:06:56,898 DEBUG          curator.indexlist     iterate_over_stats:126  Index: cloudtrail-2020.05.10  Size: 73.4MB  Docs: 6257
2020-06-19 17:06:56,898 DEBUG          curator.indexlist     iterate_over_stats:126  Index: metricbeat-7.5.0-2020.04.15-000004  Size: 276.9MB  Docs: 913723
2020-06-19 17:06:56,898 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .kibana_task_manager_1  Size: 27.9KB  Docs: 2
2020-06-19 17:06:56,898 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-es-7-2020.06.13  Size: 616.3MB  Docs: 1868761
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-es-7-2020.06.14  Size: 444.4MB  Docs: 1344214
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-es-7-2020.06.15  Size: 622.3MB  Docs: 1885249
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-es-7-2020.06.16  Size: 624.2MB  Docs: 1894112
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-es-7-2020.06.17  Size: 444.0MB  Docs: 1265658
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-es-7-2020.06.18  Size: 398.5MB  Docs: 969073
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-es-7-2020.06.19  Size: 543.0MB  Docs: 678946
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-kibana-7-2020.06.14  Size: 1.3MB  Docs: 6173
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-kibana-7-2020.06.13  Size: 1.9MB  Docs: 8635
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-kibana-7-2020.06.16  Size: 1.9MB  Docs: 8639
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: .monitoring-kibana-7-2020.06.15  Size: 1.9MB  Docs: 8639
2020-06-19 17:06:56,899 DEBUG          curator.indexlist     iterate_over_stats:126  Index: metricbeat-7.5.0-2020.04.08-000003  Size: 271.8MB  Docs: 899106
...

It applies the regex and age filters:

...
Filtering indices by regex
2020-06-19 17:06:56,908 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2020-06-19 17:06:56,908 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2020-06-19 17:06:56,908 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: logstash-2020.05.25
2020-06-19 17:06:56,908 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2020.05.25 is not actionable, removing from list.
2020-06-19 17:06:56,908 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: cloudtrail-2020.04.26
2020-06-19 17:06:56,908 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.26 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: logstash-2020.06.13
2020-06-19 17:06:56,909 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2020.06.13 is not actionable, removing from list.
2020-06-19 17:06:56,909 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: logstash-2020.06.15
2020-06-19 17:06:56,909 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2020.06.15 is not actionable, removing from list.
2020-06-19 17:06:56,909 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: cloudtrail-2020.04.18
2020-06-19 17:06:56,909 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.18 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: cloudtrail-2020.04.25
2020-06-19 17:06:56,909 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.25 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: cloudtrail-2020.04.20
2020-06-19 17:06:56,909 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.20 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: cloudtrail-2020.04.01
2020-06-19 17:06:56,909 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.01 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG          curator.indexlist        filter_by_regex:451  Filter by regex: Index: logstash-2020.06.12
...

Filtering indices by age
2020-06-19 17:06:56,923 DEBUG          curator.indexlist   _get_name_based_ages:280  Getting ages of indices by "name"
2020-06-19 17:06:56,923 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2020-06-19 17:06:56,923 DEBUG              curator.utils         get_date_regex:200  regex = \d{4}\.\d{2}\.\d{2}
2020-06-19 17:06:56,924 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2020-06-19 17:06:56,929 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2020-06-19 17:06:56,929 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.26 is actionable and remains in the list.
2020-06-19 17:06:56,929 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.04.26" age (1587859200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,929 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.18 is actionable and remains in the list.
2020-06-19 17:06:56,929 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.04.18" age (1587168000), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,929 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.25 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.04.25" age (1587772800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.20 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.04.20" age (1587340800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.01 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.04.01" age (1585699200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.06 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.04.06" age (1586131200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.03.31 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.03.31" age (1585612800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG          curator.indexlist           __actionable:35   Index cloudtrail-2020.04.12 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "cloudtrail-2020.04.12" age (1586649600), direction: "older", point of reference, (1589994416)

What I don't get is the last bit, where it seems there's another filter that eventually removes all the "cloudtrail"-related indices:

Filtering indices with index.lifecycle.name
2020-06-19 17:06:56,950 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.03.28 is not actionable, removing from list.
2020-06-19 17:06:56,950 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.03.28 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.04.24 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.04.24 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.05.13 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.05.13 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.05.12 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.05.12 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.05.04 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.05.04 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.04.06 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.04.06 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.04.22 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.04.22 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.04.28 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.04.28 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG          curator.indexlist       __not_actionable:39   Index cloudtrail-2020.04.26 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: cloudtrail-2020.04.26 has index.lifecycle.name cloudtrail

Could you let me know what that last filter is about and how I can make the indices pass it?

BTW, I took the "cloudtrail" index as an example but it happens with all the other ones except for "logstash".

Let me know if you need any more info.

Thanks a lot,

Alex

AJCandfield · June 19, 2020, 6:03pm

After a Google search, I eventually found the solution which I will post here.

As described in the documentation:

Curator will not act on any index associated with an ILM policy without setting allow_ilm_indices to true .

All I had to do was include the following line to my action.yml file:

...
options:
    allow_ilm_indices: true
...

That worked a charm.

Thanks for pointing me in the right direction and thanks for creating the curator!

Alex

theuntergeek · June 23, 2020, 2:18pm

Glad you found this. I have been out of office since Thursday last week and did not have a chance to point you to allow_ilm_indices.

system · July 21, 2020, 2:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Curator - Not deleting indices Elasticsearch	8	2059	August 9, 2019
Curator no indices found Elasticsearch	14	1780	July 29, 2019
Elasticsearch curator not deleting indices Elasticsearch	3	1776	January 2, 2018
Regarding to curator delete indices older than 5 days Elasticsearch	27	12959	April 4, 2017
Curator action, delete_indices problem Elasticsearch	5	2285	July 5, 2017

Elasticsearch Curator: Empty list

Related topics