# Elasticsearch Curator: Empty list

**URL:** <https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599>\
**Category:** Elasticsearch\
**Tags:** curator\
**Created:** [June 18, 2020, 8:53am UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599 "2020-06-18T08:53:25Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![AJCandfield](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@AJCandfield](https://discuss.elastic.co/u/AJCandfield)\
**Post date:** [June 18, 2020, 8:53am UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599/1 "2020-06-18T08:53:25Z")

</div>

Hi all,

We decided to install the Curator to delete indices with respect to age and size.

On our first run the tool was able to delete the Logstash indices that matched the criteria but when we specified any other index it would output this message:

```auto
    2020-06-18 08:43:09,446 INFO Preparing Action ID: 1, "delete_indices"
    2020-06-18 08:43:09,447 INFO Creating client object and testing connection
    2020-06-18 08:43:09,462 INFO Instantiating client object
    2020-06-18 08:43:09,462 INFO Testing client connectivity
    2020-06-18 08:43:09,473 INFO Successfully created Elasticsearch client object with provided settings
    2020-06-18 08:43:09,477 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 30 days.
    2020-06-18 08:43:10,140 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
    2020-06-18 08:43:10,140 INFO Action ID: 1, "delete_indices" completed.
    2020-06-18 08:43:10,140 INFO Job completed.

```

In this example, we tried to select all "cloudtrail" indices:

```auto
---    
    actions:
          1:
            action: delete_indices
            description: >-
              Delete indices older than 30 days.
            options:
              ignore_empty_list: true
              disable_action: false
            filters:
            - filtertype: pattern
              kind: prefix
              value: cloudtrail
            - filtertype: age
              source: name
              direction: older
              timestring: '%Y.%m.%d'
              unit: days
              unit_count: 15

```

Can anyone help us explain why it doesn't work?

Thanks for your help,

Alex

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [June 18, 2020, 1:35pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599/2 "2020-06-18T13:35:14Z")

</div>

Without debug logging enabled, I can't see what actual indices exist in your system, which reduces me to a best guess answer. In this case, that best-guess is that you do not have any `cloudtrail-%Y.%m.%d` indices older than 15 days (your `description` says `30`, but the `unit_count` says `15`). Another possibility is that your index `timestring` is connected by hyphens rather than periods, e.g. `%Y-%m-%d`.

Regardless, the reason you see an `empty list` error is that after both filters have passed over your index list, there are none remaining to act on. If you set `loglevel: DEBUG`, Curator will be much more verbose and show what it does see, and why it did or did not filter your indices.

---

<div class="post-metadata">

**Author:** ![AJCandfield](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@AJCandfield](https://discuss.elastic.co/u/AJCandfield)\
**Post date:** [June 18, 2020, 2:56pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599/3 "2020-06-18T14:56:04Z")

</div>

Hi @theuntergeek,

Thanks for getting back to me.

I will try a few more times after enabling the Debugging mode. I will keep you posted.

Thanks,

Alex

---

<div class="post-metadata">

**Author:** ![AJCandfield](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@AJCandfield](https://discuss.elastic.co/u/AJCandfield)\
**Post date:** [June 19, 2020, 5:20pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599/4 "2020-06-19T17:20:33Z")

</div>

Hi again @theuntergeek,

I followed your advice and I definitely got more info by setting the log level to DEBUG.

I think we're pretty close to a solution. This is what I see from the logs.

The curator selects all the indices:

```auto
...
Generating working list of indices
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.13 Size: 7.4GB Docs: 37510
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.12 Size: 15.8GB Docs: 79355
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.11 Size: 4.1GB Docs: 25720
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.10 Size: 73.4MB Docs: 6257
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: metricbeat-7.5.0-2020.04.15-000004 Size: 276.9MB Docs: 913723
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: .kibana_task_manager_1 Size: 27.9KB Docs: 2
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.13 Size: 616.3MB Docs: 1868761
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.14 Size: 444.4MB Docs: 1344214
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.15 Size: 622.3MB Docs: 1885249
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.16 Size: 624.2MB Docs: 1894112
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.17 Size: 444.0MB Docs: 1265658
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.18 Size: 398.5MB Docs: 969073
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.19 Size: 543.0MB Docs: 678946
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.14 Size: 1.3MB Docs: 6173
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.13 Size: 1.9MB Docs: 8635
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.16 Size: 1.9MB Docs: 8639
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.15 Size: 1.9MB Docs: 8639
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: metricbeat-7.5.0-2020.04.08-000003 Size: 271.8MB Docs: 899106
...

```

It applies the regex and age filters:

```auto
...
Filtering indices by regex
2020-06-19 17:06:56,908 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2020-06-19 17:06:56,908 DEBUG curator.indexlist working_list:237 Generating working list of indices
2020-06-19 17:06:56,908 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.05.25
2020-06-19 17:06:56,908 DEBUG curator.indexlist __not_actionable:39 Index logstash-2020.05.25 is not actionable, removing from list.
2020-06-19 17:06:56,908 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.26
2020-06-19 17:06:56,908 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.26 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.06.13
2020-06-19 17:06:56,909 DEBUG curator.indexlist __not_actionable:39 Index logstash-2020.06.13 is not actionable, removing from list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.06.15
2020-06-19 17:06:56,909 DEBUG curator.indexlist __not_actionable:39 Index logstash-2020.06.15 is not actionable, removing from list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.18
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.18 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.25
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.25 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.20
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.20 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.01
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.01 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.06.12
...

```

```auto
Filtering indices by age
2020-06-19 17:06:56,923 DEBUG curator.indexlist _get_name_based_ages:280 Getting ages of indices by "name"
2020-06-19 17:06:56,923 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2020-06-19 17:06:56,923 DEBUG curator.utils get_date_regex:200 regex = \d{4}\.\d{2}\.\d{2}
2020-06-19 17:06:56,924 DEBUG curator.indexlist working_list:237 Generating working list of indices
2020-06-19 17:06:56,929 DEBUG curator.indexlist working_list:237 Generating working list of indices
2020-06-19 17:06:56,929 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.26 is actionable and remains in the list.
2020-06-19 17:06:56,929 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.26" age (1587859200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,929 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.18 is actionable and remains in the list.
2020-06-19 17:06:56,929 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.18" age (1587168000), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,929 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.25 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.25" age (1587772800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.20 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.20" age (1587340800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.01 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.01" age (1585699200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.06 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.06" age (1586131200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.03.31 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.03.31" age (1585612800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.12 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.12" age (1586649600), direction: "older", point of reference, (1589994416)

```

What I don't get is the last bit, where it seems there's another filter that eventually removes all the "cloudtrail"-related indices:

```auto
Filtering indices with index.lifecycle.name
2020-06-19 17:06:56,950 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.03.28 is not actionable, removing from list.
2020-06-19 17:06:56,950 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.03.28 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.24 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.24 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.05.13 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.05.13 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.05.12 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.05.12 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.05.04 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.05.04 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.06 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.06 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.22 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.22 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.28 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.28 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.26 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.26 has index.lifecycle.name cloudtrail

```

Could you let me know what that last filter is about and how I can make the indices pass it?

BTW, I took the "cloudtrail" index as an example but it happens with all the other ones except for "logstash".

Let me know if you need any more info.

Thanks a lot,

Alex

---

<div class="post-metadata">

**Author:** ![AJCandfield](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@AJCandfield](https://discuss.elastic.co/u/AJCandfield)\
**Post date:** [June 19, 2020, 6:03pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599/5 "2020-06-19T18:03:34Z")

</div>

After a Google search, I eventually found the solution which I will post here.

As described in the [documentation](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/ilm-and-curator.html):

> Curator will not act on any index associated with an ILM policy without setting `allow_ilm_indices` to `true` .

All I had to do was include the following line to my `action.yml` file:

```auto
...
options:
    allow_ilm_indices: true
...

```

That worked a charm.

Thanks for pointing me in the right direction and thanks for creating the curator! 😃

Alex

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [June 23, 2020, 2:18pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599/6 "2020-06-23T14:18:12Z")

</div>

Glad you found this. I have been out of office since Thursday last week and did not have a chance to point you to `allow_ilm_indices`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2020, 2:18pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-empty-list/237599/7 "2020-07-21T14:18:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.