Hi again @theuntergeek,
I followed your advice and I definitely got more info by setting the log level to DEBUG.
I think we're pretty close to a solution. This is what I see from the logs.
The curator selects all the indices:
...
Generating working list of indices
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.13 Size: 7.4GB Docs: 37510
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.12 Size: 15.8GB Docs: 79355
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.11 Size: 4.1GB Docs: 25720
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: cloudtrail-2020.05.10 Size: 73.4MB Docs: 6257
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: metricbeat-7.5.0-2020.04.15-000004 Size: 276.9MB Docs: 913723
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: .kibana_task_manager_1 Size: 27.9KB Docs: 2
2020-06-19 17:06:56,898 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.13 Size: 616.3MB Docs: 1868761
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.14 Size: 444.4MB Docs: 1344214
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.15 Size: 622.3MB Docs: 1885249
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.16 Size: 624.2MB Docs: 1894112
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.17 Size: 444.0MB Docs: 1265658
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.18 Size: 398.5MB Docs: 969073
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2020.06.19 Size: 543.0MB Docs: 678946
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.14 Size: 1.3MB Docs: 6173
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.13 Size: 1.9MB Docs: 8635
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.16 Size: 1.9MB Docs: 8639
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2020.06.15 Size: 1.9MB Docs: 8639
2020-06-19 17:06:56,899 DEBUG curator.indexlist iterate_over_stats:126 Index: metricbeat-7.5.0-2020.04.08-000003 Size: 271.8MB Docs: 899106
...
It applies the regex and age filters:
...
Filtering indices by regex
2020-06-19 17:06:56,908 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2020-06-19 17:06:56,908 DEBUG curator.indexlist working_list:237 Generating working list of indices
2020-06-19 17:06:56,908 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.05.25
2020-06-19 17:06:56,908 DEBUG curator.indexlist __not_actionable:39 Index logstash-2020.05.25 is not actionable, removing from list.
2020-06-19 17:06:56,908 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.26
2020-06-19 17:06:56,908 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.26 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.06.13
2020-06-19 17:06:56,909 DEBUG curator.indexlist __not_actionable:39 Index logstash-2020.06.13 is not actionable, removing from list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.06.15
2020-06-19 17:06:56,909 DEBUG curator.indexlist __not_actionable:39 Index logstash-2020.06.15 is not actionable, removing from list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.18
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.18 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.25
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.25 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.20
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.20 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: cloudtrail-2020.04.01
2020-06-19 17:06:56,909 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.01 is actionable and remains in the list.
2020-06-19 17:06:56,909 DEBUG curator.indexlist filter_by_regex:451 Filter by regex: Index: logstash-2020.06.12
...
Filtering indices by age
2020-06-19 17:06:56,923 DEBUG curator.indexlist _get_name_based_ages:280 Getting ages of indices by "name"
2020-06-19 17:06:56,923 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2020-06-19 17:06:56,923 DEBUG curator.utils get_date_regex:200 regex = \d{4}\.\d{2}\.\d{2}
2020-06-19 17:06:56,924 DEBUG curator.indexlist working_list:237 Generating working list of indices
2020-06-19 17:06:56,929 DEBUG curator.indexlist working_list:237 Generating working list of indices
2020-06-19 17:06:56,929 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.26 is actionable and remains in the list.
2020-06-19 17:06:56,929 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.26" age (1587859200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,929 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.18 is actionable and remains in the list.
2020-06-19 17:06:56,929 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.18" age (1587168000), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,929 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.25 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.25" age (1587772800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.20 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.20" age (1587340800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.01 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.01" age (1585699200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.06 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.06" age (1586131200), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.03.31 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.03.31" age (1585612800), direction: "older", point of reference, (1589994416)
2020-06-19 17:06:56,930 DEBUG curator.indexlist __actionable:35 Index cloudtrail-2020.04.12 is actionable and remains in the list.
2020-06-19 17:06:56,930 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "cloudtrail-2020.04.12" age (1586649600), direction: "older", point of reference, (1589994416)
What I don't get is the last bit, where it seems there's another filter that eventually removes all the "cloudtrail"-related indices:
Filtering indices with index.lifecycle.name
2020-06-19 17:06:56,950 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.03.28 is not actionable, removing from list.
2020-06-19 17:06:56,950 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.03.28 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.24 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.24 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.05.13 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.05.13 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.05.12 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.05.12 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.05.04 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.05.04 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.06 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.06 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.22 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.22 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.28 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.28 has index.lifecycle.name cloudtrail
2020-06-19 17:06:56,951 DEBUG curator.indexlist __not_actionable:39 Index cloudtrail-2020.04.26 is not actionable, removing from list.
2020-06-19 17:06:56,951 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cloudtrail-2020.04.26 has index.lifecycle.name cloudtrail
Could you let me know what that last filter is about and how I can make the indices pass it?
BTW, I took the "cloudtrail" index as an example but it happens with all the other ones except for "logstash".
Let me know if you need any more info.
Thanks a lot,
Alex