Curator no longer working. I think it's related to a failed ILM setup

A little while back I tried to setup ILM. It didn't go so well, I asked for help here but was unable to get it working:

So it's kind of half setup but still giving errors about rolling indexes over.

I've kind of left it as it is for now but I noticed today that curator does not seem to be working any more. Now when I run curator it reports an empty list of indexes, no matter what I set unit_count to. Here's my simple curator config:

actions:
1:
action: delete_indices
description: >-
Delete indices older than 15 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 5
exclude:

And the output that I'm getting from curator:

2019-08-08 18:18:04,202 INFO Preparing Action ID: 1, "delete_indices"
2019-08-08 18:18:04,212 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 15 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2019-08-08 18:18:04,301 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-08-08 18:18:04,301 INFO Action ID: 1, "delete_indices" completed.
2019-08-08 18:18:04,301 INFO Job completed.

Here's my list of indexes currently:

root@ps-dev-elk:~# curl -X GET "localhost:9200/cat/indices?v&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-000002 KmIUUStBTkSe3NXcUy-vhQ 5 1 0 0 1.2kb 1.2kb
yellow open logstash-2019.07.03 lhehP_liSbGO5mNd8iQ6ug 5 1 34648 0 24.1mb 24.1mb
yellow open logstash-2019.06.29 9AczdsFTQk65CGyXdU6a2g 5 1 11873 0 11.9mb 11.9mb
yellow open logstash-2019.07.12 mSnVXCjLRIySS_9cG6qpXA 5 1 1218159 0 328.4mb 328.4mb
yellow open logstash-2019.07.31 LNfFP57BTd28bzW57C1f4w 5 1 11312682 0 4.6gb 4.6gb
yellow open logstash-2019.07.27 ZvZcNrT6Tq2DDjGw1I5n1w 5 1 9846601 0 3.3gb 3.3gb
yellow open logstash-2019.06.28 U430uIccRpmQEeldgTEopg 5 1 13671 0 14.1mb 14.1mb
yellow open logstash-2019.08.04 8K70cBewQ9m2kyunFR0Fnw 5 1 9712827 0 3.2gb 3.2gb
yellow open logstash-2019.08.02 fc4YutRjS_S2s8TGlXwtBQ 5 1 10279012 0 3.7gb 3.7gb
green open .kibana_task_manager xnKBV_MmSRabaCajI_9esA 1 0 2 0 12.9kb 12.9kb
yellow open logstash-2019.06.26 2zgDKUUBSPeS5YSpWqL4SQ 5 1 6013 0 6mb 6mb
yellow open logstash-2019.07.18 KwVV5VQgQgaaDup-UA5Kxg 5 1 10294809 0 3.9gb 3.9gb
yellow open logstash-2019.08.06 SL7Zka4cRtiqNDy020eqSA 5 1 1698488 0 596.2mb 596.2mb
yellow open logstash-2019.07.25 6AYLP8CzQF6-CP7ywwunog 5 1 10452281 0 3.6gb 3.6gb
yellow open logstash-2019.07.21 xm31mDXvSpu_Su7xsEp6PQ 5 1 10012414 0 3.4gb 3.4gb
yellow open logstash-2019.07.29 UaAQ1zcYRBm1PPtgyfKjYg 5 1 10975367 0 4.3gb 4.3gb
yellow open logstash-2019.07.28 1ugwhMfpSq6U22BwJ3MmfQ 5 1 9827911 0 3.3gb 3.3gb
yellow open logstash-2019.06.25 7hiw79RgROCSkk5sXdtS7Q 5 1 2870 0 4.7mb 4.7mb
yellow open logstash-2019.07.07 WicWZwu7RC2ZVdtcOEesBg 5 1 9720334 0 3.5gb 3.5gb
green open .kibana_8 8M6UJL8YRf2p4Uv3qnQftg 1 0 87 1 85kb 85kb
yellow open logstash-2019.07.30 -TpSM_BrQXKmVTghFSN5JQ 5 1 11823449 0 5gb 5gb
yellow open logstash-2019.07.26 3wcSdQLFSK2359m2ZOpCeA 5 1 10211976 0 3.6gb 3.6gb
yellow open .kibana-6 tzxLdJi8RTioz3Av0qIYVA 1 1 75 1 78.3kb 78.3kb
yellow open logstash-2019.07.24 H8Vp8ICiSDqhluGZpoILyw 5 1 10335395 0 3.7gb 3.7gb
yellow open logstash-2019.07.15 Aa4ptB3fQhmiAwupCJUnng 5 1 2412431 0 653.5mb 653.5mb
yellow open logstash-2019.07.06 knFlSPPPSj6z
-0ILMF4EQ 5 1 9525251 0 3.3gb 3.3gb
green open .kibana_7 iF0d8s65Q6-CPU4KrrG4Sw 1 0 86 1 87.2kb 87.2kb
yellow open logstash-2019.08.05 OwgaYR7oQiulhssyzaU-qg 5 1 9996109 0 3.5gb 3.5gb
yellow open logstash-2019.07.10 YgwZRYdyTtif_HOUm0zDLQ 5 1 1178881 0 307.1mb 307.1mb

I'm pretty sure that this is related to ILM because we have 2 ELK servers that are nearly identical, 1 for dev and 1 for prod. I've only tried to implement ILM on the dev server. curator is still functioning correctly on the prod server. I don't think anything else has changed, no changes to index names or anything like that.

Because it can collide with ILM, Curator will not touch indices associated with an ILM policy by default. This can be overridden by setting the option allow_ilm_indices: true

That did it, thanks @theuntergeek!

If anyone sees this and knows what's wrong with my ILM setup I'd love some tips on that as well.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.