Curator no longer working. I think it's related to a failed ILM setup

[dfinn (Daniel J Finn)]

A little while back I tried to setup ILM. It didn't go so well, I asked for help here but was unable to get it working:

Getting errors related to ILM

currently running ES 6.7.1 and we use the logstash-* indexes I recently setup ILM through the UI, using these settings: { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_age": "15d", "max_size": "50gb" }, "set_priority": { "priority": 100 } } }, "delete": { "min_age": "1d", "actions": { "delete": {} } } } } } And I noticed in our elasticsearch logs we got the following error: [2019-05-22T00:16:46,994][ERROR][o.e.x.i.IndexLifecycleR…

So it's kind of half setup but still giving errors about rolling indexes over.

I've kind of left it as it is for now but I noticed today that curator does not seem to be working any more. Now when I run curator it reports an empty list of indexes, no matter what I set unit_count to. Here's my simple curator config:

actions:
1:
action: delete_indices
description: >-
Delete indices older than 15 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 5
exclude:

And the output that I'm getting from curator:

2019-08-08 18:18:04,202 INFO Preparing Action ID: 1, "delete_indices"
2019-08-08 18:18:04,212 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 15 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2019-08-08 18:18:04,301 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-08-08 18:18:04,301 INFO Action ID: 1, "delete_indices" completed.
2019-08-08 18:18:04,301 INFO Job completed.

Here's my list of indexes currently:

root@ps-dev-elk:~# curl -X GET "localhost:9200/cat/indices?v&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-000002 KmIUUStBTkSe3NXcUy-vhQ 5 1 0 0 1.2kb 1.2kb
yellow open logstash-2019.07.03 lhehP_liSbGO5mNd8iQ6ug 5 1 34648 0 24.1mb 24.1mb
yellow open logstash-2019.06.29 9AczdsFTQk65CGyXdU6a2g 5 1 11873 0 11.9mb 11.9mb
yellow open logstash-2019.07.12 mSnVXCjLRIySS_9cG6qpXA 5 1 1218159 0 328.4mb 328.4mb
yellow open logstash-2019.07.31 LNfFP57BTd28bzW57C1f4w 5 1 11312682 0 4.6gb 4.6gb
yellow open logstash-2019.07.27 ZvZcNrT6Tq2DDjGw1I5n1w 5 1 9846601 0 3.3gb 3.3gb
yellow open logstash-2019.06.28 U430uIccRpmQEeldgTEopg 5 1 13671 0 14.1mb 14.1mb
yellow open logstash-2019.08.04 8K70cBewQ9m2kyunFR0Fnw 5 1 9712827 0 3.2gb 3.2gb
yellow open logstash-2019.08.02 fc4YutRjS_S2s8TGlXwtBQ 5 1 10279012 0 3.7gb 3.7gb
green open .kibana_task_manager xnKBV_MmSRabaCajI_9esA 1 0 2 0 12.9kb 12.9kb
yellow open logstash-2019.06.26 2zgDKUUBSPeS5YSpWqL4SQ 5 1 6013 0 6mb 6mb
yellow open logstash-2019.07.18 KwVV5VQgQgaaDup-UA5Kxg 5 1 10294809 0 3.9gb 3.9gb
yellow open logstash-2019.08.06 SL7Zka4cRtiqNDy020eqSA 5 1 1698488 0 596.2mb 596.2mb
yellow open logstash-2019.07.25 6AYLP8CzQF6-CP7ywwunog 5 1 10452281 0 3.6gb 3.6gb
yellow open logstash-2019.07.21 xm31mDXvSpu_Su7xsEp6PQ 5 1 10012414 0 3.4gb 3.4gb
yellow open logstash-2019.07.29 UaAQ1zcYRBm1PPtgyfKjYg 5 1 10975367 0 4.3gb 4.3gb
yellow open logstash-2019.07.28 1ugwhMfpSq6U22BwJ3MmfQ 5 1 9827911 0 3.3gb 3.3gb
yellow open logstash-2019.06.25 7hiw79RgROCSkk5sXdtS7Q 5 1 2870 0 4.7mb 4.7mb
yellow open logstash-2019.07.07 WicWZwu7RC2ZVdtcOEesBg 5 1 9720334 0 3.5gb 3.5gb
green open .kibana_8 8M6UJL8YRf2p4Uv3qnQftg 1 0 87 1 85kb 85kb
yellow open logstash-2019.07.30 -TpSM_BrQXKmVTghFSN5JQ 5 1 11823449 0 5gb 5gb
yellow open logstash-2019.07.26 3wcSdQLFSK2359m2ZOpCeA 5 1 10211976 0 3.6gb 3.6gb
yellow open .kibana-6 tzxLdJi8RTioz3Av0qIYVA 1 1 75 1 78.3kb 78.3kb
yellow open logstash-2019.07.24 H8Vp8ICiSDqhluGZpoILyw 5 1 10335395 0 3.7gb 3.7gb
yellow open logstash-2019.07.15 Aa4ptB3fQhmiAwupCJUnng 5 1 2412431 0 653.5mb 653.5mb
yellow open logstash-2019.07.06 knFlSPPPSj6z-0ILMF4EQ 5 1 9525251 0 3.3gb 3.3gb
green open .kibana_7 iF0d8s65Q6-CPU4KrrG4Sw 1 0 86 1 87.2kb 87.2kb
yellow open logstash-2019.08.05 OwgaYR7oQiulhssyzaU-qg 5 1 9996109 0 3.5gb 3.5gb
yellow open logstash-2019.07.10 YgwZRYdyTtif_HOUm0zDLQ 5 1 1178881 0 307.1mb 307.1mb

I'm pretty sure that this is related to ILM because we have 2 ELK servers that are nearly identical, 1 for dev and 1 for prod. I've only tried to implement ILM on the dev server. curator is still functioning correctly on the prod server. I don't think anything else has changed, no changes to index names or anything like that.

[theuntergeek (Aaron Mildenstein)]

Because it can collide with ILM, Curator will not touch indices associated with an ILM policy by default. This can be overridden by setting the option allow_ilm_indices: true

[dfinn (Daniel J Finn)]

@theuntergeek

allow_ilm_indices: true

That did it, thanks @theuntergeek!

If anyone sees this and knows what's wrong with my ILM setup I'd love some tips on that as well.