Curator not actually deleting indexes

[erikmathis]

curator it showing that it found and deleted the index, but the index isnt actually being deleted.
Also this should of found two indexes, but it only found one. This is for es 7.2.0

 2019-09-20 04:00:07,991 INFO      Preparing Action ID: 1, "delete_indices"
    2019-09-20 04:00:08,091 INFO      Trying Action ID: 1, "delete_indices": Delete infralogs indices older than 45 days (based on index name). Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
    2019-09-20 04:00:08,218 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
    2019-09-20 04:00:08,218 INFO      Action ID: 1, "delete_indices" completed.
    2019-09-20 04:00:08,218 INFO      Preparing Action ID: 2, "delete_indices"
    2019-09-20 04:00:08,306 INFO      Trying Action ID: 2, "delete_indices": Delete nginx-ingress indices older than 2 days (based on index name). Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
    2019-09-20 04:00:08,435 INFO      Deleting 1 selected indices: ['nginx-ingress-2019.09.18']
    2019-09-20 04:00:08,435 INFO      ---deleting index nginx-ingress-2019.09.18
    2019-09-20 04:00:08,718 INFO      Action ID: 2, "delete_indices" completed.
    2019-09-20 04:00:08,718 INFO      Job completed.

$ curl -s $ELASTIC_URL/_cat/indices?v | grep nginx
green  open   nginx-ingress-2019.09.18        CMidwtJHQp6WupSFjLQbqg   1   1  142400322            0    106.5gb         53.2gb
green  open   nginx-ingress-2019.09.17        appCoTJeSj6xwT3vECHKjw   1   1  137136284            0    104.1gb           52gb
green  open   nginx-ingress-2019.09.19        C0WilIOBTxGliFwC73BbMQ   1   1  128977023            0     96.8gb         48.4gb
green  open   nginx-ingress-2019.09.20        tM4dy53jSYaHzFKvSSr-YA   1   1   70825351            0     62.7gb         30.2gb


---
actions:
  1:
    action: delete_indices
    description: >-
      Delete infralogs indices older than 45 days (based on index name).
      Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: infralogs-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 45
  2:
    action: delete_indices
    description: >-
      Delete nginx-ingress indices older than 2 days (based on index name).
      Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: nginx-ingress-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 2

Deleting by hand works fine.

$ curl -XDELETE $ELASTIC_URL/nginx-ingress-2019.09.17
{"acknowledged":true}

$ curl -XDELETE $ELASTIC_URL/nginx-ingress-2019.09.18
{"acknowledged":true}

$ curl -s $ELASTIC_URL/_cat/indices?v | grep nginx
green  open   nginx-ingress-2019.09.19        C0WilIOBTxGliFwC73BbMQ   1   1  128977023            0     96.8gb         48.4gb
green  open   nginx-ingress-2019.09.20        tM4dy53jSYaHzFKvSSr-YA   1   1   73618807            0     55.6gb         27.4gb

[theuntergeek (Aaron Mildenstein)]

So, no magic here. Under the hood, Curator makes the exact same API call that you made manually. No, really. In fact, if the index does not appear to have successfully deleted, Curator will automatically retry up to 3 times.

So, why might it appear that Curator fails to delete the index? This wouldn’t be the first time I’ve heard of this. It usually happens with overtaxed clusters, where it takes a long time for the cluster state to update—which would reflect that the index is fully deleted and cleared out.