Curator delete_indices fail

[theuntergeek (Aaron Mildenstein)]

@pollux

2018-06-11 17:02:46,617 DEBUG elasticsearch log_request_fail:110 < {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"metricbeat-6.0.1-2018.05.10","index_uuid":"_na_","index":"metricbeat-6.0.1-2018.05.10"}], "type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"metricbeat-6.0.1-2018.05.10","index_uuid": "_na_","index":"metricbeat-6.0.1-2018.05.10"},"status":404}

This indicates that there's something amiss in your cluster. Something says that metricbeat-6.0.1-2018.05.10 exists to Curator (an API call), but then when it issues another API call to delete it, Elasticsearch is responding with {"type":"index_not_found_exception","reason":"no such index". It's not there. Are all three hosts in your config.yml part of the same cluster?

  hosts:
    - es1.corp.net
    - es2.corp.net
    - es3.corp.net

If these are not all members of the same cluster, and that index is not on all members, that would result in the "not found" response. Curator round-robins the requests. The first request hits the first host, and then the delete hits the second. This is the most likely explanation of what is happening.