Environment: Curator 5.5.1, Elastic Search 6.2.1
Trying to use delete_snapshots actions using curator, but it fails with below error always
2018-06-05 23:26:03,003 DEBUG curator.utils iterate_filters:528 Post-instance: [list of snapshots.........]
2018-06-05 23:26:03,003 DEBUG curator.cli process_action:99 Doing the action here.
2018-06-05 23:26:03,003 INFO curator.actions.delete_snapshots do_action:991 Deleting selected snapshots
2018-06-05 23:26:03,587 INFO curator.actions.delete_snapshots do_action:1000 Deleting snapshot <<Snapshot 1 from the list>>
2018-06-05 23:41:03,590 ERROR curator.cli run:184 Failed to complete action: delete_snapshots. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='-------', port=9200): Read timed out. (read timeout=900))
timeout_override is 900 against the default 300.
The action has couple of filters to filter snapshots based on regex and age. Looking at curator debug log, it finally lands up with the list of snapshots after applying filter. Then it loops thru' the list and issues delete snapshot for the first item. Post that it hangs and doesn't move forward with rest. Thought it is a timeout issue and increased to 900, but no result. Once the timeout overruns, it errors, but do see the first snapshot in the list is deleted successfully.
When the curator was performing the action, monitored ES using _tasks API. It did list the delete snapshot task for the first item, which was well done within ~5 mts (300 sec). The curator should have started with next, but it never did. The _task API didn't list any delete snapshot action until the timer (900) overruns.
Any thoughts/guidance is appreciated. Looked at curator code cli.py, but the log comes out from catch-all (logger.error).