Curator is not deleting ES snapshots

hudar01 · January 7, 2022, 5:21pm

Hello,

Curator is not deleting 7 day old snapshots, even though they're there.
I'm using python virtual environment where I've the following installed:

(es-curator) [ec2-user@im es-curator]$ pip list
Package               Version
--------------------- ---------
boto3                 1.20.26
botocore              1.23.26
certifi               2021.10.8
charset-normalizer    2.0.9
click                 6.7
elasticsearch         7.1.0
elasticsearch-curator 5.8.2
idna                  3.3
jmespath              0.10.0
pip                   21.3.1
python-dateutil       2.8.2
PyYAML                3.13
requests              2.26.0
requests-aws4auth     1.1.1
s3transfer            0.5.0
setuptools            59.6.0
six                   1.16.0
urllib3               1.25.11
voluptuous            0.12.2
wheel                 0.37.1

here's the VPC info:

{
  "name" : "Y4<OMITTED>",
  "cluster_name" : "96280768<OMITTED>:demo-5",
  "cluster_uuid" : "Rl-hjRTNRBiuh......<OMITTED>",
  "version" : {
    "number" : "5.6.17",
    "build_hash" : "8d99c17",
    "build_date" : "2021-10-08T10:03:42.496Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

$ cat delete_snapshots.yaml
---
actions:
  1:
    action: delete_snapshots
    description: >-
      Delete snapshots from the selected repository older than 7 days
      (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.
    options:
      repository: es_demo_backup_ca
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: es_demo_snapshot_ca_
      exclude:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
      exclude:

Here's the output of dry run:

2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        iterate_filters:536  Filter args: {'kind': 'prefix', 'value': 'es_demo_snapshot_ca_', 'exclude': False}
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        iterate_filters:537  Pre-instance: ["'es_demo_snapshot_ca_20211229'", "'es_demo_snapshot_ca_20211230'", "'es_demo_snapshot_ca_20211231'", "'es_demo_snapshot_ca_20220101'", "'es_demo_snapshot_ca_20220102'", "'es_demo_snapshot_ca_20220103'", "'es_demo_snapshot_ca_20220104'", "'es_demo_snapshot_ca_20220105'", "'es_demo_snapshot_ca_20220106'", "'es_demo_snapshot_ca_20220107'"]
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20211229'
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20211229' is not actionable, removing from list.
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20211230'
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20211230' is not actionable, removing from list.
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20211231'
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20211231' is not actionable, removing from list.
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220101'
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220101' is not actionable, removing from list.
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220102'
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220102' is not actionable, removing from list.
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220103'
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220103' is not actionable, removing from list.
2022-01-07 17:10:16,079 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220104'
2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220104' is not actionable, removing from list.
2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220105'2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220105' is not actionable, removing from list.2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220106'
2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220106' is not actionable, removing from list.
2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220107'2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220107' is not actionable, removing from list.
2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist        iterate_filters:539  Post-instance: []
                                                                                                                                       
022-01-07 17:10:16,080 DEBUG       curator.snapshotlist        iterate_filters:519  Top of the loop: []
2022-01-07 17:10:16,080 DEBUG       curator.snapshotlist        iterate_filters:520  Un-parsed filter args: {'filtertype': 'age', 'source': 'creation_date','direction': 'older', 'unit': 'days', 'unit_count': 7, 'epoch': None, 'exclude': False, 'timestring': None}
2022-01-07 17:10:16,081 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7f0b9cd31730>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']")}2022-01-07 17:10:16,081 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'epoch': None, 'exclude': False, 'timestring': None}
2022-01-07 17:10:16,082 DEBUG       curator.snapshotlist        iterate_filters:527  Parsed filter args: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'epoch': None, 'exclude': False, 'timestring': None}
2022-01-07 17:10:16,082 DEBUG       curator.snapshotlist        iterate_filters:536  Filter args: {'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'epoch': None, 'exclude': False, 'timestring': None}2022-01-07 17:10:16,082 DEBUG       curator.snapshotlist        iterate_filters:537  Pre-instance: []
2022-01-07 17:10:16,082 DEBUG       curator.snapshotlist          filter_by_age:273  Starting filter_by_age
2022-01-07 17:10:16,082 DEBUG       curator.snapshotlist          filter_by_age:276  Point of Reference: 16409706162022-01-07 17:10:16,082 DEBUG       curator.snapshotlist        iterate_filters:539  Post-instance: []
2022-01-07 17:10:16,082 INFO      curator.actions.delete_snapshots             do_dry_run:1161 DRY-RUN MODE.  No changes will be made.2022-01-07 17:10:16,082 INFO                 curator.cli                    run:225  Action ID: 1, "delete_snapshots" completed.2022-01-07 17:10:16,082 INFO                 curator.cli                    run:226  Job completed.

Will really appreciate it if you can help? thank you

hudar01 · January 10, 2022, 9:42pm

could someone please help? I'm really stuck here. thanks

theuntergeek · January 11, 2022, 3:18pm

What I'm seeing indicates something odd, but I can't tell what's going on.

Your prefix is 'es_demo_snapshot_ca_', and your snapshots are like 'es_demo_snapshot_ca_20211229', but that's effectively not matching because it's saying is not actionable, removing...

If you do not have any other snapshot names starting with es, perhaps just set the prefix to es and see what happens. I will have to go through the code otherwise.

hudar01 · January 11, 2022, 9:41pm

Hi Aaron,

Thanks a lot for your response! I set the prefix to es and still hitting the same issue (and yes my snapshots are prefixed with es_demo_snapshot_ca_).

Thanks!

I pasted the delete snapshot output below:

2022-01-11 21:32:06,613 DEBUG                curator.cli                    run:112  Client and logging options validated.
2022-01-11 21:32:06,613 DEBUG                curator.cli                    run:116  default_timeout = 30
2022-01-11 21:32:06,613 DEBUG                curator.cli                    run:120  action_file: delete_snapshots.yaml
2022-01-11 21:32:06,617 DEBUG                curator.cli                    run:122  action_config: {'actions': {1: {'action': 'delete_snapshots', 'description': "Delete snapshots from the selected repository older than 5 days (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.", 'options': {'repository': 'es_demo_backup_ca', 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern','kind': 'prefix', 'value': 'es', 'exclude': None}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': None}]}}}
2022-01-11 21:32:06,617 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <class 'dict'>}
2022-01-11 21:32:06,617 DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_snapshots', 'description': "Delete snapshots from the selected repository older than 5 days (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.", 'options': {'repository': 'es_demo_backup_ca', 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': None}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': None}]}}}
2022-01-11 21:32:06,618 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']")}
2022-01-11 21:32:06,618 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_snapshots', 'description': "Delete snapshots from the selected repository older than 5 days (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.", 'options': {'repository': 'es_demo_backup_ca', 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': None}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': None}]}
2022-01-11 21:32:06,618 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
2022-01-11 21:32:06,618 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_snapshots', 'description': "Delete snapshots from the selected repository older than 5 days (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.", 'options': {'repository': 'es_demo_backup_ca', 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': None}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': None}]}
........... <OMITTED A FEW THINGS>>>>

2022-01-11 21:32:06,622 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es'}
2022-01-11 21:32:06,622 DEBUG     curator.validators.filters                      f:48   Filter #0: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}
2022-01-11 21:32:06,622 DEBUG     curator.defaults.filtertypes.age                    age:57   AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd5891c32f0>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', msg=None)}, {'timestring': Any(None, <class 'str'>, msg=None)}]
2022-01-11 21:32:06,623 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd5891c32f0>, msg=None), msg=None), 'source': Any('name', 'creation_date', msg=None), 'timestring':Any(None, <class 'str'>, msg=None)}
022-01-11 21:32:06,623 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7}
2022-01-11 21:32:06,623 DEBUG     curator.validators.filters                      f:48   Filter #1: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'timestring': None}
2022-01-11 21:32:06,623 DEBUG                curator.cli                    run:125  Full list of actions: {1: {'action': 'delete_snapshots', 'description':"Delete snapshots from the selected repository older than 5 days (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.", 'options': {'repository': 'es_demo_backup_ca', 'continue_if_exception': False, 'disable_action': False, 'allow_ilm_indices': False, 'timeout_override': 300, 'retry_interval': 120, 'ignore_empty_list': False, 'retry_count': 3}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'timestring': None}]}}
2022-01-11 21:32:06,623 DEBUG                curator.cli                    run:130  action_disabled = False
2022-01-11 21:32:06,623 DEBUG                curator.cli                    run:134  continue_if_exception = False
2022-01-11 21:32:06,623 DEBUG                curator.cli                    run:136  timeout_override = 300
2022-01-11 21:32:06,623 DEBUG                curator.cli                    run:138  ignore_empty_list = False
2022-01-11 21:32:06,623 DEBUG                curator.cli                    run:140  allow_ilm_indices = False
2022-01-11 21:32:06,623 INFO                 curator.cli                    run:150  Preparing Action ID: 1, "delete_snapshots"
2022-01-11 21:32:06,623 INFO                 curator.cli                    run:164  Creating client object and testing connection
2022-01-11 21:32:06,623 DEBUG              curator.utils       process_ssl_args:826  Attempting to verify SSL certificate.
2022-01-11 21:32:06,685 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2022-01-11 21:32:06,687 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from before-call.apigateway to before-call.api-gateway
2022-01-11 21:32:06,688 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2022-01-11 21:32:06,691 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2022-01-11 21:32:06,691 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2022-01-11 21:32:06,692 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2022-01-11 21:32:06,692 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
022-01-11 21:32:06,696 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2022-01-11 21:32:06,696 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2022-01-11 21:32:06,696 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2022-01-11 21:32:06,696 DEBUG             botocore.hooks      _alias_event_name:421  Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2022-01-11 21:32:06,706 DEBUG             botocore.utils       _select_base_url:413  IMDS ENDPOINT: http://169.254.169.254/
2022-01-11 21:32:06,708 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: env
2022-01-11 21:32:06,709 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: assume-role
2022-01-11 21:32:06,709 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: assume-role-with-web-identity
2022-01-11 21:32:06,709 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: sso
2022-01-11 21:32:06,709 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: shared-credentials-file
2022-01-11 21:32:06,709 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: custom-process
2022-01-11 21:32:06,709 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: config-file
2022-01-11 21:32:06,710 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: ec2-credentials-file
2022-01-11 21:32:06,710 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: boto-config
2022-01-11 21:32:06,710 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: container-role
2022-01-11 21:32:06,710 DEBUG       botocore.credentials       load_credentials:1975 Looking for credentials via: iam-role
2022-01-11 21:32:06,718 DEBUG       botocore.credentials                   load:1034 Found credentials from IAM Role:<OMMITED>-ec2-role
2022-01-11 21:32:06,719 DEBUG              curator.utils           try_aws_auth:886  Checking for AWS settings
2022-01-11 21:32:06,721 INFO               curator.utils           try_aws_auth:895  Configuring client to connect to AWS endpoint
2022-01-11 21:32:06,721 WARNING            curator.utils      process_auth_args:796  Use of "http_auth" is deprecated. Please use "username" and "password" instead.
2022-01-11 21:32:06,721 DEBUG              curator.utils             get_client:1019 kwargs = {'hosts': ['vpc-<OMMITED>-demo-5-<OMMITED>.<OMMITED>.es.amazonaws.com'], 'port': 443, 'use_ssl': True, 'aws_region': 'ca-central-1', 'aws_sign_request': True, 'ssl_no_validate': False, 'aws_secret_key': '<OMMITED>', 'http_auth': <requests_aws4auth.aws4auth.AWS4Auth object at 0x7fd5891c0898>, 'aws_token': '<OMITTED>/+<OMITTED>/<OMITTED>/<OMITTED>+<OMITTED>+pHBM/<OMITTED>==', 'aws_key': '<OMITTED>', 'url_prefix': '', 'client_cert': None, 'certificate': None, 'client_key': None, 'timeout': 300, 'connection_class': <class 'elasticsearch.connection.http_requests.RequestsHttpConnection'>, 'verify_certs': True, 'ca_certs': '/home/ec2-user/es_snapshot/es-curator/lib/python3.6/site-packages/certifi/cacert.pem'}
2022-01-11 21:32:06,721 INFO               curator.utils             get_client:1024 Instantiating client object
2022-01-11 21:32:06,722 INFO               curator.utils             get_client:1027 Testing client connectivity
2022-01-11 21:32:06,758 DEBUG              curator.utils             get_client:1028 Cluster info: {'name': 'Y44J<OMITTED>', 'cluster_name': '<OMITTED>:<OMITTED>-demo-5', 'cluster_uuid': 'Rl-<OMITTED>', 'version': {'number': '5.6.17', 'build_hash': '8d99c17', 'build_date': '2021-10-08T10:03:42.496Z', 'build_snapshot': False, 'lucene_version': '6.6.1'}, 'tagline': 'You Know, for Search'}2022-01-11 21:32:06,758 INFO               curator.utils             get_client:1029 Successfully created Elasticsearch client object with provided settings
2022-01-11 21:32:06,758 DEBUG              curator.utils       do_version_check:918  Checking Elasticsearch endpoint version...2022-01-11 21:32:06,764 DEBUG              curator.utils          check_version:708  Detected Elasticsearch version 5.6.17
2022-01-11 21:32:06,765 DEBUG              curator.utils   verify_master_status:942  Not verifying local master status (master_only: false)2022-01-11 21:32:06,765 INFO                 curator.cli                    run:196  Trying Action ID: 1, "delete_snapshots": Delete snapshots from the selected repository older than 5 days (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.2022-01-11 21:32:06,765 DEBUG                curator.cli         process_action:48   Configuration dictionary: {'action': 'delete_snapshots', 'description':"Delete snapshots from the selected repository older than 5 days (based on creation_date), for 'es_demo_snapshot_ca_' prefixed snapshots.", 'options': {'repository': 'es_demo_backup_ca', 'retry_interval': 120, 'retry_count': 3}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'timestring': None}]}
2022-01-11 21:32:06,765 DEBUG                curator.cli         process_action:49   kwargs: {'master_timeout': 300, 'dry_run': False}
2022-01-11 21:32:06,765 DEBUG                curator.cli         process_action:54   opts: {'repository': 'es_demo_backup_ca', 'retry_interval': 120, 'retry_count': 3}
2022-01-11 21:32:06,765 DEBUG                curator.cli         process_action:66   Action kwargs: {'repository': 'es_demo_backup_ca', 'retry_interval': 120, 'retry_count': 3}
2022-01-11 21:32:06,765 DEBUG                curator.cli         process_action:88   Running "delete_snapshots"
2022-01-11 21:32:06,772 DEBUG              curator.utils      repository_exists:1383 Repository es_demo_backup_ca exists.
2022-01-11 21:32:07,600 DEBUG       curator.snapshotlist        iterate_filters:517  All filters: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None,'timestring': None}]
2022-01-11 21:32:07,601 DEBUG       curator.snapshotlist        iterate_filters:519  Top of the loop: ["'es_demo_snapshot_ca_20211229'", "'es_demo_snapshot_ca_20211230'", "'es_demo_snapshot_ca_20211231'", "'es_demo_snapshot_ca_20220101'", "'es_demo_snapshot_ca_20220102'", "'es_demo_snapshot_ca_20220103'", "'es_demo_snapshot_ca_20220104'", "'es_demo_snapshot_ca_20220105'", "'es_demo_snapshot_ca_20220106'", "'es_demo_snapshot_ca_20220107'", "'es_demo_snapshot_ca_20220108'", "'es_demo_snapshot_ca_20220109'", "'es_demo_snapshot_ca_20220110'", "'es_demo_snapshot_ca_20220111'"]2022-01-11 21:32:07,601 DEBUG       curator.snapshotlist        iterate_filters:520  Un-parsed filter args: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}
2022-01-11 21:32:07,602 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class '
2022-01-11 21:32:07,601 DEBUG       curator.snapshotlist        iterate_filters:519  Top of the loop: ["'es_demo_snapshot_ca_20211229'", "'es_demo_snapshot_ca_20211230'", "'es_demo_snapshot_ca_20211231'", "'es_demo_snapshot_ca_20220101'", "'es_demo_snapshot_ca_20220102'", "'es_demo_snapshot_ca_20220103'", "'es_demo_snapshot_ca_20220104'", "'es_demo_snapshot_ca_20220105'", "'es_demo_snapshot_ca_20220106'", "'es_demo_snapshot_ca_20220107'", "'es_demo_snapshot_ca_20220108'", "'es_demo_snapshot_ca_20220109'", "'es_demo_snapshot_ca_20220110'", "'es_demo_snapshot_ca_20220111'"]
2022-01-11 21:32:07,601 DEBUG       curator.snapshotlist        iterate_filters:520  Un-parsed filter args: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}2022-01-11 21:32:07,602 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7fd5886f9268>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']")}
2022-01-11 21:32:07,602 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        iterate_filters:527  Parsed filter args: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'es', 'exclude': False}
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        iterate_filters:536  Filter args: {'kind': 'prefix', 'value': 'es', 'exclude': False}
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        iterate_filters:537  Pre-instance: ["'es_demo_snapshot_ca_20211229'", "'es_demo_snapshot_ca_20211230'", "'es_demo_snapshot_ca_20211231'", "'es_demo_snapshot_ca_20220101'", "'es_demo_snapshot_ca_20220102'", "'es_demo_snapshot_ca_20220103'", "'es_demo_snapshot_ca_20220104'", "'es_demo_snapshot_ca_20220105'", "'es_demo_snapshot_ca_20220106'", "'es_demo_snapshot_ca_20220107'", "'es_demo_snapshot_ca_20220108'", "'es_demo_snapshot_ca_20220109'", "'es_demo_snapshot_ca_20220110'", "'es_demo_snapshot_ca_20220111'"]
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20211229'
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20211229' is not actionable, removing from list.
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20211230'
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20211230' is not actionable, removing from list.
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20211231'
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20211231' is not actionable, removing from list.
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220101'
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220101' is not actionable, removing from list.
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220102'
2022-01-11 21:32:07,603 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220102' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220103'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220103' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220104'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220104' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220105'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220105' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220106'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220106' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220107'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220107' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220108'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220108' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220109'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220109' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220110'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220110' is not actionable, removing from list.
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist        filter_by_regex:244  Filter by regex: Snapshot: 'es_demo_snapshot_ca_20220111'
2022-01-11 21:32:07,604 DEBUG       curator.snapshotlist       __not_actionable:51   Snapshot 'es_demo_snapshot_ca_20220111' is not actionable, removing from list.
2022-01-11 21:32:07,605 DEBUG       curator.snapshotlist        iterate_filters:539  Post-instance: []
2022-01-11 21:32:07,605 DEBUG       curator.snapshotlist        iterate_filters:519  Top of the loop: []
2022-01-11 21:32:07,605 DEBUG       curator.snapshotlist        iterate_filters:520  Un-parsed filter args: {'filtertype': 'age', 'source': 'creation_date','direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'timestring': None}
2022-01-11 21:32:07,606 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7fd58809e730>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']")}2022-01-11 21:32:07,606 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'timestring': None}
2022-01-11 21:32:07,606 DEBUG       curator.snapshotlist        iterate_filters:527  Parsed filter args: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'timestring': None}
2022-01-11 21:32:07,606 DEBUG       curator.snapshotlist        iterate_filters:536  Filter args: {'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'exclude': False, 'epoch': None, 'timestring': None}2022-01-11 21:32:07,606 DEBUG       curator.snapshotlist        iterate_filters:537  Pre-instance: []
2022-01-11 21:32:07,606 DEBUG       curator.snapshotlist          filter_by_age:273  Starting filter_by_age
2022-01-11 21:32:07,606 DEBUG       curator.snapshotlist          filter_by_age:276  Point of Reference: 1641331927
2022-01-11 21:32:07,607 DEBUG       curator.snapshotlist        iterate_filters:539  Post-instance: []
2022-01-11 21:32:07,607 DEBUG                curator.cli         process_action:103  Doing the action here.
2022-01-11 21:32:07,607 ERROR                curator.cli                    run:209  Unable to complete action "delete_snapshots".  No actionable items in list: <class 'curator.exceptions.NoSnapshots'>

hudar01 · January 17, 2022, 2:44pm

Wondering if someone got the chance to look into this ? thank you

hudar01 · January 25, 2022, 4:27pm

did someone get the chance to look into this? thanks

system · February 22, 2022, 4:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Curator is deleting all snapshots Elasticsearch	7	1035	July 5, 2017
Snapshot Delete not working as expected Elasticsearch	6	732	October 8, 2018
Curator while deleting the old snapshots, times out in middle of operation Elasticsearch	4	1801	April 17, 2018
Setup to create daily snapshots and purge old snapshots Elasticsearch	2	1002	April 11, 2017
Curator Elasticsearch 404 on delete_snapshots action in AWS Elasticsearch Elasticsearch	7	1123	June 5, 2018

Curator is not deleting ES snapshots

Related topics