Curator doesn't purge Data

asalma · May 18, 2018, 2:51pm

Hi,

I install curator and in order to test it I make : 1 day to purge data, but it seems to me that it doesn't work, my indice from yesterday is still here and not deleted.


---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 1 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1
      exclude:

theuntergeek · May 18, 2018, 3:35pm

If you turn on debug logging, you will see all of the timestamp comparisons in the log. My guess is that you haven't hit a full 24 hours past UTC day rollover (this is what Curator uses). See the age filter documentation for more information. Here's a snippet, though:

All calculations are in epoch time, which is the number of seconds elapsed since 1 Jan 1970.

asalma · May 21, 2018, 7:28am

Normallly I should have today just 21 and 20 but I still have others !!

Please, help me solve this problem !!

theuntergeek · May 21, 2018, 12:24pm

This is the _cat API output, and not debug output from Curator. I can't help debug Curator without the debug logs.

asalma · May 21, 2018, 12:25pm

How can I show you the debug log ?

theuntergeek · May 21, 2018, 12:33pm

In your client configuration file, in the logging section, you set loglevel: DEBUG. Be sure to define logfile as well, so you know where it will be logged (and this path should be somewhere the user executing the script has write privileges). If no logfile is specified, Curator will log to STDOUT.

asalma · May 21, 2018, 12:37pm

---
client:
  hosts:
    - 172.16.250.29
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3'] ```



So instead of INFO I put DEBUG 

But in the log file I don't know what to put !!

theuntergeek · May 21, 2018, 12:38pm

Yes, put DEBUG there.

Don't put anything, and just capture the logs in your terminal window.

asalma · May 21, 2018, 12:41pm

????

How can I you that this will work, should I wait again one day ?

theuntergeek · May 21, 2018, 12:53pm

How are you executing Curator now? If the action file listed above is all you have, then it won't hurt anything to run it again manually, as it won't delete anything that's not specified (i.e. age hasn't changed).

asalma · May 21, 2018, 12:57pm

I did this :

 curator --config ~/.curator/curator.yml --dry-run  ~/.curator/action.yml

and I had this :

2018-05-21 14:54:50,488 DEBUG                curator.cli                    run:108  Client and logging options validated.
2018-05-21 14:54:50,488 DEBUG                curator.cli                    run:112  default_timeout = 30
2018-05-21 14:54:50,489 DEBUG                curator.cli                    run:116  action_file: /root/.curator/action.yml
2018-05-21 14:54:50,496 DEBUG                curator.cli                    run:118  action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <class 'dict'>}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}
....

theuntergeek · May 21, 2018, 12:58pm

I would still need everything output, not just that many lines. That only tells me schema validation information.

asalma · May 21, 2018, 1:01pm

2018-05-21 14:54:50,488 DEBUG                curator.cli                    run:108  Client and logging options validated.
2018-05-21 14:54:50,488 DEBUG                curator.cli                    run:112  default_timeout = 30
2018-05-21 14:54:50,489 DEBUG                curator.cli                    run:116  action_file: /root/.curator/action.yml
2018-05-21 14:54:50,496 DEBUG                curator.cli                    run:118  action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <class 'dict'>}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
2018-05-21 14:54:50,497 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}

asalma · May 21, 2018, 1:01pm

2018-05-21 14:54:50,498 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
2018-05-21 14:54:50,498 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}
2018-05-21 14:54:50,503 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, <class 'str'>, msg=None), <function Boolean at 0x7f4f995d6950>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, <class 'str'>, msg=None), <function Boolean at 0x7f4f995d6ae8>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, <class 'str'>, msg=None), <function Boolean at 0x7f4f995d6c80>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
2018-05-21 14:54:50,503 DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'ignore_empty_list': True, 'continue_if_exception': False, 'disable_action': False}
2018-05-21 14:54:50,504 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function Filters.<locals>.f at 0x7f4f995d6268>
2018-05-21 14:54:50,504 DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]
2018-05-21 14:54:50,504 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'kind': Any('prefix', 'suffix', 'timestring', 'regex', msg=None), 'value': Any(<class 'str'>, <class 'str'>, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, <class 'str'>, msg=None), <function Boolean at 0x7f4f995d6730>, msg=None), msg=None)}
2018-05-21 14:54:50,505 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-'}
2018-05-21 14:54:50,505 DEBUG     curator.validators.filters                      f:48   Filter #0: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}
2018-05-21 14:54:50,505 DEBUG     curator.defaults.filtertypes                    age:56   AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, <class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, <class 'str'>, msg=None), <function Boolean at 0x7f4f995ead90>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'timestring': Any(<class 'str'>, <class 'str'>, msg=None)}]
2018-05-21 14:54:50,506 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, <class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, <class 'str'>, msg=None), <function Boolean at 0x7f4f995ead90>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'timestring': Any(<class 'str'>, <class 'str'>, msg=None)}
2018-05-21 14:54:50,506 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1}
2018-05-21 14:54:50,506 DEBUG     curator.validators.filters                      f:48   Filter #1: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2018-05-21 14:54:50,507 DEBUG                curator.cli                    run:121  Full list of actions: {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'continue_if_exception': False, 'disable_action': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}]}}

asalma · May 21, 2018, 1:02pm

2018-05-21 14:54:50,507 DEBUG                curator.cli                    run:126  action_disabled = False
2018-05-21 14:54:50,507 DEBUG                curator.cli                    run:130  continue_if_exception = False
2018-05-21 14:54:50,507 DEBUG                curator.cli                    run:132  timeout_override = None
2018-05-21 14:54:50,507 DEBUG                curator.cli                    run:134  ignore_empty_list = True
2018-05-21 14:54:50,507 INFO                 curator.cli                    run:144  Preparing Action ID: 1, "delete_indices"
2018-05-21 14:54:50,507 DEBUG              curator.utils             get_client:803  kwargs = {'hosts': ['172.16.250.29'], 'port': 9200, 'use_ssl': False, 'ssl_no_validate': False, 'master_only': False, 'http_auth': None, 'aws_token': None, 'aws_sign_request': False, 'client_key': None, 'aws_key': None, 'url_prefix': '', 'client_cert': None, 'aws_secret_key': None, 'certificate': None, 'timeout': 30}
2018-05-21 14:54:50,512 DEBUG              curator.utils             get_client:880  Not using "requests_aws4auth" python module to connect.
2018-05-21 14:54:50,520 DEBUG              curator.utils          check_version:689  Detected Elasticsearch version 6.2.4
2018-05-21 14:54:50,520 DEBUG                curator.cli                    run:159  client is <class 'elasticsearch.client.Elasticsearch'>
2018-05-21 14:54:50,520 INFO                 curator.cli                    run:165  Trying Action ID: 1, "delete_indices": Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2018-05-21 14:54:50,520 DEBUG                curator.cli         process_action:44   Configuration dictionary: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}]}
2018-05-21 14:54:50,520 DEBUG                curator.cli         process_action:45   kwargs: {'master_timeout': 30, 'dry_run': True}
2018-05-21 14:54:50,520 DEBUG                curator.cli         process_action:50   opts: {}
2018-05-21 14:54:50,520 DEBUG                curator.cli         process_action:62   Action kwargs: {'master_timeout': 30}
2018-05-21 14:54:50,521 DEBUG                curator.cli         process_action:91   Running "DELETE_INDICES"
2018-05-21 14:54:50,521 DEBUG          curator.indexlist          __get_indices:66   Getting all indices
2018-05-21 14:54:50,529 DEBUG              curator.utils            get_indices:644  Detected Elasticsearch version 6.2.4
2018-05-21 14:54:50,529 DEBUG              curator.utils            get_indices:646  All indices: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17', '.kibana']
2018-05-21 14:54:50,529 DEBUG          curator.indexlist     __build_index_info:81   Building preliminary index metadata for logstash-2018.05.21
2018-05-21 14:54:50,529 DEBUG          curator.indexlist     __build_index_info:81   Building preliminary index metadata for logstash-2018.05.20
2018-05-21 14:54:50,530 DEBUG          curator.indexlist     __build_index_info:81   Building preliminary index metadata for logstash-2018.05.18
2018-05-21 14:54:50,530 DEBUG          curator.indexlist     __build_index_info:81   Building preliminary index metadata for logstash-2018.05.19
2018-05-21 14:54:50,530 DEBUG          curator.indexlist     __build_index_info:81   Building preliminary index metadata for logstash-2018.05.17
2018-05-21 14:54:50,530 DEBUG          curator.indexlist     __build_index_info:81   Building preliminary index metadata for .kibana
2018-05-21 14:54:50,530 DEBUG          curator.indexlist          _get_metadata:175  Getting index metadata
2018-05-21 14:54:50,530 DEBUG          curator.indexlist       empty_list_check:224  Checking for empty list
2018-05-21 14:54:50,590 DEBUG          curator.indexlist       _get_index_stats:115  Getting index stats
2018-05-21 14:54:50,590 DEBUG          curator.indexlist       empty_list_check:224  Checking for empty list
2018-05-21 14:54:50,591 DEBUG          curator.indexlist           working_list:235  Generating working list of indices
2018-05-21 14:54:50,591 DEBUG          curator.indexlist           working_list:235  Generating working list of indices
2018-05-21 14:54:50,719 DEBUG          curator.indexlist     iterate_over_stats:124  Index: logstash-2018.05.17  Size: 147.5MB  Docs: 232782
2018-05-21 14:54:50,719 DEBUG          curator.indexlist     iterate_over_stats:124  Index: logstash-2018.05.18  Size: 319.5MB  Docs: 480004
2018-05-21 14:54:50,719 DEBUG          curator.indexlist     iterate_over_stats:124  Index: logstash-2018.05.19  Size: 161.7MB  Docs: 210114
2018-05-21 14:54:50,719 DEBUG          curator.indexlist     iterate_over_stats:124  Index: .kibana  Size: 112.7KB  Docs: 97
2018-05-21 14:54:50,719 DEBUG          curator.indexlist     iterate_over_stats:124  Index: logstash-2018.05.20  Size: 280.9MB  Docs: 432872
2018-05-21 14:54:50,720 DEBUG          curator.indexlist     iterate_over_stats:124  Index: logstash-2018.05.21  Size: 153.4MB  Docs: 231440
2018-05-21 14:54:50,720 DEBUG          curator.indexlist        iterate_filters:1131 Iterating over a list of filters
2018-05-21 14:54:50,720 DEBUG          curator.indexlist        iterate_filters:1137 All filters: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}]
2018-05-21 14:54:50,720 DEBUG          curator.indexlist        iterate_filters:1139 Top of the loop: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17', '.kibana']

asalma · May 21, 2018, 1:02pm

2018-05-21 14:54:50,720 DEBUG          curator.indexlist        iterate_filters:1140 Un-parsed filter args: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}
2018-05-21 14:54:50,722 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'aliases': Any(<class 'str'>, [<class 'str'>], <class 'str'>, [<class 'str'>], msg=None), 'allocation_type': Any(<class 'str'>, <class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(<class 'str'>, <class 'str'>, None, msg=None), 'date_from_format': Any(<class 'str'>, <class 'str'>, None, msg=None), 'date_to': Any(<class 'str'>, <class 'str'>, None, msg=None), 'date_to_format': Any(<class 'str'>, <class 'str'>, None, msg=None), 'direction': Any(<class 'str'>, <class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'int'>, <class 'str'>, <class 'str'>, <class 'bool'>, None, msg=None), 'field': Any(<class 'str'>, <class 'str'>, None, msg=None), 'intersect': Any(<class 'int'>, <class 'str'>, <class 'str'>, <class 'bool'>, None, msg=None), 'key': Any(<class 'str'>, <class 'str'>, msg=None), 'kind': Any(<class 'str'>, <class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, <class 'str'>, msg=None), 'period_type': Any(<class 'str'>, <class 'str'>, msg=None), 'reverse': Any(<class 'int'>, <class 'str'>, <class 'str'>, <class 'bool'>, None, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'source': Any(<class 'str'>, <class 'str'>, msg=None), 'state': Any(<class 'str'>, <class 'str'>, msg=None), 'stats_result': Any(<class 'str'>, <class 'str'>, None, msg=None), 'timestring': Any(<class 'str'>, <class 'str'>, None, msg=None), 'threshold_behavior': Any(<class 'str'>, <class 'str'>, msg=None), 'unit': Any(<class 'str'>, <class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, <class 'str'>, msg=None), 'use_age': <function Boolean at 0x7f4f995831e0>, 'value': Any(<class 'int'>, <class 'float'>, <class 'str'>, <class 'str'>, <class 'bool'>, msg=None), 'week_starts_on': Any(<class 'str'>, <class 'str'>, None, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']")}
2018-05-21 14:54:50,723 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}
2018-05-21 14:54:50,723 DEBUG          curator.indexlist        iterate_filters:1147 Parsed filter args: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}
2018-05-21 14:54:50,723 DEBUG              curator.utils        iterate_filters:1156 Filter args: {'kind': 'prefix', 'value': 'logstash-', 'exclude': False}
2018-05-21 14:54:50,723 DEBUG              curator.utils        iterate_filters:1157 Pre-instance: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17', '.kibana']
2018-05-21 14:54:50,723 DEBUG          curator.indexlist        filter_by_regex:424  Filtering indices by regex
2018-05-21 14:54:50,723 DEBUG          curator.indexlist       empty_list_check:224  Checking for empty list
2018-05-21 14:54:50,724 DEBUG          curator.indexlist           working_list:235  Generating working list of indices
2018-05-21 14:54:50,724 DEBUG          curator.indexlist        filter_by_regex:445  Filter by regex: Index: logstash-2018.05.21
2018-05-21 14:54:50,724 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.21 is actionable and remains in the list.
2018-05-21 14:54:50,724 DEBUG          curator.indexlist        filter_by_regex:445  Filter by regex: Index: logstash-2018.05.20
2018-05-21 14:54:50,724 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.20 is actionable and remains in the list.
2018-05-21 14:54:50,724 DEBUG          curator.indexlist        filter_by_regex:445  Filter by regex: Index: logstash-2018.05.18
2018-05-21 14:54:50,724 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.18 is actionable and remains in the list.
2018-05-21 14:54:50,725 DEBUG          curator.indexlist        filter_by_regex:445  Filter by regex: Index: logstash-2018.05.19
2018-05-21 14:54:50,725 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.19 is actionable and remains in the list.
2018-05-21 14:54:50,725 DEBUG          curator.indexlist        filter_by_regex:445  Filter by regex: Index: logstash-2018.05.17
2018-05-21 14:54:50,725 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.17 is actionable and remains in the list.
2018-05-21 14:54:50,725 DEBUG          curator.indexlist        filter_by_regex:445  Filter by regex: Index: .kibana
2018-05-21 14:54:50,725 DEBUG          curator.indexlist       __not_actionable:40   Index .kibana is not actionable, removing from list.
2018-05-21 14:54:50,725 DEBUG              curator.utils        iterate_filters:1159 Post-instance: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17']
2018-05-21 14:54:50,725 DEBUG          curator.indexlist        iterate_filters:1139 Top of the loop: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17']

asalma · May 21, 2018, 1:03pm

2018-05-21 14:54:50,725 DEBUG          curator.indexlist        iterate_filters:1140 Un-parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2018-05-21 14:54:50,728 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'aliases': Any(<class 'str'>, [<class 'str'>], <class 'str'>, [<class 'str'>], msg=None), 'allocation_type': Any(<class 'str'>, <class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(<class 'str'>, <class 'str'>, None, msg=None), 'date_from_format': Any(<class 'str'>, <class 'str'>, None, msg=None), 'date_to': Any(<class 'str'>, <class 'str'>, None, msg=None), 'date_to_format': Any(<class 'str'>, <class 'str'>, None, msg=None), 'direction': Any(<class 'str'>, <class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'int'>, <class 'str'>, <class 'str'>, <class 'bool'>, None, msg=None), 'field': Any(<class 'str'>, <class 'str'>, None, msg=None), 'intersect': Any(<class 'int'>, <class 'str'>, <class 'str'>, <class 'bool'>, None, msg=None), 'key': Any(<class 'str'>, <class 'str'>, msg=None), 'kind': Any(<class 'str'>, <class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, <class 'str'>, msg=None), 'period_type': Any(<class 'str'>, <class 'str'>, msg=None), 'reverse': Any(<class 'int'>, <class 'str'>, <class 'str'>, <class 'bool'>, None, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'source': Any(<class 'str'>, <class 'str'>, msg=None), 'state': Any(<class 'str'>, <class 'str'>, msg=None), 'stats_result': Any(<class 'str'>, <class 'str'>, None, msg=None), 'timestring': Any(<class 'str'>, <class 'str'>, None, msg=None), 'threshold_behavior': Any(<class 'str'>, <class 'str'>, msg=None), 'unit': Any(<class 'str'>, <class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, <class 'str'>, msg=None), 'use_age': <function Boolean at 0x7f4f995b06a8>, 'value': Any(<class 'int'>, <class 'float'>, <class 'str'>, <class 'str'>, <class 'bool'>, msg=None), 'week_starts_on': Any(<class 'str'>, <class 'str'>, None, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']")}
2018-05-21 14:54:50,729 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2018-05-21 14:54:50,729 DEBUG          curator.indexlist        iterate_filters:1147 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2018-05-21 14:54:50,729 DEBUG              curator.utils        iterate_filters:1156 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2018-05-21 14:54:50,729 DEBUG              curator.utils        iterate_filters:1157 Pre-instance: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17']
2018-05-21 14:54:50,729 DEBUG          curator.indexlist          filter_by_age:484  Filtering indices by age

asalma · May 21, 2018, 1:03pm

2018-05-21 14:54:50,730 DEBUG          curator.indexlist   _get_name_based_ages:278  Getting ages of indices by "name"
2018-05-21 14:54:50,730 DEBUG          curator.indexlist       empty_list_check:224  Checking for empty list
2018-05-21 14:54:50,730 DEBUG              curator.utils         get_date_regex:195  regex = \d{4}\.\d{2}\.\d{2}
2018-05-21 14:54:50,730 DEBUG          curator.indexlist           working_list:235  Generating working list of indices
2018-05-21 14:54:50,753 DEBUG          curator.indexlist           working_list:235  Generating working list of indices
2018-05-21 14:54:50,753 DEBUG          curator.indexlist       __not_actionable:40   Index logstash-2018.05.21 is not actionable, removing from list.
2018-05-21 14:54:50,753 DEBUG          curator.indexlist            __excludify:59   Removed from actionable list: Index "logstash-2018.05.21" age (1526860800), direction: "older", point of reference, (1526820890)
2018-05-21 14:54:50,754 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.20 is actionable and remains in the list.
2018-05-21 14:54:50,754 DEBUG          curator.indexlist            __excludify:59   Remains in actionable list: Index "logstash-2018.05.20" age (1526774400), direction: "older", point of reference, (1526820890)
2018-05-21 14:54:50,754 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.18 is actionable and remains in the list.
2018-05-21 14:54:50,754 DEBUG          curator.indexlist            __excludify:59   Remains in actionable list: Index "logstash-2018.05.18" age (1526601600), direction: "older", point of reference, (1526820890)
2018-05-21 14:54:50,754 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.19 is actionable and remains in the list.
2018-05-21 14:54:50,754 DEBUG          curator.indexlist            __excludify:59   Remains in actionable list: Index "logstash-2018.05.19" age (1526688000), direction: "older", point of reference, (1526820890)
2018-05-21 14:54:50,754 DEBUG          curator.indexlist           __actionable:36   Index logstash-2018.05.17 is actionable and remains in the list.
2018-05-21 14:54:50,755 DEBUG          curator.indexlist            __excludify:59   Remains in actionable list: Index "logstash-2018.05.17" age (1526515200), direction: "older", point of reference, (1526820890)
2018-05-21 14:54:50,755 DEBUG              curator.utils        iterate_filters:1159 Post-instance: ['logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17']
2018-05-21 14:54:50,755 DEBUG     curator.actions.delete_indices               __init__:470  master_timeout value: 30s
2018-05-21 14:54:50,755 INFO               curator.utils           show_dry_run:918  DRY-RUN MODE.  No changes will be made.
2018-05-21 14:54:50,755 INFO               curator.utils           show_dry_run:921  (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2018-05-21 14:54:50,755 INFO               curator.utils           show_dry_run:928  DRY-RUN: delete_indices: logstash-2018.05.17 with arguments: {}
2018-05-21 14:54:50,755 INFO               curator.utils           show_dry_run:928  DRY-RUN: delete_indices: logstash-2018.05.18 with arguments: {}
2018-05-21 14:54:50,755 INFO               curator.utils           show_dry_run:928  DRY-RUN: delete_indices: logstash-2018.05.19 with arguments: {}
2018-05-21 14:54:50,756 INFO               curator.utils           show_dry_run:928  DRY-RUN: delete_indices: logstash-2018.05.20 with arguments: {}
2018-05-21 14:54:50,756 INFO                 curator.cli                    run:194  Action ID: 1, "delete_indices" completed.
2018-05-21 14:54:50,756 INFO                 curator.cli                    run:195  Job completed.

Sorry I couldn't copy it in one post

theuntergeek · May 21, 2018, 1:06pm

So, what's the problem? This looks completely proper to me.

You start with logstash-2018.05.21, logstash-2018.05.20, logstash-2018.05.19, logstash-2018.05.18, and logstash-2018.05.17. It's marking all but logstash-2018.05.21, for deletion, which is what your action file is configured to do. So what is the problem?

Are you running with --dry-run all the time? Curator won't actually do the deletes if you run with --dry-run.

asalma · May 21, 2018, 1:08pm

But when I do the Cat, I still see the older index, also in timelion I'm seing the oldest one.

Yes, I always run it with --dry-run, how should I run it to delete the oldest logs ?

Topic		Replies	Views
Elasticsearch curator not deleting indices Elasticsearch	3	1746	January 2, 2018
Curator - Not deleting indices Elasticsearch	8	1965	August 9, 2019
Problems with curator and logstash Logstash	1	371	March 13, 2018
Unable to delete indicies using Curator Elasticsearch	3	392	May 10, 2018
Curator action, delete_indices problem Elasticsearch	5	2252	July 5, 2017

Curator doesn't purge Data

Related topics