# Curator doesn't purge Data **URL:** \ **Category:** Elasticsearch\ **Created:** [May 18, 2018, 2:51pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505 "2018-05-18T14:51:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 18, 2018, 2:51pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/1 "2018-05-18T14:51:12Z")
Hi, I install curator and in order to test it I make : 1 day to purge data, but it seems to me that it doesn't work, my indice from yesterday is still here and not deleted. ```auto --- actions: 1: action: delete_indices description: >- Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 1 exclude: ``` ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 18, 2018, 3:35pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/2 "2018-05-18T15:35:46Z")
If you turn on debug logging, you will see all of the timestamp comparisons in the log. My guess is that you haven't hit a full 24 hours past UTC day rollover (this is what Curator uses). See the [`age` filter documentation](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/filtertype_age.html#_age_calculation) for more information. Here's a snippet, though: > All calculations are in epoch time, which is the number of seconds elapsed since 1 Jan 1970. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 7:28am UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/3 "2018-05-21T07:28:35Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/8/8/886afbd5a436a8e92c5f6a4cb244dcb1247c05f8.png) Normallly I should have today just 21 and 20 but I still have others !! Please, help me solve this problem !! ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 21, 2018, 12:24pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/4 "2018-05-21T12:24:45Z")
This is the `_cat` API output, and not debug output from Curator. I can't help debug Curator without the debug logs. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 12:25pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/5 "2018-05-21T12:25:47Z")
How can I show you the debug log ? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 21, 2018, 12:33pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/6 "2018-05-21T12:33:02Z")
In your [client configuration file](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/configfile.html), in the `logging` section, you set [`loglevel: DEBUG`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/configfile.html#loglevel). Be sure to define [`logfile`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/configfile.html#logfile) as well, so you know where it will be logged (and this path should be somewhere the user executing the script has write privileges). If no `logfile` is specified, Curator will log to STDOUT. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 12:37pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/7 "2018-05-21T12:37:52Z")
````auto --- client: hosts: - 172.16.250.29 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] ``` So instead of INFO I put DEBUG But in the log file I don't know what to put !! ```` ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 21, 2018, 12:38pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/8 "2018-05-21T12:38:46Z")
Yes, put `DEBUG` there. Don't put anything, and just capture the logs in your terminal window. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 12:41pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/9 "2018-05-21T12:41:19Z")
> [@theuntergeek](#): > > Don't put anything, and just capture the logs in your terminal window. ???? How can I you that this will work, should I wait again one day ? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 21, 2018, 12:53pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/10 "2018-05-21T12:53:30Z")
How are you executing Curator now? If the action file listed above is all you have, then it won't hurt anything to run it again manually, as it won't delete anything that's not specified (i.e. age hasn't changed). ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 12:57pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/11 "2018-05-21T12:57:14Z")
I did this : ```auto curator --config ~/.curator/curator.yml --dry-run ~/.curator/action.yml ``` and I had this : ```auto 2018-05-21 14:54:50,488 DEBUG curator.cli run:108 Client and logging options validated. 2018-05-21 14:54:50,488 DEBUG curator.cli run:112 default_timeout = 30 2018-05-21 14:54:50,489 DEBUG curator.cli run:116 action_file: /root/.curator/action.yml 2018-05-21 14:54:50,496 DEBUG curator.cli run:118 action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}} 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'actions': } 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :27 "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}} 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")} 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :27 "action type" config: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]} .... ``` ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 21, 2018, 12:58pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/12 "2018-05-21T12:58:49Z")
I would still need everything output, not just that many lines. That only tells me schema validation information. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 1:01pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/13 "2018-05-21T13:01:07Z")
```auto 2018-05-21 14:54:50,488 DEBUG curator.cli run:108 Client and logging options validated. 2018-05-21 14:54:50,488 DEBUG curator.cli run:112 default_timeout = 30 2018-05-21 14:54:50,489 DEBUG curator.cli run:116 action_file: /root/.curator/action.yml 2018-05-21 14:54:50,496 DEBUG curator.cli run:118 action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}} 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'actions': } 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :27 "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]}}} 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")} 2018-05-21 14:54:50,497 DEBUG curator.validators.SchemaCheck __init__ :27 "action type" config: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]} ``` ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 1:01pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/14 "2018-05-21T13:01:26Z")
```auto 2018-05-21 14:54:50,498 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(, , msg=None), 'options': , 'filters': } 2018-05-21 14:54:50,498 DEBUG curator.validators.SchemaCheck __init__ :27 "structure" config: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}]} 2018-05-21 14:54:50,503 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'continue_if_exception': Any(, All(Any(, , msg=None), , msg=None), msg=None), 'disable_action': Any(, All(Any(, , msg=None), , msg=None), msg=None), 'ignore_empty_list': Any(, All(Any(, , msg=None), , msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)} 2018-05-21 14:54:50,503 DEBUG curator.validators.SchemaCheck __init__ :27 "options" config: {'ignore_empty_list': True, 'continue_if_exception': False, 'disable_action': False} 2018-05-21 14:54:50,504 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: .f at 0x7f4f995d6268> 2018-05-21 14:54:50,504 DEBUG curator.validators.SchemaCheck __init__ :27 "filters" config: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'exclude': None}] 2018-05-21 14:54:50,504 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'kind': Any('prefix', 'suffix', 'timestring', 'regex', msg=None), 'value': Any(, , msg=None), 'exclude': Any(, All(Any(, , msg=None), , msg=None), msg=None)} 2018-05-21 14:54:50,505 DEBUG curator.validators.SchemaCheck __init__ :27 "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-'} 2018-05-21 14:54:50,505 DEBUG curator.validators.filters f:48 Filter #0: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False} 2018-05-21 14:54:50,505 DEBUG curator.defaults.filtertypes age:56 AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(, , msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(, All(Any(, , msg=None), , msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'timestring': Any(, , msg=None)}] 2018-05-21 14:54:50,506 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(, , msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(, All(Any(, , msg=None), , msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'timestring': Any(, , msg=None)} 2018-05-21 14:54:50,506 DEBUG curator.validators.SchemaCheck __init__ :27 "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1} 2018-05-21 14:54:50,506 DEBUG curator.validators.filters f:48 Filter #1: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2018-05-21 14:54:50,507 DEBUG curator.cli run:121 Full list of actions: {1: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'continue_if_exception': False, 'disable_action': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}]}} ``` ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 1:02pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/15 "2018-05-21T13:02:19Z")
```auto 2018-05-21 14:54:50,507 DEBUG curator.cli run:126 action_disabled = False 2018-05-21 14:54:50,507 DEBUG curator.cli run:130 continue_if_exception = False 2018-05-21 14:54:50,507 DEBUG curator.cli run:132 timeout_override = None 2018-05-21 14:54:50,507 DEBUG curator.cli run:134 ignore_empty_list = True 2018-05-21 14:54:50,507 INFO curator.cli run:144 Preparing Action ID: 1, "delete_indices" 2018-05-21 14:54:50,507 DEBUG curator.utils get_client:803 kwargs = {'hosts': ['172.16.250.29'], 'port': 9200, 'use_ssl': False, 'ssl_no_validate': False, 'master_only': False, 'http_auth': None, 'aws_token': None, 'aws_sign_request': False, 'client_key': None, 'aws_key': None, 'url_prefix': '', 'client_cert': None, 'aws_secret_key': None, 'certificate': None, 'timeout': 30} 2018-05-21 14:54:50,512 DEBUG curator.utils get_client:880 Not using "requests_aws4auth" python module to connect. 2018-05-21 14:54:50,520 DEBUG curator.utils check_version:689 Detected Elasticsearch version 6.2.4 2018-05-21 14:54:50,520 DEBUG curator.cli run:159 client is 2018-05-21 14:54:50,520 INFO curator.cli run:165 Trying Action ID: 1, "delete_indices": Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. 2018-05-21 14:54:50,520 DEBUG curator.cli process_action:44 Configuration dictionary: {'action': 'delete_indices', 'description': 'Delete indices older than 1 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}]} 2018-05-21 14:54:50,520 DEBUG curator.cli process_action:45 kwargs: {'master_timeout': 30, 'dry_run': True} 2018-05-21 14:54:50,520 DEBUG curator.cli process_action:50 opts: {} 2018-05-21 14:54:50,520 DEBUG curator.cli process_action:62 Action kwargs: {'master_timeout': 30} 2018-05-21 14:54:50,521 DEBUG curator.cli process_action:91 Running "DELETE_INDICES" 2018-05-21 14:54:50,521 DEBUG curator.indexlist __get_indices:66 Getting all indices 2018-05-21 14:54:50,529 DEBUG curator.utils get_indices:644 Detected Elasticsearch version 6.2.4 2018-05-21 14:54:50,529 DEBUG curator.utils get_indices:646 All indices: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17', '.kibana'] 2018-05-21 14:54:50,529 DEBUG curator.indexlist __build_index_info:81 Building preliminary index metadata for logstash-2018.05.21 2018-05-21 14:54:50,529 DEBUG curator.indexlist __build_index_info:81 Building preliminary index metadata for logstash-2018.05.20 2018-05-21 14:54:50,530 DEBUG curator.indexlist __build_index_info:81 Building preliminary index metadata for logstash-2018.05.18 2018-05-21 14:54:50,530 DEBUG curator.indexlist __build_index_info:81 Building preliminary index metadata for logstash-2018.05.19 2018-05-21 14:54:50,530 DEBUG curator.indexlist __build_index_info:81 Building preliminary index metadata for logstash-2018.05.17 2018-05-21 14:54:50,530 DEBUG curator.indexlist __build_index_info:81 Building preliminary index metadata for .kibana 2018-05-21 14:54:50,530 DEBUG curator.indexlist _get_metadata:175 Getting index metadata 2018-05-21 14:54:50,530 DEBUG curator.indexlist empty_list_check:224 Checking for empty list 2018-05-21 14:54:50,590 DEBUG curator.indexlist _get_index_stats:115 Getting index stats 2018-05-21 14:54:50,590 DEBUG curator.indexlist empty_list_check:224 Checking for empty list 2018-05-21 14:54:50,591 DEBUG curator.indexlist working_list:235 Generating working list of indices 2018-05-21 14:54:50,591 DEBUG curator.indexlist working_list:235 Generating working list of indices 2018-05-21 14:54:50,719 DEBUG curator.indexlist iterate_over_stats:124 Index: logstash-2018.05.17 Size: 147.5MB Docs: 232782 2018-05-21 14:54:50,719 DEBUG curator.indexlist iterate_over_stats:124 Index: logstash-2018.05.18 Size: 319.5MB Docs: 480004 2018-05-21 14:54:50,719 DEBUG curator.indexlist iterate_over_stats:124 Index: logstash-2018.05.19 Size: 161.7MB Docs: 210114 2018-05-21 14:54:50,719 DEBUG curator.indexlist iterate_over_stats:124 Index: .kibana Size: 112.7KB Docs: 97 2018-05-21 14:54:50,719 DEBUG curator.indexlist iterate_over_stats:124 Index: logstash-2018.05.20 Size: 280.9MB Docs: 432872 2018-05-21 14:54:50,720 DEBUG curator.indexlist iterate_over_stats:124 Index: logstash-2018.05.21 Size: 153.4MB Docs: 231440 2018-05-21 14:54:50,720 DEBUG curator.indexlist iterate_filters:1131 Iterating over a list of filters 2018-05-21 14:54:50,720 DEBUG curator.indexlist iterate_filters:1137 All filters: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}] 2018-05-21 14:54:50,720 DEBUG curator.indexlist iterate_filters:1139 Top of the loop: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17', '.kibana'] ``` ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 1:02pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/16 "2018-05-21T13:02:56Z")
```auto 2018-05-21 14:54:50,720 DEBUG curator.indexlist iterate_filters:1140 Un-parsed filter args: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False} 2018-05-21 14:54:50,722 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'aliases': Any(, [], , [], msg=None), 'allocation_type': Any(, , msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(, , None, msg=None), 'date_from_format': Any(, , None, msg=None), 'date_to': Any(, , None, msg=None), 'date_to_format': Any(, , None, msg=None), 'direction': Any(, , msg=None), 'disk_space': , 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(, , , , None, msg=None), 'field': Any(, , None, msg=None), 'intersect': Any(, , , , None, msg=None), 'key': Any(, , msg=None), 'kind': Any(, , msg=None), 'max_num_segments': Coerce(int, msg=None), 'pattern': Any(, , msg=None), 'period_type': Any(, , msg=None), 'reverse': Any(, , , , None, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'source': Any(, , msg=None), 'state': Any(, , msg=None), 'stats_result': Any(, , None, msg=None), 'timestring': Any(, , None, msg=None), 'threshold_behavior': Any(, , msg=None), 'unit': Any(, , msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(, , msg=None), 'use_age': , 'value': Any(, , , , , msg=None), 'week_starts_on': Any(, , None, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']")} 2018-05-21 14:54:50,723 DEBUG curator.validators.SchemaCheck __init__ :27 "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False} 2018-05-21 14:54:50,723 DEBUG curator.indexlist iterate_filters:1147 Parsed filter args: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': False} 2018-05-21 14:54:50,723 DEBUG curator.utils iterate_filters:1156 Filter args: {'kind': 'prefix', 'value': 'logstash-', 'exclude': False} 2018-05-21 14:54:50,723 DEBUG curator.utils iterate_filters:1157 Pre-instance: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17', '.kibana'] 2018-05-21 14:54:50,723 DEBUG curator.indexlist filter_by_regex:424 Filtering indices by regex 2018-05-21 14:54:50,723 DEBUG curator.indexlist empty_list_check:224 Checking for empty list 2018-05-21 14:54:50,724 DEBUG curator.indexlist working_list:235 Generating working list of indices 2018-05-21 14:54:50,724 DEBUG curator.indexlist filter_by_regex:445 Filter by regex: Index: logstash-2018.05.21 2018-05-21 14:54:50,724 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.21 is actionable and remains in the list. 2018-05-21 14:54:50,724 DEBUG curator.indexlist filter_by_regex:445 Filter by regex: Index: logstash-2018.05.20 2018-05-21 14:54:50,724 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.20 is actionable and remains in the list. 2018-05-21 14:54:50,724 DEBUG curator.indexlist filter_by_regex:445 Filter by regex: Index: logstash-2018.05.18 2018-05-21 14:54:50,724 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.18 is actionable and remains in the list. 2018-05-21 14:54:50,725 DEBUG curator.indexlist filter_by_regex:445 Filter by regex: Index: logstash-2018.05.19 2018-05-21 14:54:50,725 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.19 is actionable and remains in the list. 2018-05-21 14:54:50,725 DEBUG curator.indexlist filter_by_regex:445 Filter by regex: Index: logstash-2018.05.17 2018-05-21 14:54:50,725 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.17 is actionable and remains in the list. 2018-05-21 14:54:50,725 DEBUG curator.indexlist filter_by_regex:445 Filter by regex: Index: .kibana 2018-05-21 14:54:50,725 DEBUG curator.indexlist __not_actionable:40 Index .kibana is not actionable, removing from list. 2018-05-21 14:54:50,725 DEBUG curator.utils iterate_filters:1159 Post-instance: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17'] 2018-05-21 14:54:50,725 DEBUG curator.indexlist iterate_filters:1139 Top of the loop: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17'] ``` ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 1:03pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/17 "2018-05-21T13:03:25Z")
```auto 2018-05-21 14:54:50,725 DEBUG curator.indexlist iterate_filters:1140 Un-parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2018-05-21 14:54:50,728 DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'aliases': Any(, [], , [], msg=None), 'allocation_type': Any(, , msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(, , None, msg=None), 'date_from_format': Any(, , None, msg=None), 'date_to': Any(, , None, msg=None), 'date_to_format': Any(, , None, msg=None), 'direction': Any(, , msg=None), 'disk_space': , 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(, , , , None, msg=None), 'field': Any(, , None, msg=None), 'intersect': Any(, , , , None, msg=None), 'key': Any(, , msg=None), 'kind': Any(, , msg=None), 'max_num_segments': Coerce(int, msg=None), 'pattern': Any(, , msg=None), 'period_type': Any(, , msg=None), 'reverse': Any(, , , , None, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'source': Any(, , msg=None), 'state': Any(, , msg=None), 'stats_result': Any(, , None, msg=None), 'timestring': Any(, , None, msg=None), 'threshold_behavior': Any(, , msg=None), 'unit': Any(, , msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(, , msg=None), 'use_age': , 'value': Any(, , , , , msg=None), 'week_starts_on': Any(, , None, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']")} 2018-05-21 14:54:50,729 DEBUG curator.validators.SchemaCheck __init__ :27 "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2018-05-21 14:54:50,729 DEBUG curator.indexlist iterate_filters:1147 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2018-05-21 14:54:50,729 DEBUG curator.utils iterate_filters:1156 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2018-05-21 14:54:50,729 DEBUG curator.utils iterate_filters:1157 Pre-instance: ['logstash-2018.05.21', 'logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17'] 2018-05-21 14:54:50,729 DEBUG curator.indexlist filter_by_age:484 Filtering indices by age ``` ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 1:03pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/18 "2018-05-21T13:03:39Z")
```auto 2018-05-21 14:54:50,730 DEBUG curator.indexlist _get_name_based_ages:278 Getting ages of indices by "name" 2018-05-21 14:54:50,730 DEBUG curator.indexlist empty_list_check:224 Checking for empty list 2018-05-21 14:54:50,730 DEBUG curator.utils get_date_regex:195 regex = \d{4}\.\d{2}\.\d{2} 2018-05-21 14:54:50,730 DEBUG curator.indexlist working_list:235 Generating working list of indices 2018-05-21 14:54:50,753 DEBUG curator.indexlist working_list:235 Generating working list of indices 2018-05-21 14:54:50,753 DEBUG curator.indexlist __not_actionable:40 Index logstash-2018.05.21 is not actionable, removing from list. 2018-05-21 14:54:50,753 DEBUG curator.indexlist __excludify:59 Removed from actionable list: Index "logstash-2018.05.21" age (1526860800), direction: "older", point of reference, (1526820890) 2018-05-21 14:54:50,754 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.20 is actionable and remains in the list. 2018-05-21 14:54:50,754 DEBUG curator.indexlist __excludify:59 Remains in actionable list: Index "logstash-2018.05.20" age (1526774400), direction: "older", point of reference, (1526820890) 2018-05-21 14:54:50,754 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.18 is actionable and remains in the list. 2018-05-21 14:54:50,754 DEBUG curator.indexlist __excludify:59 Remains in actionable list: Index "logstash-2018.05.18" age (1526601600), direction: "older", point of reference, (1526820890) 2018-05-21 14:54:50,754 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.19 is actionable and remains in the list. 2018-05-21 14:54:50,754 DEBUG curator.indexlist __excludify:59 Remains in actionable list: Index "logstash-2018.05.19" age (1526688000), direction: "older", point of reference, (1526820890) 2018-05-21 14:54:50,754 DEBUG curator.indexlist __actionable:36 Index logstash-2018.05.17 is actionable and remains in the list. 2018-05-21 14:54:50,755 DEBUG curator.indexlist __excludify:59 Remains in actionable list: Index "logstash-2018.05.17" age (1526515200), direction: "older", point of reference, (1526820890) 2018-05-21 14:54:50,755 DEBUG curator.utils iterate_filters:1159 Post-instance: ['logstash-2018.05.20', 'logstash-2018.05.18', 'logstash-2018.05.19', 'logstash-2018.05.17'] 2018-05-21 14:54:50,755 DEBUG curator.actions.delete_indices __init__ :470 master_timeout value: 30s 2018-05-21 14:54:50,755 INFO curator.utils show_dry_run:918 DRY-RUN MODE. No changes will be made. 2018-05-21 14:54:50,755 INFO curator.utils show_dry_run:921 (CLOSED) indices may be shown that may not be acted on by action "delete_indices". 2018-05-21 14:54:50,755 INFO curator.utils show_dry_run:928 DRY-RUN: delete_indices: logstash-2018.05.17 with arguments: {} 2018-05-21 14:54:50,755 INFO curator.utils show_dry_run:928 DRY-RUN: delete_indices: logstash-2018.05.18 with arguments: {} 2018-05-21 14:54:50,755 INFO curator.utils show_dry_run:928 DRY-RUN: delete_indices: logstash-2018.05.19 with arguments: {} 2018-05-21 14:54:50,756 INFO curator.utils show_dry_run:928 DRY-RUN: delete_indices: logstash-2018.05.20 with arguments: {} 2018-05-21 14:54:50,756 INFO curator.cli run:194 Action ID: 1, "delete_indices" completed. 2018-05-21 14:54:50,756 INFO curator.cli run:195 Job completed. ``` Sorry I couldn't copy it in one post ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 21, 2018, 1:06pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/19 "2018-05-21T13:06:37Z")
So, what's the problem? This looks completely proper to me. You start with `logstash-2018.05.21`, `logstash-2018.05.20`, `logstash-2018.05.19`, `logstash-2018.05.18`, and `logstash-2018.05.17`. It's marking all but `logstash-2018.05.21`, for deletion, which is what your action file is configured to do. So what is the problem? Are you running with `--dry-run` all the time? Curator won't actually do the deletes if you run with `--dry-run`. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [May 21, 2018, 1:08pm UTC](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505/20 "2018-05-21T13:08:52Z")
But when I do the Cat, I still see the older index, also in timelion I'm seing the oldest one. Yes, I always run it with --dry-run, how should I run it to delete the oldest logs ? [Next page](https://discuss.elastic.co/t/curator-doesnt-purge-data/132505.md?page=2)