# Understanding curator filters for aging out old indices **URL:** \ **Category:** Elasticsearch\ **Created:** [July 10, 2019, 7:54pm UTC](https://discuss.elastic.co/t/understanding-curator-filters-for-aging-out-old-indices/189862 "2019-07-10T19:54:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![luma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luma/32/49830_2.png) [@luma](https://discuss.elastic.co/u/luma)\ **Post date:** [July 10, 2019, 7:54pm UTC](https://discuss.elastic.co/t/understanding-curator-filters-for-aging-out-old-indices/189862/1 "2019-07-10T19:54:39Z")
I'm running Elastic Stack 7.2 to support an [ElastiFlow](https://github.com/robcowart/elastiflow) installation. In my use case the data being collected is ephemeral and I have no need to keep it longer than a few days. It appears that `curator` is the tool of choice for aging out old data, and I'm trying to get that setup to remove data older than 7 days. Test system is an Ubuntu 18.04 VM. Here's how I installed `curator`: ```bash wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo 'deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main' | sudo tee /etc/apt/sources.list.d/curator.list sudo apt-get update && sudo apt-get install elasticsearch-curator -y ``` ## Current indices ```bash $ curl -XGET 'localhost:9200/_cat/indices?v&pretty' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open elastiflow-3.5.0-2019.07.04 Gf6sb_MuSA-7tQba2ZFOcw 3 1 2689174 0 1.1gb 1.1gb yellow open elastiflow-3.5.0-2019.07.09 KLE16gGLROqu9KE5ypP0SA 3 1 1488207 0 758.5mb 758.5mb yellow open elastiflow-3.5.0-2019.07.08 TPZKrxHhQrObj4dy81mnJQ 3 1 245372 0 126.6mb 126.6mb green open .kibana_1 UGet4IXXQQidoIC4VBPkig 1 0 769 5 515.6kb 515.6kb yellow open elastiflow-3.5.0-2019.06.27 Q7Kcm0s5R4y5fDPZmgSmPA 3 1 210177 0 102.8mb 102.8mb yellow open elastiflow-3.5.0-2019.07.02 5cYadwh8R66hf0ZgrkL-eQ 3 1 2410084 0 1gb 1gb yellow open elastiflow-3.5.0-2019.06.30 N3cax3BhTFK9H3yTa9GPVg 3 1 2216299 0 1gb 1gb yellow open elastiflow-3.5.0-2019.06.28 _E27jhOOSR2EumJzkx3T0g 3 1 1778566 0 832.3mb 832.3mb yellow open elastiflow-3.5.0-2019.07.03 EEKZ4XiqSp2h9kOG4whP1Q 3 1 1627892 0 743.4mb 743.4mb yellow open elastiflow-3.5.0-2019.07.07 mLH5DyjISESbwIRNbRxmBQ 3 1 58706 0 30mb 30mb yellow open elastiflow-3.5.0-2019.06.29 XCypvYbEQzuLQ-s8rC5SBA 3 1 2186678 0 1015.1mb 1015.1mb yellow open elastiflow-3.5.0-1970.01.02 DJwibAyETv6JKqzmd32Tjw 3 1 10 0 142.4kb 142.4kb yellow open elastiflow-3.5.0-2019.07.10 aFJ5I3RYSomnjMkztrDcuw 3 1 1584475 0 971.4mb 971.4mb yellow open elastiflow-3.5.0-2019.07.05 XydqlPlNSsSpfb9iOfj4SQ 3 1 1340542 0 636.1mb 636.1mb green open .kibana_task_manager zx_DZzn7R1OAR8AnbFefXw 1 0 2 0 30.8kb 30.8kb yellow open elastiflow-3.5.0-2019.07.01 FNAGqbouRpi93m_3IFAptg 3 1 2346211 0 1gb 1gb ``` ## config files ### /opt/elasticsearch-curator/curator.yml ```auto client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] ``` ### /opt/elasticsearch-curator/curator-delete-indices.yml ```auto actions: 1: action: delete_indices description: Delete ElastiFlow indices older than 7 days options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: elastiflow- - filtertype: age source: creation_date direction: older unit: days unit_count: 7 ``` ## Dry run results ```bash $ sudo curator /opt/elasticsearch-curator/curator-delete-indices.yml --config /opt/elasticsearch-curator/curator.yml --dry-run 2019-07-10 15:49:50,843 INFO Preparing Action ID: 1, "delete_indices" 2019-07-10 15:49:50,850 INFO Trying Action ID: 1, "delete_indices": Delete elastiflow indices older than 7 days 2019-07-10 15:49:51,238 INFO DRY-RUN MODE. No changes will be made. 2019-07-10 15:49:51,238 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices". 2019-07-10 15:49:51,238 INFO Action ID: 1, "delete_indices" completed. 2019-07-10 15:49:51,238 INFO Job completed. ``` ## Full run results ```bash $ sudo curator /opt/elasticsearch-curator/curator-delete-indices.yml --config /opt/elasticsearch-curator/curator.yml 2019-07-10 15:50:13,961 INFO Preparing Action ID: 1, "delete_indices" 2019-07-10 15:50:13,967 INFO Trying Action ID: 1, "delete_indices": Delete elastiflow indices older than 7 days 2019-07-10 15:50:14,356 INFO Skipping action "delete_indices" due to empty list: 2019-07-10 15:50:14,357 INFO Action ID: 1, "delete_indices" completed. 2019-07-10 15:50:14,357 INFO Job completed. ``` # The Problem `Skipping action "delete_indices" due to empty list: ` sums it up nicely. I'm not sure where I'm going wrong here but it appears I have the filter conditions incorrect. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [July 10, 2019, 9:14pm UTC](https://discuss.elastic.co/t/understanding-curator-filters-for-aging-out-old-indices/189862/2 "2019-07-10T21:14:11Z")
Try setting `loglevel: DEBUG` and doing a dry run again. It will show you all of the filtering decisions. ---
**Author:** ![luma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luma/32/49830_2.png) [@luma](https://discuss.elastic.co/u/luma)\ **Post date:** [July 10, 2019, 11:02pm UTC](https://discuss.elastic.co/t/understanding-curator-filters-for-aging-out-old-indices/189862/3 "2019-07-10T23:02:05Z")
The full debug output log is a little long so I've [dropped it here](https://pastebin.com/raw/8CUZmtf5). I'm trying to understand the flow here. I'm seeing 3 passes of `iterate_filters` with 3 filters which has me confused as I've only declared 2. There's a `'filtertype': 'pattern'` which is correctly matching indices by the provided name pattern. There's a `'filtertype': 'age'` which similarly appears to be working as intended. At this point the list has the correct indices to prune. Then we have a `'filtertype': 'ilm'` which I don't understand, and which is removing the entire list, leaving nothing to delete. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [July 10, 2019, 11:25pm UTC](https://discuss.elastic.co/t/understanding-curator-filters-for-aging-out-old-indices/189862/4 "2019-07-10T23:25:02Z")
If the ILM filter is catching the indices, it means that you could potentially use Index Lifecycle Management (ILM) to manage the indices instead of Curator. Something, somewhere is applying a policy name to your indices, and because of that, Curator by default will not touch them unless you add the [`allow_ilm_indices: true`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.7/option_allow_ilm.html) directive under `options`. ---
**Author:** ![luma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luma/32/49830_2.png) [@luma](https://discuss.elastic.co/u/luma)\ **Post date:** [July 11, 2019, 10:49am UTC](https://discuss.elastic.co/t/understanding-curator-filters-for-aging-out-old-indices/189862/5 "2019-07-11T10:49:02Z")
That was exactly it! I'm a bit of an elastic newb so the "ILM" bit didn't mean anything to me. With that context, now I understand what's happening. Thanks @theuntergeek ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 8, 2019, 10:49am UTC](https://discuss.elastic.co/t/understanding-curator-filters-for-aging-out-old-indices/189862/6 "2019-08-08T10:49:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.