Understanding curator filters for aging out old indices

I'm running Elastic Stack 7.2 to support an ElastiFlow installation. In my use case the data being collected is ephemeral and I have no need to keep it longer than a few days. It appears that curator is the tool of choice for aging out old data, and I'm trying to get that setup to remove data older than 7 days.

Test system is an Ubuntu 18.04 VM. Here's how I installed curator:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main' | sudo tee /etc/apt/sources.list.d/curator.list
sudo apt-get update && sudo apt-get install elasticsearch-curator -y

Current indices

$ curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   elastiflow-3.5.0-2019.07.04 Gf6sb_MuSA-7tQba2ZFOcw   3   1    2689174            0      1.1gb          1.1gb
yellow open   elastiflow-3.5.0-2019.07.09 KLE16gGLROqu9KE5ypP0SA   3   1    1488207            0    758.5mb        758.5mb
yellow open   elastiflow-3.5.0-2019.07.08 TPZKrxHhQrObj4dy81mnJQ   3   1     245372            0    126.6mb        126.6mb
green  open   .kibana_1                   UGet4IXXQQidoIC4VBPkig   1   0        769            5    515.6kb        515.6kb
yellow open   elastiflow-3.5.0-2019.06.27 Q7Kcm0s5R4y5fDPZmgSmPA   3   1     210177            0    102.8mb        102.8mb
yellow open   elastiflow-3.5.0-2019.07.02 5cYadwh8R66hf0ZgrkL-eQ   3   1    2410084            0        1gb            1gb
yellow open   elastiflow-3.5.0-2019.06.30 N3cax3BhTFK9H3yTa9GPVg   3   1    2216299            0        1gb            1gb
yellow open   elastiflow-3.5.0-2019.06.28 _E27jhOOSR2EumJzkx3T0g   3   1    1778566            0    832.3mb        832.3mb
yellow open   elastiflow-3.5.0-2019.07.03 EEKZ4XiqSp2h9kOG4whP1Q   3   1    1627892            0    743.4mb        743.4mb
yellow open   elastiflow-3.5.0-2019.07.07 mLH5DyjISESbwIRNbRxmBQ   3   1      58706            0       30mb           30mb
yellow open   elastiflow-3.5.0-2019.06.29 XCypvYbEQzuLQ-s8rC5SBA   3   1    2186678            0   1015.1mb       1015.1mb
yellow open   elastiflow-3.5.0-1970.01.02 DJwibAyETv6JKqzmd32Tjw   3   1         10            0    142.4kb        142.4kb
yellow open   elastiflow-3.5.0-2019.07.10 aFJ5I3RYSomnjMkztrDcuw   3   1    1584475            0    971.4mb        971.4mb
yellow open   elastiflow-3.5.0-2019.07.05 XydqlPlNSsSpfb9iOfj4SQ   3   1    1340542            0    636.1mb        636.1mb
green  open   .kibana_task_manager        zx_DZzn7R1OAR8AnbFefXw   1   0          2            0     30.8kb         30.8kb
yellow open   elastiflow-3.5.0-2019.07.01 FNAGqbouRpi93m_3IFAptg   3   1    2346211            0        1gb            1gb

config files


  port: 9200
  use_ssl: False
  ssl_no_validate: False
  timeout: 30
  master_only: False

  loglevel: INFO
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']


    action: delete_indices
    description: Delete ElastiFlow indices older than 7 days
      ignore_empty_list: True
      continue_if_exception: False
      disable_action: False
    - filtertype: pattern
      kind: prefix
      value: elastiflow-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7

Dry run results

$ sudo curator /opt/elasticsearch-curator/curator-delete-indices.yml --config /opt/elasticsearch-curator/curator.yml --dry-run
2019-07-10 15:49:50,843 INFO      Preparing Action ID: 1, "delete_indices"
2019-07-10 15:49:50,850 INFO      Trying Action ID: 1, "delete_indices": Delete elastiflow indices older than 7 days
2019-07-10 15:49:51,238 INFO      DRY-RUN MODE.  No changes will be made.
2019-07-10 15:49:51,238 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2019-07-10 15:49:51,238 INFO      Action ID: 1, "delete_indices" completed.
2019-07-10 15:49:51,238 INFO      Job completed.

Full run results

$ sudo curator /opt/elasticsearch-curator/curator-delete-indices.yml --config /opt/elasticsearch-curator/curator.yml
2019-07-10 15:50:13,961 INFO      Preparing Action ID: 1, "delete_indices"
2019-07-10 15:50:13,967 INFO      Trying Action ID: 1, "delete_indices": Delete elastiflow indices older than 7 days
2019-07-10 15:50:14,356 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-07-10 15:50:14,357 INFO      Action ID: 1, "delete_indices" completed.
2019-07-10 15:50:14,357 INFO      Job completed.

The Problem

Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'> sums it up nicely. I'm not sure where I'm going wrong here but it appears I have the filter conditions incorrect.

Try setting loglevel: DEBUG and doing a dry run again. It will show you all of the filtering decisions.

The full debug output log is a little long so I've dropped it here.

I'm trying to understand the flow here. I'm seeing 3 passes of iterate_filters with 3 filters which has me confused as I've only declared 2. There's a 'filtertype': 'pattern' which is correctly matching indices by the provided name pattern. There's a 'filtertype': 'age' which similarly appears to be working as intended. At this point the list has the correct indices to prune.

Then we have a 'filtertype': 'ilm' which I don't understand, and which is removing the entire list, leaving nothing to delete.

If the ILM filter is catching the indices, it means that you could potentially use Index Lifecycle Management (ILM) to manage the indices instead of Curator. Something, somewhere is applying a policy name to your indices, and because of that, Curator by default will not touch them unless you add the allow_ilm_indices: true directive under options.

1 Like

That was exactly it! I'm a bit of an elastic newb so the "ILM" bit didn't mean anything to me. With that context, now I understand what's happening. Thanks @theuntergeek !

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.