Understanding curator filters for aging out old indices

Elastic Stack Elasticsearch
1

I'm running Elastic Stack 7.2 to support an ElastiFlow installation. In my use case the data being collected is ephemeral and I have no need to keep it longer than a few days. It appears that curator is the tool of choice for aging out old data, and I'm trying to get that setup to remove data older than 7 days.

Test system is an Ubuntu 18.04 VM. Here's how I installed curator:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main' | sudo tee /etc/apt/sources.list.d/curator.list
sudo apt-get update && sudo apt-get install elasticsearch-curator -y

Current indices

$ curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   elastiflow-3.5.0-2019.07.04 Gf6sb_MuSA-7tQba2ZFOcw   3   1    2689174            0      1.1gb          1.1gb
yellow open   elastiflow-3.5.0-2019.07.09 KLE16gGLROqu9KE5ypP0SA   3   1    1488207            0    758.5mb        758.5mb
yellow open   elastiflow-3.5.0-2019.07.08 TPZKrxHhQrObj4dy81mnJQ   3   1     245372            0    126.6mb        126.6mb
green  open   .kibana_1                   UGet4IXXQQidoIC4VBPkig   1   0        769            5    515.6kb        515.6kb
yellow open   elastiflow-3.5.0-2019.06.27 Q7Kcm0s5R4y5fDPZmgSmPA   3   1     210177            0    102.8mb        102.8mb
yellow open   elastiflow-3.5.0-2019.07.02 5cYadwh8R66hf0ZgrkL-eQ   3   1    2410084            0        1gb            1gb
yellow open   elastiflow-3.5.0-2019.06.30 N3cax3BhTFK9H3yTa9GPVg   3   1    2216299            0        1gb            1gb
yellow open   elastiflow-3.5.0-2019.06.28 _E27jhOOSR2EumJzkx3T0g   3   1    1778566            0    832.3mb        832.3mb
yellow open   elastiflow-3.5.0-2019.07.03 EEKZ4XiqSp2h9kOG4whP1Q   3   1    1627892            0    743.4mb        743.4mb
yellow open   elastiflow-3.5.0-2019.07.07 mLH5DyjISESbwIRNbRxmBQ   3   1      58706            0       30mb           30mb
yellow open   elastiflow-3.5.0-2019.06.29 XCypvYbEQzuLQ-s8rC5SBA   3   1    2186678            0   1015.1mb       1015.1mb
yellow open   elastiflow-3.5.0-1970.01.02 DJwibAyETv6JKqzmd32Tjw   3   1         10            0    142.4kb        142.4kb
yellow open   elastiflow-3.5.0-2019.07.10 aFJ5I3RYSomnjMkztrDcuw   3   1    1584475            0    971.4mb        971.4mb
yellow open   elastiflow-3.5.0-2019.07.05 XydqlPlNSsSpfb9iOfj4SQ   3   1    1340542            0    636.1mb        636.1mb
green  open   .kibana_task_manager        zx_DZzn7R1OAR8AnbFefXw   1   0          2            0     30.8kb         30.8kb
yellow open   elastiflow-3.5.0-2019.07.01 FNAGqbouRpi93m_3IFAptg   3   1    2346211            0        1gb            1gb

config files

/opt/elasticsearch-curator/curator.yml

client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

/opt/elasticsearch-curator/curator-delete-indices.yml

actions:
  1:
    action: delete_indices
    description: Delete ElastiFlow indices older than 7 days
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: elastiflow-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7

Dry run results

$ sudo curator /opt/elasticsearch-curator/curator-delete-indices.yml --config /opt/elasticsearch-curator/curator.yml --dry-run
2019-07-10 15:49:50,843 INFO      Preparing Action ID: 1, "delete_indices"
2019-07-10 15:49:50,850 INFO      Trying Action ID: 1, "delete_indices": Delete elastiflow indices older than 7 days
2019-07-10 15:49:51,238 INFO      DRY-RUN MODE.  No changes will be made.
2019-07-10 15:49:51,238 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2019-07-10 15:49:51,238 INFO      Action ID: 1, "delete_indices" completed.
2019-07-10 15:49:51,238 INFO      Job completed.

Full run results

$ sudo curator /opt/elasticsearch-curator/curator-delete-indices.yml --config /opt/elasticsearch-curator/curator.yml
2019-07-10 15:50:13,961 INFO      Preparing Action ID: 1, "delete_indices"
2019-07-10 15:50:13,967 INFO      Trying Action ID: 1, "delete_indices": Delete elastiflow indices older than 7 days
2019-07-10 15:50:14,356 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-07-10 15:50:14,357 INFO      Action ID: 1, "delete_indices" completed.
2019-07-10 15:50:14,357 INFO      Job completed.

The Problem

Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'> sums it up nicely. I'm not sure where I'm going wrong here but it appears I have the filter conditions incorrect.

2

Try setting loglevel: DEBUG and doing a dry run again. It will show you all of the filtering decisions.

3

The full debug output log is a little long so I've dropped it here.

I'm trying to understand the flow here. I'm seeing 3 passes of iterate_filters with 3 filters which has me confused as I've only declared 2. There's a 'filtertype': 'pattern' which is correctly matching indices by the provided name pattern. There's a 'filtertype': 'age' which similarly appears to be working as intended. At this point the list has the correct indices to prune.

Then we have a 'filtertype': 'ilm' which I don't understand, and which is removing the entire list, leaving nothing to delete.

4

If the ILM filter is catching the indices, it means that you could potentially use Index Lifecycle Management (ILM) to manage the indices instead of Curator. Something, somewhere is applying a policy name to your indices, and because of that, Curator by default will not touch them unless you add the allow_ilm_indices: true directive under options.

1 Like
5

That was exactly it! I'm a bit of an elastic newb so the "ILM" bit didn't mean anything to me. With that context, now I understand what's happening. Thanks @theuntergeek !

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.