Skipping action "delete_indices" due to empty list! Curator

Elastic Stack Elasticsearch
1

Hello,
I wanna delete the older data from elasticsearch, I used curator for that:
curator.yml

client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

delete_indices.yml
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1
      exclude:
#    - filtertype: space
#      disk-space: 65
#      use_age: True
#      source: creation_date

when I run this cmd I got error message:

curator delete_indices.yml --config curator.yml --dry-run
2018-08-20 10:29:10,012 INFO      Preparing Action ID: 1, "delete_indices"
2018-08-20 10:29:10,017 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2018-08-20 10:29:10,030 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-08-20 10:29:10,031 INFO      Action ID: 1, "delete_indices" completed.
2018-08-20 10:29:10,031 INFO      Job completed.

my logstash.conf

output{

  elasticsearch
    {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "test-%{+YYYY.MM.dd}"
    }
    stdout{codec=>rubydebug}}

curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-es-6-2018.08.20 RW_PLI8WT3WvKHPFFaymFg   1   0         62            0     56.4kb         56.4kb
green  open   .kibana                     qDdB3taHR_qvR79nVNPrWw   1   0         39            3     87.6kb         87.6kb
yellow open   test-2018.08.20             C4p4cFVTROK8gyT4npgYjg   5   1     141543            0     73.1mb         73.1mb
yellow open   test-2018.08.17             XlDW0lapQH2gLYm38rjCoQ   5   1     411935            0    211.1mb        211.1mb

Any idea whats wrong!

Thank you a lot

2

Yep. Here you are:

Combine that with:

And that's why it doesn't work. Your pattern filter is only going to work on indices named logstash-*, but your logstash is only creating indices named test-*.

1 Like
3

Thank you, it's working now, I have another question to delete data if disk-space is more than 65go, could I use this configuration:

#    - filtertype: space
#      disk_space: 65
#      use_age: True
#      source: creation_date

Thank you

4

I would remove the age filter in that case, but yes, you understand the concept correctly.

If you neglect to remove the age filter, the combination of age and space filters will not delete any but those both older than 1 days and greater than 65 G.

5

I think I misunderstood the use of this filtertype, I would like to delete oldest data after a week but if there is a risk to use 80% of my local disk ( let's say approximately 65go) in this case I will also delete data. So I will combine the age of data and the size of all data indexed on elasticsearch.

6

What you should do then, is have 2 actions. This will guarantee both conditions. The first covers the week of retention. The second will always delete the oldest indices in excess of 65G, should that condition prove true. If not, nothing will happen.

Sample configuration:

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days
    options:
      ignore_empty_list: true
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
  2:
    action: delete_indices
    description: >-
      Delete indices in excess of 65 gigabytes
    options:
      ignore_empty_list: true
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: space
      disk-space: 65
      use_age: true
      source: creation_date
1 Like
7

Great! Thanks a lot :blush:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.