Deleting indices with curator 4.0


#1

Hi there,

I just read this thread How do I know if curator is cleaning up? (SOLVED) and same as the last comment, when i use this command line :

curator delete indices --older-than 5 --time-unit days --timestring %Y.%m.%d

i got this :
> root@itsupervision:/# curator delete indices --older-than 7 --time-unit days --timestring %Y.%m.%d

Error: no such option: --older-than

I assume it's because of the new version but what is the equivalent for the 4.0.1 version then ?

Thanks for reading,

Josselin


(Aaron Mildenstein) #2

Curator v4 is a major release update, and it fundamentally changes the usage behavior. It uses YAML configuration files now instead of command-line flags. I recommend you read up on the new version at https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html


(Ravi Shanker Reddy) #3

Can you please explain this a little bit more. From the link you mentioned I understand that I need two files now.

Action File Configuration file

  1. In test_curator.YML
    ` 1:
    action: delete_indices
    description: >-
    Delete indices older than 45 days (based on index name), for logstash-
    prefixed indices. Ignore the error if the filter does not result in an
    actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
    ignore_empty_list: True
    timeout_override:
    continue_if_exception: False
    disable_action: True
    filters:

    • filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    • filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 45
      exclude:`
  2. How to create the config file. Whats the extension??

how to mention the file names while running in the curator????


#4

Hi,

You have to create a curator conf file in the default conf directory (~/.curator/) and it has a .yml extension


client:
hosts:
- 127.0.0.1
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
aws_key:
aws_secret_key:
aws_region:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False

logging:
loglevel: INFO
logfile:
logformat: default

and then you can run curator with

curator PATH_ACTION_FILE


(Ravi Shanker Reddy) #5

I cant find any directory in the root with that name (Searched with ls -alrt).
As you mentioned I created a file test_curator_config.yml. [quote="Jpuch, post:4, topic:55729"]
curator PATH_ACTION_FILE
[/quote]

Here I am mentioning only action file name & path. Then how the curator understand which file it has to read as configuration file. (I am not mentioned any thing about conf file in action file)

Sorry if these are very silly questions. Very new to ES


#6

I have to create this directory myself too and put my curator.yml in it with the conf i posted above.

You don't have to mention the name of the conf file in the command line because it's the default directory.


(Ravi Shanker Reddy) #7

Hi Jpuch

mkdir -p /root/.curator #Created the directory
created file curator.yml in the directory

`client:
hosts:

  • 172.16.15.153
    port: 9200
    url_prefix:
    use_ssl: False
    certificate:
    client_cert:
    client_key:
    aws_key:
    aws_secret_key:
    aws_region:
    ssl_no_validate: False
    http_auth:
    timeout: 30
    master_only: False
    logging:
    loglevel: INFO
    logfile:
    logformat: default`

And also created action file as action_file.yml

actions: 1: action: delete_indices description: >- options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: packetbeat- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 3 exclude:

and run as curator /root/action_file.yml

Then its showing an error

[root@smsc ~]# curator /root/action_file.yml Traceback (most recent call last): File "/usr/local/bin/curator", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python2.7/site-packages/curator/curator.py", line 5, in main cli() File "/usr/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/curator/cli.py", line 135, in cli yaml_log_opts = prune_nones(yaml_config['logging']) File "/usr/local/lib/python2.7/site-packages/curator/utils.py", line 1002, in prune_nones return dict([(k,v) for k, v in mydict.items() if v != None and v != 'None']) AttributeError: 'NoneType' object has no attribute 'items'

EDIT : Sorry for the reply. I missed the indent in the config file. After indent all working fine


(Mike Sparr) #8

I had similar issue when upgrading to 4.0. One thing you should do is omit all the empty lines on config file or comment them with hash (pound sign). They will load as empty strings.

I just created in my user home directory a config and action file like so:

curator_config.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- search1
port: 9200
timeout: 30

logging:
  loglevel: INFO

curator_action.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for custom-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
continue_if_exception: False
disable_action: False
filters:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 5

Then run curator, passing in the config files' paths and view output as so (I was deleting any indices with timestamp suffices older than 5 days):

curator --config ~/curator_config.yml ~/curator_action.yml
2016-09-26 14:33:20,862 INFO      Preparing Action ID: 1, "delete_indices"
2016-09-26 14:33:20,870 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 5 days (based on index name), for custom- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2016-09-26 14:33:21,592 INFO      Deleting selected indices: [u'topbeat-2016.09.17', u'topbeat-2016.09.16', u'topbeat-2016.09.15', u'topbeat-2016.09.14', u'feedbeat-2016.09.20', u'feedbeat-2016.09.21', u'topbeat-2016.09.19', u'topbeat-2016.09.18', u'feedbeat-2016.09.06', u'feedbeat-2016.09.07', u'feedbeat-2016.09.08', u'feedbeat-2016.09.09', u'feedbeat-2016.09.19', u'filebeat-2016.09.20', u'filebeat-2016.09.21', u'packetbeat-2016.09.19', u'packetbeat-2016.09.18', u'packetbeat-2016.09.12', u'packetbeat-2016.09.13', u'packetbeat-2016.09.14', u'packetbeat-2016.09.11', u'packetbeat-2016.09.17', u'packetbeat-2016.09.16', u'packetbeat-2016.09.15', u'packetbeat-2016.09.21', u'packetbeat-2016.09.08', u'packetbeat-2016.09.09', u'feedbeat-2016.09.11', u'feedbeat-2016.09.10', u'feedbeat-2016.09.13', u'feedbeat-2016.09.12', u'feedbeat-2016.09.15', u'feedbeat-2016.09.14', u'feedbeat-2016.09.17', u'feedbeat-2016.09.16', u'feedbeat-2016.09.18', u'filebeat-2016.09.17', u'filebeat-2016.09.16', u'filebeat-2016.09.15', u'filebeat-2016.09.14', u'filebeat-2016.09.19', u'filebeat-2016.09.18', u'packetbeat-2016.09.20', u'topbeat-2016.09.20', u'topbeat-2016.09.21']
2016-09-26 14:33:21,592 INFO      ---deleting index feedbeat-2016.09.20
2016-09-26 14:33:21,593 INFO      ---deleting index feedbeat-2016.09.21
2016-09-26 14:33:21,593 INFO      ---deleting index topbeat-2016.09.19
2016-09-26 14:33:21,593 INFO      ---deleting index topbeat-2016.09.18
2016-09-26 14:33:21,593 INFO      ---deleting index feedbeat-2016.09.19
2016-09-26 14:33:21,593 INFO      ---deleting index filebeat-2016.09.20
2016-09-26 14:33:21,593 INFO      ---deleting index filebeat-2016.09.21
2016-09-26 14:33:21,594 INFO      ---deleting index packetbeat-2016.09.19
2016-09-26 14:33:21,594 INFO      ---deleting index packetbeat-2016.09.21
2016-09-26 14:33:21,596 INFO      ---deleting index filebeat-2016.09.19
2016-09-26 14:33:21,596 INFO      ---deleting index packetbeat-2016.09.20
2016-09-26 14:33:21,596 INFO      ---deleting index topbeat-2016.09.20
2016-09-26 14:33:21,597 INFO      ---deleting index topbeat-2016.09.21
2016-09-26 14:33:30,677 INFO      Action ID: 1, "delete_indices" completed.
2016-09-26 14:33:30,677 INFO      Job completed.

I hope this is helpful. FYI, my old command in pre 4.0 versions was the following:

#! /bin/bash
/usr/local/bin/curator --host search1 delete indices --older-than 5 --time-unit days --timestring '%Y.%m.%d'
  • Mike

(Rohini Choudhary) #9

Thank you so much for your full answer. Worked awesome for me!


(vshah) #10

Nice..Worked for me. Thanks a lot.


(system) #11