Deleting indices with curator 4.0

Jpuch · July 18, 2016, 7:22am

Hi there,

I just read this thread How do I know if curator is cleaning up? (SOLVED) and same as the last comment, when i use this command line :

curator delete indices --older-than 5 --time-unit days --timestring %Y.%m.%d

i got this :
> root@itsupervision:/# curator delete indices --older-than 7 --time-unit days --timestring %Y.%m.%d

Error: no such option: --older-than

I assume it's because of the new version but what is the equivalent for the 4.0.1 version then ?

Thanks for reading,

Josselin

theuntergeek · July 18, 2016, 1:00pm

Curator v4 is a major release update, and it fundamentally changes the usage behavior. It uses YAML configuration files now instead of command-line flags. I recommend you read up on the new version at https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

Ravi_Shanker_Reddy · July 19, 2016, 6:21am

Can you please explain this a little bit more. From the link you mentioned I understand that I need two files now.

Action File Configuration file

In test_curator.YML
` 1:
action: delete_indices
description: >-
Delete indices older than 45 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: True
filters:
- filtertype: pattern
  kind: prefix
  value: logstash-
  exclude:
- filtertype: age
  source: name
  direction: older
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 45
  exclude:`
How to create the config file. Whats the extension??

how to mention the file names while running in the curator????

Jpuch · July 19, 2016, 10:41am

Hi,

You have to create a curator conf file in the default conf directory (~/.curator/) and it has a .yml extension

client:
hosts:
- 127.0.0.1
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
aws_key:
aws_secret_key:
aws_region:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False

logging:
loglevel: INFO
logfile:
logformat: default

and then you can run curator with

curator PATH_ACTION_FILE

Ravi_Shanker_Reddy · July 20, 2016, 6:51am

I cant find any directory in the root with that name (Searched with ls -alrt).
As you mentioned I created a file test_curator_config.yml. [quote="Jpuch, post:4, topic:55729"]
curator PATH_ACTION_FILE
[/quote]

Here I am mentioning only action file name & path. Then how the curator understand which file it has to read as configuration file. (I am not mentioned any thing about conf file in action file)

Sorry if these are very silly questions. Very new to ES

Jpuch · July 20, 2016, 7:19am

I have to create this directory myself too and put my curator.yml in it with the conf i posted above.

You don't have to mention the name of the conf file in the command line because it's the default directory.

Ravi_Shanker_Reddy · July 20, 2016, 8:55am

Hi Jpuch

mkdir -p /root/.curator #Created the directory
created file curator.yml in the directory

`client:
hosts:

172.16.15.153
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
aws_key:
aws_secret_key:
aws_region:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile:
logformat: default`

And also created action file as action_file.yml

actions: 1: action: delete_indices description: >- options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: packetbeat- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 3 exclude:

and run as curator /root/action_file.yml

Then its showing an error

[root@smsc ~]# curator /root/action_file.yml Traceback (most recent call last): File "/usr/local/bin/curator", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python2.7/site-packages/curator/curator.py", line 5, in main cli() File "/usr/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/curator/cli.py", line 135, in cli yaml_log_opts = prune_nones(yaml_config['logging']) File "/usr/local/lib/python2.7/site-packages/curator/utils.py", line 1002, in prune_nones return dict([(k,v) for k, v in mydict.items() if v != None and v != 'None']) AttributeError: 'NoneType' object has no attribute 'items'

EDIT : Sorry for the reply. I missed the indent in the config file. After indent all working fine

mikesparr · September 26, 2016, 8:42pm

I had similar issue when upgrading to 4.0. One thing you should do is omit all the empty lines on config file or comment them with hash (pound sign). They will load as empty strings.

I just created in my user home directory a config and action file like so:

curator_config.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- search1
port: 9200
timeout: 30

logging:
  loglevel: INFO

curator_action.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for custom-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
continue_if_exception: False
disable_action: False
filters:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 5

Then run curator, passing in the config files' paths and view output as so (I was deleting any indices with timestamp suffices older than 5 days):

curator --config ~/curator_config.yml ~/curator_action.yml
2016-09-26 14:33:20,862 INFO      Preparing Action ID: 1, "delete_indices"
2016-09-26 14:33:20,870 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 5 days (based on index name), for custom- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2016-09-26 14:33:21,592 INFO      Deleting selected indices: [u'topbeat-2016.09.17', u'topbeat-2016.09.16', u'topbeat-2016.09.15', u'topbeat-2016.09.14', u'feedbeat-2016.09.20', u'feedbeat-2016.09.21', u'topbeat-2016.09.19', u'topbeat-2016.09.18', u'feedbeat-2016.09.06', u'feedbeat-2016.09.07', u'feedbeat-2016.09.08', u'feedbeat-2016.09.09', u'feedbeat-2016.09.19', u'filebeat-2016.09.20', u'filebeat-2016.09.21', u'packetbeat-2016.09.19', u'packetbeat-2016.09.18', u'packetbeat-2016.09.12', u'packetbeat-2016.09.13', u'packetbeat-2016.09.14', u'packetbeat-2016.09.11', u'packetbeat-2016.09.17', u'packetbeat-2016.09.16', u'packetbeat-2016.09.15', u'packetbeat-2016.09.21', u'packetbeat-2016.09.08', u'packetbeat-2016.09.09', u'feedbeat-2016.09.11', u'feedbeat-2016.09.10', u'feedbeat-2016.09.13', u'feedbeat-2016.09.12', u'feedbeat-2016.09.15', u'feedbeat-2016.09.14', u'feedbeat-2016.09.17', u'feedbeat-2016.09.16', u'feedbeat-2016.09.18', u'filebeat-2016.09.17', u'filebeat-2016.09.16', u'filebeat-2016.09.15', u'filebeat-2016.09.14', u'filebeat-2016.09.19', u'filebeat-2016.09.18', u'packetbeat-2016.09.20', u'topbeat-2016.09.20', u'topbeat-2016.09.21']
2016-09-26 14:33:21,592 INFO      ---deleting index feedbeat-2016.09.20
2016-09-26 14:33:21,593 INFO      ---deleting index feedbeat-2016.09.21
2016-09-26 14:33:21,593 INFO      ---deleting index topbeat-2016.09.19
2016-09-26 14:33:21,593 INFO      ---deleting index topbeat-2016.09.18
2016-09-26 14:33:21,593 INFO      ---deleting index feedbeat-2016.09.19
2016-09-26 14:33:21,593 INFO      ---deleting index filebeat-2016.09.20
2016-09-26 14:33:21,593 INFO      ---deleting index filebeat-2016.09.21
2016-09-26 14:33:21,594 INFO      ---deleting index packetbeat-2016.09.19
2016-09-26 14:33:21,594 INFO      ---deleting index packetbeat-2016.09.21
2016-09-26 14:33:21,596 INFO      ---deleting index filebeat-2016.09.19
2016-09-26 14:33:21,596 INFO      ---deleting index packetbeat-2016.09.20
2016-09-26 14:33:21,596 INFO      ---deleting index topbeat-2016.09.20
2016-09-26 14:33:21,597 INFO      ---deleting index topbeat-2016.09.21
2016-09-26 14:33:30,677 INFO      Action ID: 1, "delete_indices" completed.
2016-09-26 14:33:30,677 INFO      Job completed.

I hope this is helpful. FYI, my old command in pre 4.0 versions was the following:

#! /bin/bash
/usr/local/bin/curator --host search1 delete indices --older-than 5 --time-unit days --timestring '%Y.%m.%d'

Mike

Rohini_Choudhary · February 8, 2017, 2:57pm

Thank you so much for your full answer. Worked awesome for me!

vaibhav · March 27, 2017, 9:04am

Nice..Worked for me. Thanks a lot.

Topic		Replies	Views
Using curator delete index Elasticsearch	4	631	March 11, 2018
[solved]I want to delete the indices that are older than 1 day? Elasticsearch	8	3473	March 31, 2017
Curator - Not deleting indices Elasticsearch	8	1965	August 9, 2019
Curator is not deleting the index? Elasticsearch	2	2051	October 13, 2017
Curator and ELK 5.4 Elasticsearch	7	1542	June 11, 2017

Deleting indices with curator 4.0

Related topics