How do I know if curator is cleaning up? (SOLVED)

Elastic Stack Elasticsearch
1

Hi,
I was wondering if someone could clear up a question I have with curator.

So I installed Curator

sudo apt-get install python-pip

then

pip install elasticsearch-curator

then

crontab -e

then at the end of the line added

20 0 * * * /usr/local/bin/curator --host 127.0.0.1 -d 2 -c 1

What I wanted to see if it cleans my logs to leave disk space or am I understanding the wrong concept of how curator works? Because When I go to Kibana I can still look for logs older then 5 days I thought it would of deleted it.

Thank you

2

Have you tried running it by hand? Because I'd expect Curator to complain about the command line syntax. The command should resemble curator delete indices --older-than 5 --time-unit days.

To keep track of what Curator actually does, use the --logfile option to have it log everything to a file.

3

Hi Magnusbaeck Thank you for your reply.

Well I followed this guide

What do you mean running it by hand? Do you mean something like checking if the code is working? This is how I would check if Logstash is working
Ex:

tail -f/var/log/logstash/logstash.stdout

Thank you

4

Well I followed this guide

http://www.ragingcomputer.com/2014/02/removing-old-records-for-logstash-elasticsearch-kibana

That guide is obsolete and won't work with Curator 3.0 and later.

What do you mean running it by hand?

I mean running the command in a terminal and making sure it works as expected before sticking it in a cron job. And again, make sure you use --logfile to get a log from the execution. I enable debug-level logging, enable the Logstash log format, and push the logs into Elasticsearch.

5

Hi thank you again for your reply I got it to work. This line deletes 5 days of logs perfect

curator delete indices --older-than 5 --time-unit days --timestring %Y.%m.%d

Now That I know its working and want to implement on the cron job would I add it underneath like after typing

crobtab -e

then type

curator --host localhost delete indices --older-than 5 --time-unit days --timestring %Y.%m.%d

But I think theres an error on the syntax for the cron job.

See pictures

Thank you

Clipboarder.2015.09.09-011.png956×300 10.4 KB

6

A line in a user's crontab file always starts with a five-column specification of when to run the job. Study the article you used from the beginning and what you initially had in your crontab.

7

Hi there Magnus Thanks for the reply So I think I have figured it out on my crontab -e correct me if im wrong

* * * * * /usr/local/bin/curator delete indices --older-than 1 --time-unit days --timestring '\%Y.\%m.\%d'

which this indicates to run every minute to delete any logs older then one day
I ran this command to see if cron was working which it was deleted all my logs besides today.

Next Now That i know it works I run

20 0 * * * /usr/local/bin/curator delete indices --older-than 7 --time-unit days --timestring '\%Y.\%m.\%d'

This line indicates to run curator at 20 minutes past midnight and delete any logs older then 7 days
So if I have logged day 1,2,3,4,5,6,7 and when day 8 logs curator should delete the day 1 log at 20min past midnight?
So the next day it should be logs 2,3,4,5,6,7,8?

Thank you

1 Like
8

Yes, that's right.

9

Thank you again Magnus I will be sure to add this on my guide to help others

:smile:

10

hey while running this command (curator delete indices --older-than 5 --time-unit days) am getting the following error (Error: no such option: --older-than)
what to do?

11

@prasanna_kumar, please start a new thread for your unrelated question.

12

Should you add --host to specify which host to run curator?

13

no need to add the host but whats odd.. that it wont work anymore this command

curator delete indices --older-than 5 --time-unit days --timestring %Y.%m.%d

which shows (Error: no such option: --older-than)

which is odd haven't touch curator for over months

14

@killmasta93 You've probably triggered an upgrade. Curator 3 was all command-line, while Curator 4 uses YAML configuration files. See https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html for the current documentation on Curator.

15

o no...i dont even know when it updated by itself :frowning: its running 4.0.0 is there a way to downgrade it?

:frowning:

16

It's not possible to upgrade from 3 to 4 by accident if you installed via YUM or APT as the repositories are different. I presume it may have happened via a pip upgrade statement.

You can downgrade with pip install -U elasticsearch-curator==3.5.1

1 Like
17

Thank you soo much :slight_smile: im not much for updates, still on the old kibana