Curator version headaches

Elastic Stack Elasticsearch
1

Hi forum!

Don't know whether this is the proper place to ask about curator... but I ended dealing with it since I found it widely spread among ELK users, forums, and howtos.

I installed it on a Debian server following a pip howto... it worked and got the thing ready, but found that cli syntax is completely alien to most of the forum replies and howtos.
After a while doing try / error attempts I managed to use it to delete and close indices based on time.

Neverthless, I have miserably failed on assembling a delete by space (I read it was possible, and in fact easy!) command.... Does someone know how it works with 3.0.3 version.

Also, I wonder whether such a tool allows to some degree of granularity on deletion, so, for instance, I could ask it to delete indices older than say 7 days having the "blah" tag... Is it possible? that would be amazing!

Is 3.0.3 very outdated? syntax is completely different than what I found googling around

Hope someone could put some light here...
Thank you all in advance.

Best regards!

2

...mmmm... seems it just deletes indices by text regexp against name.

I'm almost sure its not possible... I think that the elegant way to do what I want is to query elasticsearch with curl and with XDELETE option so the deleted indices will be the ones precissely matching a given query.

But maybe someone have a better idea!

Anyhow... thank you all!

3

Is 3.0.3 very outdated? syntax is completely different than what I found googling around

On the contrary, 3.0.3 is the currently most recent release. As noted in the release notes for 3.0.0, it's a major update with a new command line interface. That should explain why many examples out there won't work any more.

I'm almost sure its not possible... I think that the elegant way to do what I want is to query elasticsearch with curl and with XDELETE option so the deleted indices will be the ones precissely matching a given query.

Queries match documents and not indexes so I'm not sure exactly what you want to accomplish. Do you want to delete all indexes that contain at least one document that matches a particular query?

4

As Magnus pointed out, it's a new update. All of the Curator documentation is now on the Elastic main documentation page: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

5

Aha... good to know... so I'm steping on the bleeding edge...hope that, with time, developers and gurus make this version reach the past level of howtos, examples, guides and so that is currently around
In the meanwhile, hard time to newbies hehehe :smile:

Actually, my target was (IS), to be able to delete data from elasticsearch in a controled way.
The need comes by the fact that I have stated that there can be huge differences in data size depending on the service:
-IDS/IPS services such as snort or suricata generates me tons of data.
-Firewall log data also generates me a noticeable amount of data.
-Some web servers / streaming servers, with moderate usage create, by comparisson, just a few data...

As of today, In order to keep things light, I have to delete data in less than a week... but by doing so, I loose ALL data, since I have only find out how to use CURATOR to delete all indices older than some time.
keepling a few days of data regarding IDS/IPS/firewalling activity is good enought to me...
...But certaintly I would like to keep a whole year of web server or streaming server data archived!!!!

Since there exist distinctive fields / tags that perfectly allow to log type discrimination, my idea is to selectively delete all indices older than a few days JUST FROM IDS/IPS/Firewall activity data. while data related to web serving or stream serving could be kept longer periods of time.

So, yes... there should be some querying involved here, since I would be targeting specific tags/fields values, and yes, the ultimate reason to do this is to delete those matching log entries.
I'm not sure curator was developed with this tasks in mind...

Thank you guys!

6

Well... being reading around I have found a website were all this is discussed.

...It turns out I'm not doing things the right way:
By following most available howtos, guides and so around you end up storing everyhing on a singe, default, logstash-yyymmdd series indices ... so, I deduced, all discrimination depends on tag/type/field examination stuff.

It turns out we can configure multiple elasticsearch outputs to tha same localhost instance, but to different custom named indices!!!
by configuring output with correct usage of conditionals (here yes!, we use tags!) we can organize our data in indices depending on type or wathever:

output {
if [type] in ["access_logs", "sshd_logs", "application_a", "application_b"] {
elasticsearch_http {
host => localhost
index => "logstash-keep5day-%{+YYYY-MM-dd}"
}
}
else {
# Default index
elasticsearch_http {
host => localhost
index => "logstash-keep1day-%{+YYYY-MM-dd}"
}
}
}

I have to test it!!!!!

Afterwards, I'm almost sure that using curator to delete some indices of some type while ignoring others will be easy.

Anyhow, tomorrow I will have to test out lot of things again! :sweat_smile:

EDIT:
Found 3.0.3 documentation.... the command is clear to me right now!
To anyone searching around... I have implemented the above approach and it all results far more organized. And I can assign different delete policies depending on index/service type.
ELK stack rocks!