Delete logs in ElasticSearch after certain period

[bob-bza]

@theuntergeek

No, it doesn't.

can i know the reason please!!!

[theuntergeek (Aaron Mildenstein)]

Curator always uses the same API calls, regardless of whether it's deployed locally or remotely. AWS ES does not respond with the proper data, so Curator simply won't work with it.

[bob-bza]

@theuntergeek

Curator always uses the same API calls, regardless of whether it's deployed locally or remotely. AWS ES does not respond with the proper data, so Curator simply won't work with it.

thanks .

By using "curl -X DELETE" how can i specify target age. can you please direct me to proper documentation/page .

[theuntergeek (Aaron Mildenstein)]

You will have to script it with some other tool: bash, or some other shell; or perhaps some other program someone has written.

[godber]

@bob-bza, the thing that wasn't explicitly said here is that the way people handle data "curation" like this relies on two things:

1. Logs should be placed in a "daily" index with names like "logs-2017-01-16", "logs-2017-01-17" through "logs-2017-02-17".
2. You somehow delete the indices that fall outside of your retention period. Using "curl" to delete a whole index is just fine. So, today you might delete "logs-2017-01-16".

If you put all of your logs into the same index, then you have to go through find individual entries by timestamp and delete them. This is a much harder task. See the note at the bottom of the page here:

https://www.elastic.co/guide/en/elasticsearch/reference/current/_deleting_documents.html

See the Delete By Query API to delete all documents matching a specific query. It is worth noting that it is much more efficient to delete a whole index instead of deleting all documents with the Delete By Query API."

But like @theuntergeek says, you don't want to delete by query.

• Austin

[bob-bza]

@godber

Logs should be placed in a "daily" index with names like "logs-2017-01-16", "logs-2017-01-17" through "logs-2017-02-17".
You somehow delete the indices that fall outside of your retention period. Using "curl" to delete a whole index is just fine. So, today you might delete "logs-2017-01-16".

i was writing around 30-40 types of different indexes to ES, so its hard for us to create indexes in kibana with - all the time. is there any other solution for indexs to view in kibana. then i can go with approch - that will be easy for us to delete logs

[bhatch (Brandon Hatch)]

I'm not quite sure I understand your last question. When it comes to Kibana it shouldn't really matter whether you use daily indices or one giant index.

For example lets say you have these daily indices:

eventlogs-2017.02.20
eventlogs-2017.02.19
eventlogs-2017.02.18
syslogs-2017.02.20
syslogs-2017.02.19
syslogs-2017.02.18

In kibana you setup an index pattern of:
eventlogs-*
syslogs-*.

The * is a wildcard. Kibana will automatically match every index that matches that pattern. So whether that pattern matches 1 index or 100 indexes, there is no extra overhead for you. Once you setup the initial pattern you never have to go back and add new patterns.