I have tons of logs that was writing to elasticsearch service . i was running out of space its keep writing logs . i was looking for something to delete logs after certain period of time. Ex: 15days or 20days or 1mnth automatically .
Is there any option or way available in elasticsearch.
I was using ELKB. Filebeat-Logstash-ElasticSearch-Kibana.
I hope there was something available but i was not sure maybe because of lack of knowledge.
2 Likes
Hi,
Are you talking about Elasticsearch log files or log indexed in Elasticsearch and searchable using Kibana?
was talking about log indexed in Elasticsearch. I was sending large amount of logs to ES , so want to delete logs(mydata) after 15 or 20 days.
Ok. You can have a look at Curator (https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/index.html).
Curator 5.0 is not yet released. Try https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
I just want to delete my old data what ever i send to Elasticsearch. I was using elasticsearch 5.1.
Is there any chance to delete old data ?????? Please direct me proper way to delete those automatically.
@bob-bza, Elasticsearch Curator is the way to automatically delete your older indices.
See the delete_indices action documentation, and the example for the same action. The rest of the documentation should help with understanding.
2 Likes
I was using Elasticsearch 5.1 in AWS does it support Elasticsearch Curator ???
If you are running your own installation of Elasticsearch in your own AWS EC2 instance, then Curator will work for you.
However...
If you are running AWS ES 5.1, then Curator will not work for you. Even though AWS added the /_cluster/state endpoint—which Curator depends on—to their release of AWS ES 5.1, it still doesn't have the necessary data to support Curator. See https://github.com/elastic/curator/issues/880 for more information.
I'm sorry for the inconvenience. You will have to write your own scripts to automate index deletion, or you may be able to find some online, somewhere.
If i use AWS ES service 2.3 , does it supports to delete old data??
If it yes means , in that case i will try to use AWS E.S Service 2.3.
Curator v4 does not work with any version yet released of AWS ES. Curator v3 should work for you with AWS ES 2.x, but lacks many advanced features found in Curator 4. Curator v3 is deprecated and no longer supported in any way.
So to be clear and straight, AWS ES using any version , i cant delete my old data??
If there is by anychance to delete old data means Please let me know.
You can always use curl -XDELETE http://localhost:9200/INDEXNAME to delete indices. You just won't be able to use Elasticsearch Curator to help automate the process.
If i run this this will delete all data and index. i want to run something in corn or in settings to delete logs automatically every 15days .
1 Like
No, that curl command will only delete the specified INDEXNAME.
You're on your own for that, as I stated. Curator is normally the go-to solution for that, but AWS made their version of ES incompatible with it. You will have to run the above command manually, or find some way to script it.
I already used this , it was delete whole log in that index.
curl -XDELETE 'http://search-mydomain/s3-logs'
this will delete everything in this index right, i always want to keep for 15days logs.
you said AWS was not supporting Curator , so can i use this
DELETE /index/type/_query
{
"query": {
"range": {
"@timestamp": {
"lt": "now-7d"
}
}
}
}
To delete my old logs in elasticsearch from logstash.
It's apparent you are not using time-series indices, if that is the case. You should not be feeding a constant stream of data to a single index unless you're planning on using the rollover API.
I already responded to your other request about the delete-by-query, which is a really bad approach to data management as it heavily taxes the cluster making millions of atomic flag-for-delete operations, which then have to be singled out for deletion at the next segment merge operation. In short, do not use this approach if you want your cluster to behave in a performant way.
I highly recommend looking into the Rollover API for a way to simplify this for you. Then you can make your "non-time-series" index into a time-series index for all intents and purposes.
And ask for help with the Rollover API in a new topic, or search for an existing one, as this is off-topic here.
can i install Curator on my cluster instance and from there can do this
in that case does it works on ElasticSearchService in AWS.
No, it doesn't.