Purge elasticsearch data older than 1month

Elastic Stack Elasticsearch
1

Hi,

I am running ELK on Red Hat Enterprise Linux release 8.9 (Ootpa)

elasticsearch-8.13.1-1.x86_64
logstash-8.13.1-1.x86_64
kibana-8.13.1-1.x86_64
filebeat-8.13.1-1.x86_64

Is there a way to purge elasticsearch data older than 1month? Any REST API call or something similar.....?

Please guide me. Thanks in advance

Best Regards,

Kaushal

2

If you are using time-based indices (indices with timestamp in name, rollover or data streams) you can use index lifecycle management to delete indices that exceed a certain age. Deleting complete indices is the by far most efficient way to manage retention in Elasticsearch.

If you are not using time-based indices you need to use the delete by query API to delete data from indices. Note that this is much more expensive than deleting indices and you have to trigger these APIs yourself.

3

@Christian_Dahlqvist Thanks a lot for a detailed response. Also Is there a way to archive/backup the elasticsearch data older than 1month and move it to a centralised storage location instead of purging or deleting the data which is a specific compliance requirement as per the customer.

Please guide me. Thanks in advance

Best Regards,

Kaushal

4

Hi @kaushalshriyan

Perhaps take. Look at the documents for Snapshot and Restore this is the only / official method for backing up Elasticsearch data.

5

Thanks a lot @Christian_Dahlqvist and @stephenb for the detailed response. Much appreciated. I will go through this and try it out on my sandbox environment. Thanks once again.