Data Retention

Elastic Stack Elasticsearch
1

Hi,

How can I assign the data retention in elasticsearch for example 3 months.

Regards

2

You can't but you can use curator to clean your indices on a regular basis.

3

Elasticsearch does not handle data retention on its own. You have to do it yourself, using a tool like Elasticsearch Curator or manually use the API.

4

I wrote this script to manage my ELK indices. It's self-explanatory. It does assume that you create daily indices in the form of "logstash-*-YYYY.mm.dd"

#######################################################################
# Script to manage ELK indices
# - Close week-old ELK indicies
# - Delete 90-day-old ELK indicies
# To run daily in crontab, say at 1:00am
# 0 1 * * * /root/elk/close_last_week_to_date_elk_indices.sh
#######################################################################
CERT="/etc/elasticsearch/chain-ca.pem"
USER=elk_user
PW=elk_pw

TODAY=`date +%Y.%m.%d`
URL="https://elk-url:9200"

close_week_old_indices()
{
  WEEK_FROM_TODAY=`date +%Y.%m.%d -d -7days`
  INDEX=logstash-*-$WEEK_FROM_TODAY
  res=`curl -XGET $URL/$INDEX --cacert $CERT -u $USER:$PW`
  if [[ $res != "{}" ]]; then
    # Close indices a week old
    curl -XPOST $URL/$INDEX/_close --cacert $CERT -u $USER:$PW
  fi
}

delete_90day_old_indices()
{
  DAYS90_FROM_TODAY=`date +%Y.%m.%d -d -90days`
  INDEX=logstash-*-$DAYS90_FROM_TODAY
  res=`curl -XGET $URL/$INDEX --cacert $CERT -u $USER:$PW`
  if [[ $res != "{}" ]]; then
    # Delete indices 90 days old
    curl -XDELETE $URL/$INDEX --cacert $CERT -u $USER:$PW
  fi
}

# main()
close_week_old_indices
delete_90day_old_indices
5 Likes
5

@ZillaG Was Curator too hard?

6

It's featurr-rich and I didn't need all those features.

7

Thank You

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.