Data Retention

[saad]

Hi,

How can I assign the data retention in elasticsearch for example 3 months.

Regards

[dadoonet (David Pilato)]

You can't but you can use curator to clean your indices on a regular basis.

[theuntergeek (Aaron Mildenstein)]

Elasticsearch does not handle data retention on its own. You have to do it yourself, using a tool like Elasticsearch Curator or manually use the API.

[ZillaG]

I wrote this script to manage my ELK indices. It's self-explanatory. It does assume that you create daily indices in the form of "logstash-*-YYYY.mm.dd"

#######################################################################
# Script to manage ELK indices
# - Close week-old ELK indicies
# - Delete 90-day-old ELK indicies
# To run daily in crontab, say at 1:00am
# 0 1 * * * /root/elk/close_last_week_to_date_elk_indices.sh
#######################################################################
CERT="/etc/elasticsearch/chain-ca.pem"
USER=elk_user
PW=elk_pw

TODAY=`date +%Y.%m.%d`
URL="https://elk-url:9200"

close_week_old_indices()
{
  WEEK_FROM_TODAY=`date +%Y.%m.%d -d -7days`
  INDEX=logstash-*-$WEEK_FROM_TODAY
  res=`curl -XGET $URL/$INDEX --cacert $CERT -u $USER:$PW`
  if [[ $res != "{}" ]]; then
    # Close indices a week old
    curl -XPOST $URL/$INDEX/_close --cacert $CERT -u $USER:$PW
  fi
}

delete_90day_old_indices()
{
  DAYS90_FROM_TODAY=`date +%Y.%m.%d -d -90days`
  INDEX=logstash-*-$DAYS90_FROM_TODAY
  res=`curl -XGET $URL/$INDEX --cacert $CERT -u $USER:$PW`
  if [[ $res != "{}" ]]; then
    # Delete indices 90 days old
    curl -XDELETE $URL/$INDEX --cacert $CERT -u $USER:$PW
  fi
}

# main()
close_week_old_indices
delete_90day_old_indices

[theuntergeek (Aaron Mildenstein)]

@ZillaG Was Curator too hard?

[ZillaG]

It's featurr-rich and I didn't need all those features.

[saad]

Thank You