Automatically Delete older Documents

Elastic Stack Elasticsearch
#1

Hi Team,

I need your help to understand how to set TTL like mechanism to delete the Elastic Search documentation automatically once it's reached the expiration time.

I have read about delete by query API but the problem here is we have to execute it manually.

#2

There is no automatic way to set and enforce TTL in Elasticsearch so you will need to schedule it yourself.

#3

@Christian_Dahlqvist

Is it possible to use cron schedule to execute delete by query API, if possible please give me a reference.

#4

Yes, that should be possible although I do not have any reference.

1 Like
#5

Are you refering to purge elasticsearch documents after a period of time ?
I guess ILM is for this purpose

#6

@ylasri
ILM is used to delete entire index, but for case i need to delete some documents only not all or index.

#7

Set beats to create new index everyday, that way if you set ILM only old indexes will be deleted. You could also set a cronjob, I run a daily cronjob to delete some very specific data.

2 Likes
#8

ILM will manage the entire lifecyle of the index, it will rollover the old data to a new index and if a condition is meet it will delete it

#9

@ylasri
it is possible to delete a specific data alone in the index with ILM

for example, i have 100 documents attached with index name called test. This holds 30 older documents(last month docs). is it possible to delete only that 30 documents

#10

In that specific case, you need a script to do it. Try this:
(replace USERNAME, PASSWORD and test accordingly. mind the https, don't know if you use http or https. if you don't have username/password remove USERNAME:PASSWORD@) Credit to @ Jenni for the script, she helped me when I had a similar issue.

curl -k -X POST "https://USERNAME:PASSWORD@localhost:9200/test/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "lt": "now-30d"
            }
          }
        }
      ]
    }
  }
}

Althought I recommend setting up your beats to create new index every day so if you set up ILM, only old data will be removed.

1 Like
#11

Thanks @headtea.

1 Like