Automatically Delete older Documents

Elastic Stack Elasticsearch
1

Hi Team,

I need your help to understand how to set TTL like mechanism to delete the Elastic Search documentation automatically once it's reached the expiration time.

I have read about delete by query API but the problem here is we have to execute it manually.

2

There is no automatic way to set and enforce TTL in Elasticsearch so you will need to schedule it yourself.

3

@Christian_Dahlqvist

Is it possible to use cron schedule to execute delete by query API, if possible please give me a reference.

4

Yes, that should be possible although I do not have any reference.

1 Like
5

Are you refering to purge elasticsearch documents after a period of time ?
I guess ILM is for this purpose

6

@ylasri
ILM is used to delete entire index, but for case i need to delete some documents only not all or index.

7

Set beats to create new index everyday, that way if you set ILM only old indexes will be deleted. You could also set a cronjob, I run a daily cronjob to delete some very specific data.

2 Likes
8

ILM will manage the entire lifecyle of the index, it will rollover the old data to a new index and if a condition is meet it will delete it

9

@ylasri
it is possible to delete a specific data alone in the index with ILM

for example, i have 100 documents attached with index name called test. This holds 30 older documents(last month docs). is it possible to delete only that 30 documents

10

In that specific case, you need a script to do it. Try this:
(replace USERNAME, PASSWORD and test accordingly. mind the https, don't know if you use http or https. if you don't have username/password remove USERNAME:PASSWORD@) Credit to @ Jenni for the script, she helped me when I had a similar issue.

curl -k -X POST "https://USERNAME:PASSWORD@localhost:9200/test/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "lt": "now-30d"
            }
          }
        }
      ]
    }
  }
}

Althought I recommend setting up your beats to create new index every day so if you set up ILM, only old data will be removed.

1 Like
11

Thanks @headtea.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.