TTL Value for Documents Under the Indices

Elastic Stack Elasticsearch
1

Hi Team,

We had one requirement to set the TTL value for the documents present in a index.
Could you please help me how to achieve the same.

Thanks,
Debasis

2

You can add a field to your documents like ttl:

{
  "ttl": 3600
}

And at index time compute the date of removal with an ingest pipeline using a script processor.

This one could extract the ttl value and generate a Date field from it like (and remove the ttl field):

{
  "date_for_removal": "2023-10-17T18:10:30Z"
}

Then, you need to run a Delete By Query job every x minutes (depending on your use case), let say every hour:

POST /my-index-000001/_delete_by_query
{
  "query": {
    "range": {
      "date_for_removal": {
        "lte": "now"
      }
    }
  }
}

That should work I think (not tested).

1 Like
3

Thanks @dadoonet for your response. I had a requirement like below.

  1. We had created a index and the data is getting populated to this index via csv files(Filebeat). So how we can enforce ttl values to those documents present in index.

  2. The csv record contains a timestamp field (example:-18-10-2023) and if exactly I want to delete that particular record after 90 days from the value mentioned in timestamp field. How we can achieve the same.

Thanks,
Debasis

4

There is no support for TTL natively within Elasticsearch, so you need to follow David's recommendation and add the removal date and periodically trigger a delete by query request from outside of Elasticsearch, e.g. a script triggered by cron.

5

You need to run an update by query to set the date_for_removal field I suggested earlier on.

Follow the instructions I already shared.

6

Thank @Christian_Dahlqvist . Is time based indices are different from these TTL. Can you please help me to understand more on time based indices and the use cases of same.

Thanks,
Debasis

7

More or less.

With time based indices, you will regularly remove an entire index.
But you can use that for ttl-ing documents.

ie. If you know that a document needs to be removed in december, you could send it to an index named index-2023-11. And when in Decembre, drop the index named index-2023-11.

8

I thought ILM can help with auto-deletion/cleanup of data beyond retention. Is that not the case?

Thanks
an elasticsearch novice

9

It's the case but per index. Not a document level.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.