# TTL Value for Documents Under the Indices

**URL:** <https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194>\
**Category:** Elasticsearch\
**Created:** [October 17, 2023, 10:38am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194 "2023-10-17T10:38:33Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [October 17, 2023, 10:38am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/1 "2023-10-17T10:38:33Z")

</div>

Hi Team,

We had one requirement to set the TTL value for the documents present in a index.  
Could you please help me how to achieve the same.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 17, 2023, 11:06am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/2 "2023-10-17T11:06:17Z")

</div>

You can add a field to your documents like ttl:

```auto
{
  "ttl": 3600
}

```

And at index time compute the date of removal with an [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) using a [script processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html).

This one could extract the `ttl` value and generate a Date field from it like (and remove the `ttl` field):

```auto
{
  "date_for_removal": "2023-10-17T18:10:30Z"
}

```

Then, you need to run a Delete By Query job every x minutes (depending on your use case), let say every hour:

```auto
POST /my-index-000001/_delete_by_query
{
  "query": {
    "range": {
      "date_for_removal": {
        "lte": "now"
      }
    }
  }
}

```

That should work I think (not tested).

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [October 18, 2023, 7:16am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/3 "2023-10-18T07:16:18Z")

</div>

Thanks @dadoonet for your response. I had a requirement like below.

1. We had created a index and the data is getting populated to this index via csv files(Filebeat). So how we can enforce ttl values to those documents present in index.

2. The csv record contains a timestamp field (example:-18-10-2023) and if exactly I want to delete that particular record after 90 days from the value mentioned in timestamp field. How we can achieve the same.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 18, 2023, 7:26am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/4 "2023-10-18T07:26:26Z")

</div>

There is no support for TTL natively within Elasticsearch, so you need to follow David's recommendation and add the removal date and periodically trigger a delete by query request from outside of Elasticsearch, e.g. a script triggered by cron.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 18, 2023, 7:38am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/5 "2023-10-18T07:38:40Z")

</div>

> [@Debasis\_Mallick](#):
>
> We had created a index and the data is getting populated to this index via csv files(Filebeat). So how we can enforce ttl values to those documents present in index.

You need to run an update by query to set the `date_for_removal` field I suggested earlier on.

> [@Debasis\_Mallick](#):
>
> The csv record contains a timestamp field (example:-18-10-2023) and if exactly I want to delete that particular record after 90 days from the value mentioned in timestamp field. How we can achieve the same.

Follow the instructions I already shared.

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [October 18, 2023, 11:01am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/6 "2023-10-18T11:01:34Z")

</div>

Thank @Christian_Dahlqvist . Is time based indices are different from these TTL. Can you please help me to understand more on time based indices and the use cases of same.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 18, 2023, 11:17am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/7 "2023-10-18T11:17:47Z")

</div>

More or less.

With time based indices, you will regularly remove an entire index.  
But you can use that for ttl-ing documents.

ie. If you know that a document needs to be removed in december, you could send it to an index named `index-2023-11`. And when in Decembre, drop the index named `index-2023-11`.

---

<div class="post-metadata">

**Author:** ![pces](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pces/32/126636_2.png) [@pces](https://discuss.elastic.co/u/pces)\
**Post date:** [October 18, 2023, 11:42am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/8 "2023-10-18T11:42:26Z")

</div>

I thought ILM can help with auto-deletion/cleanup of data beyond retention. Is that not the case?

Thanks  
an elasticsearch novice

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 18, 2023, 11:54am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/9 "2023-10-18T11:54:09Z")

</div>

It's the case but per index. Not a document level.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2023, 11:54am UTC](https://discuss.elastic.co/t/ttl-value-for-documents-under-the-indices/345194/10 "2023-11-15T11:54:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.