# Automatically Delete older Documents

**URL:** <https://discuss.elastic.co/t/automatically-delete-older-documents/247078>\
**Category:** Elasticsearch\
**Created:** [September 1, 2020, 11:15am UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078 "2020-09-01T11:15:19Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arun\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_n/32/46709_2.png) [@Arun\_N](https://discuss.elastic.co/u/Arun_N)\
**Post date:** [September 1, 2020, 11:15am UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/1 "2020-09-01T11:15:19Z")

</div>

Hi Team,

I need your help to understand how to set TTL like mechanism to delete the Elastic Search documentation automatically once it's reached the expiration time.

I have read about delete by query API but the problem here is we have to execute it manually.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 1, 2020, 11:46am UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/2 "2020-09-01T11:46:50Z")

</div>

There is no automatic way to set and enforce TTL in Elasticsearch so you will need to schedule it yourself.

---

<div class="post-metadata">

**Author:** ![Arun\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_n/32/46709_2.png) [@Arun\_N](https://discuss.elastic.co/u/Arun_N)\
**Post date:** [September 1, 2020, 12:11pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/3 "2020-09-01T12:11:11Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> There is no automatic way to set and enforce TTL in Elasticsearch so you will need to schedule it yourself.

`@Christian_Dahlqvist `

Is it possible to use [cron schedule](https://www.elastic.co/guide/en/elasticsearch/reference/current/trigger-schedule.html#schedule-cron) to execute delete by query API, if possible please give me a reference.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 1, 2020, 12:16pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/4 "2020-09-01T12:16:32Z")

</div>

Yes, that should be possible although I do not have any reference.

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [September 1, 2020, 12:44pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/5 "2020-09-01T12:44:00Z")

</div>

Are you refering to purge elasticsearch documents after a period of time ?  
I guess [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html) is for this purpose

---

<div class="post-metadata">

**Author:** ![Arun\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_n/32/46709_2.png) [@Arun\_N](https://discuss.elastic.co/u/Arun_N)\
**Post date:** [September 1, 2020, 12:46pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/6 "2020-09-01T12:46:46Z")

</div>

> [@ylasri](#):
>
> Are you refering to purge elasticsearch documents after a period of time ?  
> I guess [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html) is for this purpose

`@ylasri`  
ILM is used to delete entire index, but for case i need to delete some documents only not all or index.

---

<div class="post-metadata">

**Author:** ![headtea](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/headtea/32/24271_2.png) [@headtea](https://discuss.elastic.co/u/headtea)\
**Post date:** [September 1, 2020, 12:48pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/7 "2020-09-01T12:48:19Z")

</div>

Set beats to create new index everyday, that way if you set ILM only old indexes will be deleted. You could also set a cronjob, I run a daily cronjob to delete some very specific data.

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [September 1, 2020, 12:48pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/8 "2020-09-01T12:48:49Z")

</div>

ILM will manage the entire lifecyle of the index, it will rollover the old data to a new index and if a condition is meet it will delete it

---

<div class="post-metadata">

**Author:** ![Arun\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_n/32/46709_2.png) [@Arun\_N](https://discuss.elastic.co/u/Arun_N)\
**Post date:** [September 1, 2020, 12:54pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/9 "2020-09-01T12:54:15Z")

</div>

> [@ylasri](#):
>
> ILM will manage the entire lifecyle of the index, it will rollover the old data to a new index and if a condition is meet it will delete it

@ylasri  
it is possible to delete a specific data alone in the index with ILM

for example, i have 100 documents attached with index name called `test`. This holds 30 older documents(last month docs). is it possible to delete only that 30 documents

---

<div class="post-metadata">

**Author:** ![headtea](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/headtea/32/24271_2.png) [@headtea](https://discuss.elastic.co/u/headtea)\
**Post date:** [September 1, 2020, 1:09pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/10 "2020-09-01T13:09:56Z")

</div>

In that specific case, you need a script to do it. Try this:  
(replace USERNAME, PASSWORD and test accordingly. mind the `https`, don't know if you use http or https. if you don't have username/password remove `USERNAME:PASSWORD@`) Credit to @ [Jenni](https://discuss.elastic.co/u/Jenni) for the script, she helped me when I had a similar issue.

```
curl -k -X POST "https://USERNAME:PASSWORD@localhost:9200/test/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "lt": "now-30d"
            }
          }
        }
      ]
    }
  }
}

```

Althought I recommend setting up your beats to create new index every day so if you set up ILM, only old data will be removed.

---

<div class="post-metadata">

**Author:** ![Arun\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_n/32/46709_2.png) [@Arun\_N](https://discuss.elastic.co/u/Arun_N)\
**Post date:** [September 1, 2020, 1:16pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/11 "2020-09-01T13:16:54Z")

</div>

Thanks @headtea.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 29, 2020, 1:17pm UTC](https://discuss.elastic.co/t/automatically-delete-older-documents/247078/12 "2020-09-29T13:17:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.