# Best way to do log retention

**URL:** <https://discuss.elastic.co/t/best-way-to-do-log-retention/236190>\
**Category:** Elastic Stack\
**Created:** [June 8, 2020, 2:39pm UTC](https://discuss.elastic.co/t/best-way-to-do-log-retention/236190 "2020-06-08T14:39:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![arixmf](https://avatars.discourse-cdn.com/v4/letter/a/ea5d25/32.png) [@arixmf](https://discuss.elastic.co/u/arixmf)\
**Post date:** [June 8, 2020, 2:39pm UTC](https://discuss.elastic.co/t/best-way-to-do-log-retention/236190/1 "2020-06-08T14:39:49Z")

</div>

What would be the best way to do log retention in Open Source Elasticsearch?  
I'm using Elasticsearch 7.6 and I was reading about lifecycle management, but as far as I understand, it's part of X-Pack.

What is the best way to delete indices periodically using only the open source components? Is curator still a viable option?

I'm running Elasticsearch on Docker Swarm. Should I run curator inside a container as a cron job?

Thank you

---

<div class="post-metadata">

**Author:** ![tamilsweet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tamilsweet/32/54127_2.png) [@tamilsweet](https://discuss.elastic.co/u/tamilsweet)\
**Post date:** [June 8, 2020, 3:14pm UTC](https://discuss.elastic.co/t/best-way-to-do-log-retention/236190/2 "2020-06-08T15:14:56Z")

</div>

Hi @arixmf, Welcome to Elastic community.

I use the following method in Elasticsearch 7.0.1 so it should work for you as well.

> **Create ILM policy**
>
> ```
> curl -X PUT "http://localhost:9200/_ilm/policy/general_rollover" -H 'Content-Type: application/json' -d'
> {
> "policy": {
> "phases": {
> "hot": {
> "actions": {
> "rollover": {
> "max_age": "7d",
> "max_size": "50gb"
> }
> }
> },
> "warm": {
> "min_age": "7d",
> "actions": {
> "allocate": {
> "number_of_replicas": 0,
> "include": {},
> "exclude": {}
> }
> }
> },
> "delete": {
> "min_age": "21d",
> "actions": {
> "delete": {}
> }
> }
> }
> }
> }
> '
> 
> ```

> **Create Index with Alias**
>
> ```
> curl -X PUT "http://localhost:9200/%3Cyourindex-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
> {
> "aliases": {
> "yourindex": {
> "is_write_index": true
> }
> }
> }
> '
> 
> ```

> **Add template for your index**
>
> ```
> curl -X PUT "http://localhost:9200/_template/yourindex_template" -H 'Content-Type: application/json' -d'
> {
> "index_patterns" : ["yourindex*"],
> "settings" : {
> "number_of_shards" : 5,
> "number_of_replicas" : "1",
> "index.lifecycle.name": "general_rollover",
> "index.lifecycle.rollover_alias": "yourindex"
> }
> }
> '
> 
> ```

This adds indices of the pattern  
`yourindex-2020.06.08-000001`  
`yourindex-2020.06.15-000002`  
`yourindex-2020.06.22-000003`  
...  
New index is created as per ilm rollover policy (eg. 7days or 50GB in above example).  
After 21 days the index will be deleted automatically.

---

<div class="post-metadata">

**Author:** ![arixmf](https://avatars.discourse-cdn.com/v4/letter/a/ea5d25/32.png) [@arixmf](https://discuss.elastic.co/u/arixmf)\
**Post date:** [June 8, 2020, 4:09pm UTC](https://discuss.elastic.co/t/best-way-to-do-log-retention/236190/3 "2020-06-08T16:09:30Z")

</div>

@tamilsweet Thank you, but after a while, my index starts having a lifecycle error  
it says

```auto
illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-app_logs] does not point to index [filebeat-app_logs-myapp_1.2.0.0-week-2020.24]

```

Since my indices contain the application version, I tried to create the alias like that

```auto
    curl -X PUT "http://localhost:9200/%3Cfilebeat-app_logs-*-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
    {
      "aliases": {
        "filebeat-app_logs": {
          "is_write_index": true
        }
      }
    }
    '

```

but it complained that I can't use characters like `*`

---

<div class="post-metadata">

**Author:** ![tamilsweet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tamilsweet/32/54127_2.png) [@tamilsweet](https://discuss.elastic.co/u/tamilsweet)\
**Post date:** [June 9, 2020, 4:01am UTC](https://discuss.elastic.co/t/best-way-to-do-log-retention/236190/4 "2020-06-09T04:01:45Z")

</div>

@arixmf The idea in this method is that, we use index alias to write/read the documents. Elasticsearch will handle where to write the index.  
In your case, you should use the alias name to write/read the indices. Say `filebeat-app_logs`  
Elasticsearch will handle the creation of new index and rotation based on your ILM policy.  
This way you don't need manual intervention or external handler like curator.

Please refer below doc pages.

[ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)

[Bootstrap the initial time-series index](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-bootstrap)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 8:01am UTC](https://discuss.elastic.co/t/best-way-to-do-log-retention/236190/5 "2022-11-04T08:01:36Z")

</div>