Best way to do log retention

Elastic Stack
1

What would be the best way to do log retention in Open Source Elasticsearch?
I'm using Elasticsearch 7.6 and I was reading about lifecycle management, but as far as I understand, it's part of X-Pack.

What is the best way to delete indices periodically using only the open source components? Is curator still a viable option?

I'm running Elasticsearch on Docker Swarm. Should I run curator inside a container as a cron job?

Thank you

2

Hi @arixmf, Welcome to Elastic community.

I use the following method in Elasticsearch 7.0.1 so it should work for you as well.

Create ILM policy 
curl -X PUT "http://localhost:9200/_ilm/policy/general_rollover" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "number_of_replicas": 0,
            "include": {},
            "exclude": {}
          }
        }
      },
      "delete": {
        "min_age": "21d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
'
Create Index with Alias 
curl -X PUT "http://localhost:9200/%3Cyourindex-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "yourindex": {
      "is_write_index": true
    }
  }
}
'
Add template for your index 
curl -X PUT "http://localhost:9200/_template/yourindex_template" -H 'Content-Type: application/json' -d'
{
    "index_patterns" : ["yourindex*"],
    "settings" : {
        "number_of_shards" : 5,
        "number_of_replicas" : "1",
        "index.lifecycle.name": "general_rollover",
        "index.lifecycle.rollover_alias": "yourindex"
    }
}
'

This adds indices of the pattern
yourindex-2020.06.08-000001
yourindex-2020.06.15-000002
yourindex-2020.06.22-000003
...
New index is created as per ilm rollover policy (eg. 7days or 50GB in above example).
After 21 days the index will be deleted automatically.

3

@tamilsweet Thank you, but after a while, my index starts having a lifecycle error
it says

illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-app_logs] does not point to index [filebeat-app_logs-myapp_1.2.0.0-week-2020.24]

Since my indices contain the application version, I tried to create the alias like that

    curl -X PUT "http://localhost:9200/%3Cfilebeat-app_logs-*-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
    {
      "aliases": {
        "filebeat-app_logs": {
          "is_write_index": true
        }
      }
    }
    '

but it complained that I can't use characters like *

4

@arixmf The idea in this method is that, we use index alias to write/read the documents. Elasticsearch will handle where to write the index.
In your case, you should use the alias name to write/read the indices. Say filebeat-app_logs
Elasticsearch will handle the creation of new index and rotation based on your ILM policy.
This way you don't need manual intervention or external handler like curator.

Please refer below doc pages.

ILM

Bootstrap the initial time-series index