Rollover logs indexed daily

Elastic Stack Elasticsearch
1

Hello every, I hope you all doing well during this time of confinement

So basically what I have is a series of logs indexed daily into Elasticsearch, my logs index pattern is
logs-YYYY.MM.DD
I'm trying to set up a rollover action to delete the index entirely after 30 days. From what I have read in the documentation, we simply create an index lifecycle policy with the delete phase enabled and create an index template which is using the index lifecycle policy above to apply that to all indexes matched the index pattern. My configuration for the index lifecycle policy and the index template are the following:

    PUT _ilm/policy/logspolicy
    {
      "policy": {
        "phases": {
          "hot": {
            "min_age": "0ms",
            "actions": {
              "rollover": {
                "max_age": "30d",
                "max_size": "50gb"
              },
              "set_priority": {
                "priority": 100
              }
            }
          },
          "delete": {
            "min_age": "1d",
            "actions": {
              "delete": {}
            }
          }
        }
      }
    }

    PUT _template/logs_template
    {
      "index_patterns": ["logs*"], 
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1,
        "index.lifecycle.name": "logspolicy", 
        "index.lifecycle.rollover_alias": ???
      }
    }

However, I got stuck at the configuration of index.lifecycle.rollover_alias. My case is rather different than the case in the documentation which all the events are indexed into one index logs rather than logs-YYYY.MM.DD. Hence, there is no way for me to define the lifecycle alias for indexes with the date.
Moreover, we also need to bootstrap the rollover process by creating the first index with the suffix -000001 and since my index is created daily and I don't think this will be a feasible option to do it manually every day.

Is there any way to define the index.lifecycle.rollover_alias dynamically based on the date in the index pattern? Thanks

2

You dont need a rollover alias for these kind of indexes. Just define an ilm policy with a delete phase based on creation date.

1 Like
3

I got this error instead. Is there any workaround for this ?

Index lifecycle error
illegal_argument_exception: setting [index.lifecycle.rollover_alias] for index [logs-2020.03.27] is empty or not defined
4

yes , you should only enable the DELETE phase in ILM. Dont enable the HOT phase .

Your logspolicy shows you have both HOT and DELETE phases enabled. Just disable HOT and you will be fine.

5

I tried to disable the hot phase and the rollover is not working anymore. The index ignore completely the ilm. I think it needs the hot phase to trigger the rollover action

6

You are not using rollover , if you have indices like this

logs-YYYY.MM.DD

This is just daily indices , which is fine. In which case you only need a DELETE phase to cleanup :wink:

7

Yes the thing is if I only set the DELETE phase without the HOT phase, the index will not register the ilm defined in the index template which contains the definition for the DELETE phase.
To be precise, if I enable the HOT phase and DELETE phase I have this in the index setting for logs-2020.03.27

 "settings": {
    "index": {
      "lifecycle": {
        "name": "logspolicy",
      }

If I disable the HOT phase and keep only the DELETE phase the index logs-2020.03.27 ignore the ilm from the index template and I don't have the lifecycle attribut anymore in the index setting. In the end, it acts like a normal index without any rollover policy.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.