The age of index which is using ilm policy management

[carpeDiem]

Hello,
I'm currently trying to configure and using ilm lifecycle management and data stream for my indices,we are metting a question,when the index was rollovered,the age of index will be reset,this got us into trouble.For example,the value of warm phrase's min_age is seven days,then when the index reaches to storage limit and has being rollovered,we will wait seven days again for it that enters to warm phrase.If the index never reaches to the storage limit,then it will never enter to warm phrase.In this scene,we can't do anything for it,such as deleting the index or deleting the data stream of this index.

What is the reason for this reset design? Or are there any plan I can do for solving this problem? Or will it be changed in future?
Finally,will elasticsearch manage data stream in some day?

Looking forward to any help!

[warkolm (Mark Walkom)]

Can you share your policy please?

[carpeDiem]

My policy is configured like this.I set parametermax_docs:10,then it rollovers,and the age of the index will be reset.

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1M",
            "max_docs": 10
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "20m",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "number_of_replicas": 0,
            "require":{
              "box_type":"warm"
            }
          },
          "set_priority": {
            "priority": 50
          }

        }
      },
      "cold":{
        "min_age": "30m",
        "actions":{
          "searchable_snapshot":{
            "snapshot_repository":"s3-repo"
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {
            "delete_searchable_snapshot":false
          }
        }
      }
    }
  }
}```

[warkolm (Mark Walkom)]

Do you have the max_size and min_age so small for testing?

@carpeDiem

If the index never reaches to the storage limit,then it will never enter to warm phrase

The roll over will happen whenever the first of either max_size or min_age are reached.

[carpeDiem]

Yes,I have tried,this is my test flow.

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "8k",
            "max_docs": 10
          },
          "set_priority": {
            "priority": 100
          }

        }
      },
      "warm": {
        "min_age": "2m",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "number_of_replicas": 0,
            "require":{
              "box_type":"warm"
            }
          },
          "readonly":{},
          "set_priority": {
            "priority": 50
          }

        }
      },
      "cold":{
        "min_age": "2d",
        "actions":{
          "searchable_snapshot":{
            "snapshot_repository":"s3-repo"
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot":false
          }
        }
      }
    }
  }
}
PUT _index_template/test
{
  "index_patterns": [
    "test-*"
  ],
  "data_stream": {},
  "priority": 300,
  "template": {
    "settings": {
      "index.refresh_interval": "2s",
      "number_of_shards": "1",
      "number_of_replicas": "0",
      "index.lifecycle.name": "test",
      "index.lifecycle.rollover_alias": "myindex3"
    }
  }
}

POST test-2021.01.15/_doc
{
"@version":1,
"@timestamp":"2021-01-12T08:18:25.315Z",
"用户":"admin",
"host":"135.31.49.177",
"失败原因":"",
"时间":"2021-01-14 16:18:25",
"日志类型":"访问日志",
"IP地址":"10.4.33.98",
"级别":"信息"
}

then about 30 minutes after, this is result.

as you can see,the index doesn't enter to warm phrase because it hasn't rollovered,it's age is about thirty minutes.then I write 15 documents to make it rollover.See explain result again.

the age has been reset,it changed from 0.

[carpeDiem]

So what is the reason for the age resetting? How to calculate the age of index when it rollovered?I have no idea for it.

[dakrone (Lee Hinman)]

Ages are calculated based off of index creation time prior to a rollover, and the time the index was rolled over after the rollover has occurred. (In policies with no rollover the index creation time is used).

To explain why, here's an example. Imagine that you have a max_age of 30d and max_size of 50gb, in a rollover action followed immediately with a delete phase with a min_age of 7d.

That means that documents in the index can range between 0 seconds and 30 days old (assuming the size condition is not meant). So when the index is deleted the documents are between 7 days and 37 days old. This is the current behavior.

Now, imagine that we did not calculate the age based on the rollover time. Imagine that we indexed documents constantly over a 5 day period, after 5 days, the index reaches 50gb and rolls over. This means that documents inside of the index are between 0 and 5 days old. ILM then deletes the index 2 days later (because it has reached an age of 7 days). This means that we delete the index containing documents that are between 2 days and 7 days old (even though the index is older).

We prefer to err on the side of keeping indices slightly longer but ensuring that retention is always at least as long as the min_age of the delete phase.

If you want to avoid this, you can use the index.lifecycle.origination_date setting to specify what time all the ILM phase transitions should use for the age.

Hopefully this helps.

[carpeDiem]

Thank you so much, I have known the reason for it.
I have tried the setting:index.lifecycle.origination_date
It can work well that creating the index with a template setting,but it doesn't seem to support rollover_alias. I tried to make the template with index.lifecycle.rollover_alias,Then when it was rollovering,it gave me the following information.

And this does't seem to work with a data stream,but we need data streams.Do you have some ideas I can do that manage the index with data stream and using origination_date setting for the age?