Is it possible change creation_date on reindexed Indexes to get Life cycle Management to work on old indexes

Elastic Stack Elasticsearch
1

Thanks in advance for any help.

I just created and new setup of 6.7.2 ES / logstash / kibana.
I used the reindex option to import the old data from the old 5.X cluster.
Problem is I would like to start using the ILM option to age out this old data (using 180 Day retention for example).

It seems however that the reindex process created a "creation_date" on the date of the import / reindex. It seems that this is what is used to set the "lifecycle_date_millis" to determine when actions can be taken. It also does not seem possible for me to modify the "creation_date" on any given index. If this is all true, is there any option to be able to use ILM on these reindexed indexes.

Thanks for any help!!

example of the index settings as ILM .

{
  "logstash-2018.11.14" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "delete_180_days"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "5",
        "provided_name" : "logstash-2018.11.14",
        "creation_date" : "1558472624125",
        "number_of_replicas" : "1",
        "uuid" : "hl7YIjWBSgem2B_N2y3TXw",
        "version" : {
          "created" : "6070299",
          "upgraded" : "6080099"
        }
      }
    }
  }
}

Note file date: logstash-2018.11.14
vs.
"creation_date" : "1558472624125" = 5/21/2019, 5:03:44 PM
(the time of the reindex)

ILM policy explain

{
  "indices" : {
    "logstash-2018.11.14" : {
      "index" : "logstash-2018.11.14",
      "managed" : true,
      "policy" : "delete_180_days",
      "lifecycle_date_millis" : 1558472624125,
      "phase" : "new",
      "phase_time_millis" : 1559255956785,
      "action" : "complete",
      "action_time_millis" : 1559255956785,
      "step" : "complete",
      "step_time_millis" : 1559255956785
    }
  }
}

"phase_time_millis" : 1559255956785 = 5/30/2019, 6:39:16 PM

and the policy

{
  "delete_180_days" : {
    "version" : 1,
    "modified_date" : "2019-05-30T21:00:08.400Z",
    "policy" : {
      "phases" : {
        "delete" : {
          "min_age" : "191d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

I used "min_age" : "191d" because I wanted to go a bit back in time as I only have a small set of old test indexes to play with.

I had hoped to do this, but it seems to be a protected operation.

curl -X PUT "localhost:9200/logstash-2018.11.15/_settings" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "creation_date" : "1542258000"
    }
}
'

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[localhost][IP_ADDR:9300][indices:admin/settings/update]"}],"type":"illegal_argument_exception","reason":"unknown setting [index.creation_date] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"},"status":400}

Thanks!

2

Unfortunately ILM won't help you here as you correctly observed:

  • ILM is calculating from the index create time
  • re-index creates a new index create time
  • you can not manually change the index create time

I am curious if this re-index was one time thing, or part of a larger workflow ? If it was a one time thing, perhaps just manually delete. If it is part of a larger workflow, can you help us understand ... maybe ILM needs an enhancement here ?

1 Like
3

Thanks very much for the information!

This was a one time thing to migrate OLD date to a new Cluster and also add a small painless translation to the reindex to fix the 5.X to 6.8 _type issues.

I will setup a process to manually delete the old indexes.

Thanks so much for your help so I can stop chasing dead ends.
Cheers!

4

I would note here that it would be a good idea to ad some notes to both the ILM and reindex Docs how this will interact. Once people start using ILM in a time based manner, reindexing is going to confuse the issue (such as with shrink etc?).

5

I believe Curator is able to determine the index age based on the data in a date field within the index. It would be useful if this was also supported by ILM as it would make it easier to deal with existing indices and/or reindexed data.

2 Likes
6

Yes.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.