ILM, creation_date and reindexing

Elastic Stack Elasticsearch
1

Hello Elastic people,
this is a follow-up to Is it possible change creation_date on reindexed Indexes to get Life cycle Management to work on old indexes

My usecase - so to say - is learning to use ES while trying to build a new log storage.
I am still struggling to get the mappings right, which leads to the need to reindex a lot.

I would now like to learn to use the built-in ILM feature to make a hot-cold setup,
where data moves to cold after 6 months. Unfortunately, in the current state of affairs,
it seems that I'll have to wait 6 months from last reindex to do that.

While I don't have an argument for doing reindexation as part of the final workflow,
I'd still like to plea for the feature, because it will remove a barrier to learning ES I am facing.

Thank you!

2

It seems that even Clone index API | Elasticsearch Guide [master] | Elastic resets creation_date instead of preserving it, so I must also not rename any indices for 6 months before I can learn ILM.

3

Why not try shorter time span for just testing purpose?

4

Thank you for replying.

Not a bad idea. True, I can mostly prototype the thing with shorter intervals.
So you are correct, lack of this feature is not actually preventing me from learning ILM.

It is just preventing me from going the easy "set up and forget" route. I won't be able to actually benefit from ILM until my indices stabilise enough so they won't need reindexing any more. Until that I'll have to move my indices manually. And actually it is even worse, because I recently clone-renamed about 50% of my indices withing few hours. If I leave ILM enabled, it will try to move all of them at once.

5

You can use the index setting index.lifecycle.origination_date to set a custom "age" for the index, so that if you were to reindex, you can tell ILM that it is a certain age without waiting for that long.

Here's a blog post about it: Control the phase transition timings in ILM using the origination date | Elastic Blog

And the reference documentation: Index lifecycle management settings in Elasticsearch | Elasticsearch Guide [8.0] | Elastic

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.