Index Lifecycle Management & Curator

[jong99]

I notice this morning with the release of 6.6.0 the introduction of Index Lifecycle Management which looks very interesting. Since this seems to be doing some of the same things that Elastic Curator already does, I was wondering what the relationship going forwards between ILM and Curator will be?

• Will Curator continue to provide additional functionality above ILM (with ILM being basically a "lite" version of Curator for those not needing the full functionality)?
• Will Curator be phased out over time, and support dropped to be replaced by ILM?
• Some combination of the above?

Just trying to understand whether we should be working on tighter integrations with Curator or thinking about moving over to ILM.

Thanks!

[shanec (Shane Connelly)]

You're right that ILM does some of the same things that Curator does.

The way to think of it is that Curator is kind of like a swiss army knife: it can do a ton of stuff: you can basically write a yaml that does a completely arbitrary order of actions. I suspect there will always be use cases for this kind of capability. There are advantages and disadvantages to this type of tool: on the advantage side, it can do most everything you'd want to be able to do. On the disadvantage side, the configuration can be long and sometimes confusing, it requires a separate running process, it doesn't have a UI, and it doesn't really encourage users (especially new ones) to use best practices. The virtually limitless configurability of Curator kind of works against those in some ways.

On the other hand, the way to think of ILM is that it's a built-in way to ensure most users are using best practices when dealing with time-series data. ILM knows things about the cluster and indexing that Curator doesn't and, as a result, can intervene in the "right" ways in ways that Curator can't. For example, because the shrink action requires the index to be marked in read-only and that shards should be on the same node, ILM can ensure those actions are done in the right order.

ILM is beta (as of 6.6), but we're quickly moving toward GAing it. I think ILM will be a great way for most users to manage their time-series data and I'd encourage people to use it when they can, since it helps prevent many of the administrative errors we've seen in production systems that try to do time-index management (with or without Curator). For those that need the extra configurability (or if you've got a working Curator setup that does everything you need), there's Curator.